A Survey of the Prior Programming Experience of Undergraduate Computing and Engineering Students in Ireland

Abstract

It has become apparent that increasing numbers of students arriving into undergraduate computing and engineering degree programmes in Irish 3^rd-level institutions have prior experience of computer programming. As the extent of this prior exposure as well as its nature, origins, and usefulness is not known beyond anecdotal evidence, an annual survey of prior programming experience of freshman undergraduates who study programming as part of their degree has been designed and administered. This paper reports on the first two years of this survey in 2015 and 2016. It found that around one third had some prior experience of programming with nearly half of that group reporting a reasonable level of fluency in one or more languages. The authors expect that the effect of proposed changes to primary and 2^nd-level curricula alongside the increasing popularity of informal programming clubs will be increasingly felt in coming years and therefore plan to continue and extend the survey in order to clarify the effect of such developments. The results should be of interest to 3^rd-level educators in the planning of curriculum and teaching practice.

1 Introduction

In recent years, the academic community in Ireland has become aware that many students arriving at 3^rd-level to sit computer science and engineering degree programmes do so with some prior exposure to programming. This is despite the fact that programming is not taught as a formal subject in the Leaving Certificate cycle and has only been made available since 2014 as part of an optional Junior Certificate short course in Coding [1]¹. However, outside of formal educational settings, there has been a surge of interest in learning to program as evidenced by the phenomenal success of the CoderDojo network of after-school programming clubs which since its foundation in 2011 has grown exponentially to include over 200 clubs across Ireland in January 2017 [2]. The impact of such initiatives on the intake into 3^rd-level courses has yet to be assessed. The standard introductory programming course at 3^rd-level assumes no prior knowledge of programming but, in this changing environment, this assumption needs to be questioned. Furthermore, with recent announcements that the National Council for Curriculum and Assessment is to consider approaches to introducing the teaching of programming in primary schools as well as a proposed introduction of Computer Science as a Leaving Certificate subject in 2018 [3], it is clear that 3^rd-level institutions will need to adapt to cater for a new generation of students with prior programming experience. In this evolving context, the aim of the survey discussed here is to track the state of programming knowledge among freshman (first year undergraduates) and how this is changing over time thereby providing educators with an evidence-base for decisions about future curriculum design and teaching practice.

The structure of this paper is as follows: Sect. 2 reports related research while Sect. 3 gives a short overview of the research questions and the process used to collect data to answer these questions. Section 4 reports on the results gathered in the first two years of this survey and Sect. 5 concludes the paper with a discussion of the findings from this research, its future direction and the contribution it makes to the educational research and curriculum development for teaching programming.

2 Related Research

A review of related literature revealed that, while prior knowledge of programming among first year undergraduates has been investigated in previous studies, it has generally not been the primary focus of the research. Such studies have investigated the effects of prior knowledge on such things as confidence levels [4], problem solving [5] and gender balance [8] with previous programming experience often being cited as a predictor of programming success [6, 7]. Interestingly, studies have shown that it is not prior knowledge of programming concepts alone but student self-efficacy, or their self-perception of their programming ability that has the strongest correlation with programming performance [9, 10]. Krpan et al.’s investigation into the correlation between students’ success in introductory programming and mathematical courses included an analysis of student expectations and success rates in introductory programming courses at the Faculty of Science, University of Split over three years. They found that early failure to understand basic programming concepts affects students’ confidence and leads to increased drop-out rates [11].

The background of Computer Science freshmen across two Finnish universities, was investigated in 2011 and 2012 [12]. Differing levels of prior programming experience were found between the two institutions (62% and 38%), with the university with the more selective entrance criteria having the higher percentage. It is worth noting that between 43% and 63% of those with experience gained it through formal studies, an opportunity not currently available to most Irish undergraduates.

Data relating to the prior experience of over 900 students at the Eidgenössische Technische Hochschule (ETH) Zurich was collected over seven years from 2003 and gathered from 77 students from University of York in 2008 [13]. During this time Computer Science was an optional subject not universally available in either Switzerland or the UK at primary or 2^nd level, yet in both institutions, prior experience was high, running at a stable level of 78%–84% across the 7 years. Interestingly, only 16%–25% gained their experience at school, with most reporting self-study as the source. The authors note that generalisation of their results may be limited to other universities with similar admission regulations and students who studied in a comparable school system and recommend that research be broadened to more universities and countries. Indeed, no research on collating data relating to students’ prior knowledge of programming at a national or international level was found during our literature search and the survey discussed here is believed to be the first to attempt to do so on a national level.

3 Research Question and Data Collection

This research project aims to find out what programming knowledge and experience students have before commencing undergraduate degree programmes with a programming component in Irish 3^rd-level institutions. It also explores how this exposure affects their experience of programming during their freshman year and how this picture is changing over time.

To gain answers to these questions, it was decided to undertake a repeated cross-sectional study using an annual survey. This paper reports on the first two years of results from that activity. The survey population is freshman students undertaking a third level, undergraduate degree programme in Ireland which involves the studying of programming in first year. Such programmes are typically computer science and engineering programmes. The decision was made to target students in the latter part of the second semester of their freshman year so they could make a judgement with regard to the impact of their prior experience on their current studies. While the overwhelming majority of the survey population would have undergone their second-level education in Ireland, we note that as of the 2014/2015 academic year approximately 10% of students in Irish higher education are “international” students [14]. We expect that a number of these students would have experienced some level of formal exposure to programming in their previous education.

An online survey was deemed to be the most suitable data collection instrument as this enabled easy collection of standardised data from a geographically distributed sample. The benefits of this process were deemed to outweigh our concerns about any bias resulting from the self-reporting nature of the responses. Ethical approval was sought and granted by the appropriate ethical committee with Google Forms being the platform currently used by the researchers to administer and maintain the survey.

Rather than contact the target respondents directly it was decided that a higher response rate would be likely if they were recruited by teaching staff in their own institutions. Contact was made with appropriate personnel nationwide to ask for their collaboration in introducing the goal of the survey and forwarding it to their students. In return, the researchers furnish each participating institution with their raw student data as well as a copy of the annual report of analysed results.

In 2015, a pilot study was run between two 3^rd-level institutions in Dublin in order to validate the survey design and gather some baseline data. In this pilot, categories of questions were developed to understand the profile of the student population in terms of age, gender and programme of study; to ascertain if students had prior experience; to understand the languages learned with the level of fluency and finally to examine the impact of the experience on students’ current studies. The pilot survey was issued to students in both institutions across all 4 years of relevant programmes (students were also asked to identify their year of study as part of the survey, allowing us to make direct comparisons year-by-year). Based on responses to the pilot, the survey was subsequently updated to include two extra categories of questions which examined the source of the student experience as well as their reflections regarding the usefulness of the experience.

From 2016, invitations were issued to 3^rd-level institutions nationwide aimed exclusively at freshman students with 8 institutions agreeing to participate which resulted in a sample of n = 321 respondents. The aim of the researchers is to continue to grow the institution and participant levels on an annual basis so the generalisability of trends and patterns identified in the years going forward continues to increase. The current design of the survey has 6 categories of questions with a mix of closed multiple choice and open short answer questions producing 32 questions in total. Category 1 examines the profile of the population by asking respondents for personal information such as 3^rd-level institution attended, age, gender, and degree programme. Category 2 has a singular focus on asking if respondents have prior experience. For those with experience, there is a further 4 sections. Category 3 examines the nature of their experience in terms of the language(s) learned and the degree of fluency in the language(s). Category 4 examines the origins of their experience from a school, club, online and self-taught perspective. Category 5 examines aspects of their experience that students signal they found particularly helpful in growing their knowledge of programming and finally Category 6 examines how useful students find their prior experience to be in their current freshman studies.

3.1 Data Validity

The total population size is only available for two thirds of the group surveyed, which precludes calculating confidence for our overall results, a deficit that will be addressed in future surveys. Where accurate information on cohort size is available, from three institutions (n = 222, population = 651), we calculate a confidence interval of ±5.34, at a confidence level of 95%. Our overall results do not differ significantly from the results obtained (i.e., they lie within the confidence interval reported) when only considering this restricted set of the population, nevertheless we must be cautious in interpreting the overall results as a result of this weakness.

Some respondents chose not to indicate which institution they were attending, making it impossible to include them in these figures; a requirement of the study’s ethical approval was that no question be compulsory making such occasional gaps in the data inevitable. Where it has been possible to uniquely infer a student’s institution from their stated course of study we have done so.

With respect to other threats to validity, we note that having students self-assess their level of confidence and ability, rather than applying an objective test, may introduce some bias, though it is not clear in which direction this may lie.

4 Results and Analysis

4.1 Participants

As the 2015 participants included all 4 years of a programme, we restricted our analysis of the 2015 data to the 122 freshman students, to give a truer basis for comparison. These figures are laid out in Table 1:

Table 1. Participant details

While only around 25% of respondents are female across both years of the study, this is in fact a slight overrepresentation in the survey as enrolment figures for full-time honours degrees for 2014/2015 for females are a mere 14.9% for computing degrees and 18.1% for Engineering degrees [14]. The reason for the substantial jump in the mature student rate from 2015 to 2016 is unknown.

4.2 Programming Experience

The data of most interest was the percentage of students who self-identified as having programming experience. Of the 321 students who responded to the 2016 survey, 66% indicated that they had no prior exposure to programming with 34.2% indicating they had some experience prior to the start of their 3^rd-level course. These figures broadly reflect those from the 2015 pilot survey where 63% of freshman students (n = 120) indicated they had no prior experience. A Chi-squared test showed no significant difference between the two survey results (P = 0.938).

Students were also asked to self-report their prior level of fluency in programming. The 2016 cohort reported having a much stronger level of fluency than the 2015 respondents (see Fig. 1 for figures and an explanation of fluency levels).

Fig. 1.

Level of fluency from 2015 and 2016

A Chi-squared test was performed to confirm the significance (P = 0.0176). Reasons for the stronger reported fluency in the 2016 cohort is not apparent from the survey data but it will be interesting to see if this trend continues in the years ahead and to ascertain possible reasons for this increase.

Comparing these results to the results of the ETH Zurich study [13] where 78%–84% of respondents reported prior experience highlights the relatively low level of experience of Irish students at the outset of their 3^rd-level study. We believe this is most likely due to the majority of students coming directly from Irish second-level schools where programming is not yet widely taught. Equally interesting was the level of fluency reported which suggested that for the students who did have experience their previous studies were often deep and considered.

4.3 Programming Languages

Students who reported having some programming experience were then asked to indicate which languages they had some experience with. In 2015 this was presented as multiple choice list (with an “other” option), while in 2016 students were instead presented with an open question giving respondents freedom to indicate the range of languages they had experienced. The two years’ responses are presented in Table 2.

Table 2. Programming languages experienced prior to 3^rd-level

Students have clearly engaged with a wide variety of languages which is an indicator of the diversity and range of courses on offer. As an aside, the almost complete absence of the educational language “Scratch” in the 2016 results is striking. We would speculate that this is due to the change to the open-question format in the 2016 survey causing students to omit Scratch from their answers based on a perception of what constitutes a “real” programming language.

4.4 Helpfulness of Experience

Respondents were asked to report on how (or not) their prior programming experience helped them in their freshman year of studying programming. The question was presented as an open-question prompting a varied response. Interestingly, 15% of respondents in 2015 and 10% in 2016 indicated that their prior experience didn’t help at all. The analysis of the responses given by the remaining respondents who did report an impact produced the following broad categories:

• A head start boosted confidence helping students transition to 3^rd-level.

• Not struggling with the basic concepts allowed for deeper learning to occur.

• Understanding fundamental concepts helped in adapting to new languages.

• More receptive to understanding complex concepts.

• More time could be spent on studying other modules.

As we might expect, prior exposure to programming concepts is considered helpful when dealing with introductory programming modules. Interestingly, the first group that emerged doesn’t relate directly to the conceptual knowledge and skills gained from prior experience but instead focuses on the affective impact of having such knowledge particularly in the area of confidence levels and receptiveness to new learning. This underlines the importance of educators adopting pedagogical approaches that support the growth of confidence in learners. Given the diversity of programming tools and skills encountered, it’s perhaps not surprising that familiarity with the core concepts of programming had a significant positive impact. Since many introductory programming modules focus on trying to impart these core skills, students who have a head start reported that they can focus purely on advanced constructs or even on other modules.

In 2016 only, respondents were asked, again using an open question, which parts of their prior experience were the most useful to them overall. 35% reported that nothing helped them. This could mean that they either didn’t find their experience useful or they were unable to articulate or remember any particular aspects they found useful. The remaining respondents’ responses were grouped into the categories listed in Fig. 2.

Fig. 2.

Sources of useful prior experience - 2016 (n = 80)

Over a quarter of respondents indicated that while their experience was useful, nothing specific stood out as having helped them. With the remaining responses, the most frequently cited helpful activities were time spent on projects or practice (19%) and specific online or print resources (15%) notably Codecademy and YouTube.

The fact that a substantial number of participants were unwilling (or unable) to identify individual sources of their experience as being helpful could indicate that nothing at all, or no single specific aspect was helpful. However, alternative readings could suggest that students either had difficulty in reflecting on its effect or that their overall experience - rather than specific experiences - was helpful. This question will need careful redesign in future iterations to assist students answer more accurately without biasing and guiding their responses.

4.5 Origins of Experience

In 2016, participants with prior experience were asked a series of questions to gain insight into its origins. These were categorized into school, club, online and self-directed learning with respondents also having an opportunity to indicate any other sources of their experience. This is a particularly interesting question in the Irish context where programming has not been widely available as a formal school subject and grassroots organisations like CoderDojo report considerable activity. Most respondents reported multiple sources for their experience.

School Activities:

27% indicated that they had participated in programming activities through school. These activities were identified as short summer classes, transition year courses (a one year programme in Irish second-level schools with no formal exams prior to beginning the two year terminal examination), elective classes, preparation for programming competitions, and as a formal part of school curriculum (international students). The duration of these activities ranged from 1 day to 2 years.

Club Activities:

13% of respondents reported participation in clubs or groups outside of school. Of those, 36% had attended CoderDojo; 27% were involved with other computer clubs; 19% were involved with 3^rd-level access and youth programmes with the remaining 18% citing adult education courses.

Online Activities:

Reported participation in online courses (including courses that they may not have finished) was 42%. The most common sites used were Codecademy (46%), non-specified tutorial sites (27%) and YouTube (7%) with other specified sites each representing just over 2%.

Self-directed Learning:

When asked if they had engaged with any self-directed learning such as books or building projects, 63% answered in the affirmative with the majority of those citing projects that they had designed and developed. These included games (41%), websites (22%), web applications (17%), Arduino/Raspberry Pi projects (17%) and general business applications (3%). The time spent on these projects varied significantly from a couple of hours to a couple of years with the average amount reported as approximately 40 h.

Other Sources of Experience:

30% of respondents indicated they had other sources of experience. These included post-2^nd-level school courses that were not diplomas or degrees (43%), previous 3^rd-level courses (23%), previous employment (13%) with the remaining 21% not specified.

When examining the origins of prior experience, it was not unexpected, given the lack of formal inclusion of programming in Irish schools’ curricula, that a high number (73%) of respondents gained their experience outside of school. Indeed, what was surprising was the extent and range of extra courses run by some schools both as part of their day-to-day curriculum and as extra short courses after school. The percentage of 13% of students participating in clubs/groups would appear to be low when compared to the activity that is school based (27%). This is despite the high profile of CoderDojo but may well be due to the relative infancy of this club, founded in 2011, and we would expect this number to increase in the future in line with the increasing participation and growth of dojos in Ireland. However, this is only speculative analysis and data gathered in the years going forward will allow more concrete patterns of participation in clubs to emerge. At present it is clear that most non-online computing activities undertaken by students are via school.

Not unexpectedly, there is a relatively high number of students undertaking online courses with Codecademy being the clear leader. The most popular languages being learned online are the web development scripting languages and Python. While there is no indicator in the survey regarding the reasons why students chose those languages, having an understanding of popular online programming paradigms could provide assistance both in understanding prior experience and also in the generation of online courses for distance education at 3^rd-level.

5 Discussion

When examining the profile of freshman students in any discipline, there is always diversity in terms of age, background and general experience. This is a factor that 3^rd-level educators have always had to take into consideration when designing courses and choosing appropriate pedagogical approaches. With computer science primary degrees having the highest rate of non-progression in Ireland (varying between universities (15%) and institutes of technology (26%) [15]) it is clearly important that the nature and impact of that previous experience be more fully understood so that introductory courses be made more relevant and appealing to students.

While Ireland has not as yet introduced formal computer science or programming as mainstream subjects in schools, it has become apparent to educators that a growing number of students are already presenting in Irish 3^rd-level institutions with prior programming experience. From the results of the survey presented in this paper, it can be seen that roughly a third of those surveyed have had some exposure to programming. To date, a lot of that experience is, necessarily, from self-directed and online study but a surprising amount stems from schools which emerged as an important driver and facilitator for encouraging students into their first steps in programming. Non-school clubs such as CoderDojo have grown in numbers and popularity since their foundation in 2011, however their effect in Ireland cannot be seen as yet in the 3^rd-level population. Together with the mooted introduction of programming through the new mathematics curriculum at primary level and the introduction of computer science as a Leaving Certificate subject from September 2018, it is clear that the nature and extent of the prior programming experience of incoming undergraduates will be subject to enormous change over the coming decade.

The early results of this study, while limited in their scope, can provide an indication of the current situation as well as a baseline from which to evaluate the inevitable further changes in our student profile. The authors will continue to expand the reach of this survey and administer it annually in order to track these changes and see how they impact student interest and experience in programming going forward. Given the shortage of comparable national studies from other countries, it is difficult to compare these results against international standards. However, given the growing interest in computer science and programming at school level worldwide and the resulting increase in the skills and knowledge of incoming 3^rd-level students, it is expected that research and interest in this area will grow. We would argue that this survey will lead to a better understanding of student experience prior to their arrival at 3^rd-level and help to provide an evidence-base for decisions about future curriculum design and teaching practice, consequently providing some contribution to student retention and competency levels.