Comparing the Efficiency of Different Approaches to Teach Informatics at Secondary Schools

Each of the 16 federal states of Germany has its own school system and also its own policy to integrate informatics, computer science or ICT into this system. Till present there aren't any tests of students' knowledge on a nationwide level. Therefore nationwide or international contests currently offer the only opportunities to compare the knowledge of the participating students on a large scale level. By evaluating the overall performance of the students of different federal states, we were able to compare the effectiveness of different approaches of teaching Informatics at secondary schools.


Introduction
In Germany the school systems, the subjects and the curricula are very different from state to state.At the early 70ies of the last century, nearly all federal states of Western Germany (BRD) started to integrate informatics lessons into the curricula of their secondary schools, especially at the most demanding type of these, called Gymnasium.Most of them restricted informatics to eligible courses in the highest grades 11-13, e.g., the state with the largest population, North Rhine-Westphalia (NW).After the German Reunification in 1990, most of the eastern German states simply copied an education system of one of the western states, like Brandenburg (Br) did.
Consequently, at the beginning of the new millennium, there was no German state (except Saxonia) that had installed a regular subject of informatics before grade 11.At this time, the state of Bavaria (Ba), the second largest state regarding population, decided to introduce a regular, compulsory subject of informatics into the time table of its 405 Gymnasiums, attended by about 370.000 students currently (Hubwieser and Müller, 2000).The new subject should start with grade 6 in autumn 2004.
The new subject of Informatics in Bavaria has three stages: in grade 6 and 7, all students of the Gymnasium, currently about 45.000 in each grade, have to attend one compulsory lesson per week; in grade 9 and 10, there are 2 lessons per week, compulsory for all students that have chosen natural science/technology as their direction of study (currently about 17.000 per grade); in grade 11 and 12 the students can choose an eligible Informatics course of 3 lessons per week.
The decision of the government was the consequence of the persistent lobbying of the Faculty of Informatics of the TU Munich, which had made its first proposals concerning the conception of such a subject already in 1996 (Hubwieser et al., 1996).The content and the structure of the curriculum follow more or less the proposals we have made (see Breier and Hubwieser, 2002;Hubwieser, 2003Hubwieser, , 2004Hubwieser, , 2006Hubwieser, , 2007)).Additionally we have developed a series of textbooks for the subject that has been introduced by about 50% of the schools (Hubwieser, 2008).
Meanwhile (in summer 2010), the new subject has reached grade 11, thus it seems to be high time to evaluate how it is working.In order to evaluate the notion and opinion of the teachers towards the subject, we have conducted a large scale teacher survey in December 2009 (Mühling et al., 2010).Concerning the performance of the students, there are no nation-wide performance tests in informatics until now.Thus nation-wide competitions like the Bundeswettbewerb Informatik or the Informatik-Biber (Bundeswettbewerb Informatik, 2010) represent the only opportunities to compare the knowledge of the students of different states.
In this paper we present the results of an investigation on the data of the Informatik-Biber contest of autumn 2007, which will show that the Bavarian students are performing better than those of comparable other states, as far as they have attended the new subject.

The Bebras Contest
The annual Informatik-Biber competition is the German issue of the international Bebras Contest (see Dagiene, 2008Dagiene, , 2010;;Dagiene andFutschek, 2008, 2009).It is performed in all German states and in all types of secondary schools in three different age levels: level 1 for grade 5-7 (in Germany the students of this level are 10 to 13 years old generally), level 2 for grade 8-10 (13 to 15 years) and grade 3 for grade 11-13 (15 to 19 years).
Officially the students don't need to have any obligatory prerequisite knowledge for the competition.Every participant had to answer 15 multiple choice questions in three degrees of difficulty: Five easy, five average and five difficult.In total the test is containing 30 different questions for each of the 3 age groups listed above.Every age group has to answer a different set of questions.Some items are the same for different age groups, but with different degrees of difficulty.
We examined the 30 questions of the Informatik-Biber 2007 concerning the overlapping of its topics with the Bavarian curriculum.It turned out that 9 out of the 30 test items were covered surely in the preceding lectures, 10 eventually and 11 hardly.Nevertheless contests like the Informatik-Biber allow students to use their knowledge gained in lessons in informatics (Dagiene, 2010).
The participants start with 45 points.Any correct answer adds 6 points, when it is one of the easy questions, 9 points for average questions, 12 points for difficult questions, any incorrect answer subtracts 2 resp.3 resp.4 points.Not answering at all is rated with 0 points.By this way the participants may score from a maximum of 180 points down to a minimum of 0 points.
The competition is open to all types of secondary schools.Nevertheless, we restricted this evaluation to the Gymnasium, because all other types either don't offer any systematic courses in informatics or are not comparable over different states regarding their structure.

The Situation in Autumn 2007
Because of the huge number of participants and the variety of subjects and curricula, we decided to restrict the evaluation to three states.We chose Bavaria (Ba) and NW, firstly because these are the two largest states of Germany and had the most participants in the contest, secondly because of their very different approaches concerning the integration of informatics in secondary schools.Additionally we chose Brandenburg (Br), because it has many participants on the one hand and as its school system is very similar to NW on the other hand.
As explained above, the Bavarian Gymnasium has established a compulsory subject of informatics from grade 6 to 10 and eligible courses in grade 11 and 12.At the time of the evaluated Informatik-Biber 2007 the subject had just started in grade 9. Thus the students of the age levels 5-7 and 8-10 had partly attended a regular subject of informatics (see "Schulordnung für die Gymnasien in Bayern (Gymnasialschulordnung -GSO)," 2010b; "Schulordnung für die Gymnasien in Bayern (Gymnasialschulordnung -GSO)," 2010a).We roughly estimate that about 66% of the age group 5-7 and 33% of the age group 8-10 did so, while the participants of the 3rd age level (grade 11-13) did not have any systematic lessons in informatics.
In North Rhine-Westphalia informatics is not a regular subject, but in the grades 7 to 9 (with focus on grade 8) informatics issues are integrated into the curricula of mathematics and other subjects in the extent of one hour per week.In grade 9 to 10 there are optional courses in informatics, whereas in 11 to 13 eligible advanced courses can be chosen with two or even more hours per week (Weeger, 2007).
In Brandenburg the situation is quite similar to North Rhine-Westphalia.Thus we can estimate that in North Rhine-Westphalia and Brandenburg, the participants of the first two age levels (5-7 and 8-10) did have education in informatics integrated in other subjects like mathematics, while a (unknown) part of the 3rd age group had attended advanced courses in informatics (Weeger, 2007).

Evaluating the Scores
The results of the international Bebras contest were available in a database table.We exported the test scores from the German section by using SQL statements, and analyzed them using a spreadsheet.Unfortunately the database table did not contain any information about the type of school the participants attended.In order to identify the participants as students of a Gymnasium, the name of the school was analyzed.The results are shown in Table 1.The number of participants from the Gymnasiums shown there should be the lower limit of the real number, because not every school name of a Gymnasium contains this term explicitly.
From the extracted test scores we calculated the sample mean and the 5% confidence interval, as plotted in Figs.1-4.

Numbers of Participants
The numbers of participants were varying strongly over the states.For some states the number was even too low to compare them validly with the other states (Table 1).
Column 6 in Table 1 shows the proportion of participants from Gymnasiums of a certain state compared to the number of participants from Gymnasiums over all states, while column 7 shows the proportion of the students (over all school types) of a state compared to the students of all Germany.By comparing these two proportions we can estimate, how well a state is represented in the contest.This shows that, e.g., Bavaria is strongly overrepresented, while, e.g., Baden-Wurttemberg is strongly underrepresented.Of course this is only a first, rough approximation, because we supposed, that the proportion of students  that attend a Gymnasium is nearly the same over the states (Bildungsberichterstattung Autorengruppe, 2010).

Comparison of Test Scores
As shown in Table 2 and Fig. 1 the sample mean over all three age groups for all participants is 95.9 +/-0.5, for participants from Bavaria 98.1 +/-0.8, for participants from North Rhine-Westphalia at 99.6 +/-1.1 and for participants from Brandenburg at 89.7 +/-1.4.Both the participants from North Rhine-Westphalia and Bavaria are significantly better than the average of all high school students and significantly better than the average of the students from Brandenburg.The difference between the Bavarian and North Rhine-Westphalia's high school students is not significant.
The picture is different, if we consider the three age groups separately.In grade 5-7 the means of the results from North Rhine-Westphalia and Brandenburg don't differ significantly from the overall mean, while the Bavarian results are significantly better (Fig. 2).
In grade 8-10 the sample mean of Bavaria and North Rhine-Westphalia is significantly higher than the overall result, with the Bavarian participants (although not significantly) better than the North Rhine-Westphalia's.Remarkable is the poor performance of the Brandenburg students (Fig. 3).
In grade 11-13 the situation is different again.The sample mean of Bavaria doesn't differ significantly from the overall result, while the sample mean of North Rhine-Westphalia is significantly better.The participants from Brandenburg scored significantly worse.Particular attention hast to be directed to the fact that from Bavaria only 177 students participated in this age group, while from North Rhine-Westphalia there were 1014, from Brandenburg 888 and in total 3963 (Fig. 4).
While in North Rhine-Westphalia the age groups 8-10 and 11-13 are causing the above-average performance, the Bavarian results arise from the grades 5-7 and 8-10.Brandenburg is only within 5-7 on the same level with the overall results, in 8-10 and 11-13 the sample means were significantly worse.

Interpretation
The strong performance of Bavarian students in the age groups 5-7 and 8-10 might be interpreted as an effect of the introduction of the compulsory subject informatics.Regular and autonomous lessons thus seem to result in better contest scores than integrating informatics issues into other curricula.
The strong performance of North Rhine-Westphalia in 8-10 and 11-13 are presumably a result of an effective education in informatics by elective, basic and advanced courses in informatics.
It is difficulty to argue with the differences of the curricula themselves because other influences might be found in external conditions for the implementation of the Informatik-Biber in schools such as improved IT facilities or more or less assistance in preparing the test, which were not gathered by the contest.Informatics as an autonomous subject has surly more weight and possibilities to initiate supporting settings for learning and doing informatics in schools.

Conclusion
In order to investigate the reasons for the differences in performance further research is required.Nevertheless it seems plausible, that students in those groups perform significantly better, where the newly introduced compulsory and regular subject informatics took place.This is a strong argument supporting regular lessons in computer science as a subject of its own instead of integrating informatics topics into other subjects.
Additionally it will be thrilling to evaluate the results of Informatik-Biber 2008 and 2009, as in Bavaria, the compulsory education in computer science reached grade 10 respectively grade 11 at this time.We have just acquired the data of these tests in order to compare the results.
We have to thank Wolfgang Pohl from the Bundeswettbewerb Informatik, Kirsten Schlüter from the University of Erlangen-Nuremberg and the team of the Informatik-Biber in Germany for disclosing of test results.