PREDICTING PROGRAMMER QUOTIENT

Computing education necessitates programming skill. But it varies in students. How can we quantify or measure the skill? We are yet to have a standardized measurement system for the programming ability. The concept of Programmer Quotient (PQ), which gives a measure of innate programming ability, attributes a value to one’s ability to program, just like Intelligence Quotient (IQ). This would remain the same independent of the programming experience. In this paper, we consider few inherent skills such as Analytical ability, ability to synthesize etc. and try to correlate these skills to one’s programming ability. A questionnaire was designed and used to measure the skill in these areas. Then a model was designed from the data collected. It can predict the programming skills of a student from his/her inherent skills, irrespective of the programming language. Keywords—Programming ability, Programmer Quotient PQ, Programming skill, Analytical ability.


INTRODUCTION
THE primary purpose of computing education is to impart to students a set of skills required to become a good programmer.But both teaching and learning of programming is considered to be really challenging by many [1].To learn programming, it requires correct understanding of the abstract concepts.Nature of the subject makes it difficult for many to learn it.For teaching, the size and heterogeneous nature of the student groups makes it difficult to design instruction, in such a way that it would be beneficial for everyone.[3] Students find it hard to acquire the logic of computer programming.Many research works at the end of Introductory Programming Course (IPC) have shown that the rate of failure in these programming courses is relatively high [4] [5] [7].The high dropout and failure rates in IPCs are a universal problem that motivated many researchers to propose methodologies and tools to help students.
Another perception of the difficulty faced by students in programming is that not all students have the ability or the aptitude to program well [6].This group argues that if we can measure this aptitude before admitting the students to a programming course, the failure rate and the dropout rates can be significantly reduced.Programming-related skills refer to the ability of a person in operating a computer to perform a task or multiple tasks.Handling a computer task with certain level of complexity often requires computer programming skills.[2] This paper is based on the hypothesis that the programming ability is correlated with other measurable inherent skills or aptitudes of a person.But a test for measuring programming aptitude and identifying students who have a natural aptitude for programming has not yet been standardized.
Mcgettricket.al, introduced a concept of Programmer Quotient (PQ) to measure the programming ability [1].The proposal was to develop the concept of PQ that would remain the same irrespective of the programming experience.

RELATED WORK
Research works in this field of computing education can be broadly classified into two.In the first type, the researchers have attempted to improve the methods of teaching programming.Several methods and tools have been developed to assist the students to grab the concepts of programming.And the second category focuses on identifying what differentiates successful programmers from nonsuccessful programmers.Some of the notable works are discussed below.

A. Improvement in Methodology
A great number of works deals with how the teaching methods or the delivery of the contents can be modified or improved so that the students are able to grasp the underlying concepts of programming.Simplified programming languages, environments and simulating software have been proposed to support teaching and self-learning different programming techniques.
A prominent work in early days suggests that visual way of learning programs can effectively improve the understanding.[7] In their paper, B. Kaučič et al. [16] has discussed a method to improve the teaching methodology for programming so that the success rate can be improved.They suggest that, a new generation of programming tools and activities can help to overcome the shortcomings of previous initiatives, and thus make computer programming more accessible to everyone and suitable for youngsters.
In their paper M.McCracken et al [5] conducted a multinational study of the assessment of programming skills in first year CS students.They found that the problems observed with programming skills seem to be independent of country and educational system and that the most difficult part for students seemed to be abstracting the problem to be solved from the exercise description Janna Holovikiet.al found in their work that students should have certain basic abilities like mental schemas in regard to procedural thinking and action, functional understanding of processes, and a comfortable relationship to technological artifacts when they start learning programming [9].

B. Factors Affecting Programming Ability
In Predictors of Success in a First Programming Course, [11] Simon et al, studied the relationship of programming ability with that of one's spatial visualization, behavioral aspects and Attitude.It has been found that, the behavioral aspects like searching a phone book has clear association with one's ability to program well.The learning approach also has a positive correlation on the effectiveness.
In two related papers Dehnadi, Richard et al. reported that the students can be classified based on the mental models they used to solve the questions: i) Consistent and ii) Inconsistent [6] [10].Consistent students use the same model to solve all questions while the inconsistent group used different models for different questions.They found that consistent students tend to perform slightly better than the inconsistent students and they suggested that students can be divided into programming sheep and non-programming goats [6].But in paper [10] they confirmed that this test doesn't work if the intake is already experienced.The paper [6] claimed that with the test used one can divide the students into 2 categories: the programming-sheep and the non-programming goats.
But recently, the above work was retracted by the coauthor Richard Bornet [12].While he accepts it to be true that novices who answer 'consistently' in the test are more likely to pass a programming course, he retract his claims that Dehnadi discovered a correct way to measure the programming skills and students can be divided into programming sheep and non-programming goats.
Kurland et al. [7] in a pioneer work on the development of programming skills in high school students and found that Math and reasoning abilities have direct correlation with the programming skills.Simon et al, [11] studied the relationship of programming ability with that of one's spatial visualization, behavioral aspects and attitude.They found that there are several factors positively correlate with one's programming ability.And they are logical thinking, problem solving attention to detail, consideration of alternatives, mathematics, knowledge of programming, ability to learn, knowledge of computers, modularizing, and planning.Jens Bennedsen et.al [13] claimsthat general abstraction ability has a positive impact on performance in computing science, but they did not find a correlation between abstraction ability and overall performance in learning computing science.In their work [14], Renumol et.al, has identified a set of 8 cognitive processes through which a student may go through while programming.In a similar work,[15] Ambrosio, et.al, has pointed that abstraction and command sequencing are the 2 cognitive abilities which are identified as key to explain the difficulties the students encounter when learning to program.Underlying the abstraction difficulties are the student's problem solving abilities.Problem solving is composed of Analysis and Synthesis.
Some of the existing papers on methods to test the programming skills are described by David Clark [19] and SaeedDehnadi [17].They have tried the effectiveness of testing the programming skills using MCQs from the programming knowledge.David Clark in his work [19], assessed students' programming skills with MCQs from 4 levels of cognitive skills namely, Knowledge, Comprehension, Application, and Problem-solving.It was found that comprehension, application and analysis questions were good discriminators for the students' skill, but Knowledge questions were not significant discriminators.In [17], Dehnadi tried to give an objective interpretation of the results from previous experiments using mental models for predicting success in programming course.
In their paper, 'Programming aptitude testing as a prediction of learning to program' [18], MarkkuTukiainen et.al.devised a Programming Aptitude Test (PAT) with three parts: Number series, figure analogies, and arithmetic problems.Their test was considered to predict the floor limit of performance in the first programming course for those who had prior knowledge of programming concepts.In his thesis work, Otto Seppala researched about the Advances in assessment of programming skills [20], how to improve existing assessment techniques and tools so that we can more accurately assess students' programming skills and give them better feedback.He used visual feedback, reading and tracing of the code to improve the programming skills of students.
As these works suggested there may be few factors which distinguish the successful programmers from others.The paper is a preliminary effort to identify some of these factors which contribute to PQ.The objective of this work is to create a model to predict the programming ability of a student by assessing his/her cognitive skills.The resultant model is expected to predict the programming ability with a certain level of accuracy

OVERVIEW OF THE SYSTEM
The programming ability of a person might depend on numerous factors ranging from one's gender or the type of education one has had.In this project we concentrate on various cognitive and aptitude skills of the students and their effect in their PQ such as analytical ability, ability to synthesize, inductive and deductive reasoning, translation, decision making and problem solving skills.The effects of gender, family background, emotional condition or any other similar aspects on one's PQ are not considered.
In this project we have identified few cognitive and aptitude abilities for our experiment.Some of these identified areas are part of Bloom's Taxonomy of learning objectives.Some are on the basis of our experience, assumptions and also some exploratory options.We need to verify if a correlation exists between ability in these areas and programming ability A. Phase I: Designing the Questionnaire For the modeling of the predictor, we have identified the following set of factors.Once the independent factors were identified for the model, the questions for assessing each of these abilities were identified.Datasets of each of these areas consisting of 50 questions were created.The questions were identified from text books written for the purpose of measuring these skills and also from the web.
With enough number of questions in place, the required questionnaire was designed.The questionnaires consists of five questions each from Analytical, Synthesis, Induction, Deduction, Translation, Decision Making and one question from Problem Solving.This is because; the question in the problem solving section is descriptive type while others are multiple choice questions.Problem solving questions involves giving a situation to a student and asking him/her to write down the steps that would be taken to tackle the situation effectively.The answer given by a student is evaluated using the guidelines from the method shown byNational Center for Research on Evaluation, Standards, and Student Testing (CRESST) [26] The questionnaire is then distributed among different sets of students -the known sample.These groups of students formed the training groups, about whose programming ability is already known.Thus their programming ability can be quantified by a value, which we call the programmer quotient.The questionnaire was distributed among several training groups belonging to undergraduate engineering programs in Information Technology and Electronics & Communication Departments.

B. Designing theModel
The questionnaires answered by the students with different PQs are evaluated and the results are analyzed.Then we tried to correlate or map the aptitude abilities to the programming ability.Analysis of the questionnaire evaluation gives us a general trend of the factors that might contribute to the programming ability of students.
By analyzing the data from the questionnaires, a model for Programmer quotientis built using statistical data analysis tools.We assume PQ to be a variable dependent on several independent variables.Thus the subjective assessment is translated to an objective assessment, which gives a quantitative value for the programmer quotient.The model thus predicts the PQ of a student, based on the assessment of the questionnaire answers.
This model is then tested on other student groups.The test groupswere given the questionnaire and their answers were evaluated.The data is then put to the model and the model predicts their PQ.

RESULTS
Different sets of questionnaires were created and distributed among known sample of students and the answers were analyzed.Multiple regression analysis was done on the data collected to develop the model.

A. Method
Data was collected from a known sample of nearly 100 students.The students were from undergraduate programmes of Information Technology and Electronics & Communication in School of Engineering, Cochin University of Science & Technology, India.
Questionnaires were distributed among the students and theirprogramming ability was known from their practical work, viva voce and usage of algorithms/logics in solving various problems assigned in their practical sessions.Inputs regarding their programming ability were also taken from the teaching and lab faculty, to arrive at the final Programmer Quotient assigned for the students.
The data was then stored in MS Excel.Then using the multiple regression analysis technique, a model was generated.For each sample of students, the model was generated and also, the model was generated for each of the variable.
The further details of the regression analysis are given below in the table.The value of R 2 determines the predictive ability of the model.It tells us how much of the variance of "Y" we have explained in the regression.The closer R 2 is to 1, the better the model and the prediction are.Adjusted R2 is the percentage measurement of goodness of fit.The higher this value, the more accurate the equation is.Here, the value of R 2 is 0.74, which indicates that the model has a reasonably good predictive capability.The standard error is an estimate of the standard deviation of the coefficient, the amount it varies across cases.It can be thought of as a measure of the precision with which the regression coefficient is measured.Here the SE of 1 makes the equation acceptable.

Multiple
Another value is the P-value, which is the significance factor.These values provide the likelihood that they are real results and did not occur by chance.The lower the P-Value, the higher the likelihood that the coefficient is valid.For example, a P-Value of 0.016 for a regression coefficient indicates that there is only a 1.6% chance that the result occurred only as a result of chance.With satisfying values of R 2 and p, the model thus was tested among a sample of 10 students.The PQ predicted by the model was found to be a good prediction of their programming ability.

CONCLUSION AND FUTURE WORK
The experiments conducted with a sample of about 100 students over 7 factors showed promising results.Analytical ability and deduction ability could very well be concluded as two main factors which may contribute to one's programming ability.And we could also assume that some of these factors like Induction and Translation may not be clear distinguishing factors affecting the programming ability of the student.
The model has many real-world applications.The model can be used at the admission test level to screen the students admitted to the CS & IT courses.This will ensure that only students with the programming capability join the course and thus will reduce the failure rate.It will also help in improving the morale of the students.At industry level, the model can be used to identify employees with great programming skills.They can thus be allotted to projects requiring good deal of programming skill and vice versa.
The evaluation and analysis of the method developed suggest interesting directions for future work.Our model met almost all the objectives that we aimed, to a particular extend.Next step is to concentrate on how to improve the accuracy as a future study.The work could be enhanced by considering more factors, which are considered to be affecting the programming ability.This could include various other factors like Abstraction ability, Critical thinking etc.Including some other factors like gender could also provide an interesting insight to the future work.