The Representativeness of Reading Construct in Iran's University Entrance Exam

The study aimed to analyze the reading comprehension questions of Iran's University Entrance Exams to see whether they appropriately reflect the range of constructs delineated in the pre-university English textbooks. To this end, the reading comprehension questions of university entrance exams were categorized on the basis of two TOEFL books. Meanwhile, the researcher used the teacher's manual, the student book introduction and the nationwide test specification to enumerate the emphasized objectives, strategies and skills. The frequency and percentage of each question type were calculated and the comparison between the resultant frequencies was made by the statistical procedure of Chi-Square. The results showed an unbalanced distribution among question types and, accordingly, the underrepresentation of the reading constructs. There should be more weight given to the reading comprehension part of the test, multiple measures of reading comprehension, a variety of reading passages, a detailed test specification and balanced distribution of question types. Introduction Assessment of reading comprehension, according to Paris (2007), has been a controversial issue due to the summative nature of high-stake tests for accountability and comparability and the complex interplay between many factors across texts, instruction, and response formats. The results are instruments which are susceptible to persistent complaints concerning validity, reliability and so on. To treat test validity, one can begin with an attempt at defining a construct of reading comprehension for a given purpose, or a given setting. The abstract notion of reading ability-a theory of readingis operationalized in the form of test tasks and on the basis of test specification. The validity of tests, according to Messick (1996), is affected by an inadequate or incomplete sampling of the construct (constructunder representation) and the measurement of things that are simply not relevant to our construct (construct-irrelevant variance). The assessment of reading comprehension in Iran’s university entrance exam is a part of the whole test. The test requires the candidates to read (silently and without assistance) a/ some short passage (s) and to answer a number of multiplechoice questions about implicit and explicit information in the text (s). The method is based on (a) quantitative models of normative skills used to sort students on the basis of uniform scores, (b) psychometric models with samples of text and questions in item pools that yield high reliable and validity across items and test forms, and (c) economically efficient models of standardized testing with group administration and computerized scoring. The question types in reading comprehension section of the test are meant to represent the underlying processes that readers undergo and to have the candidates to draw on a number of skills or constructs. This view of reading comprehension based on the divisibility of the construct into some skills, abilities and strategies has been adopted by a number of researchers ( Davies (1944); Harris (1948); Derrik (1953); Holmes (1950); Hunt (1957); Lenon (1962); Schreiner, Hieronymus, and Forsyth (1971); Davies (1972); Rost (1993); Carroll (1993); Weir (1994), all cited in Pearson (2005) & Alderson (2000). In this paper, a report of the investigation of reading assessment in a period of 5 years is presented to see whether the constructs, delineated in pre-university coursebook, are represented and relevant. To do this, a survey of the following is conducted: 1. Themes related to the purposes and constructs of assessment 2. The number of items assigned to particular constructs Research Questions With reference to reading constructs, skills and tests, one should take the following into account while constructing high stake tests like university entrance exams: the range of constructs, test method effects, construct under-representation and constructirrelevant variance. To my knowledge, the reading comprehension questions of university entrance exams administered in Iran have not been checked and analyzed to see whether they appropriately reflect the underlying constructs delineated in the pre-university English course book. To address this gap, the present paper aims to analyze reading comprehension questions, the starting point of which marks the introduction of the course book in Iran’s Preuniversity centers. That is to say, the researcher has chosen to investigate the representativeness of reading construct in the mentioned tests by checking if the content of tests adequately reflects the breadth and depth of reading constructs delineated in the pre-university course book. More specifically, the present study aims at answering the following questions: 1. Do the question types in the reading comprehension section of the test reflect the range of constructs focused on in the pre-university course book? 2. Is the frequency of each question type appropriate for the relevant reading construct delineated in the pre-university course book? Methodology In order to answer the research questions posed earlier, the constructs of reading focused on in the introduction of preuniversity English course book and the relevant teacher’s manual were retrieved and discussed. Then, on the basis of the guidelines offered in two TOEFL books and some other resources, the types of questions in the reading comprehension sections of universityentrance exams were identified and tallied. The resultant frequencies for the question types across the period of five years were calculated and added.


Introduction
Assessment of reading comprehension, according to Paris (2007), has been a controversial issue due to the summative nature of high-stake tests for accountability and comparability and the complex interplay between many factors across texts, instruction, and response formats. The results are instruments which are susceptible to persistent complaints concerning validity, reliability and so on.
To treat test validity, one can begin with an attempt at defining a construct of reading comprehension for a given purpose, or a given setting. The abstract notion of reading ability-a theory of reading-is operationalized in the form of test tasks and on the basis of test specification. The validity of tests, according to Messick (1996), is affected by an inadequate or incomplete sampling of the construct (constructunder representation) and the measurement of things that are simply not relevant to our construct (construct-irrelevant variance).
The assessment of reading comprehension in Iran's university entrance exam is a part of the whole test. The test requires the candidates to read (silently and without assistance) a/ some short passage (s) and to answer a number of multiplechoice questions about implicit and explicit information in the text (s). The method is based on (a) quantitative models of normative skills used to sort students on the basis of uniform scores, (b) psychometric models with samples of text and questions in item pools that yield high reliable and validity across items and test forms, and (c) economically efficient models of standardized testing with group administration and computerized scoring. The question types in reading comprehension section of the test are meant to represent the underlying processes that readers undergo and to have the candidates to draw on a number of skills or constructs. This view of reading comprehension based on the divisibility of the construct into some skills, abilities and strategies has been adopted by a number of researchers ( Davies (1944);Harris (1948);Derrik (1953);Holmes (1950);Hunt (1957); Lenon (1962);Schreiner, Hieronymus, and Forsyth (1971); Davies (1972);Rost (1993);Carroll (1993);Weir (1994), all cited in Pearson (2005) & Alderson (2000.
In this paper, a report of the investigation of reading assessment in a period of 5 years is presented to see whether the constructs, delineated in pre-university coursebook, are rep-resented and relevant. To do this, a survey of the following is conducted: 1. Themes related to the purposes and constructs of assessment 2. The number of items assigned to particular constructs

Research Questions
With reference to reading constructs, skills and tests, one should take the following into account while constructing high -stake tests like university -entrance exams: the range of constructs, test method effects, construct under-representation and construct-irrelevant variance. To my knowledge, the reading comprehension questions of university entrance exams administered in Iran have not been checked and analyzed to see whether they appropriately reflect the underlying constructs delineated in the pre-university English course book. To address this gap, the present paper aims to analyze reading comprehension questions, the starting point of which marks the introduction of the course book in Iran's Preuniversity centers. That is to say, the researcher has chosen to investigate the representativeness of reading construct in the mentioned tests by checking if the content of tests adequately reflects the breadth and depth of reading constructs delineated in the pre-university course book.
More specifically, the present study aims at answering the following questions: 1. Do the question types in the reading comprehension section of the test reflect the range of constructs focused on in the pre-university course book? 2. Is the frequency of each question type appropriate for the relevant reading construct delineated in the pre-university course book?

Methodology
In order to answer the research questions posed earlier, the constructs of reading focused on in the introduction of preuniversity English course book and the relevant teacher's manual were retrieved and discussed. Then, on the basis of the guidelines offered in two TOEFL books and some other resources, the types of questions in the reading comprehension sections of university-entrance exams were identified and tallied. The resultant frequencies for the question types across the period of five years were calculated and added.

ReseaRch PaPeR
By employing the statistical procedure of chi square, the frequencies of the question types were compared.

Reading Construct
We need to know what the construct is in order to assess it. Therefore, to devise a test or assessment procedure for reading, we must, in Alderson's terms (2000), surely appeal, if only intuitively, to some concept of what it means to read texts and to understand them. Testers have to get involved in test construction even though they know in advance that their understanding of the phenomenon-the construct-is faulty, partial and possibly never perfectible.
To get an idea about what constructs are being emphasized in the course book, the researcher referred to the teacher's manual, the introduction of the student book and the nationwide test specification.
By borrowing the idea from Alderson (2000, 3-4), the authors explain that reading, the product of which is comprehension, involves the process in which a reader interacts with the text. During the process, presumably, many things are happening. Not only is the reader looking at print, deciphering in some sense the marks on the page, 'deciding' what they 'mean' and how they relate to each other. The reader is presumably also 'thinking' about what he is reading: what it means to him, and how it relates to other things he has read, to things he knows, to what he expects to come next in texts like this.
He is presumably thinking about how useful, entertaining, boring, crazy, the text is. He may be consciously reflecting on the difficulties or ease he is experiencing when reading, and on ways of overcoming the difficulties or of continuing the pleasure.
More importantly, some general objectives have been delineated by the authors. A good reader has specific objectives on mind before reading the passage, evaluates the text to see whether his/her objectives are being fulfilled, surveys the text to recognize the organization of ideas, discerns the relevant points, predicts what comes next, and makes decisions about what to read carefully, with speed, and again; and what not to read.
Following this, they draw the student's attention to adopting important strategies of effective reading: 1. Having a purpose for reading. 2. Following the framework of "what we know, what we want to know, what we have learned and how. We can learn more." 3. Paying attention to headlines, pictures, tables and other visual aids. 4. Activating background and linguistic knowledge. 5. Paying particular attention to punctuation marks, transition words and pronouns (or more generally referents). 6. Jotting down the important points briefly in the margin. 7. Attending explicit and implicit functions of sentences in a passage. 8. Widening one's eye span to increase reading speed.
More specifically, the authors focus on specific reading skills in each lesson, which are as follows: 1. Recognizing the function of sentences, namely explanation, instruction, definition and exemplification. 2. Attending paragraph headings as a device to locate information more conveniently. 3. Guessing the meaning of unknown words by using context clues such as restatements, definitions, punctuation marks, word formation, mother tongue, and transition words. 4. Making connections between different parts of a paragraph or/and text by establishing the relationship between referents and antecedents, and by knowing the uses of punctuation marks.
5. Finding a main idea for each paragraph which is explicitly or implicitly understood and acts as a clue for finding about the organization of a passage. 6. Classifying the important points of a passage into different categories, e.g. classification of events in terms of chronological order. 7. Summarizing a text by reorganizing the important points of the passage.
Another piece of evidence about the construct comes from the exercises of course book and the nationwide test specifications which should be followed by English teachers. The ones which are not mentioned in the earlier parts are listed below: 1. Matching headings with paragraphs 2. Recognizing implicit and explicit information 3. Finding the answer directly or indirectly from the passage, i.e. restatement and inference questions. 4. Identifying the statements which are not mentioned in the passage. 5. Recognizing the statements which are supported by the passage 6. Finding paraphrases in the passage 7. Understanding the function of sentences in a passage 8. Identifying the author's attitude, opinion or purpose the examination of objectives, skills, strategies, exercises.
The examination of objectives, skills, strategies, exercises and test specifications which are mentioned above enables one to form an idea about the constructs which are delineated in the course book. The researcher can now see whether these constructs are appropriately reflected in the reading comprehension question types of university-entrance exams in terms of range and weight.

Reading Comprehension Question Types
Out of the three factors of text, task and reader which are the central issues in the research concerning reading comprehension, the present study focuses on test tasks. As we said before, there can be different constructs of reading which can be equally valid for their purpose such as the one (s) described above. However, the test should necessarily be a sample of the underlying construct (s). One of the ways to check this correspondence is through question types because their variety, range and frequency can guarantee the elicitation of reader's skills, strategies, abilities and processes underlying the constructs of interest. One the basis of the assumption that there is a correspondence between question types and skills, the attempt was made to identify reading comprehension question types in the university entrance exams.
The criteria to distinguish these question types were their defining features, wordings, scope and underlying processes. However, two TOEFL guides written by Zhong and Sullivan (1990) and Gallagher (2005)

Iran's University Entrance Exam
As a high-stake test, Iran's University Entrance Exam is annually administered to select some candidates for state universities across the country. The English section of the test comprising 24 to 70 items assesses the testees' ability through tests of vocabulary, grammar, cloze procedure, reading comprehension and functions. Reading comprehension section of the test which is the focus of this study consists of four to twenty items depending on the testees' course of study. The categorization of questions was carried out according to the criteria described above. The distribution of question types is shown in the following page. In addition, the data was submitted to the statistical procedure of Chi-square to find the ReseaRch PaPeR differences between the frequencies of question types.

Results and Discussion
As the table of data and the bar graph in the preceding page show, there is an unbalanced distribution among question types. The most frequent question out of the nine types is 'restatement', while the least frequent ones are 'support' and 'sentence function'. If we look at the proportions of question types in the following table, we can see that around 38% of the total number of reading comprehension questions belongs to 'restatement', while the approximate percent for 'support' and 'sentence function' is below 1%. Another interesting point is the proportion of RC questions to the total number of questions (i.e. 21%), which is below a quarter of the total.
The statistical procedure of Chi-Square was employed to judge whether the observed frequencies have significant differences or not. As you can see in the following table, the obtained value for x 2 is 188.15 with eight degrees of freedom. Since this value is more than the critical values for x 2 at .05 level of significance (x 2 (8) =15.51) and at .01 level of significance (x 2 (8) =20.09), the null hypothesis that there are not significant differences between the frequencies is rejected. .000

Table3: Frequency of Question Types
With reference to the first question that asks whether the question types in the reading comprehension section of the test reflect the range of constructs delineated in the pre-university course book, the results revealed that some of the constructs are represented by question types in the test. These include understanding explicit and implicit information in the text mainly through 'restatement' and 'inference' questions respectively differentiating main ideas from details through 'main idea' question, identifying what is not ReseaRch PaPeR mentioned in the text through 'negative' questions, understanding the connections between sentences and ideas in the text through 'referent' questions, guessing the meaning of unknown words in the context through 'vocabulary' questions, understanding the author's point of view and the information supported in the text through 'author' 'purpose' and 'support' questions respectively, and making a relationship between sentence form and function through 'sentence function' questions.
What is worth-mentioning is that in some cases a limited sense of some constructs are focused on. For example, most of the vocabulary questions, 5 out of 23, address 'memory recall' processes, in which the testees should retrieve dictionary definitions or synonym for the items. This is done at the expense of inferential process, which demands the testees to guess the meaning of unknown words by using context clues.
However, some important constructs, skills and strategies delineated in the course-book are totally ignored. Among these are summarizing a text, classifying events and matching headings with paragraphs.
The second research question is concerned with the appropriacy of frequencies for question types. As it was explained earlier, there is an unbalanced distribution for question types. This shows that most of the constructs are underrepresented in the test. In this regard, the difference between 'restatement' and 'inference' in terms of percentage (38% vs. 6%) is noticeable. These two questions elicit the understanding of explicit and implicit information in the text respectively. If we, for example, compare these proportions with those of TOEFL (31% to 29%, cited in Zhong & Sullivan, 1990), we can better detect the under-representative ness of 'inference' as one of the constructs delineated in the coursebook. This imposes a kind of restriction on assessment: students are exclusively involved in cut-and-paste 'and'paraphrase identification processes.
One might claim that drawing inferences from text can involve working out the main idea of the text and determining the writer's attitude to the topic, but even if we assumed that all of the questions under the headings of 'main idea' and 'authors' purpose elicit inferential process, which is not the case, the total percentage would be around 19. Examining the table of frequency and percentage also reveals that question types -'support', 'sentence function', 'author's purpose', 'inference' and 'referent'-have frequencies which are below the expected value (21.3). Again, this is another case of construct under representation. The case of 'sentence function' which is continuously presented in the units of course book but has an observed frequency of 1 out of 122 is really surprising.

Conclusion and Imploications
Although it is preferable to go beyond literal and inferential comprehension of the text and include other levels of comprehension, i.e. critical response and evaluation, it seems to be impractical for high-stake exams like university-entrance exam. Instead, we, in the first place, can increase the number of RC questions in the English test. This is in accordance with test specifications devised for the end-of-the-term exam, in which half of the points (16 out of 30) is devoted to the questions assessing reading comprehension.
Another suggestion is related to the distribution of questions which should be made more balanced by incorporating questions assessing both bottom-up and top-down elements and, more importantly, explicit and implicit information in the passage.
A comprehensive assessment of reading comprehension demands the use of multiple measures. In addition to the commonly used measure, (i.e. texts followed by multiple choice questions), cloze tasks can be a practical alternative candidate.
The cloze passages in the university-entrance exams usually measure comprehension only within sentences based on word associations and usage-intrasentential comprehension. That is to say, they are mainly accounted as measures of vocabulary and grammar. However, they can be modified in a way that they assess comprehension of meaning across sentences -intersentential comprehension.
Most of the tests under question contain only one passage. The inclusion of more texts has the benefits of having a wider range of question types and neutralizing the inhibiting effect of prior knowledge.
There should be documents of assessments laid out for test developers and made available to the public, which contain information on the constructs that should be tested, the number of items desired, and the types of items desired. This can prevent test developers from developing unbalanced tests in which some constructs are not represented or under represented. This formal test specification can be of great help for the testees focus on desired constructs.
The present research focused on the underlying constructs reflected in tasks. However, future research can focus on the examination of reading comprehension passages in terms of readability indexes, topics, syntactic tactics and organization. Another line of research can focus on statistical information obtained through examining response patterns.