An oral reading test designed to assess EFL learners' reading ability or oral reading ability requires human raters, and therefore, the inter-rater reliability must be examined. The present study firstly investigated the relation between rater training and reliability. Two English teachers evaluated 26 students' oral reading performance and the ratings given by a trained rater were compared with those given by a non-trained rater. Results of correlation analyses and generalizability (G) theory showed that the oral reading test had high reliability, and the rater training, which did not change reliability, only made wide variations in ratings. Secondly, the present study attempted to find how many criteria should be included in a rating scale, and the ideal number of raters to ensure reliability. It was shown by G theory that two raters with three criteria would show the best performance in terms of cost effectiveness. Thirdly, we examined whether teacher-rated fluency could be replaced by an oral reading rate, which was an objective measurement. In regard to the latter, implications were drawn for future research in this area.