A Quantitative Analysis of Vocabulary Taught in Japanese EFL Textbooks

This research evaluates vocabulary taught in government-approved Japanese EFL textbooks intended for senior high school students in two different ways: by calculating the proportion of high-frequency words in textbooks and by calculating how many types of core high-frequency words are taught in textbooks. To this end, vocabulary words in textbooks are compared to those in the New General Service List (NGSL), which consists of the top 2,801 high-frequency words in general English. Results show that textbooks are largely made up of words in the NGSL at a statistically signicant level with larger than 92% lexical coverage; however, textbooks do not suciently cover words in the NGSL at a statistically signicant level with up to 38%. Overall, research ndings indicate that vocabulary words textbooks provide may be what learners frequently come across in the real world, but those words cannot suce in order for learners to read English texts intended for native speakers. Therefore, language teachers need to give learners additional input of core words through homework or classroom activities to widen their vocabulary size.


Introduction
In reaction to the increasingly globalized society, the revision of the Couse of Study Guidelines this time has a strong awareness of the necessity of improving learners' English skills regardless of their occupations as English will be supposed to be used in many more situations than ever (the Ministry of Education, Culture, Sports, Science and Technology [MEXT], 2018; MEXT, 2019). To cultivate students' English skills, one area where researchers can contribute lies in the development of textbooks that have the potential to teach English effectively because for those who study English as a foreign language (EFL), textbooks used in the classroom serve as a main source of English input (Bouhlal, Horst, & Martini, 2018); this is also the case in Japan (Matsuda, 2002). In other words, the improvement of textbooks potentially has a direct contribution to that of learners' English skills.
Vocabulary, whose essentialness in language learning has been long acknowledged by many researchers (Alqahtani, 2015;Schmitt, Schmitt, & Clapham et al., 2001;Zimmerman, 1997), was investigated in this study. According to Nation (2006Nation ( , 2013, a reasonable learning target is neither mastering all of the English words nor trying to master as much vocabulary as do native speakers of English. Rather, nonnative speakers should aim to achieve vocabulary thresholds where comprehension occurs for English texts in which they are interested. If one wants to read an authentic English text such as novels or newspapers, they need to get 95% coverage (Laufer, 1989;Liu & Nation, 1985) yet ideally 98% coverage of the text (Laufer & Ravenhorst-Kalovsk, 2010); these coverage levels are provided with 4,000-5,000 and 8,000 of the most frequent word families respectively.
For developers of textbooks, this information on coverage levels would be of much use. Putting this clearly, by counting how many types of high-frequency words are treated in their textbooks, they can ascertain how close their textbooks can potentially make learners' vocabulary size to those required vocabulary sizes. Even if it is found that their textbooks do not cover high-frequency words so much, they can make the best use of that information for the development of textbooks in the future. Furthermore, textbook developers can prevent themselves from developing textbooks having a large number of words that are not worth learning. If the lexical coverage of a textbook calculated with core high-frequency words is far behind the vocabulary thresholds mentioned above, the textbook might have an inclination to teach vocabulary that learners will be less like to meet in the real world and would therefore need some modi cations. In sum, examining vocabulary in a textbook by calculating the proportion of high-frequency words and by counting the number of types of core high-frequency words can possibly offer designers of textbooks some insight into what vocabulary to teach in their textbooks.
This research, therefore, evaluated vocabulary occurring in textbooks in these two different ways.

Literature Review
Research into Vocabulary Taught in Japanese EFL Textbooks Vocabulary taught in Japanese EFL textbooks is now becoming one of the research areas where researchers show an interest. In this section, we look at how vocabulary in teaching materials has been evaluated. Chujo (2004) measured vocabulary levels of not only the combinations of junior and senior high school EFL textbooks, but also college quali cation tests, textbooks and articles used at colleges, and English pro ciency tests by using the British National Corpus (BNC). She de ned words occurring 100 times or more as high-frequency words and extracted those that met this criterion from the BNC for the creation of a benchmark word list. High-frequency words extracted from the BNC were then strati ed into frequency bands, each of which consists of 100 lemmas, for a total of 140 bands. Then, it was investigated how many of the word bands readers needed to be familiar with for reading each of the materials. Results showed varying vocabulary demands of the materials, with a low of the top 2,800 lemmas to a high of the top 15,000 lemmas. For the combinations of junior and senior high school textbooks, it was found that the 3,200 most frequent lemmas were required.
Not being at the center of his research, Underwood (2010) also measured vocabulary levels of Japanese EFL textbooks. He compared textbooks approved by MEXT (hereafter, MEXT-approved textbooks) with the National Center Exams, which are the most common university admission tests in Japan, in terms of readability, vocabulary levels, reading skills which were supposed to be required to solve comprehension questions, and types of comprehension questions. In his study, vocabulary levels were measured by referring to the General Service List (GSL; West, 1953) and the Academic Word List (AWL; Coxhead, 2000); it was observed that textbooks were largely made up of high-frequency words. Because he reported not vocabulary levels of textbooks but the average frequency of occurrence of the GSL and the AWL words per chapter of textbooks, I calculated the lexical coverage of textbooks by the word lists based on the reported average scores and found almost 90% coverage was provided with the rst and second 1,000 most frequent words in the GSL. He concluded this pervasiveness of high-frequency words in textbooks might allow students to get regular exposure to them. Wongsarnpigoon (2018) measured vocabulary levels of MEXT-approved junior high school textbooks as well as those of senior high school entrance examinations by using Heatley et al.'s (2002) software, Range. For the study, he created two different corpora: a corpus of MEXT-approved junior high school textbooks and a corpus of senior high school entrance examinations. He found that 95% coverage of textbooks was provided with the top 2,000 high-frequency words in the BNC. He thus considered textbooks to be appropriate materials for learning high-frequency words. Additionally, a comparison between the two corpora identi ed that the vocabulary demands of entrance examinations were less than those of textbooks, leading him to consider entrance examinations to be appropriate learning targets.
By measuring vocabulary levels, these previous studies offered a clear picture of the density of highfrequency words in textbooks (and other materials) as well as their vocabulary loads. Pedagogically speaking, these previous studies are noteworthy. As suggested by Underwood (2010) and Wongsarnpigoon (2018), teachers in Japan can use textbooks analyzed in these studies in order to impart high-frequency words to students. Following them, this study also measured vocabulary levels of textbooks.
In the rst place, textbooks are not reading materials but teaching materials, so the primary focus of research into vocabulary in textbooks should be on the investigation of how many types of highfrequency words students can potentially learn through textbooks, rather than that of how many vocabulary words in textbooks are high-frequency. Surprisingly, this area of research has not as yet directly investigated. This would be an area to explore since without that research, textbook developers could not notice the potential de ciency of supply of vocabulary required for reading an authentic text in their textbooks and would unintentionally leave it as it is. The following two research questions were addressed: 1. How much vocabulary in MEXT-approved textbooks is what learners are more likely to meet in the real world?
2. If Japanese students are able to master all of the vocabulary words taught in MEXT-approved EFL textbooks, is that enough for them to be able to read authentic texts?

Vocabulary Size Needed to Read English Texts
In regards to the second research question, it would be necessary to make it clear how much vocabulary is required for reading an authentic text.
At the end of the 20 th century, it was suggested that 95% of running words in a text should be known for reading a text (Laufer, 1989;Liu & Nation, 1985) and guessing from context (Liu & Nation, 1985).
Subsequently, Hu and Nation (2000) administered reading comprehension tests to 66 learners and found that 98%-99% of running words in a text should be known for reading a text for pleasure. At the same time, they observed comprehension took place for a few learners with 90% and 95% of lexical coverage. 90% coverage would be provided by the second 2,000-3,000 most frequent word families along with proper nouns, marginal words (e.g., "er"), and transparent compounds (e.g., "forever" and "lifespan") (Nation, 2013). According to Laufer and Ravenhorst-Kalovsk (2010), 95% and 98% of lexical coverage require readers to know 4,000-5,000 and 8,000 of the most frequent word families respectively. Although 98% coverage is an optimal threshold for learners ( composed of decontextualized sentences accompanied by their Japanese translations or explanations of grammar rules written in Japanese. That is, it can be said that textbooks used in the former course serve as a main source of vocabulary input rather than those in the latter course. To ensure a representative sample of Japanese learners' vocabulary input, textbooks used in the former course were solely examined. Komyunikeisyon Eigo course is further divided into three subgroups according to their course levels, namely, Komyunikeisyon Eigo 1, 2, and 3. Generally, they are intended for 1st-, 2nd-, and 3rd-year senior high school students respectively. By referring to the website of Tokyo Metropolitan Board Education (2020), the top three best-selling MEXT-approved textbooks in Tokyo in 2021 were chosen from each of the three levels, for a total of nine textbooks as shown in Table 1. Another point to mention is that the NGSL deleted "proper nouns, abbreviations, slang and other noise." (Browne, 2013, p. 15) Text data was thus so arranged that this study could fairly compare the textbooks with the NGSL, making data analysis time-consuming as closely explained in the next section.

Procedure
To make the textbooks analyzable, I rstly scanned them with OCR software; then, they were converted into .text les. Text data were gathered solely from reading passages in the textbooks. Because OCR software cannot provide 100% text compilation accuracy, I went through every page and manually modi ed errors.
There are now several corpus-analysis tools that are capable of calculating the lexical coverage of a text by the NGSL; for example, AntWordPro ler 1.5.1 (Anthony, 2021), the Online Graded Text Editor (OGTE; Waring & Browne, n.d.), VocabPro ler . Unfortunately, these tools cannot compute the coverage level of the NGSL words by the textbooks, in part because those are not meant to analyze word lists themselves. Besides, there are cases where they incorrectly identify intended meaning of homographs (e.g., "found" used as the past tense form of " nd" or the meaning of "establish") or components of contracted forms (e.g., "I'd" used as "I would" or "I had"), indicating the necessity of analyzing those words while considering the surrounding context.
For these reasons, this study adopted a unique approach by using Cobb's text analysis tool, Text Lex Compare ver. 4.2. (2021) to compare the textbooks with the NGSL as reasonably as possible. This tool compares two different texts and informs us of how many tokens, word types, word families, and word phrases are used in and unique to each text. By replacing words in the textbooks with their headwords based on the modi ed lexeme approach, the number of word types in a text counted by that tool becomes equal to that of emmas in the text, making it possible to reasonably compare vocabulary words between the NGSL and the textbooks.
As a preliminary preparation for analysis, every word in the textbooks was replaced with its headword based on the modi ed lexeme approach. In so doing, the replacement function of Microsoft Word 2016 was used (see Figure 1). This was also used to separate semantically transparent contracted forms (e.g., "I'm" into "I am") as those words are composed of two different emmas. Semantically opaque ones (e.g., "I'd" used as "I would" or "I had") and homographs were replaced with their headwords while considering the surrounding context.
Text data after these treatments were loaded into a WordList module which is part of Wordsmith 7 (Scott, 2016). Word lists produced by this module were used to visually ascertain whether or not every word occurring in the textbooks was successfully replaced with its headword.
When calculating the lexical coverage of the textbooks, the 52 supplemental words were added to the 2,801 words in order to avoid the possibility that their frequent recurrences in the textbook underestimate its lexical coverage. On the other hand, they were not added when counting the number of the NGSL words covered by the textbooks. One of the justi cations for this was that even if this research argues the de ciency of high-frequency words in the textbooks based on evidence that they do not cover all of numerals or months of the year, no one would be convinced because presenting all those words in a single textbook would not be possible as well as of much help to students. Figure 2 shows how lexical coverage was computed by using Cobb's Text Lex Compare.
According to Figure 2, 61 tokens were unique to the second text (i.e., the textbook), meaning that 1403 tokens (calculated by 1464 -61) were covered by the rst text (i.e., the NGSL). The lexical coverage of the textbook was then calculated by 1403 ÷ 1464 95.8%.

Results
Lexical Coverage of the Textbooks by the NGSL To address the rst research question, we will look at the lexical coverage of the textbooks by the NGSL. Let us bear in mind that if the textbooks had been developed with an intention to teach high-frequency words, most words appearing in the textbooks should belong to the NGSL. Table 2 reports the number of tokens and that of word types used in and unique to each of the textbooks; the lexical coverage of the textbooks; the proportion of tokens to those unique to the textbooks as well as that of word types to those unique to the textbooks. Seemingly, all the textbooks are largely composed of the NGSL words with larger than 92% coverage.
Looking at the columns showing the proportion of word types appearing in the textbooks to those unique to them, a notable nding can be that the textbooks provided words outside the NGSL in varying proportions ranging from 8% (COMET ) to 22% (ELEMENT ); however, most of those off-list words seem to have rarely recurred in the textbooks as can be inferred from the small differences in frequency of occurrence between tokens unique to the textbooks and word types unique to them. If word types unique to the textbooks had recurred many times, lexical coverage would have been lower than what was shown in the table. To see whether or not the differences in frequency of occurrence between words in the NGSL and those outside the NGSL were statistically signi cant, a Chi-square goodness of t test was performed on the data by using Mizumoto's (2015) web-based application, langtest.jp. Next, to explore the question of whether or not vocabulary words taught in the textbooks can su ce in order for learners to read English texts intended for native speakers, we look at how many of the NGSL words were covered by each of the textbooks (see Figure 3).
The textbooks covered the NGSL words with considerably different proportions ranging from 12% (COMET ) to 37% (BIG DIPPER ). These proportions might be low enough for us to insist on the de ciency of the NGSL words in the textbooks. To corroborate the results, a Chi-square goodness of t test was performed. This statistical test found statistical signi cance for all the differences with varying effect sizes from small to large. The results indicate that a large part of the NGSL is not covered by the textbooks. The combined results of the lexical coverage analysis of the textbooks and the coverage analysis of the NGSL indicate that although most words in the textbooks belonged to the NGSL, they were only part of the NGSL.

Discussion
The results of the lexical coverage analysis showed that vocabulary words taught in the nine types of textbooks under investigation were largely made up of the NGSL words at a statistically signi cant level with larger than 92% lexical coverage. Pedagogically speaking, this is good news because vocabulary words that learners can come across in textbooks would be what they are more likely to meet in the real world. Putting that nding another way, those who have mastered core high-frequency words would be able to achieve 90% coverage of the textbooks, which is a minimum requirement for comprehension, and therefore can read them without a heavy reliance on a dictionary or teacher's explanation. This is contrary to Browne's (1998) ndings. He analyzed the lexical coverage of MEXT-approved senior high school textbooks by the GSL (West, 1953) and found that they contained so many new and low-frequency words that even learners with the 2,000 most frequent words in the GSL could hardly read a text and guess word meaning from context. Although these contrasting ndings may be because this research used a different word list than Browne's study, it would not be a groundless argument that MEXT-approved senior high school textbooks have come to re ect high-frequency words in the real world over the past two decades.
By counting how many of the NGSL words were treated in the textbooks, it was found that all the textbooks did not su ciently cover the NGSL words at a statistically signi cant level. At the same time, the textbooks were found to be insu cient in order for learners to achieve the minimum threshold (i.e., 90%) for reading an authentic text. Hence, they may have to bridge a gap by themselves between vocabulary size that they can potentially learn through textbooks and it required for reading an authentic English text. Language teachers would thus need to give learners additional input of basic words through homework or classroom activities. For example, they can ask their students to watch English movies (Nation, 2006) (Elley, 1991). In these ways, providing students with opportunities for self-learning can possibly lead them to become independent English learners, which is related to learners' success in vocabulary learning (Gu & Johnson, 1996).
The insu cient supply of the NGSL words in the textbooks may be partly because MEXT-approved textbooks treat a wide variety of topics and are too short to cover all of the NGSL words. In general, they consist of several chapters accompanied by two or three sections; each chapter introduces one type of topic such as emperor penguins' ecosystem, an environmental problem, a famous Japanese tennis player, and so on. As the use of vocabulary words largely depends on language genres where they occur (Hsu, 2009;Nation, 2006), the greater the number of topics, the greater the number of topic-dependent words. Conversely, if textbooks treat a relatively narrow variety of topics, those words may rarely show up, meaning that it becomes possible to cover a greater number of high-frequency words.
For textbooks to cover core vocabulary words, another possible way is simply to increase the number of words treated in textbooks as the new Course of Study Guidelines declares. Prior to looking at this point more closely, please note that MEXT's word counting way is almost identical to yet slightly different from that of the NGSL in that MEXT does not count comparatives and superlatives of irregular adjectives as members of their positive forms. Considering that in English there are not so many irregular adjectives, it would be fair to say that the numbers of words of a text counted by MEXT and the NGSL are comparable to each other. The current Couse of Study Guidelines declares that around 3,000 words need to be learned before learners graduate senior high school (MEXT, 2008(MEXT, , 2009). The new Course of Study Guidelines (MEXT, 2017a, b; MEXT 2018), on the other hand, intends to make students familiar with 4,000-5,000 words in total through 10 years of English learning beginning from age 8 or 9. This addition would increase the possibility that MEXT-approved textbooks treat more high-frequency words.
Here, it can be interesting to report results of a survey conducted by the Benesse Educational Research and Development Institute (2014), which is one of the largest companies involved in education in Japan. According to their survey results, junior and senior high school students spend the longest time during their preparation for a lesson looking up new words. Besides, the vast majority of English teachers in Japan felt that students had an aversion to memorizing vocabulary words (Benesse Educational Research and Development Institute, 2015). Although it is a widely accepted idea that an increase in vocabulary size has a positive effect on one's reading pro ciency (Hsu, 2009  As can be seen from this table, several genres require readers to know more words than TV shows or movies. To understand a diverse range of English texts, students would also need to be familiar with words outside the NGSL, that is to say, mid-or low-frequency words. Acknowledging the necessity of learning words other than high-frequency words, this study suggests in the rest of this paper that students should learn the NGSL words rst. Learning English vocabulary is not merely an act of memorizing word meaning (Nagy, 1997). In this regard, Nation (2013) gave a detailed description of multidimensional aspects involved in one word.
Brie y explained, three aspects are involved in vocabulary knowledge, namely, form, meaning, and use; each has three components. Form refers to knowledge related to (a) pronunciation, (b) orthography, and (c) morphology; meaning refers to knowledge regarding (a) the meaning of the word form and a word form which can express the meaning, (b) concepts the word has and referents of the concept, and (c) other words semantically associated with the word; use refers to knowledge related to (a) grammatical functions, (b) collocations, and (c) registers to use the word or the frequency of occurrence of the word. Teaching all of these aspects involved in one word at a time is not a reasonable approach (McCarten, 2007;Nation, 2013), and an oversupply of new words within a single textbook possibly prevents language teachers from imparting those aspects to learners as they may unintentionally spend a huge amount of time explaining word meaning. As Nagy (1997) argues, de nition-based vocabulary learning should be avoided as much as possible because students may end up learning little about one word. What people involved in English education should do is thus giving students a proper amount of vocabulary.
"Low-frequency words tend to have fewer family members than high-frequency words" (Nation, 2006, p. 65), indicating that more time should be allocated to high-frequency words than low-frequency words so that learners can become well acquainted with at least core high-frequency words. Accordingly, MEXT should give multiple occurrences to high-frequency words, rather than increasing the number of words to teach. By so doing, students can know much about high-frequency words because "repeated meeting can have the effects of strengthening and enriching previous knowledge" (Nation, 2013, p. 127).
These arguments are not to say mid-or low-frequency words are not of high value for learners. Learning those words will be undoubtedly necessary because when they enter a university or college, they will need to be familiar with academic words or technical words in their majors. However, it would take many years for Japanese learners to master high-frequency words (Browne, 2021), asking them to learn mid-or lowfrequency words including specialized vocabulary words is impractical. According to Nation (2013), those words can be effectively learned by paying special attention to them. Fortunately, we can now have access to numerous word lists focusing on particular contexts; for example, the AWL (Coxhead, 2000), and more. High-frequency words in general English are more dominant even in such specialized contexts than specialized words (Coxhead, 2000;Nation, 2013), again indicating the importance of mastering high-frequency words rst as Nation (2013) suggests. Such high-frequency words as the NGSL words can potentially allow students to prepare for their English learning in the future. Limitations There are several limitations to note. It is general that senior high school students in Japan take English classes not for just one year but for three years. In other words, they learn vocabulary through three types of textbooks. Therefore, the evaluation of vocabulary should have been done not by textbook but by a set of three textbooks to represent learners' vocabulary input at senior high school. Due to the availability of information on which set was used by students the most, this research decided to consult the marketing data of textbooks in Tokyo.
Second, the number of textbooks examined in this study was limited to 10 types of MEXT-approved senior high school textbooks. To generalize the results, further research should be conducted for textbooks other than those analyzed in this study. The research ndings will be, then, more reliable.

Conclusion
This study has evaluated vocabulary in MEXT-approved senior high school textbooks from two different perspectives: the lexical coverage of the textbooks by the NGSL and the coverage of the NGSL by each of the textbooks. The overall research ndings indicated that it was possible for students to learn highfrequency words through textbooks, but those words would not be su cient enough for learners to read an authentic text. In reaction to these ndings, it was suggested that textbooks should teach the NGSL words rst in order to help students master at least core high-frequency words. Without these words, learners would not become able to read any kind of text because "any signi cant gap in knowledge of these ultra-high frequency words would make it virtually mathematically impossible to achieve 90% or higher coverage." (Browne, 2021, p. 3).   Proportion of NGSL Words Covered by the Textbooks to Those not Covered by Them