Hostname: page-component-848d4c4894-wzw2p Total loading time: 0 Render date: 2024-04-30T14:56:34.541Z Has data issue: false hasContentIssue false
  • English
  • Français

Assessment of L2 intelligibility: Comparing L1 listeners and automatic speech recognition

Published online by Cambridge University Press:  18 November 2022

Solène Inceoglu Open the ORCID record for Solène Inceoglu [Opens in a new window] ,
Wen-Hsin Chen Open the ORCID record for Wen-Hsin Chen [Opens in a new window]  and
Hyojung Lim Open the ORCID record for Hyojung Lim [Opens in a new window]
Solène Inceoglu
Affiliation:
Australian National University, Australia (solene.inceoglu@anu.edu.au)
Wen-Hsin Chen
Affiliation:
National Central University, Taiwan (kellychen@g.ncu.edu.tw)
Hyojung Lim
Affiliation:
Kwangwoon University, South Korea (lim@kw.ac.kr)
Get access Rights & Permissions [Opens in a new window]

Abstract

An increasing number of studies are exploring the benefits of automatic speech recognition (ASR)–based dictation programs for second language (L2) pronunciation learning (e.g. Chen, Inceoglu & Lim, 2020; Liakin, Cardoso & Liakina, 2015; McCrocklin, 2019), but how ASR recognizes accented speech and the nature of the feedback it provides to language learners is still largely under-researched. The current study explores whether the intelligibility of L2 speakers differs when assessed by native (L1) listeners versus ASR technology, and reports on the types of intelligibility issues encountered by the two groups. Twelve L1 listeners of English transcribed 48 isolated words targeting the /ɪ-i/ and /æ-ε/ contrasts and 24 short sentences that four Taiwanese intermediate learners of English had produced using Google’s ASR dictation system. Overall, the results revealed lower intelligibility scores for the word task (ASR: 40.81%, L1 listeners: 38.62%) than the sentence task (ASR: 75.52%, L1 listeners: 83.88%), and highlighted strong similarities in the error types – and their proportions – identified by ASR and the L1 listeners. However, despite similar recognition scores, correlations indicated that the ASR recognition of the L2 speakers’ oral productions mirrored the L1 listeners’ judgments of intelligibility in the word and sentence tasks for only one speaker, with significant positive correlations for one additional speaker in each task. This suggests that the extent to which ASR approaches L1 listeners at recognizing accented speech may depend on individual speakers and the type of oral speech.

Keywords

automatic speech recognitionintelligibilityCALLpronunciation learningnon-native speech
Type
Research Article
Information
ReCALL , Volume 35 , Issue 1 , January 2023 , pp. 89 - 104
Check for updates
Copyright
© The Author(s), 2022. Published by Cambridge University Press on behalf of European Association for Computer Assisted Language Learning

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Ahn, T. & Lee, S.-M. (2016) User experience of a mobile speaking application with automatic speech recognition for EFL learning. British Journal of Educational Technology, 47(4): 778–786. https://doi.org/10.1111/bjet.12354 CrossRefGoogle Scholar
Baran-Łucarz, M. (2014) The link between pronunciation anxiety and willingness to communicate in the foreign-language classroom: The Polish EFL context. The Canadian Modern Language Review, 70(4): 445–473. https://doi.org/10.3138/cmlr.2666 CrossRefGoogle Scholar
Bashori, M., van Hout, R., Strik, H. & Cucchiarini, C. (2022) Web-based language learning and speaking anxiety. Computer Assisted Language Learning, 35(5–6): 1058–1089. https://doi.org/10.1080/09588221.2020.1770293 CrossRefGoogle Scholar
Butler, Y. G. (2011) The implementation of communicative and task-based language teaching in the Asia-Pacific Region. Annual Review of Applied Linguistics, 31: 36–57. https://doi.org/10.1017/S0267190511000122 CrossRefGoogle Scholar
Chang, D. & Weng, C. (2013) Late ESL learners’ difficulties of distinction between lax and tense vowels. In Levis, J. & LeVelle, K. (eds.), Proceedings of the 4th Pronunciation in Second Language Learning and Teaching Conference. Ames: Iowa State University, 129–139.Google Scholar
Chen, W.-H., Inceoglu, S. & Lim, H. (2020) Using ASR to improve Taiwanese EFL learners’ pronunciation: Learning outcomes and learners’ perceptions. In Kang, O., Staples, S., Yaw, K. & Hirschi, K. (eds.), Proceedings of the 11th Pronunciation in Second Language Learning and Teaching Conference. Flagstaff: Northern Arizona University, 37–48.Google Scholar
Chiu, T.-L., Liou, H.-C. & Yeh, Y. (2007) A study of web-based oral activities enhanced by automatic speech recognition for EFL college learning. Computer Assisted Language Learning, 20(3): 209–233. https://doi.org/10.1080/09588220701489374 CrossRefGoogle Scholar
Derwing, T. M. & Munro, M. J. (1997) Accent, intelligibility, and comprehensibility: Evidence from four L1s. Studies in Second Language Acquisition, 19(1): 1–16. https://doi.org/10.1017/S0272263197001010 CrossRefGoogle Scholar
Derwing, T. M. & Munro, M. J. (2015) Pronunciation fundamentals: Evidence-based perspectives for L2 teaching and research. Amsterdam: John Benjamins. https://doi.org/10.1075/lllt.42 CrossRefGoogle Scholar
Derwing, T. M., Munro, M. J. & Carbonaro, M. (2000) Does popular speech recognition software work with ESL speech? TESOL Quarterly, 34(3): 592–603. https://doi.org/10.2307/3587748 CrossRefGoogle Scholar
Derwing, T. M., Munro, M. J. & Wiebe, G. (1997) Pronunciation instruction for ‘fossilized’ learners: Can it help?. Applied Language Learning, 8(2): 217–235.Google Scholar
Flege, J. E., Munro, M. J. & MacKay, I. R. A. (1995) Factors affecting strength of perceived foreign accent in a second language. The Journal of the Acoustical Society of America, 97(5): 3125–3134. https://doi.org/10.1121/1.413041 CrossRefGoogle Scholar
García, C., Nickolai, D. & Jones, L. (2020) Traditional versus ASR-based pronunciation instruction: An empirical study. CALICO Journal, 37(3): 213–232. https://doi.org/10.1558/cj.40379 CrossRefGoogle Scholar
Guskaroska, A. (2019) ASR as a tool for providing feedback for vowel pronunciation practice. Iowa State University, unpublished master’s thesis.Google Scholar
Hu, W., Tao, S., Li, M. & Liu, C. (2019) Distinctiveness and assimilation in vowel perception in a second language. Journal of Speech, Language, and Hearing Research, 62(12): 4534–4543. https://doi.org/10.1044/2019_JSLHR-H-19-0074 CrossRefGoogle Scholar
Inceoglu, S., Lim, H. & Chen, W.-H. (2020) ASR for EFL pronunciation practice: Segmental development and learners’ beliefs. The Journal of Asia TEFL, 17(2): 319–757. http://doi.org/10.18823/asiatefl.2020.17.3.5.824 Google Scholar
Kang, O., Thomson, R. I. & Moran, M. (2018) Empirical approaches to measuring the intelligibility of different varieties of English in predicting listener comprehension. Language Learning, 68(1): 115–146. https://doi.org/10.1111/lang.12270 CrossRefGoogle Scholar
Kennedy, S. & Trofimovich, P. (2008) Intelligibility, comprehensibility, and accentedness of L2 speech: The role of listener experience and semantic context. The Canadian Modern Language Review, 64(3): 459–489. https://doi.org/10.3138/cmlr.64.3.459 CrossRefGoogle Scholar
Kennedy, S. & Trofimovich, P. (2010) Language awareness and second language pronunciation: A classroom study. Language Awareness, 19(3): 171–185. https://doi.org/10.1080/09658416.2010.486439 CrossRefGoogle Scholar
Landis, J. R. & Koch, G. G. (1977) The measurement of observer agreement for categorical data. Biometrics, 33(1): 159–174. https://doi.org/10.2307/2529310 CrossRefGoogle ScholarPubMed
Levis, J. M. (2005) Changing contexts and shifting paradigms in pronunciation teaching. TESOL Quarterly, 39(3): 369–377. https://doi.org/10.2307/3588485 CrossRefGoogle Scholar
Levis, J. (2007) Computer technology in teaching and researching pronunciation. Annual Review of Applied Linguistics, 27: 184–202. https://doi.org/10.1017/S0267190508070098 CrossRefGoogle Scholar
Levis, J. & Suvorov, R. (2020) Automatic speech recognition. In Chapelle, C. A. (ed.), The concise encyclopedia of applied linguistics. Wiley-Blackwell, 149–156. https://doi.org/10.1002/9781405198431.wbeal0066.pub2 Google Scholar
Li, M. (2018) Know thyself? Self- vs. other-assessment of second language pronunciation. Boston University, unpublished PhD.Google Scholar
Liakin, D., Cardoso, W. & Liakina, N. (2015) Learning L2 pronunciation with a mobile speech recognizer: French /y/. CALICO Journal, 32(1): 1–25. https://doi.org/10.1558/cj.v32i1.25962 CrossRefGoogle Scholar
Litman, D., Strik, H. & Lim, G. S. (2018) Speech technologies and the assessment of second language speaking: Approaches, challenges, and opportunities. Language Assessment Quarterly, 15(3): 294–309. https://doi.org/10.1080/15434303.2018.1472265 CrossRefGoogle Scholar
McCrocklin, S. M. (2016) Pronunciation learner autonomy: The potential of automatic speech recognition. System, 57: 25–42. https://doi.org/10.1016/j.system.2015.12.013 CrossRefGoogle Scholar
McCrocklin, S. (2019) ASR-based dictation practice for second language pronunciation improvement. Journal of Second Language Pronunciation, 5(1): 98–118. https://doi.org/10.1075/jslp.16034.mcc CrossRefGoogle Scholar
McCrocklin, S. & Edalatishams, I. (2020) Revisiting popular speech recognition software for ESL speech. TESOL Quarterly, 54(4): 1086–1097. https://doi.org/10.1002/tesq.3006 CrossRefGoogle Scholar
Mroz, A. (2018) Seeing how people hear you: French learners experiencing intelligibility through automatic speech recognition. Foreign Language Annals, 51(3): 617–637. https://doi.org/10.1111/flan.12348 CrossRefGoogle Scholar
Mroz, A. (2020) Aiming for advanced intelligibility and proficiency using mobile ASR. Journal of Second Language Pronunciation, 6(1): 12–38. https://doi.org/10.1075/jslp.18030.mro CrossRefGoogle Scholar
Munro, M. J. & Derwing, T. M. (1995) Foreign accent, comprehensibility, and intelligibility in the speech of second language learners. Language Learning, 45(1): 73–97. https://doi.org/10.1111/j.1467-1770.1995.tb00963.x CrossRefGoogle Scholar
Neri, A., Cucchiarini, C., Strik, H. & Boves, L. (2002) The pedagogy-technology interface in computer assisted pronunciation training. Computer Assisted Language Learning, 15(5): 441–467. https://doi.org/10.1076/call.15.5.441.13473 CrossRefGoogle Scholar
Strachan, L., Kennedy, S. & Trofimovich, P. (2019) Second language speakers’ awareness of their own comprehensibility: Examining task repetition and self-assessment. Journal of Second Language Pronunciation, 5(3): 347–373. https://doi.org/10.1075/jslp.18008.str CrossRefGoogle Scholar
Trofimovich, P., Isaacs, T., Kennedy, S., Saito, K. & Crowther, D. (2016) Flawed self-assessment: Investigating self- and other-perception of second language speech. Bilingualism: Language and Cognition, 19(1): 122–140. https://doi.org/10.1017/S1366728914000832 CrossRefGoogle Scholar
Tsunemoto, A., Trofimovich, P., Blanchet, J., Bertrand, J. & Kennedy, S. (2022) Effects of benchmarking and peer-assessment on French learners’ self-assessments of accentedness, comprehensibility, and fluency. Foreign Language Annals, 55(1): 135–154. https://doi.org/10.1111/flan.12571 CrossRefGoogle Scholar
Wang, Y.-H. & Young, S. S.-C. (2015) Effectiveness of feedback for enhancing English pronunciation in an ASR-based CALL system. Journal of Computer Assisted Learning, 31(6): 493–504. https://doi.org/10.1111/jcal.12079 CrossRefGoogle Scholar