Taichi Higasa
ABSTRACT
Language learners should read challenging texts regularly. However, using dictionaries or search engines to look up difficult expressions can be time-consuming and distracting. To address this, we have developed a system combining eye tracking with Large Language Models (LLMs) to simplify sentences automatically, allowing learners to focus on the content. The system incorporates user-tailored models that estimate users’ comprehension of sentences using gaze data and sentence information. The system also features user-triggered simplification, resulting from iterative design improvements. We conducted a user study with 17 English learners where they read English text using either our system or a baseline involving online dictionaries and search engines. Our system significantly improved both reading speed and comprehension, especially for complex sentences. The gaze-based simplification improved concentration on the content, allowing for an interruption-free reading experience. It could assist in daily reading practice, particularly for extensive reading focused on large volumes of text at a rapid pace.
Footnotes
Supplemental Material
Available for Download
- David W. Aha and Richard L. Bankert. 1996. A Comparative Evaluation of Sequential Feature Selection Algorithms. Springer New York, New York, NY, 199–206. https://doi.org/10.1007/978-1-4612-2404-4_19Google Scholar
Cross Ref
- Olivier Augereau, Kai Kunze, Hiroki Fujiyoshi, and Koichi Kise. 2016. Estimation of English Skill with a Mobile Eye Tracker. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (Heidelberg, Germany) (UbiComp ’16). Association for Computing Machinery, New York, NY, USA, 1777–1781. https://doi.org/10.1145/2968219.2968275Google ScholarDigital Library
- Maria Barrett, Joachim Bingel, Nora Hollenstein, Marek Rei, and Anders Søgaard. 2018. Sequence Classification with Human Attention. In Proceedings of the 22nd Conference on Computational Natural Language Learning. Association for Computational Linguistics, Brussels, Belgium, 302–312. https://doi.org/10.18653/v1/K18-1030Google ScholarCross Ref
- R. Biedert, G. Buscher, and A. Dengel. 2010. The eyeBook – Using Eye Tracking to Enhance the Reading Experience. Informatik Spektrum 33 (2010), 272–281. https://doi.org/10.1007/s00287-009-0381-2Google ScholarCross Ref
- Ralf Biedert, Georg Buscher, Sven Schwarz, Jörn Hees, and Andreas Dengel. 2010. Text 2.0. In CHI ’10 Extended Abstracts on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI EA ’10). Association for Computing Machinery, New York, NY, USA, 4003–4008. https://doi.org/10.1145/1753846.1754093Google ScholarDigital Library
- Marc Brysbaert. 2015. SUBTLEX_US Word Frequency Database. https://osf.io/djpqz.Google Scholar
- Charles Clifton, Adrian Staub, and Keith Rayner. 2007. Chapter 15 - Eye movements in reading words and sentences. In Eye Movements, Roger P.G. Van Gompel, Martin H. Fischer, Wayne S. Murray, and Robin L. Hill (Eds.). Elsevier, Oxford, 341–371. https://doi.org/10.1016/B978-008044980-7/50017-3Google ScholarCross Ref
- Kevyn Collins-Thompson. 2014. Computational assessment of text readability: A survey of current and future research. ITL - International Journal of Applied Linguistics 165 (12 2014), 97–135. https://doi.org/10.1075/itl.165.2.01colGoogle ScholarCross Ref
- Jiexin Ding, Bowen Zhao, Yuqi Huang, Yuntao Wang, and Yuanchun Shi. 2023. GazeReader: Detecting Unknown Word Using Webcam for English as a Second Language (ESL) Learners. arxiv:2303.10443 [cs.HC]Google Scholar
- Yutao Feng, Jipeng Qiang, Yun Li, Yunhao Yuan, and Yi Zhu. 2023. Sentence Simplification via Large Language Models. arxiv:2302.11957 [cs.CL]Google Scholar
- Katsuya Fujii and Jun Rekimoto. 2019. SubMe: An Interactive Subtitle System with English Skill Estimation Using Eye Tracking. In Proceedings of the 10th Augmented Human International Conference 2019 (Reims, France) (AH2019). Association for Computing Machinery, New York, NY, USA, Article 23, 9 pages. https://doi.org/10.1145/3311823.3311865Google ScholarDigital Library
- Ana Valeria González-Garduño and Anders Søgaard. 2017. Using Gaze to Predict Text Readability. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, Copenhagen, Denmark, 438–443. https://doi.org/10.18653/v1/W17-5050Google ScholarCross Ref
- Ana González-Garduño and Anders Søgaard. 2018. Learning to Predict Readability Using Eye-Movement Data From Natives and Learners. Proceedings of the AAAI Conference on Artificial Intelligence 32, 1 (Apr. 2018). https://doi.org/10.1609/aaai.v32i1.11978Google ScholarCross Ref
- Taichi Higasa, Keitaro Tanaka, Qi Feng, and Shigeo Morishima. 2023 (to appear). Gaze-Driven Sentence Simplification for Language Learners: Enhancing Comprehension and Readability. In International Conference on Multimodal Interaction (ICMI ’23 Companion). ACM, Paris, France, 5. https://doi.org/10.1145/3610661.3616177Google ScholarDigital Library
- Rui Hiraoka, Hiroki Tanaka, Sakriani Sakti, Graham Neubig, and Satoshi Nakamura. 2016. Personalized Unknown Word Detection in Non-Native Language Reading Using Eye Gaze. In Proceedings of the 18th ACM International Conference on Multimodal Interaction (Tokyo, Japan) (ICMI ’16). Association for Computing Machinery, New York, NY, USA, 66–70. https://doi.org/10.1145/2993148.2993167Google ScholarDigital Library
- Aulikki Hyrskykari, Päivi Majaranta, Antti Aaltonen, and Kari-Jouko Räihä. 2000. Design Issues of IDICT: A Gaze-Assisted Translation Aid. In Proceedings of the 2000 Symposium on Eye Tracking Research & Applications (Palm Beach Gardens, Florida, USA) (ETRA ’00). Association for Computing Machinery, New York, NY, USA, 9–14. https://doi.org/10.1145/355017.355019Google ScholarDigital Library
- Joseph Marvin Imperial. 2021. BERT Embeddings for Automatic Readability Assessment. arxiv:2106.07935 [cs.CL]Google Scholar
- Syeda Iqbal. 2017. The Impact of Extensive Reading on Learning and Increasing Vocabulary at Elementary Level. Studies in English Language Teaching 5 (07 2017), 481. https://doi.org/10.22158/selt.v5n3p481Google ScholarCross Ref
- Shoya Ishimaru, Syed Saqib Bukhari, Carina Heisel, Nicolas Großmann, Pascal Klein, Jochen Kuhn, and Andreas Dengel. 2018. Augmented Learning on Anticipating Textbooks with Eye Tracking. Springer Fachmedien Wiesbaden, Wiesbaden, 387–398. https://doi.org/10.1007/978-3-658-19567-0_23Google ScholarCross Ref
- Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2021. Neural CRF Model for Sentence Alignment in Text Simplification. arxiv:2005.02324 [cs.CL]Google Scholar
- J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical Report. Institute for Simulation and Training, University of Central Florida.Google Scholar
- Victor Kuperman, Hans Stadthagen-Gonzalez, and Marc Brysbaert. 2012. Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods 44, 4 (2012), 978–990. https://doi.org/10.3758/s13428-012-0210-4Google ScholarCross Ref
- Shazia Maqsood, Abdul Shahid, Muhammad Afzal, Muhammad Roman, Zahid Khan, Zubair Nawaz, and Muhammad Aziz. 2022. Assessing English language sentences readability using machine learning models. PeerJ Computer Science 7 (01 2022), e818. https://doi.org/10.7717/peerj-cs.818Google ScholarCross Ref
- Louis Martin, Angela Fan, Éric de la Clergerie, Antoine Bordes, and Benoît Sagot. 2021. MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases. arxiv:2005.00352 [cs.CL]Google Scholar
- Eugene W. Myers. 1986. An O(ND) difference algorithm and its variations. Algorithmica 1 (1986), 251–266. https://doi.org/10.1007/BF01840446Google ScholarDigital Library
- Alireza Niazifar and Gholamreza Shakibaei. 2019. Effects of different text difficulty levels on Iranian EFL learners’ foreign language Reading motivation and Reading comprehension. Asian J. Second. Foreign. Lang. Educ. 4 (2019), 1–18. Issue 7. https://doi.org/10.1186/s40862-019-0070-xGoogle ScholarCross Ref
- Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. 2019. CatBoost: unbiased boosting with categorical features. arxiv:1706.09516 [cs.LG]Google Scholar
- Keith Rayner. 1998. Eye Movements in Reading and Information Processing: 20 Years of Research. Psychological Bulletin 124, 3 (November 1998), 372–422. https://doi.org/10.1037/0033-2909.124.3.372Google ScholarCross Ref
- Dario D. Salvucci and Joseph H. Goldberg. 2000. Identifying Fixations and Saccades in Eye-Tracking Protocols. In Proceedings of the 2000 Symposium on Eye Tracking Research & Applications (Palm Beach Gardens, Florida, USA) (ETRA ’00). Association for Computing Machinery, New York, NY, USA, 71–78. https://doi.org/10.1145/355017.355028Google ScholarDigital Library
- RJ Senter and Edgar A Smith. 1967. Automated readability index. Technical Report. Technical report, DTIC document.Google Scholar
- Garain Utpal, Pandit Onkar, Augereau Olivier, Okoso Ayano, and Kise Koichi. 2017. Identification of Reader Specific Difficult Words by Analyzing Eye Gaze and Document Content. 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) 01 (11 2017), 1346–1351. https://doi.org/10.1109/icdar.2017.221Google ScholarCross Ref
Index Terms
- Keep Eyes on the Sentence: An Interactive Sentence Simplification System for English Learners Based on Eye Tracking and Large Language Models
Recommendations
Gaze-Driven Sentence Simplification for Language Learners: Enhancing Comprehension and Readability
ICMI '23 Companion: Companion Publication of the 25th International Conference on Multimodal InteractionLanguage learners should regularly engage in reading challenging materials as part of their study routine. Nevertheless, constantly referring to dictionaries is time-consuming and distracting. This paper presents a novel gaze-driven sentence ...
Combining sentence similarities measures to identify paraphrases
It proposes a new paraphrase identification system based on lexical, syntactic, semantic analysis.It uses different machine learning algorithms to classify the paraphrase.The measure was evaluated using state-of-art dataset: Microsoft Paraphrase Corpus. ...
4th international workshop on pervasive eye tracking and mobile eye-based interaction
UbiComp '14 Adjunct: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct PublicationPrevious work on eye tracking and eye-based human-computer interfaces mainly concentrated on making use of the eyes in traditional desktop settings. With the recent growth of interest in smart glass devices and low-cost eye trackers, however, gaze-based ...
Comments