Abstract
This chapter aims to establish the significance of our research on the validity of TAPs. As a start, some definitional issues are clarified to ensure that further discussion of the validity issues is based on a uniform understanding of what TAPs are and how TAPs differ from other types of verbal reports, especially introspection. The second part of this chapter introduces how widely TAPs have been used in first language (L1) and L2 research, writing research in particular, to underline the importance of TAPs as a method to collect data on human thinking. Against this backdrop, this chapter then introduces the long-lasting validity issues and surmises consequences that might be caused due to TAPs’ lack of validity. The chapter finally points out the scarcity of empirical studies on the validity of TAPs in L2 writing.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Afflerbach, P., & Johnston, P. (1984). On the use of verbal reports in reading research. Journal of Reading Behavior, 16(4), 307–322.
Armengol, L., & Cots, J. (2009). Attention processes observed in think-aloud protocols: Two multilingual informants writing in two languages. Language Awareness, 18, 259–276.
Barkaoui, K. (2011). Think-aloud protocols in research on essay rating: An empirical study of their veridicality and reactivity. Language Testing, 28(1), 51–75.
Bellezza, F. S. (1986). Mental cues and verbal reports in learning. In G. H. Bower (Ed.), The psychology of learning and motivation (Vol. 20, pp. 237–273). Academic Press.
Bereiter, C., & Scardamalia, M. (1987). The psychology of written composition. Lawrence Erlbaum.
Bereiter, C., Burtis, P. J., & Scardamalia, M. (1988). Cognitive operations in constructing main points in written composition. Journal of Memory & Language, 27, 261–278.
Block, E. (1992). See how they read: Comprehension monitoring of L1 and L2 readers. TESOL Quarterly, 26, 319–343.
Bowles, M. A. (2008). Task type and reactivity of verbal reports in SLA: A first look at a L2 task other than reading. Studies in Second Language Acquisition, 30, 359–387.
Bowles, M. A. (2010). The think-aloud controversy in second language research. Routledge.
Bowles, M. A. (2018). Introspective verbal reports: Think-alouds and stimulated recall. In A. Phakiti, P. DeCosta, L. Plonsky, & S. Starfield (Eds.), The Palgrave handbook of applied linguistics research methodology (pp. 339–357). Palgrave Macmillan.
Bowles, M. A., & Leow, R. P. (2005). Reactivity and type of verbal report in SLA research methodology: Expanding the scope of investigation. Studies in Second Language Acquisition, 27, 415–440.
Breetvelt, I., Bergh, H., & Rijlaarsdam, G. (1994). Relations between writing processes and text quality: When and how? Cognition and Instruction, 12, 103–123.
Brinkman, J. A. (1993). Verbal protocol accuracy in fault diagnosis. Ergonomics, 36, 1381–1397.
Broderick, L., Bjorner, B. J., Lauher-Charest, M., White, K. M., Kosinski, M., Mulhern, B., & Brazier, J. (2022). Deciding between SF-6Dv2 health states: A think-aloud study of decision-making strategies used in discrete choice experiments. Value in Health. Advance online publication. doi:https://doi.org/10.1016/j.jval.2022.07.018.
Chang, C. C., & Johnson, T. (2021). Integrating heuristics and think-aloud approach to evaluate the usability of game-based learning material. Journal of Computers in Education, 8, 137–157.
Chenoweth, N. A., & Hayes, J. R. (2001). Fluency in writing: Generating text in L1 and L2. Written Communication, 18, 80–89.
Chon, Y. V. (2008). The electronic dictionary for writing: A solution or a problem? International Journal of Lexicography, 22(1), 23–54.
Claparede, E. (1934). Genèse de l’hypothèses (Genesis of the hypotheses). Archives de Psychologie, 24, 1–155.
Cohen, A. D. (1986). Mentalistic measures in reading strategy research: Some recent findings. English for Specific Purposes, 5, 131–145.
Cohen, A. D. (1991). Feedback on writing: The use of verbal reports. Studies in Second Language Acquisition, 13(2), 133–159.
Cohen, A. D. (2000). Exploring strategies in test taking: Fine-tuning verbal reports from respondents. In G. Ekbatani & H. Pierson (Eds.), Learner-directed assessment in ESL (pp. 127–150). Erlbaum.
Cohen, A. D. (2013). Verbal report. In C. A. Chapelle (General Ed.), The encyclopaedia of applied linguistics. Wiley-Blackwell. https://doi.org/10.1002/9781405198431.wbeal1261
Compton, B. J., & Logan, G. D. (1991). The transition from algorithm to retrieval in memory-based theories of automaticity. Memory & Cognition, 19, 151–158.
Cooper, M., & Holzman, M. (1983). Talking about protocols. College Composition and Communication, 34, 284–293.
Crutcher, R. J. (1994). Telling what we know: The use of verbal report methodologies in psychological research. Psychological Science, 5, 241–244.
Cumming, A. (1989). Writing expertise and second language proficiency. Language Learning, 39, 81–141.
Cumming, A. (1990a). Metalinguistic and ideational thinking in second language composing. Written Communication, 7(4), 482–511.
Cumming, A. (1990b). Expertise in evaluating second language compositions. Language Testing, 7, 31–51.
Cumming, A., Kantor, R., & Powers, D. (2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. Modern Language Journal, 86, 67–96.
Davis, J., & Bistodeau, L. (1993). How do L1 and L2 reading differ? Evidence from think aloud protocols. Modern Language Journal, 77, 459–472.
Deschambault, R. (2018). Actively managed products: Think-aloud data and methods in applied linguistics research. Applied Linguistics Review, 9(4), 539–562.
Duncker, K. (1926). A qualitative (experimental and theoretical) study of productive thinking (solving of comprehensible problems). Pedagogical Seminar, 33, 642–708.
Durst, R. K. (1987). Cognitive and linguistic demands of analytic writing. Research in the Teaching of English, 21, 347–376.
Ellis, R. (2001). Introduction: Investigating form-focused instruction. Language Learning, 51(Suppl.1), 1–46.
Enriquez Raido, V. (2014). Translation and web searching. Routledge.
Ericsson, K. A., & Crutcher, R. J. (1991). Introspection and verbal reports on cognitive processes—Two approaches to the study of thinking: A response to Howe. New Ideas in Psychology, 9, 57–71.
Ericsson, K. A., & Simon, H. A. (1978). RVRs as data (Unpublished Manuscript No. CIP Working Paper No°388). Carnegie-Mellon University.
Ericsson, K. A., & Simon, H. A. (1979). Thinking-aloud protocols as data: Effects of verbalization (Unpublished Manuscript No. CIP Working Paper No°397). Carnegie-Mellon University.
Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. MIT Press.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev. ed.). MIT Press.
Flower, L., & Hayes, J. (1980a). The dynamics of composing: Making plans and juggling constraints. In L. Gregg & E. Steinberg (Eds.), Cognitive processes in writing: An interdisciplinary approach (pp. 31–50). Lawrence Erlbaum.
Flower, L., & Hayes, J. (1980b). The cognition of discovery: Defining rhetorical problems. College Composition and Communication, 31, 21–32.
Flower, L., & Hayes, J. (1981a). A cognitive process theory of writing. College Composition and Communication, 32, 365–387.
Flower, L., & Hayes, J. (1981b). Plans that guide the composing process. In C. H. Frederiksen & J. F. Dominic (Eds.), Writing: The nature, development, and teaching of written communication (pp. 39–58). Erlbaum.
Flower, L., & Hayes, J. (1981c). The pregnant pause: An inquiry into the nature of planning. Research in the Teaching of English, 15(3), 229–243.
Flower, L., & Hayes, J. (1983). A cognitive model of the writing process in adults (Final report). Carnegie Mellon. (ERIC Document Reproduction Service No. ED 240 608).
Folger, T. L. (2001). Readers’ parallel text construction while talking and thinking about the reading process. Dissertation Abstracts International, 62, 1329A.
Fox, M. C., Ericsson, K. A., & Best, R. (2011). Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychological Bulletin, 137, 316–344.
Gass, S., & Mackey, A. (2000). Stimulated recall methodology in second language research. Lawrence Erlbaum.
Ghanbari, N., & Barati, H. (2020). Development and validation of a rating scale for Iranian EFL academic writing assessment: A mixed-methods study. Language Testing in Asia, 10, 17. https://doi.org/10.1186/s40468-020-00112-3
Godfroid, A., & Spino, L. (2015). Reconceptualizing reactivity of think-alouds and eye-tracking: Absence of evidence is not evidence of absence. Language Learning, 65(4), 896–928.
Green, A. (1998). Verbal protocol analysis in language testing research: A handbook. Cambridge University Press.
Green, S., & Sutton, P. (2003). What do children think as they plan their writing? Reading: Literacy and Language, 37, 32–38.
Harwood, N. (2018). What do proofreaders of student writing do to a master’s essay? Differing interventions, worrying findings. Written Communication, 35(4), 474–530.
Hayes, J. R., & Flower, L. S. (1980). Identifying the organization of writing processes. In L. W. Gregg & E. R. Steinberg (Eds.), Cognitive processes in writing: An interdisciplinary approach (pp. 3–30). Erlbaum.
Hayes, J. R., & Flower, L. S. (1983). Uncovering cognitive processes in writing: An introduction to protocol analysis. In P. Mosenthal, L. Tamor, & S. A. Walmsley (Eds.), Research on writing: Principles and methods (pp. 207–220). Longman.
Hayes, J. R., & Flower, L. S. (1986). Writing research and the writer. American Psychologist, 41, 1106–1113.
Hayes, J. R., Flower, L., Schriver, K., Stratman, J., & Carey, L. (1987). Cognitive processes in revision. In S. Rosenberg (Ed.), Advances in applied linguistics (Vol. 11: Reading, writing and language processing) (pp. 176–240). Cambridge University Press.
Jääskeläinen, R. (2000). Focus on methodology in think-aloud studies on translating. In S. Tirkkonen-Condit & R. Jääskeläinen (Eds.), Tapping and mapping the processes of translation and interpreting: Outlooks on empirical research (pp. 71–82). Benjamins.
Jääskeläinen, R. (2017). Verbal reports. In J. W. Schwieter & A. Ferreira (Eds.), The handbook of translation and cognition (pp. 213–231) Wiley-Blackwell.
Janssen, D., van Waes, L., & van den Bergh, H. (1996). Effects of thinking aloud on writing processes. In C. M. Levy & S. Ransdell (Eds.), The science of writing (pp. 233–250). Erlbaum.
Jourdenais, R. (2001). Cognition, instruction, and protocol analysis. In P. Robinson (Ed.), Cognition and second language instruction (pp. 354–375). Cambridge University Press.
Kern, R. G. (1994). The role of mental translation in second language reading. Studies in Second Language Acquisition, 16, 441–461.
Langer, J. (1986). Children reading and writing: Structures and strategies. Ablex.
Lee, C. (2020). A study of adolescent English learners’ cognitive engagement in writing while using an automated content feedback system. Computer Assisted Language Learning, 33, 26–57.
Leow, R. P. (1997). Attention, awareness, and foreign language behaviour. Language Learning, 47, 467–505.
Leow, R. P. (1998). Toward operationalizing the process of attention in SLA: Evidence for Tomlin and Villa’s (1994) fine-grained analysis of attention. Applied PsychoLinguistics, 19, 133–159.
Leow, R. P. (2000). A study of the role of awareness in foreign language behaviour: Aware versus unaware learners. Studies in Second Language Acquisition, 22, 557–584.
Leow, R. P. (2001a). Attention, awareness, and foreign language behaviour. Language Learning, 51(Suppl.1), 113–155.
Leow, R. P. (2001b). Do learners notice enhanced forms while interacting with the L2? An online and offline study of the role of written input enhancement in L2 reading. Hispania, 84, 496–509.
Leow, R. P., & Morgan-Short, K. (2004). To think aloud or not to think aloud: The issue of reactivity in SLA research methodology. Studies in Second Language Acquisition, 26, 35–57.
Lutz, J. A. (1987). A study of professional and experienced writers’ revising and editing at the computer and with pen and paper. Research in the Teaching of English, 21, 398–421.
Lyons, W. (1986). The disappearance of introspection. MIT Press.
Manchón, R. M., Roca de Larios, J., & Murphy, L. (2009). The temporal dimension and problem-solving nature of foreign language composing processes: Implications for theory. In R. M. Manchón (Ed.), Writing in foreign language contexts: Learning, teaching, and research (pp. 102–129). Multilingual Matters.
Matsuhashi, A. (1987). Revising the plan or altering the text. In A. Matsuhashi (Ed.), Writing in real time: Modeling production processes (pp. 197–223). Ablex.
McGeorge, P., & Burton, A. M. (1989). The effects of concurrent verbalization on performance in a dynamic systems task. British Journal of Psychology, 80, 455–465.
Newell, A., & Simon, H. A. (1972). Human problem solving. Prentice-Hall.
Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231–259.
Olson, G. M., Duffy, S. A., & Mack, R. L. (1984). Thinking-out-loud as a method for studying real-time comprehension processes. In D. E. Kieras & M. A. Just (Eds.), New methods in reading comprehension research (pp. 253–286). Erlbaum.
Payne, J. W. (1994). Thinking aloud: Insights into information processing. Psychological Science, 5(241), 245–248.
Payne, J. W., Braunstein, M. L., & Carroll, J. S. (1978). Exploring predecisional behaviour: An alternative approach to decision research. Organizational Behavior and Human Performance, 22, 17–44.
Qi, D., & Lapkin, S. (2001). Exploring the role of noticing in a three-stage second language writing task. Journal of Second Language Writing, 10, 277–303.
Révész, A., & Michel, M. (2019). Introduction. Studies in Second Language Acquisition, 41(3), 491–501.
Roca de Larios, J., Marín, J., & Murphy, L. (2001). A temporal analysis of formulation processes in L1 and L2 writing. Language Learning, 51, 497–583.
Ronowicz, E., Hehir, J., Kaimi, T., Kojima, K., & Lee, D.-S. (2005). Translator’s frequent lexis store and dictionary use as factors in SLT comprehension and translation speed: A comparative study of professional, paraprofessional and novice translators. Meta, 50, 580–596.
Rosa, E., & Leow, R. P. (2004a). Awareness, different learning conditions, and L2 development. Applied PsychoLinguistics, 25, 269–292.
Rosa, E., & Leow, R. P. (2004b). Computerized task-based exposure, explicitness, type of feedback, and Spanish L2 development. Modern Language Journal, 88, 192–216.
Rowe, H. (1985). The “thinking-aloud” methodology. In R. J. Sternberg (Ed.), Problem solving and intelligence (pp. 99–117). Erlbaum.
Russo, J. E., Johnson, E. J., & Stephens, D. L. (1989). The validity of verbal protocols. Memory & Cognition, 17, 759–769.
Sachs, R., & Polio, C. (2007). Learners’ uses of two types of written feedback on an L2 writing revision task. Studies in Second Language Acquisition, 29, 67–100.
Schmidt, H., Brandt, D., Bischof, A., Heidbrink, S., Bischof, G., Borgwardt, S., & Rumpf, H. (2022). Think-aloud analysis of commonly used screening instruments for Internet use disorders: The CIUS, the IGDT-10, and the BSMAS. Journal of Behavioral Addictions, 11(2), 467–480.
Selfe, C. L. (1984). The predrafting processes of four high and four low apprehensive writers. Research in the Teaching of English, 18, 45–64.
Severinson Eklundh, K., & Kollberg, P. (1996). A computer tool and framework for analyzing online revisions. In C. M. Levy & S. Ransdell (Eds.), The science of writing (pp. 163–188). Lawrence Erlbaum Associates.
Siegler, R. S. (1987). The perils of averaging data over strategies: An example from children’s addition. Journal of Experimental Psychology: General, 116, 250–264.
Siegler, R. S. (1989). Hazards of mental chronometry: An example from children’s subtraction. Journal of Educational Psychology, 81, 497–506.
Smagorinsky, P. (1989). The reliability and validity of protocol analysis. Written Communication, 6, 463–479.
Stratman, J. F., & Hamp-Lyons, L. (1994). Reactivity in concurrent think-aloud protocols: Issues for research. In P. Smagorinsky (Ed.), Speaking about writing: Reflections on research methodology (pp. 89–112). Sage.
Suh, B. (2020). Are think-alouds reactive? Evidence from an L2 written corrective feedback study. Language Teaching Research. Advance online publication. https://doi.org/10.1177/1362168820967166.
Sun, S., Li, T., & Zhou, X. (2020). Effects of thinking aloud on cognitive effort in translation. Linguistica Antverpiensia, New Series: Themes in Translation Studies, 19, 132–151.
Swain, M., & Lapkin, S. (1995). Problems in output and the cognitive processes they generate: A step towards second language learning. Applied Linguistics, 16, 371–391.
Swarts, H., Flower, L. S., & Hayes, J. R. (1984). Designing protocol studies of the writing process: An introduction. In R. Beach & L. Bridwell (Eds.), New directions in composition research (pp. 53–71). Guilford Press.
Timpe-Laughlin, V., Green, A., & Oh, S. (2021). Raising pragmatic awareness: A think-aloud study. System, 98, 1–15. https://doi.org/10.1016/j.system.2021.102470
Trabasso, T., & Suh, S. (1993). Understanding text: Achieving explanatory coherence through on-line inferences and mental operations in working memory. Discourse Processes, 16, 3–34.
Upton, T. A., & Lee-Thompson, L. C. (2001). The role of the first language in second language reading. Studies in Second Language Acquisition, 23, 469–495.
Uzawa, K. (1996). Second language learners’ processes of L1 writing, L2 writing, and translation from L1 into L2. Journal of Second Language Writing, 5, 271–294.
Vaughn, C. (1991). Holistic assessment: What goes on in the raters’ minds? In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 111–126). Ablex.
Vieira, L. N. (2017). Cognitive effort and different task foci in post-editing of machine translation: A think-aloud study. Across Languages and Cultures, 18(1), 79–105.
Wang, W. Y., & Wen, Q. F. (2002). L1 use in the L2 composing process: An exploratory study of 16 Chinese EFL writers. Journal of Second Language Writing, 11, 225–246.
Whalen, K., & Ménard, N. (1995). L1 and L2 writers’ strategic and linguistic knowledge: A model of multiple-level discourse processing. Language Learning, 44(3), 381–418.
Wigglesworth, G. (2005). Current approaches to researching second language learner processes. Annual Review of Applied Linguistics, 25, 98–111.
Wijgh, I. F. (1996). A communicative test in analysis: Strategies in reading authentic texts. In A. Cumming & R. Berwick (Eds.), Validation in language testing (pp. 154–170). Multilingual Matters.
Willey, I., & Tanimoto, K. (2015). We’re drifting into strange territory here: What think- aloud protocols reveal about convenience editing. Journal of Second Language Writing, 27, 63–83.
Wilson, T. D. (1994). The proper protocol: Validity and completeness of verbal reports. Psychological Science, 5, 249–252.
Witte, S. P. (1987). Pre-text and composing. College Composition and Communication, 38(4), 397–425.
Wolfe, E., Kao, C., & Ranney, M. (1998). Cognitive differences in proficient and nonproficient essay scorers. Written Communication, 15, 465–492.
Yang, C. S., Hu, G. W., & Zhang, L. J. (2014). Reactivity of concurrent verbal reporting in second language writing. Journal of Second Language Writing, 24(1), 51–70.
Yang, C. S., Zhang, L. J., & Parr, J. (2020). The reactivity of think-alouds in writing research: Quantitative and qualitative evidence from writing in English as a foreign language. Reading and Writing, 33(2), 451–483.
Yanguas, I., & Lado, B. (2012). Is thinking aloud reactive when writing in the heritage language? Foreign Language Annals, 45, 380–399.
Yeari, M., & Lantin, S. (2021). The origin of centrality deficit in text memory and comprehension by poor comprehenders: A think-aloud study. Reading & Writing, 34, 595–625.
Zellermayer, M., & Cohen, J. (1996). Varying paths for learning to revise. Instructional Science, 24, 177–195.
Zhang, L. J., & Zhang, D. L. (2020). Think-aloud protocols. In J. McKinley & H. Rose (Eds.), The Routledge handbook of research methods in applied linguistics (pp. 302–311). Routledge.
Zimmermann, R. (2000). L2 writing: Subprocesses, a model of formulating and empirical findings. Learning and Instruction, 10, 73–99.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Yang, C., Zhang, L.J. (2023). Introduction. In: Think-Aloud Protocols in Second Language Writing. English Language Education, vol 34. Springer, Cham. https://doi.org/10.1007/978-3-031-39574-1_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-39574-1_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-39573-4
Online ISBN: 978-3-031-39574-1
eBook Packages: EducationEducation (R0)