Skip to main content

Advertisement

SpringerLink
Log in

Building referring expression corpora with and without feedback

  • Original Paper
  • Published:
Language Resources and Evaluation Aims and scope Submit manuscript

Abstract

The design of data collection experiments involving human participants is a common task in Referring Expression Generation (REG) and related fields. Many (or most) REG data collection tasks are implemented by making use of a human–computer (e.g., web-based) communicative setting, in which participants do not have any particular addressee in mind and do not receive any feedback regarding the appropriateness (e.g., uniqueness) of the descriptions that they produce. Others, at a possibly higher cost, make use of participant pairs engaged in some form of dialogue in which hearers may provide feedback allowing speakers to rephrase ambiguous or otherwise ill-formed descriptions. Leaving the issue of cost aside, however, it remains unclear whether the two methods elicit similar referring expressions for the purpose of REG research. To shed light on this issue, this paper presents a REG corpus built under three experimental conditions: a standard human–computer (or web-based) setting in which no feedback is available to the speaker, and two settings in which feedback regarding the appropriateness of the description may be provided either by an automated parsing tool or by a second participant at the receiving end of the communication. The corpus contains fully annotated descriptions in two domains—simple geometric objects and realistic human face images—and it is provided as a resource for the training and testing of REG algorithms in these communicative settings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. When there is no risk of confusion, we will hereby use the terms ‘speaker’ and ‘hearer’ for both spoken and written communication.

  2. https://ivandreparaboni.wixsite.com/research/feedbackre.

  3. A number of studies have also considered the issue of surface realisation of referring expressions (Hervás et al. 2015; Ferreira et al. 2018).

  4. However, in certain conditions participants were allowed to refer to screen positions as in, e.g., ‘the green chair on the top row’.

  5. More recent Dutch (Koolen and Krahmer 2010) and German (Howcroft et al. 2017) extensions of the TUNA corpus made use of a dialogue setting.

  6. Stimulus images courtesy of Michael J. Tarr, Center for the Neural Basis of Cognition and Department of Psychology, Carnegie Mellon Univ. Funding provided by NSF award 0339122.

  7. Images 03 to 13 in Dale and Viethen (2009).

  8. Images 01 to 11 in Paraboni et al. (2017)

References

  • Clarke, A. D. F., Elsner, M., & Rohde, H. (2013). Where’s Wally: The influence of visual salience on referring expression generation. Frontiers in Psychology, 4, 329.

    Google Scholar 

  • Dale, R., & Reiter, E. (1995). Computational interpretations of the Gricean maxims in the generation of referring expressions. Cognitive Science, 19, 233–263.

    Article  Google Scholar 

  • Dale, R., & Viethen, J. (2009). Referring expression generation through attribute-based heuristics. In 12th European workshop on natural language generation (pp. 58–65). Athens, Greece: Association for Computational Linguistics.

  • dos Santos Silva, D., & Paraboni, I. (2015). Generating spatial referring expressions in interactive 3D worlds. Spatial Cognition & Computation, 15(03), 186–225.

    Article  Google Scholar 

  • Ferreira, T. C., Moussallem, D., Kádár, Á., Wubben, S., & Krahmer, E. (2018). NeuralREG: An end-to-end approach to referring expression generation. In 56th annual meeting of the association for computational linguistics (1959–1969). Melbourne, Australia: Association for Computational Linguistics.

  • Ferreira, T. C., & Paraboni, I. (2014). Classification-based referring expression generation. In Computational linguistics and intelligent text processing (CICLing-2014), Lecture Notes in Computer Science 8403 (pp. 481–491). Kathmandu, Nepal: Springer.

  • Gargett, A., Garoufi, K., Koller, A., & Striegnitz, K. (2010). The GIVE-2 corpus of giving instructions in virtual environments. In Proceedings of LREC-2010. Valletta, Malta: ELRA.

  • Gatt, A., Belz, A., & Kow, E. (2008). The TUNA challenge 2008: Overview and evaluation results. In Fifth international natural language generation conference (INLG-2008) (pp. 198–206). Salt Fork, Ohio, USA: The Association for Computer Linguistics.

  • Gatt, A., & Krahmer, E. (2018). Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research, 61, 65–170.

    Article  Google Scholar 

  • Gatt, A., Sluis, I., & van der Deemter, K. van. (2007). Evaluating algorithms for the generation of referring expressions using a balanced corpus. In Proceedings of ENLG-07. Schloss Dagstuhl, Germany: Association for Computational Linguistics.

  • Goldberg, L. R. (1990). An alternative description of personality: The Big-Five factor structure. Journal of Personality and Social Psychology, 59, 1216–1229.

    Article  Google Scholar 

  • Hervás, R., Arroyo, J., Francisco, V., Peinado, F., & Gervás, P. (2015), 12. Influence of personal choices on lexical variability in referring expressions. Natural Language Engineering, 1–34 (in press).

  • Howcroft, D. M., Vogels, J., Demberg, V. (2017). G-TUNA: A corpus of referring expressions in German including duration information. In Proceedings of INLG-2017 (pp. 149–153). Santiago de Compostela, Spain: Association for Computational Linguistics.

  • Kazemzadeh, S., Ordonez, V., Matten, M., & Berg, T. (2014). ReferItGame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 787–798). Doha, Qatar: Association for Computational Linguistics.

  • Koolen, R., Gatt, A., Goudbeek, M., & Krahmer, E. (2011). Factors causing overspecification in definite descriptions. Journal of Pragmatics, 43, 3231–3250.

    Article  Google Scholar 

  • Koolen, R., & Krahmer, E. (2010). The D-TUNA corpus: A Dutch dataset for the evaluation of referring expression generation algorithms. Valletta: ELRA.

    Google Scholar 

  • Krahmer, E., & van Deemter, K. (2012). Computational generation of referring expressions: A survey. Computational Linguistics, 38(1), 173–218.

    Article  Google Scholar 

  • Krahmer, E., van Erk, S., & Verleg, A. (2003). Graph-based generation of referring expressions. Computational Linguistics, 29(1), 53–72.

    Article  Google Scholar 

  • Paraboni, I., Galindo, M., & Iacovelli, D. (2017). Stars2: A corpus of object descriptions in a visual domain. Language Resources and Evaluation, 51(2), 439–462.

    Article  Google Scholar 

  • Paraboni, I., Monteiro, D. S., & Lan, A. G. J. (2017). Personality-dependent referring expression generation. In Text, Speech and Dialogue (TSD-2017) Lecture Notes in Artificial Intelligence. vol. 10415 (pp. 20–28). Prague, Czech Republic: Springer-Verlag.

  • Paraboni, I., & van Deemter, K. (1999). Issues for the generation of document deixis. In Procs. of workshop on deixis, demonstration and deictic belief in multimedia contexts, in association with the 11th Eeuropean summers school in logic, language and information (esslli99) (pp. 44–48).

  • Ramos, R. M. S., Neto, G. B. S., Silva, B. B. C., Monteiro, D. S., Paraboni, I., & Dias, R. F. S. (2018). Building a corpus for personality-dependent natural language understanding and generation. In 11th international conference on language resources and evaluation (LREC-2018) (pp. 1138–1145). Miyazaki, Japan: ELRA.

  • Righi, G., Peissig, J. J., & Tarr, M. J. (2012). Recognizing disguised faces. Visual Cognition, 20(2), 143–169.

    Article  Google Scholar 

  • Silva, B. B. C., & Paraboni, I. (2018). Learning personality traits from Facebook text. IEEE Latin America Transactions, 16(4), 1256–1262.

    Article  Google Scholar 

  • Tarenskeen, S., Broersma, M., & Geurts, B. (2014). Referential overspecification: Colour is not that special. In Refnet workshop on psychological and computational models of reference comprehension and production. Edinburgh, UK.

  • Teixeira, C. V. M., Paraboni, I., Silva, A. S. R., & da Yamasaki, A. K. (2014). Generating relational descriptions involving mutual disambiguation. In Computational linguistics and intelligent text processing (CICLing-2014), Lecture Notes in Computer Science 8403 (pp. 492–502). Kathmandu, Nepal: Springer.

  • van Deemter, K. (2002). Generating referring expressions: Boolean extensions of the incremental algorithm. Computational Linguistics, 28(1), 37–52.

    Article  Google Scholar 

  • van Deemter, K. (2016). Computational models of referring. A study in cognitive science. Cambridge: MIT Press.

    Book  Google Scholar 

  • van Gompel, R., Gatt, A., Krahmer, E., & Deemter, K. V. (2014). Testing computational models of reference generation as models of human language production: The case of size contrast. In Refnet workshop on psychological and computational models of reference comprehension and production. Edinburgh, UK.

  • Viethen, J., & Dale, R. (2010). Speaker-dependent variation in content selection for referring expression generation. In Proceedings of the Australasian language technology association workshop 2010 (pp. 81–89). Melbourne, Australia: Australasian Language Technology Association.

  • Viethen, J., Mitchell, M., & Krahmer, E. (2013). Graphs and spatial relations in the generation of referring expressions. In 14th European workshop on natural language generation (pp. 72–81). Sofia, Bulgaria: Association for Computational Linguistics.

Download references

Acknowledgements

This work has been supported by Grant # 2016/14223-0, São Paulo Research Foundation (FAPESP).

Author information

Authors and Affiliations

  1. School of Arts, Sciences and Humanities (EACH), University of São Paulo (USP), São Paulo, Brazil

    Danillo da Silva Rocha & Ivandré Paraboni

Authors
  1. Danillo da Silva Rocha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ivandré Paraboni
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ivandré Paraboni.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

da Silva Rocha, D., Paraboni, I. Building referring expression corpora with and without feedback. Lang Resources & Evaluation 54, 875–891 (2020). https://doi.org/10.1007/s10579-020-09497-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10579-020-09497-2

Keywords

Search

Navigation