Communication-based Evaluation for Natural Language Generation
- Benjamin Newman (Stanford University)
- Reuben Cohn-Gordon
- Christopher Potts (Stanford University)
Abstract
Natural language generation (NLG) systems are commonly evaluated using n-gram overlap measures (e.g. BLEU, ROUGE). These measures do not directly capture semantics or speaker intentions, and so they often turn out to be misaligned with our true goals for NLG. In this work, we argue instead for communication-based evaluations: assuming the purpose of an NLG system is to convey information to a reader/listener, we can directly evaluate its effectiveness at this task using the Rational Speech Acts model of pragmatic language use. We illustrate with a color reference dataset that contains descriptions in pre-defined quality categories, showing that our method better aligns with these quality categories than do any of the prominent n-gram overlap methods.
Keywords: NLG Evaluation, Grounded Language Understanding, Rational Speech Acts
How to Cite:
Newman, B., Cohn-Gordon, R. & Potts, C., (2020) “Communication-based Evaluation for Natural Language Generation”, Society for Computation in Linguistics 3(1), 234-244. doi: https://doi.org/10.7275/scil.1174
Downloads:
Download PDF