Abstract
In this work, we extend the task of evaluating summaries without human models by using a trivergent model. In this model, three elements are compared simultaneously: a summary to evaluate, its source document and a set of other summaries from the same source. We present in this paper, a first pilot experiment using a French corpus from which we obtained promising results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
We did not test the combinations given by the possible trivergence’s associative property.
- 5.
The corpus can be downloaded from: http://dev.termwatch.es/~fresa/CORPUS/PUCES/.
- 6.
- 7.
- 8.
We applied a Kruskal-Wallis test and a Games-Howell post hoc test as the results had heterogeneous variances.
References
Cabrera-Diego, L.A.: Automatic methods for assisted recruitment. Ph.D. thesis, Université d’Avignon et des Pays de Vaucluse, December 2015
Fernández, S., SanJuan, E., Torres-Moreno, J.M.: Textual energy of associative memories: performant applications of enertex algorithm in text summarization and topic segmentation. In: Gelbukh, A., Kuri Morales, A.F. (eds.) MICAI 2007. LNCS (LNAI), vol. 4827, pp. 861–871. Springer, Heidelberg (2007)
Gamer, M., Lemon, J., Singh, I.F.P.: irr: Various Coefficients of Interrater Reliability and Agreement (2012), rpackageversion0.84. https://CRAN.R-project.org/package=irr
Good, I.J.: The population frequencies of species and the estimation of population parameters. Biometrika 40(3–4), 237–264 (1953)
Hovy, E., Lin, C.Y., Zhou, L., Fukumoto, J.: Automated summarization evaluation with basic elements. In: Proceedings of the Fifth Conference on Language Resources and Evaluation (LREC 2006), pp. 604–611 (2006)
Jing, H., Barzilay, R., McKeown, K., Elhadad, M.: Summarization evaluation methods: experiments and analysis. In: AAAI Symposium on Intelligent Summarization, pp. 51–59 (1998)
Katz, S.M.: Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Trans. Acoustics Speech Signal Process. 35(3), 400–401 (1987)
Kendall, M.G., Stuart, A.: The Advanced Theory of Statistics, vol. 1, 2nd edn. Charles Griffin and Co., London (1948)
Kruskal, W.H., Wallis, W.A.: Use of ranks in one-criterion variance analysis. J. Am. Stat. Assoc. 47(260), 583–621 (1952)
Kullback, S., Leibler, R.A.: On information and sufficiency. Ann. Math. Stat. 22(1), 79–86 (1951)
Lin, C.Y.: Rouge: a package for automatic evaluation of summaries. In: Text Summarization Branches Out: Proceedings of the Association for Computational Linguistics 2004 Workshop, vol. 8 (2004)
Lin, J.: Divergence measures based on the Shannon entropy. IEEE Trans. Inf. Theory 37(1), 145–151 (1991)
Louis, A., Nenkova, A.: Automatically evaluating content selection in summarization without human models. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, vol. 1, pp. 306–314. Association for Computational Linguistics (2009)
Mani, I.: Summarization evaluation: an overview. In: Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL) Workshop on Automatic Summarization (2001)
Nenkova, A., Passonneau, R.: Evaluating content selection in summarization: the pyramid method. In: Susan Dumais, D.M., Roukos, S. (eds.) Proceedings of HLT-NAACL 2004, pp. 145–152. Association for Computational Linguistics, Boston (2004)
R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2016). https://www.R-project.org/
Saggion, H., Torres-Moreno, J.M., da Cunha, I., SanJuan, E., Velázquez-Morales, P.: Multilingual summarization evaluation without human models. In: 23rd International Conference on Computational Linguistics (COLING 2010), pp. 1059–1067. Association for Computational Linguistics (2010)
Spärck-Jones, K., Galliers, J.R.: Evaluating Natural Language Processing Systems: An Analysis and Review. LNCS(LNAI), vol. 1083. Springer, New York (1996)
Steinberger, J., Ježek, K.: Evaluation measures for text summarization. Comput. Inform. 28(2), 251–275 (2012)
Torres-Moreno, J.M.: Artex is another text summarizer (2012). arXiv:1210.3312
Torres-Moreno, J.M.: Automatic Text Summarization. Wiley, New York (2014)
Torres-Moreno, J.M.: Trivergence of Probability Distributions, at Glance. Computing Research Repository (CoRR) abs/1506.06205 (2015). http://arxiv.org/abs/1506.06205
Torres-Moreno, J.M., Velázquez-Morales, P., Meunier, J.G.: Condensés automatiques de textes. Lexicometrica. L’analyse de données textuelles: De l’enquête aux corpus littéraires, Special (2004)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Cabrera-Diego, L.A., Torres-Moreno, JM., Durette, B. (2016). Evaluating Multiple Summaries Without Human Models: A First Experiment with a Trivergent Model. In: Métais, E., Meziane, F., Saraee, M., Sugumaran, V., Vadera, S. (eds) Natural Language Processing and Information Systems. NLDB 2016. Lecture Notes in Computer Science(), vol 9612. Springer, Cham. https://doi.org/10.1007/978-3-319-41754-7_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-41754-7_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-41753-0
Online ISBN: 978-3-319-41754-7
eBook Packages: Computer ScienceComputer Science (R0)