Access this book
Tax calculation will be finalised at checkout
Other ways to access
Table of contents (8 chapters)
About this book
Data-driven experimental analysis has become the main evaluation tool of Natural Language Processing (NLP) algorithms. In fact, in the last decade, it has become rare to see an NLP paper, particularly one that proposes a new algorithm, that does not include extensive experimental analysis, and the number of involved tasks, datasets, domains, and languages is constantly growing. This emphasis on empirical results highlights the role of statistical significance testing in NLP research: If we, as a community, rely on empirical evaluation to validate our hypotheses and reveal the correct language processing mechanisms, we better be sure that our results are not coincidental.
The goal of this book is to discuss the main aspects of statistical significance testing in NLP. Our guiding assumption throughout the book is that the basic question NLP researchers and engineers deal with is whether or not one algorithm can be considered better than another one. This question drivesthe field forward as it allows the constant progress of developing better technology for language processing challenges. In practice, researchers and engineers would like to draw the right conclusion from a limited set of experiments, and this conclusion should hold for other experiments with datasets they do not have at their disposal or that they cannot perform due to limited time and resources. The book hence discusses the opportunities and challenges in using statistical significance testing in NLP, from the point of view of experimental comparison between two algorithms. We cover topics such as choosing an appropriate significance test for the major NLP tasks, dealing with the unique aspects of significance testing for non-convex deep neural networks, accounting for a large number of comparisons between two NLP algorithms in a statistically valid manner (multiple hypothesis testing), and, finally, the unique challenges yielded by the nature of the data and practices of the field.
Authors and Affiliations
About the authors
Segev Shlomov is a Ph.D. student under the supervision of Associate Professor Yakov Babichenko at the Technion, Israel Institute of Technology. Segev's research interests lie at the intersection of Statistics, Social Learning, and Information Theory. Segev holds an M.Sc. in Operations Research (summa cum laude) from the Technion. He was three-time a summer intern at the Artificial Intelligence department of the IBM research labs, Haifa, Israel, and he is one of the main contributors to IBM's Lambada AI service. Segev's papers have been published in top-tier conferences and journals of both the NLP community and the Economics and Computation communities. Segev is a recipient of the Jacobs Outstanding Ph.D. Scholarship for the year 2020.
Roi Reichart is an Associate Professor at the Technion, Israel Institute of Technology. Before joining the Technion, on July 2013, he was a post-doctoral researcher at the Computer Laboratory of the University of Cambridge, UK, and at the Computer Science and Artificial Intelligence Laboratory (CSAIL) of MIT. Prior to this, he was a Ph.D. student under the supervision of Professor Ari Rappoport at the Interdisciplinary Center for Neural Computation (ICNC) of Hebrew University of Jerusalem. His main research interest is NLP, with a focus on language learning in its context and designing models that integrate domain and world knowledgewith data-driven methods. He has hence worked on problems such as domain adaptation, learning with minimal human annotation (and involvement), language transfer and multilingual learning, multi-modal (text and vision) processing, and NLP of Web data. He has focused on structured aspects of language and has developed effective algorithms for inference across linguistic structures. Finally, he is interested in proper evaluation of NLP algorithms and has worked on problems such as measuring statistical significance in NLP, word embedding evaluation, and unsupervised learning (particularly clustering) evaluation.
Bibliographic Information
Book Title: Statistical Significance Testing for Natural Language Processing
Authors: Rotem Dror, Lotem Peled-Cohen, Segev Shlomov, Roi Reichart
Series Title: Synthesis Lectures on Human Language Technologies
DOI: https://doi.org/10.1007/978-3-031-02174-9
Publisher: Springer Cham
eBook Packages: Synthesis Collection of Technology (R0), eBColl Synthesis Collection 9
Copyright Information: Springer Nature Switzerland AG 2020
Softcover ISBN: 978-3-031-01046-0Published: 03 April 2020
eBook ISBN: 978-3-031-02174-9Published: 01 June 2022
Series ISSN: 1947-4040
Series E-ISSN: 1947-4059
Edition Number: 1
Number of Pages: XVII, 98
Topics: Artificial Intelligence, Natural Language Processing (NLP), Computational Linguistics