Skip to main content
SpringerLink
Log in

WiSeBE: Window-Based Sentence Boundary Evaluation

  • Conference paper
  • First Online:
Advances in Computational Intelligence (MICAI 2018)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11289))

Included in the following conference series:

Abstract

Sentence Boundary Detection (SBD) has been a major research topic since Automatic Speech Recognition transcripts have been used for further Natural Language Processing tasks like Part of Speech Tagging, Question Answering or Automatic Summarization. But what about evaluation? Do standard evaluation metrics like precision, recall, F-score or classification error; and more important, evaluating an automatic system against a unique reference is enough to conclude how well a SBD system is performing given the final application of the transcript? In this paper we propose Window-based Sentence Boundary Evaluation (WiSeBE), a semi-supervised metric for evaluating Sentence Boundary Detection systems based on multi-reference (dis)agreement. We evaluate and compare the performance of different SBD systems over a set of Youtube transcripts using WiSeBE and standard metrics. This double evaluation gives an understanding of how WiSeBE is a more reliable metric for the SBD task.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://dictionary.cambridge.org/.

  2. 2.

    https://www.ldc.upenn.edu/.

References

  1. Bohac, M., Blavka, K., Kucharova, M., Skodova, S.: Post-processing of the recognized speech for web presentation of large audio archive. In: 2012 35th International Conference on Telecommunications and Signal Processing (TSP), pp. 441–445. IEEE (2012)

    Google Scholar 

  2. Brum, H., Araujo, F., Kepler, F.: Sentiment analysis for Brazilian portuguese over a skewed class corpora. In: Silva, J., Ribeiro, R., Quaresma, P., Adami, A., Branco, A. (eds.) PROPOR 2016. LNCS (LNAI), vol. 9727, pp. 134–138. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-41552-9_14

    Chapter  Google Scholar 

  3. Che, X., Wang, C., Yang, H., Meinel, C.: Punctuation prediction for unsegmented transcript based on word vector. In: LREC (2016)

    Google Scholar 

  4. Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychol. Bull. 76(5), 378 (1971)

    Article  Google Scholar 

  5. Fohr, D., Mella, O., Illina, I.: New paradigm in speech recognition: deep neural networks. In: IEEE International Conference on Information Systems and Economic Intelligence (2017)

    Google Scholar 

  6. González-Gallardo, C.E., Hajjem, M., SanJuan, E., Torres-Moreno, J.M.: Transcripts informativeness study: an approach based on automatic summarization. In: Conférence en Recherche d’Information et Applications (CORIA), Rennes, France, May (2018)

    Google Scholar 

  7. González-Gallardo, C.E., Torres-Moreno, J.M.: Sentence boundary detection for French with subword-level information vectors and convolutional neural networks. arXiv preprint arXiv:1802.04559 (2018)

  8. Gotoh, Y., Renals, S.: Sentence boundary detection in broadcast speech transcripts. In: ASR2000-Automatic Speech Recognition: Challenges for the new Millenium ISCA Tutorial and Research Workshop (ITRW) (2000)

    Google Scholar 

  9. Hinton, G., et al.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012)

    Article  Google Scholar 

  10. Jamil, N., Ramli, M.I., Seman, N.: Sentence boundary detection without speech recognition: a case of an under-resourced language. J. Electr. Syst. 11(3), 308–318 (2015)

    Google Scholar 

  11. Kiss, T., Strunk, J.: Unsupervised multilingual sentence boundary detection. Comput. Linguist. 32(4), 485–525 (2006)

    Article  Google Scholar 

  12. Klejch, O., Bell, P., Renals, S.: Punctuated transcription of multi-genre broadcasts using acoustic and lexical approaches. In: 2016 IEEE Spoken Language Technology Workshop (SLT), pp. 433–440. IEEE (2016)

    Google Scholar 

  13. Kolář, J., Lamel, L.: Development and evaluation of automatic punctuation for French and english speech-to-text. In: Thirteenth Annual Conference of the International Speech Communication Association (2012)

    Google Scholar 

  14. Kolář, J., Švec, J., Psutka, J.: Automatic punctuation annotation in Czech broadcast news speech. In: SPECOM 2004 (2004)

    Google Scholar 

  15. Liu, Y., Chawla, N.V., Harper, M.P., Shriberg, E., Stolcke, A.: A study in machine learning from imbalanced data for sentence boundary detection in speech. Comput. Speech Lang. 20(4), 468–494 (2006)

    Article  Google Scholar 

  16. Lu, W., Ng, H.T.: Better punctuation prediction with dynamic conditional random fields. In: Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. pp. 177–186. Association for Computational Linguistics (2010)

    Google Scholar 

  17. Meteer, M., Iyer, R.: Modeling conversational speech for speech recognition. In: Conference on Empirical Methods in Natural Language Processing (1996)

    Google Scholar 

  18. Mrozinski, J., Whittaker, E.W., Chatain, P., Furui, S.: Automatic sentence segmentation of speech for automatic summarization. In: 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, vol. 1, p. I. IEEE (2006)

    Google Scholar 

  19. Palmer, D.D., Hearst, M.A.: Adaptive sentence boundary disambiguation. In: Proceedings of the Fourth Conference on Applied Natural Language Processing, pp. 78–83. ANLC 1994. Association for Computational Linguistics, Stroudsburg, PA, USA (1994)

    Google Scholar 

  20. Palmer, D.D., Hearst, M.A.: Adaptive multilingual sentence boundary disambiguation. Comput. Linguist. 23(2), 241–267 (1997)

    Google Scholar 

  21. Pearson, K.: Note on regression and inheritance in the case of two parents. Proc. R. Soc. Lond. 58, 240–242 (1895)

    Article  Google Scholar 

  22. Peitz, S., Freitag, M., Ney, H.: Better punctuation prediction with hierarchical phrase-based translation. In: Proceedings of the International Workshop on Spoken Language Translation (IWSLT), South Lake Tahoe, CA, USA (2014)

    Google Scholar 

  23. Rott, M., Červa, P.: Speech-to-text summarization using automatic phrase extraction from recognized text. In: Sojka, P., Horák, A., Kopeček, I., Pala, K. (eds.) TSD 2016. LNCS (LNAI), vol. 9924, pp. 101–108. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-45510-5_12

    Chapter  Google Scholar 

  24. Shriberg, E., Stolcke, A.: Word predictability after hesitations: a corpus-based study. In: Proceedings of the Fourth International Conference on Spoken Language, 1996. ICSLP 1996, vol. 3, pp. 1868–1871. IEEE (1996)

    Google Scholar 

  25. Stevenson, M., Gaizauskas, R.: Experiments on sentence boundary detection. In: Proceedings of the sixth conference on Applied natural language processing, pp. 84–89. Association for Computational Linguistics (2000)

    Google Scholar 

  26. Stolcke, A., Shriberg, E.: Automatic linguistic segmentation of conversational speech. In: Proceedings of the Fourth International Conference on Spoken Language, 1996. ICSLP 1996, vol. 2, pp. 1005–1008. IEEE (1996)

    Google Scholar 

  27. Strassel, S.: Simple metadata annotation specification v5. 0, linguistic data consortium (2003). http://www.ldc.upenn.edu/projects/MDE/Guidelines/SimpleMDE_V5.0.pdf

  28. Tilk, O., Alumäe, T.: Bidirectional recurrent neural network with attention mechanism for punctuation restoration. In: Interspeech 2016 (2016)

    Google Scholar 

  29. Treviso, M.V., Shulby, C.D., Aluisio, S.M.: Evaluating word embeddings for sentence boundary detection in speech transcripts. arXiv preprint arXiv:1708.04704 (2017)

  30. Ueffing, N., Bisani, M., Vozila, P.: Improved models for automatic punctuation prediction for spoken and written text. In: Interspeech, pp. 3097–3101 (2013)

    Google Scholar 

  31. Wang, W., Tur, G., Zheng, J., Ayan, N.F.: Automatic disfluency removal for improving spoken language translation. In: 2010 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pp. 5214–5217. IEEE (2010)

    Google Scholar 

  32. Xu, C., Xie, L., Huang, G., Xiao, X., Chng, E.S., Li, H.: A deep neural network approach for sentence boundary detection in broadcast news. In: Fifteenth Annual Conference of the International Speech Communication Association (2014)

    Google Scholar 

  33. Yu, D., Deng, L.: Automatic Speech Recognition. Springer, London (2015). https://doi.org/10.1007/978-1-4471-5779-3

    Book  MATH  Google Scholar 

Download references

Acknowledgments

We would like to acknowledge the support of CHIST-ERA for funding this work through the Access Multilingual Information opinionS (AMIS), (France - Europe) project.

We also like to acknowledge the support given by the Prof. Hanifa Boucheneb from VERIFORM Laboratory (École Polytechnique de Montréal).

Author information

Authors and Affiliations

  1. LIA - Université d’Avignon et des Pays de Vaucluse, 339 chemin des Meinajaries, 84140, Avignon, France

    Carlos-Emiliano González-Gallardo & Juan-Manuel Torres-Moreno

  2. Département de GIGL, École Polytechnique de Montréal, C.P. 6079, succ. Centre-ville, Montréal, Québec, H3C 3A7, Canada

    Carlos-Emiliano González-Gallardo & Juan-Manuel Torres-Moreno

Authors
  1. Carlos-Emiliano González-Gallardo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juan-Manuel Torres-Moreno
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carlos-Emiliano González-Gallardo .

Editor information

Editors and Affiliations

  1. Instituto Politécnico Nacional, Mexico City, Mexico

    Ildar Batyrshin

  2. Universidad Panamericana, Mexico City, Mexico

    María de Lourdes Martínez-Villaseñor

  3. Faculty of Engineering, Universidad Panamericana, Mexico City, Mexico

    Hiram Eredín Ponce Espinosa

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

González-Gallardo, CE., Torres-Moreno, JM. (2018). WiSeBE: Window-Based Sentence Boundary Evaluation. In: Batyrshin, I., Martínez-Villaseñor, M., Ponce Espinosa, H. (eds) Advances in Computational Intelligence. MICAI 2018. Lecture Notes in Computer Science(), vol 11289. Springer, Cham. https://doi.org/10.1007/978-3-030-04497-8_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-04497-8_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-04496-1

  • Online ISBN: 978-3-030-04497-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation