Skip to main content
SpringerLink
Log in

Improving the readability and saliency of abstractive text summarization using combination of deep neural networks equipped with auxiliary attention mechanism

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Rapid and exponential development of textual data in recent years has yielded to the need for automatic text summarization models which aim to automatically condense a piece of text into a shorter version. Although various unsupervised and machine learning-based approaches have been introduced for text summarization during the last decades, the emergence of deep learning has made remarkable progress in this field. However, deep learning-based text summarization models are still in their early steps of development and their potential has yet to be fully explored. Accordingly, a novel abstractive summarization model is proposed in this paper which utilized the combination of convolutional neural network and long short-term memory integrated with auxiliary attention in its encoder to increase the saliency and coherency of generated summaries. The proposed model was validated on CNN\Daily Mail and DUC-2004 datasets and empirical results indicated that not only the proposed model outperformed existing models in terms of ROUGE metric but also its generated summaries had higher saliency and readability compared to the baseline model according to human evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Source text pre-processing steps

Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://pypi.org/project/pyrouge/.

  2. https://github.com/summanlp/evaluation/tree/master/ROUGE-RELEASE-1.5.5.

  3. www.github.com/abisee/pointer-generator.

  4. https://stanfordnlp.github.io/CoreNLP/.

References

  1. Al-Numai AM, Azmi AM (2020) The development of single-document abstractive text summarizer during the last decade. In: Fiori A (ed) Trends and applications of text summarization techniques. IGI Global, New York, pp 32–60

    Chapter  Google Scholar 

  2. Syed AA, Gaol FL, Matsuo T (2021) A survey of the state-of-the-art models in neural abstractive text summarization. IEEE Access 9:13248–13265

    Article  Google Scholar 

  3. Song S, Huang H, Ruan T (2019) Abstractive text summarization using LSTM-CNN based deep learning. Multimed Tools Appl 78(1):857–875

    Article  Google Scholar 

  4. Magdum P, Rathi S (2020) A survey on deep learning-based automatic text summarization models. In: Magdum PG, Rathi S (eds) Advances in artificial intelligence and data engineering. Springer, pp 377–392

  5. Abualigah L, Bashabsheh MQ, Alabool H, Shehab M (2020) Text summarization: a brief review. In: Elaziz MA, Al-qaness MAA, Ewees AA, Dahou A (eds) Recent Advances in NLP: the case of Arabic language. Springer, Berlin, pp 1–15

    Google Scholar 

  6. Dey M, Das D (2020) A deep dive into supervised extractive and abstractive summarization from text. In: Hemanth J, Bhatia M, Geman O (eds) Data visualization and knowledge engineering. Springer, Berlin, pp 109–132

    Chapter  Google Scholar 

  7. Mahajani A, Pandya V, Maria I, Sharma D (2019) A comprehensive survey on extractive and abstractive techniques for text summarization. In: Hu Y-C, Tiwari S, Mishra KK, Trivedi MC (eds) Ambient communications and computer systems. Springer, Berlin, pp 339–351

    Chapter  Google Scholar 

  8. Sohail A, Aslam U, Tariq HI, Jayabalan M (2020) Methodologies and techniques for text summarization: a survey. J Crit Rev 7(11):2020

    Google Scholar 

  9. Nallapati R, Zhou B, Gulcehre C, Xiang B (2016) Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv:160206023

  10. Wang Q, Ren J (2021) Summary-aware attention for social media short text abstractive summarization. Neurocomputing 425:290–299

    Article  Google Scholar 

  11. Zhang Y, Li D, Wang Y, Fang Y, Xiao W (2019) Abstract text summarization with a convolutional Seq2seq model. Appl Sci 9(8):1665

    Article  Google Scholar 

  12. Shi T, Keneshloo Y, Ramakrishnan N, Reddy CK (2021) Neural abstractive text summarization with sequence-to-sequence models. ACM Trans Data Sci 2(1):1–37

    Article  Google Scholar 

  13. Aksenov D, Moreno-Schneider J, Bourgonje P, Schwarzenberg R, Hennig L, Rehm G (2020) Abstractive text summarization based on language model conditioning and locality modeling. arXiv:200313027

  14. Anh DT, Trang NTT (2019) Abstractive text summarization using pointer-generator networks with pre-trained word embedding. In: Proceedings of the tenth international symposium on information and communication technology, pp 473–478

  15. Celikyilmaz A, Bosselut A, He X, Choi Y (2018) Deep communicating agents for abstractive summarization. arXiv:180310357

  16. Joshi A, Fernández E, Alegre E (2018) Deep learning based text summarization: approaches databases and evaluation measures. In: International Conference of Applications of Intelligent Systems

  17. Rane N, Govilkar S (2019) Recent trends in deep learning based abstractive text summarization. Int J Recent Technol Eng (IJRTE) 8(3):3108–3115

    Article  Google Scholar 

  18. Suleiman D, Awajan A (2020) Deep learning based abstractive text summarization: approaches, datasets, evaluation measures, and challenges. In: Mathematical problems in engineering 2020

  19. Mehta P, Majumder P (2019) From extractive to abstractive summarization: a journey. Springer, Berlin

    Book  Google Scholar 

  20. Yousefi-Azar M, Hamey L (2017) Text summarization using unsupervised deep learning. Expert Syst Appl 68:93–105

    Article  Google Scholar 

  21. Shirwandkar NS, Kulkarni S (2018) Extractive text summarization using deep learning. In: 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA). IEEE, pp 1–5

  22. Nallapati R, Zhai F, Zhou B (2017) Summarunner: a recurrent neural network based sequence model for extractive summarization of documents. In: Thirty-First AAAI Conference on Artificial Intelligence

  23. Cheng J, Lapata M (2016) Neural summarization by extracting sentences and words. arXiv:160307252

  24. Litvak M, Last M (2008) Graph-based keyword extraction for single-document summarization. In: Coling 2008: Proceedings of the workshop multi-source multilingual information extraction and summarization, pp 17–24

  25. Wong K-F, Wu M, Li W (2008) Extractive summarization using supervised and semi-supervised learning. In: Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pp 985–992

  26. Cao Z, Wei F, Li S, Li W, Zhou M, Wang H (2015) Learning summary prior representation for extractive summarization. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp 829–833

  27. Ren P, Chen Z, Ren Z, Wei F, Nie L, Ma J, De Rijke M (2018) Sentence relations for extractive summarization with deep neural networks. ACM Trans Inf Syst (TOIS) 36(4):1–32

    Article  Google Scholar 

  28. Verma S, Nidhi V (2017) Extractive summarization using deep learning. arXiv:170804439

  29. Rush AM, Chopra S, Weston J (2015) A neural attention model for abstractive sentence summarization. arXiv:150900685

  30. Chopra S, Auli M, Rush AM (2016) Abstractive sentence summarization with attentive recurrent neural networks. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp 93–98

  31. Lopyrev K (2015) Generating news headlines with recurrent neural networks. arXiv:151201712

  32. Suzuki J, Nagata M (2016) Cutting-off redundant repeating generations for neural abstractive summarization. arXiv:170100138

  33. Paulus R, Xiong C, Socher R (2017) A deep reinforced model for abstractive summarization. arXiv:170504304

  34. Vinyals O, Fortunato M, Jaitly N (2015) Pointer networks. In: Advances in neural information processing systems, pp 2692–2700

  35. Gu J, Lu Z, Li H, Li VO (2016) Incorporating copying mechanism in sequence-to-sequence learning. arXiv:160306393

  36. Gulcehre C, Ahn S, Nallapati R, Zhou B, Bengio Y (2016) Pointing the unknown words. arXiv:160308148

  37. See A, Liu PJ, Manning CD (2017) Get to the point: summarization with pointer-generator networks. arXiv:170404368

  38. Yao K, Zhang L, Du D, Luo T, Tao L, Wu Y (2018) Dual encoding for abstractive text summarization. IEEE Trans Cybern 50(3):985–996

    Article  Google Scholar 

  39. Li Z, Peng Z, Tang S, Zhang C, Ma H (2020) Text summarization method based on double attention pointer network. IEEE Access 8:11279–11288

    Article  Google Scholar 

  40. Xiang X, Xu G, Fu X, Wei Y, Jin L, Wang L (2018) Skeleton to abstraction: an attentive information extraction schema for enhancing the saliency of text summarization. Information 9(9):217

    Article  Google Scholar 

  41. Zhao H, Cao J, Xu M, Lu J (2020) Variational neural decoder for abstractive text summarization. Comput Sci Inf Syst 00:12–12

    Google Scholar 

  42. Lin C-Y (2004) Rouge: a package for automatic evaluation of summaries. In: Association for computational linguistic, Barcelona, Spain

  43. Hsu W-T, Lin C-K, Lee M-Y, Min K, Tang J, Sun M (2018) A unified model for extractive and abstractive summarization using inconsistency loss. arXiv:180506266

  44. Li P, Bing L, Lam W (2018) Actor-critic based training framework for abstractive summarization. arXiv:180311070

  45. Zhou Q, Yang N, Wei F, Zhou M (2017) Selective encoding for abstractive sentence summarization. arXiv:170407073

  46. Zhang H, Xu J, Wang J (2019) Pretraining-based natural language generation for text summarization. arXiv:190209243

  47. Liao P, Zhang C, Chen X, Zhou X (2020) Improving abstractive text summarization with history aggregation. In: 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, pp 1–9

  48. Mikolov T, Chen K, Corrado G, Dean J (2013) Distributed representations of words and phrases and their compositionality, Nips

  49. Sadr H, Pedram MM, Teshnehlab M (2019) A robust sentiment analysis method based on sequential combination of convolutional and recursive neural networks. Neural Process Lett 50:1–17

    Article  Google Scholar 

  50. Hailu TT, Yu J, Fantaye TG (2020) A framework for word embedding based automatic text summarization and evaluation. Information 11(2):78

    Article  Google Scholar 

  51. Sadr H, Pedram MM, Teshnehlab M (2020) Multi-view deep network: a deep model based on learning features from heterogeneous neural networks for sentiment analysis. IEEE Access 8:86984–86997

    Article  Google Scholar 

  52. Chakraborty S, Li X, Chakraborty S (2020) A more abstractive summarization model. arXiv:200210959

  53. Zhao F, Quan B, Yang J, Chen J, Zhang Y, Wang X (2019) Document summarization using word and part-of-speech based on attention mechanism. In: Journal of Physics: Conference Series, vol 3. IOP Publishing, p 032008

  54. Over P, Dang H, Harman D (2007) DUC in context. Inf Process Manag 43(6):1506–1520

    Article  Google Scholar 

  55. Fan A, Grangier D, Auli M (2017) Controllable abstractive summarization. arXiv:171105217

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran

    Hassan Aliakbarpour & Amir Masoud Rahmani

  2. Computer Engineering Department, Sharif University of Technology, Tehran, Iran

    Mohammad Taghi Manzuri

Authors
  1. Hassan Aliakbarpour
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Taghi Manzuri
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amir Masoud Rahmani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammad Taghi Manzuri.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aliakbarpour, H., Manzuri, M.T. & Rahmani, A.M. Improving the readability and saliency of abstractive text summarization using combination of deep neural networks equipped with auxiliary attention mechanism. J Supercomput 78, 2528–2555 (2022). https://doi.org/10.1007/s11227-021-03950-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-021-03950-x

Keywords

Search

Navigation