Skip to main content
SpringerLink
Log in

A knowledge extraction framework for domain-specific application with simplified pre-trained language model and attention-based feature extractor

  • Original Research
  • Published:
Service Oriented Computing and Applications Aims and scope Submit manuscript

Abstract

With the advancement of industrial informatics, intelligent algorithms are increasingly applied in various industrial products and applications. In this paper, we proposed a knowledge extraction framework for domain-specific text. This framework can extract entities from text the subsequent tasks such as knowledge graph construction. The proposed framework contains three modules, namely domain feature pre-trained model, LSTM-based named entity recognition and the attention-based nested named entity recognition. The domain feature pre-trained model can effectively learn the features of domain corpus such as professional terms that are not included in the general domain corpus. Flat named entity recognition can use the vector from pre-trained model to obtain the entity from domain-specific text. The nested named entity recognition based on the attention mechanism and the weight sliding balance strategy can effectively identify entity types with higher nesting rates. The framework achieves good results in the field of nuclear power plant maintenance reports, and the methods for domain pre-trained model and LSTM-based flat named entity recognition have been successfully applied to practical tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. The results that are different from those in paper[19, 27] are reproduced with same embeddings.

References

  1. Akbik A, Blythe D, Vollgraf R (2018) Contextual string embeddings for sequence labeling. In: COLING, pp 1638–1649

  2. Akbik A, Bergmann T, Blythe D et al (2019) FLAIR: an easy-to-use framework for state-of-the-art NLP. In: NAACL, pp 54–59. https://doi.org/10.18653/v1/N19-4010

  3. Chen X, Qiu X, Zhu C et al (2015) Long short-term memory neural networks for Chinese word segmentation. In: EMNLP, pp 1197–1206. https://doi.org/10.18653/v1/D15-1141

  4. Collobert R, Weston J, Bottou L et al (2011) Natural language processing (almost) from scratch. J Mach Learn Res 12(null):2493–2537

    MATH  Google Scholar 

  5. de Oliveira MV, de Almeida JCS (2013) Application of artificial intelligence techniques in modeling and control of a nuclear power plant pressurizer system. Prog Nuclear Energy 63:71–85

    Article  Google Scholar 

  6. Devlin J, Chang MW, Lee K et al (2018) BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT, pp 4171–4186. https://doi.org/10.18653/v1/N19-1423

  7. Gerard SN, Heching A, Keohane SM et al (2018) Knowledge reactor: a contextual computing work in progress for eldercare. Int J Electron Commun Eng 12(4):291–298

    Google Scholar 

  8. Guo Z, Zhang Y, Lu W (2019) Attention guided graph convolutional networks for relation extraction. In: ACL, pp 241–251. https://doi.org/10.18653/v1/P19-1024

  9. Huang Z, Xu W, Yu K (2015) Bidirectional LSTM-CRF models for sequence tagging. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000274. arXiv:1508.01991

  10. Ju M, Miwa M, Ananiadou S (2018) A neural layered model for nested named entity recognition. In: Proceedings of the 2018 conference of the North American chapter of the Association for computational linguistics: human language technologies, vol 1 (long papers). Association for Computational Linguistics, New Orleans, Louisiana, pp 1446–1459. https://doi.org/10.18653/v1/N18-1131. https://www.aclweb.org/anthology/N18-1131

  11. Katiyar A, Cardie C (2018) Nested named entity recognition revisited. In: Proceedings of the 2018 conference of the North American Chapter of the Association for computational linguistics: human language technologies, vol 1 (long papers). Association for Computational Linguistics, New Orleans, Louisiana, pp 861–871. https://doi.org/10.18653/v1/N18-1079. https://www.aclweb.org/anthology/N18-1079

  12. Lample G, Ballesteros M, Subramanian S et al (2016) Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360. https://doi.org/10.18653/v1/N16-1030

  13. Li J, Sun A, Han J et al (2020) A survey on deep learning for named entity recognition. IEEE Trans Knowl Data Eng. https://doi.org/10.1109/TKDE.2020.2981314

    Article  Google Scholar 

  14. Li X, Meng Y, Sun X et al (2019) Is word segmentation necessary for deep learning of Chinese representations? In: ACL, pp 3242–3252. https://doi.org/10.18653/v1/P19-1314

  15. Liu Z, Zhu C, Zhao T (2010) Chinese named entity recognition with a sequence labeling approach: based on characters, or based on words? In: Advanced intelligent computing theories and applications. With aspects of artificial intelligence, pp 634–640

  16. Ma X, Hovy E (2016) End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In: Proceedings of the 54th annual meeting of the association for computational linguistics, vol 1: long papers, pp 1064–1074. https://doi.org/10.18653/v1/P16-1101. https://www.aclweb.org/anthology/P16-1101

  17. Peters ME, Neumann M, Gardner M et al (2018) Deep contextualized word representations. In: NAACL-HLT, pp 2227–2237. https://doi.org/10.18653/v1/N18-1202

  18. Radford A, Narasimhan K, Salimans T et al (2018) Improving language understanding with unsupervised learning. Tech. rep, OpenAI

    Google Scholar 

  19. Shen Y, Ma X, Tan Z et al (2021) Locate and label: a two-stage identifier for nested named entity recognition. In: Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing, vol 1: long papers. Association for Computational Linguistics, Online, pp 2782–2794. https://doi.org/10.18653/v1/2021.acl-long.216. https://aclanthology.org/2021.acl-long.216

  20. Sohrab MG, Miwa M (2018) Deep exhaustive model for nested named entity recognition. In: Proceedings of the 2018 conference on empirical methods in natural language processing. Association for Computational Linguistics, Brussels, Belgium, pp 2843–2849. https://doi.org/10.18653/v1/D18-1309, https://www.aclweb.org/anthology/D18-1309

  21. Viterbi A (1967) Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Trans Inf Theory 13(2):260–269. https://doi.org/10.1109/TIT.1967.1054010

    Article  MATH  Google Scholar 

  22. Webber J (2012) A programmatic introduction to neo4j. In: SPLASH, pp 217–218

  23. Xue N (2003) Chinese word segmentation as character tagging. In: IJCLCLP, pp 29–48

  24. Yang Z, Salakhutdinov R, Cohen W (2016) Multi-task cross-lingual sequence tagging from scratch. arXiv preprint arXiv:1603.06270

  25. Yang Z, Dai Z, Yang Y et al (2019) Xlnet: generalized autoregressive pretraining for language understanding. In: NIPS

  26. Zhang Y, Yang J (2018) Chinese NER using lattice LSTM. In: ACL, pp 1554–1564. https://doi.org/10.18653/v1/P18-1144

  27. Zheng C, Cai Y, Xu J et al (2019) A boundary-aware neural model for nested named entity recognition. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, pp 357–366, https://doi.org/10.18653/v1/D19-1034. https://www.aclweb.org/anthology/D19-1034

Download references

Acknowledgements

This work was funded by the National Key Research and Development Program of China (2020YFB1707803). This paper has been extended from the original conference publication in the IEEE ICEBE 2021 proceedings.

Author information

Authors and Affiliations

  1. ZJU-UIUC Institute, Zhejiang University, Haining, 314400, Zhejiang, China

    Jian Zhang, Bo Qin, Yufei Zhang & Hongwei Wang

  2. Beijing Institute of Electronic System Engineering, Beijing Simulation Center, Beijing, 100000, China

    Junhua Zhou

Authors
  1. Jian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bo Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yufei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Junhua Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hongwei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongwei Wang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Qin, B., Zhang, Y. et al. A knowledge extraction framework for domain-specific application with simplified pre-trained language model and attention-based feature extractor. SOCA 16, 121–131 (2022). https://doi.org/10.1007/s11761-022-00337-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11761-022-00337-5

Keywords

Search

Navigation