Skip to main content
SpringerLink
Log in

A Multi-teacher Knowledge Distillation Framework for Distantly Supervised Relation Extraction with Flexible Temperature

  • Conference paper
  • First Online:
Web and Big Data (APWeb-WAIM 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14332))

  • 115 Accesses

Abstract

Distantly supervised relation extraction (DSRE) generates large-scale annotated data by aligning unstructured text with knowledge bases. However, automatic construction methods cause a substantial number of incorrect annotations, thereby introducing noise into the training process. Most sentence-level relation extraction methods rely on filters to remove noise instances, meanwhile, they ignore some useful information in negative instances. To effectively reduce noise interference, we propose a Multi-teacher Knowledge Distillation framework for Relation Extraction (MKDRE) to extract semantic relations from noisy data based on both global information and local information. MKDRE addresses two main problems: the deviation in knowledge propagation of a single teacher and the limitation of traditional distillation temperature on information utilization. Specifically, we utilize flexible temperature regulation (FTR) to adjust the temperature assigned to each training instance, so as to dynamically capture local relations between instances. Furthermore, we introduce information entropy of hidden layers to gain stable temperature calculations. Finally, we propose multi-view knowledge distillation (MVKD) to express global relations among teachers from various perspectives to gain more reliable knowledge. The experimental results on NYT19-1.0 and NYT19-2.0 datasets show that our proposed MKDRE significantly outperforms previous methods in sentence-level relation extraction.

Supported by the National Natural Science Foundation of China under the Grant No. 62172451, and supported by Open Research Projects of Zhejiang Lab under the Grant No. 2022KG0AB01.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Wu, S., He, Y.: Enriching pre-trained language model with entity information for relation classification. In: CIKM, pp. 2361–2364 (2019)

    Google Scholar 

  2. Wei, Z., Su, J., Wang, Y., Tian, Y., Chang, Y.: A novel cascade binary tagging framework for relational triple extraction. In: ACL, pp. 1476–1488 (2020)

    Google Scholar 

  3. Mintz, M., Bills, S., Snow, R., Jurafsky, D.: Distant supervision for relation extraction without labeled data. In: ACL, pp. 1003–1011 (2009)

    Google Scholar 

  4. Zeng, D., Liu, K., Chen, Y., Zhao, J.: Distant supervision for relation extraction via piecewise convolutional neural networks. In: EMNLP, pp. 1753–1762 (2015)

    Google Scholar 

  5. Lin, Y., Shen, S., Liu, Z., Luan, H., Sun, M.: Neural relation extraction with selective attention over instances. In: ACL, pp. 2124–2133 (2016)

    Google Scholar 

  6. Vashishth, S., Joshi, R., Prayaga, S.S., Bhattacharyya, C., Talukdar, P.: RESIDE: improving distantly-supervised neural relation extraction using side information. In: EMNLP, pp. 1257–1266 (2018)

    Google Scholar 

  7. Lin, X., Liu, T., Jia, W., Gong, Z.: Distantly supervised relation extraction using multi-layer revision network and confidence based multi-instance learning. In: EMNLP, pp. 165–174 (2021)

    Google Scholar 

  8. Feng, J., Huang, M., Zhao, L., Yang, Y., Zhu, X.: Reinforcement learning for relation classification from noisy data. In: AAAI (2018)

    Google Scholar 

  9. Zeng, X., He, S., Liu, K.: Large scaled relation extraction with reinforcement learning. In: AAAI (2018)

    Google Scholar 

  10. He, Z., Chen, W., Wang, Y., Zhang, W., Wang, G., Zhang, M.: Improving neural relation extraction with positive and unlabeled learning. In: AAAI, pp. 7927–7934 (2020)

    Google Scholar 

  11. Ma, R., Gui, T., Li, L., Zhang, Q., Huang, X., Zhou, Y.:SENT: sentence-level distant relation extraction via negative training. In: ACL, pp. 6201–6213 (2021)

    Google Scholar 

  12. Li, R., Yang, C., Li, T., Su, S.: MidTD: a simple and effective distillation framework for distantly supervised relation extraction. ACM Trans. Inf. Syst. (TOIS) 40(4), 1–32 (2022)

    Google Scholar 

  13. Zhang, Z., et al.: Distilling knowledge from well-informed soft labels for neural relation extraction. In: AAAI, pp. 9620–9627 (2020)

    Google Scholar 

  14. Riedel, S., Yao, L., McCallum, A.: Modeling relations and their mentions without labeled text. In: Balcázar, J.L., Bonchi, F., Gionis, A., Sebag, M. (eds.) ECML PKDD 2010. LNCS (LNAI), vol. 6323, pp. 148–163. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15939-8_10

    Chapter  Google Scholar 

  15. Wang, X., Yang, J., Wang, Q., Su, C.: Threat intelligence relationship extraction based on distant supervision and reinforcement learning. In: SEKE, pp. 572–576 (2020)

    Google Scholar 

  16. Qin, P., Xu, W., Wang, W. Y.: DSGAN: generative adversarial training for distant supervision relation extraction. In: ACL, pp. 496–505 (2018)

    Google Scholar 

  17. Li, Y., et al.: Self-attention enhanced selective gate with entity-aware embedding for distantly supervised relation extraction. In: AAAI, pp. 8269–8276 (2020)

    Google Scholar 

  18. Du, J., Han, J., Way, A., Wan, D.: Multi-level structured self-attentions for distantly supervised relation extraction. In: EMNLP, pp. 2216–2225 (2018)

    Google Scholar 

  19. Wang, J., Liu, Q.: Distant supervised relation extraction with position feature attention and selective bag attention. Neurocomputing 461, 552–561 (2021)

    Article  Google Scholar 

  20. Chen, T., Shi, H., Tang, S., Chen, Z., Wu, F., Zhuang, Y.: CIL: contrastive instance learning framework for distantly supervised relation extraction. In: ACL, pp. 6191–6200 (2021)

    Google Scholar 

  21. Li, D., Zhang, T., Hu, N., Wang, C., He, X.: HiCLRE: a hierarchical contrastive learning framework for distantly supervised relation extraction. In: ACL, pp. 2567–2578 (2022)

    Google Scholar 

  22. Zhang, Y., Fei, H., Li, P.: ReadsRE: retrieval-augmented distantly supervised relation extraction. In: ACM SIGIR, pp. 2257–2262 (2021)

    Google Scholar 

  23. Peng, T., Han, R., Cui, H., Yue, L., Han, J., Liu, L.: Distantly supervised relation extraction using global hierarchy embeddings and local probability constraints. Knowl.-Based Syst. 235, 107637 (2022)

    Article  Google Scholar 

  24. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NeurIPS (2015)

    Google Scholar 

  25. Li, Y., Yang, J., Song, Y., Cao, L., Luo, J., Li, L. J.: Learning from noisy labels with distillation. In: ICCV, pp. 1910–1918 (2017)

    Google Scholar 

  26. Sarfraz, F., Arani, E., Zonooz, B.: Knowledge distillation beyond model compression. In: ICPR, pp. 6136–6143 (2021)

    Google Scholar 

  27. Zhou, H., et al.: Rethinking soft labels for knowledge distillation: a bias-variance tradeoff perspective. In: ICLR (2021)

    Google Scholar 

  28. Yim, J., Joo, D., Bae, J., Kim, J.: A gift from knowledge distillation: fast optimization, network minimization and transfer learning. In: CVPR, pp. 7130–7138 (2017)

    Google Scholar 

  29. Lei, K., et al.: Cooperative denoising for distantly supervised relation extraction. In: COLING, pp. 426–436 (2018)

    Google Scholar 

  30. Zhang, S., Zheng, D., Hu, X., Yang, M.: Bidirectional long short-term memory networks for relation classification. In: PACLIC, pp. 73–78 (2015)

    Google Scholar 

  31. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT, pp. 4171–4186 (2019)

    Google Scholar 

  32. Liu, Y., et al.: RoBERTa: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

  33. Ren, X., et al.: Cotype: joint extraction of typed entities and relations with knowledge bases. In: WWW, pp. 1015–1024 (2017)

    Google Scholar 

  34. Jia, W., Dai, D., Xiao, X., Wu, H.: ARNOR: attention regularization based noise reduction for distant supervision relation classification. In: ACL, pp. 1399–1408 (2019)

    Google Scholar 

  35. Zhou, P., et al.: Attention-based bidirectional long short-term memory networks for relation classification. In: ACL, pp. 207–212 (2016)

    Google Scholar 

  36. Qin, P., Xu, W., Wang, W. Y.: Robust distant supervision relation extraction via deep reinforcement learning. In: ACL, pp. 2137–2147 (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Central South University, Changsha, China

    Hongxiao Fei, Yangying Tan, Jun Long & Liu Yang

  2. School of Computer Science and Engineering, Hunan University of Science and Technology, Xiangtan, China

    Wenti Huang

  3. Big Data Institute, Central South University, Changsha, China

    Jun Long & Jincai Huang

Authors
  1. Hongxiao Fei
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yangying Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenti Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jun Long
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jincai Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Liu Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liu Yang .

Editor information

Editors and Affiliations

  1. Peng Cheng Laboratory, Shenzhen, China

    Xiangyu Song

  2. China University of Geosciences, Wuhan, China

    Ruyi Feng

  3. China University of Geosciences, Wuhan, China

    Yunliang Chen

  4. Deakin University, Burwood, VIC, Australia

    Jianxin Li

  5. University of Exeter, Exeter, UK

    Geyong Min

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fei, H., Tan, Y., Huang, W., Long, J., Huang, J., Yang, L. (2024). A Multi-teacher Knowledge Distillation Framework for Distantly Supervised Relation Extraction with Flexible Temperature. In: Song, X., Feng, R., Chen, Y., Li, J., Min, G. (eds) Web and Big Data. APWeb-WAIM 2023. Lecture Notes in Computer Science, vol 14332. Springer, Singapore. https://doi.org/10.1007/978-981-97-2390-4_8

Download citation

  • DOI: https://doi.org/10.1007/978-981-97-2390-4_8

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-97-2389-8

  • Online ISBN: 978-981-97-2390-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation