Abstract
Business documents are used every day by all kinds and sizes of companies and administrations, even if most of these entities have several information systems where the documents are digitilized in different formats (json, xml, database tables, ...), there still an important number of business documents that require manual processing which costs a lot and is very time consuming.
Extracting key-value information from business documents is a challenging problem due to the variety of document types and templates, in this work we will deal with the problem as a graph node classification problem using a multi “graph transformer" layers, we propose a graph construction method that focuses on the most relevant neighbours of every node while reducing the size of the graph and we use a document transformer embedding combined with some spatial and textual feature to give a better representation to each node.
Our contribution in this work was to conceive a graph neural network (GNN) achieving the highest results comparing to the rest of GNN models dealing with the same problem to our knowledge, the model is small (53,6K parameters) comparing to the recent models using transformers architectures (hundreds of millions of parameters) which is very suitable for applications when storage constraints are present, it also has a limited impact on the environment and represent an alternative to build greener AI systems.
We experiment our model on the SROIE ICDAR receipts dataset where we got an important F1 score compared to other graph neural network (GNN) based models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
References
O’Gorman, L.: The document spectrum for page layout analysis. IEEE Trans. Pattern Anal. Mach. Intell. 15(11), 1162–1173 (1993). https://doi.org/10.1109/34.244677
LeCun, Y., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Haykin, S.: Neural Networks: A Comprehensive Foundation. Prentice Hall PTR, Upper Saddle River (1994)
Vaswani, A., et al. Attention is all you need (2017). arXiv:1706.03762 [cs]
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding (2019). arXiv:1810.04805 [cs]
Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach (2019). arXiv:1907.11692 [Cs]
Ha, J., Phillips, I.T., Haralick, R.M.: Document page decomposition using bounding boxes of connected components of black pixels. In: Document Recognition II, vol. 2422, pp. 140–151. International Society for Optics and Photonics (1995). https://doi.org/10.1117/12.205816
Socher, R., Bengio, Y., Manning , C.D.: Deep learning for NLP (without magic). In: Tutorial Abstracts of ACL 2012, ACL 2012, vol. 5. Association for Computational Linguistics (2012)
Palm, R.B., Laws, F., Winther, O.: Attend, copy, parse end-to-end information extraction from documents. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 329–336 (2019). https://doi.org/10.1109/ICDAR.2019.00060
Majumder, B.P., Potti, N., Tata, S., Wendt, J.B., Zhao, Q., Najork, M.: Representation learning for information extraction from form-like documents. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6495–6504. Association for Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.acl-main.580. https://www.aclweb.org/anthology/2020.acl-main.580)
Patel, S., Bhatt, D.: Abstractive information extraction from scanned invoices (AIESI) using end-to-end sequential approach (2020). arXiv:2009.05728
Katti, A.R., Reisswig, C., Guder, C., Brarda, S., Bickel, S., Höhne, J., Faddoul, J.B.: Chargrid: towards understanding 2D documents. In: Proceedings of EMNLP, pp. 4459–4469 (2018)
Denk, T.I., Reisswig, C.: BERTgrid: contextualized embedding for 2D document representation and understanding. In: Workshop on Document Intelligence at NeurIPS 2019 (2019)
Davis, B., Morse, B., Cohen, S., Price, B., Tensmeyer, C.: Deep visual template-free form parsing. In: ICDAR, pp. 134–141 (2019)
Liu, X., Gao, F., Zhang, Q., Zhao, H.: Graph convolution for multimodal information extraction from visually rich documents. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 2 (Industry Papers), pp. 32–39. Association for Computational Linguistics, Minneapolis . https://doi.org/10.18653/v1/N19-2005
Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., Zhou, M.: Layoutlm: pre-training of text and layout for document image understanding. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2020)
Xu, Y., et al.: LayoutLMv2: multi-modal pre-training for visually-rich document understanding (2011). arXiv:2012.14740 [cs]
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks (2017). arXiv:1609.02907 [cs, stat]
Hamilton, W.L., Ying, R., Leskovec, J.: Inductive representation learning on large graphs (2018). arXiv:1706.02216 [cs, stat]
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: graph attention networks (2018). arXiv:1710.10903 [cs, stat]
Shi, Y., Huang, Z., Feng, S., Zhong, H., Wang, W., Sun, Y.: Masked label prediction: unified message passing model for semi-supervised classification (2021). arXiv:2009.03509 [cs, stat]
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS 2015), vol. 1, pp. 91–99. MIT Press, Cambridge (2015)
Riba, P., Dutta, A., Goldmann, L., Fornés, A., Ramos, O., Lladós, J.: Table detection in invoice documents by graph neural networks. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 122–127 (2019). https://doi.org/10.1109/ICDAR.2019.00028
Lohani, D., Belaïd, A., Belaïd, Y.: An invoice reading system using a graph convolutional network. In: Carneiro, G., You, S. (eds.) ACCV 2018. LNCS, vol. 11367, pp. 144–158. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-21074-8_12
Belhadj, D., Belaïd, Y., Belaïd, A.: Consideration of the word’s neighborhood in GATs for information extraction in semi-structured documents. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12822, pp. 854–869. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86331-9_55
Huang, Z., et al.: ICDAR2019 competition on scanned receipt OCR and Information extraction. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1516–1520 (2019). https://doi.org/10.1109/ICDAR.2019.00244
Defferrard, M., Bresson, X., Vandergheynst, P.: Convolutional neural networks on graphs with fast localized spectral filtering. In: Advances in Neural Information Processing Systems, pp. 3844–3852 (2016)
Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio, A., Bengio, Y.: Graph attention networks. In: Proceedings of ICLR, pp. 1–12 (2017)
Heinzerling, B., Strube, M.: BPEmb: tokenization-free pre-trained subword embeddings in 275 languages. In: Proceedings of the 11th International Conference on Language Resources and Evaluation (2018)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations (ICLR) (2017)
Li, Y., et al.: StrucTexT: structured text understanding with multi-modal transformers (2021). arXiv:2108.02923 [cs]
Li, M., et al.: DocBank: a benchmark dataset for document layout analysis (2020). arXiv:2006.01038 [cs]
Harley, A.W., Ufkes, A., Derpanis, K.G.: Evaluation of deep convolutional nets for document image classification and retrieval. In: ICDAR (2015)
Yu, W., Lu, N., Qi, X., Gong, P., Xiao, R.: PICK: processing key information extraction from documents using improved graph learning-convolutional networks (2004). arXiv:2004.07464 [cs]
Tchoubith, N., Calusha, P.: ICDAR 2019 robust reading challenge on scanned receipts OCR and information extraction (2021). https://github.com/BlackStar1313/ICDAR-2019-RRC-SROIE
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hamri, M., Devanne, M., Weber, J., Hassenforder, M. (2023). Enhancing GNN Feature Modeling for Document Information Extraction Using Transformers. In: Kerautret, B., Colom, M., Krähenbühl, A., Lopresti, D., Monasse, P., Perret, B. (eds) Reproducible Research in Pattern Recognition. RRPR 2022. Lecture Notes in Computer Science, vol 14068. Springer, Cham. https://doi.org/10.1007/978-3-031-40773-4_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-40773-4_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-40772-7
Online ISBN: 978-3-031-40773-4
eBook Packages: Computer ScienceComputer Science (R0)
