Abstract
Term frequency – Inverse Document Frequency (TFIDF) is a vital first step in text analytics for information retrieval and machine learning applications. It is a memory-intensive and complex task due to the need to create and process a large sparse matrix of term frequencies, with the documents as rows and the term as columns and populate it with the term frequency of each word in each document.
The standard method of storing the sparse array is the “Compressed Sparse Row” (CSR), which stores the sparse array as three one-dimensional arrays for the row id, column id, and term frequencies. We propose an alternate representation to the CSR: a list of lists (LIL) where each document is represented as its own list of tuples and each tuple storing the column id and the term frequency value. We implemented both techniques to compare their memory efficiency and speed. The new LIL representation increase the memory capacity by 52% and is only 12% slower in processing time. This enables researchers with limited processing power to be able to work on bigger text analysis datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Buluç, A., Fineman, J., Frigo, M., Gilbert, J., Leiserson, C.: Parallel sparse matrix-vector and matrix-transpose-vector multiplication using compressed sparse blocks. In Proceedings of the 21st Annual Symposium on Parallelism in Algorithms and Architectures (SPAA 2009), pp. 233–244. ACM, New York (2009). https://doi.org/10.1145/1583991.1584053
Kim, S.-W., Gil, J.-M.: Research paper classification systems based on TF-IDF and LDA schemes. Hum. Cent. Comput. Inf. Sci. 9(1), 1–21 (2019). https://doi.org/10.1186/s13673-019-0192-7
Koushik, G., Rajeswari, K., Muthusamy, S.: Automated hate speech detection on Twitter. In: 2019 5th International Conference On Computing, Communication, Control And Automation (ICCUBEA), Pune, India, pp. 1–4 (2019) https://doi.org/10.1109/ICCUBEA47591.2019.9128428
Michailidis, M.: Sentiment140 dataset with 1.6 million tweets. http://kaggle.com/kazanova/sentiment140. Accessed 19 Jan 2021
Romadon, A., Lhaksmana, K., Kurniawan, I., Richasdy, D.: Analyzing TF-IDF and word embedding for implementing automation in job interview grading. In: 2020 8th International Conference on Information and Communication Technology (ICoICT), Yogyakarta, Indonesia, pp. 1–4 (2020). https://doi.org/10.1109/ICoICT49345.2020.9166364
Stoer, J., Bulirsch, R.: Introduction to Numerical Analysis. Springer, New York (2002). https://doi.org/10.1007/978-0-387-21738-3ISBN 978-0-387-95452-3
Varol, C., Abdulhadi, H.: Comparision of string matching algorithms on spam email detection. In: 2018 International Congress on Big Data, Deep Learning and Fighting Cyber Terrorism (IBIGDELFT), Ankara, Turkey, pp. 6–11 (2018). https://doi.org/10.1109/IBIGDELFT.2018.8625317
Yamout, F., Lakkis, R.: Improved TFIDF weighting techniques in document Retrieval. In: 2018 Thirteenth International Conference on Digital Information Management (ICDIM), Berlin, Germany, pp. 69–73 (2018). https://doi.org/10.1109/ICDIM.2018.8847156
Yuan, H., Wang, Y., Feng, X., Sun, S.: Sentiment analysis based on weighted Word2vec and Att-LSTM. In: Proceedings of the 2018 2nd International Conference on Computer Science and Artificial Intelligence (CSAI 2018), pp. 420–424. ACM, New York (2018) https://doi.org/10.1145/3297156.3297228
Zhao, G., Liu, Y., Zhang, W., Wang, Y.: TFIDF based feature words extraction and topic modeling for short text. In: Proceedings 2018. TFIDF based Feature Words Extraction and Topic Modeling for Short Text. Proceedings of the 2018 2nd International Conference on Management Engineering, Software Engineering and Service Sciences (ICMSS 2018), pp. 188–191. ACM, New York (2018). https://doi.org/10.1145/3180374.3181354
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Senbel, S. (2021). Fast and Memory-Efficient TFIDF Calculation for Text Analysis of Large Datasets. In: Fujita, H., Selamat, A., Lin, J.CW., Ali, M. (eds) Advances and Trends in Artificial Intelligence. Artificial Intelligence Practices. IEA/AIE 2021. Lecture Notes in Computer Science(), vol 12798. Springer, Cham. https://doi.org/10.1007/978-3-030-79457-6_47
Download citation
DOI: https://doi.org/10.1007/978-3-030-79457-6_47
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-79456-9
Online ISBN: 978-3-030-79457-6
eBook Packages: Computer ScienceComputer Science (R0)