Skip to main content
SpringerLink
Log in

Nonnegative Matrix Factorization for Interactive Topic Modeling and Document Clustering

  • Chapter
  • First Online:
Partitional Clustering Algorithms

Abstract

Nonnegative matrix factorization (NMF) approximates a nonnegative matrix by the product of two low-rank nonnegative matrices. Since it gives semantically meaningful result that is easily interpretable in clustering applications, NMF has been widely used as a clustering method especially for document data, and as a topic modeling method.We describe several fundamental facts of NMF and introduce its optimization framework called block coordinate descent. In the context of clustering, our framework provides a flexible way to extend NMF such as the sparse NMF and the weakly-supervised NMF. The former provides succinct representations for better interpretations while the latter flexibly incorporate extra information and user feedback in NMF, which effectively works as the basis for the visual analytic topic modeling system that we present.Using real-world text data sets, we present quantitative experimental results showing the superiority of our framework from the following aspects: fast convergence, high clustering accuracy, sparse representation, consistent output, and user interactivity. In addition, we present a visual analytic system called UTOPIAN (User-driven Topic modeling based on Interactive NMF) and show several usage scenarios.Overall, our book chapter cover the broad spectrum of NMF in the context of clustering and topic modeling, from fundamental algorithmic behaviors to practical visual analytics systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The nonnegative rank of a matrix \(X \in \mathbb{R}_{+}^{m\times n}\) is the smallest number \(\hat{k}\) such that X = WH where \(W \in \mathbb{R}_{+}^{m\times \hat{k}}\) and \(H \in \mathbb{R}_{+}^{\hat{k}\times n}\).

  2. 2.

    http://www.cc.gatech.edu/~hpark/nmfsoftware.php

  3. 3.

    http://www.csie.ntu.edu.tw/~cjlin/nmf/index.html

  4. 4.

    The term “weakly-supervised” can be considered similar to semi-supervised clustering settings, rather than supervised learning settings such as classification and regression problems.

  5. 5.

    http://www.daviddlewis.com/resources/testcollections/reuters21578/

  6. 6.

    http://qwone.com/jason/20Newsgroups/

  7. 7.

    http://jmlr.csail.mit.edu/papers/volume5/lewis04a/lyrl2004rcv1v2README.htm

  8. 8.

    http://robotics.stanford.edu/gal/data.html

  9. 9.

    http://www.cc.gatech.edu/grads/d/dkuang3/software/kmeans3.html

  10. 10.

    https://github.com/kimjingu/nonnegfac-matlab

  11. 11.

    Results given by the sparseness measure based on L 1 and L 2 norms in [21] are similar in terms of comparison between the three NMF versions.

  12. 12.

    For LDA, we used Mallet [41], a widely-accepted software library based on a Gibbs sampling method.

  13. 13.

    http://tinyurl.com/2013utopian

References

  1. Arora S, Ge R, Kannan R, Moitra A (2012) Computing a nonnegative matrix factorization – provably. In: Proceedings of the 44th symposium on theory of computing (STOC), pp 145–162

    Google Scholar 

  2. Arora S, Ge R, Halpern Y, Mimno D, Moitra A, Sontag D, Wu Y, Zhu M (2013). A practical algorithm for topic modeling with provable guarantees. J Mach Learn Res 28(2):280–288

    Google Scholar 

  3. Berman A, Plemmons RJ (1994) Nonnegative matrices in the mathematical sciences. SIAM, Philadelphia

    Book  MATH  Google Scholar 

  4. Bertsekas DP (1999) Nonlinear programming, 2nd edn. Athena Scientific, Belmont

    MATH  Google Scholar 

  5. Blei DM, Ng AY, Jordan MI (2003) Latent Dirichlet allocation. J Mach Learn Res 3:993–1022

    MATH  Google Scholar 

  6. Brunet J-P, Tamayo P, Golub TR, Mesirov JP (2004) Metagenes and molecular pattern discovery using matrix factorization. Proc Natl Acad Sci USA 101(12):4164–4169

    Article  Google Scholar 

  7. Cai D, He X, Han J, Huang TS (2011) Graph regularized nonnegative matrix factorization for data representation. IEEE Trans Pattern Anal Mach Intell 33(8):1548–1560

    Article  Google Scholar 

  8. Choo J, Park H (2013) Customizing computational methods for visual analytics with big data. IEEE Comput Graph Appl 33(4):22–28

    Article  Google Scholar 

  9. Choo J, Lee C, Reddy CK, Park H (2013) UTOPIAN: user-driven topic modeling based on interactive nonnegative matrix factorization. IEEE Trans Vis Comput Graph 19(12):1992–2001

    Article  Google Scholar 

  10. Cichocki A, Zdunek R, Phan AH, Amari S (2009) Nonnegative matrix and tensor factorizations: applications to exploratory multi-way data analysis and blind source separation. Wiley, London

    Book  Google Scholar 

  11. Devarajan K (2008) Nonnegative matrix factorization: an analytical and interpretive tool in computational biology. PLoS Comput Biol 4(7):e1000029

    Article  Google Scholar 

  12. Dhillon IS, Sra S (2005) Generalized nonnegative matrix approximations with Bregman divergences. In: Advances in neural information processing systems (NIPS), vol 18, pp 283–290

    Google Scholar 

  13. Ding C, He X, Simon HD (2005) On the equivalence of nonnegative matrix factorization and spectral clustering. In: Proceedings of SIAM international conference on data mining (SDM), pp 606–610

    Google Scholar 

  14. Ding C, Li T, Jordan M (2008) Nonnegative matrix factorization for combinatorial optimization: spectral clustering, graph matching, and clique finding. In: Proceedings of the 8th IEEE international conference on data mining (ICDM), pp 183–192

    Google Scholar 

  15. Ding C, T Li, Jordan MI (2010) Convex and semi-nonnegative matrix factorization. IEEE Trans Pattern Anal Mach Intell 32(1):45–55

    Article  Google Scholar 

  16. Duda RO, Hart PE, Stork DG (2000) Pattern classification. Wiley-Interscience, London

    Google Scholar 

  17. Globerson A, Chechik G, Pereira F, Tishby N (2007) Euclidean embedding of co-occurrence data. J Mach Learn Res 8:2265–2295

    MATH  MathSciNet  Google Scholar 

  18. Gonzales EF, Zhang Y (2005) Accelerating the Lee-Seung algorithm for non-negative matrix factorization. Technical Report TR05-02, Rice University

    Google Scholar 

  19. Grippo L, Sciandrone M (2000) On the convergence of the block nonlinear Gauss-Seidel method under convex constraints. Oper Res Lett 26:127–136

    Article  MATH  MathSciNet  Google Scholar 

  20. Hofmann T (1999) Probabilistic latent semantic indexing. In: Proceedings of the 22nd annual international ACM SIGIR conference on research and development in information retrieval (SIGIR)

    Google Scholar 

  21. Hoyer PO (2004) Non-negative matrix factorization with sparseness constraints. J Mach Learn Res 5:1457–1469

    MATH  MathSciNet  Google Scholar 

  22. Kim H, Park H (2007) Sparse non-negative matrix factorizations via alternating non-negativity-constrained least squares for microarray data analysis. Bioinformatics 23(12):1495–1502

    Article  Google Scholar 

  23. Kim H, Park H (2008) Nonnegative matrix factorization based on alternating non-negativity-constrained least squares and the active set method. SIAM J Matrix Anal Appl 30(2):713–730

    Article  MATH  MathSciNet  Google Scholar 

  24. Kim D, Sra S, Dhillon I (2007) Fast Newton-type methods for the least squares nonnegative matrix approximation problem. In: Proceedings of SIAM international conference on data mining (SDM), pp 343–354

    Google Scholar 

  25. Kim J, He Y, Park H (2014) Algorithms for nonnegative matrix and tensor factorizations: a unified view based on block coordinate descent framework. J Global Optim 58(2):285–319

    Article  MATH  MathSciNet  Google Scholar 

  26. Kim J, Park H (2008) Sparse nonnegative matrix factorization for clustering. Technical Report GT-CSE-08-01, Georgia Institute of Technology

    Google Scholar 

  27. Kim J, Park H (2008) Toward faster nonnegative matrix factorization: a new algorithm and comparisons. In: Proceedings of the 8th IEEE international conference on data mining (ICDM), pp 353–362

    Google Scholar 

  28. Kim J, Park H (2011) Fast nonnegative matrix factorization: An active-set-like method and comparisons. SIAM J Sci Comput 33(6):3261–3281

    Article  MATH  MathSciNet  Google Scholar 

  29. Kuang D, Park H (2013) Fast rank-2 nonnegative matrix factorization for hierarchical document clustering. In: Proceedings of the 19th ACM international conference on knowledge discovery and data mining (KDD), pp 739–747

    Google Scholar 

  30. Kuang D, Ding C, Park H (2012) Symmetric nonnegative matrix factorization for graph clustering. In: Proceedings of SIAM international conference on data mining (SDM), pp 106–117

    Google Scholar 

  31. Kuhn HW (1955) The Hungarian method for the assignment problem. Nav Res Logistics Q 2:83–97

    Article  Google Scholar 

  32. Lawson CL, Hanson RJ (1974) Solving least squares problems. Prentice Hall, Englewood Cliffs

    MATH  Google Scholar 

  33. Lee DD, Seung HS (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401:788–791

    Article  Google Scholar 

  34. Lee DD, Seung HS (2001) Algorithms for non-negative matrix factorization. In: Advances in neural information processing systems (NIPS), vol 14, pp 556–562

    Google Scholar 

  35. Lewis DD, Yang Y, Rose TG, Li F (2004) Rcv1: a new benchmark collection for text categorization research. J Mach Learn Res 5:361–397

    Google Scholar 

  36. Li S, Hou XW, Zhang HJ, Cheng QS (2001) Learning spatially localized, parts-based representation. In: Proceedings of the 2001 IEEE conference on computer vision and pattern recognition (CVPR), pp 207–212

    Google Scholar 

  37. Li T, Ding C, Jordan MI (2007) Solving consensus and semi-supervised clustering problems using nonnegative matrix factorization. In: Proceedings of the 7th IEEE international conference on data mining (ICDM), pp 577–582

    Google Scholar 

  38. Li L, Lebanon G, Park H (2012) Fast Bregman divergence NMF using Taylor expansion and coordinate descent. In: Proceedings of the 18th ACM international conference on knowledge discovery and data mining (KDD), pp 307–315

    Google Scholar 

  39. Lin C-J (2007) Projected gradient methods for nonnegative matrix factorization. Neural Comput 19(10):2756–2779

    Article  MATH  MathSciNet  Google Scholar 

  40. Manning CD, Raghavan P, Schütze H (2008) Introduction to information retrieval. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  41. McCallum AK (2002) MALLET: a machine learning for language toolkit. http://mallet.cs.umass.edu

  42. Monti S, Tamayo P, Mesirov J, Golub T (2003) Consensus clustering: a resampling-based method for class discovery and visualization of gene expression microarray data. Mach Learn 52(1–2):91–118

    Article  MATH  Google Scholar 

  43. Paatero P, Tapper U (1994) Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values. Environmetrics 5:111–126

    Article  Google Scholar 

  44. Pauca VP, Shahnaz F, Berry MW, Plemmons RJ (2004) Text mining using non-negative matrix factorizations. In: Proceedings of SIAM international conference on data mining (SDM), pp 452–456

    Google Scholar 

  45. Pauca VP, Piper J, Plemmons RJ (2006) Nonnegative matrix factorization for spectral data analysis. Linear Algebra Appl 416(1):29–47

    Article  MATH  MathSciNet  Google Scholar 

  46. Shahnaz F, Berry MW, Pauca VP, Plemmons RJ (2006) Document clustering using nonnegative matrix factorization. Inf Process Manag 42:373–386

    Article  MATH  Google Scholar 

  47. Tibshirani R (1994) Regression shrinkage and selection via the lasso. J R Stat Soc Ser B Methodol 58:267–288

    MathSciNet  Google Scholar 

  48. van der Maaten L, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9:2579–2605

    MATH  Google Scholar 

  49. Vavasis SA (2009) On the complexity of nonnegative matrix factorization. SIAM J Optim 20(3):1364–1377

    Article  MATH  MathSciNet  Google Scholar 

  50. Wild S, Curry J, Dougherty A (2004) Improving non-negative matrix factorizations through structured initialization. Pattern Recognit 37:2217–2232

    Article  Google Scholar 

  51. Xie B, Song L, Park H (2013) Topic modeling via nonnegative matrix factorization on probability simplex. In: NIPS workshop on topic models: computation, application, and evaluation

    Google Scholar 

  52. Xu W, Liu X, Gong Y (2003) Document clustering based on non-negative matrix factorization. In: Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR), pp 267–273

    Google Scholar 

Download references

Acknowledgements

The work of the authors was supported in part by the National Science Foundation (NSF) grants CCF-0808863 and the Defense Advanced Research Projects Agency (DARPA) XDATA program grant FA8750-12-2-0309. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF or the DARPA.

Author information

Authors and Affiliations

  1. Georgia Institute of Technology, Atlanta, GA, USA

    Da Kuang, Jaegul Choo & Haesun Park

Authors
  1. Da Kuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jaegul Choo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haesun Park
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haesun Park .

Editor information

Editors and Affiliations

  1. Computer Science dept., Louisiana State University Shreveport, Shreveport, Louisiana, USA

    M. Emre Celebi

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Kuang, D., Choo, J., Park, H. (2015). Nonnegative Matrix Factorization for Interactive Topic Modeling and Document Clustering. In: Celebi, M. (eds) Partitional Clustering Algorithms. Springer, Cham. https://doi.org/10.1007/978-3-319-09259-1_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-09259-1_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-09258-4

  • Online ISBN: 978-3-319-09259-1

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation