Enabling Hierarchical Dirichlet Processes to Work Better for Short Texts at Large Scale

Abstract

Analyzing texts from social media often encounters many challenges, including shortness, dynamic, and huge size. Short texts do not provide enough information so that statistical models often fail to work. In this paper, we present a very simple approach (namely, bag-of-biterms) that helps statistical models such as Hierarchical Dirichlet Processes (HDP) to work well with short texts. By using both terms (words) and biterms to represent documents, bag-of-biterms (BoB) provides significant benefits: (1) it naturally lengthens representation and thus helps us reduce bad effects of shortness; (2) it enables the posterior inference in a large class of probabilistic models including HDP to be less intractable; (3) no modification of existing models/methods is necessary, and thus BoB can be easily employed in a wide class of statistical models. To evaluate those benefits of BoB, we take Online HDP into account in that it can deal with dynamic and massive text collections, and we do experiments on three large corpora of short texts which are crawled from Twitter, Yahoo Q&A, and New York Times. Extensive experiments show that BoB can help HDP work significantly better in both predictiveness and quality.

Appendix: Conversion of topic-over-biterms (distribution over biterms) to topic-over-words (distribution over words)

In BoB, after we finish training the model, we obtain topics, each is the multinomial distribution over biterms and we want to convert to the topics with distribution over words. Assume that $\varvec{\phi }_k$ is the distribution over biterms of topic k. According to probability:

$p(w_i \mid z=k) = \sum _{j=1}^{V}p(w_i,w_j \mid z=k) = \sum _{j=1}^{V}p(b_{ij} \mid z=k) = \sum _{j=1}^{V}\phi _{kb_{ij}}$,

As discussed in Sect. 4.2, in implementing BoB, we can merge $b_{ij}$ and $b_{ji}$ into $b_{ij}$ with $i<j$. Because of identical occurence in every document, after finishing training process, the value of $p(b_{ij}\mid z=k)$ will be expectedly the same as $p(b_{ji}\mid z=k)$. Therefore, in grouping these biterms into one, the conversion version of this implementation is: $p(w_i\mid z=k) = \sum _{j=1}^{V}p(b_{ij} \mid z=k) = \phi _{kb_{ii}} + \frac{1}{2}\sum _{\text {b: biterms contain } w_i}\phi _{kb}.$