Non-commutative Logic for Compositional Distributional Semantics

Abstract

Distributional models of natural language use vectors to provide a contextual foundation for meaning representation. These models rely on large quantities of real data, such as corpora of documents, and have found applications in natural language tasks, such as word similarity, disambiguation, indexing, and search. Compositional distributional models extend the distributional ones from words to phrases and sentences. Logical operators are usually treated as noise by these models and no systematic treatment is provided so far. In this paper, we show how skew lattices and their encoding in upper triangular matrices provide a logical foundation for compositional distributional models. In this setting, one can model commutative as well as non-commutative logical operations of conjunction and disjunction. We provide theoretical foundations, a case study, and experimental results for an entailment task on real data.

K. Cvetko-Vah acknowledges the financial support from the Slovenian Research Agency (research core funding No. P1-0222). M. Sadrzadeh, D. Kartsaklis and B. Blundell acknowledge financial support from AFOSR International Scientific Collaboration Grant FA9550-14-1-0079.

A Appendix

1.1 A.1 Normalisation Schemes

The raw co-occurrence counts are normalised using two measures:

• Probability Ratio

  $$\begin{aligned} \frac{P(w,f)}{P(w)P(f)} \end{aligned}$$

  where P(w, c) is the probability that words w and feature f have occurred together, and P(w) and P(f) are probabilities of occurrences of w and f. This measure tells us how often w and f were observed together in comparison to how often they would have occurred were they independent.

• Positive Pointwise Mutual Information (PPMI)

  $$\begin{aligned} \max (log(\frac{P(w,f)}{P(w)P(f)}), 0) \end{aligned}$$

  This is the positive version of the logarithm of probability ratio, where the negative logarithmic values are sent to 0.

1.2 A.2 Formulae for Computing Entailment

APinc is the average precision applied to feature inclusion. It measures a ranked version of feature inclusion on vectors $\overrightarrow{u}$ and $\overrightarrow{v}$, from highest to lowest:

$$\begin{aligned} \textit{APinc}(u,v) = \frac{\sum _r \left[ P(r) \cdot \textit{rel}'(f_r)\right] }{|F(\overrightarrow{u})|} \end{aligned}$$

(1)

In the above, $f_r$ is the feature in $\overrightarrow{u}$, denoted by $F(\overrightarrow{u})$, with rank r; P(r) is the precision at rank r, which measures how many of $\overrightarrow{v}$’s features are included at rank r in the features of $\overrightarrow{u}$, and $\textit{rel}'(f_r)$ is a relevance measure reflecting how important $f_r$ is in $\overrightarrow{v}$. It is computed as follows:

$$\begin{aligned} rel'(f) = \left\{ \begin{array}{lr} 1-\frac{\textit{rank}(f,F(\overrightarrow{v}))}{|F(\overrightarrow{v})|+1} &{} f \in F(\overrightarrow{v}) \\ 0 &{} o.w. \end{array}\right. \end{aligned}$$

(2)

BAPinc balances APinc with the LIN degree of similarity between the vectors. BAPinc was developed in [14] after realising that APinc returns poor results when the vectors had a radically different number of non-zero features; the LIN measure was included to balance out the extra dimensions of the longer vector.

$$\begin{aligned} \textit{BAPinc}(u,v) = \sqrt{\textit{LIN}(u,v) \cdot \textit{APinc}(u,v)} \end{aligned}$$

(3)

LIN is a similarity measure between vectors and was defined in [22]. It can be replaced with any other similarity measure, such as the cosine measure.

SAPinc is a measure developed in [12], based on BAPinc, but for dense vectors. Whereas APinc and BAPinc were developed to compute the degree of entailment between word vectors, which are usually sparse since word vectors live in high dimensional spaces (e.g. 5000), SAPinc was developed to deal with phrase and sentence vectors. These are obtained by composing the vectors of words in lower dimension (e.g. 300), where the compositional operators accumulate the information and return dense results.

$$\begin{aligned} \textit{SAPinc}(u,v) = \frac{\sum _r \left[ P(r) \cdot \textit{rel}'(f_r)\right] }{|\overrightarrow{u}|} \end{aligned}$$

(4)

Here, P(r) and $rel'(f_r)$ are defined differently, as shown below:

$$\begin{aligned} P(r) = \frac{\big |\{ f_r^{(u)} | f_r^{(u)} \le f_r^{(v)}, 0 < r \le |\overrightarrow{u}| \}\big |}{r} \end{aligned}$$

(5)

$$\begin{aligned} rel'(f_r) = \left\{ \begin{array}{lr} 1 &{} f_r^{(u)} \le f_r^{(v)} \\ 0 &{} o.w. \end{array} \right. \end{aligned}$$

(6)

For more explanations on these measures please see [12, 13].

1.3 A.3  Experimental Results for a Second Sample

The results of the experiment of Sect. 6, with PPMI and probability ratio matrices on the second 1000 sample of the dataset are presented in Fig. 7.

Fig. 7.

Results of the non-commutative conjunction experiment with the PPMI and probability ratio on the 2nd sample of dataset.

Similar to the results presented in the paper, the non-commutative operation performs better on recognising the non-commutative conjunctive entailments.