On the information difference between standard retrieval models

Recent work introduced a probabilistic framework that measures search engine performance information-theoretically. This allows for novel meta-evaluation measures such as Information Difference, which measures the magnitude of the difference between search engines in their ranking of documents. for which we have relevance information. Using Information Difference we can compare the behavior of search engines-which documents the search engine prefers, as well as search engine performance-how likely the search engine is to satisfy a hypothetical user. In this work, we a) extend this probabilistic framework to precision-oriented contexts, b) show that Information Difference can be used to detect similar search engines at shallow ranks, and c) demonstrate the utility of the Information Difference methodology by showing that well-tuned search engines employing different retrieval models are more similar than a well-tuned and a poorly tuned implementation of the same retrieval model.