String Retrieval for Multi-pattern Queries

Abstract

Given a collection $\mathcal D$ of string documents $\{d_1,d_2,...,d_{|\mathcal D|}\}$ of total length n, which may be preprocessed, a fundamental task is to retrieve the most relevant documents for a given query. The query consists of a set of m patterns {P ₁, P ₂, ..., P _m }. To measure the relevance of a document with respect to the query patterns, we may define a score, such as the number of occurrences of these patterns in the document, or the proximity of the given patterns within the document. To control the size of the output, we may also specify a threshold (or a parameter K), so that our task is to report all the documents which match the query with score more than threshold (or respectively, the K documents with the highest scores).

When the documents are strings (without word boundaries), the traditional inverted-index-based solutions may not be applicable. The single pattern retrieval case has been well-solved by [14,9]. When it comes to two or more patterns, the only non-trivial solution for proximity search and common document listing was given by [14], which took $\tilde{O}(n^{3/2})$ space. In this paper, we give the first linear space (and partly succinct) data structures, which can answer multi-pattern queries in $O(\sum |P_i|) + \tilde {O}(t^{1/m} n^{1-1/m})$ time, where t is the number of output occurrences. In the particular case of two patterns, we achieve the bound of $O(|P_1| + |P_2| + \sqrt{nt}\log^2 n)$. We also show space-time trade-offs for our data structures. Our approach is based on a novel data structure called the weight-balanced wavelet tree, which may be of independent interest.

This work is supported in part by Taiwan NSC Grant 96-2221-E-007-082 and US NSF Grants CCF-1017623 and CCF-0621457.