•  
  •  
 

Turkish Journal of Electrical Engineering and Computer Sciences

DOI

10.3906/elk-1906-30

Abstract

One of the critical factors that make a search engine fast and accurate is a concise and duplicate free index. In order to remove duplicate and near-duplicate (DND) documents from the index, a search engine needs a swift and reliable DND text document detection system. Traditional approaches to this problem, such as brute force comparisons or simple hash-based algorithms, are not suitable as they are not scalable and are not capable of detecting near-duplicate documents effectively. In this paper, a new signature-based approach to text similarity detection is introduced, which is fast, scalable, and reliable and needs less storage space. The proposed method is examined on standard text document datasets such as CiteseerX, Enron, Gold Set of Near-duplicate News Articles, and other similar datasets. The results are promising and comparable with the best cutting-edge algorithms considering accuracy and performance. The proposed method is based on the idea of using reference texts to generate signatures for text documents. The novelty of this paper is the use of genetic algorithms to generate better reference texts.

Keywords

Text similarity, near-duplicate, reference text, genetic algorithm

First Page

999

Last Page

1013

Download

Share

COinS