An R-tree for each object pair built on a spatial relation diagram of these two objects has been used for retrieving similar videos. The similarity between videos is measured based on the spatial relations between objects. Non-similar data can be filtered at early stages of the process of query using the R-tree. However, a better R-tree can be obtained by modifying the spatial relation diagram such that the efficiency of the process of query can be improved. Hence, we propose to use the genetic algorithm to build a near-optimal R-tree structure for each object pair for similarity retrieval of videos.