Pill-ID: Matching and retrieval of drug pill images

https://doi.org/10.1016/j.patrec.2011.08.022Get rights and content

Abstract

Worldwide, law enforcement agencies are encountering a substantial increase in the number of illicit drug pills being circulated in our society. Identifying the source and manufacturer of these illicit drugs will help deter drug-related crimes. We have developed an automatic system, called Pill-ID to match drug pill images based on several features (i.e., imprint, color, and shape) of the tablet. The color and shape information is encoded as a three-dimensional histogram and invariant moments, respectively. The imprint on the pill is encoded as feature vectors derived from SIFT and MLBP descriptors. Experimental results using a database of drug pill images (1029 illicit drug pill images and 14,002 legal drug pill images) show 73.04% (84.47%) rank-1 (rank-20) retrieval accuracy.

Highlights

► This is the first systematic study of drug pill matching and retrieval. ► We have used robust SIFT and MLBP descriptors. ► The Pill-ID system can handle pill images with missing or damaged features.

Introduction

Illicit drugs, widely circulated in the international market, are one of the major factors influencing criminal activities. They also lead to additional enforcement and tracking expense for law enforcement units. More than 35 million individuals either used illicit drugs or abused prescription drugs in the United States in 2007 alone. The US federal government allocated more than $14 billion for drug treatment and prevention, counter-drug law enforcement, drug interdiction and international counter-drug assistance in 2009.1 Drug trafficking is also one of the major factors behind violent and other illegal activities.2

US Drug Enforcement Administration’s (DEA) Office of Forensic Sciences has been publishing Microgram Bulletin and Microgram Journal to assist forensic scientists for detection and analysis of drug-related substances.3 Food and Drug Administration (FDA) regulations4 require that every prescription pill or capsule sold in the market must have unique look for easy identification in terms of size, shape, color, and imprint. An imprint is an indented or printed mark on a pill, tablet, or capsule. Imprints can be a symbol, text, a set of digits, or any combination of them.5

Illicit drug makers use imprints, color, and shape to identify the chemical substances and their quantities in each pill. Special imprints on the pills are also used for advertisement purposes. When a new illicit psychoactive pill is first detected in the market, its information is recorded in the law enforcement databases.6 This information includes chemical and physical description, where the physical description includes shape, color, imprint, etc. Fig. 1 shows example images of legal and illicit drug pills.

Law enforcement units would like to automatically extract the information about an illicit drug pill (i.e., type of pill, manufacturing location, and the manufacturer) by matching its image with known patterns in their databases. Legal drug pills can be identified from the information provided by pharmaceutical companies enrolled in the FDA database. Fig. 2 shows the FDA information of three legal pills of the same type which have different dosage.

Unlike legal drug pills, illicit drug pills do not follow any regulation about their distinctiveness for the purpose of identification. Fig. 3 shows an image of an addictive illicit drug pill, called “Rohypnol” and its attributes. Information in the DEA database indicates that Rohypnol is currently supplied with a 1-milligram dose in an olive green, oblong tablet, imprinted with the number 542. In general, matching pill images with the patterns of previously seized pills is the only effective way to identify illicit drug pills.

Consequently, it is important to develop an image based matching tool to automatically identify illicit drug pills based on their imprint, size, shape, color, etc. There are a few web sites that provide keyword-based legal drug pill identification tools5,7. The keywords are based on the size, shape, and color of the pill (e.g., round, diamond, red, etc.), but they do not utilize the imprint. Keyword-based retrieval has a number of known limitations, namely keywords are subjective and do not capture all the information about the pill for accurate retrieval.

To develop a successful automatic pill image matching system, it is important to compensate for the variations in the appearance of the pills, due, for instance, to changes in viewpoint, illumination or occlusion (Riesenhuber and Poggio, 2000). For this reason, we utilize the gradient magnitude information to characterize the imprint patterns on the drug pill images.

Gradient magnitude is more stable than color or gray scale especially against illumination variations. Given the gradient magnitude image, Scale Invariant Feature Transform (SIFT) descriptor and Multi-scale Local Binary Pattern (MLBP) descriptors are used to generate feature vectors. In addition, invariant moment features proposed by Hu (1962) and color histogram are used to generate shape and color feature vectors, respectively. With the extracted feature vectors, we use the L2-norm to compute the similarity between two pill images. The proposed retrieval system, called Pill-ID is evaluated on a database consisting of 1029 illicit drug pill images and 14,002 legal drug pill images.

The rest of the paper is organized as follows. Section 2 discusses related work, Section 3 describes feature extraction and matching scheme, Section 4 presents experimental results, and Section 5 concludes the paper.

Section snippets

Related work

Given a query image, the goal of Content Based Image Retrieval (CBIR) is to retrieve visually similar images from a large image database on the basis of low level image features (such as color, texture, and shape). Constituents of the image features are extracted from both query and gallery images and used to find gallery images which are most similar to the query. Contrary to the keyword based image retrieval, CBIR does not need any manual labeling of the query image. One of the seminal paper

Feature extraction and matching

We use the gradient magnitude image in segmentation and feature vector construction for the shape and imprint of a pill. Feature vectors for imprints, color, and shape are constructed based on distribution based descriptors (i.e., SIFT and MLBP), color histogram, and Hu moments, respectively. The overall process is shown in Fig. 4.

Database

We received 891 illicit drug pill images from the Australian Federal Police. We also downloaded 138 illicit drug pill images and 14,002 legal pill images from the following web sites: US Drug Enforcement Administration’s Office of Forensic Sciences (DEA), Drug information online,5 and pharmer.org.8 Thus, we have a total of 15,031 pill images. The image size of the pill varies from 48 × 42 to 2088 × 1550 (width × height) with 96 dpi resolution.

To evaluate the Pill-ID

Conclusions and future work

We have developed an automatic drug pill matching and retrieval system. The system has been tested on a pill image database containing a total of 15,031 images, including 1029 illicit drug pill images. Rank-1 (Rank-20) identification accuracy of 73.17% (84.47%) is obtained in the matching experiments. These accuracy values are quite good, given the similarity in shape, color, and imprint information in many of the pills. This system can be used to identify illegal drug pills to assist law

Acknowledgments

The authors thank Dr. Mark Tahtouh of the Australian Federal Police for providing drug pill images. Anil K. Jain’s research was partially supported by WCU (World Class University) program through the National Research Foundation of Korea funded by the Ministry of Education, Science and Technology (R31-10008) to Korea university.

References (16)

  • A. Jain et al.

    Score normalization in multimodal biometric systems

    Pattern Recognit.

    (2005)
  • A. Bhattacharyya

    On a measure of divergence between two statistical populations defined by probability distributions

    Bull. Calcutta Math. Soc.

    (1943)
  • S.-K. Chang et al.

    Picture indexing and abstraction techniques for pictorial databases

    IEEE Trans. Pattern Anal Machine Intell.

    (1984)
  • M. Flickner et al.

    Query by image and video content: The QBIC system

    Computer

    (1995)
  • Z. Geradts et al.

    Content based information retrieval in forensic image databases

    J. Forensic Sci.

    (2002)
  • R. Gonzalez et al.

    Digital Image Processing

    (1992)
  • M.-K. Hu

    Visual pattern recognition by moment invariants

    IEEE Trans. Inform. Theor.

    (1962)
  • Jain, A., Lee, J., Jin, R., Gregg, N., 2009. Content-based image retrieval: An application to tattoo images. In: 16th...
There are more references available in the full text version of this article.

Cited by (42)

  • Automatically classifying crime scene images using machine learning methodologies

    2021, Forensic Science International: Digital Investigation
    Citation Excerpt :

    Some of the research concerning the use of image classification and CBIR systems has targeted specific forensic applications. For example, applications include automated classification and retrieval systems for footwear impressions (Geradts et al., 1995, 1996; Srihari, 2010; Srihari et al., 2014; Alexander et al., 1999), illicit drug tablets (Geradts et al., 2001; Camargo et al., 2012; Chen et al., 2010; Geradts, 2002; Lee et al., 2012), handwriting (Srihari and Shi, 2004), tool marks (Geradts, 2002), cartridge cases (Geradts, 2002), tyre tread patterns (Liu and Li, 2014), tattoos (K Jain et al., 2007; Jain et al., 2009) and graffiti (K Jain et al., 2009). However, such retrieval systems are based on databases containing a singular or a very limited number of image categories within a specific forensic domain and designed primarily for the use of one-to-many image queries and retrieval.

  • Parameter optimization of a multiscale descriptor for shape analysis on healthcare image datasets

    2019, Pattern Recognition Letters
    Citation Excerpt :

    Thus, the automatic identification of illicit pills based on pattern recognition algorithms is of great importance for forensic science, drug monitoring and the analysis of seized drugs around the world. The development of these computer systems for pill drug image recognition relies on color, imprint and shape features [10–13]. Shape descriptors can be employed as a preliminary filter to improve the performance of this task.

  • Development of fine-grained pill identification algorithm using deep convolutional network

    2017, Journal of Biomedical Informatics
    Citation Excerpt :

    Third, the requirement of manual inputting could be very time-consuming, especially when there are multiple pills that need to be identified. Recognizing the need for accurate recognition of pills, several approaches have been proposed to identify pill automatically through computer vision [23–29]. All the existing studies design some visual features for pill identification.

  • CNN-Based Pill Image Recognition for Retrieval Systems

    2023, Applied Sciences (Switzerland)
View all citing articles on Scopus

An earlier version of this work appeared in (Lee et al., 2010).

View full text