Real-time segmentation of strawberry flesh and calyx from images of singulated strawberries during postharvest processing

Highlights

• An image processing to extract the flesh and the calyx from an image is proposed.

• It relies on image color segmentation in a two dimensional color space.

• Results are improved thanks to a blob detection and selection stage.

• The algorithm is to tune and allows accurate extraction of the areas of interest.

• It deals with natural variation in strawberry shape and visual appearance.

Abstract

This paper presents an image processing algorithm that automatically extracts the flesh and calyx areas from strawberry images. Images are captured by a camera included in a strawberry de-capping machine. Lighting is controlled and the background is known, conditions that are typical of postharvest processing. The goal is to extract as many flesh and calyx pixels as possible while rejecting any pixels belonging to the background. The proposed approach relies on image color segmentation in a two-dimensional color space, followed by a blob detection and selection stage. A set of 250 images is used to analyze the sensitivity of the algorithm with respect to user-defined parameters, and evaluate the performance of the approach. The algorithm appears to be easy to tune and allows accurate extraction of the areas of interest despite natural variation in strawberry shape and visual appearance. More than $98\%$ of the flesh area was successfully extracted by the algorithm with less than $1\%$ of the background pixels falsely included. Moreover, up to $79\%$ of the calyx area could be extracted with less than $0.25\%$ erroneous background pixels. Finally, the algorithm has been implemented using the C++ and Cuda languages and can be executed in real-time.

Introduction

Since 2000, the worldwide annual strawberry production has been constantly increasing. Recent data shows that between 2010 and 2014 production increased from 6.6 to 8.1 million tons (Food and Agriculture Organization, 2016). A significant portion of harvested strawberries is processed. For example, approximately 25% of all strawberries harvested in California (2.3 bn tons in 2014) are frozen for the processed market. Also, consumer demand for ready-to-eat, fresh-cut packaged fruits has been steadily increasing (James et al., 2011). Fresh strawberry processing tasks include capping (calyx removal), size sorting, grading, slicing, pick-and-placing and packing. Currently, these tasks (excluding size sorting) are performed manually. Automated, cost-effective execution of these tasks requires singulating, orienting and handling strawberries of varying sizes and shapes at high-throughput rates. Earlier attempts that utilized purely mechanical means such as vibration, rotation, pinching, suction to handle strawberries (e.g. Gerrans et al., 1963, Leban, 1976, Ledebuhr et al., 1978) proved impractical due to high failure rates in fruit positioning and selective handling. It seems then mandatory to consider approaches including computer vision systems, similarly to various recent food processing projects (Du and Sun, 2004, Vibhute and Bodhe, 2012).

This paper addresses the flesh-calyx segmentation problem during strawberry processing at the postharvest stage. More specifically, a real-time vision system is presented that is part of a strawberry de-capping machine (see Fig. 1). The capping machine comprises several successive stages: feeding and singulation, orientation, vision system (calyx-flesh extraction), cutting (computation of cutting line and cutting action), and ejection. The images are captured under controlled lighting conditions as the fruits lie still on a moving roller conveyor (the rollers initially rotate to orient the fruits but stop before entering the imaging section). The vision system has to calculate two binary images representing the flesh and the calyx respectively from images containing singulated strawberries at random positions and orientations (see Fig. 2). The position of the calculated cutting line determines the amount of flesh removed and depends on the accuracy of the flesh and calyx extraction. Thus, to be as accurate as possible, the algorithm has to detect as many flesh and calyx pixels as possible (i.e., “the signal”) while minimizing the number of background pixels that are erroneously labeled as flesh or calyx (i.e., the “noise”). Moreover, the image processing algorithm has to be fast enough to run in real time and not limit the overall throughput of the de-capping machine. Indeed, the cost effectiveness of such a machine is strongly related to its capacity to process a large number of berries per second.

Computer vision has already been used in strawberries harvesting and post harvest processes to provide feedback about berry orientation and position. In Nagata et al. (2000) the authors used a camera in order to compute strawberry orientation on a sorting device by extracting the flesh area. To do so, the image is segmented in gray color space based on a user-defined threshold. A similar approach is used in Saenz et al. (2013) to design a harvesting robot prototype. In the work presented in Hayashi et al. (2011), a system to handle strawberries using a suction device is presented. In this system, the flesh and calyx are obtained by segmenting the image in a color space made by a composition of the red green and blue layers. In these works, the areas of interest (flesh and/or calyx) are extracted using user-defined thresholds. In Feng et al., 2008, Han et al., 2012, Qingchun et al., 2012 similar approaches using user-defined thresholds are used in the OHTA and HSI color spaces. The works presented in Liming and Yanchao, 2010, Yamamoto et al., 2012 propose to extract the flesh with a segmentation process that uses automatically computed thresholds (Otsu’s method) either in the V color space of HSV, or in the G-R color space (green layer minus the red one). A similar approach using a variant of Otsu’s method in the RGB color space is presented in Wei et al. (2014). Finally, the work presented in Ouyang et al. (2013) proposed a different and more advanced approach. In order to identify disease, a set of processes is applied to the image to extract the flesh: median filter denoising, segmentation using Otus’s method, mean shift clustering and morphological operations. The previously presented approaches focus solely on flesh extraction, except for the work presented by Liming and Yanchao, 2010, Han et al., 2012 which propose simple methods to extract also the calyx and the stems respectively. An image processing algorithm that extracts accurately and rapidly the flesh as well as the calyx from the images has not been reported.

The algorithm proposed in this paper relies also on a threshold based segmentation for both flesh and calyx but then extends the previous works with a blobs detection step to improve the accuracy of the extraction. This choice relies on two observations. First, more advanced segmentation techniques such as clustering, edge detection, classification or region growing are more relevant in challenging environments with uncontrolled lighting and background (e.g., fruit detection in orchard) but too time consuming for real-time postharvest processing operations where lighting and background are controlled. Second, the threshold based segmentation on its own does not provide highly accurate extraction of the flesh. Indeed, in the existing literature, the processes operate always at the pixel level, and therefore cannot consider higher-level features such as size, shape, position or orientation of a blob. This limits the ability to detect whether the pixels extracted via segmentation actually belong to the area of interest, or they are irrelevant.

Thus, the image processing algorithm proposed in this work first relies on a color-based segmentation using two color spaces. The first one R-G to detect the flesh, and the second one G-B for the whole fruit. Next, the obtained images are processed to detect existing blobs and compute some of their geometrical features. Finally, thresholds-based algorithms dedicated to strawberries are designed to keep or remove blobs. Thus, they maximize the accuracy of the extraction. This blob-based approach is novel because it relies on a higher level of abstraction than previous approaches and unlike prior work does not use morphological operations that typically distort the true fruit shape. Another contribution is the sensitivity analysis of the algorithms performance with respect to user-defined thresholds, which allows for easy tuning of the algorithm. Finally, although part of a specific system, the flesh-calyx extraction algorithm is generic, in that it operates under conditions (controlled lighting and background) that are typical of other strawberry postharvest processing operations.

The work is presented as follows. In the first section, the image acquisition equipment and methods used to acquire the images of the strawberries are presented. Next, the set of image processing operations used to extract the flesh and the calyx are described in detail. In the third section, a set of 250 images is used to evaluate the performance of the algorithm and its sensitivity to the thresholds. To do so, the percentage of extracted signal and noise, i.e., the pixels respectively representing the fruit and the background, are used as performance metrics, and results are reported for several sets of features thresholds. Moreover, a comparison with results obtained using approaches from other works is presented. The last section summarizes the presented work and results, and discusses potential improvements.