Carotid Lumen Segmentation and Stenosis Grading Challenge

This paper describes the Carotid Lumen Segmentation and Stenosis Grading Challenge, which was part of the workshop ``3D Segmentation in the Clinic: A Grand Challenge’’. This workshop was part of the 12th conference on Medical Image Computing and Computer Assisted Interventions, held in London in September 2009. We introduce the purpose and background of the challenge and describe the data, reference standard and evaluation measures that were used in this challenge. We also provide the results of the data that were submitted by industrial and academic research institutes.


Introduction
This document describes the Carotid Lumen Segmentation and Stenosis Grading Challenge (CLS).The CLS is organized as one of the challenges of the 3rd MICCAI Workshop in the series "3D Segmentation in the Clinic: a Grand Challenge".
The CLS has two seperate challenges: 1. Carotid bifurcation lumen segmentation

Internal carotid artery stenosis grading
Each team can participate in either one of the challenges, or in both.This document describes respectively the challenge (i.e. the tasks to be performed), the data used, the manual annotation, the reference standard and the evaluation criteria.

Challenges 2.1 Lumen Segmentation
The Common Carotid Artery (CCA) and Internal Carotid Artery (ICA), see Fig. 1, are clinically the most relevant arteries of the Carotid Bifurcation.Therefore, the segmentation challenge focuses on these two arteries.A small part of the External Carotid Artery (ECA) is also included, to prevent evaluation issues at the location where the ECA bifurcates from the ICA.Additionally, it allows us to include a complete bifurcation in the challenge.The goal of this challenge is to accurately segment the lumen of the Carotid Bifurcation in a Computed Tomography Angiography (CTA) dataset.There are two versions of this challenge: a fully automated version, and a semi-automated version where three initial points are provided.
The region to be segmented is defined around the bifurcation slice, which we define as the first (caudal to cranial) slice where the lumen of the CCA appears as two separate lumens: the lumen of the ICA and the lumen of the ECA.The segmentation must contain the CCA, starting at least 20 mm caudal of bifurcation slice, the ICA, up to at least 40 mm cranial of bifurcation slice, and the ECA, up to between 10 and 20 mm cranial of the bifurcation slice, see also Fig. 1.
The performance measures are only determined over the region of interest as specified above.However, the bifurcation slice is not communicated to the participants of the challenge.Therefore, the participants should make sure that there segmentation at least includes this region.Our definition of the bifurcation slice, and the specified regions, should be sufficient to determine a suitable region of interest for the segmentations.
For the External Carotid Artery, the segmented lumen should be cut between 10 and 20 mm cranial of the bifurcation slice.To allow for some flexibility in cutting of the ECA, the region around the ECA between 10 and 20 mm cranial of the bifurcation slice is a "masked" region, where the evaluation measures will not be evaluated, see also Fig. 1.
The input data for participants of the challenges is (see Section 7 for more detailed information): • the CTA dataset (including header information such as voxel sizes and world coordinate system), and • three points if you join the semi-automatic method challenge: 1. a point in the Common Carotid Artery, at the level of the cranial side of the thyroid gland 2. a point in the Internal Carotid Artery, just before the artery enters the skull base 3. a point in the External Carotid Artery, where the artery is close to the mandible The participant will be asked to return the segmented lumen.This segmentation must be respresented as a (partial volume) segmentation, i.e. an image with floating point numbers, where each voxel value contains the occupancy of the voxel by the vessel lumen, where a value of 0 means no lumen present, and a value of 1 means fully occupied with lumen.The voxel value must thus be in the range [0,1].

Stenosis Grading
Two different stenosis grades have to be determined for each ICA that needs to be segmented in the segmentation challenge.We use the following NASCET-like definitions for stenosis grading: where S a is an area-based stenosis grade, and S d is a diameter-based stenosis grade.The stenosis grade is a value in the range [0 . . .100], where 0 implies no stenosis, and 100 implies a fully occluded vessel.
In the above formulations, a m is the minimal cross-sectional area along the CCA and ICA, and a r is the average cross-sectional area over a distal reference part of the Internal Carotid Artery.The default reference part has a length of 10 mm, and is 20 mm distal of the location of minimal area measured along the vessel centerline.However, observers are free to change the location and length of the reference area, with the restriction that it must be distal to the minimal area location, and not extend outside the segmented region, i.e. beyond 40 mm cranial of the bifurcation plane.
The second stenosis grade is determined using minimal diameters.The minimal diameter of a cross-section is defined as the shortest straight line that divides the contour in two equal-sized areas, see Fig. 2 for examples of minimal diameters for various contour shapes.
Similar to the lumen segmentation challenge, there are two versions of the stenosis grading challenge: a fully automated version, where the stenosis grading uses only the CTA dataset and the specification whether the left or right side needs to be graded, and a semi-automated version, where the algorithm also may use the three points in each of the arteries of the bifurcation (as supplied with the data).The input data available for this challenge is identical to the data for the lumen segmentation challenge.

Data acquisition
The CTA datasets used in this challenge have been acquired at the Erasmus MC, University Medical Center Rotterdam, The Netherlands (36 datasets), Hôpital Louis Pradel, Bron, France (10 datasets) and the Hadassah Hebrew University Medical Centre, Jerusalem, Israel (10 datasets).The datasets have been selected such that the contain a large range of stenoses: from fully open to severe stenotic.Below details of the scanning protocols are provided for each of the medical centers that provided data for this challenge.The CTA scanning parameters are summarized in Table 1.

Erasmus MC protocol
The CTA data were acquired on a 16-row CT scanner (Sensation 16 -Siemens Medical Solutions, Forchheim, Germany) with a standard scan protocol using the following parameters: 120 kV, 180 mAs, collimation 16×0.75 mm, table feed per rotation 12 mm, pitch 1.0, rotation time 0.5 seconds and scan time 10-14 seconds.The CTA scan range is from the ascending aorta to the intracranial circulation (2 cm above the sella turcica).
All patients received 80 ml contrast material (Iodixanol 320 mg/ml, Visipaque Amersham Health, Little Chalfont, UK), followed by 40 ml saline bolus chaser, both with an injection rate of 4 ml/sec.Synchronization between the passage of contrast material and data acquisition was achieved by real time bolus tracking at the level of the ascending aorta.The trigger threshold was set at an increase in attenuation of 75 Hounsfield Units (HU) above baseline attenuation (approx.150 HU in absolute HU value).Image reconstructions were made with in-plane pixel sizes of 0.23-0.26×0.23-0.26mm 2 , matrix size 512×512 (real in-plane resolution 0.6×0.6 mm), slice thickness 1.0 mm, increment 0.6 mm and with an intermediate reconstruction kernel (B30f).

Hadassah protocol
The CTA data were acquired on a 64-row CT scanner (Brilliance 64 -Philips Healthcare, Cleveland OH) with a standard scan protocol using the following parameters: 120 kV, 251 mAs, collimation 64×0.625 mm, pitch 1.20, rotation time 0.75 seconds and scan time 7.30 seconds.The CTA scan range is from the ascending aorta to the intracranial circulation (2 cm above the sella turcica).All patients received 75 ml contrast material (Iopamiro, Bracco Diagnostics, Milano Italy), with an injection rate of 3.5 ml/sec.Image reconstructions were made with in-plane pixel sizes of 0.55×0.55mm 2 , matrix size 512×512, slice thickness 1.0 mm, increment 0.5 mm and with an intermediate reconstruction kernel (B).

Louis Pradel protocol
The CTA data were acquired on a 64-row CT scanner (Brilliance 64 -Philips Healthcare, Cleveland OH) with a standard scan protocol using the following parameters: 120 kV, 300 mAs, collimation 52×1.5 mm,rotation time 0.35 seconds and scan time 10-14 seconds.The CTA scan range is from ascending aorta to the intracranial circulation (2 cm above the sella turcica).All patients received 80 ml contrast material (Iomeron 4000 mg/ml, BRACCO, Milano, Italy) followed by 40 ml saline bolus chaser, both with an injection rate of 4 ml/sec.Synchronization between the passage of contrast material and data acquisition was achieved by real time bolus tracking at the level of the ascending aorta.The trigger threshold was set at an increase in attenuation of 75 HU above baseline attenuation.Image reconstructions were made with in-plane pixel sizes of 0.414-0.547×0.450.414-0.547mm 2 , matrix size 512×512 (real inplane resolution 0.6×0.6 mm), slice thickness 0.9 mm, increment 0.45 mm with an intermediate reconstruction kernel (B).

Datasets for training, testing and on-site challenge
In total 56 datasets have been acquired for this challenge.Each dataset has a unique three-digit id.The first digit shows in which center the data were acquired (0 = Erasmus MC, 1 = Hadassah, 2 = Louis Pradel), and the other two digits are a sequence number, starting at 00.The distribution of the datasets over the training, testing and on-site sets, with their numbering, is shown in Table 2 4 Manual annotations Three different observers annotated the carotid lumen boundary and graded the stenosis in the ICA.Each contributing center performed the annotations on the data it provided, thus the observers for each of the three centers were different.The manual annotations for the lumen segmentation and stenosis grading are performed with an in-house build tool, based on MeVisLab.The annotation procedure is performed as follows: 1.After clicking a point for the bifurcation slice, positions along the centerlines for both the ICA and ECA are clicked, starting in the CCA, 20 mm caudal of the bifurcation slice, and extending to 40 mm cranial of the bifurcation slice (ICA) or ending 20 mm cranial of the bifurcation slice (ECA), see Fig. 3.
2. The resampled centerlines are used to generated Curved Multi Planar Reformatted images (CMPRs), in which longitudinal contours are drawn for three different orientations (each 60 o apart) of the CMPRs.
Cross-sectional contours orthogonal to the centerline are created at one mm intervals along the centerline.These spline-interpolated contours are initialized from the (six) positions where the cross-sectional plane intersects the longitudinal contours.The contours can be edited and updated if they do not match the luminal area, see also Fig. 4.
3. Based on the corrected cross-sectional contours, a graph for the contour area (or diameter) along the centerline is created.In the graph, the position of the minimal area (or diameter) can be selected, after which a default reference area is shown (20 mm distal, 10 mm length).This reference area can also be manually edited.The stenosis grade is determined using the values from these graphs, see also Fig. 5.
The first two steps are performed both for the ICA and the ECA.The ICA (and CCA contours) are used in the stenosis grading step, the ECA contours are not used for the stenosis grading.The contours are drawn with using standard window level settings (center = 176 HU, width = 800 HU).

Reference standard
The reference standard is created from the cross-sectional contours that result from the manual annotations.Below we describe how the contours of each of the three observers are turned into a signed distance map, and how the observer results are combined into the reference standard.

Preprocessing
The bifurcation slices are averaged, giving a reference bifurcation slice.This slice, and the bounding boxes of the contours, are used to determine the region of interest for the evaluation.The region of interest is the bounding box of the contours, extended with 15 mm both in x-and in y-direction.The z-range is determined from the reference bifurcation slice, and ranges from 20 mm caudal of the bifurcation slice to 40 mm cranial of the bifurcation slice.

Observer contour processing
The observer contours are processed as follows, see also Fig. 6:  • The contours (both for the ICA and the ECA) are turned into partial volume segmentations, using a Thin Plate Spline interpolation between the contour points [1], pv-i and pv-e.
• Signed distance maps sdm-i and sdm-e are generated from the partial volume segmentations pv-i and pv-e .
• The ICA and ECA signed distance maps are combined, giving a signed distance map for the complete bifurcation, sdm-b.
• The partial volume segmentations pv-i and pv-e are also combined (voxel wise maximum) to obtain the partial volume segmentation of the bifurcation, pv-b.This partial volume segmentation is used to rate the observer in the same way as the contestants segmentations are rated.
All signed distance maps use world distances in mm.

Reference standard lumen segmentation
The reference standard consists of a partial volume segmentation, signed distance map and an isosurface, and is constructed in the following way: sdm The observers signed distance maps of the bifurcation are averaged, to obtain an average signed distance map.
iso The zero-crossing of this reference signed distance map gives the reference lumen surface.
pv From the reference standard signed distance map, a partial volume representation is generated, by interpolating the distance map on super-resolution, and averaging the voxels that have a negative distance (i.e. that are inside) ext From the observer signed distance maps of the ICA and ECA, average distance maps of the ICA and ECA are created.The mask of the distal part of the external contains all voxels that satisfy all of the following three criteria: 1. the voxel is in the 10-20 mm range cranial of the bifurcation 2. the ECA signed distance map value of the voxel is less than 2 mm, i.e. the voxel is inside or close to the ECA 3. the ECA signed distance map value is less than the ICA signed distance map value, i.e. the voxel is closer to the ECA than to the ICA.

Stenosis values
The observer values for the stenosis are averaged to obtain the reference standard stenosis values.

Stenosis grading
The evaluation of the stenosis grade is straightforward: the absolute difference between the reference standard value and the value determined by a participant is the error in stenosis grade.As revealing the (exact) error per dataset also more or less reveals the reference stenosis grades, the stenosis errors are not communicated per dataset, but only per ensemble (testing or on-site).The same holds for the ranking.The final ranking, however, is determined by averaging the (hidden) errors per dataset and stenosis grade (diameter and area).
7 Datasets, fileformats and software This section describes the datasets that are available via the website, the format of the data, etc.

Testing and on-site challenge data
The testing data contains all data that might be needed to compute the lumen segmentation and/or stenosis grading.It is distributed in several archives that need to be extracted in the same directory.Extraction of the archives will give one subdirectory for each challenge dataset, named challenge<cid>, where <cid> stands for the challenge id, being a three-digit id for the challenge (e.g.003, 100, . . .).Per directory, the following files will be present: • cta<cid>[l|r].mhd and cta<cid> [l|r].raw: the header information, such as the world coordinate system and the voxel sizes and the pixel data, in 16-bits unsigned integers (Hounsfield units plus 1024).If the last character of the filename without extension is an l the left side (left carotid artery bifurcation) needs to be processed, if it is an r the right side needs to be processed.
• points<cid>.txt:a text file with three lines, containing the three coordinate positions that can be used for the initialization if you join one of the semi-automatic challenges.The first line contains the spatial coordinates of the point in the CCA, the second line contains the spatial coordinates of the point in the ICA, and the third line contains the spatial coordinates of the point in the ECA.All coordinates are in the world coordinate system of the CTA image.
• side<cid>.txt:a text file that contains "left" when the left carotid bifurcation needs to be processed, and "right" when the right carotid bifurcation needs to be processed.

Training data
The training data is distributed in the same structure, and contains per challenge directory the following files: • roi<cid>.txt:the region of interest (in voxels) The first line contains the minimum voxel index, and the second line contains the maximum (inclusive!) voxel index.
• ext range<cid>.txt: the region of interest (in voxels) The first line contains the minimum voxel index, and the second line contains the maximum (inclusive!) voxel index.
• pv<cid>.mhdand pv<cid>.raw: a floating point volume (in the size of the region of interest) containing the reference partial volume segmentation.
• sdm<cid>.mhdand sdm<cid>.raw:a floating point volume (in the size of the region of interest) containing the signed distance map of the segmentation (negative values are inside the lumen, positive values are outside the lumen).
• ext<cid>.mhdand ext<cid>.raw:a byte volume (in the size of the bounding box) containing the mask around the distal part of the external, where the performance measures will not be evaluated.A voxel with value 0 is a background voxel, a voxel with value 1 belongs to the mask.
• iso<cid>.vtp:a vtkPolyData file containing the isosurface of the signed distance map at the value 0.0.The surface is in world coordinates of the original dataset.
The world coordinate system of the above images that only contain a region of interest corresponds to the world coordinate system of the original CTA dataset.

Participant partial volume segmentations
The participants are required to upload their segmentation results as a floating point partial volume segmentation, using the same directory structure (challenge000 . . .challenge209) and file naming conventions: • roi<cid>.txt:the region of interest (in voxels).The first line contains the minimum voxel index, and the second line contains the maximum (inclusive!) voxel index.
• pv<cid>.mhdand pv<cid>.raw: the floating point volume (in the size of the bounding box) containing the reference partial volume segmentation.
The participants are encouraged to only upload the relevant portion of the data, to reduce the data bandwidth of our website.In case the participants region of interest does not overlap completely with the reference region of interest, the missing voxels are assumed to have a value of 0 (not containing any lumen).The world coordinate system in the participants image data will be ignored, the roi text file is the definitive source for the spatial relationship between the submitted data and the original CTA image.
Upon evaluation, a signed distance map and an isosurface representation of this partial volume segmentation will be generated for determining the performance measures.

Figure 1 -
Figure 1 -Schematic depiction of the region of interest that is relevant for the challenge, and a rendering of this region for one of the datasets.

Figure 2 -
Figure 2 -Minimal diameter lines for various cross-sectional contours.

Figure 3 -
Figure 3 -Screenshot of tool for center line drawing.

Figure 4 -
Figure 4 -Screenshot of tool for lumen contouring.

Figure 5 -
Figure 5 -Screenshot of tool for lumen stenosis grading.

Figure 6 -
Figure 6 -Processing of observer annotations, left 3D visualization and right 2D visualization.From top to bottom: initial contours, partial volume from contours (left: isosurface at 0.5) and signed distance map from partial volume (left: isosurface at 0.0).

Table 2 -
Datasets per center with their three-digit ids, and selection for training and testing