Comparing non-invasive diabetes risk scores for detecting patients in clinical practice: a cross-sectional validation study

Study population

The Cork and Kerry Diabetes and Heart Disease Study (Phase II – Mitchelstown Cohort) was a single centre study conducted between 2010 and 2011¹³. A random sample was recruited from a large primary care centre in Mitchelstown, County Cork, Ireland. The Livinghealth Clinic serves a population of approximately 20,000 Caucasian-European subjects, with a mix of urban and rural residents. Stratified sampling was employed to recruit equal numbers of men and women from all registered attending patients in the 46–73-year age group. In total, 3,807 potential participants were selected from the practice list. Following the exclusion of duplicates, deaths and subjects incapable of consenting or attending appointment, 3,051 were invited to participate in the study and of these, 2,047 (49% male) completed the questionnaire and physical examination components of the baseline assessment (response rate: 67%). Individuals with pre-existing cardiovascular disease or T2DM were not excluded from the cohort. Details of the study design, sampling procedures and methods of data collection have been reported previously¹³.

Ethics committee approval conforming to the Declaration of Helsinki was obtained from the Clinical Research Ethics Committee, University College Cork. A letter signed by the contact GP in the clinic was sent out to all selected participants with a reply slip indicating acceptance or refusal. All subjects gave signed informed consent, including permission to use their data for research purposes.

Clinical and anthropometric data

Clinical measurements were taken by researchers who were thoroughly trained according to the study research protocols. Participants attended the clinic the morning after an overnight fast (minimum 8 hours) and blood samples were taken on arrival. Data on age, sex, medical history and lifestyle behaviours were collected using a self-completed General Health Questionnaire¹⁴. Study participants answered questions regarding personal hypertension diagnosis/treatment, corticosteroid use and personal and family diabetes diagnosis/treatment. Smoking status was defined as never, former and current smokers. Diet was assessed using a modified version of the EPIC Food Frequency Questionnaire, validated for use in the Irish population¹⁵. Physical activity levels were assessed using the validated International Physical Activity Questionnaire (IPAQ)¹⁶ and subjects were classified as having low, moderate or high physical activity levels. Blood pressure (BP) was measured using an Omron M7 Digital BP monitor (Omron Healthcare Co. Ltd., Japan) on the right arm after a 5-minute rest in a seated position. The mean of the second and third measurements was used in analyses¹⁷.

The weight and height of each participant was measured to the nearest 0.1 kg and 0.1 cm respectively. Portable electronic Tanita WB-100MA weighing scales (Tanita Corporation, IL, USA) were placed on a firm, flat surface and were calibrated weekly to ensure accuracy. Height was measured using a portable Seca Leicester height/length stadiometer (Seca, Birmingham, UK) and body mass index (BMI) was calculated as weight divided by the square of height. Waist circumference was measured midway between the lowest rib and iliac crest on bare skin. Participants were instructed to breathe in, and then out, and to hold their breath while measurement was made to the nearest 0.1 cm using a Seca 200 measuring tape. The mean of two independent readings was used in analyses.

Biological analysis

Fasting glucose and glycated haemoglobin A_1c (HbA_1c) levels were measure by Cork University Hospital Biochemistry Laboratory. Fasting plasma glucose levels were collected the morning after an overnight fast using sodium fluoride EDTA collection tubes. Glucose concentrations were determined using a glucose hexokinase assay (Olympus Life and Material Science Europa Ltd., Lismeehan, Co. Clare, Ireland) and HbA_1c levels were measured in the haematology laboratory on an automated high-pressure liquid chromatography instrument Tosoh G7 [Tosoh HLC-723 (G7), Tosoh Europe N.V, Tessenderlo, Belgium]. For this study, T2DM was determined according to a fasting glucose level ≥7.0mmol/l or a HbA_1c level ≥6.5% (≥48mmol/mol). Participants with a self-reported diagnosis of diabetes, but who did not have positive test results, and subjects for whom we were missing glucose or HbA_1c data (n = 57) were excluded from the analysis. The final sample consisted of 1,990 subjects.

Diabetes risk scores

Nine non-invasive diabetes DRSs were assessed. These included the Brazil Diabetes Risk Score¹⁸, Cambridge Diabetes Risk Score¹⁹, Danish Diabetes Risk Score²⁰, the FINDRISC Models (Full and Concise)²¹, the Leicester Diabetes Risk Score²², the Rotterdam Predictive Models (1 and 2)²³ and the U.S. Diabetes Risk Score²⁴. These were chosen as they include readily available health information and have been developed to identify undiagnosed T2DM cases. Further details regarding the study populations and components used in each DRS are summarised in Table 1 and Supporting Table 1 (see data availability section)²⁵.

Statistical analysis

Descriptive characteristics were examined. Categorical features are presented as percentages and continuous variable are displayed as a mean (plus or minus one standard deviation) or a median and interquartile range for skewed data. Statistical differences were evaluated using chi-square tests, independent t-tests or a Mann-Whitney U.

Receiver operating characteristic curve (ROC) analysis was used to assess the ability of individual DRSs, and their components to discriminate prevalent T2DM cases. The area under the curve (AUC) provides a scale of 0.5 to 1.0, with 0.5 representing random chance and 1.0 indicating perfect discrimination, by which to compare the ability of a score or marker to detect a positive result. Optimal cut-off thresholds for DRSs were determined using the method that finds the cut-point which has a sensitivity and specificity closest to the top left of the ROC space. The diagnostic properties of optimal DRS cut-offs were compared by determining AUC values, sensitivity, specificity, positive and negative likelihood ratios (+LR, -LR) and positive and negative predictive values (PPV, NPV).

Using DRSs which had the highest AUC values, we determined the percentage of patients who would undergo further invasive testing, and the proportion of prevalent T2DM cases that would be correctly detected, according to stratification by DRS cut-offs corresponding to 95%, 90%, 85%, and 80% fixed sensitivities.

Data analysis was conducted using IBM SPSS Version 25 (IBM Corp, Armonk, NY, USA) for Windows. For all analyses, a P value (two-tailed) of less than .05 was considered to indicate statistical significance.

Descriptive characteristics

Baseline characteristics for the full sample, and for the participants with and without T2DM, are presented in Table 2. The number of participants with T2DM was 160, representing a prevalence of 8.0%. Participants with T2DM were more likely to be male, were older, had a greater body weight, a higher BMI and waist circumference as well as a higher systolic BP and were more likely to be taking prescribed antihypertensive medication. They also were more likely to report lower physical activity levels and a family history of diabetes compared to participants without T2DM.

Diagnostic statistics

Diagnostic statistics for DRSs, and individual DRS components, are shown in Table 3 and Supporting Table 2 (see data availability section)²⁵. Among DRSs, AUC values ranged from 0.71–0.78, with the Cambridge Diabetes Risk Score (AUC = 0.78, 95% CI: 0.75–0.82), Leicester Diabetes Risk Score (AUC = 0.78, 95% CI: 0.75–0.82), Rotterdam Predictive Model 2 (AUC = 0.78, 95% CI: 0.74–0.82) and the U.S. Diabetes Risk Score (AUC = 0.78, 95% CI: 0.74–0.81) demonstrating the largest AUC values as continuous variables and at optimal cut-offs (Figure 1). At an optimal cut-off, the Cambridge Diabetes Risk Score displayed the highest AUC value (AUC = 0.73, 95% CI: 0.69–0.77).

Figure 1. Receiver Operating Characteristic (ROC) curves to discriminate T2DM comparing DRSs using optimal cut-offs.

The area under the curve (AUC) values with 95% confidence intervals for each of the scores are as follows: Brazil Diabetes Risk Score: AUC = 0.68 (95% CI: 0.64–0.72); Cambridge Diabetes Risk Score: AUC = 0.73 (95% CI: 0.69–0.77); Danish Diabetes Risk Score: AUC = 0.70 (95% CI: 0.65–0.74); FINDRISC Concise Model: AUC = 0.68 (95% CI: 0.64–0.72); FINDRISC Full Model: AUC = 0.69 (95% CI: 0.65–0.73); Leicester Diabetes Risk Score: AUC = 0.71 (95% CI: 0.67–0.75); Rotterdam Predictive Model 1: AUC = 0.66 (95% CI: 0.61–0.70); Rotterdam Predictive Model 2: AUC = 0.72 (95% CI: 0.68–0.76); U.S. Diabetes Risk Score: AUC = 0.71 (95% CI: 0.67–0.76).

Regarding individual DRS components, anthropometric measures displayed the largest AUC values, with waist circumference (AUC = 0.74, 95% CI: 0.70–0.78) outperforming BMI (AUC = 0.71, 95% CI: 0.67–0.75) and weight (AUC = 0.68 95% CI: 0.64–0.72). Prescribed anti-hypertensive medication use (AUC = 0.65, 95% CI: 0.60–0.69) and family history of diabetes (AUC = 0.62, 95% CI: 0.57–0.67) displayed moderate discriminatory ability. Lifestyle variables displayed poor discrimination when examined individually.

Two-step screening

A comparison of the best performing DRSs at fixed sensitivities is presented in Table 4. This illustrates the percentage of patients which would undergo further testing, and the corresponding specificity, positive and negative likelihood ratios and positive and negative predictive values, using DRS cut-offs corresponding to 95%, 90%, 85% and 80% sensitivities. At a sensitivity of 95%, the percentage of patients who would undergo a further blood test ranged from 64.5–72.5%, with specificities ranging from 29.5–38.2%. Using a sensitivity of 90%, 56.2–64.6% of patients would be tested and specificities would range from 37.6–46.6%. At a sensitivity of 85%, approximately half (47.8–51.6%) of patients would undergo further invasive testing with a specificity range of 51.3–55.5%. When using DRS cut-offs corresponding to 80% sensitivity, 42.3–43.6% of patients would undergo further invasive testing, with a specificity range of 59.6–60.9%. Positive predictive values at this sensitivity for the fours scores ranged between 14.8–15.2%.

Strengths and limitations

Our study has several strengths. This research is the first to examine the ability of non-invasive DRSs to discriminate T2DM cases in a middle- to older-aged Irish population. Our results are of potential clinical significance with regards to case-finding and the use of DRSs to detect prevalent diabetes cases. The importance of identifying individuals with undiagnosed diabetes cannot be understated, given the litany of complications which arise in untreated cases. These complications, which include cardiovascular disease, vision loss, renal failure and cognitive decline, present a significant burden to healthcare systems, can have devastating consequences for individuals and importantly, are preventable^1,2. Notwithstanding these strengths, several limitations can be identified. These include measurement of HbA_1c and fasting plasma glucose at one time point and lack of Oral Glucose Tolerance test results as a comparison test. Another limitation of this study is the relatively modest number of diabetes cases available for analysis, which precluded stratification by sex and age. A larger sample would have conferred greater power and precision to our results. The cohort of this study is aged 46–73 years and therefore may not be generalizable to a younger population, however, T2DM is primarily a disease of middle-to-older aged people. An additional limitation of this study is that our data were derived from a single primary care-based sample, therefore the possibility that the sample may not be representative of the national population must be noted. However, previous research suggests that approximately 98% of Irish adults are registered with a GP and that, even in the absence of a universal patient registration system, it is possible to perform population-based epidemiological studies that are representative using our methods²⁸. In addition, as Ireland is an ethnically homogeneous population²⁹, the relationships observed in this study may be comparable to other middle- to older-aged Irish adults.

Reporting guidelines

Zenodo. TRIPOD checklist for ‘Comparing Non-invasive Diabetes Risk Scores for Detecting Patients in Clinical Practice.’ DOI: http://doi.org/10.5281/zenodo.5005201³⁰

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC BY 4.0 Public domain dedication).