Types of studies

We anticipated that the majority of the studies would be of AD‐8 test properties compared against a contemporaneous clinical diagnosis of dementia in secondary‐care settings. We included test studies performed in other healthcare settings, and classified settings as secondary care, primary care, or community.

Case‐control studies are known to potentially overestimate properties of a test and we did not include such studies in the review (Davis 2013).

We also did not include case studies or samples with very small numbers (chosen as 10 participants, for the purposes of this review).

Where settings were mixed, for example, a population study 'enriched' with additional cases from secondary care, we did not consider such studies unless separate data were presented for participants from each setting. This design can be affected by similar biases to a case‐control design.

Participants

All adults (aged over 18 years) were eligible.

We did not predefine exclusion criteria relating to the 'case mix' of the population studied, but assessed this aspect of the study as part of our assessment of heterogeneity.

Index tests

Studies must have included (not necessarily exclusively) the AD‐8, used as an informant questionnaire.

The AD‐8 has been translated into various languages to enable international administration. The properties of a translated AD‐8 in a cohort of non‐English speakers may differ from the properties of the original English‐language questionnaire. We collected data on the principal language used for AD‐8 assessment.

For this review we did not consider other cognitive screening/assessment tools. Where a paper described the AD‐8 with in‐study comparison against another screening tool, we included the AD‐8 data only. Where the AD‐8 was used in combination with another cognitive screening tool, we included the AD‐8 data only.

Target conditions

We included any clinical diagnosis of all‐cause (unspecified) dementia. We did not require a definition of a particular dementia subtype, although we recorded this information where available.

Reference standards

Our reference standard is clinical diagnosis of dementia. We recognise that clinical diagnosis itself has a degree of variability but this is not unique to dementia studies and does not invalidate the basic diagnostic test accuracy (DTA) approach.

The primary analysis for clinical diagnosis included all‐cause (unspecified) dementia, using any recognised diagnostic criteria (for example, DSM‐IV; ICD‐10). We included use of the full Clinical Dementia Rating (CDR) Scale (Morris 1993) as an acceptable method for making a dementia diagnosis, with a CDR score greater than or equal to one for a clinical diagnosis of dementia. Dementia diagnosis may have specified a pathological subtype and we included all common dementia subtypes, e.g. NINCDS‐ADRDA (Alzheimer's disease), Lund‐Manchester (frontotemporal dementia), McKeith (Dementia with Lewy bodies), NINCDS‐AIREN (vascular dementia) (McKeith 2005; McKhann 1984; McKhann 2001; Román 1993). We did not define preferred diagnostic criteria for rarer forms of dementia (e.g. alcohol‐related; HIV‐related; prion disease‐related). These were considered under our rubric of 'all‐cause' dementia, and not considered separately.

The label 'dementia' can span a range of disease severities, from mild disease to 'end stage'. The diagnostic properties of a tool will vary depending on disease stage, for example, true positives are more likely when disease is advanced and diagnosis is clear. For our primary analysis we included any dementia diagnosis, at any stage of disease.

Clinicians may use imaging, pathology or other data to aid diagnosis; however, we did not include diagnosis based only on these data without corresponding clinical assessment. We recognise that different iterations of diagnostic criteria may not be directly comparable, and that diagnosis may vary with the degree or manner in which the criteria have been operationalised (e.g. individual clinician versus algorithm versus consensus determination). We collected data on method and application of dementia diagnosis for each study and explored the validity of the dementia diagnosis as part of our 'Risk of Bias' assessment. We did not accept use of other (brief) direct performance tests in isolation as a basis for diagnosis.

We recognise that dementia diagnosis often comprises a degree of informant assessment. Thus there is potential for incorporation bias. We explored the potential effects of this bias through our 'Risk of bias' assessment.

Electronic searches

We searched the specialised register of the Cochrane Dementia and Cognitive Improvement Group, ALOIS (which includes both intervention and diagnostic accuracy studies), MEDLINE (Ovid SP), Embase (Ovid SP), BIOSIS (ISI Web of Science), Web of Science Core Collection (ISI Web of Science), PsycINFO (Ovid SP), CINAHL (EBSCOhost) and LILACS (Bireme). We designed similarly‐structured search strategies and used search terms appropriate to each database. We used MeSH words and other controlled vocabulary where appropriate.

We also searched sources specific to diagnostic accuracy or systematic review:

· MEDION database (Meta‐analyses van Diagnostisch Onderzoek www.mediondatabase.nl);

· DARE (Database of Abstracts of Reviews of Effects) and HTA Database (Health Technology Assessments Database), both The Cochrane Library;

· ARIF database (Aggressive Research Intelligence Facility; www.arif.bham.ac.uk).

See Appendix 5 and Appendix 6 for the search strategies run.

We did not apply language or date restrictions to the electronic searches, nor did we apply restrictions regarding publication status when assessing records for potential inclusion, including abstracts, conference proceedings and unpublished data. We used translation services where necessary (see: Acknowledgements), and used a translation proforma for data extraction (Appendix 7).

The Cochrane Dementia and Cognitive Impairment Group Information Specialist (ANS) ran the initial searches.The most recent search for this review was performed on 7 June 2018.

Searching other resources

Grey literature and proceedings: we searched our chosen electronic databases and included relevant assessments of conference proceedings. Where possible we accessed theses or PhD abstracts from institutions known to be involved in prospective dementia studies.

Handsearching: we did not perform handsearching as there is little published evidence of the benefits of handsearching for diagnostic studies (Glanville 2010).

Reference lists: we checked the reference lists of all relevant studies and reviews in the field for further possible titles, and repeated the process until no new titles were found (Greenhalgh 2005).

Correspondence: We contacted research groups who have published or are conducting work on AD‐8 for dementia diagnosis, informed by results of initial search (see: Acknowledgements).

We used relevant studies in PubMed to search for additional studies using the 'related article' feature. We examined key studies in citation databases, such as Science Citation Index and Scopus, to ascertain details of any further relevant studies.

Selection of studies

One review author (ANS) screened all titles generated by initial electronic database searches for relevance. The initial search was a sensitive, generic search, designed to include all potential dementia screening tools. Two review authors (KH, CG) then independently screened all remaining titles for relevance. The two review authors inspected abstracts of selected titles and selected all potentially eligible studies for full‐text review. They independently assessed full manuscripts against the inclusion criteria, and resolved disagreements by discussion or by involving an arbitrator where necessary.

Where a study included useable data but did not present these in the published manuscript, we contacted the authors directly to request further information. If the same data were presented in more than one paper we included the primary paper only.

We have detailed the study selection process in a PRISMA flow diagram.

Data extraction and management

We extracted data to a study‐specific pro forma that included clinical/demographic details of the participants; details of setting; details of AD‐8 administration and details of the dementia diagnosis process.

Where AD‐8 data were presented for differing threshold scores, we extracted data for each threshold and collated these separately. We extracted test accuracy data to a standard two‐by‐two table.

Two review authors (KH, CG), blinded to study identifiers, performed data extraction independently. They resolved disagreements by discussion, with the use of an arbitrator if necessary. As a further check of validity of search and data extraction, a third reviewer (SA) assessed all titles and papers from the most recent updated literature search.

For each included paper, we detailed the flow of participants (numbers recruited, included, assessed) in a flow diagram.

Assessment of methodological quality

We assessed the methodological quality of each study using the QUADAS‐2 tool (www.bris.ac.uk/quadas/quadas‐2) (Appendix 8). This tool incorporates domains specific to participant selection; index test; reference standard; and participant flow. Each domain is assessed for risk of bias and the first three domains are also assessed for applicability. Certain key areas important for quality assessment are participant selection, blinding, and missing data. Following a group meeting of review authors, we created guidance for the application of QUADAS‐2 to dementia screening assessments, specifically developing anchoring statements for QUADAS‐based assessment that are suited to dementia test accuracy studies. This QUADAS‐2 guidance was created through a multidisciplinary working group and has been extensively piloted. The process and resulting statements for assessment are described in Appendix 9.

We did not use QUADAS‐2 data to form a summary quality score; we produced a narrative summary describing numbers of studies for which we found high/low/unclear risk of bias, or concerns regarding applicability with corresponding tabular and graphical displays.

Paired independent raters (KH, CG), blinded to each other’s scores, performed both assessments. They resolved disagreements by further review and discussion, with recourse to a third‐party arbitrator where necessary. A third reviewer assessed all titles selected from the most recent literature search.

Statistical analysis and data synthesis

We were interested in the test accuracy of the AD‐8 for the dichotomous variable 'dementia'/'no dementia'. Thus, we applied the current Cochrane DTA framework for analysis of a single test. We extracted data from included papers to allow creation of a standard two‐by‐two data table showing dichotomised AD‐8 test results (AD‐8 positive or AD‐8 negative), cross‐classified with binary reference standard (dementia or no dementia).

We used Review Manager 5 software (RevMan 2012) to calculate sensitivity, specificity and 95% confidence intervals (CIs) from the two‐by‐two tables abstracted from the included studies. We used a threshold score of two or more on the AD‐8 for primary analyses. If data at other thresholds were presented, we examined these in separate analyses (we were able to describe accuracy at cut‐off scores of one, two, three, and seven). We presented individual study results graphically by plotting estimates of sensitivities and specificities as forest plots.

To allow for summary analysis, we used SAS release 9.4, in addition to Review Manager 5. Using the bivariate approach we described metrics of pooled sensitivity, specificity, positive and negative likelihood ratios, all with corresponding 95% confidence intervals. We plotted summary data in receiver operating characteristic (ROC) space, including 95% confidence regions. Where we were interested in comparative accuracy and where data allowed, for example differential accuracy by setting, we plotted summary accuracy data for each variable in shared ROC space and described relative sensitivity and specificity of one factor to the other.

We suspected papers would use the classical cross‐sectional test accuracy study design. An alternative is the 'delayed verification' study design, i.e. where the AD‐8 is performed at baseline and those without disease are prospectively followed up for development of incident dementia. No included studies took this approach, but if they had, we planned to use baseline (contemporaneous testing) data for our primary analysis.

Investigations of heterogeneity

Heterogeneity is expected in DTA reviews, and 'traditional' measures of heterogeneity used in meta‐analysis are not appropriate to DTA reviews.

The properties of a tool describe behaviour of the instrument under particular circumstances. We included all AD‐8 studies in narrative review. We prespecified particular areas of potential heterogeneity, as follows.

Sensitivity analyses

Where appropriate (i.e. if not already explored in our analyses of heterogeneity), and as data allowed, we explored the sensitivity of any summary accuracy estimates to other methodological aspects of the included studies. We performed a sensitivity analysis excluding a paper with a disease‐specific patient group (those with delirium). We also performed a post‐hoc sensitivity analysis excluding a paper whose test accuracy estimates were based on a more inclusive definition of dementia than was used in the other papers. We had planned to perform sensitivity analyses based on key aspects that could indicate risk of bias, such as nature of blinding and loss to follow‐up, guided by the anchoring statements developed in our QUADAS‐2 exercise. Due to the modest number of included studies, and the lack of papers deemed to have low risk of bias on all domains, we were unable to perform these analyses.

Assessment of reporting bias

We did not investigate reporting bias because of current uncertainty about how it operates in test accuracy studies, and about the interpretation of existing analytical tools such as the funnel plot (van Enst 2014).