Automating hybrid collective intelligence in open-ended medical diagnostics

Significance In the United States, an estimated 250,000 people die annually from preventable medical errors, many of which originate during the diagnostic process. A powerful approach to increase diagnostic accuracy is to combine the diagnoses of multiple diagnosticians. However, we lack methods to aggregate independent diagnoses in general medical diagnostics. Using knowledge engineering methods, we introduce a fully automated solution to this problem. We tested our solution on 1,333 medical cases, each of which was independently diagnosed by ten diagnosticians. Our solution substantially increases diagnostic accuracy: Single diagnosticians achieved 46% accuracy, pooling the decisions of ten diagnosticians increased this to 76%. These results demonstrate that collective intelligence can reduce diagnostic errors, promoting health services and trust in the global medical community.

terminology in the world (1, 2). 23 The method for constructing the KG is reported in Figure S8, depicting a workflow using the UML notation. The 24 methodology incorporates recent insights into best practices for KG construction (3, 4) and takes inspiration from previous 25 projects including ArCo (5) and ScholarlyData (6). The methodology consists of the following three activities detailed below: 26 ontology design, mapping-based KG population, and KG enrichment with SNOMED CT.

27
Ontology design. Figure S9 shows the ontology diagram. The diagram can be read from the concept :Case, which rep-  Mapping-based knowledge graph population. The second activity (cf. Figure S8)  11. w: sort the words in alphabetic order.

76
Because diagnosticians commonly refer to clinical concepts using their acronyms, we extended the SNOMED CT terminology 77 by deriving and adding acronyms for clinical concepts. This step aims at increasing the matching between elicited diagnoses and 78 SNOMED CT concepts. To derive and add acronyms for SNOMED CT concepts, we exploited the naming conventions used 79 for abbreviations and acronyms in the existing labels and synonyms ‡ ‡ to design regular expressions for extracting acronyms.

80
For example, from the label "OSA -Obstructive sleep apnea" the acronym "OSA" is extracted and added to the corresponding 81 concept. To avoid introducing ambiguities in the matching and linking process, an acronym was only added for a SNOMED

82
CT concept if that acronym did not already exist for another concept. 83 We then used the LIMES framework to align diseases in our KG to the clinical terms in SNOMED CT by using the acronyms 84 and normalized labels for computing the Jaccard similarity. For each disease associated with a diagnosis, we identified which linked to the clinical term with SCTID 520890018 which represents the "arteritis disorder" in SNOMED CT.

93
While the example in Fig. S10 shows an exact match between a disease in our KG and a SNOMED CT concept (expressed 94 with a owl:sameAs relationship), we have also designed and instantiated (as part of the overall KG) a more general ontological 95 model for representing matches between diseases and SNOMED CT concepts other than diseases (e.g., disorder, finding, 96 morphological abnormality, body structure, person, organism, or specimen). In a nutshell, the general model allows representing 97 a matching (as identified by the LIMES framework) in terms of: accuracy because it can increase the number of valid judgments that effectively enter the aggregation (especially when small 115 deviations, such as spelling errors or typos were preventing an exact match; that is, whenever they were not covered by our 116 normalization pipeline). However, Jaccard similarity threshold values below 1 may also introduce more errors when linking a 117 user's diagnosis to their intended SCTID concept thus reducing individual and collective diagnostic accuracy. 118 We performed additional analyses relaxing this strict threshold when matching users' diagnoses to a SCTID. For the correct 119 diagnoses, we kept the strict threshold of 1. This was done because we wanted to compare the results to the same set of 1,333 120 cases (i.e., obtained when using a JI of 1). Otherwise, these simulations exactly followed the simulations reported in the main 121 text, that is, we first ranked SCTIDs based on their collective support (using the three different scoring rules). In case of 122 tied scores, we ordered SCTIDs according to their semantic tags (in the order: disorder, finding, morphological abnormality, 123 organism), and when SCTIDs were still tied, we randomized the order within the respective tied SCTIDs. SI Appendix, Fig   124   S7 shows how diagnostic accuracy scales with group size when including all terms with a Jaccard similarity value of 0.6 and 125 higher. As can be seen, this substantially reduced diagnostic accuracy at all group sizes (though maintaining the positive effect 126 of increasing group size). This is most likely the result of introducing increasingly more errors when assigning users' diagnoses 127 to a SCTID. Lower values led to even worse performance. Also thresholds above 0.6 did not improve performance compared to   The likelihood that a given diagnosis was correct was highest when a user only gave one diagnosis (and by extension that diagnoses was ranked first). Users providing more than one diagnosis also had a relatively high likelihood that their first-ranked diagnosis was correct, but this likelihood quickly dropped for lower-ranked diagnoses.    Activities are depicted as light-gray boxes, while the black circle and encircled black circle represent the initial and final nodes, respectively. The arrows among activities represent the direction of the workflow execution. The dark-gray boxes pinned on the activities identifies the objects required by the activities as input. Fig. S9. Ontology. The ontology is depicted by using the Graphical Framework for OWL Ontologies (Graffoo) notation. In this notation the classes are represented as yellow rectangles, the datatypes as green parallelograms, the object properties as blue directed arrows, and the predicates as black directed arrows. The namespace adopted for defining ontology entities (i.e. classes and properties) is https://w3id.org/stlab/crome/ontology/, which is associated with the prefix : in the diagram. The sample KG is depicted by using the Graphical Framework for OWL Ontologies (Graffoo) notation. In this diagram instances are depicted as pink circles.