Improving Medical Data Annotation Including Humans in the Machine Learning Loop ^†

Abstract

At present, the great majority of Artificial Intelligence (AI) systems require the participation of humans in their development, tuning, and maintenance. Particularly, Machine Learning (ML) systems could greatly benefit from their expertise or knowledge. Thus, there is an increasing interest around how humans interact with those systems to obtain the best performance for both the AI system and the humans involved. Several approaches have been studied and proposed in the literature that can be gathered under the umbrella term of Human-in-the-Loop Machine Learning. The application of those techniques to the health informatics environment could provide a great value on prognosis and diagnosis tasks contributing to develop a better health service for Cancer related diseases.

1. Introduction

The majority of Machine Learning (ML) systems require the participation of humans at several steps of the AI pipeline. With this requirement in mind, new types of interactions between humans and machine learning algorithms are being defined, which we can group under the term Human-in-the-Loop Machine Learning (HITL-ML) [1]. The goal is to make machine learning models more accurate, obtain the desired accuracy faster, and also make humans more efficient when training or using a ML model.

In the health domain (and others), due to the reduced number of datasets, traditional ML approaches suffer from insufficient training samples [2]. Using specific techniques as the ones described in this proposal could help improving both the training process and the final user performance.

2. Materials and Methods

Hybrid Intelligence Systems include several strategies with the goal of enhancing the capabilities of either the human, the machine or both of them. A taxonomy has been proposed based on the task characteristics, learning paradigm, AI-Human interaction, and Human-AI interaction [3].

We describe below several techniques that can be grouped under the term Human-in-the-Loop Machine Learning (HITL-ML), and can be applied between others, to the Cancer prognosis and diagnosis scenarios, where training samples are scarce and domain expert knowledge is expensive.

2.1. Human-in-the-Loop Techniques

Human-in-the-Loop ML has the goal of increasing the accuracy of a ML model, reaching the target performance faster, combining human and machine intelligence to maximize accuracy, and assisting human tasks with machine learning to increase efficiency [1]. The most relevant tasks mentioned are:

• Annotating unlabeled data to create training, validation, and evaluation data.
• Sampling the most important unlabeled data items.
• Incorporating Human-Computer Interaction principles into annotation.

Depending on who is in control of the learning process, we do identify different approaches: Active Learning, Interactive Machine Learning, and Machine Teaching.

2.2. Active Learning

One of the first techniques is Active Learning (AL) [4], where the system remains in control of the learning process and treats humans as oracles to label relevant unlabeled data. It is particularly useful when the labeling example process is expensive or time-consuming, and it also applies to the scenario of scarcity of examples (e.g., cancer). AL uses an interactive/iterative process for obtaining training data, unlike passive or classical learning, where the data is provided in advance. The learner requests information from the oracle, that it selects based on different query strategies.

2.3. Interactive Machine Learning

Another approach is Interactive Machine Learning (IML), in which there is a closer interaction between users and learning systems, with people iteratively supplying information in a more focused, frequent and incremental way compared to traditional machine learning [5,6]. In this technique the learning process control is shared between the system and the users, working closely to benefit from each other.

2.4. Machine Teaching

Finally, Machine Teaching (MT) [7,8] where the idea is to focus on the teacher role a human can play to create useful information from the data available. With the aim of facilitating the construction of new models that nowadays require practitioners with deep knowledge of machine learning, this method proposes to decouple knowledge about machine learning algorithms from the process of teaching. The human would behave as a teacher guiding the learning process [9].

A particular version of MT is Iterative Machine Teaching (iMT) [10] whose goal is to obtain the optimal training set given a machine learning algorithm and a target model. The idea is to learn a target concept with a minimal number of iterations using the smallest dataset.

2.5. Applying and Interpreting the Results

Once the model is deployed and it is used in a production environment, we could use Explainable AI (XAI) [11] to make the results of AI systems more understandable to humans.

There are specific domains where the aforementioned methods could fulfill the targets of the expected model. As an example, ML-approaches can be of particular interest to solve issues in Health Informatics, where we are lacking big data sets, we need to deal with complex data and/or rare events, and traditional learning algorithms suffer due to insufficient training samples [2].

3. Results

To date, we explored two of the techniques exposed: Iterative Machine Teaching (iMT) and Active Learning (AL). We have analyzed how to integrate them in the learning process using common datasets: Gaussian, MNIST and Vehicles.

Our proposal to incorporate iMT and AL into the machine learning loop is to use iMT as a technique to obtain the “Minimum Viable Data (MVD)” for training a learning model, that is, a dataset that allows us to increase speed and reduce complexity in the learning process by allowing to build early prototypes.

The results of the application of the iMT and AL on known datasets can be found at [12]. There we can see that, in the iMT experiment, the results show—both in the example problems and in the real-world problem—that the algorithms trained by any of the proposed teachers obtain better results than those trained by randomly choosing the examples. In our AL experiment, we find that the greatest advantage of this approach is in the continuous improvement of the model, which enhances resilience and prevents obsolescence.

4. Discussion

The quality of the data is a key factor that can make the model to fail in certain scenarios. If our data is better our algorithms will generalize better. This is the idea of the so-called data-centric approach which is behind some of the techniques explored (i.e., Machine Teaching).

The methods described in this paper are not mutually exclusive, so they can be combined with the aim of obtaining better results. Some of the techniques apply at different stages of the ML pipeline. Furthermore they can be incrementally implemented enhancing the model at every step.

The outcomes of the experiments conducted were obtained using common datasets as inputs. Even if they are promising, we plan to apply these techniques to relevant medical databases as The Cancer Genome Atlas Program (TCGA).

As for future work, we would be interested in applying these techniques considering multi-class problems and utilize the TCGA datasets.

5. Conclusions

The techniques exposed (combined or individually) can be applied to a specific domain (Cancer diagnosis and prognosis) making Machine Learning (ML) methods accessible to subject-matter experts and improving the performance of both the system and the human (i.e., HITL-ML), obtaining semantic and interpretable ML models (i.e., Explainable AI).