Person-based design and evaluation of MIA, a digital medical interview assistant for radiology

Introduction Radiologists frequently lack direct patient contact due to time constraints. Digital medical interview assistants aim to facilitate the collection of health information. In this paper, we propose leveraging conversational agents to realize a medical interview assistant to facilitate medical history taking, while at the same time offering patients the opportunity to ask questions on the examination. Methods MIA, the digital medical interview assistant, was developed using a person-based design approach, involving patient opinions and expert knowledge during the design and development with a specific use case in collecting information before a mammography examination. MIA consists of two modules: the interview module and the question answering module (Q&A). To ensure interoperability with clinical information systems, we use HL7 FHIR to store and exchange the results collected by MIA during the patient interaction. The system was evaluated according to an existing evaluation framework that covers a broad range of aspects related to the technical quality of a conversational agent including usability, but also accessibility and security. Results Thirty-six patients recruited from two Swiss hospitals (Lindenhof group and Inselspital, Bern) and two patient organizations conducted the usability test. MIA was favorably received by the participants, who particularly noted the clarity of communication. However, there is room for improvement in the perceived quality of the conversation, the information provided, and the protection of privacy. The Q&A module achieved a precision of 0.51, a recall of 0.87 and an F-Score of 0.64 based on 114 questions asked by the participants. Security and accessibility also require improvements. Conclusion The applied person-based process described in this paper can provide best practices for future development of medical interview assistants. The application of a standardized evaluation framework helped in saving time and ensures comparability of results.


HEURISTICS
In the table, the heuristics underlying the heuristic evaluation are described.

Heuristic criteria
Expectations exceeded (3 points) Expectations reached (2 points) Below expectations (1 point) Visibility of system status The system always informs the user about what is happening -in good time and with appropriate feedback.
The system informs the user about what is currently happening -sometimes with a delay or without feedback.
The system rarely or never provides information about what is happening.
Feedback is non-existent or misleading.Match between system and real world The system speaks and understands the language of the user -with words, sentences and concepts that are familiar to the user (e.g.no technical terms).The information appears in a natural and logical order.The dialogue is conducted in such a way that the conversation starts and ends smoothly and is conducted appropriately.
The system mainly speaks and understands the language of the userwith words, sentences and concepts that are familiar to the user (e.g.no technical terms).The information appears in a natural and logical order.The dialogue is conducted in such a way that the conversation starts and ends smoothly and is conducted appropriately.
The system does not speak or understand the user's language -words, sentences and concepts are usually unfamiliar to the user.The information is not presented logically.The dialogue seems unnatural.

User control and freedom
The user often carries out actions unintentionally.The system offers ways to easily exit the state reached, especially without having to go through a long dialogue.Exits such as "Undo", "Redo" and "Cancel" are therefore always possible and visible.
The user often carries out actions unintentionally.The system offers partially ways to easily exit the state reached, especially without having to go through a long dialogue.Exits such as "Undo", "Redo" and "Cancel" are therefore often possible and visible.
The user often carries out actions unintentionally.The system offers no ways to easily exit the state reached, or only through a long dialogue.Exits such as "Undo", "Redo" and "Cancel" are impossible or barely visible.

Consistency and standards
The user does not have to think about whether different words, situations and actions mean the same thing.The user always receives consistent answers when they ask the same question in different ways.The system has a consistent language style and personality within the interaction.
The user does not have to think about whether different words, situations and actions mean the same thing.The user often receives consistent answers when they ask the same question in different ways.The system has often a consistent language style and personality within the interaction.
The user must consider whether different words, situations and actions mean the same thing.The user does not receive standardised answers if they ask the same question in different ways.The system does not have a consistent language style and personality within the interaction.

Error prevention
The dialogue is designed in such a way that no problems in the interaction occur.The dialogue runs undisturbed, without interruptions and pauses.Error messages do not occur.The system avoids error-prone situations situations or warns the user user before or in the event of misuse.
The dialogue is designed in such a way that often no problems in the interaction occur.The dialogue mostly runs undisturbed, only with few interruptions and pauses.
The system avoids errorprone situations situations or warns the user user before or in the event of misuse.
The dialogue is disrupted and only possible with interruptions and pauses.Error messages occur frequently.The system favours error-prone situations.It does not warn the user before or in the event of incorrect use.

Help and guidance
The system accompanies the user user during the entire dialogue dialogue by clarifying the possibilities of the system.The help functions are easy to find, focus on the user's task the user's task, list specific steps and concrete steps and are not too extensive.
Possible actions and options are explained.
The system partially accompanies the user during the dialogue by clarifying the possibilities of the system.The help functions are easy to find, focus on the user's task, usually list specific steps and are not too extensive.Possible actions and options are usually explained.
The system does not accompany the user during the dialogue.The system's options are not made clear.The help functions cannot be found, do not list specific steps or are too extensive.Possible actions and options are not explained.

Flexibility and efficiency of use
The system supports flexible interactions.Shortcuts and other shortcuts -invisible to new users -speed up operation for advanced users.Frequent actions can also be customised.
The system mostly supports flexible interactions.Shortcuts and other shortcuts -invisible to new users -speed up operation for advanced users.Frequent actions can also be customised.
There are no shortcuts or other shortcuts to make operation easier.

Aesthetic, minimalist, and engaging design
Dialogues do not contain any information that is irrelevant or rarely needed.The system supports short interactions.A more extensive conversation is possible if the user so wishes.
Dialogues do not contain any information that is irrelevant or rarely needed.The system supports short interactions.A more extensive conversation is sometimes possible if the user so wishes.

Context preservation
The context in relation to the topic of conversation is retained within the dialogue.The user has the option of referring to previous messages in further interactions.
The context in relation to the topic of conversation is retained within the dialogue.The user does not have the option of referring to previous messages in further interactions.
The context in relation to the topic of conversation is not retained within the dialogue.
The user has no option to refer to previous messages in further interactions.

Trustworthiness
The system conveys trustworthiness by ensuring the confidentiality of user data and being transparent and truthful to the user in this regard.The system does not falsely claim to be a human being.
The system explains to the user what happens to the data.The system does not falsely claim to be a human being.
The system does not convey trustworthiness.The use of user data is not transparent.The system claims to be human.
Table S3.Heuristics adapted from Langevin et al.

EVALUATION FRAMEWORK
In the following, the results for the complete evaluation framework are summarized.Judgments from study participants were aggregated in a way that number of votings "strongly agree" and "agree" were merged and number of votings "strongly disagree" and "disagree" were merged.All values related to the exploratory user study refer to n=36 participants.

Table S4 .
Detailed results from applying the evaluation framework