Leveraging Large Language Models for Realizing Truly Intelligent User Interfaces

1 INTRODUCTION

Scholarly knowledge organization becomes more important as the rate at which scholarly articles are being published keeps increasing [5]. A few approaches focus on organizing scholarly knowledge in a semantic manner using knowledge graphs. Knowledge graphs are defined as networks of entities, properties, and relationship between entities [6]. A common method to describe statements in knowledge graphs is formulating them using a subject, predicate, and object [13] in the Resource Description Framework (RDF) [7]. RDF adopts terminology from linguistics. The task of transforming scholarly knowledge from an unstructured textual format to a structured format is cumbersome, as it requires both data modeling skills and domain knowledge. The Open Research Knowledge Graph (ORKG) [1] aims to create such a knowledge graph containing knowledge presented in scholarly articles. The ORKG leverages crowdsourcing to perform the knowledge transformation process, by including article authors in this process [10]. The ORKG is a domain-agnostic knowledge graph, targeting a variety of scholarly domains. Due to the domain-agnostic nature, knowledge is not described by a predefined ontology, but is rather self-evolving based on domain requirements. Data models are crowdsourced by domain experts, which are materialized in ORKG Templates. Templates have a similar function as SHACL shapes [3] in traditional RDF knowledge graphs. Since users are able to create new ontology terms within the graph, system guidance is of importance to ensure a correct and complete structured article description. In light of the recent advancements of Natural Language Processing (NLP) and more specifically of Large Language Models (LLMs) [14], it becomes increasingly possible to use Artificial Intelligence (AI) to assist users in the transformation process. Particularly, ChatGPT gained significant attention across a plethora of domains, including health care [2], scientific publishing [4], and programming [15].

Figure 1:

View Figure

In this work, we propose an approach to integrating LLMs into existing User Interfaces (UIs). Implementing LLMs into existing UIs has several implications for example, the integration should be non-intrusive to ensure the existing workflows are not disrupted. Compared to creating completely new UIs using LLMs, the integration into existing systems entails additional challenges and constraints. We present an approach that provides best practices when implementing LLMs into existing UIs. To demonstrate the usefulness of this approach, we use the guidelines to implement LLM support into the ORKG UI. This approach resembles a neuro-symbolic hybrid approach, where the intelligence of neural models (LLM) is combined with semantic representations (knowledge graph) and human intelligence for validation and curation. The LLM is implemented in a UI component called Smart Suggestions. The Smart Suggestions provide guidance to users for a selected set of tasks that users frequently perform when interacting with the ORKG UI.

In this work, we discuss our experiences with integrating LLMs into UIs by presenting a use case for the scholarly knowledge organization domain. More specifically, we make the following contributions: 1) We present an approach comprising 22 guidelines for LLM integration into existing UIs. 2) We implement the approach in the ORKG UI by means of Smart Suggestions. 3) We evaluate the approach and implementation with a small-scale user study.

2 APPROACH AND IMPLEMENTATION

To introduce our approach, we first present guidelines for non-intrusive LLM integration into existing user interfaces. Afterwards, we explain how these guidelines are implemented in a scholarly knowledge infrastructure, by discussing use cases and implementation details.

3 EVALUATION

We conducted a user study to evaluate both the approach and implementation. Regarding the approach, our aim is to evaluate how relevant the individual guidelines are. For the implementation, we want to assess the usefulness of the Smart Suggestions when implemented in the UI. We will now discuss the evaluation in more detail.

3.2 Results

We present and discuss the results of the evaluation in three parts. Firstly, the approach evaluation. Secondly, the use case evaluation. Finally, the evaluation of the participants’ attitudes towards Smart Suggestions and ChatGPT.

3.2.1 Approach Evaluation.

The results regarding the participants’ opinions are displayed in Figure 2. As can be observed, on average the guidelines are considered to be important, and thus relevant, by the majority of the participants. Participants find guidelines 4.1, 4.2, and 6.1 to be most important. The former two guidelines are related to error management and ensuring that the system remains working. It is therefore expected that participants consider these guidelines to be important. Guideline 6.1 relates to the response time which should be short. Especially when the number of tokens increases (both for input and output), the response time can grow significantly. Therefore, it is crucial to keep the token sizes to a minimum to ensure the response time is within seconds. Participants find guideline 6.2 to be the least important. This can be partially explained by them being potential users of Smart Suggestions and therefore less concerned with the monetary costs that are associated with using LLMs.

Figure 2:

View Figure

Figure 3:

View Figure

3.2.2 Use Case Evaluation.

The use case evaluation results are displayed in Figure 3. The results are aggregated from all six use cases and the individual domain examples. In general, the perceived usefulness is high which exceeded our expectations. An interesting observation is that the perceived usefulness is higher than the correctness. This can be explained in light of our assumption that Smart Suggestions can provide inspiration to the participants even if the suggestions are not fully correct. This is also in accordance with the evaluation results from the attitudes evaluation.

3.2.3 Attitudes Evaluation.

Finally, the results of the participants’ attitudes evaluation are displayed in Figure 4. As question 1 indicates, the participants indeed consider themselves to be experienced with the ORKG, which was our target when recruiting participants. Overall, the results indicate that the participants’ attitudes are relatively positive and that they appreciate the Smart Suggestions, as indicated by questions 2 to 5. However, question 6 shows that participants think that Smart Suggestions are not able to replace human assistance while using the system. This could either mean that more Smart Suggestions are needed (also partially confirmed by question 3), or Smart Suggestions themselves are not sufficiently able to provide the same assistance as humans can provide. Questions 8 and 9 indicate that most of the participants are not using ChatGPT regularly in their daily lives, which means that participants were most likely less opinionated towards LLM usage in general. However, they do believe that ChatGPT can be successfully leveraged to organize science (question 10) and can be used effectively in the ORKG (question 11).

Figure 4:

View Figure

4 DISCUSSION

With the rise of LLMs, their presence in everyday tasks and applications is growing. However, in order to integrate LLMs into existing workflows, one has to be sure the LLM provides correct and helpful information. We applied our implementation to the scholarly knowledge organization domain, where data correctness and trustworthiness are key parts of the system. If the scholarly knowledge graph contains incorrect knowledge, it loses its potential to become a valuable asset for researchers. Therefore, integrating LLMs in this domain, with all their potential weaknesses, such as hallucinations or difficult-to-trace references, is challenging. We specifically address those weaknesses by deliberately making LLMs a non-vital part of this workflow. We decided to integrate the LLM in an assistive manner, by providing guidance to users. This largely mitigates the previously mentioned weaknesses of LLMs, as humans are verifying the correctness of the suggestions before any data is added to the knowledge graph.

Prompt Engineering Experiences and Challenges. Some tasks performed by LLMs are more suitable than others. A seemingly simple task of verifying whether a piece of text follows certain rules (e.g., not capitalized, short, reusable, etc.) turns out to be rather difficult for LLMs. Ideally, the response of an LLM would be a list of true/false values indicating whether a specific rule is met. In our experience, the responses of providing true/false results were highly variable and therefore too unreliable to be usable in a real-world environment. Sending the identical prompt multiple times resulted in opposite answers, even when the LLM was configured to reduce variations of responses (by setting the model temperature to zero). Also, asking for a percentage instead of a true/value value turned out to be too unreliable. Finally, we decided to remove the evaluation score altogether. The prompts were modified to ask for feedback instead of a scoring evaluation of an input. Even though answers from the LLM may still differ significantly for the same input, when open feedback is expected, this is acceptable, or even desirable. This enables users to get additional feedback every time a request is made. To conclude, tasks that provide specific structured evaluations are less suitable for LLMs than tasks that provide open-ended feedback to users. This is an important aspect of fostering user adoption and managing user expectation.

Another challenge related to non-conversation LLM usage is retrieving responses in a machine-readable format. Initially, we added sentences such as “return in JSON” to get the desired format. In our experience, this method was unreliable. Not only was the response frequently not clean JSON but mixed with text (e.g. the response: “Sure, I can do that. [JSON]”), also responses were often not parsing due to syntax errors. Instead, we used ChatGPT’s “function callings” which improved the reliability of the response parsing significantly. This enabled us to remove the logic from the microservice that performs fuzzy parsing of responses. Although this method also occasionally returns non-parsable formats, due to the error recovery options in the UI, this results in only a minor inconvenience for users. Finally, we further optimized our prompts using a trial and error method, ensuring the LLM was returning the desired response.

Generalizability. In our work, we used the scholarly domain as an example of a challenging application domain, but the approach is applicable outside this specific domain as well. The guidelines in Table 1 are presented in such a way that no assumptions are made about the domain. By presenting the guidelines as requirements, we facilitate adoption of the guidelines when integrating LLMs in existing UIs. The scholarly domain poses several challenging factors that are also applicable in other domains, such as correctness and trustworthiness of data. By presenting LLM responses as suggestions, and having manual verification of data by users, we propose a rather conservative approach for LLM usage in existing UIs. However, this conservative approach is easier to adopt in real-world systems, as LLM performance is less critical when LLMs are employed in a suggestive manner. We believe the guidelines offer a practical resource for the HCI community seeking to adopt the latest technologies without requiring significant changes in existing workflows and keeping the weaknesses of LLMs in mind. The use cases in which LLM supported can be integrated differs depending on the domain. The six use cases from our work are domain-specific. However, LLMs have already proven to be helpful in a plethora of other domains, including education, customer service, legal, etc. The integration of our approach for those domains is possible after domain-specific use cases are identified. Finally, the previously discussed LLM experiences are domain agnostic and provide insights for LLM integration in UIs in general.

Limitations and Future Work. The evaluation is conducted with a small number of participants, and it is therefore not possible to draw firm conclusions from the evaluation results. The relatively low participant number can be explained based on the selection criteria for our sample. The selected participants all had at least several months of experience using the ORKG and could therefore provide informed feedback about our approach. We deliberately excluded ORKG system developers from our study to prevent bias towards the platform. Additionally, we did not include inexperienced users to prevent bias caused by lack of experience. Despite the small number of participants, we consider the evaluation results to be relevant as it provides early feedback to further guide this research its development. We plan to continue the evaluation of our approach in a real-world setting. In this setting, actual system users interact with the Smart Suggestions and are thus better able assess the correctness and usefulness of the approach. Also, we hope to collect feedback to guide further prompt engineering efforts and to identify additional use cases where Smart Suggestions can be employed.

In our implementation, we only considered a single LLM as detailed in section 2.4. LLMs are varying widely, especially when fine-tuning LLMs (i.e., training them for a specific task). Therefore, some of the previously mentioned experiences might be less applicable to models other than GPT3.5-Turbo. Keeping this in mind, we tried to focus on experiences that are relevant to LLM usage in general. In future work, we plan to evaluate different open source models that are pre-trained to perform specific use case tasks. In addition to the open source benefit (i.e., transparency), less resources (i.e., computational and monetary) are consumed by smaller models.

5 CONCLUSION

In this work, we presented an approach that provides guidelines for non-intrusive integration of Large Language Models (LLMs) into existing User Interfaces (UIs). These guidelines present LLM integration as a support system for users. Some of the key elements of the guidelines include optional and on-demand usage, transparency, and distinguishability. Mainly due to these guidelines, it becomes possible to integrate LLMs into systems, even if the results of the LLMs are not always accurate. We used the guidelines to implement Smart Suggestions in the scholarly knowledge graph UI. The Smart Suggestions are designed to support users in knowledge graph curation and population. Afterwards, we conducted an evaluation, consisting of expert users, that evaluated both the approach and implementation. The evaluation results indicated that the participants were positive about the usefulness of the presented use cases. Furthermore, the participants confirmed that LLM support can be helpful, even if the suggestions are not fully accurate. In future work, we will continue with the development of Smart Suggestions, extending the implemented use cases and performing larger-scale evaluations based on real-world usage.

ACKNOWLEDGMENTS

This work was co-funded by the European Research Council for the project ScienceGRAPH (ID: 819536), by NFDI4DataScience (ID: 460234259), and by the TIB Leibniz Information Centre for Science and Technology. We want to thank the entire ORKG team for their contributions to the ORKG platform, including research and development efforts.