SimUser: Generating Usability Feedback by Simulating Various Users Interacting with Mobile Applications

2.1 Usability Feedback in Prototyping

In the prototyping stage, designers lean on the internal team or experts to acquire usability feedback, using informal techniques such as cognitive walkthroughs [44] and heuristic evaluations [65]. These methods examine the interface and judge its compliance with recognized usability principles [62]. These approaches necessitate the involvement of experienced experts, otherwise, the results may be inadequate and biased [65].

Recent research usually obtains effective usability feedback during user flow, which is a set of interactions that describe the typical set of steps needed to accomplish tasks [2, 32]. However, they focus on collecting users’ feedback toward current interfaces, yet frequently overlook the potential impact of users’ past experiences and expectations on usability feedback [96]. Researchers suggest expectation disconfirmation, which points out that people’s expectations are satisfied or not, will influence users’ feedback on usability and user experience [58, 68, 89]. Besides, it is important to consider specified user characteristics by creating user personas, which need to be described in detail and supplemented with factors relevant to the test application.

Using LLMs to simulate the above process involves two main challenges. One is to empathize with specified users and portray their characteristics. Another is to simulate how human users perceive and interact with the mobile application in the user flow. Next, we will discuss existing research that provides us with insights to address these two challenges.

2.2 Act Like a Certain User Using LLM

LLMs offer a new approach for inferring and mimicking human characteristics from textual descriptions. [37] Some studies have taught LLMs to act as a specified role with the prompts of the form "You are an expert of...", "Act as a...", etc. [74, 92]. However, LLMs can only represent a general human group. Recent studies also believe that there is a potential danger of systemic discrimination produced in the training process of LLMs. To help LLMs reduce toxicity in persona generation [17], more user data from online analytics and web technologies which liberate static personas via interactive user interfaces are utilized to improve LLMs’ ability of in-context learning [9]. Notably, opinions from stakeholders and using the most relevant opinions from individual users enable LLMs to simulate more accurately [13, 25, 56, 95]. However, for usability assessment, being merely "human-like" is insufficient. LLMs can mimic the language style of specified individuals, but simulating human cognition and behaviors remains an area for further exploration. It needs to be extended to how humans connect with their task and surrounding environment [94].

Current user modeling research provides methods to assist LLM in representing user characteristics during simulation. Nolte et al. proposed a modification of personas for hearing-impaired users by defining them in terms of ability ranges [66]. Casas et al. detailed user personas into user levels toward the system usage, interaction methods, and acceptance of audio and visual displays [10] to parameterize users and allow the system to better respond to user characteristics. Inspired by such methods, we think supplementing personas with detailed and parameterized descriptions of ability and task-related characteristics may help LLMs exhibit the characteristics of target users.

2.3 Methods to Simulate User Flow

The interaction between users and the application is another critical component. Simulated users must be capable of understanding and interacting with the interface for assessment of mobile application usability through user flow simulation.

LLMs can process textual data but cannot directly interact with GUI. Although the latest multimodal LLMs begin to process image information [49, 88], they still fall short of accurately understanding interfaces. To bridge this gap, Wang et al. contributed an algorithm that used depth-first search traversal to convert mobile UI’s view hierarchy [86] into HTML syntax. This study emphasized that LLMs require external models like SalGAN and UEyes to understand multimodal mobile applications containing text, image, and structural information [28, 70]. Transforming the combination of code files and visual models into natural language descriptions has the potential to enhance LLMs’ ability to interact with interfaces.

Automated UI testing methods measure the interaction results of interface feasibility, yet they do not focus on the interaction process and user experience. Previous methods have already been able to manipulate GUI elements like human users [22, 81, 86], but they usually require a lot of human data, step-by-step instructions, or designed task-specified reward functions for task completion [24, 46]. LLMs bring new chances with their language comprehension and inferring abilities. Liu et al. extended the research on LLM into generating semantic input text according to the GUI context based on LLM’s outstanding progress in text generation [50]. In their subsequent work, they asked the LLM to play the role of a human tester and chat with mobile apps to identify issues of usability [51]. Their LLM tool held the capability to execute more complex operations by extracting the static context of the GUI page and the dynamic context of the iterative testing process. Beyond interfaces, the mobile applications embody the logic associated with user tasks [45]. Obtaining usability feedback from aspects such as the application’s framework and interaction logic, rather than just individual interfaces, also requires exploration.

LLM-based agent framework offers chances to conduct usability tests within the context of the interactive user flow. It assigns intelligent agents with different capabilities, which can humanize interactive and emotional experiences, enhancing realism in simulating social scenarios [3, 77, 90]. Meanwhile, chain-of-thought (CoT) improves LLMs’ accuracy of inferring by zero-shot prompts [36, 91]. By achieving such a framework and structure, we think it is feasible to simulate interactions between users and mobile applications while reflecting the characteristics of users and the impact of scenarios.

3.1 Usability Evaluation Practices in Prototyping Stage

Personas require supplements. Personas are vital for practitioners to define target user groups. However, it is necessary to supplement user personas with more information. Practitioners customize user persona traits specified to the function of the prototype under test. Occasionally, they also incorporate task-related information and stakeholders’ insights to form a more comprehensive description.

User feedback is expected but absent in design practice. Practitioners recognize the importance of user feedback, yet involving users during prototyping proves challenging. Designers often substitute direct user involvement with internal estimations based on their own experience and empathy, which may lead to inaccurate judgments.

3.2 Expectations and Concerns toward LLM Agent

LLM should generate a wider range of contextual usability feedback in a quicker and lower-cost way. Promisingly, practitioners point out that one advantage of our LLM agent is its ability to quickly and cost-effectively simulate the feelings of target users. Practitioners believe they can have the opportunity to explore feedback from a wider variety of user groups. Compared to AI methods that only predict usability issues, they prefer the LLM approach that can infer user thoughts and underlying reasons. Besides, they think our demo broadened their perspectives on users by offering a new way to expand scenarios by imagining user flow in main and extreme scenarios.

LLM should infer characteristic and reasonable requirements of specified users. Although they believe our approach is inspiring, this credibility depends on reasonably reflecting user characteristics. They mention that the existing digital personas provide a solid foundation for the LLM agent to understand users’ characteristics. Based on that, they hope the LLM agent could identify users’ insights with characteristics that are distinctly related to the mobile applications, thereby enriching the diversity and completeness of the user group.

Their concerns mainly concentrate on the ability of LLM. First, they think it is very difficult for LLM to "see" or understand interfaces and interactions. They agree that using codes like HTML/CSS may enable LLM to know the functions or layouts superficially. Still, its process of understanding and interacting with the application should mimic human user perception patterns. Besides, the tactile and auditory feedback of hardware should also be felt by the LLM agent. To summarize, the key to making LLM truly understand interfaces and interactions lies in bridging the gap between language models and multimodal information such as visual, tactile, and auditory cues.

Second, practitioners are uncertain whether the LLM agent can provide reasonable usability feedback through simulating interactions in the user flow. It requires further empirical study for validation. They advise that feedback output should be described from the users’ perspective and summarized into heuristic insights. For instance, responses like "I hope I can contact parents through this interface" are more human-like than "I wish there is a ’contact parents’ button". In addition, the LLM agent should present the specific factors that lead to this usability feedback to the designer.

Figure 1:

View Figure

4.1 Interactive Process Between Mobile Application Agent and User Agent

During the simulation, MA and UA collaborate to simulate an authentic user flow, as visualized in Figure 2. The simulation starts when designers upload the prototype, user personas, and tasks. The prompt structure is shown in Figure 3 and detailed in the supplemental material.

Figure 2:

View Figure

Figure 3:

View Figure

Step 1. MA will generate interface descriptions in accordance with the high-fidelity application uploaded by the designer in advance. Meanwhile, UA will refine user characteristics based on the user personas to specified factors and expand their usage scenarios accordingly.

Step 2. Next, UA will simulate user flow by building expectations of the upcoming interface. Then, MA describes the interface information to UA. UA will then determine whether this page is available to continue the task. If the answer is negative, UA returns to the previous page and do operations again; if the answer is positive, UA continues finishing the following steps. When UA thinks that the task has been accomplished, it quits the user flow simulation after steps on the current interface.

Step 3. UA will try to continue the task. After that, UA’s operations are transmitted to MA, and MA will determine whether the operation is successful and provide potential visual feedback on the interface or haptic and auditory feedback from the hardware for UA.

Step 4. UA needs to compare its expectations with the usability categories mentioned above, providing its thoughts according to the expectation disconfirmation.

Step 5. Since UA has gone through the whole process of the task, UA will evaluate the application’s interaction logic and framework as a supplement to the usability feedback.

Throughout the process, UA should consistently play the role of the target user specified by the designer, immersing themselves in the extended usage scenarios. In each interface, UA must make three evaluations, expressing its thoughts by the "think-aloud" method. An overall assessment of the application will also be provided at the end of the simulation. When providing user’s thoughts, reasons should be given based on user-related contexts. The details of each section are described in the following.

4.2 Input from Designers

Information of the application and mobile device. Designers upload the codes along with images of each interface to SimUser for MA. Additionally, designers should define the interaction method and feedback method of the mobile device.

Information of the user personas and tasks. The processed user personas from interviews and questionnaires should be input as reference materials for SimUser to learn the characteristics of certain user groups. User personas must include ability and task-related characteristics. Additionally, designers need to define tasks that users are supposed to complete within the mobile application.

4.3 Mobile Application Agent

4.3.1 Interface description generator.

MA will generate natural language descriptions of each interface, including positions, contents, sizes, color, contrast, and visual saliency of widgets and texts based on the code files. To organize interface information, MA integrates the code files of the interface and outputs a new structured text file, encoding essential layout elements and structure while adding global style. Besides, MA will extract the absolute positions of widgets, and combine them with the image to segment the interface. The segmented interface will be processed by a visual model to calculate the visual saliency of each widget. The visual saliency will also be included in the text file as a supplement to the definitions. The most important thing is MA has to describe the interface objectively. For instance, the description should not be “icon of heart rate” but “icon looks like a heart” and evaluative adjectives are forbidden. The example is shown in Figure 4.

Figure 4:

View Figure

Figure 5:

View Figure

4.4 User Agent

User Agent (UA) serves to simulate users in the interactive user flow. The simulation method is shown in Figure 6 and described in the following:

Figure 6:

View Figure

4.4.2 User flow simulator.

Generate user expectations. We employ zero-shot prompts to assist UA in inferring user expectations for the designer’s mobile application. Compared to human users, LLM agents lack prior experience, leading to their lack of a basis for assessing usability. Therefore, enabling them to develop expectations for upcoming pages and interactions in advance may help them become more critical. The expectations generated by UA should reflect the influences of the target user and scenario. To be closer to reality, UA predicts the next interface based on its operations on the current one. UA envisions the upcoming interface from the perspective of usability categories in Table 1 before MA describes the interfaces. It is worth noting that UA should provide reasons for each user expectation it generates. These reasons must be traced back to the fundamental aspects, such as user characteristics, scenarios, device information, and so on.

Perceive and comprehend the interfaces. In our earlier demos, we found that LLMs would identify task-related elements directly from the interface, bypassing the perception process typical of human users. Therefore, we prompt SimUser to perceive the interfaces in a human-like way. First, UA gains a broad understanding of the interface layout and the relative positions of widgets and text. In this case, UA then searches for the target widget which may help it continue completing the task. Also, UA possesses decision-making capabilities. If the current interface is inappropriate, UA will return to the previous page and operate again. Otherwise, it interacts with the identified widget to proceed to the next interface or concludes the user flow simulation if it is the end.

Interact with the mobile application. When UA believes there is a widget that may be possible to enter the right page, it will choose one operation from click, slide, long press, and so on to interact with it and trigger its function. After operating, UA will receive corresponding feedback from the mobile application. UA may misoperate on the interface and will receive no feedback or be navigated to the unwanted interface.

Evaluate the single interface and corresponding interaction. After completing the perception and operation steps on a single interface, UA should compare the actual interface with user’s expectations, and determine the feelings these differences bring to him. UA objectively identifies specific differences and then explains his feelings. The five factors of usability categories listed in Table 1 should be answered one by one. To obtain interactive usability feedback, we set up various principles and standards based on different stages of tasks. Principles include how to imitate a characteristic user and how to perceive and operate on interfaces, while the standards consist of SUS [8], NASA-TLX [21], and UEQ [39] which instruct SimUser to assess usability. Detailed information on the transformation methods, principles, and standards are listed in the supplement material.

Figure 7:

View Figure

4.5 Implementation

SimUser is implemented using the following tools. We export interface images and use Locofy [53] to convert Figma files to HTML/CSS files. HTML files include information on interfaces and their interaction logic, while CSS files describe the features and layout. Other forms of code, such as React, are also feasible, as long as they include this information. After that, We logically divide functional areas based on the interface layout and obtain the absolute positions of key widgets. In conjunction with this, SalGAN [70] is employed for measuring the visual saliency proportion of widgets.

The web tool for SimUser is developed in Python and uses the Gradio library for rapid GUI construction. Designers can input test tasks and user reference documents. SimUser displays prototype previews along with three sets of resettable Users and their corresponding Scenarios. When SimUser successfully processes the structured files, three types of images for the current page (original, segmented, and saliency grayscale), as well as the natural language description will be presented. To save tokens of GPT-4 API in the user flow simulation, we deploy prompt engineering methods to remind contextual information as described above. After SimUser completes the simulation, designers can receive SimUser’s usability feedback for the current interface with the user’s perceptions and behaviors during the process. The usage method of the web tool is illustrated in Figure 7.

5.1 Apparatus: test prototypes and the mobile device

We invited design graduate students to create two simple prototypes of the same sports mobile application for the smartwatch in Figma which only included basic functions. The two prototypes (A and B) only differ in usability concerning interaction logic, interface layout, and interaction feedback, as shown in Figure 8.

In our studies, we chose Apple Watch Series 8 as the test device. Not only is it a prevalent device, but it also encompasses several typical screen interaction methods. Additionally, users can interact with physical buttons on the side of the Apple Watch. We incorporated these hardware interactions as complements to the screen-based interactions that both human users and SimUser could utilize during our testing.

The experiment was conducted in a university’s design studio. One researcher controlled the prototype interface on the watch via a laptop, while another researcher used a laptop to document participants’ feedback, with the entire session being audio recorded. The whole process of user testing was conducted in Mandarin, and transcribed by the transcription service of Tencent after removing identifiable information.

5.2 Participants

To compare and verify the effectiveness of SimUser, user tests were conducted with two groups of participants with different user types. We invited 24 university students and 24 elderly individuals with an average age of 22.58 (SD = 2.53) and 60.79 (SD = 8.87). All participants were recruited from social media and the average user test lasted 45 minutes, and participants received 50 CNY as payment for their devotion.

As the SimUser web tool was developed, we invited 21 design practitioners with average ages of 23.5 (SD = 2.25) and average experience of 5.43 years (SD = 2.06). The recruitment method was the same as the formative study.

5.3 Procedure for collecting usability feedback from human users

In human user groups, all participants were asked to fill out questionnaires about their background and demographics and perform user testing on prototypes A and B separately. In the beginning, we gave participants time to familiarize themselves with the application. Then, we introduced the test task that they should record a running exercise on the test applications and imagine themselves in their most common exercise scenarios. Afterward, the experimenter reviewed each interface with participants, retracing their experience and guiding them to identify usability issues of the interfaces. Besides, participants would state how they perceive the interface and find the target widget. In the end, we interviewed them to conclude their experience and also asked them to contemplate potential extended scenarios. The order of the two test prototypes has been counterbalanced.

Figure 9:

View Figure

5.4 Procedure for collecting usability feedback from SimUser

With the consent of the participants, we collected their information according to the user characteristics SimUser requires and processed it into the SimUser’s reference document. SimUser analyzes these data and generates deeper user characteristics, as mentioned in section 4.4.

To determine how many rounds of SimUser feedback should be collected in the study, a pre-test was conducted with different rounds of SimUser. We had SimUser simulate 10 rounds of results and compiled the generated usability feedback (the analytical method is detailed in section 6). We observed that randomly sampling five rounds of results could cover over 90% of the outcomes from ten rounds, which meant 10 rounds were sufficient. In this case, we employed SimUser to generate 10 rounds of usability feedback as elderly individuals and university students for prototypes A and B respectively (40 rounds in total). The output over 10 rounds was collected and then compared with the results from real users.

5.5 Procedure for gathering designers’ suggestions

We recorded a demonstration video, enabling these designers to remotely experience the tool after we introduced how to use SimUser, explained the input materials, and presented the generated usability feedback to them. In the following, they were asked to fill out a System Usability Scale (SUS) questionnaire [8] and conduct brief semi-structured interviews to gain their opinions. The results employed the same coding and analysis methods as the formative research.

6.1 Usage scenarios

SimUser covered over 70% of usage scenarios of both the elderly and university students, as shown in Table 3, and performed better in empathizing with university students. For example, in general scenarios, outdoor locations like "park" and "lakefront" were most frequently mentioned by simulated elderly individuals, while simulated university students often mentioned "campus playground" and "gym". Specifically, SimUser provided many special scenarios not mentioned by human users, such as running in an unfamiliar city while traveling or running with a dog.

6.2 Usability Feedback from SimUser and Human User

2,875 labels are collected from human users, and the overall coverage of SimUser is 80.0% (n = 2,299). In usability data comprising 7 categories and 2 user groups, totaling 14 data groups, 6 groups of SimUser cover over 80% of human user feedback and the other six groups have coverage of around 70%. However, 2 groups of SimUser’s coverage, the students’ views on operation method and interaction feedback, fell below 50%. The coverage rate is shown in Table 4, and we will demonstrate some interesting results behind the data.

6.3 Differences in SimUser and Human Data

SimUser provided extra information not mentioned by human users, accounting for 45.5% of all the feedback it offered, which was not counted in the coverage. Part of the feedback was considered inspiring. For instance, SimUser suggested "I have high blood pressure and I need emergency assistance", which were reasonable considerations but absent in the human data of 24 elderly individuals. However, the excessive amount of information increases the difficulty of collating and filtering, potentially obscuring valuable feedback.

6.4 Feedback of SimUser Web Tool from Design Practitioners

The web tool’s average score of the processed SUS data was 66.22 (SD = 7.55), which was close to the average performance of 68. Unlike other AI-driven design assistance tools with a high level of complexity, our web tool was easy to use as it aligns more closely with a ’designerly understanding’ [48] attributed to its intuitively explained functionalities. Designers thought functions in the tool were well-integrated and easy to use. Nonetheless, they experienced difficulty in assimilating extensive textual outputs, indicating that enhancements through visualization or integrated report generation could markedly enhance the tool’s presentation. Also, a modular approach to segregate the functionalities of the web tool was recommended for improving usability. Besides, encapsulating the reasoning process of CoT in the backend resulted in designers losing control over the process, affecting explainability and their trust in the outcomes. Some designers indicated that they would like to control intermediate steps to adjust the output.

These designers all acknowledged the value of using SimUser to generate diverse usability feedback, which was innovative and worth trying. Designers also proposed many intriguing ideas. In addition to traditional user research data, they believe that commercial data, astrological signs, and MBTI personality types could also serve as user persona inputs. Two of them even suggested that reverse-clustering users from usability feedback might iterate a more accurate categorization of user groups.

7.1 Enable LLM to Generate Usability Feedback of Mobile Application Prototypes

Designers often find it challenging to accurately predict what information users expect to receive, or how specified usage scenarios may influence the interaction. To support user-centered design, contextual usability feedback with user characteristics is becoming a trend in design practices and research [67].

After 10 rounds of feedback generated by SimUser, it could cover most of the usability issues with a similarity of 80% raised by 24 human users. SimUser reached high coverage rates in the perception processes and interactive behavior of human users, and it showed the potential to generate usability feedback in categories of information and functions, interface layout, and interaction logic. Unfortunately, SimUser faced challenges in fully capturing the nuances of real human data, primarily due to its limitations in replicating human aesthetic sensibilities, experiential knowledge, and grasp of emerging trends. Such constraints resulted in certain aspects of usability feedback that SimUser was currently unable to generate effectively. Meanwhile, the coverage indicated the similarity to human users, which does not completely reflect the output of SimUser since a lot of additional usability feedback that human participants overlooked was produced. These extra problems augment the amount of usability feedback from small sample user tests but might obscure the real usability issues as clutter also increases. Some of them may be reasonable and enlightening, but designers can only use them as heuristics inspiration, not as replacements for human results.

We find it somewhat difficult to directly identify issues within individual prototype designs without clear standards and references. Compared to human users, GPT-4 exhibits greater ’tolerance’, being willing to experience a deeper interface depth. Similarly, if only describe interface information to GPT-4 and ask it to express its feelings, it is difficult to pinpoint specific usability issues. Instead, it may unreasonably consider interface design as satisfying. We introduce the concept of expectation disconfirmation to assist SimUser in identifying issues through comparison while making it more ’critical’ through prompts. Additionally, we educate SimUser on how design factors of mobile applications impact their usability through prompts. As a result, SimUser becomes more stringent and better equipped to generate reasonable usability feedback.

In our work, we adopt two LLM models to simulate the interaction between users and mobile applications. We divide the two agents (MA & UA) into separate modules and conduct in-module inferring using the CoT approach. As tasks are decomposed into steps and we ask SimUser to record the results of every step, SimUser can create users with detailed characteristics and generate accurate interface descriptions. The rate of hallucination in SimUser’s inferring decreases, echoing the results of previous studies related to CoT [20]. The two agents complete their tasks respectively and show the potential to collaborate interactively when we establish the sequence and rules of tasks. Moreover, SimUser performs well in the interface layout feedback which relies on visual observation. We suppose that the visual saliency model predicting attention allocation on interfaces helps SimUser to understand visual information.

7.2 Reflect User Characteristics and Contextual Influence in LLM-Generated Usability Feedback

Besides simulating usage scenarios and conversations between an imaginary prototype and fictional users [35], SimUser can understand user characteristics and generate similar usage scenarios with human users. For example, the most frequent exercise scenario for the elderly is the park while for students is the playground. More promisingly, SimUser is more capable of inferring extreme scenarios. Human users find it difficult to anticipate unexpected situations that might occur during running, but SimUser can generate such scenarios, like a potential fall for the elderly or sudden social needs for students.

SimUser effectively reflects the ability characteristics, which is particularly obvious for elderly people who have noticeable limitations in their capabilities. Our results indicate that by leveraging the inferential capabilities of LLMs and the user research materials we provide, it is possible to manifest the characteristics of user groups through in-context learning without extra training [74, 83]. Besides, we draw upon theories of user modeling to add factors, that are more relevant to the applications and tasks to be tested. User modeling helps SimUser to capture the broad characteristics of users more accurately, thereby better understanding the capabilities, preferences, and attitudes of this user group that influence usability feedback. However, SimUser cannot effectively represent task-related characteristics, especially those related to experience and proficiency. Optimizing these factors may further improve the alignment between SimUser and human users.

The generated usability feedback presents the contextual influence of the physical environment and hardware. Recent AI-driven prototype evaluation tools hardly consider the context where users employ the mobile applications, but SimUser can imagine how the usage scenarios and hardware will impact usability feedback. For instance, some simulated users interacted with the mobile application by pushing the side buttons on the smartwatch and a simulated student reported that he felt auditory feedback was inefficient because the gym was very noisy.

Last but not least, it is challenging for some human users to imagine possible edge scenarios due to their lack of experience and express their expectations towards interfaces, which are both easy for SimUser to infer. Existing user research techniques still have a gap between researchers and users. For example, the elderly may not be familiar with these technologies, and there may be a generational divide in communication with younger researchers [23]. SimUser creates a channel for barrier-free communication between researchers and simulated users.

7.3 Implications for LLM-based Usability Feedback in Design Practice

Our formative study indicates that although designers want usability feedback on prototypes, it is hard for them to acquire user insights in design practice. As discussed in the first part, we achieve practitioners’ expectations that LLM tools like SimUser can help quickly diverge into various target user groups and reflect the influence of user characteristics and scenarios in usability feedback. We address the concern about how LLM understands mobile applications by generating natural language descriptions from interface code files and combining them with a visual model. For another concern about ensuring LLM infers reasonable usability feedback, we initially assist LLM in simulating user flow steps in the form of CoTs, thereby enhancing accuracy by outputting these steps. Additionally, we establish a set of principles through prompt engineering to guide LLM in reporting experiences and tracing their reasons in a user-like manner. After their concerns are addressed, they believe that we have not only essentially met the need for recognizing usability feedback issues during the prototype iteration, but also provided more insights like how usage scenarios influence usability.

Even though LLM agents are capable of generating a variety of outcomes, their hallucinations remain a concern, and designers should treat the results as heuristic insights. The LLM agent should aim to broaden designers’ insights of users, serving as an auxiliary tool for discovering user requirements as it can not replace human user research. The key is to help designers rapidly expand target users and scenarios to obtain as many usability insights as possible.

We think it is beneficial to extend LLM-based user simulation tools to other fields of design. The generation and inferring ability of LLMs may be better applied in design since the design is an ill-structured field [30]. Like other AI-driven methods, LLM-based methods have the advantages of rapid generation and low cost, especially providing designers with an ’ever-lasting’ user, which can be regarded as a supplement for human users. In design practice, experienced experts and target users are sometimes hard to recruit, but AI is always available. Some research has already used LLMs to generate user interviews [71]. After mobile applications are launched, it is also potential to use LLMs to infer users’ thoughts and feelings behind online data. It is even possibly meaningful to employ simulated users in design education for students to test their prototypes and enhance their user research skills quickly.

7.4 Limitations and Future Work

Our empirical study was conducted with a limited sample of two groups of participants and several rounds of LLM tests. The interfaces used in the study were not derived from real design practices, but rather artificially designed to simulate common issues found in mobile applications. Although we helped SimUser remember its experience, there was still hallucination in SimUser like other LLM tools [4]. Besides, we have explored the possibility of LLM generating usability feedback, but the web tool has not been fully tested [41], which requires feedback in long-term practice.

How to more effectively establish a connection between interface visual attributes and usability for SimUser requires further exploration. We think more models like activity exploration [52] and visual complexity [14] will further enhance SimUser’s ability to extend usability feedback such as aesthetic evaluations of interfaces [59].

Currently, we have only completed testing on the simple sports application of smartwatches, but the descriptive method we propose covers the basic components of mobile applications, including visual information, functions, layout, and interaction methods. The compatibility is reflected in the fact that the basic components of more complex interfaces, such as smartphones and in-car interfaces, are similar or even the same [12], only that the amount of information increases. This becomes more promising with the enhancement of LLMs’ capabilities.

There remains a more in-depth exploration of the impact of context. We have not yet instructed SimUser on the rules governing how contextual factors influence usability feedback, similar to how interface factors do. Furthermore, we have only considered how the current interface influences users’ expectations for the next one. How the dynamically accumulated factors like emotions affect usability feedback throughout the user flow is also interesting for future study [1, 75].