Benchmarking usability of patient portals in Estonia, Finland, Norway, and Sweden

Introduction: Poor usability is a barrier to widespread adoption of electronic health records (EHR). Providing good usability is especially challenging in the health care context, as there is a wide variety of patient users


Introduction
Poor usability is a continuous barrier to widespread adoption of electronic health records (EHR) [1][2][3][4][5][6].According to the ISO definition [7], usability is "the extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use".Thus, usefulness, ease of use, and satisfaction can be seen as the main dimensions of usability.Usefulness and ease of use are also the variables of the Technology Acceptance Model (TAM) [8] that is widely used to assess intentions to use health information technology [9].The connection between usability and intention to use has shown using different usability metrics [10].Usability evaluation can thus identify problems that users have and help in developing the system quality and improving user acceptance [11].
Providing good usability is especially challenging in the health care context, as there is a wide variety of patient users with different health statuses, skills, beliefs, and contexts.A part of the EHR usability is the quality of health information.Thus, patients' perception of usability depends also on health professionals and the healthcare system; unavailable, delayed, unclear, or incorrect health records are often seen negatively [12].Broekhuis et al. [10] point out that health information is sensitive and does not always support user satisfaction when the user receives unpleasant health advice or news of their health.
Despite its importance, usability is notoriously challenging to improve.For example, Kaipio et al. [2,13] found in their survey study that physicians' experiences of usability of their EHR systems did not improve between the years 2010, 2014, and 2017 in Finland.The result is particularly disappointing as the study included over 30 EHR systems, many of which are international brands.
Usability benchmarking is an approach supporting the improvement of usability by evaluating and comparing strengths and weaknesses of systems.Unless we measure usability, it is difficult to procure new systems and the software providers may not pay enough attention to developing the existing ones.Benchmarking indicates whether a system's usability is acceptable, enabling comparison of competitors or new and old versions of the same system [14].
Surveys, particularly the System Usability Scale (SUS), are the most common methods for evaluating usability [15].Surveys provide subjective evaluations of usability from large sample sizes.SUS is composed of ten survey statements that respondents evaluate using a five-point scale from strongly disagree to strongly agree and it provides one score from 0 to 100 [16].Based on extensive testing, an acceptable score level is set to 70 [17,18].
Broekhuis et al. [10] compared usability benchmarking methods in the eHealth context.They suggested that SUS is inadequate as a standalone usability benchmark for eHealth as SUS scores do not accurately reflect the task performance of the users in usability tests.Indeed, it is well known that subjective measures such as SUS and objective measures such as task completion times in usability tests are not necessarily related [19].Unlike SUS, usability testing effectively identifies concrete usability problems [10,20].However, the number of participants is low [15], the typical duration of users' interactions is short, and it is also relevant to see how user satisfaction develops over time when users learn to use the system and see its benefits and weaknesses in real life [21][22][23][24].Furthermore, subjective measures are important as they are related to users' willingness to adopt the system [9].
In this study, the aim is to test a survey approach for benchmarking the usability of national patient portals offering patient access to EHR in Estonia, Finland, Norway, and Sweden as part of the NORDeHEALTH research project [25].The Nordic countries have a long experience, over several years, being pioneers in developing eHealth services such as national patient portals.There has been a patient portal in Estonia since 2008 [26], Finland since 2010 [27], Norway since 2011 [28], and Sweden since 2012 [29].
In our survey of 29,334 Nordic patients [30], we collected both patients' quantitative ratings of patient portal usability, and their qualitative very positive and very negative peak experiences of the portal use.In analyzing the results, we proceed from the within-case analysis to cross-case analysis [31].
In the within-case analysis of the Finnish responses, we found that the system usability score (SUS) provided a comparable usability rating, and the reported experiences revealed a wide array of usability issues.Reporting very positive and very negative experiences explained 23% of the variation in perceived usability suggesting that these emotionally strong peak experiences are relevant when patients evaluate usability [32].
In this study, we conduct a cross-case analysis of the four countries to examine whether the results are comparable across different countries and patient portals.We aim to benchmark usability and learn from the qualitative comparison across countries.We expect that the survey approach allows us to identify both usability challenges and good features of the patient portals in long-term established use by a wide variety of patients.

Methods
An online survey was used to collect patients' experiences with the patient portals in Estonia, Finland, Norway, and Sweden.The respondents were invited when they had accessed the national patient portals at the start of 2022.The survey link was available after logging into the patient record part in Sweden and Norway.All in all, 29,334 patients completed the survey.The survey template consisted of 45 questions.This study focuses on the questions on usability and very positive and very negative experiences on patient portal use.The mixedmethods approach was used to cross-validate the numeric usability evaluations with qualitative ones [33].The overview of the survey, data collection, respondents, and the healthcare systems of Estonia, Finland, Norway, and Sweden are described by Hägglund et al. [30].

National patient portals in Estonia, Finland, Norway, and Sweden
Table 1 presents the main functions available in the countries.The Estonian and Finnish patient portals were implemented nationally from the start.In Sweden, the regions started to use the patient portal at their own pace and had different functions available.The patient portal design was updated two months before the survey.In Norway, three out of four regions offered digital access to hospital EHR and had different practices on providing access to the mental health notes.

Variables
The quantitative variables were perceived usability measured with a 3item version of the Usability Metric for User Experience (UMUX) [34], and reporting a very positive experience and a very negative experience with the patient portal.In Norway and Sweden, the questions referred to the part of the patient portal that provides access to EHR (Patientjournal and Journalen).UMUX was selected as two of its items provide a reliable and equivalent SUS score approximation [35] while it does not burden respondents with many questions.In addition, the ten-item SUS may not be suitable to patient groups with physical or cognitive impairments [10].After respondents were asked to evaluate usability of the patient portal with the scale, they were asked whether they had or had not a very negative experience or a very positive experience with the portal.For those who responded 'yes', qualitative data were also collected by open-ended questions asking them to describe their experience in detail to understand the reasons for the usability evaluations.

Statistical analysis
Descriptive statistics were calculated for quantitative data.The SUS scores for each country were transformed from two of the UMUX items using a corrective regression formula [36].Regression models were fitted to study the associations of reporting very positive, reporting very negative experience, and perceived usability in each country.Variance inflation factors for each regression model were calculated to test for multicollinearity [37].The Cronbach α for perceived usability was 0.76, indicating acceptable internal consistency.Statistical analyses were performed using IBM SPSS Statistics (version 29.0.0.0 (2 4 1)) and Stata 18.0.

Qualitative analysis
Inductive content analysis [38] was performed on responses to openended questions using the ATLAS.tisoftware (version 22.2.5.0:ATLAS.tiScientific Software development GmbH) and Excel (2022; Microsoft Corp), and QCAmap [39] in Sweden.In Estonia and Finland, all data collected were analyzed.Due to the large data set and limited resources available, in Sweden and Norway, a sample of 1000 very positive and 1000 very negative experiences were analyzed.The experience narratives to be analyzed were selected randomly from the data set to ensure their representativeness.
The second author started the analysis work and prepared instructions for other countries to keep the analysis consistent from a country to another.After that, the analysis was independently conducted in each country by local researchers in local languages.In Sweden, one researcher analyzed very positive experiences and another researcher very negative experiences.In other countries, one researcher analyzed all the data.In-vivo-coding [40] was used to focus on users' perspectives instead of researchers' interpretation of the codes.The number of respondents included in each code was calculated.
After the analysis in each country, the results were reviewed together with all analyzers and the first author.Each researcher reported their codes and researchers from other countries checked whether they had similar codes.The team created common English labels both for the codes across countries that we agreed to have similar meaning and country-specific codes.The large set of preliminary codes were merged into a smaller number of codes with similar meaning and were then grouped under higher level themes.

SUS scores
The Finnish patient portal received the highest System usability scale (SUS) value, 74.3, corresponding to a "good" level of usability, according to Bangor et al. [18].Estonia, Norway, and Sweden had lower SUS values of 69.4,72.1, 72.6 correspondingly, indicating "fair" level of usability [18].

Association of very positive and very negative experiences with perceived usability
The percentage of respondents who reported having very positive experiences varied from 64% in Estonia to 42-45% in other countries.The percentage of respondents reporting very negative experiences was lowest in Estonia (19%) compared to Norway (20%), Sweden (27%), and Finland (28%).
The regression analysis showed that, in each country, reporting a very positive experience with the patient portal was positively associated with its perceived usability, and reporting a very negative experience was negatively associated with the perceived usability (Appendix 1).The associations remained similar when age and gender were added to the models.The reported experiences alone explain 35% of the variation in perceived usability in Estonia, 23% in Finland, 19% in Norway, and 23% in Sweden.VIF values for the regression models were 1,13 at maximum, suggesting that multicollinearity was not a problem [41].

Very positive experiences
Based on the qualitative analysis of the experience descriptions, the most frequently mentioned very positive experience in all countries was that the records provide information (Table 2).It was also seen as important that information is available and that it is available quickly.Ease of use was also a common theme of very positive experiences in all countries.Respondents appreciated that the portal is easy to use and that information is easy to find and understand.
In addition, different functions were experienced positively.Renewing and viewing prescriptions were experienced positively, especially in Finland.Downloading a Covid vaccination certificate and booking were also appreciated functions.Overview of the information and messaging with health professionals were experienced positively, but they were only available in Norway and Sweden.Only in Sweden, respondents could also compare test results and see graphics about them.Only in Estonia, respondents received notifications of screenings such as cancer control via the patient portal.Finland was the only country offering living will which was an appreciated function.
Respondents described many benefits of accessing their health information, such as receiving a reminder of what was said during appointments, viewing the care history, and preparing for the next appointment.In contrast to these practical benefits supporting patient self-management, Swedish respondents brought out specific positive feelings they felt while reading their notes, such as feeling included, taking part in their care, increased validation of their care, empowerment, increased sense of control, and feeling safer.

Very negative experiences
In all four countries, the most common very negative experience was related to missing information (Table 3) with percentages ranging from 16.6% in Estonia to 39.8% in Norway.In addition, especially in Finland, the information was perceived to be available too slowly, with percentages of 18.3% in Finland and 7.8% in Sweden.Both in Finland and Sweden, some respondents expressed having very negative experiences regarding waiting for the acceptance from the healthcare professional (HCP) before the information is available.
Next, the patient portal was found to be challenging to use.Estonia exhibited a considerably higher percentage (37.6%) of difficulties in finding information or functions, compared to Finland (19.5%), highlighting potential usability issues specific to the Estonian patient portal.Furthermore, functions and the whole system were found to be challenging to use, terminology difficult to understand, and the first page displayed too much information according to the respondents.
The negative experiences also concerned the content of HCPs' notes.Especially in Sweden and Norway, respondents reported very negative experiences related to finding errors in their information with percentages ranging from 3.7% in Estonia to 11.5% in Norway and Sweden (Table 3), raising concerns about data accuracy.Furthermore, the errors were difficult to correct as none of the systems had that functionality.
In addition, some respondents described very negative feelings such as worry, anxiousness, anger, or frustration while reading their EHR or using the patient portal, with Finland (3.8%), Norway (5.4%), and Sweden (3.5%).Specifically, in Sweden, respondents reported experiences where they felt misunderstood or not heard, or perceived that HCPs had lied.Sometimes, respondents expressed a preference for not receiving certain information until they had discussed it with their HCP, particularly if the record contained undesirable laboratory test results or diagnoses that had not been discussed previously.Very negative experiences were also reported when HCPs used condescending, attitudinal, or offensive language, with percentages ranging from 1.2% to 2.9% across the countries.Additionally, respondents in all countries reported difficulty in understanding the medical language used in their EHRs, with percentages ranging from 2.4% to 7.1%.
A small percentage of respondents in all countries worried that somebody might access their Electronic Health Records (EHR) without permission, with percentages ranging from 0.4% in Finland to 4.5% in Norway.Specifically, some Norwegian and Finnish respondents expressed concerns about the potential negative impact of the records on their future, for example, a mental health diagnosis may prevent them from taking out health insurance.Some Swedish and Finnish respondents worried about undesired records from the past, with percentages of 2.2% and 0.8%, respectively.

Discussion
The cross-case analysis of the usability survey results in Estonia, Finland, Norway, and Sweden provided the SUS scores for each country's patient portal and rich qualitative data on the patients' very positive and very negative experiences of the patient portals.The regression analysis of results showed that very positive and very negative experiences explain 19%-35% of the variation in the SUS scores in all four countries indicating that the reported experience narratives provide relevant information on usability issues and support learning from comparison across countries.
While the Estonian patient portal received the lowest SUS score, the percentage of respondents reporting very negative experiences was the lowest in Estonia.Some experiences may have been more critical than others.This means that the number of reported experiences is not comparable across countries.However, the content of the experiences seems to be mainly in line with the SUS scores.The narratives of very positive experiences revealed that most commonly in all countries, the perceived benefits of the patient portal are related to the availability of the information and its use as a reminder of what the doctor said, as also identified by previous studies [12,[42][43][44].Most of the other benefits were also mentioned in all countries.Finland had the highest SUS score and highest percentage of respondents reporting very positive experiences related to the ease of use of the portal and useful features of renewing and viewing prescriptions.On the other hand, Estonian respondents appreciated the useful booking process that was not available in Finland, but they also had the highest number of very negative  experiences related to the difficulty of finding information or functions and other usability issues.
Norwegian and Swedish respondents' very negative experiences were related to missing information that counteracted the main perceived benefit of the patient portals.The Swedish SUS score of 72.6 dropped from 79.81 measured in 2016 [45] which may be due to the experienced discontent with the recent interface update.Compared to others, Swedish respondents had more negative experiences related to the quality of records, they found errors, difficult-to-understand medical terminology, or content discrepancies.Also, Norwegian respondents identified errors and they recognized more often unwanted access to their records or worried about it.Estonian respondents identified less often errors, possibly because they had only epicrises available.
The low response rates and different samples of the countries limit the comparability of the results across countries.Future studies are needed to find ways of reaching representative samples of users and to  A theme included only if >= 1% or mentioned more than in one country.× = A function is not available in this country.
develop methods for selecting comparable subsamples of respondents from countries to be benchmarked.For practical reasons, there were also small differences in distributing the survey [30].For example, the original English questionnaire was translated into local languages and that may have created slightly different nuances to the questions.Also, in Norway and Sweden, the questions referred to the part of the patient portal that provides access to EHR instead of the whole patient portal.Furthermore, the qualitative analysis was performed using local languages that the respondents used.In summary, the developed survey approach could be used for evaluating usability with a wide variety of users and it can support learning from comparison across the countries.In addition, the approach could potentially be used in benchmarking different systems or versions inside one country.The SUS scores provided an approximation of the level of the perceived usability and the qualitative data helped in identifying usability issues to be improved and useful features that patients appreciate or expect to have.Some of the usability issues are concrete, but many require further usability testing so that the cause of the issue can be identified.Thus, as Walji et al. [20] suggest, the survey approach and usability testing complement each other.The study contributes to theory by providing evidence for the relevance of the users' single positive and negative experiences to perceived usability.In addition, the identified usability issues support conceptualizing usability of patient portals which complements the ontology of eHealth usability problems developed by Broekhuis et al. [14].

Table 1
Main functions of the patient portals available in each country.x = the function is available, a only hospital notes are available, b only in some regions, c only epicrises, d depends on the GP.
S.Kujala et al.

Table 2
Themes and percentages of very positive experiences reported in free text in Estonia, Finland, Norway, and Sweden.

Table 2
(continued ) theme included only if >= 1% or mentioned in more than one country.× = A function is not available in this country.Acting on behalf of an adult could be done also unofficially.* = a link to get the certificate. A

Table 3
Themes and percentage of the very negative experiences reported in free text in Estonia, Finland, Norway, and Sweden.