Introduction

As digital technology continues to evolve and be utilized within the media industry, significant changes in a variety of aspects have been identified. One of these is consuming news recommended by algorithms (Kitchens et al. 2020). A recommendation algorithm is a system or technology that recommends a specific item from a set of items to a user by predicting what the user might prefer. The concept of recommendation algorithms can be classified as collaborative or content-based filtering. Collaborative filtering is a method of predicting users’ preferences based on other users’ preference data. It assumes that users with similar preferences, in general, will have similar preferences for certain items. Content-based filtering, in contrast, calculates the similarities between items. In other words, it finds and recommends highly relevant items to users by analyzing their preferred items. An algorithmic news recommendation service (ANRS) has adapted these technologies to a traditional digital news service, thereby providing news that each user may prefer. With ANRS, users can efficiently read news relevant to their interests or preferences without searching for numerous news outlets (Shin, 2020). Furthermore, as algorithmic recommendation services help increase user engagement with content (Sundar and Marathe, 2010), news service providers expect ANRS to lead users to spend more time on their services.

Along with these benefits, it is also important to note the potential risks and side effects associated with ANRS for providers and users (Makhortykh and Wijermars, 2021). For example, Pariser (2011) argues that users’ perceptions or perspectives could be biased by the selective information generated by algorithmic recommendation services, a phenomenon he defined as a filter bubble. In this regard, when ANRS users are caught in this filter bubble, they could have fewer opportunities to read the news that the algorithm judges to be less interesting or relevant to them. Moreover, since algorithmic news recommendations are based on the analysis of users’ personal information and news consumption data, users may have negative perceptions of the service, such as distrust or concerns about privacy. It indicates that the effects or result of applying algorithm technology to news services can vary and may not always be favorable to users, providers, or society.

Developing an advanced recommendation system has been considered an essential task for algorithmic service providers. From the literature on recommendation systems, many efforts have been made for system-wide improvements and more accurate recommendations (Chakraborty et al. 2019; Gabriel De Souza et al. 2019). However, no matter how systematically reliable the algorithmic recommendation service is, users could remain unsatisfied with the service or concern about how their data are collected and utilized (Parasuraman and Miller, 2004; Xiao et al. 2020). Thus, it is crucial for service providers to examine how users perceive algorithmic recommendation services (Pu et al. 2011). Then, ANRS providers can set the direction to provide a better service environment and user experience. Furthermore, it could be the basis for discussions on the impact or change ANRS can bring to society.

In particular, service providers may need to identify significant factors affecting users’ perceptions of ANRS. However, few studies have provided detailed suggestions on the important factors affecting users’ perceptions of ANRS and how to manage and improve it. Moreover, the effects of diverse contexts or users’ characteristics on their perceptions of ANRS have not been actively examined. Therefore, this paper proposes the following research question: What factors affect users’ loyalty to the algorithmic news recommendation service?

This study aims to identify the factors that could affect users’ loyalty to ANRS and verify the relationship between the factors and users’ loyalty to ANRS. This study adopts ANRS by NAVER, the dominant online portal in Korea, as its main subject and utilizes loyalty theory as a theoretical lens to examine it. This paper is structured in the following manner: First, the background of this study and the literature review are presented. Next, the research hypotheses, research model, and adopted methodology are explained. After this, the results of the data analysis are reported. Finally, conclusions, along with some implications and limitations, are suggested.

Research background and literature review

Online portal in Korea and its algorithmic news recommendation service

Online portals in the Korean media industry are positioned as news aggregators, distributing digital news to users. According to a survey from Reuters Institute’s Digital News Report 2021, 81% of Korean respondents read digital news and, among 46 countries, have the highest dependence on consuming digital news on search engines and news aggregators such as online portals but the lowest dependence on press companies’ websites or apps (Newman et al. 2021). These reports indicate that an online portal is important for the Korean media industry and greatly impacts Koreans’ daily news consumption. As the online portals’ influence has become more robust in the Korean news industry, people have begun to question the criteria for news editing and placement of online portals’ news services.

Korean online portals have changed their news service policies to handle these controversial issues for online portals’ news service. For example, NAVER, Korea’s dominant online portal, decided in 2019 to stop editing and placing news with their criteria and leave this role entirely to press companies and algorithm technology. With these changes, NAVER began to provide two types of mobile news services: one to provide news only from the press company, which users selectively choose; the other one is “My News,” which recommends news to users by analyzing users’ interest and preferences with an artificial intelligence recommendation system (AiRS) (NAVER Diary, 2017). Even though NAVER tried to free itself from the role of a press by providing ANRS, it has still been subjected to debates on other issues, such as low diversity and unfairness in news recommendations and transparency of the algorithm system (Kim and Moon, 2021).

Accordingly, NAVER organized an algorithm review committee composed of experts from academia and industry and announced the verification result to resolve the controversy over its ANRS in 2018 and 2022. Based on these verification processes and results, the committee argued that there are no major systematic problems in ANRS and filter bubble issues for users (NAVER, 2022). On the other hand, public concerns about ANRS persist. In this regard, it is necessary to conduct an in-depth examination of this service in terms of technology and users who directly interact with services.

Recommendation algorithm in news services

An algorithm is a procedure or rule to solve a particular problem. Algorithmic recommendation refers to suggesting personalized products or content to users by analyzing their preferences or interests with the algorithm system (Möller et al. 2018). Some studies have been conducted to examine the advantages of algorithmic services: Gomez-Uribe and Hunt (2015) argued that such services could predict users’ usage patterns and, by analyzing user data, recommend niche content that users would have difficulty finding but would give them greater satisfaction. Furthermore, it was verified that it could improve customer loyalty (Tam and Ho, 2005; Zhang et al. 2018) and prevent users from churning to another service (Renjith, 2017; Wang et al. 2009).

Currently, diverse online portals such as NAVER and Google have provided ANRS to their users by considering these features of the recommendation algorithm. As the number of news outlets in the digital environment increases, users have been exposed to an excessive amount of news. In this circumstance, information overload has been considered another significant problem for users (Zhang et al. 2022). ANRS can predict news articles which users might be interested in and make personalized news recommendations for individual users. This advantage helps users save time in searching for news and allows them to efficiently read relevant news (Raza and Ding, 2022).

Based on the importance and effect of ANRS, some studies have been conducted to suggest better-performing algorithm systems and methods. Gabriel De Souza et al. (2019) proposed a deep learning-based news recommendation method to overcome the limitations of the recurrent neural networks method. They also verified that considering article popularity and recency can improve the accuracy of news recommendations. Chakraborty et al. (2019) proposed an approach to recommend news by optimizing news recency and importance through diversity factor. Such studies measure the performance of algorithms by evaluating the results of simulations with prepared datasets. However, Parasuraman and Miller (2004) argued that systematic performance could not guarantee users’ satisfaction with algorithmic recommendation services.

To broaden the scope of understanding of ANRS, some studies have analyzed the relationship between ANRS and users. With the experiment method, Kim and Lee (2021) found that users perceived that news written by the algorithm was of higher quality than that written by a human. On the other hand, Tandoc et al. (2020) proved there is no difference in message and source credibility between news written by algorithms and humans. Moreover, Kwak et al. (2021) verified that ANRS could increase diversity in users’ news consumption, and demographic characteristics such as age, income, gender, occupation, news consumption, and political orientation can affect diversity in news recommendations. It was found that the perceived transparency and accountability impact users’ perception of the ANRS (Shin, 2020). These studies provided implications about how users perceive ANRS and recommended news and what these mean for industry and society. However, few studies have found a causal relationship between the factors and users’ perception of the services and suggested how to manage and improve them efficiently. Moreover, diverse contexts or user characteristics, which could be a crucial factor for the in-depth understanding of ANRS users (Kim and Lee, 2021; Tandoc et al. 2020), have been rarely examined in the context of ANRS.

The previous studies identified that ARNS has many benefits for users and providers, and there are many efforts to improve this service and related systems. However, the technical limitations of ANRS and its side effects on the user side have been discussed as well. It may be challenging to provide timely news by processing a large amount of news in real-time, and the system may fail to distinguish news from different sources with the same topic (Raza and Ding, 2022). Meng et al. (2023) indicated that ANRS could raise users’ privacy concerns as it stores consumption data and analyzes it to predict their news preferences. In addition, they claimed that technical measures are needed to prevent recommendations of fake news and users’ biased news consumption through the ANRS. Kim and Moon (2021) discussed the notion of transparency in digital content recommendation. They pointed out that the ANRS of the online portal in Korea still needs more improvement in algorithm transparency in explaining the process of data collection, involvement of (human) developers, and the range of service providers’ responsibility for side effects. Based on these arguments, failure to address these challenges of ANRS might lead to negative user perceptions, and users would seek alternative ways to read news. Thus, ANRS developers and providers need to understand how users perceive ANRS and what factors contribute to their satisfaction, trust, and consistent favorable attitude towards it. Therefore, this study examines the relationship between users and ANRS through the lens of loyalty theory.

Loyalty theory applied to algorithmic news recommendation service

Loyalty refers to users’ desire to maintain a continuous relationship with a particular service, brand, or organization (Lu et al. 2019; Oliver, 1980). Because loyal customers are more likely to devote themselves to the brand and its service (Coelho et al. 2018; Harris and Goode, 2004), and this loyalty is related to the company’s profitability (Hegner-Kakar et al. 2018; Reichheld et al. 2000), service providers endeavor to maintain customer loyalty. In the online environment, loyalty can be defined as a consumer’s intention to visit the website, use the service, or repeatedly buy a product (Flavián et al. 2006; Jeon and Jeong, 2017).

Loyalty studies have been extended to the mobile environment in alignment with loyalty in an online environment. Several studies have examined the factors affecting the loyalty of mobile services. For example, it was verified that users’ overall satisfaction and trust in service had been found to positively impact loyalty toward the mobile service (Jimenez et al. 2016; Lee et al. 2001). Moreover, it was also found that customers’ attitudes to the brand could mediate the relationship between satisfaction and loyalty to mobile services (Smith, 2020). The significant effects of system and service quality on customers’ loyalty to mobile service were also found (Zhou et al. 2021). Furthermore, perceived risk and relative advantage of mobile banking services remarkably seems to impact the users’ loyalty (Esmaeili et al. 2021). These studies commonly argued that diverse factors affect loyalty, and managing the crucial one is necessary for service providers to sustain their business.

In the Korean media industry, various news services providers such as press companies, online portals, social media, and blogs compete and provide many news services. In this competitive environment, retaining loyal users could be essential to sustain their service and related business. Especially users’ perception of a ANRS may differ from the previous one, and opinions on applying the algorithm to news services can vary depending on their preference for or resistance to this technology. Therefore, this study aims to determine which factors significantly impact users’ loyalty to ANRS.

Research hypothesis and model

Satisfaction, trust, and loyalty of algorithmic news recommendation service

Satisfaction refers to favorable responses or assessments of a particular service or product (Anderson et al. 1994). In both an offline and an online environment, satisfaction can be an important factor in estimating the customer’s behavior (Zeithaml et al. 2002). Satisfaction is known to increase loyalty to mobile services (Lee et al. 2001) and intention to use information system-related services (Delone and Mclean, 2003). Especially in the online environment, managing satisfaction is crucial for product or service providers because finding an alternative is relatively easy for the user (Shankar et al. 2003). Some studies on recommendation services have also verified that users’ service satisfaction positively affects their loyalty (Abumalloh et al. 2020; Yoon et al. 2013). In this sense, similar to other sorts of web or mobile services, one of the critical goals of ANRS would be to maximize users’ satisfaction by providing personalized news to users in an effective manner. Furthermore, it is expected that when users are satisfied with ANRS, they will be more likely to be loyal to the service. Hence, this study proposes the following hypothesis:

H1: Satisfaction with ANRS has a positive effect on loyalty to ANRS

Trust is defined by customers’ beliefs about receiving a promised service and assurance of a service or company’s reliability and integrity (Morgan and Hunt, 1994). It was verified that trust in service has a positive correlation with users’ loyalty to online services (Flavián et al. 2006). This role of trust could be critical when users perceive uncertainties and risks from the service (Ribbink et al. 2004). In the context of the automated computer system, trust can be conceptualized as an attitude or expectation toward the system to support them in accomplishing their objective in an unclear situation (Lee and See, 2004). In this sense, it has been verified that trust in a recommendation service positively influences user behavior, such as acceptance of the recommendation service and recommended contents (Cramer et al. 2008) and loyalty to it (Abumalloh et al. 2020).

Furthermore, trust has been considered to positively affect users’ satisfaction in online service environments (Harris and Goode, 2004). In this regard, it was proven that continuous trust in recommendation services positively affects satisfaction (Ashraf et al. 2020). In the context of ANRS, users might be concerned about its systematic defects or malfunctions. Moreover, trust can be a predictor of users’ online news consumption and participation (Fletcher and Park, 2017). Therefore, news service providers need to manage their service to be trustworthy so users would consistently read news from their service with satisfaction. In particular, the role of trust that affects users’ satisfaction with news services is expected to be more critical in Korea, which is one of the lowest-ranking countries for trust in overall news media (Newman et al. 2021). Based on these arguments, when users trust ANRS, they would be more likely to have loyalty and satisfaction with it. Therefore, this study proposes the following hypothesis:

H2: Trust in ANRS has a positive effect on loyalty to ANRS

H3: Trust in ANRS has a positive effect on satisfaction with ANRS

Factors affecting satisfaction with and trust in algorithmic news recommendation service: service quality and personal factors

Perceived service quality is a consequence of a comparison between customers’ expectations of service and their perception of how the service is executed. It has been considered a crucial factor positively correlated with satisfaction (Zeithaml et al. 1996) and trust in a service (Chiou and Droge, 2006). It is divided into two dimensions: technical quality and functional quality (Storbacka et al. 1994). The former is related to what users gain from the service, while the latter is related to the process of interaction with the service. In the context of ANRS, its technical quality is related to the recommended news, and the functional quality is related to the process of users’ interaction with ANRS.

Moreover, to understand users’ responses or evaluations to ANRS more precisely, this study adopts factors related to personal characteristics. Knijnenburg and Willemsen (2015) suggested a user-centric evaluation framework by adding personal characteristics to better understand the recommendation service user. In this regard, service quality and personal factors could affect the user’s perception of ANRS. Based on this assumption, this study suggests the service quality and personal factors related to ANRS. It will examine the relationship between these factors and users’ satisfaction with and trust in service, which leads to service loyalty.

Technical quality

One of the critical objectives of the algorithmic recommendation service could be predicting users’ preferences or interests and recommending content or products to users accurately (Gunawardana and Shani, 2009). By considering it, developing an accurate recommendation algorithm employing user data such as usage patterns has been considered an important research goal (Herlocker et al. 2004). In the context of diverse media content recommendations, such as movies (Bennett and Lanning, 2007) and TV programs (Chang et al. 2013), accuracy has been considered an essential component. These arguments reflect the tendency that people are likely to consume media content with a preferred genre, format, and topic. News also deals with various issues and topics, and each press company provides these based on its standards and point of view. Accordingly, users who read news through ANRS are expected to want the service to accurately predict their preferences and interests and recommend relevant news to them (Zuiderveen Borgesius et al. 2016). In this sense, when users perceive that this ANRS works consistently with high accuracy, they can be satisfied with and trust the algorithm system. Therefore, this study proposes the following hypothesis:

H4a: Perceived accuracy has a positive effect on users’ satisfaction

H4b: Perceived accuracy has a positive effect on users’ trust

In previous studies, diversity has been considered one of the crucial factors that content recommendation service providers should consider. For instance, Zhang and Hurley (2008) confirmed that excessive similarity in recommended content could make users perceive recommendations as monotonous. In this sense, Zhou et al. (2010) insisted that content recommendation services should also provide niche content that is relevant to users but can hardly be searched by themselves. Furthermore, Ekstrand et al. (2014) found that the diversity of recommended content positively impacts users’ satisfaction with the recommendation service. Pariser (2011) also argued that low-diversity recommendations might mean users have less chance to encounter diverse information, issues, and opinions, which can narrow users’ views of society. In the context of ANRS, diversity has been considered one of the essential quality factors (Karimi et al. 2018). Users would expect sufficient diversity of recommended news regarding the topic, issue, opinion, and press company so that their news consumption would not become overly biased. In this sense, users would be more satisfied and trust the ANRS when they perceive that it recommends diverse news. Based on these arguments, this study proposes the following hypothesis:

H5a: Perceived diversity has a positive effect on users’ satisfaction

H5b: Perceived diversity has a positive effect on users’ trust

News value refers to the quality of recommended news, determined by the importance of the topics and issues with which the news deals. It is considered a criterion for news editors or service providers to select the news among numerous news (Shoemaker et al. 1991). Basically, ANRS is designed to provide the most relevant news by priority based on the user’s preferences or interests (Zuiderveen Borgesius et al. 2016). It means that users are more likely to get recommended news highly related to their preferences or interests. However, it also implies that they can hardly get news from ANRS that is timely or socially important but less relevant to them. In this regard, Thurman et al. (2019) argued that users would be concerned about missing important information, which would also affect their acceptance of news recommendation technology and news selection. Even though users want ANRS to recommend relevant news to them, they would also expect the service to recommend news by considering its value. Therefore, this study adopts the news value as one of the critical technical qualities in ANRS. In this sense, users can be satisfied with, and trust the ANRS when they perceive the service recommends news that deals with socially essential and timely topics or issues. Therefore, this study proposes the following hypothesis:

H6a: Perceived news value has a positive effect on users’ satisfaction

H6b: Perceived news value has a positive effect on users’ trust

Functional quality

Usability refers to the ease with which a user can achieve a particular goal in the context of a computer-related system or service (Flavián et al. 2006). It is considered an important criterion for evaluating the quality of online services (Pu et al. 2011) and mobile services (Az-zahra et al. 2015). Regarding the role of usability and its influence in the context of online websites, Casaló et al. (2008) demonstrated that perceived usability positively affects customers’ satisfaction with and loyalty to the online website, and Flavián et al. (2006) showed that usability could also positively influence users’ trust in the website. Furthermore, usability positively affects users’ satisfaction with recommendation services (Pu et al. 2011). Especially for mobile ANRS, in which users should read recommended news through mobile devices, there might be limits on the service environment for users due to the relatively small screen size. This emphasizes the importance for ANRS providers to optimize usability to ensure a smooth and seamless service experience for users. Therefore, they should improve the overall convenience, accessibility, and layout of the service to make it a more user-friendly and efficient service environment. Based on these arguments, users will be satisfied and trust the ANRS when they perceive the service’s usability is developed well to select and read recommended news easily. Therefore, this study proposes the following hypothesis:

H7a: Perceived usability has a positive effect on users’ satisfaction

H7b: Perceived usability has a positive effect on users’ trust

Algorithmic transparency in news media refers to an explanation of how the algorithm system or service works, which helps users understand the logic of recommendation and accept the recommended news (Diakopoulos and Koliska, 2017). Some studies have verified that transparency increases users’ trust in the recommendation system (Pu et al. 2011) and intention to purchase the recommended products (Sinha and Swearingen 2002) and helps acceptance of recommendations (Cramer et al. 2008). Although disclosure of algorithms could be considered a leak of corporate confidentiality in service providers (Diakopoulos, 2015), the imbalance of information between the service provider and users can be partially resolved by explaining to users how the algorithm service system works (Diakopoulos and Koliska, 2017). Thus, perceived transparency of ANRS is expected to help users be satisfied and trust the overall service and its system. Therefore, this study proposes the following hypothesis:

H8a: Perceived transparency has a positive effect on users’ satisfaction

H8b: Perceived transparency has a positive effect on users’ trust

Personal factors

Attitude is an individual’s overall evaluation of a certain type of object (Petty et al. 1994). Customers’ attitudes can impact their intention and behavior related to service (Mazaheri et al. 2011). Moreover, a favorable attitude can increase satisfaction with service (Oliver, 1980) and loyalty to online services (Currás‐Pérez et al. 2013). Even in the context of a recommendation service, users’ attitude has been considered an essential component (Knijnenburg and Willemsen, 2015). For instance, Pu et al. (2011) insisted that a favorable attitude toward recommendation services can increase the intention to use the service and purchase a recommended product. By extension, Mazaheri et al. (2011) proved that customers’ satisfaction depends on their pre-existing attitude toward the company when provided information is ambiguous to them. In this regard, when users accept the recommendations from the algorithm system, called the ‘black box’ due to its concealed and complex recommendation logic (Diakopoulos, 2015), their pre-existing attitude toward service providers might affect their response to recommendation service. In this situation, if users have a positive attitude toward the service provider, it could help them become satisfied with and trust its ANRS. Therefore, this study proposes the following hypothesis:

H9a: Pre-existing attitude toward the service provider has a positive effect on users’ satisfaction

H9b: Pre-existing attitude toward the service provider has a positive effect on users’ trust

Privacy concerns refer to perceptions of risks associated with the situation when a person’s privacy is violated (Tan et al. 2012). In the context of online service, it is known to decrease users’ satisfaction with (Cheng and Jiang, 2020) and trust in (Martin, 2018) the service. When a recommendation service collects a lot of user-related data, the service is available to predict users’ preferences or interests more accurately, but privacy issues can also arise (Lam et al. 2006). In this sense, privacy concerns are an important personal characteristic related to a user’s evaluation or interaction with algorithmic recommendation systems (Knijnenburg and Willemsen, 2015). ANRS collects and utilizes personal data to estimate a user’s news preferences or interests. Because this service automatically provides news to users, their apparent social or political preferences can be decided and revealed through the recommended news, whether correct or not. Moreover, they are more likely to be concerned about this situation when they use ANRS through mobile devices and share it with others (Mohallick and Özgöbek, 2017). In this sense, it is reasonable to assume there is a negative effect of users’ privacy concerns on satisfaction with and trust in ANRS. Therefore, this study proposes the following hypothesis:

H10a: Privacy concerns have a negative effect on users’ satisfaction

H10b: Privacy concerns have a negative effect on users’ trust

The complete research design with the proposed hypotheses is presented in Fig. 1.

Fig. 1: Research model.
figure 1

Model with factors related to loyalty theory, service quality, and user characteristics.

Full size image

Research method

Questionnaire development

This study adopted the Likert scale for survey items because it is widely used in social science studies (Heo et al. 2022). It has been considered a convenient and appropriate method to examine human attitude, perception, or behavior and provide quantitative data for analysis and statistical inference (Chyung et al. 2017; Li, 2013). Some studies pointed out that seven responses are the most valid and reliable method to reflect survey participants’ subjective assessment of each question (Finstad, 2010; Preston and Colman, 2000). Therefore, this study used the seven-point Likert scale for all measurements, ranging from ‘1 – strongly disagree’ to ‘7 – strongly agree’.

This study developed survey items based on valid literature on algorithmic news recommendations, loyalty theory, and service quality to conduct the online survey. For example, this study measured satisfaction with adapted scales of previous research that measured the satisfaction of recommendation services and online websites (Casaló et al. 2008; Ekstrand et al. 2014; Yoon et al. 2013). Users’ trust in ANRS was measured with modified scales from previous research that tried to measure trust in recommendation services and e-services (Luarn and Lin, 2003; Pu et al. 2011). Loyalty on ANRS was measured with adapted scales from previous research that tried to measure the loyalty of recommendation services and online websites (Casaló et al. 2008; Yoon et al. 2013).

This study utilized scales from previous literature that measured the accuracy of recommendation services (Ekstrand et al. 2014) to measure the perceived accuracy of ANRS. For adapting these scales in the context of ANRS, this study referred to the discussion from the literature related to news recommendation services (Liang et al. 2006). The measurement scales from the research related to algorithmic recommendation (Ekstrand et al. 2014) were modified to measure the perceived diversity of ANRS in this study. This study also considered the discussion of Chang et al. (2013) about the concept and meaning of diversity in the context of recommendation services. This study developed the scales to measure the users’ perceived news value of ANRS based on the discussion from the literature regarding the concept of news value and its evaluation (Bednarek, 2016; Harcup and O’neill, 2017; Westerståhl and Johansson, 1994).

The perceived usability of ANRS was measured using the scales modified from the literature researched users’ perception of the website and measured the perceived usability (Casaló et al. 2008; Ranganathan and Ganapathy, 2002). The measurement scales from the studies trying to measure the transparency of the recommendation system (Cramer et al. 2008; Pu et al. 2011) were modified to measure the perceived transparency of ANRS. This study referred to the discussion from Diakopoulos (2015) about the accountability and transparency of news recommendation systems to adapt the scales appropriately.

This study adapted scales from Abzari et al. (2014) used for measuring brand attitude and from Currás‐Pérez et al. (2013) for measuring attitude toward the social network service to measure the ANRS users’ pre-existing attitude toward provider. The measurement scales from the research related to online privacy (Ekstrand et al. 2014) were modified to measure the ANRS users’ privacy concerns in this study. This study also referred to the discussion regarding the privacy concern in the news recommendation system from Mohallick and Özgöbek (2017). All measurements and references are presented in Table 1.

Table 1 Constructs and measurements.
Full size table

Sampling and data collection

To verify the proposed hypotheses, this research conducted an online survey for users of “My News”, the free mobile ANRS of NAVER, Korea’s dominant online portal site. “My News” is one of the various services embedded in the NAVER mobile app. When users access the “My News” service, recommended news is automatically displayed on its page, and they can scroll down and read various types of recommended news. This personalized news recommendation feature is unavailable for NAVER mobile app users who are not logged in. Therefore, this study targeted the users who have automatically logged in to the app as the survey participants. Before the main survey, participants were asked to answer the question, “Do you use the service of the NAVER mobile app while automatically logging in to it?”. Only those who answered “yes” to this question were eligible to participate in the main survey.

The survey was conducted with Macromill Embrain, an online research company in Korea, from June 4 to June 21, 2019. The age range of the sample in the study was limited to people in their 20s through their 60s, which corresponds to the primary demographic for the news service offered by the portal. Given that the objective of this study was to understand the general perceptions of ANRS, equal representation of both genders and age groups was ensured during data collection. The research company approached potential participants based on its’ sample pool. Random potential participants within each age and gender sample pool were asked to complete the survey until the numbers in each group were balanced.

It is acknowledged that a sample size of at least 200 is preferable when using a structural equation model (Kline, 2023). Moreover, with regard to the number of variables in the model, it has also been accepted that 10 to 20 samples per latent variable in the model are considered appropriate (Mitchell, 1998). Given the complexity of the research model and the number of latent variables and indicators, this study aimed for a sample size of 450 responses to ensure statistical significance. In total, 495 responses were collected, including an extra 45 responses.

The survey began with basic demographic information and personal factors, followed by questions about overall ANRS consumption patterns, such as preferred news categories and usage. Finally, questions related to service quality and loyalty-related factors were presented. Questions were carefully placed to ensure that respondents’ answers would not be influenced by prior questions. For this, this study placed questions about satisfaction, trust, and loyalty separately, as these factors could be highly correlated. This study also prioritized questions that required less time for respondents to answer at the front, allowing them to feel less burdened and gradually understand the survey context.

This study conducted Harman’s one-factor test with SPSS 27.0 software to check whether common method bias occurred in this survey design. Since the total variance extracted by one factor from this study was 43.8%, less than the recommended threshold of 50% (Podsakoff et al. 2003), it showed there is no issue with common method bias in this survey.

After deleting 12 inappropriate answers from the collected data set, 483 responses were included in the analysis. Table 2 shows 483 survey respondents’ profiles, including gender, age, preferred news categories, time, and daily usage frequency. The gender and age groups of respondents were evenly divided. Among 483 respondents, 50.3% were male, and 49.7% were female. In addition, 24% of respondents were in their 20s, 25.1% were in their 30s, 25.3% were in their 40s, and 25.7% were in their 50s. Among eight news categories in “My News”, the most preferred category by respondents was society (29.2%), followed by politics (25.9%) and economy (20.1%), and international news (1.0%) was the least preferred, followed by IT/science (1.4%). Regarding the average time for reading news headlines or articles through “My News” daily, 59.9% of respondents spent less than 20 min, and only 6.2% spent more than 60 min. In the average frequency of “My News” daily use, 68.1% of respondents used it fewer than five times, while 11.6% used it ten times or more. Table 3 indicates the mean value and standard deviation of each construct.

Table 2 Respondent profile.
Full size table
Table 3 Descriptive statistics.
Full size table

Research results

This study adopted partial least squares structural equation modeling (PLS-SEM) to examine the proposed research model. It is well-known to minimize the error term of endogenous latent variables and maximize the explanatory power (R-square) in estimating path coefficients. It is more suitable for developing theory or conducting exploratory research focusing on predicting or explaining latent variables rather than examining the structural characteristics of the model. Furthermore, this method can test multi-staged causal relationships simultaneously (Gefen et al. 2000) and is effectively applied when the sample size is small or the model is complicated. This method does not presume that collected data are normally distributed (Hair et al. 2011). This is why this study used PLS-SEM, in particular SmartPLS 3.3.9 software, to test the structural and measurement model and estimate the path coefficient with the collected data.

Test of measurement model

The study analyzed factor loading, Cronbach’s alpha, and composite reliability (CR) to ensure the reliability of used items for the research model. The results presented in Table 4 show that the factor loading of all items is also over 0.7, and both Cronbach’s alpha and CR values are over 0.7. As a result, all items in this study are considered to have acceptable reliability (Barclay et al. 1995; Blalock 1974).

Table 4 Reliability and convergent validity.
Full size table

This study assessed several criteria to verify the validity of the proposed research model. The proposed research model has sufficient convergent validity as the average variance extracted (AVE) for all factors exceeds the acceptable level of 0.5 (Gefen et al., 2000). The discriminant validity of the model was also verified using the Fornell-Larcker criterion and Heterotrait–monotrait ratio (HTMT). The diagonal values in Table 5, which represent the square root of the AVE, are higher than other values in the same column, which are its correlation coefficients. Additionally, all the HTMT values in Table 6 are smaller than a threshold of 0.9. These findings indicate that the model has an acceptable level of discriminant validity (Henseler et al., 2015). Therefore, this study confirms that the research model meets the major standards for reliability and validity.

Table 5 Fornell-Lacker criterion.
Full size table
Table 6 Heterotrait–monotrait ratio (HTMT).
Full size table

Test of structural model

As a result of identifying the explanatory power of the research model presented in Fig. 2, the proposed model explains 70.3% of the variance in users’ loyalty, 72.3% of the variance in satisfaction, and 58.9% of the variance in trust in ANRS. Following the standard of Hair et al. (2011), all the endogenous latent variables in the proposed structural model have moderate R-squared value, implying that this model sufficiently reflects the users’ perception of ANRS and their loyalty to it. As shown in Table 7 and Fig. 2, 11 out of 17 hypotheses were supported by the bootstrapping analysis for the testing proposed hypothesis. From this analysis, it was verified that both satisfaction (β = 0.421; t = 9.431; p < 0.001) and trust (β = .469; t = 10.587; p < 0.001) positively affect loyalty to ANRS, supporting H1 and H2. Furthermore, trust positively affected satisfaction (β = 0.422; t = 9.520; p < 0.001), supporting H3. From these findings, it was reconfirmed that when users are satisfied with and trust the service, their loyalty to it is increased, and trust is an important determinant of satisfaction and loyalty in the context of ANRS, as expected from previous studies (Harris and Goode, 2004).

Fig. 2: Result of research model analysis.
figure 2

Model with path coefficients, statistical significance (P-value), and explanatory power (R-squared).

Full size image
Table 7 Result of path analysis.
Full size table

Regarding the technical qualities of ANRS, only perceived accuracy positively affected satisfaction (β = .095; t = 2.409; p < 0.01), and perceived news value positively affected trust (β = 0.130; t = 2.842; p < 0.01), supporting H4a and H6b respectively. In addition, only perceived diversity significantly affects neither satisfaction nor trust among service quality-related factors, and therefore, H5a and H5b were not supported. In the case of functional qualities of ANRS, only perceived usability positively affects both satisfaction (β = 0.234; t = 5.561; p < 0.001) and trust (β = 0.134; t = 2.878; p < 0.01) among service quality-related factors supporting H7a and H7b. In addition, perceived transparency positively affects only trust (β = 0.205; t = 4.797; p < 0.001), supporting H8b. In terms of coefficient size for explaining satisfaction and trust that leads to the formation of loyalty to ANRS, it was shown that functional quality-related factors were more powerful than technical quality-related factors overall.

Among two personal factors, pre-existing attitude toward the service provider positively affects both satisfaction (β = 0.118; t = 2.722; p < 0.01) and trust (β = .400; t = 10.860; p < 0.001), supporting H9a and H9b. Privacy concerns, however, negatively affect only trust (β = –0.103; t = 3.289; p < 0.01), supporting H10b. It shows that not only service quality-related factors but also personal factors are important determinants for users’ satisfaction and trust that lead to the formation of loyalty to ANRS.

In addition, this study tried to verify the indirect effects relationship among users’ satisfaction, trust, and loyalty on ANRS. As shown in Table 8, the total effects of satisfaction on loyalty, the same with a direct effect, were statistically significant (β = 0.421; t = 9.431; p < 0.001). The total effects of trust in loyalty were verified to be significant (β = 0.646; t = 19.613; p < 0.001). The direct effects of trust in satisfaction (β = .469; t = 10.587; p < 0.001) and indirect effects of trust in loyalty (β = 0.178; t = 6.14; p < 0.001) were found to be statistically significant as well. These results showed that trust has larger direct effects than indirect effects on loyalty, and users’ satisfaction with ANRS partially mediates the relationship between trust and loyalty.

Table 8 Direct, indirect, and total effects of trust and satisfaction on loyalty.
Full size table

Conclusion

Discussion

This study delved into users’ perceptions of ANRS and how loyalty theory applies to ANRS context. For ANRS providers, keeping loyal users is crucial in several ways. First, collected users’ consumption and feedback data are key components of improving the system performance of algorithmic-based services (Raza and Ding, 2022). If ANRS users have loyalty to the service and keep using it, more data can be accumulated from the system consistently. Then, service providers can improve the service quality of ANRS based on these and it is essential for ensuring the sustainability of the service. Second, platform service providers like NAVER, which offers multiple services, including search, commerce, content, and news services, can utilize ANRS to help users read news efficiently and keep them in their platform environment. However, if ANRS leads to discomfort, it can harm the relationship between users and the platform, which in turn can have negative consequences for the platform’s overall business. Hence, this study emphasizes the importance of making users remain loyal to ANRS and identifying the factors that impact their perception and experience of ANRS.

The study proposed two types of service quality (technical quality and functional quality) and personal factors influencing user satisfaction and trust that lead to the formation of ANRS loyalty. This study found that satisfaction and trust positively impact user loyalty to the service. This is consistent with previous studies conducted on diverse online services, including recommendation services (Abumalloh et al. 2020; Harris and Goode, 2004). Additionally, trust was found to influence satisfaction in the ANRS environment (Ashraf et al. 2020). It is noteworthy that trust could be the more decisive factor in circumstances when users perceive service uncertainties (Ribbink et al. 2004).

This study analyzed the service quality of ANRS and divided it into technical and functional qualities. It considered its characteristics and found that functional quality factors had a greater impact on satisfaction and trust than technical quality factors. This implies that ANRS users were more concerned with the process of using the service over the results of algorithmic news recommendations. Moreover, it is notable that this study was conducted during the initial stages of ANRS adoption, so users might have lower expectations for the technical quality of the service. Since the technical quality usually needs enough time for data accumulation, the effect size of technical quality factors might be relatively small.

This study confirmed that when users perceive recommended news from ANRS as relevant, they are more satisfied with ANRS. This implies that users may expect ANRS to accurately predict their news consumption patterns and preferences and provide news selectively (Zuiderveen Borgesius et al. 2016). This is the key difference between ANRS and traditional news services which do not consider individual preferences. To increase the accuracy of news recommendations, therefore, various studies have suggested better-performing algorithms (Darvishy et al. 2020; Gabriel De Souza et al. 2019), and diverse metrics have been developed to more effectively evaluate system accuracy (Gunawardana and Shani, 2009).

Moreover, news value was the only technical quality factor found to positively affect trust. This means that while users want ANRS to provide them with news congruent with their preferences or interests, they also expect it to recommend news related to socially meaningful and important issues. This is related to the argument that users are concerned about missing important information because recommendation services provide content selectively (Thurman et al. 2019). It is also consistent with the purpose of reading news; people read the news not only for their interests but also to stay informed about social issues, even if they are less relevant to their interests or preferences.

This study confirmed that the perceived diversity of ANRS does not determine the users’ satisfaction with and trust in ANRS despite concerns about the low diversity from the filter bubble in its recommendations. While this study could not reveal whether the service faces the filter bubble problem, it did confirm that low perceived diversity in news recommendations does not negatively impact user perceptions of the service. This finding is inconsistent with previous research that claimed that diversity is important to user perceptions of recommendation services (Ekstrand et al. 2014). There are a couple of potential reasons for this difference. First, ANRS users might recognize a tradeoff relationship between recommendation diversity and accuracy (Isufi et al. 2021; Raza and Ding, 2020). Thus, although users perceive the ANRS as providing news content with low diversity, they accept its narrowed news recommendations without complaint or distrust and are relatively satisfied with relevant news recommendations. Second, in the case of “MY News,” the subject of this study, it recommends news with real-time updates. Accordingly, it might be challenging for survey respondents to assess the diversity in their service experience of ANRS through the survey. This implies that the perception, role, or effect of diversity may depend on the characteristics of the service environment.

Among service quality factors, only the perceived usability of ANRS affected both satisfaction and trust, and it was the most significant factor influencing satisfaction regarding effect size. This expands upon previous findings that usability is crucial for the formation of satisfaction and trust in online and mobile services such as websites (Az-zahra et al. 2015; Casaló et al. 2008; Flavián et al. 2006) to mobile ANRS. Mobile ANRS users might face certain limitations in news service usage with mobile devices with smaller screens. Therefore, usability could become more critical for them. Furthermore, users may have a negative perception of the service when the ANRS is inefficient in browsing, selecting, and reading recommended news articles that are normally in text format.

Based on research findings, perceived transparency was the most significant service quality factor influencing trust related to effect size. This is consistent with previous research that revealed perceived transparency can boost user trust in content recommendation services (Pu et al. 2011). Notably, the descriptive analysis of each construct revealed that perceived transparency had the lowest mean score and highest standard deviation among all the factors. This assumed that survey respondents did not highly rate ANRS’s transparency overall, and there were large differences in rating its transparency among them.

Previous studies have shown that service quality factors positively affect satisfaction and trust in service (Chiou and Droge, 2006; Zeithaml et al. 1996). However, in this study, some service quality factors were found to have varying impacts on satisfaction and trust. For example, perceived accuracy only significantly influenced satisfaction, not trust. This contradicts previous research that found users’ perception of the machine-learning system impacts their trust in it (Yin et al. 2019). Accurately personalized news recommendation is a fundamental function of ANRS. However, this result indicates that even users who perceive the service’s news recommendations as accurate may not trust the service and its system.

In contrast, perceived news value and transparency only impacted trust. The timely and socially important news topics recommended by ANRS may not be relevant to users’ interests. In addition, ANRS can accurately analyze users’ data and recommend news that meets users’ interests without explaining how the algorithmic system works. However, users seem to trust the service and system regardless of their satisfaction with the ANRS service when they perceive that the ANRS sufficiently plays its role as a media platform providing socially important news content with high transparency.

This study discovered a significant relationship between personal factors and user satisfaction and trust in ANRS. This finding aligned with the argument that personal factors play a crucial role in explaining user perceptions of algorithmic services (Knijnenburg and Willemsen, 2015). In this regard, this study found that users’ favorable pre-existing attitudes toward the service provider positively impacted their satisfaction and trust in ANRS, with the effect size on trust being the largest among all factors. This concurs with previous studies on the relationship between user attitudes toward service and their behavioral intentions (Pu et al. 2011) and satisfaction (Mazaheri et al. 2011). This study focused on “MY News,” an ANRS offered by NAVER. Korean people rely heavily on online portals (Newman et al. 2021), and NAVER provides various digital services and content through its online and mobile portal services. However, online portals have been associated with controversial issues such as fairness in news editing and manipulated comments on the news. Thus, user attitudes toward NAVER may be influenced by their daily use of its various services, and perceptions of positive or negative issues related to the provider might be cumulative. Accordingly, their attitude towards NAVER may influence their experience and perception of using ANRS operated by NAVER.

Furthermore, users’ privacy concerns negatively impact trust in ANRS, and it is related to a previous finding that high privacy concerns could decrease trust in the online service (Martin, 2018). ANRS is based on personal data such as news consumption patterns or demographic information (Helberger, 2019). In this regard, some users could be concerned that their social, political, or cultural preferences are collected and could be abused in some way. Additionally, users are expected to consider situations where their mobile devices are shared or exposed to others, allowing others to access the news they are reading or have recommended (Mohallick and Özgöbek, 2017). It is assumed that these concerns negatively impact user trust in ANRS. However, it was found that privacy concerns did not influence satisfaction. In other words, users with high privacy concerns could be satisfied with the experience of receiving news recommendations and reading them through ANRS.

Theoretical implications

Based on the findings and discussions, this study provides several academic implications. First, this study examined the ANRS with loyalty theory and found significant relationships among variables. Notably, this study identified the role and importance of satisfaction and trust in users’ loyalty to ANRS. Moreover, this study led to a discussion of the similarities and differences between service quality factors determining satisfaction or trust. These findings would help future studies to better understand the users’ perceptions of ANRS and other algorithmic services by suggesting and identifying appropriate antecedents.

Second, this study enriches ANRS studies by proposing five service quality factors and verifying their effects on users’ perception of ANRS. In addition, in this study, the service quality of ANRS was divided into technical and functional quality. This approach could be extended to research to comprehensively understand users’ perceptions of other algorithmic recommendation services or behavior.

Third, this study found that considering personal factors helps understand users of algorithmic recommendation services. Users can have diverse experiences or perceptions related to ANRS. On the other hand, the recommendation algorithm is a so-called black box, and users could have difficulty understanding how it collects or utilizes their data and offers recommendations. Accordingly, privacy concerns could negatively impact trust in ANRS. Additionally, it was also found that favorable pre-existing attitudes toward service providers could positively impact users’ satisfaction and trust in ANRS. The issues from other services or firm levels may significantly affect users’ satisfaction and trust in their ANRS.

Managerial implications

The findings related to the service quality of ANRS present several managerial implications for providers and developers of various algorithmic recommendation services, including ANRS, in effective service management and development. First, developers who want to optimize their recommendation algorithms should consider various factors, including users’ interests, lifestyles, or regions. By considering these factors, the accuracy of the recommendations can be improved. Additionally, it is crucial to utilize search history data to recommend personalized news. Platforms such as NAVER and Google can leverage this information to understand users’ interest better. Relevant news articles can be prioritized and presented to users, increasing their motivation to read the recommended news. Thus, to provide better service experiences, it is crucial to identify parameters optimized for the service environment and provider’s characteristics and integrate them into the recommendation system.

Second, it is important for ANRS to not only recommend news that is relevant to users but also to cover socially important or timely issues or events. However, users have heterogeneous preferences, and individual interests change periodically. Therefore, optimizing the accuracy and news value balance in the recommendation system can be challenging. ANRS may provide a function that allows users to set the balance or level of attributes such as accuracy or news value. This would enable users to prioritize different parameters that the recommendation system weighs and determine the direction of news recommendations themselves. The process of using this function is expected to drive more interactions between users and their recommendation service. By doing so, users might perceive the news recommendations as more personalized and have greater satisfaction and trust in the service.

Third, the transparency of the algorithm influences users’ perception of ANRS significantly. However, it is not easy for users to fully understand the mechanisms or methods of ANRS. Therefore, one solution for increasing transparency could be explaining the link between recommended news and users’ preferences or interests. For example, common keywords related to both users and recommended news can help users better understand the basic logic of algorithmic recommendations. However, a recommendation algorithm is a proprietary asset or knowledge of the business. Therefore, decisions on how much ANRS transparency should be maintained must be determined deliberately. In this sense, the service provider can consider cooperating with academia or institutions to establish guidelines for algorithm transparency for users and providers. Furthermore, this study found that the perception of transparency can vary considerably among users, so it is necessary to consistently verify how the disclosure of information by providers contributes to users’ transparency perceptions.

Fourth, usability significantly affected satisfaction and trust in ANRS. Therefore, ANRS developers should consider that the most important benefit that users expect from ANRS is efficiency and thus work toward increasing the usability of the service for users. In particular, in the case of mobile ANRS, users can access the service through various devices. Therefore, service developers must ensure that users can use the service seamlessly, even if they change devices. Tracking and displaying the news the user has previously read can also be an effective means of improving the usability of ANRS.

Lastly, there has been an intense debate regarding the potential problem of ANRS and selective news consumption. This study found that perceived diversity did not affect users’ satisfaction and trust in ANRS. However, service providers must revisit the meaning and impact of diversity by considering the role of news and the position of dominant platforms. It is still unclear whether current users’ news consumption through ANRS is problematically biased or not. Additionally, there are insufficient empirical studies to prove it. Diversifying not only the types of news but also the outlets that provide news may be one way to prevent the potential problem of ANRS. In fact, NAVER introduced another news service based on a subscription to a press company, along with ANRS (MY News). NAVER should also consider developing educational campaigns or customized services for users with low media literacy to facilitate their proper use of ANRS.

Limitations and future research directions

This study is a meaningful effort to understand the users’ perception of ANRS in Korea, which has received little precise examination. Nevertheless, this study is still not free of all limitations. First, the survey was only conducted on “My News” users from NAVER. Even though this service is provided by the most influential online portal in Korea, the results might limit the generalization of the implications to other ANRS or algorithmic services. In this sense, applying this approach or model to different contexts would be meaningful. Moreover, a series of comparative studies with different platforms or countries would be valuable.

Lastly, this study approached the online portal’s ANRS from a user perspective and might be limited in deciding the direction of service improvement or policy. Applying the algorithm technology to an online portal’s news service has been controversial in the industry, and its effect on society is broad. Therefore, a service provider should examine the response from diverse stakeholders in the media industry and manage their service with a comprehensive understanding of ANRS and its effect. For example, applying algorithm technology to an online portal’s news service is expected to affect the press companies that publish and provide news to online portals. There are cases of misuse of services by press companies through the loophole of algorithms so that their news is exposed to more users within ANRS. Therefore, this study calls for an in-depth study to examine the impact of applying algorithmic technology to news services on news producers by considering potential advantages or disadvantages. By expanding this study to understand the diverse aspects or effects of ANRS, it will be able to preemptively prepare a counterplan for conflicts or problems that may arise due to the wider dissemination of algorithm technology within the media industry.