Next Article in Journal
Have You Heard That—“GOSSIP”? Gossip Spreads Rapidly and Influences Broadly
Previous Article in Journal
Gender-Based Violence Risk Mitigation by Non-GBV Specialists Prior to and during COVID-19: A Global Survey of Knowledge, Attitudes and Practices of Humanitarian Practitioners
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 18
Issue 24
10.3390/ijerph182413388
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Public Perceptions around mHealth Applications during COVID-19 Pandemic: A Network and Sentiment Analysis of Tweets in Saudi Arabia

by
Samar Binkheder
1,†,
Raniah N. Aldekhyyel
1,†,
Alanoud AlMogbel
2,
Nora Al-Twairesh
3,4,
Nuha Alhumaid
5,
Shahad N. Aldekhyyel
5 and
Amr A. Jamal
6,7,*
1
Medical Informatics and E-learning Unit, Medical Education Department, College of Medicine, King Saud University, Riyadh 12372, Saudi Arabia
2
Freelance Research Assistant, Riyadh 12372, Saudi Arabia
3
Information Technology Department, College of Computer and Information Sciences, King Saud University, Riyadh 12372, Saudi Arabia
4
STC’s Artificial Intelligence Chair, King Saud University, Riyadh 11451, Saudi Arabia
5
College of Public Health and Health Informatics, King Saud bin Abdulaziz University for Health Sciences, Riyadh 14611, Saudi Arabia
6
Evidence-Based Health Care & Knowledge Translation Research Chair, King Saud University, Riyadh 11451, Saudi Arabia
7
Family & Community Medicine Department, College of Medicine, King Saud University, Riyadh 12372, Saudi Arabia
*
Author to whom correspondence should be addressed.
†
Samar Binkheder and Raniah N. Aldekhyyel contributed equally to this work.
Int. J. Environ. Res. Public Health 2021, 18(24), 13388; https://doi.org/10.3390/ijerph182413388
Submission received: 28 October 2021 / Revised: 7 December 2021 / Accepted: 14 December 2021 / Published: 20 December 2021
Browse Figures
Versions Notes

Abstract

:
A series of mitigation efforts were implemented in response to the COVID-19 pandemic in Saudi Arabia, including the development of mobile health applications (mHealth apps) for the public. Assessing the acceptability of mHealth apps among the public is crucial. This study aimed to use Twitter to understand public perceptions around the use of six Saudi mHealth apps used during COVID-19: “Sehha”, “Mawid”, “Sehhaty”, “Tetamman”, “Tawakkalna”, and “Tabaud”. We used two methodological approaches: network and sentiment analysis. We retrieved Twitter data using specific mHealth apps-related keywords. After including relevant tweets, our final mHealth app networks consisted of a total of 4995 Twitter users and 8666 conversational relationships. The largest networks in size (i.e., the number of users) and volume (i.e., the conversational relationships) among all were “Tawakkalna” followed by “Tabaud”, and their conversations were led by diverse governmental accounts. In contrast, the four remaining mHealth networks were mainly led by the health sector and media. Our sentiment analysis approach included five classes and showed that most conversations were neutral, which included facts or information pieces and general inquires. For the automated sentiment classifier, we used Support Vector Machine with AraVec embeddings as it outperformed the other tested classifiers. The sentiment classifier showed an accuracy, precision, recall, and F1-score of 85%. Future studies can use social media and real-time analytics to improve mHealth apps’ services and user experience, especially during health crises.

1. Introduction

The novel coronavirus disease (COVID-19), caused by severe acute respiratory coronavirus 2 (SARS-CoV-2 virus), has spread around the world causing a pandemic. In Saudi Arabia, the first COVID-19 confirmed case was reported on 2 March 2020, which was followed by a series of mitigation efforts imposed by the government. These efforts included the enforcement of social distancing, closure, and suspension of schools and universities, shopping malls, restaurants, coffee shops, public parks, sports leagues and competitions, and congregational and weekly Friday prayers [1]. On 23 March 2020, the Saudi Arabian government took extra measures by announcing a national wide curfew, which lasted about two months [2,3]. During the implementation of these precautionary measures, communication technology tools and social media platforms were among the main methods that authorities used to communicate with the public. The development of specific mobile health applications (mHealth apps) for public use was also a major pandemic response by the Saudi government [1,4].
The importance of using mHealth apps for improving public health and transforming health service delivery has been recognized by the world health organization (WHO) since 1998 [5,6,7]. The Saudi Arabian national e-health initiative has also recognized the importance of e-health by mentioning e-health as an enabler of quality and safe healthcare systems [8]. During the COVID-19 pandemic, and as a response to the public health crisis, many governments leveraged technologies that played a role in combating the COVID-19 pandemic. Such technologies were focused on developing e-health applications, which included the use of mobile integrated health care programs from home, mHealth apps, artificial intelligence (AI) and machine learning decision-making apps, robotic technologies, social networking apps, contact tracing apps, AI, and blockchain-enabled decentralized apps, and health and fitness apps [4,9,10,11,12,13].
The Saudi government, in collaboration with specialized organizations, has launched six mHealth apps, which were heavily used during the pandemic [12,14]. These mHealth apps were the official apps providing free services to the public. There were three mandated apps used for COVID-19 testing, isolation, and issuing electronic permits for movement, gathering, and work [14]. Three of the mHealth apps were specifically designed in response to the pandemic during the year 2020: “Tetamman” [15] (translated to English as “rest assured”), was launched in April 2020, “Tawakkalna” [16] (translated to English as “we trust”), launched in May 2020, and “Tabaud” [17] (translated to English as “social distancing”) was launched in June 2020. The remaining three mHealth apps were developed before the pandemic, which were designed to support telemedicine services and primary health clinic appointment scheduling: “Sehha” [18] (translated to English as “health”), launched in March 2017, “Mawid” [19] (translated to English as “appointments”), launched January 2018, and “Sehhaty” (translated to English “my health”), was launched in August 2019 [4,14,20]. However, some research studies reported usability barriers among mHealth apps users during the COVID-19 pandemic, including lack of knowledge, awareness, trust, and lower users’ satisfaction [12,21,22,23]. Therefore, a critical component for enhancing the meaningfulness of the implemented mHealth apps during the COVID-19 pandemic is to understand perceptions, experiences, and acceptance among their users [24,25].
Social media platforms are a great resource to collect information regarding user experiences and perceptions due to their popularity. Many people use the platform as a method to share their opinions, experiences, and ideas, especially during public health crises [26,27,28,29,30,31]. One of the most popular social media platforms is Twitter. Twitter can be seen as an important resource, which may be used by consumers to seek health-related information, engage in behavior change interventions, and share perceptions, and by researchers and health officials to track disease outbreaks and drug use [32,33,34,35]. For example, an infoveillance approach was used to support public health decision-makers during the 2009 H1N1 pandemic by providing near real-time content and sentiment analysis [36]. Furthermore, researchers have identified health-related keywords and hashtags, which can be used in analyzing tweets during public health pandemics or outbreaks. Signorini et al. collected tweets matching a set of 15 pre-specified search keywords including “flu”, “vaccine”, “tamiflu”, and “H1N1” and built a predictive model based on 1 million influenza-related tweets [33]. Using social media and understanding public conversations can help in gaining insights into the impact of various implemented measures during a crisis, including the use of eHealth and mHealth apps.
Social network analysis and sentiment analysis from social media data have also played a significant role in supporting stakeholders, such as governments, health authorities, and policymakers, in data-driven decision making during pandemics and outbreaks for timely responses during public health emergencies [26,27,28,29,30,31]. Social network analysis is an interdisciplinary research area that examines information flow, attitudes, and patterns gained from exchanged conversations and users characteristics [37]. For instance, Park et al. investigated information-sharing patterns during the COVID-19 pandemic by applying a network and content analysis of four networks, which suggested that the spread of information was faster in the Coronavirus network than in others [38]. Similarly, sentiment analysis can help decision-makers in understanding the sentiments of people about topics, such as medical information and public health, and to improve healthcare services [39].
Several studies applied either social network analysis or sentiment analysis to explore public perceptions toward some health-related topics, such as COVID-19 pandemic [40,41], 5G COVID-19 conspiracy theory and misinformation [31], vaccination [42,43], child physical activity [44], quality of care [45], and end-of-life care [46]. Studies that combined methodologies of social network analysis and sentiment analysis were generally lower than studies that used either social network analysis or sentiment analysis. For instance, Shams et al. experimented with the combination of sentiment analysis and social network analysis in building classification rules to represent customers’ preferences and needs and found that this combination helped in classifying products based on customers’ interests [47]. Hung et al. analyzed Twitter discussions and the related sentiments toward COVID-19 and concluded that Twitter discussions and sentiments can help officials with needed information during pandemics [35]. Furthermore, Yao et al. also applied both social network analysis and sentiment analysis to the construction safety research among the public [48].
At the time of this study, there was no research in Saudi Arabia that examined public perceptions about the use of mHealth apps during COVID-19 by probing Twitter data. Even though some published research studies have evaluated the perceptions of users on the use of mHealth apps during the pandemic, these studies relied only on surveys [23,49,50]. Unlike social media-based data collection, traditional survey-based data collection might suffer from a tendency to systematic bias due to underrepresenting the sample or fall into a systematic bias due to the survey design. Furthermore, surveys require individuals to recall their experiences and sentiments regarding a specific context, while social media collects data from real-time and real-world individual interactions on a larger scale [51]. Lastly, using the conjunction of social network analysis and sentiment analysis has not experimented with the context of mHealth apps. Therefore, understanding how these methods can help in gaining insights about users’ experiences from Twitter data is beneficial to improve the usability of mHealth apps.
Therefore, the aim of this study is to use Twitter as a source of data, to understand conversations and perceptions of users around the use of six mHealth apps during the COVID-19 pandemic by conducting a network and sentiment analysis of tweets. The specific objectives of this study are: (1) to examine the difference in communication network structure across the networks generated among the six mHealth apps included in our study; (2) to analyze the sentiment surrounding the six mHealth apps conversations; and (3) to evaluate the performance of a sentiment classifier using machine learning approaches.

2. Materials and Methods

2.1. Saudi’s mHealth Apps

The six mHealth apps (Table 1) included in our study were based on published research describing the country’s digital response to the COVID-19 pandemic [4,14,20]. These mHealth apps were used for (1) telehealth (“Sehha”), (2) digital screening (“Mawid”, “Sehhaty”), (3) follow-up (“Tetamman”), (4) Current health status and permits (“Tawakkalna”), and (5) COVID-19 contact notification (“Tabaud”).
“Sehha” is a telehealth app that provides online medical consultations, including audio-video medical consultations, AI technologies, and health assessment tools. The application “Mawid” is an e-appointment app used for scheduling appointments at one of the Ministry of Health (MOH)’s primary care centers and hospitals. During the pandemic, “Mawid” was used as a tool for virtual COVID-19 screening. The application “Sehhaty” provides e-health services and health information access, such as vital science, steps tracker, medications, and sick leaves. “Sehhaty” app was also used as a symptom checker tool and for scheduling COVID-19 tests either at Tetamman clinics (which differs from the Tetamman app) or drive-thru locations. “Tetamman” app is a preventative technological solution during the COVID-19 pandemic, which is mainly used during domestic isolation or quarantine based on exposure, such as travel. “Tetamman” app provides several services, including COVID-19 test results, symptoms check-up, educational content library, countdown indicator for isolation days, and alerts. The application “Tawakkalna” is a GPS-enabled app, which provides several services, including the COVID-19 latest health status (e.g., no record of infection, infected, exposed, arrived from abroad, and immune), requesting movement permits during curfew, and notifying the user of close contact of an infected person or any isolated areas. “Tabaud” app relies on Bluetooth technology to limit the spread of COVID-19 and notify users when exposure is detected. “Tabaud” helps in notifying people that came in contact with confirmed COVID-19 cases. People with confirmed COVID-19 tests can voluntarily share their results with people whom they had contacted during the past 14 days through the app.

2.2. Study Design and Data Collection

Our study involved a retrospective analysis of public conversations posted on Twitter [52], which mentioned one or more of the six mHealth apps during COVID-19. This study used two main methodological approaches: network analysis and sentiment analysis (Figure 1).

2.2.1. Data Collection for Network Analysis

Data were retrieved on 12 December 2020, using the Twitter search API embedded in NodeXL (Social Media Research Foundation) [53], a network analysis and visualization software package for Microsoft® Excel® that also provides a set of import tools to collect data from Twitter and other social media accounts. We retrieved Twitter users with recent tweets containing the keywords listed in Appendix A. These keywords represent terms related to mHealth apps used during COVID-19 in Saudi Arabia. Twitter limits the number of retrieved tweets to a maximum of 18,000 tweets per hour. With this limitation, a total of 73,208 tweets were retrieved. Appendix A describes the number of tweets included in our study data set. Data ranged from 9 April 2020 to 8 December 2020. We then manually reviewed the tweets to include those with the terms related to the mHealth apps for further analysis. Our final mHealth app networks consisted of a total of 4995 Twitter users and 8666 conversational relationships, which included “tweets”, “retweets”, “replies”, and “mentions”.

2.2.2. Data Collection and Annotation for Sentiment Analysis

We removed duplicate tweets based on the “unified Twitter ID” (a unique ID generated from Twitter associated with every single tweet), resulting in a total of 5048 tweets. To understand public opinions about the six mHealth apps, two annotators (S.B. and A.A.) manually annotated all the tweets in the data set. There were five annotation classes: (1) positive when the text expressed a positive opinion, (2) negative when the text expressed a negative opinion, (3) neutral when the text did not include any opinion such as facts and inquiries, (4) indeterminate when there was an opinion but there was unclear polarity or was unclear if there was an opinion [54], and (5) sarcasm when an opinion expressed the opposite in a sarcastic form and usually a negative opinion. Intercoder reliability scores were calculated using Cohen’s kappa, which is a statistical measure used for interrater reliability testing with the capability to account for chance agreement [55]. The ‘Kappa.test’ function in the ‘fmsb’ package [56] part of the statistical software R (version 4.0.3) was used [57]. There was a strong agreement between the two annotators over the five annotated classes (κ = 0.86, p < 0.001). Disagreements in annotations were resolved by discussion to reach a consensus between the two annotators. The final counts of tweets based on the five sentiment classes are shown in Table 2. After the annotation was completed, tweets that were labeled as indeterminate and sarcasm were excluded from the data set since they are expected to cause ambiguity for the machine learning classifier.
To build the sentiment classifier, classes within the data set should be balanced. However, we observed a high imbalance between the positive, negative, and neutral classes after completing the annotation in the initial data set. Therefore, data augmentation techniques were followed, which ensure the presence of sentiment polarity (positive, negative) in the tweets. This was accomplished by using a list of positive and negative Arabic words that were combined with the mHealth apps’ keywords shown in Appendix A and included as the search query using the Twitter search API. A total of 1847 additional tweets were collected for annotation. Having a high kappa score in the initial data set (Table 2), the additional set of tweets was annotated by one annotator (A.A.).
The final data set is shown in Table 3 after removing duplicate tweets and the indeterminate and sarcasm classes. The final data set was used to serve two purposes: first, to analyze the sentiment of each mHealth app using the human-in-the-loop approach, which is the process of integrating human knowledge and experience to facilitate or improve a prediction model; second, to train a machine learning classifier, which is an automated approach that inputs the training examples that were annotated by the humans and outputs the predicted classes [58], using the positive, negative, and neutral instances. Visualization was performed using Tableau Desktop (version 2021.4.1) [59].

2.3. Data Analysis

2.3.1. Social Media Network Analysis

A social network is composed of nodes and links where nodes represent users and edges represent conversations or interactions between users. We compiled users by subgroup using the Clauset–Newman–Moore cluster algorithm [60], and we visualized the networks using the Harel–Koren Fast Multiscale layout algorithm [61], which are commonly used in social communication research [31,38]. We created a network for each of the six mHealth apps, then calculated metrics for each network. To identify the influencing Twitter accounts in these conversations, the nodes were ranked using two measures: “betweenness centrality”, which is a measure of importance, and “PageRank”, which is a measure of influence [62]. We only included top Twitter accounts per measure and did not include personal accounts due to Twitter users’ privacy policies. All the tasks for the network analysis, including the calculation of network metrics and the visualization of networks, were performed using NodeXL [53].

2.3.2. Sentiment Classification

Sentiment classification is a text classification task that classifies text into polarity (positive, negative, neutral). The typical pipeline for such a task is text preprocessing, feature extraction, then the data and features are fed into the Machine Learning (ML) classifier. This is the conventional ML approach. With the recent abundance of textual data on the Web, deep learning approaches have been introduced, including context-independent models to represent words such as the Word2Vec model [63] and the context-dependent transformer-based models such as Bidirectional Encoder Representations from Transformers (BERT) [64]. These natural language processing models have the advantage that they do not require manual feature engineering (hand-driven features), and features can be learned automatically from the text. Therefore, to construct the sentiment classifier in this study, we experimented with three different approaches: classical ML with Term Frequency-Inverse Document Frequency (TF-IDF), Word2Vec model, and BERT.
The final data set (Table 3) was used to construct the sentiment classification model. First, the data set was preprocessed by removing URLs, mentions, and English characters using the natural language toolkit (NLTK), regular expression (regex), and the “genism” Python libraries. Arabic letter normalization was also performed. Then, the data set was split into 80/20 train/test splits.
For classification, we used the Support Vector Machine (SVM), one of the most used algorithms by researchers, and proved to provide high accuracy for Arabic sentiment classification [65]. The conventional TF-IDF, which relies merely on simple counts, was used in the first experiment with the SVM classifier. Then in the second experiment, we used the Word2Vec embeddings of AraVec [66] as features to represent the text of the tweets. Specifically, we used the version of AraVec that was constructed from Arabic Tweets using the Skip-gram model with dimension = 300. For the third experiment, we used AraBERT [67], an Arabic pre-trained language model. This experiment was performed using an Adam optimizer with a learning rate of 2 × 10−6 and a batch size of 8 for 10 epochs. The max sequence length was set to 256. The finetuning was performed by adding a softmax classification layer to the pre-built sentiment classification model.

3. Results

3.1. Comparing mHealth Conversations Networks

The topologies of the six networks are shown in Figure 2. The networks showed that the patterns of conversations (edges) between users (nodes) were similar across the users of the following three mHealth apps: “Sehhaty”, “Tawakkalna”, and “Tabaud”. Each of “Sehhaty”, “Tawakkalna”, and “Tabaud” network has a large cluster (community) that was connected with central nodes: Saudi Ministry of Health “@SaudiMOH” for “Sehhaty”, “@Tawakkalnaapp” for “Tawakkalna”, and “@TabaudApp” for “Tabaud”. Appendix B displays the top accounts that were found in the conversations related to the mHealth apps that were included in this study, ranked by two measures: “betweenness centrality” and “PageRank” [62]. Users with a high “betweenness centrality” score indicate that their position within the network, which is typically government accounts, allows them to become the gatekeeper or link between the communities [68]. “PageRank” high scores indicate that these users are of high importance and high influence. The users with the highest “betweenness centrality” and “PageRank” scores in “Tawakkalna” and “Tabaud” were found to be associated with education officials (e.g., ministry of education and universities), governments, hajj and umrah, media, and health-related governments or hospitals. In comparison, users with the highest “betweenness centrality” and “PageRank” scores in “Sehha”, “Mawid”, “Sehhaty”, and “Tetamman” were found to be majorly associated with media and health-related governments or hospitals accounts.
Table 4 shows the conversational relationship characteristics among users within each network. Unique edges mean fewer overlapping relationships, which reflect an instant community with many one-time conversations [38]. The network with the highest frequency of unique edges was “Sehha” followed by “Mawid”, then “Tetamman”. The networks with high frequencies of edges with duplicates indicate that conversations were continuously and more frequently exchanged among users. These networks were “Tabaud”, followed by “Tawakkalna” then “Sehhaty”.
Self-loops, known as a conversation thread starting and ending with the same user [38], was found to be more frequent in “Mawid”, followed by “Tetamman” then “Sehha” networks. The lowest frequencies of self-loops were observed in “Tawakkalna”, “Sehhaty”, and “Tabaud” networks indicating more interactions with others in conversations. “Mawid” network also had the largest percentages of isolates, users with zero connections, followed by the “Sehha” network.
Table 5 summarizes the overall metrics of each network. The “geodesic distance” is the value that reflects the shortest path between two users (nodes); also, within a network, the maximum geodesic distance (diameter) is the furthest distance [38]. “Sehha”, “Tetamman”, then “Tabaud” networks have the smallest diameter, which might indicate that users involved in these conversations communicated more frequently with each other with a faster spread of information than other networks. “Tawakkalna” and “Sehhaty” networks have the largest diameter. For the component analysis, which means that for every pair of users, there is a path, the results show that users from “Tawakkalna” had the largest numbers of connected components. “Tawakkalna” and “Tabaud” have the “largest chat rooms” [38] with the largest number of users (nodes) in a connected component and maximum edges in a connected component, followed by “Sehhaty”.
Modularity is “a measure of the network structure designed to measure the strength of division of a network into modules” [69]. The highest modularity value indicates that the strength of the connection within a sub-network (community A) is higher than across sub-networks (community B). Low modularity values indicate that the clusters are well-defined and that the users within the clusters rarely move to another cluster [38]. The modularity values of the “Sehha” and “Mawid” networks were the highest among the networks; in contrast, the modularity values of the “Tawakkalna” and “Tabaud” networks were the lowest among other networks.

3.2. Sentiment Analysis of Conversations Surrounding mHealth Apps

The manual annotation resulted in identifying conversations associated with positive, negative, or neutral sentiments (Table 2). Tweets with a positive sentiment focused on appreciation, positive opinions, and expressions around government trust. Negative tweets covered weaknesses, issues faced with apps, negative opinions, and negative psychological impact (Appendix C). Neutral tweets included facts or information pieces, neutral suggestions, and general inquires. Table 6 shows examples of positive, neutral, and negative sentiments per each mHealth app. As we evaluated the sentiment of tweets related to these networks, we found that most tweets were neutral and focused more on users seeking information and asking questions about the apps rather than providing their opinions. Within each mHealth app sentiment, the highest frequency of positive opinions was found for “Tetamman” (35.9%). The mHealth apps that had more positive sentiments than negative were: “Sehha”, “Tetamman”, and “Tabaud”. The mHealth apps that had more negative than positive sentiments were: “Mawid”, “Sehhaty”, and “Tawkkalna”.

3.3. Performance of an Automated Sentiment Classifier

Figure 3 shows the number of tweets after data augmentation for the sentiment classifier, which included unique tweets after processing. There was an increase in the numbers of positive and negative tweets across all apps except “Tetamman”. The app “Tawakkalna” remained the same, with more negative tweets than positive. Table 7 shows the performance of the three classifiers, as explained in Section 2.3.2. We observed that the best performing classifier was the SVM with AraVec embeddings (F1-score = 0.85) in contrast to SVM tf-idf (F1-score = 0.84) and AraBERT (F1-score = 0.80).

4. Discussion

4.1. Major Findings

Our study presented a novel research context by using social media conversations posted on Twitter to assess public perceptions on using mHealth apps during the COVID-19 pandemic. Two methodological approaches were used, which are the social network analysis and the sentiment analysis. Twitter data were used to identify the networks and sentiments of the public toward six mHealth apps, which were “Sehha”, “Mawid”, “Sehhaty”, “Tetamman”, “Tawakkalna”, and “Tabaud”. The social network analysis identified similar patterns in conversations among “Sehhaty”, “Tawakkalna”, and “Tabaud”. On the other hand, similar patterns were found among the following networks: “Sehha”, “Mawid”, and “Tetamman”. The apps “Tawakkalna” and “Tabaud” were the largest networks in size (the number of users) and volume (the number of conversational relationships) among all, and their conversations were led by a variety of governmental accounts. In comparison, the apps “Sehha”, “Mawid”, “Sehhaty”, and “Tetamman” networks were mainly led by a health sector or/and media. The sentiment analysis showed that conversations around the six mHealth apps were majorly neutral. Among all the six mHealth apps included in this study, we found that conversations about “Tetamman” were the highest frequency in positive sentiments. For the automated sentiment classifier, we used the SVM with AraVec embeddings as it outperformed other tested classifiers. The sentiment classifier showed an accuracy, precision, recall, and F1-score of 85%.
Overall, the social network analysis identified similar patterns in conversations among “Sehhaty”, “Tawakkalna”, and “Tabaud”. These mHealth apps had the highest number of conversations, indicating their significant role during the pandemic, and were heavily used by the public for COVID-19 health status, tracing of cases, and exposure notifications. A previous study has also reported that “Tawakkalna” and “Tabaud” mHealth apps were among the highest in the number of users during the pandemic [12]. The fact that “Tawakkalna” and “Tabaud” had distinguished Twitter user accounts while other apps did not might have also contributed to the highest number of conversations surrounding them. When examining the main conversational role-players within the networks, conversations around “Tawakkalna” and “Tabaud” were led by various governmental accounts, including education, hajj and umrah, the health sector, and media channels. This variability is most likely a result of the apps’ medical-related features and the regulations enforced by the Saudi government to combat the spread of COVID-19 through the mandatory use of “Tawakkalna” when entering universities, hospitals, workplaces, shopping malls, government buildings, and other public places.
When compared with other networks featured that had more app functionalities, the conversations of “Sehha”, “Mawid”, and “Tetamman” networks lacked interactions. A possible reason for low interactions and conversations in Twitter around “Sehha” and “Mawid” apps may be the misperceptions among the public regarding the access to MOH’s apps for only MOH patients [23]. Another reason for low interactions, some studies that surveyed physicians about their perspective on telehealth during the pandemic showed that they were concerned about the following: technological barriers, diagnostic reliability, cultural and social factors, lack of face-to-face interactions, and lack of a clear telemedicine legal framework. In addition, physicians tend to use WhatsApp® and Zoom more than the “Sehha” app [50,70,71]. Therefore, more campaigns targeting the eligibility of these mHealth apps are suggested to increase awareness about their use [23]. Lastly, even though “Tetamman” was one of the mHealth apps that were launched during the pandemic, the conversations were not as extensive as those by “Tawakkalna” and “Tabaud”. A plausible reason for this low rate of conversations is that many of the services provided by “Tetamman” were already offered by “Sehhaty” and “Tawakkalna”. This has also been found in our sentiment analysis findings where users suggested a need to integrate mHealth apps into one fully featured app.
Several findings of this study were derived from the sentiment analysis of conversations around the use of the six mHealth apps. First, the majority of conversations around mHealth apps were neutral. The dominance of neutral tweets was also reported in other similar studies depending on the research topic domains, where some topics can be more controversial than others [72,73,74]. The neutral conversations provided information or facts, neutral suggestions, and general inquires, which may indicate that Twitter can be used as an effective real-time communication platform to answer users’ questions and tackle their concerns. This is in line with many other studies that showed the use of Twitter by government officials during pandemics to communicate with the public during health crisis times [30,75,76,77]. Second, the findings also indicated several positive conversations that were relevant to the appreciation of the mHealth apps’ services and features, in addition to the positive user experiences surrounding the use of these apps. Other positive conversations were more of statements that indicated gratitude and appreciation toward healthcare providers. Many of the communication campaigns on social media platforms, and other communication outlets, which were led by the MOH during the pandemic, were focused on lifting the spirits of the public in the fight against the pandemic.
Third, when examining the type of issues raised by the public indicated by the negative sentiments, several were related to the recent digital transformation of many Saudi government services and the adaptation of mHealth and eHealth apps to facilitate health-related services, as stated by Han et al. [78]. Concerns and lack of familiarity and digital literacy by the public are expected at these early stages of adaptation [79,80]. Furthermore, the replication and overlapping of features between the mHealth apps have been a concern that was raised frequently by the public. Such duplication in services should be avoided as it could lead to confusion and avoidance of using these mHealth apps altogether, which may have contributed to the negative experience. Integrating similar features between these mHealth apps into one app may overcome these issues. Other negative conversations were related to the technical and accessibility issues experienced by users. All the mHealth apps described in our study require the use of Wi-Fi or a cellular connection, an issue with mHealth apps in general [81]. It is vital to consider this limitation when mandating the public to use a specific mHealth app, given the variability and differences in the availability of smartphones and Internet connections among the public. Another critical element raised in these conversations was the psychological impact that may be related to the use of such apps. The use of mHealth apps to track COVID-19 cases and their negative implications on the public has been a topic addressed by many researchers. Examples of these implications include increased levels of anxiety when users receive a COVID-19 exposure notification [81,82,83]. Privacy concerns have also been raised about tracking and tracing features, specifically about “Tetamman”, “Tawakkalna”, and “Tabaud”, similar to what has been reported by other mHealth apps [84,85,86]. The benefits and drawbacks of mHealth systems that raise issues with consumer privacy, must be examined critically by all stakeholders to ensure public by in and trust is not jeopardized.
To build the sentiment classifier for our data set, we experimented with the performance with different approaches. Overall, the results showed that AraVec embeddings performed better than AraBERT. This might be because AraVec embeddings were pre-trained on tweets compared to AraBERT that was pre-trained on Arabic Wikidumps and other Arabic corpora. Unlike different text sources on the Web, the nature of text in tweets is known to be informal with different characteristics. Consequently, the SVM with word embeddings sentiment classifier performed well, and it can be used in automating the detection of the sentiment of conversations around mHealth apps.

4.2. Theoretical Contributions

Social network theory, “a set of general assumptions about a domain of study and the methods suitable for its investigation”, is concerned about the relations between individuals who create, share, and disseminate knowledge as a unit of analysis [87]. The social network theory contributes to building explanations from patterns of social relations between individuals within online communities [88], where online social interactions are very similar to face-to-face communities [89]. Moukarzel et al. stated that the analysis of networks combined with other methods can provide opportunities in gaining a deeper understanding of real-time interactions and relationships among the public within a certain context [89]. This study illustrated that there is value gained from integrating and combining social network analysis with sentiment analysis to the context of mHealth apps. This integration has the potential in enabling researchers to evaluate behaviors among the public surrounding the use of mHealth apps real-world discussions.
Twitter is a potential source for big data. Unlike the traditional use of Twitter analytics (e.g., number of likes and retweets), recent directions grounded in social network theory enabled the analysis of Twitter data in robust, meaningful, and theoretically grounded methods [89]. The relationship of usability and sociability [90] within the context of mHealth app online communities is new. This relationship provides an opportunity to understand users’ needs, which may improve the usability, design, and functionalities of mHealth apps. The knowledge gained in this study has also enabled us to identify some practical strategies and recommendations (Section 4.3.) to support and improve the mHealth apps and their online communities. Unlike other cross-sectional studies that focused on collecting users’ perceptions through a survey tool [23,49,50], this study used social media as it reflects real-world interactions among people when a real-time response is needed.

4.3. Practical Implications

The study findings showed that assessing the perceptions of users surrounding the use of mHealth apps during the COVID-19 pandemic through analyzing public opinions posted on social media accounts is crucial in providing direction for future health crises and improving mHealth app services. This research supports the fact that the development of mHealth technologies requires collaborative efforts between end-users and implementers to meet person-centered needs and improve the users’ experiences in different ways. On a broader scale, health authorities and organizations can implement real-time social network analysis and sentiment analysis to automate the analysis of public perceptions and opinions related to mHealth apps. By using social networks and sentiment analyses, the results of this study provide insights into public behavior and acceptability toward the six mHealth apps included in our study. Box 1 shows a list of recommendations and lessons learned based on our study’s analyses of Twitter conversations around six mHealth apps that can be used to improve mHealth services during a pandemic.
Box 1. A list of recommendations and lessons learned that can be used to strategically improve mHealth apps services during a pandemic.
  • Using social media data as a source and a connection tool for understanding public perceptions, opinions, and acceptability around mHealth apps can serve as a real-time communication approach during pandemics to answer questions of the public and tackle users’ concerns;
  • Health authorities and organizations can implement real-time sentiment classifiers to automate the analysis of public perceptions and opinions about mHealth apps;
  • Establishing coordinated efforts among governmental entities in developing public mHealth apps, guided by the country’s digital health strategy, may have the potential to increase a positive user experience and lessen the negative experience associated with the use of mHealth apps during a pandemic;
  • Increasing campaigns targeting the public regarding mHealth apps is suggested to increase awareness about these apps;
  • Having an official Twitter account associated with a mHealth app, which is led by experts, is recommended to engage the public in conversations related to the use of the app and would serve as a platform for information distribution;
  • Enhancing mHealth apps with pandemic-related information and services may increase their use by the public (e.g., telemedicine, COVID-19 testing, health status, vaccination updates, and contact notifications);
  • Avoiding duplicate features among apps or similar app names by integrating mHealth apps with similar features into one app may increase the use among the public and positive experiences;
  • For mandated mHealth apps, app developers should consider people with limited access to Internet services, thus providing the features of the app offline;
  • Negative sentiments are likely to be driven by psychological impact, lack of familiarity and digital literacy, and technical and accessibility issues. Such sentiments may be alleviated by considering different age groups, increasing accessibility, designing educational material, and creating connection channels with the public to address their concerns;
  • Governmental communication efforts toward non-English expatriates were seen by the MOH, given that the cooperation of expatriates living in Saudi Arabia played an important role in COVID-19 mitigation and control measures. Designing mHealth apps in different languages other than English and Arabic may enhance the positive user experience among this population.

4.4. Limitations and Future Research

This work does not stand without limitations. First, the Twitter API limits the number of tweets that were retrieved. However, the API limits did not affect the applicability and performance of the sentiment classifier. Second, the extracted Twitter data might reflect Twitter users, but not all where young users tend to use Twitter more than the elderly [23,38]. Third, we extracted data from the dominant social media platform in Saudi (Twitter). Other platforms such as Instagram and Facebook may have included other sentiment topics, which were not captured in our study. Lastly, this study focused on a specific mHealth app population; users who posted their opinions on Twitter in the Arabic language. Other users who posted their opinion in English or used mHealth apps developed by private entities may have different opinions regarding the use of mHealth apps during COVID-19.
There are several opportunities for future research. First, future studies can combine more than one analytical approach to analyze users’ opinions by combining multiple data sources, such as surveys and social media. For example, surveys can be administered to mHealth apps’ users to evaluate the usability of these applications, demographic factors, and the challenges that users might face. A similar approach was used by Twitter polls to gain public insights on telemedicine [91] and can be applied to other mHealth apps. Second, future analyses can include topic modeling or content analysis for further quantitative understanding of the discussed topics, concerns, and perceptions. For instance, it can focus on standardized content analysis to assess the positive and negative tweets and explore how negative sentiments can be alleviated. Third, sentiment analysis can be combined with other Twitter metrics, such as tweets’ likes, retweets, and followers for each topic, to calculate the sentiment interaction rate [28] for each mHealth apps. Fourth, conducting an in-depth analysis of mHealth public perceptions by aggregating Twitter users’ sentiments to identify factors that might have driven the opinion change of users and identify communication structures, styles, or sources that increase positive sentiment or alleviate concerns. Fifth, future studies may focus on conducting social network and sentiment analysis centering on the general community’s sentiment by excluding tweets posted by key nodes, such as the apps’ official Twitter accounts and or governmental entities. Finally, a future study can focus on measuring the perceptions of the public around mHealth apps using similar approaches discussed in this study on a larger scale of social media data.

5. Conclusions

This study showed that social media could be used as a complementary data source and a connection tool to improve user experience and to understand public perceptions about the use of mHealth apps during a pandemic. Furthermore, there is value gained from integrating and combining social network analysis with sentiment analysis in the context of mHealth apps to enhance the understanding of their usability from real-world discussions. The network analysis showed that “Sehhaty”, “Tawakkalna”, and “Tabaud” had similar network patterns with more interactions in conversations than other networks. “Tawakkalna” and “Tabaud” were the most extensive networks among all. The sentiment analysis approach showed that most Twitter conversations around the six mHealth apps were neutral. The sentiment classifier performed well. Therefore, health authorities and organizations can implement real-time sentiment classifiers to automate the analysis of public perceptions and opinions about mHealth apps. In addition, it is imperative for developers to decrease the number of apps with shared or similar functionalities as it might lead to user confusion. Efforts should focus on enhancing positive experiences with these mHealth apps, which would hopefully lead to increased positive opinions reflected in social media conversations. Having distinguished Twitter user accounts for high-impact mHealth apps is recommended, as it provides quick access for apps’ users to express their opinions, concerns, inquires, and recommendations. What is even more important is for these accounts to be actively led by experts who interact with the public and facilitate their adaptation of such mHealth apps. The list of recommendations and lessons learned derived from Twitter conversations around six mHealth apps in this study can be used to strategically improve the user experience of the mHealth apps during pandemics.

Author Contributions

Conceptualization, S.B., R.N.A., and A.A.J.; Data curation, S.B., R.N.A., A.A., and S.N.A.; Formal analysis, S.B., R.N.A., A.A., N.A.-T., and N.A.; Funding acquisition, A.A.J.; Investigation, S.B., R.N.A., A.A., and N.A.-T.; Methodology, S.B., R.N.A., N.A.-T., and A.A.J.; Resources, A.A.J.; Supervision, A.A.J.; Validation, S.B. and A.A.; Visualization, S.B., R.N.A., and N.A.-T.; Writing—original draft, S.B., R.N.A., and N.A.-T.; Writing—review and editing, S.B., R.N.A., A.A., N.A.-T., N.A., S.N.A., and A.A.J. All authors have read and agreed to the published version of the manuscript.

Funding

This work is funded by the Vice Deanship of Scientific Research Chairs, Deanship of Scientific Research, King Saud University.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board of King Saud University (protocol code E-20-5323 on 29 October 2020).

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

The authors are grateful to the Deanship of Scientific Research, King Saud University, for funding provided through the Vice Deanship of Scientific Research Chairs.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table
Table A1. Collected tweets summary for network analysis.
Table A1. Collected tweets summary for network analysis.
mHealth AppKeywordsArabic Keywords TranslationTotal CollectedTotal Included
Sehha تطبيق صحة Sehha application77561
Mawid تطبيق موعد Mawid application22,766206
خدمة موعد Mawid service
موعد Mawid
Sehhaty صحتي Sehhaty11,254620
مركز تأكد Takkad center
تطبيق صحتي Sehhaty application
مراكز_تأكد Takkad centers
Tetamman عيادات تطمن Tetamman clinics608504
مراكز تطمن Tetamman centers
عيادة_تطمن Tetamman clinic
برنامج تطمن Tetamman program
تطبيق تطمن Tetamman application
Tawakkalna@TawakkalnaApp-26,0035755
Tawakkalnaapp-
توكلنا Tawakkalna
تطبيق توكلنا Tawakkalna application
Tabaud@TabaudApp-11,8021520
TabaudApp-
Tabaud-
تباعد Tabaud
تطبيق تباعد Tabaud application
Total73,2088666

Appendix B

Table
Table A2. Top accounts per mHealth app based on two measures: “Betweenness Centrality” and “PageRank”.
Table A2. Top accounts per mHealth app based on two measures: “Betweenness Centrality” and “PageRank”.
mHealth AppTop Accounts (Betweenness Centrality)Account TypeTop Accounts (PageRank)Account Type
Sehha@tfrabiah (11)
@ask_madinah1 (9)
@mygovsa (6)		Minister of Health
Public profile
Government (Website)		@mygovsa (2.619)
@tfrabiah (1.852)		Government (Website)
Minister of Health
Mawid@saudimoh (444.5)
@saudimoh937 (420.5)
@saudinews50 (164)
@tfrabiah (53.5)
@ask_madinah1 (42)		Government (Health)
Government (Health Call Center)
Media
Minister of Health
Public profile		@saudimoh937 (6.7)
@saudimoh (5.034)		Government (Health Call Center)
Government (Health)
Sehhaty@saudimoh (46452)
@saudimoh937 (9329)
@tfrabiah (1221)		Government (Health)
Government (Health)
Minister of Health		@saudimoh (118.867)
@saudimoh937 (14.447)
@kfshrc (5.749)		Government (Health)
Government (Health Call Center)
Public hospital
Tetamman@sparegions (887)
@saudimoh937 (531)
@saudimoh (431)
@tfrabiah (183)
@saudiatv (162)		Media
Government (Health Call Center)
Government (Health)
Minister of Health
Media		@sparegions (17.479)
@saudimoh937 (6.436)
@saudimoh (6.250)
@makkahregion (6.062)
@ajelnews24 (5.817)
@hfrmoh (4.901)
@joufhealth (4.901)		Media
Government (Health Call Center)
Government (Health)
Government (Media)
Media
Government (Health)
Government (Health)
Tawakkalna@tawakkalnaapp (3483315.84)
@tfrabiah (67779.41)
@sdaia_sa (55246.5)
@mohu_csc (43380.18)
@hajministry (32998.81)
@absher (32924.79)
@moe_gov_sa (26310)		Mobile app
Minister of Health
Government (Organization)
Government (Religious—Hajj and Umrah)
Government (Religious—Hajj and Umrah)
Government (Online services)
Government (Education)		@tawakkalnaapp (1110.747)
@jazanuniversity (21.890)
@moe_ual (14.540)
@sdaia_sa (12.332)
@hajministry (8.180)
@ask_madinah1 (7.180)
@tfrabiah (7.1300)
@mohu_csc (6.730)
@kfshrc_j (6.475)
@_ksu (6.276)
@ask_almadinah30 (6.209)		Mobile app
Public university
Government (Education)
Government (Organization)
Government (Ministry)
Public account
Minister of Health
Government (Religious—Hajj and Umrah)
Public hospital
Public university
Public account
Tabaud@tabaudapp (164419.33)
@jazanuniversity (946)
@kauweb (109.5)
@_ksu (66)		Mobile app
Public university
Public university
Public university		@tabaudapp (257.666)
@jazanuniversity (20.757)
@_ksu (6.276)		Mobile app
Public university
Public university

Appendix C

Table
Table A3. Examples of tweets for positive and negative sentiment cases.
Table A3. Examples of tweets for positive and negative sentiment cases.
PositiveNegative
  • Appreciation:
“Thank God Tawkkalna app has been launched. I can go to the mall.”
“الحمدلله فتح تطبيق توكلنا اقدر انزل للسوق”
  • Identified weaknesses:
“I wanted to do a swab test, and I remember searching between Mawid, Sehhaty, and Tetamman apps, I got confused by the abundance of applications one is enough.”
“كنت بعمل مسحه واتذكر جلست ادور بين تطبيق موعد وصحتي وتطمن لخبطونا بكثرة التطبيقات واحد يكفي”
  • Positive opinion:
“I’m astonished by @SaudiMOH amount of effort. I booked an appointment for the Corona test from the Sehhaty app. The entire trip, including the test, took only 18 min. A very great thing, thank you to the Ministry of Health. Honestly, I was not aware of the facilitation, until today”
“ انا منبهر من حجم جهود @SaudiMOH حجزت موعد فحص كورونا من تطبيق صحتيالمشوار كامل بما فيه الفحص استغرق ١٨ دقيقة فقطشيء عظيم جدا شكرا وزارة الصحة ما كنت مطلع على التسهيلات للأمانة لأني ما جربت حتى اليوم”
  • Issues faced with apps:
“Sehhay app is not responding, the activation code to reset the password is not working, the code is not sent”
“تطبيق صحتي معلق، رمز التفعيل لإعادة تعين كلمه السر لا يعمل، الرمز لا يرسل “
  • Trust expressions around government and healthcare practitioners:
“Tetamman—is an excellent app. Great service, organization and accurate appointments, loyal health practitioners. May Allah protect my country and keep you well and safe.”
“تطمن- تطبيق ممتاز. خدمة رائعة تنظيم ومواعيد مضبوطة.. ممارسين صحيين مخلصين. حفظك الله يا بلادي ودمتِ بخير وامان”
  • Negative opinion:
“A painful sight when you see an elderly man, a woman, or a child leaves the health center without treatment ... why? Not having access to the internet on their mobile or not having a mobile to access the Tawakkalna app ... they do not know that there are people who can’t afford it. For most the internet is only at home.”
“منظر مؤلم عندما ترى رجل مسن أو امرأة او طفل يخرج من المركز الصحي دون علاج ... لماذا؟ ليس لديه نت في جواله أو ليس لديه جوال اصلا بدعوى تطبيق توكلنا ... لا يعلموا أن فيه ناس عائشة بالكفاف الأغلب النت في البيت. “
  • Psychological impact:
“#Tabaud_app I do not like anxiety and I expect anxiety harms human health ... We depend on Allah and from my point of view, psychological aspects must be considered in any app, especially regarding human health. How this can be possible, and there is no accurate phone device that is capable of giving an accurate location without chances of error.”
“#تطبيق_تباعد انا ما أحب القلق واتوقع القلق يضر بصحة الانسان ... توكلنا على الله ومن وجهة نظري يجب مراعاة الجوانب النفسية في اي تطبيق خصوصا ما يتعلق بصحة الانسان كيف لا وأخطاء تحديد المواقع واردة ولا يوجد جهاز هاتف دقيق يعطي دقة في تحديد المواقع لا يمكن وجود خطأ معها”

References

  1. Algaissi, A.A.; Alharbi, N.K.; Hassanain, M.; Hashem, A.M. Preparedness and response to COVID-19 in Saudi Arabia: Building on MERS experience. J. Infect. Public Health 2020, 13, 834–838. [Google Scholar] [CrossRef]
  2. Alrashed, S.; Min-Allah, N.; Saxena, A.; Ali, I.; Mehmood, R. Impact of lockdowns on the spread of COVID-19 in Saudi Arabia. Inform. Med. Unlocked 2020, 20, 100420. [Google Scholar] [CrossRef]
  3. Yezli, S.; Khan, A. COVID-19 social distancing in the Kingdom of Saudi Arabia: Bold measures in the face of political, economic, social and religious challenges. Travel Med. Infect. Dis. 2020, 37, 101692. [Google Scholar] [CrossRef]
  4. Hassounah, M.; Raheel, H.; Alhefzi, M. Digital Response During the COVID-19 Pandemic in Saudi Arabia. J. Med. Internet Res. 2020, 22, e19338. [Google Scholar] [CrossRef] [PubMed]
  5. World Health Organization. eHealth at WHO. Available online: https://www.who.int/ehealth/about/en/ (accessed on 22 March 2021).
  6. mHealth: New Horizons for Health Through Mobile Technologies; World Health Organization: Geneva, Switzerland, 2011; pp. 8, 112.
  7. Rowland, S.P.; Fitzgerald, J.E.; Holme, T.; Powell, J.; McGregor, A. What is the clinical value of mHealth for patients? npj Digit. Med. 2020, 3, 4. [Google Scholar] [CrossRef]
  8. The Ministry of Health National E-Health Strategy. Available online: https://www.moh.gov.sa/en/Ministry/nehs/Pages/How-we-developed-the-eHealth-Strategy.aspx (accessed on 22 March 2021).
  9. Bokolo, A.J. Application of telemedicine and eHealth technology for clinical services in response to COVID-19 pandemic. Health Technol. 2021, 11, 359–366. [Google Scholar] [CrossRef] [PubMed]
  10. Alwashmi, M.F. The Use of Digital Health in the Detection and Management of COVID-19. Int. J. Environ. Res. Public Health 2020, 17, 2906. [Google Scholar] [CrossRef] [PubMed]
  11. Jabarulla, M.; Lee, H.-N. A Blockchain and Artificial Intelligence-Based, Patient-Centric Healthcare System for Combating the COVID-19 Pandemic: Opportunities and Applications. Healthcare 2021, 9, 1019. [Google Scholar] [CrossRef]
  12. Alghamdi, S.M.; Alqahtani, J.; Aldhahir, A.M. Current status of telehealth in Saudi Arabia during COVID-19. J. Fam. Community Med. 2020, 27, 208–211. [Google Scholar] [CrossRef]
  13. Petracca, F.; Ciani, O.; Cucciniello, M.; Tarricone, R. Harnessing Digital Health Technologies During and After the COVID-19 Pandemic: Context Matters. J. Med. Internet Res. 2020, 22, e21815. [Google Scholar] [CrossRef] [PubMed]
  14. Apps for your Health (COVID-19). Available online: https://covid19awareness.sa/en/apps-for-your-health-2 (accessed on 1 February 2021).
  15. E-Services. (Tetamman) App. Available online: https://www.moh.gov.sa/en/eServices/Pages/Rest-assured.aspx (accessed on 20 November 2020).
  16. Tawakkalna. Available online: https://ta.sdaia.gov.sa/en/index (accessed on 20 November 2020).
  17. Tabaud. Available online: https://tabaud.sdaia.gov.sa/IndexEn (accessed on 20 November 2020).
  18. E-Services. Sehha App. Available online: https://www.moh.gov.sa/eServices/Pages/sehha-app.aspx (accessed on 20 November 2020).
  19. E-Services. (Mawid) Service. Available online: https://www.moh.gov.sa/en/eServices/Pages/cassystem.aspx (accessed on 20 November 2020).
  20. MOH Apps for Smartphones. Available online: https://www.moh.gov.sa/en/Support/Pages/MobileApp.aspx (accessed on 22 March 2021).
  21. Alshammari, F.; Hassan, S.U.-N. Perceptions, Preferences and Experiences of Telemedicine among Users of Information and Communication Technology in Saudi Arabia. J. Health Inform. Dev. Ctries. 2019, 13. [Google Scholar]
  22. Alruzaiza, S.A.; Mahrous, R.M. Assessment of knowledge, attitude, and practice on level of awareness among pediatric emergency department visitors-Makkah City, Saudi Arabia: Cross-sectional study. Int. J. Psychosoc. Rehabil. 2020, 24, 5186–5202. [Google Scholar]
  23. Alharbi, A.; Alzuwaed, J.; Qasem, H. Evaluation of e-health (Seha) application: A cross-sectional study in Saudi Arabia. BMC Med. Inform. Decis. Mak. 2021, 21, 103. [Google Scholar] [CrossRef]
  24. Shati, A. Mhealth Applications Developed by the Ministry of Health for Public Users in KSA: A Persuasive Systems Design Evaluation. Health Inform.-Int. J. 2020, 9, 1–13. [Google Scholar] [CrossRef]
  25. Pappot, N.; Taarnhøj, G.A.; Pappot, H. Telemedicine and e-Health Solutions for COVID-19: Patients’ Perspective. Telemed. e-Health 2020, 26, 847–849. [Google Scholar] [CrossRef]
  26. MacKay, M.; Colangeli, T.; Gillis, D.; McWhirter, J.; Papadopoulos, A. Examining Social Media Crisis Communication during Early COVID-19 from Public Health and News Media for Quality, Content, and Corresponding Public Sentiment. Int. J. Environ. Res. Public Health 2021, 18, 7986. [Google Scholar] [CrossRef] [PubMed]
  27. Alamoodi, A.H.; Zaidan, B.B.; Zaidan, A.A.; Albahri, O.S.; Mohammed, K.I.; Malik, R.Q.; Almahdi, E.M.; Chyad, M.A.; Tareq, Z.; Albahri, A.S.; et al. Sentiment analysis and its applications in fighting COVID-19 and infectious diseases: A systematic review. Expert Syst. Appl. 2021, 167, 114155. [Google Scholar] [CrossRef]
  28. Abd-Alrazaq, A.; Alhuwail, D.; Househ, M.; Hamdi, M.; Shah, Z. Top Concerns of Tweeters During the COVID-19 Pandemic: Infoveillance Study. J. Med. Internet Res. 2020, 22, e19016. [Google Scholar] [CrossRef] [Green Version]
  29. Srivastava, K.C.; Shrivastava, D.; Chhabra, K.G.; Naqvi, W.; Sahu, A. Facade of media and social media during COVID-19: A review. Int. J. Res. Pharm. Sci. 2020, 11, 142–149. [Google Scholar] [CrossRef]
  30. Long, Z.; Alharthi, R.; El Saddik, A. NeedFull–A Tweet Analysis Platform to Study Human Needs During the COVID-19 Pandemic in New York State. IEEE Access 2020, 8, 136046–136055. [Google Scholar] [CrossRef] [PubMed]
  31. Ahmed, W.; Vidal-Alaball, J.; Downing, J.; Seguí, F.L. COVID-19 and the 5G Conspiracy Theory: Social Network Analysis of Twitter Data. J. Med. Internet Res. 2020, 22, e19458. [Google Scholar] [CrossRef] [PubMed]
  32. Pershad, Y.; Hangge, P.T.; Albadawi, H.; Oklu, R. Social Medicine: Twitter in Healthcare. J. Clin. Med. 2018, 7, 121. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  33. Signorini, A.; Segre, A.M.; Polgreen, P.M. The Use of Twitter to Track Levels of Disease Activity and Public Concern in the U.S. during the Influenza A H1N1 Pandemic. PLoS ONE 2011, 6, e19467. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  34. Tomazic, T.; Jerkovic, O.S. Online Interventions for the Selective Prevention of Illicit Drug Use in Young Drug Users: Exploratory Study. J. Med. Internet Res. 2020, 22, e17688. [Google Scholar] [CrossRef] [PubMed]
  35. Hung, M.; Lauren, E.; Hon, E.S.; Birmingham, W.C.; Xu, J.; Su, S.; Hon, S.D.; Park, J.; Dang, P.; Lipsky, M.S. Social Network Analysis of COVID-19 Sentiments: Application of Artificial Intelligence. J. Med. Internet Res. 2020, 22, e22590. [Google Scholar] [CrossRef]
  36. Chew, C.; Eysenbach, G. Pandemics in the Age of Twitter: Content Analysis of Tweets during the 2009 H1N1 Outbreak. PLoS ONE 2010, 5, e14118. [Google Scholar] [CrossRef]
  37. Himelboim, I. Social Network Analysis (Social Media). In The International Encyclopedia of Communication Research Methods; John Wiley & Sons: Hoboken, NJ, USA, 2017; pp. 1–15. [Google Scholar]
  38. Park, H.W.; Park, S.; Chong, M. Conversations and Medical News Frames on Twitter: Infodemiological Study on COVID-19 in South Korea. J. Med. Internet Res. 2020, 22, e18897. [Google Scholar] [CrossRef]
  39. Abualigah, L.; Alfar, H.E.; Shehab, M.; Hussein, A.M.A. Sentiment Analysis in Healthcare: A Brief Review. In Recent Advances in NLP: The Case of Arabic Language; Abd, E.M., Al-qaness, M.A.A., Ewees, A.A., Dahou, A., Eds.; Springer International Publishing: Cham, Germany, 2020; pp. 129–141. [Google Scholar]
  40. Boon-Itt, S.; Skunkan, Y. Public Perception of the COVID-19 Pandemic on Twitter: Sentiment Analysis and Topic Modeling Study. JMIR Public Health Surveill. 2020, 6, e21978. [Google Scholar] [CrossRef]
  41. Al-Shargabi, A.A.; Selmi, A. Social Network Analysis and Visualization of Arabic Tweets During the COVID-19 Pandemic. IEEE Access 2021, 9, 90616–90630. [Google Scholar] [CrossRef]
  42. Raghupathi, V.; Ren, J.; Raghupathi, W. Studying Public Perception about Vaccination: A Sentiment Analysis of Tweets. Int. J. Environ. Res. Public Health 2020, 17, 3464. [Google Scholar] [CrossRef]
  43. Milani, E.; Weitkamp, E.; Webb, P. The Visual Vaccine Debate on Twitter: A Social Network Analysis. Media Commun. 2020, 8, 364–375. [Google Scholar] [CrossRef]
  44. Prochnow, T.; Patterson, M.S.; Meyer, M.R.U. A social network analysis approach to group and individual perceptions of child physical activity. Health Educ. Res. 2020, 35, 564–573. [Google Scholar] [CrossRef]
  45. Hawkins, J.B.; Brownstein, J.S.; Tuli, G.; Runels, T.; Broecker, K.; Nsoesie, E.O.; McIver, D.J.; Rozenblum, R.; Wright, A.; Bourgeois, F.T.; et al. Measuring patient-perceived quality of care in US hospitals using Twitter. BMJ Qual. Saf. 2016, 25, 404–413. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  46. Nwosu, A.C.; Debattista, M.; Rooney, C.; Mason, S. Social media and palliative medicine: A retrospective 2-year analysis of global Twitter data to evaluate the use of technology to communicate about issues at the end of life. BMJ Support. Palliat. Care 2015, 5, 207–212. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  47. Shams, M.; Saffar, M.; Shakery, A.; Faili, H. Applying Sentiment and Social Network Analysis in User Modeling. In International Conference on Intelligent Text Processing and Computational Linguistics; Springer: Berlin/Heidelberg, Germany, 2012; pp. 526–539. [Google Scholar] [CrossRef]
  48. Yao, Q.; Li, R.Y.M.; Song, L.; Crabbe, M.J.C. Construction safety knowledge sharing on Twitter: A social network analysis. Saf. Sci. 2021, 143, 105411. [Google Scholar] [CrossRef]
  49. Nasser, A.A.; Alzahrani, R.M.; Al-Falah, C.A.; Jreash, D.M.; Almouled, N.T.; Bakula, D.S.; Ra’Oof, R.A. Measuring the Patients’ Satisfaction About Telemedicine Used in Saudi Arabia During COVID-19 Pandemic. Cureus 2021, 13. [Google Scholar] [CrossRef]
  50. Kaliyadan, F.; Al Ameer, M.A.; Al Ameer, A.; Al Alwan, Q. Telemedicine Practice in Saudi Arabia During the COVID-19 Pandemic. Cureus 2020, 12, e12004. [Google Scholar] [CrossRef] [PubMed]
  51. McCormick, T.H.; Lee, H.; Cesare, N.; Shojaie, A.; Spiro, E.S. Using Twitter for Demographic and Social Science Research: Tools for Data Collection and Processing. Sociol. Methods Res. 2017, 46, 390–421. [Google Scholar] [CrossRef]
  52. Twitter API. Available online: https://developer.twitter.com/en/docs/twitter-api (accessed on 20 November 2020).
  53. Smith, M.; Milic-Frayling, N.; Shneiderman, B.; Mendes Rodrigues, E.; Leskovec, J.; Dunne, C. NodeXL: A free and open network overview, discovery and exploration add-in for Excel 2007/2010/2013/2016. Available online: https://www.smrfoundation.org (accessed on 1 December 2020).
  54. Al-Twairesh, N.; Al-Khalifa, H.; Al-Salman, A.; Al-Ohali, Y. AraSenTi-Tweet: A Corpus for Arabic Sentiment Analysis of Saudi Tweets. Procedia Comput. Sci. 2017, 117, 63–72. [Google Scholar] [CrossRef]
  55. McHugh, M.L. Interrater reliability: The kappa statistic. Biochem. Med. 2012, 22, 276–282. [Google Scholar] [CrossRef]
  56. Nakazawa, M.; Nakazawa, M.M. Package ‘fmsb’. 2019. Available online: https://cran.r-project.org/web/packages/fmsb/fmsb.pdf (accessed on 1 December 2021).
  57. Team, R.C. R: A language and environment for statistical computing. Available online: https://www.R-project.org/ (accessed on 1 February 2021).
  58. Monarch, R.M. Human-in-the-Loop Machine Learning: Active Learning and Annotation for Human-Centered AI; Manning Publications: New York, NY, USA, 2021. [Google Scholar]
  59. Tableau. Available online: https://www.tableau.com/products/techspecs (accessed on 15 December 2021).
  60. Clauset, A.; Newman, M.E.J.; Moore, C. Finding community structure in very large networks. Phys. Rev. E 2004, 70, 066111. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  61. Harel, D.; Koren, Y. A Fast Multiscale Method for Drawing Large Graphs; Springer: Berlin/Heidelberg Germany, 2001; pp. 183–196. [Google Scholar]
  62. Golbeck, J. Introduction to Social Media Investigation: A Hands-on Approach; Syngress: Waltham, MA, USA, 2015. [Google Scholar]
  63. Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G.S.; Dean, J. Distributed representations of words and phrases and their compositionality. In Proceedings of the Advances in Neural Information Processing Systems 26 (NIPS 2013), Lake Tahoe, NV, USA, 5–10 December 2013; pp. 3111–3119. [Google Scholar]
  64. Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint. 2018, arXiv:1810.04805 2018. [Google Scholar]
  65. Oueslati, O.; Cambria, E.; Ben HajHmida, M.; Ounelli, H. A review of sentiment analysis research in Arabic language. Futur. Gener. Comput. Syst. 2020, 112, 408–430. [Google Scholar] [CrossRef]
  66. Soliman, A.B.; Eissa, K.; El-Beltagy, S.R. AraVec: A set of Arabic Word Embedding Models for use in Arabic NLP. Procedia Comput. Sci. 2017, 117, 256–265. [Google Scholar] [CrossRef]
  67. Antoun, W.; Baly, F.; Hajj, H.M. AraBERT: Transformer-based Model for Arabic Language Understanding. arXiv 2020, arXiv:abs/2003.00104. [Google Scholar]
  68. Ahmed, W.; Seguí, F.L.; Vidal-Alaball, J.; Katz, M.S. COVID-19 and the “Film Your Hospital” Conspiracy Theory: Social Network Analysis of Twitter Data. J. Med. Internet Res. 2020, 22, e22374. [Google Scholar] [CrossRef]
  69. Ji, X.; Machiraju, R.; Ritter, A.; Yen, P.-Y. Examining the Distribution, Modularity, and Community Structure in Article Networks for Systematic Reviews. AMIA Annu. Symp. Proc. AMIA Symp. 2015, 2015, 1927–1936. [Google Scholar]
  70. Eddine, I.S.; Zedan, H.S. Telehealth Role During the COVID-19 Pandemic: Lessons Learned from Health Care Providers in Saudi Arabia. Telemed. J. E-Health 2021, 27, 1249–1259. [Google Scholar] [CrossRef]
  71. Palinkas, L.A.; De Leon, J.; Salinas, E.; Chu, S.; Hunter, K.; Marshall, T.M.; Tadehara, E.; Strnad, C.M.; Purtle, J.; Horwitz, S.M.; et al. Impact of the COVID-19 Pandemic on Child and Adolescent Mental Health Policy and Practice Implementation. Int. J. Environ. Res. Public Health 2021, 18, 9622. [Google Scholar] [CrossRef]
  72. Çeliktuğ, M.F. Twitter Sentiment Analysis, 3-Way Classification: Positive, Negative or Neutral? In Proceedings of the 2018 IEEE International Conference on Big Data (Big Data), Seattle, WA, USA, 10–13 December 2018; pp. 2098–2103. [Google Scholar]
  73. Palomino, M.A.; Varma, A.P. In Any Publicity is Good Publicity: Positive, Negative and Neutral Tweets Can All Become Trends. In Proceedings of the 2020 39th International Conference of the Chilean Computer Science Society (SCCC), Coquimbo, Chile, 16–20 November 2020; pp. 1–8. [Google Scholar]
  74. Shofiya, C.; Abidi, S. Sentiment Analysis on COVID-19-Related Social Distancing in Canada Using Twitter Data. Int. J. Environ. Res. Public Health 2021, 18, 5993. [Google Scholar] [CrossRef] [PubMed]
  75. Whitelaw, S.; Mamas, M.A.; Topol, E.; Van Spall, H.G. Applications of digital technology in COVID-19 pandemic planning and response. Lancet Digit. Health 2020, 2, e435–e440. [Google Scholar] [CrossRef]
  76. Alhajji, M.; Al Khalifah, A.; Aljubran, M.; Alkhalifah, M. Sentiment analysis of tweets in Saudi Arabia regarding governmental preventive measures to contain COVID-19. Preprints 2020. [Google Scholar] [CrossRef]
  77. AlOmari, E.; Katib, I.; Albeshri, A.; Mehmood, R. COVID-19: Detecting Government Pandemic Measures and Public Concerns from Twitter Arabic Data Using Distributed Machine Learning. Int. J. Environ. Res. Public Health 2021, 18, 282. [Google Scholar] [CrossRef] [PubMed]
  78. Han, X.; Wu, Y.; Zheng, J. Demands for Healthcare Industry Changes Provide the Soil for Digital Transformation. In Disruptive Innovation through Digital Transformation: Multi-Sided Platforms of E-Health in China; Springer: Singapore, 2020; pp. 1–8. [Google Scholar]
  79. Ricciardi, W.; Barros, P.; Bourek, A.; Brouwer, W.; Kelsey, T.; Lehtonen, L.; Expert Panel on Effective Ways of Investing in Health (EXPH). How to govern the digital transformation of health services. Eur. J. Public Health 2019, 29, 7–12. [Google Scholar] [CrossRef] [PubMed]
  80. Poli, A.; Kelfve, S.; Motel-Klingebiel, A. A research tool for measuring non-participation of older people in research on digital health. BMC Public Health 2019, 19, 1487. [Google Scholar] [CrossRef] [Green Version]
  81. Cheng, W.; Hao, C. Case-Initiated COVID-19 Contact Tracing Using Anonymous Notifications. JMIR mHealth uHealth 2020, 8, e20369. [Google Scholar] [CrossRef] [PubMed]
  82. Altmann, S.; Milsom, L.; Zillessen, H.; Blasone, R.; Gerdon, F.; Bach, R.; Kreuter, F.; Nosenzo, D.; Toussaert, S.; Abeler, J. Acceptability of App-Based Contact Tracing for COVID-19: Cross-Country Survey Study. JMIR mHealth uHealth 2020, 8, e19857. [Google Scholar] [CrossRef]
  83. Kawakami, N.; Sasaki, N.; Kuroda, R.; Tsuno, K.; Imamura, K. The Effects of Downloading a Government-Issued COVID-19 Contact Tracing App on Psychological Distress During the Pandemic Among Employed Adults: Prospective Study. JMIR Ment. Health 2021, 8, e23699. [Google Scholar] [CrossRef]
  84. Elkhodr, M.; Mubin, O.; Iftikhar, Z.; Masood, M.; Alsinglawi, B.; Shahid, S.; Alnajjar, F. Technology, Privacy, and User Opinions of COVID-19 Mobile Apps for Contact Tracing: Systematic Search and Content Analysis. J. Med. Internet Res. 2021, 23, e23467. [Google Scholar] [CrossRef]
  85. Ali, A.A.; ElFadl, A.H.; Abujazar, M.F.; Aziz, S.; Abd-Alrazaq, A.; Shah, Z.; Brahim, S.; Belhaouari; Househ, M.; Alam, T. Contact Tracing Apps for COVID-19: Access Permission and User Adoption. In Proceedings of the 2020 7th International Conference on Behavioural and Social Computing (BESC), Bournemouth, UK, 5–7 November 2020; pp. 1–7. [Google Scholar]
  86. Ahmad, S.; Chitkara, P.; Khan, F.N.; Kishan, A.; Alok, V.; Ramlal, A.; Mehta, S. Mobile Technology Solution for COVID-19: Surveillance and Prevention. In Computational Intelligence Methods in COVID-19: Surveillance, Prevention, Prediction and Diagnosis; Raza, K., Ed.; Springer: Singapore, 2021; pp. 79–108. [Google Scholar]
  87. Dunn, W.N. Social Network Theory. Knowledge 1983, 4, 453–461. [Google Scholar] [CrossRef]
  88. Emirbayer, M.; Goodwin, J. Network Analysis, Culture, and the Problem of Agency. Am. J. Sociol. 1994, 99, 1411–1454. [Google Scholar] [CrossRef]
  89. Moukarzel, S.; Rehm, M.; Caduff, A.; del Fresno, M.; Perez-Escamilla, R.; Daly, A.J. Real-time Twitter interactions during World Breastfeeding Week: A case study and social network analysis. PLoS ONE 2021, 16, e0249302. [Google Scholar] [CrossRef] [PubMed]
  90. Preece, J. Online Communities: Usability, Sociabilty, Theory and Methods. In Frontiers of Human-Centered Computing, Online Communities and Virtual Environments; Earnshaw, R.A., Guedj, R.A., van Dam, A., Vince, J.A., Eds.; Springer: London, UK, 2001; pp. 263–277. [Google Scholar]
  91. Vidal-Alaball, J.; Fernandez-Luque, L.; Marin-Gomez, F.X.; Ahmed, W. A New Tool for Public Health Opinion to Give Insight into Telemedicine: Twitter Poll Analysis. JMIR Form. Res. 2019, 3, e13870. [Google Scholar] [CrossRef] [PubMed]
Ijerph 18 13388 g001 550
Figure 1. Two-phase study methodology: “Phase 1: Data collection” and “Phase 2: Data annotation and analysis”, including network and sentiment analysis. * NodeXL is a network analysis and visualization software package for Microsoft® Excel®.
Figure 1. Two-phase study methodology: “Phase 1: Data collection” and “Phase 2: Data annotation and analysis”, including network and sentiment analysis. * NodeXL is a network analysis and visualization software package for Microsoft® Excel®.
Ijerph 18 13388 g001
Ijerph 18 13388 g002 550
Figure 2. The six mHealth apps networks using Twitter conversations. “Sehha” (top left), “Mawid” (top right), “Sehhaty” (middle left), “Tetamman” (middle right), “Tawakkalna” (bottom left), and “Tabaud” (bottom right).
Figure 2. The six mHealth apps networks using Twitter conversations. “Sehha” (top left), “Mawid” (top right), “Sehhaty” (middle left), “Tetamman” (middle right), “Tawakkalna” (bottom left), and “Tabaud” (bottom right).
Ijerph 18 13388 g002
Ijerph 18 13388 g003 550
Figure 3. The sentiment of each mHealth app after data augmentation. These tweets were used for the sentiment classifier.
Figure 3. The sentiment of each mHealth app after data augmentation. These tweets were used for the sentiment classifier.
Ijerph 18 13388 g003
Table
Table 1. Description of the six mHealth apps used in response to the COVID-19 pandemic in Saudi Arabia.
Table 1. Description of the six mHealth apps used in response to the COVID-19 pandemic in Saudi Arabia.
mHealth AppTranslation in EnglishCOVID-19 Primary UseMandatoryExisted before COVID-19
SehhaHealthTelehealthNoYes
MawidAppointmentsDigital screening 1NoYes
SehhatyMy HealthDigital screeningYesYes
TetammanRest AssuredFollow-up and isolationYes 1No
TawakkalnaWe TrustCOVID-19 health status, access public places, and electronic permits for movement, gathering, and workYesNo
TabaudSocial DistancingCOVID-19 contact notificationNo 2No
1 During the time the study was conducted. 2 This app is usually linked and integrated with “Tawakkalna”.
Table
Table 2. Tweet counts from network data after manual sentiment annotation.
Table 2. Tweet counts from network data after manual sentiment annotation.
mHealth AppTweet SentimentTotal (%)
Positive (%)Neutral (%)Negative (%)Indeterminate (%)Sarcasm (%)
Sehha7 (16.7%)28 (66.7%)3 (7.1%)1 (2.4%)3 (7.1%)42 (0.8%)
Mawid8 (6.3%)89 (69.5%)22 (17.2%)5 (3.9%)4 (3.1%)128 (2.5%)
Sehhaty8 (6.3%)98 (77.2%)13 (10.2%)5 (3.9%)3 (2.4%)127 (2.5%)
Tetamman164 (35.9%)257 (56.2%)23 (5.0%)3 (0.7%)10 (2.2%)457 (9.1%)
Tawakkalna49 (1.4%)1292 (37.7%)143 (4.2%)1920 (56.0%)22 (0.6%)3426 (67.9%)
Tabaud9 (1.0%)657 (75.7%)2 (0.2%)189 (21.8%)11 (1.3%)868 (17.2%)
Total245 (4.9%)2421 (48.0%)206 (4.1%)2123 (42.1%)53 (1.0%)5048 (100%)
Table
Table 3. Tweet counts after applying data augmentation techniques.
Table 3. Tweet counts after applying data augmentation techniques.
mHealth AppTweet SentimentTotal (%)
Positive (%)Neutral (%)Negative (%)
Sehha56 (56.6%)27 (27.3%)16 (16.2%)99 (2.1%)
Mawid66 (31.0%)87 (40.8%)60 (28.2%)213 (4.5%)
Sehhaty154 (31.8%)192 (39.7%)138 (28.5%)484 (10.3%)
Tetamman69 (23.5%)141 (48.0%)84 (28.6%)294 (6.2%)
Tawakkalna523 (21.7%)1314 (54.6%)571 (23.7%)2408 (51.0%)
Tabaud385 (31.5%)704 (57.7%)132 (10.8%)1221 (25.9%)
Total1253 (26.6%)2465 (52.2%)1001 (21.2%)4719 (100.0%)
Table
Table 4. Comparing user relationships across mHealth networks.
Table 4. Comparing user relationships across mHealth networks.
Network MeasuresSehhaMawidSehhatyTetammanTawakkalnaTabaud
Nodes, n762014644443076734
Isolates, n (%)9 (11.84)29 (14.43)9 (1.94)40 (9.01)65 (2.11)14 (1.91)
Total edges, n6120662050457551520
Unique edges, n (%)55 (90.16)175 (84.95)320 (51.61)391 (77.58)2047 (35.57)536 (35.26)
Edges with duplicates, n (%)6 (9.84)27 (13.11)300 (48.39)113 (22.42)3708 (64.43)984 (64.74)
Self-loops, n (%)11 (18.03)47 (22.81)41 (6.61)99 (19.64)361 (6.27)171 (11.25)
n represents the total number per measure.
Table
Table 5. Comparing mHealth networks’ properties on Twitter.
Table 5. Comparing mHealth networks’ properties on Twitter.
PropertySehhaMawidSehhatyTetammanTawakkalnaTabaud
Maximum geodesic distance (diameter)578585
Average geodesic distance1.37822.55812.29292.03302.15152.0053
Connected components, n3069579215341
Maximum nodes in a connected component, n9383091492642575
Maximum edges in a connected component, n85248814951581355
Graph density0.01650.00730.00380.00370.00060.0027
Modularity0.79920.74850.49200.68610.34190.3354
n represents the total number per property.
Table
Table 6. Examples of positive, neutral, and negative tweets associated with each mHealth app.
Table 6. Examples of positive, neutral, and negative tweets associated with each mHealth app.
mHealth AppPositiveNeutralNegative
Sehha“Sehha app is truly great, the Dr. examined me while I was at home and gave me a prescription.”“Try Sehha app, a physician will answer you. You can have 3 consultations per month for free.”“I am physically very tired, and I do not know why until now I have not gone to the hospital, Allah, I thought I was braver than this. Even Sehha app isn’t working.”
“تطبيق صحه جميل الصدق فحصني وانا بالبيت وعطاني وصفه طبيه” “افتحي تطبيق صحة وترد عليك دكتوره او دكتور معك ٣ استشارات بالشهر ومجاني” “ انا تعبانه جسديا وواصله لمرحله كبيره ولا ادري ليه للحين مارحت للمستشفى والله احسب نفسي اشجع من كذا حتى تطبيق صحه مايشتغل”
Mawid“By using Mawid app, things are excellent”“You can book an appointment at the health center through Mawid app.”“I wanted to do a swab test, and I remember searching between Mawid, Sehhaty, and Tetamman apps, I got confused by the abundance of applications one is enough.”
“عن طريق تطبيق موعد الأمور ممتازة “ “يمكنك حجز موعد لدى المركز الصحي عبر تطبيق موعد” “كنت بعمل مسحه واتذكر جلست ادور بين تطبيق موعد وصحتي وتطمن لخبطونا بكثرة التطبيقات واحد يكفي”
Sehhaty“I’m astonished by @SaudiMOH amount of effort. I booked an appointment for the Corona test from Sehhaty app. The entire trip, including the test, took only 18 min. A very great thing, thank you to the Ministry of Health. Honestly, I was not aware of the facilitation, until today”“The Minister of Health announces it at the #HIMSS20ME conference. Sehaty app will be the unified application for all services provided by the Ministry of Health”“I have a problem logging into Sehhaty app since a week ago. The same message appears, and the information is correct ??”
“انا منبهر من حجم جهود @SaudiMOH
حجزت موعد فحص كورونا من تطبيق صحتي المشوار كامل بما فيه الفحص استغرق ١٨ دقيقة فقط
شيء عظيم جدا شكرا وزارة الصحة ما كنت مطلع على التسهيلات للأمانة لأني ما جربت حتى اليوم”		“وزير الصحة يعلنها في مؤتمر #HIMSS20ME تطبيق صحتي سيكون التطبيق الموّحد لجميع خدمات وزارة الصحة” “عندي مشكلة في تسجيل الدخول لتطبيق صحتي لمدة اسبوع نفس الرسالة تظهر والمعلومات صحيحة؟؟ “
Tetamman“Tetamman—is an excellent app. great service, organization and accurate appointments, loyal health practitioners. May Allah protect my country and keep it well and safe.”“Tetamman app is intended for those who have been invited to download it via text messages or through a designated authority (infected or suspected of being infected). If you don’t have the conditions listed above, your isolation is considered optional, and you have the option to use the application services or delete it”“I was contacted to download Tetamman app, but I previously downloaded it and deleted it, now the place of isolation has changed, and the isolation days do not appear ... and the questionnaire is blank”
“تطمن- تطبيق ممتاز.. خدمة رائعة تنظيم ومواعيد مضبوطة.. ممارسين صحيين مخلصين. حفظك الله يابلادي ودمتِ بخير وامان” “تطبيق تطمن مخصص لمن تم دعوتهم لتحميله عبر الرسائل النصية او عبر الجهة المختصة (المصابين أو المشتبه بإصابتهم) في حال لم تكن من ضمن الشروط الواردة أعلاه يعتبر عزلك اختياري ولك الخيار في استخدام خدمات التطبيق أو حذفه” “تم التواصل معي وافادتي بتحميل تطبيق تطمن مع العلم بانه تم تحميله سابقا وتم حذفه والان تغير مكان العزل ولا يظهر ايام العزل ... وكذلك لاستبيان فارغ.”
Tawakkalna“The reason for the decline of the epidemic in Medina after Allah is Tawakkalna app, which was strictly applied. It is prohibited to enter any government facility or private sector unless you have the app ... If you are infected, or exposed your entry is not allowed.”“Exposed (orange and yellow color) are converted into healthy (green color) in Tawakkalna app by the Ministry of Health after 14 days without a confirmed COVID-19 infection.”“A painful sight when you see an elderly man, a woman, or a child leaves the health center without treatment ... why? Not having access to the internet on their mobile or not having a mobile to access the Tawakkalna app ... they do not know that there are people who can’t afford it. For most the internet is only at home.”
“ سبب انحسار الوباء بالمدينة بعد الله هو تطبيق توكلنا تم تطبيقه بحذافيره
ممنوع دخول اي منشأه حكومية او قطاع خاص الا والتطبيق معك.. واذا كنت مصاب ممنوع دخولك او مخالط ممنوع دخولك”		 “المخالط (اللون البرتقالي والاصفر) يتم تحويله الى سليم (اللون الأخضر) في توكلنا من قبل وزارة الصحة بعد مرور 14 يوم وعدم ثبوت الإصابة “ “منظر مؤلم عندما ترى رجل مسن أو امرأة او طفل يخرج من المركز الصحي دون علاج ... لماذا؟ ليس لديه نت في جواله أو ليس لديه جوال اصلا بدعوى تطبيق توكلنا ... لا يعلموا أن فيه ناس عائشة بالكفاف الأغلب النت في البيت. “
Tabaud“Do you know why everyone is so proud of you @SDAIA_SA?
Because, with your effort and the perseverance of your employees, you have limited the consequences of Corona, with the grace of Allah ... and we have become the top third country in the world to implement Exposure Notification technologies.”		“Tabaud app is to assist combating the Coronavirus COVID-19, to return to normal life as soon as possible by notifying the user if they were in contact with a person who was confirmed to have the virus during the past 14 days”“#Tabaud_app I do not like anxiety, and I expect anxiety harms human health ... We depend on Allah and from my point of view, psychological aspects must be considered in any app, especially regarding human health. How this can be possible, and there is no accurate phone device that is capable of giving an accurate location without chances of error.”
“تعرفون ليش الجميع يفخر بكم @SDAIA_SA
لأنكم بجهدكم ومثابرتكم وبهمة شبابكم وشاباتكم من أبناء الوطن حديتوا من تداعيات كورونا بتوفيق الله.. وصرنا ثالث دولة في العالم تطبيقًا لتقنيات Exposure Notification”		 “تطبيق “تباعد” هو للمساعدة على احتواء فيروس كورونا كوفيد١٩ والعودة إلى للحياة الطبيعية في أقرب وقت ممكن
من خلال الإشعار بمخالطة شخص تم تأكيد إصابته بالفيروس خلال الـ ١٤ يوم الماضية” 		“#تطبيق_تباعدانا ما أحب القلق واتوقع القلق يضر بصحة الانسان ... توكلنا على اللهومن وجهة نظري يجب مراعاة الجوانب النفسية في اي تطبيق خصوصا ما يتعلق بصحة الانسان كيف لا وأخطاء تحديد المواقع واردة ولا يوجد جهاز هاتف دقيق يعطي دقة في تحديد المواقع لا يمكن وجود خطأ معها”
Table
Table 7. The performance of the three sentiment classifiers.
Table 7. The performance of the three sentiment classifiers.
ClassifierPrecisionRecallF1-Score
* SVM-AraVec0.850.850.85
SVM-tfidf0.840.840.84
AraBERT0.820.780.80
* Our selected classifier.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Binkheder, S.; Aldekhyyel, R.N.; AlMogbel, A.; Al-Twairesh, N.; Alhumaid, N.; Aldekhyyel, S.N.; Jamal, A.A. Public Perceptions around mHealth Applications during COVID-19 Pandemic: A Network and Sentiment Analysis of Tweets in Saudi Arabia. Int. J. Environ. Res. Public Health 2021, 18, 13388. https://doi.org/10.3390/ijerph182413388

AMA Style

Binkheder S, Aldekhyyel RN, AlMogbel A, Al-Twairesh N, Alhumaid N, Aldekhyyel SN, Jamal AA. Public Perceptions around mHealth Applications during COVID-19 Pandemic: A Network and Sentiment Analysis of Tweets in Saudi Arabia. International Journal of Environmental Research and Public Health. 2021; 18(24):13388. https://doi.org/10.3390/ijerph182413388

Chicago/Turabian Style

Binkheder, Samar, Raniah N. Aldekhyyel, Alanoud AlMogbel, Nora Al-Twairesh, Nuha Alhumaid, Shahad N. Aldekhyyel, and Amr A. Jamal. 2021. "Public Perceptions around mHealth Applications during COVID-19 Pandemic: A Network and Sentiment Analysis of Tweets in Saudi Arabia" International Journal of Environmental Research and Public Health 18, no. 24: 13388. https://doi.org/10.3390/ijerph182413388

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop