A Deep Learning-Based National Digital Literacy Assessment Framework Utilizing Mobile Big Data and Survey Data

With the rapid advancement of digital technology, artificial intelligence has ushered in a digital society. In this era, digital literacy has become a prerequisite for individuals, as its absence can lead to new vulnerabilities and inequalities, hindering the pursuit of sustainable development goals. Previous researches predominantly relied on questionnaires to assess digital literacy, often focusing on specific groups due to survey costs, making their methodology unsuitable for comprehensive countrywide measurement. To address these limitations, we propose FLAKE, a national digital literacy assessment framework. Within this framework, we devise a multi-task deep learning model called DLMaN, which employs mobile big data, such as users’ digital behaviors, to predict citizens’ digital literacy. FLAKE enables cost-effective assessment of digital literacy for massive citizens by surveying only a fraction of them and it also has valuable implications for other social research tasks. We test the framework’s performance using authentic survey data and mobile big data, achieving RMSE and MAPE of 5.233 and 8.65% respectively, and the improvement is significant compared to the baseline model. We further employ this model to assess the digital literacy of numerous citizens in China and explore the implications for the society and individuals based on the obtained results.


I. INTRODUCTION
With continuous breakthroughs in high-techs such as artificial intelligence and the Internet of Things, the exchange, processing, and dissemination of information by humans have become increasingly frequent and rapid.From shopping and entertainment to work, digital technology has profoundly changed our way of life, ushering in the advent of the digital society.However, further digitization has brought forth new challenges that cannot be overlooked.These challenges encompass aspects like data security and user privacy [1], [2], [3], [4], digital divide and unbalanced development [5], [6], [7], [8], as well as new demands imposed on practitioners due to the transformation of traditional industries [9], [10], [11].If we do not face up to these difficulties, it will inevitably exacerbate existing inequalities and potentially generate new ones in the digital society [12], which will also hindering The associate editor coordinating the review of this manuscript and approving it for publication was Sawyer Duane Campbell .
the sustainable development of human beings in the social and economic spheres.Especially with the emergence of ChatGPT at the end of 2022, such large language models have led to a further increase in productivity, but also to more ethical and moral issues [13], [14].Now that the fourth industrial revolution has arrived, in order to make digital technology truly serve all mankind and promote a more equitable, inclusive and sustainable digital society, we need to explore how to enhance citizens' ability to integrate into the digital society, which is commonly referred to as digital literacy.
Since entering the 21st century, digital literacy has emerged as a prominent research area, and various countries and organizations have introduced corresponding plans and policies to facilitate related research.In 2008, under its i2010 strategy, the European Commission reported on 470 digital literacy initiative while emphasizing the increasing importance of digital literacy as an essential life skill.In 2013, it proposed to improve the digital literacy among citizens as one of seven priority areas for action, and released the first version of digital competence, DigComp, at the same time [15].The United States likewise prioritized popularizing digital literacy education in the National Broadband Plan in 2010 [16].Recognizing the significance of digital literacy in national development and comprehensive human advancement, China also formulated the Action Plan for Enhancing Digital Literacy and Skills for All in 2021.In general, as a crucial component of scientific literacy, digital literacy provides a technical basis for citizens to skillfully use digital tools.It empowers individuals to comprehend and create digital content, as well as fostering innovative thinking that enables them to better understand, adapt to, and shape society.It can be seen that in the digital age, digital literacy has become an indispensable competency for every citizen.Insufficient digital literacy can lead to marginalization in this era.Consequently, the evaluation of digital literacy has become very necessary.Gaining timely insights into the public's level of digital literacy is of great significance to personal learning and career development, educational reforms and digital transformation, ultimately contributing to sustainable development of society.
A lot of previous works have been conducted on digital literacy assessment, but these studies still possess certain limitations.First of all, most of these studies focused on exploring and evaluating digital literacy within specific groups, such as teachers and students [17], [18], [19], doctors and patients [20], [21], [22], and company employees [23], [24], [25].These studies have designed a corresponding digital literacy assessment system for these specific occupations, with variations in the evaluation content depending on the occupation.Although more targeted, these studies have somewhat restricted sampling objects and evaluation content, making it challenge to reflect individuals' overall abilities and the digital literacy levels of the whole society.To gain a comprehensive understanding of a country's digital resilience, it would be beneficial to assess the digital literacy of a large number of citizens across various groups.This approach would facilitate the formulation of targeted policies and laws to promote the digital development.However, due to the high economic cost and the difficulty in formulating evaluation schemes, research in this area is very scarce.Secondly, the assessment method of the previous studies is basically to conduct a sample questionnaire survey on the research group, followed by the calculation and exploration of digital literacy according to the returned results and the predefined framework [17], [18], [19], [20], [21], [22], [23], [24], [25].The economic and time costs typically limit the number of survey users, and the length of the questionnaire cannot be too long, so the user characteristic information that can be obtained is also less.Moreover, due to the sample selection bias and the respondent's own factors, the accuracy of the obtained data may be compromised.
With the advent of the digital society, the digital divide is an important obstacle for the whole society to enjoy the achievements of scientific and technological development.
To bridge the digital divide, it is not only necessary to pay attention to the construction of basic digital facilities, but more importantly, to enhance the ability of citizens to use digital facilities, which falls within the realm of digital literacy.To eliminate old inequalities while avoiding new ones, it becomes crucial to conduct a universal assessment of the digital literacy across various groups in society.Understanding the specific weaknesses of different groups will promote the formulation of relevant policies and laws, which is of great significance for the realization of digital inclusion and sustainable development goals.Therefore, the scope of research on digital literacy should not be limited to a certain group or occupation.What we want to know is the public's adaptability to the digital society across a range of ages, regions, and occupations, which is exactly what most of the prior works could not answer.In addition, compared with relying solely on survey data for digital literacy research, the use of mobile big data allows for a more comprehensive reflection of the digital behavior patterns among a large user base, which is also helpful for the national digital literacy assessment.In recent years, some fields have turned their attention towards leveraging multi-source data, such as mobile big data and survey data, for user and social research [26], [27], [28].The utilization of diverse data sources can provide complementary advantages, significantly contributing to enhancing the accuracy of conclusions and broadening the scope and depth of research.However, when it comes to digital literacy assessment, few studies have adopted a similar approach.Moreover, by introducing big data, algorithmic models such as machine learning or deep learning can be used to assist subjective scale assessment.This approach offers the dual benefit of saving considerable time and economic costs while also rendering the conclusions more objective.
In order to address the limitations of previous studies on digital literacy assessment, this paper proposes a national digital literacy assessment framework.Our approach uses both mobile big data and survey data, employing data Fusion techniques to synthesize multi-source data for generalized digital literacy assessment.This assessment framework utilizes the correlation mechanism between digital Literacy and digital behavior that is App usage data adopting the KAP and the iceberg model as the basic theory, to construct a deep learning model that can predict citizens' digital literacy, which can simultaneously assess the digital literacy of an Extraordinary number of people at a relatively small cost.For ease of presentation, we take the initial letters of the framework's characteristic elements and name it FLAKE framework.
Digital behavior generally refers to the users' behavioral patterns in the digital environment, especially how people stay online in terms of access [29].In this paper, we use the term digital behavior to refer to the user's app usage information, which can be obtained from mobile big data, and at the same time can well reflect the user's characteristic preferences and behavioral patterns [30].According to the iceberg model proposed by the American psychologist McClelland, the quality performance of an individual can be divided into the part on the iceberg and the part below the iceberg, under the iceberg is mainly the intrinsic literacy such as character traits and values of a person, while on the iceberg is the external manifestation of these intrinsic literacy, that is behavior [31].According to the iceberg model, digital literacy and behavior are actually the two ends of the chain of conscious behavior, and digital behavior can actually be regarded as the dominant expression of the user's intrinsic literacy, so there will be a relatively strong correlation between the two which allows us to infer users' digital literacy from digital behavior data.When mastering user mobile big data, we can easily obtain the digital behavior data such as app usage of massive users.Therefore, the FLAKE framework in this paper builds a deep learning model that can predict the users' digital literacy based on their digital behavior.This model can directly infer the digital literacy approximation of massive users, and even though there is a certain degree of error unavoidably, a lot of time and economic costs can be saved for the research, which is suitable for a large-scale digital literacy census of citizens.In this paper, we test the FLAKE framework based on authentic mobile big data and survey data, and the experimental results show that our approach can predict the digital literacy of massive users with a small error.
The main contributions of this paper are summarized as follows: • We present a novel and comprehensive framework FLAKE for assessing digital literacy in a large user population, specifically designed to be cost-effective.
Our framework employs a combination of subjective assessment scales and objective deep learning models to enhance the operability of the framework as well as the result reliability.
• We discover a correlation between digital literacy and digital behaviors.Building upon this, we propose a deep learning model to predict digital literacy by leveraging big data on users' digital behaviors.To the best of the author's knowledge, this is the first attempt to predict users' digital literacy based on their app usage information.Additionally, we adopt the idea of multi-task learning to design the prediction model and enhance its accuracy in prediction.
• We validate the effectiveness of the framework and model proposed in this paper on an anonymous dataset with massive authentic users.Concurrently, we conduct an in-depth analysis and discussion of the digital literacy prediction results among a vast user population within this dataset.Our objective is to explore how the state and the society should act in the realm of digital literacy in order to bridge the digital divide and achieve the goals of broad equality in the digital society as well as sustainable societal development.The assessment encompasses over two million users, so the conclusions drawn have a higher reference value and significance.
The rest of this paper is organized as follows.In Section II, we discuss some related works on digital literacy and digital behavior.Section III presents the solution framework for national digital literacy assessment and provides model details.We explain the experimental results in Section IV.As a follow up, Section V discusses our findings from the experimental results and presents the implications for theory and practice, as well as the limitations.At last, we conclude our work in Section VI.

II. RELATED WORK
Since the beginning of the 21st century, the continuous development of digital technology, especially the widespread use of the Internet, has made digital literacy a subject of great interest across various disciplines such as social science, psychology, and pedagogy.In recent years, research on digital literacy has mainly focused on several key aspects, including the definition and conceptualization of digital literacy, competency domains, evaluation and practical implementation.
The assessment of digital literacy relies heavily on establishing its framework and dimensionality, which may vary across studies due to the evolving conceptual definition of digital literacy.Lordache et al. argued that digital literacy consists of knowledge, skills and competencies.Knowledge represented the information, awareness, and understanding of digital tools; digital skills referred to measurable applications of knowledge; and competencies involved the ability to apply knowledge and skills in various social and productive contexts [36].In the digital literacy framework for adolescents designed by the International Computer Literacy and Information Study (ICILS), digital literacy was categorized into 4 strands, while each strand contained 2 aspects, for a total of 8 aspects.These 4 strands encompassed understanding computer use, gathering information, producing information, and digital communication [38].The Digital Competence Framework (DigComp), issued by the European Commission to guide digital competence education, classified digital literacy into 21 competencies in 5 domains.These domains included information and data literacy, communication and collaboration, digital content and creation, safety, and problem solving [39].Although digital literacy frameworks vary from study to study, we are able to abstract them into three broad directions, which are digital awareness, digital skill, and digital attitude.Digital awareness represents the understanding and use of digital tools, which is a concept or fact; digital skill focuses on the practical application of knowledge in digital environments, emphasizing the ability to execute the process; digital attitude pertains to the mindset, beliefs and values when acting.These three dimensions of digital awareness, digital skill, and digital attitude are interconnected and closely related to each other.Digital awareness assists in comprehending the digital environment, digital skill entails the explicit use of digital awareness, and digital attitude guides the appropriate application of knowledge and skills.Most digital literacy frameworks can basically be mapped from these three dimensions.Therefore, the digital literacy TABLE 1.A summary of digital literacy assessment studies.It can be observed that there is a limited amount of research on using models to conduct digital literacy assessments on big data.system constructed in this paper will also be organized around these three aspects.
There are many previous studies that designed their own or referred to authoritative frameworks to assess and analyze digital literacy.Silamut and Sovajassatakul [40] conducted a questionnaire survey to evaluate the digital literacy of students in different majors during the COVID-19 pandemic.They analyzed the survey data of 259 students based on one-way analysis of variance (ANOVA) and confirmatory factor analysis (CFA) to explore the predictors of digital literacy in different types of students and the effect of education level on students' digital skills.Leader et al. [22] used a sample survey to assess digital literacy among 363 patients with cancer and caregivers at a cancer center.They used descriptive statistics and bivariate analyses to assess the significance of differences between frequencies and variables, and investigated the effect of digital literacy level on the health status of patients.Cetindamar et al. [24] utilized an online survey to measure the digital literacy of 124 company employees, using the theory of planned behavior to assess the relationship of employees' digital literacy to the use of digital skills such as cloud technology, and also explored the role of digital literacy in a company's digital transformation.
In addition to the aforementioned studies, we have conducted a lot of research on related works and classified some representative studies into several groups, which can be shown in Table 1.It can be found that most digital literacy assessments focus on specific groups or fields, especially education and healthcare.This attention stems from the rapid proliferation of digital technologies, such as online classrooms and health monitoring devices, within these fields.As a result, there is an escalating necessity for digital literacy among professionals working in these industries.Moreover, these two fields are pivotal domains influencing human development and well-being, so related works will be relatively concentrated.At the same time, with the digital transformation of various industries, there are new requirements for the digital competence of company employees in traditional industries, so there is also a considerable part of digital literacy evaluation for employees.In addition, several studies concentrate on other specific groups, particularly those categorized as vulnerable populations, such as the elderly and rural communities.These studies are basically carried out based on digital literacy and digital inclusion, aiming to explore targeted approaches for assisting disadvantaged groups in integrating themselves into the digital age.
As introduced in section I, despite the maturity of the research on digital literacy, some comprehensive problems have not been solved.It can be seen from Table 1 that most of the previous related works were carried out on a specific group with limited sample sizes, assessing digital literacy in specific domains rather than encompassing general literacy.For different occupations or identities, the definition and evaluation emphasis of digital literacy are also different.For students, digital literacy focuses on their ability to learn digital technologies and utilize them for research and communication, while for company employees, it pertains to their proficiency in using digital tools to enhance work quality and efficiency.As citizens have diverse roles, evaluating digital literacy for a single role cannot reflect their overall qualities and abilities, and lacks versatility.There are many digital literacy assessments for specific groups, which is partly due to the research themes of these works, and more importantly, limited by economic and human costs.Only national governments and some international organizations have the ability to conduct large-scale and extensive digital literacy assessment.However, with the advent of the digital era, it is crucial to assess the digital literacy of multiple groups and a significant number of users.This understanding will help identify disparities in digital literacy among different groups and facilitate the development of relevant laws and policies to assist digitally disadvantaged groups in swiftly adapting to the digital advancements.This will also bridge the digital divide and ensure that scientific and technological progress genuinely benefits society as a whole.Therefore, there is an urgent research gap on how to assess the digital literacy of massive users with less time and economic costs.
Furthermore, it can also be seen from Table 1 that most studies in the field have utilized survey data for the computation of digital literacy.This involves employing questionnaires or interviews to measure digital literacy using a scale, which is a commonly employed method in social science research.However, if we want to obtain the digital literacy of massive users, this approach becomes time-consuming and expensive.In order to compensate for these shortcomings of traditional methods, we can make use of the digital behavior information in users' mobile big data, and try to realize the prediction of digital literacy by building machine learning or deep learning models.Consequently, with access to digital behavior data, the digital literacy of a sizable user population can be calculated.While a limited number of studies have mentioned the integration of big data in assessing digital literacy [49], almost none of them have conducted experimental attempts.Additionally, a few studies have employed machine learning models for digital literacy prediction [53], [54].However, these studies have also relied on survey data for model construction, resulting in simplistic models with limited scalability due to the small sample sizes.Currently, there is a dearth of research in utilizing deep learning models based on mobile big data's digital behavior information for predicting digital literacy.This is due to the fact that only some large companies possess user-authorized access to mobile big data, and the construction of deep learning models to mine digital behavior data also needs interdisciplinary and cross-disciplinary knowledge backgrounds, which likewise makes it very difficult to conduct research in this area.
Digital behavior, comprising user-generated behavior data within the digital environment, serves as a significant external expression of digital literacy.In this paper, digital behavior specifically refers to the mobile application usage information in the user's mobile big data, encompassing the apps installed on their devices, as well as usage traffic and duration for each app.In the field of user insights and data mining, app usage information is often used to predict users' personality traits and behavior patterns, with extensive research conducted in this domain [55], [56], [57], [58], [59], [60].Kalimeri et al. [58] explored the link between digital behavior and personality characteristics of 7,633 individuals by designing a machine learning framework based on psychometric questionnaires data, web browsing data, and mobile big data collected from smartphones.Malmi and Weber [55] employed logistic regression models to predict demographics including age, gender, ethnicity, and income on a dataset containing 3,760 users based on the app installation information, and achieved good results.Razavi [59] studied the determinants of various mobile usage attributes, combining them with the Big-Five personality traits.They utilized a range of machine learning classification models to predict user personality segments, achieving an overall accuracy of 76.17%.It can be seen that digital behavior data, including users' app usage information, can provide valuable insights into users' characteristics and personality.According to the iceberg model proposed by McMillan, a large part of an individual's behavior is determined by their cognitions and attitudes.Therefore, based on the app usage information in mobile big data, our paper aims to explore the model method of using digital behavior data to predict digital literacy, so as to carry out the evaluation and analysis of digital literacy for a large user population.
There are also some previous works that use both mobile big data and survey data to conduct research, particularly focusing on user profiling and demographic prediction [26], [27], [28].These studies collected user labels through survey data and employ machine learning models to predict these labels based on the mobile data.Mobile big data captures the interactions between individuals and their mobile devices, offering substantial data volume and diverse features.Alternatively, survey data is obtained through questionnaires or interviews.While survey data may have a smaller volume, it offers flexibility and can be tailored to specific scenarios.By combining these two data sources, a more comprehensive understanding of users can be obtained while enhancing the accuracy and generalizability of the findings.Therefore, this paper aims to investigate digital literacy by integrating survey data with mobile big data.

III. METHODOLOGY A. SYSTEM FRAMEWORK
In this section, we will describe in detail the method and process of FLAKE, the national digital literacy assessment framework proposed in this paper.
Fig. 1 presents the structure of the FLAKE framework.It consists of three main parts.The first step involves acquiring and calculating the digital literacy of survey participants.In this step, our scheme is the same as most previous studies.First, we need to design a questionnaire scale to assess citizens' digital literacy.In this study, we measure the general digital literacy of citizens because the research subjects are not limited to a specific occupation or group.The index system of digital literacy in our research is designed according to the KAP theory, which believes that human behavior change is influenced by knowledge, attitude and practice, and the change of digital literacy is also a similar process.Therefore, we disassemble digital literacy into three dimensions, namely digital knowledge, digital attitude and digital skill.These three second-level indexes can be subdivided into 8 third-level indexes and 16 fourth-level indexes according to specific types of competencies, which we will introduce in detail in Section III.We employ this indicator system to construct a scale for measuring digital literacy.Once the scale design is finalized, we distribute it as a questionnaire to gather responses from a sample of users.Based on the returned data, we use entropy weighting method and analytic hierarchy process to calculate the digital literacy of the survey users.After that, according to the introduction mentioned earlier, this paper proposes a fusion of mobile big data and survey data to carry out the assessment of digital literacy.To achieve this, we extract the digital behavior data of these survey users in their mobile big datasets, which is app usage information, including name, category, data traffic and duration time.By combining these two sets of data, we are able to gain a comprehensive understanding of the research users' digital literacy, including their scores on various sub-indexes obtained through the survey, as well as their digital behaviors collected from mobile big data.
The second step in the FLAKE framework is to train a civic digital literacy prediction model using the two primary types of data obtained from the survey users in the first step.We employ a deep learning model based on multi-task learning to realize the prediction of digital literacy.Following the Iceberg theory, human digital behavior serves as an indicator of intrinsic digital literacy.Hence, the model primarily takes into account the user's digital behavior information, complemented by user profile and mobile device features.The output of the model is the user's digital literacy score based on the scale used in the study.To train the model, we divide the survey users into a training set and a test set.After completing the prediction model, we proceed to the third step of the FLAKE framework.This step involves predicting the digital literacy of a significantly larger number of users than those involved in the survey.Under conventional methods, obtaining the digital literacy of these users using the survey scale would be immensely time-consuming and financially burdensome.However, within the FLAKE framework, we can extract the digital behavior data of these predictive users from the mobile big data, and then input them into the trained civic digital literacy prediction model in the second step, enabling us to acquire these users' digital literacy scores.In this way, the digital literacy assessment of massive citizens can be realized.
Compared to previous studies, the FLAKE framework has the advantage of realizing digital literacy assessment of massive users with investigating only a fraction of them, consuming less time and economic cost.Furthermore, we introduce digital behavior data as an innovative means of predicting digital literacy.The entire framework leverages both survey data and big data, incorporating both subjective scales and objective models, thus enhancing the depth and breadth of the evaluation.

B. INDEX SYSTEM OF DIGITAL LITERACY 1) CONNOTATION OF DIGITAL LITERACY
This paper refers to Alkalai's (2004) and Goodfellow's [61] definitions of digital literacy based on the findings of the research on digital literacy conducted by other scholars and discussed above.Alkalai believed that a person's digital literacy should take into account both their particular perspectives and beliefs in the digital world [61].Goodfellow considered that a person's conduct when using digital tools in particular situations was related to their level of digital literacy [62].While referring to the connotation of digital literacy defined by official organizations such as the European Union [63], the American Library Association [64], and UNESCO [65], we come to the conclusion that digital literacy encompasses a range of skills, knowledge, and attributes that individuals need to effectively navigate and thrive in a digitally-driven society, enabling their continued personal growth and development.We define digital literacy as the synthesis of a series of qualities, such as knowledge, skill and attitude to understand, apply and assess digital information and technology.

2) CONSTRUCTION OF DIGITAL LITERACY SYSTEM
This paper develops a comprehensive digital literacy index system with one first-level index, three second-level indexes, eight third-level indexes, and sixteen fourth-level indexes.The system, illustrated in Table 2, is developed based on a meticulous analysis and understanding of the concept of digital literacy.Its foundation lies in adhering to the principles of scientificity, comprehensiveness, representativeness, and operability.
The digital literacy index system framework refers to the KAP model [66].This model assesses digital literacy by considering three aspects: knowledge, attitude, and skills, In the context of digital technology advancement in China, this study specifically focuses on digital knowledge and skill aspects.Digital knowledge refers to individuals' comprehension of technology and digital devices in the digital environment.Digital skill, on the other hand, refers to how individuals use digital tools like cell phones and computers for basic activities, including recreation, socialization, and other life behaviors.
Drawing upon the European Digital Competence Framework [67], the digital skill dimension encompasses four third-level indexes and eight fourth-level indexes, namely Information and Data Literacy, Communication and Collaboration, Digital Content Creation, and Problem Solving.These indexes are utilized to assess people's level of proficiency in using digital tools.The digital attitude dimension, influenced by social cognitive theory, focuses on self-control, self-awareness, and self-efficacy.At the same time, we make reference to Prior's study [68] on how attitude affects digital literacy.From the dimensions of digital responsibility awareness and level of digital responsibility awareness, there are two third-level indexes and four fourth-level indexes to gauge the evolution of citizens' digital attitudes.The digital knowledge dimension assesses the level of development of digital cognition from the two aspects of information knowledge and device knowledge.DigEuLip's framework [69] of knowledge necessary for digital literacy serves as the basis for these indexes, resulting in two third-level indexes and four fourth-level indexes.

3) APPLICATION AND SELECTION OF DIGITAL LITERACY MEASUREMENT METHODS
This study incorporates the data characteristics and correlation between selected indicators to comprehensively assess the suitability of each measurement method for this research.For assessing the digital literacy index, we employ a combination of the analytic hierarchy process (AHP) and the entropy weight method.The AHP divides the goal into several hierarchical structures in accordance with the target results and it is a subjective weight analysis approach.The entropy weight method is an objective weight calculation method that reflects the differences and importance of different indexes through the information entropy of the indexes.To mitigate potential biases in expert scoring, this study constructs the AHP matrix by incorporating expert scores based on the information entropy of fourth-level indexes.The subsequent calculation steps aim to avoid excessive subjectivity in expert scoring.The precise calculation procedure is as follows: Firstly, a hierarchical analysis model is built according to the index system of digital literacy.Secondly, the basic weights of fourth-level indexes are determined according to the entropy weight method.The greater the dispersion degree and the smaller the entropy value of the indexes, the greater the impact of the indexes on the comprehensive evaluation.The formula for calculating information entropy is as follows: j represents the ordinal number of indexes, i represents the number of options contained in each index, p ij refers to the probability of the i-th option appearing in the number j index.Then, we construct the judgment matrix for the analytic hierarchy model using the entropy weight results of the fourth-level indexes.We calculate the reciprocal of the information entropy ratio of the two indexes at the same level taken as the relative importance of the two indexes.In this calculation, k represents the ordinal number of the hierarchy and j represents the ordinal number of the index.Construct the following judgment matrix: We proceed by computing the maximum eigenvalue and the corresponding eigenvector for every judgment matrix.The consistency test is carried out according to the consistency index, random consistency index and consistency ratio.If the test is successful, the eigenvector becomes the weight vector.If the test fails, we construct the judgment matrix once again, referring to the expert's re-scoring method, and repeat the calculation until the test is passed.In this way, the weights of each index at each level are calculated in turn.Finally, a weighted average is employed to quantify the scores of each index.

C. DIGITAL LITERACY PREDICTION MODEL BASED ON MULTI-TASK LEARNING AND NEURAL NETWORKS 1) DESCRIPTION OF MOBILE BIG DATASETS
The FLAKE framework integrates survey data and mobile big data to assess users' digital literacy.The mobile big dataset used in this paper comes from the telecom operator, and the full-volume dataset contains more than one billion users.The dataset utilized in this study ensures anonymity and authenticity by encrypting user identity information, adhering to academic research standards.It includes three main categories of features: user profile features, mobile device features, and digital behavior features.A detailed description of these features is shown in Table 3. User profile features refer to the user's personal attribute information, including age, gender, city, etc.; mobile device features capture the user's physical hardware details and package information of the cell phone, including brand, price, 5G compatibility, etc.; digital behavior features encompass the user's app usage patterns, such as app name, category and data traffic.

2) DATA PREPROCESSING
In this study, we employ a deep learning model based on neural networks to predict citizens' digital literacy.To improve training efficiency and enhance accuracy, we perform data preprocessing operations on the original data before inputting it into the model.
The data preprocessing operation includes two main steps.The first step involves the handling of missing values and outliers.Missing values represent null data that are lost during collection, transmission, or storage, while outliers refer to values that are obviously unreasonable.In this paper, we utilize box plots combined with quartiles to identify outliers, that is to say, the data points that are more than the upper and lower quartiles plus 1.5 times of the interquartile spacing (IQR) are all regarded as outliers.For missing values and outliers, we uniformly use the mode to fill.
The second step in data preprocessing is feature encoding, which aims to transform the data into a type that the neural network can handle.For numerical features such as age and income, we can directly normalize and standardize them.For categorical features such as gender, city, occupation, and app usage information, we need to perform one-hot encoding.This process converts each category into a binary vector, where the corresponding category is assigned a value of 1, while the remains are assigned 0. Among the categorical features, especially digital behavior features, since the number of apps is enormous, it is actually a kind of high-dimensional sparse features.These features need to be dimensionally reduced through the embedding layer in the neural network.

FIGURE 2.
The structure of the digital literacy prediction model based on multi-task learning and neural networks (DLMaN).Each substructure in the model predicts a sub-index.These predictions are then fused together using an attention fusion module to obtain the final prediction result.

3) DATA MODELING
After finishing the data preprocessing operation, we can proceed to the model construction.In this paper, we propose a Digital Literacy prediction model based on Multi-task learning and Neural network, referred to as DLMaN, and the specific structure is illustrated in Fig. 2. The model takes the user's digital behavior and other features introduced in prior part as input, and generates the predicted value of digital literacy.
We choose the deep learning model to handle data processing and predict digital literacy due to the fact that the digital behavior features primarily consist of user's app usage information.Because of the large number of apps, such features are actually high-dimensional sparse features.Conventional machine learning models such as support vector machines and decision trees, are not well-suited for handling this type of features, and the model will be easily overfitted.In contrast, neural network models can effectively handle high-dimensional sparse vectors through the embedding layer, enabling deep crossover and dimensionality reduction.This capability allows the deep learning model fit these features well, which is also the current mainstream processing method.
In addition, this paper employs multi-task learning to optimize the conventional neural network model.The conventional neural network model, as shown in Fig. 3, is essentially a single-task model, that is, the overall model has only one training task, which in this paper is the prediction of digital literacy.In our research, the training data of the model is exclusively derived from the survey users.However, due to the high cost of the survey, it is impractical for us to send questionnaires to a large number of users, so the scale of the training data becomes relatively limited.In such cases, if a conventional single-task model is used for modeling, the model accuracy will be relatively low.Therefore, we have made improvements to the model based on the principles of multi-task learning.
Multi-task learning entails simultaneously performing multiple training tasks within a single model.Although each task has its own unique mode and objective, there are underlying knowledge or abstract concepts that can be shared with each other, and these can be used by multiple tasks at the same time.Exploiting this shared knowledge enables enhancement of overall training accuracy and efficiency.Numerous studies have demonstrated that as long as there is a certain correlation between different tasks, model optimisation using the strategy of multi-task learning can often achieve significantly better results than single-task learning models [70], [71], [72], [73].
In this paper, we propose a multi-task learning approach for predicting digital literacy.The main task is predicting the overall digital literacy score, while the auxiliary tasks involve predicting sub-indexes of digital literacy, which are hierarchical measures introduced in part B of section III.These sub-indexes represent specific ability domains and strongly correlate with digital behavior.We utilize features such as app usage to predict them.
As shown in Fig. 2, the substructures arranged in parallel at the bottom of the DLMaN model are auxiliary tasks for predicting these sub-indexes, while the main task is trained based on the intermediate values of these auxiliary tasks.Since digital literacy is composed of these sub-indexes, we construct the multi-task learning model in this manner.The input of auxiliary tasks is digital behavior and other user features.These features are initially passed through the embedding layer for feature crossover and dimensionality reduction.It is important to note that the embedding layer implements a parameter sharing mechanism to maintain consistent parameters across different tasks, thereby reducing the overall model complexity.The output of the embedding layer is then fed into the hidden layer for deep cross-fusion, and subsequently connected to the output layer.The output of each auxiliary task is the predicted value of its respective sub-index.On the basis of auxiliary tasks, we also need to train the main task.To achieve this, we extract the output vector of the last hidden layer from the auxiliary tasks and pass it through an attention fusion module that performs weighted fusion.The output vector of this module is further processed by another hidden layer to generate the predicted value for the main task, namely digital literacy.
In the DLMaN model, we use the huber loss as the loss function of all subtasks including the main task and all auxiliary tasks.The form of this function is as follows: y is the true value of the subtask, and F(q) represents the predicted value generated by the model.The huber loss function combines the advantages of the square difference loss function and the absolute value loss function.It is differentiable at zero and continuous when calculating the gradient, which is conducive to better optimization of the model.At the same time, the huber loss function has a smaller penalty for outliers, making the model more robust.According to the principle of multi-task learning, we need to design the overall loss function of the model based on the loss function of the subtasks.In order to achieve end-to-end integrated training of the DLMaN model, we incorporate the loss value of the main task and all auxiliary tasks when designing the loss function of the model.Additionally, we adopt the Dynamic Weight Averaging method [74], assigning different levels of emphasis to each task to optimize the training process of the model.The overall loss function is formulated as follows: Among them, L 0 is the loss function of the main task, that is, the prediction loss of digital literacy, and L 1 ∼ L n are the loss function of n auxiliary tasks which represent the prediction loss of digital literacy sub-indexes.We assign a weight value ω i (t) to each auxiliary task loss function as follows: L j (t-1) represents the loss function of auxiliary task j at step t-1, and r j (t-1) is the ratio of loss function of auxiliary task j at step t-1 and step t-2, which is actually the training speed, and T is a hyperparameter.According to the formula, it is apparent that tasks with rapidly shrinking loss functions will have smaller weights, while those with slower shrinkage will have larger weights.In this way, the dynamic weight averaging method can enable each auxiliary task of the DLMaN model to learn at a similar speed.This approach enhances the stability of model training, thereby improving overall performance.
According to the principle and structure of the DLMaN model, it can be found that after adding the sub-indexes of digital literacy to the model as training targets, the number of training samples for multi-task learning is actually the number of overall tasks multiplied by the number of survey users.This increase is achieved without the need to augment the number of survey users.Compared with the single-task model in Fig. 3 that solely focuses on training the digital literacy goal, the DLMaN model enhances the training sample size and enriches the prior information of the model.Although the model's complexity is heightened, we simplify the parameter quantity through the sharing of underlying parameters, so that the model can be fully trained and its prediction accuracy can be improved.The DLMaN model makes full use of the digital literacy index system in the questionnaire scale.It conducts modeling training on the digital literacy index and the sub-indexes that constitute digital literacy, and utilizes the principle that the underlying modes and concepts of these tasks are consistent for implicit data augmentation.The DLMaN model proposed in this paper is not only suitable for this digital literacy assessment work, but also bears reference significance for other scale data analysis.
After clarifying the overall structure and operating principle of the DLMaN model, we will provide a supplementary description of its internal modules.The main modules within the model encompass the following components: Hidden layer: The hidden layer in the DLMaN model is a fully connected layer designed to interact deeply with features and regularize vector dimension.The activation functions used in the hidden layers all select the ELU function, as depicted below [75]: Compared with the commonly used ReLU loss function, the ELU function is smooth in the entire space.The ELU function maintains non-zero values in the negative range, effectively mitigating issues related to neuron death and the vanishing/exploding gradient problems.At the same time, the non-zero gradient in the negative range enhances the convergence speed of the model.Therefore, the neural network units in the DLMaN model except the output layer employ the ELU function as the activation function.
Attention Fusion: As shown in Fig. 2, after extracting the output vectors of the last hidden layer of each auxiliary task, we use an Attention Fusion Unit to fuse these vectors into one vector.The weights assigned to each vector are calculated using an attention-based mechanism, as illustrated in the upper left section of Fig. 2. In terms of architecture design, we refer to the simple attention architecture proposed by Facebook [76], and perform a simple direct splicing of the extraction vectors of all auxiliary tasks, and then use a fully connected layer to deal with them.The number of output units in the fully connected layer is the number of auxiliary tasks.By applying the softmax function to the output of each unit, we obtain the corresponding vector weight.The weight is multiplied by the corresponding input vector and then spliced to obtain a long vector, as the output of the attention fusion unit.Based on the attention fusion unit, the DLMaN model allows for dynamic adjustment of the contribution of each auxiliary task in the output.This enables the model to prioritize sub-indexes that are most relevant to digital literacy, thereby enhancing the accuracy of its predictions.

IV. EXPERIMENTAL RESULTS
In this section, we will validate the effectiveness of our proposed FLAKE framework with authentic survey data and mobile big datasets.The data were collected in March 2023, including 51,326 users.The authorisation of the relevant users has been obtained at the time of the research, and all data have been anonymized.We tested the reliability and validity of the survey data, evaluated the performance of the DLMaN model, and studied the importance of key modules within the DLMaN model through ablation experiments.

A. QUESTIONAIRE DESIGN AND RESEARCH OBJECTS 1) QUESTIONAIRE DESIGN
The scale designed in this study is based on the Digcomp 2.1, and partially adapt to the current development of digital technology in China.The digital knowledge dimension involves two dimensions with four questions, including ''I know about the use of computers and devices such as mouse and keyboard'' and ''I know about the use of mobile devices such as smartphones'', etc.The digital skills section involves four dimensions with 17 items, including ''I can skillfully use computer-based office software (e.g.Word\Excel\PPT) to operate documents'', ''I can skillfully use online collaboration software to communicate with others'', etc.The digital attitudes section involves 4 questions with 4 dimensions, including ''I understand the use of computers and devices such as mouse and keyboard'' and ''I understand the use of mobile devices such as smartphones'', etc.The digital attitude section involves six questions with two dimensions, including ''I will pay attention to checking the security of payment by scanning codes or links'' and ''I am well aware of the legal principles and ethical boundaries of the online world and will take the initiative to abide by them'', etc.All of the above scales are 5-point Likert-type scale, with 1 indicating strongly disagree and 5 indicating strongly agree.

2) RESEARCH OBJECTS
We took the operator users as the research object.According to the age, income, and region categorization requirements from the National Bureau of Statistics of China, we used the stratified sample sampling approach and use stratified sampling of age, city class, gender, income, and other distributions.In order to ensure the comprehensiveness of the sample coverage, we adopt a combination of online and offline research.Offline questionnaires were used to reach some remote areas, the elderly or underage groups, while a kind of lottery feedback mechanism was set up to ensure the completion rate and effectiveness.
The survey covered a total of 51,326 research samples and the questionnaire survey was conducted in March 2023.According to the screening questions set in the questionnaire, we screened out invalid questionnaires, leaving a valid sample of 49,875, with an effective rate of 97.2%.The specific distribution of the samples is shown in Table 4.

3) RELIABILITY TEST
The reliability test is generally used to gauge whether the questionnaire results are reliable and consistent.Cronbach's Alpha coefficient is mainly used to evaluate the stability of the measurement tool.Nurjannah and other scholars believed that the Cronbach's Alpha coefficient greater than 0.6 indicates that there is a good consistency between the scale items [77].In this paper, we use for data analysis of the distributed questionnaire data.The overall Cronbach's Alpha of the questionnaire is 0.90, which is greater than 0.60, and the overall reliability of the questionnaire is relatively good.The results of the reliability test for each sub-dimension are displayed in Table 5 and 6.

4) VALIDITY TEST
Validity test means that the measurement tool is indeed able to measure what it is intended to measure.High validity indicates that the measurement results can well reflect the real characteristics of the measurement object.It can also ensure that different researchers have a consistent understanding of the meaning and connotation of a research variable.The calculation results showed that the KMO value was 0.829, and the approximate chi-square significance of the Bartlett's sphere test was Sig.= 0. 000 < 0.01, which indicates that there is a good correlation between the scale concepts.
As depicted in Table 7, the KMO values of all variables are greater than 0.6, and the Bartlett values of all variables are 0.00 (<0.05), indicating that the construct validity of this survey is high enough.

B. EXPERIMENTAL RESULTS OF DLMAN 1) EVALUATION METRICS
The goal of the DLMaN model is to make predictions of users' digital literacy, which takes values from 0 to 100, so this is a typical regression problem.In this paper, we use Mean Absolute Error (MAE), Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE) to be the evaluation metrics of model performance.Assuming that the true value of user digital literacy is ŷ, the model predicted value is y, the formula of these three indicators is as follows: These three metrics measure the performance of the regression model from different perspectives.MAE measures the absolute value of the error, which has the advantage of being sensitive to outliers, even if there are large individual deviations, it will not have a great impact on it.RMSE is more sensitive to outliers due to the squaring of the error in its calculation, but its function is smooth and fine-tuned, and  the overall stability is better.MAPE not only measures the deviation between the true value and the predicted value, but also considers the ratio between the deviation and the true value.The result is presented in the form of a percentage and will not be affected by the magnitude of the true value.In evaluating regression problems, these three metrics indicate better model performance as the values decrease, and each metric has its advantages and limitations.Therefore, this paper will comprehensively evaluate the model using these three metrics.

2) EVALUATION RESULTS
According to the strategy guideline of multi-task learning, there needs to be a certain correlation between each subtask, so as to ensure that we can simplify the model parameter quantity by sharing the underlying parameters without causing the model to enter a zero-sum game state.Therefore, before conducting the experiment, we first calculated the correlation coefficients between digital literacy and the second-level indexes, third-level indexes, and fourth-level indexes, as depicted in Fig. 4, 5 and 6.
It can be seen that there are positive and strong correlations between digital literacy and each sub-index.As explained in section III, we initially calculate the fourth-level index scores based on the questionnaire scale data, and then calculate the third-level indexes, the second-level indexes and the overall 5. Correlation between digital literacy and third-level indexes.
digital literacy scores through the application of methods such as entropy weighting.Therefore, the significant correlation observed between digital literacy and these sub-indexes demonstrates the suitability of the training targets derived from scales used in this study for optimizing prediction models employing multi-task learning.By treating the target to be predicted as the main task, and adding the prediction of the sub-indexes as auxiliary tasks, the prior information of the model can be greatly broadened, which helps to enhance the model accuracy.
After confirming that multi-task learning can be carried out, we conducted training and testing of the DLMaN model based on the survey and mobile data.In this experiment, we set aside 25% of the users as the test set and did not participate in the training of the model.The remaining 75% of user data was used as the training set of the model.During training, user profile features, mobile device features, and digital behavior features were used as the input of the model in the form of batches, with digital literacy as the main task learning objective, and sub-indexes for the digital literacy as the learning objectives for the subtasks.The model used the Adam optimizer to update the parameters, the learning rate was set to 0.007, and 300,000 iterations were performed.During training, we monitored the changes in model performance by observing huber loss and the three metrics, and stopped iterations in time after the loss levelled off to avoid model overfitting.In this study, digital literacy includes the second-level sub-indexes, the third-level sub-indexes and the fourth-level sub-indexes.When building a multi-task learning model, we are able to choose the predictions of the second-level indexes as auxiliary tasks, or choose the predictions of the third-level indexes or fourth-level indexes as auxiliary tasks.In order to find the optimal multi-task learning construction scheme, we used the second-level indexes, third-level indexes, and fourth-level indexes as auxiliary tasks to test the effect of the DLMaN model, and also tested the single-task learning model depicted in Fig. 3.All models underwent a 5-fold cross-validation during training, and after training, we evaluated the performance on the test set.The results are shown in Table 8.
It can be found from the table that when using multi-task learning, the model performance is the best when choosing third-level indexes as auxiliary tasks, and the MAE, RMSE and MAPE can reach 4.759, 5.233 and 8.65% respectively.The prediction error of the DLMaN model for digital literacy is already lower than 10%, which also reflects the effectiveness of the national digital literacy assessment framework FLAKE proposed in this paper.After surveying around 50,000 users, we have developed a digital literacy prediction model with an error rate of no more than 10%.This model can directly predict digital literacy based on mobile big data such as users' digital behaviors, and can realize digital literacy evaluation of millions, or even tens of millions, of users without incurring additional time or economic costs, which is of great significance for carrying out large-scale digital literacy assessment.
In addition, we compare the effects of various methods in Table 8, and find that after using multi-task learning, the model performance is significantly better than the traditional single-task model.For the single-task model, the training data of the model is only about 50,000, which is not quite enough for a deep learning model to be fully trained.However, with multi-task learning, the number of training samples of the model is expanded by N times, and N is the number of subtasks.By increasing the prior information and the number of training samples, the model can be adequately trained and converged, thereby improving its prediction accuracy.Besides that, we find that when using multi-task learning, the model with the third-level indexes as auxiliary tasks performed better than the second-level indexes and the fourthlevel indexes.The lower the level of index, the more indexes it contains, which can reasonably introduce more prior information to the model, but the increase in the number of auxiliary tasks will also make the structure of the DLMaN model more FIGURE 7. The model performances under different feature combinations.It can be concluded that digital behavior features play a critical role, followed by user profile features, the impact of mobile device features is relatively low.complex, and the number of parameters to be learned will also increase.Although we can share the parameters of the bottom layer, such as the embedding layer, the subsequent hidden layer parameters cannot be shared.Too many auxiliary tasks will make the model structure bloated and more difficult to train, so it is not the case that the more tasks, the better the model performance will be.When using the model, we also need to adjust the auxiliary task structure according to the actual conditions.
In order to learn the importance of each key module in the DLMaN model, we conducted ablation experiments to explore the model performance in different scenarios by changing the model structure and the composition of the input features.The input of the DLMaN model includes user profile features, mobile device features, and digital behavior features.After recombining these three types of features, we input them into the model and observed the changes in the model performance.The results are shown in the Fig. 7.It can be found that if we delete some features, the model performance will become worse, so these three types of features are all helpful for the prediction of users' digital literacy.In addition, when we only use digital behavior features, the performance decreases the least, which indicates that digital behavior features, that is, the user's app usage information, are the most closely related to digital literacy.When these features are deleted, even if the other two are both used, the model's error will nearly double.As we discussed in section III, according to the iceberg theory, digital behavior are the explicit manifestation of users' digital literacy and the most important features in big data for predicting digital literacy, while the other two groups of features just play an auxiliary role, which is also consistent with the design principle of our FLAKE framework.When obtaining user profile and mobile device features is challenging, relying solely on digital behavior features to predict digital literacy can yield similar performance effects.
In addition, according to the introduction in part C of section III, we added an attention fusion module when designing the DLMaN model, so as to assign different weights to different auxiliary tasks in the forward pass process.We also utilized the dynamic weight averaging method to assign dynamically adjusted weight parameters to the loss values of the auxiliary tasks when calculating the loss in the backpropagation of the model.Both of these designs are intended to allow the DLMaN model to better balance the relationship between various auxiliary tasks, avoiding that one or more tasks are trained too slowly to the extent of affecting the effectiveness of the overall model.As shown in Fig. 8, we find through ablation experiments that when we apply these two methods at the same time, the performance of the model is the best.After deleting one, the performance of the model will decline.Among them, the performance decreases more proportionally after removing dynamic weight averaging.This suggests that both methods have a positive effect on the model, and that dynamic weight averaging has a greater effect because it directly affects the design of the loss function, which in turn adjusts the parameter update of the model during the backpropagation process.

V. DISCUSSION
After completing the training and testing of the DLMaN model, we applied the digital literacy prediction model to a large-scale dataset containing approximately 2.3 million users.This dataset covers diverse populations in China, including various genders, age groups, regions, and income levels.We extracted digital behavior and other mobile big data from these users, and used the DLMaN model to predict 108672 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.their digital literacy.In this section, we will conduct a detailed discussion and analysis of the evaluation results of digital literacy based on this massive user dataset, exploring the issues of the digital divide and digital inclusion in China.

A. RATIONALITY PROOF OF RESULTS
According to the results of our measurement, the average score of Chinese citizens' quality is 56.4,and the standard deviation is 15.59.We compared the analysis result of digital literacy score predicted by the DLMaN model with the empirical research results of previous scholars on digital literacy.So as to highlight the consistency of the results and verify the rationality of the DLMaN model.

1) COMPARISON OF GENDER DIFFERENCES
According to Table 9, females exhibit higher levels of digital literacy compared to males, with a score of 57.2 for females and 55.5 for males in terms of overall digital literacy.Research has indicated that females excel in interpersonal communication, language expression, and empathy, outperforming males in these areas.Consequently, females have a developmental advantage over males in terms of digital literacy performance.The recent measurement results from the EU in 2021 suggest a 7.03% higher score for females compared to males.The similarity between these findings and our own results validates the rationality of our conclusions.

2) COMPARISON OF AGE DIFFERENCES
We categorize the age groups as follows: group aged 12 to 17 are classified as underage, group aged 18 to 35 constitutes the youth, group aged 36 to 50 represents the middleaged, and those above 50 years of age comprise the elderly group.As depicted in Table 10, the highest digital literacy score is observed among the youth group, followed by the middle-aged and underage cohorts, while the elderly group presents the lowest score.Overall, the relationship between age and digital literacy scores follows a U-shaped distribution pattern.As shown in Fig. 9, comparison with the measurement results from the EU in 2021 reveals a similar characteristic data distribution.Notably, there exists a significant disparity in scores between the elderly group and the other cohorts.Addressing the integration of the elderly population into the digital society, promoting social inclusiveness, and achieving technological inclusivity are pressing issues that require immediate attention.

B. KEY FINDINGS
Through the aforementioned analysis, we have found a widespread digital divide issue in China, which has been observed by other scholars in their own research as well.However, most of the current studies are based on small samples and primarily focus on subjective survey data, making it challenging to measure the differences in macro-level dimensions of the digital divide or to precisely describe the relationship between digital behavior and digital literacy.To address this limitation, our study proposes the FLAKE framework, which utilizes evaluation results based on a large sample of user behavior data, effectively mitigating this issue.By applying the FLAKE framework, we have discovered some new forms of digital divide are emerging with a widening trend.As urbanization progresses, the disparity between developed and underdeveloped regions continues to expand.Developed areas leverage their resources, educational infrastructure, and healthcare advantages to assimilate digital talents from less developed areas, thereby enhancing their own growth potential.Simultaneously, the growing wealth gap further perpetuates a high level of digital literacy among the affluent, enabling them to adapt to technological advancements and extract more dividends from societal progress.

1) NEW DIGITAL DIVIDE BETWEEN REGIONS
From the perspective of urban development and residents' income level, as the differences in regional and individual economic development further intensify, new types of digital divide are also emerging.
Based on the results of DLMaN model, Fig. 10 shows the elderly in developed regions have lower digital literacy scores than those in less developed regions, whereas other categories have greater scores in developed regions than in less developed ones.It demonstrates how the state of the local economy has a significant impact on digital literacy.The effect is subject to some restrictions.The distinct characteristics of China's historical development may be the cause of the variations between the older groups in China.In the 1970s, a societal phenomenon occurred where intellectuals migrated from urban centers to rural areas, resulting in a large number of educated youth moving to underdeveloped regions.This may explain why the digital literacy scores of the elderly in less developed areas are higher than those in developed areas.As times progress, the influence caused by the particularity of the times will gradually be eliminated, and the new generation of youth groups will become the main force in society.Therefore, it is essential for the government to closely monitor the growing digital divide between regions.To prevent the creation or even the growth of the new kind of digital divide, they should create an inclusive society marked by sharing, equality, and openness.

2) NEW INTRA-REGIONAL DIGITAL DIVIDE
As depicted in Table 11, the digital literacy gap among youth in undeveloped regions is more pronounced, indicated by a higher coefficient of variation compared to youth in developed regions.This can be attributed to the better digital environment and more complete digital infrastructure in developed regions, which leads to variations among younger groups.The influence of family economic circumstances on developmental variations among young people is relatively limited, with societal digital environment having a greater impact.Therefore, the difference in quality among young people is relatively small.In undeveloped region, due to the differences in digital infrastructure, the development of youth groups is more dependent on the family environment.This further accentuates the differences between groups.Furthermore, with the future impact of talent migration between regions, the development gap between developed and undeveloped regions is expected to widen.This will result in new digital divide disparities among youth groups in underdeveloped regions.As youth groups gradually grow into major social roles, the difference will be reflected in all age groups in undeveloped regions.In light of these circumstances, it behooves the government to enact legislation that embraces comprehensive social equity, providing equitable chances for youth organizations in underdeveloped areas to fully participate in the digital society.

3) NEW DIGITAL DIVIDE CAUSED BY INCOME
From Fig. 11, we can see that the digital literacy of low-income groups in developed regions is lower than low-income groups in underdeveloped regions, while the digital literacy of high-income groups in developed regions is higher than high-income groups in underdeveloped regions.The promotion effect of digital literacy on income is better than the selection of regions.Even if the low digital literacy groups work in developed regions, they cannot obtain better income treatment than those in underdeveloped regions.Therefore, in the process of future social development, digital literacy will become an important element of individual sustainable development, and groups without excellent digital literacy will find it even more difficult to obtain excellent employment opportunities.It can be seen that in the future development process, the digital literacy enrichment effect of high-income groups will continue to intensify, which will further affect the current economic distribution and further increase the social gap between rich and poor.Addressing the emerging threats of the digital divide demands urgent attention from the government.In order to support sustainable societal progress and ensure fairness, it is vital to provide low-income individuals with access to digital literacy training programs.Such initiatives empower them to enhance their skills and enhance their employability.Through promoting equal access to skill development and harnessing technology as a catalyst for advancement, the government can enhance societal well-being while fostering an inclusive and harmonized socio-economic environment.

4) NEW DIGITAL DIVIDE ASSOCIATED WITH DIGITAL BEHAVIORS
Based on the DLMaN model's prediction results, it can be found that there is a correlation between digital behavior and digital literacy, and groups in different regions show different digital behaviors.The data in Fig. 11 reveals that mobile applications for social, knowledge, and online education are generally more important in fostering personal digital literacy.In groups of different city levels, the degree of positive effect of user digital behavior on digital literacy is different.The influence coefficient of social APP usage behavior in developed regions is significantly higher than that in underdeveloped regions.That indicates the ability to skillfully use social mobile applications in developed regions is the key factor for the development of individual digital literacy.In developed regions, the urban geographical area is large, the wide range of urban construction bring about the pressure of communication.Understanding how to use  social mobile apps is essential for integrating them into urban development.An individual's digital literacy is greatly influenced by regular exposure to knowledge and online educational applications, as exemplified by the significantly higher coefficient of influence on usage behavior observed in underdeveloped regions as compared to their developed counterparts.Leveraging digital technology for online educational resources in underdeveloped fosters equitable education access and abates regional disparities.This mitigation of inequality-induced imbalances promote a more inclusive and fair education landscape.
Therefore, it is imperative for the government to proactively foster the advancement of online educational platforms, ensuring equitable access to educational resources while focusing on underserved regions including rural, remote, impoverished, and underprivileged areas.To ensure comprehensive and efficient outreach to disadvantaged groups across various educational institutions, the national education funding policy system's online education capabilities need to be enhanced.The government should actively promote the growth of online schools and online education in challenging areas, constantly striving for further development and enhancement.

C. IMPLICATION AND LIMITATIONS OF FLAKE
Based on the above results, it can be concluded that the FLAKE framework proposed in this paper, which combines mobile big data and survey data for assessment, can better and more comprehensively predict individuals' digital literacy.This framework also has implications for other similar social surveys.On one hand, utilizing big data of digital behavior for prediction can effectively avoid the subjectivity issues associated with questionnaire surveys, thus enhancing the credibility of the results.On the other hand, the predictive results based on big data have advantages in sample size and analysis dimensions, which enable a more accurate reflection of the macro-level digital literacy levels, avoiding problems such as unbalanced sampling in survey questionnaires.This approach can effectively explore potential digital divides among different regions, income levels, and groups with different digital behaviors.Furthermore, it promotes the development of digital literacy assessment towards universalization.
The FLAKE framework also provides an effective data analysis approach for governments, helping them address macro-level data gaps and timely identify disparities in regional development, talent reserves, and other aspects.This framework enables governments to make corresponding policy adjustments to achieve social equity and sustainable development.Moreover, this research can assist schools in understanding students' level of digital literacy in a timely manner and grasping the dimensions of digital literacy that individuals should improve upon.By providing specific behavioral guidance, it helps students adapt better to the development of the digital society and lays the groundwork for subsequent issues such as employment of talents.
The limitations of our study can be primarily viewed from two dimensions.Firstly, it lies in the limitation of data hierarchy.This study specifically focuses on analyzing the correlation between mobile application behavior and digital literacy.However, mobile application behavior alone cannot fully represent an individual's overall digital life development.In future research, incorporating more comprehensive digital life data into this framework could enhance the predictive accuracy.Secondly, methodologically, the FLAKE framework primarily applies to social research using scales and measures.When extending this method in future studies, certain adjustments and optimizations should be considered.Exploring the integration of other social research methodologies with big data can expand the application space of this framework.

VI. CONCLUSION
In this paper, we propose a deep learning-based national digital literacy assessment framework named FLAKE.The FLAKE framework conducts survey on selected users, utilizes scales to calculate their digital literacy, and subsequently develops a digital literacy prediction model, DLMaN, using multi-task learning and neural network training based on the digital behavior big data of these users.By employing the DLMaN model, the digital literacy evaluation research of an extraordinary number of users can be carried out leveraging mobile big data.We conducted testing on both anonymous survey datasets and large-scale mobile datasets to assess the performance of the framework and model.The prediction errors (MAE, RMSE, and MAPE) of the DLMaN model were found to be as low as 4.759, 5.233, and 8.65%, respectively.This demonstrated a strong correlation between citizens' digital literacy and their digital behavior, where intrinsic literacy can be inferred through external actions.Furthermore, we utilized the DLMaN model to assess the digital literacy of millions of citizens, followed by in-depth analysis of the results.We have found that there exists widespread digital divides among different groups in China, including age, gender, region, and income.We discussed strategies for eliminating existing inequalities and preventing new digital disparities within our country and society.Our ultimate goal is to provide recommendations for achieving sustainable development at the social level in the digital age.The FLAKE framework proposed in this paper enables a cost-effective and comprehensive survey of digital literacy.It combines deep learning techniques and big data mining with social research, making it suitable for national and city-level survey work.The framework also demonstrates practicality, portability, and reference value in social research in various domains.

FIGURE 1 .
FIGURE 1.The structure of FLAKE, the national digital literacy assessment framework.It consists of two parts: (1) utilizing a questionnaire to assess the digital literacy of a subset of research participants, and (2) training a predictive model using digital behavior data from big data, enabling the evaluation of digital literacy for a large user population.

FIGURE 3 .
FIGURE 3. The structure of the single task model.The accuracy and generalization of the model cannot be improved by utilizing multiple structures in parallel and parameter sharing.

TABLE 7 .
The validity analysis results in the survey questionnaire.The KMO value are all above 0.7 which indicates the results of survey are valid.

FIGURE 4 .
FIGURE 4. Correlation between digital literacy and second-level indexes.

FIGURE 6 .
FIGURE 6. Correlation between digital literacy and fourth-level indexes.

FIGURE 8 .
FIGURE 8.The model performances under different methods.From these results, it is evident that attention fusion and dynamic weight averaging are crucial for model performance, particularly the dynamic weight averaging technique.

FIGURE 9 .
FIGURE 9. Comparison between digital literacy of EU and this study.It can be found that the scores have a similar distribution in different age groups.

FIGURE 11 .
FIGURE 11.Compare the digital literacy scores among different income groups between developed and underdeveloped regions.It comes to the conclusion that the group of low-income in underdeveloped regions' score is higher than developed regions.

FIGURE 12 .
FIGURE 12.Digital literacy scores associated with digital behavior between residents in developed and underdeveloped regions.

TABLE 2 .
Index system of digital literacy.It consists of three second-level indexes, eight third-level indexes, and sixteen fourth-level indexes.

TABLE 3 .
Features of mobile big data.

TABLE 4 .
The distribution of the 49,875 samples surveyed by this research.

TABLE 5 .
The reliability analysis results of second-level indexes, which shows the survey results are reliable.

TABLE 6 .
The reliability analysis results of third-level indexes, which shows the survey results are reliable.

TABLE 8 .
MAE, RMSE and MAPE of different methods.It can be observed that when using third-level indexes to construct the multi-task model, the performance of each metric is the best.

TABLE 9 .
The digital literacy scores predicted by the DLMaN model note that there is an obvious difference between gender groups.

TABLE 10 .
The digital literacy scores predicted by the DLMaN model note that there is an obvious difference among age groups.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.Digital literacy scores among different age groups between developed and underdeveloped regions.Obviously, the score of the developed regions group is higher than undeveloped regions.But the old age group score has the opposite conclusion.

TABLE 11 .
The digital literacy scores for youth predicted by DLMaN model note that there is a digital divide between developed and underdeveloped area in youth.