Evaluating Quality Characteristics of Ubiquitous Application Through Means of Quality Models using Meta-metrics Approach

Ubiquitous computing shifted the way how users interact with applications. The demand of information anytime and anywhere impacts the daily life of its users, be it work related or personal. Difficulty arises when determining the quality of ubiquitous application due to lack in appropriate metrics of quality models, which serves as the motivation behind this paper. The aim of this paper is to assess the quality of ubiquitous application using comparative analysis of quality model metrics via meta-metrics approach. Preliminary review mapping was conducted where distinctive quality characteristics of ubiquitous applications from AQUARIUM model are identified. Metrics mapping was then conducted to compare metrics characteristics with quality characteristics via value assignment using meta-metrics technique. Results shows that most of the metrics mapped are not of definitive derivation, providing opportunity to have a more structured and defined measurement function.


Introduction
Ubiquitous computing is a relatively new concept introduced by Mark Weiser back in 1991. The main concept is for technology to be an enabler in providing services and information by supporting its users in tackling daily tasks with zero or minimal interaction between the users and the technologies used. The implementation of ubiquitous application is not constrained to individual users only, it could also be used to be utilized in other areas such as e-health, education and environmental monitoring [2]. To achieve this main objective, the technologies used need to recognize and learn the user's activities in order to be familiarized and self-adapt in accordance to the user's needs. The advancement and combination of different technologies such as wireless technologies, artificial intelligence, image processing, voice recognition and others play important roles in characterizing the features of ubiquitous applications across domains. These characteristics will determine how users interact with ubiquitous applications and further serves as important element in evaluating human computer interaction of these applications.
The effective usage of this concept would also help in curbing information overload faced by users in making decisions pertaining their daily activities. In fact, Bilal et. al [3] indicated that one of the reasons why projects failed was because absence of quality architectural and design documents to help guide the developers on building a more usable product.
Ubiquitous computing is also similarly termed as having interaction occurring anytime and anywhere; which put mobility as the forefront possibility of interaction to happen in any context with multiple devices. Mashal et. al [5] associated ubiquitous computing concept with Internet of Things (IoT) which seemingly has very similar concept of connectivity of smart things to provide effortless decision making in everyday life while causing as minimal as possible distraction to its users. Some authors used the term 'ubiquitous computing' and 'pervasive computing' interchangeably. Although the primary definition is similar as of Weiser's in 1991, Sadiku et al [6] argued that the concepts between these two terms are different, which leads to differences in defining characteristics of ubiquitous application.
In characterizing ubiquitous application, one must understand the nature of the application itself. Sadiku & Wang [7] describes ubiquitous application from a high level perspective concerning hardware, software and its functional and non-functional requirements. Interestingly, Sadiku & Wang [7] also mention adaptation of behaviour based on information detected from the surrounding environment, which is termed as "situatedness". The term is later known as context-awareness. Dey [8] also mention the importance of recognizing context-awareness as ubiquitous system characteristics as it relates too well with understanding the behaviour of the users to provide relevant services and adapt to the need of the users while taking into considerations the surrounding environment's activities. Considering the unique characteristics of ubiquitous application, this study attempts to assess the quality of ubiquitous application using comparative analysis of quality model metrics via meta-metrics approach.

Related Research
Software quality is a degree of a software product fulfils the requirement stated when used under specified conditions [9]. The quality of particular software is often benchmarked against prominent quality models such as McCall, Boehm, Dromey and ISO standards [10] . Gordieve et. al [11] define software quality model as a set of characteristics which serves as a basis to indicate requirements and evaluation of quality. Many software quality models have been introduced in the past, which indicated various level of hierarchy and characterization of quality elements [12].
Assessing and determining the quality of software system can also be applied through proper software measurement approach [13]. The term software measurement is described as a process to outline, gather and examine software measures about software development process with the objective of improving the process and product itself. The measures are associated with quality characteristics, which translates to elements that the users desire of a product [14]. An example of quality characteristic of ISO/IEC 25010 is maintainability which is defined as the degree of effectiveness and efficiency with which a product or system can be modified by the intended maintainers [15]. Carvalho [1] further explain that quality characteristics are categorized in software quality models, and often represented in a hierarchical manner comprised of characteristics, sub-characteristics and software measures as per exemplified in figure below. It is a concern that ubiquitous characteristics need to be considered in the evaluation process of a software product. This will contribute to a more enriched user experience and increase adoption rate of the application by the users.

Evaluation through means of quality models
The increased awareness of importance of evaluation methods is driven by improved return on investments from the implementation of these systems [16]. Carvalho et. al [1] in their research indicate two categories of evaluation methods that can be carried out in human computer interaction field namely expert based evaluation and user based evaluation. Each type of evaluation can be carried out by utilizing different techniques or formal-model approach using formalized quality models.  [17]. ISO/IEC 25010 is released under the title "Systems and Software Quality Requirements and Evaluation (SQUARE)" which specifies quality of a software need to be examined through three life cycles, namely during product development, testing and usage of system in realistic context [18]. As with ISO 9126 model, ISO/IEC 25010 evaluate the quality of software from two perspectives: product quality model and quality in use [15].
ISO/IEC 25010:2011 Systems and software Quality Requirements and Evaluation (SQuaRE) -System and software quality models is regarded as the frontrunner of quality models to determine quality of software products [15]. The aim of this model is to evaluate quality from two perspectives: software quality requirements specifications and system, and software quality evaluation. Compared to its predecessor ISO9126, ISO/IEC 25010 has simplified the division of software quality characteristics into two quality model: Product Quality and Quality in Use. Product Quality is associated with characteristics relevant to the properties of the software product, while Quality in Use relates to the degree to which a product or system can be used by specific users to meet their needs [19].  Figure 2. ISO/IEC 25010 characteristics and sub-characteristics Quality in Use model is more relevant to this study as it represents the software quality evaluation elements as compared to Product Quality model. In ISO/IEC 25010 Quality in Use model, effectiveness refers number of actions required to complete the subtasks of each task in a specified context of use. Effectiveness is measured in actions per task. The term efficiency is described as the efficiency of the user in completing the task in a specified context of use, measured in actions per second. Satisfaction is degree to which user needs are satisfied when a product or system is used in a specified context of use. Freedom from risk is termed as the safety to the user, in terms of number of errors committed in each section of each task performed in a specified context. Freedom from risk is measured by errors per action. And finally, context coverage is the degree to which a product or system can be used with effectiveness, efficiency, freedom from risk, and coverage satisfaction in both specified contexts of use and in contexts beyond those initially explicitly identified.
Quality measurement is a complicated and challenging process as it incorporates the viewpoint of users which usually is regarded as a subjective element. ISO/IEC 25010 quality model attempts to assess quality of software beyond measuring the product's requirements, hence the sub-division of Quality in Use model is integrated under the ISO/IEC 25010 umbrella. However, it is essential to mention that Product Quality has a direct impact and significant influence on Quality in Use measurement [20]. Therefore, this will increase the complexity to measurement process itself.

ISO/IEC 9241-11.
Originally published in 1998, ISO 9241-11 standard is used to provide guidance on usability. ISO 9241-11 describes usability in terms of effectiveness, efficiency and satisfaction. Bevan [21] further define usability as "The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use". From business requirement perspective, effectiveness can be translated to success in achieving goals, efficiency indicates not wasting time and satisfaction represents willingness to use the system [21,22].
ISO 9241-11 1998 has undergone a change in the revised version to include a more comprehensive definition of terms. Although the attributes remain the same (effectiveness, efficiency, satisfaction), Redefined as the resources (time, human effort, costs and material resources) that are expended when achieving a specific goal (e.g. the time to complete a specific task).

Satisfaction
Defined as freedom from discomfort, and positive attitudes towards the use of the product.
Redefined to highlight a much wider range of personal responses: the extent to which attitudes related to the use of a system, product or service and the emotional and physiological effects arising from use are positive or negative.

QiU-4-MUI.
QiU-4-MUI is a measurement model to investigate quality in use aspects of medical user interface applications [23]. The main motivation behind this study is the scarcity of measurement model to investigate the influences of HCI principles on Medical User Interface (MUI) design in healthcare domain. This model is derived from the mapping study between Quality in Use model from ISO/IEC 25010 and HCI principles illustrated below which forms as basis for QiU-4-MUI quality characteristics.    [24] conducted a study to understand the influence of u-commerce technologies to users, leading to creation of quality model to assess ubiquitous commerce applications.
Olaleye & Oyelere [24] propose a conceptual model for this purpose which consists of five elements: Usability, Privacy, Trust, Reputation and Experience. However, as with other quality models such as QiU-4-MUI and HCI quality model by Assila & de Oliveira [25], the definition for quality characteristics in this model originate from existing quality characteristics of various sources of literatures.
Usability characteristic is for this model is taken from HCI principles which indicates the degree in which a product can be used in a specific application context by a specific user to achieve a specific goal with effectiveness, efficiency, and satisfaction. Privacy is termed as user's desires to be able to make decisions in different environment and situation. It is also referring to the degree of exposure with regards to behaviors and attitude, to others. Olaleye & Oyelere [24] also enlist reputation as one of the characteristics which is defined as an assessment to determine a company's credibility, by examining the gap between the company's promises and fulfilment. Trust is a significant element for decision making, particularly involving online transactions. However, there is no definite description provided by Olaleye & Oyelere [24] for this characteristic. The final characteristic is user's experience, denoted as perception of a user from using a product, service or information system. Interestingly, this is the first quality model for ubiquitous systems which examines relationships between these elements. Olaleye & Oyelere [24] propose four hypotheses to study the relationships between the five elements as below: Usability of ubiquitous mobile devices will positively influence the shopping experience of u-commerce users.
H2: Privacy confidence of online shopping will positively enhance the experience of ucommerce users.
H3: H3a: The reputation of merchants and vendors of u-commerce will positively influence the confidence of customers.
H3b: The higher the reputation, the stronger the relationship between trust and ucommerce experience H4: A trusted u-commerce platform will positively influence online shopper's experience.

TRUU Model.
Santos & de Oliveira [26] created a quality model named TRUU to support the evaluation of human computer interaction in ubiquitous application. TRUU model consist of four main characteristics which are further expanded to sixteen sub-characteristics. TRUU model is originated from the analysis of extensive systematic mapping study with the objective to design a quality model fit to evaluate interaction in ubiquitous application. The definition of sub-characteristics of TRUU model is presented below.

Trustability
Security Assurance on secured information transfer and storage. Deals with access, utilization and modification control of context information.

Privacy
Capability to sustain protection on information and data Control Empowering users to control how data is utilized; as well as providing alternative options of resources should the data is mistreated Awareness Effortless management with multiple and concurrent users with regards to understanding of usage of data and implication of application to users Resource limitedness Device capability Device-based properties associated to executing the application such as screen size and battery life. Network capability Network information impacting the execution of application Usability Satisfaction The extent of user satisfaction from the system. Also deals with attractiveness of the system.

Ease of use
The targeted user group should have minimal difficulties in operating the system. Efficiency Completion rate (time taken with regards to effort and resources) upon tasks execution Effectiveness Completeness upon tasks execution Familiarity Enabling the improvement of quality of work via user interaction with the system. Also deals with aesthetic value of the application.

Ubiquity
Context awareness he ability of the system to use context to provide relevant services to user, where relevancy depends on the user's task Transparency The extension of the system which consists of hidden components in the physical space and interaction is performed through natural interfaces" Availability System's ability to provide uninterrupted access to information resources regardless of time and location. Focus Ability to maintain user's attention to the main task Calmness Avoid users from information-overloading Compared to other software quality model in this section, this is the first attempt to devise a model specifically to assess interaction in ubiquitous system. Santos & de Oliveira [26] believes that all TRUU elements are representation of interaction characteristics of ubiquitous application.

AQUARIUM Model.
In a more recent study, Carvalho [1] crafted AQUARIUM model with the objective to propose a well-defined software measures to assess HCI in ubiquitous application. It uses similar methodology to collate definition of characteristics as per TRUU model, however the sources are of a more recent literatures. Initial findings from the systematic mapping study suggested 27 quality characteristics for ubiquitous application, however from closer inspection, AQUARIUM model indicate five characteristics specifically associated to quality interaction of ubiquitous application namely context-awareness, mobility, attention, calmness and transparency. The characteristics are further extended to sub-characteristics as and assigned with software measures through an extensive and complex Goal-Question-Metric approach [27].

Methodology
A review mapping between several quality models were carried out to determine the set of quality characteristics for ubiquitous application. Following which the quality characteristics identified from the selected model will be used to evaluate quality using meta-metrics methodology.
Meta-metrics is an approach to identify suitable metrics to be used in quality evaluation based on specific characteristic of the metrics themselves [28]. The metrics characteristics are categorized into 10 characteristics which serves as the criteria for evaluation of system as well. Software metrics scales such as nominal, ordinal, interval, ratio and absolute.

"+" metrics on nominal and ordinal scale "-" metrics on absolute, ration and interval scale Measurement Independence (MI)
The capability of a metric to suggest the similar outcome when measured by different types of users. "+" metrics which are always measured in the same way "-" metrics for varying data collection Automation (AU) The capability of a metric measurement using a tool "+" metrics that are automated easily "=" metrics which require significant effort "-" metrics that cannot be automated Simplicity (SI) The ease of software metric. This is a measure of the clearness of the metric's meaning in order to examine how easily this definition can be understood and facilitates actions in the evaluation plan "+" for very well defined metrics "=" for fairly defined metrics "-" metrics that are difficult to be understood and interpreted Accuracy (AC) The accuracy of software metric. The metric should accurately measure what is supposed to be measured. "+" metrics with high accuracy "-" metrics with low accuracy Cost (CO) The cost to perform the metric evaluation . Depends on multiple combination of factors to extract the value of the metric. "+" requires high cost to be implemented "=" requires medium cost to be implemented "-" requires cost to be implemented Evaluation (EVAL) Assessment process type which users delivering their response through a workshop (controlled environment) or at home (uncontrolled environment) "+" needs implementation in controlled environment "-" can be implemented in uncontrolled environment

User Type (UT)
The category of user involved in the calculation of the value of a metric "+" evaluation experts and/or developers "=" mixed

Software metrics characteristics Descriptions and Value Assignments
"-" novices Target (TA) A metric may measure either data or a process "+" process metrics "-" data metrics Persuasion (PE) Metrics are directly associated to one or more quality sub characteristics of quality model such as ISO9126 "+" metrics that are associated with more than two subcharacteristics "="metrics that are associated with more two subcharacteristics "-"metrics that are associated with more one subcharacteristics Upon value assignments to each metrics' characteristics, the metrics are then mapped to quality characteristics of ubiquitous application studied across different quality models. With this methodology, ubiquitous application is examined on the connection between metrics, meta-metrics and quality characteristics and giving values to each element. This method also provide a guideline on proper selection of metric (what, where and how) when performing assessment of application. The advantage of using this method is that the comprehensiveness of evaluation performed as the software metric is evaluated based on specific characteristics of the metrics themselves.

Results and Discussion
Over the years, researchers have attempted to design quality models to assess a ubiquitous application from holistic perspective. Although ISO/IEC 25010 and ISO 9241-11 are considered as one of the most widely known and used standard for quality evaluation of software product [11, 15,29], it lacks the elements to assess ubiquitous application characteristics, hence making it unfitting to serve as quality evaluation benchmark for this type of application. Since ISO/IEC 25010 marks as a significant software standardization for quality evaluation [11], it is used to reflect as the ground of comparison against other quality models to evaluate ubiquitous application.
Although initial mapping of quality models for ubiquitous application to ISO/IEC 25010 has been constructed, it is worth to note that the mapping is constructed using sub-characteristics of some of the quality models. Although the terminology of benchmarked model is recognized in terms of its international standardization, it is not a solid assurance that the mapping is accurately done as different quality models have different interpretation of definition or terminology of these characteristics. Based on the mapping, AQUARIUM model posses the most quality characteristics of ubiquitous application, compared to other quality models. Hence, quality characteristics of AQUARIUM model were used to indicate the quality of ubiquitous application using the meta-metrics approach.  From the results depicted as above, we conclude that most of the quality characteristics require more in-depth approach of definite metrics derivation. Although some are measurement independent, the metrics that characterizes ubiquitous application should be more clearly structured with defined measurement function.

Conclusion
Ubiquitous computing applications are designed with the intention to provide continuous and shared access regardless of devices, time and location. The assessment of quality for ubiquitous application is mainly performed through benchmarking against quality model. However, a very relevant issue exists with regards recommendation of quality model for ubiquitous application as it lacks the elements characterizing ubiquitous application itself, especially on quality of interaction as the interaction between users and ubiquitous application is uniquely termed and defined as compared to traditional systems. Studies on quality model for ubiquitous application have indicated the insufficiency of quality interaction characteristics pertaining to ubiquitous computing systems. Even with the characteristics defined, some characteristics lack a defined measurement function to accurately assess the element. To further encourage adoption of ubiquitous application usage, there is a crucial need to consider having the HCI quality measurement model for ubiquitous application to assess its quality.