System performance requirements: A standards-based model for early identiﬁcation, allocation to software functions and size measurement

Background: In practice, the developers focus is on early identiﬁcation of the functional requirements (FR) allocated to software, while the system non-functional requirements (NFRs) are left to be speciﬁed and detailed much later in the development lifecycle. Aim: A standards-based model of system performance NFRs for early identiﬁcation and measurement of FR-related performance of software functions. Method: 1) Analysis of performance NFR in IEEE and ECSS standards and the modeling of the identiﬁed system/software performance functions using Softgoal Interdependency Graphs. 2) Application of the COSMIC-FSM method (e.g., ISO 19761) to measure the functional size of the performance requirements allocated to software functions. 3) Use of the COSMIC-SOA guideline to tailor this framework to service-oriented architecture (SOA) for performance requirements speciﬁcation and measurement. 4) Illustration of the applicability of the proposed approach for speciﬁcation and measurement of system performance NFR allocated to the software for an automated teller machine (ATM) in an SOA context. Result: A standards-based framework for identifying, specifying and measuring NFR system performance of software functions. Conclusion: Such a standards-based system performance reference framework at the function and service levels can be used early in the lifecycle by software developers to identify, specify and measure performance NFR.


Introduction
Over the years, system non-functional requirements (NFRs) from a variety of stakeholders have significantly increased the urgency and effort required to deliver software systems with very high-quality levels. The large and diverse body of literature on software quality and NFR makes it challenging for practitioners to figure out detailed reference works to use as a baseline for early identification, specification and measurement of any of the large number of NFRs.
Developers must take into consideration both system functional user requirements (FURs) and non-functional requirements (NFRs) early in the system requirements analysis in order to then allocate them at the software/hardware FR level [1][2][3][4][5][6] (see Figure 1).
The success of a software project depends heavily on its ability to be executed with the required functionalities while under specific constraints. Software functionalities fall under the concept of functional user requirements (FURs) and refer to the set of functions or services required from the system and allocated to the software, while constraints fall under the concept of non-functional requirements (NFRs).
In practice, requirements are usually addressed at the system level [1][2][3][4] at the start of the project either as high-level system functional user requirements (system-FURs), or as high-level system-NFRs. The latter must typically be detailed, allocated, and implemented in either hardware or software, or both -see Figure 1.
Software engineers focus on software-FURs for the early development phases, while system-NFRs are typically discussed at later development phases, such as evaluation or testing phases. To distinguish between these types of requirements, the term system-FURs is used to describe the required functions in a system, while system-NFRs is used to describe how the required functions must behave in a system.
In the software requirements engineering phase, system-NFRs are analyzed and detailed, and some may be specified as Software-FURs to allow a software engineer to develop, test, and configure the final software deliverables to system users. It should be noted that a number of such system constraints, while referred to as system-NFRs by some authors, are referred to as quality aspects by other authors.
A number of researchers have investigated issues related to NFR, such as considering them as measurable inputs to effort estimation models [1], which, although based on a different point of view, can be used concurrently with FUR, including their procedures and approaches.
This paper specifically addresses system performance NFR and extends our previous research on three other types of NFR: security [2], portability [3] and maintainability [4].
A key strength of the approach in our previous work is that it is based on the consensus documented in international standards, such as the European Cooperation for Space Standardization (ECSS), the Institute of Electrical and Electronics Engineers (IEEE) and ISO on a number of such NFR, and our proposal for a standards-based reference model for specific types of NFR.
The contribution of this work is a standards-based measurement framework of system performance requirements to be used by de- Figure 1. System performance-NFR allocated to software performance-FUR velopers in the early development stages as a generic model for the identification, specification and measurement of the system performance requirements allocated to software functions.
The proposed framework was developed in four main steps: 1. Identifying, analyzing, and categorizing into an integrated view the system performance requirement functions and services described from different perspectives into ECSS and IEEE standards. Then, modeling the identified system/software performance requirements and clarifying the relations between these requirements using SIGs. 2. Applying the COSMIC-FSM method to identify and measure the data movements derived from the allocated software performance requirements. This leads to handling the system performance requirements allocated to the measured software performance requirements as quantitative requirements. 3. Developing the proposed framework in the context of service-oriented architecture using COSMIC-SOA guidelines to support a distinct business domain. 4. Illustrating the applicability of the proposed approach for the specification and measurement of system performance NFRs allocated to the software for an automated teller machine (ATM) within an SOA context. The rest of the paper is structured as follows. Section 2 presents related work. Section 3 discusses the system performance requirements identification and related software performance requirements in international standards. Section 4 details the proposed standards-based system performance reference framework at the function and service levels in the context of a service-oriented architecture (SOA). Section 5 presents an illustrative example using the proposed standards-based framework for identifying and specifying ATM banking system performance requirements, allocating them to the software performance functions in an SOA context and measuring them with COSMIC, an ISO-recognized measurement unit. Section 6 presents conclusions and further work.

Non-functional requirements in the literature and international standards
A number of proposals for identifying and specifying different types of NFR, including different methods, approaches, views and terminologies have been made [1][2][3][4][5][6][7].
To help software project teams make the best tradeoff decisions for conflicting NFRs, Zhang and Wang [8] proposed a tradeoff model for conflicting software non-functional requirements (CNFR) using a fuzzy ranking method to express stakeholder assessments of each NFR.
To help developers prioritize such kinds of requirements early in the project cycle, Shah et al. [9] proposed an approach for specifying the NFR conflicts from previous ontological representations of the NFRs.
Daclin et al. [10] analyzed interoperability as a single NFR as a part of a complex NFR domain, linking interoperability and its impacts on the system performance requirements into a collaborative system in a crisis management framework.
Cysneiros et al. [11] highlighted the challenges facing developers of capturing NFR simultaneously with FR at the early phases of software development. They suggested the integration of NFR with FR into conceptual models based on a goal oriented strategy aimed at reducing the cost of software development as well as increasing customer satisfaction.
Various studies have focused on NFR [12][13][14][15] within the software product line process. Tawhid and Petriu [16] for example, proposed a UML model transformation framework to determine and reuse the performance requirements for a specific product.
Siegmund et al. [17] proposed a holistic approach, named SPL Conqueror, for the optimization of the specification and measurement of NFR in the SPL domain. They also carried out an analysis of the quality attributes (i.e., NFR) in SPL as well as a verification of product satisfaction of the quality conditions and constraints. Danylenko and Lowe [18] studied a context-aware recommender system with the objective to defer architectural decisions, thus permitting concentration on the core system functionality design. In early development phases this recommender system helps to ease the difficulties of NFR efficiency.
Kyo and Gil-Haeng [19] proposed a systematic software development process to support successful management and modeling for NFR. This process allows NFR to be systematically managed and efficiently modeled.
Industry, through its participation in international standards organizations, has also contributed by describing and categorizing NFR. For example: performance requirements are one of sixteen NFR types in ECSS [20][21][22][23][24][25], which have been categorized by IEEE [26] as one of thirteen NFR types, using different terms and views. Although in academia and industry NFR performance requirements are frequently discussed, there is a lack of a performance model that can be used in the early development stages. In the research reported in this paper, we propose a standards-based framework for early identification and measurement of system performance requirements by analyzing all the performance NFR related terms and views dispersed throughout the international standards, such as ECSS and IEEE.

System performance requirements in the literature
To develop new insights into performance, in this research, we analyzed related works in standards on performance in hardware domains where there is considerable, accumulated expertise. We looked for performance-related concepts and sub-concepts that were also relevant to software. System performance requirements have been discussed from various viewpoints in the literature. Shang et al. analyzed [27] the VxWorks real time operating system used in the aerospace and medical fields including five significant performance indicators: task switching time, preemption time, interrupt latency time, message communication time and semaphore shuffling time.
Alwadi et al. [28] proposed a framework for the quality of service (QoS) attributes and included performance as one of the prime system NFR, allowing performance requirements to be decomposed and allocated to a set of the system's functionalities.
Zhiwei et al. [29] proposed an approach for improving the concurrent system performance on the dynamic weighted k-out-of-n system (DWKNS). Subsequent to the state possibility and request of system components over time, this approach was combined with the Markov process with the universal generating function (UGF) method and the state probability for the performance of the system components.
Al-Sarayreh [30] considered system performance requirements to be more comprehensive than typical hardware-centric systems and proposed that dynamic system performance requirements be included with maintainability, upgradability, interface interoperability, reliability, safety and security (MUIRSS). MUIRSS should be analyzed and visibly connected to ensure that they are included with development.
The system dynamic performance of Kai and Huamin [31] for control systems indicates a relational variance of the controller design method. Their proposed method is used for the first order plus delay time system.
Krishna and Abraham [32] discussed the importance of the analysis of performance and memory NFR in real time embedded systems. Based on agile using incremental development, their development approach helps system engineers track the system performance requirements and related parameters throughout the development cycle. Their results are taken as a reference for a systematic analysis approach for memory and system performance NFR parameters using the most suitable mathematical methods.
Vila et al. [33] presented an approach for estimating the radio resource requirements for RAN slice admission control in order to describe the interference conditions of resource estimation method influences on system performance requirements extracted from data analytics collected from management plans.
Ruberg et al. [34] proposed a data processing and cleaning method for a performance and energy consumption estimation approach to manage system performance requirements. Their approach links software component feature measurements (SCFMs) and software performance quality indicators (SPQIs) to diagnose the software and functional requirements.

COSMIC functional size measurement method
There are currently five functional size measurement (FSM) methods adopted by ISO: the COS-MIC Function Points method is the only second generation of such FSM methods, and its design has corrected a number of the defects identified in the other four FSM methods of the first generation. Measuring software functional size is an important factor for managing and estimating the project budget early in the software development lifecycle. The COSMIC functional size measurement (FSM) method conforms to the measurement requirements proposed in ISO 14143-1 [35] and has been adopted as ISO 19761 [36]. This subsection presents the COSMIC generic model for software requirements and how such a model can be used to measure software functionalities with an ISO-recognized measurement unit.
In the COSMIC-FSM method, the functional user requirements are decomposed into one or more functional processes, each of which may be comprised of sub-processes and include a number of data movements. Figure 2 illustrates the COSMIC generic model of software FR. The front-end direction of the model shows that users access the software through input/output devices (such as mouse and microphone) or engineered devices (such as sensors). The back-end direction shows that the software is accessed by storage hardware (such as RAM memory). Figure 2 also illustrates the following four types of data movement: EN-TRIES (E): exchanges data groups from users or engineered devices to software (left-hand side in Figure 2). EXITS (X): exchanges data groups from software to users or engineered devices (left-hand side in Figure 2). READS (R): exchanges data groups from hardware storage to software (right-hand side in Figure 2). WRITES (W): exchanges data groups from software to hardware storage (right-hand side in Figure 2).
The core principle of the COSMIC-FSM method is to measure the size of software FR by identifying the recognized data movements (E, X, R and W). Once the data movements are identified, each type of data movement is assigned the value of one COSMIC Function Point (e.g., 1 CFP). The functional size of the software to be measured is obtained by summing the sizes of all the corresponding data movements. Since the Moreover, the COSMIC-FSM method is applicable to all the software development phases, from the analysis to implementation phases. Note that the COSMIC generic model in Figure 4 is not specific to any type of software nor to any particular method for describing functional user requirements. In the framework proposed in this paper, the COSMIC-FSM method is applied to measure the size of the software performance functional requirements with an ISO-recognized size unit.

Service-oriented architecture (SOA) and its COSMIC view
The service-oriented architecture (SOA) approach provides significant benefits to organizations, such as reducing software development and maintenance costs and increasing software quality by reusing services [37]. Various definitions have been introduced to define SOA, but none have been universally adopted. For instance, SOA has been defined as: 1) A process that involves the definition of the architecture, components, modules, interfaces, and data for a system to satisfy specified requirements [36,37]; 2) A paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains. It provides means to offer, discover, interact with, and use capabilities to produce desired effects consistent with measurable preconditions and expectations [36]; 3) Utilization of loosely coupled software services to support business processes requirements and user requirements [37]. The COSMIC-SOA guidelines document illustrates how to measure the size of software services in an SOA context [37]. The term services in the COSMIC guideline refers to a suite of related functions of software FR and also to the separation of functions into distinct units, where these services are connected with each other by exchanging data, shared format or by coordinating activities between two or more services [37].
COSMIC-SOA guidelines offer three types of data movements -exchange services, intermediary services and data exchanges, which are described in more detail in the following sections.

COSMIC-SOA exchange messages
COSMIC-SOA exchanges messages ( Figure 3) when an application needs information from a dif- Figure 3. COSMIC-SOA guidelines for modeling data movements ferent application. For instance, if application A needs to exchange data with application B, the services of application A will be invoked by the functional process of A to communicate with the services of application B to obtain the needed information. These calls between the functional processes of A and its services or between A services and B services are known as messages, where each message may involve one or more data movements [37].

COSMIC-SOA intermediary services
When services (Figure 3) of any application require data from another application in the overall SOA framework, the intermediary service will be used. For instance, if services of application A need data from services of application B, the services of application A will invoke the intermediary services to obtain the required data from the services of application B.

COSMIC-SOA data exchanges
For components in the same layer (e.g., in the application service layer) (Figure 3), two types of data movements can be used: direct and indirect message exchanges. For instance, in direct exchange, if the service of application A requires to exchange a message with a service of application B, it will use an Exit and/or an Entry for exchanging messages with the service of application B. While the indirect exchange occurs using storage, for instance, the service of application A writes the data in storage which is read later by service B [37].

Softgoal Interdependency Graphs
Softgoal Interdependency Graphs (SIGs) [38] have been proposed for analyzing and demonstrating NFR as softgoals. Each softgoal can be represented as decomposed into one or more specific goals using interdependency relations between the analyzed goals until arriving at solutions that satisfy the assigned NFR.
SIGs [39] illustrate three different types of goals at the high level: 1) Softgoals that satisfy the NFR with the software, 2) Claim softgoals which enhance the rationale between related softgoals, and 3) Operationalization of system softgoals (including a set of processes, data representations and system behavior).
These SIGs [39] at the low level (i.e., subgoals) provide both positive and negative contributions to the assigned softgoals at the high level.
Softgoals and subgoals can interact with each other using the following relations [39]: 1) AND means that each softgoal is decomposed into more than one related goal and is satisfied if all the related goals are satisfied. 2) OR means that each softgoal is decomposed into one or more related goals and is satisfied if at least one related goal is satisfied. 3) EQUAL means that each softgoal is decomposed into one related goal and is satisfied if the linked goal is satisfied.
The SIGs [38,39] approach uses the terms goals and subgoals to represent the conditions or criteria that the system should meet (e.g., non-functional requirements or quality attributes) instead of more commonly used terms in software engineering, such as functions and software specifications. In addition, the SIGs approach does not distinguish between the system view and the software view.
This research provides a mapping between some of the SIGs terms to the standards-based terms used in this paper, as presented in Figure 1. Therefore, the expression function to be specified is used instead of a functional goal while both are encoded as is in the SIG approach.

Performance requirements identification
This section introduces and discusses performance terms and views for identifying system performance NFR and related software performance FR, which may then be used for specifying and measuring the system performance requirements. Numerous terms and views are found throughout the ECSS and IEEE international standards as well as previous works in academia. These have addressed software performance FR derived from system performance FR and NFR (see Figure 1). Figure 1 also illustrates system performance requirements expressed as either system performance NFR or system performance FR.

ECSS concepts for performance requirements
ECSS standards [20][21][22][23][24][25] mention the importance of establishing performance requirements in detail at both system and software levels during the development phase so as to evaluate the consistency and cohesion of the control system within the required standards. This includes: 1) The objective(s) for each designed control system, which are normally created by the requirements engineering process; 2) The formal mathematical requirements, which are created by the requirements analysis.
Enhancing and regularly improving software applications requires system monitoring and evaluation of system performance. The performance monitor [23] provides information related to the use of processor instruction execution and storage control. For example, to provide information related to the period of time passed between events in a processing system.
The performance monitor can be used to debug the software application and analyze system faults and errors by defining a machine's state at a specific point in time. The information from the performance monitor helps system engineers to evaluate and improve the performance of a given system, or by developing enhancements of performance requirements in new system design.
ECSS standards [20][21][22][23][24][25] define the following concepts and views for system performance requirements allocated to software: 1) Frequency domain requirements such as throughput time, which includes: Workload and Bandwidth; 2) Response to reference signals for command profiles, which includes: Response time, Settling time, and Tracking errors; 3) Accuracy and stability errors in the presence of disturbances: Performance errors (absolute and stability errors) for evaluating the accuracy and Knowledge errors (absolute and relative errors) for evaluating accuracy; 4) Processing speed includes: System scalability, and System concurrency; 5) Resource consumptions include: Processor instruction execution, Main memory time, and Storage device time.

IEEE concepts for performance requirements
IEEE standard 830-1998 [26] describes the following terms and concepts for system performance requirements allocated to software as dynamic and static numerical requirements: 1) Dynamic numerical requirements, such as workload; 2) Static numerical requirements, such as capacity and concurrency. These two types of system performance requirements should be quantified with a measurable procedural method.

Describing system performance and related software functions
This section presents a brief description of system performance requirements and their allocated software performance requirements.

Performance dynamic requirements
Performance dynamic numerical requirements may involve the data amount, transaction number and tasks to be processed within a specific period of time for both normal and peak workload conditions [26]. The unified standards-based view includes two types of system requirements for dynamic requirements: the response to reference signals and throughput time.

Response to reference signals (RRS):
Response to reference signals refers to the specific values that change to a new value in a relatively short period of time, including response time or settling time values. Enhancing the response to reference signals is reflected positively on the system performance level. The unified standards-based view includes three types of functions for response to reference signals [25]: response time function, settling time function and tracking error function.
-Response time function (RTF) The response time is widely defined as the period of time that the system takes to respond to the user after receiving the user task. This relation between response time and performance is an inverse relation, since a decrease in response time leads to an increase in performance level. -Settling time function (STF) The settling time refers to the time required for the system to recover from an overload and to reach steady state. STF has also been called recovery time or reaction time [40]. It is important to reach steady state in as little time as possible. -Tracking error function (TEF) Tracking errors includes tracking performance error. The knowledge error (KE) or errors resulting from the central processing unit (CPU) and the main memory are also important to minimize system errors. System performance, therefore, can be enhanced by increasing system accuracy and speed. Throughput time (TT): Throughput time is the number of event responses carried out by the system in a specific period of time [41]. Thus, maximizing the throughput time leads to increasing performance of the system. The unified standards-based view includes two types of functions for throughput time: -Bandwidth function (BF) The bandwidth function refers to the maximum amount of data that can be carried over a network or data-transmission medium in a unit of time.
The throughput time is limited by the bandwidth function. Large bandwidth leads to more event responses over time [42]. -Workload function (WF) The workload function measures the number of transactions performed by the system within certain periods of time. The performance level is good when the system workload is significantly lower than its capacity [43]. Otherwise system performance will be slow.

Performance static requirements
Performance static numerical requirements are sometimes specified under a separate section.as capacity. They may also involve information types, the amount of time handled, the number of simultaneous users and terminals supported [26]. Based on IEEE standards, the unified standards-based view includes three types of system requirements for static numerical requirements: resource consumption, evaluation of processing speed and evaluation of accuracy.

Resource consumption (RC):
The efficiency of system resources (such as CPU, main memory and system storage) significantly affects the system performance. Heavy resource consumption can lead to the system's inability to effectively deal with its processes [44,45], therefore, slowing down or crashing causing poor system performance. Proper utilization of resources leads to high system performance.
The unified standards-based view includes three types of functions for resource consumption: main memory time function, storage device time function and processor instruction execution function.
-Main memory time function (MMTF) The main memory is also known as the system internal memory or primary memory; it is used to store the data that is in use. When the CPU requires access to specific data from the storage device, the main memory will access the storage device and retrieve the required data to be processed by the CPU [46]. The time spent to access data in the main memory needs to be as small as possible in order to optimize system performance. -Storage device time function (SDTF) Storage devices have a huge capacity to hold data in a permanent way. Fast storage devices are preferred to slower devices. Storage speed is impacted by two factors: Access time: the average time to locate data on the storage medium, and Data transfer rate: the amount of data transferred to or from the device per second [47]. -Processor instruction execution function (PIEF) Computer instructions are a set of commands executed by the processor to perform specific functions. Increasing the speed of executing such instructions can significantly contribute to improving the system performance level.

Evaluation of accuracy (EA):
The developed system should achieve a high level of accuracy (i.e., precision). The definition of accuracy varies from one system to another. For instance: In satel-lite systems, accuracy refers to the positioning accuracy provided by the system [48]. In radio systems, accuracy refers to how closely the actual output frequency matches the set frequency [48].
System accuracy may be determined through measuring system error. Based on ECSS standards, the unified standards-based view includes two error types: performance error and knowledge error.
-Performance error (PE) is defined as the functions that quantify the difference between the system's desired state and the system's actual state [27,28]. In the unified standards-based view, two common PE indices are used for measuring performance error: 1. Absolute performance error function (APEF) The absolute performance error is defined as the instantaneous value of the performance error at any given time [25]. Applying a specific mathematical operator on the performance error function determines the APE. In addition, each system has a maximum APE value, which the calculated APE should not exceed. 2. Performance stability error function (PSEF) The system stability is defined as the ability of the system to maintain a particular situation for a given time. The stability error is the peak-to-peak variation of the system attitude during the time period [25]. The PSEF is known as the change of error over a given time [25].
In addition, applying a specific mathematical operator to the performance error function and to the APE determines such a function. Just like the APE, each system has a maximum PSE value and the calculated PSE should not exceed the maximum APE. It is possible to use other performance error indices if the system so requires.
-Knowledge error (KE) is defined as the functions that quantify the difference between the system's estimated (or known) state and its actual state [27,28]. In the unified standards-based view, two common knowledge error indices are used for measuring knowledge error: 1. Absolute knowledge error function (AKEF) The absolute knowledge error (AKEF) is defined as the instantaneous value of the knowledge error at any given time [29]. Applying a specific operator to the KE function determines the AKE. Just like the performance error indices, each system has a maximum AKE value and the calculated AKE should not exceed the maximum APE.

Relative knowledge error function (RKEF)
The relative knowledge error (RKEF) refers to the difference between the instantaneous knowledge error at a specific time and its mean value over a time interval containing that time [29]. It is possible to use two other KE indices types should the system require it.

Evaluation of processing speed (EPS):
The processing speed refers to how quickly the processor handles instructions. The processing speed for a CPU is measured by the CPU clock rate. A CPU with a high clock rate leads to high speed instruction processing. The unified standards-based view includes two types of functions for EPS: system scalability function and concurrency function. 1. System scalability function (SSF) System scalability function (SSF) is the system's ability to process increased workload while maintaining the required system performance level [48]. To make the system scalable, additional hardware is added, such as CPU or memory, without making any changes to the system architecture. 2. Concurrency function (CF) The concurrency function refers to executing several instructions simultaneously, which improves the use of system resources while also reducing the system response time [49,50]. Figure 4 shows the four main phases used to determine the proposed measurement framework for system performance requirements:

Measurement framework for performance requirements
Phase 1 (Logical view): Identify and analyze the functions to be specified for system performance requirements. In this phase, the logical views are defined based on the functional user requirements view.
Phase 2 (Process view): Design and integration of the identified system performance requirements. In this phase, the process view is developed which includes default, rationale and component approaches. Phase 3 (Development view): Design a system performance requirements model using SIGs at the functional level. In this phase, the system performance model is designed and built by integrating the logical and process views [51].
Phase 4 (Deployment view): Design a measurement context at functional and service levels with COSMIC-SOA to measure the functional size of the software performance requirements. In this phase, the design measurement strategy is used to develop the proposed generic model of the system performance requirements allocated to software based on functional user requirements (FUR) views. Next, an architectural measurement context for the service levels is designed by applying the COSMIC-SOA guideline to develop a framework in an SOA context.
"For preliminary design of the performance requirements model, the SIGs tool is used. For the performance measurement model, we used another visualization tool called LibreOffice Draw Tool. This tool extends the preliminary performance model in SIGs by adding the generic COSMIC measurement procedure and adopting the detailed COSMIC-SOA to the proposed performance model."

Integration of system performance functions to be allocated to software (Phases 1 and 2)
The terms and views found in ECSS and IEEE to describe the performance NFR in Section 3.3 are combined and integrated using both their logical and process views. This leads to a dynamic view ( Figure 5) and a static view (Figure 6) of the system performance functions and related software functions, which can then be used to specify and measure them.

Design of system performance requirements at the functional level (Phase 3)
The proposed framework for system performance requirements is established at two main levels: the functions level and the services level. This section illustrates and describes in detail the framework at the functional level. In this section, the software interdependency goals (SIGs) and ISO 19761 are used to design the framework of the system performance NFR allocated to software at the functional level, divided into four sub-models, see Figure 7.

System performance dynamic requirements (SPDR)
The functions for the system performance dynamic requirements must address: The throughput time (TT): The throughput time (TT), which involves two functions: the bandwidth function and the workload function. Figure 7 illustrates the interdependency relationships between these functions: 1. The bandwidth (BF) and the workload (WF) functions may exchange data in a direct way with each other, and/or 2. may exchange data in an indirect way through the persistent storage, and as well 3. may require data from any function in the overall performance framework through intermediary services. The response to reference signals (RRS): The response to reference signals (RRS) involves three functions: the response time function (RTF), the settling time function (STF) and the tracking error function (TEF). Figure 7 shows the interdependency relations between these functions: 1. The response time, settling time and tracking error functions may exchange data in a direct way with each other; 2. may exchange data in an indirect way through the persistent storage; 3. may require data from any function in the overall performance framework through intermediary services.

System performance static requirements (SPDR)
The system performance static requirements include four function types -see Figure 7.
Resource consumption (RC): Resource consumption (RC) which involves three functions: the main memory time function (MMTF), the storage device time function (SDTF) and the processor instruction execution function (PIEF). Figure 7 illustrates the interdependency relationships between these functions: 1. They may exchange data in a direct way with each other, and/or 2. may exchange data in an indirect way through the persistent storage, and 3. may require data from any function in the overall performance framework through intermediary services.

The evaluation of accuracy (EA):
The evaluation of accuracy (EA) which is composed of performance error (PE) and knowledge error (KE). Performance error involves two specified functions: the absolute performance error function (APEF) and the performance stability error function (PSEF). The knowledge error also involves two specified functions: the absolute knowledge error function (AKEF) and the relative knowledge error function (RKEF). Figure 7 illustrates the interdependency relationships between these functions: 1. They may exchange data in a direct way with each other; 2. may exchange data in an indirect way through the persistent storage; 3. may require data from any function in the overall performance framework through intermediary services.

The evaluation of accuracy (EA):
The EPS which is composed of two functions: the system scalability function (SSF) and the concurrency function (CF). Figure 7 illustrates the interdependency relationships between these functions: Figure 7. Full view of the system performance NFR at the functional level 1. They may exchange data in a direct way with each other; 2. may exchange data in an indirect way through the persistent storage; 3. may require data from any function in the overall performance framework through intermediary services.

Design of the measurement framework for system performance NFR (Phase 4)
This section introduces the framework in an SOA context using the COSMIC-SOA guidelines. For clarity, the proposed framework is divided into the seven sub-models illustrated in Figures 8 to 13. Figure 14 shows the full view in the SOA. Figure 8 shows that all the derived functions from resource consumption have their own services. The interdependency relations between these functions and their services are: -The main memory time function, the storage device time function and the processor instruction execution function may require data from their services using EXIT and ENTRY data movements. -The main memory time service may exchange data at a service layer either in a direct way with the storage device time service using COSMIC EXIT and ENTRY data movements. Or it may exchange data in an indirect way through the persistent storage using COSMIC READ and WRITE data movements. -The storage device time service may exchange data at a service layer either in a direct way with the main memory time service and the processor instruction execution service using COSMIC EXIT and ENTRY data movements. Or it may exchange data in an indirect way through the persistent storage using COSMIC READ and WRITE data movements. -The processor instruction execution service may exchange data at a service layer either in a direct way with the storage device time service using COSMIC EXIT and ENTRY data movements. Or it may exchange data in an indirect way through the persistent storage using COSMIC READ and WRITE data movements. -The main memory time service, the storage device time service and the processor instruction execution service may require data from any service in the overall performance framework through the intermediary service using COSMIC EXIT and ENTRY data movements. Figure 9 shows that all the derived functions from the evaluation of processing speed have their own services. The interdependency relations between these functions and their services are: -The system scalability function and the concurrency function may require data from their services using EXIT and ENTRY data movements.
-The system scalability service may exchange data at a service layer either in a direct way with the concurrency service using COSMIC EXIT and ENTRY data movements. Or it may exchange data in an indirect way through the persistent storage using COSMIC READ and WRITE data movements. -The concurrency service may exchange data at a service layer either in a direct way with the system scalability service using COSMIC EXIT and ENTRY data movements. Or it may exchange data in an indirect way through the persistent storage using COSMIC READ and WRITE data movements. -The system scalability service and the concurrency service may require data from any service in the overall performance framework through the intermediary service using COS-MIC EXIT and ENTRY data movements. As mentioned in Section 4.1, the evaluation of accuracy (EA) is comprised of two error types: performance error (PE) and knowledge error (KE). Figure 10 shows that all the derived functions from performance error have their own services. The interdependency relations between these functions and their services are: -The absolute performance error function and the performance stability error function may require data from their services using EXIT and ENTRY data movements. -The absolute performance error service may exchange data at a service layer either in a direct way with the performance stability error service using COSMIC EXIT and EN-TRY data movements. Or it may exchange data in an indirect way through the persistent storage using COSMIC READ and WRITE data movements.
-The performance stability error service may exchange data at a service layer either in a direct way with the absolute performance error service using COSMIC EXIT and EN-TRY data movements. Or it may exchange data in an indirect way through the persistent storage using COSMIC READ and WRITE data movements. -The absolute performance error service and the performance stability error service may require data from any service in the overall performance framework through the inter-mediary service using COSMIC EXIT and ENTRY data movements. Figure 11 shows that all the derived functions from the knowledge error (KE)have their own services. The interdependency relations between these functions and their services are: -The absolute knowledge error function and the relative knowledge error function may require data from their services using EXIT and ENTRY data movements. Figure 11. Knowledge error sub-model in an SOA context -The absolute knowledge error service may exchange data at a service layer either in a direct way with the relative knowledge error service using COSMIC EXIT and ENTRY data movements or it may exchange data in an indirect way through the persistent storage using COSMIC READ and WRITE data movements. -The relative knowledge error service may exchange data at a service layer either in a direct way with the absolute knowledge error service using COSMIC EXIT and ENTRY data movements. Or it may exchange data in an indirect way through the persistent storage using COSMIC READ and WRITE data movements. -The absolute knowledge error service and the relative knowledge error service may require data from any service in the overall performance framework through the intermediary service using COSMIC EXIT and ENTRY data movements. Figure 12 shows that all the derived functions from the response to reference signals have their own services. The interdependency relations between these functions and their services are: -The response time function, the settling time function and the tracking error function may require data from their services using EXIT and ENTRY data movements. -The response time service may exchange data at a service layer either in a direct way with the settling time service using COSMIC EXIT and ENTRY data movements or it may exchange data in an indirect way through the persistent storage using COSMIC READ and WRITE data movements. -The settling time service may exchange data at a service layer either in a direct way with the response time service and the tracking error service using COSMIC EXIT and ENTRY data movements or it may exchange data in an indirect way through the persistent storage using COSMIC READ and WRITE data movements. -The tracking error service may exchange data at a service layer either in a direct way with the settling time service using COSMIC EXIT and ENTRY data movements. Or it may ex-change data in an indirect way through the persistent storage using COSMIC READ and WRITE data movements. -The response time service, the settling time service and the tracking error service may require data from any service in the overall performance framework through the intermediary service using COSMIC EXIT and ENTRY data movements.  Figure 13 shows that all the derived functions from the throughput time (TT) have their own services. The interdependency relations between these functions and their services are: -The bandwidth function and the workload function may require data from their services using EXIT and ENTRY data movements. -The bandwidth service may exchange data at a service layer either in a direct way with the workload service using COSMIC EXIT and ENTRY data movements. Or it may exchange data in an indirect way through the persistent storage using COSMIC READ and WRITE data movements. Figure 13. Throughput time sub-model in an SOA context -The workload service may exchange data at a service layer either in a direct way with the bandwidth service using COSMIC EXIT and ENTRY data movements. Or it may exchange data in an indirect way through the persistent storage using COSMIC READ and WRITE data movements. -The bandwidth service and the workload service may require data from any service in the overall performance framework through the intermediary service using COSMIC EXIT and ENTRY data movements.  Figure 14 illustrates the full view for the measurement framework of the system performance requirements on the basis of the previous sub-models in Figures 8-13 at functional level and in an SOA context. From Figure 14, the following points can be observed for measurement purposes: -In the direct data exchange situation, each EXIT and ENTRY data movement will be assigned a size of 1 CFP. -In the indirect data exchange situation, each READ and WRITE data movement will be assigned 1 CFP.
-Data required through intermediary services that requires using 4 EXITS and 4 ENTRIES will be assigned 8 CFP.

Overview
Banking systems provide a variety of financial services to individuals, businesses and governments. An automated teller machine (ATM) is a computerized system found in a public location. Customers are identified by inserting a smartcard that contains a unique number and some information about the customer and their account status. The services typically provided by ATM banking systems include: accepting deposits, cash withdrawal, issuing balance statements, pre-paid mobile charges and money transfers. To achieve high customer satisfaction, such systems must demonstrate high quality levels including: excellent performance, security and reliability.

Purpose and process
The purpose of this example is to present a "proof--of-concept" on a small-scale of the concepts proposed in this paper by illustrating the use of the proposed measurement framework for system performance requirements allocated to software (as in Figure 14). More specifically, to derive the system performance requirements allocated to an ATM system, and to measure the functional size of these allocated requirements using the COSMIC method. This illustrative example was realized by applying the following steps: -Analyze and specify the main components of the ATM internal structure in a physical view. -Design the workflow scenario-based application for the customer view. -Identify the ATM functional user requirements for the customer and system views. -Specify the ATM system requirements allocated to software. -Specify the ATM system performance requirements allocated to software as an extended view to step 4. -Map the allocated system performance requirements with the proposed framework. -Measure the functional size of the allocated system performance requirements to the specified banking system with the COSMIC ISO-recognized measurement unit.

ATM internal structure system
Here we detail the internal structure of the bank ATM and the relationships among its various parts. An ATM typically consists of several devices such as: central processor unit (CPU), crypto processor, memory, customer display, function key buttons (typically situated near the display), smart chip card reader, encrypting PIN pad, customer receipt printer, vault, and modem. The vault stores all devices and parts that require limited access, such as: -Cash dispensing mechanism (CDM), -Deposit mechanism (DM), -Security sensors (SS) (e.g., magnetic, thermal, seismic, gas), -Electronic journal system (EJS) to keep system log, -Cash dispenser (CD) which includes several removable cash cartridges, deposit mechanism and removable deposit cartridges. The software specifications for the ATM system are: read the ATM card, count currency notes, connect to bank network, take input from user, validate user, dispense cash to user and receive deposit envelopes from the user through deposit slot.

ATM block diagram
The set of ATM system scenarios includes authentication of the PIN entered with the one encrypted on the card. Once the PIN is confirmed, the customer can access their bank account to make the chosen transaction. Or else the system shows a suitable message to clarify rejection of access. Figure 15 illustrates the first instantiation scenario for customer authentication for the ATM as follows: -The client inserts his/her smartcard; the card reader processes the smartcard's data using the card transaction handler, and informs the system that the smartcard is valid. -The card transaction handler displays a message on the ATM screen asking for the customer PIN number. -The ATM screen asks the customer to enter the PIN and the customer enters PIN code which is passed on to the card transaction handler. -The card transaction handler verifies and gives authorization if the PIN is correct; if not, Figure 15. ATM functional requirements at the system level a message appears on the screen to inform the customer that PIN number is invalid. -The customer enters PIN again when the message appears on the screen to enter the PIN code number. -If the customer has not provided the correct PIN in three iterations, the card reader will capture the customer smartcard and the session is terminated. The second instantiation scenario after customer authentication for the ATM is as follows: -The main menu that appears on the ATM system screen contains three types of transactions: "get account balance inquiry" (choice 1), "cash withdrawal" (choice 2) and "money deposit" (choice 3). A choice to allow the user to exit the system (choice 4) appears as well. -The user at that point chooses either to make a transaction by entering one of the three choices or exits the system. -If the client enters "get account balance inquiry", the ATM retrieves the balance from the bank's database and the screen displays the client's account balance. -If the client enters "cash withdrawal", the ATM screen displays a menu holding typical withdrawal amounts such as 50, 100, 200. -The withdrawal menu also displays a choice to permit the customer to cancel the transaction.
-If the withdrawal amount selected is larger than the client's account balance, the screen displays a message telling the client to select a smaller amount. The ATM then returns to the beginning of this scenario. -If the withdrawal amount selected is less than or equal to the client's account balance, the ATM proceeds and issues the client's requested amount. -Then the ATM subtracts the withdrawal amount from the client's account in the bank's database. -The screen displays a message informing the user to take money.

ATM functional requirements (FR)
The functional requirements (FR) represent the system tasks from the stakeholder perspective and are typically derived from the context of use. The FR perspective can describe customer scenarios, system goals and objectives within the system environment and can connect these perspectives with assigned hardware resources. Customer requirements are a subset of stakeholder requirements and can be collected in the stakeholder requirements specification document together with other perspective scenarios which have been derived from the system block diagram  Figures 15 and 16, including the identification of related hardware performance.

ATM system requirements allocated to software
Each system FR should be allocated to some specific software and/or physical component. Allocation should be defined in the early phases of the system development life cycle. At a high level in the system NFR this allocation impacts the design of the system architecture. Figure 17 illustrates the customer FR perspectives connected with software goals and sub software goals to derive the system requirements allocated to software.

System performance NFR allocated to software functions
This section presents an instantiation of some system performance requirements, and an example of their allocation to software. For this example, the following system performance NFR have been selected for the ATM system: -Requirement 1: The maximum data transmission over the system network shall be 100 megabits per second. -Requirement 2: The maximum time to respond to a customer transaction shall be one second.

Mapping system performance NFR to an SOA context
This section maps the specified system performance requirements for the ATM system within an SOA context -see Figure 19. It can then be used to measure the instantiation case of the specified system performance NFR allocated to software for the banking ATM.

Measuring the specified system banking performance NFR (instantiation case)
This step identifies the detailed data movements for the allocated software performance functions in an SOA context. It is important to note that the performance NFR may require additional resources be added (i.e., hardware) to the system. In this example, for illustrative purposes, a single data group was selected for each specified performance function, while in an industrial context these performance functions may require more than one data group. Table 1 shows the COSMIC measurement of the system performance NFR allocated to the software requirements at functional and service levels.
This example shows the corresponding COS-MIC size for the selected specified requirements at the functional and services level (Figures 18  and 19). For measurement purposes they correspond to COSMIC data movements as follows: Requirement 1 (R1): is allocated to one identified function (bandwidth function and its services) for the following customer-FR (Insert E-card, check if E-card is valid and Write E-card PIN). It extracts the identification data from the smartcard and invokes its own three services between three customer-FR, and then uses the persistence storage twice (once to check if the card is valid and second to check if the PIN number is correct. Here, we consider that the PIN is entered correctly). The measurement of the functional size using ISO 19761 (COSMIC) for R1 is calculated based on data movements between groups of processes as follows: -Three identification functions (3 ENTRY and 3 EXIT). -Two identification services (4 ENTRY and 4 EXIT). -Functional and services processes use persistence storage twice to obtain information about the card (2 READ and 2 WRITE). The total functional size for R1 is 18 CFP. Requirement 2 (R2): is allocated to one identified function (response time function and its services) for the following customer-FR (Present transaction options, get account balance and Write amount of money). A. The main menu on the ATM system screen displays three types of transactions options. The measurement of the COSMIC functional size for R2-A is as follows: -One identification function (1 ENTRY and 1 EXIT). -One identification service (2 ENTRY and 2 EXIT).
-Functional and services processes use persistence storage once, which appears on the main menu of the ATM application (1 READ and 1 WRITE). The total functional size for R2-A is 8 CFP. B. In this instantiation the customer enters "get account balance inquiry". The ATM retrieves the client's account balance from the bank's database which is then displayed on the screen. The measurement of the COSMIC functional size for R2-B is as follows: -One identification function (1 ENTRY and 1 EXIT). -One identification service (2 ENTRY and 2 EXIT). -Functional and service processes use persistence storage once which appears on the main menu for the ATM application (1 READ and 1 WRITE). -One intermediary service is needed to retrieve the customer account balance from the external database (4 Entry and 4 Exit). The total functional size for R2-B is 16 CFP. C. When the customer enters "cash withdrawal", the ATM screen displays a menu showing typical Figure 19. ATM system performance requirements allocated to software functions within an SOA context withdrawal amounts, such as 50, 100, 200. In this instantiation, the customer can enter their own desired amount. The measurement of the COS-MIC functional size for R2-C calculated based on data movements between groups of processes is described as follows: -One identification function (1 ENTRY and 1 EXIT). -One identification service (2 ENTRY and 2 EXIT).
-One intermediary service is needed to subtract the amount of money from the customer account balance (i.e., the customer has 300, takes 50, the remainder is 250) (4 ENTRY and 4 EXIT). -Functional and service processes use persistence storage twice to obtain information about the card (1 READ and 1 WRITE). The total functional size for R2 (A, B, C) is (8 + 16 + 16) = 40 CFP.

Requirement 3 (R3)
: is allocated to one identified function (main memory time function and its services) for the customer-FUR (Expel money).
The measurement of the COSMIC functional size for R3 is calculated based on data movements between groups of processes and is described as follows: -One identification function (1 ENTRY and 1 EXIT). -One identification service (2 ENTRY and 2 EXIT). The total functional size for R3 is 6 CFP.
Requirement 4 (R4): is allocated to one identified function (system scalability function and its services), twice for the customer-FUR (Register transaction to local DB and Register capture to local DB).
The measurement of the COSMIC functional size for R4 is calculated based on data movements between groups of processes and is described as follows: -One identification function twice (2 ENTRY and 2 EXIT). -One identification service twice (4 ENTRY and 4 EXIT). -Twice the intermediary service while the system uses the database (8 ENTRY and 8 EXIT). The total functional size for R4 is 28 CFP.
Requirement 5 (R5): is allocated to one identified function (settling time function and its services) for the customer-FR (Process another operation).
The measurement of the COSMIC functional size for R5 is calculated based on data movements between groups of processes and is described as follows: -One identification function (1 ENTRY and 1 EXIT). -One identification service (2 ENTRY and 2 EXIT). The total functional size for R5 is 6 CFP. Requirement 6 (R6): is allocated to one identified function (concurrency function and its services) for the customer-FUR (Inform rejection, Eject E-card, Take E-card and Capture E-card).
The measurement of the COSMIC functional size for R6 is calculated based on data movements between groups of processes and is described as follows: -One identification function four times (4 ENTRY and 4 EXIT). -One identification service four times (8 ENTRY and 8 EXIT). The total functional size for R6 is 24 CFP. Requirement 7 (R7): is allocated to one identified function (storage device time function and its services) for the customer-FUR (Register capture to local DB).
The measurement of the COSMIC functional size for this process is calculated based on data movements between groups of processes and is described as follows: -One identification function (1 ENTRY and 1 EXIT). -One identification service (2 ENTRY and 2 EXIT). -Two intermediary services, one to use the local ATM database and the second to give order to storage to capture the customer smartcard (8 Entry and 8 Exit). The total functional size for R7 is 22 CFP. Requirement 8 (R8): Requirement 8 (R8): is allocated to one identified function (processor instruction execution function and its services) for the customer-FUR (Register transaction and Inform not enough money).
The measurement of the COSMIC functional size for R8 is calculated based on data movements between groups of processes and is described as follows: -One identification function twice (2 ENTRY and 2 EXIT). -One identification service twice (4 ENTRY and 4 EXIT). The total functional size for R8 is 12 CFP. Table 1 lists the 17 software subgoals to be measured from the customer-FUR perspective. These software subgoals call eight specified functions and eight specified services processes of system performance allocated to software in R1, R2 to R8. These are identified and presented in columns 1, 2 and 3. For each identified functional process, the description of the measured resource represents a data performance group moved by one data movement type, each one measured as 1 CFP (COSMIC function point).

Summary of findings
The total functional size for the performance requirements R1, R2 to R8 applied to this software in a functional level is 34 CFP and in SOA context is equal to 122 CFP, independent of the languages and technologies used to implement them. In further applications, this functional size number can be used for effort estimation models and for software benchmarking.

Limitations of the illustrative example
The ATM example illustrates how the proposed approach is applicable in a relatively simple context. Future research is needed to investigate its scalability to much larger contexts, such as complex systems that perform millions of operations per second. Naturally, for such large complex systems, organizations have more resources and can dedicate much more resources to make use of this approach. In such contexts, additional research on efficiency studies would be welcome.

Practical Implications
We think that this framework can be easily used by someone knowledgeable in performance standards, COSMIC measurement and SOA architecture. Someone familiar with other environments, such as cloud computing environments, could also use and adapt this framework to a context-specific environment. More specifically, requirement specialists, software architects, performance engineers and project managers can use the proposed standards-based performance framework. The generic approach proposed in our research program on standards-based identification, specification and measurement of system NFRs allocated to software had already influenced the COSMIC Group when it published its initial strategy on how to handle NFRs and Project Requirements from a project management perspective [52]. To facilitate industry adoption of this approach, including the performance framework presented here, we recently edited, for the COS-MIC Group, a practitioner's guide on Non-Functional Requirements and their Sizing with COS-MIC [53]: it includes a set of templates based on the performance framework documented in more detail in this study. Organizations with a very large NFR knowledge base as well as experts in performance issues can also compare their own practices with this framework and identify gaps within their practices; such gaps can be addressed, fully or partially, by using this framework. If their own practices are not structured and explicit, they can use the structure of this framework to structure and document their own related practices.

Threats to validity
The quality of an experimental case study is important as it can substantiate the limits of the study and identify threats to its validity, which could impact the results. The key types of validity threats as proposed by Wohlin [54] are: Internal validity is concerned with the reliability of the results and refers to the treatment that caused the outcome [54]. There can be other uncontrollable or not measured factors that influenced the results. The internal validity threat in this example includes the change of the design process for this example. Here, we believe that if different researchers use the same method on the same example, they will get the same results. In contrast, in the absence of a full description of some parts considered in this example, variability in the results would be expected.
External validity [54] here refers to the generalization of the results outside the scope of the example and whether or not the cause and effect established hold in other situations. An external validity threat is expressed at the outcomes level. The proposed measurements framework of system performance NFR was illustrated using only the performance requirements specifications of an ATM system. There is no claim in this study on generalizability and scalability to much larger contexts, such as complex systems that perform millions of operations per second. Scalability is outside the scope of the research presented here. To alleviate the risk of this validity threat Table 1. ATM measurement of the system performance NFR allocated to software functions additional examples and case studies could be conducted using requirement specifications of different types of software products (e.g., business application software, Telecom software, medical embedded software, etc.).
Construct validity [54] refers to the relation between the theory behind the experiment and the observation(s). The treatment and the results may not correspond to the cause and the effect controlled and measured. A limited number of standards and methods have been selected for this example. Nevertheless, there exist other standards and methods, and future research could investigate the use and relevance of such other standards and methods.

Conclusion and further work
This paper proposed a measurement framework of system performance NRF allocated to software functions. This work on system performance NFR extends our previous work on three types of NFR (security, portability and maintainability) to facilitate the early identification, specification and measurement of such kinds of NFR.
The suggested framework includes some of the consensual performance terms and concepts used by two sets of international standards (ECSS and IEEE) and some related works. They were analyzed and integrated using different design views beginning with the logical view, followed by process view, development view and ending with physical views. Next, the set of ISO 19761 (COSMIC) concepts and views were adopted for describing the framework functionality at a lower level to illustrate that the proposed framework is designed for measurement purposes as well as for capture of the performance concept. The proposed framework was designed using SIGs. This research considered system performance requirements as both static and dynamic performance types, each with its own set of candidate sub-concepts.
Additionally, for a more complete software view of a complex environment (i.e., functional services in a service-oriented architecture), COSMIC-SOA was applied to the suggested framework.
Finally, an ATM example was presented to guide developers and software engineers to use the measurement of system performance NFR allocated as performance FUR at the software level.
The main contribution of this work is its ability to assist developers, system and software engineers to specify system performance requirements early in the life cycle in order to address the specified performance functions to be allocated to software as functional requirements.
The proposed framework can also be used for identification, specification and measurement of system performance NFR using ISO 19761 independently of any programming languages, and in addition can address software performance FUR early in their implementation.
In this paper, the proposed measurement aspects addressed the system requirements allocated to software. It will be interesting in further work to extend this measurement aspect to consider other types of requirements at the system level containing hardware requirements.
Some related issues were not addressed and additional work is required such as using these functional size measurement results of system performance requirements allocated to software-FURs as new input for estimating models in software engineering projects. Additional empirical work is required to verify that such expanded size can improve the estimation models, including for testing and maintenance effort.
The ATM illustrative example showed that the proposed measurement framework can help to specify and measure the functional size of system performance-NFR allocated to software functions. Consequently, this may improve planning, managing and development of software at different phases of the software development life cycle. Furthermore, the measurement results of the proposed framework may be used in benchmarking studies.
This example was not built to learn but to demonstrate that the framework was usable, and the purpose in building this example was not to evaluate the framework. This again, would require much more empirical work with practitioners to evaluate it in a number of contexts, and with a set of criteria for evaluation.
This proposed measurement framework is limited to measuring the system performance requirements allocated to software at the functional and service levels. It will be interesting in further work to focus on its applicability to different types of software products in order to generalize the results reported in this illustrative example. In addition, further work could focus on automating the measurement of software performance requirements through building an automated measurement tool (or enhancing an existing one).
Future work on the scalability of the framework proposed in this study would be valuable for industrial research where researchers look at such practical scalability issues, with financial resources much larger than the resources available to university researchers.