Using Behavior-Driven Development (BDD) for Non-Functional Requirements

: In software engineering, there must be clarity in communication among interested parties to elicit the requirements aimed at software development through frameworks to achieve the behaviors expected by the software. Problem : A lack of clarity in the requirement-elicitation stage can impact subsequent stages of software development. Solution : We proposed a case study focusing on the performance efficiency characteristic expressed in the ISO/IEC/IEEE 25010 standard using Behavior-Driven Development (BDD). Method : The case study was performed with professionals who use BDD to elicit the non-functional requirements of a company that develops software. Summary of Results : The result obtained was the validation related to the elicitation of non-functional requirements aimed at the performance efficiency characteristic of the ISO/IEC/IEEE 25010 Standard using the BDD framework through a real case study in a software development company. Contributions and impact : The article’s main contribution is to demonstrate the effectiveness of using BDD to elicit non-functional requirements about the performance efficiency characteristic of the ISO/IEC/IEEE 25010 standard.


Introduction
The software architect is the professional responsible for building the software, considering internal and external factors such as the architectural standard and clients, respectively [1].For an architect to successfully develop software, they must know the aspects inherent to its creation, for example, the existing frameworks to achieve specific purposes to maintain quality software.
A framework is defined as a structure that serves as a basis for systematically building applications.There is also a need for frameworks used in software architecture to be able to meet such expectations external to agile methods, for example, the Behavior-Driven Development (BDD) framework, used in the life cycle of a system [2].
BDD introduces a flexible methodology that allows monitoring the characteristics to be developed by elaborating requirements through user stories.These requirements are validated through scenarios that outline the convenience criteria for the behavior of the functionalities in question.
The BDD framework defines the software system's behavior using the "given, when, then" pattern, expressed in high-level, domain-specific natural language and executed through automated tests.BDD can document non-functional requirements (quality require-ments) based on the performance efficiency characteristic presented in the ISO/IEC/IEEE 25010 [3] standard.
The ISO/IEC/IEEE 25010 standard describes the quality characteristics that can be employed to evaluate software quality, such as non-functional requirements.By aligning the characteristics of the ISO/IEC/IEEE 25010 standard with BDD examples, non-functional requirements can be documented and tested in a structured way.This technique ensures that software meets desired quality standards and provides a framework for evaluating the quality of software in use [4].
The protocol used to direct this case study was formalized using part of the Goal-Question-Metric [5] model defined below: "To analyze the application of BDD, with the purpose of automating the elicitation and testing, in relation to non-functional requirements based on the characteristics of the ISO/IEC/IEEE 25010 standard, from the point of view of those involved in the software product, in the context of the performance characteristic".Table 1 presents the research questions proposed to achieve this purpose.

Identification
Question Rationale

RQ1
How does BDD address difficulties in ensuring non-functional requirements?
This question aims to understand how BDD can mitigate situations arising from the elicitation of non-functional requirements.

RQ2
How can BDD be used to ensure quality related to the performance efficiency characteristic of the ISO/IEC/IEEE 25010 standard?
This question aims to explain how this framework could contribute to eliciting non-functional requirements.

RQ3
What are the benefits of using BDD in identifying and automating non-functional tests?
This question aims to bring positive points to adopting BDD in the testing cycle.
In order to maintain software quality, there is a need for the structures used in its development to be designed in such a way as to contribute to the delivery of the final product, aiming to achieve what the user expects from the product and reduce rework arising from flaws in these structures [6].As described in the case study below, we validate the proposed scenarios in a company that uses the BDD framework for test cycles.Thus, we ensure security in the quality of the product delivered according to testable scenarios for non-functional requirements in a real example.
This study has the following Sections: Section 2, where we address studies related to this research; Section 3, where we point out the main points related to the application of the ISO/IEC/IEEE 25010 standard and BDD; Section 4, where we explain how we carried out the methodological steps of this study; Section 5, where we discuss the user stories used in this research; Section 6, where we present the data obtained; Section 7, where we explain the contributions of this study concerning related works; Section 8, where we discuss the threats that have been identified and mitigated; and, finally, Section 9, where we present the contributions of this research to Software Architecture.

Haoues et al.
[7] conducted a study to show which type of architecture could follow the quality characteristics expressed in the ISO/IEC/IEEE 25010 standard.The authors identified that it is necessary to think about quality throughout the entire software development life cycle to deliver the final product and maintain the quality characteristics of the ISO/IEC/IEEE 25010 standard.Therefore, the architecture used must aim to meet the non-functional requirements demanded to maintain the quality of what was requested as a final product; thus, when carrying out a case study using BDD to elicit non-functional requirements, it will be possible to identify difficulties about guaranteeing such requirements.
Estdale and Georgiadou [8] explored the scope and interpretation of the quality models of the ISO/IEC/IEEE 25010 standard, presenting aspects such as non-functional requirements inherent to software development.Non-functional requirements directly improve when implemented in products through measurable characteristics that meet a fixed set of specifications, aiming to satisfy usage and consumption expectations.In this way, this case study will identify how BDD will seek to guarantee quality related to the performance efficiency characteristic of the ISO/IEC/IEEE 25010 standard.
According to Jarzkebowicz and Weichbroth [9], there is a greater need for attention when dealing with the elicitation of non-functional requirements.Therefore, the authors conducted a systematic review followed by interviews to identify how non-functional requirements were elicited and documented.The authors discovered through the results of the studies that non-functional requirements are an essential part of the software development process; however, they are carried out differently, with each company eliciting the requirements differently.Therefore, understanding the elicitation of non-functional requirements is essential, as it can impact the entire execution of software development.Thus, using BDD to elicit non-functional requirements, we can better understand how to make it using this framework with the example of a software development company.
Olsson, Sentilles, and Papatheocharous [10] also performed a systematic literature review that analyzed empirical research for non-functional requirements.They observed that the ISO/IEC/IEEE 25010 standard was mentioned in the articles analyzed by the authors, which had the function of directing the quality of the software during its development.However, researchers need to understand more when reporting possible solutions regarding non-functional requirements, which affects practitioners' acceptance.Therefore, validating research through methods such as case studies can effectively identify whether what they publish is in line with the daily challenges in the industry.Thus, this case study can contribute to the use of BDD in identifying and automating non-functional tests to maintain quality and achieve the behavior expected by the software.

Theoretical Concepts
In this section, we discuss concepts inherent to the ISO/IEC/IEEE 25010 standard and Behavior-Driven Development (BDD) in order to understand how the BDD framework can collaborate with the elicitation of non-functional requirements.

ISO/IEC/IEEE 25010 Standard
The ISO/IEC/IEEE 25010 standard addresses software quality.By relying on this standard, the software can meet the needs of stakeholders, such as performance efficiency [11].
The purpose of the ISO/IEC/IEEE 25010 standard is to guide the evaluation of the quality of software and systems.The standard is divided into eight quality attributes, or characteristics.The focus characteristic of this study was performance efficiency, which, in turn, is divided into three sub-characteristics: time behavior, resource utilization, and capacity.
This assessment uses specific techniques and tools that allow for measuring compliance with the requirements and criteria established by the standard, improving the product development process, increasing user satisfaction, and gaining more competitiveness in the software industry.
A process represents a set of activities to achieve a previously stipulated goal.The capacity and maturity of a process are verified by the level of quality of what a software achieves with respect to an expected result.The software development process focuses on delivering a software implementation [12].Thus, it is clear that the quality of a system and its maturity must meet the needs of stakeholders [8].

Behavior-Driven Development (BDD)
BDD emerged in 2003, conceived by Dan North [13] as a framework of notable flexibility.North outlined the need for a framework focused on solving communication and team integration issues.Test-Driven Development (TDD) does not address these gaps when maintaining a focus on test automation, a factor that contributes significantly to product quality.Thus, while TDD is primarily dedicated to technical aspects, BDD seeks to improve communication and collaboration between the technical team and the stakeholders.In this context, BDD is a framework mainly oriented toward observing system behavior.
In order to promote collaboration and communication, BDD introduced gherkin, a remarkable way of expressing specifications that culminate in user stories and acceptance scenarios in natural language.This approach enables integration with testing tools, allowing the adoption of BDD to monitor system requirements and evaluate its behavior, ensuring that it is validated as ready at the end of each iteration.BDD has demonstrated significant benefits, such as more effective communication, reduced release delivery time, and greater accuracy in requirements elicitation [14].
BDD uses the "3 amigos" technique, where a meeting is held between the product owner, the tester, and the developer in order to objectively and concisely specify the expected behavior by the system when eliciting the requirements, in this case, user stories and their acceptance scenarios, in order to outline the behavior expected by the software.As such stories need to be written directly and transparently to achieve the proposed objective, BDD needs to make its scenarios testable [15].In this way, it is clear how BDD can collaborate to elicit non-functional requirements.
According to Bruschi et al. [2] improved communication through BDD results in better quality collaboration between those involved in the process, as the requirements are standardized by the "3 amigos" in order to also help with the documentation and automation of tests, which are necessary factors for validity.Furthermore, with the time savings gained through reduced reworking, one can focus on other activities inherent to the software.Therefore, after outlining the requirement to be implemented, the next step is to write it through BDD following the gherkin language.The gherkin language used by BDD provides a simple everyday language to help everyone involved in the process understand what a given requirement refers to.
From this point on, the features begin to be written, implemented, and validated, seeking to maintain the behavior expected by the requirements elicited in each release in order to deliver what is expected by the system.In addition to helping with communication and collaboration between the parts involved [16], as it is written in English, the BDD also helps with the dynamic documentation of the system [14,15,17].Dynamic documentation, or living documentation, refers to the agility required for updates to process documentation.With these aspects, it is expected that the use of BDD will save time and money, as it reduces the need for reworking due to poor-quality releases, and it also has better quality validation [4,18], in addition to a better understanding of the code [3].
According to Heck and Zaidman [19], the quality of requirements is an essential aspect of the quality of the final product.There is a need for requirements to be elicited in a clear, objective, and unambiguous manner to deliver what is expected by the customer.With the aim of analyzing how a company's non-functional requirements were elicited, we conducted the following case study.

Methodology
Evidence-based software engineering is a methodology whose purpose is to deliver systems with greater reliability [20] through, for example, case studies.The methodology used in this research on the application of BDD for non-functional requirements and automation, based on the performance efficiency characteristic of the ISO/IEC/IEEE 25010 standard, was conducted through a case study in collaboration with a company that develops software.
Case studies have been increasingly used in software engineering [21] because they use empirical data collection that seeks to understand a specific population.Thus, software engineering is based on observing actions aimed at software and their respective responses [22], as, in this way, it is possible to use the observed aspects to validate how the system should behave.One of the research purposes in software engineering focuses on how software engineers and other professionals in the field carry out development, operation, and maintenance in different situations [23].Therefore, according to Perry, Sim, and Easterbrook [24], a case study is a scientific method carried out in a planned manner, from the elaboration of questions to the presentation of results.
This research is exploratory and descriptive [23,25], given the inherent need to elicit non-functional requirements through agile frameworks, in order to integrate quality validation.Some guidelines pointed out by Melegati and Wang [21] and Peterson [26] were followed, which involve a clear definition of research questions and goals, the appropriate selection of relevant cases for research, collection, and analysis of data from various sources, such as interviews, observations, and documentation.There is also a need to ensure the validity and reliability of the data through triangulation and member checking.
This research analyzed the case of a team that adopted BDD for non-functional requirements and test automation based on the performance efficiency characteristic of the ISO/IEC/IEEE 25010 standard.Experienced teams are understood to have more than one year of experience in adopting BDD; however, they still need to apply it to non-functional requirements.The case study focused on analyzing requirements elicitation in one of the products developed by a Brazilian software manufacturer.The tested product is for customer management and product pricing.This section describes the steps followed during the research, highlighting the team members' participation in the process.
Below, for better understanding, a BPMN diagram of the steps of this research is presented in Figure 1.When the research topic was defined, the method that achieved the relationship proposed as a general goal between the ISO/IEC/IEEE 25010 standard and BDD was chosen, followed by the elaboration of research questions, the case study carried out, a list of threats to validity how we mitigate them, and, finally, conclusion.

Case Study
We applied this case study to a system with mixed CRM and people-management characteristics, so we chose it due to its relevance to the company.In addition, the systems only adopt BDD for functional requirements and do not cover non-functional ones, which, according to the quality team, lacks information for tests to be carried out, generating retests in several case.A multidisciplinary team was involved during the case study with the following team members:

•
A software architect responsible for defining the architecture and non-functional requirements; • Two developers in charge of implementing the acceptance scenarios; • A team lead, responsible for leading the process and helping with communication between team members; • A test analyst tasked with automating performance tests using Locust in Python; • A DevOps, responsible for configuring the infrastructure and deploying the system.
The software architect created user stories and acceptance scenarios that described non-functional performance requirements based on the characteristics of the ISO/IEC/IEEE 25010 standard, following the gherkin language.These user stories and acceptance scenarios were presented to the team and relevant stakeholders to gain their approval, ensuring a common understanding of the non-functional requirements.
The tests were automated after the user stories were written, and the acceptance and approval criteria were set by those involved.The test analyst was responsible for automating performance tests using the Locust tool in Python, ensuring that acceptance scenarios were translated into executable tests.
The tests were carried out, and the metrics available in the acceptance criteria were monitored using analysis tools.During the execution of performance tests, the multidisciplinary team, including the software architect, test analyst, and DevOps, tracked the results and metrics generated by monitoring and analyzing the results using Grafana and Jaeger tools.

Interview
Before executing the automation tests, one of the authors interviewed the team lead who used BDD in this case study.In a brief conversation, the team lead informed us of how the requirements were approved by interested parties to achieve the behavior expected by the software.Furthermore, he mentioned the importance of integration between the team, the process, and the quality of the software, aiming to obtain releases that meet what was requested and stating the importance of using BDD from the elicitation of the requirement to the execution of the test cases.
In order to guarantee the performance efficiency characteristic expressed in the ISO/IEC/IEEE 25010 standard, as well as its sub-characteristics, based on the elicitation of requirements, user stories were created with their respective acceptance criteria reported in the following section.

Case Execution
The execution of the case study is based on the functionalities and scenarios defined following the ISO/IEC/IEEE 25010 standard using BDD.The case study intended to evaluate the system's performance efficiency concerning three sub-characteristics: time behavior, resource utilization, and capacity.User stories and acceptance criteria, following the gherkin language, a standardized and widely adopted language in BDD frameworks are described below.This standardization ensures the reliability and consistency of these concepts, instilling confidence in their use.Each subsection corresponds to one of the sub-characteristics addressed in this study in relation to the ISO/IEC/IEEE 25010 standard.Figures 2-4 below were extracted from Locust and are part of the BDD automation.

User Story: System Efficiency (Time Behavior)
Description: As a system administrator, the team seeks to ensure the system can handle 100 concurrent users for 7 days, ensuring efficiency is consistent during periods of high demand.
Scenario: Time behavior test.Given that the system is working correctly and can accommodate 100 simultaneous users, as shown in Figure 2. When 100 simultaneous users access the system for 7 consecutive days and each user creates an opportunity every 5 min, as shown in Figure 3. Then the system must maintain an average response time of less than 5 s and not present critical errors during the test, and the use of server resources must remain below 80%, as shown in Figure 4.

User Story: Resource Utilization
Description: As a system administrator, the team seeks to ensure that the system can handle 1000 concurrent users for 30 min, ensuring that server resource utilization remains below 85% during load testing.
Scenario: Resource utilization test.Given that the system is working correctly and has capacity for 1000 simultaneous users.
When 1000 simultaneous users access the system for 30 min.
Then server resource utilization should not exceed 85% and the system should not experience critical errors during testing.

User Story: Capacity
Description: As a system's administrator, the team seeks to ensure that the system can handle 2700 concurrent users for 30 min, ensuring that system capacity is sufficient to meet demand during peak usage.
Scenario: System capacity test.Given that the system is working correctly and has a capacity for 2700 simultaneous users.When 2700 simultaneous users access the system for 30 min.
Then server resource utilization should not exceed 90%, and the system should not experience critical errors during testing.

Results
Below are the answers to the research questions presented in Table 1.

QP1-How Does BDD Address Difficulties in Ensuring Non-Functional Requirements?
The BDD framework intended for test automation, such as Cucumber and JBehave, was initially designed to address the system's functional units, which resulted in significant challenges in the integration between user stories and their corresponding scenarios and subsequent automation.However, as evidenced in this investigation, it is possible to considerably mitigate the difficulties associated with the practical application of BDD in contexts involving non-functional requirements by adopting complementary tools.
In the present study, we opted for the combined use of Pytest-BDD in conjunction with Locust.While the former makes it possible to apply BDD in Python environments, the latter facilitates performance testing.As a result, an integration enables the validation of the acceptance criteria established by the BDD, focusing on evaluating the system's performance.This approach demonstrates that adopting complementary tools, such as Locust, allows the framework adopted for BDD to validate the acceptance criteria efficiently, providing more excellent reliability to the results obtained.
Adopting BDD combined with complementary tools emerges as a complete and adequate procedure to satisfy the system's non-functional requirements, allowing efficient compliance monitoring with the proposed scenarios.
6.2.QP2-How Can BDD Be Used to Ensure Quality Related to the Performance Efficiency Characteristic of the ISO/IEC/IEEE 25010 Standard?Solis and Wang [27] identified essential characteristics of BDD, highlighting scenario models and automated acceptance tests with mapping rules as preponderant elements.Such characteristics play a fundamental role in ensuring software quality.In this study, we adopted a set of tools based on user stories and acceptance criteria, enabling test automation and monitoring of user perspectives.To this end, we used Locust to evaluate system performance and Jaeger to analyze communication between services, as illustrated in Figure 5. Figure 5 presents Jaeger.This system shows communication between services, demonstrating performance and how each microservice communicates, allowing us to understand if there is an internal problem between services.This approach facilitates the work that links the need for performance that BDD pointed out and the possibility of verifying the behavior, not just of the response but also of the components linked to the response.
Figure 6 shows Grafana analyzing the containers of running services, allowing us to understand memory and processor consumption during tests, thus allowing us to crossreference information from the point of view of the user with Locust, the service with Jaeger, and the server with Grafana, with a 360 view of the system's performance.The assessment of infrastructure components, monitored by Grafana, provided a comprehensive and end-to-end view of the system's performance.The application of BDD to ensure that the quality of a system's characteristics offers a series of advantages, particularly highlighting the substantial improvement in communication between the architect and the product development team.Furthermore, its intrinsic integration with automation processes provides greater transparency and ease in validating the system, ensuring that it effectively fulfills the characteristics outlined during the design process.
As evidenced in Section 4.1, BDD becomes more transparent and accurate in the elicitation of requirements when using the gherkin language.In this way, we achieved the behaviors expected by the software, achieving success in automating non-functional tests.Furthermore, Figure 1 shows the importance of acceptance criteria meeting the tests to guarantee the quality initially proposed by eliciting the requirement.

Discussion
The execution of these test scenarios made it possible to evaluate the system's performance concerning the sub-characteristics inherent to the performance efficiency characteristic of the ISO/IEC/IEEE 25010 standard and ensure that the defined non-functional requirements were met.Thus, scripts were created for testing in Python with Locust.
As mentioned by Haoues et al. [7], to maintain software quality following the ISO/IEC/ IEEE 25010 standard, there is a need to focus on the quality aspect throughout the entire cycle of software life.In this way, BDD managed to present positive results from eliciting requirements, carried out objectively among team members, to the automation of tests to meet the proposed acceptance criteria.
According to Estdale and Georgiadou [8], non-functional requirements present improvements regarding the functioning of the software.As they are considered more complex, it is expected to have difficulty in eliciting non-functional requirements.Through the gherkin language used by BDD, there was a better understanding of what to expect from the software's behavior, so user stories were described with a clear purpose.
According to Jarzkebowicz and Weichbroth [9], it is necessary to understand how non-functional requirements are elicited.Through this study, we perceived the necessary steps for the elicitation of non-functional requirements, as shown in Figure 1, in order to realize the need to validate the requirement through user stories and their respective acceptance criteria to identify whether what was expected by the software was achieved or not.
Regarding the systematic literature review with a focus on non-functional requirements carried out by Olsson, Sentilles, and Papatheocharous [10], the authors observed the need to carry out studies that validate what has been researched in academia in order to deliver answers to assertions consistent with the reality of the industry.In this way, this research contributed to understanding the elicitation of non-functional requirements with BDD in the current situation.
As North [13] mentioned, BDD was created to mitigate problems arising from Test-Driven Development (TDD), so BDD is focused on the behavior of releases, while TDD focuses directly on testing.Thus, the author also states that BDD has the benefits of improving communication and readability, which are aspects perceived in this research in the elicitation of non-functional requirements, so there was the participation of the parts involved in the development of releases, such as what was described in Section 4.

Threats to Validity
Threats to validity are mitigated during research to carry out a study with better reliability [23].The descriptions made by Zhou et al. were used [28] regarding types of validity.

Construction Validity
Understanding the ISO/IEC/IEEE 25010 standard and the BDD was necessary for the case study of this research to be carried out.Based on what we presented in Section 3, it became possible to identify the importance of analyzing whether the elicitation of nonfunctional requirements through BDD followed the performance efficiency characteristic set out in the ISO/IEC/IEEE 25010 standard.

Internal Validity
We carried out this research with the participation of four collaborators: two carried out the collection of information in pairs regarding the ISO/IEC/IEEE 25010 standard and BDD, another carried out the case study, and, finally, the other carried out the refinement of information throughout the article writing process.

External Validity
With the aim of maintaining quality in results, we considered the members who would be part of the team that would elicit the non-functional requirements set out in Section 4. In this way, it was possible to understand each team member's role in eliciting and implementing a requirement.

Conclusions Validity
This research is replicable using the methodological steps described in Section 4 and theoretical research regarding the ISO/IEC/IEEE 25010 standard and BDD.

Conclusions
In this study, the BDD framework was adopted for the elicitation of non-functional requirements and the ISO/IEC/IEEE 25010 standard, specifically the performance efficiency characteristic with its three sub-characteristics, namely time behavior, resource utilization, and capacity.Each subcharacteristics was associated with a user story, accompanied by one or more acceptance scenarios, as exemplified in Section 5.This approach made it possible to guide user testing, establishing links between efficiency characteristics and how requirements are being elicited through the BDD gherkin language.For the company using BDD to align non-functional requirements is crucial, as it facilitates product approval.Furthermore, it fosters closer integration between the Quality team and the product since BDD requirements are directly linked to the product.
Furthermore, as it is written in gherkin language, BDD contributed to eliciting nonfunctional requirements about BDD's objectivity for writing focused on the system's behavior.In this way, the use of BDD helped those involved in the requirements elicitation process to mitigate possible ambiguities in writing user stories and their respective acceptance scenarios.
The use of BDD as a framework in the software life cycle leads to improved communication on the part of the members involved in the process to improve the software development stages, from requirements elicitation to documentation and maintenance.This research brought a holistic approach to the system's quality requirements by relating BDD to the ISO/IEC/IEEE 25010 standard.Through this study, BDD showed its effectiveness following the performance efficiency characteristic expressed in the ISO/IEC standard /IEEE 25010 as per Section 6.
The obstacles encountered in this study were mitigated according to Section 8 to guarantee the results' quality.The research questions expressed in Table 1 were answered according to Section 6, in addition to the process used for the elicitation of requirements and the test automation being detailed, aiming to clarify the step-by-step process used so that other researchers can replicate it.
As BDD is a framework used from the elicitation of requirements to software implementation and maintenance, communication between stakeholders and the development team is necessary to contribute to the quality of the software delivered.Thus, when using BDD in software development, relevant people can present its importance so that BDD can achieve its goal proposed by Dan North: a framework that assists in communication and delivers releases according to the expected behavior.Therefore, this study involves people, processes, and techniques to maintain the system's quality entirely.
As suggestions for future work, the methodology used in this research could be replicated in a new study with teams that use BDD using other quality characteristics of the ISO/IEC/IEEE 25010 standard.A survey could also be carried out with a focus on the quality characteristics of the ISO/IEC/IEEE 25010 standard with professionals who use BDD in order to identify whether software development is based on the quality characteristics expressed in the ISO/IEC/IEEE 25010 standard.Finally, an experiment could be carried out that compares the use of BDD to elicit non-functional requirements with and without the ISO/IEC/IEEE 25010 standard as a reference to observe the impact caused by the quality expressed in the standard.

Figure 1 .
Figure 1.Steps of the research.

Figure 3 .
Figure 3. CPU and memory consumption during test execution, with low memory consumption and some peaks in CPU consumption.

Figure 6 .
Figure 6.Evaluation of the infrastructure components, pods/containers, CPU, and memory consumption during test execution, with some consumption peaks.

6. 3 .
QP3-What Are the Benefits of Using BDD in Identifying and Automating Non-Functional Tests?