DEVOPS PROCESS MODEL ADOPTION IN SAUDI ARABIA: AN EMPIRICAL STUDY

Nowadays, IT organizations are not seeing DevOps as a competitive advantage or added value, but how can organizations survive if not adopting it? Many software development organizations are adopting DevOps software processes to foster better collaboration between development and operation teams, to improve the software development process’s quality and efficiency; therefore, it's very important to measure the adoption of DevOps by these organizations. Maturity models are used as a tool to assess the effectiveness of organizational processes on adopting certain practices and identify what capabilities they need to acquire next to improve their performance and reach a higher maturity level. Few DevOps maturity models have, recently, emerged as a means to assess DevOps adopted practices. This research aims to conduct an empirical field study to assess the DevOps adoption level in seven Saudi organizations using one of the published DevOps maturity models; namely, the Bucena model. The findings show that the adoption of DevOps in the surveyed Saudi organizations is promising; despite that, some factors related to DevOps culture, process and technology are weak and need more attention to enhance them to achieve better performance and continuous delivery.


INTRODUCTION
Software engineers have noticed the gap between development and operation teams. The collaboration between the two teams has been discussed where this discussion yielded a new process model called DevOps. Nowadays, IT organizations are not seeing DevOps as a competitive advantage or added value, but how can organizations survive if not adopting it? The new process model aims to achieve fast highquality releases of the software product. DevOps is the new software process that extends the agility practices within the collaborative culture to enhance the process of software development and delivery [1]- [2]. Moreover, the DevOps approach is concerned with improving the collaboration between the development and operation teams, which represents a new shift in understanding the way to build software systems. Both the development team and the operation team have different goals in the project; developers' goal is to release the new features of the software to production, whereas the operators' goal is to keep the software as stable and available as possible. To maintain these two goals, the collaboration between development and operation teams as early as possible in the project is vital. This collaboration is not considered in agile methodologies to achieve fast, high-quality software releases. Hence, DevOps changes the workflow of traditional software development to accelerate and streamline software delivery, which means changing not only the process flow, but also the organizational culture in developing and delivering software. This means that adopting DevOps may spur the organization to introduce new processes, personnel and technological change [3]. Nowadays, software delivery is treated as a continuously evolving process to meet user expectations. DevOps makes this possible by bringing the development and operation teams together to facilitate collaboration, continuous integration, continuous delivery and automation, which will result in reduced time to market, enhanced customer experience, improved quality assurance and reduced costs.
According to Forrester data [4], more than 50% of organizations across industries, including healthcare, manufacturing and banking organizations, have already incorporated DevOps as part of their digital strategy. Although DevOps has gained recently more interest in academia and practice, the literature still has a limited amount of academic publications as well as empirical studies related to DevOps [5]. Consequently, few maturity models and empirical studies related to DevOps exist in the literature. One of these maturity models will be used in this study, based on a recommendation of a previous study [6], to conduct a new empirical study to assess the maturity of DevOps adoption in several Saudi organizations. The main purpose of this study is to assess the maturity of DevOps adoption level in Saudi organizations as the DevOps process model is new and those organizations have no idea about how good they are in adopting it. For this research, a survey study has been designed to assess the maturity of DevOps adoption in Saudi organizations. The research conducted in this paper is a multicase study that collects data via interviews and surveys. The research methodology adopted in this research is summarized in steps as follows: 1. Reviewing available maturity models and choosing one to use in the empirical study. 2. Developing the assessment method (interview/survey questions).

Reporting findings
The first step of this research is a theoretical one that aims to review available DevOps maturity models, which is briefly discussed in the next section. For more details about this step and detailed comparison, please refer to [6]. Then, an assessment method is developed to be used in evaluating DevOps adoption in various Saudi organizations. The second part of this methodology is concerned conducting the empirical study that assesses the maturity of DevOps adoption in Saudi Arabia, which is discussed in Sections 4 and 5 of this research.

LITERATURE REVIEW
Software products can be improved to deliver better results by adopting DevOps practices. The software product quality will increase when adopting DevOps practices that consider the strong relationship between culture, automation, measurement and sharing, as they enhance quality [7]. DevOps is consisting of practices and cultural values to minimize the barriers between development and operation teams and DevOps adoption involves a tight relationship between agility, automation, collaborative culture, continuous measurement, quality assurance, resilience, sharing and transparency [8]. Therefore, using practices that make the software product available and ready to the requester as soon as it gets implemented leads to the importance of understanding how the deployment practices are applied in the development team and that can be achieved through establishing a proper maturity model and using it to survey the development team and operation team and their practices [9]. A qualitative case study for three software development companies in Finland was conducted to indicate the benefits and challenges involved in adopting DevOps in 2016 [10]. Another case study in Finland was conducted to measure the impact of DevOps adoption focusing on mixing responsibilities between the development and operation teams [11]. From the result of that study, DevOps practices are influencing positively software products in terms of released duration and quality. A few years ago, it was claimed that no dedicated maturity models for DevOps exist [12]. Recently, few DevOps maturity models have been developed and documented in the literature. Most of them are based on capability maturity mode as this model and its related models are found feasible to guide the process improvement for DevOps processes [13]. In the coming few paragraphs, we discuss a sample of these models briefly.
First, Bahrs form IBM provided an analysis of the adoption of the IBM DevOps approach for promoting continuous delivery of software [14]. The author identified four dimensions in adopting or implementing continuous software growth within an organization. These dimensions include: planning and measuring, developing and testing, releasing and deploying and monitoring and optimizing. The maturity mode was defined with four levels that are: practiced, consistent, reliable and scaled. The IBM DevOps maturity model is practice-based and reflects a wider context within the adoption framework of a software development organization. It focuses on defining the best practices to be applied in the adoption of new software solutions iteratively. The main strengths of this approach include the fact that it provides a well-articulated way for assessing current DevOps practices within an organization. It also helps in defining a clear roadmap for DevOps implementation.
Second, a DevOps maturity model was proposed by Mohamed and used to assess global software development practices and processes [3]. The proposed DevOps maturity model is based on the Capability Maturity Model Integration (CMMI) and is composed of five maturity levels, which are: initial, managed, defined, measured and optimized against four dimensions that include: quality, automation, collaboration and governance. CMMI helps organizations improve productivity, reduce defects, optimize the process and ensure predictability and efficiency of operations [15]. Adopting the CMMI model to develop the DevOps maturity model will help achieve these benefits in the domain of DevOps processes.
Third, another suggested DevOps maturity model based on (CMMI) process model is proposed in [12]. The model uses a combination of CMMI for development (CMMI-DEV) and CMMI for services (Dev-SVC) to evaluate the maturity of adopting DevOps. That approach was tested on a specific software project at a very large telecom organization, where there were more than 100 developers and 8 operation people working on that project. The software project was adopting DevOps and a test assessment was conducted using SCAMPI C assessment, which is the least formal assessment method. The results showed that the use of CMMI model (CMMI-Dev and CMMI-SVC) can support the purpose of evaluating the processes within software projects that are adopting DevOps practice. However, that study tested only the processes at level 2 (managed) and level 3 (defined). Fourth, another model that is aligned with the CMMI maturity for DevOps adoption was presented by the employees from Hewlett Packard Enterprise (HPE) [16]. That model is designed to cover the entire lifecycle of an application for large organizations, regardless of the change being determined by the development team or the IT operation teams. It was applied to measure process, automation and collaboration dimensions and the levels of this model are: initial, managed, defined, measured and optimized.
Fifth, a suggested maturity model with five levels is proposed in [17]. The model has three dimensions; namely: people, process and tools. The model levels are: basic, emerging, co-ordinate, enhanced and top level. Sixth, maturity model for DevOps with four levels has been proposed in [18], known as the Bucena maturity model. Its levels are: initial, repeatable, defined, managed and optimized. The model has four DevOps dimensions, which are:  Culture: Adopting DevOps requires a new culture that supports transparency and a good supporting environment between the development and operation teams [19]. This can be achieved by arranging regular meetings between both teams. The development team should be supportive of the operation team during the release and production of the software [13].  People: The team members of a DevOps process should be skilled persons with high ability in improving their skills via self-learning and team-learning. Team members should show a high level of collaboration and support for each other.  Process: The DevOps process focuses on continuous delivery, continuous testing, continuous integration and continuous monitoring. The DevOps processes adopt agile practices in development.  Technology: This dimension discusses the technologies and tools support the DevOps process in continuous delivery and to bring the development and operation environments to work collaboratively. It also provides various automation technologies to support the process dimension, hence increase productivity.
Bucena's model [18] revolves around the provision of a DevOps methodology for implementation within small enterprises. Note that the authors of this model promote this model as part of helping very small entities to adopt DevOps via three steps, see [20]. Although this maturity model is focusing on very small entities, it can still be used for anyone interested in adopting it or experiencing its benefits [20]. Accordingly, this research paper adopts the Bucena maturity model to conduct the field study, because it assesses the culture dimension which is a critical dimension to improve software quality in a DevOps environment [7]- [8] and because the model is originated from academia, not white papers. We focus in this paper only on one step, which is assessing the current organizational DevOps maturity level while leaving the issue of how to enhance the DevOps maturity level for each organization to decide on it.

DATA COLLECTION
The sampling that was chosen to be part of this study consists of seven Saudi organizations that provide software products and have started practicing DevOps. These organizations are notated as organization A, B, C, D, E, F and G and are briefly introduced as follows: 1. Organization A: A Saudi governmental organization that provides various IT serves to the public. 2. Organization B: A Saudi organization that provides IT services to governmental agencies. 3. Organization C: A Saudi organization that provides healthcare information technology (HIT) solutions 4. Organization D: A business services Saudi organization that is focusing on implementing smart solutions, business services and data services. 5. Organization E: A Saudi organization that provides and supports custom and commercial offthe-shelf solutions in different sectors, such as health sectors. 6. Organization F: Saudi organization that supports digital infrastructure, information security and system development. 7. Organization G: A Saudi organization that provides modern information technology and communication manufacturing, system integration, as well as operation and maintenance services in Saudi Arabia.
Two types of data generating methods are used in this study: First, a structured interview that consists of demographic questions with a senior-level employee in each organization to collect demographic data about the organization, like organization size, organization domain, size of the delivery team, the duration of experiencing agile development and the duration of adopting DevOps. Tables 1 summarizes this information for the seven organizations. Second, a questionnaire is developed by using an online survey tool (SurveyMonkey). This questionnaire is designed based on the Bucena DevOps maturity model. The goal of the questionnaire is to assess the DevOps maturity level of Saudi organizations. The questionnaire consists of 23 questions grouped as follows: 9 questions related to DevOps technology dimension, 6 questions related to the DevOps process dimension, 5 questions related to DevOps culture dimension and 3 questions related to DevOps people dimension. Table 2 illustrates a sample of the mapping of the DevOps dimensions, DevOps factors, questions and possible answers. Moreover, each question represents a single factor in a dimension and a score is assigned for each question/factor: level 1 for the first option, level 2 for the second option, level 3 for the third option, level 4 for the fourth option and level 5 for the fifth and other options. Appendix A provides a list of questions developed for the interview and the questionnaire. Note that the factors for each dimension are identified and discussed in the Bucena maturity model, while the description of each maturity level for each factor is used to devise the survey questions and possible answers. For more information about the maturity model refer to [18].
Note that the CMMI maturity model provides two representations; the continuous representation where the processes are assessed individually according to their capability level and the staged representation where the whole organization is assessed according to its maturity level. For more details about CMMI maturity model, refer to [13]. In this paper, where Bucena maturity model is used to assess DevOps maturity and at the time of writing this paper and conducting the case study, no details have been given about this model in terms of how to calculate the capability level or maturity level of the organization. Hence, the authors of this paper have calculated the dimension capability level based on the dimension level formula, see formula number 1. This formula is used to calculate the capability level, because some of the questions are out of four and not all of them are out of five. Moreover, the organization maturity level is calculated by summing up the four dimensions' capability levels and dividing them by four (number of DevOps dimensions).
where: f = the factor in the desired dimension, N = number of factors in the desired dimension l = 5 (number of levels).

RESULTS AND ANALYSIS
After conducting the questionnaire on the seven organizations from different industries in Saudi Arabia, the maturity level was measured (level 1: initial, level 2: repeatable, level 3: defined, level 4: managed and level 5: optimized) against the factors associated with each dimension (technology, process, people and culture). Note that the dimensions rated less than 3 out of 5 are considered weakness points and need more attention to improve them and their factors.

Technology Dimension
The technology dimension has nine factors to assess its capability. Five of these factors are having the possible maximum value of 5 and the other ones are having the possible maximum value of 4. Table 3 shows the achieved level for each factor in the technology dimension among the surveyed organizations. The analysis of the technology factors shows that six out of the nine factors are showing good average levels; i.e., achieved average level 3 or above, among the surveyed organizations.
Although all organizations achieved good capability levels for the technology dimension; i.e., achieved level 3 or above, three technology factors achieved low average levels, e.g. less than 3; namely, data management, software configuration management and issue tracking. These technology factors need more focus by all organizations. In other words, tools and automation of these factors should be activated and properly adopted by organizations.

Process Dimension
The process dimension has six factors to measure the process's maturity. These factors are delivery, development, testing, project management, documentation and organization processes, see Table 4.
The analysis of the process factors shows that two out of the six factors are showing good average levels; i.e., achieved average level 3 or above, among the surveyed organizations. The delivery and testing process factors are considered the best factors in the process dimension. Although all organizations achieved good capability levels for the process dimension; i.e., achieved level 3 or above, four process factors achieved low average levels, e.g. less than 3; namely, the development process, the project management process, the documentation process and the organization process. These process factors need more focus from the organizations.

People Dimension
This dimension has three factors to measure its capability, which are team organization, learning process and development of competencies and capabilities. Table 5 shows the achieved level for each factor in people dimension among the surveyed organizations.
The analysis of the people factors shows that two out of the three factors are showing good average levels; i.e., achieved average level 3 or above, among the surveyed organizations. Two factors achieved low average levels, e.g. less than 3; namely, team organization and development of competencies and capabilities.

Culture Dimension
This dimension has six factors to measure the culture capability, which are: communication type, requirement understanding, culture understanding, collaboration and innovation drivers. Table 6 shows the achieved level for each factor in the culture dimension among the surveyed organizations. The analysis of the cultural factors shows that three out of the five factors are showing good average levels; i.e., achieved average level 3 or above, among the surveyed organizations. Two factors achieved low average levels, e.g. less than 3; namely, cultural understanding and innovation drivers. Figure 1 illustrates the achieved capability level for each dimension in each of the surveyed organizations. Table 7 illustrates the calculated maturity levels for each organization, while Figure 2 visually illustrates the maturity levels. Even though the organizations achieved maturity levels 3 or above, Tables 3, 4, 5 and 6 depict that some factors in the different dimensions gained capability levels less than 3 and need improvement. Accordingly, organizations need to pay attention not only to their overall maturity level, but also to the individual capability of each dimension and the level achieved per factor as well. For instance, Organization A has achieved maturity level 3 and its capability level for each dimension is 3, but some factors gained a level less than 3, so these factors need to be improved for better performance of the whole DevOps process.
 Technology: Databases are a key component for the operation team, which means that it is part of the DevOps whole process. Organizations should pay attention to how they manage their databases in an agile manner; i.e., data should be flexible to change quickly and reliably. For instance, organizations should decide how to shift toward a more agile database; whether to keep using traditional relational DBMSs or shift toward other forms, such as NoSQL or Casandra. Another issue is related to adopting more automation in software configuration management as well as issue tracking for changes that may happen during continuous integration and continuous deployment and decide which tools suit them more.   3  4  5  4  2  3  3  People  3  3  4  5  3  3  3  Culture  3  3  5  5  3  4  3   Maturity level  3  3  4  4  3  3  3  Process: Organizations should improve their agile processes and make sure that agile principles and practices are implemented in the organization. Sometimes, organizations claim that they adopt agile development processes, but their processes are run traditionally. Moreover, project managers should think of how they can manage agile processes that support continuous integration and deployment, e.g. project managers may think of adopting xProcess agile planning tool. Documentation in agile processes with short iterations and releases is vital. Organizations need to specify how they document their development process when using DevOps. Organizations can make use of different documentation tools available to automate this process and achieve continuous documentation as well.
 People: Learning is seen as a process rather than an event organized once or twice. Organizations need to adopt a well-organized and planned learning process for their people and engineers to enhance their understanding of the DevOps process.
 Culture: Changing the organizational culture and engineers' mindset to adopt agility and DevOps practices is very critical for success. Engineers need to understand the culture of DevOps, where the development team is not responsible for development only and the operation team is not responsible for releasing and deploying it only, but instead, both teams are responsible for running the product and bring the system alive.

VALIDITY THREATS
This research was planned to take into account validity concerns that consist of three categories: construct validity, external validity and reliability [21].
Due to the novelty of the DevOps process, scientific papers that document the DevOps maturity models are few. The DevOps maturity model used in this research, as well as other related maturity models, are published in scientific papers, which in turn, have conducted a literature review to identify these models as well.
Regarding the conducted empirical study using interviews and survey, one of the possible threats that relate to reliability and construct validity is finding a sufficient number of Saudi organizations of different organization sizes that adopt DevOps to assure the reliability of the assessment survey. To solve this issue, we went to GITEX 2018 and visited some of the Saudi governments' booths and asked them about their preferred software provider organizations to consider them in this study. Also, we met big Saudi companies that participated in GITEX 2018 and asked them whether or not they adopt DevOps. After that, we conducted this study with three government organizations, three semigovernment organizations and one private company, with different sizes.
Moreover, another possible threat which is related to construct validity is concerned with representatives' interpretation of some questions. To avoid this, we asked the interviewees to answer the online survey during the face-to-face meeting and after the interview immediately. The questions have been explained to the organizations' representatives before they started filling the survey.
Regarding reliability, the survey is based on the Bucena DevOps maturity model, which is documented in the literature. The model has been validated by the authors in one small enterprise. Furthermore, three independent software engineers have answered the survey as a pilot study, before publishing it to verify its clearness.
Finally, another possible external validity threat is that participants may feel the need to present their company in the best light during the interview and survey. Therefore, to avoid this, it has been communicated to the participants that the results will be published anonymously and answering the questions in the most realistic view would give a better evaluation for their company which will help them identify the strengths and weaknesses of their DevOps adoption.

CONCLUSION
In this research, the Bucena DevOps maturity model was chosen to conduct an empirical study on Saudi organizations. Choosing the Bucena model does not mean that other maturity models are useless. On the contrary, other maturity modes deserve practicing to enrich the empirical studies on DevOps maturity models. The empirical study was conducted on seven Saudi organizations by interviewing these organizations' representatives and distributing surveys among them. These organizations vary in size; two organizations are medium-sized, three organizations are large-sized and two organizations are very large-sized.
This research shows a promising future for Saudi organizations in adopting DevOps. Despite this, Saudi organizations need to pay attention to various factors in the different DevOps dimensions which are found to be weakness points. Most importantly, organizations adopting DevOps need to adapt their engineers' mindset to understand and implement DevOps processes properly and enhance their learning process in this regard. Although automation and tools used are in a good position in some phases on the DevOps projects, such as testing, other phases need more work to automate them. This includes configuration management and issue tracking during the continuous delivery process. Moreover, organizations need to think about how they will manage their data and DBMS to meet agile principles and continuous delivery.
One final note for the maturity model developers is that it insufficient to develop the maturity model and its rating scale. IT is better if the assessment method that implements the model is also published along with the model. For instance, the CMMI model has its assessment method, which is known as the SCAMPI method. If the assessment method is not published or documented, researchers will develop their assessment methods to use the maturity model, as we did here and this can result in divergence among these methods.