Bivariate Software Reliability Growth Models under Budget Constraint for Development Management

Software reliability growth is observed by investing not only the testing-time but also the testing-effort in the testing-phase of software development process. If the testing-time (testing-effort) is reduced to some extent, it is possible to observe the software reliability growth by investing the amount of testing-effort (testing-time) which can compensate the insufficiency of the testing-time (testing-effort). However, most of the existing software reliability growth models (SRGMs) are constructed as univariate models and the substitutability between the testing-time and testing-effort is not considered. Additionally, it is necessary to remove many faults efficiently within the budget. In this paper, we develop bivariate Weibull type SRGMs under budget constraint based on the Cobb-Douglas type and CES (constant elasticity of substitution) type testing-time functions. Simultaneously, we evaluate the substitutability between the testing-time and testing-effort factors which are software reliability growth factors. Finally, we conduct the sensitivity analysis and show numerical examples by using actual data sets.


Introduction
Enormous testing-effort, such as CPU time, execution time, man-hour, test coverage, and executed test case, is invested in the actual testing-phase. However, it seems that existing univariate SRGMs (Pham, 2000;Yamada, 2014) cannot reflect the actual testing-phase because they assume that the reliability growth depends only on the testing-time. From these background, bivariate SRGMs have been developed in several literatures (Inoue and Yamada, 2008;Inoue et al., 2010Inoue et al., , 2017Ishii and Dohi, 2012;Kapur et al., 2010;Kapur and Pham, 2012). Especially, an approach using the Cobb-Douglas type production function (Inoue and Yamada, 2008) as a testing-time function is often used for expanding to bivariate models from univariate models. Since the Cobb-Douglas type production function is typical as a production/utility function on economics, and it has simple mathematic structure to handle. From previous research, the proposed bivariate SRGMs based on the Cobb-Douglas type testing-time function are more accurate than the existing univariate models. However, the parameter of the Cobb-Douglas type testing-time function has no economic significance. Therefore, the relationship between software reliability growth factors such as the testing-time and testing-effort cannot be described sufficiently.
Also, actual software development projects are not always proceeded as planned. For example, if enough testing period cannot be secured in the testing-process, the testing-effort which can compensate the insufficient testing-time is needed to be invested in order to observe the software reliability growth. However, the software reliability growth factor cannot substitute all of the other software reliability growth factor, realistically. Furthermore, it is needed to improve software quality/reliability within the development budget. Therefore, we need to develop bivariate SRGMs which can consider multiple software reliability growth factors under budget constraint, and understand the quantitative substitutability between software reliability growth factors.
In this paper, we apply not only the Cobb-Douglas type but also CES type production functions (Minamino et al., 2017(Minamino et al., , 2019 as testing-time functions and develop the bivariate Weibull type SRGMs based on these testing-time functions. Then, we introduce the concept of the elasticity of substitution between the software reliability growth factors in terms of economics. The elasticity of substitution means a measure of ease for substituting each software reliability growth factor in order to maintain the influence on the software reliability growth. The Cobb-Douglas type production function is greatly restricted with respect to the elasticity of substitution. However, the CES type production function which can be more flexible with respect to the elasticity of substitution. Additionally, the CES type production function generalizes various production functions including the Cobb-Douglas type production function. That is, we can assume the CES type production function as various testing-time functions.
Furthermore, we solve the optimization problem which maximizes the number of detectable faults under budget constraint in terms of software development management. Then, we propose the bivariate Weibull type SRGMs under budget constraint by introducing the optimal testing-time and the optimal amount of testing-effort to the previous bivariate Weibull type SRGMs. Finally, we show the result of sensitivity analysis by using actual data sets as numerical examples.

Bivariate Weibull Type SRGMs 2.1 Definition of Two-Dimensional Nonhomogeneous Poisson Process (NHPP)
Let { ( , ), ≥ 0, ≥ 0} denote a counting process representing the total number of faults detected up to the testing-time and the testing-effort . The probability mass function that faults are detected up to the testing-time and the testing-effort is derived as (1) where ( , ) and ℎ( , ) represent the mean value function and the intensity function of NHPP, respectively (Inoue and Yamada, 2008).

Several Testing-Time Functions
We assume that the testing-time for the existing SRGMs is expressed as a software reliability growth factor by using the following Eqs. (2) and (3) (Inoue and Yamada, 2008;Kapur et al., 2010;Kapur and Pham, 2012;Sakaguchi et al., 2018).
where is the testing-time factor (calendar time), and the testing-effort factor (test coverage). and represent the distribution parameter and the substitution parameter, respectively. Also, the CES type testing-time (production) function has the following characteristics: When approaches 1, the CES type testing-time function becomes a linear function. When approaches 0, it becomes the Cobb-Douglas type function. When approaches ∞, it becomes Leontief function. Next, the elasticity of substitution is defined as It expresses the ease of substitution between software reliability growth factors to maintain software reliability growth in terms of economics. Note that the elasticity of substitution for the Cobb-Douglas type testing-time (production) function always has 1. On the other hand, the elasticity of substitution for the CES type testing-time (production) function can take a value except 1.

Bivariate Weibull Type SRGMs Based on Several Testing-Time Functions
The univariate Weibull type SRGM is given by where represents the reliability growth parameter, and represents the scale parameter. Also, γ(0) = 0 and γ(∞) = ∞. It means that the debugging activities is imperfect. From Eq. (5), we provide the following bivariate Weibull type SRGMs based on the Cobb-Douglas type and CES type testing-time functions, respectively.

Software Reliability Analysis under Budget Constraint 4.1 Optimization Problem Based on the Bivariate C-D Weibull Model
We formulate the optimization problem based on the bivariate C-D Weibull model to estimate the number of detectable faults under the budget constraint.
where and represent the cost required in a testing-time factor and the cost required in a testing-effort factor, respectively. represents the budget for the testing-phase. From Eq. (18), we obtain the following optimal solutions ( * , * ) by using the Kuhn-Tucker condition. (20)

Optimization Problem Based on the Bivariate CES Weibull Model
Similarly, we formulate the optimization problem based on the bivariate CES Weibull model to From Eq. (21), we obtain the following optimal solutions ( * , * ) by using the Kuhn-Tucker condition.
Substituting Eq. (22) into Eq. (7), the maximum number of detectable faults under the budget constraint based on the bivariate CES Weibull type model is derived as follows:

Numerical Examples
We show numerical examples by using DS1: ( , , )( = 0,1,2, ⋯ ,24; 24 = 24 (weeks), 24 = 90.946, 24 = 296) and DS2: ( , , )( = 0,1,2, ⋯ ,21; 21 = 21 (weeks), 21 = 91.981, 21 = 212) (Fujiwara and Yamada, 2002) as actual data sets. represents the calendar time measured on the basis of weeks, represents the test coverage, represents the total number of faults detected during [0, ], [0, ]. These data sets are obtained in the testing-process for embedded software developed by software development engineers with approximately 7 to 10 years of development experience. These development scale is 1.972 × 10 5 (LOC: Lines of Code) and 1.630 × 10 5 (LOC), respectively. In order to apply the test coverage data as a testing-effort factor, we adjust the testing-time data and detected fault count data as follows: The execution rate of the specified testing period: ≡ , The expected fault detection rate: ≡ , where is the total number of data set ( , , ). Table 1 and 2 show the result of parameter estimation for the bivariate C-D Weibull model and the bivariate CES Weibull model, respectively. From the estimated parameter ̂ in Table 2, we can evaluate that the testing-environment for DS1 is easier to substitute between the software reliability growth factors than the testing-environment for DS2, relatively. Table 3 shows the result of goodness-of-fit comparison. We apply MSE (mean squared errors) as an evaluation criterion. The MSE is calculated by deriving the squared errors between the observed and estimated number of detected faults and is given by: Note that we derived the MSE of existing Weibull type SRGM after adjusting the testing-time data and detected fault count data as shown in Eqs. (24) and (25). From Table 3, we can say that the bivariate CES Weibull model has better performance than the existing Weibull type SRGM and the bivariate C-D Weibull model, and the significance of considering the bivariate model could be confirmed.   Table 4 and 5 show the result of sensitivity analysis for the maximum number of detectable faults under the budget constraint based on the bivariate C-D Weibull model. Note that , and I are given. Also, we assume that or , and I are constant in each Table. From Table 4, when the cost of the testing-effort increases, the number of detectable faults decreases and the testing-effort factor decreases. Also, the testing-time factor does not change although the testing-effort changes because the elasticity of substitution for the Cobb-Douglas type production function is 1. That is, the software reliability growth is not maintained because there is no substitution between them. The similar trend is seen in Table 5. Table 6 and 7 show the result of sensitivity analysis for the maximum number of detectable faults under the budget constraint based on the bivariate CES Weibull model. Similar to Table 4 and 5, , and I are given. Also, we assume that or , and I are constant in each Table. From Table  6, we can see that not only the testing-effort factor but also the testing-time factor changes in comparison with Table 4 because of the substitution. Note that the testing-time factor decreases because of the budget constraint. Furthermore, when the cost of the testing-effort factor decreases, the number of detectable faults increases. It means that the software reliability growth is maintained because of the substitution. The similar trend is seen in Table 7.