Optimal maintenance Of a series prOductiOn system with twO multi-cOmpOnent subsystems and an intermediate buffer

Series-parallel systems with intermediate buffers widely exist in reality. For example, a production line can have multiple production phases connected in series. Each phase can have several production units organised in parallel to enhance the performance of the system. Between these phases, some intermediate buffers are allocated to store work in process (WIP). These buffers can reduce the influence of the breakdown and maintenance of a subsystem on the production rate of the whole system. However, the effects of intermediate buffers also make the degradation process of the system more difficult to model. Some existing papers developed methods to evaluate the performance of the series-parallel or series system with intermediate buffers. Tan and Gershwin [20] investigated the steady-state of a general Markovian two-stage continuous-flow system by solving a system of differential equations that describes the dynamics of the system. After that, Tan and Gershwin [19] further applied their model to the steadystate analysis of more general situations, e.g. systems with multiple components in series or parallel in each subsystem. Alexandros and Chrissoleon[1] analysed the steady-state of a two-workstation onebuffer follow line by using the Markovian property of the system. Liu et al. [13] investigated a system similar to that in Ref. [1], which considers the asynchronous operations of independent parallel units. The system was modelled by a Quasi-Birth-Death (QBD) process that can be solved efficiently. When there are more than two subsystems (components) in a series-parallel (series) system, the above-mentioned performance evaluation approaches based on steady-state analysis become impractical. Besides methods using the Monte Carlo simulation [9], some approximate approaches e.g., the aggregation method [4, 8, 21] and the decomposition method [5, 12], are developed to evaluate the performance of the system analytically. Although the above papers addressed the performance evaluation of a series-parallel system, these papers assumed a predetermined maintenance strategy, while the maintenance strategy optimisation is not considered. Yifan Zhou Zhisheng ZhAng


Introduction
Series-parallel systems with intermediate buffers widely exist in reality.For example, a production line can have multiple production phases connected in series.Each phase can have several production units organised in parallel to enhance the performance of the system.Between these phases, some intermediate buffers are allocated to store work in process (WIP).These buffers can reduce the influence of the breakdown and maintenance of a subsystem on the production rate of the whole system.However, the effects of intermediate buffers also make the degradation process of the system more difficult to model.Some existing papers developed methods to evaluate the performance of the series-parallel or series system with intermediate buffers.Tan and Gershwin [20] investigated the steady-state of a general Markovian two-stage continuous-flow system by solving a system of differential equations that describes the dynamics of the system.After that, Tan and Gershwin [19] further applied their model to the steadystate analysis of more general situations, e.g.systems with multiple components in series or parallel in each subsystem.Alexandros and Chrissoleon [1] analysed the steady-state of a two-workstation onebuffer follow line by using the Markovian property of the system.Liu et al. [13] investigated a system similar to that in Ref. [1], which considers the asynchronous operations of independent parallel units.The system was modelled by a Quasi-Birth-Death (QBD) process that can be solved efficiently.When there are more than two subsystems (components) in a series-parallel (series) system, the above-mentioned performance evaluation approaches based on steady-state analysis become impractical.Besides methods using the Monte Carlo simulation [9], some approximate approaches e.g., the aggregation method [4,8,21] and the decomposition method [5,12], are developed to evaluate the performance of the system analytically.Although the above papers addressed the performance evaluation of a series-parallel system, these papers assumed a predetermined maintenance strategy, while the maintenance strategy optimisation is not considered.

Yifan Zhou Zhisheng ZhAng
Optimal maintenance Of a series prOductiOn system with twO multi-cOmpOnent subsystems and an intermediate buffer

Optymalna strategia utrzymania ruchu dla seryjnegO systemu prOdukcji złOżOnegO z dwóch pOdsystemów wielOskładnikOwych Oraz bufOru pOśredniegO
Intermediate buffers often exist in practical production systems to reduce the influence of the breakdown and maintenance ef subsystems on system production.At the same time, the effects of intermediate buffers also make the degradation process of the system more difficult to model.Some existing papers investigate the performance evaluation and maintenance optimisation of a production system with intermediate buffers under a predetermined maintenance strategy structure.However, only few papers pay attention to the property of the optimal maintenance strategy structure.This paper develops a method based on the Markov decision process to identify the optimal maintenance strategy for a series-parallel system with two multi-component subsystems and an intermediate buffer.The structure of the obtained optimal maintenance strategy is analysed, which shows that the optimal strategy structure cannot be modelled by a limited number of parameters.However, some useful properties of the strategy structure are obtained, which can simplify the maintenance optimisation.Another interesting finding is that a large buffer capacity cannot always bring about high average revenue even through the cost of holding an item in the buffer is much smaller than the production revenue per item.
Keywords: series-parallel systems,intermediate buffers, Markov decision process,policy iteration,generalized minimum residual method.
Słowa kluczowe: układ szeregowo-równoległy, bufor pośredni, proces decyzyjny Markowa, iteracja strategii, uogólniona metoda najmniejszego residuum sciENcE aNd tEchNology Some other research focused on the maintenance optimisation of a production system with intermediate buffers.Zhou et al. [25] developed an opportunistic preventive maintenance policy for a multiunit series systems with intermediate buffers based on the dynamic programming.The cumulative opportunistic maintenance cost savings was adopted as the objective function.Ribeiro et al. [17] proposed a mixed integer linear programming model to jointly optimise the maintenance strategy and the buffer size.Dehayem Nodem et al. [6] simultaneously optimised the production and maintenance of a system with a production unit and a buffer-inventory.Murino et al. [16] applied three thresholds (i.e.warning threshold, opportune threshold, and preventive threshold) on the condition of the components in a series system with intermediate buffers.Both the thresholds and the buffer size were optimised through a simulation approach.Zequeira et al. [23] optimised the maintenance strategy and the buffer size of a production system, where the opportunities to carry out a maintenance action were assumed to be random.Arab et al. [2] optimised the maintenance of a production system with intermediate buffers incorporating dynamics of the production system and real-time information from workstations.The maintenance optimisation was performed on a simulation optimisation platform.The degradation and failures of the units were not discussed in that paper.The above papers preliminarily addressed the maintenance optimisation problem of production systems with intermediate buffers.However, these papers adopted predetermined maintenance strategy structures that are not proved optimal.Some papers did not consider the relationship among the maintenance action to a component, the states of other components, and the buffer level.Other papers obtained a short-term optimal maintenance strategy.
Only few papers investigated the property of the optimal maintenance strategy structure of a production system with intermediate buffers.Kyriakidis and Dimitrakos [11] optimised the maintenance of a two-unit series system with an intermediate buffer, in which only the upstream unit suffered from degradation.The optimal maintenance policy of the upstream unit was proved to be a control-limit type for a fixed buffer level.Later, Dimitrakos and Kyriakidis [7] extended their research in Ref. [11] by using continuous distributions to model the repair time.During the numerical study, Dimitrakos and Kyriakidis found that the optimal strategy structure is also of a control-limit type.In Ref. [10], Karamatsoukis and Kyriakidis assumed a more general situation that both the upstream and downstream units deteriorate with time.It was proved that the optimal maintenance strategy of the two units both have a control-limit property.The above-mentioned papers largely focus on a two-unit series system.However, in practice, the buffer can often exist in a series-parallel system.To address this issue, this paper further investigates the situation that both the upstream and downstream subsystems contain multiple parallel-connected components.When multiple components are included in a subsystem, the degradation process of the subsystem becomes difficult to model and the maintenance strategy structure becomes more complex.In a previous paper of the authors [24], the optimal maintenance strategy structure of a two-unit series system without intermediate buffers was investigated by a Markov decision process (MDP).In this paper, the MDP is also adopted to model the system degradation and repairing process; similar to Ref. [24],the policy iteration is used to solve the MDP to obtain the optimal maintenance strategy.Because the transition matrix of the system states is sparse, the sparse incomplete LU factorization and the generalized minimum residual (GMRES) method are used in this paper to solve the system of linear equations during the policy iteration.Thus, the policy iteration method in this paper is more efficient than that in Ref. [24].The structure of the obtained optimal maintenance strategy is investigated, which privies a reference for other maintenance optimisation approaches (e.g., the embedded MDP and the method based on steady-state analysis).Furthermore, the influence of the buffer capacity on the optimal average revenue is analysed.The result shows that large buffer capacity can bring down the average revenue even when the inventory holding cost rate per item is considerably smaller than the production revenue per item.This counter-intuitive result indicates that the buffer capacity should be optimised according to system parameters.
The remaining parts of this paper are organised as follows.Section 2 introduces the mathematical formulation and assumptions of the maintenance optimisation problem.Then, an approach to identifying the optimal maintenance strategy is developed in Section 3.After that, numerical studies are performed in Section 4 to evaluate the performance of the proposed maintenance optimisation method.Section 4 also investigates the properties of the derived optimal maintenance strategy structure.Finally, Section 5 gives the conclusion of the whole paper.

Nomenclature c pu :
the cost rate of the preventive maintenance to a component in the upstream subsystem in one unit time c pd : the cost rate of the preventive maintenance to a component in the downstream subsystem in one unit time c cu : the cost rate of the corrective maintenance to a component in the upstream subsystem in one unit time c cd : the cost rate of the corrective maintenance to a component in the downstream subsystem in one unit time c ou (i, q): the operating cost of a component in the upstream subsystem when the component is in state I and under production rate q c od (i, q): the operating cost of a component in the downstream subsystem when the component is in state iand under production rate q c h : the cost of holding an item in the buffer per unit time K(t): the buffer level at time t N k : the buffer capacity N u : the failure state of a component in the upstream subsystem N d : the failure state of a component in the downstream subsystem p pu : the probability of successfully performing the preventive maintenance to a component in the upstream subsystem in one unit time p pd : the probability of successfully performing the preventive maintenance to a component in the downstream subsystem in one unit time p cu : the probability of successfully performing the corrective maintenance to a component in the upstream subsystem in one unit time p cd : the probability of successfully performing the corrective maintenance to a component in the downstream subsystem in one unit time P u (q): the transition matrix of a component in the upstream subsystem under production rate q P d (q): the transition matrix of a component in the downstream subsystem under production rate q u P : the transition matrix of a component in the upstream subsystem under the nominal production rate q u idel u P : the transition matrix of a component in the upstream subsystem when its production rate is zero the transition matrix of a component in the downstream subsystem when its production rate is zero P s,w (θ): the probability that the system is in state w after one unit time when the current state of the system is s and the adopted maintenance action is θ q u : the nominal production rates of a component in the upstream subsystem q d : the nominal production rates of a component in the downstream subsystem Q u,m (k,θ u ): the production rate of component m in the upstream subsystem when the buffer level is k and the maintenance action is θ u Q d,m (k,θ d ): the production rate of component m in the downstream subsystem when the buffer level is k and the maintenance action is θ d r p : the production revenue gained by an item processed by the downstream subsystem R(s, θ): the immediate revenue incurred during the next unit time when system state is s and the adopted maintenance action is θ S(t): the state of the system at time t V(w): the relative cost function when the system is in state w X um (t): the state of component m in the upstream subsystem at time t X dm (t): the state of component m in the downstream subsystem at time t θ: the maintenance action of the system θ u : the maintenance action of the upstream subsystem θ d : the maintenance action of the downstream subsystem

Problem formulation
The investigated system is illustrated in Figure 1, which contains an upstream subsystem, a downstream subsystem, and an intermediate buffer.Both the upstream and downstream subsystems consist of two identical parallel components.The upstream subsystem delivers products to the buffer, while the downstream subsystem consumes items in the buffer and processes them into final products.The nominal production rates of a component in the upstream and downstream subsystems are q u and q d , respectively.The buffer level at time t is denoted as K(t)=0,1,…,N k , where N k is the buffer capacity.The components in the upstream and downstream subsystems all suffer from degradation, and the degradation processes follow the discrete-time discrete-state Markov process.The states of the components in the upstream and downstream subsystems at time t are X um (t)=1,2,…,N u ,PM and X d,m (t)=1,2,…,N d ,PM(m=1,2), respectively.Here, state one is the faultless state, and state N u (N d ) is the failure state.The additional state PM indicates that the component is under preventive maintenance.The production rate of a component is zero when it fails or is under maintenance; otherwise, the component can work at the nominal production rate q u (q d ).However, the actual production rate of a component also depends on the current buffer level K(t).For example, when the buffer is full, i.e., K(t)=N k , the upstream subsystem is blocked, and the production rate of its two components is zero.On the other hand, when K(t)=0, the downstream subsystem is starved, and the production rate of the downstream subsystem is zero.The calculation of the production rates is discussed in Section 3. In reality, the production load of a component can affect its degradation process.Therefore, this research assumes that the transition matrix of the state of a component is the function of its production rate.The transition matrix of a component in the upstream subsystem is denoted as P u (q),where q is the current production rate of the component.In the same way, the transition matrix of a component in the downstream subsystem is described as P d (q).Two types of maintenance activities are applied to the components, i.e. the preventive maintenance and the corrective maintenance.The duration of the two types of maintenance activities follows the Geometric distribution.The probability of successfully performing the preventive and corrective maintenance to a component in the upstream (downstream) subsystem in one unit time is p pu (p pd ) and p cu (p cd ), respectively.The cost rate of the preventive and corrective maintenance to a component in the upstream (downstream) subsystem is c pu (c pd ) and c cu (c cd ), respectively.In practice, both the state and the production rate of a component affect its operation cost.Therefore, the operating cost of a component in the upstream subsystem is c ou (i,q),where i is the state of the component and the q is the current production rate of the component.This paper assumes that c ou (i, q) is a non-decrease function of i and q.The cost of holding an item in the buffer per unit time is c h , and an item processed by the downstream subsystem can gain a production revenue r p .The objective function used in maintenance optimisation is the expected revenue per unit time, which is given by: where, P R is the average production revenue per unit time; M C , O C , and H C are the average costs per unit time incurred by maintenance, operation, and inventory.

Assumptions
All the components have a non-decreasing degradation rate: • For fixed values of q and j', the quantities The number of products processed by the system is discrete.

•
The downstream subsystem can only process the product in the • buffer; an item cannot be processed by both the upstream and downstream subsystems in the same unit time.

System modelling
The change of buffer level and the degradation of the components are interrelated.Therefore, the state of the system at time t is given by a vector Because the components in a subsystem are assumed to be identical, the state space of the system can be reduced by setting X u1 (t)≥X u2 (t) and X d1 (t)≥X d2 (t).Each system state has an optimal maintenance activity that is given by a vector θ=[θ u θ d ], where θ u and θ d are the maintenance actions of the upstream and downstream subsystems, respectively.The value of θ u (θ d ) is defined as: ( ) 0 no maintenance is performed 1 the component in a better state is preventively maintained 2 the component in a worse state is preventively maintained 3 both the two components are preventively maintain u d θ θ = ed 4 one component is correctively maintained 5 one component is correctively maintained and the other is preventively maintained 6 both the two components are correctively maintained According to the assumption in this paper, the production rate of a component is a function of the maintenance action and the buffer level k.The production rates of the two components in the upstream subsystem are given by: Similarly, the production rates of the two components in the downstream subsystem can be calculated as: min , 2 2 0 , min , 1 0 2,3, 4,5,6 To simplify the formulation, this research assumes that the transition probabilities of a component are linear functions of its production rate q.The elements in the transition matrix P u (q) of a component in the upstream subsystem can be calculated as: Here, u P is the state transition matrix of a component under the nominal production rate q u , and idel u P is the state transition matrix of a component when its production rate is zero.The state transition matrix of a component in the downstream subsystem can be calculated in the same way using the two transition matrices d P and idel d

P
.The operation cost of a component is a function of the component state i and production rate q.The operation cost of a component in the upstream subsystem is assumed as: where, c ou,i is the operation cost of a component in the upstream subsystem when its state is i and its production rate is the nominal production rate q u .Similarly, the operation cost of a component in the downstream subsystem is: Other formulations of transition probabilities and operation costs can be also processed by the maintenance optimisation method developed in this paper.

Markov decision process model
The MDP is a useful tool to identify the optimal maintenance strategy when the optimal strategy structure is unknown [24].Consequently, this research adopts the MDP to investigate the properties of the optimal maintenance strategy for a series-parallel system with an intermediate buffer.
A crucial part of the MDP model is the relative cost function that formulates the relative cost of a single step in the long-run decision process [14].For the investigated maintenance optimisation problem, the relative cost function is given by: Here, s = [x u1 x u2 x d1 x d2 k] is the system state vector, while w = [x' u1 x' u2 x' d1 x' d2 k' ] is another state in the system state space S. The notations g and θ are the average revenue per unit time and the adopted maintenance action, respectively.R(s, θ) is the immediate revenue incurred during the next unit time when the system is in state s and under maintenance action θ.P s,w (θ) is the probability that the system is in state w after one unit time when the current state of the system is s and the adopted maintenance action is θ.The function V(w) is the relative cost function when the system is in state w, which is given by: where, Θ is the maintenance action space.
The immediate revenue R(s, θ) incurred during the next unit time is calculated as: The buffer level after one unit time can be calculated according to the adopted maintenance action and the current buffer level as: The degradation processes of the upstream and downstream subsystems do not depend on each other.Consequently, the transition probability of the system can be simplified as: θ θ P Pr ' , ' , , , Pr ' , ' , , , ' Here, I(A) is the indicator function given by: ( ) Because the derivation processes of the probabilities ( ) Pr ' , ' , , , Pr ' , ' , , , x x k θ are quite similar, only the calculation of ( ) Pr ' , ' , , , When θ u =0, no maintenance activities are applied to the two components.The transition probability is given by: ( ( ) Pr ' , ' , , ,0 ,0 ,0 ,0 ,0 When θ u =1, only the component in a better state is maintained.After one unit time, the component can be still under preventive maintenance or in a brand new state.The transition probability is calculated as: ( ( ) Similarly, when θ u =2, only the component in a worse state is maintained, and the transition probability is as follows: ( ( ) When both the two components are preventively maintained, i.e. θ u =3, the transition probability can be computed as: When θ u =4,a component is failed and correctively maintained.The corresponding transition probability is given by: ( 2 1 When θ u =5, a component is correctively maintained and the other one is under preventive maintenance.The transition probability is calculated as: When both the two components are correctively maintained, i.e. θ u =6, the transition probability can be computed as: After different components in the relative cost function (10) are calculated, the policy iteration modified from that in Ref. [24] is used to find the optimal maintenance strategy.In policy iteration, the relationship between the maintenance action and the system state is formulated as a policy function denoted as δ(s)=θ, where sis the system state and θ is the corresponding maintenance action.The policy iteration is used to obtain the policy function δ * (•) that incurs the largest average revenue per unit time.The process of the policy iteration is shown in Table 1.For a more detailed introduction of the policy iteration, readers can refer to [15,22].

sciENcE aNd tEchNology
In this paper, the system state is the combination of the four component states and the buffer level.Consequently, the size of the transition matrix increases quickly with the number of component states and buffer levels.For example, in the numerical study of this paper, the total number of elements in the transition matrices corresponding to different actions is 771895089.When the MATLAB is used to realise this algorithm, more than 5G memory is required to store the sevariables of type double.Fortunately, the total number of nonzero elements in these matrices is only 1121481, which requires just about 8M memory on the MATLAB platform.Another problem is that a system of linear equations that contains a large number of equations should be solved duration the policy iteration in this paper, which is computationally expensive.Fortunately, the coefficient matrix of the system of linear equations is also sparse.Subsequently, the generalized minimum residual (GMRES)method [3]that can process the large and sparse coefficient matrix efficiently is adopted.The incomplete LU factorization [18] is used to provide preconditioners for the GMRES method.The simulation study shows that the developed approach can identify the optimal maintenance strategy efficiently.

Numerical study
A numerical study is conducted to evaluate the performance of the proposed maintenance optimisation algorithm.Furthermore, the obtained optimal maintenance strategy is investigated to analyse the properties of the optimal strategy structure.Finally, the influence of system parameters on the result of maintenance optimisation is studied through sensitivity analysis.This numerical study is executed using MATLAB 7.14 on a desktop computer with an Intel i7 3770 CPU and eight Gigabytes of RAM.

System introduction
The structure of the system investigated in this numerical study is shown in Figure 1.The transition matrices of a component in the upstream subsystem when it is under nominal production rate and is idle are: 0.5 0.2 0.15 0.1 0.05 0 0.4 0.3 0.2 0.1 0 0 0.5 0.3 0.2 0 0 0 0.6 0.4 0 0 0 0 1 On the other hand, the transition matrices of a component in the downstream subsystem are: 0.6 0.18 0.1 0.07 0.05 0 0.5 0.3 0.15 0.05 0 0 0.6 0.2 0.2 0 0 0 0.5 0.5 0 0 0 0 1 Table 1.The process of policy iteration Step 1: An initial policy function δ 0 (s) is selected by the rule of thumb, and any maintenance policy that satisfies the assumptions of this research can be adopted as the initial policy.
Step 2: Obtain the relative costs s s S and the expected revenue per unit time g by solving the following system of linear equations that is constructed according to the current maintenance policy function δ k (s): The size of the state space S is (N u +1)(N u +2)/2×(N d +1)(N d +2)/2×(N k +1), which is also the number of equations in the system of linear equations.It is assumed that the relative cost function is zero when the system is brand new and the buffer in empty, i.e., V([x u1 , x u2 , x d1 , x d2 , k])=0 if x u1 =x u2 =x d1 =x d2 =1 and k=0.
Step 3: Calculate the relative costs under different maintenance actions using the relative costs ( ) { } 0.9 0.04 0.03 0.02 0.01 0 0.93 0.03 0.02 0.02 0 0 0.93 0.04 0.03 0 0 0 0.96 0.04 0 0 0 0 1 The probabilities of successful preventive and corrective maintenance in a unit time are assumed as:p pu =0.8,p pd =0.7,p cu =0.2, andp cd =0.15.The costs of maintenance activities are c pu =50, c pd =60, c cu =100, and c cd =110.The nominal production rates of a component in the upstream and downstream subsystems are q u =3 and q d =2, respectively.The operation costs of a component in different states under the nominal production rate are listedin Table 2.The buffer capacity is N k =8, and the cost of holding an item in the buffer for a unit time is c h =1.
The production revenue brought about by a product processed by the downstream subsystem is r p =150.

Maintenance optimisation and results analysis
Before the policy iteration, the transition matrices of the system under different maintenance actions and the immediate revenue should be calculated.There are 49 transition matrices of size 3969×3969 according to different maintenance actions, while the number of immediate revenue values under different combinations of system states and maintenance actions is 3969×49=194481.The computing time of the transition matrices and the immediate revenue values is about 47 second.Then, the policy iteration is performed and the optimal revenue is finally obtained as g*=200.929after five iterations, which takes 40 seconds.During the policy iteration, the GMRES method is used to solve the system of linear equations that contains 3969 equations, which takes only about eight seconds during each iteration.Some parts of the obtained maintenance strategies are demonstrated in Tables 3 to 6.The notation x u (x d ) is the state vector of the upstream (downstream) subsystem before the possible maintenance action, while x + u (x + d ) is the state vector after the beginning of the maintenance action.Here, the state of the subsystems after the beginning of the maintenance action is introduced to simplify the demonstration of the maintenance actions.For example, if the maintenance action is a d =2, the two states x d =[3,1] and [4,1] can be combined to a single state x + d =[PM,1].Some conclusions can be drawn from the result of this particular maintenance optimisation problem: The optimal maintenance activity of a subsystem is affected by 1.
the buffer level.Here, the situation that x + d =[1 1] and x u =[22] is considered as an example.As shown in Table 3, when the buffer is empty, preventive maintenance activity on the upstream subsystem is not required.Conversely, Table 5 shows that both the two components in the upstream subsystem should be preventively maintained when the buffer is full.The optimal maintenance activity of a subsystem depends on 2.
the state of the other subsystem.However, this dependence is not straightforward to explain.E.g. as shown in Table 3, when the downstream subsystem is in state ] and the upstream subsystem is in state x u =[3 1], both the two components in the upstream subsystem are not maintained.On the other hand, when the downstream subsystem is in the other states and x u =[3 1], the component in state 3 in the upstream subsystem will be preventively maintained.Therefore, it cannot be concluded that a worse health state of the downstream Table 3.The optimal maintenance actions of the upstream subsystem when k=0 sciENcE aNd tEchNology subsystem requires a more conservative or speculative maintenance strategy of the upstream subsystem.The state of the downstream subsystem has a more significant 3.
effect on the maintenance strategy of the upstream subsystem.On the other hand, according to the result of this maintenance optimisation, when the buffer level is four and eight, the opti-mal maintenance activity of the downstream subsystem does not depend on the state of the upstream subsystem.
The maintenance strategy of a component depends on the state 4.
of the other component in the same subsystem.As shown in Table 4, when x + d =[1 1] and a component in the upstream subsystem is in state one, the preventive maintenance threshold of the other component is state three.Conversely, when The highest optimal average revenue value in each row is marked with a grey background colour sciENcE aNd tEchNology x + d =[1 1] and a component in the upstream subsystem is in state four, the preventive maintenance threshold of the other component becomes state four.In this particular maintenance optimisation problem, the con-5.
trol-limit structure is always optimal for a component when the buffer level and the states of the other three components are fixed.In other words, there is always an optimal preventive maintenance threshold for a component when the states of the other components and the buffer level are determined.
In this simulation study, the maintenance activities 6.
a u =1 or a d =1 are not the optimal for all the system states.Therefore, if only one component is to be preventively maintained, it will not be the one in a better state.Conclusions one to four show that the optimal maintenance strategy cannot be described by a small number of parameters.Consequently, the MDP is an appropriate approach to optimising the maintenance strategy.Conclusions five and six can be used to simplify the MDP model.The prerequisites of conclusions five and six will be further discussed in another paper.When the number of system states is intractably large, some factors can be ignored during the maintenance optimisation to obtain an approximate optimal strategy.For example, the maintenance activity of a component can be assumed to be independent from the other in the same subsystem.

The influence of the buffer capacity on the optimal average revenue
A large capacity of the intermediate buffer can reduce the influence of the failure and maintenance of the upstream subsystem, and can thus enhance the system production rate.However, holding products in a buffer also incurs cost in some practical applications.Under these situations, the large capacity of the buffer can increase the holding cost of work in process.Therefore, it is necessary to investigate the relationship between the optimal average revenue per unit time and the buffer capacity.This section changes the production revenue per item r p , the storage cost rate per item c h , and the buffer capacity N k ; other system parameters are the same as those used in Section 4.2.The optimal average revenue g*under different system parameters is demonstrated in Table 7.
Table 7 shows that a large buffer capacity does not necessarily bring about high average revenue per unit time.When the storage cost rate c h is not ignorable compared with the production revenue per item r p , there exists an optimal buffer capacity that can result in the largest optimal average revenue per unit time.The optimal buffer capacity decreases with the storage cost rate per item and increases with the production revenue per item.As shown in Table 7, even when the storage cost rate per item c h is only 1 and the production revenue per item r p is as large as 100, the buffer capacity larger than 14 can still bring down the optimal average revenue.This counterintuitive phenomenon indicates that optimising the buffer capacity is necessary, when the holding cost of products in the buffer is not ignorable.An alternative approach to controlling the buffer level is scheduling the production according to the system state.

Sensitivity analysis of parameters of maintenance activities
The parameters of maintenance activities of the upstream and downstream subsystems can have different effects on the optimal average revenue.This section modifies the system parameters used in section 4.2.The parameters of the maintenance activities are assumed as p pu = p pd =0.7, p cu = p cd =0.175, c pu = c pd =50, and c cu = c cd =100, while the other parameters remain the same as those in section 4.2.Firstly, the sensitivity analysis about the maintenance cost rate is performed.The cost rates of preventive and corrective maintenance activities are assumed to follow the relationship c cu =2c pu and c cd =2c pd .The average revenue values under different c pu and c pd are plotted in Figure 2. Similar sensitivity analysis is conducted to the probability of successful maintenance activity.The average revenue values under different p pu and p pd are displayed in Figure 3, where p cu =0.25p pu and p cd =0.25p pd .The two figures show that the average revenue is more sensitive to the parameters of the downstream subsystem.

Conclusion
This paper formulates the degradation process of a series-parallel system with an intermediate buffer using the MDP model.The policy iteration is used to identify the optimal maintenance strategy.The GMRES method is used to solve the system of linear equations with a sparse coefficient matrix during the policy iteration to enhance the efficiency of the algorithm.A numerical study is conducted to evaluate the performance of the developed method.The result shows that the developed method can identify the optimal maintenance strategy efficiently.The numerical study also shows that the optimal maintenance activity of a component depends on the state of the other three components and the buffer level.Therefore, the optimal maintenance strategy structure cannot be modelled accurately with a small number of parameters, and the MDP is an appropriate tool to identify the optimal maintenance strategy.However, this research finds strong numeric evidence of several useful properties of the optimal maintenance strategy structure.These properties can simplify the maintenance optimisation process and provide reference to developing cost-effective maintenance strategy that is easy to implement in practice.Another interesting finding is that increasing the buffer capacity does not always enhance the average revenue when the cost of holding products in the buffer is not ignorable.Consequently, the buffer capacity should be optimised, or the production rates of the components should be controlled according to the system state.Finally, this research also finds that changing the repair rates and maintenance cost rates of the upstream and downstream subsystems can have different effects on the optimal average revenue.The outcome of this research is expected to provide foundation for more efficient maintenance optimisation methods for series-parallel systems with intermediate buffers.
of a component in the downstream subsystem under the nominal production rate q u idel d P : decreasing in i.

Fig. 2 .Fig. 3 .
Fig. 2. The average revenue per unit time under different maintenance cost rates 1

Table 2 .
operation costs of a component in different states under the nomi- The character N means that the component is not under maintenance; P denotes preventive maintenance; C standards for corrective maintenance; D is the breakdown state.The first character is the maintenance action or the state of the first component, while the second one is that of the second component.When the optimal maintenance action of a subsystem depends on the state of the other subsystem, the background of the row is grey.

Table 4 .
The optimal maintenance actions of the upstream subsystem when k=4

Table 6 .
The optimal maintenance actions of the downstream subsystem when k=0

Table 7 .
The optimal average revenue g*under different system parameters