An optimal maintenance policy for machine replacement problem using dynamic programming

Article history: Received: October 1, 2016 Received in revised format: November 16, 2016 Accepted: March 7, 2017 Available online: March 7, 2017 In this article, we present an acceptance sampling plan for machine replacement problem based on the backward dynamic programming model. Discount dynamic programming is used to solve a two-state machine replacement problem. We plan to design a model for maintenance by considering the quality of the item produced. The purpose of the proposed model is to determine the optimal threshold policy for maintenance in a finite time horizon. We create a decision tree based on a sequential sampling including renew, repair and do nothing and wish to achieve an optimal threshold for making decisions including renew, repair and continue the production in order to minimize the expected cost. Results show that the optimal policy is sensitive to the data, for the probability of defective machines and parameters defined in the model. This can be clearly demonstrated by a sensitivity analysis technique. © 2017 Growing Science Ltd. All rights reserved.


Introduction
In the past, manufacturing industries have extensively used maintenance policy under certain conditions.Nowadays, industries have moved towards policies associated with risk and uncertainty due to lack of sufficient information and economic debates in making appropriate maintenance decisions.Nowadays, maintenance problem has been converted to an issue to determine optimal maintenance policy, which has become a very complex problem.There are many literature reviews for maintenance topic; some of them are placed under certain category (Tersine, 1985;Labib, 1998;Fernandez et al., 2003;Khalil et al., 2005;Burhanuddin et al., 2007).For example, Labib (2004) proposed a decision analysis model for maintenance policy selection using a computerized maintenance management systems.Gupta et al. (2009) presented a novel graphical model using control chart to carry out the analysis on optimal maintenance.Tahir et al. (2009) proposed an improved decision making grid model for maintenance management, which uses tri-quadrant technique to cluster the data mining group.There is another category called risk mostly modeled based on mathematical techniques for determining the optimal maintenance policy.For example, De Almeida and Bohoris (1995), Lugtigheid et al. (2004), Samrout et al. (2009), Martorell et al. (1999), Gurler and Kaya (2002), Amari et al. (2006) used mathematical models to illustrate an approach to determine the optimal maintenance policy for decision making.Machine replacement problem has been studied in many researches and researchers presented different approaches to determine an optimal maintenance policy.
As mentioned earlier, maintenance policy is selected based on the terms of certainty, risk or uncertainty categories and it is necessary to consider it according to levels of information and economical debates.Maintenance in manufacturing industries is necessary because it plays an important role for decreasing expected cost and making profit in companies.Nowadays, the purpose of manufacturing industries is to reach higher operation efficiency, effectively, and to survive in the extremely competitive global economy.Basically, maintenance can be illustrated as a combination of all technical and administrative affairs including supervision, affairs intended to retain or restore the system into a state in which system can perform a required function (Swanson, 2001).However, it is necessary to underline that maintenance cost can even achieve 15 to 70% of the expenditure or even it could increase annual net profit in many instances (Madu, 2000;Mobley, 2002).Thus a suitable and optimal maintenance policy is necessary for maintenance management in fulfilling all maintenance activities to save a remarkable money (Tan & Raghavan, 2007;Lu & Sy, 2009).In addition, optimal maintenance policy can propose a targeted plan of action that usually contain a set of rules used to provide guidance for maintenance management in conducting an effective maintenance (Gupta et al., 2009;Xia et al., 2011;El-Gohary, 2004).
Over the years, researches is conducted to propose the model that can create a methodology for determining the most effective-cost components and the best policy for maintenance.For example, Ivy and Nembhard (2005) presented maintenance model for machine replacement problem under uncertainty conditions, by combining the statistical quality control technique and partially observable Markov decision processes.Kuo (2006) proposed a dynamic programming technique for integrating machine maintenance and items quality control problem in finite time horizon.
In this paper, we present a model for two-state machine replacement problem based on dynamic programming.To achieve this goal, we use sequential sampling plan and partially observable Markov decision process (POMDP) method.
The second section provides preliminaries from sequential sampling plan and presents the description of the model and the assumptions are given in the third section.The fourth section illustrates a numerical example to show the implementation of the proposed model.The results of model is presented in the fifth section and the sensitivity analysis technique is presented to determine the effect of changing any parameters on the results.Conclusion comes in the section six.

Sequential sampling plan
Acceptance sampling plan is an efficient technique for factory managers and it is important because of two reasons: 1.The quality level of the batch improves after rectifying inspection.2. Reverse relevance between quality level and average number of inspected items (Montgomery, 2007).Acceptance sampling is a useful tool in statistical quality control, which depends on the number of defective items.By obtaining the number of defective items, we may decide to reject and accept batch or continue sampling (Montgomery, 2007).
The plan is used when: 1. 100% inspecting is not possible or the related cost is high.2. The inspection has some destructive experiments.3. The inspection of all items takes a long time (Montgomery, 2007).
Sequential sampling plans are different from the other sampling techniques such as simple sampling.In simple sampling plan, the size of the sample is fixed and by inspecting the sample, it can be easily decided to reject or accept the lot; while in sequential sampling, the size of the sample is not fixed and it depends on the defective items.The plan may be continued to 100% inspecting.This is a novel acceptance sampling to optimize related costs and to reduce the number of inspected samples; in this technique, priority is to minimize the sample size (Nezhad et al., 2015).Sequential sampling plan is performed item by item and is designed to reject or accept and continue sampling according to the cumulative observed number of defectives.There is a diagram, where the x-axis shows the total number of items, and the y-axis represents the total number of observed defectives.The diagram includes accept and reject lines; where the equations are illustrated as following (Montgomery, 2007): where (1 ) log( ) ( 1) If the point of the total number of observed defectives is placed upper than the reject line, then batch is rejected.Also, if the point is placed lower than the accept line, the batch is accepted; if the point is placed between two lines then the inspection will continue until the point is placed higher than the reject line or lower than accept line until inspection is stopped.The diagram is shown in the Fig. 1.

Description of the model
In this section, we aim to describe and illustrate a model for two-state one-machine replacement problem using dynamic programming.Assumptions and formulas of the model are stated using Bayesian inference method.Then it is tried to create and implement a framework for model proposed by sequential sampling plan and partially observable Markov decision process.
The POMDP technique is applied to achieve from the partially observations to machine state; in other words, we want to gain machine state by checking partial observations; then sequential sampling plan is performed so that the machine starts to work and produces N items; by considering the framework presented, K observation is investigated; first the item is checked, then checking is continued item by item.In this article, it has been tried to present a model for two-state machine replacement problem using a dynamic programming.In fact, we aim to present a decision making model for maintenance and repairs under uncertain conditions.
It is supposed that the machine can be placed in two states of good and bad; the probability of being defective for machines depends on the amount of item produced.In fact, we are dealing with a decision tree and optimal decision is selected based on the produced parts quality.This means that if the number of nonconforming parts exceeds from the pre-defined threshold, the machine is in bad state and if it is less than this threshold, the machine is in good state.
In this model, using the statistical quality control techniques and partially observable Markov decision processes (POMDP), it is proposed to determine the probability of producing a defective product based on the machine state; if the machine is in good state, the defective observation distribution follows Bernoulli distribution with parameter p1 and if the machine is in bad state, the defective observation distribution follows Bernoulli distribution with parameter p2.Then, using the simulation technique, it has been tried to implement the proposed model in the practical cases.
To illustrate the model, some assumptions must be considered; it has been assumed that the machine can be placed in two states: good and bad.The backward dynamic programming is used; π (the probability that the machine is in bad state) is considered as state variable.Also the number of the programming periods is stage variable; programming is performed in finite time horizon.The statistical quality control is used to determine the probability distribution of defective items.This assumption has been shown in Fig. 2.

Fig. 2. Determining the parameters of the Bernoulli distribution for producing one item
Parameters and formulas of the model are illustrated in following; some of them are obtained using the Bayesian inference method.
π: Probability that the machine is in bad state; {st=0}.
1-π: Probability that the machine is in good state; {st=1}.α: Discount factor; α[0,1] p1: Probability that the observation is defective if the machine is in the good state.
p2: Probability that the observation is defective if the machine is in the bad state.
z: Probability that the observation is defective.
L: The defective observation.
pr1: The posterior probability of machine state when a nonconforming item is produced.
pr2: The posterior probability of machine state when a conforming item is produced.
Π0: Probability of defective observation for new machine.
λ: The improvement rate after repair decision.
n: The number of remained stages (the stage variable).
T: The coefficient for the cost of repair decision.

R: The fixed cost for renew decision
A: Profit of a conforming item.
C: Cost of one nonconforming item.
V0(π): The salvage value of the machine (the value of the machine when no stage is remaining and the process terminates).
The optimality equation is illustrated as following: A combination of the condition-based maintenance (CBM) and sequential sampling plan is used to illustrate the model proposed.In this article, CBM is used when the point is placed to continue sampling area and can be chosen for the decisions of repairing the machine or can be continued sampling until the point is placed in rejected area and the renew decision is chosen; if the point is placed in accept area the decisions of the Do nothing is performed.Fig. 3 clearly shows sequential sampling method in machine replacement problem and a decision tree is designed which shows the details of each decision (Fig. 4).
 A: Renew machine.
 B: Repair machine and continue production.
 C: Continue the production without any maintenance action.
Fig. 4 shows that if the renew decision is chosen, then the probability of producing defective item for new machine is equal to π0; if the repair decision is chosen, the probability of producing a defective item reaches to λπ; where λ is less than one; and if the decision of doing nothing and continuing is chosen then the machine continues production; first the machine produces one item; according to sequential sampling approach if the produced item is non-conforming, then the machine state is equal to pr1 and if the produced item is conforming then the machine state is equal to pr2.

Numerical example
In this section, a numerical example is solved for illustrating the application of proposed methodology.Input data of the problem is as follows: Assumptions and equations used in this model are simulated by MATLAB software and analyzing the model and the results are performed by this software.For example, if n=9; decision making stages are available then the results for different values of state variable π are reported in Table 1.It is seen that the optimal policy is a control threshold policy.When 0 ≤ π ≤ 0.35 , then optimal decision is to do nothing and continue; when 0.35 ≤ π ≤ 0.85 , then the optimal decision is to repair and when Π ≥ 0.85 , then the optimal decision is to renew and replace the machine (Fig. 5).Thus we can define two thresholds as π 1 * and π 2 * so that if π ≤ π 1 * then the optimal decision is to continue when π1 * ≤ π ≤ π 2 * then optimal decision is to repair and when π ≥ π 2 * then optimal decision is to renew.

Sensitivity Analysis
A sensitivity analysis is applied to analyze the effects of changing parameters on the optimal solution.
In each case of sensitivity analysis, one parameter of the model is varied.It is necessary to adjust the parameter value in a level so that one can easily interpret its behavior.Table 2 shows the results.

Table 2
The results of sensitivity analysis for the proposed sampling plan According to Table 2, it is seen the regular and irregular patterns with increasing or decreasing values of parameters; in other words, regular patterns include P1, P2, π0, R, C, A, M and irregular patterns include λ, T. For example, values of π1 * , π2 * increase and limit of the repair decision decreases by increasing P1.Also values of π1 * , π2 * reach to 0 and 1by increasing of A; it means that the repair decision is always chosen.Details of sensitive analyze technique are illustrated in Fig. 6.

Conclusion
In this article, we have presented a dynamic programming model in a finite time horizon to determine a control threshold policy using POMDP technique and sequential sampling plan for two-state machine replacement problem in the different stages.Using dynamic programming technique and sequential sampling plan by considering decision tree (including renew, repair and continue the production) is a novel and innovative method for modeling machine replacement problem.This model has been applied for optimizing expected cost in machine replacement problem based on the methods of sequential sampling and Bayesian inferences.A decision tree has been implemented to determine which decision can be chosen; if each decision is chosen the related cost is applied.A cost objective function including the costs of replacement and repair and the cost of defectives is considered.The presented model can be used in the production department that machine deterioration is investigated using the quality of produced items.

Fig. 5 .
Fig. 5. Diagram of the expected costs for each π

Table 1
Total expected costs for each Π and its optimal decision