Production, Manufacturing and Logistics
A hybrid wrapper–filter approach to detect the source(s) of out-of-control signals in multivariate manufacturing process

https://doi.org/10.1016/j.ejor.2014.02.032Get rights and content

Highlights

  • A novel fault diagnosis approach with less computational complexity has been proposed.

  • A hybrid approach has been proposed that combine the advantages of both filter and wrapper approaches.

  • Proposed heuristic score based subset generation process reduces the search space into polynomial growth.

Abstract

With modern data-acquisition equipment and on-line computers used during production, it is now common to monitor several correlated quality characteristics simultaneously in multivariate processes. Multivariate control charts (MCC) are important tools for monitoring multivariate processes. One difficulty encountered with multivariate control charts is the identification of the variable or group of variables that cause an out-of-control signal. Expert knowledge either in combination with wrapper-based supervised classifier or a pre-filter with wrapper are the standard approaches to detect the sources of out-of-control signal. However gathering expert knowledge in source identification is costly and may introduce human error. Individual univariate control charts (UCC) and decomposition of T2 statistics are also used in many cases simultaneously to identify the sources, but these either ignore the correlations between the sources or may take more time with the increase of dimensions. The aim of this paper is to develop a source identification approach that does not need any expert-knowledge and can detect out-of-control signal in less computational complexity. We propose, a hybrid wrapper–filter based source identification approach that hybridizes a Mutual Information (MI) based Maximum Relevance (MR) filter ranking heuristic with an Artificial Neural Network (ANN) based wrapper. The Artificial Neural Network Input Gain Measurement Approximation (ANNIGMA) has been combined with MR (MR-ANNIGMA) to utilize the knowledge about the intrinsic pattern of the quality characteristics computed by the filter for directing the wrapper search process. To compute optimal ANNIGMA score, we also propose a Global MR-ANNIGMA using non-functional relationship between variables which is independent of the derivative of the objective function and has a potential to overcome the local optimization problem of ANN training. The novelty of the proposed approaches is that they combine the advantages of both filter and wrapper approaches and do not require any expert knowledge about the sources of the out-of-control signals. Heuristic score based subset generation process also reduces the search space into polynomial growth which in turns reduces computational time. The proposed approaches were tested by exhaustive experiments using both simulated and real manufacturing data and compared to existing methods including independent filter, wrapper and Multivariate EWMA (MEWMA) methods. The results indicate that the proposed approaches can identify the sources of out-of-control signals more accurately than existing approaches.

Introduction

Multivariate process control techniques were established by Hotelling in his paper Hotelling (1947). He introduced the problem of correlation between the quality characteristics of a process and devised the well-known T2 statistic to identify whether the whole process is out of control. Hotelling’s T2 statistic is the optimal test statistic for detecting a general shift in the process mean vector for an individual multivariate observation. However, the technique has several practical drawbacks. Critically, when the T2 statistic indicates that a process is out of control, it does not provide information on which variable or set of variables is out of control. Moreover, it is difficult to distinguish location shifts from scale shifts since the T2 statistic is sensitive to both types of process changes.

The difficulty of interpreting an out-of-control signal on a multivariate control chart has been discussed extensively Alt, 1985, Doganaksoy et al., 1991, Murphy, 1987, Pignatiello and Runger, 1990, Lowry et al., 1992, Linderman et al., 2005 among others. When two or more correlated variables are monitored, use of a multivariate chart may cause signals at opposing times to the signals given by a set of univariate charts on the individual variables. This is because the control region for a multivariate chart on correlated variables is represented by a tilted elliptical region as opposed to the non-tilted square region obtained by the use of separate charts. In fact, the use of separate charts does not allow for the information concerning the correlation of the variables to be utilized. However, the combination of using a multivariate control chart for signalling purposes and then using separate charts for diagnostic purposes is often effective.

Given an m-dimensional quality characteristics set of data, a fault detection and diagnosis subsystem in a multivariate process needs to find the optimal sources of fault for the out-of-control signals from the 2m subsets of quality characteristics which is computationally expensive and exponential to the dimensions of characteristics set (Jackson & Morris, 1957). Moreover, the performance of the algorithms depends on its evaluation criterion and search strategies.

Murphy (1987) proposed a method to identify the out-of-control variables based on discriminant analysis. He divided the complete set of variables into two subsets and then tried to determine which one caused an out-of-control signal. Alt (1985) proposed the use of the univariate t-statistic for ranking the variables most likely to have changed. Then, to further strengthen the belief that a certain variable has changed, they applied the Bonferroni-type interval. The obvious drawback of this method is that it only tells you which variable is most likely to have shifted, which is not conclusive. Also, this method does not allow the user to study the trends.

Mason, Tracy, and Young (1995) showed that signal interpretation of the T2 statistic is greatly aided if the corresponding value is partitioned into independent parts. The characteristic that is significantly contributing to the signal is more readily identified by decomposing the T2 statistic into independent parts, each of which reflects the contribution of an individual variable. The drawback of this method is the extensive computation and its sensitivity to the number of variables. This approach reduces the search space from exponential to O(m2). However, for a large number of quality characteristics, search space still would be a cumulative factor for computational time.

Many approaches have been suggested for identifying the variable or group of variables that causes the out-of-control signals. They can be grouped broadly into three main categories: (1) the wrapper model with expert knowledge (Chen and Wang, 2004, Chen and Wang, 2004, Francisco and Jos, 2010, Jianbo et al., 2009, Low et al., 2003, Zorriassatinea et al., 2003) (2) filter models with wrapper evaluation (Rodger, 2012, Sylvain et al., 2008, Wang and Dub, 2000), and (3) statistical approaches (Francisco and Jos, 2010, Mason et al., 1995).

Wrapper approach is one of the main approaches for fault diagnosis that uses a pre-determined induction algorithm where the performances of the algorithms are used as the evaluation criteria, see for example (Chen and Wang, 2004, Chen and Wang, 2004, Deborah et al., 2006, Francisco and Jos, 2010, Jianbo et al., 2009, Lorton et al., 2013, Low et al., 2003, Wang, 2012, Zorriassatinea et al., 2003). The wrapper models use expert knowledge to build training pattern which is capable of identifying the sources of future out-of-control signals. The main disadvantage of these approaches is that they need expert inputs about the sources of out-of-control signals which may introduce human error. In addition, the wrapper models face huge computational overhead due to the use of the induction algorithm’s performance and subset generation process used in the wrapper search space. None of the above approaches use any heuristic score in subset generation process to reduce the search space for fault diagnosis in the multivariate processes.

Outside the multivariate process control domain, wrapper approaches have also been used in many classification problems. Let us refer to few recent articles by Kohavi and John, 1997, Puronnen et al., 2000, Huang et al., 2008, Hsu et al., 1999, Romero and Sopena, 2008. In particular, Hsu et al. (1999) proposed a wrapper based heuristic in artificial neural network (ANN) known as Artificial Neural Network Input Gain Measurement Approximation (ANNIGMA) for a general classification problem. Hsu et al. (1999) demonstrated that a wrapper-based heuristic can improve search performance in a standard training environment of an ANN. However, the approach (Hsu et al., 1999) also needs expert-knowledge for ANN training and ignores the problem of standard training in ANN. The other approaches use different search strategies for subset generation based-on wrapper performance function which involves numerous call of wrapper re-training in a supervised experimental environment.

Filter models are one of the computationally cheap models which have broadly been considered for many fault diagnosis problems, for example by Sylvain et al., 2008, Wang and Dub, 2000, Jackson and Morris, 1957. In contrast to wrapper models, filter models involve the implementation of a search algorithm by using different heuristics to the data set prior to its use for induction algorithm. Diverse heuristics have been reported in the literature, for example, Principal Component Analysis (PCA) by Jackson and Morris (1957) and Mutual Information by Wang and Dub (2000). Heuristics of filter models estimate the discrimination capabilities of the subsets of the quality characteristics which is followed by the ranking of the source characteristics based on the different search strategies and wrapper classifier. Filter approaches have also been applied in many other problems, for example, bankruptcy prediction, medical diagnosis Gene Selection, malware detection. Let us refer to few recent articles by Santos et al., 2010, Santos et al., 2012, Guo and Lyu, 2006, Tsai, 2009, Krier et al., 2006, Ng and Chan, 2005. In particular, Tsai (2009) provide a review of filter approaches and shows their comparative performances. Because filter models are independent from the induction algorithm, the selected quality characteristics subsets may result in poor fault diagnosis prediction accuracies. Principal components (Jackson & Morris, 1957) are not easily interpretable in many cases and do not have a one-to-one relation with the original variables. Nevertheless, in some cases, depending on the context, they can be very useful.

Therefore there is a need for development of a procedure that can avoid the drawbacks of both filter and wrapper fault diagnosis approaches and can reduce the search space for better computational complexity for the detection of the source(s) of out-of-control signals. In this paper, we propose a hybrid wrapper–filter approach by injecting the filter’s ranking score in the wrapper approach. The objective is to find a suitable heuristics score for the wrapper stage that can improve the search space complexity and can find most significant compact set of the sources of out-of-control signals without any additional information other than control list for a multivariate quality control environment. We introduce a new heuristic score by hybridizing Mutual Information (MI) based Maximum Relevance (MR) filter ranking heuristics with the Artificial Neural Network Input Gain Measurement Approximation (ANNIGMA) wrapper heuristic (MR-ANNIGMA). The justification of hybridizing of the wrapper and filter score is provided later in Section 5.

The novelty of our proposed approach is that it uses of a hybrid of wrapper and filter that utilizes the knowledge about the intrinsic pattern estimated by the filter in the wrapper search process and takes advantages of the complementary properties of both approaches. This type of hybrid approach is a new concept and to the best of our knowledge it has not been explored in the process control literature. The approach does not require any expert knowledge for detecting the out-of-control signals and is able to reduce the search space complexity by generating the subset of sources based on the hybrid score in the wrapper search.

The standard training approaches of ANN may provide locally optimized estimation for the network parameters (Marco & Alberto, 1992) which may affect the wrapper score. An additional objective of the paper is to determine whether improvement in the training problem of ANN can find better ANNIGMA score and can improve overall performance in a fault-detection process of multivariate control environment. Different heuristic approaches were applied for training problem of neural network including evolutionary approaches (Garro, Sossa, & Vzquez, 2011) and swarm optimization (Gudise & Venayagamoorthy, 2003) approaches. Considering the computational complexity of evolutionary approach and wrapper training, we propose a derivative-free global optimization technique to estimate the optimal ANNIGMA score which is later hybridized with the filter score in the MR-ANNIGMA.

This paper also deploys the MEWMA chart to monitor the multivariate data and compare the performance with the proposed approaches. The out-of-control signals are then investigated using univariate EWMA charts. Their performance is then compared with the proposed hybrid approaches in identifying the most significant variables for out-of-control signals.

The rest of the paper is organized as follows. The next two section describe the theory of the univariate and multivariate EWMA chart and their applications in multivariate control processes. Section 4 describes standard filter and wrapper approaches for multivariate processes and Artificial Neural Network Input Gain Measurement Approximation (ANNIGMA). The proposed hybrid wrapper–filter source identification algorithm for out-of-control signals based on Maximum Relevance (MR) filter heuristics and ANNIGMA wrapper heuristic (MR-ANNIGMA) is described in Section 5. Section 6 presents a Global Optimization technique for optimal estimation of ANNIGMA and its use in the proposed hybrid approach. Experimental results and discussion are described in Sections 6 Global optimization for MR-ANNIGMA, 8 Computational performances and search space complexity. Conclusions of this study are presented in the last section.

Section snippets

Univariate and multivariate Exponentially Weighted Moving Average (EWMA) charts

This chart was introduced by Worthham and Ringer (1971). The chart is effective for detecting small to moderate shifts and is very insensitive to normality assumptions; therefore it is an ideal chart for individual observations. It is proposed for applications such as chemical industries, financial and management control systems, particularly when the sample size is one. The chart can be used to control: an average (of a sample), an individual, a ratio or a proportion (of defects in the

Multivariate Exponentially Weighted Moving Average control chart (MEWMA)

To simultaneously monitor m correlated quality characteristics, the multivariate EWMA (MEWMA) chart can be applied. Ignoring correlation between variables is the main weakness of the present practice of using independent univariate charts to track each of the m quality characteristics individually. The MEWMA introduced by Lowry et al. (1992) is a logical extension of the univariate EWMA. It is defined as follows:Zi=λXi+(1-λ)Zi-1=0where λ=diag(λ1,λ2,λ3,,λm) and 0<λ1 and 1 is the identity

Filter and wrapper approaches for multivariate processes

Filter approaches (Sylvain et al., 2008, Wang and Dub, 2000) use the MEWMA statistics and the quality characteristics as the training data. A subset generation process with empty or full quality characteristics set is used to generate the subsets. This is then followed by forward, backward or bi-directional searches. The generated subsets are evaluated using filter heuristics such as co-relation measure, Principal Component Analysis and Mutual Information (Sylvain et al., 2008, Wang and Dub,

Justification of hybridization

Filter approaches can extract knowledge of the intrinsic pattern from real quality characteristics data. However filter approaches do not use any performance criteria based on predictive accuracies. Subsequently, there is no guarantee that the final subset of quality characteristics makes a better prediction and would be the most informative subset for the out-of-control signal. In contrast, the wrapper approaches (Chen and Wang, 2004, Chen and Wang, 2004, Francisco and Jos, 2010, Jianbo et

Global optimization for MR-ANNIGMA

The standard back propagation (BP) training algorithm for estimation of ANN parameters is a gradient-descent based method. One of the drawbacks of this algorithm is that it may get trapped in a local minimum (Marco & Alberto, 1992). Locally optimized network parameters can provide non-optimal values for ANNIGMA wrapper score which may affect MR-ANNIGMA combined score. Different approaches have been proposed for optimal estimation of ANN parameters including Genetic Algorithm (Dongkyu, Shingo,

Experimental results and comparison

The proposed hybrids MR-ANNIGMA and Global MR-ANNIGMA are tested using manufacturing data (Wang & Hubele, 1999). The real data set (Wang & Hubele, 1999) is from a manufacturing process with multivariate quality characteristics. It contains a sample of 100 parts that were tested on seven quality characteristics of interest to the manufacturer. The specification limits for these seven quality characteristics (q1,q2,q3,q4,q5,q6,q7) can be two-sided or one-sided, and they are 0.100.04mm,0+0.50mm,115

Computational performances and search space complexity

The hybrid algorithms run a backward elimination (BE) process where each iteration involves the computational time in training the network, the computation of MR, ANNIGMA and hybrid scores. At the beginning, when all characteristics are used, the time for training and computing scores (MR, ANNIGMA, and hybrid) will be the highest. Subsequent computation for aforementioned scores will take less time. Considering the initial computational cost as a constant-maximum value including the cost for

Conclusions

Traditional methods of detecting the sources of out-of-control signals for multivariate data sets use expert knowledge either in combination with wrapper-based supervised classifier (or a pre-filter filter with wrapper) or the simultaneous use of the individual univariate control charts (UCC). However gathering expert knowledge in source identification is not only computationally costly but may introduce human error. This may also pose exponential growth of search space in the worst case. This

References (51)

  • Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In Petrov, B. N., & Cski,...
  • F.B. Alt

    Multivariate quality control: Encyclopedia of statistical science

    (1985)
  • C.F. Deborah et al.

    Environmental statistical process control using an augmented neural network classification approach

    European Journal of Operational Research

    (2006)
  • N. Doganaksoy et al.

    Identification of out of control quality characteristics in a multivariate manufacturing environment

    Communications in Statistics-Theory and Methods

    (1991)
  • S. Dongkyu et al.

    A global optimization method rasid-ga for neural network training

    Journal of Advanced Computational Intelligence and Intelligent Informatics

    (2008)
  • A. Francisco et al.

    Interpreting the out-of-control signals of multivariate control charts employing neural networks

    World Academy of Science, Engineering and Technology

    (2010)
  • B.A. Garro et al.

    Evolving neural networks: A comparison between differential evolution and particle swarm optimization

    Advances in Swarm Intelligence, Lecture Notes in Computer Science

    (2011)
  • N. Greene William

    Econometric analysis

    (2003)
  • Gudise, V. G., & Venayagamoorthy, G. K. (2003). Comparison of particle swarm optimization and backpropagation as...
  • S.B. Guo et al.

    Gene selection based on mutual information for the classification of multi-class cancer

  • H. Hotelling

    Multivariate quality control: Techniques of statistical analysis

    (1947)
  • C. Hsu et al.

    The ANNIGMA for neural nets

    IEEE Transactions on Systems, Man and Cybernetics-Part B: Cybernetics

    (1999)
  • J.E. Jackson et al.

    An application of multivariate quality control to photographic process

    Journal of American Statistical Association

    (1957)
  • J.E. Jackson et al.

    An application of multivariate quality control to photographic process

    Journal of American Statistical Association

    (1957)
  • Y. Jian-bo et al.

    A neural network ensemble-based model for on-line monitoring and diagnosis of out-of-control signals in multivariate manufacturing processes

    Expert Systems with Applications

    (2009)
  • Cited by (29)

    • Industrial equipment reliability estimation: A Bayesian Weibull regression model with covariate selection

      2020, Reliability Engineering and System Safety
      Citation Excerpt :

      Nonetheless only a few sound methodological framework have been proposed for reliability modeling, in quite specific conditions (e.g.; [2,12,13]). To reduce the number of parameters in the Bayesian WRM, several approaches can be applied (e.g., [14,15]). According to the condition monitoring engineering practice (e.g., [16–18]), we develop a two-step covariate selection approach.

    View all citing articles on Scopus
    View full text