Spatio-temporal degradation modeling and remaining useful life prediction under multiple operating conditions based on attention mechanism and deep learning

https://doi.org/10.1016/j.ress.2022.108886Get rights and content

Highlights

  • A novel end-to-end RUL prediction method in multiple operating conditions based on the improved Transformer architecture is proposed.

  • The spatio-temporal features are extracted to capture important information in the original data. Importance of the data in different time steps and sensors are differentiated by the weights optimized automatically with two kinds of self-attention mechanism.

  • The future operating conditions leading to different RUL are integrated in the model to improve the prediction accuracy.

Abstract

Accurately predicting the Remaining Useful Life (RUL) is useful to avoid unexpected significant failure of engineering system and reduce maintenance costs effectively. Meanwhile, the diverse operating conditions and high-dimensional feature variables from diverse sensors spatially located in the system are two main obstacles for building an accurate and stable RUL prediction model. Considering that these two obstacles have not been fully considered in the state-of-art work, a novel RUL prediction method based on the improved Transformer model is proposed in this work, which resorts to the attention mechanism and deep learning considering spatio-temporal characteristics and multiple operating conditions. First, the difference of the original sequence data caused by operating conditions is eliminated by clustering and standardization in the data preprocessing process. The future condition information is also considered in RUL prediction calculation. Meanwhile, spatio-temporal feature is extracted by self-attention to realize the information fusion of multi-dimensional sensors and long-term series, in which position encoding is designed to retain the sequence information in the original sequence data. Finally, experimental results on the C-MAPSS and N-CMAPSS datasets show that the proposed method achieves a better performance compared with other existing methods.

Introduction

Remaining useful life (RUL) is an essential part in the field of condition-based maintenance (CBM) of the engineering systems. RUL is defined as the residual service lifespan of the related equipment before it is considered to be failed [1]. Accurate RUL prediction contributes to schedule maintenance in advance and reduce maintenance costs effectively, which can effectively prevent system performance deterioration or even catastrophic failures [2]. Moreover, accurate RUL prediction with fully utilizing a large amount of monitoring data is helpful to improve the safety and reliability of system operation.

Compared with traditional RUL prediction methods, deep learning based methods have strong feature learning capabilities which can automatically extract hierarchical representations from training samples, and reduce the resource consumption of human participation [3]. The RUL prediction problem is essentially a regression problem related to time series [4]. Recurrent neural network (RNN) is widely used in RUL prediction. Moreover, two typical RNN-based methods, long short-term memory (LSTM) [5] and gated recurrent unit (GRU) [6], have improved the performance in time series prediction as they can partially capture long-term information of the input. Zhang [7] et al. proposed a bidirectional LSTM (Bi-LSTM)-based method for RUL prediction, in which reverse path LSTM was used to mitigate noise effects. However, the prediction performance may not be further improved due to their limited ability in extracting features from high-dimensional data; meanwhile, neglect of operating condition information also limits the generalization performance of the model, which are two main challenges in RUL prediction at present.

In the information fusion and RUL prediction of the high-dimensional data from different sensors, it is naturally associated with CNN-based methods which can easily extract spatial feature from series data. Miao [8] et al. proposed an adaptive convolutional neural network to capture essential information by adjusting receptive fields adaptively to achieve a better prediction performance. However, CNN methods are difficult to capture the dependence between long-term data. The current methods usually assume that the contribution of temporal and spatial dimension information contained in high-dimensional data from different sensors contribute to RUL prediction equally. Actually, information in temporal and spatial dimension needs to be treated differently. Therefore, it is important to resort to attention mechanism which can highlighting the key parts of high-dimensional data information. At present, the attention mechanism is used to measure the different importance of information in the temporal dimension. Li [9] et.al proposed an attention-based method to weight different time steps in RNN to improve the performance of RUL prediction. Since transformer shined in natural language processing (NLP) field, its core structure, multi-head attention, has been more and more popular. Liu [10] et.al applied encoder-decoder architecture of the transformer to RUL predictions considering the sequential dependencies on output labels and the importance between different time steps of the inputs. Mo [11] et.al utilized the multi-head attention mechanism to extract long-term information in sequence data to achieve RUL prediction. However, the attention mechanism used in current deep learning model rarely weights important information in the spatial dimension.

In the analysis of high-dimensional monitoring data, the additional contribution on RUL prediction of some important information was taken into account, but the essential relationship between the varying operating conditions and degradation progress, which is implied in the time series, was often not given enough attention. It may result in poor prediction performance to assume that the training and testing data come from the same distribution. Therefore, mode training and testing under different operating conditions receives extensive attention, among which the transfer learning and domain adaptation are the most popular [12]. However, these methods do not have the ability to generalize to unseen conditions. To overcome this problem, the domain generalization (DG) was introduced. Zhuang [13] et.al proposed an adversarial domain generalization framework with regularization learning for RUL and health assessment. However, it is still a challenging problem to bound the risk for DG due to the absence of target data. In practice, engineering systems are always exposed to varying operational conditions, which is another challenge in accurate RUL prediction. Huang [14] et al. proposed a BLSTM-based model for RUL prediction under various operating conditions, taking normalized multiple sensors sequence data as main inputs and normalized operational conditions sequence data as auxiliary inputs. Song [15] et al. normalized the original data with operating conditions, and combined it with an improved autoencoder model to make the prediction results more accurate. Li [16] et al. considered the change of degradation rate caused by different operating conditions. Li [17] et al. integrated multiple algorithms into one model, and selected an appropriate set of algorithms for different operating conditions to mitigate the impact of multiple operating conditions. However, information on future operating conditions is rarely incorporated into the current deep learning methods.

To overcome the above issues, a novel RUL prediction method is proposed considering Multiple operating conditions and Spatio-Temporal characteristics based on the improved Transformer model (noted as MSTformer in short). In the proposed method, Multi-head attention is integrated to capture the long-term dependency of time-series data, and two kinds of self-attention are adopted to characterize the importance of data in the temporal and spatial dimensions. In addition, future operating condition information is integrated into the model to improve the performance of the prediction results. The main contributions of this work are as follows:

  • 1)

    A novel end-to-end RUL prediction method in multiple operating conditions based on the improved Transformer architecture is proposed. The improved multi-head attention mechanism MSTformer architecture is resorted to consider the influence of multiple operating conditions on degradation, thus the life prediction accuracy is improved.

  • 2)

    The spatio-temporal features are automatically extracted to capture important information in the original data with two kinds of self-attention mechanism. The multi-head attention mechanism, an effective way to extract long-distance dependencies, is used to identify important information in the time dimension. A self-attention mechanism combined with CNN is effective to weight the data of important sensors in the spatial dimension.

  • 3)

    It is the first successful attempt to combine the future operating condition information with deep learning model. The prior information is added to the deep learning model, that is, the probability of each operating condition is combined with the regression layer to improve the prediction performance. Three NASA's turbofan engine datasets under variable operating conditions are considered to verify the effectiveness of the proposed model.

The organization structure of this paper is as follows. Section II introduces the related work of the attention mechanism and Transformer architecture. Section III describes the proposed method for RUL prediction. Section IV and section V implement the case study, and analyze the results. Finally, the conclusions are drawn in section VI.

Section snippets

Attention mechanism

Attention mechanism is a resource allocation mechanism that mimics the human brain. In the recognition process, the human brain pays more attention to specific regions of the picture, by assigning larger weights [18]. The attention mechanism can highlight the key information for improving prediction performance. Fig. 1 shows the detailed structure of attention mechanism, where Q stands for query, K represents key, and V means value.

The calculation of attention mechanism can be divided into two

RUL prediction method based on the improved Transformer model

Aiming at the difficulty of modeling with multi sensors and multi operating conditions, a RUL prediction method based on the improved transformer model is proposed. Different from the encode-decode structure of transformer, the spatio-temporal characteristics are extracted by two self-attention mechanisms to capture long-term information based on transformer encoder, which reduces the complexity of the model. Meanwhile, data cleaning methods are used to eliminate the difference of sensor data

C-MAPSS dataset

C-MAPSS dataset is provided by NASA's Center for Excellence in Failure Prediction and Research, which provides four-type sub-datasets of turbine engine, as shown in Table 1. The turbine engine in the training dataset has complete run-to-failure data; while the turbine engine in the test dataset only provides data for the first segment of the full life cycle. FD002 and FD004, operating under changing conditions, are selected for case verification. The failure mode in the FD002 dataset is

Discussion

In this study, two metrics, RMSE and Score, are used to evaluate the performance of the prediction method, whose formula are as follows:RMSE=1Nm=1N(youtiyturei)2score=i=1nsi,si={e(youtiyturei)131,youti<ytureie(youtiyturei)101,youti>yturei

Conclusions

In this paper, we propose a novel RUL prediction method based on improved Transformer model considering multiple operating conditions and spatio-temporal characteristics. Without a RNN structure, the self-attention mechanism is used to extract long-term dependencies among data in our MSTformer model, which can make it possible to use parallel computing to improve computing efficiency. The proposed end-to-end deep learning method can automatically extract weighted spatio-temporal features and

CRediT authorship contribution statement

Dan Xu: Conceptualization, Methodology, Writing – original draft, Writing – review & editing, Supervision. Xiaoqi Xiao: Conceptualization, Validation, Writing – original draft, Writing – review & editing. Jie Liu: Writing – review & editing. Shaobo Sui: Conceptualization, Investigation.

Declaration of Competing Interest

The authors declared that they have no conflicts of interest to this work.

We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Acknowledgment

This work is supported by National Natural Science Foundation of China (No. 51875017), National key Laboratory of Science and Technology on Reliability and Environmental Engineering (No. 6142004180103, WDZC20220103), Fundamental Research Funds for the Central Universities (No. YWF-22-L-726).

References (34)

  • X Li et al.

    Remaining useful life estimation in prognostics using deep convolution neural networks

    Reliab Eng Syst Saf

    (2018)
  • W Yu et al.

    Remaining useful life estimation using a bidirectional recurrent neural network based autoencoder scheme

    Mech Syst Sig Process

    (2019)
  • M Benker et al.

    Utilizing uncertainty information in remaining useful life estimation via Bayesian neural networks and Hamiltonian Monte Carlo

    J Manuf Syst

    (2021)
  • W Yu et al.

    An improved similarity-based prognostic algorithm for RUL estimation using an RNN autoencoder scheme

    Reliab Eng Syst Saf

    (2020)
  • S Xiang et al.

    Multicellular LSTM-based deep learning model for aero-engine remaining useful life prediction

    Reliab Eng Syst Saf

    (2021)
  • A Rohani Bastami et al.

    Estimation of remaining useful life of rolling element bearings using wavelet packet decomposition and artificial neural network

    IJST-T ELECTR ENG

    (2019)
  • Y Cao et al.

    A novel temporal convolutional network with residual self-attention mechanism for remaining useful life prediction of rolling bearings

    Reliab Eng Syst Saf

    (2021)
  • Cited by (26)

    View all citing articles on Scopus
    View full text