Spatio-temporal degradation modeling and remaining useful life prediction under multiple operating conditions based on attention mechanism and deep learning
Introduction
Remaining useful life (RUL) is an essential part in the field of condition-based maintenance (CBM) of the engineering systems. RUL is defined as the residual service lifespan of the related equipment before it is considered to be failed [1]. Accurate RUL prediction contributes to schedule maintenance in advance and reduce maintenance costs effectively, which can effectively prevent system performance deterioration or even catastrophic failures [2]. Moreover, accurate RUL prediction with fully utilizing a large amount of monitoring data is helpful to improve the safety and reliability of system operation.
Compared with traditional RUL prediction methods, deep learning based methods have strong feature learning capabilities which can automatically extract hierarchical representations from training samples, and reduce the resource consumption of human participation [3]. The RUL prediction problem is essentially a regression problem related to time series [4]. Recurrent neural network (RNN) is widely used in RUL prediction. Moreover, two typical RNN-based methods, long short-term memory (LSTM) [5] and gated recurrent unit (GRU) [6], have improved the performance in time series prediction as they can partially capture long-term information of the input. Zhang [7] et al. proposed a bidirectional LSTM (Bi-LSTM)-based method for RUL prediction, in which reverse path LSTM was used to mitigate noise effects. However, the prediction performance may not be further improved due to their limited ability in extracting features from high-dimensional data; meanwhile, neglect of operating condition information also limits the generalization performance of the model, which are two main challenges in RUL prediction at present.
In the information fusion and RUL prediction of the high-dimensional data from different sensors, it is naturally associated with CNN-based methods which can easily extract spatial feature from series data. Miao [8] et al. proposed an adaptive convolutional neural network to capture essential information by adjusting receptive fields adaptively to achieve a better prediction performance. However, CNN methods are difficult to capture the dependence between long-term data. The current methods usually assume that the contribution of temporal and spatial dimension information contained in high-dimensional data from different sensors contribute to RUL prediction equally. Actually, information in temporal and spatial dimension needs to be treated differently. Therefore, it is important to resort to attention mechanism which can highlighting the key parts of high-dimensional data information. At present, the attention mechanism is used to measure the different importance of information in the temporal dimension. Li [9] et.al proposed an attention-based method to weight different time steps in RNN to improve the performance of RUL prediction. Since transformer shined in natural language processing (NLP) field, its core structure, multi-head attention, has been more and more popular. Liu [10] et.al applied encoder-decoder architecture of the transformer to RUL predictions considering the sequential dependencies on output labels and the importance between different time steps of the inputs. Mo [11] et.al utilized the multi-head attention mechanism to extract long-term information in sequence data to achieve RUL prediction. However, the attention mechanism used in current deep learning model rarely weights important information in the spatial dimension.
In the analysis of high-dimensional monitoring data, the additional contribution on RUL prediction of some important information was taken into account, but the essential relationship between the varying operating conditions and degradation progress, which is implied in the time series, was often not given enough attention. It may result in poor prediction performance to assume that the training and testing data come from the same distribution. Therefore, mode training and testing under different operating conditions receives extensive attention, among which the transfer learning and domain adaptation are the most popular [12]. However, these methods do not have the ability to generalize to unseen conditions. To overcome this problem, the domain generalization (DG) was introduced. Zhuang [13] et.al proposed an adversarial domain generalization framework with regularization learning for RUL and health assessment. However, it is still a challenging problem to bound the risk for DG due to the absence of target data. In practice, engineering systems are always exposed to varying operational conditions, which is another challenge in accurate RUL prediction. Huang [14] et al. proposed a BLSTM-based model for RUL prediction under various operating conditions, taking normalized multiple sensors sequence data as main inputs and normalized operational conditions sequence data as auxiliary inputs. Song [15] et al. normalized the original data with operating conditions, and combined it with an improved autoencoder model to make the prediction results more accurate. Li [16] et al. considered the change of degradation rate caused by different operating conditions. Li [17] et al. integrated multiple algorithms into one model, and selected an appropriate set of algorithms for different operating conditions to mitigate the impact of multiple operating conditions. However, information on future operating conditions is rarely incorporated into the current deep learning methods.
To overcome the above issues, a novel RUL prediction method is proposed considering Multiple operating conditions and Spatio-Temporal characteristics based on the improved Transformer model (noted as MSTformer in short). In the proposed method, Multi-head attention is integrated to capture the long-term dependency of time-series data, and two kinds of self-attention are adopted to characterize the importance of data in the temporal and spatial dimensions. In addition, future operating condition information is integrated into the model to improve the performance of the prediction results. The main contributions of this work are as follows:
- 1)
A novel end-to-end RUL prediction method in multiple operating conditions based on the improved Transformer architecture is proposed. The improved multi-head attention mechanism MSTformer architecture is resorted to consider the influence of multiple operating conditions on degradation, thus the life prediction accuracy is improved.
- 2)
The spatio-temporal features are automatically extracted to capture important information in the original data with two kinds of self-attention mechanism. The multi-head attention mechanism, an effective way to extract long-distance dependencies, is used to identify important information in the time dimension. A self-attention mechanism combined with CNN is effective to weight the data of important sensors in the spatial dimension.
- 3)
It is the first successful attempt to combine the future operating condition information with deep learning model. The prior information is added to the deep learning model, that is, the probability of each operating condition is combined with the regression layer to improve the prediction performance. Three NASA's turbofan engine datasets under variable operating conditions are considered to verify the effectiveness of the proposed model.
The organization structure of this paper is as follows. Section II introduces the related work of the attention mechanism and Transformer architecture. Section III describes the proposed method for RUL prediction. Section IV and section V implement the case study, and analyze the results. Finally, the conclusions are drawn in section VI.
Section snippets
Attention mechanism
Attention mechanism is a resource allocation mechanism that mimics the human brain. In the recognition process, the human brain pays more attention to specific regions of the picture, by assigning larger weights [18]. The attention mechanism can highlight the key information for improving prediction performance. Fig. 1 shows the detailed structure of attention mechanism, where Q stands for query, K represents key, and V means value.
The calculation of attention mechanism can be divided into two
RUL prediction method based on the improved Transformer model
Aiming at the difficulty of modeling with multi sensors and multi operating conditions, a RUL prediction method based on the improved transformer model is proposed. Different from the encode-decode structure of transformer, the spatio-temporal characteristics are extracted by two self-attention mechanisms to capture long-term information based on transformer encoder, which reduces the complexity of the model. Meanwhile, data cleaning methods are used to eliminate the difference of sensor data
C-MAPSS dataset
C-MAPSS dataset is provided by NASA's Center for Excellence in Failure Prediction and Research, which provides four-type sub-datasets of turbine engine, as shown in Table 1. The turbine engine in the training dataset has complete run-to-failure data; while the turbine engine in the test dataset only provides data for the first segment of the full life cycle. FD002 and FD004, operating under changing conditions, are selected for case verification. The failure mode in the FD002 dataset is
Discussion
In this study, two metrics, RMSE and Score, are used to evaluate the performance of the prediction method, whose formula are as follows:
Conclusions
In this paper, we propose a novel RUL prediction method based on improved Transformer model considering multiple operating conditions and spatio-temporal characteristics. Without a RNN structure, the self-attention mechanism is used to extract long-term dependencies among data in our MSTformer model, which can make it possible to use parallel computing to improve computing efficiency. The proposed end-to-end deep learning method can automatically extract weighted spatio-temporal features and
CRediT authorship contribution statement
Dan Xu: Conceptualization, Methodology, Writing – original draft, Writing – review & editing, Supervision. Xiaoqi Xiao: Conceptualization, Validation, Writing – original draft, Writing – review & editing. Jie Liu: Writing – review & editing. Shaobo Sui: Conceptualization, Investigation.
Declaration of Competing Interest
The authors declared that they have no conflicts of interest to this work.
We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.
Acknowledgment
This work is supported by National Natural Science Foundation of China (No. 51875017), National key Laboratory of Science and Technology on Reliability and Environmental Engineering (No. 6142004180103, WDZC20220103), Fundamental Research Funds for the Central Universities (No. YWF-22-L-726).
References (34)
- et al.
Machinery health prognostics: A systematic review from data acquisition to RUL prediction
Mech Syst Sig Process
(2018) - et al.
Remaining useful life prediction and predictive maintenance strategies for multi-state manufacturing systems considering functional dependence
Reliab Eng Syst Saf
(2021) - et al.
Prediction of remaining useful life of multi-stage aero-engine based on clustering and LSTM fusion
Reliab Eng Syst Saf
(2021) - et al.
Long short-term memory for machine remaining life prediction
J Manuf Syst
(2018) - et al.
A sparse domain adaption network for remaining useful life prediction of rolling bearings under different working conditions
Reliab Eng Syst Saf
(2022) - et al.
Attention-based deep survival model for time series data
Reliab Eng Syst Saf
(2022) - et al.
Aircraft engine remaining useful life estimation via a double attention-based data-driven architecture
Reliab Eng Syst Saf
(2022) - et al.
Remaining useful life prediction of machinery under time-varying operating conditions based on a two-factor state-space model
Reliab Eng Syst Saf
(2019) - et al.
Remaining useful life prediction for multi-sensor systems using a novel end-to-end deep-learning method
Measurement
(2021) - et al.
Prediction of remaining useful life based on bidirectional gated recurrent unit with temporal self-attention mechanism
Reliab Eng Syst Saf
(2022)
Remaining useful life estimation in prognostics using deep convolution neural networks
Reliab Eng Syst Saf
Remaining useful life estimation using a bidirectional recurrent neural network based autoencoder scheme
Mech Syst Sig Process
Utilizing uncertainty information in remaining useful life estimation via Bayesian neural networks and Hamiltonian Monte Carlo
J Manuf Syst
An improved similarity-based prognostic algorithm for RUL estimation using an RNN autoencoder scheme
Reliab Eng Syst Saf
Multicellular LSTM-based deep learning model for aero-engine remaining useful life prediction
Reliab Eng Syst Saf
Estimation of remaining useful life of rolling element bearings using wavelet packet decomposition and artificial neural network
IJST-T ELECTR ENG
A novel temporal convolutional network with residual self-attention mechanism for remaining useful life prediction of rolling bearings
Reliab Eng Syst Saf
Cited by (26)
A regularized constrained two-stream convolution augmented Transformer for aircraft engine remaining useful life prediction
2024, Engineering Applications of Artificial IntelligenceMachinery health prognostic with uncertainty for mineral processing using TSC-TimeGAN
2024, Reliability Engineering and System SafetyMachinery degradation trend prediction considering temporal distribution discrepancy between degradation stages
2024, Engineering Applications of Artificial IntelligenceSingle gated RNN with differential weighted information storage mechanism and its application to machine RUL prediction
2024, Reliability Engineering and System SafetyDVGTformer: A dual-view graph Transformer to fuse multi-sensor signals for remaining useful life prediction
2024, Mechanical Systems and Signal ProcessingIncorporating prior knowledge into self-supervised representation learning for long PHM signal
2024, Reliability Engineering and System Safety