Sensitivity analysis of Takagi–Sugeno fuzzy neural network☆
Introduction
Due to the advantages of interpretability and self-learning ability,Fuzzy Neural Networks (FNNs) have been widely used as machine-learning techniques to deal with pattern recognition tasks such as feature analysis [1], [2], [3], [4], clustering, classification [3], [5] and regression [4], [5], [6]. The associated learning methods can be classified as unsupervised [1] or supervised [2], [3], [4], [5] learning based on the evaluation index.
In addition to the above-mentioned software-based implementation of FNNs [1], [2], [3], [4], [5], [6], researchers have designed dedicated fuzzy inference chips using microelectronic analog or digital circuits. The analog and digital fuzzy chips have successfully been used in real applications [7], [8]. But in the hardware-based implementation of FNNs, one of the most important issues is the sensitivity arising from the noises in the weights caused by the limited precision of analog or digital electronic devices. Even for systems implemented using software, when they are designed based on data, we do not want the system to be too sensitive to small changes to its parameters. Because a small change in the training data will produce a different system (high variance) and then suggesting that the identified model is not a good approximation of the underlying system. Thus it is important to analyze the sensitivity of the system with respect to perturbations of different parameters (weights) while designing it and exploit it to obtain a robust system. This is very important because it may significantly influence the training process as well as the generalizationability of the system.
In general, sensitivity refers to how the output of a system is influenced by perturbations to its inputs, weights and structure [9], [10]. There have been some studies on the sensitivity of neural-type structured networks. In [11], the concept of sensitivity about Adaline (adaptive linear element) using n-dimensional geometry approach is presented. For the Madaline (many Adalines), an analytical expression for the probability of error caused by the weight perturbation is introduced in [12]. Inspired by [12], the sensitivity of feedforward neural networks to changes in weights is derived in [13]. To investigate the sensitivity of different optimal converged weights obtained after training, a statistics-based sensitivity measure for multilayer perceptron (MLP) is defined in [14], and the validation of the definition is illustrated using regression and classification problems. This statistical sensitivity is employed as support regularization to impose some smoothness on the ‘Don’t know’ class in regard to the samples [10]. In [15], the sensitivity of an MLP is defined by the variance of the network output and output error. Based on the partial derivative of output to input or weights, another approach to define sensitivity of an MLP is introduced in [16], and the partial derivative-based sensitivity analysis has also been employed to realize the feature selection task [17], [18] and network structure optimization [19]. In [20], the partial derivative method is integrated into a software package to perform sensitivity analyses of MLP networks. Sensitivity analysis of deep neural networks (DNNs) is presented in [21], which mainly focuses on quantifying the effects of external and internal perturbations on classifiers. In addition, the sensitivity of DNNs about input perturbations is presented in [22], [23], [24]. In [25], four different methods are presented to serve the sensitivity analysis of the neural network outputs caused by the input factors. There are some other methods to conduct sensitivity analysis [26], [27].
Sensitivity analysis of the radial basis function neural network (RBFNN) has also attracted a lot of attention. In [28],the statistical sensitivity of outputs with respect to inputs and weights of a RBFNN is presented. To realize feature selection, a RBFNN-based sensitivity measure (SM) is presented in [29]. The sensitivity-based feature selection strategy first perturbs the input feature one by one and then the SM is computed. Finally, the features with low values of SM are eliminated. In [30], a sensitivity-based RBF network construction method is proposed. Here the sensitivity is defined as the expectation of the square of the output error caused by Gaussian perturbation of centers. In addition, a stochastic sensitivity measure is used to construct a localized generalization error bound which is applied to selection of RBFNN structure in [31].
Using sigmoid function, a sensitivity parameter is designed to control the importance of rules and remove the redundant rules for a first-order TS FNN [32]. Authors in [33] study the sensitivity of different types of membership functions to select the best ANFIS (adaptive neuro-fuzzy inference system) model and then the sensitivity with respect to input data is also studied. In [35], for the multiple-input single-output (MISO) FNN, a rule-based sensitivity measure is defined and the same one is used to realize on-line structure identification of FNN, i.e., on-line rule growing and pruning. For the first-order ANFIS model, Nazari-Shirkouhi et al. [34] analyze the sensitivity of each independent variable on the customers’ satisfaction (output variable) by removing the independent variable.
In [36], the sensitivity analysis of type-1 TS fuzzy model is first studied whose least square error varies with the pre-initialized parameters. Then the first order interval type-2 TS fuzzy model is used to observe the sensitivity of RMSE (root mean square error). The simulation results show that the TS model is sensitive to the cluster center and standard deviation of Gaussian member function, but insensitive to the consequent part. In [37], sensitivity analysis is used to investigate the variation of fuzzy outputs caused by factor changes. Authors developed a Windows-based application called DANA-FLSA (DANA Fuzzy Logic Sensitivity Analysis) for analyzing the sensitivity of input parameters on the output of a fuzzy system using the DANA (Data Analysis aNd Assessment) framework and the routines for sensitivity analysis available in the SimLab Dynamic Link Library. In [38], an existing sensitivity analysis technique is used to deal with fuzzy transportation problems and the effectiveness of the strategy is verified by using a numerical example.
Under ideal conditions, an optimal solution obtainedby any learning strategy should generalize well and small perturbations in the weights/parameters or inputs should not produce significant changes in the system outputs. In other words, it is undesirable to have an optimal system that is brittle (too sensitive to minor perturbation of the weights). However, it is difficult to design a digital or analog hardware with high precision to realize such robust systems. In practice, different training trials may (usually will) result in different systems/solutions. How do we choose the best one? Which solution is the least sensitive to perturbations in weights and/or inputs? Such questions necessitate sensitivity analysis of connectionist systems. Choi and Choi in [14] propose a regularization term that has been added to the objective function to obtain networks with weights having reduced sensitivity. All these motivate us to define and study the sensitivity analysis of TS-type neuro-fuzzy systems and usethe measures of sensitivity to realize robust systems using data. Our primary contributions in this study are as follows:
- 1.
Based on the relative output error a statistical sensitivity measure is defined, which can be used to study the sensitivity of a system with respect to weights/parameters. In other words, the sensitivity can guide us to select the most robust model from a set of trained FNNs;
- 2.
For the TS FNN, we theoretically analyze and deduce the expressions for the sensitivity measure both for additive and multiplicative noise to weights and consequent parameters separately;
- 3.
The deduced sensitivity measure is used as a regularization term to the error function, which provides a novel method to design a supervised learning strategy for dealing with real-life problems;
- 4.
An absolute output error-based statistical sensitivity measure is also defined, which is employed to validate that the sensitivity-based supervised method can obtain weights with reduced sensitivity compared to the traditional backpropagation (BP) algorithm.
The organization of the remaining part of the paper is as follows. In Section 2, the structure of a zero-order TS FNN with MIMO is first introduced, then a statistics-based definition of sensitivity is presented for this model.Based on this definition, we also derive the expressions for the statistical sensitivity with respect to additive/multiplicative perturbations of the consequent parameters. Section 3 presents two novel supervised learning algorithms based on the sensitivity measures. In Section 4, another statistics-based definition of sensitivity is provided to examine the validation of the sensitivity-based regularization term. Three real problems are used to illustrate the effectiveness of the two different sensitivity measures in Section 5. Section 6 concludes the paper.
Section snippets
The structure of a zero-order TS FNN
In this paper, we consider the zero-order TS FNN whose output for every forward mapping rule is a real weight value rather than a linear combination of the input linguistic variables [39]. Actually, the zero-order TS FNN is a simplified fuzzy neural model of the classical first-order TS FNN. Suppose be the input–output dataset (training samples), where is the jth input instance, is the desired output vector corresponding to the jth
Supervised learning based on sensitivity measure
To demonstrate the utility of the measures of sensitivity, and , we use each of them as a regularizing (penalty) term so that the training can find an FNN with reduced sensitivity to small perturbations to the system parameters.Compared with other regularizers based on norm, norm, norm [45] and first/s order derivatives [46] and so on, the sensitivity measure which is designed as a new regularizing factor in this paper has the ability of improving the robustness of
The validation of sensitivity-based regularization term
We shall use the sensitivity measures defined using Definition 2.1 to train FNNs for different problems. A natural question arises: how effective the sensitivity measures are to make the trained system robust. To see the effect of the regularization term on the identified system, we propose a new very intuitive measure of sensitivity for the MIMO systems. Definition 4.1 The statistical sensitivity to weight perturbations in an MIMO fuzzy neural network for the weight vector and an input sample is defined
Simulation results
In this section, using the three algorithms, NSR-FNN, ASR-FNN and MSR-FNN, three tasks are realized to examine the utility of the measures of sensitivity:
: For NSR-FNN, Definition 2.1 is employed to estimate the robustness of different optimal models. The sensitivity measure aggregated over all the samples is defined as follows:
: We want to compute the average relative error of each output variable for each rule i for the NSR-FNN
Conclusion
In this paper, we have introduced statistical sensitivity measures for TS fuzzy neural networks. These measures can be used to analyze theoretically as well as verify empirically the noise sensitivity of the system against Gaussian additive and multiplicative noises to any parameters of the antecedents and consequents. In this context, a relative output error-based sensitivity definition of a multi-output FNN is presented and the sensitivities with respect to additive/multiplicative
CRediT authorship contribution statement
Jian Wang: Supervision. Qin Chang: Supervision. Tao Gao: Software, Writing - original draft. Kai Zhang: Supervision. Nikhil R. Pal: Methodology, Writing - review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
References (50)
Soft computing for feature analysis
Fuzzy Sets and Systems
(1999)- et al.
How to make a neural network say “Don’t know
Information Sciences
(2018) - et al.
Perturbation method for deleting redundant inputs of perceptron networks
Neurocomputing
(1997) - et al.
Sensitivity analysis applied to the construction of radial basis function networks
Neural Networks
(2005) Autostructuration of fuzzy systems by rules sensitivity analysis
Fuzzy Sets and Systems
(2001)- et al.
Hydraulic head interpolation using ANFIS-model selection and sensitivity analysis
Computers and Geosciences
(2012) - et al.
Sensitivity analysis in fuzzy systems: integration of SimLab and DANA
Environmental Modelling and Software
(2010) - et al.
A neuro-fuzzy approach to data analysis of pairwise comparisons
International Journal of Approximate Reasoning
(1993) - et al.
A modified gradient-based neuro-fuzzy learning algorithm and its convergence
Information Sciences
(2010) - et al.
Recent advances in neuro-fuzzy system: a survey
Knowledge-based Systems
(2018)
Conjugate gradient-based Takagi-Sugeno fuzzy neural network parameter identification and its convergence analysis
Neurocomputing
Unsupervised feature evaluation: a neuro-fuzzy approach
IEEE Transactions on Neural Networks
Integrated feature analysis and fuzzy rule-based system identification in a neuro-fuzzy paradigm
IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cybernetics
An integrated mechanism for feature selection and fuzzy rule extraction for classification
IEEE Transactions on Fuzzy Systems
An interval type-2 neural fuzzy system for online system identification and feature elimination
IEEE Transactions on Fuzzy Systems
A Polak-Ribière-Polyak conjugate gradient-based neuro-fuzzy network and its convergence
IEEE Access
A fuzzy inference engine in nonlinear analog mode and its application to a fuzzy logic control
IEEE Transactions on Neural Networks
Water bath temperature control by a recurrent fuzzy controller and its FPGA implementation
IEEE Transactions on Industrial Electronics
Sensitivity Analysis for Neural Networks
Madaline rule II: a new method for training networks of Adaline
Sensitivity of feedforward neural networks to weight errors
IEEE Transactions on Neural Networks
Sensitivity analysis of multilayer perceptron with differential activation functions
IEEE Transactions on Neural Networks
Using function approximation to analyze the sensitivity of MLP with antisymmetric squashing activation function
IEEE Transactions on Neural Networks
Cited by (43)
Adaptive Nonstationary Fuzzy Neural Network
2024, Knowledge-Based SystemsOptimizing FDM 3D printing parameters for improved tensile strength using the Takagi–Sugeno fuzzy neural network
2024, Materials Today CommunicationsImplementation of fuzzy associative memory toward optimizing a neural network model to predict total iron binding capacity
2023, Biomedical Signal Processing and ControlAn interpretable neural network for robustly determining the location and number of cluster centers
2024, International Journal of Machine Learning and CyberneticsPeriodic Motions and Bifurcations of a Spring-Driven Joint System with Periodic Excitation
2024, Lecture Notes in Electrical Engineering
- ☆
Qin Chang and Jian Wang contributed equally to this paper. This work was supported in part by the National Natural Science Foundation of China (No. 62173345), the Fundamental Research Funds for the Central Universities (No. 20CX05002A, 20CX05012A), and the Major Scientific and Technological Projects of China National Petroleum Corporation (CNPC) (No. ZD2019-183-008).