Allocation of Eavesdropping Attacks for Multi-System Remote State Estimation

Abstract

In recent years, the problem of cyber–physical systems’ remote state estimations under eavesdropping attacks have been a source of concern. Aiming at the existence of eavesdroppers in multi-system CPSs, the optimal attack energy allocation problem based on a SINR (signal-to-noise ratio) remote state estimation is studied. Assume that there are N sensors, and these sensors use a shared wireless communication channel to send their state measurements to the remote estimator. Due to the limited power, eavesdroppers can only attack M channels out of N channels at most. Our goal is to use the Markov decision processes (MDP) method to maximize the eavesdropper’s state estimation error, so as to determine the eavesdropper’s optimal attack allocation. We propose a backward induction algorithm which uses MDP to obtain the optimal attack energy allocation strategy. Compared with the traditional induction algorithm, this algorithm has lower computational cost. Finally, the numerical simulation results verify the correctness of the theoretical analysis.

1. Introduction

Cyber–physical systems (CPS) are considered to be among the revolutionary technologies due to the continuous technological breakthroughs and innovations in information technology and in the manufacturing industry [1]. CPS is a multidimensional and complex system that deeply integrates control, communication and computing (that is, 3C technology composed of control, communication and computing)and can realizes large-scale information acquisition and intelligent control of the physical world through the cognition, communication and control of physical objects, so that the network can monitor the specific actions of a physical entity in a real-time, reliable, remote and safe way [2,3]. CPS is widely used in aerospace, industrial production, advanced automobile systems, energy reserve, environmental monitoring, national defense and military, infrastructure construction, intelligent building, smart grids, transportation systems and telemedicine [4]. With the rapid development of network, computing, sensing and control systems, CPS technology is more and more widely used, and the emerging network attacks make the wireless CPS system very fragile, and the security of CPS becomes the primary consideration [5,6,7].

For the security issues of a system’s remote state estimation, there are many forms of malicious network attacks, but they are divided into three main and common categories: denial of service (DoS) attacks, integrity (including replay and false data injection) attacks and eavesdropping attacks [8]. DoS attacks are designed to interfere with wireless communication channels. This attack will lead to a significant decline in the estimation accuracy in CPS [9]. Peng [10] and Zhang [11] formulated the problem as a Markov decision process (MDP) problem to consider the optimal attack power allocation for remote state estimation in a multi-system. Integrity attacks can disrupt the transmitted data packets with stealth constraint [12,13]. In Ref. [14], an important scenario is designed from the attacker’s point of view, in which the false data injection attack can completely and secretly destroy CPS. In addition, the channel may be subject to eavesdropping attacks, which can lead to serious economic losses and even pose a threat to human survival by eavesdropping on personal privacy data [15,16]. For example, in the intelligent transportation system, eavesdroppers infer the path planning of vehicles by monitoring the location information, and on this basis, eavesdropping attacks will easily succeed [17,18]. In terms of existing research, data encryption is the main method to protect system privacy from eavesdropping attacks [19,20,21].

Recently, the issue of remote state estimation in the presence of eavesdroppers has attracted widespread attention from researchers. The attack types of eavesdropping attacks are divided into passive eavesdropping attacks and active eavesdropping attacks. Some estimation and control problems have been studied in the presence of active attacks. Han [22] studied the problem of active eavesdropping on fading channels, and proposed an interference-assisted eavesdropping method to improve the probability of successful monitoring. Yuan [23] constructed a two-person non-zero-sum game between the sensor and the active eavesdropper with the goal of minimizing the covariance of the self-estimated error and maximizing the covariance of the opponent’s estimated error. Ding [24] took the trade-off between stealth and eavesdropping performance as a constrained MDP, and proposed an optimal strategy for active eavesdropping.

The above literatures indicate a certain breakthrough in the design of active eavesdropping solutions. This paper mainly studies the passive attacks of eavesdroppers. Tsiamis [25] proposed a confidentiality mechanism for randomly hiding sensor information, and explored the trade-off between user utility and control theory confidentiality through optimization methods. Huang [26] proposed a new encryption strategy and considered the cost of the encryption process. Then, the optimal determinism of the encryption strategy and the existence of the Markov strategy in the finite time horizon are proven. Wang [27] theoretically proved that there are some structural properties in the optimal transmission scheduling for known and unknown eavesdropper estimation errors. In reference [28], the transmission scheduling strategy of remote state estimation systems with eavesdroppers on packet-dropping links was studied. Yuan [29] transformed the system model into MDP in order to obtain the optimal transmission scheduling to minimize the AoI of CPS and keep the AoI of eavesdroppers above a certain level, and proved that the optimal transmission scheduling strategy is a threshold behavior on the CPS and AoI of eavesdroppers, respectively. In [30], the proposed problem is formulated as a Stackelberg game, and the strategy of maximizing the secure transmission rate between sensor and controller in the presence of malicious eavesdroppers and disruptors is studied. On the basis of analyzing the influence of different strategies on eavesdropping performance, Zhou [31] studied the multi-output system and proposed a decryption scheduling scheme to minimize the expected estimation error under the condition of energy constraint.

Most of the existing literature studies the optimal transmission strategies of sensors from the remote estimator. Compared to [27,28], this paper studies the optimal attack energy allocation strategies from the perspective of eavesdroppers. Moreover, the previous literature mainly focuses on the situation that CPS has eavesdroppers in a single system and a finite time range, but does not pay too much attention to the situation when there are eavesdroppers in a multi-system and in an infinite time range. In this paper, the optimal attack allocation problem of remote state estimation in CPS with eavesdropping attacks in a multi-system in infinite time range is studied. Our goal is to maximize the state estimation error of the eavesdropper, so as to determine the optimal attack allocation of the eavesdropper. The contributions of this paper are as follows:

• We propose a multi-system eavesdropping attack model based on channel SINR, which reveals the relationship between attack power and packet arrival rate.
• In the infinite time horizon, under the condition of energy constraint, the optimal attack scheduling strategy is obtained by constructing MDP and using the Bellman equation.
• Finally, according to the given algorithm, the optimal attack energy allocation strategy is obtained, and then it is verified by simulation experiments.

Notations: The entire paper uses the following symbols. $\mathbb{N}$ is the set of natural numbers. The n-dimensional Euclidean space is denoted by ${\mathbb{R}}^n$. ${\mathbb{S}}_{+}^n$ (${\mathbb{S}}_{++}^n$) is the set of n by n positive semi-definite matricess (and positive definite matrices). $Tr\left(X\right)$ is the trace of a matrix X, and $X^T$ is the transpose of X and $X^{-1}$ denotes the inverse of matrix X. $X>0$ and $X\ge 0$ represent that X is a positive definite matrix and positive semidefinite matrix, respectively. For functions $\tilde{g}$ and h, $\tilde{g}\circ h\left(x\right)$ stands for the function composition $\tilde{g}\left(h\left(x\right)\right)$ and $h^n\left(x\right)=h\left(h^{n-1}\left(x\right)\right)$ with $h^0\left(x\right)=x$. $\mathbb{E}[·]$ indicates taking the expected value of ′·′. $\mathbb{P}[·]$ denotes the probability of ′·′.

2. Problem Setup

2.1. System Model

Figure 1 shows the system architecture. We consider N general discrete time-invariant stochastic system, which is given as follows

$$\begin{array}{ccc}\hfill x_i(k+1)& =& \hfill A_i{x}_i\left(k\right)+{\omega }_i\left(k\right),\hfill \end{array}$$

(1)

$$\begin{array}{ccc}\hfill y_i\left(k\right)& =& \hfill C_i{x}_i\left(k\right)+v_i\left(k\right)\hfill \end{array}$$

(2)

where $k\in \mathbb{N}$ is the time index and $x_i\left(k\right)\in {\mathbb{R}}^n$ and $y_i\left(k\right)\in {\mathbb{R}}^m$ refer to the state of the ith system and the system measurements vector taken by the sensor at time k, respectively. The process noise ${\omega }_i\left(k\right)\in {\mathbb{R}}^n$ and the observation noise $v_i\left(k\right)\in {\mathbb{R}}^m$ are assumed to be independent and identically distributed (i.i.d.) Gaussian noises with zero-mean and the covariances matrix $Q_i⩾0$ and matrix $R_i>0$, respectively. The initial state of the ith system $x_i\left(0\right)$ is also a zero-mean Gaussian random variable independented ${\omega }_i\left(k\right)$ and $v_i\left(k\right)$ with covariance $F_i\left(0\right)\ge 0$. We also assume that the pair $(A_i,C_i)$ is observable and $(A_i,\sqrt{Q_i})$ is stabilizable. Assuming that the sensors in the system are intelligent sensors with certain computing powers, each sensor can first use the collected observation data to calculate the local state estimation, and then transmit the local state estimation value to the remote state estimator. Therefore, we use ${\tilde{x}}_i\left(k\right)$ and ${\tilde{F}}_i\left(k\right)$ to represent the ith sensor’s local minimum mean-squared error (MMSE) estimate of the state and the corresponding error covariance [32]:

$$\begin{array}{ccc}\hfill {\tilde{x}}_i\left(k\right)& =& \mathbb{E}\left[x_i\left(k\right)\right|y_i\left(1\right),y_i\left(2\right),...,y_i\left(k\right)],\hfill \\ {\tilde{F}}_i\left(k\right)\hfill & =& \mathbb{E}[(x_i\left(k\right)-{\tilde{x}}_i\left(k\right)){(x_i\left(k\right)-{\tilde{x}}_i\left(k\right))}^T\hfill \end{array}$$

(3)

$$\begin{array}{ccc}& & \hfill |y_i\left(1\right),y_i\left(2\right),...,y_i\left(k\right)],\hfill \end{array}$$

(4)

which can be calculated based on a standard Kalman filter:

$$\begin{array}{ccc}\hfill {\tilde{x}}_i\left(k\right|k-1)& =& A_i{\tilde{x}}_i(k-1),\hfill \\ \hfill {\tilde{F}}_i\left(k\right|k-1)& =& A_i{\tilde{F}}_i(k-1)A_i^T+Q_i,\hfill \\ \hfill K_i\left(k\right)& =& {\tilde{F}}_i\left(k\right|k-1)C_i^T{[C_i{\tilde{F}}_i\left(k\right|k-1)C_i^T+R_i]}^{-1},\hfill \\ \hfill {\tilde{x}}_i\left(k\right)& =& A_i{\tilde{x}}_i(k-1)+K_i\left(k\right)(y_i\left(k\right)-C_i{A}_i{\tilde{x}}_i(k-1)),\hfill \\ \hfill {\tilde{F}}_i\left(k\right)& =& (I-K_i\left(k\right)C_i){\tilde{F}}_i\left(k\right),\hfill \end{array}$$

where

$$\begin{array}{ccc}\hfill {\tilde{x}}_i\left(k\right|k-1)& =& \mathbb{E}\left[x_i\left(k\right)\right|y_i\left(1\right),y_i\left(2\right),...,y_i(k-1)],\hfill \\ \hfill {\tilde{F}}_i\left(k\right|k-1)& =& \mathbb{E}[(x_i\left(k\right)-{\tilde{x}}_i\left(k\right)){(x_i\left(k\right)-{\tilde{x}}_i\left(k\right))}^T\hfill \\ & & |y_i\left(1\right),y_i\left(2\right),...,y_i(k-1)],\hfill \end{array}$$

$K_i\left(k\right)$ is the gain of the Kalman filter and the initial condition is ${\tilde{x}}_i\left(0\right)=0$.

For ease of representation, we can also define the Lyapunov and Riccati operators h, $\tilde{g}$: ${\mathbb{S}}_{+}^n$→${\mathbb{S}}_{+}^n$ as

$$\begin{array}{ccc}\hfill h_i\left(X\right)& \triangleq & \hfill A_{i}X{A}_i^T+Q_i,\hfill \end{array}$$

(5)

$$\begin{array}{ccc}\hfill {\tilde{g}}_i\left(X\right)& \triangleq & \hfill X-X{C}_i^T{\left[C_{i}X{C}_i^T\right]}^{-1}{C}_{i}X,\hfill \end{array}$$

(6)

$$\begin{array}{ccc}\hfill h_i^k\left(X\right)& \triangleq & \hfill \underset{\mathrm{k}\mathrm{times}}{\underbrace{h_i\circ h_i\circ ···\circ h_i\circ h_i}}\left(X\right).\hfill \end{array}$$

(7)

Under the assumptions of detectability and stability, it has been shown that the posterior estimation error covariance matrix of the Kalman filter converges exponentially from any initial condition to a unique value $\tilde{F}$, [33], i.e., ${\tilde{F}}_i\left(k\right)={\tilde{F}}_i$, $k\ge 1$, which ${\tilde{F}}_i$ is the steady-state error covariance, which is determined by the unique positive semi-definite solution of ${\tilde{g}}_i\circ h_i\left(X\right)$=X [34].

 Lemma 1

([35]). For $0\le {\xi }_1\le {\xi }_2$, it has

$$Tr\left\{\tilde{F}\right\}\le Tr\left\{h_i^{{\xi }_1}\left(\tilde{F}\right)\right\}\le Tr\left\{h_i^{{\xi }_2}\left(\tilde{F}\right)\right\}.$$

(8)

2.2. Attack Model Based on SINR

To simulate random data loss due to fading and interference, we assume that the communication between the sensor and the remote estimator or the eavesdropper is via an Additive White Gaussian Noise (AWGN) channel using quadrature amplitude modulation (QAM). Data packets sent by the sensor are quantized and mapped to QAM symbols. Then, digital communication theory reveals the relationship between symbol error rate (SER) and SINR as follows [5,36]:

$$SE{R}_i^{\wp }=2Q\left(\sqrt{\alpha SIN{R}_i}\right)$$

(9)

where $Q\left(x\right)=\frac{1}{\sqrt{2\pi }}{\int }_x^{\infty }{e}^{-t^2/2}dt$, $\wp \in \{e,a\}$ and $\alpha >0$ is a parameter. $\wp =e$ represents the remote estimator side, $\wp =a$ indicates the eavesdropper side.

Considering the remote estimator side first, the channel SINR for the remote estimator at time k is [24],

$$SIN{R}_i^e=\frac{{\Phi }_i{p}_i\left(k\right)}{{\sigma }_{i,e}^2}$$

(10)

where ${\Phi }_i$ is the channel gain of the ith communication channel between the sensor and the remote estimator. $p_i\left(k\right)\ge 0$ is the transmission power for the QAM symbol used by sensor i at time k and ${\sigma }_{i,e}^2$ is the AWGN power of the ith channel between the sensor and the remote estimator. Define a random variable ${\zeta }_i\left(k\right)\in \{0,1\}$ as to whether the remote estimator successfully receives the information at time k, i.e.,

$${\zeta }_i^e\left(k\right)=\left\{\begin{array}{cc}0,& \mathrm{otherwise}(\mathrm{regarded}\mathrm{as}\mathrm{dropout})\\ 1,& \mathrm{if}{\tilde{x}}_i\left(k\right)\mathrm{is}\mathrm{successfully}\mathrm{received}\mathrm{by}\mathrm{the}\mathrm{estimator}.\end{array}\right.$$

Then, the packet arrival rate of the remote estimator is

$$\begin{array}{cc}\hfill {\lambda }_i^e\left(k\right)& \hfill =\mathbb{P}\{{\zeta }_i^e\left(k\right)=1\}=1-SE{R}_i^e\hfill \\ & \hfill =1-2Q\left(\sqrt{\alpha \frac{{\Phi }_i{p}_i\left(k\right)}{{\sigma }_{i,e}^2}}\right).\hfill \end{array}$$

(11)

Secondly, considering the eavesdropper side, we can know that the SINR of the channel at the eavesdropper side at time k can be expressed as

$$SIN{R}_i^a=\frac{{\Psi }_i{p}_i\left(k\right)}{a_i\left(k\right)+{\sigma }_{i,a}^2}$$

(12)

where ${\Psi }_i$ is the channel gain of the ith communication channel between the sensor and the eavesdropper. $a_i\left(k\right)$ indicates the attack power to the ith channel launched by the eavesdropper. ${\sigma }_{i,a}^2$ is the AWGN power of the channel between the sensor and the eavesdropper. Similarly, we use a binary random variable ${\zeta }_i^a\left(k\right)$ to indicate whether the eavesdropper is successful in eavesdropping, i.e.,

$${\zeta }_i^a\left(k\right)=\left\{\begin{array}{cc}0,& \mathrm{otherwise}\\ 1,& \mathrm{if}{\tilde{x}}_i\left(k\right)\mathrm{is}\mathrm{successfully}\mathrm{eavesdropped}\mathrm{by}\mathrm{the}\mathrm{eavesdropper}\end{array}\right.$$

Then, the probability that an eavesdropper can successfully eavesdrop is:

$$\begin{array}{cc}\hfill {\lambda }_i^a\left(k\right)& \hfill =\mathbb{P}\{{\zeta }_i^a\left(k\right)=1\}=1-SE{R}_i^a\hfill \\ & \hfill =1-2Q\left(\sqrt{\alpha \frac{{\Psi }_i{p}_i\left(k\right)}{a_i\left(k\right)+{\sigma }_{i,a}^2}}\right).\hfill \end{array}$$

(13)

Hypothetical processes ${\zeta }_i^e\left(k\right)$ and ${\zeta }_i^a\left(k\right)$ are independent of each other.

2.3. Remote State Estimation

With the local estimate received by the remote estimator, we can determine the MMSE state estimate ${\widehat{x}}_i^e\left(k\right)$ and the corresponding estimation error covariance $F_i^e\left(k\right)$ of the remote estimator at time k, where ${\widehat{x}}_i^e\left(k\right)$ and $F_i^e\left(k\right)$ are obtained by the following iterative process:

$${\widehat{x}}_i^e\left(k\right)=\left\{\begin{array}{cccc}& A_i{\widehat{x}}_i^e(k-1),\hfill & & \mathrm{otherwise},\hfill \\ & {\tilde{x}}_i\left(k\right),\hfill & & \mathrm{if}{\zeta }_{\mathrm{i}}^{\mathrm{e}}\left(\mathrm{k}\right)=1,\hfill \end{array}\right.$$

(14)

$$F_i^e\left(k\right)=\left\{\begin{array}{cccc}& h_i\left(F_i^e(k-1)\right),\hfill & & \mathrm{otherwise},\hfill \\ & \widetilde{F_i},\hfill & & \mathrm{if}{\zeta }_{\mathrm{i}}^{\mathrm{e}}\left(\mathrm{k}\right)=1.\hfill \end{array}\right.$$

(15)

Similarly, denote ${\widehat{x}}_i^a\left(k\right)$ and $F_i^e\left(k\right)$ as the MMSE state estimation and corresponding error covariance of the eavesdropper at time k, then ${\widehat{x}}_i^a\left(k\right)$ and $F_i^a\left(k\right)$ can be expressed as

$${\widehat{x}}_i^a\left(k\right)=\left\{\begin{array}{cccc}& A_i{\widehat{x}}_i^a(k-1),\hfill & & \mathrm{otherwise},\hfill \\ & {\tilde{x}}_i\left(k\right),\hfill & & \mathrm{if}{\zeta }_{\mathrm{i}}^{\mathrm{a}}\left(\mathrm{k}\right)=1,\hfill \end{array}\right.$$

(16)

$$F_i^a\left(k\right)=\left\{\begin{array}{cccc}& h_i\left(F_i^a(k-1)\right),\hfill & & \mathrm{otherwise},\hfill \\ & \widetilde{F_i},\hfill & & \mathrm{if}{\zeta }_{\mathrm{i}}^{\mathrm{a}}\left(\mathrm{k}\right)=1.\hfill \end{array}\right.$$

(17)

Therefore,

$$\begin{array}{ccc}\hfill \mathbb{E}\left[F_i^e\left(k\right)\right]& =& \hfill {\lambda }_i^e\left(k\right){\tilde{F}}_i+(1-{\lambda }_i^e\left(k\right))h_i\left(F_i^e(k-1)\right),\hfill \end{array}$$

(18)

$$\begin{array}{ccc}\hfill \mathbb{E}\left[F_i^a\left(k\right)\right]& =& \hfill {\lambda }_i^a\left(k\right){\tilde{F}}_i+(1-{\lambda }_i^a\left(k\right))h_i\left(F_i^a(k-1)\right).\hfill \end{array}$$

(19)

Define S ≜$\{{\tilde{F}}_i,h_i\left({\tilde{F}}_i\right),h_i^2\left({\tilde{F}}_i\right)...\}$, it is composed of all possible values of $F_i^e\left(k\right)$ and $F_i^a\left(k\right)$.

2.4. Problem Formulation

Specifically, we consider the following problem: from the perspective of the eavesdropper, in an infinite time horizon, the eavesdropper finds the optimal attack allocation under the condition of limited energy to maximize the state estimation error of the eavesdropper, i.e.,

 Problem 1.

$$\begin{array}{ccc}\hfill & & \hfill \underset{\theta }{max}J\left(\pi \right),\hfill \end{array}$$

(20)

$$\begin{array}{ccc}\hfill s.t.& & \hfill \sum _{i=1}^N{a}_i\left(k\right)\le \overline{a}\hfill \end{array}$$

(21)

where $\pi =\{(a_1\left(1\right),a_2\left(1\right),\dots ,a_N\left(1\right)),(a_1\left(2\right),a_2\left(2\right),\dots ,a_N\left(2\right)),\dots \}$ is an admissible attack policy with the attack power using a time-step $1,2,....$. $\overline{a}$ is an energy constraint for the attacker at each time. The infinite horizon average expectation of the remote estimation error covariance can be expressed by the following formula:

$$\begin{array}{c}\hfill J\left(\pi \right)=\underset{T\to \infty }{liminf}\frac{1}{T+1}\left[\mathbb{E}\sum _{k=0}^T\sum _{i=1}^{N}tr\left(F_i^a\left(k\right)\right)\right]\end{array}$$

(22)

3. Optimal Attack Schedule

In this section, we formulate Problem 1 as a discrete time MDP to solve. In addition, we also give an algorithm for searching the optimal eavesdropping attack strategy.

3.1. MDP Formulation

For the convenience of notation, denote ${\tau }_i^e\left(k\right)$ (or ${\tau }_i^a\left(k\right)$) as the holding time from the estimator (or eavesdropper) to the continuous successful acquisition of data at time k, that is, the duration from the last successful transmission time to time k, which can be expressed by the following formula:

$$\begin{array}{ccc}\hfill {\tau }_i^e\left(k\right)& =& \hfill k-max\{k^{*}:{\zeta }_i^e\left(k^{*}\right)=1,0\le k^{*}\le k\},\hfill \end{array}$$

(23)

$$\begin{array}{ccc}\hfill {\tau }_i^a\left(k\right)& =& \hfill k-max\{k^{*}:{\zeta }_i^a\left(k^{*}\right)=1,0\le k^{*}\le k\}.\hfill \end{array}$$

(24)

Obviously, ${\tau }_i^e\left(k\right)\in {\mathcal{S}}_i^e=\mathbb{N}$ (or ${\tau }_i^a\left(k\right)\in {\mathcal{S}}_i^a=\mathbb{N}$), then we can get:

$$\begin{array}{ccc}\hfill F_i^e\left(k\right)& =& \hfill h^{{\tau }_i^e\left(k\right)}\left(\tilde{F}\right),\hfill \end{array}$$

(25)

$$\begin{array}{ccc}\hfill F_i^a\left(k\right)& =& \hfill h^{{\tau }_i^a\left(k\right)}\left(\tilde{F}\right).\hfill \end{array}$$

(26)

Using MDP to describe the dynamic process of CPS under eavesdropping attacks, MDP is expressed mathematically as $\langle \mathcal{S},\mathcal{A},\mathbb{P},r(·)\rangle$, and the specific elements are as follows.

State space: let $s_i\left(k\right)=({\tau }_i^e(k-1),{\tau }_i^a(k-1))\in {\mathcal{S}}_i^e\times {\mathcal{S}}_i^a={\mathbb{N}}^2$, where ${\tau }_i^e(k-1)$ and ${\tau }_i^a(k-1)$ can be considered as the state of process i at time $k-1$ at the remote estimator side and eavesdropper side, respectively. The state at time k is defined as $s\left(k\right)=(s_1\left(k\right),s_2\left(k\right),\dots ,s_N\left(k\right))$, and its value range is a countable state space ${\mathcal{S}}_i\triangleq {\mathcal{S}}_i^e\times {\mathcal{S}}_i^a$. Let $\mathcal{S}=\{{\mathcal{S}}_1,\dots ,{\mathcal{S}}_N\}$.

Action space: we can know the action space is defined as $\mathcal{A}\triangleq \{a_1\left(k\right),a_2\left(k\right),\dots ,a_N\left(k\right)\}$, where $a_i\left(k\right)\in (0,a_i^{\left(1\right)},a_i^{\left(2\right)},\dots ,a_i^{\left(l_i\right)})$, $i\in \{1,2,\dots ,N\}$, $a_i^{\left(l_i\right)}$ is the maximum attack power to channel i. Thus, the action is $a\left(k\right)\triangleq \{a_1\left(k\right),a_2\left(k\right),\dots ,a_N\left(k\right)\}\in \mathcal{A}$.

Transition probability: let the state transition introduction matrix at time k be ${\mathbb{P}}_i\left(s_i(k+1)\right|s_i\left(k\right),a_i\left(k\right))$, which represents the probability of the state changing from $s_i\left(k\right)$ to $s_i(k+1)$ under action $a_i\left(k\right)$, where $s_i\left(k\right),s_i(k+1)\in \mathcal{S}$, $a_i\left(k\right)\in \mathcal{A}$. For simplicity, let the state at time k be $s_i\left(k\right)=(j_1,j_2)$. Then, the state transition probability matrix is as follows:

$$\begin{gathered} \begin{array}{c}{\mathbb{P}}_i\left(s_i(k+1)\right|s_i\left(k\right),a_i\left(k\right))=\hfill \end{array} \\ \left\{\begin{array}{cccc}& {\lambda }_i^e\left(k\right){\lambda }_i^a\left(k\right),\hfill & & \mathrm{if}{\mathrm{s}}_{\mathrm{i}}(\mathrm{k}+1)=(0,0),\hfill \\ & {\lambda }_i^e\left(k\right)(1-{\lambda }_i^a\left(k\right)),\hfill & & \mathrm{if}{\mathrm{s}}_{\mathrm{i}}(\mathrm{k}+1)=(0,{\mathrm{j}}_2+1),\hfill \\ & (1-{\lambda }_i^e\left(k\right)){\lambda }_i^a\left(k\right),\hfill & & \mathrm{if}{\mathrm{s}}_{\mathrm{i}}(\mathrm{k}+1)=({\mathrm{j}}_1+1,0),\hfill \\ & (1-{\lambda }_i^e\left(k\right))(1-{\lambda }_i^a\left(k\right)),\hfill & & \mathrm{if}{\mathrm{s}}_{\mathrm{i}}(\mathrm{k}+1)=({\mathrm{j}}_1+1,{\mathrm{j}}_2+1).\hfill \end{array}\right. \end{gathered}$$

(27)

Payoff functions: let $r(·)$ be the immediate cost function and define it as:

$$\begin{array}{ccc}\hfill r\left(s\right(k),a(k\left)\right)& \triangleq & \hfill \sum _{i=1}^{N}Tr\left(F_i^a\left(k\right)\right)\hfill \end{array}$$

(28)

Obviously, the single-stage reward at time k is independent of the action behavior and only depends on the current state.

Note that the random decision rule of the eavesdropper is a mixed strategy sequence $\pi =\{(a_1\left(1\right),a_2\left(1\right),\dots ,a_N\left(1\right)),(a_1\left(2\right),a_2\left(2\right),\dots ,a_N\left(2\right)),\dots \}$, where $\pi$ is the random kernel from $\mathcal{H}$ to $\mathcal{A}$ and definition $\Pi$ is the set of all these feasible strategies. Based on the process state $s\left(k\right)$, the attacker chooses action $a\left(k\right)=a\left(s\right(k\left)\right)$, $\pi =\left\{\right(a\left(s\right(1\left)\right),\dots ,a\left(s\right(k\left)\right),\dots \}$. Then, for the initial state $s\left(0\right)=s\in \mathcal{S}$, we can get the sum of expected reward $r\left(k\right)$ following the action strategy $\pi \in \Pi$:

$$J(s,\pi )=\underset{T\to \infty }{liminf}\frac{1}{T+1}\mathbb{E}[\sum _{k=0}^{T}r(s\left(k\right),a\left(k\right))]$$

(29)

and its optimal value $J^{*}\left(s\right)$ is

$$J^{*}\left(s\right)=arg\underset{\pi \in \Pi }{max}J(s,\pi ).$$

(30)

Define the average value function under policy $\pi \in \Pi$ as the function V: $\mathcal{S}\to \mathbb{R}$. Therefore, we can get the following theorem.

 Theorem 1.

According to the MDP theory, we can obtain the optimal value $J^{*}\left(s\right)$ by solving the following optimality (Bellman) equation:

$$\begin{array}{cc}\hfill J^{*}\left(s\right)+V\left(s\right)& =\underset{\mathit{a}\left(\mathit{k}\right)\in \mathcal{A}}{max}\{r(s,a)\hfill \\ & +\sum _{\mathit{s}(\mathit{k}+\mathbf{1})\in \mathcal{S}}\prod _{i=1}^N{\mathbb{P}}_i\left(s_i(k+1)\right|s_i\left(k\right),a_i\left(k\right))V\left(s(k+1)\right)\}\hfill \end{array}$$

(31)

where s = $(s_1,\dots ,s_N)\in \mathcal{S}$ is the initial state.

The optimal attack strategies of the eavesdropper is:

$$\begin{array}{cc}\hfill a^{*}\left(s\left(k\right)\right)& =arg\underset{\mathit{a}\left(\mathit{k}\right)\in \mathcal{A}}{max}\{r(s\left(k\right),a\left(k\right))\hfill \\ & +\sum _{\mathit{s}(\mathit{k}+\mathbf{1})\in \mathcal{S}}\prod _{i=1}^N{\mathbb{P}}_i\left(s_i(k+1)\right|s_i\left(k\right),a_i\left(k\right))J^{*}\left(s(k+1)\right)\}\hfill \end{array}$$

(32)

Proof (Proof of Theorem 1). 

According to the eighth chapter in reference [37], Theorem 1 can be obtained by introducing our state transition probability matrix (27) and immediate cost function (28).

From Equation (8.4.2) in [37], we can get the following equation:

$$\begin{array}{cc}\hfill J^{*}\left(s\right)+V\left(s\right)& =\underset{\mathit{a}\left(\mathit{k}\right)\in \mathcal{A}}{max}\{r_{{\pi }_1}+\sum _{k=1}^{\infty }\prod _{i=1}^{k-1}{\mathbb{P}}_{{\pi }_i}{r}_{{\pi }_k}\}\hfill \\ & =\underset{\mathit{a}\left(\mathit{k}\right)\in \mathcal{A}}{max}\{r_{{\pi }_1}+{\mathbb{P}}_{{\pi }_1}{r}_{{\pi }_2}+{\mathbb{P}}_{{\pi }_1}{\mathbb{P}}_{{\pi }_2}{r}_{{\pi }_3}+···\}\hfill \\ & =\underset{\mathit{a}\left(\mathit{k}\right)\in \mathcal{A}}{max}\{r_{{\pi }_1}+{\mathbb{P}}_{{\pi }_1}[r_{{\pi }_2}+{\mathbb{P}}_{{\pi }_2}{r}_{{\pi }_3}+···]\}\hfill \\ & =\underset{\mathit{a}\left(\mathit{k}\right)\in \mathcal{A}}{max}\{r_{{\pi }_1}+{\mathbb{P}}_{{\pi }_1}V\left(s(k+1)\right)\}\hfill \end{array}$$

where ${\pi }_k$ is the strategy of the time k. r and $\mathbb{P}$ are abbreviations. Many decision rules are contained in historical strategies. So, $r_{{\pi }_1}$ and ${\mathbb{P}}_{{\pi }_1}$ can be decomposed into the following formula:

$$\begin{array}{ccc}\hfill r_{{\pi }_1}& =& r(s,a),\hfill \\ \hfill {\mathbb{P}}_{{\pi }_1}& =& \sum _{\mathit{s}(\mathit{k}+\mathbf{1})\in \mathcal{S}}\prod _{i=1}^N{\mathbb{P}}_i\left(s_i(k+1)\right|s_i\left(k\right),a_i\left(k\right)).\hfill \end{array}$$

(33)

Therefore, we can get the following:

$$\begin{array}{cc}\hfill J^{*}\left(s\right)+V\left(s\right)& =\underset{\mathit{a}\left(\mathit{k}\right)\in \mathcal{A}}{max}\{r_{{\pi }_1}+{\mathbb{P}}_{{\pi }_1}V\left(s(k+1)\right)\}\hfill \\ & =\underset{\mathit{a}\left(\mathit{k}\right)\in \mathcal{A}}{max}\{r(s,a)\hfill \\ & +\sum _{\mathit{s}(\mathit{k}+\mathbf{1})\in S}\prod _{i=1}^N{\mathbb{P}}_i\left(s_i(k+1)\right|s_i\left(k\right),a_i\left(k\right))V\left(s(k+1)\right)\}\hfill \end{array}$$

(34)

Then, we rewrite the finite-horizon optimality Equation (4.5.1) in [37] as

$$\begin{array}{cc}\hfill a^{*}\left(s\left(k\right)\right)& =arg\underset{\mathit{a}\left(\mathit{k}\right)\in \mathcal{A}}{max}\{r(s\left(k\right),a\left(k\right))\hfill \\ & +\sum _{\mathit{s}(\mathit{k}+\mathbf{1})\in \mathcal{S}}\prod _{i=1}^N{\mathbb{P}}_i\left(s_i(k+1)\right|s_i\left(k\right),a_i\left(k\right))J^{*}\left(s(k+1)\right)\}\hfill \end{array}$$

(35)

Thus, we can get the optimality (Bellman) Equation (31) and the optimal attack strategies of the eavesdropper (32). So, Theorem 1 is proved.    □

 Remark 1.

It should be noted that for finite MDP, the action $a\left(k\right)$ taken at time k is non-stationary and depends on the current state at time k.

 Remark 2.

We can get the optimal attack energy allocation strategy of (29) by using the optimality (Bellman) Equation (31); in addition, the optimal strategy is statically deterministic, which helps us to find out the structural characteristics of the optimal allocation strategy.

3.2. Policy Iteration Algorithm

MDP proposed in this paper has infinite state space. However, according to the characteristics of state transition in the system model, we can find that when the eavesdropper’s attack energy is limited, the transition rule can effectively limit the system state in a limited time range. Therefore, in the MDP proposed in this paper, although it has infinite state space, we can treat it as an MDP with a finite time domain. This is convenient for us to design the algorithm of the optimal attack strategy.

In a finite time domain, the solution of the optimal equation is the optimal quality function from the decision time k to the decision time T at the end of the process. Based on the MDP problem constructed above, we provide a specific backward induction algorithm to solve it and provide the optimal attack strategy, i.e., Algorithm 1.

In Algorithm 1, we first calculate ${\tilde{F}}_i$, the packet rate ${\lambda }_i^e\left(k\right),{\lambda }_i^a\left(k\right)$ and the hold time ${\tau }_i^e\left(k\right)$, ${\tau }_i^a\left(k\right)$ in step 1 and calculate the state transition matrix ${\mathbb{P}}_i$ in step 2. Then, in step 3, set all $k=0$ and for all $s\left(k\right)\in \mathcal{S}$, compute $J^{*}\left(s\right)+V\left(s\right)$ by (31). Next, in step 4, we set $k\leftarrow k-1$, initialize $s\left(k\right)=s(k+T)$. In step 5, let $s\left(k\right)\in \mathcal{S}$, compute $J^{*}\left(s\right)$ by (31) and $a^{*}\left(s\left(k\right)\right)$ by (32). We assess that the best action 1 is found for state $s\left(k\right)$, so in step 6, if $a^{*}\left(s\left(k\right)\right)=1$, then for all $s(k+1)\in \mathcal{S}$, let (34). After this, let $s\left(k\right)\leftarrow s(k+T)$, and go to Step 5. Otherwise, let $s\left(k\right)\leftarrow s(k+1)$, go to Step 5. Finally, in step 7, if $k=0$, output $J_T^{*}\left(s\right)$ and ${\pi }^{*}=\left(a^{*}\left(0\right),a^{*}\left(1\right),...,a^{*}(T-1)\right)$. Otherwise, go to Step 4.

Algorithm 1 Backward induction algorithm for optimal allocation strategy

Require: $A_i,C_i,Q_i,R_i,p_i\left(k\right),{\sigma }_e,{\sigma }_a,\overline{a},T,\mathcal{S},\mathcal{A},s$.

Ensure: The optimal value $J_T^{*}\left(s\right)$; optimal deterministic Markov policy ${\pi }^{*}$

Step 1: Calculate ${\tilde{F}}_i$, the packet rate ${\lambda }_i^e\left(k\right),{\lambda }_i^a\left(k\right)$ and the holding time ${\tau }_i^e\left(k\right)$, ${\tau }_i^a\left(k\right)$

Step 2: Calculate state transition matrix ${\mathbb{P}}_i$

Step 3: Set all $k=0$ and for all $s\left(k\right)\in \mathcal{S}$, compute $J^{*}\left(s\right)+V\left(s\right)$ by (31).

Step 4: Set $k\leftarrow k-1$, initialize $s\left(k\right)=s(k+T)$.

Step 5: Let $s\left(k\right)\in \mathcal{S}$, compute $$\begin{array}{cc}\hfill J^{*}\left(s\right)+V\left(s\right)& =\underset{\mathit{a}\left(\mathit{k}\right)\in \mathcal{A}}{max}\{r(s,a)\hfill \\ & +\sum _{\mathit{s}(\mathit{k}+\mathbf{1})\in \mathcal{S}}\prod _{i=1}^N{\mathbb{P}}_i(s_i(k+1)\hfill \\ & |s_i\left(k\right),a_i\left(k\right)\left)V\left(s(k+1)\right)\right\}\hfill \end{array}$$ $$\begin{array}{cc}\hfill a^{*}\left(s\left(k\right)\right)& =arg\underset{\mathit{a}\left(\mathit{k}\right)\in \mathcal{A}}{max}\{r(s\left(k\right),a\left(k\right))\hfill \\ & +\sum _{\mathit{s}(\mathit{k}+\mathbf{1})\in \mathcal{S}}\prod _{i=1}^N{\mathbb{P}}_i\left(s_i\right(k+1\left)\right|s_i\left(k\right),a_i\left(k\right))J^{*}(s(k+1))\}.\hfill \end{array}$$

Step 6: If $a^{*}\left(s\left(k\right)\right)=1$, then, for all $s(k+1)\in \mathcal{S}$, let $$\begin{array}{c}\hfill \begin{array}{c}\hfill a^{*}\left(s(k+1)\right)\\ \hfill =1,J^{*}\left(s(k+1)\right)+V\left(s(k+1)\right)=\\ & \hfill r\left(s\right(k+1),1)& +\sum _{\mathit{s}(\mathit{k}+\mathbf{2})\in \mathcal{S}}\prod _{i=1}^N{\mathbb{P}}_i\left(s_i(k+2)\right|s_i(k+1),1)V\left(s(k+2)\right)\hfill \end{array}\end{array}$$ (36)

let $s\left(k\right)\leftarrow s(k+T)$, and go to Step 5. Otherwise, let $s\left(k\right)\leftarrow s(k+1)$, go to Step 5.

Step 7: If $k=0$, then output $J_T^{*}\left(s\right)$ and ${\pi }^{*}=\left(a^{*}\left(0\right),a^{*}\left(1\right),...,a^{*}(T-1)\right)$. Otherwise, go to Step 4.

 Remark 3.

In the above Algorithm 1, it is assumed that $a^{*}\left(s\left(k\right)\right)=1$ in order to reduce the calculation cost and complexity compared with the traditional algorithm.

 Remark 4.

We can derive the state estimation of the eavesdropper each time to ensure the feasibility of Algorithm 1.

4. Discussion and Illustrative Example

In this section, according to the above MDP model, we can provide some numerical simulations to show the optimal attack energy allocation strategy of problem 1. Consider systems $\left(1\right)$ and $\left(2\right)$ with $\beta =0.5$. The parameters of the systems and channels are shown in

Table 1. Parameters for sensors and attack power.

Sensor 1								Sensor 2
Process 1				Attack Power				Process 2				Attack Power
$A_1$	$C_1$	$Q_1$	$R_1$	$a_1^{\left(1\right)}$	$a_1^{\left(2\right)}$	$a_1^{\left(3\right)}$	$a_1^{\left(4\right)}$	$A_2$	$C_2$	$Q_2$	$R_2$	$a_2^{\left(1\right)}$	$a_2^{\left(2\right)}$	$a_2^{\left(3\right)}$	$a_2^{\left(4\right)}$
1.8	1.5	0.8	1	0	5	7	10	1.2	1.0	0.9	0.8	0	3	5	10

.

Suppose the energy constraint is $\overline{a}=10$. In Table 1, $a_1^{\left(1\right)}=0,a_1^{\left(2\right)}=5,$$a_1^{\left(3\right)}=7,a_1^{\left(4\right)}=10$ indicates that the possible maximum attack power of channel 1 is 0, 5, 7 and 10, respectively, and another 0 means that there is no attack. $a_2^{\left(1\right)}=0,$$a_2^{\left(2\right)}=3,a_2^{\left(3\right)}=5,a_2^{\left(4\right)}=10$ in the same way. According to the number of existing systems and the number of channels that eavesdrop at the same time, we can divide them into the following three methods, such as numerical simulation. We use the Algorithm 1 to calculate the optimal attack energy allocation strategy and the optimal average return.

4.1. Single System

Let us first consider the case where there is only one system. When there is only one system, we consider the optimal attack energy allocation strategy when attacking this channel at different times. We can use the data in Table 1 about Sensor 1. The strategy set for the eavesdropper is $\left\{\right(0,0),(10,0),(0,10),(5,5\left)\right\}$, where $(0,0)$ means no attack, $(10,0)$ means use energy 10 to attack at the first moment and not to attack at the second moment and $(0,10)$ and $(5,5)$ have a similar meaning. Assume that the transmission power of the sensor is $p\left(k\right)=0.6$. Through the calculation, the error covariance of the steady state estimation is obtained, i.e., $\tilde{F}=1.9755$. Assume that the AWGN power of the channel between the sensor and the estimator is ${\sigma }_e^2=0.3$, and the AWGN power of the channel between the sensor and the eavesdropper is ${\sigma }_a^2=0.5$. This paper studies the infinite time domain, but in order to simplify the calculation, we use the truncated set ${\mathcal{N}}_0=\{0,1,...,16\}$. The optimal strategy is calculated through Algorithm 1.

In Figure 2, we use ${\tau }^e\left(k\right)$ and ${\tau }^a\left(k\right)$ to represent the state of the optimal strategy and the optimal strategy is shown, where the purple and red symbols represent policy 1 (policy 1 = (0, 0)) and policy 2 (policy 2 = (0, 10)).

4.2. Dual System (Not Attacking or Attacking One Channel)

Consider that when there are two systems, neither of the two channels can attack or can only attack one channel.In the numerical solution, we consider the data in Table 1 and

Table 2. AWGN power and Transmission power.

AWGN Power		Transmission Power
${\sigma }_e^2$	${\sigma }_a^2$	$p_1\left(k\right)$	$p_2\left(k\right)$
0.1	0.2	1	0.8

and use a truncated set $h_{\left(i\right)}^{35}\left({\tilde{F}}_{\left(i\right)}\right)$. We can assume that the eavesdropper’s strategy set is $\left\{\right(0,0),(5,0),(7,0),(10,0),(0,3),(0,5),(0,10\left)\right\}$, i.e.,

Table 3. Attack power levels. Dual system (not attacking or attacking one channel).

Not Attack	To Channel 1			To Channel 2
	$(a_1^{\left(1\right)},0)$	$(a_1^{\left(2\right)},0)$	$(a_1^{\left(3\right)},0)$	$(0,a_2^{\left(1\right)})$	$(0,a_2^{\left(2\right)})$	$(0,a_2^{\left(3\right)})$
(0,0)	$(5,0)$	$(7,0)$	$(10,0)$	$(0,3)$	$(0,5)$	$(0,10)$

. Where $(0,0)$ indicates that there is no attack on both channels. $(5,0),(7,0),(10,0)$ means that channel 1 is attacked with energy 5, 7 and 10, respectively, and channel 2 is not attacked; $(0,3),(0,5),(0,10)$ has the same meaning. Through our calculations, we can get ${\tilde{F}}_1=1.9755$, ${\tilde{F}}_2=1.6805$. ${\Phi }_i$ and ${\Psi }_i$ we take as random numbers.

The optimal attack energy allocation strategy when there are two systems, neither of which attack or only one attacks, is shown in Figure 3 and Figure 4. In Figure 3, with an increase of $s_1\left(k\right)$, the optimal strategies cannot be clearly identified. Therefore, we use ${\tau }^e\left(k\right)$ and ${\tau }^a\left(k\right)$ in Figure 4 to represent the state of the optimal strategies. From Figure 4, we can see that the optimal action includes $(0,10)$ and $(10,0)$.

4.3. Dual System (Not Attacking, Attacking One Channel or Attacking Two Channels)

Similarly, in this section, two systems are also considered, and we can consider three situations: no attack, only one channel attack and two channels attack at the same time. We can assume that the eavesdropper’s strategy set is $\left\{\right(0,0),(5,0),(10,0),(0,5),(0,10),(5,5),(7,3\left)\right\}$, i.e.,

Table 4. Attack power levels. Dual system (not attacking, attacking one channel or attacking two channels).

Not Attack	Attacking One Channel				Attacking Two Channels
(0,0)	To Channel 1		To Channel 2
	$(a_1^{\left(1\right)},0)$	$(a_1^{\left(3\right)},0)$	$(0,a_2^{\left(2\right)})$	$(0,a_2^{\left(3\right)})$	$(a_1^{\left(1\right)},a_2^{\left(2\right)})$	$(a_1^{\left(2\right)},a_2^{\left(1\right)})$
	$(5,0)$	$(10,0)$	$(0,5)$	$(0,10)$	$(5,5)$	$(7,3)$

. Where $(0,0)$ indicates that there is no attack on both channels. $(5,0),(10,0)$ means that channel 1 is attacked with energy 5 and 10, respectively, and channel 2 is not attacked; $(0,5),(0,10)$ has the same meaning. $(5,5)$ represents an attack on channel 1 and channel 2 using energy 5 and energy 5, respectively; $(7,3)$ has the same meaning. In the numerical solution, we consider the data in Table 1 and Table 2 and use truncated set $h_{\left(i\right)}^{35}\left({\tilde{F}}_{\left(i\right)}\right)$. ${\Phi }_i$ and ${\Psi }_i$ we take as random numbers. Through our calculations, we can get ${\tilde{F}}_1=1.9755$, ${\tilde{F}}_2=1.6805$.

Figure 5 and Figure 6 show the optimal attack energy allocation strategy when there are two systems, in which two channels are not attacked, only one channel is attacked and both channels are attacked. In Figure 5, with an increase of $s_1\left(k\right)$, the optimal strategies cannot be clearly identified. Therefore, we use ${\tau }^e\left(k\right)$ and ${\tau }^a\left(k\right)$ in Figure 6 to represent the state of the optimal strategies. From Figure 6, we can see that the optimal action includes $(0,10)$, $(5,5)$, $(7,3)$ and $(10,0)$.

5. Conclusions

The optimal attack energy allocation of multi-system remote state estimation in CPS is studied when eavesdroppers exist in an infinite time domain. Based on the channel SINR, a wireless communication model is constructed. When the eavesdropper’s energy is limited, the optimal value of the eavesdropper and the optimal attack energy allocation strategy are found by using MDP theory. Finally, the research results are numerically simulated. According to the theoretical analysis and the numerical analysis, we can draw the following conclusions: in a multi-system, the optimal energy allocation strategy is to choose a higher attack energy to attack the channel when the estimation error covariance at the eavesdropper is large, and choose a lower attack energy to attack the channel when the estimation error covariance is small. And, we can see that the optimal attack strategy has an obvious threshold structure. In the future, we will prove the threshold structure of the optimal allocation strategy and study the situation when the detector exists.