Adversarial Attacks and Defenses in Deep Learning: From a Perspective of Cybersecurity

2.2.1 Adversary’s Goal.

• Poisoning Attacks. In poisoning attacks, the attackers can access and modify the training dataset to impact the final trained models [23, 24, 25, 26]. The way that the attackers inject fake samples into the training data to generate a defective model can be viewed as “poisoning.” Poisoning attacks generally lead to a decrease of accuracy [25] or misclassification on the given test samples [26].

• Evasion Attacks. In evasion attacks, the adversary’s goal is to attack a well-trained and fixed DNN without any authority to modify the parameters of the target models [13, 18, 23]. In this way, the accessibility of the training dataset is no longer needed by the attackers. Instead, the attackers generate deceptive test samples that the target models fail to recognize to evade final detection [27, 28].

2.2.2 Adversarial Specificity.

The difference in adversarial specificity depends on whether the attacker can predefine a specific fraudulent prediction for a given adversarial sample at the inference phase.

• Untargeted Attacks. In untargeted attacks, the adversary’s only purpose is to fool the target model into generating a false prediction without caring for which label is chosen as the final output [29, 30, 31, 32].

• Targeted Attacks. In targeted attacks, for a given sample, the attacker not only wants the target model to make an incorrect prediction, but also aims to induce the model to provide a specific false prediction [33, 34, 35, 36]. Generally, targeted attacks do not succeed as often as an untargeted one.

2.2.3 Adversary’s Knowledge.

• White-box Attacks. In white-box settings, the adversary is able to access the details of the target model, including structure information, gradients, and all possible parameters [30, 34, 37]. The adversary thus can craft elaborate adversarial samples by exploiting all the information at hand.

• Black-box Attacks. In black-box settings, attackers implement attacks without any knowledge of the target model. An attacker can acquire some information by interacting with the target model [36, 38, 39]. This is done by feeding input query samples into the model and analyzing the corresponding outputs.

2.3.1 Evasion Attacks.

Szegedy et al. [2] first introduced the concept of adversarial example in adversarial attacks, which can mislead the target model with a high success rate in the inference phase. They proposed a method to search for minimal distorted adversarial examples with the targeted label through Equation (1): (1) \(\begin{equation} \begin{aligned}&{\rm minimize}\ \left\Vert x^{\prime }-x\right\Vert ^2_2\ {\rm subject\ to}\ f(x^{\prime })=t\ and\ x^{\prime }\in \left[ 0,1\right] ^m\!. \end{aligned} \end{equation}\)

Through this equation, they can find the closest \(x^{\prime }\) that has a minimal distance with benign sample x by minimizing \(\left\Vert x^{\prime }-x\right\Vert ^2_2\) and would be misclassified as targeted label t by the condition of \(f(x^{\prime })=t\). This problem can lead to the objective in Equation (2), which can be solved by L-BFGS algorithms: (2) \(\begin{equation} \begin{aligned}&{\rm minimize}\ c\left\Vert x^{\prime }-x\right\Vert ^2_2+{\mathcal {L}}(f(x^{\prime }),t)\ \ {\rm subject\ to}\ x^{\prime }\in \left[ 0,1\right] ^m\!. \end{aligned} \end{equation}\)

Evasion attacks can be loosely described as methods of crafting adversarial examples by adding imperceptible perturbations, which can result in the misbehavior of trained models.

2.3.2 Poisoning Attacks.

Unlike evasion attacks happening in the inference attacks, poisoning attacks aim to downgrade the accuracy of models by polluting the training data. Attackers need some authorities to manipulate the training data, such as data injection and data modification [11]. As a result, the goals of launching poisoning attacks can be categorized into two classes: availability violation and integrity violation. The former aims to reduce the confidence or accuracy of victim model and disrupt the entire system, while the latter tries to mislead the victim model over some specific samples by introducing a backdoor without affecting other normal samples [11]. Specifically, the poisoned instances against neural networks only involving training samples can be crafted via following two strategies: bi-level optimization and feature collision [23].

• Bi-level optimization: Classical data poisoning of modifying the data can be formalized as a bi-level optimization problem [23]. However, for non-convex neural networks, bi-level optimization problems are intractable. Mu\({\rm \tilde{n}}\)oz-Gonz\({\rm \acute{a}}\)lez et al. [24] proposed “back-gradient descent” to approximate solutions of the inner problem and then conduct gradient descent on the outer loss, although it is still computationally expensive. To speed up the production of poisoned samples, Yang et al. [25] introduced GAN to generate poisons. MetaPoison et al. [26] is also proposed as a first-order method to approximately solve the bi-level optimization of producing poisons using the ensembling strategies.

• Feature collision: Methods based on bi-level optimization usually are effective against both transfer learning and end-to-end training, while feature collision strategy can be used to design efficient attacks against transfer learning in the targeted misclassification setting. For instance, Shafahi et al. [40] developed a method to generate the poisoning sample similar to the target samples in the feature space, while it is close to the original benign sample in the input space. These two categories of methods only require permission to manipulate the training instead of the label, and the semantic of training data will remain. These poisoned samples with clean-label will be more difficult to be detected.

Moreover, in addition to poisoning attacks only manipulating training data, another type of poisoning attacks are backdoor attacks, which need the additional capacity of inserting trigger to the input in inference phase [23].

• Backdoor attacks: Adversaries in backdoor attacks usually have access to modify the label of training samples [41]. These mislabeled data with backdoor triggers will be injected into training dataset. As a result, the trained model based on this dataset will be forced to assign the new sample (with the trigger) the desired target label. Most backdoor attacks require mislabeling the training samples in the process of crafting poisons, which are more likely to be identified by defenders. Therefore, some clean-label backdoor attacks [42] are also proposed and craft the backdoor samples using the strategy of feature collision presented in Reference [40].

2.4.1 What is APT.

2.4.2 Lifecycle in APT.

To attack the target successfully, the attackers in APT need to go through a complete lifecycle with multiple stages in a sequential manner. There are several different forms of this lifecycle that vary depending on different considerations with respect to generalization or specification. We selected a representative five-stage APT lifecycle model with the goal to undermine critical infrastructure [19]. These stages are as follows.

3.3.1 Optimization-based Attacks.

As mentioned above, Szegedy et al. [2] first introduced an attack scheme, L-BFGS Attack, against DNNs in 2014. This is widely considered the first study on adversarial attacks in deep learning. Their work formulated how to craft a sample for a targeted label as a searching problem for a minimal distorted adversarial example \(x^{\prime }\). To further improve the performance of the L-BFGS method, Carlini and Wagner [33] proposed a set of optimization-based attacks, termed the Carlini and Wagner (C&W) attacks. Unlike the L-BFGS attack relying on a cross-entropy loss \({\mathcal {L}}(f(x), t)\), C&W attacks involves a margin loss \({\mathcal {L}}_m(f(x), t)\) as the loss function, which can be customized by attackers. Several different distance measures \(D(\cdot)\) including \(L_0\), \(L_2\), and \(L_\infty\) norm can be used by attackers in C&W attacks.

To reduce the size of \(L_2\) distortion and improve the imperceptibility between original samples and the adversarial samples in classification models, Moosavi-Dezfooli et al. [30] proposed an attack algorithm named DeepFool. However, few works have designed algorithms using the \(L_1\) metric to craft adversarial samples, though the \(L_1\) distortion is a distance metric that can account for the total variation. Chen et al. [56] proposed an Elastic-net attack against DNNs (EAD), which was the first to introduce the \(L_1\) norm into an adversarial attack. Their optimization problem is shown in Equation (3), where \({\mathcal {L}}(f(x^{\prime }),t)\) is the target adversarial loss function, and the additional two terms are used to minimize the perturbation in terms of \(L_1\) and \(L_2\) distance in searching: (3) \(\begin{equation} {\rm minimize}\ c\cdot {\mathcal {L}}(f(x^{\prime }),t)+\beta \left\Vert x^{\prime }-x\right\Vert _1 +\left\Vert x^{\prime }-x\right\Vert ^2_2 \ {\rm subject\ to}\ x^{\prime }\in \left[ 0,1\right] ^m\!. \end{equation}\)

The algorithms mentioned above chose the \(L_p\) norm to evaluate the perturbations. By contrast, Zhao et al. [29] used information geometry to understand the vulnerabilities of DNNs to adversarial attacks. They formalized the optimization problem as a constrained quadratic form of the Fisher Information Metric (FIM) and presented this novel attack algorithm named one-step spectral attack (OSSA) as a way of computing the optimal perturbations with the first eigenvector. Zhang et al. [57] proposed blind-spots attacks, which find some inputs that are far enough from the existing training distribution to fool the target model, because they discovered the adversarially trained network gradually loses its robustness on these data.

3.3.2 Gradient-based Attacks.

Although optimization-based L-BFGS attacks achieve high misclassification rates, an expensive linear search method is needed to find optimal hyperparameter c in Equation (2), which has a high computational cost. Thus, Goodfellow et al. [46] proposed a fast one-step method of generating adversarial perturbations called FGSM. This algorithm is described in Equation (4), where \(sign(\cdot)\) is the signum function and \(\bigtriangledown _x({\mathcal {L}}(f(x),y))\) represents the gradient of loss w.r.t. x: (4) \(\begin{equation} \begin{aligned}&\Delta x= \epsilon \cdot {\rm sign}(\bigtriangledown _x({\mathcal {L}}(f(x),y))). \end{aligned} \end{equation}\)

Because FGSM computes the perturbation with only one backpropagation step of calculating the gradient, it is much quicker at finding adversarial samples than L-BFGS attacks. However, FGSM has a low success rate. To address this shortcoming, Kurakin et al. [4] proposed an iterative version, Basic Iterative Method (BIM). To constrain the adversarial perturbations, BIM adds a clip function (Equation (5)), such that the generated sample is located in the \(\epsilon -L_\infty\) ball of the benign image, where \(x^{\prime }_i\) is the intermediate result in ith iteration, and \(\alpha\) is the size of the perturbation. In addition to BIM, Dong et al. introduced a momentum optimizer to optimize BIM, which is called momentum iterative FGSM (MI-FGSM) [58]. Madry et al. [3] presented the Projected Gradient Descent (PGD) attack. PGD replaces the Clip function in BIM with the Proj function, which is one of the strongest attacks that use the first-order information of target models. (5) \(\begin{equation} \begin{aligned}&x^{\prime }_{i+1}= {\rm Clip}\lbrace x^{\prime }_i+\alpha \cdot {\rm sign}(\bigtriangledown _x({\mathcal {L}}(f(x^{\prime }_i),y)))\rbrace \end{aligned} \end{equation}\)

Papernot et al. [34] proposed a targeted attack focusing on the perturbations under an \(L_0\) distance metric, called Jacobian-based Saliency Map Approach (JSMA). A Jacobian matrix is used to determine which element is more important for crafting effective adversarial examples. However, generated perturbations by JSMA are greater than that of DeepFool [30]. Based on the idea of DeepFool, Alaifari et al. [37] proposed a novel kind of adversarial attack, ADef, which finds “small” perturbations of the images. Due to the projection nature, iterative algorithms usually lead to large-scale distortions. Chen et al. [59] aimed to address this problem with an attack framework, Frank-Wolfe, which uses momentum mechanisms to avoid projection and leads to better distortions.

In early studies, it was common to generate attack perturbations independently for each specific input based on the loss function. However, Zheng et al. [60] proposed an algorithm called Distributionally Adversarial Attack (DAA) [60] to generate deceptive examples by introducing direct dependency between all data points, which is a variant of PGD [3] to maximally increase the generalization risk. Besides, PGD is also shown to lead to overestimation of robustness because of the sub-optimal step-size and problems of the objective loss [61]. Therefore, Croce and Hein [61] proposed a parameter-free version of PGD with an alternative objective function, Auto-PGD, avoiding the selection of step size for \(l_2\) and \(l_\infty\) perturbations. Their extensive experiments showed that Auto-PGD can decrease the accuracy of the target model under multiple existing defenses by more than 10%. In addition, when taking into account \(l_1\)-perturbations, PGD is not effective and is weaker than some state-of-the-art \(l_1\) attacks [56]. Croce and Hein [62] analyzed the reason why PGD is sub-optimal under \(l_1\) perturbations and identified the correct projection set, which is computationally feasible. Their method can encourage adversarial training to yield a more robust \(l_1\)-model with \(\epsilon =12\) when compared to the original PGD [3].

3.3.3 GAN-based Attacks.

Malicious perturbations can lead to some unnatural or not semantically meaningful examples. To craft natural adversarial samples, Zhao et al. [63] presented a framework to craft natural and legible adversarial examples using the GAN in black-box settings. A Generator \(\mathcal {G}\) on corpus \(\mathcal {X}\) and a corresponding Inverter \(\mathcal {I}\) are trained separately by minimizing the reconstruction error of x and the divergence between the sampled z and \(\mathcal {I}(\mathcal {G}(z))\). Given an instance x, they searched the perturbations with Inverter in the dense representation of \(z^{\prime } = \mathcal {I}(x)\). And then, they mapped it back to \(x^{\prime }\) with the trained Generator \(\mathcal {G}\). The perturbations from the latent low-dimensional z space can encourage these adversarial samples to be valid.

Xiao et al. [64] also proposed a GAN-based attack, AdvGAN, to generate adversarial samples with good perceptual quality efficiently. They added a loss for fooling the target model and another soft hinge loss to limit the magnitude of the perturbation. Once the generator is trained, the perturbations can be generated efficiently for any input, which can potentially accelerate some defensive methods such as Adversarial Training. Based on the architecture of Reference [64], Wei et al. [28] also proposed a GAN-based adversarial attack named Unified and Efficient Adversary (UEA) to address problems with high computation costs and the weak transferability of existing methods in image and video object detection. In their method, the generation process only involves the forward networks, so it is computationally fast. In addition, Phan et al. [65] proposed to use GAN to design black-box attacks, improving the transferability of existing attacks.

Discussion. Attack methods discussed in this section show the representative strategies of crafting adversarial examples. Despite the strong performance of optimization-based attacks, most attackers are willing to explore gradient-based attacks, because these kinds of attacks are simpler than their optimization-based counterparts. In addition, efficient attacks can be easily incorporated into defensive techniques against adversarial examples, such as adversarial training, which can increase the efficiency of defenses as an ultimate goal. However, common gradient-based methods need full knowledge of target models, which mainly consist of white-box attacks.

3.4.1 Universal Adversarial Attack.

Cross-sample transferability captures the ability of a perturbation against one benign sample to be effective against other benign instances within the same models. General methods aforementioned craft different perturbations for each single sample. Thus, it is not clear about the transferability across the benign samples. Moosavi-Dezfooli et al. [48] first showed the existence of universal adversarial perturbations and provided a systematic algorithm to create these perturbations. At each iteration, they compute the minimal perturbation according to the boundary of the current classification region and then aggregate them into the original input. When a crafted universal perturbation is added to any benign example in the training dataset, all generated adversarial examples are misclassified with a high probability. Sarkar et al. [66] proposed black-box universal adversarial attacks, which can produce targeted misclassification when compared to the initial works [48]. They used a residual generating network to produce an image-agnostic perturbation for each class to misclassify the samples with this perturbation as being from the corresponding class.

Shafahi et al. [67] proposed an efficient optimization-based method to produce the universal perturbations by solving some common problems for DNNs. Specifically, they use stochastic gradient methods to solve the optimization problem of crafting perturbations and introduce a “clipped” version of the cross-entropy loss to mitigate problems caused by unbounded cross-entropy. As a result, their methods dramatically reduce the time needed to craft adversarial examples as compared to Reference [48]. Co et al. [68] proposed to leverage procedural noise functions to generate universal adversarial perturbations. It is simpler to implement, and the smaller search space of procedural noise makes a black-box attack on large-scale applications feasible. Zhang et al. [69] reviewed existing universal adversarial attacks, discussed the challenges, and studied the underlying reasons for why universal adversarial perturbations exist.

3.4.2 Transfer-based Attacks.

So far, white-box attacks are based on the assumption that the adversary has access to information such as input data, model structure, gradients, and so on. However, in most scenarios, attackers have little information about models except the input-output pairs. The target models can only be used in a black-box manner. So, black-box attacks are far more common. Transfer-based attacks are probably the most common methods of exploring the cross-model transferability and attacking a black-box target model with the help of white-box substitute models.

Szegedy et al. [2] first described the phenomenon that adversarial examples crafted carefully for one model could be transferred to other models, regardless of its structural properties, like the number of layers. Papernot et al. [39] further explored the property to study how the adversarial samples could be transferred between different machine learning techniques and proposed the first effective algorithm to fool DNN classification models in a black-box manner. They assumed that attackers have no access to the parameters of the classifiers but do have some partial knowledge of the training data (e.g., audios, images) and the expected output (e.g., classification).

To further increase the transferability of adversarial perturbations, ensemble attacks have been a category of crafting transferable perturbations for black-box models. Che et al. [36] proposed Serial-Mini-Batch-Ensemble-Attack, where they consider the process of crafting adversarial samples to be the training of DNNs, and the transferability of the adversarial examples is thought of as the model’s generalizability. Phan et al. [65] proposed a GAN-based black-box attack method, called Content-aware Adversarial Attack Generator, which improves on the low transferability of existing attacks in the black-box settings by introducing random dropout.

Domontis et al. [8] provided a comprehensive analysis of transferability for adversarial attacks. They highlighted three metrics: the magnitude of the input gradients, the gradient alignment, and the variability of the loss landscape. In addition, Sharma et al. [70] proposed another factor, perturbation’s frequency, from the perspective of perturbations instead of models. They validated adversarial examples are more transferable and can be generated faster when perturbations are constrained to a low-frequency subspace.

To address the weak transferability of black-box attacks (especially under the existing defenses), some state-of-the-art works introduced some novel techniques to improve the cross-model transferability, such as meta learning [71], variance tuning [72], and feature importance-aware (FIA) [73].

3.4.3 Query-based Attacks.

The performance of the black-box attacks can be influenced by poor transferability in transfer-based attacks using substitute models. In addition to transfer-based attacks, black-box adversaries can use some zeroth-order optimization methods to estimate numerically the gradient through a number of queries, which are denoted as query-based attacks. To avoid using the transferability, a zeroth order optimization (ZOO)-based method has been proposed that has high visual quality [74]. However, ZOO relies on the coordinate-wise gradient estimation technique, which demands an excessive number of queries on the target model. As such, it is not query-efficient and rather impractical in the real world. Tu et al. [75] proposed a generic framework for implementing query-efficient black-box attacks, termed as Autoencoder-based Zeroth Order Optimization Method (AutoZOOM). They proposed a scaled random full gradient estimator and dimension reduction techniques (e.g., autoencoder) to reduce the query counts. Ilyas et al. [76] used natural evolutionary strategies to construct an efficient unbiased gradient estimator, which requires far fewer queries than the traditional attacks based on finite-difference.

Square attacks were proposed to further improve the query efficiency and success rate of black-box adversarial attacks, which combines the classical randomized search schemes and heuristic update rule [77]. Yatsura et al. [78] argued that the performance of attacks based on random search depends on the manual tuning of the proposal distributions. Therefore, they formalize square attack as a meta-learning problem to perform automatic optimization, which can circumvent the heuristic tuning and decrease the impact of manual design.

Query-based attacks would be ineffective in real-world scenarios due to the limited information [76]. Brendel et al. [38] proposed a decision-based attack, called a Boundary Attack, which requires less information about or from the models and solely relies on the final decision. For the label-only setting, Ilyas et al. [76] also proposed a concept of discretized score to quantify how adversarial the perturbed image is, which was used to estimate the absent output scores. Likewise, Cheng et al. [79] assumed that the adversary can only observe a final hard-label decision. They reformulated the task as a real-valued optimization problem by binary search. Cheng et al. [80] also optimized their previous work [79] and directly estimated the sign of the gradient rather than the gradient itself, which reduces the number of queries significantly.

The amount of queries for query-based attacks has decreased from millions to less than a thousand [81]. Maho et al. [81] proposed a geometrical approach, SurFree, based on the decision in a black-box setting. They bypassed the usage of surrogate of the target model and estimation of the gradient. Ilyas et al. [82] introduced a framework unifying a previous black-box attack methodology. They proposed bringing gradient priors into the problem to further improve the performance of untargeted attacks. Narodytska and Kasiviswanathan [83] proposed a black-box attack in an extremely limited scenario where only a few random pixels can be modified when crafting adversarial examples. They found that a tiny number of perturbed pixels is sufficient to fool neural networks in many cases. Su et al. [84] proposed the one-pixel attack and restricted the perturbation to only one pixel. They used differential evolution to find the optimal position of the perturbation and modified its RGB value to fool the target model.

3.4.4 Model-specific Attacks.

Designing an attack that exploits transferability to invalidate more models can be viewed as a horizontal extension to underlying attacks. Beyond this, there are studies on vertical extensionsthat exploit the properties of specific models (like the structure of GNN) to increase the chance of an attack’s success. For example, first-order optimization can not be applied directly to attacks on node classification tasks using edge manipulations for GNNs because of the discrete structure of graphs. Xu et al. [47] presented an approach to generate topology attacks (i.e., edge manipulation attacks) via convex relaxation, which empowers the gradient-based attacks that can be applied to GNNs. Chang et al. [49] provided a general framework, Graph Filter Attack, to attack graph embedding models in restricted black-box setting without requiring information in Reference [47].

The indifferentiable operations in a Deep product quantization network (DPQN) lead to one of the challenges to attack DPQN-based retrieval systems. To avoid the backpropagation, Feng et al. [85] proposed to formulate the generation problem as a minimization of similarity between the original query and the adversarial query. Tsai et al. [86] proposed an attack against a special network for point cloud classification. In addition, PGD might not perform as well on the Binarized Neural Networks (BNNs) because of their discrete and non-differentiable nature. Therefore, Khalil et al. [87] formulated the generation of adversarial samples on BNNs as a mixed integer linear programming problem and proposed integer propagation to tackle the intractability.

Moreover, Chhabra et al. [88] first investigated the adversarial robustness of unsupervised learning algorithms like clustering. Through their strategy, perturbing only one sample can lead to the perturbation of decision boundaries between clusters. Reinforcement learning often adopts some self-organization techniques to develop self-managed complex distributed systems [89, 90]. Huang et al. [91] showed the impact of existing adversarial attacks on trained policies in reinforcement learning. They applied FGSM to compute adversarial perturbations for policies. Wu et al. [92] focus on adversarial attacks in reinforcement learning by training an adversarial agent to effectively exploit the vulnerability of the victim without manipulating the environment.

Discussion. There are two challenges in designing adversarial examples for DNNs, limited knowledge and special properties of model. Universal adversarial perturbations usually generalize well across different classification models [48, 68], which can also be used to address limited knowledge. Though transfer-based attacks do not rely on the detailed information of models, the adversary needs to have some partial knowledge of the training data. Transfer-based attacks are prone to suffer from low success rates due to the lack of adjustment procedures for information from surrogate models. Query-based attacks usually achieve higher success rates while they are likely to lead to an enormous number of queries [38, 75, 79, 80]. P-RGF [93] combines transfer-based methods and query-based methods. The transfer-based prior from the surrogate model is utilized to query the target model efficiently, which simultaneously guarantees the attack success rates and query efficiency.

3.5.1 Image Domain and Video Domain.

Eykholt et al. [50] extended an existing algorithm for image classification to object detection domain. They modified the adversarial loss functions to minimize the probability of the target object appearing in the scene. Zhang et al. [32] conducted an experimental study to attacking object detectors for vehicles in the physical world. They tried to learn a camouflage pattern and painted the pattern on the vehicles, finding it could hide the vehicles effectively. Considering the limitations of static adversarial patches, Lovisotto et al. [94] proposed a novel method of generating physically robust real-world adversarial examples through a projector, which can enhance the robustness of patch-based adversarial examples by increasing the non-printability score.

Sensors are fundamental to the perception system of Autonomous Vehicles. Unlike camera-based perception, only a few papers touch on the feasibility of adversarial attacks on the sensor inputs of LiDAR perception. Cao et al. [35] conducted the first study on the security of LiDAR-based perception against adversarial perturbations by formulating adversarial attacks as an optimization problem. Hamdi et al. [95] also demonstrated that semantic attacks, including changes in camera viewpoint and lighting conditions, are more likely to occur naturally in autonomous navigation applications. Thereby, they proposed a semantic attack based on GAN, treating the process of mapping the parameters into environments as a black-box function.

Zhao et al. [1] provided systematic solutions to craft robust adversarial perturbations for practical object detectors at longer distances and wider angles. Wei et al. [101] focused on the adversarial samples in the video domain, which differs from the images domain given the temporal nature of videos. They leveraged the temporal information to improve the attacking efficiency and proposed the concept of propagating perturbations. A heuristic algorithm to further improve the efficiency of the method is in Reference [79].

3.5.2 Text Domain.

Liang et al. [96] applied adversarial attacks to DNN-based text classification. Like FGSM, they also leveraged a cost gradient to generate the adversarial examples, while keeping the text readable. They identified important text items like hot training phrases according to the gradients and proposed three strategies, including insertion, modification, and removal, to manipulate these important items. Finlayson et al. [18] reviewed the adversarial behaviors in medical billing industry, illustrating the influences on fraud detectors for medical claims.

The peculiarities like code layout in the code usually can be used in the tasks to identify authorship information, also called authorship attribution. Quiring et al. [97] proposed the first black-box adversarial attack to forge the coding style by combining compiler engineering and adversarial learning. Other than authorship attribution, it is more difficult to generate robust adversarial samples in source code processing tasks due to the constraints and discrete nature of the source domain. Zhang et al. [98] treated adversarial attacks against code processing as a sampling problem and proposed the Metropolis-Hastings Modifier algorithm, which can craft a sequence of adversarial samples of source code with a high success rate.

3.5.3 Audio Domain.

Yakura and Sakuma [51] proposed a special over-the-air condition to describe the difficulties of attacking practical Automatic Speech Recognition (ASR) systems, where the audio adversarial sample is played by the speaker and recorded by a device. In this scenario, such attacks can be impacted by reverberation and noise from the environment. Hence, they simulated the transformations caused by replaying the audio and incorporated them into adversarial audio samples. However, several hours are needed to craft just one adversarial sample. Liu et al. [99] proposed weighted-sampling adversarial audio examples, which can be computed at the minute level.

Zhang et al. [100] focused on the non-negligible noise introduced by previous works attacking ASR like Reference [51]. This noise can influence the quality of the original audio and reduce the robustness against a defensive detector by breaking temporal dependency properties. They proposed to extract Mel Frequency Cepstral Coefficient features of audio instances. For some ASR tasks with combinatorial non-decomposable loss functions, gradient-based adversarial attacks are not amenable for them [103]. Usually, a differentiable surrogate loss function is required in this case, while the poor consistency might affect the effectiveness of adversarial examples significantly. Houdini [103] is proposed to be tailored for these task losses to generate effective adversarial examples with high transferability, which can be used in the black-box scenario.

Discussion. In the practical tasks based on DNNs, it would be harder to implement adversarial attacks, because various domain-specific peculiarities and challenges have to be considered. Compared to conventional image classifiers, contextual information and location information in object detection can be used to prevent mispredictions [50]. Some factors in the diverse environments also affect the success rate of attacks, including noise and reverberation from playback in ASR [51, 100], changes in camera viewpoint and lighting conditions [95]. Likewise, generating samples of source code would also encounter rigid lexical and syntactical constraints [97].

4.3.1 Adversarial Training Techniques.

Adversarial Training is a simple and common method of decreasing the test error for adversarial examples by incorporating crafted examples into the training data. Goodfellow et al. [46] proposed an adversarial training method, where adversarial samples are generated using FGSM and then injected them into the training dataset. Adversarial training can promote regularization for DNNs. However, although these adversarially trained models have robustness against one-step attacks, they are still easy to be fooled by iterative attacks. Madry et al. [3] subsequently proposed adversarial training with adversarial examples crafted by PGD attacks. They focused on the “most adversarial” sample in the \(L_\infty\) ball around the benign sample. Thus, with this method, universally robust models can be developed against a majority of first-order attacks.

However, adversarial training still has some limitations. For example, because the process of generating each adversarial sample involves an iterative attack [3], adversarial training usually carries a high computational cost, which limits its practicality for large datasets. In addition, the effectiveness of adversarially trained models has been shown to be influenced by some factors, such as other \(L_p\) adversaries [107], more complex datasets like CIFAR [123, 124], and the perturbations occurring in the latent layer [125]. Another problem in adversarial training is a decrease in generalization [126, 127].

4.3.2 Distance Metrics and Latent Robustness.

Li et al. [107] proposed an improvement for adversarial training. They introduced Triplet Loss (one popular method in Distance Metric Learning) to the adversarial training, which enlarges the distance in embedding space between the adversarial examples and other examples. By incorporating a regularization term, this new algorithm effectively smooths the boundary and improves the robustness.

Most previous works focus on the robustness of the input layer of DNNs. Kumari et al. [125] found that the latent layers of the robust adversarial-trained model were still vulnerable to perturbations, though the input layer had high robustness. Based on this observation, they proposed Latent Adversarial Training, a fine-tuning technique, to improve the efficacy of adversarial training.

4.3.3 Complex Tasks.

Moreover, Buckman et al. [123] aimed to adapt adversarial training techniques to complex datasets where PGD-based Adversarial Training [3] is usually ineffective. Based on the hypothesis in Reference [46] that the over-generalization in models that are too linear leads to the vulnerabilities toward adversarial samples, they proposed to leverage the quantization of inputs to introduce a strong non-linearity. What they found was that combining them with the adversarial training increases adversarial accuracy. However, applying quantization alone can be broken easily. Thereby, Cai et al. [128] proposed curriculum adversarial training technique to improve the resilience of adversarial training and increase the performance on complex tasks. Specifically, they used a weak attack to train the model first and then increased the strength of the attack gradually until it reached an upper bound. Liu et al. [124] also addressed scaled up problems with complex datasets by combining the adversarial training with Bayesian learning.

4.3.4 Generalization.

Considering the decreasing performance of adversarial training in the face of random perturbations from other models, Tramèr et al. [126] proposed Ensemble Adversarial Training (EAT) to improve the generalization of defenses in the face of different attacks. The premise is to generate and transfer some one-step adversarial examples from other pre-trained models to augment the training data. Unlike the adversarial training in References [3, 46], EAT decouples the training phase from the generation process of adversarial examples. Similarly, Na et al. [127] also use already-trained models to design adversarial training, referred to as cascade adversarial training. Farnia et al. [129] proposed using regularization techniques to increase the adversarial test performance for adversarial training. They provided a theoretical analysis for the improving generalization with DNNs after introducing a computationally efficient regularization technique (spectral normalization) that significantly decreases generalization errors. Song et al. [130] also aimed to increase the generalization of existing methods based on adversarial training to resist adversarial perturbations from the perspective of domain adaptation.

4.3.5 Network Distillation.

Distillation is a popular transfer learning method, where smaller target models can be trained based on larger source DNNs. The knowledge of the source models can be transferred to the target models in the form of confidence scores. Papernot et al. [131] proposed the first defense method using network distillation against adversarial attacks in DNNs. Here, the computer could not find the gradient of the target model and, therefore, the gradient-based attacks would not work. Goldblum et al. [132] studied the distillation methods for generating robust target neural networks. They observed that the adversarial robustness could be transferred from a source model to a target model, even if the source model had been trained using clear images. They also proposed a new method, called Adversarially Robust Distillation (ARD) for distilling robustness onto smaller target networks. ARD encourages target models to imitate the output of their source model within an \(\epsilon -\)ball of training samples, which is essentially an analog of adversarial training.

Discussion. The strength of adversarial training with the worst-case perturbations is satisfactory. Adversarial training with PGD is a state-of-the-art defense, and this technique is easy to apply when compared with the certified robustness [121]. However, adversarial training requires re-training models and much computational resources, which causes the poor scalability to DNNs.

4.4.1 Architectural Defenses.

Due to the limitations of adversarial training, some defenses follow a different strategy of modifying the architecture of models without the need for additional training. Controlling Lipschitz constants layer-wise [108] and decreasing the invariance of DNNs [133, 134] can improve their robustness against adversarial perturbations. However, methods based on controlling Lipschitz constants are usually weaker than adversarial training [3]. Along these lines, Qian et al. [108] proposed a new method of controlling the Lipschitz constants of networks by modifying the regularization and loss functions. Schott et al. [109] highlighted the limitations of adversarial training especially for non-\(L_{\infty }\) perturbations and focused on the influence of unrecognizable images, modeling the class-conditional distributions by means of a Bayesian model.

Neklyudov et al. [133] also demonstrated that the variance layers they proposed (a different family of the stochastic layer) could provide higher robustness against targeted adversarial attacks. Jacobsen et al. [134] provided a novel view that the failures of machine learning models result from an excessive invariance to changes that are semantically meaningful. To address this issue, they modified the loss function by means of invertible networks, which can provably reduce the unwanted invariance.

4.4.2 Input Transformation.

The strategy of these defenses is to directly denoise the input instances and transform them into legitimate samples before they are fed into the target model. Song et al. [110] empirically evaluated a hypothesis that the adversarial examples usually lie in low probability even though the perturbations are very small. Thereby, they chose a neural density model to model image distributions to detect adversarial examples effectively, which can compute the probabilities of all images and the probability density of one input. Further, to mitigate the impact of adversarial perturbations, they proposed PixelDefend to purify adversarial examples by searching a probable image within a small deviation to the original input with a true label. Samangouei et al. [111] proposed a GAN-based denoising technique for adversarial examples regardless of the types of model. They leveraged the expressive capability of the generator to model the distribution of clean images. In addition, randomization is a commonly used technique in adversarial defenses [135, 136].

Discussion. As reactive defenses, purification including architectural defenses and input transformation are compatible with other defenses to improve the robustness further. Specifically, variance layers [133] can be combined with the idea of ensembles [126]. Input transformation techniques are model-agnostic and are regardless of whether an adversarial attacks appears or not [110, 111, 135]. Therefore, most of them can be combined with the other proactive defenses.

4.5.1 Image Domain and Audio Domain.

Face recognition is one of the simplest applications of image classifiers. Goswami et al. [113] analyzed the impacts of adversarial perturbations from preprocessing on face recognition tasks and demonstrated that simple distortions such as black grid lines could decrease the face verification accuracy. They proposed a countermeasure to detect the adversarially modified faces by evaluating the response from hidden layers of target models. For the identified faces with distortions, they exploited selective dropout to preprocess them before the recognition, which can rectify them to avoid the significant performance decrease.

Guo et al. [112] pointed that input transformations using JPEG compression have not been verified to be effective for strong adversarial attacks like FGSM. Thereby, they aimed to increase the effectiveness of input transformation-based defenses while preserving the necessary information for classification. They provided five transformations that can surprisingly defend existing attacks when the training data for DNNs was processed in a similar way before training. Xiang et al. [137] proposed a general defense framework against localized adversarial patches in the physical world, PatchGuard, which is compatible with any CNN with small receptive fields. Yang et al. [138] explored the potentials of audio data towards mitigating adversarial inputs and demonstrated the discriminative power of temporal dependency in audio data against adversarial examples. Hussain et al. [139] also studied the effectiveness of audio transformation-based defenses for detecting adversarial examples.

4.5.2 Graph-based Domain.

In addition, the structure of training data in the graph-based domain can be used in the design of adversarial defenses. Svoboda et al. [140] proposed new paradigms to develop more advanced models directly with higher robustness. The family of deep learning models on graphs named Peer-regularized Networks can exploit the information from graphs of peer samples and perform non-local forward propagation. Wu et al. [141] investigated the defense on graph data and pointed out that the robustness issue of GCN models was caused by the information aggregation of neighbors. They proposed a defense that leverages the Jaccard similarity score of nodes to detect adversarial attacks, because these attacks can improve the number of neighbors with poor similarity. Yang et al. [27] paid attention to the rumor detection problems on social media using camouflage strategies. They proposed a graph adversarial learning method to train a robust detector to resist adversarial perturbations, which increased the robustness and generalization simultaneously. Goodge et al. [142] focused on unsupervised anomaly-detecting autoencoders and analyzed its adversarial vulnerability.

4.5.3 Defenses in Privacy.

Apart from the adversarial perturbations crafted maliciously to mislead models, there are some “benign” perturbations that can cause positive influences to existing tasks based on DNNs by mitigating privacy concerns in deployment. Privacy attacks in machine learning can often be addressed using Differential Privacy (DP) technique, which has been well studied [143, 144, 145]. For example, due to the advantageous properties of differential privacy, it can also contribute to stabilize learning [146] or build heuristic models for game-theoretic solutions [147, 148]. Interestingly, benign adversarial perturbations can be used to build defenses to protect privacy in machine learning, such as membership privacy of training data [149, 150].

Discussion. Most works have been focusing on the adversarial examples in the image domain. In physical world, when conventional extensions of existing attacks for images fail to remove adversarial threats effectively in other domain, defenders can exploit some domain-specific features to resist adversarial examples. For example, due to the subtle effects of input transformation defenses from the image domain in speech recognition systems, temporal dependency in audio data can be used to improve the effectiveness of detection [138]. However, domain-specific features are usually used to improve the strength of input transformation defenses [112, 137, 138, 141] to preprocess examples, while few of them, such as information of peer samples in graphs [140], can help train model with higher robustness.