Membership Inference Attacks With Token-Level Deduplication on Korean Language Models

The confidentiality threat against training data has become a significant security problem in neural language models. Recent studies have shown that memorized training data can be extracted by injecting well-chosen prompts into generative language models. While these attacks have achieved remarkable success in the English-based Transformer architecture, it is unclear whether they are still effective in other language domains. This paper studies the effectiveness of attacks against Korean models and the potential for attack improvements that might be beneficial for future defense studies. The contribution of this study is two-fold. First, we perform a membership inference attack against the state-of-the-art Korean GPT model. We found approximate training data with 20% to 90% precision in the top-100 samples and confirmed that the proposed attack technique for naive GPT is valid across the language domains. Second, in this process, we observed that the redundancy of the selected sentences could hardly be detected with the existing attack method. Since the information appearing in a few documents is more likely to be meaningful, it is desirable to increase the uniqueness of the sentences to improve the effectiveness of the attack. Thus, we propose a deduplication strategy to replace the traditional word-level similarity metric with the BPE token level. Our proposed strategy reduces 6% to 22% of the underestimated samples among selected ones, thereby improving precision by up to 7%p. As a result, we show that considering both language- and model-specific characteristics is essential to improve the effectiveness of attack strategies. We also discuss possible mitigations against the MI attacks on the general language models.

generative LMs (e.g., GPT family) are a prime target for membership inference (MI) attacks since they can automatically yield samples from being inferred. Carlini et al. [7] identified 604 actual training data by selecting 1,800 unique candidates in GPT-2 [20]. Their follow-up study confirmed that the memorization capacity of LMs had a log-linear relationship with the model size [8]. Considering the current circumstance that GPT-based architectures are widely adopted as core engines in real applications [21], [22], the MI attacks against LMs are a substantial threat.
While these attacks have achieved remarkable success in the English-based Transformer [2] architecture, it is unclear whether they are still effective in other language domains. For example, many languages have grammatical characteristics different from English: no (or flexible) spacing, caseinsensitive characters, and relatively free word order in a sentence. These differences in grammatical characteristics lead to contrasts in preprocessing (specifically, tokenization 1 ) between English-based and Korean-based LMs, raising questions about the effectiveness of the attacks. On the contrary, Carlini et al.'s work assumed case-sensitivity and rigorous spacing by targeting the English model. To the best of our knowledge, there is no case of studying an MI attack with this language difference.
In this paper, we study the MI attacks on Korean-based generative LMs. As we mentioned above, Korean has very different characteristics compared to English. For example, a spacing is more complicated and a character is caseinsensitive. Moreover, the word order is more flexible so that its minor change may not affect the meaning of a sentence. Starting from the prior elegant work of Carlini et al. [7], we sample 100,000 texts from the LM ( §III-B). We score the texts using four metrics based on each loss of samples and select the top-100 potential members ( §III-C). Finally, we compute the approximated precision through the manual search ( §III-D). The main difference between ours and Carlini et al.'s work [7] is the subsequent verification of the inference results.
Given their interesting assumption-formalized as being k-eidetic memorization-that the less mentioned information in the documents is unintentional and potentially harmful, increasing the uniqueness of selected top-k samples is necessary. Following the assumption, we more strictly perform deduplication ( §III-D) based on the similarity between each prediction. We verify the effectiveness of our method by checking the number of underestimated samples where the newly calculated similarity exceeds a threshold so that judged to be a duplicate, unlike the previous result.
Our two-fold contribution based on the experimental results is as follows: • We verify that the existing MI attack is effective against the state-of-the-art (SOTA) Korean-based GPT model ( §IV-C). Our finding that well-defined MI attacks may not depend on the language domain motivates future research toward multilingual (or universal) MI attacks.
• We refine the existing deduplicating strategy to increase the uniqueness of selection in the MI process, resulting in improved attack precision ( §IV-D). A practical attack requires deduplication because the knowledge appearing in a few documents is likely to be meaningful information. We publish the experimental code to reproduce the main empirical results: https://github.com/seclab-yonsei/mia-kolm.

II. BACKGROUND
A. LANGUAGE MODELING 1) TRAINING OBJECTIVE An LM f explores the weightθ that maximizes the appearance probability for each element in a given training corpus D = x i N i=1 in the pre-training process. The probability that each sentence x 1:n = [x 1 , . . . , x n ] appears from f can be expressed as Eq. 1 by the chain rule.

2) INFERENCE
The LM f auto-regressively predicts the next word from the given words iteratively during the inference phase: where X is the set of all tokens that the LM can explore.

B. MEMBERSHIP INFERENCE ATTACK
A high intuition for MI is finding a distinguishable gap between the model's behavior of the training and test dataset [16], [27], [28], [29]. An adversary ( §III-A) tries to exploit loss values [7], logits/predictions [30], [31], or gradient norms [32], thereby finding or raising the gap. Since our experiments start from Carlini et al.'s work [7], we attempt MI based on sample loss.

III. METHODOLOGY
In this section, we describe an approach to attack the Korean model. We first define the attacker's capabilities and objectives ( §III-A). Afterward, we explain the method to generate texts following the threat model ( §III-B), determine the member/non-member of the sampled data through MI ( §III-C), and verify the GT of the selected data ( §III-D).

2) ADVERSARY's OBJECTIVES
Given an input x and access to the pre-trained LM f , an attacker infers whetherx ∈ D train forx ∈ D infer [34]. We assume an attacker conducting an untargeted attack that acquires data without aiming or deriving for specific data. The attacker maximizes the MI precision by selecting the set of top-k samples A among numerous samples obtained from where match is a substring matching function that determines memorization, and m is a threshold that controls the lower bound of matching characters ( §III-D2).

B. TEXT SAMPLING
For MI, we first approximate the population by generating a sufficiently large number of samples. When selecting the next word, we auto-regressively sample the words with the top-40 probabilities. The output statement is 256 tokens, excluding the input prompt. For random sampling, we use a special token [EOS] (end of the sentence) as a prompt to inform the end of the sentence. We can interpret the process of inserting only a token to trigger text in two ways. First, the sentences are likely to be familiar with the form the LM frequently sees during training. For example, if the training data contains scripts for movies or dramas, some output sentences might have a similar form. The other one is that the LM will likely guide the most grammatically complete text form. Consequently, this generation process leads to the start of extracting training data.

C. MEMBERSHIP INFERENCE
We infer membership of generated samples by selecting with a high probability of being in the training data. MI attack on machine learning (ML) starts with the assumption that the data used for training is overfitting (or memorized) more than the data not used for training (i.e., test dataset) [30]; the data already learned once will have relatively high confidence. We use four out of six evaluation metrics proposed by Carlini et al. [7] (Table 1) to capture the confidence difference. We select potential member sentences with the topk duplicates removed by scoring the samples on the four metrics. We did not use the other two metrics measuring the loss ratio of two different sizes (XL versus medium or small) models for a single sentence.

1) PERPLEXITY
We calculate the perplexity (PPL) of the output sentences from the LM as the geometric mean of the sentence probabilities over the length. Intuitively, the PPL is the number of confusing words for every time step. The lower the value, the more confident the LM is about the sentence, and the higher the value, the more confused the LM is.

2) COMPARING WITH ZLIB ENTROPY
LM sometimes yields an unexpected output; it produces an incoherent gibberish output or degenerates repetition [33]. Although the PPL effectively filters out unfamiliar phrases that do not fit the syntax, it sometimes gives a very high score to unexpectedly repeated output sentences (e.g., '' [1] [2] [3] . . . '' or ''-----. . .''). Carlini et al. attempted filtering by calculating the entropy of a sentence based on the zlib [35] compression algorithm to alleviate this phenomenon. The lower the PPL and the higher the zlib of the sentence at the same time, the closer it is to a probable member. We present a code implementation for calculating zlib entropy in the appendix ( §A-A).

3) COMPARING WITH LOWERCASE TEXTS
In the case of English text containing sufficient uppercase characters, such as newspaper article titles and product names, converting the sentences to lowercase may dramatically raise the PPL. We select the Lowercase as one of the metrics since even the Korean text can include various English characters.

4) CALCULATING MINIMUM PERPLEXITY BY SLIDING WINDOW
There is a possibility that the inference result is not wholly similar to the actual member and is included only in some segmentation. Carlini et al. calculated the lowest PPL among VOLUME 11, 2023 50 tokens by sliding the sentence. Since the stride is unclear, we set it to 16, considering the time cost.

D. VERIFICATION
We verify whether the selected top-k samples by MI are members or not. The dataset used for training is not publicly available, and the property (distribution) in which the data was collected is also unspecified. Since the GT is unidentified, we collected approximated reference documents by manually searching each sample with the Google search engine as a suboptimal approach.

1) CANDIDATE DEDUPLICATION
In order to maximize the amount of information in the unique output sentences, we deduplicate two sentences when the similarity between them exceeds a certain threshold. Two duplicated sentences are the same or similar. We deleted the identical sentences except one for efficient indexing. If they are not the same but similar to some degree, we perform deduplicating by the similarity function based on trigram multiset. We describe two deduplication method, including our new strategy with BPE tokenization, as follows: • Word-level tokenization is a traditional way of splitting each token by whitespace and punctuation [7]. Simple to use and understand, but it might not be a suitable method for tokenizing Korean text with complicated spacing.
• Byte Pair Encoding (BPE) [36] tokenization is one of NLP's most popular subword segmentation algorithms [37]. Prior knowledge of vocabulary mapping is required when pre-training the target LM, but generally, we can expect to divide the text into more precise units than word-level division. This modification was motivated by insufficient previous deduplicating, in which some similar texts were included in top-k (Table 2).
We calculate the similarity between tokenized sentences (i.e., predictions) and deduplicate them based on that. We obtain a family of sets tri by binding three adjacent tokens (i.e., trigram).
For some sentence s is similar with t if and only if |tri(s)∩tri(t)| |tri(s)| ≥ α for the threshold α = 0.5. We keep the hyperparameter α the same to verify the attack proposed in the previous study [39]. We leave the implementation for similarity calculations in our appendix ( §A-B).

2) SUBSTRING MATCHING
The raw document text (called reference) collected by the searching trivially includes the inference result (called hypothesis). Given reference r = r 1 , . . . , r p and hypothesis h = h 1 , . . . , h q where p > q and each element means a character, h is memorized if r i..i+m = h j..j+m for some indexes i and j. We maintain m to 50 for the same reason we set the hyperparameter α.

IV. EVALUATION
We answer two-fold research questions through our experiments:

B. TARGET SYSTEM
We experiment with the Korean-based SOTA generative LM, KoGPT (Korean Generative Pre-trained Transformer) [42]. The pre-trained KoGPT is open for research in HuggingFace [41], and since the number of parameters is large enough, we considered it is effective MI would be possible [8]. In addition, the corporation to which the team belongs has a search engine 2 that can collect high-quality source data. The high quality of the training data leads to more complete outputs and contributes to the consistency (i.e., reproducibility) of the experiment results.  Table 3 shows the quantitative results of MI (first and second results). We identified 89 and 90 actual members out of 100 potential members for the metrics PPL and zlib, respectively. We searched only 20 samples with the Lowercase metric, 33 fewer than the prior result. We detected 52 samples with Window, 19 more than before; yet, it did not show performance improvement as much as PPL or zlib metric. Table 4 provides examples of top-1 sentences for each metric. PPL and zlib entropy metrics showed the same output. Lowercase metric contained many English uppercase characters and had a low PPL at the same time. Window metric includes trivial (i.e., repeated and boring) segments in the middle of the sentence.
The selection of the four metrics is not focused on specific outputs but instead on looking at different parts. As shown in Fig. 1, the top-k PPL and zlib samples often overlap. Nonetheless, Lowercase and Window indicated an even distribution of the index regardless of the PPL. It is crucial to TABLE 2. An example of a non-duplicated sentence with high similarity by intrinsic evaluation. We use difflib [38] to highlight common areas. In the existing method [7], the similarity with the reference was underestimated to 0.474. However, our method raises the similarity to 0.538 and judges it as a duplicate.

TABLE 3.
Quantitative results of MI using four metrics. Each result means the number of samples found with the corresponding metric out of 100. The leading cause of finding more members than prior work in PPL, zlib, and Window metrics is the increase in the number of parameters in LM. On the other hand, the Lowercase metric shows poor performance due to differences in language domains; attempt to use the characteristics of English words from Korean-based LM.

TABLE 4.
The output sentence with the highest score for each metric. We masked the personal information included in the output with . The part obscured by the character ''*'' is presumed to be de-identified before pre-training. evaluate different aspects of the generated text as it enriches the distribution of selected samples.

2) DISCUSSION
Contrary to the previous insistence that PPL alone is challenging for high performance, we confirmed the encouraging precision of PPL on the Korean model. We interpret that the similar performance of PPL to the zlib is because the ratio of repeated phrases in the top-k sample selected by PPL was low, resulting in a similar performance to the zlib. In addition, in Figure 1, we observed that PPL and zlib metrics tend to pay attention to the same samples and even 94 of the top-100 samples selected from PPL were also selected as in zlib. On the other hand, the lowercase found only 20 samples, about 37.7%, compared to the 53 samples found in the existing GPT-2. Since web-crawled Korean text also contains English in product names or URLs, the metric will guarantee the minimum performance. Regardless, Korean is case-insensitive, so comparing PPL before and after lowercase is not enough to show outstanding performance. In summary, existing MI attacks still work well beyond language domains through these characteristics, yet we can expect a higher performance using language-specific strategies.  While we detected more redundant samples than in the previous work, it is premature to conclude that KoGPT is vulnerable. We maintained the existing Carlini et al.'s attack strategies substantially the same, but the experimental results may be exaggerated or reduced due to the following differences. First, since our target system is 4.0× larger than that of Carlini et al. ( §IV-B), the memorization capacity may differ. Carlini et al. used the GPT-2 (XL) [23] with 1542M parameters, and we used KoGPT [42] with 6167M parameters. For a fairer comparison, it is desirable to perform an empirical evaluation on the same architecture (i.e., GPT-J [25]). Second, memorization definition is also a significant factor influencing precision. Carlini et al. reported a potentially membered sample to OpenAI and received only ''member/nonmember'' results; i.e., they overlooked a precise threshold of memorization. On the other hand, we considered a case in which 50 tokens of BPE tokenized texts were duplicated ( §III-D2). Finally, the total number of sampled texts is also different; Carlini et al. sampled 200,000 texts in a sampling strategy, but we generated only 100,000 candidates.

D. RQ2: UNIQUENESS OF INFERRED SAMPLES 1) EXPERIMENTAL RESULT
As we mentioned in §III-D1, we modified the existing word-level tokenization with BPE tokenization to improve the uniqueness of inference results. The BPE token-level method found 6% to 22% more duplicated samples from each metric (Table 5), showing that the previous method underestimated the similarity of such samples.
Furthermore, we report the precision after selecting topk with the BPE tokenizer-based deduplicating strategy (the third row in Table 3). Deduplication with the BPE tokenizer improves the precision compared to the word-level tokenizer except for Lowercase; Lowercase is reduced by 1%p, PPL, zlib, and Window are increased by 2%p, 1%p, and 7%p, respectively. Accordingly, increasing uniqueness increases the precision of MI attacks.

2) DISCUSSION
Although few selected samples impede the hasty expansion interpretation of the experimental results, the existence of potentially overestimated members of about 6% per metric leaves room for further advancement. We found from further studies that a more sophisticated deduplication could increase the top-k precision of MI attacks. Therefore, increasing the uniqueness of top-k candidates for MI attacks is essential because it increases the amount of information in a sample and attack performance.
This deduplication strategy is not limited to BPE tokenization and can be generalized to all tokenizers used by the target model (except word-level, e.g.; WordPiece [4], [43]). If the pre-trained LM is public-available, the tokenizer's configuration also tends to be disclosed. Therefore, the attacker can naturally optimize the attack strategy by inferring the tokenizer used for training from the open source.

A. INFERENCE SAMPLE PROPERTIES
Since we experimented with an untargeted attack, we did not consider any properties of the inferences. A targeted attack that maximizes the risk of privacy leakage of training data resulting from MI outcomes is a promising future research area. Researchers can extend the experiment to investigate policy issues that may arise from the output, regardless of member/non-member results [44]. A simple example is an issue of copyright and intellectual property (IP) infringement driven by leakage of the memorized training data.

B. VARIOUS SAMPLING STRATEGIES
We can consider various generation methods to increase further the entropy of potential members beyond the current sampling method. For example, top-p [33] sampling can be used concurrently with top-k sampling currently in use. Alternatively, we can intentionally bias the outputs by adjusting various penalties, e.g., repetition [45], coverage [46], and length [43]. We argue that we can also increase the diversity of our output by tweaking the number of return sequences. In other words, it is possible to increase the output distribution by generating the top-n probability outputs instead of calculating the maximum probability for each input prompt.

C. APPLIED MEMBERSHIP INFERENCE
An attacker can try to make a more optimal attack based on the approximate distribution of the training dataset. Existing studies treated output length as a constant, such as limiting a sentence's maximum length or fixing the minimum/maximum length equally [7], [39]. However, assuming a limited white-box attack environment where the proper length of the desired output sentence is known [47], it would be reasonable to limit the sentence's output not to be too long/short. In other words, the attacker adjusts the min/max length of the outputs to fit the distribution of the training dataset. For example, the RealNews [48] and One-Billion Word Benchmark (LM1B) [49] datasets have 31M and 30M documents with an average of 793 and 32 BPE tokens per sample, respectively.

D. ENTIRELY THE SAME OUTPUTS
We found 35 curious sentences 3 whose all 256 tokens were utterly the same ( §III-D1). The verbatim outputs induced two or more times by LM imply that the possibility of the sentence existing in the training data is very high. In advance, we have obliterated duplicate sentences for efficient indexing, but further research is still required in future.

E. POSSIBLE MITIGATIONS
Numerous defenses and possible mitigations have been studied to prevent MI. Considering the sequential phase of DL modeling, we describe these prior works by dividing them into four attack surfaces: (i) pre-processing, (ii) training, (iii) inference, and (iv) deployment.

1) PRE-PROCESSING PHASE
Before training the model, we can attempt to preemptively reduce the memorization of the sequences that repeatedly appear by deduplicating-a step in text cleaning [50]-the given training dataset [39], [51]. Lee et al. [39] claimed that deduplicating training data reduces the training time of LMs, with little or no harm to performance, and reduces the rate of emitting memorized training data by up to 10 times. As a more proactive approach, based on the observation that samples vulnerable to MI attacks are concentrated in some outliers [52], [53], it seems possible to try to detect and remove them in advance. Regardless, these attempts are refuted head-on due to the privacy onion effect [54] that if samples vulnerable to MI attacks are removed, new samples become vulnerable to MI attacks.

2) TRAINING PHASE
Differential privacy (DP) training is one well-known method of providing strong privacy guarantees in the training dataset by adding noise to the gradient [55], [56]. While being a proven concept [34], [57], [58], DP can have tradeoffs that impair the utility of models and result in performance degradation. As an alternative to controlling this privacyutility tradeoff, there are attempts to reduce overfitting-the main cause of memorization-by modifying the loss function [28], [59], adding regularization [47], [60], [61], or quantization [62]. Controlling direct access to private training data through knowledge distillation [63] is also extensively studied [16], [64], [65].

3) INFERENCE PHASE
Extensive work has been studied on mitigating MI attacks through confidence masking, which intentionally promotes the actual confidence score returned by the target model [34]. For example, the target model could consider giving the attacker only a top-k confidence score [30], returning only the more restricted predicted labels (i.e., top-1) [31], or adding manipulated noise to the prediction vector [66], [67].

4) DEPLOYMENT PHASE
An attempt to intentionally unlearn training data exposed from a deployed model can also be seen as one of the possible mitigations of an MI attack [68], [69]. The most naive way to make a DL model forget certain data is to relearn it from scratch, but it can be challenging due to a large amount of time and computational cost. From this point of view, machine unlearning techniques that ensure that certain data is not used to train a model are not only budget-efficient but also crucial for a responsible follow-up by the model deployer.

VI. RELATED WORK A. MEMBERSHIP INFERENCE ATTACKS AGAINST MACHINE LEARNING MODELS
Shokri et al. [30] proposed a shadow training technique for training an attack model that predicts member/non-member using a prediction probability vector. They first created several shadow models that mimic the behavior of the target model. Afterward, they trained the shadow models through three methods for generating or synthesizing training data. Finally, based on the shadow models' output, they trained an attack model that predicts member/non-member and used it for the actual attack. While it showed outstanding performance in the classification task, the shadow training technique is limited in the recent LMs for two reasons. First, they achieved shadow training to target models trained by supervised learning, but recent LMs are mostly pre-trained with unsupervised learning. Second, to improve the performance of shadow training, the more the number of mimic models, the better. However, recent LMs have limited budgets since the model scale and the training data size have grown exponentially. For example, GPT-3 [24] (proposed in 2020) and Switch Transformers [70] (proposed in 2022) have parameters of 175 billion and one trillion, respectively.

B. EXTRACTING TRAINING DATA FROM LARGE LANGUAGE MODELS
Carlini et al. proposed a MI attack pipeline targeting a generative LM [7]. They deduplicated sentences by calculating a word-level trigram-multiset based on whitespaces and punctuations in a sentence. In contrast, we considered that the deduplicating strategy is an overly rough assumption for VOLUME 11, 2023 determining the similarity of sentences and improved it by the BPE token-level similarity ( §III-D1).

C. HOW BPE AFFECTS MEMORIZATION IN TRANSFORMERS
Kharitonov et al. [71] showed that the size of the subword vocabulary learned by BPE significantly affected both the ability and tendency of the Transformer models to memorize training data. They argue that large cardinality BPE vocabulary greatly encourages memorization due to the lower out-of-vocabulary (OOV) frequencies. However, unlike the character-level BPE tokenization [37] they studied, we experiment with byte-level BPE tokenization [23]; especially, our target model is pre-trained with a byte-level BPE tokenizer. Byte-level BPE tokenization rarely incurs OOV, unlike character-level BPE tokenization. Therefore, their studywhere the more extensive the capacity of the subword mapping table, the more vulnerable it is to MI attacks-is quite different from ours.

Carlini et al. performed extensive experiments on GPT-J [25]
and GPT-Neo [26] for quantitative analysis of factors that increase the memorization capability of LM. As a result, they identified three attributes that significantly influence memorization; bigger models, repeated strings, and more extended context. They confirmed that the LM could accurately reproduce the target sentence with a chosen length-k prompt. Afterward, they expanded the study to the T5 [72] masked LM (MLM) as a replication experiment. We conducted experiments only on models trained by the next word prediction (NWP) strategy ( §IV-B), not MLM. We did not study memorization according to the selection of input prompts; this is due to the characteristics of the target system for which training data is not public-available. We plan to expand the experiment to public LMs trained on public-available datasets in the future.

E. DEDUPLICATING TRAINING DATA MAKES LANGUAGE MODELS BETTER
Lee et al. showed that duplication of training data for LM reduces the diversity of outputs and increases the likelihood of exposing the members [39]. They generated 100,000 samples, each with a maximum length of 512 BPE tokens. They defined that the LM remembers tokens if each output has exactly 50 token substrings in the training data. As a result, they showed that LM memorized more than 1% of the generated tokens. We maintain their memorization definition while reducing the token length limit from ''less than 512'' to ''exactly 256''. This process prevents the experimental results (precision) from being exaggerated.

VII. CONCLUSION
This paper studied the effectiveness and improvements of the existing MI attack for the SOTA Korean-based GPT model.
We confirmed that the existing attack strategies are still sufficient for data extraction of the Korean LM. In addition, PPL and zlib metrics yield up to 80%p higher than the prior results. We established that the existing deduplicating procedure during this attack process is not rigorous enough. We identified 6% to 22% more duplicated samples than the existing method by replacing the word-level trigram similarity with the BPE token-level one. Since the knowledge that appears in fewer documents is more likely to be meaningful, increasing the uniqueness of the selected samples increases the effectiveness and diversity of the attack. Furthermore, we confirmed that increasing the uniqueness of the top-k result of the MI attack increases the precision by up to 7%p. We hope future work continues to explore whether MI attacks are still effective in other LMs and leverage new findings to improve language-specific attack and defense strategies.
We plan to gradually expand the experiment on LM memorization, one of the possible mitigations. Future studies will help understand the fundamental cause of confidentiality leaks for LM. We believe that our study on confidentiality infringement serves as an opportunity to raise the awareness of all researchers, developers, and administrators about security issues.

APPENDIX A CODE IMPLEMENTATION A. CALCULATING ZLIB ENTROPY
We obtained zlib entropy by calculating the number of bits of the compressed output after encoding the sentence in UTF-8 [73].

B. CALCULATING SIMILARITY OF TRIGRAM MULTISET
The similarity of two sentences is obtained by calculating the ratio of trigram intersection of two sentences tokenized by word-level or BPE tokenizer. Since we compute the similarity of str2 to the selected str1, the similarity operation is not commutative (i.e., the similarity between str1 and str2 is different from the similarity between str2 and str1).  He is currently a Professor of information security with Yonsei University, where he is also the Director of the Information Security Laboratory. His research interests include authentication, cryptographic protocols, system security, fuzzing, usable security, AI security, adversarial robustness, and adversarial machine learning. VOLUME 11, 2023