Machine learning in computational literary studies

1.1 Needs of CLS

As with all subject-specific research interests in the broader field of digital humanities, a key challenge of CLS is to operationalize conventional literary studies categories and theories for computational analysis. It is therefore necessary to model categories that may, at first glance, seem abstract in such a way that they can be automatically inferred from the text surface. At the same time, many CLS approaches need to make the results of studies supported by machine language processing available for interpretation in literary studies. In line with C. P. Snow’s thesis of the two diametrically opposed scientific cultures [14], it can be claimed that the interpretation of numerical, quantitative results brings together “explaining” as the central goal of the natural sciences and “understanding” as the goal of the humanities. The question of interpretability can, accordingly, also be understood more broadly as a question about the “translatability” of the results generated by computational methods into the field of traditional, non-computational literary theory. To bridge the gap between the two cultures of research, not only do the results have to be understandable but so does the process by which they were obtained. Here we see an interesting link to the field of machine learning, where explaining, understanding, and interpreting model outputs are all current fields of research.

1.2 CLS as applied NLP

The computational text analysis in CLS can be considered an application domain of natural language processing (NLP). Traditionally NLP, in conjunction with adjacent fields like computational linguistics, has been focused on analyzing all aspects of texts from the small-scale linguistic phenomena such as parts of speech^[1] to semantics of entire works. Literature presents a very challenging domain due to the complexity and in the case of prose and drama often also length of such texts, especially since many existing approaches were primarily tested on news data. For example, the popular OntoNotes [15] dataset encompasses news texts in multiple languages annotated with a wide range of linguistic phenomena. That is to say, CLS can offer a test bed for challenging tasks of natural language processing while providing the tools needed to facilitate literary analysis. One example of literary works being used in their capacity as challenging NLP problems is in the domain of abstractive text summarization. Wu et al. [16] used modern language processing models to summarize books, solving the length problem by recursively summing up previous summaries starting from small sections of text.

2.1 From operationalization to automation

Concepts of literary studies are not typically well-defined in a formal sense and rely on interpretation when applying them to an existing text. As a result, in non-digital literary studies, the focus is not on objective research findings, but on the “intersubjective comprehensibility” of the argumentation or interpretation [10], that is to say verification of a concept is done by forming a shared understanding of the subject. An operationalization specifies which aspects are to be considered in this process, it may also simplify a theoretical concept. One possible operationalization is an annotation guideline, sometimes also called a codebook.

Annotations in CLS are typically created either by domain experts (e.g. the researchers themselves) or trained annotators (e.g. student assistants), but not typically by untrained annotators such as crowd workers. Both document-level annotations and span-level annotations are common. Helling et al. [1] found the most used annotation tool among their surveyed researchers to be CATMA [21], their survey, however, only included researchers at German and Swiss Universities.

In Section 3, we will explore automation techniques in detail. Generally speaking though, in CLS, alongside modern neural-network architectures, a lot of more traditional machine learning approaches, like regressions or support vector machines (SVMs) are used; in these cases, some feature selection is typically required. Alongside these typically supervised approaches, unsupervised methods like clustering and topic modeling are also employed. Finally, not all of CLS relies on machine learning with many works instead using text mining, e.g. in the form of frequency analysis, to gain insights into the structure and content of texts.

3.1 Our survey

To provide a broader, more concrete overview of the machine learning methods in use in the field of CLS, we select a number of scientific venues that are known for significant CLS contributions, with the aim of building a broad cross-section of the CLS community. While there is no clear boundary to what constitutes a machine learning method as compared to statistical analysis, we consider all regression methods to be machine learning techniques, whereas frequency analysis of texts and significance testing are considered statistical methods. We will discuss predominant methodologies in detail, but will not explore each individual method.

We only consider publications in English, which does exclude a variety of smaller language-specific or region-specific venues; works in English do, however, often have literature in other languages as their subject of study (e.g. [22–24]). Our selection includes the newly founded “Journal of Computational Literary” studies (JCLS), the Proceedings of the international “Digital Humanities” (DH) conference, as well as the SIGHUM Workshop “on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature”, and the CHR “Computational Humanities Research” conference. The selection follows the goal of providing a cross-section of the community, with the Digital Humanities conference presenting the largest international conference on digital humanities and thereby ensuring geographic diversity. The CHR is an interdisciplinary conference focusing on all computational approaches in the humanities and receives submissions mostly from the European community. The SIGHUM workshop, on the other hand, is organized as part of the ACL and thereby connected to the NLP community. Lastly, the JCLS is the only journal in our lineup and unlike the other three is exclusively focused on CLS. We use the most recent issue of each publication, with the exception of SIGHUM, where the 2022 issue only features two CLS-focused entries, leading us to include the 2021 issue as well, as such methodological comparisons across venues, based on our survey, are potentially misleading due to the different timeframes. We cannot claim that our data is comprehensive and representative, yet it provides an overview that is not limited to a specific sub-community.

Across all publications, we start with a selection of 215 papers, only 33 % of which we considered to feature CLS in our initial screening, based on both the titles and when in doubt, the contents of the papers. This ratio differs greatly depending on the publication. For example, all submissions in the JCLS journal concern CLS, while only about a quarter of all Digital Humanities conference “Long Paper” submissions were found to be CLS-specific. In the case of CHR, we found a surprising prevalence of literary studies at the conference, with 14 of 31 papers falling into the category of CLS. After this initial selection, we checked the papers for the application of machine learning techniques, narrowing our selection of 71 CLS-related papers down to 40, which we will consider in this overview. Importantly, we also removed stylometric analyses since they, with one exception [25] that we will discuss in the Section 3.7, do not apply methodologies of machine learning. Other work discarded in this step was based on statistical evaluation of word distributions, e.g. dispersion measures, as well as conceptual work discussing the CLS research process. Detailed numbers on the occurrences of CLS and machine learning specific works are listed in Table 1.

Our collected data on the papers’ methodologies is presented in Table 2. The main machine learning methodologies are listed in the corresponding column, with the model name specifying the name of a pre-trained model, if applicable. The “LS Question/Topic” column denotes the question or topic of the paper with regard to literary studies. In some cases we did not identify a literary studies’ question, as these heavily focus on the evaluation or improvement of technical methodologies instead [26–28]. For what we list as the method we only consider the actual decision-making method, e.g. when a text is classified using an SVM that operates on word embeddings, we list SVM as the methodology. The features that are used are tracked separately. The columns “Annotations” and “Rules” indicate which of these two categories the paper falls into: (1) approaches with rule-based processing, potentially with existing pre-trained models or (2) those training their own models and scaling their annotations to more texts. One example of (1), the rule-based approach, can be found in the work by Karlińska et al. [29]. While they do not employ annotators themselves (outside of meta-data creation and validation) they instead use existing tools for named entity recognition, geographic information retrieval, and sentiment analysis. Using these tools, they analyze the sentiment towards cities as compared to other locations and conclude that, unlike suggested by previous work, cities actually are depicted more positively than villages and the countryside. While they note that this finding does need further analysis, this is an example of CLS potentially refuting a thesis previously set out. An example of (2), approaches that use their own annotations for training, can be found in the work on emotions in dramas by Schmidt et al. [30]. Some works fall into both categories, for example in the case of Abdibayev et al. [31], who use both annotations and rules for different aspects of their analysis. The column “Features” lists the features used in feature-based approaches. Although, in some cases, the list is very extensive with multiple feature sets being discussed and compared, in this case, we list them as “various”.

In our survey, transformers are by far the most popular method, followed by SVMs and logistic regression (see Figure 3). While transformers, which we will explore in detail in Section 3.4, operate more or less directly on the input text, SVMs and logistic regression both require some sort of feature engineering. We have multiple cases in which Tf-idf (term frequency – inverse document frequency) is listed as a feature, this is typically calculated on the basis of individual tokens but not in all cases clearly specified. We will consider employed features in more detail in Section 3.5.

Figure 3:

The distribution of methods in our literature review shows that transformers are by far the most popular method but still not used in even half the papers, demonstrating the diversity of methodology in CLS. We omit methods that only occured once.

Overall we found just over half of the final selection of papers to use some sort of annotation-based setup (see Table 1), allowing for supervised learning. Others built rule-based systems on top of existing machine learning approaches [29, 32, 33]; for example Piper et al. [33] use an existing sense tag system (based on WordNet) to identify “things” and the category they belong to, finding that most things referred to in literature are “human-made and supportive” meaning such objects as “rooms, houses, doors, windows, […], roads, kitchens” and others.

Some of the techniques listed in the Table 2 are not explained in detail in this document, either because they are rarely used or because they are already very well established techniques. For reference, we expand the abbreviation of those that are not explained elsewhere here. In terms of model variants, RNN refers to recurrent neural networks with GRU referring to gated recurrent units, a simplified variant of the variant of the LSTM (long short-term memory), where BiLSTM refers to the LSTM’s bidirectional variant. CRF refers to conditional random fields, a statistical model. LDA refers to Latent Dirichlet allocation a topic modeling approach, which is not to be confused with Linear Discriminant Analysis which we spell out in the one instance it is used. GRL refers to graph representation learning, NB to Naïve Bayes, and c2f to a specific coreference resolution system [34]. For the features, we use pos as short for part-of-speech, bow as short for bag-of-words and dep as short for dependency tree information.

3.2 Pre-processing

Recently, the NLP community has been slowly moving away from employing the so-called pipeline approach, where one language processing step builds on the previous ones towards end-to-end models such as transformers [64]. For example, a pipeline might consist of a tokenizer splitting the text into individual tokens, a part-of-speech tagger tagging each token, and a named entity tagger recognizing entities on the basis of the two other components’ outputs. A trend away from this kind of architecture can also be observed in CLS with the adoption of transformers; pipeline-based approaches however seem to still be popular, as evidenced by a wide range of works making use of hand-crafted features for their classifiers. One of the most popular frameworks, albeit not specific to CLS, providing a range of preprocessing options is spaCy [65], employed by a multitude of authors in our survey [25, 29, 47, 50]. While the library provides high-level capabilities like coreference resolution, it can also be used for more basic preprocessing like sentence splitting and tokenization as well as dependency parsing and part-of-speech tagging. In terms of tools focused on the processing of literature, BookNLP^[3] provides a pipeline-based approach to processing entire books (used by e.g. [48, 66]), incorporating such preprocessing tasks as named entity recognition, part-of-speech tagging (using SpaCy), and coreference resolution. Similarly, LLpro [67] provides such a pipeline approach for German language texts. The MONAPipe pipeline [68], integrates with spaCy and, in addition to some standard pre-processing steps, provides a host of literary analysis tools like event annotations, entity linking, coreference resolution, and speech attribution to name a few.

We see three main motivations for the CLS community still relying on pipeline-based approaches: (1) training data for CLS-specific problems often does not exist, meaning it either has to be time-consumingly annotated or a hybrid approach that relies on some form of rule-based inference may have to be used. Further, (2) as we discuss in detail in Section 3.4, existing language models are often not perfectly applicable to the literature domain. This fact potentially increases the viability of simpler approaches, for example on the basis of features extracted in an early pipeline step. Finally (3), feature-based methods can typically more easily be inspected with regard to their decision-making process, such as inspecting the importance of individual features, allowing researchers to interpret not only the models output but also the weight of features in its decision process.

3.3 Word embeddings

Word embeddings, in general, are vectors that represent the semantics of individual words by their position in space. Conceptually we distinguish between static word embeddings which represent each occurrence of a given word the same way and contextual word embeddings which represent each instance of a word differently, based on its context. Contextual word embeddings were, in the CLS literature we reviewed, typically produced by transformers. Overall, popular choices of static embeddings were GloVe, word2vec, and fastText. While GloVe [69] and word2vec [70] use slightly different techniques, they often perform similarly, fastText [71], on the other hand, has the added capability of processing sub-word tokens. Where any term that was not known during training to GloVe or word2vec can not be represented, fastText can represent unknown words by representing constituent character sequences individually.

A large number of works [32, 39, 47, 56] make use of static word embeddings in conjunction with classifiers, sometimes as one feature among many. In the simplest case, Algee-Hewitt [32] uses linear discriminant analysis on word embeddings to identify if a given noun refers to an object or a concept. The direct use of word embeddings and their distances as a means of analysis appears to be almost as common [41, 52, 61]. For example, Ehrmanntraut et al. [58] use GloVe and fastText embeddings to predict similarities of pairs of poems, comparing the results with human judgments. They find fastText embeddings to perform slightly better in their setup. Measuring accuracy in identifying which of two poems is overall more similar to a given anchor poem, the best fastText static embedding approach achieves scores of 0.66 without supervision and improves to 0.72 after supervised training of a siamese-network [72] that uses the existing fastText embeddings as input. Eder et al. [39] use artificially added document-specific tokens to explore the use of word embeddings in authorship attribution, with the idea that artificial tokens capture their context, which for static embeddings is true at training time on a corpus level, thereby representing the document or the author.

3.4 Transformer-based methods

Transformer-based architectures [73] have revolutionized the field of NLP in the last few years by drastically outperforming previous approaches. Recently generative models, also using transformer architectures, have demonstrated unprecedented text generation capabilities. The success of transformers as a neural-network architecture, even outside the domain of language processing, can be attributed to their attention mechanism. In contrast to previous recurrent neural network approaches, this allows the model to directly access the information of any combination of tokens at once, without relying on its own state, as a recurrent model would. The application of transformers to CLS does, however, bring a couple of challenges with it: for one, foundation models, which require vast computational resources, are typically primarily trained on data scraped from the internet, resulting in a domain mismatch. For example, only 16 % of GPT-3’s training data is sourced from two book corpora. The BookCorpus, which was used for the training of a variety of foundation models including BERT and GPT variants, has been shown to exhibit a large bias in terms of genre distribution [74]. As the selection of texts for such models is not driven by literary studies’ needs, the resulting models potentially perform poorly on the texts subject to analysis. Konle et al. [75] have shown that domain adaptation helps improving the performance of transformer models on downstream tasks in the domain of literary fiction. In another case, in the domain of German dramas, domain adaption was found to not be beneficial, with the authors hypothesizing that the additional training data was insufficient [30]. An additional issue in the application of transformers can be found in older literary texts, which often do not follow standard orthographic rules, necessitating conversion approaches like the one by de la Rosa et al. [28].

One potential impediment to the application of transformers, especially to historical literary data, is the lack of training data, as transformer models are typically trained with at least billions of tokens (e.g. the original BERT implementation [73] used a corpus of more than three billion words). ECCO-Bert [76] is an example of a domain-specific transformer model, trained on a corpus of eighteenth-century data. In the work by Zhang et al. [53], it achieves better results than transformer models trained on modern data, with the accuracy increasing from 0.94 to 0.96, exemplifying the utility of time- and domain-specific models. Even without domain specific training data, transformers can, however, typically still outperform more traditional approaches. Schmidt et al. [30], for example, found transformer models trained on contemporary language to still clearly outperform their Naïve Bayes approach with the F1 score improving from 0.50 to 0.64 for an emotion classification task.

Many of the transformer models we encountered are specific to English. In addition, we encountered language-specific models for Chinese, German, Italian, and Spanish [23, 24, 30, 51]. For Dutch, German, and Italian we also found multilingual models being applied in monolingual contexts [28, 51, 58, 59], with Parigini et al. [51] finding an Italian specific model to clearly outperform the multilingual BERT variant with performance increasing from an F1 score 0.31 of to 0.49. Multilingual models were also applied to the analysis of multilingual datasets [26].

Beyond the training data-related issues raised so far, transformers have one other major shortcoming when applied to the literature domain: their context size. Early transformers like BERT [73] only had a context window size of 512 sub-word tokens. As longer words are typically represented as combinations of two or more sub-word tokens, this means that in practice, depending on the text, only about 300 individual words will be accepted at once. This is often enough to cover short-form poetry (e.g. [55, 58]), but woefully inadequate for entire novels, as even novellas will typically have tens of thousands of tokens with novels frequently reaching the hundred thousand mark [54]. While transformer models with arbitrary input lengths can theoretically be trained, the attention mechanism leads to a memory requirement that scales quadratically with the input, making much larger sizes infeasible. More recently, approaches like Longformers [77, 78] have proposed methods to limit memory consumption, by limiting the attention mechanism, and thereby enable longer input sequences; Longformers, however, also only allow for input sequences up to 4096 tokens. We are not aware of work applying any of these approaches to CLS, presumably, at least in part, because even 4000 tokens are not typically sufficient to cover the entire subject of analysis. Another alternative exists in hierarchical transformers [79], which can potentially handle even longer sequences. But again, we have not yet encountered them in the application context of CLS. More recently, in the machine learning community, architectures incorporating explicit communication between segments have been proposed, outperforming existing models on long sequence tasks [80, 81]. In practice, the length limitation does not always present a problem, for example, Parigini et al. [51] perform token-based annotations and process the entire text in chunks. Such window-based approaches, however, bar the model from considering any text outside the current window, which might be an issue for some annotation tasks.

Another variant of the transformer model that was employed in the surveyed literature is the sentence encoder. Such models are trained to produce embeddings of entire sentences or even short documents representing, similarly to word embeddings, the sentences’ semantics in vector space. For example, Ehrmanntraut et al. [58] use them in addition to static word embeddings for identifying similarities in poetry. They find the transformer approaches to far outperform the static embedding-based ones, with performance on the overall similarity of embeddings reaching 0.79 as compared to the accuracy of 0.72 for the fastText approach. Similarly, Glass explores the use of sentence encoders to analyze if one text is an adaptation of another [40]. We did not find any instances of works making use of word mover’s distance (WMD), a technique for quantifying document distances given individual word embeddings, in our surveyed papers. Perhaps this can be explained by the approach’s time complexity, which can become prohibitive for long documents, although less expensive variants are available [82].

The mentioned shortcomings seem to not make transformers a natural fit for the domain of CLS. Despite this, as we found in our survey they are still the most prevalent machine learning methodology in recent literature in the CLS domain. We conjecture that this can be explained by the very good performance of transformers as compared to other techniques in language processing tasks (potentially outweighing domain adaptation issues) in combination with their relative ease of use.

3.5 Feature-based approaches

In our literature research, when leaving transformers aside, support vector machines (SVMs) were clearly the predominant means of approaching classification tasks. In terms of input features, we found uses of Tf-idf vectors, n-gram counts, and word counts for specific dictionary-based vocabularies. Some works also made use of preexisting classifiers or rule-sets to build features. SMVs, just as expected, perform well for classification tasks like genre identification [43].

Tf-idf is one of the most popular features in our review. It is a measure weighting the relative occurrence of a term (e.g. an individual word) in the current document by its rarity across the corpus, such that rare terms across the corpus that occur frequently in the document of interest receive a very high Tf-idf value. In this way, an individual document can be represented as a vector of real numbers where each element represents one term in the vocabulary. Orthogonal to this is the selection of what exactly constitutes a term in the vocabulary, typically individual tokens are chosen but alternatively character n-grams or word n-grams, that is to say all sequences of n characters or n tokens, could also be used. Brottrager et al. [56] employ a large variety of features to identify literary work that is likely to be externally reviewed, e.g. in a book review; from character-based metrics like the ratio of punctuation marks in the text to lexical ones like token n-grams, to semantic features like embedding cosine distance, and even complexity features like an ease of reading score. By contrast, Bonch-Osmolovskaya et al. [35] rely exclusively on Tf-idf, in their case the approach is likely to be token-based but details are not provided. With this comparatively simple method, they succeed in classifying diary entries into four classes, depending on what they describe.

Ciotti [37] made use of dictionary-based features, applying an Italian version of the LIWC [83] to collect the relative frequencies of words in specific categories (such as “Affect Words”, “Cognitive Processes”, or “Perceptual Processes”). Camps et al. [36] add musical features to their textual ones by, among others, adding bigrams of musical notes, as their subject of analysis is songs. They find character 3-g to be the best features for identifying song authors using an SVM on their dataset, with the feature reaching an F1 score of 0.79 outperforming word-lemmas at 0.67.

3.6 Topic modeling

Topic modeling is a technique to identify, for a corpus of texts, which topics are represented in which documents (see Sandhiya et al. [84] for a survey). Topics are generally inferred from the data in an unsupervised manner. In the case of Latent Dirichlet allocation (LDA), for example, topics are understood as a probability distribution across terms. A number of the works we surveyed [22, 60, 62] use LDA [85] as a means of topic modeling. Cooper et al. [22], for example, use it to segment their texts into topics to represent individual storytellers by means of the topic distribution their texts exhibit. Top2Vec [86] is an approach that, rather than relying on term distributions, encompasses existing embedding models to represent documents; it can be used in conjunction with universal sentence encoder (USE) embeddings, among other options. In our selection of papers, Top2Vec was employed by Zundert et al. [54], exploring if topic models can be used to identify literary genres as defined by publishers.

3.7 Stylometry

While stylometry does, strictly speaking, typically not qualify as a machine learning technique, we include it here for context as it is a very common technique. Stylometry in CLS is often performed using most-frequent-word (MFW) analysis, that is only the n most frequent words in the corpus are considered. Each document is represented by a vector with each element representing the occurrence count of one of the n MFWs. After normalization to account for the frequency of each token, one of a range of different distance metrics can be used. In this way, either individual documents or collections of documents (by taking the average vector), can be compared with a given document [87].

Eder [25] takes a fresh approach to this task by normalizing the number of tokens with semantically relevant words, which are extracted using static embeddings.

3.8 Metrics

In general terms, many of the metrics used in CLS are well-established in the machine learning community. For example, accuracy, F1 score, (e.g. [30]) but also BLEU (e.g. [28]), which was originally developed for machine translation evaluation. One metric that seems to generally be popular in the CLS community, as evidenced by its inclusion in MONAPipe [68], is Gamma; it can be used to evaluate classifier performance as well as inter-annotator agreement on span-based annotations. For example, Andresen et al. and Ehrmanntraut et al. [58, 88], employ it to evaluate their annotation agreement. Outside of our previously selected literature, Zehe et al. [89] employ it not only to measure inter-annotator agreements but also as a performance metric for their prediction system.

3.9 Model introspection

Some of the papers we surveyed make use of model introspection techniques. Kunilovskaya et al. [45], for example, use a linear kernel SVM to inspect feature weights and find five features that distinguish original Russian texts from translations to Russian (among them simple sentences and interrogative sentences). Steg et al. [48] take a similar approach, inspecting the importance of features (in this case individual tokens) in their Theil-Sen regressor using an existing implementation;^[4] they point out differences in the importance of first and third person pronouns depending on which theoretical approach to narrativity is taken. Crucially, both of these occurrences not only validate their models’ decision process but go so far as to derive literary insight by inspecting it.

3.10 Large and instruction-tuned language models

Since our survey considered works from 2022 and before, no instruction-tuned language models like ChatGPT were used. Instruction-tuned models [90] are Large Language Models (LLMs) that are tuned, based on human feedback, to provide helpful responses rather than exclusively being trained using self-supervised language modeling objectives. GPT-2, an LLM without instruction tuning, was used in our set of surveyed papers to generate poetry [49, 55]. In the near future, zero-shot methodology [91], that is to say inference using a pretrained model without training data, may simplify the annotation process for specific CLS tasks. Similarly, the few-shot capabilities of LLMs [92] could help scale annotations more quickly and simpler than traditional training approaches. That is to say, given only very few manually annotated examples, a large body of text could be automatically annotated with the given annotation schema for a specific concept allowing for corpus level analysis. Ziems et al. [93] evaluate the application of several LLMs, including ChatGPT, in social sciences while including a few tasks relevant to literary studies. In general, they find LLMs to potentially be ready for some zero-shot classification tasks in the context of social studies research. For example, ChatGPT perform particularly well on a stance detection tasks, where the objective is to identify a text’s position on a given topic like atheism or the legalization of abortion. Overall, in their experiments, models perform the best on misinformation classification, stance detection, and emotion classification. The authors attribute this to the fact that in each of these tasks there is either an objective knowledge-based ground truth (as is the case for misinformation) or an annotation schema aligned with colloquial definitions. They do however also find that models perform the worst on tasks that require complex expert taxonomies (meaning tasks where the annotation guidelines are informed by domain expertise), which tend to not semantically align with much of the LLM’s training data. An example of such a task is the annotation of character tropes, requiring the annotation of one of 72 tropes given a quote by a character. Aside from the large number of classes, this task is also difficult for LLMs because names of individual classes may not be easily understandable without further context, requiring expert knowledge to interpret the taxonomy. This finding seems particularly relevant in the context of the research process we outlined earlier, where theoretical work typically produces just such an annotation scheme. It remains to be seen how quickly these models are adopted in the CLS community, but we expect them to eventually see wide-spread use.

3.11 Domain-specific approaches

While some CLS methods align well with the methodology used for example in the NLP community, other methods are more specific to the field. For example, span-based text annotations are often performed in the field of NLP for tasks like named entity recognition (NER), whereas building graphs of character interactions is a technique more specific to the CLS domain.