Body language signals for rodent social communication

Integration of social cues to initiate adaptive emotional and behavioral responses is a fundamental aspect of animal and human behavior. In humans, social communication includes prominent nonverbal components, such as social touch, gestures and facial expressions. Comparative studies investigating the neural basis of social communication in rodents has historically been centered on olfactory signals and vocalizations, with relatively less focus on non-verbal social cues. Here, we outline two exciting research directions: First, we will review recent observations pointing to a role of social facial expressions in rodents. Second, we will review observations that point to a role of 'non-canonical' rodent body language: body posture signals beyond stereotyped displays in aggressive and sexual behavior. In both sections, we will outline how social neuroscience can build on recent advances in machine learning, robotics and micro-engineering to push these research directions forward towards a holistic systems neurobiology of rodent body language.


Introduction
Many social cues are nonverbal (a smile, a raised eyebrow, a shrug). A failure to correctly process and interpret social cues is thought to underlie social dysfunction in many neuropsychiatric conditions, from negatively biased interpretations of social signals in depression [1] to a nearcomplete breakdown of social understanding in some individuals with autism spectrum disorder [2]. A comparative investigation in rodents -where we have advanced tools for monitoring and manipulating neural activity during behavior -could be a powerful way to advance our understanding of the evolution and function of neural circuits for processing social cues [3].
In general, we know little about the use of posture and gesture in orchestrating social group behavior. A comparative study of body language is an old idea [4], but the systems neuroscience of rodent body language is still in early days. It is clear that rodents make use of stereotyped body postures and movements in sexual courtship (e.g. female rats darting) and in aggression and dominance (e.g. rat boxing) [5]. However, compared to our detailed knowledge about the processing of socially significant olfactory signals in aggressive [6], sexual [7] and parental [8] behaviors, we know much less about how body language signals (touch, movement, postures) are integrated by the rodent brain.

Facial expressions and whisking
Mice and rats display a variety of facial expressions. Both mice [9] and rats [10] make stereotyped expressions ('grimaces') with their facial musculature in response to pain and stress: tightening of orbital muscles, squinting eyes and retraction of the ears (Figure 1a). Rats also make facial expressions (forward movement and blushing of the ears) [11] (Figure 1b) and jumps [12 ] when experiencing positive emotions, such as after tickling. In wild mice, ear posture correlates with their behavior in tests that are thought to measure the animals' emotional state: approaching a novel odor and exploring the open arms in an elevated plus-maze. Mice with retracted ears behave more cautiously than mice with their ears in an upright, forward position [13]. Rats also display different facial expressions when presented with tastants that evoke different emotional responses (e.g. bitter, unpleasant quinine, and sweet and palatable sucrose) [14,15].
If rodent facial expressions differ between emotional states, that raises the possibility that these facial cues could be perceived by conspecifics and play a role in social communication. Increasing evidence across species suggests that facial expressions are displayed in social situations, and distinguishable by conspecifics. For example, ear wiggling is a social signal displayed by female rats during courtship [16,17]. Naked mole rats -a eusocial rodent species -have an extensive vocabulary of nonverbal body language, including elaborate facial interactions (e.g. head-on pushing, mouth gaping and tooth fencing) [18]. These facial interactions are involved in the control of 'lazy' workers [19] and help maintain reproductive suppression [20]. A landmark study found that when an intruder mouse was placed into the cage of  the resident mouse, the two mice displayed two different facial expressions, which they maintained, even during fighting: The resident displayed tightened eyes and flattened ears, while the intruder displayed widened eyes, erect ears and an open mouth [21 ] (Figure 1c). A recent study found that it was possible to train an image classifier to distinguish between facial expressions in head fixed mice in a wide range of situations (aversive and palatable tastants, LiCl-indcued nausea, painful electric shock, freezing) [22 ] (Figure 1d).
Rats are nocturnal [23], have modest visual acuity [24], and often encounter conspecifics head-on in burrows [25]. This suggests that ethologically they more often would sense faces of conspecifics with the whiskers rather than by vision. Beyond palpating the face of a social interaction partner, the whiskers themselves might also convey information, social or otherwise. During rat social facial interactions, whiskers are more protracted in aggressive than in nonaggressive interactions (Figure 1e), and female rats whisk with a lower amplitude when meeting male conspecific than when meeting female conspecific  [60][61][62][63] differ from the encoding of non-social stimuli, such as objects and textures [64,65].
Engineering advances in miniaturization make it now possible to record gaze direction and eye movements -and likely also whisker movements -by head-mounted cameras in freely-moving rats [66 ] and mice [67,68] (Figure 2f). Using this approach, it was recently reported that mice close their eyes when a conspecific is within close distance [69 ] -an unexpected and interesting observation in the context of making and recognizing facial expressions. With miniaturized, head-mounted thermocouples, accelerometers, gyrometers, and Hall-effect probes, it is possible to quantify sniffing patterns, nose movements, and head-bobbing in freely-moving animals [27,31 ] (Figure 2g-h). It remains unclear how these aspects of facial behavior vary during social interactions. Alternatively to using head-mounted cameras to record facial behavior in freely moving animals, a recently described method combines real-time tracking with motorized cameras to capture high-resolution 'close-up' images of animals moving in a large 3D arena [70 ].
Quantifying whisker movements during social facial interactions remains a challenge. Whisker tracking of solitary animals has reached high levels of accuracy. In head-fixed mice with most whiskers trimmed, simultaneous measurements of the three-dimensional shapes and kinematics of eight whiskers can be obtained automatically [71]. However, in socially interacting animals (with full, un-trimmed whisker fields), overlapping and occluded whiskers remain a major problem, and thus far social whisking patterns have either been tracked manually [26,30,60,61,63] or approximated by automatically tracking the average movement of the whisker field as a whole [61]. A promising path towards automatic whisker tracking in socially-interacting animals is to combine recent advances in automatic whisker tracking in freely-moving animals [72] (Figure 2i) with techniques for tracking the movement of single whiskers despite overlaps and occlusions by painting single whiskers with a fluorescent dye [73,74] (Figure 2j).

Posture and movement as body language signals
A role of body language in signaling distress Several studies have shown that rats will actively help conspecifics in distress. Rats will press a lever to lower a distressed and wriggling rat dangling in a harness (prodded with a sharp pencil if it did not exhibit sufficient signs of discomfort) [75 ] (Figure 3a), rats will press a lever to remove a conspecific from a water tank [76], and rats will leave a dark and comforting hiding place and stay in a brightly lit, open arena to ensure that a nearby conspecific does not receive a painful shock [77 ] (Figure 3b). More recent studies have shown that rats [78] and mice [79] will open a door to release a conspecific trapped in a small plastic tube (Figure 3c), and that rats will open a door that lets a cagemate escape a pool of water [80] (Figure 3d).
What drives the behavior of the helper animal? In rats, restraint-tube-opening behavior depends on familiarity with the strain of rat in distress [81]. Behavioral changes occur after drugging the helper rat, with benzodiazepine sedation leading to longer opening latency [82] and heroin administration abolishing opening [83]. Door-opening latency changes if there are multiple potential helpers, and depends on if these 'bystanders' are sedated [84].   (f) A head-mounted camera system for mice [67]. (g,h) Head-mounted, physiological sensors systems for recording of sniffing, nose movements and head-bobbing in rats [31 ]. (i) Automatic whisker tracking in a freely moving mouse [72]. (j) Using fluorescent dye to visualize single whiskers within the whisker field [74] Relatedly, the behaviors of the distressed animal might also be an important factor. Trapped animals produce lower-frequency distress calls in the first restraint sessions [78,81], but might also display other signs or signals of stress, such as seen in pain and sickness [23,96]. This raises the possibility that other signals such as olfactory cues [97,98] or elements of body language such as gesture and posture could be used to signal distress and solicit help. Several studies have found rodents are indeed sensitive to body language signals of distress, such as freezing [99,100], and rats prefer a room decorated with images of conspecifics in a neutral pose rather than a room decorated with images of conspecifics in pain (i.e. facial grimaces and hunched posture) [101 ].
Controlled experiments involving robotic animals [102] or virtual animals [103] is powerful way to probe the sensitivity of animals to visual social stimuli. Some studies have simulated body language distress signals by robotic animals [102]. Rats will work to release a moderately ratlike robot from a restraint tube, and rats seem to discriminate between robots based on behavior [104 ]. There is ongoing work to develop more complex rat robots, capable of realistic postures and movement patterns [105,106 ,107].

Instructing social partners through body language
Several studies have investigated the behavior of rats in artificial social games which also might involve body language. Rats will cooperate at rates above chance level in iterated prisoner's dilemma games [108][109][110], but the interpretation of such games is complex, since an iterated prisoner's dilemma can be dominated without any theory of mind [111]. Body language is usually not quantified but a classic study reported that cooperation would break down if the animals could not see each other, and that rats would engage in specific left-turning or right-turning feint behaviors apparently to influence the behavior of the partner animal [108]. The importance of visual observation has been highlighted in another social coordination nose-poke task [112].
Multiple studies have found that rats will work to deliver food to conspecifics [113-116,117 ,118], and that -when given the option to donate food to conspecifics at no extra cost to themselves -rats will prefer that conspecifics receive food also [119,120,121 ,122]. Many of these Rodent body language Ebbesen and Froemke 95  Investigating body language signals of distress. (a) A classic paradigm to investigate active helping in rats: Aversive dangling from a harness. [75 ] (b) Another classic paradigm: One rat has to decide between a brightly lit arena and a dark and comforting shelter. Entering the shelter delivers a shock to a nearby conspecific. [77 ]. (c) The aversive-restraint-tube paradigm: One rat can open a door that lets a conspecific escape a transparent tube. [78] (d) The aversive-swimming-pool paradigm: One rat can open a door that lets a conspecific escape a pool of water. [80].
studies report observations that are in line with the supposition that rats use body language signals to communicate what they want the 'chooser' animal to do, and that the chooser animal is sensitive to these signals. In one study, the likelihood of donating food by the chooser rat incurring no extra cost, was modulated by the display of food-seeking behavior by the prospective recipients, expressed as poking a nose port and by social interactions through a mesh [121 ]. In another study, where rats could work to deliver food to a conspecific only, subject rats provided food correlated with the intensity of movements and body postures displayed by the prospective recipients. These putative body language signals included stretching their paws towards the food, sniffing through the mesh in the direction of the food, and other attentiongrabbing behaviors directed at the subject rat [117 ].
Studies investigating the behavior of groups of mice in complex environments have found marked individual differences in displays of social postures and movements [123,124], and patterns in social interaction partnering [125][126][127][128]. However, while postures and movement patterns correlate with social dominance [123,129], it is still unclear if and how body language cues might help establish, maintain, or adjust the dominance hierarchy [123] or social networks of co-habituation [125].
Individual differences in movement and postures during co-housing or colony dynamics might mirror the observation that animals tend to take on different behavioral roles. When rats are moving together in dyads, some become 'leaders' and some become 'followers' [130].
In a test where rats have to dive underwater to collect morsels of food, some become 'divers' (swimming and collecting food), and other rats become non-divers which wait for the other animal to bring them food [131,132]. In wild mice performing collective nest building, some mice will become nest-builders (carrying out the vast majority of the work in collecting nesting material) and some mice will only participate weakly or not at all [133].
One important set of behavioral roles in group-housed animals is co-parenting and caretaking of infant rodent pups. Parenting behavior and active care for pups is orchestrated by innate circuits to some degree [8]. In the context of body language, however, maternal female mice can solicit the help of sexually experienced males [134][135][136] and virgin females [137][138][139] for aspects of pup caretaking (e.g. nest building, pup retrieval, crouching, and pup grooming). The parental behaviors expressed by males and female virgins develop with exposure or experience with pups (concaveation) and during co-housing with a dam and litter. The presence of experienced dams accelerates concaveation [139,140 ] indicating that dams engage in some behaviors or interactions that affect the emergence of co-parenting abilities in males or virgin females. Olfactory and auditory cues from the dam play a role. Blockade of these signals delay the development of co-parenting in males and -even without visual input from the dam -replay of dam vocalizations or dam odors can induce parenting in males [135]. Body language and motor activity of the dam also contribute, as dams will actively engage virgins in maternal care by 'shepherding' the virgins to the nest and pups. Furthermore, dams demonstrate maternal behavior in spontaneous pup retrieval episodes that allow virgin females to learn by observation [140 ]. Active social engagement and demonstration by dams might be a key driver in facilitation fast social learning of co-parenting. Free-living, wild dams selectively choose to communally nurse [141,142 ], but we do not yet know the role of body language signals in coordinating co-parenting between dams in outdoor colonies.
The apparently active demonstration of parenting is in contrast to studies reporting that wild rats do not rapidly acquire new foraging techniques by observation, even if they are performed by conspecifics [143]. It is, however, in line with other reported examples of rodents learning by observation [144]. For example, rats can learn to solve a Morris water maze by observing a trained conspecific swim to the hidden platform [145], mice can learn to solve a complex 'puzzle box' by observing conspecifics [146], and rats will imitate joystick movements that they have seen a conspecific make to receive a food reward (even though their joystick movements do not actually affect their own reward at all) [147].

Pose estimation and quantitative analysis of body language
Methodological advances in computer vision and machine learning provide new ways to monitor and analyze body language signals for social behavior. Multiple open source packages for machine-learning based markerless tracking of posture and body parts in single animals have recently been developed, including Dee-pLabCut [148 ], LEAP [178], DeepPoseKit [150], Opti-Flex [151], DeepGraphPose [152] and others [153]. However, translating single animal tracking to multiple animals is not straightforward, for at least two reasons. First, the camera view on a specific animal might be occluded by other animals (especially if any have neural implants). Second, even if all body parts are visible, the body parts have to be 'grouped' correctly and assigned to the correct animal.
One straightforward way to distinguish two interacting animals is to use animals that are physically marked or of a different coat color [154][155][156] (Figure 4a-c). This is a robust method, but excludes some use-cases (e.g. studies of behavioral genetics that require a specific background or where it is important that animals come from the same litter). Another method have used deep neural networks to recognize body parts and metrics of spatiotemporal continuity to group body parts and maintain tracking of animal identities, in unmarked animals of the same coat color [149 ] (Figure 4d). Another approach maintains the identities of multiple animals by training a network to recognize subtle differences in the appearance of individual animals [157,158 ,159 ] (Figure 4e). Another approach combines the use of implanted RFID chips [126,160]  Beyond recording raw postural and movement data, machine learning methods have also provided new ways to segment raw tracking data into behavioral categories in a principled and objective manner, and to discover behavioral structure -the building blocks of body languagein a purely data-driven way. The latter is especially promising, because it could allow discovery of new postures and movement patterns, purely from statistical properties in the behavioral kinematics and agnostic to potential observer bias.
A very effective way of automatically segmenting raw tracking data is to use a supervised approach and train a classifier to reproduce human annotation of behavioral categories [154][155][156]. This approach will, when using modern, deep-leaning based classifiers and large training sets [155,156], provide a precise way to automatically annotate behavioral data. Unsupervised approaches learn Rodent body language Ebbesen and Froemke 97  the behavioral categories from the data itself. Tracked behavioral features from an animal (e.g. 3D coordinates of many body parts) is a high-dimensional time series. To find structure in such, it is possible to draw from a recent work in laboratory studies of worm and insect behavior ([168,169]) and field ethology [170,171].
One approach to discover behavioral categories is to look for 'building blocks' of the observed behaviors that reoccur. To this end, an elegant and robust approach is to perform a nonlinear projection from the high-dimensional space of all tracked body part coordinates (often augmented with derived features, such as time derivatives and spectral components) down to a low-dimensional 2D [172 ,173-175,176 ,177-180] or 3D manifold [181,182] in a manner that preserves local similarity (e.g. t-SNE [183]). On this low-dimensional manifold, similar, reoccuring movements and postures will form clusters, that can be identified by density-based clustering algorithms. The generated clusters are manually inspected and curated (e.g. merged or split) and assigned names (e.g. 'locomotion', 'grooming', etc.).
Another approach to discover behavioral categories is to define a generative model -for example, some flavor of state space model -and fit this model to the high-dimensional time series of tracked body features [165 ,167 ,184,185 ,186,187 ,188-190,191 ,192]. This approach is attractive, because it is highly expressive: It is possible to define very complex models, for example, by adding autoregressive terms [165 ], by allowing for complex hidden dynamics [193 ], by incorporating nested structures [191 ], and by including nonlinear transformations [185 ]. It is also possible to explicitly incorporate knowledge about the animals anatomy, by writing a full generative model of the animal's body itself, akin to [175,194 ]. However, these methods also have drawbacks. First, complex models quickly become prohibitively computationally expensive to fit to data. Fitting can be accelerated, for example, by using fast modern and efficient sampling algorithms [195] or GPU-accelerated variational inference [167 ], but even these methods often show poor mixing/convergence for complex models. Second, even if a model is well fit to data, there is no principled way to discover what the 'true' latent structure is (e.g. the true number of hidden states or transition graph structure) [184,[196][197][198]. Thus, for example, the number of hidden states in a state space model of behavior -that is, the number of different behavioral categorieshas to be set using a heuristic, for example, by fitting a model with the number of latent states as a free parameter and then choosing a cutoff [165 ,190,192], by fixing the number of states based on inspection of raw data and the desired coarseness of the model [167 , 188,191 ], or -in a very elegant approach -by comparing models with different latent structure according to their ability to capture multiple aspects of the observed data, such both the most likely state and transitions between states [185 ].
Machine learning based approaches for behavioral tracking and analysis are in continual development along several directions that are of particular interest to the analysis of socially-interacting animals. For example, there are several methods for estimating 3D locations of body parts by triangulation of multiple simultaneous 2D views of the animal [199][200][201][202], but such triangulation methods are sensitive to occlusions and thus difficult to use in interacting animals. A recent report showed, that after having collected one good 'ground-truth' multi-view 3D dataset, it was possible to train a network to predict the 3D posture of an animal from a single 2D view only [203 ]. Building upon work in humans, it might even be possible to learn 3D body skeletons from only 2D views, that is, without the need to capture a ground truth 3D data from multiple cameras in the first place [204 ]. Such methods for estimating the 3D posture from a single 2D view could be a very powerful way to deal with camera occlusions in studies of interacting animals.
An elegant way to improve unsupervised behavioral clustering is to do everything in a single operation, whereby a deep neural net simultaneously learns to project the data onto a low-dimensional manifold and estimate an optimal number of latent clusters according to single objective function [205 ,206 ]. Another promising approach is to use a dictionary-based approach to identify behavioral categories as sequence 'motifs' in the raw tracking data [207 ].
When analyzing the behavior of single animals, some studies have eschewn body part tracking altogether and identified behavioral categories by fitting state space models directly to video data [165 ,190,192,208] or by training a network to replicate human labeling directly from raw video [216]. It would be very useful if these approaches can be modified to handle multiple animals in the same video. This challenge is difficult, not just due to occlusions, but because multiple animals are interaction and thus will have complicated between-animal statistics. Writing a generative model of two animals is more challenging than utilizing two copies of a generative model of a solitary animal. In fact, to understand the structure of rodent body language, and its neural basis, these between-animal statistics are critical to document in high resolution. For example, running towards a conspecific or running away from a conspecific have a very different social 'meaning', but may be identical in the kinematic space the single animal, if that animal is modeled in isolation.
Mathematical methods for understanding the behavior of interacting animals are still in active development, with many important and open questions to work on. Recent reports have used unsupervised methods to elucidate how the behavior of interacting Drosophila depends on the relative spatial location of the interacting animals [177] and the animals' behavioral state (e.g. courting or not) [176 ]. In ethology, there is related work in the use of information theory [209], modeling [210] and network theory [211 ] to understand the role of social interactions in determining collective movement, for example, in fish [212] and baboons [213]. One recent study combined information-theory and presentation of robotic conspecifics to understand the statistics of dyadic interactions in zebrafish [214 ] and another study used purely statistical methods to discover that rats rely on social information from conspecifics when exploring a maze [215 ].

Conclusions
As outlined above, some methods for automated behavioral analysis -for example, those that discover behavioral structure directly from raw video [165 ,190,192,208] -do not return an explicit physical 'body model' of the animal; only discrete behavioral categories. For many biological questions, such an 'ethogram-centric' view has no drawbacks, but when relating neural data to behavior, continuous information about movement and posture kinematics can be critical. Neural activity is modulated by motor signals [217][218][219] and vestibular signals [220][221][222] in many brain areas. To understand how neural circuits process body language cues during social interactions, these 'low-level' motor and posture related confounds must be regressed out. For example, it is in principle not enough to know that activity in a brain region is different during mutual allogrooming than during boxing to conclude that neurons in this brain region is responding to a difference in social 'meaning' (e.g. aggressive, but not agonistic behaviors). Differences in neural activity between behavioral categories might just as well be related simply to 'low-level' differences in movements and postures made by the animals in those different behavioral categories. Regressing out confounding low-level motor and postural signals is a difficult task. Without a body model, it is possible to regress out some variance by regressing the neural activity onto variance in the raw video itself, for example, regress out activity related to face movement by regressing onto principal components of a video of the face [223,224 ]. However, movement and posture signals are generally aligned to the animals own body in some form of egocentric frame of reference, for example, to muscles, posture or movement trajectories [225]. The transformation from body to video is highly non-linear and difficult to discover automatically.
There are related complexities when interpreting differences in neural activity in social situations that are associated with sensory input, for example, social events that include vocalizations and social touch. Social touch widely modulates the brain from the hypothalamus [226] to frontal and sensory cortices [62]. Moreover -in context of understanding the structure of body language -it is likely essential to know if a close contact between animals included a social touch or not. While behavioral tracking methods that can estimate the animal 3D posture as a 'skeleton' of body points are suited for regressing out signals due to the animals' own posture and movement, a full, deformable 3D surface model of the animal is required to measure social body touch. To this end, there are also promising machine learning methods on the horizon. For example, starting from a detailed, deformable 3D model of the animal's shape and color, it is possible to extract a detailed 3D model of an animal's body surface from a single 2D view, even in complex images [227 ,228-231,232 ].
Rodents display a wide range of facial expressions (grimaces and whisker movements), including during social interactions. Multiple observations suggest that body postures and movements of conspecifics function as an important social signal. Recent major advances machinelearning methods for behavioral analysis and microengineering of behavioral sensors are making it possible to quantify facial expressions and body postures during complex, social interactions. These data will reveal new questions about the neural basis of social cognition in rodents to understand the comparative neurobiology of body language.

Conflict of interest statement
Nothing declared.

57.
Goldin MA, Harrell ER, Estebanez L, Shulz DE: Rich spatiotemporal stimulus dynamics unveil sensory specialization in cortical area S2. Nat Commun 2018, 9:4053 Used a multi-whisker stimulator system that can move 24 whiskers independently to investigate how spatiotemporal features of whisker stimuli are encoded in secondary somatosensory cortex.

167.
Ebbesen CL, Froemke RC: Automatic tracking of mouse social posture dynamics by 3D videography, deep learning and GPUaccelerated robust optimization. BioRxiv 2020 Uses deep learning, physical modeling and a particle-filter based tracking algorithm with spatiotemporal constraints to automatically track the body postures of interacting mice, compatible with electrophysiology (occlusions and camera artifacts due to wires and a neural recording implant). 171. Smith JE, Pinter-ollman N: Observing the unwatchable: Integrating automated sensing, naturalistic observations and animal social network analysis in the age of big data. J Anim Ecol 2020. n/a.