Behavioral Correlates of 50-kHz Ultrasonic Vocalizations in Rats : Progressive Operant Discrimination Learning Reduces Frequency Modulation and Increases Overall Amplitude

Ultrasonic vocalizations (USVs) in rats are thought to contain ecological signals reflecting emotional states. These USVs are centered on 50-kHz, and frequency modulation (FM) is hypothesized to indicate positive emotion; however, results from recent studies are inconsistent with this hypothesis. We suspected that such inconsistencies might result from ambiguity in defining frequency modulation, and problems with acoustic analyses and behavioral protocols. We addressed these problems by applying quantitative methods for USV analyses and using a food reward operant paradigm. Our results revealed that frequency modulation varied according to the degree of positive outcomes, but the direction of change was opposite to what had been observed in previous studies. The FM in 50-kHz USVs decreased as animals learned the task and obtained more reinforcement, while USV amplitude increased as learning progressed. To reconcile these results with those from prior studies, we suggest that FM in 50-kHz USVs should be taken as an index of reward prediction errors, and USV amplitude should be considered as an index of positive emotion.

Rats (Rattus norvegicus) vocalize in the 2 -4-kHz audible range, and the ultrasonic range above 20-kHz is beyond the human audible range (Anderson, 1953).Calls above 20-kHz are called ultrasonic vocalizations (USVs).The 50-kHz USV is a type of USV among adult rats.The range of frequency at peak energy is from 32-kHz to 96-kHz, the mean bandwidth is around 1-kHz to 7-kHz, and the duration is around 30-50 ms (Portfors, 2007).Moreover, recent studies suggest that 50-kHz USVs can be subdivided into "frequency modulated" (FM) and "flat" calls (Burgdorf & Panksepp, 2006).More FM calls are produced following administration of drugs that act on dopaminergic systems (Ahrens, Ma, Maier, Duvauchelle, & Schallert, 2009;Browning et al., 2011).In contrast, fewer FM calls are produced after removing or blocking dopamine receptors in the medial forebrain bundle (Ahrens et al., 2009;Ciucci, Ma et al., 2007), which connects the ventral tegmental area (VTA) and the nucleus accumbens.Flat calls are not affected by appetitive or food reward contingencies (Burgdorf et al., 2008).Such observations indicate that 50-kHz USVs, especially FM calls, could reflect positive emotion caused by appetitive states such as reward expectation (Ahrens et al., 2009).Thus, these USVs could be a good index of vulnerability to habit-forming drugs (Taracha et al., 2012).
These 50-kHz USVs share neural foundations with human utterances and facial expressions (Burgdorf & Moskal, 2010).In animal behavior, only hedonic taste reactivity of many nonhuman species (e.g., primates to rats; Berridge, 2003), and rat 50-kHz USVs, are thoroughly parallel to be used as an index of positive emotion.While 50-kHz USVs can be evoked in many situations, hedonic taste reactivity can be evoked only in oral intake situations.Thus, 50-kHz USVs are expected to be a plausible general index of positive emotion.
However, some studies have raised doubts as to whether 50-kHz USVs function as an index of positive emotion.For instance, repeated morphine injections reduce the number of 50-kHz USVs during daily morphine administration (Hamed et al., 2012).The 50-kHz USVs do not increase during reinstatement sessions following cocaine self-administration training (Barker et al., 2013).Conditioned place preference (CPP) using morphine can be established, but rat 50-kHz USV production does not increase at the completion of CPP conditioning (Wright, Deng, & Clarke, 2012).Neither FM nor flat calls are correlated with time spent investigating a social stimulus (Willey & Spear, 2013).
Moreover, some authors have reported that non-positive events can evoke 50-kHz USVs.For example, exposure to CO2 (Niel & Weary, 2006), pain caused by antimicrobials (Dinh, Larkin, Gatlin, & Piepmeier, 1999), and invasive situations (Haney & Miczek, 1994) evoke 50-kHz USVs.Furthermore, individuals experiencing an experimental paradigm produce more flat calls than naïve individuals would (Wöhr, Houx, Schwarting, & Spruijt, 2008).Therefore, it is not conclusive whether 50-kHz USVs, especially FM calls, are completely associated with positive affect.Some of the disagreement could be caused by methodological differences between studies.For example, the amount of morphine used as a reward was smaller compared to a similar study that was consistent with the positive emotion hypothesis (Wright et al., 2012).Additionally, the experimental USV data were recorded in an experimental condition where another individual (used as a social stimulus) was present.Thus, although behavioral data was recorded only for one animal, individual vocalizing was not identified (Willey & Spear, 2013).Wright et al. (2012) claimed that these discrepancies could be accounted for by differences in recording methods.The heterodyne method and frequency-divided acoustic signals method were used in most previous studies, but these methods suffer from lower reliability (Hamdani & White, 2011;Parsons, Boonman, & Obrist, 2000) than a spectrographic analysis of broadband signals used in the Wright et al. (2012) study.Therefore, Wright et al. (2012) suggested that prior results were unreliable due to variable recording methods.
Contradictory results could also come about from differences in detection criteria or methods for classifying 50-kHz USVs.Detection and classification methods of 50-kHz USVs are mixed; they can include visual inspection (Burgdorf & Panksepp, 2006), difference between the maximum and minimum peak frequency (Burgdorf et al., 2008), or other acoustic parameters (Wright, Gourdon, & Clarke, 2010).Moreover, the proposed number of 50-kHz USV subtypes has ranged from 2 (Burgdorf & Panksepp, 2006) to 14 (Wright et al., 2010).Given such differences among studies, the reliability of detection criteria and classification methods might be limited, affecting the results.
If 50-kHz USVs, especially FM calls, are truly emitted in association with positive emotions, presenting food pellets should yield more 50-kHz USVs.Thus, we created an experimental situation where rats gradually learned to associate nose insertion into poke holes in an operant box and the presentation of pellets.If FM calls increase gradually with the number of pellets obtained by the experimental group, trained on associative learning, as compared to the control group, 50-kHz USVs should be emitted in association with positive emotions.To address the reliability problems noted by Wright et al. (2012), we conducted a spectrographic analysis of broadband signals in rat calls.Furthermore, we did not analyze calls through categorization but treated calls as continuous entities.

Method
All experiments were performed in accordance with the guidelines of the Animal Experiment Committee at the University of Tokyo (Permission number 22 -5).

Animals and Apparatus
Eight adult Long-Evans rats (males; 4 experimental and 4 controls) were used in this experiment.Water was available ad libitum, but food was restricted to maintain approximately 85 -90% of free feeding weight.All experiments and recordings were conducted in an experimental chamber (ENV-009L, Med Associates, Vermont, USA) placed in a soundproof box (Muromachi Kikai, Tokyo, Japan).The chamber had 9 poke holes (2.5 cm × 2.5 cm × 2.2 cm) on one side and a pellet dispenser on the other side.When an infrared beam passing through a poke hole was interrupted by a rat's nose, the signal was transmitted to a control device.Moreover, each hole had a light inside; thus, we were able to turn the light on or off. Figure 1A shows our experimental set up.

Procedure
After one week of handling, five days of habituation to the chamber and a one-day habituation to pellet dispensers (experimental group only), the rats were trained on a differential conditioning task with a nose poke as the operant (Figure 1B).
One trial of the task consisted of a waiting-phase and a response-phase.When a trial started, the rat was required to wait for 10 s without a nose poke.This period was the waiting-phase.If the rat's nose poked into the hole during the waiting phase, the phase restarted from the beginning.This phase finished after 10 s elapsed without a nose poke.When 2 of the 9 poke holes were lit, a nose poke was permitted.Selectively poking one of the two lit holes by the nose was defined as a correct behavior and reinforced by one pellet, and the trial was then completed.This period was the response-phase.If the rat poked his nose into an unlit hole during the response-phase, the lit holes would turn off, and a 5 s time out was inserted as punishment.Therefore, rats were reinforced by a pellet only if they poked their nose into 2 of the 9 holes lit after a 10 s suppression period.Experimental groups performed the task for 60 min per day, which was defined as one session.Sessions were conducted seven days per week.Hit rate was defined as the number of correct responses divided by the total number of trials.The suppression time was increased to 15 s to suppress excessive reward gain after the number of pellets obtained within a session exceeded 120.
Waiting time was 15 s, and a hit rate greater than 0.7 was established as successful conditioning.Thus, the number of successful trials was equal to the number of pellets obtained whereby animals showed progressive conditioning.The control group also entered the operant box for 60 min per day, but no light was turned on and actions were not reinforced.

Ultrasound Analysis
We analyzed the first 10 min of each session using SAS Lab-Pro 5.2.06 (Avisoft Bioacoustics, Berlin, Germany).The FFT length was 256, the overlapping time window was 50%, and the spectrogram resolution was 997 Hz.Prior to the main analyses, recordings were high-pass filtered at 20-kHz, and remaining noise was removed using signal processing software (Gold wave, Gold wave Inc., NL, Canada).Further noise removal, if required, was performed manually using a standard eraser cursor in Avisoft SAS Lab-Pro to remove pulse like noise above -50 dB.A call, which was separated from the previous call beyond 100 ms, was regarded as an "other call."If calls could not be differentiated from noise, or lasted less than 3 ms, they were excluded from the analysis.Thus, 6,016 calls were used for the current analysis.
We detected calls in the pre-processed data (threshold -50 dB hold time 100 ms) and calculated parameters, per ms, by applying an "automatic parameter measurements" function in the recording software.The parameters employed were bandwidth, duration, distance from the start to max, fundamental frequency, max harmonic-to-noise ratio, mean and max amplitude, and sub-element number.These parameters could be directly measured.Details of acoustic parameters are found in the SAS Lab version 5.2 manual.Briefly, bandwidth was the difference between the max and mininmum frequencies in one bin.Duration was the distance from the point where the signal exceeds -50 dB to the point where the signal falls below -50 dB.Distance from the start to max was the point from where the signal exceeds -50 dB to the point where the maximum amplitude was measured.Fundamental frequency was the lowest peak of frequency.Harmonic-to-noise ratio was the index of the degree of contamination in non-harmonic energy.Mean and max amplitude was the mean and max loudness.
A frequency modulation of 50-kHz USVs consisted of the frequency jump and frequency oscillation as defined in Brudzynski (2013).Here, the number of sub-elements should be an index of the degree of frequency jump.Next, the call that was separated from the previous call within 100 ms, and beyond 3 ms of the call, was defined as a sub-element.Thus, continuous USVs should have 1 subelement and non-continuous USVs, which have a break within 100 ms, and those beyond 3 ms should have 2 sub-elements.However, it was difficult to directly estimate the degree of frequency modulation by these default variables.Thus, we considered that the degree of vertical modulation should be reflected in the number of infraction points in frequency and the modulated frequency distance.We calculated 2 parameters using fundamental frequency per millisecond, the number of vertical modulations of the fundamental frequency, and the total modulation distance (Figure 2).The number of vertical modulations of the fundamental frequency was considered to be equal to the slope change by a drawn line between adjoining fundamental frequency points per ms.The total modulation distance was considered to be equal to the sum of the absolute difference between adjoining fundamental frequencies per ms.Prior to these analyses, these values were normalized by the duration of each USV.The number of vertical modulations (inflections) of the fundamental frequency within the call.For example, if fundamental frequency increased between 1 and 2 ms point, but decreased between 2 and 3 ms point, we regarded that there should be one vertical modulation between 1 and 3 ms.The total modulation distance (in kHz) was considered to be equal to the sum of the absolute difference between adjacent fundamental frequency points per ms.This is equal to the cumulative frequency modulations.The total modulation distance is 0-kHz if there was no modulation in fundamental frequency.

Statistical Analysis
We used the general linear mixed model (GLMM) in R.2.15.2 to evaluate the effects of operant conditioning on the quantity and quality of 50-kHz USVs.The GLMM enables us to correctly estimate the effect of the independent variable while considering individual differences.Specifically, the GLMM first estimates effect sizes of the independent variable per individual.Second, the GLMM estimates the overall effect size of the independent variable using estimated values per individual.Therefore, the overall effect size was weighted by the degree of deviation between individuals.Thus, problems caused by a small sample size were taken care by this statistical procedure.We then conducted a likelihood ratio test between the model that contained only the random effects and the model that added the fixed effect to reveal the significance of the fixed effect.Additionally, we used a single regression analysis to remove bias in the data from the effects of repeated sessions.

Descriptive Statistics
Descriptive statistics (mean ± standard deviation) were assessed before testing effects of the experimental manipulations.We confirmed that fundamental frequency was 47 ± 8.1-kHz, sub-element number was 1 ± 0.67, fundamental frequencies were vertically modulated 0.3 ± 0.04 times per ms.The fundamental frequency of USVs typically increased from the start to the end.Specifically, the fundamental frequency increased by 2.3 ± 6.0-kHz from the start to the center and increased by 1.3 ± 6.6-kHz from the center to the end.Mean bandwidth was 4.1 ± 0.62-kHz, and the duration was 26 ± 15 ms.These results were consistent with the description of 50-kHz USVs in Portfors (2007) and illustrated a normal variation of rat USVs.

Differences in 50-kHz USVs between groups
We found that the experimental group produced fewer inconsecutively modulated USVs than the control group.In this analysis, groups were used as the fixed effect.The total number of 50-kHz USVs did not differ between groups (Fig. 3E); however, the number of sub-elements was significantly lower (p < 0.005; Fig. 3A), while max-amplitude (p < 0.05; Fig. 3B) and mean amplitude (p < 0.01; Fig. 3C) were significantly higher in the experimental group, and duration from start to max was shorter (p < 0.05; Fig. 3D).

Changes in 50-kHz USVs Within the Experimental Group
As operant conditioning progressed, continuous FM in the 50-kHz USVs decreased within the experimental group.Thus, the number of pellets obtained was used as the fixed effect.The total number of 50-kHz USVs did not change with progressive conditioning (Fig. 4F), but the mean amplitude (p < 0.05; Fig. 4A), duration (p < 0.05; Fig. 4B), and the max harmonics-to-noise ratio (p < 0.001; Fig. 4C) increased.In contrast, the number of vertical modulations (p < 0.05; Fig. 4D) and the total modulation distance (p < 0.01; Fig. 4E) decreased with progressive conditioning.

The Effect of Rat Body Weight on USVs
Schwarting, Jegan, and Wöhr (2007) reported that the number of calls is negatively correlated with a rat's body weight.We confirmed this effect on the number of USVs and each acoustic parameter.Weight was measured immediately before the experiment each day.For this analysis, we added groups as random effects to the GLMM and tested the effect of body weight on all rats for each session.
Results revealed that the total number of USVs increased with body weight (p < 0.005; Fig. 5A).
Additionally, depending on body weight, the value obtained by subtracting the bandwidth at the center of the call from the start increased (p < 0.05; Fig. 5B), and the value obtained by subtracting the bandwidth at the end from the center decreased (p < 0.05; Fig. 5C).These changes were caused by a decrease in bandwidth at the center of calls (p < 0.05; Fig. 5D) as a function of body weight.The remaining acoustic parameters were unaffected by body weight.

Discussion
Some of the acoustic parameters, but not the total number of 50-kHz USVs, differed between groups and changed as conditioning progressed in the experimental group.These findings were somewhat unexpected because the number of pellets animals obtained increased as the session progressed, and this change should have yielded more FMs, which are considered to be associated with positive emotion.Simola et al. (2012) showed that amphetamine and methylphenidate increases the total number of 50-kHz USVs, as well as modifying acoustic parameters.In contrast, MDMA, morphine, and nicotine did not yield more 50-kHz USVs and did modify acoustic parameters.These results suggest that 50-kHz USVs are affected differently by various reward types.We propose that pellets presented as reward, like MDMA, morphine, and nicotine, affect some of the acoustic parameters but not the total number of 50-kHz USVs.
We found that frequency modulation differed with conditioning state and varied as conditioning progressed, which should result in positive emotion.However, the control group exhibited more noncontinuous frequency modulation than the experimental group.Furthermore, USVs gradually flattened as the number of pellets obtained increased with the progress of conditioning.It is less likely that rats were in a more positive state at the beginning of conditioning than at the end.Our results seemed to be completely opposite to prior findings that suggest frequency modulation increases with positive affect.
Most previous studies did not distinguish between continuous and non-continuous frequency modulations in USVs (Figure 6).However, we found that such modulations were affected by different factors.These factors may correspond to those identified by Riede (2011): continuous frequency modulation reflects rapid small-amplitude modulation in the sub-glottal pressure signal, but noncontinuous modulation does not reflect the pressure pattern.Fewer sub-elements and fast rise USVs in the experimental group should be attributed to rats' internal states.Progressive conditioning did not change the number of sub-elements and the distance from start to max.Thus, the number of sub-elements and the distance from start to max could be affected by specific differences between groups (e.g., such as expectation for food reward).The association between reward and behavior was unique to the experimental group and should produce higher levels of arousal than in the control group.
The flattening of 50-kHz USVs, greater harmonics-to-noise ratio, and longer duration with increasing number of pellets obtained in the experimental group should also be considered as results of specific changes within the experimental group.This is because there was no significant difference between groups in frequency modulation, the harmonics-to-noise ratio, and duration.
We suggest that these changes reflect smaller prediction errors due to the progress of conditioning.As conditioning progressed, the number of pellets obtained certainly increased; however, associative learning also progressed.At the onset of learning, there was no option to respond to all poke holes in order to gain pellets.Thus, the expectation of reward was often betrayed, and prediction error was large.As conditioning progressed, rats learned to selectively poke lit holes.This led to a decrease of both betrayed reward expectations and prediction errors.This effect may be accompanied by a reduction in novelty.
Previously, it was believed that reward solely activates dopaminergic neurons in the reward system, which subsequently increases the number of 50-kHz USVs, especially FM calls.In other words, 50-kHz USVs could be controlled mainly by the presentation and expectation of reward.However, dopaminergic neurons in the reward system can be activated by various factors.For example, with the progress of associative learning in which stimuli preceded in time rather than reward, dopaminergic neurons in the reward system began responding to the difference between the actual and expected reward based on the stimulus (i.e., prediction error; Ljungberg, Apicella, & Schultz, 1992;Schultz, 1998).Neural activity should diminish upon completion of conditioning because of the absence of prediction error.
Additionally, non-rewarding surprising and invasive stimuli, as well as novelty, can activate such neurons (see Bromberg-Martin, Matsumoto, & Hikosaka, 2010, for a review).
The present results also suggest that the number of 50-kHz USVs correlates with body weight as has been observed elsewhere (e.g., Schwarting et al., 2007).However, our findings were opposite to those described in previous studies.This contradiction might be due to the difference in situations where rats vocalized USVs.Schwarting et al. (2007) recorded USVs in a cage, and no task or manipulation was added during recording.Thus, there was a possibility that the USVs recorded in their study reflected other factors such as social wanting.
Therefore, we predicted that the amplitude of calls would reflect positive emotion induced by appetitive behavior, such as the expectation of reward, because amplitude was the only acoustic parameter that differed between the groups and changed within the experimental group.The amplitude of USVs was greater in the experimental group and increased with progressive conditioning in this group.
The social and communicative aspects of 50-kHz USVs should also be considered.Individually tested rats produce 50-kHz USV calls regardless of motivational state, and the number did not decrease over repeated sessions (Schwarting et al., 2007).More of these calls were produced when rats were placed in a cage frequently visited by other conspecifics, and calls were triggered by olfactory stimuli.Therefore, 50-kHz USVs were thought to signal the presence of a vocalizer (Brudzynski & Pniak, 2002).These social and communicative aspects of USVs may have affected our results.In both groups, individuals were solely tested in a non-communicative setup.Nevertheless, there were significant intergroup differences and changes within the experimental group.Therefore, we conclude that the differences and changes reflected rats' internal states, and these differences are not likely to reflect social signals.
It has been proposed that 50-kHz USVs, particularly subtypes containing many frequency modulations, are an index of positive emotion or high motivation.However, the opportunity to obtain food reduced non-continuous frequency modulation.Similarly, increasing the amount of reward reduced frequency modulation.The opportunity to obtain food increased mean and max amplitude, and increasing the amount of reward also increased mean amplitude.Therefore, we suggest that frequency modulation in 50-kHz USVs reflects other factors such as reinforcement prediction errors rather than positive emotion or high motivation.The amplitude of 50-kHz USVs should be taken as an index of positive emotion and expected reward.
Nevertheless, it is still too early to conclude that the amplitude of 50-kHz USVs is associated with positive affect based on this one experiment.The amplitude of 50-kHz USVs should be analyzed in other situations in which various affects are evoked.

Figure 1 .
Figure 1.The operant chamber used for the experiment A) Side view of the apparatus.The red horizontal arrow indicates the ultrasound microphone.The microphone was inserted in the center hole at the chamber roof.B) Top view.Blue horizontal arrows indicate the location of the lit nose poke holes during the differential conditioning task.

Figure 2 .
Figure2.Two parameters, frequency and time, reflect frequency modulation.The number of vertical modulations (inflections) of the fundamental frequency within the call.For example, if fundamental frequency increased between 1 and 2 ms point, but decreased between 2 and 3 ms point, we regarded that there should be one vertical modulation between 1 and 3 ms.The total modulation distance (in kHz) was considered to be equal to the sum of the absolute difference between adjacent fundamental frequency points per ms.This is equal to the cumulative frequency modulations.The total modulation distance is 0-kHz if there was no modulation in fundamental frequency.

Figure 3 .
Figure 3. Difference in USV acoustic parameters between groups.A) Sub-element number.B) Max-amplitude.C) Meanamplitude.D) Distance from start to max.E) Total USVs number.Each open square represents the averaged feature value for each day and each individual.Dashed lines represent the predicted value by the model that consisted of random effects (the length of the waiting phase and individual differences) and an intercept.Solid lines represent the predicted value when a fixed effect (Experiment group) was added to the above model and with a slope and an intercept.The magnitude of the slope of the solid line is equal to the value of the fixed effect to each acoustic parameter.A Gaussian distribution was used as the error structure.The ΔAIC is the difference between the model AIC that contained only the random effects and the model that added the fixed effect to the random effects.

Figure 4 .
Figure 4. Changes in the acoustic features with an increase in acquired reward.A) Mean-amplitude.B) Duration.C) Max harmonics to noise ratio.D) The number of vertical modulation.E) Total modulation distance.F) Total USVs number.Each open circle represents the averaged feature value for each day and each individual.Random effects were the length of the waiting phase and individual differences.The fixed effect was the number of pellets obtained.

Figure 5 .
Figure 5. Differences in USV acoustic parameters in relation to body weight.A) Total USVs number.B) Bandwidth change from start to center.C) Bandwidth change from center to end.D) Bandwidth in center.Each open circle represents the averaged feature value for each day and each individual.Random effects were the length of the waiting phase, individual differences, and group.The fixed effect was body weight.