Flexible motor sequence generation during stereotyped escape responses

Complex animal behaviors arise from a flexible combination of stereotyped motor primitives. Here we use the escape responses of the nematode Caenorhabditis elegans to study how a nervous system dynamically explores the action space. The initiation of the escape responses is predictable: the animal moves away from a potential threat, a mechanical or thermal stimulus. But the motor sequence and the timing that follow are variable. We report that a feedforward excitation between neurons encoding distinct motor states underlies robust motor sequence generation, while mutual inhibition between these neurons controls the flexibility of timing in a motor sequence. Electrical synapses contribute to feedforward coupling whereas glutamatergic synapses contribute to inhibition. We conclude that C. elegans generates robust and flexible motor sequences by combining an excitatory coupling and a winner-take-all operation via mutual inhibition between motor modules.


Introduction
Nervous systems transform sensation into a sequence of actions. The motor repertoire, constrained by the biomechanics of gait, comprises a finite number of motor primitives that are stereotyped across individuals (Ahamed et al., 2019;Berman et al., 2014;Liu et al., 2018;Stephens et al., 2008). On the other hand, behavioral flexibility allows an animal to explore the action space, and to select better strategies for acquiring rewards or avoiding danger in a changing environment (Sutton and Barto, 2017).
Many factors contribute to behavioral flexibility (Dhawale et al., 2017;Gordus et al., 2015;Remington et al., 2018a). Actions may be generated by an inherently noisy system: synapses are unreliable (Allen and Stevens, 1994), neurons generate variable spike trains (Mainen and Sejnowski, 1995), and neural circuits may operate near the edge of chaos (van Vreeswijk and Sompolinsky, 1996). On the other hand, neural networks, whether adaptive or hard-wired, have structures that shape population neural dynamics onto a low dimensional manifold, where nonrandom and ordered activity patterns emerge (Ganguli et al., 2008;Harvey et al., 2012;Inagaki et al., 2018). Computational models have promised to provide a unified view of these observations (Burak and Fiete, 2012;Brennan and Proekt, 2019;Mastrogiuseppe and Ostojic, 2018;Roberts et al., 2016), but a deep connection between theories and experiments remains to be established.
The initiation of escape responses of the nematode Caenorhabdtis elegans (C. elegans) has long been viewed as an instinctive reflex. Upon a gentle touch to its anterior body, the ventral cord-projecting premotor interneurons AVA/AVD/AVE relay mechanosensory inputs to motor neurons and reliably drive a backward movement (Chalfie et al., 1985;Pirri et al., 2009;Wicks et al., 1996). While C. elegans stays committed to its escape decision, the animal remains flexible in its approach to complete the motor sequence. After the reversal, the animal may or may not reorient its body via a deep omega (W) turn, before moving forward ( Figure 1A). This allows the animal to resume forward movement in either the original or a new direction. Notably, which action to select and when to execute exhibit trial to trial variability, and they can be coupled. For example, a and fitted type-II transition rates pass goodness-of-fit test (p>0.05).
The online version of this article includes the following video, source data, and figure supplement(s) for figure 1: Source data 1. Source data for Figure 1 and  previous study (Gray et al., 2005) has shown that a longer reversal is likely to be followed by an omega turn. We sought to understand algorithms and circuit mechanisms for motor sequence generation by investigating recurrently connected interneurons, which are positioned between sensory neurons and motor neurons in the C. elegans nervous system ( Figure 2-figure supplement  1A). Previous studies on this layer of neural network (Figure 2-figure supplement 1A) have implicated their roles in exploratory behaviors (Gray et al., 2005;Iino and Yoshida, 2009;Mori and Ohshima, 1995;Pierce-Shimomura et al., 1999). During navigation, C. elegans moves towards a new direction by making a reversal and/or a turn in a probabilistic manner. Cell ablation studies revealed that the frequencies of reversals or turns were differentially modulated by many local interneurons including AIB and RIB (Gray et al., 2005). Here we ask whether and how activities of local interneurons and their synaptic interactions shape the dynamics of a motor sequence during escape responses.
Several models have been proposed to account for motor sequence generation. In a class of synaptic chain models (Abeles, 1991;Long et al., 2010;Xiao et al., 2017), feedforward excitation between transiently activated groups of neurons controls the timing of actions hierarchically. Sequential neural activity may also emerge from a cooperation between external inputs and local synaptic interactions in a recurrent network (Rajan et al., 2016;Seeds et al., 2014). We find that neurons encoding distinct motor states, such as reversals and omega turns, use electrical coupling to reliably drive motor state transitions, whereas they exploit mutual inhibition to flexibly control the timing of an action in a sequence. We propose that a form of short-term plasticity in inhibitory synapses contributes to the time-dependent change of transition probability between motor states. Our findings provide new insights into how the nervous system organizes time-ordered and variable motor activities, by which stereotyped and flexible animal behaviors emerge.

Results
Stereotypical and flexible motor patterns constitute C. elegans escape responses A potentially threatening sensory stimulus will trigger an animal's escape response. For example, a gentle touch on the C. elegans head, which activates specific mechanosensory neurons ALM/AVM (Chalfie et al., 1985), can induce a reversal or an omega turn ( Figure 1A and Figure 1-video 1).
We quantitatively characterized the escape responses from transgenic animals in which channelrhodopsin-2 (ChR2) was expressed in ALM/AVM neurons (Pmec-4::ChR2; lite-1), and optogenetic stimulation was given to the same sensory neurons at a defined light intensity and pulse duration during forward movement (see Materials and methods) (Leifer et al., 2011). ALM/AVM-triggered backward movements responses were robust (only~10% trials did not respond, Figure 1C left), but subsequent motor sequences constituting each trial varied. Animals exhibited two main types of motor patterns: (1) backward movement was followed by a deep omega turn, and the animal moved forward in a new direction that was different from that before stimulation; (2) an animal executed backward movement and then resumed forward movement in a similar direction as that before stimulation ( Figure 1A and Figure 1-video 1). The head and the tail were diametrically opposed to each other in an omega turn; whereas they were likely aligned to each other in a backward-toforward movement (Figure 1-figure supplement 1A). Occasionally, an animal paused (1/674 trials) before resuming forward movement ( Figure 1A), which can be regarded as the third motor pattern.
The reversal length distribution is broad ( Figure 1C left) and likely bimodal (Figure 1-figure supplement 1B). This observation motivated us to describe behavior statistics by introducing two types of transitions and the corresponding transition rates r(t). Among all reversals survived to time t, r(t)Dt computes the fraction of events that will make a transition to another motor state within the time bin Dt. The type-I (RF) transition rate, r 1 , determines the transition probability from reversal to forward movement; the type-II (RT) transition rate, r 2 , determines the transition probability from reversal to omega turn ( Figure 1B and Materials and methods). r 1 (t) rapidly plateaued in about one second, while r 2 (t) increased and gradually became the dominant mode ( Figure 1C right). The escape responses induced by a focused infrared laser light (Mohammadi et al., 2013) exhibited qualitatively similar statistics to ALM/AVM-triggered responses ( Figure 1D). This quantification, which was consistent with a previous observation and description for spontaneous reversals during exploratory behaviors (Gray et al., 2005), confirms the notion that the longer a reversal, the more likely the reversal is followed by a turn.

Local interneurons in the backward module modulate motor state transitions
We ask how neural dynamics underlie the behavioral variation. Whole brain and multi-neuron calcium imaging of fixed and behaving animals suggested that population interneuron activities, which perform sensorimotor transformation, encode distinct motor states (Gordus et al., 2015;Kato et al., 2015;Kawano et al., 2011;Li et al., 2014;Luo et al., 2014;Nguyen et al., 2016;Roberts et al., 2016;Venkatachalam et al., 2016; Figure 2A and Figure 2-figure supplement 1B). Several interneurons, including the ventral-cord-projecting premotor interneurons AVA and AVE, and the local interneurons AIB and RIM, exhibited increased calcium activity during a backward movement (Kato et al., 2015;Laurent et al., 2015;Luo et al., 2014; Figure 2A and Structural and functional studies of AIB (Gray et al., 2005;White et al., 1986) indicate that they may play important roles in motor state transitions ( Figure 2A). First, AIB establish recurrent connections with the premotor interneurons AVA and AVE that potentiate backward movement either directly through chemical synapses or indirectly through electrical and chemical connections with RIM ( Figure 2A). Second, AIB form gap junctions with the inter/motor neurons RIV (White et al., 1986), which play a role in generating a ventral-biased turning behavior ( Figure 2A; Gray et al., 2005). Third, AIB exhibit ramping calcium activity during reversals (Kato et al., 2015;Laurent et al., 2015;Luo et al., 2014), and finally, laser ablation of AIB significantly reduces the frequency of reversals during food search behavior (Gray et al., 2005).
We first examined neuronal correlate of behavioral flexibility in action selection. We compared the AIB ramping activity (Pinx-1::GCaMP6; Pinx-1::wCherry) in different action sequences during either spontaneous or thermal-stimulus-triggered behaviors ( Figure 2B-C and Figure 2-figure supplement 1B-C). If the fluorescence signal (DR(t)/R 0 ) reflects a change of intracellular free calcium concentration [Ca 2+ ], the ramping rate, defined as z = dR dt ( Figure 2C), would be proportional to the calcium current. Higher z may reflect a larger depolarization of the neuronal membrane potential. In Figure 2B-C, 76% trials (91/120) in the type-I (RF) transition show a positive ramping rate, whereas the proportion rose to 95% (109/115, p < 0.0001, c 2 test) in the type-II (RT) transition. Among trials longer than 1.5 seconds, they all showed positive z, which during the type-II transition was significantly higher than that during the type-I transition ( Figure 2C and Figure 2-figure supplement 1C). These results suggest that the more active AIB are, the more likely a worm would terminate its reversal with a turn.
Optogenetic activation of AIB (Pnpr-9::ChR2 or Pnpr-9::Chrimson) alone reliably triggered reversals followed by omega turns ( Figure 2D, Figure 2-figure supplement 1D and Figure 2-video 1), whereas strong optogenetic inhibition of AIB (Pmec-4::ChR2; Pnpr-9::Arch; lite-1) during ALM/ AVM induced escape responses almost completely abolished omega turns ( Figure 2D). We also generated transgenic animals in which AIB were persistently hyperpolarized by an expression of exogenous potassium channels (Pnpr-9::TWK-18(gf)). Interestingly, the no-response fraction increased to~20% (70/332, p<0.0001, c 2 test) upon stimulating ALM/AVM in these animals and a significantly larger portion of responses were pauses (35/332, p<0.0001, c 2 test, Figure 2E). We did not observe a significant change in the type-II transition rate r 2 (Kolmogorov-Smirnov test, p=0.6, Figure 2F), which might be due to a weaker AIB inhibition in these animals. Furthermore, optogenetic ablation of AIB alone (Pnpr-9::PH-miniSOG,  Neurons were grouped into four modules based on their functional roles and activity patterns. (B) Calcium activity of AIB during spontaneous reversals before type-I (n = 23) and type-II (n = 36) transitions in unrestrained behaving animals (Pinx-1::GCaMP6;Pinx-1::wCherry). Here, data are aligned to the ends of reversals (vertical dashed line, t = 0). Heat map across trials (up) and DR(t)/R 0 (Mean ± SEM, bottom) are shown. (C) Ramping rate of calcium activity in AIB. Up, raw single trial DR(t)/R 0 from reversal start to reversal end. The ramping rate is the slope of the red line, fitted by linear regression. Bottom, ramping rates of AIB during type-I and type-II transitions. Each color (Mean ± SEM) represents single animal data across multiple trials. Total nine animals (Pinx-1::GCaMP6;Pinx-1::wCherry) were tested. Very short reversals (less than 1.5 s) are excluded, for some of them have negative ramping rates and the slope estimate is susceptible to noise (but including those trials doesn't affect our conclusion). **p<0.01, two-way ANOVA. (D) Optogenetic activation of AIB (635 nm, 4.46 mW/mm 2 , 7 s) or inhibition of AIB (561 nm, 21.71 mW/mm 2 , 12 s) during ALM/AVM (473 nm, 14.71 mW/ mm 2 , 1.5 s) triggered avoidance behaviors, reversal durations (bar graph) and fractions of animals executing omega turns (pie chart) are shown. Error bars are SEMs. Bar graph, Mann-Whitney U test. Pie chart, c 2 test. *p<0.05, ***p<0.001, ****p<0.0001. Here and below, the actual turning percentages (n turn /n total ) are noted beside the pie chart and numbers within the bars indicate the number of trials with reversal. (E-F) Reversal length distribution (E) and transition rates (F) during escape responses when AIB were persistently hyperpolarized through an exogenous expression of the potassium channel TWK-18. Control group is from Figure 1C.
The online version of this article includes the following video, source data, and figure supplement(s) for figure 2: Source data 1. Source data for Figure 2 and   Feedforward coupling between the backward module and the turning module drives the omega turn How do AIB drive turning behaviors? Whole brain imaging in immobilized animals implicated that AIB and their electrically-coupled partners RIV ( Figure 2A and Figure 3-figure supplement 1) exhibited sequentially activated patterns (Kato et al., 2015). We compared RIV activity patterns (Plim-4::GCaMP6) underlying different motor sequences during spontaneous behaviors. During the type-II (RT) transition, RIV calcium signal rose rapidly immediately before a turn began, whereas it remained largely quiescent during the type-I (RF) transition ( Figure 3A and To directly probe the functional connectivity between AIB and RIV, we performed simultaneous optogenetic stimulation of AIB (Pnpr-9::Chrimson) and calcium imaging of RIV (Plim-4::GCaMP6:: wCherry, Figure 3B). In immobilized wild-type animals, upon stimulating AIB at t = 0, RIV calcium signal rapidly rose ( Figure 3C dark blue and Figure 3-figure supplement 2C). Several innexin, including INX-1, UNC-7 and UNC-9, have been reported to be expressed in AIB and RIV (Altun et al., 2009;Bhattacharya et al., 2019). Some of these innexins were shown to form homotypic and/or heterotypic gap junctions (Kawano et al., 2011;Liu et al., 2013;Starich et al., 2009;Xu et al., 2018). To determine whether electrical synapses contribute to the observed functional coupling between AIB and RIV, we examined the effect of AIB stimulation in inx-1unc-9unc-7 triple innexin mutants. RIV remained quiescent upon AIB stimulation ( Figure 3C  UNC-7 and UNC-9 are broadly expressed in the motor circuit, and unc-7 or unc-9 mutants exhibit uncoordinated movements that prohibit them from completing a motor sequence (Barnes and Hekimi, 1997;Brenner, 1974;Kawano et al., 2011;Starich et al., 1993;Xu et al., 2018). inx-1 single mutants exhibit superficially normal forward and backward movements, allowing us to examine the behavioral requirement of INX-1. The presence of multiple innexins in many C. elegans neurons implicates that they may function redundantly at electrical synapses. Consistent with this notion, we find that optogenetic activation of AIB was capable, but with less likelihood, to trigger a turn in inx-1 mutants ( Figure 3D). Rescuing inx-1 in AIB was sufficient to restore the turning probability ( Figure 3D). Because inx-1 mutants were still capable of generating omega turns, we propose that either multiple innexins between AIB and RIV, or parallel circuit pathways are at play.
When we performed dual optogenetic activation and calcium imaging in wild-type animals that were allowed to move, an increase of RIV calcium activity was also observed. But we observed a delay in RIV calcium signal, with its increase arriving at variable times ( Figure 3E and Delayed depolarization of RIV in a moving animal may result from a convergence of excitatoryand inhibitory-inputs onto the turning module ( Figure 2A). When neural activity in behaving animals was aligned to the onset of optogenetic stimulation, a transient quiescence or decrease of RIV calcium activity indeed appeared after t = 0 ( Figure 3F and Figure 3-figure supplement 2F). We hypothesized that a rapid increase of calcium activity in Figure 3C (dark blue) could result from a stronger depolarization of RIV neurons in immobilized animals. Consistently, when the calcium imaging experiment in immobilized animals was combined with a weak and persistent optogenetic inhibition of RIV (Plim-4::Arch), we also observed a delayed and rectified excitation in RIV ( Figure 3C  Taken together, our data suggest that the feedforward excitation between the backward module and the turning module takes the form of electrical synapses, likely between AIB and RIV. We considered an effective functional coupling through polysynaptic excitation highly unlikely. First, AIB triggered ventral-biased turning behaviors did not require glutamatergic synaptic transmission ( Figure  Here, data are aligned to the ends of reversals (vertical dashed line, t = 0). Heat map across trials (left) and DR(t) (Mean ± SEM, right, also see Materials and Methods) are shown. (B) Optical neurophysiology for probing the feedforward coupling between the backward module and the turning module. (C) Simultaneous optogenetic activation of AIB (635 nm, 6.11 mW/mm 2 ) and calcium imaging of RIV in immobilized animals. DR(t)/R 0 (Mean ± SEM) under different genetic backgrounds are shown: control (ATR) is wild-type (dark blue); inx-1unc-9unc-7 triple mutant (red); calcium imaging of RIV in the presence of moderate inhibition (561 nm, 1.94 mW/mm 2 ) of RIV (light blue, Plim-4:: Arch). The control (no ATR) represents imaging data from wild-type animals without feeding all-trans retinal (grey). ****p<0.0001, two-way ANOVA. (D) Probability of a reversal followed by a turn (left) and type-II transition rates (right) for gap junction deficient mutants. AIB, expressing Chrimson, were optogenetically activated for 7 s (635 nm, 4.46 mW/mm 2 ) in wild-type animals (green), gap junction deficient mutants inx-1 (red), and inx-1 mutants, in which INX-1 channels were restored specifically in AIB (blue). Error bars indicate 95% binomial proportion confidence interval. c 2 test. ****p<0.0001. (E) Simultaneous optogenetic activation of AIB (635 nm, 6.11 mW/mm 2 ) and calcium imaging of RIV in unrestrained behaving animals. Left, calcium activity heatmap across trials. t < 0 represents reversals. Omega turns start at t = 0. Right, DRðtÞ (mean ± SEM) are shown (blue). (F) Data are related to (E), but t = 0 is aligned to the beginning of AIB stimulation. The online version of this article includes the following video, source data, and figure supplement(s) for figure 3: Source data 1. Source data for Figure 3.    and type-II transition rates (right) in glutamatergic synaptic transmission deficient animals upon strong and persistent optogenetic activation of AIB. Optogenetic stimulation was delivered for 7 s using red light (635 nm, 4.46 mW/mm 2 ). Right, compare r 2 in eat-4 mutant or in Pnpr-9::TeTx with control animals across the whole distribution (Kolmogorov-Smirnov test, p=6.2e-7, p=0.0026) or within a time window (c 2 test: * p<0.05, ** p<0.01, *** p<0.001, **** p<0.0001). (B) Reversal durations (left) and type-II transition rates (right) in GluCl receptor mutants upon optogenetic activation of AIB (635 nm, 4.46 mW/mm 2 ). Right, compare r 2 in triple receptor mutant with control animals across the whole distribution (Kolmogorov-Smirnov test, p=5.9e-8) or within a time window (c 2 test: * p<0.05, ** p<0.01).
(C) Genes encoding GluCl receptors were expressed in local interneurons RIB and AIY respectively. GFP reporter lines were constructed using avr-14, avr-15 and glc-1 promoters, respectively; mCherry (or wCherry) reporters were used for cell identification. Expression pattern from one section is showed. Scale bar, 10 mm. (D) Simultaneous optogenetic activation of AIB (635nm, 6.11 mW/mm 2 ) and calcium imaging of RIB in immobilized animals under wide-type (control (ATR)) (blue) or eat-4 glutamate deficient mutant background (red). DR(t)/R 0 (Mean ± SEM) are shown. t = 0 represents the Figure 4 continued on next page connectome (White et al., 1986). Our optogenetic activation of AIB while inhibiting RIM did not modify the turning probability, whereas activating RIM while inhibiting AIB significantly reduced the turning probability ( Figure 2-figure supplement 1F-G). Activation of RIS would drive an animal to a pause state and abolish motor actions (Steuer Costa et al., 2019). Both results argue against RIM and RIS being directly involved in driving turning behaviors.

Inhibitory glutamatergic synaptic transmission modulates the type-II transition
We next investigated behavioral flexibility in the timing of an action. Given the feedforward coupling between the backward module and the turning module, we asked why omega turns did not immediately follow the optogenetic activation of AIB. We hypothesized that a balance of feedforward excitation and an unknown inhibition provides a potential mechanism to shape the statistics of the type-II (RT) transition. Besides gap junctions, AIB make chemical synapses with neurons in other modules ( Figure 2A and C. elegans nervous system possesses a family of inhibitory glutamate-gated chloride (GluCl) channels (Dent et al., 1997). Upon optogenetic stimulation AIB, the triple GluCl mutant avr-14(ad1035); avr-15(vu227)glc-1(pk54) exhibited a behavioral phenotype resembling that of the eat-4 mutant ( Figure 4B and Figure 4-figure supplement 1B). In some trials (19/112, p<0.0001, Fisher's exact test), AIB stimulation immediately triggered omega turns without delay (Figure 3-video 1). This suggests that postsynaptic GluCl receptors work synergistically in modulating the onset timing of a turn. Using GFP reporter lines, we found avr-14, glc-1 and avr-15 expressed in many neurons. By focusing on overlaps with neurons known to encode motor states (Figure 2A), we found that Pavr-14 and Pglc-1 reporters exhibited expression in AIY interneurons, while the Pavr-15 reporter exhibited expression in RIB interneurons ( Figure 4C).
Unlike AIY, RIB receive more and invariant synaptic inputs from AIB (White et al., 1986;Witvliet et al., 2020;Figure 2A), and hence are the prominent postsynaptic partners of AIB. Consistent with a glutamate mediated feedforward inhibition, RIB calcium activity (Psto-3::GCaMP6) significantly reduced upon optogenetic activation of AIB in immobilized animals, which was not observed in the glutamate vesicular transport deficient mutant eat-4 animals ( Figure 4D and  beginning of AIB stimulation. The control group (no ATR) (grey) represents imaging data from animals without feeding all-trans retinal. ****p<0.0001, two-way ANOVA. (E) Functional coupling between AIB and RIB neurons was directly tested through glutamate imaging. Upon stimulation of AIB (Pnpr-9::Chrimson), the process (red), cell body (green) of the RIB (Psto-3::iGluSnFR) and the RIB process in eat-4 glutamate deficient mutants (grey) exhibited distinct fluorescence signals. The control group (blue) represents imaging data from animals without feeding all-trans retinal. Raw iGluSnFR imaging was recorded at 150 Hz. Trial average (bold color) and SEM (shaded region) are shown. Inset, the fluorescence signal of the process (red) is fitted with a double exponential function (blue), B e À t t 1 À e À t t 2 þ c, where B, t 1 ;t 2 and c are free parameters. t 1 represents the time constant of glutamate signal decay, and c is the baseline constant, which was subtracted in the inset plot. For reversal duration, Error bars indicate SEM. Mann-Whitney U test: ***p<0.001, ****p<0.0001. All multiple comparisons were adjusted using Bonferroni correction. The online version of this article includes the following source data and figure supplement(s) for figure 4: Source data 1. Source data for Figure 4 and To further investigate the functional connectivity between AIB and RIB, we imaged glutamate signaling (Marvin et al., 2013) at RIB (Psto-3::iGluSnFR) upon persistent optogenetic stimulation of AIB (Pnpr-9::Chrimson; see Materials and methods). After the onset of stimulation, a rapid rise ( Figure 4E inset plot,~10%DF/F 0 ) of iGluSnFR signal on RIB's neurites ( Figure 4E and Figure 4-figure supplement 2A up) was followed by a slow decay. The fluorescence signal change was well fit by (1) where t 1 ¼ 0:9 s, and t 2 ¼ 100 ms. In animals without feeding all-trans retinal (a co-factor required for AIB optogenetic stimulation), we observed random and smaller amplitude (~5%DF/F 0 ) fluctuations of iGluSnFR signals ( Figure 4E). Such dynamics was not observed in the glutamate vesicular transport deficient mutant eat-4 animals ( Figure 4E and

Local interneurons RIB promote both turning and forward behaviors
Local interneurons RIB, together with the ventral cord-projecting premotor interneurons AVB, have been previously reported to encode forward movement state (Gray et al., 2005;Kato et al., 2015;Li et al., 2014). Moreover, RIB form gap junctions with SMDV ( Figure 2A), motor neurons that have also been implicated in ventral biased omega turns (Gray et al., 2005;White et al., 1986). RIB calcium activity declined during reversals, and rose during the type-I (RF) and type-II (RT) transitions ( Figure 5A and When RIB interneurons were directly inhibited to mimic an inhibitory synaptic input, either optogenetically or by an expression of histamine-gated chloride channels (Pokala et al., 2014), the type-II (RT) transition rate r 2 plateaued at a significantly reduced value ( Figure 5C and  (Figure 4B left). The type-I transitions were also largely suppressed ( Figure 5C), as RIB also potentiate forward movement. Optogenetic ablation (Psto-3::min-iSOG) or blocking chemical synaptic transmission (Psto-3::TeTx) from RIB also led to prolonged reversals during ALM/AVM-triggered escape responses ( Figure 5F).
We asked how RIB may mediate neural activity in the turning module. Upon optogenetic stimulation of AIB ( Figure 5D-E), the rise of RIV calcium activity in RIB ablated animals showed the same rectified activation when all trials were aligned to the beginning of a turn ( Figure 5E). However, RIV activity was preceded by a longer quiescent state when trials were aligned to the stimulus onset ( Figure 5D and Figure 5-figure supplement 1C). Thus, RIB modulate motor state transitions in part through indirect modulation of the timing of RIV activation (Figure 2A).

Inhibitory feedback contributes to reversal termination
The beginning of a turn marks the end of a reversal. We next asked whether the type-II (RT) transition can be accounted for by self-termination of neural activity in the backward module (Figure 2A), analogous to a feedforward synaptic chain model, or, whether activation of the turning module provides a feedback inhibition to terminate the activity in the backward module.
In a feedforward synaptic chain model, perturbing neural activity in the downstream neurons would not affect the dynamics of upstream neurons. To test this model, we generated transgenic animals that express Archaerhodopsin in RIV/SAA/SMB neurons (Plim-4::Arch; Figure 6A). RIV/ SAAD all exhibited elevated calcium activity during omega turns, and could be regarded as downstream outputs of the backward module ( Figure 5-figure supplement 1D). However, optogenetic D Reversal duration (s) Statistics of motor state transitions (left) and representative curvature kymographs (right) upon optogenetic manipulation of RIB. Left, the probability for a transition and its 95% confidence limits were computed. RIB::Chrimson 1 , n = 65, red light (635 nm, 3.75 mW/mm 2 ); RIB::Chrimson 2 , n = 82, red light (635 nm, 1.00 mW/mm 2 ); RIB::Arch, n = 83, green light (561 nm, 8.14 mW/mm 2 ). Right, animals crawled on fresh agar plates. Body curvature was normalized by a k Á L, where L is the body length. Green (or red) shaded regions show selected spatiotemporal regions for optogenetic inhibition (or activation). The kymograph of turning behaviors exhibits longer cycles to complete body bending and larger body curvature, which are different from those during forward movement. (C) Reversal length distributions (left) and transition rates (right) when ALM/AVM activation was followed by optogenetic inhibition of RIB (12 s green light, 561 nm, 1.94 mW/mm 2 ). Pmec-4::ChR2;Psto-3::Arch, n = 173. Control group is from Figure 1C. inhibition of RIV/SAA/SMB or RIV alone by spatially patterned illumination during escape responses promoted significantly longer reversals ( Figure 6B and Figure 5-video 2).
Observations from optogenetic ablation of RIV/SAA/SMB (Plim-4::miniSOG) also argue against a pure feedforward synaptic chain model. The type-II transition was abolished since animals could no longer generate a complete omega turn ( Figure 6C upper panel), while the ability of direct transition from a backward to a forward movement remained unaffected and the type-I (RF) transition rate r 1 remained similar ( Figure 6C upper panel) to wild-type animals. Notably, the reversal duration became much longer and approached 30 s in some trials, which had not been observed in wild-type animals ( Figures 1C and 6C upper panel and Figure 6-video 1).
These results indicate that during normal type-II transitions, persistent neural activity in the upstream backward module could be abolished through inhibitory feedback from the downstream activity in the turning module.
Both the type-I (RF) transition rate ( Figure 6C upper panel) and the mirror transition rate (FR) from a forward movement to a spontaneous reversal in wild-type animals ( Figure 6C bottom panel) are consistent with the homogeneous Poisson process at long timescale, leading to exponential survival functions ( Figure 6C insets) -fraction of backward or forward movements survived to t (Berg, 1993;Stephens et al., 2011). We did not observe an exponential survival function of reversals in wild-type animals. In the absence of the turning module, the statistics of forward and backward movements ( Figure 6C) became consistent with a simple dynamic model, where a system stochastically transitions between two attractor states at constant rates.
Together, our data suggest that the feedforward inhibition ( Figure 4) and feedback inhibition ( Figure 6) between the backward module and the turning module implement a winner-take-all computation for action selection. The motor module with the highest level of activity stays active by suppressing the activities of other modules.

A biophysical model of the type-II transition
With both structural and functional evidence, we now propose a mathematical model for the type-II (RT) transition. The turning module, represented by RIV inter/motor neurons, receives opposing excitatory and inhibitory inputs during a reversal ( Figure 7A). The rapid increase of RIV activity coincides with the beginning of an omega turn ( Figure 3A and E). To capture the essential process, we assumed that the membrane potential of RIV x fluctuates around a balanced state x 0 during a reversal ( Figure 7B), and its neural dynamics is governed by the Langevin equation: where k depends on, among others, the gap junction and inhibitory synaptic conductances (see Appendix); and h could be regarded as fluctuations in synaptic currents (Lindsay et al., 2011;Narayan et al., 2011) and other sources of noises that are not explicitly considered in the model. For simplicity, h is treated as uncorrelated Gaussian white noise: hðtÞhðt 0 Þ h i ¼ 2s 2 dðt À t 0 Þ: Once the membrane potential crosses the threshold x th , RIV become rapidly depolarized due to a nonlinear rectified activation ( Figure 7B), immediately terminating the reversal via feedback inhibition and starting a turn by activating ventral muscles ( Figure 7A and D).
The next step is to calculate the type-II transition rate r 2 : the probability that x crosses x th per unit time. It is currently impossible to measure k, but we can proceed by making a prediction. Based on Source data 1. Source data for Figure 6. electrophysiological recordings of C. elegans interneurons (Lindsay et al., 2011;Roberts et al., 2016), the membrane time constant of a neuron (~10 milliseconds) is much smaller than the behavioral timescale (~seconds). As a result, the membrane potential of a model neuron rapidly approaches the fixed point x 0 . By solving this problem analytically using one-dimensional Fokker-Planck equation near the system equilibrium (see Appendix), we find  is given by the glutamate decay constant in Figure 4E.
Here erfi x ð Þ ¼ 2 ffiffi ffi p p R x 0 e z 2 dz is the imaginary error function. Equation 3, however, would lead to a constant rate on the behavioral timescale, like that during the type-I (RF) transition, as expected and confirmed by our computer simulation ( Figure 7C). To explain the experimental observation of the rising phase of r 2 ( Figure 1C-D), we incorporated a plasticity mechanism analogous to short-term synaptic depression (STD): the feedforward inhibition from the backward module to the turning module becomes weaker as the reversal lasts longer ( Figure 7D). Consequently, the membrane potential moves towards the excitation threshold x th to potentiate transition, allowing the analytical expression for r 2 , Equation 3, to become time-dependent. Our hypothesis is consistent with the decay of the glutamate sensor signal on RIB neurites upon AIB stimulation ( Figure 4E), an observation that may be explained by a depletion of available vesicles for release at the presynaptic site. Note that calcium activity in AIB cell body, like that during a reversal ( Figure 2B), kept increasing during persistent optogenetic stimulation (Figure 4-figure  supplement 2B), arguing against the possibility that an opsin-mediated membrane depolarization in the presynaptic neuron undergoes depression upon continuous light activation.
By incorporating the exponential decay of inhibitory synaptic strength (Equation 1), we found that the functional form of the transition rate (see Appendix) can be approximated by where t g ¼ 0:9 s is the decay constant of the glutamate signal ( Figure 4E). The experimentally measured type-II transition rate is well fit by Equation 4.

Discussion
Complex motor behaviors arise from continual selection and transition among a number of motor primitives. Classic synaptic chain models, in which stereotyped motor sequences arise from feedforward excitation between different groups of neurons, are thought to underlie several motor behaviors such as Zebra Finch singing (Long et al., 2010). A feedforward synaptic chain may underlie the replay of spatiotemporal activity patterns in hippocampus during sleep (Louie and Wilson, 2001;Skaggs and McNaughton, 1996), and generate temporally precise firing patterns that correspond to different actions in the motor cortex of behaving monkeys (Shmiel et al., 2006). Alternatively, when several mutually inhibited modules are co-activated by sensory inputs, motor sequences could also emerge by a winner-take-all strategy, a proposed mechanism for the grooming behavior in Drosophila (Seeds et al., 2014). In mice, mutually inhibitory neurons in the central amygdala have been shown to regulate dimorphic defensive behaviors -flight or freezing -triggered by looming visual stimuli (Fadok et al., 2017). Here, we find the two schemes are likely integrated by the C. elegans nervous system to generate robust and flexible motor sequences ( Figure 7D). In C. elegans, feedforward excitation between the backward module and the turning module ( Figure 7A) can reliably trigger an omega turn followed by forward movement through strong and persistent activation of local interneurons AIB ( Figure 2D and Figure 2-figure supplement 1D). In other words, the action in a motor sequence can be selected through feedforward excitation, triggered by either external sensory stimulus or fluctuations of internal circuit dynamics. The timing of an action can be tuned by augmenting the feedforward excitation with glutamatergic feedforward inhibition between AIB and RIB ( Figure 7A), and likely by modulating the strength of inhibitory inputs through short term synaptic plasticity. Previously, a tyraminergic feedforward inhibition (Alkema et al., 2005;Pirri et al., 2009) from the RIM interneurons in the backward module to the SMD motor neurons in the turning module was shown to suppress head movement during reversals. We propose that these functional motifs -feedforward excitation and inhibition -are combined with a nonlinear activation of turning neurons ( Figure 7D) to produce flexible type-II (RT) transitions.
A simple synaptic chain model predicts that abolishing neural activity in a downstream module would not directly affect upstream neural output. However, when RIV/SAA/SMB in the turning module were ablated or inhibited ( Figure 6B-C), we observed prolonged reversals during escape responses. Hence, the turning module may provide feedback inhibition onto the backward module, contributing to the reversal termination during the type-II transition. The cellular and molecular mechanisms for inhibitory feedback remain to be identified. One possible implementation is cholinergic synaptic outputs from SAAD onto RIM and AVA interneurons in the backward module; another possibility is synaptic outputs from RIB onto AVA/AVE ( Figure 7A). Together, the feedforward coupling between the backward module and the turning module facilitates a defined sequential activity pattern, whereas the winner-take-all operation through mutual inhibition between the two modules avoids an action conflict.
Sensorimotor transformation depends on the initial condition of the network state (Remington et al., 2018a;Remington et al., 2018b). We show that when the backward motor state is suppressed via the hyperpolarization of interneurons AIB, an identical mechanosensory stimulus is less likely to elicit an escape response ( Figure 2E). A recent study also demonstrated that mechanosensory stimuli were unlikely to drive other motor programs when C. elegans was executing a turn (Liu et al., 2018). We propose that the inhibition from the turning module to the backward module ( Figure 7D) may account for this observation.
We view omega turn, a motor state encoded by transient activity in RIV, as a special manifold connecting two attractors represented by persistent activity in the forward or backward module ( Figure 7D). Our finding -a combination of feedforward excitation and mutual inhibition between motor states -suggests a new way to control nonlinear dynamics towards a different fixed point (Morrison et al., 2020). In our simplified model, neurons within a module were treated as a homogeneous population. Nevertheless, interneurons with heterogeneous functional properties have been found. For example, laser ablation of AIB and RIM in the backward module ( Figure 2A) differentially affect the probability of spontaneous reversals (Gray et al., 2005). While RIM showed increased calcium activity during reversals (Figure 2-figure supplement 1B) and promoted reversal upon optogenetic activation (Figure 2-figure supplement 1E-G), they were less important in modulating the type-II transition than AIB did (Figure 2-figure supplement 1D-G). The impact of functional heterogeneity on the attractor dynamics and motor state transitions remains to be understood.
Our biophysical model suggests that noises in a neural circuit (Equation 2) contribute to behavior variability. We speculate that stochasticity in neural dynamics and behaviors may allow animals to efficiently explore the action space (Dhawale et al., 2017;Duffy et al., 2019;Tumer and Brainard, 2007); learning, by which functional connectivity between motor modules is modified through synaptic plasticity, may optimize action selection and timing (Sutton and Barto, 2017). We found that in C. elegans, the functional connectivity between motor modules consists of feedforward excitation and mutual inhibition. Conserved network motifs may be distributed among mammalian forebrain and midbrain circuits (Fadok et al., 2017;Klaus et al., 2019). Other animals could use similar algorithms to organize neuronal activities into sequential states to drive motor primitives, by which stereotyped and flexible behaviors emerge.

C. elegans strains
C. elegans strains including wild-type (N2), mutants, and transgenic worms were grown and cultivated according to standard procedures (Brenner, 1974). All strains used in this paper can be found in the Supplementary file 1. Transgenic worms for optogenetic experiments were cultivated in dark on NGM plates with OP50 bacteria and 0.4 mM all-trans retinal (ATR) for over 5 hr. We used young adult hermaphrodites to perform optogenetic and calcium imaging experiments, and L4 hermaphrodites to obtain expression patterns.

Molecular biology
Standard molecular biology methods were used. Details of plasmids, promoters and rescue genomic DNA (or cDNA) sequences can be found in Supplementary file 2-3.

Optogenetics
Worms were first washed in M9 buffer (or transferred onto an unseeded NGM plate for 1-3 min), then transferred onto a fresh agar plate [~0.8% (w/v) agar in M9 buffer, without food], mounted on a motorized stage. Worms were left to freely explore the new environment for 3-5 min before testing, and were automatically tracked and retained within the field of view of a 10 Â objective (Nikon Plan Apo, NA = 0.45) mounted on an inverted microscope (Nikon Ti-U, Japan) via dark field infrared illumination. Worm behaviors were recorded by a CMOS camera (Basler, aca2000-340kmNIR, Germany). MATLAB custom software (MathWorks, Inc Natick, MA, USA) was used for post-processing behavioral data and extracting moving directions and the kinematics of omega turns. For freely roaming worms without optogenetic stimulation, we only recorded them for 5-8 min.
For worms with optogenetic stimulation, lasers and a digital micromirror device (DLI4130 0.7 XGA, Digital Light Innovations, TX, USA) were used to generate a defined spatiotemporal illumination pattern (Leifer et al., 2011) at a specific wavelength (473 nm, 561 nm or 635 nm), and to manipulate the activities of neurons expressing light-activated channels (ChR2, Arch or Chrimson) (Husson et al., 2012;Nagel et al., 2005). To eliminate the effect of adaptation, single worm was stimulated 5-8 times with at least a 50 second inter-stimulus interval. For example: 1. To trigger escape responses, worms received 1.5 s blue light (short enough and over 80% trials showed responses in pilot experiments) to activate mechanosensory neurons ALM/AVM. All worms in the dataset had at least 20% probability to perform either type-I or type-II transitions. In other cases, 7 s (or 12 s) optogenetic illumination were used to ensure persistent activation/inhibition of local interneurons. 2. In some experiments, worms received sequential optogenetic stimulations with different colors and varying durations controlled by diaphragm shutters (GCI 7102M, Daheng Optics, China). For example, the 1.5 s blue light optogenetic stimulation (to trigger escape response) was followed by green light with a duration of 3-12 s to inhibit other interneurons in the motor control circuit. 3. We also performed selective optogenetic manipulation of interneurons when their cell body positions were sufficiently apart, given that the lateral resolution of our CoLBeRT system is up to~5 mm. For example, RIV and SAAD neurons are separated by at least 20 mm along the dorsal-ventral axis ( Figure 6A). In order to inhibit SAAD or RIV independently (Plim-4::Arch::GFP), we generated a spatial pattern to selectively illuminate the dorsal or ventral side ( Figure 6B right). At the same time, we monitored GFP emission signals from these neurons excited by the laser (473 nm) to ensure we targeted the correct region ( Figure 6B).

Calcium imaging
Calcium imaging was conducted on worms expressing a GCaMP6::wCherry (or mCardinal) fusion protein. Calcium activity was measured as a ratiometric change. For example, neural activity of AIB was measured as a fluorescence ratio of GFP to RFP (DR(t)/R 0 ) (GCaMP6/wCherry), where R 0 is the baseline ratio. In some cases, we define a new normalized ratiometric measure, DR(t) = [DR(t)-DR (t = 0)]/ R 0 . When cell-specific promoters are available, including AIB and RIB, we performed calcium imaging using wide-field fluorescence microscopy. Unrestrained behaving worms were placed on fresh agarose plates [2% (w/v) agarose in M9 buffer], tracked by a motorized stage using the CoLBeRT system with infrared light illumination (Leifer et al., 2011). Blue and green lights were employed to excite GCaMP6 and wCherry (or mCardinal) proteins. Green and red emission signals were captured by a 10 Â objective (Nikon Plan Apo, WD = 4 mm; NA = 0.45, Japan) at 50 fps with an exposure time of 20 ms, separated by a dichroic mirror, relayed by an optical splitter (OptoSplit II, Cairn-Research, UK), and projected onto one-half of a sCMOS sensor (Andor Zyla 4.2, UK) simultaneously. Green and red channels were aligned and processed by custom-written MATLAB scripts (Xu et al., 2018). Single worm was recorded for 3-10 min.
To image RIV neurons, which lack cell-specific promoters, we picked transgenic worms (Plim-4:: GCaMP6::wCherry) with stronger fluorescence expression on RIV, and increased the exposure time (up to 50 ms) to obtain high signal-to-noise ratio images.

Multi-neuron calcium imaging in a freely behaving worm
To image calcium activity of multiple neurons in a freely behaving worm (e.g., Figure 2-figure supplement 1B), we combined a spinning disk confocal inverted microscope (Nikon Ti-U and Yokogawa CSU-W1, Japan) for calcium imaging with a customized upright light path for worm tracking and behavior recording. A worm was placed on an agarose pad [~2% (w/v)] mounted on a motorized stage. We used a 40 Â air objective (Nikon Plan Apo, WD 0.25-0.17 mm; NA = 0.95, Japan) or a 60 Â water immersion objective (Nikon Plan Apo, WD 0.22 mm; NA = 1.20, Japan). The wavelengths of confocal excitation lasers were 488 nm and 561 nm; the emission lights were split (Andor Optosplit II, UK) in front of a sCMOS camera (Andor Zyla 4.2, UK). At the same time, we utilized a customized light path that was aligned to the same z axis to track the worm and to record behavioral data. In the upright path, a low magnification 10 Â objective (Nikon Plan Fluor, WD 16 mm; NA = 0.30, Japan) was used to gather fluorescent light excited by confocal lasers, and the fluorescent signal was processed to identify worm positions. A real time feedback signal was sent to a motorized stage to keep the worm head within the center of field of view. Meanwhile, we illuminated the worm by an infrared ring surrounding the high magnification objective to record worm behavior through the upright light path.

Simultaneous optogenetic manipulation and calcium imaging
To combine calcium imaging and optogenetic manipulation (e.g., Figure 3B), a 635 nm laser was added to activate Chrimson. Because blue light can also activate Chrimson, excitation light for GCaMP6 imaging and red light for optogenetic stimulation were synchronized using a TTL signal (LabJack Corp., U3-HV) controlled by LabVIEW (National Instruments Corp., USA). For example, to verify the connectivity between AIB and RIV, we activated AIB using red laser (635 nm) and recorded the calcium signal in RIV using blue LED excitation (M470L3-C1; Thorlabs, USA) for 7 s simultaneously. Both restrained and freely moving worms were tested. Restrained worms were placed on 10% (w/v) agarose plates with coverslips, whereas freely moving ones were placed on 2% (w/v) fresh agarose plates. Single worm was stimulated 5-8 times with at least a 50 s inter-trial interval.

Thermally-induced escape responses
In addition to optogenetic stimulation of mechanosensory neurons, we also used a thermal stimulus to trigger escape responses. We illuminated the head of a worm with a focused infrared laser light (1480 nm; spot diameter~120 mm, DT~2˚C) for 0.75 s (short enough and over 80% trials showed responses in pilot experiments), and animals responded with reversals or omega turns to avoid the stimulus (Mohammadi et al., 2013). The transition rates were qualitatively similar to ALM/AVMinduced escape responses ( Figure 1C-D). Identical experimental protocols were used when thermally-induced escape responses were followed by optogenetic manipulation or calcium imaging of interneurons.

Glutamate imaging
Glutamate imaging was conducted on worms expressing iGluSnFR (Marvin et al., 2013) on RIB and Chrimson on AIB ( Figure 4E and Figure 4-figure supplement 2A). All worms were restrained on fresh 10% (w/v) agarose plates with coverslips. We used 60 Â water immersion objective. Both processes and cell body of RIB were imaged on the same focal plane. Imaging acquisition rate is 150 fps. Like calcium imaging, blue excitation light (weak enough to reduce the bleaching effect; M470L3-C1; Thorlabs) for glutamate imaging and red light (635 nm, 6.11 mW/mm 2 ) for optogenetic stimulation were synchronized with a TTL signal. Glutamate signaling resulted in a change of the green fluorescence signal on the membrane of RIB interneurons. Stimulation and imaging sustained for more than 8 s. Single worm was stimulated 2-3 times with at least a 50 s inter-stimulus interval.

Optogenetic ablation
We used miniSOG (mini Singlet Oxygen Generator) to ablate specific neurons in C. elegans. Upon blue light stimulation, a mitochondria-targeted miniSOG (TOMM20-miniSOG) (Qi et al., 2012) or a membrane-targeted miniSOG (PH-miniSOG) (Xu and Chisholm, 2016) were employed to induce cell death in cell-autonomous manner. L2/early L3 worms were transferred onto an unseeded NGM plate, restricted by a small ring of filter paper soaked with 100 mM CuCl 2 . Worms were illuminated for 40-60 min (TOMM20-miniSOG) or 2 min (PH-miniSOG) with blue LED (M470L3-C5; Thorlabs) at the intensity of 0.46 mW/mm 2 . After illumination, worms were transplanted back to OP50-seeded NGM plates, and were allowed to recover for 1-2 days before behavior testing.

Behavioral assays and analysis
Recorded movies, in which worm body centerlines were extracted in real time, were further processed semi-automatically in MATLAB to identify locomotor states (forward locomotion, backward locomotion, pause and turn) and other statistical parameters. We also set up a Graphic User Interface (GUI) that allows human interference and proofreading.
Reversal duration was defined as the time from reversal start to reversal end. If one stimulus triggered several reversals, we always scored the first one. Omega turns were identified by either the head touching the body (or tail) or >135˚within a single head swing (Figure 1-figure supplement  1A). The end of a turn was identified when a worm opened its coiled posture and began to move forward. We group trials into time bins. For example, here we use time bin Dt ¼ 1s for illustration. Let n i forw , n i turn denote the number of trials that end with type-I or type-II transition in the i-th time bin. n 1 forw ¼ 8 means there are 8 trials which terminate its reversal with forward movement from 0.0 s to 1.0 s; n 4 turn ¼ 12 means 12 trials terminate its reversal with a turn from 3.0 s to 4.0 s. Next, we shall use S i forw , S i turn to represent the number of trials among type-I or type-II transition which survive to the start of the i-th time bin. Naturally, we have S i forw ¼ P ¥ j¼i n j forw , S i turn ¼ P ¥ j¼i n j turn . For example,

Transition rate calculation
n j turn is the total number of trials that execute a turn. S 3 forw ¼ P ¥ j¼3 n j forw represents the number of all trials among type-I transition that survive to 2.0 s.

Theoretical account of transition rates and model details
Detailed description can be found in Appendix.

Quantification and statistical analysis
Quantification and statistical parameters were indicated in the legends of each figure, including the statistical methods, error bars, n numbers (see Supplementary file 4 for more details), and p values. We applied Mann-Whitney U test, c 2 test or Fisher's exact test among samples, two-way ANOVA to determine the significance of difference between groups for two factors, and Kolmogorov-Smirnov test to compare probability distributions from two samples. All multiple comparisons were adjusted using Bonferroni correction. We considered p values of < 0.05 significant. All analyses were performed using MATLAB.

Data availability
Raw data of calcium imaging experiments and all code used for modeling or figure generation are available for download from https://github.com/Wenlab/Worm-Motor-Sequence-Generation (Xin et al., 2020; copy archived at https://github.com/elifesciences-publications/Worm-Motor-Sequence-Generation). Source data files have been provided for main figures. The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication. . Transparent reporting form

Data availability
All data generated or analysed during this study are included in the manuscript.