Stochastic Thermodynamics and Hierarchy of Fluctuation Theorems with Multiple Reservoirs

We reformulate stochastic thermodynamics in terms of noise realizations for Langevin systems in contact with multiple reservoirs and investigated the structure of the second laws of thermodynamics. We derive a hierarchy of fluctuation theorems when one degree of freedom of the system is affected by multiple reservoirs simultaneously, that is, when noise mixing occurs. These theorems and the associated second laws of thermodynamics put stricter bounds on the thermodynamics of Langevin systems. We apply our results to a stochastic machine in noise-mixing environments and demonstrate that our new bounds play a crucial role in determining the potential function and performance of the machine.


I. INTRODUCTION
In the past two decades, thermodynamics has been extended to studies of stochastic thermal processes observed at the microscopic scale [1][2][3][4][5][6][7][8][9]. These processes are referred to as stochastic thermodynamics (ST). The main achievement in ST has been the discovery of the generalized second laws of thermodynamics, which is formulated by fluctuation theorems [6,9]. As the original second law of thermodynamics initiated studies of macroscopic thermal machines, such as heat engines or refrigerators, ST studies focus on devising various thermal machines with multiple reservoirs at microscopic [10][11][12][13] and even atomic [14] scales. Therefore, it is important to establish a thermodynamic formulation of a stochastic system in contact with multiple reservoirs via a consistent ST framework.
However, theoretical difficulties arise when one degree of freedom of the machine is affected by multiple reservoirs simultaneously, that is, when thermal-noise mixing occurs. This situation can be easily encountered experimentally, for example, when a Brownian particle is immersed in a liquid with a temperature gradient [15]. The noise-mixing setup has also been proposed for a nanosized molecular motor [16] and refrigerator [17], where the rotor effectively described by a single degree of freedom is simultaneously connected to two thermal reservoirs separated by a membrane. In addition, chemical reaction between many chemical species via multiple reaction channels can be formulated as noise-mixing chemical Langevin equation [18].
When noise mixing occurs, the conventional ST approach does not reveal all thermodynamic constraints enforced by the second laws of thermodynamics. For example, the total entropy production (EP) is underestimated [19], and the simple overdamped limit is not * jslee@kias.re.kr † hgpark@kias.re.kr (d) T (1) T (2) x t y t f t (x t ) (2) dQ t (1) dQ t FIG. 1. Steady-state heat machine in the noise-mixing environment. (a) Illustration of the vane system. Two symmetric vanes, V1 and V2, are connected by a rigid axle and immersed in heat reservoirs at temperatures of T (1) and T (2) , respectively. xt is angle of the axle and ft(xt) is an external force. (b, c) are simpler schematic diagrams of the vane system connected to two different reservoirs and one effective reservoir, respectively, where dQ applicable [20]. There are two causes for this problem: i) the total EP for the noise-mixing system could not be obtained by the standard ST based on a probability ratio of system trajectories [21][22][23][24][25][26][27][28][29][30][31] and ii) the full structure of the second laws of thermodynamics for the noise-mixing system has not yet been investigated systematically.
In this study, we establish a ST formulation in terms of noise realizations instead of system trajectories. Our formulation applies to both mixing and non-mixing situ-ations. In the mixing case, many noise realizations correspond to a single system trajectory due to multiple noises affecting the same degree of freedom of the system. This approach allows us to classify a single system trajectory into many noise trajectories with different noise realizations. We derive a hierarchy of fluctuation theorems based on these noise trajectories and the associated second laws of thermodynamics. We find that the total EP could be divided into two parts, the effective and mixing EP, each of which satisfies the fluctuation theorem along with the total EP. Therefore, thermal processes are more constrained by these additional second laws in the noisemixing environment, causing a significant performancebound reduction in a steady-state thermal machine.

II. REFORMULATION OF PATH PROBABILITY IN TERMS OF NOISE REALIZATIONS
We first consider the simplest noise-mixing example, the vane system, shown in figure 1(a), which appears in a prototypical Feynman-Smoluchowski ratchet (FSR) [32,33]. One set of vanes, denoted by V1 (V2), is immersed in a heat reservoir 1 (2) at a temperature of T (1) (T (2) ). V1 and V2 are connected by a rigid axle, such that the rotational motion of the vane system at time t can be described by a single degree of freedom, the angle of the axle x t . Because V1 and V2 are affected by random noise from the respective reservoirs, x t is a stochastic variable. f t (x t ) is an external (conservative/nonconservative/time-dependent) force applied to the vane system. Figure 1(b) is a simpler schematic diagram of figure 1(a); in this diagram, dQ (ν) t represents heat transferred from the reservoirs ν to the vane system, and dW t is the work done by f t (x t ) during an infinitesimal time gap between t and t + dt.
The Langevin equation for the vane system with unit mass (moment of inertia) can be written as [20,[34][35][36][37] where v t is the angular velocity at time t, and γ (1,2) = γ (1) + γ (2) is the composite damping coefficient, which is the sum of the damping coefficient γ (ν) of each reservoir ν. dW is the composite Wiener process at t; it is the sum of two independent Wiener processes dW (ν) t of each reservoir ν, satisfying dW , expressed using the Boltzmann unit (k B = 1).
We then define a state of the vane system at time t as q t = (x t , v t ) and consider its transition from q t to q t+dt . The standard conditional probability, P(q t+dt |q t ) for the time-forward transition of the system from q t to q t+dt is usually given by the Onsager-Machlup transition probability [38]. However, as the transition of our system is implemented by the two Wiener processes simultaneously, it is convenient to introduce new conditional probabilities subject to each noise realization dW (1) t and dW (2) t for a given q t . Such a conditional probability should be given by the multiplication of the probabilities observing each Gaussian noise independently: where P . Clearly, this probability does not depend on the initial state q t (no multiplicative noises); however, we keep this variable for later discussions. The final state q t+dt is determined by a pair of noises (dW t ), starting from the initial state q t , given by equation (1). Note that there is an infinite number of pairs describing the same transition from q t to q t+dt , with the sum dW In fact, this degeneracy allows us to derive additional fluctuation theorems.
As an irreversibility measure, EP involves the probability ratio between the time-forward and time-reverse trajectories [39]. For the time-reverse process, we use qt = ( xt, vt) for a state variable at timet, obeying dynamics identical to those of the time-forward process given in equation (1) with external force ft = f t+dt such that is the composite Wiener process for the time-reverse process att, with the same statistics as d W . The conditional probability for the time-reverse transition from qt to qt +dt can be written as where P By setting qt = εq t+dt and qt +dt = εq t (ε is the parity operator as εq = (x, −v)) with dt = dt, we identify the time-reverse trajectory corresponding to the timeforward one. However, this approach does not fix each value of Gaussian noise (d W We consider the following specific pair of noises as This choice is special in that the heat transfer in the timeforward transition is exactly reversed in the time-reverse transition for each reservoir: where dQ t ) represent heat transferred from reservoir ν to the system during dt in the time-forward and the time-reverse transitions, respectively, and • denotes Stratonovich multiplication [40,41]. If we consider Hamiltonian dynamics for the total system, including heat reservoirs, the time-reverse trajectory of the total system must obey the time-reversal symmetry of heat transfers from each reservoir, so the above choice may be the most appropriate as a definition of the time-reverse process.
With this special pair of time-reverse noises, we define an irreversibility measure by the logarithmic ratio of probabilities for time-forward and time-reverse noise trajectories as where p t (pt) is the probability distribution function for the time-forward (time-reverse) transition at time t (t). With the choice ofpt(qt) = p t+dt (q t+dt ), we obtain the Shannon entropy change of the system as dS t ≡ ln(p t /p t+dt ) [21]. In the derivation of equation (8), we used the relationship dQ t ], which is easily obtained from equations (6) and (7). Our result indicates that dR t can be identified as the total EP, dS tot t (i.e., the sum of the system and environmental EP).

III. HIERARCHY OF FLUCTUATION THEOREMS
Now, we show that dS tot t satisfies the fluctuation theorem: where the Jacobian for integral variable transformations can be easily shown to be unity [42] and the last equality is derived from the probability normalization. This fluctuation theorem holds for any initial condition, any duration, and any external force. Furthermore, it is straightforward to generalize it to the case of many degrees of freedom, where different or the same multiple reservoirs are associated with each degree of freedom. We note that Murashita and Esposito [20] and Forgedby and Imparato [43] also presented the fluctuation theorem for the total EP. However, as the total EP was not defined by the path probabilities in their study, their derivation is considerably more complex than ours.
We can find another quantity satisfying the fluctuation theorem in this noise-mixing situation. We consider the conditional probability to find dW (1,2) t regardless of each value of noise dW (ν) t for a given q t , which can be obtained as where δ W ≡ δ dW is the Dirac delta function and D (1,2) = D (1) + D (2) . Note that the usual Onsager-Machlup transition probability P(q t+dt |q t ) [38] is identical to P(dW (1,2) t |q t ) with a proper Jacobian. Similarly, we obtain (11) Then, the logarithmic ratio of probabilities of the two trajectories turns out to be the following: where dQ (2) t and T (1,2) = D (1,2) /γ (1,2) . Equation (12) describes the total EP of the system connected to a single heat reservoir with an effective temperature of T (1,2) , as illustrated in figure 1(c). Note that T (1,2) is always between T (1) and T (2) .
We call dR ′ t the effective EP, dS eff t . We are then able to prove another fluctuation theorem: where the second equality was derived from equation (10) by integrating over dW  Most interesting is the third quantity, satisfying another fluctuation theorem. We call it the mixing EP and define it as The mixing EP measures the extent of thermal noise mixing and is independent of the system EP, dS t ; it satisfies the following: In summary, we derived three integral fluctuation theorems for noise-mixing systems, which can be rephrased as follows. The total EP can be divided into the two parts, dS mix t and dS eff t , each of which satisfies the fluctuation theorem as well as the total EP. This property is quite similar to the division of the total EP into the adiabatic and non-adiabatic EP in overdamped thermal systems with a single reservoir [23,24,44,45]. As dS tot t and dS eff t are written as the logarithmic ratio of two normalized probabilities, as in equations (8) and (12) with the involution property [44], it is trivial to derive the corresponding detailed fluctuation theorems, such as P (dS tot t )/P (−dS tot t ) = e dS tot t and P (dS eff t )/P (−dS eff t ) = e dS eff t in the steady state. However, the detailed fluctuation theorem does not hold for dS mix t due to the lack of the involution property.
The three integral fluctuation theorems in equations (9), (13), and (15), guarantee the three thermodynamic second laws, using Jensen's inequality [46]: which can be summarized as follows: The non-negativity of dS mix t is interpreted as information loss in the heat reservoir coarse-graining procedure. Numerical test for equation (17) is presented in appendix A. Note that dS mix t ≥ 0 can also be derived using the log sum inequality [46], because the trajectory probability p t (q t )P (1,2) t in the effective single-noise description is simply the log sum of the trajectory probabilities p t (q t )P (1),(2) t in the two-noise description. This property has been reported previously for the master equation system [44]. We also note that a richer hierarchy can be found between the total, effective, and mixing EPs in a general n-reservoir system (n ≥ 3) (see appendix B).

IV. REDUCTION OF PERFORMANCE BOUND
Each second law in equation (16) constrains a thermodynamic process in a different manner. As expected, the effective and mixing EPs enforce a tighter performance bound for a noise-mixing machine than the total EP. As an example, consider the vane system illustrated in figure 1(a) with T (1) > T (1,2) > T (2) . The three second laws of thermodynamics in the steady state (or cyclic state) are given as follows: where 1/T (m,1) = −1/T (1) + 1/T (1,2) > 0 and 1/T (m,2) = 1/T (2) − 1/T (1,2) > 0. Figure 2(a) shows a generic thermodynamically allowed region for dQ (1) t s and dQ (2) t s . From figure 2(a), we can deduce a general feature of the performance of the vane system. Defining the work extraction against the external force during the forward transition as dW t , energy conservation yields dW t = dQ (1) t +dQ (2) t . For a useful heat engine, we require that dW t s = dQ The second law of the total EP alone, equation (18), permits a useful engine in the region shown in figure 2(a). However, this result is completely prohibited by the second law of the effective EP, equation (19), manifesting the thermodynamic importance of additional second laws. In fact, our rigorous result extends the thermodynamic statement "Work cannot be extracted from an engine connected to a single reservoir" to the case of an engine connected simultaneously to multiple reservoirs which can be described by the Langevin equation (1) without any spatial asymmetry [47]. We also consider a refrigerator extracting heat from a low-temperature reservoir by external force, implying dQ (2) t s > 0. This case does not contradict dS tot t s alone, but is forbidden by the second law of the mixing EP, equation (20).
It might be useful to add another system to interact with the vane system. One of the simplest examples is an FSR-type machine, as illustrated in figure 1(d) [48,49]. The added system (a pawl) is described by another stochastic variable, y t , which is in contact with the reservoir 2. It is straightforward to show the three fluctuation theorems similarly and the corresponding second laws in the steady state as follows: where dQ R E (1) <dQt >s (2) <dQt >s (1) <dQt >s (1) <dQt >s (p) <dQt >s < 0 (p) <dQt >s > 0 (2) <dQt >s (2) <dQt >s namically allowed region bounded by the three second laws for the negative and positive dQ (p) t s , respectively. We require dW t s = dQ s > 0 for a useful refrigerator. We demonstrate a thermodynamically allowed region for a useful engine in figure 2 (b) and for a useful refrigerator in figure 2 (c).
As described above, the thermodynamically allowed region is significantly reduced in the mixing system. However, we find that heat engine efficiency, defined as η = dW t s / dQ (1) t s , can still attain the Carnot efficiency η C = 1−T (2) /T (1) at the uppermost corner of the allowed region in figure 2 (b), where all three EPs vanish simultaneously. This efficiency is higher than the maximum efficiency of the effective system described by figure 1 (f), i.e., η C > 1 − T (2) /T (1,2) , because dQ (2) t s > 0 near the corner point of the noise-mixing system, which implies that the engine absorbs extra heat from reservoir 2 to convert into work. This problem is quite similar to the famous cooling-by-heating problem, in that it introduces an additional high-temperature reservoir [50][51][52]. The refrigerator in figure 2 (c) also performs most efficiently at the uppermost corner, where the coefficient of performance, defined as K = [ dQ (2) t s is always negative, such that extra heat flows into reservoir 2 through the machine; thus, the refrigerator performs poorly compared to the effective system.

V. SUMMARY
By reformulating the stochastic thermodynamics in terms of noise realizations, we derived three fluctuation theorems, which constrain a thermal process with multi- ple reservoirs more strictly using the effective and mixing EPs than it would using only the total EP. Nevertheless, attaining ideal efficiency remains possible. Further study of the possible setup for obtaining the Carnot efficiency is necessary.
As an example, consider the vane system presented in figure 1(a) with f t (x) = −k(x − ct). This external force can be realized by a rotating torsional spring with stiffness k and constant angular velocity c. By solving the Langevin equation (1) numerically, we calculate the average rates of heat flows in the steady state as Q (1) s and Q (2) s , where Q (ν) s ≡ dQ (ν) t /dt s . For the steady-state average, we collect 10 5 data after the system reaches the steady state. From equations (18) and (19), the total and effective EP rates become respectively. For this calculation, we use the second-order integrator [53] with parameters k = 1, γ (1) = γ (2) = 1, T (1) = 3, and T (2) = 1, thus T (1,2) = 2. Figure A1 shows Ṡ tot s and Ṡ eff s , denoted by squares and circles respectively, as a function of c. At c = 0, we can analytically calculate the heat flux rate as Q (1) s = γ (1) γ (2) (T (1) − T (2) )/(γ (1) + γ (2) ) = − Q (2) s [20]. Thus, we get Q (1) s = 1 = − Q (2) s along with Ṡ tot s = 2/3 and Ṡ eff s = 0, which is confirmed in figure A1. As the external driving becomes stronger with increasing c, both Ṡ tot s and Ṡ eff s increase, still satisfying the inequality (A1). Note that the difference between Ṡ tot s and Ṡ eff s is the mixing EP rate which is always larger than zero. This numerical result clearly supports our theory for the second laws of the total, effective, and mixing EPs.