Abstract
Most natural and engineered information-processing systems transmit information via signals that vary in time. Computing the information transmission rate or the information encoded in the temporal characteristics of these signals requires the mutual information between the input and output signals as a function of time, i.e., between the input and output trajectories. Yet, this is notoriously difficult because of the high-dimensional nature of the trajectory space, and all existing techniques require approximations. We present an exact Monte Carlo technique called path weight sampling (PWS) that, for the first time, makes it possible to compute the mutual information between input and output trajectories for any stochastic system that is described by a master equation. The principal idea is to use the master equation to evaluate the exact conditional probability of an individual output trajectory for a given input trajectory and average this via Monte Carlo sampling in trajectory space to obtain the mutual information. We present three variants of PWS, which all generate the trajectories using the standard stochastic simulation algorithm. While direct PWS is a brute-force method, Rosenbluth-Rosenbluth PWS exploits the analogy between signal trajectory sampling and polymer sampling, and thermodynamic integration PWS is based on a reversible work calculation in trajectory space. PWS also makes it possible to compute the mutual information between input and output trajectories for systems with hidden internal states as well as systems with feedback from output to input. Applying PWS to the bacterial chemotaxis system, consisting of 182 coupled chemical reactions, demonstrates not only that the scheme is highly efficient but also that the number of receptor clusters is much smaller than hitherto believed, while their size is much larger.
- Received 11 April 2023
- Accepted 14 September 2023
DOI:https://doi.org/10.1103/PhysRevX.13.041017
Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.
Published by the American Physical Society
Physics Subject Headings (PhySH)
Popular Summary
The ability to process information is fundamental to all living beings as well as many devices that we rely on in our daily lives. It was Shannon who realized nearly 70 years ago that information can be precisely quantified mathematically: The performance measure for information-processing systems is the information transmission rate, which quantifies how many bits per second the system transmits. Until now, this quantity could be computed only approximately. In this work, we present a Monte Carlo simulation scheme that makes it possible, for the first time, to compute the information transmission rate exactly for a large class of systems, including biochemical and neural networks, electronic circuits, and mechanical and optical systems.
Our scheme, called path weight sampling (PWS), can not only compute the information transmission rate for theoretical models of these systems, but also estimate the rate from experimental observations using effective models obtained from empirical data. PWS is made efficient by establishing connections between quantities from information theory and statistical physics, such as path probabilities and partition functions, making it possible to leverage techniques from soft condensed-matter physics, like transition path sampling and polymer-simulation algorithms. We demonstrate the power of PWS by applying it to one of the best characterized signaling systems in biology: the bacterial chemotaxis system, consisting of 182 coupled chemical reactions.
PWS is a generic technique for computing the information transmission rate in complex nonlinear stochastic systems. We envision that PWS will become an important and flexible tool for studying information transmission in systems across such varied domains as optomechanical microresonators, stochastic heat and information engines, complex scattering media, optical cavities, and living systems.
