On a Mathematical Theory of Coded Exposure

This paper proposes a mathematical model and formalism to study coded exposure (ﬂutter shutter) cameras. The model includes the Poisson photon (shot) noise as well as any additive (readout) noise of ﬁnite variance. This is an improvement compared to our previous work that only considered the Poisson noise. Closed formulae for the mean square error and signal to noise ratio of the coded exposure method are given. These formulae take into account for the whole imaging chain, i.e., the Poisson photon (shot) noise, any additive (readout) noise of ﬁnite variance as well as the deconvolution and are valid for any exposure code. Our formalism allows us to provide a curve that gives an absolute upper bound for the gain of any coded exposure camera in function of the temporal sampling of the code. The gain is to be understood in terms of mean square error (or equivalently in terms of signal to noise ratio), with respect to a snapshot (a standard camera).


Introduction
Since the seminal papers [1,2,3,4,5,6] of Agrawal, Raskar et al. coded exposure (flutter shutter) method has received a lot of follow ups [7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39]. In a nutshell, the authors proposed to open and close the camera shutter, according to a sequence called "code", during the exposure time. By this clever exposure technique, the coded exposure method permits one to arbitrarily increase the exposure time when photographing (flat) scenes moving at a constant velocity. Note that with a coded exposure method only one picture is stored/transmitted. A rich body of empirical results suggest that the coded exposure method allows for a gain in terms of Mean Square Error (MSE) or Signal to Noise Ratio (SNR) compared to a classic camera, i.e., a snapshot. Therefore, the coded exposure method seems to be a magic tool that should equip all cameras.
We now briefly expose the different applications, variants and studies that surround the coded exposure method. An application of the coded exposure method to bar codes is given in [16,35], to fluorescent cell image in [27], to periodic events in [25,34,36], to multi-spectral imaging in [10] and to iris in [21]. Application to motion estimation/deblurring are presented in [9,19,20,26,31,33,37,38]. An extension for space dependent blur is investigated in [28]. Methods to find better or optimal sequences as investigated in [12,13,14,22,23,39] or in [15] that aims at adapting the sequence to the velocity. Diverse implementations of the method are presented in [17,18,24,32]. The method is used for spatial/temporal trade-off in [7,8,11]. A numerical and mathematical investigation of the gain of the method is in [29,30] but their camera model contains only photon (shot) noise and neglects all other noise sources, contrarily to the model we shall develop in this paper.
Therefore, as far as we know, little is known on the coded exposure method from a rigorous mathematical point of view and it seems useful for the applications to build a theory able to shed some light on this promising coded exposure method. For instance, to the best of our knowledge, little is know on the gain, in terms of MSE and SNR, of this coded exposure method compared to a standard (snapshot) camera. This paper proposes a mathematical model of photon acquisition by a light sensor. The model can cope with any additive readout noise of finite variance in addition to the Poisson photon (shot) noise. The model is compatible with the Shannon-Whittaker framework, assumes that the relative camera scene velocity is constant and known, that the sensor does not saturate, that the readout noise has finite variance and that the coded exposure method allows for an invertible transformation among the class of band limited functions (this means that the observed image can be deblurred using a filter). Note that with this model the image has a structure: the image is assumed to be band limited. This set of assumptions represent an ideal mathematical framework that allows us to give a rigorous analysis of the limits, in terms of MSE and SNR, of the coded exposure method. For instance, it is clear that the MSE (resp. SNR) will increase (resp. decrease) if one needs to estimate the velocity from the observed data, compared to the formulae we shall prove in this theoretical paper.
To be thorough, a mathematical analysis of a camera requires to go rigorously from the continuous observed scene to the discrete samples of the final restored image. This is needed to mathematically analyse the whole image chain: from the photon emission to the final restored image via the observed discrete samples measured by the camera. As far as we know, the coded exposure method is very useful for moving scenes. Consequently, we need a formalism capable of dealing with moving scenes. Since the observed scene moves continuously with respect to the time we adopt a continuous point of view. This means that we shall model the observed scene as a function s. Loosely speaking, spxq give the light intensity at a spatial position x. (By opposition, a discrete formalism would model the observed scene as a vector of R n but requires a more restrictive assumption, see below.) We shall rely on the Shannon-Whittaker framework (see, e.g, [40]) to perform the mathematical analysis of sampling-related questions. This framework requires the structure of band limited (with a cut off frequency) signals or images and will allow us to perform a rigorous mathematical analysis of the coded exposure method. Recall that a discrete formalism would model the observed scene as a vector of R n and the convolution would use Toepliz matrices. Therefore, the scene would be assumed to be periodic and also band limited for sampling purposes. Note that the continuous formalism that we shall develop in this paper does not require to assume that the observed scene s is periodic. (Most natural scenes are not periodic.) However, the adaptation of the formalism that we shall develop in this paper to periodic band limited scene is straightforward if needed for some application.
Our first goal is to provide closed mathematical formulae that give the MSE and SNR of images obtained by a coded exposure camera. Therefore, we shall start by carefully model the photon acquisition by a light sensor then deduce a mathematical model of the coded exposure method. The mathematical model of camera that we shall develop in this paper has not, to the best of our knowledge, been developed in the existing literature on the coded exposure method. Indeed, the model we shall develop in this paper is able to cope with the Poisson photon (shot) noise in addition to any additive (sensor readout) noise of finite variance and does not require to assume that the observed scene is periodic. For example, the model developed in [30] does not consider any additive (sensor readout) noise. The formulae that give the MSE and SNR of the final crisp image • Assumes the Shannon-Whittaker framework that i) requires band limited (with a frequency cut off) images, and that ii) the pixel size is designed according to the Shannon-Whittaker theory. In this paper, we prove the validity of the Shannon-Whittaker for non-stationary noises. (See also section 2.2 page 10.) • Assumes that the relative camera scene velocity is constant and known.
• Assumes that the sensor does not saturate.
• Assumes that the additive (sensor readout) noise has zero mean and finite variance. (This term contains, without loss of generality, the quantization noise.) • Assumes that the coded exposure allows for an invertible transformation among the class of band limited functions. (This means that the observed image can be deblurred using a filter.) • Neglects the boundaries effects for the deconvolution. (The inverse filter of a coded exposure camera has larger support than the inverse filter of a snapshot. Thus, this slightly overestimates the gain of the coded exposure method with respect to the snapshot.) We assume that the sensor readout (additive) noise has zero mean. However, with our formalism, the adaptation to non zero mean additive (sensor readout) noise is straightforward if needed for some application. This zero-mean assumption for the additive noise can be found in e.g. [

A mathematical model of classic cameras
The goal of this section is to provide a mathematical model of the photon acquisition by a light sensor and the formalism that we shall use to model the coded exposure method in the sequel. As usual in the coded exposure literature [2,3,9,21,22,30,35,46] and for the sake of the clarity we shall formalize the coded exposure method using a one dimensional framework. In other words, the sensor array and the observed image are assumed be one dimensional. One could think that this one dimensional framework is a limitation of the theory. However, this one dimensional framework is no limitation. Indeed, as we have seen, we assume that the image acquisition obeys the Shannon-Whittaker sampling theory. This means that the frequency cut off is compatible with the image grid sampling. The extension to any two dimensional grid (and two dimensional images) is straightforward. (The sketch of the proof is in annex A page 18.) Therefore, the one dimensional framework that we shall consider is no limitation for the scope of this paper that proposes a mathematical analysis of coded exposure cameras. A fortiori, the calculations of MSE and SNR that we shall propose in this paper remain valid for two dimensional images. The noise is, in general, non-stationary. This due both to the sensor (see, e.g., [47]) and to the observed scene. In this paper, we also prove the validity of the Shannon-Whittaker interpolation is valid for non-stationary noises. (See also section 2.2 page 10.) In addition, we shall assume that the motion blur kernel is known, i.e., the relative camera-scene velocity vector and the exposure code (or function) are known (this kernel is called "PSF motion" in, e.g., [1] and is also assumed to be known [1, p. 2]).
We now turn to the mathematical model of photons acquisition by a light sensor.

A mathematical model of photons acquisition by a light sensor
The goal of this subsection is to give a rigorous mathematical definition (see definition 2 page 10) of the samples produced by a pixel sensor that observes a moving scene. This definition of the observed sample can cope with any additive zero mean (sensor readout) noise of finite variance in addition to the standard Poisson photon (shot) noise. Note that the model developed in [30] do not consider any additive (sensor readout) noise. Therefore, the results of [30] do not include this more elaborated mathematical model. In particular, the advantages of the coded exposure method in terms of MSE, with this more elaborated set up, are, to the best of our knowledge, open questions. We consider a continuous formalism in order to ease the transition from steady scenes to scenes moving at an arbitrary real velocity. Another advantage of this continuous formalism is that it allows us to avoid the implicit periodic assumption of the observed scene needed if one uses Toeplitz matrices to represent the convolutions see, e.g., [1, equation 2, p. 3]. (This is needed because, in general, natural scenes are not periodic. ) We now sketch the construction of our camera model. We first consider the photon emission, then include the optical and sensor kernels, then include the effect of the exposure time and of the motion to our model. The lasts two steps consists in adding the Poisson photon (shot) noise and the additive (sensor readout) noise to our camera model. The camera model that we shall consider in this paper is depicted in figure 1. Figure 1 Schematic diagram of our camera model. The observed scene emits light and moves at velocity v P R. The light undergoes the blur of the optical system and is measured by a pixel sensor. The pixel sensor produces a Poisson random variable (shot noise) that is further corrupted by an additive (sensor readout) noise of finite variance to produce the observed sample.
We assume that the observed scene emits photons at a deterministic rate s defined by s : R ÝÑ p0,`8q x Þ ÝÑ spxq.
Here and in the sequel, the variable x P R represents the spatial position. (We will precise the unit of x, i.e., the unit we shall use to measure distances when we introduce the pixel sensor.) Intuitively, s represents the ideal crisp image, i.e., the image that one would observed if there were no noise whatsoever, no motion, with a perfect optical system (formally the point spread function is a Dirac-mass) and the pixel sensor has an infinitesimal area. In a nutshell, spxq would be the gray-level of the image at position x P R in the idealistic case mentioned above. The quantity spxq can also be seen as the intensity of light emission at position x. We now introduce the optical system in our model. The effect of the optical system is described by its point spread function (PSF) denoted g, and we assume that g ě 0.
Formally, the effect of the point spread function is modeled by a convolution in space (see, e.g., [48, equation 7.1 p. 171] see also, e.g. [1, equation 1, section 2]). Therefore, in the noiseless case, if there is no motion, the grey level of the acquired image at position x P R is, formally, described by pg˚sqpxq (1) where˚denotes the convolution (see (ix) for the definition). (Recall that here and in the rest of the text, Latin numerals refer to the formulae in the final glossary page 34.) We shall give the assumptions on g and s so that the quantity in (1) is well defined later on. A pixel sensor can be small but has nevertheless a positive area. Indeed, a pixel sensor integrates the incoming light g˚s (the scene is observed through the optical system) on some surface element of the form rx 1 , x 2 s Ă R with x 1 ă x 2 . Therefore, formally, the output of a pixel sensor supported by rx 1 , x 2 s, in the noiseless case and without motion, is ż x2 x1 pg˚sqpyqdy. ( In the sequel we shall assume that all the pixel sensors of the sensor array have the same length. Mathematically, we can normalize this length so that every pixel sensors of the array have unit length. This corresponds to using the pixel sensor length as unit to measure distances. Thus, this represents no limitation. Hence, from now on the unit of x is the pixel sensor length. By definition, with this unit, all the pixel sensor have lengths 1. Therefore, from now on when we speak of a pixel sensor centered at x we mean that the pixel sensor is supported on the interval rx´1 2 , x`1 2 s. Hence, from (2) we deduce that the output of a pixel sensor supported on the interval rx´1 2 , x`1 2 s, that stares at the scene s through the optical system modeled by g, is, in the noiseless case and without motion, Remark We implicitly assume a 100% fill factor for the sensor as the pixel sensor is supported on r´1 2 , 1 2 s and we have a pixel sensor at every unit. This is no loss of generality for studying the gain of the flutter with respect to a snapshot. Indeed, the fill factor impacts equally the snapshot and the flutter. In addition, the RMSE calculations are carried out using the function u in (3) as reference and using an unbiased estimator for u. Thus, all results we give in this paper hold if one replace u by u " 1 r´ε,εs˚g˚s in (3) for any ε P p0, 1 2 s. Consider the deterministic function formally defined by u :" 1 r´1 2 , 1 2 s˚g˚s . The deterministic quantity upxq represents the grey level of the image at position x if there were no noise and no motion. Indeed, u contains the kernels of the optical system g and of the sensor. Note that the quantity upxq can also be seen as an intensity of light emission received by a unit pixel sensor centered at x. With the formalism of, e.g., [1, equation 1, section 2] 1 r´1 2 , 1 2 s represents "h sensor " and g represents "h lens ". We now introduce the exposure time in our model. Indeed, the sensor accumulates the light during a time span of the form rt 1 , t 2 s Ă R, with t 1 ă t 2 . We denote by ∆t the positive quantity ∆t :" t 2´t1 that we shall call exposure time. Thus, from (3), the output of a pixel sensor centered at x that integrates on the time interval rt 1 , t 2 s is, in the noiseless case Note that the quantity in (4) is the amount of light measured by the pixel sensor, and it evolves linearly with the exposure time ∆t p" t 2´t1 q. We now extend the above formalism to cope with moving scenes. Without loss of generality (w.l.o.g.) we assume that the camera is steady while the scene s moves.
The coded exposure method permits to deal with uniform motions. Therefore, we assume that the scene s moves at a constant velocity v P R (measured in pixel per second) during the exposure time interval rt 1 , t 2 s. This means that the scene evolves with respect to the time as spx´vtq. Here and in the sequel the temporal variable is denoted by t. Therefore, from (4) we deduce that the output of a pixel pixel sensor centered at x and integrating on the time interval rt 1 , t 2 s is, in the noiseless case, For example, suppose that we take a constant velocity v " 1 in (5). In this case, the output of a pixel sensor centered at x is, in the noiseless case, where 1 ra,bs represents the characteristic function of the interval ra, bs. (See (3) for the last equality.) From this simple example we can qualitatively describe where the exposure code will act. Indeed, by a clever exposure technique, the coded exposure method will allow to replace the function 1 rt1,t2s in the above formula by a more general class of functions that does not need to be window functions. With the formalism of, e.g., [1, equation 1, section 2] 1 rt1,t2s represents "h motion " for a classic camera.
We now extend our model to cope with the Poisson photon (shot) noise and then will add the readout noise. The photon emission follows a Poisson distribution see, e.g., [49]. (If X is a random variable that follows a Poisson distribution then all the possible realization of X are in N. In addition, the probability of the event X " k is PpX " kq " λ k e´λ k! , where λ ą 0 is the intensity of the Poisson random variable.) We assume that a pixel sensor is behave as a photon counter [1] . That is to say, we assume that a pixel sensor integrates the photons that are emitted by the moving observed scene s on some surface element of the form rx 1 , x 2 s on the time span rt 1 , t 2 s and produces a sample. This sample follows a Poisson random variable. From (5), this means that the sample produced by a pixel sensor supported by rx 1 , x 2 s and that integrates on the time span rt 1 , t 2 s has law where the notation Ppλq denotes a Poisson random variable of intensity λ. With (3) the above equation can be rewritten as upx´vtqdt˙.
[1] Single photon avalanche detectors (SPADS) is a possible implementation of a photon counter. Any light sensing device that produces, when the quantization is neglected, a signal in biunivocal relationship with the photon count can be w.l.o.g. assumed to be photon counter. The quantization noise will w.l.o.g. be included in the additive (readout) noise later on.
Thus, the value of this sample can be any realization a Poisson random variable with intensity ş t2 t1 upx´vtqdt. Consequently, the probability that the sample has value k P N when observing the scene s on the time span rt 1 , t 2 s with the pixel sensor centered at x iś This quantity is nothing but the probability that the pixel sensor counts k P N photons during the time interval rt 1 , t 2 s. With the formalism we introduced we can compute the SNR of the produced image, just to verify that we retrieved the fundamental theorem of photography that underlies statements like "the capture SNR increases proportional to the square root of the exposure time" that can be found in, e.g., [2, p. 1, column 2, 1st paragraph]. To this aim, consider the case where v " 0, t 1 " 0, t 2 " ∆t in (7). If the observed value obspxq at position x P R follows P´ş ∆t 0 upxqdt¯we have Epobspxqq " ∆tupxq. This means that, in expectation, the number of photon caught by the pixel sensor centered at x increases linearly with the exposure time. If we time normalize the obtained quantity and consider, formally, a random variable u est pxq that follows Pp ş ∆t 0 upxqdtq ∆t we obtain E pu est pxqq " upxq. This means that u est pxq estimates upxq without bias. In addition, we have var pu est pxqq " ∆tupxq ∆t 2 " upxq ∆t . Consider the SNR on the spatial interval r´R, Rs given by Therefore, assuming that µ the "mean signal level" [48, p. 42

2R
ż Ŕ R E´|u est pxq´upxq| 2¯d x, whenever the limit exists, and we have MSE pu est q " µ∆t.
We are now in position to extend our model to include the additive (readout) noise. Here and in the sequel the additive (readout) noise of a pixel sensor centered at x is modelled by a zero mean real random variable of finite variance denoted by ηpxq. Therefore, from (6), the output of the pixel sensor centered at x that integrates photons on the time span rt 1 , t 2 s can be, formally, any realization of the sum of the random variables or equivalently (see (3)), Recall that the deterministic quantity upxq represents the grey level of the image at position x if there were no noise and no motion as it is seen by a pixel sensor centered at x. The quantity ş t2 t1 upx´vtqdt represents the amount of light received on the time interval rt 1 , t 2 s by a steady pixel sensor centered at x that gathers the light emitted by the observed scene that moves at velocity v P R.
We now give a mathematical framework to make precise the above formulae. We shall assume that the scene s P L 1 loc pRq so that the convolution in (3) page 6 is well defined. We shall assume that the PSF g belongs to the Schwartz class that, hereinafter, we shall denote SpRq. In addition, we shall assume that the PSF g P SpRq furnishes a cut off frequency. This assumption is needed by the Shannon-Whittaker sampling theory. We shall assume that the frequency cut off of g is π, i.e., g is r´π, πs band limited. In other words,ĝpξq " 0 for any ξ P R such that |ξ| ą π, where, here and in the sequel, we denote byĝ or Fpgq the Fourier transform of g (see (xiv) for the definition of the Fourier transform) and (here and elsewhere) ξ P R represents the (Fourier) frequency coordinate. One could think that this r´π, πs is a limitation for the theory. However, it is not. The choice of r´π, πs in the following definition is thoroughly justified in annex B page 20. Definition 1 (The observable scene u.) We call observable scene any non negative deterministic function u of the form u " 1 r´1 2 , 1 2 s˚g˚s . Recall that the 1 r´1 2 , 1 2 s denotes the characteristic function of the interval r´1 2 , 1 2 s and is related to the normalized pixel sensor. The PSF satisfies g P SpRq, g ě 0, and is r´π, πs band limited. The (non negative) photon emission intensity is denoted s P L 1 loc pRq. We have that u P L 1 loc pRq and we assume that u satisfies µ :" lim T RÑ`8

2R
ş Ŕ R upxqdx P R`. In addition, we assume that u :" u´µ P L 1 pRq X L 2 pRq. Note that u is the sum of the constant µ and ofũ P L 1 pRq. Thus, we have u P S 1 pRq (the space of tempered distributions). This means that u enjoys a Fourier transform in S 1 pRq, see, e.g., [50, p. 173], see also [51, p. 23]. In addition, u andũ inherit the frequency cut off of the PSF g. Therefore, u andũ are r´π, πs band limited. In addition note that the assumptionũ P L 2 pRq is w.l.o.g. Indeed, sinceũ P L 1 pRq, from Riemann-Lebesgue theorem (see e.g., [52, prop. 2.1]), we have thatû is continuous. In addition, sinceũ is r´π, πs is band limited we have thatû is continuous and has compact support. We deduce thatû P L 2 pRq. Therefore, we obtain thatũ P L 2 pRq w.l.o.g.
We can now give a definition of the observed sample at a pixel centered at x P R that we shall denote obspxq. Definition 2 (Observed sample of a of pixel that includes any additive (sensor readout) noise of finite variance in addition to the Poisson photon (shot) noise.) We assume that the observed sample produced by a unit pixel sensor centered at x P R is corrupted by an additive noise ηpxq that we shall call readout noise. We assume that Epηpxqq " 0 and that varpηpxqq " σ r ă`8. Hereinafter, we shall denote this observed sample by obspxq. From (9), we have that obspxq satisfies, for any x P R, where rt 1 , t 2 s is the time exposure interval, the observable scene u is defined by definition 1 page 9 and moves at velocity v P R. The notation X " Y means that the random variables X and Y have the same law.
In the sequel we will need to compute MSEs as well as SNRs. Therefore, we will need to compute expected values and variances of the observed samples. Thus, we need to justify the validity of these operations. This is done in annex C page 20.
The definition 2 entails that obspxq, the observed sample of a pixel sensor centered at position x, is a measurable function (a random variable see, e.g., [53, p. 168]) for which it is mathematically possible to compute, e.g., the expectation and the variance.
The images produced by a digital camera are discrete. In addition, the image obtained by a coded exposure camera requires to undergo a deconvolution to get the final crisp image. The calculation of the adequate deconvolution filter requires a continuous model. Thus, we now turn to the sampling and interpolation in order to go comfortably from the discrete observations to the latent continuous image.

Sampling and interpolation
This section recalls the principles of the Shannon-Whittaker interpolation that applies to, e.g., images that have the band limitedness structure. Consider a r´π, πs band limited deterministic function f P L 1 pRq X L 2 pRq. From the values f pnq for n P Z the Shannon-Whittaker interpolation of f is and the above series converges uniformly to f pxq for any x P R (see, e.g., [40, p. 354]). We recall that annex B page 20 proves that it is no loss to assume that u is r´π, πs band limited. However, the sample obspxq defined in definition 2 page 10 produced by the sensor is noisy. Indeed, the sample obspxq contains the Poisson photon (shot) noise and the additive sensor readout noise. This means that R Q x Þ Ñ obspxq is not a deterministic function and that obs does not belong to any Lebesgue space. The Shannon-Whittaker theorem is usually applied to deterministic functions. Some generalization exists in the case where the observed samples are corrupted by an additive noise, see, e.g., [54, p. 111], or to sample wide sense stationary stochastic signals, see, e.g., [54, p. 148]. However, the Poisson photon shot noise is not additive. Therefore, the first generalization is not applicable. In addition, from definition 1 page 9 we deduce that the autocorrelation function E pobspxqobspyqq is not a function of the variable x´y. This means that the samples of a coded exposure camera cannot be seen as the samples of a wide sense stationary stochastic process (see, e.g., [55, p. 17] for the definition). In addition, the sensor itself can introduce non stationary noise, see, e.g., [47]. Thus, to the best of our knowledge, the existing generalizations of the Shannon-Whittaker theorem are not sufficient to treat the observed samples of a coded exposure camera (defined in definition 2 page 10). Consequently, in the sequel, we shall carefully prove that is mathematically feasible for the obs defined in definition 2 page 10.
Therefore, in the sequel, we assume that the observed samples are obtained from a sensor array and that the sensor array is designed according to the Shannon-Whittaker sampling theory. Thus, we assume that the samples obspxq are obtained at a unit rate, i.e., for x P Z. Consequently, we shall denote the observed samples by obspnq. This means that, in the sequel, we shall neglect the boundaries effect due to the deconvolution. This is another way to get rid of the boundaries effects without assuming that the observed scene is periodic as required by linear algebra (with Toepliz matrices) model based (see, e.g., [1,2,3,12,5,25,29,30]).
(This is needed because most natural scenes are not periodic.) Note that this slightly overestimates the gain of the coded exposure method with respect to the snapshot. Indeed, the support of the coded exposure function is larger than the support of the exposure function of a snapshot. This means that in practice the boundaries artifacts due to the deconvolution are stronger with the coded exposure method.
Hereinafter, we assume that the sequence of random variables pηpnqq nPZ are mutually independent, identically distributed and independent from the shot noise, i.e., independent from P´ş t2 t1 upn´vtqdt¯where n P Z. This independence assumption represents no limitation for the model. Indeed, a photon can only be sensed once. In addition, the additive (sensor readout) noise comes from an inaccurate reading of the sample value that does not depends on the light intensity emission or on the Poisson photon (shot) noise.
We have defined the observed samples produced by a light sensor in definition 2 page 10. This definition includes both the Poisson photon (shot) noise and an additive (readout) noise of finite variance. We now turn to the mathematical formalization of the coded exposure method.

A mathematical model of coded exposure camera that includes any additive (sensor readout) noise of finite variance in addition to the Poisson photon (shot) noise
The goal of this section is to formalize the coded exposure method. In this section, we consider invertible "exposure codes" and provides the MSE and SNR of these exposure strategies. The study yields to theorem 3.4 page 15.
The coded exposure (flutter shutter) method permits to modulate, with respect to the time, the photons flux caught by the sensor array. Indeed, the Agrawal, Raskar et al. coded exposure method [1,2,3,4,5,6] consists in opening/closing the camera shutter on sub-intervals of the exposure time. In such a situation the exposure function that controls when the shutter is open or closed is binary and piecewise constant. Since it is piecewise constant it is possible to encode this function using an "exposure code". (We give a mathematical definition of these objects page 12.) Note that neither the model nor the results of [30] can be used in this paper. Indeed, in [30] the additive (sensor readout) noise is neglected. Therefore, the formalism of [30] does not hold with the more elaborated set up that we shall consider here. Indeed, this paper considers any additive sensor readout noise of finite variance in addition to the Poisson photon (shot) noise.
As we have seen, in their seminal work [1, 2, 3, 4, 5, 6], Agrawal, Raskar et al. propose to use binary exposure codes. Yet, mathematically, one could envisage smoother exposure codes that are not binary. Indeed, with a bigger searching space for the exposure code the MSE and SNR can be expected to be better than with the smaller set of binary codes. Therefore, in the sequel, we shall assume that the exposure codes have values in r0, 1s. The value 0 means that the shutter is closed while the value 1 means the shutter is open and, e.g., 1 2 means that half of the photons are allowed to reach the sensor. We do not consider the practical feasibility of these non binary exposure codes as this is out of the scope of this paper which proposes a mathematical framework and formulae.
We first formalize the fact that the exposure code method modulates temporally the flux of photons that are allowed to reach the sensor by giving an adequate definition of an "exposure function" that, hereinafter, we shall denote α. To be precise, the gain αptq at time t is defined as the proportion of photons that are allowed to travel to the sensor. We then give the formula of the observed samples taking the exposure function into account (see definition 4 page 13). Definition 3 (Exposure function, exposure code.) We call exposure function any function α of the form We assume that a k P r0, 1s, for any k, that pa k q k P 1 pZq and that ∆t ą 0. The sequence pa k q k is called exposure code.
Remark It is easy to see that α P L 1 pRq X L 2 pRq X L 8 pRq and that the above definition can cope with finitely supported codes, e.g., the Agrawal, Raskar et al. code [5, p. 5] and patent application [6].
We have defined the exposure function that controls with respect to the time the camera shutter. We now give the formula of the observed samples of a coded exposure camera.
Let α be an exposure function and upx´vtq a scene moving at velocity v P R. Recall that αptq is nothing but the percentage of photons allowed to reach the sensor at time t. Therefore, from definition 2 page 10, we deduce that obspnq, the observed sample at a position n P Z, is a random variable that satisfies, for any n P Z, This yields Definition 4 (Observed samples of a coded exposure camera.) Let α be an exposure function. We call observed samples at position n of the scene u (defined in definition 1 page 9) moving at velocity v P R the random variable Recall that the random variables obspnq observed for n P Z are mutually independent (see page 11). From definition 1 page 9 we have that u is of the form L 1 pRq plus constant and is band limited. From definition 3 page 12 we have that α P L 1 pRq.
We obtain that the convolution in (13) is well defined everywhere. In addition, note that the pixels are read only once as in, e.g., [1,2,3,4,5,6]. (Only one image is observed, stored and transmitted.) Proof The proof is a direct consequence of definition 4 page 13.
Remark (The motion blur of a standard camera is not invertible as soon as its support exceeds two pixels.) A standard camera can be seen as a coded exposure camera if the exposure function α is of the form α " 1 r0,∆ts , where ∆t ą 0 is the exposure time measured in second(s). Consider the idealistic noiseless case where, from (13), one would observe From definition 1 page 9 u is r´π, πs band limited. Therefore, we deduce that the convolution in (15) is non invertible as soon as the Fourier transform of 1 |v| 1 r0,∆ts`vȋ s zero on r´π, πs. For any ξ P R, we have F´1 |v| 1 r0,∆ts`v˘¯p ξq " F`1 r0,∆ts˘p ξvq. In addition, from the definition of the Fourier transform (xiv), for any ξ P R we have Therefore, for any ξ P R, we have F´1 |v| 1 r0,∆ts`v˘¯p ξq " ∆tsinc´ξ v∆t 2π¯e´i ξv∆t 2 . From the definition (xvi) of the sinc function, we deduce that the convolution in (15) is not invertible as soon as the blur support |v|∆t satisfies |v|∆t ě 2. Since, the velocity v is measured in pixel(s) per second, and the exposure time ∆t is measured in second(s) we deduce that as soon as the blur support |v|∆t exceeds two pixels the motion blur of a standard camera is not invertible.
The observed samples of any coded exposure camera are formalized in definition 4 page 13. We wish to compute the MSE and SNR of a deconvolved crisp image with respect to the continuous observable scene u. To this aim a continuous deconvolved crisp signal u est must be defined from the observed samples obspnq observed for n P Z. Thus, we 1) prove the mathematical feasibility of the Shannon-Whittaker interpolation "obspxq " ř nPZ obspnqsincpx´nq", 2) deduce the conditions on the exposure function α for the existence of an inverse filter γ that deconvolves the observed samples, 3) define the final crisp image u est and 4) give the formulae for the MSE and SNR of u est . The study yields to theorem 3.4 page 15.
The mathematical feasibility of the Shannon-Whittaker interpolation is formalized by Proof See annex D page 20.
We now treat the step 2). We cannot recur to a Wiener filter to define γ. Indeed, due to the Poisson photon (shot) noise, the noise in of our observations obspnq defined in definition 4 page 13 is not additive. Therefore, the Wiener filter is not defined (see, e.g., [56, p. 205], [55, p. 95], [57, p. 159] see also [58, p. 252] for a definition). Instead of using a Wiener filter we shall propose a filter designed so that the restored crisp image u est is unbiased. This is also the set up considered in, e.g, [1, section 3.1, page 6]. We now provide the condition under which an inverse filter γ will yield to an unbiased the restored crisp image u est .
If the exposure function α defined in definition 3 page 12 satisfiesαpξvq " 0 for some ξ P r´π, πs the convolution in (18) is not invertible and some information is destroyed. Therefore, it is no more possible to retrieve any observed scene u. (In a discrete setting, that would mean that the Toepliz matrix associated with the convolution kernel is not invertible.) Thus, ifαpξvq ‰ 0 for some ξ P r´π, πs there exists no inverse filter γ capable of giving back an arbitrary observed scene u. Hence, we assume that the exposure function α satisfiesαpξvq ‰ 0 for any ξ P r´π, πs. Under that condition the convolution´1 |v| α`v˘¯˚u in (18) is invertible because u is r´π, πs band-limited. Therefore, we have Definition 5 (Admissible α and definition of the inverse filter γ.) Let α as in definition 3 (page 12). If α satisfiesαpξvq ‰ 0 for any ξ P r´π, πs then the inverse filter γ, that deconvolves the observed samples (and will be proved to give back a crisp image), exists and is defined by (its inverse Fourier transform (xiv)) γpxq :" F´1ˆ1 r´π,πs pξq αpξvq˙p xq.
Remark From definition 5, we deduce that R Q ξ Þ Ñγpξq is bounded and has compact support. Hence, we haveγ P L 1 pRq X L 2 pRq and therefore γ P L 2 pRq. In addition, from Riemann-Lebesgue theorem we have that γ is continuous and bounded. Consequently, γ is r´π, πs band-limited and C 8 pRq, bounded, and belongs to L 2 pRq. We now treat the step 3). The mathematical feasibility of deconvolved crisp signal u est is formalized by converges in quadratic mean. In addition, for any x P R, we have E pu est pxqq " upxq; var pu est pxqq " var pobspnqq pγpx´nqq 2 ă`8.
This proposition means that u est is an unbiased estimator of the observable scene u.
Proof See annex F page 23.
We now treat the step 4). We have SNR coded exp. pαq:" Proof See annex I page 28.
We now connect the formulae in theorem 3.4 with the existing literature on the coded exposure method. We have that the mean photon emission µ relates toī 0 in, e.g., [2, section 2]. In addition, from (25), we have that for fixed exposure function α and additive (readout) noise variance σ 2 r the SNR evolves proportionally to with the mean photon emission µ. In particular, from (25), if σ 2 r " 0 and for a fixed α we deduce that the SNR evolves proportionally to ? µ and we retrieve the fundamental theorem of photography. Note that it is equivalent to minimize (24) or to maximize (25) with respect to the exposure function α. Therefore, in the sequel we choose w.l.o.g. to use formula (24) and to evaluate the performance of the coded exposure method in terms of MSE. The calculation for the SNR can be immediately deduced. As an easy application of theorem 3.4, we have the following corollary that provides the MSE of any invertible snapshot, i.e. that satisfies |v|∆t ă 2 (see the discussion on page 13) where ∆t is the exposure time. This corollary will also be needed to compare the coded exposure method and the snapshot, in terms of MSE, in section 4. Proof The proof is immediate combining (16) page 13 and (24).
We now turn to section 4 that proposes a theoretical evaluation of the gain of the coded exposure method, with respect a snapshot.

An upper bound of performance for coded exposure cameras
This section study the gain, in terms of MSE, of the coded exposure method, with respect to a snapshot, as a function of the exposure code sampling rate. The study yields to a theoretical bound that is formalized in theorem 4.1 page 17 and corollary 4.2 page 17. The bound is valid for any exposure code provided |v|∆t ď 1. (We recall that the exposure code sampling rate ∆t is defined in definition 3 page 12.) This means that the proposed bound is an upper bound for the gain of any coded exposure camera, provided |v|∆t ď 1. We have Proof See annex J page 29.
We also have Proof See annex K page 32 Corollary 4.2 directly provides an upper bound for the gain, of the coded exposure method, in terms of MSE with respect to a snapshot. Given our hypothesis this bound is valid for any code as soon as |v|∆t ď 1. We now depict, in figure 2, the upper bound of corollary 4.2 varying the quantity |v|∆t. Note that the quantity |v|∆t is inversely proportional to the temporal frequency sampling of the exposure code. Note that the curve is an upper bound. Thus, the actual gain of the coded exposure method is below this curve.
We now illustrate numerically corollary 4.2 in figure 3 and table 1. Table 1 This table provides the evolution of the RMSE varying the intensity of the mean photon emission for a fixed additive (readout) noise variance. The additive (readout) noise is Gaussian with zero mean and variance σ 2 r " 100. A mean photon count of k means that the camera collects k photons if it integrates 1 second. The scene moves, w.l.o.g. at velocity v " 1 pixel per second. The snapshot integrates during 1 second. The exposure code used is the Agrawal, Raskar et al. code [5,6]. This code permits to collect more photons than the snapshot but the deconvolution kernel is more severe than the deconvolution kernel of the snapshot.

Limitations and Discussion
We have proposed a mathematical model of coded exposure cameras. The model includes the Poisson photon (shot) noise and any additive readout noise of finite variance. The model is based on the Shannon-Whittaker framework that assumes band limited images. This formalism has allowed us to give closed formulae for the Mean Square Error and Signal to Noise Ratio of coded exposure cameras. In addition, we have given an explicit formula that gives an absolute upper bound for the gain of any coded exposure cameras, in terms of Mean Square Error, with respect to a snapshot. The calculations take into account the whole imaging chain that includes Poisson photon (shot) noise, any additive (readout) noise of finite variance in addition to the deconvolution. Our mathematical model doesn't allow us to prove that the coded exposure method allows for very large gains compared to an optimal snapshot. This may be the result of an imperfect model of our mathematical coded exposure camera. Indeed, our model assumes that the sensor does not saturate, that the relative camera scene velocity is known, that the scene has finite energy and is observed through an optical system that provides a cut off frequency, that the additive (readout) noise has a finite variance and neglects the boundaries effects due to the deconvolution. How the results change if one has to, e.g., estimate the velocity is, to the best of our knowledge, an open question.
Acknowledgements Yohann Tendero is happy to deeply thanks "les milieux autorisés": Igor Ciril (IPSA, MITE), Jérôme Darbon (CNRS, UCLA and the "others offices"), Marc Sigelle (Télécom ParisTech), the one who knocks and the Betches family for their careful readings of the drafts. Indeed, their feedbacks have been duly noted. Research partially supported on the ONR grants N000141410683, N000141210838 and DOE grant DE-SC00183838.. Part of this work was done at the UCLA Math. Dept.
Appendix A: Validity of the one dimensional framework Without loss of generality, we express the coordinates of the two dimensional camera scene velocity vector using the sampling axis grid as Galilean reference frame. Therefore, in this reference frame, the angle θ between the two dimensional velocity vector and, e.g., the x-axis of the sampling grid is known. Consider the rotation re-sampling operator of angle´θ on L 2 pR 2 q. This rotation operator re-sample the image so that the motion is parallel to one of the grid axis, e.g., the x-axis. The motion blur is a one dimensional phenomenon. Therefore, we can apply the motion blur model to each line of the image parallel to the x-axis. Thus, from the mathematical viewpoint, the motion blur is mathematically equivalent to a one dimensional convolution of a exposure function by the one dimensional observed stochastic scene. Indeed, the whole convolution/deconvolution model is applied to each line of Figure 3 In this experiment, we assume that the scene s moves at velocity v " 1 pixel per second. The additive (readout) noise is Gaussian with a standard deviation equal to 10. We also assume that the scene emits 625 photons per seconds (for other values see table 1 pagee 17). On the top left panel: the observed image using the Agrawal, Raskar et al. code [5,6]. On the top right panel: the observed image for snapshot an exposure time of 1 second, i.e., the blur support is 1 pixel. On the bottom left panel: the reconstructed image for the Agrawal, Raskar et al. code [5,6]. On the bottom right panel: the reconstructed image for the snapshot (blur support of 1 pixel). This means that for the Agrawal, Raskar et al. code the blur has a support of 52 pixels. In other words, this code permits to increase the exposure time by a factor 52 compared to the snapshot. The RMSE using the Agrawal, Raskar et al. code is equal to 9.84. The RMSE of the snapshot is equal to 5.96. We refer to table 1 for different values of mean photon count and additive (readout) noise variance. This simulation is based on a variant of [29]. the image parallel to the x-axis. In addition, the image rotation operator is isometric in L 2 pR 2 q. Therefore, the calculations of MSE and SNR as if the images were one dimensional signals hold for two dimensional images by easy expectation and variance calculations. This means that the theorems that we shall prove in the sequel are valid for two dimensional images.
Appendix B: It is enough to study only r´π, πs band limited functions This section justifies that it is enough to consider only r´π, πs band limited functions. As we have seen page 4, it is enough to model the coded exposure method as if the images were one dimensional signals. For the sake of the clarity, consider momentarily that the deterministic function f represents a sound. (This example is illustrative of our situation because u defined in definition 1 page 9 is one dimensional.) Since we consider a sound f , we also momentarily consider that the unit of the variable x in (xiv) is the second (denoted s). From the definition off (see (xiv)) we deduce that ξ is in Hz 2π (Hz denotes Hertz). Iff satisfiesf pξq " 0 for any ξ P R such that |ξ| ą π, i.e., is r´π, πs band limited then the maximal frequency of f is π 2π in Hz. Therefore, from the Shannon-Whittaker sampling theorem, f is well sampled with one sample every second. Consider now an arbitrary function f and a positive real number c. If f is sampled with one sample every 1 c s then, from the Shannon-Whittaker sampling theorem, one can recover any r´cπ, cπs band limited functions. This means that f must not contain any frequency above πc 2π in Hz. In other words, by diminishing adequately the sampling step one can recover any band limited function. Consider another time unit s c , and a zoomed version of f defined byf p¨q :" f`c˘. As we have seen, if we have access to the samples¨¨¨, f´´1 c¯, f`0 c˘, f`1 c˘, f`2 c˘,¨¨¨o r equivalently that we have access to¨¨¨,f p´1q,f p0q,f p1q,f p2q,¨¨¨then f and thereforef are well sampled from the Shannon-Whittaker theory point of view. One the one hand, f is r´cπ, cπs band limited with an arbitrary positive constant c. This means thatf pξq " 0 as soon as |cξ| ą |cπ|. Therefore, from its definition,f is r´π, πs band limited. In other words, f can be seen as a r´π, πs band limited function provided we use an adequate time scale, i.e., use an adequate unit to measure the time. This time scale is given by the Shannon-Whittaker sampling theorem and, implicitly, implies that ξ is measured in π 2πc Hz. Therefore, if we assume that the sampling system respects the Shannon-Whittaker theorem we can without loss of generality assume that f is r´π, πs, where the frequencies of f does not exceed π 2πˆ"sampling step in s" Hz.
In conclusion, we assume that the couple optical system/sensor array are designed according to the Shannon-Whittaker sampling theory. Therefore, the adequate unit for the distances x P R is the distance between pixel centers. With this unit of distance the signal is r´π, πs band limited, where the frequencies are measured in π 2π"pixel length" Hz. Therefore, the physical frequency cut off is arbitrary. This means that the theory we shall develop covers any band limited function, and any positive fill factors. Our hypothesis is that, the couple optical system/sensor permits to obtain well sampled signal in the sense of the Shannon-Whittaker sampling theorem. Mathematically, we can, w.l.o.g. assume that the cut off frequency comes from the PSF g as in definition 1 page 9 that is g is r´π, πs band limited.

Appendix C: Measurability
We need to prove that the samples are random variables, i.e., measurable functions (see, e.g., [53, p. 168], see also [59, p. 44]). For any x P R let the random variables ηpxq be defined by ηpxq : pΩ, Tη, Pηq Ñ pR, BorpRqq where, Ω :" R is the sample space, BorpRq is the Borel sigma algebra on R, Tη is chosen as the smallest sigma-algebra on Ω that makes ηpxq measurable i.e, Tη " tηpkq´1B : B P BorpRqu. The Poisson random variables are defined on Ppλq : pN, T N , P P oissonpλq q Ñ pR, BorpRqq, where T N is the smallest sigma-algebra (in the sense of the inclusion) that contains N. Assume that at a pixel sensor centered at x P R we have ş t 2 t 1 upx´vtqdt " λ. From equation (10) (page 10) and for any x P R, obspxq is the sum of a Poisson and of an additive random variables Ppλq`ηpxq : pΩ, T , Pq Ñ pR, BorpRqq. Therefore, obs is measurable with the sigma-algebra T " tpPpλq`ηpkqq´1B : B P BorpRqu.

Appendix D: Proof of proposition 3.2 page 14
We first give a lemma and its corollary that will be useful for the proof and then give the construction details.
Lemma D.1 The deterministic function R Q x Þ Ñ´1 |v| α`v˘˚u¯pxq is r´π, πs band limited, uniformly bounded and for any x P R we havê Proof See annex E page 23.
Corollary D.2 The deterministic function Z Q n Þ Ñ var pobspnqq is uniformly bounded.
Proof The proof is immediate from proposition 3.1 and lemma D.1.
For any n P Z, consider the centered random variables The proof is in three steps. The step a) proves that, for any x P R, the series 8 ÿ n"´8õ bspnqsincpx´nq converges in quadratic mean. The step b) proves that, for any x P R, the series converges to a deterministic constant. These two steps entail that the series in (17) (page 14) converges, for any x P R, in quadratic mean. The step c) provides explicit formulae for the expectation and variance of (17). (These calculations will be needed for the computation of MSE and SNR of the deconvolved crisp image. We now turn to the proof of step a).) Step a): From (30), for any n P Z, we have Epõ bspnqq " 0; var´õ bspnq¯" Eˆˇˇõ bspnqˇˇ2˙" var pobspnqq .
Consider the finite sums of independent random variables Note that hereinafter,õ N denotes the N -th term of the sequence defined in (32). For any N ě M and any x P R, we have that Therefore, combining (31) and (33) we deduce that for any x P R. From corollary D.2 page 20 we have that sup n varpobspnqq ă`8. Hence, from (34) we have for any x P R. The series ř`8 n"´8 sinc 2 px´nq converges for any x P R. Consequently, from (35), for any x P R and ą 0 we deduce that there exists M0 P N such that if M0 ď M ď N we have Therefore, from the Cauchy criterion (see, e.g., [60, thm 6.6.2 p. 194]) we deduce that, for any x P R, the series obspxq " ř`8 n"´8õ bspnqsincpx´nq converges in quadratic mean to a limit that we can therefore callõ bspxq. This concludes step a). We now turn to step b) that proves that, for any x P R, the series ř`8 n"´8 E pobspnqq sincpx´nq converges to a deterministic constant.
Step b): From proposition 3.1 page 13, for any x P R, we have Combining (29) page 20 and (37), we have that, for any x P R, sincpx´nq. (38) We first deal with of the first term on the right hand side (RHS) of (38). From definition 1 page 9 we havẽ u P L 1 pRq andũ is r´π, πs band limited. From definition 3 page 12 α P L 1 pRq. Therefore, from Young inequality (see, e.g., [61, p. 525]) we have that´1 |v| α`v˘˚ũ¯P L 1 pRq and is r´π, πs band limited sinceũ is band limited in this range. Hence, we deduce (see, e.g., [40, p. 354]) that, for any x P R, there holds We now deal with of the second term on the RHS of (38). For any x P R, we have ř`8 n"´8 sincpx´nq " 1. Thus, combining (38) and (39) we have that, for any x P R, there holds From definition 1 page 9, for any x P R, we have that upxq "ũpxq`µ. In addition, Hence, from (40) we obtain that, for any x P R, there holds This concludes step b). We now combine the steps a) and b). Combining (17) and (30)  converges, for any x P R, in quadratic mean to a limit that we can therefore call obspxq. Indeed, from step a), it is the sum of the quadratic mean convergent series ř N n"´Nõ bspnqsincpx´nq and, from (41), of the deterministic constant´1 |v| α`v˘˚u¯pxq. Consequently, we deduce that, for any x P R, the series defined in (17) page 14 converges in quadratic mean. Thus, for any x P R, we call obspxq this limit. We now turn to step c) that gives explicit formulae for the expectation and variance of the quadratic mean convergent series obspxq defined in (17).
Step c): The convergence in quadratic mean implies the convergence of the two firsts moments (see [59, Ex 5.6 (a)-(b), p. 158]). Therefore, for any x P R we have obspnqsincpx´nq¸.
In addition, the linearity of the expectation and the independence of the observed samples obspnq imply that, for any x P R, for any x P R. Similarly, combining (43) with (45) and the definition (17) page 14 of obspxq we have that for any x P R. Combining (41) and (46) we have that, for any x P R, Epobspxqq "ˆ1 |v| αˆv˙˚u˙pxq.
Hence, we proved (18) (page 14). From proposition 3.1 page 13 and (47) we deduce that, for any x P R, Since, for any x P R, we have ř`8 n"´8 sinc 2 px´nq " 1, from (48) we deduce that, for any x P R, Hence, we proved (19) (page 14). In addition, from corollary D.2 page 20, we deduce that, for any x P R, varpobspxqq is finite. This concludes our proof.
Appendix E: Proof of lemma D.1 page 20 From its definition 1 page 9 the deterministic functionũ " u´µ is r´π, πs band limited. Therefore, we deduce that the function R Q x Þ Ñ´1 |v| α`v˘˚ũ¯pxq is r´π, πs band limited. In addition, since up¨q "ũp¨q`µ (see definition 1) we deduce that, for any x P R, (29) holds. We now prove that u is uniformly bounded. From the remark page 12 we have α P L 1 pRq. Recall that from definition 1 we haveũ P L 1 pRq. Hence, from Young inequality (see, e.g., [61, p. 525]) we deduce that´1 |v| α`v˘˚ũ¯P L 1 pRq. Therefore, by Riemann-Lebesgue theorem (see e.g., [52, prop. 2.1]) we have that R Q ξ Þ Ñ F`α`v˘˚ũ˘pξq is continuous. In addition, from definition 1 we have thatũ is r´π, πs band limited. Therefore, R Q ξ Þ Ñ F´1 |v| α`v˘˚ũ¯pξq is compactly supported. Hence, we deduce that F´1 |v| α`v˘˚ũ¯P L 1 pRq. Thus, by Riemann-Lebesgue theorem we have that 1 |v| α`v˘˚ũ is uniformly bounded. It follows from (29) that 1 |v| α`v˘˚u is uniformly bounded. This concludes our proof.

Appendix F: Proof of proposition 3.3 page 15
We first prove (21) (page 15) then (22) and (23). We need to recur, again, to a convergence argument prove the validity of (21) (page 15). Therefore, for any x P R, we set u N est pxq : " Note that hereinafter, u N est pxq denotes the N -th term of the sequence defined in (50). From definition 5 page 15, we deduce that γ satisfies Hence, from (51) γ obeys the same decay as the sinc. Thus, we deduce that the rest of the proof follows exactly the same arguments as for the construction of obspxq leading to proposition 3.2 page 14. Therefore, we obtain that, for any x P R, u N est pxq " ř N n"´N obspnqsincpx´nq converges, in quadratic mean, to a limit that we can therefore call uestpxq. This proves (21) (page 15). Quadratic mean convergence implies the convergence of the two firsts moments (see [59, Ex 5.6 (a)-(b), p. 158]). Therefore, we again obtain that, for any x P R, It remains to prove (22) and (23) (page 15). To this aim the next lemma will be useful Lemma F.1 (The inverse filter γ gives back an unbiased estimator of u.) Let α and γ be as in definition 5 (page 15). For any x P R, we havè Proof The proof is in annex G page 24.
Combining proposition 3.1 page 13 and (50), for any x P R, we have From lemma F.1 page 23 we deduce that, for any x P R, Combining (52) and (56) we retrieve (22). The independence of obspnq implies that, for any x P R, var´u N est pxq¯" ř Ń N var pobspnqq γ 2 px´nq. Hence, from (53), for any x P R, we have that var puestpxqq " ř`8 8 var pobspnqq γ 2 px´nq. We now justify that var puestpxqq is finite for any x P R. From corollary D.2 page 20, we have sup n var pobspnqq ă`8. Thus, it remains to show that ř`8 n"´8 γ 2 px´nq is finite for any x P R. From definition 5 we have that γ is r´π, πs band-limited. In addition, from (51) page 23 we have that γ 2 P L 1 pRq. Hence, we deduce that γ 2 is such that x γ 2 pξq is supported on r´2π, 2πs. Moreover, from the definition of γ 5 (page 15) we have x γ 2 p2πq " pγ˚γq p2πq " ż π π 1 r´π,πs p2π´ξq αpvξqα pvp2π´ξqq dξ.
The integrand in (57) is non zero only on the zero Lebesgue measure set tπu. Therefore, we have x γ 2 p2πq " 0. Similarly, we have x γ 2 p´2πq " 0. Thus, from the Poisson formula (see annex L page 34) we deduce that since, from remark page 15 we have that γ P L 2 pRq. This concludes our proof.
Appendix G: Proof of lemma F.1 page 23 The proof is based on the Poisson summation formula (see annex L page 34) and on the following lemma Lemma G.1 (Poisson like formula for the inverse filter γ.) Let α and γ be as in definition 5 (page 15). For any x P R, we have ř`8 n"´8 γpx´nq " Proof The proof is in annex H page 27.
We now give the calculations details. The proof is in three steps. The step a) proves that, for any x P R, The step b) proves that, for any x P R,ˆ1 |v| αˆv˙˚ũ˙˚γ˙pxq "ũpxq.
The step c) combines the steps a) and b) with lemma G.1 page 24 and (29) page 20 to deduce (54). We now turn to step a).
Step a): For any x P R, we have To justify (61), consider the intermediate function fx defined by fxpyq "ˆ1 |v| αˆv˙˚ũ˙pyqγpx´yq for any y P R.
With the help of the function fx defined in (62), the left hand side of (61) rewrites as Using the Poisson summation formula (see annex L page 34) we will prove that Provided (64) holds, the equality in (61) follows from the definition of the Fourier transformfxp0q " ş R fxpyqdy. Indeed, from the definition (62) of fx we have ş R fxpyqdy "´1 |v| α`v˘˚ũ¯pyqγpx´yqdy which coincides with the RHS of (61). Therefore, provided we prove that (64) holds the equality in (61) holds. We now turn to the validity of (64). Using the Poisson summation formula (see annex L page 34), we now justify (64). In order to apply the Poisson formula we first need to prove that fx P L 1 pRq and band limited. From Cauchy-Schwartz inequality (see, e.g., [61, p. 140]) we have In addition, from Young inequality (see, e.g., [61, p. 525]), we have Therefore, combining (65) and (66) we have Furthermore, from definition 1 page 9 we haveũ P L 1 pRq, from remark page 12 we have α P L 2 pRq and from the remark page 15 we have γ P L 2 pRq. Therefore, from (67) we deduce that fx P L 1 pRq. From its definition 1ũ is band limited. Therefore, from its definition (62), we deduce that fx is band limited. This means that we can apply the Poisson formula to fx. The Poisson summation formula (see annex L page 34) applied to fx entails Recall that we need to prove that (64) holds. Therefore, we need to prove that the term m " 0 is the only non zero term in the RHS of (68). From the definition (62) of fx we havê fxpξq " ppαp¨vqûp¨qq˚γp¨qe´i xξ qpξq, for any ξ P R. From definition 1 we have that u is r´π, πs band-limited. This implies that the function R Q ξ Þ Ñαpξvqûpξq is r´π, πs band limited. From its definition 5 γ is also r´π, πs band-limited. Therefore, from (69), we deduce thatfxpξq " 0 for all ξ P R such that |ξ| ą 2π. Thus, from (68) we deduce that Yet, to prove that (64) holds we need to show thatfxp´2πq "fxp2πq " 0. We now show in details that fxp2πq " 0. From (69),fxpξq is given by a convolution (see (ix) for the definition). Thus, the evaluation of (69) at ξ " 2π yieldŝ Recall that, from definition 1 page 9,ũ is r´π, πs band-limited. Thus, from (71) we deduce that fxp2πq " The integrand in (72) is non zero only on the zero Lebesque measure set tπu. Hence, we deduce thatfxp2πq " 0. Similarly we obtainfxp´2πq " 0. Thus, we proved that (64) holds, i.e., we can remove the question mark in (64). Consequently, we have that (61) holds. Furthermore, from the definition (ix) of the convolution we have for any x P R. Hence, combining (61) and (73), for any x P R, we obtain (59) page 24. This concludes the step a). We now turn to step b).
Step b): From the definitions (xiv) of the Fourier and the inverse Fourier transforms, for any x P R, we havêˆ1 Since F´1 |v| α`v˘¯"αpξvq, from (74) we havêˆ1 for any x P R. From the definition of the inverse Fourier transform (xiv) and (75) we deduce that, for any x P R,ˆ1 From definition 1 page 9 we have thatũ is r´π, πs band limited. Consequently, from (76) we deduce that, for any From the definition of γ (20) page 15 we have thatγpξq " 1 αpξvq for any ξ P r´π, πs. Hence, we deduce that, for any x P R,ˆ1 From definition 1 page 9 we have thatũ is r´π, πs band limited. This implies that, for any x P R, where the last equality is justified by the definition of the inverse Fourier transform (xiv). Thus, combining (78) and (79) we obtain (60) page 24. This completes the step b). We now turn to the step c) that completes our proof.
Step c): Combining (59) and (60) page 24, for any x P R, we have From lemma G.1 page 24 we deduce that, for any x P R, Therefore, combining (29) page 20, (80) and (81) we deduce that, for any x P R, From the definition of u 1 page 9 we have up¨q "ũp¨q`µ. Thus, from (82) we obtain that, for any x P R, (54) page 23 holds. This concludes our proof.
Appendix H: Proof of lemma G.1 page 24 Consider the 2π periodic function g : R Ñ C defined as gpξq :"γpξq for any ξ P r´π, πq (83) and the Dirichlet kernel Dnpξq " 1 2π ř n k"´n e ikξ . From its definition 3 page 12 we have α P L 1 pRq. Therefore, by Riemann-Lebesque (see e.g., [52, prop. 2.1]) we have thatα is continuous. From definition 5 (page 15) and (83), we have gpξq " γpξq " 1 αpξvq for any ξ P r´π, πs. Sinceαpξvq ‰ 0 for any ξ P r´π, πs, we obtain that g is continuous on r´π, πs and that g P L 1 p´π, πq. Since g P L 1 p´π, πq it has a Fourier series decomposition (see (xviii)). From the Fourier series decomposition we have where, for any k P Z, We shall now pass to the limit n Ñ`8 in (84). The Fourier series theory entails that gp er Dnpξq :" 1 2π ş π π gpyqDnpξ´yqdy nÑ`8 Ý ÝÝÝÝ Ñ gpξq for any ξ P R where g is continuous at ξ. Therefore, we obtain that for any ξ P R where g is continuous at ξ. We have that g is continuous at ξ " 0. Therefore, from (86) we obtain that Hence, combining (83), (85) and (87) we deduce that gp0q "γp0q "`8 Since, from its definition 5 (page 15), γ is r´π, πs band limited we obtain that In addition, from (51) (page 23) we have γ P L 2 pRq. From (xiv), for any k P Z, we have Hence, combining (88) and (89) we deduce that This applies for any shift x P R of γ. Hence, from (90) we deduce that for any x P R. In addition, from definition 5, we havê (92) Thus, combining (92) and (91) we deduce that ř`8 k"´8 γpx´kq " 1 ş R αptqdt , for any x P R. This concludes our proof.
Appendix I: Proof of theorem 3.4 page 15 In order to provide closed formulae for the MSE and SNR we first evaluate varpuestpxqq and then the limit of For any n P Z, we have Therefore, from (103) we deduce that From definition 1 page 9 we haveũ P L 1 pRq andũ is r´π, πs band limited. From definition 3 page 12 α P L 1 pRq. Therefore, from the Poisson summation formula (see annex L page 34) we obtain that where dλ denotes the Lebesgue measure on pR, BpRqq. Let S :" α of the form (12), α P L 1 pRq, 0 ď α ď 1 ( . From definition 3 page 12, we deduce that optimizing the coded exposure method boils down to finding α P S that minimizes (110). We wish to prove that (27) page 17 holds true. Our proof is in 4 steps.
Step a) proves that, for any α P S, (110) is bounded from below, namely that there holds inf αPS pMSE coded exposure pαqq ě inf pα k qP 2 pZq,α k Pr´1,1sż Step b) proves that as soon as |v|∆t ď 1 the RHS of (111) can be rewritten in terms of Fourier series and that there holds Step c) proves that the RHS of (112) is attained by constructing an adequate f that allows to show that inf αPS pMSE coded exposure pαqq ě Step d) proves that (27) holds by giving a closed form of the RHS of (113), and giving a formula for the constant C. We now turn to step a).
Step c): On the one hand, α k " c k pf q P R for any k P Z and the Hermitian symmetry implies that |f p´ξq| " |f pξq|. On the other hand, the function p0,`8q Q . (121) The strict convexity of p0,`8q Q x Þ Ñ 1 x implies that the equality case in (121) is realized when ξ Þ Ñ sinc 2´ξ 2π¯| f pξq| 2 is constant say C ą 0 on the interval r´π|v|∆t, π|v|∆ts. We choose to set f pξq :" C sinc´ξ 2π¯f or any ξ P r´π|v|∆t, π|v|∆ts. It remains to extend this f on r´π, πszr´π|v|∆t, π|v|∆ts, and discuss the value of C, to give a complete definition of an f P L 2 pRq that provides, through its Fourier series, a lower bound for the MSE of any coded exposure method. From the term }f } 2 L 2 p´π,πq in (121) it is easy to see that we need to extend f by zero on r´π, πszr´π|v|∆t, π|v|∆ts. This yield to the definition of f P L 2 p´π, πq of the form f pξq :" C sinc´ξ 2π¯1 r´π|v|∆t,π|v|∆ts pξq, where C is a positive constant. Being real and even, this f therefore entails that the a k " c k pf q are real. The f given in (122) provide the equality case in (121). Thus, combining (120) and the equality case in where C is a positive constant. It remains to set the value of C in (122) so that f P U , i.e., to ensure that |α k | ď 1 (recall definition 3 page 12). From (123) it is easy to see that we would like C as large as possible, while |c k pf q| ď 1 so that f P U . From their definition (xviii) we have that |c k pf q| ď 1 2π ş π|v|∆t π|v|∆t C sinc´ξ 2π¯q dξ.
Therefore, we deduce that we need C ď in the definition of f is enough to provide a lower bound to the MSE of coded exposure cameras. That is this choice for f realizes in infinimum of the RHS of (120). In other words, from (123) we deduce that, as soon as |v|∆t ď 1, This concludes step c). We now turn to step d).
Step d): It remains to give a closed form for (126). On the one hand, we have In order to evaluate the maximal theoretical "gain" of coded exposure camera we need to compare the bound in terms of MSE given in theorem 4.1 namely (27) page 17 with the MSE of a snapshot. Therefore, the proof is in two steps. The first step prove an upper bound for the MSE of the optimal snapshot.
Step b) calculates the ratio of these MSE and prove (28) (page 17). We now turn to step a).
Step a): From corollary 3.5 (page 16) and the definition of the sinc function (xvi) we have Indeed, the optimal snapshot, if it exists has, by definition, a MSE lower or equal to the one in equation (133). In other words, (133) provides an upper bound for the MSE of the optimal snapshot. We now compare the MSE of the coded exposure method given by theorem 4.1 namely equation (27)  Hence, we deduce that Equation (138) is valid for any α that satisfies definition 3 page 12 and |v|∆t ď 1. In other words, (138) is valid for any coded exposure method that satisfies |v|∆t ď 1. Hence (28) is proven (page 17). This concludes our proof.