Analysis of a variational model for motion compensated inpainting

We study a variational problem for simultaneous video inpainting and motion estimation. We consider a functional proposed by Lauze and Nielsen [25] and we study, by means of the relaxation method of the Calculus of Variations, a slightly modified version of this functional. The domain of the relaxed functional is constituted of functions of bounded variation and we compute a representation formula of the relaxed functional. The representation formula shows the role of discontinuities of the various functions involved in the variational model. The present study clarifies the variational properties of the functional proposed in [25] for motion compensated video inpainting.


Introduction
We consider a variational model proposed by Lauze and Nielsen [25] for motion compensated video inpainting.The problem of video inpainting can be formulated as follows.Let Ω be a spatiotemporal domain, D a subset of Ω, and f a real function representing a gray-value video content which is known over Ω \ D. Hence, the subset D is a spatiotemporal region where the video data is lost.The problem consists in looking for a video u which is defined on the whole of Ω, matches f outside the region D, and has a content inside D which satisfies a regularization constraint which, typically, consists in spatial piecewise smoothness.In motion compensated video inpainting the content of the reconstructed video inside D is also required to be coherent with the apparent motion of the video in Ω\D.Apparent motion means a motion that can be estimated only through gray-value variations of the function f in Ω \ D and it is represented by a vector field σ denoted optical flow (all these notions will be carefully defined in the next section where the mathematical model will be presented).Motion compensated inpainting requires the joint estimate inside D of both the gray-value video u and the optical flow field σ associated to u.The variational model for motion compensated video inpainting, proposed by Lauze and Nielsen in [25], has been applied more recently by the same authors also to the problems of video deinterlacing [23] and video super-resolution [24].See also [4,5] for other approaches to the problem of motion estimation, and [9,14] and Chapter 5.3 of [6] to the problem of simultaneous video and motion estimation.
The variational model by Lauze and Nielsen requires the minimization of an integral functional of the Calculus of Variations whose integrand has linear growth with respect to partial derivatives of first and second order of the unknown functions.The linear growth makes it possible to reconstruct a video content which is discontinuous along the boundaries of moving objects in the video, which is an important requirement.Though a functional of this type is well defined on a Sobolev space of functions with distributional partial derivatives belonging to L 1 (Ω), nevertheless such a function space is not a reflexive Banach space.Hence, in order to achieve information about minimizing sequences of the functional we have to resort to the relaxation method of the Calculus of Variations and to compute the corresponding relaxed functional.
The main purpose of the present paper, after a suitable modification of the variational model by Lauze and Nielsen in order to achieve a functional with better variational properties, is the study of the corresponding relaxed functional.Such a study gives information about numerical algorithms designed for the original functional: a minimizing sequence of the (modified) Lauze and Nielsen's functional converges, up to a subsequence, to a minimizer of the relaxed functional in a suitable topology.Minimizers of the relaxed functional are vector valued functions of bounded variation, so that a video content which is discontinuous along the boundaries of moving objects can be reconstructed.Moreover, we find a representation formula of the relaxed functional which shows explicitly the role of discontinuities of the various functions involved in the variational model.The paper is organized as follows.In Section 2 we introduce the Lauze and Nielsen's functional and we propose a modification of the functional in order to obtain a model with improved variational properties.Then we define the relaxed functional.In Section 3 we specify notations and we give all the required mathematical preliminaries.In Section 4 we find the domain of the relaxed functional, i.e., the space of functions which make the relaxed functional finite.It turns out that the domain is constituted of functions of bounded variation.In Section 5 we find a representation formula of the relaxed functional: such a representation formula shows the role of discontinuities of the various functions in the variational model (see Remark 5.2, Remark 5.4 and Remark 5.6).Some technical details are given in the Appendix.Eventually, in Section 7 the existence of minimizers of the relaxed functional and the convergence property of minimizing sequences of the original functional are deduced.

The variational model
In this section we introduce the variational model proposed by Lauze and Nielsen [25] for motion compensated video inpainting, then we discuss a modified version of their energy functional which permits us to address the issue of existence of minimizers.In the following, the gray-value video content (called simply video in the sequel) will be represented by a real, summable function defined on a three-dimensional spatiotemporal domain.We denote by Ω s ⊂ R 2 the spatial domain, assumed bounded, open, connected and with Lipschitz boundary, and we denote by [0, T ] the time domain.Then we set Ω = Ω s × [0, T ], and we denote by (x, t) a point belonging to Ω.We denote by D ⊂ Ω the missing data locus and we assume that D is a known, open set.The degraded video is a function f ∈ L 1 (Ω \ D).For almost any t ∈ [0, T ] the function f (x, t) is defined for a.e.x ∈ Ω s \ D, and we say that such a function is a frame of the video f at time t.Then the set Ω s × {t} ∩ D represents the missing data locus in the frame of the video at time t (the hole to inpaint).We assume that both the set D and the set Ω \ D have Lipschitz boundary and that the set ∂D ∩ Ω has positive surface measure.The aim of video inpainting is to reconstruct a suitable gray-value video content on D from the datum f .We denote by u : Ω → R the restored video, and we show in the sequel that the function u has to be looked for in the class of functions of bounded variation.For almost any t ∈ [0, T ] the function u(x, t) is defined for a.e.x ∈ Ω s , and we say that such a function is a frame of the restored video u at time t.A simple approach to video inpainting tries to recover each damaged frame independently, as proposed for instance by Bertalmio et al. [7].However, such a method often yields artifacts, such as flicker.Conversely, in motion compensated video inpainting, frames of the restored video u are reconstructed from the degraded video f exploiting the property of f of being defined on a three-dimensional (spatiotemporal) domain.Particularly, coherence corresponding to the apparent motion of the video is enforced between reconstructed frames u(x, t) at different times t.For this purpose a vector field σ : Ω → R 2 , called optical flow, is estimated simultaneously with the recovered video u.The vector field σ is an approximation to the so-called 2-D motion field, which is the projection onto the image plane of the three-dimensional vector field of velocities of objects moving in the scene.The optical flow is estimated from time variation of gray-value intensity of the video f , and in this sense it corresponds to an apparent motion (see [6], Section 5.3.2).In the next subsection we discuss how the optical flow is applied in the variational model proposed by Lauze and Nielsen [25].

The Lauze and Nielsen's original model
The starting point is the assumption that the gray-value intensity of the video f is approximately constant along apparent motion trajectories, at least for a short duration.Hence, if t → x(t) denotes the trajectory in the image plane of a point belonging to the visible surface of an object moving in the scene, the constancy df /dt of the video intensity formally implies (the definition of the required derivatives in a weak sense will be given later) where σ = dx/dt gives the apparent motion, ∇ x is the spatial gradient operator, and •, • denotes the Euclidean scalar product.Equation ( 1) is called the optical flow constraint (see [22] for more details).
Based on a work by Uras et al. [27], Brox et al. [11] argue that estimation of apparent motion can be improved by requiring that also the spatial gradient of intensity be approximately constant under motion.In this case the constancy d∇ x f /dt of the intensity gradient formally implies where H x denotes the Hessian with respect to the spatial coordinates.
We are now sufficiently equipped to introduce the Lauze and Nielsen's functional [25] (we notice that in [25] the roles of the sets Ω and D are interchanged: here we prefer to denote by Ω the image domain, according to the standard notation of the calculus of variations).We denote by Σ the vector field (σ 1 , σ 2 , 1), so that the constraints (1) and ( 2) can be rewritten as ∇f, Σ = 0, (∇∇ where ∇ now denotes the spatiotemporal gradient, and ∇∇ x is a 2 × 3 matrix of mixed space-time second derivatives.Next, for t ≥ 0, we set ϕ(t) = √ t + ε, where ε is a small positive parameter.Then, the Lauze and Nielsen's functional [25] is defined by and the function u satisfies u(x, t) = f (x, t) for a.e (x, t) ∈ Ω \ D. We denote by L 3 the three-dimensional (spatiotemporal) Lebesgue measure, and we denote by | • | the Euclidean norm of both vectors and matrices.We have set equal to one all the weights that, in the functional given in [25], multiply the integrals (they are not essential in the present paper since we are interested in an existence result).The functional E LZ is well defined for u, σ belonging to the following Sobolev spaces: The meaning of the terms in E LZ is the following: (i) the first integral is a spatial regularization term which, penalizing the total variation of ∇ x u, enforces the piecewise smoothness of the frames of the reconstructed video u; (ii) the second integral penalizes global deviations from the gray-value constancy assumption and the spatial gradient constancy assumption (see (3)), thus enforcing coherence corresponding to apparent motion between frames of the reconstructed video u; (iii) the third integral is a spatiotemporal regularization term which, penalizing the total variation of ∇σ, enforces the piecewise smoothness of the optical flow σ.
In the following subsection we argue that, in order to look for the existence of minimizers in a suitable function space, it is favorable to modify the energy functional.

The modified energy functional
First we observe that it is not natural to assume the existence of derivatives (even in a weak sense) of the datum f , since it can be affected by noise which can be modeled in a deterministic setting for instance by a L ∞ function, but without derivatives.Hence we drop the condition u(x, t) = f (x, t) for a.e (x, t) ∈ Ω \ D and we add to the functional a term which penalizes a global deviation from such a condition: The function space W 2,1 (Ω) is not a reflexive Banach space, nevertheless, even enlarging W 2,1 (Ω) to the generalized space BH(Ω) of functions with bounded hessian [16], i.e., the functions u ∈ W 1,1 (Ω) whose second order distributional derivatives are bounded measures on Ω, the presence of the term |(∇∇ x u)Σ| 2 in the argument of the function ϕ makes the functional E LZ not coercive on BH(Ω).However, even adding to E LZ a term which makes the functional coercive on BH(Ω), that would not yield a satisfactory variational model.Indeed, because of the compact injection BH(Ω) → W 1,1 (Ω) [16], BH functions are not appropriate to represent a video content, which requires a function having discontinuities along the boundaries of objects in a video scene.For this purpose we need a functional having minimizers in the class of functions of bounded variation.
Then we propose an approach, inspired by the work by Bredies et al. [10], where the authors introduce a regularization functional which enforces piecewise smoothness both of the function u being reconstructed and of an approximation of its gradient.For this purpose we introduce a vector field v : Ω → R 2 which approximates the spatial gradient ∇ x u.We set 5 .For a given ρ > 0 we define where B ρ (y) = {z ∈ R 3 : |z − y| < ρ}.Then we set Σ ρ = (σ 1ρ , σ 2ρ , 1) and we assume that ρ is a fixed small parameter.We define, for any (u, v) ∈ [W 1,1 (Ω)] 3 , the functional and for any w ∈ V (Ω), the functional Eventually, for any w ∈ V (Ω) we define the overall functional where c is a small positive constant.In the sequel we also extend the choice of the function ϕ (see Section 3.2).The meaning of the terms in the functional E is the following: (i) the first integral in F (u, v) is a fidelity term which penalizes the discrepancy between the reconstructed video u and the datum f in the set Ω \ D where data are available; (ii) the second and third integrals in F (u, v) are spatial regularization terms which enforce the vector field v to approximate the spatial gradient ∇ x u, and enforce the piecewise smoothness both of the frames and of the frame gradients of the reconstructed video u; (iii) the integral in G(w) penalizes global deviations from the gray-value constancy assumption and the spatial gradient constancy assumption, thus enforcing coherence corresponding to apparent motion between frames of the reconstructed video u; (iv) the last two integrals in E(w) are, respectively, a spatiotemporal regularization term which enforces the piecewise smoothness of the optical flow σ, and a term which helps to make the functional coercive.
Eventually, the replacement of the vector field σ with the average σ ρ on a ball with fixed radius has been introduced in order to achieve the following properties: (i) uniform convergence of the sequence {σ h ρ } h when the sequence {σ h } h converges in L 1 ; (ii) as a consequence of (i), an explicit formula for the contribution of the term G to the relaxed energy density along discontinuity sets of functions u, v (essentially, the boundaries of objects in video scenes).These properties will be discussed in sections 5.2 and 6, as well as the complications that arise if the average σ ρ is not taken.

The relaxed functional
The function space W 1,1 (Ω) is not a reflexive Banach space, hence, in order to achieve information about minimizing sequences that are bounded in V (Ω) we have to resort to the relaxed functional of E [13].We set X(Ω) = [L 1 (Ω)] 5 , w ∈ X(Ω), and we extend the functional E to X(Ω) by means of the functional E : X(Ω) → [0, +∞] defined by We denote by E the relaxed functional of E, i.e., the lower semicontinuous envelope of E with respect to the strong topology of X(Ω).For every w ∈ X(Ω) we have The following properties hold ([13], Proposition 1.3.1 (iii)): E is lower semicontinuous with respect to the strong topology of X(Ω); if E has a minimizer in X(Ω), then inf In the following sections we characterize the subset Y (Ω) ⊂ X(Ω) where the relaxed functional E is finite.Then we find a representation formula of E(w) for any w ∈ Y (Ω).Eventually, we prove the existence of a minimizer of E in Y (Ω).We obtain that a minimizing sequence of E converges, up to a subsequence, to a minimizer of E in the strong topology of X(Ω).We denote by BV (Ω) the space of functions of bounded variation in Ω, i.e., the functions u ∈ L 1 (Ω) such that the distributional (spatiotemporal) gradient of u is representable as a measure Du : B(Ω) → R 3 with finite total variation.

Mathematical preliminaries
For any u ∈ BV (Ω), we denote by u − (x, t), u + (x, t) the approximate lower and upper limit of u at the point (x, t), which satisfy u − (x, t) ≤ u + (x, t).We set The set S u will be considered as the discontinuity set of u, and for H 2 -a.e.(x, t) ∈ S u a normal unit vector ν u (x, t) can be defined.If u ∈ BV (Ω), then the Lebesgue decomposition of Du is given by where D a u is the absolutely continuous part of Du with respect to L 3 , with density denoted by ∇u = dD a u/dL 3 , the measure D s u is the singular part of Du, the measure Ju is the jump part of D s u: with B ∈ B(Ω), and Cu is the Cantor part of D s u.The density ∇u coincides a.e. with the gradient of u defined in the sense of approximate limits [2].We denote by D x u : B(Ω) → R 2 the measure whose components are the distributional derivatives of u with respect to the spatial coordinates.We denote by ∇ x u the density of the absolutely continuous part of D x u with respect to L 3 , we denote by D s x u the singular part of D x u, we denote by ν u,x the orthogonal projection of ν u on the spatial subset of space-time R 3 , and we denote by J x u, C x u the jump part and the Cantor part of D s x u, respectively.We use analogous notations for the distributional derivative of u with respect to time: 2 , we denote by Dv the 2 × 3 matrix valued measure whose rows are Dv 1 , Dv 2 .For any u ∈ BV (Ω) we denote by u + ∂D , u − ∂D the inner and outer trace, respectively, of u on ∂D.The following Poincaré-Friedrichs inequality in BV will be useful (see [12], Lemma 6, for a proof): Lemma 3.1.Let u ∈ BV (D) and let Γ ⊂ ∂D be such that H 2 (Γ) > 0. Then there exists a constant C F such that for any u ∈ BV (D) the following inequality holds: We say that a sequence of functions {u h } ⊂ BV (Ω) converges to u ∈ BV (Ω) with respect to the weak- * -topology of BV (Ω), and we write where a sequence of finite Radon measures {µ h } is said to converge weakly to a finite Radon measure µ if [2], for every function ω ∈ C 0 (Ω), we have Ω ωdµ h → Ω ωdµ.

Slicing properties
For any subset Ω ⊂ R 3 and i = 1, 2, 3, we denote by Ω i the orthogonal projection of Ω on a plane perpendicular to e i , where {e 1 , e 2 , e 3 } denotes the canonical basis of R 3 .For any z ∈ Ω i we define the slice Ω i z by means of For any u ∈ BV (Ω) we define for H 2 -a.e.z ∈ Ω i , The following slicing properties hold [1]: for H 2 -a.e.z ∈ Ω i (i=1,2,3), u z ∈ BV (Ω i z ) and

Functions of measures
Let k ∈ N and ψ : R k → R be a convex function satisfying the following growth conditions: Then for any ξ ∈ R k there exists the limit [21] ψ which is said the recession function of ψ.
Given a measure µ : B(Ω) → R k , we consider its Lebesgue decomposition µ = µ a • L n + µ s , where µ a is the density of µ with respect to L n and µ s is the singular part of µ with respect to L n .According to [21] the following function of measure can be defined: and the measure ψ ∞ (µ s ) is absolutely continuous with respect to |µ s | .Hence for every set B ∈ B(Ω) we have For the properties of functions of measures we refer to [21].
In the sequel we extend the choice of the function ϕ : R + → R + by requiring that ϕ is nondecreasing and, for every k ∈ N, the function ψ : R k → R defined by means of ψ(ξ) = ϕ(|ξ| 2 ) be convex, Lipschitz, and satisfy the growth conditions (9).Such properties are satisfied for instance if ϕ(t) = √ t + ε is the function used by Lauze and Nielsen.Another example of ϕ useful in the applications is the following: 4 The domain of the relaxed functional E In this section we characterize the subset Y (Ω) ⊂ X(Ω) where the relaxed functional E is finite.We find that functions (u, v 1 , v 2 , σ 1 , σ 2 ) belonging to the subset Y (Ω) are all of bounded variation in Ω.We set First we motivate the strategy used in the proof.We need to show that a sequence {w h } = {(u h , v h , σ h )}, on which the functional E(w h ) is uniformly bounded, is also uniformly bounded in V (Ω).First we show that the sequence {v h } is uniformly bounded in W 1,1 , then we prove an analogous result for the sequence {u h } (the result for the sequence {σ h } is easy).The property that the relaxed functional E is finite Y (Ω) then follows immediately by an application of the BV compactness theorem [20].
The estimate of ∇v h in L 1 is obtained from the integrals involving the derivatives of v h using the growth properties of function ϕ, and the property that the average on a ball with radius ρ converts a sequence {σ h } uniformly bounded in L 1 into a sequence {σ h ρ } uniformly bounded in L ∞ .In order to prove that the sequence {v h } is also bounded in L 1 , first we prove that the sequence {v h i }, of mean values of v h i on Ω, is bounded for i = 1, 2, then we get the desired result by means of the Poincaré-Wirtinger inequality.The boundedness of {v h i } is achieved by resorting to a one-dimensional problem by applying a slicing argument to the functional 3 (after estimating function ϕ from below thanks to its growth properties).The estimate of ∇u h in L 1 (Ω) is obtained analogously.Then, by means of an estimate based on the fidelity term we find that {u h } is also uniformly bounded in L 1 (Ω \ D).The estimate of {u h } in W 1,1 is then completed by resorting first to the trace theorem on ∂D ∩ Ω, then to the Poincaré-Friedrichs inequality (Lemma 3.1) applied on the set D.
Proof.There exists a sequence In the following M will denote a positive constant, independent of h, which will vary from estimate to estimate.We divide the proof into three steps.
Step 1.We prove that the sequence Using (5) we have from which, using (11) and by definition of the function ϕ (see Section 3.2), we have Analogously, since E(w h ) ≥ G(w h ) and ϕ is nondecreasing on R + , we have Moreover we have from which it follows that {σ h } is uniformly bounded in [L 1 (Ω)] 2 , so that, using (4), {σ h ρ } is uniformly bounded in [L ∞ (Ω)] 2 .Then, using ( 12) and ( 13), we have that where C and K are positive constants independent of h and v h i is the mean value of v h i on Ω.Let now span({e 1 , e 2 }) be the spatial subset of space-time R 3 .We have for i = 1, 2: Then, using ( 5) and (11), by definition of the function ϕ, we have from which, using Poincaré-Wirtinger inequality ( 14) and (15), we obtain for i = 1, 2 and h large enough: We now prove, by using a slicing argument, that the sequence {v h i } is bounded for i = 1, 2. Using Fubini's theorem and the slicing properties (8), we get where z is open, and by Fatou's lemma, Then, for H 2 -a.e.z ∈ Ω i ∩ {z : , and, up to the extraction of a subsequence depending on z, we have for any h and with l z > 0 independent of h.Assume now that for some i ∈ {1, 2} the sequence {v h i } is not bounded.Then, up to a subsequence (not relabeled for simplicity), where A is a connected component of I i z , we have for any s ∈ A, Inequality (17) implies that {u h z } is uniformly bounded in L 1 (I i z ).Then, by Fatou's Lemma, for a.e.s ∈ I i z we have lim inf h→+∞ |u h z (s)| < +∞.Then we can choose s ∈ A in such a way that, up to a subsequence, |u h z ( s )| is bounded with respect to h.Hence by ( 17) and ( 18) we get that for a.e.s ∈ A there holds lim h→+∞ |u h z (s)| = +∞ and this contradicts (17).We conclude that the sequence {v h i } is bounded for i = 1, 2.
Step 2. We prove that the sequence {u h } is uniformly bounded in W 1,1 (Ω).
Using the result of Step 1 and the estimate (16) we have that ∇ x u h is uniformly bounded in L 1 (Ω; R 2 ).Moreover, using ( 5) and (11), by definition of the function ϕ, we have , this last estimate implies that also ∂ t u h is uniformly bounded in L 1 (Ω).It then follows that ∇u h is uniformly bounded in L 1 (Ω; R 3 ).Estimate (16) then implies that {u h } is uniformly bounded in L 1 (Ω \ D), so that {u h } is also uniformly bounded in W 1,1 (Ω\D).By our assumptions the set Ω\D has Lipschitz boundary, hence, using the trace theorem for W 1,1 functions, we obtain where L is a constant independent of h.Since H 2 (∂D ∩ Ω) > 0, using Lemma 3.1 in the particular case of W 1,1 functions, we get from which it follows that the sequence {u h } is uniformly bounded in L 1 (D), so that {u h } is uniformly bounded also in W 1,1 (Ω).
Step 3. We conclude the proof of the theorem.Now, using ( 5) and (11), by definition of the function ϕ, we have from which it follows that the sequence Using the results of Step 1 and Step 2, we find that the sequence {w h } is uniformly bounded in V (Ω), so that {w h } is uniformly bounded also in Y (Ω).Then, using the BV compactness theorem [20], the sequence {w h } admits a subsequence converging in X(Ω) = [L 1 (Ω)]5 to a vector valued function belonging to Y (Ω).Since w h → w in X(Ω), it follows that w = (u, v, σ) ∈ Y (Ω) and the proof of the theorem is concluded.

Relaxation of functional F
In this section we find a representation formula for the relaxed functional of F .We express the functional F by means of functions of measures and we resort to properties of functions of measures proved by Goffman and Serrin [21].First we estimate the relaxed functional from below by using the lower semicontinuity of functions of measures ( [21], Theorem 3).Then we estimate the relaxed functional from above by using an approximation property of functions of measures ([21], Theorem 4), which is based on convolutions with mollifiers.The use of convolutions permits us to obtain a local representation formula for the relaxed functional of F on open subsets A ⊂⊂ Ω.Eventually, we extend the representation formula to the whole of Ω by adapting to our problem the techniques developed by Corbo Esposito and De Arcangelis in [15].
In the following we need the notion of strongly star-shaped domain [15].We say that the spatial domain Ω s is strongly star-shaped if Ω s is star-shaped with respect to some point x 0 ∈ Ω s and, for every x ∈ Ω s , the half open line segment joining x 0 to x, and not containing x, is contained in Ω s .
It will be useful to define the following relaxed functional localized on subsets of Ω.Let First we localize the functional F on the set A: so that we have F (u, v) = F (u, v, Ω).Then we extend the functional F to Y 3 (A) by means of the functional F : Y 3 (A) → [0, +∞] defined by We denote by F the relaxed functional of F with respect to the componentwise weak- *topology: for every (u, v) ∈ Y 3 (A) we have and we set F(u, v) = F(u, v, Ω).We prove the following representation result.
where the constant C ϕ is given by the recession function of ϕ(t 2 ) evaluated at 1: Proof.The representation formula of F(u, v) is an application of the results in [21].We divide the proof into three steps.
Step 1. Estimate of F(u, v) from below.
Let A ⊂⊂ Ω be an open set.We have (u h , v h ) ∈ V 3 (A) for h large enough and, by Theorem By the properties of function ϕ, the function ϕ(t 2 ) is a Lipschitz function of the variable t.
Then we have where L is the Lipschitz constant.Then we obtain Moreover, by the properties of mollifiers, we have for y = (x, t) ∈ Ω h : and by definition of distributional gradient, Then, by using the approximation property of functions of measures ( [21], Theorem 4), we have Analogously, we find Collecting equalities (24)(25)(26), and taking into account that A ⊂ Ω h for large enough h, we get for any A ⊂⊂ Ω, from which it follows Step 3. Estimate of F(u, v, Ω) from above for a strongly star-shaped spatial domain Ω s .We assume first that the spatial domain Ω s is strongly star-shaped with respect to x 0 .In the following we adapt to our problem the argument used in Lemma 4.1 of [15] in order to obtain Since, by definition of F it immediately follows that sup{F(u, v, A) : A ⊂⊂ Ω} ≤ F(u, v, Ω), then we only need to prove that sup{F(u, v, A) : We set y = (x, t) ∈ R 3 and y 0 = (x 0 , T /2).Let r ∈ (0, 1) and let Ω r denote the set Let now p ∈ (0, r) and let Ω r,p denote the set Ω r,p = (1/p)r(Ω − y 0 ) + y 0 .Since Ω = Ω s × [0, T ] and Ω s is strongly star-shaped, then Ω ⊂⊂ Ω r,p .Then, for ξ ∈ R 2 and τ ∈ R, we set z = (ξ, τ ) ∈ R 3 , and we define the map y(z) = y 0 +p(z−y 0 ) and the functions One can check that (u h p , v h p ) BV −w * (u p , v p ) in Y 3 (Ω) as h → +∞.Then, using ( 29) and ( 23), we get: where Arguing as in the proof of inequality (23), and denoting by L the Lipschitz constant of ψ 1 , we have and, by definition of the function v p , we have lim p→1 g(p) = 0.
Taking the supremum with respect to sets A ⊂⊂ Ω, letting r → 1 − in the right-hand side of (30), and using (31), we have from which, using the continuity of Ω\D ϕ(|f − u| 2 )dL 3 with respect to the strong L 1 (Ω) topology, which is proved in ( 23) by replacing Ω h with Ω, it follows sup{F(u, v, A) : Since F(u p , v p , Ω) is lower semicontinuous, by letting p → 1 − we obtain inequality (28).
Using inequality (28) for a strongly star-shaped spatial domain Ω s , and adapting to our problem the method developed in [15], we obtain inequality (28) for a Lipschitz spatial domain Ω s .The proof is sketched in the Appendix.
The jump part of the total variation terms in the representation formula (20) characterizes the role of discontinuities of functions u and (v 1 , v 2 ) in the term F. We have We set for H 2 -a.e.(x, t) ∈ S v let ν v (x, t) denote the normal unit vector, and let ν v,x denote the corresponding orthogonal projection of ν v on the spatial subset of space-time R 3 .Then the jump part J x v of the measure D s x v and its total variation can be written as By using (32) and taking the Euclidean norm of the 2 × 2 matrix inside the above integral we obtain the following structure formulae.

Relaxation of functional G for a fixed vector field σ
In this section we find a representation formula for the relaxed functional of G when the vector field σ is taken fixed.Here we make use of the spatial average of the optical flow on a ball.Particularly, we use the property that the average on a ball with radius ρ converts a sequence {σ h } converging strongly to σ in L 1 into a sequence {σ h ρ } uniformly converging to σ ρ .This property, together with an uniform bound of ∇u h and ∇v h in L 1 , permits us to reduce the computation of the relaxed functional of G to the case of a fixed vector field σ via the equality (see next subsection for a precise formulation): In order to compute the relaxed functional we follow a different method, with respect to functional F , by expressing the functional G in such a way to resort directly to the relaxation result proved by Giaquinta, Modica and Soucek in [18], and based on theorems by Reshetnyak [26].The reason is the following.The average on a ball with radius ρ converts a L 1 vector field σ into a continuous and bounded field σ ρ .It then turns out that the functional G, considered for a fixed σ ρ as a functional of u, v, satisfies the assumptions of the relaxation theorem in [18].It follows that the proof given for functional G is more straightforward in comparison to the case of functional F .Eventually, the average of the optical flow σ on a ball, together with the application of the result in [18], permits us to obtain a representation formula for the relaxed functional of G such that the density of the jump part of the energy can be explicitly computed (see Remark 5.4).The issues that arise without taking such an average are discussed in Section 6.We extend the functional G, for a fixed function σ, from V 3 (Ω) to Y 3 (Ω) by means of relaxation with respect to the componentwise weak- * -topology.We localize the functional G on the open set A ⊂ Ω; for any w so that we have G(w) = G(w, Ω).We extend the functional G by means of the functional For every and we set G(w) = G(w, Ω).The localized versions of functionals G, G and G will be used later in the proof of Theorem 5.5 about the representation result for E.
Proof.The representation formula of G is an application of the results in [18,26].Let ξ i ∈ R 3 for i = 1, 2, 3, and let ξ ∈ R 3•3 be the vector with components ξ = (ξ 1 , ξ 2 , ξ 3 ).Let Ψ : Ω × R 3•3 → R be the function defined by Using (4) the function Σ ρ is continuous and bounded: |Σ ρ (y)| ≤ C for any y ∈ Ω, with C = C(σ, ρ, Ω) positive constant.Then, by definition of the function ϕ and using the growth conditions (9), Ψ(y, ξ) is a continuous and convex in ξ function with linear growth, i.e., For (u, v) ∈ Y 3 (Ω) we set U = (u, v 1 , v 2 ).We shall treat the matrix valued measure DU as a vector valued measure DU : B(Ω) → R 3•3 with components (Du, Dv 1 , Dv 2 ).Analogously, we denote by ∇U the vector in Following [18], we define for every (y, ξ, ξ Let now µ : Both DU and L 3 are absolutely continuous with respect to |µ|.Then, by the properties of the function Ψ and according to the result in [18], the relaxed functional G is represented by The statement of the lemma now follows from the evaluation of this integral.The measure |µ| has the Lebesgue decomposition where µ s = (D s u, D s v 1 , D s v 2 , 0).Using Proposition 3.92 of [2], we have Moreover we have Since, for any y ∈ Ω, lim then, taking into account the above properties of measures µ and DU , integral (39) takes the form where Moreover, since we have , and the following relations between Radon-Nikodym derivatives hold: Eventually, the statement of the lemma follows from equalities (43-45).
The total variation of the jump part of the measure µ s w in the representation formula (34) characterizes the role of discontinuities of functions u and (v 1 , v 2 ) in the term G.We have We denote by µ J w the jump part of µ s w and by µ C w the Cantor part.Then, using (33), the measure µ J w has components µ J w = (µ J w0 , µ J w1 , µ J w2 ) given by: for any B ∈ B(Ω).We have and we denote ν U the normal unit vector for H 2 -a.e.(x, t) ∈ S U .Then the measure µ J w and its total variation can be written as By taking the Euclidean norm of the vector (U + −U − ) Σ ρ , ν U ∈ R 3 inside the above integral we obtain the following structure formula.
Now we prove the analogous estimate for the functional G. First we prove that Let {z h 1 } be a sequence of vector fields z h 1 : Ω → R 3 with components and let {z h 2 } be a sequence of vector fields z h 2 : Ω → R 3 with components Since the function ψ : R 3 → R such that ψ(ξ) = ϕ(|ξ| 2 ) is Lipschitz, then we get: where L is the Lipschitz constant of ψ.Since (u h , v h ) BV −w * (u, v), then {u h } is uniformly bounded in W 1,1 (Ω) and {v h } is uniformly bounded in [W 1,1 (Ω)] 2 (see Proposition 3.13 of [2]), from which it follows where M is a positive constant independent of h.Since σ h → σ strongly in [L 1 (Ω)] 2 , then Σ h ρ → Σ ρ uniformly, from which equality (49) follows.Then, by the lower semicontinuity of G with respect to the componentwise weak- * -topology, we have We now consider the terms depending on the vector field σ = (σ 1 , σ 2 ).We have Dσ : B(Ω) → R 2•3 ; then let ψ : R 2×3 → R be a convex function satisfying the growth conditions (9) and such that ψ(ξ) = ϕ(|ξ| 2 ).If we consider the function of measure ψ(Dσ), then we have Since σ h BV −w * σ, then, by using the lower semicontinuity property of functions of measures ( [21], Theorem 3), we get Eventually, collecting the inequalities (48), ( 50) and (51), and by taking the infimum with respect to all sequences {w h } ⊂ V (Ω) such that w h → w in X(Ω), we obtain the estimate Step 2. Estimate of E(w) from above.
Let {ε h } be a sequence of positive numbers converging to zero as h → +∞, and let {ρ ε h } be a sequence of symmetric mollifiers, with ρ ε h having support in the sphere |(x, t)| ≤ ε h .Let u h , v h and σ h denote the convolutions Let A ⊂⊂ Ω be an open set.We set {w h } = {(u h , v h , σ h )}; we have w h ∈ V (A) for h large enough and w h BV −w * w in Y (A).Using equalities (24)(25)(26), we have Now we show that an analogous equality can be obtained for the relaxed functional G(w) by using the same sequence of functions {(u h , v h )} built by means of convolution.
We use the same notations as in the proof of Lemma 5.3; we set U = (u, v 1 , v 2 ) and . By the properties of mollifiers, and by definition of distributional gradient, we have for y ∈ Ω h : Let now µ denote the measure defined in (38); for y ∈ Ω h we define the function with values in where we have used (54) and the following property of mollifiers: Then we define the following sequence of measures {µ h }, where µ and µ h µ weakly as measures (see also proof of Theorem 4 of [21]).Then, by using the approximation property of functions of measures ( [21], Theorem 4), we have Let now Ψ and Ψ denote the functions defined in the proof of Lemma 5.3.The function Ψ is continuous in Ω × R 3•3 × R + , positively homogeneous of degree 1 in (ξ, ξ 0 ) and, using (35), satisfies the growth condition Since µ h µ weakly as measures, using (56), the representation formula (39), and the properties of function Ψ, an application of Reshetnyak continuity theorem (see Theorem 2.39 of [2] and Theorem 3 of [26]) yields Using now the definition of measures µ h , (40) and formulae (41) and (42) of Radon-Nikodym derivatives in the case (u h , v h ) ∈ V 3 (Ω h ), the definition (37) of function Ψ, and expression (36) of functional G, we have Using (55) and arguing as in the proof of (49) we get With the same method of proof we get Collecting equalities (53), (57) and (58), and taking into account that A ⊂ Ω h for large enough h, we obtain for any A ⊂⊂ Ω, from which it follows Using the same method of proof of Step 3 and of the Appendix, in the proof of Lemma 5.1, we find E(w, Ω) = sup{E(w, A) : A ⊂⊂ Ω}.
The jump part of the total variation term |D s σ|(Ω) in the representation formula (47) characterizes the role of discontinuities of the vector field σ = (σ 1 , σ 2 ) in the relaxed functional E. We have and we set Then the computation of the total variation of the measure D s σ yields the following structure formula.
6 Further remarks on the spatial average of the optical flow The integral formula in Remark 5.4 yields a representation of the contribution of G to the jump part of the energy of the type The average of the optical flow σ on a ball with a fixed radius ρ permitted us to compute an explicit expression of the function K, which is given in (46).In this section we discuss the complications that arise if we do not take the average of σ on a ball.First we discuss the behavior of the relaxed functional E when ρ tends to zero.We have ( [28], Theorem 5.14.4) where the function σ is called the precise representative of the BV function σ.
In this section we emphasize the dependence of functionals on the radius ρ of the ball.We further assume σ ∈ [BV (Ω) ∩ L ∞ (Ω)] 2 and we consequently restrict the spaces V (Ω), Y (Ω).For any w ∈ V (Ω) we denote E 0 (w) the functional obtained by replacing σ ρ with σ in E(w).
For any w ∈ Y (Ω) we denote E 0 (w) the functional obtained by replacing σ ρ with σ in E(w).Moreover, for any w ∈ Y (Ω) and any ρ > 0 we denote the relaxed functional by E ρ (w).If we now take the limit ρ → 0 + in formula (46), using Lebesgue dominated convergence theorem, we find that such a formula tends to where Σ = (σ 1 , σ2 , 1).Dealing analogously with the other terms, we find for any w = (u, v, σ) ∈ Y (Ω): lim The functional E ρ is lower semicontinuous (being a relaxed functional) for any ρ > 0. However, the functional E 0 , though well defined in Y (Ω), is not lower semicontinuous with respect to the weak- * -topology of Y (Ω) (see for instance [5] for a special case).Hence such a functional can not be the relaxed functional of E 0 .Without taking the average σ ρ , the relaxation problem is complicated by a phenomenon of fine-scale oscillations (microstructures).Such a phenomenon has been studied in detail by for every ξ ∈ C 0 (Ω j ).For p ∈ (0, 1) we set then it follows that u h p → pu in L 1 (Ω) and v h p → pv in [L 1 (Ω)] 2 as h tends to +∞.By the properties of function ϕ, let now ψ 0 : R → R be the convex and Lipschitz function such that ψ 0 (t) = ϕ(t 2 ).Then, by convexity of ψ 0 , we have: Using the functions ψ 1 and ψ 2 introduced in Step 1 of the proof of Lemma 5.1, we have by convexity of ψ 1 and ψ 2 : Hence, using inequalities (64) and (65), equality (63), (62) and Lebesgue dominated convergence theorem, the a.e.convergence of (u h j , v h j ) to (u, v) and the property of α j of being a partition of unity, it follows Let k ∈ N be fixed; we set E h = {x ∈ A : v h 1 (x) ≥ k or v h 2 (x) ≥ k}.We have where Using the Lipschitz property of ψ 1 and the growth condition (9) we have

For n = 1 , 2 , 3 ,
we denote by L n the Lebesgue n-dimensional measure in R n , and we denote by H 2 the Hausdorff two-dimensional measure in R 3 .If A and B are bounded subsets of R n and B is open, by A ⊂⊂ B we mean that A ⊂ B. Let B(Ω) be the σ-algebra of Borel subsets of Ω.For every measure µ : B(Ω) → R k , k ∈ N, we denote by |µ| its total variation.If µ is a real or vector valued measure and ν is a positive measure, then dµ/dν denotes the corresponding Radon-Nikodym derivative.If A ∈ B(Ω) we set µ A(B) = µ(A ∩ B) for every set B ∈ B(Ω).If a, b ∈ R n with n > 1 we denote by a ⊗ b their tensor product.