Modified iterated Tikhonov methods for solving systems of nonlinear ill-posed equations

We investigate iterated Tikhonov methods coupled with a Kaczmarz strategy for obtaining stable solutions of nonlinear systems of ill-posed operator equations. We show that the proposed method is a convergent regularization method. In the case of noisy data we propose a modification, the so called loping iterated Tikhonov-Kaczmarz method, where a sequence of relaxation parameters is introduced and a different stopping rule is used. Convergence analysis for this method is also provided.


Introduction
In this paper we propose a new method for obtaining regularized approximations of systems of nonlinear ill-posed operator equations.
The inverse problem we are interested in consists of determining an unknown physical quantity x ∈ X from the set of data (y 0 , . . . , y N −1 ) ∈ Y N , where X, Y are Hilbert spaces and N ≥ 1. In practical situations, we do not know the data exactly. Instead, we have only approximate measured data y δ i ∈ Y satisfying with δ i > 0 (noise level). We use the notation δ := (δ 0 , . . . , δ N −1 ). The finite set of data above is obtained by indirect measurements of the parameter, this process being described by the model where F i : D i ⊂ X → Y , and D i are the corresponding domains of definition.
Standard methods for the solution of system (2) are based in the use of Iterative type regularization methods [1,10,20]) or Tikhonov type regularization methods [10,25,29] after rewriting (2) as a single equation F (x) = y, where F := (F 0 , . . . , F N −1 ) : and y := (y 0 , . . . , y N −1 ). However these methods become inefficient if N is large or the evaluations of F i (x) and F ′ i (x) * are expensive. In such a situation, Kaczmarz type methods [18,24,26] which cyclically consider each equation in (2) separately are much faster [26] and are often the method of choice in practice.
For recent analysis of Kaczmarz type methods for systems of ill-posed equations, we refer the reader to [3,13,9,12]. The starting point of our approach is the iterated Tikhonov method [15,5,23] for solving linear ill-posed problems. This regularization method is defined by what corresponds to the iteration Motivated by the ideas in [3,12], we propose in this article an iterated Tikhonov-Kaczmarz method (iTK method) for solving (2). This iterative method is defined by Here α > 0 is an appropriate chosen number (see (9) below), [k] := (k mod N ) ∈ {0, . . . , N − 1}, and x δ 0 = x 0 ∈ X is an initial guess, possibly incorporating some a priori knowledge about the exact solution.
Remark 1.1. Notice that from the iteration formula in (4) we conclude that As usual for nonlinear Tikhonov type regularization, the global minimum for the Tikhonov functionals in (4) need not be unique. For exact data we obtain the same convergence statements for any possible sequence of iterates (see Section 3) and we will accept any global solution.
For noisy data, a (strong) semi-convergence result is obtained under a smooth assumption on the functionals F i (see assumption (A4) in Section 4), which guarantees uniqueness of global minimizers in (4).

Remark 1.2.
It is worth noticing that some authors consider iterated Tikhonov regularization with the number of iterations n ∈ N being fixed [11,21,27]. In this case, α plays the role of the regularization parameter. This regularization method is also called n-th iterated Tikhonov method.
The iTK method consists in incorporating the Kaczmarz strategy in the iterated Tikhonov method. This strategy is analog to the one introduced in [12] regarding the Landweber-Kaczmarz (LK) iteration, in [9] regarding the Steepest-Descent-Kaczmarz (SDK) iteration, in [13] regarding the Expectation-Maximization-Kaczmarz (EMK) iteration. As usual in Kaczmarz type algorithms, a group of N subsequent steps (starting at some multiple k of N ) shall be called a cycle. The iteration should be terminated when, for the first time, at least one of drops below a specified threshold within a cycle. That is, we stop the iteration at where τ > 1 still has to be chosen (see (9) below). Notice that for k = k δ * we do not necessarily In the case of noise free data, δ i = 0 in (1), the stop criteria in (6) may never be reached, i.e. k δ * = ∞ for δ i = 0. In the case of noisy data, we also propose a loping version of iTK, namely, the l-iTK iteration. In the l-iTK iteration we omit an update of the iTK iteration (within one cycle) if the corresponding i-th residual is below some threshold. Consequently, the l-iTK method is not stopped until all residuals are below the specified threshold. We provide a complete convergence analysis for both iTK and l-iTK iterations. In particular we prove that l-iTK is a convergent regularization method in the sense of [10].
The article is outlined as follows. In Section 2 we formulate basic assumptions and derive some auxiliary estimates required for the analysis. In Section 3 a convergence result for the iTK method is proved. In Section 4 a semi-convergence result for the iTK method for noisy data is proved. In Section 5 we introduce (for the case of noisy data) a loping version of the iTK method and prove a semi-convergence result for this new method. In Section 6 we discuss some possible applications related to parameter identification in elliptic PDE's. Section 7 is devoted to final remarks an conclusions.

Assumptions and preliminary results
We begin this section by introducing some assumptions, that are necessary for the convergence analysis presented in the next section. These assumptions derive from the classical assumptions used in the analysis of iterative regularization methods [10,20,27].
(A1) The operators F i are weakly sequentially continuous and Fréchet differentiable; the corresponding domains of definition D i are weakly closed. Moreover, we assume the existence of x 0 ∈ X, M > 0, and ρ > 0 such that Notice that x δ 0 = x 0 is used as starting value of the iTK iteration. (A2) This is an uniform assumption on the nonlinearity of the operators F i . We assume that the local tangential cone condition [10,20] holds for some η < 1.
We are now in position to choose the positive constants α and τ in (5), (6). For the rest of this article we shall assume where δ max := max j {δ j }. In particular, for linear problems we can choose τ = 1. Moreover, for exact data (i.e., δ j = 0, for j = 0, . . . , N − 1) we require simply α > 0.
In the sequel we verify some basic results that are necessary for the convergence analysis derived in the next section. The first result concerns the well-definiteness of the Tikhonov functionals which obviously relate to iteration (5) due to the fact that x δ k+1 ∈ arg min J k (x).
Lemma 2.1. Let assumption (A1) be satisfied. Then each Tikhonov functional J k in (10) attains a minimizer on X.
The assertion of Lemma 2.1 still holds true if, instead of (A1), we assume that the operator F [k] is continuous and weakly closed, and that D(F [k] ) is weakly closed [10]. In the next lemma we prove an estimate for the residual of the iTK iteration.
Lemma 2.2. Let x δ k and α be defined by (5) and (9) respectively. Then Proof. The inequality in (11) is a direct consequence of The following lemma is an important auxiliary result, which will be used to prove a monotony property of the iTK iteration. Lemma 2.3. Let x δ k and α be defined by (5) and (9) respectively. Moreover, assume that (A1) -(A3) hold true. If x δ k+1 ∈ B ρ (x 0 ) for some k ∈ N, then Proof. From (5) it follows that . Now, applying the Cauchy-Schwarz inequality and (8) , and (12) follows from this inequality together with (1).
It is worth noticing that the proof of Lemma 2.3 requires an assumption on x δ k+1 , namely that x δ k+1 ∈ B ρ (x 0 ). In the next lemma we make sure that this assumption is satisfied.
In the next two sections we provide a complete convergence analysis for the iTK iteration (see Theorems 3.2 and 4.3 below).

iTK Method: Convergence for exact data
Throughout this section, we assume that (A1) -(A3) hold true and that x δ k , α and τ are defined by (5) and (9). Our main goal in this section is to prove convergence of the iTK iteration for δ i = 0, i = 0, . . . , N − 1. For exact data y = (y 0 , . . . , y N −1 ), the iterates in (5) are denoted by x k to contrast with x δ k in the noisy data case.
For a detailed proof we refer the reader to [20].
Throughout the rest of this article, x † denotes the x 0 -minimal norm solution of (2). We define e k := x † − x k . From Proposition 2.5 it follows that e k is monotone non increasing.
Notice that Proposition 2.5 guarantees that (12) holds for all k ∈ N. Since the data is exact, (12) can be rewritten as 2 . By summing over all k, this leads to Equation (14) and the monotony of e k are the main arguments in the following proof of the convergence of the iTK iteration.
Theorem 3.2 (Convergence for exact data). For exact data, the iteration (x k ) converges to a solution of (2), as k → ∞. Moreover, if Proof. We have already observed that e k decreases monotonically. Therefore, e k converges to some ǫ ≥ 0. In the following we show that e k is in fact a Cauchy sequence. This is done similarly as in the proof of [9, Theorem 3.3]. The crucial difference is the fact that the term | e n − e k , e n | is here estimated by Then, it follows from (8) that Moreover, from the definition of the iterated Tikhonov method and and (7) it follows that (17), (18), (19) in (16) leads to and we finally obtain the estimate The remaining of the argumentation (including the proof of the second assertion) follows the lines of the proof of [9, Theorem 3.3].

iTK Method: Convergence for noisy data
Throughout this section, we assume that (A1) -(A3) hold true and that x δ k , α and τ are defined by (5), and (9). Our main goal in this section is to prove that x δ k δ * converges to a solution of (2) as δ → 0, where k δ * is defined in (6). Our first goal is to verify the finiteness of the stopping index k δ * .
. From Proposition 2.5 it follows that (12) can be applied recursively for k = 1, . . . , lN , and we obtain Using the fact that , we obtain the estimate Due to (9), the right hand side of (20) tends to +∞ as l → ∞, which gives a contradiction. Consequently, the minimum in (6) takes a finite value.
For the rest of this section we assume, additionally to (A1) -(A3), that (A4) The operators F i in (2) and it's derivatives F ′ i are Lipschitz continuous, i.e., there exists a constant L such that Moreover, the constants α in (9) and M in (7)   Then, for each k ∈ N we have lim j→∞ x Proof. Notice that the uniqueness of global minimizers of J k in (10) hold true. Indeed, let δ ∈ (0, ∞) N and y δ ∈ Y N be given as in (1). If x 1 , x 2 ∈ B ρ (x 0 ) are minimizers of J k , we have and from (A4) it follows that x 1 = x 2 . An immediate consequence of this uniqueness is the fact that the iterative steps x δ k+1 in (5) are uniquely defined (see (10)). The proof of Lemma 4.2 uses an inductive argument in k. First we consider the case k = 0. Notice that x δ j 0 = x 0 for j ∈ N and we can estimate Therefore, it follows from (A4) that lim j→∞ x δ j 1 = x 1 . Next, let k > 0 and assume that for all k ′ < k we have lim j→∞ x δ j k ′ +1 = x k ′ +1 . Arguing as in (21) we obtain the estimate and from the induction hypothesis we conclude that lim j→∞ x Proof. The proof is analogous to the proof of [9, Theor. 3.6] and will be omitted. In the proof, [9, Theor. 3.5] has to be replaced by Lemma 4.2 above.

Remark 4.4.
The assumption on the boundedness of the sequence {k j * } j∈N in Theorem 4.3 is crucial for the proof. This assumption is natural when dealing with ill-posed problems and noisy data, since in practical applications one generally has k δ * → ∞ as δ → 0. A similar assumption is also needed in [22] to prove convergence of the Landweber-Kaczmarz iteration for noisy data.
In Section 5 we investigate the coupling of the iTK iteration with a loping strategy, which allow us to drop the above assumption on the boundedness of {k j * } j∈N and still prove a semiconvergence result analog to Theorem 4.3.

The loping iterated Tikhonov-Kaczmarz method
Motivated by the ideas in [12,9,13,3], we investigate in this section a loping iterated Tikhonov-Kaczmarz method (l-iTK method) for solving (2). This iterative method is defined by where The positive constants α and τ are defined as in (9). The meaning of (23), (24) is the following: at each iterative step an element x k+1/2 ∈ D [k] satisfying For exact data (δ = 0) the l-iTK reduces to the iTK iteration investigated in the previous sections. For noisy data however, the l-iTK method is fundamentally different from the iTK method: The bang-bang relaxation parameter ω k effects that the iterates defined in (5) become stationary if all components of the residual vector F i (x δ k ) − y δ i fall below a prespecified threshold. This characteristic renders (5) a regularization method, as we shall see in Subsection 5.1. 2 + α x − x δ k and is not uniquely defined. For noisy data, a semi-convergence result is obtained under the smooth assumption (A4) on the functionals F i , which guarantees that the l-iTK iteration is uniquely defined.
The l-iTK iteration should be terminated when, for the first time, all x δ k are equal within a cycle. That is, we stop the iteration at Notice that k δ * is the smallest multiple of N such that

Convergence analysis
In what follows we assume that (A1) -(A3) and (A4) hold true and that x δ k , ω k , α and τ are defined by (23), (24) and (9). We start by listing some straightforward facts about the l-iTK iteration: • Lemma 2.2 holds true. Lemma 2.3 still holds true, but (12) has to be replaced by • Lemma 2.4 and Proposition 2.5 hold true.
• Theorem 3.2 holds true (for exact data, the l-iTK iteration reduces to iTK).
Before proving the main semiconvergence theorem we need two auxiliary results: the first result guarantees that, for noisy data, the stopping index k δ * in (25) is finite (compare with Proposition 4.1); the second result is the analogous of Lemma 4.2 for the l-iTK iteration. (25) is finite, and where κ : Proof. Assume by contradiction that for every l ∈ N , there exists i(l) ∈ {0, . . . , N − 1} such that x lN +i(l) = x lN . From Proposition 2.5 it follows that (27) can be applied recursively for k = 1, . . . , lN , and we obtain Using the fact that either ω k = 0 or , we obtain the estimate Equation (29) and the fact that x l ′ N +i(l ′ ) = x l ′ N for all l ′ ∈ N, imply Due to (9), the right hand side of (30) tends to +∞ as l → ∞, which gives a contradiction. Consequently, the set {l ∈ N : is not empty and the minimum in (6) takes a finite value.
We are now ready to state and prove a semiconvergence result for the l-iTK iteration.
Theorem 5.4. Let δ j = (δ j,0 , . . . , δ j,N −1 ) be a given sequence in (0, ∞) N with lim j→∞ δ j = 0, and let y δ j = (y δ j 0 , . . . , y δ j N −1 ) ∈ Y N be a corresponding sequence of noisy data satisfying y δ j i −y i ≤ δ j,i , i = 0, . . . , N −1, j ∈ N. Denote by k j * := k * (δ j , y δ j ) the corresponding stopping index defined in (25). Then x δ j k j * converges to a solution x * of (2) as j → ∞. Moreover, if Proof. The proof is analogous to the proof of [9, Theorem 3.6] and is divided in two cases. In the second case (the sequence k j * is not bounded) one has to argue with Lemma 5.3.

Applications
In this section we address parameter identification problems in elliptic equations. In the focus is the question whether the local tangential cone condition (8) is satisfied. Part of the following analysis is based on the verification of a stronger condition, which implies the local tangential cone condition, namely the (adjoint) range invariance condition: 2 There exists a family of bounded linear operators R x : Y −→ Y and a positive constant such that It is a well known fact that the range invariance condition implies that range(F ′ (x)) = range(F ′ (x † )), x ∈ B ρ (x 0 ). The model problem under investigation is an elliptic boundary value problem Here f is a given function in L 2 (0, 1) and α i , β i , g i are real numbers specified below. To simplify the discussion we consider here the one-dimensional case only, but we shall give some hints for two-and three-dimensional cases.
The equation in (35) may be considered as a simplified model for a steady state convectiondiffusion equation. The term cu is a production term where the function c depends on properties of the material. The term −(au s ) s +(bu) s results from an ansatz for the flux j := −au s +bu .
Here a, b are functions describing the diffusion and convective part, respectively. For a concrete application see for instance [2], Chapter I.2.
We want to identify the parameters a, b, c from a measurement u δ ∈ L 2 (0, 1) of the solution u ∈ L 2 (0, 1) of the boundary value problem (35), (36). We distinguish between three different inverse problems, namely the so called a/b/c-problems: The a-problem: Find a under the assumptions b ≡ 0, c ≡ 0.
The b-problem: Find b under the assumptions a ≡ 1, c ≡ 1.
The c-problem: Find c under the assumptions a ≡ 1, b ≡ 0.
Each problem may be presented by a nonlinear equation of the type F (x) = y for an appropriately chosen parameter-to-output mapping F : The a-and c-problem are considered in a huge amount of references whereas the b-problem received less attention. It seems that the tangential cone condition for this problem has not been investigated up to now; we do that below. A detailed analysis of regularization methods for the identification in elliptic and parabolic equations can be found in [4].

The c-problem
Let us start the discussion with the c-problem, the most simple one. Here the mapping F is defined as follows: F : D ∋ c → u(c) ∈ L 2 (0, 1) , D ⊂ X := Y := L 2 (0, 1) , 2 For a proof that the local tangential cone condition follows from the range invariance condition, see [16].
where u(c) solves the boundary value problem −u ss + cu = f , in (0, 1) in the weak sense. The domain of definition is chosen as a ball in X := L 2 (0, 1) (see [8]): Then the mapping F is Fréchet-differentiable in D (see [10,20]) and we have where Γ(c) : H 2 (0, 1) ∩ H 1 0 (0, 1) → L 2 (0, 1) is defined by Γ(c)u := −u ss + cu. We assume that c 0 is chosen such that u(c) ≥ κ a.e. for each c ∈ D, where κ is a positive constant. Then we have with Here κ 1 is a positive constant. As a result, we see that the range invariance condition is satisfied and the tangential cone condition follows.
Remark 6.1. The results above hold also in the two-and three-dimensional cases; no further assumptions are necessary (see, e.g., [14,19]). Clearly, the boundary conditions have now to be considered in the sense of trace operators.

The b-problem
Here the parameter-to-output mapping F is defined as follows: in the weak sense. The boundary value problem above is uniquely solvable in H 1 (0, 1) whenever b X is small enough, which can be seen from an application of the Lax-Milgram-Lemma. Therefore we choose D as a ball B ρ := {x ∈ X | x X ≤ ρ} in X with ρ small enough such that u(b) is uniquely determined for each b ∈ B ρ . Additionally, the assumption that each parameter b belongs to H 1 (0, 1) ensures that the solution u(b) is in H 2 (0, 1).
Remark 6.2. The formulation of the b-problem above can be easily generalized to the twodimensional case. 3 The convection term in this case is ∂ 1 (bu) + ∂ 2 (bu) and again a scalar function b has to be identified. The situation is different when one models the first order term in the equation by b 1 ∂ 1 u+b 2 ∂ 2 u [17]. Then one has to identify two parameters and the analysis is much more delicate. It seems that the identification problems has not been considered in the framework chosen above; see [7] for the investigation of identifiably for this inverse problem.

The a-problem
Here the parameter-to-solution mapping F is defined by where u(a) solves the boundary value problem −(au s ) s = f , in (0, 1) in the weak sense. The domain of definition is chosen as where a is a positive constant. One can prove [20] that F is Fréchet differentiable in D with where A(a) : H 2 (0, 1) ∩ H 1 0 (0, 1) → L 2 (0, 1) is defined as A(a)u := −(au s ) s and J : H 2 (0, 1) → L 2 (0, 1) is defined by Jψ := −ψ ss + ψ (J is the adjoint of the embedding of H 1 (0, 1) into L 2 (0, 1)). In [20] it is shown that the tangential cone condition is satisfied. Remark 6.3. The results in this section strongly benefit from the fact that the model is onedimensional. One can see this for instance that, due to the choice of the parameter space, each admissible parameter is a continuous function. In the two-or three-dimensional case additional assumptions are necessary in order to obtain the same results (see, e.g., [14]). Remark 6.4. It seems that the range invariance condition cannot be proved (even under stronger regularity assumptions) for the a-and the b-problem, respectively; for the a-problem see [16]. Notice that the presentation of the Fréchet-derivative in (41), (38) cannot be handled in the same way as in the case of the c-problem.

Conclusions
In this paper we propose a new iterative method for inverse problems of the form (2), namely the iTK iteration. In the case of noisy data, we also propose a loping version of iTK, namely, the l-iTK iteration.
In the particular case of dealing with a single operator equation (N = 1 in (2)), iTK and l-iTK are the same iteration and reduce to the classical iterated Tikhonov method. To the best of our knowledge this method has so far been investigated only for linear problems [5,15,23] and the convergence analysis for nonlinear operator equations was still open.

Three good reasons for using the loping iteration
The first reason is a numerical one: Notice that, (11) allow us to conclude ω k = 0 without having to compute x k+1/2 at all. Therefore, after a large number of iterations, ω k will vanish for some k within each iteration cycle and the computational expensive evaluation of x k+1/2 (solution of a nonlinear equation) might be loped, making the l-iTK method in (23) a fast alternative to the iTK method as well as to classical Kaczmarz type methods [22,6].
The second reason is of analytical nature: An alternative to relax the assumption on the boundedness of the sequence {k j * } j∈N in Theorem 4.3 and still prove a semiconvergence result, is the introduction of the loping strategy above. This is done in Theorem 5.4.
The third reason is of heuristic nature: The rules for choosing the stooping index k δ * in (6) and in (25) are quite different. According to (6) the iTK iteration should be stopped when for the first time one of the equations of system 2 is satisfied within a specified threshold. Therefore, at the iteration step x δ k δ * , we cannot control all the residuals F i (x δ k ) − y δ i within the cycle.
According to (25) however, the l-iTK iteration only stops when all the residuals F i (x δ k )− y δ i , i = 0, . . . , N − 1 drop below a specified threshold. Consequently, although the l-iTK iteration needs more steps to reach discrepancy, it produces an approximate solution x δ k δ * which better fits all the system data.