Recursive Construction of Confidence Regions

Assuming that one-step transition kernel of a discrete time, time-homogenous Markov chain model is parameterized by a parameter $\theta\in \boldsymbol \Theta$, we derive a recursive (in time) construction of confidence regions for the unknown parameter of interest, say $\theta^*\in \boldsymbol \Theta$. It is supposed that the observed data used in construction of the confidence regions is generated by a Markov chain whose transition kernel corresponds to $\theta^*$ . The key step in our construction is derivation of a recursive scheme for an appropriate point estimator of $\theta^*$. To achieve this, we start by what we call the base recursive point estimator, using which we design a quasi-asymptotically linear recursive point estimator (a concept introduced in this paper). For the latter estimator we prove its weak consistency and asymptotic normality. The recursive construction of confidence regions is needed not only for the purpose of speeding up the computation of the successive confidence regions, but, primarily, for the ability to apply the dynamic programming principle in the context of robust adaptive stochastic control methodology.


Introduction
Suppose that a set of dynamic probabilistic models is selected and that it is parameterized in terms of a finite dimensional parameter θ taking values in a known parameter space Θ. We postulate that all these models are possible descriptions of some reality, which is of interest to us, and that only one of the models, say the one corresponding to θ * ∈ Θ, is the adequate, or true, description of this reality.
Motivated by discrete time robust stochastic control problems subject to model uncertainty (cf. [BCC + 17]), we consider in the present paper discrete time, time-homogeneous Markov chain models only. Accordingly, we assume that the one-step transition kernel of a Markov chain model is parameterized by θ. We postulate that the true parameter θ * is not known, and the main goal is to derive a recursive (in time) construction of confidence regions for θ * . Needless to say, we are seeking a recursive construction of confidence regions for θ * that satisfy desired properties; in particular, some asymptotic properties, as the time series of observations increases.
The recursive construction of confidence regions is needed not only for the purpose of speeding up the computation of the successive confidence regions, but, primarily, for the ability to apply Bellman principle of optimality in the context of adaptive robust stochastic control methodology introduced in [BCC + 17]. The adaptive robust control method is a form of stochastic control problem subject to Knightian uncertainty that incorporates updating controller's knowledge about the parameter space by means of confidence regions of the true parameter θ * -a form of learning, directed towards reducing uncertainty about the unknown true parameter using the incoming information about the underlying signal process. In this case, the statistical properties of the confidence regions (decreasing volumes and weak consistency) are directly related to the reduction of model uncertainty. As with any stochastic control problem, the viability of the method relies on establishing the Bellman principle of optimality, where the authors used essentially the recursive construction of confidence regions discussed herein. Also in [BCC + 17], the authors considered a specific stochastic optimal control problem from portfolio selection problem, where, in particular, one dimensional and two dimensional recursive confidence regions where explicitly constructed. Robust stochastic control problems provide primary motivation for the present work, but, clearly, potential applications of the results presented here are far reaching.
There is a vast literature devoted to recursive computation, also known as on-line computation, of point estimators. It is fair to say though that, to the best of our knowledge, the literature regarding recursive construction of confidence regions and their asymptotic analysis is very scarce. In fact, we were able to identify only two previous works, [Yin89] and [Yin90], touching upon this subject. In this regard, our work is the first to fully concentrate on the recursive construction of confidence regions and their asymptotic analysis. The geometric idea that underlies our recursive construction is motivated by recursive representation of confidence intervals for the mean of one dimensional Gaussian distribution with known variance, and by recursive representation of confidence ellipsoids for the mean and variance of one dimensional Gaussian distribution, where in both cases observations are generated by i.i.d. random variables. The recursive representation is straightforward in the former case, but it is not so any more in the latter one.
In the already mentioned works, [Yin89] and [Yin90], a closely related idea was used for constructions of sequentially determined confidence ellipsoids based on stopped Brownian motions. Our paper and Yin's work compare as follows: We take ergodic Markov chains as our underlying processes, whereas Yin considered several other different processes such as moving average processes and stationary φ-mixing processes. While providing a formula for the confidence ellipsoids that is essentially a recursive formula, Yin was interested in computing the volume of the ellipsoids. By defining a stopping time as the first time that the volume of an ellipsoid is smaller than some threshold, in [Yin89] and [Yin90] the author proved a series of properties for such stopping time, and developed a stopping rule for recursive on-line algorithms. We, on the other hand, focus on a constructive recursive derivation of the extreme points of our confidence regions. Specifically, we provide a recursive formula for the extreme points so that we are able to efficiently compute these points and therefore to efficiently represent the points that lie in the confidence regions. This is an important new development, as it allows us to apply dynamic programming principle to the robust stochastic control problem that is studied in [BCC + 17]. From the numerical point of view, formulae for extreme points of the ellipsoids lead to efficient solution to the optimization problems that we encounter. We prove the weak consistency for the recursive confidence regions, for which having the representations of the extreme points plays an important role.
As it will be seen, one of the the key ingredients in our recursive construction of confidence regions is an appropriate recursive scheme for deriving a point estimator of θ * . In this regard, building upon classical results from inferential statistics and from the area of stochastic approximation (cf. [KY03], [LeC56], [LeC60]), in Section 3: • We introduce the concept of quasi-asymptotic linearity of a point estimator of θ * , which is satisfied by the recursive point estimation scheme that we develop in Section 3.2. This concept is related to the classic definition of asymptotic linearity of a point estimator, but it overcomes one serious drawback that the classic concept suffers from: asymptotic linearity fails to be reconciled with the full recursiveness in some applications.
• Starting from what we call the base recursive point estimation scheme, we design a quasi-asymptotically linear recursive point estimation scheme, and we prove the weak consistency and asymptotic normality of the point estimator generated by this scheme.
The main original contribution of this paper is provided in Section 4 and it can be summarized as follows: We provide a recursive construction of confidence regions for θ * . We prove that these confidence regions are weakly consistent, that is, they converge in probability (in the Hausdorff metric) to the true parameter θ * .
The paper is organized as follows. In Section 2 we introduce the Markov chain framework relevant for the present study.
In Section 3 we provide two recursive schemes for derivation of point estimators. Section 3.1 is devoted to the recursive construction of what we call the base (recursive) point estimator of θ * . In our set-up, finding a point estimator of θ * translates to finding a point estimator of the solution of equation (13). One of the most widely used iterative root finding procedures for such equations is the celebrated stochastic approximation method. Our base (recursive) point estimation scheme for θ * is an adaptation of the stochastic approximation method. Also, here we prove the strong consistency of the base point estimator. The key step to the desired recursive construction of confidence regions for θ * is to establish the asymptotic normality of the underlying recursive point estimator. In this regard one could impose additional assumptions, on top of the conditions needed for proving consistency of the point estimator, to obtain asymptotic normality for stochastic approximation estimators (see e.g. [Sac58], [Fab68], [LR79]). This however would typically result in imposing a long list of assumptions that would not be easily verifiable. Thus, we choose to go into a different direction by modifying the base estimatorθ to the effect of producing a recursive estimator that is asymptotically normal. Therefore, in Section 3.2 we appropriately modify our base (recursive) point estimator, so to construct a quasi-asymptotically linear (recursive) point estimator, for which we prove weak consistency and asymptotic normality.
The main section of this paper is Section 4, which contains the main original contribution of the paper. This section is devoted to the recursive construction of confidence regions for θ * , and to studying their asymptotic properties. In particular, we show that confidence regions derived from quasi-asymptotically linear (recursive) point estimators preserve a desired geometric structure. Such structure guarantees that we can represent the confidence regions in a recursive way in the sense that the region produced at step n is fully determined by the region produced at step n − 1 and by the newly arriving observation of the underlying reality.
Illustrating examples are provided in Section 5. The paper is completed with technical Appendices that also contain some of the proofs.

Preliminaries
Let (Ω, F ) be a measurable space. The non-empty compact hyperrectangle Θ ⊂ R d will play the role of the parameter space throughout. 1 We define C(θ), θ ∈ ∂Θ, to be the infinite convex cone generated by the outer normals at θ of the faces on which θ lies; and C(θ) = {0} if θ belongs to the interior of Θ. On the space (Ω, F ) we consider a discrete time, real valued random process Z = {Z n , n ≥ 0}. 2 We postulate that this process is observed, and we denote by F = (F n , n ≥ 0) its natural filtration. The (true) law of Z is unknown, and assumed to belong to a parameterized family of probability distributions on (Ω, F ), say {P θ , θ ∈ Θ}. It will be convenient to consider (Ω, F ) to be the canonical space for Z, and to consider Z to be the canonical process (see Appendix A for details). Consequently, the law of Z under P θ is the same as P θ . The (true) law of Z will be denoted by P θ * ; accordingly, θ * ∈ Θ is the (unknown) true parameter. We assume that θ * lies in the interior of Θ.
The set of probabilistic models that we are concerned with is {(Ω, F , F, Z, P θ ), θ ∈ Θ}. The model uncertainty addressed in this paper occurs if Θ = {θ * }, which we assume to be the case. Our objective is to provide a recursive construction of confidence regions for θ * , based on accurate observations of realizations of process Z through time, and satisfying desirable asymptotic properties.
In what follows, all equalities and inequalities between random variables will be understood in P θ * almost surely sense. We denote by E θ * the expectation operator corresponding to P θ * .
We impose the following structural standing assumption.

Assumption M:
(i) Process Z is a time homogenous Markov process under any P θ , θ ∈ Θ.
(ii) Process Z is an ergodic Markov process under P θ * . 3 (iii) The transition kernel of process Z under any P θ , θ ∈ Θ, is absolutely continuous with respect to the Lebesgue measure on R, that is, for any Borel set A ⊂ R, for some positive and measurable function p θ . 4 In other words, p θ is the conditional density function of the Markov process Z under P θ .
Remark 1. Since the process Z is ergodic then it is also a stationary process under P θ * (see Appendix A.1). Consequently, under P θ * , for each θ ∈ Θ and for each n ≥ 0, the law of π n (θ) is the same as the law of π 1 (θ).
We will also need to impose several technical assumptions, beginning with R0. For any θ ∈ Θ, π 1 (θ) is integrable under P θ * .
Note that, assuming that M and R0 hold and using Proposition 18 as well as the Kullback-Leibler Lemma (cf. [KL51]), we see that the following properties are satisfied: (1) Moreover, for any θ ∈ Θ, In the statement of the technical assumptions R1-R8 below we adapt the notations where ∇ denotes the gradient vector and H denotes the Hessian matrix with respect to θ, respectively. Due to the fact that Z is a Markov process, we have which implies that for each n ≥ 1, b n is indeed a functional of θ and Z n−1 , and we postulate that b n is continuous with respect to Z n−1 .
(iii) A detailed analysis of the ODE (6) is not given here due to space limitation. However, it proceeds in analogy to what is done in [KY03, Section 4.3]. One concludes that this equation admits a unique solution for x(0) ∈ Θ.
As stated above, our aim is to provide a recursive construction of the confidence regions for θ * . In the sequel, we will propose a method for achieving this goal that will be derived from a suitable recursive point estimator of θ * . Note that due to (2) and Assumption R3, we have that θ * is the unique solution of Therefore, constructing a point estimator of θ * is equivalent to constructing a point estimator of the solution of equation (13). Since θ * is unknown, the left-hand-side of the equation (13) is not really known to us. We will therefore apply an appropriate version of the so called stochastic approximation method, which is a recursive method used to point-estimate zeros of functions that can not be directly observed. This can be done in our set-up since, thanks to (1), we are provided with a sequence of observed random variables 1 n n i=1 ψ i (θ) that P θ * almost surely converges to E θ * [ψ 1 (θ)] -a property, which will enable us to adopt the method of stochastic approximation. Accordingly, in the next two sections, we will introduce two recursive point estimators of θ * , and we will derive properties of these estimators that are relevant for us.

Recursive point estimators
In this section two types of recursive point estimators are derived that are needed for construction of the recursive confidence regions from Section 4.

√ n-consistent base point estimator
In this section we consider a recursive point estimatorθ = {θ n , n ≥ 1} of θ * , that will be defined in (14). Towards this end, we fix a positive constant β such that βK 1 > 1 2 , where K 1 was introduced in Assumption R6. Then, we follow the idea in [KY03] and define the processθ recursively as follows, with the initial guessθ 0 being an element in Θ, where ψ n was defined in (3). The projection term J n is chosen so that β n J n is the vector of shortest Euclidean length needed to takeθ n−1 + β n ψ n (θ n−1 ) back to the set Θ. It is not hard to see that J n ∈ −C(θ n ).
Given the definition of ψ n , we see thatθ n is updated fromθ n−1 based on new observation Z n available at time n; of course, Z n−1 is used as well. We note that the recursion (14) is a version of the constrained stochastic approximation method, which is meant to recursively approximate roots of the unknown equations, such as equation (13) (see e.g. [RM51], [KW52], [LS87], [KC78], [KY03]).
Remark 3. In applications of stochastic approximation, there are two ways to deal with the case of iterates becoming too large. One is to impose some stability conditions on the problem, and the other is to make appropriate adjustments to the basic algorithm. The latter is usually called constrained or truncated stochastic approximation (see e.g. [KC78], [BK02], [SZ16]). In this work, we use the second method so that assumptions R1-R8 only need to be satisfied for θ that belongs to a compact subset of R d instead of the whole space.
Remark 4. In practice, for Θ that is defined as a hyperrectangle, the projection term J n is easily computable. See (25) and (26) as an example in the two dimensional case. It is also worth noting, as discussed in [KY03], that there are other feasible construction of Θ.
As mentioned above, we are interested in the study of asymptotic properties of confidence regions that we will construct recursively in Section 4. These asymptotic properties crucially depend on the asymptotic properties of our recursive (point) estimators. One of such required properties is asymptotic normality. As discussed earlier, we will modify the base estimatorθ to the effect of producing a recursive estimator that is asymptotically normal. In the next section we will construct such estimator, denoted there asθ, and we will study its asymptotic properties in the spirit of the method proposed by Fisher [Fis25]. Motivated by finding estimators that share the same asymptotic property as maximum likelihood estimators (MLEs), Fisher proposed in [Fis25] that if an estimator is √ n-consistent (see below), then appropriate modification of the estimator has the same asymptotic normality as the MLE. This subject was further studied by LeCam in [LeC56] and [LeC60], where a more general class of observation than i.i.d. observations are considered.
Accordingly, we will show thatθ n is strongly consistent, and, moreover it maintains √ n convergence rate, i.e.
An estimator that satisfies this equality is said to be √ n-consistent. For convenience, throughout, we will use the notation Δ n :=θ n − θ * , n ≥ 1. Next two results show thatθ is strongly consistent and √ n-consistent. The proofs of these results are deferred to Appendix A.3.

Quasi-asymptotically linear estimator
In this section we define a new estimator denoted as {θ n , n ≥ 1} and given recursively byθ where Id is the unit matrix. Sinceθ n , I n , and Γ n are updated from time n − 1 based on the new observation Z n available at time n, then the estimatorθ indeed is recursive. This estimator will be used in Section 5 for recursive construction of confidence regions for θ * . Remark 7. In the argument below we will use the following representations of Γ n and I n , Next, we will show thatθ is weakly consistent and asymptotically normal. We will derive asymptotic normality ofθ from the property of quasi-asymptotic linearity, which is related to the asymptotic linearity property (cf. [Shi84]), and which is defined as follows: Definition 8. An estimator {θ n , n ≥ 1} of θ * is called a quasi-asymptotically linear estimator if there exist a P θ * -convergent, adapted matrix valued process G, and adapted, vector valued processes ϑ and ε, such that Our definition of quasi-asymptotically linear estimator is motivated by the classic concept of asymptotically linear estimator (see e.g. [Sha10]):θ is called (locally) asymptotically linear if there exists a matrix process {Ǧ n , n ≥ 1} such thatθ Asymptotic linearity is frequently used in the proof of asymptotic normality of estimators. However, in general, asymptotic linearity can not be reconciled with the full recursiveness of the point estimator. The recursiveness of the point estimator is the key property involved in construction of recursive confidence regions. As it will be shown below, the fully recursive estimatorθ is quasi-asymptotically linear.
In what follows, we will make use of the following representation forθ Theorem 9. Assume that R1-R8 hold. Then, the estimatorθ is P θ * -weakly consistent, namelyθ n Moreover,θ is a quasi-asymptotically linear estimator for θ * .
The proof is differed to the Appendix A.3. The next result, which will be used in the analysis of asymptotic properties of the recursive confidence region for θ * in Section 6, is an application of Theorem 9.
Proposition 10. Assume that R1-R9 are satisfied. Then, there exists an adapted process ϑ such that ϑ n and See Appendix A.3 for the proof. We end this section with the following technical result, which will be used in our construction of the confidence region in Section 6. Towards this end, for any θ ∈ Θ and n ≥ 1, we define 7 where (σ ij n ) i,j=1,...,d = I(θ n ), and, as usual, we denote by χ 2 d a random variable that has the chi-squared distribution with d degrees of freedom.
Corollary 11. With ϑ n = −I −1 (θ n )I n θ * , we have that Proof. From Assumption R8, strong consistency ofθ and Proposition 10, and employing the Slutsky's theorem again, we get that

Recursive construction of confidence regions
This section is devoted to the construction of the recursive confidence region based on quasi-asymptotically linear estimatorθ developed in Section 3.2. We start with introducing the definition of the approximated confidence region.
Proof. As in Section 3.2, we take ϑ n = −I −1 (θ n )I n θ * , which in view of Proposition 10 is a weakly consistent estimator of θ * . Note that U n ( · ) is a continuous function, and thus T n is a connected set, for any n ≥ 1. By Corollary 11, U n (ϑ n ) d − → χ 2 d , and since P θ * (ϑ n ∈ T n ) = P θ * (U n (ϑ n ) < κ), we immediately have that lim n→∞ P θ * (ϑ n ∈ T n ) = 1 − α. This concludes the proof.
Next, we will show that the approximated confidence region T n can be computed in a recursive way, by taking into account its geometric structure. By the definition, the set T n is the interior of a d-dimensional ellipsoid, and hence T n is uniquely determined by its extreme 2d points. Thus, it is enough to establish a recursive formula for computing the extreme points. Let us denote by (θ 1 n,k , . . . , θ d n,k ), k = 1, . . . , 2d, the coordinates of these extreme points; that is θ i n,k , denotes the ith coordinate of the kth extreme point of the ellipsoid T n .

T. Bielecki et al.
Then, in the original system of coordinates, the extreme points (written as vectors) are given by where {e j }, j = 1, . . . , d, is the standard basis in R d . Finally, taking into account the recursive constructions (14), (17), and the representation (22), we have the following recursive scheme for computing the approximate confidence region. Output:θ n =θ n−1 + β n ψ n (θ n−1 ) + β n J n , From here, we also conclude that there exists a function τ , independent of n, such that T n = τ (T n−1 , Z n ).

Recursive construction of the confidence region
The above recursive relationship goes to the heart of application of recursive confidence regions in the robust adaptive control theory originated in [BCC + 17], since it makes it possible to take the full advantage of the dynamic programming principle in the context of such control problems. We conclude this section by proving that the confidence region converges to the singleton θ * . Equivalently, it is enough to prove that the extreme points converge to the true parameter θ * .

Proposition 14.
For any k ∈ {1, . . . , 2d}, we have that Proof. By Assumption R8 and Theorem 5 (strong consistency ofθ), we have that L n a.s. − −−− → n→∞ I 1/2 (θ * ), and consequently, we also have that Of course, the last convergence holds true in the weak sense too. Passing to the limit in (22), in P θ * probability sense, and using (24) and weak consistency of θ (Theorem 9), we finish the proof.

Examples
In this section we will present three illustrative examples of the recursive construction of confidence regions developed above. We start with our main example, Example 15, of a Markov chain with Gaussian transitional densities where both the conditional mean and conditional standard deviation are the parameters of interest. Example 16 is dedicated to the case of i.i.d. Gaussian observations, which is a particular case of the first example. Generally speaking, the simple case of i.i.d. observations for which the MLE exists and asymptotic normality holds true, one can recursively represent the sequence of confidence intervals constructed in the usual (off-line) way, and the theory developed in this paper is not really needed. The idea is illustrated in Example 17 by considering again the same experiment as in Example 16. In fact, as mentioned above, this idea served as the starting point for the general methodology presented in the paper. , n ≥ 1, and such that Z 0 ∼ N (μ, σ 2 ). We assume that the correlation parameter ρ ∈ (−1, 1) is known, and the unknown parameter is θ = (μ, σ 2 ) ∈ (−∞, ∞) × (0, ∞). The pair of true parameters (μ * , (σ * ) 2 ) lies in the interior of Θ = [a 1 , a 2 ]×[b 1 , b 2 ], and a 1 ≤ a 2 , b 1 ≤ b 2 are some fixed real numbers with b 1 > 0.
In the Appendix A.3 we show that the process Z satisfies Assumption M, and conditions R0-R8.
Thus, all the results derived in the previous sections hold true. Moreover, for a given confidence level α, we have the following explicit formulas for the nth step of the recurrent construction of the confidence regions: and, for j ∈ {1, 2, 3, 4}, (1−ρ)b1 , and P θ * (χ 2 2 < κ) = 1 − α. The projection terms J 1 n and J 2 n are defined as follows, and Example 16. Let Z n , n ≥ 0, be a sequence of i.i.d. Gaussian random variables with an unknown mean μ and unknown standard deviation σ. Clearly, this important case is a particular case of Example 15, with ρ = 0, and the same recursive formulas for confidence regions by taking ρ = 0 in the above formulas.
Therefore, taking into account (27), we have a recursive formula for computing these extreme points, and thus the desired recursive construction of the confidence regions T n .

A.1. Ergodic Markov Chains
In this section, we will briefly recall some facts from the ergodic theory of Markov processes in discrete time. Let X be a time homogeneous Markov chain on a probability space (Ω, F, P), which takes values in a measurable space (X , X). We refer to [Rev84, Chapter 4, Definition 2.6] for the definition of ergodicity for Markov processes. If X is an ergodic process under P, then it is also a stationary process, i.e. for any n ≥ 1, the law of (X j , X j+1 , . . . , X j+n ) under P is independent of j, j ≥ 0. As usual, we denote by E P the expectation under P. In view of the classical Birkhoff's Ergodic Theorem, we have the following result, that will be used in this paper.
Proposition 18. Let X be ergodic. Then, for any g such that E P [g(X 0 , . . . , Next, we provide a brief discussion regarding sufficient conditions for the Markov chain X to be ergodic. Let Q : X × X → [0, 1] be the transition kernel of X. A probability measure π on (X , X) is called an invariant measure of Q if
Proposition 19. If a transition kernel Q has a unique invariant probability measure π, then X is ergodic under P π .
One powerful tool for checking the uniqueness of invariant probability measure is the notion of positive Harris chain. There are several equivalent definitions of positive Harris Markov chain, and we will use the one from [HLL00].
Definition 20. The Markov chain X with transition kernel Q is called a positive Harris chain if (a) there exists a σ-finite measure μ on X such that for any x 0 ∈ X , and B ∈ X with μ(B) > 0 P(X n ∈ B for some n < ∞|X 0 = x 0 ) = 1, (b) there exists an invariant probability measure for Q.
Remark 21. It is well known (cf. e.g. [MT93]) that a positive Harris chain admits a unique invariant measure. Thus, in view of Proposition 19, a positive Harris chain is also ergodic.

A.2. CLT for multivariate martingales
In this section, for a matrix A with real valued entries we denote by |A| the sum of the absolute values of its entries.
In [CP05] Proposition 3.1, the authors gave the following version of the central limit theorem for discrete time multivariate martingales.
Proposition 22. On a probability space (Ω, F , P) let D = {D n,j , 0 ≤ j ≤ k n , n ≥ 1} be a triangular array of d-dimensional real random vectors, such that, for each n, the finite sequence {D n,j , 1 ≤ j ≤ k n } is a martingale difference process with respect to some filtration {F n,j , j ≥ 0}. Set Also denote by U the σ-algebra generated by j H j , where H j := lim inf n F n,j . Suppose that D * n converges in L 1 to zero and that U n converges in probability to a U measurable d-dimensional, positive semi-definite matrix U . Then, the random vector kn j=1 D n,j converges U -stably to the Gaussian kernel N (0, U). Remark 23. U -stable convergence implies convergence in distribution; it is enough to take the entire Ω in the definition of U -stable convergence. See for example [AE78] or [HL15].
We will apply the above proposition to the process {ψ n (θ * ), n ≥ 0} such that Assumption M, R8 and R9 are satisfied. To this end, let us define the triangular array {D n,j , 1 ≤ j ≤ n, n ≥ 1} as and let us take F n,j = F j . First, note that E θ * [ψ j (θ * )|F j−1 ] = 0, so that for any n ≥ 1, {D n,j , 1 ≤ j ≤ n} is a martingale difference process with respect to {F j , 0 ≤ j ≤ n}. Next, R9 implies that D * n := sup 1≤j≤n 1 √ n |ψ j (θ * )| converges in L 1 to 0. Finally, stationarity, R8 and ergodicity guarantee that The limit I(θ * ) = E θ * [ψ 1 (θ * )ψ T 1 (θ * )] is positive semi-definite, and it is deterministic, so that it is measurable with respect to any σ-algebra. Therefore, applying Proposition 22 and Remark 23 we obtain the next result Proposition 24. Assume that Assumption M, R8, and R9 are satisfied. Then,
Proposition 25. Assume that R4-R6 are satisfied. Then, for any fixed n ≥ 1, and for any random vectors θ, θ 1 , θ 2 that are F n−1 measurable and take values in Θ, we have Proof. We will only show that (28) is true. The validity of the remaining inequalities can be proved similarly. Also, without loss of generality, we assume that d = 1.
From here, using the usual limiting argument we conclude that (28) holds true for any F n−1 measurable random variable θ.
In the rest of this section we will verify that Assumption M and properties R0-R8 are satisfies in Example 15.
It is clear that the Markov chain {Z n , n ≥ 0}, as defined in Example 15, satisfies (i) and (iii) in Assumption M. Next we will show that Z is a positive Harris chain (see Definition 20). For any Borel set B ∈ B(R) with strictly positive Lebesgue measure, and any z 0 ∈ R, we have that and thus Z satisfies Definition 20.(a). Also, since the density (with respect to Lebesgue measure) of Z 1 is then Z 1 ∼ N (μ * , (σ * ) 2 ), and consequently, we get that Z n ∼ N (μ * , (σ * ) 2 ) for any n ≥ 0. This implies that N (μ * , (σ * ) 2 ) is an invariant distribution for Z. Thus, Z is a positive Harris chain, and respectively, by Remark 21, Z is an ergodic process. As far as propreties R0-R8, we fist note that We denote by Y n := Z n − ρZ n−1 − (1 − ρ)μ, and we immediately conclude that From here, and using the fact that Θ is bounded, it is straightforward, but tedious, 9 to show that R4, R5, and R6 are satisfied. Also, it is clear that R0 is true, and using (32) by direct computations we get that R1 and R2 are satisfied. Again by direct evaluations, we have that which is positive definite matrix, and thus R7 is satisfied. Since then θ * = (μ * , (σ * ) 2 ) is clearly the unique point at which left hand side of (33) vanishes.
From R2, we see that E θ * [ψ 1 (·)] is the derivative of E θ * [π 1 (·)] which is a continuously differentiable real-valued function. Then, the limit points of (6) are stationary points. By R3, we have that the only stationary point of (6) is θ * . Therefore, we conclude thatθ converges to θ * almost surely in P θ * .
In order to prove (20), we note that according to Theorem 9 we havê Next, Proposition 24 implies that Consequently, since by (42) G n P θ * − − → I −1 (θ * ), using Slutsky's theorem we get The proof is complete.