Predictive density construction and accuracy testing with multiple possibly misspecified diffusion models

Abstract

This paper develops tests for comparing the accuracy of predictive densities derived from (possibly misspecified) diffusion models. In particular, we first outline a simple simulation-based framework for constructing predictive densities for one-factor and stochastic volatility models. We then construct tests that are in the spirit of Diebold and Mariano (1995) and White (2000). In order to establish the asymptotic properties of our tests, we also develop a recursive variant of the nonparametric simulated maximum likelihood estimator of Fermanian and Salanié (2004). In an empirical illustration, the predictive densities from several models of the one-month federal funds rates are compared.

Introduction

Correct specification of models describing dynamics of financial assets is crucial for everything from pricing bonds and derivative assets to designing appropriate hedging strategies. Hence, it is of little surprise that there has been considerable attention given to the issue of testing for the correct specification of diffusion models. In this paper, we do not construct specification tests in the usual sense, but instead assume that all models are (possibly) misspecified and outline a simulation-based methodology for comparing the accuracy of predictive densities based on alternative models.

To place this paper in the correct historical context, note that a first generation of specification testing papers, initiated by the work of Aït-Sahalia (1996), compares the marginal densities implied by hypothesized null models with nonparametric estimates thereof, for the case of one-factor models (see also Pritsker (1998) and Jiang (1998)). While one-factor models may in some cases provide a reasonable representation for short-term interest rates, there is a somewhat widespread consensus that stock returns and term structures are better modeled using multifactor diffusions. To take this into account, Corradi and Swanson (2005a) outline a test for comparing the cumulative distribution (marginal or joint) implied by a hypothesized null model with the corresponding empirical distribution. Their test can be used in the context of multidimensional and/or multifactor models. Needless to say, tests based on the comparison of marginal distributions have no power against iid alternatives with the same marginal, while tests based on the comparison of joint distributions do not suffer from this problem. Nevertheless, correct specification of the joint distribution is not equivalent to that of the conditional; and hence focus in the literature now centers on comparing conditional distributions. When considering conditional distributions, a key difficulty that arises stems from the fact that knowledge of the drift and variance terms of a diffusion process does not in turn imply knowledge of the transition density, in general. Indeed, if the functional form of the transition density were known, one could test the hypothesis of correct specification of a diffusion via the probability integral transform approach of Diebold et al. (1998); the cross-spectrum approach of Hong (2001), Hong et al. (2002) and Hong and Li (2005); the martingalization-type Kolmogorov test of Bai (2003); or via the normality transformation approaches of Bontemps and Meddahi (2005) and Duan (2003). Furthermore, for the case in which the transition density is unknown, tests could be constructed by comparing the kernel (conditional) density estimator of the actual and simulated data, as in Altissimo and Mele (2009) and Thompson (2008); by comparing the conditional distribution of the simulated and of the historical data, as in Bhardwaj et al. (2008); or by using the approaches of Aït-Sahalia (2002) and Aït-Sahalia et al. (2009), where closed form approximations of conditional densities under the null are compared with data-driven kernel density estimates.

All of the papers cited above deal with testing for the correct specification of a given diffusion model. Nevertheless, and as alluded to above, we believe that all models are probably best viewed as approximations of reality and, thus, are likely to be misspecified. Therefore, we focus on choosing the “best” model from amongst (multiple) misspecified alternatives. Moreover, the “best” model is selected by constructing tests that compare both predictive densities and/or predictive conditional confidence intervals associated with alternative models.

Our approach is to measure accuracy using a distributional generalization of mean square error, as defined in Corradi and Swanson (2005b). Namely, let $F_k^{\tau }\left(u\right|X_t,{\theta }_k^{†})$ be the distribution of $X_{t+\tau }$ given $X_t$, evaluated at $u$, implied by diffusion model $k$, and let $F_0^{\tau }\left(u\right|X_t,{\theta }_0)$ be the distribution associated with the underlying and unknown “true” model. Now, choose model $k$ over model 1, say, if $E\left({(F_k^{\tau }\left(u\right|X_t,{\theta }_k^{†})-F_0^{\tau }\left(u\right|X_t,{\theta }_0))}^2\right)<E\left({(F_1^{\tau }\left(u\right|X_t,{\theta }_1^{†})-F_0^{\tau }\left(u\right|X_t,{\theta }_0))}^2\right)$. Our tests can be viewed as distributional generalizations of both Diebold and Mariano (1995) and White (2000). Note that if we knew $F_k^{\tau }\left(u\right|X_t,{\theta }_k^{†})$ in closed form, then we could proceed as in Corradi and Swanson, 2006a, Corradi and Swanson, 2006b. However, the functional form of the model implied conditional distribution is unknown in closed form, in general, and hence we rely on a simulation-based approach to facilitate testing. As is customary in the out-of-sample evaluation literature, the sample of $T$ observations is split into two subsamples, such that $T=R+P$, where only the last $P$ observations are used for predictive evaluation. We first simulate $P-\tau$$\tau$-step ahead paths, using $X_R,\dots ,X_{R+P-\tau }$ as starting values. Then, a scaled difference between the conditional distribution, estimated with historical as well as simulated data, is used to construct our test statistic. One complication that arises in this setup is that for the case of stochastic volatility (SV) models, the initial value of the volatility process is unobserved. To overcome this problem, it suffices to simulate the process using different random initial values for the volatility process. Thereafter, one simply constructs the empirical distribution of the asset price process for any given initial value of the volatility process and takes an average over the latter. This integrates out the effect of the volatility initial value.

The limiting distributions of the suggested statistics are shown to be (functionals of) Gaussian processes with covariance kernels that reflect the contribution of recursive parameter estimation error. In order to provide asymptotically (first-order) valid critical values, we introduce a new bootstrap procedure that mimics the contribution of parameter estimation error in a recursive setting. This is achieved by establishing consistency and asymptotic normality of nonparametric simulated quasi maximum likelihood (NPSQML) estimators of (possibly misspecified) diffusion models, in a recursive setting, and by establishing the first-order validity of their bootstrap analogs.

Of final note is that we test the same null hypothesis as Corradi and Swanson (2006a), and we estimate empirical conditional distributions using both historical and simulated data, as in Bhardwaj et al. (2008). However, there are many differences between those papers and this one. Five such differences are the following. First, we show the asymptotic equivalence of recursive NPSQMLE (Nonparametric Simulated Quasi Maximum Likelihood Estimators) and recursive QMLE. Second, we show the asymptotic equivalence of recursively estimated NPSQMLE and recursive QMLE for partially unobservable multidimensional diffusions (e.g. for stochastic volatility models). This extends in a non-trivial manner the NPSQMLE of Fermanian and Salanié (2004). Third, we establish the first order validity of bootstrap critical values for recursive NPSQMLE, in the case of both observable and partially unobservable diffusions. To the best of our knowledge, there are no available results on bootstrapping NPSQMLE. Fourth, we allow for jumps in the return process, and we recursively estimate the intensity and the parameters of the jump size density. Finally, we develop Diebold–Mariano type Reality Check tests for cases where (a) the CDF is not known in closed form, and (b) data are generated by partially unobservable jump diffusion processes.

The rest of the paper is organized as follows. In Section 2, we define the setup. Section 3 outlines the testing procedure for choosing between $m⩾2$ models and establishes the asymptotic properties thereof. In Section 4, we develop a recursive version of the NPSQML estimator of Fermanian and Salanié (2004) and outline conditions under which asymptotic equivalence between NPSQML and the corresponding recursive QMLE obtains. An empirical illustration is provided in Section 5, in which various models of the effective federal funds rate are compared. All proofs are collected in an Appendix. Hereafter, let $P^{\ast }$ denote the probability law governing the resampled series, conditional on the (entire) sample, let $E^{\ast }$ and ${\mathrm{V\; ar}}^{\ast }$ denote the mean and variance operators associated with $P^{\ast }$. Further, let $o_P^{\ast }\left(1\right)Pr-P$ denote a term converging to zero in $P^{\ast }$-probability, conditional on the sample except a subset of probability measure approaching zero. Finally, let $O_P^{\ast }\left(1\right)$$Pr-P$ denote a term which is bounded in $P^{\ast }$-probability, conditional on the sample, and for all samples except a subset with probability measure approaching zero.