Stochastic model specification search for Gaussian and partial non-Gaussian state space models
Introduction
State space models are widely used in time series analysis to deal with processes which gradually change over time. Model specification, however, is a challenge for these models as one has to specify which components to include and to decide whether they are fixed or time-varying. For state space models, like for many other complex models, this often leads to testing problems which are non-regular from the view-point of classical statistics. Thus, a classical approach toward model selection which is based on hypothesis testing such as a likelihood ratio test or information criteria such as AIC or BIC cannot be easily applied, because it relies on asymptotic arguments based on regularity conditions that are violated in this context.
Consider, for example, modelling a time series through the dynamic linear trend model, defined for as: where follows a random walk with a random drift starting from unknown initial values and : A typical specification problem arising for this model is to decide if the drift is time-varying rather than constant. However, testing versus results in a non-regular testing problem, because the null hypothesis lies on the boundary of the parameter space. A similar specification problem is deciding which components are present in this time series model. For instance, is it necessary to include a dynamic drift term or should be removed because the level follows a simple random walk? This is another non-regular problem, because again the null hypothesis can be rephrased as testing .
The Bayesian approach is, in principle, able to deal with such non-regular testing problems. Suppose that different models are considered to be candidates for having generated the time series . In a Bayesian setting each of these models is assigned a prior probability and the goal is to derive the posterior model probability for each model .
There are basically two strategies to cope with the challenge associated with computing the posterior model probabilities. The traditional approach dating back to Jeffreys (1948) and Zellner (1971) determines the posterior model probabilities of each model separately by using Bayes’ rule, , where is the marginal likelihood for model . An explicit expression for the marginal likelihood exists only for conjugate problems like linear regression models with normally distributed errors, whereas for more complex models numerical techniques are required. For Gaussian state space models, marginal likelihoods have been estimated using methods such as importance sampling (Frühwirth-Schnatter, 1995, Durbin and Koopman, 2000), Chib’s estimator (Chib, 1995), numerical integration (Shively and Kohn, 1997) and bridge sampling (Frühwirth-Schnatter, 2001). Recently, Frühwirth-Schnatter and Wagner (2008) considered estimation of the marginal likelihood for non-Gaussian state space models and demonstrated that the resulting estimators can be pretty inaccurate.
The modern approach to Bayesian model selection is to apply model space MCMC methods by sampling jointly model indicators and parameters, using e.g. the reversible jump MCMC algorithm (Green, 1995) or the stochastic variable selection approach (George and McCulloch, 1993, George and McCulloch, 1997). The stochastic variable selection approach is commonly applied to model selection for regression models and aims at identifying non-zero regression effects, but it is useful far beyond this problem. It allows parsimonious covariance modelling for longitudinal data as shown by Smith and Kohn (2002) and covariance selection in random effects models as shown by Chen and Dunson (2003) and Frühwirth-Schnatter and Tüchler (2008).
Shively et al. (1999) present a variable selection approach to non-parametric regression using priors for the unknown functions that are expressed in state space form, however, they did not deal explicitly with time series models. In the present paper we show that such a variable selection approach is useful for dealing with model selection problems in more general state space models.
To perform stochastic model specification search for the dynamic linear trend model defined in (1), (2), (3), for instance, we introduce three binary stochastic indicators in such a way that the unconstrained model corresponds to setting all indicators equal to 1. Reduced model specifications result by setting certain indicators equal to 0. One of those models, for instance, is the local level model, where the drift component completely disappears: Another interesting special case is the linear trend model, where The practical implementation of this approach is innovative in two respects. First, we employ a new prior for the process variances of the state space model by assuming that the square root of each process variance follows a normal distribution centered at 0. It is well-known that variable selection is, in general, sensitive to the choice of the prior, see e.g. Fernández et al. (2001). We show both for simulated as well as for real data that this prior is less influential on posterior inference when the true process variance is close to 0 than the usually applied inverted Gamma prior. This is in line with Gelman (2006) who came to similar conclusions for the related random effects model.
Second, we derive an MCMC method for Gaussian as well as partially Gaussian state space models that performs stochastic model specification search by sampling the indicators simultaneously with the models parameters. The sampler is based on a non-centered parameterization of the state space model which generalizes previous work in this area such as Pitt and Shephard (1999) and Frühwirth-Schnatter (2004). In combination with the normal prior on the square root of each process variance this leads to a Gibbs sampler that is easily implemented. This is in contrast to Shively et al. (1999) who consider a data-based prior which is motivated by the BIC criterion and leads to a sampling scheme where numerical integration has to be performed for each sweep of the MCMC scheme in order to sample indicators and parameters jointly.
To implement this approach for non-Gaussian state space modelling of time series of count data or binary, categorical, or multinomial data we make use of auxiliary mixture sampling (Frühwirth-Schnatter and Wagner, 2006, Frühwirth-Schnatter and Frühwirth, 2007, Frühwirth-Schnatter et al., forthcoming) which is a simple MCMC method for estimating a broad class of discrete-valued models.
Throughout the paper we focus on structural time series models including seasonal components, trend and an intervention effect and apply the method to various well-known time series.
Section snippets
The dynamic linear trend model
Our method is based on a non-centered parameterization of the dynamic linear trend model which is discussed in the next subsection.
The parsimonious basic structural model
In the basic structural model, a seasonal component is added to the dynamic linear trend model discussed in Section 2, see e.g. Harvey (1989): where is the same as in (2), (3) and is the number of seasons. The initial seasonal pattern is given by with . In addition to the model specification problems discussed in Section 2, a decision has to be made if a seasonal pattern is present and if this pattern is
Model selection for non-Gaussian state space models
The investigations in Shively et al. (1999) show that variable selection in state space models is also feasible for binary data. In this section we show how the variable selection approach developed in Sections 2 The dynamic linear trend model, 3 Extension to the basic structural model for Gaussian state space model may be extended to non-normal state space models using auxiliary mixture sampling (Frühwirth-Schnatter and Wagner, 2006, Frühwirth-Schnatter and Frühwirth, 2007, Frühwirth-Schnatter
Concluding remarks
The model space MCMC approach discussed in this paper could be easily adapted to other state space models. An important extension we are investigating currently is searching for fixed and time-varying coefficients in a regression model.
It is possible to extend our approach to several other non-Gaussian time series models. The method works for any partial Gaussian state space model in the sense of Shephard (1994), i.e. for any state space model that is conditionally Gaussian given a set of
Acknowledgements
We thank the anonymous reviewers and the associate editor for numerous useful suggestions and comments.
References (38)
- et al.
Benchmark priors for Bayesian model averaging
Journal of Econometrics
(2001) - et al.
Auxiliary mixture sampling with applications to logistic models
Computational Statistics and Data Analysis
(2007) - et al.
Marginal likelihoods for non-Gaussian models using auxiliary mixture sampling
Computational Statistics and Data Analysis
(2008) - et al.
A Bayesian approach to model selection in stochastic coefficient regression models and structural time series models
Journal of Econometrics
(1997) - et al.
Nonparametric regression using Bayesian variable selection
Journal of Econometrics
(1996) - et al.
A Monte Carlo approach to nonnormal and nonlinear state-space modeling
Journal of the American Statistical Association
(1992) - et al.
On Gibbs sampling for state space models
Biometrika
(1994) - et al.
Consistency of Bayesian procedures for variable selection
The Annals of Statistics
(2009) - et al.
Random effects selection in linear mixed models
Biometrics
(2003) Marginal likelihood from the Gibbs output
Journal of the American Statistical Association
(1995)