Construction of an informative hierarchical prior for a small sample with the help of historical data and application to electricity load forecasting

Abstract

We are interested in the estimation and prediction of a parametric model on a short dataset upon which it is expected to overfit and perform badly. To overcome the lack of data (relatively to the dimension of the model), we propose the construction of an informative hierarchical Bayesian prior based on another longer dataset which is assumed to share some similarities with the original, short dataset. We illustrate the performance of our prior on simulated datasets from two standard models. We then apply the methodology to a working model for the electricity load forecasting on real datasets, where it leads to a substantial improvement of the quality of the predictions.

Appendix

Using the notation $M_{i\bullet }$ for the $i$th row of a matrix $M$, the non-linear model described in (6) can be re-written in the following condensed way: for $t = 1,\ldots ,N,$

$$\begin{aligned} y_t&= (A_{t\bullet } \alpha ) (B_{t\bullet } \beta + C_{t}) + \gamma (T_t - u)1\!\!1_{[T_t,\,+\infty [}(u) + \epsilon _t. \nonumber \\&= f_t(\theta ) + \epsilon _t \end{aligned}$$

(7)

The matrices $A$ of size $N\times d_A$, $B$ of size $N\times d_{\beta }$, $C$ of size $N\times 1$, and $T$ of size $N\times 1$ are known exogenous variables while the parameters of the model to be estimated are

$$\begin{aligned} \eta = (\theta ,\sigma ^2) = (\alpha , \beta , \gamma , u, \sigma ^2) \in \mathbb {R}^{d_\alpha } \times B_+^{d_\beta }(0, 1) \times \mathbb {R}^* \times [\underline{u},\,\overline{u}] \times \mathbb {R}_+^*, \end{aligned}$$

where $B_+^{d_{\beta }}(0, 1) = \{ \beta \in (\mathbb {R}_+)^{d_{\beta }} ;\; \Vert \beta \Vert _1 \le 1 \}$ is the positive quadrant of the $\Vert \cdot \Vert _1$-unit ball of dimension $d_{\beta }$.

Proposition 1

For $(\beta , u) \in B_+^{d_{\beta }}(0, 1) \times [\underline{u},\,\overline{u}]$ denote $A_*(\beta , u)$ the matrix whose rows are

$$\begin{aligned}(A_*)_{t\bullet }(\beta , u) = \left[ ( B_{t\bullet }\beta + C_{t})A_{t\bullet }, (T_{t}-u)1\!\!1_{[T_{t},\,+\infty [}(u)\right] , \quad t=1,\ldots ,N,\end{aligned}$$

and suppose $A_*^\prime (b, u) A_*(b, u)$ has full rank for every $(\beta , u) \in B_+^{d_{\beta }}(0, 1) \times [\underline{u},\,\overline{u}]$. Assume furthermore that $N>d_\alpha +1$ and that $(y_1,\ldots ,y_N)$ are observations coming from the model (7). The posterior measure corresponding to the informative prior designed in (4) is then a well-defined (proper) probability distribution.

Proof

First notice that $\int {\pi (\theta , k, l, q, r | y)} \,\mathrm {d}\sigma ^{2}$ is proportional to

$$\begin{aligned} \Vert y - f(\theta )\Vert _2^{-N} 1\!\!1_{[0,\,1]}(\Vert \beta \Vert _1)1\!\!1_{[\underline{u},\,\overline{u}]}(u) \pi (\theta |k, l)\pi (k | q, r)\pi (l)\pi (q)\pi (r), \end{aligned}$$

for almost every $y$ and that the function $\theta \mapsto \Vert y - f(\theta )\Vert _2^{-N}$ is bounded, for almost every $y$. The posterior integrability is hence trivial as long as $\pi (\theta |k, l)\pi (k | q, r)\pi (l)\pi (q)\pi (r)$ itself is a proper distribution which is the case here. $\square$

Proposition 2

Under the same assumptions as in Proposition 1, the posterior measure corresponding to the non-informative prior $\pi (\theta , \sigma ^2) \propto \sigma ^{-2}$ is also a well-defined (proper) probability distribution.

Proof

Notice first that

$$\begin{aligned} \int {\pi (\theta , \sigma ^2 | y)} \,\mathrm {d}\sigma ^2&\propto \Vert y - f(\theta )\Vert _2^{-N} 1\!\!1_{[0,\,1]}(\Vert b\Vert _1)1\!\!1_{[\underline{u},\,\overline{u}]}(u) \quad \text {for almost every } y, \end{aligned}$$

and observe then that

$$\begin{aligned} \Vert y - f(\theta )\Vert _2^2&= \sum _{t=1}^N \left[ y_{t} - (B_{t\bullet }\beta +C_{t}) A_{t\bullet }\alpha - (T_{t}-u)1\!\!1_{[T_{t},\,+\infty [}(u)\gamma \right] ^2. \end{aligned}$$

Let $(\beta _0, u_0) \in B_+^{d_{\beta }}(0, 1) \times [\underline{u},\,\overline{u}]$ and denote $\alpha _* = (\alpha , \gamma )$. We write

$$\begin{aligned} \Vert y\! -\! f((\alpha ,\beta _0,\gamma ,u_0))\Vert _2^2&\!=\! \sum _{t=1}^N \left[ y_{t} \!-\! (B_{t\bullet }\beta _0+C_{t}) A_{t\bullet }\alpha - (T_{t}-u_0)1\!\!1_{[T_{t},\,+\infty [}(u_0)\gamma \right] ^2 \\&= \Vert y - A_*(\beta _0, u_0) \alpha _*\Vert _2^2, \end{aligned}$$

and thus obtain the following equivalence, as $(\beta , u)\rightarrow (\beta _0,u_0)$ and $\Vert \alpha _*\Vert _2\rightarrow +\infty$

$$\begin{aligned} \Vert y - f(\theta )\Vert _2^{-N} \sim \Vert y - A_*(\beta _0,u_0) \alpha _*\Vert _2^{-N}. \end{aligned}$$

(8)

The triangular inequality applied to the right-hand side of (8) gives

$$\begin{aligned} \Vert y - A_*(\beta _0,u_0) \alpha _*\Vert _2^{-N}&\le \big | \Vert y\Vert _2 - \Vert A_*(\beta _0,u_0) \alpha _*\Vert _2 \big |^{-N}. \end{aligned}$$

(9)

Since $A_*^\prime (\beta _0,u_0) A_*(\beta _0,u_0)$ has full rank, by straightforward algebra we get

$$\begin{aligned} \lambda \Vert \alpha _*\Vert _2^2&\le \Vert A_* (\beta _0,u_0)\alpha _*\Vert _2^2, \end{aligned}$$

where $\lambda$ is the smallest eigenvalue $(A_*(\beta _0,u_0))^\prime A_*(\beta _0,u_0)$ and is strictly positive. We can hence find an equivalent of the right-hand side of (9) as $\Vert \alpha _*\Vert _2\rightarrow +\infty$, which is

$$\begin{aligned} \big | \Vert y\Vert _2 - \Vert A_*(\beta _0,u_0) \alpha _*\Vert _2 \big |^{-N} \sim \lambda ^{-N/2} \Vert \alpha _*\Vert _2^{-N}. \end{aligned}$$

(10)

Combining (8), (9) and (10) together, we see that the integrability of the left-hand side of (8) as $(\beta , u)\rightarrow (\beta _0,u_0)$ and $\Vert \alpha _*\Vert _2\rightarrow +\infty$ is directly implied by that of $\Vert \alpha _*\Vert _2^{-N}$. The latter is immediate for $N > d_\alpha +1$, as can be seen via a Cartesian to hyperspherical re-parametrisation.

The previous paragraph thus ensures the integrability of $\Vert y - f(\theta )\Vert _2^{-N}$ over sets of the form

$$\begin{aligned} \{(\beta ,u)\in V(\beta _0, u_0), \Vert \alpha _*\Vert _2 \in ]M(\beta _0,u_0),\,+\infty [\}, \quad \forall (\beta _0, u_0) \in B_+^{d_{\beta }}(0, 1) \times [\underline{u},\,\overline{u}] \end{aligned}$$

where the subset $V(b_0, u_0)$ is an open neighbourhood of $(\beta _0, u_0)$ and $M(\beta _0, u_0)$ is a real number depending on $(\beta _0, u_0)$. By compactness of $B_+^{d_{\beta }}(0, 1) \times [\underline{u},\,\overline{u}]$ there exists a finite union of such $V(\beta _i, u_i)$ that covers $B_+^{d_{\beta }}(0, 1) \times [\underline{u},\,\overline{u}]$. Denoting $M$ the maximum of $M(\beta _i, u_i)$ over the corresponding finite subset of $(\beta _i, u_i)$, we finally obtain the integrability of $\Vert y - f(\theta )\Vert _2^{-N}$ over $\{(\beta ,u)\in B_+^{d_{\beta }}(0, 1), \Vert \alpha _*\Vert \in ]M,\,+\infty [\}$.

The integrability of $\Vert y - f(\theta )\Vert _2^{-N}$ over $\{(\beta ,u)\in B_+^{d_{\beta }}(0, 1), \Vert \alpha _*\Vert \in [0,\,M]\}$ is trivial, recalling that $\theta \mapsto \Vert y - f(\theta )\Vert _2$ is continuous and does not vanish over this compact for almost every $y$, meaning that its inverse shares these same properties.   $\square$

Remark 1

The condition “$A_*^\prime A_*$ has full rank” mentioned above is typically verified in our applications for the regressors used in our model. To see this, call “vector of heating degrees” the vector whose coordinates are $(T_{t}-u)1\!\!1_{[T_{t},\,+\infty [}(u)$. Then not satisfying the aforementioned condition is equivalent to saying that there exists an index $i$ and a threshold $u$ such that the family of vectors formed by the regressors $A$ and the vector of heating degrees is linearly dependent over the subset $\Psi _i$ of the calendar”.