Demand Response Management in Smart Grid Networks: a Two-Stage Game-Theoretic Learning-Based Approach

Abstract

In this paper, the combined problem of power company selection and demand response management (DRM) in a smart grid network consisting of multiple power companies and multiple customers is studied via adopting a reinforcement learning and game-theoretic technique. Each power company is characterized by its reputation and competitiveness. The customers, acting as learning automata select the most appropriate power company to be served, in terms of price and electricity needs’ fulfillment, via a reinforcement learning based mechanism. Given customers’ power company selection, the DRM problem is formulated as a two-stage game-theoretic optimization framework. At the first stage the optimal customers’ electricity consumption is determined and at the second stage the optimal power companies’ pricing is obtained. The output of the DRM problem feeds the learning system to build knowledge and to conclude to the optimal power company selection. To realize the aforementioned framework a two-stage Power Company learning selection and Demand Response Management (PC-DRM) iterative algorithm is introduced. The performance evaluation of the proposed approach is achieved via modeling and simulation and its superiority against other approaches is illustrated.

Appendices

Appendix 1

1.1 Proof of Theorem 1

Towards determining customer’s best response strategy $B{R}_i\left({\mathrm{e}}_{-\mathrm{i}}^{\left(\mathrm{t}\right)}\right)={e}_{i,j}^{(t)\ast }$, the first and the second order derivatives of customer’s utility function ${U}_i^{(t)}\left({\mathrm{e}}^{\left(\mathrm{t}\right)}\right)$with respect to ${e}_{i,j}^{(t)}$ are used.

$$ \frac{\partial {U}_i^{(t)}\left({\mathrm{e}}^{\left(\mathrm{t}\right)}\right)}{\partial {e}_{i,j}^{(t)}}=\frac{1}{E_{-i}^{(t)}}\cdot \left[{s_i}^{\prime}\left({r}_i^{(t)}\right)-{a}_i^{(t)}\cdot {p}_j^{(t)}\right] $$

(14)

$$ \frac{\partial^2{U}_i^{(t)}\left({\mathrm{e}}^{\left(\mathrm{t}\right)}\right)}{\partial {e_{i,j}^{(t)}}^2}=\frac{1}{{E_{-i}^{(t)}}^2}\cdot {s_i}^{\prime^{\prime }}\left({r}_i^{(t)}\right) $$

(15)

As stated in Section 2.2, customer’s satisfaction function ${s}_i\left({r}_i^{(t)}\right)$ is an increasing concave function with respect to ${r}_i^{(t)}$, thus ${s_i}^{\prime^{\prime }}\left({r}_i^{(t)}\right)<0$ and $\frac{\partial^2{U}_i^{(t)}\left({\mathrm{e}}^{\left(\mathrm{t}\right)}\right)}{\partial {e_{i,j}^{(t)}}^2}<0$. We set $\tau =\underset{r_i^{(t)}\to \infty }{\lim }{s^{\prime}}^{-1}\left({r}_i^{(t)}\right)$. Since ${s_i}^{\prime}\left({r}_i^{(t)}\right)$ is a strictly decreasing function (due to ${s_i}^{\prime^{\prime }}\left({r}_i^{(t)}\right)<0$) and as ${s_i}^{\prime}\left({r}_i^{(t)}\right)>0$, we know that $\tau <{s_i}^{\prime}\left({r}_i^{(t)}\right)\le {s_i}^{\prime }(0)$ and 0 ≤ τ < s_i^′(0). Hence, for $0\le {a}_i^{(t)}\cdot {p}_j^{(t)}\le \tau$, we have $\frac{\partial {U}_i^{(t)}\left({\mathrm{e}}^{\left(\mathrm{t}\right)}\right)}{\partial {e}_{i,j}^{(t)}}>0$ and thus ${U}_i^{(t)}$ is an increasing function of ${e}_{i,j}^{(t)}$. In this case, the best response strategy for customer is to demand her maximum electricity consumption, i.e., ${e}_{i,j,\max}^{(t)}$. So, for $0\le {a}_i^{(t)}\cdot {p}_j^{(t)}\le \tau$, we have $B{R}_i\left({\mathrm{e}}_{-\mathrm{i}}^{\left(\mathrm{t}\right)}\right)={e}_{i,j,\max}^{(t)}$ for all . For $\tau <{a}_i^{(t)}\cdot {p}_j^{(t)}\le {s_i}^{\prime }(0)$, the equation $\frac{\partial {U}_i^{(t)}\left({\mathrm{e}}^{\left(\mathrm{t}\right)}\right)}{\partial {e}_{i,j}^{(t)}}=0$ is equivalent to ${s_i}^{\prime}\left({r}_i^{(t)}\right)={a}_i^{(t)}\cdot {p}_j^{(t)}\iff$ . Note that as ${s_i}^{\prime}\left({r}_i^{(t)}\right)$ is a strictly decreasing function, its inverse (i.e., ${s_i^{\prime}}^{-1}$) exists, and that ${\widehat{r}}_i^{(t)}$ is a decreasing function of ${a}_i^{(t)}\cdot {p}_j^{(t)}$. Since ${s_i}^{\prime^{\prime }}\left({r}_i^{(t)}\right)<0$ for all ${r}_i^{(t)}$ and hence $\frac{\partial^2{U}_i^{(t)}\left({\mathrm{e}}^{\left(\mathrm{t}\right)}\right)}{\partial {e_{i,j}^{(t)}}^2}<0$, the roots of $\frac{\partial {U}_i^{(t)}\left({\mathrm{e}}^{\left(\mathrm{t}\right)}\right)}{\partial {e}_{i,j}^{(t)}}=0$ maximize ${U}_i^{(t)}\left({\mathrm{e}}^{\left(\mathrm{t}\right)}\right)$ for given electricity consumption of the rest of the customers, i.e., ${E}_{-i}^{(t)}$. An one-to-one relation exists between ${r}_i^{(t)}$ and ${e}_{i,j}^{(t)}$, and thus the best response electricity consumption in response to ${\mathrm{e}}_{-\mathrm{i}}^{\left(\mathrm{t}\right)}$ that maximizes ${U}_i^{(t)}\left({\mathrm{e}}^{\left(\mathrm{t}\right)}\right)$ is also unique and is equal to ${e}_{i,j}^{(t)\ast }={E}_{-i}^{(t)}\cdot {\widehat{r}}_i^{(t)}={E}_{-i}^{(t)}\cdot {s_i^{\prime}}^{-1}\left({a}_i^{(t)}\cdot {p}_j^{(t)}\right)$. If ${e}_{i,j}^{(t)\ast }>{e}_{i,j,\max}^{(t)}$ customer does not request for ${e}_{i,j}^{(t)\ast }$. In this case, since ${e}_{i,j}^{(t)\ast }$ is the unique maximizer of ${U}_i^{(t)}\left({\mathrm{e}}^{\left(\mathrm{t}\right)}\right)$, then ${U}_i^{(t)}\left({\mathrm{e}}^{\left(\mathrm{t}\right)}\right)$ is an increasing function of ${e}_{i,j}^{(t)}$ in ${e}_{i,j}^{(t)}\le {e}_{i,j,\max}^{(t)}\le {e}_{i,j}^{(t)\ast }$ for fixed ${\mathrm{e}}_{-\mathrm{i}}^{\left(\mathrm{t}\right)}$. Therefore, the best response to ${\mathrm{e}}_{-\mathrm{i}}^{\left(\mathrm{t}\right)}$ is the maximum value of customer’s electricity consumption, i.e., $B{R}_i\left({\mathrm{e}}_{-\mathrm{i}}^{\left(\mathrm{t}\right)}\right)={e}_{i,j,\max}^{(t)}$. This implies that for $\tau <{a}_i^{(t)}\cdot {p}_j^{(t)}\le {s_i}^{\prime }(0)$. For ${a}_i^{(t)}\cdot {p}_j^{(t)}>{s_i}^{\prime }(0)$, we have $\frac{\partial {U}_i^{(t)}\left({\mathrm{e}}^{\left(\mathrm{t}\right)}\right)}{\partial {e}_{i,j}^{(t)}}<0$, thus ${U}_i^{(t)}\left({\mathrm{e}}^{\left(\mathrm{t}\right)}\right)$ is a decreasing function of ${e}_{i,j}^{(t)}$. In this case, the imposed price by the companies is extremely high for customers to afford it, thus .

Appendix 2

1.1 Proof of Theorem 3

Given customers’ electricity consumption, the welfare function of each utility company is written as follows.

(16)

Considering the first order derivative of ${W}_j^{(t)}$ with respect to ${p}_j^{(t)}$, we have:

(17)

Thus, the critical points of ${W}_j^{(t)}\left({\mathrm{p}}^{\left(\mathrm{t}\right)}\right)$ are as follows.

(18)

The second order derivative of ${W}_j^{(t)}\left({\mathrm{p}}^{\left(\mathrm{t}\right)}\right)$ is as follows.

(19)

As observed via (19), we have $\frac{\partial^2{W}_j^{(t)}\left({\mathrm{p}}^{\left(\mathrm{t}\right)}\right)}{\partial {p}_j^{(t)2}}<0$, thus ${p}_j^{(t)\ast }$ as determined in Eq. (18) maximizes utility company’s welfare.