Demand side management for a residential customer in multi-energy systems

https://doi.org/10.1016/j.scs.2016.01.010Get rights and content

Highlights

  • Fully automated EMS based on a reinforcement learning (RL) is introduced.

  • Focusing on the modeling of a house as a multi-energy system to be adaptable with RL.

  • Household load is optimally managed in the stochastic environment via RL algorithm.

  • RL algorithm reduces household energy bill up to 20%.

Abstract

Today, as a consequence of the growing installation of efficient technologies, e.g. micro-combined heat and power (micro-CHP), the integration of traditionally separated electricity and natural gas networks has been attracting attentions from researchers in both academia and industry. To model the interaction among electricity and natural gas networks in distribution systems, this paper models a residential customer in a multi-energy system (MES). In this paper, we propose a fully automated energy management system (EMS) based on a reinforcement learning (RL) algorithm to motivate residential customers for participating in demand side management (DSM) programs and reducing the peak load in both electricity and natural gas networks. This proposed EMS estimates the residential customers’ satisfaction function, energy prices, and efficiencies of appliances based on the residential customers’ historical actions. Simulations are performed for the sample model and results depict how much of each energy, i.e. electricity and natural gas, the residential customer should consume and how much of natural gas should be converted in order to meet electricity and heating loads. It is also shown that the proposed RL algorithm reduces residential customer energy bill and electrical peak load up to 20% and 24%, respectively.

Introduction

CHP systems are getting increased interest in a growing number of countries. Today, the concentration of micro-CHP market is in mission-critical operations like research institutions, hospitals, and data centers. However, some manufacturers with aid of governments, proved that application of micro-CHP in home would be a financially viable option (Cho, Luck, Eksioglu, & Chamra, 2009).

The market of domestic micro-CHP is growing rapidly (Ellamla, Staffell, Bujlo, Pollet, & Pasupathi, 2015). An estimated 138,000 fuel cell micro-CHP systems had been installed in Japan by the end of 2014 (Ellamla et al., 2015). The Japanese government set a target of 1.4 million micro-CHP systems; South Korea's target is 1 million and the European Union's target is 50,000 micro-CHP systems installed by 2020 (Ellamla et al., 2015).

By raising the use of CHP systems in a near future, synergy effects of coupling between electricity and natural gas networks draw researchers’ attention to propose an integrated picture of these two physically separated networks (Geidl et al., 2007). To have this integrated picture and in order to model the interaction of these two networks properly in Mancarella (2014), an appropriate framework for a multi-energy system (MES) has been proposed.

As a simple definition, MES is whereby electricity, heating, fuels, and other type of energies optimally interact with each other at various levels. This approach of modeling creates an important opportunity to increase technical, financial, and environmental performance relative to conventional energy systems where energy systems, e.g. natural gas and electricity, are treated independently.

Along with the development of MES, smart grid concepts have been also grown. To realize smart grid concepts, demand side management (DSM) programs play a leading role. DSM commonly refers to the methods implemented by utility companies for reducing or shifting the energy consumptions at the residential customers’ side (U.S. Federal Energy Regulatory Commission, 2008).

To implement DSM techniques, the residential customer should be equipped with bidirectional communication system. There are two main approaches in DSM; direct load control (Gomes et al., 2007, Wang et al., 2011) and smart pricing techniques (Mohsenian-Rad et al., 2010, Samadi et al., 2012). In the first method, utilities are authorized to remotely control the residential customers’ energy consumptions. Alternatively in smart pricing, utilities encourage residential customers to voluntarily manage their loads, e.g. by reducing their consumptions at peak hours on a real time pricing (RTP) market. As there are not direct external forces in the latter approach, residential customers individually decide whether they participate in the DSM program or not. Therefore in this scheme, residential customers can manage their satisfaction and keep it in a standard level.

DSM in MES was introduced in Sheikhi et al., 2014, Sheikhi et al., 2015a which was called integrated DSM (IDSM). In this approach, consumers can communicate with both networks simultaneously.

In MES, for controlling and optimizing the consumption of a price taker residential customer who has no considerable effect on electricity price, different methods have been introduced and discussed (Alejandro et al., 2014, Ameri and Besharati, 2016, Mancarella and Chicco, 2013a, Mancarella and Chicco, 2013b, Martínez Ceseña et al., 2015, Pazouki et al., 2014). Mancarella and Chicco (2013a) present comprehensive techno-economic methodologies for the quantification of DSM from small commercial and residential end-users. A simple mixed integer programming (MIP) method is presented in Mancarella and Chicco (2013b), to optimally manage both heating and electrical loads in MES based on the residential customer payment cost objective function. In Martínez Ceseña et al. (2015), an analysis has been outlined to address the potential of exploiting CHP systems in providing DSM. Ameri and Besharati (2016) proposed a mixed integer linear programming (MILP) method determine the optimal capacity and operation of seven combined cooling, heating and power (CCHP) systems in the heating and cooling network of a residential district. In Pazouki et al. (2014), a Monte Carlo algorithm is applied to minimize operation costs and maximize reliability of a residential customer in a stochastic MES environment. In Alejandro et al. (2014), the optimal operation of an integrated electricity and natural gas infrastructure is presented by applying a model predictive system control approach. Regarding residential building demand side management, Rastegar and Fotuhi-Firuzabad (2015) presents a residential energy hub model for a smart multi-carrier energy home. In this reference, an optimization-based program is proposed to determine the optimal operation mode of the energy hub with household responsive demand in response to day-ahead predetermined tariffs of electricity. In order to prediction of building energy consumption, Brohus, Frier, Heiselberg, and Haghighat (2012) presents a new approach that quantifies the uncertainty of building energy consumption by means of stochastic differential equations (SDE). To optimization of thermal comfort and energy consumption in a residential house (Magnier & Haghighat, 2010) described an optimization methodology based on a combination of an Artificial Neural Network and a multi-objective evolutionary algorithm. Regarding the forecasting models, Nguyen, Reiter, and Rigo (2014) shows that the metaheuristic search algorithms (e.g. genetic algorithm and particle swarm optimization) are the most popular algorithmic technique applied to building optimization problem. Harish and Arun Kumar (2016) and Li and Wen (2014) review all the significant modeling methodologies which have been developed and adopted to model the energy systems of buildings categorically and majorly focuses on the recent studies which involved development of the control strategies by modeling the building energy systems.

Based on above mentioned approaches, there are three main shortcomings when it comes to energy management system (EMS) programming in the real world as follows:

  • 1.

    Energy system parameter definition: Almost all approaches, which are applied to optimize the energy consumption of residential customers, require modeling a building in detail which it is not be suitable for on-line operation optimization. In these approaches (Harish and Arun Kumar, 2016, Li and Wen, 2014) it is required to define some specific parameters, e.g. appliances efficiencies and energy prices, as constant (Rastegar & Fotuhi-Firuzabad, 2015); however, such an assumption would not necessary hold accurate in practice. For instance, appliances efficiencies vary with time and energy prices are intrinsically stochastic.

  • 2.

    Considering MES framework: there are some papers have been published recently considering household behavior and modeling uncertainty in predicting building energy consumption (Brohus et al., 2012, Magnier and Haghighat, 2010). Although there are valuable papers but it is required to have a holistic view and consider household in MES framework.

  • 3.

    Objective function modeling in EMS: In the past, dissatisfaction function has either been overlooked (Rastegar & Fotuhi-Firuzabad, 2015) or included, if at all, in an inaccurate fashion (Maharjan, Zhu, Zhang, Gjessing, & Basar, 2013). Without factoring in dissatisfaction function, EMS cannot deal with real world conditions. On the other hand, if modeled in a one-size-fits-all manner, dissatisfaction function does not represent the behavior of each individual because different users have different priorities.

Therefore, without tackling two mentioned deficiencies, EMS programming will arguably not lead to the accurate results in real life.

In this paper, to address these concerns, we formulate energy prices and residential customers’ loads as Markov model. We use reinforcement learning (RL) algorithm to allow the EMS learns behaviors of the residential customer, the uncertainty of energy prices, and appliances’ efficiencies to make an optimal decision.

RL is one of the machine learning techniques proposing to solve real-world problems with a dynamic environment such as electricity, heating, cooling loads, and price fluctuations without taking any system components parameters. RL differs from standard supervised learning (Nguyen et al., 2014) in that correct input/output pairs are never presented, nor sub-optimal actions explicitly corrected. Further, there is a focus on on-line performance, which involves finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge). In RL a learner is adapted to an unknown and dynamic environment by receiving rewards and punishments from the environment. Daneshfar and Bevrani (2010) and Giupponi, Agusti, Pérez-Romero, and Sallent (2005) show the application of RL in different issues.

Here, we consider the smart pricing scheme, where each user, as a price taker consumer, has been equipped with an EMS programming based on RL disciplines to minimize the electricity and natural gas bills, simultaneously. This paper firstly focus on the modeling of a house as a MES, considering different electrical and heating loads, micro-CHP, and a PHEV in an appropriate way to be adaptable with RL algorithm. Then, operation of the MES is optimized and the household load is optimally managed in the stochastic environment. The objective function is to minimize the residential customer energy payment cost and dissatisfaction level which incurs during shifting of loads in response to the time varying prices of electricity and natural gas. Solving the proposed problem determines the amount of energies following to each converter and device, the charging strategy of PHEV, and the energy consumption scheduling of all appliances at any hours. Due to curse of dimensionality in solving the optimization problem via RL algorithm at houses, we decompose this problem into clusters and use a “state reduction technique” to make the algorithm more applicable in practice.

The rest of this paper is organized as follows. The model of the price taker residential customer in MES is described in Section 2. In Section 3, the mathematical models for the loads, PHEV battery, and energy prices are given. EMS system framework and the formulation of RL algorithm is presented in Sections 4 Energy management system framework, 5 RL algorithm vs DP, respectively. Simulation results are presented in Section 6. Finally, the paper is concluded in Section 7.

Section snippets

Price taker residential customer in multi-energy systems

A price taker residential customer participates in DSM program without any communications with other ones. In fact, the price taker residential customers believe that their energy consumptions cannot affect the energy prices. Therefore, energy prices are assumed as exogenous signals for this type of residential customers.

The residential customer can connect to MES, e.g. natural gas and electricity, by means of various conversion and storage devices. A simple scheme of residential customer's

Mathematical model of stochastic environment

The EMS selects its actions based on the state of the environment as depicted in Fig. 4. Here, the environment includes a price-taker residential customer in MES. The definition of action and state signals are given in the next section.

In this study, the environment is a stochastic system. It means that the subsequent state of environment is unpredictable due to the influence of random variables. Here, these random variables are energy prices and consumption patterns of the residential customer.

Energy management system framework

EMS tries to minimize its cost function through the action signal. Here, the action signal has this vector: u(t)=uWM(t),uDW(t),uCHP(t),ueP(t)T. By means of action signal, the conversion devices and appliances are controlled. The action signal of CHP is uCHP(t) which control CHP generation level based on (16):PgCin(t)=uCHP(t)PgCin¯

As Fig. 9 shows, EMS finds the optimal policy which determines the probability of taking action u in state. Here, the state signal at each time slot is the vector s(

RL algorithm vs DP

DP algorithm requires the transition probability of environment. Moreover, it requires environment parameters, e.g. appliances efficiency, conversion devices parameters, and online electricity and natural gas prices. To practically find the optimal policy in a dynamic environment with numerous parameters that change all the time, we have to apply an algorithm that works with few parameters and adapts itself during ongoing changes. An EMS that is programmed based on Reinforcement learning (RL)

Performance evaluation

To demonstrate the effectiveness of the proposed algorithm, it is applied to a price taker residential customer in MES. The system is simulated in MATLAB Simulink-R2010a®, in the real time and stochastic environment. We assume that natural gas and electricity prices are determined by the utility companies. In this market, the EMS solves the optimization problem just by considering the historical data.

Conclusions

In this paper, we introduce a modified model of residential customer in a MES. We show that residential customers in MES can be encouraged to participate in DSM programs by both shifting their energy usage and changing their energy resources. We tackle the problem of designing optimal DSM program for a price taker residential customer who has to deal with varying component parameters, e.g. efficiency by applying RL algorithm. To evaluate the performance of the proposed method, a residential

References (38)

  • M. Rastegar et al.

    Load management in a residential energy hub with renewable distributed energy resources

    Energy and Buildings

    (2015)
  • A. Sheikhi et al.

    An autonomous demand response program for electricity and natural gas networks in smart energy hubs

    Energy

    (2015)
  • W. Yu et al.

    Modeling electricity markets with hidden Markov model

    Electric Power Systems Research

    (2006)
  • J. Alejandro et al.

    Combined environmental and economic dispatch of smart grids using distributed model predictive control

    International Journal of Electrical Power & Energy Systems

    (2014)
  • H. Cho et al.

    Cost optimized real time operation of CHP system

    Energy and Buildings

    (2009)
  • F. Daneshfar et al.

    Load–frequency control: a GA-based multi-agent reinforcement learning

    IET Generation, Transmission & Distribution

    (2010)
  • X. Fang et al.

    Smart grid—The new and improved power grid: A survey”

    IEEE Communications Surveys & Tutorials

    (2011)
  • M. Geidl et al.

    Energy hubs for the futures

    IEEE Power & Energy Magazine

    (2007)
  • L. Giupponi et al.

    A novel joint radio resource management approach with reinforcement learning mechanisms

  • Cited by (0)

    View full text