Demand side management for a residential customer in multi-energy systems
Introduction
CHP systems are getting increased interest in a growing number of countries. Today, the concentration of micro-CHP market is in mission-critical operations like research institutions, hospitals, and data centers. However, some manufacturers with aid of governments, proved that application of micro-CHP in home would be a financially viable option (Cho, Luck, Eksioglu, & Chamra, 2009).
The market of domestic micro-CHP is growing rapidly (Ellamla, Staffell, Bujlo, Pollet, & Pasupathi, 2015). An estimated 138,000 fuel cell micro-CHP systems had been installed in Japan by the end of 2014 (Ellamla et al., 2015). The Japanese government set a target of 1.4 million micro-CHP systems; South Korea's target is 1 million and the European Union's target is 50,000 micro-CHP systems installed by 2020 (Ellamla et al., 2015).
By raising the use of CHP systems in a near future, synergy effects of coupling between electricity and natural gas networks draw researchers’ attention to propose an integrated picture of these two physically separated networks (Geidl et al., 2007). To have this integrated picture and in order to model the interaction of these two networks properly in Mancarella (2014), an appropriate framework for a multi-energy system (MES) has been proposed.
As a simple definition, MES is whereby electricity, heating, fuels, and other type of energies optimally interact with each other at various levels. This approach of modeling creates an important opportunity to increase technical, financial, and environmental performance relative to conventional energy systems where energy systems, e.g. natural gas and electricity, are treated independently.
Along with the development of MES, smart grid concepts have been also grown. To realize smart grid concepts, demand side management (DSM) programs play a leading role. DSM commonly refers to the methods implemented by utility companies for reducing or shifting the energy consumptions at the residential customers’ side (U.S. Federal Energy Regulatory Commission, 2008).
To implement DSM techniques, the residential customer should be equipped with bidirectional communication system. There are two main approaches in DSM; direct load control (Gomes et al., 2007, Wang et al., 2011) and smart pricing techniques (Mohsenian-Rad et al., 2010, Samadi et al., 2012). In the first method, utilities are authorized to remotely control the residential customers’ energy consumptions. Alternatively in smart pricing, utilities encourage residential customers to voluntarily manage their loads, e.g. by reducing their consumptions at peak hours on a real time pricing (RTP) market. As there are not direct external forces in the latter approach, residential customers individually decide whether they participate in the DSM program or not. Therefore in this scheme, residential customers can manage their satisfaction and keep it in a standard level.
DSM in MES was introduced in Sheikhi et al., 2014, Sheikhi et al., 2015a which was called integrated DSM (IDSM). In this approach, consumers can communicate with both networks simultaneously.
In MES, for controlling and optimizing the consumption of a price taker residential customer who has no considerable effect on electricity price, different methods have been introduced and discussed (Alejandro et al., 2014, Ameri and Besharati, 2016, Mancarella and Chicco, 2013a, Mancarella and Chicco, 2013b, Martínez Ceseña et al., 2015, Pazouki et al., 2014). Mancarella and Chicco (2013a) present comprehensive techno-economic methodologies for the quantification of DSM from small commercial and residential end-users. A simple mixed integer programming (MIP) method is presented in Mancarella and Chicco (2013b), to optimally manage both heating and electrical loads in MES based on the residential customer payment cost objective function. In Martínez Ceseña et al. (2015), an analysis has been outlined to address the potential of exploiting CHP systems in providing DSM. Ameri and Besharati (2016) proposed a mixed integer linear programming (MILP) method determine the optimal capacity and operation of seven combined cooling, heating and power (CCHP) systems in the heating and cooling network of a residential district. In Pazouki et al. (2014), a Monte Carlo algorithm is applied to minimize operation costs and maximize reliability of a residential customer in a stochastic MES environment. In Alejandro et al. (2014), the optimal operation of an integrated electricity and natural gas infrastructure is presented by applying a model predictive system control approach. Regarding residential building demand side management, Rastegar and Fotuhi-Firuzabad (2015) presents a residential energy hub model for a smart multi-carrier energy home. In this reference, an optimization-based program is proposed to determine the optimal operation mode of the energy hub with household responsive demand in response to day-ahead predetermined tariffs of electricity. In order to prediction of building energy consumption, Brohus, Frier, Heiselberg, and Haghighat (2012) presents a new approach that quantifies the uncertainty of building energy consumption by means of stochastic differential equations (SDE). To optimization of thermal comfort and energy consumption in a residential house (Magnier & Haghighat, 2010) described an optimization methodology based on a combination of an Artificial Neural Network and a multi-objective evolutionary algorithm. Regarding the forecasting models, Nguyen, Reiter, and Rigo (2014) shows that the metaheuristic search algorithms (e.g. genetic algorithm and particle swarm optimization) are the most popular algorithmic technique applied to building optimization problem. Harish and Arun Kumar (2016) and Li and Wen (2014) review all the significant modeling methodologies which have been developed and adopted to model the energy systems of buildings categorically and majorly focuses on the recent studies which involved development of the control strategies by modeling the building energy systems.
Based on above mentioned approaches, there are three main shortcomings when it comes to energy management system (EMS) programming in the real world as follows:
- 1.
Energy system parameter definition: Almost all approaches, which are applied to optimize the energy consumption of residential customers, require modeling a building in detail which it is not be suitable for on-line operation optimization. In these approaches (Harish and Arun Kumar, 2016, Li and Wen, 2014) it is required to define some specific parameters, e.g. appliances efficiencies and energy prices, as constant (Rastegar & Fotuhi-Firuzabad, 2015); however, such an assumption would not necessary hold accurate in practice. For instance, appliances efficiencies vary with time and energy prices are intrinsically stochastic.
- 2.
Considering MES framework: there are some papers have been published recently considering household behavior and modeling uncertainty in predicting building energy consumption (Brohus et al., 2012, Magnier and Haghighat, 2010). Although there are valuable papers but it is required to have a holistic view and consider household in MES framework.
- 3.
Objective function modeling in EMS: In the past, dissatisfaction function has either been overlooked (Rastegar & Fotuhi-Firuzabad, 2015) or included, if at all, in an inaccurate fashion (Maharjan, Zhu, Zhang, Gjessing, & Basar, 2013). Without factoring in dissatisfaction function, EMS cannot deal with real world conditions. On the other hand, if modeled in a one-size-fits-all manner, dissatisfaction function does not represent the behavior of each individual because different users have different priorities.
Therefore, without tackling two mentioned deficiencies, EMS programming will arguably not lead to the accurate results in real life.
In this paper, to address these concerns, we formulate energy prices and residential customers’ loads as Markov model. We use reinforcement learning (RL) algorithm to allow the EMS learns behaviors of the residential customer, the uncertainty of energy prices, and appliances’ efficiencies to make an optimal decision.
RL is one of the machine learning techniques proposing to solve real-world problems with a dynamic environment such as electricity, heating, cooling loads, and price fluctuations without taking any system components parameters. RL differs from standard supervised learning (Nguyen et al., 2014) in that correct input/output pairs are never presented, nor sub-optimal actions explicitly corrected. Further, there is a focus on on-line performance, which involves finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge). In RL a learner is adapted to an unknown and dynamic environment by receiving rewards and punishments from the environment. Daneshfar and Bevrani (2010) and Giupponi, Agusti, Pérez-Romero, and Sallent (2005) show the application of RL in different issues.
Here, we consider the smart pricing scheme, where each user, as a price taker consumer, has been equipped with an EMS programming based on RL disciplines to minimize the electricity and natural gas bills, simultaneously. This paper firstly focus on the modeling of a house as a MES, considering different electrical and heating loads, micro-CHP, and a PHEV in an appropriate way to be adaptable with RL algorithm. Then, operation of the MES is optimized and the household load is optimally managed in the stochastic environment. The objective function is to minimize the residential customer energy payment cost and dissatisfaction level which incurs during shifting of loads in response to the time varying prices of electricity and natural gas. Solving the proposed problem determines the amount of energies following to each converter and device, the charging strategy of PHEV, and the energy consumption scheduling of all appliances at any hours. Due to curse of dimensionality in solving the optimization problem via RL algorithm at houses, we decompose this problem into clusters and use a “state reduction technique” to make the algorithm more applicable in practice.
The rest of this paper is organized as follows. The model of the price taker residential customer in MES is described in Section 2. In Section 3, the mathematical models for the loads, PHEV battery, and energy prices are given. EMS system framework and the formulation of RL algorithm is presented in Sections 4 Energy management system framework, 5 RL algorithm vs DP, respectively. Simulation results are presented in Section 6. Finally, the paper is concluded in Section 7.
Section snippets
Price taker residential customer in multi-energy systems
A price taker residential customer participates in DSM program without any communications with other ones. In fact, the price taker residential customers believe that their energy consumptions cannot affect the energy prices. Therefore, energy prices are assumed as exogenous signals for this type of residential customers.
The residential customer can connect to MES, e.g. natural gas and electricity, by means of various conversion and storage devices. A simple scheme of residential customer's
Mathematical model of stochastic environment
The EMS selects its actions based on the state of the environment as depicted in Fig. 4. Here, the environment includes a price-taker residential customer in MES. The definition of action and state signals are given in the next section.
In this study, the environment is a stochastic system. It means that the subsequent state of environment is unpredictable due to the influence of random variables. Here, these random variables are energy prices and consumption patterns of the residential customer.
Energy management system framework
EMS tries to minimize its cost function through the action signal. Here, the action signal has this vector: . By means of action signal, the conversion devices and appliances are controlled. The action signal of CHP is uCHP(t) which control CHP generation level based on (16):
As Fig. 9 shows, EMS finds the optimal policy which determines the probability of taking action in state. Here, the state signal at each time slot is the vector
RL algorithm vs DP
DP algorithm requires the transition probability of environment. Moreover, it requires environment parameters, e.g. appliances efficiency, conversion devices parameters, and online electricity and natural gas prices. To practically find the optimal policy in a dynamic environment with numerous parameters that change all the time, we have to apply an algorithm that works with few parameters and adapts itself during ongoing changes. An EMS that is programmed based on Reinforcement learning (RL)
Performance evaluation
To demonstrate the effectiveness of the proposed algorithm, it is applied to a price taker residential customer in MES. The system is simulated in MATLAB Simulink-R2010a®, in the real time and stochastic environment. We assume that natural gas and electricity prices are determined by the utility companies. In this market, the EMS solves the optimization problem just by considering the historical data.
Conclusions
In this paper, we introduce a modified model of residential customer in a MES. We show that residential customers in MES can be encouraged to participate in DSM programs by both shifting their energy usage and changing their energy resources. We tackle the problem of designing optimal DSM program for a price taker residential customer who has to deal with varying component parameters, e.g. efficiency by applying RL algorithm. To evaluate the performance of the proposed method, a residential
References (38)
- et al.
Optimal design and operation of district heating and cooling networks with CCHP systems in a residential complex
Energy and Buildings
(2016) - et al.
Quantification of uncertainty in predicting building energy consumption: A stochastic approach
Energy and Buildings
(2012) - et al.
Current status of fuel cell based combined heat and power systems for residential sector
Journal of Power Sources
(2015) - et al.
A review on modeling and simulation of building energy systems
Renewable and Sustainable Energy Reviews
(2016) - et al.
Review of building energy modelling for control and operation
Renewable and Sustainable Energy Reviews
(2014) - et al.
Multiobjective optimization of building design using TRNSYS simulation, genetic algorithm, and artificial neural network
Building and Environment
(2010) MES (multi-energy systems): An overview of concepts and evaluation models
Energy
(2014)- et al.
Electrical network capacity support from demand side response: Techno-economic assessment of potential business cases for small commercial and residential end-users
Energy Policy
(2015) - et al.
A review on simulation-based optimization methods applied to building performance analysis
Applied Energy
(2014) - et al.
Uncertainty modeling in optimal operation of energy hub in presence of wind, storage and demand response
International Journal of Electrical Power & Energy Systems
(2014)