Multi-task deep reinforcement learning for intelligent multi-zone residential HVAC control

doi:10.1016/j.epsr.2020.106959

Electric Power Systems Research

Volume 192, March 2021, 106959

https://doi.org/10.1016/j.epsr.2020.106959 Get rights and content

Highlights

•
A multi-zone residential HVAC control is proposed in this paper.
•
It is based on multi-task deep reinforcement learning (deep RL) method.
•
It minimizes energy consumption costs while maintaining users’ comfort.
•
It is compared with a rule-based case and a single-task DDPG gradient algorithm.

Abstract

In this short communication, a data-driven deep reinforcement learning (deep RL) method is applied to minimize HVAC users’ energy consumption costs while maintaining users’ comfort. The applied deep RL method's efficiency is enhanced by conducting multi-task learning that can achieve an economic control strategy for a multi-zone residential HVAC system in both cooling and heating scenarios. The applied multi-task deep RL method is compared with a rule-based benchmark case and a single-task deep deterministic policy gradient algorithm to verify its effective and generalized application in optimizing HVAC operation.

Introduction

The latest development in machine learning such as deep learning and reinforcement learning techniques are being widely discussed in many critical areas that were once dominated by human intelligence, such as robotic control and autonomous driving [1], as well as in the field of power and energy [2]. In particular, the deep reinforcement learning (deep RL) method has been implemented for controlling heating, ventilation, and air conditioning (HVAC) systems to achieve both an economic benefit and improved customer comfort. In [3], a model-free deep Q network (DQN) is applied for joint data center and HVAC load control in mixed-use buildings to reduce energy consumption. In [4], the authors compare the value-based DQN method with the policy-based deep policy gradient (DPG) method in residential energy management, and demonstrate that the latter is more suitable to perform online scheduling of energy sources. Given that many control variables in HVAC thermal control are continuous, the deep deterministic policy gradient (DDPG) method is implemented in [5,6], to avoid the discretization of the control variables and to obtain better learning performance. In [7], the authors utilize imitation learning to pre-train the HVAC control agent on historical data to make it behave similarly to the existing controller. Following this, the RL agent continues to improve its policy during the online training using a policy gradient method proximal policy optimization (PPO). In [8], the authors further extend the deep RL algorithm to optimize multi-zone HVAC system control, where a set of actor network and critic network is designed for each thermal zone, and feature extraction from selected neighbor zones is collected to better capture the mutual thermal effects between different zones for improving the control policy.

While the effectiveness of the deep RL based HVAC control methods has been illustrated in the above existing researches, one deficiency is that the majority of the researches focus on learning a single HVAC control task by merely training the algorithm in either the cooling scenario or the heating scenario. Retraining of the algorithm is required when the scenario switches. It is widely known in the RL community that the training of the algorithm can be time-consuming and resource-consuming. Solving only one task at one time is less efficient and acceptable as more complex control problems emerge. Motivated by the above concern, in this short communication, we work on teaching the RL agent to master both the cooling and heating tasks simultaneously to guarantee an optimal HVAC control regardless of the scenario. A multi-task DDPG algorithm is developed for this purpose and is further tested in a multi-zone residential HVAC system. Comparisons with a rule-based HVAC control strategy and a single-task DDPG algorithm demonstrate that the multi-task DDPG algorithm has higher generalization and enables lower energy consumption cost and less user comfort violation through intelligent scheduling.

Section snippets

Multi-task DDPG for multi-zone residential HVAC control

The changing of indoor temperature under the control of residential HVAC system can be formulated as a Markov Decision Process (MDP) [9], and the key parameters are defined as follows:

1)
State: the outdoor temperature T_out(t), the indoor temperature T_in,z(t) for each zone z, and the retail price λ_retail(t), where t is the index of time step; 2) Action: the setpoint Setpt_z(t) for each zone z; 3) Reward: the total energy consumption cost plus the temperature violation penalty, as shown below: $r (t) = - \sum t$

Simulation results

The above multi-task DDPG algorithm is tested in a two-zone residential HVAC building model [11]. The weather data and Georgia Power price data from [12], [13] are used for algorithm training and testing. The Georgia Power price contains only two price values, a peak price value at 0.2$/kWh, and an off-peak price at 0.05$/kWh. For the cooling scenario, the algorithm is trained with data from Jul. 1st, 2019 to Jul. 31st, 2019 and tested with data from Aug. 1st, 2019 to Aug. 10th, 2019, and the

Conclusions

In this short communication, a multi-task DDPG method is applied to learn the setpoint control strategies of multi-zone residential HVAC systems in both cooling and heating scenarios. The multi-task learning process can lead to a more generalized feature extraction among different tasks that share some similarities and improves learning efficiency compared to single-task learning. Comparisons with rule-based control strategies demonstrate the economy and adaptability of the RL-based HVAC

Author statement

Yan Du: Concept development, algorithm development, algorithm implementation, and writing

Fangxing (Fran) Li: Concept development, algorithm development, technical supervision, and editing

Jeffery Munk: Concept development and algorithm development

Kuldeep Kurte: Concept development and algorithm development

Olivera Kotevska: Concept development and algorithm development

Kadir Amasyali: Concept development and algorithm development

Helia Zandi: Concept development, algorithm development, and

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

This work was funded in part by the U.S. Department of Energy, Energy Efficiency and Renewable Energy, Building Technology Office under contract number DE-AC05-00OR22725, in part by CURENT which is an Engineering Research Center (ERC) funded by the U.S. National Science Foundation (NSF) and DOE under the NSF award EEC-1041877, and in part by the U.S. NSF ECCS awards 1809458 and 2033910.

References (13)

F. Li et al.
From AlphaGo to power system AI
IEEE Power Energy Mag.
(Mar. 2018)
R.S. Sutton et al.
Reinforcement Learning: An introduction
(2018)
T. Wei et al.
Deep reinforcement learning for joint datacenter and HVAC load control in distributed mixed-use buildings
IEEE Trans. Sustain. Comput.
(2019)
E. Mocanu et al.
On-line building energy optimization using deep reinforcement learning
IEEE Trans. Smart Grid
(Jul. 2019)
L. Yu et al.
Deep reinforcement learning for smart home energy management
IEEE Internet Things J.
(2019)
G. Gao et al.
DeepComfort: energy-efficient thermal comfort control in buildings via reinforcement learning
IEEE Internet Things J.
(2020)

There are more references available in the full text version of this article.

Cited by (39)

An online reinforcement learning approach for HVAC control
2024, Expert Systems with Applications
Heating, Ventilation and Air Conditioning (HVAC) optimization for energy consumption reduction is becoming ever more a topic of the utmost environmental and energetic concerns. The two most employed methodologies for optimizing HVAC systems are Model Predictive Control (MPC) and Reinforcement Learning (RL). This paper compares three different RL approaches to HVAC optimization: one based on a black-box system identification model trained on historical data, one based on a white-box model of a building and one online method based on an imitation learning pretraining phase on historical data. The three approaches are compared with a literature baseline and an EnergyPlus baseline. Results show that the overall best method in terms of energy consumption reduction (65% decrease) and thermal comfort increase (25% increase) is the approach based on the white-box model. However, the proposed methodology, based on online and imitation learning, demonstrates remarkable efficiency, achieving comparable improvements in energy consumption after just a few months of online training, while maintaining thermal comfort at around the same level as the baseline. These results prove a direct online RL approach, which avoid the use of costly simulations, can provide a reliable and inexpensive solution to the problem of HVAC optimization.
Advanced controls on energy reliability, flexibility and occupant-centric control for smart and energy-efficient buildings
2023, Energy and Buildings
Advanced controls have attracted increasing interests due to the high requirement on smart and energy-efficient (SEE) buildings and decarbonization in the building industry with optimal tradeoff strategies between energy consumption and thermal comfort of built environment. However, a state-of-the-art review is lacking on advanced controls for SEE buildings, especially considering advanced building energy systems, machine learning based advanced controls, and advanced occupant-centric controls (OCC). This study presents a comprehensive review on the latest advancement of advanced controls for SEE buildings, which covers recent research on data collection through smart metering and sensors, big data and building automation, energy digitization, and building energy simulation. Machine learning based advanced controls are comprehensively reviewed, including supervised, unsupervised and reinforcement learning, together with their roles and underlying mechanisms. In addition, advanced controls for energy security, reliability, robustness, flexibility, and resilience are further reviewed for energy-efficient and low-carbon buildings, with respect to fault detection and diagnosis, fire alarming and building energy safety, and climate change adaptation. Moreover, this study explores the advanced OCC systems and their applications in SEE buildings. Last but not the least, this study emphasizes the challenges and future prospects of the trade-off between complexity and predictive/control performance, AI-based controllers and climate change adaptation, OCC in thermal comfort and energy saving for the SEE buildings. This study offers valuable insights into the latest research progress concerning the underlying mechanisms, algorithms and applications of advanced controls for SEE buildings, paving the path for sustainable and low-carbon transition in building sectors.
Dynamic indoor thermal environment using Reinforcement Learning-based controls: Opportunities and challenges
2023, Building and Environment
Currently, the indoor thermal environment in many buildings is controlled by conventional control techniques that maintain the indoor temperature within a prescribed deadband. The latest research provides evidence that more dynamic variations of the indoor thermal environment can promote health and trigger positive thermal alliesthesia , but such an environment requires a flexible and responsive control system that can adapt to the changes in real-time. As an emerging control technique, Reinforcement Learning (RL) has attracted growing research interest and demonstrated its potential to enhance building performance while addressing some limitations of other advanced control techniques. Thus, a comprehensive review explored the boundaries and limitations of a dynamic indoor environment and the possibilities to apply RL for building controls suitable for varying the indoor thermal environment. The first part discussed the studies on the permissible limits of temperature step changes and acceptable drifts to human occupants. It also debated the flexibility of the range of human thermal comfort and adaptation. In the next part, studies on RL for HVAC controls were explored, focusing on their application in creating a dynamic indoor thermal environment. The different algorithms, HVAC systems, co-simulation environment, action spaces, and energy-saving potentials were discussed. Overall, based on the review, this work outlined a potential pathway for the RL-based controller that can dynamically vary the indoor temperature. Suitable environmental parameters to be controlled, a choice of the RL-based algorithm, action space, and co-simulation environment are discussed.
A laboratory test of an Offline-trained Multi-Agent Reinforcement Learning Algorithm for Heating Systems
2023, Applied Energy
This paper presents a laboratory study of Offline-trained Reinforcement Learning (RL) control of a Heating Ventilation and Air-Conditioning (HVAC) system. We conducted the experiments on a radiant floor heating system consisting of two temperature zones located in Denmark. The buildings are subjected to real-world weather. A previous paper describes the algorithm we tested, which we summarize in this paper. First, we present a benchmarking test which we conducted during spring 2021 and winter 2021/2022. This data is used in the Offline RL framework to train and deploy the RL policy, which we then tested during winter 2021/2022 and spring 2022. An analysis of the data shows that the RL policy showed predictive control-like behavior, and reduced the oscillations of the system by a minimum of 40%. Additionally, we show that the RL policy is minimum 14% more cost-effective than the traditional control policy used in the benchmarking test.
A Review of Reinforcement Learning for Controlling Building Energy Systems From a Computer Science Perspective
2023, Sustainable Cities and Society
Energy efficient control of energy systems in buildings is a widely recognized challenge due to the use of low temperature heating, renewable electricity sources, and the incorporation of thermal storage. Reinforcement Learning (RL) has been shown to be effective at minimizing the energy usage in buildings with maintained thermal comfort despite the high system complexity. However, RL has certain disadvantages that make it challenging to apply in engineering practices. In this review, we take a computer science approach to identifying three main categories of challenges of using RL for control of Building Energy Systems (BES). The three categories are the following: RL in single buildings, RL in building clusters, and multi-agent aspects. For each topic, we analyse the main challenges, and the state-of-the-art approaches to alleviate them. We also identify several future research directions on subjects such as sample efficiency, transfer learning, and the theoretical properties of RL in building energy systems. In conclusion, our review shows that the work on RL for BES control is still in its initial stages. Although significant progress has been made, more research is needed to realize the goal of RL-based control of BES at scale.
Energy-efficient heating control for nearly zero energy residential buildings with deep reinforcement learning
2023, Energy
Citation Excerpt :
Deep reinforcement learning (Deep RL) is a model-free method that combines a deep neural network (Q network) and reinforcement learning (RL). Deep RL have been applied to the HVAC industry [26–29]. Different from the model-based method, the model-free method represented by reinforcement learning gradually forms and perfects the control strategy in the form of interactive trial and error between the agent and the environment.
Controlling Heating, Ventilation and Air Conditioning (HVAC) systems is critical to improving energy efficiency of demand-side. In this paper, a model-free optimal control method based on deep reinforcement learning is proposed to control the heat pump start/stop and room temperature setting in residential buildings. The optimization goal of this method is to obtain the highest comprehensive reward which considering thermal comfort and energy cost. Firstly, the randomness, learning process, thermal comfort and energy consumption of the model-free controller are systematically investigated by a simulation system based on measured data. The results show that randomness has a significant impact on the initial performance and convergence speed of the model-free controller; The model-free controller has a linear accumulation of comprehensive rewards during the learning process, and the slope of the accumulated comprehensive rewards can be used to determine whether the controller converges; The model-free controller coordinates monitoring data, weather forecasts and building thermal inertia to achieve the highest comprehensive reward. Afterwards, the model-free controller was verified in a nearly zero energy residential building in Beijing, China. The results show that model-free controller improves the comprehensive reward by 15.3% compared to rule-based method.

View all citing articles on Scopus

: This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the US Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

View full text

Multi-task deep reinforcement learning for intelligent multi-zone residential HVAC control

Highlights

Abstract

Introduction

Section snippets

Multi-task DDPG for multi-zone residential HVAC control

Simulation results

Conclusions

Author statement

Declaration of Competing Interest

Acknowledgement

IEEE Power Energy Mag.

Reinforcement Learning: An introduction

Deep reinforcement learning for joint datacenter and HVAC load control in distributed mixed-use buildings

IEEE Trans. Sustain. Comput.

On-line building energy optimization using deep reinforcement learning

IEEE Trans. Smart Grid

Deep reinforcement learning for smart home energy management

IEEE Internet Things J.

DeepComfort: energy-efficient thermal comfort control in buildings via reinforcement learning

IEEE Internet Things J.