PV-EV Integrated Home Energy Management Considering Residential Occupant Behaviors

Abstract

Rooftop photovoltaics (PV) and electrical vehicles (EV) have become more economically viable to residential customers. Most existing home energy management systems (HEMS) only focus on the residential occupants’ thermal comfort in terms of indoor temperature and humidity while neglecting their other behaviors or concerns. This paper aims to integrate residential PV and EVs into the HEMS in an occupant-centric manner while taking into account the occupants’ thermal comfort, clothing behaviors, and concerns on the state-of-charge (SOC) of EVs. A stochastic adaptive dynamic programming (ADP) model was proposed to optimally determine the setpoints of heating, ventilation, air conditioning (HVAC), occupant’s clothing decisions, and the EV’s charge/discharge schedule while considering uncertainties in the outside temperature, PV generation, and EV’s arrival SOC. The nonlinear and nonconvex thermal comfort model, EV SOC concern model, and clothing behavior model were holistically embedded in the ADP-HEMS model. A model predictive control framework was further proposed to simulate a residential house under the time of use tariff, such that it continually updates with optimal appliance schedules decisions passed to the house model. Cosimulations were carried out to compare the proposed HEMS with a baseline model that represents the current operational practice. The result shows that the proposed HEMS can reduce the energy cost by 68.5% while retaining the most comfortable thermal level and negligible EV SOC concerns considering the occupant’s behaviors.

1. Introduction

There are about 100 million single-family homes in the U.S. that consume 36% of total electricity while causing the peak system load in hot summer days [1]. The home energy management system (HEMS) is one of the most promising tools to conserve electricity cost, especially in the presence of the Internet of Things, smart appliances, smart meters, wireless sensors, and energy storage. HEMS is capable of monitoring the energy usage of smart appliances and subsequently sending the control commands to each controllable appliance for energy cost savings and peak load reduction. A vast body of studies have demonstrated that HEMS can efficiently reduce the electricity cost [2,3,4,5,6,7,8] and provide residential demand flexibility and demand response (DR) (e.g., demand curtailment and demand shifting) in response to the DR signal from distribution system operators to minimize grid congestion and violations at peak load conditions [5]. In [9], a human-centered intelligent HEMS is presented while considering real-time control of DR equipment, which integrates ubiquitous sensor data from the physical layer and the network layer to obtain electricity consumption patterns and cognitively understand the behavior of residents.

The operational landscape of the distribution system is undergoing a radical transformation. Particularly, increased distributed renewable energy resources and demand diversification are challenging the status quo. The installed capacity of residential photovoltaics (PV) systems is on the rise globally since its cost has been reduced by 64% since 2010 [10]. More and more families install and utilize PV as one clean and reliable power resource. One study [11] has comprehensively demonstrated that utilizing the residential PV affects the electricity consumption for the household under the uncertainty of the cost of the PV system and occupant’s investment. Srikranjanapert et al. [12] found that the government should promote the integration of HEMS with a PV system.

Meanwhile, the transportation industry, including both commercial and residential sectors, has been experiencing one of the greatest technology transitions toward electrical vehicle (EV) [13]. By 2030, EV sales are estimated more than 10% of the US new-vehicle market share in a medium growth scenario [14]. EVs, as the home energy storage, will play a more important role in providing vehicle-to-home (V2H) services in the presence of residential PV. To coordinate the scheduling of the house load, a PV-EV integrated HEMS could positively affect the household and the distribution network, creating a ’win-win’ scenario for both [15]. Moreover, Shafie-Khah et al. [6] incorporate the uncertainties of distributed renewable energy (DRE) and EV to schedule the appliances in smart houses. Mohammadi et al. [7] use a genetic algorithm to solve scheduling of many home equipment such as EV, PV, clothes washer, dishwasher, etc., via the various residential loads. Wang et al. proposed a multiobjective optimal method for HEMS based on the Internet of Things system (i.e., ZigBee) in [16]. Jin et al. [17] integrated the machine-learning-based prediction programming into HEMS that considers thermal comfort, energy cost, carbon emission, and user convenience. Mojtaba et al. [18] applied a stochastic model predictive control strategy to minimize the electricity cost and reduce the cost of EV battery degradation. Wu et al. [8] integrate renewable energy system (RES) into EV charging station via a finite-horizon Markov decision process (MDP) model while maintaining a satisfaction rate of the demand. All the above works suggest that there has been a fast-growing interest in integrating residential PV and EV into the HEMS in the research communities.

Another primary function of the HEMS is maintaining occupants’ comfort via tracking occupants’ activities (behaviors) and understanding occupants’ preferences. References [19,20,21] presented comprehensive research about the occupant’s behavioral pattern in the HVAC system. An et al. [19] proposed a new key-performance index to represent occupant’s air conditioning (AC) usage such as AC operating hours, fan coil units’ operation hours, the ratio of AC-on days, etc. Wang et al. [20] proposed a model linking the occupancy with an energy-cyber-physical system, which can potentially save about 26.4% of energy consumption in the HVAC system. Hong et al. [21] explored the link between occupant personality types with their behaviors of sharing energy-environment control systems and interactions with their colleagues inspired by a five-factor model with four dimensions: willingness to share control, knowledge of control, group decision behavior, and adaptive strategies. However, all the above studies pertaining to the occupant’s behaviors have not factored in the occupant’s clothing behaviors, which may drastically affect the occupant’s thermal comfort.

The traditional way to design the HVAC system in the building is to consider the building insulation and indoor temperature/humidity. However, clothing insulation is also a significant factor for occupants’ thermal comfort. Specifically, Fanger et al. proposed a thermal index, i.e., predicted mean vote (PMV) for occupant [22]. PMV takes humidity, temperature, and clothing insulation, etc., into consideration to predict the thermal comfort of occupants [23,24]. PMV has been widely adopted by many researchers in the context of HVAC scheduling [25,26,27,28,29]. In reality, residential occupants can frequently adjust their clothing, depending on the thermal conditions around them, on an hourly or subhourly basis, as opposed to a constant clothing value in an entire day. The frequent clothing adjustment is even more plausible under the COVID-19 outbreak, whereby many businesses worldwide shift from traditional office-based work operations to work-from-home arrangements. Such adjustments can improve home demand flexibility and thus enable a higher energy cost saving. One relevant study that explicitly considers clothing behaviors in PMV in the context of home energy management is the authors’ previous work [30], which is first-of-its-kind in modeling the clothing behaviors as controllable variables to optimally adjust the clothing levels for occupants. The authors believe that the hourly or subhourly clothing adjustment should be fully captured and optimized in the residential home energy management to model HVAC systems realistically. Considering clothing behaviors and actions (donning/doffing) into the HVAC scheduling is meaningful and innovative to enable a sustainable energy future in the residential sector.

As the EV may become the major (only) transportation for a household and even for a sustained community, the expected EV state of charge would be another serious concern for residential occupants [31,32,33]. In [31], the authors incorporated the dynamic of driver’s behaviors into the EV charging model and proposed a stochastic game approach to address the renewable energy uncertainty. Reference [32] introduces a time anxiety concept to address the uncertain events in the charging duration and a game theory-based approach to solve the optimal EV charging problem. Yan et al. [33] proposed a new index to measure driving anxiety that to characterize the driver’s discomfort on the driving range and uncertain events which are changed by the driving experience quantitatively. However, all the related work above ignored the complex interaction between EV and other major appliances such as PV and HVAC, resulting in piecemeal studies in the context of the HEMS.

To tackle those problems, this paper studies the HEMS, consisting of HVAC, EV, and PV, by taking into account three distinct categories of occupant behaviors: (1) clothing behaviors, (2) EV SOC concerns, and (3) PMV-based thermal comforts. A stochastic Adaptive Dynamic programming (ADP) model [34] is modified and extended to optimally determine the setpoints of heating, ventilation, air conditioning (HVAC), occupant’s clothing decisions and the EV’s charge/discharge schedule while considering uncertainties in the outside temperature, PV generation, and EV’s arrival SOC. Nonlinear and nonconvex models of thermal discomfort, EV SOC concerns and clothing behaviors are holistically embedded in the ADP-HEMS model. A model predictive control framework is further proposed to simulate a residential house under the time-of-use tariff such that it continually updates with optimal appliance schedules decisions passed to the residential house model. Simulations are systematically carried out to compare the results between the proposed method and the baseline model. The contributions of this work are three-fold:

(1) To consider the occupant’s behaviors, i.e., thermal comfort, clothing behaviors, and EV SOC concerns, a stochastic ADP-HEMS model is established to include HVAC, EV, and PV under uncertainties.

(2) An MPC framework is further developed for co-simulating the proposed HEMS model with a house model such that it continuously passes the HEMS decisions and updates the house status for more accurate and realistic simulations.

(3) Cosimulations are carried out to compare the proposed ADP-HEMS with a baseline case that represents the current operation for a residential house. The result shows that a significant cost saving is achieved while the occupant’s various comfort is satisfied.

2. Background and Formulation

2.1. Background

The scheme of the proposed HEMS, shown in Figure 1, includes three interconnected parts: a preprocessor, an ADP-based MPC model, and a house model. The Preprocessor is mainly used for the analysis of the historical and forecast data collected from the cloud, such as the occupant’s desired sleeping temperature, the daily energy consumption of EV, and weather forecasts, etc. A deep neural network can be utilized to yield forecasts. The ADP-based MPC model is an algorithmic function in the application layer to determine the optimal actions for the controllable appliances. The details of this model are provided in the subsequent sections. The house model is the physical model that executes the scheduled setpoints obtained from the ADP-based MPC. In the house model, all the data are collected by a central data collector from sensors and smart meters in the home. The occupant’s daily schedules, such as commute and travel plans, can be collected from the smartphone. Such data will be feedback to the occupant-centric environment information module for data analysis and calibration. Subsequently, the optimal schedule for HVAC and EV generated by the HEMS is directly sent to the corresponding appliances, whereas the best actions for clothing behaviors are sent to the occupant’s smartphone application as notifications. The occupant can choose to either follow or ignore the notifications. Next, the occupants’ responses are recorded, and the occupant status is updated. The remainder of this section shows the description of the HEMS formulation, consisting of HVAC, EV, and PV, with detailed models of occupants’ thermal comfort, clothing behaviors, and EV SOC concerns. Finally, a discussion of the proposed HEMS-MPC framework is displayed.

2.2. ADP-HEMS Formulation

A HEMS problem is a typical optimization problem modeled as a Markov decision process (MDP) to minimize the occupant’s utility function while considering various constrains at multiple levels, e.g., the occupant level, residential home level and distribution grid level. In this paper, the proposed framework focuses on three types of controllable variables, i.e., $i=\{\mathrm{HVAC},\mathrm{Clo},\mathrm{EV}\}$. The indoor temperature $s_t^{HVAC}$, the level of clothing insulation $s_t^{Clo}$, and the SOC of EV $s_t^{EV}$ represent the variable state at time t. The associated actions for those variables are HVAC input power $a_t^{HVAC}$, clothing decision $a_t^{Clo}$, i.e., donning and doffing, and EV charging/discharging power $a_t^{EV}$. Positive $a_t^{EV}$ values denote charging actions, and negative values are for discharging actions. In addition, uncertainty is taken into account such as the outside temperature is ${\tilde{u}}_t^{\mathrm{OUT}}$, the EV arrival SOC ${\tilde{u}}_t^{\mathrm{SOC}}$, and PV generation ${\tilde{u}}_t^{\mathrm{PV}}$. Arithmetically, the set tuple $\{s_t,a_t,{\tilde{u}}_t\}$ simply depicts state, action, and uncertainty of proposed HEMS:

$$\{s_t,a_t,{\tilde{u}}_t\}=\{(s_t^{\mathrm{HVAC}},s_t^{\mathrm{Clo}},s_t^{\mathrm{EV}}),(a_t^{\mathrm{HVAC}},a_t^{\mathrm{Clo}},a_t^{\mathrm{EV}}),({\tilde{u}}_t^{\mathrm{OUT}},{\tilde{u}}_t^{\mathrm{PV}},{\tilde{u}}_t^{\mathrm{SOC}})\}$$

The primary function of HEMS is to find the optimal actions $a_t^{*}$ for minimizing the expected weighted sum of the objective function $O_t$ over the entire look-ahead horizon:

$$\begin{array}{cc}\hfill a_t^{*}=& arg\underset{a_t}{min}\mathbb{E}\{\sum _{t=1}^T{O}_t(s_t,a_t,{\tilde{u}}_t)\}\hfill \end{array}$$

(1)

where $O_t$ is a function of state $s_t$, action $a_t$, and uncertainty ${\tilde{u}}_t$, which is defined as follows:

$$\begin{array}{c}\hfill O_t(s_t,a_t,{\tilde{u}}_t)={\tau }_t^{\mathcal{D}}·{\mathcal{D}}_t\left(s_t,a_t,{\tilde{u}}_t\right)+{\tau }_t^{\mathcal{C}}·{\mathcal{C}}_t\left(s_t,a_t,{\tilde{u}}_t\right)\end{array}$$

(2)

where the objective function is composed of two components: (1) a discomfort function ${\mathcal{D}}_t$ and (2) an energy cost function ${\mathcal{C}}_t$. In (2), ${\tau }_t^{\mathcal{D}}$ and ${\tau }_t^{\mathcal{C}}$ are respectively weighted coefficients associated with discomfort and the energy cost, the values of which can represent a different type of occupants, i.e., cost-saving-seeking occupants or comfort-seeking occupants. Those coefficients can be finely tuned by using historical simulation data. The occupant discomfort function ${\mathcal{D}}_t$ is defined as:

$$\begin{array}{cc}\hfill {\mathcal{D}}_t\left(s_t,a_t,{\tilde{u}}_t\right)=& {\beta }^{PMV}·\left|PMV\left(s_t\right)\right|+{\beta }^{SOC}·E{V}_t^{concern}\hfill \\ \hfill & +{\beta }^{Clo}·Cl{o}_t^{penalty}+{\beta }^{Bat}·Ba{t}_t^{penalty}\hfill \end{array}$$

(3)

where $PM{V}_t$, $E{V}_t^{concern}$, $Cl{o}_t^{penalty}$, $Ba{t}_t^{penalty}$ are the occupant’s thermal comfort, EV SOC concern, penalties on frequent clothing adjustments, and penalties on frequent switches between charge and discharge, respectively; ${\beta }^{PMV}$, ${\beta }^{SOC}$, ${\beta }^{Clo}$, and ${\beta }^{Bat}$ are the corresponding coefficients. Additionally, $\left|PMV\left(s_t\right)\right|$ is expressed in an absolute value format since the most comfortable thermal state for the occupant is when PMV = 0. The energy cost function ${\mathcal{C}}_t$ is defined as:

$$\begin{array}{c}\hfill {\mathcal{C}}_t\left(s_t,a_t,{\tilde{u}}_t\right)={\mathrm{c}}_t{\mathrm{p}}_t^{\mathrm{G}}\Delta t\end{array}$$

(4)

where ${\mathrm{p}}_t^{\mathrm{G}}$ is the power exchange in kW between the grid and the house, $c_t$ is the electricity price in $/kW paid by the occupant, and $\Delta t$ in hr is the time interval resolution. Note that a positive ${\mathrm{p}}_t^{\mathrm{G}}$ denotes the action to purchase power from the utility and a negative ${\mathrm{p}}_t^{\mathrm{G}}$ for selling power back to the utility. The power balance equation among PV, EV, HVAC, and the power grid for the residential home is expressed as:

$$\begin{array}{c}\hfill {\mathrm{p}}_t^{\mathrm{G}}=a_t^{\mathrm{HVAC}}+a_t^{\mathrm{EV}}-{\tilde{u}}_t^{\mathrm{PV}},\forall t\end{array}$$

(5)

where the PV generation ${\tilde{u}}_t^{\mathrm{PV}}$ is taken as a stochastic input parameter with a time-series forecast over the scheduling horizon.

2.3. Occupant’s Comfort Model

2.3.1. Thermal Comfort

Here, a simplified PMV model from [25] is adapted:

$$\begin{array}{c}\hfill \mathrm{PMV}\left(s_t\right)=\mathrm{a}\left(s_t^{Clo}\right)·s_t^{\mathrm{HVAC}}+b\left(s_t^{Clo}\right)·\mathrm{Pa}(s_t^{\mathrm{HVAC}},r{h}_t)-c\left(s_t^{Clo}\right),t_w<t<t_s\end{array}$$

(6)

where $\mathrm{a}\left(s_t^{Clo}\right)$, $b\left(s_t^{Clo}\right)$ and $c\left(s_t^{Clo}\right)$ are associated coefficients, which are relevant to the clothing insulation of the occupant and can be obtained under different clothing insulation levels [25]; $r{h}_t$ is the relative humidity; $t_w$ and $t_s$ are the occupant’s times of waking up and going to bed, respectively. In general, the PMV is only considered when the occupant is active at home, excluding the sleeping time. As seen, this model is only dependent on the indoor air temperature and water vapor pressure $\mathrm{Pa}(s_t^{\mathrm{H}},r{h}_t)$ in kPa. The water vapor pressure function is defined:

$$\begin{array}{c}\hfill \mathrm{Pa}(s_t^{\mathrm{HVAC}},r{h}_t)=r{h}_t·0.61121·e^{(18.678-s_t^{\mathrm{HVAC}}/234.5)·(s_t^{\mathrm{HVAC}}/(257.14+s_t^{\mathrm{HVAC}})}\end{array}$$

(7)

2.3.2. Clothing Behavior Model

The thermal insulation offered by the occupant’s clothing state is called clothing insulation, which is quantified by the unit of clo. One unit of clo equates to 0.155 K×m²/W, indicating the amount of clothing needed by a sedentary person to maintain thermal comfort in an environment with 21 ${}^{\circ }$C of air temperature, 50% of relative humidity, and 0.1 m/s of airspeed. A comprehensive list of clo values for selected garment types and formulas can be found in the ASHRAE Standard 55 [23] and ASHRAE Handbook [35] for assessing the insulation provided by a clothing ensemble.

Table 1. Three ranges of clothing insulation for occupants.

${\mathit{s}}^{\mathit{Clo}}$ Value	clo 1	clo 2	clo 3
Clothing insulation range	0.25–0.50	0.51–1.00	1.01–1.65
Ensemble Example	(Shirt Level) short-sleeve shirt + thin trousers	(Sweater Level) thin long-sleeve sweater + long-sleeve shirt + thick trousers	(Jacket Level) thick suit jacket + long-sleeve shirt + thick trousers

lists the estimated clo values of some typical business casual clothing ensembles. The clothing insulation is typically partitioned into three ranges, i.e., Clo 1, Clo 2, and Clo 3, which are from 0.25 to 0.50, from 0.51 to 1.00, and from 1.01 to 1.65, respectively.

As seen in Table 1, if $s^{Clo}$ of a male occupant is equal to clo 2 and he is feeling cold, he can exchange his thin sweater for a thick suit. Conversely, if he is feeling hot in such a clothing state, he may take off his thin long-sleeve sweater, and replace his long-sleeve shirt for a short-sleeve shirt. Analogously, a female occupant can adjust her clothing insulation by adding or reducing layers of clothing, which in turn has a direct impact on the occupant’s thermal comfort. Therefore, the optimal indoor temperature can be drastically changed by the HEMS under different clo values. Figure 2 shows the PMV curves defined in Equation (6) versus ambient temperature at different clo levels. The clo levels impacts the associated coefficients $\mathrm{a}\left(s_t^{Clo}\right)$, $b\left(s_t^{Clo}\right)$, and $c\left(s_t^{Clo}\right)$ in Equation (6), resulting in three PMV curves for either gender of the occupant. In Figure 2, the most comfortable temperature range, that is, when the PMV value is between −0.5 and 0.5, for clo 1, clo 2, and clo 3 are highlighted in blue, yellow, and green, respectively.

$$\begin{array}{c}\hfill s_t^{Clo}=s_{t-1}^{Clo}+a_t^{Clo}\end{array}$$

(8)

$$\begin{array}{c}\hfill {\underline{\mathrm{s}}}_t^{Clo}\le s_t^{Clo}\le {\overline{\mathrm{s}}}_t^{Clo},\forall t\end{array}$$

(9)

$$\begin{array}{c}\hfill {\underline{a}}_t^{Clo}\le a_t^{Clo}\le {\overline{a}}_t^{Clo},\forall t\end{array}$$

(10)

$$Cl{o}_t^{penalty}=|a_t^{Clo}·a_{t-1}^{Clo}|,\forall t>1$$

(11)

To incorporate the occupant clothing states and actions, the clothing state transition of the occupant is expressed in (8). The upper and lower bounds on the clothing state and actions are modeled in (9) and (10). While, as is demonstrated in the numerical simulations, the clothing behavior model provides greater flexibility for the HEMS to come up with better HVAC schedules, frequent doffing and donning creates inconvenience for the occupants. As a result, adding a penalty on clothing changing behavior in consecutive time periods as modeled in (11) can avoid the problem of frequent doffing and donning. This penalty is further added in (3) as a part of the overall occupant discomfort. Notice that this clothing model of the occupant is formulated as a combinatorial optimization problem, which may suffer the computational intractability issue in a large-scale HEMS problem.

2.3.3. EV Model with the Occupant’s SOC Concern

Due to the inaccurate estimation of the EV driving range, unforeseeable traffic conditions, and potential early-than-expected departure time, an EV driver is typically fearful of completely depleting the EV battery before reaching the destination. The definition of the term, namely the SOC concern, represents the occupant’s concern caused by all these factors. The SOC concern can be viewed as a reflection of the occupant’s behavior proneness for charging the EV when it is parked at home. Therefore, a mathematical model is formulated here to describe the occupant’s EV SOC concern as follows:

$$E{V}_t^{concern}=max\left(SO{C}_t^e-s_t^{EV},0\right),t_a\le t\le t_d$$

(12)

The $SO{C}_t^e$ is the occupant’s expected SOC during the charging as follows:

$$SO{C}_t^e=\frac{k_1\left(e^{-k_2\left(t-t_a\right)/\left(t_d-t_a\right)}-1\right)}{e^{-k_2}-1},t_a\le t\le t_d$$

(13)

where $t_a$ and $t_d$ are the EV arriving time and departing time, respectively; $k_1$ and $k_2$ are the shape parameters which can be established based on insights of the occupant’s driving behavior, the occupant’s sensitivity to electricity price and the tolerance to SOC concern [33]. Figure 3 provides a visible demonstration of how the SOC concern involves overtime in a charging duration. The dark orange area represents the remaining EV SOC concern, and light orange depicts the concern that is relieved when the actual SOC ($s_t^{EV}$) is greater than the expected SOC ($SO{C}_t^e$) in the charging duration. Other EV constraints are modeled as follows:

$$\begin{array}{c}\hfill s_t^{\mathrm{EV}}={(1-\lambda }^{\mathrm{EV}})s_{t-1}^{\mathrm{EV}}+\frac{{\eta }^{\mathrm{EV}}{a}_t^{\mathrm{EV}}}{C^{\mathrm{EV}}},t_a+1\le t\le t_d\end{array}$$

(14)

$$\begin{array}{c}\hfill {\underline{\mathrm{s}}}_t^{\mathrm{EV}}\le s_t^{\mathrm{EV}}\le {\overline{\mathrm{s}}}_t^{\mathrm{EV}},\forall t\end{array}$$

(15)

$$\begin{array}{c}\hfill {\underline{a}}_t^{\mathrm{EV}}\le a_t^{\mathrm{EV}}\le {\overline{a}}_t^{\mathrm{EV}},\forall t\end{array}$$

(16)

$$Ba{t}_t^{penalty}=max\left(-(a_{t-1}^{EV}·a_t^{EV}),0\right)$$

(17)

Constraint (14) shows the state transition of EV when it is parked at home. ${\lambda }^{\mathrm{EV}}$ in (14) captures its SOC loss caused by self-discharging when transitioning from one state to the next [36]; ${\eta }^{\mathrm{EV}}$ represents the EV charge/discharge efficiency, which may differ between charge and discharge actions made by the HEMS. Constraints (15) and (16) indicate that the EV state and action (i.e., charging and discharging) must lie within its recommended SOC and power limits. It is worth mentioning that V2H is considered in this paper; therefore, the positive value of action $a_t^{EV}$ denotes charging, and negative $a_t^{EV}$ is for discharging. Equation (17) defines a penalty to prevent frequently switching between charging and discharging of the EV battery since the frequent switches would reduce the lifetime of the EV battery [37]. In Equation (17), the penalty $Ba{t}_t^{penalty}$ only occurs when the EV actions change between charging and discharging in consecutive time periods, whereas there is no penalty when it is idle (no action) or charge/discharge consistently. Note that the EV model with the occupant’s EV SOC concern in (12)–(17) are nonlinear and nonconvex, which is naturally suited for the MDP-based solution rather than using the linearization techniques.

2.4. HVAC Model

A first-order thermodynamic model is used to describe the evolution of the room temperature as a function of the previous state, the power consumed by the HVAC as well as the outdoor temperature. The thermal dynamic equation and other HVAC constraints are listed as follows:

$$\begin{array}{cc}\hfill & s_t^{\mathrm{HVAC}}=\hfill \\ \hfill & \{\begin{array}{c}{\gamma }^{RM}{s}_{t-1}^{\mathrm{HVAC}}-{\gamma }_c^{\mathrm{HVAC}}{a}_t^{\mathrm{HVAC}}+{\gamma }^{\mathrm{OUT}}{\tilde{u}}_t^{\mathrm{OUT}},\forall t,\mathrm{cooling}\\ {\gamma }^{\mathrm{RM}}{s}_{t-1}^{\mathrm{HVAC}}+{\gamma }_h^{\mathrm{HVAC}}{a}_t^{\mathrm{HVAC}}+{\gamma }^{\mathrm{OUT}}{\tilde{u}}_t^{\mathrm{OUT}},\forall t,\mathrm{heating}\end{array}\hfill \end{array}$$

(18)

$$\begin{array}{c}\hfill {\underline{\mathrm{s}}}_t^{\mathrm{HVAC}}\le s_t^{\mathrm{HVAC}}\le {\overline{\mathrm{s}}}_t^{\mathrm{HVAC}},\forall t\end{array}$$

(19)

$$\begin{array}{c}\hfill {\underline{a}}_t^{\mathrm{HVAC}}\le a_t^{\mathrm{HVAC}}\le {\overline{a}}_t^{HVAC},\forall t\end{array}$$

(20)

Equation (18) shows the state transition of the room temperature for a prescribed mode, i.e., cooling or heating. Note that in this HEMS model, the HVAC efficiency is embedded in the HVAC-related coefficients ${\gamma }_c^{\mathrm{HVAC}}$ and ${\gamma }_h^{\mathrm{HVAC}}$. Those coefficients, together with other thermal coefficients, i.e., ${\gamma }^{RM}$ and ${\gamma }^{OUT}$, can be obtained by using polynomial fitting based on historical data. In addition, the control action, $a_t^{HVAC}$, is computed in units of thermal energy added or removed. This control can be straightforwardly adapted to the corresponding thermostat settings. Equations (19)–(20) indicate that, for both the cooling and heating cases, the states and actions have to lie within the specified bounds set either by the occupants or by the HEMS.

The overall state transition diagram of the ADP-based HEMS problem consisting of HVAC, EV, and PV while considering the occupant’s thermal comfort, clothing behaviors, and EV SOC concerns on a summer day is shown in the top box of Figure 1. At the terminal state ($t=T$), $O_t$, defined in Equation (2), is calculated for all possible state in the state space. Further, for all previous states, backward induction is implemented to find the optimal objective value with the best decision. The ADP traverses backward and $a_t^{*}$ and $O_t^{*}$ at each time t is obtained. $O_1^{*}$ corresponding to $a_1^{*}$ is then obtained at the initial time interval. As seen, a highlighted red path symbolizing the optimal policy is eventually procured by the proposed ADP-HEMS.

It should be noted that the ADP-based solution is one of the most natural and suitable solutions to the proposed HEMS problem. By contrast, other types of optimization algorithms such as mixed-integer linear programming (MILP), quadratic programming (QP), and genetic algorithm (GA) are not well suited for solving the proposed HEMS problem due to the nonconvex/nonlinearization characteristics of the constraints introduced by the occupant’s thermal comfort, clothing behaviors, and EV SOC concerns. The focus of this paper is to illustrate how the proposed HEMS model can benefit residential homes, and thereby extensive simulations with and without the proposed HEMS are carried out and compared. The algorithmic comparison is beyond the scope of this paper and will be conducted in future work. For a much simpler HEMS problem (without considering occupant behaviors), interested readers can find more details about the algorithmic comparison in the prior work [34].

Note that the Sobol sampling backward induction (SSBI) and K-D tree nearest neighbor (KDNN) techniques [34] are adopted here to increase the computational efficiency of the ADP-HEMS. Analogous to classic backward induction, SSBI contains a Sobol sampling function that compresses the workload efficiently by only considering necessary state, action, and uncertainty sets. KDNN is a noteworthy value function approximation approach that traverses the nearest point in the state space and seeks an approximation value of $O_t^{*}$ instead of computing the exact value from Equation (1). The combination of KDNN and SSBI has demonstrated significant increases in the computation speed of the ADP in finding the optimal policy $a^{*}$ [34].

2.5. MPC-Based HEMS Simulation Framework

Figure 4 shows the flowchart of the developed MPC-based HEMS framework. In this framework, a rolling window is applied to implement and validate the decisions made by the HEMS for the energy cost savings and the occupant’s discomfort minimization. First, the proposed ADP-HEMS optimally searches for the best actions at the current time period based on the initial input data, and then a tuple of optimal actions is generated and passed to the house model (including the status of appliances and occupants) in real-time for execution. The simulated house status after the execution is updated and sent back to the ADP-HEMS as the initial input data for making decisions for the next time period. The above processes repeat, and the rolling window moves forward until the end of the simulation horizon. Notice that there are two types of major inputs: (1) the HEMS configuration and (2) the occupant preferences. The HEMS configuration includes all parameters required to set up the house model under various exterior system conditions, including residential appliance parameters, ToU tariff structure, weather forecasts, etc. The occupant’s preference includes the desired sleeping temperature, the parameters of clothing behaviors, and driving behaviors. Note that machine-learning-based forecast techniques can be integrated in the proposed MPC framework. The long short-term memory (LSTM) is adopted in this study as an outside temperature forecaster. LSTM is one type of recurrent neural networks that have superior time-series predictions for short-term weather conditions. The details of how to integrate LSTM into the HEMS procedure are provided in [30,38].

3. Cosimulation Results

The proposed ADP-HEMS and the MPC cosimulation framework is simulated in Matlab on a desktop computer with the Intel Xeon CPU E5-2640 dual processors and 256 GB RAM. The proposed model and framework are implemented based on the Dynamic programming for Adaptive Modeling and Optimization toolkit developed by National Renewable Energy Laboratory [39]. The HVAC and EV are controllable appliances, whereas other electrical appliances (dishwasher, water heater, etc.) are noncontrollable in the simulation. As shown in

Table 2. The parameters of HVAC and EV.

HVAC	State range	${\mathit{\gamma }}^{\mathrm{RM}}$	${\mathit{\gamma }}^{\mathrm{OUT}}$	${\mathit{\gamma }}_{\mathit{c}}^{\mathrm{HVAC}}$	COP	${\overline{\mathit{a}}}^{\mathrm{HVAC}}$(kW)	SEER	SCOP
	[18 ${}^{\circ }$C, 30 ${}^{\circ }$C]	0.9	0.1	0.116	4.75	6	18	4
EV	State range	$C^{\mathrm{EV}}$(kWh)	${\overline{a}}^{\mathrm{EV}}$(kW)	${\eta }^{\mathrm{EV}}$	${\lambda }^{EV}$	EPA est. (miles)	MPGe (city/highway)
	[20%, 100%]	82	8	0.9	0	315	141 / 127

, the HVAC parameters in Equation (18) were obtained from data of a residential house located in Hillsboro, Oregon [40,41], and the EV parameters can be found in [42,43]. Although simulations are performed only in one location in this paper, the proposed HEMS model can be effectively applied to any other locations with different climate conditions and electricity tariff structures. This will be investigated in future studies.

3.1. Simulation Scenario Setup

The consideration is a two-day cosimulation for a single-family consisting of a young couple. One works from home, and the other drives to work during the first day (Friday) and travels for a short trip for running some errands in the evening of the second day (Saturday). The scenario for the occupant’s driving and sleeping schedules is shown in

Table 3. The clothing condition and EV event.

Time Range	Occupant’s Clothing Conditions	Desired Temperature
[10 p.m., 6 a.m.]	Sleeping with Clo 1	22 °C
Time Range	EV Behavior	EV SOC Consumption
[8 a.m., 6 p.m.]	Not at home at Day 1	30%
[7 p.m., 9 p.m.]	Not home at Day 2	9.7%

for the cosimulation. In Table 3, the EV is not at home between 8 a.m. and 6 p.m. with an estimation of 30% SOC consumption for the first day, and not at home from 7 p.m. to 9 p.m., with an estimation of 9.7% SOC consumption for the second day. The desired sleeping temperature instead of the PMV model is used to validate the performance of the HEMS during the occupant’s sleeping time.

The Pacific Gas & Electric (PG&E) ToU electricity rate is applied, i.e., EToU-E6, shown in Figure 5, which includes the base (white), shoulder (light red area), and peak prices (dark red area) at $0.244, $0.32, and $0.436/kWh, respectively. The two slashed areas represent the occupant’s EV driving schedules when the EV is not at home. To compare the performance of the proposed HEMS, a baseline case is simulated without using the HEMS, in which the HVAC thermostat is set to a constant desired temperature and the EV is charged immediately after arriving at home. The female thermal comfort is used as the reference to compute the PMV in both cases. In addition, there is an assumption that these two days are hot summer days, one being a perfect sunny day and the other being a cloudy day. On both days, the peak temperature is around 34 ${}^{\circ }$C. It is worth mentioning that the use of phase change materials can stabilize daily temperature fluctuations in the residential house, which in turn reduces the temporary demand for heating and cooling. However, this is beyond the scope of this paper and will be investigated in future efforts.

3.2. Baseline Case Simulation

A baseline case is simulated without HEMS that (1) uses a fixed thermostat setpoint for a desired indoor temperature and (2) charges EV immediately after it reaches home. It is assumed that the optimal clothing level for the occupant is realized, which keeps unchanged during all nonsleeping hours. Since the HEMS is not in place in this case, the EV discharging is not considered. Therefore, the baseline case is designed to represent what a typical household does for the time being with PV and EV. Figure 6 shows the simulation results for the HVAC. As seen, the grey-dotted line, black-dotted line, and blue bar represent the outside temperature, room temperature, and HVAC power, respectively. At Day 1, there are two HVAC cooling actions between midnight and 4 a.m. to meet the requirement of the desired temperature, i.e., 22 ${}^{\circ }$C during the sleeping hours. From 6 a.m. to 1 pm, the HVAC stays in the idle status such that the outside temperature mainly affects the room temperature. The sets of cooling actions occur between 2 p.m. to 10 p.m. to maintain the room temperature at the desired temperature (24 ${}^{\circ }$C). Notice that the peak price period is from 2 p.m. to 8 p.m., as highlighted in dark pink. At 11 p.m., the beginning of the occupant’s sleeping time, there is almost rated HVAC power to ensure the room temperature to be around the desired sleeping temperature. Then, the HVAC consumes comparatively less power due to the lower outside temperature for maintaining the room temperature. The HVAC acts similarly at Day 2.

Figure 7 displays the EV simulation results in the baseline, where the maximum charging actions are carried out once the EV returns home according to the current EV operational practice. In Figure 7, the black-dotted line and red-dotted line depict the actual SOC and expected SOC, respectively. The yellow bar represents the charging actions. As seen, on both days, the EV is charged at the maximum charging power as soon as it reaches home. The resulting charging schedule brings up the EV SOC as soon as possible, representing the quickest way to relieve the EV SOC concern. However, the schedule overlaps with the peak and shoulder price in some hours.

Figure 8 shows the power of the three appliances (EV, PV, and HVAC) and the power exchange between the house and the power grid under the net metering policy. The blue bar, orange bar, and olive bar represent HVAC input power, EV charging power, and PV generation, respectively. The black-dotted line represents the net power exchange. Here, the power sink is viewed as positive while the power supply is seen as negative. Therefore, the PV generation is negative, which can be sold back to the grid in the baseline. It is seen that the PV generation occurs from 9 a.m. It powers the HVAC between 2 p.m. and 5 p.m. at Day 1. Once the EV arrives at home, the EV charging power dominates the power exchange and forms the peak demand starting from 6 p.m. at Day 1. In the baseline, the HVAC is the only working appliance during the sleeping time at Day 1 and Day 2. A slightly cloudy day and, in turn, lower PV generation is observed at Day 2. Unlike Day 1, the house purchased more power from the grid. Analogously, a large power purchase happens after 9 p.m. at Day 2 for charging the EV.

3.3. Proposed HEMS Simulation

The proposed ADP-HEMS is simulated in the exact same MPC framework with identical parameters to facilitate a fair comparison between this case and the baseline. In this case, the ADP-HEMS optimally schedule the HVAC, PV, and EV in the MPC cosimulation framework while taking into account the occupants’ thermal comfort, clothing behaviors, and EV SOC concerns. It is reasonable to assume that the occupant always follows the clothing level adjustment suggested by the proposed ADP-HEMS for the purpose of cost saving.

Figure 9 and Figure 10 show the simulation results of the HVAC and the occupant’s clothing actions by the proposed HEMS, respectively. In Figure 10, the discrete dot, red bar, and blue bar represent the clothing level, donning action, and doffing action, respectively. It is seen in Figure 9 that between midnight to noon at Day 1, the HVAC is idle while the room temperature slowly increases to 24 ${}^{\circ }$C. At 6 a.m., a donning action is suggested by the proposed HEMS immediately after the occupant wakes up. As the outside temperature increases, the room temperature climbs up to 24 ${}^{\circ }$C; thereby, a cooling action occurs at noon on Day 1 to keep the optimal temperature for the occupant who is at clo 2. As the temperature keeps increasing, the proposed HEMS recommends a doffing action at 1 p.m. instead of turning on HVAC at the upcoming peak price. In such a way, the HVAC remains on idle between 2 p.m. and 6 p.m. during the peak price. A small cooling action happens at 7 p.m. when the temperature is high up to 28 ${}^{\circ }$C. This cooling action is scheduled to be small due to the peak price for balancing cost saving and thermal comfort. Immediately following this small cooling action, one maximum cooling action at 8 p.m. takes place at the shoulder price (i.e., delayed cooling when the price decreases) to ensure the room temperature can be cooled to the desired one during sleeping. At Day 2, the proposed HEMS suggests a donning action at 6 am, which is similar to that at Day 1. As the temperature increases to 24.5 ${}^{\circ }$C at 3 p.m. at Day 2, a doffing action is proposed by the HEMS, and a small HVAC power is decided to cool the room at 4 p.m. subsequently. Then, a sequence of cooling actions occurs after the peak price period between 8 p.m. and 11 p.m. Similar to the baseline case, low-frequency cooling actions are observed during the sleeping time from 10 p.m. at Day 1 to 6 a.m. at Day 2. The results in Figure 9 and Figure 10 demonstrate that the occupant’s clothing behaviors can provide the HEMS with an additional dimension of decisions, which, in turn, enhances the cooling flexibility of the HVAC by better utilizing the thermal storage of the residential house while considering the occupant’s thermal comfort. The results here are consistent with those in the authors’ prior work, which takes into account the donning and doffing into HEMS for both genders in different seasons (but without PV, EV, or EV SOC concern). The result in [30] showed that if the occupant follows the optimal clothing decisions produced, 53.8% and 29.8% of daily electricity cost savings can be achieved for a summer-male scenario and a winter-female scenario, respectively. More simulation results can be found in [30].

Figure 11 shows the EV simulation results by the proposed HEMS. As seen, in contrast to the baseline case, the proposed HEMS enables the EV discharge actions given the efficiency of charge/discharge is 90% (i.e., the round trip efficiency of 81%). In Figure 11, the EV is discharged in the first two hours at Day 1 when the actual SOC is greater than the expected SOC for making a profit. Notice that according to Equation (12), the EV SOC concern exists only at times when the expected SOC is greater than the actual SOC. The expected SOC increases when approaching the departure time, i.e., 8 a.m. at Day 1; thereby, a series of charging actions are presented to charge the EV SOC to 98.5 % at departure and to minimize the SOC concern. After the driving event at Day 1, the EV returns home with around 60% SOC, which is higher than the expected SOC of 20% at arrival, i.e., the minimum SOC. Note that the arrival time at Day 1 coincides with the peak price followed by the shoulder price. As a result, the HEMS takes the great opportunity to discharge the EV for monetary gains during the peak and shoulder time periods. When the peak and should prices are over, the HEMS decides to charge the EV to satisfy the SOC requirement on Day 2 and to keep the zero SOC concern. At 10 a.m. on Day 2, the EV reaches 98% of SOC immediately before the shoulder price and stops charging. At 3 p.m. and 4 p.m., two discharge actions that bring the SOC to 93% are observed for a profit due to the peak price; however, those discharge actions cause a little EV SOC concern, but it does not affect the occupant’s upcoming driving event. After the EV returns at Day 2, the low expected SOC provides another chance for the HEMS to discharge the EV for more economic benefits. At the end of the cosimulation, the actual SOC is 45%, which is still higher than the expected SOC. The results in Figure 11 show that the proposed HEMS is capable of finding a balance between the occupant’s monetary gains and the EV SOC concern by optimally charging and discharging the EV. When compared with the baseline, it should be noted that the EV SOC at the end of two-day co-simulation is 45%, which is much lower than that (100%) in the baseline. The schedule by the proposed HEMS is reasonable since no driving event at Day 3 is given in the two-day cosimulation. Consequently, the difference at the ending SOC needs to be considered when comparing the EV results. This is discussed in greater detail in

Table 4. Comparison of the occupant’s energy cost, thermal comfort, clothing levels, and EV SOC concern between the baseline and proposed HEMS.

	Baseline	Proposed HEMS (Modified EV Schedule)
Avg. PMV	−0.09	0.24
Avg. Clo. Level	1.39	1.47
Avg. EV Concern	0%	1.17%
Endtime Actual SOC	100%	55% (100%)
Tot. Energy Cost	$25.40	−$0.99 ($8.01)

.

The power exchange of the proposed HEMS is shown in Figure 12. Unlike the baseline case (see Figure 8), there is no power purchase from the grid during the peak price on both days in the HEMS-optimized case. This leads to drastic cost savings. Additionally, three power purchase actions (i.e., 8 p.m. at Day 1, noon, and 8 p.m. at Day 2) are observed due to the high HVAC power at the shoulder price. Nevertheless, the EV discharge in the following base price periods could compensate for those costs. The comparative results between Figure 12 and Figure 8 demonstrate that the proposed HEMS significantly change the power exchange between the house and the grid by altering the actions from HVAC and EV, while considering the occupant’s thermal comfort, clothing behaviors, and EV’s SOC concerns.

3.4. Comparison and Discussion

Table 4 compares the occupant’s energy cost, thermal comfort, clothing levels, and EV SOC concern between the baseline and proposed HEMS. As discussed before, although the EV actions by the proposed HEMS are completely reasonable, there are differences in the ending SOC between the baseline and the proposed HEMS. Therefore, additional charge actions are included to increase the EV to the same level, i.e., 100%.

In Table 4, the numbers in the parenthesis represent the results with modified EV schedules to facilitate the comparison. As seen, the proposed HEMS leads to slightly higher average PMV and average EV concern as opposed to those in the baseline, indicating slightly higher thermal discomfort and EV SOC concern. However, for the proposed HEMS, the resultant PMV value of 0.24 is still less than the discomfort threshold of 0.5, and the EV SOC concern of 1.17% is quite low, meaning the occupant is still quite thermally comfortable and has little EV SOC concern. Additionally, the average hourly clothing levels in both cases are similar, signifying there is no significant change in the occupant’s clothing behaviors. The most prominent difference lies in total energy costs. As seen, the proposed HEMS reduces the energy cost from $25.40 in the baseline to $8.01 to the HEMS with modified EV schedules, representing a 68.5% of energy cost saving. Notice that this study only focuses on the controllable appliances while the energy consumption of noncontrollable appliances (e.g., dishwasher, water heater, etc.) is exactly the same in both the baseline and the proposed HEMS simulations. Although the 68.5% is higher than the actual percentage savings of a residential house due to the exclusion of the power consumption from noncontrollable appliances, the total dollar amount of savings for a two-day simulation is quite substantial.

To understand how the energy cost is drastically reduced, Figure 13 shows the breakdown of the total energy cost in the baseline and proposed HEMS under the modified EV schedule. The negative sign denotes a monetary gain from selling the electricity to the grid. It is seen that the energy cost for the HVAC is much lower in the proposed HEMS. The difference is mainly due to the delayed cooling to avoid the peak price in tandem with the optimized clothing behaviors. In the proposed HEMS, both the cost for charging the EV and the monetary gain for discharging the EV are much higher than those in the baseline. These results here suggest that even with 81% of round-trip efficiency, optimally charging, and most importantly, discharging the EV by the proposed HEMS can profoundly reduce the energy cost under the ToU. The overwhelming cost-saving benefits, in the long run, can well justify the necessity of using V2H services in the future, taking into account the decrease in the cost and the increase in the efficiency of the EV battery.

3.5. Comparison of One-Week Simulations

To demonstrate the efficacy of the proposed HEMS on a longer scheduling horizon, one-week simulations with and without the proposed HEMS were performed. Figure 14 shows the outside temperature, EV driving events, and electricity tariff for the one-week simulation. Figure 15 depicts the one-week simulation results for both the baseline and the proposed HEMS. As seen, the profits from PV generation are identical between the baseline and the proposed HEMS simulations. The HVAC cost is comparably small due to the lower outside temperature at Days 5–7. Analogous to previous two-day simulations, the EV charging still dominates the total electricity consumption in the one-week simulation. It can be seen that the proposed HEMS can reduce the electricity cost from $74.3 to $39.93, a 46.3% of cost reduction from the baseline simulation, representing a significant cost saving for residential occupants.

4. Conclusions

In this paper, the proposed HEMS integrates residential PV and EV into the HVAC scheduling in an occupant-centric manner. To the best of authors’ knowledge, the proposed HEMS, for the first time, takes into account the occupants’ thermal comfort, clothing behaviors, and EV’s SOC concerns. A stochastic ADP HEMS model is proposed to optimally determine the setpoints of HVAC, occupant’s clothing decisions, and the EV’s charge/discharge schedule. The uncertainty in the outside temperature, PV generation, and EV’s arrival SOC is considered. The nonlinear and nonconvex thermal comfort model, EV SOC concern model and clothing behavior model are embedded in the ADP-HEMS. An MPC framework is further adopted to simulate a residential house under the ToU tariff. Cosimulations are conducted to show the validity of the proposed ADP-HEMS. The proposed ADP-HEMS is compared with a baseline case that represents the current operational practice.

The simulation results show that the energy cost can be saved and a high level of the occupant’s comfort can be retained by comprehensively incorporating the occupants’ thermal comfort, clothing behaviors, and EV’s SOC concerns in the HEMS model. Such a user-centric manner can bring added value to the residential occupants, which, in turn, increases the adoption rate of the HEMS. In addition, when the EV discharge is enabled by the HEMS in the context of V2H, it can drastically save the electricity cost by optimally charging and discharging the EV under the ToU. Future work will focus on improving the proposed ADP-HEMS model with the help of deep machine learning techniques for a variety of applications in next-generation residential buildings.