Framework design and optimal bidding strategy for ancillary service provision from a peer-to-peer energy trading community ☆

As an innovative paradigm for electric power systems with a high penetration of distributed energy resources, peer-to-peer (P2P) energy trading enables direct energy trading between end customers, which is able to facilitate local power and energy balance and potentially support the operation of bulk power systems. In this paper, a framework was proposed to enable ancillary service provision from a P2P energy trading community, creating additional value for both customers in the community and power systems. Specifically, an ancillary service provision mechanism was designed along with P2P energy trading and residual balancing mechanisms to enable the power utility to obtain ancillary service from customers in a P2P energy trading community. Furthermore, the optimal bidding strategy of customers was figured out to maximize their benefits in the proposed mechanisms. Simulation studies were conducted based on a residential community in Great Britain. The results show that the proposed ancillary service mechanism can enable the power utility to obtain a significant or required amount of ancillary services of different types. The proposed mechanisms and optimal bidding strategy can achieve Pareto improvement for the revenue of each customer and result in significantly higher social welfare for the whole community. It is also revealed that increasing ancillary service prices and installation rate of electric vehicles can increase the total amount of ancillary service provision and thus bring higher revenue for the customers in the community. By contrast, increasing installation of PV systems does not necessarily increase the amount of service provision.


Introduction
Conventional electric power systems are characterized by centralized management and unidirectional power flow. From the perspective of power flow, a vast majority of electricity is generated by centralized large generators, transmitted through transmission and distribution networks, and finally distributed to end users. From the perspective of capital flow in deregulated electricity markets, electricity retailers buy ☆ Information on the data underpinning the results presented here, including how to access them, can be found in the Cardiff University data catalogue at http://do i.org/10.17035/d.2020.0114344992. and industrial projects on P2P energy trading, most existing work focuses on enabling and facilitating local energy trading itself, but not on manipulating the way of P2P energy trading to make the participants to provide ancillary services to support the operation of power systems. In some studies such as [6], [37] and [38], it is revealed that P2P energy trading can facilitate local balance of power and energy and reduce the peak loads, thus potentially being beneficial to the power systems. However, these potential benefits shouldn't be treated as a 'by-product' of conducting P2P energy trading. Instead this should be designed in a systematic way to contribute to the power system operation.
As proposed by T. Morstyn et al. in [39], self-organizing prosumers in P2P energy trading can further provide ancillary services for power systems. This concept is termed as 'federated power plant', which has the potential to 'address social, institutional and economic issues faced by top-down strategies for coordinating virtual power plants, while unlocking additional value for P2P energy trading' [39]. Although this concept has been proposed, there has been few solid studies depicting how to realize this in detail. The study in this paper aims to make the first attempt in this direction, through designing a framework for ancillary service provision from a P2P energy trading community. The framework is composed of three mechanisms, i.e. P2P energy trading mechanism, residual balancing mechanism and ancillary service provision mechanism.
The key research questions to be addressed in the paper are: 1) How does a power utility obtain ancillary service from customers in a P2P energy trading community? 2) What should customers do to maximize their benefits, when participating in P2P energy trading and ancillary service provision at the same time? 3) What are the outcomes when customers participate in P2P energy trading and ancillary service provision at the same time? Are they better or worse off?
To address the first question, an ancillary service provision mechanism was designed along with P2P energy trading and residual balancing mechanisms. To address the second question, the optimal bidding strategy of customers in the proposed ancillary service mechanism was figured out. Case study was conducted, and the simulation results verify the proposed methods and show the outcomes in terms of the benefits of customers, which addresses the third research question.
It is worth noting that ancillary service is a broad concept, which includes a wide range of services with various time scales (e.g. seconds, minutes, hours and even longer) and various purposes (e.g. frequency response, operating reserve, voltage support, constraint management, and peak shaving / valley filling). Ancillary services can be provided by the resources at both the supply and demand sides, and when at the demand side, ancillary services are usually provided through demand response. The ancillary services considered in this paper are those that can be provided through generation/demand increase/reduction from demand-side peer-to-peer energy trading communities, with the time scales ranging from several minutes to several hours. The ancillary services considered in this paper have some level of generality, because they cover a range of time scales and are not associated with specific purposes (e.g. they can be used for voltage support, operating reserve and peak shaving as long as the time scales fit). However, there are also some other ancillary services that have not been covered in this paper, such as frequency response with a second-by-second time scale, which are future research topics.
It is also worth noting that the characteristics and roles of customers at the demand side of electric power systems have changed with the connection of distributed generators and energy storage systems. For customers without any distributed generators, they are still conventional consumers. However, for customers with distributed generators, they can play different roles in electric power systems at different time periods, either being producers or consumers, depending on whether the onsite generation can fully satisfy the demand. Therefore, customers with distributed generators are widely referred to as 'prosumers' in many existing studies. In this paper, in the P2P energy trading mechanism, prosumers (who act as producers) directly sell their surplus generation to consumers or other prosumers (who act as consumers). In the ancillary service mechanism, both prosumers and consumers can provide ancillary services for the power utility. For providing ancillary services, consumers can increase/decrease their power consumption, while prosumers can increase/decrease their power generation or consumption. In this paper, we use the general term 'customers', which can include both prosumers and consumers.

Methodological framework
In this paper, a framework for ancillary service provision from a P2P energy trading community is presented. The overall framework and the associated timeline are illustrated in Fig. 1 and Fig. 2.
As shown in Fig. 1, the proposed framework is composed of three mechanisms, which are executed sequentially across the timeline as shown in Fig. 2. Firstly, the CDA mechanism (shown as the green dotted circle) acts as the P2P energy trading mechanism, where the customers in the community reach P2P energy trading agreements with each other. However, not all the generation / electricity demands are matched in the community, and the residual generation/demand will be balanced by the power utility in the proposed residual balancing mechanism (shown as the yellow dashed circle). After that, in the ancillary service mechanism (shown as the red dashed circle) the power utility will assess its operational needs and issue incentive signals for obtaining ancillary services, and the customers will respond to the signals by submitting bids for providing ancillary services based on the consideration of maximizing their respective economic benefits. The power utility will then assess the bids and make purchase decisions. It is worth noting that the agreements reached in the CDA and residual balancing mechanisms need to be adjusted for providing ancillary services.
Without loss of generality, the power utility is assumed to play the roles of both power system operator and electricity supplier/retailer, as the fact in many parts of the world (e.g. many places in the U.S. and China). By contrast, in some other parts of the world (e.g. the UK and some other places in the U.S. and China), the roles of power system operator and electricity supplier/retailer are played by separate companies. In this case, the proposed framework is still applicable, but the roles played by the united power utility will be played by separate companies. For example, the 'residual balancing' can be provided by the electricity supplier/retailer company and the ancillary services can be provided for the power system operator.
As shown in Fig. 2, the P2P energy trading, residual balancing and ancillary service mechanisms studied in this paper are all forward markets. The associated real-time markets and the settlement process considering the uncertainties of renewable power generation and customer behaviors are not within the scope of this paper. As shown in Fig. 2, for a future period of time, e.g. one day (shown as the 'Target Time Period T'), the agreements for P2P energy trading, residual balancing and ancillary services can be made as much as T ad in advance, from the 'Gate Opening' time to the 'Gate Closure' time. The time periods allowing to make these agreements are called 'Agreement Periods' in this paper, and it is noted that the Agreement Periods for P2P energy trading, residual balancing and ancillary services come in sequence throughout the timeline.
Note that the three mechanisms proposed in this paper are operated one after one in sequence, which may lead to some level of suboptimality from the whole system perspective. For example, in the wholesale markets in some countries, the energy and ancillary service markets are operated and cleared simultaneously, so that the power system operator and market participants are able to optimize their decisions considering the energy transaction and ancillary service provision at the same time. This paper made the first attempt to design an ancillary service mechanism along with P2P energy trading, but whether the mechanisms proposed could achieve the global optimum in terms of resource allocation remains to be further studied.
It is also an open question whether the energy market should be operated and cleared with the ancillary service market simultaneously. Operating energy and ancillary service markets simultaneously may increase the market complexity, and sometimes there are some nontechnical reasons, e.g. the institutions in the power industry, which may have an impact. For example, in reality in Great Britain, the power system operator (i.e. the National Grid Electricity System Operator) purchases different types of ancillary services through different routes (e.g. electronic tendering or direct bilateral contracts), being separate from the wholesale market. Furthermore, again setting the example in Great Britain, ancillary services may be purchased by different independent parties (e.g. the power system operator, distribution network operator or electricity suppliers), so it is difficult to operate a united market incorporating both energy and various ancillary service markets. In this sense, separate design of energy and ancillary service markets is more flexible and easier to modify and add on.

P2P energy trading mechanism and customers' bidding strategy
In this section, the P2P energy trading mechanism studied in this paper, i.e. the CDA mechanism, will be presented. The customers' bidding strategy in the CDA will also be described. Note that the work of this section is a combination of several existing methods (including the CDA in [40], optimization of customers' operational schedules [37] and bidding strategy [41]), without original innovation in each single method. However, the combination itself has some value, and furthermore, the work described in this section is the basis for the residual balancing and ancillary service mechanisms originally proposed later.

Continuous double auction mechanism
The CDA mechanism matches buyers and sellers who are interested in trading and is deemed as a highly efficient mechanism. It is widely used in the trading of various types of commodities, such as stocks as well as electricity. Recently, there have been a number of studies proposing to adopt CDA for P2P energy trading, such as those in [42][43][44][45]. In this paper, the CDA described in [40] is adopted as the mechanism for P2P energy trading. As presented in Section 2, customers trade energy with each other to reach P2P energy trading agreements by participating in the CDA mechanism. For any time slot t in the Target Time Period T, in the corresponding Agreement Period of CDA, customers submit bids or asks according to their roles being 'buyers' (who have electricity deficit at t) or 'sellers' (who have electricity surplus at t). A bid, which is submitted by a buyer, is represented by o b,t (b t , π b,t , σ b,t , τ b,t ), which means that the buyer b t would like to by the σ b,t (kWh) amount of electricity at the price π b,t (£/kWh), and the bid arrives at the 'exchange' at the time τ b,t . Note that the exchange is a centralized or decentralized platform set for P2P energy trading within the community. Similarly, an ask, which is submitted by a seller is represented by o s,t (s t , π s,t , σ s,t , τ s,t ).
Within the Agreement Period of CDA, bids and asks will arrive at the exchange asynchronously. After they arrive, the bids and asks are allocated in 'order books'. Note that for the bids and asks associated with each time slot t in the Target Time Period T, there is one order book. In each order book, the bids are sorted in the descending order of the bid prices π b,t and the asks are sorted in the ascending order of the ask prices π s,t . For both bids and asks, if several ones have the same prices, they are sorted based on the arrival time -the later a bid/ask arrives, the lower it ranks.
Every time a bid/ask arrives at the exchange, it will be allocated in the corresponding order book based on the above-described principles, and then the exchange will try to match the bids and asks in the order book. If the following relationship is satisfied for a bid o b,t and an ask o s,t : then the following amount of electricity will be matched: The trading price for the matched amount of electricity will be decided as The bid/ask that is fully matched will be removed from the order book, and the bid/ask that is not fully matched will be updated by o' s,t = ( where σ' s,t = σ s,t − σ b,t . The matching process will go from the top to the bottom of the order book, and will end once no matching can be made. The matching process will be triggered every time a new bid/ask arrives. When an amount of demand and generation, σ matched,t (kWh), is matched at the price π t (£/kWh), a P2P energy trading agreement, <σ matched,t , π t >, is established between the corresponding buyer and seller, specifying the quantity and price to be traded.
The process of CDA is summarized in Appendix A.

Bidding strategy of customers in the P2P energy trading mechanism
Bidding strategies of customers need to be studied to describe customers behaviors in the proposed P2P energy trading mechanism and therefore the performance and outcome of the mechanism can be evaluated.
First of all, each customer optimizes its operational schedule of energy storage and flexible loads to minimize the electricity cost, considering the availability of onsite generation and time-varying electricity prices. The abstract formulation of the optimization problem is presented as follows: f (x) = 0, x⊆X. (12) In Equation (8), p t (£/kWh) is the electricity price, which may take different values for importing and exporting electricity; NL i,t (kW) is the net load of the customer i at the time step t. Equation (9) calculates NL i,t by taking the sum of electricity consumption (positive values) and generation (negative values) of all the devices in the customer's premise at the time step t, where N represents the total number of devices in the premise and j is the device index. Equations (10) and (11) are the equality and inequality constraints about the device characteristics and customer comfort. Equation (12) specifies the range of decision variables. Detailed formulation refers to the Appendix, which is based on [37].
By solving the optimization problem, each customer (usually through home energy management systems), indexed by i, is able to obtain the operational schedules, and thus obtain the net load profile throughout the Target Time Period T, denoted as NL * i = {NL * i,t | t = 1,2, …, T} where NL * i,t > 0 represents that the premise imports electricity from the power utility while NL * i,t < 0 represents that the premise exports surplus generation to the power utility.
As proposed in [40], the customers decide the amount of electricity to be sold/bought in the P2P energy trading mechanism based on their net load profiles. For any time slot t, a customer will submit a bid with the amount of electricity to be bought being NL * i,t , if NL * i,t > 0. Similarly, the customer will submit an ask with the amount of electricity to be sold being The bid/ask prices are assumed to be made given the assumption that the customers act as 'zero intelligent plus (ZIP)' traders [41]. The assumption of ZIP traders can well mimic the behaviors of human traders in stock markets. As a ZIP trader, each customer will set a budget constraint, [p FiT t ,p retail t ] to ensure that it will not be worse off by participating in the P2P energy trading compared to trading with the power utility. Initially, for each customer, the price for its bid/ask is decided uniformly at random within the budget constraint. During the double auction process, the customers will dynamically adjust their bid/ ask prices based on matching results of previous bids/asks. Specifically, the bidding strategy of customers in the P2P energy trading mechanism is presented in Appendix B.
After running the CDA, a number of P2P energy trading agreements will be made between customers. In an abstract form, the set of P2P energy trading agreements regarding the customer i for the time slot t, AGRT P2P i,t , is marked as where AGRT P2P ij,t represents the P2P energy trading agreement between the customers i and j for the time slot t, which specifies the amount, price and direction (i.e. buying or selling) of the energy trading. J is the set of customers that have P2P energy trading agreements with the customer i.
In summary, each customer decides its bid in the P2P energy trading mechanism in two steps. In the first step, the customer solves the optimization problem, formulated by Formulas (8)- (12), to obtain the load schedule, so that the amount of electricity to be sold/bought in the P2P energy trading mechanism can be decided. The electricity prices used in Formula (8) are just the retail prices / feed-in tariff rates offered by the power utility. In the second step, the customer decides the price of its electricity to be sold/bought in the P2P energy trading mechanism following the rules assuming that it is a ZIP trader.
It is worth noting that this bidding strategy is not optimal for the customers, due to two reasons. First, in the P2P energy trading mechanism, what finally decides the bill of a customer is the prices specified in the P2P energy trading agreements, rather than the electricity prices offered by the power utility used in Formula (8). Second, the customers decide the amount of electricity to be sold/bought and the bid/ask price in two separate steps rather than jointly at one step. We adopted this strategy from [40] , which is feasible although not optimal. How to propose a better (even optimal) bidding strategy in the CDA-based P2P energy trading mechanism remains to be studied, but it is not the focus of this paper, which is to enable ancillary service provision from a peerto-peer energy trading community.

Residual balancing mechanism
As presented in Section 3, during the Agreement Period of CDA, a number of P2P energy trading agreements will be made between customers for the time slots in the Target Time Period T. Nevertheless, it is highly possible that there are still several bids and/or asks in the order book, which are not matched due to their prices are not matched with each other. In the original design in [40], the corresponding amount of demand and/or generation will be curtailed. However, this design will raise the risks of demand/generation curtailment for customers, and thus may hinder some of them from participating in the P2P energy trading in the first place.
In order to address this concern, in this paper, an alternative residual balancing mechanism is proposed, in which the electricity demand of the un-matched bids in the CDA is set to be fully supplied by the power utility, and the generation of the un-matched asks in the CDA is also set to be fully purchased by the power utility. That is, the power utility will act as the 'residual balancer' for the P2P energy trading community. Furthermore, in this paper, it is assumed that the power utility supplies the residual demand at the electricity retail price p retail t (£/kWh) and purchases the residual generation at the Feed-in Tariff (FiT) rate p FiT t (£/kWh), given the assumption that the entire P2P energy trading community acts as a 'price taker' in the electricity wholesale market so that the power utility can treat the P2P energy trading community as a normal customer. This assumption is basically true at the moment, when the number and scale of P2P energy trading communities are still very small in the whole power system. If P2P energy trading communities can no longer be taken as price takers in the future, their interaction with the wholesale market needs to be taken into consideration by the power utility, as discussed in [5] and [46], but this is not the focus of this paper.
In summary, in the Agreement Period of the residual balancing mechanism, for the time slots in the Target Time Period T, additional peer-to-grid (P2G) residual balancing agreements will be made between customers and the power utility, so that the electricity surplus/deficit of customers, which are not purchased/supplied in the CDA, can be balanced by the utility.
In an abstract form, the set of P2G residual balancing agreements regarding the customer i throughout the Target Time Period T, where AGRT P2G i,t represents the P2G residual balancing agreement between the customer i and the power utility for the time slot t, which specifies the amount, price and direction (i.e. buying or selling) of the energy flow for residual balancing.

Ancillary service mechanism and optimal bidding strategy of customers
In this section, the mechanism for customers in the P2P energy trading community to provide ancillary services for the power utility will be proposed. The corresponding optimal bidding strategy of customers will also be presented.

Ancillary service mechanism
After the Agreement Periods of CDA and residual balancing mechanisms, the P2P energy trading agreements between customers and the P2G residual balancing agreements between customers and the power utility have been fixed. These agreements will be made available for the power utility as the basis for it to assess its schedules for operating power systems.
Based on the results of the assessment, the power utility may foresee some operational problems such as over voltage or congestions in the power networks. Moreover, the power utility may identify some possible actions that may improve the operational economy and security of power systems, such as peak shaving. In these circumstances, the power utility is able to procure ancillary services from customers in the P2P energy trading community to help the operation of power systems. The specific mechanism is presented as follows.

Categories of ancillary services
In a general sense, ancillary services can refer to any services and functions provided to power utilities that 'facilitate and support the continuous flow of electricity so that supply will continually meet demand' [47]. There are a wide range of ancillary services for different purposes and with different time resolutions. For example, according to the U.S. Federal Energy Regulatory Commission (FERC), ancillary services include 1) scheduling, system control and dispatch, 2) reactive power supply and voltage control from generation sources, 3) regulation and frequency response, 4) energy imbalance service, 5) operating reservespinning reserve, and 6) operating reservesupplemental reserve services [48]. For another example, in the UK, the National Grid Electricity System Operator procures more than 30 types of ancillary services [47].
In this paper, two general categories of ancillary services are considered to be procured by the power utility, of which the time resolution can range from several minutes to several hours. The two categories of ancillary services considered are 1) demand reduction / generation increase (named as 'Type-1 AS'), and 2) demand increase / generation reduction (named as 'Type-2 AS'). Note that the generation increase/reduction and demand increase/reduction refer to the change of the net load of a customer as a whole, rather than the change of power output/consumption of specific appliances or devices within the customer's premise.

Procurement process
In this paper, a bidding process is designed to be used for the power utility to procure ancillary services from the customers in the P2P energy trading community. The whole bidding process will be conducted during the Agreement Period for Ancillary Service Mechanism as shown in Fig. 2. The specific steps are presented as follows.
Step 1 Publication of the demand and prices for ancillary services by the power utility As the first step, based on the assessment of the future operational status of power systems, the power utility will broadcast the total demand and prices of ancillary services to all the customers in the P2P energy trading community: where D AS is the set of the magnitude (kW) of the ancillary services required for the time slots within the Target Time Period T, and p AS is the set of prices (£/kW/h) that the power utility is willing to pay for the ancillary services provided by the customers. The power utility will also specify the type of ancillary services needed (either 'Type-1 AS' or 'Type-2 AS'). Note that how the power utility decides D AS and p AS involves comprehensive analysis of the power systems managed by the power utility, which is out of the scope of this paper.
Step 2 Bidding from the customers in the P2P energy trading community After receiving the demand and prices of ancillary services required by the power utility as described in Step 1, the customers will run optimization and submit their bids to the power utility. A bid is in the form of where B AS i is the bid of customer i for providing ancillary services for the Target Time Period T, and b AS i,t is the magnitude (kW) of the service that the customer would like to provide for the time slot t. I* is the set of the customers who submit the bids.
Note that the bids only include the amount of services to be provided. In the design of this paper, the prices of the services are fixed by the utility as published in Step 1.
Details about how a customer makes the optimal bid will be presented in Section 5.2.
Step 3 Decision on the service procurement by the power utility After collecting the bids from the customers in the P2P energy trading community (a time window can be defined in the Agreement Period for this purpose), the power utility needs to decide which of and to what extent the bids are to be accepted/declined. To achieve the fairness, the power utility in principle should distribute the amount of services to be procured averagely on the customers who have submitted the bids. However, the amount of services to be procured from each customer (called as the 'quota') calculated in this way may be higher than some customers' bids, and thus the surplus quotas can be given to other customers in need. With the above considerations, the process detailed in Fig. 3 is proposed for the power utility to decide the amount of services to be procured from the customers.

Adjustment of P2P energy trading and P2G residual balancing agreements
The ancillary services to be provided by the customers are measured by taking the net load profiles calculated in Section 3.2 as the baseline. To provide ancillary services (either generation increase/decrease or demand increase/decrease), the customer's actual net load profile will deviate from that calculated in Section 3.2. However, the net load profiles calculated in Section 3.2 have been agreed to be satisfied by other customers through the P2P energy trading agreements made in the CDA and/or by the power utility through the P2G residual balancing agreements made in the residual balancing mechanism. The deviation of the net load profiles for providing ancillary services will inevitably result in the breach of pre-made P2P energy trading agreements and/or P2G residual balancing agreements.
Therefore, after the power utility makes the procurement decisions and reaches ancillary service agreements with customers, some further arrangements need to be made to address the conflicts between the newly made ancillary service agreements and the pre-made P2P energy trading and/or P2G residual balancing agreements. The principles of the arrangements are summarized in Table 1.
The application and mathematical representation of these principles Fig. 3. Distribution of the ancillary service procurement among customers.
will be presented in the Section 5.2.1 followed.

Cost and benefit analysis
Customers decide whether and to what extent to participate in the ancillary service mechanism by weighing the associated benefit and cost. Therefore, a cost and benefit analysis is conducted for customers to provide different types of ancillary services in the proposed mechanism.
Cost and benefit analysis for Type-1 AS Without loss of generality, it is assumed that the magnitude of Type-1 AS procured from a customer i for the time slot t is q i,t (kW), as decided through the procedure shown in Fig. 3. Recall that the Type-1 AS refers to the demand reduction or generation increase of the customer i as a whole, which is measured with the baseline being the net load profile of the customer, NL * i,t , calculated in Section 3.2. First of all, because of the provision of the ancillary service, the customer will receive reward from the power utility, as presented in Section 5.1.2. The associated benefit, ΔB AS qi,t , is expressed as If NL * i,t <= 0, to provide Type-1 AS, the customer needs to export more electricity, as much as q i,t . As specified in Section 5.1.3, the power utility will purchase all the q i,t at the Feed-in Tariff rate. Therefore, the customer will receive extra benefit, ΔB EXP q i,t (£), as calculated by If NL * i,t > 0, q i,t can be divided into two parts, i.e. where In Equations (25)- (27), q' i,t (kW) represents the 'demand reduction component' that decreases the NL * i,t towards 0, while q'' i,t (kW) represents the 'generation increase component' that drives the NL * i,t towards to a more negative value (i.e. changing the customer i from a consumer to a producer). For example, if it is assumed that NL * i,t = 5 kW and q i,t = 8 kW, then q' i,t = 5 kW and q'' i,t = 3 kW, and the net load of the customer i after service provision will become − 3 kW (i.e. 5 kW -8 kW). That is, the provision of 8 kW Type-1 AS will change the customer from a consumer that imports 5 kW electricity to a producer that exports 3 kW electricity.
For another example, if it is assumed that NL * i,t = 5 kW and q i,t = 3 kW, then q' i,t = 3 kW and q'' i,t = 0 kW, and the net load of the customer i after service provision will become 2 kW (i.e. 5 kW -3 kW). That is, the provision of 3 kW Type-1 AS will reduce the electricity consumption of the customer by 3 kW but will not change the customer from a consumer to a producer.
For the additional electricity generation of the customer, i.e. q'' i,t , because of the provision of Type-1 AS, the power utility will purchase it at the Feed-in Tariff rate according to Section 5.1.3. Therefore, the additional benefit because of this, ΔB EXP For the demand reduction component q' i,t , it can be further divided into two parts: where q '− I i,t (kW) represents the component that is originally supplied by other customers through P2P energy trading agreements (denoted as (kW) represents the component that is originally supplied by the power utility through P2G residual balancing agreements (denoted as q ''− II . For the reduction of the demand which is originally supplied by the power utility, i.e. for q ''− II i,t , no penalty cost needs to be paid by the customer for the breach of the P2G residual balancing agreements, according to Section 5.1.3. Furthermore, the customer will no longer need to pay the electricity cost for the demand that has been reduced. This can be seen as a 'benefit' and is calculated by For the reduction of the demand which is originally supplied by the other customers through P2P energy trading agreements, i.e. for q '− I i,t , the customers need to pay penalty cost to those customers. The q '− I i,t may be originally supplied by multiple P2P energy trading agreements, which is expressed by where g represents a P2P energy trading agreement that contributes to q '− I i,t . According to Section 5.1.3, the level of penalty cost equals to the revenue loss of the relevant customers who have to directly trade with the power utility instead because of the demand reduction. Therefore, the penalty cost that needs to be paid by the customer i in this regard is expressed as Recall that π g is the price specified in the P2P energy trading agreement. On the other hand, the reduction of q '− I i,t makes the customer Table 1 The arrangements for addressing the conflicts among P2P energy trading, P2G residual balancing and ancillary service agreements.
The change in electricity production/consumption for providing ancillary services no longer need to pay the relevant electricity cost specified in the relevant P2P energy trading agreements. This can be seen as a sort of benefit, and is expressed as Summarizing Equations (23)-(33), the total cost and benefit of the customer i who provides q i,t of ancillary service are expressed as Cost and benefit analysis for Type-2 AS Similar analysis can be made to calculate the cost and benefit of customers in providing Type-2 AS. For simplicity, they are not presented in detail in this paper.

Optimal bidding strategy of customers
As presented in Section 5.1.2, customers can submit bids on the magnitude of ancillary services they would like to provide to the power utility, based on the types, demand and prices of the ancillary services issued by the power utility. In this paper, it is assumed that customers will make the bidding decisions to maximize their economic benefits (i. e. minimize their net costs), considering the cost and benefit analysis presented in Section 5.2.1.
Specifically, after the power utility publishes the type, demand and prices of ancillary services needed as detailed in Section 5.1.2, a customer i will run the following optimization to decide whether / how much to bid in the ancillary service mechanism: s.t. Equations (9)-(12), (34), (35), In the optimization problem, T' represents the set of time slots for which the power utility calls for ancillary service provision; NL t is the new net load of the customer to be decided; NL * t is the net load of the customer before providing any ancillary service as calculated in Section 3.2; b AS t (kW) represents the bid submitted by the customer to the power utility for ancillary service provision; ΔB b AS t and ΔC b AS t are the benefit and cost of ancillary service provision regarding b AS t , as calculated by Equations (34) and (35); q t represents the quota of ancillary service published by the power utility as explained in Section 5.1.2. Note that in the presentation of the optimization problem, the subscript i for representing the customer i is omitted for simplicity.
In the optimization problem, the operation of flexible devices is rescheduled to have the new net load NL t for reaching the minimum net cost, considering the cost and benefit of ancillary service provision. Both the time slots within and beyond the time slots when the ancillary services are called are considered in the optimization, because the power consumption of some devices (e.g. EVs) may be shifted from one time slot to another, which may result in the change of net cost both within and beyond T'. The decision variables are the operational states of flexible devices, which will decide the new net load NL t .
The objective function of the optimization problem (36) is the change of the net cost of customer i throughout the Target Time Period T, compared to that without any ancillary service provision as calculated by (8). The objective function is composed of the change of the net electricity cost beyond T' (i.e. within T\T'), and the net cost change due to ancillary service provision within T' based on the cost and benefit analysis conducted in Section 5.2.1.
As for the constraints, Equations (9)-(12) are the constraints regarding the flexible devices. Equations (34)- (35) are to calculate the cost and benefit because of ancillary service provision. Equation (37) shows the relationship between the new net load and the bid. It is seen that if it is a Type-1 AS, the bid will be a demand reduction or generation increase, and similarly, if it is a Type-2 AS, the bid will be a demand increase or generation decrease. Equation (38) requires that the bid submitted by the customer must be below the quota distributed by the power utility. Note that in the ancillary service procurement process as presented in Section 5.1.2, the optimization formulated in this section will be first run by each customer with setting q t as positive infinity to generate the original bids used for Step 2. Then the same optimization will be run again by the customers with the q t distributed by the power utility (referring to Step III of Fig. 3) to generate the final bids that will be procured by the power utility.
Remarks on deploying the proposed mechanisms in practice Note that the bidding processes for P2P energy trading and ancillary services involve intensive participation from customers, but residential customers may lack sufficient time, knowledge or interest to participate. Therefore, the practical deployment of the bidding mechanisms largely depends on the deployment of smart metering infrastructure and home energy management systems (HEMSs). Smart metering infrastructure enables automatic metering with a high time resolution, while HEMSs automatically control various appliances in smart homes and intelligently conduct bidding in electricity markets on behalf of customers. Customers only need to set their preferences occasionally through the interface of HEMSs [49]. As a result, with HEMSs, customers do not have to spend a lot of time or be very knowledgeable to participate. In recent years, as shown in another review paper of ours [50], there have been a rapidly increasing number of academic studies and industrial practice in trialing or deploying P2P energy trading within residential communities.

Case study
Ancillary service provision from a residential P2P energy trading community in Great Britain (GB) was studied to validate and assess the proposed mechanisms. The residential community was assumed to consist of 20 customers, with each customer equipped with flexible appliances and part of customers equipped with onsite PV generation and/or electric vehicles. Based on realistic statistics in GB, the CREST model [51] was used to generate the number, types and parameters of the appliances owned by each customer. As many as 34 types of appliances were included in the CREST model. The CREST model was also used to generate the occupancy and usage of appliances of each customer. For the customers with onsite PV generation, the CREST model was used again to generate the PV generation throughout the day. For the customers with electric vehicles, the parameters of electric vehicles and the travelling behaviors of customers were generated by randomly sampling from an EV database built on realistic statistics in GB [52].
In this case study, the length of the Target Time Period T was assumed to be as long as one day, before which the proposed P2P energy trading mechanism, residual balancing mechanism and ancillary service mechanism were executed in order. The length of each time step was assumed to be one hour. The simulation was conducted in a MATLAB environment and CPLEX was used to solve the mixed integer linear programming problems when needed.
Six specific cases with three scenarios for each case were designed and studied for validating and assessing the proposed mechanisms from different perspectives. They are summarized in Tables 2 and 3, with the detailed description followed.
The description of the cases is as follows: • Case 1 acts as the base case for verifying and assessing the validity and advantage of the mechanisms proposed in this paper, through comparing the performance of the three scenarios (S1-S3). It was assumed that half of the customers are equipped with onsite PV generation, and also half of the customers own electric vehicles. Whether a customer has onsite PV generation and electric vehicles was decided randomly and independently. The price of ancillary service was assumed to be 154.82 £/KW/h (with reference to the average remuneration level of the Short Term Operating Reserve service in the Great Britain [53]), and the type of ancillary service was assumed to be Type-2 AS. It was also assumed that the power utility would like to procure ancillary service for 4 h within the Target Time Period T (i.e. from 13:00 to 17:00), with no limit on the amount of ancillary service (i.e. as much as possible). • Case 2 examines the performance of the proposed mechanisms when a different type of ancillary service is provided. The settings of Case 2 were the same as those of Case 1, except that the Type-1 AS was considered. • Case 3 examines the performance of the proposed mechanisms when a limited amount of ancillary service is procured. The settings of Case 3 were the same as those of Case 1, except that the amount of ancillary service procured by the power utility was considered to have a limited value, which was 50 kW/h. • Case 4 assesses the impact of price levels of ancillary service. The settings of Case 4 were the same as those of Case 1, except that a range of ancillary service prices were considered. • Case 5 assesses the performance of the propose mechanisms with different installation rates of electric vehicles. The settings of Case 5 were the same as those of Case 1, except that different installation rates of electric vehicles were considered. • Case 6 assesses the performance of the propose mechanisms with different installation rates of PV systems. The settings of Case 6 were the same as those of Case 1, except that different installation rates of PV systems were considered.
For each case, there scenarios were studied, which are described as follows: • In Scenario 1 (S1), the customers were assumed to separately trade with the power utility at the retail price (when buying electricity) and the Feed-in Tariff rate (when selling electricity) in the conventional 'peer-to-grid (P2G)' mode. • In Scenario 2 (S2), the customers were assumed to trade with each other in the proposed P2P energy trading mechanism and with the residual balancing mechanism executed, but without the ancillary service mechanism executed. • In Scenario 3 (S3), the customers were assumed to participate in the proposed P2P energy trading mechanism, residual balancing mechanism and ancillary service mechanism in order.

Case 1: Base case assessing the performance of the proposed mechanisms
As presented in Table 2, this case acts as the base case and consists of three scenarios, for validating and assessing the performance of the proposed P2P energy trading, residual balancing and ancillary service mechanisms. The results are shown in Figs. 4 -7.
In Fig. 4, the daily revenues of each customer in the three scenarios are illustrated (note that positive revenues mean earning money while negative revenues mean paying bills). The blue solid line shows the revenue of each customer in the conventional P2G mode (S1), which acts as the reference case. The green dashed line with triangles shows the revenue of each customer with the proposed P2P energy trading and residual balancing mechanisms executed (S2). It is seen that for most customers the revenue in S2 is higher than that in the reference P2G mode (S1), demonstrating the advantage of the proposed P2P energy trading and residual balancing mechanisms. The magenta dotted line with circles shows the revenues further with the ancillary service mechanism executed (S3). It is seen that for a number of customers (e.g. Customers 2,7,8,11,13,15 and 16) their revenues are significantly higher than those in both S2 and S1, thanks to the reward in the ancillary service mechanism. This demonstrates the benefits of the proposed   Fig. 4. The revenues of each customer in Case 1.
ancillary service mechanism to customers. Moreover, it is worth noting that, as shown in Fig. 4, the revenue of each customer in S2 is always no lower than that in S1, and further, the revenue of each customer in S3 is always no lower than that in S2. This demonstrates that the proposed P2P energy trading and residual balancing mechanism are able to achieve Pareto improvement for each customer compared to the conventional P2G mode, and the proposed ancillary service mechanism can further achieve Pareto improvement for each customer compared to that without executing it.
For the overall residential community, as shown in Fig. 5, the social welfare (i.e. the total revenues of all the customers in the community) with P2P energy trading (S2) is 98% higher than that with the conventional P2G mode (S1). The social welfare further with the proposed ancillary service mechanism executed (S3) is 144% higher than that only with P2P energy trading (S2) and 194% higher than that with the conventional P2G mode (S1).
The supply and demand relationship among customers and between the customers and the external power grid is illustrated by Sankey diagrams as shown in Fig. 6.
As shown in Fig. 6(a), it is seen that the customers either sell or buy electricity to/from the external power grid in the conventional P2G mode (S1). By contrast, Fig. 6(b) shows that with the proposed P2P energy trading mechanism executed (S2), the demand of all the consumers in the community is changed to be supplied by other customers (acting as producers) in the community. It is also seen in Fig. 6(b) that the residual surplus generation of the producers will be exported to the power grid, demonstrating that the proposed residual balancing mechanism works well. Fig. 6(c) shows that, compared to Fig. 6(b), there are a lot of customers having been changed from producers to consumers and the demand of consumers has increased significantly, due to the provision of Type-2 AS (which is demand increase / generation reduction). The increased amount of demand is satisfied by the external power grid, which is consistent with that described in Section 5.1.3. In summary, Fig. 6 shows that the proposed P2P energy trading, residual balancing and ancillary service mechanisms have functioned in the expected way. Fig. 7 illustrates the total amount of ancillary service provided for different hours of the day.

Case 2: Performance with a different type of ancillary service provided
This case assesses the situation where a different type of ancillary service, i.e. Type-1 AS, is provided. The results are presented in Figs. 8-10.
In general, the results presented in Figs. 8-10 show that the validity and advantage of the proposed mechanisms still exist in the situation where a different type of ancillary service is provided. Specifically, Fig. 8 demonstrates that the proposed P2P energy trading and balancing residual mechanisms (S2) are able to achieve Pareto improvement for each customer compared to the conventional P2G mode (S1), and the proposed ancillary service mechanism (S3) can further achieve Pareto improvement for each customer compared to that without executing it (S2). Fig. 9 illustrates that the proposed mechanisms are able to result in improved social welfare for the overall community. Quantitatively, the social welfare with P2P energy trading (S2) is 98% higher than that with the conventional P2G mode (S1). The social welfare further with the proposed ancillary service mechanism executed (S3) is 85% higher than that only with P2P energy trading (S2) and 135% higher than that with the conventional P2G mode (S1). It is worth noting that the percentages of improvement in this case are lower than those in Case 1. This shows that customers might have different levels of flexibility for providing different types of ancillary services. Fig. 10 illustrates the total amount of Type-1 AS provided for the relevant hours, showing that the capability of the customers in the community to provide a different type of ancillary service (i.e. Type-1 AS).

Case 3: Ancillary service provision with limited amount
This case assesses the situation where a limited amount of ancillary service (being 50 kW/h) was needed by the power utility. Only one scenario, i.e. S3, was simulated in this case. The simulation results are shown in Fig. 11 and Fig. 12.
In Fig. 11, it is seen that for all the time steps where the ancillary service is needed, the total amount of ancillary service provided by the community is restrained below the limit 50 kW/h, compared to the unlimited case shown in Fig. 7. This demonstrates that the proposed procedure shown in Fig. 3 can effectively limit the total amount of service below the pre-defined limit. Fig. 12 compares the service provision from each customer in the situations with and without the total amount limit (50 kW/h). It is seen that with the limit, the amount of service of some customers was reduced to ensure that the total amount of service would not exceed the limit. Furthermore, it is worth noting that the reduction of the service was conducted for the customers providing the largest amount of services, guaranteeing the fairness among the customers and thus verifying the effectiveness of the proposed service distribution rules as presented in Fig. 3.

Case 4: The impact of ancillary service prices
For different levels of ancillary service prices provided by the power utility, the customers may make different decisions on whether or not and to what extent to provide ancillary service, based on the trade-off between the cost and benefit. Therefore, in this case, the performance of the proposed mechanisms is assessed given various levels of ancillary service prices. The prices range from 10% to 150% of the price in Case 1 (which is 154.82 £/KW/h based on the remuneration level of the Short Term Operating Reserve service in the Great Britain [53], called as 'based price' in the rest of this section) The results are shown in Fig. 13 and Fig. 14. Fig. 13 illustrates the social welfare of the overall community in different scenarios with various levels of ancillary service prices. In S1 and S2, the ancillary service mechanism was not executed so the social welfare will not change with the ancillary service prices (shown as the blue solid line and the green dashed line with triangles respectively). For S3 (shown as the magenta dotted line with circles), the social welfare of the community increases with the increase of ancillary service prices. The increase of social welfare is almost linear, with an obvious knee point at 70% of the base price in Case 1. Fig. 14 shows the total amount of ancillary service provided by the community given various price levels. It is seen that, within 10-60% of the base price and 70-150% of the base price, the amount of service increases marginally with the increase of service prices. This explains the near-linear increase of the social welfare in these two intervals as shown in Fig. 13. It is also observed in Fig. 14 that there is a sharp increase of the amount of service provision when the service price increases from 60% to 70% of the base price. This is because the benefit of providing ancillary service changes to be higher than the associated cost when the service price reaches 70% of the base price, so that a number of customers commit to provide ancillary service. The sharp increase of service amount also explains the knee point of the social welfare as shown in Fig. 13.

Case 5: The impact of installation rate of electric vehicles
Electrification of transportation is one important trend in many countries in the world [54]. The increasing number of electric vehicles will significantly increase the electric demand, and at the same time provide potential flexibility with smart charging and vehicle-to-grid (V2G) technologies. Therefore, this case assesses the impact of installation rate of electric vehicles in the community (10-100%) on the performance of the proposed mechanisms. The simulation results are shown in Fig. 15 and Fig. 16.
In Fig. 15, it is seen that, with increasing installation rate of electric vehicles, the social welfare of the whole community decreases in both S1 (with the conventional P2G mode) and S2 (with the proposed P2P and residual balancing mechanisms), because increasing numbers of electric vehicles increases the total electric demand of the community. However, it is worth noting that the social welfare in S2 is still always higher that of S1, demonstrating the advantage of the proposed P2P energy trading mechanism.
The situation with the ancillary service mechanism further executed (S3) is more complex. On one hand, as shown in Fig. 16, increasing installation rate of electric vehicles will bring increasing amount of demand flexibility, and thus the total amount of ancillary service provided will increase accordingly, also indicating that more reward will be obtained by the customers from the power utility due to the ancillary service provision. On the other hand, as analyzed previously, increasing numbers of electric vehicles will increase the total electric demand and thus will incur higher electricity cost as well. Therefore, the overall trend of the social welfare with increasing installation rate of electric      vehicles is not necessarily monotonical, but depending on comparison between the increasing reward due to increasing ancillary service provision and the increasing cost due to the increasing electric demand, shown as the magenta dotted line with circles in Fig. 15.

Case 6: The impact of installation rate of PV systems
This case assesses the performance of the proposed mechanisms given various installation rates of PV systems in the community (10-100%). The simulation results are shown in Fig. 17 and Fig. 18.
As shown in Fig. 17, it is seen that, in all the three scenarios (S1 -S3), the social welfare of the community increases with the increasing installation rate of PV systems. This is because increasing installation rate of PV systems increases the amount of local generation, which will result in higher revenues no matter the increased generation is consumed by customers themselves, sold to other customers or sold to the power utility.
However, as shown in Fig. 18, the increasing installation of PV systems does not necessarily result in increasing amount (and thus revenue) of ancillary service provision, because it does not bring more flexibility (in this paper, the PV systems were assumed to always generate the maximum power possible and the PV generation curtailment was not considered). The amount of service provision given different installation rates of PV systems is different, because customers will generate     different appliances schedules before and during the participation in the ancillary service mechanism with different amount of local PV generation available.

Conclusion
As an innovative paradigm for electric power systems with a high penetration of distributed energy resources, P2P energy trading enables direct energy trading between end customers, which is able to facilitate local power and energy balance and has the potential to support the operation of bulk power systems. In this paper, a framework was proposed to enable ancillary service provision from a P2P energy trading community, creating additional value for both customers in the community and power systems. Specifically, an ancillary service provision mechanism was designed along with P2P energy trading and residual balancing mechanisms to enable the power utility to obtain ancillary service from customers in a P2P energy trading community. Furthermore, the optimal bidding strategy of customers was figured out to maximize their benefits in the proposed mechanisms.
Simulation results based on a residential community in Great Britain verify the effectiveness and demonstrate the outcomes of the proposed mechanisms and bidding strategy. From the perspective of the power utility, simulation results show that the power utility was able to obtain a significant or required amount of ancillary services of different types with the proposed mechanisms applied. From the perspective of each individual customer, the proposed P2P energy trading and residual balancing mechanisms are able to achieve Pareto improvement compared to the conventional Power-to-Grid mode, and the proposed ancillary service mechanism can further achieve Pareto improvement compared to that without ancillary services. Furthermore, the social welfare of the whole community with the proposed P2P energy trading and residual balancing mechanism is significantly higher (being 98% in the case of this paper) than that with the conventional Power-to-Grid mode, and the adoption of ancillary service mechanism can further improve the social welfare (by 144% with the Type-2 Ancillary Service and by 135% with the Type-1 Ancillary Service in this paper).
Moreover, simulation results show that increasing ancillary service prices and installation rate of electric vehicles (by rescheduling EV charging and discharging) can increase the total amount of ancillary services provided for the power utility and thus can bring higher revenue for the customers in the community. By contrast, increasing installation rate of PV systems is not directly relevant to the amount of service provided.
Although this paper makes a first attempt in ancillary service provision from a P2P energy trading community, a number of studies remain to be conducted in the future, including but not limited to the following topics. First of all, all the mechanisms presented in this paper are forward markets. The uncertainties of generation and load forecast, and the corresponding real-time / settlement mechanisms are to be figured out. Moreover, the constraints and costs related to electric power networks are to be considered. Last but not the least, more specific types of ancillary services remain to be studied, such as frequency control, voltage support, etc.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
(continued ) CDA process for customers to reach P2P energy trading agreements 7: make the trading price for the matched electricity: π t = (π b,t + π s,t )/2;

8:
if σ b,t > σ s,t 9: In Equations (A1) and (A2), T is the total number of time slots within the Target Time Period T; A f refers to the set of flexible loads; x δ (kW) is the power of the flexible load; x ch (kW) and x dis (kW) are the charging and discharging power of the electric vehicle, and η ch and η dis represent the charging and discharging efficiency; P must-run (kW) is the sum of inflexible demand; P PV (kW) is the power output of PV panels; Δt (hour) is the length of each time slot. p t (£/kWh) is the electricity prices offered by the power utility: where the premise sells the electricity at the feed-in tariff rate p FiT t (£/kWh) and buys electricity at the retail price p retail t (£/kWh). The constraints include the operational constraints of flexible loads and the requirements of customers. Specifically, the constraints regarding noninterruptible flexible loads (i.e. δ ∈ A NL ⊂ A f ) are as follows: x δ,t = {0, P δ } ∀t ∈ T (A7) In Equations (A4)-(A7), b and e represent the allowed start time and required end time of the task respectively; L δ represents the time duration of the task, measured by the number of time slots (note that eb >= L δ ); P δ (kW) represents the rate power of the appliance. Equation (A4) guarantees that the appliance will not operate beyond the permissioned time interval. Equation (A5) requires that the task must be accomplished before the deadline. Equation (A6) guarantees that the task cannot be interrupted. Finally, Equation (A7) denotes the possible power status of the appliance. It is worth noting that interruptible flexible appliances can be seen as a special type of non-interruptible flexible appliance and can be described by Equations (A4), (A5) and (A7) but without (A6).
The constraints regarding thermostatically controlled loads, i.e. δ ∈ A TCL ⊂ A f , are presented as follows (electric water heaters are considered as an example in this paper): ∑ t s=1 x δ,s ⋅Δt + ρ⋅M⋅c water ⋅(θ 0 − θ low ) ≥ x δ,t = [0, P δ ] ∀t ∈ T (A11) In Equations (A8)-(A11), ρ represents a constant coefficient for unit conversion between J and kWh; M (kg) represents the mass of water in full storage; c water [J/(kg⋅ • C)] represents the specific heat of water; θ 0 ( • C), θ req ( • C) and θ en ( • C) are the initial water temperature, required water temperature and cold water temperature respectively; θ low ( • C) and θ up ( • C) are the lower and upper limits of water temperature; C (kWh) is the heat energy demand for hot water use; d (kg) is the amount of hot water drawn. Inequation (A8) requires that the total heat energy gained from the heater plus the initial heat energy stored in the tank till the t-th time slot should be no less than the total heat demand till the same time slot. Similarly, Inequation (A9) requires that the total heat stored in the tank cannot be over the upper limit. Equation (A10) calculates the heat demand due to hot water use. Equation (A11) specifies the range of the heating power. The constraints regarding the electric vehicle are shown in (A12)-(A17). It is worth noting that the same set of equations, i.e. (A12)-(A17) can be used to model household battery energy storage systems as well by setting t in = t out and ΔSOC = 0.
SOC min ≤ SOC t ≤ SOC max ∀t ∈ [1, t out ] ∪ (t in , T], (A14) 0 ≤ x ch t ≤ P ch max , In Equations (A12)-(A17), SOC is the state of charge (SOC) of the battery of the electric vehicle; SOC 0 represents the initial SOC; E (kWh) represents the rated capacity of the battery; t in and t out represent the time slots at which the electric vehicle plugs in and out (i.e. return and leave home); ΔSOC represents the energy consumption of the electric vehicle during the travel, measured by SOC; SOC min and SOC max represent the lower and upper limits of SOC; P ch max (kW) and P dis max (kW) are the upper limits of charging and discharging power. Equation (A12) calculates the SOC of the battery on the electric vehicle at any time slot t. Equations (A13) presents the relationship between the SOC at plug-in time and plug-out time. Inequation (A14) specifies the range of the SOC. Equation (A15) requires that the SOC at the end of the considered time horizon should be equal to that at the beginning of the horizon, ensuring that the scheduling can keep going beyond the current scheduling horizon. Equations (A16) and (A17) presents the upper limits of charging and discharging power, in which the charging power is denoted as positive while the discharging power is negative. It is worth noting that in the formulation composed of Equations (A12)-(A17), the electric vehicle just leaves home once. Nevertheless, this will not lose any generality, because it is straightforward to model the situations where the electric vehicle leaves/returns home several times, by adding more formulas that are similar to Equation (A13). It is also worth noting that in Equations (A12)-(A17), it is assumed that t out < t in , but it is straightforward how to formulate the situations where t out > t in , so it is not presented in the paper.
Note that the optimization problem formulated by Equations (A1)-(A17) can be transformed as an equivalent mixed integer linear programming (MILP) problem, and thus can be solved by off-the-shelf optimization tools.