Demand-Response Round-Trip Latency of IoT SmartGrid Network Topologies

Smart grids are the next generation of power distribution network, using information and communications technologies to increase overall energy efficiency and service quality of the power grid. A significant challenge in smart grid development is the rapidly rising number of smart devices and how to meet the associated load on the backbone communication infrastructure. This paper designs an Internet-of-Things smart grid testbed simulator to provide crucial insight into communication network optimization. Simulation for a large number of smart devices under various heterogeneous network topologies is used to analyze the maximum number of clients supportable for a given demand-response latency requirement. This latency includes all protocol overheads, retransmissions and traffic congestion, and simulator processing time is successfully eliminated from the final delay calculation via data post-processing. For a specific three-tier topology, given a round-trip latency requirement, the effect of number of smart devices per local hub and overall number of local hubs on network performance is analyzed, and crucial design insights are drawn relevant to cost-efficiency optimization of network deployment.


I. INTRODUCTION
The global interconnectedness of machines and devices over the internet, often called the 'internet-of-things' (IoT), proves to be a defining characteristic of 21 st century technology [1].Smart Grids, the extension of this idea to the power distribution network, use two-way information exchange between connected components to optimise energy flow.Application of Smart Grid systems are capable of dramatically reducing energy consumption and improving overall service quality [2].However, as the prevalence of Smart Devices grows exponentially, the problem of increasing traffic load and addressable end-points is of rising concern.
Internet and communications technology (ICT) infrastructure is the backbone of future Smart Grid enhancement, providing scalable and reliable services to all kinds of IoT applications.In light of the rapid growth of IoT and Smart Grid technology, the infrastructural challenges to be addressed come down to three aspects: Firstly, the fast growing amount of power system data needs to be supported by the network [3].A persisting challenge in IoT systems, both in Smart Grids and otherwise, is that the deployment of numerous and diverse interconnected devices is accompanied by equally numerous traffic increase with equally diverse set of Quality of Service (QoS) requirements, such as reliability, throughput, latency and security.The number of these Smart Devices is predicted to increase dramatically in the near future, and this will inevitably add load to ICT infrastructure.How best to optimise the Smart Devices' network topology to meet these QoS requirements remains an open issue.Secondly, there are many competing communication standards for IoT, but with which to communicate these data is still an open issue.Twoway information flow between Smart Devices is enabled by integration of many advanced communication technologies.A cooperation of multiple technologies are required to meet the Smart Grid requirements, and a single industry standard unifying all these technologies has yet to emerge.Finally, the cost of deploying the necessary supporting hardware is high.Deployment of devices and infrastructure also involves significant financial investment.Once the infrastructure has been deployed, any modifications can also be costly.
Thus, simulation is required to tackle these three infrastructural challenges.The simulation must be able to test varieties of communication standards combined in assorted arranged network topologies, so as to find the optimum solutions prior to capital investment in hardware and deployment.In this context, Smart Grid simulations have gained significant attention in the recent years: A testbed for demandfocused energy management in the end-user environment is designed and implemented in [4].The testbed consists of three levels -the base station, gateways and Smart Devices.A testbed based on wireless communication technology involving both centralised and distributed architectures is studied in [5].Hardware interfaces between energy and communication components is designed and implemented, and a small scale laboratory test is performed investigating realtime demand response and disruption resilience.There are also works that focus on ICT architecture.In [6], a threetier framework is proposed based on the Internet of Things, while in [7], an interoperability framework based on data distribution services is proposed.Meanwhile, in [8], a comprehensive survey on Smart Grid Cyber-Physical System testbeds is performed.Existing testbeds are compared and discussed from several design aspects, including heterogeneous communication support, security and privacy, multiple protocol support and remote connection access.These trials and tests pave the way for the successful deployment of Smart Grid ICT architectures, however these are restricted mainly to small or medium scale studies, and large scale simulation has not been fully addressed.Analysis of a large scale network is of great importance to physical implementation of Smart Grid ICT architectures from the perspective of communication service providers.
In this respect, this paper presents the smart grid testbed design toward large scale network simulations.The testbed is designed to be flexible, scalable and reconfigurable, oriented by communication providers to optimise for large scale metrics.The major contributions are listed as follows.
• An IoT smart grid testbed simulator is designed and developed to provide crucial insight into the effect of ICT backbone topology on overall network performance.Simulations for a large number of smart devices under various heterogenous network topologies are used to analyse critical performance limits given minimum QoS requirements.
• For a specific three-tier heterogeneous topology using point to point, unicast and multicast communication, given a demand-response latency requirement, investigations are carried out on the number of smart devices that a local hub can support, and with a fixed number of smart devices, the number of local hubs that a central server can support.
• A model is provided for demand-response round-trip latency, where the latency includes all congestion delays, protocol overhead and retransmissions, and the processing time of the testbed computers is successfully eliminated from analysis with data post-processing.
Using the model, critical design constraints concerning network topology are optimised.
The rest of the paper is laid out as follows.Section II describes the testbed simulation model of this paper.Demand-response round-trip latency modelling is presented in Section III, and analysis of simulation results are presented in Section IV, where critical findings are demonstrated.Finally, Section VI summarises the key points of this study.

II. SIMULATOR OVERVIEW
The aims of the smart grid testbed simulator are to analyse the effect of the telecommunications infrastructure on the overall network performance and be used to provide crucial insight into network optimisation.Scope of the simulator is to simulate networks where: up to hundreds of nodes are organized in a specific topology; point to point, unicast and also multicast (or even broadcast) communication, between nodes is being considered; and traditional layer 3 Internet protocols (TCP/IP) with IPv4 (upgradeable to also use IPv6) address space are used.

A. SIMULATION TOPOLOGY
The smart grid testbed needs to be flexible, scalable and reconfigurable.To meet these needs, the simulator consists of three component types forming a three-tier heterogenous network: a Smart Device module, a Substation or Local Hub module and a Central Hub module.To enable large scale testing, these modules have been implemented using Microsoft Visual Studio VB.NET platform [9], [10].The chosen system topology is a three tier tree-star network.A tree-star network topology can be considered as a combination of two or more star networks connected together.In each star network comprising the tree, there is a Local Hub to which all the lower tier Smart Devices are directly linked.The Local Hubs of each star network are then directly connected to a central administrator node call the Central Hub, as in Fig. 1.This topology is ideal when the nodes are located in groups, with each group occupying a relatively small physical region, such as households or groups of households.

B. COMMUNICATION PROTOCOLS
The simulation supports communication via wireless or wired communication, requiring only input of data rate and bit error rate (BER) (or packet error rate (PER)) on each communication link.Transmission Control Protocol (TCP) / Internet Protocol (IP) [11] communication is used to simulate communication on the ICT backbone.The TCP/IP protocol has several advantages for a Smart Grid testbed.It is widely used and well understood, so it is supported by the majority of available routers and servers, which will serve to reduce the cost of system development.Furthermore, since it is transparent to real networks, more routes will be available for the transmission.A wide choice of encryption algorithms can guarantee information security, the complexity of which can be assigned according to the specific application.Finally, IPv6 is already rolled out for commercial use and is able to support vast amount of devices within the network.A main criticism of TCP/IP in IoT applications is it's energy performance on power-limited devices, however the simulation can also utilize the User Datagram Protocol (UDP).

C. SIMULATION MODULE STRUCTURE
The simulation is developed in a windows application environment.In this application a user selects all the simulation parameters through the application graphic user interface (GUI).The application then calls the three described functions (SmartMeterModule.exe,SubstationModule.exe,Cen-tralHubModule.exe) in order to build the 3-level Tree-Star topology.For example, if a user has selected 1 node for the Central-hub in the upper layer, 10 nodes for the Substations in the middle layer and 10 nodes (Smart Meters) per Substation in lower layer, then the application will call Cen-tralHubModule.exe once, SubstationModule.exeten times, SmartMeterModule.exe 100 times.The central-hub knows the IP addresses of its Substation nodes.Each Substation node knows the IP address of the Central-hub, as well as the IP addresses of its regional Smart Meters.Each Smart Meter knows the IP address of its Substation.The function of each module will be described in turn, before the operation of the GUI inputs and outputs is explained.

1) SMART METER/DEVICE MODULE
Smart Meters (or Smart Devices) are the lower tier nodes, conducting measurements and/or actuations within a Smart Grid system.They receive and respond to control commands from the Central Hub, via their Local Hub.The proposed Smart Meter module consists of two submodules, the Listener and Sender, as illustrated in Fig. 2.
Each Smart Meter is assigned a unique IP address which is shared by the two submodules.In theory, the maximum supported port number is 65,536, although this simulation requires only one per node for the listener submodule.This feature also makes the proposed testbed highly flexible, since more functions can easily be added and assigned with different ports to cooperate with existing modules.Each submodule has functions to generate and receive TCP/IP packets and log device events.
The Smart Meter module algorithm has several capabilities.The Listener sub module listens to a specific IP:port address and receives data (command packets) from the upper layers.The Sender sub-module can send data (measurement packets) to the upper layer in two ways, either periodically every predefined fixed period of time, or on demand, upon receipt of a measurement command from the upper layer by the Listener sub-module.There are several measurement commands, chosen for the needs of the simulation, for example ''send now values'', ''send last values'', ''send a fix value'', ''send a random value'', etc.The Listener and Sender sub-module write a record into the SmartMeter_Listener.log file and SmartMeter_Sender.log file, respectively, for every received data packet.

2) SUBSTATION/LOCAL HUB MODULE
Substation modules are the mid-tier nodes, which aggregate measurements from their associated Smart Meters and forward the data to the Central Hub.Like the Smart Meter module, the Substation module consists of a Listener submodule and a Sender submodule, as illustrated in Fig. 2.
Capabilities of the Local Hub involve TCP/IP traffic coordination, Smart Meter control and device events logging, and the algorithm has a number of functions.On starting the Substation module, both Listener and Sender sub-modules start.The Listener sub-module listens to a specific IP:port address and receives data (command packets) from the Central Hub and data (measurement packets) from its Smart Meters.For every received packet it writes a record of data into Substation_Listener.log file.
When the Substation Listener receives a measurement command from the Central Hub, the Sender submodule forwards the command to its regional Smart Meters in a broadcast manner.Upon receipt of the measurement packets from all of its affiliated Smart Devices, the Sender submodule aggregates and sends the data to the Central Hub in two ways: • Periodically (every predefined fixed period of time): The substation collects all the data received from its regional Smart Devices and sends them to the Central Hub.
• On demand: Once the substation has sent a measurement command to its regional Smart Devices, it waits to collect all the replies before sending an array of data to the Central Hub.The Sender sub-module writes log data to the Substa-tion_Sender.logfile.

3) CENTRAL HUB MODULE
The Central Hub module is the main network coordinator, where data from the entire network is gathered.Again it consists of Listener and Sender submodules, shown in Fig. 2. It's functions are to send commands to the lower tier nodes and receive replies.It also logs device events into a Sender and Listener log file and may be instructed to request Smart Meter data either periodically, on demand, or randomly during the overall simulation time period.
In the Central Hub module algorithm, the Listener submodule listens to a specific IP:port address and receives data (aggregated measurement packets) from the the Local Hubs, then writes a record of the data into the Central-Hub_Listener.log file.The Sender submodule, sends measurement commands to all Substations with a broadcast function, and writes log data to the Central-Hub_Sender.logfile.

D. GRAPHIC USER INTERFACE (GUI)
Simulation data is input by the user in four GUI forms: In the first form, shown in Fig. 3 (a), the user selects if the simulation environment should be distributed over multiple computers or stand-alone on a single computer.In the stand-alone environment the Upper layer, Middle layer and Lower layer will all be built in the same computer, with the same IP address.Each node will then use different ports.In the distributed environment, any of the three layers Upper, Middle or Lower may be built in different computers (and hence with different IP addresses) as required.In the second form, shown in Fig. 3 (c), the network topology is chosen by selecting the number of nodes for each layer.In the third form, shown in Fig. 3 (b), the user decides the type of Smart Device measurement data (either fixed values or random values for purposes of simulation) and activity of the Central Hub measurement commands (either periodically, randomly or on demand).In the last form, shown in Fig. 3 (d), the simulation time preferences are specified, which includes when, if at all, the nodes in each tier should send their periodic messages, whether there should be a random time delay before each transmission (so that the computer's processor is not overloaded by a large number of nodes sending messages all at once) and how long the total simulation should run for.
While the simulation is running, the GUI displays the output window shown in Fig. 4 (a).During this time, the user can chose to send additional commands from the Central Hub, such as 'send now', 'send alive', etc.Once the simulation is complete, the GUI displays a bar graph of the round-trip latencies from all the commands sent from the Central Hub during the simulation, and displays the mean with a red line, as shown in Fig. 4 (b).With this testbed simulation platform it is possible to analyse the effect of Smart Grid network topology on roundtrip latency, and use this to gauge the maximum number of devices that can be supported for a given network structure.Analysis can then serve to optimise the system and gain crucial insights into relevant design characteristics.

III. ROUND-TRIP LATENCY MODELLING
It is of great importance to know the maximum number of devices that a Smart Grid network can support for a given set of QoS requirements.The maximum number of Smart Devices per Substation and Substations per Central Hub will be a target metric in this experiment, and finding the optimal topology to maximise the number of supportable Smart Devices is key.In this section, we seek to find how the three-tier centralised topology and Smart Device aggregation affects the resulting demand-response or round-trip latency.
The round-trip latency in this case consists of the time required for the Central Hub to request measurements from all subsidiary Smart Metres and receive a complete reply.This includes all protocol overheads, retransmissions and traffic congestion at the respective nodes.For total number of Smart Devices N D , the number of Smart Devices per Substation (Local Hub) S L and number of Local Hubs per Central Hub L are related by the formula Given that measurement data from each Smart Device must pass through two hops in order to reach the Central Hub (Smart Device to Substation and Substation to Central Hub), the bottleneck in the system occurs in the middle tier, at the Substation level.Thus S L is a critical network parameter.
The traffic load ρ in a network is defined as the ratio between the packet arrival rate λ and the packet service rate µ.
Assuming an M/M/1 queueing process in this case, since there is one queue per substation server, the average delay due to congestion at the local hub is given by Assuming constant average packet service rate from Substation to Central Hub, the arrival rate at each local hub should increase linearly with S L , and the delay should increase proportional to the inverse.However, this is only one link.The round-trip delay incurs a transmission delay from central hub to substation, from substation to smart device, from smart device to substation and substation to central hub.Each node has only one listener and one sender submodule, so there may be congestion at any of these steps.And that is without considering the delay incurred by processing time, transmission time and the number of retransmissions.Hence the need for a simulation testbed.

IV. NETWORK SIMULATION RESULTS & ANALYSIS
Using the testbed described in Section II, ten different network sizes were simulated using N D from 100 to 1000.Within each network size, S L and L were varied, to gain an idea of the effect of network topology on round-trip delay.The data links were modelled with a data rate of 50 Mbps to simulate the VDSL backbone, and BER of 10 −7 to allow for zero PER with convolutional coding.Fig. 5 shows how the round-trip delay D rt varies with S L for the different network sizes.It can be seen that the delay increases linearly with S L of the form where C n are fitting coefficients.This relation is seen even more clearly when D rt is plotted against L, following an inverse relationship of the form shown in Fig. 6.In this case C n are found by nonlinear least squares regression.The fewer devices per local hub, or the more local hubs for a given number of devices, the less congestion during the aggregation process, and less overall delay.This makes intuitive sense.However, Fig. 6 also shows a crucial design  insight: for any constant number of devices, the delay benefits of using more local hubs diminishes with the inverse of the number of substations.This is important, since the number of substations within a smart grid network will have a significant effect on deployment cost.
Also notice that the round-trip delay appears to rise with the number of nodes.This is unexpected, since the bottleneck occurs at the local hub, which is unaffected by the number of parallel data streams in different substations.This proportional increase is accountable to the processing time of the computer itself: Generating measurement packets for 1000 nodes will take ten times the time to generate packets for 100 nodes.It is desirable to remove the processing time from the overall delay calculation, since packet generation for the entire system would not normally be undertaken by only one processor.
To remove the processing delay, the average delay for each fit in Fig. 5 was used to compute the average difference between each successive network size, which is taken to be the processing time for 100 nodes T p100 , in this case 457 milliseconds.This time delay was then subtracted from the data for every 100 node increase, giving the plots shown in Fig. 7   and 8.This allows the coefficients C 1 and C 2 to be estimated as the average gradient and y-intercept.The average curve in each case is shown by the black dotted line, which runs approximately through the origin in Fig. 7, and tends to zero and infinity in Fig. 8. Fig. 9 shows the same in a 3D surface plot for the number of substations and total number of devices.For comparison, plots both with and without the processing time T p100 are included.From this it is clearly visible that for any number of devices N D , the minimum delay will occur with the maximum number of local hubs.For L > 15 there is a constant convergence with very low variation.The exact number must then be chosen accounting for cost efficiency of the overall system.

V. DISCUSSION
How best to structure the IoT power and communications network remains a significant challenge for future smart grid systems.This paper improves analysis over existing models by successfully simulating demand-response latency for various large scale network topologies, where all protocol overheads, retransmissions and traffic congestion are taken into account, and simulator processing time is removed from the final latency assessment.Analysis of large scale smart grid architectures is of great interest to communication service providers for optimisation of deployment cost-efficiency.Deployment of ICT devices and infrastructure, such as local hubs in a network, involves significant financial investment.The analysis in this paper exposes an opportunity to compromise between latency gain and deployment cost, on the grounds that the demand-response latency falls by an increasingly smaller amount with each addition of a mid-layer node.

VI. CONCLUSION
This paper presents a large scale simulation testbed to study the demand-response latency of various heterogeneous smart grid network topologies.A three-tier tree-star topology is simulated with three distinct node types.Networks with up to 1000 client nodes are simulated, and various lower and mid-tier node configurations are studied.Round trip delay includes all congestion delays, protocol headers and retransmissions, and the processing time of the testbed computers was successfully eliminated from analysis with data postprocessing.It was found that the round-trip delay varies with the inverse of number of substation nodes, meaning that the system is optimised with the maximum of local hubs.Moreover, regardless of the number of Smart Devices in a Smart Grid, beyond a certain number of middle layer local-hubs (in this simulation 15 and above), the Demand-Response round trip delay changes very little, slowly converging to zero.This can be used for easy analysis of network implementation costefficiency and network performance limitations given certain QoS requirements, quantity of client devices and implementation budget.

FIGURE 2 .
FIGURE 2. Overview of the simulation module structure.

FIGURE 5 .
FIGURE 5. Round-trip delay variation for smart devices per local hub.

FIGURE 6 .
FIGURE 6. Round-trip delay variation for local hubs per central hub.

FIGURE 7 .
FIGURE 7. Round-trip delay minus processing time for devices per local hub.

FIGURE 8 .
FIGURE 8. Round-trip delay minus processing time for substation number.

FIGURE 9 .
FIGURE 9. 3D plot of D rt with L and N D with and without processing time.(Uses the same data point colour scheme as previous graphs).