Software-defined dynamic QoS provisioning for smart metering in energy Internet using fog computing and network calculus

: Energy Internet (EI) is a revolution of power systems due to its flexible energy control. The smart meters are the key components in EI which perform real-time monitoring and require high demands on quality of services (QoSs). Besides, due to the great performance on resolving device-interconnecting and realising overall network control, software-defined networks (SDNs) are considered as a trend for future EI. However, most existing software-defined EI have all the data processed at SDN controller which makes it difficult to realise fine-grained and deep edge computation and control. Meanwhile, most existing QoS provisioning cannot provide satisfying service for EI because various metering data from smart meters require different time-variant QoS guarantees. To address above challenges, the authors propose a software-defined dynamic QoS provisioning for smart metering in EI using fog computing and network calculus (NC). First, a software-defined EI architecture, where fogs are deployed on smart meters and base stations to realise edge services, is proposed to perform deep edge control in EI. Second, the software-defined dynamic QoS provisioning based on NC theory is proposed to guarantee QoS. Simulations of latency as well as bandwidth utilisation show the advantages of the proposed scheme.


Introduction
Recently, more and more researches have focused on energy Internet (EI) which integrate various energy forms into communication systems and provide two-way real-time communication between the grid and users [1][2][3].As EI is still a new type of electronic technology, there still exists several challenges that need to be solved such as complexity, efficiency, reliability and security [1].To improve the efficiency in EI, software-defined networks (SDNs) which have a great performance on resolving devices interconnecting, realising monitoring and controlling for the overall network is considered as a trend for future EI.In [4], an SDN-based EI architecture is proposed to realise local energy information monitoring.In [5], an energyaware routing model is proposed combined with SDN to take the consideration of energy when deciding communication path.Besides, the implementation of programmability of energy flows into EI is proposed in [6].However, most existing software-defined EIs process all the data at SDN controller which has exerted all data processing and monitoring pressure, thus these schemes cannot realise fine-grained and deep edge computation and control.
Besides, smart meters and sensors in EI, which are deployed to monitor the real-time power consumption, require totally different quality of services (QoSs) due to various energy infrastructures they linked to [1,7].Although software-defined EI has a great performance on energy routing, it ignores the concerning of increasing number of smart meters and sensors in EI which require high demands for dynamic QoS.However, software-defined EIs mostly focus on the energy router without the concern of increasing number of smart meters and sensors in EI which require high demands for dynamic QoS.Existing dynamic QoS provisioning methods in traditional networks cannot be directly used in software-defined EI.These QoS policies mainly have two limitations.First, most of them only change one abnormal QoS policy without global control [8,9].Second, only one QoS parameter is concerned in guaranteeing QoS such as latency or bandwidth [10,11].Therefore, the issue about dynamic QoS provisioning in software-defined EI is still unresolved.
To address these challenges, we propose a hierarchical fog computing advanced software-defined EI architecture.On the basis of this architecture, a network calculus (NC)-based QoS anomaly prediction mechanism and dynamic software-defined QoS provisioning scheme are proposed.
The main contributions of this paper are as follows.
Hierarchical fog computing advanced software-defined EI architecture: We introduce fog computing into software-defined EI architecture to realise locally information processing and reduce the latency.The fogs are deployed on two layers as base stations layer and smart meter layer.For the underlying energy infrastructures, they communicate with the nearest meter fogs.For meter fogs, they can manage the states of underlying infrastructures and handle some tasks which can be processed locally.Before sending the data to base station fogs, they will preprocess the data due to the content aware function of fog computing.For base station fogs, they can communicate with the nodes from same layer and adjacent layers.Base station fogs are responsible for data forwarding and QoS guarantees.Besides, the SDN controller is responsible for managing the overall network states, deciding the QoS provisioning policies and controlling of the whole network.
Dynamic QoS anomaly prediction and provisioning: First of all, QoS policies are designed based on NC which makes it possible to predict the network anomalies such as latency and backlog and lets the base station fog initiatively requesting for QoS provisioning.Besides, dynamic QoS provisioning is realised by SDN controller and the request as well as response of QoS provisioning are in the form of FlowTables.For our proposed architecture, meter fog plays a role of content classification.The base station fog is responsible for QoS anomaly prediction and QoS guarantee.The SDN controller needs to generate QoS policies and provisioning and make an overall control.Therefore, this scheme can realise a twoway communication and aware of anomaly in time.
The remainder of this paper is organised as follows.Section 2 introduces related works.Our proposed hierarchical fog computing advanced software-defined EI architecture is described in Section 3. Section 4 describes the NC-based QoS anomaly prediction.The software-defined QoS provisioning is presented in Section 5. Section 6 presents the design of fogs and SDN controller.Simulation and analysis are demonstrated in Section 7. Finally, Section 8 draws a conclusion.EI is the comprehensive use of advanced power electronics technology and information technology to achieve multiple ways of energy sharing and information exchange [2].In [12], a future renewable electric energy delivery and management system is proposed to integrate energy generating sources and storage with existing power systems.A distributed energy resources services platform is proposed to coordinate the distributed energy resources and renewable energy integration [13].
Besides, as it is complex to configure traditional networks, SDN is proposed to separate the control plane and data plane to achieve a flexible network traffic control [14].It offers a chance to change the limitations of current network resources [15,16].For example, the authors apply the SDN into wireless sensor networks (WSN) to realise the network management in [17].Therefore, SDN has also been widely used into EI to virtualise networks and leave the underlying sensors only data-related functions [18][19][20].In [6], Zhong et al. proposed the software defined energy internet (SDEI) architecture which separates the control, data and energy planes to realise flexibility and efficiency.In [18], the network subsystem utilised SDN to connect all the devices together.In [19], SDN enables the dynamic network control in the consideration of specific communication requirements in power grid.
As we can see, the SDN offers EI better network management and configuration.Although many researches focus on softwaredefined smart grid or EI, the dynamic QoS provisioning problems for smart metering in EI have not been solved yet.

Fog computing
With the explosively increasing number of sensors and data, cloud computing is utilised on many traditional sensor networks [21][22][23].As sensors are heterogeneous and dynamically changing which contribute to numerous different types of data and cost massive time to process, cloud computing cannot well solve them and the sensor networks require a more flexible and compatible architecture [24].To deal with this issue, many researches are focusing on fog computing, which is localised and extends the distant cloud to the edge of network [25,26].In [27], edge computing is utilised to realise edge caching for mobile social networks.Furthermore, fog computing has a better performance on processing data and controlling underlying sensors as it is more efficient and easier to access compared with cloud computing.The fog layer is deployed between WSNs and cloud layer in architecture [28].Besides, a data delivery mechanism is designed.In [29], the fog nodes are deployed above the WSNs and manage all the nodes inside a fog coverage area.As the researches on fogadvanced EI have not been proposed, the architecture of it and the specific structure of each component are blurry and need to be further explored.

QoS provisioning methods
Guaranteeing QoS for the numerous types of data is an important goal in network [1,[30][31][32].Recently, researches are focusing on dynamic QoS mechanism as different data types require different QoS requirements [33,34].A multi-path routing protocol is proposed which can make different data choose different routing mechanisms but it is not suitable for real-time applications [35].Kader et al. in [36] propose to consider QoS each time receiving a packet but this may cause data backlog when mass data arrive.Some consider to change QoS policies timely in [37] but it is not suitable for the network which covers several data types.In [38], a QoS provisioning based on energy aware is proposed.These QoS provisioning methods only care about each signal data packet without concern of overall network states.As every packet will be influenced by other packets arrived at the same node, how to provision QoS still needs future consideration.Moreover, as fog is content aware, it has the ability to distinguish the data into more types in a fine-grained way.Besides, these data may change their QoS requirements with the changes of network and application states.Therefore, the existing QoS mechanisms cannot be directly utilised in EI and make full advantages of SDN.

Proposed fog computing advanced softwaredefined EI architecture
In this section, the architecture of fog computing advanced software-defined EI architecture and a brief introduction of QoS provisioning are proposed.
This fog computing advanced software-defined EI architecture includes the basic components as energy infrastructures, meter nodes, base stations and cloud centre and attaches them new responsibilities.Fog computing is deploying on meter nodes and base stations.The cloud centre also plays a role as SDN controller.This architecture contains four layers, as shown in Fig. 1 including energy infrastructures layer, meter fog layer, base station fog layer and SDN controller in cloud layer.The energy infrastructures layer contains numerous underlying energy infrastructures.The smart meters play a role in acquiring electricity information, monitoring electricity quality and managing energy.Therefore, sensors are running on different protocols to support different functions or applications and need the route information when sending data.However, in our proposed architecture, the underlying sensors do not need to care about the heterogeneous network convergence problems and route information and only need to send the energyrelated data to the local meter fog nodes.The first fog layer is deployed at meter fog nodes which are originally local based.As fog computing is content aware, the nodes complete the data preprocessing which helps classify data into different types based on content.Also, they monitor underlying sensors and upload these informations.After classifying, meter fog will attach a type ID for the upper fog to distinguish the type and do further processing.
The second fog layer is deployed at base stations which have more flexible resources.The data forwarding and QoS guarantee are realised in this layer.The nodes inside this layer have the ability to communicate with each other among the same layer.The base station fogs service for data forwarding and QoS anomaly prediction.Different data types are allocated different amounts of computing and communicating resources.The computing resources are represented as data processors, and the quantity of processors allocated data type equals to the allocated number of computing resources.Also, each data processor corresponds to a QoS anomaly prediction table which can detect network anomaly based on NC theory.Once anomaly occurs, the base station fog will initiatively request for QoS provisioning.The requested object is the SDN controller which is deployed at cloud centre.It contacts to all the base station fogs and has a strong ability of computing and storage.Therefore, it regulates the whole network state and issues QoS policies generation and provisioning.
It is clear to see that the meter fogs are at the edge of underlying energy infrastructure sensors, the base station fogs are at the edge of meter fogs.They are localised relatively and preprocess part of local data, which is not limited to QoS provisioning, before sending to next layer.The detailed information about anomaly prediction, QoS provisioning and deployment systems of fogs and SDN controller will be introduced in the remaining sections.

QoS anomaly prediction based on NC
NC theory is a network performance analysis tool which can be used to calculate the bounds of performance parameters based on arrival curve and service curve [39].Therefore, with the upper bounds of latency, backlog and so on, systems can process network abnormal on time to guarantee the overall performance of the network [40,41].
In our proposed QoS anomaly prediction mechanism, we utilised two main concepts from NC theory as arrival curve and service curve.The QoS policies are designed in the form of service curves.Along with arrival curves computed from each processor in base station fog, many parameters related to QoS can be easily computed and thus in future predict QoS anomalies.

Data shaping
Processor m is assigned a time slot ∂ i to forward data type i in one time period T. Obviously, the most efficient way to send data packets is sending as many packets as possible during allocated time.For this idea, we forward data after shaping into same size packets in one processor.The time interval ∂ i is set in the consideration of jitter in order to avoid packet loss.As we can see from Fig. 2a, the data shaping in single processor allows the processor try its best to send packets in the allowed time ∂ i .When all processors follow the data shaping rule, the bandwidth can be well utilised in one base station fog as shown in Fig. 2b.
To simplify our description, the size of packets of each data type i is defined as The S i represents the size of packets the processor i shaping the data into, B represents the bandwidth of the base station fog, and T represents the time interval for all processors finishing sending one shaped packet.The ∂ i represents the allocated time for processor i sending shaped packets in one period T. σ i represents the jitter limit value.

Arrival curve and service curve in NC
The data flow inside base station fog is described in Fig. 3.When data flows arrived at the fog, they will be sorted and shaped based on the type label.Each processor has time interval ∂ i to send the packets in a time period T. Therefore, the queue delay can be concerned as time period T.During this process, each processor has the responsibility to record the arrival data amount from the packet analysis module as arrival curve AC i (t).The service curve SC i (t) is defined as As we know, the QoS is guaranteed based on the allocated computing resources, bandwidth and so on.The allocated time to send packet for data i in time T is ∂ i , the jitter limit value is σ i , the allocated computing resource is represented as j, which means data i is allocated j processors to process it.Therefore, the QoS policies can be described as a set of each processor's service curve along with the data type it processed.

QoS anomaly prediction
Network parameters concerned with QoS can be calculated by arrival curve and service curve.The QoS anomaly prediction and new QoS provisioned based on the computed QoS parameters are shown in Fig. 4.
At the very beginning, several critical QoS parameters values are set such as backlog critical value BL crit and latency critical value DL crit .As we can see from Fig. 4, the current delay DL compand and backlog BL comp can be easily computed from arrival curve and service curve.When anyone of the computed value is larger than the critical one, the processor will predict it as QoS anomaly and request for new provisioning.These QoS policies include the resources allocation and the service curves for different data types.

QoS provisioning
As is described above, base station fogs act as predictors for future QoS anomalies and the SDN controller acts as a solver providing the solutions, which means provisioning new QoS policies.

Modelling of resource configuration
As we know, the QoS provisioning includes two parts as resource configuration and forwarding policy provisioning.The configuration of resource will be discussed first.The latter part of QoS provisioning is mainly about forwarding policy.
As we concerned, different data types should be configured different resources.Therefore, we consider a network of a set of N data types which is defined as Also, the resources in the fog node are divided into numerous processors.We consider a base station fog node with M processors as Once the corresponding data type processing is allocated to each processor, for example, whether the data type ty i is allocated to processor pro j to process can be defined as f (x) = 1 pro j is allocated to ty i 0 pro j is not allocated to ty i (5) Therefore, the overall allocated matrix Cor can be defined as The number of processors allocated to data type ty i is defined as The Sum i represents the amount of resources allocated to process that type of data.The more processors allocated to one data type are equal to more resources configured to that data type.

Procedures and FlowTables design of QoS provisioning
For one data type, QoS policy includes the service curve which decides the network parameters for how to send it and the resource allocation which decides how much processors will be allocated to process it.For one processor, QoS policy means which data it needs to process and how to send the data.
The main procedures and related FlowTables participate into the provisioning are described in Fig. 5.The detailed information about procedures and FlowTables related to QoS provisioning are described in the following section.

Acquiring and processing classified data in base station
fog: Acquisition module stores the table which describes the data type in corresponded processors.Once a data packet arrives, this module will extract the type label and then send to related processor according to its data type.Data processing includes three steps which are executed in the processors.Processors first will change the size of shaped data based on service curve.At the same time, each processor saves a QoS anomaly prediction table, in which the anomaly values are set by the SDN controller.
Once the computed value of parameter is bigger than the critical one, the processor will send the anomaly message to configuration module.Then, each processor will send its arrival curve and QoS anomaly prediction table to configuration module once required by configuration module.

Configuring QoS policies in base station fog:
As shown in Fig. 5, configuration module serves for receiving the anomaly messages from one processor and then asks all processors to send the arrival curve and QoS anomaly prediction table.After collecting all the information, it transfers these information together into the form of FlowTable and then sends to the SDN controller.Once received the FlowTable of QoS provisioning, configuration module transfers the table into QoS anomaly prediction tables, service curves and sends them to corresponding data processors.Also the relationship between data types and processors will be sent to acquisition module.

Packet analysing and QoS provisioning in SDN controller:
When a packet arrived at the SDN controller, once packet analysis module finds that it is a request for QoS provisioning, this module will send this packet to QoS provisioning module.QoS provisioning module is the key part during the whole provisioning stage.The basic step is the transfer between the FlowTable and provisioning information.When the request table arrived, it will extract the data process information including arrival curves, QoS-related parameters value.Then, it will obtain previous emplace of service curves and parameters set logs from the log database.
On the basis of these informations and the information from the FlowTable of QoS provisioning request, the QoS provisioning module computes the best combination of QoS policies which will then be sent in the form of the FlowTable of QoS provisioning to the fog nodes.

Table of QoS anomaly prediction:
The table includes the critical values of QoS parameters such as latency, jitter and backlog.Once these parameters are computed abnormal by the service curve and arrival curve, the anomaly sign will be set to 1.This table will be sent to the SDN controller when there exists QoS provisioning requesting.

FlowTable of QoS provisioning response:
The FlowTable includes three components as match, action and counter.Obviously, the action represents QoS provision requests and the counter is SDN controller.Besides, the match contains the information required for QoS provisioning.As SDN controller has saved the previous policies, the flow table only needs to contain the ID of fog and those that are currently calculated as arrival curve, QoS-related parameters value.

FlowTable of QoS provisioning response:
The action in this FlowTable is QoS provision and the counter is the ID of fog.The match part has to contain all sets of each processor as is shown in Fig. 5.The parts of processor ID and type ID allocate computing resources to each data type.The parts of time and jitter decide the service curve.The parts of jitter, latency and backlog decide the

Dynamic QoS provisioning sequences
In this section, the detailed QoS provisioning and data forwarding sequences will be described chronologically and are shown in Fig. 6: Step 1: The energy-related data is sent to the nearest meter fog node with the information of source address and destination address.The underlying energy infrastructure sensors do not need to care about how to forward data any more.
Step 2: When the meter fog receives the data packets, as it is content aware, it classifies data type according to content and attaches a label which records the data type at the end of each packet.After classification, the meter fog sends packets to the nearest base station fog.
Step 3: The base station fog starts to extract label from packets in the acquisition module which stores a table recording the relationship between processors and data types.Once obtained the label, the module sends data to the corresponding processor.As the sizes of packets are different, data shaping is the first step when the data arrives at processor.Different processors will shape the packets into different sizes according to the allocated bandwidth and jitter; therefore, the service curve can be computed based on them.Each processor will record the arrival amount of packets as the arrival curve.
Step 4: With the arrival curve and service curve, latency, buster and other parameters can be predicted.Different types of data will be set different abnormal values according to QoS policies.Once the parameters are close to the abnormal threshold, the processor will proactively send the QoS provisioning request.Then, the base station fog collects all processors' states and sends as the form of FlowTable.The detail of FlowTable for requests of QoS provisioning is described in the previous section.
Step 5: The SDN controller will reconfigure the QoS policies, resource allocations and anomaly values of each data type.These parameters will be transferred in the form of FlowTable.The FlowTable of QoS provisioning is described in the previous section.
Step 6: When the FlowTable of QoS provisioning is sent to the nearest base station fog, the configuration module of base station fog will abstract the data process information including arrival curves, QoS anomaly prediction tables and the corresponding relationships between processors and data types.Then, it resets the configuration of each processors and acquisition module.
Step 7: When the destination sensor does not belong to the base station fog, the base station fog will request SDN controller for the route information.
Step 8: As SDN controller has the overall sight, it can find out which base station fog the destination sensor it belongs to and then feedback the route information to it.
Step 9: The base station fog will then forward this packet to another base station fog which covers the destination node.In this section, the detailed designs of two kinds of fogs and SDN controller which are related to QoS provisioning are presented.

SDN controller involves dynamic QoS provisioning
SDN controller layer serves dynamic QoS provisioning, router information providing and the whole network monitoring and managing.
As is shown in Fig. 7, on one side, SDN controller is responsible for the QoS provisioning.With the QoS abnormal detection mechanism, SDN controller does not need to care about the data processing in base station fogs until the request of QoS provisioning comes.After receiving the request, the SDN controller will generate new QoS provision policy and then send to base station fogs in the form of FlowTable.Also, the request of QoS provisioning is sent in the form of FlowTable.On the basis of the local management idea, which means meter fogs are responsible for the management of underlying sensors and base station fogs are responsible for the management of meter fog nodes, SDN controller can only manage the states of base station fog nodes.On the other side, based on a global sight of the network, SDN controller can manage the whole network and provide route forwarding information.Also, base station fogs can ask SDN controller for route information.That is to say, more data can be forwarded by the base station fog instead of the central controller.

Base station fog involves QoS anomaly prediction
The base station fog is one of the most important components in the network and serves as QoS anomaly prediction and QoS provisioning requesting.
As described in Fig. 8, the base station fog contains four main modules as storage module, acquisition module, data processor module and management.After the preprocessing of the data in meter fogs, the acquisition module in base station fogs can easily distinguish data types and process them according to types based on the labels attached on the data packets.Then, processes will process the data based on QoS policies.The storage module saves the QoS policies related data as QoS service curves, computing resources and QoS policies logs.The requesting of QoS provisioning, monitoring of processors' states, managing of local network topology and sensors' state are executed in managements.
The software platform inside the base station fog has several processors.Each processor has same amount of resources and processes one data type at one time.The number of processors is more than the number of data types and one data type may be processed by different number of processors at different conditions according to the QoS polices provisioned by SDN controller.That is to say, each type of data will be processed under different compute resources, bandwidth to meet different QoS requirement according to dynamic QoS polices and QoS abnormal detection based on NC.The configuration of processors are completed by configuration module and meter fogs' state are monitored by state management module.Besides communicating with underlying meter fogs and SDN controller, base station fogs also have the ability to communicate with other fogs in the same layer.

Meter fog involves content classification
In traditional EI, all the energy infrastructures send energy-related data to the nearest meter node and then forward to core network.These sensors follow different transmission protocols and make it a problem for meter node to forward them.The meter fog is suitable to be deployed on the meter node and solve this issue.Besides, as fog nodes are content aware, they can classify data according to content.Therefore, the base station which manages numbers of meter nodes can reduce the time to process data.
The heterogeneous network convergence module and fog smart gateway contact underlying energy infrastructures and meter nodes for receiving and forwarding data.The monitoring module collects sensors' state includes power, resources, service and so on.Also, it monitors local network topology.As one base station fog may manage several meter fogs, this preprocessing can prevent large number of data waiting from being sorted and attached labels at the base station fog node and reduce the delay.Also, it makes full use of local resources.Besides the data classification role based on the fog's ability of content aware, the meter fogs play another role to monitor the status of the underlying nodes including energy, position and so on.This information will finally be collected in the SDN controller.

Simulation and results
In this section, we simulate the proposed hierarchical fog computing advanced software-defined EI and software-defined dynamic QoS provisioning under the consideration of the scalability of network and fine-grained control.

Evaluations of scalability
In EI, new energy infrastructures will join into the network continuously depending on the change of time domain.Therefore, to verify the scalable capability of the proposed hierarchical fog computing advanced EI architecture as well as the dynamic QoS provisioning scheme, bandwidth utilisation simulations are done with different network scales.In this circumstance, bandwidth utilisation is observed which is on the communication link between the meter node and the base station.
At the beginning of the simulation, there are ten sensors in the network, each of them sending packets in different sizes.The number of sensors in the experiment is doubled periodically from 20 to 80 sensor nodes, whereas the packets transmission rate remains the same.Also, without loss of generality, we assume that the anomaly critical value of bandwidth utilisation is 75%, which is taken as an example for simulation.When the predictive value of bandwidth utilisation is <75%, the communication bandwidth in our proposed network will increase around two times than before.
As shown in Fig. 9, the traditional network cannot adapt the QoS policies when the network state changes.Therefore, the rate of bandwidth utilisation remains substantially double increase in traditional EI.When 80 sensor nodes are included in the network, the current network bandwidth is unable to meet sensors' transmission needs though the rate of bandwidth utilisation has come to 100%.However, for the proposed software-defined QoS provisioning scheme, the bandwidth increased to leave fluctuating space for future network demands and the participating of more sensor nodes, because energy infrastructures growth and the lack of bandwidth in the future are predicted.

Evaluations of fine-grained control
It is important for EI to manage the network under a stable state when facing network burst situations.To verify the fine-grained control of our architecture and dynamic software-defined QoS provisioning, end-to-end delay and bandwidth utilisation simulations are done in the consideration of data mutation.
In this simulation, the network includes four underlying sensors, one meter node and one base station node which are taken as examples for simulation.Besides, four types of data are set in the network transmission.The sizes of different data types are, respectively, as 512, 768, 1024 and 1500 bits.Each node randomly sends packets to other terminal nodes.The interval time of sending packets will be shorted and modified to half of the previous one when the simulation comes to 137 s.In this network, the data rate of the link between meter node and base station is 4096 Mbits/s and the bandwidth of the link between underlying energy infrastructure and meter node is 2048 Mbit/s.
The simulation results include two parameters as network delay, which means the overall propagating and processing delay from the source sensor to the terminal sensor, and bandwidth utilisation occupied on the communication link from base station fog to the terminal sensor.

End-to-end communication delay:
The end-to-end communication delay is the total time D which includes the transmission delay t tran , the propagation delay through the entire path t prap and the process delay t proc .The process delay has two main parts as data classification delay t proc_s in the meter fog and the QoS guaranteed process and queue delay t proc_b in base station fog nodes.Therefore, the delay can be represented as On the basis of delay parameters above, the software-defined QoS will modify the bandwidth from the base station to the terminal node and thus future influences the value of t proc_b .
In the simulation, the results of end-to-end communication delay of the data packets under the environment with and without software-defined QoS provisioning are shown in Fig. 10.End-toend communication delay of traditional QoS provisioning link is expressed as blue curve.When the network load grows abruptly, the time cost to send the packets will increase proportionally.In the first half of the simulation time when the network is under normal state, the total delay fluctuates up and down within 5 µs.However, when facing huge traffic load, the end-to-end communication delay increases from 5 to 30 µs.Thus, the network is finally in the congestion state and the QoS is obviously not satisfied.End-to-end delay of data packets of software-defined QoS provisioning link is expressed as red curve in Fig. 10.When the experiment progresses to half of the total time, the load increases and the end-to-end communication delay starts to increase slightly but quickly be suppressed.The network anomalies can be well predicted and eventually avoided.Therefore, the network returns to normal state.

Bandwidth utilisation:
As the software-defined QoS is provisioned about the resources in base station fogs and communication links between base station fogs and terminal nodes.Therefore, we analyse the communication link between the base station fog and terminal nodes.The initial data rates are set as 1024 Mbit/s and the simulation results are shown in Fig. 11.
The bandwidth utilisation of traditional link is expressed as blue curve.With the rapid increasing of network load, the bandwidth utilisation also rises.In the extreme case, the network even becomes saturation.This is because when the amount of data packets sent by the sensors increases, the expected bandwidth also increases which will further cause the rise of bandwidth utilisation to one single data type.That means the link can only be distributed to limited users which is an undesirable situation for the network operators.Besides, the bandwidth utilisation of software-defined provisioning link is expressed as red curve.As is shown in Fig. 11,   though the amount of data transmission surges, the corresponding bandwidth utilisation only rises briefly during a very short period and soon returns to about 50% of the stable level.Since the timely prediction of the rising rate of sending packets, though the bandwidth occupied by a single data type increases, the bandwidth utilisation still maintains with the previous state.Therefore, the network congestion is avoided.

Conclusion
In this paper, a fog computing advanced software-defined EI architecture based on SDN was proposed which deploys fog on the meter nodes and base stations and this architecture makes each layer able to preprocess data with the local knowledge.Then, the NC theory-based QoS anomaly prediction mechanism was designed which made the base station fogs have the ability of predicting the anomaly of network and actively requesting for the QoS provisioning.Meanwhile, software-defined QoS provisioning mechanism which realises dynamically provision QoS policies were proposed.Finally, the implement models of fogs and SDN controller which related to QoS provisioning were presented.In our experiments, the ability of network scalability and fine-grained control in the consideration of varying latency and bandwidth utilisation were analysed.Simulations showed that this architecture and software-defined QoS provisioning mechanism enable scalability and fine-grained control well.

Fig. 1
Fig. 1 Architecture of fog computing advanced software-defined EI architecture

Fig. 9
Fig. 9 Scalability of the proposed architecture

2 Related work
2.1 EI and software-defined EI