On Application of Least-delay Variation Problem in Ethernet Networks Using SDN Concept

The goal of this paper is to present an application idea of SDN in Smart Grids, particularly, in the area of L2 multicast as defined by IEC 61850-9-2. Authors propose an Integer Linear Formulation (ILP) dealing with a Least-Delay-Variation multicast forwarding problem that has a potential to utilize Ethernet networks in a new way. The proposed ILP formulation is numerically evaluated on random graph topologies and results are compared to a shortest path tree approach that is traditionally a product of Spanning Tree Protocols. Results confirm the correctness of the ILP formulation and illustrate dependency of a solution quality on the selected graph models, especially, in a case of scale-free topologies.


Introduction
The applicability of the ISO/IEC/IEEE 8802-3 Ethernet in mission-critical environments like Smart Grids (SG) has been immensely studied for one decade already.For the popularity, manageability, and low cost of Ethernet in data networks the industrial systems become more Ethernet orientated.Published works in power engineering shown the use of Ethernet, marked out by the IEC 61850, for a real-time control with strict limits of transmission delays [1].Additionally, network components have to support many distinct protocols or technologies to run real-time applications on the Ethernet reliably.Typically, these protocols seek to prevent loops, increase network resilience, propagate forwarding information, or to optimize traffic flows to reach desired Quality of Service (QoS) in the network.Although these tasks are already solved, the complexity of such systems requires highly specialized personnel, and troubleshooting is difficult.
Currently, many network problems and challenges are intensively researched using a concept of Software-Defined Networking (SDN).The SDN concept has a potential to fulfill SG demands and to simplify overall network management.SDN is still subject of research, and it spreads into many forms depending on the researcher's or vendor's perspective.Regardless of the used means, from our point of view, the SDN presents a tool that allows implementing advanced control algorithms increasing an automation of processes and introducing new services in data networks.The level of automation is even more strengthened by mechanisms contained in the IEC 61850 standard as describes Molina et al. in [2].
One of the potential SDN applications is a multicast distribution of sampled values in the form of a constant data stream from non-conventional transformers as described in IEC 61850-9-2LE [19].Although the problem of L2 multicast forwarding in Ethernet networks is tackled by Multiple MAC Registration Protocol [3], it still relies on an underlying spanning tree that is rigid.This approach using spanning tree underutilizes network resources.In contrast, SDN allows implementing an arbitrarily calculated multicast tree in the network giving the possibility to optimize traffic flow according to relevant requirements.Therefore, authors propose an ILP formulation minimizing the variation of propagation delay along branches of a multicast tree for studied network topologies.The ILP formulation was specifically designed for a Least-Delay-Variation Steiner problem.The motivation is to find such a network configuration that ensures delivery of generated packets to multicast subscribers most likely at the same time.This approach has a direct implication in power engineering, e.g.multisynchrophasor SAS in island operation during a blackout and reconnecting phase.The goal of this work is to investigate an impact of different random graph topologies on qualitative parameters of computed multicast trees.
The paper is structured as follows.In Section 2. , we discuss related published works.Section 3. details network model and closes up proposed ILP formulations.Section 4. presents achieved results, and we conclude our paper with a summary in Section 5.

Related Work
The network control in SG based on SDN is promising and supported by recent publications.The main research topics are focused on how SDN can help to increase SG resilience, meaning both data network and power grids, for example in [4].The crucial problem of ISO/IEC/IEEE 8802-3 Ethernet-based networks is their non-deterministic nature, but critical services run on them; therefore, researchers have focused on delivering required level of QoS to application flows [5].Dorsch et al. introduced and evaluated algorithms for fast rerouting of cases when critical flows are placed in a regular network [6].
The objective of finding a multicast tree as a subgraph which takes into account only a subset of nodes is defined as Steiner problem.The Steiner problem is a very well studied N P-complete problem regarding proposed algorithms.Authors of heuristics often consider different objectives and constraints when optimizing a multicast forwarding problem such as path delay, total cost of the tree or maximal congestion.The vast majority of algorithms is source-specific focusing on minimization of tree cost with at least one constraint.Almost all basic heuristics use or come out from Minimum-Spanning Tree (MST) [7].
Historically, two kinds of problems related to the delay variation were studied: Delay-and Delay Variation-Bounded Multicast Tree (DVBMT) and (DVBST).The latter additionally considers tree cost for an objective in optimization formulation [15].Rouskas and Baldine defined the DVBMT in [9], where they proved the DVMA to be N P-complete [9].Authors proposed Delay Variation Multicast Algorithm (DVMA) that has a series of successors improving time complexity.These are Core Based Tree algorithms presented in [10], or metaheuristics for example Genetic Algorithm published in [11] or an approach using Simulated Annealing proposed in [12].The DVBST problem became a main topic of various algorithms [13], [14] and [15].Currently, authors focus on more complex variations of the multicast forwarding problem considering mul-tiple objectives with multiple constraints.Published algorithms address the increasing level of the problem complexity by combined metaheuristics.Recently, Xu and Qu presented in [16] an extensive survey of metaheuristics together with a proposal of multiobjective simulated annealing based genetic local search algorithm that represent the combined approach.Concurrently, the multicast forwarding problem is drifting from the traditional network layers up to the application layer, since the trend of overlay networks and applications not relying on lower layers is more apparent than ever as published in [17] by Lin et al.In the context of methods used in this paper, Park et al. published an ILP formulation of a multi-QoS DVBST variant used for multicast routing in sparse-splitting optical networks [18].Authors claim the ILP is widely used to solve multicast routing optimization problems in optical networks.

Mathematical Formulation
A multicast publisher generates a stream of sampled values defined by IEC 61850-9-2LE [19], forming a constant flow, to the group of L2 multicast subscribers.
For the purpose of modeling, the algorithm anticipates a static set of subscribers.The static set of subscribers is assumed as well in [19].In contrast to some papers, network nodes which are subscribers can forward the traffic to further nodes.
The following text describes a network model that is used later in ILP formulations.To compare qualitative parameters on various graphs and setups, we propose ILP formulations for the Least-Delay-Variation (LDV) problem and ILP formulation of an agnostic approach based on Shortest Path Tree (SPT).The delay and delay-variation constraints are not considered in any of the following mathematical formulations.

Network Model
Let's consider directed connected graph G = (V, L) where V is a set of network nodes and L is a set of network links.The set of nodes V represents interconnecting nodes, e.g., Ethernet switches.The publisher and subscribers are connected to these nodes.
The multicast tree T (v p , S) is a sub-graph of G compounded of a multicast source node (publisher) v p ∈ V , and multicast destination nodes (subscribers) S ⊆ V \{v p } where the set S ∪ {v p } is called multicast group.The set S and the publisher node v p are interconnected by links through a subset of Steiner tree nodes M ⊂ V which form a part of T (v p , S).
All links are bidirectional, each directed link Let P T (v p , s), s ∈ S be a set of links ∈ L on a path from node v p to node s in the tree T (v p , S) and M (v p , s) ⊂ V is a set of Steiner nodes along this particular path.The total end-to-end transmission delay D T (P T (v p , s)) is then a sum of all link delays along the path as given in expression Eq. ( 1). ( The delay-variation δ T of the multicast tree T (v p , S) is defined as a maximum difference among end-to-end delays along paths of all node pairs in v p × S as is described by expression Eq. ( 2).

1) SPT-Based Formulation
At first, we define an ILP formulation producing SPT with a root node in the multicast publisher v p .The objective is to minimize total tree size, as defined by expression Eq. ( 3), where y ∈ {0, 1} with y = 1 if a traffic from v p to v s ∈ S is forwarded on link .
This formulation uses flow constraints approach Eq. ( 4).Each flow, from the publisher to a subscriber, is a set of node pairs PS = {{v p , v s }|v s ∈ S}, and these sets of node pairs are used for calculation of the objective function.A flow at link from node v p to destination v s ∈ S is denoted as ϕ ps and this variable can take a value of forwarded bandwidth b, i.e., ϕ ps is defined in positive domain Eq.(8).
The rest of the ILP formulation in Eq. ( 5) and Eq. ( 6) and Eq. ( 7) ensures that the found solution will be a tree aggregating all flows through the binary vector y .
2) Least-Delay-Variation Formulation The objective of the Least-Delay-Variation multicast forwarding problem is to minimize variation of total propagation delay along all paths in T (v p , S), as expressed in Eq. (9).
The ILP fomulation of LDV multicast forwarding is based on the SPT-formulation, but this algorithm is extended by a set of constrains detailed in this sections.The expression Eq. ( 10) tightens δ T (v p , S) for all pairs in PS using link delays defined in vector d .In order to map propagation delays to links selected in a solution process, we use in formulations Eq. ( 11) and Eq. ( 12) an additional conversion vector x ps , defined in Eq. ( 13), with x ps = 1 if a flow is forwarded at link from node v p to v s ∈ S.
x ps < ϕ ps + 1 ∈ L, (p, s) ∈ PS, (12) The modification of the objective function from Eq. (3) to Eq. ( 9) may cause the emergence of loops in a final solution.Therefore, we introduce auxiliary constraints that help to avoid stand-alone loops in the solution.The primary constraint Eq. ( 18) assures that all links assigned to a particular flow ϕ ps i are virtually labeled in non-decreasing order in vector o ps .The constraints are limited only to a set of neighboring pairs of ingress/egress links The constrains Eq. ( 14), Eq. ( 15) and Eq. ( 16) express that i and o are ingress and egress links along a specific flow (p, s) ∈ PS.Typically, this information can be obtained by logical operation AND for these decision variables.However, operation AND is non-linear operation; therefore, we applied a standard linearization approach.
To accomplish this side step, we have used an auxiliary variable a ps , defined in Eq. (17), that bonds similarly to x ps links with an assigned flow and the order constraint.

Evaluation
The variation of packet propagation delay is impacted by various network parameters such as network topology and its parameters.Four different random models were chosen with respect to a sufficient variability of evaluated instances.Random graph topologies were obtained using Erdos-Renyi [21], Watts and Strogatz as a small-world model [22], Barabasi-Albert [23] and a planar Dorogovtsev-Mendes [24] model as scalefree representatives.Graphs and their main robustness characteristics are listed in Tab. 1.
Each graph model was randomly generated 10 times with similar parameters and for each instance was generated 9 multicast groups with uniformly placed subscribers in a range from 10 % to 90 % of graph nodes.A propagation delay was randomly assigned to each link in a range from 5 to 500 ns.This propagation delay is approximately proportional to a delay on an Ethernet segments with length from 1 to 100 m.The multicast publisher was randomly placed in a graph center.Due to the exponential time complexity of the ILP, the network size was limited.Depending on the network model and link density, we were able to numerically evaluate, in a reasonable time, instances with sizes in a range from 10 to 20 nodes.

Numerical Results
In order to show benefits of suggested algorithm, we implemented agnostic based Shortest-Path Tree (SPT) and objective-aware called Least-Delay Variation (LDV) algorithms.Formulations proposed in Section 3. were implemented in OPL language and evaluated using CPLEX Optimizer.Implemented algorithms were run on 21600 instances in total.The following algorithm outputs on these instances are statistically evaluated and compared.
Results of the LDV algorithm give the best possible solution for a given network and multicast group.The resultant configuration can be proactively deployed into an SDN-enabled network and with proper QoS settings it leads in long-term to the desired LDV multicast tree.On the other hand, the SPT approach can be seen as the best approximation of an arbitrary Spanning-Tree Protocol.The difference in solutions provided by algorithms is shown in Fig. 1.At first, we analyze an effect of the multicast group size on the achieved delay variation that is depicted in Fig. 2. LDV delivers better results than the SPT approach.That can be seen in the lower delay variation for LDV and is mainly due to the character of graph models.The Watts-Strogatz network has highest diameter and lowest nodal connectivity thus it produces solutions with long paths.Simply, there is not enough of alternative paths in the topology.On the other hand, the Barabasi-Albert model shows much better results, since the model generates a lot of links and the LDV can find alternative paths.Remaining two topologies show delay-variation differences somewhere in the middle, proving that the number of links in a graph is a major factor in this setup.The impact of multicast group size is evident.The higher penetration of subscribers, the greater the delay variation.The growth is slowing down with an amount of subscribers, as the number of free links is decreasing.
The second case, where we investigate the effect of the multicast group size on a mean path delay D T (v p , u), u ∈ S, is rather opposite in its progress.The chart in Fig. 3 shows an unusual drop in the path delay for instances with low penetration of subscribers at graph models with the power-law distribution of node degrees (Barabasi-Albert, Dorogovtsev-Mendes).The path delay is decreasing significantly at LDV in contrast to the SPT approach.The LDV uses more links, particularly in the beginning when the link variability is higher; therefore, it produces paths with higher propagation delays.As the network topologies are always finite, the number of links in the solution is limited as well, and paths cannot grow to infinite lengths.Due to the uniformly placed multicast subscribers, SPT fluctuates almost at constant levels in all graph instances.
Tab. 1: Graph models used for evaluation purposes.All values are means of 10 generated graph instances.The probability of link selection in Erdos-Renyi is 7 %.Each node in Watts-Strogatz is connected to 3 nearest neighbors in a ring topology and each link is rewired with the probability of 30 %.Each new node in Barabasi-Albert is attached by 3 links to existing nodes.Graph characteristics presented in the table describes fundamental properties.Refer to [20] for a detailed explanation.Although the higher link density gives better results concerning LDV, the multicast tree can contain paths infeasible from a jitter perspective.Since none of the ISO/IEC/IEEE 8802-3 Ethernet QoS mechanisms can guarantee exact priority packet handling (switching fabric latency, various queue mechanisms), each node added to the solution potentially increases jitter along the path to a multicast subscriber.The impact of higher link variability in instances with lower penetration of subscribers is depicted in Fig. 4. The LDV on Barabasi-Albert model produces trees with a greater number of links, i.e., a higher number of hops a multicast packet has to pass.On the other hand, the chart proves that the SPT-based formulation produces trees with a lower number of links in all cases.All curves are converging at the higher number of subscribers since it is not possible to build a tree with a number of links > |V | − 1.Whereas the impact of multicast group size on the tree size is very well identifiable, the investigation into the effect of the network size was limited only to the window of three sizes (10,15,20).Although the range is not excessive the chart in Fig. 5 indicates, that the conclusions from previous perspectives were right.The LDV formulation on Barabasi-Albert and Dorogovtsev-Mendes models tends to construct longer paths as the growing tree size suggests.The size of the LDV tree is almost two times larger than the SPT of those models.Interestingly, all curves seem to be linear in this detail.

Conclusion
The ISO/IEC/IEEE 8802-3 Ethernet as a nondeterministic communication bus poses many challenges in the area of mission-critical applications, for example, L2 multicast defined by IEC 61850-9-2LE.The implementation of the SDN concept in data networks can increase the quality of services in Smart Grids limited by a transfer of communication technologies to Ethernet in last decade.Although SDN is very well studied nowadays, it is only a tool and for such a powerful tool new applications have to be adopted.
In this paper, authors proposed an ILP formulation of Least-Delay-Variation (LDV) multicast forwarding problem in Section 3. as a potential application of SDN in SG.Results obtained from a significant number of numerical evaluations were compared with an agnostic approach based on a Shortest-Path Tree (SPT) in Section 4. The analysis of random graph instances using the proposed LDV ILP minimizing delay variation shows improvement compared to currently used approaches based on SPT.Interestingly, the results indicate that scale-free topologies with a higher number of links lead to lower delay variations.We assume that in closed network environments as the IEC 61850 local networks are with specific traffic patterns is possible to avoid high jitter values since the local QoS mechanisms can be tuned very precisely to fulfill the LDV goal by SDN or by traditional management tools.
Authors plan to address this potential jitter problem in future research and to focus on metaheuristics as well as exact algorithms on a multi-tree problem con-sidering a combination of processing and queuing delays with traffic priorities, which could be implemented by SDN.
and egress links ω − (v).A real nonnegative value is assigned to every link ∈ L in form of a link delay d → R + .The link delay function d is a measure of link propagation delay.The function is naturally symmetrical, therefore d = d , ∈ L, ∈ L.

Fig. 2 :
Fig. 2: Effect of multicast group size on the mean value of least delay variations at graph size n = 20.
LDV multicast tree for a Barabasi-Albert graph.
SPT for a Dorogovtsev-Mendes graph.
LDV multicast tree for a Dorogovtsev-Mendes graph.

Fig. 1 :
Fig.1: An example of the difference between solutions found by SPT-based Fig.1a, Fig.1cand LDV formulations Fig.1b, Fig.1dfor Barabasi-Albert model and Dorogovtsev-Mendes models with n = 10 % and 30 % penetration of subscribers.In each graph, the orange node is the multicast publisher, and green nodes are multicast subscribers, and blue nodes represent Steiner nodes.Numbers next to links represent their weight in ns.

Fig. 3 :
Fig. 3: Effect of multicast group size on the mean value of path delays at graph size n = 20.

Fig. 4 :
Fig. 4: Effect of multicast group size on mean tree size at graph size n = 20.

Fig. 5 :
Fig.5: Effect of network size on mean tree size at the multicast group size of 20 %.