Assessment and analysis of different process bus redundancy networks performance for IEC61850-based digital substation

: To achieve a highly reliable data communication network, Future Intelligent Transmission Network Substation (FITNESS) project has considered three different substation data communication redundancy networks. These include two independent parallel star networks, High-availability Seamless Redundancy (HSR) network, and Parallel Redundancy Protocol (PRP) network. This paper presents the assessment and comparison analysis of the data flow performance for these three networks. A typical 275 kV substation with the consideration of three redundancy networks is modelled and simulated using OPNET a communication network simulation tool. The data flow capability performance with respect of the network time delay for the typical protection and control equipment requirements (e.g. IEDs and MUs) in one bay are simulated and analysed. Results show that two independent parallel star networks have the highest data flow capacity. The capacity of PRP network is reduced by a half, while the capacity of HSR network is reduced to a quarter. The results also reveal that Ethernet Switch can consume a part of network bandwidth as well. The work would help power utilities to determine an optimal number of devices to be installed and to ensure highly reliable data communication network for protection and control functions in an IEC 61850 based digital substation..


Introduction
Multi-function microprocessor-based intelligent electronic devices (IEDs) are widely installed and used in substation automation systems (SASs) to provide measurements, monitoring and control functions for power systems. The IEDs can hardly stay in service for >10 years before becoming obsolescent compared with the conventional analogue relays due to fast technology change demand and lack of availability of technical support. In contrast, the lifespan of primary plants can reach over 40-50 years [1]. Since the life mismatch between IEDs and primary equipment, the replacements or maintenance for IEDs to connecting and disconnecting primary equipment will require outage time. The high cost, high risk associated with outage time will reduce the availability and reliability of electricity transmission and distribution networks. In addition, the data communication between different vendor IEDs is often vendor specific. Due to lack of interoperability between different vendor IEDs, it has been difficulty to consider any new equipment from other vendors during the replacement or maintenance of IEDs.
To address these issues, IEC 61850 standards were released in 2003 [2] with the main aim to promote the interoperability among multiple vendors IEDs. To improve the flexibility and reliability performance, UK National Grid completed Architecture for Substation Secondary System (AS 3 ) project to standardise digital interfaces between IEDs and primary equipment [3]. However, AS 3 project proposed two independent parallel star networks at process bus level. The performance of two independent process buses network is evaluated and analysed [4]. However, it focused on protection control equipment redundancy and the communication network redundancy was not considered. Two reliable networks with redundancy protocols are published in IEC 62439-3, they are High-availability Seamless Redundancy (HSR) and Parallel Redundancy Protocol (PRP) [5]. Huu-Dung [6] have assessed and evaluated the HSR network performance on process level by OPNET simulation tool. Xi and Peter [7] designed a test platform in the lab for a process bus with PRP to access the IEDs interoperbility performance from different suppliers. The authors of [7,8] present a substation with the implementation of HSR protocol and PRP, respectively. Recently, Scottish Power Energy Networks (SPENs') has considered to deploy a real live full digital substation for a 275 kV substation in Scotland through UK government Ofgem' Network Innovation Competition (NIC) funded Future Intelligent Transmission Network Substation (FITNESS) project [9]. In order to test the reliability and data capability performance, FITNESS architecture considers all three redundancy networks and will test their performances in a live substation.
This paper presents the assessment of process bus network capability performance and the comparison between the mentioned three different process bus redundancy networks. First, the FITNESS architecture with three redundancy networks is presented and explained. The protection scheme for the three different process bus redundancy network topologies and protocols are modelled and implemented using a data communication network simulation software tool, OPENT. The case studies for the network capability performance with respect of the system time delay for the typical protection and control equipment requirements (including IEDs and merging units -MUs) in one bay are considered and analysed. In addition, the network bandwidth utilisation study has been carried out. Fig. 1 shows the single-line diagram for a typical UK 275 kV substation, which contains a bus coupler bay, two feeder bays, a bus section bay and a transformer bay. The protection and control scheme in each bay is independent with other bays.

FITNESS architecture
FITNESS architecture has implemented two feeder bays for the 275 kV substation. The two bay configurations in FITNESS architecture are presented in Fig. 2.
FITNESS architecture uses two vendors' IEDs. Each vendor takes the responsibility for its own designated bay, but the bay needs to incorporate the other vendor's IEDs. Vendor A bay is applying HSR protocol and vendor B considers two independent parallel star networks at process bus level and PRP at station level.

Process bus level network model
In the structure of FITNESS architecture, each bay consists of two main protections (MP1 and MP2), two MUs and two circuit breaker controllers (CBCs) which are connected to the process bus network or the station bus.

Process bus with two independent parallel star networks:
The two independent parallel star networks are the most straightforward configurations which consist of two separated star process buses 1 and 2. As shown in Fig. 3, the equipment, e.g. IEDs and MUs are connected to their corresponding process bus 1 or 2, respectively. Failure of any devices in one of the networks will not affect the operation of the other network. Therefore, process bus 2 can be regarded as a backup solution for process bus 1 failure and vice versa. Hence the reliability performance for the protection and control function can be improved effectively. Fig. 4, HSR process bus network basically forms a ring network that allows two independent communication paths (clockwise and anticlockwise) between the source and destination. In order to realise the function of HSR configuration, devices have to be able to support a doubly attached node with HSR (DANH) [10]. Devices that are unable to support DANH are known as a single attached node (SAN), SANs cannot connect to HSR network directly and would require pairing with Redundancy Boxes (RedBoxes) in order to transmit message packets in both directions within the ring network. Moreover, for the communication between the HSR process bus and the Station Bus, a Redbox is necessary as well.

Process bus using a parallel star network with PRP:
High availability for the process bus can be provided by two fully connected parallel star networks with PRPs. As the message packets are duplicated and sent via two star networks, any one of networks failure will not affect the protection and control function. Similarly, PRP network also requires software functions to support Doubly Attached Node with PRP (DANP) regards as DANP devices. IEDs without DANP function are regarded as SAN devices which will require a RedBox to be connected into the PRP network, in order to transmit two copies of message packets into two separate paths. Fig. 5 shows the overview of the process bus with PRP and DANP devices.

Substation process bus modelling
The process buses with three described redundancy communication networks described above are modelled and simulated, respectively, using OPNET.
The time triggered message Sampled Value (SV) packets are published in MUs which carries a fixed data set containing voltage and current information from instruction transformers. SV multicast capabilities permit the same frame transmit to multiple IEDs. The IEDs subscribe to the desired SV streams from the process bus. Event triggered message is modelled as GOOSE. Since GOOSE trip messages are mainly from IEDs to CBCs, the time requirement for the GOOSE messages is very short. This can ensure the status of the circuit breaker to be updated immediately. Once a fault is detected by an IED, a sequence of GOOSE messages will be generated to the CBC via the process bus. The CBC will then trip the CB in a short time. The packet size and interval time of each message are shown in Table 1. And 100 Mbps fully-duplex Ethernet links are utilised to connect devices in the network.

Process bus with two independent parallel star networks modelling:
In this case, two star networks forming process buses 1 and 2 are running separated without affecting the other operation. Thus, one of the star networks, either process bus 1 or 2 can be chosen for the process bus capability performance studies. As shown in Fig. 6, MP is connected with its corresponding an MU and a CBC via central Ethernet switch. These four devices are modelled using OPNET model library. Continuous SVs are generated in MU while GOOSE messages are sent from MP and CBC. Rapid Spanning Tree Protocol (RSTP) is utilised in process bus with two independent parallel star networks. Process bus 1 has been acted as the main protection in the substation. Therefore, in case of any Ethernet link disconnected in process bus 1, a short period of time is needed for using backup protection (process bus 2). As a result, the recovery time of RSTP may not meet the IEC 61850 SAS communication network performance requirements.

Process bus using a ring network with HSR protocol modelling:
The process bus using a ring network with HSR protocol modelling is presented in Fig. 7, each device model is programed and configured as DANH to realise the following functions. The message frame will be generated and duplicated as HSR frame A and HSR frame B in the source IED, and these two frames will be circulated in separate directions into the HSR ring simultaneously. Intermediate IEDs will receive and forward the frame to the adjacent IEDs within 5 μs [11]. Thus, after propagating in the whole ring, the frame will return to the source IED, and then be discarded. For the destination IED, it will be reached by the duplicated frame successively, in the case of one frame loss. The first frame is considered to accept by an end node and the other copy of frame should be discarded. Therefore, HSR is efficient to cope with single point failure on the Ethernet link and zero recovery time is provided. Fig. 8 presents the simulation scenario for process bus using a parallel star network with PRP. DANP has been configured in this network. And DANP has two individual communication ports in each device. Two independent ports only transmit two copies of frames without receiving and forwarding. However, all frames will be duplicated and transmit into two separate networks as LAN A and LAN B simultaneously. It should be noted the topologies of LAN A and LAN B are not required to be identical. The frame arriving first at the destination node is received whilst the later one will be discarded. All traffic will never return to the source. In addition, normal Ethernet switches can be utilised in this network.

IEC 61850 SAS network performance requirement
The system time delay is an important indicator to reflect the performance of a substation communication network, which is defined in IEC 61850-5 [12]. The message time delay consists of the total time it generated in source IED and collect by destination IED completely through a communication network. In this paper, three layers in the IED transmission stack are considered. As Fig. 9 shows, they are an Ethernet layer, TCP/IP layer and application, respectively. During the message transmission process, the time delay may raise in three different stages which are shown in below: Stage 1: Time delay in the source IED T a =T sa + T st + T se Stage 2: Time delay in the network transmission (Ethernet link, Ethernet switch) T b Stage 3: Time delay in destination IED T c = T da + T dt + T de Finally, the end-to-end time delay is given T=T a + T b + T c . Time requirements for different messages have been illustrated in Table 2. The maximum transfer time of each message is specified. The time is depended on the message applications within the network. It can be obtained that substation communication network time delay should within 3 ms to avoid message packets delay or even loss.

Results and analysis
The simulations have been conducted for several times with a different number of MUs to access and evaluate the performance of three kinds of process bus network. During the simulation, each scenario has been run for 30 s. Results of network time delay and communication network bandwidth utilisation under different situation are presented.

System time delay study
Fig. 10 shows the system time delay for three redundancy networks with basic protection scheme as two MUs connected. It can be inferred that the time delay for the HSR network is 0.095 ms which is longer than PRP (0.075 ms) and two independent parallel star networks (0.065 ms). This is because message packets need long distance to arrive destination in a ring topology, while message packets are propagated through central Ethernet switch to be received by IEDs in a star topology. And PRP is more complex than a star network. In this scenario, all three networks are performed well. Fig. 11 presents the simulation results of system time delay for three networks under increasing number of equipment being connected. As can be seen that the system time delay will be increased with more devices (MUs) connected to the network. While the relation between system time delay and number of MUs is almost linear. According to the network performance requirement, the maximum number of MUs is able to connect to each network can be easily found. The capacity of the HSR network is 8 MUs, and the capacity for the PRP network and single star network has doubled as 17 MUs. Therefore, the capacity of two independent parallel networks can be reached up to 34 MUs based on the assumption. In the case of one more MU linked to the network, the system time delay will be over 3 ms. It is necessary to emphasise the duplicated frames have a strong occupation in the Ethernet link in the HSR network. Therefore, the capacity of the HSR network will be limited by these extra data. Although all frames are duplicated in the source IEDs for PRP network, they are transmitted into two star networks (LAN A and LAN B). Thus, the capacity could be doubled. Compared with the two independent parallel star networks, the capacity of the PRP network has been reduced by half. It is because two copies of frames are generated in a DANP node and propagated into two separate central Ethernet switches. However, in the two independent parallel star networks, each time an MU will only transmit one frame to the Ethernet central switch.

Network bandwidth utilisation study
As the network utilisation varies with a different number of equipment connected. Table 3 shows the network bandwidth utilisation in each redundancy network. For HSR network, when nine MUs have been connected, 82.2% communication channel is occupied but system delay time is far more than the performance requirements. This is because DANH can occupy a part of network bandwidth for message receiving and forwarding. Under this circumstance, message packets are queueing to pass through DANH. In the same way, it can be found in the table that the central Ethernet switches in PRP network and single star network are taking over a part of the network bandwidth as well. And the simulation results have revealed that with the same number of MUs connected, the network bandwidth utilisation of HSR network is more than doubled compared with the other two networks, since the message packets will be duplicated and propagated into separate paths for the whole ring in the HSR network, the packets from the source have to return to the source and discard while the packet from the source will determinate at the receiver. In PRP network and single star network, the message packets will never return to source IEDs.

Conclusion
This paper presents the FITNESS architecture with the three different redundancy networks including process bus with two independent parallel star networks with no redundancy protocol, process bus with a ring network with HSR protocol, process bus using a parallel star network with PRP. The methodology for the data flow capability performance assessment based on a typical the protection and control equipment requirements is described and discussed. Three redundancy network models based on process level are developed in the OPNET simulation tool. The data flow capability performance for the maximum number of devices (MUs) to be integrated into a full digital substation is also evaluated and analysed. Results show the maximum number of equipment for process bus with two independent parallel star networks has doubled as 34 MUs in comparing them with PRP. The process bus with a ring network and HSR protocol has the lowest network capacity as only eight MUs. For the network bandwidth utilisation study, it can be concluded that DANH and Ethernet switches will consume a part of network bandwidth for message packet  transferring. The results show when considers network redundancy protocols to improve the network reliability, the data transmission capability of the network will be reduced. The results also show that PRP has better performance than HSR. The work will help future digital substation design with considering a trade-off between the network reliability, availability, flexibility and the protection and control data flow capability performance.

Acknowledgments
The author thanks Scottish Power Energy Networks (SPENs), United Kingdom to support this work through FITNESS project.