Metric Indices for Performance Evaluation of a Mixed Measurement based State Estimator

With the development of synchronized phasor measurement technology in recent years, it gains great interest the use of PMU measurements to improve state estimation performances due to their synchronized characteristics and high data transmission speed. The ability of the Phasor Measurement Units (PMU) to directly measure the system state is a key over SCADA measurement system. PMU measurements are superior to the conventional SCADA measurements in terms of resolution and accuracy. Since the majority of measurements in existing estimators are from conventional SCADA measurement system, it is hard to be fully replaced by PMUs in the near future so state estimators including both phasor and conventional SCADA measurements are being considered. In this paper, a mixed measurement (SCADA and PMU measurements) state estimator is proposed. Several useful measures for evaluating various aspects of the performance of the mixed measurement state estimator are proposed and explained. State Estimator validity, performance and characteristics of the results on IEEE 14 bus test system and IEEE 30 bus test system are presented.


Introduction
Phasor measurement units (PMU) are considered to be important devices used as measurement technology on power systems, due to its unique ability to sample analogue voltage and current waveform data taken at distant points.
PMUs are equipped with Global Positioning System receivers allowing the synchronization of the several readings using the GPS-clock as it provides a precise timing pulse time-stamping power system information [1]. All information is sent to phasor data concentrators that compute the sinusoidal wave and the phasors representing the magnitude and phase angle of the voltage and current from widely dispersed locations in the system. As PMU technology is widely available and PMUs are beginning to be extensively deployed in electric power industry there is an increasing use of this measurement units in modern power systems for a wide range of applications. One well known application of the PMU bus phase angles is within the state estimator. State estimation is an important EMS application that provides real time, reliable and qualitative information on the system state. Many studies have been made on the subject [2], [3], [4], [5], [6], [7], [8], [9], [10], [11].
The worldwide deregulation process has significantly affected power system management and energy markets, as competitive markets will lead to more efficient power generation, more technological innovations and, eventually, to lower retail prices. In addition the power system network is growing larger and more complex as there are many independent power generators adding generation to the electric system and a continuous increase of load demand. In this situation, the function of state estimation is becoming more important, because it is the main tool for monitoring and control based on the real-time data received from the measurement system.
Due to their cost, the integration of these PMU devices will not be instantaneous and it is unlikely that they will replace SCADA system for state estimation and other EMS functions in a near future too. Based on this idea, this paper, under given the placement of PMU in the power system, presents a simple WLS mixed measurement (with SCADA and PMU measurements) based power system state estimation model and algorithm. Due to the simplicity of the relations between PMU measurements and the state variables, such approach opens the possibility of obtaining contribution to an effective improvement on state estimator performances. A key aspect in the evaluation of state estimation algorithms is the selection and proper interpretation of the metrics used for measuring and determining the performance and characteristics of the algorithm. As the estimator is predominantly datadependent, that is of a statistical nature, its performance should be evaluated in a statistical sense, based on statistical measures. Various metrics have been adopted for assessing the effectiveness of state estimator in other technology areas.
To obtain the results stated, a MATLAB software package was developed and used in this work. The paper is divided into four sections. The first section presents an introduction to the subject and explains the motivation of the developed work. The second section deals with the mathematical formulation of the problem. The third section deals with issues related to the problem implementation. State estimator simulation results on IEEE 14 and IEEE 30 bus test system are also presented in the third section. The simulation results are examined and discussed exploring the statistical measures for evaluating various aspects of the performance of the implemented estimator. 14 IEEE bus test system and 30 IEEE bus test system are used as test systems. Conclusions are presented in the fourth section.

WLS State Estimation Method
The power system state estimation is formulated based on the measurement equations that, for a given set of bus voltage, line flows and injection measurements, z, related to the vectors of state variables, x, and measurement noise e, such as [12]: The function h( x) corresponds to the nonlinear function relating measurements to the system states. The state vector x is of dimension n and the measurement vector z of dimension m where n < m.
The voltage magnitude and phase angle at all buses are chosen as state variables, defined by a vector x.
The achievement of an estimate for the system state variables vector using the weighted least square method (WLS) consists in determining the state vector x that minimizes the following function: ( This estimate should satisfy at least the first order optimality constraints as: where H( x) is the jacobian matrix of h( x) for x, i.e: Due to the non-linearity of the function h( x), expanding the nonlinear function g( x) into its Taylor series expansions we have: The first-order Taylor series expansions of (1) yield: where x 0 is initial estimate for the start of an iterative process, k is iterations counter and x k is solution vector obtained at k-th iteration.
PMUs have the ability to measure the voltage magnitude on buses and also the currents in adjacent lines. The incorporation of these phasor measurements will extend the measurement vector as it contains voltage phase angle and current phasor measurements, besides conventional SCADA measurements (voltage magnitude, active/reactive power injections and power flow measurement). The approach followed considers the injected current at the buses that have a PMU placed. The use of the PMU data further complicates the approach because it creates conflicts between quantities expressed in rectangular coordinates and polar coordinates. System state is expressed in polar coordinates, thus the injected currents are expressed as nonlinear functions of magnitude and phase angle voltages at buses. Evidently, phase angle and magnitude of injected currents can be calculated from the rectangular components, but the problem lies in the characterization of the covariance of measurement errors, [13].
Adding phasor measurements, due to the use of PMU, to an existing system which already contains m measurements causes the jacobian matrix to augment with added rows corresponding to the partial derivatives of real and imaginary parts of the injected current in order to voltage magnitude and its phase angle (7), (8), (9), (10). The susceptance of the branches were considered as b 0i >> g 0i .
The measurements received from PMUs are more accurate when compared to conventional SCADA measurements. Therefore, including PMU measurements in state estimator is expected to produce more accurate estimates. The structure of the modified measurement jacobian matrix will be as follows: The gain matrix is formed using the measurement jacobian matrix and the measurement error covariance matrix R. This covariance matrix is assumed to be the diagonal measurement variances entries. The variances of the measurements are typically given in terms of variance or standard deviation on the magnitude and angle. The approach followed requires covariance matrix elements in corresponding to phasor rectangular components. Thus, it is necessary to transform them. Since voltage phasor measurements are direct measures, its error covariance matrix can be calculated based on the error distribution. According to [14], the standard deviations of the errors of voltage phasor measurement can be set as 0,0017 rad (phase angle) and 0,002 p.u. (magnitude), and thus their squares are the corresponding diagonal elements of error covariance matrix. The error covariance matrix for phasor currents measurements are calculated as covariance matrix of indirect measurements according to the known error variances of the direct measurements. The variance assigned to each measurement provides an indication of the certainty about that particular measurement.
The variance errors due to the measurement transformation can be calculated by: where σ 2 Ii real and σ 2 Ii imag are the error variances of I i real and I iimag respectively. Thus, with σ θ I i and σ |Ii| the corresponding diagonal elements of error covariance matrix σ 2 Ii imag can be calculated.

Observability Analysis
Observability problem consists on identifying a set of available measurements enough to be able to estimate system state. For observability purposes not only is important the number of measurements but also their types and locations. If the system state is observable, it is relevant to identify measurements that, if missing, render the state unobservable. In this paper, an algorithm based on observability analysis method introduced earlier in [12] is used for observability analysis. It simultaneously allows checking observability, determining all observable islands and irrelevant boundary injections, and identifying pseudo-measurements needed to restore system observability. These pseudomeasurements are taken from available PMU in the system. For observability purposes and without loss of generality [12], (1) can be linearized, thus: where the vector z is linearly related to the ndimensional state vector x containing N -bus phase angle state variables, H( x) is decoupled jacobian matrix of real power measurements relating to the phase angles, e is the additive measurement error vector and the measurement error covariance matrix is assumed to be the identity matrix. The observability of the system is decided through Jacobian matrix analysis. Since measures of active and reactive power are available at the system, observability P − θ and Q − V can be tested separately. An P − θ observability analysis is made. The decoupled gain matrix for the real power measurements is formed as: where the decoupled jacobian matrix for the real power measurements is formed as, Unobservable branches can be easily identified by the factorization of the gain matrix. Cholesky factorization of the gain matrix will be interrupted as soon as a zero pivot is found. Consider the step where the first zero pivot is encountered during the factorization of the singular gain matrix, as shown below: When a zero pivot is encountered, a 1.0 replaces it and the corresponding entry of the right hand side vector t p will be assigned an arbitrary value. These arbitrary values should remain distinct from each other. One possible choice is integers in increasing order, such as 1, 2, 3, etc. Then, forming the following vector can identify the unobservable branches: where A inc is the branch-bus incidence matrix. If an element of P b is nonzero, the corresponding branch will be unobservable. Removing the unobservable branches and injections incident on them, the procedure is repeated until no more branches are found unobservable. Unobservable branches will separate observable islands. Identifying unobservable branches will lead to the identification of the observable islands. A system with no unobservable branches will be fully observable.
The method used selects a measurement set that is able to render possible system observability without contaminating observable region. These measurements are taken from available PMUs in the system. The measurements will make observable all non-observable branches, thus adding the various observable regions of the system into one full observable island. Measurements obtained directly from PMUs as voltage phasors and currents phasors are added to the measurement set. The procedure which realizes complete observability follows the steps listed below [15]: • Use of PMUs measurements available on the system. It's assumed that a PMU at a bus makes that bus and its neighbors observable.
• Establish a measurements set which contain the measures candidates to link the observable regions of the system. These candidates are measures of unobservable branches and buses on the border of observable islands.
• Judge observability of the system according to observability analysis.
• If complete observability has not been achieved then the first steps should be preceded again for the unobservable region until complete observability is realized.
The system observability can be achieved according to what was stated before and it serves the purpose of having a measurement set that guarantees full system observability. It can precede other system analysis as state estimation. Once identified the system observable islands, other measures may be added to the measurement set in order to join these observable regions and establish a single observable region. These measures correspond to flows in the identified unobservable branches and bus injections at buses that are on the border of the observable islands.
The use of PMU at zero injection buses for system observability has the effect of being necessary the use of less PMUs to reestablish system observability. As the intent is to minimize the number of PMUs placed in an N bus system, the use of PMU located at zero injection buses, buses with a large number of connected branches, will maximize the coverage allowing the use of a minimal PMU set.

Measures of Accuracy
The performance of the estimator determines the estimator capability to provide the outputs in due time to be use by other applications in the control center.
The value of the indirectly J x expresses how good the solution of the state estimation is, i.e., the fit between the measurements and linking of system states. This widely accepted index does not fully serve the purposes as this index shows how properly the solution fits to the measurements but it does not reflect how distant the estimate is from the real values. Contrary to that, in this paper it is discussed several indexes that quantify the performance of the state estimator with respect to the deviation of the state vector from the real conditions. Also the convergence capability of the estimator is analyzed resorting to the use of specific indicators.
The metric M conv obj evaluates the relative change in the value of the objective function at the k-th iteration, evaluating its ability to converge [16]. Such metric is defined as: The metric M conv θ measures the largest final relative change in bus voltage phase angle; it uses the absolute difference to avoid problems when the angle is near zero, which will occur near the system reference bus, Other metrics are used to indicate the deviation of estimated state variables from real values. With this purpose it can be used a norm metric M acc v that captures the effect of both voltage magnitude errors and voltage angle errors, as: To evaluate solution accuracy related to voltage magnitude it is used the metric MAPE, which expresses voltage magnitude mean absolute percentage error: In terms of voltage phase angle, estimated values are evaluated using the metrics M AE θ , which expresses voltage phase angle mean absolute error, measuring how close estimated voltage phase angles are to the actual values in the system.
The simulation procedure followed, the test system used and the results of the state estimator simulation analyzed by the metrics referred are presented in the next section.

Simulation Considerations and Results
The proposed method for mixed measurements based state estimator is implemented in Matlab and tested on the IEEE 14 bus and IEEE 30 bus test systems [17].

Simulation Considerations
A set of SCADA measurements mostly power flows and power injections are arbitrarily distributed in the system. A load flow is carried out using the load and generation values and its outcome serves as the true state and true value of measurements that include bus voltage magnitude, bus power injections and power flows. The actual measurements z are obtained by adding true value of measurements with normally distributed noise of 1 % standard deviation for power injections, 0,8 % for power flows and 8 % for voltage measurements. All measurements have the exactly same error characteristics.
For buses that don't have any load or generation, the injection at those buses into the rest of the system is known to be exactly zero. As in these zero injection buses the injection is zero, it can be interpreted the zero injection as an exact measurement of the real and reactive power. One possibility to deal with these implicit measurements on state estimation problem is to ignore the zero injection buses in the formulation. The drawback of this consideration is that if there are many zero injections buses and if they are all ignored, then the remaining system is likely to be unobservable because the remaining measurements will be insufficient to completely determine the variables unknowns.
In order to identify the observable islands in the system a numerical observability analysis as described at Section 2.2 is carried out. The PMUs are placed according to the placement methodology described, so that the total number of PMUs is minimized, and the measurement redundancy at the buses maximized. Voltage phasor measurement errors standard deviations are set to 0,0017 rad for phase angle and 0,002 p.u. for magnitude.
Taking into account the statistical nature of the measurements errors used in the simulation, a large number of trials is performed. For each trial, the sample of a measurement is randomly taken from the normal distribution of the measurement around the measured value. So on each trial state estimator has the same measurement set but new errors are added to the measurements. It was simulated 150 trials. Thus it is possible to discard the statistical nature of measurements error on estimator outcomes. To make it easier to understand and interpret the large set of data from the 150 state estimation trials it was used box plots. It is a very useful tool for graphically portraying the distribution of state estimator results.
The voltage phase angles obtained by state estimator are all with respect to a reference bus. In SCADA measurement system, usually one bus (slack bus for most cases) is chosen as the reference bus to get the relative phase angles of all other buses in the system. Synchronized phasor measurements might have a different reference which is determined by the instant synchronized sampling initiated. If phase angle measurements are added, without considering the different references, the algorithm is not likely to converge. State estimation results will be incorrect if the PMU measurements are used without dealing with the reference problem. The solution followed, was to measure the phase angle of the slack bus (considered the reference in SCADA system) by placing a PMU at that bus and consider it as a common reference. State estimator algorithm convergence tolerance considered was 10 −5 .

IEEE 14 Bus Test System Simulation Results
The IEEE 14 bus test system measurement set with SCADA measurements is illustrated in Fig. 1. There is only one zero injection bus, bus 7, which was taken into consideration in setting initial SCADA measurements allocation. Power flows in all adjacent branches to bus 7 are part of the initial measurement set. After observability procedure completed all IEEE 14 bus test system observable regions are identified, as they are separated by the unobservable branches (see Fig. 1). It was verified that PMU'S were needed to satisfy system observability.
To deal with the reference problem it is required to place a PMU at bus 1, as bus 1 is the slack bus and so it will be considered as a common reference. It was used measures from available PMUs located at the buses 6, 5, 9, 10 and 14. The following locations for PMU were simulated to verify the effectiveness of the performance indexes in state estimator evaluation, creating 8 scenarios: • PMUs at buses 1, 6, 9, • PMUs at buses 1, 6, 14, • PMUs at buses 1, 6, 10, • PMUs at buses 1, 5, 9, • PMUs at buses 1, 5, 14, • PMUs at buses 1, 5, 10, • PMUs at buses 1, 6, • PMUs at buses 1, 9. Figure 2 illustrates the convergence behavior of the state estimator when applied to the IEEE 14 bus test system. It describes the relative change in the value of function J x at the last two iterations for all scenarios, evaluating state estimator ability to converge. Throughout careful analysis of Fig. 2, it can be seen how data from 150 state estimator trials related to the index M conv obj are distributed relative to the position of the measure of central tendency. For the scenario PMUs at buses 1, 6, 9 and PMUs at buses 1, 6 M conv obj data are spread more to the lower part of the center. Also it can be seen that scenario PMUs at buses 1, 5, 10, and scenario PMUs at buses 1, 9 M conv obj data are clustered closer to the median. Scenario PMUs at buses 1, 6, 14 presents the lower relative change of function J x at the last two iterations but also it expresses the more spread out relative to the mean values of relative change of function J x of the 150 trials. It is possible to identify outliers on M conv obj for scenario PMUs at buses 1, 5, 10 and scenario PMUs at buses 1, 9.    Fig. 4 shows a similar behavior on all scenarios results related to both the largest final relative change on bus voltage magnitude and on bus phase angle. Scenario PMUs at buses 1, 6, 9 and PMUs at buses 1, 6 present M conv v data spread more to the upper part of the center. The more clustered closer to the median data is a better choice. It seems to be because the index does not vary as much from simulation trial to simulation trial. Outliers are not present in every scenario. When they are present they are outside of the boundaries of the whiskers. These outliers are extreme values that deviate significantly from the rest of results. This is consistent with the results on measures of the largest final relative change in bus voltage magnitude and in bus voltage phase angle showed by Fig. 3 and Fig. 4 for scenarios PMUs at buses 1, 6, 14, PMUs at buses 1, 5, 9, PMUs at buses 1, 5, 14, PMUs at buses 1, 5, 10 and PMUs at buses 1, 6, 9, PMUs at buses 1, 6, 10, PMUs at buses 1, 9, respectively.
The state estimator performance in terms of accuracy for all simulations is graphically presented by metrics M AP E, M AE θ and M acc v in Fig. 5, Fig. 6 and Fig. 7 respectively.
The mean absolute percentage error (M AP E) is useful because it is expresses, in generic percentage terms, the voltage magnitude error. It is easily demonstrated the estimator results accuracy with respect to voltage magnitude estimates. Figure 5 shows the M AP E for voltage magnitude errors for all simulations scenarios. It can be seen that for almost every simulation scenario M AP E is less than 0,35 %, indicating a very good estimator accuracy on voltage magnitude estimated values. The worst case scenario is for measurement set with PMUs at buses 1, 6. Problems can occur when calculating the M AP E value with a series of small denominators such as voltage phase angle at reference bus. Thus a singularity problem of the form 'one divided by zero' caused by a small deviation in error occurs in this case. Due to that, the metric use to analyze phase angle error was Mean Average voltage phase angle Error (M AE θ ). Figure 6 shows voltage phase angle errors for all scenarios trials simulations. The metric used, M AE θ , indicates the average of the absolute values of the differences between estimated voltage phase angle and the corresponding real voltage phase angle.
Results indicate that the worst case in terms of voltage phase angle estimates is the simulation with measurement set with PMUs at buses 1,6 and PMUs at buses 1, 9. Note that the remaining voltage phase angle errors are lower than 2e −3 rad (approximately 0,1 deg). It is reasonable to compare voltage magnitude errors and voltage phase angle errors separately. M acc v combines both capturing the effect of both, so state estimator should return low values on that metrics.  Figure 7 shows that M acc v is lower than 0,015 for all simulations scenarios. The median data values of M acc v are around 0,01 and the simulations with PMU at bus 1, 6, 10 and PMU at bus 1, 5, 9, are the ones that present M acc v values from the 150 trials more clustered closer to the median value, which indicates that the state estimator for these simulation scenarios has a greater ability to match the controlled test model in the aggregate.

IEEE 30 Bus Test System Simulation Results
The state estimator algorithm is also applied on the IEEE 30 bus test system. Figure 8 shows the bus test system measurement configuration used.
Observability analysis was carried out as described in section 2.2. Once identified the network observable regions, measurements are added to the measurement set in order to join these observable regions and establish a single observable region. These measurements correspond to flows in the identified unobservable branches and bus injections at buses that are on the border of the observable islands. Figure 8 illustrates the observable islands identified. The measurements used to establish a single observable system were from available PMUs located at the selected buses, creating 3 simulation scenarios: • PMUs at buses 1, 2, 6, 10, 12, 15, 16, 28.
A PMU is placed at bus 1 to measure the phase at system slack bus. The other PMU location where chosen to maximize the coverage allowing the use of a minimal PMU set. 150 simulations trials were carried out for the three scenarios. The simulation considerations assumed for IEEE 14 bus test system are the same for IEEE 30 bus test system. Simulation results are analyzed through the use of state estimation performance evaluation metrics when using phasor measurements on state estimator measurement set.
Metrics analysis is presented for IEEE 30 bus test system state estimation solution. Visual comparison is enhanced when using box plots to show metrics distribution calculated from state estimator outputs for each of 150 simulation trials.  Fig. 10 and Fig. 11 show convergence comparison for all three simulation scenarios. It can be said that the relative change of J x at the last iteration is lower for scenario PMUs at buses 1, 2, 6, 10, 12, 15, 16, 28. Scenario PMUs at buses 1, 2, 5, 10, 12, 25, 28, exhibits data related with largest final relative   change on bus voltage magnitude clustered closer to the median. Also scenario PMUs at buses 1, 2, 5, 10, 12, 25, 28 presents the lower values for the largest final relative change on bus phase angle.
From plots presented on Fig. 13 and Fig. 14, scenario PMU at buses 1, 2, 5, 10, 12, 25, 28 has the better state estimator performance in terms of voltage magnitude and voltage phase angle error. For this scenario metrics M AP E and M AE θ have the lower median value and the outcomes of the 150 simulation trials are more clustered closer to the median value when comparing with the other scenarios. M acc v metric box plot also indicates that a very high number of simulation trials results are contained within a very small segment of the sample for scenario PMU at buses 1, 2, 5, 10, 12, 25, 28. Also this scenario presents the higher relative change of function J x at the last two iterations, meaning that state estimation presents better performance in terms of convergence for the scenario PMUs at buses 1, 2, 6, 10, 12, 15, 16, 28.

Conclusion
This paper presents a methodology for a mixed measurements based state estimator. Several practical metrics for evaluating state estimator performance were discussed. They are useful for measuring different aspects of the estimator performance. The algorithm implemented was tested in IEEE 14 bus test system and for IEEE 30 bus test system with different number of PMU measurements and constant SCADA measurement set. Estimator performance evaluation was carried out for the several measurements sets for comparison purpose. Simple metrics cannot capture a complete picture of the estimator but those that are more complete are also more complex and subject to subjective interpretations. The state estimation performance evaluation through the metrics shows that the implemented algorithm for mixed measurements based state estimator can be applied with good results.