Total variation as a metric for complementarity in energy resources time series

,


Introduction
The proliferation of energy sources caused by technological advances and their consequent cost reduction brought the concept of complementarity to the energy jargon.This concept is of particular interest when the resource's availability or their costs are variable in time.Although, up to now, there is no formal definition of resource complementarity in energy supply systems, intuition suggests that complementarity refers to the possibility of alternatively using different resources according to their availability.
Although complementarity entails correlation, correlation does not entail complementarity.For example, the correlation between speed and height in a pendulum is negative, but they are not complementary.
2. Dimensions matter.In the above example, speed and height have different units.Obviously, for two or more variables to be complementary, they must have the same dimension.If they have different dimensions, one can not sum them, and their complementarity has no sense.In the pendulum example above, the complementary variables are kinetic energy and potential energy.3. The scale of the variables does matter.For example, streamflows from a large river and a small creek could exhibit a sizeable negative correlation, but the large one dominates their sum, and hence the complementarity value should be low.

Linearity of the relation is also an issue. Even
if correlation coefficients were suitable for assessing complementarity, the selection of Pearson's, Kendall's, Spearman's or any other coefficient should be done on the basis of some physical or mathematical considerations.
Besides correlation, among the other methods used to evaluate complementarity, one can mention the one proposed by Beluco et al. (2008) as the product of three partial indices: (i) partial time complementarity index, which evaluates the time interval between the minimum values of two sources; (ii) partial energy complementarity index, which evaluates the relationship between the average values of two sources; (iii) partial 85 amplitude-complementarity index, which assesses the differences between maximum and minimum values of the two energy sources.This idea works well for simple annual sinusoidal series but fails for more complicated series.Borba and Brito (2017) 90 proposed the dimensionless ratio between the actual generation discarding excess power and the average generation.This index is more a measure of volatility than complementarity.Han et al. (2019) presented two indices of the complementary rate 95 of fluctuation (CROF) and complementary rate of the ramp (CROR) to analyze the complementary degree of sequences with different fluctuation characteristics.Their rate of fluctuation seems to follow the concept of total variation that we 100 present.Neto et al. (2020) used the concept of Daily Physical Guarantee (DPG), similar to firm power, to evaluate different renewable energy source's complementarity.The general idea is interesting but depends on the system's particular characteristics 105 and does not provide a direct complementarity metric.Canales et al. (2020a); Arias et al. (2015); Canales et al. (2020b) introduced a method that assesses temporal complementarity between three variable energy sources, using a combination 110 of correlation coefficients, Euclidean vectors, compromise programming, and normalization.The method's basis is correlation analysis, and therefore their proposal shares the criticism we made above.For a complete review of different 115 methods to evaluate complementarity see Jurasz et al. (2020).We think that none of those proposals is a satisfactory way of measuring complementarity.
Looking to solve these issues, we propose two novel indices based on the Total Variation theory, 120 described in section 2. To illustrate the use and properties of the new indices, we apply them to four cases of study (section 3): 1) two autoregresive processes; 2) wind vs. hydro in the Colombian electricity market; 3) two hydropower sources in 125 the Colombian electricity market; and 4) fifteen hydropower sources in the Colombian electricity market.

Method
In ordinary language, a complement of something is a thing that completes or brings perfection.In math, two numbers complement each other when their sum is constant, usually 1 or 100.A measure of complementarity or complementariness between two functions should assess the perfection of their sum.For perfection here, we will take the regularity of the sum.Therefore, the measure of complementarity is related to the measure of regularity.A satisfactory way of measuring the regularity of a function is through its total variation, which allows for simple discontinuities and has more satisfactory properties than the variance (Kolmogorov and Fomin, 1970, p. 328) (Lasota and Mackey, 1994, p. 140).
In section 2.1 we review the concept and main mathematical properties of Total Variation.Then, we propose two indices, φ and φ , as novel metrics of complementarity (sections 2.2 and 2.3).Afterwards, we present one theoretical and two real case studies to evaluate the suitability of φ and φ to quantify complementarity: 1) two AR(1) processes (section 3.1), 2) daily wind and hydropower generation time series in Colombia (section 3.2), and 3) the daily inflows to two hydropower reservoirs in Colombia (section 3.3).
Results are compared between φ and φ .In section 3.4 we present the application of φ index for the complementarity evaluation of multiple time series.

Total variation
According to Kolmogorov and Fomin (1970); Lasota and Mackey (1994), the total variation of a function f defined on an interval where the supremum is taken over all possible finite partitions a If the sum is bonded, f is said to be of bounded variation on [a, b].Some important properties of the total variation are worth recalling: 1.The total variation of a constant function is zero, and conversely, From the above properties it is clear that the total variation of a function f with values f (t k ) at the points a = t 0 < t 1 < . . .< t n = b, and linear interpolation in between the given points is This property is useful to compute the total variation of a time series.
To illustrate, consider the following sequence of functions g n (t) = sin(2πnt), n ≥ 1.Clearly, in [0, 1] all the functions have 0 mean and variance Lasota and Mackey (1994).Taking into account the above, it is possible to define a complementarity index.

Total Variation Complementarity Index
Clearly, 0 ≤ φ(f 1 , f 2 ) ≤ 1, and when φ(f 1 , f 2 ) = 1 there is perfect complementary between f 1 and 195 f 2 , for in that case f 1 + f 2 is a constant.The generalization to any number of functions is direct.
As an especial case, consider the Total Variation Complementarity index between the series of anomalies.Clearly, the total variation 200 This is a non-peer reviewed preprint.complementarity φ between the series f 1 , f 2 , . . ., f n , is equal to the complementarity between the series of anomalies with respect to a constant mean for Summary of the φ index characteristics For the extreme cases φ = 1 means that there is perfect complementarity and for φ = 0 there is no complementarity.3. φ is symmetric.4. The φ index presents the same result when is evaluated on anomaly series compared to their corresponding pure series.
5. The φ index could be applied to two or more series.

Variance Complementarity Index
If one wants to stay within the correlation concept, it is possible to correct the use of ρ as a metric for complementarity to consider the variables' scale.For that, considering Eq. 5, one can replace total variation by variance.This is, given two functions f 1 (t) and f 2 (t), not both constant, their variance complementarity index which simplifies to Notice that φ v so defined is the negative of the correlation coefficient times the ratio of the geometric and arithmetic means of the variances.This last factor is always less or equal to one, with equality when the two variances are equal.
Therefore, φ v (f 1 , f 2 ) varies between −1 and 1.It is convenient to re-scale it to the range zero to one by means of Summary of the φ index characteristics means that there is perfect complementarity and for φ = 0 there is no complementarity.3. φ is symmetric.4. The φ index presents the same result when is evaluated on anomaly series compared to their corresponding pure series.5.The φ index could be applied just to two series.
3 Cases of study 245

Auto-regressive process
Auto-regressive processes are a well-known time series method widely used for all sorts of applications (Box et al., 2011).This section shows an expression for the total variance complementarity 250 index between two cross-correlated auto-regressive processes of order one.The purpose is to clarify the two ways of measuring complementarity, namely the total variation and the variance complementarity index.Let z t be a vector time series with two 255 components, x t and y t , both with zero mean and unit variance.The multivariate auto-regressive process of order one is where the matrices A and B are constant parameters, and v t is a vector time series of uncorrelated random variables with zero mean and unit variance.This is: for all j = 0, and I the identity matrix.For j > 0 the vector v t+j is uncorrelated with z t , The second order correlation structure of the process z t depends on the matrices , with components that is symmetric and t-independent.Whereas is not symmetric.
To find expressions for the parametric matrices post-multiply eq. ( 10) by z t and take the expected value Similarly, post-multiply eq. ( 10) by z t+1 and take expected values to get One can then obtain B from Eq. 17 assuming it is triangular.A simple way is to assume that ρ x t+1 ,y t = ρ xy ρ x and ρ y t+1 ,x t = ρ xy ρ y (Matalas, 1967).In that case, M 1 = DM 0 , with D a diagonal matrix Therefore eq. ( 16) is now Similarly, eq. ( 17) simplifies to that one can solve for The expected value of the total variation of an uni-275 variate auto-regressive process (the first component of eq. ( 10)) is Fig. 1 shows the expected value as a function of the lag-one autocorrelation coefficient ρ x .
The result in eq. ( 21) comes realizing that |x t+1 − , where both x t and v t are independent random variables, with zero mean and unit variance.In addition, one usually assumes that their distribution is Gaussian.Therefore, the expected value of n 1 (x t ) is A simple calculation of the expected value in eq. ( 22) gives eq. ( 21).
Similar arguments apply to the calculation of the expected value of the total variation of the sum x t + y t : where the a i,j and b i,j are respectively the components of the matrices A and B in eqs.( 18) and (20).All the random variables in eq. ( 23) are Gaussian, with zero mean and unit variance.v t+1 and w t+1 are mutually independent and independent of x t , y t .The covariance between x t and y t is ρ xy .Therefore, using properties of the Gaussian distribution, the absolute value in eq. ( 23) corresponds to a Gaussian distribution with zero mean and variance equal to Finally, the expected value of the total variation of the sum is Figure 2 illustrates this result.
Using the above results, eqs.( 21) and ( 24), the total variation complementarity index between two auto-regressive time series with lag-one correlation coefficients ρ x and ρ y , and cross correlation ρ xy , is Figure 3 illustrate some particular cases of this result.We are interested in the comparison of φ with φ .Recall that the unit variance of x t and y t implies that φ (x t , y t ) = (1 − ρ xy )/2.A first observation is that for ρ xy = −1 both φ and φ are one, that is perfect complementarity.If ρ x = ρ y the total variation complementarity is independent of ρ x , with φ = 1 − 2(1 + ρ xy )/2 ≤ φ = (1 − ρ xy )/2 (left panel of fig.3).Still within the equality of the lag-one correlation coefficients, the case ρ xy = 0 and therefore φ = 1 − √ 2/2 ≈ 0.29 is interesting in comparison with φ = 0.5.That high a variance complementarity index in the absence of cross-correlation may come from how the range of the index was put between zero and one in eq. ( 9).In fact, positive cross correlations give 0 < φ < 1/2, something that seems high.An alternative to eq. ( 9) could be 1 − ρ xy for negative correlations and zero for positive ones.We think that both alternatives are inferior than the total variation complementarity index.For null cross correlation, of the order of 30% of the total variation of the sum cancels out just by chance, whereas either 50% or 0% seem off.
For constant ρ xy = −0.7 in the center panel of fig.3, the total variation complementarity does depend on both ρ x and ρ y although φ = 0.85.
For the case ρ x = −ρ y (right panel of fig. 3) φ depends on ρ x except for φ close to one.In all cases φ is less than φ .
We conclude from this comparison that the total variation complementarity index has more information than the variance complementarity.

Colombian electricity market
The total installed capacity for electricity generation in Colombia is approximately 17.5 GW, of which 68.3% is hydro while the only one wind plant accounts for near 0.1%.In fact, this big scale 340 difference impacts the complementarity assessment.
To quantify the complementarity between wind and hydro-resources in the Colombian electricity system, we use daily time series provided by XM, the electric system administrator and operator 345 in Colombia.For hydro-resources, we analyze natural inflows (in units of energy) to the largest 15 hydropower reservoirs, with a total effective installed capacity of 7,668 MW (≈ 70% of total hydro-capacity) and records from 2000 to 2020.

350
These rivers are San Lorenzo, Nare, Guatapé, San Carlos, Tenche, Guadalupe, Grande, Porce II, Miel I, Urrá, Betania, Batá, Guavio, Alto Anchicayá, and Salvajina.For details on the hydropower system of Colombia, see XM the internet site.For wind 355 resources, we analyze daily wind power generation from the Jepirachi plant, with 19.5 MW of effective installed capacity and records for the period 2004-2020.Notice that physical dimensions of both series are the same (energy).
Figure 4 shows the estimation of φ and φ indices to the daily series and the annual cycles of hydroinflows and wind generation with moving averages of 31 days (top panel).Considering the large scale difference of the accumulated hydro-inflows and the Jepirachi plant, the scaled series's complementarity was also studied (bottom panel), being the scale factor 1/100 for hydro-inflow series and 1/10 for wind generation series.Table 1 presents the results for all analyzed cases.
As expected, because of the scale differences, φ does not show complementarity, while φ index indicates a complementarity of 50% for the daily and annual cycles.When one uses the scaled series, the φ index increases to 39%.We interpret this scaled analysis to evaluate the resources' potential complementarity if the resources' scale were comparable.
The difference in the results may come because the φ index depends on each series's serial correlation, while φ does not.Besides, there is a tendency of φ to exceed φ, overestimating the complementarity quantification.

market
Nare and Guavio are two important rivers of the hydropower reservoir system of Colombia.The Andes mountain range separates the two basins, causing important differences in their hydroclimatic 390 patterns.For instance, seasonality of rainfall is unimodal in the Guavio basin and bimodal in the Nare basin (Urrea et al., 2019), and streamflows show these regimes too (Fig. 5).According to XM, these rivers belong to different hydrological regions 395 (see details in the XM website).
We computed the complementarity indices between the daily natural inflows (in energy units) of Nare and Guavio rivers for the period 2000-2020.
Figure 5 shows the results, where the left panel 400 shows the 31-day moving average daily inflows and the right panel the annual cycle at daily time-step for both rivers.The scale of both resources is not as different as in the previous case.Consequently, the φ index is similar for the daily and scaled series with values close to 20%.Using the equivalent to eq. ( 25) that considers the values of variance (not shown), with the parameters contained in table 2, one gets φ = 0.20.The same value that one obtains using the time series and eqs.( 4) and ( 5).This correspondence suggests that the auto-regressive model is good to estimate complementarity.Again, φ shows a higher result than φ for all cases, quantifying the complementarity around 38%.The pattern is that the variance complementarity index tends to overestimate complementarity.

Case of study 4: Hydropower integrated sources in the Colombian electricity market
One of the main advantages of the φ index is that one can evaluate the complementarity not just between two but among any number of series.To illustrate this capability, we quantify the complementarity among the 15 largest rivers energy inflows in the Colombian hydroelectric system (described in section 3.2).These rivers are in different Colombia regions with different  Figure 6 present the daily series and annual cycle of the integrated hydro system with moving averages of 31 days.The φ index shows a complementarity of 34% for the smoothed series and 30% for the annual cycles.For the unsmoothed series (not shown), the complementarity is somewhat higher, reaching 38% and 46%, respectively.

Conclusions
Correlation coefficients between time series are the common practice to evaluate and quantify complementarity.Pearson's correlation coefficient, which is the most frequently used metric of complementarity, has several fundamental issues that undermine its usefulness for that purpose.As it is scaled independent, one could use it for nondimensionally homogeneous variables, which does not make any sense.Also, if the series's scale is very different, the correlation coefficient may 450 mislead the real complementarity.We showed that negative correlation is a necessary but not sufficient condition to uphold complementarity, demonstrated that scale has a large effect on complementarity magnitude.Also, that dimensional homogeneity 455 between the variables is a requisite to provide physical sense to the concept of complementarity.
We propose a novel method based on the concept and main mathematical properties of Total Variation.The total variation complementarity 460 index (φ) is a consistent and satisfactory metric for complementarity between any number of series.Index φ measures the sum of the variables' regularity and is unaffected by the issues pointed out regarding correlation coefficients.Some essential 465 properties of φ are: 1) φ is sensitive to the scale of the variables; 2) φ works only with dimensionally homogeneous variables; 3) φ can be used with any number of variables.
Given the unreliability of renewable energy 470 sources, consistent evaluation of complementarity is a critical issue.Our proposed index fulfills this need, making a substantial contribution to energy planning.
Inspired by the total variation complementarity 475 index, we proposed the variance complementarity index to correct the issues identified about Pearson's correlation coefficient in complementarity quantification.
We obtained analytical results for the expected 480 value of two cross-correlated auto-regressive series, demonstrating that the variance complementarity index does not capture all the complementarity's dependencies.In particular, the example shows that serial auto-correlation does affect complementarity.

485
Besides, the applications to the Colombian power system show that the variance complementarity overestimates complementarity.
Finally, the proposed complementarity indices quantify the Colombian electricity system's 490 complementarity due to its hydrological and climatic differences.The total variation complementarity index among the daily series of the hydro resources is 30%.This figure is the first estimation of the complementarity among the 15 main rivers feeding the Colombian power system.The total variation complementarity index, φ, between hydro and wind resources is almost null due to the wind component's negligible contribution.When one uses scaled variables, it is 39%, representing the potential complementarity between these resources and could stimulate a higher level of investment in wind capacity development.The corresponding figures for the variance complementarity index between hydro and wind resources are 50% and 63%.The φ index between Nare and Guavio is 20%, whereas φ = 38%.These two rivers are often considered complementary because of their annual cycles.The indices put a solid basis and a concrete number on those intuitive ideas. 240

Figure 1 .
Figure1.Expected value of the total variation of an univariate auto-regressive process as a function of the lag-one auto-correlation coefficient.The points are the average of 100 simulations.The continuous line is eq.(21).

Figure 2 .
Figure 2. Expected value of the total variation of the sum of two AR(1) series as a function of ρ xy for different values of ρ y = −ρ x (left panel), and as a function of ρ y = −ρ x for different vales of ρ xy (right panel).Points are average of 100 simulations and the continuous line is eq.(24).

Figure 3 .
Figure 3. Expected value of total variation complementarity index as a function of ρ x , ρ y and φ for the sum of two AR(1) series: left panel, ρ y = ρ x ; center panel, ρ xy = −0.7,φ = 0.85; and right panel, ρ y = −ρ x .Notice a region of negative values that correspond to non-admissible parameter values for the AR(1) process.

Figure 4 .
Figure 4. Comparison of φ and φ indices between wind and hydraulic resources in Colombia electricity market, with smoothed moving averages of 31 days.The scale factor in the case of scaled annual cycle is 1/100 for hydro-inflow series and 1/10 for wind-power generation series.

Figure 5 .
Figure 5. Assessment of complementarity by the φ and φ indices between the Nare and Guavio rivers.Left panel: 31day moving average daily inflows.Right panel: annual cycle at daily time-step.

Figure 6 .
Figure 6.Quantification of the complementarity with φ Index among the energy inflows of the main 15 rivers of the Colombian electrical system, with smoothed moving averages of 31 days.

Table 1 .
Table2lists the results for all analyzed cases.Complementarity between wind and hydro resources in Colombia.The daily time series were smoothed, obtaining moving averages of 31 days.Variance has units of GW h 2 in the case of smoothed time series and annual cycle.In all other cases, the variables are dimensionless.ρ 1 represents the exponential value of the slope of the linear regression for the autocorrelograms of both resources.

Table 2 .
Complementarity between 31-day moving average daily streamflows of Nare and Guavio rivers.Variance has units of GW h 2 in the case of smoothed time series and annual cycle.In all other cases, the variables are dimensionless.ρ 1 represents the linear regression slope for the auto-correlograms of both resources.