Type-A Worst-Case Uncertainty for Gaussian noise instruments

An analytical type-A approach is proposed for predicting the Worst-Case Uncertainty of a measurement system. In a set of independent observations of the same measurand, modelled as independent- and identically-distributed random variables, the upcoming extreme values (e.g. peaks) can be forecast by only characterizing the measurement system noise level, assumed to be white and Gaussian. Simulation and experimental results are presented to validate the model for a case study on the worst-case repeatability of a pulsed power supply for the klystron modulators of the Compact LInear Collider at CERN. The experimental validation highlights satisfying results for an acquisition system repeatable in the order of ±25 ppm over a bandwidth of 5 MHz.


Introduction
The crucial specification of Worst-Case Uncertainty (WCU) [14] is often required in industrial production for several engineering fields, such as in some sensitive cases as nuclear or aero-space applications. However, WCU is not included into the Guide to the Expression of Uncertainty in Measurements (GUM) [9], which mainly focuses on providing the standard uncertainty associated with a best estimate of a measurand. More recently, the focus of the GUM working group has shifted towards the probability density function (pdf ) as a comprehensive way to express the experimenter's knowledge of a measurand [8]. Moreover, in the measurement practice, the WCU is usually determined as a Type-B uncertainty [10], e.g. on the basis of manufacturer's production specification of the particular instrument. In many cases of research, when custom-designed instruments have to be characterized, manufacturer's specifications are not available, thus Type-A methods have to be defined for determining WCU. In fact, this turns into the problem of characterizing the extreme values (e.g. the peaks) considered as the worst cases of a measurement system working in its nominal conditions [5].
The Uni-variate Extreme Value Theory, or simply Extreme Value Theory, is widely used for predicting the peak values of a given phenomenon in different fields (e.g., risk management [16,26], finance, economics, or even for estimating the fastest human time on the 100 m sprint [13]), and specifically in engineering, e.g., studies ranging from tides [23] to accelerator technology [12]. As an example, in telecommunications, an accurate expression for the peak distribution of the Orthogonal Frequency Division Multiplexing (OFDM) envelope was determined by the University of Massachusetts [25]. In fact, the main problem about the applicability of OFDM systems in low-power wireless applications is the highly-variable amplitude of transmitted signals. The above study defined a rigorous method for predicting the upcoming peak values of the envelope by the previous peak observations. However, this means that each prediction needs to be supported by an adequate data collection which, in some cases, requires very-long observations of the phenomena, resulting in infeasible test durations.
In this paper, the exact Cumulative Distribution Function (CDF) of the uncertainty of several measurements, modelled as independent and identically distributed (iid) normal random variables, is presented, together with an approximated pdf of the Worst Case Uncertainty. In a set of replicated measurements of the same measurand, the upcoming peaks can be forecast without any knowledge about preliminary observations or data collections, thus overcoming the main issue related to the Extreme Value Theory. In section 2, the metrological problem is formalized and an analytical WCU model is proposed. In section 3, a case study at CERN on a reference acquisition system for assessing the performance of a high-voltage pulsed power supply for the klystron modulators of the Compact LInear Collider (CLIC) is introduced. In particular, in section 4, the numerical results of performance analysis of the proposed model, aimed at confirming the validity of the underlying approximations, are reported. In section 5, the distribution approximated through the model is validated by comparison with experimental results obtained with the reference acquisition system for the CLIC klystron modulators. Finally, in the last section, the rigorous WCU equation is derived and the model approximation validity is demonstrated.

Worst-Case Uncertainty theoretical model
In this section, an approximated pdf of the type-A WCU is defined for Gaussian-noise measurement systems by deriving the exact CDF of replicated measurements, modelled as independent and identically distributed (iid) normal random variables. In particular, after stating the metrological problem, the WCU definition is given, by analysing the three random variables corresponding to its three main operations of difference, absolute value and maximum. Finally, the analytical model is derived, by leaving to the appendix A the rigorous proofs of both its equation and the Gaussian noise approximation.

WCU definition
In many situations, a periodic signal is to be acquired in a given time window, and measurements have to be taken in different periods. Among these periods, the measured instantaneous values at the same (in equivalent time) sampling instant can vary significantly owing to several uncertainty sources. In many research cases when custom-designed instruments have to be characterized, manufacturer's specifications are not available, thus Type-A methods have to be defined for determining WCU. In fact, this turns into the problem of characterizing the extreme values (e.g. the peaks) considered as the worst cases of a measurement system working under its nominal conditions [5].
If the measurand is assumed to be ideal and the measurement system has negligible instability within the signal period, then the measurement is affected only by the instrumental noise, defined according to the Standard IEEE 1057-07 [17] as: "Any deviation between the output signal (converted to input units) and the input signal except deviations caused by linear time invariant system response (gain and phase shift), a dc level shift, total harmonic distortion (THD), or an error in the sample rate".
Hereafter, all the samples of the measurement system noise (and their random effect) are modelled as independent and identically distributed (iid) random variables. By acquiring N s samples per period and a total number N p of periods, the WCU can be defined as in equation (2.1), where three operations are carried out on each of the N o pairs of acquired periods (N o = N p − 1), (i) the difference, (ii) the absolute value, and (iii) the maximum: where V i, j and V i, j+1 are the values of the i th samples of the j th and ( j + 1) th acquired periods, respectively (figure 1), with: In the following subsections, the three random variables, corresponding to the above operations, are analysed for deriving the WCU model equation.

The difference random variable
For a given index j, N s random variables Y i = V i, j − V i, j+1 with 1 ≤ i ≤ N s are defined. Actually, Y i coincides with Y i, j , therefore,in the following, the latter full notation will be used only when necessary for the sake of clarity. The generic V q,k =V q,k + n q,k is the sum of the deterministic sample of the ideal measurandV q,k and the random variable n q,k which is a sample of the stationary stochastic process n(t) that represents the time-domain noise of the measurement system.

The absolute value random variable
Under the assumption of statistical independence upon the index j (the issue of statistic dependence will be discussed in detail in the subsequent sections), the Cumulative Distribution Function (CDF) is computed: By assuming that each V i, j is symmetrically distributed around its mean value, each Y i is symmetrically distributed around zero: Finally, the absolute value is only defined for positive values of x: The pd f is given by the derivative of the CDF: -4 -2015 JINST 10 P07007

The maximum random variable
Let us consider now the maximum M j between the N s independent and identically distributed (i.i.d.) random variables |Y 1 |, |Y 2 |, . . . , |Y N s | (taken from the j th and j +1 th periods). The dependence upon j has been already neglected, therefore M j = M = max {|Y 1 |, |Y 2 |, . . . , |Y N s |}. Given the i.i.d. assumptions, the CDF of M is expressed as [11]: The CDF of M can be now expressed as a function of the CDF of the underlying distribution Y i .
The pd f is simply given by the derivative of the CDF: Equation (2.10) describes the distribution of WCU j , as the maximum of the absolute value of the difference of all the samples between two consecutive periods (the index j is fixed).

Analytical model
Once the WCU was estimated for a given sample size N o , the variability of M over the N s periods is considered: Having assumed the independence on j, the CDF of Z = Z ind can be computed straightforwardly by simply raising (2.9) to the N th o power: In the particular case of independent random variables, the analytical equation of the PDF of the WCU distribution is finally assessed by computing the derivative of the equation (2.12): So far, the validity of all the equations is guaranteed by assuming that: (i) all the variables V i, j are symmetrically distributed around their mean value (or equivalently, each Y i, j is symmetrically distributed around zero), and (ii) the dependence on j of Y i, j can be neglected. Whereas the first assumption is very weak and fully realistic, the independence on j is very strong and not credible, therefore it must be dealt with properly. In appendix A, the actual WCU distribution will be presented and discussed. In addition, it will be also proved that the distribution (2.13) is a worst case approximation of the actual distribution in the case of white and Gaussian noise n(t) of the measurement system. In section 4, it will be shown that the worst-case approximation accurately models simulation data, whereas in section 5, also the hypothesis of white Gaussian noise will be tested against experimental data of the case study.

Repeatability case study at CERN
At CERN, a new particle accelerator is currently under study, the Compact LInear Collider (CLIC). This new electron-positron collider would provide significant fundamental physics information even beyond that available from the Large Hadron Collider as a result of its unique combination of experimental precision and high energy [6]. The accelerating principle relies on energy transfer from a drive beam, which is generated in a classic linear accelerator using more than 1600 klystrons [20] in synchronized pulsed mode for RF power production, to a main beam. The nominal drive beam energy can only be reached with more than 1,000 klystron modulators, thus, the overall efficiency of the klystron and the modulator has to be maximized in order to confine the overall power consumption within an acceptable range [3,6].
In table 1, the CLIC klystron modulators pulses specifications are reported. The very challenging requirements for the RF power quality, derived directly from the accelerator performance specifications, are met by imposing a flat-top repeatability (according to the related standard [1] the flat-top is the pulse level), down-stream of the modulators, better than ±100ppm. A specifically designed reference acquisition system [4] (figure 2) was proven experimentally capable of assessing this performance. In this context, the focus is mostly dedicated to the Worst-Case Repeatability (WCR) which is defined as: ) are the instantaneous voltage values at the same (in equivalent time) sampling instant i within two consecutive pulses flat-tops ( j and j + 1). The difference between WCU and WCR is analogous to the difference between precision and repeatability as defined in the International Vocabulary of Metrology (V IM) [2]. In particular, according to the approach of the VIM, the repeatability can be seen as the most restrictive case of the uncertainty, namely when the operating conditions are not varying during the measurement. In addition, within this case study, the general WCU definition, discussed in 2.1, is applied to the particular case of a train of pulses. By assuming the measurand pulses as perfectly repeatable, the differences defined in (3.1) should be exactly equal to zero. However, even under this assumption, the WCR measurement is affected by the instrument noise. In fact, the reference acquisition system developed at CERN -6 -  has suitable stability within the repetition period of the train of pulses (20 ms) [4], thus its noise is the only factor affecting WCR, because all the long-term effects (e.g. temperature drift) can be neglected. In particular, its analogue front-end exploits a very low-noise difference amplifier to implement the difference between the input pulse and a DC voltage reference set at the nominal amplitude of the pulse itself (it also introduces an amplification down-stream of the difference operation). This allows the acquisition board to acquire only the interesting part of the pulse, the flat-top, by accordingly taking advantage of its high-resolution. In table 2, the full specifications of the reference acquisition system [4] are reported.
The theory reported in the last section can be applied to this experimental case study; in fact, the definitions of WCU in (2.1) and WCR in (3.1) (symbols are the same on purpose), are exactly equivalent: the only difference is that in the CLIC context, the measurements are taken under repeatability conditions [2].

Noise standard deviation
From the WCR definition (3.1), the differences of two flat-tops samples are distributed as Y i . In addition, under the ideal hypothesis of perfectly repeatable flat-tops, two counterparts samples will have exactly the same value. However, the acquisition system will introduce some additive noise on the flat-tops (mostly due to the analogue front-end [4]).

Number of samples
Each pulse has a Flat-Top Duration (FT D) of 150 µs (table 1) and it is acquired at a Sampling Rate (SR) of 15 MS/s. By considering the flat-top samples as iid random variables, the numerosity N s should be therefore chosen as:

Number of observations
In the context of the case study, a pulse has a repetition period of 20 ms; the number of observations N o (sample size) is therefore strictly related to the time duration of the acquisition (observation time). As an example, if the WCR has to be verified over s, 50 pulses have to be acquired. This parameter should be chosen according to the WCR (or, in general, WCU) specification requested by the specific application.

Numerical results
In this section, a simulation analysis for (i) validating the assumptions underlying the model, and (ii) characterizing the model at varying its main parameters is reported.

Model goodness
In the following, two simulation trials in MATLAB proving the effectiveness of the approximated equation (2.13) are reported. Two sets of samples, each of them composed by N s Normal random numbers, are generated N o times by the MATLAB's randn function for emulating the acquisition of N o consecutive pulse flat-tops. randn is not guaranteed to generate a zero mean sample, or a negligible mean value, therefore this effect is always compensated before further processing. The WCR definition (2.11) is applied to calculate WCR(N o ). The statistical sample as a whole is generated by reiterating the simulation N test times. A χ 2 test [18] is then carried out in order to verify that the data in the sample under study are distributed according to the proposed distribution. The χ 2 test is carried out at a significance level α = 0.1%, in order to control the nonreproducibility rate to usual levels of scientific evidence, as recently suggested in [18]. Compatible results between simulation and experimental tests with the CLIC reference acquisition system are obtained by assuming an additive white Gaussian noise with σ = 3.12 ppm of Full Scale as the underlying distribution with CDF equal to F Y i (z) (in [4], the acquisition system's RMS noise was assessed to be equal to 3.12 ppm). In table 3, the model parameters for the simulations are reported.

WCR over 2 pulses
For the first simulation trial, the distribution of WCR, estimated with a sample size N o = 1 (corresponding to N p = 2 pulses), is depicted in figure 3, where both the normalized histogram obtained in simulation and the analytical formula (2.13) are shown. A χ 2 test gave not reason to doubt about the validity of the proposed analytical formula, at a significance level α = 0.1% (the goodness of the fit can be evaluated also visually).  figure 4, the maximum value among 180, 000 pairs of pulses is represented as a single count in the normalized histogram. This results in a higher mode of the distribution. Also in this case, the proposed model fits the simulated distribution, as confirmed by a χ 2 test at the above significance level α = 0.1%.

Model characterization
Once the Once the assumptions underlying the proposed model were validated by simulation, the mode of the WCR distribution can be predicted at varying (i) the sample size, and (ii) the RMS noise.

Mode of WCR versus sample size
In figure 5, the trend of the mode is depicted for N o ranging from 1 to 10 5 (with σ = 1 ppm). However, the model can be generalized for higher σ , for systems with higher noise. A logarithmic dependency of the WCR versus the sample size N o is also highlighted.

Mode of WCR versus RMS noise
The trend of the mode of the WCR distribution versus σ was estimated for N o = 1 ( figure 6, left). The linear relation between the two variables is highlighted by the goodness of linear fitting based -10 - on Least Squares Errors (LSE), as well as by its residuals (figure 6, right). In this case, a model without known term was used to consider the limit (ideal) condition σ = 0 which has to return a mode exactly equal to zero.

Experimental proof demonstration
An experimental proof demonstration was carried out for the above mentioned case study at CERN. The reference acquisition system for the CLIC klystron modulators is composed by a custom ultra low-noise analogue front-end [4], and a high-speed high-resolution acquisition board, the NI − PXI 5922. The acquisition system has suitable stability within the period of the pulse's train (20 ms), thus all the long-terms effects (e.g. temperature drift) can be neglected and the only factor affecting WCR is the instrumental noise.
In the following, the model is characterized by verifying (i) the assumption of additive white Gaussian noise, and (ii) the behaviour for the required sample size (short-term prediction) and an 1-year long time window (long-term).

Additive white Gaussian noise
The reference acquisition system was carefully characterized in terms of its internal noise by means of the test set-up sketched in figure 7 with shorted and grounded inputs. In this section, the hypotheses of additive Gaussian and white noise are verified by means of a χ 2 test (Gaussianity), and an auto-correlation test (whiteness).

Gaussian noise model
Under the hypotheses that (i) the acquisition board NI − PXI 5922 does not saturate during the sampling, and (ii) the internal noise of the analogue front-end (figure 7) has a standard deviation greater than 0.4∆, with ∆ the quantization step, it has been reported in [19] and demonstrated in [7] that its internal noise can be accurately modeled as Gaussian noise. In [4], the noise standard  deviation was assessed to be 3.12 ppm of Full Scale, much greater than 0.4∆ 5922 ≈ 0.4 ppm of Full Scale (all values reported are Referred To Input, RT I), according to the specification of effective number of bits in the data-sheet. The goodness of the Gaussian model with σ = 3.12 ppm of Full Scale for the system's noise was also verified by means of a χ 2 test (α = 0.1%). The test results are depicted in figure 8, where the normalized noise histogram built by a 10,000 sample size is compared with the Normal fitting.

White noise model
The other assumption to be verified is the whiteness of the instrumental noise. By definition, a white noise has a normalized impulsive auto-correlation ideally equal to 1 at zero-lags and 0 elsewhere. In the practice, noise whiteness is tested by verifying that, for every lag = 0, the absolute value of its auto-correlation function is below a given threshold corresponding to the confidence interval.
Thus, the noise was acquired within the flat-band of the instrument, namely 5 MHz [4]. The results of figure 9 highlight clearly the impulsive shape of the auto-correlation function, confirming the initial assumption of white instrument noise.

WCR prediction
In the following, the model is verified for the required sample size (short-term prediction) and an 1-year long time window (long-term).

Short term
By means of the same test set-up of figure 7, the intrinsic WCR of the system was characterized according to equation (2.11) over the desired sample size N o = 3000 − 1 (1 minute of acquisition). The statistical sample is generated by reiterating the whole measurement N test = 500 times (note that such a measurement has a duration of about 500 minutes).
In figure 10, the measured WCR is compared with the proposed model. Also in this case, the χ 2 test did not reject the hypothesis that the acquisition system's WCR is actually distributed according to the proposed distribution at a significance level α = 0.1%.

Long term
The agreement shown during these tests allows the WCR of the system to be predicted for larger N o , when the actual measurement would be infeasible due to the extremely long test duration. Assum- ing the total RMS noise of the reference acquisition system to be not greater than 3.12 ppm of Full Scale, the estimated WCR distribution is depicted in figure 11 for N o = 1.5768×10 9 , corresponding to 1 year of acquisition considering the CLIC case study (1.5768 × 10 9 · RP = 365 days).

Conclusion
An approximated statistical model for predicting the distribution of the Worst-Case Uncertainty, based on a type-A approach, has been proposed. Numerical simulations demonstrated its effective capability of fitting the actual distribution and, thus, validated the model approximation. The model was also verified by comparing predicted and experimental distributions measured by means of the reference acquisition system for the CLIC klystron modulators at CERN (where the requirements concern the Worst-Case Repeatability). However, the model can be generalized for any measurement system by simply characterizing its instrumental noise (assumed to be white and Gaussian) in terms of standard deviation, to be provided as input to the model. As a matter of fact, the model allows the WCU of a measurement system to be predicted for any sample size and this analytical tool can be used to formalize the uncertainty requirements of a measurement system.

A Analytical derivations
In this section, the actual WCU distribution is derived analytically, by proving also that the distribution (2.13) is a worst-case approximation of the actual distribution in the case of white and Gaussian noise n(t) of the measurement system.

A.1 WCU for a given sample size with Gaussian noise
In this subsection, the rigorous equation of the WCU distribution is presented and discussed.
The assumption of whiteness and Gaussianity of the stochastic process n(t) (introduced in section 2.1.1) guarantees both the symmetry and the i.i.d. hypotheses, therefore each V i, j is distributed -14 -

JINST 10 P07007
as N (µ i , σ 2 ). These samples, 1 ≤ j ≤ N p , can be rearranged in a random vector V i ∼ N N p (µ i µ i µ i ,Σ Σ Σ), with N p × N p co-variance matrix Σ Σ Σ: can also be arranged as a random vector: where the transformation matrix M is described as: The random vector Y i is therefore jointly Gaussian and The expression (A.4) exploits the following identities: The covariance matrix of Y i also has a very simple structure: Expression (A.7) clearly shows that adjacent samples, with respect to j, are not independent since the covariance matrix is not diagonal, but statistic dependence is limited to adjacent samples only, therefore the covariance matrix structure is rather simple.
The PDF of Y i is therefore completely known, and the independence on i can be now exploited fully to obtain the distribution of WCU(N o ). WCU(N o ) can be rewritten as WCU(N o ) = Z = max j max i Y i, j , where 1 ≤ j ≤ N o , 1 ≤ i ≤ N s .
-15 -By swapping the order of the maximization, the CDF of z i = max j Y i, j can be calculated: The independence upon i allows the CDF of WCU(N o ) to be calculated by simply raising to the N th s power the CDF of z i written in (A.8). The next equation summarizes all the involved parameters: It is worth noting that the approach used to derive (A.9) is reversed with respect to the one used in 2.2; first the maximization over j, dealing with statistic dependence, and then over i exploiting the statistic independence. As asserted in 2.2, the A.2 proves that (2.12) is the CDF of a random variable Z ind that is a worst case of the random variable WCU(N o ) in the sense of the stochastic dominance.
A.2 Dominant approximation of WCU for a given sample size with Gaussian noise As asserted in 2.2, it can be proven that (2.12) is the CDF of a random variable Z ind that is a worst case of the random variable WCU(N o ) in the sense of the stochastic dominance: From equations (A.8) and (A.9) it yields that F WCU(N o ) (z) = F N s z i (z). In [21] and [15] an inequality, found in the early 60 s, is presented, which is equivalent to Slepian's inequality [22] for the absolute value of Gaussian random vectors. For any N-dimensional Gaussian vector U, this inequality states that: Pr {|U 1 | ≤ u 1 , |U 2 | ≤ u 2 , · · · , |U N | ≤ u N , } ≥ N ∏ j=1 Pr U j ≤ u j (A.11) Inequality (A.11) shows that the joint CDF of dependent and jointly Gaussian random variables is always greater than or equal to the joint CDF of the same random variables assumed independent in the sense of factorizing the marginal CDFs. Such inequality can be particularized for N = N o , u j = z,∀ j and U j = Y i, j as follows: The right hand equality exploits equation (2.5) and the now fully justified i.i.d. hypothesis (actually only identical distributions are needed given the already factorized probabilities). Given the independence on i, both the sides of inequality (A.12) can now be raised to N th s power; by means of (2.12) and (A.9) it is eventually possible to write: It is worth noting that inequality (A.11) is not a direct result of Slepian's inequality; indeed for the case under study here, extending the lower integration limit to −∞ in (A.9) would have given exactly the opposite result.
The covariance matrix of Y i , assumed to be independent as done in 2.2, would be: Hence, Σ Σ Σ Y i (q, k) ≤ σ 2 M M T (q, k) ∀q, k and, due to Slepian's inequality, the exact CDF of the maximum of Y i (instead of |Y i |) would have been smaller or equal to the approximated one obtained by neglecting statistical dependence.
In conclusion, for white Gaussian instrumental noise, the PDF of WCU(N o ) can therefore be approximated in the worst-case sense through (2.13), by simply assuming Y i ∼ N (0, 2σ 2 ).