Characterization of an analog-to-digital converter frequency response by a Josephson arbitrary waveform synthesizer

The capability to generate up to 1 V pure AC signals based on quantum standards marked a milestone on electrical metrology opening new applications that were not possible without this standard. Frequency response characterization of analog-to-digital converters (ADC) is fundamental for precision digital metrology. Several methods have been investigated for this characterization based on thermal converters, programmable Josephson voltage standard or input impedance measurements. This paper describes the method, the results obtained and the uncertainty estimation for the characterization of the amplitude frequency response at different aperture times of the DCV sampling function of the Keysight 3458-A using, for the first time, a Josephson arbitrary waveform synthesizer. This new standard allows one to extend the characterization to a higher frequency range and lower aperture times. The results show that the frequency response does not depend on aperture time and the same frequency correction can be applied in an extended frequency range. The knowledge of this correction will facilitate the application of the ADCs to higher frequencies, where low aperture times are required with accuracy in the order of V/V.

to the measured magnitude. Dynamic measurements are currently critical in many applications where the RMS value of the electrical signal does not provide enough information and the signal needs to be sampled and processed to obtain the required information. Analog-to-digital converters (ADCs) play a key role. The industry has been able to develop and produce converters whose characteristics cover a wide range of metrological and industrial requirements. The accurate characterization of the ADCs is fundamental to improve their accuracy and to extend their practical application. For dynamic measurements (time varying signals), a limiting parameter is the frequency amplitude variation of the ADC. Metrology research has developed quantum standards at the highest level and directly traceable to the future International System of Units (SI) redefinition [1][2][3], that provide the necessary reference for the accurate ADC characterization. In particular, the direct current volts (DCV) sampling function of the Keysight 3458-A digital multimeter (DMM) [4] is widely used for high-accuracy sampling measurements in National Metrology Institutes (NMIs) and calibration laboratories. Several works regarding properties of this DMM function have been published. Some of its dynamic characteristics (RMS measurements, gain variation with aperture time, hysteresis, and integral non-linearity) have been evaluated by means of a programmable Josephson voltage standard (PJVS) from 11 Hz up to about 400 Hz [5]. A PJVS is also used to evaluate the influence of the DMM synchronization and measurement parameters on the root-mean-square (RMS) measurements for different frequencies [6]. Thermal converters have been used for the characterization of the frequency range from 20 Hz to 400 Hz [7]. The noise performance of the 1 V range DMM at various sampling frequencies and aperture time settings has been obtained applying signals generated by a JAWS [8]. Evaluation of Keysight 3458-A time jitter performance has been reported in [9]. The effect of Keysight 3458-A jitter in precision phase difference measurement and a method for its improvement have been described in [10]. A complex waveform generated by JAWS to calibrate frequency and amplitude of a digitizer simultaneously was described in [36]. 100 ns sample clock time jumps for Keysight 34580-A DMM with an internal sampling clocking was detected and measured in [37]. The frequency response of a Keysight 3458-A is measured by comparing it with a calibrated AC Voltage Standard F-5790. From these results, model parameters are identified for three different frequency response modelling [38].
Despite the number of research articles in this field, due to the available standards limitation, the frequency response characterization was limited to the order of 1 kHz. The frequency response of an ADC is a key issue for arbitrary waveform signal sampling. This paper presents the characterization of the frequency response of a Keysight 3458-A digital multimeter using a Josephson arbitrary waveform synthesizer (JAWS). This new standard overcomes the previous limitation and the characterization can be done at frequencies up to 20 kHz, only limited by the performance of ADC. This method can be applied to any precise ADC. The JAWS, developed by Physikalisch-Technische Bundesanstalt (PTB), is able to generate 1 V RMS pure spectral signals up to 1 MHz. The characterization has been done in the frequency domain for different aperture times, carrying out asynchronous measurements. The aperture time is the time when a multimeter is actually sampling the input signal and for DCV digitizing, the aperture time is equal to the ADCs integration time, that for the Keysight 3458-A can be varied from 500 ns to 1 s. The study includes the uncertainty evaluation budget. This work demonstrates the possibility of a significant improvement of the ADCs accuracy for dynamic measurements and in the case of Keysight 3458-A applying the same frequency correction for all the aperture times below 100 µs and a different one for aperture times above 100 µs, from DC to 20 kHz.
In this paper, we summarize and compare the different methods available for this characterization (section 2). The new method, based on JAWS, is described together with its application to the characterization of the amplitude frequency response at different aperture times of the Keysight 3458-A DMM analog-to-digital converter (ADC) (section 3), the uncertainty estimation (section 4), and the results obtained (section 5).

Based on thermal converters
The AC reference is provided from the known DC value by power dissipated equivalence on a thermal converter. In this method, the ADC is connected in parallel to the thermal converter and both are alternatively connected to an AC and DC source by means of and AC-DC switch. Several cycles AC, DC+, DC−, AC are applied. From the DC value and the known AC-DC difference of the thermal converter, the RMS AC value of the source is obtained. From the ADC samples obtained during the AC source application, the ADC RMS value of the same source is obtained. The comparison of these two values provides the ADC error [7]. The same process can be done for a set of frequencies and aperture times. This method has the following limitations.
• Each AC accuracy thermal converter measurement takes very long so it would take several days to perform all the measurement points proposed in this work and it will not be possible to distinguish the aperture time changes, due to the frequency behavior of the ADC, from its stability. • Thermal converter response to the RMS of the input signal and harmonic content of the AC source can introduce errors in the main frequency component measurement. • To extend the study to the higher aperture times at 20 kHz it is not possible to meet the Nyquist requirements. To avoid errors on sampling results a very low distortion AC source is necessary.

Based on PJVS
The AC voltage source is obtained using binary-divided arrays of damped Josephson junctions. Voltage waveforms are synthesised by periodic switching of the binary-divided arrays. It is equivalent to a digital-to-analog converter where each step of the wave has the accuracy directly traceable to a quantum standard.
The main limitation of the PJVS is the transition time from one step to the other. During this time there is a non-calculable oscillation on the signal. In addition the rise time of the transitions are usually below 10 ns, such a fast rise time typically is above the bandwidth limit of the ADC affecting the samples taken during this time. To overcome this limitation solutions as 'differential measurements' [11][12][13][14] or a Josephson-locked sythesizer (JoLoS) [15] can be used for ADC characterization.
In differential measurements a source is calibrated using an ADC to measure the difference between the source and the PJVS. The samples are taken during the quantized part of the PJVS. From the samples of the difference, the AC source can be corrected with direct traceability to the quantum standard. The uncertainty of the AC source correction corresponds to the uncertainty of the measured samples so its influence is divided by the source-sample amplitude relation.The calibrated source is later used for the ADC under test characterization. The main limitation of the differential method is due to the response of the ADC used to measure the difference, a delay time after the step change and sampling is necessary. This delay depends on the magnitude of the voltage change. This limits the number of samples for period of the signal. Besides the PJVS needs a time to set each quantum value. As the frequency increases the number of samples for period decreases. This means an increase on the magnitude of the difference limiting the maximum applicable frequency, to characterize the ADC under test, to a few kHz.
JoLoS is based on a low harmonic content synthesizer controlled by a PJVS that can be used as quantum precision low distortion source for ADC characterization. The main limitation of this system, as the differential, is that its operating frequency is limited to the order of 1 kHz.

Based on input impedance measurements and circuit model
In this method an input impedance model of the ADC is assumed. The parameters of this model are obtained by means of complex impedance measurements. The method can be used to frequencies up to 100 kHz. The main limitation of this method is its high uncertainty that is estimated to be in the order of 120 µV V −1 at 10 kHz [16].

Based on JAWS
The operating principle is based on the use of Josephson junctions as pulse quantizer. The AC voltage source is obtained applying high-speed pulses to Josephson junctions, the output voltage is quantized and proportional to the pulse repetition rate and the number of junction [17][18][19]. Recently such systems have reached the breakthrough of achieving 1 V-2 V RMS, however due to the required compensation method (AC coupling) at 1 MHz only smaller voltages have been published [2,20]. The increase to the 1 V level opened the possibility of its use in many metrological applications: AC voltage in a wide range of frequencies available and traceable to a quantum standard reference, quantum precision digital impedance bridges.
The JAWS is able to generate pure sine waves at desired frequencies. The only limitation of this cryogenic standard is the need of using relatively long cables from the output of the quantized voltage inside a cryostat to the instrument under test. These leads produce a frequency dependence voltage error. This error scales with the square of the frequency. Several methods have been described for this error evaluation and compensation [21][22][23][24][25]. These comments show that JAWS is the unique source capable to provide low uncertainties up to the frequency of 20 kHz. For this particular work, this standard is employed to determine the ADC frequency response of two DMMs.

Method
To characterize the frequency response at different aperture times, for each frequency the AC JAWS signal is applied to the digital converter and the aperture time (T a ) is varied. For each T a , the signal is sampled and processed to obtain the measured amplitude. From the direct comparison of the applied and processed signals at several frequencies, the error is obtained for a set of frequencies. A curve fitting of these points provides the frequency response.
In this research, the characterization of the 1 V range of the DCV sampling function of two Keysight 3458-A digital multimeters is performed applying a 0.8 V RMS pure spectral signal from 10 Hz to 20 kHz. The aperture times vary from 4 ms to 140 ms.

PTB Josephson arbitrary waveform synthesizer
The PTB 1V JAWS system is based on an 8-channel commercial pulse pattern generator (PPG) from Sympuls that delivers ternary pulses (−1/0/+1) to eight JAWS arrays in series arranged on four chips. The AC coupling method proposed by Benz et al [27] was employed with four dual arbitrary waveform generators for the compensation signals, to suppress higher harmonics. In total, there are 63 000 SNS (superconductor, normal conductor, superconductor) Josephson junctions with Nb0.8Si0.2 as a normal-conducting barrier (N) operated at a PPG clock-frequency of 15 GHz return-to-zero (RTZ)-pulses and an operating temperature of 4.2 K (superconductor: Nb). The non-hysteretic current-voltage characteristic of the SNS junctions allows the positive respectively to negative Shapiro-steps to be addressed by respectively '+1' and '−1' PPG pulses. For each array, there are four bias parameters to adjust (pos. and neg. pulse amplitude, compensation amplitude, and phase), i.e. in total 32 bias parameters. All parameters are time stable and the complete system is easily controlled by a LabView software. The analog waveforms are encoded into digital patterns by a higher-order Σ∆-modulation. The characterization of the output waveform in frequency-and time-domain is obtained by sampling using a battery-driven PXI 5922 spectrum analyzer. For each signal frequency, the Σ∆-code was calculated and transferred to the code-memory of the PPG [26].
The quantized output waveform of the PTB JAWS system is then applied to the DMM. Different computers are used to run the JAWS system and the sampling system.

Sampling system
The sampling system consists of the DMM under characterization operating on its 1 V range of the DCV mode [4]. The number of samples (n), measured periods (M), sampling frequency ( f s ), aperture time (T a ), number of measurements etc can be selected. The time reference for sampling is provided by the Keysight 3458-A internal clock. The measurement conditions depend on the signal frequency to be measured. The main parameters are: sampling frequency, aperture time, dead time, (aperture time and dead time must be compatible with the sampling frequency), and the number of signal periods to be sampled. The measurement settings are configured via computer communications. The ADC is configured so that its internal clock establishes the sampling instants. The samples are stored in the DMM internal memory.
The output signal is obtained from the samples taken using two different methods, a sine-wave fitting algorithm [28,31] and artificial neural network [29], in order to check the influence of the selected method on the results. Table 1 shows the parameters configuration to obtain the frequency response at different aperture times. To avoid its possible influence, all the measurement have been performed with the same number of samples (n = 5120). The number of samples n was fixed to the maximum value of the DMM internal memory. The number of periods M were selected to avoid common factors between n and M, so that all samples obtained can be set at equivalent points of a unique fundamental period, and also that the sampling interval (t s = M/(n · f a )) allows for longer apertures.

Uncertainty evaluation
The uncertainty evaluation has been performed according to the supplement 1 to the guide to the expression of uncertainty in measurement [30]. The present section describes the uncertainty evaluation corresponding to the ADC error on the amplitude measurement of a pure sine signal. The error and uncertainty are obtained at several frequencies. The ADC error and uncertainty for any other frequency can be obtained from the least-square fit of the error and uncertainty determination at several frequencies within the required frequency range of interest.
The output quantity is the gain error of the ADC at a defined frequency ( f a ), aperture time (T a ) and temperature (T) :  E( f a , T a , T).
The input quantities are the following. The temperature: T The model relating the output quantity and input quantities is as follows: (1) where E( f a , T a , T): ADC gain error at a given frequency, aperture time and temperature. A( f a ): For a specific T a and T, amplitude value obtained from the n samples, V n , at times t n by means of a least squares sine-wave fitting or an artificial neural network.
The function f 1 denotes the least squares sine fitting algorithm procedure or neural approach. V i are the mean DC values obtained by the ADC at time t i during the integration time T a at temper ature T.
T a stands for aperture time.
V I ( f a ) is the reference applied signal from the JAWS. C a ( f a ): The correction term arrives from the fact that the ADC measures the mean value during the integration time and this value is assigned to the mean time of the aperture time t i . For a sine signal at frequency f , there is a difference between the mean value and the signal value at the time t i [31]. C alg : Correction due to the performance of the applied data processing method other than C a (f ). C T : Correction due to temperature influence. C R : Variations in repeated observations under apparently identical conditions. The evaluation of the sources of uncertainties is the following: V I ( f a ): It comes from the JAWS reference signal. The error and uncertainty of this component are only due to voltage lead error. Several works regarding this error estimation were published [25]. The relative uncertainty introduced for the PTB system has been estimated to be 2 mV V −1 at 20 kHz [24]. At other frequencies this uncertainty, u f , scales with frequency squared (u f = 2 µV V −1 (f /20 kHz) 2 ).  20 18 A( f a ) depends on V i and t i uncertainties.
In relation with V i values: • Linearity. The error due to the linearity of the ADC is intrinsic to its frequency response so the error and its uncertainty are not considered. • DC calibration. The error due to the DC calibration does not affect the frequency response. • Gain variation with T a . The T a value is measured internally for the ADC, the systematic error in T a is intrinsic to the ADC and is part of the frequency response. • Quantization. There is an uncertainty contribution due to quantization.
In relation with t i : The time when the samples are taken, variation of t i during the measurement, due to random time jitter, contributes to the uncertainty.
The time jitter performance has been evaluated for the DCV function of a Keysight 3458-A. Taken into account that according to [9] it is far from normal and due to the negligible jitter uncertainty component in relation with the total uncertainty, assuming rectangular distribution is conservative enough.
According to the above, the two uncertainty components for this function are time jitter and quantization. It has been verified [32] that the distribution due to the contribution of time jitter and quantization can be combined as a quadratic sum of the two terms obtained separately. The two components are then evaluated independently.
Regarding jitter, based on the work reported in [9], a 150 ps rectangular distribution has been considered. To obtain the uncertainty contribution for each configuration trials of 10 000 iterations of the Monte Carlo method (MCM) have been performed to obtain the numerical representation of their PDF output and from that the standard uncertainty. An adaptive MC method was used to ensure that an increased number of trials does not improve the obtained relative standard uncertainty. The simulations have been done, for input sinusoids, at 53 Hz, 1 kHz, 5 kHz and 20 kHz, with the same parameters defined in table 1. Figure 1 shows the relative standard uncertainty obtained at the different frequencies. From figure 1, it can be seen that the variation of standard uncertainty due to jitter with the frequency is linear. It is not surprising as the variance scales as the product of the jitter variance and the slope squared. Since the slope is proportional to the frequency, taking the square root gives a linear dependence. From this relation, the jitter relative uncertainty contribution at the different frequencies under test can be obtained.
The influence of quantization has been evaluated and the results show that the relative standard uncertainty can be represented by the Wagdy equation [33] for the DFT as follows: where D is the dynamic range, n is the number of quantization bits and N the number of samples. For the Keysight 3458-A the number of bits depends on aperture time, it is 16 bits for 4 µs and 6 µs and 18 bits for the values up to 120 µs.
C a ( f a ): This correction is due to the integrating ADC conversion and is not intrinsic to the ADC.
The correction term is the following: There is an uncertainty on the ADC aperture time value (T a ), the influence of this uncertainty becomes more significant as the product ( f a · T a ) approaches unity. The frequencies of the source (JAWS) f a , is locked to a 10 MHz reference with a negligible uncertainty. It is assumed that the error on T a comes from the error on the Keysight 3458-A internal time reference. This error can be obtained from the frequency measurement function of the multimeter applying a 10 MHz reference. Considering that after a DMM frequency adjustment, a correction constant is applied to the frequency measurements function of the DMM but not to the sampling timing. This correction constant needs to be read from the internal memory and be compensated from the DMM frequency reading to obtain the internal time reference error [34]. This error can be also obtained from the difference in the frequency generated by the JAWS and the measurement frequency for any of the algorithms, four pole sine fitting or neural network. The K correction is applied using the corrected T a in the equation (4). The relative uncertainty of the frequency standard is negligible. If the correction is obtained from the DMM frequency function the unique uncertainty component comes from the resolution. If the correction is obtained from the difference in the frequency generated by the JAWS and the measured frequency for any of the algorithms, the uncertainty component is due to the algorithm error. The uncertainty is estimated to be 1 µHz Hz −1 . A rectangular distribution for T a has been simulated with 100 000 trials for different values of the product f a T a in the K-function. An adaptive MC method was used to ensure that an increased number of trials does not improve the obtained relative standard uncertainty. The output for all the values is a rectangular distribution. Figure 2 shows the relative standard uncertainty for the different T a /T ac values. T ac stands for the T a value where ( f a T a ) = 1. In the worst case (20 kHz and 18 µs) the relative standard uncertainty is in the order of 0.1 µV V −1 .
The influence of the temperature in the amplitude, referred in other papers [35], especially for aperture times below 100 µs, needs further investigation. Without this knowledge in this paper it has been assumed that temperature and other input quantities are no correlated.
C alg : Data from several simulated ideal signals are processed by the three methods, sine-wave fitting, ANN and discrete Fourier transform (DFT). 100 ns jitter and 16 bits quantization noise was added to an ideal sine signal. The results shows that differences within the three methods and the ideal signal are negligible [32]. C T : To characterize the relation between the gain and the temper ature, a climatic chamber was used, the temper ature was varied from 20 °C to 26 °C in steps of 1 °C. Before performing the measurement, each temperature was maintained for sufficient time to stabilize the internal temperature of the DMM [35]. For each temperature a 1 V Zener reference was applied to the DMMs. The aperture time was varied from 4 µs to 140 µs. The results show that for aperture time above 100 µs the temperature coefficient is below 0.5 µV (V °C) −1 . The temperature influence for aperture times below 100 µs is relatively high, and similar for all the aperture times. For the DMMs under evaluation the coefficient were, −4 µV (V °C) −1 and −6 µV (V °C) −1 . The temperature at the PTB laboratory, where the JAWS is operated, is not controlled. To correct this influence, the internal DMMs temperature indications was used. During the climatic chamber measurements the external temperature was measured using a calibrated PT-100, after waiting for the inertial stabilization the PT-100 values and the DMMs indications were in agreement. Due to the thermal inertia of the DMMs, the internal indication cannot follow the effective temperature of the multimeter. The difference between the effective temperature and the indication is estimated to be not higher than ±1 °C. C R : For each frequency and aperture time the measurement were repeated five times for all the frequencies, except at 10 Hz and 20 kHz where the dispersion is higher and ten repetitions have been done. The higher dispersion at 10 Hz can be explained for the difficulty to generate JAWS low distortion signal at low frequencies due to bit-multiply function of the PPG to virtually increase code memory. The higher dispersion at 20 kHz is due to the behavior of the ADC.
According to equation (1), the combined relative standard uncertainty can be obtained as the quadratic sum of the relative standard uncertainties related to the amplitude of the signal. The only possible mutual dependence of the uncertainty components comes from the temperature influence. The only information about this influence is in [35] and is related to the amplitude variation with temperature that is relevant for aperture times below 100 µs. The understanding of this influence needs further investigation and it is not possible to evaluate the temperature influence on each uncertainty component. In this work a general component has been considered to cover the difference between the DMM indication and the actual temperature due to thermal inertia. A very conservative ±1 °C has been considered as a whole. The combined uncertainty increase due to any possible correlation will be well within this value.
Tables 2-6, summarize the input quantities relative uncertainties at different frequencies and aperture times. Table 7 summarizes the relative combined standard uncertainty for the different combinations of frequency and aperture times.
There is not any previous work that covers the range, frequencies, and aperture times presented in this paper. However, Figure 2. Contribution of the DMM T a error to the relative standard uncertainty at different T a /T ac .  there are some estimations in some ranges, for instance in [16], up to 5 kHz in the 1 V range the standard relative uncertainty is 23 µV V −1 . The frequency response or bandwidth is the most limiting factor for the use of ADC. The results and uncertainties obtained in this paper would contribute to the improvement of precise measurement of arbitrary waveforms by means of an ADC.

Results and discussion
There is no difference between the results obtained from the sine-wave fitting algorithm and those obtained from the artificial neural network. Therefore, in the following evaluation the method is not distinguished. It is specified by the manufacturer of the Keysight 3458-A and have been reported in several papers [6,7,16], that there is a change in the DMM internal configuration for aperture times above and below 100 µs that modifies its input impedance, resulting in two distinctly different regions in the frequency response. For this reason the study of the results is divided according to each configuration.
In order to graphically compare, with high resolution, the variation of the frequency response at different aperture times, for each frequency, the mean of the values obtained at all the aperture times is taken as reference value, and for each aperture time its difference from the reference value is represented at all the frequencies.
As it is shown in figures 3(a) and (b), the frequency response is very similar for all the aperture times below 100 µs up to 1 kHz for the two DMMs. As the frequency increases, the frequency response starts to vary with the aperture time. This variation is noticeable at 2 kHz in figure 3(b) and is very clear in figures 4(a) and (b). This variation can be due to the correction terms C a ( f a ) explained in section 4. It was considered that the error in the effective aperture time is mainly due to the multimeter internal clock error, but there are other systematic errors as in the opening and shutting of the integrator, rise and fall times, etc. From the results and using equation (4), the delay to compensate this difference has been calculated to be of 7 ns and 9 ns, respectively.   Frequency response up to 2 kHz, as relative difference from the mean value, for DMM1 (a) and DMM2 (b) for different T a values below 100 µs. As can be seen from the figures for the DMM1 there is a noticeable gain decrease as the T a increases, for the DMM2 the gain slightly increase as the T a increase.  Table 6. Relative standard uncertainty due to non-reproducibility. be concluded that constant delays to be added to the timing clock errors are very likely to explain the difference found at frequencies above 1 kHz. Figures 6(a) and (b) show AC gain variation for aperture times above 100 µs. As expected the frequency response is similar for all the aperture times, the discrepancy at 140 µs at 1 kHz can be again explained by the systematic delays in aperture duration.

Conclusions and future work
The breakthrough of a JAWS system achieving 1 V RMS in a frequency range up to 1 MHz opened the opportunity for the first quantum precision characterization in the frequency domain of frequency gain variation of an ADC. This new characterization considerably improves the knowledge about the frequency response of the Keysight 3458-A. In previous work, it was concluded that same gainaperture variation can be applied for apertures times below and above 100 µs. However, the frequency range was limited to frequencies up to 400 Hz and for aperture times above 50 µs. This work extends this conclusion up to 20 kHz and for aperture times as low as 6 µs. The precision provided by the new standard has showed the need to take into consideration two influence parameters on the Keysight 3458-A. For low aperture times, below 100 µs, due to the unexpected relatively high temperature coefficient, it is necessary to perform the measurements in a temperature-controlled environment and to report the temperature with the results. For higher frequencies, above 1 kHz or 2 kHz, it is necessary to precisely Frequency response from 1 kHz to 20 kHz, as relative difference from the mean value, for DMM1 (a) and DMM2 (b) for different T a values below 100 µs after applying correction for time delays on T a . As can be seen from the figures for both multimeters there is no frequency response variation with T a .  Frequency response from 1 kHz to 20 kHz, as relative difference from the mean value, for DMM1 (a) and DMM2 (b) for different T a values below 100 µs. As can be seen from the figures for both multimeters there is a variation in the frequency response with T a . know the effective aperture times that can be different from nominal. Finally, the results show that the JAWS can considerably improve the ADCs characterization. This would allow the progressive replacement of thermal converters by ADCs providing the required metrological information. In the future, similar work will be performed on other precision digital converters extending the frequency range to 1 MHz. In relation with the Keysight 3458-A, the measurement will be repeated in a temperature controlled environment. Also a method to precisely estimate the effective aperture time needs to be developed.