Performance characterization of network timing with remote traceability via GNSS time transfer

One credible method for performance evaluation of network timing service was proposed, based on one kind of disciplined time standard with GNSS time transfer, NIMDO. The performance of the NTP timing service were characterized over multiple baselines on both the Internet and the Intranet, and the network delay and the timing offset were analyzed in detail. The results show that on the Internet the averaged timing offsets via the NTP servers are roughly hundreds of μs to several ms level, and on the Intranet the averaged timing offsets via the NTP servers are at about 20 μs level. NTP timing service was mainly affected by white phase noise, and flicker phase noise at the different sites. The maximum change of the timing offset was one-half of the maximum change of RTDelay. The uncertainties have been separately evaluated as less than 30 ms on the Internet and less than 60 μs on the Intranet for NTP timing service.


Introduction
Network timing service, especially Network Time Protocol (NTP) 1 timing service widely used around the world is a significant method for many national metrology institutes (NMIs) to disseminate the standard time and for many industrial and commercial users to trace the standard time, described as references. [2][3][4][5][6][7] In the actual NTP timing service, the client clock after NTP time synchronization is still deviated from the reference time of the NTP server because of the primary limiting factors, which are network asymmetry, server instability, and client instability. 8 To ensure the reliability of NTP timing service and provide the real performance characteristics for users' references, it is necessary to effectively evaluate its performance, such as timing offset or timing accuracy and uncertainty of NTP timing service. In 2008, in Europe, an NTP timing accuracy measurement and calibration device based on GPS disciplined oscillators (GPSDOs) was established, and the test experiments in collaboration with five European timekeeping laboratories were conducted. The network influence on the timing offset for NTP timing service was analyzed. 9 However, the round-trip delay involving network asymmetry was not combined for further analysis. Moreover, in terms of the definition in International Vocabulary in Metrology (VIM) for traceability, the metrology traceability must be acquired through constructing a documented unbroken chain of calibrations, each contributing to the measurement uncertainty. 10 For a standalone GPSDO disciplined by GPS, to achieve the metrology traceability, establishing the unbroken calibration link with UTC is required as described in Defraigne et al., 11 which is the task for the task group of Consultative Committee for Time and Frequency (CCTF) Working Group (WG) on Global Navigation Satellite System (GNSS). Thus, referenced to GPSDO, the traceability of the measurement and evaluation for NTP timing service cannot be declared. In 2014, in America, the National Institute of Standards and Technology (NIST), Colorado, USA designed a local system at NIST to measure the timing accuracy of NTP timing service via different NTP servers located in North, Central and South America. The system compared the timestamps received via NTP servers to UTC(NIST) located in Boulder, Colorado and monitored the working status of multiple NTP servers in the same place. 12 The uncertainty, including network asymmetry from round-trip delay on the Internet, was evaluated at milliseconds-level. However, since the measurement system was always at NIST and the NTP servers were at remote sites, the evaluation on the Intranet mainly reflecting the server instability was not implemented.
In order to have important time and frequency quantities traceable to the UTC(NIM) (National Metrology Primary Standard for Atomic Time Scale), then to UTC, and finally to SI (Syste`me International d'Unite´s), the transfer hierarchy of time and frequency quantity values is being constructed. One of the most significant time transfer links is NTP time transfer via the NTP servers, and more and more remote NTP servers have been equipped. The performance of NTP timing services of these servers has to be evaluated. In this paper, a traceable method for remote and local evaluation of network timing service was proposed using UTC(NIM) disciplined oscillator (NIMDO), and the influence on network asymmetry and server instability can be evaluated for NTP timing service. UTC(NIM) is National Metrology Primary Standard for Atomic Time Scale in China and is legally and directly traced to UTC via TAI cooperation, which is the only key comparison for time and frequency, named CCTF-K001.UTC. NIMDO gets traced to UTC(NIM) through GNSS time transfer with the time offset at several nanoseconds level. Thus, with NIMDO, the performance for NTP timing service can be evaluated and characterized with traceability, mainly including the timing offset and the time stability for the client after NTP time synchronization and the timing uncertainty. The NTP server located at NIM and the remote ones at the remote test sites were involved in the evaluation.

Principles for NTP timing service
The typical working mode for NTP is client/server mode. As shown in Figure 1, the client first sends an NTP request packet to the NTP server, which stamped with the time epoch T 1 when the packet leaves for the server. It is stamped with T 2 when the packet arrives at the server. When the reply packet leaves for the client, T 3 is stamped on the packet. When the client receives the packet, the packet is stamped with T 4 . The client calculates the round-trip delay of the NTP packet and the clock deviation of the client from the server using the above four time parameters (T 1 , T 2 , T 3 , T 4 ). With the information, the client adjusts the local clock so that its time is consistent with the server time as soon as possible. d 1 and d 2 are separately the transmission delays of the NTP request packet from the client to the server and the NTP reply packet from the server to the client; TD is the time difference between the client and the server, RT Delay is the round-trip delay of the NTP packets, and they can be calculated as (1)- (5).
Assuming that the transmission delays for the NTP request packet and the NTP reply packet are equal (d 1 = d 2 ), then Principles for performance characterization In terms of the principles of NTP timing service, when the round-trip delay is asymmetric (d 1 6 ¼ d 2 ) and T 1 2T 4 can be stamped in nanoseconds levels, the time difference TD between the client and the server is as follows.
Network asymmetry introduces NTP timing offset as follows.
It can be also concluded from formula (7) that the NTP timing offset standard deviation is equal to half of RT Delay standard deviation. The derivation process is as follows.
We could get the relationship between the maximum change Doffset in the NTP timing offset and the maximum change DRT Delay in RT Delay obtained from formula (3), (7), and (8) as follows.
We could get the absolute value for the maximum change of the measured offsets to evaluate the uncertainty of NTP timing service u NTPTT as (10) shows.

Methodology and experiments
Taking NTP timing service as an instance, we evaluated its performance both through the NTP server located (type Symmetricom SyncServer S250) at NIM on the Internet and through the ones at the remote test sites on the Intranet. To acquire the traceability to UTC(NIM) and then to UTC, we use NIMDO to link with UTC(NIM). It was used as the time reference to measure the timing offset of NTP testing equipment (type Titan TimeAcc 007) through NTP timing service. Moreover, one standard test suite including one NTP server (type Symmetricom SyncServer S350) and one self-developed pulse distribution amplifier at the remote sites was used in the evaluation for NTP timing service on the Intranet.

GNSS time transfer and NIMDO
The basic principle for GNSS time transfer is shown in Figure 3.
The GNSS time and frequency transfer receivers R 1 and R 2 are separately used in the reference station and test station, which are referenced to the time and frequency standards LTR 1 (the reference standard) and LTR 2 (the standard to be tested). The time difference DT 1 or DT 2 can be obtained by the two receivers, which is the difference between the local time and frequency standard for each station and GNSS system time (GNSST). The time and frequency transfer result is  calculated as (11)2(13). That is to say, the difference between LTR 1 and LTR 2 is acquired by the difference between DT 1 and DT 2 .
NIMDO is a significant extensional application of GNSS time transfer as shown in Figure 4. 13 If the GNSS time transfer data from the reference station can be acquired via a certain kind of nearly real-time communication method by the remote station, together with the GNSS time transfer data from the remote station itself, the time difference between the two stations could be calculated in real-time. Then, the clock difference and clock rate of the steerable oscillator could be predicted for the present at the remote station and the oscillator may be steered to the reference at the reference station, such as UTC(NIM). When the oscillator is operated continuously based on the above principles, it will be really a time scale disciplined by and traced to UTC(NIM). NIMDO has been verified and demonstrated on its performance referenced to UTC(NIM) and has been used at different sites and in several industries, 14 and it is much more accurate and precise than the GPS disciplined oscillator (GPSDO). During the test, the time difference between the NIMDO at each site and UTC(NIM) is better than 610 ns for around 95% of the time.

Experiment 1: NTP timing service on the Internet
The NTP server at the Changping campus of NIM in Beijing mostly transfers its time to the clients on the Internet, and the experiments schemed in Figure 5 were performed for the direct verification of time synchronization performance with an NTP test equipment (type Titan TimeAcc 007) at the remote sites.
The test equipment at the test site is used as the client, which was connected with the 1PPS and 10 MHz signals of the NIMDO as the time and frequency reference. It sends a request packet and obtains a reply packet from the NTP server via the Internet and then its time difference TD from the time of the NTP server with reference to UTC(NIM) can be calculated. Since its ideal time difference TD 0 from the time of the NTP server is the time deviation of the NIMDO from UTC(NIM) at nanoseconds level and negligible compared to the timing offset of NTP at microseconds level at best, TD could be regarded as the timing offset for NTP time synchronization (offset'TD).

Experiment 2: NTP timing service on the Intranet
To evaluate the performance of time transfer via the local NTP server, the evaluation experiment was designed as shown in Figure 6, and the standard test suite was employed except that during the test at IMGZ the NTP server (type THC-200AB) was used instead of the NTP server (type Symmetricom SyncServer S350). TimeAcc synchronizes the time via the NTP server on the Intranet.
As the NTP server and TimeAcc were both with reference to the NIMDO, the ideal time difference TD 0 between the client and the NTP server should be 0. TD   by TimeAcc is the timing offset of the NTP server at the test site.

Numerical results
For the two experiments, the timing offset and the round-trip delay via the network have been acquired and the statistics for them have been done. As well, the stability based on the timing offset has been analyzed.

Experiment 1: Evaluation of NTP timing service on the Internet
The experiments were performed at each site for a few days, and data were recorded every 10 s. To eliminate abnormal network data and accurately evaluate NTP timing service performance, the experimental data of each group are sorted in ascending order according to RT Delay , and only the first 95% of the data is used. It could be observed that the measurements are in 1-day cycle and the offset and the delay were larger during rush hours. Some big steps occurred, possibly due to the network path change deriving from network jams or network faults. The statistical results are shown in Table 1 (Mean: mean value, Std: standard deviation). Except the results at the Chijiu site, on the Internet, the timing offsets and the round-trip delays for NTP timing service at other sites are separately roughly at hundreds of microseconds to milliseconds level and tens of milliseconds level, and are basically proportional to the baselines from NIM, that is, farther away sites lead to larger delays and larger absolute timing offsets.
For experiment 1, Table 1 summarizes the statistics of the timing offset and the round-trip delay in Figures  7 and 8 (shifted for a better view). It can be seen that, in general, the longer the distance is from NIM, the greater the delay is in general. The distance between Chijiu and NIM is 547 km, which is the closest distance from NIM among all the test sites. However, the delay is the largest, inferring that there could be the worst network environment which had a greater influence on the delay than the distance. The results of the experiment 2 stressed the effect of the background network traffic, as experiment 2 can be regarded as the measurement without the background network traffic. In general, the larger the delay standard deviation is, the larger the standard deviation of the offset. As shown in Figure 8, the round-trip delay data points of each station have dense parts and sparse parts. In this paper, the dense part of the data points is called the basic area. In addition, the basic areas of the same site may jump due to changes in the network path during NTP timing. For example, as shown in Figure 9, there are seven basic areas (in the red boxes), and there are jumps between the basic areas in the LIM site test. It can be    seen from the results for 1 week that both the timing offset and the delay have similar large periodic changes. This phenomenon reflects that the network is more crowded during the day than at night, which causes the increase in delay and the change in offset. The periods when the largest offsets occurred correspond to the periods when the largest round-trip delay occurred during the rush hours for Internet surfing, which are roughly 7:00 to 23:00 (local time zone, Beijing time). The basic areas for the round-trip delay and the timing offset measured at HNIMT, XJMT, LIM, QIMT, IMGZ, SCM, and CMQ all have large but different jumps, anyway, it is evident that the jumps are due to the network rather than the NTP servers. From the above results, it can be found that network asymmetry is an important factor determining the NTP timing performance. Figure 10 reflects the numerical relationship of formula (9) in the corresponding time period for the same basic area. Figure 11 describes the time stability (Time Deviation, TDEV) and the frequency stability (Modified Allan Deviation, MDEV) of the NTP timing offsets at all the sites, and these two results at XJMT are both less than those at the other sites for all the averaging intervals. Except for at the Chijiu site, the time stability of the offset at all the other sites is less than 1 ms for all the averaging intervals; the frequency stability of the offset at all the sites is at the level of 1e 28 after the averaging time of approximately 1 day. Network asymmetry is the main factor that affects the NTP timing offset. By analyzing the slope of the frequency stability curves at the different sites in Figure 11, we found that NTP timing service on the Internet is mainly affected by white phase noise, flicker phase noise and white frequency noise as shown in Table 2, and the noise parameters are estimated with t in seconds.

Experiment 2: Evaluation of NTP timing service on the Intranet
The experiment was performed at each site for 1 week, and data were recorded every 10 s. Data selection and processing principles are the totally same as those in section 4.1. The experimental results are shown in Table 3 and Figures 12 and 13 (shifted for a better view). Table 3 summarizes the statistics of the timing offset and the round-trip delay. The standard deviation for offset does not increase as that of RT Delay becomes larger. However, it is larger than half of the standard deviation for RT Delay , which indicates that there is almost no network asymmetry in this configuration, and the timing offset can be attributed to other factors, such as the performance of the server instability and the client instability.
From Table 3 and the Figures 12 and 13, in the case of one-hop or two-hop on the Intranet, the offsets and the delays at different sites are basically at separately about 20 ms and 100 ms level. Figure 14 reflects the time stability and the frequency stability of the timing offsets for all the sites. The time stability is less than 1 ms after the averaging time of approximately 1 day; the frequency stability of the offset for all the sites is approximately 1e 211 after the averaging time of approximately 1 day. The NTP timing offset can be mainly attributed to the factors, such as the server instability and the client instability. By analyzing the slope of the frequency stability curves in Figure 14, we found that NTP timing service on the Intranet is also mainly affected by white phase noise and flicker phase noise at the different sites as shown in Table 4, which are estimated with t in seconds. The noise parameters of NTP timing service on the Intranet  are about 100 times less than those on the Internet, which are roughly consistent with what happens on the offsets and the delays.

Uncertainty evaluation
In terms of section 5.2, we could evaluate the uncertainty for NTP timing service in terms of formula (10), as Figure  15 shows. It can be concluded that in the case one-hop or two-hop on the Intranet, the timing offsets for NTP timing service at the different sites are basically at about 20 ms on average, and the uncertainty for NTP timing service could be evaluated as less than 50 ms. As well, on the Internet, the timing offsets for NTP timing service at different sites are basically at about milliseconds level on average, and the uncertainty is evaluated as less than 35 ms.

Summary
Via the device, such as NIMDO, the evaluation for network time synchronization could be legally traced to national time scale and then to UTC.
In our experiments via NIMDO, on the Internet, on average the timing offsets and the round-trip delays for NTP timing service at the different sites are separately roughly at hundreds of microseconds to milliseconds and tens of milliseconds level, and are basically proportional to the baselines from NIM, that is, farther away sites lead to larger delays and absolute offsets. The time stability of the offset for the different sites is almost less than 1 ms for all the averaging times and the frequency stability of the offset at different sites is less than 1e 28 after the averaging time of approximately 7 days. Network asymmetry is the main factor that determines the NTP timing offset.
On the Intranet, in the case of one-hop or two-hop, on average the timing offsets and the round-trip delays for NTP timing service at the different sites are basically at separately about 20 ms and 100 ms level. The time stability of the offsets for different sites is less than 1 ms after the averaging time of approximately 7 days and the frequency stability of the offsets at the different sites is approximately 1e 211 after the averaging time of approximately 7 days. The NTP timing offset can be attributed to the factors, such as server instability and client instability.
NTP timing service on the Internet and Intranet is mainly affected by white phase noise and flicker phase noise at the different sites. The noise parameters of NTP timing service on the Intranet are about 100 times less than those of NTP timing service on the Internet, which are roughly consistent with the phenomenon on the timing offsets and the round-trip delays.
The maximum change of the timing offset is one-half of the maximum change of RT Delay . On the Internet and Intranet, the uncertainties have been separately evaluated as less than 30 ms and less than 60 ms for NTP timing service.    Figure 13. Round-trip delay for experiment 2.

Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Key R&D Program of China with grant no. 2021YFB3900704.