Evaluation of 400 m, 4.8 Gbit/s Versatile Link lengths over OM3 and OM4 fibres for the LHCb upgrade

The LHCb experiment will upgrade its DAQ system to a trigger-less, 40 mHz readout in 2018. To be able to process the approximately 40 Tbit/s of data we will require a massive computing farm. This computing farm can no longer be installed underground in the vicinity of the detector anymore due to the increase in power and cooling requirements. An affordable, optical data transport solution has to be found to carry the data from the detector to the new data center on the surface. The distance to cover is 300 m with an additional 100 m of safety margin. This document covers the results of our measurements of the 4.8 Gbit/s Versatile Link signal over 400 m of OM3 and OM4 fibres.


Motivation and description
The LHCb collaboration has decided to upgrade the LHCb detector during the 2018 shutdown of the LHC for a trigger-less readout [1]. Data will be acquired at the LHC bunch crossing frequency of 40 mHz. This means that data is sent directly to the readout network, without any reduction in volume. Current estimates put the traffic volume of this readout mode to about 40 Tbit/s. Similar to the current High Level Trigger farm, a dedicated computing farm will process and filter the data on an event by event basis. This filter process is expected to reduce the data rate to about 10 Gbit/s. The data will go over two different kinds of link level technologies before it reaches the compute nodes. The first leg will utilise the Versatile Link [2], developed at CERN for radiation tolerant data transport. The link is based on a 4.8 Gbit/s optical signal running on current 850 nm, Multimode technology. Dedicated converter boards / readout boards called TELL40 [3] will be used to receive the Versatile Link data and convert it to a commercial link technology. It has yet to be decided what this link will be. Current candidates are Ethernet, Infiniband or PCIe. Since the LHCb experiment is approximately 100 m underground, it is rather intricate to support a large computing farm in close proximity to the detector. In the current incarnation this is still possible. In the future the throughput will be scaled up by almost a factor of 100 from currently 500 Gbit/s to 40 Tbit/s. Even taking Moore's law into consideration this is still a significant increase in the number of compute nodes and consequently power and cooling requirements. Since it would be too costly to augment the infrastructure for this, it was decided to install the new computing farm on the surface. As a result, the data will have to be transported over approximately 300 m of distance to reach the new computing data center. This distance has to be bridged somehow and there are several different flavours which are all derived from transporting the data via current 850 nm multimode optical cables from underground to the surface.
Since the exact distance was unknown to us at the time we performed the following lab tests, we assumed a total length of 400 m for our measurements. An overview of the proposed fibre system is illustrated in figure 1. The system contains a minimum of 3 break points. Data is transferred via a 4.8 Gbit/s signal produced on the detector by Versatile Link transmitters (VTTx). Control signals are sent from the data center to the detector by the same carrier system and received on the -1 -   detector with Versatile Link receivers (VTRx). The complete system will contain approximately 17.000 individual fibres. The VTTx/VTRx components are compatible with the LC form factor and connect to LC terminated fibres. The readout and control boards in the data center are based on MPO 12 connectors. The conversion between the two form factors is foreseen on the last leg of the fibre installation on the detector itself. The whole system is very similar to the current LHCb fibre infrastructure but longer.

Lab setup
The goal of the setup is to determine two unknown parameters related to 4.8 Gbit/s operation. The first is the dispersion behaviour of OM3 and OM4 fibres at 4.8 Gbit/s. We assume that the behaviour is much better than the one specified for 10 Gbit/s but we would like to know how much. In particular we want to know the difference between OM3 and OM4 fibres. An OM3 installation would be about 30% cheaper than OM4.
The second unknown parameter is the receiver sensitivity behaviour at 4.8 Gbit/s. Since the receiver we tested here is designed for 10 Gbit/s operation, we suspect that it will perform better at 4.8 Gbit/s which should give us additional optical power margin. The reasoning here is that the electrical propagation within the optical receiver and the signal paths on the PCB will profit from lower than designed signaling rates.
A secondary goal of this measurement campaign was to determine the quality spread of fibres obtained by different manufacturers. For this purpose several 400 m OM3 and OM4 samples where obtained from the major fibre producers. Figures 2 and 3 show schematics of the lab setups that were used for the measurements. Setup A is used for MPO transmitters and LC receivers. Setup B is for testing the reverse direction with LC transmitters and MPO receivers. The measurement principle is the same in both cases. A pattern generator is generating a random pattern (PBRS7) and transmits it to a carrier board (MCB) for optical transceivers. The random pattern is converted to an optical signal by the transmitter and After the test fibre the signal is brought into a programmable, variable optical attenuator (VOA) and split into an attenuated signal that goes to the receiver and a proportionally attenuated signal that goes into an optical power meter (OPM). For MPO receivers the signal is then sent through another fan-out to change back to a multi-fibre ribbon.
Setup A simulates the monitoring and control link of the detector. The transmitter is an Avago MiniPOD [5] multi-lane transmitter and the receiver is a VTRx Versatile Link receiver as it would go onto the detector. The transmitter in Setup B is a Versatile Link VTTx transmitter, the receiver is a an Avago MiniPOD multi-lane receiver which is mounted on a prototype readout board, called AMC40. The AMC40 board is the current baseline readout board for the proposed LHCb upgrade. This setup is equivalent to the planned Data Acquisition (DAQ) link. For the 10 Gbit/s measurements, the VTTx transmitter in Setup B was exchanged for a MiniPOD transmitter on a test board for multi-lane optics. Since the MiniPOD has already an MPO connector, the fan-out cassette was not necessary on the transmitter side of the 400 m cable. To keep the number of breakpoints consistent, the signal was sent through two additional short cables instead.
-3 - To determine dispersion behaviour and receiver sensitivity we measure BER vs. OMA. OMA is estimated by measuring the extinction ratio of the transmitter first, and then measuring the average optical power with the OPM. Both values are then used to calculate the approximate OMA according to ref. [4]. This measurement should give a fair estimate of the effects that dispersion has on the optical signal in the fibre under testing. For strong dispersion, we are expecting a worse bit error rate (BER) at the same OMA than for a fibre with less dispersion. In the following diagrams this means that the typical BER curve is shifted to the left for less dispersion.

Optical dispersion measurement
The optical dispersion is determined by measuring the bit error rate curves of a 400 m fibre and a short, 30 cm reference fibre. The bit error rate is plotted vs. the received OMA. Since the 30 cm are too short to introduce significant distortion, any measured increase of BER at the same receive OMA can be attributed to dispersion. Figure 4 shows a sample measurement for 400 m of OM4 fibre done with Setup A. We tested two kinds of data generators in this plot. The blue markers were done with a dedicated pattern generator on an FMC carrier board, while the green points were obtained by using a prototype, AMC40 readout board as data generator. 1 In both cases we used an Avago MiniPOD [5] as the optical transmitter.
It becomes quite clear, that the distortion introduced by the 400 m fibre has almost no effect on the signal. There is at most a 0.2 dB shift visible between the short (open circles) and long (filled circles) cables. A much greater shift seems to come from the combination of driver components and transmitter variances. We have noticed that different MiniPODs come with different factory defaults for de-emphasis and equalisation settings which can lead to such variances. The difference in this case is approximately 0.6 dB but even with these variances the VTRx receiver sensitivity target of −13.1 dBm at 10 −13 is reached with ample margin.   The situation in the DAQ direction can be derived from figure 5a, where we also measured different qualities of fibres from different manufacturers. Here we are using a VTTx transmitter and MiniPOD receiver. Since the dispersion caused by the fibre is of the same magnitude as the variation between individual 400 m fibres the reference has been omitted from this plot. Again, the receiver sensitivity of −11.1 dBm at 10 −13 is reached with ample margin.

OM3 vs. OM4
Another important aspect is the question of OM3 vs. OM4 fibres. OM4 was developed specifically for current high speed signaling frequencies of 10 Gbit/s and upward. OM3 on the other hand is much more economical and made for lower transmission frequencies. We tested several 400 m OM3 and OM4 fibres of different manufacturers to find out if there are significant differences at 4.8 Gbit/s and if it is worth investing in OM4. Figure 5a shows the results of these measurements. The blue points are a sample of OM3 fibres while the other coloured points are OM4 fibre samples from major fibre manufacturers. There is no significant difference visible between the two types of fibres and also between the manufacturers. The eye diagrams in figure 5b show that the rise time in the OM3 case is slightly worse. The lower signaling rate however compensates for the slower rise time and the eye is still opened sufficiently enough, leaving the error rate unaffected.
To convince ourselves that we were really measuring OM3 and OM4 fibres we repeated the same BER test with a 10 Gbit/s transmitter and receiver. Since the Versatile Link transmitter is not capable of 10 Gbit/s operation, we used an Avago MiniPOD transmitter as optical source. The result can be seen in figure 6. The figure shows the best, average and worst OM4 channels of a 12 Fibre cable in green and all OM3 channels in blue. While the signal on OM4 is stable between channels, the OM3 signal is very erratic. In fact two channels have been omitted here, because the signal was so bad that no transmission was possible at all.
From the point of view of OM3 vs. OM4 there seems to be no difference for our application. While the expected degradation due to dispersion is visible in the OM3 eye diagram, it has no effect on the error rate at 4.8 Gbit/s.  Figure 6. BER of a 10 Gbit/s signal over OM3 and OM4 fibres. Another important result of this test is the sensitivity of the MiniPOD receiver at 4.8 Gbit/s. The 400 m fibre introduces only marginal dispersion and so the receiver sensitivity can be directly obtained from figure 5a. Extending the curve in 5a to 10 −13 yields a sensitivity of approximately −15 dBm at 4.8 Gbit/s instead of the −11.1 dBm specified by the manufacturer for 10 Gbit/s operation.

Optical power budget calculation
To determine the feasibility of a 400 m readout we calculated the optical link budget available in the system with the help of the parameters obtained from the previous measurements. For the items in the budget we did not measure we use the specs of the component in question. Table 1 summarises the result of the available budget. The VTTx to MP, spec. column contains the values specified by the standards and manufacturers for 10 Gbit/s operation. The VTTx to MP, meas. column contains the values we measured. For the values we did not measure (NM) we use the values from the spec column.
The biggest unknown in the proposed optical system was the distortion behaviour of the optical fibres at 4.8 Gbit/s. There are no commercial products running at this signaling rate and consequently the manufacturers do not give any specs for these rates. After our measurements we think that a value of 0.5 dB for OM3 and OM4, twice the measured value, is a reasonable assumption for the final system.
Another unknown was the receiver sensitivity at 4.8 Gbit/s. In case of 4.8 Gbit/s, see figure 5a, the receive OMA at 10 −13 is −15 dBm. This value by itself is already enough to ensure the successful operation of the proposed system with sufficient margin. It would however also be interesting to know quantitatively how much improvement we can expect by lowering the signaling rate.
A worst case simulation with the 10Gbit/s model in [6] for 400 m over OM4 yields a dispersion penalty of 2.4 dB. In figure 6 we see a value of approximately −9.5 dBm at 10 −13 . Subtracting the simulated value of 2.4 dB gives an estimated receiver sensitivity of −11.9 dBm. This is in good agreement with the −11.1 dBm the manufacturer quotes. 2 This also means, that the −15 dBm we measured in the 4.8 Gbit/s case is really an improvement due to the lower signaling rate and not because the receiver by itself is much better than specified. By using the lower signaling rate, we gain at least another 3.1 dB of optical power margin over the 10 Gbit/s performance of the receiver. To be conservative we use these 3.1 dB on top of the specified value of −11.1 dBm since this is the worst case the manufacturer guarantees.
For other, smaller penalties due to radiation damage, we rely on the specs given by the Versatile Link [7,8] for the optical receiver and experience with similar fibre installations in other HEP experiments.
In this paper we have mostly focused on the DAQ direction of the data flow. After measuring the fibre losses for the controls direction (MP to VRx) it became clear, that this direction is safe for 400 m operation. On top of the ample margin of 4.4 dB, this direction will also be protected by strong Forward Error Correction (FEC).
To summarize, the operation of the proposed, upgraded LHCb readout and control is possible over a 400 m link of either OM3 or OM4 with sufficient margin. Industry standard is at 3.0 dB of margin, we achieve at least 4.4 dB plus FEC on the controls path and 6.0 dB on the DAQ path.

Conclusions
We have shown that a long distance readout based on a 4.8 Gbit/s Versatile Link is feasible over a distance of at least 400 m. Especially the following points have been settled: • There is no significant difference between OM3 and OM4 at 4.8 Gbit/s. Degradation of a 4.8 Gbit/s signal seems mostly dominated by attenuation and not by optical dispersion. Signal degradation due to dispersion is approximately equal to a single additional breakpoint. As expected there were significant differences for 10 Gbit/s.
• There is no significant difference between the three largest fibre manufacturers. All measured channels of all fibres are identical within the error margins of the setup.
• The readout of the LHCb detector over 300 m seems to be feasible. In the light of planning for an installation of 17.000 fibres, more statistics on receiver sensitivity and fibre quality spreads should be acquired though.
In the future we will test more fibres and transmitter -receiver pairings. This is especially important since the receivers are multi-lane receivers and only very few lanes have been tested so far. The current study draws a very promising picture, but needs more statistics.