The management of large cabling campaigns during the Long Shutdown 1 of LHC

The Large Hadron Collider at CERN entered into its first 18 month-long shutdown period in February 2013. During this period the entire CERN accelerator complex will undergo major consolidation and upgrade works, preparing the machines for LHC operation at nominal energy (7 TeV/beam). One of the most challenging activities concerns the cabling infrastructure (copper and optical fibre cables) serving the CERN data acquisition, networking and control systems. About 1000 kilometres of cables, distributed in different machine areas, will be installed, representing an investment of about 15 MCHF. This implies an extraordinary challenge in terms of project management, including resource and activity planning, work execution and quality control. The preparation phase of this project started well before its implementation, by defining technical solutions and setting financial plans for staff recruitment and material supply. Enhanced task coordination was further implemented by deploying selected competences to form a central support team.


Introduction
The Large Hadron Collider (LHC) has reached 77% of nominal luminosity at half energy, enabling fundamental discoveries. After three years of operation, the Long Shutdown 1 (LS1) will allow the complete maintenance of all equipment, as well as the necessary consolidation and upgrade activities in order to ensure a reliable and safe operation of the LHC at nominal performance from 2015 [1].
In this framework the consolidation and upgrade of the cabling infrastructure, copper cables and optical fibres, of the CERN accelerator complex is one of the most critical activities of the LS1. The activity volume will be about four times more than the normal yearly workload [2], significantly higher than what occurred in previous shutdowns. The activity requires substantial project management efforts and a massive deployment of resources, consisting of CERN staff and external contractors, to complete all the expected tasks within the planned dates.
The structure of the CERN cabling and optical fibre team was enhanced, including the configuration of the labour force by autonomous cells and introducing support functions for assisting the management in the coordination aspects. Five contracts were set up with external firms to carry out -1 -

JINST 9 C03042
the planned cabling projects. Supply of qualified material was organized as from six months to one year in advance.
All the cabling installations were grouped by geographical campaigns to facilitate the resource and material distribution and to optimize the intervention time in different areas. The campaign time slots have been fitted in the planning by increasing as much as possible parallel tasks and by looking for a good compromise between the priorities of the accelerators and the requests of the users. Resource optimization was finally achieved by smoothing manpower peaks where possible. The development of a specific progress monitoring tool has guaranteed the weekly monitoring of the cabling activities, allowing a dynamic redistribution of the resources in case of delayed or anticipated schedule changes and for any occurrences of conflicting activities.
This paper describes in detail the strategy implemented to cope with such a large cabling campaign and the measures taken to make it happen.

Challenges
In the course of LS1 about 1000 km of copper and optical fibre cables will be replaced or newly installed in the CERN accelerator complex, counting roughly 10.000 cables. This project deserves special care for achieving the required reliability of the final cabling infrastructures. In particular, the coordination of the large cabling campaigns and the organization of activities in a harsh radioactive environment or in coactivity with other working teams are important elements of the organization and the execution of the cabling campaigns.

Reliability of the installations
The cabling system (copper and optical fibres) plays a vital role in the operation of the whole CERN accelerator complex. The LS1 represents a unique opportunity to plan for an intense renovation and upgrade of the infrastructures. It is a fundamental part of a 15-year programme aiming at increasing the availability of the cabling infrastructure and at guaranteeing the top quality reliability of all subsystems. For these reasons CERN has organized massive replacement of irradiated and aged cables and non-radiation resistant fibres, as well as the upgrade of the existing installations and the removal of unused cables in particularly overcrowded areas.

Coordination of large cabling campaign
The exceptionally large cabling activity (37 large projects) to carry out during the LS1 requires a wide coordination effort in order to guarantee meeting all the deadlines.
The LS1 management carefully evaluated the workload to decide what is compulsory or what can be postponed after the LS1 in order to ensure the overall programme feasibility. 75% of the proposed projects were accepted, the remaining 25% was postponed to the post-LS1 period.
Moreover, the team (hereinafter the "section") coordination had to face the following main challenges: forecast of the needs, management of external contractors, real time monitoring of the on-going activities and fast adaptability to condition changes during the LS1.

Organization of the field activities
The coactivity with other groups working in the same machine areas, added to the high radiation dose level of certain zones, requires an accurate preparation of the activity in terms of procedures, training, tooling and schedule. The cable trays are often overloaded with cables no longer in use, preventing efficient cable removal or installation (see figure 1 and 2). For some old cables it is even observed moisture in the screen layer as well as deterioration of cable joints. This is mainly the case of the PS accelerator complex and of the SPS, which have been operating respectively for over 50 and 35 years.
Moreover, in those areas the documentation is often unavailable. This has demanded an additional effort to the CERN groups, owners of the equipment, to identify all unused cables onsite and to check them before ultimately being removed.
Performing the cabling activities in these conditions turns out to be very demanding with respect to the accuracy for the work preparation, the efforts for coordination and the repeated evaluation of the execution phase.

Strategy
Well before the beginning of the LS1 it was studied and put in place a precise strategy, mainly affecting the section organization, the activity planning, the material procurement, the work preparation and the strict respect of the CERN safety rules. The details of this strategy are explained in the following paragraphs.

Section organization
During normal years, out of LS1, the cabling and optical fibre section is typically composed by 15 staff (see figure 3) and is partitioned in three main technical units dealing respectively with copper cabling, optical cabling and purchasing of cables and connectors.   During the preparation of LS1, which lasted about 1 year, the section was reorganized both in terms of manpower consolidation and by deploying a new support unit structure, providing more strength to the section Leader's coordination activities. The new section organization chart is represented in figure 4.
The project manager cells were configured in a modular shape to contain all technical competences and resources to perform the given projects.
To face the increasing number of installation requests, project managers and work supervisors were progressively doubled by means of new recruits and by temporary internal mobility from other CERN groups. The number of 35 staff, exceptionally large for a single section, was attained on the activity peak during LS1.
A variable number of projects can be assigned to each cell according to the technical skills and experience of the project manager, obviously by re-shaping the cell resource profile. Project managers are supposed to keep up-to-date with the state of the art of the cabling techniques and, if necessary, to arrange for technical training of their work supervisors. New installation requests will result in adding more cells to the structure, nonetheless without impacting on the organization and coordination capacity of the whole section.
One of the major challenges to manage this large modular structure is to provide to the section Leader the assistance of a proven support team. A Support Unit Coordinator, with established seniority in project technical coordination and CERN administration domains, coordinates a team of 5 support functions. The support functions consists in monitoring the schedule, the contracts and the budget, quality assurance, safety and access procedures and documentation archiving.
The Support Unit takes care of surveying the activities developed across the three main technical units, evaluates working methods, consults regularly the section Leader by reporting on activity progress, proposing project organization improvements and helps the Unit Technical Coordinators in the implementation of recommended actions.

Planning agility
The large number of cabling activities required the implementation of a very dense planning, promoting as much as possible parallel tasks and setting the best compromise between the accelerators priorities and users requests [3].
With agile planning management, activities can be time tuned to new access slots becoming available in the schedule. This action allows smoothing the contractor load curve and building up a contingency reserve for coping with unforeseen circumstances. Figure 5 shows the contractor load curve with a focus on the three CERN main accelerators (i.e. LHC, SPS and PS). The curve was built up according to the workload estimation for the installation projects and shows the number of external staff employed along all the LS1. Figure 6 shows two load curves of the contractor installing optical cables. They are extracted respectively on November 2012 and February 2013. Here the work optimization to smooth as much as possible the contractors load curve can be seen. In the extraction of February 2013 the curve appears much more flat, preventing wherever possible the presence of troublesome peaks and holes.

Procurement anticipation
A large quantity of material is purchased and handled during the LS1. The procurement process has begun two years before the LS1 to avoid all possible lack of material during the cabling activity.
All the installation requests have been received from the users and processed before September 2011. New copper cabling and optical fibre contracts were awarded and led off in 2011 and 2012 as well as most of the orders for material procurement (see figure 7). Since July 2011, 104 contracts and 635 material orders have been signed and led off. Storage areas for the supplied material -5 -   were organized to host roughly 4000 cable drums plus connectors and installation tools. Logistics for ensuring the stock availability and delivery of the material to the worksites were defined and managed by means of dedicated software called GESMAR [4].
A process of standardization has been followed to reduce the diversity of the material across the CERN machines and to simplify the installation process. New cables and connectors have been proposed to all the CERN users recommending selected types of components when possible.
-6 - As direct consequence of this, the organization of the storage has been simplified (fewer articles to manage), the contractors' effort has been reduced (less technical documents and material type to handle) and the purchasing action has been optimized (less contracts to sign and at more advantageous prices). Practical examples are the radiation-resistant cables, the fire-resistant cables and the standard cables, which were regrouped under one common layout and under one common connection convention, with all the benefits mentioned above.

Geographical campaigns
Activities sharing the same working areas are grouped by "geographical campaigns" and accommodated into the same planning slot. The project breakdown and re-organization into geographical campaigns is managed with the aid of GESMAR and represented in figure 8.
For the specific case of the activities located in the SPS area, about 67 installation requests, coming from different users, have been grouped in 11 cable campaigns reducing significantly the time to complete the entire work and simplifying the organization of the material distribution to that area.

Work preparation and optimization
CERN strict rules in terms of safety and quality standards require an accurate preparation of the cabling activities. This can only be achieved with a strict team approach among users, group responsible, contractors, radiation protection and safety officers. Working in high radioactive environments implies, for example, a very precise estimation of the radiation doses absorbed by the workers and the development of specific work procedures aiming at keeping As Low As Reasonably Achievable (ALARA principle) the dose absorption [4]. This is the case of the cable campaign located in the Point 1 of SPS (i.e. TS1+). Thanks to a meticulous preparation of the work, the cumulative dose received by the personnel involved is reduced from 168 mSv to 91.55 mSv, keeping the dose absorbed by each single worker lower than the CERN safety limit (i.e. < 3 mSv/person/12 months).

Training and tests
Training courses and test classes were arranged for contractors and CERN staff. More than 12 training courses, mainly focused on connector fitting, were organized. Only the technicians passing these tests were authorized to work in the machine areas and in the technical galleries. Additional test sessions were arranged to evaluate the realistic time for basic operations in order to minimize the dose absorption.
Finally, in preparation for LS1, the overall working method and contractor reliability were tested during two large pilot projects (LINAC4 and PSS) in 2012, confirming the solidity of the working method.

Execution
Copper and optical cabling supply and installation activities are carried out through industrial support contracts managed by CERN staff of the cabling and optical fibre section. Based on early estimation of the project size, CERN hired five external contractors, four for copper cables and one for optical fibres. More than 100 people at the peak of the activity (see figure 5) had to install and test roughly 10.000 copper cables (20.000 connectors) and 500 multi-fibre optical cables (9.000 optical connectors).
Copper cabling installations are typically constituted by: mechanical works (installation of cable trays and racks), removal of old cables, laying of new cables, fitting and testing of connectors. A similar scheme applies also for optical fibre installations. The main difference consists in the "blowing technique", which is used for putting in place optical fibre cables through pre-installed ducts by means of compressed air.
Well before starting the work, a folder listing all the cables to be installed in the campaign is given to the contractors. Each cable includes the specific installation information such as: starting and ending racks, connector types and pin-out conventions, path in the cable trays and installation layout. This document guides the contractors to organize their installation teams.
CERN staff supervises daily the progress and organize the work reception meeting at the completion of each installation to ensure that the technical specifications were followed. Reworks are decided in case of non-conformities. Figures 9 and 10 show two examples of copper and optical cabling installation.

Activity monitoring and follow-up
An automated tool monitors weekly the progress of all the cable campaigns. This guarantees the respect of the deadlines and prevents that delays provoke a domino effect on the whole schedule. The tool displays the actual progress of the installations in comparison with the baseline and forecast curves (see figure 11).
During the work execution, the contractor regularly updates GESMAR, introducing the information for each pulled cable (type, length and date), connected cable ends (connectors used on both ends and date) and tested assemblies (date). They also have the possibility to update the progress of the mechanical installation, the cable removal and the reworking activities; this allows the monitoring of activities which consist only of mechanical and removal work or following the reworking scheduled after the validation.
-8 -    The collective radioactive dose absorbed by the personnel can be monitored through an additional tool (see figure 12). This is used to alert in case of accumulation of doses higher than expected and to plan an appropriate turnover of the contractor's teams. This tool becomes particularly important for activities located in areas with high radiation dose rates.

Contingency measures
Various contingency measures have been added to cope with unexpected problems. For instance, a third contractor for copper cabling activities was added to compensate for activity delays and lack -9 -of manpower in the main contractor teams. In different situations, due to the LS1 planning density, working in 2 shifts can be allowed for solving coactivity conflicts. Finally, the temporary unavailability of internal resources can be compensated from CERN staff with fully interchangeable expertise.

Conclusions
The management of very large cabling campaigns during LS1 is a challenging activity requiring anticipated strategic actions and significant deployment of resources on-site.
The traditional method for installing copper cables and optical fibres was enhanced in view of LS1, by reinforcing the resource and planning coordination and by optimizing the work organization. A tool for monitoring the LS1 activity progress was also developed.
The new methodologies, procedures and tools developed for LS1, represent a valuable testbench for building experience and addressing LS2 on a much more informed basis.