Deep Trouble ! Common Problems for Ocean Observatories

● The ocean environment is “brutal” for long term device deployments ● Recovery from or repair of equipment failures is typically very expensive. ● Equipment vendors charge top dollar for equipment that is not necessarily well vetted for observatory use. Quality varies widely both across vendors and across time. ● Equipment deployed in remote locations is subject to accidental and intentional damage from human interactions ● Management of observatory deployment and maintenance is typically quite complex

• The ocean environment is "brutal" for long term device deployments • Recovery from or repair of equipment failures is typically very expensive.
• Equipment vendors charge top dollar for equipment that is not necessarily well vetted for observatory use.Quality varies widely both across vendors and across time.• Equipment deployed in remote locations is subject to accidental and intentional damage from human interactions • Management of observatory deployment and maintenance is typically quite complex Past attempts at ocean observation were often temporally limited relative to the yearlong and multi-year environmental cycles known to exist.Oceanographic instruments typically expended their battery energy within a few months.Instruments that were capable of operating for a year necessarily had to severely limit their sample rates.Data from deployed instruments could only be examined if and when the device was recovered, typically adding up to a year's delay from the time of observation.In recent years, a few large scale observatories have leveraged telecom industry expertise at providing continuous power and communications to undersea nodes and have deployed observatories that can operate 24/7 while providing nearly unlimited power and communication bandwidth.Now that long term ocean observatory technology has been established, new problems have arisen which need to be addressed: • The failure rate of oceanographic instruments during long term deployments is disappointingly high.• The failure rate of connectors, molded cable terminations is disappointingly high.
• Non-profit organizations are not always organized or funded to handle the implementation and support of large scale observatories.• It is difficult to learn from "prior art", partly because long term observatories are relatively new, partly because organizations naturally do not want to disclose failures and partly because the focus is generally on making science data readily available but not necessarily operational data.
This situation presents the conundrum which was the basis of the workshop: How can we improve the success of current and future ocean observatories?
The Workshop Invitations were sent to representatives of the world's major existing ocean observatories.Invitations were also extended to certain vendors of oceanographic equipment.The reason for these latter invitations will become obvious.The workshop conveniently followed the Ocean 2016 conference which occurred earlier in the week in Monterey, CA.Workshop attendees were requested to present brief reports of problems they had encountered and what their solutions were.Following these presentations, work groups were formed to discuss solutions in three major areas (see Working Group Reports later in this paper).

Observatory Reports
Reports from individual observatories are presented in this section.The corresponding slide presentations referenced in these reports are available in http://www.soest.hawaii.edu/Workshop_OceanTech_Lessons_Learned/Presentations.

ALOHA
ALOHA, the the deepest of the represented cabled observatories at 4728 m, highlighted the challenges of replicating the operational environment during the predeployment testing cycle.The observatory suffered failures in instruments that tested correctly at the surface.One sadly spectacular failure was caused by the failure of a stainless steel pressure housing.The metallurgy of the housing led to brittle failure allowing the entire electronics and power suite to be flooded with seawater.This housing was successfully tested to working pressure plus a margin in a freshwater test vessel prior to deployment.It is not known whether additional test cycles, longer duration cycles, different temperatures, and/or testing with saltwater would have uncovered this flaw.The presentation pointed out the need for a standard pressure test program for observatory equipment.
Testing that simulates all of the planned deployment environmental factors is desirable as the combination of temperature, pressure and water composition may create unexpected failure modes.
The deployment methodology can introduce equipment failures and should be factored into any predeployment test plan.Also mentioned in this report were SMART cables, which would could be an add-on to subsea telecommunications cables if and only if, the technology can establish sufficient reliability.

DONET/DONET2
DONET is Japan's national earthquake and tsunami detection network covering the Nankai trough region.Since it is specifically targeted at seismic and pressure sensing, the network has better control over instrumentation and has amassed a nearly 99% data availability record in 5 years of operation.Successful design of the subsea infrastructure for maintainability resulted in very short (10 hr) science node repair time.Cables and connectors have been the weakest elements of the system.A flood tolerant cable assembly that failed due to an insulation fault on one of the internal conductors illustrated the importance of using standard procedures to fully test all elements of a fault tolerant assembly.

KM3NET
KM3NET is a large scale undersea neutrino telescope.Optical sensors scattered throughout a large and deep volume of Mediterranean seawater detect flashes caused by neutrino interactions.The sensors are connected by a seafloor network at a depth of 3500m.The large water volume of the array and its connecting infrastructure introduces significant planning, deployment and life-cycle maintenance challenges.After initial deployments, maintenance operations uncovered problems that had not been considered in the design.Timely repair of failed infrastructure is difficult since observatory class vessels are typically booked far into the future.The system will ultimately have large numbers of sensors and interconnect nodes and will require an advanced maintenance tracking system.

Monterey Accelerated Research System
The MARS observatory is a single node at 890m and located within 2 hours of the Moss Landing harbor in California.The system intent was to provide a high power and bandwidth platform for testing of observatory instruments.The design of the observatory focused on upgrade strategies and the operational challenges of cabled systems It was known beforehand that the deployment would be in a known trawling area.Although MBARI followed best practices to mitigate external fishing aggression, trawl strikes have occurred.As a mitigation effort, MBARI elected to extend cable burial efforts beyond  the primary infrastructure to include the secondary infrastructure.This resulted in the development of a cable burial system suitable for a scientific ROV.The system is in operation and has reduced the risk of trawl damage.The failure of an expensive wet-mate connector caused a loss of availability.This failure is consistent with experience at other observatories that indicts manufacturers of sub-sea connectors.
MBARI, like several of the other observatories represented at the conference, is sometimes in the position of implementing instruments which were conceived by people who understand the science but not necessarily the techniques for design of deep sea instruments.The difference between a conceptual instrument that works in a test pool and an instrument that can survive the rigors of transport, deployment, and long term operation on the seafloor is immense.It is important to realize who is good at what and to pass on responsibility accordingly.

Ocean Networks Canada
ONC manages an ensemble of regional and community observatories, the largest two of which are NEPTUNE and VENUS, both of which provide continuous power and communications to instruments.The deepwater observatory NEPTUNE extends west from Vancouver Island in the form of a ring and reaching depths of 2600m.The VENUS coastal observatory lies in the Strait of Georgia at depths up to 300m.
ONC, like other observatories, has suffered unreasonable failure rates of commercial oceanographic sensors.The more complex the instrument, the more likely it is to fail during the course of long term observatory deployment.Inventory management of instruments is complicated.Instruments must often be returned for vendors for refurbishment and calibration.Each instrument has an individual record of calibrations, failures, and maintenance.
One spectacular example of the forces ocean observatories have to contend was the trawling of a seafloor node.The node in question was placed in a trawl resistant frame.However, the trawler's wire managed to flip the entire frame.The encased node popped out and complicated the recovery and repair.Given the time required for the recovery of this frame and node, the node design was subsequently simplified to reduce future costs of production and recovery.Deployment of NEPTUNE, the earliest of the large undersea observatories, provided several "opportunities to learn".Early efforts at cable laying met with unacceptable failure rates.Engineering shortcuts that had initially seemed good, ultimately led to problems in the unfriendly ocean environment.Reversion to tried and true methods resulted in better success rates.Bathymetry along cable paths led to unacceptable open expanses.This led to improvements in cable lay planning and better monitoring of the cable lay process.
As with other observatory experience, ONC has experienced unacceptable cable and connector failure rates.Connectors from certain manufacturers were much more likely to fail.Improvements in cleaning and greasing before deployment helped mitigate the problems somewhat.Rubber molded cables eventually absorb seawater and fail so re-use is not recommended.Polyurethane jackets are tough and resist water absorption.
Finally, some ONC deployments have been damaged by strikes or entanglement from commercial fishing activities.Surface markings and Notices to Mariners are not sufficient to keep fishing activity away from deployed equipment.Instead, the deployment site must be vetted against known fishing activity.An unmarked subsurface platform is asking for trouble.

Ocean Observatories Initiative Cabled Array
The OOI Cabled Array is a large scale observatory in the northeastern Pacific.This observatory provides continuous power and communications to its instruments.The core infrastructure has been extremely reliable but the observatory has suffered from high rates of commercial instrument and cable/connector failures.During the 2015 deployment year, 25% of deployed instruments failed.Detailed tracking and analysis of instrument failures has guided improvements in instrument testing and deployment processes.
Engineers designing the cabling for the infrastructure of the observatory made extensive use of experiences from other earlier observatories.The operative phrase for cable laying was "Plan Plan Plan".All routes were surveyed 2 years in advance.Meetings with cable and connector vendors and ROV operators were common.Every connector was given a detailed visual inspection prior to deployement.The secondary infrastructure involved the routing of 60 cables totaling 65Km length.Some cables were installed by ROV and some by a cable ship.The only failure during installation was a single "hockle" -a loop twisted into the cable which exceeded the bend radius allowed for the internal optical fibers.
Electronic design and manufacture of all but the primary cabling and node structure was handled by the University of Washington Applied Physics Laboratory.The team all had commercial production expertise but had to face difficult procedural choices due to the limited scale of production.Many operations were "manual" until the team learned where improvement was most needed.For example, internal mechanical assemblies used stainless steel parts while those exposed to seawater always used titanium.The two metals are sometimes difficult to distinguish and a few stainless bolts found their way into exposed assemblies.Fortunately, none of these caused failures but indicated a need for tight procedural control in the assembly area.
Experience with various subsea connectors has shown that certain vendors and certain connector types should be avoided.Many subsea connectors look good on the bench but suffer from poor design in practice.One connector class, including a very expensive hybrid electro-optical connector, regularly releases its internal o-ring if inadvertently tapped.Many titanium connector pairs are difficult to mate if one cannot visually align them first.Visual alignment is often not possible in oceanographic deployments.

Ocean Observatories Initiative Coastal Global Scale Nodes
The OOI CGSN observatory is comprised of a wide variety of oceanographic components including buoys, moorings, gliders, and AUVs.The system used numerous innovative designs but these often suffered from immature development and lack of testing in the deployment environment.Logistics and management of widely varying instruments and platforms was complicated.Instrument vendors made firmware changes that changed the control interface.As in other observatories, connector failures were too common.Biofouling was a particular issue since many of the moorings had permanent surface expressions.
Non-commercial observatories do not have the financial resources to have sufficient spares.If a device fails near deployment time, it often eliminates a part of the science program.One incident represents an issue common to all observatories but especially those implemented by academic institutions or other noncommercial organizations.In the CGSN case, a buoy went through complete testing but stopped working after deployment.The culprit was an untightened vacuum plug.Production checklists are so long that technicians suffer checklist fatigue.Resulting in the grouping of several steps in the list and then checking them all off at once.With this approach, it's easy to check off a step that was not actually completed.
CGSN also reported on the difficulty of supporting software for a wide range of instruments and platforms.For example, the following platforms and languages must be managed at CGSN: Each of these systems may also have one or more configuration files that must be tracked.Some configuration files are changed daily.Given limited personnel resources, the task of managing software and configuration files across these various platforms is very difficult.

South China Sea Experimental Cabled Ocean Observatory
China has established a cabled observatory in the South China Sea.The design is modeled off ONC's Neptune equipment and consists of a single 150 Km cable with two primary and 4 secondary nodes.
Observatory representatives described a cascade failure where a shunt fault in the high voltage supply of a primary backbone cable caused an unexpected failure in a science node further downstream than the short.While the component responsible for the science node failure was quickly determined -it was a shorted TVS diode, determining why the diode failed was difficult.The conditions existing on the seafloor along a 150 Km cable are difficult to replicate in the lab.Theoretical analysis can be used but doesn't always result in an obvious fault mechanism.In this case, two possible causes were posited but testing in attempt to replicate either cause was not possible.Instead, minor hardware changes were made to cover either case and the system has been successfully redeployed.

NOAA
NOAA maintains a large array of observation buoys widely scattered throughout the world's oceans.The buoys collect data and are visited periodically for maintenance and data recovery.Human aggression is a significant source of damage and destruction of this equipment.In the Indian Ocean, between 2004 and 2016, 34 of 148 buoys were destroyed or damaged and the rate of incidence is increasing.Aggression types include accidental snagging by fishing gear, fishing gear purposely attached to buoys or use of buoy to assist in fishing, and malicious mechanical destruction, sometimes by firearms.Data stored in the buoys is often lost.Since the buoys are typically in remote locations, aggression is difficult to prevent or prosecute.A multifaceted approach is being taken to reduce the Damage/loss rate.In some cases, video cameras which upload images via satellite have led to a few prosecutions of vandalism.Use of smooth and tough covers which both protect and hide onboard instrumentation has help.These covers also make it difficult to tie lines to the buoy.Some buoys have been deployed which transmit data via satellite link -thereby eliminating the risk of lost data if the buoys are destroyed.
Vendor Reports ALCATEL ALCATEL-Lucent provided an overview of the evolution of power systems for cabled observatory systems.A new generation of power systems that removes some of the challenges associated with the current systems was described.The new constant current system uses a dual coaxial conductor (DCC) backbone cable and significantly reduces the complexity of the sub-sea power infrastructure.LINK: F13_Lecroart

L3-Maripro
L3-Maripro reviewed the evolution of various node designs from MARS through to the OOI Cable Array.
Lessons learned through the design, development and deployment of each system were applied to follow on systems.This resulted in substantial reductions in size and significant improvements in maintainability as the node design evolved.

LINK: F12_Reardon
Teledyne ODI Teledyne reported on a quality improvement program for their Optical Wet-Mate connectors.The changes were driven particularly by feedback from ROV operations.Basically, the ROV-Mate connectors needed certain improvements to be more reliable in common use cases.

ROPOS
ROPOS is one of the major ROV service providers for undersea construction.This report focused on the importance of early involvement of the ROV service provider in planning for undersea deployments.Cable deployments are particularly troublesome.Cables which are so easily handled in air are unwieldy and uncooperative under deep sea conditions.Extensive joint planning between ROV operators and the clients is critical to successful cabling.LINK: S09_Tamburri Severn Marine This report focused on various methods to prevent marine biofouling.Coatings, UV light exposure and mechanical design to eliminate "quiet water" are covered.The latter is particularly interesting.Marine microorganisms have a difficult time attaching to surfaces that are in continuously current flow.LINK: S06_Lobe

Related Presentations
Finding the right Industrial/Academic Balance Gene Massion addressed the need to combined academic and industrial cooperation.Academic workers may be good at designing proof of concept devices but production is a different issue.The mindset and infrastructure for deploying large observatories is not typically found in academic toolboxes.Engineers who are knowledgeable in ocean operations need to be involved.For example, the successful deployment of the OOI Cabled array was an effort involving scientists from UW oceanography, engineers from UW Applied Physics Laboratory and staff from L3-Maripro.LINK: S08_Massion

Connectors and Corrosion
Kevin Hardy summarized several spectacular moments in his career where something failed that shouldn't have.The resulting investigation revealed a design flaw in a commercial component.Also included in this report is an extensive discussion of corrosion and its prevention as related to ocean observation.LINK: S05_Hardy

Top Problem Areas
Through the course of the reports, it became clear that the problems faced by those implementing ocean observatories covered a broad range.The following list highlights major problem fields.
• The high pressure, corrosive environment causes leaks and destruction of housings

Working Groups
After all the reports were presented and discussed, attendees selected one of three working groups: Testing and Operations, Systems (Moorings, Instruments, Vehicles), Cables and Connectors.These groups met separately to consider how their respective problems might be mitigated.Their meeting notes and conclusions are detailed in the following documents.LINK: WG1_Cables_and_Connectors LINK: WG2_Systems LINK: WG3_Testing_and_Operations The workshop then re-assembled as a unit and heard the reports from the working groups.Much discussion ensued regarding the best steps to mitigate known problems.The following list shows the most supported recommendations.

Recommendations
• International Observatory committee to learn from and leverage expertise from existing organizations, learn from what has been done and facilitate additional recommendations listed below • Create a Wiki on commercial science instruments.This Wiki would include operations tips, test manuals, and a failure database • Create a quality certification process for instruments destined for observatory use.
• Catalog and publish best practices, procedures, checklists and workflows known to be successful in observatory deployment and maintenance • Create an online forum for information interchange related to observatory • Define standards for instrument readiness for deployment • Require instruments ready far ahead of deployment (months) • Where coordination with shore staff is required, have telepresence on ships doing deployments • Develop practices for vandalism protection, identification and/or capture of perpetrators observatory equipment.• Use AIS projection and other notifications when moorings placed • Require all atmospheric or dangerous housings to have pressure relief valves • Drive commercial vendors towards smaller, lower-power instruments with no reagent limitations • For instruments that are live connected in observatory networks, require the ability to safely update instrument firmware via the communications channel.

Figure 1 :
Figure 1: Image showing a crack in the bottom of the pressure housing for the ALOHA MARS Mooring.

Figure 2 :
Figure 2: Image showing arc perforation (small brown spot) in the insulation of a wire from a flood tolerant connector.

Figure 4 :Figure 5 :
Figure 4: Arc damage in a failed wet-mate connector at MARS

Figure 7 :Figure
Figure 7: Cabled Array Instrument Status for deployment ending in July 2016

Figure 8 :
Figure 8: Titanium and stainless steel bolts after a year's deployment.

Figure 9 :
Figure 9: The plug in the center was not fully tightened.

Figure 10 :
Figure 10: Instrument package completely torn off the top of this buoy • Connectors and cable moldings fail at relatively high rates • Instruments often operate correctly in lab tests but fail during deployment • Biofouling limits the operational life to undesirably short timespans • Trawlers, Vandals, other Human Intervention cause substantial damage to deployed equipment • The cost to recover deployed but inoperable equipment is very high • Poor software/firmware supplied by some equipment vendors makes successful long term deployment improbable • Vendors sometimes do not to notify observatory equipment managers of changes made to equipment during refurbishment/recalibration • Launch/Recovery/Shipboard handling, if not planned correctly, can lead to equipment damage.• Ship operations too intense for late planning • Data quality is a big issue.Observatory data must be good or science cannot use it.