Tuesday, January 6, 2015

ccmi2 buoy offline, power failure suspected

On January 5th I sent out a general email alert to LCRC, YSI, C-ARMS and NOAA personnel, noting that the Little Cayman CREWS buoy had gone offline on December 29th (at 21:25 UTC) and asking whether anyone had further information about what might be going on.

Laura Wright of LCRC replied that they still haven't fixed their RF (radio) connection to the buoy and were therefore unable to communicate with it in any way.  She did report that the buoy was still physically in place, though.

Matt reminded me that AOML had put certain firewall kludges in place to allow the buoy to call into our government servers.  This setup is somewhat volatile as it is based on our best guess about what range of IP addresses might possibly be assigned dynamically to the buoy's modem by LCRC's cellular provider.  [This buoy is different from the other CREWS buoys that communicate by cellular modem in that LCRC's cellular provider could not assign them a static IP address.]  I contacted AOML's security guru and he has confirmed that there is no incoming traffic from the buoy on any IP, either within or outside of the expected range.

Matt later was able to check their YSI server's connection to the buoy and report that they too were unable to connect to it.  He suggested that I review the most recent transmissions to look for evidence of a possible power failure.

I reported that yes, the last several days of transmissions before failure indicated that the buoy's power levels were dropping steadily with no sign of charging from its solar panels.  Here is a graph of power levels (plotted in Volts against day-of-year in 2014) for the year:

Click on this image to see a larger version.

Note the unexplained, slow downward trend of low voltages throughout the year.  This is not obviously related to the final loss of power but it is still curious.  The data gaps and very low voltages in October were related to the buoy's recovery to land, its redeployment, and the data black-out from a week in November when the main logger's clock was running five hours behind UTC.  Those data are archived but require further manual intervention before reintegration.

Here is a close-up of the power levels from only the last month before failure:

Click on this image to see a larger version.

Note that the buoy's normal diurnal pattern of daytime charging and nighttime power use was interrupted a bit more than five days before communications ceased, and thereafter there was only a slow but steady power drain.  I would also note that we have observed a handful of cases at other CREWS stations where an underwater instrument or cable has failed, and generally the resulting power drain is much quicker (in those cases the stations went offline within a few hours of the first sign of a power drain).

I also mentioned by email that the "panel humidity" diagnostic appeared to be trending high, both before and after October's annual maintenance operation.  I had thought this was a known issue, because I remembered seeing some email mention of it, but today when I looked for that mention all I could find was a December 8th email from Matt stating that the Barbados buoy's panel humidity was too high.  So perhaps there is an unnoticed/undiagnosed problem with the Cayman buoy's panel humidity, here seen in a percentage-vs.-day-of-year plot for all of 2014:

Click on this image to see a larger version.

Note that we have no reason to think the issue revealed by the panel humidity diagnostic, if there is one, is at all related to the loss of communications with the buoy.

Jon and Matt agree that the Low Voltage Disconnect has likely tripped, but that does not answer the question of what caused the power loss in the first place.  Possible causes include problems with the solar panels, batteries, regulator, or a short in one of the underwater cables or sensors, as speculated by Matt.

Laura reports that, weather permitting, the LCRC team is scheduled to visit the buoy next week, so we may learn more at that time.

(signed)
Mike Jankulak