ccmi2 buoy offline, power failure suspected

On January 5th I sent out a general email alert to LCRC, YSI, C-ARMS and NOAA personnel, noting that the Little Cayman CREWS buoy had gone offline on December 29th (at 21:25 UTC) and asking whether anyone had further information about what might be going on.

Laura Wright of LCRC replied that they still haven't fixed their RF (radio) connection to the buoy and were therefore unable to communicate with it in any way.  She did report that the buoy was still physically in place, though.

Matt reminded me that AOML had put certain firewall kludges in place to allow the buoy to call into our government servers.  This setup is somewhat volatile as it is based on our best guess about what range of IP addresses might possibly be assigned dynamically to the buoy's modem by LCRC's cellular provider.  [This buoy is different from the other CREWS buoys that communicate by cellular modem in that LCRC's cellular provider could not assign them a static IP address.]  I contacted AOML's security guru and he has confirmed that there is no incoming traffic from the buoy on any IP, either within or outside of the expected range.

Matt later was able to check their YSI server's connection to the buoy and report that they too were unable to connect to it.  He suggested that I review the most recent transmissions to look for evidence of a possible power failure.

I reported that yes, the last several days of transmissions before failure indicated that the buoy's power levels were dropping steadily with no sign of charging from its solar panels.  Here is a graph of power levels (plotted in Volts against day-of-year in 2014) for the year:

Click on this image to see a larger version.

Note the unexplained, slow downward trend of low voltages throughout the year.  This is not obviously related to the final loss of power but it is still curious.  The data gaps and very low voltages in October were related to the buoy's recovery to land, its redeployment, and the data black-out from a week in November when the main logger's clock was running five hours behind UTC.  Those data are archived but require further manual intervention before reintegration.

Here is a close-up of the power levels from only the last month before failure:

Click on this image to see a larger version.

Note that the buoy's normal diurnal pattern of daytime charging and nighttime power use was interrupted a bit more than five days before communications ceased, and thereafter there was only a slow but steady power drain.  I would also note that we have observed a handful of cases at other CREWS stations where an underwater instrument or cable has failed, and generally the resulting power drain is much quicker (in those cases the stations went offline within a few hours of the first sign of a power drain).

I also mentioned by email that the "panel humidity" diagnostic appeared to be trending high, both before and after October's annual maintenance operation.  I had thought this was a known issue, because I remembered seeing some email mention of it, but today when I looked for that mention all I could find was a December 8th email from Matt stating that the Barbados buoy's panel humidity was too high.  So perhaps there is an unnoticed/undiagnosed problem with the Cayman buoy's panel humidity, here seen in a percentage-vs.-day-of-year plot for all of 2014:

Click on this image to see a larger version.

Note that we have no reason to think the issue revealed by the panel humidity diagnostic, if there is one, is at all related to the loss of communications with the buoy.

Jon and Matt agree that the Low Voltage Disconnect has likely tripped, but that does not answer the question of what caused the power loss in the first place.  Possible causes include problems with the solar panels, batteries, regulator, or a short in one of the underwater cables or sensors, as speculated by Matt.

Laura reports that, weather permitting, the LCRC team is scheduled to visit the buoy next week, so we may learn more at that time.

(signed)
Mike Jankulak

one of the buoy loggers misset to local Cayman/Miami time

[This post was created in January/2015 but back-dated to the time of the incidents described.]

On November 7th, 2014, I noticed that the CCMI2 buoy's two dataloggers were running on different time zones.  The "met" logger continued to (correctly) operate in UTC but the "main" logger was running five hours behind, which coincided with local time at the Little Cayman Research Centre and here at AOML in Miami.  Regarding the only other servers known to connect to these loggers, those of YSI, I'm not sure where they are located or what time zone they run in.

The reset happened on Wednesday, November 5th, at some time between 2:20pm and 2:30pm local Cayman/Miami time.

The most obvious explanation for this problem would be if someone had connected to the main datalogger from a computer running in this -0500 time zone whose local loggernet settings were configured to automatically "correct" the logger clock if it were found to be out of sync with the server.  This is not a default setting on loggernet but neither is it an uncommon setting for people to choose.

However, we were unable to identify any computer that might have reset this datalogger's clock.  Consider:

• Laura Wright reported, on November 12th, that their land-based servers were completely unable to connect to the station in any way:  "The antennae cable/connector end is literally broken. I'm waiting on parts to fix. There is only one computer on station that connects to the buoy via the antennae cable which is currently not connecting due to the cable issue and was the only computer connecting since July 14, 2014 when Rob transferred everything over before that previous computer died."
• The AOML loggernet server runs exclusively in UTC and anyway its settings were verified as not auto-correcting any loggers clocks even if they were found to be out of sync.
• Jon Fajans, on November 8th, suggested the possibility that the main datalogger might have reset its clock based on a time signal provided by the modem, since the modem likely "knows" what local Cayman time is.  However there is no known mechanism for the datalogger and modem to trade time information in this way.  The one bit of clock-sync code that used to exist, that which every hour reset the "met" logger clock using the "main" logger time, was in fact deleted in the last reprogramming because of problems it had caused at one of the Tobago sites.
• On November 12th, Matt Previte of YSI reported: "Verified that our Loggernet is not set to sync clock for Cayman."  He suggested asking CCCCC while saying that he didn't recall setting up the logger to call out to any servers but AOML's and YSI's.  I checked the programming and confirmed that there were no other callouts except for AOML and YSI, and did not pursue the matter further with CCCCC.

In the absence of any explanation for how the clock was reset in the first place, we decided that AOML would correct the time on the main logger through our IP/modem connection.  This was done at 2014/11/12 17:56:46 UTC.  Since then (as of January 2015) we have not noticed any further problems with either logger's clock.

The data from November 5th - 12th are all archived but all of their timestamps are wrong so they have not been loaded or integrated in any way.  Eventually I will write a simple script to correct the timestamps on those records so that this hole in the dataset can be patched.

Just to reiterate, we still have no theory about how the main logger's clock could have been changed on November 5th.

(signed)
Mike Jankulak

2014 Annual Maintenance

The 2014 annual maintenance of the Crews Buoy was completed this October by CCMI staff members Jon Clamp, Laura Wright and Lowell Forbes along with remote assistance from NOAA tech Mike Jankulak and field assistance from C-ARMS principal Jon Fajans.

 

The buoy was released from its mooring and towed to Salt Rock Dock on October 14 where it was lifted out onto our truck with the help of CB Power and Light. Once at the station the buoy was lifted off the truck and set down for maintenance. This included a thorough power washing and fresh bottom paint for the float ring. New zincs were attached as needed along with zip ties as well. The CTD was replaced after some programming issues were updated remotely by Mike Jankulak and was then launched and tested by Jon Fajans in Little Cayman (October 24 at 17:03 UTC). Our Bics were cleaned and left in place for another season due to their reliability. These will be replaced in 2015 or if and when needed during the coming year. A new RM Young Temp/RH sensor was also attached as this had failed earlier in the year (June 6). Jon Fajans completed an instrument test and all was pronounced operational.

 

Finally the buoy was given a pressure test on the battery and electronics housing and given the green light for redeployment. We launched the buoy on October 29 in a fresh Easterly breeze and had to manage the re-attachment of the buoy in less than optimal conditions however no problems were encountered. We were also glad to have the help on this leg by CCMI long time volunteer Greg Locher.

 

Although NOAA is receiving the data stream, CCMI has not received any data updates since redeployment.  We are currently working to resolve this issue.

Please view the video clip of the removal and still photos of the redeployment.

CREWS buoy redeployed, data feeds resume

From examination of the data record, here is a timeline of buoy refurbishment operations:

• 2014-10-14 1800 UTC: buoy towed from its deployment location
• 2014-10-15 1400 UTC: buoy removed from water
• 2014-10-24 1700 UTC: programming updated on both dataloggers
• 2014-10-29 1800 UTC: buoy once again moored at deployment site and fully operational

As of October 31st, 2014, all buoy data feeds are once again operational at AOML.  This includes the near-realtime web reports, the feed to the CHAMP database (as accessible via the CHAMP Portal), and the feed to NDBC's page for this buoy.

(signed)
Mike Jankulak

Data Sharing: CoRIS and NDBC

This is a followup announcement to an earlier post of mine, the one that talked about fully integrating the Little Cayman CREWS station's ocean and light data into CHAMP processing.  These data appear on our web page reports, are loaded into our expert system for ecoforecasting analysis, and are queryable from our CHAMP Portal web site.  Please see this prior post for details on the original causes of the missing ocean/light data and how we worked around it.

One thing that has changed since last month's post is that the concurrency problem with the CHAMP Portal is now resolved.  While there may still be some inefficiencies in serving very large data queries, there is no longer any user concurrency problem.  The CHAMP Portal can be found here:

http://www.coral.noaa.gov/champportal/

Another development is that the CREWS data and metadata reports were submitted to NOAA's Coral Reef Information System (CoRIS) on September 30th, 2014.  This submission covered data collected during the 2013 calendar year, and includes the first report of data from CCMI2 (for a period covering October 23rd through December 31st).

One way to access these data reports is to visit the CoRIS web site:

http://www.coris.noaa.gov/

Once at this site, click on "CoRIS Geoportal Search Tool" (prominently announced in the middle of the page, noted as "New!"), and then as a search term enter "CREWS AND Little Cayman AND 2013" (note that the capitalization of AND is required).

As of this writing this query brings back exactly one record, which is the submitted report for the CCMI2 site for 2013:

http://www.coris.noaa.gov/geoportal/catalog/search/resource/details.page?uuid=%7B8CB71F6D-1371-43E3-9CC7-C812E95BFAAC%7D

Other 2013 (and 2012) CREWS metadata/data submissions may be found on the CHAMP web site as follows:

http://www.coral.noaa.gov/noaa/coris/2013/
http://www.coral.noaa.gov/noaa/coris/2012/

On another topic, the feed of Little Cayman data from AOML to NDBC has begun, dating back to September 24th.  As is the convention with buoys, the Little Cayman station has been given an all-numeric designation by NWS.  This designation is 42089, although we will continue to use CCMI2 informally.  The NDBC data site can be found here:

http://www.ndbc.noaa.gov/station_page.php?station=42089

Please note that all of these data submission and feeds -- the CoRIS reports, the NDBC feed, the online CHAMP hourly and daily web reports, and the CHAMP Portal's data archives -- they all include the complete parameter set from the Little Cayman buoy, including the ocean and light data.

(signed) Mike Jankulak

Routine Cleaning

It is time once again for the CREWS buoy routine cleaning.  As you can see in the photos, we stir up quite a mess but it's all in the name of science!  Nothing out of the ordinary was observed and all reported data remains in the expected ranges.

Integration of Ocean and Light Data

It's not gone unnoticed that the light and ocean data from this buoy have until now not been included in AOML's reports and database.  This was due to a miscommunication between AOML, which processes the output of the buoy, and YSI, which created the buoy programming.  AOML's near-real time processing of CREWS data depends on having one hourly summary table with data from all station instruments, but the buoy's current program only provides 10-minute data summaries for light and ocean data and spreads these data summaries across three data tables, one per instrument.

I am in the process of altering the buoy's CRBasic programming to correct this problem.  Once these changes are complete the programs will be handed off to CCMI/LCRC for upload to the buoy.  Very likely this programming update will be timed to coincide with the buoy's annual refurbishment, when it will be brought to shore for cleaning, repainting, and instrumentation swapout.

We always knew that new routines would have to be created to process the first year's worth of buoy data after the fact.  Those routines have now been created, and with their creation it now becomes possible (through a minorly hackish kludge) to process ocean and light data in near-real time as well.

Complete data reports, including light and ocean data, are now available here for the first time:

http://www.coral.noaa.gov/static/data_ccmi2_Web_12.html

See also the links at the bottom of that page for "Detail" reports that include every parameter reported by the buoy, including diagnostics, and also the "Ten-Minute" reports that show some of the higher-frequency data that are summarized in the "Hourly" reports.

The light+ocean data have also been integrated into the "Ecoforecast" expert system that is running on our Gensym/G2 server:

http://ecoforecast.coral.noaa.gov/index/0/CCMI2/station-home

And the light+ocean data are also queryable through our beta-versioned CHAMP Portal product:

http://www.coral.noaa.gov/champportal/

Note that the CHAMP Portal currently has known issues with user concurrency -- only one visitor at a time may run a query, while others who attempt to connect will see a progress message that will remain fixed in place until the first visitor's query has finished.  Developers are hard at work on this issue and we hope to release a fix very soon.

As a final note about CCMI2 data integration, I have now set up a feed of data from this station from AOML to the National Data Buoy Center (NDBC).  As of this writing the feed is undergoing quality assurance on the NDBC end.  I will post an update (including a link) once the feed has gone live on the NDBC web site.

cheers,
Mike J+