December 13th 2015

Shortly after I left to head back to the US from our annual maintenance of the Crews Buoy our buoy developed a communications problem. In as much that we lost all RF and Cellular comms rendering any contact impossible. This could have been due to any number of issues so a return trip was planned to get the buoy up and running as soon as possible.
             The buoy was pulled out by the Little Cayman team and brought on shore Monday December 2nd. Shortly after I arrived we opened up the Battery Junction Box and had a thorough inspection of the 12v power supply and controller. Upon closer inspection it was found that the fuse that protects the CR1000 data logger had blown. This in turn controls the cellular modem and RF401 so was immediately diagnosed as the issue concerning communications. With this fuse replaced we waited 48hrs to determine if the fuse would blow again and if there was an underlying issue with the fuse circuit. After 48hrs we determined that the buoy was fit to be re-deployed and that the issue was an anomaly perhaps created by a weak fuse or a small power spike. I also took this opportunity to inspect the Met Junction box fuses and replace the desiccant in both Junction boxes.
              The buoy will be re-deployed when a suitable weather window is observed and with any luck we will get back to business of collecting data!

Annual Crews maintenance Oct 2015

Greetings From the Crews project in Little Cayman. During the month of October we completed the annual maintenance of our Crews buoy. The buoy has now been successfully deployed for 2 years and is holding up well given the rigors of an open ocean environment. The running gear and attachment system is in excellent condition.
             The team at NOAA and AOML continue to support our staff with the various issues that pop up when sophisticated ocean and meteorological instruments are deployed. Below is a list of the procedures followed to keep the buoy in optimal condition for the coming year.

1. New bottom paint and general spruce up.

2. Install WXT520, replace WXT520 power and comms cable, configure WXT520 and test. OK

3. Install re-calibrated Surface Bic. 

4. Install re-calibrated U/W Bic.

5. Make new instrument mount for CTD, Program and install and test re-calibrated CTD. OK

6. Inspect battery junction box, found to be dry, no need to wipe with cloth although desiccant was shot. Replace desiccant, new 'O'ring and pressure test vessel to -5psi for 20 mins. OK

7. Current RH in the Met junction box is around 25% so deemed to be good. This was repaired extensively  on my last trip in July so should be good until Oct 2016

8. Check solar junction box. OK

9. Check RF comms. OK

10. Check Cel comms (Mike J)

misc operations during July of 2015

[The following post is a back-dated reconstruction based on email records. In examining this buoy's 2015 data set I realized that there is a gap in data that did not have any explanation recorded on the maintenance log.]

Between July 6th and 22nd, 2015, there were a number of operations carried out on the buoy (including a recovery to land and subsequent redeployment).

Things kicked off on July 6th when Jon Clamp attempted to resolved the WXT failure (see this blog post for details) by replacing the sensor on-site.  Details can be found in an email of July 6th:

Here is the latest update. The WXT that was loaned by AOML has been swapped out on the buoy on site July 6th at around 11am. We did not bring the buoy to shore as I am hoping that this will be the hoped for quick fix. As far as the cellular telemetry is concerned I was not aware of any problems, although this seems to be becoming an increasingly common occurrence. I can get in touch with the cellular supplier to see if there are any issues on our end. Other than that I am hoping the WXT is sending data. I also plan on installing our new base station RF 401 so that we can start receiving data again via radio.

Another email from Jon on July 7th contained some more details:

The visit to the buoy was on Monday July 6th at about 11am which was when the WXT swap out was achieved. Just as a heads up for all. The YSI style buoy is very difficult to perform on site intricate maintenance due to it being a floating buoy as opposed to the previous static stick. There is far too much movement in the buoy to perform intricate electrical diagnostics as there is a real danger of sea water ingress.

The swap out was done by removing the old WXT via the twist bayonet style mount and the cable was removed via the threaded collar. Prior to installing the refurbished WXT I applied some electrical contact grease to the female part of the WXT plug. I was sitting atop the buoy whilst in a fair sea and installed the refurbished WXT. At no point during this visit did I power down the station. I have not looked at any of the internal connections in the junction boxes. The WXT cable was not visually in bad shape and the coupling was also free and easy to manipulate.

The plan was to swap out the instrument as I knew this could be achieved on site. If we are not receiving data then we will have to go to the next step and pull the buoy for a complete inspection. Bearing this in mind we are due to swap out the CTD and PAR in October so we were trying to minimize the number of times we have to pull the buoy out.

We do have pictures and video which we can incorporate into the blog.

I have also brought  2lb of desiccant with me for replacement of the existing desiccant if we end up pulling the buoy out.

The overly large cellular bills were a clerical error and have been cleared up. The changes in the IP address are something that is beyond my knowledge so would need some guidance on how to establish a better system for getting timely data. I assume this would be a dedicated IP address. It may be that we could inquire with the other cellular provider on island. We are currently with Digicel so will check with Lime.

I will be installing the new radio in the next 2 days so hopefully will be able to look at the data ourselves to establish if the buoy instruments are performing as they should.

Jon's references to "cellular telemetry" and "changes in the IP address" refer to questions that I raised on July 6th when I was first told about the WXT swap and asked to check if the new instrument was working correctly.  I replied:

AOML lost contact with the Little Cayman buoy about 18 hours ago, at UTC 22:51 on July 5th.  This would have been about 5:51pm yesterday, local Cayman time.  I checked the last-reported data and there is no sign of any improvement from the WXT.

In retrospect this appears to have been entirely coincidental. CCMI's cellular provider gives them what is known as a "dynamic" IP address.  This means the provider owns a range of IP addresses and, whenever a subscriber device connects, it is assigned one of those IP addresses randomly.  The problem is that neither CCMI nor AOML knows what the range of possible IP addresses is. We have made our best guess based on observations of what IP addresses have been used in the past, but every now and then the buoy's IP address randomly jumps outside of that range and AOML's firewall blocks its attempts to send us data.

It now appears that the buoy jumped to an unexpected IP address on the day before the WXT swap operation.  On July 13th I was able to confirm this with our AOML firewall manager and we opened up access for the buoy to call us from its new address.  At this time I was able to report that the new WXT was not communicating any better than the old one had.

Judging from the data record, the next thing that happened was that the buoy was recovered to land on July 17th at about UTC 1400. On July 18th I received the following email (and attached diagrams) from Jon Clamp:

As the follow up to the conversation below here are the results of my investigation into the failed WXT comms.

Monday July 6th.
The WXT 520 was swapped out with a known repaired and calibrated unit from AOML. This was done on site at the buoy in the hopes of a quick fix. This was not the case as the WXT did not resume its connection.

Friday July 17th.
I pulled the buoy out and have inspected the brain canister. This was found to be in good condition. Upon opening it there was some humidity but nothing to cause concern, a paper towel was all that was needed. The CR1000 and all connections are corrosion free and in good order. All the fuses are intact and corrosion free. I would say with confidence that there are no connection issues inside the canister.

The canister itself had a good seal at the o-ring and there are no nicks or cuts. The glued seal at the top of the canister seems to be holding and looks in good condition.

I next moved to the WXT connection cable and did a continuity test to see if we have an issue with the cable. I have attached the YSI connection drawing and also my findings with respect to the continuity. I might be wrong but think the issue is in the cable. Please advise on this based on the attachments.

My next scheduled task on the buoy is in October where I will be replacing the U/W Bic, surface Bic and CTD. It would seem that it would be good policy to replace all the cables at this time to avoid any other potential connection issues. I am unable to get the cable made up in the Cayman Islands so am leaning towards a) putting the buoy back together with a known fault and rectifying with a new set of cables in October b) making a separate trip back with a new cable and replacing it on site, perhaps August.

I have also installed a new RF401A base station, new cable and antenna so comms to the station are available. Due to having to install new drivers etc for this to run I am not able to automatically connect the Buoy every 10mins for downloads. This is a programming issue with loggernet and I am a bit of a neanderthal when it comes to this so if anyone can help that would be great (manual connection is available and I have been doing this everyday). I am concerned with interrupting the cellular connection so have not made any attempt to sync this. The current RF401A operates at baud rate 38400, pak bus 1, comm port 5 if that helps.

On July 20th I replied by email to Jon, saying:

Looks like your WXT cable is missing a connection to the WXT plug's pin #2.  That is Vin+, or the one that supplies power to the WXT.  This means the WXT has no power, so it's not surprising that it isn't operational.

I also sought some clarification about the language we were using to describe the three junction boxes.  Jon had been talking about a "brain canister" and said that it was the box with the datalogger, but there are loggers in both the "met junction box" (an external canister that is hose-clamped to one of the tower legs) and the "main junction box" (a large enclosure hollowed out in the body of the buoy itself).  Jon followed up that same day with more information about the "canister" he'd been inspecting:

Thanks for the clarification, I will refer to it as the "met junction box” (MJB). However, I decided to perform a pressure test rather than a vacuum test on the “MJB” and unfortunately we have numerous leaks at the glued joint that attaches the cover to the body of the MJB. At this point we will either have to try and seal this (Thinking JB weld for a quick field fix) or replace the box.

Later on July 20th, Matt Previte of YSI weighed in with this email and picture:

1) As Mike mentioned in one of his responses, we DID find a problem with some of the MJB’s with the glue application. This was found after I came to Little Cayman because I was testing/replacing them, as necessary, on subsequent deployments. I think adding some kind of caulk or silicone around the outer edge of that seal should help temporarily. I’m not sure whether JB Weld is appropriate for plastic or not. Long term we need to look at fixing or replacing that cylinder for your October haul out. I’ll inquire here for that and let you know.

2) I did want to point out that there is a fuse within the Met Junction for the WXT. Though that won’t matter if you can’t get continuity through the cable between V+ on the WXT connector and pin 2 on the Pi-connector. However, there may be a grey inline junction on the WXT cable (see attached picture). It may be worth opening that cylinder to see if moisture has somehow seeped in and caused corrosion. I do not hear about this frequency of moisture intrusion, but it may be that these Caribbean conditions are simply pushing past our normal experiences. And we will need to find ways to beef-up the builds to be extra robust.

In the late evening of July 20th Jon sent this update (and photos):

I have attached pictures of my MJB work around. I have put a bead of 5200 on the inside of the canister and put 6 stainless screws on the cover to hold it in place as I do not want the cover to pop off in the future. I have exposed it to a pressure test of +5psi for 20 mins and all seems to be well. The solar jnx box that we repaired April 15 is still in good condition since the repair and the battery box is also good and holds a -5psi vacuum for 20 mins. All desiccant has been has been exchanged. I plan on re-deploying without the WXT in the next 48 hours.

Hopefully we are back on line shortly.

On the next day, July 21st, Jon sent another update about what he'd learned concerning the WXT cable, after having been prompted by Matt's talk about the grey inline junction on the WXT cable:

Based on your thoughts I investigated the inline junctions. There are two grey cylinders incorporated into the WXT cable. The first one I opened, the wiring and soldered joints were in good shape. The second one however had water inside and the soldered connection at the white wire had corroded away causing the loss of power to the WXT. I decided to cut the cable back at six inch intervals toward the pie connector in an attempt to get good wire.Unfortunately this corrosion of the white wire had worked its way all the way back to pin 2 on the pie connector and had actually detached from pin 2 (I cut open the pie connector to confirm this) So after thinking I could possibly get a fix on the cable we are back to re-deployment without the WXT. We will need to get a new cable made up for this.

I will let you all know shortly when the buoy is re-deployed and back on site.

That same afternoon Jon sent another update:

Ok all inspections are done, I briefly disconnected the batteries for inspection so you may see that in the data. As of July 21st 13.30pm (Cayman) all systems are up (except WXT) The buoy is located on land at the station. The plan is to launch the buoy July 22nd at 13.00 (Cayman) The buoy should be on site at its mooring by 16.00 (Cayman). Please can you check to see that there is communication via cellular before I splash the buoy.

Judging from the data record (with particular attention to how the directions reported by the compass are hugely variable while under tow and then settle down once the buoy is moored again), the buoy was redeployed and reporting from site as of UTC 2100 July 22 2015.

One last note, after the buoy was redeployed I was examining its data feed on July 31st and I noticed a jump in compass/wind directions.  I sent Jon the following email and graph:

I got the chance this week to sit down with the post-operations Cayman data feed, and I noticed something is up with the electronic compass.

For the station's entire lifetime to date, the compass has averaged around 236°. This is just an arbitrary direction that is used to correct the wind directions on the two wind sensors (analog and WXT) so that they are calibrated to magnetic north.

After your recent maintenance operations on land, the compass readings jumped about 55° higher, to 291°. I cannot be sure but this *appears* to have changed the trends in normal wind directions reported by the analog anemometer, too. I am attaching graphs of these two parameters (wDir1, ECompDir) over the entire lifetime of the buoy. Obviously I cannot be sure about the WXT directions since that sensor is offline.

The most likely explanation is that the electronic compass is pointed differently than it was before your operations. I know you said that you removed the "met" junction box and relocated it temporarily to a lab that was entirely separate from the buoy. It seems likely that when the met JB was remounted onto the buoy, it was rotated about 55° from where it used to point.

I do not know if this box-pointing procedure is documented anywhere. We are right now meeting with Matt Previte of YSI here in AOML and he is going to send us any "deployment instructions" documentation that he can lay hands on, so that might be mentioned somewhere. But if I understand his explanation correctly, then the Met JB, the analog anemometer, and the Vaisala WXT must all be mounted to point in the same direction. Not north, necessarily, but you must choose an arbitrary "fake north," either the leg of the buoy or possibly a spot on the horizon, and point all three of these sensors in the same direction. I'm told that the top of the Met JB is translucent specifically so that you can see the markings of the electronic compass "north" on the inside when you mount it.

Matt says when he deploys buoys in person, he points the compass directly to the tower leg that it is clamped to. Then the other two instruments are pointed, not at that same tower leg (because then they'd both be 45° off from where they should be) but best-guess "parallel" to how the compass/JB is pointed. The reason this sounds so awkwardly phrased is that I'm describing a process that I've never witnessed and may not fully understand myself. Matt will hopefully chime in with any needed corrections or clarifications, and if there exists any documentation of this wind-sensor-orientation procedure I'm sure he can forward you copies as well.

Anyhow for now I can just apply an ad hoc wind direction correction to the data feed. This will correct the data and "fix" the graphs like the ones I've attached here to eliminate this data jump. But you should make a note and, the next time the buoy comes to land, you will want to have a closer look at the direction that the Met JB is pointing. You will know that it should be rotated about 55° from where it points presently and for guidance you can try to gauge how the anemometer and WXT are pointing.

Jon and I exchanged a bunch of messages that day but in the afternoon he sent me this:

It is a calm day today so I am going out to the buoy to re- orient the MJB. This only requires loosening two hose clamps and rotating the MJB 55* center clockwise. I have made up a pattern to perform this. Expect to see the changes by 1600 Cayman time.

This adjustment corrected the compass/wind direction discrepancies I had noted and the 55° adjustments I had coded were applied only to data in the period between July 20th and 31st.

-- Mike Jankulak

Voltage trends at Little Cayman, 2013-present

This post is expected to be the last of a series of posts to share the results of my recent evaluation of data produced by all of the CREWS/CCCCC buoys over their lifetimes, from 2013 to the present.  This post will briefly discuss the curious downward trend over time in voltage minima that is common to all three operational buoys.

This trend was first remarked upon in an email conversation between myself and Matt Previte of YSI on January 7th and 8th, 2015.  We had had occasion to examine the voltage levels at the Little Cayman (CCMI2) buoy because on December 29th, 2014 it had suffered a complete loss of power.  Subsequent to discovering that power failure I posted an analysis of 2014 voltage levels for CCMI2 with particular attention to the final month of data.  In this post I remarked:

Note the unexplained, slow downward trend of low voltages throughout the year.  This is not obviously related to the final loss of power but it is still curious.

Matt's email to me on January 7th touched upon that subject very briefly:

I'm also surprised by the gradual, overall decline in min/max of the daily battery voltage. I'll ask around to see if anyone else has thoughts on that. It wasn't below operational levels and batteries due wear, but seemed a little odd.

My own January 8th reply to this remark included the following:

I'm pretty sure I've seen similar patterns at (some of?) the other buoys, but I will have to let you know next week if I can back up that statement with real data. [...] I agree that the gradual low-voltages decline is mildly worrying without being hugely alarming.

In fact I did not follow up on this subject as promised until now, since I've just spent several weeks looking at trends in all of the CREWS/CCCCC data, and indeed the gradually-declining trend of voltage minima appears in the data from all three operational buoys.

For this post, we examine the voltage trends at Little Cayman, Cayman Islands (CCMI2).  Voltages are sampled every five seconds and then at 10-minute intervals the minimum voltage from the last ten minutes is reported.  This graph shows voltage minima reported by the Met datalogger (green) and the Main datalogger (red) as well as their difference (in blue, equal to Met - Main).  The first two parameters are graphed on the left axis and the third on the right, with both axes sharing the same scale but offset from one another by 11V.

Please click on this image to see it in larger form.

The most notable features of this graph are as follows: an obvious dip in voltage from October 14th - 29th while the station was on land for maintenance; a loss of data during the period from December 29th, 2014 until March 12th, 2015, caused by the station's power loss and subsequent redeployment; and an unexplained voltage dip at the end of the dataset beginning June 1st, 2015.  Data for this analysis were last refreshed on June 9th, 2015.

At this station the Met voltages were slightly lower than the Main voltages (by 0.057V on average) so my subsequent analysis of battery minima focuses on the Met voltages.

My informal analysis looked for the 'lower edges' of the minima to try to quantify how much they were decreasing over time and how quickly.  This is a largely subjective evaluation.  For CCMI2, this 'lower edge' was about 12.79V at deployment time.  This edge crept lower still by about 0.1V every 3-6 months until the power failure on December 29th, 2014, at which point (apart from obvious power irregularities) I estimate this lower edge to have been at about 12.52V, for a loss of about 0.33V overall.  Following redeployment this trend appears to have reversed itself somewhat, with a lower edge at about 12.60V for a short-term gain of about 0.08V.

Similar analyses were carried out for this buoy's sister stations at Buccoo Reef, Tobago (BUTO1) and at Speyside / Angel's Reef, Tobago (ARTO1).  Two of the stations (BUTO1, ARTO1) reported lower Main voltages on average and one (CCMI2) reported lower Met voltages.  All three stations exhibited a gradual downward trend in voltage minima, losing on average 0.1V every 4-8 months, with some slight changes in pace noted (decelerating at BUTO1, accelerating at ARTO1, constant at CCMI2).  There was also one reversal of this trend noted at CCMI2 following that station's power loss and redeployment in early 2015.

The complete analyses for the other voltage minima, including graphs, may be found at this link for BUTO1 and at this link for ARTO1.

(signed)
Mike Jankulak

Junction Box Humidities at Little Cayman, 2013-present

This post is part of a series of posts to share the results of my recent evaluation of data produced by all of the CREWS/CCCCC buoys over their lifetimes, from 2013 to the present.  This post will discuss the diagnostic relative humidity (RH) data collected from inside two of the buoy's junction boxes: the 'Main' and 'Met' junction boxes which house the Main and Met dataloggers, respectively.  Overly high humidities within either of these junction boxes could lead to a failure of the buoy's controlling electronics and lengthy interruptions in the data stream.

By way of example please see this post from the Little Cayman station log (including photos), which concludes that a "catastrophic power loss" was caused by "condensation" within the "solar panel junction box."  To my knowledge there are no diagnostic RH sensors deployed in the solar panel junction boxes at any CREWS/CCCCC station but this serves as an important lesson about the damage that moisture incursion can have on station operations.  In this case the Cayman station was nonoperational for 73 days and when redeployed it was found that communications with the WXT (Vaisala's 'Weather Transmitter') had failed, which may indicate another yet-undiagnosed effect of junction box condensation at that buoy.

The following graph shows the Little Cayman (CCMI2) diagnostic RH values plotted over the buoy's deployment lifetime to date (through June 9th, 2015).  The red line is RH maxima as measured within the Main junction box and the green line is RH maxima as measured within the Met junction box.

Please click on this image to see it in larger form.

Some obvious features of the above graph are as follows: initial buoy deployment was on October 23rd, 2013; the Main RH levels (red) ceased to fall below 20% on November 16th, 2013; there is a nine-month period when Main RH levels (red) remain strictly above 60% (January 17th - October 27th, 2014); there is a break in data lasting from the station's power failure (December 29th, 2014) until redeployment (March 12th, 2015); and there is a two-week period (April 30th - May 14th, 2015) during which there were no Met RH updates (green) because the Met logger's program stopped running (a situation believed to have been caused by excessive watchdog errors, indicating a potential programming/timing problem).  Note that the station's annual maintenance operations took place October 14th - 29th, 2014, throughout which RH data continued to be collected.

Note that these data report only the maximum RH seen in a ten-minute period of those raw values collected every five seconds.

A natural question is how humid is too humid?  I have heard it suggested that these junction box humidity maxima should not exceed 20%, and the lifetime of Met junction box RH data from the Buccoo Reef, Tobago CREWS/CCCCC buoy shows that this is an entirely attainable goal and can be regarded as a reasonable target.  However, at what point should overly-high RH values prompt remedial intervention?  I have for many years run CREWS programming tests inside my office which has had the side-effect of collecting a long-term dataset of indoor RH values, in an environment that is dry enough to prevent any damage from moisture or condensation.  Based on these somewhat accidental datasets I would suggest that RH values up to 50% may be considered tolerable, but that prolonged measurements of diagnostic humidity in excess of 50% should be considered cause for immediate reparative action.

The story told by these data, then, is twofold:  the Met junction box (green line) begins nicely stable and largely below 10% for about six months, a pattern which starts to be disrupted on April 10th, 2014.  On June 6th, 2014 the pattern shifts significantly above 10% for the first time, and this increasing pattern becomes quite obvious on August 7th, 2014, which is the last Met RH report to fall below 10%.  This only becomes worrisome on October 29th, 2014, the date of buoy redeployment after its annual maintenance operations, which also happens to be the Met RH's last report to fall below the 20% humidity threshold.  Met RH values remain in the worrisome-but-not-alarming range (above 20% but below 50%) from October of 2014 to May of 2015, but following the two-week period of non-updates of Met RH data (i.e. beginning May 14th, 2015, see above) Met RH values are well into the >50% alarming range, with 97.2% of Met RH reports rising above 50% humidity.  In the final 6 days of data examined in this report, in fact, all of the Met RH reports are above 90% humidity.

The Main RH numbers for this station are even more concerning.  These numbers started low but spiked quickly.  Less than one month after initial deployment the Main RH numbers rose above 20% and never recovered, beginning November 16th, 2013.  Two months after that, on January 17th, 2014, the Main RH numbers rose above 60% humidity and remained there until the buoy's annual maintenance operations nine months later in October.  Following those operations Main RH levels dropped slightly but remained at >50% humidity for 97.2% of the time.

These diagnostics indicate that there are presently serious humidity problems in both of the Main and Met junction boxes at this site, with the Main junction box problems being pretty much constant over the lifetime of the buoy to date and the Met junction box problems starting slow but becoming very serious over the last few months.  Given that this is the site where junction box condensation was blamed for a very serious power failure, both of these humidity concerns should be attended to at the earliest possible opportunity.

Similar analyses have been conducted at this station's sister buoys located at Buccoo Reef, Tobago (BUTO1) and at Speyside / Angel's Reef, Tobago (ARTO1).  A pattern that is common to all three of these buoys is that the Main RH levels are all presently at alarming levels, after starting out acceptably low during initial deployment and increasing much more quickly than the Met RH levels do.  This might suggest a design or construction problem with the moisture seals on the Main junction box, or a lack of clear deployment instructions regarding proper sealing of the junction boxes and the use of fresh desiccant.

The Met RH patterns at the three buoys range from BUTO1, where Met RH levels start low and stay low throughout the buoy's entire lifetime, to ARTO1, showing a mildly-increasing trend of Met RH levels that is not yet any cause for alarm, to CCMI2, where Met RH levels began low but increased quickly and are presently at levels that are alarmingly high.  There does not seem to be any reason to suspect a systemic problem with the Met junction box design, construction, or deployment practices as there is in the case of the Main junction boxes.

The complete analyses for the other RH diagnostics, including graphs, may be found at this link for BUTO1 and at this link for ARTO1.

(signed)
Mike Jankulak

WDirDiff/Compass data from Little Cayman, 2013-present

This post is part of a series of posts to share the results of my recent evaluation of data produced by all of the CREWS/CCCCC buoys over their lifetimes, from 2013 to the present.  This post will discuss the offsets (WDirDiffs) between the wind directions reported by the analog anemometer manufactured by RM Young (RMY) and the sonic wind sensors on Vaisala's Weather Transmitter (WXT).  Ideally these offsets should be less than 5° in absolute value.  This post will further discuss the raw directions reported by the buoy's Compass.

For reference, some important milestones in this station's lifetime are as follows:

• 10/23/2013: initial deployment
• 10/14/2014 - 10/29/2014: buoy brought to land for a maintenance operation
• 12/29/2014 - 3/12/2015: station offline due to a power failure, brought to land before redeployment
• 3/12/2015 - 6/9/2015 (present): station's WXT non-operational (no redundant wind data)

The following graph shows the differences in wind directions reported by the two wind sensors (red, on the left axis) and the raw directions reported by the compass (blue, on the right axis).  All directions are reported in degrees of compass but note where the scales are different by a factor of 6x and the zeroes offset, with the WDirDiff axis running on the left from -30° to +30° but the Compass axis running on the right from 0° to 360°.  A negative WDirDiff would indicate that the reported WXT wind directions are lower than the corresponding analog anemometer values.

Please click on this image to see it in larger form.

First of all the Compass averages suggest that this buoy has been deployed in the same orientation throughout its entire lifetime to date.  See the report of WDirDiff/Compass averages for the Buccoo Reef station for an example where this does not appear to be the case.

The second thing to note from this graph is that the WDirDiffs average through the end of 2014 (after which time WXT wind directions are not available for comparison) is +1.5°.  This is entirely reasonable and falls within a range explainable by the specifications of the anemometer (± 5° accuracy) and the WXT (± 3° accuracy).

Similar analyses carried out at this buoy's sister stations at Buccoo Reef, Tobago (BUTO1) and Speyside / Angel's Reef, Tobago (ARTO1) found that the BUTO1 Compass directions can be divided into four distinct "regimes" with subsequent regime averages offset from one another by roughly 180°, and the ARTO1 Compass directions were stable throughout its deployment lifetime to date.  At BUTO1 the lifetime WDirDiff average is -18.6° and at ARTO1 the lifetime WDirDiff average is -11.4°, which suggests that at both Tobago buoys the wind instruments may not be properly oriented with the divergence being more significant at BUTO1 compared to ARTO1.

The complete analyses for the other WDirDiff/Compass averages, including graphs, may be found at this link for BUTO1 and at this link for ARTO1.

(signed)
Mike Jankulak

AirT/RH performance at Little Cayman, 2013-present

This post is part of a planned series of posts to share the results of my recent evaluation of data produced by all of the CREWS/CCCCC buoys over their lifetimes, from 2013 to the present.  This post will discuss the performance of the analog instruments which measure air temperature (AirT) and relative humidity (RH).  These analog reading serve as a basis of comparison for AirT/RH measurements reported by the Vaisala Weather Transmitter (WXT) which also reports wind, barometric pressure and precipitation data.

At Little Cayman the analog AirT/RH sensor's first deployment lasted 226 days before both the AirT and RH data simultaneously went bad on June 6, 2014.  [All instruments on a CREWS/CCCCC buoy are intended to produce usable data for an entire year.]  The buoy was brought to land for an annual maintenance operation on October 14-29, 2014 and suffered a power loss on December 29, 2014 that was repaired on land before redeployment on March 12, 2015.  The AirT/RH appears to have been successfully repaired/replaced during the October 2014 operation and has produced reasonable data since then.  This amounts to 150 days of reasonable AirT/RH data and counting, or 223 days and counting if you assume that the sensor would have operated properly during the buoy's extended power outage.

Note that at the time of the buoy's March 2015 redeployment it was discovered that the WXT communications had failed, so as of this writing there are no redundant AirT or RH readings available to compare against the analog AirT/RH readings.

The following are graphs of AirT (top, in °C) and RH (bottom, in %) from the Little Cayman buoy's lifetime, from 2013 to the present.  Values reported from the analog sensor under discussion are in blue and values from the WXT are in red.  Data are shown through June 9, 2015.

Please click on this image to see it in larger form.

Based on this data record the CCMI2 (Little Cayman) buoy's AirT/RH sensor performed reasonably well for 376 days out of the buoy's 506 operational days, or about 74% of the time.  Its longest stretch of proper operation was 226 days, or about 7.4 months.

Similar analysis performed on this buoy's sister stations at Buccoo Reef, Tobago (BUTO1) and Speyside / Angel's Reef (ARTO1) found that the BUTO1 instrument performed reasonably well for 90 days out of the buoy's 469 operational days, or about 19% of the time, and the ARTO1 instrument performed reasonably well for 181 days out of the buoy's 557 operational days, or about 32% of the time.  The BUTO1 sensor's longest stretch of proper operation was 90 days, or about 3.0 months, and the ARTO1 sensor's longest stretch of proper operation was 181 days, or about 6.0 months.

The complete analyses for the other AirT/RH sensors, including graphs, may be found at this link for BUTO1 and at this link for ARTO1.

(signed)
Mike Jankulak

wxt failure, resumptions of data feeds

This is catch-up post about the state of things with the Little Cayman CREWS buoy, which we at CHAMP refer to informally as CCMI2 and is known to NDBC and the US National Weather Service as buoy #42089:

March 12th, 2015: judging from the way the compass directions settled down, the buoy was redeployed on site at about 2300 UTC.  This would have been at about 6pm local time on this Thursday evening.

March 31st, 2015: in catching up my data spreadsheets before preparing our CoRIS metadata submissions, I noticed that the buoy's WXT appear to be mostly uncommunicative.  The WXT is the instrument made by Vaisala that measures winds and rains (both acoustically) as well as air temperature and pressure and relative humidity.  The few measurements that did make it through appeared to agree with readings from the analog sensors, so this looks like primarily an issue of communications with the instrument.  In fact there are signs of WXT trouble dating back to last December so this problem may have developed concurrently with the corrosion that caused the December power outage, and may have more or less the same underlying cause (terminal corrosion).

At the same time I mentioned that the panel humidity diagnostic seems to be running high, in the 30% - 40% range, which may indicate an improper seal on that chamber or perhaps insufficient desiccant packs.

April 17th, 2015: I was reminded by Kristi Foster of CCMI that the data feeds had never been restarted from this buoy post-redeployment.  Rather than pass along the bad WXT data in our feeds I'd been waiting for word about a possible fix for the WXT problem.  In talking over the status again, Jon Clamp stated that the WXT issue would likely not be resolved before early June at the earliest, and that resolving that problem would probably involve recovering the buoy to land again for repair.  That being the case, I undertook to work around the WXT failure as much as possible and restart the data feeds.

April 24th, 2015: feeds of Cayman data have restarted to NDBC, the CHAMP database and CHAMP Portal, and CHAMP's G2 Ecoforecasting system.  NDBC often chooses to withold data from a new or long-silent station while they perform their own QA process, so our data may not appear on their site until next week.  The CHAMP Portal once again should be loading buoy data in near real-time, although fields sourced from the WXT will continue to be missing or (in some cases) corrupted.  Data loads into G2 have already resumed although this may have limited effect because there does not at this time seem to be any ecoforecasts defined in G2 for this site.

Some links:

• the last 12 hours of data as posted on the CHAMP web page
• the NDBC page for this station 
• the CHAMP Portal, which features data from this site
• the G2 Ecoforecasting page for this site (under construction)

(signed)
Mike Jankulak

With the LC Crews Station being offline since December 29th 2014 and subsequently losing all power and ability to communicate it was decided that we would need to pull the buoy out and do a thorough investigation into the cause. Consequently the buoy was pulled on February 27th 2015.
               The problem was found at the first point of inspection and was due to a failure in the solar panel junction box whereby condensation had entered and corroded the connection terminals and diodes. This caused the catastrophic power loss.

                                          

A new terminal bar was installed along with new diodes which quickly had the buoy systems back up and running. Further comms tests and instrument tests were performed and it was found that our RF comms were inoperable. This issue has been identified solely to the receiving RF unit at the Little Cayman Research Center. Given that this is not a critical issue it was decided to leave the RF comms offline until a repair can be made at Campbell Scientific. The RF should be back up and running in the near future.

                 Upon confirming that the cellular telemetry was working and that NOAA was receiving our data once again it was decided to redeploy the buoy. This was achieved on March 12th 2015.

We are now using a converted boat trailer to make our deployments and retrievals somewhat easier.

ccmi2 buoy offline, power failure suspected

On January 5th I sent out a general email alert to LCRC, YSI, C-ARMS and NOAA personnel, noting that the Little Cayman CREWS buoy had gone offline on December 29th (at 21:25 UTC) and asking whether anyone had further information about what might be going on.

Laura Wright of LCRC replied that they still haven't fixed their RF (radio) connection to the buoy and were therefore unable to communicate with it in any way.  She did report that the buoy was still physically in place, though.

Matt reminded me that AOML had put certain firewall kludges in place to allow the buoy to call into our government servers.  This setup is somewhat volatile as it is based on our best guess about what range of IP addresses might possibly be assigned dynamically to the buoy's modem by LCRC's cellular provider.  [This buoy is different from the other CREWS buoys that communicate by cellular modem in that LCRC's cellular provider could not assign them a static IP address.]  I contacted AOML's security guru and he has confirmed that there is no incoming traffic from the buoy on any IP, either within or outside of the expected range.

Matt later was able to check their YSI server's connection to the buoy and report that they too were unable to connect to it.  He suggested that I review the most recent transmissions to look for evidence of a possible power failure.

I reported that yes, the last several days of transmissions before failure indicated that the buoy's power levels were dropping steadily with no sign of charging from its solar panels.  Here is a graph of power levels (plotted in Volts against day-of-year in 2014) for the year:

Click on this image to see a larger version.

Note the unexplained, slow downward trend of low voltages throughout the year.  This is not obviously related to the final loss of power but it is still curious.  The data gaps and very low voltages in October were related to the buoy's recovery to land, its redeployment, and the data black-out from a week in November when the main logger's clock was running five hours behind UTC.  Those data are archived but require further manual intervention before reintegration.

Here is a close-up of the power levels from only the last month before failure:

Click on this image to see a larger version.

Note that the buoy's normal diurnal pattern of daytime charging and nighttime power use was interrupted a bit more than five days before communications ceased, and thereafter there was only a slow but steady power drain.  I would also note that we have observed a handful of cases at other CREWS stations where an underwater instrument or cable has failed, and generally the resulting power drain is much quicker (in those cases the stations went offline within a few hours of the first sign of a power drain).

I also mentioned by email that the "panel humidity" diagnostic appeared to be trending high, both before and after October's annual maintenance operation.  I had thought this was a known issue, because I remembered seeing some email mention of it, but today when I looked for that mention all I could find was a December 8th email from Matt stating that the Barbados buoy's panel humidity was too high.  So perhaps there is an unnoticed/undiagnosed problem with the Cayman buoy's panel humidity, here seen in a percentage-vs.-day-of-year plot for all of 2014:

Click on this image to see a larger version.

Note that we have no reason to think the issue revealed by the panel humidity diagnostic, if there is one, is at all related to the loss of communications with the buoy.

Jon and Matt agree that the Low Voltage Disconnect has likely tripped, but that does not answer the question of what caused the power loss in the first place.  Possible causes include problems with the solar panels, batteries, regulator, or a short in one of the underwater cables or sensors, as speculated by Matt.

Laura reports that, weather permitting, the LCRC team is scheduled to visit the buoy next week, so we may learn more at that time.

(signed)
Mike Jankulak