misc operations during July of 2015

[The following post is a back-dated reconstruction based on email records. In examining this buoy's 2015 data set I realized that there is a gap in data that did not have any explanation recorded on the maintenance log.]

Between July 6th and 22nd, 2015, there were a number of operations carried out on the buoy (including a recovery to land and subsequent redeployment).

Things kicked off on July 6th when Jon Clamp attempted to resolved the WXT failure (see this blog post for details) by replacing the sensor on-site.  Details can be found in an email of July 6th:

Here is the latest update. The WXT that was loaned by AOML has been swapped out on the buoy on site July 6th at around 11am. We did not bring the buoy to shore as I am hoping that this will be the hoped for quick fix. As far as the cellular telemetry is concerned I was not aware of any problems, although this seems to be becoming an increasingly common occurrence. I can get in touch with the cellular supplier to see if there are any issues on our end. Other than that I am hoping the WXT is sending data. I also plan on installing our new base station RF 401 so that we can start receiving data again via radio.

Another email from Jon on July 7th contained some more details:

The visit to the buoy was on Monday July 6th at about 11am which was when the WXT swap out was achieved. Just as a heads up for all. The YSI style buoy is very difficult to perform on site intricate maintenance due to it being a floating buoy as opposed to the previous static stick. There is far too much movement in the buoy to perform intricate electrical diagnostics as there is a real danger of sea water ingress.

The swap out was done by removing the old WXT via the twist bayonet style mount and the cable was removed via the threaded collar. Prior to installing the refurbished WXT I applied some electrical contact grease to the female part of the WXT plug. I was sitting atop the buoy whilst in a fair sea and installed the refurbished WXT. At no point during this visit did I power down the station. I have not looked at any of the internal connections in the junction boxes. The WXT cable was not visually in bad shape and the coupling was also free and easy to manipulate.

The plan was to swap out the instrument as I knew this could be achieved on site. If we are not receiving data then we will have to go to the next step and pull the buoy for a complete inspection. Bearing this in mind we are due to swap out the CTD and PAR in October so we were trying to minimize the number of times we have to pull the buoy out.

We do have pictures and video which we can incorporate into the blog.

I have also brought  2lb of desiccant with me for replacement of the existing desiccant if we end up pulling the buoy out.

The overly large cellular bills were a clerical error and have been cleared up. The changes in the IP address are something that is beyond my knowledge so would need some guidance on how to establish a better system for getting timely data. I assume this would be a dedicated IP address. It may be that we could inquire with the other cellular provider on island. We are currently with Digicel so will check with Lime.

I will be installing the new radio in the next 2 days so hopefully will be able to look at the data ourselves to establish if the buoy instruments are performing as they should.

Jon's references to "cellular telemetry" and "changes in the IP address" refer to questions that I raised on July 6th when I was first told about the WXT swap and asked to check if the new instrument was working correctly.  I replied:

AOML lost contact with the Little Cayman buoy about 18 hours ago, at UTC 22:51 on July 5th.  This would have been about 5:51pm yesterday, local Cayman time.  I checked the last-reported data and there is no sign of any improvement from the WXT.

In retrospect this appears to have been entirely coincidental. CCMI's cellular provider gives them what is known as a "dynamic" IP address.  This means the provider owns a range of IP addresses and, whenever a subscriber device connects, it is assigned one of those IP addresses randomly.  The problem is that neither CCMI nor AOML knows what the range of possible IP addresses is. We have made our best guess based on observations of what IP addresses have been used in the past, but every now and then the buoy's IP address randomly jumps outside of that range and AOML's firewall blocks its attempts to send us data.

It now appears that the buoy jumped to an unexpected IP address on the day before the WXT swap operation.  On July 13th I was able to confirm this with our AOML firewall manager and we opened up access for the buoy to call us from its new address.  At this time I was able to report that the new WXT was not communicating any better than the old one had.

Judging from the data record, the next thing that happened was that the buoy was recovered to land on July 17th at about UTC 1400. On July 18th I received the following email (and attached diagrams) from Jon Clamp:

As the follow up to the conversation below here are the results of my investigation into the failed WXT comms.

Monday July 6th.
The WXT 520 was swapped out with a known repaired and calibrated unit from AOML. This was done on site at the buoy in the hopes of a quick fix. This was not the case as the WXT did not resume its connection.

Friday July 17th.
I pulled the buoy out and have inspected the brain canister. This was found to be in good condition. Upon opening it there was some humidity but nothing to cause concern, a paper towel was all that was needed. The CR1000 and all connections are corrosion free and in good order. All the fuses are intact and corrosion free. I would say with confidence that there are no connection issues inside the canister.

The canister itself had a good seal at the o-ring and there are no nicks or cuts. The glued seal at the top of the canister seems to be holding and looks in good condition.

I next moved to the WXT connection cable and did a continuity test to see if we have an issue with the cable. I have attached the YSI connection drawing and also my findings with respect to the continuity. I might be wrong but think the issue is in the cable. Please advise on this based on the attachments.

My next scheduled task on the buoy is in October where I will be replacing the U/W Bic, surface Bic and CTD. It would seem that it would be good policy to replace all the cables at this time to avoid any other potential connection issues. I am unable to get the cable made up in the Cayman Islands so am leaning towards a) putting the buoy back together with a known fault and rectifying with a new set of cables in October b) making a separate trip back with a new cable and replacing it on site, perhaps August.

I have also installed a new RF401A base station, new cable and antenna so comms to the station are available. Due to having to install new drivers etc for this to run I am not able to automatically connect the Buoy every 10mins for downloads. This is a programming issue with loggernet and I am a bit of a neanderthal when it comes to this so if anyone can help that would be great (manual connection is available and I have been doing this everyday). I am concerned with interrupting the cellular connection so have not made any attempt to sync this. The current RF401A operates at baud rate 38400, pak bus 1, comm port 5 if that helps.

On July 20th I replied by email to Jon, saying:

Looks like your WXT cable is missing a connection to the WXT plug's pin #2.  That is Vin+, or the one that supplies power to the WXT.  This means the WXT has no power, so it's not surprising that it isn't operational.

I also sought some clarification about the language we were using to describe the three junction boxes.  Jon had been talking about a "brain canister" and said that it was the box with the datalogger, but there are loggers in both the "met junction box" (an external canister that is hose-clamped to one of the tower legs) and the "main junction box" (a large enclosure hollowed out in the body of the buoy itself).  Jon followed up that same day with more information about the "canister" he'd been inspecting:

Thanks for the clarification, I will refer to it as the "met junction box” (MJB). However, I decided to perform a pressure test rather than a vacuum test on the “MJB” and unfortunately we have numerous leaks at the glued joint that attaches the cover to the body of the MJB. At this point we will either have to try and seal this (Thinking JB weld for a quick field fix) or replace the box.

Later on July 20th, Matt Previte of YSI weighed in with this email and picture:

1) As Mike mentioned in one of his responses, we DID find a problem with some of the MJB’s with the glue application. This was found after I came to Little Cayman because I was testing/replacing them, as necessary, on subsequent deployments. I think adding some kind of caulk or silicone around the outer edge of that seal should help temporarily. I’m not sure whether JB Weld is appropriate for plastic or not. Long term we need to look at fixing or replacing that cylinder for your October haul out. I’ll inquire here for that and let you know.

2) I did want to point out that there is a fuse within the Met Junction for the WXT. Though that won’t matter if you can’t get continuity through the cable between V+ on the WXT connector and pin 2 on the Pi-connector. However, there may be a grey inline junction on the WXT cable (see attached picture). It may be worth opening that cylinder to see if moisture has somehow seeped in and caused corrosion. I do not hear about this frequency of moisture intrusion, but it may be that these Caribbean conditions are simply pushing past our normal experiences. And we will need to find ways to beef-up the builds to be extra robust.

In the late evening of July 20th Jon sent this update (and photos):

I have attached pictures of my MJB work around. I have put a bead of 5200 on the inside of the canister and put 6 stainless screws on the cover to hold it in place as I do not want the cover to pop off in the future. I have exposed it to a pressure test of +5psi for 20 mins and all seems to be well. The solar jnx box that we repaired April 15 is still in good condition since the repair and the battery box is also good and holds a -5psi vacuum for 20 mins. All desiccant has been has been exchanged. I plan on re-deploying without the WXT in the next 48 hours.

Hopefully we are back on line shortly.

On the next day, July 21st, Jon sent another update about what he'd learned concerning the WXT cable, after having been prompted by Matt's talk about the grey inline junction on the WXT cable:

Based on your thoughts I investigated the inline junctions. There are two grey cylinders incorporated into the WXT cable. The first one I opened, the wiring and soldered joints were in good shape. The second one however had water inside and the soldered connection at the white wire had corroded away causing the loss of power to the WXT. I decided to cut the cable back at six inch intervals toward the pie connector in an attempt to get good wire.Unfortunately this corrosion of the white wire had worked its way all the way back to pin 2 on the pie connector and had actually detached from pin 2 (I cut open the pie connector to confirm this) So after thinking I could possibly get a fix on the cable we are back to re-deployment without the WXT. We will need to get a new cable made up for this.

I will let you all know shortly when the buoy is re-deployed and back on site.

That same afternoon Jon sent another update:

Ok all inspections are done, I briefly disconnected the batteries for inspection so you may see that in the data. As of July 21st 13.30pm (Cayman) all systems are up (except WXT) The buoy is located on land at the station. The plan is to launch the buoy July 22nd at 13.00 (Cayman) The buoy should be on site at its mooring by 16.00 (Cayman). Please can you check to see that there is communication via cellular before I splash the buoy.

Judging from the data record (with particular attention to how the directions reported by the compass are hugely variable while under tow and then settle down once the buoy is moored again), the buoy was redeployed and reporting from site as of UTC 2100 July 22 2015.

One last note, after the buoy was redeployed I was examining its data feed on July 31st and I noticed a jump in compass/wind directions.  I sent Jon the following email and graph:

I got the chance this week to sit down with the post-operations Cayman data feed, and I noticed something is up with the electronic compass.

For the station's entire lifetime to date, the compass has averaged around 236°. This is just an arbitrary direction that is used to correct the wind directions on the two wind sensors (analog and WXT) so that they are calibrated to magnetic north.

After your recent maintenance operations on land, the compass readings jumped about 55° higher, to 291°. I cannot be sure but this *appears* to have changed the trends in normal wind directions reported by the analog anemometer, too. I am attaching graphs of these two parameters (wDir1, ECompDir) over the entire lifetime of the buoy. Obviously I cannot be sure about the WXT directions since that sensor is offline.

The most likely explanation is that the electronic compass is pointed differently than it was before your operations. I know you said that you removed the "met" junction box and relocated it temporarily to a lab that was entirely separate from the buoy. It seems likely that when the met JB was remounted onto the buoy, it was rotated about 55° from where it used to point.

I do not know if this box-pointing procedure is documented anywhere. We are right now meeting with Matt Previte of YSI here in AOML and he is going to send us any "deployment instructions" documentation that he can lay hands on, so that might be mentioned somewhere. But if I understand his explanation correctly, then the Met JB, the analog anemometer, and the Vaisala WXT must all be mounted to point in the same direction. Not north, necessarily, but you must choose an arbitrary "fake north," either the leg of the buoy or possibly a spot on the horizon, and point all three of these sensors in the same direction. I'm told that the top of the Met JB is translucent specifically so that you can see the markings of the electronic compass "north" on the inside when you mount it.

Matt says when he deploys buoys in person, he points the compass directly to the tower leg that it is clamped to. Then the other two instruments are pointed, not at that same tower leg (because then they'd both be 45° off from where they should be) but best-guess "parallel" to how the compass/JB is pointed. The reason this sounds so awkwardly phrased is that I'm describing a process that I've never witnessed and may not fully understand myself. Matt will hopefully chime in with any needed corrections or clarifications, and if there exists any documentation of this wind-sensor-orientation procedure I'm sure he can forward you copies as well.

Anyhow for now I can just apply an ad hoc wind direction correction to the data feed. This will correct the data and "fix" the graphs like the ones I've attached here to eliminate this data jump. But you should make a note and, the next time the buoy comes to land, you will want to have a closer look at the direction that the Met JB is pointing. You will know that it should be rotated about 55° from where it points presently and for guidance you can try to gauge how the anemometer and WXT are pointing.

Jon and I exchanged a bunch of messages that day but in the afternoon he sent me this:

It is a calm day today so I am going out to the buoy to re- orient the MJB. This only requires loosening two hose clamps and rotating the MJB 55* center clockwise. I have made up a pattern to perform this. Expect to see the changes by 1600 Cayman time.

This adjustment corrected the compass/wind direction discrepancies I had noted and the 55° adjustments I had coded were applied only to data in the period between July 20th and 31st.

-- Mike Jankulak