RBC Status Log

Power Again

2011-11-10T13:55:00.000-05:00

Power to our main server rack in Middlebury was interrupted again this morning a few minutes before 9am--compounded by a battery failure in one of our rackmount UPSes . Once more, this appears to be related to ongoing electrical work elsewhere in the building, but the battery failure was key. We were able to restore power and full network availability by 10:20am.

We just ordered a new "hot spare" backup UPS and replacement batteries and should have them in place this weekend.

Email Server Issue

2011-11-07T07:53:00.005-05:00

GMAVT is having problems with the email server this morning. At this time we are unable to send or receive ripton-coop.net and nbnworks.net emails. As of two minutes ago, they report that they are working on the problem and we expect resolution shortly.

This issue was resolved within 30 minutes.

2011-11-07T07:53:00.001-05:00

POWER!

2011-10-27T10:39:00.002-04:00

Power to our main server rack in Middlebury was interrupted this morning--incredibly, both our main grid power AND our backup power. This appears to be related to ongoing electrical work elsewhere in the building, and we are still working out the details. Fortunately, we were able to restore power and full network availability by 10am.
We will continue to investigate the causes and take appropriate measures.

Concentrator Issue

2011-10-10T22:11:00.002-04:00

We had a problem this evening with a routing device at the W. Cornwall tower which required an onsite fix. All subscribers might have had trouble connecting for a period of one to two hours.

Post Hurricane Irene Report

2011-09-01T10:08:00.001-04:00

We're pleased to report that the network weathered the week's disasters without incident, although a handful of users at the far end of secondary links with limited backup battery power would have been unable to connect, owing to the extended power outages. We will be fitting those secondary relay locations with more robust battery backups before the end of the year.
Almost all users who had power--including their own generator backup--would have had uninterrupted access. We did see a handful of burned or defaulted routers owing to the grid power flickers. This a good time to remind all that a good quality UPS with a healthy battery in it will generally prevent these problems.

Our hearts go out to all of our neighbors who suffered--and the many who continue to suffer--from the terrible losses of this event.

For current road, bridge, and river status Google has stepped up with good online tool, which you can find here:

http://crisislanding.appspot.com/?crisis=2011_flooding_vermont

Micro-Outage

2011-07-30T17:52:00.001-04:00

We experienced an outage this afternoon between approximately 4:51pm and 4:59pm. Our W. Cornwall backhaul radios were unreachable for this period, and this would have caused a system-wide outage. We recovered them with automatic processes in place. We are investigating the causes.

AP Change Out

2011-07-13T22:45:00.003-04:00

Early this morning, between 6:30 and 7:10am, we changed out the original AP radio we installed in Ripton, which lately has been serving just ten clients as a sector facing south from the Hanson tower. Later in the day we replaced some additional associated equipment. This has solved some intermittent service problems we had been seeing recently.

Server Bug

2011-04-16T13:33:00.002-04:00

It appears that the updated NAS server we deployed earlier this week has either a software or configuration bug, which caused it to start authenticating users unreliably starting late this morning. We have replaced that server temporarily while we diagnose and correct the problem. Many Ripton users would have been unable to log in and start a new session between 11:15 this morning and about 1:15 this afternoon. We apologize for the disruption and sincerely thank you for your understanding.

Thunderboomers!

2011-04-11T10:29:00.002-04:00

A severe electrical storm has just passed through the area. It seems to have found a weakness in the oldest of the Ripton APs, which required several manual reboots. The ten services on that AP, who are generally clients south of Hanson, would have had intermittent service through the storm, but that AP has been stabilized and we will be evaluating it for damage as soon as weather permits.

Minor Backhaul Glitch--Last Time!!

2011-04-06T11:47:00.003-04:00

The Han-Bied backhaul link went down again this morning around 8:15am, taking out service for a dozen customers in the village and Breadloaf areas of Ripton. We had a tough time uncovering the problem, but we did definitively this time and implemented a definitive fix and had the link restored by 10:45. We apologize for the inconvenience, but are happy to report that the problem is now truly resolved.

Ugly Power Event

2011-03-11T10:33:00.002-05:00

Last night Ripton experienced serious grid power fluctuations before most of the town lost power mid-evening. Power was restored somewhat later that night, most people reconnected and service appeared normal. There was, however some damage which became evident early this morning, initially on the Ripton village and Breadloaf portions of the network, and then in the authentication server for all of Ripton. A damaged power distribution unit was replaced at the Biederman location by 8:20am, and the server problem was fixed at approximately 9:15.

PERFECT STORM

2011-03-09T12:18:00.003-05:00

From Monday morning through 6:45pm yesterday evening we experienced the longest, most widespread outage in our five-year history. A mega-snowstorm, which prevented vehicle access to our W. Cornwall tower location until it was cleared with a front end loader early Tuesday morning, a concealed equipment problem on the Battell roof pushed over the edge by the drifted snow there, and finally, a defect in a brand new unused spare backhaul radio we attempted to deploy to replace the unit at Battell conspired to lengthen our concerted efforts at restoration.

The only good news is that the discovery of the problem on the Battell roof has already solved a couple of nagging minor network issues, including the past month's problem with the W. Cornwall link, and improved network performance perceptibly.

We are aware that constant broadband connectivity is a necessity, and that this kind of event is absolutely unacceptable. We are evaluating the lessons learned and implementing policy changes, which, in combination with this year's backhaul redundancy plan, will make a recurrence of this disaster impossible. Thank you for your continued patronage and support as we move forward.

Backhaul Outage III

2011-02-25T18:53:00.002-05:00

We experienced an brief backhaul outage this evening at the same location as last time (high on the W. Cornwall tower), from approximately 5:45pm to shortly after 6:00pm when we were able to revive it. We continue to look for a good weather opportunity to change out that problem radio with our standby replacement, and are monitoring it closely in the meantime.

Blip

2011-02-20T11:28:00.003-05:00

We experienced a brief outage of of the Hanson-Biederman link this morning between approximately 9:50 and 10:30am. This may have cut off service to a dozen customers in the village and far eastern areas of Ripton.

Backhaul Outage II

2011-02-19T09:18:00.002-05:00

We experienced an overnight backhaul outage at the same location as last time, compounded by a failure of the automatic alert system. We were able to restore service at around 8:45am. We are upgrading and reconfiguring the alert system, and will be changing out the problematic radio as soon as weather cooperates.

Backhaul Outage

2011-02-08T10:58:00.002-05:00

We experienced a full network outage this morning. A weather issue took down a backhaul radio at Oxford shortly after 7am , and service was not restored until approximately 8:45.

Cable Cut

2011-01-28T18:31:00.002-05:00

A handful of customers in the Breadloaf area of Ripton suffered an outage from around 3:30 to 5:30 this afternoon when a satellite television installer working at the Billings relay location accidentally cut two cables to RBC radios.

Radio Link Problem

2011-01-20T18:40:00.002-05:00

The (up to) 12 clients downstream of the Hanson-Biederman radio link were out today for nearly 12 hours--from around 6:30am to 6:30pm while we struggled to find what turned out to be a very obscure configuration anomaly in that link, with the help of our trusty consultants in St. Louis. We very deeply regret the interruption in service to those customers in the village area of Ripton and higher on the mountain.

Minor AP Restored

2011-01-17T15:58:00.001-05:00

That AP has been restored to service. We had a cabling break plus some damage to the associated lightning arrestor.

Minor AP Down

2011-01-17T11:26:00.002-05:00

We just lost connectivity to a minor AP in Ripton that connects four customers. We are going to attempt to access the unit locally. Please check back for updates.

Backhaul Freezeup

2010-12-28T09:00:00.003-05:00

We experienced a backhaul radio problem at the Oxford location this morning which required a manual reboot. It appears the radio went offline sometime between 7:30 and 8:00am. We were able to locate the problem and bring the radio back up by 8:40. We apologize for the inconvenience, and hope that you all are staying warm on this chilly morning.

Post-Windstorm Update

2010-12-05T18:40:00.002-05:00

Wednesday's windstorm and its hurricane-force winds caused widespread power outages. Most Ripton residents lost power for over 24 hours, many over 40 hours. The network suffered no significant problems, however. Many Ripton residents with backup generators saw no interruption of service. One subscriber in the valley had a damaged tree install, and minor APs in Ripton went down when battery power at those locations went out. We noted that some of these UPS backups did not come on automatically when power was restored, and we will be attempting to figure out why. The more important of those will soon be getting upgraded battery backups for longer run times.

RADIO OUTAGE

2010-10-28T13:17:00.003-04:00

We experienced a rare outage of the entire network beginning early this morning. The problem involved a backhaul radio on the Oxford tower in W. Cornwall, and was corrected with a full restoral of service at a couple of minutes after Noon. This backhaul link is scheduled for dual radio redundancy in the near future, which will make the possibility of this kind of outage negligible. We thank you for your support and patience.

Blip

2010-09-19T09:43:00.002-04:00

We suffered an outage to four subscribers in the village area of Ripton yesterday morning. We discovered the Leeds-Grip link down at about 9am (it appears to have gone down around 8) and had it restored by 10am.

AP Lightning Repair

2010-08-10T23:15:00.002-04:00

A near-direct hit at the Campbell location in Ripton last mid-evening took out the backhaul radio and an ethernet switch, affecting service to four customers. We built a replacement radio and installed it, replaced the ethernet switch, and improved the lightning protection today, restoring full functionality by late afternoon.

NS Again

2010-07-19T16:05:00.002-04:00

For most of the last two days, it appears we have been suffering from a problem with the name server for ripton-coop.net and nbnworks.net domains. Email using those domains has been affected and access to the respective websites. We ordered a move of this functionality to servers at our partners at GMAVT after suffering the same problem last month. We have ordered this change again and will be following up this time to make sure that it is completely effected by the end of the week. Thank you for your patience.

Local Power Outages, Network is Fine

2010-06-24T18:36:00.003-04:00

There are no network problems despite the severe thunderstorms that just passed through the area. However, network monitoring shows several residents of North Branch Road in Ripton down, presumably indicating a local electrical power outage. If you are one of these services, and you find yourself unconnected when power is restored, please reboot your radio (PoE injector), your router, and your computer--in that order. If you continue to have problems, please call 382-8700, press 1 to hear network status, and then leave a message.

Issues Resolved

2010-06-18T19:54:00.003-04:00

The authoritative Name Server for ripton-coop.net and nbnworks.net addresses was restored by our NH vendor this afternoon. As a result, email and the few websites with those addresses are once again fully functional.
We will be moving this function to GMAVT servers in the beginning of the coming week to minimize the possibility of this kind of problem recurring.

Email, RBC website issues

2010-06-17T19:09:00.003-04:00

It appears that the authoritative Name Server for ripton-coop.net and for nbnworks.net addresses has been down since sometime last night. This means that users of email accounts with those domain names may not function with regular (POP/SMTP) clients. Users will be able to access some email with the WebMail client. Additionally, users may be unable to browse to websites with those domain names, such as the RBC website. We have informed the owners of the Name Server in New Hampshire of the problem and they are addressing the issue. When it is solved we will update here and on the status phone line.

Minor Blip

2010-05-30T15:51:00.003-04:00

We just had a very brief outage in our Ripton PPPoE concentrator (5 minutes). RBC customers may have been unable to connect for this period. The server has been restabilized.

Backhaul Breakdown

2010-03-31T15:51:00.002-04:00

At 11:00pm last evening the link from W. Cornwall to Ripton went down, taking out service to all Ripton subscribers. As the problem was high on the W. Cornwall tower, repairs had to wait until daylight and then until the rain abated. We were finally able to diagnose the radio issue and create a temporary workaround, bringing service back up at around 3:45. Thank you all for your patience.

THE NBN TEAM

Email Delays

2010-03-29T15:25:00.002-04:00

It appears that the GMAVT email server got attacked early this morning and bogged down the Red Condor filtering system. Email delivery to ripton-coop.net addresses was held up for some; things appear to be returning to normal at this time.

Server Problem

2009-12-31T11:51:00.002-05:00

A configuration error in our principal bandwidth management server caused it to go down around 9:30 this morning. This caused difficulty in connecting for many subscribers. The problem was resolved at approximately 11:30am. We apologize for inconvenience.

Server Meltdown

2009-12-23T13:14:00.002-05:00

Yesterday, we experienced a failure of a main network server, despite its being equipped with a redundant RAID drive array. As it hosted our Radius authentication server, some customers would have been unable to log back on if they had been off the network during the day, and others would have been prevented from accessing the Internet from later in the evening until sometime after midnight. Fifteen hours of work produced a completely rebuilt server shortly after midnight, and service is now back to normal.

Post Storm Update

2009-12-12T14:49:00.004-05:00

On Wednesday a storm moved through Ripton (as well as East Middlebury and Lincoln) accompanied by extremely high winds in exposed locations. One subscriber whose anemometer failed reported winds estimated above 80mph, another reported her anemometer hit its maximum of 105mph. Some time after the entire town lost power around 3:00 in the afternoon, network monitors indicated a problem at Hanson, the principal Ripton node. Onsite access required rerouting to avoid large treefalls and downed powerlines, and also some chainsaw work to clear the driveway. Inspection of the site indicated some infrastructure had been completely blown off the base of the tower, including all of the lightning protection, and all of the cabling connections had ripped apart.
By shortly after 2am all of the connections had been temporarily respliced, restoring service to 75% of the Ripton network. The remainder of the network traverses repeater nodes without backup generation; existing battery backup will keep those nodes powered for only 20-40minutes into an outage. Of course, subscriber locations without independent backup had to await the restoration of grid power, which occurred for most locations well into Thursday evening, and power to the relay nodes generally occured around the same time.
At this time we are aware of two remaining problems due to blowdowns of trees hosting radios. Unfortunately, both serve multiple customers. The Mullholland backhaul and AP are still out, and we are working to restore service to the three affected subscribers. One client radio serving two customers far north on Natural Turnpike has been recovered and awaits the cleanup of several blowdowns and direction from the subscribers regarding restoral.

DNS Tweaks, Overnight Generator Issue

2009-10-15T17:44:00.001-04:00

It appears that yesterday one of the DNS servers that we use had a problem. This may have caused some subscribers to have difficulty reaching destinations on the Web. The problems were corrected last evening.

Early this morning, a backup generator failed to start properly, causing a network outage for a period between 4:15 and 5:15AM. This, too has been corrected. Given the hour, we hope no one was inconvenienced.

SCHEDULED MAINTENANCE

2009-10-03T14:08:00.000-04:00

Our fiber partner TelJet has scheduled maintenance work beginning at midnight tonight:

Start time (local): 04-Oct-09 at 00:01
End time (local): 04-Oct-09 at 04:00
Work order number: ALBY/0409
Expected Outage/Downtime: 30 minutes

Most Internet customers will be affected by this maintenance, with some destinations being briefly inaccessible as equipment is brought online.

The purpose of this work is: upgrading core hardware.

During this maintenance window, you may experience one or more brief interruptions in service while we complete the maintenance activities; the interruptions are expected to last less than 30 minutes total

Lightning Damage

2009-07-04T14:32:00.003-04:00

A severe thunderstorm late yesterday afternoon damaged both a switch and a minor backhaul radio at the Biederman location. This link was restored at about 2pm this afternoon. Nine customers would have been without service for this period; the network is back to full functionality at this time.

Localized Problem

2009-06-13T22:05:00.000-04:00

It's Saturday, June 13th, at 9:55 pm and network operations are normal at this time. Early this morning, one of the network radios began experiencing problems which affected subscribers in the northwest part of Ripton. AFter a day of troubleshooting, network functionality has been restored. It appears that one of the causes may have been unidentified interfering 2.4 GHZ equipment.

If you are in that part of Ripton and are experiencing problems connecting, please reboot your radio, router and computer in that order. If you continue to experience problems, please call 382-8700 to report it.

OUTAGE

2009-06-05T14:37:00.003-04:00

We experienced a network-wide outage from approximately 12:15 to 2:15pm today due to cascading power and backup power failures at the NOC. We are analyzing the problem and implementing measures to prevent a recurrence.
Thank you for your patience and understanding.

Hiccup

2009-05-22T12:37:00.002-04:00

We experienced an automatic reboot of one of our main servers at approximately 12:05pm today. This would have resulted in a lack of access for many users for a period of 1-2 minutes.

Secondary Problem Corrected

2009-04-14T14:32:00.002-04:00

We just got a definitive fix on the Hanson-Biederman link, which has been problematic for a few days (see the 4/11 posts). The real problem was an internal wire break in a cable which has been replaced. Remaining customers in the Old Town Road and Village areas of Ripton should be able to reconnect.

Service Outage: Backhaul Radio Down and Up

2009-04-14T12:46:00.003-04:00

One of our main backhaul radios at Bicentennial Hall went down sometime between 10:30 and 11:00am this morning. We were able to restore it shortly before Noon. All RBC and NBN customers would have been unable to connect during that period. In the meantime the local point-to-point radio between Hanson and Beiderman has become intermittent; we are addressing that problem now. Customers in the Old Town Road and Village areas of Ripton may continue to have trouble connecting.

Radio Restored

2009-04-11T12:14:00.002-04:00

We restored connectivity through the problem radio shortly before noon. It appears that the problem was simply a failed ethernet port in a local switch in the Hanson rack. Thank you for your patience.

Radio Down

2009-04-11T11:16:00.002-04:00

We just lost ethernet connectivity to the upstream side of the point-to-point radio from Hanson to Biederman. This means that all customers on the Old Town Road branch and in the village on the Grip branch are down. We are working to assess and correct the problem, and will update as soon as we know more.

Local Power Outage

2009-03-08T12:04:00.002-04:00

A localized electrical power outage on North Branch Road in Ripton this morning from about 3am through shortly before 9am (Daylight Savings Time) prevented access for four or five members. Service recovered automatically with restoration of power.

Branch Problem

2009-02-10T12:19:00.001-05:00

We experienced another round of instability in the backhaul radio between Hanson and Biederman between approximately 10:15 and 11:00AM this morning. This would have affected a total of nine subscriber locations.
We have made channel and modulation adjustments to that radio and it is restabilized. Given the occasional interference problems with this link, we are exploring replacement with radios in the newly licensed 3.65 GHz spectrum. Thank you for your patience.

Local Blip

2009-02-05T14:45:00.002-05:00

At about 2:00pm today, one of the sector radios serving about 15 clients in Ripton had to be rebooted manually. This would have caused about 5 minutes of network unavailablity to any of those members who might have been online. We apologize for the inconvenience.

Radius Server Problem

2009-01-25T17:08:00.002-05:00

We had a radius server problem this afternoon that appeared at about 4:10pm and was corrected by about 4:45. During this time, some members who attempted to join or rejoin the network would have been unable to do so. We apologize for the inconvenience.

North Branch Slowdown

2009-01-01T13:44:00.002-05:00

The Northern branch of the RBC network, serving seven households, experienced slow speed problems New Years Eve through the morning of New Years Day. The problem has been alleviated somewhat, but NBN will close down the associated AP radio for maintenance at 4am tomorrow, January 2nd, for some maintenance work. The expected period of network unavailability should last no longer than 30 minutes.

Branch Blips

2008-12-17T17:07:00.002-05:00

It looks like we have been experiencing frequent unnecessary reboots of the Billings backhaul radio, causing frequent but very brief interruptions, affecting only the nine members downstream of that link. This may have manifested itself as frequent brief drops of streaming radio, having to repeatedly request web pages, or similar problems. We've reset the PDU at Billings and the problem is not recurring, but we'll keep an eye on it. Thanks for your understanding, and remember--if you are having a problem please let us know by calling the status line at 382-8700 xt1 and leaving a message.

Ripton Outage and Restoration

2008-11-26T12:24:00.004-05:00

We suffered an outage over the entire Ripton network this morning from shortly before 9am to around 10:10am. The problem was a circuit breaker failure at Hanson, and it was bypassed shortly after 10 and then replaced .

Of late the morning network technician has been getting lax about wearing his pager while having coffee downstairs and starting work, on the assumption that he would hear the pager. Today he learned the hard way that that is not the case. Also, we discovered a fault in the other alert system, and will change the forwarding path for the voicemail messages on the status line so that they go through a backup copper phone line (instead of a VoIP line) before forwarding to email in case the network and/or email is down at the notification target site. We will be making these adjustments immediately. We are also investigating changing out the UPS at Hanson for one that will send us an email alert when it goes over to battery power. The problem was a faulty breaker on the circuit supplying our cabinet there. In all probability the UPS supplied power until the battery was exhausted after the breaker failed. An email alert would quite possibly have given us time to change over the circuit without suffering any down time.

Outage and Restoration, Ripton Backhaul

2008-11-17T15:32:00.003-05:00

The backhaul link from W. Cornwall to Ripton went out yesterday shortly after 1:00 pm. It was well into the night by the time we finally concluded that the problem was between the radio and parabolic antenna on the W. Cornwall tower. We were able to climb this morning and discovered moisture leakage in the lightning arrestor to cable connection at the antenna. This was repaired and waterproofed adequately to prevent a recurrence, and service was restored at about 11:45am.

We will be rechecking all similar connections this week to prevent any further problems of this kind.

Smooth Sailing

2008-11-04T10:21:00.004-05:00

Things are running smoothly. Remember for up-to-date status by phone, call NBN at 382-8700 and press 1 at the prompt.

Server Fixed

2008-10-15T15:07:00.001-04:00

Our bandwidth manager has been fixed, and is back online. We hope that disruptions during this period were minimal and we thank you for your patience.

Server Problem

2008-10-15T01:36:00.002-04:00

Our Bandwidth management apppliance developed a problem around 8:30 last evening and took access to most hosts down for most of an hour before service was restored. We scheduled emergency work for midnight and took the nextwork down for approximately 20 minutes while we attempted to fix the problem. Eventually we decided to take the device off line. We will be continuing work in the morning with more support resources. In the meantime members may notice some irregularity in speeds. We will keep you informed of progress.

Lightning Hit

2008-10-01T15:41:00.002-04:00

We just suffered a very close lightning strike at the W. Cornwall location which appears to have taken out the PDU there. We were able to take it offline quickly but most customers would have been unable to acces the Internet from about 3:17 to 3:28.

OVERCLOCKING

2008-09-28T19:19:00.004-04:00

Sorry, folks. We just found found that we can't trust the auto rate-shifting setting on the new backhaul radio the hard way. This caused a 25 minute period of instability from approximately 6:45 to 7:10 In general, we hope that you are seeing significantly improved speeds, and that this is the last of the bumps associated with this upgrade.

THE NBN TEAM

GROWING PAINS

2008-09-22T14:47:00.002-04:00

We're back online after going down from about 11:00am to 1:15pm due to faulty cabling to the new backhaul receiving unit. We've corrected the problem. At this point we're almost through with the upgrade and would like to thank everyone for their patience--we know it's been trying for the last few days. We hope that you'll find that it's all been worth it

Outage

2008-09-13T11:45:00.002-04:00

The Ripton network just suffered a 17 minute outage from 11:21am through 11:38am. This was caused by unexpected interference between the new backhaul radio being tested and the old backhaul radio which is still in service. As a consequence we will move the remainder of the backhaul testing and changeover to a pre-announced time in the overnight hours when usage is minimal. Thank you for your patience.

Partial 30 Minute Outage

2008-09-12T10:32:00.002-04:00

A problem in the process of the Hanson POP upgrade caused the Main AP there to lose its clients from about 9:55 through 10:25 this morning. We apologize for the inconvenience.

Lightning Redux

2008-06-27T14:50:00.002-04:00

We just had another violent lightning storm move through
West Cornwall and knock out a radio. We had to reboot on site as we are still waiting for our new PDU, which we lost in this weeks earlier lightning storm. Total outage: 1 hr. 13 min.

Back Up

2008-06-25T00:10:00.000-04:00

Well, it appears we took quite a hit at Oxford's. The wind turbine is still down, so we continue to run on propane-fired backup generation, but the network equipment seems to be restabilized. We lost a power distribution unit, a POE injector and some ethernet ports on the big Cisco switch there, but we seem to have finally located most of the problems and implemented fixes. We'll be replacing some equipment and hoping that our steady pattern of violent lightning storms changes sometime soon.

Lightning Damage?

2008-06-24T13:40:00.000-04:00

It appears that we just took a hit at the West Cornwall tower, and lost a backhaul radio. We are on our way there and will report in as soon as we know more.

Email Server Issue

2008-06-18T13:22:00.000-04:00

Yesterday, users of ripton-coop email accounts were unable to send mail as they were not connecting to the SMTP server. This was due to a firewall change at GMAVT...the problem has been corrected. We apologize for the inconvenience.

Rough Week

2008-06-09T10:13:00.000-04:00

Last week was a tough week all the way around. On Monday, June 2, at around 10am, an excavator working near the Hanson tower pulled down a multiport lightning arrestor box, tearing out the wiring for three separate radios including the backhaul to
Ripton and the principal AP. With all NBN personnel out of town, Coop members Charles Billings and Matt and Alison Joseph Dickinson came to the rescue. With a little long distance coordination, they were able locate the spare arrestor box and the necessary tools and install and wire the replacement. The network was back up by 3:30 pm. A great big thanks to those heroes of the day!

Also last week, DNS server issues caused many ripton-coop.net email clients to be unable to access the email servers at GMAVT from sometime Monday through the end of the day Wednesday. The problem has been corrected and procedures are in place to prevent a recurrence.

The above problems were further exacerbated by a failure in the support alert chain, at a time when all NBN personnel were in remote locations, and that problem is being addressed as well.

We thank all of you for your patience and understanding through this most difficult of weeks.

March-ing Along

2008-03-05T10:40:00.000-05:00

There are no known network problems at this time.

Reconfiguring bandwidth distribution pools and Netequalizer settings was completed in the last week. Subscribers should be seeing more bandwidth in general, and speeds more closely tuned to their subscription levels as a result.

Smooth Sailing

2008-02-06T08:41:00.001-05:00

There are no known network problems at this time.

Unscheduled Service Interuption

2008-01-04T12:20:00.000-05:00

The network experienced an interruption in service between 5:20AM and 6:25AM on the morning of Wednesday January 2nd, due to a backhaul radio problem. Automatic rebooting of the radio in question has been implemented and should prevent a recurrence of the problem.
Some subscribers may have experienced slow network performance for a period last evening owing to bandwidth shaping problems, which have been corrected.

Radius Concentrator Hiccups

2007-11-03T20:26:00.000-04:00

We experienced an interruption in function of our Radius concentrator this evening between approximately 6 and 8PM. Some members may have been unable to get back online during this period if they had been off. The problem has been corrected.

BACK UP!!!

2007-10-24T11:10:00.000-04:00

We are back online. The TelJet fiber work was significantly delayed by access issues around the accident site, but technicians have finally completed the last splice. Thanks to you all for your patience.

EMERGENCY NETWORK MAINTENANCE UPDATE

2007-10-24T08:16:00.000-04:00

TelJet crews are still working on the fiber bypass. The emergency command center has been limiting access to the site through the night and consequently progress has been slower than anticipated. The current prediction is for restoration of service by 9:00am.
Thank you for your patience as we work to address this extraordinary situation.

EMERGENCY NETWORK MAINTENANCE

2007-10-23T15:38:00.000-04:00

Emergency network maintenance will be performed by TelJet Longhaul on our fiber feed to Battell Block beginning at midnight tonight.

Start time (local): 10/24/07 at 12:00 am
End time (local): 10/24/07 at 4:00 am
Expected Outage/Downtime: 4 Hours

All RBC and NBN customers receiving Internet service will be affected by this maintenance. The purpose of this work is to install and splice a temporary cable to get around the derailment area in Middlebury and prevent further disturbance to service. The derailment is directly alongside our underground fiber “tap”.

During this maintenance window, you WILL experience an interruption in service while we complete the maintenance activities; the interruption is expected to last until 4:00am. We will post any updates to that prediction.

Network operations engineers closely monitor the work and will do everything possible to minimize any inconvenience to you. If you have any problems with your connection after this time, or if you have any questions regarding the maintenance at any point, please let us know at:

info@nbnworks.net

We thank you for your patience.

BANDWIDTH MANAGEMENT WORK

2007-10-14T11:18:00.000-04:00

We are continuing to reconfigure and refine bandwidth management on the network. New policies will continue to be aimed and controlling file-sharing applications in favor of mainstream applications such as general browsing, voice, IM, streaming audio, etc.

SPEED ISSUES

2007-10-03T08:10:00.000-04:00

We are seeing some negative impacts on network speed as we work out the bugs with our new bandwidth management tools. We are working on the issue and will be reporting here on progress.

NO PROBLEMS

2007-09-10T09:38:00.000-04:00

There are no known network problems at this time.

OUTAGE

2007-09-04T09:01:00.000-04:00

We had a technical problem this morning which would have prevented many of you from going online from late last evening until we discovered and corrected it shortly before 8:00 this morning. We apologize for the inconvenience.

Intermittent issues with GMA

2007-08-22T15:52:00.000-04:00

Green Mountain Access is having some network issues. This only effects email and
web space. They have issued the following statement:

Please note our email server is experiencing a delay of approx. 1 hour
due to a network issue earlier today.

We will post an update when we have it.

Winslow

Smooth Sailing

2007-08-17T17:48:00.001-04:00

We have no known network problems at this time.

LAST NIGHT'S STORM

2007-07-28T09:16:00.000-04:00

Last night a severe thunderstorm moved through Ripton, causing some lightning damage at the Hanson location a few minutes before 5:00PM. After the next cell moved through, we managed a tower climb to change out the AP radio and the problem was corrected at about 8:50PM. Most subscribers came back on line immediately; a few had to wait until later in the evening for corrections to the database in the new broadcast radio. We had already scheduled a major upgrade to the grounding at this tower location for this coming week, and will be refining the lightning protection based on what we have seen in this incident. Thank you for your patience and understanding.

POWER OUTAGE REDUX

2007-07-21T10:46:00.000-04:00

Once again, downtown Middlebury (along with the College) has experienced a sustained power outage. Yesterday, the power went out at about 9:30AM and was not restored until around 12:30PM. Although we do not yet have our permanent propane backup generator installation at Battell Block permitted, we attempted a temporary backup with a small gasoline generator set up outside that location. Unfortunately, this didn’t work, as we discovered some compatibility problems between this unit and our rackmount UPS units, which we believe we have corrected now. Consequently, our network went down when the UPS batteries fully discharged sometime between 10 and 10:30. We came back up shortly after utility power was restored.

As we told you last time this happened, we have moved up the priority of the Battell generator installation, since grid power in Middlebury lately seems to be going through a period of low reliability by historical standards. In the meantime, we are better prepared for an emergency solution. Thank you for your patience.

POWER OUTAGE

2007-07-05T05:33:00.000-04:00

Last evening, July 4th, at 4:30PM, CVPS power went out in downtown Middlebury and remained out until shortly before midnight. At that time our TelJet feed went down but our UPS batteries kept the network going for about twenty minutes-- without internet access. Shortly before 5:00 we went down and didn't come back up until power was restored. When our new propane generator goes online at the Battell location we will be able to ride out these long power outages. TelJet, our bandwidth provider, is addressing the gap in its backup as well.

WEDNESDAY 6/27

2007-06-28T07:41:00.000-04:00

Yesterday (Wednesday, June 27) saw another round of reboots on the primary (Hanson) AP. This despite the elimination of proven and probable interfering devices last week and despite a continuing round of attempts to modify settings and firmware revisions in coordination with Trango technical support all week. This cycle of crashes began at around 2:00PM and continued in a downward spiral until 8:00PM. Subscribers would have experienced unstable connection through that period. At 8:00, the radio appeared to stop passing traffic completely, and we decided to replace the radio on the tower with the remaining light. The new AP went live at 8:51PM and has been stable since.

2007-06-20T07:36:00.000-04:00

OUTAGES and INSTABILITY, PROGRESS
Yesterday, Tuesday June 20, we went through a long period of instability and multiple reboots of the Hanson AP lasting from mid-morning until shortly after midnight. We had in fact been waiting for this period--the first significant wind in a week or so--to test our suspicion that we were seeing a growing problem with interference from the wind turbine that might be responsible for the periods of AP reboots that we have been seeing intermittently. After a 15-hour marathon of on-site babysitting, managing, and testing this event, we were certain that this was the case. We have taken the turbine off-line while we investigate methods for mitigating the interference with shielding, and the system has restabilized. We realize that this period has been frustrating for our member/subscribers , but we are probably at , or near, the end of it. Remember to check here at the Network Status link on the website for updates.

2007-06-05T15:46:00.000-04:00

WE SUFFERED AN EXTENDED OUTAGE LAST FRIDAY (6/1) WHEN THE POWER WENT OUT TO MOST OF MIDDLEBURY. The power went out at around 3:00PM and our battery backup kept us up until around 4:00, but we were down from around 4:00PM through 6:30PM when CVPS restored power. The Middlebury network operations center remains the only major node in the network without backup generation, as installation of the propane generator has been delayed by location and permitting complications.

2007-05-29T04:33:00.000-04:00

Maintenance work went well last night, and we came back up sometime around 3am.

2007-05-28T16:39:00.000-04:00

MAINTENANCE ALERT:

WE WILL BE PERFORMING SOME NETWORK EQUIPMENT UPGRADES TONIGHT, MONDAY MAY 28th, BEGINNING AT MIDNIGHT. THE NETWORK MAY BE DOWN FOR AN HOUR OR MORE. PLEASE PLAN ACCORDINGLY.

2007-05-26T15:17:00.000-04:00

THE NETWORK EXPERIENCED A BRIEF (<5 MIN.) OUTAGE AT ABOUT 12:10 PM TODAY, CAUSED BY OPERATOR ERROR. WE APOLOGIZE FOR THE INCONVENIENCE.

2007-05-22T20:57:00.000-04:00

THERE ARE CURRENTLY NO KNOWN NETWORK PROBLEMS.

2007-05-22T20:53:00.000-04:00

Our network suffered a 9-minute outage yesterday (Monday 5/21/07) between approximately 1:05 and 1:25 PM.

2007-05-16T09:35:00.001-04:00

THERE ARE CURRENTLY NO KNOWN NETWORK PROBLEMS

2007-05-15T23:29:00.000-04:00

OUTAGE: THE LIGHTNING STORM THIS EVENING SHUT DOWN THE INVERTER AT THE HANSON LOCATION THIS EVENING CAUSING AN OUTAGE FROM APPROXIMATELY 9:05 THROUGH 11:06 THIS EVENING. WE ARE TAKING STEPS TO UNDERSTAND THIS EVENT AND PREVENT AND/OR MINIMIZE THE POSSIBILITY OF ITS RECURRENCE.

2007-05-15T19:37:00.000-04:00

WE HAVE FINALLY GOTTEN A FUNCTIONAL FIRMWARE UPGRADE INTO ALMOST ALL OF THE 915SU RADIO UNITS. THIS SEEMS TO HAVE SOLVED MOST OF OUR SLOW SPEED ISSUES. A FEW SUBSCRIBERS ARE SEEING PROBLEMS AS SPRING POPS OUT ALL THOSE DECIDUOUS LEAVES AND WE ARE ADJUSTING INSTALLATIONS APPROPRIATELY.
THANKS FOR YOUR PATIENCE AND SUPPORT. ENJOY THE GREAT SPRING WEATHER!

2007-05-08T10:40:00.000-04:00

WE ARE CURRENTLY EXPERIENCING UNUSUALLY SLOW NETWORK SPEEDS. WE ARE WORKING TO DIAGNOSE AND CORRECT THE PROBLEM.

2007-04-16T10:30:00.000-04:00

THE NETWORK IS UP. WE SURVIVED THE SPRING NOR'EASTER WITH ONE VERY BRIEF REBOOT EVENT. IF YOU ARE NOT CONNECTED, PLEASE GO THROUGH YOUR NORMAL REBOOT SEQUENCE, MAKING SURE TO DO A SLOW POWER CYCLE OF THE RADIO--YOU PROBABLY GOT DISSOCIATED BY ONE OF THE MANY POWER OUTAGES AND SWITCHES WE HAVE EXPERIENCED IN THE LAST 12 HOURS OR SO.

2007-04-05T16:23:00.000-04:00

THERE ARE CURRENTLY NO KNOWN NETWORK PROBLEMS