Statement From FiberNet:
As of 6 am this morning, FiberNet has successfully restored telephone service to over 99% of the customers that had been impacted by the recent outage. Of the small number of customers still without service, many will require an on-premise service call in order for FiberNet to complete service restoration. These service calls are currently underway and will be on-going throughout the day. FiberNet anticipates 100% service restoration by this evening. In the interim, FiberNet continues to devote all of its resources and efforts toward restoring service to all of its loyal customers. FiberNet again wishes to apologize for any inconvenience attribritable to this recent service outage.
UPDATE WEDNESDAY @ 5:00PM
FiberNet says it is working on a "fair compensation package" for its customers after unexpected equipment failure left thousands without telephone service.
The company now reports that 80-85% of of its customers affected by the telephone outage have service. The rest of FiberNet's customers are expected to have service returned sometime Wednesday night.
RELEASE & FREQUENTLY ASKED QUESTIONS (from FiberNet):
At this time, we estimate that 80-85% of our customers affected by the outage have had their service restored. The team will keep working until all customers have restoration of their service. Following are some common questions that we have been receiving from our customers and the media about the outage that occurred yesterday along with answers. We hope that this information will help in understanding the outage.
• What happened?
o At approximately 5:30 am yesterday morning, FiberNet experienced a network outage impacting a majority of its service area in West Virginia. Failure of both the primary and redundant card in a network switching component was responsible for the outage situation.
• Why did it happen?
o It was a combination of failed hardware and a processor switching failure. This was an equipment failure that could not have been predicted or foreseen.
• When will I be back in service?
o Critical customers were restored early this morning. Customers still out of service should be restored by this evening.
• What did FiberNet do – or how did they react – when they became aware a problem existed?
o We dispatched technicians immediately, and determined within the hour that there was a total system failure in that piece of equipment. At that point, two external technical support teams were brought in to assist FiberNet with the restoration. New equipment was ordered, and installation crews were brought in for wiring preparation. Around 3:30 a.m. on Wednesday, July 11, FiberNet began moving customer circuits to the new equipment.
• What corrective action has been or will be put in place to insure or mitigate the possibility of this happening again?
o We brought in additional electronic equipment to further diversify the equipment and we have updated some processes and procedures to further reduce the chances of this happening again.
• What are you going to do for me?
o FiberNet is working on a fair compensation package for our customers affected by this outage.
• Why wasn’t FiberNet prepared?
o We are prepared for the majority of outage situations; however, this was a very unusual outage that should not have happened. According to the equipment manufacturer, this was an extremely abnormal situation. From this experience, we have learned to prepare for even those abnormal situations.
• Why wasn’t there a back up in place?
o There was redundancy built in to the piece of equipment that failed; however, both the primary and redundant card failed at the same time.
• Will there be an announcement when service is restored?
o Yes, we have been updating the media throughout this entire event as new information has been available.
• Why weren’t there reports of the outage sooner?
o Reports were given as soon as we determined there was an issue. Priority was focused on restoration of service.
• Why wasn’t the website updated sooner?
o We have been posting information on our website as new information was received. Again, our main focus was on getting services restored.
• Is this outage an effect of the purchase of FiberNet by One Communications?
o No, One Communications assisted FiberNet with a quick resolution to the problem. The majority of the services affected by this outage were restored in less than 24 hrs, which is an exceptional timeframe for an outage of this magnitude. This equipment failure has absolutely nothing to do with the ownership of the company.
FiberNet has also posted this list on its website. Click here to read all of the news releases from FiberNet.
FiberNet says its still working to get all of its customers back online. About 50% of the customers affected are still without service.
The FiberNet team has worked around the clock. The equipment is programmed and the service that is critical to public health and safety has been installed in the new equipment. We are testing and verifying that each customer’s service is working. FiberNet will continue to work around the clock until service is restored to all our customers. Critical customers (such as hospitals, 911 centers, fire depts) should still call FiberNet Customer Care at 800/320-6144 if they have questions or problems.
UPDATE @ 9:15PM
After the failure of its primary and backup systems, telephone provider FiberNet says customers are starting to get service again. The outage has affected thousands.
A spokesperson with FiberNet says new equipment is in place and they're beginning to bring customers back online.
We're told that because of the nature of the failure, workers have to provision each individual line from scratch. So, some customers may not have service until tomorrow.
FiberNet says its prioritizing which customers will get service first. Hospitals and emergency lines are at the top of the list. However, the company says technicians are working around the clock to get service restored to everyone.
Here is the latest statement from FiberNet:
The new equipment has arrived and has been installed. It has been determined by the manufacturer that this was an equipment failure that never should have happened. FiberNet personnel are in the process of installing customers into the new equipment. Customers critical to public health and safety will receive first priority. After all customers are restored to service, FiberNet will be further evaluating the incident and will develop a corrective action plan to further reduce the likelihood of a reoccurrence of this type of outage again. Updates will be provided when available.
The outage only affects FiberNet customers, but not all of them. A spokesperson with the company says a piece of equipment failed early Tuesday morning -- and the backup also failed.
The company, which is based in Charleston, now estimates 15-20,000 customers are without phone service. FiberNet has about 92,000 customer lines and a network with about 28-hundred miles of fiber optic cable.
We're told St. Francis Hospital in Charleston is without phone service. NEW: the hospital has set up a temporary phone number to call: (304) 414-5908.
Several people have contacted WSAZ in Charleston, Huntington, and Portsmouth to say their phone service is also out.
Phone service is also out for WSAZ's Charleston station. If you need to reach someone, you can call our backup line: (304) 346-0131.
At approximately 5:30 am this morning, FiberNet experienced a network outage potentially impacting a majority of its service area in West Virginia. It is suspected that the failure of both the primary and redundant card in a network switching component is responsible for the current outage situation. Although no estimated time to repair is available at this point, FiberNet is working diligently to resolve this problem just as quickly as possible. FiberNet apologizes to its loyal customers for the inconvenience caused by this situation and will provide further updates as they become available.
Stay with WSAZ.com for the latest information.