Think it can't happen?
On 3/12/15 5:22, William Talanian wrote:
Think it can't happen?
It's interesting what happens when a somewhat technical issue becomes significant enough to reach the popular press. The outages.org mailing list tracked this pretty much in real time and the reports by general news outlets sounded like the results of a game of "telephone" fueled by speculation rumor, and FUD.
* The outage only affected customers singly homed to Centurylink, not "all" voice and data services. There are several other providers with alternate paths.
* Many merchant payment terminals are satellite-based and wouldn't have been affected. Others with non-Centurylink Internet would have also been fine.
* 9-1-1 emergency calls from most landlines and some cell phones continued to function, however the ALI (automatic location identification) data was unavailable for most callers.
* Determining the location of the outage wasn't done by "inspecting mile-by-mile". That assertion is ridiculous on its face. A device called an OTDR is used to determine the location of the break quickly and accurately before going into the field. Measuring from both ends resolves any inaccuracy and will tell if there is more than one break. It isn't unusual for long-haul fiber-optic lines to extend several hundred miles without a repeater, and walking it mile-by-mile is simply not practical. http://en.wikipedia.org/wiki/Optical_time-domain_reflectometer . Fiber runs are typically long MTBF but also long MTTR. After determining the location of the break (not by walking mile-by-mile) trucks need to roll with splicing gear, repair parts, and the like. The break needs to be isolated, the fiber prepped, and sometimes repair requires two splices with a new section in the middle if there isn't a slack loop near enough to pull through. If easily accessible, four to six hours is typical. If the reason for the break is something major like a train wreck, fire, etc. it may be several hours or even days before the repair crew can approach the break to begin work. This happened with the hazmat train wreck at La Conchita locally several years ago. Fiber cuts attributed to vandalism are often really due to copper wire thieves snipping the wrong cable.
* "Five-nines" is a nice idea and is described as the "holy grail", but from a practical standpoint virtually impossible to guarantee to a specific endpoint. That's under six minutes a year. Even in this case, if Centurylink were to total all of their customers worldwide and the number of customer-minutes of downtime, overall their network makes the 99.999% grade. Each extra "9" is ten times more uptime at typically ten times the cost.
Centurylink shouldn't run collapsed rings, but stuff happens and when it happens it will wind up hitting a fan along the way eventually. When the redundancy scheme is more complex than the failure it protects against, it isn't unusual for something in the redundant design to go wrong and itself cause an outage.
For what it's worth, my employer Impulse has a point-of-presence at a data center in Phoenix and several customers in Scottsdale, and neither was impacted at all. Not even a blip.
This thread has some comments as to the CBS News and AP reporting vs. reality. https://puck.nether.net/pipermail/outages/2015-February/007640.html
A power outage lasting for the same duration is much more common and has a much greater impact on business continuity and daily life.
P.S. Yes, public wi-fi is insecure. However, email sent via webmail interface is SSL encrypted end-to-end automatically, and most traditional email clients are encrypted by IMAP-S in the same way. No VPN needed. Not that it really matters as the boot firmware of your hard drive is likely compromised from the factory by our government.
http://www.theregister.co.uk/2015/02/17/kaspersky_labs_equation_group/
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
On 3/12/15 5:22, William Talanian wrote:
Think it can't happen?
It's interesting what happens when a somewhat technical issue becomes significant enough to reach the popular press. The outages.org mailing list tracked this pretty much in real time and the reports by general news outlets sounded like the results of a game of "telephone" fueled by speculation rumor, and FUD.
* The outage only affected customers singly homed to Centurylink, not "all" voice and data services. There are several other providers with alternate paths.
* Many merchant payment terminals are satellite-based and wouldn't have been affected. Others with non-Centurylink Internet would have also been fine.
* 9-1-1 emergency calls from most landlines and some cell phones continued to function, however the ALI (automatic location identification) data was unavailable for most callers.
* Determining the location of the outage wasn't done by "inspecting mile-by-mile". That assertion is ridiculous on its face. A device called an OTDR is used to determine the location of the break quickly and accurately before going into the field. Measuring from both ends resolves any inaccuracy and will tell if there is more than one break. It isn't unusual for long-haul fiber-optic lines to extend several hundred miles without a repeater, and walking it mile-by-mile is simply not practical. http://en.wikipedia.org/wiki/Optical_time-domain_reflectometer . Fiber runs are typically long MTBF but also long MTTR. After determining the location of the break (not by walking mile-by-mile) trucks need to roll with splicing gear, repair parts, and the like. The break needs to be isolated, the fiber prepped, and sometimes repair requires two splices with a new section in the middle if there isn't a slack loop near enough to pull through. If easily accessible, four to six hours is typical. If the reason for the break is something major like a train wreck, fire, etc. it may be several hours or even days before the repair crew can approach the break to begin work. This happened with the hazmat train wreck at La Conchita locally several years ago. Fiber cuts attributed to vandalism are often really due to copper wire thieves snipping the wrong cable.
* "Five-nines" is a nice idea and is described as the "holy grail", but from a practical standpoint virtually impossible to guarantee to a specific endpoint. That's under six minutes a year. Even in this case, if Centurylink were to total all of their customers worldwide and the number of customer-minutes of downtime, overall their network makes the 99.999% grade. Each extra "9" is ten times more uptime at typically ten times the cost.
Centurylink shouldn't run collapsed rings, but stuff happens and when it happens it will wind up hitting a fan along the way eventually. When the redundancy scheme is more complex than the failure it protects against, it isn't unusual for something in the redundant design to go wrong and itself cause an outage.
For what it's worth, my employer Impulse has a point-of-presence at a data center in Phoenix and several customers in Scottsdale, and neither was impacted at all. Not even a blip.
This thread has some comments as to the CBS News and AP reporting vs. reality. https://puck.nether.net/pipermail/outages/2015-February/007640.html
A power outage lasting for the same duration is much more common and has a much greater impact on business continuity and daily life.
P.S. Yes, public wi-fi is insecure. However, email sent via webmail interface is SSL encrypted end-to-end automatically, and most traditional email clients are encrypted by IMAP-S in the same way. No VPN needed. Not that it really matters as the boot firmware of your hard drive is likely compromised from the factory by our government.
http://www.theregister.co.uk/2015/02/17/kaspersky_labs_equation_group/
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
-
Jay Hennigan -
William Talanian