On August 30th, CenturyLink/Level 3 experienced a widespread internet outage lasting nearly 5 hours. This not only impacted their network and their direct customers, but it also affected many other ISPs and services that connected to their network, directly or indirectly.

Each year, there are numerous notable internet disruptions like that one, which can bring businesses to a grinding halt. In just the past few months, Comcast has experienced widespread outage, and AT&T internet service was interrupted or slowed throughout Florida.

Events like these affect every organization that relies on cloud-based applications and video-based communications to maintain day-to-day operations and serve customers. Outages reveal the increased power and indispensability of these tools for business and highlight the importance of internet performance — frequently shining a spotlight on poor WAN performance. This can cost a business even more than the estimated $5,600 per minute that Gartner calculated back in 2014.

While outages are show stoppers, they can still be considered relatively rare. However, jittery VoIP, flaky video calls, or lagging ERP and point-of-sale tools are everyday occurrences, with a significant impact on productivity and the bottom line. Depending on the size and nature your operation, poor network performance can cost hundreds or even thousands of dollars per minute.

It is an ongoing challenge for businesses to keep mission-critical operations running smoothly over internet connections, especially as more tools are regularly spun up as needed at the department, team, or even user level without IT involvement. And as adoption of cloud-based applications and SaaS solutions increases, traditional networks become even more complex and difficult to manage. Network engineers and IT teams are increasingly strained by dynamic technology stacks (cloud-based, on-premises, and hybrid) and a workforce that is more geographically dispersed.

Smarter network operations

Bigleaf offers a cloud-first SD-WAN solution that improves network efficiency, optimizes IT resources, and helps you create a more resilient and dependable network for your mission-critical applications.

We make it easy to build a transparent, worry-free network that frees IT teams from cumbersome network management tasks and support calls. With our intelligent software, owned and operated Cloud Access Network, 99.99% SLA-backed uptime, tier one support, and firewall-friendly design, Bigleaf helps IT teams to deliver solid internet and cloud app performance more easily in an uncertain environment.

Bigleaf recognizes that tech stacks constantly evolve. Apps are deployed and retired almost daily because individual users have different requirements. If a webinar presenter is dissatisfied with one video conferencing tool, you can be sure they’ll deploy a different one next time. These types of changes don’t need to be a four-alarm fire that spurs the IT department to make manual SD-WAN configuration changes for QoS and firewall compatibility. Bigleaf’s intelligent solution makes it easy for businesses to scale and adjust to changes, in many cases automatically, while maintaining uptime and performance.

These applications and technologies represent investments that business need to see pay off every day. That requires reliable performance at headquarters, branch offices, and home offices no matter what tools are being used. Bigleaf meets those needs with redundant, dependable SD-WAN that’s smarter than your average network architecture.

By stabilizing the network for the technology that powers your business, you can rest easy knowing VoIP won’t fail during an all-hands conference call, video won’t get jittery or freeze during the next big sales presentation, and sales and inventory transactions will go through every time.

To learn more about Bigleaf’s intelligent networking solution from a real-world perspective, watch this webinar — Creating a resilient network: Q&A with Lionakis IT Director Matthew Onken.

So what happened? Bigleaf’s founder and CEO took a few minutes to share some insights from our perspective running a network that peers with CenturyLink and many others.

Video transcript

Hi, I’m Joel Mulkey, founder and CEO of Bigleaf Networks. You may be wondering what happened with the CenturyLink / Level 3 outage that happened yesterday, on Sunday. We definitely saw that going on. We own and operate our own backbone network and had a lot of visibility and were able to respond to that. I’ll talk you through a little bit of that now.

So we had a couple responses to that. One that was automated. So our SD-WAN software that is managing traffic constantly – 10 times a second – detected that and responded by rerouting customer traffic wherever possible, keeping folks up and running, which was great. We also had a manual response. We have a skilled network engineering and operations team who was alerted and went and dug in and found some optimizations they could make in how traffic was flowing.

And through that, we saw some different customer experiences. So the nature of this issue that CenturyLink / Level 3 had – it was actually the Level 3 network which is now owned by CenturyLink but not fully incorporated – they had this BGP issue. BGP is the routing protocol that runs the internet, and you can think of it as a sort of black hole sort of an issue where, like an onramp on a freeway. If Waze is sending all the traffic to that onramp but there’s actually an accident, this was a similar scenario where CenturyLink was saying “Hey, get to all these networks through me,” yet their network wasn’t functioning right.

And so, it was a very difficult time for network operators – kind of unprecedented with CenturyLink even telling other big carriers, “Hey, shut off your connections to us,” which was a pretty substantial move, disconnecting one of the world’s biggest networks from the internet. But they had about a 4-hour outage, from 4 am Pacific Time to about 8 am Pacific Time.

And we were able to respond to that. So our customers, they saw and experienced – if they had multiple WAN circuits, maybe they had AT&T and something else, generally they stayed up and running. Although because of that black hole-ing, no exclusively. Some customers had outages because their traffic was flowing through CenturyLink and CenturyLink was just dropping it. And then on the content side, if you were trying to reach content that was hosted by CenturyLink, even as one of the paths to that content, you may have been able to not reach it. So it was quite the dramatic moment.

Thankfully, most of our customers stayed up and running, and were happy. If you only had one WAN circuit with CenturyLink, obviously you were down. In that case, I certainly do recommend – take a look at diverse WAN connections, take look at Bigleaf as an intelligent SD-WAN platform that can automatically mitigate these kinds of issues. And let us know if you have any questions. We’re happy to share more about this, what we saw, and how our platform can help. Thanks.

Sometimes the internet breaks. 

Over the course of two hours on 24 June 2019, the internet broke down for most of the United States. Popular websites and apps were inaccessible on browsers and phones.  

The cause was achingly human while also being deeply technical. It is called a route leak: A Border Gateway Protocol (BGP) route list that was intended as a map to guide traffic between a few networks was published to networks that should not use those directions. It is like all the rush hour freeway traffic being routed to a suburban side street.  

As a result, traffic for 2,400 networks was unfortunately sent through the network of Allegheny Technologies in Pennsylvania. Their infrastructure was not up to the task and most requests failed. 

BGP is one of the many arcane arts that usher traffic across the internet. The “inter-net” is a connection of many autonomous networks, and BGP provides rules for how to get from here to there by moving data from one network to another. A BGP route is somewhat like the turn-by-turn directions you get from Google Maps, only it tells data how to get from a server in Bellevue, Washington to your customer support desk in Trenton, New Jersey.

Propagation of a bad BGP table is preventable. This was clearly an error that everyone agrees never should have happened, but it did. And while the Allegheny incident was a high-profile breakage whose source we can identify, this sort of thing happens in harder-to-diagnose ways all the time.  

Due to the nature of internet infrastructure and the laws of probability, they are inevitable. The internet will break, connections will drop, services will fail for no obvious reason. 

The more you know about how the internet functions the more difficult it is to believe that it works at all. Along with leaky BGP routes, services depend on DNS, content delivery networks, cloud service providers, and a variety of technologies run by different companies falling well beyond the reach of the customer support or sales person whose web browser is displaying a cute “504, timed out” message instead of the new customer’s loan document.  

Where does that leave your business operations, particularly now that cloud-based SaaS applications are taking over?  

If your vendor is not taking your concerns about outages seriously, they clearly don’t know much about the “modern” internet. 

The concern naturally increases when the risks are greater. The closer the cloud-based solution is to customer engagement where customers are won and lost, the more reasonably nervous you would be about uptime.  

• If you are a car dealer and your parts lookup is cloud-based, short downtime is awkward and undesirable.  
• If your customer-facing staff rely on a scheduling system based in the cloud, downtime is an absolutely terrible prospect.
• If your medical clinic’s electronic health records or electronic medical records are cloud-based, downtime is completely unacceptable. Significant downtime needs to be beyond belief.    

For some locations, such as many rural and suburban areas of the US, the internet breaks worse and more often. When considering a cloud-based or SaaS solution for a business, concerns about downtime are legitimate and substantiated. Regardless of the technical advantages, inconveniencing customers isn’t worth it. Putting the weak links of the internet between the business and customer interaction at the service counter isn’t worth it. 

As technologists, we can’t just complain and shirk connectivity. These applications are the key to being competitive in the modern marketplace. We have to make cloud solutions functional and reliable. They simplify business operations, keep technology up to date, and save money.  

Despite everything fragile and subject to failure between that key service and our users, we have to create resilience the right level of resilience.  

Key Network Issues for SaaS Deployments 

• Uptime and bandwidth 
• Management and support requirements 
• Security 

Uptime and bandwidth 

Some things you don’t want to know, such as how many problems the internet has at any one time. Not every issue makes the news, but even very short incidents can cause problems for mission-critical real-time applications. A hiccup at the ISP can be enough to drop a call or tangle up a customer service response.  

A study of Bigleaf router performance data shows that a typical single-ISP business experiences 3.5 hours of internet downtime a month. What’s more, they experience an additional 23 hours of severely degraded service from jitter, low throughput, and other internet problems that don’t register as downtime but the effect on applications – and thus customer experience – is the same. It is downtime by another name. 

Calculating management and support 

When networking gets critical, the solutions can be very involved. They can become a problem in themselves. When deciding on quality of service (QoS) settings to optimize a Voice over IP (VOIP) system, are you impacting another mission-critical system? Is YouTube video downloading important to a business operation or can you lower its priority? Do you have to manually tweak and then stress test these applications to see how they interact?

As new applications emerge and the business develops new expectations of network performance, maintaining the network, troubleshooting problems, and new installations can be significant time and budget burdens.   

Security in all things 

Security has to be a part of every conversation now, and the resolution of our network challenges is no exception. The perimeter firewall is a centerpiece of current network security strategies. Particularly in regulated industries with compliance requirements, the business needs to have control over their firewall to keep rules and monitors up to snuff. Network solutions can interfere with existing firewalls and potentially provide a new attack vector. 

The Uptime Reality 

Bigleaf Networks was built with all of these concerns in mind. Our SD-WAN platform allows clients to seamlessly use multiple ISPs for higher reliability and performance of their network making them more reliable than any one ISP by itslef.

In the course of our business, we have a window into the reliability of the internet. In a recent month, all the circuits that our clients used averaged 92.5 percent reliability. That is not measuring just major outages but also moments when throughput, errors, or jitter is preventing the internet from being usable. 

Our data also shows the solution: with Bigleaf  implemented, uptime at the client location was 99.88 percent.  

Bringing a business-critical SaaS application into the office is exciting but scary. There are no guarantees in this world, but using the right SD-WAN solution means that, the next time someone transposes a couple numbers on a BGP table, your operation is more likely to stay up and running.