Bandwidth VoIP Backbone Outage

UPDATED: 21/09/29 14:25:00 MST

Bandwidth.com, a national backbone carrier serving most if not all phone carriers in the USA has been experiencing a service outage since Saturday, 9/25/21. The cause of the outage has been reported to be a cyberattack.

Below are the communications that have been sent out by Phoneware to customers, sorted in order of newest to oldest:

Update email sent out from Phoneware on Wed. 9/29 at 4PM MST:

We have received a positive update from Saddleback Communications regarding the Bandwidth outage. We immediately performed tests and our test calls were 100% successful.

From Saddleback: “Information from the 2PM MST (5PM EDT) Bandwidth update stated that at about 12PM MST (3PM EDT), Bandwidth had implemented mitigation that they could not disclose, which made a significant impact in stabilizing the call issues that we have been experiencing. They stated there were also other non-DDoS attack-related issues that affected call service quality, that were corrected in parallel. Based on this most recent implementation, Bandwidth believes that service has been stabilized across their platform. Encouraging words for sure, but actions will speak louder than words, so only time will tell.. If they have an attack again tomorrow and it is minimized then that will speak volumes of their positive news today.”

We will continue to monitor and test and report status of this issue.

Thank you for your patience and understanding.

Update email sent out from Phoneware on Wed, 9/29:

This morning around 6AM MST, Phoneware testing showed that the problem was still occurring, causing a high percentage of calls to Bandwidth phone numbers to fail to connect.

We continue to test throughout the day and as of noon today Wed 9/29, testing shows a significantly lower call connect failure rate, which is an improvement but still reflects impairment and a partial outage in the Bandwidth network.

We placed a request to port all customer main phone numbers that are served by Bandwidth over to another backbone carrier, Inteliquent. Our goal is for this port to occur tomorrow and we are awaiting confirmation.

Customers whose phone numbers are served by Bandwidth are the most affected. This will include nearly all Phoneware customers outside of the Phoenix metro area, and some customers in the Phoenix area as well. If your business is in the Phoenix area, and you have received updates via text message from Phoneware, this means that some of your phone numbers are with Bandwidth and you should be aware of the impact.

For Phoneware customers in the Phoenix area whose phone numbers are not on Bandwidth, there should be no direct impact, since inbound calls do not route across Bandwidth, and outbound calling from Phoneware now uses alternative carriers. However, you may still experience calling problems because the general communications network across the greater USA is affected. Many other VoIP carriers do not have the variety of upstream carriers for routing calls that Phoneware does. So you may be calling a phone number that uses Bandwidth and it could affect your call, even though everything on the Phoneware side is performing at 100%. This problem is widespread, affecting most if not all phone carriers and most end users are experiencing problems as well.

We hope Bandwidth is able to solve this problem soon, and we will continue to do everything we can to help our customers avoid the impact of this nationwide phone service outage.

Update email sent out from Phoneware on Tue, 9/28:

Latest Update: Bandwidth continued to experience problems with the cyberattack throughout the day today, Tuesday 9/28. They have reported several times throughout the day that they were seeing the effects of the attack subside, but we continue to test and we still see frequent call failures as recently as 3:45PM MST today.

We had an in-depth conference today to review the details of this outage and what actions can be taken to mitigate its impact on our customers. Moving phone numbers away to another carrier has always been an option, but that typically takes two weeks. Our upstream provider Saddleback Communications had a discussion with Bandwidth.com and another major backbone carrier about a way to expedite the porting process and they believe it can be completed in two days due to this emergency.

Therefore, we are proceeding with moving the main phone numbers for all customers who have their phone numbers served by Bandwidth.com over to Inteliquent (alternative backbone carrier).

We hope that Bandwidth.com can resolve their outage before the two days that this porting process will take, but we want to take proactive measures regardless. The porting process will be transparent to our customers and will provide relief.

This attack is beginning to be reported more widely on online news sources. Here are some example articles:

https://www.zdnet.com/article/bandwidth-ceo-confirms-outages-caused-by-ddos-attack/

https://www.bibbvoice.com/2021/09/28/voip-providers-nationwide-targeted-by-ransom-demands/

https://www.newsobserver.com/news/business/article254588122.html

https://www.ozarkradionews.com/local-news/voip-services-are-disrupted-nationwide

https://www.techradar.com/news/ddos-assaults-against-voip-providers-continues

https://www.bleepingcomputer.com/news/security/bandwidthcom-is-latest-victim-of-ddos-attacks-against-voip-providers/

As stated in our earlier update today, this problem is affecting most if not all VoIP carriers, including Microsoft Teams, Google Voice, RingCentral, Nextiva, Zoom, and countless others. This will likely go down in history as the largest phone outage ever experienced in the USA.

The Phoneware and Saddleback networks are performing well, are stable, and are well-secured. However, the effect of this Bandwidth.com outage is widespread. Even when our customers are completely moved off of Bandwidth.com, calling problems may still occur until the outage is completely resolved. Phone calls may still attempt to route through Bandwidth.com as they traverse other carriers to reach their destination.

Again, we regret the impact of this outage and we stress that we are doing everything in our power to help our customers recover. We hope that Bandwidth.com resolves their problem soon and will continue to keep you updated as well as we can.

Update email sent out from Phoneware on Mon, 9/27:

What is happening?
On Saturday 9/25, Sunday 9/26, and Monday 9/27, Bandwidth.com, one of the largest backbone carriers serving nearly all VoiP phone service providers, experienced outages as the result of multiple distributed denial of service (DDoS) cyberattacks. As of this writing at on the morning of Tuesday, 9/28, we are seeing another outage develop and we suspect that Bandwidth.com continues to be the victim of a cyberattack. If you do a web search for “bandwidth outage” you will find many articles and blog posts.

Who is Bandwidth.com?

Bandwidth is the underlying carrier or tandem provider (path that phone calls route through) for a portion of nearly all VOIP carriers’ traffic. In many areas, the only choices are porting phone numbers and routing calls through carriers such as Bandwidth.com, Onvoy, Peerless, and Flowroute. Bandwidth.com is one of the largest of these top-level carriers and is used by nearly all voice over IP phone service providers.

What is a DDoS Cyber Attack?

A distributed denial-of-service (DDoS) attack is a malicious attempt to disrupt the normal traffic of a targeted server, service, or network by overwhelming the target or its surrounding infrastructure with a flood of Internet traffic. DDoS attacks achieve effectiveness by utilizing multiple compromised computer systems as sources of attack traffic. Exploited machines can include computers and other networked resources such as IoT (Internet of things aka smart home) devices.
From a high level, a DDoS attack is like an unexpected traffic jam clogging up the highway, preventing regular traffic from arriving at its destination.

Who does this outage impact?

Most if not all VoIP carriers are affected: Microsoft Teams Voice, Google Voice, Vonage, RingCentral, Nextiva, and most wholesale VoIP service providers have some or all phone numbers on Bandwidth.com and route national and international call traffic through Bandwidth. Phoneware and their upstream carrier, Saddleback Communications, use Bandwidth.com as one of their backbone carriers.

What is the relationship between Phoneware and Bandwidth.com?

Phoneware and their upstream carrier Saddleback Communications use Bandwidth.com to terminate incoming calling for some customers and as a primary outbound call route for all traffic outside of the Phoenix Arizona metro area. Phoneware and Saddleback also have other carriers used for providing inbound and outbound routing. Most local calling service in the Phoenix Metro Area is routed through Lumen, so the effect of the Bandwidth.com outage has been limited. Most services provided to Phoneware customers outside of the Phoenix Metro area are served by Bandwidth.com and are the most affected by the Bandwidth.com outage.

How is this outage affecting Phoneware customers?

On 9/27, customers in the Phoenix Metro area were experiencing intermittent call failures on outbound calls such as connection failure, no audio, and disconnects, mostly to phone numbers outside of the Phoenix area. Some local calling within the Phoenix metro area was also affected. A routing change was made in the afternoon on 9/27 to redirect calls away from Bandwidth to use different carriers, which improved outbound calling to this customers in the Phoenix area.

Customers outside of the Phoenix area whose phone numbers are served by Bandwidth.com are the most severely impacted. Incoming calls will fail intermittently, but outbound calling should be mostly unaffected as these calls could be sent via other carriers.

What is Bandwidth doing to resolve the outage?

The information we are receiving from Bandwidth is indirect and guarded. Their status page reports that they are working to block the attack and resolve the issue. We received the following update from Saddleback Communications last night:

“Bandwidth has been working on mitigation and advised us that they will continue to work through the night to ensure they close any holes in their environment to stop any additional attacks. While that is a valiant effort, there is still a chance that the hackers may disrupt Bandwidth again. Hackers continue to be more sophisticated in finding new ways into networks and all carriers need to be attentive to the threats and fortify their environments. For Saddleback/Phoneware, this has been an ongoing effort since our 2019 DDoS attacks and while we fortified our network after the attacks, we continue to look for vulnerabilities and have budgeted a security audit again this year to ensure we are as secure as we can.

What we cannot tell you is where Bandwidth is in the overall mitigation and prevention for future attacks, other than they are addressing them.“

What is Phoneware doing to help mitigate the effects of the Bandwidth.com outage?

On Monday, 9/27, Phoneware worked closely with Saddleback throughout the day to modify the outbound call routing so that Bandwidth would be used as little as possible and so that outbound calling would flow to a different carrier that was not affected. There was little that could be done to resolve inbound calling problems for those Phoneware customers whose phone numbers are served by Bandwidth.com, other than porting phone numbers away from Bandwidth.com to a different carrier. But typically, this takes 10 business days to complete and requires the active participation of Bandwidth.com, who is currently under attack and therefore limited in their ability to assist with porting numbers to a different carrier. We are assessing all options.

We deeply regret the impact that this outage which is out of our control is having to our customers and we will continue to do everything in our power to help resolve the issue and re-route services away from Bandwidth.