Skip to main content
Question

Ringcentral outage -   Things I have learned about RingCentral


Forum|alt.badge.img
  1. Ringcentral is not as redundant as they said they are. 
  2. There is no way to put a message on your main lines for an outage.
  3. There is no way to forward numbers to a cell phone or to another call que system outside of RingCentral.  
  4. When major downtime - calls to ringcentral support is down as well since they are on the same system.
  5. Major lack of communication other than will let you know something in an hour over and over.

6 replies

  • New Participant
  • 3 replies
  • January 29, 2025

What I would say about this is that if you have the ability to keep your phone numbers within your control (do not port your numbers to RC) and use Remote Call Forwarding (or Customer Redirect Service), which is done at your carrier level BY YOU, you can then redirect your calls to another number at anytime via an automated system.  That outage caused a lot of business disruption for many of us and everyone is having to rethink their backup contingency plans (which included RingEX, and those services were also down).   


  • New Participant
  • 1 reply
  • January 29, 2025

I haven’t seen any update from RC regarding an RCA for the outage.  Does anyone know if one was sent?


  • New Participant
  • 3 replies
  • January 29, 2025

I just had a meeting with my RC Support and she told me it was due to the scheduled maintenance. She is supposed to email me the RCA this afternoon. 
 


  • New Participant
  • 3 replies
  • January 29, 2025

They now have the RCA available for this outage - although I am not sure why they didn’t provide it when it was completed.  I just received it from my Customer Success Manager - it’s dated 1/25/25 - today is 1/29. . .


Joe Cache
Forum|alt.badge.img
  • Brainy
  • 150 replies
  • January 29, 2025

Here’s the line of BS I received:

25 January 2025
A message from our Chief Innovation Officer & CMO on RingCentral’s Service Interruption
We sincerely apologize for the recent RingCentral service interruption that took place on January
22, 2025. Events like this are incredibly rare, and reliability is something we take very seriously.
We know you count on the reliability of our system, and we let you down.
RingCentral’s service is fully restored, and we have confirmed that no security breaches or data
leaks occurred. As mentioned in the message from our CEO, we are committed to full
transparency and taking corrective action and steps to ensure this does not happen again.
Please see the enclosed Root Cause Analysis (RCA) that details the findings and mitigations
pertaining to this incident.
The trust you put in RingCentral is not taken lightly, and we understand the impact this
interruption has caused. On behalf of myself and the entire RingCentral team, I want to again
extend our deepest apologies.


 

Preliminary Root Cause Analysis
Incident Summary
A portion of RingCentral customers, primarily in North America, experienced intermittent inbound and
outbound call failures and drops. This issue began on January 22, 2025, at 10:13 AM PT and was resolved at 4:58 PM PT. The total impact duration was 6 hours and 45 minutes.

Customer Impact
During the incident, some RingCentral customers experienced intermittent service availability issues with inbound and outbound calls, including failures and drops, across the RingCentral Phone and contact center products. While some calls were completed successfully, the issue impacted customers across the network. We have confirmed that no security breaches or data leaks occurred.

Root Cause
The issue stemmed from a planned change during a regular maintenance window to optimize call routing.
Unfortunately, this change triggered a call loop between multiple network components. This led to rapid call flooding within our network, compromising our existing redundancy mechanisms, and causing intermittent call failures impacting a portion of our customers. This behavior was unexpected, and therefore, we had to deploy multiple troubleshooting methods to identify the loop source and mitigate the issue.


Steps were taken to roll back the changes, shifting traffic between redundant data centers, and
performing re-starts of impacted components. Despite the rollback, an unexpected domino effect in the network prevented the rollback from working completely and required additional steps to resolve the call loops. The need to restore multiple affected components contributed to the time needed to recover.

RingCentral continued monitoring the system for stability and complete mitigation. Following the
monitoring period, RingCentral confirmed system stability and return to full availability, at which point the incident was deemed resolved.

Preventative Measures
The RingCentral Operations team is conducting an in-depth audit of our change management procedures and implementing process improvements. Immediate measures include a freeze of all non-critical changes, a review of Standard Operating Procedure (SOP) to consider additional steps in planning and execution stages, and a review of the customer “incident status” communications process.


Further preventative measures to improve recovery time include improving loop detection mechanisms across internal components, implementing additional fault isolation, reducing the failure domain of the telephony components, and continuing audits of internal playbooks.

 


 

“Planned change” - in the middle of the day?!?  Really?!?  Or did someone insert the maintenance with 10am rather than 10pm?

We are definately looking at backup solutions, and/or ways to get our DIDs brought back in house and just pointed to the underlying carrier.  (which could be shunted at a moments notice)

So… we’ll see how this all pans out.

 

Good luck.

 

 


  • New Participant
  • 4 replies
  • January 29, 2025

That was my thought as well on the line “planned change during a regular maintenance window”. 

What sort planned maintenance happens literally in the middle of the work week?   It happened around 1pm on a Wednesday EST.   Anyone doing maintenance at that time that might affect key systems should be fired for even attempting it.

I hope this is a major wake-up call for RingCentral but judging from the minimal feedback we have received since the outage, I’ll believe it when I see it.


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings