Today, we began receiving reports from customers unable to log calls using App Connect. This issue is affecting all CRMs supported by App Connect. The root cause appears to be a recent change made to our networking layer to protect the RingCentral Platform from denial of service attacks. This problem is unrelated to version 1.3.6, which was released last night.
Important Information
- The RingCentral operations team is working diligently to roll back the change causing this call logging disruption.
- The App Connect team is scheduling an emergency patch release (version 1.3.7) tonight to further mitigate the issue.
- All calls that could not be logged due to this disruption will be logged once the systems are restored. Calls will be logged automatically if you have Automatic Call Logging enabled. Alternatively, they can be logged manually.
- We will continue to update this post as we learn more.
We sincerely apologize for this inconvenience. We understand that you rely on RingCentral and App Connect to operate your business, and disruptions like this can erode trust. We take your trust seriously and deeply regret this incident. Thank you for your patience and understanding.
Update 2/11/2025, 10:07PM PST - we just entered a maintenance window in which the team will be deploying a fix to the network configuration error. We hope that this combined with our emergency patch will greatly alleviate the call logging issues.
Update 2/11/2025, 11:26PM PST - we released version 1.3.7 which focuses on optimizing the call log synchronization process to increase reliability, improve performance and reduce load.
Update 2/14/2025, 3:27PM PST - the RIngCentral operations team deployed a final fix for this around 9:30PM PST on 2/13/2025. If you are still experiencing problems, please let us know.
So what exactly is going on?
For the nerds and technical people out there, let me explain what is happening.
Our network operations team recently implemented additional safeguards to protect our platform from potential threats, such as denial-of-service attacks. These safeguards are positioned in front of our API gateway, which already has its own rate-limiting mechanisms to regulate how often a single user or app can call the API.
However, our load balancer—positioned ahead of the API gateway—does not have the same granularity in identifying individual users or applications. This means that if multiple users within the same company access App Connect behind a shared firewall, the load balancer perceives all of this traffic collectively and may incorrectly flag it as excessive, triggering access restrictions.
This is not how the system is intended to function, and we recognize the disruption this causes. Our team is actively fine-tuning the load balancer to better align with the API gateway’s protections. However, due to a recent outage at RingCentral, we are proceeding with extra caution, and all network changes must be implemented during designated maintenance windows. As a result, adjustments are being made overnight, with results assessed the following day.
To mitigate this issue in the short term, the App Connect engineering team is working to optimize API calls by reducing their frequency and refining the types of calls being made to improve efficiency. These optimizations are being rolled out progressively as we fine-tune the system. Additionally, the team is developing a more comprehensive long-term plan that may involve significant architectural changes to enhance performance.
We understand that this issue is frustrating, and we appreciate your patience. Resolving this is our top priority, and we will continue to provide updates as we make progress.
