question

Julio Toledo avatar image
Julio Toledo asked benjamin-dean commented

Batch retrieving message attachments

We are retrieving user message log for 6 month period containing as many as 2,000 records and hundreds of message attachment URIs (mpegs and pdfs).


Our follow-up attempts to retrieve actual message attachments in a loop is causing rate exceed limit errors.


Does the Batch Request GET method support fetching actual binary text files (i.e.in blocks of 50)?


If not, is there another batch operation we can use to overcome rate exceed limit and accelerate the retrieval of large quantities of attachment binaries?


errorsrate limits
1 |3000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

benjamin-dean avatar image
benjamin-dean answered
Batch requests are counted as a single API request regardless of the total number of items requested in the batch.

According to our documentation about batch requests (following sentence is URL), the Message Store should support batch requests do.  https://developers.ringcentral.com/api-docs/latest/index.html#!#BatchRequests.html

HOWEVER, I do not see this referenced in the section of Message Store about Attachments:  https://developers.ringcentral.com/api-docs/latest/index.html#!#RefGetMessageAttachment

You could try by sending a small batch of four or five items and see if it works, but I think you will receive an HTTP 400 response.

Instead I would recommend one of the following:

A: Create additional admin users (service users essentially) and execute multiple instances of this application in server-side code using each of these users to fetch the data.
B: Submit a Developer Support case to request that your Application's Usage Plan be reviewed for increase considerations.



Knowing you can use batch requests to obtain the list, and to get the individual Message Info items, you can pre-fetch all the data you know you'll need to get the attachment information.

Using your numbers from above (and doing some rough math):

2000 records * 300 attachments per record === 600,000 attachments (roughly)

The Basic Usage Plans permit 40 requests per minute: 40 * 60 (1 hour) * 24 (1 day) * 7 (1 week) === 403,200

This means you should be able to fetch all 600,000+ attachments in under two weeks (if you ALWAYS honor the rate limiting and do not get throttled).

Is that not sufficient (considering you have not attempted to download these attachments for at least the past 6 months)?
1 |3000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Julio Toledo avatar image
Julio Toledo answered
Hi Benjamin,

Thanks for your response.

Fortunately not all message records in the message store contain attachments, as a large percentage of them are SMS text messages :)

Actual figures (per user) are approximately 2,000 messages (or about 7 pages of message store records at 300 per page), and of those, roughly 25% (or about 500) of them contain pdf and mpeg attachments. So if my math is correct, it should take about 12-15 minutes to retrieve all 500 attachments.

The use case is for local caching of message store (on device) upon first time login. Following the first mass retrieval each device will continue to update its local message repository differentially. However, thanks to ubiquity, users will sign in and out of multiple devices, and they expect their data to follow them, so message store will most likely need to be re-fetched multiple times (once per device upon first login).

Even if we were to cut off message store caching in half (3 months), that would still mean retrieving about 250 attachments...

We will experiment with your recommendations.
1 |3000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

benjamin-dean avatar image
benjamin-dean answered
Let me know how things go and if you need further assistance.
1 |3000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Julio Toledo avatar image
Julio Toledo answered
Thanks Benjamin,

I do have a related question about rate limits. I am told all responses to all endpoints include headers that contain rate limit metadata. However I have never seen such headers in any of the API responses we are getting, nor are they included in the API Explorer when you test endpoints there.

Where can I find these elusive headers?
1 |3000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

benjamin-dean avatar image
benjamin-dean answered
I had not noticed that the API Explorer does not show these new response headers. I have created a ticket to remedy this in the future.

There are four (4) different response headers returned by the RingCentral API:
  • X-Rate-Limit-Group (string)
    Provides you with the human-readable name which you can reference in the Developer Portal's Rate Limit section for your application for a particular API resource's limits

  • X-Rate-Limit-LImit (integer, fixed number)
    Although a little redundant, this is the header which defines the actual count of requests which can be made based upon the X-Rate-Limit-Group during the X-Rate-Limit-Window value (in seconds)

  • X-Rate-Limit-Remaining (integer, decremented count)
    This header is an approximate value of the number of calls you have remaining in the current limit-measuring time window

  • X-Rate-Limit-Window (integer, in seconds)
    This header defines the length of time the maximum number of requests (as defined by the X-Rate-Limit-Limit header indicates) can be executed before your requests will be throttled
1 |3000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Julio Toledo avatar image
Julio Toledo answered
Right, I am aware of these response headers, I just have never seen them in any of the API responses that we get for our app.

Our RingCentral responses look identical to the API Explorer responses (no headers).
1 |3000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

benjamin-dean avatar image
benjamin-dean answered
Well...that's a problem, isn't it. :)

Could you do me a favor and submit a Developer Support case ( https://developers.ringcentral.com/api/support-cases/create) and request that your account is inspected and resolved as to why your API responses are missing the X-Rate-Limit-* headers?
1 |3000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Julio Toledo avatar image
Julio Toledo answered
Will do.

I am going to guess our API responses may be missing the headers for the same/similar reason that your API Explorer is also missing the response headers :)

1 |3000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

John Wang avatar image
John Wang Deactivated answered
The X-Rate-Limit-* headers are generated for both API calls and API Explorer calls, however, without the proper inspection you will not see them.

While API Explorer API calls will return the X-Rate-Limit-* headers, it seems that the API Explorer UI does not display them which we will look into. You can view the response headers using Google Chrome's Network Inspection of the API Explorer 'account/~/extension/~/message-store' API call as shown in the screen shot below:



Additionally, when I make an API call to 'account/~/extension/~/message-store', I get the following X-Rate-Limit-* headers (lowercased by Ruby) using the Ruby SDK ( https://github.com/grokify/ringcentral-sdk-ruby/) using "pp res.headers":

 "x-rate-limit-group"=>"light",
 "x-rate-limit-limit"=>"50",
 "x-rate-limit-remaining"=>"49",
 "x-rate-limit-window"=>"60"

How are you making API calls and to which endpoints where you are not seeing the X-Rate-Limit-* response headers?

If you create a support case, please reference this post.

1 |3000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Julio Toledo avatar image
Julio Toledo answered benjamin-dean commented
We're using php.

I created a support case Friday. Will add this post reference to it.

Support case 04731530
10 comments
1 |3000

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

John Wang avatar image John Wang ♦♦ commented ·
Using the PHP SDK ( https://github.com/ringcentral/ringcentral-php), I'm able to get access to the X-Rate-Limit-* headers for the message-store endpoint with the following code. Can you give it a try?

$sdk = new RingCentral\SDK\SDK('appKey', 'appSecret', RingCentral\SDK\SDK::SERVER_SANDBOX); $sdk->platform()->login('username', 'extension', 'password'); $res = $sdk->platform()->get('/account/~/extension/~/message-store'); var_dump($res->response()->getHeaders()); echo $res->response()->getHeader('X-Rate-Limit-Group')[0] . PHP_EOL; echo $res->response()->getHeader('X-Rate-Limit-Remaining')[0] . PHP_EOL;
0 Likes 0 ·
Julio Toledo avatar image Julio Toledo commented ·
Thanks. We will try this and get back to you.
0 Likes 0 ·
benjamin-dean avatar image benjamin-dean commented ·
How are things going please?
0 Likes 0 ·
Julio Toledo avatar image Julio Toledo commented ·
Trying this, will know if we succeeded tomorrow, but knowing Rate Limit and Remaining Limit is only half the battle.

Seems we will actually need a rate increase, which we have already requested. Especially for message log /SMS log retrieval, which are in the Medium Rate category, for two reasons:
  1. Our app is not authenticating as Admin(s) to retrieve message logs by proxy, each user is responsible for retrieving their own message store. I don't think the scheme of using running multiple instances of the app will work under the same (unprivileged) user account.
  2. Our app is not capable of subscribing to SMS threads, so we are polling the message log instead.
0 Likes 0 ·
John Wang avatar image John Wang ♦♦ commented ·
Regarding your app:

  1. Rate limiting is per app and user. So if your app is running for each user, the rate limit will apply separately for each user, not across all users. For example, if you have a rate of 40 requests per minute and have 40 users, each user will have 10 requests per minute. So if you needed to retrieve 40 messages each for 10 users, you only need a rate of 40 per minute, not 400 per minute.
  2. How often are you polling? To subscribe to SMS threads, you need to subscribe to message-store events, e.g. '. When a new SMS message, you will receive events for the message-store. Have you tried this yet?

Here's the link for subscriptions which shows the message-store event filter:

https://developer.ringcentral.com/api-docs/latest/index.html#!#RefCreateSubscription
0 Likes 0 ·
Show more comments

Developer sandbox tools

Using the RingCentral Phone for Desktop, you can dial or receive test calls, send and receive test SMS or Fax messages in your sandbox environment.

Download RingCentral Phone for Desktop:

Tip: switch to the "sandbox mode" before logging in the app:

  • On MacOS: press "fn + command + f2" keys
  • On Windows: press "Ctrl + F2" keys