question

Julio Toledo avatar image
Julio Toledo asked ·

Batch retrieving message attachments

We are retrieving user message log for 6 month period containing as many as 2,000 records and hundreds of message attachment URIs (mpegs and pdfs).

Our follow-up attempts to retrieve actual message attachments in a loop is causing rate exceed limit errors.

Does the Batch Request GET method support fetching actual binary text files ( i.e.in blocks of 50)?

If not, is there another batch operation we can use to overcome rate exceed limit and accelerate the retrieval of large quantities of attachment binaries?

message-storeattachmentsbatch requests
1 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

benjamin-dean avatar image
benjamin-dean answered ·
Batch requests are counted as a single API request regardless of the total number of items requested in the batch.

According to our documentation about batch requests (following sentence is URL), the Message Store should support batch requests do.  https://developers.ringcentral.com/api-docs/latest/index.html#!#BatchRequests.html

HOWEVER, I do not see this referenced in the section of Message Store about Attachments:  https://developers.ringcentral.com/api-docs/latest/index.html#!#RefGetMessageAttachment

You could try by sending a small batch of four or five items and see if it works, but I think you will receive an HTTP 400 response.

Instead I would recommend one of the following:

A: Create additional admin users (service users essentially) and execute multiple instances of this application in server-side code using each of these users to fetch the data.
B: Submit a Developer Support case to request that your Application's Usage Plan be reviewed for increase considerations.



Knowing you can use batch requests to obtain the list, and to get the individual Message Info items, you can pre-fetch all the data you know you'll need to get the attachment information.

Using your numbers from above (and doing some rough math):

2000 records * 300 attachments per record === 600,000 attachments (roughly)

The Basic Usage Plans permit 40 requests per minute: 40 * 60 (1 hour) * 24 (1 day) * 7 (1 week) === 403,200

This means you should be able to fetch all 600,000+ attachments in under two weeks (if you ALWAYS honor the rate limiting and do not get throttled).

Is that not sufficient (considering you have not attempted to download these attachments for at least the past 6 months)?
Share
1 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Julio Toledo avatar image
Julio Toledo answered ·
Hi Benjamin,

Thanks for your response.

Fortunately not all message records in the message store contain attachments, as a large percentage of them are SMS text messages :)

Actual figures (per user) are approximately 2,000 messages (or about 7 pages of message store records at 300 per page), and of those, roughly 25% (or about 500) of them contain pdf and mpeg attachments. So if my math is correct, it should take about 12-15 minutes to retrieve all 500 attachments.

The use case is for local caching of message store (on device) upon first time login. Following the first mass retrieval each device will continue to update its local message repository differentially. However, thanks to ubiquity, users will sign in and out of multiple devices, and they expect their data to follow them, so message store will most likely need to be re-fetched multiple times (once per device upon first login).

Even if we were to cut off message store caching in half (3 months), that would still mean retrieving about 250 attachments...

We will experiment with your recommendations.
Share
1 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

benjamin-dean avatar image
benjamin-dean answered ·
Let me know how things go and if you need further assistance.
Share
1 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Julio Toledo avatar image
Julio Toledo answered ·
Thanks Benjamin,

I do have a related question about rate limits. I am told all responses to all endpoints include headers that contain rate limit metadata. However I have never seen such headers in any of the API responses we are getting, nor are they included in the API Explorer when you test endpoints there.

Where can I find these elusive headers?
Share
1 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

benjamin-dean avatar image
benjamin-dean answered ·
I had not noticed that the API Explorer does not show these new response headers. I have created a ticket to remedy this in the future.

There are four (4) different response headers returned by the RingCentral API:
  • X-Rate-Limit-Group (string)
    Provides you with the human-readable name which you can reference in the Developer Portal's Rate Limit section for your application for a particular API resource's limits

  • X-Rate-Limit-LImit (integer, fixed number)
    Although a little redundant, this is the header which defines the actual count of requests which can be made based upon the X-Rate-Limit-Group during the X-Rate-Limit-Window value (in seconds)

  • X-Rate-Limit-Remaining (integer, decremented count)
    This header is an approximate value of the number of calls you have remaining in the current limit-measuring time window

  • X-Rate-Limit-Window (integer, in seconds)
    This header defines the length of time the maximum number of requests (as defined by the X-Rate-Limit-Limit header indicates) can be executed before your requests will be throttled
Share
1 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Julio Toledo avatar image
Julio Toledo answered ·
Right, I am aware of these response headers, I just have never seen them in any of the API responses that we get for our app.

Our RingCentral responses look identical to the API Explorer responses (no headers).
Share
1 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

benjamin-dean avatar image
benjamin-dean answered ·
Well...that's a problem, isn't it. :)

Could you do me a favor and submit a Developer Support case ( https://developers.ringcentral.com/api/support-cases/create) and request that your account is inspected and resolved as to why your API responses are missing the X-Rate-Limit-* headers?
Share
1 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Julio Toledo avatar image
Julio Toledo answered ·
Will do.

I am going to guess our API responses may be missing the headers for the same/similar reason that your API Explorer is also missing the response headers :)

Share
1 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

John Wang avatar image
John Wang answered ·
The X-Rate-Limit-* headers are generated for both API calls and API Explorer calls, however, without the proper inspection you will not see them.

While API Explorer API calls will return the X-Rate-Limit-* headers, it seems that the API Explorer UI does not display them which we will look into. You can view the response headers using Google Chrome's Network Inspection of the API Explorer 'account/~/extension/~/message-store' API call as shown in the screen shot below:



Additionally, when I make an API call to 'account/~/extension/~/message-store', I get the following X-Rate-Limit-* headers (lowercased by Ruby) using the Ruby SDK ( https://github.com/grokify/ringcentral-sdk-ruby/) using "pp res.headers":

 "x-rate-limit-group"=>"light",
 "x-rate-limit-limit"=>"50",
 "x-rate-limit-remaining"=>"49",
 "x-rate-limit-window"=>"60"

How are you making API calls and to which endpoints where you are not seeing the X-Rate-Limit-* response headers?

If you create a support case, please reference this post.

Share
1 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Julio Toledo avatar image
Julio Toledo answered ·
We're using php.

I created a support case Friday. Will add this post reference to it.

Support case 04731530
10 comments Share
1 |1000 characters needed characters left characters exceeded

Up to 8 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.

Using the PHP SDK ( https://github.com/ringcentral/ringcentral-php), I'm able to get access to the X-Rate-Limit-* headers for the message-store endpoint with the following code. Can you give it a try?

$sdk = new RingCentral\SDK\SDK('appKey', 'appSecret', RingCentral\SDK\SDK::SERVER_SANDBOX); $sdk->platform()->login('username', 'extension', 'password'); $res = $sdk->platform()->get('/account/~/extension/~/message-store'); var_dump($res->response()->getHeaders()); echo $res->response()->getHeader('X-Rate-Limit-Group')[0] . PHP_EOL; echo $res->response()->getHeader('X-Rate-Limit-Remaining')[0] . PHP_EOL;
0 Likes 0 · ·
Thanks. We will try this and get back to you.
0 Likes 0 · ·
How are things going please?
0 Likes 0 · ·
Trying this, will know if we succeeded tomorrow, but knowing Rate Limit and Remaining Limit is only half the battle.

Seems we will actually need a rate increase, which we have already requested. Especially for message log /SMS log retrieval, which are in the Medium Rate category, for two reasons:
  1. Our app is not authenticating as Admin(s) to retrieve message logs by proxy, each user is responsible for retrieving their own message store. I don't think the scheme of using running multiple instances of the app will work under the same (unprivileged) user account.
  2. Our app is not capable of subscribing to SMS threads, so we are polling the message log instead.
0 Likes 0 · ·
Regarding your app:

  1. Rate limiting is per app and user. So if your app is running for each user, the rate limit will apply separately for each user, not across all users. For example, if you have a rate of 40 requests per minute and have 40 users, each user will have 10 requests per minute. So if you needed to retrieve 40 messages each for 10 users, you only need a rate of 40 per minute, not 400 per minute.
  2. How often are you polling? To subscribe to SMS threads, you need to subscribe to message-store events, e.g. '. When a new SMS message, you will receive events for the message-store. Have you tried this yet?

Here's the link for subscriptions which shows the message-store event filter:

https://developer.ringcentral.com/api-docs/latest/index.html#!#RefCreateSubscription
0 Likes 0 · ·
Show more comments

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 10 attachments (including images) can be used with a maximum of 1.0 MiB each and 10.0 MiB total.