My app is occasionally (once per day) running a bulk insert of around 1,000 files. After a handful of inserts I start getting 403 rate limit responses. Since my app does the inserts sequentially, my attempted insert rate is never higher than 1 per second.
I've checked that I have billing enabled and that my quota limits are 100+ per second, so I don't understand why I'm getting throttled so aggressively. The consequence is that the insert is taking over an hour which isn't a great advert for Drive :-(
Seems the answer is that Drive will allow up to 30 or so inserts before rejecting with 403 errors. The precise figure, and the rate at which the rate limit is de-restricted are not made public. See also 403 rate limit on insert sometimes succeeds
You need to implement exponential backoff as Google describes in their documentation.
You can change the rate in the api console. Set it to a larger value like 10000/sec.
Related
As the title states, I'm trying to create a local copy of our entire google drive, we currently are using it as a file storage service which is obviously not the best use-case, but to migrate else where I of course need to get all the files, the entire google drive is around 800gb~.
I am using rclone specifically the copy command to copy the files FROM google drive TO the local server, however I am constantly running into user Rate Limit errors.
I am using a google service account to authenticate this as well, which I believe should provide more usage limits.
2021/11/22 07:39:50 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User
Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may
consider re-evaluating expected per-user traffic to the API and adjust project quota
limits accordingly. You may monitor aggregate quota usage and adjust limits in the API
Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?
project=, userRateLimitExceeded)
But I don't really understand since according to my usage it is not even coming close, I am just wondering what exactly can I do to either increase my rate limit (even if that means paying) or is there some sort of solution to this issue? Thanks
Error 403: User
Rate Limit Exceeded. Rate of requests for user exceed configured project quota.
User rate limit is the number of requests your user is making per second. its basically flood protection. You are flooding the server. It is unclear how google calculates this, beyond the 100 requests per user per second. If you are getting the error there is really nothing you can do besides slow down your code. Its also unclear from your question how you are running these requests.
If you could include the code we could see how the requests are being preformed. How ever as you state you are using something called rclone so there is no way of knowing how that works.
Your only option would be to slow your code down if you have any control over that though this third party application. If not you may want to contact the owner of the product for direction as to how to fix it.
i currently have a google web app that takes a value then checks it against 14,0000 cells in a master google sheet. i've had the idea to break the function down into 14 functions that run simultaneously(each function searching 1000) therefore cutting the runtime from 6 seconds to below 1. The issue i am concerned about is the limit of 30 simultaneous functions running. this web app will be used by more than one person at a time so it needs to be able to do more than 30. My idea is to create an account for each user with a copy of the web app for each account, so that the simultaneous executions will stay around 14 for each user. so my question is this, Does google sheets have a limit to the amount of simultaneous read/write? any help is greatly appreciated.
If you can implement this I think there are no limits.
This version of the Google Sheets API has a limit of 500 requests per
100 seconds per project, and 100 requests per 100 seconds per user.
Limits for reads and writes are tracked separately. There is no daily
usage limit.
https://developers.google.com/sheets/api/limits
scenario: MySQL database with a million entries, containing Youtube channel IDs and subscribers counts. Now I would like to update the subscribers counts periodically (maybe once a week).
I am wondering if such large requests are even possible with the API? And if so, what would be the most efficient way to get the subscribers count of a million channels? And if not so, can you think of a work-around?
Thank you very much!
I think you may want to consider doing them in batches you should check the Google developer console see what your quota is for the API currently. If there is a pencil icon next to the quota you can increase it.
Also checking the YouTube Data API (v3) - Quota Calculator might help you decide how many to run each day.
I am creating a games comparison website and would like to get Amazon prices included within it. The problem I am facing is using their API to get the prices for the 25,000 products I already have.
I am currently using the ItemLookup from Amazons API and have it working to retrieve the price, however after about 10 results I get an error saying 'You are submitting requests too quickly. Please retry your requests at a slower rate'.
What is the best way to slow down the request rate?
Thanks,
If your application is trying to submit requests that exceed the maximum request limit for your account, you may receive error messages from Product Advertising API. The request limit for each account is calculated based on revenue performance. Each account used to access the Product Advertising API is allowed an initial usage limit of 1 request per second. Each account will receive an additional 1 request per second (up to a maximum of 10) for every $4,600 of shipped item revenue driven in a trailing 30-day period (about $0.11 per minute).
From Amazon API Docs
If you're just planning on running this once, then simply sleep for a second in between requests.
If this is something you're planning on running more frequently it'd probably be worth optimising it more by making sure that the length of time it takes the query to return is taken off that sleep (so, if my API query takes 200ms to come back, we only sleep for 800ms)
Since it only says that after 10 results you should check how many results you can get. If it always appears after 10 fast request you could use
wait(500)
or some more ms. If its only after 10 times, you could build a loop and do this every 9th request.
when your request A lot of repetition.
then you can create a cache every day clear context.
or Contact the aws purchase authorization
I went through the same problem even if I put 1 or more seconds delay.
I believe when you begin to make too much requests with only one second delay, Amazon doesn't like that and thinks you're a spammer.
You'll have to generate another key pair (and use it when making further requests) and put a delay of 1.1 second to be able to make fast requests again.
This worked for me.
Does anyone know if the limit touches an entire day or a period of 24 hour starting from the first request you do and until tomorrow at the same hour in the day?
Example:
I make 2500 request today from 14h00 to 19h00.
Result:
Tomorrow at 2h00AM I will be able to make more request?
OR
I have to wait until 19h00 tomorrow before I can make more request?
Thank you.
According to the documentation, the usage limits are :
Users of the free API: 2,500 requests per 24 hour period.
Maps for Business customers: 100,000 requests per 24 hour period.
We are speaking about a 24 hour period, so in your example, if you make 2500 requests from 2PM to 7PM, you won't be able to do any request since tomorrow 7PM.
I found another page where they talk about the rate and daily limit.
Usage Limits for Google Maps API Web Services
Sending too many requests per day. Sending requests too fast, i.e. too
many requests per second. Sending requests too fast for too long or
otherwise abusing the web service.
If you exceed the usage limits you will get an OVER_QUERY_LIMIT status
code as a response.
This means that the web service will stop providing normal responses
and switch to returning only status code OVER_QUERY_LIMIT until more
usage is allowed again. This can happen:
Within a few seconds, if the error was received because your
application sent too many requests per second.
Some time in the next 24 hours, if the error was received because your application sent too many requests per day. The time of day at
which the daily quota for a service is reset varies between customers
and for each API, and can change over time.
Upon receiving a response with status code OVER_QUERY_LIMIT, your
application should determine which usage limit has been exceeded. This
can be done by pausing for 2 seconds and resending the same request.
If status code is still OVER_QUERY_LIMIT, your application is sending
too many requests per day. Otherwise, your application is sending too
many requests per second.
On the same page there is also a video.
:-)
Maps API for Business: Using the v3 Geocoding API
Update: The limit is now 5000 queries per day for the free tier.