I am creating a games comparison website and would like to get Amazon prices included within it. The problem I am facing is using their API to get the prices for the 25,000 products I already have.
I am currently using the ItemLookup from Amazons API and have it working to retrieve the price, however after about 10 results I get an error saying 'You are submitting requests too quickly. Please retry your requests at a slower rate'.
What is the best way to slow down the request rate?
Thanks,
If your application is trying to submit requests that exceed the maximum request limit for your account, you may receive error messages from Product Advertising API. The request limit for each account is calculated based on revenue performance. Each account used to access the Product Advertising API is allowed an initial usage limit of 1 request per second. Each account will receive an additional 1 request per second (up to a maximum of 10) for every $4,600 of shipped item revenue driven in a trailing 30-day period (about $0.11 per minute).
From Amazon API Docs
If you're just planning on running this once, then simply sleep for a second in between requests.
If this is something you're planning on running more frequently it'd probably be worth optimising it more by making sure that the length of time it takes the query to return is taken off that sleep (so, if my API query takes 200ms to come back, we only sleep for 800ms)
Since it only says that after 10 results you should check how many results you can get. If it always appears after 10 fast request you could use
wait(500)
or some more ms. If its only after 10 times, you could build a loop and do this every 9th request.
when your request A lot of repetition.
then you can create a cache every day clear context.
or Contact the aws purchase authorization
I went through the same problem even if I put 1 or more seconds delay.
I believe when you begin to make too much requests with only one second delay, Amazon doesn't like that and thinks you're a spammer.
You'll have to generate another key pair (and use it when making further requests) and put a delay of 1.1 second to be able to make fast requests again.
This worked for me.
Related
I have an express.js backend server, and I am using MySQL to store the users' information. I want to implement an API rate limit that limits a certain number of requests a user can make in a minute; however, if I store the request count per minute in a database and get the count every time an API request is made, this system can easily be abused. Is there a better way to do this?
How to limit the number of transcations per second on a table in Mysql?
Like to prevent brute force login via an API
As David says, do this on the API. You cannot and should not limit your database. There's no way to distinguish the origin of the query, so you'll just shut down the database for everyone if one person decides to flood it, making a denial-of-service attack easier.
As for a solution there are many examples.
Nginx has a rate-limiting feature built in that can limit requests per interval of time, and is very flexible. This can be focused on particular endpoints, paths, or other criteria, making it easy to protect whatever parts of your system are vulnerable.
You'll also need to block clients that are trying to attack your system. Consider something like fail2ban which can read logs and automatically block source traffic from offenders. Log every failed attempt and this tool can do the rest.
scenario: MySQL database with a million entries, containing Youtube channel IDs and subscribers counts. Now I would like to update the subscribers counts periodically (maybe once a week).
I am wondering if such large requests are even possible with the API? And if so, what would be the most efficient way to get the subscribers count of a million channels? And if not so, can you think of a work-around?
Thank you very much!
I think you may want to consider doing them in batches you should check the Google developer console see what your quota is for the API currently. If there is a pencil icon next to the quota you can increase it.
Also checking the YouTube Data API (v3) - Quota Calculator might help you decide how many to run each day.
Does anyone know if the limit touches an entire day or a period of 24 hour starting from the first request you do and until tomorrow at the same hour in the day?
Example:
I make 2500 request today from 14h00 to 19h00.
Result:
Tomorrow at 2h00AM I will be able to make more request?
OR
I have to wait until 19h00 tomorrow before I can make more request?
Thank you.
According to the documentation, the usage limits are :
Users of the free API: 2,500 requests per 24 hour period.
Maps for Business customers: 100,000 requests per 24 hour period.
We are speaking about a 24 hour period, so in your example, if you make 2500 requests from 2PM to 7PM, you won't be able to do any request since tomorrow 7PM.
I found another page where they talk about the rate and daily limit.
Usage Limits for Google Maps API Web Services
Sending too many requests per day. Sending requests too fast, i.e. too
many requests per second. Sending requests too fast for too long or
otherwise abusing the web service.
If you exceed the usage limits you will get an OVER_QUERY_LIMIT status
code as a response.
This means that the web service will stop providing normal responses
and switch to returning only status code OVER_QUERY_LIMIT until more
usage is allowed again. This can happen:
Within a few seconds, if the error was received because your
application sent too many requests per second.
Some time in the next 24 hours, if the error was received because your application sent too many requests per day. The time of day at
which the daily quota for a service is reset varies between customers
and for each API, and can change over time.
Upon receiving a response with status code OVER_QUERY_LIMIT, your
application should determine which usage limit has been exceeded. This
can be done by pausing for 2 seconds and resending the same request.
If status code is still OVER_QUERY_LIMIT, your application is sending
too many requests per day. Otherwise, your application is sending too
many requests per second.
On the same page there is also a video.
:-)
Maps API for Business: Using the v3 Geocoding API
Update: The limit is now 5000 queries per day for the free tier.
My app is occasionally (once per day) running a bulk insert of around 1,000 files. After a handful of inserts I start getting 403 rate limit responses. Since my app does the inserts sequentially, my attempted insert rate is never higher than 1 per second.
I've checked that I have billing enabled and that my quota limits are 100+ per second, so I don't understand why I'm getting throttled so aggressively. The consequence is that the insert is taking over an hour which isn't a great advert for Drive :-(
Seems the answer is that Drive will allow up to 30 or so inserts before rejecting with 403 errors. The precise figure, and the rate at which the rate limit is de-restricted are not made public. See also 403 rate limit on insert sometimes succeeds
You need to implement exponential backoff as Google describes in their documentation.
You can change the rate in the api console. Set it to a larger value like 10000/sec.