Because this product is new, I am looking forward to develop an app on it so is there any limit on the API usage such as:
upload and download quota
requests per app
etc
To view your allowed quota please create a project in the Google APIs Console. In the "Service" tab, the default quota allowed for each service is indicated.
Currently for the Drive API it reads "Courtesy limit: 1,000,000,000 queries/day". It's a per app quota.
After you've enabled the Drive API you can also set a per user rate limit (by default 1000 req per 100 sec) to prevent one user from depleting your app's quota. That's available in the "Quotas" tab.
There is also a link to request more quota in the "Quotas" tab in case you need more than the default 10M req/day such requests will go through a (light) manual review process.
Also files have per-files playback limit which depends on many factors (Is the file shared publicly or just to your domain/users? Is it a video? An audio file? etc...). These rules are not disclosed at this point unfortunately but for instance a publicly shared video can't be viewed by millions of anonymous sessions per day (use Youtube for that). Nor can an image be used on a high traffic website. Google Drive cannot be used as a Web scale CDN, it is scaled for personal content sharing (you share files with friends/work group/company).
I'm getting a message saying "Sorry, you can't view or download this file at this time.
Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator."
so there IS a limit, just undocumented :e
this is a file only i downloaded about 50 times in the last hour (testing some stuff)
There's no mention of rate limiting on the best practices page, or in the performance tips but—most conspicuously of all—the documentation on handling errors does not contain any errors for going over rate limits.
I seen the free request quota as for Google Drive Api's is.
1,000,000,000 requests/day.
and Default Per-user limit is:(you can increase it)
10 requests/second/user
you can visit this and login with valid account for more information
https://console.developers.google.com/project/bionic-path-686/apiui/apiview/drive/quotas
It's not only about API-Queries .. there is also a Download-Limit, i always ran into .. But Google doesn't publish the allowed amount
There are some types of limits which exists in addition to "Quotas".
Error ex.:
code=403
reason=subscriptionRateLimitExceeded
message=Rate limit exceeded for creating file subscriptions.
Similar rate limits are not well documented and can make some pain to app developers.
Read the documentation and you should find what you're looking for.
"Any type of file can be stored in Drive, up to the user's storage limit or maximum file size of 10GB"
https://developers.google.com/drive/
Related
We have hundreds of users and we occasionally get blocked because we exceed the quota for UrlFetch. Is it possible to increase this limit? Or is there another workaround so we don't get throttled?
If I am not mistaken, it is possible, you would need to associate your script with a standard project from GCP (in case you have not done this yet). What I mean by this is that you need to create a separate project on GCP, then enable the APIs and configure the scopes and everything manually, and then associate it with your script from the script settings (see screenshot).
After all that just go to the project, then IAM & Admin >> Quotas and click "Edit quotas". This is also officially documented here and make sure that billing is enabled for the project.
What you would be doing with all this is submitting a request for the Google team in charge to review it and decide if approve it or not. It can take a couple of days for you to get a response from them.
I can't guarantee that the quota that you need to increase would be available, but at least this is how you can request any quota increase for any script you may have.
If you already have a project from GCP with the Apps Script API enabled, you can just follow the instructions to submit the request without having to follow the other steps.
I have a program where I have to copy about 500,000 files onto google drive to different folders. I use the google drive v3 nodejs api. I issue about 2 uploads per second (every 450ms). After a while, I get ECONNRESET or socket hang up from API.
When I look at the quota on the console.cloud.google.com. I am nowhere near my quota. Why is it failing?
For kicks, I have tried google filestream and it has no problems pushing into the drive under my user account. It's about 5 times faster.
Did anyone run into this problem?
I think your quota per se is not the problem here. This is happening when you're writing too much data within a short time frame. Try to slow it down and try to shard the requests across different user accounts. This should help with the heavy lifting of the many requests you are performing. Also, don't forget to implement exponential backoff for 4xx error retries. My two cents.
This does happen when I call passing a stream. There is no warning in the developers.google.com but there is a warning at their github repository.
You can also upload media by specifying media.body as a Readable stream. This can allow you to upload very large files that cannot fit into memory.
Note: Your readable stream may be unstable. Use at your own risk.
Once I have changed it not to use the streams, I started getting the proper error message such as status code 403, going over your rate limit.
I simply changed my code to use a straight buffer. Buffer is read via fs.readFileSync before the call.
media: {
mimeType: 'text/plain',
body: buf
}
We have a client-server architecture that uses Google Drive for sharing files between the client and the server, without having to actually send them.
The client uses the Google Drive API to get a list of file IDs of all files it wants to share with the server.
The server then downloads the files with the appropriate authorization token.
Server response time is crucial for user experience.
We tried a few approaches:
First, we used the webContentLink. This worked until we started receiving large files from the client. Instead of getting the files' content, we got an html with a warning "exceeds the maximum size that Google can scan". We could not find a header we can use to skip this check.
Second, we switched to the Google API resource URL with the alt=media query param. This works, but we then hit API quota errors (User Rate Limit Exceeded). Since this is server code, it was identified as a single user for all requests.
Then we added the quotaUser param to represent on behalf of which user each request is. We still got many 403 responses.
In addition, we implemented exponential backoff for the failed requests.
We also added a cache for the successful requests.
Our current solution is a combination of the two. Using the webContentLink whenever possible (which appears not to affect the Google API quota). If the response is not as expected, (i.e. an html, wrong size, etc.), we try the Google API resource URL (with exponential backoff).
(Most of the files are small enough to not exceed the scan size limit)
Both client and server uses the same OAuth 2.0 client ID.
Here are my questions:
1. Is it possible to skip the virus scan, so that all files can be downloaded using the webContentLink?
2. Is the size threshold for the virus scan documented? Assuming we know the file size we can then save the round-trip of the first request (using the webContentLink)
3. Is there anything else we can do other than applying for a higher quota?
Is it possible to skip the virus scan, so that all files can be downloaded using the webContentLink?
If it is greater than 25MB it is not possible with webContentLink but since you are using authorized request use files.get with alt=media. Apply appropriate error handling options (which you have done using exponential backoff). The next step would be checking if you code is optimized then after checking and applied recommended optimization and still received Error 403 Limit Exceed, time to apply for a higher quota.
Is the size threshold for the virus scan documented? Assuming we know the file size we can then save the round-trip of the first request (using the webContentLink)
To answer this, you can refer to the Google Drive Help Forum : How can I successfully download large files from google drive without network errors at the most end of the download:
Only files smaller than 25 MB can be scanned for viruses.
Is there anything else we can do other than applying for a higher quota?
You can do the following before applying for a higher quota:
Performance Tips
Drive Platform Best Practices
Handling API Errors
After all optimization is done, the only option is to apply for higher quota limit.
Hope this helps!
for my Google App Engine application I would like to use the Service Account's own Google Drive, i.e. I do not wish to create a "regular" gmail account and grant privileges to my Service Account, nor I want to share files with anyone else, I just wish to use the Drive as a "internal filesystem" for the application, accessed by the application only.
My questions:
1) is such service still provided, i.e. are Service Accounts granted to use their own private Drive, or am I forced to use the "new" Google Cloud Storage?
2) where can I find any relevant & updated information regarding the storage quota for such Drive & possible options for expanding it? I browsed many forums, but all posts I found (including on SO) contains dead links and/or states various information ("I think its 5 GB", "I believe its 15 GB" etc.)
Thanks in advance to anyone who can unravel the mystery :-)
You can use the Drive resource about.get and find all the information related to your Drive.
If you make the call from your service account, you can know what is the total amount of available space returned in the parameter quotaBytesTotal for that account.
I just checked and for me it was 15 GB, so that's likely the same available space you will get.
I am using Google Drive SDK for .NET. Everything is working as expected, except that whenever I get the permission feed for a particular document. I get the Id, kind, name, role, selflink and type fields on permission.
There is no mention of the email address of the user which is stopping me from recognizing by reading the permissions, whether a file has been shared inside the domain or outside of the domain.
I can't use Google Docs API to get the ACL on doc because I am writing an app for users over 200000 users and I will need speed which is provided by Google API Console.
What should I do?
Thanks
You can use the Documents List API to get the email addresses. I'm curious why you believe Drive is faster than Documents List, for most API calls they have a comparable response time.
https://developers.google.com/google-apps/documents-list/#retrieving_the_acl_for_a_document_file_or_collection
Thanks for the idea but can I use Document List API with the Service Accounts provided through API Console?
Why do I believe that Google Drive SDK is better? Because we don't have any control over Document List API where we can set QPS (Query per second) limits or not that I know of.
Secondly, with Document List API when you make request to servers, in the past I had to create a fault tollerent algorithm in such a way that if one request fails second should go after 2 seconds if that fails then make your next request after 5 seconds until 7 second delay.
So, I don't think that Document List API would be a good fit for processing documents over 200000 users everyday unless Google has changed the way their API used to behave?