Google suggests to retrieve the base url again when needed after 60 minutes after the origin query because the url's expire.
So far, so good. But what if I'm developing a photo gallery and I'm displaying 5000 or them in a grid? Should I query the API again and again? They use a maximum page size of 100 (instead pf 1000 for google drive), so we're starting many requests if that's true.
I'm already caching the photos locally, but when the user scrolls to another section, the url will be expired after one hour.
What is the best solution for that?
Google has a batch request for that use case :
https://developers.google.com/photos/library/reference/rest/v1/mediaItems/batchGet
Anyway the maximum number of items you can request per call is 50, so you should anyway queue some of those requests.
Personally I made it automatic in my code. When the baseUrl i'm loading gives me a 403, it automatically get the mediaitem updated object and retry.
Also you should not cache base urls or generally mediaobjects within different app launches (assuming you're writing an app). You should retain only the mediaItem id and reload it or the entire collection when needed.
Related
I want to ask for help/ideas on the issue I will describe below.
Our iOS app allows users to access their Google Drive files.
We use Changes API (https://developers.google.com/drive/api/v3/reference/changes). The main pre-condition to using this API is to build a local DB that holds the snapshot of the user's Drive file tree and the token. To initially fill the DB we must request the list of all files from user's Drive. Getting the list of all files (with metadata) takes too long for many of our users. This is the issue I want to address.
We request files with the series of Files requests (https://developers.google.com/drive/api/v3/reference/files/list). Most requests are plain files?q=trashed%20%3D%20false.
For example, at my own private Google Drive:
69K files
initial request of all files takes 5+ minutes with my current network speed (Download 527 Mbps, Upload 417 Mbps; ping www.googleapis.com – 40–45 ms)
~150 requests
each request brings information about ~460 files
each request takes around 2-2.5 seconds
Sometimes I observed requests to take up to 6 seconds, which means that getting all files list took 15 minutes at my account.
If I look at the Developer Console, the latency is below 0.1s
Many of our users have Drives far bigger than mine. Standard iOS app user's session is not long enough to complete the initial request. We do save every intermediate page token so that all data received during single app session is not lost if user leaves the app – next session we will keep downloading data from the last saved token. But still there're some cases when our app needs the DB to be filled out with data before starting some operations – in that case our users see "Pending..." progress and they complain that our app is slow.
So, questions:
is it possible to improve the described request speed/latency?
maybe there's some quota that we are missing and it can be changed?
maybe someone can advice a more effective way of getting all files list?
P.S. We could potentially reduce the amount of requests. We have to perform some double checks for Shared with Me folders as we observed that sometimes request of all files doesn't list all files from Shared folders. That's a bit of a side story, and I don't think this will dramatically improve situation for us. I can provide more details on the actual set of requests we perform if necessary.
Are you returning all the fields - I would assume so since the only query param provided is trashed=false as the query param. Do you need all the fields? Can you try to reduce the query to only return the fields you really care about (using a field mask) and see if that improves your performance?
I'm currently using the Google Places API to pull reviews onto a webpage. Everything is working fine except for the Photos of the people leaving reviews. When trying to get the photo of the reviewer, it's returning a 403 Forbidden on every other page load. It seems that there might be a rate limit perhaps?
The problem is I can't find any documentation about rate limits and how to get the picture to display without issue. Am I missing something in the docs?
My API call is this;
https://maps.googleapis.com/maps/api/place/details/json?placeid=PLACE_ID&key=API_KEY
That returns quite a long JSON array (I've cut it down). One of those fields is;
{
"result" : {
"reviews" : [
{
"profile_photo_url" : "//lh5.googleusercontent.com/url/photo.jpg"
}
}
}
}
Like I said, if I refresh a couple of times it'll cause a 403 error for the images get request. Anyway to cache or allow more requests?
I found out why this was happening. It's to do with rate limits on the photo media which is why it was giving a 403 error in the console. The developer docs outline the limits for the media requested.
An excerpt of the docs...
The Google Places API Web Service enforces a default limit of 1,000
free requests per 24 hour period, calculated as the sum of client-side
and server-side requests. If your app exceeds the initial limit, the
app will start failing. You can increase this limit free of charge, up
to 150,000 requests per 24 hour period, by enabling billing on the
Google API Console to verify your identity. A credit card is required
for verification. We ask for your credit card purely to validate your
identity. Your card will not be charged for use of the Google Places
API Web Service.
Best thing to do is to cache the media once it's requested to avoid going over the limit which is especially useful if you're reloading the page lots of times for testing local development changes.
We have a client-server architecture that uses Google Drive for sharing files between the client and the server, without having to actually send them.
The client uses the Google Drive API to get a list of file IDs of all files it wants to share with the server.
The server then downloads the files with the appropriate authorization token.
Server response time is crucial for user experience.
We tried a few approaches:
First, we used the webContentLink. This worked until we started receiving large files from the client. Instead of getting the files' content, we got an html with a warning "exceeds the maximum size that Google can scan". We could not find a header we can use to skip this check.
Second, we switched to the Google API resource URL with the alt=media query param. This works, but we then hit API quota errors (User Rate Limit Exceeded). Since this is server code, it was identified as a single user for all requests.
Then we added the quotaUser param to represent on behalf of which user each request is. We still got many 403 responses.
In addition, we implemented exponential backoff for the failed requests.
We also added a cache for the successful requests.
Our current solution is a combination of the two. Using the webContentLink whenever possible (which appears not to affect the Google API quota). If the response is not as expected, (i.e. an html, wrong size, etc.), we try the Google API resource URL (with exponential backoff).
(Most of the files are small enough to not exceed the scan size limit)
Both client and server uses the same OAuth 2.0 client ID.
Here are my questions:
1. Is it possible to skip the virus scan, so that all files can be downloaded using the webContentLink?
2. Is the size threshold for the virus scan documented? Assuming we know the file size we can then save the round-trip of the first request (using the webContentLink)
3. Is there anything else we can do other than applying for a higher quota?
Is it possible to skip the virus scan, so that all files can be downloaded using the webContentLink?
If it is greater than 25MB it is not possible with webContentLink but since you are using authorized request use files.get with alt=media. Apply appropriate error handling options (which you have done using exponential backoff). The next step would be checking if you code is optimized then after checking and applied recommended optimization and still received Error 403 Limit Exceed, time to apply for a higher quota.
Is the size threshold for the virus scan documented? Assuming we know the file size we can then save the round-trip of the first request (using the webContentLink)
To answer this, you can refer to the Google Drive Help Forum : How can I successfully download large files from google drive without network errors at the most end of the download:
Only files smaller than 25 MB can be scanned for viruses.
Is there anything else we can do other than applying for a higher quota?
You can do the following before applying for a higher quota:
Performance Tips
Drive Platform Best Practices
Handling API Errors
After all optimization is done, the only option is to apply for higher quota limit.
Hope this helps!
Does anyone know if it's possible to dive deeper into our static maps API usage data than we get through the console? We're being charged and I'm trying to make sure that I understand where all the requests are coming from!
First stop was to prevent bots from trying to index static map content but the requests remained high. At 50k pageviews for the whole site per day from visitors I'm just a little suspicious that we'd be hitting 112k static map views per day...
I'm hoping there are ways/strategies to figure out which pages are causing the most views and which user agent strings/IPs.
Just an idea:
Add a serverside script as icon to the StaticMap-URL.
The url of this script(icon) should contain a random parameter to prevent google from caching this script(icon), and additionally
parameters that are populated with the desired details(url of the current page, IP, UserAgent)
The response of the script should be an image that is suitable to be used as a icon for a static map(may be a 1x1 transparent png)
The script may store the supplied parameters in a DB, you should be able to create detailed statistics of the StaticMaps-loads.
I am using Google Drive SDK for .NET. Everything is working as expected, except that whenever I get the permission feed for a particular document. I get the Id, kind, name, role, selflink and type fields on permission.
There is no mention of the email address of the user which is stopping me from recognizing by reading the permissions, whether a file has been shared inside the domain or outside of the domain.
I can't use Google Docs API to get the ACL on doc because I am writing an app for users over 200000 users and I will need speed which is provided by Google API Console.
What should I do?
Thanks
You can use the Documents List API to get the email addresses. I'm curious why you believe Drive is faster than Documents List, for most API calls they have a comparable response time.
https://developers.google.com/google-apps/documents-list/#retrieving_the_acl_for_a_document_file_or_collection
Thanks for the idea but can I use Document List API with the Service Accounts provided through API Console?
Why do I believe that Google Drive SDK is better? Because we don't have any control over Document List API where we can set QPS (Query per second) limits or not that I know of.
Secondly, with Document List API when you make request to servers, in the past I had to create a fault tollerent algorithm in such a way that if one request fails second should go after 2 seconds if that fails then make your next request after 5 seconds until 7 second delay.
So, I don't think that Document List API would be a good fit for processing documents over 200000 users everyday unless Google has changed the way their API used to behave?