I have a client that send me an excel of all his employees with CVs links from google driver,
So I have to download the cv and upload it to our S3 servers, but after maybe 50 files google start returning 403 I have to wait a bit of time then download again.
Is there any way to download all files faster?
Thanks
403 (maybe this one) you are hitting is probably a rate limiting error. These errors are used by google as flood protection. The google drive api is a free api, which means you need to share it with other developers.
Is this really a time critical service?
There really is no way to speed it up, even if you put up multiple services google is going to put on the breaks by detecting your ip address or the user your authenticated with.
Related
I want to download a Google Sheet (and/or Doc, or Colab Notebook) from an "Anyone can View" sharing URL, if the file is newer than my local copy. To do that, I need to find out when the remote file was last modified. Which I thought shouldn't be hard.
There are threads explaining how to do this for regular files on websites that make use of the HTML Last-Modified property, but Google doesn't provide this field in its headers. It provides a Date: but that's just the download date/time that updates every moment.
I see threads about doing this from within the Doc or Sheet itself. My question is not about that. I'm talking about getting the info remotely by running a python script on my local machine.
I see a thread about using the Google Drive API v3, but....is it really necessary to go through all that (e.g. install oauth, register an API key, etc. effectively create an entire Google app *) just to find out when a publicly-available file was last modified? Is there an easier way?
Thanks!
EDIT: * I started down the road of Google Drive API but I find it confusing and overwhelming. It's like they think I'm trying to create an app for general users for the Android Store, instead of just myself. (??)
I'm trying to write a Python program that uses Google Drive to do some custom backup and sync work from my Linux desktop.
I started using the PyDrive2 library, and followed its recommendations on how to sign up for Drive API access. When I run my program, it accesses Drive as I'd expect it to, but my authorization expires after a few hours, which is not going to work for a program that runs in cron. I thought this might be because my app is not verified, so I looked into verification, but it's asking for things like a my application's home page. Mine isn't a web-based application, so I don't have one.
Is there a good way to access Google Drive APIs from a non-web application?
if your access is expireing after only a few hours it sounds like you are not requesting offline access and storing the refresh token for later use.
If you followed Quickstart pyton You may have noticed this section. The users credetinals that being the refresh otken and the access token are stored in token.json for future use by your application.
# The file token.json stores the user's access and refresh tokens, and is
# created automatically when the authorization flow completes for the first
# time.
if os.path.exists('token.json'):
creds = Credentials.from_authorized_user_file('token.json', SCOPES)
I thought this might be because my app is not verified, so I looked into verification, but it's asking for things like a my application's home page.
Your application only really needs to be verified if you are going to have other uses using it. If your only using it yourself then you really only need to remove it from testing phase and put it into production then your refresh tokens will last longer then seven days.
You could try to use a service account but i don't know of anyone who has gotten service account authorization to work with curl.
I have an application that provides protection through encryption to files.
One of my clients requested that I shall add an ability to his Google Drive to protect documents once they are downloaded from his account.
The scenario is as following:
- My client uploads files (documents, excel sheets etc.) into his google drive account.
- My client send share invites to his peers.
- My client peers can view the files online, and they can download the files.
- When my client peers try to downloade the files, I need to capture a download event, protect the file and then continue with the download process.
I searched everywhere but cannot find an answer that is even close to what I need to acheive.
Your'e help is much appreciated.
Unless they are downloading the files through your application, you cannot intercept requests like this.
Your only option is to have them protected in Drive before they are shared/downloaded.
I uploaded a utility in the last few days to google cloud storage.
It's a zip file containing two executables and a readme file.
I tested the download and it worked fine. I then looked into how I could see the download stats and yesterday I enabled logging.
I posted the link to a mailing list this afternoon and clicked it to verify that I had the right link and the download in chrome reports "xxx.zip appears to be malicious".
This did not happen prior to when I enabled logging, but I don't know for sure that is what caused it.
I am using a CNAME alias for the download, and I am a paying google apps customer.
The executables are not malicious in any way. They are simple utilities for doing replacements in text files. They do not access the network at all.
My question is "Why is my zip file being reported as malicious?" and is there any way to remedy this situation?
I looked around for a solution to this problem and I found the following advice:
1) Sign your EXEs. As it turns out, this advice is incorrect. While it has worked for some people, there are people who report that even signed executables are reported as malicious downloads.
2) Use SSL. SSL access is not available for google cloud storage unless you use the commondatastorage.googleapis.com or sandbox.google.com URLs. While this does might work, it doesn't resolve my problem.
3) Use the commondatastorage.googleapis.com URL. This works. The same file using the commondatastorage.googleapis.com url rather than my custom CNAME record does not report that it "appears malicious".
4) Register your site with Google Webmaster Tools. Getting around Chrome's Malicious File Warning According to this stackoverflow entry, the solution is to sign up for Google Webmaster Tools and add your site.
I have tried this one, but it has not made a change just yet. Because this is google cloud storage and not a main site, I added an index.html page, a 404 page, and ran the gsutil commands to enable web configuration within google cloud storage. I added the site to Webmaster Tools and additionally added it to Google Analytics.
I'll give solution 4 a few days to see if it pans out.
It seems like this is more of an issue with Google Chrome and not necessarily Google Cloud Storage. Chrome's methods for identifying malicious files are less than desirable right now.
We're working on an app to sell our music and was wondering if Google Drive can be used as an online storage solution.
The user would complete the transaction on our site, and then authorize us to save the file (or multiple files) to their Google Drive.
The appeal to us is to solve downloading problems via the browser. I believe the Google Drive api returns a successful response when the delivery is complete. If incomplete, we would then either resend or update.
One other requirement is whether we can set permissions to not allow sharing after save (and for that setting to be permanent).
You can do everything you want. The last part about not allowing users to reshare, you can do this if you still own the file, but cannot do it if you have made the user own the file. I am not sure you could ever achieve that - a user can always download a file and share it themselves, whether you are using Drive or your own custom system.