We are planning on opening a company account on google drive which will be accessible to only company people.
The issue is we want to put several files on our drive and download them programatically. We tried using google drive APIs but the download speed it very low.
Then we also tried wget but that requires that all the files are made public which we cannot do.
Is there any way to use wget with credentials which will allow a file to be downloaded via an URL.
Our typical file size is 50GB.
There is actually a command from wget to specify user and password. Have you tried the following?
wget --user='username' --ask-'password' https://docs.google.com/'ItemType'/Export='DocumentId'
Related
Using Google Takeout usually gives you zip or tar.gz in Google Drive.
Is there any way, by API or any programming methods, to decompress those files and put them back in Drive without downloading them locally?
You can't decompress a file in Drive, but there are some workarounds for that. Using the API you could easily develop a solution that downloads the file, then decompresses it locally and finally uploads it again. As an alternative, you could speed up that process by using a sync folder that allows you to extract the file locally. Feel free to ask any doubts about those approaches.
Sometimes is hard to do just simple stuff with puppeteer. I was wondering if it is possible to use puppeteer to login to a dropbox link with password and then use someting like wget or curl to do the rest. I imagine that I would need to read and then pass on on some sort of access token after the login.
Would this be possible?
(yes, I know that using the dropbox API would perhaps be an easier and more correct solution)
I am not familiar how puppeteer stores cookies, but I am sure you can do that, see references below, and e.g. the Puppeteer API docu on cookies.
Here my 50 cents on wget and/or cURL with access control. In both cases it is possible to load cookies which provide you access to dropbox. Assume you have them stored in myAccessCookies.txt, you can reload them and use them e.g. with wget:
wget -qO- --load-cookies myAccessCookies.txt http://www.example.com/replaceWithDropboxLink
Another hint: To my knowledge, Dropbox allows to share direct, temporary links to files which do not require further authentication, see e.g. https://help.dropbox.com/files-folders/share/set-link-permissions - if this is not a security threat, and you can influence your counter-party to use these, everything is easy.
References and further reading
Superuser: How to download dropbox files using wget
https://www.apharmony.com/software-sagacity/2014/10/using-wget-with-cookies/ on how to use wget with cookies
Download Folder including Subfolder via wget from Dropbox link to Unix Server
How to save cookies and load it in another puppeteer session?
https://stackoverflow.com/a/56515357/6189678 - how to store and reuse cookies (suggestion by V.Kostenko)
I have a pdf link like www.xxx.org/content/a.pdf, and I know that there are many pdf files in www.xxx.org/content/ directory but I don't have the filename list. And When I access www.xxx.org/content/ using browser, it will redirect to www.xxx.org/home.html.
I tried to use wget like "wget -c -r -np -nd --accept=pdf -U NoSuchBrowser/1.0 www.xxx.org/content", but it returns nothing.
So does any know how to download or list all the files in www.xxx.org/content/ directory?
If the site www.xxx.org blocks the listing of files in HTACCESS, you can't do it.
Try to use File Transfer Protocol with FTP path you can download and access all the files from the server. Get the absolute path of of the same URL "www.xxx.org/content/" and create a small utility of ftp server and get the work done.
WARNING: This may be illegal without permission from the website owner. Get permission from the web site first before using a tool like this on a web site. This can create a Denial of Service (DoS) on a web site if not properly configured (or if not able to handle your requests). It can also cost the web site owner money if they have to pay for bandwidth.
You can use tools like dirb or dirbuster to search a web site for folders/files using a wordlist. You can get a wordlist file by searching for a "dictionary file" online.
http://dirb.sourceforge.net/
https://sectools.org/tool/dirbuster/
I use Xampp for localhost. I would like to have my files on localhost synced on different computers. I figure that I could just install Google Drive inte the Xampp directory eg. "localhost/Google Drive" (of course I will have to do this on all computers).
Before I do so I wonder if there would be any disadvantages doing so?
Also I wonder how to get "localhost/Google Drive/some-website/index.php" to work (note the space in "Google Drive")?
The best way to do this is to use a local service that calls your google drive files by url, check the google drive documentation so you can do this integration.
Couldn't wait.
Works like a CHARM! Did not experience any disadvantages.
I solved the "Google Drive space problem" like this:
Quit Google Drive
Move the folder to your "htdocs" folder and rename the folder to eg. "google-drive" (my location and filename is "C:\xampp\htdocs\google-drive")
Restart Google Drive (it will say that folder could not be found)
Choose the option to relink to the new folder.
Since I installed the Google Drive folder in "C:\xampp\htdocs\google-drive" I have the option to work on localhost without syncing files eg. files in "C:\xampp\htdocs\my-uncynked-folder".
I've built nice browsing window which shows all of the pdf files on my (or any user) Google Drive for managing purposes.
What i looking to do is simple, i want to take a pdf file from my google drive (i have all the info related to this file - "downloadUrl","webContentLink" etc) and just copy it to my server (remote).
Any thoughts?
I guess I'm pretty late here, but this may help other people too.
You could try using Grive. Here's a straightforward tutorial: http://xmodulo.com/2013/05/how-to-sync-google-drive-from-the-command-line-on-linux.html
Even if you don't have root access on the server, you can simply build from source, and:
$ mkdir ~/google_drive
$ cd ~/google_drive
$ grive -a
You'll receive an auth URL which you need to paste on your browser and click on "Allow Access" and you're done. Go to the google_drive dir on your server and run grive to sync between your local dir and your GDrive.