Download a file from Google Drive using google-api-ruby-client - google-drive-api

I try to download files from a directory on Google Disk.
The code, mostly copied from official quickstart guide, works fine:
# ... code from official quickstart tutorial...
# Initialize the API
service = Google::Apis::DriveV3::DriveService.new
service.client_options.application_name = APPLICATION_NAME
service.authorization = authorize
# now the real deal
response = service.list_files(q: "'0ByJUN4GMe_2jODVJVVpmRE1VTDg' in parents and trashed != true",
page_size: 100,
fields: 'nextPageToken, files(id, name)')
puts 'Files:'
puts 'No files found' if response.files.empty?
response.files.each do |file|
puts "#{file.name} (#{file.id})"
# content = service.get_file(file.id, download_dest: StringIO.new)
end
The output looks fine:
Files:
k.h264 (0B4D93ILRdf51Sk40UzBoYmZKMTQ)
output_file.h264 (0B4D93ILRdf51V1RGUDFIWFQ5cG8)
test.mp4 (0B4D93ILRdf51dWpoZWdBV3l4WjQ)
test2.mp4 (0B4D93ILRdf51ZmN4ZGlwZjBvR2M)
test3.mp4 (0B4D93ILRdf51ZXo0WnVfdVBjTlk)
12.mp4 (0ByJUN4GMe_2jRzEwS1FWTnVkX00)
01.mp4 (0ByJUN4GMe_2jSlRWVEw4a1gxa2s)
01.mp4 (0ByJUN4GMe_2jOFpPMW9YNjJuY2M)
But once I uncomment content = service.get_file(file.id, download_dest: StringIO.new), I get a lot of errors:
Files:
k.h264 (0B4D93ILRdf51Sk40UzBoYmZKMTQ)
/Users/mvasin/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/google-api-client-0.9.15/lib/google/apis/core/http_command.rb:211:in `check_status': Invalid request (Google::Apis::ClientError)
[...lots of other 'from' stuff...]
from /Users/mvasin/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/google-api-client-0.9.15/generated/google/apis/drive_v3/service.rb:772:in `get_file'
from quickstart.rb:56:in `block in <main>'
from quickstart.rb:54:in `each'
from quickstart.rb:54:in `<main>'
But that's the way it should work according to ruby section in "download files" official tutorial.
I also tried content = service.get_file(file.id, download_dest: "/tmp/#{file.name}"), but it failed with the same set of errors.
UPDATE: Here are my findings. If you start with Google Drive API Ruby quick start tutorial, and want make it download something,
1) change scope to not just let your script read meatadata, but read files contents as well:
SCOPE = Google::Apis::DriveV3::AUTH_DRIVE_METADATA_READONLY
to at least
SCOPE = Google::Apis::DriveV3::AUTH_DRIVE_READONLY
2) Filter out google docs files, because you can't download them this way, you have to convert them. To filer them:
2.1) Add mime_type to fileds set:
response = service.list_files(page_size: 10, fields: 'nextPageToken, files(id, name, mime_type)')
2.2) and in the final loop where you print files' ids and names, put something like
service.get_file(file.id, download_dest: "/your/path/#{file.name}") unless file.mime_type.include? "application/vnd.google-apps"

From the error that you got, it says that your request is invalid. So make sure that your request there are correct. Here is the documentation on how to download files using Ruby(just click the Ruby on the example to view the ruby code.)
Take NOTE: Downloading the file requires the user to have at least
read access. Additionally, your app must be authorized with a
scope
that allows reading of file content.
For more information, check these threads:
How to download file from google drive api
A Ruby library to read/write files

Related

Google API to create/update files on 'Shared with me' folders

I have been trying to use the Google API to create files on a folder that's been shared with me by another user (I made sure I have edit permissions on it). When I was using the files.create module with supportsAllDrives=True I got the following error message:
{
"errorMessage": "<HttpError 404 when requesting https://www.googleapis.com/upload/drive/v3/files?supportsTeamDrives=true&alt=json&uploadType=multipart returned "File not found: 1aLcUoiiI36mbCt7ZzWoHr8RN1nIPlPg7.". Details: "[{'domain': 'global', 'reason': 'notFound', 'message': 'File not found: 1aLcUoiiI36mbCt7ZzWoHr8RN1nIPlPg7.', 'locationType': 'parameter', 'location': 'fileId'}]">",
"errorType": "HttpError",
"requestId": "fc549b9e-9590-4ab4-8aaa-f5cea87ba4b6",
"stackTrace": [
" File "/var/task/lambda_function.py", line 154, in lambda_handler\n upload_file(service, download_path, file_name, file_name, folder_id, 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')\n",
" File "/var/task/lambda_function.py", line 78, in upload_file\n file = service.files().create(\n",
" File "/opt/python/googleapiclient/_helpers.py", line 131, in positional_wrapper\n return wrapped(*args, **kwargs)\n",
" File "/opt/python/googleapiclient/http.py", line 937, in execute\n raise HttpError(resp, content, uri=self.uri)\n"
]
}
After a bit of digging in, I found that 'Shared Drives' is different from 'Shared with me' and all the APIs I found so far apply to the 'Shared Drives' only. The supportsTeamDrives=True has been deprecated and I was not able to find a related replacement parameter in the docs. There is a parameter sharedWithMe=True for the file.list api and I'm not sure how I can use this in my code because file.create doesn't see the folderID for a 'Shared with me' folder anyway. Any suggestions are appreciated in advance!
My current code:
def upload_file(service, file_name_with_path, file_name, description, folder_id, mime_type):
media_body = MediaFileUpload(file_name_with_path, mimetype=mime_type)
body = {
'name': file_name,
'title': file_name,
'description': description,
'mimeType': mime_type,
'parents': [folder_id]
}
file = service.files().create(
supportsAllDrives=True,
supportsTeamDrives=True,
body=body,
media_body=media_body).execute()
Modified answer to include more details:
You are correct 'Shared Drives' are different from 'Shared With Me'. First off, you need to get the ID from the shared with you folder, for this you can use files:list. To upload files to that folder or any type of folder you can use the modified code below:
from __future__ import print_function
import pickle
import os.path
from googleapiclient.http import MediaFileUpload
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
from google.oauth2 import credentials, service_account
# Scopes required by this endpoint -> https://developers.google.com/drive/api/v3/reference/files/create
SCOPES = ['https://www.googleapis.com/auth/drive']
"""
To upload/create a file in to a 'Shared with me' folder this script has the following configured:
1. Project:
* Create project
* Enable the Google Workspace API the service account will be using: https://developers.google.com/workspace/guides/create-project
2.Consent screen:
* Configure the consent screen for the application
* Create credentials for your service account depending on the type of application to be used with https://developers.google.com/workspace/guides/create-credentials#create_a_service_account
Once your Service Account is created you are taken back to the credentials list (https://console.cloud.google.com/apis/credential) click on the created Service Account, next click on ‘Advanced settings’ and copy your client ID
3. Scopes
* Collect the scopes needed for your service account/application
https://developers.google.com/identity/protocols/oauth2/scopes
4. Grant access to user data to a service account in Google Workspace https://admin.google.com/ac/owl/domainwidedelegation
* In the "Client ID" field, paste the client ID from your service account
* In the "OAuth Scopes" field, enter a comma-delimited list of the scopes required by your application. This is the same set of scopes you defined when configuring the OAuth consent screen.
* Click Authorize.
5. In your code you need to impersonate the account the folder was shared with, if it was your account, you add your account here:
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
delegated_creds = credentials.with_subject('user#domain.info')
"""
def main():
SERVICE_ACCOUNT_FILE = 'drive.json' #Service Account credentials from Step 2
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
delegated_creds = credentials.with_subject('user#domain.xyz')
service = build('drive', 'v3', credentials=delegated_creds)
media = MediaFileUpload(
'xfiles.jpg',
mimetype='image/jpeg',
resumable=True
)
request = service.files().create(
media_body=media,
body={'name': 'xfile new pic', 'parents': ['1Gb0BH1NFz30eau8SbwMgXYXDjTTITByE']} #In here 1Gb0BH1NFz3xxxxxxxxxxx is the 'Shared With ME'FolderID to upload this file to
)
response = None
while response is None:
status, response = request.next_chunk()
if status:
print("Uploaded %d%%." % int(status.progress() * 100))
print("Upload Complete!")
if __name__ == '__main__':
main()
Where:
parents is the ID of the folder shared with you.
See here for more documentation details
After a chat with a Google Workspace API specialist, turns out there is no API available to perform the above task. For clarity, refer the picture where my target folder lies.
Difference between 'Shared Drive' and 'Shared with me' (image)
Here's the response from the Support Agent:
I reviewed your code and everything was done perfectly, so I spoke to
our Drive Specialists, and they have explained to me that "Shared with
me" it's more than anything a label, and because you are not the owner
of the file, (like you would be if they were in "My Drive" )nor the
co-owner (if they were located in "Shared Drive") it does not allow
you to use any type of API in order to automate file creation or
deletion or anything for that matter.
In this case you can either make a copy on your Drive and automate it
there, and just update it every now and then in the file that was
shared with you, or just ask the user to move it to the "Shared Drive"
and access it from there.
I confess I'm a little disappointed that there is no API way to add/delete/edit in another user's folder in spite of having permissions to do so. My understanding as a developer is that the CLI is the ultimate most powerful way to interact with any service. GUI comes second to CLI, it's just a more visually appealing medium. Often times, when we are not able to perform a task using the GUI, we turn to CLI and manage high granularity and precision.
But this was a completely upside down scenario! I'm failing to understand how I'm able to access the 'shared folder' and make adds and deletes through the GUI but unable to do the same using a script. I understand now that 'Shared with me' is just a label and not a 'location' for me to access the folder but surely I would have assumed there was another API way to access a folder that belonged to another user (using the person's username/ID for identification, folder path as target, verifying if I have permissions to make said changes for authentication, returning an error if I don't, lastly executing the API).
If someone's able to explain to me if there is a specific reason why this is not made available to end users, I would love to learn about it please.
EDIT
I'm a bit late posting the solution here, but the issue turned out to be that the google workspace service account that was being used by my API did not have write permissions to the Shared Drive I was trying to query. Once the service account was given the required edit permissions, my code worked perfectly.

Is there a way to bulk subscribe to all subreddits in a particular list programmatically (there is no built-in way to do this)

Currently the list is return-separated and "
(break)" separated as well, but of course it could be in any other format such as csv. or whatever.
Hoping someone here could help me. I scoured google to try and find a solution but couldn't for the life of me, I must be missing something in my search! I'm surprised that nobody else would like this functionality.
here is a naive solution with praw, it assumes you have a text file containing the subreddits you wish to subscribe to (separate lines for each subreddit) and a custom application added to your reddit account:
import praw
reddit = praw.Reddit(
user_agent="mass sub",
# visit https://old.reddit.com/prefs/apps/ to add a new script
# choose http://localhost:8080 as a random and unused callback url
# fill in the correct credentials
client_id="",
client_secret="",
username="",
password=""
)
file1 = open('./some_txt_file_of_subreddits', 'r')
Lines = file1.readlines()
for line in Lines:
print("Line: {}".format(line.strip()))
reddit.subreddit(line.strip()).subscribe()
if you need to generate a list of subreddits from an existing account you can use this bookmarklet
The bookmarklet script below should be run on
https://www.reddit.com/subreddits/mine/
javascript:$('body').replaceWith('<body>'+$('.subscription-box').find('li').find('a.title').map((_, d) => $(d).text()).get().join("<br>")+'</body>');javascript.void()
where the output can be saved to a text file

Getting "The provided image is in an unsupported format" error when trying to insert an image in a Google Slides presentation

I am trying to insert an image (PNG) in a Google Slide presentation using the Slides API. I do this by first uploading the image to the user's Drive, obtaining the url, passing that along to the Slide API via the correct request and then deleting the image file.
What used to work as of a few weeks ago:
image_url = '%s&access_token=%s' % (
drive_service.files().get_media(fileId=image_file_id).uri,
creds.token)
However, there have been changes to the Drive API, such that URLS constructed this way no longer work.
I am having difficulty figuring out the new correct URL to use here. The options as per the doc that describes the change are:
Use webContentLink -- Downloads
Use webViewLink -- View
Use exportLinks -- Export
I use code that looks like this to get these links:
upload = drive_service.files().create(
body={'name': 'My Image File'},
media_body=media_body,
fields='webContentLink, id, webViewLink').execute()
image_url = upload.get('webContentLink')
I have tried both #1 and #2 and get the following error:
"Invalid requests[0].createImage: The provided image is in an unsupported format."
I have also been receiving the following error intermittently:
"Invalid requests[0].createImage: Access to the provided image was forbidden."
I verified that I am able to download / view the image from the URLs generated in #1 and #2. I didn't try #3 since I am not trying to export to a different format.
What would be the best way to go about figuring out the correct URL to use?
From your script, I think that the reason of your issue is due to this. By this, the query parameter of access_token cannot be used. Under this situation, when image_url = '%s&access_token=%s' % (drive_service.files().get_media(fileId=image_file_id).uri,creds.token) is used, the login page is returned. By this, such error occurs. So as a workaround, how about the following flow?
Flow:
Upload a PNG file.
Publicly share the PNG file by creating a permission.
Insert the PNG file to Slides.
Close the shared PNG file by deleting the permission.
When the image file is put to the Slides, even when the permission of file is deleted, the image is not removed from the Slides. This workaround uses this.
Sample script:
For above flow, the sample script of python is as follows. Please set the variables of uploadFilename, presentation_id and pageObjectId
uploadFilename = './sample.png' # Please set the filename with the path.
presentation_id = '###' # Please set the Google Slides ID.
pageObjectId = '###' # Please set the page ID of the Slides.
drive = build('drive', 'v3', credentials=creds)
slides = build('slides', 'v1', credentials=creds)
# 1. Upload a PNG file from local PC
file_metadata = {'name': uploadFilename}
media = MediaFileUpload(uploadFilename, mimetype='image/png')
upload = drive.files().create(body=file_metadata, media_body=media, fields='webContentLink, id, webViewLink').execute()
fileId = upload.get('id')
url = upload.get('webContentLink')
# 2. Share publicly the uploaded PNG file by creating permissions.
drive.permissions().create(fileId=fileId, body={'type': 'anyone', 'role': 'reader'}).execute()
# 3. Insert the PNG file to the Slides.
body = {
"requests": [
{
"createImage": {
"url": url,
"elementProperties": {
"pageObjectId": pageObjectId
}
}
}
]
}
slides.presentations().batchUpdate(presentationId=presentation_id, body=body).execute()
# 4. Delete the permissions. By this, the shared PNG file is closed.
drive.permissions().delete(fileId=fileId, permissionId='anyoneWithLink').execute()
Note:
I thought that from your script, you might be using google-api-python-client with python. So I proposed the sample script for python.
In this case, the scopes for using Slides API and Drive API are required. Please be careful this.
In the case of Google Apps Script, you can see the sample script at here.
References:
Upcoming changes to the Google Drive API and Google Picker API
Permissions: create
Permissions: delete
If I misunderstood your question and this was not the direction you want, I apologize.
I was running into the same error even when using the flow involving granting temporary permissions access then removing the permissions after calling .createImage() or .replaceAllShapesWithImage()
I also ran into this error when creating permissions for a folder containing those images: "Invalid requests[0].replaceAllShapesWithImage: Access to the provided image was forbidden." Not sure why the permissions are not propagating to the images...
Following Kos' comment, switching to jpg file type worked for me.
Edit:
It appears I am also required to set the scope to 'https://www.googleapis.com/auth/drive' in order for it to work, which isn't ideal, but is sufficient for now.
Edit 2:
Nevermind it appears to be inconsistent. I am running into the permissions access error again. Deleting my token.pickle does not seem to fix either

Fatfree routing with PHP built-in web server

I'm learning fatfree's route and found it behaves unexpected.
Here is my code in index.php:
$f3 = require_once(dirname(dirname(__FILE__)). '/lib/base.php');
$f3 = \Base::instance();
echo 'received uri: '.$_SERVER['REQUEST_URI'].'<br>';
$f3->route('GET /brew/#count',
function($f3,$params) {
echo $params['count'].' bottles of beer on the wall.';
}
);
$f3->run();
and here is the URL which I access: http://xx.xx.xx.xx:8090/brew/12
I get a 404 error:
received uri: /brew/12
Not Found
HTTP 404 (GET /12)
the strange thing is that the URI in F3 is now "/12" instead of "/brew/12" and I guess this is the issue.
When I check the base.php (3.6.5), $this->hive['BASE'] = "/brew" and $this->hive['PATH'] = "/12".
But if F3 only uses $this->hive['PATH'] to match the predefined route, it won't be able to match them.
If I change the route to:
$f3->route('GET /brew',
and use the URL: http://xx.xx.xx.xx:8090/brew, then the route matches without issue.
In this case, $this->hive['BASE'] = "" and $this->hive['PATH'] = "/brew". If F3 compares the $this->hive['PATH'] with predefined route, they match each other.
BTW, I'm using PHP's built-in web server and since $_SERVER['REQUEST_URI'] (which is used by base.php) returns the correct URI, I don't think there is anything wrong with the URL rewrite in my .htrouter.php.
Any idea? What did I miss here?
add the content of .htrouter.php here
<?php
#get the relative URL
$uri = urldecode(parse_url($_SERVER['REQUEST_URI'], PHP_URL_PATH));
#if request to a real file (such as a html, image, js, css) then leave it as it is
if ($uri !== '/' && file_exists(__DIR__ . $uri)) {
return false;
}
#if request virtual URL then pass it to the bootstrap file - index.php
$_GET['_url'] = $_SERVER['REQUEST_URI'];
require_once __DIR__ . './public/index.php';
Your issue is directly related to the way you're using the PHP built-in web server.
As stated in the PHP docs, here's how the server handles requests:
URI requests are served from the current working directory where PHP was started, unless the -t option is used to specify an explicit document root. If a URI request does not specify a file, then either index.php or index.html in the given directory are returned. If neither file exists, the lookup for index.php and index.html will be continued in the parent directory and so on until one is found or the document root has been reached. If an index.php or index.html is found, it is returned and $_SERVER['PATH_INFO'] is set to the trailing part of the URI. Otherwise a 404 response code is returned.
If a PHP file is given on the command line when the web server is started it is treated as a "router" script. The script is run at the start of each HTTP request. If this script returns FALSE, then the requested resource is returned as-is. Otherwise the script's output is returned to the browser.
That means that, by default (without a router script), the web server is doing a pretty good job for routing unexisting URIs to your document root index.php file.
In other words, provided your file structure is like:
lib/
base.php
template.php
etc.
public/
index.php
The following command is enough to start your server and dispatch the requests properly to the framework:
php -S 0.0.0.0:8090 -t public/
Or if you're running the command directly from the public/ folder:
cd public
php -S 0.0.0.0:8090
Beware that the working directory of your application depends on the folder from which you call the command. In order to leverage this value, I strongly advise you to add chdir(__DIR__); at the top of your public/index.php file. This way, all subsequent require calls will be relative to your public/ folder. For ex: $f3 = require('../lib/base.php');
Routing file-style URIs
The built-in server, by default, won't pass unexisting file URIs to your index.php, as stated in:
If a URI request does not specify a file, then either index.php or index.html in the given directory are returned
So if you plan to define some routes with dots, such as:
$f3->route('GET /brew.json','Brew->json');
$f3->route('GET /brew.html','Brew->html');
Then it won't work because PHP won't pass the request to index.php.
In that case, you need to call a custom router, such as the .htrouter.php you were trying to use. The only thing is that your .htrouter.php has obviously been designed for a different framework (F3 doesn't care about $_GET['url'] but cares about $_SERVER['SCRIPT_NAME'].
Here's an exemple of .htrouter.php that should work with F3:
// public directory definition
$public_dir=__DIR__.'/public';
// serve existing files as-is
if (file_exists($public_dir.$_SERVER['REQUEST_URI']))
return FALSE;
// patch SCRIPT_NAME and pass the request to index.php
$_SERVER['SCRIPT_NAME']='index.php';
require($public_dir.'/index.php');
NB: the $public_dir variable should be set accordingly to the location of the .htrouter.php file.
For example if you call:
php -S 0.0.0.0:8090 -t public/ .htrouter.php
it should be $public_dir=__DIR__.'/public'.
But if you call:
cd public
php -S 0.0.0.0:8090 .htrouter.php
it should be $public_dir=__DIR__.
OK, I checked the base.php and found out when f3 calculates the base URI, it uses $_SERVER['SCRIPT_NAME'].
$base='';
if (!$cli)
$base=rtrim($this->fixslashes(
dirname($_SERVER['SCRIPT_NAME'])),'/');
if we have web server directly forward all requests to index.php, then
_SERVER['SCRIPT_NAME'] = /index.php, and in this this case, base is ''.
if we use URL rewriting via .htrouter.php to index.php, then
_SERVER['SCRIPT_NAME'] = /brew/12, and in this this case, base is '/brew' which causes the issue.
Since I'm going to use the URL rewrite, I have to comment out the if statement and make sure base =''.
Thanks xfra35 for providing the clue.
Apache like php router here:
It can url rewrite.
https://github.com/kyesil/QPHP/blob/master/router.php
Usage:
php -S localhost:8081 router.php

Not receiving "webViewLink" in response?

After turning on Google Drive API access from the management console and getting my Client ID keys, I followed the sample code (using Python 2.7) and I am able to insert a folder, set the appropriate permissions (type=anyone,role=reader), and insert a text/html type file into the new folder.
However the JSON file resource objects I receive from executing insert on the drive service have no 'webViewLink' field! There are 'webContentLink' and 'selfLink' fields but 'webViewLink', which is necessary for static HTML publishing, seems to be missing.
Most perplexing. If this feature hasn't been turned on yet or if I need to configure my account settings to allow HTML publishing please let me know. Any other help would be most appreciated ;)
The webViewLink property is only returned for public folders, and not the single files inside such folders. You can use that as the base url to construct links to your files.
The WebViewLink file property can be retrieved by doing something like this:
$file = $service->files->get($file_id, array('fields' => 'webViewLink'));
$web_link_view = $file->getWebViewLink();
OR
$sheetsList = $drive_service->files->listFiles([
'fields' => 'files(id, name, webViewLink, webContentLink)',
]);
$web_link_view = $sheetsList->current()->getWebViewLink();
Pay attention that you should load the file specifying which fields you wanna bring with it (In this case, webViewLink). If you don't do that, only id and name will be available.
If you also need to configure file permissions, you can do something like:
$permissions = new \Google_Service_Drive_Permission();
$permissions->setRole('writer');
$permissions->setType('anyone');
$drive_service->permissions->create($file_id, $permissions);
Possible values for setRole() and setType() can be found here: https://developers.google.com/drive/api/v3/reference/permissions/create