I am planning to deploy a small project on Google Cloud Functions and is using the Google Cloud Storage as well.
I notice that instead of using the os module to call the credential, I need to call from the service account file.
e.g.
credentials = service_account.Credentials.from_service_account_file('xx.json')
storage_client = storage.Client(credentials=credentials)
Then, the errorI　dont' have much experence to tell which point shall I check.
I have do the from google.oauth2 import service_account and from google.cloud import storage and put the credential json key under the cloud functions' temp.
I wonder which process or details have I missed?
Do I need to manually activate the Oauth permision on somewhere or something?
Any pointers? Welcome for any advices and comments.
Related
I have a Firebase (node.js) cloud function that pulls in some data from my app's Firestore database and builds some static content for the web. I'd like that same cloud function to deploy the static content to Firebase hosting via the Firebase Hosting API, creating a static portion of my site with user generated content.
I understand the general flow thanks to the somewhat clear walkthrough, but am stuck on the first step: getting an access token to call the API. Obviously I'm not going to insecurely put my service account key in the cloud function itself, so the example in the walkthrough doesn't apply. And as I understand it, Firebase cloud functions are already associated with a service account, so presumably there's some way to get an access token to call other Google Cloud services from a cloud function.
So how do I get an access token to call the hosting API from a Cloud Function?
There are some red flags that make me think this isn't possible. For example, all of the uses cases in the walkthrough allude to other server environments, as opposed to Google Cloud environments. And yet, this use case is the third bullet in the use case list in the walkthrough.
I've searched extensively here and elsewhere for some guidance, but aren't finding anything. There are some older questions about accessing hosted files from a cloud function that aren't relevant. This promising question from 5 years ago about this exact use case only has dead ends.
You can use the google-auth-library package in Cloud Functions to a get a token as shown below:
import { GoogleAuth } from "google-auth-library";
const token = await new GoogleAuth({
scopes: ["https://www.googleapis.com/auth/cloud-platform"],
}).getAccessToken();
If you use Firebase Admin SDK in the Cloud Functions, then you can get an access token of the default service account as shown below (do ensure the service account has required permissions):
import { initializeApp } from "firebase-admin/app";
const admin = initializeApp();
const token = await admin.options.credential?.getAccessToken();
// ^ Google OAuth2 access token object used to authenticate with Firebase services.
I need to build an API layer to expose a set of files stored within a Private Google Cloud Storage bucket, using Python. I may be using FastAPI for this.
The API would most likely be hosted as GCP Run application.
Could I get some guidance on the best practices for this?
My thinking was, from within the Cloud function to return a FastAPI File Response
e.g. return FileResponse(some_file_path)
Is this a valid approach?
If you have file of "moderate" size, you can load the file from Cloud Storage and serve it directly with your service (Cloud Run, or Cloud Functions, or other).
In case of large file, above a few Mb, it's recommended to use signedURL. On the client request, you generate a signed request and you reply it. For a better experience, you can set the HTTP code 307 (temporary redirect) to indicate to the client to query the new (signed) URL
I have a Google Cloud Function. I also have a web application. I want to authenticate requests to the cloud function by using a service account.
I have the json key file.
I know that I have to follow https://cloud.google.com/functions/docs/securing/authenticating#service-to-function. But that is leading me to an IAP page that does not apply to google cloud functions.
Another similar instructions are found in https://developers.google.com/identity/protocols/oauth2/service-account
But if I am following the python library code, I end up with the sample code there :
import googleapiclient.discovery
sqladmin = googleapiclient.discovery.build('sqladmin', 'v1beta3', credentials=credentials)
response = sqladmin.instances().list(project='exciting-example-123').execute()
This does not directly relate to invoking a cloud function.
This question's answer somewhat deals with my requirement but is using a Call API which is only suitable for testing.
Also, I want to expose this API to multiple applications using another tech like .net. So I believe the best option for me will be to use the HTTP method (given on the same page):
https://developers.google.com/identity/protocols/oauth2/service-account#httprest
But whatever I do I am unable to get the signature right.
Any help to get this sorted will be highly appreciated as I am stuck on this for the past few days.
You can use the Google auth library like this
from google.oauth2.id_token import fetch_id_token
from google.auth.transport import requests
audience="my_audience"
r = requests.Request()
token=fetch_id_token(r,audience)
print(token)
The fetch_id_token method will use the default credentials
The service account key file defined in the environment variable GOOGLE_APPLICATION_CREDENTIALS
The service account loaded in the Google Cloud environment
For now, I followed this answer in PHP
In the claims section, I removed the scope. Instead added a claim of target_audience.
"target_audience" => "google-function-http-trigger"
the cloud function http trigger will look like https://us-central1-test-project-name.cloudfunctions.net/function-name",
This will give the required assertion key.
Then I follow https://developers.google.com/identity/protocols/oauth2/service-account#httprest to get the id_token
Then with the id_token as the bearer token we can call the cloud function.
please note that the token expires depending on the time set in the "exp" claim. Once expired you have to redo the steps to generate the new id_token
I want to authenticate requests to the cloud function by using a service account.
I am not sure I understand the context correctly, but I would try to assign a roles/cloudfunctions.invoker IAM role to that service account (which is used to run your code in the web application) - see Cloud Functions IAM Roles .
In that case a code under that service account "Can invoke an HTTP function using its public URL"
I reckon no json keys are required in this case.
I'm trying to download a Google doc to PDF or Sheet to XLS given an ID programmatically from the CLI.
Steps I've tried so far:
Contact support, but can't see a (?) help icon
Google for 10 minutes... I think Google Drive API does this (not sure)
Enable the Google Drive API
Signed up for a GCP project
Navigated thought the UI to enable the API
Trying the GET API results in 400 Invalid field selection using the fields for the ID of the document
I'm a bit stuck now and I am not sure how to proceed. Any suggestions?
Warning: hopefully
informative wall of text ahead! I've also uploaded the full Jupyter Notebook for you to clone and run here since, as you've realized, putting this sort of stuff together can be challenging.
Since we're going to be exporting files via the google drive API, we need credentials for that scope as detailed in https://developers.google.com/drive/api/v3/reference/files/export#auth.
However, first we need to choose an authentication method as detailed in https://developers.google.com/identity/protocols/oauth2#scenarios.
Since you mentioned creating a GCP project, I assume you're interested in using a GCP service account
as detailed in https://developers.google.com/identity/protocols/oauth2#serviceaccount
You can create a service account at https://console.developers.google.com/apis/credentials
or as explained in https://developers.google.com/identity/protocols/oauth2/service-account#creatinganaccount
Make sure to enable domain-wide-delegation for that service account while creating it and grant it https://www.googleapis.com/auth/drive scope under https://admin.google.com/ac/owl/domainwidedelegation since you otherwise won't be able to impersonate other users, including yourself, and download their files.
We then use the SERVICE_ACCOUNT_FILE we just downloaded and the SCOPES we defined to create a Credentials object.
However, you'll need to first install the Python bindings for the Google API as per https://developers.google.com/drive/api/v3/quickstart/python (pip3 install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib)
With that, the following should be enough to authenticate to the API:
from googleapiclient.discovery import build
from google.oauth2 import service_account
SCOPES = ['https://www.googleapis.com/auth/drive']
SERVICE_ACCOUNT_FILE = 'credentials.json'
credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE,
scopes=SCOPES)
# Remember, you must have created credentials.json with domain-wide delegation!
credentials = credentials.with_subject('user#example.com')
# We then build a drive_v3 service using the credentials we just created
service = build('drive', 'v3', credentials=credentials)
We can access the files resource as shown in https://developers.google.com/drive/api/v3/reference/files/get and request the metadata of a file to which user#example.com has access https://docs.google.com/document/d/fileId/edit. In your case fileId=141g8UkQfdMQSTfIn475gHj1ezZVV16f5ONDxpWrrvts.
files = service.files()
print(service.files().get(fileId='1U3eMevKxTwDxzvOBUsqa36zvwBzKPVYOFgy3k_9vxb8').execute())
{'kind': 'drive#file', 'id':
'1U3eMevKxTwDxzvOBUsqa36zvwBzKPVYOFgy3k_9vxb8', 'name': 'empty',
'mimeType': 'application/vnd.google-apps.document'}
We access the files resource again but this time to export the file as detailed in
https://developers.google.com/resources/api-libraries/documentation/drive/v3/python/latest/drive_v3.files.html#export
This could also be achieved using https://developers.google.com/drive/api/v3/manage-downloads.
Valid MIME types are listed in https://developers.google.com/drive/api/v3/ref-export-formats.
fconr = files.export(fileId='1U3eMevKxTwDxzvOBUsqa36zvwBzKPVYOFgy3k_9vxb8',
mimeType='application/vnd.openxmlformats-officedocument.wordprocessingml.document')
fcont = fconr.execute()
print('{}...'.format(fcont[:10]))
file = open("/tmp/sample.doc", "wb")
file.write(fcont)
file.close()
b'MN\xc30\x10\x85O\xc0\x1d"'...
As you can see, fcont contains a binary blob that corresponds to the document and of which I'm showing the first 10 bytes. Finally, the blob is saved to sample.doc.
ls -alh1 /tmp/sample.doc
-rw-rw-r-- 1 jdsalaro jdsalaro 6,0K Jan 20 23:38 /tmp/sample.doc
As mentioned above, I encourage you to experiment with the Jupyter notebook once you've created the service account with domain-wide delegation, have saved it to credentials.json and have granted it the https://www.googleapis.com/auth/drive scope.
I'm trying to use the what Google terms a 'Domain-wide delegation' service account: https://developers.google.com/admin-sdk/directory/v1/guides/delegation
The specific API I'm trying to access with this delegation is: https://developers.google.com/apps-script/api/
Here's the code:
from google.oauth2 import service_account
import googleapiclient.discovery
import json
import os
SCOPES = ['https://www.googleapis.com/auth/script.projects', 'https://www.googleapis.com/auth/drive']
SERVICE_KEY = json.loads(os.environ['SERVICE_KEY'])
credentials = service_account.Credentials.from_service_account_info(SERVICE_KEY, scopes=SCOPES)
delegated_credentials = credentials.with_subject('fred.bloggs#my-gapps-domain.com')
script = googleapiclient.discovery.build('script', 'v1', credentials=delegated_credentials)
response = script.projects().get(scriptId='<myscriptId>').execute()
print json.dumps(response)
This fails with:
google.auth.exceptions.RefreshError: ('unauthorized_client: Client is unauthorized to retrieve access tokens using this method.', u'{\n "error" : "unauthorized_client",\n "error_description" : "Client is unauthorized to retrieve access tokens using this method."\n}')
I'm pretty sure I've followed all the steps at https://developers.google.com/api-client-library/python/auth/service-accounts, including authorizing the 'https://www.googleapis.com/auth/script.projects' scope with the client ID of the service account json key I downloaded.
Note, I was able to successfully get this particular snippet to work by skipping with_subject, and going in to the Script dashboard as the user, and 'sharing' the script project.
Unfortunately though that still doesn't allow upload a new set of files (as 'sharing' doesn't give the ability to delete). It does at least confirm my calling code is correct, albeit not authenticating properly with the json service key.
To clarify:
The script in question is what I believe is termed 'standalone' (not a web app)
It's owned by a bot user I've setup just like a regular GSuite user (as I didn't want scripts in regular user's Google Drives)
The script started in a Google Cloud Project that seemed to be automatically created, and with 'No organisation'. I then created a new project manually within the organisation, and moved the script to that project.
There's an official Google Apps Script client now, so I asked there too https://github.com/google/clasp/issues/225#issuecomment-400174500 - although they use the Javascript API (via Typescript), the principles should be the same.