Unable to authenticate google api - google-contacts-api

I have followed everything on this website here. I have also downloaded the .json file with my client ID and client secret and put it in the same folder as my python file. When I run my code, I am able to login, but after I continue to let my google app access my google account, it crashes. I am not receiving any access tokens at all either. Does anyone know why this is happening?
This is the quickstart.py file
from __future__ import print_function
import datetime
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
# If modifying these scopes, delete the file token.json.
SCOPES = ['https://www.googleapis.com/auth/calendar']
def main():
"""Shows basic usage of the Google Calendar API.
Prints the start and name of the next 10 events on the user's calendar.
"""
creds = None
# The file token.json stores the user's access and refresh tokens, and is
# created automatically when the authorization flow completes for the first
# time.
if os.path.exists('token.json'):
creds = Credentials.from_authorized_user_file('token.json', SCOPES)
# If there are no (valid) credentials available, let the user log in.
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
'credentials.json', SCOPES)
creds = flow.run_local_server(port=8080)
# Save the credentials for the next run
with open('token.json', 'w') as token:
token.write(creds.to_json())
service = build('calendar', 'v3', credentials=creds)
# Call the Calendar API
now = datetime.datetime.utcnow().isoformat() + 'Z' # 'Z' indicates UTC time
print('Getting the upcoming 10 events')
events_result = service.events().list(calendarId='primary', timeMin=now,
maxResults=10, singleEvents=True,
orderBy='startTime').execute()
events = events_result.get('items', [])
if not events:
print('No upcoming events found.')
for event in events:
start = event['start'].get('dateTime', event['start'].get('date'))
print(start, event['summary'])
if __name__ == '__main__':
main()

I was used google contacts API to export contacts with nodeJS. I made documentation also where you may find out how you generate access tokens.
How does it work?
The first time you run the sample, it will prompt you to authorize access:
I assume that you got that step and accept/allow button. Now you are so close to generating the access token.
Now,
Copy the code you're given, paste it into the command-line prompt, and press Enter. After this, The file token.json stores the user's access and refresh tokens, and is created automatically when the authorization flow completes for the first time.
For further details see this, maybe it will help you.
google-contacts-api

Related

Google API to create/update files on 'Shared with me' folders

I have been trying to use the Google API to create files on a folder that's been shared with me by another user (I made sure I have edit permissions on it). When I was using the files.create module with supportsAllDrives=True I got the following error message:
{
"errorMessage": "<HttpError 404 when requesting https://www.googleapis.com/upload/drive/v3/files?supportsTeamDrives=true&alt=json&uploadType=multipart returned "File not found: 1aLcUoiiI36mbCt7ZzWoHr8RN1nIPlPg7.". Details: "[{'domain': 'global', 'reason': 'notFound', 'message': 'File not found: 1aLcUoiiI36mbCt7ZzWoHr8RN1nIPlPg7.', 'locationType': 'parameter', 'location': 'fileId'}]">",
"errorType": "HttpError",
"requestId": "fc549b9e-9590-4ab4-8aaa-f5cea87ba4b6",
"stackTrace": [
" File "/var/task/lambda_function.py", line 154, in lambda_handler\n upload_file(service, download_path, file_name, file_name, folder_id, 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')\n",
" File "/var/task/lambda_function.py", line 78, in upload_file\n file = service.files().create(\n",
" File "/opt/python/googleapiclient/_helpers.py", line 131, in positional_wrapper\n return wrapped(*args, **kwargs)\n",
" File "/opt/python/googleapiclient/http.py", line 937, in execute\n raise HttpError(resp, content, uri=self.uri)\n"
]
}
After a bit of digging in, I found that 'Shared Drives' is different from 'Shared with me' and all the APIs I found so far apply to the 'Shared Drives' only. The supportsTeamDrives=True has been deprecated and I was not able to find a related replacement parameter in the docs. There is a parameter sharedWithMe=True for the file.list api and I'm not sure how I can use this in my code because file.create doesn't see the folderID for a 'Shared with me' folder anyway. Any suggestions are appreciated in advance!
My current code:
def upload_file(service, file_name_with_path, file_name, description, folder_id, mime_type):
media_body = MediaFileUpload(file_name_with_path, mimetype=mime_type)
body = {
'name': file_name,
'title': file_name,
'description': description,
'mimeType': mime_type,
'parents': [folder_id]
}
file = service.files().create(
supportsAllDrives=True,
supportsTeamDrives=True,
body=body,
media_body=media_body).execute()
Modified answer to include more details:
You are correct 'Shared Drives' are different from 'Shared With Me'. First off, you need to get the ID from the shared with you folder, for this you can use files:list. To upload files to that folder or any type of folder you can use the modified code below:
from __future__ import print_function
import pickle
import os.path
from googleapiclient.http import MediaFileUpload
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
from google.oauth2 import credentials, service_account
# Scopes required by this endpoint -> https://developers.google.com/drive/api/v3/reference/files/create
SCOPES = ['https://www.googleapis.com/auth/drive']
"""
To upload/create a file in to a 'Shared with me' folder this script has the following configured:
1. Project:
* Create project
* Enable the Google Workspace API the service account will be using: https://developers.google.com/workspace/guides/create-project
2.Consent screen:
* Configure the consent screen for the application
* Create credentials for your service account depending on the type of application to be used with https://developers.google.com/workspace/guides/create-credentials#create_a_service_account
Once your Service Account is created you are taken back to the credentials list (https://console.cloud.google.com/apis/credential) click on the created Service Account, next click on ‘Advanced settings’ and copy your client ID
3. Scopes
* Collect the scopes needed for your service account/application
https://developers.google.com/identity/protocols/oauth2/scopes
4. Grant access to user data to a service account in Google Workspace https://admin.google.com/ac/owl/domainwidedelegation
* In the "Client ID" field, paste the client ID from your service account
* In the "OAuth Scopes" field, enter a comma-delimited list of the scopes required by your application. This is the same set of scopes you defined when configuring the OAuth consent screen.
* Click Authorize.
5. In your code you need to impersonate the account the folder was shared with, if it was your account, you add your account here:
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
delegated_creds = credentials.with_subject('user#domain.info')
"""
def main():
SERVICE_ACCOUNT_FILE = 'drive.json' #Service Account credentials from Step 2
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
delegated_creds = credentials.with_subject('user#domain.xyz')
service = build('drive', 'v3', credentials=delegated_creds)
media = MediaFileUpload(
'xfiles.jpg',
mimetype='image/jpeg',
resumable=True
)
request = service.files().create(
media_body=media,
body={'name': 'xfile new pic', 'parents': ['1Gb0BH1NFz30eau8SbwMgXYXDjTTITByE']} #In here 1Gb0BH1NFz3xxxxxxxxxxx is the 'Shared With ME'FolderID to upload this file to
)
response = None
while response is None:
status, response = request.next_chunk()
if status:
print("Uploaded %d%%." % int(status.progress() * 100))
print("Upload Complete!")
if __name__ == '__main__':
main()
Where:
parents is the ID of the folder shared with you.
See here for more documentation details
After a chat with a Google Workspace API specialist, turns out there is no API available to perform the above task. For clarity, refer the picture where my target folder lies.
Difference between 'Shared Drive' and 'Shared with me' (image)
Here's the response from the Support Agent:
I reviewed your code and everything was done perfectly, so I spoke to
our Drive Specialists, and they have explained to me that "Shared with
me" it's more than anything a label, and because you are not the owner
of the file, (like you would be if they were in "My Drive" )nor the
co-owner (if they were located in "Shared Drive") it does not allow
you to use any type of API in order to automate file creation or
deletion or anything for that matter.
In this case you can either make a copy on your Drive and automate it
there, and just update it every now and then in the file that was
shared with you, or just ask the user to move it to the "Shared Drive"
and access it from there.
I confess I'm a little disappointed that there is no API way to add/delete/edit in another user's folder in spite of having permissions to do so. My understanding as a developer is that the CLI is the ultimate most powerful way to interact with any service. GUI comes second to CLI, it's just a more visually appealing medium. Often times, when we are not able to perform a task using the GUI, we turn to CLI and manage high granularity and precision.
But this was a completely upside down scenario! I'm failing to understand how I'm able to access the 'shared folder' and make adds and deletes through the GUI but unable to do the same using a script. I understand now that 'Shared with me' is just a label and not a 'location' for me to access the folder but surely I would have assumed there was another API way to access a folder that belonged to another user (using the person's username/ID for identification, folder path as target, verifying if I have permissions to make said changes for authentication, returning an error if I don't, lastly executing the API).
If someone's able to explain to me if there is a specific reason why this is not made available to end users, I would love to learn about it please.
EDIT
I'm a bit late posting the solution here, but the issue turned out to be that the google workspace service account that was being used by my API did not have write permissions to the Shared Drive I was trying to query. Once the service account was given the required edit permissions, my code worked perfectly.

Is there any way to track file information (size,name,bucket location,create/update timestamp) in multiple google buckets

I'm able to track the file ingress in a single bucket using below code. But I want to track all the files going into different buckets of my project on Google Cloud..is there any way?
def hello_gcs(event, context):
"""Triggered by a change to a Cloud Storage bucket.
Args:
event (dict): Event payload.
context (google.cloud.functions.Context): Metadata for the event.
"""
import os
from google.cloud import bigquery
BQ = bigquery.Client()
table_id = 'xx.DW_STAGE.Bucket_Monitor'
table = BQ.get_table(table_id)
bucket=event['bucket']
file_nm=event['name']
create_ts=event['timeCreated']
update_ts=event['updated']
size=event['size']
contentType=event['contentType']
crc32c=event['crc32c']
etag=event['etag']
generation=event['generation']
file_id=event['id']
kind=event['kind']
md5Hash=event['md5Hash']
medialink=event['mediaLink']
metageneration=event['metageneration']
selfLink=event['selfLink']
storageClass=event['storageClass']
timeStorageClassUpdated=event['timeStorageClassUpdated']
errors = BQ.insert_rows(table, [(bucket,file_nm,create_ts,update_ts,size,contentType,crc32c,etag,generation,file_id,kind,md5Hash,medialink,metageneration,selfLink,storageClass,timeStorageClassUpdated)]) # Make an API request.
if errors == []:
print("New rows have been added.")
else:
print("Encountered errors while inserting rows: {}".format(errors))
Maybe you will be interested in Pub/Sub. This mechanism is used underneath by Cloud Function Storage triggers (description is here).
For every of your buckets you can create Pub/Sub notification in the same topic (How to create notification you may find here). Every notification can has attributes and payload which contain information about the object which changed.
Such notification can be used as trigger for Cloud Function similar to yours.

Scrape with scrapy using saved html pages

I'm looking to find a way to use scrapy with html pages that I saved on my computer. As far as I am, I got an error :
requests.exceptions.InvalidSchema: No connection adapters were found for 'file:///home/stage/Guillaume/scraper_test/mypage/details.html'
SPIDER_START_URLS = ["file:///home/stage/Guillaume/scraper_test/mypage/details.html"]
I have had great success with using request_fingerprint to inject existing HTML files into HTTPCACHE_DIR (which is almost always .scrapy/httpcache/${spider_name}). Then, turning on the aforementioned http cache middleware which defaults to the file based cache storage, and the "Dummy Policy" which considers the on-disk file authoritative and won't make a network request if it finds the URL in the cache.
I would expect the script would something like (this is just the general idea, and not guaranteed to even run):
import sys
from scrapy.extensions.httpcache import FilesystemCacheStorage
from scrapy.http import Request, HtmlResponse
from scrapy.settings import Settings
# this value is the actual URL from which the on-disk file was saved
# not the "file://" version
url = sys.argv[1]
html_filename = sys.argv[2]
with open(html_filename) as fh:
html_bytes = fh.read()
req = Request(url=url)
resp = HtmlResponse(url=req.url, body=html_bytes, encoding='utf-8', request=req)
settings = Settings()
cache = FilesystemCacheStorage(settings)
spider = None # fill in your Spider class here
cache.store_response(spider, req, resp)

Accessing Google API from a web application

I've been trying for a couple of days now to crack this but have not had any success.
I have a web application that I want to use with Google Drives API.
I want the web application to check if there is an access token it can use and if not redirect to Google so the user can log in and grant access.
Seemingly a simple task but it's driving me mad! I've checked the Google documentation but it all seems to be geared around console applications
Google provides an interface UserService which stores details of the users using the application. If the users is not logged in redirect the user to login page using:
response.sendRedirect(userService.createLoginURL(request.getRequestURI()))
Later or if the user is logged in, redirect him to "Request for Permission" page using:
List<String> scopes = Arrays.asList(PlusScopes.PLUS_LOGIN,PlusScopes.PLUS_ME,PlusScopes.USERINFO_EMAIL,PlusScopes.USERINFO_PROFILE......); // Add/remove scopes as per your requirement
List<String> responseTypes = Arrays.asList("code");
GoogleAuthorizationCodeRequestUrl gAuthCode = new GoogleAuthorizationCodeRequestUrl(Google project client id, redirect url, scopes);
gAuthCode.setAccessType("offline");
gAuthCode.setClientId(Google project client id);
gAuthCode.setResponseTypes(responseTypes);
gAuthCode.setApprovalPrompt("force");
authURl = gAuthCode.toURL().toString();
response.sendRedirect(authURl);
Make sure you add all required scopes of the API methods you will be using. After the user has accepted, you will have to create a servlet with "/oauth2callback" mapping to get the authorization code.
request.getParameter("code")
In the same servlet using the code obtained, get refresh and access token making a rest call.
URL url = new URL("https://www.googleapis.com/oauth2/v3/token");
HttpURLConnection connection= (HttpURLConnection)url.openConnection();
connection.setRequestMethod("post");
connection.setDoInput(true);
connection.setDoOutput(true);
DataOutputStream dw= new DataOutputStream(connection.getOutputStream());
dw.writeBytes("code="+authorizationCode+"&client_id="+CLIENT_ID+"&client_secret="+CLIENT_SECRET+"&redirect_uri="+REDIRECT_URL+"&grant_type=authorization_code");
dw.flush();
dw.close();
InputStream inputStream= connection.getInputStream();
Parse the input stream to get your refresh token and access token and redirect the user to your landing page.
Now you have access token to query your api whose scopes were provided in authorization flow. Also you have a refresh token which can be used to regenerate new access token if the previously issued access token has expired.
You should be able to implement the OAuthHandshake using HTTP requests and a redirect URL to your web application. You can play around with the requests here to see what the headers and responses look like: https://developers.google.com/oauthplayground/
You can store the authorization code and tokens any way you like. You would have your web application refer to these tokens to see if they are expired. For example:
def getTokenFromFile(self):
creds = self.readCredsFromDisk()
# check if token is expired
expiration_time = datetime.datetime.strptime(creds['token_expiry'], '"%Y-%m-%dT%H:%M:%S.%f"')
if expiration_time < datetime.datetime.now():
self.refreshToken()
# reload creds
creds = self.readCredsFromDisk()
return creds['access_token']
I'm writing just a python script that does the handshake and saves the token to a plain text file. Any time the script runs a function to the Google API it will use this function.
The refresh function:
def refreshToken(self):
with open('client_secret.json') as s:
secret = json.load(s)
secret = secret['installed']
creds = self.readCredsFromDisk()
refresh_url = secret['token_uri']
post_data = {'client_id':secret['client_id'],
'client_secret':secret['client_secret'],
'refresh_token':creds['refresh_token'],
'grant_type':'refresh_token'}
headers = {'Content-type':'application/x-www-form-urlencoded'}
(resp, content) = self.http.request(refresh_url,
method='POST',
body=urlencode(post_data),
headers=headers)
content = json.loads(content)
creds['access_token'] = content['access_token']
date = datetime.datetime.now() + datetime.timedelta(seconds=content['expires_in'])
creds['token_expiry'] = json.dumps(date.isoformat())
self.writeCredsToDisk(json.dumps(creds))
You would write a function similar to this to trade the original authorization code and access code following the logic the OAuth Playground shows you.

"Error generating the Discovery document for this api" when trying to build a drive service, starting 2/14/2013

I am intermittently getting this error when calling build on a drive service. I am able to reproduce this with a simple program which has the JSON credentials stored to a file.
#!/usr/bin/python
import httplib2
import sys
from apiclient.discovery import build
from oauth2client.client import Credentials
json_creds = open('creds.txt', 'r').read()
creds = Credentials.new_from_json(json_creds)
http = httplib2.Http()
http = creds.authorize(http)
try:
drive_service = build('drive', 'v2', http=http)
except Exception:
sys.exit(-1)
When I run this in a loop, I am seeing a rather high number of errors, this code in a loop fails 15-25% of the time for me.
i=0; while [ $i -lt 100 ]; do python jsoncred.py || echo FAIL ; i=$(( $i + 1 )); done | grep FAIL | wc -l
Now when I take this same code, and just replace 'drive' by 'oauth2', the code runs without problems
I have confirmed that the OAuth token that I am using is valid and have the correct scopes:
"expires_in": 2258,
"scope": "https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/drive https://www.googleapis.com/auth/userinfo.email",
Looking at my app logs, this seems to have started 2/14/2013 1PM PST. I did not push any new code, so I wonder if this a problem with the API. Is there a bug in the API causing this ?
Google are seeing some reports of increased error rates for the discovery document. Please just retry on 500 error for now, and you should be successful.
One could argue that you should have retry logic for this call anyway, since it is good practice, but the current levels are too high, so, sorry about that.
Update: this should now be fixed.