How do I append content to the end of a file in Google Drive using the API ?
Do I really have to download the whole thing, then edit the local copy, and then re-upload the whole thing again?
Yes you really have to download edit the file and upload it again. There is no way to programmatically edit a file. Except maybe a spreadsheet but then you would be using the Google sheets API and not the Google drive API.
You can use the drive apis resumable upload with some restrictions:
(I don't quite remember if its true) Minimum bytes uploaded have to be 262144 except for the final upload which "creates" the file, which can contain less
An upload-session expires after one week,
you can set Content-Range to */* if you don't know the final filesize
The file wont show up in google drive in the ui until complete,Upload using a similiar header to the example below, where /* is the final byte length of the file. It has to be one byte less, like in the example below: 262146/262147
I recommend getting a service account for gcp project, you can create a folder in your personal drive and share it with the service account email.
To save some time, because the drive api documentation is not the best, here in "pure" python http requests:
First you have to create the file and get the session_url:
headers = {"Authorization": "Bearer "+myAccesstoken,
"Content-Type": "application/json"}
file_metadata = {
'name': "myFile.txt",
'mimeType': "text/plain",
'parents': [myFolderid],
"uploadType": "resumable"
}
r = requests.post(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable",
headers=headers,
data=json.dumps(file_metadata)
)
session_url=r.headers['Location']
Then you can upload data to it:
headers = {
"Authorization": "Bearer "+myAccesstoken,
"Content-Range": 'bytes 0-262144/*'}
if is_final_data:
headers = {
"Authorization": "Bearer "+myAccestoken,
"Content-Range": 'bytes 262144-262146/262147'}
sd = io.BytesIO()
sd.write(bytes("Wurst", "ASCII"))
sd.seek(0)
r = requests.put(
session_url,
headers=headers,
data=sd
)
To get last uploaded byte position, if you are resuming an upload, send an empty put request only with the session url and authorization headers and read its response headers afterwards.
You can store the session url in a file and resume upload for one week.
Note: You will need something like below, since the access token is only valid for a limited amount of time.
if credentials.access_token_expired:
credentials.refresh(httplib2.Http())
Related
I want to update an existing spreadsheet in a drive folder but have trouble implementing the http request. I followed the documentation and was able to update a spreadsheet but the request body, which I tried to send in JSON, is always converted to CSV. This results in the distribution of the JSON parts into individual cells depending on present commas.
For instance, cell1= "{key1" and cell2= "value1" and so on. However, this prevents me from specifying the style of the sheet and values within the cells.
I found the possibility to send multipart request which, however, results in the same result. Now the first boundary string and the initial information until the first comma are included in the first cell and the rest ist divided according to existing commas.
What I want to do ist send an HTTP request with the body consisting of a JSON-File of specified information for the spreadsheet as described in the Sheets API of Google, but cannot find my current mistake. Even with mimetype set to "application/vnd.google-apps.spreadsheet" the json is always converted to csv.
mimetype "application/vnd.google-apps.spreadsheet"
If the file in question is an actuall google sheets file type. For example the mime type is "application/vnd.google-apps.spreadsheet". Then you should go though the google sheets api to update it. Other wise updating it though google drive you will need to load the file itself into a file stream and then upload it that way. You cant pick and choose what parts are uploaded with drive its all or nothing. Drive doesn't have the power to format things like cells and stuch it just uploads the raw file data.
Mimetype "text/plain"
If the file is in fact a csv file so the mime type is "text/plain" then you can update the text directly. by turning the text into a stream.
You have not said what language you are using so here is my sample for C#. The code is ripped from How to upload to Google Drive API from memory with C#
var uploadString = "Test";
var fileName = "ploadFileString.txt";
// Upload file Metadata
var fileMetadata = new Google.Apis.Drive.v3.Data.File()
{
Name = fileName,
Parents = new List<string>() { "1R_QjyKyvET838G6loFSRu27C-3ASMJJa" } // folder to upload the file to
};
var fsSource = new MemoryStream(Encoding.UTF8.GetBytes(uploadString ?? ""));
string uploadedFileId;
// Create a new file, with metadata and stream.
var request = service.Files.Create(fileMetadata, fsSource, "text/plain");
request.Fields = "*";
var results = await request.UploadAsync(CancellationToken.None);
if (results.Status == UploadStatus.Failed)
{
Console.WriteLine($"Error uploading file: {results.Exception.Message}");
}
// the file id of the new file we created
uploadedFileId = request.ResponseBody?.Id;
I am trying to download a file from Google Drive API v3. I have to do this by finding a file by name.
This is request url to get a file general information:
https://www.googleapis.com/drive/v3/files?q=name+%3D+'fileName.json'
and to download a file I have to use parameter alt=media.
It works but only when I am finding this file by id. I mean:
https://www.googleapis.com/drive/v3/files/3Gp-A4t6455kGGGIGX_gg63454354YD?alt=media
Anyone know how to download a file by name so using this?
https://www.googleapis.com/drive/v3/files?q=name+%3D+'fileName.json'
Answer:
Unfortunately it isn't possible to download a file from Google Drive using the file name in the URL itself.
Reasoning:
As can be seen in the image below, Google Drive supports multiple files having the same name. Each file, instead of being identified exclusively by its name, has a unique ID which tells it apart from other files. As it is possible to have, for example, three files all with the same name, making a request and only referring to the file name doesn't give enough information to identify exactly which file you want to download.
Workaround:
You can still use the filename to build the request, but first you need to make a list request to the Drive API so that you can obtain the specific file ID for the file you wish to download.
I'll assume the file you want to download is called fileName.json as in your question.
First, you'll want to make a list request to the server to obtain the initial file name. The scope you will need for this is:
https://www.googleapis.com/auth/drive.readonly
The request itself is as you placed in the question. Once you have obtained your token, you must make a GET request:
GET https://www.googleapis.com/drive/v3/files?q=name+%3D+'fileName.json'&key=[YOUR API KEY]
You must replace [YOUR API KEY] with your actual API key here. You can obtain a temporary one over at the OAuth Playground.
From this request you will get a JSON reponse of all the files in your Drive with the requested filename. This is an important point - if you only have one file with this filename, you have nothing to worry about and can continue from here. If more than one file exists then the JSON response will contain all these files and so extra code will need to be added here to retrieve the one you want.
Continuing on - the response you get back is of the following form:
{
"incompleteSearch": false,
"files": [
{
"mimeType": "application/json",
"kind": "drive#file",
"id": "<your-file-ID>",
"name": "fileName.json"
}
],
"kind": "drive#fileList"
}
From here, you can start to build your URL.
Building the download URL:
After retrieving the JSON response from the API, you need to extract the File ID to put into a URL. The following example is written in JavaScript/Google Apps Script, but can be built in whichever language suits your needs:
function buildTheUrl() {
var url = "https://www.googleapis.com/drive/v3/files";
var fileName = "fileName.json"
var apiKey = "your-api-key";
var parameters = "?q=name+%3D+'";
var requestUrl = url + parameters + fileName + "&key=" + apiKey;
var response = JSON.parse(UrlFetchApp.fetch(requestUrl).getContentText());
var fileId = response.files.id;
var downloadUrl = "https://www.googleapis.com/drive/v3/files/";
var urlParams = "?alt=media";
return downloadUrl + fileId + urlParams + "&key=" + apiKey;
}
This returns a string which is the download URL for the file. This is a Files: get request that can then be used to download the file in question.
References:
Google OAuth Playground
Google Drive API Files: list
Google Drive API Files: get
Google Apps Script: UrlFetchApp
w3schools: JSON objects
I'm at my wits end here so any pointers much appreciated.
I'm querying the Google Analytics API, converting the response to appropriate JSON format and loading it into bigQuery using using multipart requests using Urlfetchapp. But this leads to me hitting the Urlfetchapp 100MB quota per day very quickly so I'm looking at ways to compress the JSON to GZIP and load that into bigQuery (I considered Google Cloud Storage but I'd have the same problem as saving the data to GCS first requires Urlfetchapp as well so that's why this is a Google Apps Scripts issue).
I've converted the data to blob, then zipped it using Utilities.zip and sent the bytes but after much debugging it turns out that the format is .zip, not .gzip..
Here is the json string created in my Apps Script (NEWLINE_DELIMITED_JSON)
{"ga_accountname":"photome","ga_querycode":"493h3v63078","ga_startdate":"2013-10-23 00:00:00","ga_enddate":"2013-10-23 00:00:00","ga_segmentname":"#_all_visits","ga_segmentexp":"ga:hostname=~dd.com","ga_landingPagePath":"/","ga_pagePath":"/","ga_secondPagePath":"(not set)","ga_source":"(direct)","ga_city":"Boden","ga_keyword":"(not set)","ga_country":"Sweden","ga_pageviews":"1","ga_bounces":"0","ga_visits":"1"}
I've got the rest of the API requests worked out (using uploadType resumable, job configuration sending okay, zipped blob bytes getting uploaded okay but bigQuery says "Input contained no data". Here are my Urlfetchapp parameters.
// Sending job configuration first
var url = 'https://www.googleapis.com/upload/bigquery/v2/projects/' + bqProjectId +'/jobs?uploadType=resumable';
var options = {
'contentType': 'application/json; charset=UTF-8',
'contentLength': newJobSize,
'headers': {
'Accept-Encoding': 'gzip, deflate',
'Accept': 'application/json',
'X-Upload-Content-Length': zipSize,
'X-Upload-Content-Type': 'application/octet-stream'
},
'method' : 'post',
'payload' : jobData,
'oAuthServiceName' : 'bigQuery',
'oAuthUseToken' : 'always'
};
// Sending job data
var url = jobReq.getHeaders().Location;
var options = {
'contentType': 'application/octet-stream',
'contentLength': zipSize,
'contentRange': '0-'+zipSize,
'method' : 'put',
'payload' : zipBytes,
'oAuthServiceName' : 'bigQuery',
'oAuthUseToken' : 'always'
};
What options have I got? I'm fairly new to APIs but can I get Urlfetchapp to compress the payload to GZIP for me?
There isn't any way to work with gzip in Google Apps Scripts right now - the UtilitiesApp.zip() method only uses regular zip compression, not gzip.
Rather than use the UrlFetchApp to form multipart uploads, why not use the BigQuery library that is present in Google Apps Scripts?
var projectId = "Bigquery-Project-Id";
var job = {
configuration: {
load: {
destinationTable: {
projectId: projectId,
datasetId: datasetId,
tableId: tableId
},
sourceFormat: "NEWLINE_DELIMITED_JSON",
writeDisposition: "WRITE_APPEND"
}
}
};
var data = jobData;
job = BigQuery.Jobs.insert(job, projectId, data);
To enable it, you will need to turn BigQuery access on in two places.
First, you need to go to the Resources drop down menu in the Apps UI, and then select Advanced Google Services... . Find Big Query on the list, and toggle the On/Off switch for it.
Before you close the advanced services window, you will need to click on the Google's Developer Console link at the bottom. This will open the developers console for your Google Apps Script project. Find Big Query on the APIs list in the console and enable it.
That's it - from there you can can pass data to the BigQuery API using the BigQuery Apps class, rather than UrlFetchApp.
2020 status
For those viewing the question in 2020, the support for gzip has been added for a while now and is available under Utilities service method gzip() and its corresponding override.
GCF option
The other alternative to using the BigQuery advanced service is to change from UrlFetchApp and Google Apps Script project to cloud functions. From there, one can choose the preferred language to write in and utilize the libraries / packages needed for compressing (for example, NodeJS has a Zlib module out of the box).
References
gzip method reference
Cloud functions reference
I am working on a Java application that will move files and folders to box.com. I am consuming the REST API V2 and was able to upload a single file by making a multpart post to the endpoint: https://upload.box.com/api/2.0/files/content.
Is it possible to upload multiple files to box.com by making a single post call? If so, what would the post call look like?
Here is a code snippet showing how I upload a single file:
Client client = Client.create();
File thefile = new File(PATH_TO_FILE/FILE_NAME.pdf);
WebResource webResource = client.resource("https://upload.box.com/api/2.0/files/content");
FormDataMultiPart form = new FormDataMultiPart();
form.bodyPart(new FileDataBodyPart("filename", thefile, MediaType.APPLICATION_OCTET_STREAM_TYPE));
form.field("filename", "test.pdf");
form.field("parent_id", parentId);
ClientResponse response = webResource.type(MediaType.MULTIPART_FORM_DATA).header(
"Authorization", "Bearer " + getBoxTokenProperty(GRANT_TYPE_ACCESS_VAL)).post(ClientResponse.class, form);
Thanks in advance!
Currently the Box API only supports uploading a single file per request.
I'm trying to understand how the Google API works server side in order to allow me to implement my own type of resumable upload. I understand that I can use the MediaFileUpload or MediaInMemoryUpload mechanism, but I am looking for something much more raw. For example, I want to deliberately upload 1k from a file, then later on (like days later), append another 1k of the file. Obviously not real figures here, but hopefully you get the idea. Well here is where I am with the code:
headers = {
'range': 'bytes=%d-%d' % (
offset,
offset + len(data)
)
}
body = {
'title': "MyFile.bin",
'description': "",
'modifiedDate': datetime.datetime.now().isoformat(),
'mimeType': 'application/octet-stream',
'parents': [{ 'id': parentId }]
}
res = http.request(
url, method="PUT", body=body, headers=headers
).execute()
So as you can see, it is clear where you specify the parameters for the file (file attributes) and the header specification for the request. But where do you specify the actual data stream to be uploaded in that request? Is it the case that I can just specify a media_body in the request?
You need to implement a multipart HTTP request which is explained on https://developers.google.com/drive/manage-uploads#multipart
I'd recommend you to use our JS client library and use the existing implementation on the API reference right under the JavaScript tab.
It is not possible and is not formally on Google's roadmap to introduce this functionality. The only way to append to a file is to update the entire file again from scratch.