I'm trying to understand how the Google API works server side in order to allow me to implement my own type of resumable upload. I understand that I can use the MediaFileUpload or MediaInMemoryUpload mechanism, but I am looking for something much more raw. For example, I want to deliberately upload 1k from a file, then later on (like days later), append another 1k of the file. Obviously not real figures here, but hopefully you get the idea. Well here is where I am with the code:
headers = {
'range': 'bytes=%d-%d' % (
offset,
offset + len(data)
)
}
body = {
'title': "MyFile.bin",
'description': "",
'modifiedDate': datetime.datetime.now().isoformat(),
'mimeType': 'application/octet-stream',
'parents': [{ 'id': parentId }]
}
res = http.request(
url, method="PUT", body=body, headers=headers
).execute()
So as you can see, it is clear where you specify the parameters for the file (file attributes) and the header specification for the request. But where do you specify the actual data stream to be uploaded in that request? Is it the case that I can just specify a media_body in the request?
You need to implement a multipart HTTP request which is explained on https://developers.google.com/drive/manage-uploads#multipart
I'd recommend you to use our JS client library and use the existing implementation on the API reference right under the JavaScript tab.
It is not possible and is not formally on Google's roadmap to introduce this functionality. The only way to append to a file is to update the entire file again from scratch.
Related
I am using the google-drive-sdk with our company-made device. We upload pictures made by our device to google drive. After that I try to list the files with a GET request to https://www.googleapis.com/drive/v2/files to get thumbnailLink and webContentLink. Everything is working fine except that when I switch to v3 I don't get the response I should. The documentation says I should get a metadata response like https://developers.google.com/drive/v3/reference/files
but I only get: id, kind, name and mimeType. What am I doing wrong?
As stated in Migrate to Google Drive API v3 documentation, there are changes on how fields were returned.
Full resources are no longer returned by default. You need to use the fields query parameter to request specific fields to be returned. If left unspecified only a subset of commonly used fields are returned.
You can see examples on Github. This SO question might also help.
In v3 they made all the queries parametric. So you can query passing some parameter like
var request = gapi.client.drive.files.list({
'pageSize': 10,
'fields': 'files,kind,nextPageToken'
});
This block of code will return you all the information of every file just like v2.
If you are sending a get request then for fetching all the information you can try GET https://www.googleapis.com/drive/v3/files?fields=files%2Ckind%2CnextPageToken&key={YOUR_API_KEY}
Suppose you need ownsers and permissions only then set
var request = gapi.client.drive.files.list({
'pageSize': 10,
'fields':'files(owners,permissions),kind,nextPageToken'
});
For GET request use GET https://www.googleapis.com/drive/v3/files?fields=files(owners%2Cpermissions)%2Ckind%2CnextPageToken&key={YOUR_API_KEY}
for reference you can use Google Developers Documentation for fetching File list
How do I append content to the end of a file in Google Drive using the API ?
Do I really have to download the whole thing, then edit the local copy, and then re-upload the whole thing again?
Yes you really have to download edit the file and upload it again. There is no way to programmatically edit a file. Except maybe a spreadsheet but then you would be using the Google sheets API and not the Google drive API.
You can use the drive apis resumable upload with some restrictions:
(I don't quite remember if its true) Minimum bytes uploaded have to be 262144 except for the final upload which "creates" the file, which can contain less
An upload-session expires after one week,
you can set Content-Range to */* if you don't know the final filesize
The file wont show up in google drive in the ui until complete,Upload using a similiar header to the example below, where /* is the final byte length of the file. It has to be one byte less, like in the example below: 262146/262147
I recommend getting a service account for gcp project, you can create a folder in your personal drive and share it with the service account email.
To save some time, because the drive api documentation is not the best, here in "pure" python http requests:
First you have to create the file and get the session_url:
headers = {"Authorization": "Bearer "+myAccesstoken,
"Content-Type": "application/json"}
file_metadata = {
'name': "myFile.txt",
'mimeType': "text/plain",
'parents': [myFolderid],
"uploadType": "resumable"
}
r = requests.post(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable",
headers=headers,
data=json.dumps(file_metadata)
)
session_url=r.headers['Location']
Then you can upload data to it:
headers = {
"Authorization": "Bearer "+myAccesstoken,
"Content-Range": 'bytes 0-262144/*'}
if is_final_data:
headers = {
"Authorization": "Bearer "+myAccestoken,
"Content-Range": 'bytes 262144-262146/262147'}
sd = io.BytesIO()
sd.write(bytes("Wurst", "ASCII"))
sd.seek(0)
r = requests.put(
session_url,
headers=headers,
data=sd
)
To get last uploaded byte position, if you are resuming an upload, send an empty put request only with the session url and authorization headers and read its response headers afterwards.
You can store the session url in a file and resume upload for one week.
Note: You will need something like below, since the access token is only valid for a limited amount of time.
if credentials.access_token_expired:
credentials.refresh(httplib2.Http())
I'm creating an API which should return data in a CSV format. I set the content-type header to text/csv but this forces a download of the contents as a csv file.
I'm using NodeJS and the express framework. It could be that this is standard behaviour. However I would like to know how you guys solved this issue.
This is a sample of the code that I'm using:
res.set('Content-Type', 'text/csv');
var toCsv = require('to-csv');
// obj is a just a standard JavaScript object.
res.send(toCsv(obj));
I would like that the person using the API can retrieve data in a CSV format without actually downloading a file
Maybe have a look at this question:
How does browser determine whether to download or show
It's your browser that decides that content of the type "text/csv" should be downloaded.
You should simply consider using another content-type, if you just want the csv to show in the browser as plain text.
Try this instead:
res.set('Content-Type', 'text/plain');
I am having a problem integrating Couchdb and D3. D3 is a Javascript library that performs document driven data visualization. Couchdb is a document database. They were made for each other.
D3 binds an array of data to DOM elements of a web page. In most of the examples I have seen on the web or in books, people are working on a static data set. Generally, examples will show an array written into the Javascript or a text.csv file loaded into the page.
I would like to take data directly from database documents and load it into D3. I'm uncertain how to do it. I have seen one example on the web where a person has loaded all of their data as an array into one couchdb document and then brought the data into index.html with a couchdb.jquery call:
/ This function replaces the d3.csv function.
$.couch.db("d3apps3").openDoc("sp500", {
success : function (doc) {
var data = doc.data;
data.forEach(function(d) {
d.date = formatDate.parse(d.date);
d.price = +d.price;
})
I tried something similar with db.allDocs:
<script type="text/javascript">
$dbname = "dataset2";
$appname = "dataset2";
$db = $.couch.db("dataset2");
$db.allDocs({
success: function (data) {
console.log(data)
}
});
</script>
I could get the data to render in console.log, but could not get it into D3 and index.html. I also realized that the datastream resulting from db.allDocs is limited to the _id and _rev of each document.
I also tried to GET the data from a Couchdb view with a d3.json call. That wouldn't work because d3.json is looking for an existing .json file.
It's funny, I can call the view with cURL using a GET command and see the datastream, but can't seem to bind it with D3.
~$ curl -X GET http://anywhere.com:5984/dataset2/_desing/list_view/_view/arnold
{"total_rows":25,"offset":0,"rows":[
{"id":"dataset.csv1","key":"0","value":null},
{"id":"dataset.csv2","key":"1","value":null},
{"id":"dataset.csv11","key":"10","value":null},
{"id":"dataset.csv12","key":"11","value":null},
Any ideas would be appreciated.
Part four of https://gist.github.com/anonymous/9275891 has an example that I think you'd appreciate. You don't need to rely on the jquery.couchdb library at all - d3 knows enough abuot http and json to work right out the box. The relevant piece of code is:
d3.json("_view/pricetimeseries", function(viewdata) {
// We just want rows from the view in the visualisation
data = viewdata["rows"];
data.forEach(function(d) {
// the key holds the date, in seconds
d.date = new Date(d.key);
d.price = +d.value;
});
// rest of the visalisation code
HTH
If the page in which your D3 code is embedded is not served from the same domain (+ port) than CouchDB you will have to enable Cross-Origin Resource Sharing.
Assume your page is at http://example.com/data.html which contains JavaScript D3 code that acesses data from http://db.example.com/ or http://example.com:5984/. In that case your browser (which is executing the JavaScript) will by default deny such (cross-origin) requests unless the requested domain explicitly allows it.
There are basically two solutions to this:
Serve both the data and the page from the same domain, either by
putting a reverse proxy in between that maps resources to upstream servers (eg /couch to your CouchDB server and everything else to your web server)
serving your static files directly from CouchDB
or by allowing Cross-Origin Resource Sharing, which is available in CouchDB since version 1.3. You can find a list of relevant settings in the CouchDB docs on CORS.
I'm having some trouble with the UrlFetchApp class, fetch() method. I've singled out the issue, and it seems to be the fact that the actual link I'm fetching is just too long
When I eliminate some needed data(resulting in ~1900 characters), it send the fetch request fine
The length limit is somewhere between 2040 and 2060 characters, as that is where it stops working and I receive a "Bad request" error. I'm assuming it's 2048, as that seems to have been the industry standard some time ago.
I'm needing to fetch data from a link that's upwards of 3400 characters! Is this just too long? 2048 characters might have been understandable a while back, but in this day in age it's a limit that is going to be met quite often
My question is this: Is there a way around this? I'm assuming Google set the limit, is there some way to request this limit be raised?
Thank you!
The restriction is on the size (2kB) and not on the length of the url.
On March 30, 2018, Google deprecated the URL Shortener service that was used in the accepted answer.
I wrote a script to use the Firebase Dynamic Links Short Links API service.
The docs are here if you want to cook your own.
You can try UrlShortener to shorten the URL and then use UrlFetchApp with the shortened URL
I used the POST method with payload data instead, showed here:
Google Apps Script POST request UrlFetchApp
The classic code is:
// Make a POST request with form data.
var resumeBlob = Utilities.newBlob('Hire me!', 'text/plain', 'resume.txt');
var formData = {
'name': 'Bob Smith',
'email': 'bob#example.com',
'resume': resumeBlob
};
// Because payload is a JavaScript object, it is interpreted as
// as form data. (No need to specify contentType; it automatically
// defaults to either 'application/x-www-form-urlencoded'
// or 'multipart/form-data')
var options = {
'method' : 'post',
'payload' : formData
};
UrlFetchApp.fetch('https://httpbin.org/post', options);