I am trying to retrieve data by ID. Use the 3rd method in this link: How to speed up the search data in sheet
I run the function and err : Could not parse text.
I do not understand why I have used this method so many times and ran well, but this case is faulty.
This is my code:
function loadDataOfThread() {
var ss = SpreadsheetApp.openByUrl(url);
var ws = ss.getSheetByName("sheet1");
var ID = "12345";
var formatRange = ws.getRange(1, 1, ws.getLastRow() ,ws.getLastColumn()).setNumberFormat("#STRING#");
var query = "select * where A ='" + ID + "'";
var url = "https://docs.google.com/spreadsheets/d/" + ss.getId() + "/gviz/tq?gid=" + ws.getSheetId() + "&tqx=out:csv&tq=" + encodeURIComponent(query);
var options = {
headers: {
'Authorization': 'Bearer ' + ScriptApp.getOAuthToken()
}
};
var csv = UrlFetchApp.fetch(url, options);
var f = Utilities.parseCsv(csv); // err this line
var dataArr = [];
if (f.length > 0) {
for (var i = 0; i < f.length; i++) {
dataArr.push(f[i][1]);
}
}
}
I think in a spreadsheet whose data type is the date time column and it make err my function but i have convert to string !!! I do not understand why ?
How about this answer? Your issue might be able to be removed with "PasteDataRequest" because "PasteDataRequest" is better than parseCsv() as the parser of CSV data. In this answer, I would like to propose a method for using "PasteDataRequest" of Sheets API. Please think of this as just one of several answers. The flow of this method is as follows.
Insert a sheet as a temporal sheet.
Put the CSV data to the inserted sheet using "PasteDataRequest" of Sheets API.
Retrieve values from the temporal sheet.
Delete the temporal sheet.
Modified script:
When your script is modified, please modify as follows.
Before you use this script, please enable Sheets API at Advanced Google services.
From:
var f = Utilities.parseCsv(csv);
To:
var temp = ss.insertSheet("temp");
var sheetId = temp.getSheetId();
var resource = {requests: [{pasteData: {data: csv.getContentText(), coordinate: {sheetId: sheetId}, delimiter: ","}}]};
Sheets.Spreadsheets.batchUpdate(resource, ss.getId());
var f = temp.getDataRange().getValues();
ss.deleteSheet(temp);
Note:
Of course, I think that the issue can be also removed by modifying csv of var csv = UrlFetchApp.fetch(url, options);. But from your question, I cannot image the values of your issue. So I proposed above method. If you want to use other method, can you provide a sample Spreadsheet for replicating your issue? Of course, please remove your personal information. By this, I would like to think of the issue.
References:
Method: spreadsheets.batchUpdate
PasteDataRequest
If I misunderstood your question and this was not the direction you want, I apologize.
Related
I have a Google sheet which has a large number of versions. Now, I want to get each version one by one, extract data of that version into Google sheet. I have tried to follow the below solution from #Tanaike:
function myFunction() {
var spreadsheetId = "###"; // Please set the Spreadsheet ID.
var revisionId = "###"; // Please set the revision ID.
var url = "https://docs.google.com/spreadsheets/export?id=" + spreadsheetId + "&revision=" + revisionId + "&exportFormat=xlsx";
var blob = UrlFetchApp.fetch(url, {headers: {Authorization: "Bearer " + ScriptApp.getOAuthToken()}}).getBlob();
var id = Drive.Files.insert({mimeType: MimeType.GOOGLE_SHEETS, title: revisionId}, blob).id;
var spreadsheet = SpreadsheetApp.openById(id);
var sheet = spreadsheet.getSheets()[0];
var sheetdata = sheet.getDataRange().getValues();
}
It would be helpful if you can guide me/suggest it to me about it. I just want to get all the data for each version history and put it into Google sheets. Thank you for your guidance.
Following the advice in the comments. The sample code requires the revision ID. A sample code to gather all the revisions ID:
function listRevisions(fileId) {
var fileId = 'yourfileid';
var revisions = Drive.Revisions.list(fileId);
if (revisions.items && revisions.items.length > 0) {
for (var i = 0; i < revisions.items.length; i++) {
var revision = revisions.items[i];
var date = new Date(revision.modifiedDate);
Logger.log(revision.id);
}
} else {
Logger.log('No revisions found.');
}
}
Get the File ID of the Spreadsheet that you want to get its revision (the one you would be adding in the first sample code)
Revisions: list, you can get here the list of all revision that you do for that file in particular and is presented in the sample code above or you can manually test it over the web
By using Revisions: get, you should be able to get an specific revision details of the file by adding both the file id and revision id.
There is also an excellent thread with information about it where the sample code was taken (besides the ones linked in the revision list official documentation, there are javascript samples that can also be used) feel free to review the thread
This question already has answers here:
Scraping data to Google Sheets from a website that uses JavaScript
(2 answers)
Closed last month.
I'm attempting to scrape options pricing data from Yahoo Finance in Google Sheets. Although I'm able to pull the options chain just fine, i.e.
=IMPORTHTML("https://finance.yahoo.com/quote/TCOM/options?date=1610668800","table",2)
I find that it's returning results that don't completely match what's actually shown on Yahoo Finance. Specifically, the scraped results are incomplete - they're missing some strikes. i.e. the first 5 rows of the chart may match, but then it will start returning only every other strike (aka skipping every other strike).
Why would IMPORTHTML be returning "abbreviated" results, which don't match what's actually shown on the page? And more importantly, is there some way to scrape complete data (i.e. that doesn't skip some portion of the available strikes)?
In Yahoo finance, all data are available in a big json called root.App.main. So to get the complete set of data, proceed as following
var source = UrlFetchApp.fetch(url).getContentText()
var jsonString = source.match(/(?<=root.App.main = ).*(?=}}}})/g) + '}}}}'
var data = JSON.parse(jsonString)
You can then choose to fetch the informations you need. Take a copy of this example https://docs.google.com/spreadsheets/d/1sTA71PhpxI_QdGKXVAtb0Rc3cmvPLgzvXKXXTmiec7k/copy
edit
if you want to get a full list of available data, you can retrieve it by this simple script
// mike.steelson
let result = [];
function getAllDataJSON(url = 'https://finance.yahoo.com/quote/TCOM/options?date=1610668800') {
var source = UrlFetchApp.fetch(url).getContentText()
var jsonString = source.match(/(?<=root.App.main = ).*(?=}}}})/g) + '}}}}'
var data = JSON.parse(jsonString)
getAllData(eval(data),'data')
var sh = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet()
sh.getRange(1, 1, result.length, result[0].length).setValues(result);
}
function getAllData(obj,id) {
const regex = new RegExp('[^0-9]+');
for (let p in obj) {
var newid = (regex.test(p)) ? id + '["' + p + '"]' : id + '[' + p + ']';
if (obj[p]!=null){
if (typeof obj[p] != 'object' && typeof obj[p] != 'function'){
result.push([newid, obj[p]]);
}
if (typeof obj[p] == 'object') {
getAllData(obj[p], newid );
}
}
}
}
Here's a simpler way to get the last market price of a given option. Add this function to you Google Sheets Script Editor.
function OPTION(ticker) {
var ticker = ticker+"";
var URL = "finance.yahoo.com/quote/"+ticker;
var html = UrlFetchApp.fetch(URL).getContentText();
var count = (html.match(/regularMarketPrice/g) || []).length;
var query = "regularMarketPrice";
var loc = 0;
var n = parseInt(count)-2;
for(i = 0; i<n; i++) {
loc = html.indexOf(query,loc+1);
}
var value = html.substring(loc+query.length+9, html.indexOf(",", loc+query.length+9));
return value*100;
}
In your google sheets input the Yahoo Finance option ticker like below
=OPTION("AAPL210430C00060000")
I believe your goal as follows.
You want to retrieve the complete table from the URL of https://finance.yahoo.com/quote/TCOM/options?date=1610668800, and want to put it to the Spreadsheet.
Issue and workaround:
I could replicate your issue. When I saw the HTML data, unfortunately, I couldn't find the difference of HTML between the showing rows and the not showing rows. And also, I could confirm that the complete table is included in the HTML data. By the way, when I tested it using =IMPORTXML(A1,"//section[2]//tr"), the same result of IMPORTHTML occurs. So I thought that in this case, IMPORTHTML and IMPORTXML might not be able to retrieve the complete table.
So, in this answer, as a workaround, I would like to propose to put the complete table parsed using Sheets API. In this case, Google Apps Script is used. By this, I could confirm that the complete table can be retrieved by parsing the HTML table with Sheet API.
Sample script:
Please copy and paste the following script to the script editor of Spreadsheet, and please enable Sheets API at Advanced Google services. And, please run the function of myFunction at the script editor. By this, the retrieved table is put to the sheet of sheetName.
function myFunction() {
// Please set the following variables.
const url ="https://finance.yahoo.com/quote/TCOM/options?date=1610668800";
const sheetName = "Sheet1"; // Please set the destination sheet name.
const sessionNumber = 2; // Please set the number of session. In this case, the table of 2nd session is retrieved.
const html = UrlFetchApp.fetch(url).getContentText();
const section = [...html.matchAll(/<section[\s\S\w]+?<\/section>/g)];
if (section.length >= sessionNumber) {
if (section[sessionNumber].length == 1) {
const table = section[sessionNumber][0].match(/<table[\s\S\w]+?<\/table>/);
if (table) {
const ss = SpreadsheetApp.getActiveSpreadsheet();
const body = {requests: [{pasteData: {html: true, data: table[0], coordinate: {sheetId: ss.getSheetByName(sheetName).getSheetId()}}}]};
Sheets.Spreadsheets.batchUpdate(body, ss.getId());
}
} else {
throw new Error("No table.");
}
} else {
throw new Error("No table.");
}
}
const sessionNumber = 2; means that 2 of =IMPORTHTML("https://finance.yahoo.com/quote/TCOM/options?date=1610668800","table",2).
References:
Method: spreadsheets.batchUpdate
PasteDataRequest
Since yesterday I get the problem when I make a query to a sheet to bring information. The code to bring information is the following:
var s_usuario="c123";
var spreadsheetId = 'myid'; // Please set the spreadsheet ID.
var targetSheet = "Usuarios"; // Please set the sheet name.
var usuario =s_usuario.toUpperCase();
var query = 'select B where A = "'+usuario+'"'; // Please set the query for retrieving the values.
var ss = SpreadsheetApp.openById(spreadsheetId);
var sheetId = ss.getSheetByName(targetSheet).getSheetId();
var url = "https://docs.google.com/spreadsheets/d/" + spreadsheetId + "/gviz/tq?gid=" + sheetId + "&tqx=out:csv&tq=" + encodeURIComponent(query);
var res = UrlFetchApp.fetch(url, {headers: {Authorization: "Bearer " + ScriptApp.getOAuthToken()}});
var row = Utilities.parseCsv(res.getContentText());
var maximo=row.length;
for (var i = 0; i < maximo; i++) {
var mostrar =row[i][0].toString();
}
return mostrar
What should I modify so that the query can be given? I think there was an update in the language
Perhaps you want:
var mostrar='';
for (var i=0;i<maximo; i++) {
mostrar +=row[i][0].toString();
}
return mostrar;
or even:
var mostrar=row.map(r=>{return r[0];});
return mostrar.join(',');
Probably everything else is okay since you copied it from here
This is a new bug related to the UrlFetchApp
It seems to already worked on, so hopefully the issue is of temporarily nature.
I don't know the solution, but I can also say that in the past 2-3 days my scripts also all started failing that use var file = UrlFetchApp.fetch(url); My scripts are on a schedule and all of them fail at any line that uses this. I see that you are using UrlFetchApp as well. My first failure happened at Dec 8, 2020, 3:53:50 AM
Hopefully this can help figure out what is going on. Here is the specific error I get on mine:
Exception: Unexpected error: https://...org/..xxxx/..xxx.csv (line 242, file "Code")
I am very interested in the solution that works for you, perhaps it can help me as well.
Either using rest API, Google Scripts, Node SDK, whatever works.
I'm seeing this in the docs but that doesn't seem to tell me the duration:
function watchFile(fileId, channelId, channelType, channelAddress) {
var resource = {
'id': channelId,
'type': channelType,
'address': channelAddress
};
var request = gapi.client.drive.files.watch({
'fileId': fileId,
'resource': resource
});
request.execute(function(channel){console.log(channel);});
}
I found this link but it doesn't seem to help https://apis-nodejs.firebaseapp.com/drive/classes/Resource$Files.html#watch
You want to retrieve the duration of the video on your Google Drive.
You want to achieve this using Google Apps Script.
If my understanding is correct, how about this sample script? In this modification, I used files.get and files.list methods of Drive API. From your question, I thought that the script that the endpoint is directly requests might be useful for your situation. So I proposed the following script.
1. Using files.get method
In this sample script, the duration is retrieved from a video file.
Sample script:
function sample1() {
var fileId = "###"; // Please set the file ID of the video file.
var fields = "mimeType,name,videoMediaMetadata"; // duration is included in "videoMediaMetadata"
var url = "https://www.googleapis.com/drive/v3/files/" + fileId + "?fields=" + encodeURIComponent(fields) + "&access_token=" + ScriptApp.getOAuthToken();
var res = UrlFetchApp.fetch(url);
var obj = JSON.parse(res);
Logger.log("filename: %s, duration: %s seconds", obj.name, obj.videoMediaMetadata.durationMillis / 1000);
// DriveApp.getFiles() // This line is put for automatically detecting the scope (https://www.googleapis.com/auth/drive.readonly) for this script.
}
2. Using files.list method
In this sample script, the durations are retrieved from a folder including the video files.
Sample script:
function sample2() {
var folderId = "###"; // Please set the folder ID including the video files.
var q = "'" + folderId + "' in parents and trashed=false";
var fields = "files(mimeType,name,videoMediaMetadata)"; // duration is included in "videoMediaMetadata"
var url = "https://www.googleapis.com/drive/v3/files?q=" + encodeURIComponent(q) + "&fields=" + encodeURIComponent(fields) + "&access_token=" + ScriptApp.getOAuthToken();
var res = UrlFetchApp.fetch(url);
var obj = JSON.parse(res);
for (var i = 0; i < obj.files.length; i++) {
Logger.log("filename: %s, duration: %s seconds", obj.files[i].name, obj.files[i].videoMediaMetadata.durationMillis / 1000);
}
// DriveApp.getFiles() // This line is put for automatically detecting the scope (https://www.googleapis.com/auth/drive.readonly) for this script.
}
Note:
These are simple sample scripts. So please modify them for your situation.
I'm not sure about the format of your video files. So if above script cannot be used for your situation, I apologize.
References:
Files of Drive API
Class UrlFetchApp
If I misunderstood your question and this was not the result you want, I apologize.
Updated: March 19, 2020
From January, 2020, the access token cannot be used with the query parameter like access_token=###. Ref So please use the access token to the request header instead of the query parameter. It's as follows.
var res = UrlFetchApp.fetch(url, {headers: {Authorization: "Bearer " + ScriptApp.getOAuthToken()}});
I am trying to query a spreadsheet with gviz (Google Visualization), using UrlFetchApp, but no result so far.
Could you help me to fix this code?
(the query Url works fine in the browser)
function queryTest() {
var onlyforscope = SpreadsheetApp.getActiveSpreadsheet();
var template = "https://docs.google.com/spreadsheets/d/%s/gviz/tq?gid=%s&tq=select C,E,K,M,N,O where C contains '%s'";
var query = Utilities.formatString(template, docId, sheetId, value);
var param = {
method : "get",
headers : {"Authorization": "Bearer " + ScriptApp.getOAuthToken()},
muteHttpExceptions:true,
};
var r = UrlFetchApp.fetch(query, param).getContentText();
// var j = JSON.parse(r);
Logger.log(r);
return;
}
Thanks in advance, Fausto
it was trivial, though hard to find out for me
the required scope is Drive !!!
I just add this line and it worked
var onlyforscope = DriveApp.getRootFolder();