I have the following data example in a google sheet:
url
https://www.testwebsite.com/compute/v1/test/images-prd-5d4d/glob/images/testimage-vsfd
https://www.testwebsite.com/compute/v1/test/images-prd-5d4d/glob/images/testimage-sdawr|
What I need is to extract the data after the substring "images/" and have something like this:
url
extract
https://www.testwebsite.com/compute/v1/test/images-prd-5d4d/glob/images/testimage-vsfd
testimage-vsfd
https://www.testwebsite.com/compute/v1/test/images-prd-5d4d/glob/images/testimage-sdawr|testimage-sdawr
I have created the following function to get this but is only extracting everything after the last "-":
function strip() {
const ss = SpreadsheetApp.getActive();
const sh = ss.getSheetByName("Sheet6");
const vs = sh.getRange(2,1,sh.getLastRow() - 1).getDisplayValues().flat();
let vo = vs.map(s => [s.match(/\b[0-9A-Zaz/]+$/gi)[0]]);
sh.getRange(2,2,vo.length,1).setValues(vo);
}
What is the proper way to extract the data it's mentioned above?
You could use this on Apps Script:
function strip() {
const ss = SpreadsheetApp.getActive();
const sh = ss.getSheetByName("Sheet6");
const vs = sh.getRange(2,1,sh.getLastRow() - 1).getDisplayValues().flat();
const string= "/images/";
for (i = 0; i < vs.length; i++){
//Using substrings:
const extract = vs[i].substring(vs[i].indexOf(string) + string.length);
sh.getRange(i+2,2).setValue(extract);
//Using .split():
// const extract = vs[i].split(string); //This splits the string in 2.
// sh.getRange(i+2,2).setValue(extract[1]); //Adding the second part of the array;
}
}
If you want to do it like a custom function you can try the following code:
function strip(url) {
var text = url;
var splittedValue = text.split("/images/");
return splittedValue[1];
}
It would work something like this:
Input:
Result:
The script can also be changed to get a specific range of data automatically so that every time you add a new URL you get the result in the next column automatically, but this is just for you to get the idea.
References:
Custom Functions in Google Sheets
Say your URLs are in A2:A. You can use
=arrayformula(if(isblank(A2:A),,substitute(REGEXEXTRACT(A2:A,"/images/[A-Za-z0-9-_|/\.]+"),"/images/","")))
Use native formulas where possible. That is more efficient.
If you already dealt with the issue with run time delay, and have a need to use custom function for other reasons, you can match with the "/image/" part and then remove it, or, alternatively, specifying a capturing group. Also don't forget other value characters such as _, |.
Related
I am fetching data from an api to google sheet. all its getting is data string on a single cell. How can I format it into a table with headers. Part of the long string I am getting and the code are below.
String - {instantBuy=5679.95, pricechange=0, instantSell=5623.39, currency=AUD, 24hoursHigh=5848.86, market=5848.86, virtualCurrency=ETH, 24hoursLow=5848.86, sell=5848.86, volume=0.341947, buy=5848.86, pair=ETH-AUD} ...
Code:
function myQuotes() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var mainSheet = ss.getSheetByName('Sheet1');
var url = "https://www.zebapi.com/pro/v1/market/";
var response = UrlFetchApp.fetch(url);
var json = response.getContentText();
var data = JSON.parse(json);
mainSheet.getRange(4,2).setValue(data);
Thanks!
Use Object.entries(), like this:
function myQuotes() {
const url = "https://www.zebapi.com/pro/v1/market/";
const response = UrlFetchApp.fetch(url);
const json = response.getContentText();
const data = transpose_(Object.entries(JSON.parse(json)));
const targetRange = SpreadsheetApp.getActive().getRange('Sheet1!B4');
targetRange.offset(0, 0, data.length, data[0].length).setValues(data);
}
function transpose_(a) {
// #see https://stackoverflow.com/a/13241545/1536038
return Object.keys(a[0]).map(c => a.map(r => r[c]));
}
You may also want to take a look at ImportJSON.
I'm not very familiar with JavaScript, but you could try the search() method, which I believe returns the index at which the word is found. So for example, if you were trying to put data from "instantBuy", and put it in it's own row or column, you could do search("instantBuy"), get the string between that and the next index that search will return, search("pricechange")
Sorry if that was confusing!
This question already has answers here:
Scraping data to Google Sheets from a website that uses JavaScript
(2 answers)
Closed last month.
I'm attempting to scrape options pricing data from Yahoo Finance in Google Sheets. Although I'm able to pull the options chain just fine, i.e.
=IMPORTHTML("https://finance.yahoo.com/quote/TCOM/options?date=1610668800","table",2)
I find that it's returning results that don't completely match what's actually shown on Yahoo Finance. Specifically, the scraped results are incomplete - they're missing some strikes. i.e. the first 5 rows of the chart may match, but then it will start returning only every other strike (aka skipping every other strike).
Why would IMPORTHTML be returning "abbreviated" results, which don't match what's actually shown on the page? And more importantly, is there some way to scrape complete data (i.e. that doesn't skip some portion of the available strikes)?
In Yahoo finance, all data are available in a big json called root.App.main. So to get the complete set of data, proceed as following
var source = UrlFetchApp.fetch(url).getContentText()
var jsonString = source.match(/(?<=root.App.main = ).*(?=}}}})/g) + '}}}}'
var data = JSON.parse(jsonString)
You can then choose to fetch the informations you need. Take a copy of this example https://docs.google.com/spreadsheets/d/1sTA71PhpxI_QdGKXVAtb0Rc3cmvPLgzvXKXXTmiec7k/copy
edit
if you want to get a full list of available data, you can retrieve it by this simple script
// mike.steelson
let result = [];
function getAllDataJSON(url = 'https://finance.yahoo.com/quote/TCOM/options?date=1610668800') {
var source = UrlFetchApp.fetch(url).getContentText()
var jsonString = source.match(/(?<=root.App.main = ).*(?=}}}})/g) + '}}}}'
var data = JSON.parse(jsonString)
getAllData(eval(data),'data')
var sh = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet()
sh.getRange(1, 1, result.length, result[0].length).setValues(result);
}
function getAllData(obj,id) {
const regex = new RegExp('[^0-9]+');
for (let p in obj) {
var newid = (regex.test(p)) ? id + '["' + p + '"]' : id + '[' + p + ']';
if (obj[p]!=null){
if (typeof obj[p] != 'object' && typeof obj[p] != 'function'){
result.push([newid, obj[p]]);
}
if (typeof obj[p] == 'object') {
getAllData(obj[p], newid );
}
}
}
}
Here's a simpler way to get the last market price of a given option. Add this function to you Google Sheets Script Editor.
function OPTION(ticker) {
var ticker = ticker+"";
var URL = "finance.yahoo.com/quote/"+ticker;
var html = UrlFetchApp.fetch(URL).getContentText();
var count = (html.match(/regularMarketPrice/g) || []).length;
var query = "regularMarketPrice";
var loc = 0;
var n = parseInt(count)-2;
for(i = 0; i<n; i++) {
loc = html.indexOf(query,loc+1);
}
var value = html.substring(loc+query.length+9, html.indexOf(",", loc+query.length+9));
return value*100;
}
In your google sheets input the Yahoo Finance option ticker like below
=OPTION("AAPL210430C00060000")
I believe your goal as follows.
You want to retrieve the complete table from the URL of https://finance.yahoo.com/quote/TCOM/options?date=1610668800, and want to put it to the Spreadsheet.
Issue and workaround:
I could replicate your issue. When I saw the HTML data, unfortunately, I couldn't find the difference of HTML between the showing rows and the not showing rows. And also, I could confirm that the complete table is included in the HTML data. By the way, when I tested it using =IMPORTXML(A1,"//section[2]//tr"), the same result of IMPORTHTML occurs. So I thought that in this case, IMPORTHTML and IMPORTXML might not be able to retrieve the complete table.
So, in this answer, as a workaround, I would like to propose to put the complete table parsed using Sheets API. In this case, Google Apps Script is used. By this, I could confirm that the complete table can be retrieved by parsing the HTML table with Sheet API.
Sample script:
Please copy and paste the following script to the script editor of Spreadsheet, and please enable Sheets API at Advanced Google services. And, please run the function of myFunction at the script editor. By this, the retrieved table is put to the sheet of sheetName.
function myFunction() {
// Please set the following variables.
const url ="https://finance.yahoo.com/quote/TCOM/options?date=1610668800";
const sheetName = "Sheet1"; // Please set the destination sheet name.
const sessionNumber = 2; // Please set the number of session. In this case, the table of 2nd session is retrieved.
const html = UrlFetchApp.fetch(url).getContentText();
const section = [...html.matchAll(/<section[\s\S\w]+?<\/section>/g)];
if (section.length >= sessionNumber) {
if (section[sessionNumber].length == 1) {
const table = section[sessionNumber][0].match(/<table[\s\S\w]+?<\/table>/);
if (table) {
const ss = SpreadsheetApp.getActiveSpreadsheet();
const body = {requests: [{pasteData: {html: true, data: table[0], coordinate: {sheetId: ss.getSheetByName(sheetName).getSheetId()}}}]};
Sheets.Spreadsheets.batchUpdate(body, ss.getId());
}
} else {
throw new Error("No table.");
}
} else {
throw new Error("No table.");
}
}
const sessionNumber = 2; means that 2 of =IMPORTHTML("https://finance.yahoo.com/quote/TCOM/options?date=1610668800","table",2).
References:
Method: spreadsheets.batchUpdate
PasteDataRequest
I want to retrieve a list of files from google drive using GAS.
So I want the list of files based on the owner who is empty.
For files that we create on the share drive, the owner doesn't exist.
If we try to force the owner name, it will get an error.
For example like this:
file.getOwner().getName();
it cannot be retrieved from the file list.
I have tried adding a filter based on an empty owner, I would like to say Unknown.
function getfilesUnknown () {
var ss = SpreadsheetApp.getActiveSpreadsheet ();
var st = ss.getActiveSheet ();
var files = DriveApp.getFiles ();
var counter = 0;
while (files.hasNext () && counter <21) {
var file = files.next ();
var data = [
file.getName (),
file.getUrl (),
file.getOwner ()? file.getOwner (). getName (): 'Unknown'
];
var filter = data.filter (v => v [2] === 'Unknown');
counter ++;
st.appendRow (filter);
}
}
But the list of files won't retrieve.
It seems that the filter in while is not recommended.
Or maybe, folks here have another solution.
I am not sure if this is the best approach.
But since you are getting an error here file.getOwner().getName() when the owner does not exist, then try to use try...catch to catch the error.
Namely, if there is an error pass Unknown, otherwise pass the name of the file owner.
function getfilesUnknown() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var st = ss.getActiveSheet();
var files = DriveApp.getFiles();
var counter = 0;
var data = [];
while (files.hasNext() && counter<21) {
var file = files.next();
try{
data.push([file.getOwner().getName()]);
}
catch(e){
data.push(['Unknown']);
}
counter++;
}
st.getRange(st.getLastRow()+1,1,data.length,1).setValues(data);
}
Please note that your code is not very well optimized. You add information in the data array but then you filter on a particular value only. Therefore there is no need of using the other items as they will slow down your code.
Another note is that you are using appendRow iteratively which is not recommended. Instead use setValues outside of the for loop once.
I am currently working on a semester project for my university in which we want to log data from an Arduino to a Google Sheet.
I was following the numerous tutorials and examples that I could find on Google and it worked so far really, really well. My Arduino is able to upload data to said spreadsheet.
Unfortunately all those examples always only deal with one row to be filled. For our project we would like to fill 2 or 3 lines simultaneously.
I will shortly show what I have done so far and maybe you can help me solve my (probably easy) problem.
I created a google spreadsheet in which I want to log my data
I used the script from a tutorial that should fill one row.
By typing the following line in my browserhttps://script.google.com/macros/s/<gscript id>/exec?tempData=datahereI am now able to fill row one with my data in enter in the end of the url.
But how do I progress now, when I want to fill two or three rows of the table? I say that the author of the code already implemented an option to fill the third row, yet I can't find out what to input in my url then to fill it with data.
All my attempts to write something like
https://script.google.com/macros/s/<gscript id>/exec?tempData=datahere&tempData1=value2
just ended in writing
datahere&tempData1=value2
in my first row, not filling datahere into the first and value2 in to the second row.
How can I provide and write multiple rows of data?
The code in this script is:
/*
GET request query:
https://script.google.com/macros/s/<gscript id>/exec?tempData=data_here
*/
/* Using spreadsheet API */
function doGet(e) {
Logger.log( JSON.stringify(e) ); // view parameters
var result = 'Ok'; // assume success
if (e.parameter == undefined) {
result = 'No Parameters';
}
else {
var id = '<ssheet id>'; // Spreadsheet ID
var sheet = SpreadsheetApp.openById(id).getActiveSheet();
var newRow = sheet.getLastRow() + 1;
var rowData = [];
//var waktu = new Date();
rowData[0] = new Date(); // Timestamp in column A
for (var param in e.parameter) {
Logger.log('In for loop, param='+param);
var value = stripQuotes(e.parameter[param]);
//Logger.log(param + ':' + e.parameter[param]);
switch (param) {
case 'tempData': //Parameter
rowData[1] = value; //Value in column B
break;
case 'tempData1':
rowData[2] = value; //Value in column C
break;
default:
result = "unsupported parameter";
}
}
Logger.log(JSON.stringify(rowData));
// Write new row below
var newRange = sheet.getRange(newRow, 1, 1, rowData.length);
newRange.setValues([rowData]);
}
// Return result of operation
return ContentService.createTextOutput(result);
}
/**
* Remove leading and trailing single or double quotes
*/
function stripQuotes( value ) {
return value.replace(/^["']|['"]$/g, "");
}
I would suggest the following:
Create a 2d array of your data you wish to write to the spreadsheet. If your client on Arduino were using JavaScript this might look like :
var data = [
["row1value1", "row1value2"],
["row2value1", "row2value2"]
];
Convert this to JSON, again in JavaScript this might look like:
var json = JSON.stringify(data);
This gives you a string representation of your array.
Now make your request using this data. I would suggest you should look at using doPost instead of doGet, as you are sending data to the spreadsheet that updates state. However, for the purposes of getting something working, your URL would look like:
https://script.google.com/<.....>/exec?myarray=<stringified JSON>
In Apps Script, in your doGet (again, consider using doPost instead), you could then use:
// Get the JSON representation of the array:
var json = e.parameter.myarray;
// Convert back to 2d array
var data = JSON.parse(json);
Now you can write this to a Range in Sheets using setValues, e.g. assuming a rectangular 2d array:
sheet.getRange(1, 1, data.length, data[0].length).setValues(data);
Hope this helps
i got a quick one for someone who can help. I've downloaded some data from yahoo. I want to split the data into a N x 7 array. (is that the correct term?).
I want it to look like this:
[[2013-01-29,64.25,65.03,64.00,64.24,4883100,64.24],[2013-01-28,64.51,64.87,63.27,64.59,7591300,64.59],...]
but now, as you can see, it's not in that format. Novice to javascript. Please help.
function function() {
var ticker='YUM';
var startMonth=0; var startDate=1; var startYear=2013;
var endMonth=0; var endDate=25; var endYear=2013;
var fetchString="http://ichart.finance.yahoo.com/table.csv?s="+ticker+"&a="+startMonth+"&b="+startDate+"&c="+startYear+"&d="+endMonth+"e="+endDate+"&f="+endYear+"&g=d";
var response = UrlFetchApp.fetch(fetchString);
a=response.getContentText();
var allData = a.slice(a.indexOf("2013"));
}
Assuming you don't want the column headers, this is a one line change:
var allData = a.match(/(.*?)\n/g) // convert each line to a row
.splice(1) // remove headers row
.map(function(row){
return row.replace(/\n/,'').split(',');
}); // convert row string to array