I have a code in Google Apps Scripts that takes a Spotify URL and pastes the amount of followers in the cell where you reference the url. =SAMPLE(A2) (where A2 holds the URL)
It seems that the specification has changed again. And I'm not sure how to fix it. Any guidance or help would be greatly appreciated!
Here's the original post = FetchingURL Using JSON on Google Sheets
Here's the code I have currently in Google Apps Scripts. (courtesy of #Tanaike)
function SAMPLE(url) {
const res = UrlFetchApp.fetch(url).getContentText();
const v = res.match(/"followers":({[\s\S\w]+?})/);
return v && v.length == 2 ? JSON.parse(v[1].trim()).total : "Value cannot be
retrieved.";
}
Thanks!
In the current stage, it seems that the JSON data is put as base64 data, and also, the structure of the object was changed. So, as the current sample script, how about the following sample script?
Sample script:
function SAMPLE(url) {
const res = UrlFetchApp.fetch(url).getContentText();
const v = res.match(/<script id\="initial-state" type\="text\/plain">([\s\S\w]+?)<\//);
if (!v || v.length != 2) return "Value cannot be retrieved.";
const obj = JSON.parse(Utilities.newBlob(Utilities.base64Decode(v[1].trim())).getDataAsString());
const value = Object.entries(obj.entities.items).reduce((n, [k, o]) => {
if (k.includes("spotify:playlist:")) n.push(o.followers.total);
return n;
}, []);
return value.length > 0 ? value[0] : "Value cannot be retrieved.";
}
Note:
This sample script is for the current HTML data. So, I think that this might be changed in the future update. So, I also would like to recommend using the API for your future work as it has already been mentioned in the comment.
Related
A previously working solution that was resolved here by #tanaike suddenly returns an empty cell upon execution. I don't get an error message and in the google apps scripts edit page I get "Notice Execution completed".
It looks like it's working in the background but having trouble returning a value to the cell, my guess would be something wrong with the last line that may resolve it?
function pressReleases(code) {
var url = 'https://finance.yahoo.com/quote/' + code + '/press-releases'
var html = UrlFetchApp.fetch(url).getContentText().match(/root.App.main = ([\s\S\w]+?);\n/);
if (!html || html.length == 1) return;
var obj = JSON.parse(html[1].trim());
// --- I modified the below script.
const { _cs, _cr } = obj;
if (!_cs || !_cr) return;
const key = CryptoJS.algo.PBKDF2.create({ keySize: 8 }).compute(_cs, JSON.parse(_cr)).toString();
const obj2 = JSON.parse(CryptoJS.enc.Utf8.stringify(CryptoJS.AES.decrypt(obj.context.dispatcher.stores, key)));
var res = obj2.StreamStore.streams["YFINANCE:" + code + ".mega"].data.stream_items[0].title;
// ---
return res || "No value";
}
The CryptoJS code saved as a script in google apps script is here
When I tested this script, at if (!_cs || !_cr) return;, I confirmed that the values of _cs and _cr are undefined. From this result, I understood that recently, the specification of the key for decrypting the data has been changed at the server side. When I saw this thread, I confirmed the same situation. In the thread, I noticed that, in the current stage, the key can be simply retrieved from the HTML data. So, as with the current script, how about the following modification?
Usage:
1. Get crypto-js.
Please access https://cdnjs.cloudflare.com/ajax/libs/crypto-js/4.1.1/crypto-js.min.js. And, copy and paste the script to the script editor of Google Apps Script, and save the script.
2. Modify script.
function pressReleases(code) {
var url = 'https://finance.yahoo.com/quote/' + code + '/press-releases'
var html = UrlFetchApp.fetch(url).getContentText().match(/root.App.main = ([\s\S\w]+?);\n/);
if (!html || html.length == 1) return;
var obj = JSON.parse(html[1].trim());
var key = Object.entries(obj).find(([k]) => !["context", "plugins"].includes(k))[1];
if (!key) return;
const obj2 = JSON.parse(CryptoJS.enc.Utf8.stringify(CryptoJS.AES.decrypt(obj.context.dispatcher.stores, key)));
var res = obj2.StreamStore.streams["YFINANCE:" + code + ".mega"].data.stream_items[0].title;
// console.log(res); // Check the value in the log.
return res || "No value";
}
When this script is run with code = "PGEN", the value of Precigen Provides Pipeline and Corporate Updates at the 41st Annual J.P. Morgan Healthcare Conference is obtained.
Note:
If you want to load crypto-js directly, you can also use the following script. But, in this case, the process cost becomes higher than that of the above flow. Please be careful about this.
const cdnjs = "https://cdnjs.cloudflare.com/ajax/libs/crypto-js/4.1.1/crypto-js.min.js";
eval(UrlFetchApp.fetch(cdnjs).getContentText());
I can confirm that this method can be used for the current situation (January 14, 2023). But, when the specification in the data and HTML is changed in the future update on the server side, this script might not be able to be used. Please be careful about this.
Reference:
crypto-js
I need some help with Apps Script and Google Sheets. I don't know much about coding and programming, but I'm trying to tweak a script. I have a website that sends variables to Apps Script and then posted in Google Sheets. But I want these variables in two separate groups. Let's call them Data 1 and Data 2. Data 1 goes in Sheet 1, and Data 2 goes in Sheet 2. Currently, both data are sent and processed under one function, meaning Sheet 1 and Sheet 2 are populated at the same time. Also both data include a timestamp per row. However, if there is no Data 2, Sheet 2 is still being populated with timestamps. But I only want either Sheet 1 or Sheet 2 (no timestamps or anything else in the other sheet) depending on what the website sends.
I tried to split the script into two functions, but only one ended up working, perhaps because I'm using the same function name? I tried to rename the second function, but it didn't work either, possibly because I may be breaking naming conventions or the way App Script doesn't allow two postData per script? If empty variables are not a valid factor for Apps Script to stop populating the other sheet, do I need to use conditionals, and if yes, how? I'm not sure where to go. Could you help me with this? Here's a script:
function doPost(postData) {
let doc = SpreadsheetApp.openById(MY_SHEET_ID);
let sheet = doc.getSheetByName("Sheet 1");
let sheet2 = doc.getSheetByName("Sheet 2");
let parsedData = postData.parameter;
parsedData = JSON.parse(Object.keys(parsedData));
let userBlue = parsedData["Blue"]
let userPurple = parsedData["Purple"]
let userRed = parsedData["Red"]
let userOrange = parsedData["Orange"]
sheet.appendRow([new Date(), userBlue, userPurple]);
sheet2.appendRow([new Date(), userRed, userOrange]); return ContentService.createTextOutput("Success");
}
Thank you in advance
Although I am not able to fully reproduce using a website I believe a conditional statement would work here:
function myFunction() {
const MY_SHEET_ID = "1m9nLp5sgpignwB1ddddxxxxxxxxx8";
let doc = SpreadsheetApp.openById(MY_SHEET_ID);
let sheet = doc.getSheetByName("Sheet1");
let sheet2 = doc.getSheetByName("Sheet2");
var myOB = { "Blue": 'BlueBerries', "Purple": 'grapes', "Red": 'apple', "Orange": '' }
if(myOB.Blue != "" && myOB.Purple != ""){ //Use && if you want to make sure no data is posted if both variables are empty
sheet.appendRow([new Date(),myOB.Blue,myOB.Purple])
}if(myOB.Red != "" || myOB.Orange != ""){ //Use && if you want to make sure no data is posted if both variables are empty
sheet2.appendRow([new Date(),myOB.Red,myOB.Orange])
}else{
ContentService.createTextOutput("DataSet1 Empty - No changes made");
}
}
So in your case, you can try this:
if (userBlue != "" && userPurple != "") {
sheet.appendRow([new Date(), userBlue, userPurple]);
ContentService.createTextOutput("Success");
}
else {
return ContentService.createTextOutput("Failed");
}
if (userRed != "" && userOrange != "") {
sheet2.appendRow([new Date(), userRed, userOrange]);
ContentService.createTextOutput("Success");
} else {
return ContentService.createTextOutput("Failed");
}
In the above, variables will be evaluated if they are not empty then data will be added to its respective Sheet.
Using if-else
This question already has an answer here:
How to pull Yahoo Finance Historical Price Data from its Object with Google Apps Script?
(1 answer)
Closed last month.
I'm trying to get an entire table data from https://finance.yahoo.com/quote/CL%3DF/history?p=CL%3DF. On a browser, the webpage shows 1 year data down to Oct 12, 2020 as a default. But the following code didn't pull the whole table data for some reason. It pulled only partial data, just less than 5 month data only down to May 20, 2021. What am I missing? Can anyone help fix anything wrong in the code? Thank you!
function test() {
const url = 'https://finance.yahoo.com/quote/CL%3DF/history?p=CL%3DF';
const res = UrlFetchApp.fetch(url, { muteHttpExceptions: true }).getContentText();
const $ = Cheerio.load(res);
// The URL webpage shows one year data down to Oct 12, 2021 on the browser.
// But the code below got data only down to May 20, 2020. Why am I mssing?
var data = $('table').find('td').toArray().map(x => $(x).text());
console.log(data[data.length-8]); // Print the last row date other than the web note
}
When I saw the HTML data, it seems that the table tab has not all data. But fortunately, I noticed that the object in the Javascript has all data you expect. So how about the following modified script?
Modified script:
In this modified script, the container-bound script of Spreadsheet is used. Of course, you can use the standalone type. But in that case, please modify SpreadsheetApp.getActiveSpreadsheet().
When you use this script, please copy and paste the following modified script to the script editor of Spreadsheet and set the sheet name, and run. By this, all data is retrieved and put to the Spreadsheet.
function test() {
const url = 'https://finance.yahoo.com/quote/CL%3DF/history?p=CL%3DF';
const res = UrlFetchApp.fetch(url, { muteHttpExceptions: true }).getContentText();
const $ = Cheerio.load(res);
// I modified below script
const data = $('script').toArray().reduce((ar, x) => {
const c = $(x).get()[0].children;
if (c.length > 0) {
const d = c[0].data.trim().match(/({"context"[\s\S\w]+);\n}\(this\)\);/);
if (d && d.length == 2) {
ar.push(JSON.parse(d[1]));
}
}
return ar;
}, []);
if (data.length == 0) throw new Error("No data.");
const header = ["date","open","high","low","close","adjclose","volume"];
const ar = data[0].context.dispatcher.stores.HistoricalPriceStore.prices.map(o => header.map(h => h == "date" ? new Date(o[h] * 1000) : (o[h] || "")));
const sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1"); // <--- Please set the sheet name you want to put the values.
sheet.getRange(1, 1, ar.length, ar[0].length).setValues(ar);
}
Result:
When the above script is run, the following result is obtained.
References:
reduce()
map()
I want to retrieve a table from the URL of https://s.cafef.vn/screener.aspx#data using VBA. This task is difficult because the table contains JSON data embedded in an html file, but it was so kind of Tanaike, an GAS expert who helped me to create a custom function for Google Apps Scripts.
(IMPORTHTML() doesn't work in this webpage structure)
The function looks like:
function SAMPLE(url) {
const res = UrlFetchApp.fetch(url, {muteHttpExceptions: true});
const html = res.getContentText().match(/var jsonData \=([\S\s\w]+\}\])/);
if (!html) return "No tables. Please confirm URL again.";
const table = JSON.parse(html[1].replace(/\n/g, ""));
const header = ["", "FullName", "Symbol", "CenterName", "ChangePrice", "VonHoa", "ChangeVolume", "EPS", "PE", "Beta", "Price"];
return table.reduce((ar, e, i) => {
const temp = header.map(f => f == "" ? i + 1 : e[f]);
ar.push(temp);
return ar;
}, [header]); }
This function works perfect in Google Sheets environment, but my goal now is to convert it into VBA, or in other words, writing a VBA module which can get the table at https://s.cafef.vn/screener.aspx#data.
Many thanks for any help or suggestions
Cao Doremi
This question already has answers here:
Scraping data to Google Sheets from a website that uses JavaScript
(2 answers)
Closed last month.
I'm attempting to scrape options pricing data from Yahoo Finance in Google Sheets. Although I'm able to pull the options chain just fine, i.e.
=IMPORTHTML("https://finance.yahoo.com/quote/TCOM/options?date=1610668800","table",2)
I find that it's returning results that don't completely match what's actually shown on Yahoo Finance. Specifically, the scraped results are incomplete - they're missing some strikes. i.e. the first 5 rows of the chart may match, but then it will start returning only every other strike (aka skipping every other strike).
Why would IMPORTHTML be returning "abbreviated" results, which don't match what's actually shown on the page? And more importantly, is there some way to scrape complete data (i.e. that doesn't skip some portion of the available strikes)?
In Yahoo finance, all data are available in a big json called root.App.main. So to get the complete set of data, proceed as following
var source = UrlFetchApp.fetch(url).getContentText()
var jsonString = source.match(/(?<=root.App.main = ).*(?=}}}})/g) + '}}}}'
var data = JSON.parse(jsonString)
You can then choose to fetch the informations you need. Take a copy of this example https://docs.google.com/spreadsheets/d/1sTA71PhpxI_QdGKXVAtb0Rc3cmvPLgzvXKXXTmiec7k/copy
edit
if you want to get a full list of available data, you can retrieve it by this simple script
// mike.steelson
let result = [];
function getAllDataJSON(url = 'https://finance.yahoo.com/quote/TCOM/options?date=1610668800') {
var source = UrlFetchApp.fetch(url).getContentText()
var jsonString = source.match(/(?<=root.App.main = ).*(?=}}}})/g) + '}}}}'
var data = JSON.parse(jsonString)
getAllData(eval(data),'data')
var sh = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet()
sh.getRange(1, 1, result.length, result[0].length).setValues(result);
}
function getAllData(obj,id) {
const regex = new RegExp('[^0-9]+');
for (let p in obj) {
var newid = (regex.test(p)) ? id + '["' + p + '"]' : id + '[' + p + ']';
if (obj[p]!=null){
if (typeof obj[p] != 'object' && typeof obj[p] != 'function'){
result.push([newid, obj[p]]);
}
if (typeof obj[p] == 'object') {
getAllData(obj[p], newid );
}
}
}
}
Here's a simpler way to get the last market price of a given option. Add this function to you Google Sheets Script Editor.
function OPTION(ticker) {
var ticker = ticker+"";
var URL = "finance.yahoo.com/quote/"+ticker;
var html = UrlFetchApp.fetch(URL).getContentText();
var count = (html.match(/regularMarketPrice/g) || []).length;
var query = "regularMarketPrice";
var loc = 0;
var n = parseInt(count)-2;
for(i = 0; i<n; i++) {
loc = html.indexOf(query,loc+1);
}
var value = html.substring(loc+query.length+9, html.indexOf(",", loc+query.length+9));
return value*100;
}
In your google sheets input the Yahoo Finance option ticker like below
=OPTION("AAPL210430C00060000")
I believe your goal as follows.
You want to retrieve the complete table from the URL of https://finance.yahoo.com/quote/TCOM/options?date=1610668800, and want to put it to the Spreadsheet.
Issue and workaround:
I could replicate your issue. When I saw the HTML data, unfortunately, I couldn't find the difference of HTML between the showing rows and the not showing rows. And also, I could confirm that the complete table is included in the HTML data. By the way, when I tested it using =IMPORTXML(A1,"//section[2]//tr"), the same result of IMPORTHTML occurs. So I thought that in this case, IMPORTHTML and IMPORTXML might not be able to retrieve the complete table.
So, in this answer, as a workaround, I would like to propose to put the complete table parsed using Sheets API. In this case, Google Apps Script is used. By this, I could confirm that the complete table can be retrieved by parsing the HTML table with Sheet API.
Sample script:
Please copy and paste the following script to the script editor of Spreadsheet, and please enable Sheets API at Advanced Google services. And, please run the function of myFunction at the script editor. By this, the retrieved table is put to the sheet of sheetName.
function myFunction() {
// Please set the following variables.
const url ="https://finance.yahoo.com/quote/TCOM/options?date=1610668800";
const sheetName = "Sheet1"; // Please set the destination sheet name.
const sessionNumber = 2; // Please set the number of session. In this case, the table of 2nd session is retrieved.
const html = UrlFetchApp.fetch(url).getContentText();
const section = [...html.matchAll(/<section[\s\S\w]+?<\/section>/g)];
if (section.length >= sessionNumber) {
if (section[sessionNumber].length == 1) {
const table = section[sessionNumber][0].match(/<table[\s\S\w]+?<\/table>/);
if (table) {
const ss = SpreadsheetApp.getActiveSpreadsheet();
const body = {requests: [{pasteData: {html: true, data: table[0], coordinate: {sheetId: ss.getSheetByName(sheetName).getSheetId()}}}]};
Sheets.Spreadsheets.batchUpdate(body, ss.getId());
}
} else {
throw new Error("No table.");
}
} else {
throw new Error("No table.");
}
}
const sessionNumber = 2; means that 2 of =IMPORTHTML("https://finance.yahoo.com/quote/TCOM/options?date=1610668800","table",2).
References:
Method: spreadsheets.batchUpdate
PasteDataRequest