How to extract data from website to Google Sheet [duplicate] - html

Question

I try to print historic adjusted close prices from Yahoo finance to Google Sheets.
=ImportXML("https://sg.finance.yahoo.com/quote/"&B57&"/history?p="&B57, "//tbody/tr[21]/td[6]")
Cell B57 is for example "SPY".
This works fine for historic prices up to 100 days. (it is adjusted here: tr[100])
When I try to get prices later 100 days it returns "N/A".
These prices are visible on yahoo finance.
It there a way to adjust XPATH that it works?
I noticed, that in the html code of yahoo pices about 100 days don't have this "data-reactid=1520" in the tr tag.

Answer 1

In the current stage, it seems that your expected values are included in the HTML data as a JSON object for Javascript. In this case, when the JSON object is retrieved with Google Apps Script, the value can be retrieved. When this is reflected in a sample Google Apps Script, how about the following sample script?
Sample script:
Please copy and paste the following script to the script editor of Google Spreadsheet and save the script. When you use this script, please put a custom function of =SAMPLE("https://sg.finance.yahoo.com/quote/SPY/history?p=SPY") to a cell. By this, the script is run.
function SAMPLE(url) {
const html = UrlFetchApp.fetch(url).getContentText().match(/root.App.main = ([\s\S\w]+?);\n/);
if (!html || html.length == 1) return "No data";
const tempObj = JSON.parse(html[1].trim());
const obj = tempObj.context.dispatcher.stores;
const header = ["date", "amount", "open", "high", "low", "close", "adjclose", "volume"];
return [header, ...obj.HistoricalPriceStore.prices
.map(o => header.map(h => {
if (h == "date") {
return new Date(o[h] * 1000)
} else if (h == "amount" && o[h]) {
return `${o[h]} ${o.type}`;
}
return o[h];
}))];
}
Testing:
When this script is run with =SAMPLE("https://sg.finance.yahoo.com/quote/SPY/history?p=SPY"), the following result is obtained.
Note:
The above script is for a custom function. If you want to use this script with the script editor, you can also the following sample script.
function myFunction() {
const url = "https://sg.finance.yahoo.com/quote/SPY/history?p=SPY"; // This URL is from your question.
const html = UrlFetchApp.fetch(url).getContentText().match(/root.App.main = ([\s\S\w]+?);\n/);
if (!html || html.length == 1) return;
const tempObj = JSON.parse(html[1].trim());
const obj = tempObj.context.dispatcher.stores;
const header = ["date", "amount", "open", "high", "low", "close", "adjclose", "volume"];
const values = [header, ...obj.HistoricalPriceStore.prices
.map(o => header.map(h => {
if (h == "date") {
return new Date(o[h] * 1000)
} else if (h == "amount" && o[h]) {
return `${o[h]} ${o.type}`;
}
return o[h];
}))];
const sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1"); // Please set your sheet name.
sheet.getRange(sheet.getLastRow() + 1, 1, values.length, values[0].length).setValues(values);
}
Note:
If const obj = tempObj.context.dispatcher.stores is the salted base64 data, please check this answer.
References:
Custom Functions in Google Sheets
map()

Answer 2

not possible because yahoo site uses JavaScript element - the infinity scroll - which kicks in after 100th value and that's the reason why you can't get past that point. you can test this by disabling JS for a given site and what's left can be scraped:

Answer 3

It's possible with a workaround :
Later than 100 days :
Cell with green background : the code to search
Cells with orange backgound : cells containing formulas
Cells with yellow background : data returned
Formulas used :
=IMPORTXML(A1;"substring-before(substring-after(//script[#id='fc'],'{""prices"":'),',""isPending')")
=SUBSTITUE(SUBSTITUE(SUBSTITUE(A3;"},{";"|");",";";");".";",")
=REGEXREPLACE(A4;"[a-z:{}\[\]""]+";"")
=TRANSPOSE(SPLIT(A5;"|"))
=(((C8/60)/60)/24)+DATE(1970;1;1)
IMPORTXML to import the data.
SUBSTITUE AND REGEXREPLACE to prepare the TRANSPOSE step.
TRANSPOSE to "build" the lines and SPLIT to "build" the columns.
DATE to transform timestamp to date.
Sheet

Answer 4

Answer:
IMPORTXML can not retrieve data which is populated by a script, and so using this formula to retrieve data from this table is not possible to do.
More Information:
As the first 100 values are loaded into the page without the use of JavaScript (as you can see by disabling JavaScript for https://sg.finance.yahoo.com/quote/SPY/history?p=SPY and reloading the page), the information can be retrieved by IMPORTXML.
As the data after the first 100 results is generated on-the-fly after scrolling down the page, the newly available data is not retrievable by IMPORTXML - as far as the formula sees, there is no 101st <tr> element and so it displays N/A: Imported content is empty .
References:
IMPORTXML - Docs Editors Help
Related Questions:
Google Sheets importXML Returns Empty Value