I make a request to Go to Webinar API in Apps Script and the response is something like this:
{"joinUrl":"https://example.com","asset":true,"registrantKey":3669516009270899211,"status":"APPROVED"}
When I make a JSON.parse i get something like this:
{joinUrl=https://example.com, asset=true, registrantKey=3.6695160092708992E18, status=APPROVED}
The RegistrantKey changes, I dont know why.
Thats my code:
try{
var resp = UrlFetchApp.fetch(url,options)
var json=JSON.parse(resp);
return json
}catch(err){
Logger.log(err);
return err;
}
How about this answer? Please think of this as just one of several possible answers.
Issue and workaround:
I think that the reason of your issue is that 3669516009270899211 of "registrantKey":3669516009270899211 is used as the number. In Javascript, the maximum integer value is 9007199254740991. Ref So when 3669516009270899211 is converted to the number, it becomes 3.6695160092708992E18 and when this is converted to the string, it becomes 3669516009270899000.
So in this case, as one of several workaround, how about enclosing 3669516009270899211 by the double quotes like "registrantKey":"3669516009270899211"? By this, "3669516009270899211" is used ad the string, and you can retrieve the value of 3669516009270899211.
Modified script:
When your script is modified, how about the following modification?
From:
var resp = UrlFetchApp.fetch(url,options)
var json=JSON.parse(resp);
return json
To:
var resp = UrlFetchApp.fetch(url,options);
resp = resp.getContentText().replace(/registrantKey":(\d.+),/, "registrantKey\":\"$1\","); // Added
var json=JSON.parse(resp);
return json
Note:
As the test script, you can also use the following script.
var str = '{"joinUrl":"https://example.com","asset":true,"registrantKey":3669516009270899211,"status":"APPROVED"}';
str = str.replace(/registrantKey":(\d.+),/, "registrantKey\":\"$1\",");
var obj = JSON.parse(str);
Logger.log(obj)
Reference:
Number.MAX_SAFE_INTEGER
If I misunderstood your question and this was not the direction you want, I apologize.
Related
considering the website: https://www.coindesk.com/price/bitcoin/
I'm trying to extract the value $23,073.15 (that changes continuosly) of the following span class.
span class="typography__StyledTypography-owin6q-0 fZpnIj">$23,073.15</span
I tried the following code with Google script using Cheerio without success:
var URL = "https://www.coindesk.com/price/bitcoin/";
var response = UrlFetchApp.fetch(URL);
var $ = Cheerio.load(response.getContentText());
var itemsOfInterest = $('span.typography__StyledTypography-owin6q-0 fZpnIj').text();```
any help about?
When I saw the HTML data from your URL using Google Apps Script, unfortunately, it seems that the data is retrieved from an API by Javascript. By this, in this case, your expected value cannot be retrieved from your URL with cheerio. So, in this answer, I would like to propose retrieving your expected value using the API. The sample script is as follows.
Sample script:
function sample() {
const url = "https://production.api.coindesk.com/v2/tb/price/ticker?assets=BTC";
const res = UrlFetchApp.fetch(url);
const obj = JSON.parse(res.getContentText());
const value = obj.data.BTC.ohlc.l;
console.log(value)
}
I think that when this script is run, you can see your expected value in the log.
I am fetching data from an api to google sheet. all its getting is data string on a single cell. How can I format it into a table with headers. Part of the long string I am getting and the code are below.
String - {instantBuy=5679.95, pricechange=0, instantSell=5623.39, currency=AUD, 24hoursHigh=5848.86, market=5848.86, virtualCurrency=ETH, 24hoursLow=5848.86, sell=5848.86, volume=0.341947, buy=5848.86, pair=ETH-AUD} ...
Code:
function myQuotes() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var mainSheet = ss.getSheetByName('Sheet1');
var url = "https://www.zebapi.com/pro/v1/market/";
var response = UrlFetchApp.fetch(url);
var json = response.getContentText();
var data = JSON.parse(json);
mainSheet.getRange(4,2).setValue(data);
Thanks!
Use Object.entries(), like this:
function myQuotes() {
const url = "https://www.zebapi.com/pro/v1/market/";
const response = UrlFetchApp.fetch(url);
const json = response.getContentText();
const data = transpose_(Object.entries(JSON.parse(json)));
const targetRange = SpreadsheetApp.getActive().getRange('Sheet1!B4');
targetRange.offset(0, 0, data.length, data[0].length).setValues(data);
}
function transpose_(a) {
// #see https://stackoverflow.com/a/13241545/1536038
return Object.keys(a[0]).map(c => a.map(r => r[c]));
}
You may also want to take a look at ImportJSON.
I'm not very familiar with JavaScript, but you could try the search() method, which I believe returns the index at which the word is found. So for example, if you were trying to put data from "instantBuy", and put it in it's own row or column, you could do search("instantBuy"), get the string between that and the next index that search will return, search("pricechange")
Sorry if that was confusing!
I'm working on a small project using Google App Script now. What I need to do is make both GET and POST request to API. I'm working on a GET request for now and wrote the code below.
var myApiKey = "123456789"
const requestServer = (path, method, params = {}) => {
if (method.toUpperCase() === "GET"){
var url = "https://api.xyz.com/" + path + "apikey=" + myApiKey;
var response = UrlFetchApp.fetch(url);
var json = response.getContentText();
var data = JSON.parse(json);
} else if (method.toUpperCase() === "POST"){
}
}
I used fake url and apikey for asking this question.
My question is if there is a place or a way to store myApiKey in Google App Script, like .env file?
If so how should I store and make my key hidden...
In order to achieve your goal, how about the following 3 patterns.
Pattern 1:
In this pattern, how about using Properties Service? Ref When the script is run, the value can be put and retrieved using Properties Service.
In this case, at first, it is required to input the value to Properties Service.
Sample script:
var scriptProperties = PropertiesService.getScriptProperties();
// Put value.
scriptProperties.setProperty("key", "value");
// Get value.
var value = scriptProperties.getProperty("key");
Pattern 2:
In this pattern, how about using the custom file properties? Ref This can be put and retrieve the value using Drive API.
In this case, at first, it is required to input the value to the custom file properties.
Sample script:
var fileId = ScriptApp.getScriptId();
// Put value.
Drive.Files.update({properties: [{key: "key", value: "value"}]}, fileId);
// Get value.
var savedKey = "key";
var obj = Drive.Files.get(fileId).properties.filter(({key}) => key == savedKey);
if (obj.length == 0) throw new Error("No property for the inputted key");
var value = obj[0].value;
Pattern 3:
In this pattern, how about using a temporal file? In this case, this can be used like .env file. At first, the value is put to the file (in this case, I think that the text file and Google Docs file can be used.). And, when the script is run, the value is retrieved from the file and use it in the script.
Sample script:
// Put value.
DriveApp.createFile("sample.txt", "value");
// Get value.
var value = DriveApp.getFileById("###fileId###").getBlob().getDataAsString().trim();
References:
Properties Service
Add custom file properties
I am trying to scrape a table of price data from this website using the following code;
function scrapeData() {
// Retrieve table as a string using Parser.
var url = "https://stooq.com/q/d/?s=barc.uk&i=d";
var fromText = '<td align="center" id="t03">';
var toText = '</td>';
var content = UrlFetchApp.fetch(url).getContentText();
var scraped = Parser.data(content).from(fromText).to(toText).build();
//Parse table using XmlService.
var root = XmlService.parse(scraped).getRootElement();
}
I have taken this method from an approach I used in a similar question here however its failing on this particular url and giving me the error;
Error on line 1: Content is not allowed in prolog. (line 12, file "Stooq")
In related questions here and here they talk of textual content that is not accepted being submitted to the parser however, I am unable to apply the solutions in these questions to my own problem. Any help would be much appreciated.
How about this modification?
Modification points:
In this case, it is required to modify the retrieved HTML values. For example, when var content = UrlFetchApp.fetch(url).getContentText() is run, each attribute value is not enclosed. These are required to be modified.
There is a merged column in the header.
When above points are reflected to the script, it becomes as follows.
Modified script:
function scrapeData() {
// Retrieve table as a string using Parser.
var url = "https://stooq.com/q/d/?s=barc.uk&i=d";
var fromText = '#d9d9d9}</style>';
var toText = '<table';
var content = UrlFetchApp.fetch(url).getContentText();
var scraped = Parser.data(content).from(fromText).to(toText).build();
// Modify values
scraped = scraped.replace(/=([a-zA-Z0-9\%-:]+)/g, "=\"$1\"").replace(/nowrap/g, "");
// Parse table using XmlService.
var root = XmlService.parse(scraped).getRootElement();
// Retrieve header and modify it.
var headerTr = root.getChild("thead").getChildren();
var res = headerTr.map(function(e) {return e.getChildren().map(function(f) {return f.getValue()})});
res[0].splice(7, 0, "Change");
// Retrieve values.
var valuesTr = root.getChild("tbody").getChildren();
var values = valuesTr.map(function(e) {return e.getChildren().map(function(f) {return f.getValue()})});
Array.prototype.push.apply(res, values);
// Put the result to the active spreadsheet.
var ss = SpreadsheetApp.getActiveSheet();
ss.getRange(1, 1, res.length, res[0].length).setValues(res);
}
Note:
Before you run this modified script, please install the GAS library of Parser.
This modified script is not corresponding to various URL. This can be used for the URL in your question. If you want to retrieve values from other URL, please modify the script.
Reference:
Parser
XmlService
If this was not what you want, I'm sorry.
I have created a sheet to keep my crypto holdings. I use this importJSON function I found on youtube : (I have changed the help text for myself)
/**
* Imports JSON data to your spreadsheet Ex: IMPORTJSON("https://api.coinmarketcap.com/v2/ticker/1/?convert=EUR","data/quotes/EUR/price")
* #param url URL of your JSON data as string
* #param xpath simplified xpath as string
* #customfunction
*/
function IMPORTJSON(url,xpath){
try{
// /rates/EUR
var res = UrlFetchApp.fetch(url);
var content = res.getContentText();
var json = JSON.parse(content);
var patharray = xpath.split("/");
//Logger.log(patharray);
for(var i=0;i<patharray.length;i++){
json = json[patharray[i]];
}
//Logger.log(typeof(json));
if(typeof(json) === "undefined"){
return "Node Not Available";
} else if(typeof(json) === "object"){
var tempArr = [];
for(var obj in json){
tempArr.push([obj,json[obj]]);
}
return tempArr;
} else if(typeof(json) !== "object") {
return json;
}
}
catch(err){
return "Error getting data";
}
}
I use this function to readout an API :
This is a piece of my script :
var btc_eur = IMPORTJSON("https://api.coinmarketcap.com/v2/ticker/1/?convert=EUR","data/quotes/EUR/price");
var btc_btc = IMPORTJSON("https://api.coinmarketcap.com/v2/ticker/1/?convert=BTC","data/quotes/BTC/price");
ss.getRange("B2").setValue([btc_eur]);
ss.getRange("H2").setValue([btc_btc]);
var bhc_eur = IMPORTJSON("https://api.coinmarketcap.com/v2/ticker/1831/?convert=EUR","data/quotes/EUR/price");
var bhc_btc = IMPORTJSON("https://api.coinmarketcap.com/v2/ticker/1831/?convert=BTC","data/quotes/BTC/price");
ss.getRange("B3").setValue([bhc_eur]);
ss.getRange("H3").setValue([bhc_btc]);
The last few days I get "Error getting data" errors. When I start manualy the script it works.
I than tried this code I found here :
ImportJson
function IMPORTJSON(url,xpath){
var res = UrlFetchApp.fetch(url);
var content = res.getContentText();
var json = JSON.parse(content);
var patharray = xpath.split("/");
var res = [];
for (var i in json[patharray[0]]) {
res.push(json[patharray[0]][i][patharray[1]]);
}
return res;
}
But this gives an error about : TypeError: Cannot read property "quotes" from null. What am I doing wrong ?
The big problem is your script call API at least 4 times. When few users do it too, the Google server call API too much times.
The API of Coinmarketcap has limited bandwidth. When any client reach this limit, the API return HTTP error 429. Google Scripts is on shared Google servers, that means lot of users looks as one client for Coinmarketcap API.
When API decline your request, your script fails – the error message corresponds to the assumed error (xpath cant find quotes component in empty varible).
This is ruthless behavior. Please, don't ruin API via mass calls.
You can load data from API at once and re-use it angain for each finding in data.
I have similar Spreadsheet automatically filled from Coinmarketcap API, you can copy it for your:
Coins spreadsheet
Google Script on GitHub.
This my script is strictly ask API only once for whole runtime and reusing one response for all queries.
Change of your script
Also you can make few changes in your Code for saving resources:
Change IMPORTJSON function from this:
function IMPORTJSON(url,xpath){
var res = UrlFetchApp.fetch(url);
var content = res.getContentText();
var json = JSON.parse(content);
...
to this:
function IMPORTJSON(json, xpath) {
...
and rutime section of code you can change like this:
var res = UrlFetchApp.fetch("https://api.coinmarketcap.com/v2/ticker/1/?convert=EUR");
var content = res.getContentText();
var json = JSON.parse(content);
var btc_eur = IMPORTJSON(json,"data/quotes/EUR/price");
var btc_btc = IMPORTJSON(json,"data/quotes/BTC/price");
ss.getRange("B2").setValue([btc_eur]);
ss.getRange("H2").setValue([btc_btc]);
...
Main benefit is: the UrlFetchApp.fetch is called only once.
Yes, I know, this code is not works 1:1 like your. That because that receive prices only for EUR and not for BTC. Naturally fetching comparation between BTC and BTC is unnecessary because it is always 1 and other values you can count matematically from EUR response – please don't abuse an api for such queries.
As Jakub said, the main issue is that all requests are counted as coming from the same Google server.
One solution which I consider easier is to put a proxy server in the middle, this can be done by either purchasing a server and setting it up (which is quite complex) or using a service like Proxycrawl which includes some free requests and after that, unless you run thousands of queries per month, it should cost you less than 1 USD per month.
To do that you just need to edit one line of the script:
var res = UrlFetchApp.fetch(url);
This line, becomes this:
var res = UrlFetchApp.fetch(`https://api.proxycrawl.com/?token=YOUR_TOKEN&url=${encodeURIComponent(url)}`);
Make sure to replace YOUR_TOKEN with your actual service token
Just this simple change will make the requests never fail as each request will be sent from a different IP instead of all coming from Google.