Use importData to fetch and parse JSON - google-apps-script

Question

I had the following function running perfectly:
var ss = SpreadsheetApp.getActiveSpreadsheet();
var habSheet = ss.getSheetByName("Harvests");
var bVals = habSheet.getRange("b2:b").getValues();
var habs = bVals.filter(String).length;
var habitats = habSheet.getRange("B2:B"+habs+1).getDisplayValues();
var data = [];
var traitNames = habSheet.getRange("D1:U1").getValues();
var values = new Array(habs);
for (i = 0; i < habs; i++) {
values[i] = new Array(traitNames[0].length);
for (j=0; j<traitNames[0].length; j++){
values[i][j] = [""];
}
}
var rawData = "";
var names = new Array(habs);
for (i = 0; i < habs; i++) {
names[i] = new Array(1);
}
for (i=0; i<habs; i++){
try{
rawData = UrlFetchApp.fetch("https://api.genopets.me/habitat/"+habitats[i]);
data[i] = JSON.parse(rawData.getContentText());
names[i][0] = data[i].name;
for (j=0; j<data[i].attributes.length; j++){
value = data[i].attributes[j].value;
trait = data[i].attributes[j].trait_type;
for (k=0; k<=21; k++){
if (traitNames[0][k] == trait){
values[i][k] = value;
}
}
}
}
catch(err){
But I'm exceeding max fetch calls daily. I'm in an emergency situation because this needs to run again within an hour.
I'm trying to build a temporary fix, so I'm using importData to call the API with the following formula:
=join(",",IMPORTDATA("https://api.genopets.me/habitat/"&B2,","))
Then, I want to just replace rawData in the code with this imported data. However, now it comes in as text and can't be parsed in the same way. Is there a quick way to force it into JSON format or otherwise convert to a dictionary as before so that I can parse it with the same code?
I'm getting stuck because .name, .length, etc. are failing as the "rawData" is now just a string.
This is the code snippet I'm playing with to try and get this right and build the quick patch for right now:
// for (i=0; i<habs; i++){
var i=0;
importData = habSheet.getRange("AL1").getDisplayValue();
rawData = JSON.stringify(importData);
// Logger.log(rawData);
data[i] = rawData;
// data[i] = JSON.parse(rawData.getContentText());
names[i][0] = data[i].name;
for (j=0; j<data[i].attributes.length; j++){
value = data[i].attributes[j].value;
trait = data[i].attributes[j].trait_type;
for (k=0; k<=21; k++){
if (traitNames[0][k] == trait){
values[i][k] = value;
}
}
}
I've tried as above, and also without stringify, but I can't get this yet.
For reference, this is an example of the API response:
https://api.genopets.me/habitat/7vTz9dniU14Egpt8XHkMxP1x36BLRd15C11eUTaWhB19
Appreciate any help!

Answer 1

I have done a lot of testing to find a simple workaround, but could not find one, the string resulting from the =join(",",IMPORTDATA(url,",")) (and none of the other =IMPORTXXX functions) will work for your code. When using these IMPORT functions the data is interpreted and certain characters are removed or the values formatted, it is NOT recommended to use these functions.
Since you mentioned the message you are getting is related to quota limits you should consider splitting the load of this script in multiple Apps Script projects. As a possible immediate solution you can make a copy of the script (or file bound to the script), authorize the new copy and try again.
To increase performance you could try using the calls in bulk, use this other function fetchAll (https://developers.google.com/apps-script/reference/url-fetch/url-fetch-app#fetchallrequests). There is a 100 request limit for this method. This will result in the same quota usage.