Google Sheets Script Timeout Workaround - google-apps-script

I'm building a script that uses a looping ImportHTML command to web scrape weather data based on zip code, and am currently running into an issue with the execution timing out every time the script is run.
The current way I have the script set up produces a correct result when run, but given that the script is pulling data from several hundred sources, it is taking a while and will not complete within the current time limit of Google scripts.
The sheet running the script utilizes 3 tabs:
ZIPS, which contains a list of zip code link values pulled from the
site that weather data is to be pulled from
Blank, which is simply an intermediary sheet used in the execution of the script
Result, where the final output is to be placed
In order to try and reduce the amount of read/write as much as possible, I changed the code from writing each result of the ImportHTML command as it was executed to appending to an array and writing this array at the end of the script. The code in it's current form is as follows:
function getTemps() {
var googleSheet = SpreadsheetApp.getActive();
// Read in Zip code link values
var sheet = googleSheet.getSheetByName('ZIPS');
var zipArray = sheet.getDataRange().getValues();
var arrayLength = zipArray.length;
//Set up sheet values
var blankSyntaxA = 'ImportHtml("https://www.wunderground.com/cgi-bin/findweather/getForecast?query=pz:';
var blankSyntaxB = '&zip=1", "table", 1)';
var tempResult = [];
// Writing Section
var sheet = googleSheet.getSheetByName('Blank');
for (var i = 0; i < arrayLength; i++)
{
var liveSyntax = blankSyntaxA+zipArray[i][0]+blankSyntaxB;
sheet.getRange('A1').setFormula(liveSyntax);
var importedData = sheet.getDataRange().getValues();
tempResult = tempResult.concat(importedData);
}
var sheet = googleSheet.getSheetByName('Result');
sheet.getRange(1,1,tempResult.length,8).setValues(tempResult);
}
I know the run time of the script could be reduced by eliminating the read/write contained within the For loop, but I'm not sure how to obtain the necessary HTML table without running the ImportHTML command within the 'Blank' sheet. Is there a way to run that command to fill the 'importedData' array without writing to a sheet?
Alternatively, I had considered utilizing a check on the runtime of the function and implementing a break as it neared the ~5 minute runtime limit, followed by a recursive call back to the original function, but I wasn't sure if this would actually mitigate the runtime issue, or even be possible given the nature of the recursive call.
Any advice on how this script could be modified to run within the script timeout parameter or modified to produce the complete desired outcome with all the necessary imported data would be appreciated. Thanks!

Is there a way to run that command to fill the 'importedData' array without writing to a sheet?
IMPORTHMTL is a Google Sheets built-in spreadsheet function. This kind of functions can't be ran / evaluated by Google Apps Script.
Related
How to evaluate a spreadsheet formula within a custom function?
Alternatively, I had considered utilizing a check on the runtime of the function and implementing a break as it neared the ~5 minute runtime limit, followed by a recursive call back to the original function, but I wasn't sure if this would actually mitigate the runtime issue, or even be possible given the nature of the recursive call.
Rather than a "mitigator" this is a workaround. There are several techniques like batch processing and parallel processing.
Reference
Exceeded maximum execution time in Google Apps Script
From answer to Threading in Google App Script
There is a great example from Bruce
Mcphearson. His example
Parallel Processing in Apps
Script uses
Map Reduce in exercise. He is utilizing triggers as well, but perhaps
may provide some different perspective.
Another alternative is to sign in to the Early Access Program to extend the execution time limit to 30 minutes.

Related

Run formula calculation in Google Sheets without to open file

I have a scheduled addon, which writes data periodically into a Google Sheet, without to open file. I have a formula, which should make some calculations on written data and write calculated values into the sheet, so then Data Studio with this sheet as data source updates the visualization.
What is the way to periodically run a formula calculation and write calculated values without manual steps like opening file?
I'm pretty new to this topic: my search approaches to find something like convert formula to app script or schedule formula execution brought me not to any fruitful idea.
PS_ the formula, which should do the calculation is:
=IFERROR(((VLOOKUP(A2,'[1]SV'!$A:$B,2.0))/B2)+(IF(B2=1,"33,9",IF(B2=2,"16,28",IF(B2=3,"10,36",IF(B2=4,"7",IF(B2=5,"5,64")))))+IF(B2=6,"4,13",IF(B2=7,"3,27",IF(B2=8,"2,61",IF(B2=9,"2,18",IF(B2=10,"1,82")))))+IF(B2=11,"1,77",IF(B2=12,"1,81",IF(B2=13,"1,85",IF(B2=14,"1,9",IF(B2=15,"2,04")))))+IF(B2=16,"1,68",IF(B2=17,"1,61",IF(B2=18,"1,65",IF(B2=19,"1,62",IF(B2=20,"1,59","0")))))),0)
If i would know, how to convert the formula to app script, i would manage the rest - i'm familiar with running scripts with time based trigger. Or, maybe, there is a method to run formula on the same scheduled way, like scripts...?
*Disclaimer: I have not tried this and it might not work, if it doesn't, comment and I will try to provide another solution
OK here's what i would try:
In google scripts there is a trigger function, manly used for testing. what it does is it runs your script and sends you an email if there is a error, and hypothetically, you could use this to run you script periodically.
A couple of other things I may try if this doesn't work, leave a comment if you would like me to go in depth about these:
depending on how you want this to work, you could set up a web app to do this, and instead of opining google sheets open the web app with a timer built into the script, that would trigger as long as you had the web page opened.
It might also be possible to run the script automatically with a JavaScript function, requires more research.
If Google Data Studio is not showing you the correct value then you could use a time-driven trigger to save the formula results in another cell. Example:
function respondToTimeDrivenTrigger(e){
var spreadsheet = SpreadsheetApp.getActiveSpreadsheet();
/** The range holding the formula */
var range = spreadsheet.getRange('Sheet1!A1');
/** The value returned by the formula */
var value = range.getValue();
/** Write the value to the cell to the right or the source cell */
range.offset(0,1).setValue(value);
}
Resources
https://developers.google.com/apps-script/guides/sheets
https://developers.google.com/apps-script/guides/triggers/installable

Stop Googe Sheet from refreshing HTML data

I have a sheet that pulls alot of finance data from alot of web pages using HTML. Problem is that it slows down alot. Plus it starts giving errors due to high number of HTMLs and etc.. i looked for the following solutions:
Having an app script to make refreshes controlled..(not so effective)
Any way i could copy paste the old data automatically till new data comes in and updates old pasted values..(also not effective as i couldnt find an automated method)
Tried finding a way to play with google sheet auto-calculations to help....(also failed in that)
IS there a way to ristrict the sheet from auto refreshing so many times...?
In the documentation you can see that:
Functions that pull data from outside the spreadsheet recalculate at the following times:
ImportRange: 30 minutes
ImportHtml, ImportFeed, ImportData, ImportXml: 1 hour
GoogleFinance: may be delayed up to 20 minutes
If you want to recalculate the value for your importHTML functions with a lower frequency you should definitely use Apps Script to do the data fetch and then populate your Spreadsheet with the information.
You can use UrlFetchApp class on Apps Script to get the data and define your timing logic for the updates.
function myAppScriptFunction() {
var urls = ["your-website-url1", "...", "your-website-urlN"];
var response = UrlFetchApp.fetchAll(urls);
//... Parse the response and store the information you need in a table-like data structure
// Let's assume the variable parsedResponseData is created
var ss = SpreadsheetApp.getActiveSheet();
ss.getRange("your-range").setValues(parsedResponseData);
}
You should now wrap your function in your custom time logic to manage the updates.
You can achieve this with a time-driven trigger on Apps Script.
function triggerSetup() {
ScriptApp.newTrigger('myAppScriptFunction')
.timeBased()
.everyHours(6)
.create();
}
References:
Time-driven triggers
UrlFetchApp
Quota Limits
you may try this addon which freezes your spreadsheet on demand:
https://gsuite.google.com/marketplace/app/spreadsheet_freezer/526561533622

Is there a way to convert google-class (objects) into Strings? (In Google App Scripts)

For Example, a common workflow that I use is :
var spreadsheet = SpreadsheetApp.openById(spreadsheetId);
var sheet = spreadsheet.getSheetByName(sheetName);
I call these 2 functions every time I want to read/write to my sheet. However, if there was a way to cache the result of
SpreadsheetApp.openById(spreadsheetId), which returns a Spreadsheet Object, then I would assume to have increased performance, as it forgoes function calls. So far, both PropertiesService and Cache only support strings, hence the need for a conversion. Using JSON.stringify on spreadsheet or sheet doesn't seem to work... any ideas?

App script access to data when loading data in spreadsheet via HTML import

I am using the Spreadsheet Services in Google app script to retrieve some data from the Internet and then mess a bit with it. The problem is when I set the ImportHtml value if the data set is larger than say a few rows I do not have access right away to the imported range and thus an error is thrown in the script:
example:
// create tmp sheet and import some data.
var sheet = this.createTmpSheet(); // custom method to create a temp sheet.
sheet.getRange('A1').setValue('=ImportHtml("someUrl","table",1)');
// at this point usually I can access the range
var range_to_copy = sheet.getDataRange();
// However if the data is more than 10-15 rows I get invalid dimention for range..
Any ideas how to wait for the 'readiness' of the import? None of the usuall triggers seemd like an appropriate choice. All I need is to have flow control in such a way as to be notified once the import completes, usually under 10 seconds.
As you noted, there's no trigger that will tell you when a spreadsheet recalculation has completed. (That's what is going on after you update a formula in a cell.)
You can induce your script to wait 10 seconds by using Utilities.sleep(10000);. Generally it's bad programming practice to rely on delays, but it's almost your only option here.
The other option would be to perform the html query yourself, parse the table of data into an array, and write it to the new sheet. If you're interested in that, have a look at html div nesting? using google fetchurl, to see how you could obtain your table of information.
Dont do it like that. Use urlFetch to get the data and write it yourself to the spreadsheet.

How to pause App Scripts until spreadsheet finishes calculation

Because google spreadsheets does not support iterations, I wrote my own simple app script to adjust an input based upon the calculation of the spreadsheet output. However, after I change the input variable, the spreadsheet recalculates but app scripts does not seem to wait for that recalculation so I end up retrieving values such as "Thinking..." or "#NA". Is there a way to pause a script and wait for the calculation to complete before moving to the next line in the script?
Currently, I am just using a loop to watch the cell but I wanted to find out if there was a more elegant way to pause the execution until the sheet was done calculating.
I write a lot of Excel Macros and Excel VBA always waits for the calculation to complete before moving to the next line in the code. Apps Script does not seem to do this so I am hoping there is an easy way to do this.
A second question: Because this iteration can take some time, how does one interrupt and terminate a script from running? I can't seem to find a way to do this.
Here is a very simple way of preventing the next script from starting until the current script completes in google apps scripts. Just add a call for testWait() after each script you are processing successively. The SpreadsheetApp.flush() also seems to reset the timeout timer on the spreadsheet back to the default 5min so you have more time to process multiple scripts in one go.
//holds processing of next script till last one has completed
function testWait(){
var lock = LockService.getScriptLock(); lock.waitLock(300000);
SpreadsheetApp.flush(); lock.releaseLock();
}
Scripts timeout after about 6 minutes to prevent infinite loops and constantly running programs. I don't think there's a manual way to stop a script.
Edit: Oops, forgot to answer your first question: I thought onEdit ran after values were recalculated, but apparently I don't use enough formulas to see this. If it's not waiting, then the best way is to do something like this:
while(value === "Thinking..." || value === "#NA") {
Utilities.sleep(10000);
}
It pauses the script for a few seconds and then checks again.
I also write a bit in Excel VBA, where the native Calculation functions come in handy. I've run across the same problem in Google Apps/Docs several times, wanting to execute code whenever calculations are complete. I wonder why there's no native function in Google Apps/Docs to handle this. Anyhow, I wrote this code to solve the problem. Hope it helps.
function onEdit() {
Refresh();
};
function Refresh () {
var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1");
var sheet2 = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet2");
// set the range wherever you want to make sure loading is done
var range = sheet.getRange('A:A')
var values = range.getValues();
var string = values.toString();
var loading = "Loading";
do{
var randomWait = Math.floor(Math.random()*1+0); randomWait;
} while (string.search(loading) ==! 0);
range.copyTo(sheet2.getRange('A1'), {contentsOnly:true});
customMsgBox();
};
function customMsgBox() {
Browser.msgBox("Data refreshed.");
};
Here's an example in action:
https://docs.google.com/spreadsheet/ccc?key=0AkK50_KKCI_pdHJvQXdnTmpiOWM4Rk5PV2k5OUNudVE#gid=0
Make a copy if you want to play around with it.
I had the same problem. I resolved it by using 2 scripts as I needed to be sure the spreadsheet has the data I need.
The first script provides a seed value to populate the spreadsheet through a IMPORTXML function.
The second script processes this data.
I used time based triggers to run the scripts allowing for sufficient time for the first script to complete
You could put the recursive structure in the code.js file, rather than on the spreadsheet. Javascript can handle recursion really well. You could then have the script update the spreadsheet on each iteration (or each 100).