I am trying to scrape data of a website once day automatically. In Google Spreadsheets, i use the =ImportHTML() function to import data tables, and then I extract the relevant data with a =query(). These functions take between 10 and 30 seconds to complete calculation, every time I open the spreadsheet.
I use a scheduled Google Apps Script, to copy the data into a different sheet (where it is stored, so i can run statistics) every day.
My problem is that I am having trouble to make the script wait for the calculations to be finished, before the data is copied. The Result is that my script just copies the error Message "N/A".
I tried just adding a Utilities.sleep(60000);, but it didn't work.
Is it possible to create a loop, that checks for the calculation to finish? I tried this without success:
function checkForError() {
var spreadsheet = SpreadsheetApp.getActive();
var source = spreadsheet.getRange ("Today!A1");
if (source = "N/A") {
Utilities.sleep(6000);
checkForError();
} else {
moveValuesOnly();
}
}
Locks are for this. Look up lock services in the docs. Use a public lock.
Here's how I used Zig's suggestion (combined with my own check loop) to solve my similar problem:
// Get lock for public shared resourse
var lock = LockService.getPublicLock();
// Wait for up to 120 seconds for other processes to finish.
lock.waitLock(120000);
// Load my values below
// something like sheet.getRange("A1").setFormula('= etc...
// Now force script to wait until cell D55 set to false (0) before
// performing copy / pastes
var current = SpreadsheetApp.setActiveSheet(sheet.getSheets()[1]);
var ready = 1;
var count = 0;
while (true) {
// break out of function if D55 value has changed to zero or counter
// has hit 250
if (count >= 250) break;
// otherwise keep counting...
ready = current.getRange("D55").getValue();
if (ready == 0) {count = 400;}
Utilities.sleep(100);
++count;
}
// wait for spreadsheet to finish... sigh...
Utilities.sleep(200);
// Do my copy and pastes stuff here
// for example sheet.getRange("a1:b1").copyTo(sheet.getRange("a3"), {contentsOnly:true});
// Key cells are updated so release the lock so that other processes can continue.
lock.releaseLock();
// end script
return;
}
This has worked fantastic for me, stopped Google's sporadic service from ruining my work!
Thanks goes to Zig's suggestion!
Related
In a self-developed add-on for Google Sheets, the functionality has been added that a sound file will be played from a JavaScript audio player in the sidebar, depending on the selection in the table. For the code itself see here.
When a line is selected in the table the corresponding sound file is played in the sidebar. Every time the next line is selected it takes around 2 seconds before the script will start to run and load the sound file into the sidebar. As the basic idea of the script is to quickly listen through long lists of sound files, it is crucial to reduce the waiting time as fare as possible.
A reproducible example is accessible here; Add-ons > 'play audio' (Google account necessary). To reproduce the error, the sheet has to be opened two times (e.g. in two browsers).
In order to reduce the latency you might try to reduce interval on your poll function as suggested by Cooper on a comment to the question and to change the getRecord function.
poll
At this time the interval is 2 seconds. Please bear in mind that reducing the interval too much might cause an error and also might have an important impact on the consume of the daily usage quotas. See https://developers.google.com/apps-script/guides/services/quotas
getRecord
Every time it runs it make multiple calls to Google Apps Script which are slow so you should look for a way to reduce the number of Google Apps Script calls. In order to do this you could store the spreadsheet table data in the client side code and only read it again if the data was changed.
NOTE: The Properties Service has a 50,000 daily usage quota for consumer accounts.
One way to quickly implement the above is to limit the getRecord function to read the current cell and add a button to reload the data from the table.
Function taken from the script bounded to the demo spreadsheet linked in the question.
function getRecord() {
var scriptProperties = PropertiesService.getScriptProperties();
var sheet = SpreadsheetApp.getActiveSheet();
var data = sheet.getDataRange().getValues();
var headers = data[0];
var rowNum = sheet.getActiveCell().getRow(); // Get currently selected row
var oldRowNum = scriptProperties.getProperty("selectedRow"); // Get previously selected row
if(rowNum == oldRowNum) { // Check if the was a row selection change
// Function returns the string "unchanged"
return "unchanged";
}
scriptProperties.setProperty("selectedRow", rowNum); // Update row index
if (rowNum > data.length) return [];
var record = [];
for (var col=0;col<headers.length;col++) {
var cellval = data[rowNum-1][col];
if (typeof cellval == "object") {
cellval = Utilities.formatDate(cellval, Session.getScriptTimeZone() , "M/d/yyyy");
}
record.push({ heading: headers[col],cellval:cellval });
}
return record;
}
Related
Problems when using a Google spreadsheet add-on by multiple users
i'm struggling with problem of running script by 2 or more users in the same time. Script adds data to spreadsheet and creates folders in google drive.
I've tried using Lockservice, but it didn't work. And I've made my own 'Lock' which change value in sheet and when this Busy status is true script won't run, but it changes too slow.
function thisRunsClient(){
var lock = LockService.getScriptLock();
if(lock.hasLock()){return 'END')};
else{
lock.tryLock(10000)};
//rest of code
}
What I use is
var lock = LockService.getScriptLock();
try {
lock.waitLock(30000); // wait 30 seconds for others' use of the code section and lock to stop and then proceed
} catch (e) {
Logger.log('Could not obtain copy lock after 20 seconds.');
return;
}
//REST OF CODE
I'm not sure what you are trying to do with the haslock there, it looks to me like if you get the lock you immediately end the function, which doesn't seem like what you want to do.
I have the following code set to run based on a onFormSubmit trigger but it will sometimes run multiple times with the same submission. I want to verify if it already copied the row and if so to stop the script.
function toDo(){
var responses = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Form Responses 1");
var jobs = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Jobs");
var lastrow = responses.getLastRow();
var col = responses.getLastColumn();
var row = responses.getRange(lastrow, 1, 1, 19).getValues();
jobs.appendRow(row[0]);
//copyValuesOnly(copyFromRange, copyToRangeStart);
var si = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Jobs');
var range = si.getRange("A2:R");
range.sort({column: 5,ascending: true}),({column: 1, ascending:true});
}
this is a known problem with GAS + Forms. The way that you solve it is by creating a script lock that rejects (causing them to return early) all other attempts within a period of time.
function toDo(){
SpreadsheetApp.flush();
var lock = LockService.getScriptLock();
try {
lock.waitLock(5000);
} catch (e) {
Logger.log('Could not obtain lock after 5seconds.');
return HtmlService.createHtmlOutput("<b> Server Busy please try after some time <p>")
// In case this a server side code called asynchronously you return a error code and display the appropriate message on the client side
return "Error: Server busy try again later... Sorry :("
}
var responses = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Form Responses 1");
var jobs = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Jobs");
var lastrow = responses.getLastRow();
var col = responses.getLastColumn();
var row = responses.getRange(lastrow, 1, 1, 19).getValues();
jobs.appendRow(row[0]);
//copyValuesOnly(copyFromRange, copyToRangeStart);
var si = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Jobs');
var range = si.getRange("A2:R");
range.sort({column: 5,ascending: true}),({column: 1, ascending:true});
Utilities.sleep(5000);
lock.releaseLock)();
}
I've had scripts that do this up to 8 times, and usually do it every 2-3 seconds. With this solution you are making a lock at the beginning and then sleeping at the end to make sure that the process time is greater than the wait time. (Here I used 5 seconds, that should prevent the double entry).
I have noticed that if you just make another copy of the sheet with the script, this error goes away. Seems to reset whatever was the issue in the original copy. Also try dis-abling the response receipts on your google forms.
How to check:
Go to script editor and check under execution logs. If you see multiple instances of on form submit, then you probably have multiple triggers somehow and the trigger is running multiple times legitimately.
To fix:
Go to triggers tab and delete all unwanted triggers.
Check code if you are creating new trigger through code. And comment that out.
Possible Reason:
I would have expected the code such as below to overwrite existing trigger. I had 19 triggers created. This was because every time I generated the link, it called Initialize and I got a new trigger added. Thus I could see script running 19 times.
const initialize = () => { const form = FormApp.getActiveForm(); //ScriptApp.newTrigger('onFormSubmit').forForm(form).onFormSubmit().create(); };
I noticed the problem got solved if I renamed the function called by onFormSubmit and saved the script.
If I click "run" in the script editor I get 2 emails per form submit. If I click "run" again I get 3 emails per form submit. I reset to only 1 email per form submit if I change the function name again.
Somehow when I run the script it duplicates the triggers.
I am using the app script provided by Google to access their prediction API through sheets. I am trying to predict thousands of rows at once, however, after 6 minutes the maximum execution time is reached at the code stops.
I implemented a solution that I found using clock trigger builder. Once I run the function it goes for 5 mins, then it stops sets a trigger to recall the function within 2 mins.
The major problem is that the function is not called when scheduled. I see it in the current triggers list, but it never gets called again. Can you please explain why this is occurring.
My intention is to predict as many lines as possible in 5 min then stop set a trigger to call the predict function again within a few minutes start where it left off and continue until ever element has been predicted.
I also need to know how would I store then values in cache so that it would know all the information that it needs when the function is called again.
//This is the function that is used to predict a selection of data
function predict() {
try {
clearOutput();
var startTime= (new Date()).getTime();
var sheet = SpreadsheetApp.getActiveSheet();
var selection = sheet.getActiveSelection();
var instances = selection.getValues();
var project_number = getProjectNumber();
var model_name = getModelName();
var startRow = stRow;
var MAX_RUNNING_TIME = 300000;
var REASONABLE_TIME_TO_WAIT = 60000;
for (var i = startRow; i < instances.length; ++i) {
var currTime = (new Date()).getTime();
if(currTime - startTime >= MAX_RUNNING_TIME) {
var builder = ScriptApp.newTrigger('predict').timeBased().after(REASONABLE_TIME_TO_WAIT);
builder.create();
break;
} else {
var result = predictSingleRow(project_number, model_name, instances[i]);
selection.getCell(i + 1, 1).setValue(result);
}
}
} catch(e) {
Browser.msgBox('ERROR:' + e, Browser.Buttons.OK);
}
}
Few things as to why your code is not functioning as intended:
1) Since you mentioned,"I see it in the current triggers list, but it never gets called again" and looking at your code, I am unsure whether you intended to call the function again after it's execution has completed. If you do, this is because your for loop runs for a while until the length of the instances is obtained. Nothing in the script suggests that the function needs to be run again once it has finished iterating through instances. Refer to this link to see how to Manage Trigger Programmatically.
2) var builder = ScriptApp.newTrigger('predict').timeBased().after(REASONABLE_TIME_TO_WAIT);
This line of your code falls under an if condition which stops the execution for 1 minute (value is 60000). Hence, adding 1 minute to the time since execution started. Nowhere are you resetting the startTime counter to the time after the waiting time since once the value of currTime - startTime has exceeded MAX_RUNNING_TIME, the function will keep calling the if loop for all iterations of the for loop after that. Simply put, if startTime was 9:35 and currTime was 9:40, after waiting for 1 minute the currTime is 9:41 which is still more than the MAX_RUNNING_TIME(5 minutes) because value of startTime still remains 9:35. Resetting it to 9:41 at this point should resolve your problem.
3) Loosing the break in the if loop would probably help fix that as well.
EDIT:
Add a function as shown in the link I mentioned above:
function callTrigger(){
ScriptApp.newTrigger('predict')
.timeBased()
.everyMinutes(30)
.create();
}
Run the function callTrigger once from your editor and you should be good to go. Remember, for minutes you can only pass values 1,5,15 or 30.
is it allowed to chain time triggers in Google App script like this :
function doGet(e){ //first invocation by user, HTTP GET
if (e == null || e.parameters == null) {
return ContentService.createTextOutput("EMPTY");
}
saveGetParametersForUser(Session.getUser().getEmail(), e);
//trigger 10 seconds
var timeTrigger = ScriptApp.newTrigger("timeDrivenEvent").timeBased().after(10 * 1000).create();
}
function timeDrivenEvent() { //runs until there are some data in ScriptDB
Logger.log("INVOKED AT " + new Date());
removeAllPreviousTriggers(); //removes old time triggers
var somedata = loadTaskData({email: "" + Session.getUser().getEmail()});
var remainingData = processTaskData(somedata);
if(remainingData == null){
return; //we are finished here
}
removePreviousAndSaveRemainingTaskData(remainingData);
var timeTrigger = ScriptApp.newTrigger("timeDrivenEvent").timeBased().after(10 * 1000).create();
}
First invocation by user doGet()
Until all data are processed script invokes itself with 10 sec intervals (e.g. 2minutes of processing, 10 seconds nothing happens, then again 2 minutes of processing...)
size of processed data is ~ few kilobytes and processing time takes usually 1-2mins.
What happens to me that sometimes script is interrupted and data are not fully processed ! I am not getting any email alerts and nothing is in log or execution transcript - everything looks fine.
I am starting to think that maybe 10 seconds is quite quick to start script method but it`s in the API after all...
Any ideas ?
This is ONLY POSSIBLE solution how to chunk big task into smaller pieces as Google App Script cannot run for more than ~ 5-6 minutes (see quotas).
Having one periodic time trigger worked well as it was recommended in comments.
I just wonder why time trigger chaining didnt worked well ! What principle i did broke that Google App Script didnt like that.
Documentation : https://developers.google.com/apps-script/class_clocktriggerbuilder
method everyMinutes Sets the trigger to be created to fire on an interval of the passed in number of minutes which must be one of 1, 5, 10, 15 or 30.