Google Script timeout after 6 min - google-apps-script

I want to list all the folderName and their folderID present in a team Drive(more than 3000 folders). I am using speedsheet and running following code in script-
function listFilesInFolder(folderName) {
var sheet = SpreadsheetApp.getActiveSheet();
sheet.appendRow(['Name','File-Id']);
//change the folder ID below to reflect your folder's ID (look in the URL when you're in your folder)
var folder = DriveApp.getFolderById('folder ID');
var contents = folder.getFolders();
var cnt = 0;
var folder;
while (contents.hasNext()) {
var folder = contents.next();
cnt++;
data = [
folder.getName(),
folder.getId(),
];
sheet.appendRow(data);
};
}
But this is getting Error Exceeded maximum execution time which is 6 min by default.
I tried adding triggers from script app, but after triggering it get start from beginning and script still ends after 6min.
How to add a triggers which starts from where it left?

Answer:
The slow part of this script is the repeated call to sheet.appendRow(). You can speed this up by pushing the results to an array and setting the values at the end of the loop, rather than appending a row on each iteration of the while loop.
More Information:
Using the built-in services such as SpreadsheetApp often can be slow when making many changes to a sheet in a short space of time. You can combat this by minimising the number of calls to the built-in Apps Script services as possible, relying on pure JavaScript to do your processing.
Code Change:
function listFilesInFolder(folderName) {
const sheet = SpreadsheetApp.getActiveSheet()
//change the folder ID below to reflect your folder's ID (look in the URL when you're in your folder)
let folder = DriveApp.getFolderById('')
let contents = folder.getFolders()
let cnt = 0
let data = [['Name','File-Id']]
while (contents.hasNext()) {
folder = contents.next()
cnt++
data.push([
folder.getName(),
folder.getId(),
])
}
sheet.insertRows(sheet.getMaxRows(), data.length)
sheet.getRange(2, 1, data.length, 2).setValues(data)
}
Code changes:
data is declared as an array initialised with the header row, as opposed to appending it directly to the sheet
On each iteration of the loop, the current folder's name and ID is appended to the data array as a new row of data.
After all folders have been looped through, the number of rows in the sheet is extended by the number of rows in data so to not hit an out of bounds error
All rows inside data are added to the sheet using setValues().
In my test environment, I had the following set up:
Drive folder containing 3424 folders
Using the appendRow() method inside the while loop, execution took 1105.256 seconds (or 18 minutes)
Using push() with the .setValues() method outside the loop, execution took 4.478 seconds.
References:
Class Range - setValues() | Apps Script | Google Developers

Related

Google Script Function Timing

I am struggling to find the cause of the "mis-timing" of the following two function calls. I am attempting to have google look for a csv file in a particular gmail folder and insert the data into a google sheet. That works fine. Then, once the data is in the sheet, I call "UpdateComplete" whose sole purpose is to sort the sheet by column K (where there is a vlookup function that looks at a sheet with completed rows that have been moved there) so that rows that have the vlookup function already are sort to the top, and it then copies the formula into the rows that are new and don't already have it. However, if the google sheet has, say, 2000 rows, and the csv file contains 2100, for some reason the new 100 rows are being added after the call to UpdateComplete. So the new 100 rows are added, but they do not get the vlookup like all of the other rows. This issue only happens when the google sheet does not have enough rows, initially, for the csv data.
If, however, I comment out the call to "UpdateComplete" from within "RetrieveAwardData", and manually run that first, and then manually run "UpdateComplete", it works perfectly. I have tried adding a Utilities.Sleep call before the call to "UpdateComplete" (but after the csv setvalues line), in case it was a timing thing, but when I do that, the system waits that amount of time before adding the 100 new rows, even though the line for sleep comes after the line to add the csv data. I also tried creating a new function that calls "RetrieveAwardData" first (with the UpdateComplete call commented out) and then calls UpdateComplete 2nd, but the same issue happens. Why does it work properly if I run them, separately, manually, but not one after the other programmatically?
function RetrieveAwardData(){
var threads = GmailApp.search('is:unread subject:VA Benefit Aid');
var message = GmailApp.getMessagesForThreads(threads); //retrieve all messages in the specified threads.
var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('AwardData');
if(message[0] != null){
Logger.log(message[0]);
for (var i = 0 ; i < message.length; i++) {
for (var j = 0; j < message[i].length; j++) {
var attachment = message[i][j].getAttachments()[0];
var csvData = Utilities.parseCsv(attachment.getDataAsString('ISO-8859-1'), ",");
sheet.getRange(1, 1, csvData.length, csvData[0].length).setValues(csvData);
UpdateComplete();
GmailApp.markMessageRead(message[i][j]);
}
}
}
else{Logger.log("No file available.");}
}
function UpdateComplete(){
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName('AwardData');
sheet.sort(11);
var LastAwardRow = sheet.getLastRow();
var Avals = ss.getRange("K1:K").getValues();
var LastCompleteRow = Avals.filter(String).length;
if(LastAwardRow != LastCompleteRow){
sheet.getRange("K"+(LastCompleteRow+1) + ":K" + LastAwardRow).setFormulaR1C1(sheet.getRange("K"+LastCompleteRow).getFormulaR1C1());
}
}
Posting this for documentation purposes.
As mentioned by Cooper, use SpreadsheetApp.flush().
This method ensures that later parts of the script work with updated data, since changes made by previous parts are applied to the spreadsheet.

Script time limit for Google sheet

I need to change some formulas at the same cells at an specific sheet (LISTAFINAL) present in a great number of spreadsheets, these one located at the same folder. But it stops at Google script time limit of 6 minutes, making changes only in 9 spreadsheets, and comes the message: Erro - Exceeded maximum execution time.
My goals:
I would like to know if there's any way to or speed up this process or make changes in a bigger number of spreadsheets or both. Here is the code:
function validacao(){
var folder = DriveApp.getFolderById("FOLDER ID");
var x = folder.getFilesByType(MimeType.GOOGLE_SHEETS);
while (x.hasNext()) {
SpreadsheetApp.open(x.next()).getSheets().forEach(sheet => {
sheet.getRange('LISTAFINAL!F5:F15').activate();
sheet.getRange('LISTAFINAL!F5').setValue('=ALUNO1!$F$167');
sheet.getRange('LISTAFINAL!F6').setValue('=ALUNO2!$F$167');
sheet.getRange('LISTAFINAL!F7').setValue('=ALUNO3!$F$167');
sheet.getRange('LISTAFINAL!F8').setValue('=ALUNO4!$F$167');
sheet.getRange('LISTAFINAL!F9').setValue('=ALUNO5!$F$167');
sheet.getRange('LISTAFINAL!F10').setValue('=ALUNO6!$F$167');
sheet.getRange('LISTAFINAL!F11').setValue('=ALUNO7!$F$167');
sheet.getRange('LISTAFINAL!F12').setValue('=ALUNO8!$F$167');
sheet.getRange('LISTAFINAL!F13').setValue('=ALUNO9!$F$167');
sheet.getRange('LISTAFINAL!F14').setValue('=ALUNO10!$F$167');
sheet.getRange('LISTAFINAL!F15').setValue('=ALUNO11!$F$167');
});
}
}
Explanation:
You iterate over all sheets for every spreadsheet file. Your goal is to just get a single sheet and put the formulas in specific cells. Therefore, you can get rid of the forEach loop.
It is a best practice to work with arrays instead of iteratively using multiple google apps script functions.
For example, in your code you are using getRange and setValue 11 times. If you store the formula values in an array you will be able to store them by using only a single getRange and setValues.
Solution:
function validacao(){
const folder = DriveApp.getFolderById("FOLDER ID");
const x = folder.getFilesByType(MimeType.GOOGLE_SHEETS);
const formulas = Array(11).fill().map((_, i) => [`=ALUNO${i+1}!$F$167`]);
while (x.hasNext()) {
let ss_target = SpreadsheetApp.open(x.next());
let sh = ss_target.getSheetByName("LISTAFINAL");
sh.getRange('F5:F15').setValues(formulas);
}
}

Google Apps Script trigger - run whenever a new file is added to a folder

I want to execute a google apps script whenever a new file is added to a specific folder.
Currently I'm using a run-every-x-minutes clock trigger, but I only need to run the script whenever I add a file to a folder. Is there a way to do this?
The same as this question - which is now almost 3 years old. The comment below the question states that:
There's not a trigger for that, if that's what you're hoping. How are
things getting into the folder, and do you have any control over that?
– Jesse Scherer Apr 8 '18 at 3:02
I wonder if this comment is still valid, and if it is, then if there's a workaround.
Issue:
Unfortunately, the comment you read is still true. Here is a list of all the available triggers and a trigger for a new file added to a folder is not one of them.
Workaround / Explanation:
I can offer you a workaround which is usually used by developers when they built their add-ons. You can take advantage of the PropertiesService class. The logic is quite simple.
You will store key-value pairs scoped to the script:
In your case, the key will be the folder id, and the value will be the number of files under this folder.
You will setup a time-driven trigger to execute mainFunction for example every one minute.
The script will count the current number of files within the selected folder. The function responsible for that is countFiles.
The checkProperty function is responsible for checking if the current number of files under this folder matches the old number of files. If there is a match, meaning no files were added, then checkProperty returns false, otherwise return true and update the property for the current folder ID, so when the script runs after 1 minute, it will compare with the fresh value.
If checkProperty returns true, then execute the desired code.
Code snippet:
Set up a time-driven trigger for mainFunction. Whatever code you put inside the brackets of the if(runCode) statement will be executed if the number of files under the folderID has changed.
function mainFunction(){
const folderID = 'folderID'; //provide here the ID of the folder
const newCounter = countFiles(folderID);
const runCode = checkProperty(folderID, newCounter);
if(runCode){
// here execute your main code
//
console.log("I am executed!");
//
}
}
And here are the helper functions which need to be in the same project (you can put them in the same script or different scripts but in the same "script editor").
function countFiles(folderID) {
const theFolder = DriveApp.getFolderById(folderID);
const files = theFolder.getFiles();
let count = 0;
while (files.hasNext()) {
let file = files.next();
count++;
};
return count;
}
function checkProperty(folderID, newC){
const scriptProperties = PropertiesService.getScriptProperties();
const oldCounter = scriptProperties.getProperty(folderID);
const newCounter = newC.toString();
if(oldCounter){
if(oldCounter==newCounter){
return false;
}
else{
scriptProperties.setProperty(folderID, newCounter);
return true;
}
}
else{
scriptProperties.setProperty(folderID, newCounter);
return true;
}
}

Script to open a long list of URLs one at a time every 5 seconds and then open the next URL

I have a google spreadsheet with a large number of API Urls.
They look like this => http://oasis.caiso.com/oasisapi/SingleZip?resultformat=6&queryname=PRC_LMP&version=1&startdatetime=20160101T08:00-0000&enddatetime=20160103T08:00-0000&market_run_id=DAM&grp_type=ALL
The database I am drawing from limits requests to one every 5 seconds.
When you follow the link it will download a zip file with cvs files.
I would like to write a script that will follow a URL, wait 6 seconds and then move on to the next URL on the list.
I would like it to stop when it gets to the last URL
I am imagining that I would need to use a "when" loop, but I cannot figure out how to install a wait period, or how to get it to open the URL.
HELP!!!!
I tried a batch URL follow, which failed because of the timing issue.
I began to write the When loop, but I am totally stuck.
I would like to run through the huge list of links fully once. To date I cannot make anything work.
function flink(){
var app = spreasheetapp
//access the current open sheet
var activesheet = app.getactivespreadsheet().getactivesheet()
var activecell= activesheet.getrange(11,11).openurl
//I am getting totally stuck here
I have tried using an iterator but I have no idea how to add the time delay and then I cannot seem to get the syntax for the iterator correct.
To access a url from AppsScript, you can use UrlFetchApp.fetch(url).
To force the script to wait for a certain amount of time, you can use Utilities.sleep(milliseconds).
References:
https://developers.google.com/apps-script/reference/url-fetch/url-fetch-app#fetch(String)
https://developers.google.com/apps-script/reference/utilities/utilities#sleepmilliseconds
I came up with this code that accesses each URL in the sheet. I added some comments so you can see what the script is doing step by step:
function accessURLs() {
// Copy urls
var ss = SpreadsheetApp.getActive();
var sheet = ss.getSheetByName("Links up to today");
// Copy the contents of the url to another column removing the formulas (your script will appreciate this)
sheet.getRange("C3:C").copyTo(sheet.getRange("D3:D"), {contentsOnly:true});
var column = 4; // URLs are in column 4
var row = 3; // URLs start at row 3
var url = sheet.getRange(row, column).getValue();
var responses = []
// Loop through all urls in the sheet until it finds a cell that is blank (no more urls left)
do {
var options = {
muteHttpExceptions: true
}
var response = UrlFetchApp.fetch(url, options); // Script makes a request to the url
responses.push(response); // The latest response is added to the responses array
Utilities.sleep(6000); // The script stops for 6 seconds
row++;
url = sheet.getRange(row, column).getValue();
} while (url != ""); // Cell is not blank
}
Take into account that if we are to access ~1400 URLs and the script stops for 6 seconds after each fetch, it will take more than 2 hours to access all URLs.
Also, take into account that this script is just getting the data coming from the URL requests (it's stored in the variable responses), but it is not doing anything else. Depending on what you want to do with this data, you might want to add some extra stuff there.
I hope this is of any help.

Google Apps Script withSuccessHandler is running before function fully executes

I have two files in an Apps Script project. One is a .gs file acting as a "server" and one is a .html file containing JavaScript as per Google's Apps Script guidelines.
Everything has been going swimmingly for the first 40 hours of development on this project. I have the following line of code outside of any function, in between two tags in the .html file:
google.script.run.withSuccessHandler(setSheetData).getSheetData();
Documentation: https://developers.google.com/apps-script/guides/html/reference/run#withSuccessHandler(Function)
According to the documentation, getSheetData() should first execute in the .gs file, and return a value that is then passed into setSheetData which exists in the .html file.
Server file:
function getSheetData() {
var ss = SpreadsheetApp.getActive();
var activeSheet = ss.getActiveSheet();
var sheetName = activeSheet.getName();
var sheetVals = activeSheet.getDataRange().getValues();
return [sheetName, sheetVals];
}
Html file:
function setSheetData(data) {
alert(data);
sheetName = data[0];
sheetData = data[1];
headers = sheetData[0];
document.getElementById('sheetLook').innerHTML = 'Looking at sheet: ' + sheetName;
}
How I know it is a matter of execution speed:
Currently the alert() call just prints out null. The sheet it is drawing from contains 4 rows of data. However, all other things remaining the same, if I simply am looking at a sheet with 0-1 rows of data, it correctly alerts the entire data vals.
Inside of getSheetData() if I add Logger.log(sheetVals) it correctly logs the entire sheet's data regardless of size. The issue is that the successhandler is executing before it has time to evaluate.
Possible cause(s):
Illegal return values in rows 3-4 of data
No active sheet is present.
Solution:
Avoid returning illegal values like Date objects. Or JSON.stringify() them before returning to client.
getSheetByName or number instead of getting it by activeness.
References:
google.script.run § myFunction § return