Gmail Snooze error: moveSnoozes "service invoked too many times in a short time" - google-apps-script

I trying to find a way to snooze e-mail in my Gmail. Yes, I had Mailbox.app on my iDevice, but I want a way to Snooze or "Laterize" some emails.
I found a Blog Spot post and a Lifehacker one. The Lifehacker article just adds more pictures to the Blog Sport one. I followed the Lifehacker instructions. I made only one deviation from the code. In the two instances that the code says "7", I replaced it with "200".
The code I have is:
var MARK_UNREAD = false;
var ADD_UNSNOOZED_LABEL = false;
function getLabelName(i) {
return "Snooze/Snooze " + i + " days";
}
function setup() {
// Create the labels we’ll need for snoozing
GmailApp.createLabel("Snooze");
for (var i = 1; i <= 200; ++i) {
GmailApp.createLabel(getLabelName(i));
}
if (ADD_UNSNOOZED_LABEL) {
GmailApp.createLabel("Unsnoozed");
}
}
function moveSnoozes() {
var oldLabel, newLabel, page;
for (var i = 1; i <= 200; ++i) {
newLabel = oldLabel;
oldLabel = GmailApp.getUserLabelByName(getLabelName(i));
page = null;
// Get threads in "pages" of 100 at a time
while(!page || page.length == 100) {
page = oldLabel.getThreads(0, 100);
if (page.length > 0) {
if (newLabel) {
// Move the threads into "today’s" label
newLabel.addToThreads(page);
} else {
// Unless it’s time to unsnooze it
GmailApp.moveThreadsToInbox(page);
if (MARK_UNREAD) {
GmailApp.markThreadsUnread(page);
}
if (ADD_UNSNOOZED_LABEL) {
GmailApp.getUserLabelByName("Unsnoozed")
.addToThreads(page);
}
}
// Move the threads out of "yesterday’s" label
oldLabel.removeFromThreads(page);
}
}
}
}
But I'm getting an error:
Your script, Gmail Snooze, has recently failed to finish
successfully. A summary of the failure(s) is shown below.
Details:
Start: 9/10/13 12:16 AM
Function: moveSnoozes
Error Message: Service invoked too many times in a short time: gmail rateMax. Try Utilities.sleep(1000) between calls. (line 24, file "Code")
Trigger: time-based
End: 9/10/13 12:22 AM
Does anybody know how I can make use of Utilities.sleep(1000)? Where do I type that in?

The error you got is because creating 200 labels at the speed-of-code is too fast for Gmail and considered abuse of it's API. SO, as the error message suggests, you could use Utilities.sleep(1000) to wait a second between each label creation.
I'm not familiar with the script (Boomerang works well for me), but you might try making your 30, 60, 90 180 day labels by hand and letting the script pick up on them. Then you don't need to worry about the label creation at all.
In an effort to help you better understand the code you are reading. Below is an explanation of what's going on and how to insert the Utilities.sleep(). You will need to decide for how long you want to wait 1000ms means that you're waiting for 200 seconds, and if you have 3 min to spare that will do just fine. You could experiment to see at what point the Gmail app says 'Woah that's enough there. Take it slower please.'
NOTE: I am not attempting to re-engineer this to allow for a better creation/management of labels. There are a probably a number of enhancements that can be made to the code, and although tempting, they will be left as an exercise for the reader.
RE your pastie:
function setup() {
// This creates 1 label called "Snooze"
// it probably isn't causing the problem
GmailApp.createLabel("Snooze");
// This piece is a loop that will execute 200 times.
// It will do so as fast as GAS will let it.
// This is probably where the problem is...
for (var i = 1; i <= 200; ++i) {
// Create the label
GmailApp.createLabel(getLabelName(i));
// We should wait a bit before making the next one
// Utilities.sleep() will do well to go here
Utilities.sleep(/*Milliseconds to wait*/);
}
// I doubt this is causing the problem either.
// It's only one label creation
if (ADD_UNSNOOZED_LABEL) {
GmailApp.createLabel("Unsnoozed");
}
}

Related

Save Number with Shared Object and Add to that Saved Number.

Hey everyone so basically what I am trying to accomplish is saving a number with the Shared Object which the the coins that the player collects in the game and if they player quits out of the game and comes back to play again the amount of coins he had when he quit will still show and he will be able to add to that amount if the player picks up more coins. I had this working for a bit but then the textfield started displaying "NAN".
Here is how I have it set up.
The Variable private var nCoins:Number;
In the constructor function I have:
sharedObjectCoins = SharedObject.getLocal("CoinsData");
nCoins = 0 + sharedObjectCoins.data.tCoins;
if (sharedObjectCoins.data.tCoins == null)
{
sharedObjectCoins.data.tCoins = nCoins;
}else
{
trace("Save data found."); // if we did find data...
loadDataTimeAttack(); // ...load the data
}
and in the games Enter.Frame Loop I have the function saveDataCoins which is setup like so:
private function saveDataCoins():void
{
if (nCoins > sharedObjectCoins.data.tCoins )
{
sharedObjectCoins.data.tCoins = nCoins;
}
coinsGraphic.coinsText.text = " " + sharedObjectCoins.data.tCoins;
sharedObjectCoins.flush();
}
not sure if you need the function to where the hitTest takes place between the coins and player but here it is:
private function checkPlayerHitCoins():void
{
for (var i:int = 0; i < aCoinsArray.length; i++)
{
//get current point in i loop
var currentCoins:mcCoin = aCoinsArray[i];
//test if player is hitting current point
if(player.hitTestObject(currentCoins))
{
nCoins += 1;
updatecoinsTextScore();
updateCoinsPauseScreen();
//Add points sound effects
var coinsSEffect:Sound = new coinsSound();
coinsSEffect.play();
//remove point on stage
currentCoins.destroyCoins();
//remove points from array
aCoinsArray.splice(i, 1);
trace("Hit: " + aCoinsArray.length);
}
}
}
Please if anyone could help me with this maybe point something out that I am doing wrong. This code worked perfect one time and when I closed the screen and came back to re test it the textfield displayed NAN and thats it when I hitTest the coins sometimes the NAN switches to a number for like a second but then goes back to NAN.
The first time (or rather every time it creates a new shared object) you will be trying to add undefined to 0, which will result in either a runtime error or NaN.
You need to check if the value exists before attempting to do addition with it.
if(sharedObjectCoints.data && sharedObjectCoins.data.tCoins && !isNaN(sharedObjectCoins.data.tCoins)){
nCoins = Number(sharedObjectCoins.data.tCoins); //there's not point in adding 0
trace("Save data found."); // if we did find data...
loadDataTimeAttack(); // ...load the data
}else{
sharedObjectCoins.data.tCoins = nCoins;
}
Also, if you don't manually set a value to a number var, it will start off life as NaN. eg var nCoins:Number will be NaN until you set it to something.
That said, working with the sharedObject directly like this is a very sloppy way to code your program. Really you should just use shared object to load and save the value, and everything in between use a strongly typed variable.
var nCoins:int = 0;
var tCoins:int = 0;
sharedObjectCoins = SharedObject.getLocal("CoinsData");
if(sharedObjectCoins.data && sharedObjectCoins.data.tCoins && !isNaN(sharedObjectCoins.data.tCoins){
tCoins = int(sharedObjectCoins.data.tCoins);
}else{
//no shared object, use default value for tCoins
tCoins = 0; //or whatever it should start off as.
}
Then write a save function
private function saveSharedObject():void {
sharedObjectCoins.data.tCoins = tCoins;
sharedObjectCoins.flush();
}
Then replace all other instances of sharedObjectCoins.data.tCoins with the var tCoins
It's probably best not to flush the shared object every frame for performance purposes.
Also, shared objects may or may not actually save, depending on user preferences, storage space available, etc. They should not be relied upon for critical data retention.
You can listen for problems with the shared object with AsyncErrorEvent.ASYNC_ERROR I believe (It's been a while since I've worked with AS3 Shared Objects)

How to measure (or calculate) the interval between two js-function?

My friend asked me a special javascript question yesterday, and i found i could not solve it because nobody will write code like this:
function fun(){
fun1();
}
function fun1(){
setTimeout(fun(), 10000);
}
$(document).ready(function(){
fun();
});
And his question was: Could you tell me that how many seconds between first time execute fun1 and second time execute fun1? 10 seconds?
I don't think his code will execute in correct, and i test his code via chrome dev-tools for inspecting that i am right.Chrome dev-tools print a message:
Uncaught RangeError: Maximum call stack size exceeded
I also wanna know how many seconds between first time execute fun1 and second time execute fun1 even though i'll never write code like that. Is there anybody know the answer about this question?
You can achieve it this way:
var tm = null;
var times = 0;
function fun()
{
var d = new Date();
times++;
if (times == 2)
alert((d.getTime() - tm) + " milliseconds");
else
{
tm = d.getTime();
setTimeout(function () { fun(); }, 10000);
}
}
fun();
Basically you're storing last time value and calculating the time difference the second time the function is called.
JSFiddle example

How to create an auto click button

I have a list of cases in the queue that I need to grab. As you can imagine, it's a bit repetitive and time consuming. I'm new at programming and haven't figured a way to create a script that auto-click/grab these cases. Can someone help?
Code to:
1) Search and Click "Grab"
- will take 4 seconds for the page to refresh
2) Click grab again
3) stop after 50 cases are grabbed
This code doesn't work
window.setTimeout("pushSubmit()",3000);
function pushSubmit()
{document.getElementById('Grab').click();
Assuming your page is not refreshed in the process, you could keep a counter of how many "Grabs" you have done:
var counter = 0;
var maxCount = 50;
function pushSubmit() {
if(counter++ < maxCount) {
document.getElementById('Grab').click();
window.setTimeout(pushSubmit,3000);
}
}
//start the process
pushSubmit();
Here is a jsfiddle example
EDIT:
Or what I would probably prefer, set up the function so it can be used with any number of iterations.
function pushSubmit(max, count) {
count = typeof count !== 'undefined' ? count : 1;
if(count <= max) {
document.getElementById('Grab').click();
window.setTimeout(function() { pushSubmit(max, ++count) },3000);
}
}
//start the process with the max number of iterations it should perform
pushSubmit(50);
Example

Game Simulator Times Out

This is my first post on here. Thank you in advance for taking the time to read my question.
I am a novice coder. I have a minor in Computer Science that I got a decade ago. I had an urge to do some simple coding, and an opportunity came up, so I did!
In developing a game, I wanted to run a program to determine the chances of given outcomes with given parameters. I excitedly reached the point where it was a go, but Google Scripts couldn't handle running the 60,000,000 possible scenarios in order to compute a win%.
I got, "error: Exceeded maximum execution time."
I'm just trying to find the shortest path between me and running this program. Ideas:
1) Is there a way to remove the maximum execution time and let it just take all day? Is there some other way I can get it to run in Google Scripts?
2) Perhaps I can run a smaller number of trials by inputing random numbers. Is there a way to generate random numbers in Google Scripts?
3) Should I be doing this kind of thing in something besides Google Scripts? If so, is there a free/affordable compiler for Mac I should look into? I tried importing it into Xcode, but I'm bewildered and can't seem to get to a simple place to compile. Also, importing it to "C" is creating some compatibility issues; though I may just have to suck it up and retool it here.
For reference, here's the function that's timing it out:
function dieFeeder(winCount, fSkill, fMagnitude, fHeart, fDie1, fDie2, fDie3, fDie4, fDie5, cSkill, cMagnitude, cHeart, cDie1, cDie2, cDie3, cDie4, cDie5){
// a parent function to function questionMatrix, feeds the changing dice into it
var matrixWinner;
//This 'for' clause keeps going until all dice permutations have been tried out
for (var i=0; i<60466176; i++){
//This part changes the dice to go through all combiations in a way similar to counting in base 6
if (cDie5 == 7){
cDie5 = 1;
cDie4 = cDie4+1;
}
if (cDie4 == 7){
cDie4 = 1;
cDie3 = cDie3 +1;
}
if (cDie3 == 7){
cDie3 = 1;
cDie2 = cDie2 +1;
}
if (cDie2 == 7){
cDie2 = 1;
cDie1 = cDie1 +1;
}
if (cDie1 == 7){
cDie1 = 1;
fDie5 = fDie5 +1;
}
if (fDie5 == 7){
fDie5 = 1;
fDie4 = fDie4 +1;
}
if (fDie4 == 7){
fDie4 = 1;
fDie3 = fDie3 +1;
}
if (fDie3 == 7){
fDie3 = 1;
fDie2 = fDie2 +1;
}
if (fDie2 == 7){
fDie2 = 1;
fDie1 = fDie1 +1;
}
cDie5 = cDie5 + 1;
//This part checks to see who wins and increases the winCount if it was the Favorite
matrixWinner = questionMatrix(fSkill, fMagnitude, fHeart, fDie1, fDie2, fDie3, fDie4, fDie5, cSkill, cMagnitude, cHeart, cDie1, cDie2, cDie3, cDie4, cDie5);
if (matrixWinner == 'favorite'){
winCount = winCount +1;
}
}
return winCount;
}
There is no way to lift the maximum execution time. The limit is there so that other users (like me) can have time to run our scripts too! The solution to this problem is to break up your problem into many subproblems.
One solution would be to continue executing your script while your running time is under some threshold (say, 3 minutes) (i.e. keep track of how long your script has been running). Then save all state relating to your script (variables, etc.). Save these to ScriptDb. Then have your script run on a 5-minute trigger. When your script runs again, it reads the values from ScriptDb and picks up where it left off.
If you're looking for random numbers, use Math.random(). Google Apps Scripts is built off of javascript, so basic javascript functions are available.
Relating to my answer to #2, what you have shown is entirely javascript, so you can just copy your code over to some webpage to run it. (For testing, you can use jsfiddle).
Also you need to define if (cDie5 == 7){

Exceeded maximum execution time in Google Apps Script [duplicate]

My Google Apps Script is iterating through the user's Google Drive files and copying and sometimes moving files to other folders. The script is always stopped after certain minutes with no error message in the log.
EDITOR's NOTE: The time limit have varied over the time and might vary between "consumer" (free) and "Workspace" (paid) accounts but as of December 2022 most of the answers are still valid.
I am sorting tens or sometimes thousands files in one run.
Are there any settings or workarounds?
One thing you could do (this of course depends on what you are trying to accomplish) is:
Store the necessary information (i.e. like a loop counter) in a spreadsheet or another permanent store(i.e. ScriptProperties).
Have your script terminate every five minutes or so.
Set up a time driven trigger to run the script every five minutes(or create a trigger programmatically using the Script service).
On each run read the saved data from the permanent store you've used and continue to run the script from where it left off.
This is not a one-size-fit-all solution, if you post your code people would be able to better assist you.
Here is a simplified code excerpt from a script that I use every day:
function runMe() {
var startTime= (new Date()).getTime();
//do some work here
var scriptProperties = PropertiesService.getScriptProperties();
var startRow= scriptProperties.getProperty('start_row');
for(var ii = startRow; ii <= size; ii++) {
var currTime = (new Date()).getTime();
if(currTime - startTime >= MAX_RUNNING_TIME) {
scriptProperties.setProperty("start_row", ii);
ScriptApp.newTrigger("runMe")
.timeBased()
.at(new Date(currTime+REASONABLE_TIME_TO_WAIT))
.create();
break;
} else {
doSomeWork();
}
}
//do some more work here
}
NOTE#1: The variable REASONABLE_TIME_TO_WAIT should be large enough for the new trigger to fire. (I set it to 5 minutes but I think it could be less than that).
NOTE#2: doSomeWork() must be a function that executes relatively quick( I would say less than 1 minute ).
NOTE#3 : Google has deprecated Script Properties, and introduced Properties Service in its stead. The function has been modified accordingly.
NOTE#4: 2nd time when the function is called, it takes the ith value of for loop as a string. so you have to convert it into an integer
Quotas
The maximum execution time for a single script is 6 mins / execution
- https://developers.google.com/apps-script/guides/services/quotas
But there are other limitations to familiarize yourself with. For example, you're only allowed a total trigger runtime of 1 hour / day, so you can't just break up a long function into 12 different 5 minute blocks.
Optimization
That said, there are very few reasons why you'd really need to take six minutes to execute. JavaScript should have no problem sorting thousands of rows of data in a couple seconds. What's likely hurting your performance are service calls to Google Apps itself.
You can write scripts to take maximum advantage of the built-in caching, by minimizing the number of reads and writes. Alternating read and write commands is slow. To speed up a script, read all data into an array with one command, perform any operations on the data in the array, and write the data out with one command.
- https://developers.google.com/apps-script/best_practices
Batching
The best thing you can possibly do is reduce the number of service calls. Google enables this by allowing batch versions of most of their API calls.
As a trivial example, Instead of this:
for (var i = 1; i <= 100; i++) {
SpreadsheetApp.getActiveSheet().deleteRow(i);
}
Do this:
SpreadsheetApp.getActiveSheet().deleteRows(i, 100);
In the first loop, not only did you need 100 calls to deleteRow on the sheet, but you also needed to get the active sheet 100 times as well. The second variation should perform several orders of magnitude better than the first.
Interweaving Reads and Writes
Additionally, you should also be very careful to not go back and forth frequently between reading and writing. Not only will you lose potential gains in batch operations, but Google won't be able to use its built-in caching.
Every time you do a read, we must first empty (commit) the write cache to ensure that you're reading the latest data (you can force a write of the cache by calling SpreadsheetApp.flush()). Likewise, every time you do a write, we have to throw away the read cache because it's no longer valid. Therefore if you can avoid interleaving reads and writes, you'll get full benefit of the cache.
- http://googleappsscript.blogspot.com/2010/06/optimizing-spreadsheet-operations.html
For example, instead of this:
sheet.getRange("A1").setValue(1);
sheet.getRange("B1").setValue(2);
sheet.getRange("C1").setValue(3);
sheet.getRange("D1").setValue(4);
Do this:
sheet.getRange("A1:D1").setValues([[1,2,3,4]]);
Chaining Function Calls
As a last resort, if your function really can't finish in under six minutes, you can chain together calls or break up your function to work on a smaller segment of data.
You can store data in the Cache Service (temporary) or Properties Service (permanent) buckets for retrieval across executions (since Google Apps Scripts has a stateless execution).
If you want to kick off another event, you can create your own trigger with the Trigger Builder Class or setup a recurring trigger on a tight time table.
Also, try to minimize the amount of calls to google services. For example, if you want to change a range of cells in the spreadsheets, don't read each one, mutate it and store it back.
Instead read the whole range (using Range.getValues()) into memory, mutate it and store all of it at once (using Range.setValues()).
This should save you a lot of execution time.
Anton Soradoi's answer seems OK but consider using Cache Service instead of storing data into a temporary sheet.
function getRssFeed() {
var cache = CacheService.getPublicCache();
var cached = cache.get("rss-feed-contents");
if (cached != null) {
return cached;
}
var result = UrlFetchApp.fetch("http://example.com/my-slow-rss-feed.xml"); // takes 20 seconds
var contents = result.getContentText();
cache.put("rss-feed-contents", contents, 1500); // cache for 25 minutes
return contents;
}
Also note that as of April 2014 the limitation of script runtime is 6 minutes.
G Suite Business / Enterprise / Education and Early Access users:
As of August 2018, max script runtime is now set to 30 minutes for these users.
Figure out a way to split up your work so it takes less than 6 minutes, as that's the limit for any script. On the first pass, you can iterate and store the list of files and folders in a spreadsheet and add a time-driven trigger for part 2.
In part 2, delete each entry in the list as you process it. When there are no items in the list, delete the trigger.
This is how I'm processing a sheet of about 1500 rows that gets spread to about a dozen different spreadsheets. Because of the number of calls to spreadsheets, it times out, but continues when the trigger runs again.
I have used the ScriptDB to save my place while processing a large amount of information in a loop. The script can/does exceed the 5 minute limit. By updating the ScriptDb during each run, the script can read the state from the db and pick up where it left off until all processing is complete. Give this strategy a try and I think you'll be pleased with the results.
If you are using G Suite Business or Enterprise edition.
You can register early access for App Maker after App maker enabled your script run runtime will increase run time from 6 minutes to 30 minutes :)
More details about app maker Click here
Here's an approach based very heavily on Dmitry Kostyuk's absolutely excellent article on the subject.
It differs in that it doesn't attempt to time execution and exit gracefully. Rather, it deliberately spawns a new thread every minute, and lets them run until they are timed out by Google. This gets round the maximum execution time limit, and speeds things up by running processing in several threads in parallel. (This speeds things up even if you are not hitting execution time limits.)
It tracks the task status in script properties, plus a semaphore to ensure no two threads are editing the task status at any one time. (It uses several properties as they are limited to 9k each.)
I have tried to mimick the Google Apps Script iterator.next() API, but cannot use iterator.hasNext() as that would not be thread-safe (see TOCTOU). It uses a couple of facade classes at the bottom.
I would be immensely grateful for any suggestions. This is working well for me, halving the processing time by spawning three parallel threads to run through a directory of documents. You could spawn 20 within quota, but this was ample for my use case.
The class is designed to be drop-in, usable for any purpose without modification. The only thing the user must do is when processing a file, delete any outputs from prior, timed out attempts. The iterator will return a given fileId more than once if a processing task is timed out by Google before it completes.
To silence the logging, it all goes through the log() function at the bottom.
This is how you use it:
const main = () => {
const srcFolder = DriveApp.getFoldersByName('source folder',).next()
const processingMessage = processDocuments(srcFolder, 'spawnConverter')
log('main() finished with message', processingMessage)
}
const spawnConverter = e => {
const processingMessage = processDocuments()
log('spawnConverter() finished with message', processingMessage)
}
const processDocuments = (folder = null, spawnFunction = null) => {
// folder and spawnFunction are only passed the first time we trigger this function,
// threads spawned by triggers pass nothing.
// 10,000 is the maximum number of milliseconds a file can take to process.
const pfi = new ParallelFileIterator(10000, MimeType.GOOGLE_DOCS, folder, spawnFunction)
let fileId = pfi.nextId()
const doneDocs = []
while (fileId) {
const fileRelativePath = pfi.getFileRelativePath(fileId)
const doc = DocumentApp.openById(fileId)
const mc = MarkupConverter(doc)
// This is my time-consuming task:
const mdContent = mc.asMarkdown(doc)
pfi.completed(fileId)
doneDocs.push([...fileRelativePath, doc.getName() + '.md'].join('/'))
fileId = pfi.nextId()
}
return ('This thread did:\r' + doneDocs.join('\r'))
}
Here's the code:
const ParallelFileIterator = (function() {
/**
* Scans a folder, depth first, and returns a file at a time of the given mimeType.
* Uses ScriptProperties so that this class can be used to process files by many threads in parallel.
* It is the responsibility of the caller to tidy up artifacts left behind by processing threads that were timed out before completion.
* This class will repeatedly dispatch a file until .completed(fileId) is called.
* It will wait maxDurationOneFileMs before re-dispatching a file.
* Note that Google Apps kills scripts after 6 mins, or 30 mins if you're using a Workspace account, or 45 seconds for a simple trigger, and permits max 30
* scripts in parallel, 20 triggers per script, and 90 mins or 6hrs of total trigger runtime depending if you're using a Workspace account.
* Ref: https://developers.google.com/apps-script/guides/services/quotas
maxDurationOneFileMs, mimeType, parentFolder=null, spawnFunction=null
* #param {Number} maxDurationOneFileMs A generous estimate of the longest a file can take to process.
* #param {string} mimeType The mimeType of the files required.
* #param {Folder} parentFolder The top folder containing all the files to process. Only passed in by the first thread. Later spawned threads pass null (the files have already been listed and stored in properties).
* #param {string} spawnFunction The name of the function that will spawn new processing threads. Only passed in by the first thread. Later spawned threads pass null (a trigger can't create a trigger).
*/
class ParallelFileIterator {
constructor(
maxDurationOneFileMs,
mimeType,
parentFolder = null,
spawnFunction = null,
) {
log(
'Enter ParallelFileIterator constructor',
maxDurationOneFileMs,
mimeType,
spawnFunction,
parentFolder ? parentFolder.getName() : null,
)
// singleton
if (ParallelFileIterator.instance) return ParallelFileIterator.instance
if (parentFolder) {
_cleanUp()
const t0 = Now.asTimestamp()
_getPropsLock(maxDurationOneFileMs)
const t1 = Now.asTimestamp()
const { fileIds, fileRelativePaths } = _catalogFiles(
parentFolder,
mimeType,
)
const t2 = Now.asTimestamp()
_setQueues(fileIds, [])
const t3 = Now.asTimestamp()
this.fileRelativePaths = fileRelativePaths
ScriptProps.setAsJson(_propsKeyFileRelativePaths, fileRelativePaths)
const t4 = Now.asTimestamp()
_releasePropsLock()
const t5 = Now.asTimestamp()
if (spawnFunction) {
// only triggered on the first thread
const trigger = Trigger.create(spawnFunction, 1)
log(
`Trigger once per minute: UniqueId: ${trigger.getUniqueId()}, EventType: ${trigger.getEventType()}, HandlerFunction: ${trigger.getHandlerFunction()}, TriggerSource: ${trigger.getTriggerSource()}, TriggerSourceId: ${trigger.getTriggerSourceId()}.`,
)
}
log(
`PFI instantiated for the first time, has found ${
fileIds.length
} documents to process. getPropsLock took ${t1 -
t0}ms, _catalogFiles took ${t2 - t1}ms, setQueues took ${t3 -
t2}ms, setAsJson took ${t4 - t3}ms, releasePropsLock took ${t5 -
t4}ms, trigger creation took ${Now.asTimestamp() - t5}ms.`,
)
} else {
const t0 = Now.asTimestamp()
// wait for first thread to set up Properties
while (!ScriptProps.getJson(_propsKeyFileRelativePaths)) {
Utilities.sleep(250)
}
this.fileRelativePaths = ScriptProps.getJson(_propsKeyFileRelativePaths)
const t1 = Now.asTimestamp()
log(
`PFI instantiated again to run in parallel. getJson(paths) took ${t1 -
t0}ms`,
)
spawnFunction
}
_internals.set(this, { maxDurationOneFileMs: maxDurationOneFileMs })
// to get: _internal(this, 'maxDurationOneFileMs')
ParallelFileIterator.instance = this
return ParallelFileIterator.instance
}
nextId() {
// returns false if there are no more documents
const maxDurationOneFileMs = _internals.get(this).maxDurationOneFileMs
_getPropsLock(maxDurationOneFileMs)
let { pending, dispatched } = _getQueues()
log(
`PFI.nextId: ${pending.length} files pending, ${
dispatched.length
} dispatched, ${Object.keys(this.fileRelativePaths).length -
pending.length -
dispatched.length} completed.`,
)
if (pending.length) {
// get first pending Id, (ie, deepest first)
const nextId = pending.shift()
dispatched.push([nextId, Now.asTimestamp()])
_setQueues(pending, dispatched)
_releasePropsLock()
return nextId
} else if (dispatched.length) {
log(`PFI.nextId: Get first dispatched Id, (ie, oldest first)`)
let startTime = dispatched[0][1]
let timeToTimeout = startTime + maxDurationOneFileMs - Now.asTimestamp()
while (dispatched.length && timeToTimeout > 0) {
log(
`PFI.nextId: None are pending, and the oldest dispatched one hasn't yet timed out, so wait ${timeToTimeout}ms to see if it will`,
)
_releasePropsLock()
Utilities.sleep(timeToTimeout + 500)
_getPropsLock(maxDurationOneFileMs)
;({ pending, dispatched } = _getQueues())
if (pending && dispatched) {
if (dispatched.length) {
startTime = dispatched[0][1]
timeToTimeout =
startTime + maxDurationOneFileMs - Now.asTimestamp()
}
}
}
// We currently still have the PropsLock
if (dispatched.length) {
const nextId = dispatched.shift()[0]
log(
`PFI.nextId: Document id ${nextId} has timed out; reset start time, move to back of queue, and re-dispatch`,
)
dispatched.push([nextId, Now.asTimestamp()])
_setQueues(pending, dispatched)
_releasePropsLock()
return nextId
}
}
log(`PFI.nextId: Both queues empty, all done!`)
;({ pending, dispatched } = _getQueues())
if (pending.length || dispatched.length) {
log(
"ERROR: All documents should be completed, but they're not. Giving up.",
pending,
dispatched,
)
}
_cleanUp()
return false
}
completed(fileId) {
_getPropsLock(_internals.get(this).maxDurationOneFileMs)
const { pending, dispatched } = _getQueues()
const newDispatched = dispatched.filter(el => el[0] !== fileId)
if (dispatched.length !== newDispatched.length + 1) {
log(
'ERROR: A document was completed, but not found in the dispatched list.',
fileId,
pending,
dispatched,
)
}
if (pending.length || newDispatched.length) {
_setQueues(pending, newDispatched)
_releasePropsLock()
} else {
log(`PFI.completed: Both queues empty, all done!`)
_cleanUp()
}
}
getFileRelativePath(fileId) {
return this.fileRelativePaths[fileId]
}
}
// ============= PRIVATE MEMBERS ============= //
const _propsKeyLock = 'PropertiesLock'
const _propsKeyDispatched = 'Dispatched'
const _propsKeyPending = 'Pending'
const _propsKeyFileRelativePaths = 'FileRelativePaths'
// Not really necessary for a singleton, but in case code is changed later
var _internals = new WeakMap()
const _cleanUp = (exceptProp = null) => {
log('Enter _cleanUp', exceptProp)
Trigger.deleteAll()
if (exceptProp) {
ScriptProps.deleteAllExcept(exceptProp)
} else {
ScriptProps.deleteAll()
}
}
const _catalogFiles = (folder, mimeType, relativePath = []) => {
// returns IDs of all matching files in folder, depth first
log(
'Enter _catalogFiles',
folder.getName(),
mimeType,
relativePath.join('/'),
)
let fileIds = []
let fileRelativePaths = {}
const folders = folder.getFolders()
let subFolder
while (folders.hasNext()) {
subFolder = folders.next()
const results = _catalogFiles(subFolder, mimeType, [
...relativePath,
subFolder.getName(),
])
fileIds = fileIds.concat(results.fileIds)
fileRelativePaths = { ...fileRelativePaths, ...results.fileRelativePaths }
}
const files = folder.getFilesByType(mimeType)
while (files.hasNext()) {
const fileId = files.next().getId()
fileIds.push(fileId)
fileRelativePaths[fileId] = relativePath
}
return { fileIds: fileIds, fileRelativePaths: fileRelativePaths }
}
const _getQueues = () => {
const pending = ScriptProps.getJson(_propsKeyPending)
const dispatched = ScriptProps.getJson(_propsKeyDispatched)
log('Exit _getQueues', pending, dispatched)
// Note: Empty lists in Javascript are truthy, but if Properties have been deleted by another thread they'll be null here, which are falsey
return { pending: pending || [], dispatched: dispatched || [] }
}
const _setQueues = (pending, dispatched) => {
log('Enter _setQueues', pending, dispatched)
ScriptProps.setAsJson(_propsKeyPending, pending)
ScriptProps.setAsJson(_propsKeyDispatched, dispatched)
}
const _getPropsLock = maxDurationOneFileMs => {
// will block until lock available or lock times out (because a script may be killed while holding a lock)
const t0 = Now.asTimestamp()
while (
ScriptProps.getNum(_propsKeyLock) + maxDurationOneFileMs >
Now.asTimestamp()
) {
Utilities.sleep(2000)
}
ScriptProps.set(_propsKeyLock, Now.asTimestamp())
log(`Exit _getPropsLock: took ${Now.asTimestamp() - t0}ms`)
}
const _releasePropsLock = () => {
ScriptProps.delete(_propsKeyLock)
log('Exit _releasePropsLock')
}
return ParallelFileIterator
})()
const log = (...args) => {
// easier to turn off, json harder to read but easier to hack with
console.log(args.map(arg => JSON.stringify(arg)).join(';'))
}
class Trigger {
// Script triggering facade
static create(functionName, everyMinutes) {
return ScriptApp.newTrigger(functionName)
.timeBased()
.everyMinutes(everyMinutes)
.create()
}
static delete(e) {
if (typeof e !== 'object') return log(`${e} is not an event object`)
if (!e.triggerUid)
return log(`${JSON.stringify(e)} doesn't have a triggerUid`)
ScriptApp.getProjectTriggers().forEach(trigger => {
if (trigger.getUniqueId() === e.triggerUid) {
log('deleting trigger', e.triggerUid)
return ScriptApp.delete(trigger)
}
})
}
static deleteAll() {
// Deletes all triggers in the current project.
var triggers = ScriptApp.getProjectTriggers()
for (var i = 0; i < triggers.length; i++) {
ScriptApp.deleteTrigger(triggers[i])
}
}
}
class ScriptProps {
// properties facade
static set(key, value) {
if (value === null || value === undefined) {
ScriptProps.delete(key)
} else {
PropertiesService.getScriptProperties().setProperty(key, value)
}
}
static getStr(key) {
return PropertiesService.getScriptProperties().getProperty(key)
}
static getNum(key) {
// missing key returns Number(null), ie, 0
return Number(ScriptProps.getStr(key))
}
static setAsJson(key, value) {
return ScriptProps.set(key, JSON.stringify(value))
}
static getJson(key) {
return JSON.parse(ScriptProps.getStr(key))
}
static delete(key) {
PropertiesService.getScriptProperties().deleteProperty(key)
}
static deleteAll() {
PropertiesService.getScriptProperties().deleteAllProperties()
}
static deleteAllExcept(key) {
PropertiesService.getScriptProperties()
.getKeys()
.forEach(curKey => {
if (curKey !== key) ScriptProps.delete(key)
})
}
}
If you're a business customer, you can now sign up for Early Access to App Maker, which includes Flexible Quotas.
Under the flexible quota system, such hard quota limits are removed. Scripts do not stop when they reach a quota limit. Rather, they are delayed until quota becomes available, at which point the script execution resumes. Once quotas begin being used, they are refilled at a regular rate. For reasonable usage, script delays are rare.
If you are using G Suite as a Business, Enterprise or EDU customer the execution time for running scripts is set to:
30 min / execution
See: https://developers.google.com/apps-script/guides/services/quotas
The idea would be to exit gracefully from the script, save your progress, create a trigger to start again from where you left off, repeat as many times as necessary and then once finished clean up the trigger and any temporary files.
Here is a detailed article on this very topic.
As many people mentioned, the generic solution to this problem is to execute your method across multiple sessions. I found it to be a common problem that I have a bunch of iterations I need to loop over, and I don't want the hassle of writing/maintaining the boilerplate of creating new sessions.
Therefore I created a general solution:
/**
* Executes the given function across multiple sessions to ensure there are no timeouts.
*
* See https://stackoverflow.com/a/71089403.
*
* #param {Int} items - The items to iterate over.
* #param {function(Int)} fn - The function to execute each time. Takes in an item from `items`.
* #param {String} resumeFunctionName - The name of the function (without arguments) to run between sessions. Typically this is the same name of the function that called this method.
* #param {Int} maxRunningTimeInSecs - The maximum number of seconds a script should be able to run. After this amount, it will start a new session. Note: This must be set to less than the actual timeout as defined in https://developers.google.com/apps-script/guides/services/quotas (e.g. 6 minutes), otherwise it can't set up the next call.
* #param {Int} timeBetweenIterationsInSeconds - The amount of time between iterations of sessions. Note that Google Apps Script won't honor this 100%, as if you choose a 1 second delay, it may actually take a minute or two before it actually executes.
*/
function iterateAcrossSessions(items, fn, resumeFunctionName, maxRunningTimeInSeconds = 5 * 60, timeBetweenIterationsInSeconds = 1) {
const PROPERTY_NAME = 'iterateAcrossSessions_index';
let scriptProperties = PropertiesService.getScriptProperties();
let startTime = (new Date()).getTime();
let startIndex = parseInt(scriptProperties.getProperty(PROPERTY_NAME));
if (Number.isNaN(startIndex)) {
startIndex = 0;
}
for (let i = startIndex; i < items.length; i++) {
console.info(`[iterateAcrossSessions] Executing for i = ${i}.`)
fn(items[i]);
let currentTime = (new Date()).getTime();
let elapsedTime = currentTime - startTime;
let maxRunningTimeInMilliseconds = maxRunningTimeInSeconds * 1000;
if (maxRunningTimeInMilliseconds <= elapsedTime) {
let newTime = new Date(currentTime + timeBetweenIterationsInSeconds * 1000);
console.info(`[iterateAcrossSessions] Creating new session for i = ${i+1} at ${newTime}, since elapsed time was ${elapsedTime}.`);
scriptProperties.setProperty(PROPERTY_NAME, i+1);
ScriptApp.newTrigger(resumeFunctionName).timeBased().at(newTime).create();
return;
}
}
console.log(`[iterateAcrossSessions] Done iterating over items.`);
// Reset the property here to ensure that the execution loop could be restarted.
scriptProperties.deleteProperty(PROPERTY_NAME);
}
You can now use this pretty easily like so:
let ITEMS = ['A', 'B', 'C'];
function execute() {
iterateAcrossSessions(
ITEMS,
(item) => {
console.log(`Hello world ${item}`);
},
"execute");
}
It'll automatically execute the internal lambda for each value in ITEMS, seamlessly spreading across sessions as needed.
For example, if you use a 0-second maxRunningTime it would run across 4 sessions with the following outputs:
[iterateAcrossSessions] Executing for i = 0.
Hello world A
[iterateAcrossSessions] Creating new session for i = 1.
[iterateAcrossSessions] Executing for i = 1.
Hello world B
[iterateAcrossSessions] Creating new session for i = 2.
[iterateAcrossSessions] Executing for i = 2.
Hello world C
[iterateAcrossSessions] Creating new session for i = 3.
[iterateAcrossSessions] Done iterating over items.