I am trying to limit concurrent use of apps script for different parts of the same script, but no matter what I do, one lock will lock the whole thing down. Here's my sample script:
function lockTest(event) {
var r_eventRange = event.range;
var value = r_eventRange.getValue();
if (value == "A"){
var lock1 = LockService.getScriptLock();
lock1.waitLock(10000);
//do stuff
lock1.releaseLock();
} else if (value == "B") {
var lock2 = LockService.getScriptLock();
lock2.waitLock(10000);
//do different stuff
lock2.releaseLock();
}
}
Function 'lockTest' is triggered by an onEdit event. The intent for this script is that there can be two values: "A", and "B". If I get consecutive calls where both have value "A", then force the calls to wait to execute one at a time. Same thing if I get consecutive calls where both have value "B". But if I get consecutive calls where value is "A" in one and "B" in the other, then go ahead and let the code run consecutively without waiting.
However, the with this script the 'lock service' will not allow consecutive runs regardless of the value. In other words, if the first call has value "A", and the second call comes before the first is done but the value is "B", the lock service is forcing the second call to wait for the first call to complete even though the locks are supposed to be different.
Is it possible to control the locking on different parts of the script?
Basically, I need value "A" to trigger lock1 which edits a specific set of shared resources. value "B" should trigger lock2 which edits a completely different set of shared resources than lock1. That means that lock1 and lock2 should be able to run at the same time because the resources they edit are completely different. In fact I need them to run at the same time because lock1 will be triggered a lot more frequently than lock2 and lock1 takes only about 2 seconds to execute. lock2 is triggered much less frequently but takes 30-150 seconds to execute. So I need lock1 to be able to run even if lock2 is already running because otherwise lock1 will either time out or there could be so many instances of lock1 waiting in line while lock2 finishes that the 'lock service' will start throwing errors.
Interestingly enough, if I put a logger before the if() statement, I get logs instantly every time an edit event happens - meaning that the 'lock service' in only locking down the if statement and not the lines before it.
Google's documentation for 'lock service' says
getScriptLock() Gets a lock that prevents any user from concurrently
running a section of code. A code section guarded by a script lock
cannot be executed simultaneously regardless of the identity of the
user.
It seems like from this that we should be able to lock specific sections of code. I just can't figure out how to designate what those individual sections are. I assumed the releaseLock() method would tell it where to end the section. And maybe it does - I'm concerned that the 'lock service' doesn't support designating multiple independent locks, which is what I need here.
According to Oleg's suggestion:
function lockTest(event) {
var r_eventRange = event.range;
var value = r_eventRange.getValue();
if (value == "A"){
checkFormatting();
} else {
doAction();
}
}
function checkFormatting(){
var lock1 = LockService.getScriptLock();
lock1.waitLock(10000);
//do stuff
lock1.releaseLock();
}
function doAction(){
var lock2 = LockService.getScriptLock();
lock2.waitLock(10000);
//do different stuff
lock2.releaseLock();
}
Unfortunately I'm having the same issue with the above code.
I have the same problem as you do. It seems the locks provided by LockService are singletons, and thus your lock.waitLock() calls will always wait irregardless of where the lock was grabbed initially.
I recommend you look at this library providing "named locks" which is what you are looking for here. Have not used it yet, but looks promising.
https://ramblings.mcpher.com/gassnippets2/using-named-locks-with-google-apps-scripts/
LockService provides a single mutex (aka lock) which you can lock/unlock.
Recall that multiple JS runtimes can run in your Google Sheet at a time. The guarantee that LockService provides is that when waitLock() succeeds, you know that you are the only JS runtime that has obtained the lock. All other runtimes are either doing something which doesn't care about the lock, or are waiting for you to release the lock.
const mutex = LockService.getScriptLock();
const TIMEOUT_MS = 10*1000;
function doStuff() {}
function doTheFirstThing() {
// Process some local data. Don't need the lock yet.
doStuff();
// Increment a cell. We definitely need the lock here,
// because we need to read and write in a row without
// somebody changing the value under our nose.
mutex.waitLock(TIMEOUT_MS);
const range = SpreadsheetApp.getActiveSpreadsheet.getRange('A1');
range.setValue(range.getValue()+1);
mutex.releaseLock();
}
function doTheSecondThing() {
// Process some different local data.
// We also don't care about the lock yet.
// Decrement a cell. We definitely need the lock here,
// because we need to read and write in a row without
// somebody changing the value under our nose.
mutex.waitLock(TIMEOUT_MS);
const range = SpreadsheetApp.getActiveSpreadsheet.getRange('A1');
range.setValue(range.getValue()-1);
mutex.releaseLock();
}
So a critical section is not defined really by a code block persay - it is defined by where you lock and unlock a specific mutex. In my example, doTheFirstThing() and doTheSecondThing() share a critical section. What you can do to achieve named locks (and thus distinct critical sections) is maintain a (synchronized) map of (lockName => isLocked) and use the one global lock you get to access said map.
However, be wary of performance here, as locking any named lock will lock the global lock.
Related
Each time we deploy our Google AppScript Project, our custom function starts returning #ERROR!. This will happen regardless of whether a code change was made.
See photo below:
NOTE: Internal error executing the custom function. is not one of our error strings.
This is very strange because the function does not seem to be executing. I say this because #ERROR! is returned immediately, with 0 processing time. See failures in photo below:
This issue resolves itself after some seemingly arbitrary amount of time. Meaning the custom function will run normally, after some seemingly arbitrary amount of time.
This has become a very large problem because we have uncontrollable downtime after each deployment, and it does not seem to be an issue with our code considering this happens every time we deploy the code, regardless of whether the code actually changed.
This Google document states A custom function call must return within 30 seconds. If it does not, the cell will display an error: Internal error executing the custom function.. Our custom function does not take 30s to run. We actually can't even find an instance where our function runs longer than 5s.
NOTE: the only thing that fails is our custom function, our task pane that interacts with the Google Sheet remains functional.
According to the Optimization section of Custom Functions:
Each time a custom function is used in a spreadsheet, Google Sheets
makes a separate call to the Apps Script server.
Having multiple custom functions means you are making multiple calls to the server simultaneously. This could lead to a slow process or sometimes Error.
Solution:
The workaround to this is to lessen the use of custom functions. Custom function can accept range as parameter and it will be translated as two-dimensional array in Apps Script. Use the array to calculate the values and return a two-dimensional array that can overflow into the appropriate cells in your spreadsheet.
Example:
Code:
function multiplicationTable(row, col){
var table = [];
var rowArr = row.flat();
var colArr = col.flat();
for (var i = 0; i < rowArr.length ; i++) {
table.push([]);
for (var j = 0; j < col.length; j++) {
table[i].push(rowArr[i] * colArr[j]);
}
}
return table;
}
My situation is pretty simple. I'm using lockservice to avoid concurrent update of one row by different instances of same script, like this:
LockService.getScriptLock().waitLock(5000);
if(rowIndex > 1) {
row = sheet.loadRowAt(rowIndex);
sheet.saveRowAt(_.assign(row, newValues), rowIndex);
} else {
sheet.appendRow(newValues)
}
SpreadsheetApp.flush();
LockService.getScriptLock().releaseLock();
(sheet is my own obj, not internal Sheet so don't worry for unknown methods)
The question is - do I need a line with SpreadsheetApp.flush() to make sure changes by this section would be immediately available to parralel script run, which might already be waiting for the lock.
Apparently, I want to keep the code inside lock as quick as possible, so if this line is redundant, I want to get rid of it.
To re-phrase the question, does the SpreadsheetApp.flush() actually affects sheet values in memory OR it only relates to display of them to end-user currently observing the sheet and I can forget about it until I don't care about end-user display delay?
I have a piece of Go code with 3 different functions, insertIntoMysql, updateRowMysql and deleteRowmysql. I check for operation type and run one of these functions as needed.
I want to convert my normal functions into go routines to be able to handle more operations.
But here is the issue:
If I convert into goroutines, I will lose the sequence of operations.
For example, the insert operations are much more frequent than the delete operations and insert operations are being queued in the insert channel while the delete channel is empty it is possible for my code to try to delete a row before it gets inserted (e.g. a row is inserted and the deleted 1 sec later).
Any ideas on how to make sure the sequence of my operations on mysql is the same as the received operations.
here is the code:
go insertIntoMysql(insertChan, fields, db, config.DestinationTable)
go updatefieldMysql(updateChan, fields, db, config.DestinationTable)
go deleteFromMysql(deleteChan, fields, db, config.DestinationTable)
for !opsDone {
select {
case op, open := <-mysqlChan:
if !open {
opsDone = true
break
}
switch op.Operation {
case "i":
//fmt.Println("got insert operation")
insertChan <- op
break
case "u":
fmt.Println("got update operation")
updateChan <- op
break
case "d":
fmt.Println("got delete operation")
deleteChan <- op
break
case "c":
fmt.Println("got command operation")
//no logic yet
break
}
break
}
}
close(done)
close(insertChan)
close(deleteChan)
close(updateChan)
}
Looking at your code and your question/requirement: Stay in order as the original go channel delivered the data.
Staying in sync and still firing off multiple go routines to process the data, could be done like this:
func processStatements(mysqlChan chan *instructions) {
var wg sync.WaitGroup
prevOp := "" // Previous operation
for {
op, open := <-mysqlChan
if !open {
break
}
if prevOp!=op.Operation {
// Next operation type, unknown side effects on previous operations,
// wait for previous to finish before continuing
wg.Wait()
}
switch op.Operation {
case "i":
wg.Add(1)
go processInsert(op,wg)
break
case "u":
wg.Add(1)
go processUpdate(op,wg)
break
case "d":
wg.Add(1)
go processDelete(op,wg)
break
case "c":
break
}
prevOp = op.Operation
}
// Previous go routines might still be running, wait till done
wg.Wait()
// Close any channels
}
func processInsert(c chan *sqlCall,wg *sync.WaitGroup) {
defer wg.Done()
// Do actual insert etc
}
The main differences with your program are:
Processing is in order, safe against your base scenario
Process of next operation type will wait until previous operation type is finished
Multiple of the same operations run in parallel (as for in your code a single operation per type would run in parallel). Depending on the data distribution either of them can be faster or slower (i,i,i,d,u = 3 waits in this code, while it would also be 3 waits in your code, however in your code the 3 i would run sequential, here they run parallel) (Look into insert or update for mysql, since now your inserts could suddenly be updates depending on your data).
How would the one that calls delete know about the row in the first place? From a SQL perspective: if it is not committed, it does not exist. Something calls your code to insert. Your code should return only after the transaction is completed. At which point the caller could start select, update or delete.
For example, in a web server: Only the client that called insert knows of the existence to the record initially. Other clients will only learn of its existence after running select. That select will only return the new record if it was committed in the transaction that inserted it. Now they might decide to upsert or delete.
Many SQL databases take care of proper row locking, which will ensure data integrity during concurrent access. So if you use Go routines to write to the same record, they will become sequential (in any given order) on the DB side. Concurrent reads could still happen.
Please note that net/http already runs a Go routine for every request. And so do most other server front-ends for Go. If you are writing something different altogether, like a custom TCP listener, you could initiate your own Go routine for each request.
As far as I understand the problem, the following can be done as a resolution :
Call Insert, Delete or Update as go routines and they will run concurrently and performing all these operations.
Ensure you have row-level locking in your MySQL DB (I am sure InnoDb provides that)
For your delete and update operations, have an isExist check to check if the row you are updating/deleting exists before deleting and updating the row, this allows for some level of control.
You can implement a retry mechanism in your channel (with jitter preferably) to ensure that even if Delete Operation goes before insert operation, it will fail the exist check and then can be retried(after a second or 0.5 second or some config) and this will ensure insert record is already performed before delete or update operation is retried.
Hope this helps.
So I have some code running off a button click. There are a number of functions, called one after another. In some of them there are calls to REST services. Occasionally, the whole shebang runs over 6 minutes thus hitting the timeout barrier which, unlike the sound-barrier, cannot be broken.
Is the timeout trappable? Can I wrap the most likely offender in a try/catch and have the catch block evaluate at the appropriate moment? Or am I stuck with trying to make things just so incredibly performant that I never hit the wall?
Given that you're calling multiple functions, I'd recommend creating a trigger instead using ScriptApp.newTrigger, which could be triggered by introducing another function which checks whether or not the script has exceeded the predetermined execution time (if the max is 6, you could set it to be 5 to be safe).
Referring to something like this -
// var today = new Date();
// 'today' is declard in the original script
function isTimeUp(today) {
var now = new Date();
return now.getTime() - today.getTime() > 30000;
// 30000 = 30 seconds; this is the threshold limit
// you are free to setup your own threshold limit
}
I've written more on this topic here where i've used a .timeBased() trigger; however, you may use a different kind of trigger altogether.
I'm using Google Spreadsheet for my tax administration. All financial transactions that I have done in a year are in the spreadsheet. Because the tax rules in my country are quite complex, I've developed a number of JavaScript functions to help me with my calculations. There are many rows in my spreadsheet, about 1000. Each row has multiple references to those JavaScript functions.
This system worked beautifully before, but today I've found that Google installed some kind of runtime limiting system into Google Spreadsheet, causing many of my columns to abort with the following error:
Service invoked too many times in a short time: exec maxSimultaneous.
Try Utilities.sleep(1000) between calls.
After some investigation, it would appear that this time limit is there to protect against scripts that take too long time to run. My case is different: my scripts are all short, O(1) algorithms that do nothing but calculating some numbers. A typical script looks like this:
// Calculates the total amount excluding Value Added Tax, given
// an amount including Value Added Tax, and other sorts of information.
function ex_btw(inc_btw, commentaar, soort, btw_verlegd, btw_pct) {
Utilities.sleep(1000);
var soort = soort.toLowerCase();
if (soort == 'eten en drinken'
|| commentaar.match(/treinkaartje/i)
|| commentaar.match(/treinticket/i)
|| commentaar.match(/taxi/i)
|| commentaar.match(/ boek /i))
{
return inc_btw / 1.06;
} else if (soort == 'priveonttrekking'
|| soort == 'boete'
|| soort == 'belasting'
|| commentaar.match(/postzegel/i)
|| btw_verlegd == 'Ja')
{
return inc_btw;
} else {
return inc_btw / (1 + btw_pct);
}
}
The script is then invoked like this from a cell:
=IF(B6<>""; ex_btw(B6;D6;E6;J6;S6); "")
Maybe my problem is that I have too many script calls. Every single row calls about 6 of such scripts, so with 1000 rows I call 6000 times per spreadsheet.
How do I solve this problem? Is there a way to increase the execution limit, or is there a way to make the scripts run slower so that they don't hit the execution limit? As you can see in the example code I've already tried inserting Utilities.sleep(1000), but that doesn't seem to solve the problem. I don't care whether the scripts run slowly, I just just care that they finish without errors.
Can I pay to have the limit increased? I need to hand in my taxes in a few days.
Other alternatives that I've considered, but that are not feasible.
Using non-JavaScript functions. Not feasible because: they don't support collaboration like Google Spreadsheet does. I regularly go over the spreadsheet with a colleague to check whether we've made any mistakes. It helps that the both of us can immediately see any changes the other makes.
Have one huge-ass JavaScript function that iterates over rows and populates cells. Not feasible because:
Too error prone, it's very easy to make mistakes compared to my current method.
Does not update cells automatically until I re-run the script. I want to see any calculations immediately after I update other cells, just like a spreadsheet is supposed to do.
Using other spreadsheets like Excel and OpenOffice Calc. Not feasible because: they don't appear to offer the same scripting capabilities.
Writing my own financing app. Not feasible because: it takes too much take, it's not my core business, and tax rules change almost every year so I will have to constantly update the app. I can update a spreadsheet very quickly, but writing a financing app takes too much time.
I solved it by making every function sleep for a random period, like this:
Utilities.sleep(Math.random() * 5000);
It is important that the sleeping time is random, not constant. Apparently Google limits the maximum number of functions that may simultaneously be using CPU.
an alternative to the custom function might be to have an onEdit function trigger and then process either just the entered data or the whole column of numbers and place the results of the function in the target cell(s) as a number.
might be quicker