Google Apps Script - Set Values randomly failing - google-apps-script

Someone on stack overflow from 2016 reported this issue happening in debug mode only but this isn't debug mode, it is happening in my live code (run from webapp) running in backend AND debug mode.
The frustrating thing is that this isn't every call to setValues. There is one array call to setvalues that does work (writes to the spreadsheet) higher in the script.
This code has not been modified in more than a month and was working until at least 10-26 (first failure reported 11-2).
Here is a small example:
function setValuesIsFailing(sheet, array, row, column){
//newIDR_SHEET.getRange(3, 9, formulaColumns.length, 2).setValues(formulaColumns);//old command this function replaces
const targetRange = sheet.getRange(row,column,array.length,array[0].length);
console.log(139,targetRange.getA1Notation());
sheet.getRange("C5").setValue("UGH");
targetRange.setValues(array);
console.log(sheet.getRange("C5").getValue());
}
The result of this code is the expected console log of :
10:46:10 AM Info 139 'I3:J49'
10:46:11 AM Error
Exception: Service Spreadsheets failed while accessing document with id 12shsTE-Mxxx
setValuesIsFailing # Code.gs:143
So line 139 implies that it does have the sheet it is writing to and then it
doesn't throw an error when told to write to those two locations (but it does
not succeed in writing to them) and after that it throws an error when asked
to read from one of the locations.
To recap:
code was working just fine. Had not been modified for a month and was working correctly most of that month.
setValue / setValues commands fail silently without throwing an error (nothing is written to target)
service spreadsheet error occurs later when trying to read from places that should have been modified by the script but were not
Moving the problem to a new function obviously didn't fix it.
Other things I tried, logging the sheet name, reassigning a new variable to the sheet (originalyl concerned that somehow the function was losing the sheet but debugging shows that isn't the case), moving the failing piece of code to a new function for easier logging, waiting for the problem to resolve itself.
Any ideas?
=====
Solved:
It turned out that the weird behavior was caused by a failed attempt to apply data validation to the sheet.
Unlikely that anyone else will have this behavior, but I was pulling my data validations down from a sheet a single column at a time.
Either it was accepting a 2d array of one column as acceptable to the data validation setup command OR it was automatically flattening the array since it was only one column. This has changed. Manually flattening my column data with a .flat() fixed the issue. I would really have expected the bad validation setting call to throw an error though.

I discovered that a bad validation application was causing the spreadsheet to basically stop being able to be updated.
It was reading a 2d array (one column) and applying that as a validation list. When I flattened the validation list it worked.

Related

Has my self-invoking workaround worked or is it coincidence?

I've been having an issue with a Google Apps Script I'm running that kept throwing the following error when ever I triggered the code onChange. No such errors were returned when I manually ran the code.
Exception: Service Documents failed while accessing document with id XXXX
The error was happening on the same line of code each time and I've subsequently found that this is a relatively common error and is perhaps happening because my script is quite lengthy and perhaps not as efficient as it should be (I'm very much an Apps Script and JS novice). Interestingly I have a very similar script (with perhaps 20% less variables) running in another file that executes as expected every time.
I had no naming or scoping issues with the variable that was being used to call the document but I thought that perhaps wrapping the troublesome part of the script into a self-invoking function within the larger function would minimise hoisting and improve the efficiency of the script - at least enough to let it run consistently without errors. My understanding of JS is that when a script initiates variable declarations are hoisted within their scope and therefore by creating a self-invoking function I could reduce the hoisting within the main function which contains in excess of 100 variables and therefore reduce the initial demand.
So far the script does appear to be running more quickly and avoiding the error that I'd previously being seeing - the last run was just over 63 seconds whereas the previous successful manual run without the self-invoking function was just under 103 seconds.
I do believe this can be an intermittent error and I'm trying to find a robust, longer term fix without having to rewrite all of my code.
I have detailed the self-invoking code below with any IDs redacted. The part that was causing the error within the script was "var docFinal = DocumentApp.openById(docFinalId);
Do you think this could be a genuine fix or has the code started to work coincidently because of the intermittent nature of this error?
var docTempId = "XXXX";//Template File Id
var docFinalId = "XXXX"; //Final File Id
var sheetId = "XXXX";
(function () { // self-invoked within larger function to minimise hoisting
var docTemp = DocumentApp.openById(docTempId);
var docFinal = DocumentApp.openById(docFinalId);
docFinal.getBody().clear();
var templateParagraphs = docTemp.getBody().getParagraphs();
createMailMerge(Company,Branch,PropertyID,PropertyAddress,ApplicantName,Dateofbirth1,EmailAddress,PhoneNumber,Nationality,PassportNumber,NationalInsuranceNumber,Currentresidentialstatus,LeaseRequested,RentOfferedPCM,TotalNumberofAdultOccupants,Proposedleasecommencementdate1,TotalRentPayers,RentPayer2Name,RentPayer2Phone,RentPayer2Email,RentPayer3Name,RentPayer3Phone,RentPayer3Email,Relationshipwithadultoccupants,Numberofchildren,Ageofchildchildren,Currenteconomicstatus,Applicantoccupation,Applicantemployedorselfemployed,ApplicantDeclaredIncome,SelfEmployedDocuments,EmploymentPartorFullTime,EmploymentContractorPermanent,MainEmploymentCompany,Mainemploymentaddress,Mainemploymentstartdate1,Mainemploymemtpensionpayrollnumber,MainEmploymentManager,ManagerEmail,ManagerPhoneNumber,ApplicantPaymentType,ApplicantHourlyRate,Applicantprimaryaveragehourspermonth,Applicantsalary,ReceivesHousingBenefit,HousingBenefitAmount,Anyadditionalincome,Typeofadditonalincome,Secondemploymentcompany,Rolewithin2ndCompany,ndEmployeraddress,ndEmploymentstartdate1,ndEmployerpensionpayrollnumber,ndEmploymentContact,ndEmploymentEmail,ndEmploymentphonenumber,Additionalincomeamount,Additionalincomedetails,TotalDeclaredGrossIncome,Applicansavingsdeclared,MostRecentAddress,DateStartedlivingincurrentaddress1,Liveanywhereelseinlast3years,Applicant2ndresidingaddress,Applicant2ndaddressmoveindate1,Applicant2ndaddressmoveoutdate1,Applicantadditionaladdressdeclared,Applicantadditionaladdressdetails,Applicantadditonaladdressmoveindate1,Applicantadditionaladdressmoveoutdate1,Applicantpreviouslandlordreference,Landlordreferenceaddress,referencefromlandlordoragent,LandlordAgentName,lengthoftimeatproperty,LandlordAgentphonenumber,LandlordAgentemailaddress,Previouslandlordreferencepreventionreason,Anypets,Petdetails,Applicantsmoke,Applicantsmokeinside,Applicantadversecredit,Adversecreditdetails,Applicantprovidecurrentaccount,Applicantcurrentaccountname,Applicantcurrentaccountbank,Applicantcurrentaccountnumber,Applicantcurrentaccountsortcode,UKbasedguarantor,GuarantorName,GuarantorEmail,GuarantorPhoneNo,Noguarantorreason,NextofKinName,NextofKinrelationship,NextofKinEmail,NoNextofKinPhoneNo,Applicantadditionalinfo,Applicantdocuments,Applicantaccurateinformationdeclaration,Applicantaccepttermsandconditions,submittedatt,Token,maidenname,ApplicantReferencingChoice,Applicantcanprovide,Applicantdocumentlink,ApplicantacceptsHomelet,ApplicantallowsHomelettocontactreferences,ApplicanthappyforHomelet,templateParagraphs,docFinal);
docFinal.saveAndClose();
createPDF(); // calls the next function
}) ();
The script has just failed again. After working on a number of occasions which hadn't been happening, it has now failed and return the same error as before. I can therefore only assume that creating a self-invoking function within the main function has made no material difference to the efficiency of the script.
The error you are receiving does not seem to be the expected one.
I think the best solution in this situation is to file a bug on Google's Issue Tracker by using the template here.

Data Studio Connector can't get access token for BigQuery Service Account: Access not granted or expired

I'm trying to make a community connector to connect my database in BigQuery to data studio with the service account that I hooked up as the Owner/DataViewer/JobUser of the BigQuery project. I know that the service account works when connecting to BigQuery because I've tested it elsewhere. I copied from the connector code from this tutorial (https://developers.google.com/datastudio/solution/blocks/using-service-accounts) almost exactly, replacing the SQL string with my query and adding some different query parameters. I also stored the service account's credentials in my script properties by pasting the json object and storing it like:
var service_account_creds_obj = {
"type": "service_account",
"project_id": ...
...
}
scriptProperties.setProperty('SERVICE_ACCOUNT_CREDS', JSON.stringify(service_account_creds_obj));
However, I always get stuck in the flow when my getData function calls getOauthService().getAccessToken(), which doesn't ever successfully return. When I create a report using the connector, I get this error: "Access not granted or expired." I can't find the documentation for getAccessToken and I'm having trouble understanding why it won't terminate. I can see that it doesn't return because a console.log immediately before that line displays but it never gets to the log on the next line. Then my try-catch block catches the error that I'm seeing. Note that my getOauthService function is exactly the same as the one from the documentation/tutorial example, except that I've played around with the input text in the call to createService. That input text shouldn't really matter though right?
Please, I've been trying to debug this for hours, but the documentation on this is pretty horrible, and it's really hard to debug since the flow of the code is handled in the background and stackdriver logging is really buggy.
I figured out my problem. The documentation posted above said to set the OAuth2 scope to https://www.googleapis.com/auth/bigquery.readonly. However, I naively included
"oauthScopes": ["https://www.googleapis.com/auth/bigquery.readonly"]
in my manifest file. Meanwhile, the code I copied over from the documentation already included this line:
.setScope(['https://www.googleapis.com/auth/bigquery.readonly']);
So I'm not sure exactly why this caused a problem. But it must have prevented the OAuth2.createService function from properly getting set.

Exception: The data execution is not started because the data object is currently in error state. DataSource.updateSpec()

Trying to assist an analyst with this error it's written in google app scripts. The job is scheduled every hour and it occasionally fails with. This executes a sql in bigquery and populates a google sheet.
Exception: The data execution is not started because the data object is currently in error state. Please check and update specification with DataSource.updateSpec().
at RefreshDataSource(macros:9:57)
You can rerun it and it works perfectly. Not sure where to start debugging this, any assistance is appreciated. I can run the query in BQ anytime.
/** #OnlyCurrentDoc */
function RefreshDataSource() {
var spreadsheet = SpreadsheetApp.getActive();
spreadsheet.getRange('D1').activate();
spreadsheet.getRange('A1').activate();
spreadsheet.setActiveSheet(spreadsheet.getSheetByName('JIRA Security Review'), true);
SpreadsheetApp.enableAllDataSourcesExecution();
spreadsheet.getCurrentCell().getDataSourceTables()[0].refreshData();
};
This does not answer why it is happening, but as for a way to deal with it, you can try using forceRefreshData() in place of refreshData().
forceRefreshData() will refresh your data regardless of the error state (it refreshes data regardless of the current state).
Here's a link to documentation:
https://developers.google.com/apps-script/reference/spreadsheet/data-source-table#forceRefreshData()
Hope this is helpful!

Range.getValues() fails with valid cell (Service error: Spreadsheets)

I have a script that's been run daily for at least the past year which has suddenly started failing (seemingly at random).
It fails when trying to call getValue() on a range that contains one cell. The code looks like this:
var ss = SpreadsheetApp.getActiveSpreadsheet();
var configSheet = ss.getSheetByName("Config");
Logger.log(configSheet);
Logger.log(configSheet.getRange(3,2));
Logger.log(configSheet.getRange(3,2).getValue());
It always fails on the last step with "Service error: Spreadsheets" on that line (which isn't particularly helpful as far as error messages go). Any ideas?
Other notes:
One interesting thing is that it runs for quite a while before giving up and raising this error (longer than a single read of a single cell range should take).
Changing the range doesn't help.
Other spreadsheets that perform a similar step are running fine.
It started working again in a few hours with no intervention (same code as it used to be). I'll close this out - if anyone else has this problem, it's worth seeing if it eventually goes away (I believe it might be an internal problem).
what triggers the script to run?
You might try:
openById(id)
in lieu of getActiveSpreadsheet()

New Google Sheets custom functions sometimes display "Loading..." indefinitely

SPECIFIC FOR: "NEW" google sheets only.
This is a known issue as highlighted by google in the new sheets.
Issues: If you write complex* custom functions in google-apps-script for google sheets, you will occasionally run into cells which display a red error box around the cell with the text "Loading..."
Google has suggested:
If this occurs, try reloading the page or renaming the function and changing all references to the new name.
However for other developers experiencing this issue (and who are unable to escape the "loading..." error), I've written my findings in the answer below on how to get past this (with limitations) consistently.
*We're treating this question as the canonical answer for Google Sheet's indefinite "Error... Loading data" problem. It's not limited to complex or slow functions.
Important Tip: Create multiple copies of your entire spreadsheet as you experiment. I have had 3 google spreadsheets corrupted and rendered completely in-accessible (stuck in a refresh loop). This has happened when I was experimenting with custom functions so YOU HAVE BEEN WARNED!
You will want to try one or many of the following ways to fix this issue:
As suggested by google, try re-loading the spreadsheet or re-naming the function or changing the parameters in the cell to see if this fixes the issue.
Surround ALL your custom functions in a try-catch block. This will help detect code issues you may not have tested properly. Eg:
try{
//methods
}catch(ex){
return "Exception:"+ex;
}
Revert to the old sheets and test your functions and check for any other type of error such as an infinite loop or invalid data format. If the function does not work in the old sheets, it will not work in the new sheets and it will be more difficult to debug.
Ensure NONE of your parameters refer to, can expect to or will ever contain a number larger than 1 million (1000000). No idea why but using a number larger than a million as any parameter will cause your function to fail to execute. If you have to, ask the input to be reduced in size (maybe divide by 1000 or ask for M instead of mm).
Check for numeric or floating point issues where numbers may exceed a normal set of significant figures. The new sheets seems to be a little glitchy with numbers so if you are expecting very large or very complex numbers, your functions may not work.
Finally, if none of the above work, switch to the old google sheets and continue working.
If you find any other limitations or causes for functions to fail to execute, please write them below for me and other users who are heavy g-sheet users!
I also had the infinite loading issue with the following function.
// check if an item can be checked off
function checkedOff( need, have ) {
var retStr = "nope";
if( have >= need ){
retStr = "yep";
}
return retStr;
};
Turns out you shouldn't have a trailing ";". Removing the semicolon solved the problem.
// check if an item can be checked off
function checkedOff( need, have ) {
var retStr = "nope";
if( have >= need ){
retStr = "yep";
}
return retStr;
}
This runs as one would expect.
FWIW, I just ran into this and the culprit ended up being a getRange() call that pulled several thousand rows into an array. Periodically it would get hung on the "Loading..." message.
I worked around it by putting that range into the document cache. It's a little kludgy because the cache only stores strings, not arrays, but you can force it back into an array using .split(',') when you need to access the array.
(In my case it's a single array. There's probably a way to do it using a double array, either by sending each row or column into its own cache, or reading the cache value back N items at a time, each N becoming its own array.)
Here's the relevant bit from my code:
var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("mySheet"); //search the "mySheet" sheet
// is the big list already in the cache?
var cache = CacheService.getDocumentCache();
var cached = cache.get("columnValues");
if (cached != null) {
var columnValues = cached.split(','); // take the cached string and make it an array
} else { // it's not in the cache, so put it there
var column = 1; // the column with your index
var columnValues = sheet.getRange(2, column, sheet.getLastRow()).getValues(); // first row is header
cache.put("columnValues",columnValues,60000); // this forces the array into a string as if you used .join() on it
}
This is definitely a bug in Apps Script -- getRange() shouldn't hang without a timeout or error message. But at least there's a workaround. Here's the bug I opened against it, where I've also put the full code.gs from my sheet.
One cause: Permissions needing authorizing.
As far as {this problem, better phrased the cell result(s) of a custom function displaying the disgustingly-vague message ‘Loading... Error: loading data...’}, indeed in the case where all instances of the same/similar custom function call displaying this error, is that Google Sheets needs permissions to run the script (often additionally: meaning in the past it didn't need these), so instead of {acting appropriately: then prompting the user for these permissions else returning that error}, Sheets instead hangs with this disgustingly vague error.
Additional permissions can be needed from 1 or more:
Google App Scripts has since rewriting their permission structure --how this problem now just happened to me, per my internal note O80U3Z.
Your code or some library it uses made changes to require more access ...but in this case you have a much better chance of guessing the cause of this disgustingly-vague error, so hopefully won't be reading here.
To fix, I explicitly ran my GAS spreadsheet code by both: clicking one of my custom menu functions and, in the ‘script editor’, running one of my custom JS functions notably the ‘onOpen()’ since that is most comprehensive. The first promoted me for indeed new permissions, via popup ‘Authorization RequiredThe application "MM6ZBT(MM6Z83 script)" needs authorization to run.’, though onOpen() also did this in cases of GAS revising its permissions since we used that sheet. Then, as I was still getting this ‘Loading...’ error, I reloaded the web page (so the sheet), and, at least for these cases of this disgustingly vague error, it was gone and the computations worked fine :-)
TL;DR - Try duplicating the sheet tab and delete the old one
I struggled with this issue today, and tried some of the approaches mentioned. For various reasons, renaming the function wasn't possible for me.
It was very clear that if I called a my function like this in cell X25:
=myFunction("a", 1, "b", 2, "c", 3)
The cell would be stuck "Loading...", while just changing a parameter slightly (e.g. converting a number to a string) the cell would evaluate fine.
=myFunction("a", "" & 1, "b", 2, "c", 3)
Just copying the code into another cell (e.g. X24) and executing it there seemed to bypass the problem. As soon as I moved it back to the original parameters or cell, it got stuck "Loading..." again.
So I would assume it's some kind of caching of "Cell ID", function and parameters that go bonkers on Google's side.
My good-enough solution was to simply duplicate the Sheet tab, delete the old one, and finally rename the new one back to the original name. This solved the problem for me.
I also had the "loading data..." error but none of the fixes described here worked for me. It didn't seem to be caused by the issues described here. In my case, I narrowed it down to a specific floating point operation issue (it looks like a real bug in Google Sheets to me), and documented one possible work around at
Google Sheets / Apps "Loading data" error: is there a better workaround?
To summarize (at the request of commenter Steve), if a cell with
= myfunction(B10)
generated a "loading data" error, then for me it could be fixed by wrapping the argument in a "value()" function:
= myfunction(value(B10))
which converts the number in cell B10 (which seemed like a normal number but generated problems somehow) into a normal number that works fine.
I also had the problem that you explained. It seems that it can be caused in more than one way.
I ended up finding that my custom function was displaying that error because it relied on data from an =IMPORTRANGE() call, and that call was failing.
I eventually found that the =IMPORTRANGE() call was failing because I had forgotten to update the URL that it was importing from when I had uploaded a new version of that imported-from sheet. It seems that trying to IMPORTRANGE from a trashed file can cause the infinite "Loading..." error.
Update 2022
It looks like this bug is still happening. I tried ALL the solutions mentioned here but none worked.
What worked was to start with a blank slate. I recreated the file, copy-pasted my data, reapplied my preferred style and format, and lo-and-behold the sheet finally managed to pull the data using my custom functions.
This is a definitely a bug on Google's end - and it's all the more annoying because they removed the "Report a problem" button from the "Help" section.
Nevermind
The newer sheet has stopped working too. This is so annoying ..
The problem is that when a custom function formula cell starts showing Loading..., the custom function does not get called at all. The code in the script project thus does not come into play. Even the simplest custom functions sometimes suffer from the issue.
The problem usually goes away if you clear the formula cell and undo, or slightly edit the custom function's parameters to cause it to get re-evaluated. But that does not solve the issue. Google has been dragging their feet solving the underlying cause for many years.
To help the issue get Google's attention, star issue 233124478 in the issue tracker. Click the star icon ☆ in the top left-hand corner to vote for fixing the issue and get notified of status changes. Please do not post a "me too" or "+1" reply, but just click the star icon. Google prioritizes issues with the most stars.
Add-ons
I had two add-ons, and no function was loading.
I removed them, and all is well!
For me, renaming the custom function solved the problem. For now at least.
Just to add to Azmo 's answer...
I in fact removed all trailing semi-colons from the code:
// check if an item can be checked off
function checkedOff( need, have ) {
var retStr = "nope"
if( have >= need ){
retStr = "yep"
}
return retStr
}
And discovered, that when doing this over a large range you can also max out the acceptable number of calls to the API.
To get around it I added an IF THEN check around my custom script call.
So instead of:
=checkedOff(H10,H11)
Use something like this to check for a populated field before execution:
=if(H17<>"-",checkedOff(H10,H11),0)
My app script pulling data from my MSSQL database displayed just fine on GoogleSheets my laptop browser but then did not display on the Android GS app.
Per this thread it looks like there's a number of issues that could cause this, but #DestinyArchitect's answer above re: Permissions seemed like the simplest fix.
While testing my app script, Sharing was off for this GoogleSheet file.
Once I moved it to my team's folder where we have default Sharing switched on with a few team members, the MSSQL data showed right up on the GoogleSheet in my Android GS app.
Easy fix, this time...
In my case, the cell was stuck with a Loading... message due to "probably" a race condition between functions and formulas resolutions.
This is my custom function:
function ifBlank(value1, value2) {
return !!value1 ? value1 : value2;
}
This is the formula calling it:
=IFBLANK(VLOOKUP($A2,Overrides!$A$2:$E,5,FALSE),VLOOKUP($A2,'_resourceReq'!$A$2:$C,3))
Those VLOOKUP values could be pretty complex and could also take some time to resolve.
Solution: (in my case)
Wrapping the VLOOKUP() into TO_TEXT() or VALUE() for example.
So, changing the formula to =IFBLANK(TO_TEXT(VLOOKUP($A2,Overrides!$A$2:$E,5,FALSE)),TO_TEXT(VLOOKUP($A2,'_resourceReq'!$A$2:$C,3))) worked well.
If that doesn't work maybe try resolving the value from a function into a cell before using it as the argument of your custom function.
In my case, multiple cells using functions experienced this issue, but the simple answer was... wait.
In my case, I was scraping data via importXML functions across multiple rows and
columns. I was thrilled with the results, feeling on top of the world, then "Loading..." started showing its ugly face. For way too long. That's how I wound up here in troubleshooting mode, impatient and upset that Google was doing me wrong.
I tried many of the solutions here, only to find my "Loading..." antagonist acting unpredictably, popping up randomly like whacamoles, nothing to do with the code itself.
So. In my case, it was a matter of waiting it out (towards an hour for some rows, but I had so many cells fetching url data).
My layman's guess is that fetching data like this gets put in their bandwidth pipeline, lesser priority than typing a url into a search bar or other user requests.