Automation script no longer functions, issue isolated to the spreadsheet itself - google-apps-script

I've been using Google Apps Script for about 2 months now to automate updating spreadsheets with information from a database. The scripts have been running this amount of time without any problems, until this week. Out of nowhere three of the scripts started failing. After diving into it I managed to fix two of the three spreadsheets, but one remains unresolved. One of the spreadsheets I fixed by manually clearing the sheet, despite that the script itself already does that. The other I had to create a new sheet for.
When running the script the log specifies that it is inserts the data as it should, but it clearly doesn't. It also displays an incorrect runtime (says between 15-20 seconds, but reality is 2-3 minutes). The exact error is
Execution failed: We're sorry, a server error occurred. Please wait a bit and try again.
The script itself triggers on a weekly basis. It duplicates the previous weeks sheet to maintain certain formatting and renames it, runs a query and inserts the resultset into the spreadsheet. Then it inserts a formula and that's it. The script still works. When using the script on a new spreadsheet it runs without a problem. It simply refuses to run on the existing spreadsheet, therefore I believe that that is where the problem lies.
I've tried manually clearing the old sheet (so before it gets duplicated), creating a new one, removing older sheets but all to no avail. The only solution I've encountered so far seems to be creating a new spreadsheet, but this isn't really an option. Some of these sheets are used by 40+ people on a daily basis. Having to replace and reshare a spreadsheet when this happens makes it a less reliable solution.
So my questions are why is this happening, and are there any solutions that will allow the use of the existing spreadsheet? Even if there is background data somewhere causing an error shouldn't that be removed when clearing a sheet.
Just in case I'll post the code snippit that writes to the spreadsheet, but I'm confident that this isn't the cause as it runs fine on a new spreadsheet and has run without a problem for nearly 2 months:
function buildWeek() {
var doc = SpreadsheetApp.getActiveSpreadsheet();
latestWeek();
var sheets = doc.getSheets();
var latestSheet = sheets[0];
Logger.log(latestSheet.getName());
var conn = Jdbc.getConnection("connection details");
var stmt = conn.createStatement();
var rs = stmt.executeQuery("query");
//Writing range start
var cell = latestSheet.getRange('A2');
var row = 0;
while (rs.next()) { // Inserts data from rows
var colcount = rs.getMetaData().getColumnCount();
for (var col = 0; col < colcount; col++) {
Logger.log(rs.getString(col + 1));
cell.offset(row, col).setValue(rs.getString(col + 1));
}
row++;
}

Related

Google Script pulling cached data

I have a sheet that is pulling data from an API every minute. Each row contains information on an individual NFT that shows what has been done to the NFT for a given day. The data resets once per day at a different time for each row/NFT. The script I have archives each row just before it resets to log what happened with it during that 24-hour period.
The issue is, the script does not always pull the live time updates in my helper cells. I have been logging debugs for the last day and I can see that sometimes it pulls live data, sometimes it pulls the same data for up to 4 minutes. In other words, the countdown clock for a row may show 10 minutes on the first script run, then in the sheet it changes to 9 minutes, but the script will pull 10 minutes again. This may continue again at 8 and 7 minutes remaining, where the script still says 10 minutes. Finally, at 6 minutes, the script will get live data again and show 6 minutes. My script begins with a Spreadsheet flush as well.
I have only found this similar issue reported a year ago, but I did not see it resolved anywhere:
Script pulling old data
I agree with that poster's assumption, it seems Google Script is pulling from a cached version of the sheet. The randomness of this bug is causing major problems with my archive, and so I'm trying to figure out if it is possible to force a fresh read of the sheet, or else if there is known information about this delay so that I can account for this with proper error handling (for example, currently the helper cells continuously flag a row to be archived for 4 minutes, and remove the flag once it's been archived, but I'm not sure if that needs to be even longer)
For reference, this is the code that pulls cached data sometimes, in case there is something I can add to it to force a fresh read (perhaps adding and removing data from a cell just to make it access the sheet?):
function saveHabs() {
var dateStamp = Utilities.formatDate(new Date(), "GMT", "yyyy-MM-dd");
var ss = SpreadsheetApp.getActiveSpreadsheet();
var habSheet = ss.getSheetByName("Harvests");
SpreadsheetApp.flush;
var beVals = habSheet.getRange("be2:be").getDisplayValues();
var habs = beVals.filter(String).length;
var dataSheet = ss.getSheetByName("New Historical");
var aVals = dataSheet.getRange("a:a").getDisplayValues();
var firstRow = aVals.filter(String).length+1;
Logger.log(habs)
var archiveDetails;
if (habs>0){
archiveDetails = habSheet.getRange("be2:bi"+(habs+1)).getDisplayValues();
dataSheet.getRange(firstRow,1,habs,5).setValues(archiveDetails);
dataSheet.getRange(firstRow,6,habs,1).setValue(dateStamp);
dataSheet.getRange(firstRow,6,habs,1).setNumberFormat("#");
}
Logger.log(archiveDetails);
//debugging logs
var totalCount = habSheet.getRange("BM:BM").getDisplayValues().filter(String).length;
Logger.log(totalCount);
var logFlags = habSheet.getRange("BM2:BP"+totalCount).getDisplayValues();
console.log(habs+" habitats to archive, below are the flags and time remaining (ignore second column, that's only for unharvested habs)");
console.log(logFlags);
}
This is a simplified version of the sheet for a MRE: https://docs.google.com/spreadsheets/d/1od2G4i2YOJleQ6Ph0ICH-XC7L-SsINF8uBEivOsicTY/edit?usp=sharing
It is currently running every minute with logs that should show the same error - that is, the countdown in column BM will not always go down on each new script run, but rather the cached time remaining will be pulled by the script.
I'm not positive if this will fix the issue in all cases, but I came across this problem and solved it as you mentioned. I added a simple value (in my case dummyCell.setValue("Running")) to a blank cell on the source sheet. At the end of the script, the script deletes it. That has, so far, caused the script to always read the live sheet, nothing cached.
This seems to only happen when the source tab is only read from, never written to, so it would appear forcing the write action updates the sheet to the live version.
Hopefully this will solve this for you as well!

Long processing time when inserting data in google sheets through google scripts

I have just built my first google script for a dashboard I am building in google sheets. Essentially, this dashboard has cells which act as filters for calculations in other cells. I am trying to build two buttons, one which returns the filter-cells values to a default value and one which copies the values for the filter from another sheet in the same google spreadsheet.
Problem
The script is taking between 2 and 120s to finish, for an action that done manually takes about 20s, essentially making the button "useless" when it takes more than 30s. Some times the script is incredibly fast, but I would need it to be that fast consistently for it to be worth having. I have tried optimizing the code to the best of my ability (which is not much) based on other threads I have found, but I haven't been able to optimize it yet and I was hoping to find a way to improve my code. Any suggestions?
Code
Default button
function Filters() {
var spreadsheet = SpreadsheetApp.getActiveSheet();
range_apply_1 = spreadsheet.getRange('C2:C8'),
range_apply_2 = spreadsheet.getRange('E4:E6');
var values_1 = SpreadsheetApp.getActiveSheet().getRange('\'Aux AM\'!A11:A17').getValues()
values_2 = SpreadsheetApp.getActiveSheet().getRange('\'Aux AM\'!A20:A22').getValues();
range_apply_1.setValues(values_1),
range_apply_2.setValues(values_2);
};
Copy values button
function test() {
var spreadsheet = SpreadsheetApp.getActiveSheet();
range_apply_1 = spreadsheet.getRange('C2:C8'),
range_apply_2 = spreadsheet.getRange('E4:E6');
var values_1 = SpreadsheetApp.getActiveSheet().getRange('\'AM Metrics - Orders\'!C2:C8').getValues()
values_2 = SpreadsheetApp.getActiveSheet().getRange('\'AM Metrics - Orders\'!E4:E6').getValues();
range_apply_1.setValues(values_1),
range_apply_2.setValues(values_2);
};
The basic to reduce the execution time in Google Apps Script is to to reduce the number of calls to Google Apps Script services. In your specific case you could reduce the calls by using a single getRangeList instead of using multiple getRange.
I think that the following modification should work:
function Filters() {
var sheet = SpreadsheetApp.getActiveSheet();
var name = sheet.getName();
var [range_apply_1, range_apply_2, range_1, range_2] = sheet
.getParent()
.getRangeList([
`${name}!C2:C8`,
`${name}!E4:E6`,
'Aux AM!A11:A17',
'Aux AM!A20:A22'
]).getRanges();
var values_1 = range_1.getValues();
var values_2 = range_2.getValues();
range_apply_1.setValues(values_1);
range_apply_2.setValues(values_2);
};
Regarding variation on the execution time that is normal. It's caused by things on your control and things out of it. One example of things that might be on your control is your PC load as the spreadsheet recalculation might be affected by the PC resources availability. An example of things out of your control is the Google servers responsiveness among other things.

Google Sheets App Script Edit Protected Cell or Object Error

I have a Google Sheets script that I've been using for the past 2+ years that grabs a chunk of data and transcribes it into a pair of tally sheets, then clears the cells that had been filled creating said chunk of data so the process can be started over. This is used for an inventory calculation system. The script has reliably worked up until roughly 3 weeks ago, but I now encounter an edit protection error when users with limited access to the sheet attempt to run the script. None of the editable cells/ranges reference in the script are locked to any user, but the sheet does have protection on cells I do not want anybody to make inadvertent changes to.
The script is:
function CopyErADD() {
var sss=SpreadsheetApp.getActiveSpreadsheet();
var sheet = sss.getSheetByName('ADDER'); //Entry Sheet
var rangeIn = sheet.getRange('B32:N45'); //Range to copy into "Incoming" sheet
var dataIn = rangeIn.getValues();
var ts = sss.getSheetByName('Incoming'); //Tally sheet in
ts.getRange(ts.getLastRow()+1, 1, dataIn.length, dataIn[0].length).setValues(dataIn);
var rangeOut = sheet.getRange('B48:O54'); //Range to copy into "Outgoing" sheet
var dataOut = rangeOut.getValues();
var tss = sss.getSheetByName('Outgoing'); //Tally sheet out
tss.getRange(tss.getLastRow()+1, 1, dataOut.length, dataOut[0].length).setValues(dataOut);
SpreadsheetApp.flush() // NEWLY ADDED PER METAMAN'S SUGGESTION
sheet.getRange('E2:E5').clearContent();
sheet.getRange('B7:B20').clearContent();
sheet.getRange('E7:H20').clearContent();
sheet.getRange('I7:I20').setValue('kg');
sheet.getRange('L7:L20').clearContent();
sheet.getRange('B24:B29').clearContent();
sheet.getRange('J24:J29').clearContent();
}
I also have an "erase only" script that runs the second second part of the script only (if data has been entered incorrectly) that executes perfectly fine. It is only when coupled with the copy/transcribe portion of the script that the protection error occurs.
function ErADD() {
var sss=SpreadsheetApp.getActiveSpreadsheet();
var sheet = sss.getSheetByName('ADDER');
sheet.getRange('E2:E5').clearContent();
sheet.getRange('B7:B20').clearContent();
sheet.getRange('E7:H20').clearContent();
sheet.getRange('I7:I20').setValue('kg');
sheet.getRange('L7:L20').clearContent();
sheet.getRange('B24:B29').clearContent();
sheet.getRange('J24:J29').clearContent();
}
Commenting out the sheet.getRange.... etc clearContent and setValue portion of the first script allows it to complete successfully, so I attempted to create a master function that calls the CopyErAdd script (sans clear portion) and then calls the ErADD script, and I encounter the same error. Both script can be run on their own successfully, but when combined the erase portion encounters the error.
Does anybody see any issues that I am missing, or did something occur over the past few weeks that I'm not aware of that could cause this protection error?
I appreciate any ideas anybody might have.
Edit - Thank you MetaMan for the tips for making the script more efficient.
As for the protection, "Incoming" and "Outgoing" copy destination sheets are completely open, no protection at all. "ADDER" sheet is protected except certain cells that are edited by user, and are referenced in the script.
E2:E5 (actually E2:I5 open, since the macro button sits on top of the F2:I5 range), B7:B20, E7:I20, L7:L20, B24:B29, J24:J29, All unprotected.
Adding the line
SpreadsheetApp.flush()
in between the copy section and clearing section of the code worked (as shown in the updated code snippet). I'm no longer encountering a range protection error.
I really appreciate the fix, but I'm boggled over why I've never needed that line until recently. I guess you don't need flush() until you do.
You could replace this:
sheet.getRange('I7').setValue('kg');
sheet.getRange('I8').setValue('kg');
sheet.getRange('I9').setValue('kg');
sheet.getRange('I10').setValue('kg');
sheet.getRange('I11').setValue('kg');
sheet.getRange('I12').setValue('kg');
sheet.getRange('I13').setValue('kg');
sheet.getRange('I14').setValue('kg');
sheet.getRange('I15').setValue('kg');
sheet.getRange('I16').setValue('kg');
sheet.getRange('I17').setValue('kg');
sheet.getRange('I18').setValue('kg');
sheet.getRange('I19').setValue('kg');
sheet.getRange('I20').setValue('kg');
with this:
sheet.getRange('I7:I20).setValue('kg');
also replace this:
var ss = sss.getSheetByName('ADDER'); //Entry sheet
var sheet = SpreadsheetApp.getActive().getSheetByName('ADDER');
with this:
var sheet = sss.getSheetByName('ADDER);
If you are doing anything immediate after this function with the data then you might wish to add SpreadsheetApp.flush()
I can't comment on the protection since none of that information was provided.

Google Apps script setValues() issue: timing out intermittently

I have a Google Apps script that has been running without issues for 4 years. However, since 3 weeks I have this problem: the script is running for a very long time and failing. This happens every 3 out of 10 runs. The error message is “Service Spreadsheets timed out while accessing spreadsheet with id [spreadsheet id here]”.
The actual script, which is elaborate (thousands of lines) and runs on hundreds of spreadsheets takes the data using fetchUrl() and populates the sheet with setValues(). This actual script used to work fine on spreadsheets with 10 sheets and could update the 180k cells in each sheet without a problem for the past 4 years. Now, I can't update even one sheet.
The script below replicates this issue: it copies 1300 rows by 140 columns from Sheet1 to Sheet2 using .getValues() and .setValues().The script starts to fail when the number of rows is increased above 800. When it runs fine the execution logs show it takes 8 seconds. When it fails the logs show run times of up to 900 seconds. During that time, you can’t access the spreadsheet for more than 10 minutes, if you try to load the spreadsheet in a different tab it doesn’t load at all.
I have opened an issue with Google Support, I got no timeline, but profuse apologies for the inconvenience. This happens on all domains I have tried the script on, not only mine. You need to try running the script 10 times to see the failures.
I would greatly appreciate if someone could suggest a workaround or provide some insight about this issue.
Here is the link to the spreadsheet replicating the issue: https://docs.google.com/spreadsheets/d/1jea15rtjv85YIZumABMfFKESb2_QmX0-7zC-KchWeDc/edit?usp=sharing
function myFunction() {
var row1 = 1;
var col1 = 1;
var row2 = 1300;
var col2 = 140;
console.log({numrows:row2, numcols:col2} );
var rng = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1").getRange(row1,col1,row2,col2);
var values_to_set = rng.getValues();
var rng2 = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet2").getRange(row1,col1,row2,col2);
rng2.setValues(values_to_set);
console.log('done');
}
According to this comparison of read/write methods, using advanced services to write is faster than setValues().
Using the following modified version of your original snippet worked for your sample spreadsheet:
function myFunction() {
var row1 = 1;
var col1 = 1;
var row2 = 1300;
var col2 = 140;
Logger.log({numrows:row2, numcols:col2} );
var rng = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1").getRange(row1,col1,row2,col2);
var values_to_set = rng.getValues();
var rng2 = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet2").getRange(row1,col1,row2,col2);
//rng2.setValues(values_to_set);
// Based on https://developers.google.com/apps-script/advanced/sheets
var request = {
'valueInputOption': 'USER_ENTERED',
'data': [
{
'range': 'Sheet2!' + rng2.getA1Notation(),
'majorDimension': 'ROWS',
'values': values_to_set
}
]
};
Sheets.Spreadsheets.Values.batchUpdate(request, SpreadsheetApp.getActiveSpreadsheet().getId());
Logger.log('done');
}
This issue is already reported to Google in Issuetracker. Add a star(on top left) and +1(on top right) to the issue to request Google developers to prioritize the issue and fix it.
In the mean time, Consider using Advanced Google services using google-sheets-api to do massive operations on a spreadsheet.
The problem seems to stem from set* methods. Another alternative in your specific case would be to use range.copyTo(instead of getValues() and setValues()), which works without issues (tested upto 15 times)
/**#OnlyCurrentDoc*/
function myFunction() {
var row1 = 1;
var col1 = 1;
var row2 = 1300;
var col2 = 140;
console.log({numrows:row2, numcols:col2} );
var rng = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1").getRange(row1,col1,row2,col2);
/*var values_to_set = rng.getValues();*/
var rng2 = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet2").getRange(row1,col1,row2,col2);
/*rng2.setValues(values_to_set);*/
/*Added*/rng.copyTo(rng2, SpreadsheetApp.CopyPasteType.PASTE_VALUES, false)
console.log('done');
}
function test_myFunction(i=15){
while(i--){
myFunction();
}
}
There seems to be a problem with google v8 engine. I recently experienced same problem. after disabling v8 engine its working fine now.
Go to tools->script editor.
In the script editor window, Click Run then disable v8 engine. See the attachment.
The issue is not the code itself, is the type of account that runs it.
I have two spreadsheets, owned by my personal , p............#gmail.com, account.
The script wrote for this 2 sheets to interact, is very simple, and consist on getting the range and values from sheet 1 (about 150000 cells) and setting the values on sheet 2.
Now, if i try to run the script using my personal account, i get the "service spreadsheets timed out while accessing document with id ..." error. The error shows at about 140 seconds.
At work , i have been issued a corporate paid account , that uses my work email,, p.....#c..mi.com. I shared sheets 1 and 2, with editing privileges with this account,.
When i run this same script, with this work paid account, the error never shows up, and the script finishes successfully after about 50 seconds.
I had this same problem with inserting a dozen records into a sheet with 150k rows. It would read the first file, insert and then fail on the second or third. I had several tabs in this sheet doing some very complicated nested queries and arrayformula vlookups and had set the sheet settings to update every minute and on change.
My solution- which fixed the problem, was to make a copy of my original sheet with the complex queries and replace the source data tab with an importRange (pulling the data from the raw data sheet). I then removed all the other tabs from my raw data sheet. Keeping the raw data sheet isolated from the dynamic calculations.
Short answer: remove tabs with complex queries, arrayformulas and vlookup to another sheet.

Time triggered script to paste value if condition is met (absence of #N/A)

Here's the idea: I want to automate the data entry from one cell that changes constantly to a range in another sheet, so the script fires at specific times based on a timed trigger, copies the value of the changing cell and fills a column in another sheet with the value of the changing cell. I should mention the reason I use a timed trigger is because I do not want to have the Spreadsheet opened at all times on a computer but rather have it happen automatically without me opening it.
Here's the Problem: Because the changing cells is based on Google/Yahoo Finance data, it can take time to load once the Spreadsheet is opened. Typically the cells will read "Loading" for a few seconds until the data is downloaded. During this time, the cells I'm looking to copy and paste read either "Loading" or "#N/A". As a result these are the "values" copied and pasted into my recorded data sheet. Sometimes the data isn't downloaded at all and the cells reads "#N/A" until I reload the Spreadsheet.
Here's my progress so far: So far my script is mostly functional, if the timed trigger is set to a short period (5 minutes) the results are perfect and all the data is recorded correctly, however I'm looking into this script running once or twice a day and when I increase the time to hours some data appears as "#N/A" or "Loading". Sometimes reloading the Spreadsheet solves this.
I get the impression that Google Sheets doesn't keep the data cached for more than a few minutes, so when the time between triggers is short the data is still available and everything works, but when the time is increased (for example to 1 hour) everything is re-downloaded, resulting in the "Loading".
Here's the code script I'm using to copy/paste the values from one sheet to another:
function PasteValue() {
var ss = SpreadsheetApp.openById("my spreadsheet ID");
var nextRow = ss.getSheetByName('My recorded data sheet').getLastRow()+1;
ss.getRange("Changing Data Sheet!F6").copyTo(ss.getRange("My recorded data sheet!A" + nextRow),{contentsOnly:true});
}
My idea: The solution I'm thinking about is to have a second function before the "PasteValue()" one which would reload the spreadsheet until no "#N/A" can be found in the "Changing Data Sheet". This would be done by timed trigger as well so:
at constant interval the spreadsheet is opened
the data sheet is searched for "#N/A"
if one #N/A is found the spreadsheet is reloaded until none can be
found
if no #N/As can be found, the values are copy/pasted in the recorded
data sheet
I found a script that reloads the spreadsheet until no more #N/As are found, however when I combine my script and the following one I still get missing data on my recorded data sheet.
The refresh script I found on Stack Overflow:
function refreshSheet(spreadsheet, sheet) {
var dataArrayRange = sheet.getRange(1,1,sheet.getLastRow(),sheet.getLastColumn());
var dataArray = dataArrayRange.getValues(); // necessary to refresh custom functions
var nanFound = true;
while(nanFound) {
for(var i = 0; i < dataArray.length; i++) {
if(dataArray[i].indexOf('#N/A') >= 0) {
nanFound = true;
dataArray = dataArrayRange.getValues();
break;
} // end if
else if(i == dataArray.length - 1) nanFound = false;
} // end for
} // end while
}
Am I not getting something?
Is there an easier way, or is this not possible?