Google App Script Service error: Spreadsheets - google-apps-script

Were not set up to use a proper SQL database or anything so were working with google sheets.
I've been trying to avoid importrange as I have a large amount of data constantly being updated and more rows added to Form responses every day. Importrange constantly fails with "importrange internal server error"
I found this fantastic code to copy from one source spreadsheet to another (as static text) so I can further manipulate the data :
function CopyTaskSource() {
var sss = SpreadsheetApp.openById('1OPnw_7vTCFkChy8VUKhAG5QRhcpKnDbmod0ZxjG----'); //replace with source ID
var ss = sss.getSheetByName('TASK Status'); //replace with source Sheet tab name
var range = ss.getRange('E:L'); //assign the range you want to copy
var data = range.getValues();
var tss = SpreadsheetApp.openById('1T3tqsHvKxuulYxDnaR3uf-wjVdXwLHBcUgI7tgN----'); //replace with destination ID
var ts = tss.getSheetByName('TaskSource'); //replace with destination Sheet tab name
ts.getRange(1, 1, data.length, data[0].length).setValues(data); //you will need to define the size of the copied data see getRange()
}
Now it copies about 15,000 rows of data, and I expect I will end up at 50,000 rows of data (and some other sheets go up to 27 columns).
I started getting this Service error: Spreadsheets line 9 (last line of the code).
Can someone please advise me a workaround to get bulk data transferred to multiple Google spreadsheet files?
importrange doesn't work well, and I have a few Google Forms that I need to combine the source responses to manipulate the data to output presentable spreadsheets.
Thank you

So I am working currently on a script that sends out emails when there is an issue, it then adds an array of values ,containing three values [type, ID, status], to an existing array ending with [[values1],[values2],etc...].
I have gotten the same error when I leave the third parameter of getRange as the array.length. I got it to work once yesterday by subtracting the array.length by 1 as I will show below. Maybe you can try this on line 9 and see if that fixes it?
It is important to mention that today after running the exact same script, it gave me an error stating that the range size was incorrect (due to the same subtraction that seemed to fix the service error)
I think that it may be broken on Google's side, but that is not something I can confirm.
This:
ts.getRange(1, 1, data.length, data[0].length).setValues(data);
Becomes This:
ts.getRange(1, 1, data.length - 1, data[0].length).setValues(data);
Hope that fixes it for you, I am truly stumped as to why it decides to work one day but not another...
I also added a waitLock to make sure it waits for other processes to be finished before trying to write it, but realize the data I write is much smaller, only 3 columns by 6-10 rows at a time. Here is the code for that, though this is to insert the data at the top of the sheet, not the bottom. (From Henrique G. Abreu, Original Post)
function insertRow(sheetI, rowData, optIndex) {
var lock = LockService.getScriptLock();
lock.waitLock(30000);
try {
var index = optIndex || 1;
sheetI.insertRowsBefore(index,rowData.length).getRange(index, 1, rowData.length, 3).setValues(rowData)
SpreadsheetApp.flush();
} finally {
lock.releaseLock();
}
}

Related

How Do I Prevent copyValuesToRange from Overwriting Destination with Blank or Null Data

For the most part, using copyValuesToRange works very well (99.9% of the time) in a function with the following statements:
let source = ss.getSheetByName("Update List");
let destination = ss.getSheetByName("Power Level");
source.getRange('TIER1DescImport').copyValuesToRange(destination, 12, 12, 6, 29); // 'PowerLevel'!L6:L29
The source is on a sheet "Update List" with a named data range "TIER1DescImport". The imported data is from a separate Google Sheet file using IMPORTRANGE.
ONCE, the destination range was entirely overwritten with empty or blank cells. How do I prevent this from happening again? I don't know why this occurred. The source range seemed fine. After closing the Google sheet and opening it 30 minutes later, everything was working properly again.
Is there a way of determining if the source range is okay, and if so, proceed with copyValuesToRange?
I'm a novice using Apps Script. I didn't know how to debug this issue. The only thing that worked was closing the files and coming back to it later. After that, it all seemed to be working again. I don't know why the destination was overwritten with blank or empty data but I'd like to prevent it from happening again.
You can test whether the source range has a meaningful amount of data with something like this:
const targetSheet = ss.getSheetByName('Power Level');
const sourceSheet = ss.getSheetByName('Update List');
const sourceRange = sourceSheet.getRange('TIER1DescImport');
if (sourceRange.getDisplayValues.join('').length > 10) {
sourceRange.copyValuesToRange(targetSheet, 12, 12, 6, 29);
// ...or: sourceRange.copyTo(targetSheet.getRange('L6'));
}

Query Import Range not updating when script runs automatically - error loading

I have a script to paste the raw data from a csv received by email. When the raw data is pasted on the sheet, I expected that another sheet with a query import range formula updates automatically with the new data.
I have a second script to read data from a pivot table that comes from the sheet with those formulas. However when it tries to read the data from the pivot table I get the error Exception: The number of rows in the range must be at least 1.. This happens because my variable numRows is equal to zero.
When I open the g-doc manually I see an error on the sheet with the formulas mentioned: error loading.
However, after really a few seconds that I open the gdoc, the range updates almost instantaneally without any problem, and If I manually run the script after this happening it runs without any problem.
How can I make sure that after updating the raw data I don't get the formulas stucked on error loading? I would like to run the script automatically and not manually. Any tip is more than welcome.
Notes:
I've tried already every type of recalculations but didn't work (on change, on change and every hour, on change and every minute)
The raw data has arround 2300 rows
The formula I am using is the following: =QUERY(IMPORTRANGE("1OpF8gcrV1Yj8bYP1j5PsHM4VRw2pKZOUmJf6VxGeFdY","raw_data!A2:G"), "select Col1,Col2,Col3,Col4,Col5,Col6,Col7 where Col2 is not null order by Col4 asc, Col1 asc, Col5 asc",0)
function sending_emails(){
var ss=SpreadsheetApp.openById("1OpF8gcrV1Yj8bYP1j5PsHM4VRw2pKZOUmJf6VxGeFdY");
var today = new Date();
if(today.getDay() != 6 && today.getDay() != 0){
//Sending emails to reps:
var data_sheet = ss.getSheetByName("Copy of sending_emails");
var aux = data_sheet.getRange("B3:B").getValues();
var startRow = 3; // First row of data to process
var numRows = aux.filter(String).length;
Logger.log('numRows' + numRows);
// Fetch the range of cells
var dataRange = data_sheet.getRange(startRow, 1, numRows, 5); //I get the error here because startRow = 3 and I get numRows = 0
// Fetch values for each row in the Range.
var data = dataRange.getValues();
for (var a in data) {
var row = data[a];
var message = row[3];
var emailAddress = row[0];
Logger.log('emailAddress'+ emailAddress);
MailApp.sendEmail({
to: emailAddress,
subject: 'Task Manager',
htmlBody: message,
cc: row[4]
});
}
}
}
The issue is likely with IMPORTRANGE
The class of functions IMPORTHTML, IMPORTRANGE etc have been the subject of many questions about auto updating - this approach generally seems to be quite flaky. I can't find it documented anywhere but I suspect that these functions stop calculating when they are closed. Or if a recalculation happens, for some reason they are not authorized because they are no longer linked to a user session.
That said, although I don't use this approach, I have tested it various times and it seems to work for me, though I know there are many people for whom it does not.
Some people have found that by removing all protections and making the sheet public removes errors, though in my experience its just best to remove formulae from the equation (no pun intended).
Suggested fix
In your chain of Mail > Apps Script > Sheet > FORMULA > Sheet.
Change it to Mail > Apps Script > Sheet > Apps Script > Sheet.
I don't have your source data to test with, but to implement your query in Apps Script would look something like this:
const ss = SpreadsheetApp.openById("YOUR ID");
const dataRange = ss.getSheetByName("Sheet1").getRange("A2:G");
const data = dataRange.getValues()
const filteredData = data.filter(row => row[1] !== "")
You could potentially sort the data with formulae once it has been imported with the script.
TLDR: Chaining IMPORTRANGE may work sometimes, but it doesn't seem very reliable. In my opinion, you are better off moving everything to Apps Script at this point.

Google Apps Script execution time issue

Im currently using Google Apps Script to implement a viewer of a supply chain database.
To synchronize the viewer with the current database (a google spreadsheet) I import the values and all the formatting it into a new sheet, this means the viewer basically is a copy of the current database.
However executing the script always takes something about 1 minute in time. I tried to find the issue with logging some debug messages at various positions in the code.
At first it seemed that the line with Viewer.setFrozenRows(1); (which is strange since I actually only freeze the first row) was the issue, however when commenting out this line the line afterwards (Viewer.setFrozenColumns(Database.getFrozenColumns());) seemed to be the issue.
Unfortuanetly I'm not able to share the database sheet with you, but maybe somebody can already spot the issue from the code.
Some additional Info: The database sheet has 1300 rows and 100 columns, and I added a picture of the log of the current code below.
function LoadViewer(view) {
Logger.log("LoadViewer Start");
if (view == null) {
view = 0;
}
var Database = SpreadsheetApp.openByUrl('[SHEET_URL].getSheetByName('Database');
var Viewer = SpreadsheetApp.getActiveSpreadsheet().getSheets()[0];
var numberOfColms = Database.getLastColumn();
var numberOfRows = Database.getLastRow();
var rules = Database.getConditionalFormatRules();
var headerRowHeight = Database.getRowHeight(1);
var dataRowHeight = Database.getRowHeight(2);
var Values = Database.getRange(1, 1, numberOfRows, numberOfColms).getValues();
Logger.log("Declarations Finished");
Viewer.getRange(1, 1,numberOfRows,numberOfColms).setValues(Values);
if(!Viewer.getRange(1, 1,numberOfRows,numberOfColms).getFilter())
Viewer.getRange(1, 1,numberOfRows,numberOfColms).createFilter();
Viewer.setConditionalFormatRules(rules);
Viewer.getRange(1, 1, 1, numberOfColms).setFontWeight('bold');
Viewer.autoResizeColumns(1, numberOfColms);
Viewer.setRowHeight(1, headerRowHeight);
Logger.log("1st Half of functions finished");
Viewer.setRowHeights(2, numberOfRows-1, dataRowHeight);
Logger.log("Freeze Rows");
//Viewer.setFrozenRows(1);
Logger.log("Freeze Columns");
Viewer.setFrozenColumns(Database.getFrozenColumns());
Logger.log("Loop Start");
for(var i = 1; i<=numberOfColms; i++){
Viewer.setColumnWidth(i, Database.getColumnWidth(i));
}
Logger.log("Loop End");
Viewer.getRange(1, 1,1,numberOfColms).setVerticalAlignment('middle').setWrap(true);
Logger.log("Load Viewer End");
}
Two optimization points I can see for your code:
Requests to the any external service including SpreadsheetApp make your code slow - see Best Practices.
Thus, making calls to a SpreadsheetApp method within a for loop will slow your code down.
You will be able to accelerate your code by replacing multiple setColumnWidth() requests within the loop by a single setColumnWidths(startColumn, numColumns, width) - avoiding iteration.
Log the number of columns and rows in your sheet.
A common problem is that the sheet contains a significant amount of empty rows and columns that increase your detected data range and consequently apply the subsequent calls to a bigger range than necessary.
If that you case - either delete the spare rows and columns manually, or use getNextDataCell() instead of getLastRow() or getLastColumn()

Get Range from GetRangeByName, activate range

My goal is to automate adding 10 rows to a Google Sheet used by a nonprofit organization's business and then replicate the formulas sequences needed in the newly created rows. I had all the code working to accomplish the task and prevent users from messing up the spreadsheet formulas when they manually insert rows. However, the code time out due to the number of rows in the spreadsheet with the looping use of getRange(). My new approach is to jump to a named cell as a starting point instead of the the really slow looping cell search.
I have created a name "EndData", read all the stuff I can find online, trialed and errored the syntax for hours to get the named_cell range into myrange and then activate the range on the worksheet...
Here is the current coding attempt (which leaves the cursor at the top of the column and an
"TypeError: Cannot find function getRangeByName in object Sheet. (line 170, file "macros")"
//Get EOD range, select, index up 3 rows to start row insertions
function getEOD() {
var ss = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
Logger.log(ss); //lOG is not helpful, says, "sheet", not SheetName
var MyRange = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet().getRangeByName("EndData");
Logger.log(MyRange); //lOG is not helpful, says, "Range", not RangeAddress
//Activate the named cell, moves with the spreadsheet
MyRange.activate();
};
Had a new idea after I asked for help, here's the working code that gets the job done:
//Get EOD range, select, index up 3 rows to start row insertions
function getEOD() {
var ss = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
//Logger.log(ss); //lOG is not helpful, says, "sheet", not SheetName
var rg = SpreadsheetApp.getActiveSpreadsheet().getRangeByName('EndData');
if (rg != null) {
Logger.log(rg.getNumColumns());
}
//Logger.log(rg); //lOG is not helpful, says, "Range", not RangeAddress
//Referenced the Named the EOD cell
//Future, Trying to create a debug status bar
//SpreadsheetApp.getActiveSpreadsheet().toast("looking at row" & i "now")
//Activate the named cell, which moves with spreadsheet growth, down
rg.activate();
//Uncomment for testing purposes, places a Yes on the row 4 columns over
//sheet.setActiveRange(sheet.getRange(0, 4)).setValue('Yes');
//turned off durning testing, writes in data range with this trial code
//Reposition from named cell to insert lines location
ss.getRange(sheet.getCurrentCell().getRow() -2, 1, 1,
sheet.getMaxColumns()).activate();
//Insert ten lines, copy and paste special formulas only
Insert_10_Lines()
//https://stackoverflow.com/questions/59772934/get-range-from-
getrangebyname-activate-range
};
My answer is I have tenacity, I try to limit the variables one at a time to prove how things really work, moved the code that did work in the test script file to a production script file and it didn't work for copy of what I had in test, went back to the test script and it didn't work either, after it had... There are variables in play that seem to be happening at the scripting, savings and running steps that change the interactive responses I am getting. Trial and error again through the production environment came up with a working combination by going to the simples code suggested and getting a combination that works. Here is the the code that is running in product to the main need of my question...
function InsertRows() {
var spreadsheet = SpreadsheetApp.getActive();
var sheet = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
//Get range from named cell
var range = SpreadsheetApp.getActiveSpreadsheet().getRangeByName("EOD");
if (range != null) {
Logger.log(range.getNumColumns());
}
//Select the cell as a starting point for code to follow
range.activate(); //start location cell is active and row is selected
Thanks for those that responded for the help! Looks like it is going to take a while to recognize the patterns and figure out what to do to get consistent results in a timely manner...
Try this:
function getEOD() {
var ss=SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
var MyRange=ss.getRangeByName("EndData");
MyRange.activate();
}

persistently sporadic out of memory error in Google Apps Scripts when copying data validations from a template

I have hundreds of Spreadsheets that were made from a template sheet. They all have the same number/name of sheets, rows, columns, etc...
I have added some data validations to the template. I want to copy the data validations from the template to each of the Spreadsheets. I have the code, and it works, but it throws a memory error.
It always throws the error -- the only thing that changes is how many destination Spreadsheets it has processed before it throws the error. Sometimes it'll process 4 Spreadsheets before it throws the error, sometimes 50, sometimes more/less. I cannot figure out why.
I trimmed my code down to a working sample. I can't share the source files but they are just normal Spreadsheets with 5 sheets/tabs and various data validations. If it matters, the data validations do use named ranges. For example: =REGEXMATCH(LOWER(google_drive_url) , "^https:\/\/drive\.google\.com\/drive\/folders\/[a-z0-9_-]{33}$").
I have commented the below code but here is a recap:
Get the template Spreadsheet and cache all of the data validations in it
Go through each destination Spreadsheet:
Alear all of the data validations
Apply the data validations from the template
In my real code I have an array of destination file IDs. For testing purposes I am just using one destination file and applying the data validations from the template multiple times.
function myFunction() {
var sourceFileID = "1rB7Z0C615Kn9ncLykVhVAcjmwkYb5GpYWpzcJRjfcD8";
var destinationFileID = "1SMrwTuknVa1Xky9NKgqwg16_JNSoHcFTZA6QxzDh7q4";
// get the source file
var sourceSpreadsheet = SpreadsheetApp.openById(sourceFileID);
var sourceDataValidationCache = {};
// go through each sheet and get a copy of the data validations
// cache them for quick access later
sourceSpreadsheet.getSheets().forEach(function(sourceSheet){
var sheetName = sourceSheet.getName();
// save all the data validations for this sheet
var thisSheetDataValidationCache = [];
// get the full sheet range
// start at first row, first column, and end at max rows and max columns
// get all the data validations in it
// go through each data validation row
sourceSheet.getRange(1, 1, sourceSheet.getMaxRows(), sourceSheet.getMaxColumns()).getDataValidations().forEach(function(row, rowIndex){
// go through each column
row.forEach(function(cell, columnIndex){
// we only need to save if there is a data validation
if(cell)
{
// save it
thisSheetDataValidationCache.push({
"row" : rowIndex + 1,
"column" : columnIndex + 1,
"dataValidation" : cell
});
}
});
});
// save to cache for this sheet
sourceDataValidationCache[sheetName] = thisSheetDataValidationCache;
});
// this is just an example
// so just update the data validations in the same destination numerous times to show the memory leak
for(var i = 0; i < 100; ++i)
{
// so we can see from the log how many were processed before it threw a memory error
Logger.log(i);
// get the destination
var destinationSpreadsheet = SpreadsheetApp.openById(destinationFileID);
// go through each sheet
destinationSpreadsheet.getSheets().forEach(function(destinationSheet){
var sheetName = destinationSheet.getName();
// get the full range and clear existing data validations
destinationSheet.getRange(1, 1, destinationSheet.getMaxRows(), destinationSheet.getMaxColumns()).clearDataValidations();
// go through the cached data validations for this sheet
sourceDataValidationCache[sheetName].forEach(function(dataValidationDetails){
// get the cell/column this data validation is for
// copy it, build it, and set it
destinationSheet.getRange(dataValidationDetails.row, dataValidationDetails.column).setDataValidation(dataValidationDetails.dataValidation.copy().build());
});
});
}
}
Is there something wrong with the code? Why would it throw an out of memory error? Is there anyway to catch/prevent it?
In order to get a better idea of what's failing I suggest you keep a counter of the iterations, to know how many are going through.
I also just noticed the line
sourceSheet.getRange(1, 1, sourceSheet.getMaxRows(), sourceSheet.getMaxColumns()).getDataValidations().forEach(function(row, rowIndex){
This is not a good idea, because getMaxRows() & getMaxColumns() will get the total number of rows in the sheet, not the ones with data, meaning, if your sheet is 100x100 and you only have data in the first 20x20 cells, you'll get a range that covers the entire 10,000 cells, and then calling a forEach means you go through every cell.
A better approach to this would be using getDataRange(), it will return a range that covers the entirety of your data (Documentation). With that you can use a much smaller range and considerably less cells to process.