Improving efficiency of a Google sheets script - google-apps-script

I'm fairly new to Google script and have been basically cobbling together code by finding it in Stack Overflow answers or the Google documentation. I have a working script, but it's very slow, so when I try to run it on a large file, it times out.
I know there must be inefficiencies in my code given how I went about writing it, but since I'm not super familiar with the language and learning as I go, I need help understanding how to find them (or help finding them, or both).
The code basically goes through a file that is a series of datasheets, each with a block of data that I need to copy into a "master sheet". Basically it's copy and paste, but from sometimes over a hundred sheets (which is why I'm not just doing this by hand).
Code in question (excessively commented, I know, but I'm not the only one using it):
function CleanAllTheThings(){
//Define all the variables first thing
var ss = SpreadsheetApp.getActive();
var allsheets = ss.getSheets();
//Get rid of blank rows and columns so Google doesn't get upset
for (var s in allsheets){
var sheet=allsheets[s]
var maxColumns = sheet.getMaxColumns()
var lastColumn = sheet.getLastColumn()
if (maxColumns-lastColumn != 0){
sheet.deleteColumns(lastColumn+1, maxColumns-lastColumn)
}
var maxRows = sheet.getMaxRows()
var lastRow = sheet.getLastRow()
if (maxRows-lastRow != 0){
sheet.deleteRows(lastRow+1, maxRows-lastRow)
}
}
//Continue defining variables
var create = ss.insertSheet("Merged",0);
var destsheet = ss.getSheetByName("Merged");//Name of sheet you’re putting everything into. If you don’t like the name, change it, but make sure you change it in the for loop too, or it’ll give you an error
//var StartRow = 4;//Row to start with, must be the same for every sheet; change it if the data start in a different row
//var StartCol = 3;//Column where the actual data start, ditto above
//var EndCol = 11;//End Column of all data, ditto above, but shouldn't ever change unless there are more columns--then just change this number accordingly
var prOne = ss.getSheetByName("Profile 1");
var rows_deleted = 0;
//Create a Merged sheet
for (var s in allsheets){
var sheet=allsheets[s]
if (sheet.getName() == "Merged"){//If the sheet is called “Merged”, don’t do anything to it!
}
else{//If the sheet is not called “Merged”, do stuff
var EndRow = sheet.getLastRow()
var copyRng = sheet.getRange(4,3,EndRow,11)//If the data start in a different place than row 4 column 3 and end in a different column than column 11, you need to change these numbers. Leave EndRow alone because it changes
var nextRow = 1 + destsheet.getLastRow()
destsheet.insertRowsAfter(destsheet.getMaxRows(), EndRow)//insert rows so Google doesn’t have fits. You end up with extras but you can either take them out with the add-on or just leave them be. I don’t think they’ll fill with ###### when you bring them back to Excel, if that’s what you decide to do
copyRng.copyTo(destsheet.getRange(nextRow,2))//Row and Column of where to paste the information; you’re pasting into column 2 because you’re about to fill column one with the sheet name the data came from
var LastRow = destsheet.getLastRow()//New last Row after pasting the data
var Sample = sheet.getRange(1,1)
sheet.getRange(4,2).copyTo(destsheet.getRange(nextRow,1,LastRow-nextRow+1,1))//Paste the name of the sheet in column 1 for every row copied
}
}
//Add a Header
prOne.getRange(3,3,3,11).copyTo(destsheet.getRange(1,2,1,10))
prOne.getRange(4,1).copyTo(destsheet.getRange(1,1,1,1))
//Get rid of the Solvent Peak rows
var range = destsheet.getDataRange();
var values = range.getValues();
for (var i = 0; i < values.length; i++){
for (var j = 0; j < values[i].length; j++){
var value = values[i][j];//row numbers are 1-based, not zero-based like this for-loop, so we add one AND every time we delete a row, all of the rows move down one, so we will subtract this count
var row = i + 1 - rows_deleted;//if the type is a number, we don't need to look
if (typeof value === 'string'){
var result = value.search("SOLVENT PEAK");//the .search() method returns the index of the substring, or -1 if it is not found we only care if it is found, so test for not -1
if (result !== -1){
destsheet.deleteRow(row)
rows_deleted++
}
}
}
}
//Delete blank rows and columns in Merged
var maxColumns = destsheet.getMaxColumns()
var lastColumn = destsheet.getLastColumn()
if (maxColumns-lastColumn != 0){
destsheet.deleteColumns(lastColumn+1, maxColumns-lastColumn)
}
var maxRows = destsheet.getMaxRows()
var lastRow = destsheet.getLastRow()
if (maxRows-lastRow != 0){
destsheet.deleteRows(lastRow+1, maxRows-lastRow)
}
}
For one thing, I know adding blank rows that I later delete is just a waste of time, but I'm not sure if it's such a big waste of time that fixing that alone would help this.
Thanks in advance for any help!

Related

Google Sheet speedup Loops

I have around 5000 rows. Now I want to move data for one sheet to another if their status is Delivered.
This is what I am doing:
function move(){
var activeSheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet2");
var sourceSheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1");
var lastSourceRow = sourceSheet.getLastRow();
var lastSourceCol = sourceSheet.getLastColumn();
var sourceRange = sourceSheet.getRange(1, 1, lastSourceRow, lastSourceCol);
var sourceData = sourceRange.getValues();
var activeRow = 0;
//Loop through every retrieved row from the Source
for (row = lastSourceRow; row > 1; row--) {
//IF Column B in this row has 'deal', then work on it.
if (sourceData[row-1][1] === 'Delivered') {
//then push that into the variables which holds all the new values to be returned
activeSheet.appendRow(sourceData[row-1]);
//delete current
sourceSheet.deleteRow(row);
}
Logger.log(row);
}
}
My problems:
Script is so slow it takes so much time to execute.
As per google policy script timeout in 5 mins.
So the loop has to be started again. So also index is set to null. So I can't check the record at the top if they are delivered or not.
Please can anyone provide me a better solution for this? I search for a faster loop but still no luck.
function move(){
const ash=SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet2");
const ssh=SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1");
var data=ssh.getDataRange().getValues();
var d=0;
var oA=[];
data.forEach(function(r,i){
if(r[1]=="Delivered") {
oA.push(r);
ssh.deleteRow(i+1-d++);
}
});
ash.getRange(ash.getLastRow()+1,1,oA.length,oA[0].length).setValues(oA);
}
The deletion of the rows always takes time. You could splice them out of the data array and clear the sheet and place them back in with a setvalues(). But if you have formulas this may mess some things up. I'll leave that up to you.

Fetch data from source sheet based on emails provided in the Emails tab in the target spreadsheet

I have a problem where I have two sheets. one sheet is the source spreadsheet and another is a target spreadsheet. The source spreadsheet has a source sheet has which is the master database and the target spreadsheet has the target where we want to fetch data from source sheet based on emails provided in the Emails tab in the target spreadsheet.
I want the following things to happen with a script and not with IMPORTRANGE or QUERY:
The target spreadsheet will have multiple copies so I want to connect the target spreadsheet with the source spreadsheet based on the source spreadsheet's id.
I want the email matches to be case insensitive so that the users of the target spreadsheet can type emails in any case.
The Emails can go up to 50 or let's say get the last row for that column.
It will be great if the script shows a pop up saying updated after it has fetched the data.
The source sheet might have data up to 15000 rows so I am thinking about speed too.
I have shared both of the spreadsheets with hyperlinks to their names. I am not really great at scripts so it will be helpful if you can leave comments in it wherever you feel like. I would truly appreciate your help.
Thanks in advance!
Script here:
function fetch() {
//get the sheets
var source_Ssheet = SpreadsheetApp.openById('19FkL3rsh5sxdujb6x00BUPvXEEhiXfAeURTeQi3YWzo');
var target_Ssheet = SpreadsheetApp.getActiveSpreadsheet();
//get the tabs
var email_sheet = target_Ssheet.getSheetByName("Emails");
var target_sheet = target_Ssheet.getSheetByName("Target Sheet");
var source_sheet = source_Ssheet.getSheetByName("Source Sheet");
//get ranges
var email_list = email_sheet.getRange("B2:B");
var target_sheet_range = target_sheet.getRange("A1:F100");
var source_sheet_range = source_sheet.getRange("A1:F100");
//get last rows
var last_email_name = email_list.getLastRow();
var last_target_sheet_range = target_sheet_range.getLastRow();
var last_source_sheet_range = source_sheet_range.getLastRow();
//start searching for emails
for (var i=3; i < last_email_name.length+1; i++)
{
for(varj=3; j< last_source_sheet_range.length+1; j++ )
{
if(source_sheet_range[j][3].getValue() == email_list[i][3].getValue())
{
//copy matches to target sheet
target_sheet.getRange((last_target_sheet_range + 1),1,1,10).setValues(master_sheet_range[j].getValues());
}
}
}
}
Several things
last_email_name and last_source_sheet_range are numbers - they do not have any length, this is why your first forloops are not working
You are missing a space in varj=3;
email_list[i][3].getValue() does not exist because email_list only includes B - that only one column. I assume you meant email_list[i][0].getValue()
ranges cannot be addressed with the indices [][], you need to retrieve the values first to have a 2D value range.
You email values in the different sheets do not follow the same case. Apps Script is case sensitive, to suee the == comparison you need to use the toLowerCase() method.
Also mind that defining getRange("B2:B") will include many empty rows that you don't need and will make your code very slow. Replace it through getRange("B2:B" + email_sheet.getLastRow());
Have a look here at the debugged code - keep in mind that there is still much room for improvement.
function fetch() {
//get the sheets
var source_Ssheet = SpreadsheetApp.openById('19FkL3rsh5sxdujb6x00BUPvXEEhiXfAeURTeQi3YWzo');
var target_Ssheet = SpreadsheetApp.getActiveSpreadsheet();
//get the tabs
var email_sheet = target_Ssheet.getSheetByName("Emails");
var target_sheet = target_Ssheet.getSheetByName("Target Sheet");
var source_sheet = source_Ssheet.getSheetByName("Source Sheet");
//get ranges
var email_list = email_sheet.getRange("B2:B" + email_sheet.getLastRow()).getValues();
var target_sheet_range = target_sheet.getRange("A1:F100").getValues();
var source_sheet_range = source_sheet.getRange("A1:F100").getValues();
var last_target_sheet_range = target_sheet.getLastRow();
//start searching for emails
for (var i=1; i < email_list.length; i++)
{
for(var j=1; j< source_sheet_range.length; j++ )
{
if(source_sheet_range[j][0].toLowerCase() == email_list[i][0].toLowerCase())
{
target_sheet.getRange((last_target_sheet_range + 1),1,1,6).setValues([source_sheet_range[j]]);
}
}
}
}

Moving rows between sheets based on cell edit + blank row found by column lookup not full row

I have been working on this for a few days and have turned the internet and these forums upside down, but can't find the answer anywhere. I've learned some scripting along the way, but have determined that what I need to do is way beyond my limits. Any help at all or guidance would be SUPER well received!!
I'm setting up an order sheet system for my and my wife's handmade 'business' (project). Essentially I need to move order 1 (columns a-w) from 'ready to ship" when "done" is entered into column X, onto the next blank row on "sent", discounting the formulas in columns w-z. So when its checking for a blank row, it should only check columns a-w.
This means that once the order info has moved across onto the new sheet, we have extra functionality added in those columns (I hooked it up to send an email apologising, if an order will be late - hope not, but better to be prepared, with our postal service :) )
Sample sheet here.
I originally learned how to write a 'move row if value added in column x - and learned about triggers and how to call the sheets etc.
Then, because column x, y and z have formulas, I either end up deleting them when I move the whole row - not cool. Or I end up (if i move only columns a- w) having the new (moved) row, showing up in row 9 million, because it sees the formulas as not blank rows. Ive tried editing a 'find row based on column' script, that seemed to work okay, but then i tried to put the two together and thats when it blew up.
Can anyone tell me where i'm going wrong here?
function lastRowForColumn(sheet, column){
// Get the last row with data for the whole sheet.
var ss = spreadSheetApp.getActiveSpreadSheet();// this gets you the
active spreadsheet in which you are working
var sheet = ss.getSheetByName('Sent');
var r = event.source.getActiveRange();
var data = sheet.getRange(1, "A", numRows).getValues();
var numRows = sheet.getLastRow();
// Iterate backwards and find first non empty cell
for(var i = data.length - 1 ; i >= 0 ; i--){
if (data[i][0] != null && data[i][0] != ""){
return i + 1;
var ss = spreadSheetApp.getActiveSpreadSheet();// this gets you the
active spreadsheet in which you are working
var sheet = ss.getSheetByName('Sent');
var r = event.source.getActiveRange();
if(s.getName() == "Ready To Send" && r.getColumn() == 25 &&
r.getValue() == "Sent") {
var row = r.getRow();
var numColumns = s.getLastColumn();
var targetSheet = ss.getSheetByName("Sent");
var lastRow = lastRowForColumn();
var target = targetSheet.getRange(targetSheet, i, 1);
s.getRange(row, 1, 1, 10).moveTo(target);
s.deleteRow(row);
}
}
}
}
Try this:
function onEdit(e){
var rg=e.range;
var sh=rg.getSheet();
var name=sh.getName();
var col=rg.columnStart;
if(name!='Sheet To Send'){return;}
if(col==24 && e.value=='done') {
var ss=e.source;
var sheet=ss.getSheetByName('Sent');
var row=rg.rowStart;
var srg=sh.getRange(row,1,1,23);
var data=srg.getValues();
var drg=sheet.getRange(sheet.getLastRow()+1,1,1,23);
drg.setValues(data);
sh.deleteRow(row);
}
}
I think what you want is to use getDisplayValues. That solves both your (formulas aren't blank problem (make sure your formulas display blank when the row is unused) and also the it isn't moving my numbers problem).
Try:
moveMe = [];
rowArray = s.getRange(row,1,1,10).getDisplayValues();
moveMe.push(rowArray);
targetSheet.appendRow(moveMe);
s.deleteRow(row);
You might have to wrestle the appendRow a bit but it is the cleanest. If it doesn't work as is try skipping the push step and doing
targetSheet.appendRow(rowArray);
BETTER (edit):
You need to change first your data row to:
var data = sheet.getRange(1, "A", numRows).getDisplayValues();
and then later you can simply do:
targetSheet.appendRow(data[i]);
s.deleteRow(row);
This function takes an edit event and checks if the values edited was changed to 'Sent'. It then copies that row into the sent sheet in the same spreadsheet, maintaining the values.
function onEdit(e){
if(e.value != 'Sent') return false; //If not changed to 'Sent' then stop.
var range = e.range; //get the range the edit occured in.
if(range.getColumn() != 25) return false;//see if it was column number 25, ie. Column Y
var sheet = range.getSheet();
if(sheet.getName() != 'Ready To Send') return false;
var editedRow = sheet.getRange(range.getRow(), 1, 1, sheet.getMaxColumns()); //Select the row.
var sent = sheet.getParent().getSheetByName('Sent');
sent.appendRow(editedRow.getValues()[0]); //Copy row to Sent sheet.
sheet.deleteRow(range.getRow()); //Deleted the original row
}

Improving Apps Script flexibility by using a column of sheet data instead of hard-coded IDs

Background: My coworkers originally each had a worksheet within the same Google Sheets file that makes a lot of calculations (and was getting unusable). Now, everyone has their own (known) Google Sheets file. To run the same calculations, we need to consolidate all that data into a master sheet (image ref below). We tried =importrange(...), but it's too heavy and breaks often (i.e., Loading... and other unfilled cells).
I've written some code to do this import, but right now its only manual: manually repeating the code and manually add the sheet IDs and changing the destrange.getRange(Cell range) each time. We have 80+ analysts, and fairly high turnover rates, so this would take an absurd amount of time. I'm new to Sheets and Apps Script, and know how to make the script use a cell as reference for a valid range or a valid ID, but I need something that can move a cell down and reference the new info.
Example:
Sheet 1 has a column of everyone Sheet ID
Script Pseudocode
get first row's id(Row 1), get sheet tab, get range, copies to active sheet's corresponding row(Row 1).
gets second row's id(Row 2), get sheet tab, get range, copies to active sheet's corresponding row (Row 2)
etc.
My script understanding is way to low to know how to process this. I have no idea what to read and learn to make it work properly.
function getdata() {
var confirm = Browser.msgBox('Preparing to draw data','Draw the data like your french girls?', Browser.Buttons.YES_NO);
if(confirm == 'yes'){
// I eventually want this to draw the ID from Column A:A, not hard-coded
var sourcess = SpreadsheetApp.openById('1B9sA5J-Jx0kBLuzP5vZ3LZcSw4CN9sS6A_mSbR9b26g');
var sourcesheet = sourcess.getSheetByName('Data Draw'); // source sheet name
var sourcerange = sourcesheet.getRange('E4:DU4'); // range
var sourcevalues = sourcerange.getValues();
var ss = SpreadsheetApp.getActiveSpreadsheet(); //
var destsheet = ss.getSheetByName('Master Totals'); //
// This range needs to somehow move one down after each time it pastes a row in.
var destrange = destsheet.getRange('E4:DU4');
destrange.setValues(sourcevalues); // Data into destsheet
}
}
Any suggestions are greatly appreciated!
Thanks to tehhowch for pointing me in the right direction!
function getdata() {
var ss = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
var destsheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Master Totals');
var confirm = Browser.msgBox('Drawing Data','Would you like to update the sheet? It may take 2 to 5 minutes.', Browser.Buttons.YES_NO);
if(confirm =='yes'){
var lr = ss.getLastRow();
for (var i = 4; i<=lr; i++) {
var currentID = ss.getRange(i, 1).getValue();
var sourcess = SpreadsheetApp.openByUrl(currentID);
var sourcesheet = sourcess.getSheetByName('Data Draw');
var sourcerange = sourcesheet.getRange('E4:DU4');
var sourcevalues = sourcerange.getValues();
var destrange = destsheet.getRange('E' +i+':'+ 'DU'+ i);
destrange.setValues(sourcevalues);
I just had to learn how to use a variable loop.
Edit: thanks also to Phil for making my question more presentable!
Now that you've figured out one way to do it, I'll offer an alternative that uses batch methods (i.e. is much more time- and resource-efficient):
function getData() {
var wb = SpreadsheetApp.getActive();
var ss = wb.getActiveSheet();
var dest = wb.getSheetByName('Master Totals');
if (!dest || "yes" !== Browser.msgBox('Drawing Data', 'Would you like to update the sheet? It may take 2 to 5 minutes.', Browser.Buttons.YES_NO))
return;
// Batch-read the first column into an array of arrays of values.
var ssids = ss.getSheetValues(4, 1, ss.getLastRow() - 4, 1);
var output = [];
for (var row = 0; row < ssids.length; ++row) {
var targetID = ssids[row][0];
// Open the remote sheet (consider using try-catch
// and adding error handling).
var remote = SpreadsheetApp.openById(targetID);
var source = remote.getSheetByName("Data Draw");
var toImport = source.getRange("E4:DU4").getValues();
// Add this 2D array to the end of our 2D output.
output = [].concat(output, toImport);
}
// Write collected data, if any, anchored from E4.
if(output.length > 0 && output[0].length > 0)
dest.getRange(4, 5, output.length, output[0].length).setValues(output);
}
Each call to getRange and setValues adds measurable time to the execution time - i.e. on the order of hundreds of milliseconds. Minimizing use of the Google interface classes and sticking to JavaScript wherever possible will dramatically improve your scripts' responsiveness.

Google sheet script, times out. Need a new way or flip it upside down

noob here, so the script works great if there are less than 800 rows on a sheet, however this time I have a sheet of almost 1500 rows and the script times out.
Basically it is a quick way to get a quote. (quick here means 5-6mins, not an issue) It hides columns with calculations, hides columns with sensitive information and rows where there was no value in column H.
What I want to know is, if I can do the same with a different code, or if someone knows how to make getRange().getValue(); start at the bottom of the sheet, then I could have two scripts starting one after the other to finish the sheet and produce a printable quote.
Any help is greatly appreciated.
Many Thanks
here is the script:
function Quote()
{
var ss = SpreadsheetApp.getActiveSpreadsheet();
var s = ss.getSheetByName("Quote"); `
var datarange = s.hideColumns(6);
var datarange = s.hideColumns(9);
var datarange = s.hideColumns(10);
var datarange = s.hideColumns(12);
var datarange = s.hideColumns(13);
var datarange = s.hideColumns(14);
var lastRow = s.getLastRow();
for( i=1 ; i<=lastRow ; i++) {
var status = s.getRange("H"+i).getValue();
if (status == "") {
s.hideRows(i);
}
}
}
Your problem is row:
s.getRange("H"+i).getValue()
This code takes data from spreadsheet, this is very slow process when you use it inside loop. You may use this conctruction:
var data = s.getDataRange().getValues();
for (var i=0; i < data.length; i++) {
var status = data[i][7]; // takes column H
// other code goes here...
}
This way you read data from the spreadsheet only once using getDataRange(), and convert it into array with getValues(). Then loop should work way more faster.
When hiding rows, remember to add 1 because array starts from 0, heres code for hiding rows inside loop:
if (status == "") {
s.hideRows(i + 1); // adding 1
}