I have a script that I run on multiple Spreadsheets... it auto hides rows that contain a certain value. Currently this script it setup to run daily around 3:00 am, to ensure no one is active in it while it processes. The issue is I am now running into is these sheets are getting too large to use my current script, which runs line by line. The script times out and doesn't finish. I'm guessing it still runs the script on all the lines that are already hidden.
Here is my current script, which is pretty basic:
function autoHide() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName("SHIPPING");
//get data from column
var data = sheet.getRange('AD:AD').getValues();
//iterate over all rows
for(var i=0; i< data.length; i++){
//compare first character, if greater than 0, then hide row
if(data[i][0] > 0){
sheet.hideRows(i+1);
}
}
}
I have tried searching for better options, and found where people were talking about using array filters, or running in batches, just different things that didn't seem to be explained enough for me to translate to what I was working on. I know running this line by line isn't the best way, especially with over a 1,000 rows and growing.
For a best case scenario, I would like to have a very efficient script that uses fractions of the processing my current script does. Otherwise, if there was just a way to run the script on the rows that are visible, that would be almost as good. Worst case scenario, if there is just a way to tell it to pick up where it left off when it gave a time out error... by placing some type of tag or something to know where to start back up.
I don't think linking a sheet is necessary, I just need to be able to hide any row that has a number greater than 0 in column AD, on a sheet called "SHIPPING".
While you could speed up the script itself (for example by batching consecutive rows that need to be hidden) that will also evenctually time out.
Instead, your script should remember in script properties the last row it processed, ao that if the script times out it will continue starting from that row.
You will also need to change the trigger times. Make it run every 10 minutes but only start processing if 1) its past 3am and b) the last row processed is not yet the very last row (which you reset to zero when finished).
this should handle huge sheets just fine. by 5am it would have run 12 times since 3am so it should be able to process 12 times more rows.
Note I chose 10min trigger so that a previous trigger (which could run for 6 minutes) wont ever overlap the next trigger.
Do make sure to set your timezone in the sheet and script file properties so that you use your timezones when checking if its past 3am already.
Related
I have a sheet that is pulling data from an API every minute. Each row contains information on an individual NFT that shows what has been done to the NFT for a given day. The data resets once per day at a different time for each row/NFT. The script I have archives each row just before it resets to log what happened with it during that 24-hour period.
The issue is, the script does not always pull the live time updates in my helper cells. I have been logging debugs for the last day and I can see that sometimes it pulls live data, sometimes it pulls the same data for up to 4 minutes. In other words, the countdown clock for a row may show 10 minutes on the first script run, then in the sheet it changes to 9 minutes, but the script will pull 10 minutes again. This may continue again at 8 and 7 minutes remaining, where the script still says 10 minutes. Finally, at 6 minutes, the script will get live data again and show 6 minutes. My script begins with a Spreadsheet flush as well.
I have only found this similar issue reported a year ago, but I did not see it resolved anywhere:
Script pulling old data
I agree with that poster's assumption, it seems Google Script is pulling from a cached version of the sheet. The randomness of this bug is causing major problems with my archive, and so I'm trying to figure out if it is possible to force a fresh read of the sheet, or else if there is known information about this delay so that I can account for this with proper error handling (for example, currently the helper cells continuously flag a row to be archived for 4 minutes, and remove the flag once it's been archived, but I'm not sure if that needs to be even longer)
For reference, this is the code that pulls cached data sometimes, in case there is something I can add to it to force a fresh read (perhaps adding and removing data from a cell just to make it access the sheet?):
function saveHabs() {
var dateStamp = Utilities.formatDate(new Date(), "GMT", "yyyy-MM-dd");
var ss = SpreadsheetApp.getActiveSpreadsheet();
var habSheet = ss.getSheetByName("Harvests");
SpreadsheetApp.flush;
var beVals = habSheet.getRange("be2:be").getDisplayValues();
var habs = beVals.filter(String).length;
var dataSheet = ss.getSheetByName("New Historical");
var aVals = dataSheet.getRange("a:a").getDisplayValues();
var firstRow = aVals.filter(String).length+1;
Logger.log(habs)
var archiveDetails;
if (habs>0){
archiveDetails = habSheet.getRange("be2:bi"+(habs+1)).getDisplayValues();
dataSheet.getRange(firstRow,1,habs,5).setValues(archiveDetails);
dataSheet.getRange(firstRow,6,habs,1).setValue(dateStamp);
dataSheet.getRange(firstRow,6,habs,1).setNumberFormat("#");
}
Logger.log(archiveDetails);
//debugging logs
var totalCount = habSheet.getRange("BM:BM").getDisplayValues().filter(String).length;
Logger.log(totalCount);
var logFlags = habSheet.getRange("BM2:BP"+totalCount).getDisplayValues();
console.log(habs+" habitats to archive, below are the flags and time remaining (ignore second column, that's only for unharvested habs)");
console.log(logFlags);
}
This is a simplified version of the sheet for a MRE: https://docs.google.com/spreadsheets/d/1od2G4i2YOJleQ6Ph0ICH-XC7L-SsINF8uBEivOsicTY/edit?usp=sharing
It is currently running every minute with logs that should show the same error - that is, the countdown in column BM will not always go down on each new script run, but rather the cached time remaining will be pulled by the script.
I'm not positive if this will fix the issue in all cases, but I came across this problem and solved it as you mentioned. I added a simple value (in my case dummyCell.setValue("Running")) to a blank cell on the source sheet. At the end of the script, the script deletes it. That has, so far, caused the script to always read the live sheet, nothing cached.
This seems to only happen when the source tab is only read from, never written to, so it would appear forcing the write action updates the sheet to the live version.
Hopefully this will solve this for you as well!
function All() {
var ss = SpreadsheetApp.getActive();
ss.getRange('Página1!A6').setFormula('=IF(ISEVEN(MINUTE(NOW())),"Ok","Error")')
ss.getRange('Página1!A6').copyTo(ss.getRange('Página1!A6'), SpreadsheetApp.CopyPasteType.PASTE_VALUES, false);
if (ss.getSheetByName('Página1').getRange("A6").getValues()[0][0]=="Ok"){
//History
ss.getRange('Página1!D1').setFormula('=IFERROR(FILTER({C:C;H1},{C:C;H1}<>""))');
ss.getRange('Página1!D1:D').copyTo(ss.getRange('Página1!C1'),
}
}
Google has triggers every 1 minute, 5 minutes, 10 minutes, 15 minutes and 30 minutes.
For this model, I use the 1 minute trigger!
To bypass this and be able to turn it on every 2 minutes instead of 1 minute (because it weighs a lot in the spreadsheet and occasionally creates errors), to deflect this I created this model where it analyzes if the minute of the current time is odd or even. If even, it activates the rest of the script, if odd it ends without doing anything else.
I would like to know if I could do this same thing, but instead of throwing the function into a cell, copy the value so that the formula NOW() doesn't keep updating all the time and so on ... same step but directly in the script, without moving the spreadsheet with unnecessary calls.
And if it would also be possible to do this to set the script to work every 3 minutes instead of 2 minutes as I managed to do.
Instead of using a sheet with a formula to determine if the minute is even or odd, you can use the Apps Script alternative.
I am using the %(Remainder) operator to get the reminder of a division by 2. If it's zero then the number is odd.
The equivalent for MINUTE(NOW()) is achieved with the Javascript Date new Date().getMinutes()
function myFunction() {
if (new Date().getMinutes()%2==0) { //If the minute is odd.
//Your code here
} //No need for else.
}
Instead of modifying your spreadsheet use the Properties Service to store the last time you script ran. Bear in mind that the Properties Service only stores strings, so you will have to convert the Date object to an string an viceversa.
Related
How can I modify a trigger so that it emails upon edit, but not so quickly?
I have a server side apps script that inserts a new row into a Google sheet, and then adds data to corresponding columns. Here is a short code snippet of what it does:
//check to see what is the last row of the sheet
var last_row = sheet.getLastRow();
//then, insert a new row after the last detected row
sheet.insertRowAfter(last_row);
//then, set current work row to last_row + 1, then do some adding data.
//Here, we add a current_timedate Date object that was created earlier.
var current_row = last_row + 1;
var current_timedate_cell = 'A' + current_row;
sheet.getRange(current_timedate_cell).setValue(current_timedate);
This is all pretty standard stuff. However, I noticed if I call the function that does all this pretty quickly from the front end interface (I have another client side javascript / html interface that calls the server side javascript function with onClick method on a button), it starts to skip rows. There also seems to be some bizarre concurrency issues. Some rows actually have partial data inserted, but only for checkboxes that I'm adding. None of the other stuff gets added. I can't tell if there is mixing of data between rows since all of the data is repeated. So, I can't tell if row 2's column H is now in row 3's column H, etc... To be fair, I am inserting one LARGE cell per row up to the 50k character limit. So, each row is probably close to 50k. That could be causing these issues
Can anyone provide some insights? I know Google Sheets is not supposed to be a relational database with ACID integrity and concurrent performance. But, I thought it would do a better job of handling 50kb inserts.
I've tried SpreadsheetApp.flush(). I'm somewhat aware of the rate limitations, and perhaps that might be it. I'm sometimes doing the inserts at a rate of about 3 inserts / second. I'll go double check quota rate limits. But, 3 / second doesn't seem incredibly high.
Expected results: all data requested to be inserted is inserted
Actual results: lots of skipped rows on faster insert rates
I'm doing all of this in a try / catch block, and so far I'm not getting any errors thrown. If there is an error thrown, I both log it in Logger and email it to myself via the MailApp.
Just going to add it as an answer instead of comment since it answered the question:
Use appendRow() instead of insertRowAfter() for this purpose.
I have a really very odd problem, which seems to have to do with the sequence i execute scripts. After investigating for hours, I cant explain it at all.
I have a google sheets script which gets emails from an email account and parses them according to given rules into a speadsheets.
I have many of those methods, all leveraging common classes like getEmails, etc.
Every single method of parsing works well and delivers the respected result. But when I run them in a big method one after the other it reports the error
"The coordinates of the range are outside the dimensions of the sheet."
after executing some of the methods correctly. The error occurs in the following line:
var resultArray = sheet.getRange(startrow, column, sheet.getLastRow(), 1).getValues();
and is based on the call
sheet.getLastRow()
(I can not even call this in the logger, it works for lets say 5 out of the 10 methods and then all the sudden i get the error)
Every of those methods parse a different email with a different pattern but does this only for new emails. Therefore I have to get the hashs of the old emails (thats the call) from the google sheets column 1 to work only on new email hashes. This process breaks somehow.
What is striking me is that i can execute any of the methods isolated without an error.
Any ideas?
As mentioned I have tried isolated and i have tried to change order or to run only 2 of the methods.. with the same result. I assume some variable is not set back properly... but i have no idea how that can lead to this error.
By the way: the code was working for the past few weeks without error (also for the combined method). The errors have started like a week ago without any code changes.
I came across the same issue.
This is a sample of the code that was causing the bug for me:
var priceSheet = ss.getSheetByName(priceSheetName);
var rangeToSort = priceSheet.getRange(2,1,priceSheet.getLastRow(),priceSheet.getLastColumn());
rangeToSort.sort(1);
ss.getSheetByName("my sheet").getRange(startingRow,pasteColumn,pasteHoldings.length,1).setValues(pasteHoldings);
The error The coordinates of the range are outside the dimensions of the sheet was raised on line 4 (similar to your scenario), but the issue was occurring when I was trying to sort a range that extended beyond the last row of the sheet i.e.
last row with contents = last row of the sheet
Sort row start range = 2
Number of rows = priceSheet.getLastRow() <-- this is impossible because the sort row start range is great than 1
The fix for me was to adjust the sort range down by the start row - 1
var priceSheet = ss.getSheetByName(priceSheetName);
var rangeToSort = priceSheet.getRange(2,1,priceSheet.getLastRow()-1,priceSheet.getLastColumn());
rangeToSort.sort(1);
ss.getSheetByName("my sheet").getRange(startingRow,pasteColumn,pasteHoldings.length,1).setValues(pasteHoldings);
This appears to be a bug in Google Sheets script: either the sort functionality, or at the very least the error handling is raising the error with reference to the wrong row.
My recommendation would be to check your use of .getLastRow() and see if it corresponds to a starting row greater than 1. Then adjust the .getLastRow() by starting row - 1
Would you happen to have your sort range starting on something other than row 1?
I was testing this out and I have my data to sort on A7:N100.
What I found was that the max row that I can have is the last row in the range minus the header rows that are not in the range. For example, I have the first 6 rows that aren't in the range. I have my last row in row 100. So my range is only working with A7:N94.
To solve, I ended up adding 6 blank rows to the bottom of my page and set sort range to what I wanted (A7:N100) and this worked.
I'm trying to add rows to a google spreadsheet through an adwords script which runs daily.
Some of the code so far is:
var report_iter = AdWordsApp.report(
'SELECT ' + columns_str +
'FROM ACCOUNT_PERFORMANCE_REPORT ' +
'DURING YESTERDAY', { apiVersion: 'v201302' }
).rows();
while(report_iter.hasNext()) {
var row = report_iter.next();
var row_array = [""]; // This is deliberate to include an empty cell in column A in the spreadsheet.
for(var i in columns) {
row_array.push(row[columns[i]]);
}
sheet.appendRow(row_array); // I think this line might be the problem
}
It is working properly, however, it has some unwanted outcomes as well. What I want to stop is every time I append a row to the bottom of the spreadsheet, it also appends 50 other rows of blank cells. Then, the next time I append a row, it appends it to the bottom of the spreadsheet, 50 rows after the one before... Is there any way to stop this. I had a look at using feed lists but I don't know how to do that or if you can with adwords scripts.
The only other thing I was thinking was to insert a new row to the bottom every time, find the row number, then insert data based on the row number.
What I had was working perfectly 3 days ago, but now it has just stopped.
Any suggestions or ideas?
1) you should not use appendRow. It would be much faster and efficient if you write all the rows at once using getRange setValues and using sheet.getLastRow to calculate the starting row.
2) if you do use appendRow, it will append to the last non-empty row, so it should work ok and shouldnt be writting past the blank rows that are autoinserted. However maybe you have something like an arrayformula or something else operating on an entire column that is writting in those empty rows at the bottom. Even if it writes blank values those rows wont be considered empty so the next appendRow will write below them.
The something is probably one of the calls setting the array to 30 rows, then only emptying the array, but not creating a new resized one.
But #1 is the way to go...