I have a sheet that is pulling data from an API every minute. Each row contains information on an individual NFT that shows what has been done to the NFT for a given day. The data resets once per day at a different time for each row/NFT. The script I have archives each row just before it resets to log what happened with it during that 24-hour period.
The issue is, the script does not always pull the live time updates in my helper cells. I have been logging debugs for the last day and I can see that sometimes it pulls live data, sometimes it pulls the same data for up to 4 minutes. In other words, the countdown clock for a row may show 10 minutes on the first script run, then in the sheet it changes to 9 minutes, but the script will pull 10 minutes again. This may continue again at 8 and 7 minutes remaining, where the script still says 10 minutes. Finally, at 6 minutes, the script will get live data again and show 6 minutes. My script begins with a Spreadsheet flush as well.
I have only found this similar issue reported a year ago, but I did not see it resolved anywhere:
Script pulling old data
I agree with that poster's assumption, it seems Google Script is pulling from a cached version of the sheet. The randomness of this bug is causing major problems with my archive, and so I'm trying to figure out if it is possible to force a fresh read of the sheet, or else if there is known information about this delay so that I can account for this with proper error handling (for example, currently the helper cells continuously flag a row to be archived for 4 minutes, and remove the flag once it's been archived, but I'm not sure if that needs to be even longer)
For reference, this is the code that pulls cached data sometimes, in case there is something I can add to it to force a fresh read (perhaps adding and removing data from a cell just to make it access the sheet?):
function saveHabs() {
var dateStamp = Utilities.formatDate(new Date(), "GMT", "yyyy-MM-dd");
var ss = SpreadsheetApp.getActiveSpreadsheet();
var habSheet = ss.getSheetByName("Harvests");
SpreadsheetApp.flush;
var beVals = habSheet.getRange("be2:be").getDisplayValues();
var habs = beVals.filter(String).length;
var dataSheet = ss.getSheetByName("New Historical");
var aVals = dataSheet.getRange("a:a").getDisplayValues();
var firstRow = aVals.filter(String).length+1;
Logger.log(habs)
var archiveDetails;
if (habs>0){
archiveDetails = habSheet.getRange("be2:bi"+(habs+1)).getDisplayValues();
dataSheet.getRange(firstRow,1,habs,5).setValues(archiveDetails);
dataSheet.getRange(firstRow,6,habs,1).setValue(dateStamp);
dataSheet.getRange(firstRow,6,habs,1).setNumberFormat("#");
}
Logger.log(archiveDetails);
//debugging logs
var totalCount = habSheet.getRange("BM:BM").getDisplayValues().filter(String).length;
Logger.log(totalCount);
var logFlags = habSheet.getRange("BM2:BP"+totalCount).getDisplayValues();
console.log(habs+" habitats to archive, below are the flags and time remaining (ignore second column, that's only for unharvested habs)");
console.log(logFlags);
}
This is a simplified version of the sheet for a MRE: https://docs.google.com/spreadsheets/d/1od2G4i2YOJleQ6Ph0ICH-XC7L-SsINF8uBEivOsicTY/edit?usp=sharing
It is currently running every minute with logs that should show the same error - that is, the countdown in column BM will not always go down on each new script run, but rather the cached time remaining will be pulled by the script.
I'm not positive if this will fix the issue in all cases, but I came across this problem and solved it as you mentioned. I added a simple value (in my case dummyCell.setValue("Running")) to a blank cell on the source sheet. At the end of the script, the script deletes it. That has, so far, caused the script to always read the live sheet, nothing cached.
This seems to only happen when the source tab is only read from, never written to, so it would appear forcing the write action updates the sheet to the live version.
Hopefully this will solve this for you as well!
Related
I am importing JSON data to Google Sheets using this solution (https://blog.fastfedora.com/projects/import-json).
It is vital that this data is accurate all the time, however, I have noticed that the data provided through this function lags behind the actual API feeds.
The issue can be sorted if I delete and Ctrl+Z, but obviously I am not available 24/7 to constantly do that :).
A solution I have is that in the cells with the IMPORTJSON function, I have placed the following before it: IF(A1=1,"",IMPORTJSON....
So if 1 is entered in A1, everything is deleted, and once the 1 is deleted, the feeds refresh with the correct data.
Again the issue is that I have to manually enter this 1. I would like to create a method of this one is entered automatically. Like every minute or five minutes.
How do I go about creating this time triggering cell?
function updateCell() {
var range = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Name of Sheet").getRange("A1");
range.clear({contentsOnly: true});
range.setValue(1);
}
And set that function to trigger on a timer every x minutes.
See the documentation for further information if you need finer revisions.
I've been using Google Apps Script for about 2 months now to automate updating spreadsheets with information from a database. The scripts have been running this amount of time without any problems, until this week. Out of nowhere three of the scripts started failing. After diving into it I managed to fix two of the three spreadsheets, but one remains unresolved. One of the spreadsheets I fixed by manually clearing the sheet, despite that the script itself already does that. The other I had to create a new sheet for.
When running the script the log specifies that it is inserts the data as it should, but it clearly doesn't. It also displays an incorrect runtime (says between 15-20 seconds, but reality is 2-3 minutes). The exact error is
Execution failed: We're sorry, a server error occurred. Please wait a bit and try again.
The script itself triggers on a weekly basis. It duplicates the previous weeks sheet to maintain certain formatting and renames it, runs a query and inserts the resultset into the spreadsheet. Then it inserts a formula and that's it. The script still works. When using the script on a new spreadsheet it runs without a problem. It simply refuses to run on the existing spreadsheet, therefore I believe that that is where the problem lies.
I've tried manually clearing the old sheet (so before it gets duplicated), creating a new one, removing older sheets but all to no avail. The only solution I've encountered so far seems to be creating a new spreadsheet, but this isn't really an option. Some of these sheets are used by 40+ people on a daily basis. Having to replace and reshare a spreadsheet when this happens makes it a less reliable solution.
So my questions are why is this happening, and are there any solutions that will allow the use of the existing spreadsheet? Even if there is background data somewhere causing an error shouldn't that be removed when clearing a sheet.
Just in case I'll post the code snippit that writes to the spreadsheet, but I'm confident that this isn't the cause as it runs fine on a new spreadsheet and has run without a problem for nearly 2 months:
function buildWeek() {
var doc = SpreadsheetApp.getActiveSpreadsheet();
latestWeek();
var sheets = doc.getSheets();
var latestSheet = sheets[0];
Logger.log(latestSheet.getName());
var conn = Jdbc.getConnection("connection details");
var stmt = conn.createStatement();
var rs = stmt.executeQuery("query");
//Writing range start
var cell = latestSheet.getRange('A2');
var row = 0;
while (rs.next()) { // Inserts data from rows
var colcount = rs.getMetaData().getColumnCount();
for (var col = 0; col < colcount; col++) {
Logger.log(rs.getString(col + 1));
cell.offset(row, col).setValue(rs.getString(col + 1));
}
row++;
}
I have a script that I run on multiple Spreadsheets... it auto hides rows that contain a certain value. Currently this script it setup to run daily around 3:00 am, to ensure no one is active in it while it processes. The issue is I am now running into is these sheets are getting too large to use my current script, which runs line by line. The script times out and doesn't finish. I'm guessing it still runs the script on all the lines that are already hidden.
Here is my current script, which is pretty basic:
function autoHide() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName("SHIPPING");
//get data from column
var data = sheet.getRange('AD:AD').getValues();
//iterate over all rows
for(var i=0; i< data.length; i++){
//compare first character, if greater than 0, then hide row
if(data[i][0] > 0){
sheet.hideRows(i+1);
}
}
}
I have tried searching for better options, and found where people were talking about using array filters, or running in batches, just different things that didn't seem to be explained enough for me to translate to what I was working on. I know running this line by line isn't the best way, especially with over a 1,000 rows and growing.
For a best case scenario, I would like to have a very efficient script that uses fractions of the processing my current script does. Otherwise, if there was just a way to run the script on the rows that are visible, that would be almost as good. Worst case scenario, if there is just a way to tell it to pick up where it left off when it gave a time out error... by placing some type of tag or something to know where to start back up.
I don't think linking a sheet is necessary, I just need to be able to hide any row that has a number greater than 0 in column AD, on a sheet called "SHIPPING".
While you could speed up the script itself (for example by batching consecutive rows that need to be hidden) that will also evenctually time out.
Instead, your script should remember in script properties the last row it processed, ao that if the script times out it will continue starting from that row.
You will also need to change the trigger times. Make it run every 10 minutes but only start processing if 1) its past 3am and b) the last row processed is not yet the very last row (which you reset to zero when finished).
this should handle huge sheets just fine. by 5am it would have run 12 times since 3am so it should be able to process 12 times more rows.
Note I chose 10min trigger so that a previous trigger (which could run for 6 minutes) wont ever overlap the next trigger.
Do make sure to set your timezone in the sheet and script file properties so that you use your timezones when checking if its past 3am already.
My google spreadsheet is using GOOGLEFINANCE('symbol','price) function to retrieve stock prices of my portfolio. Unfortunately, I have to refresh manually now. How can I make the spreadsheet refresh itself every 1 minute?
Thank you for your help.
If you're on the New Google Sheets, this is all you need to do, according to the docs:
change your recalculation setting to "On change and every minute" in your spreadsheet at File > Spreadsheet settings.
This will make the entire sheet update itself every minute, on the server side, regardless of whether you have the spreadsheet up in your browser or not.
If you're on the old Google Sheets, you'll want to add a cell with this formula to achieve the same functionality:
=GoogleClock()
EDIT to include old and new Google Sheets and change to =GoogleClock().
If you are only looking for a refresh rate for the GOOGLEFINANCE function, keep in mind that data delays can be up to 20 minutes (per Google Finance Disclaimer).
Single-symbol refresh rate (using GoogleClock)
Here is a modified version of the refresh action, taking the data delay into consideration, to save on unproductive refresh cycles.
=GoogleClock(GOOGLEFINANCE(symbol,"datadelay"))
For example, with:
SYMBOL: GOOG
DATA DELAY: 15 (minutes)
then
=GoogleClock(GOOGLEFINANCE("GOOG","datadelay"))
Results in a dynamic data-based refresh rate of:
=GoogleClock(15)
Multi-symbol refresh rate (using GoogleClock)
If your sheet contains a number of rows of symbols, you could add a datadelay column for each symbol and use the lowest value, for example:
=GoogleClock(MIN(dataDelayValuesNamedRange))
Where dataDelayValuesNamedRange is the absolute reference or named reference of the range of cells that contain the data delay values for each symbol (assuming these values are different).
Without GoogleClock()
The GoogleClock() function was removed in 2014 and replaced with settings setup for refreshing sheets. At present, I have confirmed that replacement settings is only on available in Sheets from when accessed from a desktop browser, not the mobile app (I'm using Google's mobile Sheets app updated 2016-03-14).
(This part of the answer is based on, and portions copied from, Google Docs Help)
To change how often "some" Google Sheets functions update:
Open a spreadsheet. Click File > Spreadsheet settings.
In the RECALCULATION section, choose a setting from the drop-down menu.
Setting options are:
On change
On change and every minute
On change and every hour
Click SAVE SETTINGS.
NOTE External data functions recalculate at the following intervals:
ImportRange: 30 minutes
ImportHtml, ImportFeed, ImportData, ImportXml: 1 hour
GoogleFinance: 2 minutes
The references in earlier sections to the display and use of the datadelay attribute still apply, as well as the concepts for more efficient coding of sheets.
On a positive note, the new refresh option continues to be refreshed by Google servers regardless of whether you have the sheet loaded or not. That's a positive for shared sheets for sure; even more so for Google Apps Scripts (GAS), where GAS is used in workflow code or referenced data is used as a trigger for an event.
[*] in my understanding so far (I am currently testing this)
GOOGLEFINANCE can have a 20 minutes delay, so refreshing every minute would not really help.
Instead of GOOGLEFINANCE you can use different source. I'm using RealTime stock prices(I tried a couple but this is the easiest by-far to implement. They have API that return JSON { Name: CurrentPrice }
Here's a little script you can use in Google Sheets(Tools->Script Editor)
function GetStocksPrice() {
var url = 'https://financialmodelingprep.com/api/v3/stock/real-time-
price/AVP,BAC,CHK,CY,GE,GPRO,HIMX,IMGN,MFG,NIO,NMR,SSSS,UCTT,UMC,ZNGA';
var response = UrlFetchApp.fetch(url);
// convert json string to json object
var jsonSignal = JSON.parse(response);
// define an array of all the object keys
var headerRow = Object.keys(jsonSignal);
// define an array of all the object values
var values = headerRow.map(function(key){ return jsonSignal[key]});
var data = values[0];
// get sheet by ID -
// you can get the sheet unqiue ID from the your current sheet url
var jsonSheet = SpreadsheetApp.openById("Your Sheet UniqueID");
//var name = jsonSheet.getName();
var sheet = jsonSheet.getSheetByName('Sheet1');
// the column to put the data in -> Y
var letter = "F";
// start from line
var index = 4;
data.forEach(function( row, index2 ) {
var keys = Object.keys(row);
var value2 = row[keys[1]];
// set value loction
var cellXY = letter + index;
sheet.getRange(cellXY).setValue(value2);
index = index + 1;
});
}
Now you need to add a trigger that will execute every minute.
Go to Project Triggers -> click on the Watch icon next to the Save icon
Add Trigger
In -> Choose which function to run -> GetStocksPrice
In -> Select event source -> Time-driven
In -> Select type of time based trigger -> Minutes timer
In -> Select minute interval -> Every minute
And your set :)
I had a similar problem with crypto updates. A kludgy hack that gets around this is to include a '+ now() - now()' stunt at the end of the cell formula, with the setting as above to recalculate every minute. This worked for my price updates, but, definitely an ugly hack.
use now() in any cell. then use that cell as a "dummy" parameter in a function.
when now() changes every minute the formula recalculates.
example:
someFunction(a1,b1,c1) * (cell with now() / cell with now())
I have a few queries on the sheet where I extract info to a database.
The problem I have is that the code implemented was fine till some sheets won't display the query showing the: "warning: one or more of these results' entries may not be displayed. Select ctrl+Shift+E to show them.
I tried clearing the values and pasting them back to no avail.
what else can I do?
IMHO I don't think that there is a canonical answer to this question - there has always been a loud clamour from users around certain topics which merely dissipate into an eerie silence of the Google docs engineering team (this issue seems to be shrouded in the same eternal mystery as IMPORTRANGE frequently failing or SPLIT not creating a perfect matrix).
I have been bitten by this myself several times so have some experience. I have used a heath-robinson workaround, which has worked for me. But YMMV.
The OP's question discusses the situation where there is a formula which was working well, but after some time of operation, a Ctrl+Shift+E situation has started to occur.
(There are other situations where Ctrl+Shift+E might occur due to results of one formula attempting to overwrite onto cells written by another formula. In this scenario, sometimes adding EXPAND at the beginning of the desired formula does the trick. Also using Filter functions, rather than 'IF' comparisons, does mop-up previous =CONTINUE(1,2,3) cells ... but perhaps neither of these are likely to be of help to the OPs question).
In my experience the spreadsheet might develop such a 'confused' behaviour over time, so requiring Ctrl+Shift+E, meaning a period of time where there are several uses-of-the-spreadsheet resulting in changes to the data. I might further speculate that there is more likelihood of this confused behaviour when the referred-to data is more dynamic e.g. rows added, rows removed OR that there are other formulas which also calculate from the same data set (let alone that these two formulas relate to each other).
I think that the idiom 'confused-behaviour' is appropriate since the actual engineering of the spreadsheet is an open box only to google engineers; we cannot rationalise its workings. So it seems mysterious and magical how, in the OP's case and my experience, the spreadsheet was working well, but then altered behaviour to require Ctrl+Shift+E without there being any change in the formulas, only in the sheets' accumulated data changes through usage.
I am giving an example of my workaround using an example case of mine. Note that in my case the data and also the problematic formulas are in one single sheet which I am calling the production sheet
The workaround involves using a script to duplicate a 'template-sheet', in the same spreadsheet as the production sheet. The template-sheet is structurally identical to the production sheet with the same column headings, but with only a few lines of sample data. It has the same formulas as the production sheet, which refer to the sample data in the self-same template sheet (not to the production sheet). Importantly, the template sheet is not showing 'confused-behaviour' - no Ctrl+Shift+E is required in the formula cells should there be any changes to the template's sample data. It also contains the formating of the production sheet.
So, when the script runs, it creates a duplicate of the template-sheet; it subsequently copies over the production sheet's data to this template duplicate and reapplies the formats. This duplicate becomes the new production sheet. The old production sheet is hidden (in my case, but could be deleted).
I have this system operational on a Tasks List spreadsheet being used by 12 people daily, where Tasks come in by google form throughout the day (and are themselves copied over to the production sheet by another script, not shown as not relevant to this OP). Once a task is completed, the task's row is removed from the production sheet. So the data grows and shrinks constantly throughout the day. The duplicate sheet is created each night, by timed trigger upon the insertSheet() function. Here is the script I use:
function insertSheet(){
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet =ss.getSheets()[0]; //first sheet is the production sheet
var sheetRange = sheet.getRange('A3:P');
var sheetValues= sheetRange.getValues();
var d = Utilities.formatDate(new Date(), Session.getTimeZone(), 'ddMMMyyyy-hh:mm:ss');
var ex = sheet.setName('CCEs' + d);
var templateSheet = ss.getSheetByName('templateSheet');
var s2 = ss.insertSheet(0,{template: templateSheet});
var height = sheetRange.getHeight();
s2.insertRowsAfter(2, height)
s2.setName('CCEs');
templateSheet.hideSheet();
ex.hideSheet();
s2.getRange('A3:P' + (sheetValues.length+2)).setValues(sheetValues);
format(); //
templateSheet.hideSheet();
ex.hideSheet();
}
function format(){
var ss = SpreadsheetApp.getActiveSpreadsheet();
var formatRange = ss.getSheetByName('templateSheet').getRange('B2:N2').copyTo(ss.getSheetByName('CCEs').getRange(2,2, ss.getSheetByName('CCEs').getMaxRows() -1,14), {formatOnly:true});
}
Prior to deploying this strategy, the spreadsheet would experience 'confused-behaviour' after a couple of days or so.
The sheet contained three formulas which would become confused. They are not important to the OP, but I duplicate here just to give the reader a feeling for the success of this strategy:
=arrayformula(IFERROR(FILTER(if(row(O:O) =1,"Auto Time Stamp ",iferror(1/0)) &O:O&if(row(O:O) =1,"copy",),len(A:A)),"Error"))
=Arrayformula(iferror(if(filter(L:L, len(A:A)) - filter(A:A, len(A:A))>0, if( int(filter(L:L, len(A:A)) - filter(A:A, len(A:A))) = 0 , text( text(filter(L:L, len(A:A)), "HH:mm") -text(filter(A:A, len(A:A)), "HH:mm") , "H:mm") , int(filter(L:L, len(A:A)) - filter(A:A, len(A:A))) & "Day(s), " & text( text(filter(L:L, len(A:A)), "HH:mm") -text(filter(A:A, len(A:A)), "HH:mm") , "H:mm") ) ,iferror(1/0)),"Time Taken"))
=arrayformula( IFERROR(if((ISBLANK(FILTER(J1:J,LEN(A1:A))) * (FILTER(G1:G,LEN(A1:A)) = "Normal")* (now()-FILTER(A1:A,LEN(A1:A))> OverdueTimings!A1 )),"OVERDUE Normal", if((ISBLANK(FILTER(J1:J,LEN(A1:A))) * (FILTER(G1:G,LEN(A1:A)) = "Urgent")* (now()-FILTER(A1:A,LEN(A1:A))> OverdueTimings!A2 )),"OVERDUE Urgent", if((ISBLANK(FILTER(J1:J,LEN(A1:A))) * (FILTER(G1:G,LEN(A1:A)) = "Very Urgent")* (now()-FILTER(A1:A,LEN(A1:A))> OverdueTimings!A3 )),"OVERDUE V. Urgent", IFERROR(1/0)))),countif(if((ISBLANK(FILTER(J1:J,LEN(A1:A))) * (FILTER(G1:G,LEN(A1:A)) = "Normal")* (now()-FILTER(A1:A,LEN(A1:A))> OverdueTimings!A1 )),"OVERDUE Normal", if((ISBLANK(FILTER(J1:J,LEN(A1:A))) * (FILTER(G1:G,LEN(A1:A)) = "Urgent")* (now()-FILTER(A1:A,LEN(A1:A))> OverdueTimings!A2 )),"OVERDUE Urgent", if((ISBLANK(FILTER(J1:J,LEN(A1:A))) * (FILTER(G1:G,LEN(A1:A)) = "Very Urgent")* (now()-FILTER(A1:A,LEN(A1:A))> OverdueTimings!A3 )),"OVERDUE V. Urgent", iferror(1/0)))),"OVERDUE *")& " OVERDUE"))
After several days, we delete all the accumulated old production sheets.