I have a bunch of workbooks that have dozens... up to 100... sheets in them, and we do an import process, loading one sheet after the other, on these workbooks that tends to get throttled when we collate our entire body of data. In other words, we read every sheet in every workbook and partway through the process, we get bogged down by the fact that Google is putting off our calls.
What I'd like to do is look at the last modification time for each sheet before I request it. If it is more recent than the data I've cached for that sheet, I'll download it. Otherwise, I'll skip it and use my cached version.
It's absolutely possible to get the last mod time for the workbook, but if I used that, and only one sheet in the book changed, I'd be stuck downloading all 100 sheets in that book when I may only need one.
I've looked at the idea of having Google notify me any time a cell changes, but there are limits on how many notifications are sent in a day. If I were to activate the notification for all the sheets I'm watching, and turn them off as I mark cached data as being dirty, I would still have the problem that I wouldn't know if I'd suddenly stopped getting notifications because I was being throttled. There are some sneaky workarounds to this, but none of them would notice if I'd been throttled for a while, but wasn't anymore.
So, how do I find the last modification for a sheet - preferably via one of the Python libraries?
Related
I have a container-bound Apps Script Project contained to a Form Response Google Sheet, triggered on form submit. The script runs as me. I'm dealing with execution runtimes 6-8x the nominal run time during peak hours of the day, which seems largely correlated to increased traffic of form submissions. The program follows this series of steps:
Form response collects a string of information and triggers the Apps Script Project to execute
Project creates a copy of a complex Google sheet of a few tabs and a lot of complex formulas
Project pastes the string collected from the form response into this google sheet copy, flushes the sheet, and then returns a string of results calculated within the sheet
Google sheet file is named and the Project creates a unique Drive folder where this sheet eventually gets moved to.
Run complete
The Project performs a wide variety of getValue and SetValue calls throughout the run as it's updating cell values or reading calculated results. Over the last year, I've improved optimization in many ways (i.e. batch calls to getValues or setValues, batch API calls, etc). It's normal run time is 25-45 seconds, but increases to 200+ seconds during my company's peak business hours. Using logs, there is no one particular step that gets hung up. But rather the script lags in all aspects (When it creates the file copy, SpreadsheetApp.flush(), append or delete rows in google other sheets it references by Sheet ID, etc). Although, I'll say a common reason for a failed execution returns the error message "Service Spreadsheets timed out while accessing document with id..." But most of the executions complete successfully, just after a lengthy run.
It seems there is a clear correlation between execution time and traffic. So my questions are:
Can anyone confirm that theory?
Is there a way I can increase the service's bandwidth for my project?
Is there a better alternative to having the script run as me? (mind you I'm performing the rest of my job using Chrome throughout the day while this program continues to automatically run in the background)
This is an Apps Script managed project; it is not tied to a personal GCP Project. Is that a potential factor at play here?
I have experienced a case where a Google Apps Script that moves data from one Google Sheet to another Google Sheet, is not pulling the current data in the sheet. I can verify this by checking the Version History in both sheets.
The issue occurred when the script ran from Google Trigger. Though I could not replicate when forcing the script.
I recorded a video to explain the issue:
https://drive.google.com/file/d/1HChy4WOR5ehg8EA6rlmvSH7Lqjg9bd30/view?usp=drivesdk
As the issue of "pulling old data", is only occurring on ~50 rows. And these are all items that we recently edited. So, that tells me that instead of pulling the "current data" in the sheet, Google pulled some "cached version of the data" instead (i.e. data from a few days, or week before).
I also experienced the same type of old data / caching issue with "IMPORTRANGE" formulas last week. Though this is first instance I have encountered the issue from a script.
This problem has me very worried, as I have setup many business systems to run on Google Sheets and Google Apps Script and rely GAS scripts moving the current/up to date data... for obvious reasons.
I am business owner, not a developer - but I would bet that Google has changed something in recent weeks relating to 'caching of data in Google Sheets'. I have tried reaching out to Google Workplace support to raise this issue but they don't seem to care, and they don't offer any support for GAS.
Hopefully others have this issue and we can get Google to resolve. In the meantime, I have to start preparing for a move away from the unreliable GAS and Google Sheets. Airtable is what I will be looking at firstly.
I just came across this same issue with a script of mine. It pulls cached data for about 4 minutes, then updates to the live version, then repeats that for the next few runs.
For my script in particular, I solved it based on an answer I received here: Script Pulling Cached Data. I set the text value "Running" to an empty cell on the source sheet at the start of the script, then flush, then delete that value from the dummy cell at the end of the script. Forcing it to write to the source tab seems to force the script to pull live data. It's been working consistently for a few hours now, so I believe this is a viable solution.
I hope this helps!
I am familiar with the Lock Service but that is only for locking scripts.
I have some code that will "process" a large Google Sheet. My script needs to re-order the rows. I need/want to make it so while the script is running nobody else can change the order. However, I still need another script to be able to append rows.
We use a Google Form for our team's intake. It appends rows to a sheet. I have an hourly job that will go through all the rows/records and "process them". I have a column that stores the last time a record/row was "processed". I want to sort on that column such that the "oldest" records are on top and then start processing from the top down. If the script fails or times out then the next iteration will just start over...
I know I could use getValues or getDisplayValues to get an array and then write the array back but I worry what would happen if someone sorted the rows as it would muck things up when writing the array back.
Is there some way to accomplish my goal? I want to be able to process the records, and maintain row order to avoid breaking my processing.
The way to block a spreadsheet "completely" is by changing the spreadsheet sharing settings. Remove all editors or change them to viewers, once your script finish, change them back as editors. In a extreme case, usa a second account to act as the owner of the critical files / spreadsheets and only use it for this purpose,so you could block your regular account for doing changes to the spreadsheet.
NOTE: A Google Form editResponseUrl could be used to edit the linked spreadsheet.
I'm facing a similar situation but I took a different approach, I'm using an index/key column (you could use the timestamp column) and using the index/key to save each edited row to the right position, then write the whole resulting array in a single operation (by using setValues()). In my case this is simple because I only require values, I'm not worried about notes, data validation, conditional formatting, comments, etc. and there isn't a Google Form linked to my spreadsheet.
Related
Google Spreadsheet -- get sharing permissions by script
Any way to share google docs programmatically?
Google sheets changed the size of data that can be imported using the 'importrange' function in late March. We want to write a formula that can search through all our sheets and identify any that will be affected by Google's latest change.
We know there has been a change, as Importranges we had previously setup, are now returning "... to large" type error.
It appears to a limit of data size, not cells - as we experienced the issue on importrange of only 2 columns, but one contained large amount of HTML code in each cell.
Does anyone know what the data size limit is?
If not, then does anyone have an idea for best way to write a script that can find the limit?
Or next step will be to write a script that can search through our 'network analysis' sheet (sheet/tool that shows all Google sheets that are connected by importrange), and identify those with the importrange issue.
FYI - Google appears to have changed quite a bit relating to Google sheets and apps script.
a) We lost the ability to save changes in a sheet (making it basically unusable), which we think is related to having >50 importranges referencing that sheet (google support advised to avoid >50 importranges going to one sheet).
b) Other change we noticed is that google apps script can definitely now run for longer than the previous limit of 5min. We previously were seeing some scripts, sometimes running >5min, all the way up to a max of 30min. But now we see it consistently on some scripts. We built a tool, that can automatically get a script to 'run again' if it did not complete last time. We have had to tweak that 'run again script', to ensure it does not try to run the script again within 30min of the first run, to ensure that first script is definitely still not running.
I have a report file get the content from other files in a Google drive to generate a report. The script do the following
Get list of files in folder (and their IDs)
Create new sheets using predetermined form
Get content from files in folder into the sheets (via IMPORTRANGE)
Get summary of files' content in each sheet (progress of projects)
Get the summary in sheets into one single sheet (via INDIRECT)
However, it seems that the result could behave wildly. Sometimes IMPORTRANGE will not get the correct file content even all the results lead to it are all correct. Sometimes INDIRECT will not get error even when enter the formula exactly the same manually will get the correct result. Sometimes the function getSheets() will be error but most of the time it will not.
When the result is not correct, I just delete the newly created sheets and run the code all over again. After awhile, they all will be correct.
So my question is what's the problem/limitation with google sheet or script that they will have results varies wildly between run?
After a lot of trying and searching, it ALL comes down to IMPORTRANGE. It has a specific cycle of updating value of 30 mins. To force the function into update more regularly, just put in a variable that will.
The solution is something like this
=IF(MINUTES(NOW())=MINUTES(NOW()),IMPORTRANGE(.....)
After I implement this, everything goes smoothly.