I have a sheet that pulls alot of finance data from alot of web pages using HTML. Problem is that it slows down alot. Plus it starts giving errors due to high number of HTMLs and etc.. i looked for the following solutions:
Having an app script to make refreshes controlled..(not so effective)
Any way i could copy paste the old data automatically till new data comes in and updates old pasted values..(also not effective as i couldnt find an automated method)
Tried finding a way to play with google sheet auto-calculations to help....(also failed in that)
IS there a way to ristrict the sheet from auto refreshing so many times...?
In the documentation you can see that:
Functions that pull data from outside the spreadsheet recalculate at the following times:
ImportRange: 30 minutes
ImportHtml, ImportFeed, ImportData, ImportXml: 1 hour
GoogleFinance: may be delayed up to 20 minutes
If you want to recalculate the value for your importHTML functions with a lower frequency you should definitely use Apps Script to do the data fetch and then populate your Spreadsheet with the information.
You can use UrlFetchApp class on Apps Script to get the data and define your timing logic for the updates.
function myAppScriptFunction() {
var urls = ["your-website-url1", "...", "your-website-urlN"];
var response = UrlFetchApp.fetchAll(urls);
//... Parse the response and store the information you need in a table-like data structure
// Let's assume the variable parsedResponseData is created
var ss = SpreadsheetApp.getActiveSheet();
ss.getRange("your-range").setValues(parsedResponseData);
}
You should now wrap your function in your custom time logic to manage the updates.
You can achieve this with a time-driven trigger on Apps Script.
function triggerSetup() {
ScriptApp.newTrigger('myAppScriptFunction')
.timeBased()
.everyHours(6)
.create();
}
References:
Time-driven triggers
UrlFetchApp
Quota Limits
you may try this addon which freezes your spreadsheet on demand:
https://gsuite.google.com/marketplace/app/spreadsheet_freezer/526561533622
Related
I have a Google Sheet that uses an IMPORTRANGE query to combine data from multiple other sheets. This combined import sheet is read by Google AppSheet. We have realized that the data AppSheet is reading is always outdated. It only reads the data as of the last time the sheet was manually opened.
I followed the steps in this post to try to fix this issue by creating this function: function refresh() {SpreadsheetApp.flush()}. I then set up a timed trigger to activate it once an hour. Logs show the function is running, but the data is still not updating until I manually open the sheet.
This is my first time using Apps Script. Any tips/ideas? Is there a different or better way to have the formulas update without opening the file?
Thank you for reading.
SpreadsheetApp.flush() only works for the script execution that calls it. If you need to refresh the data results from a formula it's uncertain how exactly the spreadsheet will respond as most of the formula calculations are done on the client side. You could verify this by yourself by using your web browser developer tools.
Anyway, spreadsheet formulas have several caveats so it will not be extrange that at some point you will have to rethink your solution. Assuming that you want to keep using AppSheet:
Use AppSheet for your front end and some no-code / low-code automation. Keep your app small, if you need many forms / views consider to distribute them among several apps.
Use Google Sheets only for data storage for your AppSheet app. Please bear in mind that it has 10 million cells limit for the whole spreadsheet, so you might want to delete the unused sheets and delete the unused columns and rows on each sheet.
You might use Google Apps Script to do the data import and transformation tasks. If you need that something be updated based on actions done on the AppSheet app, you might use an installable change trigger or use webhook from the AppSheet side to and a "simple" web application using Google Apps Script (you could use GET / POST http requests to trigger some Google Apps Script functions).
Also you might use other programming platforms for the data import / transformation tasks and keep using Google Sheets as your AppSheet database by using the Google Sheets API or other automation tools like Zappier, IFTTT, Integromat among many others.
solution #1
You can try this solution :
define a checkbox (for instance in A1 in tab Sheet1)
set this script
function myFunction() {
var chk = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Sheet1').getRange('A1')
chk.setValue(false);
SpreadsheetApp.flush();
Utilities.sleep(500);
chk.setValue(true);
}
define a trigger on it
define the formula as follows
=if(A1,importrange("1n-rjSYb63Z2jySS3-M0BQ78vu8DTPOjG-SZM4i8IxXI","A:Z"),"")
when A1 is unchecked, the result will be empty, then check A1 to fill once again the result as expected
solution #2
by script, try for instance
function myFunction() {
var sh = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Sheet9')
var data = SpreadsheetApp.openById('1n-rjSYb63Z2jySS3-M0BQ78vu8DTPOjG-SZM4i8IxXI').getSheets()[0].getDataRange().getValues()
sh.getRange(1,1,data.length,data[0].length).setValues(data)
}
put a daily triger as needed
I've been trying to get Yahoo Finance ES symbol daily OHLC data into Google Sheet.
I've checked the disabling javascript and reloading the page method by #Rubén and the OHLC data displays after reloading the page. But the IMPORT formulas returns #N/A Resource at URL Not Found.
I also tried #user0 methods (save the IMPORTJSON one) here but no luck there too.
Then I've tested the script by #JohnA and it does retrieve the data.
Here's #JohnA Script output I got.
But now I'm wondering how to make it auto-update at least every day (it is daily data).
I'm not sure how to proceed since #JohnA's Script involves handling a CSV file.
I've enabled calculation on change and every minute as detailed here: How to Configure Auto Refresh for Googlesheets
Is there any workaround allowing to just get the data into Google Sheet without needing the csv step?
Or is it possible to still pull the data from the CSV while getting Google sheet to auto-update daily? If yes, how to do it?
Thanks for any advice, always very much appreciated!
The test Sheet1 and Sheet2
Related scripts by #Tanaike 1 #Tanaike 2
In script editor look for the clock
Then complete the form
Optionally, you can set a trigger via a script
/**
* Creates time-driven triggers
*
* https://developers.google.com/apps-script/reference/script/clock-trigger-builder
*/
function createTimeDrivenTriggers() {
// Trigger every day at 04:00AM CT.
ScriptApp.newTrigger('csvDaily')
.timeBased()
.everyDays(1)
.atHour(4)
.create();
}
I am using the script of https://github.com/bradjasper/ImportJSON to import JSON data (in my case crypto currency prices) into google sheets. However I notice that the ImportJSON function I use does not always get called when I reload the sheet so I don't know if I have the last data or not. Approx. half of the time I get the new data and the other half it keeps the old data. The trick I use to have it load the new data is edit the script and rename the ImportJSON to something else such that the spreadsheet fails and then undo the change and then ImportJSON gets called. However this is quite tedious.
Is there a way such that with every reload google sheets reevaluates all formulas and loads the new JSON data
Thanks
Jaap
Is there a way such that with every reload google sheets reevaluates all formulas and loads the new JSON data?
There is a Simple Trigger called onOpen which executes every time a sheet is opened or when the page is reloaded.
function onOpen() {
//Write your codeblocks here. For this sample, I'm just using an alert..
SpreadsheetApp.getUi().alert('Function Triggered on Reload!');
}
I have created a form that pushes data to a Google Spreadsheet. The data is latitude, longitude, location, and other identifying data. The spreadsheet is then published as a .CSV file and imported into ARC GIS to be displayed on an interactive map. It works exactly as I wanted and I set it to republish after each change.
The problem is that when the spreadsheet has rows appended by the script, it is not seeing it as a change and republishing. In order to get the updated data imported to the map, I need to go in and manually republish. Is there anyway through the Google Apps Script that I could make a few lines of code to force a republish? I could then add that to the "on form submit" script I have or another time based one that already runs at 3 am everyday.
I have looked through the Google Apps Script documents and not found anything. When searching for help on the web, the overwhelming majority of responses are for how to publish your script as a template for other.
My testing sheet was republished after the following function was executed by either a menu entry or a time-based trigger.
function ChangeIt() {
var sheet = SpreadsheetApp.getActiveSpreadsheet()
var t = new Date()
var x = 'upd: ' + t
var range = sheet.getRange('a3')
range.setValue(x)
}
If I were in your shoes, I'd add an extra column to the end of the sheet with some benign constant data that a script can change without affecting the systems consuming the data. If an extra column isn't an option, try modifying my sample to read in a current value, change it, and immediately change it back.
Also, I'd see if the spreadsheet onEdit() trigger fires when the form submit adds a new row. If so, tie your GAS function to it to force the republish. If not, setup a timed trigger to execute the GAS function.
A quick workaround for this issue that doesn't require scripting is to simply make an array copy of the data.
For example, I made a new tab and in A1 put this: =ArrayFormula('Form Responses 1'!A1:Z1000)
While the main Form responses tab will insert rows and not play nice with formulas this new tab stay nice and constant and updates automatically when new data is added.
Because google spreadsheets does not support iterations, I wrote my own simple app script to adjust an input based upon the calculation of the spreadsheet output. However, after I change the input variable, the spreadsheet recalculates but app scripts does not seem to wait for that recalculation so I end up retrieving values such as "Thinking..." or "#NA". Is there a way to pause a script and wait for the calculation to complete before moving to the next line in the script?
Currently, I am just using a loop to watch the cell but I wanted to find out if there was a more elegant way to pause the execution until the sheet was done calculating.
I write a lot of Excel Macros and Excel VBA always waits for the calculation to complete before moving to the next line in the code. Apps Script does not seem to do this so I am hoping there is an easy way to do this.
A second question: Because this iteration can take some time, how does one interrupt and terminate a script from running? I can't seem to find a way to do this.
Here is a very simple way of preventing the next script from starting until the current script completes in google apps scripts. Just add a call for testWait() after each script you are processing successively. The SpreadsheetApp.flush() also seems to reset the timeout timer on the spreadsheet back to the default 5min so you have more time to process multiple scripts in one go.
//holds processing of next script till last one has completed
function testWait(){
var lock = LockService.getScriptLock(); lock.waitLock(300000);
SpreadsheetApp.flush(); lock.releaseLock();
}
Scripts timeout after about 6 minutes to prevent infinite loops and constantly running programs. I don't think there's a manual way to stop a script.
Edit: Oops, forgot to answer your first question: I thought onEdit ran after values were recalculated, but apparently I don't use enough formulas to see this. If it's not waiting, then the best way is to do something like this:
while(value === "Thinking..." || value === "#NA") {
Utilities.sleep(10000);
}
It pauses the script for a few seconds and then checks again.
I also write a bit in Excel VBA, where the native Calculation functions come in handy. I've run across the same problem in Google Apps/Docs several times, wanting to execute code whenever calculations are complete. I wonder why there's no native function in Google Apps/Docs to handle this. Anyhow, I wrote this code to solve the problem. Hope it helps.
function onEdit() {
Refresh();
};
function Refresh () {
var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1");
var sheet2 = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet2");
// set the range wherever you want to make sure loading is done
var range = sheet.getRange('A:A')
var values = range.getValues();
var string = values.toString();
var loading = "Loading";
do{
var randomWait = Math.floor(Math.random()*1+0); randomWait;
} while (string.search(loading) ==! 0);
range.copyTo(sheet2.getRange('A1'), {contentsOnly:true});
customMsgBox();
};
function customMsgBox() {
Browser.msgBox("Data refreshed.");
};
Here's an example in action:
https://docs.google.com/spreadsheet/ccc?key=0AkK50_KKCI_pdHJvQXdnTmpiOWM4Rk5PV2k5OUNudVE#gid=0
Make a copy if you want to play around with it.
I had the same problem. I resolved it by using 2 scripts as I needed to be sure the spreadsheet has the data I need.
The first script provides a seed value to populate the spreadsheet through a IMPORTXML function.
The second script processes this data.
I used time based triggers to run the scripts allowing for sufficient time for the first script to complete
You could put the recursive structure in the code.js file, rather than on the spreadsheet. Javascript can handle recursion really well. You could then have the script update the spreadsheet on each iteration (or each 100).