I've a spreadsheet that uses a function from another external script (https://github.com/Eloise1988/CRYPTOBALANCE) which grabs the balance from a wallet.
I want to snapshot this value daily on another column, so I've created the following script:
function snapshot() {
SpreadsheetApp.flush()
// Assign 'dashboard' the Dashboard sheet.
var Carteiras = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Carteiras');
attempts = 0;
while (
(Carteiras.getRange("C2:C2").getValue() === '#NAME'
|| Carteiras.getRange("C2:C2").getValue() === "#NAME?"
|| Carteiras.getRange("C2:C2").getValue() === 'Loading...'
|| Carteiras.getRange("C2:C2").getValue() === 'Contact for BALANCE : t.me/TheCryptoCurious')
&& attempts < 60
) {
console.log('Formula is not yet ready... re-attempting in 1seg');
console.log('C2 value is ', Carteiras.getRange("C2:C2").getValue());
Utilities.sleep(1000)
attempts++;
}
console.log('C2 value is ', Carteiras.getRange("C2:C2").getValue());
if (attempts < 60) {
Carteiras.getRange("D2:D23").setValues(Carteiras.getRange("C2:C23").getValues());
console.log('Values updated successfully!');
} else {
console.error('Failed to grab the formula values.');
}
}
This script basically attempts to grab the balance from the wallet (Columns C2:C) , i know that once C2 is loaded all the others are loaded too, so I'm checking that C2 is in a valid state (e.g.: Not loading, no #Name or anything)
I've set a time driven trigger to run this snapshot function every day in the morning (10am to 11am) -- The problem is that the column is always on #NAME?
I think at some point google is not allowing this other external script to run, any ideas how can i make sure how to run this other script?
Also any improvements on my code will be welcomed as i never did anything on google spreadsheets.
Appreciated!
Instead of trying to read the result of custom function from the spreadsheet, call the custom function as a "normal" function.
function snapshot(){
const spreadsheet = SpreadsheetApp.getActiveSpreadshet();
var Carteiras = spreadsheet.getSheetByName('Carteiras');
const values = Carteiras.getRange("C2:C23").getValues();
const balance = values.map(row => {
const ticker = 'a-ticker'; // or use row[some-index-1] or other way to get the ticker
const address = 'a-address'; // or use row[some-index-2] or other way to get the address
const refresh_cell = null; // not needed in this context
return [CRYPTOBALANCE(ticker,address, refresh_cell)]
});
Carteiras.getRange("D2:D23").setValues(balance);
}
The above because Google Apps Script officials docs have not disclosed how exactly the formula recalculation works when the spreadsheet is opened by the script when the spreadsheet has not been first opened by the active user as usually occurs when a daily time-driven trigger is executed. I guess that the custom functions are loaded into the active spreadsheet function list when the formula recalculation is triggered by Google Sheets web client-side code.
Related
Custom formula returning #NAME when Google Sheets is published to web
Google Sheet Script Returning #NAME?
Why am I getting #NAME? retrieving google sheets cell values that depend on custom functions via API?
Google sheets custom function data disappears when viewing site as html after 20 minutes
Are there methods to measure the execution time when built-in functions completed for Spreadsheet? When I use several built-in functions (For example, IMPORTHTML and IMPORTXML), if I know the average execution-time, it is easy for me to use and design data sheet.
I measure it of custom functions using this script.
function myFunction() {
var start = new Date();
// do something
var end = new Date();
var executiontime = end - start;
}
Thank you so much for your time and advices.
Unfortunately, there are not measurement tools for retrieving the execution time of built-in functions. This has already been commented by #Rubén. So I thought of about the workarounds. How about the following workaround?
Flow :
Import a value to a cell. The value is anything good, because this is used as a trigger. Please do this by yourself.
Custom functions cannot use setValue(). So I used onEdit().
func1() imports a formula that you want to measure the execution time by the script launched by the trigger.
At func2(), after set the formula, the measurement is started. The confirmation when built-in function was completed is carried out using loop.
By measuring the cost per one call for getValue(), it was found that that was about 0.0003 s. So I thought that this can be used.
The result of measurement can be seen at Stackdriver as milliseconds.
Sample script :
function func1(range, formula){
range.setFormula(formula);
}
function func2(range){
var d = range.getValue();
while (r == d) {
var r = range.getValue();
}
}
function onEdit(){
var formula = '### Built-in function ###'; // Please set the built-in function you want to measure the execution time.
var label = "Execution time for built-in functions.";
var ss = SpreadsheetApp.getActiveSheet();
var cell = ss.getActiveCell();
var range = ss.getRange(cell.getRow(), cell.getColumn());
func1(range, formula);
console.time(label);
func2(range);
console.timeEnd(label);
}
Note :
When built-in functions with very long time is measured, an error may occur at getValue().
In my environment, the built-in function for 10 seconds worked fine.
Updated at November 11, 2020:
As the additional information, I would like to add one more sample script for measuring the execution time of when built-in functions completed for Spreadsheet and the result using the script.
This is a simple sample script for measuring the process cost of functions on a cell. At first, in order to confirm whether this script can be used for measuring the process cost of the function put in a cell, a custom function was used. Because when the custom function is used, the process time of the script can be known by using Utilities.sleep(time).
Sample script:
When you test this script, please copy and paste the following script to the container-bound script of Google Spreadsheet. When you run the function of main(), the process cost of =SAMPLE(5000) can be obtained.
// This is a sample custom formula. This is used for testing.
function SAMPLE(time) {
Utilities.sleep(time);
return "ok";
}
// This is a script for measuring the process cost.
function main() {
const obj = { formula: `=SAMPLE(5000)`, returnValue: "ok" }; // Set formula and response value.
const label = "Execution time";
const sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet1");
const range = sheet.getRange("A1");
range.clear();
// start --- measure
console.time(label);
range.setFormula(obj.formula);
SpreadsheetApp.flush();
while (range.getDisplayValue() != obj.returnValue) {}
console.timeEnd(label);
// end --- measure
range.clear();
}
In this sample, when =SAMPLE(5000) is put to a cell, the value of ok is shown in the cell after 5 seconds. main() measures the process time for this.
This sample script checks the output value from the function. So please set returnValue. Please be careful this.
At the formula of Spreadsheet, when the formula is put to the cell, the measurement of process time is started. So in this sample, I included setFormula and flush to the measured process cost.
Experimental result:
As an experiment, it shows the change of process time with increasing the sleep time of the custom function as above image. This result indicates that the process time is linearly increased with the increase in the sleep time. It was found that the process time was at least, more than the sleep time, and the process time was large in about 0.5 s for each sleep time as the offset. It is considered that this offset includes the process costs of setFormula, flush, getDisplayValue and the while loop. But, from this image, it is considered that when the process costs of various formulas are measured using above script, those can be compared by the relative comparison. And from this result, it is considered that above script can be used for measuring the execution time of the function in a cell of Spreadsheet.
Reference:
Benchmark: Measuring Process Costs for Formulas in Cells on Google Spreadsheet using Google Apps Script
Google Sheets doesn't include a built-in tool to measure the recalculation time.
One alternative is to use the Chrome Developers Tools Timeline but bear in mind that functions like IMPORTHTML and IMPORTXML are not recalculated every time that the spreadsheet does (reference Set a spreadsheet’s location and calculation settings).
Related Q&A
SO
Google spreadsheet constantly recalculates functions
Web Applications
What's causing / how to get rid of long web calls in Google Spreadsheets
Where does the calculation take place?
One can make a Named Function (just to have it easily reusable) to measure formula execution using only formulas, no scripts needed.
BENCHMARK()
BENCHMARK() (returns execution time in seconds):
=N(
REDUCE(
0,
{0; 1; 0},
LAMBDA(
acc,
cur,
IF(
cur = 0,
LAMBDA(x, x)(NOW()) - acc,
acc + 0 * ROWS(func())
)
)
) * 24 * 60 * 60
)
Its single parameter is func - a function to run and time its execution. You just get a working formula, enclose it in LAMBDA() and pass to BENCHMARK(). Examples are at the bottom.
BENCHMARKN()
BENCHMARKN():
=N(
REDUCE(
0,
{0; SEQUENCE(number); 0},
LAMBDA(
acc,
cur,
IF(
cur = 0,
LAMBDA(x, x)(NOW()) - acc,
acc + 0 * ROWS(func())
)
)
) * 24 * 60 * 60 / number
)
There's another parameter number: func will be executed in a loop number times. Then total execution time will be divided by number to get an average execution time.
UPD: enclosing NOW() in LAMBDA(x, x)(NOW()) freezes the value and prevents recalculation caused by any change in the sheet. Got it from here.
Examples
Say, we want to sum all squares of natural numbers from 1 to 1000000. So we have our formula:
=SUM(ARRAYFORMULA(SEQUENCE(1000000)))
We time it like so:
=BENCHMARK(LAMBDA(SUM(ARRAYFORMULA(SEQUENCE(1000000)^2))))
And in a 10-step loop like so:
=BENCHMARKN(LAMBDA(SUM(ARRAYFORMULA(SEQUENCE(1000000)^2))), 10)
Here's my option to caclulate the difference between ArrayFormulas:
Create a template with sample data
Set the script: change array with formulas, template Spreadsheet Id, cell address to set formulas, cell address to iterate values.
This function will test 2 or more formulas:
Copy sample file to isolate the influence of previous iterations.
Set formula and calculate the time to calculate it the first time
Edit source data in a loop and calculate time needed for the formula to recalculate.
Sample output, plotted in Sheets:
↑ formulas for the tests are taken from this question.
The y-axis is the processing time, and bars are the iterations of the script.
the low bars are the the times to insert the formula,
and higher bars are the times to re-calculate the formula when the value changed.
The Code:
function test_formulas_speed() {
/** Options ↓ ********************************************************************************************* */
// https://docs.google.com/spreadsheets/d/1vQu7hVr7FwH8H5N8JOlOGfvjlJgKtpfoM2DPPjgUaLo/template/preview
var testSpreadsheetId = '1vQu7hVr7FwH8H5N8JOlOGfvjlJgKtpfoM2DPPjgUaLo';
var formulas = [
'=INDEX(if(A2:A="",,LAMBDA(srt, SORT(SCAN(0,MAP(SEQUENCE(ROWS(A2:A)),LAMBDA(v,if(v=1,0,if(INDEX(srt,v,1)<>INDEX(srt,v-1,1),1,0)))),LAMBDA(ini,v,IF(v=1,1,ini+1))),index(srt,,2),1) ) (SORT({A2:A,SEQUENCE(ROWS(A2:A))}))))',
'=LAMBDA(a,INDEX(if(a="",,COUNTIFS(a,a,row(a),"<="&row(a)))))(A2:A)'
];
// value from the first cell of arrayformula ↓
var returnValue = 1;
// range to insert the formula
var rA1 = 'B2';
var sheetName = 'Sheet1';
var iterations = 5;
var testChangeRaangeA1 = 'A5';
var sub_iterations = 5; // the number of times to change the value
/** Options ↑ ********************************************************************************************* */
var results = [];
var file = DriveApp.getFileById(testSpreadsheetId);
var results = []
var testOne_ = function(formula, indx) {
// prepare
var copy = file.makeCopy();
var id = copy.getId();
var ss = SpreadsheetApp.openById(id);
var s = ss.getSheetByName(sheetName);
var r = s.getRange(rA1);
var rCh = s.getRange(testChangeRaangeA1);
var addToResult = function(t) {
var result = new Date() - t;
if (results[indx]) {
results[indx].push(result);
} else {
results[indx] = [result];
}
}
// measure time
var t = new Date();
r.setFormula(formula);
SpreadsheetApp.flush();
// loop until expected value is returned
while (r.getDisplayValue() != returnValue) {}
addToResult(t);
for (var i = 0; i < sub_iterations; i++) {
t = new Date();
// clear the cells, because of the internal cache
rCh.clearContent();
rCh.setValue(i);
SpreadsheetApp.flush();
addToResult(t);
}
// clean up
copy.setTrashed(true);
return 0;
}
for (var i = 0; i < iterations; i++) {
formulas.forEach(testOne_);
}
console.log(results);
}
Ref:
This benchmark by #Tanaikech
To add another option you can do:
function myFunction() {
console.time('someFunction');
// do something
console.timeEnd('someFunction');
}
And then look at the Stackdriver logs for the Function Execution.
Example output would be:
Jul 3, 2020, 1:03:00 AM Debug someFunction: 80ms
I have an object called dataMapEN --> var dataMapEN = {};
This object has say 30000 entries.
I am trying to write them to Google spreadsheet using the following code:
var sheet = ss.getSheetByName("EN");
populateSheet(sheet, dataMapEN);
function populateSheet(sheet, dataSource){
for(item in dataSource){
sheet.appendRow([item, dataSource[item].clicks, dataSource[item].impressions, dataSource[item].cost, dataSource[item].conversionValue, dataSource[item].conversions]);
}
}
What I am seeing is that to write 200 entries the script takes about 2 minutes.
The script times out everytime I write to the sheet.
Is there a faster way to write to the sheet so that the script finishes in the 30min window ?
Thanks
Your script is so slow because it's doing 30k write calls to the spreadsheet.
Try putting everything into a 2d array first and then doing only one call.
Something like this (not tested because no example):
var sheet = ss.getSheetByName("EN");
populateSheet(sheet, dataMapEN);
function populateSheet(sheet, dataSource){
var dataToWrite = dataSource.map(funtion(row, item) {
return [item, row.clicks, row.impressions, row.cost, row.conversionValue, row.conversions];
});
sheet.getRange(sheet.getLastRow(), 1, dataToWrite.length, dataToWrite[0].length).setValues(dataToWrite);
}
If your data is actually an object of objects, not an array of objects iterate over Object.keys(dataSource) and return dataSource[key].clicks etc.
You may run into the danger of that being too much data to write in one go so you may want to write in bulks of 1k rows or something like that.
I am trying to scrape data of a website once day automatically. In Google Spreadsheets, i use the =ImportHTML() function to import data tables, and then I extract the relevant data with a =query(). These functions take between 10 and 30 seconds to complete calculation, every time I open the spreadsheet.
I use a scheduled Google Apps Script, to copy the data into a different sheet (where it is stored, so i can run statistics) every day.
My problem is that I am having trouble to make the script wait for the calculations to be finished, before the data is copied. The Result is that my script just copies the error Message "N/A".
I tried just adding a Utilities.sleep(60000);, but it didn't work.
Is it possible to create a loop, that checks for the calculation to finish? I tried this without success:
function checkForError() {
var spreadsheet = SpreadsheetApp.getActive();
var source = spreadsheet.getRange ("Today!A1");
if (source = "N/A") {
Utilities.sleep(6000);
checkForError();
} else {
moveValuesOnly();
}
}
Locks are for this. Look up lock services in the docs. Use a public lock.
Here's how I used Zig's suggestion (combined with my own check loop) to solve my similar problem:
// Get lock for public shared resourse
var lock = LockService.getPublicLock();
// Wait for up to 120 seconds for other processes to finish.
lock.waitLock(120000);
// Load my values below
// something like sheet.getRange("A1").setFormula('= etc...
// Now force script to wait until cell D55 set to false (0) before
// performing copy / pastes
var current = SpreadsheetApp.setActiveSheet(sheet.getSheets()[1]);
var ready = 1;
var count = 0;
while (true) {
// break out of function if D55 value has changed to zero or counter
// has hit 250
if (count >= 250) break;
// otherwise keep counting...
ready = current.getRange("D55").getValue();
if (ready == 0) {count = 400;}
Utilities.sleep(100);
++count;
}
// wait for spreadsheet to finish... sigh...
Utilities.sleep(200);
// Do my copy and pastes stuff here
// for example sheet.getRange("a1:b1").copyTo(sheet.getRange("a3"), {contentsOnly:true});
// Key cells are updated so release the lock so that other processes can continue.
lock.releaseLock();
// end script
return;
}
This has worked fantastic for me, stopped Google's sporadic service from ruining my work!
Thanks goes to Zig's suggestion!
is it allowed to chain time triggers in Google App script like this :
function doGet(e){ //first invocation by user, HTTP GET
if (e == null || e.parameters == null) {
return ContentService.createTextOutput("EMPTY");
}
saveGetParametersForUser(Session.getUser().getEmail(), e);
//trigger 10 seconds
var timeTrigger = ScriptApp.newTrigger("timeDrivenEvent").timeBased().after(10 * 1000).create();
}
function timeDrivenEvent() { //runs until there are some data in ScriptDB
Logger.log("INVOKED AT " + new Date());
removeAllPreviousTriggers(); //removes old time triggers
var somedata = loadTaskData({email: "" + Session.getUser().getEmail()});
var remainingData = processTaskData(somedata);
if(remainingData == null){
return; //we are finished here
}
removePreviousAndSaveRemainingTaskData(remainingData);
var timeTrigger = ScriptApp.newTrigger("timeDrivenEvent").timeBased().after(10 * 1000).create();
}
First invocation by user doGet()
Until all data are processed script invokes itself with 10 sec intervals (e.g. 2minutes of processing, 10 seconds nothing happens, then again 2 minutes of processing...)
size of processed data is ~ few kilobytes and processing time takes usually 1-2mins.
What happens to me that sometimes script is interrupted and data are not fully processed ! I am not getting any email alerts and nothing is in log or execution transcript - everything looks fine.
I am starting to think that maybe 10 seconds is quite quick to start script method but it`s in the API after all...
Any ideas ?
This is ONLY POSSIBLE solution how to chunk big task into smaller pieces as Google App Script cannot run for more than ~ 5-6 minutes (see quotas).
Having one periodic time trigger worked well as it was recommended in comments.
I just wonder why time trigger chaining didnt worked well ! What principle i did broke that Google App Script didnt like that.
Documentation : https://developers.google.com/apps-script/class_clocktriggerbuilder
method everyMinutes Sets the trigger to be created to fire on an interval of the passed in number of minutes which must be one of 1, 5, 10, 15 or 30.