copyToRange (and similar) execution at end of script? - google-apps-script

I am working on a kinda large sheet which I would like to update on a regular basis (weekly) via Google Apps Scripts.
Every week I need to add a new column at the "end" (lastDataColumn) of my sheet and then move the last two colums (with Fomulas to calculate weekly relative changes) to the "end" (ie move them one column to the right). This leaves me with a blank column addressed with lastDataColumn - 2. This is where the new report data will go.
I have two functions.
copyCols and getReports.
They both work fine on their own, so copyCols creates a new empty column at position lastDataColumn - 2 using the method explained above - and getReports fetches report Data from Analytics, third party APIs and other sheets and then writes these values in the column at position lastDataColumn - 2.
However, if I group these two functions in let's say my main function which I then want to trigger on a 7-day basis, copyCols seems to only execute to the point of creating a new empty column. Then getReports executes fully and writes all data in lastDataColumn - 2. But no columns were moved, so getReports overwrites last weeks data. After executing everything from getReports copyCols starts moving the rows (ie copying).
This leaves me with a duplicate column of lastDataColumn - 3 (which should have last weeks data, but was overwritten with this weeks data because it was still in lastDataColumn - 2 before the execution of getReports) in lastDataColumn - 2.
To clarify: Executing copyCols and getReports afterwards (each on it's own) works perfectly fine.
Is Google Apps Script threadless? If so why does the problem described happen? Do "bulk" operations (like copying ranges) execute at the end of the script?
Code:
var today = new Date();
var start = new Date(today.getTime() - 7 * 24 * 60 * 60 * 1000);
var end = new Date(today.getTime() - 1 * 24 * 60 * 60 * 1000);
var dateString=Utilities.formatDate(start, Session.getScriptTimeZone(),'dd.MM.')+'-'+Utilities.formatDate(end, Session.getScriptTimeZone(),'dd.MM.')
var nc=2 //num of cols with growth rates to move
function main() {
copyCols();
getReports();
}
function copyCols(){
var ss=SpreadsheetApp.openById('1dSpy7teczLwViKbfr-VjfQRuOq3iaRfSm3LghFldjZk')
var sh_DB=ss.getSheetByName('data')
var w=sh_DB.getLastColumn(); //width
var h = sh_DB.getLastRow(); //heigth
// insert new column
sh_DB.insertColumnAfter(w);
// copy last n cols to next col
for (i=0;i<=nc;i++){
sh_DB.getRange(1,w-i,h,1).copyTo(sh_DB.getRange(1,w-i+1,h,1));
}
sh_DB.getRange(1,w-nc+1).setValue(dateString);
}
function getReports(){
var sh_DB=SpreadsheetApp.getActiveSpreadsheet().getSheetByName('data')
var w=sh_DB.getLastColumn(); //width
var h = sh_DB.getLastRow(); //heigth
dc=sh_DB.getRange(1,w-nc); //lastDataColumn
data = [50, 60, 870, 2];
report = {'rows':[2,3,4,5]};
for (i in data){
sh_DB.getRange(report['rows'][i],w-nc).setValue(data[i]);
}
}
Thank you for any help provided.

Add SpreadsheetApp.flush() between copyCols() and getReports() in order to tell to the Google Apps Script engine to apply the changes made by the first before running the second.

Related

Can Sheet-A reference a total that might come from 2 different Sheets (B&C), but display the total from the Sheet (A or B) currently being used?

I would appreciate it if the experts can tell me whether or not it will be possible to do what I would like to!
Here at the University of Johannesburg Library, I have put in place a simple system in order to count and capture students detail when entering & exciting the Libraries.
(In order to control numbers due to Covid)
So I have 3 separate Google Excel files.
File-1 with Sheet-1 runs on a PC where students Enter the Library
File-2 with Sheet-1 runs on a PC where students Exit the Library
File-3 with Sheet-1 runs on a big 65” screen above Circulation – displaying the total number of Students in the Library (simple formula that minus File-2’s total from that of File-1)
Then I have 2 scripts running on Files 1 & 2:
Script-1 (TimeStamp) to trigger on Edit, so that I capture date/time when a student swipe his/her card and enter or exit the Library.
Script-2 (Copy & Clean) that runs 1AM to copy the previous day’s data to backup files and then clean Files-1 & 2
Problem:
For some reason this Copy & Clean script will run fine for a week and then suddenly stop.
This means that the next day, Files-1 & 2 will not be fresh/clean.
So I have created a 2nd sheet for files-1 & 2 that I called “Emergency” on which staff can then click and students can scan their cards.
(This will give me time to clean Sheets-1 manually)
However, if staff uses the “Emergency” sheets, obviously File-3 will still display the numbers coming from Sheets-1.
Question:
Is there a way that one can program Sheet-3 to display the totals coming from the “Emergency” sheets, when they are in used?
And then again from Sheets-1 when they are normally in use?
Yes, I use a Time-Based Trigger for the "myCopySheet" script that runs between 01h00 and 02h00 AM
Script to copy the content of the "Enter" sheet to a backup sheet and then to clear the "Enter" sheet
Error:
"myCopySheet":
function myCopySheet() {
// source spreadsheets
var sourceSpreadSheet = SpreadsheetApp.getActiveSpreadsheet();
// Get previous days date for the new tab
var MILLIS_PER_DAY = 1000 * 60 * 60 * 24;
var now = new Date();
var yesterday = new Date(now.getTime() - MILLIS_PER_DAY);
var date = Utilities.formatDate(yesterday, "GMT+2", "yyyy-MM-dd")
var srcSheet = sourceSpreadSheet.getSheets()[0];
// Change ID to destination Sheet - One for Entry and one for Exit
var destSpreadsheet =
SpreadsheetApp.openById('15bev166RYSHNlWhnLIBHNtLVMbQyWjzJDOW0R-NBIOg');
// Copy to new spreadsheet sheet
srcSheet.copyTo(destSpreadsheet).setName(date);
// Clear Sheet Column A
var firstCell = srcSheet.getRange("A2");
var numRows = srcSheet.getLastRow()+1;
var range = srcSheet.getRange(firstCell.getRow(), firstCell.getColumn(), numRows);
range.clear();
// Clear Sheet Column B
var firstCell = srcSheet.getRange("B2");
var numRows = srcSheet.getLastRow()+1;
var range = srcSheet.getRange(firstCell.getRow(), firstCell.getColumn(), numRows);
range.clear();
}
The error message Service Spreadsheets failed while accessing document with id ID you are receiving might in fact be related to this known issue on Google's side.
I suggest you star the issue here as all the updates regarding this will be posted there.

Alternatives to google finance stock prices

I'm trying to make a Stock portfolio tracker for my investments. I already understand how to use google finance but the 20 min delay on prices is a drag. I've updated the recalculation for my sheets to on change and every minute but the delay will still happen. So my real question is if and how could I pull a stock price from the Nasdaq into google sheets for a more real time feel.
What you can try is having script and have it triggered every minute.
I tried fetching from NASDAQ but it takes too long. I used WSJ instead but the issue is it randomly errors out due to something I was never able to point out. Thus I added a column to identify when the data was last updated. Don't worry though, it will be updated within the next run.
Code:
function getPrice() {
var sheet = SpreadsheetApp.getActiveSheet();
// tickers will be checked on column A starting 2nd row
var values = sheet.getRange("A2:A" + sheet.getLastRow()).getValues().flat();
sheet.getRange("A1:C1").setValues([["Tickers", "Price", "Last Updated"]]);
values.forEach(function (ticker, index){
var url = "https://www.wsj.com/market-data/quotes/" + ticker;
Utilities.sleep(2000);
var html = UrlFetchApp.fetch(url).getContentText();
Utilities.sleep(2000);
var price = html.match(/quote_val">([\d ,.]+)/);
// since it randomly errors out on [1], check if price is null
// add date and time too on column C to confirm when was the data last updated
if(price){
sheet.getRange(index + 2, 2).setValue(price[1].trim());
sheet.getRange(index + 2, 3).setValue(new Date());
}
});
// call to remove existing triggers and create another one
createTrigger();
}
function createTrigger(){
// Delete all existing triggers before creating one
// Ensuring none will exist before creating trigger.
var triggers = ScriptApp.getProjectTriggers();
triggers.forEach(function (trigger){
ScriptApp.deleteTrigger(trigger);
});
// Create trigger after every run which is per minute
ScriptApp.newTrigger('getPrice')
.timeBased()
.after(60 * 1000)
.create();
}
Sample Output:
Note:
There are 1440 minutes in a day, and quota for Url Fetch calls is 20000 a day, so you must only have at most 13 tickers as when you reach the quota, it will not behave as expected. If you want more tickers, you need to adjust the frequency. (e.g. update every 2 minutes will allow you to have at most 27 tickers)
Resource:
Quota
GOOGLEFINANCE recalculation is set to 20 minutes and cant be re-set
you will need to either scrape it from somewhere (yahoo finance, coinmarketcap) or like MK mentioned you will need to pay for (API) it because free plans may not be enough for you. see: https://coinmarketcap.com/api/pricing/

Google Apps Script execution time issue

Im currently using Google Apps Script to implement a viewer of a supply chain database.
To synchronize the viewer with the current database (a google spreadsheet) I import the values and all the formatting it into a new sheet, this means the viewer basically is a copy of the current database.
However executing the script always takes something about 1 minute in time. I tried to find the issue with logging some debug messages at various positions in the code.
At first it seemed that the line with Viewer.setFrozenRows(1); (which is strange since I actually only freeze the first row) was the issue, however when commenting out this line the line afterwards (Viewer.setFrozenColumns(Database.getFrozenColumns());) seemed to be the issue.
Unfortuanetly I'm not able to share the database sheet with you, but maybe somebody can already spot the issue from the code.
Some additional Info: The database sheet has 1300 rows and 100 columns, and I added a picture of the log of the current code below.
function LoadViewer(view) {
Logger.log("LoadViewer Start");
if (view == null) {
view = 0;
}
var Database = SpreadsheetApp.openByUrl('[SHEET_URL].getSheetByName('Database');
var Viewer = SpreadsheetApp.getActiveSpreadsheet().getSheets()[0];
var numberOfColms = Database.getLastColumn();
var numberOfRows = Database.getLastRow();
var rules = Database.getConditionalFormatRules();
var headerRowHeight = Database.getRowHeight(1);
var dataRowHeight = Database.getRowHeight(2);
var Values = Database.getRange(1, 1, numberOfRows, numberOfColms).getValues();
Logger.log("Declarations Finished");
Viewer.getRange(1, 1,numberOfRows,numberOfColms).setValues(Values);
if(!Viewer.getRange(1, 1,numberOfRows,numberOfColms).getFilter())
Viewer.getRange(1, 1,numberOfRows,numberOfColms).createFilter();
Viewer.setConditionalFormatRules(rules);
Viewer.getRange(1, 1, 1, numberOfColms).setFontWeight('bold');
Viewer.autoResizeColumns(1, numberOfColms);
Viewer.setRowHeight(1, headerRowHeight);
Logger.log("1st Half of functions finished");
Viewer.setRowHeights(2, numberOfRows-1, dataRowHeight);
Logger.log("Freeze Rows");
//Viewer.setFrozenRows(1);
Logger.log("Freeze Columns");
Viewer.setFrozenColumns(Database.getFrozenColumns());
Logger.log("Loop Start");
for(var i = 1; i<=numberOfColms; i++){
Viewer.setColumnWidth(i, Database.getColumnWidth(i));
}
Logger.log("Loop End");
Viewer.getRange(1, 1,1,numberOfColms).setVerticalAlignment('middle').setWrap(true);
Logger.log("Load Viewer End");
}
Two optimization points I can see for your code:
Requests to the any external service including SpreadsheetApp make your code slow - see Best Practices.
Thus, making calls to a SpreadsheetApp method within a for loop will slow your code down.
You will be able to accelerate your code by replacing multiple setColumnWidth() requests within the loop by a single setColumnWidths(startColumn, numColumns, width) - avoiding iteration.
Log the number of columns and rows in your sheet.
A common problem is that the sheet contains a significant amount of empty rows and columns that increase your detected data range and consequently apply the subsequent calls to a bigger range than necessary.
If that you case - either delete the spare rows and columns manually, or use getNextDataCell() instead of getLastRow() or getLastColumn()

Google Apps Script extract specific columns from sheet and put into new sheet using filter & map -- slow

Thanks in advance for your help. I'm new to apps script.
I have a gsheet with 98 columns and 25000 rows. All I want to do is copy 24 of the 98 columns to a new sheet.
Currently I am using filter & map, and it works:
var data = sourceSheet.getDataRange().getValues(); //read all columns & rows into array
var filterData = data.filter(function(e, j){return j > 0 && e}).map(function(e){return [e[88], e[14], e[13], e[4], e[17], e[87], e[91], e[48], e[57], e[31], e[89], e[82], e[70], e[97], e[47], e[30], e[72], e[71], e[67], e[34], e[33], e[00], e[38], e[39]]}); //extract just the columns I want into a new array
but that 2nd line takes almost an hour to execute! I presume because it is processing every element 1-by-1, even though it is just to return each one.
I haven't tried getValue'ing and setValue'ing columns one at a time because everything I read says limit external calls and do everything in memory. And I can't imagine pushing elements 1-by-1 would be faster than filtering.
Suggestions for faster execution?
function doCopy(SpreadID, OrgSheet, DestSheet, OrgRange, Sql, DestCell) {
var mySpread = SpreadsheetApp.openById(SpreadID);
var myQry = '=QUERY(' + OrgSheet + "!" + OrgRange + ',\"'+ Sql + '\")';
var myDestSheet = mySpread.getSheetByName(DestSheet);
myDestSheet.getRange(DestCell).setFormula(myQry);
}
Sample to call, but the destination sheet must be blank:
doCopy(spreadsheetId,"Sheet1", "Sheet2", "A:G", "Select A, C, F", "A1");
Another way to look at the problem which might be more time-effective and easy to implement is the following :
Duplicate the original sheet into a new one with copyTo(spreadsheet) (https://developers.google.com/apps-script/reference/spreadsheet/sheet#copytospreadsheet). You might also want to check this post for more information about creating a sheet into the same spreadsheet (Google Script to Duplicate Sheet that is Not Active)
Delete the columns you don't want to keep with deleteColumn(columnPosition) or deleteColumns(columnPosition, howMany)(see https://developers.google.com/apps-script/reference/spreadsheet/sheet#deletecolumncolumnposition).
You might also want to start deleting columns from the right side of the sheet to make the implementation easier.
Let me know if it helps.

Enter a break into a formula [duplicate]

I am working on a kinda large sheet which I would like to update on a regular basis (weekly) via Google Apps Scripts.
Every week I need to add a new column at the "end" (lastDataColumn) of my sheet and then move the last two colums (with Fomulas to calculate weekly relative changes) to the "end" (ie move them one column to the right). This leaves me with a blank column addressed with lastDataColumn - 2. This is where the new report data will go.
I have two functions.
copyCols and getReports.
They both work fine on their own, so copyCols creates a new empty column at position lastDataColumn - 2 using the method explained above - and getReports fetches report Data from Analytics, third party APIs and other sheets and then writes these values in the column at position lastDataColumn - 2.
However, if I group these two functions in let's say my main function which I then want to trigger on a 7-day basis, copyCols seems to only execute to the point of creating a new empty column. Then getReports executes fully and writes all data in lastDataColumn - 2. But no columns were moved, so getReports overwrites last weeks data. After executing everything from getReports copyCols starts moving the rows (ie copying).
This leaves me with a duplicate column of lastDataColumn - 3 (which should have last weeks data, but was overwritten with this weeks data because it was still in lastDataColumn - 2 before the execution of getReports) in lastDataColumn - 2.
To clarify: Executing copyCols and getReports afterwards (each on it's own) works perfectly fine.
Is Google Apps Script threadless? If so why does the problem described happen? Do "bulk" operations (like copying ranges) execute at the end of the script?
Code:
var today = new Date();
var start = new Date(today.getTime() - 7 * 24 * 60 * 60 * 1000);
var end = new Date(today.getTime() - 1 * 24 * 60 * 60 * 1000);
var dateString=Utilities.formatDate(start, Session.getScriptTimeZone(),'dd.MM.')+'-'+Utilities.formatDate(end, Session.getScriptTimeZone(),'dd.MM.')
var nc=2 //num of cols with growth rates to move
function main() {
copyCols();
getReports();
}
function copyCols(){
var ss=SpreadsheetApp.openById('1dSpy7teczLwViKbfr-VjfQRuOq3iaRfSm3LghFldjZk')
var sh_DB=ss.getSheetByName('data')
var w=sh_DB.getLastColumn(); //width
var h = sh_DB.getLastRow(); //heigth
// insert new column
sh_DB.insertColumnAfter(w);
// copy last n cols to next col
for (i=0;i<=nc;i++){
sh_DB.getRange(1,w-i,h,1).copyTo(sh_DB.getRange(1,w-i+1,h,1));
}
sh_DB.getRange(1,w-nc+1).setValue(dateString);
}
function getReports(){
var sh_DB=SpreadsheetApp.getActiveSpreadsheet().getSheetByName('data')
var w=sh_DB.getLastColumn(); //width
var h = sh_DB.getLastRow(); //heigth
dc=sh_DB.getRange(1,w-nc); //lastDataColumn
data = [50, 60, 870, 2];
report = {'rows':[2,3,4,5]};
for (i in data){
sh_DB.getRange(report['rows'][i],w-nc).setValue(data[i]);
}
}
Thank you for any help provided.
Add SpreadsheetApp.flush() between copyCols() and getReports() in order to tell to the Google Apps Script engine to apply the changes made by the first before running the second.