I have an object called dataMapEN --> var dataMapEN = {};
This object has say 30000 entries.
I am trying to write them to Google spreadsheet using the following code:
var sheet = ss.getSheetByName("EN");
populateSheet(sheet, dataMapEN);
function populateSheet(sheet, dataSource){
for(item in dataSource){
sheet.appendRow([item, dataSource[item].clicks, dataSource[item].impressions, dataSource[item].cost, dataSource[item].conversionValue, dataSource[item].conversions]);
}
}
What I am seeing is that to write 200 entries the script takes about 2 minutes.
The script times out everytime I write to the sheet.
Is there a faster way to write to the sheet so that the script finishes in the 30min window ?
Thanks
Your script is so slow because it's doing 30k write calls to the spreadsheet.
Try putting everything into a 2d array first and then doing only one call.
Something like this (not tested because no example):
var sheet = ss.getSheetByName("EN");
populateSheet(sheet, dataMapEN);
function populateSheet(sheet, dataSource){
var dataToWrite = dataSource.map(funtion(row, item) {
return [item, row.clicks, row.impressions, row.cost, row.conversionValue, row.conversions];
});
sheet.getRange(sheet.getLastRow(), 1, dataToWrite.length, dataToWrite[0].length).setValues(dataToWrite);
}
If your data is actually an object of objects, not an array of objects iterate over Object.keys(dataSource) and return dataSource[key].clicks etc.
You may run into the danger of that being too much data to write in one go so you may want to write in bulks of 1k rows or something like that.
Related
I am working on a Google sheet script to manage stocks of items in a game, which is supposed to work as such:
People can make request to deposite or withdraw items using a Google form, which send all the infos, including what resource and in what amount, to a first "log" sheet. I then want a script to read these logs, and use them to update a different sheet, which show the actual stocks.
I should mention, there's about 800 different items to stock, and we like to move them around (up or down the list) because we're dumb.
So my idea what the have the script first retrieve the name of the item we made a request for, then try to match it in the stock sheet.
If it can, it should then add or substract the amount to the stock.
If it can't, it should just colour the log line in red so we can see it and redo the request.
My first problem is that I have no idea if a script in Gsheet can stay active for a long time, and the second is that I have even less of an idea how to properly retrieve a string of text and store it, then compare it with others, and that +800 times each time.
Thank you !
From the question
My first problem is that I have no idea if a script in Gsheet can stay active for a long time,
Google Apps Script have quotas. In this case, the corresponding quota is the execution time limit. For free accounts the limit is 6 minutes, for Workspace accounts the limit is 30 minutes.
and the second is that I have even less of an idea how to properly retrieve a string of text and store it, then compare it with others, and that +800 times each time.
Start by reading https://developers.google.com/apps-script/guides/sheets
Tl;Dr.
You need to learn the pretty basics of JavaScript.
You might use the Spreadsheet Service (Class SpreadsheetApp) or the Advanced Sheets Service, i.e.
/**
* Returns the values from the data range of the active sheet
*
*/
function readData(){
const sheet = spreadsheet.getActiveSheet();
const values = sheet.getDataRange().getValues();
return values;
}
You should decide where do you will store the values, then use JavaScript comparison expressions. You might use loops (for, while, do..while, or use Array methods like Array.prototype.forEach()
Here is an example how it could be done for simplest case, for manual firing of the functions.
Let's say you have the log sheet that look like this:
And your data sheet looks like this:
Here is the function that takes all items from the log sheet, sums them and put on the data sheet:
function add_all_items_from_log() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var log = ss.getSheetByName('log').getDataRange().getValues();
// put all data into the object {item1:q, item2:q, item3:q, ...etc}
var obj = {};
for (let [date, item, q] of log) {
if (item in obj) obj[item] += q; else obj[item] = q;
}
console.log(obj);
// convert the object into a 2d array [[item1,q], [item2,q], [item3,q], ...]
var array = Object.keys(obj).map(key => [key, obj[key]]);
console.log(array);
// put the array on the data sheet (starting from second row)
var sheet = ss.getSheetByName('data');
sheet.getRange(2,1,sheet.getLastRow()).clearContent();
sheet.getRange(2,1,array.length, array[0].length).setValues(array);
}
The result:
Here is the function that takes item from the last line of the log sheet and add the item to the data sheet:
function add_last_item_from_log() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
// get item from the last row of the log sheet
var [date, item, q] = ss.getSheetByName('log').getDataRange().getValues().pop();
console.log(date, item, q);
// get data from the data sheet
var sheet = ss.getSheetByName('data');
var [header, ...data] = sheet.getDataRange().getValues();
// put the data into the object {item1:q, item2:q, item3:q, ...etc}
var obj = {};
data.forEach(row => obj[row[0]] = row[1]);
console.log(obj);
// add the item to the object
if (item in obj) obj[item] += q; else obj[item] = q;
// convert the object into a 2d array [[item1,q], [item2,q], [item3,q], ...]
var array = Object.keys(obj).map(key => [key, obj[key]]);
console.log(array);
// put the array on the sheet (starting from second row)
var sheet = ss.getSheetByName('data');
sheet.getRange(2,1,sheet.getLastRow()).clearContent();
sheet.getRange(2,1,array.length, array[0].length).setValues(array);
}
Here is my sheet.
You can run these function manually from Text Editor. Just to see how it works. But actually, as far as I can tell, you better to run the last function (or its variant) automatically every time the log sheet is updated from the Form submit. It can be done with the trigger onFormSubmit().
And this is a simplest case. If you have 800+ items and many columns the code may require some optimizations.
I've setup a google app script that would be triggered from an external system. This script would fetch the details from the third party system and add them to google sheet row.
function doPost(request) {
try{
var jsonString = request.postData.getDataAsString(); //get the request from KF as JSON String
setLog("\n postData*********************"+jsonString+"************************************* \n");
setLog("before the row append");
ss.appendRow([jsonString["Name"], jsonString["Age"], jsonString["Contact"]]);
setLog("After the row append");
var returnJson = '{"status": "success"}';
//used to send the return value to the calling function
setLog("/n returnJson****************************"+returnJson+"************************************* /n")
return ContentService.createTextOutput(returnJson).setMimeType(ContentService.MimeType.JSON);
}
There's absolutely no errors or warnings, but somehow it keeps adding the blank rows into the sheet.
Note: setLog() is a function where I print the values into google doc for debugging.
Maybe the reason your script is not working has to do with the value of jsonString.
I could not find any reference to request.postData.getDataAsString() inside GAS Documentation, so maybe you are trying to call a method on an object which does not support it, which would not raise an Error, but would return undefined.
One quick way to debug this would be to LOG the value (using your custom function or Logger.log(jsonString)) BEFORE you call .appendRow(). Then, you can verify if your variable has the value you expect it to have.
On the other hand, my suggestion is to use this method:
var jsonString = JSON.parse(request.postData.contents) //Gets the content of your request, then parses it
This method is present in the Documentation, and has been consistently working on all of my projects.
I think you should sort the coulmns with google app script. Write this code after ss.appendRow. The column will be sorted and all blank rows gets down.
// Sorts the sheet by the first column, ascending
ss.sort(1)
or if errors try this one also
var fl = SpreadsheetApp.getActiveSpreadsheet();
var sheet = fl.getSheets()[0];
fl.sort(1)
I need to change some formulas at the same cells at an specific sheet (LISTAFINAL) present in a great number of spreadsheets, these one located at the same folder. But it stops at Google script time limit of 6 minutes, making changes only in 9 spreadsheets, and comes the message: Erro - Exceeded maximum execution time.
My goals:
I would like to know if there's any way to or speed up this process or make changes in a bigger number of spreadsheets or both. Here is the code:
function validacao(){
var folder = DriveApp.getFolderById("FOLDER ID");
var x = folder.getFilesByType(MimeType.GOOGLE_SHEETS);
while (x.hasNext()) {
SpreadsheetApp.open(x.next()).getSheets().forEach(sheet => {
sheet.getRange('LISTAFINAL!F5:F15').activate();
sheet.getRange('LISTAFINAL!F5').setValue('=ALUNO1!$F$167');
sheet.getRange('LISTAFINAL!F6').setValue('=ALUNO2!$F$167');
sheet.getRange('LISTAFINAL!F7').setValue('=ALUNO3!$F$167');
sheet.getRange('LISTAFINAL!F8').setValue('=ALUNO4!$F$167');
sheet.getRange('LISTAFINAL!F9').setValue('=ALUNO5!$F$167');
sheet.getRange('LISTAFINAL!F10').setValue('=ALUNO6!$F$167');
sheet.getRange('LISTAFINAL!F11').setValue('=ALUNO7!$F$167');
sheet.getRange('LISTAFINAL!F12').setValue('=ALUNO8!$F$167');
sheet.getRange('LISTAFINAL!F13').setValue('=ALUNO9!$F$167');
sheet.getRange('LISTAFINAL!F14').setValue('=ALUNO10!$F$167');
sheet.getRange('LISTAFINAL!F15').setValue('=ALUNO11!$F$167');
});
}
}
Explanation:
You iterate over all sheets for every spreadsheet file. Your goal is to just get a single sheet and put the formulas in specific cells. Therefore, you can get rid of the forEach loop.
It is a best practice to work with arrays instead of iteratively using multiple google apps script functions.
For example, in your code you are using getRange and setValue 11 times. If you store the formula values in an array you will be able to store them by using only a single getRange and setValues.
Solution:
function validacao(){
const folder = DriveApp.getFolderById("FOLDER ID");
const x = folder.getFilesByType(MimeType.GOOGLE_SHEETS);
const formulas = Array(11).fill().map((_, i) => [`=ALUNO${i+1}!$F$167`]);
while (x.hasNext()) {
let ss_target = SpreadsheetApp.open(x.next());
let sh = ss_target.getSheetByName("LISTAFINAL");
sh.getRange('F5:F15').setValues(formulas);
}
}
I have a multifaceted question.
I'm attempting to have a script which pulls the tab names, and uses that info to pull specific cells from each one (tabs for the doc change frequently) in order to create a Master Sheet. The Master Sheet is meant to display all open action items, and filter out any closed items / blank rows. The script I have so far works, but it pulls all empty rows from each tab - which I don't want.
Here's 1 of the 2 current scripts I have for the master sheet:
function onEdit(e) {
//set variable
const masterSheet = "Open Action Items";
const mastersheetFormulaCell = "E2";
const ignoreSheets = ["Template", "Blank"];
const dataRange = "I2:K";
const checkRange = "J2:J"
//end set variables
const ss = SpreadsheetApp.getActiveSpreadsheet();
ignoreSheets.push(masterSheet);
const allSheets = ss.getSheets();
const filteredListOfSheets = allSheets.filter(s => ignoreSheets.indexOf(s.getSheetName()) == -1);
let formulaArray = filteredListOfSheets.map(s => `iferror(FILTER('${s.getSheetName()}'!${dataRange}, NOT(ISBLANK('${s.getSheetName()}'!${checkRange}))),{"","",""})`);
let formulaText = "={" + formulaArray.join(";") + "}";
ss.getSheetByName(masterSheet).getRange(mastersheetFormulaCell).setFormula(formulaText);
}
The other part of this is another script that has been running ok when it was the only script running, but has since stopped working when the other script were introduced. This script added a checkbox to column C based on criteria in column B.
Here's that script:
function onEdit(e) {
if (e.range.getColumn() == 2) {
var sheet = e.source.getActiveSheet();
if (e.value === "Tech Note" ||
e.value === "Intake Process")
sheet.getRange(e.range.getRow(),3).insertCheckboxes();
else
sheet.getRange(e.range.getRow(),3).removeCheckboxes();
}
}
Here's a sample sheet
The "Open Action Items" tab is the master sheet the script is meant to update. It should list all the open items from other tabs (explained below)
The "Copy of E3-O Case Notes" is a tab which is the basis of what every tab will eventually look like. Columns F-K of this tab pull open items from A-E. There may likely be a more efficient way to sort this whole sheet...
Any help appreciated, thank you!
I'll address the second question first, as it's a more fundamental problem.
The other part of this is another script that has been running ok when it was the only script running, but has since stopped working when the other script were introduced.
In the script project attached to your sample, you have 3 files which each define an onEdit() function. This is problematic because each time you define onEdit() you're redefining the same identifier. The project only has a single global scope, so there can only be 1 onEdit() function defined, regardless of how many files your project contains.
Essentially, this is equivalent to what you've defined in your project:
function onEdit(e) {
console.log("onEdit #1");
}
function onEdit(e) {
console.log("onEdit #2");
}
function onEdit(e) {
console.log("onEdit #3");
}
onEdit();
Running the above snippet will only execute the last definition of onEdit().
To accomplish what you're trying to do, you can instead define unique functions for all the actions you want to perform and then, in a single onEdit() definition, you can call those functions. Something like:
function editAction1(e) {
console.log("edit action #1");
}
function editAction2(e) {
console.log("edit action #2");
}
function editAction3(e) {
console.log("edit action #3");
}
function onEdit(e) {
editAction1(e);
editAction2(e);
editAction3(e);
}
onEdit();
When defining an onEdit() trigger, you really want to optimize it so that it can complete its execution as quickly as possible. From the Apps Script best practices, you want to pay particular attention to "Minimize calls to other services" and "Use batch operations".
A few specific tips for you:
Avoid repeated calls to the same Apps Script API (e.g. Sheet.getName()). Instead, run it once and store the value in local variable.
As much as possible, avoid making Apps Script API calls within loops and in callback functions passed to methods such as Array.prototype.filter() and Array.prototype.map().
When you do need to loop through data, especially when Apps Script API calls are involved, minimize the number of times you iterate through the data.
With onEdit() triggers, try to structure the logic so that you identify cases where you can exit early (similar to how you perform the column check before going ahead with manipulating checkboxes). I doubt you actually need to iterate through all of the sheets and update the "Open Action Items" formula for every single edit. If I'm interpreting the formula properly, it's something that should only be done when sheets are added or removed.
Finally, to address the blank rows in your formula output, instead of using SORT() to group the blank rows you can use QUERY() to actually filter them out.
Something like:
=QUERY({ <...array contents...> }, "select * where Col1 is not null")
Note that when using QUERY() you need to be careful that the input data is consistent in regards to type. From the documentation (emphasis mine):
In case of mixed data types in a single column, the majority data type
determines the data type of the column for query purposes. Minority
data types are considered null values.
In your sample sheet, a lot of the example data varies and doesn't match what you'd actually expect to see (e.g. "dghdgh" as a value in a column meant for dates). This is important given the warning above... when you have mixed data types for a given column (i.e. numbers and strings) whichever type is least prevalent will silently be considered null.
After taking a closer, end-to-end look at your sample, I noticed you're performing a very convoluted series of transformations (e.g. in the data sheets there's the hidden "D" column, the QUERY() columns to the right of the actual data, etc.). This all culminates in a large set of parallel QUERY() calls that you're generating via your onEdit() implementation.
This can all be made so much simpler. Here's a pass at simplifying the Apps Script code, which is dependent on also cleaning up the spreadsheet that it's attached to.
function onEdit(e) {
/*
Both onEdit actions are specific to a subset of the sheets. This
regular expression is passed to both functions to facilitate only
dealing with the desired sheets.
*/
const validSheetPattern = /^E[0-9]+/;
updateCheckboxes(e, validSheetPattern);
updateActionItems(e, validSheetPattern);
}
function updateCheckboxes(e, validSheetPattern) {
const sheet = e.range.getSheet();
// Return immediately if the checkbox manipulation is unnecessary.
if (!validSheetPattern.exec(sheet.getName())) return;
if (e.range.getColumn() != 2) return;
const needsCheckbox = ["Tech Note", "Intake Process"];
const checkboxCell = sheet.getRange(e.range.getRow(), 3);
if (needsCheckbox.includes(e.value)) {
checkboxCell.insertCheckboxes();
} else {
checkboxCell.removeCheckboxes();
}
}
function updateActionItems(e, validSheetPattern) {
const masterSheetName = "Open Action Items";
const dataLocation = "A3:E";
/*
Track the data you need for generating formauls in an array
of objects. Adding new formulas should be as simple as adding
another object here, as opposed to duplicating the logic
below with a growing set of manually indexed variable names
(e.g. cell1/cell2/cell3, range1/range2/range3, etc.).
*/
const formulas = [
{
location: "A3",
code: "Tech Note",
},
{
location: "E3",
code: "Intake Process",
},
];
const masterSheet = e.source.getSheetByName(masterSheetName);
const sheets = e.source.getSheets();
/*
Instead of building an array of QUERY() calls, build an array of data ranges that
can be used in a single QUERY() call.
*/
let dataRangeParts = [];
for (const sheet of sheets) {
// Only call getSheetName() once, instead of multiple times throughout the loop.
const name = sheet.getSheetName();
// Skip this iteration of the loop if we're not dealing with a data sheet.
if (!validSheetPattern.exec(name)) continue;
dataRangeParts.push(`'${name}'!${dataLocation}`);
}
const dataRange = dataRangeParts.join(";");
for (const formula of formulas) {
/*
And instead of doing a bunch of intermediate transformations within the sheet,
just query the data directly in this generated query.
*/
const query = `SELECT Col5,Col1,Col4 WHERE Col2='${formula.code}' AND Col3=FALSE`;
const formulaText = `IFERROR(QUERY({${dataRange}},"${query}"),{"","",""})`;
formula.cell = masterSheet.getRange(formula.location);
formula.cell.setFormula(formulaText);
}
}
Here's a modified sample spreadsheet that you can reference.
The one concession I made is that the data sheets still have a "Site Code" column, which is automatically populated via a formula. Having all the data in the range(s) you feed into QUERY() makes the overall formulas for the "Open Action Items" sheet much simpler.
Im currently using Google Apps Script to implement a viewer of a supply chain database.
To synchronize the viewer with the current database (a google spreadsheet) I import the values and all the formatting it into a new sheet, this means the viewer basically is a copy of the current database.
However executing the script always takes something about 1 minute in time. I tried to find the issue with logging some debug messages at various positions in the code.
At first it seemed that the line with Viewer.setFrozenRows(1); (which is strange since I actually only freeze the first row) was the issue, however when commenting out this line the line afterwards (Viewer.setFrozenColumns(Database.getFrozenColumns());) seemed to be the issue.
Unfortuanetly I'm not able to share the database sheet with you, but maybe somebody can already spot the issue from the code.
Some additional Info: The database sheet has 1300 rows and 100 columns, and I added a picture of the log of the current code below.
function LoadViewer(view) {
Logger.log("LoadViewer Start");
if (view == null) {
view = 0;
}
var Database = SpreadsheetApp.openByUrl('[SHEET_URL].getSheetByName('Database');
var Viewer = SpreadsheetApp.getActiveSpreadsheet().getSheets()[0];
var numberOfColms = Database.getLastColumn();
var numberOfRows = Database.getLastRow();
var rules = Database.getConditionalFormatRules();
var headerRowHeight = Database.getRowHeight(1);
var dataRowHeight = Database.getRowHeight(2);
var Values = Database.getRange(1, 1, numberOfRows, numberOfColms).getValues();
Logger.log("Declarations Finished");
Viewer.getRange(1, 1,numberOfRows,numberOfColms).setValues(Values);
if(!Viewer.getRange(1, 1,numberOfRows,numberOfColms).getFilter())
Viewer.getRange(1, 1,numberOfRows,numberOfColms).createFilter();
Viewer.setConditionalFormatRules(rules);
Viewer.getRange(1, 1, 1, numberOfColms).setFontWeight('bold');
Viewer.autoResizeColumns(1, numberOfColms);
Viewer.setRowHeight(1, headerRowHeight);
Logger.log("1st Half of functions finished");
Viewer.setRowHeights(2, numberOfRows-1, dataRowHeight);
Logger.log("Freeze Rows");
//Viewer.setFrozenRows(1);
Logger.log("Freeze Columns");
Viewer.setFrozenColumns(Database.getFrozenColumns());
Logger.log("Loop Start");
for(var i = 1; i<=numberOfColms; i++){
Viewer.setColumnWidth(i, Database.getColumnWidth(i));
}
Logger.log("Loop End");
Viewer.getRange(1, 1,1,numberOfColms).setVerticalAlignment('middle').setWrap(true);
Logger.log("Load Viewer End");
}
Two optimization points I can see for your code:
Requests to the any external service including SpreadsheetApp make your code slow - see Best Practices.
Thus, making calls to a SpreadsheetApp method within a for loop will slow your code down.
You will be able to accelerate your code by replacing multiple setColumnWidth() requests within the loop by a single setColumnWidths(startColumn, numColumns, width) - avoiding iteration.
Log the number of columns and rows in your sheet.
A common problem is that the sheet contains a significant amount of empty rows and columns that increase your detected data range and consequently apply the subsequent calls to a bigger range than necessary.
If that you case - either delete the spare rows and columns manually, or use getNextDataCell() instead of getLastRow() or getLastColumn()