I am trying to import a CSV of approx. 7500 lines from a GMail attachment into Google Sheets and it is taking longer than the allowed 6 minutes to execute.
I am following this tutorial: https://developers.google.com/apps-script/articles/docslist_tutorial#section2 and my code is listed below
The section consuming all the time is the pushing of data into the new sheet:
// Push data into the sheet
for ( var n=0, lenCsv=csvData.length; n<lenCsv; n++ ) {
newsheet.getRange(n+1, 1, 1, csvData[n].length).setValues(new Array(csvData[n]));
}
Is there any way I can improve the performance of this and make it faster?
Thanks in advance.
My code is listed below.
var attachmentData = attachments[k].getDataAsString();
var attachmentClean = attachmentData.split(' ').join(',');
var attachmentCleanA = attachmentClean.split(',');
var csvData = Utilities.parseCsv(attachmentCleanA);
var SheetName = String(csvData[1]).substring(5,7)+ "-" + String(csvData[1]).substring(0,4);
Logger.log("SheetName:"+SheetName);
ss = SpreadsheetApp.openById('XXXXXXXXX');
var sheet = ss.getSheetByName(SheetName);
if (sheet == null){
var newsheet = ss.insertSheet(SheetName);
}else{
sheet.clearContents();
var newsheet = sheet;
}
Logger.log("Entering "+csvData.length+" rows");
// Push data into the sheet
for ( var n=0, lenCsv=csvData.length; n<lenCsv; n++ ) {
newsheet.getRange(n+1, 1, 1, csvData[n].length).setValues(new Array(csvData[n]));
}
This line:
var csvData = Utilities.parseCsv(attachmentCleanA);
Creates a two dimensional array. Google Documentation - parseCsv
If your csvData variable truly is a 2D array, then all you need to do is use this line of code without the for loop:
newsheet.getRange(1, 1, csvData.length, csvData[0].length).setValues(csvData);
The range starts in row one, column one, and sets a range that is the length of the outer array of the csvData, and the number of columns to the number of elements in the first inner array.
Note that if your data has inner arrays of different lengths, setValues() will produce an error.
If all of your inner arrays are not the same length, then you can not avoid looping through every inner array. But, I don't know what your data looks like. If each line of data in your CSV has the same number of vales in the row, then you don't need the for loop.
So, improving the part of the code that is taking all the time, is totally dependent upon whether the rows are all the same length or not.
You shouldn't need the new Array() part in the setValues() method. If the csvData array truly is a good 2D array, then it's already an array. You don't need to create another array from something that's already an array.
Related
I'm using Google Sheets and I have this code below to copy the last row (via Form On Submit trigger) onto another sheet (not tab). I'm not sure how to copy over the last row and skip the data in Column C. It's currently copying the entire row to another sheet successfully, fyi. Any help would be appreciated, thank you!
function CopyToAnotherSheet() {
// Delaying this script so that there is enough time to get the Google Doc URL before it starts to fill out the template file
Utilities.sleep(10000); // 10 second delay
// Get Source Spreadsheet
var source = SpreadsheetApp.getActiveSpreadsheet();
// Get Source Sheet from Spreadsheet
var source_sheet = source.getActiveSheet();
// Get Last Row
var lastRow = source_sheet.getLastRow();
// Get Last Column
var lastColumn = source_sheet.getLastColumn();
// Get Last Row of Data
var lastRowOfData = source_sheet.getRange(lastRow, 1, 1, lastColumn).getValues();
// Creates a one dimensional array
var oneD_array = lastRowOfData.join().split(",");
// Get the Value of the Manufacturer Cell
var cellValue = source_sheet.getRange(lastRow,3).getValues();
// Copy Last Row to First Sheet
if ( cellValue == "First" ) {
var target = SpreadsheetApp.openById("xxxyyyzzz");
var target_sheet = target.getSheetByName("First");
target_sheet.appendRow(oneD_array);
}
// Copy Last Row to Second Sheet
if ( cellValue == "Second" ) {
var target = SpreadsheetApp.openById("aabbcc");
var target_sheet = target.getSheetByName("Second");
target_sheet.appendRow(oneD_array);
}
}
One option you can do is to remove column C value in your 1-d array. Instead of skipping it.
Sample code:
var lastRowOfData = source_sheet.getRange(lastRow, 1, 1, lastColumn).getValues().flat();
Logger.log("Before removal: "+lastRowOfData)
// Remove index 2 (Column C value) in the array
lastRowOfData.splice(2,1)
Logger.log("After removal: "+lastRowOfData)
What it does?
Using array.flat(), change 2-d array to 1-d array
Using array.splice(), remove an element in the array
Output:
Execution log
5:10:50 AM Notice Execution started
5:10:51 AM Info Before removal: a,b,c,d,e
5:10:51 AM Info After removal: a,b,d,e
5:10:52 AM Notice Execution completed
Note:
I removed the oneD_array variable in the sample code. If you really want a separate variable for your 1-d array you can use this one:
var oneD_array = lastRowOfData.flat();
oneD_array.splice(2,1);
I'm fetching data from this binance API(https://api1.binance.com/api/v3/klines?symbol=ADAUSDT&interval=1d&limit=14).
I've, successfully, looped through the 'limit=14' parameter, which returns 0-13 indecis(or 14 indeces all together). These indeces contain various data on the pair specified in the 'symbol=ADAUSDT' parameter.
Now, theres 2 particular values I'm interested in: the high and the low of the day, which are in positions 2 and 3, respectively, in each index 0-13.
So, I've been able to log 0-13 results with the below script, but when I try to return the results to my sheet, it only populates the value in only one index, instead of 0-13.
How would I populate the results of my loop into the cells on my sheet?
function atrCalculation() {
var dailyHigh = UrlFetchApp.fetch("https://api1.binance.com/api/v3/klines?symbol=ADAUSDT&interval=1d&limit=14");
var jsonHighPrice = JSON.parse(dailyHigh);
for (var i = 0; i < jsonHighPrice.length; i++) {
data = jsonHighPrice[i];
Logger.log(data[2]);
Return data[i][2];
function in cell
only one value produced
Logger.log result
Thanks for any help!
You can simply retrieve all the data in your sheet 'klines' this way
function myKlines(){
var url='https://api1.binance.com/api/v3/klines?symbol=ADAUSDT&interval=1d&limit=14'
var source = UrlFetchApp.fetch(url).getContentText()
var json = JSON.parse(source)
var sh = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('klines')
sh.getRange(1, 1, json.length, json[0].length).setValues(json);
}
I am trying to figure out how can I export the text from a specific column from Google Sheets to a Google Drive .txt file. I am trying to export all the text from all the columns so there would be a different .txt file on google drive for each column. Also would want to skip first 2 rows.
I found this:
function saveToTextfile() {
var ss = SpreadsheetApp.getActive();
var sheet = ss.getActiveSheet();
var range = sheet.getRange(1, 1, sheet.getLastRow(), sheet.getLastColumn());
var rows = range.getValues();
var folder = DriveApp.getFoldersByName("folderName").next();
var files = folder.getFiles();
while(files.hasNext()) files.next().setTrashed(true);
rows.forEach(function(row, index) {
folder.createFile("row" + index + ".txt", row.join(", "));
});
}
This function is doing a text file for all the rows instead of columns and also adds a lot of "," in the resulted txt file. I can't figure it out how to change so it will do the columns, basically I want the function to create a txt file for columns (example: D3 to D100, E3 to E100, F3 to E100 and so on).
Also want this to update the text files when any changes are made on the sheet.
Thanks :)
Transposing the array:
getValues() returns a 2D array where each element in the outer array corresponds to a row. You want to have a 2D array where the outer array elements correspond to columns instead. That is, you need to transpose the 2D array.
An easy way to do that can be found in this answer:
array[0].map((_, colIndex) => array.map(row => row[colIndex]));
Other issues:
If you want to skip the first two rows, the starting row in your range (defined in getRange) should be 3 instead of 1, and correct the number of rows accordingly (check code sample below).
It adds a lot of , because you wrote it that way (see row.join(", ")). If instead you want each value to occupy a new line, use \n instead (check code sample below).
Updating the text files would require you to install an onEdit trigger which would fire the function saveToTextfile every time the spreadsheet is edited. Since this function requires authorization, it cannot be a simple trigger, but an installable one. If you have problems installing it, please post a new question (each question should deal with a specific issue, not with several of them).
Code sample:
function saveToTextfile() {
var ss = SpreadsheetApp.getActive();
var sheet = ss.getActiveSheet();
var firstRow = 3; // Skip first two rows => start at 3rd
var range = sheet.getRange(firstRow, 1, sheet.getLastRow() - firstRow + 1, sheet.getLastColumn());
var rows = range.getValues();
var columns = rows[0].map((_, colIndex) => rows.map(row => row[colIndex]));
var folder = DriveApp.getFoldersByName("folderName").next();
var files = folder.getFiles();
while(files.hasNext()) files.next().setTrashed(true);
columns.forEach(function(column, index) {
folder.createFile("column" + index + ".txt", column.join("\n")); // New line
});
}
Reference:
Transposing a 2D-array in JavaScript
a really quick question... I want to read rows in sheets one by one, (preferably 'live' so rather than putting it in an array or something). Besides using a loop, how do I read the contents of a whole row? Is it even possible?
You can read about it here
function readData() {
var sht = SpreadsheetApp.getActiveSheet();
var rng = sht.getRange(rownumber, 1, 1, numberofcolums)
var rangeArray = rng.getValues();
//now all your data for that row is in a two Dimensional array [[1,2,3,4,'My Data','etc']]
}
var
spreadsheetID =
"1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms",
rowNum = 1,
rowVals =
SpreadsheetApp.openById(spreadsheetID)
.getRange(rowNum +":"+ rowNum)
.getValues()
;
The other answers to date don't get a row, they get a range of specfic columns within a row. The code in this answer gets the entire unbounded row.
Not sure exactly what your end goal is, but I assume you mean something like this:
var values = sheet.getRange(startRow, startCol, numRows, numCols).getValues(); // returns 2d array
values.forEach(function(row){
Logger.log(row); // prints the contents of the row
});
You could also use the getCell() method of the Range class to iterate over cells in a range. The return type for getCell() is also Range. See this thread for details on the implementation Google Spreadsheets: Iterate over range, append string to each
I want to merge several sheets into 1 spreadsheeet and want to paste all data below each other.
My current code looks like that:
function updateMaster() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var allSheets = ss.getSheets();
var repArray = new Array();
// build array of all sheets
for (i in allSheets) {
repArray.push(allSheets[i].getName());
}
// store all sheets in array
var sheetArray = [];
// loop through all rep sheets
for (var j in allSheets) {
// get each sheet
var tempSheet = ss.getSheetByName(repArray[j]);
// get sheet data
var dataRange = tempSheet.getDataRange().getValues();
// remove the first header row
dataRange.splice(parseInt(0), 1);
// append sheet data to array
var sheetArray = sheetArray.concat(dataRange);
}
// Time to update the master sheet
var mSheet = ss.getSheetByName("Master");
// clear the whole sheet
mSheet.clear({contentsOnly:true});
// write to the Master sheet via the array
mSheet.getRange(1, 1, sheetArray.length, 4).setValues(sheetArray); //I GET THE ERROR BECAUSE OF THIS LINE
// force spreadsheet updates
SpreadsheetApp.flush();
// pause (1,000 milliseconds = 1 second)
Utilities.sleep("200");
// delete empty rows at bottom
var last = mSheet.getLastRow();
var max = mSheet.getMaxRows();
if (last !== max) {mSheet.deleteRows(last+1,max-last);}
}
However, I get an error, because other sheets have not the same column length.
The error message:
Incorrect range width, was 5 but should be 4
Here I get the error:
mSheet.getRange(1, 1, sheetArray.length, 4).setValues(sheetArray);
I basically, just want to post the data below. Any recommendation how I could make the column length variable?
Here is a link to my spreadsheet:
https://docs.google.com/spreadsheets/d/1TuHaqs20PvNwOJ46bQOAwN3Nt5vyRl7Win1Lul4m-e8/edit?usp=sharing
I appreciate your replies!
The sheetArray array has inner row elements of different lengths. Some inner arrays have 4 elements, some have 5 elements. You can't set values from an array with different inner element lengths. If you change the column width to 5, the error states that it should be 4, if you change it to 4, it says that it should be 5. You need to either add an extra empty element to the range with 4 columns, or write each sheet to the master individually.
Get one of the inner arrays, and find the length of an inner array. That is your column length.
On this line:
var dataRange = tempSheet.getDataRange().getValues();
You are using the getDataRange() method. That causes the range to be different dimensions for each sheet that has a different number of columns. If you set the column width to the biggest number of columns in the sheet with the most columns, then all your inner elements would be the same length. Your code would write a lot of empty cells. But don't know if there is any downside to that .
var columsToWriteTo = sheetArray[0].length
mSheet.getRange(1, 1, sheetArray.length, columsToWriteTo).setValues(sheetArray);
The above code gets the first inner array of the two dimensional sheetArray, and gets the length of the inner array.
The other answer pointed the issue but did not provide a solution.
What you should simply do is to make the final array homogeneous before setting its value with setValues().
Below is a code demo that does it by adding empty items in rows that are too short. I added comments in code to explain
function ArrayNormalize() {
var sh = SpreadsheetApp.getActive();
var data1 = sh.getSheets()[0].getDataRange().getValues(); // get data from sheet1
var data2 = sh.getSheets()[1].getDataRange().getValues();// same from sheet2 that has a different width
var combine = data1.concat(data2);
Logger.log("combine = "+JSON.stringify(combine));
//get Max array width
var maxWidth=0
for(var n=0;n<combine.length;n++){
Logger.log(n+' - '+combine[n]+" length = "+combine[n].length);
if(combine[n].length>=maxWidth){maxWidth=combine[n].length};
}
Logger.log("maxWidth = "+maxWidth);
//normalize
for(var n=0;n<combine.length;n++){
while(combine[n].length<maxWidth){
combine[n].push('');// while row is shorter add empty "cells" (=array elements)
}
}
sh.insertSheet().getRange(1,1,combine.length,combine[0].length).setValues(combine);// insert the result in new sheet to check the result but you can insert it anywhere else of course
}