Google Script - arrayofarrays - incorrect range width - google-apps-script

I have a Google Script that imports a TSV file and copies the contents into a Google Spreadsheet (code isn't mine, I copied it off someone else's project after googling a bit).
After some tinkering, the script works great but I keep getting a "Incorrect range width, was 1 but should be 6 (line 46, file "Test")" error.
Now, the TSV file getting imported is basically a list consisting of 6 columns, ie:
fruits vegetables cars countries colors names
pear carrot ford nicaragua yellow frank
After doing some reading up on how arrayofarrays works and googling for the same error, I've concluded that the problem is that somewhere in the importing/outputting process, a final empty row gets added. ie: instead of having 2 rows of 6 columns, I have 2 rows of 6 columns and a 3rd row with 1 empty column.
Apparently arrayofarrays needs all rows to be the exact same width.
As far as I can tell, this extra column is not present in the source file. I also cannot modify the source file so any fix needs to be done by the script itself.
Like I said, the script works but I'd like to a) stop getting emails about errors in my script, and b) understand how to fix this.
Any suggestions?
Thanks in advance.
Here's the code (there's some commented lines where I tried some fixes which clearly didn't work but I'm leaving them there for the sake of completion).
function getFileIDFromName(fileName) {
// Return the file ID for the first
// matching file name.
// NB: DocsList has been deprecated as of 2014-12-11!
// Using DriveApp instead.
var files = DriveApp.getFiles();
while (files.hasNext()) {
var file = files.next();
if (file.getName() === fileName) {
return file.getId();
}
}
return;
}
function getTsvFileAsArrayOfArays(tsvFileID) {
// Read the file into a single string.
// Split on line char.
// Loop over lines yielding arrays by
// splitting on tabs.
// Return an array of arrays.
var txtFile = DriveApp.getFileById(tsvFileID),
fileTextObj = txtFile.getAs('text/plain'),
fileText = fileTextObj.getDataAsString(),
lines = fileText.split('\n'),
lines2DArray = [];
lines.forEach(function (line) {
lines2DArray.push(line.split('\t'));
});
return lines2DArray;
}
function writeArrayOfArraysToSheet(tsvFileID) {
// Target range dimensions are determine
// from those of the nested array.
var sheet = SpreadsheetApp.getActiveSheet(),
arrayOfArrays = getTsvFileAsArrayOfArays(tsvFileID),
dimensions = {rowCount: Math.floor(arrayOfArrays.length),
colCount: Math.floor(arrayOfArrays[0].length)},
targetRng;
sheet.clearContents();
targetRng = sheet.getRange(1, 1,
dimensions.rowCount,
dimensions.colCount);
// .setValues(arrayOfArrays);
//targetRng = sheet.getRange(1, 1, arrayOfArrays.length, 6).setValues(arrayOfArrays);
//targetRng.setValues(arrayOfArrays);
targetRng.setValues(arrayOfArrays);
}
function runTsv2Spreadsheet() {
// Call this function from the Script Editor.
var fileName,
fileID,
file;
//fileName = Browser.inputBox('Enter a TSV file name:',
// Browser.Buttons.OK_CANCEL);
//fileID = getFileIDFromName(fileName);
//Logger.log(fileID)
fileID = "0B-9wMGgdNc6CdWxSNGllNW5FZWM"
if (fileID) {
writeArrayOfArraysToSheet(fileID);
}
}

Don't know if this is helpful. But, have you tried the appendRow() method for Google sheets? You can manipulate your array into the item format you want, then loop through it appending into the next empty row in the sheet.
In the past I looped though arrays trying to figure out the positions, but then started using the appendRow() method, and it made things easier for me.

Related

Convert a TXT delimited TAB to Google Sheets

I'm looking for a mean to convert my TXT file into a Google Sheets :
function convert_txt_gsheets(){
var file = DriveApp.getFilesByName('file.txt').next();
var body = file.getBlob().getDataAsString().split(/\n/);
var result = body.map( r => r.split(/\t/));
SpreadsheetApp.getActive().getSheets()[0].getRange(1,1,result.length,result[0].length).setValues(result);
return;
}
An error occured "The number of columns in the data does not match the number of columns in the range. The data has 1 but the range has 18."
Does someone have an idea ?
If I import the txt file manually it works but I need to do it through an G apps script.
I only see typos/wrong method names for getBlob and getFilesByName (you used getBlobl and getFileByName), but aside from that, the only possible issue that will cause this is that something from the file is written unexpectedly.
Update:
Upon checking, your txt file has a line at the bottom containing a blank row. Delete that and you should successfully write the file. That's why the error is range is expecting 18 columns but that last row only has 1 due to not having any data.
You could also filter the file before writing. Removing rows that doesn't have 18 columns will fix the issue. See code below:
Modification:
var result = body.map( r => r.split(/\t/)).filter( r => r.length == 18);
Successful run:

Google Script Function Timing

I am struggling to find the cause of the "mis-timing" of the following two function calls. I am attempting to have google look for a csv file in a particular gmail folder and insert the data into a google sheet. That works fine. Then, once the data is in the sheet, I call "UpdateComplete" whose sole purpose is to sort the sheet by column K (where there is a vlookup function that looks at a sheet with completed rows that have been moved there) so that rows that have the vlookup function already are sort to the top, and it then copies the formula into the rows that are new and don't already have it. However, if the google sheet has, say, 2000 rows, and the csv file contains 2100, for some reason the new 100 rows are being added after the call to UpdateComplete. So the new 100 rows are added, but they do not get the vlookup like all of the other rows. This issue only happens when the google sheet does not have enough rows, initially, for the csv data.
If, however, I comment out the call to "UpdateComplete" from within "RetrieveAwardData", and manually run that first, and then manually run "UpdateComplete", it works perfectly. I have tried adding a Utilities.Sleep call before the call to "UpdateComplete" (but after the csv setvalues line), in case it was a timing thing, but when I do that, the system waits that amount of time before adding the 100 new rows, even though the line for sleep comes after the line to add the csv data. I also tried creating a new function that calls "RetrieveAwardData" first (with the UpdateComplete call commented out) and then calls UpdateComplete 2nd, but the same issue happens. Why does it work properly if I run them, separately, manually, but not one after the other programmatically?
function RetrieveAwardData(){
var threads = GmailApp.search('is:unread subject:VA Benefit Aid');
var message = GmailApp.getMessagesForThreads(threads); //retrieve all messages in the specified threads.
var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('AwardData');
if(message[0] != null){
Logger.log(message[0]);
for (var i = 0 ; i < message.length; i++) {
for (var j = 0; j < message[i].length; j++) {
var attachment = message[i][j].getAttachments()[0];
var csvData = Utilities.parseCsv(attachment.getDataAsString('ISO-8859-1'), ",");
sheet.getRange(1, 1, csvData.length, csvData[0].length).setValues(csvData);
UpdateComplete();
GmailApp.markMessageRead(message[i][j]);
}
}
}
else{Logger.log("No file available.");}
}
function UpdateComplete(){
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName('AwardData');
sheet.sort(11);
var LastAwardRow = sheet.getLastRow();
var Avals = ss.getRange("K1:K").getValues();
var LastCompleteRow = Avals.filter(String).length;
if(LastAwardRow != LastCompleteRow){
sheet.getRange("K"+(LastCompleteRow+1) + ":K" + LastAwardRow).setFormulaR1C1(sheet.getRange("K"+LastCompleteRow).getFormulaR1C1());
}
}
Posting this for documentation purposes.
As mentioned by Cooper, use SpreadsheetApp.flush().
This method ensures that later parts of the script work with updated data, since changes made by previous parts are applied to the spreadsheet.

persistently sporadic out of memory error in Google Apps Scripts when copying data validations from a template

I have hundreds of Spreadsheets that were made from a template sheet. They all have the same number/name of sheets, rows, columns, etc...
I have added some data validations to the template. I want to copy the data validations from the template to each of the Spreadsheets. I have the code, and it works, but it throws a memory error.
It always throws the error -- the only thing that changes is how many destination Spreadsheets it has processed before it throws the error. Sometimes it'll process 4 Spreadsheets before it throws the error, sometimes 50, sometimes more/less. I cannot figure out why.
I trimmed my code down to a working sample. I can't share the source files but they are just normal Spreadsheets with 5 sheets/tabs and various data validations. If it matters, the data validations do use named ranges. For example: =REGEXMATCH(LOWER(google_drive_url) , "^https:\/\/drive\.google\.com\/drive\/folders\/[a-z0-9_-]{33}$").
I have commented the below code but here is a recap:
Get the template Spreadsheet and cache all of the data validations in it
Go through each destination Spreadsheet:
Alear all of the data validations
Apply the data validations from the template
In my real code I have an array of destination file IDs. For testing purposes I am just using one destination file and applying the data validations from the template multiple times.
function myFunction() {
var sourceFileID = "1rB7Z0C615Kn9ncLykVhVAcjmwkYb5GpYWpzcJRjfcD8";
var destinationFileID = "1SMrwTuknVa1Xky9NKgqwg16_JNSoHcFTZA6QxzDh7q4";
// get the source file
var sourceSpreadsheet = SpreadsheetApp.openById(sourceFileID);
var sourceDataValidationCache = {};
// go through each sheet and get a copy of the data validations
// cache them for quick access later
sourceSpreadsheet.getSheets().forEach(function(sourceSheet){
var sheetName = sourceSheet.getName();
// save all the data validations for this sheet
var thisSheetDataValidationCache = [];
// get the full sheet range
// start at first row, first column, and end at max rows and max columns
// get all the data validations in it
// go through each data validation row
sourceSheet.getRange(1, 1, sourceSheet.getMaxRows(), sourceSheet.getMaxColumns()).getDataValidations().forEach(function(row, rowIndex){
// go through each column
row.forEach(function(cell, columnIndex){
// we only need to save if there is a data validation
if(cell)
{
// save it
thisSheetDataValidationCache.push({
"row" : rowIndex + 1,
"column" : columnIndex + 1,
"dataValidation" : cell
});
}
});
});
// save to cache for this sheet
sourceDataValidationCache[sheetName] = thisSheetDataValidationCache;
});
// this is just an example
// so just update the data validations in the same destination numerous times to show the memory leak
for(var i = 0; i < 100; ++i)
{
// so we can see from the log how many were processed before it threw a memory error
Logger.log(i);
// get the destination
var destinationSpreadsheet = SpreadsheetApp.openById(destinationFileID);
// go through each sheet
destinationSpreadsheet.getSheets().forEach(function(destinationSheet){
var sheetName = destinationSheet.getName();
// get the full range and clear existing data validations
destinationSheet.getRange(1, 1, destinationSheet.getMaxRows(), destinationSheet.getMaxColumns()).clearDataValidations();
// go through the cached data validations for this sheet
sourceDataValidationCache[sheetName].forEach(function(dataValidationDetails){
// get the cell/column this data validation is for
// copy it, build it, and set it
destinationSheet.getRange(dataValidationDetails.row, dataValidationDetails.column).setDataValidation(dataValidationDetails.dataValidation.copy().build());
});
});
}
}
Is there something wrong with the code? Why would it throw an out of memory error? Is there anyway to catch/prevent it?
In order to get a better idea of what's failing I suggest you keep a counter of the iterations, to know how many are going through.
I also just noticed the line
sourceSheet.getRange(1, 1, sourceSheet.getMaxRows(), sourceSheet.getMaxColumns()).getDataValidations().forEach(function(row, rowIndex){
This is not a good idea, because getMaxRows() & getMaxColumns() will get the total number of rows in the sheet, not the ones with data, meaning, if your sheet is 100x100 and you only have data in the first 20x20 cells, you'll get a range that covers the entire 10,000 cells, and then calling a forEach means you go through every cell.
A better approach to this would be using getDataRange(), it will return a range that covers the entirety of your data (Documentation). With that you can use a much smaller range and considerably less cells to process.

Apps Script Utilities.parseCsv assumes new row on line breaks within double quotes

When using Utilities.parseCsv() linebreaks encased inside double quotes are assumed to be new rows entirely. The output array from this function will have several incorrect rows.
How can I fix this, or work around it?
Edit: Specifically, can I escape line breaks that exist only within double quotes? ie.
/r/n "I have some stuff to do:/r/n Go home/r/n Take a Nap"/r/n
Would be escaped to:
/r/n "I have some stuff to do://r//n Go home//r//n Take a Nap"/r/n
Edit2: Bug report from 2012: https://code.google.com/p/google-apps-script-issues/issues/detail?id=1871
So I had a somewhat large csv file about 10MB 50k rows, which contained a field at the end of each row with comments that users enter with all sorts of characters inside. I found the proposed regex solution was working when I tested a small set of the rows, but when I threw the big file to it, there was an error again and after trying a few things with the regex I even got to crash the whole runtime.
BTW I'm running my code on the V8 runtime.
After scratching my head for about an hour and with not really helpful error messages from AppsSript runtime. I had an idea, what if some weird users where deciding to use back-slashes in some weird ways making some escapes go wrong.
So I tried replacing all back-slashes in my data with something else for a while until I had the array that parseCsv() returns.
It worked!
My hypothesis is that having a \ at the end of lines was breaking the replacement.
So my final solution is:
function testParse() {
let csv =
'"title1","title2","title3"\r\n' +
'1,"person1","A ""comment"" with a \\ and \\\r\n a second line"\r\n' +
'2,"person2","Another comment"';
let sanitizedString =
csv.replace(/\\/g, '::back-slash::')
.replace(/(?=["'])(?:"[^"\\]*(?:\\[\s\S][^"\\]*)*"|'[^'\\]\r?\n(?:\\[\s\S][^'\\]\r?\n)*')/g,
match => match.replace(/\r?\n/g, "::newline::"));
let arr = Utilities.parseCsv(sanitizedString);
for (let i = 0, rows = arr.length; i < rows; i++) {
for (let j = 0, cols = arr[i].length; j < cols; j++) {
arr[i][j] =
arr[i][j].replace(/::back-slash::/g,'\\')
.replace(/::newline::/g,'\r\n');
}
}
Logger.log(arr)
}
Output:
[20-02-18 11:29:03:980 CST] [[title1, title2, title3], [1, person1, A "comment" with a \ and \
a second line], [2, person2, Another comment]]
It may be helpful for you to use Sheets API.
In my case, it works fine without replacing the CSV text that contains double-quoted multi-line text.
First, you need to make sure of bellow:
Enabling advanced services
To use an advanced Google service, follow these instructions:
In the script editor, select Resources > Advanced Google services....
In the Advanced Google Service dialog that appears,
click the on/off switch next to the service you want to use.
Click OK in the dialog.
If it is ok, you can import a CSV text data into a sheet with:
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName('some_name');
const resource = {
requests: [
{
pasteData: {
data: csvText, // Your CSV data string
coordinate: {sheetId: sheet.getSheetId()},
delimiter: ",",
}
}
]
};
Sheets.Spreadsheets.batchUpdate(resource, ss.getId());
or for TypeScript, which can be used by clasp:
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName('some_name');
const resource: GoogleAppsScript.Sheets.Schema.BatchUpdateSpreadsheetRequest = {
requests: [
{
pasteData: {
data: csvText, // Your CSV data string
coordinate: {sheetId: sheet.getSheetId()},
delimiter: ",",
}
}
]
};
Sheets.Spreadsheets.batchUpdate(resource, ss.getId());
I had this same problem and have finally figured it out. Thanks Douglas for the Regex/code (a bit over my head I must say) it matches up nicely to the field in question. Unfortunately, that is only half the battle. The replace shown will simply replaces the entire field with \r\n. So that only works when whatever is between the "" in the CSV file is only \r\n. If it is embedded in the field with other data it silently destroys that data. To solve the other half of the problem, you need to use a function as your replace. The replace takes the matching field as a parameter so so you can execute a simple replace call in the function to address just that field. Example...
Data:
"Student","Officer
RD
Special Member","Member",705,"2016-07-25 22:40:04 EDT"
Code to process:
var dataString = myBlob().getDataAsString();
var escapedString = dataString.replace(/(?=["'])(?:"[^"\](?:\[\s\S][^"\])"|'[^'\]\r\n(?:\[\s\S][^'\]\r\n)')/g, function(match) { return match.replace(/\r\n/g,"\r\n")} );
var csvData = Utilities.parseCsv(escapedString);
Now the "Officer\r\nRD\r\nSpecial Member" field gets evaluated individually so the match.replace call in the replace function can be very straight forward and simple.
To avoid trying to understand regular expressions, I found a workaround below, not using Utilities.parseCsv(). I'm copying the data line by line.
Here is how it goes:
If you can find a way to add an extra column to the end of your CSV, that contains the exact same value all the time, then you can force a specific "line break separator" according to that value.
Then, you copy the whole line into column A and use google app script' dedicated splitTextToColumns() method...
In the example below, I'm getting the CSV from an HTML form. This works because I also have admin access to the database the user takes the CSV from, so I could force that last column on all CSV files...
function updateSheet(form) {
var fileData = form.myFile;
// gets value from form
blob = fileData.getBlob();
var name = String(form.folderId);
// gets value from form
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.setActiveSheet(ss.getSheetByName(name), true);
sheet.clearContents().clearFormats();
var values = [];
// below, the "Dronix" value is the value that I could force at the end of each row
var rows = blob.contents.split('"Dronix",\n');
if (rows.length > 1) {
for (var r = 2, max_r = rows.length; r < max_r; ++r) {
sheet.getRange(r + 6, 1, 1, 1).setValue(String(rows[r]));
}
}
var spreadsheet = SpreadsheetApp.getActive();
spreadsheet.getRange("A:A").activate();
spreadsheet.getRange("A:A").splitTextToColumns();
}
Retrieved and slightly modified a regex from another reply on another post: https://stackoverflow.com/a/29452781/3547347
Regex: (?=["'])(?:"[^"\\]*(?:\\[\s\S][^"\\]*)*"|'[^'\\]\r\n(?:\\[\s\S][^'\\]\r\n)*')
Code:
var dataString = myBlob.getDataAsString();
var escapedString = dataString.replace(/(?=["'])(?:"[^"\\]*(?:\\[\s\S][^"\\]*)*"|'[^'\\]\r\n(?:\\[\s\S][^'\\]\r\n)*')/g, '\\r\\n');

Iterate over sheet using GS while there is any data in columns

I need to do a small script, its basic idea is to have a sheet with names in one columns and some random info in another (to the right). The sheet has some count (unknown in advance) of such records, so they are like
John 39483984
George 3498349
Layla 23948
So that a user can enter any number of such simple records, my script must create a file for each name (file is of the same name) and write the number into that file. I managed to find how to create files (though still couldn't find out how to create file in the current folder, same as where the sheet is located - this is just a side question, but if you know how to do it, please tell me). The only real problem is iterating through the records. My idea was to go through them one by one and stop when there is an empty record - basic strategy, but I wasn't able to find how to implement it (yep!). There are range functions, but there I should know in advance the range; also there is a function to get selected cells but that will require a user to select the records, which is strange.
So please suggest me a solution if it exists in this frustration Google Script.
function createFilesForEachNameInSheet() {
// First, you connect to the spreadsheet, and store the connection into a variable
var ss = SpreadsheetApp.openById("SPREADSHEET_KEY_GOES_HERE"); // you do know how to get the spreadsheet key, right?
// Then, you take the sheet from that spreadsheet
var sheet = ss.getSheetByName("Sheet1");
// Then, the "problematic range". You get the ENTIRE range, from end to end, as such:
var wholeRange = sheet.getRange(1,1,sheet.getLastRow(),sheet.getLastColumn());
// Then, you fetch its values:
var rangeValues = wholeRange.getValues();
// At this point you have a bi-dimensional array, representing the rows and columns.
// Assuming you have 2 columns, in the first column you have the names, and in the second you have the unknown value
// You need to use the already known for loop, iterate over all the data, and store it first in an object, so that you create the file only ONCE.
var objectData = {};
for (var i=0;i<rangeValues.length;i++) {
var thisName = rangeValues[i][0];
var thisValue = rangeValues[i][1];
if (objectData.thisName == undefined) objectData.thisName = [];
objectData.thisName.push(thisValue);
}
// Now we have our values grouped by name.
// Let's create a file for each name.
for (var name in objectData) {
DriveApp.createFile(name, objectData.name.join("\n"));
}
// NOTE: if you want to create into a specific folder, you first target it, using the DriveApp:
var folders = DriveApp.getFoldersByName("the folder name goes here");
// folders variable is now an iterator, containing each folder with that name.
// we will iterate over it as follows, and select the one we want.
// the logic for it, you'll select one of your choice:
while (folders.hasNext()) {
var thisFolder = folders.next();
if (/* condition to check if we found the right folder */) {
thisFolder.createFile(name, objectData.name.join("\n"))
}
}
}