Automated MD5 and Hex Encoding of Spreadsheet Columns - csv

I am receiving a CSV file, and in that file I need to Hex Encode one column, and MD5 Hash another column.
Final Outcome I am looking for is to take an incoming CSV file from an FTP account, somehow take data from two columns and encode (as above) the data into two other columns automatically.
While I can do this using a script I found in Excel manually, I really need to automate this process some other way so i am looking for help in knowing how to do this, perhaps sending the CSV file from FTP to Google Sheets and having a script there that automatically adds the two calculations as data is imported - so I can then pull from Google Sheets as the source to the program that will use it (automated)?
Or if there is a program out there that can do this?
Any help is much appreciated!

You can do it inside google sheets
Hex encode
If you want to hex encode decimal digit, you can use this built-in function: DEC2HEX(). Documentation about that function https://support.google.com/docs/answer/3093137?hl=en
MD5 hash
Open Tools > Script Editor then paste the following code:
function MD5 (input) {
var rawHash = Utilities.computeDigest(Utilities.DigestAlgorithm.MD5, input);
var txtHash = '';
for (i = 0; i < rawHash.length; i++) {
var hashVal = rawHash[i];
if (hashVal < 0) {
hashVal += 256;
}
if (hashVal.toString(16).length == 1) {
txtHash += '0';
}
txtHash += hashVal.toString(16);
}
return txtHash;
}
Save the script after that and then use the MD5() function in your spreadsheet while referencing a cell.

In case you need a unique identifier for a Google sheets cell I just got this:
Formula:
=IMPORTXML(https://md5.gromweb.com/?string=&C2, "//*[#id='content']/p[1]/em[2]")
Reference:
https://support.google.com/docs/answer/3093342?hl=en&ref_topic=9199554
Detail:
The idea is to have a column with a standard length string that is unique, the MD5 function is not in Google sheets unless you create a formula.
What did I do:
With the IMPORTXML function, I created a formula and used the URL of the site https://md5.gromweb.com, and took the XPath path of the query result which is the second parameter of the formula, for the query parameter I concatenated several cells in one and from that concatenated cell extract an MD5 from the string resulting from the concatenation, in this way I manage to have an md5 hash that varies each time the cell is updated.

I ran into a need for MD5 hashes myself and since there still isn't a built-in MD5 function provided by Google, I implemented a custom function.
To use it in your Google Sheets spreadsheet, go to Extensions > Apps Script in the menu and in the Apps Script editor that opens add a new file with this content:
/**
* Compute MD5 digest
*
* #param {string} input Input for the MD5 digest algorithm
* #customfunction
*/
const MD5 = (input) => Utilities
.computeDigest(Utilities.DigestAlgorithm.MD5, input)
.map((hashByte) => ((hashByte + 256) % 256).toString(16).padStart(2, '0'))
.join('')
.toUpperCase();
You can then use =MD5 in the sheet cell(s).
For example =MD5("My own string") will result in 1C978176903D20CD5EA1B40FD9FE909D MD5 hash.

Related

Prevent Auto-Format DriveApi 3 Google Apps script

Using the Drive API3, I'm looking for a way to make a copy of a CSV file in Google Sheets format, without having to convert the text to numbers, nor the functions and dates as it can be proposed in the Google Sheets menu:
File>Import>(Select your CSV file)> Untick "Convert text to number, dates and formula".
At the moment, I've got something such as :
function convert(){
var file = DriveApp.getFileById('1234');
var resource = { title : "Title", mimeType : MimeType.GOOGLE_SHEETS,parents : [{id: file.getParents().next().getId()}],}
Drive.Files.copy(resource,file.getId())
}
To illustrate my example : I've got a text in my CSV file "2021-25-03", if I run my macro, the new spreadsheet will automaticaly format my text to a Date and that's not my goal.
TFR.
There doesn't seem to be a setting in the API or in Apps Script to prevent the automatic conversion of numbers and dates, but we can build a script to work around this. Two tools are useful:
Apps Script's Utilities.parseCsv() method, which will build a 2D array of the values in the CSV file (as pure text--it does not interpret numbers and dates).
The fact that Google Sheets interprets any value starting with a single quote ' as text. This is true whether the value is entered in the UI or programmatically.
So the overall strategy is:
Copy the file as you are doing (or just create a new blank file, as we will write the values to it).
Parse the CSV values and prepend a ' to each one.
Write these modified values to the sheet.
Something like this:
function convert(){
var file = DriveApp.getFileById(CSV_FILE_ID);
// Create the copy:
var resource = { title : "Title", mimeType : MimeType.GOOGLE_SHEETS,parents : [{id: file.getParents().next().getId()}],}
var sheetsFile = Drive.Files.copy(resource,file.getId())
// Parse the original csv file:
var csv = Utilities.parseCsv(file.getBlob().getDataAsString())
// csv is a 2D array; prepend each value with a single quote:
csv.forEach(function(row){
row.forEach(function(value, i){
row[i] = "'" + value
})
})
// Open the first (and only) sheet in the file and overwrite the values with these modified ones:
var sheet = SpreadsheetApp.openById(sheetsFile.id).getSheets()[0]
sheet.getRange(1,1,csv.length, csv[0].length).setValues(csv)
}

Getting an error parsing data in google sheet custom script function

I created a simple custom function to test in google sheets script functions. The function definition is :
/**
* convert duration such as 1:30 to 1.5
*
* #customfunction
*/
function SIMPLETEST(input) {
// simply return the input for now to test.
return input;
}
and in my spread sheet I have a cell A2 that have value 3:30:00. when I apply this function on B2 for example set b2 to: =DURATION_DECIMAL(A2) it returns 12/30/1899 which I believe is base date.
Why is this happening?
It's because you must have the data type for that cell set to "automatic" or "duration", and Google Sheets will guess that "3:30:00" is a type of date/time for automatic, and for duration it converts it to date/time for passing to your function. It lets you keep it in your format (#:##:##), but when you pass it to your custom formula, Sheets first converts it to a Javascript Date object, which your function then returns, and sheets automatically displays as a regular date (12/30/1899). See Google's warning about Date conversions with custom functions here.
The easiest solution is to just explicitly set your input format to "plain text" using the format selection dropdown, and then in your custom function code, you can parse it as you see fit.
For example, I used this StackOverflow answer to write your custom function:
function DURATION_DECIMALS(input){
// https://stackoverflow.com/a/22820471/11447682
var arr = input.split(':');
var dec = parseInt((arr[1]/6)*10, 10);
return parseFloat(parseInt(arr[0], 10) + '.' + (dec<10?'0':'') + dec);
}
And here it is working with format set to plain text:
This works for me:
function durdechrs(dt) {
return Number((dt.valueOf()-new Date(dt.getFullYear(),dt.getMonth(),dt.getDate()).valueOf())/3600000).toFixed(2);
}

Google App Script - compare and copy data (URL Redirect list)

Please, do you have any idea how to solve the following work-flow in Google Spreadsheets or in JavaScript?
1) I have a list(sheet) of old URLs (1 URL per row)
2) I have a list(sheet) of new URLs (1 URL per row) which I want to pair to old similar URLs (for 301 redirect)
3) I have selected from URLs part of string which could be used for search and pair
4) I need to search the string in the list of new URLs and if there is a match, I need to copy this new URL and copy and paste it to the new column (D) next to the old URL.
I have tried google and lots of Google Spreadsheets Add-ons, but I cannot find anything that works.
I have the idea of the algorithm, but no clue how to write it in Google App Script (and the documentation didn't help)
1) for loop to select data from each row in column B
2) another for loop to select each row in column F
3) compare, if the data from step 1 is inside any cell in column F (for loop from step 2)
4) if there is a match, copy the cell from column F to column D, next to the substring which it found
Please, any ideas how to make it work?
Thank you for any tips.
function redirectSearch()
{
var ss=SpreadsheetApp.getActiveSpreadsheet();
var sht=ss.getSheetByName('Redirect');
var rng=sht.getRange(2,1,sht.getLastRow(),sht.getLastColumn());
var rngA=rng.getValues();
var substringA=[];
for(var i=0;i<rngA.length;i++)//load the substring Array
{
substringA.push(rngA[i][1]);
}
for(var i=0;i<rngA.length;i++)
{
for(var j=0;j<substringA.length;j++)
{
if(String(rngA[i][3]).indexOf(substringA[j])>-1)//look for substring in all of the new urls
{
rngA[j][2]=rngA[i][3];//if found copy them to column C next to substring
}
}
}
rng.setValues(rngA);//reload results and original data
}

What is the max size of a string in Google Apps Script?

I'm importing csv files from Gmail attachments into an existing Google Spreadsheet.
I use the getDataAsString() to hold the entire csv contents. I've tried it varying sizes up to ~6000 characters. Is there a maximum number of characters this string can take?
The limit for strings in Google Apps Script is 67,108,864 (67 million) characters. You can try it yourself with this function:
function strtest() {
var str = "a";
while (1) {
str = str + str;
}
}
Outputs:
I am not sure if there is a way to increase this through other means (for example, there is a fix for this in Wordpress).
GAS is based on Javascript, and apparently there is no JS string variable limit:
javascript object max size limit

Using built-in spreadsheet functions in a script

I'm using Google App Script for the first time.
I'm using it on a Google Doc spreadsheet.
I'm trying very simple functions, just to learn the basics. For example this works:
function test_hello() {
return 'hello';
}
But I'm puzzled by this simple one :
function test_today() {
return today();
}
It makes an #ERROR! wherever I use it.
And when I put my cursor on it, it says :
error : ReferenceError: "today" is not defined.
While the today() function works when used directly in the spreadsheet.
Does this mean that in scripts, I cannot use spreadsheet built-in functions?
Is there any elegant way around this?
Some spreadsheet functions are quite useful to me (I like weekday() for example).
A non-elegant way could be to create columns to calculate intermediate values that I need, and that can be calculated with spreadsheet functions. But I'd rather avoid something this dirty and cumbersome.
Google Apps Script is a subset of JavaScript, spreadsheet functions are currently not supported.
For example, if you want to create a function that returns today's date you should write :
function test_today(){
return new Date()
}// note that this will eventually return a value in milliseconds , you'll have to set the cell format to 'date' or 'time' or both ;-)
syntax is the same as with sheet functions : =test_today() see tutorial
There are many internet ressources on javascript, one of the most useful I found is w3school
Google Apps Script still does not (1/7/20) include an API to Google Sheets native functions.
But you can set the formula (native functions) of a cell named as a named range in a spreadsheet.
Then in the GAS:
var nativeOutput = spreadsheet.getRangeByName("outputCell").getValue();
Voila! Your GAS is calling the native function in the cell.
You can send data from the GAS to the native function in the cell, by naming another cell in the sheet (or in any sheet) referred to by the formula in the other cell:
spreadsheet.getRangeByName("inputCell").setValue(inputData);
Your GAS can dynamically create these cells, rather than hardcoding them, eg:
// Create native function, its input and output cells; set input value; use native function's output value:
// Use active spreadsheet.
var spreadsheet = SpreadsheetApp.getActive();
// Name input, output cells as ranges.
spreadsheet.setNamedRange("inputCell", spreadsheet.getRange("tuples!F1"));
spreadsheet.setNamedRange("outputCell", spreadsheet.getRange("tuples!F2"));
var outputCell = spreadsheet.getRangeByName("outputCell");
var inputCell = spreadsheet.getRangeByName("inputCell");
// Set native formula that consumes input cell's value, outputting in formula's cell.
outputCell.setFormula("=WEEKNUM(inputCell)");
// Call native function by setting input cell's value for formula to consume.
// Formula sets its cell's value to formula's output value.
inputCell.setValue(15);
// Consume native function output.
var nativeOutput = outputCell.getValue();
Logger.log("nativeOutput: "+ JSON.stringify(nativeOutput)); // Logs "nativeOutput: 3"
Beware: this technique exposes the code in cells that a spreadsheet user can access/change, and other spreadsheet operations could overwrite these cells.
What the spreadsheet functions can do, Javascript can do. I just have to replace var day_num = weekday() by var day_num = new Date(date).getDay()
Here is the result :
/**
* Writes the day of the week (Monday, Tuesday, etc), based on a date
*/
function day_name(date) {
// calculate day number (between 1 and 7)
var day_num = new Date(date).getDay();
// return the corresponding day name
switch(day_num) {
case 0: return 'Sunday'; break;
case 1: return 'Monday'; break;
case 2: return 'Tuesday'; break;
case 3: return 'Wednesday'; break;
case 4: return 'Thursday'; break;
case 5: return 'Friday'; break;
case 6: return 'Saturday'; break;
}
return 'DEFECT - not a valid day number';
};