MySQL: LOADing a SHA1 hash into a BINARY(20) column - mysql

Question

I'm going to be loading a billion rows into a mySQL table, one column of which - BINARY(20) - is the SHA1 hash of several other columns, concatenated. Offhand I don't see how to use the LOAD command to load binary values, because it seems to rely upon delimiters.
Obviously, speed is important here, which is why I want to use LOAD. Does anyone know how to load a fixed-length binary value with LOAD? Is this perhaps a job for a trigger? (I've never used triggers before.) Or can I invoke a function (e.g. UNHEX) in the LOAD command?
(Since it seems to be a common question: no, I don't want to store it in base64 or hex notation. BINARY(20) is a requirement.)

Answer 1

Binary data and LOAD DATA INFILE are not friends. The file format specifiers need a delimiter, and arbitrary binary data is length delimited, not field delimited.
Your best bet is to use large multi-INSERT statements and tough it out. These can handle having hex-encoded strings decoded and dropped into BINARY columns automatically.
I'm not sure why anyone would wish this misery upon themselves, though. Saving twenty bytes a row versus standard hex notation is not worth the trouble.
If you really need to load in kajillions of rows, maybe MySQL is not the best platform to do it on. What you should be doing is either sharding that data into multiple tables or databases, or using a NoSQL store to split it up more effectively.

Answer 2

This seems to be a reasonable approach: to use the SET form of LOAD, using variables and invoking functions such as UNHEX and CONCAT.
For example:
Suppose mytable has four columns:
mysha1 BINARY(20)
a VARCHAR(20)
b VARCHAR(20)
c VARCHAR(20)
Column mysha1 is the sha1 hash of a, b, and c concatenated with '|' as a separator.
And suppose the input file is tab-delimited text lines of three fields apiece:
abel\tbaker\tcharlie\t\n
dog\teasy\tfor\t\n
etc\tetc\tetc\t\n
Here's how I'd load the table
LOAD DATA INFILE '/foo/bar/input.txt' INTO TABLE mytable
FIELDS TERMINATED BY '\t' ESCAPED BY '\\' LINES TERMINATED BY '\n'
(#f1, #f2, #f3) SET mysha1 = UNHEX(SHA1(CONCAT_WS('|', #f1, #f2, #f3))),
a=#f1, b=#f2, c=#f3;
UPDATE: in the general case, for the arbitrary binary value that can't be computed with a builtin function such as SHA1, the binary value must be expressed in the INFILE as a displayable-hex string, read into an #variable, and then converted into binary with the UNHEX function. E.g.:
mytable:
mybin8 BINARY(8)
a VARCHAR(20)
b VARCHAR(20)
c VARCHAR(20)
input file:
abel\tbaker\tcharlie\t0123456789abcdef\n
dog\teasy\tfox\t2468ace13579bdf\n
etc\tetc\tetc\t0000000000000000\n
load command:
LOAD DATA INFILE '/foo/bar/input.txt' INTO TABLE mytable
FIELDS TERMINATED BY '\t' ESCAPED BY '\\' LINES TERMINATED BY '\n'
(a, b, c, #myhex) SET mybin8 = UNHEX(#myhex);