Can this query or table schema be optimized? - mysql

Question

I am running this procedure a few million times, and although each time it takes a few ms, eventually it takes a couple of weeks to run all of them. I was wondering if anyone could help me optimizing or improving its performance. Any improvement might save days!
CREATE PROCEDURE process_parameters(IN parameter1 VARCHAR(128), IN parameter2 VARCHAR(128), IN combination_type CHAR(1))
BEGIN
SET #parameter1_id := NULL, #parameter2_id := NULL;
SET #parameter1_hash := "", #parameter2_hash := "";
IF parameter1 IS NOT NULL THEN
SET #parameter_hash := parameter1;
INSERT IGNORE INTO `collection1` (`parameter`) VALUES (parameter1);
SET #parameter1_id := (SELECT `id` FROM `collection1` WHERE `parameter` = parameter1);
END IF;
IF parameter2 IS NOT NULL THEN
SET #parameter2_hash := parameter2;
INSERT IGNORE INTO `collection2` (`parameter`) VALUES (parameter2);
SET #parameter2_id := (SELECT `id` FROM `collection2` WHERE `parameter` = parameter2);
END IF;
SET #hash := MD5(CONCAT(#parameter1_hash, #parameter2_hash));
INSERT IGNORE INTO `combinations` (`hash`,`type`,`parameter1`,`parameter2`) VALUES (#hash, combination_type, #parameter1_id, #parameter2_id);
END
The logic behind of it is: I store unique combinations of (parameter1, parameter2) in combinations, where parameter1 or paramter2 can be NULL (but never both at the same time). I store a type in combinations to know later which parameter has value. To ensure that a combination is unique I added an MD5 field (a primary key (parameter1,parameter2) will not work because of comparison with NULL always returns NULL). Each parameter has a separate table (collection1 and collection2 respectively) to store their unique id. There are hundreds/thousands of unique parameter1 and parameter2, but their combinations are highly repeated and are much below the cardinal multiplication.
As an example, ("A", "1"), ("A", "2"), ("B", "1"), ("A", "1"), ("A", NULL), (NULL, "2") would yield:
`collection1` (`id`, `parameter`)
1, "A"
2, "B"
`collection2` (`id`, `parameter`)
1, "1"
2, "2"
`combinations` (`type`, `parameter1`, `parameter2`)
"P1andP2", 1, 1,
"P1andP2", 1, 2,
"P1andP2", 2, 1,
"P1Only", 1, NULL
"P2Only", NULL, 2
These are the definitions of the tables:
DESCRIBE `combinations`;
+-------------+-----------------------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+-----------------------------------+------+-----+---------+----------------+
| combination | int(11) | NO | PRI | NULL | auto_increment |
| hash | char(32) | NO | UNI | NULL | |
| type | enum('P1andP2','P1Only','P2Only') | NO | | NULL | |
| parameter1 | int(11) | YES | | NULL | |
| parameter2 | int(11) | YES | | NULL | |
+-------------+-----------------------------------+------+-----+---------+----------------+
DESCRIBE `collection1`; (`collection2` is identical)
+-----------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| parameter | varchar(255) | NO | UNI | NULL | |
+-----------+--------------+------+-----+---------+----------------+
Any help will be appreciated!

Answer 1

Please use SHOW CREATE TABLE; it is more descriptive than DESCRIBE.
Use LAST_INSERT_ID()
SET #parameter1_id := (SELECT `id` FROM `collection1`
WHERE `parameter` = parameter1);
can be replaced by
SELECT #parameter1_id := LAST_INSERT_ID();
It will avoid a round trip to the server.
Oops... The OP points out that the id won't be returned if the row is a dup. This is a workaround that might run faster:
INSERT INTO `collection1` (`parameter`)
VALUES (parameter1)
ON DUPLICATE KEY UPDATE
id = LAST_INSERT_ID(id);
SELECT #parameter1 := LAST_INSERT_ID(id);
It's a kludgy trick that is documented somewhere in the documentation. But; more below...
Shrink table
Do you really need combination? You have another UNIQUE key that could be used as the PRIMARY KEY. This might cut in half the time taken for the final INSERT.
This may (or may not) speed things up, but only because the row size shrinks: Instead of storing the md5 into CHAR(32), store UNHEX(md5) into BINARY(16).
Batch INSERT
Can you gather a bunch of these to INSERT at once? If you gather 1000 rows and string them into a single INSERT (actually 3 INSERTs, since 3 tables are involved), it will run literally 10 times as fast.
Because of needing the ids, it gets more complicated. You would need to batch things into collection1 and collection2; then work on combinations.
Since the "combination*" tables are essentially "normalization", see my discussion of how to batch them very efficiently: http://mysql.rjweb.org/doc.php/staging_table#normalization It involves 2 statements, one to insert new rows, the other to grab all the ids for the batch.
COALESCE
Get rid of #parameter*_hash and #hash completely. Change the use of #hash call to:
INSERT IGNORE INTO combinations (...) VALUES
( CONCAT(COALESCE(parameter1,''), COALESCE(parameter2, '')),
...)
Think of it this way... Each statement takes a non-trivial amount of time. (This shows up significantly in batching of inserts.) I'm getting rid of 4 statements at some expense due to adding complexity to one statement.
Settings
The most important might be innodb_flush_log_at_trx_commit = 2.
3 Streams
Write 3 procedures, each one with the code simplified to the particular type. Combining this with batching should further speed things up.
Potential issues
I think these two will get the same hash. Hence, only one row for these two:
("xyz", NULL)
(NULL, "xyz")
Be aware that INSERT IGNORE will burn ids if there is already a row with the given unique key. Because of this, keep an eye on running out of values with INT (only 2 billion). Changing to INT UNSIGNED would up it to 4B, still in 4 bytes.