I want to use BINARY UUIDs as my primary key in my tables, but using my own custom functions that generates optimised UUIDs loosely based on this article: https://mariadb.com/kb/en/guiduuid-performance/
The table structure and two main functions of interest here are:
CREATE TABLE `Test` (
`Id` BINARY(16),
`Data` VARCHAR(100)
) ENGINE=InnoDB
ROW_FORMAT=DYNAMIC CHARACTER SET 'utf8mb4' COLLATE 'utf8mb4_unicode_ci';
CREATE DEFINER = 'user'#'%' FUNCTION `OPTIMISE_UUID_STR`(`_uuid` VARCHAR(36))
RETURNS VARCHAR(32) CHARACTER SET utf8mb4
DETERMINISTIC
NO SQL
SQL SECURITY INVOKER
COMMENT ''
BEGIN
/*
FROM
00 10 20 30
123456789012345678901234567890123456
====================================
AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE
TO
00 10 20 30
12345678901234567890123456789012
================================
CCCCBBBBAAAAAAAADDDDEEEEEEEEEEEE
*/
RETURN UCASE(CONCAT(
SUBSTR(_uuid, 15, 4), /* Time nodes reversed */
SUBSTR(_uuid, 10, 4),
SUBSTR(_uuid, 1, 8),
SUBSTR(_uuid, 20, 4), /* MAC nodes last */
SUBSTR(_uuid, 25, 12)));
END;
CREATE DEFINER = 'user'#'%' FUNCTION `CONVERT_OPTIMISED_UUID_STR_TO_BIN`(`_hexstr` BINARY(32))
RETURNS BINARY(16)
DETERMINISTIC
NO SQL
SQL SECURITY INVOKER
COMMENT ''
BEGIN
/*
Convert optimised UUID from string hex representation to binary. If the UUID is not optimised, it makes no sense to convert
*/
RETURN UNHEX(_hexstr);
END;
I cannot use my custom functions in column definition as shown below
CREATE TABLE `Test` (
`Id` BINARY(16) NOT NULL DEFAULT CONVERT_OPTIMISED_UUID_STR_TO_BIN(OPTIMISE_UUID_STR(UUID())),
I get the error "Function or expression 'OPTIMISE_UUID_STR()' cannot be used in the DEFAULT clause of Id"
So I tried using the same in Triggers:
CREATE DEFINER = 'user'#'%' TRIGGER `Test_before_ins_tr1` BEFORE INSERT ON `Test`
FOR EACH ROW
BEGIN
IF (new.Id IS NULL) OR (new.Id = X'0000000000000000') OR (new.Id = X'FFFFFFFFFFFFFFFF') THEN
SET new.Id = CONVERT_OPTIMISED_UUID_STR_TO_BIN(OPTIMISE_UUID_STR(UUID()));
END IF;
END;
The above works pretty good, but the issue is that I cannot define the Id column as PRIMARY KEY, which I want to because PRIMARY KEYs have to be NOT NULL, and setting this means I have to pre-generate optimised UUIDs. I do not want to do this as I would like the DB to take care of generating the optimised UUIDs.
As you might have inferred looking at the above Trigger definition, I tried setting a default value on the Id column, such as:
Id` BINARY(16) NOT NULL DEFAULT X'0000000000000000'
and
Id` BINARY(16) NOT NULL DEFAULT X'FFFFFFFFFFFFFFFF'
and
Id` BINARY(16) NOT NULL DEFAULT '0' /* I tried setting 0, but always seem to revert to '0' */
and this default value would be picked up by the trigger and a correct optimised UUID assigned. But that also does not work as the DB complains "Column 'Id' cannot be null" even though a DEFAULT value has been set.
So my actual question is: Can I generate a custom (optimised UUID) BINARY value for a PRIMARY KEY column?
Short answer: Yes
Long answer:
The PRIMARY KEY can be almost any datatype with whatever values you can create.
TEXT or BLOB are not allowed. Not even TINYTEXT.
VARCHAR (etc) are not allowed beyond some size (depends on version and CHARACTER SET). VARCHAR(191) (or smaller) works in all combinations. The ubiquitous VARCHAR(255) works in many situations.
MySQL 8.0 has builtin functions for converting between binary and string UUIDs. This also provides functions (like yours) for such: UUIDs
Yes, it's doable even without triggers and/or stored functions:
MariaDB from version 10.6:
Use function SYS_GUID() which returns same result as UUID() but without - characters. The result of this function can be directly converted to a 16-byte value with UNHEX() function.
Example:
CREATE TABLE test (a BINARY(16) NOT NULL DEFAULT UNHEX(SYS_GUID()) PRIMARY KEY);
INSERT INTO test VALUES (DEFAULT);
INSERT INTO test VALUES (DEFAULT);
SELECT HEX(a) FROM test;
+----------------------------------+
| HEX(a) |
+----------------------------------+
| 53EE84FB733911EDA238D83BBF89F2E2 |
| 61AC0286733911EDA238D83BBF89F2E2 |
+----------------------------------+
MariaDB from version 10.7 (as mentioned in danielblack's comment):
Use UUID datatype which stores UUID() (and SYS_GUID()) values as 16 byte:
CREATE TABLE test (a UUID not NULL default UUID() PRIMARY KEY);
INSERT INTO test VALUES (DEFAULT);
INSERT INTO test VALUES (DEFAULT);
SELECT a FROM test;
+--------------------------------------+
| a |
+--------------------------------------+
| 6c42e367-733b-11ed-a238-d83bbf89f2e2 |
| 6cbc0418-733b-11ed-a238-d83bbf89f2e2 |
+--------------------------------------+
Addendum: If you are using a version < 10.6 and your requirements match the following limitations, you could also use UUID_SHORT() function, which generates a 64-bit identifier.
Related
I've created a table in mysql by
create table test (id int primary key not null auto_increment, vs varchar(255) not null);
when run
insert into test (vs) values (null);
It throws an error:
ERROR 1048 (23000): Column 'vs' cannot be null
But when I try to insert two rows by
insert into test (vs) values (null),(null);
It works and result is:
mysql> select * from test;
+----+----+
| id | vs |
+----+----+
| 1 | |
| 2 | |
+----+----+
The field vs is not nullable, I wonder whether it is a feature.
This is neither a bug nor a feature. It's just how it works in this case. This behavior is documented in MySQL documentation - Constraints on Invalid Data.
If you try to store NULL into a column that doesn't take NULL values, an error occurs for single-row INSERT statements. For multiple-row INSERT statements or for INSERT INTO ... SELECT statements, MySQL Server stores the implicit default value for the column data type. In general, this is 0 for numeric types, the empty string ('') for string types, and the “zero” value for date and time types. Implicit default values are discussed in Section 11.7, “Data Type Default Values”.
This happens when you have strict mode turned off.
I have user table. The primary key is user_id which has a datatype bigint(20).
I generate a user_id using UUID_SHORT() via trigger below. The issue is I get a warning when I try to insert a record as follows:
Warning: #1366 Incorrect integer value: '' for column 'user_id' at row 1
My phpMyAdmin trigger is as follows:
BEGIN
SET NEW.user_id=UUID_SHORT();
END
Any reason why I am getting this warning? Have I set the datatype correctly?
INT is a four-byte signed integer, while UUID_SHORT() returns a 64-bit (i.e. 8 byte) unsigned integer. You are trying to store a 64-bit data type into a 4-byte INT and MySQL appears to be storing an empty string '' into the field user_id.
From the MySQL manual:
UUID_SHORT()
Returns a “short” universal identifier as a 64-bit unsigned integer (rather than a string-form 128-bit identifier as returned by the UUID() function).
You should use the BIGINT UNSIGNED type instead.
will not work if you try to insert from PMA, as it generate this kind of query :
INSERT INTO `users` (`user_id`, `name`) VALUES ('', 'sami')
you can replace the '' with NULL, or remove user_id from the insert
INSERT INTO `users` (`name`) VALUES ('sami')
this will solve the warning problem.
I have a weird problem with a MySQL users table. I have quickly created a simplified version as a testcase.
I have the following table
CREATE TABLE IF NOT EXISTS `users` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`identity` varchar(255) NOT NULL,
`credential` varchar(255) NOT NULL,
`credentialSalt` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=ucs2 AUTO_INCREMENT=2 ;
INSERT INTO `users` (`id`, `identity`, `credential`, `credentialSalt`) VALUES
(1, 'test', '7288edd0fc3ffcbe93a0cf06e3568e28521687bc', '123');
And I run the following query
SELECT id,
IF (credential = SHA1(CONCAT('test', credentialSalt)), 1, 0) AS dynamicSaltMatches,
credentialSalt AS dynamicSalt,
SHA1(CONCAT('test', credentialSalt)) AS dynamicSaltHash,
IF (credential = SHA1(CONCAT('test', 123)), 1, 0) AS staticSaltMatches,
123 AS staticSalt,
SHA1(CONCAT('test', 123)) AS staticSaltHash
FROM users
WHERE identity = 'test'
Which gives me the following result
The dynamic salt does NOT match while the static salt DOES match.
This is blowing my mind. Can someone help me point out the cause of this?
My MySQL version is 5.5.29
It's because of the default character set of your table. You appear to be running this on a UTF8 database and something in SHA1() is having problems with the differing character sets.
If you change your table declaration to the following it will match again:
CREATE TABLE IF NOT EXISTS `users` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`identity` varchar(255) NOT NULL,
`credential` varchar(255) NOT NULL,
`credentialSalt` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=2 ;
SQL Fiddle
As robertklep commented explicitly casting your string to a character will also work, basically ensure you're using the same characterset when doing comparisons using SHA1()
As the encryption functions documentation says:
Many encryption and compression functions return strings for which the result might contain arbitrary byte values. If you want to store these results, use a column with a VARBINARY or BLOB binary string data type. This will avoid potential problems with trailing space removal or character set conversion that would change data values, such as may occur if you use a nonbinary string data type (CHAR, VARCHAR, TEXT).
This was changed in version 5.5.3:
As of MySQL 5.5.3, the return value is a nonbinary string in the connection character set. Before 5.5.3, the return value is a binary string; see the notes at the beginning of this section about using the value as a nonbinary string.
My projects requires to start inputs from the user with the spacing on the left and spacing on the right of a word, for example 'apple'. If the user types in ' apple' or 'apple ', whether it is one space or multiple space on the left or right of the word, I need to store it that way.
This field has the Unique attribute, but I attempt to insert the word with spacing on the left, and it works fine. But when I attempt to insert the word with spacing on the right it trims off all the spacing from the right of the word.
So I am thinking of adding a special character to the right of the word after the spacing. But I am hoping there is a better solution for this issue.
CREATE TABLE strings
( id bigint(20) unsigned NOT NULL AUTO_INCREMENT,
string varchar(255) COLLATE utf8_bin NOT NULL,
created_ts timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (id), UNIQUE KEY string (string) )
ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_bin
The problem is that MySQL ignores trailing whitespace when doing string comparison. See
http://dev.mysql.com/doc/refman/5.7/en/char.html
All MySQL collations are of type PADSPACE. This means that all CHAR, VARCHAR, and TEXT values in MySQL are compared without regard to any trailing spaces.
...
For those cases where trailing pad characters are stripped or comparisons ignore them, if a column has an index that requires unique values, inserting into the column values that differ only in number of trailing pad characters will result in a duplicate-key error. For example, if a table contains 'a', an attempt to store 'a ' causes a duplicate-key error.
(This information is for 5.7; for 8.0 this changed, see below)
The section for the like operator gives an example for this behavior (and shows that like does respect trailing whitespace):
mysql> SELECT 'a' = 'a ', 'a' LIKE 'a ';
+------------+---------------+
| 'a' = 'a ' | 'a' LIKE 'a ' |
+------------+---------------+
| 1 | 0 |
+------------+---------------+
1 row in set (0.00 sec)
Unfortunately the UNIQUE index seems to use the standard string comparison to check if there is already such a value, and thus ignores trailing whitespace.
This is independent from using VARCHAR or CHAR, in both cases the insert is rejected, because the unique check fails. If there is a way to use like semantics for the UNIQUE check then I do not know it.
What you could do is store the value as VARBINARY:
mysql> create table test_ws ( `value` varbinary(255) UNIQUE );
Query OK, 0 rows affected (0.13 sec)
mysql> insert into test_ws (`value`) VALUES ('a');
Query OK, 1 row affected (0.08 sec)
mysql> insert into test_ws (`value`) VALUES ('a ');
Query OK, 1 row affected (0.06 sec)
mysql> SELECT CONCAT( '(', value, ')' ) FROM test_ws;
+---------------------------+
| CONCAT( '(', value, ')' ) |
+---------------------------+
| (a) |
| (a ) |
+---------------------------+
2 rows in set (0.00 sec)
You better do not want to do anything like sorting alphabetically on this column, because sorting will happen on the byte values instead, and that will not be what the users expect (most users, anyway).
The alternative is to patch MySQL and write your own collation which is of type NO PAD. Not sure if someone wants to do that, but if you do, let me know ;)
Edit: meanwhile MySQL has collations which are of type NO PAD, according to https://dev.mysql.com/doc/refman/8.0/en/char.html :
Most MySQL collations have a pad attribute of PAD SPACE. The exceptions are Unicode collations based on UCA 9.0.0 and higher, which have a pad attribute of NO PAD.
and https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-sets.html
Unicode collations based on UCA versions later than 4.0.0 include the version in the collation name. Thus, utf8mb4_unicode_520_ci is based on UCA 5.2.0 weight keys, whereas utf8mb4_0900_ai_ci is based on UCA 9.0.0 weight keys.
So if you try:
create table test_ws ( `value` varbinary(255) UNIQUE )
character set utf8mb4 collate utf8mb4_0900_ai_ci;
you can insert values with and without trailing whitespace
You can find all available NO PAD collations with:
show collation where Pad_attribute='NO PAD';
This is not about CHAR vs VARCHAR. SQL Server does not consider trailing spaces when it comes to string comparison, which is applied also when checking a unique key constraint. So it is not that you cannot insert value with trailing spaces, but once you insert, you cannot insert another value with more or fewer spaces.
As a solution to your problem, you can add a column that keeps the length of the string, and make the length AND the string value as a composite unique key constraint.
In SQL Server 2012, you can even make the length column as a computed column so that you don't have to worry about the value at all. See http://sqlfiddle.com/#!6/32e94 for an example with SQL Server 2012. (I bet something similar is possible in MySQL.)
You probably need to read about the differences between VARCHAR and CHAR types.
The CHAR and VARCHAR Types
When CHAR values are stored, they are right-padded with spaces to the specified length. When CHAR values are retrieved, trailing spaces are removed unless the PAD_CHAR_TO_FULL_LENGTH SQL mode is enabled.
For VARCHAR columns, trailing spaces in excess of the column length are truncated prior to insertion and a warning is generated, regardless of the SQL mode in use. For CHAR columns, truncation of excess trailing spaces from inserted values is performed silently regardless of the SQL mode.
VARCHAR values are not padded when they are stored. Trailing spaces are retained when values are stored and retrieved, in conformance with standard SQL.
Conclusion: if you want to retain whitespace on the right side of a text string, use the CHAR type (and not VARCHAR).
Thanks to #kennethc. His answer works for me.
Add a string length field to the table and to the unique key.
CREATE TABLE strings
( id bigint(20) unsigned NOT NULL AUTO_INCREMENT,
string varchar(255) COLLATE utf8_bin NOT NULL,
created_ts timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
string_length int(3),
PRIMARY KEY (id), UNIQUE KEY string (string,string_length) )
ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_bin
In MySQL it's possible to update the string length field with couple of triggers like this:
CREATE TRIGGER `string_length_insert` BEFORE INSERT ON `strings` FOR EACH ROW SET NEW.string_length = char_length(NEW.string);
CREATE TRIGGER `string_length_update` BEFORE UPDATE ON `strings` FOR EACH ROW SET NEW.string_length = char_length(NEW.string);
I have seen this several times. I have one server that allows me to insert some of the values, without specifying the others like so: INSERT INTO table SET value_a='a', value_b='b'; (value_c is a field that does not have a default value set, but it works fine here). When the script is moved to a new server some INSERT queries break because it requires the query to specify all non-default values, giving me the following error for the first occurrence of not specifying a non-default value:
#1364 - Field 'value_c' doesn't have a default value
Setting default values for the table might break functionality in other areas, otherwise I would just do that. I would love to know what exactly is going on here.
One of your servers is running in strict mode by default and the other not.
If a server runs in strict mode (or you set it in your connection) and you try to insert a NULL value into a column defined as NOT NULL you will get #1364 error. Without strict mode your NULL value will be replaced with empty string or 0.
Example:
CREATE TABLE `test_tbl` (
`id` int(11) NOT NULL,
`someint` int(11) NOT NULL,
`sometext` varchar(255) NOT NULL,
`somedate` datetime NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
SET sql_mode = '';
INSERT INTO test_tbl(id) VALUES(1);
SELECT * FROM test_tbl;
+----+---------+----------+---------------------+
| id | someint | sometext | somedate |
+----+---------+----------+---------------------+
| 1 | 0 | | 0000-00-00 00:00:00 |
+----+---------+----------+---------------------+
SET sql_mode = 'STRICT_ALL_TABLES';
INSERT INTO test_tbl(id) VALUES(2);
#1364 - Field 'someint' doesn't have a default value
INSERT INTO table SET value_a='a', value_b='b'
Will perform the following query:
On the following table
TABLE `table`
a varchar
b varchar
c varchar NOT NULL -- column is listed as NOT NULL !
And you are doing the following query on it
INSERT INTO `table` (a,b,c) VALUES ('a', 'b', null)
So you are trying to insert a null value in a not null column that does not have a default value listed. This is why you are getting the error.
But why am I getting this error on server B and not on server A
Possible reasons are:
The create table statements are not the same: do a show create table table1 on each server and compare the output carefully.
There is a before insert trigger present on server A but not on server B (or visa versa) that is changing the input you are inserting. do a show triggers in databasex on each server and compare the output.
The table on server A is using a different engine from server B. This will show up in the output of show create table as the last line e.g. ENGINE=MyISAM
See: http://dev.mysql.com/doc/refman/5.0/en/show-triggers.html
http://dev.mysql.com/doc/refman/5.0/en/show-create-table.html