MySQL strict string comparison the semantically correct way? - mysql

Question

The **compare-on-binary way** is **NOT semantically-correct**
For example, when you want to do strict-string-comparison with different encoded strings, the compare-on-binary way's mistake comes out. The following test case illustrates why:
In this case, I want to replace the string '北京 ' (with a tailling space) in the field 城市 into string '北京111', but keep the string '北京' unchanged, so I write down the following sql:
SELECT CASE WHEN BINARY `城市` = BINARY '北京 ' THEN '北京111' ELSE `城市` END
FROM `中文测试表1`
GROUP BY BINARY CASE WHEN BINARY `城市` = BINARY '北京 ' THEN '北京111' ELSE `城市` END
The underlying table definition and data (the session encoding is setted to 'utf8mb4'):
CREATE TABLE `中文测试表1` (
`城市` varchar(50) CHARACTER SET gbk DEFAULT NULL,
`销量` int(11) DEFAULT NULL
) ENGINE=InnoDB;
INSERT INTO `中文测试表1` VALUES ('杭州', '111');
INSERT INTO `中文测试表1` VALUES ('北京', '345');
INSERT INTO `中文测试表1` VALUES ('北京 ', '123');
And what actually happened is that the string '北京 ' is not replaced by '北京111' and still keep what it was in the result set.
The reason is, the string literal '北京 ' is encoded using utf8mb4 (which is decided by the session) and the string value '北京 ' from field 城市 is encoded using gbk (which is decided by the table definition), and when they are converted to binary, they are not the same every byte, but the two string is indeed semantically equal every character (no matter what the underlying encoding method is used).
So, what is the semantically correct way to compare strings strictly in MySQL?

Answer 1

See the TRIM() function for removing spaces from the start/end of strings.
Converting between gbk and utf8mb4 leaves you at the mercy of the conversion tables; you may (or may not) get the desired transliteration.
'北京' is HEX E58C97 E4BAAC for utf8/utf8mb4
'北京 ' is HEX E58C97 E4BAAC 20 for utf8/utf8mb4 -- as found in the query
'北京' is HEX B1B1 BEA9 for gbk
'北京 ' is HEX B1B1 BEA9 20 for gbk -- as found in the table
When you say SELECT ... BINARY '北京 ' ..., the encoding for the string is based on the connection, not the column encoding. So it is utf8mb4.
Instead of ... WHEN BINARY 城市 = BINARY '北京 ' THEN ..., do one of these:
Plan A, Let the conversions happen automatically: ... WHEN 城市 = '北京 ' THEN ...
Plan B, Explicitly convert: ... WHEN 城市 = CONVERT('北京 ' USING gbk) THEN ...
Plan C, Use HEX: ... WHEN HEX(城市) = HEX(CONVERT('北京 ' USING gbk)) THEN ...
Plan D, closer to your attempt: ... WHEN BINARY 城市 = BINARY(CONVERT('北京 ' USING gbk)) THEN ...
There are other ways, using COLLATE utf8_bin, COLLATE gbk_bin, etc.