strange "Duplicate entry" error of mysql - mysql

Question

This might be an unicode related problem.
I have a mysql source file:
set names utf8;
STATUS;
drop table tianya;
create table tianya ( name char(50) not null primary key, passwd char (50) not null, email char(50));
insert into tianya values ("■■■■■■■■","68221",""),("12345678","098",""),("〡〢〣〤〥〦〧〨","1","");
And when I run this sql file, mysql reports:
ERROR 1062 (23000) at line 5: Duplicate entry '〡〢〣〤〥〦〧〨' for key 'PRIMARY'
as this image:
However, as we can see, the table is whole new, and the 3 primary keys are different from each other.
So, what causes this error?
=======add some extra info on 2015-Aug-28
mysql> create table tianya ( name char(50) not null primary key, passwd char (50) not null, email char(50));
Query OK, 0 rows affected (0.16 sec)
mysql>
mysql>
mysql> insert into tianya values ("■■■■■■■■","68221",""),("12345678","098","");
Query OK, 2 rows affected (0.19 sec)
Records: 2 Duplicates: 0 Warnings: 0
mysql> select name,hex(name) from tianya;
+--------------------------+--------------------------------------------------+
| name | hex(name) |
+--------------------------+--------------------------------------------------+
| ■■■■■■■■ | E296A0E296A0E296A0E296A0E296A0E296A0E296A0E296A0 |
| 12345678 | 3132333435363738 |
+--------------------------+--------------------------------------------------+
2 rows in set (0.03 sec)
mysql> insert into tianya values ("1234567","098",""),("〡〢〣〤〥〦〧〨","1","");
ERROR 1062 (23000): Duplicate entry '〡〢〣〤〥〦〧〨' for key 'PRIMARY'
mysql> delete from tianya;
Query OK, 2 rows affected (0.02 sec)
mysql> insert into tianya values ("1234567","098",""),("〡〢〣〤〥〦〧〨","1","");
Query OK, 2 rows affected (0.00 sec)
Records: 2 Duplicates: 0 Warnings: 0
mysql> select name,hex(name) from tianya;
+--------------------------+--------------------------------------------------+
| name | hex(name) |
+--------------------------+--------------------------------------------------+
| 1234567 | 31323334353637 |
| 〡〢〣〤〥〦〧〨 | E380A1E380A2E380A3E380A4E380A5E380A6E380A7E380A8 |
+--------------------------+--------------------------------------------------+
2 rows in set (0.00 sec)
mysql> describe tianya;
+--------+----------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------+----------+------+-----+---------+-------+
| name | char(50) | NO | PRI | NULL | |
| passwd | char(50) | NO | | NULL | |
| email | char(50) | YES | | NULL | |
+--------+----------+------+-----+---------+-------+
3 rows in set (0.05 sec)
mysql> show create table tianya ( name char(50) not null primary key, passwd char (50) not null, email char(50));
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '( name char(50) not null primary key, passwd char (50) not null, email char(50))' at line 1
mysql> show create table tianya;
+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tianya | CREATE TABLE `tianya` (
`name` char(50) COLLATE utf8_unicode_ci NOT NULL,
`passwd` char(50) COLLATE utf8_unicode_ci NOT NULL,
`email` char(50) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
+--------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
====and another test:
xungeng#fl-ubuntu:~/tmp/a$ mysql -u root test < a.sql
Table Create Table
t2 CREATE TABLE `t2` (\n `name` varchar(10) COLLATE utf8_unicode_ci NOT NULL,\n UNIQUE KEY `name` (`name`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
ERROR 1062 (23000) at line 5: Duplicate entry 'hì' for key 'name'
xungeng#fl-ubuntu:~/tmp/a$ cat a.sql
set names utf8;
drop table if exists t2;
create table t2 (name varchar(10) not null unique key);
show create table t2;
insert into t2 values ("hí"),("hì");

Answer 1

Finally I know what had happened.
The collate utf_unicode_ci is case insensitive. This makes an insert of primary or unique key of 'TEST' failed if there is a key 'test' or 'Test'. I thought the collision is caused by '■■■■■■■■' and '〡〢〣〤〥〦〧〨', this misled you and also me. In fact the collision is caused by '12345678' and '〡〢〣〤〥〦〧〨'.
The symbols '〡〢〣〤〥〦〧〨' look so curious. They are ancient chinese symbols for number 12345678, respectively. [refer: https://zh.wikipedia.org/wiki/%E7%AD%B9%E7%AE%97 (chinese edition only)]
To resolve this problem, I changed utf8_unicode_ci to utf8_bin.

Answer 2

You can add to the charset of the table in the create statement. Append 'Set charset=utf8'. Maybe that helps. See http://dev.mysql.com/doc/refman/5.1/en/create-table.html
Section table-options