I am trying to insert emoji's into mysql but it turns to question marks, I have changed mysql connection server collation, database collation , table collation and column collation. I used these to change the items
# For each database:
ALTER DATABASE database_name CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
# For each table:
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
# For each column:
ALTER TABLE table_name CHANGE column_name column_name VARCHAR(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
I have done all these but emoji's in mysql still show question marks. Please what should I do to make mysql show the emojis. Thanks in advance
Little late to answer the question. But I hope it will be useful for others...
Above configuration makes the database tables to store utf8 encoded data. But, the database connection(JDBC) should be able to transfer the utf8 encoded data to client. For that the JDBC connection parameter charset should be set to utf8mb4.
The default encoding for inbound connections isn't set properly. DEFAULT CHARSET will return as utf8 however character_set_server will be something different.
So, Set default-character-set=utf8.
Related
Here is the command I use:
ALTER TABLE <table_name> MODIFY <column_name> VARCHAR(255) CHARACTER SET utf8 COLLATE utf8_unicode_ci;
It works well. Now I need to set utf8mb4_unicode_ci for a column (since currently characters are shown as ???). Anyway here is my new command:
ALTER TABLE <table_name> MODIFY <column_name> VARCHAR(255) CHARACTER SET utf8 COLLATE utf8mb4_unicode_ci;
But sadly MySQL throws:
ERROacR 1253 (42000): COLLATION 'utf8mb4_unicode_ci' is not valid for CHARACTER
Any idea?
The first part of the COLLATION name must match the CHARACTER SET name.
CHARACTER SET utf8mb4 is needed for Emoji and some Chinese characters.
Let's back up to the 'real' problem -- of question marks.
COLLATION refers to the rules of ordering and sorting, not encoding.
CHARACTER SET refers to the encoding. This should be consistent at all stages. Question Marks come from inconsistencies.
Trouble with UTF-8 characters; what I see is not what I stored points out that these are the likely suspects for Question Marks:
The bytes to be stored are not encoded as utf8/utf8mb4. Fix this.
The column in the database is not CHARACTER SET utf8mb4. Fix this if you need 4-byte UTF-8. (Use SHOW CREATE TABLE.)
Also, check that the connection during reading is UTF-8. The details depend on the application doing the connecting.
This worked for me:
ALTER TABLE <table_name> MODIFY <column_name> VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
I am developing with mariadb and Spring, JdbcTemplate.
At first, we made DB charset as utf8, but now we have to change it into utf8mb4 because of emojis.
Till now I update individual charset with query something like below.
ALTER TABLE WT_WORKS CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE WT_WORKS CHANGE WORKS_TITLE WORKS_TITLE VARCHAR(300) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE WT_WORKS CHANGE WORKS_DESC WORKS_DESC VARCHAR(500) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
However It is now efficient because of table relations. For example when I insert into WT_WORKS, also need to into WT_WRITERS. It looks impossible to find every tables and columns.
So I want to know change these at once.(Including Procedures and Functions). -- something like (v_name VARCHAR(10)) into (v_name VARCHAR(10) CHARSET utf8mb4).
Thanks for answer.
FYI. my my.cnf got follow settings
[client]
default-character-set=utf8mb4
[mysqld]
collation-server = utf8mb4_unicode_ci
character-set-server = utf8mb4
[client]
default-character-set = utf8mb4
SELECT DISTINCT TABLE_SCHEMA, TABLE_NAME
FROM information_schema.COLUMNS
WHERE CHARACTER_SET_NAME = 'utf8'
will list all the tables that still have some column set to utf8. When them, do ALTER TABLE .. CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci;
This should generate all the ALTERs you need:
SELECT DISTINCT
CONCAT(
"ALTER TABLE ", TABLE_SCHEMA, ".", TABLE_NAME,
" CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci;"
)
FROM information_schema.COLUMNS
WHERE CHARACTER_SET_NAME = 'utf8'
Then copy and paste them into the mysql command line tool.
Caveat: If you have some columns with charsets other than utf8, they will be blindly converted to utf8mb4. This would be bad for hex, ascii, etc., columns, such as country_code, uuid, md5, etc.
You could do something similar to change individual columns instead.
You do not need to do both.
I cannot insert Chinese in the table. I have already config the charset and collation as utf8 and utf8_general_ci. The database and table is created as utf8.
Also, I have tried charset as utf8mb4 and collation as utf8_unicode_ci, utf8mb4_general_ci, utf8mb4_unicode_ci. All of them point to same error : warning 1366 Incorrect String value.
Big5 and GBK encoding work in this case but I would like to use utf8, as simplified, traditional chinese will be allow to insert.
some of the code in mysql:
create database db1 DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
create table table1 (
lastName varchar(255),
firstName varchar(255)
) ENGINE = MYISAM CHARSET=utf8 COLLATE utf8_general_ci;
INSERT INTO table1 VALUES ('陳','小文');
Warning | 1366 | Incorrect string value: '\xB3\xAF' for column 'lastName' at r
w 1 |
Warning | 1366 | Incorrect string value: '\xA4p\xA4\xE5' for column 'firstName
at row 1 |
configuration in my.ini:
[client]
default-character-set=utf8
[mysqld]
default-storage-engine=MyISAM
collation-server=utf8_general_ci
init-connect='SET NAMES utf8'
character-set-server=utf8
skip-character-set-client-handshake=1
[mysql]
default-character-set=utf8
Using Mysql 5.6.8 version.
CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci set this collation on three places:
1- In database collation under operation or setting tab you can find collation dropbox select "utf8mb4_general_ci" from there.
2- In table again under operation tab you need to change collation value from dropdown for same "utf8mb4_general_ci".
3- Don't forget now to change the collation of your column that hold your data click on structure tab of your table edit the columns that need to be hold Chinese data and change collation for each as same: "utf8mb4_general_ci".
Try to place chinese character directly from phpmyadmin if you can successfully insert chinese character that's means you're done with database then try with server or from your website. If doesn't work from wesbite, Then May you need to add utf 8 meta tag on website script.
I have a utf8_general_ci database that I'm interested in converting to utf8_unicode_ci.
I've tried the following commands
ALTER DATABASE dbname CHARACTER SET utf8 COLLATE utf8_unicode_ci;
ALTER TABLE tbl_name CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci; (for every single table)
But that seems to change the charset for future data but doesn't convert the actual existing data from utf8_general_ci to utf8_unicode_ci.
Is there any way to convert the existing data to utf8_unicode_ci?
SHOW CREATE TABLE to see if it really set the CHARACTER SET and COLLATION on the columns, not just the defaults.
What was the CHARACTER SET before the ALTERs?
Do SELECT col, HEX(col) ... for some field that should have utf8 in it. This will help us determine if you really have utf8 in the table. The encoding for characters is different based on CHARACTER SET; the HEX helps discover such.
The ordering (WHERE, ORDER BY, etc) is controlled by COLLATION. The indexes probably had to be rebuilt based on your ALTER TABLE. Did big tables with indexes take a 'long' time to convert?
To actually see the difference between utf8_general_ci and utf8_unicode_ci, you need a "combining accent" or, more simply, the German ß versus ss:
mysql> SELECT 'ß' = 'ss' COLLATE utf8_general_ci,
'ß' = 'ss' COLLATE utf8_unicode_ci;
+-------------------------------------+-------------------------------------+
| 'ß' = 'ss' COLLATE utf8_general_ci | 'ß' = 'ss' COLLATE utf8_unicode_ci |
+-------------------------------------+-------------------------------------+
| 0 | 1 |
+-------------------------------------+-------------------------------------+
However, to test that in your tables, you would need to store those values and use WHERE or GROUP_CONCAT or something else to determine the equality.
What 'proof' do you have that the ALTERs failed to achieve the collation change?
(Addressing other comments: REPAIR should be irrelevant. CONVERT TO tells the ALTER to actually modify the data, so it should have done the desired action.)
You have to change the collation of every field in every table. As you say, the collation of the table is only the default value for fields created later, and the collation of the database is only the default value for tables created later.
As Lorenz Meyer said, the collation of the table is only the default value for fields created later and you need to set the defaults for the columns explicitly too.
Such a change looks like:
ALTER TABLE mytable CHANGE mycolumn mycolumn varchar(15) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci
I have a table temp. I applied new character set and collation in MYSQL with following query:
ALTER TABLE temp CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin;
Now I want to revert this back to what the table was before I changed the attributes. Is there a way I can do that?
Thanks for help in advance.
Not without knowing what the previous value was. I think the default charset for mysql is latin1 and the default collation is case-insensitive (latin1_swedish_ci for latin1)