I am developing with mariadb and Spring, JdbcTemplate.
At first, we made DB charset as utf8, but now we have to change it into utf8mb4 because of emojis.
Till now I update individual charset with query something like below.
ALTER TABLE WT_WORKS CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE WT_WORKS CHANGE WORKS_TITLE WORKS_TITLE VARCHAR(300) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE WT_WORKS CHANGE WORKS_DESC WORKS_DESC VARCHAR(500) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
However It is now efficient because of table relations. For example when I insert into WT_WORKS, also need to into WT_WRITERS. It looks impossible to find every tables and columns.
So I want to know change these at once.(Including Procedures and Functions). -- something like (v_name VARCHAR(10)) into (v_name VARCHAR(10) CHARSET utf8mb4).
Thanks for answer.
FYI. my my.cnf got follow settings
[client]
default-character-set=utf8mb4
[mysqld]
collation-server = utf8mb4_unicode_ci
character-set-server = utf8mb4
[client]
default-character-set = utf8mb4
SELECT DISTINCT TABLE_SCHEMA, TABLE_NAME
FROM information_schema.COLUMNS
WHERE CHARACTER_SET_NAME = 'utf8'
will list all the tables that still have some column set to utf8. When them, do ALTER TABLE .. CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci;
This should generate all the ALTERs you need:
SELECT DISTINCT
CONCAT(
"ALTER TABLE ", TABLE_SCHEMA, ".", TABLE_NAME,
" CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci;"
)
FROM information_schema.COLUMNS
WHERE CHARACTER_SET_NAME = 'utf8'
Then copy and paste them into the mysql command line tool.
Caveat: If you have some columns with charsets other than utf8, they will be blindly converted to utf8mb4. This would be bad for hex, ascii, etc., columns, such as country_code, uuid, md5, etc.
You could do something similar to change individual columns instead.
You do not need to do both.
I have an existing table and I want to convert the charset only for one specific column to utf-8.
I know that this command ALTER TABLE table_name CONVERT TO CHARACTER SET utf8 does it for the whole table but I'm looking for a column-specific command.
Is there a command for that?
Try this:
ALTER TABLE t MODIFY col1 CHAR(50) CHARACTER SET utf8mb4;
I share that, it can always help...
I modified a database recently; moving from utf8 to utf8mb4; here is the script that allowed me to generate the alters...
Generate SQL commands to alter the tables:
SELECT CONCAT("ALTER TABLE `",`TABLE_NAME`,"` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;")
FROM `information_schema`.`TABLES`
WHERE `TABLE_SCHEMA` = 'xxxx';
Generate SQL commands to alter each column:
SELECT CONCAT("ALTER TABLE `",`TABLE_NAME`,"` MODIFY `",`COLUMN_NAME`,"` ",COLUMN_TYPE," CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci ",IF(`IS_NULLABLE`='YES', 'NULL', 'NOT NULL')," ",IF(`COLUMN_DEFAULT` IS NOT NULL, CONCAT(" DEFAULT '", `COLUMN_DEFAULT`, "'"), ''),";")
FROM `information_schema`.`COLUMNS`
WHERE `TABLE_SCHEMA` = 'xxx' AND `TABLE_NAME` = 'xxxx' AND (`CHARACTER_SET_NAME` IS NOT NULL OR `COLLATION_NAME` IS NOT NULL);
Note that for foreign keys and primary keys that make a relationship, you will need to disable foregin key checks before modifying the column
SET FOREIGN_KEY_CHECKS=0;
and enable afterwards.
SET FOREIGN_KEY_CHECKS=1;
Below one worked for me.
ALTER TABLE table_name
MODIFY column_name VARCHAR(255) CHARACTER SET utf8 COLLATE utf8_unicode_ci;
This one worked for me.
ALTER TABLE `table_name` CHANGE `column_name` `column_name` TEXT CHARACTER SET utf8 COLLATE utf8mb4_unicode_ci;
How can I convert entire MySQL database character-set to UTF-8 and collation to UTF-8?
Use the ALTER DATABASE and ALTER TABLE commands.
ALTER DATABASE databasename CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE tablename CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
Or if you're still on MySQL 5.5.2 or older which didn't support 4-byte UTF-8, use utf8 instead of utf8mb4:
ALTER DATABASE databasename CHARACTER SET utf8 COLLATE utf8_unicode_ci;
ALTER TABLE tablename CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;
Make a backup!
Then you need to set the default char sets on the database. This does not convert existing tables, it only sets the default for newly created tables.
ALTER DATABASE dbname CHARACTER SET utf8 COLLATE utf8_general_ci;
Then, you will need to convert the char set on all existing tables and their columns. This assumes that your current data is actually in the current char set. If your columns are set to one char set but your data is really stored in another then you will need to check the MySQL manual on how to handle this.
ALTER TABLE tbl_name CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
On the commandline shell
If you're one the commandline shell, you can do this very quickly. Just fill in "dbname" :D
DB="dbname"
(
echo 'ALTER DATABASE `'"$DB"'` CHARACTER SET utf8 COLLATE utf8_general_ci;'
mysql "$DB" -e "SHOW TABLES" --batch --skip-column-names \
| xargs -I{} echo 'ALTER TABLE `'{}'` CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;'
) \
| mysql "$DB"
One-liner for simple copy/paste
DB="dbname"; ( echo 'ALTER DATABASE `'"$DB"'` CHARACTER SET utf8 COLLATE utf8_general_ci;'; mysql "$DB" -e "SHOW TABLES" --batch --skip-column-names | xargs -I{} echo 'ALTER TABLE `'{}'` CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;' ) | mysql "$DB"
You can create the sql to update all tables with:
SELECT CONCAT("ALTER TABLE ",TABLE_SCHEMA,".",TABLE_NAME," CHARACTER SET utf8 COLLATE utf8_general_ci; ",
"ALTER TABLE ",TABLE_SCHEMA,".",TABLE_NAME," CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci; ")
AS alter_sql
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = "your_database_name";
Capture the output and run it.
Arnold Daniels' answer above is more elegant.
Before proceeding, ensure that you: Have completed a full database backup!
Step 1: Database Level Changes
Identifying the Collation and Character set of your database
SELECT DEFAULT_CHARACTER_SET_NAME, DEFAULT_COLLATION_NAME FROM
information_schema.SCHEMATA S
WHERE schema_name = 'your_database_name'
AND
(DEFAULT_CHARACTER_SET_NAME != 'utf8'
OR
DEFAULT_COLLATION_NAME not like 'utf8%');
Fixing the collation for the database
ALTER DATABASE databasename CHARACTER SET utf8 COLLATE utf8_unicode_ci;
Step 2: Table Level Changes
Identifying Database Tables with the incorrect character set or collation
SELECT CONCAT(
'ALTER TABLE ', table_name, ' CHARACTER SET utf8 COLLATE utf8_general_ci; ',
'ALTER TABLE ', table_name, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci; ')
FROM information_schema.TABLES AS T, information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` AS C
WHERE C.collation_name = T.table_collation
AND T.table_schema = 'your_database_name'
AND
(C.CHARACTER_SET_NAME != 'utf8'
OR
C.COLLATION_NAME not like 'utf8%')
Adjusting table columns' collation and character set
Capture upper sql output and run it. (like following)
ALTER TABLE rma CHARACTER SET utf8 COLLATE utf8_general_ci;ALTER TABLE rma CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
ALTER TABLE rma_history CHARACTER SET utf8 COLLATE utf8_general_ci;ALTER TABLE rma_history CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
ALTER TABLE rma_products CHARACTER SET utf8 COLLATE utf8_general_ci;ALTER TABLE rma_products CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
ALTER TABLE rma_report_period CHARACTER SET utf8 COLLATE utf8_general_ci;ALTER TABLE rma_report_period CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
ALTER TABLE rma_reservation CHARACTER SET utf8 COLLATE utf8_general_ci;ALTER TABLE rma_reservation CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
ALTER TABLE rma_supplier_return CHARACTER SET utf8 COLLATE utf8_general_ci;ALTER TABLE rma_supplier_return CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
ALTER TABLE rma_supplier_return_history CHARACTER SET utf8 COLLATE utf8_general_ci;ALTER TABLE rma_supplier_return_history CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
ALTER TABLE rma_supplier_return_product CHARACTER SET utf8 COLLATE utf8_general_ci;ALTER TABLE rma_supplier_return_product CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
refer to: https://confluence.atlassian.com/display/CONFKB/How+to+Fix+the+Collation+and+Character+Set+of+a+MySQL+Database
Use HeidiSQL. Its free and a very good db tool.
From tools menu, enter Bulk table editor
Select the complete database or pick tables to convert,
tick Change default collation: utf8mb4_general_ci
tick Convert to charset: utf8
Execute
This converts complete database from latin to utf8 in just a few seconds.
Works like a charm :)
HeidiSQL connects by default as utf8 so any special characters should now be seen as the character (æ ø å) and not as encoded when inspecting the table data.
The real pitfall when moving from latin to utf8 is to make sure pdo connects with utf8 charset. If not you will get rubbish data inserted to the utf8 table and question marks all over the place on your web page, making you think the table data is not utf8...
DELIMITER $$
CREATE PROCEDURE `databasename`.`update_char_set`()
BEGIN
DECLARE done INT DEFAULT 0;
DECLARE t_sql VARCHAR(256);
DECLARE tableName VARCHAR(128);
DECLARE lists CURSOR FOR SELECT table_name FROM `information_schema`.`TABLES` WHERE table_schema = 'databasename';
DECLARE CONTINUE HANDLER FOR SQLSTATE '02000' SET done = 1;
OPEN lists;
FETCH lists INTO tableName;
REPEAT
SET #t_sql = CONCAT('ALTER TABLE ', tableName, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci');
PREPARE stmt FROM #t_sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
FETCH lists INTO tableName;
UNTIL done END REPEAT;
CLOSE lists;
END$$
DELIMITER ;
CALL databasename.update_char_set();
Inspired by #sdfor comment, here is a bash script that does the job
#!/bin/bash
printf "### Converting MySQL character set ###\n\n"
printf "Enter the encoding you want to set: "
read -r CHARSET
# Get the MySQL username
printf "Enter mysql username: "
read -r USERNAME
# Get the MySQL password
printf "Enter mysql password for user %s:" "$USERNAME"
read -rs PASSWORD
DBLIST=( mydatabase1 mydatabase2 )
printf "\n"
for DB in "${DBLIST[#]}"
do
(
echo 'ALTER DATABASE `'"$DB"'` CHARACTER SET utf8 COLLATE `'"$CHARSET"'`;'
mysql "$DB" -u"$USERNAME" -p"$PASSWORD" -e "SHOW TABLES" --batch --skip-column-names \
| xargs -I{} echo 'ALTER TABLE `'{}'` CONVERT TO CHARACTER SET utf8 COLLATE `'"$CHARSET"'`;'
) \
| mysql "$DB" -u"$USERNAME" -p"$PASSWORD"
echo "$DB database done..."
done
echo "### DONE ###"
exit
In case the data is not in the same character set you might consider this snippet from http://dev.mysql.com/doc/refman/5.0/en/charset-conversion.html
If the column has a nonbinary data type (CHAR, VARCHAR, TEXT), its
contents should be encoded in the column character set, not some other
character set. If the contents are encoded in a different character
set, you can convert the column to use a binary data type first, and
then to a nonbinary column with the desired character set.
Here is an example:
ALTER TABLE t1 CHANGE c1 c1 BLOB;
ALTER TABLE t1 CHANGE c1 c1 VARCHAR(100) CHARACTER SET utf8;
Make sure to choose the right collation, or you might get unique key conflicts. e.g.
Éleanore and Eleanore might be considered the same in some collations.
Aside:
I had a situation where certain characters "broke" in emails even though they were stored as UTF-8 in the database. If you are sending emails using utf8 data, you might want to also convert your emails to send in UTF8.
In PHPMailer, just update this line: public $CharSet = 'utf-8';
For databases that have a high number of tables you can use a simple php script to update the charset of the database and all of the tables using the following:
$conn = mysqli_connect($host, $username, $password, $database);
if ($conn->connect_error) {
die("Connection failed: " . $conn->connect_error);
}
$alter_database_charset_sql = "ALTER DATABASE ".$database." CHARACTER SET utf8 COLLATE utf8_unicode_ci";
mysqli_query($conn, $alter_database_charset_sql);
$show_tables_result = mysqli_query($conn, "SHOW TABLES");
$tables = mysqli_fetch_all($show_tables_result);
foreach ($tables as $index => $table) {
$alter_table_sql = "ALTER TABLE ".$table[0]." CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci";
$alter_table_result = mysqli_query($conn, $alter_table_sql);
echo "<pre>";
var_dump($alter_table_result);
echo "</pre>";
}
The safest way is to modify the columns first to a binary type and then modify it back to it type using the desired charset.
Each column type have its respective binary type, as follows:
CHAR => BINARY
TEXT => BLOB
TINYTEXT => TINYBLOB
MEDIUMTEXT => MEDIUMBLOB
LONGTEXT => LONGBLOB
VARCHAR => VARBINARY
Eg.:
ALTER TABLE [TABLE_SCHEMA].[TABLE_NAME] MODIFY [COLUMN_NAME] VARBINARY;
ALTER TABLE [TABLE_SCHEMA].[TABLE_NAME] MODIFY [COLUMN_NAME] VARCHAR(140) CHARACTER SET utf8mb4;
I tried in several latin1 tables and it kept all the diacritics.
You can extract this query for all columns doing this:
SELECT
CONCAT('ALTER TABLE ', TABLE_SCHEMA,'.', TABLE_NAME,' MODIFY ', COLUMN_NAME,' VARBINARY;'),
CONCAT('ALTER TABLE ', TABLE_SCHEMA,'.', TABLE_NAME,' MODIFY ', COLUMN_NAME,' ', COLUMN_TYPE,' CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;')
FROM information_schema.columns
WHERE TABLE_SCHEMA IN ('[TABLE_SCHEMA]')
AND COLUMN_TYPE LIKE 'varchar%'
AND (COLLATION_NAME IS NOT NULL AND COLLATION_NAME NOT LIKE 'utf%');
After you do this on all your columns then you do it on all tables:
ALTER TABLE [TABLE_SCHEMA].[TABLE_NAME] CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
To generate this query for all your table, use the following query:
SELECT
CONCAT('ALTER TABLE ', TABLE_SCHEMA, '.', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;')
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_COLLATION NOT LIKE 'utf8%'
and TABLE_SCHEMA in ('[TABLE_SCHEMA]');
And now that you modified all your columns and tables, do the same on the database:
ALTER DATABASE [DATA_BASE_NAME] CHARSET = utf8mb4 COLLATE = utf8mb4_general_ci;
mysqldump -uusername -ppassword -c -e --default-character-set=utf8 --single-transaction --skip-set-charset --add-drop-database -B dbname > dump.sql
cp dump.sql dump-fixed.sql
vim dump-fixed.sql
:%s/DEFAULT CHARACTER SET latin1/DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci/
:%s/DEFAULT CHARSET=latin1/DEFAULT CHARSET=utf8/
:wq
mysql -uusername -ppassword < dump-fixed.sql
from utf8 to utf8mb4:
1.show all DATABASE default characterset:
SELECT SCHEMA_NAME 'YOUR_DATABASE_NAME',
default_character_set_name 'charset',
DEFAULT_COLLATION_NAME 'collation'
FROM information_schema.SCHEMATA;
2.show all tables status(character set), focus on column 'collation':
use YOUR_DATABASE_NAME;
SHOW TABLE STATUS ;
3.generate convert sql:
convert database & all tables to utf8mb4,utf8mb4_unicode_ci
USE information_schema;
SELECT CONCAT("ALTER DATABASE `",table_schema,"` CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;") AS _sql
FROM `TABLES` WHERE table_schema LIKE "YOUR_DATABASE_NAME" AND TABLE_TYPE='BASE TABLE' GROUP BY table_schema UNION
SELECT CONCAT("ALTER TABLE `",table_schema,"`.`",table_name,"` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;") AS _sql
FROM `TABLES` WHERE table_schema LIKE "YOUR_DATABASE_NAME" AND TABLE_TYPE='BASE TABLE' GROUP BY table_schema, TABLE_NAME
/*include all columns, commonly don't need this.*/
/*
UNION
SELECT CONCAT("ALTER TABLE `",`COLUMNS`.table_schema,"`.`",`COLUMNS`.table_name, "` CHANGE `",column_name,"` `",column_name,"` ",data_type,"(",character_maximum_length,") CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci",IF(is_nullable="YES"," NULL"," NOT NULL"),";") AS _sql
FROM `COLUMNS` INNER JOIN `TABLES` ON `TABLES`.table_name = `COLUMNS`.table_name WHERE `COLUMNS`.table_schema like "YOUR_DATABASE_NAME" and data_type in ('varchar','char') AND TABLE_TYPE='BASE TABLE' UNION
SELECT CONCAT("ALTER TABLE `",`COLUMNS`.table_schema,"`.`",`COLUMNS`.table_name, "` CHANGE `",column_name,"` `",column_name,"` ",data_type," CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci",IF(is_nullable="YES"," NULL"," NOT NULL"),";") AS _sql
FROM `COLUMNS` INNER JOIN `TABLES` ON `TABLES`.table_name = `COLUMNS`.table_name WHERE `COLUMNS`.table_schema like "YOUR_DATABASE_NAME" and data_type in ('text','tinytext','mediumtext','longtext') AND TABLE_TYPE='BASE TABLE';
*/
4.run the sql generated.
5.refresh your database.
6.check:
SHOW TABLE STATUS ;
If you cannot get your tables to convert or your table is always set to some non-utf8 character set, but you want utf8, your best bet might be to wipe it out and start over again and explicitly specify:
create database database_name character set utf8;
To change the character set encoding to UTF-8 for the database itself, type the following command at the mysql> prompt. Replace DBNAME with the database name:
ALTER DATABASE DBNAME CHARACTER SET utf8 COLLATE utf8_general_ci;
The only solution that worked for me: http://docs.moodle.org/23/en/Converting_your_MySQL_database_to_UTF8
Converting a database containing tables
mysqldump -uusername -ppassword -c -e --default-character-set=utf8 --single-transaction --skip-set-charset --add-drop-database -B dbname > dump.sql
cp dump.sql dump-fixed.sql
vim dump-fixed.sql
:%s/DEFAULT CHARACTER SET latin1/DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci/
:%s/DEFAULT CHARSET=latin1/DEFAULT CHARSET=utf8/
:wq
mysql -uusername -ppassword < dump-fixed.sql
alter table table_name charset = 'utf8';
This is a simple query i was able to use for my case, you can change the table_name as per your requirement(s).
You can also DB tool Navicat, which does it more easier.
Siva.
Right Click Your Database & select DB Properties & Change as you desired in Drop Down
Command Line Solution and Exclude Views
I am simply completing #Jasny's answer for others like #Brian and I who have views in our database.
If you have an error like this:
ERROR 1347 (HY000) at line 17: 'dbname.table_name' is not of type 'BASE TABLE'
It's because you probably have views and you need to exclude them.
But when trying to exclude them, MySQL returns 2 columns instead of 1.
SHOW FULL TABLES WHERE Table_Type = 'BASE TABLE';
-- table_name1 BASE TABLE
-- table_name2 BASE TABLE
So we have to adapt Jasny's command with awk to extract only the 1st column which contains the table name.
DB="dbname"
(
echo 'ALTER DATABASE `'"$DB"'` CHARACTER SET utf8 COLLATE utf8_general_ci;'
mysql "$DB" -e "SHOW FULL TABLES WHERE Table_Type = 'BASE TABLE'" --batch --skip-column-names \
| awk '{print $1 }' \
| xargs -I{} echo 'ALTER TABLE `'{}'` CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;'
) \
| mysql "$DB"
One-liner for simple copy/paste
DB="dbname"; ( echo 'ALTER DATABASE `'"$DB"'` CHARACTER SET utf8 COLLATE utf8_general_ci;'; mysql "$DB" -e "SHOW FULL TABLES WHERE Table_Type = 'BASE TABLE'" --batch --skip-column-names | awk '{print $1 }' | xargs -I{} echo 'ALTER TABLE `'{}'` CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;' ) | mysql "$DB"
To change the character set encoding to UTF-8 follow simple steps in PHPMyAdmin
Select your Database
Go To Operations
In operations tab, on the bottom collation drop down menu, select you desire encoding i.e(utf8_general_ci), and also check the checkbox (1)change all table collations, (2) Change all tables columns collations. and hit Go.