Related
I have a database I am using that has support for different languages, the issue I am running into is, in the source SQL data, the format is correct.
MariaDB [stmtransit]> SELECT * FROM routes WHERE route_id = 181;
+----------+-----------+------------------+------------------+------------+------------+------------------------------------------+-------------+------------------+
| route_id | agency_id | route_short_name | route_long_name | route_desc | route_type | route_url | route_color | route_text_color |
+----------+-----------+------------------+------------------+------------+------------+------------------------------------------+-------------+------------------+
| 181 | 1 | 369 | Côte-des-Neiges | NULL | 3 | http://www.stm.info/fr/infos/reseaux/bus | 009EE0 | NULL |
+----------+-----------+------------------+------------------+------------+------------+------------------------------------------+-------------+------------------+
1 row in set (0.00 sec)
When I move do the query and move it into CouchDB, it changes accents and anything other than plain characters to
Côte-des-Neiges
My request is
function queryRouteTable(db, route_id) {
return db.query({
sql: "SELECT * FROM routes WHERE route_id = ?;",
values: [route_id],
})
.take(1);
}
Then my upload to couch uses rx and rx-couch with the code, and no matter where I view the document.route_long_name after the initial grab, its always formatted wrong.
What am I missing, why does it change after initial grab.
To display the current character encoding set for a particular database, type the following command at the mysql> prompt. Replace DBNAME with the database name:
SELECT default_character_set_name FROM information_schema.SCHEMATA S WHERE schema_name = "DBNAME";
If you have your encoding set per table use the following command. Replace DBNAME with the database name, and TABLENAME with the name of the table:
SELECT CCSA.character_set_name FROM information_schema.`TABLES` T,information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA WHERE CCSA.collation_name = T.table_collation AND T.table_schema = "DBNAME" AND T.table_name = "TABLENAME";
IMPORTANT: BACKUP YOUR DATABASE
If you have a working backup of your database you can convert it from your current encoding to UTF-8 by issuing the following commands:
mysql --database=DBNAME -B -N -e "SHOW TABLES" | awk '{print "SET foreign_key_checks = 0; ALTER TABLE", $1, "CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci; SET foreign_key_checks = 1; "}' | mysql --database=DBNAME
And in prompt:
ALTER DATABASE DBNAME CHARACTER SET utf8 COLLATE utf8_general_ci;
Now you should be able to export using UTF-8 and import into couch using UTF-8 encoding...
Hope that helps...
Turns out MariaDB has a bug which turns the database formating to latin1 intead of utf8
To correct for this you must go to /etc/my.cnf
remove all instances of
default-character-set=utf8
find title "mysqld" in my.cnf and put under it
init_connect='SET collation_connection = utf8_unicode_ci'
init_connect='SET NAMES utf8'
character-set-server=utf8
collation-server=utf8_unicode_ci
skip-character-set-client-handshake
and save.
Then restart mariadb.
In a nutshell:
Within a Perl-Script: How do I connect to MySQL in a way that allows to transmit the four-byte unicode character U+1F61C ("😜") from the perl script to a MySQL-Table where this character should be stored?
Using {mysql_enable_utf8 => 1} doesn't solve the problem.
In detail:
I have exactly the same problem as described in the Question ERROR 1366 (HY000): Incorrect string value: '\xF0\x9F\x98\x9C' for column 'comment' at row 1 and even with the same Unicode character (😜 = U+1F61C = FACE WITH STUCK-OUT TONGUE AND WINKING EYE) which produces the error message
DBD::mysql::st execute failed: Incorrect string value: '\xF0\x9F\x98\x9C' for column ...
But I don't use PHP, I use Perl.
The accepted answer in the other question says:
Run MySQL 5.5 or later.
I check the version:
mysql> select version();
+-------------------------+
| version() |
+-------------------------+
| 5.7.13-0ubuntu0.16.04.2 |
+-------------------------+
So it is 5.7 which is later than 5.5.
✅checked
Set table's character to utf8mb4.
I check the character set of my database, my table and even of the reported column:
mysql> SELECT default_character_set_name FROM information_schema.SCHEMATA
-> WHERE schema_name = "myDatabase";
+----------------------------+
| default_character_set_name |
+----------------------------+
| utf8mb4 |
+----------------------------+
mysql> SELECT CCSA.character_set_name FROM information_schema.`TABLES` T,
-> information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA
-> WHERE CCSA.collation_name = T.table_collation
-> AND T.table_schema = "myDatabase"
-> AND T.table_name = "myTable";
+--------------------+
| character_set_name |
+--------------------+
| utf8mb4 |
+--------------------+
mysql> SELECT character_set_name FROM information_schema.`COLUMNS`
-> WHERE table_schema = "myDatabase"
-> AND table_name = "myTable"
-> AND column_name = "myColumn";
+--------------------+
| character_set_name |
+--------------------+
| utf8mb4 |
+--------------------+
So my database, my table and the reported column all use the character set utf8mb4.
✅checked
Enable UTF8 on your MySQL connection.
This seems to be the problem. The answer to the other question says:
SET NAMES utf8, or use an option when connecting that similarly enables it.
I don't know how to SET NAMES utf8 within a perl script, so I did it how I did it over the last years. I think that this is "an option when connecting that similarly enables it".
It's at the end of the long line that begins with my $dbh = DBI->connect:
#!/usr/bin/perl -w
use strict;
use warnings;
use utf8;
use Encode;
use DBI;
binmode STDOUT, ":utf8";
#Here I connect using the parameter mysql_enable_utf8 (create database handle):
my $dbh = DBI->connect('DBI:mysql:database=myDatabase;host=localhost','aUser','aPassword',{mysql_enable_utf8 => 1});
#Prepare the statement (create statement handle):
my $sth = $dbh->prepare('INSERT INTO `myTable` (`myColumn`) VALUES(?);');
#This doesn't work:
$sth->execute('😜');
#This doesn't work either:
$sth->execute(encode_utf8('😜'));
#end processing:
$dbh->disconnect();
exit(0);
Both executes throw the same error (only the line number at the end changes):
DBD::mysql::st execute failed: Incorrect string value: '\xF0\x9F\x98\x9C' for column 'myColumn' at row 1 at myTestScript.pl line 16.
What am I doing wrong?
How can I do it better?
The problem is with the SET NAMES utf8 command. In MySQL the utf8 character set is not truly utf8, it supports characters up 3 bytes only and the character in question has 4 bytes:
The utf8 character set in MySQL has these characteristics:
• No support for supplementary characters (BMP characters only).
• A maximum of three bytes per multibyte character.
The true utf8 is the utf8mb4 that you use as character set in the field itself. So, use SET NAMES utf8mb4
So from Perl you should use {mysql_enable_utf8mb4 => 1} instead of {mysql_enable_utf8 => 1}.
I tried so many times, in so many different ways, to cgi script works correctly to read input from STDIN, read html file, print it to STDOUT and search the inputed text on mysql correctly. The attribute mysql_enable_utf8mb4 and "SET NAMES utf8mb4" after the mysql connection works correctly with "meta charset='UTF-8'".
#!/usr/bin/perl
print "Content-type: text/html; charset=UTF-8\n\n";
#use utf8;
#use open ':utf8';
#binmode STDOUT, ":utf8";
#binmode STDIN , ":utf8";
#use encoding 'utf8';
our $dbh = DBI->connect("DBI:mysql:database=$database;host=$servername;port=$port",$username,$password, {PrintWarn => 0, PrintError => 0, mysql_enable_utf8mb4 => 1}) || die;
$dbh->do("SET NAMES utf8mb4");
I try to save names from users from a service in my MySQL database. Those names can contain emojis like 🙈😂😱🍰 (just for examples)
After searching a little bit I found this stackoverflow linking to this tutorial. I followed the steps and it looks like everything is configured properly.
I have a Database (charset and collation set to utf8mb4 (_unicode_ci)), a Table called TestTable, also configured this way, as well as a "Text" column, configured this way (VARCHAR(191) utf8mb4_unicode_ci).
When I try to save emojis I get an error:
Example of error for shortcake (🍰):
Warning: #1300 Invalid utf8 character string: 'F09F8D'
Warning: #1366 Incorrect string value: '\xF0\x9F\x8D\xB0' for column 'Text' at row 1
The only Emoji that I was able to save properly was the sun ☀️
Though I didn't try all of them to be honest.
Is there something I'm missing in the configuration?
Please note: All tests of saving didn't involve a client side. I use phpmyadmin to manually change the values and save the data. So the proper configuration of the client side is something that I will take care of after the server properly saves emojis.
Another Sidenote: Currently, when saving emojis I either get the error like above, or get no error and the data of Username 🍰 will be stored as Username ????. Error or no error depends on the way I save. When creating/saving via SQL Statement I save with question marks, when editing inline I save with question marks, when editing using the edit button I get the error.
thank you
EDIT 1:
Alright so I think I found out the problem, but not the solution.
It looks like the Database specific variables didn't change properly.
When I'm logged in as root on my server and read out the variables (global):
Query used: SHOW VARIABLES WHERE Variable_name LIKE 'character\_set\_%' OR Variable_name LIKE 'collation%';
+--------------------------+--------------------+
| Variable_name | Value |
+--------------------------+--------------------+
| character_set_client | utf8mb4 |
| character_set_connection | utf8mb4 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | utf8mb4 |
| character_set_server | utf8mb4 |
| character_set_system | utf8 |
| collation_connection | utf8mb4_unicode_ci |
| collation_database | utf8mb4_unicode_ci |
| collation_server | utf8mb4_unicode_ci |
+--------------------------+--------------------+
10 rows in set (0.00 sec)
For my Database (in phpmyadmin, the same query) it looks like the following:
+--------------------------+--------------------+
| Variable_name | Value |
+--------------------------+--------------------+
| character_set_client | utf8 |
| character_set_connection | utf8mb4 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| collation_connection | utf8mb4_unicode_ci |
| collation_database | utf8mb4_unicode_ci |
| collation_server | utf8mb4_unicode_ci |
+--------------------------+--------------------+
How can I adjust these settings on the specific database?
Also even though I have the first shown settings as default, when creating a new database I get the second one as settings.
Edit 2:
Here is my my.cnf file:
[client]
port=3306
socket=/var/run/mysqld/mysqld.sock
default-character-set = utf8mb4
[mysql]
default-character-set = utf8mb4
[mysqld_safe]
socket=/var/run/mysqld/mysqld.sock
[mysqld]
user=mysql
pid-file=/var/run/mysqld/mysqld.pid
socket=/var/run/mysqld/mysqld.sock
port=3306
basedir=/usr
datadir=/var/lib/mysql
tmpdir=/tmp
lc-messages-dir=/usr/share/mysql
log_error=/var/log/mysql/error.log
max_connections=200
max_user_connections=30
wait_timeout=30
interactive_timeout=50
long_query_time=5
innodb_file_per_table
character-set-client-handshake = FALSE
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
!includedir /etc/mysql/conf.d/
character_set_client, _connection, and _results must all be utf8mb4 for that shortcake to be eatable.
Something, somewhere, is setting a subset of those individually. Rummage through my.cnf and phpmyadmin's settings -- something is not setting all three.
If SET NAMES utf8mb4 is executed, all three set correctly.
The sun shone because it is only 3-bytes - E2 98 80; utf8 is sufficient for 3-byte utf8 encodings of Unicode characters.
For me, it turned out that the problem lied in mysql client.
mysql client updates my.cnf's char setting on a server, and resulted in unintended character setting.
So, What I needed to do is just to add character-set-client-handshake = FALSE.
It disables client setting from disturbing my char setting.
my.cnf would be like this.
[mysqld]
character-set-client-handshake = FALSE
character-set-server = utf8mb4
...
Hope it helps.
It is likely that your service/application is connecting with "utf8" instead of "utf8mb4" for the client character set. That's up to the client application.
For a PHP application see http://php.net/manual/en/function.mysql-set-charset.php or http://php.net/manual/en/mysqli.set-charset.php
For a Python application see https://github.com/PyMySQL/PyMySQL#example or http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#mysql-unicode
Also, check that your columns really are utf8mb4. One direct way is like this:
mysql> SELECT character_set_name FROM information_schema.`COLUMNS` WHERE table_name = "user" AND column_name = "displayname";
+--------------------+
| character_set_name |
+--------------------+
| utf8mb4 |
+--------------------+
1 row in set (0.00 sec)
Symfony 5 answer
Although this is not what was asked, people can land up here after searching the web for the same problem in Symfony.
1. Configure MySQL properly
☝️ See (and upvote if helpful) top answers here.
2. Change your Doctrine configuration
/config/packages/doctrine.yaml
doctrine:
dbal:
...
charset: utf8mb4
I'm not proud of this answer, because it uses brute-force to clean the input. It's brutal, but it works
function cleanWord($string, $debug = false) {
$new_string = "";
for ($i=0;$i<strlen($string);$i++) {
$letter = substr($string, $i, 1);
if ($debug) {
echo "Letter: " . $letter . "<BR>";
echo "Code: " . ord($letter) . "<BR><BR>";
}
$blnSkip = false;
if (ord($letter)=="146") {
$letter = "´";
$blnSkip = true;
}
if (ord($letter)=="233") {
$letter = "é";
$blnSkip = true;
}
if (ord($letter)=="147" || ord($letter)=="148") {
$letter = """;
$blnSkip = true;
}
if (ord($letter)=="151") {
$letter = "–";
$blnSkip = true;
}
if ($blnSkip) {
$new_string .= $letter;
break;
}
if (ord($letter) > 127) {
$letter = "�" . ord($letter) . ";";
}
$new_string .= $letter;
}
if ($new_string!="") {
$string = $new_string;
}
//optional
$string = str_replace("\r\n", "<BR>", $string);
return $string;
}
//clean up the input
$message = cleanWord($message);
//now you can insert it as part of SQL statement
$sql = "INSERT INTO tbl_message (`message`)
VALUES ('" . addslashes($message) . "')";
ALTER TABLE table_name CHANGE column_name column_name
VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NULL
DEFAULT NULL;
example query :
ALTER TABLE `reactions` CHANGE `emoji` `emoji` VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NULL DEFAULT NULL;
after that , successful able to store emoji in table :
Consider adding
init_connect = 'SET NAMES utf8mb4'
to all of your your db-servers' my.cnf-s.
(still, clients can (so will) overrule it)
I was importing data via command:
LOAD DATA LOCAL INFILE 'abc.csv' INTO TABLE abc
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(col1, col2, col3, col4, col5...);
This didnt work for me:
SET NAMES utf8mb4;
I had to add the CHARACTER SET to make it working:
LOAD DATA LOCAL INFILE
'E:\\wamp\\tmp\\customer.csv' INTO TABLE `customer`
CHARACTER SET 'utf8mb4'
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES;
Note, the target column must be also utf8mb4 not utf8, or the import will save (without errors thought) the question marks like "?????".
For codeigniter user, ensure your character set and collate setting in database.php is set properly, which is worked for me.
$db['default']['char_set'] = 'utf8mb4';
$db['default']['dbcollat'] = 'utf8mb4_unicode_ci';
I'm getting error:
Illegal mix of collations (utf8_general_ci,IMPLICIT) and (utf8_unicode_ci,IMPLICIT) for operation '='"
I tried changing both tables manually to utf8_general_ci,IMPLICIT but I'm still getting the error.
Is there a way to convert all tables to utf8_general_ci,IMPLICIT and be done with it?
You need to execute a alter table statement for each table. The statement would follow this form:
ALTER TABLE tbl_name
[[DEFAULT] CHARACTER SET charset_name]
[COLLATE collation_name]
Now to get all the tables in the database you would need to execute the following query:
SELECT *
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA="YourDataBaseName"
AND TABLE_TYPE="BASE TABLE";
So now let MySQL write the code for you:
SELECT CONCAT("ALTER TABLE ", TABLE_SCHEMA, '.', TABLE_NAME," COLLATE your_collation_name_here;") AS ExecuteTheString
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA="YourDatabaseName"
AND TABLE_TYPE="BASE TABLE";
You can copy the results and execute them. I have not tested the syntax but you should be able to figure out the rest. Think of it as a little exercise.
Better option to change also collation of varchar columns inside table also
SELECT CONCAT('ALTER TABLE `', TABLE_NAME,'` CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;') AS mySQL
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA= "myschema"
AND TABLE_TYPE="BASE TABLE"
Additionnaly if you have data with forein key on non utf8 column before launch the bunch script use
SET foreign_key_checks = 0;
It means global SQL will be for mySQL :
SET foreign_key_checks = 0;
ALTER TABLE `table1` CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;
ALTER TABLE `table2` CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;
ALTER TABLE `tableXXX` CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;
SET foreign_key_checks = 1;
But take care if according mysql documentation http://dev.mysql.com/doc/refman/5.1/en/charset-column.html,
If you use ALTER TABLE to convert a column from one character set to another, MySQL attempts to map the data values, but if the character sets are incompatible, there may be data loss. "
EDIT: Specially with column type enum, it just crash completly enums set (even if there is no special caracters)
https://bugs.mysql.com/bug.php?id=26731
#Namphibian's suggestion helped me a lot...
went a little further though and added columns and views to the script
just enter your schema's name below and it will do the rest
-- set your table name here
SET #MY_SCHEMA = "";
-- tables
SELECT DISTINCT
CONCAT("ALTER TABLE ", TABLE_NAME," CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;") as queries
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA=#MY_SCHEMA
AND TABLE_TYPE="BASE TABLE"
UNION
-- table columns
SELECT DISTINCT
CONCAT("ALTER TABLE ", C.TABLE_NAME, " CHANGE ", C.COLUMN_NAME, " ", C.COLUMN_NAME, " ", C.COLUMN_TYPE, " CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;") as queries
FROM INFORMATION_SCHEMA.COLUMNS as C
LEFT JOIN INFORMATION_SCHEMA.TABLES as T
ON C.TABLE_NAME = T.TABLE_NAME
WHERE C.COLLATION_NAME is not null
AND C.TABLE_SCHEMA=#MY_SCHEMA
AND T.TABLE_TYPE="BASE TABLE"
UNION
-- views
SELECT DISTINCT
CONCAT("CREATE OR REPLACE VIEW ", V.TABLE_NAME, " AS ", V.VIEW_DEFINITION, ";") as queries
FROM INFORMATION_SCHEMA.VIEWS as V
LEFT JOIN INFORMATION_SCHEMA.TABLES as T
ON V.TABLE_NAME = T.TABLE_NAME
WHERE V.TABLE_SCHEMA=#MY_SCHEMA
AND T.TABLE_TYPE="VIEW";
Below is the more accurate query.
I am giving example how to convert it to utf8
SELECT CONCAT("ALTER TABLE `", TABLE_NAME,"` DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;") AS mySQL
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA="myschema"
AND TABLE_TYPE="BASE TABLE"
If you're using PhpMyAdmin, you can now:
Select the database.
Click the "Operations" tab.
Under "Collation" section, select the desired collation.
Click the "Change all tables collations" checkbox.
A new "Change all tables columns collations" checkbox will appear.
Click the "Change all tables columns collations" checkbox.
Click the "Go" button.
I had over 250 tables to convert. It took a little over 5 minutes.
You can use this BASH script:
#!/bin/bash
USER="YOUR_DATABASE_USER"
PASSWORD="YOUR_USER_PASSWORD"
DB_NAME="DATABASE_NAME"
CHARACTER_SET="utf8" # your default character set
COLLATE="utf8_general_ci" # your default collation
tables=`mysql -u $USER -p$PASSWORD -e "SELECT tbl.TABLE_NAME FROM information_schema.TABLES tbl WHERE tbl.TABLE_SCHEMA = '$DB_NAME' AND tbl.TABLE_TYPE='BASE TABLE'"`
for tableName in $tables; do
if [[ "$tableName" != "TABLE_NAME" ]] ; then
mysql -u $USER -p$PASSWORD -e "ALTER TABLE $DB_NAME.$tableName DEFAULT CHARACTER SET $CHARACTER_SET COLLATE $COLLATE;"
echo "$tableName - done"
fi
done
For phpMyAdmin I figured this out:
SELECT GROUP_CONCAT("ALTER TABLE ", TABLE_SCHEMA, '.', TABLE_NAME," CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" SEPARATOR ' ') AS OneSQLString
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA="yourtableschemaname"
AND TABLE_TYPE="BASE TABLE"
Just change yourtableschemaname and you're fine.
Taking the answer from #Petr Stastny a step further by adding a password variable. I'd prefer if it actually took it in like a regular password rather than as an argument, but it's working for what I needed.
#!/bin/bash
# mycollate.sh <database> <password> [<charset> <collation>]
# changes MySQL/MariaDB charset and collation for one database - all tables and
# all columns in all tables
DB="$1"
PW="$2"
CHARSET="$3"
COLL="$4"
[ -n "$DB" ] || exit 1
[ -n "$PW" ]
[ -n "$CHARSET" ] || CHARSET="utf8mb4"
[ -n "$COLL" ] || COLL="utf8mb4_bin"
PW="--password=""$PW"
echo $DB
echo "ALTER DATABASE $DB CHARACTER SET $CHARSET COLLATE $COLL;" | mysql -u root "$PW"
echo "USE $DB; SHOW TABLES;" | mysql -s "$PW" | (
while read TABLE; do
echo $DB.$TABLE
echo "ALTER TABLE $TABLE CONVERT TO CHARACTER SET $CHARSET COLLATE $COLL;" | mysql "$PW" $DB
done
)
PW="pleaseEmptyMeNow"
This is my version of a bash script. It takes database name as a parameter and converts all tables to another charset and collation (given by another parameters or default value defined in the script).
#!/bin/bash
# mycollate.sh <database> [<charset> <collation>]
# changes MySQL/MariaDB charset and collation for one database - all tables and
# all columns in all tables
DB="$1"
CHARSET="$2"
COLL="$3"
[ -n "$DB" ] || exit 1
[ -n "$CHARSET" ] || CHARSET="utf8mb4"
[ -n "$COLL" ] || COLL="utf8mb4_general_ci"
echo $DB
echo "ALTER DATABASE $DB CHARACTER SET $CHARSET COLLATE $COLL;" | mysql
echo "USE $DB; SHOW TABLES;" | mysql -s | (
while read TABLE; do
echo $DB.$TABLE
echo "ALTER TABLE $TABLE CONVERT TO CHARACTER SET $CHARSET COLLATE $COLL;" | mysql $DB
done
)
Following on from G H I've added the user and host parameters incase you need to do this on a remote server
#!/bin/bash
# mycollate.sh <database> <user> <password> [<host> <charset> <collation>]
# changes MySQL/MariaDB charset and collation for one database - all tables and
# all columns in all tables
DB="$1"
USER="$2"
PW="$3"
HOST="$4"
CHARSET="$5"
COLL="$6"
[ -n "$DB" ] || exit 1
[ -n "$USER" ] || exit 1
[ -n "$PW" ] || exit 1
[ -n "$HOST" ] || HOST="localhost"
[ -n "$CHARSET" ] || CHARSET="utf8mb4"
[ -n "$COLL" ] || COLL="utf8mb4_general_ci"
PW="--password=""$PW"
HOST="--host=""$HOST"
USER="--user=""$USER"
echo $DB
echo "ALTER DATABASE $DB CHARACTER SET $CHARSET COLLATE $COLL;" | mysql "$HOST" "$USER" "$PW"
echo "USE $DB; SHOW TABLES;" | mysql "$HOST" "$USER" "$PW" | (
while read TABLE; do
echo $DB.$TABLE
echo "ALTER TABLE $TABLE CONVERT TO CHARACTER SET $CHARSET COLLATE $COLL;" | mysql "$HOST" "$USER" "$PW" $DB
done
)
PW="pleaseEmptyMeNow"
If you want a copy-paste bash script:
var=$(mysql -e 'SELECT CONCAT("ALTER TABLE ", TABLE_NAME," CONVERT TO CHARACTER SET utf8 COLLATE utf8_czech_ci;") AS execTabs FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="zabbix" AND TABLE_TYPE="BASE TABLE"' -uroot -p )
var+='ALTER DATABASE zabbix CHARACTER SET utf8 COLLATE utf8_general_ci;'
echo $var | cut -d " " -f2- | mysql -uroot -p zabbix
Change zabbix to your database name.
I will share my answer using MySQL procedure.
You need to run 3 sql command.
1.
DROP PROCEDURE IF EXISTS UpdateTable;
2.
DELIMITER $$
CREATE PROCEDURE UpdateTable()
BEGIN
DECLARE done INT DEFAULT FALSE;
DECLARE _table_name CHAR(255);
DECLARE cur CURSOR FOR
SELECT table_name FROM information_schema.tables
WHERE table_schema = 'my_db_name' AND table_type = "BASE TABLE";
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
OPEN cur;
My_loop: LOOP
FETCH cur INTO _table_name;
SET #my_table_name = _table_name;
IF done THEN
LEAVE My_loop;
END IF;
SET FOREIGN_KEY_CHECKS = 0;
SET #stmt = CONCAT('ALTER TABLE ', #my_table_name, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;');
PREPARE stmt1 FROM #stmt;
EXECUTE stmt1;
DEALLOCATE PREPARE stmt1;
SET FOREIGN_KEY_CHECKS = 1;
END LOOP;
CLOSE cur;
END$$
DELIMITER ;
3.
CALL UpdateTable();
Then run first one again. If you don't want to store the procedure.
The database is latin1_general_ci now and I want to change collation to utf8mb4_general_ci.
Is there any setting in PhpMyAdmin to change collation of database, table, column? Rather than changing one by one?
I am contributing here, as the OP asked:
How to change collation of database, table, column?
The selected answer just states it on table level.
Changing it database wide:
ALTER DATABASE <database_name> CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
Changing it per table:
ALTER TABLE <table_name> CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
Good practice is to change it at table level as it'll change it for columns as well. Changing for specific column is for any specific case.
Changing collation for a specific column:
ALTER TABLE <table_name> MODIFY <column_name> VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
You need to either convert each table individually:
ALTER TABLE mytable CONVERT TO CHARACTER SET utf8mb4
(this will convert the columns just as well), or export the database with latin1 and import it back with utf8mb4.
You can run a php script.
<?php
$con = mysql_connect('localhost','user','password');
if(!$con) { echo "Cannot connect to the database ";die();}
mysql_select_db('dbname');
$result=mysql_query('show tables');
while($tables = mysql_fetch_array($result)) {
foreach ($tables as $key => $value) {
mysql_query("ALTER TABLE $value CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci");
}}
echo "The collation of your database has been successfully changed!";
?>
To change collation for tables individually you can use,
ALTER TABLE mytable CONVERT TO CHARACTER SET utf8
To set default collation for the whole database,
ALTER DATABASE `databasename` DEFAULT CHARACTER SET utf8 COLLATE utf8_bin
or else,
Goto PhpMyAdmin->Operations->Collation.
There you an find the select box which contains all the exsiting collations. So that here you can change your collation. So here after database table will follows this collation while you are creating new column . No need of select collation while creating new columns.
The following query will generate ALTER queries that change the collation for all appropriate columns in all tables to a certain type (utf8_general_ci in my example below).
SELECT concat
(
'ALTER TABLE ',
t1.TABLE_SCHEMA,
'.',
t1.table_name,
' MODIFY ',
t1.column_name,
' ',
t1.data_type,
'(' ,
CHARACTER_MAXIMUM_LENGTH,
')',
' CHARACTER SET utf8 COLLATE utf8_general_ci;'
)
from
information_schema.columns t1
where
t1.TABLE_SCHEMA like 'you_db_name_goes_here' AND
t1.COLLATION_NAME IS NOT NULL AND
t1.COLLATION_NAME NOT IN ('utf8_general_ci');
Generates query to update each table and column of each table.
I have used this to some of my projects before and was able to solved most of my COLLATION problems. (especially on JOINS)
To use, just export results to delimited text (probably new line '\n')
EACH TABLE
SELECT CONCAT('ALTER TABLE `', TABLE_NAME,
'` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;')
AS 'USE `DATABASE_NAME`;'
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA = 'DATABASE_NAME'
AND TABLE_TYPE LIKE 'BASE TABLE'
EACH COLUMN
SELECT CONCAT('ALTER TABLE `', TABLE_NAME, '` MODIFY COLUMN `', COLUMN_NAME,'` ',
DATA_TYPE, IF(CHARACTER_MAXIMUM_LENGTH IS NULL
OR DATA_TYPE LIKE 'longtext', '', CONCAT('(', CHARACTER_MAXIMUM_LENGTH,
')')
), ' COLLATE utf8mb4_unicode_ci;') AS 'USE `DATABASE_NAME`;'
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = 'DATABASE_NAME'
AND (SELECT INFORMATION_SCHEMA.TABLES.TABLE_TYPE
FROM INFORMATION_SCHEMA.TABLES
WHERE INFORMATION_SCHEMA.TABLES.TABLE_SCHEMA =
INFORMATION_SCHEMA.COLUMNS.TABLE_SCHEMA
AND INFORMATION_SCHEMA.TABLES.TABLE_NAME =
INFORMATION_SCHEMA.COLUMNS.TABLE_NAME
LIMIT 1) LIKE 'BASE TABLE'
AND DATA_TYPE IN ( 'char', 'varchar' ) /* include other types if necessary */
If you run phpMyAdmin >> select database >> select table >> go to "Operations" tab >> in "Table options" section >> you can pick Collation from the drop down list >> and once you press {Go} at the top of the screen you will see a message:
Your SQL query has been executed successfully
and a script
ALTER TABLE `tableName` CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci
But it will NOT change the collations of existing columns.
To do so you can use this script (this one also came from phpMyAdmin)
ALTER TABLE `tableName` CHANGE `Name` `Name` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL
you can set default collation at several levels:
http://dev.mysql.com/doc/refman/5.0/en/charset-syntax.html
1) client
2) server default
3) database default
4) table default
5) column
Just run this SQL to convert all database tables at once. Change your COLLATION and databaseName to what you need.
SELECT CONCAT("ALTER TABLE ", TABLE_SCHEMA, '.', TABLE_NAME," COLLATE utf8_general_ci;") AS ExecuteTheString
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA="databaseName"
AND TABLE_TYPE="BASE TABLE";
Better variant to generate SQL script by SQL request. It will not ruin defaults/nulls.
SELECT concat
(
'ALTER TABLE ',
t1.TABLE_SCHEMA,
'.',
t1.table_name,
' MODIFY ',
t1.column_name,
' ',
t1.column_type,
' CHARACTER SET utf8 COLLATE utf8_general_ci',
if(t1.is_nullable='YES', ' NULL', ' NOT NULL'),
if(t1.column_default is not null, concat(' DEFAULT \'', t1.column_default, '\''), ''),
';'
)
from
information_schema.columns t1
where
t1.TABLE_SCHEMA like 'your_table_here' AND
t1.COLLATION_NAME IS NOT NULL AND
t1.COLLATION_NAME NOT IN ('utf8_general_ci');
You can change the CHARSET and COLLATION of all your tables through PHP script as follows. I like the answer of hkasera but the problem with it is that the query runs twice on each table. This code is almost the same except using MySqli instead of mysql and prevention of double querying. If I could vote up, I would have voted hkasera's answer up.
<?php
$conn1=new MySQLi("localhost","user","password","database");
if($conn1->connect_errno){
echo mysqli_connect_error();
exit;
}
$res=$conn1->query("show tables") or die($conn1->error);
while($tables=$res->fetch_array()){
$conn1->query("ALTER TABLE $tables[0] CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci") or die($conn1->error);
}
echo "The collation of your database has been successfully changed!";
$res->free();
$conn1->close();
?>
You can simple add this code to script file
//Database Connection
$host = 'localhost';
$db_name = 'your_database_name';
$db_user = 'your_database_user_name';
$db_pass = 'your_database_user_password';
$con = mysql_connect($host,$db_user,$db_pass);
if(!$con) { echo "Cannot connect to the database ";die();}
mysql_select_db($db_name);
$result=mysql_query('show tables');
while($tables = mysql_fetch_array($result)) {
foreach ($tables as $key => $value) {
mysql_query("ALTER TABLE $value CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci");
}
}
echo "The collation of your database has been successfully changed!";
I was surprised to learn, and so I had to come back here and report, that the excellent and well maintained Interconnect/it SAFE SEARCH AND REPLACE ON DATABASE script has some options for converting tables to utf8 / unicode, and even to convert to innodb. It's a script commonly used to migrate a database driven website (Wordpress, Drupal, Joomla, etc) from one domain to another.
https://github.com/interconnectit/Search-Replace-DB
https://interconnectit.com/products/search-and-replace-for-wordpress-databases/
I read it here, that you need to convert each table manually, it is not true. Here is a solution how to do it with a stored procedure:
DELIMITER $$
DROP PROCEDURE IF EXISTS changeCollation$$
-- character_set parameter could be 'utf8'
-- or 'latin1' or any other valid character set
CREATE PROCEDURE changeCollation(IN character_set VARCHAR(255))
BEGIN
DECLARE v_finished INTEGER DEFAULT 0;
DECLARE v_table_name varchar(255) DEFAULT "";
DECLARE v_message varchar(4000) DEFAULT "No records";
-- This will create a cursor that selects each table,
-- where the character set is not the one
-- that is defined in the parameter
DECLARE alter_cursor CURSOR FOR SELECT DISTINCT TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = DATABASE()
AND COLLATION_NAME NOT LIKE CONCAT(character_set, '_%');
-- This handler will set the value v_finished to 1
-- if there are no more rows
DECLARE CONTINUE HANDLER
FOR NOT FOUND SET v_finished = 1;
OPEN alter_cursor;
-- Start a loop to fetch each rows from the cursor
get_table: LOOP
-- Fetch the table names one by one
FETCH alter_cursor INTO v_table_name;
-- If there is no more record, then we have to skip
-- the commands inside the loop
IF v_finished = 1 THEN
LEAVE get_table;
END IF;
IF v_table_name != '' THEN
IF v_message = 'No records' THEN
SET v_message = '';
END IF;
-- This technic makes the trick, it prepares a statement
-- that is based on the v_table_name parameter and it means
-- that this one is different by each iteration inside the loop
SET #s = CONCAT('ALTER TABLE ',v_table_name,
' CONVERT TO CHARACTER SET ', character_set);
PREPARE stmt FROM #s;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
SET v_message = CONCAT('The table ', v_table_name ,
' was changed to the default collation of ', character_set,
'.\n', v_message);
SET v_table_name = '';
END IF;
-- Close the loop and the cursor
END LOOP get_table;
CLOSE alter_cursor;
-- Returns information about the altered tables or 'No records'
SELECT v_message;
END $$
DELIMITER ;
After the procedure is created call it simply:
CALL changeCollation('utf8');
For more details read this blog.
My solution is a combination of #Dzintars and #Quassnoi Answer.
SELECT CONCAT("ALTER TABLE ", TABLE_SCHEMA, '.', TABLE_NAME," CONVERT TO CHARACTER SET utf8mb4 ;") AS ExecuteTheString
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_SCHEMA="<your-database>"
AND TABLE_TYPE="BASE TABLE";
By using CONVERT TO, this generates a scripts, which converts all the Tables of <your-database> to your requested encoding. This also changes the encoding of every column!
if you want to update the default charset on a schema:
ALTER SCHEMA MYSCHEMA DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_unicode_ci;
I used the following shell script. It takes database name as a parameter and converts all tables to another charset and collation (given by another parameters or default value defined in the script).
#!/bin/bash
# mycollate.sh <database> [<charset> <collation>]
# changes MySQL/MariaDB charset and collation for one database - all tables and
# all columns in all tables
DB="$1"
CHARSET="$2"
COLL="$3"
[ -n "$DB" ] || exit 1
[ -n "$CHARSET" ] || CHARSET="utf8mb4"
[ -n "$COLL" ] || COLL="utf8mb4_general_ci"
echo $DB
echo "ALTER DATABASE $DB CHARACTER SET $CHARSET COLLATE $COLL;" | mysql
echo "USE $DB; SHOW TABLES;" | mysql -s | (
while read TABLE; do
echo $DB.$TABLE
echo "ALTER TABLE $TABLE CONVERT TO CHARACTER SET $CHARSET COLLATE $COLL;" | mysql $DB
done
)
Quick way - export to SQL file, use search and replace to change the text you need to change. Create new database, import the data and then rename the old database and the new one to the old name.
To change the collation of all fields in all tables of a database at once:
I was just adding another loop for the fields within the tables to the solution via Php before mentioned. This has helped, all fields in the tables are also converted.
<?php
$con = mysql_connect('localhost','user','pw');
if(!$con) { echo "Cannot connect to the database ";die();}
mysql_select_db('database_name');
$result=mysql_query('show tables');
while($tables = mysql_fetch_array($result)) {
foreach ($tables as $key => $table) { // for each table
$sql = "ALTER TABLE $table CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci";
echo "\n".$sql;
mysql_query($sql);
$sql = "show fields in ".$table." where type like 'varchar%' or type like 'char%' or type='text' or type='mediumtext';";
$rs2=mysql_query($sql);
while( $rw2 = mysql_fetch_array($rs2) ){ // for each field in table
$sql = "ALTER TABLE `".$table."` CHANGE `".$rw2['Field']."` `".$rw2['Field']."` ".$rw2['Type']." CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL;";
echo "\n".$sql;
mysql_query($sql);
}
}
}
echo "The collation of your database has been successfully changed!";
?>}
Note, after changing the charset for database/table/column, you might need to actually convert the existing data (if you see, for example, something like "مطلوب توريد جÙ") with something like this:
update country set name = convert(cast(convert(name using latin1) as binary) using utf8), cn_flag = convert(cast(convert(cn_flag using latin1) as binary) using utf8), and so on..
While for converting database, tables and fields, I would suggest this answer from this thread which would generate a big set of queries that you will just copy at paste, here I couldn't find an automatic solution yet.
Also be warned, if you will convert the same field twice you will get unrecoverable question marks: "???". You will also get this question marks if you will convert data before converting fields/tables.
I had to change the collates of all databases, tables and columns in a cluster with many bases.
I used a script running on php 8.1 and mysql 8.0
function changeCollate() {
$databases = $this->fetchQueryToArray("SHOW DATABASES LIKE 'nova_%'")->rows;
foreach ($databases as $value) {
$db = $value['Database (nova_%)'];
$this->LOG("-- banco de dados --- " . $db);
$this->exeQuery("ALTER DATABASE `$db` COLLATE utf8mb4_0900_ai_ci;");
$this->exeQuery("use $db");
$tables = $this->fetchQueryToArray("SHOW tables")->rows;
foreach ($tables as $table) {
$tb_name = $table["Tables_in_$db"];
$this->exeQuery("ALTER TABLE `$tb_name` COLLATE utf8mb4_0900_ai_ci;");
$QUERY = "ALTER TABLE `$db`.`$tb_name`\n";
$columns = $this->fetchQueryToArray("SHOW FULL COLUMNS FROM $tb_name WHERE Type LIKE 'varchar%' OR Type = 'text' OR Type like 'enum%' OR Type = 'longtext' OR Type = 'mediumtext'")->rows;
foreach ($columns as $column) {
$QUERY .= "CHANGE `{$column['Field']}` `{$column['Field']}` {$column['Type']} COLLATE 'utf8mb4_0900_ai_ci'";
$QUERY .= ($column['Null'] == 'YES') ? " NULL" : " NOT NULL";
if ($column['Default']) $QUERY .= " DEFAULT '{$column['Default']}'";
if ($column['Comment']) $QUERY .= " COMMENT '{$column['Comment']}'";
$QUERY .= ",\n";
}
if ($QUERY == "ALTER TABLE `$db`.`$tb_name`\n") continue;
$QUERY = substr($QUERY, 0, -2) . ";\n\n";
$this->exeQuery($QUERY);
}
}
}
I've just written a bash script to find all tables in a given database and covert them (and its columns).
Script is available here: https://github.com/Juddling/mysql-charset