Set CENTOS 6 mysql character sets to UTF8 - mysql

I have a website, and I realised when you copy some characters(*,' " - _) from specific applications like Microsoft word, into a search box on my website, it returns this error:
ERROR org.hibernate.util.JDBCExceptionReporter - Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation 'like'
So I went to check out my database and I wanted to see if the database used UTF-8.
mysql> SHOW VARIABLES LIKE '%character%';
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)
mysql> SHOW VARIABLES LIKE '%collation%';
+----------------------+-------------------+
| Variable_name | Value |
+----------------------+-------------------+
| collation_connection | utf8_general_ci |
| collation_database | latin1_swedish_ci |
| collation_server | latin1_swedish_ci |
+----------------------+-------------------+
As you can see, the database is using latin1 and I wanted to set it to use utf8. So firstly, I'm on a Centos 6.2 server and the file my.cnf file resides in /etc/my.cnf and the file is as follows under the [mysqld]:
[mysqld]
local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
init_connect ='SET NAMES utf8'
character_set_server = utf8
collation_server = utf8_general_ci
P.S I am not worried about the [client] section since it shows that it uses utf8 under the value of character_set_client.
The issue:
Although I've tried to set the server in my my.cnf file (and closed the file, shutdown my tomcat and restarted my tomcat). Nothing is changing. And when I run the first query I displayed, it still shows that character_set_server is still using latin1

Although restarting my TOMCAT didn't have any effect, I actually had to restart my 'mySql' in order for the changes to take place. So you'd have to stop the main server first (in my case Stop TOMCAT) and then restart mySql and then Start TOMCAT again. After, in your mySql if you type: show VARIABLES like %character%; you should see your database datatype with utf-8 and if you type: show VARIABLES like %collation% you should also see your database collation data type which should be utf-8_general_ci.

Related

Change default character set on MySQL Workbench

I am trying to connect to my MYSQL database using a utf8mb4 charset (Note the global settings for the database charset is already utf8mb4).
I can do this quite easily using the CLI like so:
mysql -h myhostname -u myuser -p --default-character-set=utf8mb4
When I do the following query:
SHOW VARIABLES WHERE Variable_name LIKE 'character\_set\_%' OR Variable_name LIKE 'collation%';
I get the correct output as expected:
+--------------------------+--------------------+
| Variable_name | Value |
+--------------------------+--------------------+
| character_set_client | utf8mb4 |
| character_set_connection | utf8mb4 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | utf8mb4 |
| character_set_server | utf8mb4 |
| character_set_system | utf8 |
| collation_connection | utf8mb4_general_ci |
| collation_database | utf8mb4_unicode_ci |
| collation_server | utf8mb4_unicode_ci |
+--------------------------+--------------------+
However, when I connect to my MySQL database using MySQL Workbench, and perform the same query I get the following:
+--------------------------+--------------------+
| Variable_name | Value |
+--------------------------+--------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8mb4 |
| character_set_system | utf8 |
| collation_connection | utf8_general_ci |
| collation_database | latin1_swedish_ci |
| collation_server | utf8mb4_unicode_ci |
+--------------------------+--------------------+
The issue here is that I am struggling to change the default-character-set in MySQL Workbench GUI. I tried appending the following:
default-character-set=utf8mb4
in Manage Server Connections > Connection > Advanced > Others section,
but does not seem to have any affect.
How can I change the default character set on the MySQL Workbench GUI.
AFAIK you have to execute this command each time you start a new Workbench session:
SET NAMES 'utf8mb4' COLLATE 'utf8mb4_unicode_ci';
UPDATE
The following is useful if you need to use Workbench to do exports: (I haven't found a similar way to cause all it's connections to default to utf8mb4)
The default charset that is used is to export data is utf8. To support full Unicode though we need utf8mb4. To achieve this it's possible to modify Workbench to use utf8mb4 manually.
Go to C:\Program Files\MySQL\MySQL Workbench 6.3 CE\modules
open this file wb_admin_export.py.
Create a backup copy
Replace all occurrences "default-character-set":"utf8" with "default-character-set":"utf8mb4".
Save the file.
Restart Workbench.
The next time you run the export you will see in the log results like this:
Running: mysqldump.exe --defaults-file="c:\users\jonathan\appdata\local\temp\tmpidlh7a.cnf" --host=localhost --protocol=tcp --user=root --allow-keywords=TRUE --port=3306 --default-character-set=utf8mb4 --routines --skip-triggers "databasename"
In MySQL Workbench (8.0), you can click the Administration tab, select Options File under Instance, scroll to the International section and you'll find character-set-server and collation-server, which you can set to your desired charset and collation. Click the Apply button to save the changes.
This will set the values in /etc/mysql/my.cnf, or wherever your config file is.

Force MariaDB clients to use utf8mb4

I'm running into an issue where I'm getting differently ordered results when querying with PHP Versus the command line. From my research, it appears that in some cases that bad encoding can cause problems with the order of the results.
That said, all my DB tables are encoded as utf8mb4, with the collation utf8mb4_general_ci. However, it doesnt seem that the mysql variables are set correctly.
I'm on Mysql 5.5.5-10.1.26-MariaDb.
Here are my CNF settings, but to be honest I don't know what I'm doing here:
[client]
default-character-set=utf8mb4
[mysql]
default-character-set=utf8mb4
[mariadb]
[mysqld]
character-set-server=utf8mb4
character_set_client=utf8mb4
collation-server=utf8mb4_general_ci
The variables output from mysql:
character_set_client utf8
character_set_connection utf8
character_set_database utf8mb4
character_set_filesystem binary
character_set_results utf8
character_set_server utf8mb4
character_set_system utf8
collation_connection utf8_general_ci
collation_database utf8mb4_unicode_ci
collation_server utf8mb4_general_ci
Update: A person has asked for how I'm connecting to the database:
$this->connection = new PDO('mysql:host='.DB_SERVER.';dbname='.DB_NAME.';port='.DB_PORT, DB_USER, DB_PASS, $options);
Update: I've switched to utf8mb4_unicode_ci (as per suggestions in answers below).
You want to have character-set-client-handshake = FALSE as well.
With /etc/my.cnf.d/character-set.cnf
# https://scottlinux.com/2017/03/04/mysql-mariadb-set-character-set-and-collation-to-utf8/
# https://mariadb.com/kb/en/library/setting-character-sets-and-collations/
# https://medium.com/#adamhooper/in-mysql-never-use-utf8-use-utf8mb4-11761243e434
# https://stackoverflow.com/questions/47566730/force-mariadb-clients-to-use-utf8mb4
[client]
default-character-set = utf8mb4
[mysql]
default-character-set = utf8mb4
[mysqld]
character-set-client-handshake = FALSE
collation-server = utf8mb4_unicode_ci
init-connect = 'SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci'
character-set-server = utf8mb4
I get everything to be utf8mb41
MariaDB [(none)]> show variables like 'char%'; show variables like 'collation%';
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8mb4 |
| character_set_connection | utf8mb4 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | utf8mb4 |
| character_set_server | utf8mb4 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)
+----------------------+--------------------+
| Variable_name | Value |
+----------------------+--------------------+
| collation_connection | utf8mb4_unicode_ci |
| collation_database | utf8mb4_unicode_ci |
| collation_server | utf8mb4_unicode_ci |
+----------------------+--------------------+
3 rows in set (0.00 sec)
MariaDB [(none)]>
however without the character-set-client-handshake line some are still utf8
MariaDB [(none)]> show variables like 'char%'; show variables like 'collation%';
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8mb4 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)
+----------------------+--------------------+
| Variable_name | Value |
+----------------------+--------------------+
| collation_connection | utf8_general_ci |
| collation_database | utf8mb4_unicode_ci |
| collation_server | utf8mb4_unicode_ci |
+----------------------+--------------------+
3 rows in set (0.01 sec)
MariaDB [(none)]>
1 character_set_system is always utf8.
You should probably use utf8mb4_unicode_ci instead of utf8mb4_general_ci as it's more accurate. Unless you're running MariaDB on a system with an old/limited CPU and performance is a huge concern.
That being said, the solution is to set init_connect in your MariaDB configuration (or --init-connect on the command line):
init_connect = "SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci"
Either way is fine. I am not recommending one way over the other. Both are equally valid approaches.
Your MariaDB configuration may be in my.cnf or a file included by my.cnf, typically found under /etc/mysql. Check your system documentation for details. Because you are configuring a server variable, as indicated by the MariaDB documentation linked to above, you should set the variable in the server part of the configuration file. The server part of the configuration files is indicated by the INI section names ending in "d". An INI section is denoted by a keyword surrounded by square brackets, e.g. "[section]". The "d" stands for "daemon", which is standard UNIX nomenclature for a server process. You can set the variable in either the [mysqld] section or the [mariadb] section. Because the init_connect server variable is common to both MySQL and MariaDB, I would recommend you put it under [mysqld].
I see that you are setting character_set_client=utf8mb4 in your pasted configuration. You don't need to do this. You can delete or comment out the line. Comments are lines starting with pound symbol (#), also known as a hash mark, octothorp, or number sign.
Any and all clients that connect to the server will execute these command(s) before any other commands are processed.
init_connect is not performed by anyone connecting as root, so it is not as universal as you would like.
SET NAMES utf8mb4 sets 3 things; experiment to see that. You need all 3.
If you weren't as far back as 5.5, I would recommend utf8mb4_unicode_520_ci as being a better collation: "Unicode collation names now may include a version number to indicate the Unicode Collation Algorithm (UCA) version on which the collation is based. Initial collations thus created use version UCA 5.2.0. For example, utf8_unicode_520_ci is based on UCA 5.2.0. UCA-based Unicode collation names that do not include a version number are based on version 4.0.0."
Version 8.0 has Unicode 9.0 standard.
Back to the question: There is no perfect solution; the user can override whatever you do -- either through ignorance or through malice.
You could police the tables created, but that won't keep them from connecting incorrectly. Or correctly, but with a different charset. It is valid to do SET NAMES latin1, then provide latin1-encode bytes. MySQL will convert as it stores/fetches.
But if they have utf8-encoded bytes, but say SET NAMES latin1, you get "double encoding". This "bug" destroys any chance of collating correctly, but is otherwise (usually) transparent. That is, stuff is messed up as it is stored, then un-messed up as it is fetched.
To fix this warning you should edit
/etc/my.cnf (my.ini on Windows)
Simply add/set in the file
[client]
default-character-set=utf8mb4
[mysql]
default-character-set=utf8mb4
[mysqld]
collation-server=utf8mb4_unicode_ci
init-connect='SET NAMES utf8mb4'
character-set-server=utf8mb4

Saving emoji characters in mysql database

I am trying to save some data on mysql database, input contains emoji characters like this : '\U0001f60a\U0001f48d' and I'm getting this error:
1366, "Incorrect string value: '\\xF0\\x9F\\x98\\x8A\\xF0\\x9F...' for column 'caption' at row 1"
I searched over net and read a lot of answers include these:
MySQL utf8mb4, Errors when saving Emojis or MySQL utf8mb4, Errors when saving Emojis or https://mathiasbynens.be/notes/mysql-utf8mb4#character-sets or http://www.java2s.com/Tutorial/MySQL/0080__Table/charactersetsystem.htm but nothing worked !!
I have different problems:
here is mydb info:
mysql> SHOW VARIABLES WHERE Variable_name LIKE 'character\_set\_%' OR Variable_name LIKE 'collation%';
+--------------------------+--------------------+
| Variable_name | Value |
+--------------------------+--------------------+
| character_set_client | utf8mb4 |
| character_set_connection | utf8mb4 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | utf8mb4 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| collation_connection | utf8mb4_general_ci |
| collation_database | utf8mb4_general_ci |
| collation_server | utf8_general_ci |
+--------------------------+--------------------+
10 rows in set (0.00 sec)
I tried to change character_set_server value to utf8mb4 by
mysql>SET character_set_server = utf8mb4
Query OK, 0 rows affected (0.00 sec)
But when restart mysqld everything revert !
I don't have any /etc/my.cnf file in also, and I edited /etc/mysql/my.cnf file instead.
What should I do?
How can I save emoji file in my database?
1st or 2nd line in source code (to have literals in the code utf8-encoded: # -- coding: utf-8 --
Your columns/tables need to be CHARACTER SET utf8mb4
The python package "MySQL-python" version needs to be at least 1.2.5 in order to handle utf8mb4.
self.query('SET NAMES utf8mb4') may be necessary.
Django needs client_encoding: 'UTF8' -- I don't know if that should be 'utf8mb4`.
References:
https://code.djangoproject.com/ticket/18392
http://mysql.rjweb.org/doc.php/charcoll#python

How do I convert the OpenShift MySQL 5.1 cartridge to UTF-8

The default MySQL 5.1 cartridge apparently creates all its tables with the latin1 character set. I have an application (Review Board, a python/Django application) that has some issues unless the DB is running as UTF-8. How do I change that? I can't just edit my.cnf because it will be wiped at the next cartridge restart.
mysql> show variables like 'character_set%';
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | latin1 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)
I cannot change this setting in my.cnf, because to the best of my knowledge, there exists no OpenShift environment variable to set the character encoding. How do I persistently change this (ideally in my OpenShift hooks so this will persist into future deployments) and update my existing tables to UTF-8?
I found a solution but not a perfect one :
In openshift installing phpMyAdmin,
Find and change server settings, the relevant character variables changed from latin1 to utf8.
Hope that helps

mysql default charset different when invoked by php

As many others, I'm having some problems with mysql charset. As many others, I want everything to be UTF-8, but mysql was installed with latin-1, and no matter how I try/google/experiment with mysql config there is still latin-1 lurking in client settings.
Ok, here is the setup. I have a (non-root) mysql user 'usr' with a password 'pwd'. Whenever I access mysql via terminal (mysql -uusr -p) and then ask him nicely about his charsets, he tell that he is in love with utf8 (as he ought to be):
mysql> SHOW VARIABLES LIKE 'character%';
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.00 sec)
mysql> SHOW VARIABLES LIKE 'collation%';
+----------------------+-----------------+
| Variable_name | Value |
+----------------------+-----------------+
| collation_connection | utf8_general_ci |
| collation_database | utf8_unicode_ci |
| collation_server | utf8_unicode_ci |
+----------------------+-----------------+
3 rows in set (0.00 sec)
However, if I use PHP to access mysql (via the very same user):
$mysql_link=mysql_connect('localhost','usr','pwd');
$result1=mysql_query("show variables like 'character%'");
$result2=mysql_query("show variables like 'collation%'");
mysql_close($mysql_link)
And print_r $result1, $result 2, it magically falls back to latin-1:
character_set_client => latin1
character_set_connection => utf8
character_set_database => utf8
character_set_filesystem => binary
character_set_results => latin1
character_set_server => utf8
character_set_system => utf8
character_sets_dir => /usr/share/mysql/charsets/
collation_connection => utf8_unicode_ci
collation_database => utf8_general_ci
collation_server => utf8_unicode_ci
This happens regardless whether I invoke php via browser (as php-cgi) or via terminal (as php-cli).
Kinda fix for that is to set charset manually at each connection:
mysql_set_charset('utf8',$mysql_link);
That works. But I feel like there should be a way to do that via mysql config.
For reference, Mysql config (my.cfg) includes:
[client]
default_character_set = utf8
[mysqld]
init_connect='SET collation_connection = utf8_unicode_ci'
character-set-server = utf8
collation-server = utf8_unicode_ci
And PHP config (php.ini) includes
default_charset = "utf-8"
Thank forward! =)
P.S. I know that mysql_ functions are deprecated and should be replaced with mysqli_ ones. But hopefully that doesn't have anything to do with this exact problem =)
If you're like most people, you use the root account to get to MySQL. This little snippet from the docs might be your smoking gun.
It is still necessary for applications to configure their connection using SET NAMES or equivalent after they connect, as described previously. You might be tempted to start the server with the --init_connect="SET NAMES 'utf8'" option to cause SET NAMES to be executed automatically for each client that connects. However, this will yield inconsistent results because the init_connect value is not executed for users who have the SUPER privilege.