I recently switched my MariaDB database to UTF-8 from Latin1. Read a bunch of checklists and carefully updated my character set, collation, my.cnf and php.ini. I have php forms for most of my data entry on the site, but sometimes for quick little changes, it's easier to go into a program like Adminer or phpmyadmin.
With the UTF-8 in place, I wanted to change director Alfonso Cuaron's name to Cuarón. I went to his entry in Adminer. Edit. Cuar[alt+0243]n. It showed in the edit box as Cuarón. But when I saved the change, Adminer showed it as Cuarón. Okay. Looked at page info in Firefox. Says the character encoding of the page is UTF-8. So all should be well, right?
I went to one of my php data entry forms and created a Bob Cuarón. It showed up fine.
I SSH'd into the server fired up a mysql command line and ran an update sql line with Cuarón. That worked. But trying to change it in Adminer still kept giving me Cuarón. I installed phpmyadmin (which was giving me some issues with my nginx config) but I was able to edit his name and...sigh...it too gave me Cuarón. I installed SQLbuddy and...success...I was able to make the changes, but the program is lacking some of the things I need, like the ability to edit search results.
I'm sure I've nailed everything down:
nginx.conf:
charset UTF-8;
my.cnf:
[client]
default-character-set=utf8
[mysqld]
character-set-server=utf8
collation-server=utf8_general_ci
init-connect='SET NAMES utf8'
/etc/php5/fpm/php.ini
mbstring.language = Neutral
mbstring.internal_encoding = UTF-8
mbstring.encoding_translation = On
mbstring.http_input = auto
mbstring.http_output = UTF-8
mbstring.detect_order = auto
mbstring.substitute_character = none
default_charset = UTF-8
SHOW VARIABLES LIKE "%character_set%";
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
I can't see what I could be missing. Both Adminer and phpmyadmin handle UTF-8 so I don't know why it's not working. It worked right out of the box with SQLBuddy, but as I said it's missing some features.
Any thoughts where I should look?
UPDATE: turns out that an article I had read (forgot to bookmark, sorry) on UTF-8 migrations had me change some of the mbstring setting away from PHP's defaults. Someone at adminer noticed that and all was good. See their response here:
https://sourceforge.net/p/adminer/discussion/960418/thread/33595373/#42df
Related
I'm connecting to a MySQL server (v5.7.21) that is configured to use utf8mb4 encoding and to ignore encoding set by the client. Here are the relevant sections of my.ini config file (I'm running Windows) :
[client]
default-character-set=utf8mb4
[mysql]
default-character-set=utf8mb4
[mysqld]
character-set-client-handshake=FALSE
skip-character-set-client-handshake #I've added this but it has no effect
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
My client is an ETL (Talend) which uses the JDBC driver : mysql-connector-java-5.1.30-bin.jar.
If I set the driver property: characterEncoding=utf8, this overrides the character encoding set by the server (utf8mb4), as shown by this query :
SHOW VARIABLES
WHERE Variable_name LIKE 'character\_set\_%' OR Variable_name LIKE 'collation%'
|=-----------------------+-----------------=|
|Variable_Name |Value |
|=-----------------------+-----------------=|
|character_set_client |utf8 |
|character_set_connection|utf8 |
|character_set_database |utf8mb4 |
|character_set_filesystem|binary |
|character_set_results | |
|character_set_server |utf8mb4 |
|character_set_system |utf8 |
|collation_connection |utf8_general_ci |
|collation_database |utf8mb4_unicode_ci|
|collation_server |utf8mb4_unicode_ci|
'------------------------+------------------'
If I don't set any driver properties on the connection, the utf8mb4 encoding is used (which is what is expected).
It seems setting character-set-client-handshake=FALSE and skip-character-set-client-handshake has no effect.
How to prevent the client from changing the encoding and force it to always use utf8mb4 ?
Hiho,
I'm trying to get some Asian UTF-8 characters, which are read from a mysql database, to display properly in the command line prompt. I'll go through the steps I've gone through in case someone else is after the same thing...
1: First I made sure the database was encoded, and set the globals utf8
mysql> show variables like 'char%';
+--------------------------+---------------------------------------------------------+
| Variable_name | Value |
+--------------------------+---------------------------------------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
After that I could compare entries in my tables, but they still weren't displayed properly.
2: In the properties of cmd, I noticed it was using code page 850, so I decided to see how change that and changed that before running mysql
cmd> chcp 65001
cmd> "C:\Program Files\MySQL\MySQL Server 5.7\bin\mysql.exe" "--defaults-file=C:\ProgramData\MySQL\MySQL Server 5.7\my.ini" "-uroot" "-p" "--default-character-set=utf8"
3: After checking the font used by the command line, neither font available could display what I needed. Unifont is a fixed size font that seems to be able to though
And that's where I am now. After installing it; I just can't seem to be able to add the Unifont to the list of fonts in command prompt. I found this tutorial on registry editing to add it but no luck. The other available fonts were there though. Even replacing the current one just caused it to use the default font instead. It seems strange Any ideas?
The font in cmd needs to be Lucida Console.
More discussion.
I'm in the process of upgrading an old legacy Rails 2.3 app to something more modern and running into an encoding issue. I've read all the existing answers I can find on this issue but I'm still running into problems.
Rails ver: 2.3.17
Ruby ver: 1.9.3p385
My MySQL tables are default charset: utf8, collation: utf8_general_ci. Prior to 1.9 I was using the original mysql gem without incident. After upgrading to 1.9 when it retrieved anything with utf8 characters in it would get this well-documented problem:
ActionView::TemplateError (incompatible character encodings: ASCII-8BIT and UTF-8)
I switched to the mysql2 gem for it's superior handling and I no longer see exceptions but things are definitely not encoding correctly. For example, what appears in the DB as the string Repoussé is being rendered by Rails as Repoussé, “Boat” appears as “Boatâ€, etc.
A few more details:
I see the same results when I use the ruby-mysql gem as the driver.
I've added encoding: utf8 lines to each entry in my database.yml
I've also added the following to my environment.rb:
Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8
It has occurred to me that I may have some mismatch where latin1 was being written by the old version of the app into the utf8 fields of the database or something, but all of the characters appear correctly when viewed in the mysql command line client.
Thanks in advance for any advice, much appreciated!
UPDATE: I now believe that the issue is that my utf8 data is being coerced through a binary conversion into latin1 on the way out of the db, I'm just not sure where.
mysql> SELECT CONVERT(CONVERT(name USING BINARY) USING latin1) AS latin1, CONVERT(CONVERT(name USING BINARY) USING utf8) AS utf8 FROM items WHERE id=myid;
+-------------+----------+
| latin1 | utf8 |
+-------------+----------+
| Repoussé | Repoussé |
+-------------+----------+
I have my encoding set to utf8 in database.yml, any other ideas where this could be coming from?
I finally figured out what my issue was. While my databases were encoded with utf8, the app with the original mysql gem was injecting latin1 text into the utf8 tables.
What threw me off was that the output from the mysql comand line client looked correct. It is important to verify that your terminal, the database fields and the MySQL client are all running in utf8.
MySQL's client runs in latin1 by default. You can discover what it is running in by issuing this query:
show variables like 'char%';
If setup properly for utf8 you should see:
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
If these don't look correct, make sure the following is set in the [client] section of your my.cnf config file:
default-character-set = utf8
Add add the following to the [mysqld] section:
# use utf8 by default
character-set-server=utf8
collation-server=utf8_general_ci
Make sure to restart the mysql daemon before relaunching the client and then verify.
NOTE: This doesn't change the charset or collation of existing databases, just ensures that any new databases created will default into utf8 and that the client will display in utf8.
After I did this I saw characters in the mysql client that matched what I was getting from the mysql2 gem. I was also able to verify that this content was latin1 by switching to "encoding: latin1" temporarily in my database.conf.
One extremely handy query to find issues is using char length to find the rows with multi-byte characters:
SELECT id, name FROM items WHERE LENGTH(name) != CHAR_LENGTH(name);
There are a lot of scripts out there to convert latin1 contents to utf8, but what worked best for me was dumping all of the databases as latin1 and stuffing the contents back in as utf8:
mysqldump -u root -p --opt --default-character-set=latin1 --skip-set-charset DBNAME > DBNAME.sql
mysql -u root -p --default-character-set=utf8 DBNAME < DBNAME.sql
I backed up my primary db first, then dumped into a test database and verified like crazy before rolling over to the corrected DB.
My understanding is that MySQL's translation can leave some things to be desired with certain more complex characters but since most of my multibyte chars are fairly common things (accent marks, quotes, etc), this worked great for me.
Some resources that proved invaluable in sorting all of this out:
Derek Sivers guide on transforming MySQL data latin1 in utf8 -> utf8
Blue Box article on MySQL character set hell
Simple table conversion instructions on Stack Overlow
You say it all looks OK in the command line client, but perhaps your Terminal's character encoding isn't set to show UTF8? To check in OS X Terminal, click Terminal > Preferences > Settings > Advanced > Character Encoding. Also, check using a graphical tool like MySQL Query Browser at http://dev.mysql.com/downloads/gui-tools/5.0.html.
I have a mysql dump generated from phpmyadmin in a windows environment,
when i try to import in osx (using mysql command line) there are encoding
problems, the databases have the same encoding and collation.
I've also noticed that this problems occurs also when i try to import a
diferent database from a unix virtual machine.
When i try to import the same databases in the Windows with the same commands everything is ok.
Anyone have a ideia about whats going on?
Thanks.
Both databases have the following configuration:
character_set_client | latin1
character_set_connection | latin1
character_set_database | utf8
character_set_filesystem | binary
character_set_results | latin1
character_set_server | utf8
character_set_system | utf8
character_sets_dir | /usr/local/mysql-5.1.43-osx10.6-x86_64/share/charsets/
I have a staging Rails site up that's running on MySQL 5.0.32-Debian.
On this particular site, all of my tables are using utf8 / utf8_general_ci encoding.
Inside that database, I have some data that looks like so:
mysql> select * from currency_types limit 1,10;
+------+-----------------+---------+
| code | name | symbol |
+------+-----------------+---------+
| CAD | Canadian Dollar | $ |
| CNY | Chinese Yuan | å…ƒ |
| EUR | Euro | € |
| GBP | Pound | £ |
| INR | Indian Rupees | ₨ |
| JPY | Yen | ¥ |
| MXN | Mexican Peso | $ |
| USD | US Dollar | $ |
| PHP | Philippine Peso | ₱ |
| DKK | Denmark Kroner | kr |
+------+-----------------+---------+
Here's the issue I'm having
On staging (with the db and Rails site running on the debian box), the characters for symbols are appearing correctly when displayed from Rails. For instance, the Chinese Yuan is appearing as 元 in my browser, not å…ƒ as it shows inside the database.
When I download that data to my local OS X development machine and run the db and Rails locally, I see the representation from inside the DB (å…ƒ) on my browser, not the character 元 as I see in staging.
Debugging I've done
I've ensured all headers for Content-Type are coming back as utf8 from each webserver (local, staging).
My local mysql server and the staging server are both setup to use utf8 as the default charset. I'm using "set names 'utf8'" before I make any calls.
I can even connect to my staging db from my OS X Rails host, and I still see the characters å…ƒ representing the yuan. I'm guessing then, perhaps there's an issue with my mysql local client, but I can't figure out what the issue is.
Perhaps this might lend a clue
To make it even more confusing, if I paste the character 元 into the db on my local machine, I see that in the web browser fine. --- YET if I paste that same character into my staging db, I get a ? mark in it's place on the page from my staging Rails site.
Also, locally on my OS X rails machine if I use "set names 'latin1'" before my queries, the characters all come back properly. I did have these tables set as latin1 before - could this be the issue?
Someone please help me out here, I'm going crazy trying to figure out what's wrong!
AHA! Seems I had some table information encoded in latin1 before, and stupidly changed the databases to utf8 without converting.
Running the following fixed that currency_types table:
mysqldump -u root -p --opt --default-character-set=latin1 --skip-set-charset DBNAME > DBNAME.sql
mysql -u root -p --default-character-set=utf8 DBNAME < DBNAME.sql
Now I just have to ensure that the other content generated after the latin1 > utf8 switch isn't messed up by that :(
Do you have these two lines in your database.yml under the proper section?
encoding: utf8
collation: utf8_general_ci
The problem could have been with you MySQL client in staging it does not support UTF-8.
Your local OSX ruby installation configuration might not have declared the proper configs.
You should have "encoding: utf8" in "config/database.yml" for the MySQL database.
You should have "$KCODE = 'u'" in "config/environment.rb" for the ruby enviroment.
Another simple approach is to set the encode type by using SQL Alter statement. You can do this using the below bash script.
for t in $(mysql --user=root --password=admin --database=DBNAME -e "show tables";);do echo "Altering" $t;mysql --user=root --password=admin --database=DBNAME -e "ALTER TABLE $t CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;";done
prettified
for t in $(mysql --user=root --password=admin --database=DBNAME -e "show tables";);
do
echo "Altering" $t;
mysql --user=root --password=admin --database=DBNAME -e "ALTER TABLE $t CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;";
done
My DB was already set by default to utf8, but I encountered the same problem.
Also after adding the following usual meta tag, the problem was still there:
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
Then I created a dedicated connection.php to ensure all communication with MySQL is set to charset utf8. Note that there is no - in utf8 in mysqli_set_charset($bd, 'utf8')!
Here is my Connection.php:
<?php
$mysql_hostname = "localhost";
$mysql_user = "username";
$mysql_password = "password";
$mysql_database = "dbname";
$prefix = "";
$bd = mysqli_connect($mysql_hostname, $mysql_user, $mysql_password) or die("Could not connect database");
mysqli_select_db($bd, $mysql_database) or die("Could not select database");
if(!mysqli_set_charset($bd, 'utf8')) {
exit() ;
}
?>
Another php file:
<?php
//Include database connection details
require_once('connection.php');
//Enter code here...
//Create query
$qry = "SELECT * FROM subject";
$result = mysqli_query($bd, $qry);
?>
//Other stuff
For Rails run the following code snippet into rails console. It will generate an sql for all tables. Then log in to mysql and execute copied sql from rails console. It will alter all tables encoding.
schema = File.open('db/schema.rb', 'r').read
rows = schema.split("\n")
table_name = nil
rows.each do |row|
if row =~ /create_table/
table_name = row.match(/create_table "(.+)"/)[1]
puts "ALTER TABLE `#{table_name}` CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;"
end
end
You can generate a migration, the Rails way, to change the collation type on your databases:
rails generate migration ChangeDatabaseCollation
Then you can edit the generated file and paste:
def change
# for each table that will store the new collation execute:
execute "ALTER TABLE my_table CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci"
end
And run the migration:
rake db:migrate
You can also enforce the new collation on your database.yml:
development:
adapter: mysql2
encoding: utf8
collation: utf8_general_ci
For more information on Rails migrations:
http://edgeguides.rubyonrails.org/active_record_migrations.html
For more information on collation types:
http://collation-charts.org/