Issue when deploying mysql db (utf8mb4_unicode_520_ci -> utf8mb4_unicode_ci) - mysql

I started working on a wordpress on my dev machine. mysql version is 5.6, and worpdress is 4.7 so its already using the utf8mb4_unicode_520_ci encoding if it detects its possible.
My problem is that on my hosting (mysql 5.5) utf8mb4_unicode_520_ci is not recognized as a valid encoding. So I'm trying to target utf8mb4_unicode_ci encoding as my hosting knows about this one, and if I understand correctly, this would - in opposition to going to utf8 - allow me to keep the 4 bytes.
I tried several different combinaison of encoding and collation set up for the db, but nothing successful (from here How to convert an entire MySQL database characterset and collation to UTF-8?).
I tried several combination of encoding and collation in the wp-config, but nothing.
Everything that is coming from the database (like post titles and post contents displays badly encoded char for all diatrics, anything else is displayed appropriately )
menu label from the database display incorrectly, where the hardcoded/translated label display correctly
I think I need to convert the actual content of the database, changing charset and collation does not seems to be enough.
I found this but it does not address my problem directly, or if it does I missed it.
Any help would be appreciated
————————————————————————————————
UPDATE :
here is the precise procedure I went through:
Initial situation:
I installed a wordpress (4.6.1) locally (on my dev machine, mysql 5.6.28).
I worked on the theme and plugin locally
(at this moment I have, locally, a database that is utf8_general_ci and tables that are utf8mb4_unicode_520_ci
Problem:
I want to deploy my wordpress on my hosting (mysql: 5.5 - db collation seems to be utf8mb4_unicode_ci).
I mysqldump the db locally, then try to import it on my hostings' phpmyadmin.
This gives error :
Unknown collation: 'utf8mb4_unicode_520_ci'
solution 1 change the tables charset to utf8mb4_unicode_ci:
On my hosting sql server, utf8mb4_unicode_520_ci is not available and I can't get a more recent version of mysql.
utf8mb4_unicode_ci seems like the closest and is available on my hosting sql server.
from various so question, I adapt a bash script to change charset and collation of my tables
for tbl in wp_sij2017_commentmeta wp_sij2017_comments wp_sij2017_cwa wp_sij2017_links wp_sij2017_options wp_sij2017_postmeta wp_sij2017_posts wp_sij2017_term_relationships wp_sij2017_term_taxonomy wp_sij2017_termmeta wp_sij2017_terms wp_sij2017_usermeta wp_sij2017_users wp_sij2017_woocommerce_api_keys wp_sij2017_woocommerce_attribute_taxonomies wp_sij2017_woocommerce_downloadable_product_permissions wp_sij2017_woocommerce_order_itemmeta wp_sij2017_woocommerce_order_items wp_sij2017_woocommerce_payment_tokenmeta wp_sij2017_woocommerce_payment_tokens wp_sij2017_woocommerce_sessions wp_sij2017_woocommerce_shipping_zone_locations wp_sij2017_woocommerce_shipping_zone_methods wp_sij2017_woocommerce_shipping_zones wp_sij2017_woocommerce_tax_rate_locations wp_sij2017_woocommerce_tax_rates; do
mysql --execute="ALTER TABLE wp_sij_2017_original_copy.${tbl} CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"
done
I run this script on the local db
I now have all my tables set to collation utf8mb4_unicode_ci
My db collation is still utf8
I mysqldump the db, then import it to my hosting and...
Import is successful.
I search and replace siteurl in the db.
I then visit the online website, I got SOME diatrics that renders a "question mark char"
Any text coming from the db has decoding issue AT SOME POINT
The source/html markup also has those "question mark char"
I have no idea where to look or what to do next

Clarification: CHARACTER SETs utf8 and utf8mb4 specify how characters are encoded into bytes. COLLATIONs *_unicode_*, etc, specify how those character compare.
The encoding for utf8mb4_unicode_ci and utf8mb4_unicode_520_ci are the same because they are encoded in the character set utf8mb4.
"database that is utf8_general_ci and tables that are utf8mb4_unicode_520_ci" -- that probably means that new tables in that database, unless specifically stated, will be CHARACTER SET utf8 COLLATION utf8_general_ci. That is the database setting is just a default for CREATE TABLE. Since your tables are already CHARACTER SET utf8mb4 COLLATION utf8mb4_unicode_520_ci, the database default is not relevant to them.
As long as the CHARACTER SET stays utf8mb4, no Emoji, Chinese, etc will be lost or otherwise mangled.
Do not use mysql40; it did not know about any CHARACTER SETs. Do not use CONVERT or CAST. Etc.
I assume the 520 is coming from the output of mysqldump? Do you have an editor that can handle a file that big? If so, simply edit it to change utf8mb4_unicode_520_ci to utf8mb4_unicode_ci throughout. Then load the dump. Problem solved?
Your fix
You did ALTER ... CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci on your local machine. That is probably an even better way -- since it will put your dev and prod machine in line with each other. That should have worked. Don't worry about what the "database" claims.

I'm find 'utf8mb4_unicode_520_ci' and replace with 'utf8mb4_unicode_ci' in .sql file.
Its simplest why to solve this.

Related

MySQL character set with issue

I have an issue with the character set for my MySQL database. I am trying to support all languages so I have set the collation to utf8_general_ci and the character set variables to utf8.
The columns of the tables (and the tables themselves) are also set to utf8...
This seems to work fine on my local database...
...but not on the live database...
The live database is hosted on AWS RDS and I have set the character set and collation parameters to utf8/utf8_general_ci as above and rebooted.
This is both an issue in Workbench as well as when the data is queried from code. The value has not been saved properly to the database like it has been locally.
Is there something I'm missing?
Something is set up differently. See "best practice" and "question mark" in Trouble with UTF-8 characters; what I see is not what I stored
If you need "all" languages, you should use CHARACTER SET utf8mb4 so that you can handle the 4-byte characters in Chinese.

#1115 - Unknown character set: 'utf8mb4'

I have a local webserver running on my pc to which I use for local development. I'm now at the stage of exporting the database and importing onto my hosted VPS.
When exporting then importing I get the following error!
1115 - Unknown character set: 'utf8mb4'
Can somebody point me in the right direction?
The error clearly states that you don't have utf8mb4 supported on your stage db server.
Cause: probably locally you have MySQL version 5.5.3 or greater, and on stage/hosted VPS you have MySQL server version less then 5.5.3
The utf8mb4 character sets was added in MySQL 5.5.3.
utf8mb4 was added because of a bug in MySQL's utf8 character set.
MySQL's handling of the utf8 character set only allows a maximum of 3
bytes for a single codepoint, which isn't enough to represent the
entirety of Unicode (Maximum codepoint = 0x10FFFF). Because they
didn't want to potentially break any stuff that relied on this buggy
behaviour, utf8mb4 was added. Documentation here.
Solution 1:
Simply upgrade your MySQL server to 5.5.3 (at-least) - for next time be conscious about the version you use locally, for stage, and for prod, all must have to be same.
A suggestion - in present the default character set should be utf8mb4.
Solution 2 (not recommended): Convert the current character set to utf8, and then export the data - it'll load ok.
Sometimes I get similar problems while using HeidiSQL which by default exports in utf8mb4 character encoding. Not all MySQL installations support this encoding and importing such data leads to similar error messages. My workaround then is to export data using phpMyAdmin, which exports in utf8. There are problably other tools and possible ways like manually editing dump file, converting it from utf8mb4 to utf8 (if needed) and changing SET NAMES utf8mb4 to SET NAMES utf8. Utf8mb4 is a superset of utf8, so if you're absolutely sure, that your data is just utf8, then you can simply change SET NAMES in dump file to utf8.
Open sql file by text editor find and replace all
utf8mb4 to utf8
Import again.
This helped me
ALTER TABLE your_table_name CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;

incorrect output 4 bytes symbols in mysql table with utf8mb4 encode

I want to insert via phpmyadmin 4 bytes character in the tabel. (phpmyadmin version is 5.5.33).
I assigned Server connection collation to utf8mb4_general_ci collation;
Database has utf8mb4 encode;
Table and column has utf8mb4 encode;
I tryed to insert 𩸽 symbol and it was success and without any errors! But this symbol in the table is displayed as ????.
Can someone help, please?
So I would reccomend you to check what is the application web encoding because your problem is not the data itself is the program that is printing it. If your php administration tool or the web container (apache most probably) that is hosting this application doesn't have your character encoding you wont see your character. Most of theese application use just UTF8 as encoding therefore I suggest you to change your database to this encoding just UTF8 and the collation to utf8_general_ci.
Your question is most probably related with this one How to display UTF-8 characters in phpMyAdmin?

default database collation not respected while importing

In my database, the collation was originally utf8_general_ci. However, I noticed that utf8_unicode_ci is necessary because of better sorting accuracy.
So I exported all database using phpmyadmin and checked that the word "COLLATION" does not appear in the exported sql file (except for only 2 times in one table where it is set to binary) so generally this script is collation agnostic and should not imply any specific collation when importing but use database default.
After dropping all tables, the database collation was changed to utf8_unicode_ci and then the import script was run from phpmyadmin. But as a result, all tables and all columns are shown again with utf8_general_ci collation (and sorting is incorrect). Why?? And what to do to change it?
P.S. The export/import script contains commented line at the beginning:
/*!40101 SET #OLD_COLLATION_CONNECTION=##COLLATION_CONNECTION */;
I don't know if it has any impact while importing, but after opening mysql console, the command show variables like 'collation_connection'shows COLLATION_CONNECTION as cp852_general_ci.
However, in phpmyadmin->variables the variable 'collation_connection' is set to utf8_general_ci. But there is no way to change it.
That happens because the database export is setting the character set on every table, and such a clause comes with a default collation that depends on the character set, not on the collation of your connection. utf8_general_ci is the default collation for utf8.
You'll have to convert your tables with something like ALTER TABLE tablename CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci; or edit your database export if this is affordable.
As for the MySQL console: the command-line client is pretty much broken on Windows. It'll never support, display or read Unicode, and you're getting a per-connection collation for that client that matches your Windows so-called OEM character set for your locale. This is a Windows misfeature that's difficult to workaround in portable software. PHPMyAdmin uses a web server and doesn't suffer from this problem. I advise you to use a UNIX-like operating system like GNU/Linux for any serious work in any case, not just for this reason. As an added benefit, MySQL, Apache and your whole application stack perform better on Linux.

what to use latin1 or utf8 for rails 3 & ruby 1.8.7

I found that utf-8 is the standard , but Mysql doesn't fully support utf-8(4bytes) .
Now here is my situation.
My collation variables of mysql show latin1 but the encoding mechanism in "database.yml"(rails) is utf8 .
I have a database with around 20 tables and around 1k rows in each table.
I am using Mysql 5.0 , ruby 1.8.7 and rails 3 .
I would like to support at least a few non printable charters.
What would be the best option ?
Should I change the whole database to utf-8 ( converting is painful and also mysql fully doesn't support utf-8).
Should I change the encoding mechanism in "database.yml" to latin1 ( will the new setting be compatible with old data that is already stored ).
Is there any other solution ?
Thank you.
I think that the :encoding field in database.yml
is just what is used when creating a new database.
It won't break your app if you change this field
(in fact I think it won't do anything, unless you use rake db:create)
I suggest, if your app is targeting anyone outside of the US or western europe
you should be using utf8.
I find it surprising that Ubuntu still ships with the default as latin1.
This may be "optimised" for space
but causes a hassle for almost anyone with a client facing website.
There are various links on google for how to convert your database to UTF8
http://www.devcha.com/2008/03/convert-existing-mysql-database-from.html
something like
/* convert the default character set (used for new tables) */
ALTER DATABASE db_name DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
/* convert a specific table */
ALTER TABLE db_table CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
of course....
always back up your data, and try it out on a staging machine first