i have a problem with inserting data with polish chars to Mysql DB. Im working on windows 8 and Ubuntu. At Windows there is no problem but on ubuntu i can not insert that kind of chars: "żąśźćłż" in place of them i get: "?????". I have checked with TRACE lvl of logging. My application put correct Strings to prepared query but in db i see "???????". I can insert that kind of chars via cmd and its ok, so problably there is some problem with connector? Or some other settings. I have tried change:
mysql> show variables like "collation%";;
+----------------------+--------------------+
| Variable_name | Value |
+----------------------+--------------------+
| collation_connection | utf8_general_ci |
| collation_database | latin1_swedish_ci |
| collation_server | latin1_swedish_ci |
+----------------------+--------------------+
to
utf8_general_ci
every where but after service(mysql) restart its come back with the same with
mysql> show variables like "character%";
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
I can not set utf8 for database and server.
Anyone have some ideas?
Adding the line
character_set_server = utf8
in the [mysqld] section of the MySQL configuration file (my.ini or my.cnf) should set the new value the next time the MySQL server is started.
Related
I am trying to create a database using the utf8mb4 character set and utf8mb4_unicode_ci collation. However, I don't seem to be able to insert unicode characters into my tables.
What I have done:
SET NAMES utf8mb4;
CREATE DATABASE mydb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
USE mydb;
CREATE TABLE test (val VARCHAR(16));
INSERT INTO test (val) VALUES ("á");
ERROR 1366 (22007): Incorrect string value: '\xA0' for column `mydb`.`test`.`val` at row 1
If I don't use SET NAMES utf8mb4;, then I can insert the "á" character without issue.
These are my default character set variables:
show variables like 'char%'; show variables like 'collation%';
+--------------------------+-----------------------------------------------+
| Variable_name | Value |
+--------------------------+-----------------------------------------------+
| character_set_client | cp850 |
| character_set_connection | cp850 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | cp850 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | C:\Program Files\MariaDB 10.5\share\charsets\ |
+--------------------------+-----------------------------------------------+
8 rows in set (0.000 sec)
+----------------------+--------------------+
| Variable_name | Value |
+----------------------+--------------------+
| collation_connection | cp850_general_ci |
| collation_database | utf8mb4_unicode_ci |
| collation_server | utf8_general_ci |
+----------------------+--------------------+
3 rows in set (0.000 sec)
And after using SET NAMES:
show variables like 'char%'; show variables like 'collation%';
+--------------------------+-----------------------------------------------+
| Variable_name | Value |
+--------------------------+-----------------------------------------------+
| character_set_client | utf8mb4 |
| character_set_connection | utf8mb4 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | utf8mb4 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | C:\Program Files\MariaDB 10.5\share\charsets\ |
+--------------------------+-----------------------------------------------+
8 rows in set (0.000 sec)
+----------------------+--------------------+
| Variable_name | Value |
+----------------------+--------------------+
| collation_connection | utf8mb4_general_ci |
| collation_database | utf8mb4_unicode_ci |
| collation_server | utf8_general_ci |
+----------------------+--------------------+
3 rows in set (0.000 sec)
How can I fix this issue so I can insert characters in the utf8mb4 character set?
Your text (or .sql) file itself is encoded in cp850 and not in utf-8.
You can see that encoded value is a single byte - UTF-8 encoding should be at least 2 bytes.
In order to use SET NAMES utf8mb4; command, your file needs to be converted to utf-8. Some advanced editors allow that, and even windows notepad can save a text file as utf-8 in modern versions.
If you are using Windows cmd, the command "chcp" controls the "code page". chcp 65001 provides utf8, but it needs a special charset installed, too.
To set the font in the console window: Right-click on the title of the window → Properties → Font → pick Lucida Console
I am running a webapp on Ubuntu 16.04.4.
The stack is as follows
Python 3.5.2
MySQL 5.7.22
Flask
Flask-SQLAlchemy
The webapp has a feature for admins to upload some text using a xlsx. file which is read with openpyxl inside the webapp. However while saving I am getting errors like:
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1366, "Incorrect string value: '\\xC4\\x9B nep...'
In the beginning I was able to delete the characters which were making troubles (e.g. zero width whitespace). But now I am not able to do it anymore like this.
Reading a bit on the internet I think it could be that my db is not using utf8mb4. Could someone lead me to update my db and all its tables? Because I do not know anything about SQL and stuff.
As the webapp is used in production I do not like to try tutorials which are outdated.
Seems to work now. I did following steps:
Started the mysql cli with:
mysql -u root -p
Logged in using the root pw.
Checked the default parameters using
show variables like "%character%";
which gave me:
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
and
show variables like "%collation%";
which gave me
+----------------------+-------------------+
| Variable_name | Value |
+----------------------+-------------------+
| collation_connection | utf8_general_ci |
| collation_database | latin1_swedish_ci |
| collation_server | latin1_swedish_ci |
+----------------------+-------------------+
So I edited /etc/mysql/my.cnf
I added:
[client]
default-character-set = utf8mb4
[mysql]
default-character-set = utf8mb4
[mysqld]
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
Restarting mysql (sudo service mysql restart) and running the same commands as above now gave me
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8mb4 |
| character_set_connection | utf8mb4 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | utf8mb4 |
| character_set_server | utf8mb4 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
and
+----------------------+--------------------+
| Variable_name | Value |
+----------------------+--------------------+
| collation_connection | utf8mb4_general_ci |
| collation_database | utf8mb4_unicode_ci |
| collation_server | utf8mb4_unicode_ci |
+----------------------+--------------------+
So I looked up the table settings using
SHOW TABLE STATUS FROM databasename;
They still used stuff like latin1_swedish_ci
I used following to change the database setting:
ALTER DATABASE databasename CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
and following for each table:
use databasename;
ALTER TABLE assessments CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
Looking up again the table settings showed that latin1_swedish_ci was now changed to utf8mb4_unicode_ci
Then I changed the sqlalchemy connection url to use ?encoding=utf8mb4 at the end.
Restarted mysql again and the webapp. Since then it's working properly.
I have a weird issue. I have an embedded system running Linux, QT 4.8 (for touch screen) and MySQL 5.7.8. In my database, I have some entries that contained French accent characters such as "é, à, À, etc"
So, I start mysqld process first, then start my application running QT that will open the database.
This will give me corrupted characters on the screen. For example, the "É" will become "Ã%".
If I restart mysqld, all the French characters will be displayed properly on my screen.
So, it seems my QT application can handle French characters, but I need to restart mysqld to make it happened!
I wanted to start my QT app first and then mysqld after, but I need to get access to the database and because it is not started yet, my app gives an error.
Any clue why?
UPDATE 20170524
Here is my database info:
mysql> show variables like "collation_database";
+--------------------+-------------------+
| Variable_name | Value |
+--------------------+-------------------+
| collation_database | latin1_swedish_ci |
+--------------------+-------------------+
mysql> show variables like "%character%"; show variables like "%collation%";
+--------------------------+-------------------------------------------------+
| Variable_name | Value |
+--------------------------+-------------------------------------------------+
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | latin1 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /mnt/data/part1/usr/local/mysql/share/charsets/ |
+--------------------------+-------------------------------------------------+
+----------------------+-------------------+
| Variable_name | Value |
+----------------------+-------------------+
| collation_connection | latin1_swedish_ci |
| collation_database | latin1_swedish_ci |
| collation_server | latin1_swedish_ci |
+----------------------+-------------------+
I'm trying to change my RDS instance default character set to utf8mb4 so I can support emojis. I have a repo with all my recipes, which I've used in the past to customize my deployments. I followed this guide from AWS, but when I deploy the app, the changes aren't reflected in the database. I also made sure to create a metadata.rb file in the root of the mysql directory in my custom cookbooks repo.
I also setup a new RDS instance using a new property group where the appropriate character sets and collations are set to utf8mb4. This DB is also set as my datasource in my app in OpsWorks.
In Rails, I also set the encoding and collation to utf8mb4...
default: &default
adapter: mysql2
encoding: utf8mb4
collation: utf8mb4_general_ci
...
If I ssh into my application server and then connect to MySql, this is what I see when querying for global variables...
mysql> SHOW GLOBAL VARIABLES WHERE Variable_name LIKE 'character\_set\_%' OR Variable_name LIKE 'collation%';
+--------------------------+--------------------+
| Variable_name | Value |
+--------------------------+--------------------+
| character_set_client | utf8mb4 |
| character_set_connection | utf8mb4 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | utf8mb4 |
| character_set_server | utf8mb4 |
| character_set_system | utf8 |
| collation_connection | utf8mb4_unicode_ci |
| collation_database | utf8mb4_unicode_ci |
| collation_server | utf8mb4_unicode_ci |
+--------------------------+--------------------+
But when I do the same for non-global variables, I see this:
mysql> show variables like 'char%';
+--------------------------+-------------------------------------------+
| Variable_name | Value |
+--------------------------+-------------------------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8mb4 |
| character_set_system | utf8 |
| character_sets_dir | /rdsdbbin/mysql-5.6.23.R1/share/charsets/ |
+--------------------------+-------------------------------------------+
8 rows in set (0.00 sec)
I should also mention that this works fine locally after I set the appropriate tables and columns to utf8mb4 using a migration.
At this point, I can't figure out why I can't get the character sets to apply correctly. Hopefully someone smarter than me can help me figure this out!
Thanks
After digging a bit deeper, I realized that I missed a critical step when updating custom cookbooks. I didn't realize that simply doing a deployment wouldn't retrieve the new recipes. After I ran the command to "Update Custom Cookbooks" and then did a deployment, it worked.
Hopefully this helps someone in the future.
I've implemented simple web server to have proxy to MySQL using 'sqljocky' package.
And I have issue with character encoding, cyrillic glyphs displays incorrectly:
ÐавÑдов ÐиÑалий instead of Давыдов Витайлий
EDIT: Table collation is utf8_general_ci.
I've tried to query SET NAMES UTF8:
pool.query('set names utf8');
[UPDATED] Then I've created my.cnf in /etc/ directory with this content:
[mysqld]
character-set-server=utf8
collation-server=utf8_general_ci
Output of show variables like "%char%";
mysql> show variables like "%char%";
+--------------------------+--------------------------------------------------------+
| Variable_name | Value |
+--------------------------+--------------------------------------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /usr/local/mysql-5.6.12-osx10.7-x86_64/share/charsets/ |
+--------------------------+--------------------------------------------------------+
8 rows in set (0,00 sec)
Output of show variables like 'collation%'
mysql> show variables like 'collation%';
+----------------------+-----------------+
| Variable_name | Value |
+----------------------+-----------------+
| collation_connection | utf8_general_ci |
| collation_database | utf8_general_ci |
| collation_server | utf8_general_ci |
+----------------------+-----------------+
3 rows in set (0,00 sec)
But still having incorrect displayed characters.
How to get it displayed correctly?
There was a bug in sqljocky which meant that unicode characters weren't being encoded properly. I have fixed some of the places where the bug was occurring in v0.5.5 which I have just published. When I have more time I will make sure that it is fixed everywhere.