I'm using Squeryl to work with a MySQL database. The tables are in utf8mb4 encoding. Now I want to insert some utf8 (4 byte) strings into the db through Squeryl. How do I do that?
I tried to set ?useUnicode=yes&characterEncoding=UTF-8 to my connection url but apparently, UTF-8 here is 3 byte to MySQL so it doesn't work.
I found this StackOverflow answer, but after some digging, I don't see anyway to append my queries with SET NAMES utf8mb4; (changing database config and environment is not an option)
Example string: อลิซร้องเพลงตามเลยค่ะ😂😂😂
Error when trying to insert the string:
Exception in thread "main" org.squeryl.SquerylSQLException: Exception while executing statement : Incorrect string value
Be sure not to connect as root.
Have this in my.cnf (in the [mysqld] section)
init_connect = SET NAMES utf8mb4
Related
The mysql5 db has been used to latin1 and latin1_general_ci character encoding settings, all tables are used also this character set.
After migration to mysql8(create and restore dump from mysql5-to-mysql8) the error: Unknown character set index for field '255' received from server. occurs when doing connection to db.
Why does it happens?, I suppose that it may be related with a fact that mysql8 uses utf8mb4 character set encoding as a default.
But utf8mb4 encoding is more wide than latin1 so it must to be support to migration from latin1(1Byte) to utf8mb4(4Bytes) not vice verse.
The changing character set of the db to latin1 and collation to latin1_swedish_ci doesn't take affect.
There is some mysql8 params:
SHOW VARIABLES LIKE 'char%';
character_set_client utf8
character_set_connection utf8
character_set_database utf8mb4
character_set_filesystem binary
character_set_results utf8
character_set_server utf8mb4
character_set_system utf8
character_sets_dir C:\Program Files\MySQL\MySQL Server 8.0\share\charsets\
MySql8 connector driver is:
mysql-connector-java v. 3.1.14
I have this issue when I am executing scripts from my Java application. It is showing the same error as mentioned in the subject line.
The fix is(in my case,as we earlier use latin1): provide character encoding type in connection url.
?characterEncoding=latin1
Here is the list of available types which are used between java and mysql. This is the official documentation of java-mysql connector jar.
The only solution for this problem is
download the latest version of connector from the following website..
https://dev.mysql.com/downloads/connector/j/
When you go there there was a option to select operating system if you use windows then select the platform independent option then a download option appears ... download the zip file and extract it.Then go to your project and add this jar file into the libraries.So that's all your problem will be solved.
Hope this solution work for you...
When deploying my jhipster based application to cloud foundry (in my case Pivotal with ClearDB service) I don't have option to change the DB character set and not to update the JDBC parameters as it shared DB.
the charset of the DB is latin1 and I need it to be utf-8 to be able support languages like Arabic and Hebrew.
So the only option I think about to support those languages is to init the DB session/connection when it's created, like running below sqls:
SET session character_set_client = charset_name;
SET session character_set_results = charset_name;
SET session character_set_connection = charset_name;
How this can be done in jhipster I don't see place where we can set DB connection/session init sqls and if you have any other recommendation?
Currently what happen is that Arabic/Hebrew input data coming from client saved in the DB as ????
BTW if I will update the DB entries using MYSQL Workbench the Arabic/Hebrew values are save correctly and also displayed correctly.
Thanks,
Rabiaa
The data was destroyed during INSERTion.
The bytes to be stored are not encoded as utf8/utf8mb4. Fix this.
The column in the database is CHARACTER SET utf8 (or utf8mb4). Fix this.
Also, check that the connection during reading is UTF-8.
See "question marks" in http://stackoverflow.com/questions/38363566/trouble-with-utf8-characters-what-i-see-is-not-what-i-stored
Is that one question mark per character? Or two (in the case of Arabic / Hebrew)?
Workaround to solve the issue, by creating proxy service:
1 Unbind the clearDB service (but keep it, just unbind)
2 Create new user provided service which will call the clearDB service with custom uri.
See the following commands:
cf create-user-provided-service mysql-db -p '{"uri":"mysql://<uri of the clearDB service>?useUnicode=true&characterEncoding=utf8&reconnect=true"}'
cf bind-service <app name> mysql-db
cf restart <app name>
We are using Dropwizard, JDBI, MySql 5.6 and mysql connector 5.1.32 and use a Pooled data source. In order to support emojis, the only way I have found is to call the query "SET NAMES utf8mb4" on the connection whenever the connection is obtained.
But under load we are observing that this query takes a long time (around 222 ms).. Is there any alternative to this query?
Things tried so far:
1. Tried setting charSet, characterEncoding on jdbc connection url
2. The columns in the table use utf8mb4 encoding and utf8mb4_unicide_ci collation
3. MySql is on RDS, not yet changed the character_set_server etc. variables on RDS
I'm running java over jetty, on an EC2 linux instance, using MySQL DB. The column is a VARCHAR, set to accept utf8mb4 encoding.
After playing around with stuff, I've found out that it works when I run this code through gradle jettyRunWar, or even when running the same code on a tomcat server.
It doesn't work when I place the exact same war that was working before in $JETTY_HOME/webapps/root.war, then run jetty with sudo service jetty start.
The error shown is -
java.sql.BatchUpdateException: Incorrect string value: '\xF0\x9F\x99\x89' for column 'name' at row 1
Current column definition -
`name` varchar(50) CHARACTER SET utf8mb4 DEFAULT NULL
Value is set in SQL through preparedStatement.setString(...) and I made sure that mysql connector JAR is the same.
Any ideas?
Problem resolved once I've set all character_set_... MySQL DB variables to utf8mb4.
That probably means that newer version of jetty (8+) treat JDBC connection different than Tomcat or than older version, because the exact connection string, beans definition & DB was used at all cases.
Until now, character_set_... params were set to utf8 with specific columns defined as utf8mb4, that used to be enough.
Hopes that saves anyone else the day I was stuck on it.
My problem:
I'v been given a mysql database in a non-English language (Persian or Farsi, & if you don't know what kind of language is that, it's like Arabic). The records were entered through a web interface by php, using a windows machine. When I want to view the database using phpMyAdmin in xampp, the records look like this:
مرکز آموزش توپخانه نزاجا
If I edit the records in phpMyAdmin, I can add non-English (Persian) characters and they look fine, only the existing data is incorrectly displayed.
I've been provided with a .sql backup file as well, but when I open it in Notepad++ it doesn't look right either. I also tried "Encode in UTF-8" in Notepad++, but no use.
What I want:
A correct representation in phpMyAdmin or a healthy .sql file.
What I have:
xampp 1.8.2 (Apache 2.4.4, MySQL 5.5.32, PHP 5.4.16, phpMyAdmin 4.0.4), win 7 x64
The files I have:
.frm .MYD .MYI files (which I copied to xampp\mysql\data\mxpro), the .sql file i mentioned (mxpro.sql) & db.opt file containing these 2 lines:
default-character-set=utf8
default-collation=utf8_general_ci
I've found this line included inside the .sql file:
CHARSET=latin1
All of these files are inside a folder called 'mxpro' located in xampp\mysql\data\.
The collation of the table columns in the phpMyAdmin are: latin1_swedish_ci
What I have tried:
First of all, when I open the MYI file in Notepad++ and use "Encode in UTF-8", I can see most of the data sitting there in the correct format (Persian).
I've tried the following based on my research:
1) Changing whatever I see to utf8_general_ci, including: database (mxpro) collation (through operations), table collation (through operations), columns collation & server connection collation (in general settings)
2) Changing these server variables to utf8: character set client, character set connection, character set database, character set results, character set server & character set system.
3) Changing these server variables to utf8_unicode_ci: collation connection, collation database & collation server.
4) Adding this line:
#MySQL_Query("SET NAMES utf8");
to xampp\php\pear\MDB2\Driver\mysql.php after this line:
$connection = #call_user_func_array($connect_function, $params);
5) Adding these 3 lines to my.ini:
collation_server=utf8_unicode_ci
character_set_server=utf8
skip-character-set-client-handshake
6) Adding these 2 lines:
mysqli_query($link, "SET SESSION CHARACTER_SET_RESULTS =latin1;");
mysqli_query($link, "SET SESSION CHARACTER_SET_CLIENT =latin1;");
to xampp\phpMyAdmin\libraries\dbi\mysql.dbi.lib.php, below the following line:
PMA_DBI_postConnect($link, $is_controluser);
7) Changing this:
'utf-8' => 'utf8',
to this:
'utf-8' => 'latin1',
in xampp\phpMyAdmin\libraries\select_lang.lib.php
despite my efforts, no outcome yet.
Thank you in advance.
On the phpMyAdmin wiki, there is an article explaining this issue:
http://wiki.phpmyadmin.net/pma/Garbled_data
I used this query on my db and it's worked perfectly
ALTER DATABASE yourDB CHARACTER SET utf8 COLLATE utf8_bin