Inserting Emoji into MySQL [duplicate] - mysql

I have a MySQL database configured with the default collation utf8mb4_general_ci. When I try to insert a row containing an emoji character in the text using the following query
insert into tablename
(column1,column2,column3,column4,column5,column6,column7)
values
('273','3','HdhdhdhπŸ˜œπŸ˜€πŸ˜ŠπŸ˜ƒhzhzhzzhjzj ζˆ‘ηˆ±δ½  ❌',49,1,'2016-09-13 08:02:29','2016-09-13 08:02:29');
MySQL is raising the following error
1366 Incorrect string value: '\xF0\x9F\x98\x83\xF0\x9F...' for column
'comment' at row 1

1) Database: Change Database default collation as utf8mb4.
2) Table: Change table collation as CHARACTER SET utf8mb4 COLLATE utf8mb4_bin.
Query:
ALTER TABLE Tablename CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin
3) Code:
INSERT INTO tablename (column1, column2, column3, column4, column5, column6, column7)
VALUES ('273', '3', 'HdhdhdhπŸ˜œπŸ˜€πŸ˜ŠπŸ˜ƒhzhzhzzhjzj ζˆ‘ηˆ±δ½  ❌', 49, 1, '2016-09-13 08:02:29', '2016-09-13 08:02:29')
4) Set utf8mb4 in database connection:
$database_connection = new mysqli($server, $user, $password, $database_name);
$database_connection->set_charset('utf8mb4');

Step 1, change your database's default charset:
ALTER DATABASE database_name CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
if the db is not created yet, create it with correct encodings:
CREATE DATABASE database_name DEFAULT CHARSET = utf8mb4 DEFAULT COLLATE = utf8mb4_unicode_ci;
Step 2, set charset when creating table:
CREATE TABLE IF NOT EXISTS table_name (
...
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE utf8mb4_unicode_ci;
or alter table
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE table_name MODIFY field_name TEXT CHARSET utf8mb4;

The command to modify the column is:
ALTER TABLE TABLE_NAME MODIFY COLUMN_NAME TYPE;
And we need to use type = BLOB
Example to modify is as under:-
ALTER TABLE messages MODIFY content BLOB;
I checked that latest mySQL and other databases don't need '' to use in command on table_name, column_name etc.
Fetch and Save data:
Directly save the chat content to column and to retrieve data, fetch data as byte array (byte[]) from db column and then convert it to string e.g. (Java code)
new String((byte[]) arr)

Both the databases and tables should have character set utf8mb4 and collation utf8mb4_unicode_ci.
When creating a new database you should use:
CREATE DATABASE mydb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
If you have an existing database and you want to add support:
ALTER DATABASE database_name CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
You also need to set the correct character set and collation for your tables:
CREATE TABLE IF NOT EXISTS table_name (
...
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE utf8mb4_unicode_ci;
or change it if you've got existing tables with a lot of data:
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
Note that utf8_general_ci is no longer recommended best practice. See the related Q & A:
What's the difference between utf8_general_ci and utf8_unicode_ci on Stack Overflow.

If you are using Solr + Mysql + Java, you can use:
This can be Used :
case1: When you don`t want to alter DB.
case2: when you have to import emoticons from your Mysql to Solr core.
In above case this is one of the solutions to store your emoticons in your system.
Steps to use it:
Library used: import java.net.URLDecoder;
import java.net.URLEncoder;
Use urlEncoder to encode your String having emoticons.
Store it in DB without altering the MysqlDB.
You can store it in solr core(decoded form)if you want or you can store
encoded form.
When fetching these emoticons from DB or Solr core you can now decode it
Using urlDecoder.
Code example:
import java.net.URLDecoder;
import java.net.URLEncoder;
public static void main(String[] args) {
//SpringApplication.run(ParticipantApplication.class, args);
System.out.println(encodeStringUrl("πŸ‡ΊπŸ‡ΈπŸ‡¨πŸ‡³πŸ‡―πŸ‡΅πŸ‡©πŸ‡ͺπŸ”³πŸ”ΊπŸ†”πŸ†”πŸ†‘3⃣5⃣3βƒ£β€Όγ€½βž—βž—πŸŽ¦πŸ”†πŸŽ¦πŸ”†β™‹β™β™‹β™β¬…β¬†β¬…β¬…πŸ›‚πŸšΉπŸ›‚πŸ›„πŸš³πŸš¬πŸ’ŠπŸ”§πŸ’ŠπŸ—Ώ "));
System.out.println(decodeStringUrl("Hello+emoticons%2C%2C%F0%9F%98%80%F0%9F%98%81%F0%9F%98%8A%F0%9F%98%8B%F0%9F%98%8E%F0%9F%98%8A%F0%9F%98%8D%E2%98%BA%F0%9F%98%98%E2%98%BA%F0%9F%98%91%F0%9F%98%87%F0%9F%98%98%F0%9F%98%8B%F0%9F%90%84"));
}
public static String encodeStringUrl(String url) {
String encodedUrl =null;
try {
encodedUrl = URLEncoder.encode(url, "UTF-8");
} catch (UnsupportedEncodingException e) {
return encodedUrl;
}
return encodedUrl;
}
public static String decodeStringUrl(String encodedUrl) {
String decodedUrl =null;
try {
decodedUrl = URLDecoder.decode(encodedUrl, "UTF-8");
} catch (UnsupportedEncodingException e) {
return decodedUrl;
}
return decodedUrl;
}

I have updated my database and table to upgraded from utf8 to utf8mb4. But nothing works for me. Then I tried to update column datatype to blob, luckily it worked for me and data has been saved. Even my database and table both are CHARACTER SET utf8 COLLATE utf8_unicode

My answer only adds to Selvamani P answer.
You might also need to change any SET NAMES utf8 queries with SET NAMES utf8mb4. That did the trick for me.
Also, this is a great article to port your website from utf8 to utf8mb4. In particular the article makes 2 good points on indexes and repairing tables after converting them to utf8mb4:
INDEXES
When converting from utf8 to utf8mb4, the maximum length of a column
or index key is unchanged in terms of bytes. Therefore, it is smaller
in terms of characters, because the maximum length of a character is
now four bytes instead of three. [...] The InnoDB storage engine has a maximum index length of 767 bytes, so for utf8 or utf8mb4 columns, you can index a maximum of 255 or 191 characters, respectively. If you currently have utf8 columns with indexes longer than 191 characters, you will need to index a smaller number of characters when using utf8mb4.
REPAIRING TABLES
After upgrading the MySQL server and making the necessary changes
explained above, make sure to repair and optimize all databases and
tables. I didn’t do this right away after upgrading (I didn’t think it
was necessary, as everything seemed to work fine at first glance), and
ran into some weird bugs where UPDATE statements didn’t have any
effect, even though no errors were thrown.
Read more about the queries to repair tables on the article.

I have a good solution to save your time. I also meet the same problem but I could not solve this problem by the first answer.
Your defualt character is utf-8. But emoji needs utf8mb4 to support it.
If you have the permission to revise the configure file of mysql, you can follow this step.
Therefore, do this following step to upgrade your character set ( from utf-8 to utf8mb4).
step 1. open your my.cnf for mysql, add these following lines to your my.cnf.
[mysqld]
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
init_connect='SET NAMES utf8mb4'
[mysql]
default-character-set = utf8mb4
[client]
default-character-set = utf8mb4
step2. stop your mysql service, and start mysql service
mysql.server stop
mysql.server start
Finished!
Then you can check your character are changed into utf8mb4.
mysql> SHOW VARIABLES LIKE 'character_set%';
+--------------------------+----------------------------------------------------------+
| Variable_name | Value |
+--------------------------+----------------------------------------------------------+
| character_set_client | utf8mb4 |
| character_set_connection | utf8mb4 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | utf8mb4 |
| character_set_server | utf8mb4 |
| character_set_system | utf8 |
| character_sets_dir | /usr/local/Cellar/mysql#5.7/5.7.29/share/mysql/charsets/ |
+--------------------------+----------------------------------------------------------+
8 rows in set (0.00 sec)

Well, you need not to change the Whole DB Charset. Instead of that you can do it by changing column to blob type.
ALTER TABLE messages MODIFY content BLOB;

There are two ways-->
# Way one
The simplest is to follow below steps:
Step 1:
SET NAMES utf8mb4;
Step 2:
ALTER DATABASE database_name CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
Step 3:
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
Step 4:
ALTER TABLE table_name CHANGE column column VARCHAR(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL;
That's all!!
#Way Two (For Python)
This is a hack and its work like a charm
Step 1:
Encode your string and decode it in ASCII, and save it to your database.
content = 'πŸ₯³πŸ₯³ Content to be save in πŸ₯³πŸ₯³ Database πŸ₯³πŸ₯³'
encoded_content = content.encode('unicode-escape').decode('ASCII'))
This simply store encoded_content string in DB
Step 2:
While fetch this column data to show your user, simply convert it,
here content is the data, fetched from the database.
c = bytes(encoded_content, 'utf-8')
original_content = c.decode('unicode-escape')
Done!!

Emoji support for application having tech stack - mysql, java, springboot, hibernate
Apply below changes in mysql for unicode support.
ALTER DATABASE <database-name> CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
ALTER TABLE <table-name> CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
DB Connection - jdbc url change:
jdbc:mysql://localhost:3306/<database-name>?useUnicode=yes&characterEncoding=UTF-8
Note - If the above step is not working please update mysql-connector version to 8.0.15. (mysql 5.7 works with connector version 8.0.15 for unicode support)

The main point hasn't been mentioned in the above answers that,
We need to pass query string with the options "useUnicode=yes" and "characterEncoding=UTF-8" in connection string
Something like this
mysql://USERNAME:PASSWORD#HOSTNAME:PORT/DATABASE_NAME?useUnicode=yes&characterEncoding=UTF-8

The simplest solution what works for me is to store the data as json_encode.
later when you retrieve just make sure you json_decode it.
Here you don't have to change the collation or the character set of the database and the table.

For Rails, next to the accepted answer, don't forget to add:
encoding: utf8mb4
collation: utf8mb4_bin
to your database.yml

For anyone trying to solve this on a managed MySQL instance (in my case on AWS RDS), the easiest way was to modify the parameter group and set the server character set and collation to be utf8mb4 and utf8mb4_bin, respectively. After rebooting the server, a quick query verifies the settings for system databases and any newly created ones:
SELECT * FROM information_schema.SCHEMATA S;

If you are inserting using PHP, and you have followed the various ALTER database and ALTER table options above, make sure your php connection's charset is utf8mb4.
Example of connection string:
$this->pdo = new PDO("mysql:host=$ip;port=$port;dbname=$db;charset=utf8mb4", etc etc
Notice the "charset" is utf8mb4, not just utf8!

Today I am facing the same question, but solutions in other answers don't work for me. Here is my solution.
First of all, changing charset in mysql/my.ini, database, and the table is necessary, as described in other answers.
Second, if you have created your tables before you want to saving emoji, you can use
SHOW FULL COLUMNS FROM `yourcolumn`;
To check whether the column you want to save emoji is set in utf8mb4. You can find that most of your columns are still in utf8 charset.
Use
ALTER TABLE `yourtable` CHANGE `yourcolumn` `yourcolumn` VARCHAR(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

I'm facing this issue when upgrading MySQL 5.0 to MySQL 8.0 AWS RDS, trying many things finally what works for me share with you folks.
Error:
Warning: PDOStatement::execute(): SQLSTATE[HY000]: General error:
3988 Conversion from collation utf8_unicode_ci into utf8mb4_general_ci
impossible for parameter in /var/www/html/pdo_con.php on line 87
Array (
[0] => HY000
[1] => 3988
[2] => Conversion from collation utf8_unicode_ci into utf8mb4_general_ci impossible for parameter )
Backend: PHP5/php7 + PDO is giving trouble.
Solution: only two thing needs to do
Add a code in line after your pdo connection
$conn->exec("set names utf8mb4");
where $conn is connection handler in PDO
Alter the table and set charset utf8mb4 and collate utf8mb4_unicode_ci.
ALTER TABLE mytable CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
This command will change your every column charset to utf8mb4 and collation too.

Put this right before your database call:
mysqli_set_charset($db, "utf8mb4");
This will allow you to input emojis directly into the database table that has been set to Collation: utfmb4_bin. Make sure to set your column to utfmb4 as well.

Hi my friends
This is how I solved this problem and I was happy to teach it to you as well
I am in the Android application
I encrypt a string containing text and emoj and send it to the server and save it in the mysql table and after receiving it from the server I decrypt it and display it in the textview.
encoded and decoded my message before request and after response:
I send Android app messages to mysql via pdo through this method and receive them with pdo. And I have no problem.
I think it was a good way. Please like
Thankful
public void main()
{
String message="hi mester ali moradi 🌦️🌦️ how are you ?";
String encoded_message=encodeStringUrl(message);
String decode_message=decodeStringUrl(encoded_message);
}
public static String encodeStringUrl(String message) {
String encodedUrl =null;
try {
encodedUrl = URLEncoder.encode(message, "UTF-8");
} catch (UnsupportedEncodingException e) {
return encodedUrl;
}
return encodedUrl;
}
public static String decodeStringUrl(String message) {
String decodedUrl =null;
try {
decodedUrl = URLDecoder.decode(message, "UTF-8");
} catch (UnsupportedEncodingException e) {
return decodedUrl;
}
return decodedUrl;
}
message : hi mester ali moradi 🌦️🌦️ how are you ?
encoded : ghgh%F0%9F%98%AE%F0%9F%A4%90%F0%9F%98%A5
decoded :hi mester ali moradi 🌦️🌦️ how are you ?

If you use command line interface for inserting sql file to database.
Be sure your table charset utf8mb4 and column collation utf8mb4_unicode_ci or utf8mb4_bin
mysql -u root -p123456 my_database < profiles.sql
ERROR 1366 (HY000) at line 1679: Incorrect string value: '\xF0\x9F\x98\x87\xF0\x9F...' for column 'note' at row 328
we can solve the problem with this parameter
--default-character-set=name (Set the default character set)
mysql -u root -p123456 --default-character-set=utf8mb4 my_database < profiles.sql

Actually i'm using mysql Ver 8.0.23
I had created the both Database and the Table, without Altering them :
mysql> CREATE DATABASE tp2;
Query OK, 1 row affected (0.30 sec)
mysql> INSERT INTO tweetsRep(username, content) VALUES ('ibrahim', '🀣 oh my god');
Then after select, i thing it just worked fine !
I don't know if it is requested to enter Emoji as a hexadecimal or other encoding string or just copy it as it is... just correct me if i'm wrong, thank you !

I tried different methods and approaches and found a way that worked for me.
The SQL for the update query:
ALTER DATABASE YOUR_DB_NAME_HERE CHARACTER SET = utf8mb4 COLLATE =
utf8mb4_unicode_ci;
You can see in the table, the emoji's are present
And if you go to this page: https://www.thecookingcat.com/recipes/thai-green-curry.php#comments
You can see the emojis in the comments.
I also have an RSS feed on the site and the emojis are included in the RSS feed XML code.

If anyone searching this in 2022 just follow these steps and no need to do any modification on Database
Name Space
using System.Web;
Your normal text like this :
String encode = "thank you 😊"
encode = HttpUtility.UrlEncode(encode);
It will store in Database like this : "thank+you+%f0%9f%98%8a"
And next fetch that data form your Database and do UrlDecode like this
DataSet ds = "Fetch your Encoded data form your Database";
String decode = HttpUtility.UrlDecode(ds.Tables[0].Rows[i]["YourColumnName"].ToString().Trim());
And your output is :-
decode = "thank you 😊".
It is working fine for me and saved time.

Related

MySql and JDBC: change charset of a single column to utf8mb4

I have an "old" database (in utf 8) where I read and write on by using JDBC. Now, I must be able to also store emoji into a column of my table.
I have changed the charset of involved columns to utf8mb4:
ALTER TABLE
myTable
CHANGE column_name column_name
longtext
CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci
NOT NULL;
However, when I try to insert an emoticon into that column, I get the famous error
java.sql.SQLException: Incorrect string value: '\xF0\x9F\x91\x8D\xF0\x9F...'
Should I convert entire database, or am I doing something wrong?
Need to connect with utf8mb4 to get πŸ‘, etc.
?useUnicode=yes&characterEncoding=UTF-8 in the getConnection() call.
As a fallback, execute SET NAMES utf8mb4 after connecting.
(See Comment.)
"For Connector/J 8.0.12 and earlier: In order to use the utf8mb4 character set for the connection, the server MUST be configured with character_set_server=utf8mb4; if that is not the case, when UTF-8 is used for characterEncoding in the connection string, it will map to the MySQL character set name utf8, which is an alias for utf8mb3.'

Error Code: 1267. Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) [duplicate]

I'm getting this strange error while processing a large number of data...
Error Number: 1267
Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '='
SELECT COUNT(*) as num from keywords WHERE campaignId='12' AND LCASE(keyword)='hello again Γ¦Λœβ€ Γ£β€ΉΓ£β€šβ€° Γ£β€šΓ£β€šβ€Ή Γ₯ ´æ‰€'
What can I do to resolve this? Can I escape the string somehow so this error wouldn't occur, or do I need to change my table encoding somehow, and if so, what should I change it to?
SET collation_connection = 'utf8_general_ci';
then for your databases
ALTER DATABASE your_database_name CHARACTER SET utf8 COLLATE utf8_general_ci;
ALTER TABLE your_table_name CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
MySQL sneaks swedish in there sometimes for no sensible reason.
CONVERT(column1 USING utf8)
Solves my problem. Where column1 is the column which gives me this error.
You should set both your table encoding and connection encoding to UTF-8:
ALTER TABLE keywords CHARACTER SET UTF8; -- run once
and
SET NAMES 'UTF8';
SET CHARACTER SET 'UTF8';
Use following statement for error
be careful about your data take backup if data have in table.
ALTER TABLE your_table_name CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
In general the best way is to Change the table collation. However I have an old application and are not really able to estimate the outcome whether this has side effects. Therefore I tried somehow to convert the string into some other format that solved the collation problem.
What I found working is to do the string compare by converting the strings into a hexadecimal representation of it's characters. On the database this is done with HEX(column). For PHP you may use this function:
public static function strToHex($string)
{
$hex = '';
for ($i=0; $i<strlen($string); $i++){
$ord = ord($string[$i]);
$hexCode = dechex($ord);
$hex .= substr('0'.$hexCode, -2);
}
return strToUpper($hex);
}
When doing the database query, your original UTF8 string must be converted first into an iso string (e.g. using utf8_decode() in PHP) before using it in the DB. Because of the collation type the database cannot have UTF8 characters inside so the comparism should work event though this changes the original string (converting UTF8 characters that are not existend in the ISO charset result in a ? or these are removed entirely). Just make sure that when you write data into the database, that you use the same UTF8 to ISO conversion.
I had my table originally created with CHARSET=latin1. After table conversion to utf8 some columns were not converted, however that was not really obvious.
You can try to run SHOW CREATE TABLE my_table; and see which column was not converted or just fix incorrect character set on problematic column with query below (change varchar length and CHARSET and COLLATE according to your needs):
ALTER TABLE `my_table` CHANGE `my_column` `my_column` VARCHAR(10) CHARSET utf8
COLLATE utf8_general_ci NULL;
I found that using cast() was the best solution for me:
cast(Format(amount, "Standard") AS CHAR CHARACTER SET utf8) AS Amount
There is also a convert() function. More details on it here
Another resource here
Change the character set of the table to utf8
ALTER TABLE your_table_name CONVERT TO CHARACTER SET utf8
My user account did not have the permissions to alter the database and table, as suggested in this solution.
If, like me, you don't care about the character collation (you are using the '=' operator), you can apply the reverse fix. Run this before your SELECT:
SET collation_connection = 'latin1_swedish_ci';
After making your corrections listed in the top answer, change the default settings of your server.
In your "/etc/my.cnf.d/server.cnf" or where ever it's located add the defaults to the [mysqld] section so it looks like this:
[mysqld]
character-set-server=utf8
collation-server=utf8_general_ci
Source: https://dev.mysql.com/doc/refman/5.7/en/charset-applications.html

How to store Emoji Character in MySQL Database

I have a MySQL database configured with the default collation utf8mb4_general_ci. When I try to insert a row containing an emoji character in the text using the following query
insert into tablename
(column1,column2,column3,column4,column5,column6,column7)
values
('273','3','HdhdhdhπŸ˜œπŸ˜€πŸ˜ŠπŸ˜ƒhzhzhzzhjzj ζˆ‘ηˆ±δ½  ❌',49,1,'2016-09-13 08:02:29','2016-09-13 08:02:29');
MySQL is raising the following error
1366 Incorrect string value: '\xF0\x9F\x98\x83\xF0\x9F...' for column
'comment' at row 1
1) Database: Change Database default collation as utf8mb4.
2) Table: Change table collation as CHARACTER SET utf8mb4 COLLATE utf8mb4_bin.
Query:
ALTER TABLE Tablename CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin
3) Code:
INSERT INTO tablename (column1, column2, column3, column4, column5, column6, column7)
VALUES ('273', '3', 'HdhdhdhπŸ˜œπŸ˜€πŸ˜ŠπŸ˜ƒhzhzhzzhjzj ζˆ‘ηˆ±δ½  ❌', 49, 1, '2016-09-13 08:02:29', '2016-09-13 08:02:29')
4) Set utf8mb4 in database connection:
$database_connection = new mysqli($server, $user, $password, $database_name);
$database_connection->set_charset('utf8mb4');
Step 1, change your database's default charset:
ALTER DATABASE database_name CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
if the db is not created yet, create it with correct encodings:
CREATE DATABASE database_name DEFAULT CHARSET = utf8mb4 DEFAULT COLLATE = utf8mb4_unicode_ci;
Step 2, set charset when creating table:
CREATE TABLE IF NOT EXISTS table_name (
...
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE utf8mb4_unicode_ci;
or alter table
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE table_name MODIFY field_name TEXT CHARSET utf8mb4;
The command to modify the column is:
ALTER TABLE TABLE_NAME MODIFY COLUMN_NAME TYPE;
And we need to use type = BLOB
Example to modify is as under:-
ALTER TABLE messages MODIFY content BLOB;
I checked that latest mySQL and other databases don't need '' to use in command on table_name, column_name etc.
Fetch and Save data:
Directly save the chat content to column and to retrieve data, fetch data as byte array (byte[]) from db column and then convert it to string e.g. (Java code)
new String((byte[]) arr)
Both the databases and tables should have character set utf8mb4 and collation utf8mb4_unicode_ci.
When creating a new database you should use:
CREATE DATABASE mydb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
If you have an existing database and you want to add support:
ALTER DATABASE database_name CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
You also need to set the correct character set and collation for your tables:
CREATE TABLE IF NOT EXISTS table_name (
...
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE utf8mb4_unicode_ci;
or change it if you've got existing tables with a lot of data:
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
Note that utf8_general_ci is no longer recommended best practice. See the related Q & A:
What's the difference between utf8_general_ci and utf8_unicode_ci on Stack Overflow.
If you are using Solr + Mysql + Java, you can use:
This can be Used :
case1: When you don`t want to alter DB.
case2: when you have to import emoticons from your Mysql to Solr core.
In above case this is one of the solutions to store your emoticons in your system.
Steps to use it:
Library used: import java.net.URLDecoder;
import java.net.URLEncoder;
Use urlEncoder to encode your String having emoticons.
Store it in DB without altering the MysqlDB.
You can store it in solr core(decoded form)if you want or you can store
encoded form.
When fetching these emoticons from DB or Solr core you can now decode it
Using urlDecoder.
Code example:
import java.net.URLDecoder;
import java.net.URLEncoder;
public static void main(String[] args) {
//SpringApplication.run(ParticipantApplication.class, args);
System.out.println(encodeStringUrl("πŸ‡ΊπŸ‡ΈπŸ‡¨πŸ‡³πŸ‡―πŸ‡΅πŸ‡©πŸ‡ͺπŸ”³πŸ”ΊπŸ†”πŸ†”πŸ†‘3⃣5⃣3βƒ£β€Όγ€½βž—βž—πŸŽ¦πŸ”†πŸŽ¦πŸ”†β™‹β™β™‹β™β¬…β¬†β¬…β¬…πŸ›‚πŸšΉπŸ›‚πŸ›„πŸš³πŸš¬πŸ’ŠπŸ”§πŸ’ŠπŸ—Ώ "));
System.out.println(decodeStringUrl("Hello+emoticons%2C%2C%F0%9F%98%80%F0%9F%98%81%F0%9F%98%8A%F0%9F%98%8B%F0%9F%98%8E%F0%9F%98%8A%F0%9F%98%8D%E2%98%BA%F0%9F%98%98%E2%98%BA%F0%9F%98%91%F0%9F%98%87%F0%9F%98%98%F0%9F%98%8B%F0%9F%90%84"));
}
public static String encodeStringUrl(String url) {
String encodedUrl =null;
try {
encodedUrl = URLEncoder.encode(url, "UTF-8");
} catch (UnsupportedEncodingException e) {
return encodedUrl;
}
return encodedUrl;
}
public static String decodeStringUrl(String encodedUrl) {
String decodedUrl =null;
try {
decodedUrl = URLDecoder.decode(encodedUrl, "UTF-8");
} catch (UnsupportedEncodingException e) {
return decodedUrl;
}
return decodedUrl;
}
I have updated my database and table to upgraded from utf8 to utf8mb4. But nothing works for me. Then I tried to update column datatype to blob, luckily it worked for me and data has been saved. Even my database and table both are CHARACTER SET utf8 COLLATE utf8_unicode
My answer only adds to Selvamani P answer.
You might also need to change any SET NAMES utf8 queries with SET NAMES utf8mb4. That did the trick for me.
Also, this is a great article to port your website from utf8 to utf8mb4. In particular the article makes 2 good points on indexes and repairing tables after converting them to utf8mb4:
INDEXES
When converting from utf8 to utf8mb4, the maximum length of a column
or index key is unchanged in terms of bytes. Therefore, it is smaller
in terms of characters, because the maximum length of a character is
now four bytes instead of three. [...] The InnoDB storage engine has a maximum index length of 767 bytes, so for utf8 or utf8mb4 columns, you can index a maximum of 255 or 191 characters, respectively. If you currently have utf8 columns with indexes longer than 191 characters, you will need to index a smaller number of characters when using utf8mb4.
REPAIRING TABLES
After upgrading the MySQL server and making the necessary changes
explained above, make sure to repair and optimize all databases and
tables. I didn’t do this right away after upgrading (I didn’t think it
was necessary, as everything seemed to work fine at first glance), and
ran into some weird bugs where UPDATE statements didn’t have any
effect, even though no errors were thrown.
Read more about the queries to repair tables on the article.
I have a good solution to save your time. I also meet the same problem but I could not solve this problem by the first answer.
Your defualt character is utf-8. But emoji needs utf8mb4 to support it.
If you have the permission to revise the configure file of mysql, you can follow this step.
Therefore, do this following step to upgrade your character set ( from utf-8 to utf8mb4).
step 1. open your my.cnf for mysql, add these following lines to your my.cnf.
[mysqld]
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
init_connect='SET NAMES utf8mb4'
[mysql]
default-character-set = utf8mb4
[client]
default-character-set = utf8mb4
step2. stop your mysql service, and start mysql service
mysql.server stop
mysql.server start
Finished!
Then you can check your character are changed into utf8mb4.
mysql> SHOW VARIABLES LIKE 'character_set%';
+--------------------------+----------------------------------------------------------+
| Variable_name | Value |
+--------------------------+----------------------------------------------------------+
| character_set_client | utf8mb4 |
| character_set_connection | utf8mb4 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | utf8mb4 |
| character_set_server | utf8mb4 |
| character_set_system | utf8 |
| character_sets_dir | /usr/local/Cellar/mysql#5.7/5.7.29/share/mysql/charsets/ |
+--------------------------+----------------------------------------------------------+
8 rows in set (0.00 sec)
Well, you need not to change the Whole DB Charset. Instead of that you can do it by changing column to blob type.
ALTER TABLE messages MODIFY content BLOB;
There are two ways-->
# Way one
The simplest is to follow below steps:
Step 1:
SET NAMES utf8mb4;
Step 2:
ALTER DATABASE database_name CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
Step 3:
ALTER TABLE table_name CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
Step 4:
ALTER TABLE table_name CHANGE column column VARCHAR(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL;
That's all!!
#Way Two (For Python)
This is a hack and its work like a charm
Step 1:
Encode your string and decode it in ASCII, and save it to your database.
content = 'πŸ₯³πŸ₯³ Content to be save in πŸ₯³πŸ₯³ Database πŸ₯³πŸ₯³'
encoded_content = content.encode('unicode-escape').decode('ASCII'))
This simply store encoded_content string in DB
Step 2:
While fetch this column data to show your user, simply convert it,
here content is the data, fetched from the database.
c = bytes(encoded_content, 'utf-8')
original_content = c.decode('unicode-escape')
Done!!
Emoji support for application having tech stack - mysql, java, springboot, hibernate
Apply below changes in mysql for unicode support.
ALTER DATABASE <database-name> CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
ALTER TABLE <table-name> CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
DB Connection - jdbc url change:
jdbc:mysql://localhost:3306/<database-name>?useUnicode=yes&characterEncoding=UTF-8
Note - If the above step is not working please update mysql-connector version to 8.0.15. (mysql 5.7 works with connector version 8.0.15 for unicode support)
The main point hasn't been mentioned in the above answers that,
We need to pass query string with the options "useUnicode=yes" and "characterEncoding=UTF-8" in connection string
Something like this
mysql://USERNAME:PASSWORD#HOSTNAME:PORT/DATABASE_NAME?useUnicode=yes&characterEncoding=UTF-8
The simplest solution what works for me is to store the data as json_encode.
later when you retrieve just make sure you json_decode it.
Here you don't have to change the collation or the character set of the database and the table.
For Rails, next to the accepted answer, don't forget to add:
encoding: utf8mb4
collation: utf8mb4_bin
to your database.yml
For anyone trying to solve this on a managed MySQL instance (in my case on AWS RDS), the easiest way was to modify the parameter group and set the server character set and collation to be utf8mb4 and utf8mb4_bin, respectively. After rebooting the server, a quick query verifies the settings for system databases and any newly created ones:
SELECT * FROM information_schema.SCHEMATA S;
If you are inserting using PHP, and you have followed the various ALTER database and ALTER table options above, make sure your php connection's charset is utf8mb4.
Example of connection string:
$this->pdo = new PDO("mysql:host=$ip;port=$port;dbname=$db;charset=utf8mb4", etc etc
Notice the "charset" is utf8mb4, not just utf8!
Today I am facing the same question, but solutions in other answers don't work for me. Here is my solution.
First of all, changing charset in mysql/my.ini, database, and the table is necessary, as described in other answers.
Second, if you have created your tables before you want to saving emoji, you can use
SHOW FULL COLUMNS FROM `yourcolumn`;
To check whether the column you want to save emoji is set in utf8mb4. You can find that most of your columns are still in utf8 charset.
Use
ALTER TABLE `yourtable` CHANGE `yourcolumn` `yourcolumn` VARCHAR(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
I'm facing this issue when upgrading MySQL 5.0 to MySQL 8.0 AWS RDS, trying many things finally what works for me share with you folks.
Error:
Warning: PDOStatement::execute(): SQLSTATE[HY000]: General error:
3988 Conversion from collation utf8_unicode_ci into utf8mb4_general_ci
impossible for parameter in /var/www/html/pdo_con.php on line 87
Array (
[0] => HY000
[1] => 3988
[2] => Conversion from collation utf8_unicode_ci into utf8mb4_general_ci impossible for parameter )
Backend: PHP5/php7 + PDO is giving trouble.
Solution: only two thing needs to do
Add a code in line after your pdo connection
$conn->exec("set names utf8mb4");
where $conn is connection handler in PDO
Alter the table and set charset utf8mb4 and collate utf8mb4_unicode_ci.
ALTER TABLE mytable CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
This command will change your every column charset to utf8mb4 and collation too.
Put this right before your database call:
mysqli_set_charset($db, "utf8mb4");
This will allow you to input emojis directly into the database table that has been set to Collation: utfmb4_bin. Make sure to set your column to utfmb4 as well.
Hi my friends
This is how I solved this problem and I was happy to teach it to you as well
I am in the Android application
I encrypt a string containing text and emoj and send it to the server and save it in the mysql table and after receiving it from the server I decrypt it and display it in the textview.
encoded and decoded my message before request and after response:
I send Android app messages to mysql via pdo through this method and receive them with pdo. And I have no problem.
I think it was a good way. Please like
Thankful
public void main()
{
String message="hi mester ali moradi 🌦️🌦️ how are you ?";
String encoded_message=encodeStringUrl(message);
String decode_message=decodeStringUrl(encoded_message);
}
public static String encodeStringUrl(String message) {
String encodedUrl =null;
try {
encodedUrl = URLEncoder.encode(message, "UTF-8");
} catch (UnsupportedEncodingException e) {
return encodedUrl;
}
return encodedUrl;
}
public static String decodeStringUrl(String message) {
String decodedUrl =null;
try {
decodedUrl = URLDecoder.decode(message, "UTF-8");
} catch (UnsupportedEncodingException e) {
return decodedUrl;
}
return decodedUrl;
}
message : hi mester ali moradi 🌦️🌦️ how are you ?
encoded : ghgh%F0%9F%98%AE%F0%9F%A4%90%F0%9F%98%A5
decoded :hi mester ali moradi 🌦️🌦️ how are you ?
If you use command line interface for inserting sql file to database.
Be sure your table charset utf8mb4 and column collation utf8mb4_unicode_ci or utf8mb4_bin
mysql -u root -p123456 my_database < profiles.sql
ERROR 1366 (HY000) at line 1679: Incorrect string value: '\xF0\x9F\x98\x87\xF0\x9F...' for column 'note' at row 328
we can solve the problem with this parameter
--default-character-set=name (Set the default character set)
mysql -u root -p123456 --default-character-set=utf8mb4 my_database < profiles.sql
Actually i'm using mysql Ver 8.0.23
I had created the both Database and the Table, without Altering them :
mysql> CREATE DATABASE tp2;
Query OK, 1 row affected (0.30 sec)
mysql> INSERT INTO tweetsRep(username, content) VALUES ('ibrahim', '🀣 oh my god');
Then after select, i thing it just worked fine !
I don't know if it is requested to enter Emoji as a hexadecimal or other encoding string or just copy it as it is... just correct me if i'm wrong, thank you !
I tried different methods and approaches and found a way that worked for me.
The SQL for the update query:
ALTER DATABASE YOUR_DB_NAME_HERE CHARACTER SET = utf8mb4 COLLATE =
utf8mb4_unicode_ci;
You can see in the table, the emoji's are present
And if you go to this page: https://www.thecookingcat.com/recipes/thai-green-curry.php#comments
You can see the emojis in the comments.
I also have an RSS feed on the site and the emojis are included in the RSS feed XML code.
If anyone searching this in 2022 just follow these steps and no need to do any modification on Database
Name Space
using System.Web;
Your normal text like this :
String encode = "thank you 😊"
encode = HttpUtility.UrlEncode(encode);
It will store in Database like this : "thank+you+%f0%9f%98%8a"
And next fetch that data form your Database and do UrlDecode like this
DataSet ds = "Fetch your Encoded data form your Database";
String decode = HttpUtility.UrlDecode(ds.Tables[0].Rows[i]["YourColumnName"].ToString().Trim());
And your output is :-
decode = "thank you 😊".
It is working fine for me and saved time.

Incorrect string value: '\xF0\x9F\x8E\xB6\xF0\x9F...' MySQL

I am trying to store a tweet in my MYSQL table. The tweet is:
quiero que me escuches, no te burles no te rias, anoche tuve un sueño que te fuiste de mi vida 🎢🎢
The final two characters are both 'MULTIPLE MUSICAL NOTES' (U+1F3B6), for which the UTF-8 encoding is 0xf09f8eb6.
The tweet_text field in my table is encoded in utf8mb4. But when I try to store the tweet in that column I get the following error message:
Incorrect string value: '\xF0\x9F\x8E\xB6\xF0\x9F...' for column 'tweet_text' at row 1.
What is going wrong? How can I fix this? I need to store multiple languages as well and this character set works for all languages but not for the special characters like emoticons and emojis.
This is my create table statement:
CREATE TABLE `twitter_status_data` (
`unique_status_id` bigint(20) NOT NULL AUTO_INCREMENT,
`metadata_result_type` text CHARACTER SET utf8,
`created_at` text CHARACTER SET utf8 NOT NULL COMMENT 'UTC time when this Tweet was created.',
`id` bigint(20) unsigned NOT NULL COMMENT 'Unique tweet identifier',
`id_str` text CHARACTER SET utf8 NOT NULL,
`tweet_text` text COMMENT 'Actual UTF-8 text',
`user_id_str` text CHARACTER SET utf8,
`user_name` text COMMENT 'User''s name',
`user_screen_name` text COMMENT 'Twitter handle',
`coordinates` text CHARACTER SET utf8,
PRIMARY KEY (`unique_status_id`),
KEY `user_id_index` (`user_id`),
FULLTEXT KEY `tweet_text_index` (`tweet_text`)
) ENGINE=InnoDB AUTO_INCREMENT=82451 DEFAULT CHARSET=utf8mb4;
I was finally able to figure out the issue.
I had to change some settings in mysql configuration my.ini
This article helped a lot
http://mathiasbynens.be/notes/mysql-utf8mb4#character-sets
First i changed the character set in my.ini to utf8mb4
Next i ran the following commands in mysql client
SET NAMES utf8mb4;
ALTER DATABASE dreams_twitter CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci;
Use the following command to check that the changes are made
SHOW VARIABLES WHERE Variable_name LIKE 'character\_set\_%' OR Variable_name LIKE 'collation%';
I had hit the same problem and learnt the following-
Even though database has a default character set of utf-8, it's possible for database columns to have a different character set in MySQL.
Modified dB and the problematic column to UTF-8:
mysql> ALTER DATABASE MyDB CHARACTER SET 'utf8' COLLATE 'utf8_unicode_ci'
mysql> ALTER TABLE database.table MODIFY COLUMN column_name VARCHAR(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL;
Now creating new tables with:
> CREATE TABLE My_Table_Name (
twitter_id_str VARCHAR(255) NOT NULL UNIQUE,
twitter_screen_name VARCHAR(512) CHARACTER SET utf8 COLLATE utf8_unicode_ci,
.....
) CHARACTER SET utf8 COLLATE utf8_unicode_ci;
It may be obvious, but it still was surprising to me, that SET NAMES utf8 is not compatible with utf8mb4 encoding. So for some apps changing table/column encoding was not enough. I had to change encoding in app configuration.
Redmine (ruby, ROR)
In config/database.yml:
production:
adapter: mysql2
database: redmine
host: localhost
username: redmine
password: passowrd
encoding: utf8mb4
Custom Yii application (PHP)
In config/db.php:
return [
'class' => yii\db\Connection::class,
'dsn' => 'mysql:host=localhost;dbname=yii',
'username' => 'yii',
'password' => 'password',
'charset' => 'utf8mb4',
],
If you have utf8mb4 as a column/table encoding and still getting errors like this, make sure that you have configured correct charset for DB connection in your application.
Change database charset and collation
ALTER DATABASE
database_name
CHARACTER SET = utf8mb4
COLLATE = utf8mb4_unicode_ci;
change specific table's charset and collation
ALTER TABLE
table_name
CONVERT TO CHARACTER SET utf8mb4
COLLATE utf8mb4_unicode_ci;
change connection charset in mysql driver
before
charset=utf8&parseTime=True&loc=Local
after
charset=utf8mb4&collation=utf8mb4_unicode_ci&parseTime=True&loc=Local
From this article https://hackernoon.com/today-i-learned-storing-emoji-to-mysql-with-golang-204a093454b7
According to the create table statement, the default charset of the table is already utf8mb4. It seems that you have a wrong connection charset.
In Java, set the datasource url like this:
jdbc:mysql://127.0.0.1:3306/testdb?useUnicode=true&characterEncoding=utf-8`.
?useUnicode=true&characterEncoding=utf-8 is necessary for using utf8mb4.
It works for my application.
FOR SQLALCHEMY AND PYTHON
The encoding used for Unicode has traditionally been 'utf8'. However, for MySQL versions 5.5.3 on forward, a new MySQL-specific encoding 'utf8mb4' has been introduced, and as of MySQL 8.0 a warning is emitted by the server if plain utf8 is specified within any server-side directives, replaced with utf8mb3. The rationale for this new encoding is due to the fact that MySQL’s legacy utf-8 encoding only supports codepoints up to three bytes instead of four. Therefore, when communicating with a MySQL database that includes codepoints more than three bytes in size, this new charset is preferred, if supported by both the database as well as the client DBAPI, as in:
e = create_engine(
"mysql+pymysql://scott:tiger#localhost/test?charset=utf8mb4")
All modern DBAPIs should support the utf8mb4 charset.
enter link description here
I had use an emoji in my string that was the reason for this error.
So make sure you are not using some incorrect string that is not valid to save into the database.
As others said, it's because you are trying to save a 4 bytes of data into less space.
If you are facing the similar issue in java and don't have the flexibility to change the charset and collate encoding of database than this answer is for you.
you can use the Emoji Java library to achieve the same. You can convert into alias before saving/updating into database and convert back to unicode post save/update/load from database. The main benefit is readability of the text even after the encoding because this library only alias the emoji's rather than whole string.
I changed MySQL settings and still the same. Finally I used the function utf8_decode() on the string before insert.

Illegal mix of collations MySQL Error

I'm getting this strange error while processing a large number of data...
Error Number: 1267
Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '='
SELECT COUNT(*) as num from keywords WHERE campaignId='12' AND LCASE(keyword)='hello again Γ¦Λœβ€ Γ£β€ΉΓ£β€šβ€° Γ£β€šΓ£β€šβ€Ή Γ₯ ´æ‰€'
What can I do to resolve this? Can I escape the string somehow so this error wouldn't occur, or do I need to change my table encoding somehow, and if so, what should I change it to?
SET collation_connection = 'utf8_general_ci';
then for your databases
ALTER DATABASE your_database_name CHARACTER SET utf8 COLLATE utf8_general_ci;
ALTER TABLE your_table_name CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
MySQL sneaks swedish in there sometimes for no sensible reason.
CONVERT(column1 USING utf8)
Solves my problem. Where column1 is the column which gives me this error.
You should set both your table encoding and connection encoding to UTF-8:
ALTER TABLE keywords CHARACTER SET UTF8; -- run once
and
SET NAMES 'UTF8';
SET CHARACTER SET 'UTF8';
Use following statement for error
be careful about your data take backup if data have in table.
ALTER TABLE your_table_name CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
In general the best way is to Change the table collation. However I have an old application and are not really able to estimate the outcome whether this has side effects. Therefore I tried somehow to convert the string into some other format that solved the collation problem.
What I found working is to do the string compare by converting the strings into a hexadecimal representation of it's characters. On the database this is done with HEX(column). For PHP you may use this function:
public static function strToHex($string)
{
$hex = '';
for ($i=0; $i<strlen($string); $i++){
$ord = ord($string[$i]);
$hexCode = dechex($ord);
$hex .= substr('0'.$hexCode, -2);
}
return strToUpper($hex);
}
When doing the database query, your original UTF8 string must be converted first into an iso string (e.g. using utf8_decode() in PHP) before using it in the DB. Because of the collation type the database cannot have UTF8 characters inside so the comparism should work event though this changes the original string (converting UTF8 characters that are not existend in the ISO charset result in a ? or these are removed entirely). Just make sure that when you write data into the database, that you use the same UTF8 to ISO conversion.
I had my table originally created with CHARSET=latin1. After table conversion to utf8 some columns were not converted, however that was not really obvious.
You can try to run SHOW CREATE TABLE my_table; and see which column was not converted or just fix incorrect character set on problematic column with query below (change varchar length and CHARSET and COLLATE according to your needs):
ALTER TABLE `my_table` CHANGE `my_column` `my_column` VARCHAR(10) CHARSET utf8
COLLATE utf8_general_ci NULL;
I found that using cast() was the best solution for me:
cast(Format(amount, "Standard") AS CHAR CHARACTER SET utf8) AS Amount
There is also a convert() function. More details on it here
Another resource here
Change the character set of the table to utf8
ALTER TABLE your_table_name CONVERT TO CHARACTER SET utf8
My user account did not have the permissions to alter the database and table, as suggested in this solution.
If, like me, you don't care about the character collation (you are using the '=' operator), you can apply the reverse fix. Run this before your SELECT:
SET collation_connection = 'latin1_swedish_ci';
After making your corrections listed in the top answer, change the default settings of your server.
In your "/etc/my.cnf.d/server.cnf" or where ever it's located add the defaults to the [mysqld] section so it looks like this:
[mysqld]
character-set-server=utf8
collation-server=utf8_general_ci
Source: https://dev.mysql.com/doc/refman/5.7/en/charset-applications.html