Good Morning!
I've been trying to implement emojis in my applications for a while, but I'm having a lot of difficulties. I have already tested several internet solutions, but none has been effective.
I will try to detail as much as I am doing:
I'm using Classic ASP and MySQL in versions
5.6.40-84.0-log and 5.6.26-log.
The application is hosted on a Plesk Windows (I tried to run locally
and the same goes for the database). Notepad ++ pages have already
been tested in UTF-8 and UTF8 without BOM.
In the HTML <head> there is <meta charset = "utf-8">.
The <form> has the tag accept-charset="UTF-8".
The ASP has Response.AddHeader "Content-Type", "text/html;charset=UTF-8", Response.CodePage=65001,Response.LCID=1060 and Response.Charset="utf-8".
Now for the problems:
In both versions of MySQL, when I change COLLATE to utf8mb4_unicode_* or utf8mb4_bin, it returns to utf8mb4_0900_ai_ci automatically.
In version 5.6.40-84.0-log, if I register the emojis directly in the database, they are like ??????????????.
In version 5.6.26-log, if I register directly with the database, the error returns:
Executing:
INSERT INTO `db`.`table` (` emoji`) VALUES ('tion 123 😀😉😙😐🤐 ão');
Operation failed: There was an error while applying the SQL script to the database.
ERROR 1366: 1366: Incorrect string value: '\ xF0 \ x9F \ x98 \ x80 \ xF0 \ x9F ...' for column 'emoji' at row 1
SQL Statement:
INSERT INTO 'db'. 'Table' ('emoji') VALUES ('tion 123 😀😉😙😐🤐 ão')
In this same version, registering through the of the page, using SET NAMES 'utf8mb4' or SET NAMES' utf8' in the ODBC connection string, the following error is returned:
Microsoft OLE DB Provider for ODBC Drivers error '80004005'
[MySQL] [ODBC 3.51 Driver] [mysqld-5.6.22] Incorrect string value: '\ xE7 \ xE3o' for column 'field' at row 1
EDIT 1:
When I consult SHOW SESSION VARIABLES LIKE 'character_set%'; and SHOW SESSION VARIABLES LIKE "%collation%"; the database it returns the following results:
Thanks!
Translation by Google Translate (rsrs)
For Emoji and some Chinese, you need utf8mb4, not utf8 in MySQL.
Since you are running the old 5.6, you may stumble over another problem. http://mysql.rjweb.org/doc.php/limits#767_limit_in_innodb_indexes
If that E7E3 should have been çã, then you have some latin1-encoded text, too.
Do not mix encodings in a single column.
Your output for SHOW SESSION VARIABLES LIKE 'char%' shows that the connection is just utf8; it needs to be utf8mb4. The main difference between them is in Emoji.
Related
I have a chat app that is giving an error message when a user posts 4 byte emojis. I verified that the database, table and column are all set to utf8mb4. I can tell that the character_set_connection is set to utf8 by default, so I want to override that. We are using MyXQL and I thought that I could add this to my config:
+config :myxql,
+ charset: "utf8mb4",
+ collation: "utf8mb4_unicode_ci"
It didn’t help. Are there any suggestions?
The actual error message is:
** (exit) an exception was raised:
** (MyXQL.Error) (1366) Incorrect string value: '\xF0\x9F\x8F\x86\x0A\x0A...' for column `my_db`.`messages`.`body` at row 1
(ecto_sql 3.6.2) lib/ecto/adapters/myxql.ex:270: Ecto.Adapters.MyXQL.insert/6
(ecto 3.7.1) lib/ecto/repo/schema.ex:744: Ecto.Repo.Schema.apply/4
(ecto 3.7.1) lib/ecto/repo/schema.ex:367: anonymous fn/15 in Ecto.Repo.Schema.do_insert/4
(ecto 3.7.1) lib/ecto/multi.ex:716: Ecto.Multi.apply_operation/5
(elixir 1.13.3) lib/enum.ex:2396: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto 3.7.1) lib/ecto/multi.ex:690: anonymous fn/5 in Ecto.Multi.apply_operations/5
(ecto_sql 3.6.2) lib/ecto/adapters/sql.ex:1017: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
(db_connection 2.4.0) lib/db_connection.ex:1512: DBConnection.run_transaction/4
What do I need to do to get Ecto/MyXQL to connect to the database with utf8mb4? Or is this error message indicating something else?
Thanks.
UPDATE:
I added charset: "utf8mb4", collation: "utf8mb4_unicode_ci", to the configuration for my repository, and judging by the errors from myxql when I set the collation just to “utf8mb4”, I’m in the right place. So it appears that the connection should now SET NAMES "utf8mb4", collation "utf8mb4_unicode_ci" which seems like the ideal operation per the mysql docs, but I get the same error. What other setting could I need? And how do I change it?
I have a problem with MySql.
My version of MYSql is : 5.7.33 - MySQL Community Server (GPL)
I have create a discord Bot in node.js, and i have a mistake when a new user with pseudo like this : legoshi🌌🌧
So i have try to follow this topic : How to fix "Incorrect string value" errors?
So i convert my Database in : utf8mb4_unicode_ci
And my error is still here.
At the begin my database was in utf8 and i have the error too.
code: 'ER_TRUNCATED_WRONG_VALUE_FOR_FIELD',
errno: 1366,
sqlMessage: "Incorrect string value: '\\xF0\\x9F\\x8C\\x8C\\xF0\\x9F...' for column 'user' at row 1",
sqlState: 'HY000',
index: 0,
sql: 'INSERT INTO registre (id, user, autohit, ultimate, platinium, `Date Inscription`) VALUES (210490816542670849, "legoshi🌌🌧", 0, 0, 0, CURRENT_TIMESTAMP())'
}
So i don't no how to change this. I have see a lot of topic and all seems to be fix with utf8mb4_unicode_ci but not in my case.
Thanks for you're help.
In MySQL, there are several places where you can set up a character set:
On the server level
On the database level
On the table level (for each table)
On the field level for all character-based fields
On your connection (telling the server what charset will be used in packets you send to the server)
Basically, server-level, database-level and table-level are just defaults for newly created items: New databases are generated with the server's default. New tables are created with the database's default, new fields are created with the table's default. However, only the field-level charset is what actually counts.
So first, you should make sure that the fields you want to store the data in actually are set up to utf8mb4_unicode_ci. Then, you need to connect to the server using exactly the same charset. Be aware that also the collation should match.
You can find out what character set is in use by issuing the following query:
SHOW VARIABLES LIKE 'character_set_%'
You'll see several variables indicating which default is set for various scopes. Have a look especially to the variables character_set_client and character_set_connection. If the connection does not have the correct character set specified, you need to set it up on connection.
It's a good practice to have all character sets match identically. Mixed values will sooner or later cause trouble.
To check the character set which is set up for the field, have it displayed with the command
SHOW CREATE TABLE registre
I am using jupyter & python 3.6.4 via anaconda.
I want to be able to process and store data from python to a MySQL database.
The libraries I am using to do this arepymysql and sqlalchemy.
For now, I am testing this localy with wamp (mysql version : 5.7.21), later I will apply it to a distant server.
Database creation function:
def create_raw_mysql_db(host,user,password,db_name):
conn=pymysql.connect(host=host,user=user,password=password)
conn.cursor().execute('DROP DATABASE '+db_name)
conn.cursor().execute('CREATE DATABASE '+db_name+' CHARACTER SET utf8mb4')
Function to convert a Dataframe to a relational table in MySql:
def save_raw_to_mysql_db(df,table_name,db_name,if_exists,username,password,host_ip,port):
engine = create_engine("mysql+pymysql://"+username+":#"+host_ip+":"+port+"/"+db_name+"?charset=utf8mb4")
df.to_sql(name=table_name,con=engine,if_exists=if_exists,chunksize=10000)
The execution code:
#DB info & credentials
host = "localhost"
port = "3306"
user= "root"
password= ""
db_name= "raw_data"
exade_light_tb = "exade_light"
#A simple dataframe
df = pd.DataFrame(np.random.randint(low=0, high=10, size=(5, 5)),columns=['a', 'b', 'c', 'd', 'e'])
create_raw_mysql_db(host,user,password,db_name)
save_raw_to_mysql_db(df,exade_light_tb,db_name,"replace",user,password,host,port)
The warning I receive when I run this code:
C:\Users.... : Warning: (1366, "Incorrect string value: '\x92\xE9t\xE9)' for column 'VARIABLE_VALUE' at row 481")
result = self._query(query)
From these threads: /questions/34165523/ questions/47419943 questions/2108824/, I could conclude the problem must be related to the utf8 charset, but I am using utf8mb4 to create my db and I am not using Django (which supposedly also needed to be configured according to questions/2108824/).
My questions :
How is this warning really impacting my data and its integrity?
How come even though I change charset from utf8 to utf8mb4, it
doesn't seem to solve the warning? Do I need to configure something
further? In this case, what are the parameters I should keep in mind
to apply the same configuration to my distant server?
How do I get rid of this warning?
Annex:
I don't think title could be more described better as tl;dr, because problem is a bit deeper.
I've got two databases (finnish language):
MSSQL (collation: SQL_Latin1_General_CP437_CI_AI)
MySQL (collation: utf_general_ci)
I've created BI project in vs2017, connected two databases and transfered tables from one to another, no problem. Except for 1 letter: "å" - instead it was "?". I cannot change any database collation so I am trying to find a way to transfer words with this letter.
What I've tried:
OLD DB Source -> ODBC Destination
Point "1" with "Data Conversion" block in between (with code page 1252)
Script Component, in which I have tried:
Insert with "_latin"
sql= "INSERT INTO db.words(Name) VALUES(_latin1'å')";
byte[] b = Encoding.UTF8.GetBytes(sql);
odbcCmd = new OdbcCommand(Encoding.UTF8.GetString(b), odbcConn);
odbcCmd.ExecuteNonQuery();
Insert without it
sql= "INSERT INTO db.words(Name) VALUES('å')";
byte[] b = Encoding.UTF8.GetBytes(sql);
odbcCmd = new OdbcCommand(Encoding.UTF8.GetString(b), odbcConn);
odbcCmd.ExecuteNonQuery();
Diferent ways of encoding
byte[] bytes = Encoding.GetEncoding(1252).GetBytes("å");
var myString = Encoding.GetEncoding(1252).GetString(bytes);
byte[] bytes2 = Encoding.Default.GetBytes("å");
var myString2 = Encoding.Default.GetString(bytes2);
Insert with COLLATE which got me error
insert into db.words(Name) values ("å" COLLATE latin1_swedish_ci) ;
and error:
System.Data.Odbc.OdbcException: „ERROR [HY000] [MySQL][ODBC 5.3(a) Driver][mysqld-5.7.21-log]COLLATION 'latin1_swedish_ci' is not valid for CHARACTER SET 'cp1250'”
Here is interesting part:
I can make insert with this letter in MySQL Workbench without a problem, and it will be inserted, but when I try to pass it from one database to another it is lost. I've set Data Viewers between Data Conversion and the letter was still there, and also when debugging script it was after encoding in string that were inserted to database.
Maybe someone got any idea what else I can try, because I feel like I have tried everything, and feel that the resolve of this problem is really close, but I just don't see it.
CP1250 does not include å; CP437 and utf8 do include it.
COLLATE is irrelevant -- it applies only to comparing and sorting.
Don't use any encode/conversion functions; instead, specify how the data is encoded.
I see 'code' -- but what is the encoding for the source in that language and/or editor?
Show us the hex of any strings in question.
Which direction are you trying to transfer?
What are the connection parameters for each database?
I am using sql server database drivers in codeigniter and I am executing the following query:
select A.inst_name,Substring((Select ',' + cast(B.program_id as varchar(15))
From k12_dms_inst_programs B
Where B.inst_id=A.id For XML Path('')),2,8000) As EmployeeList
From k12_dms_institution_master A
Group by A.inst_name,A.id
which is working absolutely fine in SSMS.
But when I am trying to execute the same query using Codeigniter I am getting the following error:--
Unicode data in a Unicode-only collation or ntext data cannot be sent to clients using DB-Library (such as ISQL) or ODBC version 3.7 or earlier.
Found some solutions on php.net
MSSQLNewbie 19-Sep-2011 06:34
In /etc/freetds/freetds.conf add these two lines (last two):
[global]
;tds version = 4.2
tds version = 8.0
client charset = UTF-8
You can edit "charset" in php.ini too (but you don't need if you did it previously in freetds.conf):
; Specify client character set..
; If empty or not set the client charset from freetds.comf is used
; This is only used when compiled with FreeTDS mssql.charset = "UTF-8"
Use nchar/nvarchar/ntext column types if you need unicode support.
dann dot farquhar at iteams dot org 24-Sep-2009 11:45
I found that changing the version in /etc/freetds.conf from 4.2 to 8.0
fixes this problem without having to make any changes to the SELECT
statement
huberkev11 at hotmail dot com 12-May-2006 01:47
This is because you are using column types of like ntext instead of
text. There are 2 solutions.
1 Change all ntext column types to text or
2 Your query must look like: SELECT CAST(field1 AS TEXT) AS field1 FROM table
Hope they will help.