Incorrect string value - MySql - mysql

I have a problem with MySql.
My version of MYSql is : 5.7.33 - MySQL Community Server (GPL)
I have create a discord Bot in node.js, and i have a mistake when a new user with pseudo like this : legoshi🌌🌧
So i have try to follow this topic : How to fix "Incorrect string value" errors?
So i convert my Database in : utf8mb4_unicode_ci
And my error is still here.
At the begin my database was in utf8 and i have the error too.
code: 'ER_TRUNCATED_WRONG_VALUE_FOR_FIELD',
errno: 1366,
sqlMessage: "Incorrect string value: '\\xF0\\x9F\\x8C\\x8C\\xF0\\x9F...' for column 'user' at row 1",
sqlState: 'HY000',
index: 0,
sql: 'INSERT INTO registre (id, user, autohit, ultimate, platinium, `Date Inscription`) VALUES (210490816542670849, "legoshi🌌🌧", 0, 0, 0, CURRENT_TIMESTAMP())'
}
So i don't no how to change this. I have see a lot of topic and all seems to be fix with utf8mb4_unicode_ci but not in my case.
Thanks for you're help.

In MySQL, there are several places where you can set up a character set:
On the server level
On the database level
On the table level (for each table)
On the field level for all character-based fields
On your connection (telling the server what charset will be used in packets you send to the server)
Basically, server-level, database-level and table-level are just defaults for newly created items: New databases are generated with the server's default. New tables are created with the database's default, new fields are created with the table's default. However, only the field-level charset is what actually counts.
So first, you should make sure that the fields you want to store the data in actually are set up to utf8mb4_unicode_ci. Then, you need to connect to the server using exactly the same charset. Be aware that also the collation should match.
You can find out what character set is in use by issuing the following query:
SHOW VARIABLES LIKE 'character_set_%'
You'll see several variables indicating which default is set for various scopes. Have a look especially to the variables character_set_client and character_set_connection. If the connection does not have the correct character set specified, you need to set it up on connection.
It's a good practice to have all character sets match identically. Mixed values will sooner or later cause trouble.
To check the character set which is set up for the field, have it displayed with the command
SHOW CREATE TABLE registre

Related

Ecto MySql database and table set for utf8mb4, now how do I connect with utf8mb4?

I have a chat app that is giving an error message when a user posts 4 byte emojis. I verified that the database, table and column are all set to utf8mb4. I can tell that the character_set_connection is set to utf8 by default, so I want to override that. We are using MyXQL and I thought that I could add this to my config:
+config :myxql,
+ charset: "utf8mb4",
+ collation: "utf8mb4_unicode_ci"
It didn’t help. Are there any suggestions?
The actual error message is:
** (exit) an exception was raised:
** (MyXQL.Error) (1366) Incorrect string value: '\xF0\x9F\x8F\x86\x0A\x0A...' for column `my_db`.`messages`.`body` at row 1
(ecto_sql 3.6.2) lib/ecto/adapters/myxql.ex:270: Ecto.Adapters.MyXQL.insert/6
(ecto 3.7.1) lib/ecto/repo/schema.ex:744: Ecto.Repo.Schema.apply/4
(ecto 3.7.1) lib/ecto/repo/schema.ex:367: anonymous fn/15 in Ecto.Repo.Schema.do_insert/4
(ecto 3.7.1) lib/ecto/multi.ex:716: Ecto.Multi.apply_operation/5
(elixir 1.13.3) lib/enum.ex:2396: Enum."-reduce/3-lists^foldl/2-0-"/3
(ecto 3.7.1) lib/ecto/multi.ex:690: anonymous fn/5 in Ecto.Multi.apply_operations/5
(ecto_sql 3.6.2) lib/ecto/adapters/sql.ex:1017: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
(db_connection 2.4.0) lib/db_connection.ex:1512: DBConnection.run_transaction/4
What do I need to do to get Ecto/MyXQL to connect to the database with utf8mb4? Or is this error message indicating something else?
Thanks.
UPDATE:
I added charset: "utf8mb4", collation: "utf8mb4_unicode_ci", to the configuration for my repository, and judging by the errors from myxql when I set the collation just to “utf8mb4”, I’m in the right place. So it appears that the connection should now SET NAMES "utf8mb4", collation "utf8mb4_unicode_ci" which seems like the ideal operation per the mysql docs, but I get the same error. What other setting could I need? And how do I change it?

Problems when registering emojis in MySQL with ASP Classic

Good Morning!
I've been trying to implement emojis in my applications for a while, but I'm having a lot of difficulties. I have already tested several internet solutions, but none has been effective.
I will try to detail as much as I am doing:
I'm using Classic ASP and MySQL in versions
5.6.40-84.0-log and 5.6.26-log.
The application is hosted on a Plesk Windows (I tried to run locally
and the same goes for the database). Notepad ++ pages have already
been tested in UTF-8 and UTF8 without BOM.
In the HTML <head> there is <meta charset = "utf-8">.
The <form> has the tag accept-charset="UTF-8".
The ASP has Response.AddHeader "Content-Type", "text/html;charset=UTF-8", Response.CodePage=65001,Response.LCID=1060 and Response.Charset="utf-8".
Now for the problems:
In both versions of MySQL, when I change COLLATE to utf8mb4_unicode_* or utf8mb4_bin, it returns to utf8mb4_0900_ai_ci automatically.
In version 5.6.40-84.0-log, if I register the emojis directly in the database, they are like ??????????????.
In version 5.6.26-log, if I register directly with the database, the error returns:
Executing:
INSERT INTO `db`.`table` (` emoji`) VALUES ('tion 123 😀😉😙😐🤐 ão');
Operation failed: There was an error while applying the SQL script to the database.
ERROR 1366: 1366: Incorrect string value: '\ xF0 \ x9F \ x98 \ x80 \ xF0 \ x9F ...' for column 'emoji' at row 1
SQL Statement:
INSERT INTO 'db'. 'Table' ('emoji') VALUES ('tion 123 😀😉😙😐🤐 ão')
In this same version, registering through the of the page, using SET NAMES 'utf8mb4' or SET NAMES' utf8' in the ODBC connection string, the following error is returned:
Microsoft OLE DB Provider for ODBC Drivers error '80004005'
[MySQL] [ODBC 3.51 Driver] [mysqld-5.6.22] Incorrect string value: '\ xE7 \ xE3o' for column 'field' at row 1
EDIT 1:
When I consult SHOW SESSION VARIABLES LIKE 'character_set%'; and SHOW SESSION VARIABLES LIKE "%collation%"; the database it returns the following results:
Thanks!
Translation by Google Translate (rsrs)
For Emoji and some Chinese, you need utf8mb4, not utf8 in MySQL.
Since you are running the old 5.6, you may stumble over another problem. http://mysql.rjweb.org/doc.php/limits#767_limit_in_innodb_indexes
If that E7E3 should have been çã, then you have some latin1-encoded text, too.
Do not mix encodings in a single column.
Your output for SHOW SESSION VARIABLES LIKE 'char%' shows that the connection is just utf8; it needs to be utf8mb4. The main difference between them is in Emoji.

Compromised saveguard of data due to bad encoding usage?

I am using jupyter & python 3.6.4 via anaconda.
I want to be able to process and store data from python to a MySQL database.
The libraries I am using to do this arepymysql and sqlalchemy.
For now, I am testing this localy with wamp (mysql version : 5.7.21), later I will apply it to a distant server.
Database creation function:
def create_raw_mysql_db(host,user,password,db_name):
conn=pymysql.connect(host=host,user=user,password=password)
conn.cursor().execute('DROP DATABASE '+db_name)
conn.cursor().execute('CREATE DATABASE '+db_name+' CHARACTER SET utf8mb4')
Function to convert a Dataframe to a relational table in MySql:
def save_raw_to_mysql_db(df,table_name,db_name,if_exists,username,password,host_ip,port):
engine = create_engine("mysql+pymysql://"+username+":#"+host_ip+":"+port+"/"+db_name+"?charset=utf8mb4")
df.to_sql(name=table_name,con=engine,if_exists=if_exists,chunksize=10000)
The execution code:
#DB info & credentials
host = "localhost"
port = "3306"
user= "root"
password= ""
db_name= "raw_data"
exade_light_tb = "exade_light"
#A simple dataframe
df = pd.DataFrame(np.random.randint(low=0, high=10, size=(5, 5)),columns=['a', 'b', 'c', 'd', 'e'])
create_raw_mysql_db(host,user,password,db_name)
save_raw_to_mysql_db(df,exade_light_tb,db_name,"replace",user,password,host,port)
The warning I receive when I run this code:
C:\Users.... : Warning: (1366, "Incorrect string value: '\x92\xE9t\xE9)' for column 'VARIABLE_VALUE' at row 481")
result = self._query(query)
From these threads: /questions/34165523/ questions/47419943 questions/2108824/, I could conclude the problem must be related to the utf8 charset, but I am using utf8mb4 to create my db and I am not using Django (which supposedly also needed to be configured according to questions/2108824/).
My questions :
How is this warning really impacting my data and its integrity?
How come even though I change charset from utf8 to utf8mb4, it
doesn't seem to solve the warning? Do I need to configure something
further? In this case, what are the parameters I should keep in mind
to apply the same configuration to my distant server?
How do I get rid of this warning?
Annex:

SISS MSSQL to MySQL with different collation is not copying finnish letter å

I don't think title could be more described better as tl;dr, because problem is a bit deeper.
I've got two databases (finnish language):
MSSQL (collation: SQL_Latin1_General_CP437_CI_AI)
MySQL (collation: utf_general_ci)
I've created BI project in vs2017, connected two databases and transfered tables from one to another, no problem. Except for 1 letter: "å" - instead it was "?". I cannot change any database collation so I am trying to find a way to transfer words with this letter.
What I've tried:
OLD DB Source -> ODBC Destination
Point "1" with "Data Conversion" block in between (with code page 1252)
Script Component, in which I have tried:
Insert with "_latin"
sql= "INSERT INTO db.words(Name) VALUES(_latin1'å')";
byte[] b = Encoding.UTF8.GetBytes(sql);
odbcCmd = new OdbcCommand(Encoding.UTF8.GetString(b), odbcConn);
odbcCmd.ExecuteNonQuery();
Insert without it
sql= "INSERT INTO db.words(Name) VALUES('å')";
byte[] b = Encoding.UTF8.GetBytes(sql);
odbcCmd = new OdbcCommand(Encoding.UTF8.GetString(b), odbcConn);
odbcCmd.ExecuteNonQuery();
Diferent ways of encoding
byte[] bytes = Encoding.GetEncoding(1252).GetBytes("å");
var myString = Encoding.GetEncoding(1252).GetString(bytes);
byte[] bytes2 = Encoding.Default.GetBytes("å");
var myString2 = Encoding.Default.GetString(bytes2);
Insert with COLLATE which got me error
insert into db.words(Name) values ("å" COLLATE latin1_swedish_ci) ;
and error:
System.Data.Odbc.OdbcException: „ERROR [HY000] [MySQL][ODBC 5.3(a) Driver][mysqld-5.7.21-log]COLLATION 'latin1_swedish_ci' is not valid for CHARACTER SET 'cp1250'”
Here is interesting part:
I can make insert with this letter in MySQL Workbench without a problem, and it will be inserted, but when I try to pass it from one database to another it is lost. I've set Data Viewers between Data Conversion and the letter was still there, and also when debugging script it was after encoding in string that were inserted to database.
Maybe someone got any idea what else I can try, because I feel like I have tried everything, and feel that the resolve of this problem is really close, but I just don't see it.
CP1250 does not include å; CP437 and utf8 do include it.
COLLATE is irrelevant -- it applies only to comparing and sorting.
Don't use any encode/conversion functions; instead, specify how the data is encoded.
I see 'code' -- but what is the encoding for the source in that language and/or editor?
Show us the hex of any strings in question.
Which direction are you trying to transfer?
What are the connection parameters for each database?

Unknown character set index for field received from server

I have an instance of MySQL 5.0.4.1 with an application written in Hibernate. On one of the pages, I get the following error message in the server log:
Unknown character set index for field '123' received from server.at com.mysql.jdbc.Connection.getCharsetNameForIndex(Connection.java:1664)at com.mysql.jdbc.Field.(Field.java:144)at com.mysql.jdbc.MysqlIO.unpackField(MysqlIO.java:506)at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:280)at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1319)at com.mysql.jdbc.MysqlIO.sqlQuery(MysqlIO.java:1218)at com.mysql.jdbc.Connection.execSQL(Connection.java:2233)at com.mysql.jdbc.Connection.execSQL(Connection.java:2193)at com.mysql.jdbc.Connection.execSQL(Connection.java:2174)at com.mysql.jdbc.Connection.setAutoCommit(Connection.java:536)at org.apache.commons.dbcp.DelegatingConnection.setAutoCommit(DelegatingConnection.java:268)at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.setAutoCommit(PoolingDataSource.java:293)at org.hibernate.transaction.JDBCTransaction.toggleAutoCommit(JDBCTransaction.java:194)at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:186)at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:162)at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:603)at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:579)at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:556)at org.springframework.transaction.interceptor.TransactionAspectSupport.doCloseTransactionAfterThrowing(TransactionAspectSupport.java:284)at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:100)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)at org.acegisecurity.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:66)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)at org.acegisecurity.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:66)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:176)at $Proxy6.getDataFromDatabase(Unknown Source)at org.myCompany.myAction.load(Unknown Source)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)at java.lang.reflect.Method.invoke(Method.java:324)at org.apache.struts.actions.DispatchAction.dispatchMethod(DispatchAction.java:274)at org.apache.struts.actions.DispatchAction.execute(DispatchAction.java:194)at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)at javax.servlet.http.HttpServlet.service(HttpServlet.java:787)at javax.servlet.http.HttpServlet.service(HttpServlet.java:908)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:264)at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:110)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:217)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:229)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:148)at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:213)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:280)at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:509)at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:509)at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:209)at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:509)at com.iplanet.ias.web.connector.nsapi.NSAPIProcessor.process(NSAPIProcessor.java:157)at com.iplanet.ias.web.WebContainer.service(WebContainer.java:579)
This error occurs on a production environment so I don't know which query produced the problem, but I do know that none of my tables has a column named '123'. Do you have any suggestions about what might be causing this problem?
Edit: As a followup, I found that this behavior is a known bug in MySQL 5.0.4, but I haven't found a good wokaround since I don't have the option of upgrading mySQL.
Could be that somehow the character set for the field has been set to an invalid value, run this SQL on the server with the problem replacing the table name and the column 'Collation' will show the characterset for each varchar/char fields
SHOW FULL COLUMNS IN table_name;
You can then change the character set of a field by using the following:
ALTER TABLE t MODIFY col1 VARCHAR(50) CHARACTER SET latin1;
Internally, MySQL is mapping out the indexes across the columns. Normally this is done using integers that represent the column index. Most indexes are actually a compound index once built (col 1 + col 3) which form something like field 13.
Likely this happened when migrating data from dev into production when the stack is not an exact replica.
As 3urdoch mentioned you can pull the charset using internal MySQL function; and then change the table's charset to a compatible encoding.
Alternatively (where I'm willing to bet this issue came from) you can check if the loaded driver for the Production matches the loaded driver for the Other server. This will prevent re-occurring issue if the Development/Other server is still being used for testing, and migrated into Production.