I have written hibernate query to save data in a mysql data table. UTF-8 is used inside code. The letter Δ is inserted into table as ?. I read several threads regarding this issue and tried following solutions.
Adding following line to the end of the jdbc url
?useEncoding=true&characterEncoding=UTF-8"
Adding following lines to the hibernate-config.xml file
**
"<prop key="hibernate.connection.useUnicode">true</prop>
<prop key="hibernate.connection.characterEncoding" >utf8</prop>
<prop key="hibernate.connection.charSet">utf8</pop>"
**
And also I have set the charset to utf8.
None of above didn't work. Are there any solution to resolve this issue?
Related
Good Morning!
I've been trying to implement emojis in my applications for a while, but I'm having a lot of difficulties. I have already tested several internet solutions, but none has been effective.
I will try to detail as much as I am doing:
I'm using Classic ASP and MySQL in versions
5.6.40-84.0-log and 5.6.26-log.
The application is hosted on a Plesk Windows (I tried to run locally
and the same goes for the database). Notepad ++ pages have already
been tested in UTF-8 and UTF8 without BOM.
In the HTML <head> there is <meta charset = "utf-8">.
The <form> has the tag accept-charset="UTF-8".
The ASP has Response.AddHeader "Content-Type", "text/html;charset=UTF-8", Response.CodePage=65001,Response.LCID=1060 and Response.Charset="utf-8".
Now for the problems:
In both versions of MySQL, when I change COLLATE to utf8mb4_unicode_* or utf8mb4_bin, it returns to utf8mb4_0900_ai_ci automatically.
In version 5.6.40-84.0-log, if I register the emojis directly in the database, they are like ??????????????.
In version 5.6.26-log, if I register directly with the database, the error returns:
Executing:
INSERT INTO `db`.`table` (` emoji`) VALUES ('tion 123 πππππ€ Γ£o');
Operation failed: There was an error while applying the SQL script to the database.
ERROR 1366: 1366: Incorrect string value: '\ xF0 \ x9F \ x98 \ x80 \ xF0 \ x9F ...' for column 'emoji' at row 1
SQL Statement:
INSERT INTO 'db'. 'Table' ('emoji') VALUES ('tion 123 πππππ€ Γ£o')
In this same version, registering through the of the page, using SET NAMES 'utf8mb4' or SET NAMES' utf8' in the ODBC connection string, the following error is returned:
Microsoft OLE DB Provider for ODBC Drivers error '80004005'
[MySQL] [ODBC 3.51 Driver] [mysqld-5.6.22] Incorrect string value: '\ xE7 \ xE3o' for column 'field' at row 1
EDIT 1:
When I consult SHOW SESSION VARIABLES LIKE 'character_set%'; and SHOW SESSION VARIABLES LIKE "%collation%"; the database it returns the following results:
Thanks!
Translation by Google Translate (rsrs)
For Emoji and some Chinese, you need utf8mb4, not utf8 in MySQL.
Since you are running the old 5.6, you may stumble over another problem. http://mysql.rjweb.org/doc.php/limits#767_limit_in_innodb_indexes
If that E7E3 should have been çã, then you have some latin1-encoded text, too.
Do not mix encodings in a single column.
Your output for SHOW SESSION VARIABLES LIKE 'char%' shows that the connection is just utf8; it needs to be utf8mb4. The main difference between them is in Emoji.
I'm using Flyway test extensions with H2 database and MySQL dialect.
Unfortunately, #FlywayTest annotation executes database clean with H2 dialect and ends with error:
org.flywaydb.core.internal.dbsupport.FlywaySqlException:
Unable to drop "PUBLIC"."site"
------------------------------
SQL State : 42S02
Error Code : 42102
Message : Table "site" not found; SQL statement:
DROP TABLE "PUBLIC"."site" CASCADE [42102-193]
at org.flywaydb.core.internal.dbsupport.SchemaObject.drop(SchemaObject.java:82)
at org.flywaydb.core.internal.dbsupport.h2.H2Schema.doClean(H2Schema.java:69)
at org.flywaydb.core.internal.dbsupport.Schema.clean(Schema.java:148)
at org.flywaydb.core.internal.command.DbClean$4.call(DbClean.java:184)
at org.flywaydb.core.internal.command.DbClean$4.call(DbClean.java:181)
at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.exec
However, when I manually run DROP TABLE PUBLIC.site CASCADE (no quotes) from console it ends successfully.
How to force dialect in Flyway test extensions?
This is no problem of the annotation #FlywayTest.
The annotation #FlywayTest call flyway.clean() method.
And Flyway supports for H2 only quoted names for schema and tables.
See class source code on Github https://github.com/flyway/flyway/blob/master/flyway-core/src/main/java/org/flywaydb/core/internal/dbsupport/h2/
I think there will be no plan to support H2 feature modes, such like Mysql, Oracle, ....
You can open a issue here https://github.com/flyway/flyway/issues
I've made an application in Java EE that uses Hibernate to communicate with MySQL. It works perfectly on my Windows development machine, but I have problem on debian, where the application is deployed.
When I search for keyword with Polish letters(like Ε, Δ
, Δ, Γ³ etc,) the result is ok on Windows, but on server, where I have imported the database it does not work.
Hibernate query looks like this:
#NamedQuery(name = "Keyword.findByKeyword", query = "SELECT k FROM Keyword k WHERE k.keyword = :keyword")
and is called like this:
myEntityManager.createNamedQuery("Keyword.findByKeyword").setParameter("keyword", keyword).getSingleResult();
When I use mysql on debian via SSH and type in SELECT query manually:
SELECT * FROM keywords WHERE keyword = 'ser ΕΌΓ³Εty';
it also works and return single result. Encoding and collations of tables and columns are also ok. In datasource configuration I've added
?UseUnicode=true&characterEncoding=utf8
parameters, but it also did not help. I thought that maybe there is a problem with encoding in data from request send by form, but the problem appears even if the parameter i.e. "ser ΕΌΓ³Εty" is hardcoded in my repository class.
I also use Hibernate Search for indexing and the FullTextEntityManager return correct results with Polish letters.
I assume that there is some problem between Hibernate and MySQL, but I have no more ideas what could I change. Any suggestions?
Server Wildfly9.0.1, MySQL 5.6
Ok the problem was in encoding on the mysql server level. It was set to latin1 by default. To fix this follow this question Change MySQL default character set to UTF-8 in my.cnf? and edit your my.cnf file.
I have an instance of MySQL 5.0.4.1 with an application written in Hibernate. On one of the pages, I get the following error message in the server log:
Unknown character set index for field '123' received from server.at com.mysql.jdbc.Connection.getCharsetNameForIndex(Connection.java:1664)at com.mysql.jdbc.Field.(Field.java:144)at com.mysql.jdbc.MysqlIO.unpackField(MysqlIO.java:506)at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:280)at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1319)at com.mysql.jdbc.MysqlIO.sqlQuery(MysqlIO.java:1218)at com.mysql.jdbc.Connection.execSQL(Connection.java:2233)at com.mysql.jdbc.Connection.execSQL(Connection.java:2193)at com.mysql.jdbc.Connection.execSQL(Connection.java:2174)at com.mysql.jdbc.Connection.setAutoCommit(Connection.java:536)at org.apache.commons.dbcp.DelegatingConnection.setAutoCommit(DelegatingConnection.java:268)at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.setAutoCommit(PoolingDataSource.java:293)at org.hibernate.transaction.JDBCTransaction.toggleAutoCommit(JDBCTransaction.java:194)at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:186)at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:162)at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:603)at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:579)at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:556)at org.springframework.transaction.interceptor.TransactionAspectSupport.doCloseTransactionAfterThrowing(TransactionAspectSupport.java:284)at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:100)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)at org.acegisecurity.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:66)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)at org.acegisecurity.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:66)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:176)at $Proxy6.getDataFromDatabase(Unknown Source)at org.myCompany.myAction.load(Unknown Source)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)at java.lang.reflect.Method.invoke(Method.java:324)at org.apache.struts.actions.DispatchAction.dispatchMethod(DispatchAction.java:274)at org.apache.struts.actions.DispatchAction.execute(DispatchAction.java:194)at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)at javax.servlet.http.HttpServlet.service(HttpServlet.java:787)at javax.servlet.http.HttpServlet.service(HttpServlet.java:908)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:264)at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:110)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:217)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:229)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:148)at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:213)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:280)at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:509)at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:509)at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:209)at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:509)at com.iplanet.ias.web.connector.nsapi.NSAPIProcessor.process(NSAPIProcessor.java:157)at com.iplanet.ias.web.WebContainer.service(WebContainer.java:579)
This error occurs on a production environment so I don't know which query produced the problem, but I do know that none of my tables has a column named '123'. Do you have any suggestions about what might be causing this problem?
Edit: As a followup, I found that this behavior is a known bug in MySQL 5.0.4, but I haven't found a good wokaround since I don't have the option of upgrading mySQL.
Could be that somehow the character set for the field has been set to an invalid value, run this SQL on the server with the problem replacing the table name and the column 'Collation' will show the characterset for each varchar/char fields
SHOW FULL COLUMNS IN table_name;
You can then change the character set of a field by using the following:
ALTER TABLE t MODIFY col1 VARCHAR(50) CHARACTER SET latin1;
Internally, MySQL is mapping out the indexes across the columns. Normally this is done using integers that represent the column index. Most indexes are actually a compound index once built (col 1 + col 3) which form something like field 13.
Likely this happened when migrating data from dev into production when the stack is not an exact replica.
As 3urdoch mentioned you can pull the charset using internal MySQL function; and then change the table's charset to a compatible encoding.
Alternatively (where I'm willing to bet this issue came from) you can check if the loaded driver for the Production matches the loaded driver for the Other server. This will prevent re-occurring issue if the Development/Other server is still being used for testing, and migrated into Production.
I have a db in production where all of my tables are using utf8 / utf8_general_ci encoding. This is basically working fine except in one scenario.
What happens is that ??? are being returned for some characters (Chinese, etc); however, they are also returned correctly for the same table but via a different criteria.
I've double checked the connection parameters from Hibernate to MySQL and they have the good charset set.
I cannot understand how this can be happening. The criteria that returns the bad characters is just a simple findById:
Criteria criteria = getHibernateSession().createCriteria(CalendarEvent.class);
criteria.add(Restrictions.eq("id", id));
return (CalendarEvent) criteria.uniqueResult();
This is only happening in production on Solaris - I cannot reproduce it locally.
In your connection-string have you tried
jdbc:mysql://localhost/dbname?characterEncoding=utf8
or add JVM parameter -Dfile.encoding=utf-8 when starting your application / server
Try setting the following properties in your hibernate configuration file:
<property name="hibernate.connection.useUnicode">true</property>
<property name="hibernate.connection.characterEncoding">UTF-8</property>
<property name="hibernate.connection.charSet">UTF-8</property>