I have a db in production where all of my tables are using utf8 / utf8_general_ci encoding. This is basically working fine except in one scenario.
What happens is that ??? are being returned for some characters (Chinese, etc); however, they are also returned correctly for the same table but via a different criteria.
I've double checked the connection parameters from Hibernate to MySQL and they have the good charset set.
I cannot understand how this can be happening. The criteria that returns the bad characters is just a simple findById:
Criteria criteria = getHibernateSession().createCriteria(CalendarEvent.class);
criteria.add(Restrictions.eq("id", id));
return (CalendarEvent) criteria.uniqueResult();
This is only happening in production on Solaris - I cannot reproduce it locally.
In your connection-string have you tried
jdbc:mysql://localhost/dbname?characterEncoding=utf8
or add JVM parameter -Dfile.encoding=utf-8 when starting your application / server
Try setting the following properties in your hibernate configuration file:
<property name="hibernate.connection.useUnicode">true</property>
<property name="hibernate.connection.characterEncoding">UTF-8</property>
<property name="hibernate.connection.charSet">UTF-8</property>
Related
Python 3.8.8 programm with Flask 2.0.1 and Flask-SQLAlchemy 2.5.1
MySql database, collation of the tables: utf8_general_ci.
I'm using two other sqlserver DB with SQLALCHEMY_BINDS. Everything runs on Windows 10.
Some chars from select queries on the MySql DB comes wrong: "situazione è decisamente migliorata"
should be: "situazione è decisamente migliorata"
This would solve the problem:
mystring.encode('cp1252').decode('utf8')
but I need a solution at program level. I tried:
appending to SQLALCHEMY_DATABASE_URI connection string:
"?charset=utf8" or "?charset=cp1215" and others
setting app.config['MYSQL_CHARSET'] and
app.config['MYSQL_DATABASE_CHARSET'] to 'utf8', 'utf8mb4', 'latin1', 'cp1252'
...
passing a parameter to SQLAlchemy like db = SQLAlchemy(use_native_unicode="utf8"), many variations here too
No attemp worked. Please I need suggestions.
Are you looking for a way to specify per database connection encoding ?
For all connections try to use
app.config['SQLALCHEMY_ENGINE_OPTIONS'] = {'encoding': 'cp1252'}
For specific connections to different DBs you can also use engine_options:
engine = create_engine('mysql://user:password#hostname/dbname',
encoding='cp1252')
Got the solution.
The problem was not a problem.
The person who build the original database (that is quite old) coded wrong some characters.
Some of my approaches and the one suggested by olegsv, worked, I checked that debugging deep down into into sqlalchemy data structures, the driver accepted the characters encoding, but the very chars in data were themself worong.
This was unespected.
Maybe I should delete the whole question.
I have written hibernate query to save data in a mysql data table. UTF-8 is used inside code. The letter č is inserted into table as ?. I read several threads regarding this issue and tried following solutions.
Adding following line to the end of the jdbc url
?useEncoding=true&characterEncoding=UTF-8"
Adding following lines to the hibernate-config.xml file
**
"<prop key="hibernate.connection.useUnicode">true</prop>
<prop key="hibernate.connection.characterEncoding" >utf8</prop>
<prop key="hibernate.connection.charSet">utf8</pop>"
**
And also I have set the charset to utf8.
None of above didn't work. Are there any solution to resolve this issue?
I've made an application in Java EE that uses Hibernate to communicate with MySQL. It works perfectly on my Windows development machine, but I have problem on debian, where the application is deployed.
When I search for keyword with Polish letters(like ł, ą, ć, ó etc,) the result is ok on Windows, but on server, where I have imported the database it does not work.
Hibernate query looks like this:
#NamedQuery(name = "Keyword.findByKeyword", query = "SELECT k FROM Keyword k WHERE k.keyword = :keyword")
and is called like this:
myEntityManager.createNamedQuery("Keyword.findByKeyword").setParameter("keyword", keyword).getSingleResult();
When I use mysql on debian via SSH and type in SELECT query manually:
SELECT * FROM keywords WHERE keyword = 'ser żółty';
it also works and return single result. Encoding and collations of tables and columns are also ok. In datasource configuration I've added
?UseUnicode=true&characterEncoding=utf8
parameters, but it also did not help. I thought that maybe there is a problem with encoding in data from request send by form, but the problem appears even if the parameter i.e. "ser żółty" is hardcoded in my repository class.
I also use Hibernate Search for indexing and the FullTextEntityManager return correct results with Polish letters.
I assume that there is some problem between Hibernate and MySQL, but I have no more ideas what could I change. Any suggestions?
Server Wildfly9.0.1, MySQL 5.6
Ok the problem was in encoding on the mysql server level. It was set to latin1 by default. To fix this follow this question Change MySQL default character set to UTF-8 in my.cnf? and edit your my.cnf file.
I have an instance of MySQL 5.0.4.1 with an application written in Hibernate. On one of the pages, I get the following error message in the server log:
Unknown character set index for field '123' received from server.at com.mysql.jdbc.Connection.getCharsetNameForIndex(Connection.java:1664)at com.mysql.jdbc.Field.(Field.java:144)at com.mysql.jdbc.MysqlIO.unpackField(MysqlIO.java:506)at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:280)at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1319)at com.mysql.jdbc.MysqlIO.sqlQuery(MysqlIO.java:1218)at com.mysql.jdbc.Connection.execSQL(Connection.java:2233)at com.mysql.jdbc.Connection.execSQL(Connection.java:2193)at com.mysql.jdbc.Connection.execSQL(Connection.java:2174)at com.mysql.jdbc.Connection.setAutoCommit(Connection.java:536)at org.apache.commons.dbcp.DelegatingConnection.setAutoCommit(DelegatingConnection.java:268)at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.setAutoCommit(PoolingDataSource.java:293)at org.hibernate.transaction.JDBCTransaction.toggleAutoCommit(JDBCTransaction.java:194)at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:186)at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:162)at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:603)at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:579)at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:556)at org.springframework.transaction.interceptor.TransactionAspectSupport.doCloseTransactionAfterThrowing(TransactionAspectSupport.java:284)at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:100)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)at org.acegisecurity.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:66)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)at org.acegisecurity.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:66)at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:176)at $Proxy6.getDataFromDatabase(Unknown Source)at org.myCompany.myAction.load(Unknown Source)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)at java.lang.reflect.Method.invoke(Method.java:324)at org.apache.struts.actions.DispatchAction.dispatchMethod(DispatchAction.java:274)at org.apache.struts.actions.DispatchAction.execute(DispatchAction.java:194)at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196)at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)at javax.servlet.http.HttpServlet.service(HttpServlet.java:787)at javax.servlet.http.HttpServlet.service(HttpServlet.java:908)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:264)at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:110)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:217)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:229)at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:148)at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:213)at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:280)at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:509)at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:509)at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:209)at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:509)at com.iplanet.ias.web.connector.nsapi.NSAPIProcessor.process(NSAPIProcessor.java:157)at com.iplanet.ias.web.WebContainer.service(WebContainer.java:579)
This error occurs on a production environment so I don't know which query produced the problem, but I do know that none of my tables has a column named '123'. Do you have any suggestions about what might be causing this problem?
Edit: As a followup, I found that this behavior is a known bug in MySQL 5.0.4, but I haven't found a good wokaround since I don't have the option of upgrading mySQL.
Could be that somehow the character set for the field has been set to an invalid value, run this SQL on the server with the problem replacing the table name and the column 'Collation' will show the characterset for each varchar/char fields
SHOW FULL COLUMNS IN table_name;
You can then change the character set of a field by using the following:
ALTER TABLE t MODIFY col1 VARCHAR(50) CHARACTER SET latin1;
Internally, MySQL is mapping out the indexes across the columns. Normally this is done using integers that represent the column index. Most indexes are actually a compound index once built (col 1 + col 3) which form something like field 13.
Likely this happened when migrating data from dev into production when the stack is not an exact replica.
As 3urdoch mentioned you can pull the charset using internal MySQL function; and then change the table's charset to a compatible encoding.
Alternatively (where I'm willing to bet this issue came from) you can check if the loaded driver for the Production matches the loaded driver for the Other server. This will prevent re-occurring issue if the Development/Other server is still being used for testing, and migrated into Production.
Is it possible to set connection collation within MySql connection string and how, since there's a default setting on the server that's used for new connections.
Two things I can't do:
Can't call SET COLLATION_CONNECTION after I open a connection, because I'm using Entity Framework that does all the calls for me not entirely true as you may see in the edit
Can't change server default connection collation because of other databases and their respected applications that use them.
All I'd like to specify is a certain connection string parameter in my web.config file like:
"User id=dbuser;Password=dbpass;Host=dbserver;Database=testung;Collation=utf8_general_ci"
but Collation setting/variable isn't recognised.
Technologies used
Asp.net MVC 2
IIS 7
Entity Framework 1
DevArt dotConnect MySql connector
MySql 5.1
EDIT 1
I've tried this code as #Devart suggested but to no avail:
partial void OnContextCreated()
{
System.Data.Common.DbCommand command = this.Connection.CreateCommand();
command.CommandText = "set collation_connection = utf8_slovenian_ci;";
command.CommandType = System.Data.CommandType.Text;
this.Connection.Open();
command.ExecuteNonQuery();
// this.Connection.Close();
}
We recommend you to implement the OnContextCreated partial method.
You have access to the store connection in it and you can execute ADO.NET command "SET COLLATION = ..." using this connection.
If anyone else stumbles over this problem or wants to issue a command when opening a connection: The answer regarding OnContextCreated does no longer work as the method does no longer exist/is no longer supported.
An alternative, which I use for executing SET NAMES <character set used by the database> is to append ;initialization command=\"SET NAMES '" + CharSet + "';\" to your connection string. According to Devart's documentation this also works for PostgreSQL, MSSQL and Oracle
This property can also be set inside EntityDeveloper when accessing the properties of the database connection and clicking on the Advanced button.