I am trying to save some Indian language content (read Hindi) from a website into a column in MySQL database. I am using SpringBoot and JPA to scrape the website and write the content.
Everything works fine on my local system (OSX). The same code is deployed on Ubuntu 16.04 LTS. It works fine except that the content of the relevant db column shows '????'. I presumed that it might be a problem with the COLLATION and CHARACTER SET of MySQL of the prod instance.
Indeed MYSQL version on my local machine was 8.x. The COLLATION on local db was utf8mb4_0900_ai_ci. The version on prod was 5.6, the COLLATION being utf8_x_x. Apparently, utf8mb4_0900_ai_ci, is not available for versions prior to 8.x.
I tried changing the COLLATION of columns (the entire schema for that matter) on local machine to utf8mb4_general_ci, as well as utf8mb4_unicode_ci. Both works fine on local DB. But the same collation, namely utf8mb4_general_ci & utf8mb4_unicode_ci still gives '?????' in the prod environment!
Could this be a problem other than 'COLLATION'? I am able to print the Hindi content on terminal, both on local and Ubuntu deployment. The problem is less likely with the client, as the same client (Datagrip) shows local schema content nicely and ubuntu DB content as cryptic question marks.
Have googled for two days. Any help would be appreciated.
Related
SonarQube's install guide says that UTF-8 must be set when setting up MySQL. Is there a way I can check if my MySQL instance is compatible before attempting the upgrade?
I've done a few checks and we seem to have UTF8 setup for the client, connection, results, and system. At the database and server level we have swedish case insensitive. We do have UTF8_general_ci set for the collation_collection.
This instance will not be used in a multi language situation (only Us English). Do my settings align with what they should or do I need to spin up an entirely new instance?
You need to configure the url using jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8 (see for example here). Read also the MySql Character Set Configuration documentation page. The settings are used by the SonarQube program, plugins and upgradescripts.
I have strange problem, meaning:
in my company we have 2 kinds of databases 1 oracle and 1 mysql and 1 remote in hosting.
when ERP which is on oracle codding is set to utf-8 is making update or insert to remote mysql db everything is ok all chars like ńłóść are coded correctly, but doing the same update to my local mysql db chars like ńłóść are replaced by ?. I tried to
change coding from utf8_general_ci to utf8_unicode_ci didn't work,
change engine from InnoDB to MylSam didn't work
varsion of my local MySQL is 5.6.12
version of my remote MYSQL is 5
but i don't think there is problem, i use wamp 2.4
Can i check in what encoding query is ?
Short answer: use utf8mb4. Search for it on Stack Overflow, or see How to support full Unicode in MySQL databases.
I'm using MySQL Workbench and successfully migrating a SQL Server database used with SilverStripe PHP CMS to MySQL on Linux. Problem is when the content is displayed on the Linux web server I have to change encoding to Western (Windows-1252) to get the content to display correctly. The site on Windows IIS with SQL Server displays correctly with the default UTF-8 encoding.
In the manual migration editing section on MySQL Workbench some columns say Collation Latin1_General_CI_AS migrated to utf8_general_ci so I gather this is correct.
The site is setting <meta http-equiv="Content-type" content="text/html; charset=utf-8" /> in the HTML.
Migration process:
Use MySQL Workbeanch to migrate from SQL Server to MySQL on
production server
Export MySQL database on production server to *.sql file
Import *.sql file into Linux server using PHPMyAdmin and default UTF-8 encoding
I'm not sure where in the migration process I need to fix this?
This might not be about the migration process at all.
If you are using PHP to access the new database, the connection charset might be incorrect.
After setting up the connection you should set the connection charset to UTF-8.
$db = new MySQLi(HOST, USER, PASSWORD, DATABASE);
$db->set_charset('utf8');
Or, if you are not using MySQLi:
mysql_connect(/*...*/);
mysql_set_charset('utf8');
There are 3 possibilities of what could have gone wrong:
Encoding miss-configuration at the original SQL-server setup. Look at this post for more details on how this can happen: https://stackoverflow.com/a/20824533/684229. In this case, the encoding in the SQL-database is incorrect, but due to wrong setup it displays correctly.
You made a mistake in the migration process. Then the encoding in the new MySQL database is incorrect.
The encoding in the new MySQL database is correct, but there is an encoding miss-configuration at the new Linux MySQL setup, which makes it look incorrect.
To check which case applies, you have to check the encoding in both databases by some independent tool which for sure (200% at least!!!) has the encoding configured correctly. I would use PHPMyAdmin in case of Linux, I don't know what's available on SQL server. But make sure that this tool is configured correctly, otherwise you will get fooled!!!
Post the result and I will expand my answer accordingly.
EDIT: Dave, I have numbered the steps of your migration process. Please check the encoding of your MySQL database at two points - right after step 1 (before you do the export & import) as well as after you export & import in step 3. This will have to detect the exact point where it went wrong.
I have deployed my Rails 3.1 app with the MySQL database to Heroku and there everything works fine. I mean, into database are saved the chars right (seems to be used UTF charset on a databases on Heroku).
But when I will run the command heroku db:pull (this command will download a whole database from Heroku into the database on localhost), so the downloaded data stored in databased have bad coding - a chars are displayed bad (it looks like my local MySQL database have a different set up of charset than the MySQL on Heorku).
Could anyone give me a tip, how I can find the set up of charset used on Heroku database and how to use it on my local MySQL database?
Many thanks!
All is not lost - you really don't have to use PostgreSQL if you don't want to.
If your database is small enough (which it will have to be since the Heroku PostGres DB is also 5Mb) and you would prefer to remain on mySQL then you could use the ClearDB mySQL addon - http://addons.heroku.com/cleardb - their entry level DB is free and is the same size as the Heroku Shared PostGres DB that you get by default but be careful that the number of connections is limited so don't be going crazy with your web dyno counts.
Once you add the addon if you look at the output of heroku config then you can use the DATABASE_URL to create a connection in your favourite mySQL administration tool locally to restore/backup etc data to ClearDB. You may even find heroku db:push would work but personally I've not tried that so would be guessing.
The problem is that Heroku does not use a MySQL database in production, but a PostgreSQL database.
Therefore you will run into all sorts of issues pulling and pushing data from a different database engine. Taps is an activerecord based process that will reduce this problem but not all the time.
Ideally you want to use PostgresSQL on your development machine (install via Homebrew for simplicity on OSX) and you'll not see any more of these problems.
Alternatively, use one of the MySQL addons as described in the comments in the question.
I have some trouble showing right single quote from mysql database.
Screenshot:
This problem appears only when I deploy the application on a server.
If I start the application locally, connected to the database on the server, it's ok.
Any ideas?
Perhaps the encoding is set different on the server than your system. Check what the default encoding is on your system and then the server. The MySQL will just use what is set in the DB not the system.
http://msdn.microsoft.com/en-us/library/system.text.encoding.default.aspx