I am a beginner when it comes to databases, so please bear with me. I'm trying to set up a database and import some tables from a file tables.sql. Some of the Columns in tables.sql have Swedish letters in them (Ä, Ö) and the problem is that I get the following:
Ä = ä
Ö = ö
First I begin to check the character set of the server:
mysql> show variables like 'character_set_server';
The server is configured to character set 'Latin-1'. I must mention that I have no control over the server more than to create a database. So I guess I have to create my database and specify the character set of the database.
This is how I proceed:
mysql> create database db;
mysql> alter database db character set utf8 collate utf8_swedish_ci;
I double checked that my tables.sql have charset utf-8 by executing:
file -bi allsok_tables.sql
And then I load it into the database by:
$ mysql -u [username] -h [hostname] -P [port] -p db < tables.sql
when I create my tables in tables.sql I use engine = InnoDB (don't know if this is relevant or not). However if I now select everything from the table TableTest
mysql> select * from TableTest
I get these weird characters instead of the Swedish characters. I appreciate any help right now.
Thanks in advance!
UPDATE:
If I insert a value manually into a table it works e.g.
mysql> insert into TableTest values ('åäö');
So the problem seems to be with the .sql-file. Right?
$ mysql ... --default-character-set=utf8 < tables.sql
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
MySQL needs to know what encoding the data you're sending it is in. To do this, you need to set the connection encoding. When connecting to MySQL from a client, you usually run a SET NAMES query or use an equivalent call on your API of choice to do so. On the command line, the --default-character-set option does this. It needs to be set to whatever encoding your file is in.
Related
I have Percona XtraDB 5.6 server with very old database with charset set to utf8 and data encoded in different charset (probably latin1).
I tried to migrate the database to new Percona 8.0 server but after importing the SQL file, all diacritic marks become broken on the 8.0 server. I have resolved the issue by executing this query on every column in every table:
UPDATE table SET col = convert(cast(convert(col using latin1) as binary) using UTF8) WHERE 1;
BUT there is one table with binary data (specifically GZIP compressed data) saved into LONGTEXT columns. Data from this columns always becomes damaged after import on the new server.
This is what I tried so far:
changing column type to LONGBLOB before dump.
using the above query to convert the data before/after column type change.
This is the command I'm using to export DB:
mysqldump --events --routines --triggers --add-drop-database --hex-blob --opt --skip-comments --single-transaction --skip-set-charset --default-character-set=utf8 --databases "%s" > db.sql
Please note the "--hex-blob" option which still results in binary data being exported as strings, instead of hex.
It would not have been damaged by zip/unzip. But it could have been damaged in any of a number of other ways.
"--hex-blob" turns data into strings such that they will not be mangled until you reload them.
Dumping, loading, INSERTing, SELECTing all need to be told what character set to use.
The particular UPDATE you did may or may not have made things worse. Here is a list of cases that I have identified, and the minimal "fix":
http://mysql.rjweb.org/doc.php/charcoll#fixes_for_various_cases
I'm migrating a small Gyroscope app from MySQL to MariaDB. I asked this question earlier on why MariaDB would not start on latin1 as server encoding (MariaDB won't start when character-set-server = latin1 is in my.cnf), but now I've come to terms with using just utf8.
After importing the data dump:
mysql -uwebuser -p ezine < dump.sql
, what's supposed to be autotööstus shows up as autot??stus. In addition I got this error:
illegal mix of collations (latin1_general_ci implicit) and (utf8_general_ci coercible), storing ??
Attempt 1: I removed all the occurrences of charset=latin1 in the data dump, and the import terminated early with this error:
Specified key was too long; max key length is 767 bytes
Attempt 2: added back charset, but set it to charset=utf8, according to this post: MySQL Convert latin1 data to UTF8
The import works, but autotööstus is now showing as autotööstus
How do I interpret the data dump as latin1 encoded content but deposit into a utf8 storage?
If you cannot use latin1 as the default system encoding for whatever reason, here's a quick solution:
Leave the data dump as-is. Even though it's latin1 encoded, it can be imported as utf8.
mysql> create database ezine character set utf8 collate utf8_general_ci;
mysql -uwebuser -p --default-character-set=utf8 ezine < dump.sql
Now in your web app, you should correctly see autotööstus
From now on, the "native currency" of your database is utf8. When dumping the database, you have to be careful, make sure the exported file is still latin1 encoded.
Otherwise you'll end up with the over-encoded data, such as autotööstus.
mysqldump -uwebuser -p --default-char-set=latin1 ezine > dump.sql
At shell command prompt:
mysqladmin -u"username" -p"password" --default-character-set=utf8 CREATE my_db_schema
--default-character-set=utf8 seems to have no effect and I don't understand why.
Database gets created, but character set is latin1 with collation latin1_swedish_ci.
I found this question, which would seem to be the same issue, but even when I tried a non-root user as the selected answer suggested, I get identical behavior:
MySQL connection character set problems
(I'm using Windows and MariaDB if that makes any difference)
I have tried these mysqladmin.exe clients:
MariaDB 5.3.2 for Win32 (ia32) with default character set latin1 (no .ini)
MySQL 5.0.77 for linux-gnu (i686) with default character set utf8
In both cases, --default-character-set=utf8 or --default-character-set=latin1 do NOT override the MySQL server's .ini/.cnf settings.
As a workaround I'd suggest running:
echo "CREATE DATABASE my_db_schema DEFAULT CHARACTER SET utf8" | mysql -uusername -ppassword
--default-character-set=utf8 seems to have no effect and I don't understand why.
Database gets created, but character set is latin1 with collation latin1_swedish_ci.
This options does not influence the character of a datatabase, table or column when they are created.
The default-character-set is the character set of the connection to the server -- it ensures values you select from the database come through to the client with the correct encoding for display.
On the surface I'd say this appears to be a mysqladmin bug. I would let the MariaDB devs know about it.
http://kb.askmonty.org/en/reporting-bugs has general instructions about reporting bugs (ignore the bit about using the mysqlbug script, since it is not available on Windows).
P.S. And if the bug exists in MariaDB it likely also exists in MySQL.
The title saiz it all, right? :)
Details:
I'm looking for a SQL query I can use in phpmyadmin. Need to Change all my fields in one database from latin1_swedish_ci to utf8_general_ci
The main problem in your case is when you change the collation/character set from the table the data inside the table are latin1 and the table is utf8 until then.
What you can try is to make a dump of the table with the old character set.
mysqldump -uuser -p --default-character-set=latin1 dbname > dump.sql
and then import the database with the new character set like this:
mysql -uuser -p --default-character-set=utf8 dbname_test < dump.sql
To test i would import the dump to a test database. If the characters are not correct after the import. Use an Editor like pspad or notepad++ and change the file encoding to UTF-8. After this you can try to import another time.
The most times encodings are very annoying but i hope you can solve it.
When you only have phpmyadmin use the "Export" function and make the same with phpmyadmin. Export in the actual encoding and try to import in the new encoding but then you have to change the file encoding i think.
Change the default collation to utf8_general_ci in MySql:
Open the my.ini file. (C:\xampp\mysql\bin\my.ini)
Find the text [mysqld] and add the below lines.
[mysqld]
character-set-server = utf8
collation-server = utf8_general_ci
The above two lines will select a character set and collation at server startup. These settings apply server-wide and apply as the defaults for databases created by any application, and for tables created in those databases.
I had an old table with latin1 charset. Using phpmyadmin, I convert it to utf-8
After that, when I read data with php, my data shows as ???? ????? question marks
my page charset is utf-8 there is no problem with my php , and i also tried :
#mysqli_query("SET NAMES 'utf8'", $db);
#mysqli_query("SET CHARACTER SET 'utf8'", $db);
#mysqli_query("SET character_set_client = utf8 ",$db);
#mysqli_query("SET character_set_results = utf8 ",$db) ;
#mysqli_query("SET character_set_connection = utf8 ",$db);
before any query
seems doesn't work, still showing as ???? ??????
there is no problem for new records, but old records are not readable
they are stored in db like : غلامی
Is there any way to retrieve those old data?
What I used to do is go on the command line
mysqldump --default-character-set=latin1 ... > file.sql
then convert file.sql to UTF (using ICONV or any other option) then alter the table's charset and use
mysql --default-character-set=utf8 < file.sql
You could just use the export option from phpMyAdmin and converting like in the above example, change the charset on the table from phpMyAdmin and then import it again.
I usually did a backup before doing this cause mistakes happen but its very simple