I'm writing a program. This program transfers data to MySql database which is in SQL Server Datas.
MySql database default charset is Latin1. Latin5 charset is usually used for Turkish characters. But I can't change mysql table's charset because it's a very old database.
Is there any way to import turkish chars to mysql database correctly?
To test try:
CREATE TABLE newtable LIKE oldtable;
-- change the character latin character set to latin5
ALTER TABLE newtable MODIFY latin1_text_col TEXT CHARACTER SET latin5;
INSERT INTO newtable
SELECT * from oldtable;
If everything looks good you can drop the old table and rename the newtable to have the same name as the oldtable.
Related
I have approx 70 databases, each database contains 750+ tables (exact same structure), and lot of data stored but, the problem is only few databases set to utf8 and others are latin1, so latin1 database saved double encoded values like 接近åˆå ± for 接近初報
So i want to convert all my databases to utf8mb4 so it should save correct data, but this will obviously requires existing double encoded data to convert to utf8mb4
I have following sql query to convert data.
UPDATE table SET col = IFNULL(CONVERT(CONVERT(CONVERT(col USING latin1) USING binary) USING utf8), col )
But the problem is my databases are very large and this will take lot of time to convert data to utf8. so is there any easy way to update data for whole database in one go or something else which is easy?
Many thanks
You really should be using utf8mb4 for Chinese; some Chinese characters are not representable in MySQL's 3-byte utf8.
A slightly shorter expression:
CONVERT(BINARY(CONVERT(col USING latin1)) USING utf8mb4)
Which case? see http://mysql.rjweb.org/doc.php/charcoll#fixes_for_various_cases -- You probably need the 3rd of these:
CHARACTER SET latin1, but have utf8 bytes in it; leave bytes alone while fixing charset:
First, lets assume you have this declaration for tbl.col:
col VARCHAR(111) CHARACTER SET latin1 NOT NULL
Then, to convert the column without changing the bytes:
ALTER TABLE tbl MODIFY COLUMN col VARBINARY(111) NOT NULL;
ALTER TABLE tbl MODIFY COLUMN col VARCHAR(111) CHARACTER SET utf8mb4 NOT NULL;
Note: If you start with TEXT, use BLOB as the intermediate definition. Since ALTER needs to know all the details (size, nullness, etc), it is quite messy to dynamically create the ALTERs.
CHARACTER SET utf8mb4 with double-encoding:
UPDATE tbl SET col = CONVERT(BINARY(CONVERT(col USING latin1)) USING utf8mb4);
CHARACTER SET latin1 with double-encoding: Do the 2-step ALTER, then fix the double-encoding.
Going through the tables:
SELECT CONCAT("UPDATE ", table_schema, ".", table_name, "
SET ", column_name, " = CONVERT(BINARY(CONVERT(", column_name,
" USING latin1)) USING utf8mb4);")
FROM information_schema.columns
WHERE character_set_name = 'latin1';
Then copy & paste the output. (Or write a Stored Procedure to do the execute.)
Caveat: The SELECTs may pick more tables/columns than it should.
I would like to save extra rare Chinese character in database using mysql,
however keep showing ???? after the text is inserted into the table.
Previously I have searched how to insert any 4byte Chinese character into mysql data, and following the process ("Incorrect string value" when trying to insert UTF-8 into MySQL via JDBC?) but still no use. Is there something I miss?
Fyi, I have updated mysql version into 5.7.18, and I try to insert the extra rare Chinese character from phpmyadmin directly.
thanks in advance!
As per the CREATE TABLE Syntax:
CREATE [TEMPORARY] TABLE [IF NOT EXISTS] tbl_name
(create_definition,...)
[table_options]
[partition_options]
[...]
table_option:
ENGINE [=] engine_name
| AUTO_INCREMENT [=] value
| AVG_ROW_LENGTH [=] value
| [DEFAULT] CHARACTER SET [=] charset_name
In other words:
CREATE TABLE test (
some_column VARCHAR(100)
)
CHARACTER SET = utf8mb4;
It won't hurt either to pick a specific collation.
You also must change the connection to be utf8mb4. See http://stackoverflow.com/questions/38363566/trouble-with-utf8-characters-what-i-see-is-not-what-i-stored for discussion of "question marks".
Note: The data failed to be stored correctly; it cannot be recovered without re-inserting the data.
My Table collation is utf8_general_ci and character set is utf8. If I run a query like the following, it does not show me any value.
SELECT * FROM mytable WHERE myfield = "Björn Borg"
OR myfield = "FrüFrü & Tigerlily";
So how I can fetch the values from table ?
The problem can be in your connection charset. Try to run
SET NAMES 'utf8mb4'
And then your query.
There are more places where MySQL defines character sets, including at the column level.
To check the database settings:
SHOW VARIABLES LIKE 'char%';
SHOW VARIABLES LIKE 'collation%';
To check the table settings: SHOW CREATE TABLE tablename;
To check the individual columns: SHOW FULL COLUMNS IN tablename;
If you created a database with the wrong encoding, you will have to change every single occurence of a bad charset in all of these.
I managed to insert special characters into a table by setting the charset with
CHARSET=utf8;
Thing is, when I run the following query on the table
SELECT * FROM table WHERE word = 'francais';
it returns both "francais" and "français"!
This is not quite desirable for my situation.. I have no idea why it does this because they're just different...
Can anyone tell me how to avoid this? Would be much appreciated.
lordstyx
Try using collation, e.g.,
select *
from table
where word = 'francais' collate utf8_bin;
When I encoding the following character to UTF-8:
º
I get:
º
Then with º stored as a field value, I select the field with the LOWER() function and get
âº
I was expecting it to respect that the value is a multi-byte character and thus will not perform the LOWER on it.
Expected:
º
I am I not understanding correctly that the LOWER() function is suppose to be multi-byte safe as stated in the manual? (http://dev.mysql.com/doc/refman/5.1/en/string-functions.html#function_lower)
Or am I doing something wrong here?
I am running MySQL 5.1.
EDIT
The encoding on the table is set to UTF-8. The session encoding is default latin1.
Here are my repro steps.
CREATE TABLE test_table (
test_field VARCHAR(1000) DEFAULT NULL
) ENGINE=INNODB DEFAULT CHARSET=utf8;
INSERT INTO test_table(test_field) VALUES('º');
SELECT LOWER(test_field) FROM test_table;
INSERT INTO test_table(test_field) VALUES('º');
Will insert a 2 character string, which has the correct LOWER() of "âº"
Lower("Â") is "â"
Lower("º") is "º"
If you want to insert "º" then make sure you have
SET NAMES 'utf-8';
and
INSERT INTO test_table(test_field) VALUES('º');