I am using Django on Bluehost. I created a form for user generated input, but unicode inputs from this form fails to be stored or displayed of characters. So I did a SO and google search that I should change the Collate and Character set of my database. I run this sql
ALTER DATABASE learncon_pywithyou CHARACTER SET utf8 COLLATE utf8_unicode_ci;
from python27 manage.py dbshell, which initiated a mysql shell, what shows on screen is
Query OK, 1 row affected (0.00 sec).
So I assume the problem is solved, but it is not actually. This sql has not done anything, as I later find it in phpMyAdmin provided by Bluehost. All the Varchar fields of all the tables are still in lantin1_swedish_ci collate.
So assume that alter table should work instead. I run this on mysql
alter table mytable character set utf8 collate utf8_unicode_ci;
although on screen it shows Query OK. 4 rows affected, it actually did nothing either, the collate of those fields in mytable did not change at all.
So I finally manually change the fields in phpMyAdmin for mytable and this works, now I am able to insert in this table with unicode and also they display correctly, but I have around 20 tables of such, I don't want to change them one by one manually.
Do we at all have a simple and effective way of changing Collate of each field to store and display correct unicodes?
Changing collation at the database level sets the default for new objects - existing collations will not be changed.
Similarly, at a table level, only new columns (See comment at the bottom) are affected with this:
alter table mytable character set utf8 collate utf8_unicode_ci;
However, to convert the collation of existing columns, you need to add convert to:
alter table mytable convert to character set utf8 collate utf8_unicode_ci;
In addition to #StuartLC ,
For Changing All 20 tables charset and collation use below query, Here world is database name
SELECT
CONCAT("ALTER TABLE ",TABLE_SCHEMA , ".",TABLE_NAME," CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci") AS AlterSQL
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = "world";
The above will prepare all ALTER queries which you need to run.
Related
I was getting an error on inserting emoji through my android app in mysql db that :
java.sql.SQLException: Incorrect string value: '\xF0\x9F\x98\x8D\xF0\x9F...' for column 'question_text' at row 1
so i searched for it and found out that i had to Change the character set and collation properties of the databases, tables, and columns to use utf8mb4 instead of utf8.
Well i thought of applying to just one column first so i executed the query:
ALTER TABLE question MODIFY question_text varchar(100)
CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
But got the same error , than i thought of applying to the entire database so i executed the query:
ALTER DATABASE my_database CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci;
but still getting the same error . Can someone explain to me what i could be missing?
I have a utf8_general_ci database that I'm interested in converting to utf8_unicode_ci.
I've tried the following commands
ALTER DATABASE dbname CHARACTER SET utf8 COLLATE utf8_unicode_ci;
ALTER TABLE tbl_name CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci; (for every single table)
But that seems to change the charset for future data but doesn't convert the actual existing data from utf8_general_ci to utf8_unicode_ci.
Is there any way to convert the existing data to utf8_unicode_ci?
SHOW CREATE TABLE to see if it really set the CHARACTER SET and COLLATION on the columns, not just the defaults.
What was the CHARACTER SET before the ALTERs?
Do SELECT col, HEX(col) ... for some field that should have utf8 in it. This will help us determine if you really have utf8 in the table. The encoding for characters is different based on CHARACTER SET; the HEX helps discover such.
The ordering (WHERE, ORDER BY, etc) is controlled by COLLATION. The indexes probably had to be rebuilt based on your ALTER TABLE. Did big tables with indexes take a 'long' time to convert?
To actually see the difference between utf8_general_ci and utf8_unicode_ci, you need a "combining accent" or, more simply, the German ß versus ss:
mysql> SELECT 'ß' = 'ss' COLLATE utf8_general_ci,
'ß' = 'ss' COLLATE utf8_unicode_ci;
+-------------------------------------+-------------------------------------+
| 'ß' = 'ss' COLLATE utf8_general_ci | 'ß' = 'ss' COLLATE utf8_unicode_ci |
+-------------------------------------+-------------------------------------+
| 0 | 1 |
+-------------------------------------+-------------------------------------+
However, to test that in your tables, you would need to store those values and use WHERE or GROUP_CONCAT or something else to determine the equality.
What 'proof' do you have that the ALTERs failed to achieve the collation change?
(Addressing other comments: REPAIR should be irrelevant. CONVERT TO tells the ALTER to actually modify the data, so it should have done the desired action.)
You have to change the collation of every field in every table. As you say, the collation of the table is only the default value for fields created later, and the collation of the database is only the default value for tables created later.
As Lorenz Meyer said, the collation of the table is only the default value for fields created later and you need to set the defaults for the columns explicitly too.
Such a change looks like:
ALTER TABLE mytable CHANGE mycolumn mycolumn varchar(15) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci
Does MySQL treat the default character setting in a cascading type of way?
For example, I'm looking at a script that generates the full db and it starts off with a statement like this:
CREATE SCHEMA IF NOT EXISTS `xyz` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci ;
so does that imply that all tables that are created as part of this db will use the UTF8 charset by default? And if I wanted to use a different charset for a given table I would simply have to define it on the CREATE TABLE statement? Is that accurate? Can I override on a specific field of a specific table too? thanks!
Yes. If you set the charset and collation at the database (schema) level, it will apply to any newly created tables inside that database, unless those tables specify their own charset/collation in the CREATE TABLE statement.
And yes, you can also specify charset/collation on a per-field basis. Example from the MySQL manual:
CREATE TABLE t1
(
col1 VARCHAR(5)
CHARACTER SET latin1
COLLATE latin1_german1_ci
);
I have a database that is uft8_general_ci, only problem is up until this point an application coded by a previous developer seems to have been working with the database in latin-1.
I have since changed how the app uses the database and can now store an umlaut as an umlaut instead of ü. The problem now is that the application reads the previously existing data from the database as (example) 'Süddeutsche' instead of 'Süddeutsche'.
Is there anyway to convert the data inside the database from one format to the other?
Regards
Edit:
ALTER TABLE TableName MODIFY ColumnName ColumnType CHARACTER SET latin1;
ALTER TABLE TableName MODIFY ColumnName ColumnType CHARACTER SET binary;
ALTER TABLE TableName MODIFY ColumnName ColumnType CHARACTER SET utf8;
This worked for me.
You could try SET NAMES to let the Database talk in latin-1 with your application while storing in utf-8 or you will need to convert all previous Datasets to utf-8-Strings
try
ALTER DATABASE your_db DEFAULT CHARACTER SET = 'utf8' COLLATE 'utf8_unicode_ci';
and
ALTER TABLE a CONVERT TO CHARACTER SET 'utf8' COLLATE 'utf8_unicode_ci';
ALTER TABLE b CONVERT TO CHARACTER SET 'utf8' COLLATE 'utf8_unicode_ci';
ALTER TABLE c CONVERT TO CHARACTER SET 'utf8' COLLATE 'utf8_unicode_ci';
don't forget to replace the 'ß':
UPDATE a SET field_1 = REPLACE(field_1, 'ß', 'ss') WHERE label LIKE '%ß%';
http://blog.hno3.org/2010/04/22/fixing-double-encoded-utf-8-data-in-mysql/
Using what is listed here has fixed all my problems. I used this with my live data and have had no issues!
Recently I changed a bunch of columns to utf8_general_ci (the default UTF-8 collation) but when attempting to change a particular column, I received the MySQL error:
Column 'node_content' cannot be part of FULLTEXT index
In looking through docs, it appears that MySQL has a problem with FULLTEXT indexes on some multi-byte charsets such as UCS-2, but that it should work on UTF-8.
I'm on the latest stable MySQL 5.0.x release (5.0.77 I believe).
Oops, so I have found the answer to my problem:
All columns of a FULLTEXT index must have not only the same character set but also the same collation.
My FULLTEXT index had utf8_unicode_ci on one of its columns, and utf8_general_ci on its other columns.
Just to add to Thomas's good advice: And to sort things out in PHPMyAdmin you have to change the characterset for all columns AT THE SAME TIME.
Just wasted half a day trying again and again to change the columns one at a time and continually getting the error message about the FULLTEXT index.
For DBeaver/database tool users.
When you use interface to modify more than one column, the tool generate commands like this :
ALTER TABLE databaseName.tableName MODIFY COLUMN columnName1 text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL;
ALTER TABLE databaseName.tableName MODIFY COLUMN columnName2 varchar(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL;
This is not working because you must modify the charsets at the same time.
So, you have to change it manually, in one command :
ALTER TABLE databaseName.tableName
MODIFY COLUMN columnName1 text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL,
MODIFY COLUMN columnName2 text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL;
utf8 or utf8mb4 ? See here.