MySql VIEW created with charset latin1 all configs are set to utf8 - mysql

I'm trying to create a simple view but I'm getting error because the view is created with latin1 instead of utf8.
The View looks something like this:
create or replace view
my_view
as
select * from my_table
group by some_field
collate utf8_unicode_ci
;
The error I'm getting is:
COLLATION 'utf8_unicode_ci' is not valid for CHARACTER SET 'latin1'
What I did, was check multiple configuration options:
Global
show variables like "%char%";
character_set_client,utf8
character_set_connection,utf8
character_set_database,utf8
character_set_filesystem,binary
character_set_results,utf8
character_set_server,utf8
character_set_system,utf8
The table I'm using to create the view:
SELECT TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, TABLE_COLLATION
FROM INFORMATION_SCHEMA.TABLES
where TABLE_NAME in ('my_table');
;
def,my_database,my_table,utf8_unicode_ci
The Columns of that table:
SELECT TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME, COLLATION_NAME
FROM INFORMATION_SCHEMA.COLUMNS
where TABLE_NAME in ('my_table');
def,my_database,my_table,id,null
def,my_database,my_table,active,null
def,my_database,my_table,title,utf8_unicode_ci

I had the same issue in an MySQL database, where I had Umlauts/special charaters within a CASE like this (reduced sample):
CASE
WHEN cfs.value = "ungeklärt" THEN "" -- the character 'ä' was an issue for me here
ELSE cfs.value
END AS "Category"
The error popped up for me, when savin this statement as a view.
Can you check if it solves the issue for you, when you set the names accordingly first? In my case this was latin1 instead of utf8 due to an old legacy database.
set names latin1;
If I understood your case correclty it would be this for you instead:
set names utf8;

Related

COLLATION 'utf8_bin' is not valid for CHARACTER SET 'utf8mb4'

I am trying to add emoji like 😋 in my application. In order to make it work, I have to use charset : 'utf8mb4' in database connection. But then my other search query doesn't work and throughs error like this
select id, full_name, profile_pic from users where school_id = 1 and exists (select * from groupmembers where (group_id = '110') and (users.id = groupmembers.user_id)) and full_name like '%d%' COLLATE utf8_bin - ER_COLLATION_CHARSET_MISMATCH: COLLATION 'utf8_bin' is not valid for CHARACTER SET 'utf8mb4
How can I make both works together? I am using adonis.js framework and it uses knex query/
Collations are specific to a character set. If your character set is utf8mb4, then you could use collation utf8mb4_bin but not utf8_bin.
This will become a bit more confusing someday, because MySQL has the intention to change the name of utf8mb4 to utf8. Cf. https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8.html
You can check all the collations allowed by this query:
select collation_name
from information_schema.collation_character_set_applicability
where character_set_name = ##character_set_connection;

Get a Table's Character Set

In MySQL, I can get a table's name, engine and collation like so:
SELECT TABLE_NAME, TABLE_SCHEMA, ENGINE, TABLE_COLLATION
FROM information_schema.tables
WHERE table_name = 'tbl_name';
But how how do I get a table's character set, not just collation? Is it possible to get it from information_schema.tables?
Each collation is used for only one character set, so it's not necessary to record the character set in the INFORMATION_SCHEMA.TABLES. The table collation is enough to indicate both the collation and the character set unambiguously.
You can check INFORMATION_SCHEMA.COLLATIONS or INFORMATION_SCHEMA.COLLATION_CHARACTER_SET_APPLICABILITY to get the mapping from a given collation to its character set.
Try running this:
SELECT default_character_set_name FROM information_schema.SCHEMATA S WHERE schema_name = "DBNAME";

Information Schema showing two character encodings for column

I'm in the process of migrating a MySQL database from the utf8 character set to uft8mb4, following this guide (https://mathiasbynens.be/notes/mysql-utf8mb4). For one of the tables I updated (table1) I get weird output from the information_schema. table1 has four columns, each listed below:
data_store VARCHAR(24)
data_group VARCHAR(24)
source_count INT
load_count INT
I have validated that only 4 columns appear through SELECT * on the table. However, running the following query on information_schema produces odd output.
SELECT column_name, character_set_name FROM information_schema.COLUMNS
WHERE table_name = "table1";
COLUMN_NAME CHARACTER_SET_NAME
--------------------------------------------------------------------------------------
data_store utf8
data_group utf8
source_count <null>
load_count <null>
data_store utf8mb4
data_group utf8mb4
source_count <null>
load_count <null>
I don't see duplicate rows (with differing character sets) for any other table that I have updated and am at a loss in regards to what is wrong and/or how to fix it. Any help would be much appreciated!
Notes: I believe I could just remove the unwanted columns from information_schema, but I'm not sure if this would break anything.
I think you're seeing the columns from different table1 tables in several different database schemas.
Try this to verify that claim.
SELECT table_schema, column_name, character_set_name
FROM information_schema.COLUMNS
WHERE table_name = 'table1'
Try this query to filter by the current database.
SELECT column_name, character_set_name
FROM information_schema.COLUMNS
WHERE table_name = 'table1'
AND table_schema = DATABASE()
Do not try to alter the INFORMATION_SCHEMA database in any way. Don't delete rows, don't add columns, or anything else. It's supposed to be readonly. But sometimes it isn't, and altering it can trash your MySQL instance. Don't ask me how I know that. :-)

PHP Mysql - Illegal mix of collations when using CONCAT

Iv'e been trying to concat two columns from my table and perform a LIKE on them
Here is the mysql query:
SELECT * FROM stops WHERE CONCAT_WS(name, ' ', desc) LIKE
'%This%' AND CONCAT_WS(name, ' ', desc) LIKE '%is%' AND
CONCAT_WS(name, ' ', desc) LIKE '%a%'
AND CONCAT_WS(name, ' ', desc) LIKE '%test%'
When I run it via my php code or phpmyadmin I get the following error:
Illegal mix of collations (utf8_bin,NONE) and
(utf8_general_ci,COERCIBLE) for operation 'like'
Iv'e been checking similar questions about this topic and saw that a lot of
the answers told to change the collation of the database.
I already changed all database, tables and fields collcation to utf8_general_ci
When I run the same query exactly in Navicat it works just fine.
Am I missing something here?
Thanks!
set the database and tables -> character to utf8
ALTER DATABASE db CHARACTER SET utf8 COLLATE utf8_general_ci
ALTER TABLE table CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci

How to change all the tables in my database to UTF8 character set?

My database is not in UTF8, and I'd like to convert all the tables to UTF8, how can I do this?
For single table you can do something like this:
ALTER TABLE tab CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;
For the whole database I don't know other method than similar to this:
http://www.commandlinefu.com/commands/view/1575/convert-all-mysql-tables-and-fields-to-utf8
replace my_database_name with your database name
SELECT CONCAT('ALTER TABLE ', TABLE_NAME, ' CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;')
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = 'my_database_name' AND TABLE_TYPE != 'VIEW';
this will build lots of queries which you can run
mysqldump --user=username --password=password --default-character-set=latin1 --skip-set-charset dbname > dump.sql
sed -r 's/latin1/utf8/g' dump.sql > dump_utf.sql
mysql --user=username --password=password --execute="DROP DATABASE dbname; CREATE DATABASE dbname CHARACTER SET utf8 COLLATE utf8_general_ci;"
mysql --user=username --password=password --default-character-set=utf8 dbname < dump_utf.sql
To change the collation in phpMyAdmin just follow this simple steps:
Method 1
open phpMyAdmin and select your database.
After click on database click on operation tab.
Next, Scroll down the page, you will see the collation section.
set the collation do you want and click the checkbox.
your collation will be updated.
Note: If you find any difficulty using method 1 follow method 2 using sql command line.
Method 2
Using command Line
Open phpMyAdmin and click on SQL tab.
Next, write command for changing the collation for your database.
# syntax command:
ALTER DATABASE your_db_name CHARACTER SET utf8 COLLATE write_collation;
# e.g:
ALTER DATABASE temprory CHARACTER SET utf8 COLLATE utf8_general_ci;
Better yet, use Percona's tool kit. I'd audit your indices before updating to utf8mb4 as there are issues with key length.
SELECT CONCAT('pt-online-schema-change --alter "CONVERT TO CHARACTER SET utf8
COLLATE utf8_unicode_ci" t=', TABLE_NAME, ',D=DB_NAME,u=USER_NAME --statistics --execute')
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = 'DB_NAME' AND TABLE_TYPE != 'VIEW' AND TABLE_COLLATION NOT LIKE '%utf8%';
in my case I have several schemas due to a bad migration.
Joining the other answers I did an ALTER command generator in this case to charset=latin1 and collate=latin1_general_ci for all the tables of all the schemas that doesn`t match latin1_general_ci
Hope it helps.
SELECT CONCAT('ALTER TABLE ', tbl.ts,'.',tbl.tn, ' CHARACTER SET latin1
COLLATE latin1_general_ci, CHANGE COLUMN \`', tbl.tc, '\` \`'
,tbl.tc,'\` ', tbl.tct, ' CHARACTER SET latin1 COLLATE
latin1_general_ci;' ) command FROM (SELECT table_schema ts, table_name
tn, column_name tc, COLUMN_TYPE tct FROM information_schema.columns
WHERE collation_name != 'latin1_general_ci' AND table_schema not in
('information_schema','mysql','performance_schema','sys')) tbl;