I have a table in MySQL that contains people's names and now some people are putting in characters with accents. Whatever original character set/collation this database used, it could not handle characters with accents and characters. Characters such as 'é' turned into 'é', characters such as 'ü' turned into 'ü' in the front-end applications that connect to this database. The database has since been modified (as well as most front-end applications that needed it) so that these fields use a UTF8 encoding and collation. Most names with accents now render correctly.
The problem is, there are some fields that now have a literal 'é' (some weird double-encoding caused by manually copying and pasting stuff? I don't know, I wasn't there!). I now need to find all of these and modify these fields to use the correct accented characters. To find them, I wrote a query (based on the table I found here):
select count(*), bad_char
from some_table
inner join (
select '€' as bad_char union
select '‚' as bad_char union
select 'Æ’' as bad_char union
select '„' as bad_char union
select '…' as bad_char union
...
-- snip a whole bunch
...
select 'þ' as bad_char union
select 'ÿ' as bad_char ) bad_chars
where some_table.some_text_field like CONCAT('%',bad_chars.bad_char,'%')
group by bad_char
order by count(*);
And I get results like this:
count(*), bad_char
------------------
'1', '¯'
'1', 'Ñ'
'1', 'Ö'
'1', 'Ž'
'1', 'Ç'
'1', '¬'
...snip...
'1797', 'ß'
'4450', 'Ê'
But I have a feeling this doesn't quite work the way I think it does. One of the "bad_char" combinations I'm searching for is 'Ê' but when I run a query like this:
select some_text_field from some_table where some_text_field like '%Ê%';
I get many results that are 'as' which are the same letters but without the accents. But for other cases such as '¯' the query seems to work fine.
How can I get this query to not treat 'as' the same as 'Ê'?
I get many results that are 'as' which are the same letters but without the accents.
That would be an issue of the collation used - those are rule sets for character comparison, and they define which characters are to be treated as equal in different languages.
But you can use the BINARYoperator to change that directly within the query.
For Mojibake ('é' turned into 'é'), the fix for the data is
ALTER TABLE Tbl MODIFY COLUMN col VARBINARY(...) ...;
ALTER TABLE Tbl MODIFY COLUMN col VARCHAR(...) ... CHARACTER SET utf8 ...;
Your SELECTs sound like Mojibake -- € should have been €, correct?
The fix for the code is to understand that
The bytes you have in the client are correctly encoded in utf8 (good).
You connected with SET NAMES latin1 (or set_charset('latin1') or ...), probably by default. (It should have been utf8.)
The column in the tables may or may not have been CHARACTER SET utf8, but it should have been that.
The SET NAMES is often done by language-specific code; what programming language are you using.
Use SHOW CREATE TABLE to see what CHARACTER SET you are using.
For anyone else trying to fix broken UTF8 accented text on latin1 fields, when some of them have proper latin1 and others have broken UTF8, the solution is this:
update posts set post_text = #txt where length(post_text) = length(#txt := convert(binary post_text using utf8));
This will fix only those rows with wrongly encoded UTF8 back into proper latin1 encoding, and á will become á, etc.
Related
Note that this question is NOT about searching for (non)accented characters.
Suppose I have a table where there is a column name, with collation utf8mb4_unicode_ci.
This collation works perfectly for the purpose of selecting the base selection
in a case-insensitive, accent-insensitive way.
The problem is that I need to order the results in an accent-sensitive and case-insensitive way.
The purpose of this is to select every name starting with some character/string and sort them "alphabetically", first should be not-accented, then accented.
From selection e.g.:
Črpw
Cewo
céag
čefw
The final results should be:
Cewo
céag -- because accented e is more than non-accented
čefw
Črpw -- because r is more than e
Note that c/C < č/Č , but lower/upper cases are handled as equals.
I tried searching for this problem, but there are only popping similar questions or questions about searching, which is not the case, the searching itself is fine.
From mentioned I've tried this test query:
SELECT * FROM
(SELECT 'Črpw' as t
UNION SELECT 'Cewo'
UNION SELECT 'céag'
UNION SELECT 'čefw')virtual
ORDER BY t COLLATE utf8mb4_czech_ci ASC
Which produces something very similar to what I want
céag
Cewo
čefw
Črpw
But note that é gets ordered before e.
Is there a way how to get to the results order I want to have?
Using: MySQL 5.5.54 (Debian)
I have a database table which represent people and the records have people's names in them. Some of the names have accented characters in them. Some do not. Some are non-accented duplicates of the accented version.
I need to generate a report of all of the potential duplicates by finding names that are the same (first, middle, last) except for the accents so that someone else can go through this list and verify which are true duplicates, and which are actually different people (I'm assuming they have some other way of knowing).
For example: Jose DISTINCT-LAST-NAME and José DISTINCT-LAST-NAME should be picked up as potential duplicates because they have the same characters, but one has an accented character.
How can this type of query by written in MySQL?
This question: How to remove accents in MySQL? is not the same. It is asking about de-accenting strings in-place and the poster already has a second column of data that has been de-accented. Also, the accepted answer to that question is to set the character set and collation. I have already set the character set and collation.
I am trying to generate a report that finds strings in different records that are the same except for their accents.
I found your question very interesting.
According to this article Accents in text searches, using "like" condition with some character collation adjustments will solve your problem. I have not tested this solution, so if it helps you, please come back and tell us.
Here is a similar question: Accent insensitive search query in MySQL,
according to that, you can use something like:
where 'José' like 'Jose' collate utf8_general_ci
Well, I found something that seems to work (the real query involves a few more other fields, but the same basic idea):
select distinct p1.person_id, p1.first_name, p1.last_name, p2.last_name
from people as p1, people as p2
where binary p1.last_name <> binary p2.last_name
and p1.last_name = p2.last_name
and p1.first_name = p2.first_name
order by p1.last_name, p1.first_name, p2.last_name, p2.first_name;
The results look like this:
12345 Bob Jose José
56789 Bob José Jose
...
This makes sense as there are 2 records for Bob José and I know that in this case, it is the same person but one record is missing the accent.
The trick is to do a binary and non-binary compare on the "last_name" field as well as matching on all other fields. This way we can find everything that is "equal" and also not binary-equal. This works because with the current character-set/collation (utf8/utf8_general_ci), Jose and José are equal but are not binary-equal. you can try it out like this:
select 'Jose' = 'José', 'Jose' like 'José', binary 'Jose' = binary 'José';
The Bane of Character Encodings
There are a wide variety of character-sets and encodings that may be used in MySQL, and when dealing with encoding it is important to learn what you can about them. In particular, take a close look at the differences between:
utf8_unicode_ci
utf8_general_ci
utf8_unicode_520_ci
utf8mb4_general_ci
Some character sets are built to include as many printable characters as possible, to support a wider range of uses, while others are built with the intent of portability and compatibility between systems. In particular, utf8_unicode_ci maps most accented characters to non-accented equivalents. Alternatively, you could use uft8_ascii_ci which is even more restrictive.
Take a look at the utf8_unicode_ci collation chart, and What's the difference between utf8_general_ci and utf8_unicode_ci .
The best answer is from a similar question, "How to remove accents in MySQL?"
If you set an appropriate collation for the column then the value
within the field will compare equal to its unaccented equivalent
naturally.
mysql> SET NAMES 'utf8' COLLATE 'utf8_unicode_ci';
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT 'é' = 'e';
+------------+
| 'é' = 'e' |
+------------+
| 1 |
+------------+
1 row in set (0.05 sec)
How to apply this to your situation?
SELECT id, last-name
FROM people
WHERE last-name COLLATE utf8_unicode_ci IN
(
SELECT last-name
FROM people
GROUP BY last-name COLLATE utf8_unicode_ci
HAVING COUNT(last-name)>1
)
I have a simple table of a single column with rows of char(12) like:
DRF4482
DRF4497
DRF451
DRF4515
EHF452
FJF453
GKF4573
I want to select all of the rows that are between D and F, and have 4 numbers at the end. Like DRF4482, DRF4497, DRF4515, etc. I've tried a number of different wildcard combinations but I get no rows. I'm using:
SELECT * FROM `expired` WHERE id like '%[D-F][A-Z][A-Z]____';
I've even tried to broaden it to:
SELECT * FROM `expired` WHERE id like '%[D-F]%';
and that returns nothing as well.
I've even tried COLLATE latin1_bin based on some other posts but that didn't work either. My table is utf8, but I've created a second table as latin1 and tried a few different collations with the same results - no rows.
Where is my error?
You need to use REGEXP instead of LIKE. Notice that the syntax is a little different; it doesn't do anything with the SQLish % wildcard characters.
So, you want
id REGEXP '[D-F][A-Z][A-Z][0-9]{4}'
for this app. Hopefully you don't have multibyte characters in these strings, because MySQL's regexp doesn't work correctly in those circumstances.
I need to select columns that end in an uppercase extension.
for example, look at the following table.
id - picture
1 - abc.JPG
2 - def.jpg
3 - 123.jpg
4 - xyz.JPG
the results should give me the rows 1 and 4 because JPG are in uppercase.
can anyone help?
I'm far from an expert, but case sensitivity has hung me up before. Are you able to modify the structure of your table? One thing that may help is changing the collation of the table as follows (SQLFiddle here):
CREATE TABLE pics (id INT, picture VARCHAR(200))
CHARACTER SET latin1 COLLATE latin1_general_cs;
INSERT INTO pics VALUES
(1, 'abc.JPG'),
(2, 'def.jpg'),
(3, '123.jpg'),
(4, 'xyz.JPG')
The _cs stands for case sensitive, and I believe the default is case insensitive, which makes case-based comparisons a bit trickier. You can then use the following query to get your rows:
SELECT *
FROM pics
WHERE picture REGEXP '\.[[:upper:]+]$'
If you do not have access to your underlying table, you could try the following, which casts the column in a different character set (latin1), and then changes the collation to support case-insensitive comparisons (SQLFiddle here):
SELECT *
FROM pics
WHERE CAST(picture AS CHAR CHARACTER SET latin1)
COLLATE latin1_general_cs REGEXP '\.[[:upper:]+]$'
Some REGEXPs:
'[.][[:upper:]]+$' -- just uppercase letters
'[.].*[[:upper:]]' -- at least one uppercase letter
'[.].*[[:lower:]]' -- at least one lowercase letter; AND together with previous to get upper and lower, etc.
If there could be two "." in the filename, then consider using
SUBSTRING_INDEX(picture, '.', -1)
to isolate the 'extension'.
Most SQL languages have a UCASE or UPPER function to convert text to uppercase. I'm also taking advantage of the RIGHT function which isn't in all SQL dialects. If your SQL doesn't have a RIGHT function you'll have to futz with SUBSTRING and LENGTH to get the right three characters in picture.
Select id, picture
from table
where UPPER(RIGHT(TRIM(picture),3)) = RIGHT(TRIM(picture),3)
If the same text converted to uppercase is the same as the unconverted text then it is uppercase in the database and will be selected.
As it can not always be assumed that the file extension will be 3 letters you may use the following to get the first chart after the period and compare it to see if it is uppercase:
select * from table where SUBSTRING(picture,CHARINDEX('.',picture) + 1,1)
= upper(SUBSTRING(picture,CHARINDEX('.',picture) + 1,1)) collate SQL_Latin1_General_CP1_CS_AS
My Table collation is "utf8_general_ci". If i run a query like:
SELECT * FROM mytable WHERE myfield = "FÖÖ"
i get results where:
... myfield = "FÖÖ"
... myfield = "FOO"
is this the default for "utf8_general_ci"?
What collation should i use to only get records where myfield = "FÖÖ"?
SELECT * FROM table WHERE some_field LIKE ('%ö%' COLLATE utf8_bin)
A list of the collations offered by MySQL for Unicode character sets can be found here:
http://dev.mysql.com/doc/refman/5.0/en/charset-unicode-sets.html
If you want to go all-out and require strings to be absolutely identical in order to test as equal, you can use utf8_bin (the binary collation). Otherwise, you may need to do some experimentation with the different collations on offer.
For scandinavian letters you can use utf8_swedish_ci fir example.
Here is the character grouping for utf8_swedish_ci. It shows which characters are interpreted as the same.
http://collation-charts.org/mysql60/mysql604.utf8_swedish_ci.html
Here's the directory listing for other collations. I'm no sure which is the used utf8_general_ci though. http://collation-charts.org/mysql60/