MySQL Database text encoding errors - mysql

Question

I'm trying to clean up a table I inherited. There's a text column with text in languages other than English and often times the text will look like this: Phénix
I know that it's supposed to be the French word: phénix
So I guess the é would be a failed encoding for the letter é
Does anyone know why this would happen, and is there any way to fix it? The same encoding errors keep on popping up, so is there something like an alphabet equivalent for these encoding errors that I could use to match up against the correct characters?
thanks

Answer 1

CONVERT(BINARY(CONVERT(CONVERT(BINARY(CONVERT('é' USING latin1)) USING utf8) USING latin1)) USING utf8)
--> 'é'
You have Double-Encoding.
Here's what probably happened.
The client had characters encoded as utf8 (good); and
SET NAMES latin1 lied by claiming that the client had latin1 encoding; and
The column in the table declared CHARACTER SET utf8 (good).
Let's walk through what happens to e-acute: é.
The hex for that, in utf8 is 2 bytes: C3A9.
SET NAMES latin1 saw it as 2 latin1-encoded characters à and © (hex: C3 and A9)
Since the target was CHARACTER SET utf8, those 2 characters needed to be converted.
à was converted to utf8 (hex C383) and © (hex C2A9)
So, 4 bytes were stored (hex C383C2A9 for é)
When reading it back out, the reverse steps were performed,
and the end user possibly noticed nothing wrong. What is wrong:
The data stored is 2 times as big as it should be (3x for Asian languages).
Comparisions for equal, greater than, etc may not work as expected.
ORDER BY may not work as expected.
The fix (2 parts):
Be sure to do SET NAMES utf8; (or equivalent, such as mysqli_set_charset('utf8')).
Something like this will repair your data:
UPDATE ... SET col = CONVERT(BINARY(CONVERT(
CONVERT(UNHEX(col) USING utf8)
USING latin1)) USING utf8);