I have a mysql dump generated from phpmyadmin in a windows environment,
when i try to import in osx (using mysql command line) there are encoding
problems, the databases have the same encoding and collation.
I've also noticed that this problems occurs also when i try to import a
diferent database from a unix virtual machine.
When i try to import the same databases in the Windows with the same commands everything is ok.
Anyone have a ideia about whats going on?
Thanks.
Both databases have the following configuration:
character_set_client | latin1
character_set_connection | latin1
character_set_database | utf8
character_set_filesystem | binary
character_set_results | latin1
character_set_server | utf8
character_set_system | utf8
character_sets_dir | /usr/local/mysql-5.1.43-osx10.6-x86_64/share/charsets/
Related
i have a problem...
we moved site from server to server...
and after that our site ascii code not working true...
if i add an ascii code to editor on vbulletin there is problem after saving post...
i saw on database..
▄ ; must be save but saving „â–
there is a problem but how can i solve this problem..
i looked show variables ...
character_set_client utf8
character_set_connection latin1
character_set_database latin1
character_set_filesystem binary
character_set_results utf8
character_set_server latin1
character_set_system utf8
character_sets_dir /usr/share/mysql/charsets/
collation_connection latin1_swedish_ci
collation_database latin1_swedish_ci
collation_server latin1_swedish_ci
but i looked on vbulletin sql query
all of them latin1...only character_set_system utf8 but i cant latin1 all of them..
how can i fix it...
please help me ...
i looked 3 months ...
sorry my bad english.
I am trying to transfer one of my databases from one host (home.pl) to another (my newly set server). The script that I am trying to transfer is wordpress. Unluckily irrespective of the method used I am struggling with encoding problems.
New host configuration
In my new server I am using the following directives in my.cnf:
[mysql]
default-character-set=utf8
[mysqld]
collation-server = utf8_general_ci
character-set-server = utf8
init_connect='SET collation_connection = utf8_general_ci'
init_connect='SET NAMES utf8'
[client]
default-character-set=utf8
My mySQL vars:
character_set_client utf8
character_set_connection utf8
character_set_database utf8
character_set_filesystem binary
character_set_results utf8
character_set_server utf8
character_set_system utf8
collation_connection utf8_general_ci
collation_database utf8_general_ci
collation_server utf8_general_ci
Php.ini on new server:
; PHP's default character set is set to UTF-8.
; http://php.net/default-charset
default_charset = "UTF-8"
Old host configuration
I have runned SHOW VARIABLES in my old host from which I am trying to transfer database and I got the following:
character_set_client utf8
character_set_connection utf8mb4
character_set_database utf8
character_set_results utf8
character_set_server latin2
character_set_system utf8
/usr/local/pssql55/share/charsets/
collation_connection utf8mb4_general_ci
collation_database utf8_polish_ci
collation_server latin2_general_ci
Transfer methods tried out
1) Transfer via phpmyadmin
I have tried using PHPMYADMIN export/import. In particular I have pointed out UTF-8 as file character set both during export and import via phpmyadmin.
What is strange both in phpmyadmin on source server and new host I don't see polish chars (the output is the same without polish chars).
2) Export / Import via mysql dump
I have tried also to use:
mysqldump -h OLD_HOST -u OLD_USER -p DB | mysql -h localhost -u root NEW DATABASE
but the encoding also fails.
Tried to use also encoding variables but it also failed:
mysqldump --default-character-set=latin1 | mysql --default-character-set=utf8
Dump file
In my dump file using Programers Notepad with UTF-8 encoding set, charcters look like this:
"Ä" instead of "ę"
Opening them in microsoft word I see
Ä™ instead of "ę"
The encoding converter (gżegżółka) recognises that the file is in:
C:\Users\mkondej001\Desktop\14271425_mk.sql
Kodowanie: Unicode UTF-8
EOL: LF (Unix)
Any clues how to transfer DB / set server variables correctly ?
At the end I have founded out that the problem was related to the fact that the data was written to SQL incorrectly in my original server.
I ended up with transferring DB using:
mysqldump --default-character-set=utf8 [ORYGINAL_DB] | mysql [TARGET_DB] --default-character-set=utf8
and the executing:
UPDATE [table name] SET [field] = CONVERT(BINARY CONVERT([field] USING latin2) USING utf8)
as it was advices here:
strange character encoding of stored data , old script is showing them fine new one doesn't
Hope that the above solution will be helpful for others too.
SET NAMES utf8;
(The default is latin11, which leads to Ä™.)
Note: init_connect is not executed for root (or any SUPER) user. So this failed you:
init_connect='SET NAMES utf8'
I'm in the process of upgrading an old legacy Rails 2.3 app to something more modern and running into an encoding issue. I've read all the existing answers I can find on this issue but I'm still running into problems.
Rails ver: 2.3.17
Ruby ver: 1.9.3p385
My MySQL tables are default charset: utf8, collation: utf8_general_ci. Prior to 1.9 I was using the original mysql gem without incident. After upgrading to 1.9 when it retrieved anything with utf8 characters in it would get this well-documented problem:
ActionView::TemplateError (incompatible character encodings: ASCII-8BIT and UTF-8)
I switched to the mysql2 gem for it's superior handling and I no longer see exceptions but things are definitely not encoding correctly. For example, what appears in the DB as the string Repoussé is being rendered by Rails as Repoussé, “Boat” appears as “Boatâ€, etc.
A few more details:
I see the same results when I use the ruby-mysql gem as the driver.
I've added encoding: utf8 lines to each entry in my database.yml
I've also added the following to my environment.rb:
Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8
It has occurred to me that I may have some mismatch where latin1 was being written by the old version of the app into the utf8 fields of the database or something, but all of the characters appear correctly when viewed in the mysql command line client.
Thanks in advance for any advice, much appreciated!
UPDATE: I now believe that the issue is that my utf8 data is being coerced through a binary conversion into latin1 on the way out of the db, I'm just not sure where.
mysql> SELECT CONVERT(CONVERT(name USING BINARY) USING latin1) AS latin1, CONVERT(CONVERT(name USING BINARY) USING utf8) AS utf8 FROM items WHERE id=myid;
+-------------+----------+
| latin1 | utf8 |
+-------------+----------+
| Repoussé | Repoussé |
+-------------+----------+
I have my encoding set to utf8 in database.yml, any other ideas where this could be coming from?
I finally figured out what my issue was. While my databases were encoded with utf8, the app with the original mysql gem was injecting latin1 text into the utf8 tables.
What threw me off was that the output from the mysql comand line client looked correct. It is important to verify that your terminal, the database fields and the MySQL client are all running in utf8.
MySQL's client runs in latin1 by default. You can discover what it is running in by issuing this query:
show variables like 'char%';
If setup properly for utf8 you should see:
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
If these don't look correct, make sure the following is set in the [client] section of your my.cnf config file:
default-character-set = utf8
Add add the following to the [mysqld] section:
# use utf8 by default
character-set-server=utf8
collation-server=utf8_general_ci
Make sure to restart the mysql daemon before relaunching the client and then verify.
NOTE: This doesn't change the charset or collation of existing databases, just ensures that any new databases created will default into utf8 and that the client will display in utf8.
After I did this I saw characters in the mysql client that matched what I was getting from the mysql2 gem. I was also able to verify that this content was latin1 by switching to "encoding: latin1" temporarily in my database.conf.
One extremely handy query to find issues is using char length to find the rows with multi-byte characters:
SELECT id, name FROM items WHERE LENGTH(name) != CHAR_LENGTH(name);
There are a lot of scripts out there to convert latin1 contents to utf8, but what worked best for me was dumping all of the databases as latin1 and stuffing the contents back in as utf8:
mysqldump -u root -p --opt --default-character-set=latin1 --skip-set-charset DBNAME > DBNAME.sql
mysql -u root -p --default-character-set=utf8 DBNAME < DBNAME.sql
I backed up my primary db first, then dumped into a test database and verified like crazy before rolling over to the corrected DB.
My understanding is that MySQL's translation can leave some things to be desired with certain more complex characters but since most of my multibyte chars are fairly common things (accent marks, quotes, etc), this worked great for me.
Some resources that proved invaluable in sorting all of this out:
Derek Sivers guide on transforming MySQL data latin1 in utf8 -> utf8
Blue Box article on MySQL character set hell
Simple table conversion instructions on Stack Overlow
You say it all looks OK in the command line client, but perhaps your Terminal's character encoding isn't set to show UTF8? To check in OS X Terminal, click Terminal > Preferences > Settings > Advanced > Character Encoding. Also, check using a graphical tool like MySQL Query Browser at http://dev.mysql.com/downloads/gui-tools/5.0.html.
I'm currently attempting to switch from my shared inmotionhosting account (have received AWEFUL service lately) to an Amazon EC2 server that I've set up. I'm having trouble with getting the encryption function working in the EC2 account.
In my PHP code, all text gets encrypted by mcrypt before being put into the SQL. I have deduced that those mcrypt characters are responsible for all my queries throwing errors. (I know it's because of encoding issues, but Google searches on the subject aren't very clear on where I need to focus my attention.)
A more simplified way of explaining the problem. On my new hosting account this SQL query doesn't work:
UPDATE mydatabase.clients SET firstname='\'å».”é¶Q' WHERE id_client=65
But this does
UPDATE mydatabase.clients SET firstname='Test' WHERE id_client=65
So that tells me the mcrypt function is using characters that the SQL database doesn't understand and thus the queries aren't working.
Some other info for you...
When I run "SHOW VARIABLES LIKE 'character_set_%'" on the working database I get this:
Variable_name Value
character_set_client utf8
character_set_connection utf8
character_set_database latin1
character_set_filesystem binary
character_set_results utf8
character_set_server latin1
character_set_system utf8
When I do that on the nonworking database I get:
Variable_name Value
character_set_client utf8
character_set_connection utf8
character_set_database utf8
character_set_filesystem binary
character_set_results utf8
character_set_server utf8
character_set_system utf8
I saw the difference in character_set_database and ran this line of code:
ALTER DATABASE mydatabase DEFAULT CHARACTER SET latin1
It successfully changed the character_set_database to "latin1" to match the other, but didn't solve the problem.
Finally, all my columns in my tables are using the Collation "latin1_swedish_ci"
Any help you could give would be very very appreciated!
Store your encrypted strings as binary (or a similar) type. Also make sure you are escaping the encrypted string. Both are important parts to doing this right!
I've been working with MySQL and Mcrypt and I store my encrypted data and initialization vectors as binary and I escape all of these strings before they get put in a query. Works like a charm.
I'm a little worried about the absence of this variable when I execute the show variables command. This is what I get when I execute show variables like 'char%':
character_set_client utf8
character_set_connection utf8
character_set_database utf8
character_set_results utf8
character_set_server utf8
character_set_system utf8
character_sets_dir /usr/share/mysql/charsets/
I wonder why this is happening. What does it mean? can I just add it to the my.cnf file?
Thank you...
Edit: Sorry, I recently noted that I didn't specify which variable we're talking about (though I said it in the title). The variable is character_set_filesystem. Thanks.
http://dev.mysql.com/doc/refman/5.1/en/server-options.html#option_mysqld_character-set-filesystem
Version Introduced 5.1.6
So most likely you have mysql < 5.1.6