For a very long time i was suffering form the Latin-1 encoding in My Django web application DB causing this error when trying to select a matching string using LIKE :
-- UnicodeEncodeError:'latin-1' codec can't encode character--
i've tried every solution from setting (charset='utf8') in the connection to applying cursor.execute("set names 'utf8'") before my actual query but nothing seams to work.
Until i came across this blog post: https://www.whitesmith.co/blog/latin1-to-utf8/
about the encoding problem and it is the same problem that i have because when i looked in phpMyAdmin a saw that by default my DB i Latin-1 encoding:
chatbot_brain=>utf8_general_ci
information_schema=>utf8_general_ci
Total: 2=> latin1_swedish_ci
So, the solution is to dump the DB and change the description in the schema file:
# Login into your future database host to create a new database with an UTF-8 charset
$ mysql -h FUTURE_HOST -u FUTURE_USER -p
mysql> CREATE DATABASE `FUTURE_DB` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
# Flush the current database schema on the future host, replacing all CHARSET=latin1 occurrences along the way
mysqldump -h CURRENT_HOST -u CURRENT_USER -p CURRENT_DB --no-data --skip-set-charset --default-character-set=latin1 \
| sed 's/CHARSET=latin1/CHARSET=utf8/g' \
| mysql -h FUTURE_HOST -u FUTURE_USER -p FUTURE_DB --default-character-set=utf8
# Flush the current database data on the future host
mysqldump -h CURRENT_HOST -u CURRENT_USER -p --no-create-db --no-create-info --skip-set-charset --default-character-set=latin1 CURRENT_DB \
| mysql -h FUTURE_HOST -u FUTURE_USER -p FUTURE_DB --default-character-set=utf8
Now i know what is the problem and the solution, But my question is how i can applied to my Django project-- Do i have to use my computer terminal and SSH session or is there any application of that?
this is a sceen shot of my DB in phpmyAdmin:
https://ibb.co/dDu7D5
Thank you
PS(I am using Django10 , Python3.5,Mysql,Webfaction sherd host)
Start with the "best practice" in Trouble with utf8 characters; what I see is not what I stored
Check for improper setup in Python: http://mysql.rjweb.org/doc.php/charcoll#python
Hopefully that 2-step mysqldump will work.
After loading, check some of the data by using "Test the data" in my first link. (SELECT HEX(col)...)
Related
I am trying to create a dump file with mysqldump.exe
I tried it like this:
C:\Program Files\MySQL\MySQL Workbench 8.0 CE>mysqldump --column-statistics=0 --single-transaction -p3306 -h10.10.10.10 -u username -p dbName > backup.sql
mysqldump: [Warning] Using a password on the command line interface can be insecure.
Enter password: ****************
mysqldump: Couldn't execute 'SHOW CREATE TABLE `PLZ_Stra├ƒen`': Table 'dbName.PLZ_Stra├â┼©en' doesn't exist (1146)
Because of the special character, the dump process fails, How can I solve this problem?
try to specify the character set with --default-character-set=utf8mb4 option when using mysqldump
mysqldump --default-character-set=utf8mb4 --column-statistics=0 --single-transaction -p3306 -h10.10.10.10 -u username -p dbName > backup.sql
or you can set any character set in mysql using that
ß, when interpreted as CHARACTER SET cp850 is hex c39f.
ß, which is presumably the character you wanted, when interpreted as CHARACTER SET utf8 (or utf8mb4) is hex c39f.
One hand is talking cp850; the other hand is utf8. Be consistent.
You seem to be using the Windows cmd. The command chcp 65001 provides utf8, but it needs a special charset installed.
I am trying to take mysql dump with command:
mysqldump -u xxxx -p dbxxx > xxxx270613.sql
what is command to take mysqldump with UTF8 ?
Hi please try the following.
mysqldump -u [username] –p[password] --default-character-set=utf8 -N --routines --skip-triggers --databases [database_name] > [dump_file.sql]
I had the problem, that even with applied utf-8 flags when creating the dump I could not avoid broken characters importing a dump that was created from a DB with many text columns using latin1.
Some googling and especially this site helped me to finally figure it out.
mysqldump with --skip-set-charset --default-character-set=latin1 flags,
to avoid MySQL attempt of reconversion and setting a charset.
fix the dump by replacing the charset strings using sed on terminal
sed -i 's/latin1_swedish_ci/utf8mb4/g' mysqlfile.sql
sed -i 's/latin1/utf8mb4/g' mysqlfile.sql
to make sure you don't miss anything you can do grep -i 'latin1' mysqlfile.sql before step 2 - and then come up with more sed orders. Introduction to sed here
create a clean DB
CREATE DATABASE mydatabase CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci;
apply fixed dump
--default-character-set=utf8 is the option you are looking for the one can be used together with these others:
mysqldump --events \
--routines \
--triggers \
--add-drop-database \
--compress \
--hex-blob \
--opt \
--skip-comments \
--single-transaction \
--skip-set-charset \
--default-character-set=utf8 \
--databases dbname > my.dump
Also, check the --hex-blob it helps to dump binary strings in hexadecimal format, so I can guaranty (be more portable) making the import to work.
The --databases option causes all names on the command line to be treated as database names. Without this option, mysqldump treats the first name as a database name and those following as table names.
With --all-databases or --databases, mysqldump writes CREATE DATABASE and USE statements prior to the dump output for each database. This ensures that when the dump file is reloaded, it creates each database if it does not exist and makes it the default database so database contents are loaded into the same database from which they came. If you want to cause the dump file to force a drop of each database before recreating it, use the --add-drop-database option as well. In this case, mysqldump writes a DROP DATABASE statement preceding each CREATE DATABASE statement.
This helps to restore using:
# mysql < dump.sql
Instead of:
# mysql dbname < dump.sql
I am using MySQL 5.5.29, utf8mb4 charset, there is a table user containing a field nickname with value hex F09F988EF09F988E that translates to the emojis 😎😎.
Now open MySQL console, and execute:
set names utf8mb4;
select nickname, hex(nickname) from user;
nickname | hex(nickname)
---------+-----------------
😎😎 | F09F988EF09F988E
And then execute:
mysqldump --default-character-set=utf8 -utest -ptest test_dev user > user.sql
Check the user.sql and find the nickname display ?? which hex string is 3f
So, how can mysqldump with UTF8 export the right emojis string?
btw, the database charset envionments configured as follow:
show variables like 'character_set_%':
'character_set_client', 'utf8mb4'
'character_set_connection', 'utf8mb4'
'character_set_database', 'utf8mb4'
'character_set_filesystem', 'binary'
'character_set_results', 'utf8mb4'
'character_set_server', 'utf8mb4'
'character_set_system', 'utf8'
'character_sets_dir', '/data/mysql/share/charsets/'
Thanks Danack!
Thru specifying utf8mb4 charset and upgrading mysqldump version to 5.5.3+, mysqldump & mysql work well for 4 bytes emojis.
[tomcat#localhost ~]$ mysqldump --default-character-set=utf8mb4 -utest -ptest test_dev user > user.sql
If it shows an error like:
mysqldump: Character set 'utf8mb4' is not a compiled character set and is not specified in the '/usr/share/mysql/charsets/Index.xml' file
check your mysqldump version (mysqldump --version)
[tomcat#localhost ~]$ mysqldump --version
mysqldump Ver 10.11 Distrib 5.0.95, for redhat-linux-gnu (x86_64)
It works after upgrading mysqldump to 5.5.33.
[tomcat#localhost ~]$ mysqldump --version
mysqldump Ver 10.13 Distrib 5.5.33, for Linux (x86_64)
It's true that you need to use mysqldump --default-character-set=utf8mb4 (notice the --default-character-set option) when exporting.
But then importing is still super tricky. I tried so many different approaches and had no success.
Finally, I discovered that you need to create an importer.sql file like this:
USE my_example_db_name;
# Select the right charset
SET NAMES 'utf8mb4';
# Import from SQL file
SOURCE /somewhere/dump.sql;
# Disconnect from SQL server
EXIT
Then, to import, run this:
mysql -u my_user my_example_db_name < /somewhere/importer.sql
Thank you to https://korobochkin.wordpress.com/2017/02/25/import-and-export-wordpress-database-with-utf8mb4-charset/
Was struggling with this for a while as well. The other solutions in this thread still caused dump.sql to still have multiple wrong characters for emojis.
Turns out, using > is not a safe way of exporting (at least not on my machine, Windows 10). Using -r dump.sql instead of > dump.sql did the trick.
This command exports all the tables of the database:
mysqldump --default-character-set=utf8mb4 -h [host] -u [username] -p [database_name] --set-gtid-purged=OFF --port=3306 --protocol=tcp --skip-triggers -r dump.sql
Then for importing, first run this command:
mysql -h [host_name] -u [username] -p [database_name] --binary-mode -o
And then, when in mysql mode, type the following:
USE database_name;
SET NAMES 'utf8mb4';
SOURCE /dump.sql;
EXIT
Hope this helps others that had the same issue!
I back up my production database with the following command:
mysqldump -u root --opt --skip-extended-insert --databases my_production_db
The resulting dump file has the following lines near the top:
CREATE DATABASE /*!32312 IF NOT EXISTS*/ `my_production_db` /*!40100 DEFAULT CHARACTER SET latin1 */;
USE `my_production_db `;
In order to restore the database to a different destination ie. my_debvelopment_db I have to open the dump file and edit the bits where the database is named.
Then I run:
mysql -u root -p <password> < mydumpfile
I have not figured out another way to do it.
As the database gets bigger this becomes impractical.
Am I missing something? Cant I somehow specify where I want to restore the database? Would I need a different backup command?
#minaz answer was good, but I want to append a little bit more.
The problem was caused by --databases keyword. If you omit the keyword, it will not contain any database creation contents.
So, Dump without --databases keyword.
mysqldump -u username -p database_name > dump.sql
And restore it with the target database name.
mysql -u username -p target_database_name < dump.sql
Also, there are several ways to do this. See the similar problem on here (dba.stackexchange).
If you drop the option --databases but still specify the database name, you will NOT get the create database statements. ie:
mysqldump -u root --opt --skip-extended-insert my_production_db
On your dev machine simply create any database you wish to restore to.
If you already have your dump you can strip the commands for creating and using the database. Simply remove the fourth and the new fifth line.
sed '4d' dump.sql | sed '5d' > dump-striped.sql
On windows xampp I used following commands to achieve this
Export
mysqldump -u root -p mydb > mydb.sql
Import
mysql -u root -p mynewdb < mydb.sql
I want to copy a mysql database from my local computer to a remote server.
I am trying to use the mysql dump command. All the examples on the internet suggest doing something like
The initial mysql> is just the prompt I get after logging in.
mysql> mysqldump -u user -p pass myDBName | NewDBName.out;
But when I do this I get You have an error in your SQL syntax; check the manual that corresponds ... to use near 'mysqldump -u user -p pass myDBName | NewDBName.out'
Since I have already logged in do I need to use -u and -p? Not doing so gives me the same error. Can you see what is wrong?
In addition to what Alexandre said, you probably don't want to pipe (|) output to NewDBName.out, but rather redirect it there (>).
So from the Windows/Unix command line:
mysqldump -u user -p pass myDBName > NewDBName.out
Note that if you have large binary fields (e.g. BLOBS) in some columns you may need to set an additional option (I think it was --hex-blob, but there might have been another option too). If that applies to you, add a comment and I'll research the setting.
mysqldump is not an SQL statement that you execute inside a mysql session but a distinct binary that should be started from your OS shell.
The are a few ways to use this. One of them is to pipe the output of mysqldump to another MySQL instance:
echo CREATE DATABASE remote_db | mysql -h remote_host -u remote_user -premote_password
mysqldump -h source_host -u root -ppassword source_db | mysql -h remote_host -u remote_user -premote_password -D remote_db
I have had to dump large sets of data recently. From what I have found on a 200Mb database with 10,000+ records in many of the tables is the following. I used the linux 'time' command to get actual time.
12 minutes using:
mysqldump -u user -p pass myDBName > db-backups.sql
7 minutes to clone the database:
mysqldump -u user -p pass myDBName | mysql -u user -p pass cloneDBName
And in less than a second:
mysqlhotcopy -u user -p pass myDBName cloneDBName
The last one blew my mind, but you have to be logged in locally where the database server resides. Personally I think this is much faster than remotely doing a dump, the you can compress the .sql file and transfer it manually.