For a very long time i was suffering form the Latin-1 encoding in My Django web application DB causing this error when trying to select a matching string using LIKE :
-- UnicodeEncodeError:'latin-1' codec can't encode character--
i've tried every solution from setting (charset='utf8') in the connection to applying cursor.execute("set names 'utf8'") before my actual query but nothing seams to work.
Until i came across this blog post: https://www.whitesmith.co/blog/latin1-to-utf8/
about the encoding problem and it is the same problem that i have because when i looked in phpMyAdmin a saw that by default my DB i Latin-1 encoding:
chatbot_brain=>utf8_general_ci
information_schema=>utf8_general_ci
Total: 2=> latin1_swedish_ci
So, the solution is to dump the DB and change the description in the schema file:
# Login into your future database host to create a new database with an UTF-8 charset
$ mysql -h FUTURE_HOST -u FUTURE_USER -p
mysql> CREATE DATABASE `FUTURE_DB` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
# Flush the current database schema on the future host, replacing all CHARSET=latin1 occurrences along the way
mysqldump -h CURRENT_HOST -u CURRENT_USER -p CURRENT_DB --no-data --skip-set-charset --default-character-set=latin1 \
| sed 's/CHARSET=latin1/CHARSET=utf8/g' \
| mysql -h FUTURE_HOST -u FUTURE_USER -p FUTURE_DB --default-character-set=utf8
# Flush the current database data on the future host
mysqldump -h CURRENT_HOST -u CURRENT_USER -p --no-create-db --no-create-info --skip-set-charset --default-character-set=latin1 CURRENT_DB \
| mysql -h FUTURE_HOST -u FUTURE_USER -p FUTURE_DB --default-character-set=utf8
Now i know what is the problem and the solution, But my question is how i can applied to my Django project-- Do i have to use my computer terminal and SSH session or is there any application of that?
this is a sceen shot of my DB in phpmyAdmin:
https://ibb.co/dDu7D5
Thank you
PS(I am using Django10 , Python3.5,Mysql,Webfaction sherd host)
Start with the "best practice" in Trouble with utf8 characters; what I see is not what I stored
Check for improper setup in Python: http://mysql.rjweb.org/doc.php/charcoll#python
Hopefully that 2-step mysqldump will work.
After loading, check some of the data by using "Test the data" in my first link. (SELECT HEX(col)...)
I am using MySQL 5.5.29, utf8mb4 charset, there is a table user containing a field nickname with value hex F09F988EF09F988E that translates to the emojis 😎😎.
Now open MySQL console, and execute:
set names utf8mb4;
select nickname, hex(nickname) from user;
nickname | hex(nickname)
---------+-----------------
😎😎 | F09F988EF09F988E
And then execute:
mysqldump --default-character-set=utf8 -utest -ptest test_dev user > user.sql
Check the user.sql and find the nickname display ?? which hex string is 3f
So, how can mysqldump with UTF8 export the right emojis string?
btw, the database charset envionments configured as follow:
show variables like 'character_set_%':
'character_set_client', 'utf8mb4'
'character_set_connection', 'utf8mb4'
'character_set_database', 'utf8mb4'
'character_set_filesystem', 'binary'
'character_set_results', 'utf8mb4'
'character_set_server', 'utf8mb4'
'character_set_system', 'utf8'
'character_sets_dir', '/data/mysql/share/charsets/'
Thanks Danack!
Thru specifying utf8mb4 charset and upgrading mysqldump version to 5.5.3+, mysqldump & mysql work well for 4 bytes emojis.
[tomcat#localhost ~]$ mysqldump --default-character-set=utf8mb4 -utest -ptest test_dev user > user.sql
If it shows an error like:
mysqldump: Character set 'utf8mb4' is not a compiled character set and is not specified in the '/usr/share/mysql/charsets/Index.xml' file
check your mysqldump version (mysqldump --version)
[tomcat#localhost ~]$ mysqldump --version
mysqldump Ver 10.11 Distrib 5.0.95, for redhat-linux-gnu (x86_64)
It works after upgrading mysqldump to 5.5.33.
[tomcat#localhost ~]$ mysqldump --version
mysqldump Ver 10.13 Distrib 5.5.33, for Linux (x86_64)
It's true that you need to use mysqldump --default-character-set=utf8mb4 (notice the --default-character-set option) when exporting.
But then importing is still super tricky. I tried so many different approaches and had no success.
Finally, I discovered that you need to create an importer.sql file like this:
USE my_example_db_name;
# Select the right charset
SET NAMES 'utf8mb4';
# Import from SQL file
SOURCE /somewhere/dump.sql;
# Disconnect from SQL server
EXIT
Then, to import, run this:
mysql -u my_user my_example_db_name < /somewhere/importer.sql
Thank you to https://korobochkin.wordpress.com/2017/02/25/import-and-export-wordpress-database-with-utf8mb4-charset/
Was struggling with this for a while as well. The other solutions in this thread still caused dump.sql to still have multiple wrong characters for emojis.
Turns out, using > is not a safe way of exporting (at least not on my machine, Windows 10). Using -r dump.sql instead of > dump.sql did the trick.
This command exports all the tables of the database:
mysqldump --default-character-set=utf8mb4 -h [host] -u [username] -p [database_name] --set-gtid-purged=OFF --port=3306 --protocol=tcp --skip-triggers -r dump.sql
Then for importing, first run this command:
mysql -h [host_name] -u [username] -p [database_name] --binary-mode -o
And then, when in mysql mode, type the following:
USE database_name;
SET NAMES 'utf8mb4';
SOURCE /dump.sql;
EXIT
Hope this helps others that had the same issue!
I have a .sql file with an export from phpMyAdmin. I want to import it into a different server using the command line.
I have a Windows Server 2008 R2 installation. I placed the .sql file on the C drive, and I tried this command
database_name < file.sql
It is not working. I get syntax errors.
How can I import this file without a problem?
Do I need to create a database first?
Try:
mysql -u username -p database_name < file.sql
Check MySQL Options.
Note 1: It is better to use the full path of the SQL file file.sql.
Note 2: Use -R and --triggers with mysqldump to keep the routines and triggers of the original database. They are not copied by default.
Note 3 You may have to create the (empty) database from MySQL if it doesn't exist already and the exported SQL doesn't contain CREATE DATABASE (exported with --no-create-db or -n option) before you can import it.
A common use of mysqldump is for making a backup of an entire database:
mysqldump db_name > backup-file.sql
You can load the dump file back into the server like this:
Unix
mysql db_name < backup-file.sql
The same in the Windows command prompt:
mysql -p -u [user] [database] < backup-file.sql
PowerShell
cmd.exe /c "mysql -u root -p db_name < backup-file.sql"
MySQL command line
mysql> use db_name;
mysql> source backup-file.sql;
Regarding the time taken for importing huge files: most importantly, it takes more time because the default setting of MySQL is autocommit = true. You must set that off before importing your file and then check how import works like a gem.
You just need to do the following thing:
mysql> use db_name;
mysql> SET autocommit=0 ; source the_sql_file.sql ; COMMIT ;
Among all the answers, for the problem above, this is the best one:
mysql> use db_name;
mysql> source file_name.sql;
Easiest way to import into your schema:
Login to mysql and issue below mention commands.
mysql> use your_db_name;
mysql> source /opt/file.sql;
We can use this command to import SQL from the command line:
mysql -u username -p password db_name < file.sql
For example, if the username is root and password is password. And you have a database name as bank and the SQL file is bank.sql. Then, simply do like this:
mysql -u root -p password bank < bank.sql
Remember where your SQL file is. If your SQL file is in the Desktop folder/directory then go the desktop directory and enter the command like this:
cd ~/Desktop
mysql -u root -p password bank < bank.sql
And if you are in the Project directory and your SQL file is in the Desktop directory. If you want to access it from the Project directory then you can do like this:
cd ~/Project
mysql -u root -p password bank < ~/Desktop/bank.sql
If you already have the database, use the following to import the dump or the sql file:
mysql -u username -p database_name < file.sql
if you don't you need to create the relevant database(empty) in MySQL, for that first log on to the MySQL console by running the following command in terminal or in cmd
mysql -u userName -p;
And when prompted provide the password.
Next, create a database and use it:
mysql>create database yourDatabaseName;
mysql>use yourDatabaseName;
Then import the sql or the dump file to the database from
mysql> source pathToYourSQLFile;
Note: if your terminal is not in the location where the dump or sql file exists, use the relative path in above.
Open the MySQL command line
Type the path of your mysql bin directory and press Enter
Paste your SQL file inside the bin folder of mysql server.
Create a database in MySQL.
Use that particular database where you want to import the SQL file.
Type source databasefilename.sql and Enter
Your SQL file upload successfully.
A solution that worked for me is below:
Use your_database_name;
SOURCE path_to_db_sql_file_on_your_local;
While most answers here just mention the simple command
mysql -u database_user -p [db_name] < database_file.sql
today it's quite common that databases and tables have utf8-collation where this command is not sufficient.
Having utf8-collation in the exported tables it's required to use this command:
mysql -u database_user -p --default-character-set=utf8 [db_name] < database_file.sql
An according export can be done with
mysqldump -u database_user -p --default-character-set=utf8 [db_name] > database_file.sql
Surely this works for other charsets too, how to show the right notation can be seen here:
https://dev.mysql.com/doc/refman/5.7/en/show-collation.html
One comment mentioned also that if a database never exists an empty database had to be created first. This might be right in some cases but depends on the export file. If the exported file includes already the command to create the database then the database never has to be created in a separate step, which even could cause an error on import. So on import, it's advisable to have a look first in the file to know which commands are included there, on export, it's advisable to note the settings, especially if the file is very large and hard to read in an editor.
There are still more parameters for the command which are listed and explained here:
https://dev.mysql.com/doc/refman/5.7/en/mysql-command-options.html
If you use another database version consider searching for the corresponding version of the manual too. The mentioned links refer to MySQL version 5.7.
EDIT:
The same parameters are working for mysqldump too. So while the commands for export and import are different, the mentioned parameters are not.
Nevertheless there exists a special site in the manual that describes the options for mysqldump: https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html
To dump a database into an SQL file use the following command.
mysqldump -u username -p database_name > database_name.sql
To import an SQL file into a database (make sure you are in the same directory as the SQL file or supply the full path to the file), do:
mysql -u username -p database_name < database_name.sql
I think it's worth mentioning that you can also load a gzipped (compressed) file with zcat like shown below:
zcat database_file.sql.gz | mysql -u username -p -h localhost database_name
Go to the directory where you have the MySQL executable. -u for username and -p to prompt for the password:
C:\xampp\mysql\bin>mysql -u username -ppassword databasename < C:\file.sql
To import a single database, use the following command.
mysql -u username -p password dbname < dump.sql
To import multiple database dumps, use the following command.
mysql -u username -p password < dump.sql
To import a database, use the following command.
mysql> create new_database;
mysql> use new_database;
mysql> source (Here you need to import the path of the SQL file);
E.g.:
mysql> source E:/test/dump.sql;
You need to use forward slashes (/) even on Windows, e.g., E:/test/dump.sql instead of E:\test\dump.sql
Or double backslashes (\\) because of escaping, i.e., E:\\test\\dump.sql
mysql --user=[user] --password=[password] [database] < news_ml_all.sql
I kept running into the problem where the database wasn't created.
I fixed it like this:
mysql -u root -e "CREATE DATABASE db_name"
mysql db_name --force < import_script.sql
For exporting a database:
mysqldump -u username -p database_name > file.sql
For importing a database:
mysql -u username -p database_name < file.sql
For importing multiple SQL files at one time, use this:
# Unix-based solution
for i in *.sql ; do mysql -u root -pPassword DataBase < $i ; done
For simple importing:
# Unix-based solution
mysql -u root -pPassword DataBase < data.sql
For WAMP:
REM mysqlVersion - replace with your own version
C:\wamp\bin\mysql\mysqlVersion\bin\mysql.exe -u root -pPassword DataBase < data.sql
For XAMPP:
C:\xampp\mysql\bin\mysql -u root -pPassword DataBase < data.sql
You do not need to specify the name of the database on the command line if the .sql file contains CREATE DATABASE IF NOT EXISTS db_name and USE db_name statements.
Just make sure you are connecting with a user that has the permissions to create the database, if the database mentioned in the .sql file does not exist.
Import a database
Go to drive:
d:
MySQL login
c:\xampp\mysql\bin\mysql -u root -p
It will ask for pwd. Enter it:
pwd
Select the database
use DbName;
Provide the file name
\.DbName.sql
Use:
mysql -u root -p password -D database_name << import.sql
Use the MySQL help for details - mysql --help.
I think these will be useful options in our context:
[~]$ mysql --help
mysql Ver 14.14 Distrib 5.7.20, for osx10.12 (x86_64) using EditLine wrapper
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Usage: mysql [OPTIONS] [database]
-?, --help Display this help and exit.
-I, --help Synonym for -?
--bind-address=name IP address to bind to.
-D, --database=name Database to use.
--delimiter=name Delimiter to be used.
--default-character-set=name Set the default character set.
-f, --force Continue even if we get an SQL error.
-p, --password[=name] Password to use when connecting to server.
-h, --host=name Connect to host.
-P, --port=# Port number to use for connection or 0 for default to, in order of preference, my.cnf, $MYSQL_TCP_PORT, /etc/services, built-in default (3306).
--protocol=name The protocol to use for connection (tcp, socket, pipe,
-s, --silent Be more silent. Print results with a tab as separator, each row on new line.
-v, --verbose Write more. (-v -v -v gives the table output format).
-V, --version Output version information and exit.
-w, --wait Wait and retry if connection is down.
What is fun, if we are importing a large database and not having a progress bar. Use Pipe Viewer and see the data transfer through the pipe
For Mac, brew install pv
For Debian/Ubuntu, apt-get install pv.
For others, refer to pv - Pipe Viewer
pv import.sql | mysql -u root -p password -D database_name
1.45GiB 1:50:07 [339.0KiB/s] [=============> ] 14% ETA 11:09:36
1.46GiB 1:50:14 [ 246KiB/s] [=============> ] 14% ETA 11:09:15
1.47GiB 1:53:00 [ 385KiB/s] [=============> ] 14% ETA 11:05:36
Go to the directory where you have MySQL.
c:\mysql\bin\> mysql -u username -p password database_name <
filename.sql
Also to dump all databases, use the -all-databases option, and no databases’ name needs to be specified anymore.
mysqldump -u username -ppassword –all-databases > dump.sql
Or you can use some GUI clients like SQLyog to do this.
You can try this query.
Export:
mysqldump -u username –-password=your_password database_name > file.sql
Import:
mysql -u username –-password=your_password database_name < file.sql
and detail following this link:
https://chartio.com/resources/tutorials/importing-from-and-exporting-to-files-using-the-mysql-command-line/
Add the --force option:
mysql -u username -p database_name --force < file.sql
The following command works for me from the command line (cmd) on
Windows 7 on WAMP.
d:/wamp/bin/mysql/mysql5.6.17/bin/mysql.exe -u root -p db_name < database.sql
Providing credentials on the command line is not a good idea. The above answers are great, but neglect to mention
mysql --defaults-extra-file=etc/myhost.cnf database_name < file.sql
Where etc/myhost.cnf is a file that contains host, user, password, and you avoid exposing the password on the command line. Here is a sample,
[client]
host=hostname.domainname
user=dbusername
password=dbpassword
Import into the database:
mysql -u username -p database_name < /file path/file_name.sql
Export from the database:
mysqldump -u username -p database_name > /file path/file_name.sql
After these commands, a prompt will ask for your MySQL password.
Similarly to vladkras's answer to How do import an SQL file using the command line in MySQL?.
Key differences for me:
The database has to exist first
No space between -p and the password
shell> mysql -u root -ppassword #note: no space between -p and password
mysql> CREATE DATABASE databasename;
mysql> using databasename;
mysql> source /path/to/backup.sql
I am running Fedora 26 with MariaDB.
I thought it could be useful for those who are using Mac OS X:
/Applications/xampp/xamppfiles/bin/mysql -u root -p database < database.sql
Replace xampp with mamp or other web servers.