Mission: Migrate data from MySQL to PostgreSQL using pgloader. Pgloader is installed using this guide
Context: I have installed pgloader in the machine where MySQL server exists, and I have also a VM Linux that runs postgres.
The MySQL server is accessed from MySQL Workbench. Apparently, the MySQL user does not have privileges to create tables in that MySQL server.
So what I would like to do is to execute the command below:
pgloader mysql://mysql_username:password#mysql_server_ip_/source_database_name postgresql://postgresql_role_name:password#postgresql_server_ip/target_database_name
and load a sample of the MySQL table to Postgres table like a SQL query:
select *
from source_database_name.table1
where year = 2021
limit 1000;
Is that feasible with pgloader or should I migrate the whole MySQL table which probably is 100gb.
Temporary approach
Load the data using a csv file.
Basically, I executed the SQL query to yield 1000 sample rows from a MySQL table.
Then on the same directory with the csv file I have created a csv_file.load file. The file includes the following commands
LOAD CSV
FROM './csv_file_with_data.csv'
INTO postgresql://<my_user>:<my_pass>#<public_ip>:5432/<db_name>?table1_1000
WITH truncate,
skip header = 1,
fields optionally enclosed by '"',
fields escaped by double-quote,
fields terminated by ','
SET client_encoding to 'latin1',
standard_conforming_strings to 'on'
BEFORE LOAD DO
$$ drop table if exists public.table1_1000; $$,
$$ CREATE TABLE IF NOT EXISTS public.table1_1000(
a varchar(4) DEFAULT NULL,
b varchar(20) DEFAULT NULL,
c varchar(15) DEFAULT NULL,
d int DEFAULT NULL,
...
);
$$;
Executed the file like pgloader csv_file.load
I just started using MySQL Workbench and I'm experiencing multiple issues while Synchronizing.
I modelled the schema using Workbench and Forward Engineered it and ran it successfully in the database server. Later edited a Trigger and tried to synchronise it with the database server
So I went to Database -> Synchronize Model...
I have only edited one trigger(just added +1 to an already existing variable in the Trigger) but as you can see it shows that all the tables must be changed/updated(with a little yellow exclamation mark). It shouldn't be right?
And when I click continue, I noticed in the generated script that it is deleting foreign keys and again adding them.
And when I clicked on Execute, it is showing an error:
ERROR: Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'Default Collation' at line 1
Also, it adds DEFAULT COLLATE Default Collation for all ALTER statements for schemas and tables, which is causing error 1064
ALTER SCHEMA `myschema` DEFAULT COLLATE Default Collation ;
.....
.....
.....
ALTER TABLE `table1`
COLLATE = Default Collation ;
ALTER TABLE `table2`
COLLATE = Default Collation ;
ALTER TABLE `table3`
COLLATE = Default Collation ;
....
....
....
....
MySQL Community Edition: 8.0.19
MySQL Workbench: 8.0.19
macOS: Catalina 10.15.3
I've researched on other SO posts about error: 1064 and many suggested to change the version. So, I've also changed the MySQL version in Workbench Model -> Model Options... -> MySQL -> Target MySQL Version: 8.0.19 but still the error exists.
Am I missing something or doing it wrong?
I tried (the other way around) to synchronize model(as destination) with the database server( as source) to make sure at least the database server is as expected. So, when I did that, all the foreign keys are deleted in the EER Diagram.
I checked on the database server are there even foreign keys present using this post and everything is fine on the server.
I don't know whether it's supposed to do like this or just a hack but the thing is, I got what I want for now atleast.
I used trail and error. So, I don't know why or how but it works.
Open the workbench, get the schema from the database server, edit the schema in your workbench and synchronize. This way even if someone changed on the server, you get the latest version of the schema.
Go to Database -> Reverse Engineer. It creates EER diagram from your database server schema.
Once you get the EER diagram generated, try to edit your changes and go to Database -> Synchronize Model... Now you only have the yellow exclamation mark before a table name which you have modified(instead of before all the tables as you said)
Also when I did this, there are no DEFAULT COLLATE Default Collation lines or foreign keys deletion and addition lines in the final script. The final script only contained which is required.
I hope it works for you too.
Being a designer, I don't understand MySQL and have no ability to alter the database. Go easy!
CREATING THE ERROR
Exported a Concrete5 db (modified to run on php 7) from a server running MySQL 5.7.27
Imported the db into MAMP Pro 5.5.1 running MySQL 5.7.26 for local development
Import fails on time-related rows like this one...
CREATE TABLE `atDateTime` (
`avID` int(10) unsigned NOT NULL,
`value` datetime DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`avID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
ANSWERS FOUND
1. MYSQL 5.7+
Many answers say the error is because MySQL 5.7+ no longer accepts blank dates but the db is being exported from a server already running MySQL 5.7.27?
2. TIMEZONE ISSUE
Other answers suggest it's a Timezone difference but both the server and local Mac are running NZDT. MAMP Pro seems to use the system timezone.
3. QUERIOUS/SEQUAL PRO
Multiple db apps were tried incase issue was application-specific but both show the same errors.
QUESTION
Can anyone explain what the exact issue is (to a designer) and how to fix without altering the db?
MySQL 5.7 doesn't allow blank dates by default.
The db doesn't work in MAMP Pro because MySQL strict mode is enabled by default.
The db works on the live server because MySQL strict mode has been disabled.
The solution is to disable strict mode in MAMP Pro.
Disable MySQL strict mode and allow blank dates in MAMP Pro:
Shut down the MAMP Pro servers
Go File > Edit Templates> MySQL (my.conf) > MySQL Version you're using
In the first third of the file, look for (with square brackets)...
[mysqld]
Directly underneath, add sql_mode="" like this...
[mysqld]
sql_mode=""
This disables strict mode and allows blank dates
Save the file and close
Start the servers
Import your db
If you don't know the correct search terms; 'disable MySQL strict mode in MAMP Pro', you're left wandering in the wilderness for hours. Hope this helps other designers.
More reference: https://mampsupportforum.com/forums/latest/mamp-mamp-pro-disable-mysql-strict-mode
I have a WordPress website on my local WAMP server. But when I upload its database to live server, I get error
#1273 – Unknown collation: ‘utf8mb4_unicode_520_ci’
Any help would be appreciated!
I believe this error is caused because the local server and live server are running different versions of MySQL. To solve this:
Open the sql file in your text editor
Find and replace all utf8mb4_unicode_520_ci with utf8mb4_unicode_ci
Save and upload to a fresh mySql db
You can solve this by finding
ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci;
in your .sql file, and swapping it with
ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;
In my case it turns out my
new server was running MySQL 5.5,
old server was running MySQL 5.6.
So I got this error when trying to import the .sql file I'd exported from my old server.
MySQL 5.5 does not support utf8mb4_unicode_520_ci, but
MySQL 5.6 does.
Updating to MySQL 5.6 on the new server solved collation the error !
If you want to retain MySQL 5.5, you can:
- make a copy of your exported .sql file
- replace instances of utf8mb4unicode520_ci and utf8mb4_unicode_520_ci
...with utf8mb4_unicode_ci
- import your updated .sql file.
Open the sql file in your text editor;
1. Search: utf8mb4_unicode_ci Replace: utf8_general_ci (Replace All)
2. Search: utf8mb4_unicode_520_ci Replace: utf8_general_ci (Replace All)
3. Search: utf8mb4 Replace: utf8 (Replace All)
Save and upload!
I experienced a challenge importing data into mysql exported using mysql workbench. It is a collation issue.
I solved this error by:
Opening the .sql file using text editor
Replacing "utf8mb4_0900_ai_ci" with "utf8mb4_general_ci".
Saving the file as .sql and importing it
It worked
easy replace
sed -i 's/utf8mb4_unicode_520_ci/utf8mb4_unicode_ci/g' your_sql_file.sql
just remove "520_"
utf8mb4_unicode_520_ci → utf8mb4_unicode_ci
I just opened the dump.sql file in Notepad++ and hit CTRL+H to find and replace the string "utf8mb4_0900_ai_ci" and replaced it with "utf8mb4_general_ci". Source link https://www.freakyjolly.com/resolved-when-i-faced-1273-unknown-collation-utf8mb4_0900_ai_ci-error/
this error is caused because the conflict of different versions of MySQL. To solve this:
Open the sql file in any text editor
Find and replace all utf8mb4_0900_ai_ci with utf8mb4_unicode_ci
Save and try new mySql db file
after that try again,it works fine for me
find and replace:
utf8mb4_unicode_520_ci
with
utf8_general_ci
in whole sql file
Getting collation error #1273 - Unknown collation: 'utf8mb4_unicode_520_ci' is caused by the difference of the MySQL version from which you export and our MySQL server to which you import. Basically, the Wordpress library for newer version checks to see what version of SQL your site is running on. If it uses MySQL version 5.6 or more, it assumes the use of a new and improved Unicode Collation Algorithm (UCA) called “utf8mb4_unicode_520_ci”. This is great unless you end up moving your WordPress site from a newer 5.6 version of MySQL to an older, pre 5.6 version of MySQL.
To resolve this you will either have to edit your SQL export file and do a search and replace, changing all instances of ‘utf8mb4_unicode_520_ci’ to ‘utf8mb4_unicode_ci’. Or follow the steps below if you have a PHPMyAdmin:
Click the Export tab for the database
Click the Custom radio button.
Go the section titled Format-specific options and change the drop-down for Database system or older MySQL server to maximize output compatibility with: from NONE to MYSQL40.
Scroll to the bottom and click GO.
I solved it this way, I opened the .sql file in a Notepad and clicked CTRL + H to find and replace the string "utf8mb4_0900_ai_ci" and replaced it with "utf8mb4_general_ci".
In my case I substitute it with utf8_general_ci with sed like this:
sed -i 's/utf8mb4_0900_ai_ci/utf8_general_ci/g' MY_DB.sql
sed -i 's/utf8mb4_unicode_520_ci/utf8_general_ci/g' MY_DB.sql
After that, I can import it without any issue.
Late to the party, but in case this happens with a WORDPRESS installation :
#1273 - Unknown collation: 'utf8mb4_unicode_520_ci
In phpmyadmin, under export method > Format-specific options( custom export )
Set to : MYSQL40
If you will try to import now, you now might get another error message :
1064 - You have an error in your SQL syntax; .....
That is because The older TYPE option that was synonymous with ENGINE was removed in MySQL 5.5.
Open your .sql file , search and replace all instances
from TYPE= to ENGINE=
Now the import should go smoothly.
1273 - Unknown collation: 'utf8mb4_0900_ai_ci'
in my case I was unable to import DB using
ENGINE = InnoDB AUTO_INCREMENT = 1 DEFAULT CHARSET = utf8 COLLATE = utf8_general_ci;
and
ENGINE = InnoDB AUTO_INCREMENT = 1 DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci;
both. But changing it to this in .SQL File resolved the problem
ENGINE=InnoDB DEFAULT CHARSET=latin1;
UPDATED
using 'utf8mb4_general_ci'resolved the problem
ENGINE = InnoDB AUTO_INCREMENT = 1 DEFAULT CHARSET = utf8mb4 COLLATE = utf8mb4_general_ci;
Very strange that all answers recommend replacing collation. Which is a very bad practice because you want to use the same MySQL version as the one in development and the one in production. Therefore, your local mysql server should be the same.
First of all, Execute the query SHOW COLLATION to check all the collations your server supports. If you're using xampp or any other similar tool to start your server, it might come shipped with maria db server instead of mysql server.
What you should do is replace your current mysql (which is really mariadb) by the real mysql one.
So what you should do is simply replace your maria db server by mysql server.
After a little investigation, I found that the MySQL server running on the destination is an older version than the source. So we got that the destination server doesn’t contain the required database collation.
Then we do a little tweak in the backup file to resolve this. Edit the database backup file(your_sql_file.sql) in a text editor and replace utf8mb4_0900_ai_ci with utf8mb4_general_ci and CHARSET=utf8mb4 with CHARSET=utf8.
I hope this solution might help you.
Use the sed command to replace text in files directly
Linux OS
sed -i 's/utf8mb4_unicode_520_ci/utf8mb4_general_ci/g' YOUR_SQL_FILE.sql
Mac OS
sed -i '' s/utf8mb4_unicode_520_ci/utf8mb4_general_ci/g' YOUR_SQL_FILE.sql
The help of this command i have fixed issue ERROR 1273 (HY000) at line 51: Unknown collation: 'utf8mb4_0900_ai_ci'
According to my experience, the destination's MySQL server is an older version than the source. The required database collation is not present on the destination server.
To fix this, we can make a small change to the backup file. Replace "utf8mb4 0900 ai ci" with "utf8mb4 general ci" and "CHARSET=utf8mb4" with "CHARSET=utf8" in the database backup file.
Replace the below string:
ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;
with:
ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;
Save your file and restore the database.
I simply removed the COLLATE and other attributes and left only till ENGINE.
like the following
FROM:
ENGINE=InnoDB AUTO_INCREMENT=429 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
TO:
ENGINE=InnoDB;
and it worked for me just fine.
I am trying to backup my MySql databases on Windows via a batch file. It works fine.
but i want to remove the default use "database" and create "database" command.from that .sql file which is by default created while creating backup.
these are the lines i want to remove before taking backup for each .sql file
CREATE DATABASE /*!32312 IF NOT EXISTS*/ `database name` /*!40100 DEFAULT CHARACTER SET utf8 */;
USE database name;
Ya I got the answer from mysql official site.
I have used '--databases' in my batch file. to backup all databases.present in server.
just removing that and now it works fine.
--no-create-db
Do not write CREATE DATABASE statements
https://dev.mysql.com/doc/refman/5.6/en/mysqldump.html#option_mysqldump_create-options