I have a database hosted on Amazon RDS. I am developing a Rails application with primary database as PostgreSQL.
The Postgres database needs to be updated with data from RDS. I need to pull records from MySQL and then load it into Postgres.
Also, this is not database migrations.
One option is to create a secondary database (MySQL) in my rails application. But this does not seem a good option as I need to transfer data only when needed and not very frequently.
What is the best way to achieve this? Can this be done through pgloader?
UPDATE
This solved my issue. I just needed to read from MySQL and this is working fine. Are there any drawbacks for this approach?
You can use PostgreSQL's Foreign Data Wrappers : your MySQL tables will be viewed as PostgreSQL tables.
See : https://www.postgresql.org/docs/current/static/postgres-fdw.html
Related
I have SonarQube MySQL DB (450GB) running and I want to Migrate to PostgreSQL DB.
I tried to migrate using MySQL-migrator (https://github.com/SonarSource/mysql-migrator), it takes long time and fails, we cannot RESUME as well. What is the simple and best way to Migrate from MySQL to PostgrSQl?
500 GB is not large these days.
You should first try to get the schema definition right. Try to migrate without the data, that will be fast, and you can debug the migration of indexes, constraints, views and stored code. Once you have that, try to migrate the data (but of course you migrate the data before creating constraints and indexes).
I have no experience with the tools available, but you could investigate mysql_fdw. A foreign data wrapper has the advantage that you can migrate data in one step, without the need for intermediary storage. Also, it is easy to migrate several tables in parallel.
I am having one MySQL Server and one PostgreSQL server.
Need to replicate or re-insert set of data from multiples tables
of MySQL to be Streamed/Synced to the PostgreSQL table.
This replication can be based on time(Sync) or event such as
a new insert in the table(Stream).
I tried using the below replication tools but all these tools will be able to sync table to table only.Its not allowing to choose the columns from different tables of the source database(MySQL) and insert in to different tables in the destination database(PostgreSQL).
Symmetricds
dbconvert
pgloader
Postgresql FDW
Now I have to write an application to query the data from MySQL
and insert in to PostgreSQL as a cron job .
Its cumbersome and error prone to sync the data.
This is not able to stream(Event based) the data for realtime replication.
it would be great if some tools already solving this problem.
Please let me know if there is opensource library or tool can do this for me.
Thanks in advance.
To achieve a replication with one of tools you proposed you can do the following:
Create a separate schema in PostgreSQL and add views so that they completely copy the table structure of MySQL. You will then add rules or triggers to the views to handle inserts/updates/deletes and redirect them to the tables of your choice.
This way you have the complete freedom to transform your data during the replication, yet still use the common tools.
maybe this tool can help you. https://github.com/the4thdoctor/pg_chameleon
Pg_chameleon is a replication tool from MySQL to PostgreSQL developed in Python 2.7/3.5. The system relies on the mysql-replication library to pull the changes from MySQL and covert them into a jsonb object. A plpgsql function decodes the jsonb and replays the changes into the PostgreSQL database.
The tool can initialise the replica pulling out the data from MySQL but this requires the FLUSH TABLE WITH READ LOCK; to work properly.
The tool can pull the data from a cascading replica when the MySQL slave is configured with log-slave-updates.
I have a table in MySQL that I need to join with a couple of tables in a different server. The catch is that these other tables are in Informix.
I could make it work by selecting the content of a MySQL table and creating a temp table in Informix with the selected data, but I think in this case it would be too costly.
Is there an optimal way to join MySQL tables with Informix tables?
I faced a similar problem a number of years ago while developing a Rails app that needed to draw data from both an Informix and a MySQL database. What I ended up doing was using of an ORM library that could connect to both databases, thereby abstracting away the fact that the data was coming from two different databases. Not sure if this will end up as a better technique than your proposed temp table solution. A quick google search also brought up this, which might be promising.
This can sometimes be solved in the database management system with a technique called federation. The idea is that you create virtual tables in one of the two systems that makes queries to the other database system on demand.
For both MySQL and MariaDB there is the FederatedX storage engine that unfortunately only works with other MySQL/MariaDB systems. This is a fork of the older, but as far as I know unmaintained, Federated storage engine.
Some might also consider migrating to MariaDB where you can use the CONNECT storage engine which contains an ODBC client.
What I ended up doing is manually (that is, from the php app) keeping in sync the mysql tables with their equivalents in informix, so I didn't need to change older code. This a temporary solution, given that the older system, which is using informix, is going to be replaced.
Is it possible create/configure MySQL for functionality like SQL Server's Linked Server?
If yes, would you please tell me how? I'm using MySQL 5.5.
MySQL's FEDERATED engine provides functionality similar to SQL Server's Linked Server (and Oracle's dblink) functionality, but doesn't support connecting to vendors other than MySQL. It's not clear from the question if you need the functionality to connect to vendors other than MySQL.
You might want to look into MySQL Proxy. This doesn't match the architecture of Linked Servers/dblink, but you can probably solve a similar set of problems that you would use Linked Servers/dblink to solve.
I am the developer of the MySQL Data Controller. Unfortunately, since we had lack of requests we have stopped development on it. The plugin was 100% functional with MySQL connecting to Oracle, MSSQL or MySQL.
Base on some requests, we had added back a blog and video about it :
http://www.acentera.com/mysql-datacontroller/
Regards,
Francis L.
Unfortunately you cannot link an entire MySQL database to another MySQL database like you can with MS SQL. However, you can link individual tables. A federated table is a local table you create that points to a table on another server.
You can run queries and stored procedures just like any other table. The two tables must have the same structure, except the Federated table uses a different database engine: Federated. If you make ANY changes to the structure of the remote table, you should re-create the local federated table.
The process is actually quite easy, here is an example: https://docs.oracle.com/cd/E17952_01/mysql-5.0-en/federated-use.html
In my experience, the time needed to create and implement this process is minimal, even when compared to linked servers. It should take you less than 30 minutes to get your first federated table working, after that its a 5 min process. Last item, when naming your federated table, I give it the same name as the remote table with a "federated_" in front, like federated_customer.
Also, store your federated table definitions as separate stored procedures so you can reuse them anytime you need to create the federated table again, AND so that other developers can see how you generated the federated table.
The MySQL Data Controller is a follow-on to the federated engine that allows connection to different database types, such as Microsoft SQL Server or Oracle. I am not sure how development is going, yet.
See: http://en.wikipedia.org/wiki/MySQL_DataController
or: https://launchpad.net/datacontroller
HI everybody,
i have a question
if i have a computer run sqlite , and i want to make sqlite sync Mysql server in the external network.
If the data in the sqlite have been (changed/ modified), how can I sync my MYSQL DB and sqlite so that the data in sqlite will be (changed/ modified) when the data in MYSQL DB is changed/ modified?
thanks all.
You can try greplicator
geplicator is a real-time solution
designed to replicate data from MySQL
database to any other relational
database, such as Oracle, Microsoft
SQL Server, IBM DB2 UDB and MySQL.
I have a similar problem of 2-way syncing between an Android sqlite DB and a central MySql DB. The problem really comes when inserts are made, because it admits the possibility of replicated primary keys.
I did build a system a while back that used hardware and time-based GUID primary keys instead of simple, monotonically increasing primary keys. The theory is that if new records are inserted either on the handset using sqlite, or in the central server using mysql, the keys will never conflict because they are globally unique.
Unfortunately, this means you have get in the primary key management business instead of just taking the automatic one that comes with both DB engines. But in theory this approach allows data from many different handsets to be melded easily in the central database and redistributed back out to all the handsets. I am considering this approach for my Android problem although I'm still hoping to find a solution already crafted out there.