I have a regular Rails application that uses a Postgres database, but I have the following requirement: every day I have to transfer data from all tables of this Postgres database to the customer's MySQL database.
There's no API available, so I have to connect to the customer's database and perform create/update queries for the new/updated rows. They will allow my IP for these operations.
What would be the best way to achieve that? I thought of some options:
1) Schedule a job on my Rails application to perform that operation (the con here is: this logic is specific for one customer, so I don't like the idea of having this on the main codebase)
2) Deploy a lightweight application (maybe node/express) that reads from one database and sends to another (the con here is: I'll have to maintain another server to keep this running)
Are there any other options that I am not considering?
You could use a foreign data wrapper to connect to the MySQL database from your PostgreSQL database. That would allow you to read and write to the customer database with very little that you would need to write or maintain.
It looks like there is a well maintained wrapper for MySQL.
Related
I need to create a system with local webservers on Raspberry Pi 4 running laravel for API calls, websockets, etc. Each RPI will be installed in multiple customers places.
For this project i want to have the abality to save/sync the database to a remote server (when the local system is connected to internet).
Multiple locale databases => One remote database cutomers based
The question is, how to synchronize databases and identify properly each customers data and render them in a mutualised remote dashboard.
My first thought was to set a customer_id or a team_id on each tables but it seems dirty.
The other way is to create multiple databases on the remote server for the synchronization and one extra database to set customers ids and database connection informations...
Someone has already experimented something like that? Is there a sure and clean way to do this?
You refer to locale but I am assuming you mean local.
From what you have said you have two options at the central site. The central database can either store information from the remote databases into a single table with an additional column that indicates which remote site it's from, or you can setup a separate table (or database) for each remote site.
How do you want to use the data?
If you only ever want to work with the data from one remote site at a time it doesn't really matter - in both scenarios you need to identify what data you want to work with and build your SQL statement to either filter by the appropriate column, or you need to direct it to the appropriate table(s).
If you want to work on data from multiple remote sites at the same time, then using different tables requires tyhat you use UNION queries to extract the data and this is unlikely to scale well. In that case you would be better off using a column to mark each record with the remote site it references.
I recommend that you consider using Uuids as primary keys - it may be that key collision will not be an issue in your scenario but if it becomes one trying to alter the design retrospectively is likely to be quite a bit of work.
You also asked about how to synchronize the databases. That will depend on what type of connection you have between the sites and the capabilities of your software, but typically you would have the local system periodically talk to a webservice at the central site. Assuming you are collecting sensor data or some such the dialogue would be something like:
Client - Hello Server, my last sensor reading is timestamped xxxx
Server - Hello Client, [ send me sensor readings from yyyy | I don't need any data ]
You can include things like a signature check (for example an MD5 sum of the records within a time period) if you want to but that may be overkill.
I am looking for a tip how to synchronize data from a local firebird database into online db? Few comments:
On a local machine I use sales software which keeps data on firebird db. There is an internet connection, but I want to avoid direct db access (as the PC after 9pm is being turned off).
I would like to create an online app (based on foundation + php + database) in which I will be able to view daily sales and explore past data.
In local db, I will need to pull data from several different tables, and I would like to keep them in online/final db as a single table (with fields: #id, transaction date, transaction value, sales manager).
While mostly I know how to create frontend of the app, and partially backend still I wonder what would be best choice in terms of db - mysql? (it was my first thought). Or rather I should focus on NoSQL?
What's your recommendation on data sync? I should use symmetricsDB (pretty hard to configure) or equivalent, I should write a script which will push data from firebird into json/xml? I'm referring to your knowledge and best practices
Put a scheduled job that will invoke a simple data pump / replication script.
From the script, connect to the source sales db, retrieve the joined data added from last replication and insert them into the "online" database.
You may keep also Firebird as online DB as it works great with PHP.
Firebird also in version 2.5 has all technology already build in to implement a fully functional replication. We have implemented this in the largest installation for a big restaurant company with about 0.6 billion records, daily about 1 million new records and 150 locations where replicated servers are working online or offline with the back office software.
If you simply want to upload the data from your local db to a remote db, you can rent a virtual server at a provider you like, install firebird there, create a secure connection (we use ssh, but any tcp over vpn can be used). copy your local database to the remote server, if required open firewall fb port (3050 or other) and when you a low number of writes on your local database, simply implement a trigger on each table, that does the same insert/update/delete with the same values using the "execute statement on external" feature.
When your local database has higher workload, it is better to put the change data (table name and pk values) from trigger into a log table and let a second connection upload the records to the target db, where the same "execute statement on external" can be used.
this is just a hint how to do that, if budget allows, we can do it for you, but stopping the database pc in the evening seems to be only typical for smaller companies
I am looking for a tip how to synchronize data from a local firebird database into online db? Few comments:
On a local machine I use sales software which keeps data on firebird db. There is an internet connection, but I want to avoid direct db access (as the PC after 9pm is being turned off).
I would like to create an online app (based on foundation + php + database) in which I will be able to view daily sales and explore past data.
In local db, I will need to pull data from several different tables, and I would like to keep them in online/final db as a single table (with fields: #id, transaction date, transaction value, sales manager).
While mostly I know how to create frontend of the app, and partially backend still I wonder what would be best choice in terms of db - mysql? (it was my first thought). Or rather I should focus on NoSQL?
What's your recommendation on data sync? I should use symmetricsDB (pretty hard to configure) or equivalent, I should write a script which will push data from firebird into json/xml? I'm referring to your knowledge and best practices
Put a scheduled job that will invoke a simple data pump / replication script.
From the script, connect to the source sales db, retrieve the joined data added from last replication and insert them into the "online" database.
You may keep also Firebird as online DB as it works great with PHP.
Firebird also in version 2.5 has all technology already build in to implement a fully functional replication. We have implemented this in the largest installation for a big restaurant company with about 0.6 billion records, daily about 1 million new records and 150 locations where replicated servers are working online or offline with the back office software.
If you simply want to upload the data from your local db to a remote db, you can rent a virtual server at a provider you like, install firebird there, create a secure connection (we use ssh, but any tcp over vpn can be used). copy your local database to the remote server, if required open firewall fb port (3050 or other) and when you a low number of writes on your local database, simply implement a trigger on each table, that does the same insert/update/delete with the same values using the "execute statement on external" feature.
When your local database has higher workload, it is better to put the change data (table name and pk values) from trigger into a log table and let a second connection upload the records to the target db, where the same "execute statement on external" can be used.
this is just a hint how to do that, if budget allows, we can do it for you, but stopping the database pc in the evening seems to be only typical for smaller companies
I run a game community (with servers). I have two hosts, the first one is the "community" host which pretty much hosts the website/forums/etc, in the other host I run my game servers, along with a MySQL database containing playerdata.
In short, I want to create a ranking on the website based on the stats in the gameserver database.
In order to not put much strain to the gameservers I was wondering if it's possible (with a Cronjob maybe?, to make it periodically) to create a dump of a specific SELECT query.
However I do not know how to use the SELECT INTO query to select data and insert it between remote servers.
Is there a way to accomplish this or I'm forced to use MySQL's Federated tables?
Thank you in advance
The thing, you are talking about is called ETL in the database world (which translates to Extract, Transform, Load). Forget the transform part for now.
You are not forced to anything.... well.. Yeah, you will need a user on the source server which have access to the data you want to transfer and a user on the destination server too.
You can find several ETL tools written in almost all available languages (some of them are cool and working fine, others... not always)
The 'easiest' (= without extra tools) way to achive your goal is to:
(aka these are the steps you are looking for)
create a separate user on the game (source) server
Username: etl (for example)
Host: the application (web) server
Pass: supersecretstuff
Minimum required SELECT privileges (table level if possible)
write your extract query
create a table on the destination server which structure matches the structure determined by your extract query
extract the data using the command line client on the application server (mysql -h gamesrv.mynet database < my_extract_query.sql > extracted_data.csv <- this is a tab separated file)
use the extracted_data.csv to import the data you need (LOAD DATA LOCAL INFILE 'extracted_data.csv' INTO TABLE myimported data; http://dev.mysql.com/doc/refman/5.6/en/loading-tables.html)
To implement an incremental extract-load process, you can use timestamps or the ID-s (in case of some kind of monoton ID generation such as the auto_increment in mysql) in the tables to extract to determine point from where you should continue the copy process.
For actual recommendations or solutions, you have to know your data, but the basics are these.
I agree with the above answer. But I think you're looking to automate this process (cron job).
You could create a cron job on your application server (web) that periodically checks for updates from the game db and updates its own db.
I have task to implement particular database structure:
Multiple mysql servers with data using the same schema. Each server can see and edit only his particular part of data.
And
One master server with his own data that can run queries using data from all previously mentioned servers, but cannot edit them.
Example would be multiple hospitals with data of their patients and master server that can use combined data from all hospitals.
Previous system was written using mysql cluster, so i tried it naturally. I can create mysql cluster with multiple nodes and maybe even partition data so i can have particular set of data in particular node, but as far as i know i can't connect to single node using mysql, because it is already connected to cluster.
Can it be done with mysql cluster? Is there other framework that can do that easily?
You could try to use http://galeracluster.com/. You can perform updates on all slaves and every server has all data, but it might still meet your requirements.