Is it possible to link tables from other databases (MS SQL, Sybase, etc.) inside a MySQL database, on a Debian server?
I am thinking this could be possible using ODBC.
Out of the box, I don't think so.
AFAIK, while its possible to implement your own functions in MySQL, these can only return single values - not tables of data.
It should be possible using a custom storage engine plugin. I believe there's one written for DB2 but a quick google turned up nothing for ODBC. It'd be a useful thing to have - so you could write one.
The only caveat is that neither the local MySQL nor the remote ODBC connected database would be able to optimise queries spanning engines properly - so it might be more efficient to handle the two systems in a progrmanning language which supports both.
Related
I want to query data from 2 different database server using mysql. Is there a way to do that without having to create Federated database as Google Cloud Platform does not support Federated Engine.
Thanks!
In addition to #MontyPython's excellent response, there is a third, albeit a bit cumbersome, way to do this if by any chance you cannot use Federated Engine and you also cannot manage your databases replication.
Use an ETL tool to do the work
Back in the day, I faced a very similar problem: I had to join data from two separate database servers, neither of which I had any administrative access to. I ended up setting up Pentaho's ETL suite of tools to Extract data from both databases, Transform if (basically having Pentaho do a lot of work with both datasets) and Loading it on my very own local database engine where I ended up with exactly the merged and processed data I needed.
Be advised, this IS a lot of work (you have to "teach" your ETL tool what you need and depending on what tool you use, it may involve quite some coding) but once you're done, you can schedule the work to happen automatically at regular intervals so you always have your local processed/merged data readily accesible.
FWIW, I used Pentaho's community edition so free as in beer
You can achieve this in two ways, one you have already mentioned:
1. Use Federated Engine
You can see how it is done here - Join tables from two different server. This is a MySQL specific answer.
2. Set up Multi-source Replication on another server and query that server
You can easily set up Multi-source Replication using Replication channels
Check out their official documentation here - https://dev.mysql.com/doc/refman/8.0/en/replication-multi-source-tutorials.html
If you have an older version of MySQL where Replication channels are not available, you may use one of the many third-party replicators like Tungsten Replicator.
P.S. - There is no such thing in MySQL as a FDW in PostgreSQL. Joins across servers are easily possible in other database management systems but not in MySQL.
I have a table in MySQL that I need to join with a couple of tables in a different server. The catch is that these other tables are in Informix.
I could make it work by selecting the content of a MySQL table and creating a temp table in Informix with the selected data, but I think in this case it would be too costly.
Is there an optimal way to join MySQL tables with Informix tables?
I faced a similar problem a number of years ago while developing a Rails app that needed to draw data from both an Informix and a MySQL database. What I ended up doing was using of an ORM library that could connect to both databases, thereby abstracting away the fact that the data was coming from two different databases. Not sure if this will end up as a better technique than your proposed temp table solution. A quick google search also brought up this, which might be promising.
This can sometimes be solved in the database management system with a technique called federation. The idea is that you create virtual tables in one of the two systems that makes queries to the other database system on demand.
For both MySQL and MariaDB there is the FederatedX storage engine that unfortunately only works with other MySQL/MariaDB systems. This is a fork of the older, but as far as I know unmaintained, Federated storage engine.
Some might also consider migrating to MariaDB where you can use the CONNECT storage engine which contains an ODBC client.
What I ended up doing is manually (that is, from the php app) keeping in sync the mysql tables with their equivalents in informix, so I didn't need to change older code. This a temporary solution, given that the older system, which is using informix, is going to be replaced.
I work for a large organization that has an established and well populated MS-SQL server. However, I am not a Microsoft user, and my database of choice is MySQL. I am looking for a solution that will allow me to either...
-Directly query our MS-SQL server from my MySQL server
and/or
-Set up some sort of job that will copy data systematically from the MS-SQL server to our MySQL server.
It looks like Linked Servers may be part of the solution, however everything I have found describes scenarios where MS-SQL is accessing MySQL, not the other way around.
To be clear I want my MySQL server to talk to/query/pull data from my MS-SQL server.
Any help appreciated!
As far as I'm aware, you can't query any other RDBMS vendor from MySQL. MySQL's remote access feature is FEDERATED tables, which only work with other MySQL databases as far as I know.
About the simplest way you could do this would be to use SQL Server's Import/Export Wizard to create a simple package that copies the data to your MySQL server through an ODBC or ADO.NET connection to the MySQL database.
To be clear I want my MySQL server to talk to/query/pull
data from my MS-SQL server.
I think it is hard to even assume this is the best decision. Without a TON more context of what the real problem is and/or the real "need", answers vary widely from "just use ms-sql" to other levels of ad-hoc ETL. That said, some abstract feedback.
There is nothing wrong with MS-SQL, as long as you are (a) not paying for it and (b) have a clean solution to use it from a real POSIX based system. Technically, MS-SQL is a great database, I just dislike Windows. To that end, I made sure that working with MS-SQL from Ruby was done well at both the C extension layer with TinyTDS and the ActiveRecord adapter.
Sadly, I have personally stopped maintaing the later, but the C extensions are strong and even used by great projects like Sequel which if you had to some sort of raw ETL without the overhead of ActiveRecord is a great choice since it has adapter for all DBs, TinyTDS included.
There's a project I'm working on, kind of a distributed Database thing.
I started by creating the conceptual schema, and I've partitioned the tables such that I may require to perform joins between tables in MySQL and PostgreSQL.
I know I can write some sort of middleware that will break down the SQL queries and issue sub-queries targeting individual DBs, and them merge the results, but I'd like to do do this using SQL if possible.
My search so far has yielded this (Federated storage engine for MySQL) but it seems to work for MySQL databases.
If it's possible, I'd appreciate some pointer's on what to look at, preferably in Python.
Thanks.
It might take some time to set up, but PrestoDB is a valid OpenSource solution to consider.
see https://prestodb.io/
You connect connect to Presto with JDBC, send it the SQL, it interprets the different connections, dispatches to the different sources, then does the final work on the Presto node before returning the result.
From the postgres side, you can try using a foreign data wrapper such as mysql_ftw (example). Queries with joins can then be run through various Postgres clients, such as psql, pgAdmin, psycopg2 (for Python), etc.
This is not possible with SQL.
Your options are to write your own "middleware" as you hinted at. To do that in Python, you would use the standard DB-API drivers for both databases and write individual queries; then merge their results. An ORM like sqlalchemy will go a long way to help with that.
The other option is to use an integration layer. There are many options out there, however, none that I know that are written in Python. mule esb, apache servicemix, wso2 and jboss metamatrix are some of the more popular ones.
You can colocate the data on a single RDBMS node (either PostgreSQL or MySQL for example).
Two main approaches
Readonly - You might want to use read-replicas of both source systems, then use a process to copy the data to a new writeable converged node; OR
Primary - You might chose a primary database of 2. Move the data from one to the primary using a conversion process (eg. ETL or off the shelf table-level replication)
Then you can just run the query on the one RDBMS with JOINs as usual.
BONUS: You can also do log reading from RDBMS that can ship logs through Kafka. You can make it really complex as required.
Is it possible create/configure MySQL for functionality like SQL Server's Linked Server?
If yes, would you please tell me how? I'm using MySQL 5.5.
MySQL's FEDERATED engine provides functionality similar to SQL Server's Linked Server (and Oracle's dblink) functionality, but doesn't support connecting to vendors other than MySQL. It's not clear from the question if you need the functionality to connect to vendors other than MySQL.
You might want to look into MySQL Proxy. This doesn't match the architecture of Linked Servers/dblink, but you can probably solve a similar set of problems that you would use Linked Servers/dblink to solve.
I am the developer of the MySQL Data Controller. Unfortunately, since we had lack of requests we have stopped development on it. The plugin was 100% functional with MySQL connecting to Oracle, MSSQL or MySQL.
Base on some requests, we had added back a blog and video about it :
http://www.acentera.com/mysql-datacontroller/
Regards,
Francis L.
Unfortunately you cannot link an entire MySQL database to another MySQL database like you can with MS SQL. However, you can link individual tables. A federated table is a local table you create that points to a table on another server.
You can run queries and stored procedures just like any other table. The two tables must have the same structure, except the Federated table uses a different database engine: Federated. If you make ANY changes to the structure of the remote table, you should re-create the local federated table.
The process is actually quite easy, here is an example: https://docs.oracle.com/cd/E17952_01/mysql-5.0-en/federated-use.html
In my experience, the time needed to create and implement this process is minimal, even when compared to linked servers. It should take you less than 30 minutes to get your first federated table working, after that its a 5 min process. Last item, when naming your federated table, I give it the same name as the remote table with a "federated_" in front, like federated_customer.
Also, store your federated table definitions as separate stored procedures so you can reuse them anytime you need to create the federated table again, AND so that other developers can see how you generated the federated table.
The MySQL Data Controller is a follow-on to the federated engine that allows connection to different database types, such as Microsoft SQL Server or Oracle. I am not sure how development is going, yet.
See: http://en.wikipedia.org/wiki/MySQL_DataController
or: https://launchpad.net/datacontroller