I need to join table from other database and sometimes other server - mysql

My project has its own database. Also, I use table of users, which is on other database. Two offices have their data on the same server, but third one has its own user table on other server.
So, in lots of queries I need to join either table some_db.users or other_server.some_db.users
What solution would you advise for this scenario?
I use MySQL.

There is Federated tables in MySQL:
The FEDERATED storage engine lets you access data from a remote MySQL
database without using replication or cluster technology. Querying a
local FEDERATED table automatically pulls the data from the remote
(federated) tables. No data is stored on the local tables.
First, you must have a table on the remote server that you want to access by using a FEDERATED table. Suppose that the remote table is in the sakila database and is defined like this:
CREATE TABLE test_table (
id INT(20) NOT NULL AUTO_INCREMENT,
name VARCHAR(32) NOT NULL
PRIMARY KEY (id)
)
ENGINE=MyISAM
DEFAULT CHARSET=latin1;
Next, create a FEDERATED table on the local server for accessing the remote table:
CREATE TABLE federated_table (
id INT(20) NOT NULL AUTO_INCREMENT,
name VARCHAR(32) NOT NULL
PRIMARY KEY (id)
)
ENGINE=FEDERATED
DEFAULT CHARSET=latin1
CONNECTION='mysql://fed_user:fed_user#197.186.1.199:3306/sakila/test_table';
Sample connection strings:
CONNECTION='mysql://username:password#hostname:port/database/tablename'
CONNECTION='mysql://username#hostname/database/tablename'
CONNECTION='mysql://username:password#hostname/database/tablename'
The basic structure of this table should match that of the remote table, except that the ENGINE table option should be FEDERATED.
Execute:
show variables like '%federated%';
to check if FEDERATED storage engine is available on your local server.
The table federated_table in localhost becomes virtual table of test_table in remote server.
Now you can use the JOIN between the tables in a DB in the localhost server. If there is a table called test in your localhost server, and you want to JOIN with the former sakila.test_table which is in the remote server, write a query like the one shown below:
SELECT * FROM `federated_table` JOIN `test`;
The federated_table in the query will actually refer to test_table in remote server.
On enabling FEDERATED Storage Engine
The FEDERATED storage engine is not enabled by default in the running server; to enable FEDERATED, you must start the MySQL server binary using the --federated option.
NOTE:
Optional storage engines require privileges and will fail to load when --skip-grant-tables is specified.
The result the entire db will fail to load and the following error will appear in the logs:
110318 21:37:23 [ERROR] /usr/local/libexec/mysqld: unknown option '--federated'
This in turn means that an upgrade from 5.x needs to be done in two steps if you have federated tables. Once with --skip-grant-tables and without --federated, the once without --skip-grant-tables and with --federated.
Source: The FEDERATED Storage Engine

Please mention the databse also.
In SQLServer you can use Linked sever. http://msdn.microsoft.com/en-us/library/ms188279.aspx

In MySQL, you can join tables from different databases using fully qualified names like
`database_name1`. `table_name1` JOIN `database_name2`.`table_name2`
But i fear, you cant join tables from different servers because for that you need to have two different connections and as per my knowledge there are no fully qualified connection names to be used in the query.
Alternatively, you can create local temporary table(s) on one of the servers and run the query there on. But in this case you will need to transfer data from one server to another. You can use MySQL GUI tool like SQLyog or MySQL admin, to transfer data from one server to another and to synchronize databases on two servers.
Hope it helps....

Federated tables are your solution for tables on other servers. They are very slow though if you perform joins on them.
If you just want to read data from another database on the same server you can use a view. This way you have all tables virtually in one database and you have to open only one connection in your application.
CREATE
VIEW `my_db`.`table_name`
AS
(SELECT * FROM `other_db`.`table_name`);

Related

What is the connection in mysql?

I'm trying to do a query like this:
CREATE TABLE Connection_Properties (
COL VARCHAR(30) NULL
)
CONNECTION = 'mysql://localhost:3306/DOCKER2'
PASSWORD = '1234';
Although the schema called docker2 entered in connection does not exist, it is shown that the above query has been performed.
What is the role of that keyword, and if that is done, how to retrieve connection information and password information from that table (Connection_Properties)?
An example of creating and accessing the table with FEDERATED engine.
Enabling FEDERATED Engine.
Stop the server.
Open my.ini and add federated line to [mysqld] section.
Start server and ensure that it is started normally.
Execute show engines; and ensure that FEDERATED storage engine support is enabled.
Creating and accessing FEDERATED table.
Create a table and insert some data in it. The table will be created using default engine (InnoDB in my case).
USE test
CREATE TABLE local_test (id INT, val INT);
INSERT INTO local_test VALUES (1,11), (2,22);
Create FEDERATED table connected to the above table.
CREATE TABLE federated_test (id INT, val INT)
ENGINE=FEDERATED
CONNECTION='mysql://username:password#localhost:3306/test/local_test';
Pay attention - only table structure with specified connection properties is created, the connection is not established and is not even checked!
Work with FEDERATED table.
SELECT * FROM federated_test; -- 2 rows are selected
INSERT INTO federated_test VALUES (3,33); -- insert 3rd row
SELECT * FROM local_test; -- and check that it was inserted successfully
Drop FEDERATED table.
DROP TABLE federated_test; -- drop FEDERATED table definition
SELECT * FROM local_test; -- ensure that this is not effected the local table
local_test is accessed directly.
federated_test is accessed remotely. When the query which needs to access this table is executed then local server establishes a connection to remote server according the connection options (protocol, address and port of the remote server, username and password for to authenticate), accesses specified remote table in specified remote database, receives the answer and uses it in the query. In the query text there is no difference does the table is local or remote.

How can I scan for new data in one database and send it to another one?

I have two databases on two servers.
The first one contains many tables which contains many codes.
An example is "Products" Table which contains the column "ProductCode".
Lets say there are 5 distinct records in that column i.e ProductCode1 -> ProductCode5.
The second database contains all the fields from each table defined in the first database.
I use the second database to provide definitions for each code found in the first database. I have migrated all the data from all the tables in the first db over to the new one manually via an excel file and script.
However, I would like to create an SQL function which scans the first database and when it finds new rows of data, it adds that data to the second database.
This would save me the hassle of querying all the tables individually and then adding them manually, as i originally did.
Please note that both databases are stored on separate servers.
Is this possible to achieve?
Or is there any better options?
While there are recovery backup and replication methods, if both databases maintain different data, there is no single, convenient SQL function to migrate new data in all tables from one database to another. However, you can build an .sql script or stored procedure that runs duplicate-avoid queries for new data.
Consider following steps where 1 and 2 are to be run for each table:
Create Federated Table from remote MySQL database to be locally available for querying but physical storage remains in remote database. See overview of Federated Storage Engine. Note: this step needs to only be run once for each needed table.
CREATE TABLE federated_table (
id INT(20) NOT NULL AUTO_INCREMENT,
name VARCHAR(32) NOT NULL DEFAULT '',
other INT(20) NOT NULL DEFAULT '0',
PRIMARY KEY (id), INDEX name (name),
INDEX other_key (other)
)
ENGINE=FEDERATED
DEFAULT CHARSET=utf8mb4
CONNECTION='mysql://fed_user#remote_host:9306/federated/test_table
Federated table schema must be identical to remote table. Therefore, align data types of CREATE TABLE to output of SHOW CREATE TABLE in remote database.
Run SQL duplicate-avoid queries such as NOT IN vs. NOT EXISTS vs. LEFT JOIN / IS NULL. One other method is EXCEPT (same family as UNION and INTERSECT operators):
INSERT INTO Products (Col1, Col2, Col3, ...)
SELECT Col1, Col2, Col3, ...
FROM my_federated_products_table
EXCEPT
SELECT Col1, Col2, Col3, ...
FROM Products
Automate step 2 for each table into a single stored procedure (or .sql script) to be run multiple times in future.
DELIMITER $
CREATE PROCEDURE migrate_new_data()
BEGIN
-- ALL INSERT INTO STATEMENTS FROM FEDERATED TABLES
END $
DELIMITER ;
Run procedure each time from Excel, workbench, command line, or elsewhere which can serve as your SQL function:
CALL migrate_new_data;
Looks like FEDERATED Storage is what your are looking for.
From the docs:
The FEDERATED storage engine lets you access data from a remote MySQL database without using replication or cluster technology. Querying a local FEDERATED table automatically pulls the data from the remote (federated) tables. No data is stored on the local tables.
Here is and article that shows how to configure it: https://medium.com/#techrandomthoughts/setting-up-federated-tables-in-mysql-8a17520b988c

How to Insert into another Database Server

Is it possible to INSERT query to another db server?
Current Db server: 192.168.59.2
Example:
Insert into 192.168.1.1.Testing.Student (id) values (1)
Looks like you need to use MySQL The FEDERATED Storage Engine. Per documentation
The FEDERATED storage engine lets you access data from a remote MySQL
database without using replication or cluster technology. Querying a
local FEDERATED table automatically pulls the data from the remote
(federated) tables. No data is stored on the local tables.
It kind a similar concept like Linked Server in Microsoft SQL Server.
Simply Use Generate Script Feature of the SQL server :
Follow Link
I'm afraid you can't do something like that. Maybe you should look into FEDERATED tables, where you can copy values to a table from one server to another.
You could have something like this on the table, which you're trying to map:
CREATE TABLE federated_table (
id INT(20) NOT NULL AUTO_INCREMENT,
name VARCHAR(32) NOT NULL DEFAULT '',
PRIMARY KEY (id)
)
ENGINE=FEDERATED
DEFAULT CHARSET=latin1
CONNECTION='mysql://user#your_host:3306/federated/test_table';
This thread might help you.
Thank you for all the answers. I've answered my question by installing a trial version of navicat and export to excel then import to another db server. Thank you again guys!

MySql Remotely access database

I have a two DB server say Server-1 & Server-2, I have installed MySql yog into it, and the servers are able to connect with each other as I have already provided Grant privileges for the servers.
But, I need to build a query which can extract data from more than one servers like -
Select * from Server1.db.Table1, Server2.db.Table2
Is this possible in mysql, if yes, can you please help me to achieve the same.
Thanks
federated storage engine should help to full fill your requirement.
Follow link to enable Federated storage engine in your server from where you are accessing remote table.
--In Server1:
CREATE DATABASE fed_remote_db
CREATE USER 'fed_remote_user'#'%' IDENTIFIED BY 'fed_remote_password';
GRANT ALL ON fed_remote_db.* TO 'fed_remote_user'#'%';
CREATE TABLE fed_remote_db.fed_table(id INT,NAME VARCHAR(50));
--In Server2:
CREATE TABLE fed_remote_db.Server1_fed_table (
id INT NOT NULL AUTO_INCREMENT,
NAME VARCHAR(50) NOT NULL,
PRIMARY KEY (id)
) ENGINE=FEDERATED CONNECTION='mysql://fed_remote_user:fed_remote_password#Server1_IP:3306/fed_remote_db/fed_table';
SELECT *
FROM
fed_remote_db.Server1_fed_table a /* Remember Server1_fed_table table referring to remote server /,
fed_remote_db.server2_fed_table b / server2_fed_table table referring to local server */
WHERE a.id=b.id;
I believe you can convert above example for your requirements.

Federated Table Clarification

In my prior job, I was able to copy data from our production environment in a breeze by using the following statements:
from tablename#UNIXPROD2
INSERT INTO tablename#UNIXTEST2
My current job's databases aren't setup in this fashion.
So, I did some research on MySQL 5.0+ because that's what we are using for one of our customers. And I came across FEDERATED tables, so as I was reading, I found this (here):
As of MySQL 5.0.46, FEDERATED performs bulk-insert handling such that multiple rows are sent to the remote table in a batch. This provides a performance improvement. Also, if the remote table is transactional, it enables the remote storage engine to perform statement rollback properly should an error occur. This capability has the following limitations:
To me, this indicates that (A) I can copy the data from our prod database to our test database; (B) any actions performed on the federated table will also be processed on the source table, which is not what I want to do. I have some scripts that I need to run and I want to run it against actual prod data to make sure it works before I use it in the prod environment.
My question: Is my interpretation correct?
Assuming it is, I've tried:
select * from database.tablename#ipaddress, but received an error message that told me to check the MySQL manual for the version I'm running, which is what I'm going to do after I hit "Post Your Question."
I would appreciate any help in this matter.
EDIT: After further research, I think might be able to do what I need using OUTFILE and INFILE whereby I would use OUTFILE on the prod table(s) and then INFILE those rows on the test table(s). Thoughts?
My answer:
A - correct
B - correct.
You could set the user permission to read-only, but in your situation I would not use federated tables, instead dump the whole db into file and then restore it on the other server. Easiest way - use MySql Workbench.
and some info about federated tables:
You need federated enabled just on server B
You can access a view on A by making a federated table on B
You can do INSERT UPDATE DELETE on federated table
If you need read-only access you can limit the user privileges
BUT! You can't do any aggregate func. on a view which will be federated (ex. COUNT(), MAX(), UNION...) (you can, however it will lag)
Remember to set the KEY's on the federated table you are creating. (or it will lag horr.)
Remember to use ALGORITHM=MERGE on views
Remember to grant acces to USERNAME(from connection string) on server A
example of a federated table on server B:
delimiter $$
CREATE TABLE `schemaName`.`tableName`(
`keyName` VARCHAR(10) NOT NULL,
`key2Name` DATE DEFAULT '2012-01-01',
KEY `keyName` (`keyName`)
)
ENGINE=FEDERATED
DEFAULT CHARSET=utf8
CONNECTION='mysql://USERNAME:PASSWORD#IPADDRESS:PORTNUMBER/baseSchema/baseTable'
$$
And the view on server A:
CREATE
ALGORITHM = MERGE
DEFINER = `ANOTHERUSERNAME`#`%`
SQL SECURITY DEFINER
VIEW `baseSchema`.`baseTable` AS
SELECT
... AS `keyName`,
... AS `key2Name`
FROM
...