How to Get Data from Multiple Database Dynamically? - mysql

I have multiple databases in MySQL from Diffrent Companies.
Ex.
1.Company1
2.Company2
3.Company3
4.Company4
In every database the table and column's structure are same but data is stored for different companies.
Now if i have to get a count of EmployeeID's sales for diffrent companies then i need to write the queries like below.
Select a.EmployeeID,Count(b.TransactionDate)
From Company1.Employee as a
Inner Join Company1.Sales as b
On a.EmployeeID=b.EmployeeID
Group By a.EmployeeID
Union
Select a.EmployeeID,Count(b.TransactionDate)
From Company2.Employee as a
Inner Join Company2.Sales as b
On a.EmployeeID=b.EmployeeID
Group By a.EmployeeID
Union
Select a.EmployeeID,Count(b.TransactionDate)
From Company3.Employee as a
Inner Join Company3.Sales as b
On a.EmployeeID=b.EmployeeID
Group By a.EmployeeID
Union
Select a.EmployeeID,Count(b.TransactionDate)
From Company4.Employee as a
Inner Join Company4.Sales as b
On a.EmployeeID=b.EmployeeID
Group By a.EmployeeID
Notice i am changing the database in "FROM" Clause and "INNER JOIN" with hard-coded value.
In future further database may be added and i don't want to change the code behind or i don't want add code with another "union".
Is there anything which we can do to do it dynamically. i mean if we can store database name in a table and query should automatically pick those database information from the table.

I'm always waiting for upvotes and acknowledgements ;)
With this databases (they are all the same of course, so i post only one of them.
use `company3`;
DROP TABLE IF EXISTS Employee;
CREATE TABLE Employee
(`EmployeeID` int, `LastName` varchar(40), `Firstname` varchar(40), `Age` int)
;
INSERT INTO Employee
(`EmployeeID`, `LastName`, `Firstname`, `Age`)
VALUES
(1, 'Hansen', 'Han', 30),
(2, 'Svendson', 'Sven', 23),
(3, 'Pettersen', 'Peter', 20)
;
DROP TABLE IF EXISTS Sales;
CREATE TABLE Sales
(`EmployeeID` int, `TransactionDate` datetime)
;
INSERT INTO Sales
(`EmployeeID`, `TransactionDate`)
VALUES
(1, '2015-12-20 10:01:00'),
(1, '2015-12-20 10:01:00'),
(2, '2015-12-20 10:01:00'),
(2, '2015-12-20 10:01:00'),
(2, '2015-12-20 10:01:00')
;
And this stored procedure
CREATE DEFINER=`root`#`localhost` PROCEDURE `GetSakesConut`()
BEGIN
DECLARE bDone INT;
DECLARE DBname TEXT;
DECLARE sqlstement LONGTEXT;
DECLARE n INT;
DECLARE curs CURSOR FOR SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA
WHERE SCHEMA_NAME LIKE 'company%';
DECLARE CONTINUE HANDLER FOR NOT FOUND SET bDone = 1;
OPEN curs;
SET bDone = 0;
SET n =0;
SET sqlstement = '';
SALESloop: LOOP
FETCH curs INTO DBname;
IF bDone = 1 THEN
LEAVE SALESloop;
END IF;
IF n>0 THEN
SET sqlstement = CONCAT(sqlstement,' UNION ');
END IF;
SET sqlstement = CONCAT(sqlstement,'Select "',DBname,'",a.EmployeeID,');
SET sqlstement = CONCAT(sqlstement,'Count(b.TransactionDate) ');
SET sqlstement = CONCAT(sqlstement,'From ',DBname,'.Employee as a ');
SET sqlstement = CONCAT(sqlstement,'Inner Join ',DBname,'.Sales as b ');
SET sqlstement = CONCAT(sqlstement,'On a.EmployeeID=b.EmployeeID ');
SET sqlstement = CONCAT(sqlstement,'Group By a.EmployeeID ');
SET n =n+1;
END LOOP SALESloop;
CLOSE curs;
SET #sqlstement = sqlstement;
PREPARE stmt FROM #sqlstement;
EXECUTE stmt;
END
For the explanation:
For the cursor curs i get all Database names that start with compan
In the loop i get one Dataase name after another and i build with it
your select statement with the correct database names.
And of course ou have to add union to all Select without the first
you get folloowing Result
company1 EmployeeID Count(b.TransactionDate)
company1 1 2
company1 2 3
company2 1 2
company2 2 3
company3 1 2
company3 2 3
Of course i had to adept the select statement because yours didn't work properly.

Related

Batch MYSQL inserts for performance in DB structure migration

I need to restructure my MYSQL InnoDB database.
At the moment I have a customer table holding 3 product names.
I need to extract these names to a new product table. The product table should hold each name currently held in the customer table and be linked to the customer table via a new customer_product table. While the product names may not be unique, they don't have anything to do with each other, meaning for each customer there will need to be inserted 3 new entries into the product table and 3 new entries into the customer_product table.
So instead of this:
customer
| id | product_name_a | product_name_b | product_name_c |
I need this:
customer
| id |
customer_product
| customer_id | product_id | X3
product
| id | name | X3
I've written the following MYSQL procedure that works:
BEGIN
DECLARE nbr_of_customers BIGINT(20);
DECLARE customer_count BIGINT(20);
DECLARE product_id BIGINT(20);
DECLARE customer_id BIGINT(20);
DECLARE product_name_a VARCHAR(500);
DECLARE product_name_b VARCHAR(500);
DECLARE product_name_c VARCHAR(500);
SELECT COUNT(*) FROM customer INTO nbr_of_customers;
SET customer_count = 0;
SET product_id = 1;
WHILE customer_count < nbr_of_customers DO
SELECT
customer.id,
customer.product_name_a,
customer.product_name_b,
customer.product_name_c
INTO
customer_id,
product_name_a,
product_name_b,
product_name_c
FROM customer
LIMIT customer_count,1;
INSERT INTO product(id, name)
VALUES(product_id, product_name_a);
INSERT INTO customer_product(customer_id, product_id)
VALUES(customer_id, product_id);
SET product_id = product_id + 1;
INSERT INTO product(id, name)
VALUES(product_id, product_name_b);
INSERT INTO customer_product(customer_id, product_id)
VALUES(customer_id, product_id);
SET product_id = product_id + 1;
INSERT INTO product(id, name)
VALUES(product_id, product_name_c);
INSERT INTO customer_product(customer_id, product_id)
VALUES(customer_id, product_id);
SET product_id = product_id + 1;
SET customer_count = customer_count + 1;
END WHILE;
END;
This is too slow.
I've run this locally and estimate that my ~15k customers would take ~1h to complete. And my VPS server is far slower than that, so it could take upward to 10h to complete.
The problem seem to be the inserts taking a long time. I've would therefore like to store all the inserts during the procedure and execute them all in batch after the loop is complete and I know what to insert.
I there a way to perform all the ~100k inserts in batch to optimize performance, or is there a better way to do it?
FINAL EDIT:
I marked the correct solution based on that it did an excellent job of speeding up the process massively, which was the main focus of the question. In the end I ended up performing the migration using modified production code (in Java), due to the solution's limitations regarding not escaping the inserted strings.
First, use a cursor to process the results of a single query, rather than performing a separate query for each row.
Then concatenate the VALUES lists into strings that you execute using PREPARE and EXECUTE.
My code does the inserts in batches of 100 customers, because I expect there's a limit on the size of a query.
BEGIN
DECLARE product_id BIGINT(20);
DECLARE customer_id BIGINT(20);
DECLARE product_name_a VARCHAR(500);
DECLARE product_name_b VARCHAR(500);
DECLARE product_name_c VARCHAR(500);
DECLARE done INT DEFAULT FALSE;
DECLARE cur CURSOR FOR SELECT c.id, c.product_name_a, c.product_name_b, c.product_name_c FROM customer AS c;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
SET product_id = 1;
OPEN cur;
SET #product_values = '';
SET #cp_values = '';
read_loop: LOOP
FETCH cur INTO customer_id, product_name_a, product_name_b, product_name_c;
IF done THEN
LEAVE read_loop;
END IF;
SET #product_values = CONCAT(#product_values, IF(#product_values != '', ',', ''), "(", product_id, ",'", product_name_a, "'), (", product_id + 1, ",'", product_name_b, "'), (", product_id + 2, ",'", product_name_c, "'), ");
SET #cp_values = CONCAT(#cp_values, IF(#cp_values != '', ',', ''), "(", customer_id, ",", product_id, "), (", customer_id, ",", product_id + 1, "), (", customer_id, ",", product_id + 2, "),");
SET product_id = product_id + 3;
IF product_id % 300 = 1 -- insert every 100 customers
THEN BEGIN
SET #insert_product = CONCAT("INSERT INTO product(id, name) VALUES ", #product_values);
PREPARE stmt1 FROM #insert_product;
EXECUTE stmt1;
SET #insert_cp = CONCAT("INSERT INTO customer_product(customer_id, product_id) VALUES ", #cp_values);
PREPARE stmt2 FROM #insert_cp;
EXECUTE stmt2;
SET #product_values = '';
SET #cp_values = '';
END IF;
END LOOP;
IF #product_values != '' -- Process any remaining rows
THEN BEGIN
SET #insert_product = CONCAT("INSERT INTO product(id, name) VALUES ", #product_values);
PREPARE stmt1 FROM #insert_product;
EXECUTE stmt1;
SET #insert_cp = CONCAT("INSERT INTO customer_product(customer_id, product_id) VALUES ", #cp_values);
PREPARE stmt2 FROM #insert_cp;
EXECUTE stmt2;
SET #product_values = '';
SET #cp_values = '';
END IF;
END;
Beware that, using this solution, the product names will not be properly escaped before inserting. This solution will therefore not work if any of the product names contains special characters, such as single quote '.
Perhaps you could do this in three separate inserts (instead of ~100K) as follows:
INSERT INTO customer_product (customer_id, product_id)
SELECT customer.id as customer_id, product.id as product_id
FROM customer
JOIN product on customer.product_name_a = product.name
INSERT INTO customer_product (customer_id, product_id)
SELECT customer.id as customer_id, product.id as product_id
FROM customer
JOIN product on customer.product_name_b = product.name
INSERT INTO customer_product (customer_id, product_id)
SELECT customer.id as customer_id, product.id as product_id
FROM customer
JOIN product on customer.product_name_c = product.name
Of course, you would have to set up your product table ahead of time, and you'd want to drop your de-normalized columns from your customer table after the fact.
This could be further sped up if you create an index on the customer.product_name_X columns (and possibly the product.name column, though it's so few, idk if it would be significant). EXPLAIN can help with that.

IF ELSE STATEMENT for the Differnce of two columns in MYSQL

I have 3 tables : badge_master, match_result_updation and team_badges.
The badge_master is the master table where I manually insert the data acc to the excel. On match_result_updation I have some columns, if sum(goal_column1) - sum(goal_column2) = 10, then the value column1 will get the badge and it will be inserted in team_badges table. I am handling it through trigger but unable to proceed after certain time.
The trigger which I tried :
CREATE TRIGGER `afterinsert_teamgoals` AFTER INSERT ON `match_result_updation` FOR EACH ROW
BEGIN
DECLARE goalCount1 INT(10);
DECLARE goalCount2 INT(10);
DECLARE badgeId BIGINT(20);
DECLARE teamId bigint(20) default 0 ;
SELECT team1_goal INTO goalCount1 FROM match_result_updation WHERE team1_id = NEW.team1_id ;
SELECT team2_goal INTO goalCount2 FROM match_result_updation WHERE team2_id = NEW.team2_id ;
IF (goalCount1 - goalCount2 >= 10)
Then
Insert into team_badges(team_id,badge_id,match_id,timestamp)
SELECT teamId , badgeId , match_id FROM match_result_updation limit 1;
END IF;
Please Assist.

count number of hierarchical childrens in sql

I have a table that stores parent and left child and right child information. How do i count number of children belongs that parent?
for example my table structure is:
parent left right
--------------------
1 2 3
3 4 5
4 8 9
5 10 11
2 6 7
9 12 null
How do I count number of sub nodes for any parent. For example 4 contains following hierarchical child nodes - 8,9,12 so number of children are 3.
3 contains following sub nodes -> 4,5,10,11,8,9,12 so total number of children 7.
How do I achieve this using SQL query?
create table mytable
( parent int not null,
cleft int null,
cright int null
)
insert into mytable (parent,cleft,cright) values (1,2,3);
insert into mytable (parent,cleft,cright) values (2,6,7);
insert into mytable (parent,cleft,cright) values (3,4,5);
insert into mytable (parent,cleft,cright) values (4,8,9);
insert into mytable (parent,cleft,cright) values (5,10,11);
insert into mytable (parent,cleft,cright) values (6,null,null);
insert into mytable (parent,cleft,cright) values (7,null,null);
insert into mytable (parent,cleft,cright) values (8,13,null);
insert into mytable (parent,cleft,cright) values (9,12,null);
insert into mytable (parent,cleft,cright) values (10,null,null);
insert into mytable (parent,cleft,cright) values (12,null,null);
insert into mytable (parent,cleft,cright) values (13,null,17);
insert into mytable (parent,cleft,cright) values (17,null,null);
DELIMITER $$
CREATE procedure GetChildCount (IN parentID INT)
DETERMINISTIC
BEGIN
declare ch int;
declare this_left int;
declare this_right int;
declare bContinue boolean;
declare count_needs_scan int;
create temporary table asdf999 (node_id int,processed int);
-- insert into asdf999 (node_id,processed) values (1,0);
-- update asdf999 set processed=1;
SET ch = parentID;
set bContinue=true;
while bContinue DO
-- at this point you are sitting at a ch (anywhere in hierarchy)
-- as you are looping and getting/using children
-- save non-null children references: -----------------------------
select cleft into this_left from mytable where parent=ch;
if !isnull(this_left) then
insert asdf999 (node_id,processed) select this_left,0;
end if;
select cright into this_right from mytable where parent=ch;
if !isnull(this_right) then
insert asdf999 (node_id,processed) select this_right,0;
end if;
-- -----------------------------------------------------------------
select count(*) into count_needs_scan from asdf999 where processed=0;
if count_needs_scan=0 then
set bContinue=false;
else
select node_id into ch from asdf999 where processed=0 limit 1;
update asdf999 set processed=1 where node_id=ch;
-- well, it is about to be processed
end if;
END WHILE;
select count(*) as the_count from asdf999;
drop table asdf999;
END $$
DELIMITER ;
call GetChildCount(2); -- answer is 2
call GetChildCount(4); -- answer is 5
I could supply a version that creates a dynamically named table (or temp table) and clobbers it at end if you want . "dynamic sql / prepare statment" inside of a procedure. that way users won't step on each other with shared use of the work table asdf999. so this is not production ready. but the above gives you an idea of the concept

MySQL stored procedure pass select as parameter

could you please give me an advice how to CALL prcd with SELECT results? Or advice me pls better solution.. I am open minded to all working solution
I have a procedure to control inserting data ...
CREATE PROCEDURE control_insert (
)
And I need to pass data from SELECT results to procedure ...
SELECT t.c1, t.c2
FROM table t1
LEFT JOIN other_table t2
ON t1.id = t2.id
WHERE 1=1
The point is, I need to get some data via SELECT (around 6 tables joined to the base table) and I need to do control for each row before insert.. each row should meet some conditions .. if it doesn't meet them, it should just skip it and process next one ...
The procedure should look like:
CREATE PROCEDURE control_insert (
IN v_c1 INT,
IN v_c2 INT
)
BEGIN
IF v_c1 > 1 THEN
INSERT INTO controlled_table (id, type) VALUES (v_c1, v_c2);
ELSE
/* do nothing */
END IF;
END;
CALL control_insert ( SELECT .... );
Could you help me with that? Is there any possibility to do this via MySQL? I can write a PERL skript, but I want to avoid this type of solution ... I just one to do it only in MySQL way
Thank you
EDIT1: I need to check if ID of the SELECT result and LABEL is already in this table for specific date ... this code above is only an example to demonstrate the situation
SOLUTION
I've found the solution ... so for the other visitors:
calling procedure:
CALL controlInsert();
procedure body:
CREATE PROCEDURE controlInsert()
BEGIN
DECLARE done INT DEFAULT FALSE;
DECLARE v_id INT;
DECLARE v_id_dupl INT;
DECLARE v_label INT;
DECLARE v_date DATE;
DECLARE v_type VARCHAR(100);
DECLARE v_category VARCHAR(255);
DECLARE v_user VARCHAR(255);
DECLARE v_country VARCHAR(255);
DECLARE c1 CURSOR FOR SELECT id, label, date, type, category, user, country FROM t1 LEFT JOIN ... /* whole select with 6 joins ended by ; */
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
## open cursor
OPEN c1;
## loop through the cursor
read_loop: LOOP
## fetch cursor into variables
FETCH c1 INTO v_id , v_label, v_date, v_type, v_category, v_user, v_country;
## check if there is any record
IF done THEN
LEAVE read_loop;
END IF;
## get count of existing records
SELECT count(*) INTO v_id_dupl
FROM
WHERE 1=1
AND id = v_id
AND label= v_label
AND date = v_date;
## if v_id_dupl = 0 => no rows found (ok to load)
IF (v_id_dupl = 0) THEN
INSERT INTO target_table (id, label, date, type, category, user, country)
VALUES (v_id , v_label, v_date, v_type, v_category, v_user, v_country);
END IF;
END LOOP;
CLOSE c1;
END
If that is all your stored procedure is doing, then you don't actually need it. You can do the whole thing in a single statement:
INSERT INTO controlled_table (id, type)
SELECT t.c1, t.c2
FROM table t1
LEFT JOIN other_table t2 ON t1.id = t2.id
WHERE something = somethingElse
AND t.c1 > 1
Essentially, I've just combined your original query with the INSERT statement in your procedure.
If your procedure is more complex and needs to do multiple operations on each row, then you should look into using a cursor.

No data return after calling the stored procedures with multiple cursors in it

I have a stored procedures with the following code. The reason i use cursor is to join table which something will return NULL value and cause the record to be disappear. By using this method, I am able to get all data without losing any.
The only problem now is that when i try to call the stored precedures, it return
Error Code : 1329
No data - zero rows fetched, selected, or processed
but when i do a manual select * from TMOMain, the table is created and there is data in it but no data from SignUpCur and UnSubCur mean it was not updated.
1st time using mysql stored procedures so there might be something i miss out.
My Code
ROOT:BEGIN
DECLARE pTotal,pShortCode,pSignUp,pUnSub,pJunk,pT INT;
DECLARE pTc NVARCHAR(10);
DECLARE SignTotal,UnSubTotal, JunkTotal INT;
DECLARE pSignTotal,pSignTeamID,pUnSubTotal,pUnSubT,pSignUpS,pUnSubS INT;
DECLARE pSignTeam,pUnSubTeam NVARCHAR(10);
DECLARE no_more_rows BOOLEAN;
DECLARE MoMainCur CURSOR FOR
SELECT COUNT(*) AS GrandTotal,pShort,(CASE WHEN r= 1 THEN 'A'
WHEN r= 2 THEN 'B' WHEN r= 3 THEN 'C' ELSE 'UV' END) AS Team,recvTeamID
FROM tbli
INNER JOIN tblK ON keywordid = rkey
WHERE recvDate >='2011-11-15' AND recvDate < '2011-11-16'
GROUP BY pShort,Team,recvTeamID;
DECLARE SignUpCur CURSOR FOR
SELECT COUNT(*) AS SignUp,(CASE WHEN r= 1 THEN 'A'
WHEN r= 2 THEN 'B' WHEN r= 3 THEN 'C' ELSE 'UV' END) AS Team,
recvTeamID,pShort
FROM tbli INNER JOIN tbl_user ON recvphone = userphone
INNER JOIN tblK ON keywordid = userpublicstatus
WHERE userdatejoined >='2011-11-15' AND userdatejoined < '2011-11-16'
AND recvdate >='2011-11-15' AND recvdate < '2011-11-16'
GROUP BY Team,recvTeamID,pShort;
DECLARE UnSubCur CURSOR FOR
SELECT COUNT(*) AS UnSub,(CASE WHEN r= 1 THEN 'A'
WHEN r= 2 THEN 'B' WHEN r= 3 THEN 'C' ELSE 'UV' END) AS Team,
recvTeamID,pShort
FROM tbliINNER JOIN tbl_user ON recvphone = userphone
INNER JOIN tblK ON keywordid = userpublicstatus
WHERE userdateExpire >='2011-11-15' AND userdateExpire <'2011-11-16'
AND recvdate >='2011-11-15' AND recvdate < '2011-11-16'
GROUP BY Team,recvTeamID,pShort;
DROP TABLE IF EXISTS `TMoMain`;
CREATE TEMPORARY TABLE TMOMain
(GrandTotal INT,ShortCode INT,Team NVARCHAR(10),SignUp INT,UnSub INT, Junk INT, TeamID INT);
OPEN MoMainCur;
-- Main Table
read_loop:LOOP
FETCH MoMainCur INTO pTotal,pShortCode,pTc,pT;
INSERT INTO TMOMain
VALUES
(pTotal,pShortcode,pTc,0,0,0,pT);
END LOOP read_loop;
CLOSE MoMainCur;
-- Insert Signup Details into Main Table
OPEN SignUpCur;
SignUp_Loop:LOOP
FETCH SignUpCur INTO pSignTotal,pSignTeam,pSignTeamID,pSignUpS;
UPDATE TMOMain
SET SignUp = pSignTotal
WHERE Team = pSignTeam AND Shortcode =pSignUpS;
END LOOP SignUp_Loop;
CLOSE SignUpCur;
-- Insert UnSub Details into Main Table
OPEN UnSubCur;
UnSub_Loop:LOOP
FETCH UnSubCur INTO pUnSubTotal,pUnSubTeam,pUnSubT,pUnSubS;
UPDATE TMOMain
SET UnSub = pSignTotal
WHERE Team = pUnSubTeam AND pShort = pUnSubShortCode;
END LOOP UnSub_Loop;
CLOSE UnSubCur;
SELECT * FROM TMOMain;
END$$
Please try this out:
Add this declaration once (at the top):
DECLARE curIsDone INT DEFAULT FALSE;
Then after you declare your cursor add this:
DECLARE CONTINUE HANDLER FOR NOT FOUND SET curIsDone = TRUE;
After your FETCH commands and before the actions you intend to perform:
IF curIsDone THEN
LEAVE read_loop;
END IF;