MySQL - Stored Procedure - mysql

I need to fetch data from the stored procedure and present it in a dashboard. So the background for this is.
For every select query in the Stored procedure will return only one record I've around 10 queries like that. Apart from that I've also a query which will return say some 5 rows in the same stored procedure.
I want this 5 records to be presented as 1 line along with the other queries in the same stored procedure.
Example:
CALL dashboard_proc() Will return
Date Amount total_orders
1-1-2017 500.00 50
A part from above output I want to append another select query to the same stored procedure (dashboard_proc). The other query which will return
Place Total
Hyderabad 100
Bangalore 100
Chennai 200
Mumbai 100
And I need below
Date Amount total_orders Hyderabad Bangalore Chennai Mumbai
1-1-2016 500.00 50 100 100 200 100

Based on your inputs:
First Part of the report has only one record so I have used the below structure
CREATE TABLE `ResultPart1` (
`ReportDate` date NOT NULL,
`Amount` float DEFAULT NULL,
`total_orders` int(11) DEFAULT NULL
)
insert into ResultPart1 select '2017-01-01', 500.00 , 50;
2. Second Part of the report will have distinct Places, here the places used are single word and no keyword used.
CREATE TABLE `ResultPart2` (
`Place` varchar(50) NOT NULL,
`Total` int(11) DEFAULT NULL
)
insert into ResultPart2()
Select 'Hyderabad', 100 union
Select 'Bangalore' , 100 union
Select 'Chennai' , 200 union
Select 'Mumbai' , 100;
3. Create Stored Procedure
CREATE PROCEDURE `new_procedureDashboard`()
BEGIN
Declare vPlaceCount int;
Declare vquery varchar(1000);
declare vPlace varchar(1000);
declare vPlaceTotal int;
-- select * from ResultPart1;
-- Select * from ResultPart2;
CREATE TEMPORARY TABLE IF NOT EXISTS Table_ResultPart2
(
Id int NOT NULL AUTO_INCREMENT,
Place varchar(100),
PlaceTotal int ,
PRIMARY KEY (id)
);
insert into Table_ResultPart2(Place,PlaceTotal)
Select distinct Place,Total from ResultPart2;
SET vPlaceCount=(Select count(*)from Table_ResultPart2);
WHILE vPlaceCount>0 DO
SET vPlace=(Select Place from Table_ResultPart2 where Id=vPlaceCount);
SET vPlaceTotal=(Select PlaceTotal from Table_ResultPart2 where Id=vPlaceCount);
SET vquery=concat("", "alter table ResultPart1 add " ,vPlace ," int ;");
Select vquery into #AddColumn;
PREPARE stmt FROM #AddColumn;
-- Select #AddColumn;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
SET vquery="";
-- Select vPlace,vPlaceTotal;
-- Update
SET SQL_SAFE_UPDATES = 0;
SET vquery=concat("", "update ResultPart1 SET " ,vPlace ," = " ,vPlaceTotal ," where 1=1 ;");
Select vquery into #UpdateColumn;
-- select #UpdateColumn;
PREPARE stmt1 FROM #UpdateColumn;
-- Select #AddColumn;
EXECUTE stmt1;
DEALLOCATE PREPARE stmt1;
SET SQL_SAFE_UPDATES = 1;
SET vPlaceCount=vPlaceCount-1;
SET vPlace="" ;
END WHILE;
Select * from ResultPart1;
drop TEMPORARY TABLE IF EXISTS Table_ResultPart2;
END
Explanation:
Added the required column in the main resultant table using loop, and updated the values of the newly place(added) column using the loop. You will require to make changes in your stored procedure as I have used permanent tables (ResultPart1 and ResultPart2 ) .
4. Result
Alter Statements to drop column Just in case you want to rerun the stored procedure.
alter table ResultPart1 drop column Hyderabad;
alter table ResultPart1 drop column Bangalore;
alter table ResultPart1 drop column Chennai;
alter table ResultPart1 drop column Mumbai;

Related

How to find the sum of all the numeric values present in a MySQL table's column which has datatype as varchar and are separated by commas

I have a table which has been created using the following query
create table damaged_property_value
(case_id int, property_value varchar(100) );
insert into damaged_property_value (1,'2000'),(2,'5000,3000,7000');
The problem is I need to find the total value of all the properties that have been damaged.
I am writing the following query to return the sum:
select SUM(cast(property_value as unsigned)) from damaged_property_value;
It returns the sum as 7000, i.e , 2000+5000. It is not considering the value of property which are separated by commas.
Note that 5000,3000 and 7000 are values of three different properties that have been damaged in a particular case. It should have produced 17000 as an answer.
How to solve this problem.
Please help!
As was said, the best solution would be to fix the data structure.
Now, just for the fun of solving the problem, and after much research, I managed to do the following (it requires the case_id to be sequential, starting at 1) that calculates the values of the property_value strings and puts them into the new actual_value field.
drop table if exists damaged_property_value;
create table damaged_property_value
(case_id int primary key, property_value varchar(100), actual_value int );
insert into damaged_property_value (case_id, property_value) values (1,'2000'),(2,'5000,3000,7000'),(3, '7000, 2000'),(4, '100,200,300,400,500,600');
drop procedure if exists Calculate_values;
DELIMITER $$
CREATE PROCEDURE Calculate_values()
BEGIN
DECLARE count INT;
SET count = 1;
label: LOOP
select
concat('update damaged_property_value set actual_value = ',
replace((select property_value from damaged_property_value where case_id = count), ",", "+"),
' where case_id = ', count, ';')
into #formula;
#select #formula;
prepare stmt from #formula;
execute stmt;
deallocate prepare stmt;
SET count = count +1;
IF count > (select count(*) from damaged_property_value) THEN
LEAVE label;
END IF;
END LOOP label;
END $$
DELIMITER ;
CALL Calculate_values();
select * from damaged_property_value;
/* select SUM(actual_value) from damaged_property_value; */

How to `SELECT FROM` a table that is a part of a query itself using MySQL?

Say, if I have multiple tables that have the same schema:
CREATE TABLE `tbl01`
(
`id` INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
`name` TINYTEXT,
`data` INT
);
CREATE TABLE `tbl02`
(
`id` INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
`name` TINYTEXT,
`data` INT
);
CREATE TABLE `tbl03`
(
`id` INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
`name` TINYTEXT,
`data` INT
);
-- etc. ------------------
INSERT INTO `tbl01` (`name`, `data`) VALUES
('row 1', 1),
('row 2', 1),
('row 3', 3);
INSERT INTO `tbl02` (`name`, `data`) VALUES
('cube', 1),
('circle', 0);
INSERT INTO `tbl03` (`name`, `data`) VALUES
('one', 1);
and then one table that contains names of all other tables in one of its columns:
CREATE TABLE `AllTbls`
(
`id` INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
`tblnm` VARCHAR(64) NOT NULL UNIQUE,
`desc` TINYTEXT,
`flgs` BIGINT UNSIGNED
);
INSERT INTO `AllTbls` (`tblnm`, `desc`, `flgs`) VALUES
('tbl01', 'Table 1', 0),
('tbl02', 'Table two', 1),
('tbl03', '3rd table', 0);
So if I want to write a query to retrieve contents of AllTbls and also in one column to include count of rows in each of corresponding tables, I thought the following would be the way to do it:
SELECT *, `tblnm` as TblName, (SELECT COUNT(*) FROM TblName) as cntRws
FROM `AllTbls` ORDER BY `id` ASC LIMIT 0,30;
But this returns an error:
#1146 - Table 'database.TblName' doesn't exist
I know that I can do this in multiple queries (using a loop in a programming language), but is it possible to do it in one query?
PS. I'm using MySQL v.5.7.28
The simple answer is: "you can't"
Table names are not supposed to be used like variables, to hold data, in this way. What you're supposed to have is one table:
tblContractCounts
Client, ContractCount
-------------------
IBM, 1
Microsoft, 3
Google, 2
Not three tables:
tblIBMContractCounts
ContractCount
1
tblMicrosoftContractCounts
ContractCount
3
tblGoogleContractCounts
ContractCount
2
If your number of tables is known and fixed you can perhaps remedy things by creating a view that unions them all back together, or embarking on an operation to put them all into one table, with separate views named the old names so things carry in working til you can change them. If new tables are added all the time it's a flaw in the data modelling and need to be corrected. In that case you'd have to use a programming language (front end or stored procedure) to build a single query:
//pseudo code
strSql = ""
for each row in dbquery("Select name from alltbls")
strSql += "select '" + row.name + "' as tbl, count(*) as ct from " + row.name + " union all "
next row
strSql += "select 'dummy', 0"
result = dbquery(strSql)
It doesn't have to be your front end that does this - you could also do this in mysql and leverage the dynamic sql / EXECUTE. See THIS ANSWER how we can concatenate a string using logic like above so that the string contains an sql query and then execute the query. The information schema will give you the info you need to get a list of all current table names
But all you're doing is working around the fact that your data modelling is broken; I recommend to fix that instead
ps: the INFORMATION_SCHEMA has rough counts for tables with their names, which may suffice for your needs in this particular case
select table_name, table_rows from infornation_schema.tables where table_name like ...
I managed to solve the problem using the following stored procedure.
-- DROP PROCEDURE sp_Count_Rows;
Delimiter $$
CREATE PROCEDURE sp_Count_Rows()
BEGIN
DECLARE table_name TEXT DEFAULT "";
DECLARE finished INTEGER DEFAULT 0;
DECLARE table_cursor
CURSOR FOR
SELECT tblnm FROM alltbls;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET finished = 1;
OPEN table_cursor;
DROP TABLE IF EXISTS RowsCount;
CREATE TABLE IF NOT EXISTS RowsCount(Tlbnm text, ctnRws int);
table_loop: LOOP
FETCH table_cursor INTO table_name;
IF finished = 1 THEN
LEAVE table_loop;
END IF;
SET #s = CONCAT("insert into RowsCount select '", table_name ,"', count(*) as cntRws from ", table_name);
PREPARE stmt1 FROM #s;
EXECUTE stmt1;
DEALLOCATE PREPARE stmt1;
END LOOP table_loop;
CLOSE table_cursor;
SELECT * FROM RowsCount;
DROP TABLE RowsCount;
END
$$
And then when you call the procedure
CALL sp_Count_Rows();
You get this result

Use string from select to select a specific column in mysql

I'm trying to select a column based on a string I get from another select:
SELECT id, (
SELECT COLUMN_NAME
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA = 'database_name'
AND TABLE_NAME = 'a'
AND DATA_TYPE = 'varchar'
ORDER BY `COLUMNS`.`ORDINAL_POSITION` ASC LIMIT 1) AS `name`
FROM `a`
The problem is like this I'm getting only the string from the sub select not the column content of the main query.
(what I want to do in this case, is to return the value of the first column that is varchar)
You can do this with a dynamic query constructed in a stored procedure which you then call with the appropriate parameters.
I'd put the example in a sqlfiddle if I could find a way of querying the information_schema but you can just copy/paste this into your normal client and it should work.
I've gone a little further than your example by allowing you to select the id and first varchar column from any table but you can easily adjust the procedure and hardcode the database and table names if that's all you require.
CREATE DATABASE IF NOT EXISTS `mydb`;
USE `mydb`;
CREATE TABLE IF NOT EXISTS `mydb`.`tableA` (
id INT auto_increment primary key,
char_col char(3),
varchar_col varchar(25)
);
-- clear out any existing records
TRUNCATE `mydb`.`tableA`;
INSERT INTO `mydb`.`tableA` VALUES (null,'abc','varchar value abc');
INSERT INTO `mydb`.`tableA` VALUES (null,'def','varchar value def');
INSERT INTO `mydb`.`tableA` VALUES (null,'ghi','varchar value ghi');
INSERT INTO `mydb`.`tableA` VALUES (null,'klm','varchar value klm');
DELIMITER //
DROP PROCEDURE IF EXISTS `mydb`.`dyntest` //
CREATE PROCEDURE `mydb`.`dyntest` (IN dbname VARCHAR(64), IN tname VARCHAR(64))
BEGIN
DECLARE colname VARCHAR(64);
-- get the column name (as your example)
SELECT `COLUMN_NAME` INTO colname
FROM `information_schema`.`COLUMNS`
WHERE `TABLE_SCHEMA` = dbname
AND `TABLE_NAME` = tname
AND `DATA_TYPE` = 'VARCHAR'
ORDER BY `COLUMNS`.`ORDINAL_POSITION` ASC LIMIT 1;
-- construct the query
SET #sqlquery = CONCAT('SELECT `id`,`', colname , '` FROM `' , dbname, '`.`', tname, '`');
-- Prepare and execute
PREPARE stmt FROM #sqlquery;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END //
DELIMITER ;
This returns
mysql> CALL `mydb`.`dyntest`('mydb','tableA');
+----+-------------------+
| id | varchar_col |
+----+-------------------+
| 1 | varchar value abc |
| 2 | varchar value def |
| 3 | varchar value ghi |
| 4 | varchar value klm |
+----+-------------------+
4 rows in set (0.06 sec)

MYSQL Stored Procedure failed on second run

I have a stored procedure to create a roster table. Source is a view which collects the needed data. The stored procedure should prepare the data to be displayed to the team. Either for a specific team member or an overview for all. When I call the procedure the first time anything is fine and I receive the data as expected. But on the second run with a different parameter it fails with an error 'SQL Error [1054] [42S22]: (conn:218) Unknown column 'd0294220.duties.TM 3' in 'field list' Query is : call somp_roster('Ritchie')'
The missing column depends on the parameter in the first run.
The view collects the data from 4 tables:
CREATE VIEW somv_roster AS
SELECT bd.businessday AS Datum,
v.venue AS Dropzone,
d.duty AS Service,
c.nickname AS Staff
FROM som_roster r
INNER JOIN som_businessday bd ON bd.businessdayID = r.ID_businessday
INNER JOIN som_venue v ON v.venueID = bd.ID_venue
INNER JOIN som_duty d ON d.dutyID = r.ID_duty
INNER JOIN som_contact c ON c.contactID = r.ID_contact
ORDER BY bd.businessday, v.venue
WHERE bd.businessday >= CURDATE();
Result looks like this:
Datum Dropzone Service Staff
2018-04-28 Illertissen TM 2 Manu
2018-04-28 Illertissen Packer Martina B.
2018-04-28 Illertissen TM 1 Rued
2018-04-28 Illertissen TM 3 Hane
2018-04-29 Illertissen Manifest Sissi
2018-04-29 Illertissen TM 2 Ritchie
2018-04-29 Illertissen Packer Martina B.
2018-04-29 Illertissen TM 1 Rued
2018-04-29 Illertissen TM 3 Hane
2018-05-01 Illertissen TM 1 Ritchie
2018-05-01 Illertissen TM 3 Hane
2018-05-01 Illertissen TM 2 Purzl
2018-05-01 Illertissen Packer Martina B.
2018-05-26 Illertissen TM 1 Rued
2018-05-26 Kempten TM 1 Ritchie
2018-05-26 Kempten TM 2 Hane
2018-05-26 Kempten Manifest Sissi
Now I'm going to build a new table which should show this:
Datum Dropzone Manifest Packer TM 1 TM 2 TM 3
2018-04-28 Illertissen [NULL] Martina B. Rued Manu Hane
2018-04-29 Illertissen Sissi Martina B. Rued Ritchie Hane
2018-05-01 Illertissen [NULL] Martina B. Ritchie Purzl Hane
2018-05-26 Illertissen [NULL] [NULL] Rued [NULL] [NULL]
2018-05-26 Kempten Sissi [NULL] Ritchie Hane [NULL]
Or for a specific team member it should look like this:
Datum Dropzone TM 2 TM 3
2018-04-28 Illertissen [NULL] Hane
2018-04-29 Illertissen [NULL] Hane
2018-05-01 Illertissen [NULL] Hane
2018-05-26 Kempten Hane [NULL]
In the stored procedure I define a cursor and loop through that cursor. In a temp table I add the results and display the table at the end.
The stored procedure:
/*
* Create routine somp_roster()
* Requires somp_roster_et() to expand the columns dynamically.
* Requires somv_roster view to read the rosters.
* 4Air 2018 (c)
*/
DROP PROCEDURE IF EXISTS somp_roster;
DELIMITER $$
CREATE PROCEDURE somp_roster(IN staff_name VARCHAR(25))
BEGIN
/*
* variables have to be declared at the start of the procedure
* declare variables for the loops
* flag to be set at the end of the loop
*/
DECLARE finishedloop INTEGER DEFAULT 0;
-- flag to loop through the duty table to
DECLARE temp_date DATE DEFAULT NULL;
DECLARE temp_venue VARCHAR(25) DEFAULT '';
DECLARE temp_duty VARCHAR(25) DEFAULT '';
DECLARE temp_staff VARCHAR(25) DEFAULT '';
-- this var holds the string for the alter table statement
DECLARE alttable VARCHAR(250) DEFAULT '';
-- the first cursor to prepare the temp table
DECLARE roster_cursor CURSOR FOR
SELECT * FROM somv_roster WHERE Datum >= CURDATE() AND Staff LIKE CONCAT('%',staff_name) ORDER BY Service;
-- to recognize the end of the loops. Set to 0 before starting the loop!
DECLARE CONTINUE HANDLER FOR NOT FOUND SET finishedloop = 1;
-- cleanup first from a interrupted previous run
DROP TEMPORARY TABLE IF EXISTS duties;
-- now we need a temp table for the duties
CREATE TEMPORARY TABLE duties (Datum DATE, Dropzone VARCHAR(25), PRIMARY KEY(Datum, Dropzone)) ENGINE=MEMORY DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;
-- loop through the temp table
SET finishedloop = 0;
OPEN roster_cursor;
SET #alttable = '';
get_roster_loop: LOOP
-- now read the duties and create the new rows in the temp rosters
FETCH roster_cursor INTO temp_date, temp_venue, temp_duty, temp_staff;
IF finishedloop = 1 THEN
LEAVE get_roster_loop;
END IF;
/*
* fist step - try to expand the table with needed column
* the statement has to be this:
* call somp_rosters_et('tablename','columnname','columndefs')
* example:
* CALL somp_rosters_et('duties','TM 1','VARCHAR(25)');
*/
CALL somp_roster_et('duties',temp_duty,'VARCHAR(25)');
/*
* next prepare the statement to fill the data...
* implies the test if the date and venue already exists
* if so, then use UPDATE TABLE rather then INSERT TABLE
*/
SET #alttable = CONCAT('INSERT INTO duties (`Datum`,`Dropzone`,`', temp_duty , '`) VALUES (''' , temp_date , ''',''' , temp_venue , ''',''' , temp_staff , ''') ON DUPLICATE KEY UPDATE `', temp_duty , '` = ''', temp_staff , ''';') ;
PREPARE stmt FROM #alttable;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END LOOP get_roster_loop;
CLOSE roster_cursor;
SELECT * FROM duties ORDER BY Datum, Dropzone;
DROP TEMPORARY TABLE IF EXISTS duties;
END $$
DELIMITER ;
The second stored procedure to add the needed column to the temp table:
I build this to have an second handler for the sql exception when I try to add an existing column.
/*
* Stored Procedure to expand the table and do not care about an error if the column exists.
* Used by somp_roster();
* 4Air 2018 (c).
*/
DROP PROCEDURE IF EXISTS somp_roster_et;
DELIMITER $$
CREATE PROCEDURE somp_roster_et(IN t_name VARCHAR(25), IN c_name VARCHAR(25), IN c_attribute VARCHAR(25))
BEGIN
DECLARE eflag INTEGER DEFAULT 0;
DECLARE modtable VARCHAR(150) DEFAULT "";
DECLARE CONTINUE HANDLER FOR SQLEXCEPTION SET eflag = 1;
SET #modtable = CONCAT('ALTER TABLE ' , t_name , ' ADD COLUMN `' , c_name , '` ' , c_attribute , ';');
-- INSERT INTO som_log (Entry) VALUES(CONCAT('somp_roster_et: ',#modtable));
PREPARE stmt FROM #modtable;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END $$
DELIMITER ;
This works for the first run. On the second run with a different team member as a parameter for the first procedure the SELECT statement at the end of the procedure throws an error:
SQL Error [1054] [42S22]: (conn:218) Unknown column 'd0294220.duties.TM 3' in 'field list' Query is : call somp_roster('Ritchie')
When I remove the last SELECT from the procedure and also remove the drop table command I can receive the data manually:
call somp_roster('Ritchie');
SELECT * FROM duties;
call somp_roster('Martina B.');
SELECT * FROM duties;
This works fine. Why do I get the exception when I select the data inside the procedure?
Richard
2018-dec-04
Still fighting with this... I've changed the procedure a bit but the strange behavior is still present.
New procedure:
/*
* Create routine somp_roster()
*
* 2018-02-15 / implement a second handler for sql exceptions to handle the expand and select in this script
*
* somp_roster_et is depricated with this change.
*
* Requires somv_roster view to read the rosters.
* 4Air 2018 (c)
*/
DROP PROCEDURE IF EXISTS somp_roster;
DELIMITER $$
CREATE PROCEDURE somp_roster(IN staff_name VARCHAR(25))
BEGIN
/*
* variables have to be declared at the start of the procedure
* declare variables for the loops
* flag to be set at the end of the loop
*/
DECLARE finishedloop,eflag, loopcount INTEGER DEFAULT 0;
-- flag to loop through the duty table to
DECLARE temp_date DATE DEFAULT NULL;
DECLARE temp_venue, temp_duty, temp_staff VARCHAR(25) DEFAULT '';
-- this var holds the string for the alter table statement
DECLARE alttable,modtable VARCHAR(250) DEFAULT '';
-- the first cursor to prepare the temp table
DECLARE roster_cursor CURSOR FOR
SELECT * FROM somv_roster WHERE date >= CURDATE() AND staff LIKE CONCAT('%',staff_name) ORDER BY service;
-- to recognize the end of the loops. Set to 0 before starting the loop!
DECLARE CONTINUE HANDLER FOR NOT FOUND SET finishedloop = 1;
-- to recognize a sql exception and continue
DECLARE CONTINUE HANDLER FOR SQLEXCEPTION SET eflag = 1;
-- cleanup first from a interrupted previous run
DROP TEMPORARY TABLE IF EXISTS duties;
-- now we need a temp table for the duties
CREATE TEMPORARY TABLE duties (Datum DATE, Dropzone VARCHAR(25), PRIMARY KEY(Datum, Dropzone)) ENGINE=MEMORY DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;
-- loop through the temp table
SET finishedloop = 0;
OPEN roster_cursor;
SET #alttable = '';
get_roster_loop: LOOP
-- now read the duties and create the new rows in the temp rosters
FETCH roster_cursor INTO temp_date, temp_venue, temp_duty, temp_staff;
IF finishedloop = 1 THEN
LEAVE get_roster_loop;
END IF;
SET #modtable = CONCAT('ALTER TABLE duties ADD COLUMN `' , temp_duty , '` VARCHAR(25);');
PREPARE stmt FROM #modtable;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
/*
* next prepare the statement to fill the data...
* implies the test if the date and venue already exists
* if so, then use UPDATE TABLE rather then INSERT TABLE
*/
SET #alttable = CONCAT('INSERT INTO duties (`Datum`,`Dropzone`,`', temp_duty , '`) VALUES (''' , temp_date , ''',''' , temp_venue , ''',''' , temp_staff , ''') ON DUPLICATE KEY UPDATE `', temp_duty , '` = ''', temp_staff , ''';') ;
PREPARE stmt FROM #alttable;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END LOOP get_roster_loop;
CLOSE roster_cursor;
SELECT * FROM duties ORDER BY Datum, Dropzone;
DROP TEMPORARY TABLE IF EXISTS duties;
END $$
DELIMITER ;
Now I changed the temporary table to a persistent INNODB table to see what the procedure creates and this is always the result I expect. Now I change the SELECT statement at the end to this:
SELECT COUNT(*) FROM duties;
and I always get the correct count.
Any ideas what is the reason for this?
Richard

Insertion in MySQL takes a lot of time

I have a lot of tables and few tables have around 20 million records.
Now I do some calculations on these tables before showing the result to the UI.
For that I have created stored procedure.
In the stored procedure I am using temprorary tables to store select query records, do join, do processing and then return result.
I have written query like this
Insert INTO A
SELECT * from B JOIN c ....
Now my select query here does not take time (I have optimize it using explain extended) but my Insert into takes a lot of time since the output of select is in million. My table A is temprorary.
And Also I have do some processing on this A table after that.
My query is is there a way where I can skip inserting this insert.
I can take the result in a seperate table variable(if any) and then do processing instead of inserting all that in temprorary and then again doing some processing on it.
Adding my stored procedure here
DROP PROCEDURE IF EXISTS 6_4_1n2_PortUtil_temp;
DELIMITER |
CREATE PROCEDURE 6_4_1n2_PortUtil_temp(utilType VARCHAR(100),service varchar(5000),p_networkType VARCHAR(20),inputCity varchar(5000),inputNodeName varchar(5000), startTime timestamp,endTime timestamp)
BEGIN
DROP TEMPORARY TABLE IF EXISTS TEMP_SERVICE_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEIF_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEIF_TBL_1;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEName_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEANDIF_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_ROUTERTRAFFIC_VLANPRT_SCALE1_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_TRAFFIC_TBL;
DROP TEMPORARY TABLE IF EXISTS TRAFFIC_TBL;
DROP TEMPORARY TABLE IF EXISTS FINAL_FETCH;
SELECT now();
CREATE TEMPORARY TABLE TEMP_SERVICE_TBL(nodeName varchar(256),NodeNumber int) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_NODEName_TBL(nodeName varchar(256),NodeNumber int) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_NODEIF_TBL(NodeNumber int,IfIndex INTEGER,IfSpeed FLOAT,IfDescr VARCHAR(100),IfAlias VARCHAR(100)) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_NODEIF_TBL_1(NodeNumber int,IfIndex INTEGER,IfSpeed FLOAT,IfDescr VARCHAR(100),IfAlias VARCHAR(100)) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_NODEANDIF_TBL(NodeName Varchar(256),NodeNumber int,IfIndex INTEGER,IfSpeed FLOAT,IfDescr VARCHAR(100),IfAlias VARCHAR(100),PortID BIGINT(20) DEFAULT 0) ENGINE INNODB;
CREATE TEMPORARY TABLE TRAFFIC_TBL(
PortID BIGINT(20),
NodeName VARCHAR(100),
IfDescr VARCHAR(100),
IfSpeed VARCHAR(100),
InErrPkts BIGINT(20),
RcvOctets BIGINT(20),
TxOctets BIGINT(20),
Time_1 TIMESTAMP) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_TRAFFIC_TBL(
PortID BIGINT(20),
maxTrafficValueIn BIGINT(20) DEFAULT 0,
maxOutTrafficValueOut BIGINT(20) DEFAULT 0,
avgTrafficValueIn BIGINT(20) DEFAULT 0,
avgTrafficValueOut BIGINT(20) DEFAULT 0,
CRCError BIGINT(20) DEFAULT 0,
UpTime INTEGER DEFAULT 0,
Reliablity INTEGER DEFAULT 0,
AvgUtilIn float DEFAULT 0,
AvgUtilOut float DEFAULT 0,
PeakUtilIn float DEFAULT 0,
PeakUtilOut float DEFAULT 0,
ThresholdExceed INTEGER DEFAULT 0,
inPeakTime timestamp DEFAULT '0000-00-00 00:00:00',
outPeakTime timestamp DEFAULT '0000-00-00 00:00:00') ENGINE INNODB;
SET #where = '';
IF service='ALL'
THEN
SET #where = '';
ELSE
set #a=1;
set #like="";
select REPLACE(SUBSTRING(SUBSTRING_INDEX(service, ',', #a),LENGTH(SUBSTRING_INDEX(service, ',', #a -1)) + 1),',','') into #service;
while(#service != "")
DO
IF(#like = "")
THEN
SET #like = CONCAT("NodeName like '%",SUBSTRING(#service,2,3),"%'");
ELSE
SET #like = CONCAT(#like," or NodeName like '%",SUBSTRING(#service,2,3),"%'");
END IF;
set #a=#a+1;
select REPLACE(SUBSTRING(SUBSTRING_INDEX(service, ',', #a),LENGTH(SUBSTRING_INDEX(service, ',', #a -1)) + 1),',','') into #service;
END WHILE;
SET #where = CONCAT(" where",#like);
END IF;
Set #where2 = '';
IF inputCity='ALL'
THEN
set #where2 = '';
ELSE
set #where2 = CONCAT(' and substring(NodeName,1,3) in (',inputCity,')');
END IF;
SET #where3 = '';
IF inputNodeName='ALL'
THEN
set #where3 = '';
ELSE
set #where3 = CONCAT(' and NodeName in (',inputNodeName,')');
END IF;
SET #query1 := CONCAT("INSERT INTO TEMP_NODEName_TBL SELECT distinct NodeName,NodeNumber from NODE_TBL",#where, #where2, #where3);
SELECT #query1;
PREPARE statement1 from #query1;
EXECUTE statement1;
DEALLOCATE Prepare statement1;
CREATE INDEX n1 ON TEMP_NODEName_TBL(NodeNumber);
CREATE INDEX i1 ON TEMP_NODEIF_TBL(NodeNumber,IfIndex);
CREATE INDEX portIDIndex1 on TEMP_NODEANDIF_TBL(PortID);
SET #where4 = '';
IF (utilType='ALL')
THEN
SET #where = '';
ELSE
If (utilType = "'AESI-IN'")
THEN
SET #where4 = " where IfAlias like '%AESI-IN%'";
ELSE
SET #where4 = " where IfAlias NOT like '%AESI-IN%'";
END IF;
END IF;
CREATE INDEX i2 ON TEMP_NODEIF_TBL_1(NodeNumber,IfIndex);
CREATE INDEX i3 ON TEMP_NODEANDIF_TBL(NodeNumber,IfIndex);
SET #where5 = '';
IF(p_networkType != 'ALL')
THEN
set #r1= SUBSTRING(p_networkType,2,3);
set #r2= SUBSTRING(p_networkType,8,3);
set #r3= SUBSTRING(p_networkType,14,3);
SET #where5 = CONCAT(" and IfAlias like '%",#r1,"%'");
if(#r2 != "")
THEN
SET #where5 = CONCAT(" and IfAlias like '%",#r2,"%'");
IF(#r3 != "")
THEN
SET #where5 = CONCAT(" and IfAlias like '%",#r3,"%'");
END IF;
END IF;
END IF;
SET #query2 := CONCAT("INSERT INTO TEMP_NODEANDIF_TBL(NodeName,NodeNumber,IfIndex,IfSpeed,IfDescr,IfAlias) SELECT distinct b.NodeName, a.NodeNumber, a.IfIndex,a.IfSpeed,a.IfDescr,a.IfAlias from NODEIF_TBL a JOIN TEMP_NODEName_TBL b ON a.NodeNumber = b.NodeNumber ", #where4, #where5);
SELECT #query2;
PREPARE statement1 from #query2;
EXECUTE statement1;
DEALLOCATE Prepare statement1;
SELECT "DROP TEMPORARY TABLES";
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEIF_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEIF_TBL_1;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEName_TBL;
update TEMP_NODEANDIF_TBL a,VLANPRT_TBL b set a.PortID = PrtID where a.NodeNumber = b.NodeID and a.IfIndex = b.IfIndex;
SELECT "Update Temporary tables";
delete from TEMP_NODEANDIF_TBL where PortID = 0;
SELECT now();
INSERT INTO TRAFFIC_TBL
SELECT a.PortID,NodeName,IfDescr,IfSpeed,InErrPkts,RcvOctets,TxOctets,Time_1 FROM ROUTERTRAFFIC_VLANPRT_SCALE1_TBL a JOIN TEMP_NODEANDIF_TBL b ON a.PortID = b.PortID where Time_1>startTime and Time_1<endTime;
SELECT now();
DROP TEMPORARY TABLE IF EXISTS EXCEED_COUNT;
CREATE TEMPORARY TABLE EXCEED_COUNT (PortID BIGINT(20), Exceed INTEGER);
INSERT INTO EXCEED_COUNT
select PortID,count(RcvOctets) from TRAFFIC_TBL where (RcvOctets/(IfSpeed*10)>70 or TxOctets/(IfSpeed*10)>70) group by PortID;
INSERT INTO TEMP_TRAFFIC_TBL (PortID, maxTrafficValueIn,maxOutTrafficValueOut,avgTrafficValueIn,avgTrafficValueOut,CRCError,AvgUtilIn,AvgUtilOut,PeakUtilIn,PeakUtilOut)
SELECT PortID,max(RcvOctets),max(TxOctets),avg(RcvOctets),avg(TxOctets), sum(InErrPkts),
IF((IfSpeed=0),"0",(avg(RcvOctets)/(IfSpeed*10))),
IF((IfSpeed=0),"0",(avg(TxOctets)/(IfSpeed*10))),
IF((IfSpeed=0),"0",(max(RcvOctets)/(IfSpeed*10))),
IF((IfSpeed=0),"0",(max(TxOctets)/(IfSpeed*10)))
from TRAFFIC_TBL group by PortID;
CREATE INDEX portIDIndex2 on TEMP_TRAFFIC_TBL(PortID);
CREATE INDEX portIDIndex3 on EXCEED_COUNT(PortID);
UPDATE TEMP_TRAFFIC_TBL A JOIN TRAFFIC_TBL B ON A.PortId=B.PortId SET inPeakTime=B.Time_1 where B.RcvOctets=A.maxTrafficValueIn;
UPDATE TEMP_TRAFFIC_TBL A JOIN TRAFFIC_TBL B ON A.PortId=B.PortId SET outPeakTime=B.Time_1 where B.TxOctets=A.maxOutTrafficValueOut;
UPDATE TEMP_TRAFFIC_TBL A JOIN EXCEED_COUNT B ON (A.PortID = B.PortID)
set ThresholdExceed = Exceed;
SELECT substring(NodeName,5,3) as ServiceType,
CASE
WHEN IfAlias like '%SWH%' THEN "Trunk"
WHEN IfAlias like '%AES%' THEN "Backbone"
WHEN IfAlias like '%RTR%' THEN "Back-to-Back"
ELSE "-"
END ,
NodeName,IfDescr,ROUND(maxTrafficValueIn/1000,2),ROUND(maxOutTrafficValueOut/1000,2),ROUND(avgTrafficValueIn/1000,2),ROUND(avgTrafficValueOut/1000,2),ROUND(CRCError/1000,2),0,0,ROUND(AvgUtilIn,2),ROUND(AvgUtilOut,2),ROUND(PeakUtilIn,2),ROUND(PeakUtilOut,2),ThresholdExceed,inPeakTime,outPeakTime from TEMP_TRAFFIC_TBL a ,TEMP_NODEANDIF_TBL b where a.PortID = b.PortID ;
SELECT now();
END |
DELIMITER ;
My ROUTERTRAFFIC_VLANPRT_SCALE1_TBL contains around 20 million records and NodeIF_TBL around 1 lakhs records. My VLANPRT_TBL also contains around 1 lakh records.
Also I have BTree indexing set on time_1 so that the join on the big table do not take time
Temporary tables:
Temporary tables start out in the memory and if the surpass a certain limitation they will be written to disk. If the select yields millions and you're writing those millions to disk then that's your problem.
Needing a temporary table with millions of records is usually a sign of doing something wrong. Perhaps you should further filter the data before inserting it into the temporary table for processing.
Disk I/O thrashing:
We have a few INSERT cases:
INSERT INTO ... VALUES
INSERT INGORE INTO ... SELECT
INSERT INTO ... SELECT
For 1. each insert has its own transaction and writes to disk individually meaning lots of tiny I/O operations.
For 2. the same applies except the data is also being read from the disk (slightly more I/O operations).
For 3. the whole operation is one big transaction meaning one big disk I/O at the very end (the commit) of the transaction. More often it happens that the information contained in the big transaction cannot be held entirely in memory so it is written to disk temporarily before the final commit and then copied from there into the designated area where it should reside.
In all 3 cases disk I/O thrashing occurs, to prevent this you would group small operations into average sized operations, and break huge operations into average sized operations.
For 1. you need to wrap every N inserts in a transaction.
For 2. and 3. you need to LIMIT the SELECT clause to N and repeat the INSERT INTO ... SELECT with an added OFFSET of N until you finish all records.
Either should be fairly easy to do if you're using a scripting language to run your queries.
Its not needed to store in a temporary table the output from a query in order to use it in another query. You can make this:
Select t1.* from (select * from A where condition1)as t1 where condition2
Here is an example of a query i had to do.
select avg(v1) as v1avg ,start_date_time as time, timekey as time_key from (select ATable.*, ROUND(UNIX_TIMESTAMP(start_date_time)/(60*60)) as timekey from ATable where start_date_time between '2014-01-01 00:00:00' and '2014-01-10 00:00:00')as t1 group by timekey;