Insertion in MySQL takes a lot of time - mysql

Question

I have a lot of tables and few tables have around 20 million records.
Now I do some calculations on these tables before showing the result to the UI.
For that I have created stored procedure.
In the stored procedure I am using temprorary tables to store select query records, do join, do processing and then return result.
I have written query like this
Insert INTO A
SELECT * from B JOIN c ....
Now my select query here does not take time (I have optimize it using explain extended) but my Insert into takes a lot of time since the output of select is in million. My table A is temprorary.
And Also I have do some processing on this A table after that.
My query is is there a way where I can skip inserting this insert.
I can take the result in a seperate table variable(if any) and then do processing instead of inserting all that in temprorary and then again doing some processing on it.
Adding my stored procedure here
DROP PROCEDURE IF EXISTS 6_4_1n2_PortUtil_temp;
DELIMITER |
CREATE PROCEDURE 6_4_1n2_PortUtil_temp(utilType VARCHAR(100),service varchar(5000),p_networkType VARCHAR(20),inputCity varchar(5000),inputNodeName varchar(5000), startTime timestamp,endTime timestamp)
BEGIN
DROP TEMPORARY TABLE IF EXISTS TEMP_SERVICE_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEIF_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEIF_TBL_1;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEName_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEANDIF_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_ROUTERTRAFFIC_VLANPRT_SCALE1_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_TRAFFIC_TBL;
DROP TEMPORARY TABLE IF EXISTS TRAFFIC_TBL;
DROP TEMPORARY TABLE IF EXISTS FINAL_FETCH;
SELECT now();
CREATE TEMPORARY TABLE TEMP_SERVICE_TBL(nodeName varchar(256),NodeNumber int) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_NODEName_TBL(nodeName varchar(256),NodeNumber int) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_NODEIF_TBL(NodeNumber int,IfIndex INTEGER,IfSpeed FLOAT,IfDescr VARCHAR(100),IfAlias VARCHAR(100)) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_NODEIF_TBL_1(NodeNumber int,IfIndex INTEGER,IfSpeed FLOAT,IfDescr VARCHAR(100),IfAlias VARCHAR(100)) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_NODEANDIF_TBL(NodeName Varchar(256),NodeNumber int,IfIndex INTEGER,IfSpeed FLOAT,IfDescr VARCHAR(100),IfAlias VARCHAR(100),PortID BIGINT(20) DEFAULT 0) ENGINE INNODB;
CREATE TEMPORARY TABLE TRAFFIC_TBL(
PortID BIGINT(20),
NodeName VARCHAR(100),
IfDescr VARCHAR(100),
IfSpeed VARCHAR(100),
InErrPkts BIGINT(20),
RcvOctets BIGINT(20),
TxOctets BIGINT(20),
Time_1 TIMESTAMP) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_TRAFFIC_TBL(
PortID BIGINT(20),
maxTrafficValueIn BIGINT(20) DEFAULT 0,
maxOutTrafficValueOut BIGINT(20) DEFAULT 0,
avgTrafficValueIn BIGINT(20) DEFAULT 0,
avgTrafficValueOut BIGINT(20) DEFAULT 0,
CRCError BIGINT(20) DEFAULT 0,
UpTime INTEGER DEFAULT 0,
Reliablity INTEGER DEFAULT 0,
AvgUtilIn float DEFAULT 0,
AvgUtilOut float DEFAULT 0,
PeakUtilIn float DEFAULT 0,
PeakUtilOut float DEFAULT 0,
ThresholdExceed INTEGER DEFAULT 0,
inPeakTime timestamp DEFAULT '0000-00-00 00:00:00',
outPeakTime timestamp DEFAULT '0000-00-00 00:00:00') ENGINE INNODB;
SET #where = '';
IF service='ALL'
THEN
SET #where = '';
ELSE
set #a=1;
set #like="";
select REPLACE(SUBSTRING(SUBSTRING_INDEX(service, ',', #a),LENGTH(SUBSTRING_INDEX(service, ',', #a -1)) + 1),',','') into #service;
while(#service != "")
DO
IF(#like = "")
THEN
SET #like = CONCAT("NodeName like '%",SUBSTRING(#service,2,3),"%'");
ELSE
SET #like = CONCAT(#like," or NodeName like '%",SUBSTRING(#service,2,3),"%'");
END IF;
set #a=#a+1;
select REPLACE(SUBSTRING(SUBSTRING_INDEX(service, ',', #a),LENGTH(SUBSTRING_INDEX(service, ',', #a -1)) + 1),',','') into #service;
END WHILE;
SET #where = CONCAT(" where",#like);
END IF;
Set #where2 = '';
IF inputCity='ALL'
THEN
set #where2 = '';
ELSE
set #where2 = CONCAT(' and substring(NodeName,1,3) in (',inputCity,')');
END IF;
SET #where3 = '';
IF inputNodeName='ALL'
THEN
set #where3 = '';
ELSE
set #where3 = CONCAT(' and NodeName in (',inputNodeName,')');
END IF;
SET #query1 := CONCAT("INSERT INTO TEMP_NODEName_TBL SELECT distinct NodeName,NodeNumber from NODE_TBL",#where, #where2, #where3);
SELECT #query1;
PREPARE statement1 from #query1;
EXECUTE statement1;
DEALLOCATE Prepare statement1;
CREATE INDEX n1 ON TEMP_NODEName_TBL(NodeNumber);
CREATE INDEX i1 ON TEMP_NODEIF_TBL(NodeNumber,IfIndex);
CREATE INDEX portIDIndex1 on TEMP_NODEANDIF_TBL(PortID);
SET #where4 = '';
IF (utilType='ALL')
THEN
SET #where = '';
ELSE
If (utilType = "'AESI-IN'")
THEN
SET #where4 = " where IfAlias like '%AESI-IN%'";
ELSE
SET #where4 = " where IfAlias NOT like '%AESI-IN%'";
END IF;
END IF;
CREATE INDEX i2 ON TEMP_NODEIF_TBL_1(NodeNumber,IfIndex);
CREATE INDEX i3 ON TEMP_NODEANDIF_TBL(NodeNumber,IfIndex);
SET #where5 = '';
IF(p_networkType != 'ALL')
THEN
set #r1= SUBSTRING(p_networkType,2,3);
set #r2= SUBSTRING(p_networkType,8,3);
set #r3= SUBSTRING(p_networkType,14,3);
SET #where5 = CONCAT(" and IfAlias like '%",#r1,"%'");
if(#r2 != "")
THEN
SET #where5 = CONCAT(" and IfAlias like '%",#r2,"%'");
IF(#r3 != "")
THEN
SET #where5 = CONCAT(" and IfAlias like '%",#r3,"%'");
END IF;
END IF;
END IF;
SET #query2 := CONCAT("INSERT INTO TEMP_NODEANDIF_TBL(NodeName,NodeNumber,IfIndex,IfSpeed,IfDescr,IfAlias) SELECT distinct b.NodeName, a.NodeNumber, a.IfIndex,a.IfSpeed,a.IfDescr,a.IfAlias from NODEIF_TBL a JOIN TEMP_NODEName_TBL b ON a.NodeNumber = b.NodeNumber ", #where4, #where5);
SELECT #query2;
PREPARE statement1 from #query2;
EXECUTE statement1;
DEALLOCATE Prepare statement1;
SELECT "DROP TEMPORARY TABLES";
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEIF_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEIF_TBL_1;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEName_TBL;
update TEMP_NODEANDIF_TBL a,VLANPRT_TBL b set a.PortID = PrtID where a.NodeNumber = b.NodeID and a.IfIndex = b.IfIndex;
SELECT "Update Temporary tables";
delete from TEMP_NODEANDIF_TBL where PortID = 0;
SELECT now();
INSERT INTO TRAFFIC_TBL
SELECT a.PortID,NodeName,IfDescr,IfSpeed,InErrPkts,RcvOctets,TxOctets,Time_1 FROM ROUTERTRAFFIC_VLANPRT_SCALE1_TBL a JOIN TEMP_NODEANDIF_TBL b ON a.PortID = b.PortID where Time_1>startTime and Time_1<endTime;
SELECT now();
DROP TEMPORARY TABLE IF EXISTS EXCEED_COUNT;
CREATE TEMPORARY TABLE EXCEED_COUNT (PortID BIGINT(20), Exceed INTEGER);
INSERT INTO EXCEED_COUNT
select PortID,count(RcvOctets) from TRAFFIC_TBL where (RcvOctets/(IfSpeed*10)>70 or TxOctets/(IfSpeed*10)>70) group by PortID;
INSERT INTO TEMP_TRAFFIC_TBL (PortID, maxTrafficValueIn,maxOutTrafficValueOut,avgTrafficValueIn,avgTrafficValueOut,CRCError,AvgUtilIn,AvgUtilOut,PeakUtilIn,PeakUtilOut)
SELECT PortID,max(RcvOctets),max(TxOctets),avg(RcvOctets),avg(TxOctets), sum(InErrPkts),
IF((IfSpeed=0),"0",(avg(RcvOctets)/(IfSpeed*10))),
IF((IfSpeed=0),"0",(avg(TxOctets)/(IfSpeed*10))),
IF((IfSpeed=0),"0",(max(RcvOctets)/(IfSpeed*10))),
IF((IfSpeed=0),"0",(max(TxOctets)/(IfSpeed*10)))
from TRAFFIC_TBL group by PortID;
CREATE INDEX portIDIndex2 on TEMP_TRAFFIC_TBL(PortID);
CREATE INDEX portIDIndex3 on EXCEED_COUNT(PortID);
UPDATE TEMP_TRAFFIC_TBL A JOIN TRAFFIC_TBL B ON A.PortId=B.PortId SET inPeakTime=B.Time_1 where B.RcvOctets=A.maxTrafficValueIn;
UPDATE TEMP_TRAFFIC_TBL A JOIN TRAFFIC_TBL B ON A.PortId=B.PortId SET outPeakTime=B.Time_1 where B.TxOctets=A.maxOutTrafficValueOut;
UPDATE TEMP_TRAFFIC_TBL A JOIN EXCEED_COUNT B ON (A.PortID = B.PortID)
set ThresholdExceed = Exceed;
SELECT substring(NodeName,5,3) as ServiceType,
CASE
WHEN IfAlias like '%SWH%' THEN "Trunk"
WHEN IfAlias like '%AES%' THEN "Backbone"
WHEN IfAlias like '%RTR%' THEN "Back-to-Back"
ELSE "-"
END ,
NodeName,IfDescr,ROUND(maxTrafficValueIn/1000,2),ROUND(maxOutTrafficValueOut/1000,2),ROUND(avgTrafficValueIn/1000,2),ROUND(avgTrafficValueOut/1000,2),ROUND(CRCError/1000,2),0,0,ROUND(AvgUtilIn,2),ROUND(AvgUtilOut,2),ROUND(PeakUtilIn,2),ROUND(PeakUtilOut,2),ThresholdExceed,inPeakTime,outPeakTime from TEMP_TRAFFIC_TBL a ,TEMP_NODEANDIF_TBL b where a.PortID = b.PortID ;
SELECT now();
END |
DELIMITER ;
My ROUTERTRAFFIC_VLANPRT_SCALE1_TBL contains around 20 million records and NodeIF_TBL around 1 lakhs records. My VLANPRT_TBL also contains around 1 lakh records.
Also I have BTree indexing set on time_1 so that the join on the big table do not take time

Answer 1

Temporary tables:
Temporary tables start out in the memory and if the surpass a certain limitation they will be written to disk. If the select yields millions and you're writing those millions to disk then that's your problem.
Needing a temporary table with millions of records is usually a sign of doing something wrong. Perhaps you should further filter the data before inserting it into the temporary table for processing.
Disk I/O thrashing:
We have a few INSERT cases:
INSERT INTO ... VALUES
INSERT INGORE INTO ... SELECT
INSERT INTO ... SELECT
For 1. each insert has its own transaction and writes to disk individually meaning lots of tiny I/O operations.
For 2. the same applies except the data is also being read from the disk (slightly more I/O operations).
For 3. the whole operation is one big transaction meaning one big disk I/O at the very end (the commit) of the transaction. More often it happens that the information contained in the big transaction cannot be held entirely in memory so it is written to disk temporarily before the final commit and then copied from there into the designated area where it should reside.
In all 3 cases disk I/O thrashing occurs, to prevent this you would group small operations into average sized operations, and break huge operations into average sized operations.
For 1. you need to wrap every N inserts in a transaction.
For 2. and 3. you need to LIMIT the SELECT clause to N and repeat the INSERT INTO ... SELECT with an added OFFSET of N until you finish all records.
Either should be fairly easy to do if you're using a scripting language to run your queries.

Answer 2

Its not needed to store in a temporary table the output from a query in order to use it in another query. You can make this:
Select t1.* from (select * from A where condition1)as t1 where condition2
Here is an example of a query i had to do.
select avg(v1) as v1avg ,start_date_time as time, timekey as time_key from (select ATable.*, ROUND(UNIX_TIMESTAMP(start_date_time)/(60*60)) as timekey from ATable where start_date_time between '2014-01-01 00:00:00' and '2014-01-10 00:00:00')as t1 group by timekey;