I am trying to do something a little weird and cannot figure out the right method for getting it done. Essentially I am trying to pull all tables/views and columns where the column name is like some string. In addition to that I would like to pull 1 row of data from that table/view and column combination. The second part is where I am lost. I know I can pull the necessary tables/views and columns with the below select statement.
SELECT COLUMN_NAME AS 'ColumnName'
,TABLE_NAME AS 'TableName'
FROM INFORMATION_SCHEMA.COLUMNS
WHERE COLUMN_NAME LIKE '%email%'
ORDER BY TableName,ColumnName;
So that I get something like the below
|ColumnName |TableName |
|emailAddress |all_emails |
....
But I want to get something like this:
|ColumnName |TableName |Example |
|emailAddress |all_emails |first.last#gmail.com|
....
Can anyone offer any insight?
I can't think of a simple way to do this within a query, but here's one option...
Put the list of the columns and tables into a temp table and run them through a loop, using dynamic SQL to select the max row for each.
I've added plenty of comments below to explain it.
DECLARE #SQL NVARCHAR(1000)
DECLARE #TABLE NVARCHAR(1000)
DECLARE #COLUMN NVARCHAR(1000)
DECLARE #SAMPLE NVARCHAR(1000)
DROP TABLE IF EXISTS ##TABLELIST
SELECT COLUMN_NAME AS 'ColumnName'
,TABLE_NAME AS 'TableName'
,ROW_NUMBER() OVER (ORDER BY COLUMN_NAME,TABLE_NAME)[RN]
INTO ##TABLELIST
FROM INFORMATION_SCHEMA.COLUMNS
WHERE COLUMN_NAME LIKE '%email%';
ALTER TABLE ##TABLELIST
ADD [Sample] NVARCHAR(1000) -- Add a column for your sample row.
DECLARE #ROWCOUNTER INT = 1 -- Add a counter for the loop to use.
WHILE #ROWCOUNTER <= (SELECT MAX([RN]) FROM ##TABLELIST) -- Keep the loop running until the end of the list.
BEGIN
UPDATE ##TABLELIST
SET #TABLE = TableName WHERE [RN] = #ROWCOUNTER -- Get the table name into a variable.
UPDATE ##TABLELIST
SET #COLUMN = ColumnName WHERE [RN] = #ROWCOUNTER -- Get the column name into a variable.
SET #SQL = 'SELECT #SAMPLE = MAX([' + #COLUMN + ']) FROM [' + #TABLE + ']' -- Create SQL statement to pull max column from table specified in variables.
EXEC SP_EXECUTESQL #SQL, N'#SAMPLE NVARCHAR(1000) OUTPUT', #SAMPLE OUTPUT -- Execute SQL and put the output into the #SAMPLE variable.
UPDATE ##TABLELIST
SET [Sample] = CAST(#SAMPLE AS NVARCHAR(1000)) WHERE [RN] = #ROWCOUNTER -- Insert the SQL output into the sample column.
SET #ROWCOUNTER = #ROWCOUNTER+1 -- Add one to the row counter to move to the next column and table.
END
SELECT * FROM ##TABLELIST -- Select final output.
I have databases named company_abc, company_xyz, etc. Those company_* databases have all the same structure and they contain users table.
What I need to do is to aggregate all users data from just company_* databases and replicate this view to another server. The view would just be something like
COMPANY NAME | USERNAME
abc | user#email.com
abc | user1#email.com
xyz | user2#email.com
company3 | user3#email.com
Is something like that possible in MySQL?
The databases are created dynamically, as well as the users so I can't create a view with just a static set of databases.
As you say you want to create view with dynamic database names - so the result you want to achieve is not possible in current versions of mysql.
So you have example following options:
Option 1
If you want to get result of all databases users tables you could define a stored procedure that uses prepared statement. This procedure needs parameter db_prefix what in your case is company_%. Basicly this procedure selects all tables named as users from information_schema when database name is like db_prefix parameter value. After that it loops through results and creates query string as union all users tables and executes this query. When creating a query string i also add field called source, so i can identify from what database this result is coming. In my example my databases are all in default collation utf8_unicode_ci.
In this case you can define procedure example "getAllUsers"
-- Dumping structure for procedure company_abc1.getAllUsers
DELIMITER //
CREATE DEFINER=`root`#`localhost` PROCEDURE `getAllUsers`(IN `db_prefix` TEXT)
DETERMINISTIC
COMMENT 'test'
BEGIN
DECLARE qStr TEXT DEFAULT '';
DECLARE cursor_VAL VARCHAR(255) DEFAULT '';
DECLARE done INTEGER DEFAULT 0;
DECLARE cursor_i CURSOR FOR SELECT DISTINCT (table_schema) FROM information_schema.tables WHERE table_name = 'users' AND table_schema LIKE db_prefix COLLATE utf8_unicode_ci;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
OPEN cursor_i;
read_loop: LOOP
FETCH cursor_i INTO cursor_VAL;
IF done = 1 THEN
LEAVE read_loop;
END IF;
IF qStr != '' THEN
SET qStr = CONCAT(qStr, ' UNION ALL ');
END IF;
SET qStr = CONCAT(qStr, ' SELECT *, \'', cursor_VAL ,'\' as source FROM ', cursor_VAL, '.users');
END LOOP;
CLOSE cursor_i;
SET #qStr = qStr;
PREPARE stmt FROM #qStr;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
SET #qStr = NULL;
END//
DELIMITER ;
Now you can get your all users result as:
CALL getAllUsers('company_%');
In my example database it results as:
id name source
1 User 1 company_abc1
2 User 2 company_abc1
3 User 3 company_abc1
1 User 1 company_abc2
2 User 2 company_abc2
3 User 3 company_abc2
1 User 1 company_abc3
2 User 2 company_abc3
3 User 3 company_abc3
1 User 1 company_abc4
2 User 2 company_abc4
3 User 3 company_abc4
1 User 1 company_abc5
2 User 2 company_abc5
3 User 3 company_abc5
Option 2
If you really, really need view then you can modify first procedure and instead of executeing select you can create view. Example like this:
-- Dumping structure for procedure company_abc1.createAllUsersView
DELIMITER //
CREATE DEFINER=`root`#`localhost` PROCEDURE `createAllUsersView`(IN `db_prefix` TEXT)
DETERMINISTIC
COMMENT 'test'
BEGIN
DECLARE qStr TEXT DEFAULT '';
DECLARE cursor_VAL VARCHAR(255) DEFAULT '';
DECLARE done INTEGER DEFAULT 0;
DECLARE cursor_i CURSOR FOR SELECT DISTINCT (table_schema) FROM information_schema.tables WHERE table_name = 'users' AND table_schema LIKE db_prefix COLLATE utf8_unicode_ci;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
OPEN cursor_i;
read_loop: LOOP
FETCH cursor_i INTO cursor_VAL;
IF done = 1 THEN
LEAVE read_loop;
END IF;
IF qStr != '' THEN
SET qStr = CONCAT(qStr, ' UNION ALL ');
END IF;
SET qStr = CONCAT(qStr, ' SELECT *, \'', cursor_VAL ,'\' as source FROM ', cursor_VAL, '.users');
END LOOP;
CLOSE cursor_i;
SET #qStr = CONCAT('CREATE OR REPLACE VIEW allUsersView AS ', qStr);
PREPARE stmt FROM #qStr;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
SET #qStr = NULL;
END//
DELIMITER ;
In this stored procedure we create/replace view called allUsersView, so basicly every time you will execute this procedure it will updates view.
In my test case it creates view like this:
CREATE OR REPLACE VIEW `allusersview` AS
SELECT *, 'company_abc1' as source FROM company_abc1.users
UNION ALL SELECT *, 'company_abc2' as source FROM company_abc2.users
UNION ALL SELECT *, 'company_abc3' as source FROM company_abc3.users
UNION ALL SELECT *, 'company_abc4' as source FROM company_abc4.users
UNION ALL SELECT *, 'company_abc5' as source FROM company_abc5.users ;
And now you can use view.
SELECT * FROM allusersview
And result is same as in first option.
All tested on:
Mysql 5.6.16
To find the list of database names:
SELECT SCHEMA_NAME
FROM information_schema.`SCHEMATA`
WHERE SCHEMA_NAME LIKE 'company%';
If you can code in something like PHP, the rest is pretty easy -- build a UNION of SELECTs from each database. But, if you must do it just in SQL...
To build the UNION, write a Stored Procedure. It will do the above query in a CURSOR. Inside the loop that walks through the cursor, CONCAT() a constructed SELECT onto a UNION you are building.
When the loop is finished, PREPARE and EXECUTE the constructed UNION. That will deliver something like the output example you had.
But, if you now need to INSERT the results of that into another server, you should leave the confines of the Stored Procedure and use some other language.
OK, OK, if you must stay in SQL, then you need some setup: Create a "Federated" table that connects to the other server. Now, in your SP, concatenate INSERT INTO fed_tbl in front of the UNION. Then the execute should do the entire task.
If you have trouble with the FEDERATED Engine, you may need to switch to FederatedX in MariaDB.
"The details are left as an exercise to the reader."
I already marked this as duplicate of Mysql union from multiple database tables
(SELECT *, 'abc' as COMPANY_NAME from company_abc.users)
union
(SELECT *, 'xyz' as COMPANY_NAME from company_xyz.users)
union
(SELECT *, 'company3' as COMPANY_NAME from company_company3.users)
...
I think that the only method to make this is to write a stored procedure that read all database and table name from information_schema.table, build a string with union select * from company_abc.users union all select * from company_xyz and then execute the command with prepared statement: http://dev.mysql.com/doc/refman/5.0/en/sql-syntax-prepared-statements.html
I'm looking for a way to easily check each table of a MySQL database and make sure that a certain field contains one value only. I have tables named Authors, Titles, Places, etc.
Each table contains a field called xuser and it needs to ask "does the field xuser contain the value xy in all records of all tables".
Can someone push me in the right direction how to do this with a SQL query if this is possible?
Thanks for reading, regards
Nico
I've created stored procedure which checks all table for provided db:
DELIMITER $$
DROP PROCEDURE IF EXISTS `UTL_CHECK_BACKUP_FOR_USER` $$
CREATE PROCEDURE `UTL_CHECK_BACKUP_FOR_USER`(
IN i_database_name VARCHAR(255),
IN i_user_column_name VARCHAR(255),
IN i_user_column_value VARCHAR(255),
OUT o_result TINYINT(1)
)
BEGIN
DECLARE v_table_name VARCHAR(255);
DECLARE v_last_row_fetched TINYINT(3) DEFAULT 0;
DECLARE tables_cursor CURSOR FOR
SELECT table_name
FROM information_schema.tables
WHERE table_schema = i_database_name
;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET v_last_row_fetched = 1;
SET v_last_row_fetched = 0;
OPEN tables_cursor;
SET #query =
CONCAT(
'SELECT SUM(IF(user_column=''',
i_user_column_value,
''', 1, -1)) = 1 INTO #o_result FROM ( SELECT ''test'' AS user_column FROM information_schema.tables WHERE 1<>1 '
)
;
table_loop: LOOP
FETCH tables_cursor INTO v_table_name;
IF (v_last_row_fetched = 1) THEN
LEAVE table_loop;
END IF;
SET #query =
CONCAT(
#query,
' UNION SELECT DISTINCT(',
i_user_column_name,
') AS user_column FROM ',
v_table_name
)
;
END LOOP table_loop;
CLOSE tables_cursor;
SET v_last_row_fetched=0;
SET #query =
CONCAT(
#query,
' ) all_xusers;'
)
;
PREPARE stmt FROM #query;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
SET o_result = COALESCE(#o_result, 0);
END $$
DELIMITER ;
Just deploy this stored procedure to database.
And then it could be executed in the following way:
-- db_name, user_column_name, user_column_value, result
call UTL_CHECK_BACKUP_FOR_USER('test', 'xuser', 'xxx', #result);
select #result;
To get the rows from all three tables where xuser has the same value in all three tables you could use:
SELECT *
FROM authors a
JOIN titles t
ON t.xuser = a.xuser
JOIN places p
ON p.xuser = t.xuser
If you want to look at a specific xuser value you could add the following WHERE clause:
WHERE a.xuser = 'xy'
The first thing comes to my mind:
select sum(if(xuser='xxx', 1, -1)) = 1
from (
select distinct(xuser) from authors
union
select distinct(xuser) from titles
union
select distinct(xuser) from places
) all_xusers;
This will return 1 (true) if all tables contains records belonging ONLY to 'xxx' user. Otherwise (if there is no 'xxx' records or there is some other user records) it will return 0 (false).
I have a lot of tables and few tables have around 20 million records.
Now I do some calculations on these tables before showing the result to the UI.
For that I have created stored procedure.
In the stored procedure I am using temprorary tables to store select query records, do join, do processing and then return result.
I have written query like this
Insert INTO A
SELECT * from B JOIN c ....
Now my select query here does not take time (I have optimize it using explain extended) but my Insert into takes a lot of time since the output of select is in million. My table A is temprorary.
And Also I have do some processing on this A table after that.
My query is is there a way where I can skip inserting this insert.
I can take the result in a seperate table variable(if any) and then do processing instead of inserting all that in temprorary and then again doing some processing on it.
Adding my stored procedure here
DROP PROCEDURE IF EXISTS 6_4_1n2_PortUtil_temp;
DELIMITER |
CREATE PROCEDURE 6_4_1n2_PortUtil_temp(utilType VARCHAR(100),service varchar(5000),p_networkType VARCHAR(20),inputCity varchar(5000),inputNodeName varchar(5000), startTime timestamp,endTime timestamp)
BEGIN
DROP TEMPORARY TABLE IF EXISTS TEMP_SERVICE_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEIF_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEIF_TBL_1;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEName_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEANDIF_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_ROUTERTRAFFIC_VLANPRT_SCALE1_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_TRAFFIC_TBL;
DROP TEMPORARY TABLE IF EXISTS TRAFFIC_TBL;
DROP TEMPORARY TABLE IF EXISTS FINAL_FETCH;
SELECT now();
CREATE TEMPORARY TABLE TEMP_SERVICE_TBL(nodeName varchar(256),NodeNumber int) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_NODEName_TBL(nodeName varchar(256),NodeNumber int) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_NODEIF_TBL(NodeNumber int,IfIndex INTEGER,IfSpeed FLOAT,IfDescr VARCHAR(100),IfAlias VARCHAR(100)) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_NODEIF_TBL_1(NodeNumber int,IfIndex INTEGER,IfSpeed FLOAT,IfDescr VARCHAR(100),IfAlias VARCHAR(100)) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_NODEANDIF_TBL(NodeName Varchar(256),NodeNumber int,IfIndex INTEGER,IfSpeed FLOAT,IfDescr VARCHAR(100),IfAlias VARCHAR(100),PortID BIGINT(20) DEFAULT 0) ENGINE INNODB;
CREATE TEMPORARY TABLE TRAFFIC_TBL(
PortID BIGINT(20),
NodeName VARCHAR(100),
IfDescr VARCHAR(100),
IfSpeed VARCHAR(100),
InErrPkts BIGINT(20),
RcvOctets BIGINT(20),
TxOctets BIGINT(20),
Time_1 TIMESTAMP) ENGINE INNODB;
CREATE TEMPORARY TABLE TEMP_TRAFFIC_TBL(
PortID BIGINT(20),
maxTrafficValueIn BIGINT(20) DEFAULT 0,
maxOutTrafficValueOut BIGINT(20) DEFAULT 0,
avgTrafficValueIn BIGINT(20) DEFAULT 0,
avgTrafficValueOut BIGINT(20) DEFAULT 0,
CRCError BIGINT(20) DEFAULT 0,
UpTime INTEGER DEFAULT 0,
Reliablity INTEGER DEFAULT 0,
AvgUtilIn float DEFAULT 0,
AvgUtilOut float DEFAULT 0,
PeakUtilIn float DEFAULT 0,
PeakUtilOut float DEFAULT 0,
ThresholdExceed INTEGER DEFAULT 0,
inPeakTime timestamp DEFAULT '0000-00-00 00:00:00',
outPeakTime timestamp DEFAULT '0000-00-00 00:00:00') ENGINE INNODB;
SET #where = '';
IF service='ALL'
THEN
SET #where = '';
ELSE
set #a=1;
set #like="";
select REPLACE(SUBSTRING(SUBSTRING_INDEX(service, ',', #a),LENGTH(SUBSTRING_INDEX(service, ',', #a -1)) + 1),',','') into #service;
while(#service != "")
DO
IF(#like = "")
THEN
SET #like = CONCAT("NodeName like '%",SUBSTRING(#service,2,3),"%'");
ELSE
SET #like = CONCAT(#like," or NodeName like '%",SUBSTRING(#service,2,3),"%'");
END IF;
set #a=#a+1;
select REPLACE(SUBSTRING(SUBSTRING_INDEX(service, ',', #a),LENGTH(SUBSTRING_INDEX(service, ',', #a -1)) + 1),',','') into #service;
END WHILE;
SET #where = CONCAT(" where",#like);
END IF;
Set #where2 = '';
IF inputCity='ALL'
THEN
set #where2 = '';
ELSE
set #where2 = CONCAT(' and substring(NodeName,1,3) in (',inputCity,')');
END IF;
SET #where3 = '';
IF inputNodeName='ALL'
THEN
set #where3 = '';
ELSE
set #where3 = CONCAT(' and NodeName in (',inputNodeName,')');
END IF;
SET #query1 := CONCAT("INSERT INTO TEMP_NODEName_TBL SELECT distinct NodeName,NodeNumber from NODE_TBL",#where, #where2, #where3);
SELECT #query1;
PREPARE statement1 from #query1;
EXECUTE statement1;
DEALLOCATE Prepare statement1;
CREATE INDEX n1 ON TEMP_NODEName_TBL(NodeNumber);
CREATE INDEX i1 ON TEMP_NODEIF_TBL(NodeNumber,IfIndex);
CREATE INDEX portIDIndex1 on TEMP_NODEANDIF_TBL(PortID);
SET #where4 = '';
IF (utilType='ALL')
THEN
SET #where = '';
ELSE
If (utilType = "'AESI-IN'")
THEN
SET #where4 = " where IfAlias like '%AESI-IN%'";
ELSE
SET #where4 = " where IfAlias NOT like '%AESI-IN%'";
END IF;
END IF;
CREATE INDEX i2 ON TEMP_NODEIF_TBL_1(NodeNumber,IfIndex);
CREATE INDEX i3 ON TEMP_NODEANDIF_TBL(NodeNumber,IfIndex);
SET #where5 = '';
IF(p_networkType != 'ALL')
THEN
set #r1= SUBSTRING(p_networkType,2,3);
set #r2= SUBSTRING(p_networkType,8,3);
set #r3= SUBSTRING(p_networkType,14,3);
SET #where5 = CONCAT(" and IfAlias like '%",#r1,"%'");
if(#r2 != "")
THEN
SET #where5 = CONCAT(" and IfAlias like '%",#r2,"%'");
IF(#r3 != "")
THEN
SET #where5 = CONCAT(" and IfAlias like '%",#r3,"%'");
END IF;
END IF;
END IF;
SET #query2 := CONCAT("INSERT INTO TEMP_NODEANDIF_TBL(NodeName,NodeNumber,IfIndex,IfSpeed,IfDescr,IfAlias) SELECT distinct b.NodeName, a.NodeNumber, a.IfIndex,a.IfSpeed,a.IfDescr,a.IfAlias from NODEIF_TBL a JOIN TEMP_NODEName_TBL b ON a.NodeNumber = b.NodeNumber ", #where4, #where5);
SELECT #query2;
PREPARE statement1 from #query2;
EXECUTE statement1;
DEALLOCATE Prepare statement1;
SELECT "DROP TEMPORARY TABLES";
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEIF_TBL;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEIF_TBL_1;
DROP TEMPORARY TABLE IF EXISTS TEMP_NODEName_TBL;
update TEMP_NODEANDIF_TBL a,VLANPRT_TBL b set a.PortID = PrtID where a.NodeNumber = b.NodeID and a.IfIndex = b.IfIndex;
SELECT "Update Temporary tables";
delete from TEMP_NODEANDIF_TBL where PortID = 0;
SELECT now();
INSERT INTO TRAFFIC_TBL
SELECT a.PortID,NodeName,IfDescr,IfSpeed,InErrPkts,RcvOctets,TxOctets,Time_1 FROM ROUTERTRAFFIC_VLANPRT_SCALE1_TBL a JOIN TEMP_NODEANDIF_TBL b ON a.PortID = b.PortID where Time_1>startTime and Time_1<endTime;
SELECT now();
DROP TEMPORARY TABLE IF EXISTS EXCEED_COUNT;
CREATE TEMPORARY TABLE EXCEED_COUNT (PortID BIGINT(20), Exceed INTEGER);
INSERT INTO EXCEED_COUNT
select PortID,count(RcvOctets) from TRAFFIC_TBL where (RcvOctets/(IfSpeed*10)>70 or TxOctets/(IfSpeed*10)>70) group by PortID;
INSERT INTO TEMP_TRAFFIC_TBL (PortID, maxTrafficValueIn,maxOutTrafficValueOut,avgTrafficValueIn,avgTrafficValueOut,CRCError,AvgUtilIn,AvgUtilOut,PeakUtilIn,PeakUtilOut)
SELECT PortID,max(RcvOctets),max(TxOctets),avg(RcvOctets),avg(TxOctets), sum(InErrPkts),
IF((IfSpeed=0),"0",(avg(RcvOctets)/(IfSpeed*10))),
IF((IfSpeed=0),"0",(avg(TxOctets)/(IfSpeed*10))),
IF((IfSpeed=0),"0",(max(RcvOctets)/(IfSpeed*10))),
IF((IfSpeed=0),"0",(max(TxOctets)/(IfSpeed*10)))
from TRAFFIC_TBL group by PortID;
CREATE INDEX portIDIndex2 on TEMP_TRAFFIC_TBL(PortID);
CREATE INDEX portIDIndex3 on EXCEED_COUNT(PortID);
UPDATE TEMP_TRAFFIC_TBL A JOIN TRAFFIC_TBL B ON A.PortId=B.PortId SET inPeakTime=B.Time_1 where B.RcvOctets=A.maxTrafficValueIn;
UPDATE TEMP_TRAFFIC_TBL A JOIN TRAFFIC_TBL B ON A.PortId=B.PortId SET outPeakTime=B.Time_1 where B.TxOctets=A.maxOutTrafficValueOut;
UPDATE TEMP_TRAFFIC_TBL A JOIN EXCEED_COUNT B ON (A.PortID = B.PortID)
set ThresholdExceed = Exceed;
SELECT substring(NodeName,5,3) as ServiceType,
CASE
WHEN IfAlias like '%SWH%' THEN "Trunk"
WHEN IfAlias like '%AES%' THEN "Backbone"
WHEN IfAlias like '%RTR%' THEN "Back-to-Back"
ELSE "-"
END ,
NodeName,IfDescr,ROUND(maxTrafficValueIn/1000,2),ROUND(maxOutTrafficValueOut/1000,2),ROUND(avgTrafficValueIn/1000,2),ROUND(avgTrafficValueOut/1000,2),ROUND(CRCError/1000,2),0,0,ROUND(AvgUtilIn,2),ROUND(AvgUtilOut,2),ROUND(PeakUtilIn,2),ROUND(PeakUtilOut,2),ThresholdExceed,inPeakTime,outPeakTime from TEMP_TRAFFIC_TBL a ,TEMP_NODEANDIF_TBL b where a.PortID = b.PortID ;
SELECT now();
END |
DELIMITER ;
My ROUTERTRAFFIC_VLANPRT_SCALE1_TBL contains around 20 million records and NodeIF_TBL around 1 lakhs records. My VLANPRT_TBL also contains around 1 lakh records.
Also I have BTree indexing set on time_1 so that the join on the big table do not take time
Temporary tables:
Temporary tables start out in the memory and if the surpass a certain limitation they will be written to disk. If the select yields millions and you're writing those millions to disk then that's your problem.
Needing a temporary table with millions of records is usually a sign of doing something wrong. Perhaps you should further filter the data before inserting it into the temporary table for processing.
Disk I/O thrashing:
We have a few INSERT cases:
INSERT INTO ... VALUES
INSERT INGORE INTO ... SELECT
INSERT INTO ... SELECT
For 1. each insert has its own transaction and writes to disk individually meaning lots of tiny I/O operations.
For 2. the same applies except the data is also being read from the disk (slightly more I/O operations).
For 3. the whole operation is one big transaction meaning one big disk I/O at the very end (the commit) of the transaction. More often it happens that the information contained in the big transaction cannot be held entirely in memory so it is written to disk temporarily before the final commit and then copied from there into the designated area where it should reside.
In all 3 cases disk I/O thrashing occurs, to prevent this you would group small operations into average sized operations, and break huge operations into average sized operations.
For 1. you need to wrap every N inserts in a transaction.
For 2. and 3. you need to LIMIT the SELECT clause to N and repeat the INSERT INTO ... SELECT with an added OFFSET of N until you finish all records.
Either should be fairly easy to do if you're using a scripting language to run your queries.
Its not needed to store in a temporary table the output from a query in order to use it in another query. You can make this:
Select t1.* from (select * from A where condition1)as t1 where condition2
Here is an example of a query i had to do.
select avg(v1) as v1avg ,start_date_time as time, timekey as time_key from (select ATable.*, ROUND(UNIX_TIMESTAMP(start_date_time)/(60*60)) as timekey from ATable where start_date_time between '2014-01-01 00:00:00' and '2014-01-10 00:00:00')as t1 group by timekey;
Example: I have a table with 5 fields, named id, field_1, field_2, field_3, field_4
And I am searching for 'foo' across all fields.
SELECT ID FROM table WHERE field_1 LIKE ('%foo%') OR field_2 LIKE ('%foo%') OR ...
I'd like to return the IDs, as well as which fields the term was found in.
What would be the most efficient way to do this?
Note: I am looking for a solution that could dynamically accommodate adding new DB fields, without having to manually update the SQL.
One possible approach is to map these matches in returned columns:
SELECT ID,
field_1 LIKE '%foo%' AS field_1_match,
field_2 LIKE '%foo%' AS field_2_match
...
... so you can just check each corresponding column_match value to know, well, was it matched or not.
It's easy to extend this into returning a string with columns (separated by ,, for example) with CONCAT_WS:
SELECT ID,
CONCAT_WS(',',
IF(field_1 LIKE '%foo%', 'field_1', NULL),
IF(field_2 LIKE '%foo%', 'field_2', NULL)
...
)
... but, honestly speaking, I doubt it'll be easier to process data formatted this way.
Use unions:
SELECT id, 'field_1' as 'fieldName' FROM table WHERE field1 LIKE '%foo%' UNION
SELECT id, 'field_2' as 'fieldName' FROM table WHERE field2 LIKE '%foo%' UNION
...
This will return the id and column name wherever it is found. If it happens multiple times on the same row, multiple results will be returned for that row.
Updated:
It is possible to dynamically search all of the tables using a stored procedure and a cursor.
DELIMITER //
CREATE PROCEDURE search_all_fields(IN search VARCHAR(100), IN tableName VARCHAR(100), IN idColumnName VARCHAR(100))
BEGIN
DECLARE sqlQuery VARCHAR(200);
DECLARE done INT DEFAULT 0;
DECLARE columnName VARCHAR(30);
DECLARE cur CURSOR FOR SELECT `COLUMN_NAME` FROM `INFORMATION_SCHEMA`.`COLUMNS` WHERE `TABLE_NAME`= tableName; # cursor will iterate over the column names
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
CREATE TEMPORARY TABLE temp ( id VARCHAR(100), FieldName VARCHAR(100) ); # procedure returns multiple result sets, so we'll dump them in a temp table and get them at the end
OPEN cur;
read_loop: LOOP # iterates through column names
FETCH cur INTO columnName;
IF done THEN
LEAVE read_loop;
END IF;
# execute search
SET #sqlQuery = CONCAT("INSERT INTO temp SELECT ", idColumnName, ", '", columnName, "' as 'FieldName' FROM ", tableName, " WHERE ", columnName, " LIKE '%", search, "%'");
PREPARE stmt FROM #sqlQuery;
EXECUTE stmt;
END LOOP;
CLOSE cur;
# grab the results
SELECT * FROM temp;
END;//
DELIMITER ;