I am creating some reporting which summarises user actions (clicks) from an online job board.
The MYSQL tables are structured so that there is one table for each days activities. The table names are standard (except for the date) e.g. 'User_Clicks_DD_MM_YYY'
I want to select from all of the tables (including future tables that have not yet been created) without having to revisit the code each day.
Does anyone know of a way that I can do this?
here is one way to approach this, it is a stored procedure that builds a query
what this does is, loops through a range of dates (you can substitute now() for the last date) and checks if a table exists for a particular date. if it does exist then it adds it to a select statement string. then the string is executed at the end of the proc.
create table tab_01_01_2020 (id int);
create table tab_02_01_2020 (id int);
insert into tab_01_01_2020 values (1);
insert into tab_02_01_2020 values (1);
DELIMITER $$
DROP PROCEDURE IF EXISTS test$$
CREATE PROCEDURE test()
BEGIN
DECLARE count INT DEFAULT 0;
DECLARE mydate DATE;
set mydate = '2020/01/01';
set #qry='select 0 as id where 1 = 0';
WHILE mydate < '2020/02/01' DO
SET count = count + 1;
set #day = (SELECT RIGHT(CONCAT('0',DAY(mydate)), 2));
set #mon = (SELECT RIGHT(CONCAT('0',MONTH(mydate)), 2));
set #tab = CONCAT('tab_',#day,'_',#mon,'_',YEAR(mydate));
IF (SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = DATABASE() AND table_name = #tab) = 1
THEN
set #qry = CONCAT(#qry, ' union select id from ', #tab);
END IF;
SET mydate = DATE_ADD(mydate, INTERVAL 1 DAY);
END WHILE;
select #qry;
prepare stmt from #qry;
execute stmt;
deallocate prepare stmt;
END$$
DELIMITER ;
call test();
Related
I have two stored procedures in MySql. One of them does the calculation and returns everything in groups. With another stored procedure, I have a WHILE statement that runs for each month and calls the first stored procedure. I want to get the result of each group and save them in a table.
The code is something like:
CREATE PROCEDURE `first`(IN `start` date, **it should be list of items** )
begin
SET result = 0;
SELECT
item.name,
sum(amount) *INTO result*
FROM
FOO
INNER JOIN
BAR ON id = id
WHERE
date(someDate) < date(start)
group by something;
end
And the runner is something like:
CREATE PROCEDURE `runner`(IN `from` date, IN `to` date)
BEGIN
set dateTo = date(to);
set dateFrom = date(from);
WHILE DATE(dateFrom) <= DATE(dateTo) DO
call first(dateFrom, #res);
// here I need another loop through all results of the first procedure to insert each one of them in the following line.
insert into table_x (**some values which have been returned**);
SET dateFrom = DATE_ADD(dateFrom, INTERVAL 1 month);
END WHILE;
END
I don-' think that the loop and the second procedure ist really necessary
MySQL can't return table arrays or something of that kind, but you can use temporary tables
DELIMITER $$
CREATE PROCEDURE `first`(IN `start` date )
begin
DROP TEMPORARY TABLE IF EXISTS myTable;
CREATE TEMPORARY TABLE myTABLE
SELECT
item.name,
sum(amount)
FROM
FOO
INNER JOIN
BAR ON id = id
WHERE
date(someDate) < date(start)
group by something;
end$$
DELIMITER ;
Outer procdudre
DELIMITER $$
CREATE PROCEDURE `runner`(IN `_from` date, IN `_to` date)
BEGIN
set dateTo = date(_to);
set dateFrom = date(_from);
WHILE DATE(dateFrom) <= DATE(dateTo) DO
call first(dateFrom, #res);
insert into table_x (SELECT * FROM myTable);
SET dateFrom = DATE_ADD(dateFrom, INTERVAL 1 month);
END WHILE;
END$$
DELIMITER ;
You ca make dynamic sql work with different variables
DELIMITER $$
CREATE PROCEDURE `first`(IN `_start` date , IN _group varchar(100))
begin
DROP TEMPORARY TABLE IF EXISTS myTable;
SET #sql := CONCAT("
CREATE TEMPORARY TABLE myTABLE
SELECT
item.name,
sum(amount)
FROM
FOO
INNER JOIN
BAR ON id = id
WHERE
date(someDate) < date(",_gRoup,")
group by",_goup,";");
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
end$$
DELIMITER ;
This might work, if all ** some values ** are replaced.
Minimal version of MySQL needed is 8.0 (or an equivalent MariaDB version).
The two variables #from and #to are set to let this script know from which data to which date is needed.
This code is untested, because OP wrote "I have two stored procedures in MySql", but he uses things like "** some values **", which make the two stored procedures invalid (sigh).
Final remark: It is, of course, possible to wrap this complete statement into a stored procedure. (If above corrections are made)
set #from = '2021-05-01';
SET #to = '2021-06-13';
insert into table_x (** some values **)
WITH recursive dates AS (
SELECT #from as d
UNION ALL
SELECT date_add(d, INTERVAL 1 DAY)
FROM dates
WHERE d<#to
)
SELECT d, **some values which have been returned**
FROM dates
CROSS JOIN (
SELECT
item.name,
sum(amount)
FROM
FOO
INNER JOIN
BAR ON id = id
WHERE
date(someDate) < date(d)
group by something) sq
Let's say I query this:
SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE COLUMN_NAME like 'coduser';
it returns a list of tables which contains the column "coduser":
users
messages
passwords
photos
It's a 20+ items list.
I need to search on those tables all occurrences where "coduser" is equal to "5OEWP1BPSV".
SELECT * FROM tablenamehere WHERE coduser = "5OEWP1BPSV";
but I'm not using anything other than MySQL to do this.
Basically just search all tables where there is a column called "coduser" and coduser = "5OEWP1BPSV".
You will need to use dynamic sql and given the small numbers involved a cursor would be appropriate
drop table if exists t,t1;
create table t(id int auto_increment primary key , codeuser varchar(20));
create table t1(id int auto_increment primary key , codeuser varchar(20));
insert into t(codeuser) values
('aaa'),('5OEWP1BPSV');
insert into t(codeuser) values
('bbb');
drop procedure if exists p;
delimiter $$
create procedure p()
begin
declare tablename varchar(20);
declare finished int;
DECLARE GRD_CUR CURSOR FOR
SELECT TABLE_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE COLUMN_NAME like 'codeuser';
DECLARE CONTINUE HANDLER FOR NOT FOUND SET FINISHED = 0;
OPEN GRD_CUR;
LOOPROWS:LOOP
FETCH GRD_CUR INTO tablename;
IF FINISHED = 0 THEN
LEAVE LOOPROWS;
END IF;
#build and execute dynamic sql
set #sql = concat('SELECT * FROM ' , tablename, ' WHERE codeuser = "5OEWP1BPSV"');
#select #sql;
prepare sqlstmt from #sql;
execute sqlstmt;
deallocate prepare sqlstmt;
END LOOP;
close grd_cur;
end $$
delimiter ;
call p();
Loop over the returned list of tables, and for each table in that list perform a
SELECT *
WHERE columnnamehere = "valueofcolumnhere";
Bear in mind that this will give you separate lists for each table within that FOR loop, so you would have to concatenate these in some way to get a complete list.
Is it possible to loop through the all column names while inside a trigger?
Scenario:
To log all the columns of a table that have been modified.
If some values did not change, do not log those ones.
DROP TRIGGER IF EXISTS t_before_update_test;
DELIMITER $$
CREATE TRIGGER t_before_update_test
BEFORE UPDATE ON test
FOR EACH ROW
BEGIN
-- Loop here for all columns, not just col1
IF OLD.col1 <> NEW.col1 THEN
INSERT INTO change_logs(
log_on, user_id,
table_name, colum_name,
old_data, new_data
) VALUES (
UNIX_TIMESTAMP(NOW()), '0',
'test', 'col1',
OLD.col1, NEW.col1
);
END IF;
-- process looping all columns
-- col1, col2, ... should be dynamic per loop
END $$
This is working copy example, where I now need to loop through all columns available in OLD or NEW.
Unfortunately, using dynamic SQL (i.e PREPARED STATEMENT) in MySQL trigger is not allowed.(This can not be bypassed by calling a stored procedure which has dynamic SQL ). Therefore, we have to hardcode the column name in the trigger. However, if the columns are to change, the trigger will break due to the unmatchable columns, which simply stops the UPDATE trasaction. Therefore, we need to check if it's legit to do the logging job in the change_logs table. If legit, then insert into the change_logs table; else just send a warning message into a warning table. Supposing the test table has two columns namely id and datetm. And a warning table with 3 columns (table_name,log_time,log_content) is created beforehand. The change_logs table is identical to the OP's. The rest is creating the trigger (written and tested in workbench):
delimiter //
drop trigger if exists t_before_update_test//
create trigger t_before_update_test before update on test for each row begin
if
'id' not in (select column_name from information_schema.columns where table_name='test')
or 'datetm' not in (select column_name from information_schema.columns where table_name='test')
or (select count(column_name) from information_schema.columns where table_name='test') !=2
then
insert into warning_table values ('test',now(),'Table column structure has been changed!!');
else
IF old.id <> new.id THEN
INSERT INTO change_logs(
log_on, user_id,
`table_name`, colum_name,
old_data, new_data
) VALUES (
UNIX_TIMESTAMP(NOW()), '0',
'test', 'id',
old.id, new.id
);
END IF;
IF old.datetm <> new.datetm THEN
INSERT INTO change_logs(
log_on, user_id,
`table_name`, colum_name,
old_data, new_data
) VALUES (
UNIX_TIMESTAMP(NOW()), '0',
'test', 'datetm',
old.datetm, new.datetm
);
END IF;
end if;
end //
I don't have enough time to finish this right now, but I think that using CONCAT() to prepare a statement and using the result of that for a conditional might enable you to do what you want. Something along these lines:
DECLARE num_rows INT DEFAULT 0;
DECLARE cols CURSOR FOR SELECT column_name FROM information_schema.columns WHERE table_name = 'table_name' ORDER BY ordinal_position;
OPEN cols;
SELECT FOUND_ROWS() INTO num_rows;
SET #i = 1;
cols_loop: LOOP
IF #i > num_rows THEN
CLOSE cols;
LEAVE cols_loop;
END IF;
FETCH cols INTO col;
SET #do_stuff = 0;
SET #s = CONCAT('SELECT IF(NEW.', col, ' <> OLD.', col, ', 1, 0) INTO #do_stuff');
PREPARE stmt1 FROM #s;
EXECUTE stmt1;
DEALLOCATE PREPARE stmt1;
IF #do_stuff = 1 THEN
SET #s2 = CONCAT('INSERT INTO change_logs(log_on, user_id, table_name, colum_name, old_data, new_data )
VALUES (UNIX_TIMESTAMP(NOW()), ''0'', ''test'', ''', col,''', OLD.', col, ', NEW.', col, ');');
PREPARE stmt2 FROM #s2;
EXECUTE stmt2;
DEALLOCATE PREPARE stmt2;
END IF;
SET #i = #i + 1;
END LOOP cols_loop;
CLOSE cols;
Unfortunately you can't do that. You can get the column names by accessing INFORMATION_SCHEMA but it's not possible to access the OLD and NEW values from that column names. I think it make sense because unlike stored procedure you're creating a trigger for a specific table not for the database . Calling a Stored procedure inside the trigger will help you to reduce code up-to some extend.
DROP TRIGGER IF EXISTS t_before_update_test;
DELIMITER $$
CREATE TRIGGER t_before_update_test
BEFORE UPDATE ON test
FOR EACH ROW
BEGIN
IF OLD.col1 <> NEW.col1 THEN
/*pseudo*/
CALL SP_insert_log (
'test',
'colum_name',
'old_value',
''old_value');
ELSEIF OLD.col2 <> NEW.col2 THEN
//call above sp with this column related data
END IF;
END $$
yes, a cursor can be added within a trigger to loop through columns.
here are a couple of links :
mysql, iterate through column names
https://dba.stackexchange.com/questions/22925/mysql-loop-over-cursor-results-ends-ahead-of-schedule
from experience, it might be easier to create a stored procedure that does the looping and inserts and call it from the trigger
I have a table autos that has a column name, I want to check first 5 rows in the table and if name value is "toyota", in table mytable write "yes", else write "no".
I write stored procedure, but mysqli_error() returns error in line, where I have EXECUTE ....
If in WHEN I write not PREPARED STATEMENT, but directly the query, the procedure works.
Please see my code and tell me, where is it wrong?
CREATE PROCEDURE proc_auto()
BEGIN
DECLARE start INT;
SET start = 0;
PREPARE stmt FROM ' SELECT name FROM autos ORDER BY id LIMIT ?,1 ';
WHILE start < 5 DO
CASE
WHEN (EXECUTE stmt USING #start ) = 'toyota'
THEN INSERT INTO mytable (log) VALUES('yes');
ELSE
INSERT INTO mytable (log) VALUES('no');
END CASE;
SET start = start + 1;
END WHILE;
END;
(The suggestion about EXECUTE is removed as incorrect and potentially confusing.)
The problem you are trying to solve with a stored procedure could in fact be solved without it, using an entirely different approach: just use a single INSERT ... SELECT statement instead:
INSERT INTO mytable (log)
SELECT
CASE name
WHEN 'toyota' THEN 'yes'
ELSE 'no'
END
FROM autos
ORDER BY id
LIMIT 5
That is, the above statement does the same as your stored procedure: it retrieves first 5 rows from autos and inserts 5 rows into mytable. Depending on the value of name it generates either yeses or nos.
Andriy M's INSERT statement is the most elegant solution, but if you still want to use a procedure, this will work:
CREATE PROCEDURE proc_auto()
BEGIN
DECLARE start INT DEFAULT 0;
PREPARE stmt FROM
'SELECT name INTO #name FROM autos ORDER BY id LIMIT ?,1';
WHILE start < 5 DO
SET #start = start;
EXECUTE stmt USING #start;
IF #name = 'toyota' THEN
INSERT INTO mytable (log) VALUES('yes');
ELSE
INSERT INTO mytable (log) VALUES('no');
END IF;
SET start = start + 1;
END WHILE;
END;
but, in this case, using a CURSOR would yield better performance:
CREATE PROCEDURE proc_auto()
BEGIN
DECLARE start INT DEFAULT 0;
DECLARE b_not_found BOOL DEFAULT FALSE;
DECLARE cur CURSOR FOR
'SELECT name FROM autos ORDER BY id LIMIT 5';
DECLARE CONTINUE HANDLER FOR NOT FOUND SET b_not_found = TRUE;
OPEN cur;
loop1: WHILE start < 5 DO
FETCH cur INTO #name;
IF b_not_found THEN
LEAVE loop1;
END IF;
IF #name = 'toyota' THEN
INSERT INTO mytable (log) VALUES('yes');
ELSE
INSERT INTO mytable (log) VALUES('no');
END IF;
SET start = start + 1;
END WHILE;
CLOSE cur;
END;
Hallo and good sunday to everybody.
I need to select N random records from each group.
Starting from the query of Quassnoi
http://explainextended.com/2009/03/01/selecting-random-rows/
to select X random record I wrote this store procedure
delimiter //
drop procedure if exists casualiPerGruppo //
create procedure casualiPerGruppo(in tabella varchar(50),in campo varchar(50),in numPerGruppo int)
comment 'Selezione di N record casuali per gruppo'
begin
declare elenco_campi varchar(255);
declare valore int;
declare finite int default 0;
declare query1 varchar(250);
declare query2 varchar(250);
declare query3 varchar(250);
declare query4 varchar(250);
declare cur_gruppi cursor for select gruppo from tmp_view;
declare continue handler for not found set finite = 1;
drop table if exists tmp_casuali;
set #query1 = concat('create temporary table tmp_casuali like ', tabella);
prepare stmt from #query1;
execute stmt;
deallocate prepare stmt;
set #query2 = concat('create or replace view tmp_view as select ',campo,' as gruppo from ',tabella,' group by ',campo);
prepare stmt from #query2;
execute stmt;
deallocate prepare stmt;
open cur_gruppi;
mio_loop:loop
fetch cur_gruppi into valore;
if finite = 1 then
leave mio_loop;
end if;
set #query3 = concat("select group_concat(column_name) into #elenco_campi
from information_schema.columns
where table_name = '",tabella,"' and table_schema = database()");
prepare stmt from #query3;
execute stmt;
deallocate prepare stmt;
set #query4 = concat('insert into tmp_casuali select ',
#elenco_campi,' from (
select #cnt := count(*) + 1,
#lim :=', numPerGruppo,
' from ',tabella,
' where ',campo,' = ', valore,
' ) vars
straight_join
(
select r.*,
#lim := #lim - 1
from ', tabella, ' r
where (#cnt := #cnt - 1)
and rand() < #lim / #cnt and ', campo, ' = ', valore ,
') i');
prepare stmt from #query4;
execute stmt;
deallocate prepare stmt;
end loop;
close cur_gruppi;
select * from tmp_casuali;
end //
delimiter ;
that I recall in this way to give you an idea:
create table prova (
id int not null auto_increment primary key,
id_gruppo int,
altro varchar(10)
) engine = myisam;
insert into prova (id_gruppo,altro) values
(1,'aaa'),(2,'bbb'),(3,'ccc'),(1,'ddd'),(1,'eee'),(2,'fff'),
(2,'ggg'),(2,'hhh'),(3,'iii'),(3,'jjj'),(3,'kkk'),(1,'lll'),(4,'mmm');
call casualiPerGruppo('prova','id_gruppo',2);
My problem is that Quassnoi query, even though is very performant, it takes even 1 second on a large recorset. So if I apply it within my sp several times, the total time increases a lot.
Can you suggest me a better way to solve my problem?
Thanks in advance
EDIT.
create table `prova` (
`id` int(11) not null auto_increment,
`id_gruppo` int(11) default null,
`prog` int(11) default null,
primary key (`id`)
) engine=myisam charset=latin1;
delimiter //
drop procedure if exists inserisci //
create procedure inserisci(in quanti int)
begin
declare i int default 0;
while i < quanti do
insert into prova (id_gruppo,prog) values (
(floor(1 + (rand() * 100))),
(floor(1 + (rand() * 30)))
);
set i = i + 1;
end while;
end //
delimiter ;
call inserisci(1000000);
#Clodoaldo:
My stored procedure
call casualipergruppo('prova','id_gruppo',2);
gives me 200 records and takes about 23 seconds. Your stored procedure keeps on giving me Error Code : 1473 Too high level of nesting for select even though I increase varchar value to 20000. I don't know if there is any limit on unions involved in a query.
I removed the tabella and campo parameters from the procedure just to make it easier to understand. I'm sure you can bring them back.
delimiter //
drop procedure if exists casualiPerGruppo //
create procedure casualiPerGruppo(in numPerGruppo int)
begin
declare valore int;
declare finite int default 0;
declare query_part varchar(200);
declare query_union varchar(2000);
declare cur_gruppi cursor for select distinct id_gruppo from prova;
declare continue handler for not found set finite = 1;
create temporary table resultset (id int, id_gruppo int, altro varchar(10));
set #query_part = 'select id, id_gruppo, altro from (select id, id_gruppo, altro from prova where id_gruppo = #id_gruppo order by rand() limit #numPerGruppo) ss#id_gruppo';
set #query_part = replace(#query_part, '#numPerGruppo', numPerGruppo);
set #query_union = '';
open cur_gruppi;
mio_loop:loop
fetch cur_gruppi into valore;
if finite = 1 then
leave mio_loop;
end if;
set #query_union = concat(#query_union, concat(' union ', #query_part));
set #query_union = replace(#query_union, '#id_gruppo', valore);
end loop;
close cur_gruppi;
set #query_union = substr(#query_union, 8);
set #query_union = concat('insert into resultset ', #query_union);
prepare stmt from #query_union;
execute stmt;
deallocate prepare stmt;
select * from resultset order by id_gruppo, altro;
drop table resultset;
end //
delimiter ;
Wow. That's a complicated way to do something very simple. Try this:
Assuming you have sequential ids (otherwise you could get no rows).
create view random_prova as
select * from prova
where id = (select min(id) from prova) +
floor(RAND(0) * (select max(id) - min(id) from prova));
This will give you 1 random row.
To get multiple rows, either loop in a stored procedure or server program until you get enough rows, or programatically create a query that employs union.
eg, this will give you 3 random rows:
select * from random_prova
union
select * from random_prova
union
select * from random_prova;
Note that using RAND(0) instead of RAND() means getting a different random number for each invocation. RAND() will give the same value for each invocation in the one statement (so using RAND() with union won't give you multiple rows).
There are some shortcomings with using union - it is possible to get the same row twice by chance. Programatically calling this until you get enough rows is safer.
To give better performance, use something like java to randomly select the ids for a simple query like
select * from prova where id in (...)
and have java (or perl or whatever) fill in the list with random ids - you would avoid the inefficiency of having to get the id range every time.
Post if your ids are not sequential - there is an efficient way, but I its explanation is long.