I have my data in a key-value table (in MySql) which has the shape:
id, key, value
Now for export to my users I need to transform it into a table with all the keys as columns. (actually only ~20 of the ~100 keys need to be in that table)
The workflow would be to provide it my users so that they can correct the table and to reimport the tables.
I'm just in writing a quite complex select command to give me such a table. It already is some heck of long command and I hope it will not need to be debugged.
I can't help but think this should be an already solved problem ;)
So I'm hoping anyone can provide me with some clues.
So far I hacked together a script that will construct a view.
DROP PROCEDURE IF EXISTS view_test;
DELIMITER //
CREATE PROCEDURE view_test()
BEGIN
DECLARE done INT DEFAULT 0;
DECLARE curr_prop VARCHAR(31);
DECLARE curr_table VARCHAR(31);
DECLARE fixed_prop VARCHAR(31);
DECLARE statement_a LONGTEXT DEFAULT 'SELECT pd.id,t0.date';
DECLARE statement_b LONGTEXT DEFAULT "FROM xxx.codes AS pd INNER JOIN
xxx.eav AS t0 ON (pd.id = t0.idX)";
DECLARE n_prop INT DEFAULT 0;
DECLARE cur1 CURSOR FOR SELECT DISTINCT prop FROM xxx.eav LIMIT 59;
DECLARE CONTINUE HANDLER FOR SQLSTATE '02000' SET done = 1;
OPEN cur1;
main_loop: LOOP
FETCH cur1 INTO curr_prop;
IF done
THEN LEAVE main_loop;
END IF;
SET fixed_prop = curr_prop;
SET fixed_prop = REPLACE(fixed_prop,'-','_');
SET fixed_prop = REPLACE(fixed_prop,'+','_');
SET fixed_prop = REPLACE(fixed_prop,' ','_');
SET n_prop = n_prop + 1;
SET curr_table = CONCAT('t',n_prop);
SET statement_a = CONCAT(statement_a,',\n',curr_table,'.value AS ', fixed_prop);
SET statement_b = CONCAT(statement_b,' \nLEFT JOIN xxx.eav AS ',curr_table,
' ON (',curr_table,'.idX=pd.id AND t0.date=',curr_table,'.date AND
',curr_table,'.prop="',curr_prop,'")');
END LOOP;
-- cleanup
CLOSE cur1;
SET #S = CONCAT('CREATE OR REPLACE VIEW auto_flat_table AS\n',statement_a,
'\n',statement_b,
'\nGROUP BY pd.id,t0.date'); -- ,'\nGROUP BY pd.id'
PREPARE stmt_auto_demo FROM #S;
EXECUTE stmt_auto_demo;
DEALLOCATE PREPARE stmt_auto_demo;
END //
DELIMITER ;
CALL view_test();
That will create such a view for myself - though I'm a bit confused why this GROUP BY is neccesary.
Not being a SQL developer I'm pretty sure I took the long way in this program.
Also that I'm only able to use 61 tables for joins is a bummer. I think we will soon need more.
Related
I have within a database several tables where they all have username column. I would like to update one username and naturally I should update it in all tables.
I have this working solution:
UPDATE `user`,
`user_images`,
`user_comments`
SET `user`.`username` = 'new_name',
`user_images`.`username` = 'new_name',
`user_comments`.`username` = 'new_name'
WHERE `user`.`username` = 'old_name'
AND `user_images`.`username` = 'old_name'
AND `user_comments`.`username` = 'old_name'
I am hoping for a better query that can do the same action, as if table numbers got increased, do I really need to do this in 100 lines?
It sounds painful if you have to update each table. I would suggest using a stored procedure to finish the tedious job. Fisrt of all , make a table(named tablelist) which list all the tablename you would like to update. Then call the procedure by providing the two parameters where the o_name is the name you would like to change and the n_name is the new name to be changed into.
delimiter //
drop procedure if exists update_name//
create procedure update_name (o_name varchar(30),n_name varchar(30))
begin
declare t_name varchar(30);
declare done bool default false;
declare csr cursor for select tablename from tablelist;
declare continue handler for not found set done=true;
open csr;
lp: loop
fetch csr into t_name;
if done=true then
leave lp;
end if;
set #prep=concat('update ',t_name,' set `username`= "',n_name,'" where `username`= "',o_name,'";');
prepare prep_stat from #prep;
execute prep_stat;
deallocate prepare prep_stat;
end loop lp;
close csr;
end//
delimiter ;
The following call will change the name from john(case insensitive) to Xero in all tables listed in the tablelist table.
call update_name('John','Xero');
I intend to write a procedure to run a query on each of the tables in a provided list (can be a comma separated list or a table - undecided on that yet)
I started off with creating a while loop to iterate through each element in the provided list. Have been able to extract each element but I don't know how to run a query for that extracted element/table.
DELIMITER $$
DROP PROCEDURE IF EXISTS retain_demo_clients$$
CREATE PROCEDURE retain_demo_clients()
BEGIN
DECLARE counter INT(10);
DECLARE client_tables VARCHAR(255);
DECLARE table_count INT(10);
DECLARE table_in_process VARCHAR(255);
SET counter = 1;
SET table_count = 3;
SET client_tables = 'client_table, somerandomstuff, somemorestuff';
WHILE (counter < table_count +1) DO
SET table_in_process = substring_index(substring_index(client_tables, ',',counter),',',-1);
SELECT table_in_process;
SET counter = counter +1;
END WHILE;
END$$
DELIMITER ;
CALL retain_demo_clients();
I expect to do something like 'select * from table_in_process'. Would also appreciate if there is a better way to loop through the list of tables.
Here is DBFiddle link, if someone wants to tinker: https://www.db-fiddle.com/f/v6EMsiWvXFrBoNLgoZwDVX/1
You can use EXECUTE to run a text that represent a single statement
SET #someQuery = CONCAT('SELECT * FROM ', table_in_process ) ;
PREPARE preparable_stmt FROM #someQuery;
EXECUTE preparable_stmt;
Currently, our application's database stores all DATETIME values in GMT-8, which is generally very bad. We're trying to convert the application and all data to use UTC.
I am trying to write a one-off stored procedure that will query my database's schema, find all columns of type DATETIME or TIMESTAMP, and automatically UPDATE the columns by converting values with the CONVERT_TZ() function.
Using all of the information I've found so far about cursors, dynamic statement preparation, and error handling, I've come up with this:
DELIMITER $$
CREATE PROCEDURE `sp_convertDBtoUTC`()
BEGIN
DECLARE `_rollback` BOOL DEFAULT 0;
DECLARE done INT DEFAULT 0;
DECLARE tname VARCHAR(64);
DECLARE cname VARCHAR(64);
DECLARE dt_columns CURSOR FOR
SELECT table_name, column_name
FROM information_schema.columns
WHERE table_schema = 'my_schema'
AND data_type IN ('datetime', 'timestamp')
ORDER BY table_name, column_name;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
DECLARE CONTINUE HANDLER FOR SQLEXCEPTION SET `_rollback` = 1;
START TRANSACTION;
OPEN dt_columns;
table_loop:
LOOP
FETCH dt_columns INTO tname, cname;
IF done = 1 THEN
LEAVE table_loop;
END IF;
SET #stmt = CONCAT('UPDATE ',tname,' SET ',cname, ' = CONVERT_TZ(',cname, ', \'America/Los_Angeles\', \'UTC\')');
PREPARE stmt FROM #stmt;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END LOOP;
CLOSE dt_columns;
IF `_rollback` THEN
ROLLBACK;
ELSE
COMMIT;
END IF;
END$$
This seems to run without errors, but does nothing. If I add a SELECT #stmt statement in there, I get multiple result sets with the correct SQL, e.g. UPDATE tz_test SET dts = CONVERT_TZ(dts, 'America/Los_Angeles', 'UTC'). If I run this statement manually, the table gets updated as expected.
However, when running the stored procedure to update the entire database, the data in the tables doesn't actually get updated, and I don't understand why.
I'm using the following as a test:
CREATE TABLE tz_test (id INT auto_increment, dts DATETIME, PRIMARY KEY (id));
INSERT INTO tz_test (dts) VALUES (NOW());
SELECT * FROM tz_test;
CALL sp_convertDBtoUTC();
SELECT * FROM tz_test;
I would have expected the second result set to contain the modified datetime value, but the two result sets are identical. Why is this?
When trying to run these queries through MySQL Workbench, they were actually failing silently because of Safe Mode (no mass update/delete without a reference to a primary key).
Once I turned it off and reconnected, the entire process worked flawlessly.
I would like to use LIMIT in a cursor. The cursor should be used and updated several times within a loop, each time with different parameters of LIMIT. Here some code:
DELIMITER $$
CREATE PROCEDURE `updateIt`() READS SQL DATA
BEGIN
declare done int(1) default 0;
declare counter int(10) default 0;
declare xabc int(10) default 0;
declare tab1Cursor cursor for select abc from tab1 limit 100000*counter, 100000;
declare continue handler for not found set done=1;
loopCounter: LOOP
set done = 0;
open tab1Cursor;
igmLoop: loop
fetch tab1Cursor into xabc;
if done = 1 then leave igmLoop; end if;
-- do something
end loop igmLoop;
close tab1Cursor;
if (counter = 1039)
leave loopCounter;
end if;
set counter = counter + 1;
END LOOP loopCounter;
END $$
DELIMITER ;
This, however, does not work (I also tried it with the cursor in the LOOP counterLoop). Can Mysql deal with dynamic cursors?
From the MySQL Manual
a cursor cannot be used for a dynamic statement that is prepared and
executed with PREPARE and EXECUTE. The statement for a cursor is
checked at cursor creation time, so the statement cannot be dynamic.
However there are 2 ways, according to this post in mysql forums:
The first is for cases where absolutely only one user at a time will be running the procedure. A prepare statement can be used to create a view with the dynamic SQL and the cursor can select from this statically-named view. There's almost no performance impact. Unfortunately, these views are also visible to other users (there's no such thing as a temporary view), so this won't work for multiple users.
Analogously, a temporary table can be created in the prepare statement and the cursor can select from the temporary table. Only the current session can see a temporary table, so the multiple user issue is resolved. But this solution can have significant performance impact since a temp table has to be created each time the proc runs.
Bottom line: We still need cursors to be able to be created dynamically!
Here's an example of using a view to pass the table name and column name into a cursor.
DELIMITER //
DROP PROCEDURE IF EXISTS test_prepare//
CREATE PROCEDURE test_prepare(IN tablename varchar(255), columnname varchar(50))
BEGIN
DECLARE cursor_end CONDITION FOR SQLSTATE '02000';
DECLARE v_column_val VARCHAR(50);
DECLARE done INT DEFAULT 0;
DECLARE cur_table CURSOR FOR SELECT * FROM test_prepare_vw;
DECLARE CONTINUE HANDLER FOR cursor_end SET done = 1;
SET #query = CONCAT('CREATE VIEW test_prepare_vw as select ', columnname, ' from ', tablename);
select #query;
PREPARE stmt from #query;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
OPEN cur_table;
FETCH cur_table INTO v_column_val;
WHILE done = 0 DO
SELECT v_column_val;
FETCH cur_table INTO v_column_val;
END WHILE;
CLOSE cur_table;
DROP VIEW test_prepare_vw;
END;
//
DELIMITER ;
I'm working on an old database already in use for years and really crappy designed.
There is a table, "Articles", which contains a "code" column that will be our PK.
And many tables like "idXXXXX" where XXXXX is a "code" value with exactly the same structure.
I looked at the application using this database and saw that relations between tables is made there.
I'm not affraid of redesign the database access in the application, but I don't want to lose years of entries in the database.
I want to create a "campain" table which will have an "id" PK and a "id_code" as FK linking "campain" to "articles"
I'm not a SQL master but I know I can get tables names with
SELECT TABLE_NAME FROM INFORMATION_SCHEMA WHERE TABLE_NAME LIKE 'id%'
But I have really no idea about how to deal with the result (which is fine).
So how can I access to every tables named "idXXX" and insert every rows in the "campain" table + set "id_code" column to "XXX"?
Here is the procedure I saved (I didn't add every fields in the INSERT line for testing purpose) :
CREATE PROCEDURE JoinAllTables()
BEGIN
DECLARE done INT default 0;
DECLARE tableName CHAR(9);
DECLARE buffStr CHAR(7);
DECLARE buffId INT default 0;
DECLARE cur1 CURSOR FOR SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE 'id%';
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
OPEN cur1;
read_loop: LOOP
FETCH cur1 INTO tableName;
IF done THEN
LEAVE read_loop;
END IF;
SET buffStr = SUBSTRING(tableName, 3);
SET buffId = CAST(buffStr AS SIGNED);
set #sql = CONCAT("INSERT INTO campagnes(id, id_code) SELECT null, bufId FROM ",tableName); # Dynamically building sql statement
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END LOOP;
CLOSE cur1;
END;
As u can see, I sub 'idXXXXX' to 'XXXXX' then CAST it AS INTEGER (SIGNED).
But I guess that in the "INSERT INTO" line, second tableName doesn't point to the variable. That's why I'm getting a
"#1446 - Tabble 'bddsoufflage.tablename'doesn't exist" Error :) Any idea ?
Edit: Updated answer
We can't have the tableName dynamically changed inside a prepared statement, so we must go through DynamicSQL to build the query using CONCAT, then compile the SQL with PREPARE, EXECUTE it and DEALLOCATE it.
DELIMITER //
CREATE PROCEDURE JoinAllTables()
BEGIN
DECLARE done INT default 0;
DECLARE tableName CHAR(9);
DECLARE buffStr CHAR(7);
DECLARE buffId INT default 0;
DECLARE cur1 CURSOR FOR SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE 'id%';
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
OPEN cur1;
read_loop: LOOP
FETCH cur1 INTO tableName;
IF done THEN
LEAVE read_loop;
END IF;
SET buffStr = SUBSTRING(tableName, 3);
SET buffId = CAST(buffStr AS SIGNED);
set #sql = CONCAT("INSERT INTO campagnes(id, id_code) SELECT null, ", buffId, " FROM ",tableName); # Dynamically building sql statement
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END LOOP;
CLOSE cur1;
END; //
See also this answer MySQL Pass table name to cursor select
Old answer
The procedure should look something like this. Thanks Mchl for providing an Insert Into query example, I simply added it to the rest of the procedure.
DELIMITER //
CREATE PROCEDURE JoinAllTables()
BEGIN
DECLARE done INT default 0;
DECLARE tableName CHAR(7); # Variable to contain table names CHAr(7) is assuming id + 5Xs as characters.
DECLARE cur1 CURSOR FOR SELECT TABLE_NAME FROM INFORMATION_SCHEMA WHERE TABLE_NAME LIKE 'id%'; # Create a cursor to iterate over the tables
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
OPEN cur1;
read_loop: LOOP
FETCH cur1 INTO tableName;
IF done THEN
LEAVE read_loop;
END IF;
#Your Insert statement here, using tableName as a field.
INSERT INTO campain (id, id_code, otherfields) SELECT null, tableName, otherfields FROM tableName;
END LOOP;
CLOSE cur1;
END;//
Easiest way would be to run the information_schema query you have within some script (PHP,Python,Perl - whichever suits you best) and use it's results to create queries like:
INSERT INTO
campain (id, id_code, otherfields)
SELECT
null, 'idXXXX', otherfields FROM idXXXX