I'm performing some database clean up and have noticed that there are a lot of columns that have both empty strings and NULL values in various columns.
Is it possible to write an SQL statement to update the empty strings to NULL for each column of each table in my database, except for the ones that do not allow NULL's?
I've looked at the information_schema.COLUMNS table and think that this might be the place to start.
It's not possible to do this with one simple SQL statement.
But you can do it using one statement for each column.
UPDATE TABLE SET COLUMN = NULL
WHERE LENGTH(COLUMN) = 0
or, if you want to null out the items that also have whitespace:
UPDATE TABLE SET COLUMN = NULL
WHERE LENGTH(TRIM(COLUMN)) = 0
I don't think it's possible within MySQL but certainly with a script language of your choice.
Start by getting all tables SHOW TABLES
Then for each table get the different columns and find out witch ones allow null, either with DESC TABLE, SHOW CREATE TABLE or SELECT * FROM information_schema.COLUMNS, take the one you rather parse
Then for each column that allows null run a normal update that changes "" to null.
Prepare to spend some time waiting :)
I figured out how to do this using a stored procedure. I'd definitely look at using a scripting language next time.
DROP PROCEDURE IF EXISTS settonull;
DELIMITER //
CREATE PROCEDURE settonull()
BEGIN
DECLARE done INT DEFAULT FALSE;
DECLARE _tablename VARCHAR(255);
DECLARE _columnname VARCHAR(255);
DECLARE cur1 CURSOR FOR SELECT
CONCAT(TABLE_SCHEMA, '.', TABLE_NAME) AS table_name,
COLUMN_NAME AS column_name
FROM information_schema.COLUMNS
WHERE IS_NULLABLE = 'YES'
AND TABLE_SCHEMA IN ('table1', 'table2', 'table3');
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
OPEN cur1;
read_loop: LOOP
FETCH cur1 INTO _tablename, _columnname;
IF done THEN
LEAVE read_loop;
END IF;
SET #s = CONCAT('UPDATE ', _tablename, ' SET ', _columnname, ' = NULL WHERE LENGTH(TRIM(', _columnname, ')) = 0' );
PREPARE stmt FROM #s;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END LOOP;
CLOSE cur1;
END//
DELIMITER ;
CALL settonull();
Related
I want a procedure that will search all columns for non keyboard ascii characters (Dec 16 to Dec 31 or DLE to US) and update the column by replacing them with a space ' ' or nothing ''.
I have a SELECT statement that is finding the rows I need to update, but I have to manually change all columns myself.
SELECT column_name
FROM table_name
WHERE column_name REGEXP '[[.DLE.]-[.US.]]'
Here's the UPDATE script for modifying the column values
UPDATE table
SET
column = replace(column,char(16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31), '')
I want those two to be fused together to a single function or stored procedure but I don't know how, since I'm just starting to learn MySQL.
Disclaimer
Between using REGEXP and CURSORs to loop through each table and column, these examples are not going to be lightning fast. The speed will obviously vary depending on your environment and I suggest testing them out on development before production
One column in one table
To search a single column on a single table, you basically had the UPDATE as you needed it.
UPDATE t1
SET
column_name = replace(column_name,
char(16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31), '')
WHERE column_name REGEXP '[[.DLE.]-[.US.]]'
All columns in one table
To do all columns in a table, you need to identify the table, then loop through the columns using a cursor
DELIMITER $$
CREATE PROCEDURE table_regexp_replace(in_table VARCHAR(128))
BEGIN
DECLARE done INT DEFAULT FALSE;
DECLARE search_column VARCHAR(64);
DECLARE cur1 CURSOR FOR
SELECT DISTINCT `COLUMN_NAME` FROM `information_schema`.`COLUMNS`
WHERE `TABLE_NAME` = in_table ORDER BY `ORDINAL_POSITION` ;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
OPEN cur1;
read_loop: LOOP
-- Process the next column
FETCH cur1 INTO search_column;
-- If we're done, stop the loop
IF done THEN
LEAVE read_loop;
END IF;
-- Replace everything in this column matching the regexp
SET #new_query := CONCAT ('UPDATE ', in_table,
' SET `', search_column, '` = replace(', search_column,
', char(16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31), \'\')
WHERE ', search_column, ' REGEXP \'[[.DLE.]-[.US.]]\'') ;
PREPARE stmt FROM #new_query;
EXECUTE stmt ;
END LOOP;
CLOSE cur1;
END$$
DELIMITER ;
Then usage
CALL table_regexp_replace('my_table');
How it works
Looks convoluted, it's actually pretty straight forward.
We create a procedure with one parameter in_table which is used to specify the table to work with.
Setup a cursor that pulls the column names from the information_schema table, in their correct order
Loop through each of those columns, executing the manually created UPDATE statement against each one.
You'll notice anywhere in the UPDATE query that required quotes, they've had to be escaped using \.
\'[[.DLE.]-[.US.]]\'
All columns in all tables
You could then use this procedure in a loop through all tables, using a similar method to above. Below is how you'd pull all the table names from information_schema:
SELECT DISTINCT TABLE_NAME FROM information_schema.TABLES
WHERE TABLE_SCHEMA = 'your_database_name';
We are cleaning our MySQL database and finding many rows with empty string (as value) in ENUM columns. MySQL has the nasty habit of ignoring errors like that, you know.
So, since we have hundreds of tables, I was wondering if there is a way to find which tables have this problem. Solutions involving queries, procedures or phpMyAdmin commands are welcome.
Thanks in advance!
P.S. My first question here! Yay!
So, following the suggestions in the comments, I made a procedure that prints the table name, the column name and the number of rows with empty strings instead of a ENUM value.
It works in mysql terminal app, but not in phpMyAdmin.
USE __DATABASE_NAME_HERE__;
DELIMITER //
DROP PROCEDURE IF EXISTS hunt //
CREATE PROCEDURE hunt()
BEGIN
DECLARE done INT DEFAULT false;
DECLARE current_table_name VARCHAR(255);
DECLARE current_column_name VARCHAR(255);
DECLARE my_cursor CURSOR FOR
SELECT table_name, column_name
FROM information_schema.columns
WHERE information_schema.columns.table_schema = "__DATABASE_NAME_HERE__" AND information_schema.columns.column_type LIKE "%ENUM%";
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
OPEN my_cursor;
my_loop: LOOP
FETCH my_cursor INTO current_table_name, current_column_name;
IF done THEN
LEAVE my_loop;
END IF;
SET #sql = CONCAT(
' SELECT COUNT(*) as column_count, "', current_table_name, '" AS table_name, "', current_column_name, '" AS column_name',
' FROM ', current_table_name,
' WHERE `', current_column_name , '` = ""',
' HAVING column_count > 0'
);
PREPARE stmt FROM #sql;
EXECUTE stmt;
DROP PREPARE stmt;
END LOOP;
CLOSE my_cursor;
END //
DELIMITER ;
CALL hunt();
Seen a lot for dropping tables using a wildcard but not a direct SQL statement except this one:
http://azimyasin.wordpress.com/2007/08/11/mysql-dropping-multiple-tables/
It says:
SHOW TABLES LIKE ‘phpbb_%’;
then DROP TABLES, is there a neat way to combine this all into one SQL Statement?
You could use dynamic SQL to do it, inside a stored procedure. It'd look something like this (untested):
CREATE PROCEDURE drop_like (IN pattern VARCHAR(64))
BEGIN
DECLARE q tinytext;
DECLARE done INT DEFAULT FALSE;
DECLARE cur CURSOR FOR
SELECT CONCAT('DROP TABLE "', table_schema, '"."', table_name, '"')
FROM information_schema.tables WHERE table_type = 'BASE TABLE' AND table_name LIKE pattern;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
OPEN cur;
drop_loop: LOOP
FETCH cur INTO q;
IF done THEN
LEAVE drop_loop;
END IF;
PREPARE stmt FROM #q;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END LOOP;
CLOSE cur;
END;
Using dynamic SQL in a query, as per derobert's answer, is the only to do this with pure SQL (no app code).
I wrote a generalized procedure to do this sort of thing (run a query for every table in a database) that you can find here - to use it, you would just need to run this query:
CALL p_run_for_each_table('databasename', 'DROP TABLE `{?database}`.`{?table}`');
It works in essentially the same way as derobert's answer.
However, the writer of that blog post was probably expecting you to write app code to turn the names of tables into a single DROP statement.
To do this, you would iterate over the results of the SHOW TABLE in your code and build a single query like this:
DROP TABLE table1, table2, tablewhatever;
This can be achieved via stored procedure, for example:
CREATE DEFINER=`some_user`#`%` PROCEDURE `drop_tables`()
LANGUAGE SQL
NOT DETERMINISTIC
MODIFIES SQL DATA
SQL SECURITY DEFINER
COMMENT ''
BEGIN
#We need to declare a variable with default 0 to determine weather to continue the loop or exit the loop.
DECLARE done INT DEFAULT 0;
DECLARE archive_table_name VARCHAR(100);
#Select desired tables from `information_schema`
DECLARE cur CURSOR FOR
SELECT t.`TABLE_NAME` FROM information_schema.`TABLES` t WHERE t.`TABLE_NAME` LIKE 'some_table_name%'
AND t.CREATE_TIME BETWEEN DATE_SUB(NOW(), INTERVAL 9 MONTH) AND DATE_SUB(NOW(), INTERVAL 6 MONTH);
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
OPEN cur;
read_loop: LOOP
#Fetch one record from CURSOR and set variable (if not found, then variable `done` will be set to 1 by continue handler)
FETCH cur INTO archive_table_name;
IF done THEN
LEAVE read_loop; #If done is set to 1, then exit the loop, else continue
END IF;
#Do your work
-- Create the truncate query
SET #s = CONCAT('DROP TABLE IF EXISTS ', archive_table_name);
-- Prepare, execute and deallocate the truncate query
PREPARE drop_statement FROM #s;
EXECUTE drop_statement;
DEALLOCATE PREPARE drop_statement;
END LOOP;
CLOSE cur; #Closing the cursor
END
Pay attention to the database user, which is creating/executing the stored routine: it must have appropriate credentials for executing/dropping tables.
I'm working on an old database already in use for years and really crappy designed.
There is a table, "Articles", which contains a "code" column that will be our PK.
And many tables like "idXXXXX" where XXXXX is a "code" value with exactly the same structure.
I looked at the application using this database and saw that relations between tables is made there.
I'm not affraid of redesign the database access in the application, but I don't want to lose years of entries in the database.
I want to create a "campain" table which will have an "id" PK and a "id_code" as FK linking "campain" to "articles"
I'm not a SQL master but I know I can get tables names with
SELECT TABLE_NAME FROM INFORMATION_SCHEMA WHERE TABLE_NAME LIKE 'id%'
But I have really no idea about how to deal with the result (which is fine).
So how can I access to every tables named "idXXX" and insert every rows in the "campain" table + set "id_code" column to "XXX"?
Here is the procedure I saved (I didn't add every fields in the INSERT line for testing purpose) :
CREATE PROCEDURE JoinAllTables()
BEGIN
DECLARE done INT default 0;
DECLARE tableName CHAR(9);
DECLARE buffStr CHAR(7);
DECLARE buffId INT default 0;
DECLARE cur1 CURSOR FOR SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE 'id%';
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
OPEN cur1;
read_loop: LOOP
FETCH cur1 INTO tableName;
IF done THEN
LEAVE read_loop;
END IF;
SET buffStr = SUBSTRING(tableName, 3);
SET buffId = CAST(buffStr AS SIGNED);
set #sql = CONCAT("INSERT INTO campagnes(id, id_code) SELECT null, bufId FROM ",tableName); # Dynamically building sql statement
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END LOOP;
CLOSE cur1;
END;
As u can see, I sub 'idXXXXX' to 'XXXXX' then CAST it AS INTEGER (SIGNED).
But I guess that in the "INSERT INTO" line, second tableName doesn't point to the variable. That's why I'm getting a
"#1446 - Tabble 'bddsoufflage.tablename'doesn't exist" Error :) Any idea ?
Edit: Updated answer
We can't have the tableName dynamically changed inside a prepared statement, so we must go through DynamicSQL to build the query using CONCAT, then compile the SQL with PREPARE, EXECUTE it and DEALLOCATE it.
DELIMITER //
CREATE PROCEDURE JoinAllTables()
BEGIN
DECLARE done INT default 0;
DECLARE tableName CHAR(9);
DECLARE buffStr CHAR(7);
DECLARE buffId INT default 0;
DECLARE cur1 CURSOR FOR SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME LIKE 'id%';
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
OPEN cur1;
read_loop: LOOP
FETCH cur1 INTO tableName;
IF done THEN
LEAVE read_loop;
END IF;
SET buffStr = SUBSTRING(tableName, 3);
SET buffId = CAST(buffStr AS SIGNED);
set #sql = CONCAT("INSERT INTO campagnes(id, id_code) SELECT null, ", buffId, " FROM ",tableName); # Dynamically building sql statement
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END LOOP;
CLOSE cur1;
END; //
See also this answer MySQL Pass table name to cursor select
Old answer
The procedure should look something like this. Thanks Mchl for providing an Insert Into query example, I simply added it to the rest of the procedure.
DELIMITER //
CREATE PROCEDURE JoinAllTables()
BEGIN
DECLARE done INT default 0;
DECLARE tableName CHAR(7); # Variable to contain table names CHAr(7) is assuming id + 5Xs as characters.
DECLARE cur1 CURSOR FOR SELECT TABLE_NAME FROM INFORMATION_SCHEMA WHERE TABLE_NAME LIKE 'id%'; # Create a cursor to iterate over the tables
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
OPEN cur1;
read_loop: LOOP
FETCH cur1 INTO tableName;
IF done THEN
LEAVE read_loop;
END IF;
#Your Insert statement here, using tableName as a field.
INSERT INTO campain (id, id_code, otherfields) SELECT null, tableName, otherfields FROM tableName;
END LOOP;
CLOSE cur1;
END;//
Easiest way would be to run the information_schema query you have within some script (PHP,Python,Perl - whichever suits you best) and use it's results to create queries like:
INSERT INTO
campain (id, id_code, otherfields)
SELECT
null, 'idXXXX', otherfields FROM idXXXX
I have a joomla mysql database with a table name prefix of "jos_" on all of my table names. But I would like to remove it from all of my tables. I understand how to rename each table, one at a time, but I have 600 tables. Is there an easy to run a sql query to do this.
If someone has a solution, could you please post the exact sql query I can use?
In phpmyadmin select all tables of your database.
From the dropdown 'With selected:' choose 'Replace table prefix'
Set from->to replacement.
DONE
You can generate the necessary statements with a single query:
select 'RENAME TABLE ' || table_name || ' TO ' || substr(table_name, 5) ||';'
from information_schema.tables
Save the output of that query to a file and you have all the statements you need.
Or if that returns 0s and 1s rather the statemenets, here's the version using concat instead:
select concat('RENAME TABLE ', concat(table_name, concat(' TO ', concat(substr(table_name, 5), ';'))))
from information_schema.tables;
You can create your own stored procedure to rename your tables, with that you don't need to open an external editor everything will be done on the server:
delimiter //
CREATE PROCEDURE rename_tables( IN db CHAR(255), IN srch CHAR(255), IN rplc CHAR(255) )
BEGIN
DECLARE done INT DEFAULT 0;
DECLARE from_table CHAR(255);
DECLARE cur1 CURSOR FOR SELECT TABLE_NAME FROM information_schema.TABLES WHERE TABLE_SCHEMA=db;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
OPEN cur1;
read_loop: LOOP
IF done THEN
LEAVE read_loop;
END IF;
FETCH cur1 INTO from_table;
SET #to_table = REPLACE(from_table, srch, rplc);
IF from_table != #to_table THEN
SET #rename_query = CONCAT('RENAME TABLE ', db, '.', from_table, ' TO ', #to_table, ';');
PREPARE stmt FROM #rename_query;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END IF;
END LOOP;
CLOSE cur1;
END//
delimiter ;
Usage:
CALL rename_tables('test', 'jos_', '');
Update: This was my first MySQL stored procedure and I ran into the 6 years old bug #5967 which was quite annoying, your variable names must be different from the field names, because if they aren't you'll get NULL values in your variables.
So be aware of that if you decide to write a MySQL stored procedure.