What is the best way to do a find replace of an entire database in MySQL, without having to actually script it in a language (I mean, if that's what has to be done, so be it).
Basically I want to replace a site name, with another site name (I'm setting up a dev server for upgrades) in Drupal.
My thoughts are that all links in the database need to be changed. I would prefer to use PHPMyAdmin to do this, but I'm happy to use the command line as well.
Assuming you have a table setup like:
sites
site_id
url
You could use REPLACE() in an UPDATE command:
UPDATE sites SET url=REPLACE(url, 'findUrl.com', 'replaceUrl.com');
UPDATE: To support the need to "find & replace" on every column in every table in a given database, a Stored Procedure is required (if you want to do it directly in MySQL, of course). Here's a rough-draft (untested) of what could do it:
DELIMITER $$
CREATE PROCEDURE `find_replace_in_all_tables`()
BEGIN
DECLARE done INT DEFAULT 0;
DECLARE TABLE_NAME CHAR(255);
DECLARE COLUMN_NAME CHAR(255);
DECLARE tables CURSOR for
SELECT table_name, column_name FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = DATABASE();
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
OPEN tables;
WHILE done = 0 DO
FETCH NEXT FROM tables INTO TABLE_NAME, COLUMN_NAME;
IF done = 0 THEN
SET #SQL_TEXT = CONCAT("UPDATE `", TABLE_NAME, "` SET `", COLUMN_NAME, "`=REPLACE(`", COLUMN_NAME, "`, 'findUrl.com', 'replaceUrl.com');");
PREPARE statement FROM #SQL_TEXT;
EXECUTE statement;
DEALLOCATE PREPARE statement;
END IF;
END WHILE;
CLOSE tables;
END
The stored procedure iterates through the information_schema database for every table/column in your current database. With this list, it builds an UPDATE query similar to my original answer and executes it. This could be made more efficient if you limit the column-types in the SELECT query, or if you know the exact column names the replacement can take place on (but I guess then you wouldn't need to do a find+replace.
Related
I'm trying to write a MySQL stored proceedure that loops through all existing tables in my database and creates a copy/clone of each table. I'm using a cursor to loop through the table names then create a new table like this:
DELIMITER //
CREATE PROCEDURE CopyTables()
BEGIN
DECLARE finished INT DEFAULT 0;
DECLARE tableName VARCHAR(100);
DECLARE copyTableName VARCHAR(100);
DECLARE curTables
CURSOR FOR
SELECT table_name FROM INFORMATION_SCHEMA.TABLES;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET finished = 1;
OPEN curTables;
create_loop: LOOP
FETCH curTables INTO tableName;
IF finished THEN LEAVE create_loop; END IF;
SELECT concat('Processing table ', tableName);
SET copyTableName = CONCAT('copy_',tableName);
SELECT concat('Creating table ', copyTableName);
CREATE TABLE copyTableName LIKE tableName;
END LOOP;
CLOSE curTables;
END //
DELIMITER;
But I get an error when calling the stored procedure:
> call CopyTables()
[2020-12-08 18:16:03] 1 row retrieved starting from 1 in 77 ms (execution: 15 ms, fetching: 62 ms)
[2020-12-08 18:16:03] [S1000] Attempt to close streaming result set com.mysql.cj.protocol.a.result.ResultsetRowsStreaming#7a714591 that was not registered. Only one streaming result set may be open and in use per-connection. Ensure that you have called .close() on any active result sets before attempting more queries.
Is the result set exception effectively complaining because I'm creating new tables which is effectively messing with the cursor/select? I've got additional table changes on both the original and copied table to perform, like adding new columns, creating triggers, modifying constraints.
The list of table names is not static, and this should be able to run on whatever database I need it.
Can you suggest another way to achieve this without the cursor perhaps?
The problem is that the procedure is returning multiple result sets, but your Java client is not handling that correctly.
Refer to How do you get multiple resultset from a single CallableStatement?
Another problem with your procedure is that you aren't creating tables the way you think you are.
This statement:
CREATE TABLE copyTableName LIKE tableName;
will only create a table named literally copyTableName that is like another table that is literally tableName. It will NOT use the values of variables by those names.
To do what you want, you need to use a prepared statement:
SET #sql = CONCAT('CREATE TABLE `', copyTableName, '` LIKE `', tableName, '`');
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
This way the value of your variables is concatenated into an SQL statement.
Note that PREPARE only accepts a user-defined session variable, the type with the # sigil. It doesn't work with local variables you create in your procedure with DECLARE. Read https://dev.mysql.com/doc/refman/8.0/en/prepare.html and https://dev.mysql.com/doc/refman/8.0/en/user-variables.html
I want to find all the tables in my db that contain the column name Foo, and update its value to 0, I was thinking something like this, but I don't know how to place the UPDATE on that code, I plan on having this statement on the Events inside the MySQL database, I'm using WAMP, the idea is basically having an event run daily which sets all my 'Foo' Columns to 0 without me having to do it manually
SELECT TABLE_NAME, COLUMN_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE column_name LIKE 'Foo'
No, not in a single statement.
To get the names of all that tables that contain column named Foo:
SELECT table_schema, table_name
FROM information_schema.columns
WHERE column_name = 'Foo'
Then, you'd need an UPDATE statement for each table. (It's possible to do update multiple tables in a single statement, but that would need to be an (unnecessary) cross join.) It's better to do each table separately.
You could use dynamic SQL to execute the UPDATE statements in a MySQL stored program (e.g. PROCEDURE)
DECLARE sql VARCHAR(2000);
SET sql = 'UPDATE db.tbl SET Foo = 0';
PREPARE stmt FROM sql;
EXECUTE stmt;
DEALLOCATE stmt;
If you declare a cursor for the select from information_schema.tables, you can use a cursor loop to process a dynamic UPDATE statement for each table_name returned.
DECLARE done TINYINT(1) DEFAULT FALSE;
DECLARE sql VARCHAR(2000);
DECLARE csr FOR
SELECT CONCAT('UPDATE `',c.table_schema,'`.`',c.table_name,'` SET `Foo` = 0') AS sql
FROM information_schema.columns c
WHERE c.column_name = 'Foo'
AND c.table_schema NOT IN ('mysql','information_schema','performance_schema');
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
OPEN csr;
do_foo: LOOP
FETCH csr INTO sql;
IF done THEN
LEAVE do_foo;
END IF;
PREPARE stmt FROM sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END LOOP do_foo;
CLOSE csr;
(This is just an rough outline of an example, not syntax checked or tested.)
FOLLOWUP
Some brief notes about some ideas that were probably glossed over in the answer above.
To get the names of the tables containing column Foo, we can run a query from the information_schema.columns table. (That's one of the tables provided in the MySQL information_schema database.)
Because we may have tables in multiple databases, the table_name is not sufficient to identify a table; we need to know what database the table is in. Rather than mucking with a "use db" statement before we run an UPDATE, we can just reference the table UPDATE db.mytable SET Foo....
We can use our query of information_schema.columns to go ahead and string together (concatenate) the parts we need to create for an UPDATE statement, and have the SELECT return the actual statements we'd need to run to update column Foo, basically this:
UPDATE `mydatabase`.`mytable` SET `Foo` = 0
But we want to substitute in the values from table_schema and table_name in place of mydatabase and mytable. If we run this SELECT
SELECT 'UPDATE `mydatabase`.`mytable` SET `Foo` = 0' AS sql
That returns us a single row, containing a single column (the column happens to be named sql, but name of the column isn't important to us). The value of the column will just be a string. But the string we get back happens to be (we hope) a SQL statement that we could run.
We'd get the same thing if we broke that string up into pieces, and used CONCAT to string them back together for us, e.g.
SELECT CONCAT('UPDATE `','mydatabase','`.`','mytable','` SET `Foo` = 0') AS sql
We can use that query as a model for the statement we want to run against information_schema.columns. We'll replace 'mydatabase' and 'mytable' with references to columns from the information_schema.columns table that give us the database and table_name.
SELECT CONCAT('UPDATE `',c.table_schema,'`.`',c.table_name,'` SET `Foo` = 0') AS sql
FROM information_schema.columns
WHERE c.column_name = 'Foo'
There are some databases we definitely do not want to update... mysql, information_schema, performance_schema. We either need whitelist the databases containing the table we want to update
AND c.table_schema IN ('mydatabase','anotherdatabase')
-or- we need to blacklist the databases we definitely do not want to update
AND c.table_schema NOT IN ('mysql','information_schema','performance_schema')
We can run that query (we could add an ORDER BY if we want the rows returned in a particular order) and what we get back is list containing the statements we want to run. If we saved that set of strings as a plain text file (excluding header row and extra formatting), adding a semicolon at the end of each line, we'd have a file we could execute from the mysql> command line client.
(If any of the above is confusing, let me know.)
The next part is a little more complicated. The rest of this deals with an alternative to saving the output from the SELECT as a plain text file, and executin the statements from the mysql command line client.
MySQL provides a facility/feature that allows us to execute basically any string as a SQL statement, in the context of a MySQL stored program (for example, a stored procedure. The feature we're going to use is called dynamic SQL.
To use dynamic SQL, we use the statements PREPARE, EXECUTE and DEALLOCATE PREPARE. (The deallocate isn't strictly necessary, MySQL will cleanup for us if we don't use it, but I think it's good practice to do it anyway.)
Again, dynamic SQL is available ONLY in the context of a MySQL stored program. To do this, we need to have a string containing the SQL statement we want to execute. As a simple example, let's say we had this:
DECLARE str VARCHAR(2000);
SET str = 'UPDATE mytable SET mycol = 0 WHERE mycol < 0';
To get the contents of str evaluated and executed as a SQL statement, the basic outline is:
PREPARE stmt FROM str;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
The next complicated part is putting that together with the query we are running to get string value we want to execute as SQL statements. To do that, we put together a cursor loop. The basic outline for that is to take our SELECT statement:
SELECT bah FROM humbug
And turn that into a cursor definition:
DECLARE mycursor FOR SELECT bah FROM humbug ;
What we want to is execute that and loop through the rows it returns. To execute the statement and prepare a resultset, we "open" the cursor
OPEN mycursor;
When we're finished with it, we're goin to issue a "close", to release the resultset, so the MySQL server knows we don't need it anymore, and can cleanup, and free up the resources allocated to that.
CLOSE mycursor;
But, before we close the cursor, we want to "loop" through the resultset, fetching each row, and do something with the row. The statement we use to get the next row from the resultset into a procedure variable is:
FETCH mycursor INTO some_variable;
Before we can fetch rows into variables, we need to define the variables, e.g.
DECLARE some_variable VARCHAR(2000);
Since our cursor (SELECT statement) is returning only a single column, we only need one variable. If we had more columns, we'd need a variable for each column.
Eventually, we'll have fetched the last row from the result set. When we attempt to fetch the next one, MySQL is going to throw an error.
Other programming languages would let us just do a while loop, and let us fetch the rows and exit the loop when we've processed them all. MySQL is more arcane. To do a loop:
mylabel: LOOP
-- do something
END LOOP mylabel;
That by itself makes for a very fine infinite loop, because that loop doesn't have an "exit". Fortunately, MySQL gives us the LEAVE statement as a way to exit a loop. We typically don't want to exit the loop the first time we enter it, so there's usually some conditional test we use to determine if we're done, and should exit the loop, or we're not done, and should go around the the loop again.
mylabel: LOOP
-- do something useful
IF some_condition THEN
LEAVE mylabel;
END IF;
END LOOP mylabel;
In our case, we want to loop through all of the rows in the resultset, so we're going to put a FETCH a the first statement inside the loop (the something useful we want to do).
To get a linkage between the error that MySQL throws when we attempt to fetch past the last row in the result set, and the conditional test we have to determine if we should leave...
MySQL provides a way for us to define a CONTINUE HANDLER (some statement we want performed) when the error is thrown...
DECLARE CONTINUE HANDLER FOR NOT FOUND
The action we want to perform is to set a variable to TRUE.
SET done = TRUE;
Before we can run the SET, we need to define the variable:
DECLARE done TINYINT(1) DEFAULT FALSE;
With that we, can change our LOOP to test whether the done variable is set to TRUE, as the exit condition, so our loop looks something like this:
mylabel: LOOP
FETCH mycursor INTO some_variable;
IF done THEN
LEAVE mylabel;
END IF;
-- do something with the row
END LOOP mylabel;
The "do something with the row" is where we want to take the contents of some_variable and do something useful with it. Our cursor is returning us a string that we want to execute as a SQL statement. And MySQL gives us the dynamic SQL feature we can use to do that.
NOTE: MySQL has rules about the order of the statements in the procedure. For example the DECLARE statement have to come at the beginning. And I think the CONTINUE HANDLER has to be the last thing declared.
Again: The cursor and dynamic SQL features are available ONLY in the context of a MySQL stored program, such as a stored procedure. The example I gave above was only the example of the body of a procedure.
To get this created as a stored procedure, it would need to be incorporated as part of something like this:
DELIMITER $$
DROP PROCEDURE IF EXISTS myproc $$
CREATE PROCEDURE myproc
NOT DETERMINISTIC
MODIFIES SQL DATA
BEGIN
-- procedure body goes here
END$$
DELIMITER ;
Hopefully, that explains the example I gave in a little more detail.
This should get all tables in your database and append each table with update column foo statement Copy and run it, the copy the output and run as sql
select concat('update ',table_name,' set foo=0;') from information_schema.tables
where table_schema = 'Your database name here' and table_type = 'base table';
I've found another thread on this question, but I wasn't able to use its solutions, so I thought I'd ask with more clarity and detail.
I have a large MySQL database representing a vBulletin forum. For several years, this forum has had an error generated on each view, each time creating a new table named aagregate_temp_1251634200, aagregate_temp_1251734400, etc etc. There are about 20,000 of these tables in the database, and I wish to delete them all.
I want to issue a command that says the equivalent of DROP TABLE WHERE TABLE_NAME LIKE 'aggregate_temp%';.
Unfortunately this command doesn't work, and the Google results for this problem are full of elaborate stored procedures beyond my understanding and all seemingly tailored to the more complex problems of different posters.
Is it possible to write a simple statement that drops multiple tables based on a name like match?
There's no single statement to do that.
The simplest approach is to generate a set of statements, and execute them individually.
We can write a simple query that will generate the statements for us:
SELECT CONCAT('DROP TABLE `',t.table_schema,'`.`',t.table_name,'`;') AS stmt
FROM information_schema.tables t
WHERE t.table_schema = 'mydatabase'
AND t.table_name LIKE 'aggregate\_temp%' ESCAPE '\\'
ORDER BY t.table_name
The SELECT statement returns a rowset, but each row conveniently contains the exact SQL statement we need to execute to drop a table. (Note that information_schema is a builtin database that contains metadata. We'd need to replace mydatabase with the name of the database we want to drop tables from.
We can save the resultset from this query as a plain text file, remove any heading line, and voila, we've got a script we can execute in our SQL client.
There's no need for an elaborate stored procedure.
A little googling found this:
SELECT 'DROP TABLE "' + TABLE_NAME + '"'
FROM INFORMATION_SCHEMA.TABLES
WHERE TABLE_NAME LIKE 'prefix%'
This should generate a script.
Source: Drop all tables whose names begin with a certain string
From memory you have to use prepared statements, for example: plenty of samples on stack exchange
I would recommend this example:
SQL: deleting tables with prefix
The SQL from above, this one includes the specific databasename - it builds it for you
SELECT CONCAT( 'DROP TABLE ', GROUP_CONCAT(table_name) , ';' )
AS statement FROM information_schema.tables
WHERE table_schema = 'database_name' AND table_name LIKE 'myprefix_%';
Here is a different way to do it:
MySQL bulk drop table where table like?
This will delete all tables with prefix "mg_"
No need to copy and paste rowsets and in phpadmin copying and pasting is problematic as it will cut off long table names and replace them with '...' ruining set of sql commands.
Also note that '_' is a special character so thats why 'mg_' should be encoded as 'mg\_'
(and FOREIGN_KEY_CHECKS needs to be disabled in order to avoid error messages)
SET FOREIGN_KEY_CHECKS = 0;
SET GROUP_CONCAT_MAX_LEN=32768;
SET #tables = NULL;
SELECT GROUP_CONCAT('`', table_name, '`') INTO #tables
FROM information_schema.tables
WHERE table_schema = (SELECT DATABASE()) and table_name like 'mg\_%';
SELECT IFNULL(#tables,'dummy') INTO #tables;
SET #tables = CONCAT('DROP TABLE IF EXISTS ', #tables);
PREPARE stmt FROM #tables;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
SET FOREIGN_KEY_CHECKS = 1;
After some searching here on stackoverflow and on the web, I couldn't find the answer to my question. I'm not a real SQL talent, but I'm trying to covert all the columns in my table to varchar (255). It has about 600 columns which are all varchar, but the size limit varies. I would like them all to be 255. Is there a way to not having to do this manually? I work with MySQL.
Thanks!
You need to generate the alter table statement by pulling the data from the database.
select 'alter table MyTableName modify column ' + column_name + ' varchar(255);'
from information_schema where table_name = 'MyTableName'
And then paste the results of this command into the query window and run it -- making sure it does what you want it to do. Do a backup first.
Or you could make one big alter statement (if MySql wouldn't choke on it) by replacing the semicolon with a comma.
This isn't what you really need to do. You have something more important to do: NORMALIZE YOUR DATABASE
Now, It's impossible that you have a normalized table with 600 columns. Split your entities in that table correctly, following at least the 3rd normal form rules. After that, you'll have a much better database which is easier to mantain.
To do this, you'll need to drop your current table, therefore, you don't need to change all the types to varchar(255) because you'll fix them during the creation of other tables.
This would be a good start to read: http://en.wikipedia.org/wiki/Database_normalization (thanks to #Tim Schmelter from question's comments)
First of all as mentioned by others you better off normalize you data.
In the meantime you can achieve your goal with dynamic SQL like this
DELIMITER $$
CREATE PROCEDURE change_to_varchar255(IN _tname VARCHAR(64))
BEGIN
SET #sql = NULL;
SELECT GROUP_CONCAT(
CONCAT_WS(' ', 'CHANGE', COLUMN_NAME, COLUMN_NAME, 'VARCHAR(255)'))
INTO #sql
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = _tname
AND DATA_TYPE = 'varchar'
AND CHARACTER_MAXIMUM_LENGTH < 255
AND TABLE_SCHEMA = SCHEMA();
SET #sql = CONCAT_WS(' ', 'ALTER TABLE', _tname, #sql);
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END$$
DELIMITER ;
Sample usage:
CALL change_to_varchar255('table1');
Here is SQLFiddle demo
If you are using PhpMyAdmin or other, you can also click on the button to modify the table.
When you are on the web page, press Ctrl+Shift+J under Windows or Cmd+Opt+J under Mac to open the console window in the Chrome Developer tools. Now enter the following command to replace all occurrences of the number 255 with 100 :
document.body.innerHTML = document.body.innerHTML.replace(/255/g, "100").
Finally, click on the button to execute the query.
In our Wordpress 3.0 multi-site installation, we have a custom option for all of our blogs called something like 'platform'. Admins can enter in a value for this platform when creating or editing a blog. Some blogs may have no platform.
We need to be able to create a list of all platforms, and their associated blogs. The problem is, we dynamically create and delete blogs through other site mechanisms, so we have lots of blog options tables with numbers that are not necessarily contiguous. (ie wp_2_options, wp_4_options, wp_12_options, etc.)
My question is this, is there a way in Wordpress to grab an option across all blogs? Conversely, is there a query I could run that would do this manually? I've tried something like this to no effect:
SELECT * FROM
(SELECT table_name
FROM information_schema.tables
WHERE table_name like 'wp_%_options') as t
WHERE option_name='platform'
Does it make sense what I'm trying to do? Again, I apologize for my lack of MySql knowledge, but I haven't been able to find any answers about how to do this. I could also query all these table names first, and then query each table separately, but thats not really an option because we have many blogs, and we may need to run this query for many page requests simultaneously, and this would be adding hundreds of queries to each of these requests.
Any advice or help you guys could give would be greatly appreciated.
In case anyone is interested, I ended up doing it like this (but I would still like to know if its possible to do a search on table names using LIKE and then query those tables, if anyone knows).
// so get all the blog ids from the blogs table
$blogs = $wpdb->get_results("SELECT blog_id FROM {$wpdb->blogs}", ARRAY_A);
// build a sql statement for each blog options table, adding in the blog id for each row
$select_statements = array();
foreach ($blogs as $blog_row) {
$select_statements[] = 'SELECT option_value, CAST( '.$blog_row['blog_id'].' AS UNSIGNED INTEGER ) AS blog_id FROM '.$wpdb->get_blog_prefix($blog_row['blog_id'])."options WHERE option_name='$option_name'";
}
// cache the results of the union of all these select statements
$option_results = $wpdb->get_results(implode(' UNION ALL ', $select_statements), ARRAY_A);
If you want to query directly MySQL database, you can create a procedure and use it:
use wordpress;
Drop Procedure IF EXISTS wordpress.MyProcedure;
DELIMITER | ;
CREATE PROCEDURE MyProcedure (param1 VARCHAR(30))
BEGIN
DECLARE tbname CHAR(50);
DECLARE endfetch INT DEFAULT 0;
DECLARE cur1 CURSOR FOR
SELECT table_name FROM information_schema.tables WHERE table_schema='wordpress' and table_name like '%options';
DECLARE CONTINUE HANDLER FOR SQLSTATE '02000'
SET endfetch = 1;
OPEN cur1;
FETCH cur1 INTO tbname;
fetchloop: WHILE NOT endfetch DO
SELECT tbname ;
SET #opt = param1;
SET #table_name = tbname;
SET #sql_text = concat('SELECT option_value FROM ',#table_name,' WHERE option_name=''',#opt,'''');
PREPARE stmt FROM #sql_text;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
FETCH cur1 INTO tbname;
END WHILE fetchloop;
END
|
DELIMITER ; |
CALL MyProcedure('siteurl');