I have 2 columns in the table which have incorrect entries. The size of the table runs in to billions of records. I had like to swap data between two columns (c1 and C2)
The approach taken is to export the data in small chunks in to CSV files and then import it back with corrected entries. For example, below is the data set
--------
|C1 | C2 |
|3 | 4 |
|4 | 6 |
I would then export the data in to a semicolon delimted CSV file (complete command NOT shown) as below
SELECT C2,C1 FROM TABLE temp INTO OUTFILE /tmp/test.csv
The output of such command would be
4;3
6;4
When I import back the data (after deleting the data in question), the data will be corrected as follows
| C1 C2 |
| 3 4 |
| 4 6 |
It is really a matter of OUTFILE and INFILE operation, I believe
Question
Does the approach makes sense? The real data also expects NULL, int
values in some of the columns apart from data swaps.
The other complexity is in the production database, I will need to
use the WHERE clause. The table name would also be fetched
dynamically.
With reference to point 2, how do I add dynamicity to the queries.
Should I use a STORED procedure or SHELL SCRIPT? STORED Procedure
does not seem to support LOAD DATA INFILE functionality.
If I am left with shell, any sample script that I can reuse? The CSV
filename, table name and WHERE clause will have to be built at run
time.
Also the size of the chunk to be exported and imported will be calculated dynamically.
Any other approach?
Note - This is a INFOBRIGHT column based table on top of mysql. The UPDATE query is non-performant and ALTER TABLE is not supported by INFOBRIGHT.
you can use this approach
create a temporary table 'temp_table' and the use this procedure, then call it from anywhere with the name of the table like this.
call change_fields('table_origen');
the procedure works like this
delete the data in the temp_table
insert into this temp_table with the order changed from the table_origen
delete from the table_origen
insert into the table_origen the information from the temp_table
You can go crazy and how many variables you want to accept
CREATE DEFINER=`root`#`%` PROCEDURE `change_fields`(IN `tableName` VARCHAR(200))
LANGUAGE SQL
NOT DETERMINISTIC
CONTAINS SQL
SQL SECURITY DEFINER
COMMENT ''
BEGIN
SET #hr1 = CONCAT('Delete from temp_table;');
PREPARE hrStmt1 FROM #hr1;
EXECUTE hrStmt1;
DEALLOCATE PREPARE hrStmt1;
SET #hr1 = CONCAT('insert into temp_table (C_1, C_2) Select C2, C1 from `',tableName,'`;');
PREPARE hrStmt1 FROM #hr1;
EXECUTE hrStmt1;
DEALLOCATE PREPARE hrStmt1;
SET #hr1 = CONCAT('Delete from `',tableName,'`;');
PREPARE hrStmt1 FROM #hr1;
EXECUTE hrStmt1;
DEALLOCATE PREPARE hrStmt1;
SET #hr1 = CONCAT('insert into `',tableName,'` (C1,C2) select C_1, C_2 from temp_table;');
PREPARE hrStmt1 FROM #hr1;
EXECUTE hrStmt1;
DEALLOCATE PREPARE hrStmt1;
END;
Related
I have a question that is plaguing me. I have a mysql database that is being used for historical data. Basically every year the system writes a new table in the database housing the time data.
so it is like this:
tables:
year_2017
year_2018
year_2019
and in each table there is data.
What I would like to do is build a query that will search through these specific tables.
Now there are other tables in there I don't want to search through. I only want the year_* tables.
Basically I would want SELECT * FROM year_*
I know you could do SELECT * FROM year_2017, year_2018, year_2019 but the problem is that every year there is going to be a new table so I would have to edit the query every year and I don't want to do that.
Any help would be great.
From the information_schema.tables, find the names of the tables that contain the year_ and then place the table names in a temporary table. Then fetch each table with a data loop.
CREATE PROCEDURE myproc()
BEGIN
CREATE TEMPORARY TABLE temp(
tablename varchar(100)
);
INSERT INTO temp
SELECT t.TABLE_NAME
FROM information_schema.tables t
WHERE t.TABLE_NAME LIKE '%year_%';
WHILE (Select Count(*) From temp) > 0 DO
SET #tablename= (Select tablename From temp limit 1);
SET #sql= concat("select * from ", #tablename);
prepare exec from #sql;
execute exec;
Delete From temp Where tablename=#tablename;
END WHILE;
END;
CALL myproc();
demo in db<>fiddle
You can use a variable to store the table name. But you'd need to come up with an automation to fill the table name variables.
mysql prepared statements
/* MYSQL */
SET #table = 't1';
SET #s = CONCAT('SELECT * FROM ', #table);
PREPARE stmt3 FROM #s;
EXECUTE stmt3;
I'm trying to write a MySQL stored proceedure that loops through all existing tables in my database and creates a copy/clone of each table. I'm using a cursor to loop through the table names then create a new table like this:
DELIMITER //
CREATE PROCEDURE CopyTables()
BEGIN
DECLARE finished INT DEFAULT 0;
DECLARE tableName VARCHAR(100);
DECLARE copyTableName VARCHAR(100);
DECLARE curTables
CURSOR FOR
SELECT table_name FROM INFORMATION_SCHEMA.TABLES;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET finished = 1;
OPEN curTables;
create_loop: LOOP
FETCH curTables INTO tableName;
IF finished THEN LEAVE create_loop; END IF;
SELECT concat('Processing table ', tableName);
SET copyTableName = CONCAT('copy_',tableName);
SELECT concat('Creating table ', copyTableName);
CREATE TABLE copyTableName LIKE tableName;
END LOOP;
CLOSE curTables;
END //
DELIMITER;
But I get an error when calling the stored procedure:
> call CopyTables()
[2020-12-08 18:16:03] 1 row retrieved starting from 1 in 77 ms (execution: 15 ms, fetching: 62 ms)
[2020-12-08 18:16:03] [S1000] Attempt to close streaming result set com.mysql.cj.protocol.a.result.ResultsetRowsStreaming#7a714591 that was not registered. Only one streaming result set may be open and in use per-connection. Ensure that you have called .close() on any active result sets before attempting more queries.
Is the result set exception effectively complaining because I'm creating new tables which is effectively messing with the cursor/select? I've got additional table changes on both the original and copied table to perform, like adding new columns, creating triggers, modifying constraints.
The list of table names is not static, and this should be able to run on whatever database I need it.
Can you suggest another way to achieve this without the cursor perhaps?
The problem is that the procedure is returning multiple result sets, but your Java client is not handling that correctly.
Refer to How do you get multiple resultset from a single CallableStatement?
Another problem with your procedure is that you aren't creating tables the way you think you are.
This statement:
CREATE TABLE copyTableName LIKE tableName;
will only create a table named literally copyTableName that is like another table that is literally tableName. It will NOT use the values of variables by those names.
To do what you want, you need to use a prepared statement:
SET #sql = CONCAT('CREATE TABLE `', copyTableName, '` LIKE `', tableName, '`');
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
This way the value of your variables is concatenated into an SQL statement.
Note that PREPARE only accepts a user-defined session variable, the type with the # sigil. It doesn't work with local variables you create in your procedure with DECLARE. Read https://dev.mysql.com/doc/refman/8.0/en/prepare.html and https://dev.mysql.com/doc/refman/8.0/en/user-variables.html
create table a.a {
db varchar(255)
}
//first record of a.a is db = b
create table b.a {
id varchar(255)
}
mysql to run:
use concat((select * from a.a limit 1),".a"); select * from a;
How can i achieve the above? Using the returned results of one table to access another database without doing round trips to mysql connection.
Dynamic SQL requires using PREPARE in a stored procedure.
SET #sql = (SELECT CONCAT('SELECT * FROM `', a.a, '`')
FROM a
LIMIT 1);
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
It's generally considered best to avoid designing your database so that it requires this. Don't spread the data among multiple tables with the same structure, use a single table and add another column to distinguish the records.
I have create a table in MySQL where i have all the names of data I'm going to bring in from FRED data.
Now I want to make a new table, where the 1504 names in my example table above, each has one column.
AAA AAA10M AAAFFM ADBJORNS and so on.
So every name in fred_namecol should get one column each with numeric as value. Is there a easy way to do this instead of writing everyone manually?
Maybe there is a way too loop trough each name and make a column for them into a new table?
Yes. There is a way.
In the following query; replace WRITE_YOUR_TABLE_NAME_HERE with the name of the table which contains fred_namecol column:
drop table if exists columns_over_1000;
set #prefix = 'CREATE TABLE columns_over_1000 (';
set #suffix = '\n);';
select #create_table_sql := concat(
#prefix,
group_concat('\n `', fred_namecol, '` INT'),
#suffix)
from `WRITE_YOUR_TABLE_NAME_HERE`;
PREPARE stmt1 FROM #create_table_sql;
EXECUTE stmt1;
DEALLOCATE PREPARE stmt1;
select * from columns_over_1000;
I have the following query I want to execute in my stored procedure WITHOUT PREPARING the query, since this gives me problems with OUT to pass back parameters.
DELIMITER //
CREATE PROCEDURE Test (
IN CID BIGINT(20),
IN IDs LONGTEXT
)
BEGIN
#EXECUTE UNDERNEATH QUERY
SELECT * FROM CONCAT('Part1_OfTableName', CID); #CID IS CustomerID
END //
DELIMITER ;
However, this fails and I don't know how to fix the problem.
(Note that in the example I have no spaces in my table name, however in my situation I might have a space in my table name though)
PREPARE should have no bearing on your ability to successfully set OUT parameters of your procedure
SET DELIMITER //
CREATE PROCEDURE test(IN cid INT, IN ids TEXT, OUT out_int INT)
BEGIN
SET #sql = CONCAT('SELECT * FROM `table_', cid, '`', CASE WHEN ids IS NULL THEN '' ELSE CONCAT(' WHERE id IN( ', ids, ')') END);
PREPARE stmt FROM #sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
SET out_int = 1;
END//
SET DELIMITER ;
Sample usage:
mysql> CALL test(1, '2,3', #out_int);
+------+
| id |
+------+
| 2 |
| 3 |
+------+
2 rows in set (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT #out_int;
+----------+
| #out_int |
+----------+
| 1 |
+----------+
1 row in set (0.00 sec)
If you need to return results from a stored procedure using sql statement that must be prepared, you can use an intermediate temp table.
BEGIN
CREATE TEMPORARY TABLE `myresults` blah blah....;
//construct and prepare select you would've used, but start it with an insert like so...
// INSERT INTO `myresults` SELECT ....
// Execute the prepared query
SELECT * FROM `myresults`;
DROP TEMPORARY TABLE `myresults`;
END
...at least I am pretty sure this technique used to work; I've been working more in MSSQL the last couple years.
Something to note:
Temporary tables are connection/session specific, so while safe from a global perspective using a generic name like myresults can be problematic if queries executed earlier on the connection/session (or by a procedure calling this one) use the same name; in practice/paranoia, I tended to use a different guid (in each procedure using this technique) as a prefix for any temporary tables generated within it.