I have a big data raw(bronze) table with ~400 columns. In preparation for this table moving forward to other level tables in prepared (or silver level), I am picking up, let's say, 395 columns from the raw table; however, I don't like to type the name of all 399 columns in my SQL query.
Is there any solution in SQL to save some time?
Instead of
SELECT col1, col2, col3, ..., col395 FROM table
something like
SELECT * EXCEPT col400 FROM table
SELECT CONCAT_WS(' ',
'SELECT',
GROUP_CONCAT(column_name),
'FROM database_name.table_name') query_text
FROM INFORMATION_SCHEMA.COLUMNS
WHERE table_schema = 'database_name'
AND table_name = 'table_name'
AND column_name NOT IN ('excess_column_1', 'excess_column_2', ...);
Insert your database and table names, fill the list of the columns to be excluded, execute the query - and it will produce needed query text.
You may convert this to the stored procedure which composes and executes needed query dynamically and call this SP instead of the query.
First do
EXPLAIN SELECT * FROM tbl;
SHOW WARNINGS;
Then edit the output to remove the column(s) you don't want.
(Next time, think about whether it is wise to have that many columns in a table; 400 is very high.)
this is my problem: I want to check rows in a table which name is parameterized, something like table_X. The value of X comes from another table, so for example in my main table I have a column c_id and a column X, the table to join has name table_X, it EXISTS with no doubt, and it has the same column c_id, which I shall join on, to check if there are values of c_id in that table.
I've tried a view, but without success, because I can't put a parameterized table name in a view. I can parameterize where clauses and other things, but no table names.
I've tried a procedure, with
SET #q = CONCAT('select blabla from table_', X);
PREPARE stmt FROM #q;
EXECUTE stmt;
but procedures can't return values, and I need it, because I need to know if there is the c_id value in the parameterized table, else it is useless.
I've tried a function, but "Dynamic SQL is not allowed in stored function or trigger"
So what can I do to extract this data? I'm calling this view/function/whatever from PHP, and I know I can do it from PHP side, with two queries, but I need to do it db-side, for future implementations.
Is it possible?
NOTE: I can't modify the structure of the DB :) btw, it's the Limesurvey db, sounds like a crazy db structure, huh?
The only way, without dynamically building queries, is to hard code in every combination and pick out the one you want.
If the table name is a parameter to a stored procedure, this can be in IF blocks. But it feels clunky.
If the fields from each table are the same, you can union the tables together and select from those...
CREATE VIEW myUnifiedStructure AS
SELECT 'Table1' AS tableName, * FROM Table1
UNION SELECT 'Table2' AS tableName, * FROM Table2
UNION SELECT 'Table3' AS tableName, * FROM Table3
-- etc
SELECT * FROM myUnifiedStructure WHERE tableName = 'Table1'
If the fields are different in each table, you may only be interested in a subset of the fields...
CREATE VIEW myUnifiedStructure AS
SELECT 'Table1' AS tableName, field1 AS field1, field4 AS field2 FROM Table1
UNION SELECT 'Table2' AS tableName, field2 AS field1, field3 AS field2 FROM Table2
UNION SELECT 'Table3' AS tableName, field2 AS field1, field4 AS field2 FROM Table3
-- etc
Or you can pass in NULLs for fields that don't exist in the source table...
CREATE VIEW myUnifiedStructure AS
SELECT 'Table1' AS tableName, NULL AS field1, field2 AS field2 FROM Table1
UNION SELECT 'Table2' AS tableName, field1 AS field1, field2 AS field2 FROM Table2
UNION SELECT 'Table3' AS tableName, field1 AS field1, NULL AS field2 FROM Table3
-- etc
Why do you need separate tables like this? It's usually a sign of bad design. Wouldn't it just be easier to create a single table with an identifier field for whichever X` value that record belongs to, that you can join/filter on?
That'd reduce the query to something like
SELECT ...
FROM othertable
JOIN bigtable ON othertable.c_id = bigtable.c_id AND othertable.fieldName = bigtable.fieldName
I know that I can copy the rows into the same table while specifying different values for columns that need to contain different values by actually writing ALL the column names in the query like so:
INSERT INTO
my_table (col1, col2, col3, col4)
SELECT
col1,
col2,
[value_for_col_3],
col4
FROM
my_table;
WHERE [value_for_col_3] is the raw value that I want for the column col3. This works fine, but in cases where my table contains a lot of columns, it would be cumbersome to write all the column names. Is there a way to perform the same operation without typing all the column names of my table (while still being able to specify different values for certain columns)?
Thanks in advance for any help.
In this answer, I'm giving a basic outline instead of writing out all of the code since you're looking to see if what you want can be done. This is a possbile method of doing so.
The method is to get the list of all of the names of the columns except for the ones you don't want, then use the output of those column names in a query.
This shows how to select all but certain columns in SQL Server, but I'm sure syntax could be modified to work in MySQL.
(Copy and pasting the below code from one of the answers here: SQL exclude a column using SELECT * [except columnA] FROM tableA?)
declare #cols varchar(max), #query varchar(max);
SELECT #cols = STUFF
(
(
SELECT DISTINCT '], [' + name
FROM sys.columns
where object_id = (
select top 1 object_id from sys.objects
where name = 'MyTable'
)
and name not in ('ColumnIDontWant1', 'ColumnIDontWant2')
FOR XML PATH('')
), 1, 2, ''
) + ']';
SELECT #query = 'select ' + #cols + ' from MyTable where';
EXEC (#query);
Just the SELECT statement,
SELECT #query = 'select ' + #cols + ' from MyTable where';
would be modified to your insert into statement.
I have a large data set in a denormalized format. Here is an example of the column names:
foreign_key_ID, P1, P2, P3, P4, P5.... D1, D2, D3.... etc..
These fields all contain similar type of data.
I need to normalize this into my existing table structure:
insert into new_table (id, name, index)
select foreign_key_id, P1, 1
from denormalized_table;
But that means that I need to run separate queries for each field in my denormalized table, just changing a few things:
insert into new_table (id, name, index)
select foreign_key_id, P2, 2
from denormalized_table;
This is getting tedious considering how many of these fields I have.
Is there a way this can be automated into a single operation? I.e.: iterate through the fields (I don't mind creating a list of eligible fields once, somewhere), pull off the last digit of that field name (ie "1" in "P1" and "2" for "P2") use the field name and the extracted index # in the sub-select.
Here's a start:
SELECT column_name, substr(column_name,2) AS `index`
FROM information_schema.columns
WHERE table_schema = 'mydatabasename'
AND table_name = 'denormalized_table'
AND column_name REGEXP '^[PD][0-9]+$'
ORDER BY column_name
You can modify the select list in that statement, to have MySQL generate statements for you:
SELECT CONCAT('INSERT INTO new_table (id, name, `index`) SELECT foreign_key_id, '
,column_name,', ',substr(column_name,2)
,' FROM denormalized_table ;') AS stmt
FROM information_schema.columns
WHERE table_schema = 'mydatabasename'
AND table_name = 'denormalized_table'
AND column_name REGEXP '^[PD][0-9]+$'
ORDER BY column_name
The output from that would be a set of MySQL INSERT statements that you could then execute.
If the number of rows and total size of the data to be inserted is not too large, you could and you want to get the whole conversion done in "one operation", then you could generate a single INSERT INTO ... SELECT statement, using the UNION ALL operator. I would get the majority of the statement like this:
SELECT CONCAT('UNION ALL SELECT foreign_key_id, '
,column_name,', ',substr(column_name,2)
,' FROM denormalized_table ') AS stmt
FROM information_schema.columns
WHERE table_schema = 'mydatabasename'
AND table_name = 'denormalized_table'
AND column_name REGEXP '^[PD][0-9]+$'
ORDER BY column_name
I would take the output from that, and replace the very first UNION ALL with an INSERT INTO .... That would give me a single statement to run to get the whole conversion done.
hat you're looking for is Dynamic SQL. This is where you execute SQL statements that you can assemble programmatically. You can run any arbitrary SQL code that's in a string, as long as you're in a Stored Procedure. See this link: How To have Dynamic SQL in MySQL Stored Procedure
Basically, you can build a string using mySQL statements by iterating over a set of columns. You can use the SHOW COLUMNS syntax (see http://dev.mysql.com/doc/refman/5.0/en/show-columns.html) to return a collection then loop over that resultset and build your dynamic query string and execute that way.
SHOW COLUMNS FROM myTable WHERE Field NOT IN (pkey, otherFieldIDontWantToInclude)
In MySQL I am trying to copy a row with an autoincrement column ID=1 and insert the data into same table as a new row with column ID=2.
How can I do this in a single query?
Use INSERT ... SELECT:
insert into your_table (c1, c2, ...)
select c1, c2, ...
from your_table
where id = 1
where c1, c2, ... are all the columns except id. If you want to explicitly insert with an id of 2 then include that in your INSERT column list and your SELECT:
insert into your_table (id, c1, c2, ...)
select 2, c1, c2, ...
from your_table
where id = 1
You'll have to take care of a possible duplicate id of 2 in the second case of course.
IMO, the best seems to use sql statements only to copy that row, while at the same time only referencing the columns you must and want to change.
CREATE TEMPORARY TABLE temp_table ENGINE=MEMORY
SELECT * FROM your_table WHERE id=1;
UPDATE temp_table SET id=0; /* Update other values at will. */
INSERT INTO your_table SELECT * FROM temp_table;
DROP TABLE temp_table;
See also av8n.com - How to Clone an SQL Record
Benefits:
The SQL statements 2 mention only the fields that need to be changed during the cloning process. They do not know about – or care about – other fields. The other fields just go along for the ride, unchanged. This makes the SQL statements easier to write, easier to read, easier to maintain, and more extensible.
Only ordinary MySQL statements are used. No other tools or programming languages are required.
A fully-correct record is inserted in your_table in one atomic operation.
Say the table is user(id, user_name, user_email).
You can use this query:
INSERT INTO user (SELECT NULL,user_name, user_email FROM user WHERE id = 1)
This helped and it supports a BLOB/TEXT columns.
CREATE TEMPORARY TABLE temp_table
AS
SELECT * FROM source_table WHERE id=2;
UPDATE temp_table SET id=NULL WHERE id=2;
INSERT INTO source_table SELECT * FROM temp_table;
DROP TEMPORARY TABLE temp_table;
USE source_table;
For a quick, clean solution that doesn't require you to name columns, you can use a prepared statement as described here:
https://stackoverflow.com/a/23964285/292677
If you need a complex solution so you can do this often, you can use this procedure:
DELIMITER $$
CREATE PROCEDURE `duplicateRows`(_schemaName text, _tableName text, _whereClause text, _omitColumns text)
SQL SECURITY INVOKER
BEGIN
SELECT IF(TRIM(_omitColumns) <> '', CONCAT('id', ',', TRIM(_omitColumns)), 'id') INTO #omitColumns;
SELECT GROUP_CONCAT(COLUMN_NAME) FROM information_schema.columns
WHERE table_schema = _schemaName AND table_name = _tableName AND FIND_IN_SET(COLUMN_NAME,#omitColumns) = 0 ORDER BY ORDINAL_POSITION INTO #columns;
SET #sql = CONCAT('INSERT INTO ', _tableName, '(', #columns, ')',
'SELECT ', #columns,
' FROM ', _schemaName, '.', _tableName, ' ', _whereClause);
PREPARE stmt1 FROM #sql;
EXECUTE stmt1;
END
You can run it with:
CALL duplicateRows('database', 'table', 'WHERE condition = optional', 'omit_columns_optional');
Examples
duplicateRows('acl', 'users', 'WHERE id = 200'); -- will duplicate the row for the user with id 200
duplicateRows('acl', 'users', 'WHERE id = 200', 'created_ts'); -- same as above but will not copy the created_ts column value
duplicateRows('acl', 'users', 'WHERE id = 200', 'created_ts,updated_ts'); -- same as above but also omits the updated_ts column
duplicateRows('acl', 'users'); -- will duplicate all records in the table
DISCLAIMER: This solution is only for someone who will be repeatedly duplicating rows in many tables, often. It could be dangerous in the hands of a rogue user.
If you're able to use MySQL Workbench, you can do this by right-clicking the row and selecting 'Copy row', and then right-clicking the empty row and selecting 'Paste row', and then changing the ID, and then clicking 'Apply'.
Copy the row:
Paste the copied row into the blank row:
Change the ID:
Apply:
insert into MyTable(field1, field2, id_backup)
select field1, field2, uniqueId from MyTable where uniqueId = #Id;
A lot of great answers here. Below is a sample of the stored procedure that I wrote to accomplish this task for a Web App that I am developing:
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON
-- Create Temporary Table
SELECT * INTO #tempTable FROM <YourTable> WHERE Id = Id
--To trigger the auto increment
UPDATE #tempTable SET Id = NULL
--Update new data row in #tempTable here!
--Insert duplicate row with modified data back into your table
INSERT INTO <YourTable> SELECT * FROM #tempTable
-- Drop Temporary Table
DROP TABLE #tempTable
You can also pass in '0' as the value for the column to auto-increment, the correct value will be used when the record is created. This is so much easier than temporary tables.
Source:
Copying rows in MySQL
(see the second comment, by TRiG, to the first solution, by Lore)
I tend to use a variation of what mu is too short posted:
INSERT INTO something_log
SELECT NULL, s.*
FROM something AS s
WHERE s.id = 1;
As long as the tables have identical fields (excepting the auto increment on the log table), then this works nicely.
Since I use stored procedures whenever possible (to make life easier on other programmers who aren't too familiar with databases), this solves the problem of having to go back and update procedures every time you add a new field to a table.
It also ensures that if you add new fields to a table they will start appearing in the log table immediately without having to update your database queries (unless of course you have some that set a field explicitly)
Warning: You will want to make sure to add any new fields to both tables at the same time so that the field order stays the same... otherwise you will start getting odd bugs. If you are the only one that writes database interfaces AND you are very careful then this works nicely. Otherwise, stick to naming all of your fields.
Note: On second thought, unless you are working on a solo project that you are sure won't have others working on it stick to listing all field names explicitly and update your log statements as your schema changes. This shortcut probably is not worth the long term headache it can cause... especially on a production system.
INSERT INTO `dbMyDataBase`.`tblMyTable`
(
`IdAutoincrement`,
`Column2`,
`Column3`,
`Column4`
)
SELECT
NULL,
`Column2`,
`Column3`,
'CustomValue' AS Column4
FROM `dbMyDataBase`.`tblMyTable`
WHERE `tblMyTable`.`Column2` = 'UniqueValueOfTheKey'
;
/* mySQL 5.6 */
Try this:
INSERT INTO test_table (SELECT null,txt FROM test_table)
Every time you run this query, This will insert all the rows again with new ids. values in your table and will increase exponentially.
I used a table with two columns i.e id and txt and id is auto increment.
I was looking for the same feature but I don't use MySQL. I wanted to copy ALL the fields except of course the primary key (id). This was a one shot query, not to be used in any script or code.
I found my way around with PL/SQL but I'm sure any other SQL IDE would do. I did a basic
SELECT *
FROM mytable
WHERE id=42;
Then export it to a SQL file where I could find the
INSERT INTO table (col1, col2, col3, ... , col42)
VALUES (1, 2, 3, ..., 42);
I just edited it and used it :
INSERT INTO table (col1, col2, col3, ... , col42)
VALUES (mysequence.nextval, 2, 3, ..., 42);
insert into your_table(col1,col2,col3) select col1+1,col2,col3 from your_table where col1=1;
Note:make sure that after increment the new value of col1 is not duplicate entry if col1 is primary key.
CREATE TEMPORARY TABLE IF NOT EXISTS `temp_table` LIKE source_table;
DELETE FROM `purchasing2` ;
INSERT INTO temp_table SELECT * FROM source_table where columnid = 2;
ALTER TABLE temp_table MODIFY id INT NOT NULL;
ALTER TABLE temp_table DROP PRIMARY KEY;
UPDATE temp_table SET id=NULL ;
INSERT INTO source_table SELECT * FROM temp_table;
DROP TEMPORARY TABLE IF EXISTS temp_table ;
Dump the row you want to sql and then use the generated SQL, less the ID column to import it back in.