I have a big data raw(bronze) table with ~400 columns. In preparation for this table moving forward to other level tables in prepared (or silver level), I am picking up, let's say, 395 columns from the raw table; however, I don't like to type the name of all 399 columns in my SQL query.
Is there any solution in SQL to save some time?
Instead of
SELECT col1, col2, col3, ..., col395 FROM table
something like
SELECT * EXCEPT col400 FROM table
SELECT CONCAT_WS(' ',
'SELECT',
GROUP_CONCAT(column_name),
'FROM database_name.table_name') query_text
FROM INFORMATION_SCHEMA.COLUMNS
WHERE table_schema = 'database_name'
AND table_name = 'table_name'
AND column_name NOT IN ('excess_column_1', 'excess_column_2', ...);
Insert your database and table names, fill the list of the columns to be excluded, execute the query - and it will produce needed query text.
You may convert this to the stored procedure which composes and executes needed query dynamically and call this SP instead of the query.
First do
EXPLAIN SELECT * FROM tbl;
SHOW WARNINGS;
Then edit the output to remove the column(s) you don't want.
(Next time, think about whether it is wise to have that many columns in a table; 400 is very high.)
Related
I am fairly new to SQL and im not sure if this is possible, but as the title suggests I want to combine the results of a query into one "entity" that I can then later insert into another table.
Here is an example:
SELECT COLUMN1
FROM TABLE1
Let's say the output is this:
COLUMN1
-------
data1
data2
data3
What I want to do is to take the result of this query and turn it into this:
data1,data2,data3
Which I can then insert into another table as one row like this:
INSERT INTO TABLE2(RESULT)
VALUES ('data1,data2,data3')
You can use GROUP_CONCAT to get a list of all your column values:
SELECT GROUP_CONCAT(COLUMN1 SEPARATOR ',') FROM TABLE1
demo: http://sqlfiddle.com/#!9/3cf4ecc/5/0
To directly INSERT the result of this SELECT you can use the following:
INSERT INTO TABLE2 SELECT GROUP_CONCAT(COLUMN1 SEPARATOR ',') FROM TABLE1
demo: http://sqlfiddle.com/#!9/466869/1/1
You can also use a VIEW to keep your tables normalized:
CREATE VIEW V_TABLE1 AS SELECT GROUP_CONCAT(COLUMN1 SEPARATOR ',') FROM TABLE1;
The advantage of a VIEW is that it is always generated from the current values of your table.
... but be careful: As marc_s already mentioned in the comments, it is not recommended to store multiple values as a list in a single column. You should always design normalized databases.
What you need is an aggregate function such as string_agg in postgresql , simply as SELECT string_agg(COLUMN1, ',') FROM TABLE1;
Certainly you can combine it with INSERT, i.e INSERT ...SELECT.
I am running some SQL queries from command line as follows:
cat my_query.sql | mysql --defaults-file=my_conf.cnf
I need to print column names whether the query returns any data or not.
Currently when there is no data to return, i.e when query returns an empty result set this command does not print anything.
For example my query is:
-- id=-1 doesn't exist
SELECT col1, col2 FROM table WHERE id=-1
I need this query to return
col1 col2
Instead it returns nothing.
Is it possible to do that using purely mysql and standard unix commands?
Thanks.
Adding a UNION ALL to a SELECT with "dummy"/blank data might work:
SELECT col1, col2 FROM table WHERE id=-1
UNION ALL
SELECT '', '' -- OR 0, 0 whatever is appropriate
;
I don't run queries from the command line, so this is assuming it normally would give you the column names if you had at least one row in the results.
I have a large data set in a denormalized format. Here is an example of the column names:
foreign_key_ID, P1, P2, P3, P4, P5.... D1, D2, D3.... etc..
These fields all contain similar type of data.
I need to normalize this into my existing table structure:
insert into new_table (id, name, index)
select foreign_key_id, P1, 1
from denormalized_table;
But that means that I need to run separate queries for each field in my denormalized table, just changing a few things:
insert into new_table (id, name, index)
select foreign_key_id, P2, 2
from denormalized_table;
This is getting tedious considering how many of these fields I have.
Is there a way this can be automated into a single operation? I.e.: iterate through the fields (I don't mind creating a list of eligible fields once, somewhere), pull off the last digit of that field name (ie "1" in "P1" and "2" for "P2") use the field name and the extracted index # in the sub-select.
Here's a start:
SELECT column_name, substr(column_name,2) AS `index`
FROM information_schema.columns
WHERE table_schema = 'mydatabasename'
AND table_name = 'denormalized_table'
AND column_name REGEXP '^[PD][0-9]+$'
ORDER BY column_name
You can modify the select list in that statement, to have MySQL generate statements for you:
SELECT CONCAT('INSERT INTO new_table (id, name, `index`) SELECT foreign_key_id, '
,column_name,', ',substr(column_name,2)
,' FROM denormalized_table ;') AS stmt
FROM information_schema.columns
WHERE table_schema = 'mydatabasename'
AND table_name = 'denormalized_table'
AND column_name REGEXP '^[PD][0-9]+$'
ORDER BY column_name
The output from that would be a set of MySQL INSERT statements that you could then execute.
If the number of rows and total size of the data to be inserted is not too large, you could and you want to get the whole conversion done in "one operation", then you could generate a single INSERT INTO ... SELECT statement, using the UNION ALL operator. I would get the majority of the statement like this:
SELECT CONCAT('UNION ALL SELECT foreign_key_id, '
,column_name,', ',substr(column_name,2)
,' FROM denormalized_table ') AS stmt
FROM information_schema.columns
WHERE table_schema = 'mydatabasename'
AND table_name = 'denormalized_table'
AND column_name REGEXP '^[PD][0-9]+$'
ORDER BY column_name
I would take the output from that, and replace the very first UNION ALL with an INSERT INTO .... That would give me a single statement to run to get the whole conversion done.
hat you're looking for is Dynamic SQL. This is where you execute SQL statements that you can assemble programmatically. You can run any arbitrary SQL code that's in a string, as long as you're in a Stored Procedure. See this link: How To have Dynamic SQL in MySQL Stored Procedure
Basically, you can build a string using mySQL statements by iterating over a set of columns. You can use the SHOW COLUMNS syntax (see http://dev.mysql.com/doc/refman/5.0/en/show-columns.html) to return a collection then loop over that resultset and build your dynamic query string and execute that way.
SHOW COLUMNS FROM myTable WHERE Field NOT IN (pkey, otherFieldIDontWantToInclude)
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
How can I merge two MySql tables?
I want to merge multiple tables that have the same structure and make one large table. The tables have similar names, so I want to use the LIKE statement. Can anyone tell me how I can do this?
The tables are very simple, each having an ID column and a few other columns, but there are a large amount of tables, all of which have names like 'TX-xxx', where 'TX' means Texas, and 'xxx' are the counties in Texas; you know there are more than 200 counties in Texas. (In fact, I have to do this for all the states.) So I want to use the statement "LIKE 'TX-___'".
Thanks!
You would have to give more information so we know exactly what you want but you could create a view
CREATE VIEW myViewName AS
select *
from table1
union all
select * from
table2
This way it would show the information from all your tables (and can be limited so in the selects to not show everything) and when table1, table2, etc are changed the view will reflect this. You can change it at anytime and fetch from it as you would a table:
select * from myViewName
Now for grabbing from specific tables I am not sure how you can do this in mysql though I have done it in tsql. This previous question would help you so you might have something like:
-- Create temporary table of varchar(200) to store the name of the tables. Depending on how you want to go through the array maybe an id number (int).
insert into tempTableName (name)
SELECT table_name FROM information_schema.tables WHERE table_schema = 'database_name' and table_name like 'TX_%';
declare #sqlQuery varchar(max)
--Then you will want to loop through the array and build up an sql statement
-- For each loop through:
if len(#sqlQuery) = 0 begin -- first time through
set #sqlQuery = 'select col1, col2, col3 from ' + currentTableName
end else begin -- second+ time through
set #sqlQuery = 'union all select col1, col2, col3 from ' + currentTableName
end
-- after the loop add the create view. Could double check it worked by checking length = 0 again
set #sqlQuery = 'CREATE VIEW myViewName AS ' + #sqlQuery
Once the query string is built up you will execute it with
PREPARE stmt FROM #sqlQuery;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
If I understand your question correctly UNION is what you need. Something like
SELECT field1, field2
FROM (
SELECT field1, field2 from table1
UNION
SELECT field1, field2 from table2
) all_tables
WHERE all_tables.field1 like "%whatever%
Assuming they have the same columns or similar:
insert into #table
Select * from (Select * from tbl1
Union
select * from tbl2
Union
select * from tbl3)
If they don't have the same number/type of columns then you should provide us with that information.
Current situation:
INSERT INTO othertbl
SELECT *
FROM tbl
WHERE id = '1'
So i want to copy a record from tbl to othertbl. Both tables have an autoincremented unique index. Now the new record should have a new index, rather then the value of the index of the originating record else copying results in a index not unique error.
A solution would be to not use the * but since these tables have quite some columns i really think it's getting ugly.
So,.. is there a better way to copy a record which results in a new record in othertbl which has a new autoincremented index without having to write out all columns in the query and using a NULL value for the index.
-hope it makes sense....-
Think you're gonna have to drop the * and specify the columns fella
If you're using SQL Server you could get a list of columns using
SELECT column_name+', ' from INFORMATION_SCHEMA.COLUMNS where table_name = 'tbl'
Building an insert statement using the result from the above should be easy.
You can dump the list of non-auto-increment columns for your table with this query, and then use it in the insert and select statements:
SELECT group_concat(column_name)
from INFORMATION_SCHEMA.COLUMNS
where table_schema = 'myschema'
and table_name = 'tbl'
and extra != 'auto_increment';
Try:
INSERT ...
SELECT *
FROM tbl
ON DUPLICATE KEY UPDATE `id`=NULL
http://dev.mysql.com/doc/refman/5.1/en/insert-select.html