I have a large database (~50,000 rows) with 20 columns, and I want to "split" the data based upon the values in the third column (called FEATURE_CLASS). The values of FEATURE_CLASS are all of type VARCHAR(), and I want to create however many tables I'd need to replace the single, large table with many smaller tables each entitled with whatever the original table's FEATURE_CLASS value was.
Not sure of the best way to go about this, I was thinking something along the lines of creating a temporary table which would serve as an index, each row carrying a unique value of FEATURE_CLASS, then to iterate over the temp table and perform copying operations for each row of the temp table. I'm not sure really where to go from here, any help/ideas would be appreciated. Thanks!
The basic idea is to create a result set of statements that will create the tables for you and populate with the correct data. Run the below to generate the statements. You can then execute these statements manually (copy/paste) or from within a script.
SQL Server Example
SELECT DISTINCT
'SELECT * INTO [' + TableName.FEATURE_CLASS + '] FROM TableName WHERE FEATURE_CLASS = ''' + TableName.FEATURE_CLASS + ''';'
FROM
TableName
If you have any special characters in the FEATURE_CLASS column, you might want to consider removing them in the script above to prevent table names that are either invalid or tough to work with.
For example:
...
'SELECT * INTO [' + REPLACE(TableName.FEATURE_CLASS, '.', '') + '] FROM TableName WHERE FEATURE_CLASS = ''' + TableName.FEATURE_CLASS + ''';'
...
MySQL Example
SELECT DISTINCT
CONCAT('CREATE TABLE `', DB1.FEATURE_CLASS,
'` AS SELECT * FROM DB1 WHERE FEATURE_CLASS = ''',
DB1.FEATURE_CLASS, ''';') AS statements
FROM DB1;
This will give you a MySQL command something like this:
CREATE TABLE `feature_class_value` AS
SELECT * FROM DB1
WHERE FEATURE_CLASS = 'feature_class_value';
Check out the MySQL docs for more info on CREATE TABLE SELECT options https://dev.mysql.com/doc/refman/5.7/en/create-table-select.html.
Related
In my MySQL database, I have a lot of data with the timestamp in this format:
2017.07.13 - 12:00:00:000
I want to change it to be in this format:
2017-07-13T12:00:00:000-0400
I know I need SELECT to get the data from the table and UPDATE to change it to a new record. But I don't know how to edit the timestamp using SQL commands.
How can I edit a string using SQL commands?
You don't need to run a select prior to an update. The update has the data already available. You could do something like this:
update table
set column = concat(replace(replace(column, ' - ', 'T'), '.', '-'), '-0400')
to alter the format of your all dates in the column column of the table table.
Demo: http://sqlfiddle.com/#!9/2699e9/2 (using select because the update wouldnt show anything)
https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat
https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_replace
https://dev.mysql.com/doc/refman/5.7/en/update.html
If you only want to update 1 specific row use a where clause to limit the update affects.
Should work (tested in SQL-Server)
Update tableName SET col = Replace(col," - ","T") + '-0400'
If + is not work for mysql then try CONCAT
Update: Solution for MySQL is
UPDATE tableName SET col = CONCAT(REPLACE(REPLACE(col, ' - ', 'T'), '.', '-'), '-0400')
I know that I can copy the rows into the same table while specifying different values for columns that need to contain different values by actually writing ALL the column names in the query like so:
INSERT INTO
my_table (col1, col2, col3, col4)
SELECT
col1,
col2,
[value_for_col_3],
col4
FROM
my_table;
WHERE [value_for_col_3] is the raw value that I want for the column col3. This works fine, but in cases where my table contains a lot of columns, it would be cumbersome to write all the column names. Is there a way to perform the same operation without typing all the column names of my table (while still being able to specify different values for certain columns)?
Thanks in advance for any help.
In this answer, I'm giving a basic outline instead of writing out all of the code since you're looking to see if what you want can be done. This is a possbile method of doing so.
The method is to get the list of all of the names of the columns except for the ones you don't want, then use the output of those column names in a query.
This shows how to select all but certain columns in SQL Server, but I'm sure syntax could be modified to work in MySQL.
(Copy and pasting the below code from one of the answers here: SQL exclude a column using SELECT * [except columnA] FROM tableA?)
declare #cols varchar(max), #query varchar(max);
SELECT #cols = STUFF
(
(
SELECT DISTINCT '], [' + name
FROM sys.columns
where object_id = (
select top 1 object_id from sys.objects
where name = 'MyTable'
)
and name not in ('ColumnIDontWant1', 'ColumnIDontWant2')
FOR XML PATH('')
), 1, 2, ''
) + ']';
SELECT #query = 'select ' + #cols + ' from MyTable where';
EXEC (#query);
Just the SELECT statement,
SELECT #query = 'select ' + #cols + ' from MyTable where';
would be modified to your insert into statement.
I would like to examine every field in every table row in a MySQL database and:
see whether it contains a certain REGEXP pattern
store the table_name, column_name and contents in a separate table:
create sep_table ( pattern text, newtext test, tablename text, columnname text);
How can I achieve this?
I have a mysql database with hundreds of tables. I want to convert this into a single table, where the table name will be added to the records, meaning that for each row / data in a particular table, I will add a new field, where in this new field the value will be the table name. Is there any way to do this automatically, rather than I have to create a new field on every table and insert the table name manually, this will take year. The record that I have on this db is around 400000 records. Can mysqldump with a script do this? or any other way ?
My suggestion is to generate syntax of the form:
create table xxx as
select t.*, 'table1' as TableName from table1 t union all
select t.*, 'table2' as TableName from table2 t union all
. . .
You can create the select statements using:
select concat('select t.*, ''', Table_Name, ''' as TableName from ', Table_Name, ' union all '
from Information_Schema.Tables
You can then put the create table line in front and remove the final union all.
I am wondering if it is possible to use SQL to create a table that name columns by index(number). Say, I would like to create a table with 10 million or so columns, I definitely don't want to name every column...
I know that I can write a script to generate a long string as SQL command. However, I would like to know if there is a more elegant way to so
Like something I make up here:
CREATE TABLE table_name
(
number_columns 10000000,
data_type INT
)
I guess saying 10 million columns caused a lot of confusion. Sorry about that. I looked up the manual of several major commercial DBMS and seems it is not possible. Thank you for pointing this out.
But another question, which is most important, does SQL support numerical naming of columns, say all the columns have the same type and there is 50 columns. And when referring it, just like
SELECT COL.INDEX(3), COL.INDEX(2) FROM MYTABLE
Does the language support that?
Couldn't resist looking into this, and found that the MySQL Docs say "no" to this, that
There is a hard limit of 4096 columns per table, but the effective
maximum may be less for a given table
You can easily do that in Postgres with dynamic SQL. Consider the demo:
DO LANGUAGE plpgsql
$$
BEGIN
EXECUTE '
CREATE TEMP TABLE t ('
|| (
SELECT string_agg('col' || g || ' int', ', ')
FROM generate_series(1, 10) g -- or 1600?
)
|| ')';
END;
$$;
But why would you even want to give life to such a monstrosity?
As #A.H. commented, there is a hard limit on the number of columns in PostgreSQL:
There is a limit on how many columns a table can contain. Depending on
the column types, it is between 250 and 1600. However, defining a
table with anywhere near this many columns is highly unusual and often
a questionable design.
Emphasis mine.
More about table limitations in the Postgres Wiki.
Access columns by index number
As to your additional question: with a schema like the above you can simply write:
SELECT col3, col2 FROM t;
I don't know of a built-in way to reference columns by index. You can use dynamic SQL again. Or, for a table that consists of integer columns exclusively, this will work, too:
SELECT c[3] AS col3, c[2] AS col2
FROM (
SELECT translate(t::text, '()', '{}')::int[] AS c -- transform row to ARRAY
FROM t
) x
Generally when working with databases your schema should be more or less "defined" so dynamic column adding isn't a built in functionality.
You can, however, run a loop and continually ALTER TABLE to add columns like so:
BEGIN
SET #col_index = 0;
start_loop: LOOP
SET #col_index = #col_index + 1;
IF #col_index <= num_columns THEN
SET #alter_query = (SELECT CONCAT('ALTER TABLE table_name ADD COLUMN added_column_',#col_index,' VARCHAR(50)'));
PREPARE stmt FROM #alter_query;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
ITERATE start_loop;
END IF;
LEAVE start_loop;
END LOOP start_loop;
END;
But again, like most of the advice you have been given, if you think you need that many columns, you probably need to take a look at your database design, I have personally never heard of a case that would need that.
Note: As mentioned by #GDP you can have only 4096 cols and definitely the idea is not appreciated and as again #GDP said that database design ideas need to be explored to consider if something else could be a better way to handle this requirement.
However, I was just wondering apart from the absurd requirement if ever I need to do this how can I do it? I thought why not create a custom / user defined MySQL function e.g. create_table() tht will receive the parameters you intend to send and which will in turn generate the required CREATE TABLE command.
This is an option for finding columns using ordinal values. It might not be the most elegant or efficient but it works. I am using it to create a new table for faster mappings between data that I need to parse through all the columns / rows.
DECLARE #sqlCommand varchar(1000)
DECLARE #columnNames TABLE (colName varchar(64), colIndex int)
DECLARE #TableName varchar(64) = 'YOURTABLE' --Table Name
DECLARE #rowNumber int = 2 -- y axis
DECLARE #colNumber int = 24 -- x axis
DECLARE #myColumnToOrderBy varchar(64) = 'ID' --use primary key
--Store column names in a temp table
INSERT INTO #columnNames (colName, colIndex)
SELECT COL.name AS ColumnName, ROW_NUMBER() OVER (ORDER BY (SELECT 1))
FROM sys.tables AS TAB
INNER JOIN sys.columns AS COL ON COL.object_id = TAB.object_id
WHERE TAB.name = #TableName
ORDER BY COL.column_id;
DECLARE #colName varchar(64)
SELECT #colName = colName FROM #columnNames WHERE colIndex = #colNumber
--Create Dynamic Query to retrieve the x,y coordinates from table
SET #sqlCommand = 'SELECT ' + #colName + ' FROM (SELECT ' + #colName + ', ROW_NUMBER() OVER (ORDER BY ' + #myColumnToOrderBy+ ') AS RowNum FROM ' + #tableName + ') t2 WHERE RowNum = ' + CAST(#rowNumber AS varchar(5))
EXEC(#sqlCommand)