Is there an accepted practice for bulk inserting values that don't exist into a table without auto-incrementing when you attempt to insert a row that already exists?
There's a great answer for the single-row insert case at:
Prevent auto increment on MySQL duplicate insert
However, for insertion efficiency I'd like to insert a large number of rows with a single SQL command. (i.e.:INSERT INTO myBigTable VALUES ((value1_row1,value2_row1),(value1_row2,value2_row2) ... )
ADDITIONAL INFO:
I would like to have all the ID's available, since my table has the potential of becoming extremely large. Changing the auto_increment variable size to a BIGINT would be a last resort. My insertion application will attempt to insert a large number of already-existing rows on a regular basis (think stock price updates), so I'll effectively have a large number of auto-incremented ID's skipped.
WHY I'M USING AUTO-INCREMENT:
I believe that for query speed, I should be using an integer index into my (very large) table as the primary key instead of a combination of string fields. I also believe that I should use auto_increment, so MySQL handles concurrency for me.
You can use the technique in the other question, using a UNION query:
INSERT INTO yourTable (col1, col2, ...)
SELECT $row1_value1, $row1_value2, ...
FROM DUAL
WHERE NOT EXISTS (
SELECT 1
FROM yourTable
WHERE unique_col = $row1_value1)
UNION ALL
SELECT $row2_value1, $row2_value2, ...
FROM DUAL
WHERE NOT EXISTS (
SELECT 1
FROM yourTable
WHERE unique_col = $row2_value1)
UNION ALL
SELECT $row3_value1, $row3_value2, ...
FROM DUAL
WHERE NOT EXISTS (
SELECT 1
FROM yourTable
WHERE unique_col = row1_value1)
UNION ALL
SELECT $row4_value1, $row4_value2, ...
FROM DUAL
WHERE NOT EXISTS (
SELECT 1
FROM yourTable
WHERE unique_col = $row4_value1)
I looked into MySQL duplicate key but cant figure it out.
I have a table like below:
id series chapter path(can be unique)
I want only insert data and not update. Lets say I have data like below:
seri:Naruto, klasor:567 ==> If both of these exist in table then do not insert.
seri:Naruto, klasor:568 ==> If Naruto exist but 568 does not exist then do insert.
How can I achieve this?
Easiest way would be to define unique index with two columns on that table:
ALTER TABLE yourtable ADD UNIQUE INDEX (seri,klasor);
You may also define two column primary key, which would work just as well.
Then use INSERT IGNORE to only add rows when they will not be duplicates:
INSERT IGNORE INTO yourtable (seri, klasor) VALUES ('Naruto',567);
INSERT IGNORE INTO yourtable (seri, klasor) VALUES ('Naruto',568);
Edit: As per comments, you can't use UNIQUE INDEX which complicates things.
SET #seri='Naruto';
SET #klasor=567;
INSERT INTO yourtable
SELECT seri,klasor FROM (SELECT #seri AS seri, #klasor AS klasor)
WHERE NOT EXISTS (SELECT seri, klasor FROM yourtable WHERE seri=#seri AND klasor=#klasor);
You may use the above query with two local variables or convert it to single statement by replacing the local variables with actual values.
Better way would be to use stored procedure:
CREATE PROCEDURE yourinsert (vseri VARCHAR(8), vklasor INT)
BEGIN
DECLARE i INT;
SELECT COUNT(*) INTO i FROM yourtable WHERE seri=vseri AND klasor=vklasor;
IF i=0 THEN
INSERT INTO yourtable (seri,klasor) VALUES (vseri, vklasor);
END IF;
END;
This would allow you to perform the INSERT using:
CALL yourinsert('Naruto',567);
INSERT INTO table_name (seri, klasor) VALUES ('Naruto',567)
WHERE NOT EXISTS( SELECT seri,klasor FROM table_name WEHERE seri='Naruto' AND klasor=567
)
Hope this helps..
Hey Stackoverflow community!
I currently use a temporary table to store the list of arrays are compare which ones don't exist within the table and insert those but I'm hoping there is a quicker more optimized way.
My goal is to get this down to one query
1.) I have an array of c_no INTs (1,2,3,4,5) I would like to add to a group_join table.
2.) In the groups_join I need to preform the following query to get all c_no values currently in the table.
select c_no from groups_join where g_no = (INT)
to see what INT's are in the groups
3.) I need someway to check if any of the c_no INTs I will be inserting are already in the groups_join table with same the g_no so I don't have duplicates.
I guess you are looking for the IN Keyword.
SELECT * FROM myTable WHERE field IN (myList)
If you want the g_no field to have unique values, why do you don't add the unique index ? MySQL Document - Create Index
I cannot comment due to rep, but in response to your comment on Didier's answer, instead of INSERT you can use INSERT IGNORE after you've created your unique index. When you use INSERT IGNORE MySQL will not insert the value, but will also suppress the error thrown due to attempting to insert a duplicate unique index. This will allow you to add your unique index without have to write code to handle the error.
INSERT [LOW_PRIORITY | DELAYED | HIGH_PRIORITY] [IGNORE]
[INTO] tbl_name [(col_name,...)]
{VALUES | VALUE} ({expr | DEFAULT},...),(...),...
[ ON DUPLICATE KEY UPDATE
col_name=expr
[, col_name=expr] ...
https://dev.mysql.com/doc/refman/5.5/en/insert.html
To insert array values that doesn't already exist in your groups_join table, you would need to use NOT EXISTS something like this:
INSERT INTO groups_join(c_no)
SELECT c_no
FROM yourTempTable yt
WHERE NOT EXISTS (
SELECT 1
FROM groups_join gj
WHERE yt.c_no = gj.c_no
);
Is there a way to use a MySQL INSERT similar to the following:
INSERT INTO doc_details SELECT * FROM doc_details WHERE dd_id = 1
This doesn't work because the primary key is being repeated and it can get very long-winded expanding the columns out.
The purpose of this is to duplicate rows in the same table which will get modified later, retrieving the last_insert_id for the new record. So ideas for other ways to do this would be appreciated too.
Thanks.
Simply name the columns you want to duplicate and omit the primary key:
INSERT INTO doc_details (col1, col2, col3)
SELECT col1, col2, col3
FROM doc_details
WHERE dd_id = 1
I'd suggest you to make ID field with AUTO_INCREMENT option, then use NULL values when inserting -
INSERT INTO doc_details(id, column1, column2)
SELECT NULL, column1, column2 FROM doc_details WHERE dd_id = 1;
In this case old ID will be changed with new ones.
You can depend on temporary table to copy from old record and omitting the key field value.
You have to use at least one named column, i.e. the key field name, to omit its repeating values.
See the following example:
CREATE TEMPORARY TABLE tmp SELECT * from doc_details WHERE dd_id = ?;
ALTER TABLE tmp drop pk_field_name_here; -- drop the key field for not repeating
INSERT INTO doc_details SELECT 0, tmp.* FROM tmp;
DROP TABLE tmp;
You can observe that no other filed names are used but the key field name to omit it's value.
You can also refer to my answer to a similar posting at: Mysql: Copy row but with new id.
Thanks for the answers. Really appreciated. Because most answers specify the column, this led to some extra research that said 'wildcards cannot be used in INSERT statements. Select, Modify and insert into the same table
I managed to solve this in my application with a separate SELECT then the INSERT with the columns expanded with a Perl map function:
SELECT * FROM doc_details WHERE dd_id = 1
Then in Perl, with the row as a hash reference in $data:
$data->{'dd_id'} = 0;$columns = join(',', map {$_ .'='. $dbh->quote( $data->{$_} ) } keys %{$cdh} );
Does the trick nicely - it copies the row regardless of changes to the column structure/order as long as the auto_increment column is maintained.
I know it's not a pure SQL solution - although Ravinder provided one that was.
Thanks to all!
In MySQL I am trying to copy a row with an autoincrement column ID=1 and insert the data into same table as a new row with column ID=2.
How can I do this in a single query?
Use INSERT ... SELECT:
insert into your_table (c1, c2, ...)
select c1, c2, ...
from your_table
where id = 1
where c1, c2, ... are all the columns except id. If you want to explicitly insert with an id of 2 then include that in your INSERT column list and your SELECT:
insert into your_table (id, c1, c2, ...)
select 2, c1, c2, ...
from your_table
where id = 1
You'll have to take care of a possible duplicate id of 2 in the second case of course.
IMO, the best seems to use sql statements only to copy that row, while at the same time only referencing the columns you must and want to change.
CREATE TEMPORARY TABLE temp_table ENGINE=MEMORY
SELECT * FROM your_table WHERE id=1;
UPDATE temp_table SET id=0; /* Update other values at will. */
INSERT INTO your_table SELECT * FROM temp_table;
DROP TABLE temp_table;
See also av8n.com - How to Clone an SQL Record
Benefits:
The SQL statements 2 mention only the fields that need to be changed during the cloning process. They do not know about – or care about – other fields. The other fields just go along for the ride, unchanged. This makes the SQL statements easier to write, easier to read, easier to maintain, and more extensible.
Only ordinary MySQL statements are used. No other tools or programming languages are required.
A fully-correct record is inserted in your_table in one atomic operation.
Say the table is user(id, user_name, user_email).
You can use this query:
INSERT INTO user (SELECT NULL,user_name, user_email FROM user WHERE id = 1)
This helped and it supports a BLOB/TEXT columns.
CREATE TEMPORARY TABLE temp_table
AS
SELECT * FROM source_table WHERE id=2;
UPDATE temp_table SET id=NULL WHERE id=2;
INSERT INTO source_table SELECT * FROM temp_table;
DROP TEMPORARY TABLE temp_table;
USE source_table;
For a quick, clean solution that doesn't require you to name columns, you can use a prepared statement as described here:
https://stackoverflow.com/a/23964285/292677
If you need a complex solution so you can do this often, you can use this procedure:
DELIMITER $$
CREATE PROCEDURE `duplicateRows`(_schemaName text, _tableName text, _whereClause text, _omitColumns text)
SQL SECURITY INVOKER
BEGIN
SELECT IF(TRIM(_omitColumns) <> '', CONCAT('id', ',', TRIM(_omitColumns)), 'id') INTO #omitColumns;
SELECT GROUP_CONCAT(COLUMN_NAME) FROM information_schema.columns
WHERE table_schema = _schemaName AND table_name = _tableName AND FIND_IN_SET(COLUMN_NAME,#omitColumns) = 0 ORDER BY ORDINAL_POSITION INTO #columns;
SET #sql = CONCAT('INSERT INTO ', _tableName, '(', #columns, ')',
'SELECT ', #columns,
' FROM ', _schemaName, '.', _tableName, ' ', _whereClause);
PREPARE stmt1 FROM #sql;
EXECUTE stmt1;
END
You can run it with:
CALL duplicateRows('database', 'table', 'WHERE condition = optional', 'omit_columns_optional');
Examples
duplicateRows('acl', 'users', 'WHERE id = 200'); -- will duplicate the row for the user with id 200
duplicateRows('acl', 'users', 'WHERE id = 200', 'created_ts'); -- same as above but will not copy the created_ts column value
duplicateRows('acl', 'users', 'WHERE id = 200', 'created_ts,updated_ts'); -- same as above but also omits the updated_ts column
duplicateRows('acl', 'users'); -- will duplicate all records in the table
DISCLAIMER: This solution is only for someone who will be repeatedly duplicating rows in many tables, often. It could be dangerous in the hands of a rogue user.
If you're able to use MySQL Workbench, you can do this by right-clicking the row and selecting 'Copy row', and then right-clicking the empty row and selecting 'Paste row', and then changing the ID, and then clicking 'Apply'.
Copy the row:
Paste the copied row into the blank row:
Change the ID:
Apply:
insert into MyTable(field1, field2, id_backup)
select field1, field2, uniqueId from MyTable where uniqueId = #Id;
A lot of great answers here. Below is a sample of the stored procedure that I wrote to accomplish this task for a Web App that I am developing:
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
SET NOCOUNT ON
-- Create Temporary Table
SELECT * INTO #tempTable FROM <YourTable> WHERE Id = Id
--To trigger the auto increment
UPDATE #tempTable SET Id = NULL
--Update new data row in #tempTable here!
--Insert duplicate row with modified data back into your table
INSERT INTO <YourTable> SELECT * FROM #tempTable
-- Drop Temporary Table
DROP TABLE #tempTable
You can also pass in '0' as the value for the column to auto-increment, the correct value will be used when the record is created. This is so much easier than temporary tables.
Source:
Copying rows in MySQL
(see the second comment, by TRiG, to the first solution, by Lore)
I tend to use a variation of what mu is too short posted:
INSERT INTO something_log
SELECT NULL, s.*
FROM something AS s
WHERE s.id = 1;
As long as the tables have identical fields (excepting the auto increment on the log table), then this works nicely.
Since I use stored procedures whenever possible (to make life easier on other programmers who aren't too familiar with databases), this solves the problem of having to go back and update procedures every time you add a new field to a table.
It also ensures that if you add new fields to a table they will start appearing in the log table immediately without having to update your database queries (unless of course you have some that set a field explicitly)
Warning: You will want to make sure to add any new fields to both tables at the same time so that the field order stays the same... otherwise you will start getting odd bugs. If you are the only one that writes database interfaces AND you are very careful then this works nicely. Otherwise, stick to naming all of your fields.
Note: On second thought, unless you are working on a solo project that you are sure won't have others working on it stick to listing all field names explicitly and update your log statements as your schema changes. This shortcut probably is not worth the long term headache it can cause... especially on a production system.
INSERT INTO `dbMyDataBase`.`tblMyTable`
(
`IdAutoincrement`,
`Column2`,
`Column3`,
`Column4`
)
SELECT
NULL,
`Column2`,
`Column3`,
'CustomValue' AS Column4
FROM `dbMyDataBase`.`tblMyTable`
WHERE `tblMyTable`.`Column2` = 'UniqueValueOfTheKey'
;
/* mySQL 5.6 */
Try this:
INSERT INTO test_table (SELECT null,txt FROM test_table)
Every time you run this query, This will insert all the rows again with new ids. values in your table and will increase exponentially.
I used a table with two columns i.e id and txt and id is auto increment.
I was looking for the same feature but I don't use MySQL. I wanted to copy ALL the fields except of course the primary key (id). This was a one shot query, not to be used in any script or code.
I found my way around with PL/SQL but I'm sure any other SQL IDE would do. I did a basic
SELECT *
FROM mytable
WHERE id=42;
Then export it to a SQL file where I could find the
INSERT INTO table (col1, col2, col3, ... , col42)
VALUES (1, 2, 3, ..., 42);
I just edited it and used it :
INSERT INTO table (col1, col2, col3, ... , col42)
VALUES (mysequence.nextval, 2, 3, ..., 42);
insert into your_table(col1,col2,col3) select col1+1,col2,col3 from your_table where col1=1;
Note:make sure that after increment the new value of col1 is not duplicate entry if col1 is primary key.
CREATE TEMPORARY TABLE IF NOT EXISTS `temp_table` LIKE source_table;
DELETE FROM `purchasing2` ;
INSERT INTO temp_table SELECT * FROM source_table where columnid = 2;
ALTER TABLE temp_table MODIFY id INT NOT NULL;
ALTER TABLE temp_table DROP PRIMARY KEY;
UPDATE temp_table SET id=NULL ;
INSERT INTO source_table SELECT * FROM temp_table;
DROP TEMPORARY TABLE IF EXISTS temp_table ;
Dump the row you want to sql and then use the generated SQL, less the ID column to import it back in.