Related
TL;DR (i.e. asking the question first):
Is there any way to write an INSERT INTO...SELECT FROM...GROUP BY...ON DUPLICATE KEY UPDATE statement using row alias(es) in the ON DUPLICATE KEY UPDATE clause instead of the col1 = VALUES(col1) syntax that has been deprecated and will be removed from future MySQL releases?
My searches of SO relating to this issue tend to all suggest using the deprecated VALUES() function, which is why I believe that my question is not a duplicate.
BACKGROUND (i.e. more info on how to reproduce the issue)
I have a table that comprises grouped records from another table. For simplicity in describing this issue, I've created two sample tables purely to illustrate:
items:
item_groups (below) was populated using the following SQL:
insert into item_groups (item_type,quantity) (select item_type, count(*) from items group by item_type order by item_type)
It also has a unique index on item_type:
Now, let's say that I add two more items to the items table, one with an item_type of 4 and one with a new item_type of 5. The quantity of item_type 4 in item_groups should be updated to 3 and a new row inserted for the item_type of 5 with quantity of 1.
Using the same INSERT statement I used above to initially populate the item_groups table, I now get an error, which is expected because of a duplicate key (4 of the 5 item_types currently in the items table are duplicates of the item_types that currently exist in the item_groups table):
Zero updates or inserts were completed due to this error. To remedy this, we would have historically used the ON DUPLICATE KEY UPDATE (occasionally abbreviated to ODKU below) clause like so including the VALUES() function:
insert into item_groups (item_type,quantity) (select item_type, count(*) from items group by item_type order by item_type) ON DUPLICATE KEY UPDATE quantity = VALUES(quantity);
The above INSERT...ON DUPLICATE KEY UPDATE statement with VALUES() DOES work (currently)...
However, I am also greeted with the following warning:
'VALUES function' is deprecated and will be removed in a future
release. Please use an alias (INSERT INTO ... VALUES (...) AS alias)
and replace VALUES(col) in the ON DUPLICATE KEY UPDATE clause with
alias.col instead
Now, I know how to write a simple INSERT...ODKU statement to be future-proof against the warning above (generically):
INSERT INTO `my_table` (col1,col2,col3) VALUES (1,2,3) AS new ON DUPLICATE KEY UPDATE col1 = new.col1, col2 = new.col2, col3 = new.col3
But let's insert more items into my items table and then use the above syntax for my more complicated INSERT...SELECT...ODKU statement into item_groups:
insert into item_groups (item_type,quantity) (select item_type, count(*) from items group by item_type order by item_type) AS new ON DUPLICATE KEY UPDATE quantity = new.quantity;
I get this error:
ERROR 1064 (42000): You have an error in your SQL syntax; check the
manual that corresponds to your MySQL server version for the right
syntax to use near 'AS new ON DUPLICATE KEY UPDATE quantity =
new.quantity' at line 1
Adding "VALUES" prior to my SELECT subquery, like so...
insert into item_groups (item_type,quantity) VALUES (select item_type, count(*) from items group by item_type order by item_type) AS new ON DUPLICATE KEY UPDATE quantity = new.quantity;
I now get a new syntax error:
ERROR 1064 (42000): You have an error in your SQL syntax; check the
manual that corresponds to your MySQL server version for the right
syntax to use near 'select item_type, count(*) from items group by
item_type order by item_type) AS ' at line 1
Finally, at my wit's end, I try adding another set of parentheses around the SELECT sub-query...
insert into item_groups (item_type,quantity) VALUES ((select item_type, count(*) from items group by item_type order by item_type)) AS new ON DUPLICATE KEY UPDATE quantity = new.quantity;
...and I still get an error:
ERROR 1136 (21S01): Column count doesn't match value count at row 1
This appears to be "progress" as I'm no longer getting syntax errors; however, I don't understand why the column count doesn't match the value count. My SELECT subquery pulls in 2 values for each row and the INSERT attempts to insert those into 2 columns for each row. So it would seem to me that 2 values -> 2 columns should not be an issue; yet it is.
CONCLUSION
I'm frankly not even sure what else to try, and I'm about ready to give up doing it this way and just write a simple SELECT, store those retrieved values in variables, and then use a simple INSERT to insert those values (wrapping everything in a transaction). However, if there is a way to do what I'm trying to do in one statement, I would appreciate anyone who can help me to do this.
From MySQL docs
Beginning with MySQL 8.0.20, an INSERT ... SELECT ... ON DUPLICATE KEY
UPDATE statement that uses VALUES() in the UPDATE clause, like this
one, throws a warning:
INSERT INTO t1 SELECT c, c+d FROM t2 ON DUPLICATE KEY UPDATE b =
VALUES(b); You can eliminate such warnings by using a subquery
instead, like this:
INSERT INTO t1 SELECT * FROM (SELECT c, c+d AS e FROM t2) AS dt ON
DUPLICATE KEY UPDATE b = e;
In simple words you could use a subquery as follows:
insert into item_groups (item_type,
quantity)
select * from ( select item_type , count(*) as new_quantity from items group by item_type ) as tbl
ON DUPLICATE KEY UPDATE quantity = new_quantity;
https://dbfiddle.uk/HoMLKMfd
You need a version mysql that is newer
8.0.30 and 8.0,31 this works
The use of VALUES() to refer to the new row and columns is deprecated beginning with MySQL 8.0.20, and is subject to removal in a future version of MySQL. Instead, use row and column aliases, as described in the next few paragraphs of this section.
so it shpuld work with 8.0.20 too
Besides security risks, with very update come new functions and old bugs are fixed.
Deploying for two Versions is bad, as you need more and more code to support more database version or to simulate functions you need, that you open your code to more and more bugs and insecurities.
So make a cut and use the latest Version
CREATE TABLE `my_table` (col1 int unique,col2 int ,col3 int)
INSERT INTO `my_table` (col1,col2,col3) VALUES (1,2,3) AS new
ON DUPLICATE KEY UPDATE col1 = new.col1, col2 = new.col2, col3 = new.col3
SELECT * FROM `my_table`
col1
col2
col3
1
2
3
fiddle
Clearly the following is incorrect.
INSERT INTO `aTable` (`A`,`B`) VALUES((SELECT MAX(`A`) FROM `aTable`)*2),'name');
I get the value:
SQL query:
INSERT INTO `aTable` (`A`, `B` )
VALUES
(
(
SELECT MAX(`A`)
FROM `aTable`
) *2
, 'name'
)
MySQL said:
1093 - You can't specify target table 'aTable' for update in FROM clause
So, I'm trying to make a bitmap table, each row corresponds to one Bit, and has a 'map' value.
To insert in the table, I don't want to do two queries, I want to do one.
How should I do this?
No one commented on this, but since I am trying to make a bitmap, it should be * 2 not ^ 2, my mistake, please note that is why the comments often say ^ 2, it was an error in the version that the commenters read.
try:
insert into aTable select max(a)^2, 'name' from aTable;
or
insert into aTable select max(a)^2, 'name' from aTable group by B;
If you need a join, you can do this:
insert into aTable select max(a)^2, 'name' from aTable, bTable;
My "Server version" is "5.0.51b-community-nt MySQL Community Edition (GPL)"
Actually, you can alias the table on the insert. I've seen this question all over the place, but no one seems to have tried that. Use a subquery to get the max from the table, but alias the table in the subquery.
INSERT INTO tableA SET fieldA = (SELECT max(x.fieldA) FROM tableA x)+1;
A more complex example, where you have a corresponding secondary key and might be inserting the FIRST record for the corresponding secondary key:
INSERT INTO tableA SET secondaryKey = 123, fieldA = COALESCE((SELECT max(x.fieldA) FROM tableA x WHERE x.secondaryKey = 123)+1,1);
By aliasing the table, it doesn't throw the error and seems to work. I just did this while coding something, although I can't see if there area any silly syntax errors above, I would try that type of syntax.
I take it that INSERT ... SELECT isn't working? I see this in the documentation for it:
The target table of the INSERT
statement may appear in the FROM
clause of the SELECT part of the
query. (This was not possible in some
older versions of MySQL.) In this
case, MySQL creates a temporary table
to hold the rows from the SELECT and
then inserts those rows into the
target table.
Out of curiosity, which version of MySQL are you using?
I think you need to drop the "VALUES", and have a valid select statement.
see this link
I'm not particularly a mySQL guy, I use MSSQL mostly. But If you format the select statement correctly, It should work.
as soon as the Select is correct you can do this.
I started by googling and found the article How to write INSERT if NOT EXISTS queries in standard SQL which talks about mutex tables.
I have a table with ~14 million records. If I want to add more data in the same format, is there a way to ensure the record I want to insert does not already exist without using a pair of queries (i.e., one query to check and one to insert is the result set is empty)?
Does a unique constraint on a field guarantee the insert will fail if it's already there?
It seems that with merely a constraint, when I issue the insert via PHP, the script croaks.
Use INSERT IGNORE INTO table.
There's also INSERT … ON DUPLICATE KEY UPDATE syntax, and you can find explanations in 13.2.6.2 INSERT ... ON DUPLICATE KEY UPDATE Statement.
Post from bogdan.org.ua according to Google's webcache:
18th October 2007
To start: as of the latest MySQL, syntax presented in the title is not
possible. But there are several very easy ways to accomplish what is
expected using existing functionality.
There are 3 possible solutions: using INSERT IGNORE, REPLACE, or
INSERT … ON DUPLICATE KEY UPDATE.
Imagine we have a table:
CREATE TABLE `transcripts` (
`ensembl_transcript_id` varchar(20) NOT NULL,
`transcript_chrom_start` int(10) unsigned NOT NULL,
`transcript_chrom_end` int(10) unsigned NOT NULL,
PRIMARY KEY (`ensembl_transcript_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Now imagine that we have an automatic pipeline importing transcripts
meta-data from Ensembl, and that due to various reasons the pipeline
might be broken at any step of execution. Thus, we need to ensure two
things:
repeated executions of the pipeline will not destroy our
> database
repeated executions will not die due to ‘duplicate
> primary key’ errors.
Method 1: using REPLACE
It’s very simple:
REPLACE INTO `transcripts`
SET `ensembl_transcript_id` = 'ENSORGT00000000001',
`transcript_chrom_start` = 12345,
`transcript_chrom_end` = 12678;
If the record exists, it will be overwritten; if it does not yet
exist, it will be created. However, using this method isn’t efficient
for our case: we do not need to overwrite existing records, it’s fine
just to skip them.
Method 2: using INSERT IGNORE Also very simple:
INSERT IGNORE INTO `transcripts`
SET `ensembl_transcript_id` = 'ENSORGT00000000001',
`transcript_chrom_start` = 12345,
`transcript_chrom_end` = 12678;
Here, if the ‘ensembl_transcript_id’ is already present in the
database, it will be silently skipped (ignored). (To be more precise,
here’s a quote from MySQL reference manual: “If you use the IGNORE
keyword, errors that occur while executing the INSERT statement are
treated as warnings instead. For example, without IGNORE, a row that
duplicates an existing UNIQUE index or PRIMARY KEY value in the table
causes a duplicate-key error and the statement is aborted.”.) If the
record doesn’t yet exist, it will be created.
This second method has several potential weaknesses, including
non-abortion of the query in case any other problem occurs (see the
manual). Thus it should be used if previously tested without the
IGNORE keyword.
Method 3: using INSERT … ON DUPLICATE KEY UPDATE:
Third option is to use INSERT … ON DUPLICATE KEY UPDATE
syntax, and in the UPDATE part just do nothing do some meaningless
(empty) operation, like calculating 0+0 (Geoffray suggests doing the
id=id assignment for the MySQL optimization engine to ignore this
operation). Advantage of this method is that it only ignores duplicate
key events, and still aborts on other errors.
As a final notice: this post was inspired by Xaprb. I’d also advise to
consult his other post on writing flexible SQL queries.
Solution:
INSERT INTO `table` (`value1`, `value2`)
SELECT 'stuff for value1', 'stuff for value2' FROM DUAL
WHERE NOT EXISTS (SELECT * FROM `table`
WHERE `value1`='stuff for value1' AND `value2`='stuff for value2' LIMIT 1)
Explanation:
The innermost query
SELECT * FROM `table`
WHERE `value1`='stuff for value1' AND `value2`='stuff for value2' LIMIT 1
used as the WHERE NOT EXISTS-condition detects if there already exists a row with the data to be inserted. After one row of this kind is found, the query may stop, hence the LIMIT 1 (micro-optimization, may be omitted).
The intermediate query
SELECT 'stuff for value1', 'stuff for value2' FROM DUAL
represents the values to be inserted. DUAL refers to a special one row, one column table present by default in all Oracle databases (see https://en.wikipedia.org/wiki/DUAL_table). On a MySQL-Server version 5.7.26 I got a valid query when omitting FROM DUAL, but older versions (like 5.5.60) seem to require the FROM information. By using WHERE NOT EXISTS the intermediate query returns an empty result set if the innermost query found matching data.
The outer query
INSERT INTO `table` (`value1`, `value2`)
inserts the data, if any is returned by the intermediate query.
In MySQL, ON DUPLICATE KEY UPDATE or INSERT IGNORE can be viable solutions.
An example of ON DUPLICATE KEY UPDATE update based on mysql.com:
INSERT INTO table (a,b,c) VALUES (1,2,3)
ON DUPLICATE KEY UPDATE c=c+1;
UPDATE table SET c=c+1 WHERE a=1;
An example of INSERT IGNORE based on mysql.com
INSERT [LOW_PRIORITY | DELAYED | HIGH_PRIORITY] [IGNORE]
[INTO] tbl_name [(col_name,...)]
{VALUES | VALUE} ({expr | DEFAULT},...),(...),...
[ ON DUPLICATE KEY UPDATE
col_name=expr
[, col_name=expr] ... ]
Or:
INSERT [LOW_PRIORITY | DELAYED | HIGH_PRIORITY] [IGNORE]
[INTO] tbl_name
SET col_name={expr | DEFAULT}, ...
[ ON DUPLICATE KEY UPDATE
col_name=expr
[, col_name=expr] ... ]
Or:
INSERT [LOW_PRIORITY | HIGH_PRIORITY] [IGNORE]
[INTO] tbl_name [(col_name,...)]
SELECT ...
[ ON DUPLICATE KEY UPDATE
col_name=expr
[, col_name=expr] ... ]
Any simple constraint should do the job, if an exception is acceptable. Examples:
primary key if not surrogate
unique constraint on a column
multi-column unique constraint
Sorry if this seems deceptively simple. I know it looks bad confronted to the link you share with us. ;-(
But I nevertheless give this answer, because it seems to fill your need. (If not, it may trigger you updating your requirements, which would be "a Good Thing"(TM) also).
If an insert would break the database unique constraint, an exception is throw at the database level, relayed by the driver. It will certainly stop your script, with a failure. It must be possible in PHP to address that case...
Try the following:
IF (SELECT COUNT(*) FROM beta WHERE name = 'John' > 0)
UPDATE alfa SET c1=(SELECT id FROM beta WHERE name = 'John')
ELSE
BEGIN
INSERT INTO beta (name) VALUES ('John')
INSERT INTO alfa (c1) VALUES (LAST_INSERT_ID())
END
REPLACE INTO `transcripts`
SET `ensembl_transcript_id` = 'ENSORGT00000000001',
`transcript_chrom_start` = 12345,
`transcript_chrom_end` = 12678;
If the record exists, it will be overwritten; if it does not yet exist, it will be created.
Here is a PHP function that will insert a row only if all the specified columns values don't already exist in the table.
If one of the columns differ, the row will be added.
If the table is empty, the row will be added.
If a row exists where all the specified columns have the specified values, the row won't be added.
function insert_unique($table, $vars)
{
if (count($vars)) {
$table = mysql_real_escape_string($table);
$vars = array_map('mysql_real_escape_string', $vars);
$req = "INSERT INTO `$table` (`". join('`, `', array_keys($vars)) ."`) ";
$req .= "SELECT '". join("', '", $vars) ."' FROM DUAL ";
$req .= "WHERE NOT EXISTS (SELECT 1 FROM `$table` WHERE ";
foreach ($vars AS $col => $val)
$req .= "`$col`='$val' AND ";
$req = substr($req, 0, -5) . ") LIMIT 1";
$res = mysql_query($req) OR die();
return mysql_insert_id();
}
return False;
}
Example usage:
<?php
insert_unique('mytable', array(
'mycolumn1' => 'myvalue1',
'mycolumn2' => 'myvalue2',
'mycolumn3' => 'myvalue3'
)
);
?>
There are several answers that cover how to solve this if you have a UNIQUE index that you can check against with ON DUPLICATE KEY or INSERT IGNORE. That is not always the case, and as UNIQUE has a length constraint (1000 bytes) you might not be able to change that. For example, I had to work with metadata in WordPress (wp_postmeta).
I finally solved it with two queries:
UPDATE wp_postmeta SET meta_value = ? WHERE meta_key = ? AND post_id = ?;
INSERT INTO wp_postmeta (post_id, meta_key, meta_value) SELECT DISTINCT ?, ?, ? FROM wp_postmeta WHERE NOT EXISTS(SELECT * FROM wp_postmeta WHERE meta_key = ? AND post_id = ?);
Query 1 is a regular UPDATE query without any effect when the data set in question is not there. Query 2 is an INSERT which depends on a NOT EXISTS, i.e. the INSERT is only executed when the data set doesn't exist.
Something worth noting is that INSERT IGNORE will still increment the primary key whether the statement was a success or not just like a normal INSERT would.
This will cause gaps in your primary keys that might make a programmer mentally unstable. Or if your application is poorly designed and depends on perfect incremental primary keys, it might become a headache.
Look into innodb_autoinc_lock_mode = 0 (server setting, and comes with a slight performance hit), or use a SELECT first to make sure your query will not fail (which also comes with a performance hit and extra code).
Update or insert without known primary key
If you already have a unique or primary key, the other answers with either INSERT INTO ... ON DUPLICATE KEY UPDATE ... or REPLACE INTO ... should work fine (note that replace into deletes if exists and then inserts - thus does not partially update existing values).
But if you have the values for some_column_id and some_type, the combination of which are known to be unique. And you want to update some_value if exists, or insert if not exists. And you want to do it in just one query (to avoid using a transaction). This might be a solution:
INSERT INTO my_table (id, some_column_id, some_type, some_value)
SELECT t.id, t.some_column_id, t.some_type, t.some_value
FROM (
SELECT id, some_column_id, some_type, some_value
FROM my_table
WHERE some_column_id = ? AND some_type = ?
UNION ALL
SELECT s.id, s.some_column_id, s.some_type, s.some_value
FROM (SELECT NULL AS id, ? AS some_column_id, ? AS some_type, ? AS some_value) AS s
) AS t
LIMIT 1
ON DUPLICATE KEY UPDATE
some_value = ?
Basically, the query executes this way (less complicated than it may look):
Select an existing row via the WHERE clause match.
Union that result with a potential new row (table s), where the column values are explicitly given (s.id is NULL, so it will generate a new auto-increment identifier).
If an existing row is found, then the potential new row from table s is discarded (due to LIMIT 1 on table t), and it will always trigger an ON DUPLICATE KEY which will UPDATE the some_value column.
If an existing row is not found, then the potential new row is inserted (as given by table s).
Note: Every table in a relational database should have at least a primary auto-increment id column. If you don't have this, add it, even when you don't need it at first sight. It is definitely needed for this "trick".
INSERT INTO table_name (columns) VALUES (values) ON CONFLICT (id) DO NOTHING;
I'm trying to insert new rows into a MySQL table, but only if one of the values that I'm inserting isn't in a row that's already in the table.
For example, if I'm doing:
insert into `mytable` (`id`, `name`) values (10, `Fred`)
I want to be able to check to see if any other row in the table already has name = 'Fred'. How can this be done?
Thanks!
EDIT
What I tried (can't post the exact statement, but here's a representation):
INSERT IGNORE INTO mytable (`domain`, `id`)
VALUES ('i.imgur.com', '12gfa')
WHERE '12gfa' not in (
select id from mytable
)
which throws the error:
#1064 - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'WHERE '12gfa' not in ( select id from mytable)' at line 3
First of all, your id field should be an autoincrement, unless it's a foreign key (but I can't assume it from the code you inserted in your question).
In this way you can be sure to have a unique value for id for each row.
If it's not the case, you should create a primary key for the table that includes ALL the fields you don't want to duplicate and use the INSERT IGNORE command.
Here's a good read about what you're trying to achieve.
You could use something like this
INSERT INTO someTable (someField, someOtherField)
VALUES ("someData", "someOtherData")
ON DUPLICATE KEY UPDATE someOtherField=VALUES("betterData");
This will insert a new row, unless a row already exists with a duplicate key, it will update it.
DELIMITER |
CREATE PROCEDURE tbl_update (IN id INT, IN nm VARCHAR(15))
BEGIN
DECLARE exst INT;
SELECT count(name) INTO exst FROM mytable WHERE name = nm;
IF exst = 0 THEN
INSERT INTO mytable VALUES(id, name);
END IF;
END
|
DELIMITER ;
or just make an attribute name as UNIQUE
Query:
INSERT INTO `metadata` (`group_id`, `key`, `value`)
VALUES ("19", "originality", "2")
ON DUPLICATE KEY UPDATE (`group_id` = `19`, `key`=`originality`, `value`=`2`)
The table:
group_id | key | value
----------------------------------------
group_id and key both have a UNIQUE index.
The error happens when I try to run the query when a row already exists with the id 19. The way I want the query to function is, if there is no row with that id, insert it and if there is update it instead of inserting a new row.
The error message I get is the typical:
I'm not sure if a ( should follow the UPDATE keyword - I think not. So try
ON DUPLICATE KEY UPDATE `group_id` = 19, `key`='originality', `value`=2
(or replace group_id with submission_group_id - your error message doesn't seem to match the original query)
you can only use ` on table columns and table names, not for data.
data should use ' or "
like:
ON DUPLICATE KEY UPDATE `group_id` = 19, `key`="originality", `value`=2
The quote tag must be the ' character not the ` character.
if there is no row with that id, insert it and if there is update it instead of inserting a new row.
If you want to do this you should try statement like:
IF EXISTS (SELECT * FROM sometable WHERE ColumnName='somevalue')
UPDATE sometable SET (...) WHERE ColumnName='somevalue'
ELSE
INSERT INTO Table1 VALUES (...)