I need to implement concurrent-safe UPSERT using a non-unique key and avoid unnecessary auto-increment of ID.
Traditional INSERT ... ON DUPLICATE KEY doesn't work for me, so I'm performing:
INSERT INTO table (col1, col2, col3, col4, col5)
SELECT 1, 2, 'value3', 'value4', 'value5'
WHERE NOT EXISTS (SELECT 1
FROM table
WHERE col3 = 'value3'
AND col4 = 'value4'
AND col5 = 'value5')
then if it results in no row inserted, I'm performing:
UPDATE table
SET col1 = col1 + 1,
col2 = MAX(col2, 2)
WHERE col3 = 'value3'
AND col4 = 'value4'
AND col5 = 'value5'
There's an index:
CREATE INDEX ON table (col3, col4, col5)
It is non-unique as there are legacy data that does not allow me to declare it unique. Newer records, however, should not have duplicated (col3, col4, col5) rows.
Unsurprisingly, using the given INSERT statement I'm getting mixed results trying to execute it concurrently from two sessions. I can see the second session blocking until the first one commits its transaction, but then the second transaction is also able to insert a new row sometimes (or sometimes it achieves the expected of avoiding to insert a duplicate (col3, col4, col5) row).
I'm currently performing manual unique-check after the insert:
SELECT COUNT(1)
FROM table
WHERE col3 = 'value3'
AND col4 = 'value4'
AND col5 = 'value5'
but I've also tried:
INSERT INTO table (col1, col2, col3, col4, col5)
SELECT 1, 2, 'value3', 'value4', 'value5'
WHERE NOT EXISTS (SELECT 1
FROM table
WHERE col3 = 'value3'
AND col4 = 'value4'
AND col5 = 'value5'
FOR UPDATE)
which appears to work with the examples I'm always getting a duplicate (col3, col4, col5) row, otherwise. Is the given FOR UPDATE usage reliable for the purpose of ensuring no duplicate (col3, col4, col5) row will be inserted?
I'm using READ-COMMITTED transaction isolation.
MySQL 8.0.13 and higher supports functional key parts that index expression values rather than column or column prefix values. (link)
Because you have a unique field, i am assuming this is col, you can add an index like:
CREATE unique INDEX idx2 ON `mytable` ((col1>42),col3, col4, col5);
Where 42 should be the next auto-increment for col1.
Newly create records will be unique on the 3 columns, without affecting your 'old' data.
It is even possible to update the old data (as long as col1<=42).
Related
I have a table with lets say 4 fields.
table:
id(autoincremental), col1, col2, col3
There are several rows with data for id, col1 and col2.
Col3 is empty.
I would like to fill col3 with values, in the existing rows with one query like this:
INSERT INTO table(id, col3)
VALUES
(1, 'value1'),
(2, 'value2'),
(3, 'value3'),
...
ON DUPLICATE KEY UPDATE
id = VALUES(id),
col3 = VALUES(col3);
But I get an error because col1 and col2 don't have a default value.
I just want to update col3, and preserve other column values.
How can I do?
You can use a simple update statement :
Update yourtable t
Set t.col3 = case when t.id = 1 then ‘value1’
case when t.id = 2 then ‘value2’
.....
else t.col3 end;
And you can also filter the desired IDs for better performance
I believe that #Lukasz Szozda comment is the best approach for this issue. So I'll choose it as a solution. Thanks.
Mark col1, col2 as nullable.
You can update like this :
ON DUPLICATE KEY UPDATE
col3 = VALUES(col3);
This will keep the current values for the fields that are not in the update statement.
I read the question again and it seems like your insert would need col1 and col2 as well. You said that they don't have a default value, so I would either
add a default value for each
mark them as nullable
make these parameters mandatory in your application.
So you can't change your table structure my final query would be the following :
INSERT INTO table(id, col1, col2, col3) VALUES
(1, '', '', 'value1'),
(2, '', '', 'value2'),
(3, '', '', 'value3')
-- ...
ON DUPLICATE KEY UPDATE
col1 = col1,
col2 = col2,
col3 = VALUES(col3);
When a field does not have a default value and is not nullable, the insert query must include a value for that field. So your problem is not in the update part of your query, it is in the insert part.
I have a situation where I have a table (col1 (pk), col2, col3, col4) and a set of records which I need to insert into a table and on duplicate key update them. I would like to do a batch query to speed things up. However, col4 doesn't have a NOT NULL constraint. The problem arises when I want to update with records (val1, val2, val3, None), (val4, val5, val6, val7). For the first record, I don't want column 4 to be updated (If (val1, val2, val3, val8) existed in DB I wouldn't want to override val8 because None would signify lack of value as opposed to explicit setting to Null). However, for the second record, I would want to update col4 because an explicit value is passed. This would be fine with one record where I would just set the update columns to be col2, col3, and not col4, but I want to batch this query and would need to have col4update when a value is passed for it and not update when I don't have a value. I would logically need something like given below.
INSERT INTO table1
(col1, col2, col3, col4)
VALUES
('val1', 'val2', 'val3'), ON DUP KEY UPDATE col2, col3
('val5', 'val6', 'val7', 'val8'), ON DUP KEY UPDATE col2, col3, col4
('val9', 'val10', 'val11') ON DUP KEY UPDATE col2, col3
Clearly this can be done by just making it a series of separate statements, but I would like to find a way to batch this. Is there any way this, or a different method, can be done in sql?
In the on duplicate key update part of the insert, you can refer to the inserted values with values. You can use coalesce to preserve the pre-update value in case of null:
INSERT INTO YourTable (col1, col2, col3, col4) VALUES
('val1', 'val2', 'val3', null)
, ('val5', 'val6', 'val7', 'val8')
, ('val9', 'val10', 'val11', null)
ON DUPLICATE KEY UPDATE
col1 = values(col1)
, col2 = values(col2)
, col3 = values(col3)
, col4 = coalesce(values(col4), col4)
Example on SQL Fiddle.
In reply to your comment, you can set null explicitly with a case:
, col4 = case values(col4)
when 'None' then null
else coalesce(values(col4), col4)
end
The obvious risk here is that you can no longer update to None :)
The number of fields in VALUES must be the same as in INSERT. But you can just pass NULL for the col4 and use COALESCE in the UPDATE part.
INSERT INTO table1
(col1, col2, col3, col4)
VALUES
('val1', 'val2', 'val3', NULL),
('val5', 'val6', 'val7', 'val8'),
('val9', 'val10', 'val11', NULL)
ON DUPLICATE KEY UPDATE
col2 = VALUES(col2),
col3 = VALUES(col3),
col4 = COALESCE(VALUES(col4), col4)
Is this the thing that you are looking for?
INSERT INTO table1
(col1, col2, col3, col4)
VALUES
('val1', 'val2', 'val3', null)
('val5', 'val6', 'val7', 'val8')
('val9', 'val10', 'val11', null)
ON DUPLICATE KEY UPDATE
col2 = values(col2),
col3 = values(col3),
col4 = coalesce(values(col4), col4)
;
If you don't know what are the columns(or how many columns are there) in a table, how to check whether a particular column exists or not in that table using MySQL?
For example,
A table "DemoTable" contains columns Col1, Col2, Col3 with some data. And you write a select query like,
Select col1, col2, col3, col4 from DemoTable. But col4 does not exists in the DemoTable, so it will through an error. How to check whether col4 exists in the DemoTable in the select query or before the select query? Without a procedure.
There are two tables with same name in two DB's, in those tables there are few columns with same name, while others columns have different names. For example, DemoTable has columns col1, col2, col3, col4 in one DB and in other DB, DemoTable has columns col1, col2, col3, col5, col6. But the condition is, a single select query should be used to fetch data from the table without an error, so we need to check whether that columns first exists or not. Query like Select col1, col2, col5 from DemoTable from any DB.
I have two tables,
tblA(id, num, col1, col2, col3),
tblB(col1, col2, col3)
col1, col2 and col3 are the same in both tables. Now I have following sql:
declare #num...(same type as num)
insert into tblA
select #num, * from tblB
id in tblA is an indentity column.
But I got following error,
Column name or number of supplied values does not match table definition.
Can anyone help me to fix it?
Can you please try providing the column names as well,
declare #num...(same type as num)
insert into tblA(num, col1, col2, col3)
select #num, * from tblB
Please don't worry about identity column as it will get filled automatically.
Just INSERT using named columns, and skip the identity column - it will be filled automatically:
INSERT INTO tblA (num, col1, col2, col3) SELECT #Num, col1, col2, col3 FROM tblB
I think the error message is quite explicative: the SELECT and the INSERT has to have the same number of columns.
in your case
declare #num...(same type as num)
insert into tblA(num,col1, col2, col3)
select #num,col1, col2, col3 from tblB
if the key on tblA is not auto-generated you have to consider it in the INSERT
more info here
It simply based on your column name they should be of same type:
insert into tblA(col1,col2,col3)
select col1,col2,col3
from tblB
How can I at an insert query, check if a specific column has a specific value and then update the row. Otherwise it should insert a new row.
Somethings like this is what I want:
INSERT INTO table (col1, col2, col3) VALUES (val1, val2, val3) UPDATE IF col3 = $number
Create a UNIQUE key over your col3:
ALTER TABLE `table` ADD UNIQUE KEY (col3)
Then use INSERT ... ON DUPLICATE KEY UPDATE:
INSERT INTO `table` (col1, col2, col3) VALUES (val1, val2, val3)
ON DUPLICATE KEY UPDATE col1 = VALUES(col1), col2 = VALUES(col2)
Have a look at the REPLACE INTO command (MySQL docu).
REPLACE INTO table (col1, col2, col3) VALUES (val1, val2, val3);
It checks primary keys and unique indexes and when there is a matching entry already present, that row is replaced by the new one, else a new row is inserted.
EDIT
As #eggyal mentioned this the behaviour in the case, when the row is replaced, is actually a deletion of the old row and insertion of the new row afterwards. This may lead to some problems, when you're using triggers, foreign keys or alike.
Actually you might loose the content of some columns as well. Suppose you have a table with 3 columns (col1 to col3), but the REPLACE just sets two of them (col1, col2), the third one col3 will receive the default value specified and not retain the old value.
if you have unique index on that column then use this
INSERT INTO table (col1, col2, col3) VALUES (val1, val2, val3)
ON DUPLICATE KEY UPDATE col3 = $number