My app (which uses MySQL) is doing a large number of subsequent upserts. Right now my SQL looks like this:
INSERT IGNORE INTO customer (name,customer_number,social_security_number,phone) VALUES ('VICTOR H KINDELL','123','123','123')
INSERT IGNORE INTO customer (name,customer_number,social_security_number,phone) VALUES ('VICTOR H KINDELL','123','123','123')
INSERT IGNORE INTO customer (name,customer_number,social_security_number,phone) VALUES ('VICTOR H KINDELL OR','123','123','123')
INSERT IGNORE INTO customer (name,customer_number,social_security_number,phone) VALUES ('TRACY L WALTER PERSONAL REP FOR','123','123','123')
INSERT IGNORE INTO customer (name,customer_number,social_security_number,phone) VALUES ('TRACY L WALTER PERSONAL REP FOR','123','123','123')
So far I've found INSERT IGNORE to be the fastest way to achieve upserts. Selecting a record to see if it exists and then either updating it or inserting a new one is too slow. Even this is not as fast as I'd like because I need to do a separate statement for each record. Sometimes I'll have around 50,000 of these statements in a row.
Is there a way to take care of all of these in just one statement, without deleting any existing records?
You can put everything in 1 insert
INSERT IGNORE INTO table_1 (field1, field2) VALUES ('val1', 'val2'), ('val3', 'val4'), etc. You may also want to check INSERT ... ON DUPLICATE KEY UPDATE if you need to either updater or insert a record.
Related
i had query like this
CREATE TRIGGER `tambah_riwayatobat` AFTER INSERT ON `obat`
FOR EACH ROW insert into riwayat_obat(nama, keterangan, distributor,tanggal)
(select new.nama, 'Masuk', d.nama ,now()
From distributor d
join obat ON new.id_distributor = d.id_distributor)
i try to insert data with trigger and one of part data i fetch with constraint, but why the data be duplicate entry ?
Output :
example, if i try to insert data obat 1st time, data on tambah_riwayatobat insert 1 too
if i try to insert data obat 2nd time, data on tambah_riwayatobat insert 2 times with same data
if i try to insert data obat 3rd time, data on tambah_riwayatobat insert 3 times with same data
I'm not certain exactly what's happening, but it's a result of the join in your trigger code. You’re joining obat to distributor, but your join condition makes no mention of obat so you're getting some sort of cross-product where on the second and subsequent INSERT your SELECT subquery is selecting more than one row.
You shouldn't (and don't need to) use the join, since all the data you need from obat is already in the pseudorecord NEW. The following code should work much better:
CREATE TRIGGER `tambah_riwayatobat`
AFTER INSERT ON `obat`
FOR EACH ROW
INSERT INTO riwayat_obat
(nama, keterangan, distributor, tanggal)
(SELECT NEW.nama, 'Masuk', d.nama, now()
FROM distributor d
WHERE new.id_distributor = d.id_distributor
LIMIT 1);
The LIMIT clause will ensure that the SELECT selects only one row, so the INSERT inserts only one row; if distributor.id_distributor is a primary key the LIMIT clause is unnecessary.
I have a table in mysql. It has a few records from before. Its fields are a, b and c. Sometimes I insert a record in this table like for example this with different values:
$query = "INSERT INTO table(a, b, c) VALUES('1', '0', '2')";
Values are characters. Seems last record in the table is 5, 6, 4. I mean a=5, b=6 and c=4 and I want to insert a new record. My values are 1 and 0 and 2 but I want you to help me for this method:
When b == 0, I don't want to save it and instead, I want to save the last field in the table. For example I am inserting 1, 0, 2 and it just insert a=1 and c=2 but it inserts the last field in table instead this 0 that it is 6.
something like this:
if(false)
{
$query = "INSERT INTO table(a, b, c) VALUES('1', The Last Value In Table, '2')";
}
I would rather not to read last record of table and use its value because it can decrease the speed for me and speed is very important. Its better to use a mysql command that does it automatically.
As you need the data of the last inserted record you must get it from somewhere. The first thing that comes to mind is: read the table and look for the last inserted record, which is something you don't want, maybe because the table is too large to access the data quickly.
So you need a lookup table, containing only the last inserted values (i.e. one record):
create table last_insert_mytable(a varchar, b varchar, c varchar);
Which you fill with a trigger:
create trigger trg_mytable_last_insert after insert on mytable
for each row begin
delete from last_insert_mytable;
insert into last_insert_mytable (a, b, c) values (new.a, new.b, new.c);
end;
So your insert statement looks like this:
insert into mytable(a, b, c)
values ('1', (select b from last_insert_mytable), '2');
or (provided there is already a record in last_insert_mytable):
insert into mytable(a, b, c)
select '1', b, '2' from last_insert_mytable;
Keep in mind that every insert gets a bit slowed down due to the trigger. As there is only one record to deal with, it can be faster than to have to look up the last inserted record in mytable. This depends on the size of mytable. This happens for every insert into mytable. If it is rather seldom to have to look up the latest record, it may be better to have a slow lookup every now and then, than to have a slightly slower insert every time. Well, just try it.
I tried to insert from one table into another and im having with the redundancy..
I came up with a query but every time when i execute it, It cannot deals with duplicate.
here's my query...
INSERT INTO balik ( balik_date, balik_time, balik_cardID, balik_status,balik_type)
select current_date(), '00:00:00', L_CardID, 'BELUM BALIK', L_Type
FROM logdetail t1
LEFT JOIN balik t2 ON (t1.L_CardID = t2.balik_cardID)
WHERE t1.L_Type = 'IN'
any help will be greatly appreciated
Use INSERT IGNORE instant of INSERT.
Use INSERT IGNORE rather than INSERT. If a record doesn't duplicate an
existing record, MySQL inserts it as usual. If the record is a
duplicate, the IGNORE keyword tells MySQL to discard it silently
without generating an error.
OR
Check row count for unique field. If row exist don't insert or update.
OR
Use REPLACE rather than INSERT. If the record is new, it's inserted
just as with INSERT. If it's a duplicate, the new record replaces the
old one:
Source for definitions MySQL Handling Duplicates
I have three tables in my database. A users table, StoreA and StoreB
StoreA and StoreB both have a unique key which is the user ID value.
What I want is; When I create a user and insert them into the database, how can I Insert a row into the other two tables without too many additional queries.
I figure I can do this by inserting the user in one query,
then in another return the newly created user ID,
then in another, using said ID, create rows in StoreA and StoreB
Can I cut out the middle query?
Can I cut out the middle query?
YES
START TRANSACTION;
INSERT INTO user (id, name, other)
VALUES (null, 'John','rest of data');
INSERT INTO storeA (id, user_id, other)
VALUES (null, #user_id:= LAST_INSERT_ID(), 'rest of data');
INSERT INTO storeB (id, user_id, other)
VALUES (null, #user_id, 'rest of data');
COMMIT;
See: http://dev.mysql.com/doc/refman/5.0/en/getting-unique-id.html
It's a good idea to do this in a transaction, you you're not stuck with just a user with no other data if something goes wrong.
It's not a DB requirement though.
Yes - there should be a function available to get the last inserted ID (assuming it's an autoincrement field) without another query. In PHP, it's mysql_insert_id(). Just call that after the first query.
YES
Q1: insert into table1 values (...);
Q2: insert into table2 values (last_insert_id(), ...);
last_insert_id is the default mysql build-in function
Most of the mysql libraries in various programming language did support return last insert id.
But You did not mention what sort of language you are using to connect to mysql.,
so cannot provide any example
I just wanted to share a php solution.
If you're using mysqli, first execute your insert query.
Then do
$db_id = $this->db->insert_id;
Why don't you use their username as the primary key instead of creating an arbitrary user_id field thats auto incremented? Their user names are unique, right?
If I insert multiple records with a loop that executes a single record insert, the last insert id returned is, as expected, the last one. But if I do a multiple records insert statement:
INSERT INTO people (name,age)
VALUES ('William',25), ('Bart',15), ('Mary',12);
Let's say the three above are the first records inserted in the table. After the insert statement I expected the last insert id to return 3, but it returned 1. The first insert id for the statement in question.
So can someone please confirm if this is the normal behavior of LAST_INSERT_ID() in the context of multiple records INSERT statements. So I can base my code on it.
Yes. This behavior of last_insert_id() is documented in the MySQL docs:
Important
If you insert multiple rows using a single INSERT statement, LAST_INSERT_ID() returns the value generated for the first inserted row only. The reason for this is to make it possible to reproduce easily the same INSERT statement against some other server.
This behavior is mentioned on the man page for MySQL. It's in the comments but is not challenged, so I'm guessing it's the expected behavior.
I think it's possible if your table has unique autoincrement column (ID) and you don't require them to be returned by mysql itself. I would cost you 3 more DB requests and some processing. It would require these steps:
Get "Before MAX(ID)" right before your insert:
SELECT MAX(id) AS before_max_id FROM table_name`
Make multiple INSERT ... VALUES () query with your data and keep them:
INSERT INTO table_name
(col1, col2)
VALUES
("value1-1" , "value1-2"),
("value2-1" , "value2-2"),
("value3-1" , "value3-2"),
ON DUPLICATE KEY UPDATE
Get "After MAX(ID)" right after your insert:
SELECT MAX(id) AS after_max_id FROM table_name`
Get records with IDs between "Before MAX(ID)" and "After MAX(ID)" including:
SELECT * FROM table_name WHERE id>$before_max_id AND id<=$after_max_id`
Do a check of retrieved data with data you inserted to match them and remove any records that were not inserted by you. The remaining records have your IDs:
foreach ($after_collection as $after_item) {
foreach ($input_collection as $input_item) {
if ( $after_item->compare_content($input_item) ) {
$intersection_array[] = $after_item;
}
}
}
This is just how a common person would solve it in a real world, with parts of code. Thanks to autoincrement it should get smallest possible amount of records to check against, so they will not take lot of processing. This is not the final "copy & paste" code - eg. you have to create your own function compare_content() according you your needs.