I want to insert or update, then insert a new log into another table.
I'm running a nifty little query to pull information from a staging table into other tables, something like
Insert into
select
on duplicate key update
What I'd like to do without php, or triggers (the lead dev doesn't like em, and I'm not that familiar with them either) is insert a new record into a logging table. Needed for reporting on what data was updated or inserted and on what table.
Any hints or examples?
Note: I was doing this with php just fine, although it was taking about 4 hours to process on 50K rows. Using the laravel php framework, looping over each entry in staging update 4 other tables with the data and log for each one was equalling 8 queries for each row (this was using laravel models not raw sql). I was able to optimise by pushing logs into an array and batch processing. But you can't beat 15sec processing time in mysql by bypassing all that throughput. Now I'm hooked on doing awesome things the sql way.
If you need executing more than one query statement. I refer to use transaction then trigger to guarantee Atomicity (part of ACID). Bellow code is sample for MySql transaction:
START TRANSACTION;
UPDATE ...
INSERT ...
DELETE ...
Other query statement
COMMIT;
Statements inside transaction will be executed all or nothing.
If you want to do two things (insert the base row and insert a log row), you'll need two statements. The second can (and should) be a trigger.
It would be better to use a Trigger, it is often used for Logging purposes
Related
I hope you can offer some words of wisdom on an issue i've been struggling with. I am using a MySQL 5.6 trigger to copy data during inserts into a separate table (not the one i'm inserting into).
I'm also modifying the data as it's being copied and need to compare rows within the insert to each over. Due to lack of support for "FOR EACH STATEMENT", i cannot act on the entire insert dataset while the transaction is still in progress. It appears i can only work on the current row as part of the supported FOR EACH ROW syntax.
Does anybody know a way to overcome this?
thanks!
UPDATE 18/01/18: #solarflare thanks for your answers, I looked into splitting the operation into an insert then a call to a stored procedure. It would work but it's not a path i want to go down as it breaks the atomicity of the operation. I tested the same code on PostgreSQL and it works fine.
It appears that when performing a bulk insert, an AFTER INSERT..FOR EACH ROW trigger in MySQL takes a snapshot of the table as it was before the bulk insert TXN started and allows you to query the snapshot but you cannot query the other rows of the insert (even if they have been inserted).
In postgresql this is not the case, as rows are inserted the trigger can see them, even if the transaction is not fully committed. Have you ever seen this / do you know is this a configurable param in MySQL or is it a design choice?
I have made one database. I normalized it to 3NF. I have 5 tables that have been normalized. Now I want to perform insert query at once on 5 tables. I am making a json service which I am going to implement in my Android Application.
Now my question is how will I execute 5 queries at once, in such a way that if one fails then other can not be get executed?
P.S. I am working with phpmyadmin. And using json.
Thanks a lot. :P
You need to wrap all inserts in a transaction. Basically:
BEGIN;
INSERT INTO ....
INSERT INTO ....
INSERT INTO ....
INSERT INTO ....
COMMIT;
If one of the inserts fails, all will fail.
Use Transaction,
Do a START TRANSACTION at start of the sequence and COMMIT at the end. If any error occurs do ROLLBACK.
I've came across the situation, where I need to select huge amount of data (say 100k records which look like ID | {"points":"9","votes":"2","breakdown":"0,0,0,1,1"}), process it in PHP and then put it back. Question is about putting it back efficiently. I saw a solution using INSERT ... ON DUPLICATE KEY UPDATE, I saw a solution with UPDATE using CASE. Are there any other solutions? Which would be the most efficient way to update huge data array?
Better choice is using simple update.
When you try to put data with insert exceptions your DB will do more additional work: try to insert, verify constraints, raise exception, update row, verify constraints again.
Update
Run tests on my local PC for insert into ... ON DUPLICATE KEY UPDATE and UPDATE statements against the table with 43k rows.
the first approach works on 40% faster.
But both worked faster then 1.5s. I suppose, you php code will be bottleneck of your approach and you should not worry about speed of MySQL statements. Of course, it works if you table not huge and does not had dozens millions rows.
Update 2
My local PC uses MySQL 5.6 in default configuration.
CPU: 8Gb
I'm trying to insert pages into a table with a sort column that I auto-increment by 2000 in this fashion:
INSERT INTO pages (sort,img_url,thumb_url,name,img_height,plank_id)
SELECT IFNULL(max(sort),0)+2000,'/image/path.jpg','/image/path.jpg','name',1600,'3'
FROM pages WHERE plank_id = '3'
The trouble is I trigger these inserts on the upload of images, so 5-10 of these queries are run almost simultaneously. This triggers a deadlock on some files, for some reason.
Any idea what is going on?
Edit: I'm running MySQL 5.5.24 and InnoDB. The sort column has an index.
What I made for myself is setting sort to 0 on insert, retrieve id of inserted row and set sort to id*2000. But, also, you can try to use transactions:
BEGIN;
INSERT INTO pages (sort,img_url,thumb_url,name,img_height,plank_id)
SELECT IFNULL(max(sort),0)+2000,'/image/path.jpg','/image/path.jpg','name',1600,'3'
FROM pages WHERE plank_id = '3';
COMMIT;
Note that not all of the MySQL client libraries support multiqueris, so you may have to execute them separately but in stream of one connection.
Another approach is to lock the whole table for the time INSERT is executed, but this will lead to increase of queries queue because they will have to wait until insert is performed
I was going through some code and noticed that UPDATE LOW_PRIORITY and INSERT DELAYED INTO are used for updating the database. What is is the use of these statements? Should I use these in every insert and update statement for various tables in the same database?
With the LOW_PRIORITY keyword, execution of the UPDATE is delayed until no other clients are reading from the table. Normally, reading clients are put on hold until the update query is done. If you want to give the reading clients priority over the update query, you should use LOW_PRIORITY.
The DELAYED option for the INSERT statement is a MySQL extension to standard SQL that is very useful if you have clients that cannot or need not wait for the INSERT to complete. This is a common situation when you use MySQL for logging and you also periodically run SELECT and UPDATE statements that take a long time to complete.
LOW_PRIORITY, HIGH_PRIORITY and DELAYED are only useful in a few circustamces. If you don't have a BIG load they can't help you. If you have, don't do anything you don't fully understand.
All of these otpions only work with MyISAM, not InnoDB, not views.
DELAYED doesn't work with partitioned tables, and it's clearly designed for dataware house. The client sends the insert and then forgets it, without waiting for the result. So you won't know if the insert succeded, if there were duplicate values, etc. It should never be used while other threads could SELECT from that table, because an insert delayed is never concurrent.
LOW_PRIORITY waits until NO client is accessing the table. But if you have a high traffic, you may wait until the connection times out... that's not what you want, I suppose :)
Also, note that DELAYED will be removed in Oracle MySQL 5.7 (but not in MariaDB).
If you need to use these, then you have a big load on your server, and you know that some UPDATE or INSERT statements are not high priority and they can act on load.
Example: SQL that generates some statistics or items top. They are slow, and do not need to be executed immediately.
If your UPDATEs on MySQL a read intensive environment are taking as much as 1800 seconds then it is advisable to use the UPDATE LOW_PRIORITY.