I have an SP with a set of 3 temp tables created within the SP with no indexes. All three are inserted into using INSERT INTO ... WITH (TABLOCK). The database recovery model is SIMPLE for userDB as well as tempDB.
This SP is generating and inserting new data and a Transaction Commit/Rollback is good enough to maintain data integrity. So I want it to do minimal logging which I think I have enabled by using the TABLOCK hint.
I am checking the log generation before and after execution of the SP using below query and see no difference in log generation after adding the TABLOCK hint. (Checking in tempDB as tables are temp tables)
SELECT count(1) as NumLogEntries
, sum("log record length") as TotalLengthWritten
FROM fn_dblog(null, null);
Is there anything else I need to do in order to enable minimal logging?
PN: I am able to see reduced logging if I use this hint to do the same INSERT INTO separately in management studio, but not if I do the same within the SP.
I have also tried adding the trace flag 610 ON before the insert statement but to no effect.
Related
I'm using an Aurora DB (ie MySQL version 5.6.10) as a queue, and I'm using a stored procedure to pull records out of a table in batches. The sproc works with the following steps...
Select the next batch of data into a temptable
Write the IDs from the records from the temp table into to a log table
Output the records
Once a record has been added to the log, the sproc won't select it again next time it's called, so multiple servers can call this sproc, and both deal with batches of data from the queue without stepping on each others toes.
The sproc runs in a fraction of a second, but my company is now spinning up servers automatically, and these cloned servers are calling the sproc at exactly the same time, and the result is the same records are being selected twice
Is there a way I can make this sproc be limited to one call at a time? Ideally, any additional calls should wait until the first call is finished, and then they can run
Unfortunately, I have very little experience working with MySQL, so I'm not really sure where to start. I'd much appreciate it if anyone could point me in the right direction
This is a job for MySQL table locking. Try something like this. (You didn't show us your queries so there's a lot of guesswork here.)
SET autocommit = 0;
LOCK TABLES logtable WRITE;
CREATE TEMPORARY TABLE temptable AS
SELECT whatever FROM whatevertable FOR UPDATE;
INSERT INTO logtable (id)
SELECT id FROM temptable;
COMMIT;
UNLOCK TABLES;
If more than one connection tries to run this sequence concurrently, one will wait for the other's UNLOCK TABLES; to proceed. You say your SP is quick, so probably nobody will notice the short wait.
Pro tip: When you have the same timed code running on lots of servers, it's best to put in a short random delay before running the job. That way the shared resources (like your MySQL database) won't get hammered by a whole lot of requests precisely timed to be simultaneous.
SO, we are trying to run a Report going to screen, which will not change any stored data.
However, it is complex, so needs to go through a couple of (TEMPORARY*) tables.
It pulls data from live tables, which are replicated.
The nasty bit when it comes to take the "eligible" records from
temp_PreCalc
and populate them from the live data to create the next (TEMPORARY*) table output
resulting in effectively:
INSERT INTO temp_PostCalc (...)
SELECT ...
FROM temp_PreCalc
JOIN live_Tab1 ON ...
JOIN live_Tab2 ON ...
JOIN live_Tab3 ON ...
The report is not a "definitive" answer, expectation is that is merely a "snapshot" report and will be out-of-date as soon as it appears on screen.
There is no order or reproducibility issue.
So Ideally, I would turn my TRANSACTION ISOLATION LEVEL down to READ COMMITTED...
However, I can't because live_Tab1,2,3 are replicated with BIN_LOG STATEMENT type...
The statement is lovely and quick - it takes hardly any time to run, so the resource load is now less than it used to be (which did separate selects and inserts) but it waits (as I understand it) because of the SELECT that waits for a repeatable/syncable lock on the live_Tab's so that any result could be replicated safely.
In fact it now takes more time because of that wait.
I'd like to SEE that performance benefit in response time!
Except the data is written to (TEMPORARY*) tables and then thrown away.
There are no live_ table destinations - only sources...
these tables are actually not TEMPORARY TABLES but dynamically created and thrown away InnoDB Tables, as the report Calculation requires Self-join and delete... but they are temporary
I now seem to be going around in circles finding an answer.
I don't have SUPER privilege and don't want it...
So can't SET BIN_LOG=0 for this connection session (Why is this a requirement?)
So...
If I have a scratch Database or table wildcard, which excludes all my temp_ "Temporary" tables from replication...
(I am awaiting for this change to go through at my host centre)
Will MySQL allow me to
SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED;
INSERT INTO temp_PostCalc (...)
SELECT ...
FROM temp_PreCalc
JOIN live_Tab1 ON ...
JOIN live_Tab2 ON ...
JOIN live_Tab3 ON ...
;
Or will I still get my
"Cannot Execute statement: impossible to write to binary log since
BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine
limited to row-based logging..."
Even though its not technically true?
I am expecting it to, as I presume that the replication will kick in simply because it sees the "INSERT" statement, and will do a simple check on any of the tables involved being replication eligible, even though none of the destinations are actually replication eligible....
or will it pleasantly surprise me?
I really can't face using an unpleasant solution like
SELECT TO OUTFILE
LOAD DATA INFILE
In fact I dont think I could even use that - how would I get unique filenames? How would I clean them up?
The reports are run on-demand directly by end users, and I only have MySQL interface access to the server.
or streaming it through the PHP client, just to separate the INSERT from the SELECT so that MySQL doesnt get upset about which tables are replication eligible....
So, it looks like the only way appears to be:
We create a second Schema "ScratchTemp"...
Set the dreaded replication --replicate-ignore-db=ScratchTemp
My "local" query code opens a new mysql connection, and performs a USE ScratchTemp;
Because I have selected the default database of the "ignore"d one - none of my queries will be replicated.
So I need to take huge care not to perform ANY real queries here
Reference my scratch_ tables and actual data tables by prefixing them all on my queries with the schema qualified name...
e.g.
INSERT INTO LiveSchema.temp_PostCalc (...) SELECT ... FROM LiveSchema.temp_PreCalc JOIN LiveSchema.live_Tab1 etc etc as above.
And then close this connection just as soon as I can, as it is frankly dangerous to have a non-replicated connection open....
Sigh...?
After consulting the MySQL manual [1,2], it seems like performing a rename of a table needs a global table lock. Some quick experiments confirm this. This means that if any SELECT statements are executing against this table, the RENAME operation will block until the SELECT statements finish executing.
I was wondering whether it is possible with InnoDB or any other database engine (PostgreSQL maybe?) where read-only operations do not block the database from performing a RENAME table.
I would imagine that the SELECT continues to read from the new table name if the two operations happen concurrently.
[1] https://dev.mysql.com/doc/refman/5.0/en/rename-table.html
[2] https://dev.mysql.com/doc/refman/5.0/en/select.html
I want to insert or update, then insert a new log into another table.
I'm running a nifty little query to pull information from a staging table into other tables, something like
Insert into
select
on duplicate key update
What I'd like to do without php, or triggers (the lead dev doesn't like em, and I'm not that familiar with them either) is insert a new record into a logging table. Needed for reporting on what data was updated or inserted and on what table.
Any hints or examples?
Note: I was doing this with php just fine, although it was taking about 4 hours to process on 50K rows. Using the laravel php framework, looping over each entry in staging update 4 other tables with the data and log for each one was equalling 8 queries for each row (this was using laravel models not raw sql). I was able to optimise by pushing logs into an array and batch processing. But you can't beat 15sec processing time in mysql by bypassing all that throughput. Now I'm hooked on doing awesome things the sql way.
If you need executing more than one query statement. I refer to use transaction then trigger to guarantee Atomicity (part of ACID). Bellow code is sample for MySql transaction:
START TRANSACTION;
UPDATE ...
INSERT ...
DELETE ...
Other query statement
COMMIT;
Statements inside transaction will be executed all or nothing.
If you want to do two things (insert the base row and insert a log row), you'll need two statements. The second can (and should) be a trigger.
It would be better to use a Trigger, it is often used for Logging purposes
To select information related to a list of hundreds of IDs... rather than make a huge select statement, I create temp table, insert the ids into it, join it with a table to select the rows matching the IDs, then delete the temp table. So this is essentially a read operation, with no permanent changes made to any persistent database tables.
I do this in a transaction, to ensure the temp table is deleted when I'm finished. My question is... what happens when I commit such a transaction vs. let it roll it back?
Performance-wise... does the DB engine have to do more work to roll back the transaction vs committing it? Is there even a difference since the only modifications are done to a temp table?
Related question here, but doesn't answer my specific case involving temp tables: Should I commit or rollback a read transaction?
EDIT (Clarification of Question):
Not looking for advice up to point of commit/rollback. Transaction is absolutely necessary. Assume no errors occur. Assume I have created a temp table, assume I know real "work" writing to tempdb has occurred, assume I perform read-only (select) operations in the transaction, and assume I issue a delete statement on the temp table. After all that... which is cheaper, commit or rollback, and why? What OTHER work might the db engine do at THAT POINT for a commit vs a rollback, based on this specific scenario involving temp-tables and otherwise read-only operations?
If we are talking about local temporary table (i.e. the name is prefixed with a single #), the moment you close your connection, SQL Server will kill the table. Thus, assuming your data layer is well designed to keep connections open as short a time as possible, I would not worry about wrapping the creation of temp tables in a transaction.
I suppose there could be a slight performance difference of wrapping the table in a transaction but I would bet it is so small as to be inconsequential compared to the cost of keeping a transaction open longer due to the time to create and populate the temp table.
A simpler way to insure that the temp table is deleted is to create it using the # sign.
CREATE TABLE #mytable (
rowID int,
rowName char(30) )
The # tells SQL Server that this table is a local temporary table. This table is only visible to this session of SQL Server. When the session is closed, the table will be automatically dropped. You can treat this table just like any other table with a few exceptions. The only real major one is that you can't have foreign key constraints on a temporary table. The others are covered in Books Online.
Temporary tables are created in tempdb.
If you do this, you won't have to wrap it in a transaction.