I'm using JMeter to do some load testing, and need to run a test which inserts a few thousand records. Part of the requirement is that it should not commit each individual update, but only commit every thousand records. Seems pretty straightforward, but I cannot find any examples or documentation explaining how to do a manual commit in JMeter, and I can't get it to work.
I've got it set up to read a few entries from a csv file, plus a counter variable for an incrementing ID field. If auto commit is enabled, that works fine.
When I disable auto commit, nothing gets committed -- as you'd expect. But how DO I get it committed? I can't even get it to commit every update without auto commit, let alone a specified rate.
I tried adding a JDBC post processor with a commit statement and/or
commit query type, on the same DB connection pool, only one thread on
that pool and in the test, but that doesn't work.
Tried doing the same as a separate JDBC request, that didn't work either.
Tried adding the commit to my update statement itself, no luck.
Tried changing transaction isolation, but none of those values seem to make
a difference.
Tried opening a transaction with a pre-processor or a
request or within the existing request, but that also hasn't helped.
I assume my problem might be that the commit is going into a separate transaction from the actual update statements...but I can't find any way to change that...
Try doing the following:
Add a JDBC Request with Query Type set to AutoCommit(false)
Put your JDBC Requests as you want
Add a JDBC Request with Query Type set to Commit
Your plan should look like this:
Fist set autocommit to false:
Then call your update queries:
Finally commit:
Reset autocommit to true:
If this does not work, I suggest you report an issue to JMeter.
Related
I am having trouble finding an answer to this using google or Stack Overflow, so perhaps people familiar with Percona XtraDB can help answer this. I fully understand how unexpected deadlocks can occur as outlined in this article, and the solution is to make sure you wrap your transactions with retry logic so you can restart them if they fail. We already do that.
https://www.percona.com/blog/2012/08/17/percona-xtradb-cluster-multi-node-writing-and-unexpected-deadlocks/
My questions is about normal updates that occur outside of a transaction in auto commit mode. Normally if you are writing only to a single SQL DB and perform an update, you get a last in wins scenario so whoever executes the statement last, is golden. Any other data is lost so if two updates occur at the same time, one of them will take hold and the others data is essentially lost.
Now what happens in a multi master environment with the same thing? The difference in cluster mode with multi master is that the deadlock can occur at the point where the commit happens as opposed to when the lock is first taken on the table. So in auto commit mode, the data will get written to the DB but then it could fail when it tries to commit that to the other nodes in the cluster if something else modified the exact same record at the same time. Clearly the simply solution is to re-execute the update again and it would seem to me that the database itself should be able to handle this, since it is a single statement in auto commit mode?
So is that what happens in this scenario, or do I need to start wrapping all my update code in retry handling as well and retry it myself when this fails?
Autocommit is still a transaction; a single statement transaction. Your single statement is just wrapped up in BEGIN/COMMIT for you. I believe your logic is inverted. In PXC, the rule is "commit first wins". If you start a manual transaction on node1 (ie: autocommit=0; BEGIN;) and UPDATE id=1 and don't commit then on node2 you autocommit an update to the same row, that will succeed on node2 and succeed on node1. When you commit the manual UPDATE, you will get a deadlock error. This is correct behavior.
It doesn't matter if autocommit or not; whichever commits first wins and the other transaction must re-try. This is the reason why we don't recommend writing to multiple nodes in PXC.
Yes, if you want to write to multiple nodes, you need to adjust your code to "try-catch-retry" handle this error case.
I made a wrong update query in my table.
I forgot to make an id field in the WHERE clause.
So that updated all my rows.
How to recover that?
I didn't have a backup....
There are two lessons to be learned here:
Backup data
Perform UPDATE/DELETE statements within a transaction, so you can use ROLLBACK if things don't go as planned
Being aware of the transaction (autocommit, explicit and implicit) handling for your database can save you from having to restore data from a backup.
Transactions control data manipulation statement(s) to ensure they are atomic. Being "atomic" means the transaction either occurs, or it does not. The only way to signal the completion of the transaction to database is by using either a COMMIT or ROLLBACK statement (per ANSI-92, which sadly did not include syntax for creating/beginning a transaction so it is vendor specific). COMMIT applies the changes (if any) made within the transaction. ROLLBACK disregards whatever actions took place within the transaction - highly desirable when an UPDATE/DELETE statement does something unintended.
Typically individual DML (Insert, Update, Delete) statements are performed in an autocommit transaction - they are committed as soon as the statement successfully completes. Which means there's no opportunity to roll back the database to the state prior to the statement having been run in cases like yours. When something goes wrong, the only restoration option available is to reconstruct the data from a backup (providing one exists). In MySQL, autocommit is on by default for InnoDB - MyISAM doesn't support transactions. It can be disabled by using:
SET autocommit = 0
An explicit transaction is when statement(s) are wrapped within an explicitly defined transaction code block - for MySQL, that's START TRANSACTION. It also requires an explicitly made COMMIT or ROLLBACK statement at the end of the transaction. Nested transactions is beyond the scope of this topic.
Implicit transactions are slightly different from explicit ones. Implicit transactions do not require explicity defining a transaction. However, like explicit transactions they require a COMMIT or ROLLBACK statement to be supplied.
Conclusion
Explicit transactions are the most ideal solution - they require a statement, COMMIT or ROLLBACK, to finalize the transaction, and what is happening is clearly stated for others to read should there be a need. Implicit transactions are OK if working with the database interactively, but COMMIT statements should only be specified once results have been tested & thoroughly determined to be valid.
That means you should use:
SET autocommit = 0;
START TRANSACTION;
UPDATE ...;
...and only use COMMIT; when the results are correct.
That said, UPDATE and DELETE statements typically only return the number of rows affected, not specific details. Convert such statements into SELECT statements & review the results to ensure correctness prior to attempting the UPDATE/DELETE statement.
Addendum
DDL (Data Definition Language) statements are automatically committed - they do not require a COMMIT statement. IE: Table, index, stored procedure, database, and view creation or alteration statements.
Sorry man, but the chances of restoring an overwritten MySQL database are usually close to zero. Different from deleting a file, overwriting a record actually and physically overwrites the existing data in most cases.
To be prepared if anything comes up here, you should stop your MySQL server, and make a copy of the physical directory containing the database so nothing can get overwritten further: A simple copy+paste of the data folder to a different location should do.
But don't get your hopes up - I think there's nothing that can be done really.
You may want to set up a frequent database backup for the future. There are many solutions around; one of the simplest, most reliable and easiest to automate (using at or cron in Linux, or the task scheduler in Windows) is MySQL's own mysqldump.
Sorry to say that, but there is no way to restore the old field values without a backup.
Don't shoot the messenger...
Do you have binlogs enabled? You can recover by accessing the binlogs.
Setup:
Python script A inserts data to a database every 15 minutes
Python script B queries a few of the latest entries from the database every few minutes
Both use Django's ORM, run on the same machine and use a local MySQL database.
The Problem:
B fetches entries, except for the latest one, even though A saves it minutes before.
I suspected that A doesn't close the transaction, thus B sees the database without the last entry. Indeed when examining the MySQL logs, I noticed the commit for each INSERT happens right before the next INSERT.
Even though it's supposed to be redundant, I added #commit_on_success decorator to the A function that includes the save(), but it did not help.
How can I force Django (or MySQL?!) to commit right after the save()?
UPDATE:
I discovered that the commits DO happen - I was mislead to believe they don't because MySQL's General Query Log only has 1 sec resolution.
In light of this and other new information, I've reasked the question here.
You can use the commit_manually decorator and call it whenever you want.
Straight from the documentation:
from django.db import transaction
#transaction.commit_manually
def viewfunc(request):
...
# You can commit/rollback however and whenever you want
transaction.commit()
...
# But you've got to remember to do it yourself!
try:
...
except:
transaction.rollback()
else:
transaction.commit()
This answers the question you asked, though I wonder if there might be something else at work.
NOTE: commit_manually was deprecated in 1.6 and removed in 1.8.
The problem is caused by that MySQL by default has a REPEATABLE-READ transaction isolation level. That means that the same query is expected to return the same values. Changes won't return. You can do two things here:
Set transaction isolation level to READ-COMMITTED in the settings. As explained here.
Force a commit, thus close the transaction on script B, so as a new transaction starts, you will see all changes before this new transaction. Model.objects.update() does the trick.
I wrote a tool for our project, for applying sql update files that were committed, to the DB. Whenever run (on deployment), it calculates the list of update files which need to be applied, and applies them iniside a transaction.
Recently I became aware of an issue: mysql would implicitly commit a transaction, whenever DDL statements (like create) are executed. http://dev.mysql.com/doc/refman/5.0/en/implicit-commit.html
This is an issue for me, as sometimes an sql update file contains several statements, which as I understand will result in committing the transaction in the middle of executing the update file. This is a problem, because whenever a subsequent update will fail (which happens from time to time) I want to be able to rollback the transaction, or at least track which update files where applied (completely) and which were not.
Is there a way around the implicit transactions issue? I.e. is there a way to rollback a sequence of DDL statements whenever one of them fail?
Any other suggestions how I can handle the issue?
Thanks
Gidi
No. MySQL does not support transactional DDL. You either need to separate your DDL statements from DML statements, or perhaps try to use migration tool like RuckUsing
I made a wrong update query in my table.
I forgot to make an id field in the WHERE clause.
So that updated all my rows.
How to recover that?
I didn't have a backup....
There are two lessons to be learned here:
Backup data
Perform UPDATE/DELETE statements within a transaction, so you can use ROLLBACK if things don't go as planned
Being aware of the transaction (autocommit, explicit and implicit) handling for your database can save you from having to restore data from a backup.
Transactions control data manipulation statement(s) to ensure they are atomic. Being "atomic" means the transaction either occurs, or it does not. The only way to signal the completion of the transaction to database is by using either a COMMIT or ROLLBACK statement (per ANSI-92, which sadly did not include syntax for creating/beginning a transaction so it is vendor specific). COMMIT applies the changes (if any) made within the transaction. ROLLBACK disregards whatever actions took place within the transaction - highly desirable when an UPDATE/DELETE statement does something unintended.
Typically individual DML (Insert, Update, Delete) statements are performed in an autocommit transaction - they are committed as soon as the statement successfully completes. Which means there's no opportunity to roll back the database to the state prior to the statement having been run in cases like yours. When something goes wrong, the only restoration option available is to reconstruct the data from a backup (providing one exists). In MySQL, autocommit is on by default for InnoDB - MyISAM doesn't support transactions. It can be disabled by using:
SET autocommit = 0
An explicit transaction is when statement(s) are wrapped within an explicitly defined transaction code block - for MySQL, that's START TRANSACTION. It also requires an explicitly made COMMIT or ROLLBACK statement at the end of the transaction. Nested transactions is beyond the scope of this topic.
Implicit transactions are slightly different from explicit ones. Implicit transactions do not require explicity defining a transaction. However, like explicit transactions they require a COMMIT or ROLLBACK statement to be supplied.
Conclusion
Explicit transactions are the most ideal solution - they require a statement, COMMIT or ROLLBACK, to finalize the transaction, and what is happening is clearly stated for others to read should there be a need. Implicit transactions are OK if working with the database interactively, but COMMIT statements should only be specified once results have been tested & thoroughly determined to be valid.
That means you should use:
SET autocommit = 0;
START TRANSACTION;
UPDATE ...;
...and only use COMMIT; when the results are correct.
That said, UPDATE and DELETE statements typically only return the number of rows affected, not specific details. Convert such statements into SELECT statements & review the results to ensure correctness prior to attempting the UPDATE/DELETE statement.
Addendum
DDL (Data Definition Language) statements are automatically committed - they do not require a COMMIT statement. IE: Table, index, stored procedure, database, and view creation or alteration statements.
Sorry man, but the chances of restoring an overwritten MySQL database are usually close to zero. Different from deleting a file, overwriting a record actually and physically overwrites the existing data in most cases.
To be prepared if anything comes up here, you should stop your MySQL server, and make a copy of the physical directory containing the database so nothing can get overwritten further: A simple copy+paste of the data folder to a different location should do.
But don't get your hopes up - I think there's nothing that can be done really.
You may want to set up a frequent database backup for the future. There are many solutions around; one of the simplest, most reliable and easiest to automate (using at or cron in Linux, or the task scheduler in Windows) is MySQL's own mysqldump.
Sorry to say that, but there is no way to restore the old field values without a backup.
Don't shoot the messenger...
Do you have binlogs enabled? You can recover by accessing the binlogs.