Setup:
Python script A inserts data to a database every 15 minutes
Python script B queries a few of the latest entries from the database every few minutes
Both use Django's ORM, run on the same machine and use a local MySQL database.
The Problem:
B fetches entries, except for the latest one, even though A saves it minutes before.
I suspected that A doesn't close the transaction, thus B sees the database without the last entry. Indeed when examining the MySQL logs, I noticed the commit for each INSERT happens right before the next INSERT.
Even though it's supposed to be redundant, I added #commit_on_success decorator to the A function that includes the save(), but it did not help.
How can I force Django (or MySQL?!) to commit right after the save()?
UPDATE:
I discovered that the commits DO happen - I was mislead to believe they don't because MySQL's General Query Log only has 1 sec resolution.
In light of this and other new information, I've reasked the question here.
You can use the commit_manually decorator and call it whenever you want.
Straight from the documentation:
from django.db import transaction
#transaction.commit_manually
def viewfunc(request):
...
# You can commit/rollback however and whenever you want
transaction.commit()
...
# But you've got to remember to do it yourself!
try:
...
except:
transaction.rollback()
else:
transaction.commit()
This answers the question you asked, though I wonder if there might be something else at work.
NOTE: commit_manually was deprecated in 1.6 and removed in 1.8.
The problem is caused by that MySQL by default has a REPEATABLE-READ transaction isolation level. That means that the same query is expected to return the same values. Changes won't return. You can do two things here:
Set transaction isolation level to READ-COMMITTED in the settings. As explained here.
Force a commit, thus close the transaction on script B, so as a new transaction starts, you will see all changes before this new transaction. Model.objects.update() does the trick.
Related
I'm using JMeter to do some load testing, and need to run a test which inserts a few thousand records. Part of the requirement is that it should not commit each individual update, but only commit every thousand records. Seems pretty straightforward, but I cannot find any examples or documentation explaining how to do a manual commit in JMeter, and I can't get it to work.
I've got it set up to read a few entries from a csv file, plus a counter variable for an incrementing ID field. If auto commit is enabled, that works fine.
When I disable auto commit, nothing gets committed -- as you'd expect. But how DO I get it committed? I can't even get it to commit every update without auto commit, let alone a specified rate.
I tried adding a JDBC post processor with a commit statement and/or
commit query type, on the same DB connection pool, only one thread on
that pool and in the test, but that doesn't work.
Tried doing the same as a separate JDBC request, that didn't work either.
Tried adding the commit to my update statement itself, no luck.
Tried changing transaction isolation, but none of those values seem to make
a difference.
Tried opening a transaction with a pre-processor or a
request or within the existing request, but that also hasn't helped.
I assume my problem might be that the commit is going into a separate transaction from the actual update statements...but I can't find any way to change that...
Try doing the following:
Add a JDBC Request with Query Type set to AutoCommit(false)
Put your JDBC Requests as you want
Add a JDBC Request with Query Type set to Commit
Your plan should look like this:
Fist set autocommit to false:
Then call your update queries:
Finally commit:
Reset autocommit to true:
If this does not work, I suggest you report an issue to JMeter.
I encounter in my project a need to implement transactions - thing that I never did before. I already checked that my aytocommit is set to 1 - and I'm not sure if I nneed to touch it at all?
Right now I have set of scripts that all does include function that connnects to database first. There is a perfect place to put mysqli_begin_transaction($link);, and mysqli_autocommit($link, FALSE);, so I'd have transactions everywhere regardless if specific script does need it, or not, and turn off autocommit - the documentation on php.net if very poor there but AFAIR I should do this. So my question no. 1 & 2 would be: Is it fine to start transaction everywhere regardless if script does need it or not? And should i disable autocommit like this as well?
Now let's say, that I have such script (sorry for not providing actual code, but my question is about how transaction works, not about code itself):
~insert and/or update things
~do something aka "line 2"
~insert and/or update things again
Seems like example taken right from the book. I obviously want all inserts and updates, or none to happen. Since I already started transaction, I assume, that nothing will commit, unless I call mysqli_commit ($link);. But here we have a little problem: I do not include any 'footer' at the end of my scripts and doing so seems like a nightmare now, so I don't have any place to put commit. So question no. 3 is: Will my queries commit automaticly after script ends (or I call exit; or die();) even if i set autocommit to false? Or do I need to call commit/do not turn off autocommit?
Now comes time for case when something fails and I need to rollback. Same as above - do I need to call mysqli_rollback (mysqli $link);, or pure fact that I did not call commit will be sufficient? I'm refering here to an situation where script does not end normally. Situations like power off server while working on "line 2", or was stopped because it took to much time (set_time_limit stopped it).
This is a somewhat broad question, so I'll try to cover all the things as much as I can.
At first you can ignore the mysqli api (the api specific transaction functions are just wrappers), and go straight to the MySQL manual. The important thing here is that disabling autocommit and starting a transaction are the same thing. Also a single query (including modifications by triggers) is always a transaction.
The answer you question 1 & 2 is "probably not". It very much depends on what your existing code assumes about the database connection, and how your application is structured.
From what you mentioned in the question, the answer would be: it will be better if you only put transactions in the places that need them.
For question 3: it will not commit automatically. You can however make it do so, by using register_shutdown_function, although I don't recommend doing that.
There are statements (implicit commits) which will commit the transaction automatically. These include all DDL statements (CREATE,ALTER...) and also TRUCNATE, LOCK TABLES and others. This basically means those statements can't be used in transactions.
MySQL rolls back transactions when the connection is terminated.
I would recommend to add transactions only to the code which needs them (to be safe you can do this for all code which does more than one write query to the db).
The classic approach is:
START TRANSACTION
query
other things
another query
some other stuff
3-rd query
...
COMMIT
The main thing here is to make sure you only commit if no errors have occurred.
Leave the rollback to either connection termination (or register_shutdown_function if you are using persistent connections), because making sure each and every script will have a correctly working rollback logic is hard :)
This will make sure that nothing is committed if bad things happen (exceptions, fatal errors, time/mem limits, power outages, meteors...).
It is also possible to have transactions at a function/method level (nested and stack-like), but thats out of the scope for this question.
I am having trouble finding an answer to this using google or Stack Overflow, so perhaps people familiar with Percona XtraDB can help answer this. I fully understand how unexpected deadlocks can occur as outlined in this article, and the solution is to make sure you wrap your transactions with retry logic so you can restart them if they fail. We already do that.
https://www.percona.com/blog/2012/08/17/percona-xtradb-cluster-multi-node-writing-and-unexpected-deadlocks/
My questions is about normal updates that occur outside of a transaction in auto commit mode. Normally if you are writing only to a single SQL DB and perform an update, you get a last in wins scenario so whoever executes the statement last, is golden. Any other data is lost so if two updates occur at the same time, one of them will take hold and the others data is essentially lost.
Now what happens in a multi master environment with the same thing? The difference in cluster mode with multi master is that the deadlock can occur at the point where the commit happens as opposed to when the lock is first taken on the table. So in auto commit mode, the data will get written to the DB but then it could fail when it tries to commit that to the other nodes in the cluster if something else modified the exact same record at the same time. Clearly the simply solution is to re-execute the update again and it would seem to me that the database itself should be able to handle this, since it is a single statement in auto commit mode?
So is that what happens in this scenario, or do I need to start wrapping all my update code in retry handling as well and retry it myself when this fails?
Autocommit is still a transaction; a single statement transaction. Your single statement is just wrapped up in BEGIN/COMMIT for you. I believe your logic is inverted. In PXC, the rule is "commit first wins". If you start a manual transaction on node1 (ie: autocommit=0; BEGIN;) and UPDATE id=1 and don't commit then on node2 you autocommit an update to the same row, that will succeed on node2 and succeed on node1. When you commit the manual UPDATE, you will get a deadlock error. This is correct behavior.
It doesn't matter if autocommit or not; whichever commits first wins and the other transaction must re-try. This is the reason why we don't recommend writing to multiple nodes in PXC.
Yes, if you want to write to multiple nodes, you need to adjust your code to "try-catch-retry" handle this error case.
I am working with MySQL 5.0 from python using the MySQLdb module.
Consider a simple function to load and return the contents of an entire database table:
def load_items(connection):
cursor = connection.cursor()
cursor.execute("SELECT * FROM MyTable")
return cursor.fetchall()
This query is intended to be a simple data load and not have any transactional behaviour beyond that single SELECT statement.
After this query is run, it may be some time before the same connection is used again to perform other tasks, though other connections can still be operating on the database in the mean time.
Should I be calling connection.commit() soon after the cursor.execute(...) call to ensure that the operation hasn't left an unfinished transaction on the connection?
There are thwo things you need to take into account:
the isolation level in effect
what kind of state you want to "see" in your transaction
The default isolation level in MySQL is REPEATABLE READ which means that if you run a SELECT twice inside a transaction you will see exactly the same data even if other transactions have committed changes.
Most of the time people expect to see committed changes when running the second select statement - which is the behaviour of the READ COMMITTED isolation level.
If you did not change the default level in MySQL and you do expect to see changes in the database if you run a SELECT twice in the same transaction - then you can't do it in the "same" transaction and you need to commit your first SELECT statement.
If you actually want to see a consistent state of the data in your transaction then you should not commit apparently.
then after several minutes, the first process carries out an operation which is transactional and attempts to commit. Would this commit fail?
That totally depends on your definition of "is transactional". Anything you do in a relational database "is transactional" (That's not entirely true for MySQL actually, but for the sake of argumentation you can assume this if you are only using InnoDB as your storage engine).
If that "first process" only selects data (i.e. a "read only transaction"), then of course the commit will work. If it tried to modify data that another transaction has already committed and you are running with REPEATABLE READ you probably get an error (after waiting until any locks have been released). I'm not 100% about MySQL's behaviour in that case.
You should really try this manually with two different sessions using your favorite SQL client to understand the behaviour. Do change your isolation level as well to see the effects of the different levels too.
We just switched our MySQL database from MyIsam to Innodb, and we are seeing an odd issue arise in Django. Whenever we make a database transaction, the existing sessions do not pick it up...ever. We can see the new record in the database from a mysql terminal, but the existing django sessions (ie a shell that was already open), would not register the change. For example:
Shell 1:
>>> my_obj = MyObj.objects.create(foo="bar")
>>> my_obj.pk
1
Shell 2 (was open before the above)
>>> my_obj = MyObj.objects.filter(pk=1)
[]
Shell 3 (MySQL):
mysql> select id from myapp_my_obj where id = 1;
id
1
Does anyone know why this might be happening?
EDIT: To clarify, Shell 2 was opened before Shell 1, then I make the create Shell 1, then I try to view the object that I created in Shell 2.
EDIT2: The big picture is that I have a celery task that is being passed the primary key from the object that is created. When I was using MyISAM, it found it every time, and now it throws ObjectDoesNotExist, even though I can see that the object is created in the database.
Your create() command commits the transaction for the current shell, but doesn't do anything to the transaction in the second shell.
https://docs.djangoproject.com/en/dev/topics/db/transactions/
Your second thread that can't see what's done in the first because it is in a transaction of its own. Transactions isolate the database so that when a transaction is committed, everything happens at a single point in time, including select statements. This is the A in ACID. Try running
from django.db import transaction; transaction.commit()
in the second shell. That should commit the current transaction and start a new one. You can also use transaction.rollback() to acheive the same thing if you haven't modified anything in the db in the current shell.
Edit Edit:
You may need to grab your specific db connection to make this work. Try this:
import django.db
django.db.connection._commit()
More information about this problem here:
http://groups.google.com/group/django-users/msg/55fa3724d2754013
The relevant bit is:
If you want script1.py (using an InnoDB table) to see committed updates from
other transactions you can change the transaction isolation level like so:
from django.db import connection
connection.cursor().execute('set transaction isolation level read
committed')
Alternatively you can enable the database's version of auto-commit, which
"commits" queries as well as updates, so that each new query by script1 will
be in its own transaction:
connection.cursor().execute('set autocommit=1')
Either one allows script1 to see script2's updates.
So, the tl;dr is that you need to set your InnoDB transaction isolation to READ-COMMITTED.