I'm new to Titan. Sometimes I get
"getcom.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception"
OR
"com.thinkaurelius.titan.diskstorage.PermanentBackendException: Permanent failure in storage backend"
when executing a query command especially the number of edges is more than one million. But it does not always happen.
Does anybody know how to avoid it?
Related
Logs showing that from time to time this error is raised.
I'm reading the docs and it's very confusing because we're not locking any tables to do inserts and we have no transactions beyond individual SQL calls.
So - might this be happening because we're running out of the mySQL connection pool in Node? (We've set it to something like 250 simultaneous connections).
I'm trying to figure out how to replicate this but having no luck.
Every query not run within an explicit transaction runs in an implicit transaction that immediately commits when the query finishes or rolls back if an error occurs... so, yes, you're using transactions.
Deadlocks occur when at least two queries are in the process of acquiring locks, and each of them holds row-level locks that they happened to acquire in such an order that they each now need another lock that the other one holds -- so, they're "deadlocked." An infinite wait condition exists between the running queries. The server notices this.
The error is not so much a fault as it is the server saying, "I see what you did, there... and, you're welcome, I cleaned it up for you because otherwise, you would have waited forever."
What you aren't seeing is that there are two guilty parties -- two different queries that caused the problem -- but only one of them is punished. The query that has accomplished the least amount of work (admittedly, this concept is nebulous) will be killed with the deadlock error, and the other query happily proceeds along its path, having no idea that it was the lucky survivor.
This is why the deadlock error message ends with "try restarting transaction" -- which, if you aren't explicitly using transacrions, just means "run your query again."
See https://dev.mysql.com/doc/refman/5.6/en/innodb-deadlocks.html and examine the output of SHOW ENGINE INNODB STATUS;, which will show you the other query -- the one that helped cause the deadlock but that was not killed -- as well as the one that was.
I am trying to have batch process a number of insert queries by using addbatch.
During the execute, it throws out the exception at the first query which causes the exception
However the rest of batch will not be processed. What would be the best approach to handle this matter? If using rollback and isolate the error query before reprogress, it may repeat the same situation at the next error query. It will be very inefficient, particularly if the batch insert volume is huge.
Thanks for any advice.
Can you try to surround you first query with an try catch block and duck the exception. Although I suggest it is not a best coding practice. When you have list of insert statements to be executed always we need to take care that the whole got executed successfully or none of the operation got successeded.In addition as per your business scenario you have have a look at batch SKIP and RESTART ability if it works for you
I'm using JDBC with mysql. I have a pretty complex series of inserts and updates that I'm doing in a single transaction. This seems to work for the most part, but about 1% of the time I find that the data in one of my tables is in an inconsistent state.
I'm rolling back the transaction if an error occurs, but am not sure how to start debugging. My setup generally looks like:
try {
conn.setAutoCommit(false);
PreparedStatement stmt1 = conn.prepareStatement("insert into table1");
stmt1.executeUpdate();
stmt1.close();
PreparedStatement stmt2 = conn.prepareStatement("update table2");
stmt2.executeUpdate();
stmt2.close();
... more statements ...
conn.commit();
}
catch (Exception ex) {
conn.rollback();
}
I'm using a 2010 version of mysql. I might try updating that, but I have a feeling it's more something in my application code that's causing the inconsistency.
Are there any debugging tools I might find helpful at the database level to help? Any other pointers? I'm using JDBC with all default settings, I wonder if there is any stricter transaction level I need to use for this kind of scenario?
Thanks
----- Note -----
All my tables are InnoDb.
Hm.. Interesting. Yes, it should work. We used really huge transactions accross multiple tables many times, never experienced even strange things...
Are you sure it is not you who produce the inconsistency (whatever this means here, you didn't specify this)? By simply inserting/updating wrong things? :-)
Just an idea - we ran into this several times. Deadlock resolving. DB servers used to handle that. The chance a deadlock occurs is higher if you have several parallel threads and the transaction blocks are manipulating more tables. In this case some of your transactions could be aborted (and rolled back) by the DB server itself. And those transactions will result in an error.
The code you wrote above only rollbacks in the exception case (aborted transaction already rolled back, so it doesn't do too much..), but have you tried to print/log the exceptions? If not you should.
Of course transactions are running separated from each other. But this could explain why you experience this strange behaviour in only 1-2% of the cases...
You should check the logs of your mysql server too. It is also possible the server itself fails for any reason. And one more tip: you may try to run "mysqltop" (or "mtop", hope I remember the name of this tool correctly..). This is able to monitor and show you what happens inside the DB server. However it is mostly used to track the performance of our sqls, this also shows failures. Maybe running this could also help you out...
Perhaps you use DDL (create table, alter table, and so on) in your statements?
I am not sure about MySQL but it may not be able to roll back DDL statements.
For example:
PostgreSQL can rollback DDL,
Oracle performs commit before executing DDL.
See here: http://dev.mysql.com/doc/refman/5.0/en/cannot-roll-back.html
Once in a while I get following error in production enviornment which goes away on running the same stored procedure again.
Transaction (Process ID 86) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction
Someone told me that if I use NOLOCK hint in my stored procedures, it will ensure it will never be deadlocked. Is this correct? Are there any better ways of handling this error?
Occasional deadlocks on an RDBMS that locks like SQL Server/Sybase are expected.
You can code on the client to retry as recommended my MSDN "Handling Deadlocks".
Basically, examine the SQLException and maybe a half second later, try again.
Otherwise, you should review your code so that all access to tables are in the same order. Or you can use SET DEADLOCK_PRIORITY to control who becomes a victim.
On MSDN for SQL Server there is "Minimizing Deadlocks" which starts
Although deadlocks cannot be completely avoided
This also mentions "Use a Lower Isolation Level" which I don't like (same as many SQL types here on SO) and is your question. Don't do it is the answer... :-)
What can happen as a result of using (nolock) on every SELECT in SQL Server?
https://dba.stackexchange.com/q/2684/630
Note: MVCC type RDBMS (Oracle, Postgres) don't have this problem. See http://en.wikipedia.org/wiki/ACID#Locking_vs_multiversioning but MVCC has other issues.
While adding NOLOCK can prevent readers and writers from blocking each other (never mind all of the negative side effects it has), it is not a magical fix for deadlocks. Many deadlocks have nothing at all to do with reading data, so applying NOLOCK to your read queries might not cause anything to change at all. Have you run a trace and examined the deadlock graph to see exactly what the deadlock is? This should at least let you know which part of the code to look at. For example, is the stored procedure deadlocking because it is being called by multiple users concurrently, or is it deadlocking with a different piece of code?
Here is a good link on learning to troubleshoot deadlocks. I always try avoid using nolock for the reasons above. Also you might want to better understand Lock Compatibility.
In our applications we don't use either ADO.NET transaction or SQL Server transactions in procedures and now we are getting the below error in our website when multiple people are using.
Transaction (Process ID 73) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction
Is this error due to the lack of transactions? I thought the consistency will be handled by the DB itself.
And one thing I noticed that SQLCommand.Timeout property has set to 10000. Will this be an issue for the error?
I am trying to solve this issue ASAP. Please help.
EDIT
I saw the Isolationlevel property of ADO.NET transaction, so if I use ADO.NET transaction with proper isolationlevel property like "ReadUncommitted" during reading and "Serializable" during writing?
Every SQL DML (INSERT, UPDATE, DELETE) or DQL (SELECT) statement runs inside a transaction. The default behaviour for SQL Server is for it to open a new transaction (if one doesn't exist), and if the statement completes without errors, to automatically commit the transaction.
The IMPLICIT_TRANSACTIONS behaviour that Sidharth mentions basically gets SQL Server to change it's behaviour somewhat - it leaves the transaction open when the statement completes.
To get better information in the SQL Server error log, you can turn on a trace flag. This will then tell you which connections were involved in the deadlock (not just the one that got killed), and which resources were involved. You may then be able to determine what pattern of behaviour is leading to the deadlocks.
If you're unable to determine the underlying cause, you may have to add some additional code to your application - that catches sql errors due to deadlocks, and retries the command multiple times. This is usually the last resort - it's better to determine which tables/indexes are involved, and work out a strategy that avoids the deadlocks in the first place.
IsolationLevel is your best bet. Default serialization level of transactions is "Serializable" which is the most stringent and if at this level there is a circular reference chances of deadlock are very high. Set it to ReadCommitted while reading and let it be Serializable while writing.
Sql server can use implicit transactions which is what might be happening in your case. Try setting it off:
SET IMPLICIT_TRANSACTIONS OFF;
Read about it here: http://msdn.microsoft.com/en-us/library/ms190230.aspx