I am developing application that will run from multiple comuters. I want to lock mysql tables, so there won't be process concurrency issues, like one process is writing and other process is reading at the same time. Or what is even worse both process simultaneously writing (updating) different values. MySQL provides locks, but documentation says that we should avoid using locks with InnoDB. Read here. Please provide some advices what to do in this situation. Thanks everyone.
InnoDB is a transactional storage engine with full ACID support. One of the properties of InnoDB is that it handles the concurrent updates. How exactly depends on the Isolation level, but generally InnoDB disallow two transactions to modify the same row by locking the row. It does not lock the whole table so other records can be modified by other transactions.
If you set the isolation level to serializable the application will work as there is no concurrency at all, but still will allow some concurrency.
The higher the isolation level, the less concurrency you have, still you have more then if you lock the table.
Related
I have a webservice that produces a lot of updates in my database. After, it will do some other things (like calculus, call another webservice, etc). At the end, it contacts the database again.
The problem is that the tables are locked during the entire webservice life. So if the "other things" takes longer, I can´t work with the tables for this time.
There is a way to lock just the register, not the tables ?
How can I avoid this situation ?
I´m using Hibernate and MYSQL.
Pro JPA 2 book says:
The reality is that very few applications actually need pessimistic locking, and those > that do only need it for a limited subset of queries. The rule is that if you think
you need pessimistic locking, think again. If you are in a situation
where you have a very high degree of write concurrency on the same
object(s) and the occurrence of optimistic failures is high, then you
might need pessimistic locking because the cost of retries can become
so prohibitively expensive that you are better off locking
pessimistically . If you absolutely cannot retry your transactions and
are willing to sacrifice some amount of scalability for it, this also
might lead you to use pessimistic locking.
so I suggest you think again about your need.
i´m using PESSIMISTIC_WRITE
The hibernate acquires exclusive locks by using ‘SELECT … FOR UPDATE‘ statements(when using pessimistic lock)
4:Connection.TRANSACTION_REPEATABLE_READ lock your selected data(during the transaction). so you do not need to use pessimistic_lock.
the pessimistic lock is usually used for repeateabe_read while the transaction isolation is not repeatable_read( when is Read Committed)
following links describe mysql locking mechanism
https://dev.mysql.com/doc/refman/8.0/en/innodb-locking-reads.html
https://dev.mysql.com/doc/refman/8.0/en/innodb-transaction-isolation-levels.html#isolevel_repeatable-read
There is a way to lock just the register, not the tables?
the selected row should be locked not the tables( check your selects)
What transaction isolation level are you using? Please refer the documentation to see how this affects locking and how this can be changed.
Check your application. Transactions should be as short as possible. Consider a redesign, if needed. You might even consider using BASE instead of ACID.
I know there is one issue in MySQL with concurrent SELECT and INSERT. However, my question is if I open up two connections with MySQL and keep loading data using both of them, does MySQL takes data concurrently or waits for one to finish before loading another?
I’d like to know how MySQL behaves in both cases. Like when I am trying to load data in the same table or different tables concurrently when opening separate connections.
If you will create a new connection to the database and perform inserts from both the links, then from the database's perspective, it will still be sequential.
The documentation of Concurrent Inserts for MyISAM on the MySQL's documentation page says something like this:
If MyISAM storage is used and table has no holes, multiple INSERT statements are queued and performed in sequence, concurrently with the SELECT statements.
Mind that there is no control over the order in which two concurrent inserts will take place. The order in this concurrency is at the mercy of a lot of different factors. To ensure order, by default you will have to sacrifice concurrency.
MySQL does support parallel data inserts into the same table.
But approaches for concurrent read/write depends upon storage engine you use.
InnoDB
MySQL uses row-level locking for InnoDB tables to support simultaneous write access by multiple sessions, making them suitable for multi-user, highly concurrent, and OLTP applications.
MyISAM
MySQL uses table-level locking for MyISAM, MEMORY, and MERGE tables, allowing only one session to update those tables at a time, making them more suitable for read-only, read-mostly, or single-user applications
But, the above mentioned behavior of MyISAM tables can be altered by concurrent_insert system variable in order to achieve concurrent write. Kindly refer to this link for details.
Hence, as a matter of fact, MySQL does support concurrent insert for InnoDB and MyISAM storage engine.
You ask about Deadlock detection, ACID and particulary MVCC, locking and transactions:
Deadlock Detection and Rollback
InnoDB automatically detects transaction deadlocks and rolls back a
transaction or transactions to break the deadlock. InnoDB tries to
pick small transactions to roll back, where the size of a transaction
is determined by the number of rows inserted, updated, or deleted.
When InnoDB performs a complete rollback of a transaction, all locks
set by the transaction are released. However, if just a single SQL
statement is rolled back as a result of an error, some of the locks
set by the statement may be preserved. This happens because InnoDB
stores row locks in a format such that it cannot know afterward which
lock was set by which statement.
https://dev.mysql.com/doc/refman/5.6/en/innodb-deadlock-detection.html
Locking
The system of protecting a transaction from seeing or changing data
that is being queried or changed by other transactions. The locking
strategy must balance reliability and consistency of database
operations (the principles of the ACID philosophy) against the
performance needed for good concurrency. Fine-tuning the locking
strategy often involves choosing an isolation level and ensuring all
your database operations are safe and reliable for that isolation
level.
http://dev.mysql.com/doc/refman/5.5/en/glossary.html#glos_locking
ACID
An acronym standing for atomicity, consistency, isolation, and
durability. These properties are all desirable in a database system,
and are all closely tied to the notion of a transaction. The
transactional features of InnoDB adhere to the ACID principles.
Transactions are atomic units of work that can be committed or rolled
back. When a transaction makes multiple changes to the database,
either all the changes succeed when the transaction is committed, or
all the changes are undone when the transaction is rolled back. The
database remains in a consistent state at all times -- after each
commit or rollback, and while transactions are in progress. If related
data is being updated across multiple tables, queries see either all
old values or all new values, not a mix of old and new values.
Transactions are protected (isolated) from each other while they are
in progress; they cannot interfere with each other or see each other's
uncommitted data. This isolation is achieved through the locking
mechanism. Experienced users can adjust the isolation level, trading
off less protection in favor of increased performance and concurrency,
when they can be sure that the transactions really do not interfere
with each other.
http://dev.mysql.com/doc/refman/5.5/en/glossary.html#glos_acid
MVCC
InnoDB is a multiversion concurrency control (MVCC) storage engine
which means many versions of the single row can exist at the same
time. In fact there can be a huge amount of such row versions.
Depending on the isolation mode you have chosen, InnoDB might have to
keep all row versions going back to the earliest active read view, but
at the very least it will have to keep all versions going back to the
start of SELECT query which is currently running
https://www.percona.com/blog/2014/12/17/innodbs-multi-versioning-handling-can-be-achilles-heel/
It depends.
It depends on the client -- some clients allow concurrent access; some will serialize access, thereby losing the expected gain. You have not even specified PHP vs Java vs ... or Apache vs ... or Windows vs ... Many combinations simply do not provide any parallelism.
If different tables, there is only general contention for I/O, CPU, Mutexes on the buffer_pool, etc. A reasonable amount of parallelism is possible.
If same table, it depends on the indexes and access patterns. In some cases the threads will block each other. In some cases it will even "deadlock" and rollback one of the transactions. Deadlocks not only slow you down, but make you retry the inserts.
If you looking for high speed ingestion of a lot of rows, see my blog. It lays out techniques, and points out sever of the ramifications, such as replication, Engine choice, multi-threading.
Multiple threads inserting into the same tables -- It depend a lot on the values you are providing for any PRIMARY or UNIQUE keys. It depends on whether other actions are taken in the same transaction. It depends on how much I/O is involved. It depends on whether you are doing single-row inserts, or batching. It depends on ... (Sorry to be vague, but your question is not very specific.)
If you would like to present specifics on two or three designs, we can discuss the specifics.
I have a Django app where the default "REPEATABLE READ" transaction isolation level in InnoDB is causing different processes to have different views of the data than that current in the database.
e.g. Process 1 has made a change but Process 2 isn't seeing it.
I don't need transactional integrity in the app; can I just turn off transactions altogether so that all processes doing a SELECT see the same data?
Any downside to doing this?
Is this what is meant by "READ UNCOMMITTED"?
Any pointers welcome
Rachel
I'd suggest that you just convert the InnoDB tables to myISAM. If your criteria is speed, you are wasting alot of potential by using a transaction oriented table type (InnoDB) and just disabling transactions. You would gain alot if you just converted the tables to myISAM. It's designed with lack of transactions in mind, while still being able to lock changes (i.e. table locks).
A clean
ALTER TABLE table_name ENGINE = MyISAM;
can do the trick for a single table, dumping, changing type and loading the table does the trick as well.
Autocommit is on by default in InnoDB. Transactions are still used for updates (which is necessary), but they are committed immediately after each statement.
The READ UNCOMMITTED isolation level allows a transaction to read rows which have been written by other transactions but haven't yet been committed. This point is irrelevant however if you're not explicitly using transactions and autocommit is on.
Unfortunately, I'm not too familiar with Django, but from the documentation I see:
How to globally deactivate transaction management
Control freaks can totally disable all transaction management by setting
DISABLE_TRANSACTION_MANAGEMENT to True in the Django settings file.
Hope that helps.
I want to program a real-time application using MySQL.
It needs a small table (less than 10000 rows) that will be under heavy read (scan) and write (update and some insert/delete) load. I am really speaking of 10000 updates or selects per second. These statements will be executed on only a few (less than 10) open mysql connections.
The table is small and does not contain any data that needs to be stored on disk. So I ask which is faster: InnoDB or MEMORY (HEAP)?
My thoughts are:
Both engines will probably serve SELECTs directly from memory, as even InnoDB will cache the whole table. What about the UPDATEs? (innodb_flush_log_at_trx_commit?)
My main concern is the locking behavior: InnoDB row lock vs. MEMORY table lock. Will this present the bottleneck in the MEMORY implementation?
Thanks for your thoughts!
If you're really having to have that much concurrent updates, it's almost certain that innodb will perform better, as HEAP tables only have table-level locks, not row-level locks like Innodb.
If you're starting from scratch I would investigate using MySQL 5.5 or Percona's XtraDB as they both contain many scalability improvements over the stock MySQL 5.1.
It's not just a question of row locks - InnoDB also has MVCC http://en.wikipedia.org/wiki/Multiversion_concurrency_control so the readers won't even block writers.
But I think your question is missing the all important detail - what sort of data are you storing? If you need to be able to recover post-crash MEMORY is not an option.
If you don't need to recover post crash, then why are you using a database? Why not use something like memcached or redis?
I was wondering if anyone has a suggestion for what kind of storage engine to use. The programs needs to perform a lot of writes to database but very few reads.
[edit] No foreign keys necessary. The data is simple, but it needs to preform the writes very fast.
From jpipes:
MyISAM and Table-Level Locks
Unlike InnoDB, which employs row-level
locking, MyISAM uses a much
coarser-grained locking system to
ensure that data is written to the
data file in a protected manner.
Table-level locking is the only level
of lock for MyISAM, and this has a
couple consequences:
Any connection issuing an UPDATE or DELETE against a MyISAM table will
request an exclusive write lock on the
MyISAM table. If no other locks (read
or write) are currently placed on the
table, the exclusive write lock is
granted and all other connections
issuing requests of any kind (DDL,
SELECT, UPDATE, INSERT, DELETE) must
wait until the thread with the
exclusive write lock updates the
record(s) it needs to and then
releases the write lock.
Since there is only table-level locks, there is no ability (like there
is with InnoDB) to only lock one or a
small set of records, allowing other
threads to SELECT from other parts of
the table data.
The point is, for writing, InnoDB is better as it will lock less of the resource and enable more parallel actions/requests to occur.
"It needs to perform the writes very fast" is a vague requirement. Whatever you do, writes may be delayed by contention in the database. If your application needs to not block when it's writing audit records to the database, you should make the audit writing asynchronous and keep your own queue of audit data on disc or in memory (so you don't block the main worker thread/process)
InnoDB may allow concurrent inserts, but that doesn't mean they won't be blocked by contention for resources or internal locks for things like index pages.
MyISAM allows one inserter and several readers ("Concurrent inserts") under the following circumstances:
The table has no "holes in it"
There are no threads trying to do an UPDATE or DELETE
If you have an append-only table, which you recreate each day (or create a new partition every day if you use 5.1 partitioning), you may get away with this.
MyISAM concurrent inserts are mostly very good, IF you can use them.
When writing audit records, do several at a time if possible - this applies whichever storage engine you use. It is a good idea for the audit process to "batch up" records and do an insert of several at once.
You've not really given us enough information to make a considered suggestion - are you wanting to use foreign keys? Row-level locking? Page-level locking? Transactions?
As a general rule, if you want to use transactions, InnoDB/BerkeleyDB. If you don't, MyISAM.
In my experience, MyISAM is great for fast writes as long as, after insertion, it's read-only. It'll keep happily appending faster than any other option I'm familiar with (including supporting indexes).
But as soon as you start deleting records or updating index keys, and it needs to refill emptied holes (in tables or indexes) the discussion gets a lot more complicated.
For classic log-type or journal-type tables, though, it's very happy.