I have a project in which I insert a lot if info to a table with an auto increment primary key per second and do that with multi-threading which means that there are many threads that tries to insert a new row to that table. Because there is a lock on the table for insert queries, I cant perform inserts concurrently and therefore I cant get maximum performance from the threads...
Is there a way to overcome this lock?
You can set the innodb_autoinc_lock_mode to 1 in the my.cnf. Then there is no LOCK for Auto_increment. Then it is possible that you have holes in the Values if one Thead rollback an see sample
MariaDB [test]> show variables like 'innodb_autoinc_lock_mode';
+--------------------------+-------+
| Variable_name | Value |
+--------------------------+-------+
| innodb_autoinc_lock_mode | 1 |
+--------------------------+-------+
1 row in set (0.00 sec)
sample 1
MariaDB [test]> start transaction;
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> insert into autoinc VALUES(NULL,'hello');
Query OK, 1 row affected (0.01 sec)
thread 2 ------------------> MariaDB [test]> start transaction;
thread 2 ------------------> Query OK, 0 rows affected (0.00 sec)
thread 2 ------------------> MariaDB [test]> insert into autoinc VALUES(NULL,'world');
thread 2 ------------------> Query OK, 1 row affected (0.00 sec)
thread 2 ------------------> MariaDB [test]> commit;
thread 2 ------------------> Query OK, 0 rows affected (0.00 sec)
thread 2 ------------------> MariaDB [test]>
MariaDB [test]> commit;
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> select * from autoinc;
+----+-------+
| id | d |
+----+-------+
| 1 | hello |
| 2 | world |
+----+-------+
2 rows in set (0.00 sec)
MariaDB [test]>
sample 2 with rollback
MariaDB [test]> start transaction;
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> insert into autoinc VALUES(NULL,'Guten');
Query OK, 1 row affected (0.00 sec)
thread 2 ------------------> MariaDB [test]> start transaction;
thread 2 ------------------> Query OK, 0 rows affected (0.00 sec)
thread 2 ------------------> MariaDB [test]> insert into autoinc VALUES(NULL,'Tag');
thread 2 ------------------> Query OK, 1 row affected (0.00 sec)
thread 2 ------------------> MariaDB [test]> commit;
thread 2 ------------------> Query OK, 0 rows affected (0.01 sec)
MariaDB [test]> rollback;
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> select * from autoinc;
+----+-------+
| id | d |
+----+-------+
| 1 | hello |
| 2 | world |
| 4 | Tag |
+----+-------+
3 rows in set (0.00 sec)
MariaDB [test]>
Related
I have two Tables (account2017) and (account2018) and I want to add a 2 at the end of the names in the name row for one of the tables, but only if both tables contain the same name. Adding the 2 for that name, which has lower points value.
Overall a solution to merge two tables with unique key (name), but to decide which row gets a 2 added at his end of name, should be followed by lower points.
For example if table account2017 and account2018 have both "Alex" in the name column, add a 2 at the end of the name (=Alex2) to that table, which has lower value in points column. As Alex in accounts2017 has 20 points, and Alex in accounts2018 has only 15 points, Alex name will be changed to Alex2 for accounts2018. Accounts2017 will be untouched.
Any idea how it can work?
If I'm understanding correctly, sounds like you need to use 2 separate update statements, using exists to match the criteria:
update account2017
set name = concat(name, '2')
where exists (
select 1
from account2018
where account2017.name = account2018.name and account2017.score < account2018.score)
update account2018
set name = concat(name, '2')
where exists (
select 1
from account2017
where account2018.name = account2017.name and account2018.score < account2017.score)
You will need to do this in 2 queries. Syntax will depend on what you are using (MySQL, SQL Server, SQLite) but here is the MySQL Version:
UPDATE accounts2017 table1 SET name = concat(name, '2') WHERE exists (SELECT 1 FROM accounts2018 table2 WHERE table1.name = table2.name AND table1.score < table2.score);
Then you can just flip the query around to update the 2018 table.
You can do this using a multi-table update. This relies on mysql recognising no change has been made to account2017 (proved by the fact that timestamp has not changed) but beware both before and after triggers fire.
MariaDB [sandbox]> drop table if exists account2017,account2018;
Query OK, 0 rows affected (0.39 sec)
MariaDB [sandbox]> create table account2017(name varchar(10), points int,ts timestamp DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP);
Query OK, 0 rows affected (0.20 sec)
MariaDB [sandbox]> create table account2018(name varchar(10), points int,ts timestamp DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP);
Query OK, 0 rows affected (0.29 sec)
MariaDB [sandbox]>
MariaDB [sandbox]> drop trigger if exists t;
Query OK, 0 rows affected, 1 warning (0.00 sec)
MariaDB [sandbox]> delimiter $$
MariaDB [sandbox]> create trigger t before update on account2017
-> for each row
-> begin
-> insert into debug_table(msg) values (concat('before:',old.name,':',new.name));
-> end $$
Query OK, 0 rows affected (0.07 sec)
MariaDB [sandbox]> delimiter ;
MariaDB [sandbox]>
MariaDB [sandbox]> drop trigger if exists t1;
Query OK, 0 rows affected, 1 warning (0.00 sec)
MariaDB [sandbox]> delimiter $$
MariaDB [sandbox]> create trigger t1 after update on account2017
-> for each row
-> begin
-> insert into debug_table(msg) values (concat('after:',old.name,':',new.name));
-> end $$
Query OK, 0 rows affected (0.08 sec)
MariaDB [sandbox]> delimiter ;
MariaDB [sandbox]>
MariaDB [sandbox]> insert into account2017 (name,points) values('alex',20);
Query OK, 1 row affected (0.02 sec)
MariaDB [sandbox]> insert into account2018 (name,points) values('alex',15);
Query OK, 1 row affected (0.01 sec)
MariaDB [sandbox]> truncate table debug_table;
Query OK, 0 rows affected (0.17 sec)
MariaDB [sandbox]>
MariaDB [sandbox]> select * from account2017;
+------+--------+---------------------+
| name | points | ts |
+------+--------+---------------------+
| alex | 20 | 2018-12-11 16:49:25 |
+------+--------+---------------------+
1 row in set (0.00 sec)
MariaDB [sandbox]> select * from account2018;
+------+--------+---------------------+
| name | points | ts |
+------+--------+---------------------+
| alex | 15 | 2018-12-11 16:49:25 |
+------+--------+---------------------+
1 row in set (0.00 sec)
MariaDB [sandbox]>
MariaDB [sandbox]> select sleep(60);
+-----------+
| sleep(60) |
+-----------+
| 0 |
+-----------+
1 row in set (1 min 0.00 sec)
MariaDB [sandbox]> update account2017 join account2018 on account2017.name = account2018.name
-> set account2017.name = case when account2017.points < account2018.points then concat(account2017.name,'2') else account2017.name end,
-> account2018.name = case when account2018.points < account2017.points then concat(account2018.name,'2') else account2018.name end
-> where 1 = 1;
Query OK, 1 row affected (0.04 sec)
Rows matched: 2 Changed: 1 Warnings: 0
MariaDB [sandbox]>
MariaDB [sandbox]>
MariaDB [sandbox]>
MariaDB [sandbox]> select * from account2017;
+------+--------+---------------------+
| name | points | ts |
+------+--------+---------------------+
| alex | 20 | 2018-12-11 16:49:25 |
+------+--------+---------------------+
1 row in set (0.00 sec)
MariaDB [sandbox]> select * from account2018;
+-------+--------+---------------------+
| name | points | ts |
+-------+--------+---------------------+
| alex2 | 15 | 2018-12-11 16:50:26 |
+-------+--------+---------------------+
1 row in set (0.00 sec)
MariaDB [sandbox]> select * from debug_table;
+----+------------------+------+
| id | msg | MSG2 |
+----+------------------+------+
| 1 | before:alex:alex | NULL |
| 2 | after:alex:alex | NULL |
+----+------------------+------+
2 rows in set (0.00 sec)
I have table_a with and auto_increment column named id and string column named name.
Running the statement:
INSERT INTO table_a(id, name)VALUES(DEFAULT, 'test');
Results to (MySQL):
+----+------+
| id | name |
+----+------|
| 1 | test |
+----+------+
Running the similar statement in MariaDB results to:
+----+------+
| id | name |
+----+------|
| 0 | test |
+----+------+
Other scenario:
I tried editing the AUTO_INCREMENT value of the table to 30. MySQL inserts 30 while MariaDB inserts 0.
What is the difference of DEFAULT value in INSERT statement of MySQL and MariaDB? Is this a bug in MariaDB or it is working as intended?
This behavior is controlled by SQL_MODE='NO_AUTO_VALUE_ON_ZERO', both in MySQL and MariaDB. If you observe the difference, it's most likely because you have different sql_mode on the instances.
MariaDB [test]> CREATE TABLE t (id BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY);
Query OK, 0 rows affected (0.20 sec)
MariaDB [test]> SET SQL_MODE='';
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> INSERT INTO t (id) VALUES (DEFAULT);
Query OK, 1 row affected (0.05 sec)
MariaDB [test]> SELECT * FROM t;
+----+
| id |
+----+
| 1 |
+----+
1 row in set (0.00 sec)
MariaDB [test]> DROP TABLE t;
Query OK, 0 rows affected (0.14 sec)
MariaDB [test]> CREATE TABLE t (id BIGINT(20) UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY);
Query OK, 0 rows affected (0.30 sec)
MariaDB [test]> SET SQL_MODE='NO_AUTO_VALUE_ON_ZERO';
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> INSERT INTO t (id) VALUES (DEFAULT);
Query OK, 1 row affected (0.03 sec)
MariaDB [test]> SELECT * FROM t;
+----+
| id |
+----+
| 0 |
+----+
1 row in set (0.00 sec)
MariaDB [(none)]> SET #good = 10;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> SHOW VARIABLES LIKE 'good';
Empty set (0.07 sec)
HOW show list all variables initialized by SET operator ?
For MySQL, use this:
select * from performance_schema.user_variables_by_thread;
MySQL Demo:
mysql> SET #good = 10;
Query OK, 0 rows affected (0.00 sec)
mysql> SET #bad = 0;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from performance_schema.user_variables_by_thread;
+-----------+---------------+----------------+
| THREAD_ID | VARIABLE_NAME | VARIABLE_VALUE |
+-----------+---------------+----------------+
| 36922 | bad | 0 |
| 36922 | good | 10 |
+-----------+---------------+----------------+
2 rows in set (0.00 sec)
For MariaDB, it is not supported so far:
https://mariadb.com/kb/en/mariadb/information-schema-user_variables-table/
The USER_VARIABLES table will be introduced in MariaDB 10.2.0 as part of the user_variables plugin.
WHy Flush Tables don't worK ? I can INSERT /SELECT into TABLE.
mysql> FLUSH TABLES;
Query OK, 0 rows affected (0.00 sec)
mysql> SELECT COUNT(*) FROM big_table;
+----------+
| COUNT(*) |
+----------+
| 1054155 |
+----------+
1 row in set (1.13 sec)
mysql> INSERT INTO exept VALUES(1);
Query OK, 1 row affected (0.02 sec)
I have all privileges.
When I use FLUSH TABLES WITH READ LOCK I can't insert but can select queries:
mysql> FLUSH TABLES WITH READ LOCK;
Query OK, 0 rows affected (0.00 sec)
mysql> INSERT INTO exept VALUES(1);
ERROR 1223 (HY000): Can't execute the query because you have a conflicting read lock
mysql> SELECT * FROM exept;
+----+
| id |
+----+
| 1 |
+----+
1 row in set (0.03 sec)
How disabled INSERT/SELECT queries ?
I opened two command windows to work with my Database (MySQL5).
Below is table structure I'm working with (It should be noted that I've turned off the auto commit by executing set autocommit=0;):
Table Structure:
CREATE TABLE `ajax`.`zipcodes` (
`ZIPCODE` varchar(5) NOT NULL,
`CITY` varchar(50) DEFAULT NULL,
`STATE` varchar(2) DEFAULT NULL,
PRIMARY KEY (`ZIPCODE`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Below is sequence of activities:
Step 1:
In command window 1, I executed below commands and you could also see the output:
mysql> insert into ajax.zipcodes values(5, 'Wil', 'AK');
Query OK, 1 row affected (0.00 sec)
Step 2
In second command window, I fired below command and it hangs (it seems waiting for commit command to be issues from previous window)
mysql> update ajax.zipcodes set city='Dublin' where zipcode=5;
Step 3
I went to Command window#1, and executed commit; you could see the output below:
mysql> commit;
Query OK, 0 rows affected (0.00 sec)
At the same time, i could see that second window that was earlier hanging, also executed the command and printed below output:
mysql> update ajax.zipcodes set city='Dublin' where zipcode=5;
Query OK, 1 row affected (3.63 sec)
Rows matched: 1 Changed: 1 Warnings: 0
Step 4
Now I issues commit in my second window to ensure that all the changes gets commited properly even the second session:
mysql> commit;
Query OK, 0 rows affected (0.00 sec)
Step 5
Now since commit has been issued from both windows I thought that everything is all right and the two session also must be in sync, so I went to 1st command window and issues below command:
mysql> select * from zipcodes where zipcode=5;
+---------+------+-------+
| ZIPCODE | CITY | STATE |
+---------+------+-------+
| 5 | Wil | AK |
+---------+------+-------+
1 row in set (0.00 sec)
I was surprised because I was expecting City value to be 'Dublin' because changes from second command window (i.e. update) has been commited in Step 4, but i'm still getting Wil in City column.
What am I doing wrong here?
This is to do with isolation levels. If you raise your isolation level to SERIALIZABLE (the default in MySQL is REPEATABLE READS) you won't get "phantom reads".
Isolation levels and phantom reads are described on the Wikipedia page for database transaction isolation.
If I run this through as you did, but with the higher isolation level, I get the result you were expecting.
Session 1
mysql> set autocommit=0;
Query OK, 0 rows affected (0.00 sec)
mysql> SET SESSION TRANSACTION ISOLATION LEVEL SERIALIZABLE;
Query OK, 0 rows affected (0.00 sec)
mysql> CREATE TABLE `ajax`.`zipcodes` (
-> `ZIPCODE` varchar(5) NOT NULL,
-> `CITY` varchar(50) DEFAULT NULL,
-> `STATE` varchar(2) DEFAULT NULL,
-> PRIMARY KEY (`ZIPCODE`)
-> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Query OK, 0 rows affected (0.07 sec)
mysql> insert into ajax.zipcodes values(5, 'Wil', 'AK');
Query OK, 1 row affected (0.00 sec)
Session 2
mysql> set autocommit=0;
Query OK, 0 rows affected (0.00 sec)
mysql> SET SESSION TRANSACTION ISOLATION LEVEL SERIALIZABLE;
Query OK, 0 rows affected (0.00 sec)
mysql> update ajax.zipcodes set city='Dublin' where zipcode=5;
Session 1
mysql> commit;
Query OK, 0 rows affected (0.04 sec)
Session 2
/* continued from previous (was frozen) */
Query OK, 1 row affected (7.54 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> select * from zipcodes;
+---------+--------+-------+
| ZIPCODE | CITY | STATE |
+---------+--------+-------+
| 5 | Dublin | AK |
+---------+--------+-------+
1 row in set (0.00 sec)
mysql> commit;
Query OK, 0 rows affected (0.04 sec)
mysql> select * from zipcodes;
+---------+--------+-------+
| ZIPCODE | CITY | STATE |
+---------+--------+-------+
| 5 | Dublin | AK |
+---------+--------+-------+
1 row in set (0.00 sec)
Session 1
mysql> select * from zipcodes;
+---------+--------+-------+
| ZIPCODE | CITY | STATE |
+---------+--------+-------+
| 5 | Dublin | AK |
+---------+--------+-------+
1 row in set (0.00 sec)
NB: This doesn't necessarily mean that you should always be using SERIALIZABLE - there are trade-offs. Most notable is that the database will acquire a range lock when executing a SELECT and you'll get more locking-based conflicts.
Update - Explicitly Handling Transactions
Since we have autocommit=0; set in these scripts, we really should handle the transactions explicitly, rather than expecting a START TRANSACTION - although in most cases, the database behaves as you'd expect if you'd executed START TRANSACTION.
However, run the original example while explicitly starting and ending all transactions (including those that are just SELECT, and you get a different result:
Session 1
mysql> set autocommit=0;
Query OK, 0 rows affected (0.00 sec)
mysql> SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ;
Query OK, 0 rows affected (0.00 sec)
mysql> CREATE TABLE `ajax`.`zipcodes` (
-> `ZIPCODE` varchar(5) NOT NULL,
-> `CITY` varchar(50) DEFAULT NULL,
-> `STATE` varchar(2) DEFAULT NULL,
-> PRIMARY KEY (`ZIPCODE`)
-> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
Query OK, 0 rows affected (0.07 sec)
mysql> START TRANSACTION;
Query OK, 0 rows affected (0.00 sec)
mysql> insert into ajax.zipcodes values(5, 'Wil', 'AK');
Query OK, 1 row affected (0.00 sec)
Session 2
mysql> set autocommit=0;
Query OK, 0 rows affected (0.00 sec)
mysql> SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ;
Query OK, 0 rows affected (0.00 sec)
mysql> START TRANSACTION;
Query OK, 0 rows affected (0.00 sec)
mysql> update ajax.zipcodes set city='Dublin' where zipcode=5;
Session 1
mysql> commit;
Query OK, 0 rows affected (0.04 sec)
Session 2
/* continued from previous (was frozen) */
Query OK, 1 row affected (8.32 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> select * from zipcodes;
+---------+--------+-------+
| ZIPCODE | CITY | STATE |
+---------+--------+-------+
| 5 | Dublin | AK |
+---------+--------+-------+
1 row in set (0.00 sec)
mysql> commit;
Query OK, 0 rows affected (0.04 sec)
mysql> START TRANSACTION;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from zipcodes;
+---------+--------+-------+
| ZIPCODE | CITY | STATE |
+---------+--------+-------+
| 5 | Dublin | AK |
+---------+--------+-------+
1 row in set (0.00 sec)
mysql> commit;
Query OK, 0 rows affected (0.04 sec)
Session 1
mysql> START TRANSACTION;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from zipcodes;
+---------+--------+-------+
| ZIPCODE | CITY | STATE |
+---------+--------+-------+
| 5 | Dublin | AK |
+---------+--------+-------+
1 row in set (0.00 sec)