Mysql optimize subquery in where clause - mysql

If I execute a query e.g
SELECT *
FROM `table1`
WHERE 1 OR EXISTS (
SELECT show
FROM `table2`
WHERE id IN(1,2,3)
)
I would expect the optimiser to realise that the where clause allways resolves to true and that it is therefore unnecessary to run the subquery. As running the query with EXPLAIN shows this is not the case and the subquery is executed anyway.
This is a narrowed down example of a more complex problem where I tryed to execute different subqueries bases on a column value of an outer query like:
SELECT value FROM table t
LEFT JOIN...
WHERE
(SELECT
IF(t.value = 1,
(SELECT ...),
(SELECT ...)
)
)
The intension was that only one of the inners subqueries in the where condition is executed, but the same happens here, both get executed but only the value of one is used. So the result is correct but useless queries are run. I have tryed with CASE WHEN as well same problem. Not sure if it is because I using MariaDB or something I'm missing here.

Sorry, i hope that i understand your question correctly. In the first query with WHERE 1 MySQL NEVER executes the EXISTS subquery. EPLAIN only show you whats possible to do. You can see it simple with SHOW status LIKE 'Handler_read_first'; I have 2 tables. if i test with WHERE 1 its only increment by 1 (1 Table read) and without it increments by 2
sample
the tables
MariaDB [test]> select * from index_usage;
+----+------+------+------+-------+
| id | a | b | c | dummy |
+----+------+------+------+-------+
| 11 | 1 | 1 | 1 | a |
+----+------+------+------+-------+
1 row in set (0.00 sec)
MariaDB [test]> select * from index_usage_copy;
+----+------+------+------+-------+
| id | a | b | c | dummy |
+----+------+------+------+-------+
| 99 | 1 | 1 | 1 | a |
+----+------+------+------+-------+
1 row in set (0.00 sec)
the counter
MariaDB [test]> SHOW status LIKE 'Handler_read_first';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| Handler_read_first | 59 |
+--------------------+-------+
1 row in set (0.00 sec)
first query (without WHERE 1) increment by 2
MariaDB [test]> SELECT * from index_usage WHERE EXISTS ( SELECT 1 FROM index_usage_copy where id in (1,2,99)); SHOW status LIKE 'Handler_read_first';
+----+------+------+------+-------+
| id | a | b | c | dummy |
+----+------+------+------+-------+
| 11 | 1 | 1 | 1 | a |
+----+------+------+------+-------+
1 row in set (0.00 sec)
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| Handler_read_first | 61 |
+--------------------+-------+
1 row in set (0.00 sec)
MariaDB [test]> SELECT * from index_usage WHERE EXISTS ( SELECT 1 FROM index_usage_copy where id in (1,2,99)); SHOW status LIKE 'Handler_read_first';
+----+------+------+------+-------+
| id | a | b | c | dummy |
+----+------+------+------+-------+
| 11 | 1 | 1 | 1 | a |
+----+------+------+------+-------+
1 row in set (0.00 sec)
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| Handler_read_first | 63 |
+--------------------+-------+
1 row in set (0.00 sec)
MariaDB [test]> SELECT * from index_usage WHERE EXISTS ( SELECT 1 FROM index_usage_copy where id in (1,2,99)); SHOW status LIKE 'Handler_read_first';
+----+------+------+------+-------+
| id | a | b | c | dummy |
+----+------+------+------+-------+
| 11 | 1 | 1 | 1 | a |
+----+------+------+------+-------+
1 row in set (0.00 sec)
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| Handler_read_first | 65 |
+--------------------+-------+
1 row in set (0.00 sec)
now with WHERE 1 - increments only by one
MariaDB [test]> SELECT * from index_usage WHERE 1 OR EXISTS ( SELECT 1 FROM index_usage_copy where id in (1,2,99)); SHOW status LIKE 'Handler_read_first';
+----+------+------+------+-------+
| id | a | b | c | dummy |
+----+------+------+------+-------+
| 11 | 1 | 1 | 1 | a |
+----+------+------+------+-------+
1 row in set (0.00 sec)
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| Handler_read_first | 66 |
+--------------------+-------+
1 row in set (0.00 sec)
MariaDB [test]> SELECT * from index_usage WHERE 1 OR EXISTS ( SELECT 1 FROM index_usage_copy where id in (1,2,99)); SHOW status LIKE 'Handler_read_first';
+----+------+------+------+-------+
| id | a | b | c | dummy |
+----+------+------+------+-------+
| 11 | 1 | 1 | 1 | a |
+----+------+------+------+-------+
1 row in set (0.00 sec)
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| Handler_read_first | 67 |
+--------------------+-------+
1 row in set (0.00 sec)
MariaDB [test]>

Also, EXPLAIN FORMAT=JSON says "optimized_away_subqueries" for 5.6 and 5.7. 8.0 and MariaDB 10.2 and 10.3 do not say that but seem to get rid of the subquery anyway.

Related

mysql: Reset counter after deleting rows

Dropping or truncating a table resets the counter(AUTO_INCREMENT), but deleting selected rows (with a WHERE clause) doesn’t reset the counter.
I want it to continue it from the point where rows were deleted.
Is there any way to reset the counter? Please follow the example below for better understanding.
mysql> create table dummy(id int NOT NULL AUTO_INCREMENT PRIMARY KEY);
Query OK, 0 rows affected (2.09 sec)
mysql> insert into dummy values (),(),(),(),(),();
Query OK, 6 rows affected (0.19 sec)
Records: 6 Duplicates: 0 Warnings: 0
mysql> select * from dummy;
+----+
| id |
+----+
| 1 |
| 2 |
| 3 |
| 4 |
| 5 |
| 6 |
+----+
6 rows in set (0.00 sec)
mysql> delete from dummy where id>4;
Query OK, 2 rows affected (0.23 sec)
mysql> select * from dummy;
+----+
| id |
+----+
| 1 |
| 2 |
| 3 |
| 4 |
+----+
4 rows in set (0.00 sec)
mysql> insert into dummy values (),(),(),(),(),();
Query OK, 6 rows affected (0.18 sec)
Records: 6 Duplicates: 0 Warnings: 0
mysql> select * from dummy;
+----+
| id |
+----+
| 1 |
| 2 |
| 3 |
| 4 |
| 7 |
| 8 |
| 9 |
| 10 |
| 11 |
| 12 |
+----+
10 rows in set (0.00 sec)
What I want:
+----+
| id |
+----+
| 1 |
| 2 |
| 3 |
| 4 |
| 5 |
| 6 |
| 7 |
| 8 |
| 9 |
| 10 |
+----+
We can query alter table dummy AUTO_INCREMENT=1; after delete statement;
So, as soon as we add new values to the table, it will check in the existing table, what value should be assign to the id starting from 0, and will assign the one which is exactly 1 greater than the max id of the table.
So, it will be like:
mysql> delete from dummy where id>4;
Query OK, 2 rows affected (0.14 sec)
mysql> select * from dummy;
+----+
| id |
+----+
| 1 |
| 2 |
| 3 |
| 4 |
+----+
4 rows in set (0.00 sec)
mysql> alter table dummy AUTO_INCREMENT=1;
Query OK, 0 rows affected (0.21 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> insert into dummy values (),(),(),(),(),();
Query OK, 6 rows affected (0.18 sec)
Records: 6 Duplicates: 0 Warnings: 0
And we will get in sequential form:
select * from dummy;
+----+
| id |
+----+
| 1 |
| 2 |
| 3 |
| 4 |
| 5 |
| 6 |
| 7 |
| 8 |
| 9 |
| 10 |
+----+
Thanks #jspcal

mysql sqlmode is non strict_mode. But it check not null. and return error

I've changed my.cnf. and check sql_mode use below command
select ##global.sql_mode;
and It says,
and Also I've tried
set global sql_mode='';
SET sql_mode = 'NO_ENGINE_SUBSTITUTION';
But... when I query to insert null values into not null column, It returns error...
How Can I turn off strict mode????????
Please help me.....
You cannot set a not null column to null. the effect of turning ofF sql mode is described https://dev.mysql.com/doc/refman/5.7/en/constraint-invalid-data.html,
If you are not using strict mode, then whenever you insert an “incorrect” value into a column, such as a NULL into a NOT NULL column or a too-large numeric value into a numeric column, MySQL sets the column to the “best possible value” instead of producing an error ... If you try to store NULL into a column that doesn't take NULL values, an error occurs for single-row INSERT statements. For multiple-row INSERT statements or for INSERT INTO ... SELECT statements, MySQL Server stores the implicit default value for the column data type. In general, this is 0 for numeric types, the empty string ('') for string types, and the “zero” value for date and time types
This is easily demonstrated:-
MariaDB [sandbox]> SET SESSION SQL_MODE = '';
Query OK, 0 rows affected (0.00 sec)
MariaDB [sandbox]> SELECT ##sESSION.SQL_MODE;
+--------------------+
| ##sESSION.SQL_MODE |
+--------------------+
| |
+--------------------+
1 row in set (0.00 sec)
MariaDB [sandbox]> select * from t;
+----+------------+---------+
| id | dt | meeting |
+----+------------+---------+
| 1 | 2017-12-20 | 1 |
| 2 | 2017-12-20 | 1 |
| 3 | 2017-12-20 | 1 |
| 4 | 2017-12-22 | 1 |
| 5 | 2017-12-25 | 1 |
+----+------------+---------+
5 rows in set (0.00 sec)
MariaDB [sandbox]> INSERT INTO T (DT,MEETING) VALUES ('2018-01-01',NULL);
ERROR 1048 (23000): Column 'meeting' cannot be null
MariaDB [sandbox]>
MariaDB [sandbox]>
MariaDB [sandbox]> INSERT INTO T (DT,MEETING) VALUES ('2018-01-01',NULL),('2018-01-02',NULL);
Query OK, 2 rows affected, 2 warnings (0.41 sec)
Records: 2 Duplicates: 0 Warnings: 2
MariaDB [sandbox]>
MariaDB [sandbox]> SELECT * FROM T;
+----+------------+---------+
| id | dt | meeting |
+----+------------+---------+
| 1 | 2017-12-20 | 1 |
| 2 | 2017-12-20 | 1 |
| 3 | 2017-12-20 | 1 |
| 4 | 2017-12-22 | 1 |
| 5 | 2017-12-25 | 1 |
| 13 | 2018-01-01 | 0 |
| 14 | 2018-01-02 | 0 |
+----+------------+---------+
7 rows in set (0.00 sec)
For single inserts the ignore extension
MariaDB [sandbox]> INSERT ignore INTO T (DT,MEETING) VALUES ('2018-01-10',NULL);
Query OK, 1 row affected, 1 warning (0.02 sec)
MariaDB [sandbox]> select * from t;
+----+------------+---------+
| id | dt | meeting |
+----+------------+---------+
| 1 | 2017-12-20 | 1 |
| 2 | 2017-12-20 | 1 |
| 3 | 2017-12-20 | 1 |
| 4 | 2017-12-22 | 1 |
| 5 | 2017-12-25 | 1 |
| 13 | 2018-01-01 | 0 |
| 14 | 2018-01-02 | 0 |
| 15 | 2018-01-10 | 0 |
+----+------------+---------+
8 rows in set (0.00 sec)
Allows the insert but stores the implicit value as described above.(this also works for multiple row inserts AND doesn't care about strict mode)
Warning turning off sql mode and using insert ignore are very blunt tools - and may be harmful to your db.
The problem here is that MySQL 5.1 and before allowed INSERT NULL values into NOT NULL fields and a lot of apps were written with this precept.
For DBAs, it is not easy to say to all the developers to upgrade MySQL when they have to change so much code.

a test about mysql innodb mvcc

I am learning InnoDB mvcc now and i have try a test show as follows:
Mysql version:
[root#mysql-test ~]# mysql --version
mysql Ver 15.1 Distrib 5.5.52-MariaDB, for Linux (x86_64) using readline 5.1
table schema:
MariaDB [liruifeng]> show create table test_a;
+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| test_a | CREATE TABLE `test_a` (
`id` int(11) NOT NULL DEFAULT '0',
`a` char(10) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
then init with data like this:
MariaDB [liruifeng]> select * from test_a;
+----+------+
| id | a |
+----+------+
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
+----+------+
3 rows in set (0.00 sec)
and at first i have open two session in different terminals, the test step show as bellows:
t1:
MariaDB [liruifeng]> start transaction;
Query OK, 0 rows affected (0.00 sec)
MariaDB [liruifeng]> select * from test_a;
+----+------+
| id | a |
+----+------+
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
+----+------+
3 rows in set (0.00 sec)
t2:
MariaDB [liruifeng]> insert into test_a values (4,4);
Query OK, 1 row affected (0.01 sec)
MariaDB [liruifeng]> select * from test_a;
+----+------+
| id | a |
+----+------+
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
| 4 | 4 |
+----+------+
4 rows in set (0.00 sec)
t1:
MariaDB [liruifeng]> select * from test_a;
+----+------+
| id | a |
+----+------+
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
+----+------+
3 rows in set (0.00 sec)
MariaDB [liruifeng]> update test_a set a = 444 where id = 4;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
MariaDB [liruifeng]> select * from test_a;
+----+------+
| id | a |
+----+------+
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
| 4 | 444 |
+----+------+
4 rows in set (0.00 sec)
it makes me puzzled that why a t1 can update the row insert by t2 before t1 has committed? my tx_isolation level is repeatable read and why will this update sql works?
my isolation show as bellows:
MariaDB [liruifeng]> show variables like 'tx_isolation';
+---------------+-----------------+
| Variable_name | Value |
+---------------+-----------------+
| tx_isolation | REPEATABLE-READ |
+---------------+-----------------+
1 row in set (0.00 sec)
thanks in advances :)
REPEATABLE-READ says that the select * from test_a; will say the same thing until t1 COMMITs. The fact that the UPDATE can see row 4 but the identical SELECT cannot is weird, but valid.
this is strange, in my experiment with auto-commit = false the update will block because of a record X lock on the inserted entry.

Can I define a lifetime for each row of MySQL's table?

I have a table like this:
// banned
+----+---------+---------------+
| id | user_id | unix_time |
+----+---------+---------------+
| 1 | 32534 | 1467066745524 |
| 2 | 43535 | 1467066745541 |
| 3 | 24352 | 1467066745618 |
| 4 | 88734 | 1467066746093 |
+----+---------+---------------+
Actually I need to define a expire time for each row when I insert it. Is that possible in MySQL? (I heard it is possible in Redis, well what about MySQL?)
So I want something like this:
// banned
+----+---------+---------------+
| id | user_id | unix_time |
+----+---------+---------------+
| 1 | 32534 | 1467066745524 | -- removing this row automatically in 10 min
| 2 | 43535 | 1467066745541 | -- removing this row automatically in 1 hour
| 3 | 24352 | 1467066745618 | -- removing this row automatically 2 day
| 4 | 88734 | 1467066746093 | -- removing this row automatically 8 hours min
+----+---------+---------------+
As you see, each row has a arbitrary lifetime.
you can create a EVENT that running every minute and delete old records like this:
enable scheduler
SET GLOBAL event_scheduler = ON;
create EVENT runs every minute
CREATE EVENT cleanup
ON SCHEDULE EVERY 1 MINUTE
DO
DELETE
FROM yourTable
WHERE unix_time < UNIX_TIMESTAMP();
Sample
MariaDB [yourschema]> SET GLOBAL event_scheduler = ON;
Query OK, 0 rows affected (0.00 sec)
MariaDB [yourschema]> SHOW GLOBAL VARIABLES LIKE 'event_scheduler';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| event_scheduler | ON |
+-----------------+-------+
1 row in set (0.00 sec)
MariaDB [yourschema]> SHOW PROCESSLIST;
+-----+-----------------+-----------------+------------+---------+------+-----------------------------+------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+-----+-----------------+-----------------+------------+---------+------+-----------------------------+------------------+----------+
| 28 | root | localhost:53255 | yourSchema | Sleep | 11 | | NULL | 0.000 |
| 29 | root | localhost:53256 | NULL | Sleep | 28 | | NULL | 0.000 |
| 34 | event_scheduler | localhost | NULL | Daemon | 6603 | Waiting for next activation | NULL | 0.000 |
| 316 | root | localhost | yourschema | Query | 0 | init | SHOW PROCESSLIST | 0.000 |
+-----+-----------------+-----------------+------------+---------+------+-----------------------------+------------------+----------+
4 rows in set (0.00 sec)
MariaDB [yourschema]>
MariaDB [yourschema]> SHOW events;
Empty set (0.01 sec)
MariaDB [yourschema]> DROP EVENT IF EXISTS cleanup;
Query OK, 0 rows affected, 1 warning (0.00 sec)
MariaDB [yourschema]> CREATE EVENT cleanup
-> ON SCHEDULE EVERY 5 SECOND ON COMPLETION PRESERVE ENABLE
-> DO
-> DELETE
-> FROM schedulerTable
-> WHERE expire_unix_time < UNIX_TIMESTAMP() LIMIT 10;
Query OK, 0 rows affected (0.00 sec)
MariaDB [yourschema]> INSERT INTO schedulerTable VALUES
-> (NULL,100,UNIX_TIMESTAMP(now() + INTERVAL 30 SECOND)),
-> (NULL,101,UNIX_TIMESTAMP(now() + INTERVAL 40 SECOND)),
-> (NULL,111,UNIX_TIMESTAMP(now() + INTERVAL 1 MINUTE));
Query OK, 3 rows affected (0.01 sec)
Records: 3 Duplicates: 0 Warnings: 0
MariaDB [yourschema]> SELECT * FROM schedulerTable;SELECT SLEEP(20);
+-------------+---------+------------------+
| id_rubrique | id_stat | expire_unix_time |
+-------------+---------+------------------+
| 1 | 100 | 1467074632 |
| 2 | 101 | 1467074642 |
| 3 | 111 | 1467074662 |
+-------------+---------+------------------+
3 rows in set (0.00 sec)
+-----------+
| SLEEP(20) |
+-----------+
| 0 |
+-----------+
1 row in set (20.00 sec)
MariaDB [yourschema]> SELECT * FROM schedulerTable;SELECT SLEEP(20);
+-------------+---------+------------------+
| id_rubrique | id_stat | expire_unix_time |
+-------------+---------+------------------+
| 2 | 101 | 1467074642 |
| 3 | 111 | 1467074662 |
+-------------+---------+------------------+
2 rows in set (0.00 sec)
+-----------+
| SLEEP(20) |
+-----------+
| 0 |
+-----------+
1 row in set (20.01 sec)
MariaDB [yourschema]> SELECT * FROM schedulerTable;SELECT SLEEP(20);
+-------------+---------+------------------+
| id_rubrique | id_stat | expire_unix_time |
+-------------+---------+------------------+
| 3 | 111 | 1467074662 |
+-------------+---------+------------------+
1 row in set (0.00 sec)
+-----------+
| SLEEP(20) |
+-----------+
| 0 |
+-----------+
1 row in set (20.01 sec)
MariaDB [yourschema]> SELECT * FROM schedulerTable;
Empty set (0.00 sec)
MariaDB [yourschema]>
You could create a view, something like this:
create view v_table as
select (case when unix_time < UNIX_TIMESTAMP() then id end) as id,
(case when unix_time < UNIX_TIMESTAMP() then user_id end) as user_id
from banned;
Or:
create view v_table as
select b.*
from banned b
where unixtime < UNIX_TIME();
This returns no information about a user after the timestamp. You can then schedule an event to periodically delete rows with expired information.

MySQL: "= true" vs "is true" on BOOLEAN. When is it advisable to use which one? And Which one is vendor independent?

MySQL provides 2 ways to check truth value of boolean columns, those are column_variable = true and column_variable is true. I created a table, inserted few values & tried a few select statements. Here are the results:
First I created this table:
mysql> create table bool_test (
-> id int unsigned not null auto_increment primary key,
-> flag boolean );
Query OK, 0 rows affected (0.13 sec)
Then I inserted 4 rows:
mysql> insert into bool_test(flag) values (true),(false),(9),(null);
mysql> select * from bool_test;
+----+------+
| id | flag |
+----+------+
| 1 | 1 |
| 2 | 0 |
| 3 | 9 |
| 4 | NULL |
Here are all the select queries I fired on this table:
mysql> select * from bool_test where flag;
+----+------+
| id | flag |
+----+------+
| 1 | 1 |
| 3 | 9 |
+----+------+
2 rows in set (0.49 sec)
mysql> select * from bool_test where flag = true;
+----+------+
| id | flag |
+----+------+
| 1 | 1 |
+----+------+
1 row in set (0.02 sec)
mysql> select * from bool_test where flag is true;
+----+------+
| id | flag |
+----+------+
| 1 | 1 |
| 3 | 9 |
+----+------+
2 rows in set (0.04 sec)
mysql> select * from bool_test where flag = false;
+----+------+
| id | flag |
+----+------+
| 2 | 0 |
+----+------+
1 row in set (0.01 sec)
mysql> select * from bool_test where flag is false;
+----+------+
| id | flag |
+----+------+
| 2 | 0 |
+----+------+
1 row in set (0.00 sec)
mysql> select * from bool_test where !flag;
+----+------+
| id | flag |
+----+------+
| 2 | 0 |
+----+------+
1 row in set (0.00 sec)
mysql> select * from bool_test where not flag;
+----+------+
| id | flag |
+----+------+
| 2 | 0 |
+----+------+
1 row in set (0.00 sec)
mysql> select * from bool_test where flag != true;
+----+------+
| id | flag |
+----+------+
| 2 | 0 |
| 3 | 9 |
+----+------+
2 rows in set (0.00 sec)
mysql> select * from bool_test where flag is not true;
+----+------+
| id | flag |
+----+------+
| 2 | 0 |
| 4 | NULL |
+----+------+
2 rows in set (0.00 sec)
mysql> select * from bool_test where flag != false;
+----+------+
| id | flag |
+----+------+
| 1 | 1 |
| 3 | 9 |
+----+------+
2 rows in set (0.04 sec)
mysql> select * from bool_test where flag is not false;
+----+------+
| id | flag |
+----+------+
| 1 | 1 |
| 3 | 9 |
| 4 | NULL |
+----+------+
3 rows in set (0.00 sec)
My Question is: when is it advisable to use is/is not and when is it advisable to use =/!= with true/false ? Which one is vendor independent?
MySQL is actually fooling you. It doesn't have a boolean column type at all:
BOOL, BOOLEAN
These types are synonyms for TINYINT(1). A value of zero is considered
false. Nonzero values are considered true:
Also, the boolean literals are not such:
The constants TRUE and FALSE evaluate to 1 and 0, respectively.
Considering that:
Many database systems do not have booleans either (not at least in standard SQL and column types)
MySQL doesn't have an easy way to enforce 0 or 1 in BOOLEAN
My conclusion would be:
You'll have to use WHERE IS flag or just WHERE flag because = simply doesn't work correctly. Which one, is possibly a matter of preference.
Whatever you choose, no option will be vendor independent. For instance, Oracle won't even run either of them.
Edit: if cross-platform is a must, I'd go for this:
WHERE flag=0
WHERE flag<>0
I'm sure we've all done it lots of times.
If the flag column is indexed and all values are either 0 or 1, where flag = true is much faster than where flag is true.
During our testing, is true resulted in a “full table scan” and took 1.121 seconds, while = true was executed with “key lookup” and only took 0.167 seconds. The table had about 3 million rows.