How to estimate SQL query timing? - mysql

I'm trying to get an rough (order-of-magnitude) estimate of how long time the following query could take:
mysql> EXPLAIN SELECT t1.col1, t1_col4 FROM t1 LEFT JOIN t2 ON t1.col1=t2.col1 WHERE col2=0 AND col3 IS NULL;
+----+-------------+--------------------+------+---------------+------------+---------+-----------------------------+---------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------------+------+---------------+------------+---------+-----------------------------+---------+--------------------------+
| 1 | SIMPLE | t1 | ref | foobar | foobar | 4 | const | 9715129 | |
| 1 | SIMPLE | t2 | ref | col1 | col1 | 4 | db2.t1.col1 | 42318 | Using where; Using index |
+----+-------------+--------------------+------+---------------+------------+---------+-----------------------------+---------+--------------------------+
2 rows in set (0.00 sec)
mysql>

This can be done when using SHOW PROFILES syntax.
When you open a MySQL session, you could set the variable "profiling" to 1 or ON.
mysql> SET profiling = 1;
So all the statements sent to the server will be profiled and stored in a historical and shown later by typing the command:
mysql> SHOW PROFILES;
See, from MySQL manual:
mysql> SET profiling = 1;
Query OK, 0 rows affected (0.00 sec)
mysql> DROP TABLE IF EXISTS t1;
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> CREATE TABLE T1 (id INT);
Query OK, 0 rows affected (0.01 sec)
mysql> SHOW PROFILES;
+----------+----------+--------------------------+
| Query_ID | Duration | Query |
+----------+----------+--------------------------+
| 0 | 0.000088 | SET PROFILING = 1 |
| 1 | 0.000136 | DROP TABLE IF EXISTS t1 |
| 2 | 0.011947 | CREATE TABLE t1 (id INT) |
+----------+----------+--------------------------+
3 rows in set (0.00 sec)
mysql> SHOW PROFILE;
+----------------------+----------+
| Status | Duration |
+----------------------+----------+
| checking permissions | 0.000040 |
| creating table | 0.000056 |
| After create | 0.011363 |
| query end | 0.000375 |
| freeing items | 0.000089 |
| logging slow query | 0.000019 |
| cleaning up | 0.000005 |
+----------------------+----------+
7 rows in set (0.00 sec)
mysql> SHOW PROFILE FOR QUERY 1;
+--------------------+----------+
| Status | Duration |
+--------------------+----------+
| query end | 0.000107 |
| freeing items | 0.000008 |
| logging slow query | 0.000015 |
| cleaning up | 0.000006 |
+--------------------+----------+
4 rows in set (0.00 sec)
mysql> SHOW PROFILE CPU FOR QUERY 2;
+----------------------+----------+----------+------------+
| Status | Duration | CPU_user | CPU_system |
+----------------------+----------+----------+------------+
| checking permissions | 0.000040 | 0.000038 | 0.000002 |
| creating table | 0.000056 | 0.000028 | 0.000028 |
| After create | 0.011363 | 0.000217 | 0.001571 |
| query end | 0.000375 | 0.000013 | 0.000028 |
| freeing items | 0.000089 | 0.000010 | 0.000014 |
| logging slow query | 0.000019 | 0.000009 | 0.000010 |
| cleaning up | 0.000005 | 0.000003 | 0.000002 |
+----------------------+----------+----------+------------+
References (updated at: 2014-09-04):
- SHOW PROFILE Syntax
- The INFORMATION_SCHEMA PROFILING Table
- How To Use MySQL Query Profiling (The Digital Ocean recently published a great article concerning this issue.)

Related

Can I create MySQL View with SHOW keywords?

I need to create view like
create view GetProcessList
as
show processlist
Is it possible?
Is it possible?
No you can only create a view base on a SELECT statement , not a SHOW statement. But if your question is really how you can access the processlist as if it were a table....then this is already available in the information_schema meta-database:
MariaDB [information_schema]> SELECT * FROM information_schema.PROCESSLIST;
+----+------+-----------+--------------------+---------+------+----------------------+----------------------------------------------+---------+-------+-----------+----------+-------------+---------------+----------+
| ID | USER | HOST | DB | COMMAND | TIME | STATE | INFO | TIME_MS | STAGE | MAX_STAGE | PROGRESS | MEMORY_USED | EXAMINED_ROWS | QUERY_ID |
+----+------+-----------+--------------------+---------+------+----------------------+----------------------------------------------+---------+-------+-----------+----------+-------------+---------------+----------+
| 42 | root | localhost | information_schema | Query | 0 | Filling schema table | SELECT * FROM information_schema.PROCESSLIST | 0.561 | 0 | 0 | 0.000 | 83104 | 0 | 204 |
+----+------+-----------+--------------------+---------+------+----------------------+----------------------------------------------+---------+-------+-----------+----------+-------------+---------------+----------+
1 row in set (0.00 sec)

Select count takes two minutes on a two column table

I have a MariaDB table with just under 100000 rows, and selecting the count takes a very long time (almost 2 minutes).
Selecting anything by id from the table though takes only 4 milliseconds.
The text field here contains on average 5000 characters.
How can I speed this up?
MariaDB [companies]> describe company_details;
+---------+------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------+------------------+------+-----+---------+-------+
| id | int(10) unsigned | NO | PRI | NULL | |
| details | text | YES | | NULL | |
+---------+------------------+------+-----+---------+-------+
MariaDB [companies]> explain select count(id) from company_details;
+------+-------------+-----------------+-------+---------------+---------+---------+------+-------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-----------------+-------+---------------+---------+---------+------+-------+-------------+
| 1 | SIMPLE | company_details | index | NULL | PRIMARY | 4 | NULL | 71267 | Using index |
+------+-------------+-----------------+-------+---------------+---------+---------+------+-------+-------------+
MariaDB [companies]> analyze table company_details;
+---------------------------+---------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+---------------------------+---------+----------+----------+
| companies.company_details | analyze | status | OK |
+---------------------------+---------+----------+----------+
1 row in set (0.098 sec)
MariaDB [companies]> select count(id) from company_details;
+-----------+
| count(id) |
+-----------+
| 96544 |
+-----------+
1 row in set (1 min 43.199 sec)
This becomes an even bigger problem when I try to join the table.
For example, to find the number of companies which do not have associated details:
MariaDB [companies]> SELECT COUNT(*) FROM company c LEFT JOIN company_details cd ON c.id = cd.id WHERE cd.id IS NULL;
+----------+
| count(*) |
+----------+
| 42178 |
+----------+
1 row in set (10 min 28.846 sec)
Edit:
After running OPTIMIZE on the table, the select count has improved speed from 1min 43sec to just 5 sec, and the join has improved speed from 10 minutes to 25 seconds.
MariaDB [companies]> optimize table company_details;
+---------------------------+----------+----------+-------------------------------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+---------------------------+----------+----------+-------------------------------------------------------------------+
| companies.company_details | optimize | note | Table does not support optimize, doing recreate + analyze instead |
| companies.company_details | optimize | status | OK |
+---------------------------+----------+----------+-------------------------------------------------------------------+
2 rows in set (11 min 21.195 sec)
I think there's OPTIMIZE -command for rebuilding indexes.
OPTIMIZE company_details;
This usually takes some time to complete. More details: https://mariadb.com/kb/en/optimize-table/

MySQL ROLLBACK not deleting new records - sequelize

I'm using sequelize v5.1.0 to create a transaction in MySQL 5.7.25-0ubuntu0.18.04.2. It appears to be executing the correct commands according to the MySQL 5.7 Documentation, however, the record that was inserted and rolled back still exists in the database afterwards.
javascript
let promises = []
models.sequelize.transaction(function (t) {
promises.push(models.alert.create(setter, { transaction: t }))
promises.push(new Promise((resolve, reject) => {
reject(new Error('roll it back yall'))
}))
return Promise.all(promises)
}).then(function () {
console.log('SUCCESS!!! (will commit)')
}).catch(function (err) {
console.log('FAILURE !!! (will rollback)')
next(err)
})
SQL query log
| 2019-03-21 12:55:17.798200 | root[root] # [10.211.55.2] | 2151 | 0 | Query | START TRANSACTION |
| 2019-03-21 12:55:19.597304 | root[root] # [10.211.55.2] | 2151 | 0 | Prepare | INSERT INTO `alerts` (`id`,`user_id`,`alert_name`,`reading_type`,`reading_condition`,`reading_value`,`always_active`,`sensors_global`,`enabled`,`last_updated`,`updated`) VALUES (DEFAULT,?,?,?,?,?,?,?,?,?,?) |
| 2019-03-21 12:55:19.616278 | root[root] # [10.211.55.2] | 2151 | 0 | Execute | INSERT INTO `alerts` (`id`,`user_id`,`alert_name`,`reading_type`,`reading_condition`,`reading_value`,`always_active`,`sensors_global`,`enabled`,`last_updated`,`updated`) VALUES (DEFAULT,21,'Test Alert','temperature','below',60,1,0,1,'2019-03-21 12:55:17.781','2019-03-21 12:55:17') |
| 2019-03-21 12:55:19.619249 | root[root] # [10.211.55.2] | 2151 | 0 | Query | ROLLBACK
record in database after
mysql> select * from alerts where alert_name='Test Alert';
+-------+---------+------------+--------------+-------------------+---------------+---------------+---------------+----------------+---------+---------------------+---------------------+
| id | user_id | alert_name | reading_type | reading_condition | reading_value | alert_message | always_active | sensors_global | enabled | updated | last_updated |
+-------+---------+------------+--------------+-------------------+---------------+---------------+---------------+----------------+---------+---------------------+---------------------+
| 48689 | 21 | Test Alert | temperature | below | 60.00 | NULL | 1 | 0 | 1 | 2019-03-21 06:55:17 | 2019-03-21 12:55:18 |
+-------+---------+------------+--------------+-------------------+---------------+---------------+---------------+----------------+---------+---------------------+---------------------+
1 row in set (0.00 sec)
Update:
Poking around in MySQL CLI gives a warning:
mysql> start transaction;
Query OK, 0 rows affected (0.00 sec)
mysql> insert into contacts ( user_id, contact_name ) values (21, 'Some Person' );
Query OK, 1 row affected (0.00 sec)
mysql> rollback;
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> show warnings;
+---------+------+---------------------------------------------------------------+
| Level | Code | Message |
+---------+------+---------------------------------------------------------------+
| Warning | 1196 | Some non-transactional changed tables couldn't be rolled back |
+---------+------+---------------------------------------------------------------+
1 row in set (0.00 sec)
What makes some tables non-transactional?
It appears that the alerts table is using the MyISAM engine, which does not support transactions:
mysql> show table status like 'alerts';
+--------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+--------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
| alerts | MyISAM | 10 | Dynamic | 18 | 136 | 2712 | 281474976710655 | 2048 | 256 | 48690 | 2019-03-19 10:38:39 | 2019-03-21 06:55:19 | NULL | utf8_general_ci | NULL | | |
+--------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+------------+-----------------+----------+----------------+---------+
1 row in set (0.00 sec)
To change the db Engine, follow the guidelines here and:
mysql> ALTER TABLE alerts ENGINE=InnoDB;

MySQL - slow access to the table index or table data

I have MyISAM tables with a big number of rows - 10M to 500M. The tables are used to store time series data infrequently and I would like to optimize it for SELECT, which I am doing throung 2 indexes: the epoch and a classifier column (which contain a few thousand categories).
My issue is that the first SELECT I do for a particular category is quite long (10 to 50 sec) while the subsequent ones are really fast, even when using SQL_NO_CACHE. Such a query will typically return between 100,000 and 1M elements.
Profiling shows that MySQL spends a lot of time "Sending data". This would indicate that a lot of the time spent is done doing disk I/O. But I do not really understand where the bottleneck is:
Is the bottleneck in the BTREE read? The tree has only a few thousands nodes and then less than 1M points in the selected node. I cannot believe it would take 30 seconds to do that on a modern machine, even with old school hard drives.
Is it in reading the rows in the table? That is again less than a million rows with an average length of ~40bytes.
Something else I am not accounting for?
Here are the query results:
mysql> SELECT SQL_NO_CACHE COUNT(`Time`) FROM archive_1 WHERE Channel=63;
+---------------+
| COUNT(`Time`) |
+---------------+
| 450619 |
+---------------+
1 row in set (28.67 sec)
mysql> SELECT SQL_NO_CACHE COUNT(`Time`) FROM archive_1 WHERE Channel=63;
+---------------+
| COUNT(`Time`) |
+---------------+
| 450619 |
+---------------+
1 row in set (2.20 sec)
mysql> SELECT SQL_NO_CACHE COUNT(`Time`) FROM archive_1 WHERE Channel=63;
+---------------+
| COUNT(`Time`) |
+---------------+
| 450619 |
+---------------+
1 row in set (0.88 sec)
mysql> SHOW PROFILES;
+----------+-------------+-----------------------------------------------------------------------------------+
| Query_ID | Duration | Query |
+----------+-------------+-----------------------------------------------------------------------------------+
| 1 | 28.66720725 | SELECT SQL_NO_CACHE COUNT(`Time`) FROM archive_1 WHERE Channel=63 |
| 2 | 2.19872350 | SELECT SQL_NO_CACHE COUNT(`Time`) FROM archive_1 WHERE Channel=63 |
| 3 | 0.87811475 | SELECT SQL_NO_CACHE COUNT(`Time`) FROM archive_1 WHERE Channel=63 |
+----------+-------------+-----------------------------------------------------------------------------------+
3 rows in set (0.00 sec)
mysql> SHOW PROFILE FOR QUERY 1;
+----------------------+-----------+
| Status | Duration |
+----------------------+-----------+
| starting | 0.000113 |
| checking permissions | 0.000010 |
| Opening tables | 0.000027 |
| System lock | 0.000017 |
| init | 0.000030 |
| optimizing | 0.000018 |
| statistics | 0.055731 |
| preparing | 0.000024 |
| executing | 0.000008 |
| Sending data | 28.611161 |
| end | 0.000019 |
| query end | 0.000005 |
| closing tables | 0.000014 |
| freeing items | 0.000021 |
| logging slow query | 0.000003 |
| logging slow query | 0.000004 |
| cleaning up | 0.000005 |
+----------------------+-----------+
17 rows in set (0.00 sec)
mysql> SHOW PROFILE FOR QUERY 2;
+----------------------+----------+
| Status | Duration |
+----------------------+----------+
| starting | 0.000105 |
| checking permissions | 0.000011 |
| Opening tables | 0.000036 |
| System lock | 0.000015 |
| init | 0.000028 |
| optimizing | 0.000019 |
| statistics | 0.032255 |
| preparing | 0.000024 |
| executing | 0.000007 |
| Sending data | 2.166140 |
| end | 0.000020 |
| query end | 0.000004 |
| closing tables | 0.000014 |
| freeing items | 0.000025 |
| logging slow query | 0.000003 |
| cleaning up | 0.000018 |
+----------------------+----------+
16 rows in set (0.00 sec)
mysql> SHOW PROFILE FOR QUERY 3;
+----------------------+----------+
| Status | Duration |
+----------------------+----------+
| starting | 0.000071 |
| checking permissions | 0.000009 |
| Opening tables | 0.000018 |
| System lock | 0.000012 |
| init | 0.000021 |
| optimizing | 0.000014 |
| statistics | 0.000059 |
| preparing | 0.000020 |
| executing | 0.000007 |
| Sending data | 0.877795 |
| end | 0.000021 |
| query end | 0.000004 |
| closing tables | 0.000015 |
| freeing items | 0.000029 |
| logging slow query | 0.000015 |
| cleaning up | 0.000006 |
+----------------------+----------+
16 rows in set (0.00 sec)
The particular table I am querying contains 107,407,213 rows with a data length of 4,237,427,600 bytes and an index length of 4,255,541,248 bytes. I optimized it yesterday and there where no added data since then.
If the query is I/O bound, I can always switch to a SSD and I also have the option to store the time index as an integer instead of a double. But so far I do not understand where my bottleneck is and I would like to avoid major changes before knowing more.
SQL_NO_CACHE means that mysql should not use the query cache.
The disk/buffer cache is still used, which is why the first query takes longer time, and the subseqent queries are faster.

Can I define a lifetime for each row of MySQL's table?

I have a table like this:
// banned
+----+---------+---------------+
| id | user_id | unix_time |
+----+---------+---------------+
| 1 | 32534 | 1467066745524 |
| 2 | 43535 | 1467066745541 |
| 3 | 24352 | 1467066745618 |
| 4 | 88734 | 1467066746093 |
+----+---------+---------------+
Actually I need to define a expire time for each row when I insert it. Is that possible in MySQL? (I heard it is possible in Redis, well what about MySQL?)
So I want something like this:
// banned
+----+---------+---------------+
| id | user_id | unix_time |
+----+---------+---------------+
| 1 | 32534 | 1467066745524 | -- removing this row automatically in 10 min
| 2 | 43535 | 1467066745541 | -- removing this row automatically in 1 hour
| 3 | 24352 | 1467066745618 | -- removing this row automatically 2 day
| 4 | 88734 | 1467066746093 | -- removing this row automatically 8 hours min
+----+---------+---------------+
As you see, each row has a arbitrary lifetime.
you can create a EVENT that running every minute and delete old records like this:
enable scheduler
SET GLOBAL event_scheduler = ON;
create EVENT runs every minute
CREATE EVENT cleanup
ON SCHEDULE EVERY 1 MINUTE
DO
DELETE
FROM yourTable
WHERE unix_time < UNIX_TIMESTAMP();
Sample
MariaDB [yourschema]> SET GLOBAL event_scheduler = ON;
Query OK, 0 rows affected (0.00 sec)
MariaDB [yourschema]> SHOW GLOBAL VARIABLES LIKE 'event_scheduler';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| event_scheduler | ON |
+-----------------+-------+
1 row in set (0.00 sec)
MariaDB [yourschema]> SHOW PROCESSLIST;
+-----+-----------------+-----------------+------------+---------+------+-----------------------------+------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+-----+-----------------+-----------------+------------+---------+------+-----------------------------+------------------+----------+
| 28 | root | localhost:53255 | yourSchema | Sleep | 11 | | NULL | 0.000 |
| 29 | root | localhost:53256 | NULL | Sleep | 28 | | NULL | 0.000 |
| 34 | event_scheduler | localhost | NULL | Daemon | 6603 | Waiting for next activation | NULL | 0.000 |
| 316 | root | localhost | yourschema | Query | 0 | init | SHOW PROCESSLIST | 0.000 |
+-----+-----------------+-----------------+------------+---------+------+-----------------------------+------------------+----------+
4 rows in set (0.00 sec)
MariaDB [yourschema]>
MariaDB [yourschema]> SHOW events;
Empty set (0.01 sec)
MariaDB [yourschema]> DROP EVENT IF EXISTS cleanup;
Query OK, 0 rows affected, 1 warning (0.00 sec)
MariaDB [yourschema]> CREATE EVENT cleanup
-> ON SCHEDULE EVERY 5 SECOND ON COMPLETION PRESERVE ENABLE
-> DO
-> DELETE
-> FROM schedulerTable
-> WHERE expire_unix_time < UNIX_TIMESTAMP() LIMIT 10;
Query OK, 0 rows affected (0.00 sec)
MariaDB [yourschema]> INSERT INTO schedulerTable VALUES
-> (NULL,100,UNIX_TIMESTAMP(now() + INTERVAL 30 SECOND)),
-> (NULL,101,UNIX_TIMESTAMP(now() + INTERVAL 40 SECOND)),
-> (NULL,111,UNIX_TIMESTAMP(now() + INTERVAL 1 MINUTE));
Query OK, 3 rows affected (0.01 sec)
Records: 3 Duplicates: 0 Warnings: 0
MariaDB [yourschema]> SELECT * FROM schedulerTable;SELECT SLEEP(20);
+-------------+---------+------------------+
| id_rubrique | id_stat | expire_unix_time |
+-------------+---------+------------------+
| 1 | 100 | 1467074632 |
| 2 | 101 | 1467074642 |
| 3 | 111 | 1467074662 |
+-------------+---------+------------------+
3 rows in set (0.00 sec)
+-----------+
| SLEEP(20) |
+-----------+
| 0 |
+-----------+
1 row in set (20.00 sec)
MariaDB [yourschema]> SELECT * FROM schedulerTable;SELECT SLEEP(20);
+-------------+---------+------------------+
| id_rubrique | id_stat | expire_unix_time |
+-------------+---------+------------------+
| 2 | 101 | 1467074642 |
| 3 | 111 | 1467074662 |
+-------------+---------+------------------+
2 rows in set (0.00 sec)
+-----------+
| SLEEP(20) |
+-----------+
| 0 |
+-----------+
1 row in set (20.01 sec)
MariaDB [yourschema]> SELECT * FROM schedulerTable;SELECT SLEEP(20);
+-------------+---------+------------------+
| id_rubrique | id_stat | expire_unix_time |
+-------------+---------+------------------+
| 3 | 111 | 1467074662 |
+-------------+---------+------------------+
1 row in set (0.00 sec)
+-----------+
| SLEEP(20) |
+-----------+
| 0 |
+-----------+
1 row in set (20.01 sec)
MariaDB [yourschema]> SELECT * FROM schedulerTable;
Empty set (0.00 sec)
MariaDB [yourschema]>
You could create a view, something like this:
create view v_table as
select (case when unix_time < UNIX_TIMESTAMP() then id end) as id,
(case when unix_time < UNIX_TIMESTAMP() then user_id end) as user_id
from banned;
Or:
create view v_table as
select b.*
from banned b
where unixtime < UNIX_TIME();
This returns no information about a user after the timestamp. You can then schedule an event to periodically delete rows with expired information.