RDS High CPU utilization - mysql

I am facing high CPU utilization issue, is too many concurrent create temporary table statement cause high CPU utilization?
Is there any query through that we can capture queries which causing high CPU utilization?
Variable we set:-
tmp_table_size = 1G
max_heap_table_size = 1G
innodb_buffer_pool_size = 145 G
innodb_buffer_pool_instance = 8
innodb_page_cleaner = 8
Status Variables:-
mysql> show global status like '%tmp%';
+-------------------------+-----------+
| Variable_name | Value |
+-------------------------+-----------+
| Created_tmp_disk_tables | 60844516 |
| Created_tmp_files | 135751 |
| Created_tmp_tables | 107643364 |
+-------------------------+-----------+
mysql> show global status like '%innodb_buffer%';
+---------------------------------------+--------------------------------------------------+
| Variable_name | Value |
+---------------------------------------+--------------------------------------------------+
| Innodb_buffer_pool_dump_status | Dumping of buffer pool not started |
| Innodb_buffer_pool_load_status | Buffer pool(s) load completed at 170917 19:11:45 |
| Innodb_buffer_pool_resize_status | |
| Innodb_buffer_pool_pages_data | 8935464 |
| Innodb_buffer_pool_bytes_data | 146398642176 |
| Innodb_buffer_pool_pages_dirty | 18824 |
| Innodb_buffer_pool_bytes_dirty | 308412416 |
| Innodb_buffer_pool_pages_flushed | 122454921 |
| Innodb_buffer_pool_pages_free | 188279 |
| Innodb_buffer_pool_pages_misc | 377817 |
| Innodb_buffer_pool_pages_total | 9501560 |
| Innodb_buffer_pool_read_ahead_rnd | 0 |
| Innodb_buffer_pool_read_ahead | 585245 |
| Innodb_buffer_pool_read_ahead_evicted | 14383 |
| Innodb_buffer_pool_read_requests | 304878851665 |
| Innodb_buffer_pool_reads | 10537188 |
| Innodb_buffer_pool_wait_free | 0 |
| Innodb_buffer_pool_write_requests | 14749510186 |
+---------------------------------------+--------------------------------------------------+

Step 1 -
show processlist
Find is any process is locking table if yes than change it to myisam.
Step 2 -
Check Ram and your db size
Step 3 -
Explain complex queries and check if file sort or maximum number od rows are getting scan remove it either by making table flat , not more than 4 sub queries
Step 4 -
Use of joins efficiently

Related

MySQL to handle large amount of data

Currently, my application is generating around a 40million records every hour, I have created a partition for every hour so it will be easier for me to drop the partition as and when required, also using the partition to aggregate the data.
My aggregation procedure will run fine if there are not any queries happening but once those queries are started, the aggregation code takes more than an hour to complete.
in MySQL is there a process to freeze and not impact based on the queries happening to the database?
reply to #rick
RAM: 32GB
innodb_buffer_pool_size : 20GB
SSD: Yes
type of read: it's
a mix of group by and update over primary key
I don't want to do aggregation for every 5 minutes, since that would also generate a large number of records and it's not achievable for application, I'm actually saving 5 partitions and running the oldest every hour, my application required a minimum of 5 hours of non-aggregated data.
for my application, no need for the ACID type of characteristics so changed the default isolation to READ-UNCOMMITTED, and auto-commit to 0 that improved the aggregation code running faster but inserts took a hit, which is taking more than 2s.
here updating the profile info for the aggregation query
+----------+-----+---------------------------+-----------+------------+------------+-------------------+---------------------+--------------+---------------+---------------+-------------------+-------------------+-------------------+-------+-----------------------+----------------------+-------------+
| QUERY_ID | SEQ | STATE | DURATION | CPU_USER | CPU_SYSTEM | CONTEXT_VOLUNTARY | CONTEXT_INVOLUNTARY | BLOCK_OPS_IN | BLOCK_OPS_OUT | MESSAGES_SENT | MESSAGES_RECEIVED | PAGE_FAULTS_MAJOR | PAGE_FAULTS_MINOR | SWAPS | SOURCE_FUNCTION | SOURCE_FILE | SOURCE_LINE |
+----------+-----+---------------------------+-----------+------------+------------+-------------------+---------------------+--------------+---------------+---------------+-------------------+-------------------+-------------------+-------+-----------------------+----------------------+-------------+
| 50754 | 2 | continuing inside routine | 0.000015 | 0.000197 | 0.000036 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | NULL | NULL | NULL |
| 50754 | 3 | checking permissions | 0.000007 | 0.000005 | 0.000001 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | check_access | sql_authorization.cc | 809 |
| 50754 | 4 | checking permissions | 0.000006 | 0.000006 | 0.000000 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | check_access | sql_authorization.cc | 809 |
| 50754 | 5 | Opening tables | 0.000017 | 0.000013 | 0.000003 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | open_tables | sql_base.cc | 5815 |
| 50754 | 6 | init | 0.000260 | 0.000400 | 0.000073 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | handle_query | sql_select.cc | 128 |
| 50754 | 7 | System lock | 0.000011 | 0.000009 | 0.000001 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | mysql_lock_tables | lock.cc | 330 |
| 50754 | 8 | optimizing | 0.000115 | 0.000098 | 0.000017 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | optimize | sql_optimizer.cc | 158 |
| 50754 | 9 | statistics | 0.001624 | 0.003051 | 0.000552 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | optimize | sql_optimizer.cc | 374 |
| 50754 | 10 | preparing | 0.000158 | 0.000134 | 0.000024 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | optimize | sql_optimizer.cc | 482 |
| 50754 | 11 | Sorting result | 0.000009 | 0.000007 | 0.000001 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | make_tmp_tables_info | sql_select.cc | 3849 |
| 50754 | 12 | executing | 0.000006 | 0.000005 | 0.000001 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | exec | sql_executor.cc | 126 |
| 50754 | 13 | Sending data | 40.298694 | 144.161765 | 12.297466 | 596361 | 261826 | 265128 | 2899384 | 0 | 0 | 0 | 328 | 0 | exec | sql_executor.cc | 202 |
| 50754 | 14 | end | 0.000031 | 0.000024 | 0.000005 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | handle_query | sql_select.cc | 206 |
| 50754 | 15 | query end | 0.000016 | 0.000013 | 0.000003 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | mysql_execute_command | sql_parse.cc | 4959 |
| 50754 | 16 | closing tables | 0.000055 | 0.000048 | 0.000007 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | mysql_execute_command | sql_parse.cc | 5018 |
| 50754 | 17 | query end | 0.000007 | 0.000005 | 0.000002 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | mysql_execute_command | sql_parse.cc | 4959 |
| 50754 | 18 | closing tables | 0.000012 | 0.000009 | 0.000002 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | mysql_execute_command | sql_parse.cc | 5018 |
+----------+-----+---------------------------+-----------+------------+------------+-------------------+---------------------+--------------+---------------+---------------+-------------------+-------------------+-------------------+-------+-----------------------+----------------------+-------------+
aggregation code looks like this and each time takes around 100 clientKey around 100K clientKey available in each hour
insert into DB.NETWORK_USAGE_FINAL(clientKey,k1,k2,k3,k4,k5,createdAt)
select clientKey, sum(k1) as k1, sum(k2) as k2, sum(k3) as k3 ,
k4, k5 , "',startTime,'" from DB.NETWORK_USAGE_F1 partition (',partitionKey,')
where clientKey in (',selectedClientKey,')
group by clientKey,k4,k5
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
#
# * Basic Settings
#
innodb_buffer_pool_size=20G
innodb_buffer_pool_instances=20
max_connections=100
query_cache_size=0
query_cache_type=0
query_cache_limit=2M
innodb_log_file_size=3G
innodb_read_io_threads = 8
innodb_write_io_threads = 8
innodb_io_capacity = 2000
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
back_log = 1000
tmp_table_size = 1G
max_heap_table_size = 1G
join_buffer_size=1G
sort_buffer_size=512M
innodb_lru_scan_depth=100
table_open_cache=4000
max_allowed_packet=1G
innodb_file_per_table=1
character-set-server = utf8
collation-server = utf8_unicode_ci
event_scheduler = ON
transaction_isolation = READ-COMMITTED
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
key_buffer_size = 1G
thread_stack = 128M
thread_cache_size = 8
myisam-recover-options = BACKUP
log_error = /var/log/mysql/error.log
expire_logs_days = 10
max_binlog_size = 100M
show create table
CREATE TABLE `NETWORK_USAGE_F1` (
`id` char(15) NOT NULL,
`clientKey` int(11) NOT NULL,
`k4` int(11) NOT NULL,
`k5` char(50) NOT NULL,
`createdAt` datetime NOT NULL,
`partitionKey` int(11) NOT NULL,
`k1` bigint(20) NOT NULL,
`k2` bigint(20) NOT NULL,
`k3` int(11) NOT NULL,
PRIMARY KEY (`id`,`partitionKey`),
KEY `key2` (`clientKey`,`k4`,`k5`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
/*!50100 PARTITION BY RANGE (partitionKey)
*/
#reply to Rick updates:
Aggregation code is running on 100 clientkey's(limit) at a time, for an hour there will be around 100K unique clientKey exists, but the overall number of rows in the DB is around 40 million rows for an hour/partition(so its nothing but each clientKey would have around 400 rows)
id used is 15 character length only
Currently, I have 5 partitions, the partition key is in the format of YYYYMMMDDHH
Not using MyISAM
Let's see the aggregation code. And SHOW CREATE TABLE.
Other things to try:
Aggregate every 5 minutes instead.
Aggregate "continually". That is, do all the rows since you last did an aggregation.
Collect the rows into a 'staging' table. Then do the aggregation from that and finally copy the rows into the main table. Here's a discussion of a "ping-pong" technique for such: http://mysql.rjweb.org/doc.php/staging_table
11K rows per second is a lot, but not impossible.
Further questions:
How much RAM do you have?
What is the setting of innodb_buffer_pool_size?
Do you have SSD?
What type of reads are going on at the same time? (reading from big table? reading from summary table?)
(re my.cnf)
I recommend you limit each of these to 1% of RAM (unless you have some identified reason for being larger):
tmp_table_size = 1G
max_heap_table_size = 1G
join_buffer_size=1G
sort_buffer_size=512M
max_allowed_packet=1G
key_buffer_size = 1G -- for MyISAM only
I hope you are not using MyISAM. If not lower that one to 50M.
thread_stack = 128M
Ouch! Leave that at its default!
(Schema)
Use VARCHAR, not CHAR unless the strings are truly fixed-length.
Since your table will be huge, use the smallest practical INT size.
How many PARTITIONs are there? What version of MySQL? ("Too many" partitions can, by itself, be a performance drag.)
What is the partitionkey set to?
(other)
Is "around 100 clientKey around 100K clientKey" a typo, or am I confused by what it is saying?
Also, that seems to conflict with "40million records every hour".

xtrabackup does not write gtid information in xtrabackup_binlog_info

Software versions:
xtrabackup 8.0.12
percona-xtradb-cluster-server 8.0.18-9
I am running xtrabackup with this options:
--defaults-file=/etc/mysql/my.cnf --backup --user=backup
--password=**** --parallel=4 --no-timestamp --target-dir=/my-backup-dir
Some of server options:
binlog_format | ROW
gtid_mode | ON_PERMISSIVE
enforce_gtid_consistency | ON
File xtrabackup_binlog_info has only binlog file name and position:
mysql-bin.000159 251
No GTID, so I can not create GTID-based replication restoring a slave from this backup.
What should I do to make xtrabackup include this information?
UPDATE:
Check if GTIDs are enabled:
show global variables like '%gtid%';
| Variable_name | Value |
| binlog_gtid_simple_recovery | ON |
| enforce_gtid_consistency | ON |
| gtid_executed | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-3399594,
de211648-2642-ee18-628d-dc48283b005c:1-3697598:3877279-10141440 |
| gtid_executed_compression_period | 1000 |
| gtid_mode | ON_PERMISSIVE |
| gtid_owned | |
| gtid_purged | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-2661056,
de211648-2642-ee18-628d-dc48283b005c:1-3697598:3877279-10141440 |
| session_track_gtids | OFF |
show master status;
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
| mysql-bin.000166 | 15285372 | | | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-3358798,
de211648-2642-ee18-628d-dc48283b005c:1-3697598:3877279-10141440 |
(pavel.selivanov#localhost) [kassa_prod]> show binlog events in 'mysql-bin.000166' limit 7;
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info
| mysql-bin.000166 | 4 | Format_desc | 4315 | 124 | Server ver: 8.0.18-9, Binlog ver: 4 |
| mysql-bin.000166 | 124 | Previous_gtids | 4315 | 251 | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-3347159,
de211648-2642-ee18-628d-dc48283b005c:3697598:3877279-10141440 |
| mysql-bin.000166 | 251 | Gtid | 4315 | 330 | SET ##SESSION.GTID_NEXT= 'c0e3de06-a2a6-11ea-913c-c7b046cf5782:3347160' |
| mysql-bin.000166 | 330 | Query | 4315 | 411 | BEGIN |
| mysql-bin.000166 | 411 | Table_map | 4315 | 499 | table_id: 150 (db.table) |
| mysql-bin.000166 | 499 | Update_rows | 4315 | 3475 | table_id: 150 flags: STMT_END_F |
| mysql-bin.000166 | 3475 | Xid | 4315 | 3506 | COMMIT /* xid=9611331 */
From the manual:
ON_PERMISSIVE: New transactions are GTID transactions. Replicated transactions can be either anonymous or GTID transactions.
Check with show global variables like 'gt%'; or in your binary logs, if you actually have GTID transactions.
You actually don't have anything special to do, to have xtrabackup include GTIDs in the xtrabackup_binlog_info file: How to create a new (or repair a broken) GTID based slave

Slow InnoDB aggregation queries comparing to Postgresql

MySQL gurus
We have readonly dataset, which contains around 4 million rows. This dataset is represented as 1 InnoDB table with a bunch of indices, like this:
CREATE TABLE IF NOT EXISTS dim_users (
ds varchar(10),
user_id varchar(32),
date_joined varchar(10),
payer boolean,
source varchar(12),
country varchar(2),
is1dayactive boolean,
is7dayactive boolean,
is28dayactive boolean,
istest boolean
) ENGINE = InnoDB;
ALTER TABLE dim_users ADD INDEX (ds);
ALTER TABLE dim_users ADD INDEX (user_id);
ALTER TABLE dim_users ADD INDEX (date_joined);
ALTER TABLE dim_users ADD INDEX (source);
ALTER TABLE dim_users ADD INDEX (country);
MySQL8 is running on Ubuntu 18.04 in servers.com provider on a cloud server with 8GB of RAM and swap turned off.
The database has default config, except
[mysqld]
innodb-buffer-pool-size=6G
read_only=1
The problem is that aggregation queries like select count(*) from dim_users or select country,count(*) from dim_users group by 1 order by 1 desc; take too much time (up to 12 seconds) comparing to default Postgres 11 setup on the same vhost.
Default psql11 performs the last query for less than 2 seconds.
We were trying to profile and analyze this request on MySQL:
Explain:
mysql> explain analyze select country,count(*) from dim_users group by 1 order by 1 desc;
+---------------+
| EXPLAIN
+---------------+
| -> Group aggregate: count(0) (actual time=241.091..14777.679 rows=11 loops=1)
| -> Index scan on dim_users using country (reverse) (cost=430171.90 rows=4230924) (actual time=0.054..11940.166 rows=4228692 loops=1)
|
+---------------+
1 row in set (14.78 sec)
Profile:
mysql> SELECT SEQ,STATE,DURATION,SWAPS,SOURCE_FUNCTION,SOURCE_FILE,SOURCE_LINE FROM INFORMATION_SCHEMA.PROFILING WHERE QUERY_ID=1;
+-----+--------------------------------+-----------+-------+-------------------------+----------------------+-------------+
| SEQ | STATE | DURATION | SWAPS | SOURCE_FUNCTION | SOURCE_FILE | SOURCE_LINE |
+-----+--------------------------------+-----------+-------+-------------------------+----------------------+-------------+
| 2 | starting | 0.000115 | 0 | NULL | NULL | NULL |
| 3 | Executing hook on transaction | 0.000008 | 0 | launch_hook_trans_begin | rpl_handler.cc | 1119 |
| 4 | starting | 0.000013 | 0 | launch_hook_trans_begin | rpl_handler.cc | 1121 |
| 5 | checking permissions | 0.000010 | 0 | check_access | sql_authorization.cc | 2176 |
| 6 | Opening tables | 0.000050 | 0 | open_tables | sql_base.cc | 5591 |
| 7 | init | 0.000012 | 0 | execute | sql_select.cc | 677 |
| 8 | System lock | 0.000020 | 0 | mysql_lock_tables | lock.cc | 331 |
| 9 | optimizing | 0.000007 | 0 | optimize | sql_optimizer.cc | 282 |
| 10 | statistics | 0.000046 | 0 | optimize | sql_optimizer.cc | 502 |
| 11 | preparing | 0.000035 | 0 | optimize | sql_optimizer.cc | 583 |
| 12 | executing | 13.552128 | 0 | ExecuteIteratorQuery | sql_union.cc | 1409 |
| 13 | end | 0.000037 | 0 | execute | sql_select.cc | 730 |
| 14 | query end | 0.000008 | 0 | mysql_execute_command | sql_parse.cc | 4606 |
| 15 | waiting for handler commit | 0.000018 | 0 | ha_commit_trans | handler.cc | 1589 |
| 16 | closing tables | 0.000015 | 0 | mysql_execute_command | sql_parse.cc | 4657 |
| 17 | freeing items | 0.000039 | 0 | mysql_parse | sql_parse.cc | 5330 |
| 18 | cleaning up | 0.000020 | 0 | dispatch_command | sql_parse.cc | 2184 |
+-----+--------------------------------+-----------+-------+-------------------------+----------------------+-------------+
As you can see, group by query takes ages to complete.
So, my questions are the following.
I've never been using MySQL before, my main DB has always been Postgresql for years and I don't know whether it's normal behaviour for this DB or not - it's very slow when we want to execute group by queries.
Otherwise, if I'm missing something very important here, eg DB or OS configuration, could you please give me a pieces of advice on how to properly setup MySQL for read only purposes.
Thank you!

How do I check the value of 'binlog_do_db' in MySQL?

I am trying to debug why my DML statements aren't ending up in the binlogs. The master and slave appear to be talking to eachother and the slave status indicates that it caught up properly to the most recent log file.
I did a show variables LIKE '%binlog%'; with this output, but binlog_do_db is not there.
+-----------------------------------------+----------------------+
| Variable_name | Value |
+-----------------------------------------+----------------------+
| binlog_cache_size | 32768 |
| binlog_checksum | CRC32 |
| binlog_direct_non_transactional_updates | OFF |
| binlog_error_action | ABORT_SERVER |
| binlog_format | ROW |
| binlog_group_commit_sync_delay | 0 |
| binlog_group_commit_sync_no_delay_count | 0 |
| binlog_gtid_simple_recovery | ON |
| binlog_max_flush_queue_time | 0 |
| binlog_order_commits | ON |
| binlog_row_image | FULL |
| binlog_rows_query_log_events | OFF |
| binlog_stmt_cache_size | 32768 |
| innodb_api_enable_binlog | OFF |
| innodb_locks_unsafe_for_binlog | OFF |
| log_statements_unsafe_for_binlog | ON |
| max_binlog_cache_size | 18446744073709547520 |
| max_binlog_size | 104857600 |
| max_binlog_stmt_cache_size | 18446744073709547520 |
| sync_binlog | 1 |
+-----------------------------------------+----------------------+
After much effort and grief, I found that in the previous administrator must have been trying to get replication to work and put these lines into the /etc/mysql/mysql.conf.d/mysqld.cnf file
# Setup Incremental Backups
log-bin = /srv/ddisk/backups/mysql/mysql_inc
expire_logs_days = 3
max_binlog_size = 100M
server-id = 1
binlog_do_db = include_database_name
The last line is probably the culprit (though I will likely clean out that whole config block since I have them in the my.cnf config file already. In any case, I am now trying to verify that this configuration is actually being read by MySQL, but I'm unable to figure out what query to run to find this variable.
Would someone happen to know how to find binlog_do_db?
Thanks ahead of time
You can check using mysql prompt on the master node below:
mysql> SHOW MASTER STATUS;
It will show Binary log file, position, Binlog_Do_DB, Binlog_Ignore_DB parameter and so on.

Huge Temp table going on disk In mysql

How to avoid temporary table creation on disk in mysql if table has multiple text column.
I have given tmp_table_size to 1 GB and Max_heap_size = GB
Still tables going on Disk.
+-------------------------+-------+
| Variable_name | Value |
+-------------------------+-------+
| Created_tmp_disk_tables | 70742 |
| Created_tmp_files | 6 |
| Created_tmp_tables | 71076 |
Thanks :-)