Mysql slow query on large table - mysql

With Mysql 5.6.10, I have a table like this:
CREATE TABLE `es_user_action` (
`id` bigint(11) unsigned NOT NULL AUTO_INCREMENT,
`user_id` bigint(11) unsigned NOT NULL,
`company_id` bigint(11) unsigned NOT NULL,
`work_id` bigint(11) unsigned NOT NULL,
`action` tinyint(2) NOT NULL COMMENT '10, 20, 30, 40',
`action_id` bigint(11) unsigned DEFAULT NULL,
`apply_id` bigint(11) unsigned DEFAULT '0',
`apply_display_id` varchar(100) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`action_time` datetime NOT NULL,
`scout_time` datetime NOT NULL,
`register_datetime_sorting` datetime DEFAULT NULL,
`score` tinyint(2) DEFAULT '0',
`is_pending` tinyint(2) DEFAULT '0',
`apply_status` tinyint(2) DEFAULT '2' COMMENT '1: paid, 2: free',
`has_response` tinyint(2) DEFAULT '0',
`response_time` datetime DEFAULT NULL,
`is_shown` tinyint(2) DEFAULT '1',
`source` varchar(15) COLLATE utf8mb4_unicode_ci NOT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `user_company_work` (`user_id`,`work_id`,`company_id`,`apply_id`,`source`),
KEY `IDX_2` (`company_id`,`is_shown`,`apply_status`,`apply_id`,`work_id`,`action_time`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=436896779 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
The full query I want to perform on the table would be the following:
SELECT
`id` AS `seqNo`,
`company_id` AS `companyId`,
`work_id` AS `workId`,
`action` AS `userActionType`,
`action_time` AS `userActionDatetime`,
`is_pending`,
`apply_status`,
`action_id`,
`source`,
`user_id` AS `userId`,
`apply_display_id`
FROM
`es_user_action`
WHERE
`company_id` = 449664
AND `is_shown` = 1
AND `apply_status` = 1
AND `apply_id` = 0
AND `action_time` >= '2021-01-05 15:56:14'
ORDER BY
is_pending ASC,
action ASC,
score DESC,
CASE
source
WHEN "entenshoku" THEN
action_time
END DESC,
CASE
WHEN source <> "entenshoku" THEN
action_time
END ASC
LIMIT 100 OFFSET 0;
The table has around 15 million rows and the following query takes around 15 seconds. The query becomes very slow.
Can anyone help out? Thanks in advance.
UPDATED:
This is the explain query result:

KEY `IDX_2` (`company_id`,`is_shown`,`apply_status`,`apply_id`,`work_id`,`action_time`)
Work_id is in the way of that index being effective for that SELECT. Remove it. That should speed up the query some. Still the are too many columns and expressions involved in the WHER and ORDER BY to do much more to help.
Changing BIGINTs to smaller INT types will help some (by shrinking the table).

I'm pretty sure that what you have cannot work.
You have is a dynamically changing ORDER BY, depending on the value of source. But which row's source?
If you could decide before issuing the query whether to scan action_time ascending or descending, which value of source would you pick?
Delete this Question, rewrite the query, then open another Question if you still have an issue.

Related

Optimise a MySQL query

I am using MySQL 8.0.20 and I have a huge table with some data in it.
CREATE TABLE `db_table` (
`utcDateTime` datetime DEFAULT NULL,
`dateTime` datetime DEFAULT NULL,
`col1` varchar(250) DEFAULT NULL,
`col2` varchar(10) DEFAULT NULL,
`col3` varchar(5) NOT NULL DEFAULT '1',
`col4` varchar(5) NOT NULL DEFAULT '1',
`id` int NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_dateIns` (`utcDateTime`,`col2`)
) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8'
Below is the query I am trying to optimise,
SELECT col3,col4
from db_table
WHERE col2='123456'
AND (dateTime BETWEEN '2020-11-25 00:01' AND '2020-11-26 00:00')
group by utcDateTime
order by utcDateTime asc limit 1;
This query takes about 12 seconds to execute. But if I change the sorting order to desc, it takes about 0.015 seconds.
Can someone please help optimise the query or the database table?
Thank you

MySQL Composite Index Question - Performance issue

I have done some research on the index before posting the question here. So far I believe I have done this correctly, but for some reason, the performance of a query that returns around 2400 records has not been good.
Here is the table schema
CREATE TABLE `tblCheck` (
`id` VARCHAR(50) NULL DEFAULT NULL COLLATE 'latin1_swedish_ci',
`token` VARCHAR(50) NULL DEFAULT NULL COLLATE 'latin1_swedish_ci',
`domainId` VARCHAR(50) NULL DEFAULT NULL COLLATE 'latin1_swedish_ci',
`time` DATETIME NULL DEFAULT NULL,
`responseCode` INT(11) NULL DEFAULT NULL,
`totalTime` DECIMAL(10,2) NULL DEFAULT NULL,
`namelookupTime` INT(11) NULL DEFAULT NULL,
`connectTime` INT(11) NULL DEFAULT NULL,
`pretransferTime` INT(11) NULL DEFAULT NULL,
`startTransferTime` INT(11) NULL DEFAULT NULL,
`redirectTime` INT(11) NULL DEFAULT NULL,
`appconnectTime` INT(11) NULL DEFAULT NULL,
`responseText` TEXT(65535) NULL DEFAULT NULL COLLATE 'latin1_swedish_ci',
`agentId` VARCHAR(50) NULL DEFAULT NULL COLLATE 'latin1_swedish_ci',
`isHealthy` CHAR(50) NULL DEFAULT NULL COLLATE 'latin1_swedish_ci',
`ftp_connect_time` DECIMAL(10,4) NULL DEFAULT NULL,
`ftp_login_time` DECIMAL(10,4) NULL DEFAULT NULL,
`ftp_change_mode_time` DECIMAL(10,4) NULL DEFAULT NULL,
`ftp_list_time` DECIMAL(10,4) NULL DEFAULT NULL,
`syntheticToken` VARCHAR(50) NULL DEFAULT NULL COLLATE 'latin1_swedish_ci',
UNIQUE INDEX `id` (`id`) USING BTREE,
INDEX `domainId` (`domainId`) USING BTREE,
INDEX `deleteTime` (`time`) USING BTREE,
INDEX `SearchIndex` (`domainId`, `time`, `agentId`) USING BTREE
)
COLLATE='latin1_swedish_ci'
ENGINE=InnoDB
ROW_FORMAT=COMPACT
;
The Query
SELECT *
FROM `tblCheck`
WHERE (`time` BETWEEN '2020-05-04 22:15:04' AND '2020-05-05 22:15:04')
AND `domainId` = '03d4c1ce-8b13-11ea-abf5-124e96b5f417'
AND `agentId` != '145a-f6bb-11e8-983f-1231322cbdb6'
ORDER BY `time` DESC
;
/* Affected rows: 0 Found rows: 2,418 Warnings: 0 Duration for 1 query: 0.109 sec. (+ 10.360 sec. network) */
It returned 2418 rows but took almost 10s.
Running it with EXPLAIN
EXPLAIN SELECT *
FROM `tblCheck`
WHERE (`time` BETWEEN '2020-05-04 22:15:04' AND '2020-05-05 22:15:04')
AND `domainId` = '03d4c1ce-8b13-11ea-abf5-124e96b5f417'
AND `agentId` != '145a-f6bb-11e8-983f-1231322cbdb6'
ORDER BY `time` DESC
Returns this
This looks like it is using the index "SearchIndex". However, I can't figure out why it would take to process 10s for 2k rows
For this query:
SELECT *
FROM `tblCheck`
WHERE
`time` BETWEEN '2020-05-04 22:15:04' AND '2020-05-05 22:15:04'
AND `domainId` = '03d4c1ce-8b13-11ea-abf5-124e96b5f417'
AND `agentId` != '145a-f6bb-11e8-983f-1231322cbdb6'
ORDER BY `time` DESC
The right index would be: (domainId, agentId, time), or (domainId, time, agentId). You have an the second index in place, and the query plan shows that MySQL happily uses it.
Looking at the explain summary, you can see:
Duration for 1 query: 0.109 sec. (+ 10.360 sec. network)
The query runs fast in the database. The bottleneck is the network, that is the time taken to return the 2000+ rows from the database to the client. Not much can be done from database perspective. Speed up your network, or switch to a local database if you can.
As a side note: select * is a not good for performance; you should try and reduce the number of columns that the query returns (this might also reduce the amount that needs to transmitted over the network).
You don't have the primary key.
You only have a UNIQUE INDEX.
If you con't nee that bulky responseText column in the results, don't include it. This may dramatically speed up the query.
(This is because large columns are stored "off-record", thereby taking an extra disk read if the table is huge.)

Huge speed difference in two similar queries (MySQL ORDER clause)

Important updade (explanation):
I realized that my query having single DESC order is 10 times slower that the same query with ASC order. The ordered field has an index. Is it normal behavior?
Original question with queries:
I have a mysql table with a few hundred of product items. It's suprising (for me) how 2 similar sql queries differs in terms of performance. I don't know why. Can you please give me a hint or explain why the difference is so huge?
This query takes 3ms:
SELECT
*
FROM
`product_items`
WHERE
(product_items.shop_active = 1)
AND (product_items.active = 1)
AND (product_items.active_category_id is not null)
AND (has_picture is not null)
AND (price_orig is not null)
AND (category_min_discount IS NOT NULL)
AND (product_items.slug is not null)
AND `product_items`.`active_category_id` IN (6797, 5926, 5806, 6852)
ORDER BY
price asc
LIMIT 1
But the following query takes already 169ms... Only difference is that the order clause contains 2 columns. "Price" value has each product, while "price top" has roughly only 1% of products.
SELECT
*
FROM
`product_items`
WHERE
(product_items.shop_active = 1)
AND (product_items.active = 1)
AND (product_items.active_category_id is not null)
AND (has_picture is not null)
AND (price_orig is not null)
AND (category_min_discount IS NOT NULL)
AND (product_items.slug is not null)
AND `product_items`.`active_category_id` IN (6797, 5926, 5806, 6852)
ORDER BY
price asc,
price_top desc
LIMIT 1
The table structure looks like this:
CREATE TABLE `product_items` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`shop_id` int(11) DEFAULT NULL,
`item_id` varchar(255) DEFAULT NULL,
`productname` varchar(255) DEFAULT NULL,
`description` text,
`url` text,
`url_hash` varchar(255) DEFAULT NULL,
`img_url` text,
`price` decimal(10,2) DEFAULT NULL,
`price_orig` decimal(10,2) DEFAULT NULL,
`discount` decimal(10,2) DEFAULT NULL,
`discount_percent` decimal(10,2) DEFAULT NULL,
`manufacturer` varchar(255) DEFAULT NULL,
`delivery_date` varchar(255) DEFAULT NULL,
`categorytext` text,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`active_category_id` int(11) DEFAULT NULL,
`shop_active` int(11) DEFAULT NULL,
`active` int(11) DEFAULT '0',
`price_top` decimal(10,2) NOT NULL DEFAULT '0.00',
`attention_priority` int(11) DEFAULT NULL,
`attention_priority_over` int(11) DEFAULT NULL,
`has_picture` varchar(255) DEFAULT NULL,
`size` varchar(255) DEFAULT NULL,
`category_min_discount` int(11) DEFAULT NULL,
`slug` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `index_product_items_on_url_hash` (`url_hash`),
KEY `index_product_items_on_shop_id` (`shop_id`),
KEY `index_product_items_on_active_category_id` (`active_category_id`),
KEY `index_product_items_on_productname` (`productname`),
KEY `index_product_items_on_price` (`price`),
KEY `index_product_items_on_discount_percent` (`discount_percent`),
KEY `index_product_items_on_price_top` (`price_top`)
) ENGINE=InnoDB AUTO_INCREMENT=1715708 DEFAULT CHARSET=utf8;
UPDATE
I realized that the difference is mainly in the type of ordering: if I use asc+asc for both columns the query takes around 6ms, if I use asc+desc or desc+asc, the query takes around 160ms..
Thank you.
If creating an index to help ORDER BY doesn't help, try creating an index that helps both WHERE and ORDER BY:
CREATE INDEX product_items_i1 ON product_items (
shop_active,
active,
active_category_id,
has_picture,
price_orig,
category_min_discount,
slug,
price,
price_top DESC
)
Obviously, this is a bit clunky, and you'll have to balance the performance gain for the query with the price of maintaining the index.

query slows down if add field to where condition

I have a table Mysql fiddle with about 500k records.
CREATE TABLE IF NOT EXISTS `p_transactions` (
`transaction_id` bigint(10) unsigned NOT NULL,
`amount` decimal(19,2) NOT NULL,
`dt` bigint(1) NOT NULL,
`transaction_status` int(1) NOT NULL,
`transaction_type` varchar(15) NOT NULL,
`payment_method` varchar(25) NOT NULL,
`notes` text NOT NULL,
`member_id` int(10) unsigned NOT NULL,
`new_amount` decimal(19,2) NOT NULL,
`paid_amount` decimal(19,2) NOT NULL,
`secret_code` char(40) NOT NULL,
`internal_status` varchar(40) NOT NULL,
`ip_addr` varchar(15) NOT NULL,
`description` text NOT NULL,
`seller_transaction_id` varchar(50) DEFAULT NULL,
`return_url` varchar(255) DEFAULT NULL,
`fail_url` varchar(255) DEFAULT NULL,
`success_url` varchar(255) DEFAULT NULL,
`result_url` varchar(255) DEFAULT NULL,
`user_fee` decimal(19,3) DEFAULT '0.000',
`currency` char(255) DEFAULT 'USD',
`gateway_transaction_id` char(255) DEFAULT NULL,
`load_amount` decimal(19,2) NOT NULL,
`transaction_mode` varchar(1) NOT NULL DEFAULT '',
`p_fee` decimal(19,2) NOT NULL,
`country` varchar(2) NOT NULL,
`email` varchar(255) NOT NULL,
`vat` decimal(19,2) NOT NULL DEFAULT '0.00',
`name` varchar(255) NOT NULL,
`bdate` varchar(255) NOT NULL,
`child_method` varchar(255) NOT NULL,
`processing_fee` decimal(19,2) NOT NULL DEFAULT '0.00',
`flat_fee` varchar(1) NOT NULL DEFAULT 'n',
`user_fee_sum` decimal(19,2) NOT NULL DEFAULT '0.00',
`p_fee_sum` decimal(19,2) NOT NULL DEFAULT '0.00',
`dt_open` bigint(1) NOT NULL DEFAULT '0',
`user_fee_type` varchar(1) NOT NULL DEFAULT 'r',
`custom_gateway_fee` decimal(19,2) NOT NULL DEFAULT '0.00',
`paid_currency` varchar(3) NOT NULL DEFAULT 'USD',
`paid_microtime` bigint(10) unsigned NOT NULL,
`check_ballance` varchar(1) NOT NULL DEFAULT 'n',
PRIMARY KEY (`transaction_id`),
KEY `member_id` (`member_id`),
KEY `payment_method` (`payment_method`),
KEY `child_method` (`child_method`),
KEY `check_ballance` (`check_ballance`),
KEY `dt` (`dt`),
KEY `transaction_type` (`transaction_type`),
KEY `paid_microtime` (`paid_microtime`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
When I execute query
SELECT *
FROM `p_transactions`
WHERE dt >= 1517443200
AND dt <= 1523404799
AND member_id = 2051
ORDER BY `paid_microtime` DESC
LIMIT 50;
it runs 0,000 sec. (+ 0,016 sec. network)
but if I add to query this condition AND transaction_status = 7
SELECT *
FROM `p_transactions`
WHERE dt >= 1517443200
AND dt <= 1523404799
AND member_id = 2051
AND transaction_status = 7
ORDER BY `paid_microtime` DESC
LIMIT 50
query run 12,938 sec. (+ 0,062 sec. network)
Please help me to find out the reason of such behavior
PS. There was index on transaction_status and it increased execution time even more.
Add a suitable index, such as:
ON payzoff_transactions (member_id, dt)
or
ON payzoff_transactions (member_id, dt, transaction_status)
We want member_id column as the leading column in the index, because of the equality comparison, and we expect the result to be a substantially smaller subset of the entire table. We want dt column after that, because of the "range scan" on that.
Including additional columns in the index may allow MySQL to check that condition using values from the index, without a visit/lookup of the row in the underlying table pages.
Either of these indexes would be suitable for both of the queries shown in the question.
Use EXPLAIN to see the execution plan... which index is being used.
There's really no getting around the "Using filesort" operation, since we're pulling a small subset of the entire table.
(If we were pulling the entire table (or a huge subset), we might be able to avoid an expensive sort operation with an access plan that pulls rows in reverse index order, with that has an index with leading column of paid_microtime.)
For the original query have these
INDEX(member_id, dt)
INDEX(member_id, paid_microtime)
For the secondary query, have
INDEX(transaction_status, member_id, dt)
INDEX(transaction_status, member_id, paid_microtime)
Without getting into the details of the distribution of the data values, we cannot explain why one query so much slower; however, my 4 indexes should make both queries run faster most of the time.
More discussion of how I came up with those indexes (and why (member_id, dt, transaction_status) is not so good): http://mysql.rjweb.org/doc.php/index_cookbook_mysql

Slow update Query

When i try to update my torrents table (My torrent site permits to share only open source stuff) with the following query
UPDATE `torrents` SET `leech` = '0', `seed` = '1' WHERE `id` = '26784'
It take approximaty 0.5 seconds to update a table which contains only 20,000 records . My other queries are executed in less than 0.0478s (SELECT queries)
CREATE TABLE IF NOT EXISTS `torrents` (
`id` int(11) NOT NULL,
`hash_info` varchar(255) NOT NULL,
`category_slug` varchar(255) NOT NULL,
`name` varchar(255) NOT NULL,
`size` bigint(20) NOT NULL,
`age` int(11) NOT NULL,
`description` text NOT NULL,
`trackers` longtext NOT NULL,
`magnet` longtext,
`files` longtext,
`parent_category` int(11) NOT NULL,
`category` int(11) NOT NULL,
`publish_date` int(11) DEFAULT NULL,
`uploader` int(11) NOT NULL,
`seed` int(11) DEFAULT '0',
`leech` int(11) DEFAULT '0',
`file_key` varchar(255) DEFAULT NULL,
`comments_count` int(11) DEFAULT '0'
) ENGINE=InnoDB AUTO_INCREMENT=26816 DEFAULT CHARSET=latin1;
Any solution ?
Lookups based on the indexed columns are much faster than the lookups on the non-indexed columns. This behavior can be case more visible with the growing amount of the data.
Create an index on Id column and check if it helps you improve the performance of the query.
id is declared to be an integer. So, first your comparison should be to an integer not a string:
UPDATE `torrents`
SET `leech` = '0', `seed` = '1'
WHERE `id` = 26784;
Second, you need an index on the id. You can create one by:
create index idx_torrents_id on torrents(id);
Alternatively, make it a primary key in the table.