Below is the table structure. MySql table contains 11 million data. All the data may be inserted into the table in one day only.
I tried executing the below query and it took 5-7 mins to finish. I hav created the index based on the query and even it is taking 3-5 mins to finish.Index is being used.
What should i do to increase the query performance ? any suggestions ?
CREATE TABLE `service_collection_data` (
`service_name` varchar(512) NOT NULL,
`device_id` bigint(20) NOT NULL,
`dataset_name` varchar(1024) NOT NULL,
`dataset_title` varchar(1024) NOT NULL,
`dataset_type` varchar(32) NOT NULL,
`dataset_command` varchar(4000) DEFAULT NULL,
`tag_name` varchar(4000) DEFAULT NULL,
`command_status` varchar(256) NOT NULL,
`command_result` longtext,
`command_error` longtext,
`command_context` longtext,
`collection_time` timestamp NOT NULL
DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
KEY `device_id` (`device_id`),
KEY `service_tag_command_index` (`service_name`(300),`tag_name`(200),
`command_status`(200),`device_id`),
CONSTRAINT `service_collection_data_ibfk_1`
FOREIGN KEY (`device_id`)
REFERENCES `nodes` (`id`) ON DELETE CASCADE
)
ENGINE=InnoDB DEFAULT CHARSET=latin1
Below is query is executed
SELECT COUNT(DISTINCT device_id)
FROM service_collection_data
WHERE service_name='NOS'
AND tag_name='Config'
AND command_status = 'Successful'
As debugging scenarios, what is the speed of:
1)
SELECT COUNT(device_id)
FROM service_collection_data
2)
SELECT COUNT(DISTINCT device_id)
FROM service_collection_data
3)
SELECT COUNT(*)
FROM service_collection_data
WHERE service_name='NOS'
AND tag_name='Config'
AND command_status = 'Successful'
4)
SELECT COUNT(DISTINCT device_id)
FROM
(SELECT service_name, tag_name, command_status, device_id
FROM service_collection_data
GROUP BY service_name, tag_name, command_status, device_id
WHERE service_name='NOS'
AND tag_name='Config'
AND command_status = 'Successful')
Other:
Can device_id be specified as primary key (thus guaranteeing a distinct value)?
Can service_name, tag_name, and command_status be changed to integer foreign keys?
Related
I am trying to update my table:
UPDATE offers t1
SET t1.deleted_at = NOW()
WHERE t1.id
NOT IN
(
SELECT f.id
FROM (
SELECT ean, MIN(net_price) as minprice
FROM offers group BY ean
)
as x inner join offers as f on f.ean = x.ean and f.net_price = x.minprice
);
can anybody help me with this issue?
I realized that I needed to solve the issue through left join, but did not understand how to apply it in my case. Thank you.
Here is create table:
CREATE TABLE `offers` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`net_price` decimal(10,2) unsigned DEFAULT NULL,
`retail_price` decimal(10,2) unsigned DEFAULT NULL,
`supplier_id` bigint unsigned DEFAULT NULL,
`manufacturer_id` bigint unsigned DEFAULT NULL,
`stock` int DEFAULT NULL,
`ean` varchar(14) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`mnn_id` bigint unsigned DEFAULT NULL,
`source_id` bigint unsigned NOT NULL,
`nds` smallint unsigned DEFAULT NULL,
`to_export` tinyint(1) DEFAULT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
`deleted_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `offers_supplier_id_foreign` (`supplier_id`),
KEY `offers_manufacturer_id_foreign` (`manufacturer_id`),
KEY `offers_mnn_id_foreign` (`mnn_id`),
KEY `offers_source_id_foreign` (`source_id`),
CONSTRAINT `offers_manufacturer_id_foreign` FOREIGN KEY (`manufacturer_id`) REFERENCES `manufacturers` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `offers_mnn_id_foreign` FOREIGN KEY (`mnn_id`) REFERENCES `mnns` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `offers_source_id_foreign` FOREIGN KEY (`source_id`) REFERENCES `sources` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `offers_supplier_id_foreign` FOREIGN KEY (`supplier_id`) REFERENCES `suppliers` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=25762 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
The error you are getting is
You can't specify target table 't1' for update in FROM clause
This is a problem specific to MySQL. You cannot directly access the updated table in a subquery, you need a sub subquery :-)
Replace
inner join offers as f
by
inner join (select * from offers) as f
hence to get the query working.
You don't need the join by the way. You can just write
UPDATE offers
SET deleted_at = NOW()
WHERE (ean, net_price) NOT IN
(
SELECT ean, MIN(net_price)
FROM (select * from offers) o
GROUP BY ean
);
Another way to write this (update all rows for which exists a lower price for the same EAN):
UPDATE offers t1
SET t1.deleted_at = NOW()
WHERE EXISTS
(
SELECT NULL
FROM (SELECT * FROM offers) t2
WHERE t2.ean = t1.ean
AND t2.net_price < t1.net_price
);
Demo: https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=bfa6d74a72d40fa445ea5fa9f95803d1
By the way: You may want to add the condition deleted_at IS NULL to your update statement in order not to update rows that are already logically deleted, if such rows exists.
I have a accounts table. every account creation I am pushing treeRight id and treeLeft id into account_device_tree table.
Now I have more than 10M accounts under first parent account. when I select all the subaccouts it is taking more than a min to execute.
my query is given below
select *
FROM
accounts acc
JOIN
account_device_tree ON acc.tree_id = account_device_tree.tree_id
WHERE
(acc.account_id = 1 OR (account_device_tree.tree_left >= 1 AND account_device_tree.tree_right <= 748534))
I need to optimize as much as possible.
schema of account_device_tree
CREATE TABLE `account_device_tree` (
`tree_id` int(11) NOT NULL AUTO_INCREMENT,
`tree_left` int(11) NOT NULL,
`tree_right` int(11) NOT NULL,
PRIMARY KEY (`tree_id`),
KEY `tree_left` (`tree_left`),
KEY `tree_right` (`tree_right`)
) ENGINE=InnoDB AUTO_INCREMENT=388173 DEFAULT CHARSET=latin1
accounts table Schema
CREATE TABLE `accounts` (
`account_id` int(11) NOT NULL AUTO_INCREMENT,
`parent_id` int(11) NOT NULL,
`name` varchar(64) NOT NULL,
`tree_id` int(11) NOT NULL,
PRIMARY KEY (`account_id`),
UNIQUE KEY `tree_id` (`tree_id`),
KEY `name` (`name`),
KEY `idx_parent_id` (`parent_id`)
) ENGINE=InnoDB AUTO_INCREMENT=389739 DEFAULT CHARSET=latin1
Hard to suggest something without table structures and query plan..
But rewritting the query into UNION ALL instead of using OR tends to optimize beter assuming the correct indexes are in the table.
select *
FROM
accounts acc
JOIN
account_user_device_tree ON acc.tree_id = account_user_device_tree.tree_id
WHERE
acc.account_id = 1
UNION ALL
select *
FROM
accounts acc
JOIN
account_user_device_tree ON acc.tree_id = account_user_device_tree.tree_id
WHERE
account_user_device_tree.tree_left >= 1
AND
account_user_device_tree.tree_right <= 748534
I am trying to improve performance of following query which took 93.2 sec to execute query below:
SELECT year(date), month(date), `country_name_name`,
CEIL(count(res.`user_xmpp_login`) /DAY(LAST_DAY(date))) as avgUser,
CEIL(count(res.user)/DAY(LAST_DAY(date))) as avgPurchase
FROM
( SELECT DATE(`user_registration_timestamp`) as date,
user_country,
NULL as user, `user_xmpp_login`
FROM users
WHERE `user_registration_timestamp` >= "2015-01-01 00:00:00"
AND `user_registration_timestamp` < "2016-01-01 00:00:00"
UNION ALL
SELECT DATE(`ts`) as date, user_country, user, NULL as `user_xmpp_login`
FROM purchase_log p
INNER JOIN users u ON u.`user_xmpp_login` = p.`user`
WHERE `ts` >= "2015-01-01 00:00:00"
AND `ts` < "2016-01-01 00:00:00"
AND result in ('ok', 'cancelled', 'pending')
) AS res
INNER JOIN countries c ON c.`country_id` = res.`user_country`
INNER JOIN country_names cn
ON (cn.`country_name_country` = c.`country_id`
AND cn.`country_name_language` = 'en')
GROUP BY 1,2,3
ORDER BY 4 DESC,5 DESC, 3 ASC;
Explain command shows:
And structure of each table is:
purchase table:
CREATE TABLE `purchase` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`result` varchar(32) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
KEY `iuser` (`user`),
) ENGINE=InnoDB AUTO_INCREMENT=12710221 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
users table:
CREATE TABLE `users` (
`user_id` int(11) NOT NULL AUTO_INCREMENT,
`user_country` int(11) DEFAULT NULL,
`user_xmpp_login` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`user_registration_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`user_id`),
UNIQUE KEY `user_xmpp_login_UNIQUE` (`user_xmpp_login`),
KEY `user_country_FK` (`user_country`),
KEY `user_registration_timestamp` (`user_registration_timestamp`),
CONSTRAINT `users_country_FK` FOREIGN KEY (`user_country`)
REFERENCES `countries` (`country_id`) ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=33504745 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
countries table
CREATE TABLE `countries` (
`country_id` int(11) NOT NULL AUTO_INCREMENT,
`country_code` varchar(2) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`country_id`),
) ENGINE=InnoDB AUTO_INCREMENT=508 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
country names
CREATE TABLE `country_names` (
`country_name_id` int(11) NOT NULL AUTO_INCREMENT,
`country_name_country` int(11) NOT NULL,
`country_name_language` char(2) COLLATE utf8_unicode_ci NOT NULL,
`country_name_name` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`country_name_id`),
UNIQUE KEY `country_name_country_language_UNIQUE`
(`country_name_country`,`country_name_language`),
KEY `country_name_language` (`country_name_language`),
CONSTRAINT `country_name_country` FOREIGN KEY (`country_name_country`)
REFERENCES `countries` (`country_id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=45793 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Is there any recommendations?
If you time each subquery, I think you will find users is the slowest component.
The purchase_log subquery can probably be improved with this "covering" INDEX(result, ts, user).
Combine the two "country" tables!. Use CHAR(2) CHARACTER SET ascii for the PRIMARY KEY and the JOINs to other tables. It is only 2 bytes, unlike INT, which is 4 bytes and VARCHAR..., which is 3 bytes (in this case).
You mention ts, but I don't see where it is coming from. If it is in purchase_log, then that table needs INDEX(user, ts).
What percentage of the users involved 2015? If it is more than about 20%, the INDEX(user_registration_timestamp) won't help.
Consider: Get rid of PRIMARY KEY (country_name_id), and promote the UNIQUE key to PRIMARY.
The biggest problem seems to be in your users table. Remember, mysql can only use one index per table for most situations. On your users table, the user_xmpp_login_UNIQUE column has been used to join it to the purchase_log table. There fore, the user_registration_timestamp index is not being used on the comparison involving the timestamp column.
One suggestion is to create a composite index on the user_xmpp_login and user_registration_timestamp columns.
I want to exectute the following query:
SELECT *
FROM `bm_tracking`
WHERE `oid` IN
(SELECT `oid`
FROM `bm_tracking`
GROUP BY `oid` HAVING COUNT(*) >1)
The subquery:
SELECT `oid`
FROM `bm_tracking`
GROUP BY `oid`
HAVING COUNT( * ) >1
executes in 0.0525 secs
The whole query "stucks" (still processing after 3 minutes...). Column oid is indexed.
Table bm_tracking contains around 64k rows.
What could be the reason for this "stuck"?
[Edit: Upon request]
CREATE TABLE `bm_tracking` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`oid` varchar(10) NOT NULL,
`trk_main` varchar(50) NOT NULL,
`tracking` varchar(50) NOT NULL,
`label` text NOT NULL,
`void` int(11) NOT NULL DEFAULT '0',
`created` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `oid` (`oid`),
KEY `trk_main` (`trk_main`),
KEY `tracking` (`tracking`),
KEY `created` (`created`)
) ENGINE=MyISAM AUTO_INCREMENT=63331 DEFAULT CHARSET=latin1
[Execution Plan]
Generally exists EXISTS faster than IN so you can try this and see if it executes better for you
SELECT *
FROM `bm_tracking` bt
WHERE EXISTS
( SELECT 1
FROM `bm_tracking` bt1
WHERE bt.oid = bt1.oid
GROUP BY `oid`
HAVING COUNT(*) >1
)
EDIT:
if you notice from the EXPLAIN you posted... the IN() is considered as a DEPENDENT SUBQUERY which is a correlated subquery... meaning that for every row in the table all rows in the table are pulled and compared... so for example 1000 rows in the table would mean 1000 * 1000 = 1 million comparisons -- thats why its taking such a long time
I have a database with the records near about 2.7 milion . I need to fetch records from that for that i am using the below query
for result
SELECT r3.original_image_title,r3.uuid,r3.original_image_URL FROM `image_attributes` AS r1 INNER JOIN `filenames` as r3 WHERE r1.`uuid` = r3.`uuid` and r3.`status` = 1 and r1.status=1 and (r1.`attribute_name` like "Quvenzhané Wallis%" or r3.original_image_URL like "Quvenzhané Wallis%") group by r3.`uuid` limit 0,20
for total count
SELECT count(DISTINCT(r1.`uuid`)) as count FROM `image_attributes` AS r1 INNER JOIN `filenames` as r3 WHERE r1.`uuid` = r3.`uuid` and r3.`status` = 1 and r1.status=1 and (r1.`attribute_name` like "Quvenzhané Wallis%" or r3.original_image_URL like "Quvenzhané Wallis%")
table structures are as below
CREATE TABLE IF NOT EXISTS `image_attributes` (
`index` int(11) NOT NULL AUTO_INCREMENT,
`attribute_name` text NOT NULL,
`attribute_type` varchar(255) NOT NULL,
`uuid` varchar(255) NOT NULL,
`status` tinyint(1) NOT NULL DEFAULT '1',
PRIMARY KEY (`index`),
KEY `attribute_type` (`attribute_type`),
KEY `uuid` (`uuid`),
KEY `status` (`status`),
KEY `attribute_name` (`attribute_name`(50))
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=2730431 ;
CREATE TABLE IF NOT EXISTS `filenames` (
`index` int(11) NOT NULL AUTO_INCREMENT,
`original_image_title` text NOT NULL,
`original_image_URL` text NOT NULL,
`uuid` varchar(255) NOT NULL,
`status` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`index`),
KEY `uuid` (`uuid`),
KEY `original_image_URL` (`original_image_URL`(50))
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=591967 ;
please suggest me how can i optimize the queries to make the search result faster
I would recommend to you a book called 'High Performance MySql'. There is a section called Optimize databases and queries, or something like that.