MySQL - Any addtional indexes would speed up this query? - mysql

I see that my query does full table scan and takes a lot of time. I heard that making indexes would speed this up and I have added some to the tables. Is there any other indexes I should create to make this query faster?
My query is:
SELECT p.id, n.people_type_id, n.full_name, n.post, p.nick,
p.key_name, p.email, p.internal_user_id FROM email_routing e
JOIN people_emails p ON p.id=e.receiver_email_id
JOIN people n ON n.id = p.people_id
WHERE e.message_id = 897360 AND e.basket=1
Here is the EXPLAIN result:
EXPLAIN SELECT p.id, n.people_type_id, n.full_name, n.post, p.nick,
p.key_name, p.email, p.internal_user_id FROM email_routing e
JOIN people_emails p ON p.id=e.receiver_email_id
JOIN people n ON n.id = p.people_id
WHERE e.message_id = 897360 AND e.basket=1
id select_type table partitions type possible_keys key key_len ref rows filtered Extra
1 SIMPLE n NULL ALL PRIMARY NULL NULL NULL 1 100.00 NULL
1 SIMPLE p NULL ALL PRIMARY NULL NULL NULL 3178 10.00 Using where; Using join buffer (Block Nested Loop)
1 SIMPLE e NULL ref bk1 bk1 4 server.p.id 440 1.00 Using where; Using
And here are the tables strucutre:
SHOW CREATE TABLE people_emails;
CREATE TABLE `people_emails` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`nick` varchar(255) NOT NULL,
`email` varchar(255) NOT NULL,
`key_name` varchar(255) NOT NULL,
`people_id` int(11) NOT NULL,
`status` int(11) NOT NULL DEFAULT '0',
`activity` int(11) NOT NULL,
`internal_user_id` int(11) NOT NULL,
PRIMARY KEY (`id`),
FULLTEXT KEY `email` (`email`)
) ENGINE=MyISAM AUTO_INCREMENT=22114 DEFAULT CHARSET=utf8
SHOW CREATE TABLE email_routing;
CREATE TABLE `email_routing` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`message_id` int(11) NOT NULL,
`sender_email_id` int(11) NOT NULL,
`receiver_email_id` int(11) NOT NULL,
`basket` int(11) NOT NULL,
`status` int(11) NOT NULL,
`popup` int(11) NOT NULL DEFAULT '0',
`tm` int(11) NOT NULL DEFAULT '0',
KEY `id` (`id`),
KEY `bk1` (`receiver_email_id`,`status`,`sender_email_id`,`message_id`,`basket`),
KEY `bk2` (`sender_email_id`,`tm`)
) ENGINE=InnoDB AUTO_INCREMENT=1054618 DEFAULT CHARSET=utf8
SHOW CREATE TABLE people;
CREATE TABLE `people` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`fname` varchar(255) CHARACTER SET cp1251 NOT NULL,
`lname` varchar(255) CHARACTER SET cp1251 NOT NULL,
`patronymic` varchar(255) CHARACTER SET cp1251 NOT NULL,
`gender` tinyint(1) NOT NULL,
`full_name` varchar(255) NOT NULL DEFAULT ' ',
`category` int(11) NOT NULL,
`people_type_id` int(255) DEFAULT NULL,
`tags` varchar(255) CHARACTER SET cp1251 NOT NULL,
`job` varchar(255) CHARACTER SET cp1251 NOT NULL,
`post` varchar(255) CHARACTER SET cp1251 NOT NULL,
`profession` varchar(255) CHARACTER SET cp1251 DEFAULT NULL,
`zip` varchar(16) CHARACTER SET cp1251 NOT NULL,
`country` int(11) DEFAULT NULL,
`region` varchar(10) NOT NULL,
`city` varchar(255) CHARACTER SET cp1251 NOT NULL,
`address` varchar(255) CHARACTER SET cp1251 NOT NULL,
`address_date` date DEFAULT NULL,
`inner` tinyint(4) NOT NULL,
`contact_through` varchar(255) DEFAULT '',
`next_call` date NOT NULL,
`additional` text CHARACTER SET cp1251 NOT NULL,
`user_id` int(11) NOT NULL,
`changed` datetime NOT NULL,
`status` int(11) DEFAULT NULL,
`nick` varchar(255) DEFAULT NULL,
`birthday` date DEFAULT NULL,
`last_update_ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`area` text NOT NULL,
`reviewed_` tinyint(4) NOT NULL,
`phones_old` text NOT NULL,
`post_sticker` text NOT NULL,
`permissions` int(120) NOT NULL DEFAULT '0',
`internal_user_id` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `most_used` (`category`,`status`,`city`,`lname`,`next_call`),
KEY `registrars` (`category`,`status`,`contact_through`,`next_call`),
FULLTEXT KEY `lname` (`lname`),
FULLTEXT KEY `fname` (`fname`),
FULLTEXT KEY `mname` (`patronymic`),
FULLTEXT KEY `Full Name` (`full_name`)
) ENGINE=MyISAM AUTO_INCREMENT=415009 DEFAULT CHARSET=utf8
How to choose columns for building indexes, should I pick text columnts too or that only will work with numer columnts

The table email_routing seem to have 1054618 rows .
And you try to find one row , by message_id .
e.message_id = 897360
BUT message_id must be indexed to speed-up the query .
message_id is part of the index bk1 , but this is not enough because message_id is not the first columns of the index .

email_routing needs
INDEX ( message_id, basket, -- first, in either order
receiver_email_id ) -- for "covering"
Your bk1 starts with receiver_email_id; this is not nearly as good.
Include column(s) in WHERE that are tested with =.
Include other columns from WHERE, GROUP BY, and ORDER BY (none in your case); the order is important, but beyond the scope of this discussion.
Include any other columns of the same table used anywhere in the query -- this is to make it a "covering" index. But don't bother if this would lead to more than, say, 5 columns or would involve TEXT, which cannot be in an index.
Then move on to the other tables. In both JOINs, it seems that they would be hit by their PRIMARY KEYs (JOIN x ON x.id = ...)
More discussion: Cookbook for creating indexes
On other issues...
You really should move to InnoDB. As of 5.6, it includes FULLTEXT, but there are some differences. In particular, you may need more fulltext indexes. For example, MATCH(lname, fname) requires FULLTEXT(lname, fname).
Do you really want to stick to cp1251? It limits your internalization mostly to English, Russian, Bulgarian, Serbian and Macedonian. And it is unclear how well FULLTEXT (MyISAM or InnoDB) will work with those non-English languages.
INTs are always 4 bytes; consider using smaller versions.
Is there really only one people? The Optimizer decided that was the best table to start with, but it wasn't. I'm hoping my improved index on email_routing will trick it into starting with that table -- which will definitely be optimal.

Related

Why is this SQL query extremely slow?

UPDATE: seems that the Query Cache is not activated, otherwise the performance would be much better from the second execution of the same query, since MySQL should have cached it already. I have MariaDB version: 10.2.41-
Protocol version: 10 , indeed the Query Cache is disabled by default starting with MariaDB 10.1.7.
Is it a good idea to enable it on my VPS config? Or i better upgrade it first?
I have a MySQL query performing very slowly on a CentOS VPS with 1GB RAM and 1 CPU Core.
The query takes 400 - 900 ms to execute (!!), which i want to figure out how to optimize.
It's over several tables, some of them pretty huge (Wordpress' wp_postmeta for example).
But apparently all columns involved are indexed properly, so i'm asking for advice on what to do, or if the only optimization possible is to upgrade the VPS.
Thanks in advance!
Here is the query and the EXPLAIN output
EXPLAIN SELECT c.post_id AS wc_variation_id,v.original_price_rand,v.price,c.meta_value AS color,s.meta_value AS size,bp.tier,t.discount_percent,MAX(IFNULL(tag.extra_price_tag,0)) AS extra_price_tag,MAX(IFNULL(lab.extra_price_label,0)) AS extra_price_label,
i.uri,i.branded,i.design_cost,ib.uri AS uri_back,ib.branded AS branded_back,ib.design_cost AS design_cost_back
FROM wp_postmeta c
JOIN brandly_tiers_bp_balances bp
JOIN brandly_tiers t ON t.role_name=bp.tier
JOIN wix_images i
JOIN wix_images_back ib
JOIN wix_imported_product_variations v ON v.wc_id=c.post_id
LEFT JOIN brandly_reseller_tags rt ON rt.user_id=bp.user_id
LEFT JOIN brandly_reseller_branded_packaging_sizes tag ON tag.size_label=rt.size_label
LEFT JOIN brandly_reseller_labels rl ON rl.user_id=bp.user_id
LEFT JOIN brandly_reseller_branded_packaging_sizes lab ON lab.size_label=rl.size_label
JOIN wp_postmeta s ON s.post_id=c.post_id
WHERE c.post_id IN (SELECT DISTINCT(wc_id) FROM wix_imported_product_variations WHERE wc_product_id = 181189)
AND s.meta_key = 'attribute_pa_size'
AND c.meta_key = 'attribute_pa_colour'
AND i.wix_variation_id=v.wix_id
AND ib.wix_variation_id=v.wix_id
AND v.store_id=36
AND bp.user_id=11
GROUP BY c.post_id
ORDER BY color,size
Output:
id
select_type
table
type
possible_keys
key
key_len
ref
rows
Extra
1
PRIMARY
bp
const
PRIMARY,tier
PRIMARY
8
const
1
Using temporary; Using filesort
1
PRIMARY
t
const
PRIMARY
PRIMARY
52
const
1
1
PRIMARY
wix_imported_product_variations
ref
wc_product_id,wc_id
wc_product_id
8
const
3
Start temporary
1
PRIMARY
rt
ref
PRIMARY
PRIMARY
8
const
5
Using index
1
PRIMARY
tag
eq_ref
PRIMARY
PRIMARY
32
brandly_xvfkg.rt.size_label
1
Using where
1
PRIMARY
c
ref
post_id,meta_key
post_id
8
brandly_xvfkg.wix_imported_product_variations.wc_id
14
Using index condition; Using where; End temporary
1
PRIMARY
v
ref
PRIMARY,store_id,wc_id
wc_id
8
brandly_xvfkg.c.post_id
1
Using index condition; Using where
1
PRIMARY
ib
ref
wix_variation_id
wix_variation_id
103
brandly_xvfkg.v.wix_id
3
1
PRIMARY
i
ref
wix_variation_id
wix_variation_id
103
brandly_xvfkg.v.wix_id
3
1
PRIMARY
s
ref
post_id,meta_key
post_id
8
brandly_xvfkg.c.post_id
14
Using where
1
PRIMARY
rl
ref
PRIMARY
PRIMARY
8
const
5
Using index
1
PRIMARY
lab
eq_ref
PRIMARY
PRIMARY
32
brandly_xvfkg.rl.size_label
1
Using where
EDIT: The subquery takes 0.0005 seconds , so it's not the culprit
EDIT2: the wp_postmeta table has 1709642 records and it's MyISAM:
SHOW CREATE TABLE `wp_postmeta`;
CREATE TABLE `wp_postmeta` (
`meta_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`post_id` bigint(20) unsigned NOT NULL DEFAULT 0,
`meta_key` varchar(255) DEFAULT NULL,
`meta_value` longtext DEFAULT NULL,
PRIMARY KEY (`meta_id`),
KEY `post_id` (`post_id`),
KEY `meta_key` (`meta_key`(191))
) ENGINE=MyISAM AUTO_INCREMENT=3945402 DEFAULT CHARSET=utf8
EDIT3: actually all tables are MyISAM except wix_images, wix_images_back, wix_imported_product_variations which are innoDB
EDIT4: just converted all tables to innoDB. The performance is better now, it takes 200-400 ms. Still slow though.
EDIT5: EXPLAIN output changed a bit now, after switching the tables to innoDB. See https://codebeautify.org/htmlviewer/y225302bb
EDIT6: here are the definitions of the tables involved
CREATE TABLE `wp_postmeta` (
`meta_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`post_id` bigint(20) unsigned NOT NULL DEFAULT 0,
`meta_key` varchar(255) DEFAULT NULL,
`meta_value` longtext DEFAULT NULL,
PRIMARY KEY (`meta_id`),
KEY `post_id` (`post_id`),
KEY `meta_key` (`meta_key`(191))
) ENGINE=InnoDB AUTO_INCREMENT=3945402 DEFAULT CHARSET=utf8
CREATE TABLE `brandly_tiers_bp_balances` (
`user_id` bigint(20) NOT NULL,
`bp_balance` bigint(20) NOT NULL DEFAULT 0,
`bp_pending_balance` bigint(20) NOT NULL DEFAULT 0,
`tier` varchar(50) COLLATE latin1_general_ci NOT NULL DEFAULT 'Earth',
`tier_change_ts` bigint(20) DEFAULT NULL,
PRIMARY KEY (`user_id`),
KEY `tier` (`tier`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci
CREATE TABLE `brandly_tiers` (
`role_name` varchar(50) COLLATE latin1_general_ci NOT NULL,
`bp_threshold` int(11) NOT NULL DEFAULT 0,
`discount_percent` smallint(6) NOT NULL,
`bp_percent_branded_orders` smallint(6) NOT NULL,
`bp_percent_bulk_orders` smallint(6) NOT NULL,
`bp_percent_courier` smallint(6) NOT NULL,
`bp_percent_bulk_branded_packaging` smallint(6) NOT NULL,
`bp_percent_monthly_subscription_fee` smallint(6) NOT NULL,
`bulk_order_minimum_quantity` int(11) NOT NULL DEFAULT 10,
`bp_bulk_order_cap` int(11) NOT NULL DEFAULT 7500,
`tier_monthly_subscription_price` int(11) NOT NULL DEFAULT 0,
PRIMARY KEY (`role_name`),
UNIQUE KEY `bp_threshold` (`bp_threshold`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci
CREATE TABLE `wix_images` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`filename` text COLLATE latin1_general_ci NOT NULL,
`uri` longtext COLLATE latin1_general_ci NOT NULL,
`background_image_uri` longtext COLLATE latin1_general_ci NOT NULL,
`branded` tinyint(1) DEFAULT 0,
`wix_id` varchar(100) COLLATE latin1_general_ci DEFAULT NULL,
`wix_product_id` varchar(100) COLLATE latin1_general_ci DEFAULT NULL,
`wix_variation_id` varchar(100) COLLATE latin1_general_ci DEFAULT NULL,
`design_cost` decimal(18,2) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `wix_id` (`wix_id`),
KEY `wix_product_id` (`wix_product_id`),
KEY `wix_variation_id` (`wix_variation_id`),
KEY `branded` (`branded`),
KEY `background_image_uri` (`background_image_uri`(8))
) ENGINE=InnoDB AUTO_INCREMENT=10680 DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci
CREATE TABLE `wix_images_back` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`filename` text COLLATE latin1_general_ci NOT NULL,
`uri` longtext COLLATE latin1_general_ci NOT NULL,
`background_image_uri` longtext COLLATE latin1_general_ci NOT NULL,
`branded` tinyint(1) DEFAULT 0,
`wix_id` varchar(100) COLLATE latin1_general_ci DEFAULT NULL,
`wix_product_id` varchar(100) COLLATE latin1_general_ci DEFAULT NULL,
`wix_variation_id` varchar(100) COLLATE latin1_general_ci DEFAULT NULL,
`design_cost` decimal(18,2) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `wix_id` (`wix_id`),
KEY `wix_product_id` (`wix_product_id`),
KEY `wix_variation_id` (`wix_variation_id`),
KEY `branded` (`branded`),
KEY `background_image_uri` (`background_image_uri`(8))
) ENGINE=InnoDB AUTO_INCREMENT=7733 DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci
CREATE TABLE `wix_imported_product_variations` (
`wix_id` varchar(100) COLLATE latin1_general_ci NOT NULL,
`store_id` int(11) NOT NULL,
`wc_id` bigint(20) NOT NULL,
`wc_product_id` bigint(20) NOT NULL,
`sku` varchar(100) COLLATE latin1_general_ci NOT NULL,
`price` decimal(18,2) NOT NULL,
`original_price_rand` decimal(18,2) NOT NULL,
`has_markup` tinyint(1) DEFAULT 1,
`wix_image_id` varchar(1000) COLLATE latin1_general_ci NOT NULL,
`imported_at` datetime DEFAULT NULL,
`tmp_design_image` text COLLATE latin1_general_ci DEFAULT NULL,
`tmp_branded` tinyint(1) DEFAULT NULL,
`tmp_design_cost` decimal(18,2) DEFAULT NULL,
`tmp_design_image_back` text COLLATE latin1_general_ci DEFAULT NULL,
`tmp_branded_back` tinyint(1) DEFAULT NULL,
`tmp_design_cost_back` decimal(18,2) DEFAULT NULL,
PRIMARY KEY (`wix_id`),
KEY `sku` (`sku`),
KEY `store_id` (`store_id`),
KEY `has_markup` (`has_markup`),
KEY `imported_at` (`imported_at`),
KEY `wc_product_id` (`wc_product_id`),
KEY `wc_id` (`wc_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci
CREATE TABLE `brandly_reseller_tags` (
`user_id` bigint(20) NOT NULL,
`size_label` varchar(30) COLLATE latin1_general_ci NOT NULL,
`image_front1` mediumtext COLLATE latin1_general_ci NOT NULL,
`image_back1` mediumtext COLLATE latin1_general_ci NOT NULL,
`image_front2` mediumtext COLLATE latin1_general_ci NOT NULL,
`image_back2` mediumtext COLLATE latin1_general_ci NOT NULL,
`image_front3` mediumtext COLLATE latin1_general_ci NOT NULL,
`image_back3` mediumtext COLLATE latin1_general_ci NOT NULL,
`default_index` int(11) DEFAULT NULL,
`design_cost` decimal(18,2) DEFAULT NULL,
PRIMARY KEY (`user_id`,`size_label`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci
CREATE TABLE `brandly_reseller_labels` (
`user_id` bigint(20) NOT NULL,
`size_label` varchar(30) COLLATE latin1_general_ci NOT NULL,
`image_front1` mediumtext COLLATE latin1_general_ci NOT NULL,
`image_back1` mediumtext COLLATE latin1_general_ci NOT NULL,
`image_front2` mediumtext COLLATE latin1_general_ci NOT NULL,
`image_back2` mediumtext COLLATE latin1_general_ci NOT NULL,
`image_front3` mediumtext COLLATE latin1_general_ci NOT NULL,
`image_back3` mediumtext COLLATE latin1_general_ci NOT NULL,
`default_index` int(11) DEFAULT NULL,
`design_cost` decimal(18,2) DEFAULT NULL,
PRIMARY KEY (`user_id`,`size_label`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci
CREATE TABLE `brandly_reseller_branded_packaging_sizes` (
`size_label` varchar(30) COLLATE latin1_general_ci NOT NULL,
`size1_cm` int(11) NOT NULL,
`size2_cm` int(11) NOT NULL,
`size3_cm` int(11) NOT NULL,
`is_bag` tinyint(1) DEFAULT NULL,
`max_weight_kg` decimal(18,2) NOT NULL,
`extra_price` decimal(18,2) NOT NULL,
`extra_price_gift` decimal(18,2) NOT NULL,
`extra_price_tag` decimal(18,2) NOT NULL,
`extra_price_label` decimal(18,2) NOT NULL,
PRIMARY KEY (`size_label`),
KEY `size1_cm` (`size1_cm`),
KEY `size2_cm` (`size2_cm`),
KEY `size3_cm` (`size3_cm`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci
EDIT7:
moved the ON conditions from the JOINs to the WHERE. Execution time 200-400ms like before.
EXPLAIN SELECT c.post_id AS wc_variation_id,v.original_price_rand,v.price,c.meta_value AS color,s.meta_value AS size,bp.tier,t.discount_percent,MAX(IFNULL(tag.extra_price_tag,0)) AS extra_price_tag,MAX(IFNULL(lab.extra_price_label,0)) AS extra_price_label,
i.uri,i.branded,i.design_cost,ib.uri AS uri_back,ib.branded AS branded_back,ib.design_cost AS design_cost_back
FROM wp_postmeta c
JOIN brandly_tiers_bp_balances bp
JOIN brandly_tiers t
JOIN wix_images i
JOIN wix_images_back ib
JOIN wix_imported_product_variations v
LEFT JOIN brandly_reseller_tags rt ON rt.user_id=bp.user_id
LEFT JOIN brandly_reseller_branded_packaging_sizes tag ON tag.size_label=rt.size_label
LEFT JOIN brandly_reseller_labels rl ON rl.user_id=bp.user_id
LEFT JOIN brandly_reseller_branded_packaging_sizes lab ON lab.size_label=rl.size_label
JOIN wp_postmeta s
WHERE c.post_id IN (SELECT DISTINCT(wc_id) FROM wix_imported_product_variations WHERE wc_product_id = 181189)
AND s.meta_key = 'attribute_pa_size'
AND c.meta_key = 'attribute_pa_colour'
AND i.wix_variation_id=v.wix_id
AND ib.wix_variation_id=v.wix_id
AND v.store_id=36
AND bp.user_id=11
AND s.post_id=c.post_id
AND t.role_name=bp.tier
AND v.wc_id=c.post_id
GROUP BY c.post_id
ORDER BY color,size
EXPLAIN output
id
select_type
table
type
possible_keys
key
key_len
ref
rows
Extra
1
PRIMARY
bp
const
PRIMARY,tier
PRIMARY
8
const
1
Using temporary; Using filesort
1
PRIMARY
t
const
PRIMARY
PRIMARY
52
const
1
1
PRIMARY
wix_imported_product_variations
ref
wc_product_id,wc_id
wc_product_id
8
const
3
Start temporary
1
PRIMARY
rt
ref
PRIMARY
PRIMARY
8
const
5
Using index
1
PRIMARY
tag
eq_ref
PRIMARY
PRIMARY
32
brandly_xvfkg.rt.size_label
1
Using where
1
PRIMARY
c
ref
post_id,meta_key
post_id
8
brandly_xvfkg.wix_imported_product_variations.wc_id
8
Using index condition; Using where; End temporary
1
PRIMARY
v
ref
PRIMARY,store_id,wc_id
wc_id
8
brandly_xvfkg.c.post_id
1
Using index condition; Using where
1
PRIMARY
ib
ref
wix_variation_id
wix_variation_id
103
brandly_xvfkg.v.wix_id
3
1
PRIMARY
i
ref
wix_variation_id
wix_variation_id
103
brandly_xvfkg.v.wix_id
3
1
PRIMARY
s
ref
post_id,meta_key
post_id
8
brandly_xvfkg.v.wc_id
8
Using index condition; Using where
1
PRIMARY
rl
ref
PRIMARY
PRIMARY
8
const
5
Using index
1
PRIMARY
lab
eq_ref
PRIMARY
PRIMARY
32
brandly_xvfkg.rl.size_label
1
Using where
Spending more money on a bigger server is unlikely to do you much good.
200-400ms for a query like this is not bad. There's probably no SQL magic that will get it to under 100ms; it's doing a lot. Often the words extremely slow refer to queries that take minutes or longer.
MAX(IFNULL(column, 0)) yields precisely the same result as MAX(column) as long as all values of column are positive.. Aggregate functions skip over nulls. You might be able to simplify the two places you use that expression.
Your query is hard to reason about. If this were my query I would rewrite it to get rid of the short table aliases, and rather spell them out. And, I would rewrite it so every JOIN (both LEFT and ordinary) had an ON-clause with it. Notice that ON-clauses can be compound:
FROM a
JOIN wp_postmeta ON a.id = wp_postmeta.postid
AND wp_postmeta.meta_key = 'attribute_pa_size'
That's a query pattern we WordPress hackers instantly recognize.
For ordinary JOINs, moving ON-clauses to WHERE-clauses has no effect on execution plans.
Your join types, in your plan, are all "const", "eq-ref", or "ref". That is very good.
There's a field-proven rework of WordPress's postmeta table indexes. Your postmeta table is big enough -- almost four megarows -- that you probably can use it. It changes the primary key to a compound key that meets the requirement of your query pattern. Yours is a common pattern in WooCommerce installations. It is this for MySQL 5.7 and above.
ALTER TABLE wp_postmeta
ADD UNIQUE KEY meta_id (meta_id),
DROP PRIMARY KEY,
ADD PRIMARY KEY (post_id, meta_key, meta_id),
DROP KEY meta_key,
ADD KEY meta_key (meta_key, meta_value(32), post_id, meta_id),
ADD KEY meta_value (meta_value(32), meta_id),
DROP KEY post_id;
The Index WP MySQL For Speed plugin is designed to install this set of indexes for you, and do its best with MySQL / MariaDB 5.5 and up.
I don't see any big problems with indexing on your non-WordPress tables. There may be some opportunities for compound covering indexes, but, again, your query is hard to read.
95% of systems are better off with the Query Cache turned off. Remember that any modification to a table causes the QC to purge all entries for that table. If there are a lot of writes to a table, that could be a lot of CPU cycles for purging.
If you have only 1GB of RAM, there is not much room for the QC; use the room for InnoDB's buffer_pool instead. When you changed from MyISAM did you lower key_buffer_size and raise innodb_buffer_pool_size?
Some of these composite indexes may help:
c: INDEX(post_id, meta_key, meta_value)
v: INDEX(wix_id, store_id, wc_id, original_price_rand, price)
s: INDEX(meta_key, post_id, meta_value)
bp: INDEX(user_id, tier)
t: INDEX(role_name, discount_percent)
tag: INDEX(size_label, extra_price_tag)
lab: INDEX(size_label, extra_price_label)
i: INDEX(wix_variation_id, uri, branded, design_cost)
ib: INDEX(wix_variation_id, uri, branded, design_cost)
rt: INDEX(user_id, size_label)
rl: INDEX(user_id, size_label)
When adding a composite index, DROP index(es) with the same leading columns.
That is, when you have both INDEX(a) and INDEX(a,b), toss the former.

Improving the performance of a MYSQL query with a one-to-many relationship

I have a query in my DB that is taking 25 seconds to return results, which is way too long. It seems like it should be pretty simple. Two tables; the main table (document) is a standard table with some data columns, the join table is a mapping table with only two columns (parent_id, division_id). Previously there wasn't an index on the mapping table so I added one and that changed the "explain" to include the index but doesn't seem to have had an impact on the performance.
The query looks like this:
explain SELECT DISTINCT doc.*
FROM document doc
LEFT JOIN multi_division_mapper divisions ON doc.id = divisions.parent_id
WHERE doc.clientId = 'SOME_GUID'
AND (divisions.division_id IS NULL OR divisions.division_id IN ('SOME_GUID'));
and the results of explain are:
Total number of rows in document: 6720
Total number of rows in mapper: 6173
From what I've been able to gather I need to improve either the "type" or the "extra" to make the query faster. What can I do here?
Create table statements:
CREATE TABLE `document` (
`id` varchar(36) NOT NULL,
`addedBy` varchar(255) DEFAULT NULL,
`addedDate` datetime NOT NULL,
`editedBy` varchar(255) DEFAULT NULL,
`editedDate` datetime NOT NULL,
`deleted` bit(1) DEFAULT NULL,
`clientId` varchar(36) NOT NULL,
`departmentId` varchar(36) DEFAULT NULL,
`documentParentId` varchar(36) DEFAULT NULL,
`documentParent` varchar(50) DEFAULT NULL,
`fileId` varchar(255) DEFAULT NULL,
`fileUrl` varchar(600) DEFAULT NULL,
`documentName` varchar(500) NOT NULL,
`displayName` varchar(255) NOT NULL,
`documentId` varchar(45) DEFAULT NULL,
`notes` varchar(1000) DEFAULT NULL,
`visibility` varchar(45) NOT NULL DEFAULT 'PRIVATE',
`documentType` varchar(45) NOT NULL,
`restrictDelete` bit(1) NOT NULL,
`customData` text,
`releaseDate` datetime NOT NULL,
`expirationDate` datetime NOT NULL,
`isApproved` bit(1) NOT NULL DEFAULT b'0',
`userSupplier` varchar(36) DEFAULT NULL,
`complianceCertificateId` varchar(36) DEFAULT NULL,
`Status` varchar(50) DEFAULT 'NEUTRAL',
PRIMARY KEY (`id`),
KEY `idx_client` (`clientId`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `multi_division_mapper` (
`parent_id` varchar(36) NOT NULL,
`division_id` varchar(36) NOT NULL,
PRIMARY KEY (`parent_id`,`division_id`),
KEY `idx_parent` (`parent_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
I was able to get a more favorable EXPLAIN report in a test by creating the following index:
ALTER TABLE multi_division_mapper
DROP INDEX idx_parent,
ADD INDEX (division_id, parent_id);
I also dropped idx_parent because it's redundant; it's a prefix of the primary key.
id
select_type
table
partitions
type
possible_keys
key
key_len
ref
rows
filtered
Extra
1
SIMPLE
doc
NULL
ref
idx_client
idx_client
110
const
1
100.00
Using temporary
1
SIMPLE
divisions
NULL
ref
PRIMARY,division_id
division_id
38
const
1
100.00
Using where; Using index; Distinct
The type: ref is better than type: index.
The query I tested is slightly different, but I believe it returns the same result:
SELECT DISTINCT doc.*
FROM document doc
LEFT JOIN multi_division_mapper divisions
ON doc.id = divisions.parent_id AND divisions.division_id in ('SOME_GUID')
WHERE doc.clientId = 'SOME_GUID'

MYSQL INNER JOIN is slow with index

this is my simple inner join:
SELECT
SUM(ASSNZ.assenzeDidattiche) AS TotaleAssenze,
SUM(ASSNZ.ore) AS totale_parziale,
FLOOR(((SUM(ASSNZ.assenzeDidattiche) / SUM(ASSNZ.ore)) * 100)) AS andamento,
MAX(ASSNZ.dataLezione) AS ultima_lezione,
ASSNZ.idServizio,
ASSNZ.idUtente
FROM
ciac_corsi_assenze AS ASSNZ
INNER JOIN
ciac_serviziAcquistati_ITA AS ACQ
ON ACQ.idContatto = ASSNZ.idUtente
AND ACQ.idServizio = ASSNZ.idServizio
AND ACQ.stato_allievo <> 'ritirato'
GROUP BY
ASSNZ.idServizio,
ASSNZ.idUtente
table "ASSNZ" has 213886 rows with index "idUtente", "idServizio"
table "ACQ" has 8950 rows with index "idContatto", "idServizio"
ASSNZ table:
CREATE TABLE `ciac_corsi_assenze` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`idUtente` int(11) DEFAULT NULL,
`idServizio` int(11) DEFAULT NULL,
`idCorso` int(11) DEFAULT NULL,
`idCalendario` int(11) DEFAULT NULL,
`modalita` varchar(255) DEFAULT NULL,
`ore` int(11) DEFAULT NULL,
`assenzeDidattiche` float DEFAULT NULL,
`assenzeAmministrative` float DEFAULT NULL,
`dataLezione` date DEFAULT NULL,
`ora_inizio` varchar(8) DEFAULT NULL,
`ora_fine` varchar(8) DEFAULT NULL,
`dataFineStage` date DEFAULT NULL,
`giustificata` varchar(128) DEFAULT NULL,
`motivazione` longtext,
`grouped` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `idUtente` (`idUtente`) USING BTREE,
KEY `idServizio` (`idServizio`) USING BTREE,
KEY `dataLezione` (`dataLezione`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=574582 DEFAULT CHARSET=utf8;
ACQ table:
CREATE TABLE `ciac_serviziacquistati_ita` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`idServizio` int(11) NOT NULL,
`idContatto` int(11) NOT NULL,
`idAzienda` int(11) NOT NULL,
`idSede` int(11) NOT NULL,
`tipoPersona` int(11) NOT NULL,
`num_registro` int(11) NOT NULL,
`codice` varchar(255) CHARACTER SET latin1 DEFAULT NULL,
`dal` date NOT NULL,
`al` date NOT NULL,
`ore` int(11) NOT NULL,
`costoOrario` decimal(10,0) NOT NULL,
`annoFormativo` varchar(128) CHARACTER SET latin1 NOT NULL,
`stato_attuale` int(11) NOT NULL,
`datore_attuale` int(11) NOT NULL,
`stato_allievo` varchar(64) CHARACTER SET latin1 NOT NULL DEFAULT 'corsista',
`data_ritiro` date DEFAULT NULL,
`crediti_formativi` int(11) NOT NULL,
`note` longtext CHARACTER SET latin1 NOT NULL,
`valore_economico` decimal(10,2) NOT NULL,
`dataInserimento` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idServizio` (`idServizio`) USING BTREE,
KEY `idAzienda` (`idAzienda`) USING BTREE,
KEY `idContatto` (`idContatto`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=9542 DEFAULT CHARSET=utf8;
this is my EXPLAIN of the select
Now because the query is slow, during 1.5s / 2.0s??
Something wrong?
UPDATE
added new index (with the John Bollinger's answer) to the table ciac_corsi_assenze:
PRIMARY KEY (`id`),
KEY `dataLezione` (`dataLezione`) USING BTREE,
KEY `test` (`idUtente`,`idServizio`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=574582 DEFAULT CHARSET=utf8;
added new index to the table ciac_serviziAcquistati_ITA:
PRIMARY KEY (`id`),
KEY `idAzienda` (`idAzienda`) USING BTREE,
KEY `test2` (`idContatto`,`idServizio`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=9542 DEFAULT CHARSET=utf8;
New EXPLAIN:
But it's always slow :(
Your tables have separate indexes on various columns of interest, but MySQL will use at most one index per table to perform your query. This particular query would probably be sped by table ciac_corsi_assenze having an index on (idUtente, idServizio) (and such an index would supersede the existing one on (idUtente) alone). That should allow MySQL to avoid sorting the result rows to perform the grouping, and it will help more in performing the join than any of the existing indexes do.
The query would probably be sped further by table ciac_serviziAcquistati_ITA having an index on (idContatto, idServizio), or even on (idContatto, idServizio, ritirato). Either of those would supersede the existing index on just (idContatto).
John went the right direction. However the order of columns in the composite index needs changing.
For the GROUP BY, this order is needed (on ASSNZ):
INDEX(idServizio, idUtente)
(and that should replace KEY(idServizio), but not KEY(idUtente))
Then ACQ needs, in this order:
INDEX(idContatto, idServizio, stato_allievo)
replacing only KEY(idContatto).

MySQL use separate indices for JOIN and GROUP BY

I am trying to execute following query
SELECT
a.sessionID AS `sessionID`,
firstSeen, birthday, gender,
isAnonymous, LanguageCode
FROM transactions AS trx
INNER JOIN actions AS a ON a.sessionID = trx.SessionID
WHERE a.ActionType = 'PURCHASE'
GROUP BY trx.TransactionNumber
Explain provides the following output
1 SIMPLE trx ALL TransactionNumber,SessionID NULL NULL NULL 225036 Using temporary; Using filesort
1 SIMPLE a ref sessionID sessionID 98 infinitiExport.trx.SessionID 1 Using index
The problem is that I am trying to use one field for join and different field for GROUP BY.
How can I tell MySQL to use different indices for same table?
CREATE TABLE `transactions` (
`SessionID` varchar(32) NOT NULL DEFAULT '',
`date` datetime DEFAULT NULL,
`TransactionNumber` varchar(32) NOT NULL DEFAULT '',
`CustomerECommerceTrackID` int(11) DEFAULT NULL,
`SKU` varchar(45) DEFAULT NULL,
`AmountPaid` double DEFAULT NULL,
`Currency` varchar(10) DEFAULT NULL,
`Quantity` int(11) DEFAULT NULL,
`Name` tinytext NOT NULL,
`Category` varchar(45) NOT NULL DEFAULT '',
`customerInfoXML` text,
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`id`),
KEY `TransactionNumber` (`TransactionNumber`),
KEY `SessionID` (`SessionID`)
) ENGINE=InnoDB AUTO_INCREMENT=212007 DEFAULT CHARSET=utf8;
CREATE TABLE `actions` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`sessionActionDate` datetime DEFAULT NULL,
`actionURL` varchar(255) DEFAULT NULL,
`sessionID` varchar(32) NOT NULL DEFAULT '',
`ActionType` varchar(64) DEFAULT NULL,
`CustomerID` int(11) DEFAULT NULL,
`IPAddressID` int(11) DEFAULT NULL,
`CustomerDeviceID` int(11) DEFAULT NULL,
`customerInfoXML` text,
PRIMARY KEY (`id`),
KEY `ActionType` (`ActionType`),
KEY `CustomerDeviceID` (`CustomerDeviceID`),
KEY `sessionID` (`sessionID`)
) ENGINE=InnoDB AUTO_INCREMENT=15042833 DEFAULT CHARSET=utf8;
Thanks
EDIT 1: My indexes were broken, I had to add (SessionID, TransactionNumber) index to transactions table, however now, when I try to include trx.customerInfoXML table mysql stops using index
EDIT 2 Another answer does not really solved my problem because it's not standard sql syntax and generally not a good idea to force indices.
For ORM users such syntax is a unattainable luxury.
EDIT 3 I updated my indices and it solved the problem, see EDIT 1

Data Structure causing impossible joins

Tables:
nodes
data_texts
data_profiles
data_locations
data_profiles
data_media
data_products
data_metas
categories
tags
categories_nodes
tags_nodes
This question is a generalized question and is on the back of another question
Explanation:
Each of the "data" tables has a node_id that refers back to the id of the nodes table (hasMany/belongsTo association).
A "Node" can be anything - a TV Show, a Movie, a Person, an Article...etc (all generated via a CMS, so the user can control what type of "Nodes" they want).
When pulling data, I want to be able to query against certain fields. For example if they do a search, I want to be able to pull nodes that have data_texts.title = '%george%' or order by the datetime field in data_locations.
The problem is, when I do a join on all seven data tables (or more), the query has to hit so many combined rows that it just times out (even with a nearly empty database.... total 200 rows across the entire database).
I realize I can determine IF I need a join depending on what I'm doing - but even with five or six joins (once the database gets to 10k+ records), it's going to be horribly slow, if it works at all. Per this question, the query I'm using just doing a join on these tables times out completely.
Each node can have multiple rows of each data type (for multi-language reasons among others).
I'm completely defeated - I'm at the point where I think I need to restructure the entire thing, but don't have the time for that. I've thought about combining all into one table, but aren't sure how....etc
nodes
CREATE TABLE `nodes` (
`id` CHAR(36) NOT NULL,
`name` VARCHAR(100) NOT NULL,
`slug` VARCHAR(100) NOT NULL,
`node_type_id` CHAR(36) NOT NULL,
`site_id` CHAR(36) NOT NULL,
`created` DATETIME NOT NULL,
`modified` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `nodeTypeId` (`node_type_id`),
INDEX `slug` (`slug`),
INDEX `nodeId` (`id`)
)
COLLATE='latin1_swedish_ci'
ENGINE=MyISAM;
data_texts:
CREATE TABLE `data_texts` (
`id` CHAR(36) NOT NULL,
`title` VARCHAR(250) NULL DEFAULT NULL,
`subtitle` VARCHAR(500) NULL DEFAULT NULL,
`content` LONGTEXT NULL,
`byline` VARCHAR(250) NULL DEFAULT NULL,
`language_id` CHAR(36) NULL DEFAULT NULL,
`foreign_key` CHAR(36) NULL DEFAULT NULL,
`model` VARCHAR(40) NULL DEFAULT NULL,
`node_id` CHAR(36) NULL DEFAULT NULL,
`created` DATETIME NOT NULL,
`modified` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `nodeId` (`node_id`),
INDEX `languageId_nodeId` (`language_id`, `node_id`),
INDEX `foreignKey_model` (`foreign_key`, `model`)
)
COLLATE='utf8_general_ci'
ENGINE=MyISAM;
data_profiles
CREATE TABLE `data_profiles` (
`id` CHAR(36) NOT NULL,
`name` VARCHAR(80) NULL DEFAULT NULL,
`email_personal` VARCHAR(100) NULL DEFAULT NULL,
`email_business` VARCHAR(100) NULL DEFAULT NULL,
`email_other` VARCHAR(100) NULL DEFAULT NULL,
`title` VARCHAR(100) NULL DEFAULT NULL,
`description` LONGTEXT NULL,
`prefix` VARCHAR(40) NULL DEFAULT NULL,
`phone_home` VARCHAR(40) NULL DEFAULT NULL,
`phone_business` VARCHAR(40) NULL DEFAULT NULL,
`phone_mobile` VARCHAR(40) NULL DEFAULT NULL,
`phone_other` VARCHAR(40) NULL DEFAULT NULL,
`foreign_key` CHAR(36) NULL DEFAULT NULL,
`model` VARCHAR(40) NULL DEFAULT NULL,
`node_id` CHAR(36) NULL DEFAULT NULL,
`language_id` CHAR(36) NULL DEFAULT NULL,
`created` DATETIME NOT NULL,
`modified` DATETIME NOT NULL,
`user_id` CHAR(36) NULL DEFAULT NULL,
PRIMARY KEY (`id`),
INDEX `nodeId` (`node_id`),
INDEX `languageId_nodeId` (`node_id`, `language_id`),
INDEX `foreignKey_model` (`foreign_key`, `model`)
)
COLLATE='latin1_swedish_ci'
ENGINE=MyISAM;
categories
CREATE TABLE `categories` (
`id` CHAR(36) NOT NULL,
`name` VARCHAR(100) NOT NULL,
`node_type_id` CHAR(36) NOT NULL,
`site_id` CHAR(36) NOT NULL,
`slug` VARCHAR(100) NOT NULL,
`created` DATETIME NOT NULL,
`modified` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `nodeTypeId` (`node_type_id`),
INDEX `slug` (`slug`)
)
COMMENT='Used to categorize nodes'
COLLATE='utf8_general_ci'
ENGINE=MyISAM;
categories_nodes
CREATE TABLE `categories_nodes` (
`id` CHAR(36) NOT NULL,
`category_id` CHAR(36) NOT NULL,
`node_id` CHAR(36) NOT NULL,
PRIMARY KEY (`id`),
INDEX `categoryId_nodeId` (`category_id`, `node_id`)
)
COLLATE='utf8_general_ci'
ENGINE=MyISAM;
node_tags
CREATE TABLE `node_tags` (
`id` CHAR(36) NOT NULL,
`name` VARCHAR(40) NOT NULL,
`site_id` CHAR(36) NOT NULL,
`created` DATETIME NOT NULL,
`modified` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `siteId` (`site_id`)
)
COLLATE='utf8_general_ci'
ENGINE=MyISAM;
nodes_node_tags
CREATE TABLE `nodes_node_tags` (
`id` CHAR(36) NOT NULL,
`node_id` CHAR(36) NOT NULL,
`node_tag_id` CHAR(36) NOT NULL,
PRIMARY KEY (`id`),
INDEX `node_id_node_tag_id` (`node_id`, `node_tag_id`)
)
COLLATE='utf8_general_ci'
ENGINE=MyISAM;
MySQL:
SELECT `Node`.`id`, `Node`.`name`, `Node`.`slug`, `Node`.`node_type_id`, `Node`.`site_id`, `Node`.`created`, `Node`.`modified`
FROM `mysite`.`nodes` AS `Node`
LEFT JOIN `mysite`.`data_date_times` AS `DataDateTime` ON (`DataDateTime`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_locations` AS `DataLocation` ON (`DataLocation`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_media` AS `DataMedia` ON (`DataMedia`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_metas` AS `DataMeta` ON (`DataMeta`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_profiles` AS `DataProfile` ON (`DataProfile`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_products` AS `DataProduct` ON (`DataProduct`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_texts` AS `DataText` ON (`DataText`.`node_id` = `Node`.`id`)
WHERE 1=1
GROUP BY `Node`.`id`
Firstly, try InnoDB, not MyISAM.
Secondly, remove the group by, see how well it runs then, and how many rows are involved. Shouldn't be that many, but it's interesting.
You don't need the 'nodeId' index on node (as you already have it as a primary key). Again, shouldn't make any difference.
The where clause is irrelevant. You can remove it with no effect one way or another.
Thirdly, well, something is seriously broken.
Have a quick look on how to start profiling (e.g. http://dev.mysql.com/doc/refman/5.0/en/show-profile.html) , and run a profile command to see where all the time is going. Post it here if it doesn't immediately show that something is broken.
I'm unfortunately not in a position where I can do any tests right now. I'll just throw out some ideas. I might be able to do some tests later.
Be suspicious of different collations.
Some of your ids are useless. For example, you should drop the column categories_nodes.id, and put a primary key constraint on {category_id, node_id} instead.
Be suspicious of any design that requires joining all the tables at run time. There are better ways.
Use innodb and foreign key constraints.