Optimize a joined MySQL query - mysql

There is this query that have been bugging me in two days now, it used to work good but now it slows down the entire cluster environment, the query is as seen below:
SELECT userUploads.*,
users_avatar.avatar AS avatar
FROM userUploads
LEFT JOIN users_avatar
ON userUploads.udid = users_avatar.udid
INNER JOIN user_subscription
ON (
user_subscription.sub_1 = 'G:123456789'
AND user_subscription.sub_2 = userUploads.udid
)
WHERE userUploads.platform = 'Private'
AND userUploads.STATUS IN ( 'featured', 'approved' )
ORDER BY userUploads.id DESC
LIMIT 50 OFFSET 0
I would really appreciate if anyone can help out with this query.
Below is the explain of the query:
+----+-------------+-------------------+--------+----------------------+----------+---------+------------------------+------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------------+--------+----------------------+----------+---------+------------------------+------+-----------------------------+
| 1 | SIMPLE | userUploads | range | platform,udid,status | platform | 154 | NULL | 12 | Using where; Using filesort |
| 1 | SIMPLE | users_avatar | eq_ref | PRIMARY | PRIMARY | 182 | Seeds.userUploads.udid | 1 | |
| 1 | SIMPLE | user_subscription | ref | sub_1,sub_2 | sub_1 | 93 | const | 7 | Using where |
+----+-------------+-------------------+--------+----------------------+----------+---------+------------------------+------+-----------------------------+
Thanks in advance
EDIT** show create table can seen below
Below is the show create table for the tables hope you have any ideas dancrumb.
| users_avatar | CREATE TABLE `users_avatar` (
`udid` varchar(60) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
`avatar` varchar(448) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`udid`)
) ENGINE=ndbcluster DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
| userUploads | CREATE TABLE `userUploads` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`bdaha` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`user` varchar(60) COLLATE utf8_unicode_ci DEFAULT NULL,
`direktoren` text COLLATE utf8_unicode_ci,
`filnamnet` varchar(180) COLLATE utf8_unicode_ci DEFAULT NULL,
`karhes` varchar(150) COLLATE utf8_unicode_ci DEFAULT NULL,
`version` char(10) COLLATE utf8_unicode_ci DEFAULT NULL,
`rostat` int(10) DEFAULT NULL,
`stars` int(11) DEFAULT NULL,
`statyn` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`platform` char(30) COLLATE utf8_unicode_ci DEFAULT NULL,
`images` int(2) DEFAULT NULL,
`date` char(10) COLLATE utf8_unicode_ci DEFAULT NULL,
`udid` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`favorirepris` int(8) DEFAULT NULL,
`hikes` char(4) COLLATE utf8_unicode_ci DEFAULT 'no',
`dbn` char(6) COLLATE utf8_unicode_ci DEFAULT NULL,
`timestamp` char(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`comments` int(5) DEFAULT NULL,
`klistret` enum('no','yes') COLLATE utf8_unicode_ci NOT NULL DEFAULT 'no',
PRIMARY KEY (`id`),
KEY `platform` (`platform`,`status`),
KEY `udid` (`udid`),
KEY `hikes` (`hikes`),
KEY `bdaha` (`bdaha`),
KEY `statyn` (`statyn`),
KEY `version` (`version`)
) ENGINE=ndbcluster AUTO_INCREMENT=118831 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
| user_subscription | CREATE TABLE `user_subscription` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`sub_1` varchar(30) COLLATE utf8_unicode_ci DEFAULT NULL,
`sub_2` varchar(30) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `sub_1` (`sub_1`),
KEY `sub_2` (`sub_2`)
) ENGINE=ndbcluster AUTO_INCREMENT=155184 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |

Well, you have a filesort on userUploads which is always slow. You may want to play with indices to remove that. For example, you may want to start with an index on udid, platform, and status.

In general when querying you want to perform the most limiting operations in terms of data returned first so that other operations are only performed on the data that is actually in the results.
In this case try reordering the inner join to user_subscription and the left join to users_avatar. This way it will only attempt to even get the avatar for a user if they are actually in the result set rather than it looking up all the avatars first then filtering based on joins and where clauses.
SELECT userUploads.*,
users_avatar.avatar AS avatar
FROM userUploads
INNER JOIN user_subscription
ON (
user_subscription.sub_1 = 'G:123456789'
AND user_subscription.sub_2 = userUploads.udid
)
LEFT JOIN users_avatar
ON userUploads.udid = users_avatar.udid
WHERE userUploads.platform = 'Private'
AND userUploads.STATUS IN ( 'featured', 'approved' )
ORDER BY userUploads.id DESC
LIMIT 50 OFFSET 0

Related

speed up join between two databases

I have this query and takes 40 seconds, Is there a way to speed up? thank you
SELECT *, last.Date
FROM constant.derogation der
LEFT JOIN variable.last last
ON der.code = last.code
WHERE 1 = 1
AND status != 'removed'
ORDER BY status;
+------+-------------+-------+------+---------------+------+---------+------+--------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------+------+---------------+------+---------+------+--------+----------------------------------------------+
| 1 | SIMPLE | der | ALL | NULL | NULL | NULL | NULL | 318 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | last | ALL | NULL | NULL | NULL | NULL | 250950 | Using where |
+------+-------------+-------+------+---------------+------+---------+------+--------+----------------------------------------------+
This is the structure of both tables, both databases are on the same server.
I will only get a value from Last Table
DLL
| derogation | CREATE TABLE `derogation` (
`xxx` char(10) NOT NULL,
`xxx` varchar(100) DEFAULT NULL,
`code` char(17) DEFAULT NULL,
`xxx` varchar(20) DEFAULT NULL,
`xxx` char(6) DEFAULT NULL,
`xxx` varchar(50) DEFAULT NULL,
`xxx` varchar(200) DEFAULT NULL,
`xxxx` varchar(200) DEFAULT NULL,
`xxxx` varchar(100) DEFAULT NULL,
`xxxx` datetime DEFAULT NULL,
`xxx` varchar(100) DEFAULT NULL,
`xxxx` varchar(20) DEFAULT NULL,
`xxx` varchar(50) DEFAULT NULL,
`xxx` varchar(10) DEFAULT NULL,
`xxx` datetime DEFAULT NULL,
`xxx` varchar(10) DEFAULT NULL,
`xxx` datetime DEFAULT NULL,
`xxx` datetime DEFAULT NULL,
`status` varchar(20) DEFAULT NULL,
`xxx` varchar(1000) DEFAULT NULL,
KEY `code_index_derogation` (`code`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
| Last | CREATE TABLE `Last` (
`code` char(17) DEFAULT NULL,
`xxx` decimal(10,2) DEFAULT NULL,
`Date` datetime DEFAULT NULL,
`xxxx` datetime DEFAULT NULL,
`xxxx` datetime DEFAULT NULL,
`xxxx` text DEFAULT NULL,
`xxxxx` datetime DEFAULT NULL,
`xxxx` char(6) DEFAULT NULL,
KEY `idx_Last_Code` (`code`),
KEY `idx_Last_xxx` (`xxx`),
KEY `code_index` (`code`),
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
you can use temporary tables, staging tables CTE or you can add indexes for the columns of the tables.
But using temporary tables, it's always an excelent choice to accelerate the excecution of the the SQL query.
(The problem is subtle, and often overlooked in this forum.)
Don't mix collations. You are joining on code, but the CHARACTER SET and COLLATION are different between the two tables. Suggest using `ALTER TABLE .. CONVERT TO .. to get them to be the same.
Other tips:
Use CHAR only for truly fixed-length columns. Use VARCHAR otherwise.
Have a PRIMARY KEY on every table.
Don't say LEFT JOIN if you really expect JOIN; it adds confusion to the reader.

how to convert a field to string in a mysql join

I try to join a filed that is a int(13) on to a field that is varchar(50).
If I only use (a.id = b.id) the DESCRIBE says type: ref.
If I use (a.id = CONCAT(b.id)) the DESCRIBE says type: eq_ref. (where b.id is the integer)
The use of CONCAT to cast a field is ugly, so I tried to use CAST() or CONVERT().
If I use (a.id = CAST(b.id AS CHAR(50))) the DESCRIBE says type: ref.
How do I write a correct cast/convert, that gives a eq_ref join?
UPDATE 1:
DESCRIBE SELECT.. with CONCAT
+------+-------------+-----------------------+--------+-----------------------------------+----------------+---------+-------------------------------------+------+----------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-----------------------+--------+-----------------------------------+----------------+---------+-------------------------------------+------+----------------------------------------+
| 1 | SIMPLE | ext_icecat_prodmatch | ref | PRIMARY,our_article_id,product_id | our_article_id | 152 | const | 3016 | Using index condition; Using temporary |
| 1 | SIMPLE | ext_icecat_product | eq_ref | PRIMARY,product_id | PRIMARY | 4 | ext_icecat_prodmatch.product_id | 1 | |
| 1 | SIMPLE | ext_icecat_supplier | eq_ref | PRIMARY | PRIMARY | 4 | ext_icecat_product.supplier_id | 1 | |
| 1 | SIMPLE | products | eq_ref | PRIMARY | PRIMARY | 152 | ext_icecat_prodmatch.our_article_id | 1 | |
| 1 | SIMPLE | partner_product_saved | eq_ref | PRIMARY | PRIMARY | 155 | const,func | 1 | Using where |
| 1 | SIMPLE | category_names | eq_ref | PRIMARY | PRIMARY | 6 | products.category_id,const | 1 | Using where |
+------+-------------+-----------------------+--------+-----------------------------------+----------------+---------+-------------------------------------+------+----------------------------------------+
The Select:
SELECT
partner_product_saved.*,
ext_icecat_product.product_id,
CONCAT(ext_icecat_supplier.name, ' ', ext_icecat_product.name) AS export_product_name,
ext_icecat_product.catid_match AS category_id,
GROUP_CONCAT(ext_icecat_prodmatch.our_article_id) AS oais,
products.file_name,
category_names.category_path
FROM ext_icecat_product
LEFT JOIN ext_icecat_prodmatch USING (product_id)
LEFT JOIN ext_icecat_supplier USING (supplier_id)
LEFT JOIN products USING (our_article_id)
LEFT JOIN partner_product_saved ON (partner_product_saved.partner_id = 29 AND partner_product_saved.product_id = CONCAT(ext_icecat_product.product_id))
LEFT JOIN category_names ON (category_names.category_id = products.category_id AND category_names.language_id = 2)
WHERE ext_icecat_prodmatch.our_article_id = '0EF03850-D25A-1174-BCDC-EC67352010A6'
GROUP BY ext_icecat_product.product_id
ORDER BY NULL;
SHOW CREATE TABLE
CREATE TABLE `partner_product_saved` (
`partner_id` mediumint(8) NOT NULL,
`product_id` varchar(50) CHARACTER SET utf8 NOT NULL,
`product_name` varchar(100) CHARACTER SET utf8 NOT NULL,
`our_article_id` varchar(50) CHARACTER SET utf8 DEFAULT NULL,
`our_category_id` mediumint(8) DEFAULT NULL,
`manufacture_id` mediumint(8) DEFAULT NULL,
`manufacturer_partnr` varchar(255) COLLATE utf8_bin NOT NULL,
`manufacturer_upc` varchar(255) COLLATE utf8_bin NOT NULL,
`image` tinytext COLLATE utf8_bin NOT NULL,
`image_small` tinytext COLLATE utf8_bin NOT NULL,
`image_big` tinytext COLLATE utf8_bin NOT NULL,
`image_200` tinytext COLLATE utf8_bin NOT NULL,
`image_original` tinytext COLLATE utf8_bin NOT NULL,
`image_width` int(11) DEFAULT NULL,
`image_height` int(11) DEFAULT NULL,
`birth` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`last_updated` timestamp NULL DEFAULT NULL,
`saved` tinyint(3) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`partner_id`,`product_id`),
KEY `our_article_id` (`our_article_id`),
KEY `our_category_id` (`our_category_id`),
KEY `manufacture_id` (`manufacture_id`,`manufacturer_partnr`),
KEY `manufacturer_upc` (`manufacturer_upc`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
CREATE TABLE `ext_icecat_product` (
`product_id` int(13) NOT NULL,
`supplier_id` int(13) NOT NULL DEFAULT '0',
`prod_id` varchar(235) COLLATE utf8_bin NOT NULL DEFAULT '',
`prod_id_clean` varchar(255) CHARACTER SET utf8 NOT NULL,
`catid` int(13) NOT NULL DEFAULT '0',
`catid_match` varchar(50) CHARACTER SET utf8 NOT NULL,
`name` varchar(255) CHARACTER SET utf8 NOT NULL,
`name_clean` varchar(255) CHARACTER SET utf8 NOT NULL,
`low_pic` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '',
`high_pic` varchar(255) COLLATE utf8_bin NOT NULL DEFAULT '',
`thumb_pic` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`family_id` int(13) NOT NULL DEFAULT '0',
`low_pic_size` int(13) DEFAULT '0',
`high_pic_size` int(13) DEFAULT '0',
`thumb_pic_size` int(13) DEFAULT '0',
`import_date` datetime NOT NULL,
`release_date` datetime NOT NULL,
`updated` datetime NOT NULL,
`need_update` tinyint(1) NOT NULL DEFAULT '0',
`deleted` tinyint(1) NOT NULL DEFAULT '0',
`keyword` tinyint(1) NOT NULL DEFAULT '0',
`special_match` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`product_id`),
KEY `supplier_id` (`supplier_id`),
KEY `catid` (`catid`),
KEY `prod_id` (`prod_id`),
KEY `product_id` (`product_id`,`prod_id`,`supplier_id`),
KEY `release_Date` (`release_date`),
KEY `prod_id_clean` (`prod_id_clean`),
KEY `name_clean` (`name_clean`),
KEY `need_update` (`need_update`),
KEY `deleted` (`deleted`),
KEY `keyword` (`keyword`),
KEY `catid_2` (`catid`,`import_date`),
KEY `catid_match` (`catid_match`),
KEY `special_match` (`special_match`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
WHERE indexed_column = any_function(any_column) -- can use index. WHERE non_indexed_column = any_function(indexed_column) -- cannot use index.
The difference between ref and eq_ref is minor. I think that eq_ref is where the optimizer decides that there cannot be more than one match, often because of UNIQUE.
WHERE ext_icecat_prodmatch.our_article_id = '0EF03850-D25A-1174-BCDC-EC67352010A6' -- is our_article_id INDEXed? or UNIQUE? Sounds like it is only an INDEX, so multiple rows might ensue. To make it eq_ref, you need UNIQUE. But only if the data supports such. The stats imply there might be 3016 rows with that article_id.
Do not use LEFT unless you need it. Note how the Optimizer turned LEFT JOIN ext_icecat_prodmatch USING (product_id) into JOIN and decided (rightly) to start with ext_icecat_prodmatch.
Back to other discussions...
AND partner_product_saved.product_id = CONCAT(ext_icecat_product.product_id))
can go one way, but not the other. That is, it can efficiently go from eip to pps, but not the other way. And EXPLAIN indicated such with const,func.

Table indexed well but query taking long time to execute?

one of our query is scanning full table and it was taking around
15mins and it was impacted on our application performance. The table
having proper indexes but it was scanning complete table. How to
rewrite or is there any best way indexing?
Query:
select count(u.user_id)
from iflora_user_newsletter_map unm,
users u ,
addresses a
where unm.user_id=u.user_id
and unm.newsletter_id=1
and unm.active=1
and u.user_id=a.user_id
and a.type='billing';\
Explain Plan:
+----+-------------+-------+------+------------------+----------+---------+----------------------+---------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+------+------------------+----------+---------+----------------------+---------+--------------------------+
| 1 | SIMPLE | unm | ref | idx_2243,idx_747 | idx_2243 | 4 | const | 2960628 | Using where |
| 1 | SIMPLE | u | ref | idx_747 | idx_747 | 5 | shopcart.unm.user_id | 1 | Using where; Using index |
| 1 | SIMPLE | a | ref | idx_747 | idx_747 | 4 | shopcart.unm.user_id | 1 | Using where |
+----+-------------+-------+------+------------------+----------+---------+----------------------+---------+--------------------------+
3 rows in set (0.05 sec)
Tables structure:
mysql> show create table iflora_user_newsletter_map\G
*************************** 1. row ***************************
Table: iflora_user_newsletter_map
Create Table: CREATE TABLE `iflora_user_newsletter_map` (
`row_mod` datetime DEFAULT NULL,
`row_create` datetime DEFAULT NULL,
`user_id` int(11) NOT NULL,
`newsletter_id` int(11) NOT NULL,
`active` int(11) DEFAULT NULL,
`date_subscribed` datetime DEFAULT NULL,
`date_unsubscribed` datetime DEFAULT NULL,
`email_preference` int(11) DEFAULT NULL,
`mail_preference` int(11) DEFAULT NULL,
`sms_preference` int(11) DEFAULT NULL,
`phone_preference` int(11) DEFAULT NULL,
UNIQUE KEY `idx_2243` (`newsletter_id`,`user_id`),
KEY `idx_1571` (`date_subscribed`),
KEY `idx_1798` (`date_unsubscribed`),
KEY `idx_747` (`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_bin
1 row in set (0.00 sec)
mysql> show create table users\G
*************************** 1. row ***************************
Table: users
Create Table: CREATE TABLE `users` (
`row_mod` datetime DEFAULT NULL,
`row_create` datetime DEFAULT NULL,
`user_id` int(11) DEFAULT NULL,
`name` varchar(50) COLLATE latin1_bin DEFAULT NULL,
`password` varchar(255) COLLATE latin1_bin DEFAULT NULL,
`salt` varchar(12) COLLATE latin1_bin DEFAULT NULL,
`last_visit` int(11) DEFAULT NULL,
`member_since` int(11) DEFAULT NULL,
`email` varchar(100) COLLATE latin1_bin DEFAULT NULL,
`active` int(11) DEFAULT NULL,
`transient` int(11) DEFAULT NULL,
`fname` varchar(50) COLLATE latin1_bin DEFAULT NULL,
`lname` varchar(50) COLLATE latin1_bin DEFAULT NULL,
`default_address_id` decimal(12,0) DEFAULT NULL,
`opt_in_email` char(1) COLLATE latin1_bin DEFAULT NULL,
`opt_in_email2` char(1) COLLATE latin1_bin DEFAULT NULL,
`opt_in_email3` char(1) COLLATE latin1_bin DEFAULT NULL,
`email_promotions` char(1) COLLATE latin1_bin DEFAULT NULL,
`textonly_email` char(1) COLLATE latin1_bin DEFAULT NULL,
`website_id` int(11) DEFAULT NULL,
`express_checkout` int(11) DEFAULT NULL,
`webstore_id` int(11) DEFAULT NULL,
UNIQUE KEY `idx_747` (`user_id`),
KEY `idx_417` (`name`),
KEY `idx_delete_transient` (`transient`,`last_visit`),
KEY `users_n1` (`email`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_bin
1 row in set (0.00 sec)
mysql> show create table addresses\G
*************************** 1. row ***************************
Table: addresses
Create Table: CREATE TABLE `addresses` (
`row_mod` datetime DEFAULT NULL,
`row_create` datetime DEFAULT NULL,
`address_id` decimal(12,0) NOT NULL,
`user_id` int(11) NOT NULL,
`name` varchar(30) COLLATE latin1_bin DEFAULT NULL,
`type` varchar(20) COLLATE latin1_bin DEFAULT NULL,
`fname` varchar(50) COLLATE latin1_bin DEFAULT NULL,
`lname` varchar(50) COLLATE latin1_bin DEFAULT NULL,
`company` varchar(100) COLLATE latin1_bin DEFAULT NULL,
`email` varchar(100) COLLATE latin1_bin DEFAULT NULL,
`street_address` varchar(100) COLLATE latin1_bin DEFAULT NULL,
`address2` varchar(100) COLLATE latin1_bin DEFAULT NULL,
`city` varchar(30) COLLATE latin1_bin DEFAULT NULL,
`state` varchar(40) COLLATE latin1_bin DEFAULT NULL,
`zip` varchar(10) COLLATE latin1_bin DEFAULT NULL,
`country` varchar(20) COLLATE latin1_bin DEFAULT NULL,
`day_phone` varchar(20) COLLATE latin1_bin DEFAULT NULL,
`day_phone_ext` varchar(10) COLLATE latin1_bin DEFAULT NULL,
`evening_phone` varchar(20) COLLATE latin1_bin DEFAULT NULL,
`mobile_phone` varchar(20) COLLATE latin1_bin DEFAULT NULL,
`fax` varchar(15) COLLATE latin1_bin DEFAULT NULL,
`location_name` varchar(40) COLLATE latin1_bin DEFAULT NULL,
`personal_notes` varchar(100) COLLATE latin1_bin DEFAULT NULL,
`complete_flag` varchar(1) COLLATE latin1_bin DEFAULT 'N',
`credit_card_number` varchar(20) COLLATE latin1_bin DEFAULT NULL,
`credit_card_type` varchar(10) COLLATE latin1_bin DEFAULT NULL,
`credit_card_exp_month` int(11) DEFAULT NULL,
`credit_card_exp_year` int(11) DEFAULT NULL,
`birthday` datetime DEFAULT NULL,
`title` varchar(10) COLLATE latin1_bin DEFAULT NULL,
`street_address2` varchar(100) COLLATE latin1_bin DEFAULT NULL,
`district` varchar(30) COLLATE latin1_bin DEFAULT NULL,
`verified` int(11) DEFAULT NULL,
`email_type` varchar(1) COLLATE latin1_bin DEFAULT NULL,
`location_type` varchar(20) COLLATE latin1_bin DEFAULT NULL,
`birth_day` varchar(4) COLLATE latin1_bin DEFAULT NULL,
`birth_month` varchar(4) COLLATE latin1_bin DEFAULT NULL,
`birth_year` varchar(4) COLLATE latin1_bin DEFAULT NULL,
`skip_queries` int(11) DEFAULT NULL,
`has_qas_results` int(11) DEFAULT NULL,
`qas_queried` int(11) DEFAULT NULL,
`debit_card_issue_number` varchar(6) COLLATE latin1_bin DEFAULT NULL,
`credit_card_start_year` varchar(4) COLLATE latin1_bin DEFAULT NULL,
`credit_card_start_month` varchar(4) COLLATE latin1_bin DEFAULT NULL,
`in_the_name_of` varchar(30) COLLATE latin1_bin DEFAULT NULL,
`parameters` varchar(1024) COLLATE latin1_bin DEFAULT NULL,
UNIQUE KEY `idx_1042` (`address_id`),
KEY `idx_747` (`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_bin
1 row in set (0.00 sec)
#morilo,
here is explain after adding the index
create index idx_748 on iflora_user_newsletter_map(user_id,newsletter_id,active);
explain Plan
mysql> explain select count(u.user_id) from iflora_user_newsletter_map unm, users u , addresses a where unm.user_id=u.user_id and unm.newsletter_id=1 and unm.active=1 and u.user_id=a.user_id and a.type='billing';
+----+-------------+-------+--------+-----------------------------------+----------+---------+--------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------------+----------+---------+--------------------------+------+-------------+
| 1 | SIMPLE | u | index | idx_747 | idx_747 | 5 | NULL | 2575 | Using index |
| 1 | SIMPLE | a | ref | idx_747,idx_ads | idx_747 | 4 | shopcart.u.user_id | 1 | Using where |
| 1 | SIMPLE | unm | eq_ref | idx_2243,idx_news,idx_747,idx_748 | idx_2243 | 8 | const,shopcart.u.user_id | 1 | Using where |
+----+-------------+-------+--------+-----------------------------------+----------+---------+--------------------------+------+-------------+
3 rows in set (0.00 sec)
Explain Plan use force index
mysql> explain select count(u.user_id) from iflora_user_newsletter_map unm, users u , addresses a where unm.user_id=u.user_id and unm.newsletter_id=1 and unm.active=1 and u.user_id=a.user_id and a.type='billing';
+----+-------------+-------+--------+-----------------------------------+----------+---------+--------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-----------------------------------+----------+---------+--------------------------+------+-------------+
| 1 | SIMPLE | u | index | idx_747 | idx_747 | 5 | NULL | 2575 | Using index |
| 1 | SIMPLE | a | ref | idx_747,idx_ads | idx_747 | 4 | shopcart.u.user_id | 1 | Using where |
| 1 | SIMPLE | unm | eq_ref | idx_2243,idx_news,idx_747,idx_748 | idx_2243 | 8 | const,shopcart.u.user_id | 1 | Using where |
+----+-------------+-------+--------+-----------------------------------+----------+---------+--------------------------+------+-------------+
3 rows in set (0.00 sec)
This is your query written with proper join syntax:
select count(u.user_id)
from iflora_user_newsletter_map unm join
users u
on unm.user_id = u.user_id join
addresses a
on u.user_id = a.user_id
where unm.newsletter_id = 1 and unm.active = 1 and a.type = 'billing';
This query would potentially benefit from the following indexes:
addresses(user_id, type)
iflora_user_newsletter_map(newsletter_id, active, user_id)
or:
addresses(type, user_id)
iflora_user_newsletter_map(active, user_id, newsletter_id)
I do not see the columns in the where clause in the indexes in your table. Which of these is better depends on how much data in each table is being used for the indexes. The larger table appears to be unm, so I would try the first set of indexes first.
The select is scanning all the rows because of the conditions "unm.newsletter_id=1 and unm.active=1": those fields are not indexed and therefore, the engine has to scan all rows.
include those fields too in the index 'user_id' to solve the problem.
I recommend changing:
KEY idx_747 (user_id)
To
KEY idx_747 (user_id,newsletter_id,active)

Optimize query -- It is using filesort. (Explain, query, and show create table included)

I am wondering why my query isn't using the index "created_2" which covers all the fields used in the query. It seems to use a filesort. What are the rules for picking an index?
Query:
SELECT * FROM (`stories`) WHERE `image_full_url` != '' AND `order` != 0 ORDER BY `created` DESC, `order` DESC LIMIT 5
Create Table:
| stories | CREATE TABLE `stories` (
`id` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`news_type` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`title` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`created` datetime DEFAULT NULL,
`author` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`author_title` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`image_caption` text COLLATE utf8_unicode_ci,
`image_credit` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`image_full_url` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`body` text COLLATE utf8_unicode_ci,
`summary` text COLLATE utf8_unicode_ci,
`external_url` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`order` int(10) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `news_type` (`news_type`),
KEY `created` (`created`),
KEY `news_type_2` (`news_type`,`created`),
KEY `created_2` (`created`,`image_full_url`,`order`),
KEY `image_full_url` (`image_full_url`,`order`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
Explain:
mysql> explain SELECT * FROM (`stories`) WHERE `image_full_url` != '' AND `order` != 0 ORDER BY `created` DESC, `order` DESC LIMIT 5;
+----+-------------+---------+-------+----------------+----------------+---------+------+------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------+-------+----------------+----------------+---------+------+------+-----------------------------+
| 1 | SIMPLE | stories | range | image_full_url | image_full_url | 768 | NULL | 25 | Using where; Using filesort |
+----+-------------+---------+-------+----------------+----------------+---------+------+------+-----------------------------+
1 row in set (0.00 sec)
The full set of rules are here, and state that an order by can't use the index if it references non-consecutive parts of the index. If you change the index from (created,image_full_url,order) to (created,order,image_full_url), that will probably let it be used.

More Odd MySQL Behavior - Query Optimization Help

We have a central login that we use to support multiple websites. To store our users' data we have an accounts table which stores each user account and then users tables for each site for site specific information. We also have a simple connections table which stores the connections between users.
We noticed that one query that is joining the tables on their primary key user_id is executing slowly. I'm hoping that some SQL expert out there can explain why it's using WHERE to search the users_site1 table and suggest how we can optimize it. Here is the slow query & the explain results:
mysql> explain select a.username,a.first_name,a.last_name,a.organization_name,a.organization,a.city,a.state,a.zip,a.country,a.profile_photo,a.facebook_id,a.twitter_id,u.reviews from accounts a join users_site1 u ON a.user_id=u.user_id where a.user_id IN (select cid2 from connections where cid1=10001006 AND type="MM" AND status="A") OR a.user_id IN (select cid1 from connections where cid2=10001006 AND type="MM" AND status="A") order by RAND() LIMIT 4;
+----+--------------------+-------------+--------+-------------------+---------+---------+-----------------------+-------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-------------+--------+-------------------+---------+---------+-----------------------+-------+----------------------------------------------+
| 1 | PRIMARY | u | ALL | PRIMARY | NULL | NULL | NULL | 79783 | Using where; Using temporary; Using filesort |
| 1 | PRIMARY | a | eq_ref | PRIMARY | PRIMARY | 4 | exampledb.u.user_id | 1 | |
| 3 | DEPENDENT SUBQUERY | connections | ref | PRIMARY,cid1,cid2 | cid2 | 6 | const,const | 2 | Using where |
| 2 | DEPENDENT SUBQUERY | connections | ref | PRIMARY,cid1,cid2 | cid1 | 6 | const,const | 1 | Using where |
+----+--------------------+-------------+--------+-------------------+---------+---------+-----------------------+-------+----------------------------------------------+
4 rows in set (0.00 sec)
Here are the definitions for each table:
CREATE TABLE `accounts` (
`user_id` int(9) unsigned NOT NULL AUTO_INCREMENT,
`username` varchar(40) DEFAULT NULL,
`facebook_id` bigint(15) unsigned DEFAULT NULL,
`facebook_username` varchar(30) DEFAULT NULL,
`password` varchar(20) DEFAULT NULL,
`profile_photo` varchar(100) DEFAULT NULL,
`first_name` varchar(40) DEFAULT NULL,
`middle_name` varchar(40) DEFAULT NULL,
`last_name` varchar(40) DEFAULT NULL,
`suffix_name` char(3) DEFAULT NULL,
`organization_name` varchar(100) DEFAULT NULL,
`organization` tinyint(1) unsigned DEFAULT NULL,
`address` varchar(200) DEFAULT NULL,
`city` varchar(40) DEFAULT NULL,
`state` varchar(20) DEFAULT NULL,
`zip` varchar(10) DEFAULT NULL,
`province` varchar(40) DEFAULT NULL,
`country` int(3) DEFAULT NULL,
`latitude` decimal(11,7) DEFAULT NULL,
`longitude` decimal(12,7) DEFAULT NULL,
`phone` varchar(20) DEFAULT NULL,
`sex` char(1) DEFAULT NULL,
`birthday` date DEFAULT NULL,
`about_me` varchar(2000) DEFAULT NULL,
`activities` varchar(300) DEFAULT NULL,
`website` varchar(100) DEFAULT NULL,
`email` varchar(150) DEFAULT NULL,
`referrer` int(4) unsigned DEFAULT NULL,
`referredid` int(9) unsigned DEFAULT NULL,
`verify` int(6) DEFAULT NULL,
`status` char(1) DEFAULT 'R',
`created` datetime DEFAULT NULL,
`verified` datetime DEFAULT NULL,
`activated` datetime DEFAULT NULL,
`network` datetime DEFAULT NULL,
`deleted` datetime DEFAULT NULL,
`logins` int(6) unsigned DEFAULT '0',
`api_logins` int(6) unsigned DEFAULT '0',
`last_login` datetime DEFAULT NULL,
`last_update` datetime DEFAULT NULL,
`private` tinyint(1) unsigned DEFAULT NULL,
`ip` varchar(20) DEFAULT NULL,
PRIMARY KEY (`user_id`),
UNIQUE KEY `username` (`username`),
KEY `facebook_id` (`facebook_id`),
KEY `status` (`status`),
KEY `state` (`state`)
);
CREATE TABLE `users_site1` (
`user_id` int(9) unsigned NOT NULL,
`facebook_id` bigint(15) unsigned DEFAULT NULL,
`facebook_username` varchar(30) DEFAULT NULL,
`facebook_publish` tinyint(1) unsigned DEFAULT NULL,
`facebook_checkin` tinyint(1) unsigned DEFAULT NULL,
`facebook_offline` varchar(300) DEFAULT NULL,
`twitter_id` varchar(60) DEFAULT NULL,
`twitter_secret` varchar(50) DEFAULT NULL,
`twitter_username` varchar(20) DEFAULT NULL,
`type` char(1) DEFAULT 'M',
`referrer` int(4) unsigned DEFAULT NULL,
`referredid` int(9) unsigned DEFAULT NULL,
`session` varchar(60) DEFAULT NULL,
`api_session` varchar(60) DEFAULT NULL,
`status` char(1) DEFAULT 'R',
`created` datetime DEFAULT NULL,
`verified` datetime DEFAULT NULL,
`activated` datetime DEFAULT NULL,
`deleted` datetime DEFAULT NULL,
`logins` int(6) unsigned DEFAULT '0',
`api_logins` int(6) unsigned DEFAULT '0',
`last_login` datetime DEFAULT NULL,
`last_update` datetime DEFAULT NULL,
`ip` varchar(20) DEFAULT NULL,
PRIMARY KEY (`user_id`)
);
CREATE TABLE `connections` (
`cid1` int(9) unsigned NOT NULL DEFAULT '0',
`cid2` int(9) unsigned NOT NULL DEFAULT '0',
`cid3` int(9) unsigned NOT NULL DEFAULT '0',
`type` char(2) NOT NULL,
`status` char(1) NOT NULL,
`created` datetime DEFAULT NULL,
`updated` datetime DEFAULT NULL,
PRIMARY KEY (`cid1`,`cid2`,`type`,`cid3`),
KEY `cid1` (`cid1`,`type`),
KEY `cid2` (`cid2`,`type`)
);
Instead of WHERE a.userid IN( ... ) OR a.userid IN( ... ) you should use another join:
select
a.username,a.first_name,a.last_name,a.organization_name,a.organization,a.city,
a.state,a.zip,a.country,a.profile_photo,a.facebook_id,a.twitter_id,u.reviews
from accounts a
join users_site1 u ON a.user_id=u.user_id
join ( select cid2 as id from connections
where cid1=10001006 AND type="MM" AND status="A"
union
select cid1 as id from connections
where cid2=10001006 AND type="MM" AND status="A" ) c
on a.user_id = c.id
order by RAND() LIMIT 4;
have you tried remove order by RAND() and run again?
my result is below:
+----+--------------------+-------------+----------------+-------------------+---------+---------+------------------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-------------+----------------+-------------------+---------+---------+------------------+------+----------------------------------------------+
| 1 | PRIMARY | a | ALL | PRIMARY | NULL | NULL | NULL | 2 | Using where; Using temporary; Using filesort |
| 1 | PRIMARY | u | ALL | PRIMARY | NULL | NULL | NULL | 2 | Using where; Using join buffer |
| 3 | DEPENDENT SUBQUERY | connections | index_subquery | PRIMARY,cid1,cid2 | PRIMARY | 14 | func,const,const | 1 | Using where |
| 2 | DEPENDENT SUBQUERY | connections | ref | PRIMARY,cid1,cid2 | PRIMARY | 14 | const,func,const | 1 | Using where |
+----+--------------------+-------------+----------------+-------------------+---------+---------+------------------+------+----------------------------------------------+
I am not a MySQL guru by any means but been have involved more than once in optimization of high performance applications, though I was more on the implementation end of the optimisation process versus finding what needed to be optimized.
The firt thing I see is the subqueries seem efficient, but the way the first query is run with this where clause: ... where a.user_id IN (select cid2 ...) or a.user_id IN (select cid1 from ...) is a performance killer in my very humble opinion.
The first thing I would try to optimise performance, consider trying join decomposition , split your request in 2 or even 3 queries. The code is less pretty, but the db will be able to work more efficiently. It is a myth that doing everything in one query is better.
What can this bring you? Caching will be more efficient, if using MyISam tables the locking srategy is more efficient when you have less tables in your query, and you will reduce the redundant row accesses. If you can get your main query ( that would be the last one if you decompose ) from Using where; Using temporary; Using filesort, you will have much faster response.
Profile the different options you try with SHOW SESSION STATUS and FLUSH status, also you can disable caching to get true comparison of different options you try by adding SQL_NO_CACHE in your query, ie SELSECT SQL_NO_CACHE a.username ... etc..
Profiling and measuring the results is the only way you will be able to determine the performance gains. Unfortunately this step is often overlooked.
Good luck!