I have a query which looks something like this:
SELECT
STRAIGHT_JOIN `reviewApp_review`.`id`, `reviewApp_review`.`reviewTitle`,
`reviewApp_review`.`reviewContent`, `reviewApp_review`.`translatedEnTitle`,
`reviewApp_review`.`translatedEnContent`, `reviewApp_review`.`translatedEnDate`,
`reviewApp_review`.`reviewLink`, `reviewApp_review`.`reviewUser`,
`reviewApp_review`.`reviewUserProfile`, `reviewApp_review`.`reviewDataCreated`,
`reviewApp_review`.`reviewDataDiscovered`, `reviewApp_review`.`reviewData`,
`reviewApp_review`.`reviewRating`, `reviewApp_review`.`reviewSignature`,
`reviewApp_review`.`reviewSignature2`, `reviewApp_review`.`reviewExternalId`,
`reviewApp_review`.`reviewStatus`, `reviewApp_review`.`reviewWebsite`,
`reviewApp_review`.`language`, `reviewApp_review`.`helpfulVotes`,
`reviewApp_review`.`verified`, `reviewApp_review`.`color`,
`reviewApp_review`.`style`, `reviewApp_review`.`size`,
`reviewApp_review`.`lastUpdated`, `reviewApp_review`.`lastSeen`,
`reviewApp_review`.`deleted`, `reviewApp_review`.`alerted`,
`reviewApp_review`.`analyzed`, `reviewApp_review`.`productLink_id`
FROM `reviewApp_review`
INNER JOIN `reviewApp_productlink` ON (`reviewApp_review`.`productLink_id` = `reviewApp_productlink`.`id`)
INNER JOIN `reviewApp_product` ON (`reviewApp_productlink`.`product_id` = `reviewApp_product`.`id`)
WHERE (`reviewApp_product`.`owner` = 'my product'
AND `reviewApp_productlink`.`customer_id` = '1241'
AND (`reviewApp_review`.`translatedEnContent` LIKE '%%urban%%'
OR (`reviewApp_review`.`reviewContent` LIKE '%%urban%%'
AND `reviewApp_review`.`translatedEnDate` = '0'))
)
ORDER BY `reviewApp_review`.`reviewRating` ASC
LIMIT 10
I want to select reviews from the database and when a result is a number of reviews under 10, it takes a lot of time, more than a minute. And I'm wondering if there is a solution to optimize this query.
I tried to make different operations in the SQL query, as you can see I tried to use STRAIGHT_JOIN, and INNER_JOIN but I didn't have a better time.
The result of the query should be a querySet that contains a list of reviews, matching all those conditions from the query.
From Django I'm using something like that to run this query:
querySet.model.objects.raw(rawQuery)
where rawQuery is the query described above.
Here I have a python function and I want to optimize it because it takes to much time.
def fixQS(querySet):
"""
Optimisation for SQL QUERY using order by
https://dba.stackexchange.com/a/40195/246455
https://docs.djangoproject.com/en/2.1/ref/models/querysets/#extra
STRAIGHT_JOIN
"""
# complete the SQL with params encapsulated in quotes
sql, params = querySet.query.sql_with_params()
newParams = ()
for param in params:
if not str(param).startswith("'"):
if isinstance(param, str):
param = re.sub("'", "\\'", param)
newParams = newParams + ("'{}'".format(param),)
else:
newParams = newParams + (param,)
rawQuery = sql % newParams
# escape the percent used in SQL LIKE statements
rawQuery = re.sub('%', '%%', rawQuery)
# replace SELECT with SELECT STRAIGHT_JOIN
rawQuery = rawQuery.replace('SELECT', 'SELECT STRAIGHT_JOIN')
return querySet.model.objects.raw(rawQuery)
I'm calling it like this :
currentReviews = fixQS(currentReviews)
And when the debugger evaluates this it takes a lot of time.
Here is the result of the command SHOW CREATE TABLE:
CREATE TABLE `reviewApp_review` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`reviewTitle` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewContent` longtext COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewLink` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewDataCreated` int(11) NOT NULL,
`reviewDataDiscovered` int(11) NOT NULL,
`reviewRating` varchar(30) COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewSignature` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewStatus` varchar(200) COLLATE utf8mb4_unicode_ci NOT NULL,
`productLink_id` int(11) NOT NULL,
`reviewWebsite` varchar(200) COLLATE utf8mb4_unicode_ci NOT NULL,
`language` varchar(200) COLLATE utf8mb4_unicode_ci NOT NULL,
`verified` tinyint(1) NOT NULL,
`color` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`size` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`style` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`alerted` tinyint(1) NOT NULL,
`reviewUser` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewUserProfile` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewSignature2` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`reviewData` int(11) NOT NULL,
`reviewExternalId` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`helpfulVotes` int(11) NOT NULL,
`lastSeen` int(11) NOT NULL,
`lastUpdated` int(11) NOT NULL,
`translatedEnContent` longtext COLLATE utf8mb4_unicode_ci NOT NULL,
`translatedEnDate` int(11) NOT NULL,
`translatedEnTitle` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`deleted` tinyint(1) NOT NULL,
`analyzed` datetime(6) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `reviewApp_re_productLink_id_084fa2ec_fk_reviewApp_productlink_id` (`productLink_id`),
KEY `reviewApp_review_reviewSignature_2c31b21d_uniq` (`reviewSignature`),
KEY `reviewApp_review_reviewSignature_224ab822_idx` (`reviewSignature`,`productLink_id`),
KEY `reviewApp_review_fa0816ae` (`reviewSignature2`),
KEY `reviewApp_review_alerted_57344ef1_uniq` (`alerted`),
KEY `reviewApp_review_reviewData_5dd52e46_uniq` (`reviewData`),
KEY `reviewApp_review_reviewRating_d418fcfc_uniq` (`reviewRating`),
KEY `reviewApp_review_reviewWebsite_25d7fd04_uniq` (`reviewWebsite`),
KEY `reviewApp_review_reviewExternalId_32dfb169_uniq` (`reviewExternalId`),
KEY `reviewApp_reviewContent` (`reviewContent`(100)),
KEY `reviewApp_translatedEnContent` (`translatedEnContent`(100)),
KEY `reviewApp_translatedEnDate` (`translatedEnDate`),
KEY `reviewApp_review_da602f0b` (`deleted`),
KEY `reviewApp_review_language_3de815cd_uniq` (`language`),
KEY `reviewApp_review_order_index` (`reviewData`),
KEY `reviewApp_review_rating_order_index` (`reviewRating`),
KEY `reviewApp_review_rating_order_index_desc` (`reviewRating`),
CONSTRAINT `reviewApp_re_productLink_id_084fa2ec_fk_reviewApp_productlink_id` FOREIGN KEY (`productLink_id`) REFERENCES `reviewApp_productlink` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=24274768 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
And this one is for the product:
CREATE TABLE `reviewApp_product` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`owner` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`customer_id` int(11) DEFAULT NULL,
`ean` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`internalCode` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`sku` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
`asin` varchar(500) COLLATE utf8mb4_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `reviewApp_product_name_37c04b41_uniq` (`name`,`customer_id`),
KEY `reviewApp_product_cb24373b` (`customer_id`),
KEY `reviewApp_product_name_23da262c_uniq` (`name`),
KEY `reviewApp_product_owner_680ac2b4_uniq` (`owner`),
KEY `reviewApp_product_index` (`owner`,`id`),
CONSTRAINT `reviewApp_product_customer_id_7663d434_fk_reviewApp_customer_id` FOREIGN KEY (`customer_id`) REFERENCES `reviewApp_customer` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=31556 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
And also for productLink I have this:
CREATE TABLE `reviewApp_productlink` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`productLink` varchar(2000) COLLATE utf8mb4_unicode_ci NOT NULL,
`customer_id` int(11) NOT NULL,
`productDataCreated` int(11) NOT NULL,
`product_id` int(11) DEFAULT NULL,
`domain` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`httpStatus` int(11) NOT NULL,
`externalProductId` varchar(1000) COLLATE utf8mb4_unicode_ci NOT NULL,
`ratingsNumber` int(11) NOT NULL,
`rating` double DEFAULT NULL,
`fb_data` longtext COLLATE utf8mb4_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
KEY `reviewApp_productl_customer_id_f0141212_fk_reviewApp_customer_id` (`customer_id`),
KEY `reviewApp_productlink_9bea82de` (`product_id`),
CONSTRAINT `reviewApp_productl_customer_id_f0141212_fk_reviewApp_customer_id` FOREIGN KEY (`customer_id`) REFERENCES `reviewApp_customer` (`id`),
CONSTRAINT `reviewApp_productlin_product_id_6123214e_fk_reviewApp_product_id` FOREIGN KEY (`product_id`) REFERENCES `reviewApp_product` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=127823 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
Remove STRAIGHT_JOIN so that the Optimizer is allowed to try looking at the tables in different orders. The following composite indexes may give the Optimizer better options:
reviewApp_review: INDEX(translatedEnContent, reviewContent,
translatedEnDate, productLink_id, reviewRating)
reviewApp_productlink: INDEX(customer_id, product_id, id)
reviewApp_product: INDEX(owner, id)
translatedEnDate is a DATE? Yet you are testing for "0"??
More
(repeating original, but with aliases to shorten the text)
SELECT
STRAIGHT_JOIN rr.((lots of stuff))
FROM `reviewApp_review` AS rr
INNER JOIN `reviewApp_productlink` AS pl ON (rr.`productLink_id` = pl.`id`)
INNER JOIN `reviewApp_product` AS p ON (pl.`product_id` = p.`id`)
WHERE ( p.`owner` = 'my product'
AND pl.`customer_id` = '1241'
AND (rr.`translatedEnContent` LIKE '%%urban%%'
OR ( rr.`reviewContent` LIKE '%%urban%%'
AND rr.`translatedEnDate` = '0')
) )
ORDER BY rr.`reviewRating` ASC
LIMIT 10
You seem to be gathering all those columns only to throw away all but 10 rows. An optimization is to first find the 10 ids, then fetch the rest of the columns:
SELECT rr.((lots of stuff))
FROM ( SELECT rr2.id
FROM reviewApp_review` AS rr2
INNER JOIN `reviewApp_productlink` AS pl
ON (rr2.`productLink_id` = pl.`id`)
INNER JOIN `reviewApp_product` AS p
ON (pl.`product_id` = p.`id`)
WHERE ( p.`owner` = 'my product'
AND pl.`customer_id` = '1241'
AND ( rr2.`translatedEnContent` LIKE '%%urban%%'
OR ( rr2.`reviewContent` LIKE '%%urban%%'
AND rr2.`translatedEnDate` = '0')
) )
ORDER BY rr2.`reviewRating` ASC
LIMIT 10
) AS ids
INNER JOIN reviewApp_review` AS rr ON ids.id = rr.id
ORDER BY rr.`reviewRating` ASC -- Yes, this needs repeating
p: INDEX(owner)
pl: INDEX(customer_id, product_id)
reviewApp_review: INDEX(productLink_id, reviewRating)
reviewApp_review: INDEX(reviewRating)
These are useless (at least for the current query):
KEY `reviewApp_reviewContent` (`reviewContent`(100)),
KEY `reviewApp_translatedEnContent` (`translatedEnContent`(100)),
Better might be
FULLTEXT(translatedEnContent),
FULLTEXT(reviewContent)
together with
MATCH(reviewContent) AGAINST ('+urban' IN BOOLEAN MODE)
MATCH(translatedEnContent) AGAINST ('+urban' IN BOOLEAN MODE)
But, unfortunately, it won't be cheap to do the subquery. The hope is that 'my product' and '1241' do enough filtering to minimize the usage of LIKE or MATCH.
A minor side note: When you have INDEX(a,b) or UNIQUE(a,b), it is unnecessary to also have INDEX(a). (I see several instances of this.)
Even better...
If possible, get rid of the test on translatedEnDate and combine the two text columns:
FULLTEXT(reviewContent, translatedEnContent)
MATCH(reviewContent, translatedEnContent)
AGAINST ('+urban' IN BOOLEAN MODE)
This will make it possible to start the search with an efficient FT lookup, followed by checking the owner and customer_id. This would avoid a full table scan of the multi-GB reviewApp_review.
Related
I have the following MySQL query:
SELECT
`influencers`.*,
`locations`.`country_name`
FROM
`influencers`
LEFT JOIN `locations` ON `influencers`.`country_id` = `locations`.`id`
WHERE
`is_dead` = 0
AND `influencers`.`is_private` = 0
AND `influencers`.`country_id` = '31'
AND influencers.uuid IN(
SELECT
`influencer_uuid` FROM `category_influencer`
WHERE
`category_influencer`.`category_id` = 17
AND `category_influencer`.`is_main` = 1)
ORDER BY
`influencers`.`followed_by` DESC
LIMIT 7 OFFSET 6
I have identified the IN subquery is causing a lag of around 10s for this query to complete. Here is the EXPLAIN:
I have indexes on all columns being queried.
How can I significantly speed this query up?
Updated with SHOW CREATE TABLE for both:
locations
CREATE TABLE `locations` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`country_name` varchar(255) DEFAULT NULL,
`city_name` varchar(255) DEFAULT NULL,
`type` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `unique_index` (`city_name`, `country_name`),
KEY `type` (`type`)
USING BTREE) ENGINE = InnoDB AUTO_INCREMENT = 6479 DEFAULT CHARSET = utf8mb4
influencers
CREATE TABLE `influencers` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`bio` varchar(255) CHARACTER
SET utf8mb4 DEFAULT NULL,
`url` varchar(255) CHARACTER
SET utf8mb4 DEFAULT NULL,
`followed_by` int(11) DEFAULT NULL,
`follows` int(11) DEFAULT NULL,
`full_name` varchar(255) CHARACTER
SET utf8mb4 NOT NULL,
`social_id` varchar(255) DEFAULT NULL,
`is_private` tinyint (1) DEFAULT NULL,
`avatar` varchar(255) NOT NULL,
`username` varchar(30) NOT NULL,
`text_search` text CHARACTER
SET utf8mb4 NOT NULL,
`updated_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
`uuid` varchar(255) DEFAULT NULL,
`is_dead` tinyint (4) DEFAULT NULL,
`country_id` int(11) DEFAULT NULL,
`city_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `username` (`username`),
UNIQUE KEY `uuid` (`uuid`),
KEY `is_dead` (`is_dead`),
KEY `updated_at` (`updated_at`),
KEY `followed_by` (`followed_by`),
KEY `social_id` (`social_id`),
KEY `is_private` (`is_private`),
KEY `country_id` (`country_id`),
FULLTEXT KEY `text_search` (`text_search`)) ENGINE = InnoDB AUTO_INCREMENT = 2278376 DEFAULT CHARSET = utf8
You could avoid the in clause using an inner join
SELECT
`influencers`.*,
`locations`.`country_name`
FROM
`influencers`
INNER JOIN (
SELECT
`influencer_uuid` FROM `category_influencer`
WHERE
`category_id` = 17
AND `is_main` = 1
) T ON T.influencer_uuid = influencers.uuid
LEFT JOIN `locations` ON `influencers`.`country_id` = `locations`.`id`
WHERE
`is_dead` = 0
AND `is_private` = 0
AND `country_id` = '31'
ORDER BY
`followed_by` DESC
LIMIT 7 OFFSET 6
This way instead of an iteration on all the IN result you use just a single relational match based on join
Unless I missed something, you can replace the subselect with JOIN:
SELECT influencers.*,
locations.country_name
FROM influencers
JOIN category_influencer T ON (
T.influencer_uuid = influencers.uuid
AND category_id = 17
AND is_main = 1)
LEFT JOIN locations ON influencers.country_id = locations.id
WHERE is_dead = 0
AND is_private = 0
AND country_id = '31'
ORDER BY followed_by DESC
LIMIT 7 OFFSET 6
I'm building a twitter-like app using node.js for fun and I have multiple tables :
users: to store users' data.
tweets: to store tweets.
likes: to store what users liked what tweet.
retweets: to store what users retweeted what tweet.
following: to store what user is following other users.
CREATE TABLE IF NOT EXISTS `following` (
`user_id` varchar(50) NOT NULL,
`followed_id` varchar(50) NOT NULL,
`date_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`user_id`,`followed_id`)
)
CREATE TABLE IF NOT EXISTS `likes` (
`user_id` varchar(50) NOT NULL,
`tweet_id` varchar(50) NOT NULL,
`date_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`user_id`,`tweet_id`)
)
CREATE TABLE IF NOT EXISTS `retweets` (
`user_id` varchar(50) NOT NULL,
`tweet_id` varchar(50) NOT NULL,
`date_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`user_id`,`tweet_id`)
)
CREATE TABLE IF NOT EXISTS `tweets` (
`tweet_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
`created_at` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`reply_to_tweet_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
`reply_to_user_id` varchar(50) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
`truncated` tinyint(1) NOT NULL,
`author` varchar(30) NOT NULL,
`text` varchar(255) NOT NULL,
`media` varchar(255) NOT NULL,
`entities` json NOT NULL,
PRIMARY KEY (`tweet_id`)
)
CREATE TABLE IF NOT EXISTS `users` (
`user_id` varchar(30) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
`email` varchar(30) NOT NULL,
`username` varchar(30) NOT NULL,
`password` varchar(255) NOT NULL,
`handelname` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL,
`account_url` varchar(255) NOT NULL,
`bio` varchar(255) NOT NULL,
`profile_pic` varchar(255) NOT NULL,
`cover_pic` varchar(255) NOT NULL,
`protected` tinyint(1) NOT NULL,
`created_at` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`user_id`),
UNIQUE KEY `email` (`email`),
UNIQUE KEY `username` (`username`)
)
and I want to create a new view for the users containing there information along with how many tweets
they did like, the number of users they follow, the number of users follow them and how many tweets they have created.
I want this information in one table but I tried many ways and none of them worked fine for me!
what is the proper solution for such a problem?
You may try below query -
SELECT `user_id`
,LI.LI_CNT no_of_tweets_liked
,FL.FL_CNT no_of_followers
,FL2.FL2_CNT no_of_followee
,RT.RT_CNT no_of_retweets
FROM `users` U
JOIN (SELECT `user_id`, COUNT(`tweet_id`) LI_CNT
FROM `likes`
GROUP BY `user_id`) LI ON U.`user_id` = LI.`user_id`
JOIN (SELECT `user_id`, COUNT(`followed_id`) FL_CNT
FROM `following`
GROUP BY `user_id`) FL ON U.`user_id` = FL.`user_id`
JOIN (SELECT `followed_id`, COUNT(`followed_id`) FL2_CNT
FROM `following`
GROUP BY `followed_id`) FL2 ON U.`user_id` = FL2.`followed_id`
JOIN (SELECT `user_id`, COUNT(`tweet_id`) RT_CNT
FROM `retweets`
GROUP BY `user_id`) RT ON U.`user_id` = RT.`user_id`;
query is simple, as below:
select count(1) from ec_account a join ec_card b on a.id = b.AccountId
there are 2.5 million rows in either ec_account and ec_card.(InnoDB)
here is the execution plan:
execution plan
as you see,
it already added index and used it, but the query still costed almost 60 seconds, is there any way could optimize it except changing database(mariadb has no such choke point as far as i know).
here is table DDL,ec_ccount:
CREATE TABLE `ec_account` (
`Id` varchar(64) NOT NULL,
`AccountType` varchar(32) NOT NULL,
`Name` varchar(32) NOT NULL,
`Status` tinyint(3) unsigned NOT NULL,
`IDCardType` varchar(32) DEFAULT NULL,
`IDCardNo` varchar(64) DEFAULT NULL,
`Password` varchar(256) DEFAULT NULL,
`PasswordHalt` varchar(128) DEFAULT NULL,
`Sex` varchar(8) DEFAULT NULL,
`BirthDay` datetime NOT NULL,
`Mobile` varchar(16) DEFAULT NULL,
`Address` varchar(64) DEFAULT NULL,
`Linkman` varchar(32) DEFAULT NULL,
`LinkmanRelation` varchar(16) DEFAULT NULL,
`LinkmanTel` varchar(16) DEFAULT NULL,
`Remark` varchar(128) DEFAULT NULL,
`Nationality` varchar(32) DEFAULT NULL,
`Nation` varchar(32) DEFAULT NULL,
`MaritalStatus` varchar(8) DEFAULT NULL,
`NativePlace` varchar(64) DEFAULT NULL,
`Occupation` varchar(32) DEFAULT NULL,
`BloodType` varchar(8) DEFAULT NULL,
`Education` varchar(8) DEFAULT NULL,
`LinkmanAddress` varchar(64) DEFAULT NULL,
`HomeAddress` varchar(128) DEFAULT NULL,
`Email` varchar(64) DEFAULT NULL,
`CompanyName` varchar(64) DEFAULT NULL,
`CompanyAddress` varchar(128) DEFAULT NULL,
`CompanyTel` varchar(16) DEFAULT NULL,
`Creator` char(36) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`CreateTime` datetime NOT NULL,
`LastModifier` char(36) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`LastModifyTime` datetime DEFAULT NULL,
`Avatar` longblob,
PRIMARY KEY (`Id`),
KEY `IX_Name` (`Name`) USING HASH,
KEY `Idx_IDCard_Account` (`IDCardType`,`IDCardNo`) USING HASH,
KEY `Idx_Mobile` (`Mobile`) USING HASH,
KEY `Idx_CreateTime` (`CreateTime`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
and ec_card :
CREATE TABLE `ec_card` (
`Id` char(36) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '',
`AccountId` varchar(64) NOT NULL,
`CardType` varchar(32) NOT NULL,
`CardNo` varchar(32) NOT NULL,
`Status` tinyint(3) unsigned NOT NULL,
`IsPasswordAuth` tinyint(1) NOT NULL,
PRIMARY KEY (`Id`),
UNIQUE KEY `Idx_Unique_AccountId_CardType` (`AccountId`,`CardType`) USING HASH,
UNIQUE KEY `Idx_Unique_CardType_CardNo` (`CardType`,`CardNo`) USING HASH,
KEY `Idx_Uniques_AccountId` (`AccountId`) USING BTREE,
CONSTRAINT `FK_ec_card_ec_account_AccountId` FOREIGN KEY (`AccountId`) REFERENCES `ec_account` (`Id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Not without fundamentally changing the query.
There are no conditions on your query! It selects all 2.5 million rows from ec_card, as well as every matching row from ec_account. Reading all this data from disk and sending it over the network is the bottleneck; there is no way to change that without changing what the query does.
Here is a workaround for you. I think it would run much faster, and get the same result.
Calculate the total count of ec_account:
SELECT count(1) AS total_count FROM ec_account;
Calculate the amount of records those existed in ec_account but not existed in ec_card:
SELECT count(1) AS missing_count
FROM ec_account a LEFT JOIN ec_card b on a.id = b.AccountId
WHERE b.AccountId IS NULL;
Matched count = total_count - missing_count
The core problem here is that you combined two large table together, it requires a lot of memory and it apparently needs a lot of time to finish.
try it using correlated subquery. This might help:
select count(1) from ec_account a where exists (select * from ec_card b
where b.AccountId=a.id)
Also, other than indexing following strategies generally help:
- Denormalization
- Caching results
- Using a NoSQL database
I have the following schema:
CREATE TABLE `news` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`news_category_id` int(10) unsigned NOT NULL,
`news_type_id` int(10) unsigned NOT NULL,
`news_pictures_main_id` int(10) unsigned DEFAULT NULL,
`title` tinytext COLLATE latin1_general_ci,
`body` text COLLATE latin1_general_ci,
`tmstp` timestamp NULL DEFAULT NULL,
`subcategory` varchar(64) COLLATE latin1_general_ci DEFAULT NULL,
`source` varchar(128) COLLATE latin1_general_ci DEFAULT NULL,
`old_id` int(10) unsigned DEFAULT NULL,
`tags` text COLLATE latin1_general_ci,
PRIMARY KEY (`id`),
KEY `news_time_idx` (`tmstp`),
KEY `fk_news_news_pictures1` (`news_pictures_main_id`),
KEY `fk_news_news_category1` (`news_category_id`),
KEY `fk_news_news_type1` (`news_type_id`),
CONSTRAINT `fk_news_news_category1` FOREIGN KEY (`news_category_id`) REFERENCES `news_category` (`id`) ON UPDATE CASCADE,
CONSTRAINT `fk_news_news_pictures1` FOREIGN KEY (`news_pictures_main_id`) REFERENCES `news_pictures` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
CONSTRAINT `fk_news_news_type1` FOREIGN KEY (`news_type_id`) REFERENCES `news_type` (`id`) ON UPDATE CASCADE
) ENGINE=InnoDB
CREATE TABLE `news_pictures` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`path` text COLLATE latin1_general_ci,
`description` text COLLATE latin1_general_ci,
`author` varchar(45) COLLATE latin1_general_ci DEFAULT NULL,
`news_id` int(10) unsigned DEFAULT NULL,
`temp_id` varchar(40) COLLATE latin1_general_ci DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `filename_old_id_unq` (`path`(20),`temp_id`(6)),
KEY `fk_news_pictures_news1` (`news_id`),
KEY `temp_id_idx` (`temp_id`(8)),
CONSTRAINT `fk_news_pictures_news1` FOREIGN KEY (`news_id`) REFERENCES `news` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB
CREATE TABLE `news_category` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(45) COLLATE latin1_general_ci DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB
CREATE TABLE `news_type` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(45) COLLATE latin1_general_ci DEFAULT NULL,
`slug` varchar(45) COLLATE latin1_general_ci DEFAULT NULL,
PRIMARY KEY (`id`)
KEY `news_type_slug_idx` (`slug`)
) ENGINE=InnoDB
From that, there is derived the following view:
CREATE OR REPLACE VIEW `news_full` AS select `n`.`id` AS `id`,
`n`.`title` AS `title`,
`n`.`body` AS `body`,
`n`.`tmstp` AS `tmstp`,
`n`.`subcategory` AS `subcategory`,
`n`.`source` AS `source`,
`n`.`old_id` AS `old_id`,
`n`.`news_type_id` AS `news_type_id`,
`n`.`tags` AS `tags`,
`nt`.`name` AS `news_type_name`,
`nt`.`slug` AS `news_type_slug`,
`n`.`news_pictures_main_id` AS `news_pictures_main_id`,
`np`.`path` AS `news_pictures_main_path`,
`np`.`description` AS `news_pictures_main_description`,
`np`.`author` AS `news_pictures_main_author`,
`np`.`temp_id` AS `news_pictures_main_temp_id`,
`n`.`news_category_id` AS `news_category_id`,
`nc`.`name` AS `news_category_name`
from (((`news` `n`
left join `news_pictures` `np` on((`n`.`news_pictures_main_id` = `np`.`id`)))
join `news_category` `nc` on((`n`.`news_category_id` = `nc`.`id`)))
join `news_type` `nt` on((`n`.`news_type_id` = `nt`.`id`)));
However, if I try to run the following query:
select * from news_full order by tmstp limit 100
I get the following execution plan (please click on the image to expand it):
Notice the Using temporary; Using filesort field in the first step. But this is weird, because tmstp field is indexed on the base table.
First I thought this was due the left join on the view, but I've changed it to inner join and I got the same results.
Edit
As #Michael-sqlbot cleverly noticed, the query optimizer is inverting the order of the base tables, putting news_category (nc) first.
If I change the query that creates the view to use only LEFT JOINs it seems to work:
The execution times, as expected, as blatantly different:
Not satisfied, I created another view with the original query, adding the STRAIGHT_JOIN statement. So, the query plan comes as follows:
So, it's not using the index.
However, if I run the plan for the base query adding the same ORDER BY and LIMIT clauses, it does uses the index:
(Not an answer, but some other issues to bring up...)
UNIQUE KEY `filename_old_id_unq` (`path`(20),`temp_id`(6))
That constrains the first 20 characters of path, together with the first 6 characters of temp_id to be unique across the table. Did you really want that?
I suspect the optimizer will never use both columns of that index. (In general, prefixing is useless.)
And...
`title` tinytext COLLATE latin1_general_ci
Change to VARCHAR(255). There are disadvantages of TINYTEXT and perhaps no advantages.
I need to make a query that will return me a count of messages for every user's dialog. I tried to make it by myself, but it gives me count of messages from all dialogs.
Here is my query that contains params that I need.
SELECT COUNT(*)
FROM message, dialog
WHERE (dialog.username LIKE 'acid')
AND (message.checkms=0)
AND (message.messender NOT LIKE 'acid')
AND (dialog.iddialog=message.iddialog)
It's result. Pls help!
CREATE TABLE `users` (
`username` varchar(45) CHARACTER SET utf8 NOT NULL,
`enabled` bit(1) NOT NULL,
`password` varchar(60) CHARACTER SET utf8 NOT NULL,
`name` varchar(30) CHARACTER SET utf8 NOT NULL,
`surname` varchar(30) CHARACTER SET utf8 NOT NULL,
`email` varchar(50) CHARACTER SET utf8 NOT NULL,
`gender` varchar(30) CHARACTER SET utf8 NOT NULL,
`age` varchar(2) NOT NULL,
`weight` varchar(2) NOT NULL,
`height` varchar(3) NOT NULL,
`sport` varchar(50) CHARACTER SET utf8 NOT NUlL,
`place` varchar(400) CHARACTER SET utf8 NOT NULL,
`photo` varchar(100) NOT NULL,
PRIMARY KEY (`username`)
) ENGINE=InnoDB DEFAULT CHARSET=UTF8;
create table dialog
(
`iddialog` int NOT NULL auto_increment,
`reciever` varchar(50) CHARACTER SET utf8 NOT NULL,
`username`varchar(45) CHARACTER SET utf8 NOT NULL,
PRIMARY KEY(`iddialog`),
foreign key (`username`) references users(`username`)
)ENGINE=InnoDB DEFAULT CHARSET=UTF8;
create table message(
`idmessage` BIGINT not null auto_increment,
`text` varchar(300) character set utf8 NOT NULL,
`date` varchar(40) NOT NULL,
`iddialog` int NOT NULL,
`messender` varchar(50),
`checkms` boolean,
primary key (`idmessage`),
foreign key (`iddialog`) references dialog(`iddialog`)
)ENGINE=InnoDB DEFAULT CHARSET=UTF8;
If I'm understanding you correctly, it sounds like you need to group by username:
SELECT d.username, COUNT(*)
FROM message m
join dialog d on m.iddialog = d.iddialog
WHERE d.username = 'acid'
AND m.checkms=0
AND m.messender != 'acid'
GROUP BY d.username
A couple of quick notes. It's better to use the more standard ansi join syntax. Also no need to use like if you aren't using a wildcard -- it's equivalent to equals without.
Your question might need a little more clarification. Perhaps you need to also group by d.iddialog. That would return the results per user per dialog:
SELECT d.username, d.iddialog, COUNT(*)
FROM message m
join dialog d on m.iddialog = d.iddialog
WHERE d.username = 'acid'
AND m.checkms=0
AND m.messender != 'acid'
GROUP BY d.username, d.iddialog