Query result is very slow - mysql

I am developing a website like any other social networking site using mysql.
I wish it give people suggestion to my users, and I have implemented this functionality in my application, but It is working very slow. this process take 2-3 seconds to fetch result from server. It has all the necessary indexes, relations on table. I have used EXPLAIN command to understand it, but I got nothing problematic in it.
I can't understand what is the basic problem in it. Please help me.
here is my table structure :
Table : UserMaster
~~~~~~~~~~~~~~~~~~
CREATE TABLE `UserMaster` (
`UserID` mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
`UserName` varchar(20) DEFAULT NULL,
`EMailID` varchar(50) DEFAULT NULL,
`FirstName` varchar(20) NOT NULL,
`LastName` varchar(20) NOT NULL,
`CityID` mediumint(8) unsigned DEFAULT NULL,
PRIMARY KEY (`UserID`),
UNIQUE KEY `UX_UserMaster_UserName` (`UserName`),
UNIQUE KEY `UX_UserMaster_EMailID` (`EMailID`),
KEY `FK_UserMaster_CityMst_CityID_idx` (`CityID`),
KEY `FK_UserMaster_CountryMst_CountryID_idx` (`CountryID`),
CONSTRAINT `FK_UserMaster_CityMst_CityID`
FOREIGN KEY (`CityID`) REFERENCES `CityMst` (`CityID`) ON DELETE NO ACTION,
CONSTRAINT `FK_UserMaster_CountryMst_CountryID` FOREIGN KEY CountryID REFERENCES CountryMst (CountryID) ON DELETE NO ACTION ON UPDATE CASCADE
)
ENGINE=InnoDB AUTO_INCREMENT=19722 DEFAULT CHARSET=utf8$$
Table : UserFriends
~~~~~~~~~~~~~~~~~~~
CREATE TABLE `UserFriends` (
`FriendID` mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
`UserID` mediumint(8) unsigned NOT NULL,
`UserID2` mediumint(8) unsigned NOT NULL,
`RequestDate` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
`Status` tinyint(3) unsigned NOT NULL DEFAULT '2',
`ApprovalDate` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`FriendID`),
UNIQUE KEY `UX_UserFriends_UserID_UserID2` (`UserID`,`UserID2`),
KEY `IX_UserFriens_UserID_ApprovalStatus` (`UserID`,`ApprovalStatus`,`UserID2`,`FriendID`,`RequestDate`,`ApprovalDate`),
KEY `FK_UserFriends_UserMaster_UserID_idx` (`UserID`),
KEY `FK_UserFriends_UserMaster_UserID2_idx` (`UserID2`),
CONSTRAINT `FK_UserFriends_UserMaster_UserID` FOREIGN KEY (`UserID`) REFERENCES `UserMaster` (`UserID`) ON DELETE NO ACTION ON UPDATE CASCADE,
CONSTRAINT `FK_UserFriends_UserMaster_UserID2` FOREIGN KEY (`UserID2`) REFERENCES `UserMaster` (`UserID`) ON DELETE NO ACTION ON UPDATE CASCADE
)
ENGINE=InnoDB AUTO_INCREMENT=50825 DEFAULT CHARSET=utf8$$
UserID & UserID2 both fields are linked with UserMaster.UserID
Here is my select query :
~~~~~~~~~~~~~~~~~~~~~~~~~
SELECT
upm.UserID,
upm.UserName,
upm.FirstName,
COALESCE(mf.TotMutualFriends,0) TotMutualFriends
FROM UserMaster upm
LEFT JOIN CityMst ct on ct.CityID = upm.CityID
LEFT JOIN StateMst st on st.StateID = ct.StateID
LEFT JOIN (
SELECT uf.UserID, COUNT(1) TotMutualFriends
FROM (
SELECT uf.UserID, uf.UserID2, uf.ApprovalStatus
FROM UserFriends uf
UNION ALL
SELECT uf.UserID2 UserID, uf.UserID UserID2, uf.ApprovalStatus
FROM UserFriends uf
) uf
INNER JOIN (
SELECT IF(uf.UserID = 1, uf.UserID2, uf.UserID) UserID2
FROM UserFriends uf
WHERE (uf.UserID = 1 OR uf.UserID2 = 1)
AND uf.ApprovalStatus = 1
) uf1 on uf1.UserID2 = uf.UserID2
WHERE uf.ApprovalStatus = 1
GROUP BY uf.UserID
) mf on mf.UserID = upm.UserID
LEFT JOIN (
SELECT DISTINCT usar.UserID2
FROM UserSuggAutoRejct usar
WHERE usar.UserID = 1
UNION
SELECT IF(uf.UserID = 1, uf.UserID2, uf.UserID) UserID2
FROM UserFriends uf
WHERE (uf.UserID = 1 OR uf.UserID2 = 1)
) usar ON usar.UserID2 = upm.UserID
WHERE upm.UserStatus IN(10,11)
AND upm.UserID <> 1
AND upm.UserID NOT IN (1221,2191)
AND usar.UserID2 IS NULL
ORDER BY
(CASE WHEN COALESCE(mf.TotMutualFriends,0) > 0 THEN 0 ELSE 1 END),
(CASE WHEN COALESCE(mf.TotMutualFriends,0) > 0 THEN RAND() ELSE NULL END),
(CASE upm.CityID WHEN 1 THEN 0 ELSE 1 END),
(CASE upm.CityID WHEN 1 THEN RAND() ELSE NULL END),
(CASE ct.StateID WHEN 1 THEN 0 ELSE 1 END),
(CASE ct.StateID WHEN 1 THEN RAND() ELSE NULL END),
(CASE st.CountryID WHEN 91 THEN 0 ELSE 1 END),
(CASE st.CountryID WHEN 91 THEN RAND() ELSE NULL END)
LIMIT 10
This is performing very slow. It takes 2-3 seconds to evolute.

relational database might not be relevant for social networking sites.
because joining table is very slow, try using other no-sql databases(NoSql type database).
if u still insist using mysql, then try not to much join in your query.
sorry for my bad english, if any.

Related

SQL Views for single row (with where clause)

I'm working on a system that administrates courses, with multiple classes, with multiple lessons and multiple consumers on them. As the system grows more data were required so with some performance issues I've decided to go with SQL Views. We're using MySQL.
So I've replaced old calls to the DB (for example for the single lesson)
select * from `courses_classes_lessons` where `courses_classes_lessons`.`deleted_at` is null limit 1;
select count(consumer_id) as consumers_count from `courses_classes_lessons_consumers` where `lesson_id` = '448' limit 1;
select `max_consumers` from `courses_classes` where `id` = '65' limit 1;
select `id` from `courses_classes_lessons` left join `courses_classes_lessons_consumers` on `courses_classes_lessons_consumers`.`lesson_id` = `courses_classes_lessons`.`id` where `id` = '448' group by `courses_classes_lessons`.`id` having count(courses_classes_lessons_consumers.consumer_id) < '4' limit 1;
select courses_classes.max_consumers - LEAST(count(courses_classes_lessons_consumers.consumer_id), courses_classes.max_consumers) as available_spaces from `courses_classes_lessons` left join `courses_classes_lessons_consumers` on `courses_classes_lessons_consumers`.`lesson_id` = `courses_classes_lessons`.`id` left join `courses_classes` on `courses_classes_lessons`.`class_id` = `courses_classes`.`id` where `courses_classes_lessons`.`id` = '448' group by `courses_classes`.`id` limit 1;
The above took around 4-5ms
with the SQL View as follow:
CREATE OR REPLACE VIEW `courses_classes_lessons_view` AS
SELECT
courses_classes_lessons.id AS lesson_id,
(SELECT
max_consumers
FROM
courses_classes
WHERE
id = courses_classes_lessons.class_id
LIMIT 1) AS class_max_consumers,
(SELECT
count(consumer_id)
FROM
courses_classes_lessons_consumers
WHERE
lesson_id = courses_classes_lessons.id) AS consumers_count,
(SELECT
CASE WHEN consumers_count >= class_max_consumers THEN
TRUE
ELSE
FALSE
END AS is_full) AS is_full,
(CASE WHEN courses_classes_lessons.completed_at > NOW() THEN
'completed'
WHEN courses_classes_lessons.cancelled_at > NOW() THEN
'cancelled'
WHEN courses_classes_lessons.starts_at > NOW() THEN
'upcoming'
ELSE
'incomplete'
END) AS status,
(SELECT
class_max_consumers - LEAST(consumers_count, class_max_consumers)) AS available_spaces
FROM
courses_classes_lessons
The problem I'm having is that doesn't matter if I'm loading the whole View or a single row from it - it always takes about 6-9s to load! But when I've tried the same query with a WHERE clause it takes about 500μs. I'm new to SQL View and confused - why there are no indexes/primary keys that I could use to load a single row quickly? Am I doing something wrong?
EXPLAIN RESULT
INSERT INTO `courses_classes` (`id`, `select_type`, `table`, `partitions`, `type`, `possible_keys`, `key`, `key_len`, `ref`, `rows`, `filtered`, `Extra`) VALUES
(1, 'PRIMARY', 'courses_classes_lessons', NULL, 'ALL', NULL, NULL, NULL, NULL, 478832, 100.00, NULL),
(3, 'DEPENDENT SUBQUERY', 'courses_classes_lessons_consumers', NULL, 'ref', 'PRIMARY,courses_classes_lessons_consumers_lesson_id_index', 'courses_classes_lessons_consumers_lesson_id_index', '4', 'api.courses_classes_lessons.id', 3, 100.00, 'Using index'),
(2, 'DEPENDENT SUBQUERY', 'courses_classes', NULL, 'eq_ref', 'PRIMARY,courses_classes_id_parent_id_index', 'PRIMARY', '4', 'api.courses_classes_lessons.class_id', 1, 100.00, NULL);
TABLE STRUCTURE
Lessons
CREATE TABLE `courses_classes_lessons` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`franchisee_id` int(10) unsigned NOT NULL,
`class_id` int(10) unsigned NOT NULL,
`instructor_id` int(10) unsigned NOT NULL,
`instructor_rate` int(10) unsigned NOT NULL DEFAULT '0',
`instructor_total` int(10) unsigned NOT NULL DEFAULT '0',
`instructor_paid` tinyint(1) NOT NULL DEFAULT '0',
`starts_at` timestamp NULL DEFAULT NULL,
`ends_at` timestamp NULL DEFAULT NULL,
`completed_at` timestamp NULL DEFAULT NULL,
`cancelled_at` timestamp NULL DEFAULT NULL,
`cancelled_reason` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
`cancelled_reason_extra` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
`deleted_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `courses_classes_lessons_franchisee_id_foreign` (`franchisee_id`),
KEY `courses_classes_lessons_class_id_foreign` (`class_id`),
KEY `courses_classes_lessons_instructor_id_foreign` (`instructor_id`),
KEY `courses_classes_lessons_starts_at_ends_at_index` (`starts_at`,`ends_at`),
KEY `courses_classes_lessons_completed_at_index` (`completed_at`),
KEY `courses_classes_lessons_cancelled_at_index` (`cancelled_at`),
KEY `courses_classes_lessons_class_id_deleted_at_index` (`class_id`,`deleted_at`),
KEY `courses_classes_lessons_deleted_at_index` (`deleted_at`),
KEY `class_ownership_index` (`class_id`,`starts_at`,`cancelled_at`,`deleted_at`),
CONSTRAINT `courses_classes_lessons_class_id_foreign` FOREIGN KEY (`class_id`) REFERENCES `courses_classes` (`id`) ON DELETE CASCADE,
CONSTRAINT `courses_classes_lessons_franchisee_id_foreign` FOREIGN KEY (`franchisee_id`) REFERENCES `franchisees` (`id`) ON DELETE CASCADE,
CONSTRAINT `courses_classes_lessons_instructor_id_foreign` FOREIGN KEY (`instructor_id`) REFERENCES `instructors` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=487853 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
Lessons consumers
CREATE TABLE `courses_classes_lessons_consumers` (
`lesson_id` int(10) unsigned NOT NULL,
`consumer_id` int(10) unsigned NOT NULL,
`present` tinyint(1) DEFAULT NULL,
`plan_id` int(10) unsigned DEFAULT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`lesson_id`,`consumer_id`),
KEY `courses_classes_lessons_consumers_consumer_id_foreign` (`consumer_id`),
KEY `courses_classes_lessons_consumers_plan_id_foreign` (`plan_id`),
KEY `courses_classes_lessons_consumers_lesson_id_index` (`lesson_id`),
KEY `courses_classes_lessons_consumers_present_index` (`present`),
CONSTRAINT `courses_classes_lessons_consumers_consumer_id_foreign` FOREIGN KEY (`consumer_id`) REFERENCES `customers_consumers` (`id`) ON DELETE CASCADE,
CONSTRAINT `courses_classes_lessons_consumers_lesson_id_foreign` FOREIGN KEY (`lesson_id`) REFERENCES `courses_classes_lessons` (`id`) ON DELETE CASCADE,
CONSTRAINT `courses_classes_lessons_consumers_plan_id_foreign` FOREIGN KEY (`plan_id`) REFERENCES `customers_plans` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
From classes it's only using max_consumers int(10) unsigned NOT NULL DEFAULT '0',
UPDATE 1
I've changed SQL View to the following one:
CREATE OR REPLACE VIEW `courses_classes_lessons_view` AS
SELECT
courses_classes_lessons.id AS lesson_id,
courses_classes.max_consumers AS class_max_consumers,
lessons_consumers.consumers_count AS consumers_count,
(
SELECT
CASE WHEN consumers_count >= class_max_consumers THEN
TRUE
ELSE
FALSE
END AS is_full) AS is_full,
(
CASE WHEN courses_classes_lessons.completed_at > NOW() THEN
'completed'
WHEN courses_classes_lessons.cancelled_at > NOW() THEN
'cancelled'
WHEN courses_classes_lessons.starts_at > NOW() THEN
'upcoming'
ELSE
'incomplete'
END) AS status,
(
SELECT
class_max_consumers - LEAST(consumers_count, class_max_consumers)) AS available_spaces
FROM
courses_classes_lessons
JOIN courses_classes ON courses_classes.id = courses_classes_lessons.class_id
JOIN (
SELECT
lesson_id,
count(*) AS consumers_count
FROM
courses_classes_lessons_consumers
GROUP BY
courses_classes_lessons_consumers.lesson_id) AS lessons_consumers ON lessons_consumers.lesson_id = courses_classes_lessons.id;
and even though the SELECT query itself seems to be way slower than the previous one then as the View it seems to perform way better. It's still not as fast as I wish it will be but it's a step forward.
Overall improvement jumps from 6-7s to around 800ms, the aim here is in the area of 500μs-1ms. Any adivces how I can improve my SQL View more?
UPDATE 2
Ok, I've found the bottleneck! Again - it's kinda similar to the last one (SELECT query works fast for a single row, but SQL VIEW is trying to access the whole table at once every time.
My new lesson SQL VIEW:
CREATE OR REPLACE VIEW `courses_classes_lessons_view` AS
SELECT
courses_classes_lessons.id AS lesson_id,
courses_classes.max_consumers AS class_max_consumers,
IFNULL(lessons_consumers.consumers_count,0) AS consumers_count,
(
SELECT
CASE WHEN consumers_count >= class_max_consumers THEN
TRUE
ELSE
FALSE
END AS is_full) AS is_full,
(
CASE WHEN courses_classes_lessons.completed_at > NOW() THEN
'completed'
WHEN courses_classes_lessons.cancelled_at > NOW() THEN
'cancelled'
WHEN courses_classes_lessons.starts_at > NOW() THEN
'upcoming'
ELSE
'incomplete'
END) AS status,
(
SELECT
IFNULL(class_max_consumers, 0) - LEAST(IFNULL(consumers_count,0), class_max_consumers)) AS available_spaces
FROM
courses_classes_lessons
JOIN courses_classes ON courses_classes.id = courses_classes_lessons.class_id
LEFT JOIN courses_classes_lessons_consumers_view AS lessons_consumers ON lessons_consumers.lesson_id = courses_classes_lessons.id;
Another SQL View - this time for consumers:
CREATE OR REPLACE VIEW `courses_classes_lessons_consumers_view` AS
SELECT
lesson_id,
IFNULL(count(
consumer_id),0) AS consumers_count
FROM
courses_classes_lessons_consumers
GROUP BY
courses_classes_lessons_consumers.lesson_id;
And looks like this one is the trouble maker! The consumers table is above, and here is the explain for the above SELECT query:
INSERT INTO `courses_classes_lessons_consumers` (`id`, `select_type`, `table`, `partitions`, `type`, `possible_keys`, `key`, `key_len`, `ref`, `rows`, `filtered`, `Extra`)
VALUES(1, 'SIMPLE', 'courses_classes_lessons_consumers', NULL, 'index', 'PRIMARY,courses_classes_lessons_consumers_consumer_id_foreign,courses_classes_lessons_consumers_plan_id_foreign,courses_classes_lessons_consumers_lesson_id_index,courses_classes_lessons_consumers_present_index', 'courses_classes_lessons_consumers_lesson_id_index', '4', NULL, 1330649, 100.00, 'Using index');
Any idea how to spread up this count?
Consider writing a Stored procedure; it may be able to get the 448 put into place to be better optimized.
If you know there will be only one row (such as when doing COUNT(*)), skip the LIMIT 1.
Unless consumer_id might be NULL, use COUNT(*) instead of COUNT(consumer_id).
A LIMIT without an ORDER BY leaves you getting a random row.
If courses_classes_lessons_consumers is a many-to-many mapping table, I will probably have some index advice after I see SHOW CREATE TABLE.
Which of the 5 SELECTs is the slowest?
After many attempts, it looks like that the Procedure way is the best approach and I won't be spending more time on the SQL Views
Here's the procedure I wrote:
CREATE PROCEDURE `LessonData`(
IN lessonId INT(10)
)
BEGIN
SELECT
courses_classes_lessons.id AS lesson_id,
courses_classes.max_consumers AS class_max_consumers,
IFNULL((SELECT
count(consumer_id) as consumers_count
FROM
courses_classes_lessons_consumers
WHERE
lesson_id = courses_classes_lessons.id
GROUP BY
courses_classes_lessons_consumers.lesson_id), 0) AS consumers_count,
(
SELECT
CASE WHEN consumers_count >= class_max_consumers THEN
TRUE
ELSE
FALSE
END) AS is_full,
(
CASE WHEN courses_classes_lessons.completed_at > NOW() THEN
'completed'
WHEN courses_classes_lessons.cancelled_at > NOW() THEN
'cancelled'
WHEN courses_classes_lessons.starts_at > NOW() THEN
'upcoming'
ELSE
'incomplete'
END) AS status,
(
SELECT
class_max_consumers - LEAST(consumers_count, class_max_consumers)) AS available_spaces
FROM
courses_classes_lessons
JOIN courses_classes ON courses_classes.id = courses_classes_lessons.class_id
WHERE courses_classes_lessons.id = lessonId;
END
And the execution time for it is around 500μs-1ms.
Thank you all for your help!

Select data using if statement to show Not Available for missing values

I have a database table in mysql that store types like "temperature" , "oxygen_saturation" , "heart_rate" , "systolic_pressure" , "diastolic_pressure".
However, for a given checkup_id, some types may not stored. Those I want to show them as N/A.
My table is the following:
CREATE TABLE `exam` (
`id` int NOT NULL AUTO_INCREMENT,
`checkup_id` int NOT NULL,
`type` varchar(100) NOT NULL,
`result` longtext NOT NULL,
`combined_exam_id` int NOT NULL DEFAULT '-1',
`status` tinyint NOT NULL DEFAULT '1',
`version_id` int NOT NULL,
PRIMARY KEY (`id`,`version_id`),
UNIQUE KEY `exam_id` (`id`,`version_id`) /*!80000 INVISIBLE */,
KEY `checkup_idx` (`checkup_id`) /*!80000 INVISIBLE */,
KEY `version_id_fk_exam_idx` (`version_id`),
CONSTRAINT `exam_ibfk_1` FOREIGN KEY (`checkup_id`) REFERENCES `endorse_checkup` (`id`),
CONSTRAINT `version_id_fk_exam` FOREIGN KEY (`version_id`) REFERENCES `ps_versioning` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=65 DEFAULT CHARSET=utf8;
I have the following select statement:
SELECT end_up.id as checkup_id,
end_e.type,
end_e.result,
FROM endorse_exam as end_e
left JOIN endorse_checkup as end_up on end_up.id = end_e.checkup_id
left JOIN ps_versioning as v ON v.id = end_e.version_id
left JOIN u_users as u ON u.user_id = v.user_id
WHERE end_up.patient_id = 50 AND v.id <= (SELECT IF(164 > -1, 164, max(v.id))) and checkup_id = 25;
Note: I put 164 value by hand, but actually it comes through an API.
Using the SQL code above, I get a result like
However, I want to create a query to get info like:

Possible multiple returned matches joined to a MySQL table

I am not even sure how to ask this question but here's my situation. I use Plex to stream movies at home. I've built a database which I translate to a webpage that I use as an index. With in this database I have a few tables. The main one is called movie_list. 1 of the fields is called Rating which has an association table called assc_movie_genre which simply stores the movie id generated from the main table and a genre id which is read from another association table. There can be multiple movie Id's that are the same which match a Genre, for instance let's say The Matrix falls under the category Action and Sci Fi there will be 2 entries for MovieId each on matching the corresponding genre code. Anyways, my question is I need a query (if possible) that can join all genres to the appropriate row. Right now I have the following query
SELECT a.`Title`,a.`Year`,b.`Rating` FROM movie_list a, assc_rating b WHERE b.`Id` = a.`Rating
But would need to expand it to I guess join the multiple genres that match. I hope that all makes sense.
Thanks in advance
Update
Thanks to your help I am also there. Here is my current query
SELECT a.Title, c.Rating,
GROUP_CONCAT(DISTINCT b.GenreId ORDER BY b.GenreId)
AS Genres FROM assc_movie_genre b, movie_list a, assc_rating c
WHERE a.Id = b.MovieId AND a.Rating = c.Id group by a.Title
ORDER BY a.Title;
But the issue remains where I am just getting the GenreId instead of the genre name. I would assume I need to put a select in there somewhere so that it is pulling the name from the assc_genres tables just not 100% sure where.
Here's what the current output looks like
Title Rating Genres
28 Days Later... R 11,16,17
The concat works great and I'm so close. Thanks again
Update
Here are the queries to create my tables, you can get the structure from here (obviously)
CREATE TABLE IF NOT EXISTS `assc_genres` (
`Id` int(11) NOT NULL AUTO_INCREMENT,
`Genre` varchar(50) NOT NULL DEFAULT '0',
PRIMARY KEY (`Id`)
) ENGINE=InnoDB AUTO_INCREMENT=20 DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS `assc_movie_genre` (
`MovieId` int(11) NOT NULL DEFAULT '0',
`GenreId` int(11) NOT NULL DEFAULT '0',
KEY `FK_assc_movie_genre_movie_list` (`MovieId`),
KEY `FK_assc_movie_genre_assc_genres` (`GenreId`),
CONSTRAINT `FK_assc_movie_genre_movie_list` FOREIGN KEY (`MovieId`) REFERENCES `movie_list` (`Id`),
CONSTRAINT `FK_assc_movie_genre_assc_genres` FOREIGN KEY (`GenreId`) REFERENCES `assc_genres` (`Id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS `assc_rating` (
`Id` int(11) NOT NULL AUTO_INCREMENT,
`Rating` char(50) NOT NULL DEFAULT '0',
PRIMARY KEY (`Id`)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS `assc_status` (
`Id` tinyint(4) NOT NULL,
`Status` char(50) NOT NULL,
PRIMARY KEY (`Id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE IF NOT EXISTS `movie_list` (
`Id` int(11) NOT NULL AUTO_INCREMENT,
`Title` varchar(100) NOT NULL DEFAULT '0',
`Year` year(4) NOT NULL DEFAULT '2000',
`Rating` int(11) NOT NULL DEFAULT '0',
`Folder` varchar(50) NOT NULL DEFAULT '0',
PRIMARY KEY (`Id`),
UNIQUE KEY `Title_Year` (`Title`,`Year`),
KEY `FK_movie_list_assc_rating` (`Rating`),
CONSTRAINT `FK_movie_list_assc_rating` FOREIGN KEY (`Rating`) REFERENCES `assc_rating` (`Id`)
) ENGINE=InnoDB AUTO_INCREMENT=614 DEFAULT CHARSET=latin1;
I am not sure as well if this is what you are asking, but you can join all 3 tables to get the data like
SELECT a.`Title`,
a.`Year`,
b.`Rating`
FROM movie_list a
JOIN assc_movie_genre c ON a.Id = c.movie_id
JOIN assc_rating b ON b.`Id` = c.genre_id;
Per your comment you can use GROUP_CONCAT() like
SELECT a.`Title`,
a.`Year`,
b.`Rating`,
xx.genre_list
FROM movie_list a
JOIN ( select movie_id, genre_id, group_concat(genre) as genre_list
from assc_movie_genre
group by movie_id) xx ON a.Id = xx.movie_id
JOIN assc_rating b ON b.`Id` = xx.genre_id;
You can modify your query like
SELECT a.Title, c.Rating,
GROUP_CONCAT(DISTINCT d.`Genre` ORDER BY d.`Genre`) AS Genres
FROM movie_list a
JOIN assc_movie_genre b ON a.Id = b.MovieId
JOIN assc_rating c ON a.Rating = c.Id
JOIN `assc_genres` d ON b.`GenreId` = d.Id
group by a.Title
ORDER BY a.Title;

MySql Query Optimization with LEFT JOIN

I have a MySQL query to optimized because when this query runs it increase MySQL process CPU usage to 100%. So I did some modification on that query and now it decrease its running time but still CPU usage is near to 100%. I index all search fields in both tables.
These 'milestone_and_events' and 'milestone_and_events_reminders' has one to many relationship ('milestone_and_events' have one or more 'milestone_and_events_reminders') and 'milestone_and_events_reminders' table have nearly 1.5 million records.
Is there any way to further optimize this query, can any one give me a guide ?? thank you
This is original query
SELECT MAE.*
, MAER.reminder_user_id
, MAER.reminder_type
, MAER.id as reminder_id
FROM milestone_and_events AS MAE
LEFT
JOIN milestone_and_events_reminders MAER
ON MAE.id = MAER.milestone_and_events_id
WHERE MAER.alert_time < '$currentTime'
AND MAER.issued ! = 1
AND MAE.completed = 0
GROUP
BY MAE.id
, MAER.reminder_user_id
, MAER.reminder_type
This is my current query
$currentTime = date('Y-m-d H:i:s');
$query = 'SELECT MAE.id, MAE.time, MAER.reminder_user_id, MAER.reminder_type, MAER.id AS reminder_id
FROM milestone_and_events AS MAE
LEFT JOIN milestone_and_events_reminders MAER ON
MAE.id = MAER.milestone_and_events_id AND
MAER.issued =0 AND
MAER.alert_time < "' . $currentTime . '" AND
MAE.completed =0
WHERE MAER.reminder_type != "onTime"
GROUP BY MAER.milestone_and_events_id,MAER.reminder_user_id,MAER.reminder_type';
UPDATE 1
this 'milestone_and_events' table have nearly 200 entries.
CREATE TABLE `milestone_and_events` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`description` varchar(250) NOT NULL,
`time` datetime NOT NULL,
`attached_type` int(11) DEFAULT NULL,
`attached_type_value` int(11) DEFAULT NULL,
`completed` int(11) NOT NULL DEFAULT '0',
`type` int(11) NOT NULL DEFAULT '1',
PRIMARY KEY (`id`),
KEY `time` (`time`),
KEY `completed` (`completed`),
KEY `id` (`id`,`completed`)
) ENGINE=InnoDB AUTO_INCREMENT=154 DEFAULT CHARSET=utf8
'milestone_and_events_reminders' table have nearly 1.5 million entries.
CREATE TABLE `milestone_and_events_reminders` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`reminder_user_id` int(11) NOT NULL,
`milestone_and_events_id` int(11) NOT NULL,
`alert_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`issued` tinyint(11) NOT NULL DEFAULT '0',
`reminder_type` enum('upComming','delayed','onTime') NOT NULL,
PRIMARY KEY (`id`),
KEY `user_id` (`reminder_user_id`),
KEY `reminder_type` (`reminder_type`),
KEY `issued` (`issued`),
KEY `milestone_and_events_id` (`milestone_and_events_id`, `issued`,`reminder_type`),
CONSTRAINT `milestone_and_events_reminders_ibfk_1` FOREIGN KEY (`milestone_and_events_id`)
REFERENCES `milestone_and_events` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=3323544 DEFAULT CHARSET=utf8
This is your query:
SELECT MAE.id, MAE.time, MAER.reminder_user_id, MAER.reminder_type, MAER.id AS reminder_id
FROM milestone_and_events AS MAE LEFT JOIN
milestone_and_events_reminders MAER
ON MAE.id = MAER.milestone_and_events_id AND
MAER.issued = 0 AND
MAER.alert_time < "' . $currentTime . '"
WHERE MAE.completed = 0 AND MAER.reminder_type <> "onTime"
GROUP BY MAE.id, MAER.reminder_user_id, MAER.reminder_type;
Note: I moved the condition MAE.completed = 0 from the on clause to the where clause. It does nothing in the on clause. And, I changed the first key in the group by to match the SELECT.
The best indexes for this query are composite indexes : milestone_and_events(completed, id) and milestone_and_events_reminders(milestone_and_events_id, issued, alert_time, reminder_type).
My guess is that putting the completed = 0 where it belongs will reduce the amount of data and improve the performance of the query.

MySQL : Avoid Temporary/Filesort Caused by GROUP BY Clause in Views

I'm planning on creating a view using this SQL SELECT, but the explain for it shows it's using temporary and using filesort. I can't figure out what indices I need in order to fix this problem. Mostly, I'm wondering why it's using filesort instead of using an index to sort.
Table bn_ms_gt_generaltransaction has 7 million records
Table bn_pd_mp_masterprofile has few hundred records
Table bn_pd_sc_scheme also has few hundred records
Here are create table statements:
CREATE TABLE `bn_ms_gt_generaltransaction` (
`GNTRDATE` date NOT NULL,
`GNTRNUMBER` decimal(10,0) NOT NULL,
`GNTRSUBNUMBER` decimal(5,0) NOT NULL,
`ORGACODE` varchar(20) NOT NULL,
`LOCACODE` varchar(20) NOT NULL,
`PRODCODE` varchar(20) DEFAULT NULL,
`SCHEMCODE` varchar(20) DEFAULT NULL,
`BKMSNUMBER` varchar(50) DEFAULT NULL,
`TRANCODE` varchar(20) NOT NULL,
`GNTRVALUEDAT` date NOT NULL,
`GNTRDRCR` varchar(20) NOT NULL,
`GLACCODE` varchar(50) NOT NULL,
`CURRCODE` varchar(20) NOT NULL,
`GNTRAMTFC` decimal(15,2) NOT NULL,
`GNTRSTATUS` varchar(2) DEFAULT NULL,
`TMODCODE` varchar(20) DEFAULT NULL,
PRIMARY KEY (`ORGACODE`,`LOCACODE`,`GNTRDATE`,`GNTRNUMBER`,`GNTRSUBNUMBER`),
KEY `CC10921796764` (`TMODCODE`),
KEY `SQL0407170412326` (`TRANCODE`),
KEY `SQL0407170412328` (`CURRCODE`),
KEY `SQL0407170412329` (`ORGACODE`,`GLACCODE`),
KEY `GNTRDRCR` (`GNTRDRCR`),
KEY `GLACCODE` (`GLACCODE`),
KEY `GT_ACC_NUM` (`PRODCODE`,`SCHEMCODE`,`BKMSNUMBER`),
KEY `GT_PK` (`GNTRDATE`,`GNTRNUMBER`,`GNTRSUBNUMBER`),
KEY `GT_EFFECTDATE` (`GNTREFFECTDAT`),
KEY `GT_GL_CODE` (`ORGACODE`,`LOCACODE`,`CURRCODE`,`GLACCODE`),
KEY `INDEX_GNTRDATE` (`GNTRDATE`),
KEY `INDEX_GNTRVALUEDAT` (`GNTRVALUEDAT`),
KEY `GT_VOUCHER` (`GNTRDATE`,`GNTRNUMBER`),
KEY `REPORT_INDEX_2` (`PRODCODE`,`SCHEMCODE`,`BKMSNUMBER`,`GNTRDATE`,`TRANCODE`,`GLACCODE`,`GNTRSTATUS`,`SGT_REVERSALTAG`,`GNTRAMTFC`),
KEY `INDEX_LRDL_GROUPBY` (`PRODCODE`,`SCHEMCODE`,`BKMSNUMBER`,`GNTRDATE`,`GNTRVALUEDAT`,`GNTREFFECTDAT`,`GNTRNUMBER`,`TRANCODE`,`TMODCODE`),
KEY `INDEX_LRDL_GROUPBY_LOOSE` (`PRODCODE`,`SCHEMCODE`,`BKMSNUMBER`,`GNTRDATE`,`GNTRVALUEDAT`,`GNTREFFECTDAT`,`GNTRNUMBER`,`TRANCODE`,`TMODCODE`,`GNTRAMTFC`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
CREATE TABLE `bn_pd_mp_masterprofile` (
`PRODCODE` varchar(20) NOT NULL,
`ORGACODE` varchar(20) DEFAULT NULL,
`GLACCODE` varchar(50) DEFAULT NULL,
PRIMARY KEY (`PRODCODE`),
KEY `SQL040126114206101` (`ORGACODE`,`GLACCODE`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
CREATE TABLE `bn_pd_sc_scheme` (
`PRODCODE` varchar(20) NOT NULL,
`SCHEMCODE` varchar(20) NOT NULL,
`ORGACODE` varchar(20) DEFAULT NULL,
`GLACCODE` varchar(50) DEFAULT NULL,
`CURRCODE` varchar(20) DEFAULT NULL,
`SCHEPURPOSE` varchar(100) DEFAULT NULL,
`CODE_PENALTY` varchar(50) DEFAULT NULL,
PRIMARY KEY (`PRODCODE`,`SCHEMCODE`),
KEY `SQL040126114217543` (`CURRCODE`),
KEY `CC1318422668625` (`PST_PSTYCODE`),
KEY `SQL040126114217560` (`ORGACODE`,`GLACCODE`),
KEY `CC1205848198000` (`ORGACODE`,`CODE_PENALTY`),
CONSTRAINT `CC1205848198000` FOREIGN KEY (`ORGACODE`, `CODE_PENALTY`) REFERENCES `pr_gl_ca_account` (`ORGACODE`, `GLACCODE`),
CONSTRAINT `SQL040126114217543` FOREIGN KEY (`CURRCODE`) REFERENCES `pr_gn_cr_currency` (`CURRCODE`),
CONSTRAINT `SQL040126114217560` FOREIGN KEY (`ORGACODE`, `GLACCODE`) REFERENCES `pr_gl_ca_account` (`ORGACODE`, `GLACCODE`),
CONSTRAINT `SQL040126114217562` FOREIGN KEY (`PRODCODE`) REFERENCES `bn_pd_mp_masterprofile` (`PRODCODE`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
And following are two views select
SELECT
`a`.`PRODCODE` AS `PRODCODE`,
`a`.`SCHEMCODE` AS `SCHEMCODE`,
`a`.`BKMSNUMBER` AS `BKMSNUMBER`,
`a`.`GNTRDATE` AS `GNTRDATE`,
`a`.`GNTRVALUEDAT` AS `GNTRVALUEDAT`,
`a`.`GNTREFFECTDAT` AS `GNTREFFECTDAT`,
`a`.`GNTRNUMBER` AS `GNTRNUMBER`,
`a`.`TRANCODE` AS `TRANCODE`,
`a`.`TMODCODE` AS `TMODCODE`,
(CASE WHEN ((`a`.`TRANCODE` = '29') AND (`a`.`GNTRDRCR` = 'DR') AND (`a`.`GLACCODE` = `sc`.`GLACCODE`)) THEN COALESCE(`a`.`GNTRAMTFC`,0) END) AS `DISB_AMOUNT`,
(CASE WHEN ((`a`.`TRANCODE` = '29') AND (`a`.`GNTRDRCR` = 'CR') AND (`a`.`GLACCODE` = `sc`.`GLACCODE`)) THEN COALESCE(`a`.`GNTRAMTFC`,0) END) AS `DISB_AMOUNT_REV`,
(CASE WHEN ((`a`.`TRANCODE` IN ('30','32')) AND (`a`.`GNTRDRCR` = 'CR') AND (COALESCE(`a`.`GNTRSTATUS`,'P') 'C') AND (`a`.`GLACCODE` = `sc`.`GLACCODE`)) THEN COALESCE(`a`.`GNTRAMTFC`,0) END) AS `PRINCIPAL_AMOUNT`,
(CASE WHEN ((`a`.`TRANCODE` IN ('30','32')) AND (`a`.`GNTRDRCR` = 'CR') AND (COALESCE(`a`.`GNTRSTATUS`,'P') = 'C') AND (`a`.`GLACCODE` = `sc`.`GLACCODE`)) THEN COALESCE(`a`.`GNTRAMTFC`,0) END) AS `PRINCIPAL_AMOUNT_CANCEL`,
(CASE WHEN ((`a`.`TRANCODE` IN ('30','32')) AND (`a`.`GNTRDRCR` = 'DR') AND (`a`.`GLACCODE` = `sc`.`GLACCODE`)) THEN COALESCE(`a`.`GNTRAMTFC`,0) END) AS `PRINCIPAL_AMOUNT_REV`,
(CASE WHEN ((`a`.`TRANCODE` IN ('30','73','32')) AND (`a`.`GNTRDRCR` = 'CR') AND (`a`.`GLACCODE` = `sc`.`SCHEPURPOSE`)) THEN COALESCE(`a`.`GNTRAMTFC`,0) END) AS `MARKUP_AMOUNT`,
(CASE WHEN ((`a`.`TRANCODE` IN ('30','73','32')) AND (`a`.`GNTRDRCR` = 'DR') AND (`a`.`GLACCODE` = `sc`.`SCHEPURPOSE`)) THEN COALESCE(`a`.`GNTRAMTFC`,0) END) AS `MARKUP_AMOUNT_REV`,
(CASE WHEN ((`a`.`TRANCODE` IN ('30','55','32')) AND (`a`.`GNTRDRCR` = 'CR') AND (`a`.`GLACCODE` = `sc`.`CODE_PENALTY`)) THEN COALESCE(`a`.`GNTRAMTFC`,0) END) AS `OTHER_AMOUNT`,
(CASE WHEN ((`a`.`TRANCODE` IN ('30','55','32')) AND (`a`.`GNTRDRCR` = 'DR') AND (`a`.`GLACCODE` = `sc`.`CODE_PENALTY`)) THEN COALESCE(`a`.`GNTRAMTFC`,0) END) AS `OTHER_AMOUNT_REV`
FROM ((`bn_ms_gt_generaltransaction` `a`
JOIN `bn_pd_mp_masterprofile` `mp`
ON (((`mp`.`PRODCODE` = `a`.`PRODCODE`)
AND (`mp`.`PPC_PRCGCODE` = 'LOAN'))))
JOIN `bn_pd_sc_scheme` `sc`
ON (((`a`.`PRODCODE` = `sc`.`PRODCODE`)
AND (`a`.`SCHEMCODE` = `sc`.`SCHEMCODE`))))
WHERE ((`a`.`TRANCODE` IN('29','30','55','73','32'))
AND ((`a`.`GLACCODE` = `sc`.`GLACCODE`)
OR (`a`.`GLACCODE` = `sc`.`SCHEPURPOSE`)
OR (`a`.`GLACCODE` = `sc`.`CODE_PENALTY`))
AND ((COALESCE(`a`.`GNTRSTATUS`,'P') 'C')
OR ((COALESCE(`a`.`GNTRSTATUS`,'P') = 'C')
AND (`a`.`GNTRVALUEDAT` = `a`.`GNTRDATE`)
AND (`a`.`GNTRVALUEDAT` `a`.`GNTREFFECTDAT`))))
SELECT
`a`.`DMP_PRODCODE` AS `DMP_PRODCODE`,
`a`.`DCS_SCHEMCODE` AS `DCS_SCHEMCODE`,
`a`.`MBM_BKMSNUMBER` AS `MBM_BKMSNUMBER`,
`a`.`SGT_GNTRDATE` AS `SGT_GNTRDATE`,
`a`.`SGT_GNTRVALUEDAT` AS `SGT_GNTRVALUEDAT`,
`a`.`SGT_GNTREFFECTDAT` AS `SGT_GNTREFFECTDAT`,
`a`.`SGT_GNTRNUMBER` AS `SGT_GNTRNUMBER`,
`a`.`PTR_TRANCODE` AS `PTR_TRANCODE`,
`a`.`PTM_TMODCODE` AS `PTM_TMODCODE`,
SUM(COALESCE(`a`.`DISB_AMOUNT`,0)) AS `DISB_AMOUNT`,
SUM(COALESCE(`a`.`DISB_AMOUNT_REV`,0)) AS `DISB_AMOUNT_REV`,
(SUM(COALESCE(`a`.`DISB_AMOUNT`,0)) - SUM(COALESCE(`a`.`DISB_AMOUNT_REV`,0))) AS `TOTAL_DISB`,
SUM(COALESCE(`a`.`PRINCIPAL_AMOUNT`,0)) AS `PRINCIPAL_AMOUNT`,
SUM(COALESCE(`a`.`PRINCIPAL_AMOUNT_REV`,0)) AS `PRINCIPAL_AMOUNT_REV`,
SUM(COALESCE(`a`.`PRINCIPAL_AMOUNT_CANCEL`,0)) AS `PRINCIPAL_AMOUNT_CANCEL`,
(SUM(COALESCE(`a`.`PRINCIPAL_AMOUNT`,0)) - SUM(COALESCE(`a`.`PRINCIPAL_AMOUNT_REV`,0))) AS `TOTAL_PRINCIPAL`,
SUM(COALESCE(`a`.`MARKUP_AMOUNT`,0)) AS `MARKUP_AMOUNT`,
SUM(COALESCE(`a`.`MARKUP_AMOUNT_REV`,0)) AS `MARKUP_AMOUNT_REV`,
(SUM(COALESCE(`a`.`MARKUP_AMOUNT`,0)) - SUM(COALESCE(`a`.`MARKUP_AMOUNT_REV`,0))) AS `TOTAL_MARKUP`,
SUM(COALESCE(`a`.`OTHER_AMOUNT`,0)) AS `OTHER_AMOUNT`,
SUM(COALESCE(`a`.`OTHER_AMOUNT_REV`,0)) AS `OTHER_AMOUNT_REV`,
(SUM(COALESCE(`a`.`OTHER_AMOUNT`,0)) - SUM(COALESCE(`a`.`OTHER_AMOUNT_REV`,0))) AS `TOTAL_OTHER`
FROM `test_view` `a`
GROUP BY `a`.`DMP_PRODCODE`,`a`.`DCS_SCHEMCODE`,`a`.`MBM_BKMSNUMBER`,`a`.`SGT_GNTRDATE`,`a`.`SGT_GNTRVALUEDAT`,`a`.`SGT_GNTREFFECTDAT`,`a`.`SGT_GNTRNUMBER`,`a`.`PTR_TRANCODE`,`a`.`PTM_TMODCODE`$$
The second view uses the first.
Output from explain:
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY ALL 3564
2 DERIVED mp ref PRIMARY,SQL040126114206120 SQL040126114206120 63 const 3 Using where; Using index; Using temporary; Using filesort
2 DERIVED sc ref PRIMARY PRIMARY 62 mp.PRODCODE 2
2 DERIVED a ref SQL0407170412326,PCA_GLACCODE,GT_ACC_NUM,REPORT_INDEX_2,INDEX_LRDL_GROUPBY,INDEX_LRDL_GROUPBY_LOOSE INDEX_LRDL_GROUPBY 126 mp.PRODCODE,sc.SCHEMCODE 594 Using index condition; Using where
Now, the top derived table shows using temporary; using filesort.
Even creating index doesn't help. I tried to create index on group by for loose index but doesn't work. AM I missing something?
I have explored about it and found many workarounds but none fits my example. Some suggest to do aggregation in subquery before join which is not possible as MySQL doesn't allow subquery in FROM clause.
Creating another view doesn't work as I need to filter records before summation.
So, can someone help me with fixing the issue.