I'm working on a system that administrates courses, with multiple classes, with multiple lessons and multiple consumers on them. As the system grows more data were required so with some performance issues I've decided to go with SQL Views. We're using MySQL.
So I've replaced old calls to the DB (for example for the single lesson)
select * from `courses_classes_lessons` where `courses_classes_lessons`.`deleted_at` is null limit 1;
select count(consumer_id) as consumers_count from `courses_classes_lessons_consumers` where `lesson_id` = '448' limit 1;
select `max_consumers` from `courses_classes` where `id` = '65' limit 1;
select `id` from `courses_classes_lessons` left join `courses_classes_lessons_consumers` on `courses_classes_lessons_consumers`.`lesson_id` = `courses_classes_lessons`.`id` where `id` = '448' group by `courses_classes_lessons`.`id` having count(courses_classes_lessons_consumers.consumer_id) < '4' limit 1;
select courses_classes.max_consumers - LEAST(count(courses_classes_lessons_consumers.consumer_id), courses_classes.max_consumers) as available_spaces from `courses_classes_lessons` left join `courses_classes_lessons_consumers` on `courses_classes_lessons_consumers`.`lesson_id` = `courses_classes_lessons`.`id` left join `courses_classes` on `courses_classes_lessons`.`class_id` = `courses_classes`.`id` where `courses_classes_lessons`.`id` = '448' group by `courses_classes`.`id` limit 1;
The above took around 4-5ms
with the SQL View as follow:
CREATE OR REPLACE VIEW `courses_classes_lessons_view` AS
SELECT
courses_classes_lessons.id AS lesson_id,
(SELECT
max_consumers
FROM
courses_classes
WHERE
id = courses_classes_lessons.class_id
LIMIT 1) AS class_max_consumers,
(SELECT
count(consumer_id)
FROM
courses_classes_lessons_consumers
WHERE
lesson_id = courses_classes_lessons.id) AS consumers_count,
(SELECT
CASE WHEN consumers_count >= class_max_consumers THEN
TRUE
ELSE
FALSE
END AS is_full) AS is_full,
(CASE WHEN courses_classes_lessons.completed_at > NOW() THEN
'completed'
WHEN courses_classes_lessons.cancelled_at > NOW() THEN
'cancelled'
WHEN courses_classes_lessons.starts_at > NOW() THEN
'upcoming'
ELSE
'incomplete'
END) AS status,
(SELECT
class_max_consumers - LEAST(consumers_count, class_max_consumers)) AS available_spaces
FROM
courses_classes_lessons
The problem I'm having is that doesn't matter if I'm loading the whole View or a single row from it - it always takes about 6-9s to load! But when I've tried the same query with a WHERE clause it takes about 500μs. I'm new to SQL View and confused - why there are no indexes/primary keys that I could use to load a single row quickly? Am I doing something wrong?
EXPLAIN RESULT
INSERT INTO `courses_classes` (`id`, `select_type`, `table`, `partitions`, `type`, `possible_keys`, `key`, `key_len`, `ref`, `rows`, `filtered`, `Extra`) VALUES
(1, 'PRIMARY', 'courses_classes_lessons', NULL, 'ALL', NULL, NULL, NULL, NULL, 478832, 100.00, NULL),
(3, 'DEPENDENT SUBQUERY', 'courses_classes_lessons_consumers', NULL, 'ref', 'PRIMARY,courses_classes_lessons_consumers_lesson_id_index', 'courses_classes_lessons_consumers_lesson_id_index', '4', 'api.courses_classes_lessons.id', 3, 100.00, 'Using index'),
(2, 'DEPENDENT SUBQUERY', 'courses_classes', NULL, 'eq_ref', 'PRIMARY,courses_classes_id_parent_id_index', 'PRIMARY', '4', 'api.courses_classes_lessons.class_id', 1, 100.00, NULL);
TABLE STRUCTURE
Lessons
CREATE TABLE `courses_classes_lessons` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`franchisee_id` int(10) unsigned NOT NULL,
`class_id` int(10) unsigned NOT NULL,
`instructor_id` int(10) unsigned NOT NULL,
`instructor_rate` int(10) unsigned NOT NULL DEFAULT '0',
`instructor_total` int(10) unsigned NOT NULL DEFAULT '0',
`instructor_paid` tinyint(1) NOT NULL DEFAULT '0',
`starts_at` timestamp NULL DEFAULT NULL,
`ends_at` timestamp NULL DEFAULT NULL,
`completed_at` timestamp NULL DEFAULT NULL,
`cancelled_at` timestamp NULL DEFAULT NULL,
`cancelled_reason` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
`cancelled_reason_extra` varchar(255) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
`deleted_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `courses_classes_lessons_franchisee_id_foreign` (`franchisee_id`),
KEY `courses_classes_lessons_class_id_foreign` (`class_id`),
KEY `courses_classes_lessons_instructor_id_foreign` (`instructor_id`),
KEY `courses_classes_lessons_starts_at_ends_at_index` (`starts_at`,`ends_at`),
KEY `courses_classes_lessons_completed_at_index` (`completed_at`),
KEY `courses_classes_lessons_cancelled_at_index` (`cancelled_at`),
KEY `courses_classes_lessons_class_id_deleted_at_index` (`class_id`,`deleted_at`),
KEY `courses_classes_lessons_deleted_at_index` (`deleted_at`),
KEY `class_ownership_index` (`class_id`,`starts_at`,`cancelled_at`,`deleted_at`),
CONSTRAINT `courses_classes_lessons_class_id_foreign` FOREIGN KEY (`class_id`) REFERENCES `courses_classes` (`id`) ON DELETE CASCADE,
CONSTRAINT `courses_classes_lessons_franchisee_id_foreign` FOREIGN KEY (`franchisee_id`) REFERENCES `franchisees` (`id`) ON DELETE CASCADE,
CONSTRAINT `courses_classes_lessons_instructor_id_foreign` FOREIGN KEY (`instructor_id`) REFERENCES `instructors` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=487853 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
Lessons consumers
CREATE TABLE `courses_classes_lessons_consumers` (
`lesson_id` int(10) unsigned NOT NULL,
`consumer_id` int(10) unsigned NOT NULL,
`present` tinyint(1) DEFAULT NULL,
`plan_id` int(10) unsigned DEFAULT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`lesson_id`,`consumer_id`),
KEY `courses_classes_lessons_consumers_consumer_id_foreign` (`consumer_id`),
KEY `courses_classes_lessons_consumers_plan_id_foreign` (`plan_id`),
KEY `courses_classes_lessons_consumers_lesson_id_index` (`lesson_id`),
KEY `courses_classes_lessons_consumers_present_index` (`present`),
CONSTRAINT `courses_classes_lessons_consumers_consumer_id_foreign` FOREIGN KEY (`consumer_id`) REFERENCES `customers_consumers` (`id`) ON DELETE CASCADE,
CONSTRAINT `courses_classes_lessons_consumers_lesson_id_foreign` FOREIGN KEY (`lesson_id`) REFERENCES `courses_classes_lessons` (`id`) ON DELETE CASCADE,
CONSTRAINT `courses_classes_lessons_consumers_plan_id_foreign` FOREIGN KEY (`plan_id`) REFERENCES `customers_plans` (`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
From classes it's only using max_consumers int(10) unsigned NOT NULL DEFAULT '0',
UPDATE 1
I've changed SQL View to the following one:
CREATE OR REPLACE VIEW `courses_classes_lessons_view` AS
SELECT
courses_classes_lessons.id AS lesson_id,
courses_classes.max_consumers AS class_max_consumers,
lessons_consumers.consumers_count AS consumers_count,
(
SELECT
CASE WHEN consumers_count >= class_max_consumers THEN
TRUE
ELSE
FALSE
END AS is_full) AS is_full,
(
CASE WHEN courses_classes_lessons.completed_at > NOW() THEN
'completed'
WHEN courses_classes_lessons.cancelled_at > NOW() THEN
'cancelled'
WHEN courses_classes_lessons.starts_at > NOW() THEN
'upcoming'
ELSE
'incomplete'
END) AS status,
(
SELECT
class_max_consumers - LEAST(consumers_count, class_max_consumers)) AS available_spaces
FROM
courses_classes_lessons
JOIN courses_classes ON courses_classes.id = courses_classes_lessons.class_id
JOIN (
SELECT
lesson_id,
count(*) AS consumers_count
FROM
courses_classes_lessons_consumers
GROUP BY
courses_classes_lessons_consumers.lesson_id) AS lessons_consumers ON lessons_consumers.lesson_id = courses_classes_lessons.id;
and even though the SELECT query itself seems to be way slower than the previous one then as the View it seems to perform way better. It's still not as fast as I wish it will be but it's a step forward.
Overall improvement jumps from 6-7s to around 800ms, the aim here is in the area of 500μs-1ms. Any adivces how I can improve my SQL View more?
UPDATE 2
Ok, I've found the bottleneck! Again - it's kinda similar to the last one (SELECT query works fast for a single row, but SQL VIEW is trying to access the whole table at once every time.
My new lesson SQL VIEW:
CREATE OR REPLACE VIEW `courses_classes_lessons_view` AS
SELECT
courses_classes_lessons.id AS lesson_id,
courses_classes.max_consumers AS class_max_consumers,
IFNULL(lessons_consumers.consumers_count,0) AS consumers_count,
(
SELECT
CASE WHEN consumers_count >= class_max_consumers THEN
TRUE
ELSE
FALSE
END AS is_full) AS is_full,
(
CASE WHEN courses_classes_lessons.completed_at > NOW() THEN
'completed'
WHEN courses_classes_lessons.cancelled_at > NOW() THEN
'cancelled'
WHEN courses_classes_lessons.starts_at > NOW() THEN
'upcoming'
ELSE
'incomplete'
END) AS status,
(
SELECT
IFNULL(class_max_consumers, 0) - LEAST(IFNULL(consumers_count,0), class_max_consumers)) AS available_spaces
FROM
courses_classes_lessons
JOIN courses_classes ON courses_classes.id = courses_classes_lessons.class_id
LEFT JOIN courses_classes_lessons_consumers_view AS lessons_consumers ON lessons_consumers.lesson_id = courses_classes_lessons.id;
Another SQL View - this time for consumers:
CREATE OR REPLACE VIEW `courses_classes_lessons_consumers_view` AS
SELECT
lesson_id,
IFNULL(count(
consumer_id),0) AS consumers_count
FROM
courses_classes_lessons_consumers
GROUP BY
courses_classes_lessons_consumers.lesson_id;
And looks like this one is the trouble maker! The consumers table is above, and here is the explain for the above SELECT query:
INSERT INTO `courses_classes_lessons_consumers` (`id`, `select_type`, `table`, `partitions`, `type`, `possible_keys`, `key`, `key_len`, `ref`, `rows`, `filtered`, `Extra`)
VALUES(1, 'SIMPLE', 'courses_classes_lessons_consumers', NULL, 'index', 'PRIMARY,courses_classes_lessons_consumers_consumer_id_foreign,courses_classes_lessons_consumers_plan_id_foreign,courses_classes_lessons_consumers_lesson_id_index,courses_classes_lessons_consumers_present_index', 'courses_classes_lessons_consumers_lesson_id_index', '4', NULL, 1330649, 100.00, 'Using index');
Any idea how to spread up this count?
Consider writing a Stored procedure; it may be able to get the 448 put into place to be better optimized.
If you know there will be only one row (such as when doing COUNT(*)), skip the LIMIT 1.
Unless consumer_id might be NULL, use COUNT(*) instead of COUNT(consumer_id).
A LIMIT without an ORDER BY leaves you getting a random row.
If courses_classes_lessons_consumers is a many-to-many mapping table, I will probably have some index advice after I see SHOW CREATE TABLE.
Which of the 5 SELECTs is the slowest?
After many attempts, it looks like that the Procedure way is the best approach and I won't be spending more time on the SQL Views
Here's the procedure I wrote:
CREATE PROCEDURE `LessonData`(
IN lessonId INT(10)
)
BEGIN
SELECT
courses_classes_lessons.id AS lesson_id,
courses_classes.max_consumers AS class_max_consumers,
IFNULL((SELECT
count(consumer_id) as consumers_count
FROM
courses_classes_lessons_consumers
WHERE
lesson_id = courses_classes_lessons.id
GROUP BY
courses_classes_lessons_consumers.lesson_id), 0) AS consumers_count,
(
SELECT
CASE WHEN consumers_count >= class_max_consumers THEN
TRUE
ELSE
FALSE
END) AS is_full,
(
CASE WHEN courses_classes_lessons.completed_at > NOW() THEN
'completed'
WHEN courses_classes_lessons.cancelled_at > NOW() THEN
'cancelled'
WHEN courses_classes_lessons.starts_at > NOW() THEN
'upcoming'
ELSE
'incomplete'
END) AS status,
(
SELECT
class_max_consumers - LEAST(consumers_count, class_max_consumers)) AS available_spaces
FROM
courses_classes_lessons
JOIN courses_classes ON courses_classes.id = courses_classes_lessons.class_id
WHERE courses_classes_lessons.id = lessonId;
END
And the execution time for it is around 500μs-1ms.
Thank you all for your help!
Related
I am working with agricultural product management system. I have a question regarding a MySQL query. I would like to know how to create the same query using Laravel query builder:
SELECT
vegitables.name, vegitables.image, vegitables.catagory,
AVG(price_wholesale),
SUM(CASE WHEN rank = 1 THEN price_wholesale ELSE 0 END) today,
SUM(CASE WHEN rank = 2 THEN price_wholesale ELSE 0 END) yesterday
FROM (
SELECT
veg_id, price_wholesale, price_date,
RANK() OVER (PARTITION BY veg_id ORDER BY price_date DESC) as rank
FROM old_veg_prices
) p
INNER JOIN vegitables ON p.veg_id = vegitables.id
WHERE rank in (1,2)
GROUP BY veg_id
This Output result get when run query in database:
Following two table are used to get today price yesterday price and price average get from each product.
CREATE TABLE `vegitables` (
`id` bigint(20) UNSIGNED NOT NULL,
`name` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`image` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`catagory` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`total_area` int(11) NOT NULL COMMENT 'Total area of culativate in Sri Lanka (Ha)',
`total_producation` int(11) NOT NULL COMMENT 'Total production particular product(mt)',
`annual_crop_count` int(11) NOT NULL COMMENT 'how many time can crop pre year',
`short_dis` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
ALTER TABLE `vegitables`
ADD PRIMARY KEY (`id`);
ALTER TABLE `vegitables`
MODIFY `id` bigint(20) UNSIGNED NOT NULL AUTO_INCREMENT, AUTO_INCREMENT=3;
COMMIT;
CREATE TABLE `old_veg_prices` (
`id` bigint(20) UNSIGNED NOT NULL,
`veg_id` int(11) NOT NULL,
`price_wholesale` double(8,2) NOT NULL,
`price_retial` double(8,2) NOT NULL,
`price_location` int(11) NOT NULL,
`price_date` date NOT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
ALTER TABLE `old_veg_prices`
ADD PRIMARY KEY (`id`);
ALTER TABLE `old_veg_prices`
MODIFY `id` bigint(20) UNSIGNED NOT NULL AUTO_INCREMENT, AUTO_INCREMENT=6;
COMMIT;
I try this site to convert to MySQL query to query builder code. But it show some error's could find it out. Any Way i want to run this code in Laravel with any method??
Your query will not return the data for yesterday and today; it will return the data for two most recent dates (e.g. if today is 2021-11-01 and most recent two dates for for carrots are 2021-10-25 and 2021-10-20 it will use those two dates). Using RANK() ... IN (1, 2) is also incorrect because it can return ranks such as 1 followed by 3 instead of 2.
To get today and yesterday prices you don't need window functions. Just use appropriate where clause and conditional aggregation:
SELECT vegitables.name
, vegitables.image
, vegitables.catagory
, AVG(old_veg_prices.price_wholesale) AS avgwholesale
, SUM(CASE WHEN old_veg_prices.price_date = CURRENT_DATE - INTERVAL 1 DAY THEN old_veg_prices.price_wholesale END) AS yesterday
, SUM(CASE WHEN old_veg_prices.price_date = CURRENT_DATE THEN old_veg_prices.price_wholesale END) AS today
FROM vegitables
INNER JOIN old_veg_prices ON vegitables.id = old_veg_prices.veg_id
WHERE old_veg_prices.price_date IN (CURRENT_DATE - INTERVAL 1 DAY, CURRENT_DATE)
GROUP BY vegitables.id -- other columns from vegitables table are functionally dependent on primary key
The Laravel equivalent would be:
DB::table('vegitables')
->Join('old_veg_prices', 'old_veg_prices.veg_id', '=', 'vegitables.id')
->whereRaw('old_veg_prices.price_date IN (CURRENT_DATE - INTERVAL 1 DAY, CURRENT_DATE)')
->select(
'vegitables.name',
'vegitables.image',
'vegitables.catagory',
DB::raw('AVG(old_veg_prices.price_wholesale) AS avgwholesale'),
DB::raw('SUM(CASE WHEN old_veg_prices.price_date = CURRENT_DATE - INTERVAL 1 DAY THEN old_veg_prices.price_wholesale END) AS yesterday'),
DB::raw('SUM(CASE WHEN old_veg_prices.price_date = CURRENT_DATE THEN old_veg_prices.price_wholesale END) AS today')
)
->groupBy(
'vegitables.id',
'vegitables.name',
'vegitables.image',
'vegitables.catagory'
)
->get();
"Query builder" features of abstraction products often leave out some possible SQL constructs. I recommend you abandon the goal of reverse engineering SQL back to Laravel and simply perform the "raw" query.
Also...
rank() OVER (PARTITION BY veg_id ORDER BY price_date DESC) as rank
requires MySQL 8.0 (MariaDB 10.2).
And suggest you avoid the alias "rank" since that is identical to the name of a function.
I have two tables, main_part (3k records) and part_details (25k records)
I tried the following indexes but explain always returns full table scan of 25k records as opposed to about 2k of matched records and Using where; Using temporary; Using filesort
ALTER TABLE `main_part` ADD INDEX `main_part_index_1` (`unit`);
ALTER TABLE `part_details` ADD INDEX `part_details_index_1` (`approved`, `display`, `country`, `id`, `price`);
Here is my query:
SELECT a.part_id, b.my_title,
b.price, a.type,
a.unit, a.certification,
b.my_image,
b.price/a.unit AS priceW
FROM main_part AS a
INNER JOIN part_details AS b ON a.part_id=b.id
WHERE b.approved = 'Yes'
AND b.display = 'On'
AND b.country = 'US'
AND a.unit >= 300
ORDER BY b.price ASC LIMIT 50
One thing that I am aware of is that a.part_id is not a Primary Key in main_part table. Could this be a culprit?
Create tables SQL:
CREATE TABLE `main_part` (
`id` smallint(6) NOT NULL AUTO_INCREMENT,
`part_id` mediumint(9) NOT NULL DEFAULT '0',
`type` varchar(50) NOT NULL DEFAULT '',
`unit` varchar(50) NOT NULL DEFAULT '',
`certification` varchar(50) NOT NULL DEFAULT '',
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `part_details` (
`id` mediumint(9) NOT NULL AUTO_INCREMENT,
`asn` varchar(50) NOT NULL DEFAULT '',
`country` varchar(5) NOT NULL DEFAULT '',
`my_title` varchar(200) NOT NULL DEFAULT '',
`display` varchar(5) NOT NULL DEFAULT 'On',
`approved` varchar(5) NOT NULL DEFAULT 'No',
`price` decimal(7,3) NOT NULL DEFAULT '0.000',
`my_image` varchar(250) NOT NULL DEFAULT '',
`update_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `countryasn` (`country`,`asn`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
For your query the more important index is the JOIN condition and as you are already aware a.part_id isn't primary key, so doesn't have a default index and your first try should be:
ALTER TABLE `main_part` ADD INDEX `main_part_index_1` (`part_id`,`unit`);
Because we are interested on the JOIN condition first you should also change the second index to
ALTER TABLE `part_details` ADD INDEX `part_details_index_1`
(`id`, `approved`, `display`, `country`, `price`);
order matters in the index
Another tip is you start with the basic query:
SELECT *
FROM main_part AS a
INNER JOIN part_details AS b ON a.part_id=b.id
Add index for part_id and id check the explain plan and then start adding condition and updating the index if required.
It seems that most columns used for filtering in part_details aren't very selective (display is probably an On/off switch, country is probably very similar in many products, etc.).
In some cases, when the WHERE clause is not very selective, MySQL may choose to use an index that better suits the ORDER BY clause.
I would try to create this index as well and check in the explain plan if there is any changes:
ALTER TABLE `part_details` ADD INDEX `part_details_price_index` (`price`);
For this query:
SELECT mp.part_id, pd.my_title, pd.price, mp.type,
mp.unit, mp.certification, pd.my_image,
pd.price/mp.unit AS priceW
FROM main_part mp INNER JOIN
part_details pd
ON mp.part_id = pd.id
WHERE pd.approved = 'Yes' AND
pd.display = 'On' AND
pd.country = 'US' AND
mp.unit >= 300
ORDER BY pd.price ASC
LIMIT 50;
For this query, I would start with indexes on part_details(country, display, approved, id, price) and main_part(part_id, unit).
The index on part_details can be used for filtering before the join. It is not easy to get rid of the sort for the order by.
I have a mysql table:
CREATE TABLE `templates_assignments` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`type` int(11) NOT NULL DEFAULT '2'
`assignment_id` int(11) NOT NULL DEFAULT '1',
`template_id` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `template_id` (`template_id`),
) ENGINE=InnoDB AUTO_INCREMENT=122 DEFAULT CHARSET=latin1;
/*Data for the table `templates_assignments` */
insert into `templates_assignments`
(`id`,`type`,`assignment_id`,`template_id`)
values
(15,1,1,1),
(16,1,1,2),
(19,1,1,6),
(54,2,30,6),
(55,1,5,11),
(56,1,5,15),
(57,1,5,22);
I want to select the template that qualifies for both conditions:
type=2 AND assignment_id=30
type=1 AND assignment_id=1
the only template_id that should come back is 6, but i keep getting all or none.
My query condition was something like:
WHERE
(
(templatesAssignments.type=2 AND templatesAssignments.assignment_id=30) AND (templatesAssignments.type=1 AND templatesAssignments.assignment_id=1)
)
But no luck...what am i missing?
SELECT ta1.template_id
FROM templatesAssignments ta1
INNER JOIN templatesAssignments ta2 ON ta1.template_id = ta2.template_id
WHERE (ta1.type=1 AND ta1.assignment_id=1)
AND (ta2.type=2 AND ta2.assignment_id=30)
select template_id
from templatesAssignments
group by template_id
having sum(type=2 AND assignment_id=30) > 0
and sum(type=1 AND assignment_id=1) > 0
table:
CREATE TABLE IF NOT EXISTS `l_not_200_page` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`server` tinyint(3) unsigned NOT NULL,
`domain` tinyint(3) unsigned NOT NULL,
`page` varchar(128) NOT NULL,
`query_string` varchar(384) NOT NULL,
`status` smallint(5) unsigned NOT NULL,
PRIMARY KEY (`id`),
KEY `idx_time_domain_status_page` (`time`,`domain`,`status`,`page`),
KEY `page` (`page`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;
explain:
EXPLAIN SELECT *
FROM `l_not_200_page`
WHERE TIME
BETWEEN TIMESTAMP( '2014-03-25' )
AND TIMESTAMP( '2014-03-25 23:59:59' )
AND domain =1
AND STATUS = 404
GROUP BY PAGE
1
SIMPLE
l_not_200_page
range
idx_time_domain_status_page
idx_time_domain_status_page
7
NULL
1
Using where; Using temporary; Using filesort
it's very slow, how to optimize ?
sql:
SELECT PAGE , COUNT( * ) AS cnt
FROM l_not_200_page
WHERE TIME
BETWEEN TIMESTAMP( '2014-03-26 12:00:00' )
AND TIMESTAMP( '2014-03-26 12:30:00' )
AND domain =1
AND STATUS = 499
GROUP BY PAGE ORDER BY cnt DESC
LIMIT 100
the daily amount of data about 900w
Change the index to:
create index `idx_domain_status_time_page` on l_not_200_page(`domain`, `status`, `time`, `page`)
When MySQL uses an index for a where clause, the best index has all the fields in equality comparisons followed by one with an inequality, such as between. With time as the first element, it doesn't use the index for domain and status (well, it uses an index scan instead of a direct lookup).
For further optimization, you can get rid of the group by by choosing one row per page:
SELECT lp.* FROM l_not_200_page lp WHERE TIME BETWEEN TIMESTAMP( '2014-03-25' ) AND TIMESTAMP( '2014-03-25 23:59:59' ) AND
domain = 1 AND STATUS = 404 AND
NOT EXISTS (select 1
from l_not_200_page lp2
where lp2.page = lp.page and
lp2.domain = 1 and lp2.status = 404 and
lp2.TIME BETWEEN TIMESTAMP('2014-03-25) AND TIMESTAMP('2014-03-25 23:59:59') AND
lp2.id > lp.id
)
For this, an additional index on (page, domain, status, time) would help.
I am developing a website like any other social networking site using mysql.
I wish it give people suggestion to my users, and I have implemented this functionality in my application, but It is working very slow. this process take 2-3 seconds to fetch result from server. It has all the necessary indexes, relations on table. I have used EXPLAIN command to understand it, but I got nothing problematic in it.
I can't understand what is the basic problem in it. Please help me.
here is my table structure :
Table : UserMaster
~~~~~~~~~~~~~~~~~~
CREATE TABLE `UserMaster` (
`UserID` mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
`UserName` varchar(20) DEFAULT NULL,
`EMailID` varchar(50) DEFAULT NULL,
`FirstName` varchar(20) NOT NULL,
`LastName` varchar(20) NOT NULL,
`CityID` mediumint(8) unsigned DEFAULT NULL,
PRIMARY KEY (`UserID`),
UNIQUE KEY `UX_UserMaster_UserName` (`UserName`),
UNIQUE KEY `UX_UserMaster_EMailID` (`EMailID`),
KEY `FK_UserMaster_CityMst_CityID_idx` (`CityID`),
KEY `FK_UserMaster_CountryMst_CountryID_idx` (`CountryID`),
CONSTRAINT `FK_UserMaster_CityMst_CityID`
FOREIGN KEY (`CityID`) REFERENCES `CityMst` (`CityID`) ON DELETE NO ACTION,
CONSTRAINT `FK_UserMaster_CountryMst_CountryID` FOREIGN KEY CountryID REFERENCES CountryMst (CountryID) ON DELETE NO ACTION ON UPDATE CASCADE
)
ENGINE=InnoDB AUTO_INCREMENT=19722 DEFAULT CHARSET=utf8$$
Table : UserFriends
~~~~~~~~~~~~~~~~~~~
CREATE TABLE `UserFriends` (
`FriendID` mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
`UserID` mediumint(8) unsigned NOT NULL,
`UserID2` mediumint(8) unsigned NOT NULL,
`RequestDate` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
`Status` tinyint(3) unsigned NOT NULL DEFAULT '2',
`ApprovalDate` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`FriendID`),
UNIQUE KEY `UX_UserFriends_UserID_UserID2` (`UserID`,`UserID2`),
KEY `IX_UserFriens_UserID_ApprovalStatus` (`UserID`,`ApprovalStatus`,`UserID2`,`FriendID`,`RequestDate`,`ApprovalDate`),
KEY `FK_UserFriends_UserMaster_UserID_idx` (`UserID`),
KEY `FK_UserFriends_UserMaster_UserID2_idx` (`UserID2`),
CONSTRAINT `FK_UserFriends_UserMaster_UserID` FOREIGN KEY (`UserID`) REFERENCES `UserMaster` (`UserID`) ON DELETE NO ACTION ON UPDATE CASCADE,
CONSTRAINT `FK_UserFriends_UserMaster_UserID2` FOREIGN KEY (`UserID2`) REFERENCES `UserMaster` (`UserID`) ON DELETE NO ACTION ON UPDATE CASCADE
)
ENGINE=InnoDB AUTO_INCREMENT=50825 DEFAULT CHARSET=utf8$$
UserID & UserID2 both fields are linked with UserMaster.UserID
Here is my select query :
~~~~~~~~~~~~~~~~~~~~~~~~~
SELECT
upm.UserID,
upm.UserName,
upm.FirstName,
COALESCE(mf.TotMutualFriends,0) TotMutualFriends
FROM UserMaster upm
LEFT JOIN CityMst ct on ct.CityID = upm.CityID
LEFT JOIN StateMst st on st.StateID = ct.StateID
LEFT JOIN (
SELECT uf.UserID, COUNT(1) TotMutualFriends
FROM (
SELECT uf.UserID, uf.UserID2, uf.ApprovalStatus
FROM UserFriends uf
UNION ALL
SELECT uf.UserID2 UserID, uf.UserID UserID2, uf.ApprovalStatus
FROM UserFriends uf
) uf
INNER JOIN (
SELECT IF(uf.UserID = 1, uf.UserID2, uf.UserID) UserID2
FROM UserFriends uf
WHERE (uf.UserID = 1 OR uf.UserID2 = 1)
AND uf.ApprovalStatus = 1
) uf1 on uf1.UserID2 = uf.UserID2
WHERE uf.ApprovalStatus = 1
GROUP BY uf.UserID
) mf on mf.UserID = upm.UserID
LEFT JOIN (
SELECT DISTINCT usar.UserID2
FROM UserSuggAutoRejct usar
WHERE usar.UserID = 1
UNION
SELECT IF(uf.UserID = 1, uf.UserID2, uf.UserID) UserID2
FROM UserFriends uf
WHERE (uf.UserID = 1 OR uf.UserID2 = 1)
) usar ON usar.UserID2 = upm.UserID
WHERE upm.UserStatus IN(10,11)
AND upm.UserID <> 1
AND upm.UserID NOT IN (1221,2191)
AND usar.UserID2 IS NULL
ORDER BY
(CASE WHEN COALESCE(mf.TotMutualFriends,0) > 0 THEN 0 ELSE 1 END),
(CASE WHEN COALESCE(mf.TotMutualFriends,0) > 0 THEN RAND() ELSE NULL END),
(CASE upm.CityID WHEN 1 THEN 0 ELSE 1 END),
(CASE upm.CityID WHEN 1 THEN RAND() ELSE NULL END),
(CASE ct.StateID WHEN 1 THEN 0 ELSE 1 END),
(CASE ct.StateID WHEN 1 THEN RAND() ELSE NULL END),
(CASE st.CountryID WHEN 91 THEN 0 ELSE 1 END),
(CASE st.CountryID WHEN 91 THEN RAND() ELSE NULL END)
LIMIT 10
This is performing very slow. It takes 2-3 seconds to evolute.
relational database might not be relevant for social networking sites.
because joining table is very slow, try using other no-sql databases(NoSql type database).
if u still insist using mysql, then try not to much join in your query.
sorry for my bad english, if any.