multiple indexing with slow query - mysql

In following query I indexed every field like:
menu.id
pricelist.menu_id
vendors.id
pricelist.vendor
orders.pricelist_id
pricelist.id
users.id
orders.user_id
orders.free
pricelist.menu_id.
When I run the below query it takes much time, we have 13 million records in orders table and other table has few thousands.
SELECT
`orders`.`itusername`,
`orders`.`iturl`,
`orders`.`error_message`,
`orders`.`return` AS return1,
`orders`.`coupon`,
DATEDIFF( users.reseller_expiry, now( ) ) AS edays,
`users`.`email`,
`menu`.`menuname`,
`orders`.`error_status`,
`orders`.`auto_status`,
`vendors`.`name`,
`vendors`.`id` AS venderid,
`pricelist`.`servicename`,
`orders`.`email_order` AS paypal_order_email,
`orders`.`user_id`,
`orders`.`services_order`,
`orders`.`created_dt`,
`orders`.`id`,
`orders`.`transaction_comment`,
`orders`.`url`,
`orders`.`requireviews`,
`orders`.`youtubeviews`,
`orders`.`total_views_completed`,
`orders`.`aff`,
`orders`.`is_package`,
`orders`.`price`,
`orders`.`order_from_site`,
`orders`.`cost_per_unit_order`,
`orders`.`service_name_order`,
`orders`.`status`,
`orders`.`start_api_date`,
`orders`.`end_api_date`,
`orders`.`allow_setting`,
`orders`.`return2`,
`users`.`balance` AS user_balance
FROM
( `pricelist` )
JOIN `menu` ON `menu`.`id` = `pricelist`.`menu_id`
JOIN `vendors` ON `vendors`.`id` = `pricelist`.`vendor`
JOIN `orders` ON `orders`.`pricelist_id` = `pricelist`.`id`
JOIN `users` ON `users`.`id` = `orders`.`user_id`
WHERE
`orders`.`free` != 1
AND pricelist.menu_id = 3
ORDER BY
`orders`.`id` DESC
LIMIT 10.
.................................

The trick is to do the LIMIT before running around to 5 tables.
SELECT ...
FROM
( SELECT orders.pricelist_id AS id
FROM pricelist
JOIN `orders` ON `orders`.`pricelist_id` = `pricelist`.`id`
WHERE `orders`.`free` != 1
AND pricelist.menu_id = 3
ORDER BY `orders`.`id` DESC
LIMIT 10
) AS x
JOIN `pricelist` ON pricelist.id = x.id
JOIN `menu` ON `menu`.`id` = `pricelist`.`menu_id`
JOIN `vendors` ON `vendors`.`id` = `pricelist`.`vendor`
JOIN `orders` ON `orders`.`pricelist_id` = x.`id`
JOIN `users` ON `users`.`id` = `orders`.`user_id`
ORDER BY `orders`.`id` DESC
There is a unknown -- are the tables 1:1 or 1:many or many:1 or many:many? With the 'wrong' answers, you will get more than 10 rows, and the design is flawed.

Related

Sequelize: Force left join on nested subquery which is produced by limit

Hello I have this query generated by sequelize
`Orders` .*,
`OrderPriority`.`id` as `OrderPriority.ID`,
`OrderPriority`.`priority` as `OrderPriority.priority`,
`Employees`.`employee_alias` as `Employees.alias`,
`Employees`.`employee_id` as `Employees.ID` from
(
select
`Orders`.`id` as `ID`,
`Orders`.`due_date` as `dueDate`,
`Orders`.`creation_date` as `creationDate`,
`Orders`.`priority_id` as `priorityID`,
`OrderStatus`.`id` as `OrderStatus.ID`,
`OrderStatus`.`status` as `OrderStatus.status`
from
`orders` as `Orders`
inner join `statuses` as `OrderStatus` on
`Orders`.`status_id` = `OrderStatus`.`id`
where
(`Orders`.`creation_date` >= '2022-07-06 00:00:00'
and `Orders`.`creation_date` < '2022-07-14 00:00:00')
order by
`Orders`.`creation_date` desc
limit 0,
50) as `Orders`
left outer join `order_priorities` as `OrderPriority` on
`Orders`.`priorityID` = `OrderPriority`.`id`
left outer join ( `order_employees` as `Employees->OrderEmployees`
inner join `employes` as `Employees` on
`Employees`.`employee_id` = `Employees->OrderEmployees`.`employee_id`
and (`Employees->OrderEmployees`.`state` = 'Pending'
or `Employees->OrderEmployees`.`state` = 'Accepted')) on
`Orders`.`ID` = `Employees->OrderEmployees`.`order_id`
order by
`creationDate` desc;
I need to filter by null priority but the generated query has priority on the outer query if required:false which is logically wrong and the only way that I found to put the priority inside the subquery is to use required:true but this results in an inner join and I want left join in order to take the null values. Is there a way to force the include as left join inside the subquery?The nested subquery is generated because it exists a many to many relationship between Orders and Employees and limit must be applied to orders. I want to generate the following Query:
`Orders` .*,
`OrderPriority`.`id` as `OrderPriority.ID`,
`OrderPriority`.`priority` as `OrderPriority.priority`,
`Employees`.`employee_alias` as `Employees.alias`,
`Employees`.`employee_id` as `Employees.ID` from
(
select
`Orders`.`id` as `ID`,
`Orders`.`due_date` as `dueDate`,
`Orders`.`creation_date` as `creationDate`,
`Orders`.`priority_id` as `priorityID`,
`OrderStatus`.`id` as `OrderStatus.ID`,
`OrderStatus`.`status` as `OrderStatus.status`
from
`orders` as `Orders`
inner join `statuses` as `OrderStatus` on
`Orders`.`status_id` = `OrderStatus`.`id`
left outer join `order_priorities` as `OrderPriority` on
`Orders`.`priorityID` = `OrderPriority`.`id`
where
(`Orders`.`creation_date` >= '2022-07-06 00:00:00'
and `Orders`.`creation_date` < '2022-07-14 00:00:00' and `Orders`.`priorityID` is null)
order by
`Orders`.`creation_date` desc
limit 0,
50) as `Orders`
left outer join ( `order_employees` as `Employees->OrderEmployees`
inner join `employes` as `Employees` on
`Employees`.`employee_id` = `Employees->OrderEmployees`.`employee_id`
and (`Employees->OrderEmployees`.`state` = 'Pending'
or `Employees->OrderEmployees`.`state` = 'Accepted')) on
`Orders`.`ID` = `Employees->OrderEmployees`.`order_id`
order by
`creationDate` desc;

how optimize prestashop category get product for random

this is prestashop 1.7 version category get product query. if use random, it is very slow, how optimize it?
SELECT
cp.id_category,
p.*,
product_shop.*,
stock.out_of_stock,
IFNULL( stock.quantity, 0 ) AS quantity,
IFNULL( product_attribute_shop.id_product_attribute, 0 ) AS id_product_attribute,
product_attribute_shop.minimal_quantity AS product_attribute_minimal_quantity,
pl.`description`,
pl.`description_short`,
pl.`available_now`,
pl.`available_later`,
pl.`link_rewrite`,
pl.`meta_description`,
pl.`meta_keywords`,
pl.`meta_title`,
pl.`name`,
image_shop.`id_image` id_image,
il.`legend` AS legend,
m.`name` AS manufacturer_name,
cl.`name` AS category_default,
DATEDIFF(
product_shop.`date_add`,
DATE_SUB( "2019-11-30 00:00:00", INTERVAL 7 DAY )) > 0 AS new,
product_shop.price AS orderprice
FROM
`ps_category_product` cp
LEFT JOIN `ps_product` p ON p.`id_product` = cp.`id_product`
INNER JOIN ps_product_shop product_shop ON ( product_shop.id_product = p.id_product AND product_shop.id_shop = 1 )
LEFT JOIN `ps_product_attribute_shop` product_attribute_shop ON ( p.`id_product` = product_attribute_shop.`id_product` AND product_attribute_shop.`default_on` = 1 AND product_attribute_shop.id_shop = 1 )
LEFT JOIN ps_stock_available stock ON ( stock.id_product = `p`.id_product AND stock.id_product_attribute = 0 AND stock.id_shop = 1 AND stock.id_shop_group = 0 )
LEFT JOIN `ps_category_lang` cl ON ( product_shop.`id_category_default` = cl.`id_category` AND cl.`id_lang` = 11 AND cl.id_shop = 1 )
LEFT JOIN `ps_product_lang` pl ON ( p.`id_product` = pl.`id_product` AND pl.`id_lang` = 11 AND pl.id_shop = 1 )
LEFT JOIN `ps_image_shop` image_shop ON ( image_shop.`id_product` = p.`id_product` AND image_shop.cover = 1 AND image_shop.id_shop = 1 )
LEFT JOIN `ps_image_lang` il ON ( image_shop.`id_image` = il.`id_image` AND il.`id_lang` = 11 )
LEFT JOIN `ps_manufacturer` m ON m.`id_manufacturer` = p.`id_manufacturer`
WHERE
product_shop.`id_shop` = 1
AND cp.`id_category` = 12
AND product_shop.`active` = 1
AND product_shop.`visibility` IN ( "both", "catalog" )
ORDER BY
RAND()
LIMIT 50
Please provide SHOW CREATE TABLE for each table. Meanwhile, ...
Let's start by optimizing the joins.
LEFT JOIN `ps_product_lang` pl ON ( p.`id_product` = pl.`id_product`
AND pl.`id_lang` = 11
AND pl.id_shop = 1 )
That needs INDEX(id_product, id_lang, id_shop) (The columns may be in any order.)
Don't use LEFT unless you really need to fetch a row from the righthand table as NULLs when it does not exist. In particular,
LEFT JOIN `ps_product` p
is probably getting in the way of optimization.
WHERE product_shop.`id_shop` = 1
AND product_shop.`active` = 1
AND product_shop.`visibility` IN ( "both", "catalog" )
would probably benefit from these indexes
INDEX(id_shop, active, visibility, id_product)
INDEX(id_product, id_shop, active, visibility)
product_category needs
INDEX(id_category, id_product) -- in this order.
In general many-to-many mapping tables need to follow the tips here: http://mysql.rjweb.org/doc.php/index_cookbook_mysql#many_to_many_mapping_table
The query has the "explode-implode" syndrome. This is where it first does a JOINs, collecting a lot of data, then throws away much of it due, in your case, to the LIMIT 10. It can probably be cured by turning the query inside-out. The general ID is to start with a derived table that gets the 10 rows desired, then reaches into the other table for the rest of the desired columns. This "reaching" need happen only 10 times, not however many the JOINs currently require.
SELECT ...
FROM ( SELECT <<primary key columns from cp, p, and product_shop>>
FROM cp
JOIN p ON ...
JOIN product_shop ON ...
ORDER BY RAND()
LIMIT 10 ) AS x
JOIN <<p, product_shop ON their PKs>> -- to get p.*, product_shop.*>>
[LEFT] JOIN << each of the other tables>> -- to get the other tables
You should start by testing the subquery (a "derived" table) to verify that it is noticeably faster than the original query.

Optimize SQL query with 2 selects

I am trying to update a single campaign.id with minimum used_time (datetime) based on user.id but the following code need about 5 seconds to execute. Backlinks table contains 1 million rows.
UPDATE `backlinks`
SET
`backlinks`.`crawler_id` = 'test',
`backlinks`.`used_time`=NOW()
WHERE
`backlinks`.`campaign_id`=(
SELECT `id` FROM `campaigns`
WHERE `campaigns`.`completed`=false
AND `campaigns`.`status`=true
GROUP BY `campaigns`.`user_id`
ORDER BY `campaigns`.`used_time` ASC
limit 1
)
AND `backlinks`.`googlebot_id` IS NULL
AND `backlinks`.`used_time` IS NULL
LIMIT 1;
You can try to UPDATE with JOIN by a subquery.
UPDATE `backlinks` b
JOIN (
SELECT c.id
FROM campaigns c
WHERE exists (
SELECT 1
FROM campaigns cc
WHERE c.user_id = cc.user_id
GROUP BY cc.user_id
HAVING min(cc.used_time) = c.used_time
)
) t1 on b.`campaign_id` = t1.id
SET
b.`crawler_id` = 'test',
b.`used_time`=NOW()
WHERE
b.`googlebot_id` IS NULL
AND
b.`used_time` IS NULL

Optimizing django has_perm database query

Model method has perm looks for all the permission from Group and Permission. which is a heavy query.How can i optimized query time or any other solution to check user permission.
Example:
SELECT "auth_permission"."id", "auth_permission"."name", "auth_permission"."content_type_id", "auth_permission"."codename" FROM "auth_permission" INNER JOIN "auth_group_permissions" ON ( "auth_permission"."id" = "auth_group_permissions"."permission_id" ) INNER JOIN "auth_group" ON ( "auth_group_permissions"."group_id" = "auth_group"."id" ) INNER JOIN "auth_user_groups" ON ( "auth_group"."id" = "auth_user_groups"."group_id" ) INNER JOIN "django_content_type" ON ( "auth_permission"."content_type_id" = "django_content_type"."id" ) WHERE "auth_user_groups"."user_id" = 1235 ORDER BY "django_content_type"."app_label" ASC, "django_content_type"."model" ASC, "auth_permission"."codename" ASC
SELECT "auth_permission"."id", "auth_permission"."name", "auth_permission"."content_type_id", "auth_permission"."codename", "django_content_type"."id", "django_content_type"."name", "django_content_type"."app_label", "django_content_type"."model" FROM "auth_permission" INNER JOIN "auth_user_user_permissions" ON ( "auth_permission"."id" = "auth_user_user_permissions"."permission_id" ) INNER JOIN "django_content_type" ON ( "auth_permission"."content_type_id" = "django_content_type"."id" ) WHERE "auth_user_user_permissions"."user_id" = 1235 ORDER BY "django_content_type"."app_label" ASC, "django_content_type"."model" ASC, "auth_permission"."codename" ASC

Mysql query optimization

HI im running socialengine3 and need optimization on the a custom Mutual friends query.
Its currently taking 15 seconds to execute
Friends table
friend_id
friend_user_id1
friend_user_id2
friend_status
friend_type
users
user_id
Edited
I have converted in into exists and still its now executing in 20 seconds.
below is the updated query.
SELECT friendlist.friend_user_id2, se_users.username, se_users.id, se_users.image, se_users.name, se_users.surname, count( * ) AS mutral_friends
FROM `se_friends` friendlist
INNER JOIN `users` se_users ON friendlist.friend_user_id2 = `se_users`.id
WHERE EXISTS (
SELECT se.friend_user_id2
FROM se_friends se
WHERE se.friend_user_id1 = '105012'
AND se.friend_status = '1'
AND se.friend_user_id2 = friendlist.friend_user_id1
) AND NOT EXISTS (
SELECT se1.friend_user_id2
FROM `se_friends` se1
WHERE se1.friend_user_id1 = '105012'
AND friendlist.friend_user_id2 = se1.friend_user_id2
)
AND NOT (
friendlist.friend_user_id2 = '105012'
)
AND friendlist.friend_status = '1'
GROUP BY friendlist.friend_user_id2, se_users.username, se_users.id, se_users.image, se_users.name, se_users.surname
ORDER BY mutral_friends DESC
LIMIT 0 , 20
Orignal query
SELECT DISTINCT `se_friends`.friend_user_id2, se_users.username, se_users.id, se_users.image, se_users.name, se_users.surname, count(*) as mutral_friends
FROM `se_friends`
INNER JOIN `users` se_users` ON `se_friends`.friend_user_id2=`se_users`.id
WHERE
(se_friends.friend_user_id1 <> '30355' or se_friends.friend_user_id2 <> '30355') AND
se_friends.friend_user_id1 IN
(SELECT se_friends.friend_user_id2
FROM `se_friends`
WHERE se_friends.friend_user_id1='".$user_id."' AND se_friends.friend_status='1')
AND `se_friends`.friend_user_id2 NOT IN
(SELECT se_friends.friend_user_id2
FROM `se_friends`
WHERE se_friends.friend_user_id1='".$user_id."'
)
AND NOT(se_friends.friend_user_id2='".$user_id."') AND se_friends.friend_status='1'
GROUP BY `se_friends`.friend_user_id2, se_users.username, se_users.id, se_users.image, se_users.name, se_users.surname
ORDER BY mutral_friends DESC
LIMIT 0, 20
IN is very expensive operation.
try to replace it with EXISTS. eg
select * from table where user_id in (select user_id from users where active='A')
and
select * from table t where exists (select user_id from users u where t.user_id = u.user_id and u.active='A')
if it won't be helpful, it's better to look at execution plan