Counting quantities using join and nested queries - mysql

I'm attempting to count stock quantities from active sales orders, which are stored in xcart_order_details. I also want to only count stock from orders placed within x days which have a status of processed 'P' or queued 'Q', and also if the stock type matches certain locations; C1 stock, C2 stock, and so on.
This used to be relatively simple in our old xcart database;
SELECT COUNT(`amount`)
FROM `xcart_order_details`
WHERE `productid` IN (
SELECT `productid`
FROM `xcart_products` WHERE `orderid` IN
(SELECT `orderid`
FROM `xcart_orders`
WHERE `date` > ".$date_range."
AND (`status` = 'P'
OR `status` = 'Q'))
AND (LOWER(param01) = 'c1 stock'
OR LOWER(param01) = 'c2 stock'
OR LOWER(param01) = 'g stock'
OR LOWER(param01) = 'stock')
AND `productid` = ".$safe_prodid.")
This query worked. But now our stock locations are stored in another table called xcart_extra_field_values, and must be retrieved where fieldid = 5;
I've tried using a join to get the value field from xcart_extra_field_values where the fieldid = 5, and attempt to do what I'm doing with the query above, but it doesn't work.
SELECT COUNT(`a.amount`)
FROM xcart_order_details a,
xcart_extra_field_values b
WHERE a.productid IN (
SELECT productid
FROM xcart_products WHERE orderid IN
(SELECT orderid
FROM xcart_orders
WHERE date > 1409529600
AND (status = 'P'
OR status = 'Q'))
AND (LOWER(b.value) = 'c1 stock'
OR LOWER(b.value) = 'c2 stock'
OR LOWER(b.value) = 'g stock'
OR LOWER(b.value) = 'stock')
AND (a.productid = b.productid)
AND (a.productid = 4169)
AND (b.fieldid = 5)
Unfortunately this query does not work, and I know it's a syntax issue, but the error I'm getting is no help whatsoever.
Can anyone shed any light on what I'm doing wrong?

You are missing the criteria how to join the two tables. Let's assume there is an xcart_order_details_id in table xcart_extra_field_values, then use that.
After all, you don't even have to join. You want results from xcart_order_details for which certain detail value records EXIST:
SELECT COUNT(`amount`)
FROM `xcart_order_details`
WHERE `productid` IN
(
SELECT `productid`
FROM `xcart_products`
WHERE `orderid` IN
(
SELECT `orderid`
FROM `xcart_orders`
WHERE `date` > ".$date_range."
AND `status` IN ('P','Q')
)
AND `productid` = ".$safe_prodid."
AND EXISTS
(
SELECT *
FROM xcart_extra_field_values v
WHERE v.xcart_order_details_id = xcart_order_details.id
AND fieldid = 5
AND LOWER(param01) IN ('c1 stock','c2 stock','g stock','stock')
);
BTW: You count how many order detail records have an amount?

Resolved.
SELECT COUNT(a.amount)
FROM xcart_order_details a,
xcart_extra_field_values b
WHERE a.productid IN
(SELECT productid
FROM xcart_products
WHERE orderid IN
(SELECT orderid
FROM xcart_orders
WHERE date > 1409529600
AND (status = 'P'
OR status = 'Q'))
AND (LOWER(b.value) = 'c1 stock'
OR LOWER(b.value) = 'c2 stock'
OR LOWER(b.value) = 'g stock'
OR LOWER(b.value) = 'stock'))
AND (a.productid = b.productid)
AND (a.productid = 4169)
AND (b.fieldid = 5)
Firstly, I was missing a bracket at the end of my query as I was nesting two IN queries, and I needed to remove the Grave accents around a.amount within COUNT, which it didn't like. I only noticed the missing bracket when I looked at my queries side by side.
Edit: Moved my join outside of my nested IN clause, as it made no sense to have it nested.

Related

MySQL query taking too much time

query taking 1 minute to fetch results
SELECT
`jp`.`id`,
`jp`.`title` AS game_title,
`jp`.`game_type`,
`jp`.`state_abb` AS game_state,
`jp`.`location` AS game_city,
`jp`.`zipcode` AS game_zipcode,
`jp`.`modified_on`,
`jp`.`posted_on`,
`jp`.`game_referal_amount`,
`jp`.`games_referal_amount_type`,
`jp`.`status`,
`jp`.`is_flaged`,
`u`.`id` AS employer_id,
`u`.`email` AS employer_email,
`u`.`name` AS employer_name,
`jf`.`name` AS game_function,
`jp`.`game_freeze_status`,
`jp`.`game_statistics`,
`jp`.`ats_value`,
`jp`.`integration_id`,
`u`.`account_manager_id`,
`jp`.`model_game`,
`jp`.`group_id`,
(CASE
WHEN jp.group_id != '0' THEN gm.group_name
ELSE 'NA'
END) AS group_name,
`jp`.`priority_game`,
(CASE
WHEN jp.country != 'US' THEN jp.country_name
ELSE ''
END) AS game_country,
IFNULL((CASE
WHEN
`jp`.`account_manager_id` IS NULL
OR `jp`.`account_manager_id` = 0
THEN
(SELECT
(CASE
WHEN
account_manager_id IS NULL
OR account_manager_id = 0
THEN
`u`.`account_manager_id`
ELSE account_manager_id
END) AS account_manager_id
FROM
user_user
WHERE
id = (SELECT
user_id
FROM
game_user_assigned
WHERE
game_id = `jp`.`id`
LIMIT 1))
ELSE `jp`.`account_manager_id`
END),
`u`.`account_manager_id`) AS acc,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '1'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS somewhatgame,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '2'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS verygamecommitted,
(SELECT
COUNT(recach_limit_id)
FROM
recach_limit
WHERE
recach_limit = '3'
AND recach_limit_game_id = rpr.recach_limit_game_id) AS notgame,
(SELECT
COUNT(joa.id) AS applicationcount
FROM
game_refer_to_member jrmm
INNER JOIN
game_refer jrr ON jrr.id = jrmm.rid
INNER JOIN
game_applied joa ON jrmm.id = joa.referred_by
WHERE
jrmm.STATUS = '1'
AND jrr.referby_user_id IN (SELECT
ab_testing_user_id
FROM
ab_testing)
AND joa.game_post_id = rpr.recach_limit_game_id
AND (rpr.recach_limit = 1
OR rpr.recach_limit = 2)) AS gamecount
FROM
(`game_post` AS jp)
JOIN
`user_info` AS u ON `jp`.`user_user_id` = `u`.`id`
JOIN
`game_functional` jf ON `jp`.`game_functional_id` = `jf`.`id`
LEFT JOIN
`group_musesm` gm ON `gm`.`group_id` = `jp`.`group_id`
LEFT JOIN
`recach_limit` rpr ON `jp`.`id` = `rpr`.`recach_limit_game_id`
WHERE
`jp`.`status` != '3'
GROUP BY `jp`.`id`
ORDER BY `posted_on` DESC
LIMIT 10
I would first suggest not nesting select statements because this will cause an n^x performance hit on every xth level and I see at least 3 levels of selects inside this query.
Add index
INDEX(status, posted_on)
Move LIMIT inside
Then, instead of saying
FROM (`game_post` AS jp)
say
FROM ( SELECT id FROM game_post
WHERE status != 3
ORDER BY posted_on DESC
LIMIT 10 ) AS ids
JOIN game_post AS jp USING(id)
(I am assuming that the PK of jp is (id)?)
That should efficiently use the new index to get the 10 ids needed. Then it will reach back into game_post to get the other columns.
LEFT
Also, don't say LEFT unless you need it. It costs something to generate NULLs that you may not be needing.
Is GROUP BY necessary?
If you remove the GROUP BY, does it show dup ids? The above changes may have eliminated the need.
IN(SELECT) may optimize poorly
Change
AND jrr.referby_user_id IN ( SELECT ab_testing_user_id
FROM ab_testing )
to
AND EXISTS ( SELECT * FROM ab_testing
WHERE ab_testing_user_id = jrr.referby_user_id )
(This change may or may not help, depending on the version you are running.)
More
Please provide EXPLAIN SELECT if you need further assistance.

SUM with CASE Statement in mysql

I have following Mysql query
SELECT c.`id`
,c.`category_name`
,c.`category_type`
,c.bookmark_count
,f.category_id cat_id
,f.unfollow_at
,(
CASE WHEN c.id = f.follower_category_id
THEN (
SELECT count(`user_bookmarks`.`id`)
FROM `user_bookmarks`
WHERE (`user_bookmarks`.`category_id` = cat_id)
AND ((`f`.`unfollow_at` > `user_bookmarks`.`created_at`) || (`f`.`unfollow_at` = '0000-00-00 00:00:00'))
)
ELSE 0 END
) counter
,c.id
,f.follower_category_id follow_id
,c.user_id
FROM categories c
LEFT JOIN following_follower_categories f ON f.follower_category_id = c.id
WHERE c.user_id = 26
ORDER BY `category_name` ASC
and here is output what i am getting after execuation
now i just want to count . here i have field id having value 172 against it i have counter 30,3, 2 and Bookmark_count is 4( i need to include only once)
and i am accepting output for id 172 is 30+3+2+4(bookmark_count only once).
I am not sure how to do this.
Can anybody help me out
Thanks a lot
The following may be the most inefficient query for that purpose, but I added a cover to your query in order to hint at grouping the results.
(I removed the second c.id, and my example may have errors since I couldn't try it.)
SELECT `id`,
`category_name`,
`category_type`,
max(`bookmark_count`),
`cat_id`,
`unfollow_at`,
sum(`counter`)+max(`bookmark_count`) counter,
follow_id`, `user_id`
FROM
(SELECT c.`id`
,c.`category_name`
,c.`category_type`
,c.bookmark_count
,f.category_id cat_id
,f.unfollow_at
,(
CASE WHEN c.id = f.follower_category_id
THEN (
SELECT count(`user_bookmarks`.`id`)
FROM `user_bookmarks`
WHERE (`user_bookmarks`.`category_id` = cat_id)
AND ((`f`.`unfollow_at` > `user_bookmarks`.`created_at`) || (`f`.`unfollow_at` = '0000-00-00 00:00:00'))
)
ELSE 0 END
) counter
,f.follower_category_id follow_id
,c.user_id
FROM categories c
LEFT JOIN following_follower_categories f ON f.follower_category_id = c.id
WHERE c.user_id = 26)
GROUP BY `id`, `category_name`, `category_type`, `cat_id`, `unfollow_at`, `follow_id`, `user_id`
ORDER BY `category_name` ASC

Query and get three expected results

SELECT
(
`members`.`id`
SELECT COUNT(`members`.`id`) FROM `members` WHERE `gender` = 0 AS `Unknown`
SELECT COUNT(`members`.`id`) FROM `members` WHERE `gender` = 1 AS `Female`
SELECT COUNT(`members`.`id`) FROM `members` WHERE `gender` = 2 AS `Male`
) FROM `members` INNER JOIN `mapMember`
ON `mapMember`.`id` = `members`.`id`
WHERE `mapMember`.`mapper_id` = 3
My expected result:
Unknown Female Male
0 1 3
However I get SYNTAX error. Cant' figure out what's wrong.
I also tried:
SELECT COUNT(`members`.id) AS `members`, `gender`
FROM `members` INNER JOIN `mapMember`
ON `mapMember`.`id` = `members`.`id`
WHERE `mapMember`.`mapper_id` = 3 GROUP BY `gender` ORDER BY `gender` ASC
Which gives me almost the result I want to have, the only difference is If the there are no members with the given gender, there won't be a 0 result back. (no row that is) I always expect three rows back.
SELECT
sum(if (`gender` = 0, 1,0)) as `Unknown`,
sum(if (`gender` = 1, 1,0)) as `Female`,
sum(if (`gender` = 2, 1,0)) as `Male`
FROM `members` INNER JOIN `mapMember`
ON `mapMember`.`id` = `members`.`id`
WHERE `mapMember`.`mapper_id` = 3
SELECT * FROM
(
SELECT `members`.`id`,COUNT(`members`.`id`) AS `Unknown` FROM `members` WHERE `gender` = 0
UNION
SELECT `members`.`id`,COUNT(`members`.`id`) AS `Female` FROM `members` WHERE `gender` = 1
UNION
SELECT `members`.`id`,COUNT(`members`.`id`) AS `Male` FROM `members` WHERE `gender` = 2
) Z INNER JOIN `mapMember`
ON `mapMember`.`id` = `Z`.`id`
WHERE `mapMember`.`mapper_id` = 3
Others have given you solutions, so I mainly tell you where you went wrong with your own statement.
COUNT(column_name) simply counts records where column_name is not null. members.id is not null, so you simply count all records from members. You need a where clause instead limiting the counted records to the member id in question.
Sub queries must be in parentheses.
Here is your statement re-written. It is not good though, because you query the same table again and again. I just wanted to use your statement and only correct errors:
SELECT
`members`.`id`,
(SELECT COUNT(*) FROM `members` u WHERE `gender` = 0 AND u.id = members.id) AS `Unknown`
(SELECT COUNT(*) FROM `members` f WHERE `gender` = 1 AND f.id = members.id) AS `Female`
(SELECT COUNT(*) FROM `members` m WHERE `gender` = 2 AND m.id = members.id) AS `Male`
FROM `members` INNER JOIN `mapMember`
ON `mapMember`.`id` = `members`.`id`
WHERE `mapMember`.`mapper_id` = 3;
Now it's syntactically correct. However, as a member record has exactly one gender, you will always get records with 0-0-1 or 0-1-0 or 1-0-0. So you don't really want to select members and have the counts per member.
Here is a better statement querying the tables just once, counting over all records rather than per member and providing better readabilty by using an IN clause for mapmember. (You can as well replace the IN clause with an EXISTS clause, which is sometimes faster.)
select
sum( case when gender = 0 then 1 else 0 end ) as unknown,
sum( case when gender = 1 then 1 else 0 end ) as female,
sum( case when gender = 2 then 1 else 0 end ) as male
from members
where id in (select id from mapmember where mapper_id = 3);
(BTW: Is the mapmember id really a members id? It looks strange to have a table with a column named id and this not being the id of the table itself but the id of another table actually.)
EDIT: I just notice you use MySQL. There you have a boolean data type you can use:
select
sum( gender = 0 ) as unknown, sum( gender = 1 ) as female, sum( gender = 2 ) as male
from members
where id in (select id from mapmember where mapper_id = 3);
This is no longer standard SQL, because it uses an enhancement from MySQL.

MySQL: How to optimize this query?

I almost spent a day to optimize this query:
SELECT
prod. *,
cat.slug category_slug,
sup.bname bname,
sup.slug bname_slug
FROM bb_admin.bb_directory_products AS prod
LEFT JOIN bb_admin.bb_categories_products AS cat
ON prod.primary_category_id = cat.category_id
LEFT JOIN bb_admin.bb_directory_suppliers AS sup
ON prod.supplier_id = sup.supplier_id
LEFT JOIN bb_admin.bb_directory_suppliers AS credit_sup
ON prod.credit_supplier_id = credit_sup.supplier_id
LEFT JOIN bb_admin.bb_directory_suppliers AS photo_sup
ON prod.photo_supplier_id = photo_sup.supplier_id
WHERE (
prod.status = '1'
OR prod.status = '3'
OR prod.status = '5'
)
AND (
sup.active_package_id != '1'
OR sup.active_package_id != '5'
OR sup.active_package_id != '6'
OR prod.supplier_id = '0'
)
AND (
sup.supplier_id = '1989'
OR credit_sup.supplier_id = '1989'
OR photo_sup.supplier_id = '1989'
)
GROUP BY prod.product_id
ORDER BY prod.priority_index ASC
Can you help me to optimized this query?
Update your column data types to be INT or one of its variants, since the ones you are checking against are all integer IDs (assumption).
Create indexes on following columns(if possible in all tables):
prod.status
supplier_id
active_package_id
Use IN clause instead of concatenating OR segments.
I'll be putting the updated WHERE clause here:
WHERE prod.status IN(1, 3, 5)
AND ( sup.active_package_id NOT IN(1, 5, 6)
OR prod.supplier_id = 0
)
AND 1989 IN (prod.supplier_id, prod.credit_supplier_id, prod.photo_supplier_id)

MySQL 500 million rows table in select query with join

I'm concerned about the performance of the query below once the tables are fully populated. So far it's under development and performs well with dummy data.
The table "adress_zoo" will contain about 500 million records once fully populated. "adress_zoo" table looks like this:
CREATE TABLE `adress_zoo`
( `adress_id` int(11) NOT NULL, `zoo_id` int(11) NOT NULL,
UNIQUE KEY `pk` (`adress_id`,`zoo_id`),
KEY `adress_id` (`adress_id`) )
ENGINE=InnoDB DEFAULT CHARSET=latin1;
The other tables will contain maximum 500 records each.
The full query looks like this:
SELECT a.* FROM jos_zoo_item AS a
JOIN jos_zoo_search_index AS zsi2 ON zsi2.item_id = a.id
WHERE a.id IN (
SELECT r.id FROM (
SELECT zi.id AS id, Max(zi.priority) as prio
FROM jos_zoo_item AS zi
JOIN jos_zoo_search_index AS zsi ON zsi.item_id = zi.id
LEFT JOIN jos_zoo_tag AS zt ON zt.item_id = zi.id
JOIN jos_zoo_category_item AS zci ON zci.item_id = zi.id
**JOIN adress_zoo AS az ON az.zoo_id = zi.id**
WHERE 1=1
AND ( (zci.category_id != 0 AND ( zt.name != 'prolong' OR zt.name is NULL))
OR (zci.category_id = 0 AND zt.name = 'prolong') )
AND zi.type = 'telefoni'
AND zsi.element_id = '44d3b1fd-40f6-4fd7-9444-7e11643e2cef'
AND zsi.value = 'Small'
AND zci.category_id > 15
**AND az.adress_id = 5**
GROUP BY zci.category_id ) AS r
)
AND a.application_id = 6
AND a.access IN (1,1)
AND a.state = 1
AND (a.publish_up = '0000-00-00 00:00:00' OR a.publish_up <= '2012-06-07 07:51:26')
AND (a.publish_down = '0000-00-00 00:00:00' OR a.publish_down >= '2012-06-07 07:51:26')
AND zsi2.element_id = '1c3cd26e-666d-4f8f-a465-b74fffb4cb14'
GROUP BY a.id
ORDER BY zsi2.value ASC
The query will usually return about 25 records.
Based on your experience, will this query perform acceptable (respond within say 3 seconds)?
What can I do to optimise this?
As adviced by #Jack I ran the query with EXPLAIN and got this:
This part is an important limiter:
az.adress_id = 5
MySQL will limit the table to only those records where adress_id matches before joining it with the rest of the statement, so it will depend on how big you think that result set might be.
Btw, you have a UNIQUE(adress_id, zoo_id) and a separate INDEX. Is there a particular reason? Because the first part of a spanning key can be used by MySQL to select with as well.
What's also important is to use EXPLAIN to understand how MySQL will "attack" your query and return the results. See also: http://dev.mysql.com/doc/refman/5.5/en/execution-plan-information.html
To avoid subquery you can try to rewrite your query as:
SELECT a.* FROM jos_zoo_item AS a
JOIN jos_zoo_search_index AS zsi2 ON zsi2.item_id = a.id
INNER JOIN
(
SELECT ** distinct ** r.id FROM (
SELECT zi.id AS id, Max(zi.priority) as prio
FROM jos_zoo_item AS zi
JOIN jos_zoo_search_index AS zsi ON zsi.item_id = zi.id
LEFT JOIN jos_zoo_tag AS zt ON zt.item_id = zi.id
JOIN jos_zoo_category_item AS zci ON zci.item_id = zi.id
**JOIN adress_zoo AS az ON az.zoo_id = zi.id**
WHERE 1=1
AND ( (zci.category_id != 0 AND ( zt.name != 'prolong' OR zt.name is NULL))
OR (zci.category_id = 0 AND zt.name = 'prolong') )
AND zi.type = 'telefoni'
AND zsi.element_id = '44d3b1fd-40f6-4fd7-9444-7e11643e2cef'
AND zsi.value = 'Small'
AND zci.category_id > 15
**AND az.adress_id = 5**
GROUP BY zci.category_id ) AS r
) T
on a.id = T.id
where
AND a.application_id = 6
AND a.access IN (1,1)
AND a.state = 1
AND (a.publish_up = '0000-00-00 00:00:00' OR a.publish_up <= '2012-06-07 07:51:26')
AND (a.publish_down = '0000-00-00 00:00:00' OR a.publish_down >= '2012-06-07 07:51:26')
AND zsi2.element_id = '1c3cd26e-666d-4f8f-a465-b74fffb4cb14'
GROUP BY a.id
ORDER BY zsi2.value ASC
This approach don't perform subquery for each candidate row. Performance may be increased only if T is calculated in few milliseconds.