sql selecting a lot of counts - mysql

Question

I have a problem driving me nuts for the last 2 days. I basically have 4 tables with inheritance in the following order:
users
|
categories blogs
| | |
---- pages visits
So a user has many blogs which has many pages and visits. Each page also belongs to a category.
All I want is to extract all users with the following counts associated:
total number of blogs each user has
total number of pages each user has
total number of categories each user has blogs in
total number of visits each user has
total number of visitors each user has (visits but we count by distinct ip_address)
My query is as follows:
SELECT
u.id
u.username,
COUNT(b.id) as blogs_count,
COUNT(p.id) as pages_count,
COUNT(v.id) as visits_count,
COUNT(distinct ip_address) as visitors_count
COUNT(c.id) as categories_count
FROM
users u
LEFT JOIN
blogs b ON(b.user_id=u.id)
LEFT JOIN
pages p ON(p.blog_id=b.id)
LEFT JOIN
visits v ON(v.blog_id=b.id)
LEFT JOIN
categories c ON(v.category_id=c.id)
GROUP BY u.id, blogs_count, pages_count, visits_count,
visitors_count, categories_count
I should get 24 users with their counts but, given the fact that I have almost 300,000 visits I get my SQL database hanging in forever probably trying to pull millions of rows.
I'm not a db guru and it's obvious. Can someone point me to the right direction somehow so I can make a good query able to perform well on even millions of records (with the right hardware of course)?

Answer 1

Try this:
SELECT u.id,
u.username,
COUNT(b.id) AS blogs_count,
COALESCE(MAX(p.pagecnt), 0) AS pages_count,
COALESCE(MAX(v.visitscnt), 0) AS visits_count,
COALESCE(MAX(v.visitorscnt), 0) AS visitors_count,
COALESCE(MAX(c.catcnt), 0) AS categories_count
FROM users u
LEFT JOIN blogs b ON u.id = b.user_id
LEFT JOIN (
SELECT blog_id,
COUNT(*) AS pagecnt
FROM pages
GROUP BY blog_id
) p ON b.id = p.blog_id
LEFT JOIN (
SELECT blog_id,
COUNT(*) AS visitscnt,
COUNT(DISTINCT ip_address) AS visitorscnt
FROM visits
GROUP BY blog_id
) v ON b.id = v.blog_id
LEFT JOIN (
SELECT aa.id,
COUNT(DISTINCT dd.id) AS catcnt
FROM users aa
JOIN blogs bb ON aa.id = bb.user_id
JOIN pages cc ON bb.id = cc.blog_id
JOIN categories dd ON cc.category_id = dd.id
GROUP BY aa.id
) c ON u.id = c.id
GROUP BY u.id,
u.username
Breakdown
This should also work across different DBMSs like PGSQL, SQL-Server, etc.
The challenge is that you have this sort of hierarchy of 1:M relationships in which joining them all together can easily throw off the different types of counts (as you want distinct counts in some places, but total counts in others).
What I've decided to do is first subselect the count of each page and visit / distinct visitors, grouping by the blog_id. This ensures that we get only one row per blog_id, even after joining the subselects on the blogs table.
For the category count, you want a count of distinct categories per user, but the challenge is that categories is linked deep within the relationship hierarchy (to the pages table), so you have to make a separate subselect that joins on the user_id instead of the blog_id.
Even with as many subselects as this query contains, it should still be quite fast as no two subselects are joining against each other. As long as there is an indexed table (subselects are actually unindexed temporary tables) on either side of the join, you should be fine.

Answer 2

SELECT
u.id
u.username,
COUNT(b.id) as blogs_count,
COUNT(p.id) as pages_count,
COUNT(v.id) as visits_count,
COUNT(distinct ip_address) as visitors_count
COUNT(c.id) as categories_count
FROM
users u
LEFT JOIN
blogs b ON(b.user_id=u.id)
LEFT JOIN
pages p ON(p.blog_id=b.id)
LEFT JOIN
visits v ON(v.blog_id=b.id)
LEFT JOIN
categories c ON(v.category_id=c.id)
GROUP BY u.id
Try with removing blogs_count, pages_count, visits_count, visitors_count, categories_count from your group by statment.