Slow database queries - mysql

Question

Since my website database has grown very large the performance for certain queries has become terrible. Some queries are taking over 30 seconds to perform. I'm wondering if someone can help me optimize my query or make a suggestion on how I can improve performance? I have set an index on all the foreign keys and ids.
SELECT p.*
, u.unique_id
, u.nick_name
, u.avatar_thumb
, t.desc as tag_desc
, pt.post_id as tag_post_id
from tt_post_tags pt
LEFT
JOIN tt_posts p
ON p.id = pt.post_id
RIGHT
JOIN tt_users u
ON p.user_id = u.user_id
LEFT
JOIN tt_tags t
ON t.name = "gameday"
WHERE pt.name = "gameday"
ORDER
BY create_date DESC
LIMIT 100
The above query takes 29 seconds to complete. If I remove the "create_date DESC" from the query it runs in .3 seconds. I've added an index to create_date but still, it takes 30 seconds for the query to run. The tt_posts table contains about 1.6 million records.
My database has the following tables: Posts, Users, Tags, and PostTags.
Posts table contains a foreign key for the users table.
Tags table contains a unique id and name for each tag
Post_tags table contains the foreign key from the Tags table aswell as a foreign key for the post that the tag is for.
I can include a diagram tomorrow if it's not easy to understand. Hopefully, someone can assist me. Thanks in advance.
CREATE TABLE `tt_posts` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`post_id` bigint(30) NOT NULL,
`user_id` bigint(30) NOT NULL,
`create_date` datetime NOT NULL,
`cover` varchar(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
`duration` int(10) DEFAULT NULL,
`desc` text CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci,
PRIMARY KEY (`id`),
UNIQUE KEY `post_id` (`post_id`),
KEY `user_id` (`user_id`),
KEY `create_date` (`create_date`)
) ENGINE=InnoDB AUTO_INCREMENT=4641550 DEFAULT CHARSET=utf8
CREATE TABLE `tt_tags` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`tt_tag_id` BIGINT(30) NULL DEFAULT NULL,
`name` VARCHAR(100) NOT NULL COLLATE 'utf8mb4_unicode_ci',
PRIMARY KEY (`id`),
UNIQUE INDEX `name` (`name`),
UNIQUE INDEX `tt_tag_id` (`tt_tag_id`),
INDEX `tt_tag_id_key` (`tt_tag_id`),
INDEX `name_key` (`name`)
)
COLLATE='utf8mb4_unicode_ci'
ENGINE=InnoDB
AND
CREATE TABLE `tt_post_tags` (
`post_id` INT(11) NOT NULL,
`name` VARCHAR(100) NOT NULL COLLATE 'utf8mb4_unicode_ci',
INDEX `post_id` (`post_id`),
INDEX `name` (`name`)
)
COLLATE='utf8_general_ci'
ENGINE=InnoDB;
AND
CREATE TABLE `tt_users` (
`id` BIGINT(20) NOT NULL AUTO_INCREMENT,
`user_id` BIGINT(30) NOT NULL,
`unique_id` VARCHAR(190) NOT NULL COLLATE 'utf8mb4_unicode_ci',
`nick_name` VARCHAR(100) NOT NULL COLLATE 'utf8mb4_unicode_ci',
`avatar` VARCHAR(190) NOT NULL COLLATE 'utf8mb4_unicode_ci',
`signature` TEXT NOT NULL COLLATE 'utf8mb4_unicode_ci',
PRIMARY KEY (`id`),
UNIQUE INDEX `user_id` (`user_id`),
UNIQUE INDEX `unique_id` (`unique_id`),
INDEX `unique_id_index` (`unique_id`),
INDEX `user_id_index` (`user_id`)
)
COLLATE='utf8mb4_unicode_ci'
ENGINE=InnoDB

Answer 1

In my opinion, the main issue with your query is the mix of left and right outer joins. Honestly, are you able to read this correctly?
The first join alone seems weird. You outer join a post to its post tags. But can a post tag without a post even exist? What would it refer to? (The other way round would make more sense: to also select posts that have no tags.) If I am not mistaken here, your join is rendered to a mere inner join. In your where clause you further limit this result to post tags named 'gameday'.
Then you right outer join users. We avoid right outer joins for being by far less readable than left outer joins, but well, you select all users, even those without 'gameday' post tags here.
Then you left outer join all 'gameday' tags. This looks completely unrelated to the other tables (i.e. you either find 'gameday' tags or not). But in your explanation you say "Post_tags table contains the foreign key from the Tags", so I surmise there is no tag_id in your post tags table, but the name is the tag ID really (and thus also the foreign key in your post tags table). This again leads to the question: Why would a post tag even exist, when it has no related tag? Probably this is not possible, and again all this is boiled down to a mere inner join. (I would recommend here to have a tag_id instead of the names in both tables, just for readability. The column name name kind of hides the foreign key relationship.)
In your query, you don't show any information of the post tags table, but I see you select pt.post_id as tag_post_id, which of course is just p.id as tag_post_id again. I suppose this is a typo and you want to show pt.id as tag_post_id instead?
I understand that you want to see all users, but are only interested in 'gameday' post tags. This makes writing the query a little complicated. I would probably just select users and outer join the complete post tag information.
Your create_date is not qualified with a table. I suppose it is a column in the posts table?
This is the query I am coming up with:
select
gdp.*,
u.unique_id,
u.nick_name,
u.avatar_thumb
from tt_users u
left join
(
select
p.*,
t.desc as tag_desc,
pt.id as tag_post_id
from tt_tags t
join tt_post_tags pt on pt.name = t.name
join tt_posts p on p.id = pt.post_id
where t.name = 'gameday'
) gdp on gdp.user_id = u.user_id
order by p.create_date desc;
There has been a lot of guessing on my side, so this query may still be a little different from what you need. I don't know.
Now let's look at which table columns are accessed, to provide good indexes for the query. Let's particularily look at the subquery where we collect all post tags:
We only want 'gameday' tags. As this seems to be the primary key for tt_tags, there should already be a unique index on tt_tags(name).
Being the foreign key, there should also be an index on tt_post_tags(name). This is good, but as we want to continue joining on the post_id, it would be beneficial to have this in the index, too: create unique index idx on tt_post_tags(name, post_id). However, as this is the table's natural key, this index should also already exist in order to ensure data integrity. If it doesn't exist yet, hurry up to provide it.
At last we join tt_posts on its primary key (i.e. there should be an index on tt_posts(id)). Once more: Nothing for us to do here.
You select all users and you select all 'gameday' tags. Then you must join all found tags to the users, which already is some work. You can imagine this as ordering all found tags by user_id first in order to join. Then you want to sort your result by post date. This means that the DBMS must again sort all result rows. Sorting takes time; that's just the way it is. How many rows does the result contain? If we are talking about millions of rows to sort, then this will probably remain slow. And if many post tags are 'gameday' tags, then even the indexes may not help much reading the tables and the DBMS may go for full sequential table reads instead. Make sure the statistics are up-to-date (https://dev.mysql.com/doc/refman/8.0/en/analyze-table.html).

Answer 2

(Iteration one of Answering the Question)
First, let's look at the query without users:
select p.id
from post_tags AS pt
join posts AS p ON p.id = pt.post_id
join tags AS t ON t.name = "gameday"
where pt.name = "gameday"
ORDER BY p.create_date
LIMIT 100;
It is not possible to have a single index that handles both pt.name and p.create_date. Is there any way to get them into the same table? I see, for example, that name seems to be redundantly in t and p.
tt_post_tags sounds like a many-to-many mapping table between posts and tags; is it? If so, what is name, the seems to be in tags and post_tags?
I think this
join tags AS t ON t.name = "gameday"
should be
join tags AS t ON t.name = "gameday" AND pt.tag_id = t.tag_id
If so, that might be the main problem. Please provide SHOW CREATE TABLE for the rest of the tables.
The following indexes may (or may not) help:
tags: (post_id, name)
tags: (name, tag_id)
posts: (create_date, id)
post_tags: (name, post_id)
More
A UNIQUE INDEX is an INDEX, so the second of these is redundant and should be dropped: UNIQUE(x), INDEX(x)
Index Cookbook: http://mysql.rjweb.org/doc.php/index_cookbook_mysql