MYSQL order large database by decimal 10,10 - mysql

I have about 25 million rows containing 0.183463545, 0.183423545, 0.183443545, 0.183443445, 0.183447545 and so on.
I need to order these however it currently takes around 20 seconds. Any way to speed it up? AFAIK, I have my index put it place correctly.
Thank you!
SELECT `a`.`float_val`,
`a`.`num_id`,
`b`.`userID`,
`c`.`img`,
`c`.`username`,
`b`.`img`,
`d`.`exterior`
FROM `a`
INNER JOIN `b` ON `b`.`num_id` = `a`.`num_id`
INNER JOIN `d` ON `d`.`id` = `b`.`item`
INNER JOIN `c` ON `c`.`userID` = `b`.`userID`
WHERE `float_val` IS NOT NULL
AND `float_val` BETWEEN 0 AND 1
AND `username` = 'ABC'
ORDER BY `float_val` LIMIT 100
Index is on float_val, num_id, userID
CREATE TABLE `float` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`num_id` bigint(11) DEFAULT NULL,
`float_val` decimal(10,10) DEFAULT NULL,
`userID` char(17) DEFAULT NULL,
`last_checked` datetime DEFAULT NULL,
`index10` smallint(11) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `floatID` (`num_id`),
UNIQUE KEY `num_id` (`num_id`,`userID`),
KEY `userID` (`userID`),
KEY `float_val` (`float_val`),
KEY `last_checked` (`last_checked`),
KEY `index10` (`index10`)
) ENGINE=InnoDB AUTO_INCREMENT=25750916 DEFAULT CHARSET=latin1;

Edited to reflect table definitions shown.
Start by doing a query to obtain the lowest float_val rows from table a.
SELECT a.id
FROM a
INNER JOIN b ON b.num_id = a.num_id
INNER JOIN d ON d.id = b.item
INNER JOIN c ON c.userID = b.userID
WHERE c.username = 'ABC'
AND a.float_val BETWEEN 0.0 AND 1.0
ORDER BY a.float_val
LIMIT 100
If you have an index on a(float_val,num_id), and another on c.username this will be reasonably fast. It will spit out the id values for the rows of a that are candidates for your query. (If you're using MyISAM, you need an index on a(float_val, num_id, id). By the way, the BETWEEN clause also implies IS NOT NULL.
Then use that as a subquery to complete your query, as follows.
SELECT a.float_val,
a.num_id,
b.userID,
c.img,
c.username,
b.img,
d.exterior
FROM a
INNER JOIN (
SELECT a.id
FROM a
INNER JOIN b ON b.num_id = a.num_id
INNER JOIN d ON d.id = b.item
INNER JOIN c ON c.userID = b.userID
WHERE c.username = 'ABC'
AND a.float_val BETWEEN 0.0 AND 1.0
ORDER BY a.float_val
LIMIT 100
) q ON a.ID = q.id
INNER JOIN b ON b.num_id = a.num_id
INNER JOIN d ON d.id = b.item
INNER JOIN c ON c.userID = b.userID
WHERE c.username = 'ABC'
AND a.float_val BETWEEN 0.0 AND 1.0
ORDER BY a.float_val LIMIT 100
This kind of query contains a deferred join. That dramatically reduces the number of rows of the full query that need to be subjected to ORDER BY ... LIMIT. Without the deferred join, your original query sorts an enormous mess of quite long rows just to discard all but the first hundred of them. That's why it takes so long.
This should help. The next optimization step is to look at the EXPLAIN output from this query and the exact definitions of your tables.
Pro tip: In queries of this complexity, always qualify column names with table names or aliases. That is, use a.float_val throughout, rather than just float_val. This is a kindness to the next person to look at the query.

Related

JOIN query taking long time and creating issue "converting HEAP to MyISAM

My query like below. here I used join query to take data. can u pls suggest how can I solve "converting HEAP to MyISAM" issue.
Can I use subquery here to update it? pls suggest how can I.
Here I have joined users table to check user is exist or not. can I refine it without join so that "converting HEAP to MyISAM" can solve.
Oh one more sometimes I will not check with specific user_id. like here I have added user_id = 16082
SELECT `user_point_logs`.`id`,
`user_point_logs`.`user_id`,
`user_point_logs`.`point_get_id`,
`user_point_logs`.`point`,
`user_point_logs`.`expire_date`,
`user_point_logs`.`point` AS `sum_point`,
IF(sum(`user_point_used_logs`.`point`) IS NULL, 0, sum(`user_point_used_logs`.`point`)) AS `minus`
FROM `user_point_logs`
JOIN `users` ON ( `users`.`id` = `user_point_logs`.`user_id` )
LEFT JOIN (SELECT *
FROM user_point_used_logs
WHERE user_point_log_id NOT IN (
SELECT DISTINCT return_id
FROM user_point_logs
WHERE return_id IS NOT NULL
AND user_id = 16082
)
)
AS user_point_used_logs
ON ( `user_point_logs`.`id` = `user_point_used_logs`.`user_point_log_used_id` )
WHERE expire_date >= 1563980400
AND `user_point_logs`.`point` >= 0
AND `users`.`id` IS NOT NULL
AND ( `user_point_logs`.`return_id` = 0
OR `user_point_logs`.`return_id` IS NULL )
AND `user_point_logs`.`user_id` = '16082'
GROUP BY `user_point_logs`.`id`
ORDER BY `user_point_logs`.`expire_date` ASC
DB FIDDLE HERE WITH STRUCTURE
Kindly try this, If it works... will optimize further by adding composite index.
SELECT
upl.id,
upl.user_id,
upl.point_get_id,
upl.point,
upl.expire_date,
upl.point AS sum_point,
coalesce(SUM(upl.point),0) AS minus -- changed from complex to readable
FROM user_point_logs upl
JOIN users u ON upl.user_id = u.id
LEFT JOIN (select supul.user_point_log_used_id from user_point_used_logs supul
left join user_point_logs supl on supul.user_point_log_id=supl.return_id and supl.return_id is null and supl.user_id = 16082) AS upul
ON upl.id=upul.user_point_log_used_id
WHERE
upl.user_id = 16082 and coalesce(upl.return_id,0)= 0
and upl.expire_date >= 1563980400 -- tip: if its unix timestamp change the datatype and if possible use range between dates
#AND upl.point >= 0 -- since its NN by default removing this condition
#AND u.id IS NOT NULL -- removed since the inner join matches not null
GROUP BY upl.id
ORDER BY upl.expire_date ASC;
Edit:
Try adding index in the column return_id on the table user_point_logs.
Since this column is used in join on derived query.
Or use composite index with user_id and return_id
Indexes:
user_point_logs: (user_id, expire_date)
user_point_logs: (user_id, return_id)
OR is hard to optimize. Decide on only one way to say whatever is being said here, then get rid of the OR:
AND ( `user_point_logs`.`return_id` = 0
OR `user_point_logs`.`return_id` IS NULL )
DISTINCT is redundant:
NOT IN ( SELECT DISTINCT ... )
Change
IF(sum(`user_point_used_logs`.`point`) IS NULL, 0,
sum(`user_point_used_logs`.`point`)) AS `minus`
to
COALESCE( ( SELECT SUM(point) FROM user_point_used_logs ... ), 0) AS minus
and toss LEFT JOIN (SELECT * FROM user_point_used_logs ... )
Since a PRIMARY KEY is a key, the second of these is redundant and can be DROPped:
ADD PRIMARY KEY (`id`),
ADD KEY `id` (`id`) USING BTREE;
After all that, we may need another pass to further simplify and optimize it.

MySQL - add clauses to left join

I have a table called properties (p) and another table called certificates (c). There can be more than one certificate allocated against each property or no certificate at all. I need to produce a query that uses a join and only displays one certificate from the certificates table per property. The one certificate that is shown needs to be the one with the most recent expiry date. There is a field in the certificates table named 'certificate_expiry_date'. The simple join would be p.property_id = c.certificate_property but this currently outputs all certificates.
My Query Attempt
Here's my query so far;
SELECT DISTINCT t.tenancy_property, t.*, p.*, c.* FROM tenancy t
INNER JOIN property p
on t.tenancy_property = p.property_id
LEFT JOIN
(
SELECT *
FROM certificate
WHERE certificate_expiry_date > CURDATE()
ORDER BY certificate_expiry_date DESC
LIMIT 1
) c ON p.property_id = c.certificate_property
WHERE t.tenancy_type='1' AND p.property_mains_gas_supply='1' AND p.property_availability='2' ORDER BY t.tenancy_id DESC LIMIT {$startpoint} , {$per_page}
This query executes fine but doesn't seem to take into account the left join on the certificates table.
Table structure for table certificate
CREATE TABLE IF NOT EXISTS `certificate` (
`certificate_id` int(11) NOT NULL AUTO_INCREMENT,
`certificate_property` int(11) DEFAULT NULL,
`certificate_type` tinyint(4) DEFAULT NULL,
`certificate_reference` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`certificate_start_date` date DEFAULT NULL,
`certificate_expiry_date` date DEFAULT NULL,
`certificate_notes` text COLLATE utf8_bin,
`certificate_renewal_instructed` tinyint(4) DEFAULT NULL,
`certificate_renewal_contractor` int(11) DEFAULT NULL,
PRIMARY KEY (`certificate_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=219 ;
If we only need to return one or two columns from the certificates table, we can sometimes use correlated subqueries in the SELECT list.
This approach has some performance implications for large tables; but for some use cases, with appropriate indexes available, this can be a workable approach.
SELECT p.id
, p.somecol
, ( SELECT c.col
FROM certificate c
WHERE c.property_id = p.id
ORDER BY c.date_added DESC, c.id DESC
LIMIT 1
) AS most_recent_cert_col
, ( SELECT c.date_added
FROM certificate c
WHERE c.property_id = p.id
ORDER BY c.date_added DESC, c.id DESC
LIMIT 1
) AS most_recent_cert_date_added
FROM property p
WHERE ...
ORDER BY ...
Updated answer with your updated information
Something like this?
(Note: This answer assumes that each property has at least one certificate, or else the sub-query qMostRecentExpire may fail)
select
p.property_id
, p.*
, ( select
c.certificate_id
from
certificates as c
where
c.certificate_property = p.property_id -- all the cert of this property
and c.certificate_expiry_date < CURDATE() -- cert has expired
order by c.certificate_expiry_date desc
limit 1 -- most recent one
) as qMostRecentExpire
from
properties as p
Updated answer after knowing that some properties may have no certificates
select
p.property_id
, p.*
, ( select
c.certificate_id
from
certificates as c
where
c.certificate_property = p.property_id -- all the cert of this property
and c.certificate_expiry_date < CURDATE() -- cert has expired
order by c.certificate_expiry_date desc
limit 1 -- most recent one
) as qMostRecentExpire
from
properties as p
, certificates as c -- inner join : properties that
where -- has not cert will be dropped
p.property_id = c.certificate_property

Optimize table to avoid using temporary and using filesort

I have a messages table
CREATE TABLE `messages` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`author` int(11) DEFAULT NULL,
`time` int(10) unsigned DEFAULT NULL,
`text` text CHARACTER SET latin1,
`dest` int(11) unsigned DEFAULT NULL,
`type` tinyint(4) unsigned DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `author` (`author`),
KEY `dest` (`dest`)
) ENGINE=InnoDB AUTO_INCREMENT=2758 DEFAULT CHARSET=utf8;
I need to get messages between two users
SELECT
...
FROM
`messages` m
LEFT JOIN `people` p ON m.author = p.id
WHERE
(author = 1 AND dest = 2)
OR (author = 2 AND dest = 1)
ORDER BY
m.id DESC
LIMIT 0, 25
When I EXPLAIN this query I get
Please excuse any ignorance, but is there a way I could optimize this table to avoid using a temporary table and filesort for this query, for now it is not causing a problem but I'm pretty sure in future it is going to be troublesome?
First, I'm guessing the left join is not necessary. Second, consider using union all instead. Then one approach is:
(SELECT ...
FROM messages m JOIN
people p
ON m.author = p.id
WHERE author = 1 AND dest = 2
ORDER BY id DESC
LIMIT 25
)
UNION ALL
(SELECT ...
FROM messages m JOIN
people p
ON m.author = p.id
WHERE author = 2 AND dest = 1
ORDER BY id DESC
LIMIT 25
)
ORDER BY m.id DESC
LIMIT 0, 25
With this query, an index on messages(author, dest, id) should make it fast. (Note: you might need to include m.id in the SELECT list.)
To build on Gordon's answer:
SELECT m2..., p...
FROM
(
( SELECT id
FROM messages
WHERE author = 1
AND dest = 2
ORDER BY id DESC
LIMIT 75
)
UNION ALL
(
SELECT id
FROM messages
WHERE author = 2
AND dest = 1
ORDER BY id DESC
LIMIT 75
)
) ORDER BY id DESC
LIMIT 50, 25 ) AS m1
JOIN messages AS m2 ON m2.id = m1.id
JOIN people p ON p.id = m2.author
ORDER BY m1.id DESC
Notes:
Gordon's index is now "covering". (This adds efficiency, thereby masking some of the other stuff I added.)
Lazy evaluation means that it does not need to shovel all the bulky fields of more than 25 rows around. Instead, only 25 need to be handled. Also, I avoid touching people to start with.
The code shows what "page 3" should look like. Note LIMIT 75 versus LIMIT 50,25.
"Pagination via OFFSET" has several problems. See my blog.
This formulation still will not avoid "filesort" and "using temp". But speed is the real goal, correct? ("Filesort" is a misnomer -- if you don't include that TEXT column, the sort will be done in RAM.)
When you add INDEX(author, dest, id), INDEX(author) becomes redundant; drop it.
The ALL after UNION is not the default for UNION, but it avoids an extra pass (and temp table) to de-duplicate the data.
There will still be 2 or 3 temp tables involved. See EXPLAIN FORMAT=JSON SELECT ... for details.

How to speed up left join queries by indexing?

At the moment I am experiencing some slower MySQL queries in my application which I want to speed up. Unfortunately I’m not quite sure which is the correct way to do it.
I have the following (fictitious) tables: Book, Page and Word.
Word is child of Page by word_page_id
Page is child of Book by page_book id
I already have individual indexes on page_book_id, word_page_id, book_user_id and book_flag_delete.
SELECT `book`.*, COUNT(word_id) AS `word_amount` FROM `book`
LEFT JOIN `page` ON page_book_id = book_id
LEFT JOIN `word` ON word_page_id = paragraph_id
WHERE (book_user_id = 1) AND (book_flag_delete IS NULL)
GROUP BY `book_id`
ORDER BY `book_id` ASC LIMIT 100
SELECT COUNT(DISTINCT `book_id`) AS `book_row_count` FROM `book`
LEFT JOIN `page` ON page_book_id = book_id
LEFT JOIN `word` ON word_page_id = page_id
WHERE (book_user_id = 59) AND (book_flag_delete IS NULL)
Any ideas how to speed up such queries?
Is there extra indexing involved?
Set indexes on the fields you use for joining.
Further make sure that these have both the same datatype, encoding, and collation, else the index will also not be used.
mysql> EXPLAIN <query> will show you the actually used fields (key column in output) and the available indexes (possible_keys output field).
For this query:
SELECT b.*, COUNT(w.word_id) AS `word_amount`
FROM `book` b LEFT JOIN
`page` p
ON p.page_book_id = b.book_id LEFT JOIN
`word` w
ON w.word_page_id = p.paragraph_id
WHERE (b.book_user_id = 1) AND (b.book_flag_delete IS NULL)
GROUP BY b.`book_id`
ORDER BY b.`book_id` ASC
LIMIT 100;
The best indexes are: book(user_id, book_flag_delete, book_id), page(page_book_id, paragraph_id), and word(word_page_id, word_id).
However, the overall group by might be expensive. You might try writing the query as:
SELECT b.*,
(SELECT COUNT(w.word_id)
FROM `page` p JOIN
`word` w
ON w.word_page_id = p.paragraph_id
WHERE p.page_book_id = b.book_id
) AS `word_amount`
FROM `book` b LEFT JOIN
WHERE (b.book_user_id = 1) AND (b.book_flag_delete IS NULL)
ORDER BY b.`book_id` ASC
LIMIT 100;
The same indexes indexes work here. But, this query should avoid a group by on all the data at once (instead, it uses the indexes for the aggregation).
The optimal schema for a many-to-many mapping table is
CREATE TABLE XtoY (
# No surrogate id for this table
x_id MEDIUMINT UNSIGNED NOT NULL, -- For JOINing to one table
y_id MEDIUMINT UNSIGNED NOT NULL, -- For JOINing to the other table
# Include other fields specific to the 'relation'
PRIMARY KEY(x_id, y_id), -- When starting with X
INDEX (y_id, x_id) -- When starting with Y
) ENGINE=InnoDB;
The details on 'why' are in my index cookbook
In your select you're gonna want to refrain from using the wildcard "*" to grab columns. Plus utilize aliases ALWAYS!! This will keep your db from having to create a "virtual" alias.
select book1.column1, book1.column2, page1.column1
from book book1
left join page page1
on page1.page_book_id = book1.book_id
..... blah

MySQL Query is Extremely Slow

Hello I am looking for ways to optimize the mysql query, basically I am fetching the articles for the user which belong to category_id = 25 and source_id not in a table where I store source id's from which user has unsubscribed.
select
a.article_id,
a.article_title,
a.source_id,
a.article_publish_date,
a.article_details,
n.source_name
from sources n
INNER JOIN articles a
ON (a.source_id = n.source_id)
WHERE n.category_id = 25
AND n.source_id NOT IN(select
source_id
from news_sources_deselected
WHERE user_id = 5)
ORDER BY a.article_publish_date DESC
Schema for Articles Table
CREATE TABLE IF NOT EXISTS `articles` (<br>
`article_id` int(255) NOT NULL auto_increment,<br>
`article_title` varchar(255) NOT NULL,<br>
`source_id` int(255) NOT NULL,<br>
`article_publish_date` bigint(255) NOT NULL,<br>
`article_details` text NOT NULL,<br>
PRIMARY KEY (`article_id`),<br>
KEY `source_id` (`source_id`),<br>
KEY `article_publish_date` (`article_publish_date`)<br>
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Contains articles.';
Structure for Sources table
CREATE TABLE IF NOT EXISTS `sources` (<br>
`source_id` int(255) NOT NULL auto_increment,<br>
`category_id` int(255) NOT NULL,<br>
`source_name` varchar(255) character set latin1 NOT NULL,<br>
`user_id` int(255) NOT NULL,<br>
PRIMARY KEY (`source_id`),<br>
KEY `category_id` (`category_id`),<br>
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='News Sources.'
The articles table has around 0.3 Million records and sources table contains around 1000 records, the query takes around 180 seconds to execute.
Any help will be greatly appreciated.
Try using a derieved query with IS NULL condition. You explain says there is a dependent subquery. Ignore using it and use derieved query for your problem. This will increase the performance
select
a.article_id,
a.article_title,
a.source_id,
a.article_publish_date,
a.article_details,
n.source_name
from sources n
INNER JOIN articles a
ON (a.source_id = n.source_id)
LEFT JOIN (SELECT *
FROM news_sources_deselected
WHERE user_id = 5) AS nsd
ON nsd.source_id = n.source_id
WHERE n.category_id = 25
AND nsd.source_id IS NULL
ORDER BY a.article_publish_date DESC
Use EXPLAIN in front of your query and analyze results.
Here you can find how to start your optimization work.
I see few issues you could check.
You're not using relations despite using InnoDB engine.
You're selecting fields without index.
You're selecting all rows at once.
Do you need all those rows at once? Maybe consider splitting this query to multiple shards (paging)?
Try this query
select
a.article_id,
a.article_title,
a.source_id,
a.article_publish_date,
a.article_details,
n.source_name
from
sources n
INNER JOIN
articles a
ON
n.category_id = 25 AND
a.source_id = n.source_id
INNER JOIN
news_sources_deselected nsd
ON
nsd.user_id <> 5 AND n.source_id = nsd.source_id
ORDER BY
a.article_publish_date DESC
I have removed the extra query and added news_sources_deselected in join by accepting all source_id for user_id other than with id 5.
Or we can go for using only needed records for join as user raheelshan has mentioned
select
a.article_id,
a.article_title,
a.source_id,
a.article_publish_date,
a.article_details,
n.source_name
from
(select
*
from
sources
where
category_id = 25) n
INNER JOIN
articles a
ON
a.source_id = n.source_id
INNER JOIN
(select
*
from
news_sources_deselected
where
user_id <> 5) nsd
ON
n.source_id = nsd.source_id
ORDER BY
a.article_publish_date DESC
Hope this helps..
I fixed the issue by partitioning the table, but I am still open to suggestions.