MySQL innoDB: Long time of query execution - mysql

I'm having troubles to run this SQL:
I think it's a index problem but I don't know because I dind't make this database and I'm just a simple programmer.
The problem is, that table has 64260 records, so that query gets crazy when executing, I have to stop mysql and run again because the computer get frozen.
Thanks.
EDIT: table Schema
CREATE TABLE IF NOT EXISTS `value_magnitudes` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`value` float DEFAULT NULL,
`magnitude_id` int(11) DEFAULT NULL,
`sdi_belongs_id` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`reading_date` datetime DEFAULT NULL,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1118402 ;
Query
select * from value_magnitudes
where id in
(
SELECT min(id)
FROM value_magnitudes
WHERE magnitude_id = 234
and date(reading_date) >= '2013-04-01'
group by date(reading_date)
)
EDIT2

First, add an index on (magnitude_id, reading_date):
ALTER TABLE
ADD INDEX magnitude_id__reading_date__IX -- just a name for the index
(magnitude_id, reading_date) ;
Then try this variation:
SELECT vm.*
FROM value_magnitudes AS vm
JOIN
( SELECT MIN(id) AS id
FROM value_magnitudes
WHERE magnitude_id = 234
AND reading_date >= '2013-04-01' -- changed so index is used
GROUP BY DATE(reading_date)
) AS vi
ON vi.id = vm.id ;
The GROUP BY DATE(reading_date) will still need to apply the function to all the selected (thorugh the index) rows and that cannot be improved, unless you follow #jurgen's advice and split the column into date and time columns.

Since you want to get results for every day you need to extract the date from a datetime column with the function date(). That makes indexes useless.
You can split up the reading_date column into reading_date and reading_time. Then you can run the query without a function and indexes will work.
Additionally you can change the query into a join
select *
from value_magnitudes v
inner join
(
SELECT min(id) as id
FROM value_magnitudes
WHERE magnitude_id = 234
and reading_date >= '2013-04-01'
group by reading_date
) x on x.id = v.id

For starters, I would change your query to:
select * from value_magnitudes where id = (
select min(id) from value_magnitudes
where magnitude_id = 234
and DATE(reading_date) >= '2013-04-01'
)
You don't need to use the IN clause when the subquery is only going to return one record.
Then, I would make sure you have an index on magnitude_id and reading_date (probably a two field index) as that's what you are querying against in the subquery. Without that index, you are scanning the table each time.
Also if possible change magnitude_id and reading_date to non null. Null values and indexes are not great fits.

Related

MySQL never completes query with a subquery, but completes when queries are run separately

This is the query that never finishes:
SELECT modelId, `timestamp` FROM thread_view WHERE id IN
(SELECT max(id) FROM thread_view
WHERE viewerId = 1 AND modelType = 'LEASE' AND modelId IN
(15,615,618,660) GROUP BY modelId);
When I run just the subquery, it completes in 12ms and returns two ids. When I hardcode those 2 ids in place of the subquery, the outerquery returns two rows and completes in 0.3ms.
As you can see, both parts of the query run incredibly quickly and it's not like we're dealing with a lot of data here. When I run the EXPLAIN:
The explain didn't reveal any problems to me. Is this just a bug with MySQL?
EDIT: mysql --version gives: mysql Ver 8.0.25 for macos11.3 on x86_64 (Homebrew)
EDIT #2: As requested, here is the create table statement:
CREATE TABLE `thread_view` (
`id` int unsigned NOT NULL AUTO_INCREMENT,
`timestamp` bigint DEFAULT NULL,
`viewerId` bigint DEFAULT NULL,
`modelType` varchar(16) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL,
`modelId` bigint DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `viewerId_modelType_modelId` (`viewerId`,`modelType`,`modelId`)
) ENGINE=InnoDB AUTO_INCREMENT=50582 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;
If the goal is really to grab the greatest timestamp per model (also noted by #Shaharyar):
SELECT modelId
, MAX(`timestamp`)
FROM thread_view
WHERE modelId IN (15,615,618,660)
AND viewerId = 1
AND modelType = 'LEASE'
GROUP BY modelId
;
I would suggest that you instead phrase this using window functions. I'm not 100% sure that this is equivalent (you don't explain the data or what your query is supposed to be doing).
But assuming that id is unique, then try:
select t.*
from (select t.*,
row_number() over (partition by modelId order by id desc) as seqnum
from thread_view t
where viewerId = 1 and
modelType = 'LEASE' and
modelId in (15, 615, 618, 660)
) t
where seqnum = 1;

Mysql query with multiple selects results in high CPU load

I'm trying to do a link exchange script and run into a bit of trouble.
Each link can be visited by an IP address a number of x times (frequency in links table). Each visit costs a number of credits (spend limit given in limit in links table)
I've got the following tables:
CREATE TABLE IF NOT EXISTS `contor` (
`key` varchar(25) NOT NULL,
`uniqueHandler` varchar(30) DEFAULT NULL,
`uniqueLink` varchar(30) DEFAULT NULL,
`uniqueUser` varchar(30) DEFAULT NULL,
`owner` varchar(50) NOT NULL,
`ip` varchar(15) DEFAULT NULL,
`credits` float NOT NULL,
`tstamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`key`),
KEY `uniqueLink` (`uniqueLink`),
KEY `uniqueHandler` (`uniqueHandler`),
KEY `uniqueUser` (`uniqueUser`),
KEY `owner` (`owner`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE IF NOT EXISTS `links` (
`unique` varchar(30) NOT NULL DEFAULT '',
`url` varchar(1000) DEFAULT NULL,
`frequency` varchar(5) DEFAULT NULL,
`limit` float NOT NULL DEFAULT '0',
PRIMARY KEY (`unique`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
I've got the following query:
$link = MYSQL_QUERY("
SELECT *
FROM `links`
WHERE (SELECT count(key) FROM contor WHERE ip = '$ip' AND contor.uniqueLink = links.unique) <= `frequency`
AND (SELECT sum(credits) as cost FROM contor WHERE contor.uniqueLink = links.unique) <= `limit`")
There are 20 rows in the table links.
The problem is that whenever there are about 200k rows in the table contor the CPU load is huge.
After applying the solution provided by #Barmar:
Added composite index on (uniqueLink, ip) and droping all other indexes except PRIMARY, EXPLAIN gives me this:
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY l ALL NULL NULL NULL NULL 18
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 15
2 DERIVED pop_contor index NULL contor_IX1 141 NULL 206122
Try using a join rather than a correlated subquery.
SELECT l.*
FROM links AS l
LEFT JOIN (
SELECT uniqueLink, SUM(ip = '$ip') AS ip_visits, SUM(credits) AS total_credits
FROM contor
GROUP BY uniqueLink
) AS c
ON c.uniqueLink = l.unique AND ip_visits <= frequency AND total_credits <= limit
If this doesn't help, try adding an index on contor.ip.
The current query is of the form:
SELECT l.*
FROM `links` l
WHERE l.frequency >= ( SELECT COUNT(ck.key)
FROM contor ck
WHERE ck.uniqueLink = l.unique
AND ck.ip = '$ip'
)
AND l.limit >= ( SELECT SUM(sc.credits)
FROM contor sc
WHERE sc.uniqueLink = l.unique
)
Those correlated subqueries are going to each your lunch. And your lunchbox too.
I'd suggest testing an inline view that performs both of the aggregations from contor in one pass, and then join the result from that to the links table.
Something like this:
SELECT l.*
FROM ( SELECT c.uniqueLink
, SUM(c.ip = '$ip' AND c.key IS NOT NULL) AS count_key
, SUM(c.credits) AS sum_credits
FROM `contor` c
GROUP
BY c.uniqueLink
) d
JOIN `links` l
ON l.unique = d.uniqueLink
AND l.frequency >= d.count_key
AND l.limit >= d.sum_credits
For optimal performance of the aggregation inline view query, provide a covering index that MySQL can use to optimize the GROUP BY (avoiding a Using filesort operation)
CREATE INDEX `contor_IX1` ON `contor` (`uniqueLink`, `credits`, `ip`) ;
Adding that index renders the uniqueLink index redundant, so also...
DROP INDEX `uniqueLink` ON `contor` ;
EDIT
Since we have a guarantee that contor.key column is non-NULL (i.e. the NOT NULL constraint), this part of the query above is unneeded AND c.key IS NOT NULL, and can be removed. (I also removed the key column from the covering index definition above.)
SELECT l.*
FROM ( SELECT c.uniqueLink
, SUM(c.ip = '$ip') AS count_key
, SUM(c.credits) AS sum_credits
FROM `contor` c
GROUP
BY c.uniqueLink
) d
JOIN `links` l
ON l.unique = d.uniqueLink
AND l.frequency >= d.count_key
AND l.limit >= d.sum_credits

Optimize a query

How can I proceed to make my response time more faster, approximately the average time of response is 0.2s ( 8039 records in my items table & 81 records in my tracking table )
Query
SELECT a.name, b.cnt FROM `items` a LEFT JOIN
(SELECT guid, COUNT(*) cnt FROM tracking WHERE
date > UNIX_TIMESTAMP(NOW() - INTERVAL 1 day ) GROUP BY guid) b ON
a.`id` = b.guid WHERE a.`type` = 'streaming' AND a.`state` = 1
ORDER BY b.cnt DESC LIMIT 15 OFFSET 75
Tracking table structure
CREATE TABLE `tracking` (
`id` bigint(11) NOT NULL AUTO_INCREMENT,
`guid` int(11) DEFAULT NULL,
`ip` int(11) NOT NULL,
`date` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `i1` (`ip`,`guid`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=4303 DEFAULT CHARSET=latin1;
Items table structure
CREATE TABLE `items` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`guid` int(11) DEFAULT NULL,
`type` varchar(255) DEFAULT NULL,
`name` varchar(255) DEFAULT NULL,
`embed` varchar(255) DEFAULT NULL,
`url` varchar(255) DEFAULT NULL,
`description` text,
`tags` varchar(255) DEFAULT NULL,
`date` int(11) DEFAULT NULL,
`vote_val_total` float DEFAULT '0',
`vote_total` float(11,0) DEFAULT '0',
`rate` float DEFAULT '0',
`icon` text CHARACTER SET ascii,
`state` int(11) DEFAULT '0',
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=9258 DEFAULT CHARSET=latin1;
Your query, as written, doesn't make much sense. It produces all possible combinations of rows in your two tables and then groups them.
You may want this:
SELECT a.*, b.cnt
FROM `items` a
LEFT JOIN (
SELECT guid, COUNT(*) cnt
FROM tracking
WHERE `date` > UNIX_TIMESTAMP(NOW() - INTERVAL 1 day)
GROUP BY guid
) b ON a.guid = b.guid
ORDER BY b.cnt DESC
The high-volume data in this query come from the relatively large tracking table. So, you should add a compound index to it, using the columns (date, guid). This will allow your query to random-access the index by date and then scan it for guid values.
ALTER TABLE tracking ADD INDEX guid_summary (`date`, guid);
I suppose you'll see a nice performance improvement.
Pro tip: Don't use SELECT *. Instead, give a list of the columns you want in your result set. For example,
SELECT a.guid, a.name, a.description, b.cnt
Why is this important?
First, it makes your software more resilient against somebody adding columns to your tables in the future.
Second, it tells the MySQL server to sling around only the information you want. That can improve performance really dramatically, especially when your tables get big.
Since tracking has significantly fewer rows than items, I will propose the following.
SELECT i.name, c.cnt
FROM
(
SELECT guid, COUNT(*) cnt
FROM tracking
WHERE date > UNIX_TIMESTAMP(NOW() - INTERVAL 1 day )
GROUP BY guid
) AS c
JOIN items AS i ON i.id = c.guid
WHERE i.type = 'streaming'
AND i.state = 1;
ORDER BY c.cnt DESC
LIMIT 15 OFFSET 75
It will fail to display any items for which cnt is 0. (Your version displays the items with NULL for the count.)
Composite indexes needed:
items: The PRIMARY KEY(id) is sufficient.
tracking: INDEX(date, guid) -- "covering"
Other issues:
If ip is an IP-address, it needs to be INT UNSIGNED. But that covers only IPv4, not IPv6.
It seems like date is not just a "date", but really a date+time. Please rename it to avoid confusion.
float(11,0) -- Don't use FLOAT for integers. Don't use (m,n) on FLOAT or DOUBLE. INT UNSIGNED makes more sense here.
OFFSET is naughty when it comes to performance -- it must scan over the skipped records. But, in your query, there is no way to avoid collecting all the possible rows, sorting them, stepping over 75, and only finally delivering 15 rows. (And, with no more than 81, it won't be a full 15.)
What version are you using? There have been important changes to the Optimization of LEFT JOIN ( SELECT ... ). Please provide EXPLAIN SELECT for each query under discussion.

Sqlite alias column give an error

this is my table
CREATE TABLE IF NOT EXISTS `calls` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`date` timestamp NOT NULL,
`type` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB
and there are data on it its
now i use this query for get type data with same day
SELECT user.name_first calls.date as days, (SELECT GROUP_CONCAT(type) FROM calls WHERE DAY(date) = DAY(days) ) FROM calls JOIN user ON user.user_id = calls.user_id WHERE calls.id IN (SELECT MAX(id) FROM calls GROUP BY DAY(calls.date))
as we see i do this calls.date as days and try to get type for this data by
(SELECT GROUP_CONCAT(type) FROM calls WHERE DAY(date) = DAY(days) )
its work on mysql result in mysql
but when i try it on sqlite its say
no such column: days (code 1): , while compiling
i read sqlite support alias

mysql query with multiple groups?

I have a query that gets product IDs based on keywords.
SELECT indx_search.pid
FROM indx_search
LEFT JOIN word_index_mem ON (word_index_mem.word = indx_search.word)
WHERE indx_search.word = "phone"
GROUP BY indx_search.pid
ORDER BY indx_search.pid ASC
LIMIT 0,20
This works well but now I'm trying to go a step further and implement "price range" into this query.
CREATE TABLE `price_range` (
`pid` int(11) NOT NULL,
`range_id` tinyint(3) NOT NULL,
PRIMARY KEY (`pid`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
This table simply contains product IDs and a range_id. The price range values are stored here:
CREATE TABLE `price_range_values` (
`ID` tinyint(3) NOT NULL AUTO_INCREMENT,
`rangeFrom` float(10,2) NOT NULL,
`rangeTo` float(10,2) NOT NULL,
PRIMARY KEY (`ID`)
) ENGINE=MyISAM AUTO_INCREMENT=32 DEFAULT CHARSET=latin1
I want to GROUP price_range.range_id with COUNT() of how many products match the certain price range within the current query. I still want to receive my 20 results of product IDs.
So something along the lines of:
SELECT indx_search.pid, price_range.range_id as PriceRangeID, COUNT(price_range.range_id) as PriceGroupTotal
FROM indx_search
LEFT JOIN windex_mem ON ( windex_mem.word = indx_search.word )
LEFT JOIN price_range ON ( price_range.pid = indx_search.pid )
WHERE indx_search.word = "memory"
GROUP BY indx_search.pid, PriceRangeID
ORDER BY indx_search.pid ASC
LIMIT 0 , 20
Is this possible to accomplish without busting an additional query?
Try to use a subquery with a LIMIT clause, e.g. -
SELECT
i_s.pid, p_r.range_id as PriceRangeID, COUNT(p_r.range_id) as PriceGroupTotal
FROM
(SELECT * FROM indx_search WHERE i_s.word = 'memory' ORDER BY i_s.pid ASC LIMIT 0 , 20) i_s
LEFT JOIN
windex_mem w_m ON w_m.word = i_s.word
LEFT JOIN
price_range p_r ON p_r.pid = i_s.pid
GROUP BY
i_s.pid, PriceRangeID