There are some very large tables (TargetTable) that I am querying against, and there is a particular procedure that get stuck in its second iteration and never finishes nor crashes. The first iteration always finishes in less than a few minutes, regardless of the start of the range (loopIndex) or size of the of the range (loopStepShort).
I look forward hearing your thoughts and suggestions.
[Update 1] This problem goes away if I do one of the following:
Remove the nested part of the inner-join;
Use an in-memory temporary table for the nested part of the inner join (thanks to #SashaPachev);
Run each loop iteration outside of the while loop;
Use a smaller TargetTable.
[Update 2] Solved! I think the problem might have been occurred, when some of databases indexes were not copied on the process of database transition. Because, when I tried to reproduce an example, it was occurring for non-indexed tables (High CPU usage and virtually infinite loop step) MariaDB Server, JIRA.
The custom configurations of the MySQL InnoDB engine (10.0.21-MariaDB Server, Linux x86_64, Fedora v.21) is as the following:
innodb_buffer_pool_size = 2G
net_write_timeout = 1800
net_read_timeout = 1800
join_buffer_size = 2G
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 4M
max_allowed_packet = 4G
key_buffer = 2G
sort_buffer_size = 512K
And the procedure body is as the following:
SET loopIndex = 0;
SET loopMax = 20000000;
SET loopStepShort = 10000;
WHILE loopIndex < loopMax do
UPDATE TargetTable AS t0,
(SELECT __index, sessionStartAge
FROM SubjectTable AS t0
INNER JOIN (SELECT t0.id, t0.admission,
if(t0.startage is null and t0.endage is null, 21,
if(least(t0.startage, t0.endage) <= 1, 1,
if(least(t0.startage, t0.endage) <= 4, 2,
if(least(t0.startage, t0.endage) <= 9, 3,
if(least(t0.startage, t0.endage) <= 14, 4,
if(least(t0.startage, t0.endage) <= 19, 5,
if(least(t0.startage, t0.endage) <= 24, 6,
if(least(t0.startage, t0.endage) <= 29, 7,
if(least(t0.startage, t0.endage) <= 34, 8,
if(least(t0.startage, t0.endage) <= 39, 9,
if(least(t0.startage, t0.endage) <= 44, 10,
if(least(t0.startage, t0.endage) <= 49, 11,
if(least(t0.startage, t0.endage) <= 54, 12,
if(least(t0.startage, t0.endage) <= 59, 13,
if(least(t0.startage, t0.endage) <= 64, 14,
if(least(t0.startage, t0.endage) <= 69, 15,
if(least(t0.startage, t0.endage) <= 74, 16,
if(least(t0.startage, t0.endage) <= 79, 17,
if(least(t0.startage, t0.endage) <= 84, 18,
if(least(t0.startage, t0.endage) <= 89, 19,
if(least(t0.startage, t0.endage) <= 120, 20, 21))))))))))))))))))))) AS sessionStartAge
FROM SubjectTable AS t0
INNER JOIN ids AS t1 ON t0.id = t1.id
AND t1.id >= loopIndex
AND t1.id < (loopIndex + loopStepShort)
GROUP BY t0.id, t0.admission) AS t1
ON t0.id = t1.id AND t0.admission = t1.admission) AS t1
SET t0.sessionStartAge = t1.sessionStartAge
WHERE t0.__index = t1.__index;
SET loopIndex = loopIndex + loopStepShort;
END WHILE;
Finally, below are approximate dimensions of the tables:
TABLE: ids:
TABLE ROWS: ~1,500,000 records,
DATA LENGTH: ~250 MB,
INDEX LENGTH: ~140 MB,
TABLE SIZE: ~400 MB
TABLE: TargetTable:
TABLE ROWS: ~6,500,000 records,
DATA LENGTH: ~4 GB,
INDEX LENGTH: ~350 MB,
TABLE SIZE: ~4.35 MB
TABLE: SubjectTable:
TABLE ROWS: ~6,500,000 records,
DATA LENGTH: ~550 MB,
INDEX LENGTH: N/A,
TABLE SIZE: ~550 MB
Many thanks in advance.
I guess I have to raise a bug report to Oracle/MariaDB, and update the post.
Try this (disclaimer - untested, may contain syntax errors or bugs):
DROP TABLE IF EXISTS t1;
CREATE TEMPORARY TABLE t1 (key(id)) ENGINE=MEMORY SELECT t0.id, t0.admission,
if(t0.startage is null and t0.endage is null, 21,
if(least(t0.startage, t0.endage) <= 1, 1,
if(least(t0.startage, t0.endage) <= 4, 2,
if(least(t0.startage, t0.endage) <= 9, 3,
if(least(t0.startage, t0.endage) <= 14, 4,
if(least(t0.startage, t0.endage) <= 19, 5,
if(least(t0.startage, t0.endage) <= 24, 6,
if(least(t0.startage, t0.endage) <= 29, 7,
if(least(t0.startage, t0.endage) <= 34, 8,
if(least(t0.startage, t0.endage) <= 39, 9,
if(least(t0.startage, t0.endage) <= 44, 10,
if(least(t0.startage, t0.endage) <= 49, 11,
if(least(t0.startage, t0.endage) <= 54, 12,
if(least(t0.startage, t0.endage) <= 59, 13,
if(least(t0.startage, t0.endage) <= 64, 14,
if(least(t0.startage, t0.endage) <= 69, 15,
if(least(t0.startage, t0.endage) <= 74, 16,
if(least(t0.startage, t0.endage) <= 79, 17,
if(least(t0.startage, t0.endage) <= 84, 18,
if(least(t0.startage, t0.endage) <= 89, 19,
if(least(t0.startage, t0.endage) <= 120, 20, 21)))))))))))))))))))) as sessionStartAge,
FROM SubjectTable AS t0
INNER JOIN ids AS t1 ON t0.id = t1.id
AND t1.id >= loopIndex
AND t1.id < (loopIndex + loopStepShort)
GROUP BY t0.id, t0.admission;
UPDATE TargetTable AS t0,
(SELECT __index, sessionStartAge
FROM SubjectTable AS t0
INNER JOIN t1 ON t0.id = t1.id AND t0.admission = t1.admission) AS t2
SET t0.sessionStartAge = t1.sessionStartAge
WHERE t0.__index = t2.__index;
The idea is to replace the inner sub-query with a temporary table with a key, so that the outside join could use that key.
Related
I am trying to get enquiries that their teams are in 15, 9, 25, 26, 23, 18, 12 and null. The problems is the query doesn't retrieves the NULL records.
I have tried to use OR team IS NULL the query gives me all other statuses.
SELECT e.*,
FROM enquiries e
WHERE e.timestamp BETWEEN '1293840000' AND '1469055599'
AND e.status IN(1) AND e.team IN(15, 9, 25, 26, 23, 18, 12, NULL)
ORDER BY e.timestamp ASC
Any help please
Place OR condition in brackets
SELECT e.*,
FROM enquiries e
WHERE e.timestamp BETWEEN '1293840000' AND '1469055599'
AND e.status IN(1) AND ( e.team IN (15, 9, 25, 26, 23, 18, 12) OR e.team IS NULL)
ORDER BY e.timestamp ASC
I made the following case when statement in my sql:
SELECT
*,
CASE
WHEN lead_time < 14 THEN 0
WHEN (14 <= lead_time < 21 AND days_passed = 8) THEN 1
WHEN
(21 <= lead_time < 28
AND days_passed = 15)
THEN
1
WHEN
(28 <= lead_time < 42
AND days_passed = 22)
THEN
1
WHEN
(42 <= lead_time < 56
AND days_passed = 36)
THEN
1
WHEN
(56 <= lead_time < 84
AND days_passed = 29)
THEN
1
WHEN
(56 <= lead_time < 84
AND days_passed = 50)
THEN
1
WHEN (lead_time > 84 AND days_passed = 36) THEN 1
WHEN (lead_time > 84 AND days_passed = 57) THEN 1
ELSE 0
END AS send_email
I do not get any error. However, when I check the results I get:
# lead_time, days_passed, send_email
99, 15, 1
99, 22, 1
99, 15, 1
99, 8, 1
99, 8, 1
99, 8, 1
99, 8, 1
85, 29, 1
57, 50, 1
18, 36, 1
99, 22, 1
99, 22, 1
99, 22, 1
99, 22, 1
99, 22, 1
15, 15, 1
15, 15, 1
99, 8, 1
99, 8, 1
99, 8, 1
99, 8, 1
It seems as if the 'and' in the query behaves as an 'or'. Any idea why?.
Thanks,
MySQL doesn't support a syntax:
min <= expr <= max
Such kinds of expressions can be written as:
WHERE expr >= min AND expr <= max
or with using operator BETWEEN (please note that BETWEEN is inclusive):
WHERE expr BETWEEN min AND max
Comparison operators in MySQL: http://dev.mysql.com/doc/refman/5.5/en/comparison-operators.html
The comparison 14 <= lead_time < 21 AND days_passed = 8 is checked in sequence, so in a way you have:
((14 <= lead_time) < 21) AND (days_passed = 8)
Which is always true because 14 <= lead_time equals 1 and thus your comparison is equal to:
( 1 < 21 ) AND days_passed = 8
You should use a between or an and for each comparison.
Considering a simple MySQL table with a id column and a int column, I need to count how many times I have a gap equal or greater than certain value.
Let's say that value will be 10.
Given the following sample records:
{1, 2, 3} = 1 time
{1, 2, 3, 4, 5, 6, 7, 8, 9} = 1 time;
{1, 2, 3, 14, 17} = 2 times (1, 2, 3 and 14, 17);
{1, 2, 3, 14, 20, 40, 42} = 3 times (1, 2, 3 and 14, 20 and 40, 42);
Is it possible resolve that with mysql?
Yes. For table t with columns id and num this will be seems like this:
SET #n = 10;
SELECT 1 + SUM(COALESCE(t3.f, 0))
FROM (
SELECT DISTINCT t1.num, (
SELECT CASE WHEN t2.num - t1.num > #n THEN 1 ELSE 0 END
FROM t t2
WHERE t2.num > t1.num
ORDER BY num LIMIT 1
) AS f
FROM t t1
) t3
SELECT id, server_id, start_time, end_time
FROM errors
WHERE server_id in (3, 12, 24, 25, 26, 27, 28, 29, 30)
ORDER BY id DESC
LIMIT 9
This is the query I'm trying to run to give me results where the server_id = 3, 12, 24, 25, 26, 27, 28, 29, 30. Instead, what I receive is server_id = 25, 25, 12, 25, 27, 27, 28, 28, 27. Note the repeating server_ids. The query gives me unique id but duplicate server_id.
Is there a way I can get results that would show me the last id for each server_id?
I've tried doing ORDER BY server_id but that gives me the same issue.
I tried running DISTINCT but that also does not work.
you'll have to use some aggregation functions.
Something like
select
server_id,
max(id),
avg(start_time),--for example
avg(end_time)--for example
from errors
where server_id in (3, 12, 24, 25, 26, 27, 28, 29, 30)
group by server_id
order by id desc
if you need tht start_time and end_time corresponding to the max id by server_id, you may do
select e.id, e.server_id, e.start_time, e.end_time
from errors e
join (select server_id, max(id) maxid
from errors
group by server_id) t
on t.maxid = e.id and e.server_id = t.server_id
where e.server_id in (3, 12, 24, 25, 26, 27, 28, 29, 30)
order by e.id DESC
The issue you have is that you need only one record from each server with the max ID.. and relative information. You need to limit the results to just that max ID... Here's one way...
SELECT id, server_id, start_time, end_time
FROM errors
WHERE server_id in (3, 12, 24, 25, 26, 27, 28, 29, 30)
and ID = (Select max(ID) from errors E2 where E2.server_ID=errors.server_ID)
ORDER BY id DESC
LIMIT 9
How can I avoid eliminating the users with zero meetings? I'm aware there are similar questions, but this code is quite a bit more complex.
SELECT user.userID, user.contactName, user.email, COUNT( * ) AS meetingsCount
FROM user
LEFT OUTER JOIN meeting ON user.userID = meeting.userID
WHERE user.userID NOT
IN ( 1, 2, 3, 4, 5, 59, 62, 63, 64, 66, 69, 71, 72, 73, 78, 107 )
AND SUBSTRING( meeting.meetingCode, 5, 2 )
BETWEEN 12
AND 22
AND SUBSTRING( meeting.meetingCode, 7, 2 )
BETWEEN 01
AND 12
AND SUBSTRING( meeting.meetingCode, 9, 2 )
BETWEEN 01
AND 31
GROUP BY user.userID, contactName, email
ORDER BY meetingsCount DESC
You need to put the logic for the meeting code table in your join. Otherwise users matching the records you are filtering out from the meeting table will be filtered out of your results. Making your JOIN essentially an INNER join. I think you also should put single quotes around the values in your BETWEEN clauses.
SELECT user.userID, user.contactName, user.email, COUNT( meeting.userID ) AS meetingsCount
FROM user
LEFT OUTER JOIN meeting ON user.userID = meeting.userID
AND SUBSTRING( meeting.meetingCode, 5, 2 ) BETWEEN '12' AND '22'
AND SUBSTRING( meeting.meetingCode, 7, 2 ) BETWEEN '01' AND '12'
AND SUBSTRING( meeting.meetingCode, 9, 2 ) BETWEEN '01' AND '31'
WHERE user.userID NOT IN ( 1, 2, 3, 4, 5, 59, 62, 63, 64, 66, 69, 71, 72, 73, 78, 107 )
GROUP BY user.userID, contactName, email
ORDER BY meetingsCount DESC