Related
I have a stored procedure. A little problematic in terms of performance. I want to improve the performance of the stored procedure, but I could not figure out what to do. There are approximately 3 million records in my database. When I run this query one by one, it's good in performance. But when 150 people run this stored procedure at the same time, there are spikes in the CPU.
As an example, I created my procedure and table structures.
My Stored Procedure:
BEGIN
SELECT ss.car_route from person o
inner join car_time ss on ss.inst_id =o.inst_id
and ss.start_time<=DATE_FORMAT(CURTIME(),'%H:%i') AND ss.finish_time>= date_format(curtime() ,'%H:%i') AND ss.car_id=carid
and ss.days like concat('%',(select WEEKDAY(now())+1),'%')
where (o.car_id=carid or o.back_car_id=carid ) LIMIT 1 into #route_;
select sf.stop_service from car_comp sf
inner join cars s on s.inst_id = sf.id and s.id=carid and s.active=1 limit 1
into #stop_ser;
if #route_ = 1 and #stop_ser=0 THEN
select DISTINCT ss.start_time,ss.finish_time ,o.id,o.name,r.photo, oh.state ,oh.datee,ss.car_route,
ifnull(bh.id,0) AS called,
ifnull(mh.excuse_id,0) AS excuse_id,
ifnull(o.latitude_1,0) AS latitude_1,
ifnull(o.longitude_1,0) AS longitude_1,
ifnull(o.latitude_2,0) AS latitude_2,
ifnull(o.longitude_2,0) AS longitude_2,
case when (ifnull(o.call_notify,0)=1 or ifnull(o.mes_notify,0)=1) then 1 else 0 end AS call_notify ,
ifnull(o.rownumber,0) AS rownumber,
ifnull(o.number_1,0) AS number_1,
ifnull(o.number_2,0) AS number_2,
ifnull(o.brownumber,0) AS brownumber,
ifnull(ROUND(o.notify_meter_1/2),0) AS notify_meter_1,
ifnull(ROUND(o.notify_meter_2/2),0) AS notify_meter_2
from person o
inner join car_time ss on ss.inst_id =o.inst_id and o.car_id=ss.car_id
and ss.start_time<=DATE_FORMAT(CURTIME(),'%H:%i') AND ss.finish_time>= date_format(curtime() ,'%H:%i')
and ss.days like concat('%',(select WEEKDAY(now())+1),'%')
LEFT JOIN notify_records bh ON bh.table_id=o.id AND bh.car_route=#route_
and bh.table_name='person' AND bh.notify=4 AND bh.car_id=o.car_id and bh.date_ >= CURDATE() and bh.date_ < CURDATE() + INTERVAL 1 DAY
left join person_records oh on oh.person_id=o.id
and oh.car_id=o.car_id
and date_format(oh.datee,'%H:%i') >=ss.start_time
and date_format(oh.datee,'%H:%i') <=ss.finish_time
AND oh.car_route= #route_
and
oh.id in(select max(id) from person_records
where date_time >= CURDATE() and date_time < CURDATE() + INTERVAL 1 DAY and car_id = carid and car_id = carid
GROUP by person_id
)
left join inst ok on o.inst_id = ok.id and o.car_id=carid
left join excuse_records mh on mh.person_id=o.id and mh.date_time >= CURDATE() and mh.date_time < CURDATE() + INTERVAL 1 DAY and (mh.car_route=ss.car_route)
left join photo_ r on r.table_id = o.id and r.table_name = 'person'
where
(ss.car_route=o.cars_route_ or o.cars_route_=3) and
o.car_id = carid and o.active=1
AND o.work_time=ss.work_time;
elseif #route_ = 2 and #stop_ser=0 then
select DISTINCT ss.start_time,ss.finish_time ,o.id,o.name,r.photo, oh.state ,oh.datee,ss.car_route,
ifnull(bh.id,0) AS called,
ifnull(mh.excuse_id,0) AS excuse_id,
ifnull(o.latitude_1,0) AS latitude_1,
ifnull(o.longitude_1,0) AS longitude_1,
ifnull(o.latitude_2,0) AS latitude_2,
ifnull(o.longitude_2,0) AS longitude_2,
case when (ifnull(o.call_notify,0)=1 or ifnull(o.mes_notify,0)=1) then 1 else 0 end AS call_notify ,
ifnull(o.rownumber,0) AS rownumber,
ifnull(o.number_1,0) AS number_1,
ifnull(o.number_2,0) AS number_2,
ifnull(o.brownumber,0) AS brownumber,
ifnull(ROUND(o.notify_meter_1/2),0) AS notify_meter_1,
ifnull(ROUND(o.notify_meter_2/2),0) AS notify_meter_2
from person o
inner join car_time ss on ss.inst_id =o.inst_id and o.back_car_id=ss.car_id
and ss.start_time<=DATE_FORMAT(CURTIME(),'%H:%i') AND ss.finish_time>= date_format(curtime() ,'%H:%i')
and ss.days like concat('%',(select WEEKDAY(now())+1),'%')
LEFT JOIN notify_records bh ON bh.table_id=o.id AND bh.car_route=#route_
and bh.table_name='person' AND bh.notify=4 AND bh.car_id=o.back_car_id and bh.date_ >= CURDATE() and bh.date_ < CURDATE() + INTERVAL 1 DAY
left join person_records oh on oh.person_id=o.id
and oh.car_id=o.back_car_id and oh.car_route=2
and date_format(oh.datee,'%H:%i') >=ss.start_time
and date_format(oh.datee,'%H:%i') <=ss.finish_time
AND oh.car_route= #route_
and
oh.id in (select max(id) from person_records
where date_time >= CURDATE() and date_time < CURDATE() + INTERVAL 1 DAY and car_id = carid
GROUP by person_id
)
left join inst ok on o.inst_id = ok.id and o.car_id=carid
left join excuse_records mh on mh.person_id=o.id and mh.date_time >= CURDATE() and mh.date_time < CURDATE() + INTERVAL 1 DAY and (mh.car_route=ss.car_route)
left join photo_ r on r.table_id = o.id and r.table_name = 'person'
where
(ss.car_route=o.cars_route_ or o.cars_route_=3) and
o.back_car_id = carid and o.active=1
AND o.work_time=ss.work_time;
END IF;
end
I have a database example here.
I made my.cnf improvement but still have difficulties with performance. What is wrong with this query? What can I change?
Thank you from now.
Edit:
Server version: 10.1.41-MariaDB - MariaDB Server
I have indexes. I forgot to add indexes while creating test data.
What the heck is this?
ss.days like concat('%',(select WEEKDAY(now())+1),'%')
It can at least be sped up by changing to
ss.days like concat('%',WEEKDAY(now()),'%')
And, won't that lead to checking against 2, 21, 20, 12, ... if the WEEKDAY is "2"?
These might be useful for ss:
(car_id, inst_id, start_time)
(inst_id, car_id, finish_time)
LIMIT 1 without ORDER BY leads to some random row being returned? Is the LIMIT redundant? Or is an ORDER BY needed?
Suggest you get some timings -- It is not obvious which of the SELECTs is chewing up the most CPU.
If the PRIMARY KEY of cars is id, then why test for inst_id and active? Yikes! You don't seem to have a PK for cars! Please verify that every table has a PK.
Redundant:
and car_id = carid
and car_id = carid
Why twice? And what tables are those columns in? Please qualify columns so we can understand what is going on.
When #stop_ser=0, the procedure does nothing? In which case, perform that test first, so you can avoid computing #route.
Change start_time to datatype TIME; then you can get rid of DATE_FORMAT in
and ss.start_time<=DATE_FORMAT(CURTIME(),'%H:%i')
AND ss.finish_time>= date_format(curtime() ,'%H:%i')
Also, beware of the inequality tests, it may lead to some edge cases you did not want.
Don't use (m,n) on FLOAT (eg, float(11,7)); it does rounding that is unnecessary. Also, you can't get 7 decimal places for lat/lng except very near the equator and longitude=0. More on precision: http://mysql.rjweb.org/doc.php/latlng#representation_choices
After you have cleaned up those and provided the requested info, I will take another look.
I am just learning MySQL. I need to find out rank of deals by day. Here I am adding the corresponding MYSQL query for my requirement that currently ranks all sales highest to lowest by day. Please help me to add a column that gives the rank to the deal highest to lowest and resetting the next day.
Here is my current working query,..
single day with title, price
SELECT
DATE(order_items.created_at) AS the_day,
order_items.deal_id,
SUM(order_items.item_total) AS daily_total,
SUM(order_items.qty) AS units_sold,
deals.price,
deals.title
FROM
order_items JOIN deals ON order_items.deal_id = deals.id
WHERE
order_items.created_at >= '2016-01-01 00:00:00' AND order_items.created_at < '2016-01-30 00:00:00'
AND
order_items.status=1
AND
order_items.paid=1
GROUP BY
order_items.deal_id
ORDER BY
the_day,
daily_total DESC;
The easiest way to do is that:
Use your existing SQL - I guess you need to update your SQL, make sure any non-aggregated columns in select should be in group by as well
Use PHP to loop (1-5), it works for multiple days
If you are happy to get top 5 for a single day, you can add limit 5 at end of your SQL
If you need top 5 results for each day in multiple days in one SQL, you need to update SQL to be more complicated. And here is a hint to use row id see example:
select increment counter in mysql
OK - Since you updated your question to return top 1 result per day, this is easier:
Step 1: get each day, each deal, report:
SELECT deal_id, date(created_at) ymd, sum(item_total) daily_total, sum(qty) units_sold
FROM order_items
WHERE substr(created_at,1,7) = '2016-01'
AND status = 1
AND paid = 1
GROUP BY 1,2
Step 2: Find the best deal of each day from step 1:
SELECT aa.ymd, max(aa.daily_total) max_total
FROM (
SELECT deal_id, date(created_at) ymd, sum(item_total) daily_total, sum(qty) units_sold
FROM order_items
WHERE substr(created_at,1,7) = '2016-01'
AND status = 1
AND paid = 1
GROUP BY 1,2
) as aa
GROUP BY 1;
Please note that max(item_total) not necessary same row as max(unit_sold), so you need to choose one, and cannot run togather
Step 3: Join step 2 with step 1 and deals to find out the rest of information:
SELECT aa.*, deals.price, deal.title
FROM (
SELECT aa.ymd, max(aa.daily_total) max_total
FROM (
SELECT deal_id, date(created_at) ymd, sum(item_total) daily_total, sum(qty) units_sold
FROM order_items
WHERE substr(created_at,1,7) = '2016-01'
AND status = 1
AND paid = 1
GROUP BY 1,2
) as aa
GROUP BY 1
) as bb
JOIN (
SELECT deal_id, date(created_at) ymd, sum(item_total) daily_total, sum(qty) units_sold
FROM order_items
WHERE substr(created_at,1,7) = '2016-01'
AND status = 1
AND paid = 1
GROUP BY 1,2
) as aa ON bb.ymd = aa.ymd and bb.max_total = aa.daily_total
JOIN deals ON aa.deal_id = deals.id
ORDER BY aa.ymd, aa.max_total
This is an extension of the earlier question here
I have two tables, yesterday and today and in each table there are two columns
StockNumber, StockLevel
How do I find out
Only the StockNumbers that have differences in their StockLevel's
Return the affected StockNumber & StockLevel values
For example:
todays table :
StockNumber, StockLevel
ABC, 10
CBA,5
123,5
yesterdays table :
StockNumber, StockLevel
ABC, 9
CBA,5
123,10
To return the following wi
StockNumber, StockLevel
ABC, 10
123, 5
Noticing now StockNumber "ABC" is returning todays value 10 and StockNumber "123" is also returning todays value 5 and StockNumber "CBA" there was no change and is not returned.
Many thanks,
Matt
This will do it for stocks that existed both days;
SELECT t.StockNumber, t.StockLevel
FROM today t
JOIN yesterday y
ON t.StockNumber=y.StockNumber
AND t.StockLevel <> y.StockLevel
If you (as in your earlier question) also want the new stocks from today included;
SELECT t.StockNumber, t.StockLevel
FROM today t
LEFT JOIN yesterday y
ON t.StockNumber = y.StockNumber
WHERE y.StockNumber IS NULL
OR t.StockLevel <> y.StockLevel
An SQLfiddle to test both with.
You can join the tables on StockNumber and select those rows WHERE NOT A.StockLevel = B.StockLevel.
This is a bit tricky, because you might have different items in the two tables. I recommend a union all and group by for this:
select stockNumer,
max(case when which = 'yesterday' then stocklevel end) as yesterdaylevel,
max(case when which = 'today' then stocklevel end) as todaylevel
from ((select StockNumber, StockLevel, 'yesterday' as which
from yesterday
) union all
(select StockNumber, StockLevel, 'today' as which
from today
)
) t
group by stockNumber
having min(StockLevel) <> max(stockLevel) and count(*) = 2
select T.StockNumber, T.StockLevel
from Today as T
left join Yesterday as Y
on T.StockNumber = Y.StockNumber
WHERE T.StockLevel <> Y.StockLevel
play with the results here : http://sqlfiddle.com/#!2/367b0/3/0
The following query will return stocks that are either different (by StockLevel) from yesterday, or didn't exist yesterday (this is what LEFT and IS NULL are for):
SELECT Today.*
FROM Today LEFT JOIN Yesterday
ON Today.StockNumber = Yesterday.StockNumber
WHERE
Today.StockLevel <> Yesterday.StockLevel
OR Yesterday.StockLevel IS NULL
Please consider the following two tables:
Holidays
HolidayID (PK)
Destination
Length
MaximumNumber
...
Bookings
BookingID (PK)
HolidayID (FK)
Name
...
Customers can book holidays (e.g. go to Hawaii). But, suppose that a given holiday has a maximum number of places. e.g. there are only 75 holidays to Hawaii this year (ignoring other years).
So if some customer wants to book a holiday to Hawaii. I need to count the records in Bookings table, and if that number is greater than 75 I have to tell the customer it's too late.
This I can do using 2 MySQL queries (1 to get MaximumNumber for the holiday, 2 to get the current total from Bookings) and PHP (for example) to compare the count value with the maximum number of Hawaii holidays.
But I want to know if there is a way to do this purely in SQL (MySQL in this case)? i.e. count the number of bookings for Hawaii and compare against Hawaii's MaximumNumber value.
EDIT:
My method:
$query1 = "SELECT MaximumNumber FROM Holidays WHERE HolidayID=$hawaiiID";
$query2 = "SELECT COUNT(BookingID) FROM Bookings WHERE HolidayID=$hawaiiID";
So if the first query gives 75 and the second query gives 75 I can compare these values in PHP. But I wondered if there was a way to do this somehow in SQL alone.
Maybe I am missing something, but why not use a subquery to determine the total bookings for each holidayid:
select *
from holidays h
left join
(
select count(*) TotalBooked, HolidayId
from bookings
group by holidayId
) b
on h.holidayId = b.holidayId
WHERE h.HolidayID=$hawaiiID;
See SQL Fiddle with Demo.
Then you could use a CASE expression to compare the TotalBooked to the MaxNumber similar to this:
select h.destination,
case
when b.totalbooked = h.maxNumber
then 'Not Available'
else 'You can still book' end Availability
from holidays h
left join
(
select count(*) TotalBooked, HolidayId
from bookings
group by holidayId
) b
on h.holidayId = b.holidayId
WHERE h.HolidayID=$hawaiiID;
See SQL Fiddle with Demo.
You will notice that I used a LEFT JOIN which will return all rows from the Holidays table even if there are not matching rows in the Bookings table.
Something like this will work. You can fill in the details:
select case
when
(select count(*)
from Bookings
where holidayID = $hawaiiid)
<= MaximumNumber then 'available' else 'sold out' end status
from holidays
etc
You might try something like this:
select case when b.freq < h.MaximumNumber
then 'Available'
else 'Not Available'
end as MyResult
from Holidays h
left join (
select HolidayID
, count(*) as freq
from Bookings
where HolidayID=$hawaiiID
group by HolidayID
) b
on h.HolidayID=b.HolidayID
I have a table that looks like this:
studentID | subjectID | attendanceStatus | classDate | classTime | lecturerID |
12345678 1234 1 2012-06-05 15:30:00
87654321
12345678 1234 0 2012-06-08 02:30:00
I want a query that reports if a student has been absent for 3 or more consecutive classes. based on studentID and a specific subject between 2 specific dates as well. Each class can have a different time. The schema for that table is:
PK(`studentID`, `classDate`, `classTime`, `subjectID, `lecturerID`)
Attendance Status: 1 = Present, 0 = Absent
Edit: Worded question so that it is more accurate and really describes what was my intention.
I wasn't able to create an SQL query for this. So instead, I tried a PHP solution:
Select all rows from table, ordered by student, subject and date
Create a running counter for absents, initialized to 0
Iterate over each record:
If student and/or subject is different from previous row
Reset the counter to 0 (present) or 1 (absent)
Else, that is when student and subject are same
Set the counter to 0 (present) or plus 1 (absent)
I then realized that this logic can easily be implemented using MySQL variables, so:
SET #studentID = 0;
SET #subjectID = 0;
SET #absentRun = 0;
SELECT *,
CASE
WHEN (#studentID = studentID) AND (#subjectID = subjectID) THEN #absentRun := IF(attendanceStatus = 1, 0, #absentRun + 1)
WHEN (#studentID := studentID) AND (#subjectID := subjectID) THEN #absentRun := IF(attendanceStatus = 1, 0, 1)
END AS absentRun
FROM table4
ORDER BY studentID, subjectID, classDate
You can probably nest this query inside another query that selects records where absentRun >= 3.
SQL Fiddle
This query works for intended result:
SELECT DISTINCT first_day.studentID
FROM student_visits first_day
LEFT JOIN student_visits second_day
ON first_day.studentID = second_day.studentID
AND DATE(second_day.classDate) - INTERVAL 1 DAY = date(first_day.classDate)
LEFT JOIN student_visits third_day
ON first_day.studentID = third_day.studentID
AND DATE(third_day.classDate) - INTERVAL 2 DAY = date(first_day.classDate)
WHERE first_day.attendanceStatus = 0 AND second_day.attendanceStatus = 0 AND third_day.attendanceStatus = 0
It's joining table 'student_visits' (let's name your original table so) to itself step by step on consecutive 3 dates for each student and finally checks the absence on these days. Distinct makes sure that result willn't contain duplicate results for more than 3 consecutive days of absence.
This query doesn't consider absence on specific subject - just consectuive absence for each student for 3 or more days. To consider subject simply add .subjectID in each ON clause:
ON first_day.subjectID = second_day.subjectID
P.S.: not sure that it's the fastest way (at least it's not the only).
Unfortunately, mysql does not support windows functions. This would be much easier with row_number() or better yet cumulative sums (as supported in Oracle).
I will describe the solution. Imagine that you have two additional columns in your table:
ClassSeqNum -- a sequence starting at 1 and incrementing by 1 for each class date.
AbsentSeqNum -- a sequence starting a 1 each time a student misses a class and then increments by 1 on each subsequent absence.
The key observation is that the difference between these two values is constant for consecutive absences. Because you are using mysql, you might consider adding these columns to the table. They are big challenging to add in the query, which is why this answer is so long.
Given the key observation, the answer to your question is provided by the following query:
select studentid, subjectid, absenceid, count(*) as cnt
from (select a.*, (ClassSeqNum - AbsentSeqNum) as absenceid
from Attendance a
) a
group by studentid, subjectid, absenceid
having count(*) > 2
(Okay, this gives every sequence of absences for a student for each subject, but I think you can figure out how to whittle this down just to a list of students.)
How do you assign the sequence numbers? In mysql, you need to do a self join. So, the following adds the ClassSeqNum:
select a.StudentId, a.SubjectId, count(*) as ClassSeqNum
from Attendance a join
Attendance a1
on a.studentid = a1.studentid and a.SubjectId = a1.Subjectid and
a.ClassDate >= s1.classDate
group by a.StudentId, a.SubjectId
And the following adds the absence sequence number:
select a.StudentId, a.SubjectId, count(*) as AbsenceSeqNum
from Attendance a join
Attendance a1
on a.studentid = a1.studentid and a.SubjectId = a1.Subjectid and
a.ClassDate >= a1.classDate
where AttendanceStatus = 0
group by a.StudentId, a.SubjectId
So the final query looks like:
with cs as (
select a.StudentId, a.SubjectId, count(*) as ClassSeqNum
from Attendance a join
Attendance a1
on a.studentid = a1.studentid and a.SubjectId = a1.Subjectid and
a.ClassDate >= s1.classDate
group by a.StudentId, a.SubjectId
),
a as (
select a.StudentId, a.SubjectId, count(*) as AbsenceSeqNum
from Attendance a join
Attendance a1
on a.studentid = a1.studentid and a.SubjectId = a1.Subjectid and
a.ClassDate >= s1.classDate
where AttendanceStatus = 0
group by a.StudentId, a.SubjectId
)
select studentid, subjectid, absenceid, count(*) as cnt
from (select cs.studentid, cs.subjectid,
(cs.ClassSeqNum - a.AbsentSeqNum) as absenceid
from cs join
a
on cs.studentid = a.studentid and cs.subjectid = as.subjectid
) a
group by studentid, subjectid, absenceid
having count(*) > 2