I am using MySQL database to query one of the huge tables and it's taking too long 5-6 seconds and sometimes 10-20 seconds.
SELECT theminutes.theminute, IFNULL(summary.`costweek`,0) total_cost,IFNULL(summary.`timeweek`,0) total_time
FROM (
SELECT DATE(NOW())-INTERVAL seq.seq DAY theminute
FROM (
SELECT 0 AS seq
UNION ALL SELECT 1 UNION ALL SELECT 2
UNION ALL SELECT 3 UNION ALL SELECT 4
UNION ALL SELECT 5 UNION ALL SELECT 6
) seq
) theminutes
LEFT JOIN (
SELECT DATE(cdr.starttime) theminute,
COALESCE(SUM(`sessionbill`),0) `costweek`,
COALESCE(SUM(`sessiontime`),0) `timeweek`
FROM cc_call cdr
WHERE (DATE(cdr.starttime) >= DATE(NOW()) - INTERVAL 6 DAY) AND cdr.accountcode = {$user_id} AND cdr.direction = 'outbound'
GROUP BY DATE(cdr.starttime)
) summary USING (theminute)
ORDER BY theminutes.theminute
When I ran "EXPLAIN" on this query I got this:
Live table contains approx 10 million rows. Please guide me if I can improve my query or if I need to optimize my table.
Edit:
Output of
SHOW CREATE TABLE cc_calls
Related
Here's my "customers" table:
To get number of enquiries per for a particular month and year, I'm using following query:
SELECT YEAR(customer_date) AS Year, MONTH(customer_date) AS Month, COUNT(customer_id) AS Count FROM customers WHERE customer_product = 6 GROUP BY YEAR(customer_date), MONTH(customer_date)
I get following result:
You can see that as there is no enquery in the April month, so no row fetched for month number 4. But I want 0 value in Count column if there is no record found in that particular month and year.
This is what I want:
One option uses a calendar table to represent all months and years, even those which do not appear in your data set:
SELECT
t1.year,
t2.month,
COUNT(c.customer_id) AS Count
FROM
(
SELECT 2017 AS year UNION ALL
SELECT 2018
) t1
CROSS JOIN
(
SELECT 1 AS month UNION ALL
SELECT 2 UNION ALL
SELECT 3 UNION ALL
SELECT 4 UNION ALL
SELECT 5 UNION ALL
SELECT 6 UNION ALL
SELECT 7 UNION ALL
SELECT 8 UNION ALL
SELECT 9 UNION ALL
SELECT 10 UNION ALL
SELECT 11 UNION ALL
SELECT 12
) t2
LEFT JOIN customers c
ON t1.year = YEAR(c.customer_date) AND
t2.month = MONTH(c.customer_date)
WHERE
c.customer_product = 6
GROUP BY
t1.year,
t2.month
ORDER BY
t1.year,
t2.month;
Note: The above query can probably be made faster by actually creating dedicated calendar tables in your MySQL schema.
The following index on the customers table might help:
CREATE INDEX idx ON customers(customer_product, customer_id);
This might make the join between the calendar tables and customers faster, assuming that the customer_product = 6 condition is restrictive.
So I have a table, PowerUse, in MySQL. It has two important columns, ID (long int) and timeRecorded (timeStamp). There are often long time gaps between entries. My objective is to query the table to get a result tells me how many records there are for each minute over 20 minute - including a 0 response for minutes without records.
Thanks to MySQL: Select All Dates In a Range Even If No Records Present I've got to the query below, but I can't get the dratted thing to work. Could someone point to the obvious error that I'm making, so I can stop banging my head against a brick wall? I know it's in the JOIN, I just can't see it. Thanks!
select count(PowerUse.id) + 0 as counter, timestamp((PowerUse.timeRecorded div 100)*100) as time
from PowerUse right join
(
select date_add(NOW(), INTERVAL -n2.num*10-n1.num MINUTE) as dateb from
(select 0 as num
union all select 1
union all select 2
union all select 3
union all select 4
union all select 5
union all select 6
union all select 7
union all select 8
union all select 9) n1,
(select 0 as num
union all select 1
union all select 2
) n2
as ) datea
on (timestamp(PowerUse.timeRecorded div 100)*100) = dateb.datea
group by timeRecorded div 100 ;
I guess your best option is to create another table with:
start_date
end_date
and fill it with your ranges....
then you can join:
select start_date, count(PowerUse.id) + 0 as counter, timestamp((PowerUse.timeRecorded div 100)*100) as time
from PowerUse
right join timeframes on ( timestamp((PowerUse.timeRecorded div 100)*100) >= start_date and timestamp((PowerUse.timeRecorded div 100)*100) < end_date )
group by start_date
Something like that
I've a table where there's two column:
MARKS
CREAT_TS
I want to daily average marks for between two date range (e.g. startDate & endDate)
I've made the following query:
select SUM(MARKS)/ COUNT(date(CREAT_TS)) AS DAILY_AVG_MARKS,
date(CREAT_TS) AS DATE
from TABLENAME
group by date(CREAT_TS)
With this query I can get the daily average only if there's a row in the database for the date. But my requirement is that even if there's no row, I want to show 0 for that date.
I mean I want the query to return X rows if there are X days between (startDate, endDate)
Can anyone help me. :(
You need to create a set of integers that you can add to the dates. The following will give you an idea:
select thedate, avg(Marks) as DAILY_AVG_MARKS
from (select startdate+ interval num day as thedate
from (select d1.d + 10 * d2.d + 100*d3.d as num
from (select 0 as d union select 1 union select 2 union select 3 union select 4 union
select 5 union select 6 union select 7 union select 8 union select 9
) d1 cross join
(select 0 as d union select 1 union select 2 union select 3 union select 4 union
select 5 union select 6 union select 7 union select 8 union select 9
) d2 cross join
(select 0 as d union select 1 union select 2 union select 3 union select 4 union
select 5 union select 6 union select 7 union select 8 union select 9
) d3
) n cross join
(select XXX as startdate, YYY as enddate) const
where startdate + num <= enddate
) left outer join
tablename t
on date(CREAT_TS) = thedate
group by thedate
All the complication is in creating a set of sequential dates for the report. If you have a numbers table or a calendar table, then the SQL looks much simpler.
How does this work? The first big subquery has two parts. The first just generates the numbers from 0 to 999 by cross joining the digits 0-9 and doing some arithmetic. The second joins this to the two dates, startdate and enddate -- you need to put the correct values in for XXX and YYY. With this table, you have all the dates between the two values. If you need more than 999 days, just add in another cross join.
This is the left joined to your data table. The result is that all dates appear for the group by.
In terms of reporting, there are advantages and disadvantages to doing this in the presentation layer. Basically, the advantage to doing it in SQL is that the report layer is simpler. The advantage to doing it in the reporting layer is that the SQL is simpler. It is hard for an outsider to make that judgement.
My suggestion would be to create a numbers table that you can just use in reports like this. Then the query will look simpler and you won't have to change the reporting layer.
Been trying to sort this one out for a while. I'd really appreciate any help.
I've got this table where I'm getting 2 columns with date and int values respectively. The problem is that mysql skips the date values wherever the int value is null.
Here the sql statement
SELECT DATE_FORMAT(sales_date_sold, '%b \'%y')
AS sale_date, sales_amount_sold
AS sale_amt
FROM yearly_sales
WHERE sales_date_sold BETWEEN DATE_SUB(SYSDATE(), INTERVAL 2 YEAR) AND SYSDATE()
GROUP BY YEAR(sales_date_sold), MONTH(sales_date_sold)
ORDER BY YEAR(sales_date_sold), MONTH(sales_date_sold) ASC;
There aren't any values for feb 2011 so that month gets skipped, along with a few others. Coalesce and if_null don't work too.
You need a row source that provides values for all of the months in the dimension, and then left join your yearly_sales table to that.
You are doing a GROUP BY, you most likely want an aggregate on your measure (sales_amount_sold), or you don't want a GROUP BY. (The query in your question is going to return a value from sales_amount_sold for only one row in a given month. That may be what you want, but its a very odd resultset to return.)
One approach is to have a "calendar_month" table that contains DATE values all of the months you want returned. (There are other ways to generate this, existing answers to questions elsewhere on stackoverflow)
SELECT m.month AS sale_date
, IFNULL(SUM(s.sales_amount_sold),0) AS sale_amt
FROM calendar_months m
LEFT
JOIN yearly_sales s
ON s.sales_date_sold >= m.month
AND s.sales_date_sold < DATE_ADD(m.month,INTERVAL 1 MONTH)
WHERE m.month BETWEEN DATE_SUB(SYSDATE(), INTERVAL 2 YEAR) AND SYSDATE()
GROUP BY m.month
ORDER BY m.month
This query returns a slightly different result, you are only going to get rows in groups of "whole months", rather than including partial months, as in your original query, because the WHERE clause on sale_date references two years before the current date and time, rather than the "first of the month" two years before.
A calendar_months table is not necessarily required; this could be replaced with a query that returns the row source. In that case, the predicate on the month value could be moved from the outer query into the subquery.
Addendum: if you use a calendar_month table as a rowsource, you'll need to populate it with every possible "month" value you want to return.
CREATE TABLE calendar_month
(`month` DATE NOT NULL PRIMARY KEY COMMENT 'one row for first day of each month');
INSERT INTO calendar_month(`month`) VALUES ('2011-01-01'),('2011-02-01'),('2011-03-01')
As an alternative, you can specify a dynamically generated rowsource, as an inline view, rather than a table reference. (You could use a similar query to quickly populate a calendar_months table.)
You can wrap this query in parenthesis, and paste it between FROM and calendar_months in the previous query I provided.
SELECT DATE_ADD('1990-01-01',INTERVAL 1000*thousands.digit + 100*hundreds.digit + 10*tens.digit + ones.digit MONTH) AS `month`
FROM ( SELECT 0 AS digit UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9 ) ones
JOIN ( SELECT 0 AS digit UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9 ) tens
JOIN ( SELECT 0 AS digit UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9 ) hundreds
JOIN ( SELECT 0 AS digit UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9 ) thousands
The problem is not that the value is NULL, the problem is that you are selecting data off your Database. If you don't have data for a specific month, MySQL has no way of selecting data that is not there.
The only way to solve this completely in MySQL is already answered in a very similar question
I have had this problem before with timestamps. The solution I used was to create a reference table with all of your months. This could be a table with just the numbers 1-12 (12 rows) or you could go one step further and put the month names. Then you can left join your yearly_sales table to the 1_through_12 table to get every month.
Why don't you just use 0 instead of NULL?
My problem:
I got a MySQL database that stores a great amount of meteorological data in chronological order (New data are inserted every 10 min). Unfortunately there have been several blackouts and hence certain rows are missing. I recently managed to obtain certain backup-files from the weather station and now I want to use these to fill in the missing data.
The DB ist structures like this (example):
date* the data
2/10/2009 10:00 ...
2/10/2009 10:10 ...
( Missing data!)
2/10/2009 10:40 ...
2/10/2009 10:50 ...
2/10/2009 11:00 ...
...
*=datatime-type, primary key
My idea:
Since backup and database are located on different computers and traffic is quite slow, I thought of creating a MySQL-query that, when run, will return a list of all missing dates in a specified range of time. I could then extract these dates from the backup and insert them to the database.
The question:
How to write such a query? I don't have the permission to create any auxilary table. Is it possible to formulate a "virtual table" of all required dates in the specified interval and then use it in a JOIN? Or are there entirely different propositions for solving my problem?
Edit:
Yes, the timestamps are consistently in the form shown above (always 10 minutes), except that some are just missing.
Okay, what about the temporary tables? Is there an elegant way of populating them with the time-range automatically? What if two scripts try to run simultaneously, does this cause problems with the table?
select t1.ts as hival, t2.ts as loval
from metdata t1, metdata t2
where t2.ts = (select max(ts) from metdata t3
where t3.ts < t1.ts)
and not timediff(t1.ts, t2.ts) = '00:10:00'
This query will return couplets you can use to select the missing data. The missing data will have a timestamp between hival and loval for each couplet returned by the query.
EDIT - thx for checking, Craig
EDIT2 :
getting the missing timestamps - this SQL gets a bit harder to read, so I'll break it up a bit. First, we need a way to calculate a series of timestamp values between a given low value and a high value in 10 minute intervals. A way of doing this when you can't create tables is based on the following sql, which creates as a resultset all of the digits from 0 to 9.
select d1.* from
(select 1 as digit
union select 2
union select 3
union select 4
union select 5
union select 6
union select 7
union select 8
union select 9
union select 0
) as d1
...now by combining this table with a copy of itself a couple of times means we can dynamically generate a list of a specified length
select curdate() +
INTERVAL (d1.digit * 100 + d2.digit * 10 + d3.digit) * 10 MINUTE
as date
from (select 1 as digit
union select 2
union select 3
union select 4
union select 5
union select 6
union select 7
union select 8
union select 9
union select 0
) as d1
join
(select 1 as digit
union select 2
union select 3
union select 4
union select 5
union select 6
union select 7
union select 8
union select 9
union select 0
) as d2
join
(select 1 as digit
union select 2
union select 3
union select 4
union select 5
union select 6
union select 7
union select 8
union select 9
union select 0
) as d3
where (d1.digit * 100 + d2.digit * 10 + d3.digit) between 1 and 42
order by 1
... now this piece of sql is getting close to what we need. It has 2 input variables:
a starting timestamp (I used
curdate() in the example); and a
number of iterations - the where
clause specifies 42 iterations in
the example, maximum with 3 x digit tables is 1000 intervals
... which means we can use the original sql to drive the example from above to generate a series of timestamps for each hival lowval pair. Bear with me, this sql is a bit long now...
select daterange.loval + INTERVAL (d1.digit * 100 + d2.digit * 10 + d3.digit) * 10 MINUTE as date
from
(select t1.ts as hival, t2.ts as loval
from metdata t1, metdata t2
where t2.ts = (select max(ts) from metdata t3
where t3.ts < t1.ts)
and not timediff(t1.ts, t2.ts) = '00:10:00'
) as daterange
join
(select 1 as digit
union select 2
union select 3
union select 4
union select 5
union select 6
union select 7
union select 8
union select 9
union select 0
) as d1
join
(select 1 as digit
union select 2
union select 3
union select 4
union select 5
union select 6
union select 7
union select 8
union select 9
union select 0
) as d2
join
(select 1 as digit
union select 2
union select 3
union select 4
union select 5
union select 6
union select 7
union select 8
union select 9
union select 0
) as d3
where (d1.digit * 100 + d2.digit * 10 + d3.digit) between 1 and
round((time_to_sec(timediff(hival, loval))-600) /600)
order by 1
...now there's a bit of epic sql
NOTE : using the digits table 3 times gives a maximum gap it will cover of a bit over 6 days
If you can create a temporary table, you can solve the problem with a JOIN
CREATE TEMPORARY TABLE DateRange
(theDate DATE);
Populate the table with all 10 minute intervals between your dates, then use the following
SELECT theDate
FROM DateRange dr
LEFT JOIN Meteorological mm on mm.date = dr.theDate
WHERE mm.date IS NULL
The result will be all of the date/times that do not have entries in your weather table.
If you need to quickly find days with missing data, you can use
select Date(mm.Date),144-count(*) as TotMissing
from Meteorological mm
group by Date(mm.Date)
having count(*) < 144
This is assume 24 hour a day, 6 entries per hour (hence 144 rows). – Sparky 0 secs ago
Create a temporary table (JOIN). Or take all the dates and query them locally, where you should have free reign (loop/hash).
For the JOIN, your generated reference of all dates is your base table and your data is your joined table. Seek out pairs where the joined data does not exist and select the generated date.
As a quick solotion using Sql Server, check for dates that do not have a follower of date+interval. I think MySql does have some sort of dateadd function, but you can try something like this. This will show you the ranges where you have missing data.
DECLARE #TABLE TABLE(
DateValue DATETIME
)
INSERT INTO #TABLE SELECT '10 Feb 2009 10:00:00'
INSERT INTO #TABLE SELECT '10 Feb 2009 10:10:00'
INSERT INTO #TABLE SELECT '10 Feb 2009 10:40:00'
INSERT INTO #TABLE SELECT '10 Feb 2009 10:50:00'
INSERT INTO #TABLE SELECT '10 Feb 2009 11:00:00'
SELECT *
FROM #TABLE currentVal
WHERE ((SELECT * FROM #TABLE nextVal WHERE DATEADD(mi,10,currentVal.DateValue) = nextVal.DateValue) IS NULL AND currentVal.DateValue != (SELECT MAX(DateValue) FROM #TABLE))
OR ((SELECT * FROM #TABLE prevVal WHERE DATEADD(mi,-10,currentVal.DateValue) = prevVal.DateValue) IS NULL AND currentVal.DateValue != (SELECT MIN(DateValue) FROM #TABLE))
Note: uses MSSQL syntax. I think MySQL uses DATE_ADD(T1.date, INTERVAL 10 MINUTE) instead of DATEADD, but I haven't tested this.
You can get the missing timestamps with two self-joins:
SELECT T1.[date] AS DateFrom, MIN(T3.[date]) AS DateTo
FROM [test].[dbo].[WeatherData] T1
LEFT JOIN [test].[dbo].[WeatherData] T2 ON DATEADD(MINUTE, 10, T1.date) = T2.date
LEFT JOIN [test].[dbo].[WeatherData] T3 ON T3.date > T1.Date
WHERE T2.[value] IS NULL
GROUP BY T1.[date]
If you have a lot of data, You might want to try restricting the range to one month at a time to avoid heavy load on your server, as this operation could be quite intensive.
The results will be something like this:
DateFrom DateTo
2009-10-02 10:10:00.000 2009-10-02 10:40:00.000
2009-10-02 11:00:00.000 NULL
The last row represents all data from the last timestamp into the future.
You can then use another join to get the rows from the other database that have a timestamp in between any of these intervals.
This solution uses sub-queries, and there is no need for any explicit temporary tables. I've assumed your backup data is in another database on the other machine; if not you'd only need to do up to step 2 for the result-set you need, and write your program to update the main database accordingly.
The idea is to start out by producing a 'compact' result-set summarising the gap-list. I.e. the following data:
MeasureDate
2009-12-06 13:00:00
2009-12-06 13:10:00
--missing data
2009-12-06 13:30:00
--missing data
2009-12-06 14:10:00
2009-12-06 14:20:00
2009-12-06 14:30:00
--missing data
2009-12-06 15:00:00
Would be transformed into the following where actual gaps are strictly between (i.e. exclusive of) the endpoints:
GapStart GapEnd
2009-12-06 13:10:00 2009-12-06 13:30:00
2009-12-06 13:30:00 2009-12-06 14:10:00
2009-12-06 14:30:00 2009-12-06 15:00:00
2009-12-06 15:00:00 NULL
The solution query is built up as follows:
Obtain all MeasureDates that don't have an entry 10 minutes later as this will be the start of a gap. NOTE: The last entry will be included even though not strictly a gap; but this won't have any adverse effects.
Augment the above by adding the end of the gap using the first MeasureDate after the start of the gap.
NOTE: The gap-list is compact, and unless you have an exceptionally high prevalence of fragmented gaps, it should not consume much bandwidth in passing that result-set to the backup machine.
Use an INNER JOIN with inequalities to identify any missing data that may be available in the backup. (Run tests and checks to verify the integrity of your backup data.)
Assuming your backup data is sound, and won't produce anomalous unfounded spikes in your measurements, INSERT the data in your main database.
The following query should be tested (preferably adjusted to run on the backup server for performance reasons).
/* TiC Copyright
This query is writtend (sic) by me, and cannot be used without
expressed (sic) written permission. (lol) */
/*Step 3*/
SELECT gap.GapStart, gap.GapEnd,
rem.MeasureDate, rem.Col1, ...
FROM (
/*Step 2*/
SELECT gs.GapStart, (
SELECT MIN(wd.MeasureDate)
FROM WeatherData wd
WHERE wd.MeasureDate > gs.GapStart
) AS GapEnd
FROM (
/*Step 1*/
SELECT wd.MeasureDate AS GapStart
FROM WeatherData wd
WHERE NOT EXISTS (
SELECT *
FROM WeatherData nxt
WHERE nxt.MeasureDate = DATEADD(mi, 10, wd.MeasureDate)
)
) gs
) gap
INNER JOIN RemoteWeatherData rem ON
rem.MeasureDate > gap.GapStart
AND rem.MeasureDate < gap.GapEnd
The insert...
INSERT INTO WeatherData (MeasureDate, Col1, ...)
SELECT /*gap.GapStart, gap.GapEnd,*/
rem.MeasureDate, rem.Col1, ...
...
Do a self join and then calculate the max values that are smaller and have a difference larger than your interval.
In Oracle I'd do it like this (with ts being the timestamp column):
Select t1.ts, max(t2.ts)
FROM atable t1 join atable t2 on t1.ts > t2.ts
GROUP BY t1.ts
HAVING (t1.ts - max(t2.ts))*24*60 > 10
There will be better ways to handle the difference calculation in mySql, but I hope the idea comes across.
This query will give you the timestamps directly after and before outage, and you can build from there.