I need help with some dates in mysql
I have this query that brings me the count of the messages of the corresponding dates of every message from the last 30 days.
SELECT DATE_FORMAT(time, '%m/%d/%Y') AS Dates, count(*) as count
FROM ma_messages
WHERE time BETWEEN NOW() - INTERVAL 30 DAY AND NOW()
AND usersid_send = 110
Group by Dates
ORDER BY Dates ASC
query image
But I need that also bring me the other dates of the last 30 days with a value in the count of 0, for example that in the query also bring me 05/26/2021 date and the dates before 05/25/2021 and after 05/28/2021.
I don't know if this is possible, but I will apreciate any help.
Thanks.
try this
SELECT
DATE_FORMAT(time, '%m/%d/%Y') AS Dates,
count(if(usersid_send='110',true,null)) as count
FROM ma_messages
WHERE time BETWEEN NOW() - INTERVAL 30 DAY AND NOW()
Group by Dates
ORDER BY Dates ASC
I have a table with dates .. with all the dates of the year – Alexis Murillo
SELECT d.`date`, COUNT(m.`time`) AS `count`
FROM dates_table d
LEFT JOIN ma_messages m ON d.`date` = DATE(m.`time`)
WHERE d.`date` BETWEEN NOW() - INTERVAL 30 DAY AND NOW()
AND m.usersid_send = 110
GROUP BY d.`date`
ORDER BY d.`date` ASC
Using your table "with all the dates of the year":
WITH
message_counts AS ( SELECT DATE_FORMAT("time", '%m/%d/%Y') AS "day",
COUNT(*) AS "count"
FROM ma_messages
WHERE "time" BETWEEN NOW() - INTERVAL 30 DAY AND NOW()
AND
usersid_send = 110
GROUP BY Dates)
SELECT all_dates."day", COALESCE(message_counts."count", 0)
FROM all_dates
LEFT JOIN message_counts
ON all_dates."day" = message_counts."day"
WHERE all_dates."day" BETWEEN BETWEEN NOW() - INTERVAL 30 DAY AND NOW()
ORDER BY all_dates."day" ASC
I currently have a MariaDB database that gets populated every day with different products (around 800) and also gets the price updates for these products.
I've created a view on top of the prices/products table that generates statistics such as the avg, mean and mode for the last 7, 15 and 30 days, and calculates the difference from today's price to the averages of 7, 15 and 30 days.
The problem is that whenever I run this view it takes almost 50 seconds to generate the data. I saw some comments about switching over to a calculated table, in which the calculations would be updated when new data is entered into the table, however I'm quite skeptical in doing that, as I'm inserting around 1000 price points at one specific time of the day that will impact all the calculations on the table. Is a calculated table something that updates only the rows that were updated, or it would recalculate everything? I'm worried about the overhead this might cause (memory is not an issue with the server).
I've pasted the products and prices tables and the view to DBFiddle, here: https://dbfiddle.uk/?rdbms=mariadb_10.2&fiddle=4cf594a85f950bed34f64d800601baa9
Calculations can be seen for product code 22141
Just to give an idea these are some of the calculations done by the view (available on the fiddle as well):
ROUND((((SELECT preconormal
FROM precos
WHERE codigowine = vinhos.codigowine
AND timestamp >= CURRENT_DATE - INTERVAL 9 HOUR) / (SELECT AVG(preconormal)
FROM precos
WHERE codigowine = vinhos.codigowine
AND timestamp >= CURRENT_DATE - INTERVAL 7 DAY) - 1) * 100), 2) as dif_7_dias,
ROUND((((SELECT preconormal
FROM precos
WHERE codigowine = vinhos.codigowine
AND timestamp >= CURRENT_DATE - INTERVAL 9 HOUR) / (SELECT AVG(preconormal)
FROM precos
WHERE codigowine = vinhos.codigowine
AND timestamp >= CURRENT_DATE - INTERVAL 15 DAY) - 1) * 100), 2) as dif_15_dias,
ROUND((((SELECT preconormal
FROM precos
WHERE codigowine = vinhos.codigowine
AND timestamp >= CURRENT_DATE - INTERVAL 9 HOUR) / (SELECT AVG(preconormal)
FROM precos
WHERE codigowine = vinhos.codigowine
AND timestamp >= CURRENT_DATE - INTERVAL 30 DAY) - 1) * 100), 2) as dif_30_dias
If switching to a calculated table, is there an optimal way to do this?
A "calculated table" isn't a MySQL / MariaDB feature. So I guess you mean another table derived from your raw data, that you use when you need those statistics.
You say the table is "populated every day...". Do you mean it's reloaded from scratch, or do you mean 800 more rows are added? By "every day" do you mean at a particular time of day, or ongoing throughout the day.
Do you always have to select all rows from your view, or can you sometimes do SELECT columns FROM view WHERE something = 'constant';' This matters because optimization techniques differ between the all-rows case and the few-rows case.
How can you handle this problem efficiently?
You could work to optimize the query used to define your view, making it faster. That is very likely a good approach.
MariaDB has a type of column known as a Persistent Computed Column. These are computed when rows are INSERTED or UPDATED. Then they are available for quick reference. But they have limitations; they cannot be defined with subqueries.
You could define an EVENT (a scheduled SQL job) to do the following.
Create a new, empty, "calculated" table with a name like tbl_new.
Use your (slow) view to insert the rows it needs.
Roll over your tables, so the new one replaces the current one and you keep a couple of older ones. This will give you a brief window where tbl doesn't exist.
DROP TABLE IF EXISTS tbl_old_2;
RENAME TABLE tbl_old TO tbl_old_2, tbl TO tbl_old, tbl_new TO tbl;
That's a whole boatload of correlated subqueries, crying out for appropriate indexing.
For a reasonable number of rows being returned by the query, the correlated subqueries can give reasonable performance. But if the outer query is returning thousands of rows, that will be thousands of executions of the subqueries.
I would tend to avoid running multiple SELECT against the same table, to get the last 7 days, the last 15 days, the last 30 days, and then repeating that to get AVG, repeating that to get MAX, and again to get MIN.
Instead, I would tend towards using conditional aggregation, to get all of the stats AVG, MAX, MIN, for all of the time periods 30 days, 15 days, and 7 days, in a single pass through the table.
... pause to note that views can be a problematic for performance; predicates from the outer query may not get pushed into the view query. We're not seeing what the whole view definition is doing, but I suspect we may be materializing a large set.
Consider a query like this:
SELECT ...
, ROUND( ( n.mal / a.avg_07_day - 1)*100 ,2) AS dif_7_dias
, ROUND( ( n.mal / a.avg_15_day - 1)*100 ,2) AS dif_15_dias
, ROUND( ( n.mal / a.avg_30_day - 1)*100 ,2) AS dif_30_dias
, ...
FROM vinhos
LEFT
JOIN ( SELECT h.codigowine
, AVG(IF( h.timestamp >= CURRENT_DATE + INTERVAL -30 DAY, h.preconormal, NULL)) AS avg_30_day
, MAX(IF( h.timestamp >= CURRENT_DATE + INTERVAL -30 DAY, h.preconormal, NULL)) AS max_30_day
, MIN(IF( h.timestamp >= CURRENT_DATE + INTERVAL -30 DAY, h.preconormal, NULL)) AS min_30_day
, AVG(IF( h.timestamp >= CURRENT_DATE + INTERVAL -15 DAY, h.preconormal, NULL)) AS avg_15_day
, MAX(IF( h.timestamp >= CURRENT_DATE + INTERVAL -15 DAY, h.preconormal, NULL)) AS max_15_day
, MIN(IF( h.timestamp >= CURRENT_DATE + INTERVAL -15 DAY, h.preconormal, NULL)) AS min_15_day
, AVG(IF( h.timestamp >= CURRENT_DATE + INTERVAL -7 DAY, h.preconormal, NULL)) AS avg_07_day
, MAX(IF( h.timestamp >= CURRENT_DATE + INTERVAL -7 DAY, h.preconormal, NULL)) AS max_07_day
, MIN(IF( h.timestamp >= CURRENT_DATE + INTERVAL -7 DAY, h.preconormal, NULL)) AS min_07_day
FROM precos h
GROUP
BY h.codigowine
HAVING h.codigowine IS NOT NULL
) a
ON a.codigowine = vinhos.codigowine
LEFT
JOIN ( SELECT s.codigowine
, MAX(s.precnormal) AS mal
, MIN(s.precnormal) AS mil
FROM precos s
WHERE s.timestamp >= CURRENT_DATE - INTERVAL 9 HOUR
GROUP
BY s.codigowine
HAVING s.codigowine IS NOT NULL
) n
ON n.codigowine = vinhos.codigowine
Consider the inline view query a.
Note that we can run that SELECT separately, and get a resultset returned, like we would return a result from a table. We expect this to do a single pass through the referenced table. There may be some predicates (conditions in the WHERE clause) that will filter our row, or enable us to make better use of an index. As currently written, the query could make use of an index with leading column of codigowine to avoid a (potentially expensive) "Using filesort" operation to satisfy the GROUP BY.
I'm a bit confused by the queries the - INTERVAL 9 HOUR. It looks to me like those subqueries could potentially return more than one row. There's no LIMIT clause (and no ORDER BY)... but it looks like we are expecting a single value (scalar), given the division operation.
Without an understanding of what we're trying to achieve there, not knowing the specification, I've wrapped my confusion and put that into another inline view n... not that this is what we want to do, but just to illustrate (again) an inline view returning a resultset. Whatever value(s) we're trying to get from the - INTERVAL 9 HOUR subquery, I think we can return those as a set as well.
With all that said, we can now get around to answering the question that was asked: adding a "calculated table".
If we don't require up to the second results, but can work with cached statistics, I would be looking at materializing the resultset from inline view a into a table, and then re-writing the query above to replace the inline view a with a reference to the cache table.
CREATE TABLE calc_stats_n_days
( codigowine <datatype> PRIMARY KEY
, avg_30_day DOUBLE
, max_30_day DOUBLE
, min_30_day DOUBLE
, avg_15_day DOUBLE
, ...
For the initial population...
INSERT INTO calc_stats_n_days
( codigowine, avg_30_day, maxg_30_day, min_30_day, avg_15_day, ... )
SELECT h.codigowine
, AVG(IF( h.timestamp >= CURRENT_DATE + INTERVAL -30 DAY, h.preconormal, NULL)) AS avg_30_day
, MAX(IF( h.timestamp >= CURRENT_DATE + INTERVAL -30 DAY, h.preconormal, NULL)) AS max_30_day
, MIN(IF( h.timestamp >= CURRENT_DATE + INTERVAL -30 DAY, h.preconormal, NULL)) AS min_30_day
, AVG(IF( h.timestamp >= CURRENT_DATE + INTERVAL -15 DAY, h.preconormal, NULL)) AS avg_15_day
, ...
For ongoing sync, I'd probably create a temporary table, populate it with the same query, and then do a sync between the temporary table and the target table. Maybe an INSERT ... ON DUPLICATE KEY and DELETE anti-join (to remove old rows).
Before considering other options, try and make the query more efficient. This is beneficial on the long term: even if you eventually move to a calculated table, you will still take advantage of a more efficient refresh query.
Your query has 15-20 inline subqueries that all address the same dependant table (as far as I read) and do aggregate computations for the same column precos(preconormal) (min, max, avg, most occuring value). Each metric is computed several times in a date range that varies from 9 hours back to 1 month back. So it goes:
SELECT
codigowine,
nomevinho,
DATE(timestamp) AS data_adc,
-- ...
/* Medidas estatÃsticas para 7 dias - min, max, media e moda */
ROUND(
(
SELECT MIN(preconormal)
FROM precos
WHERE
codigowine = vinhos.codigowine
AND timestamp >= CURRENT_DATE - INTERVAL 7 DAY
),
2
) AS min_7_dias,
ROUND(
(
SELECT MAX(preconormal)
FROM precos
WHERE
codigowine = vinhos.codigowine
AND timestamp >= CURRENT_DATE - INTERVAL 7 DAY
),
2
) AS max_7_dias,
-- ... and so on ...
FROM vinhos
It seems like it could be more efficient to do all computation at once, using conditional aggregation:
select
codigowine,
min(preconormal) min_30d
max(preconormal) max_30d,
avg(preconormal) avg_30d,
min(case when timestamp >= current_date - interval 15 day) min_15d,
max(case when timestamp >= current_date - interval 15 day) max_15d,
avg(case when timestamp >= current_date - interval 15 day) avg_15d,
min(case when timestamp >= current_date - interval 7 day) min_07d,
max(case when timestamp >= current_date - interval 7 day) max_07d,
avg(case when timestamp >= current_date - interval 7 day) avg_07d
from precos
where timestamp >= current_date - interval 30 day
group by codigowine
For performance, you want an index on (codigowine, timestamp, preconormal).
Then you can join it with the original table:
select
v.nomevinho,
date(v.timestamp) data_adc,
p.*
from vinhos v
inner join (
select
codigowine,
min(preconormal) min_30d
max(preconormal) max_30d,
avg(preconormal) avg_30d,
min(case when timestamp >= current_date - interval 15 day then preconormal end) min_15d,
max(case when timestamp >= current_date - interval 15 day then preconormal end) max_15d,
avg(case when timestamp >= current_date - interval 15 day then preconormal end) avg_15d,
min(case when timestamp >= current_date - interval 7 day then preconormal end) min_07d,
max(case when timestamp >= current_date - interval 7 day then preconormal end) max_07d,
avg(case when timestamp >= current_date - interval 7 day then preconormal end) avg_07d
from precos
where timestamp >= current_date - interval 30 day
group by codigowine
) p on p.codigowine = v.codigowine
This should be a sensible base query to build upon. To get the other computed values (most occuring value per period, latest value), you may add additional joins, or use inline queries.
To finish: here is another version of the base query, that aggregates after the join. Depending on how your data spreads across the two tables, this may, or may not be more efficient (and will not be equivalent if there are duplicates codigowine in table vinhos):
select
v.nomevinho,
date(v.timestamp) data_adc,
p.codigowine,
date(v.timestamp) data_adc,
min(p.preconormal) min_30d
max(p.preconormal) max_30d,
avg(p.preconormal) avg_30d,
min(case when p.timestamp >= current_date - interval 15 day then p.preconormal end) min_15d,
max(case when p.timestamp >= current_date - interval 15 day then p.preconormal end) max_15d,
avg(case when p.timestamp >= current_date - interval 15 day then p.preconormal end) avg_15d,
min(case when p.timestamp >= current_date - interval 7 day then p.preconormal end) min_07d,
max(case when p.timestamp >= current_date - interval 7 day then p.preconormal end) max_07d,
avg(case when p.timestamp >= current_date - interval 7 day then p.preconormal end) avg_07d
from vinhos v
inner join precos p
on p.codigowine = v.codigowine
and p.timestamp >= current_date - interval 30 day
group by v.codigowine, v.nomevinho
Looking at your query: Try refactoring it to eliminate as many dependent subqueries as possible, and instead JOINing to subqueries. Eliminating those dependent subqueries will make a vast performance difference.
Figuring the mode is an application of finding the detail record for an extreme value in a dataset. If you use this as a subquery
WITH freq AS (
SELECT COUNT(*) freq,
ROUND(preconormal, 2) preconormal,
codigowine
FROM precos
WHERE timestamp >= CURRENT_DATE - INTERVAL 7 DAY
GROUP BY ROUND(preconormal, 2), codigowine
),
most AS (
SELECT MAX(freq) freq,
codigowine
FROM freq
GROUP BY codigowine
),
mode AS (
SELECT GROUP_CONCAT(preconormal ORDER BY preconormal DESC) modeps,
freq.codigowine
FROM freq
JOIN most ON freq.freq = most.freq
GROUP BY freq.codigowine
)
SELECT * FROM mode
You can find the most frequent price for each item. The first CTE, freq, gets the prices and their frequencies.
The second CTE, most, finds the frequency of the most frequent price (or prices).
The third CTE, mode, extracts the most frequent prices from freq using a JOIN. It also uses GROUP_CONCAT() because it's possible to have more than one mode--most frequent price.
For your stats you can do this:
WITH s7 AS (
SELECT ROUND(MIN(preconormal), 2) minp,
ROUND(AVG(preconormal), 2) meanp,
ROUND(MAX(preconormal), 2) maxp,
codigowine
FROM precos
WHERE timestamp >= CURRENT_DATE - INTERVAL 7 DAY
GROUP BY codigowine
),
s15 AS (
SELECT ROUND(MIN(preconormal), 2) minp,
ROUND(AVG(preconormal), 2) meanp,
ROUND(MAX(preconormal), 2) maxp,
codigowine
FROM precos
WHERE timestamp >= CURRENT_DATE - INTERVAL 15 DAY
GROUP BY codigowine
),
s30 AS (
SELECT ROUND(MIN(preconormal), 2) minp,
ROUND(AVG(preconormal), 2) meanp,
ROUND(MAX(preconormal), 2) maxp,
codigowine
FROM precos
WHERE timestamp >= CURRENT_DATE - INTERVAL 30 DAY
GROUP BY codigowine
),
m7 AS (
WITH freq AS (
SELECT COUNT(*) freq,
ROUND(preconormal, 2) preconormal,
codigowine
FROM precos
WHERE timestamp >= CURRENT_DATE - INTERVAL 7 DAY
GROUP BY ROUND(preconormal, 2), codigowine
),
most AS (
SELECT MAX(freq) freq,
codigowine
FROM freq
GROUP BY codigowine
),
mode AS (
SELECT GROUP_CONCAT(preconormal ORDER BY preconormal DESC) modeps,
freq.codigowine
FROM freq
JOIN most ON freq.freq = most.freq
GROUP BY freq.codigowine
)
SELECT * FROM mode
)
SELECT v.codigowine, v.nomevinho, DATE(timestamp) AS data_adc,
s7.minp min_7_dias, s7.maxp max_7_dias, s7.meanp media_7_dias, m7.modeps moda_7_dias,
s15.minp min_15_dias, s15.maxp max_15_dias, s15.meanp media_15_dias,
s30.minp min_30_dias, s30.maxp max_30_dias, s30.meanp media_30_dias
FROM vinhos v
LEFT JOIN s7 ON v.codigowine = s7.codigowine
LEFT JOIN m7 ON v.codigowine = m7.codigowine
LEFT JOIN s15 ON v.codigowine = s15.codigowine
LEFT JOIN s30 ON v.codigowine = s30.codigowine
I'll leave it to you to do the modes for 15 and 30 days.
This is quite the query. You better hope the next guy to work on it doesn't curse your name. :-)
I'm know this can be written as a single SQL statement, but I just don't know how to do it. I have two separate queries. Ont that pulls all orders from a specific period of last year
SELECT * FROM `order` WHERE date_added BETWEEN '2014-10-01' AND '2014-11-01';
and one that pulls from the last month
SELECT * FROM `order` WHERE date_added BETWEEN DATE_SUB( now(), INTERVAL 1 MONTH) AND Now() ORDER BY date_added ASC
What I want to do is now join the two so that I only get the customer_id of orders that were placed inside of the date range last year (query 1), but haven't placed an order in the last month (query 2).
I know there is a way to set this up as a join, but my knowledge on sql join's is not very limited. Thanks for any help.
http://sqlfiddle.com/#!9/35ed0/1
SELECT * FROM `order`
WHERE date_added BETWEEN '2014-10-01' AND '2014-11-01'
AND customer_id NOT IN (
SELECT DISTINCT customer_id FROM `order`
WHERE date_added BETWEEN DATE_SUB( now(), INTERVAL 1 MONTH) AND Now())
UPDATE If you need only 1 records per customer_id, here is an example . It is not very best from performance perspective. But it returns only last (according to the date_added column) order per customer.
SELECT t.*,
if(#fltr=customer_id, 0, 1) fltr,
#fltr:=customer_id
FROM (SELECT *
FROM `order`
WHERE date_added BETWEEN '2014-10-01' AND '2014-11-01'
AND customer_id NOT IN (
SELECT DISTINCT customer_id FROM `order`
WHERE date_added BETWEEN DATE_SUB( now(), INTERVAL 1 MONTH) AND Now())
ORDER BY customer_id, date_added DESC
) t
HAVING (fltr=1);
I usually use a correlated not exists predicate for this as I feel that it corresponds well with the intent of the question:
SELECT *
FROM `order` o1
WHERE date_added BETWEEN '2014-10-01' AND '2014-11-01'
AND NOT EXISTS (
SELECT 1
FROM `order` o2
WHERE date_added BETWEEN DATE_SUB(NOW(), INTERVAL 1 MONTH) AND NOW()
AND o1.customer_id = o2.customer_id
);
I like to approach these questions using group by and having. You are looking for customer ids, so:
select o.customer_id
from orders o
group by o.customer_id
having sum(date_added BETWEEN '2014-10-01' AND '2014-11-01') > 0 and
sum(date_added BETWEEN DATE_SUB( now(), INTERVAL 1 MONTH) AND Now()) = 0;