Here is my data structure :
name value date_received
foo 100 2013-09-19 10:00:00
bar 200 2013-09-19 10:00:00
foo 100 2013-09-19 10:05:00 //no change
bar 200 2013-09-19 10:05:00 //no change
foo 110 2013-09-19 10:08:00 // foo changed
bar 200 2013-09-19 10:08:00 // no change
......
Question:
I want a query (mysql) which can do something like:
select date_received where anyOf(foo, bar) changed from the previous
specified value in the past N hours.
There could be other names in the table but we are only interested in foo and bar.
Any pointers. To me it looks like we'll need a self join - but don't know how.
EDIT: looks like the below query is just a good starting point.
select date_received from (SELECT DISTINCT name, value from data) a
INNER JOIN (select DISTINCT name, value, date_received from data)b
on (a.name=b.name and a.value=b.value)
update Looks like below query works - easier than I thought it would be.
SELECT DISTINCT a.tr FROM (
SELECT name, value, MAX(date_received) dr from data
where date_received > now() - INTERVAL 2 hour
GROUP BY name, value order by dr desc)a;
I do not see how your edited query solves the problem. Where does the "last N hours" come in, for instance?
I would approach this by looking at the previous value, then using logic around the datetime constraints and value changes to see if there has been a change. Your question is ambiguous: Are you looking for changes only in the last N hours? Are you looking for a change from the last value before N hours? What happens if the value changes back?
All of these, though, could be answered by having the previous value and previous time on each row. Here is an example of how to get this:
select t.*,
(select t.date_received
from t t2
where t2.date_received < t.date_received and
t2.name = t.name
order by t2.date_received desc
limit 1
) as prev_date_received,
(select t.value
from t t2
where t2.date_received < t.date_received and
t2.name = t.name
order by t2.date_received desc
limit 1
) as prev_value
from t
having <your logic goes here for the date time and changes you care about>;
This is using the having clause instead of a subquery, just out of convenience (this is not supported by other databases).
For instance, if you want any changes in the last N hours:
having date_received > now() - interval N hour and prev_value <> value
Related
I have three columns User_ID, New_Status and DATETIME.
New_Status contains 0(inactive) and 1(active) for users.
Every user starts from active status - ie. 1.
Subsequently table stores their status and datetime at which they got activated/inactivated.
How to calculate number of active users at the end of each date, including dates when no records were generated into the table.
Sample data:
| ID | New_Status | DATETIME |
+----+------------+---------------------+
| 1 | 1 | 2019-01-01 21:00:00 |
| 1 | 0 | 2019-02-05 17:00:00 |
| 1 | 1 | 2019-03-06 18:00:00 |
| 2 | 1 | 2019-01-02 01:00:00 |
| 2 | 0 | 2019-02-03 13:00:00 |
Format the date time value to a date only string and group by it
SELECT DATE_FORMAT(DATETIME, '%Y-%m-%d') as day, COUNT(*) as active
FROM test
WHERE New_Status = 1
GROUP BY day
ORDER BY day
In MySQL 8 you can use the row_number() window function to get the last status of a user per day. Then filter for the one that indicate the user was active GROUP BY the day and count them.
SELECT date(x.datetime),
count(*)
FROM (SELECT date(t.datetime) datetime,
t.new_status,
row_number() OVER (PARTITION BY date(t.datetime)
ORDER BY t.datetime DESC) rn
FROM elbat t) x
WHERE x.rn = 1
AND x.new_status = 1
GROUP BY x.datetime;
If not all days are in the table you need to create a (possibly derived) table with all days and cross join it.
Find out the last activity status of users whose activity was changed for each day
select User_ID, New_Status, DATE_FORMAT(DATETIME, '%Y-%m-%d')
from activity_table
where not exists
(
select 1
from activity_table at
where at.User_ID = activity_table.User_ID and
DATE_FORMAT(at.DATETIME, '%Y-%m-%d') = DATE_FORMAT(activity_table.DATETIME, '%Y-%m-%d') and
at.DATETIME > activity_table.DATETIME
)
order by DATE_FORMAT(activity_table.DATETIME, '%Y-%m-%d');
This is not the solution yet, but a very very useful information before solution. Note that here not all dates are covered yet and the values are individual records, more precisely their last values on each day, ordered by the date.
Let's get aggregate numbers
Using the query above as a subselect and aliasing it into a table, you can group by DATETIME and do a select sum(new_Status) as activity, count(*) total, DATETIME so you will know that activity - (total - activity) is the difference in comparison to the previous day.
Knowing the delta for each day present in the result
At the previous section we have seen how the delta can be calculated. If the whole query in the previous section is aliased, then you can self join it using a left join, with pairs of (previous date, current date), still having the gaps of dates, but not worrying about that just yet. In the case of the first date, its activity is the delta. For subsequent records, adding the previous day's delta to their delta yields the result you need. To achieve this you can use a recursive query, supported by MySQL 8, or, alternatively, you can just have a subquery which sums the delta of previous days (with special attention to the first date, as described earlier) will and adding the current date's delta yields the result we need.
Fill the gaps
The previous section would already perfectly work (assuming the lack of integrity problems), assuming that there were activity changes for each day, but we will not continue with the assumption. Here we know that the figures are correct for each date where a figure is present and we will need to just add the missing dates into the result. If the results are properly ordered, as they should be, then one can use a cursor and loop the results. At each record after the first one, we can determine the dates that are missing. There might be 0 such dates between two consequent dates or more. What we do know about the gaps is that their values are exactly the same as the previous record, that do has data. If there were no activity changes on a given date, then the number of active users is exactly the same as in the previous day. Using some structure, like a table you can generate the results you have with the knowledge described here.
Solving possible integrity problems
There are several possibilities for such problems:
First, a data item might exist prior to the introduction of this table's records were started to be spawned.
Second, bugs or any other causes might have made a pause in creating records for this activity table.
Third, the addition of user is or was not necessarily generating an activity change, since its popping into existence renders its previous state of activity undefined and subject to human standards, which might change over time.
Fourth, the removal of user is or was not necessarily generating an activity change, since its popping out of existence renders is current state of activity undefined and subject to human standards, which might change over time.
Fifth, there is an infinity of other issues which might cause data integrity issues.
To cope with these you will need to comprehensively analyze whatever you can from the source-code and the history of the project, including database records, logs and humanly available information to detect such anomalies, the time they were effective and figure out what their solution is if they exist.
EDIT
In the meantime I was thinking about the possibility of a user, who was active at the start of the day being deactivated and then activated again by the end of the day. Similarly, an inactive user during a day might be activated and then finally deactivated by the end of the day. For users that have more than an activation at the start of the day, we need to compare their activity status at the start and the end of the day to find out what the difference was.
SELECT
DATE(DATETIME),
COUNT(*)
FROM your_table
WHERE New_Status = 1
GROUP BY User_ID,
DATE(DATETIME)
For MySQL
WITH RECURSIVE
cte AS (
SELECT MIN(DATE(DT)) dt
FROM src
UNION ALL
SELECT dt + INTERVAL 1 DAY
FROM cte
WHERE dt < ( SELECT MAX(DATE(DT)) dt
FROM src )
),
cte2 AS
(
SELECT users.id,
cte.dt,
SUM( CASE src.New_Status WHEN 1 THEN 1
WHEN 0 THEN -1
ELSE 0
END ) OVER ( PARTITION BY users.id
ORDER BY cte.dt ) status
FROM cte
CROSS JOIN ( SELECT DISTINCT id
FROM src ) users
LEFT JOIN src ON src.id = users.id
AND DATE(src.dt) = cte.dt
)
SELECT dt, SUM(status)
FROM cte2
GROUP BY dt;
fiddle
Do not forget to adjust max recursion depth.
Here is what I believe is a good solution for this problem of yours:
SELECT SUM(New_Status) "Number of active users"
, DATE_FORMAT(DATEC, '%Y-%m-%d') "Date"
FROM TEST T1
WHERE DATE_FORMAT(DATEC,'%H:%i:%s') =
(SELECT MAX(DATE_FORMAT(T2.DATEC,'%H:%i:%s'))
FROM TEST T2
WHERE T2.ID = T1.ID
AND DATE_FORMAT(T1.DATEC, '%Y-%m-%d') = DATE_FORMAT(T2.DATEC, '%Y-%m-%d')
GROUP BY ID
, DATE_FORMAT(DATEC, '%Y-%m-%d'))
GROUP BY DATE_FORMAT(DATEC, '%Y-%m-%d');
Here is the DEMO
I'm pretty bad with dates.
I have a mysql table with one field, which is OF DateTime type, called HoraRegistratBBDD.
What I want to do is to select data (any kind of data) from a specific day. So far I was doing this:
SELECT COUNT(*)
FROM
(
SELECT mydata
FROM mytable
WHERE DATE(`HoraRegistratBBDD`) = '".$fecha."' AND
FetOPerdutIMotiu = '1'
GROUP BY Partit,
mydata
) AS Col;
Where $fecha is something like "2016-09-03". THIS WORKS.
But I have a problem. When my HoraRegistratBBDD has (for example) this value:
2016-09-02 10:28:41
I would like to substract 15 hours from it. Meaning that I would like to treat this value like it's actually
2016-09-01 19:28:41
How can I do my query considering that I want to substract hours from it (therefore, day will change sometimes)?
If you want to subtract 15 hours from the HoraRegistratBBDD column, then you can use DATE_SUB:
SELECT mydata FROM mytable
WHERE DATE_SUB(HoraRegistratBBDD, INTERVAL 15 HOUR) = ...
The function that you are looking for is DATE_SUB.
Here are a few links:
http://www.w3schools.com/sql/func_date_sub.asp
How to subtract 3 hours from a datetime in MySQL?
The first one shows you how it works and the other one is a similar question and it has been answered.
SELECT COUNT(*)
FROM
(
SELECT mydata, DATE_FORMAT(HoraRegistratBBDD,'%Y-%m-%d') AS niceDate
FROM mytable
WHERE
FetOPerdutIMotiu = '1'
HAVING niceDate = '".$fecha."'
GROUP BY Partit,
mydata
) AS Col;
I've been trying to work this one out for a while now, maybe my problem is coming up with the correct search query. I'm not sure.
Anyway, the problem I'm having is that I have a table of data that has a new row added every second (imagine the structure {id, timestamp(datetime), value}). I would like to do a single query for MySQL to go through the table and output only the first value of each minute.
I thought about doing this with multiple queries with LIMIT and datetime >= (beginning of minute) but with the volume of data I'm collecting that is a lot of queries so it would be nicer to produce the data in a single query.
Sample data:
id datetime value
1 2015-01-01 00:00:00 128
2 2015-01-01 00:00:01 127
3 2015-01-01 00:00:04 129
4 2015-01-01 00:00:05 127
...
67 2015-01-01 00:00:59 112
68 2015-01-01 00:01:12 108
69 2015-01-01 00:01:13 109
Where I would want the result to select the rows:
1 2015-01-01 00:00:00 128
68 2015-01-01 00:01:12 108
Any ideas?
Thanks!
EDIT: Forgot to add, the data, whilst every second, is not reliably on the first second of every minute - it may be :30 or :01 rather than :00 seconds past the minute
EDIT 2: A nice-to-have (definitely not required for answer) would be a query that is flexible to also take an arbitrary number of minutes (rather than one row each minute)
SELECT t2.* FROM
( SELECT MIN(`datetime`) AS dt
FROM tbl
GROUP BY DATE_FORMAT(`datetime`,'%Y-%m-%d %H:%i')
) t1
JOIN tbl t2 ON t1.dt = t2.`datetime`
SQLFiddle
Or
SELECT *
FROM tbl
WHERE dt IN ( SELECT MIN(dt) AS dt
FROM tbl
GROUP BY DATE_FORMAT(dt,'%Y-%m-%d %H:%i'))
SQLFiddle
SELECT t1.*
FROM tbl t1
LEFT JOIN (
SELECT MIN(dt) AS dt
FROM tbl
GROUP BY DATE_FORMAT(dt,'%Y-%m-%d %H:%i')
) t2 ON t1.dt = t2.dt
WHERE t2.dt IS NOT NULL
SQLFiddle
In MS SQL Server I would use CROSS APPLY, but as far as I know MySQL doesn't have it, so we can emulate it.
Make sure that you have an index on your datetime column.
Create a table of numbers, or in your case a table of minutes. If you have a table of numbers starting from 1 it is trivial to turn it into minutes in the necessary range.
SELECT
tbl.ID
,tbl.`dt`
,tbl.value
FROM
(
SELECT
MinuteValue
, (
SELECT tbl.id
FROM tbl
WHERE tbl.`dt` >= Minutes.MinuteValue
ORDER BY tbl.`dt`
LIMIT 1
) AS ID
FROM Minutes
) AS IDs
INNER JOIN tbl ON tbl.ID = IDs.ID
For each minute find one row that has timestamp greater than the minute. I don't know how to return the full row, rather than one column in MySQL in the nested SELECT, so at first I'm making a temp table with two columns: Minute and id from the original table and then explicitly look up rows from original table knowing their IDs.
SQL Fiddle
I've created a table of Minutes in the SQL Fiddle with the necessary values to make example simple. In real life you would have a more generic table.
Here is SQL Fiddle that uses a table of numbers, just for illustration.
In any case, you do need to know in advance somehow the range of dates/numbers you are interested in.
It is trivial to make it work for any interval of minutes. If you need results every 5 minutes, just generate a table of minutes that has values not every 1 minute, but every 5 minutes. The main query would remain the same.
It may be more efficient, because here you don't join the big table to itself and you don't make calculations on the datetime column, so the server should be able to use the index on it.
The example that I made assumes that for each minute there is at least one row in the big table. If it is possible that there are some minutes that don't have any data at all you'd need to add extra check in the WHERE clause to make sure that the found row is still within that minute.
select * from table where timestamp LIKE "%-%-% %:%:00" could work.
This is similar to this question: Stack Overflow Date SQL Query Question
Edit: This probably would work better:
`select , date_format(timestamp, '%Y-%m-%d %H:%i') as the_minute, count()
from table
group by the_minute
order by the_minute
Similar to this question here: mysql select date format
i'm not really sure, but you could try this:
SELECT MIN(timestamp) FROM table WHERE YEAR(timestamp)=2015 GROUP BY DATE(timestamp), HOUR(timestamp), MINUTE(timestamp)
I have a table recording the accumulative total visit numbers of some web pages every day. I want to fetch the real visit numbers in a specific day for all these pages. the table is like
- record_id page_id date addup_number
- 1 1 2012-9-20 2110
- 2 2 2012-9-20 1160
- ... ... ... ...
- n 1 2012-9-21 2543
- n+1 2 2012-9-21 1784
the result I'd like to fetch is like:
- page_id date increment_num(the real visit numbers on this date)
- 1 2012-9-21 X
- 2 2012-9-21 X
- ... ... ...
- N 2012-9-21 X
but I don't want to do this in php, cause it's time consuming. Can I get what I want with SQL directives or with some mysql functions?
Ok. You need to join the table on itself by joining on the date column and adding a day to one side of the join.
Assuming:
date column is a legitimate DATE Type and not a string
Every day is accounted for each page (no gaps)
addup_number is an INT of some type (BIGINT, INT, SMALLINT, etc...)
table_name is substituted for your actual table name which you don't indicate
Only one record per day for each page... i.e. no pages have multiple counts on the same day
You can do this:
SELECT t2.page_id, t2.date, t2.addup_number - t1.addup_number AS increment_num
FROM table_name t1
JOIN table_name t2 ON t1.date + INTERVAL 1 DAY = t2.date
WHERE t1.page_id = t2.page_id
One thing to note is if this is a huge table and date is an indexed column, you'll suffer on the join by having to transform it by adding a day in the ON clause, but you'll get your data.
UPDATED:
SELECT today.page_id, today.date, (today.addup_number - yesterday.addup_number) as increment
FROM myvisits_table today, myvisits_table yesterday
WHERE today.page_id = yesterday.page_id
AND today.date='2012-9-21'
AND yesterday.date='2012-9-20'
GROUP BY today.page_id, today.date, yesterday.page_id, yesterday.date
ORDER BY page_id
Something like this:
SELECT date, SUM(addup_number)
FROM your_table
GROUP BY date
How do I modify this MySQL query to only count leadIDs from table leads where column 'Date' contains the newest (youngest) date?
SELECT COUNT(leadID) as accepted FROM leads WHERE change like '%OK%'
The problem is that leadID can have multiple instances in table leads. The original query result is "4" because of one duplicate. The correct result is "3".
The date is stored in this format: 2011-10-26 18:23:52. The result should take hours and minutes into consideration when determining the youngest date.
TABLE leads:
leadID | date | change
1 | 2011-10-26 18:23:52 | BAD
1 | 2011-10-26 17:00:00 | OK
2 | 2011-10-26 19:23:52 | OK
3 | 2011-10-26 20:23:52 | OK
4 | 2011-10-26 21:23:52 | OK
5 | 2011-10-26 22:23:52 | BAD
I think this is what you're looking for:
select count(distinct l1.leadId) as accepted from leads l1
left join leads l2
on l1.leadId = l2.leadId and l1.date < l2.date
where l2.date is null and l1.`change` like '%OK%'
You must decide what you mean by newest date: the single latest? yesterday? today?
if yesterday, then add this to your query clause
select * from mytable where date >= date_sub(now(), interval 1 day)
if you are using oracle database you can use max() function to extract newest date from the table, further to check with the table for this newest date :-
SELECT COUNT(leadID) as accepted FROM leads WHERE change like '%OK%'
and date_col = (select max(date_col) from leads)
I am assuming that with newest date your mean is about newest in the table data..
changes :- as per changes in question and as per mentioned in commends ..
I think you want to take newest date among the records having "change" column value like '%OK%' and want to count distinct leadId
please try the following query-
SELECT COUNT(distinct leadID) as accepted FROM leads WHERE change like '%OK%'
and date_col = (select max(date_col) from leads WHERE change like '%OK%')
You can try (in case your date is a int like return by time() function)
$sql = "SELECT COUNT(leadID) as accepted FROM leads WHERE change like '%OK%' ORDER BY Date DESC LIMIT 1"
You will only extract the newest entry.
Edit: This shouldalso works for your date format YYYY-MM-DD hh:mm:ss
Edit 2: Okay, I did not understood your question.
You have a table lead: leadid date
You want to count the number of row for the newset date.
Like another pointed out you can use the MAX operator:
SELECT COUNT(distinct leadid)
FROM LEAD AS l,
( SELECT MAX(Date) mdate FROM Lead ) AS MaxDate
WHERE l.date = MaxDate.mdate
AND l.change like '%OK%'