I have a MySQL table "page" that contains fields "page_name" and "page_timestamp". The table stores page and times requests that were made on a website. I'm trying to write a query that gives me the most hit page for every 5 min period.
Output like
Time Page Hits
12:00 index.html 34
12:05 page1.html 11
12:10 index.html 44
This is one attempt, but no output.
select pages_timestamp,
(select count( pages_name)
from pages t2
where UNIX_TIMESTAMP(t2.pages_timestamp) DIV 300 =
UNIX_TIMESTAMP(pages_timestamp)
group by pages_name
order by count(pages_name) desc
limit 1 )
from pages
where ...
group by UNIX_TIMESTAMP(pages_timestamp ) DIV 300
I'm sure there is a better approach
This assumes page_timestamp is a DATETIME type.
Here is a sample with some data I created, it helps to visualize the output.
So first off, lets create a function for time rounding, this will declutter our query:
drop function if exists rtime;
create function rtime (time DATETIME)
returns DATETIME
BEGIN
DECLARE newtime DATETIME;
set newtime = from_unixtime(floor(unix_timestamp(time)/300)*300);
return newtime;
END;
Next let's explore our data a little bit. I want to pull all Time, Page, and Count(Page_Name), grouping by interval and page.
SELECT
Time, Page, Hits
from (
select rtime(pages_timestamp) as time
, page_name as page
, count(page_name) as hits
from pages
group by page_name, rtime(pages_timestamp)
order by rtime(pages_timestamp), hits desc) g ;
This orders our aggregated table by time interval then the number of hits. Since our groups are ordered by most hits desc we can pull the first row. In MySQL we can non-aggregate non-group by columns (src. This gives us the first row per group, the one with the most hits. We just SELECT * from the above table grouping only by time:
select * from
(select time, page, hits
from (
select rtime(pages_timestamp) as time
, page_name as page
, count(page_name) as hits
from pages
group by page_name, rtime(pages_timestamp)
order by rtime(pages_timestamp), hits desc
) g
) h group by time;
Disclaimer!: If there is a tie, if two pages both have the most hits, this will only pull one record.
Related
I have users and orders tables with this structure (simplified for question):
USERS
userid
registered(date)
ORDERS
id
date (order placed date)
user_id
I need to get array of users (array of userid) who placed their 25th order during specified period (for example in May 2019), date of 25th order for each user, number of days to place 25th order (difference between registration date for user and date of 25th order placed).
For example if user registered in April 2018, then placed 20 orders in 2018, and then placed 21-30th orders in Jan-May 2019 - this user should be in this array, if he placed 25th (overall for his account) order in May 2019.
How I can do this with MySQL request?
Sample data and structure: http://www.sqlfiddle.com/#!9/998358 (for testing you can get 3rd order as ex., not 25th, to not add a lot of sample data records).
One request is not required - if this can't be done in one request, few is possible and allowed.
You can use a correlated subquery to get the count of orders placed before the current one by a user. If that's 24 the current order is the 25th. Then check if the date is in the desired range.
SELECT o1.user_id,
o1.date,
datediff(o1.date, u1.registered)
FROM orders o1
INNER JOIN users u1
ON u1.userid = o1.user_id
WHERE (SELECT count(*)
FROM orders o2
WHERE o2.user_id = o1.user_id
AND o2.date < o1.date
OR o2.date = o1.date
AND o2.id < o1.id) = 24
AND o1.date >= '2019-01-01'
AND o1.date < '2019-06-01';
The basic inefficient way of doing this would be to get the user_id for every row in ORDERS where the date is in your target range AND the count of rows in ORDERS with the same user_id and a lower date is exactly 24.
This can get very ugly, very quickly, though.
If you're calling this from code you control, can't you do it from the code?
If not, there should be a way to assign to each row an index describing its rank among orders for its specific user_id, and select from this all user_id from rows with an index of 25 and a correct date. This will give you a select from select from select, but it should be much faster. The difficulty here is to control the order of the rows, so here are the selects I envision:
Select all rows, order by user_id asc, date asc, union-ed to nothing from a table made of two vars you'll initialize at 0.
from this, select all while updating a var to know if a row's user_id is the same as the last, and adding a field that will report so (so for each user_id the first line in order will have a specific value like 0 while the other rows for the same user_id will have a 1)
from this, select all plus a field that equals itself plus one in case the first added field is 1, else 0
from this, select the user_id from the rows where the second added field is 25 and the date is in range.
The union thingy is only necessary if you need to do it all in one request (you have to initialize them in a lower select than the one they're used in).
Edit: Well if you need the date too you can just select it along with the user_id, but calculating the number of days in sql will be a pain. Just join the result table to the users table and get both the date of 25th order and their date of registration, you'll surely be able to do the difference in code.
I'll try building an actual request, however if you want to truly understand what you need to make this you gotta read up on mysql variables, unions, and conditional statements.
"Looks too complicated. I am sure that this can be done with current DB structure and 1-2 requests." Well, yeah. Use the COUNT request, it will be easy, and slow as hell.
For the complex answer, see http://www.sqlfiddle.com/#!9/998358/21
Since you can use multiple requests, you can just initialize the vars first.
It isn't actually THAT complicated, you just have to understand how to concretely express what you mean by "an user's 25th command" to a SQL engine.
See http://www.sqlfiddle.com/#!9/998358/24 for the difference in days, turns out there's a method for that.
Edit 5: seems you're going with the COUNT method. I'll pray your DB is small.
Edit 6: For posterity:
The count method will take years on very large databases. Since OP didn't come back, I'm assuming his is small enough to overlook query speed. If that's not your case and let's say it's 10 years from now and the sqlfiddle links are dead; here's the two-queries solution:
SET #PREV_USR:=0;
SELECT user_id, date_ FROM (
SELECT user_id, date_, SAME_USR AS IGNORE_SMUSR,
#RANK_USR:=(CASE SAME_USR WHEN 0 THEN 1 ELSE #RANK_USR+1 END) AS RANK FROM (
SELECT orders.*, CASE WHEN #PREV_USR = user_id THEN 1 ELSE 0 END AS SAME_USR,
#PREV_USR:=user_id AS IGNORE_USR FROM
orders
ORDER BY user_id ASC, date_ ASC, id ASC
) AS DERIVED_1
) AS DERIVED_2
WHERE RANK = 25 AND YEAR(date_) = 2019 AND MONTH(date_) = 4 ;
Just change RANK = ? and the conditions to fit your needs. If you want to fully understand it, start by the innermost SELECT then work your way high; this version fuses the points 1 & 2 of my explanation.
Now sometimes you will have to use an API or something and it wont let you keep variable values in memory unless you commit it or some other restriction, and you'll need to do it in one query. To do that, you put the initialization one step lower and make it so it does not affect the higher statements. IMO the best way to do this is in a UNION with a fake table where the only row is excluded. You'll avoid the hassle of a JOIN and it's just better overall.
SELECT user_id, date_ FROM (
SELECT user_id, date_, SAME_USR AS IGNORE_SMUSR,
#RANK_USR:=(CASE SAME_USR WHEN 0 THEN 1 ELSE #RANK_USR+1 END) AS RANK FROM (
SELECT DERIVED_4.*, CASE WHEN #PREV_USR = user_id THEN 1 ELSE 0 END AS SAME_USR,
#PREV_USR:=user_id AS IGNORE_USR FROM
(SELECT * FROM orders
UNION
SELECT * FROM (
SELECT (#PREV_USR:=0) AS INIT_PREV_USR, 0 AS COL_2, 0 AS COL_3
) AS DERIVED_3
WHERE INIT_PREV_USR <> 0
) AS DERIVED_4
ORDER BY user_id ASC, date_ ASC, id ASC
) AS DERIVED_1
) AS DERIVED_2
WHERE RANK = 25 AND YEAR(date_) = 2019 AND MONTH(date_) = 4 ;
With that method, the thing to watch for is the amount and the type of columns in your basic table. Here orders' first field is an int, so I put INIT_PREV_USR in first then there are two more fields so I just add two zeroes with names and call it a day. Most types work, since the union doesn't actually do anything, but I wouldn't try this when your first field is a blob (worst comes to worst you can use a JOIN).
You'll note this is derived from a method of pagination in mysql. If you want to apply this to other engines, just check out their best pagination calls and you should be able to work thinks out.
I am using the Graph Reports for the select below. The MySQL database only has the active records in the database, so if no records are in the database from X hours till Y hours that select does not return anything. So in my case, I need that select return Paypal zero values as well even the no activity was in the database. And I do not understand how to use the UNION function or re-create select in order to get the zero values if nothing was recorded in the database in time interval. Could you please help?
select STR_TO_DATE ( DATE_FORMAT(`acctstarttime`,'%y-%m-%d %H'),'%y-%m-%d %H')
as '#date', count(*) as `Active Paid Accounts`
from radacct_history where `paymentmethod` = 'PayPal'
group by DATE_FORMAT(`#date`,'%y-%m-%d %H')
When I run the select the output is:
Current Output
But I need if there are no values between 2016-07-27 07:00:00 and 2016-07-28 11:00:00, then in every hour it should show zero active accounts Like that:
Needed output with no values every hour
I have created such select below , but it not put to every hour the zero value like i need. showing the big gap between the 12 Sep and 13 Sep anyway, but there should be the zero values every hour
(select STR_TO_DATE ( DATE_FORMAT(acctstarttime,'%y-%m-%d %H'),'%y-%m-%d %H')
as '#date', count(paymentmethod) as Active Paid Accounts
from radacct_history where paymentmethod <> 'PayPal'
group by DATE_FORMAT(#date,'%y-%m-%d %H'))
union ALL
(select STR_TO_DATE ( DATE_FORMAT(acctstarttime,'%y-%m-%d %H'),'%y-%m-%d %H')
as '#date', 0 as Active Paid Accounts
from radacct_history where paymentmethod <> 'PayPal'
group by DATE_FORMAT(#date,'%y-%m-%d %H')) ;
I guess, you want to return 0 if there is no matching rows in MySQL. Here is an example:
(SELECT Col1,Col2,Col3 FROM ExampleTable WHERE ID='1234')
UNION (SELECT 'Def Val' AS Col1,'none' AS Col2,'' AS Col3) LIMIT 1;
Updated the post: You are trying to retrieve data that aren't present in the table, I guess in reference to the output provided. So in this case, you have to maintain a date table to show the date that aren't in the table. Please refer to this and it's little bit tricky - SQL query that returns all dates not used in a table
You need an artificial table with all necessary time intervals. E.g. if you need daily data create a table and add all day dates e.g. start from 1970 till 2100.
Then you can use the table and LEFT JOIN your radacct_history. So for each desired interval you will have group item (group by should be based on the intervals table.
I have a table called 'events'.
It contains eventID (INT), eventDateTime(DATETIME), and eventMessage(VARCHAR).
I want to be able group the rows by eventDateTime where there is another row with eventDateTime within 1 hour each side. This should propogate forever (for example a group should be able go on for years, as long as there is never a gap longer than an hour between a linking chain of eventDateTime values within that time period. Ideally I want to end up selecting MIN(eventID) for each group, and both the MIN and MAX of eventDateTime which will give me the time span in which the group runs.
I assume I need some kind of iterating loop to do this? Where would I start?
Let's start from subqueries we need
SET #row_number1 = 0;
SET #row_number2 = 0;
The query returns us the events table ordered with row numbers (rn)
SELECT
(#row_number1:=#row_number1 + 1) AS rn, eventID, eventDateTime
FROM
events
ORDER BY eventDateTime
Let's mar them as SUB1 and SUB2
Then let's join them
select *
from SUB1 join SUB2 on sub1.rn=sub2.rn+1
So we have in one row 2 eventDateTime of current and next row and can calculate time difference
TIMESTAMPDIFF(HOUR, SUB1.eventDateTime, SUB2.eventDateTime) as hoursDiff
Then we can add HAVING hourDiff>1 to have rule breaking intervals. For such records SUB1.eventDateTime is the end of previous group but SUB2.eventDateTime is the beginning of next group.
So our query will return us
SUB1.eventID as previousGroupEndEventId,
SUB1.eventDateTime as previousGroupEndeventDateTime,
SUB2.eventID as currentGroupStartEventId,
SUB2.eventDateTime as currentGroupStarteventDateTime,
TIMESTAMPDIFF(HOUR, SUB1.eventDateTime, SUB2.eventDateTime) as breakInterval
And you can use the query results to get all your info
For complex problems requiring some form of looping, some databases allow recursive queries, but apparently not mysql.
Fortunately, in your case I don't think it is necessary. You can instead look for any rows which don't have another row in the preceeding hour thus:
select *
from events as A
where not exists (
select 1
from events as B
where B.eventDateTime < A.eventDateTime
and B.eventDateTime > DATE_ADD(A.eventDateTime, INTERVAL -1 HOUR)
)
Example kept simple. Fix up the details to meet your requirements.
Working example is here: http://sqlfiddle.com/#!9/c3b73c/1
So I get a large amount of data from server using this SQL:
SELECT value,DATE_FORMAT(`time`,'%Y-%m-%dT%H:%i:%sZ') AS `time`
FROM history WHERE :id=reference AND
(time BETWEEN :start AND :end) ORDER BY time LIMIT 100 ";
Limit is set to fixed 100 entries.
But in given time range there could be 5 000 entries.
Here's my goal: I want to sample these entries by time between each entry.
So for example this interval between each entry will be 60 seconds (let's say it is parameter), then I will receive 100 entries (from 5000), but there will be always one minute difference between each one of them.
E.g.
value1,14:40:40
value2,14:41:40
...
value100,16:20:40
Is this doable via SQL? Or do I have to parse through this large data with PHP?
If it is not doable just with SQL, is it possible to get this 100 entries equally spread across this 5000 entries? (so not by time, but I'd get fixed entry id1,id50,id100,id150,...,id5000). Again just with sql.
Thanks!
Just as Kristof sais in his answer: Order the rows and take each nth row by applying a row number. This is how it is done in MySQL:
select
rows.value,
date_format(rows.`time`,'%Y-%m-%dT%H:%i:%sZ') AS `time`
from
(
select
#row_number := #row_number + 1 as row_number,
history.*
from history
cross join (select #row_number := 0) as t
where reference = :id and `time` between :start and :end
order by `time`
) as rows
cross join
(
select count(*) as cnt
from history
where reference = :id and `time` between :start and :end
) as rowcount
where mod(rows.row_number - 1, ceil(rowcount.cnt / 100)) = 0;
And this is how the same would look in another dbms, Oracle for instance, using analytic functions:
select
rows.value,
to_char(rows."time",'yyyy-mm-dd hh24:mi:ss') AS "time"
from
(
select
row_number() over (order by "time") as rown,
count(*) over () as cnt,
history.*
from history
where reference = :id and "time" between :start and :end
) rows
where mod(rows.rown - 1, ceil(rows.cnt / 100)) = 0;
These queries result in 100 records or a little less, depending on how many rows the table contains exactly. You can also use TRUNCATE(rowcount.cnt,0) instead of CEIL(rowcount.cnt) in MySQL, thus getting hundred rows or a little more and additionally apply LIMIT 100 to get exactly 100 rows (provided there are at least 100 rows in the table).
What you could is select the rowNumber and calculate the modulo of that rowNumber.
Not sure how it would be done in mysql but t-sql goes like this :
SELECT ROW_NUMBER() over( order by idField) % 50 as selector, *
FROM history
WHERE selector = 1
This will count the rows and reset the counter every 50th record, giving you a spread out result.
I have a table with a date stamp E.g (1241037505). There's also a column with the number of views.
The data stamp resembles when it was created.
So I want to select the top viewed threads from the past week.
How do I do this?
Try this:
SELECT * WHERE
DATEDIFF(NOW(),created_date) < 7
SELECT * FROM table WHERE createdon > SUBDATE(NOW(), '7 day') ORDER BY hits DESC;
See: http://dev.mysql.com/doc/refman/5.1/en/date-and-time-functions.html#function_subdate
The data you're currently tracking isn't going to allow you to select the top viewed in the last week. It will show you the top viewed over all time, or the most viewed items created in the last week. If something was created two weeks ago, but was viewed more than anything else during the last week you cannot determine that from the data you're tracking. One way I can see to do it would be to track the number of hits each content item gets each day of the week.
create table daily_hits {
cid integer, -- content id points to the table you already have
dotw smallint, -- 0-6 or similar
hits integer
PRIMARY KEY (cid, dotw)
}
Whenever you increase the hit count on the content item, you would also update the daily_hits table for the given content id and day of the week. You would need a function that converted the current date/time to a day of the week. MySql provides DAYOFWEEK for this purpose.
To get the most viewed in the last week, you could query like this:
SELECT cid, SUM(hits) FROM daily_hits GROUP BY cid ORDER BY SUM(hits) DESC
You will need some type of scheduled job that deletes the current day of the week at midnight so you aren't accumulating forever and essentially performing the same accumulation happening on the hits column of the current table.
SELECT * FROM table WHERE Date_Created > (7 days ago value) ORDER BY Hits LIMIT 0,100
or you could use this (per WishCow's Answer)
SELECT * FROM table WHERE Date_Created > SUBDATE(NOW(), '7 day') ORDER BY Hits LIMIT 0,100