I have a record set for inspections of many pieces of equipment. The four cols of interest are equip_id, month, year, myData.
My requirement is to have EXACTLY ONE record per month for each piece of equipment.
I have a query that makes the data unique over equip_id, month, year. So there is no more than one record for each month/year for a piece of equipment. But now I need to simulate data for the missing month. I want to simply go back in time to get the last piece of my data.
So that may seem confusing, so I'll show by example.
Given this sample data:
equip_id month year myData
-----------------------------
1 1 2010 500
1 2 2010 600
1 5 2010 800
2 2 2010 300
2 4 2010 400
2 6 2010 500
I want this output:
equip_id month year myData
-----------------------------
1 1 2010 500
1 2 2010 600
1 3 2010 600
1 4 2010 600
1 5 2010 800
2 2 2010 300
2 3 2010 300
2 4 2010 400
2 5 2010 400
2 6 2010 500
Notice that I'm filling in missing data with the data from the month (or two months etc.) before. Also note that if the first record for equip 2 is in 2/2010 than I don't need a record for 1/2010 even though I have one for equip 1.
I just need exactly one record for each month/year for each piece of equipment. So if the record does not exist I just want to go back in time and grab the data for that record.
Thanks!
By no means perfect:
SELECT equip_id, month, mydata
FROM (
SELECT equip_id, month, mydata FROM equip
UNION ALL
SELECT EquipNum.equip_id, EquipNum.Num,
(SELECT Top 1 mydata
FROM equip
WHERE equip.month<n.num And equip.equip_id=equipnum.equip_id
ORDER BY equip.month desc) AS Data
FROM
(SELECT e.equip_id, n.Num
FROM
(SELECT DISTINCT equip_id FROM equip) AS e,
Numbers AS n) AS EquipNum
LEFT JOIN equip
ON (EquipNum.Num = equip.month)
AND (EquipNum.equip_id = equip.equip_id)
WHERE EquipNum.Num<DMax("month","equip")
AND
(SELECT top 1 mydata
FROM equip
WHERE equip.month<n.num And equip.equip_id=equipnum.equip_id
ORDER BY equip.month desc) Is Not Null
AND equip.equip_id Is Null AND equip.Month Is Null) AS x
ORDER BY equip_id, month
For this to work you need a Numbers table, in this case it needs only hold integers from 1 to 12. The numbers table I used is called Numbers and the field is called Num.
EDIT re years comment
SELECT equip_id, year, month, mydata
FROM (
SELECT equip_id, year, month, mydata FROM equip
UNION ALL
SELECT en.equip_id, en.year, en.Num, (SELECT Top 1 mydata
FROM equip e
WHERE e.month<n.num And e.year=en.year And e.equip_id=en.equip_id
ORDER BY e.month desc) AS Data
FROM (SELECT e.equip_id, n.Num, y.year
FROM
(SELECT DISTINCT equip_id FROM equip) AS e,
Numbers AS n,
(SELECT DISTINCT year FROM equip) AS y) AS en
LEFT JOIN equip AS e ON en.equip_id = e.equip_id
AND en.year = e.year
AND en.Num = e.month
WHERE en.Num<DMax("month","equip") AND
(SELECT Top 1 mydata
FROM equip e
WHERE e.month<n.num And e.year=en.year And e.equip_id=en.equip_id
ORDER BY e.month desc) Is Not Null
AND e.equip_id Is Null
AND e.Month Is Null) AS x
ORDER BY equip_id, year, month
I've adjusted to account for year and month... The primary principles remain the same as the original queries presented where just the month. However, for applying a month and year, you need to test for the SET of YEAR + MONTH, ie: what happens if Nov/2009, then jump to Feb/2010, You can't rely on just a month being less than another, but the "set". So, I've apply the year * 12 + month to prevent a false value such as Nov=11 + year=2009 = 2009+11 = 2020, then Feb=2 of year=2010 = 2010+2 = 2012... But 2009*12 = 24108 + Nov = 11 = 24119 compared to 2010*12 = 24120 + Feb =2 = 24122 -- retains proper sequence per year/month combination. The rest of the principles apply. However, one additional, I created a table to represent the span of years to consider. For my testing, I added a sample Equip_ID = 1 entry with a Nov-2009, and Equip_ID = 2 with a Feb-2011 entry and the proper roll-over works too. (Table C_Years, column = year and values of 2009, 2010, 2011)
SELECT
PYML.Equip_ID,
PYML.Year,
PYML.Mth,
P1.MyData
FROM
( SELECT
PAll.Equip_ID,
PAll.Year,
PAll.Mth,
( SELECT MAX( P1.Year*12+P1.Mth )
FROM C_Preset P1
WHERE PAll.Equip_ID = P1.Equip_ID
AND P1.Year*12+P1.Mth <= PAll.CurYrMth) as MaxYrMth
FROM
( SELECT
PYM1.Equip_ID,
Y1.Year,
M1.Mth,
Y1.Year*12+M1.Mth as CurYrMth
FROM
( SELECT p.equip_id,
MIN( p.year*12+p.mth ) as MinYrMth,
MAX( p.year*12+p.mth ) as MaxYrMth
FROM
C_Preset p
group by
1
) PYM1,
C_Years Y1,
C_Months M1
WHERE
Y1.Year*12+M1.Mth >= PYM1.MinYrMth
AND Y1.Year*12+M1.Mth <= PYM1.MaxYrMth
) PAll
) PYML,
C_Preset P1
WHERE
PYML.Equip_ID = P1.Equip_ID
AND PYML.MaxYrMth = P1.Year*12+P1.Mth
If this is going to be a repetative thing/report, I would just create a temporary table with 12 months -- then use that as the primary table, and do a left OUTER join to the rest of your data. This way, you know you'll always get every month, but only when a valid join to the "other side" is identified, you'll get that data too. Ooops... missed your point about the filling in missing elements from the last element... Thinking...
The following works... and I'll describe the elements to what is going on. First, I created a temp table "C_Months" with a column Mth (month) with numbers 1-12. I used "Mth" as an abbreviation of Month to not cause possible conflict with POSSIBLE reserved word MONTH. Additionally, in my query, the table reference "C_Preset" is the prepared set of data you mentioned you already have of distinct elements.
SELECT
LVM.Equip_ID,
LVM.Mth,
P1.Year,
P1.MyData
FROM
( SELECT
JEM.Equip_ID,
JEM.Mth,
( SELECT MAX( P.Mth )
FROM C_Preset P
WHERE P.Equip_ID = JEM.Equip_ID
AND P.Mth <= JEM.Mth ) as MaxMth
FROM
( SELECT distinct
p.equip_id,
c.mth
FROM
C_months c,
C_Preset p
group by
1, 2
HAVING
c.mth >= MIN( p.Mth )
and c.mth <= MAX( p.Mth )
ORDER BY
1, 2 ) JEM
) LVM,
C_Preset P1
WHERE
LVM.Equip_ID = P1.Equip_ID
AND LVM.MaxMth = P1.Mth
ORDER BY
1, 2
The inner most query is a query of the available months (C_Months) associated with a given equipment ID. In your example, equipment ID 1 had a values of 1,2,5. So this would return 1, 2, 3, 4, 5. And for Equipment ID 2, it started with 2, but ended with 6, so it would return 2, 3, 4, 5, 6. Hence the aliased reference JEM (Just Equipment Months)
Then, the field selection for MaxMth (Maximum month)... This is the TRICKY ONE
( SELECT MAX( P.Mth )
FROM C_Preset P
WHERE P.Equip_ID = JEM.Equip_ID
AND P.Mth <= JEM.Mth ) as MaxMth
From this, stating I want the maximum month AVAILABLE (from JEM) associated with the given equipment that is AT OR LESS than the month In question (detecting the highest "valid" equipment item/month within the qualified list. The result of this would result in...
Equip_ID Mth MaxMth
1 1 1
1 2 2
1 3 2
1 4 2
1 5 5
2 2 2
2 3 2
2 4 4
2 5 4
2 6 6
So, for your example of ID = 1, you had months 1, 2, 5 (3 and 4 were missing), so the last valid month that 3 and 4 would refer to is sequence #2's month. Likewise for ID = 2, you had months 2, 4 and 6... Here, 3 would refer back to 2, 5 would refer back to 4.
The rest is the easy part. Now, we join your LVM (Last Valid Month) result as shown above to your original C_Preset (less records). But since we now have the last valid month to directly associate to an existing record in the C_Preset, we join by equipment id and the MaxMth colum, and NOT THE ACTUAL month.
Hope this helps... Again, you'll probably have to change my "mth" column references to "month" to match your format.
Related
I am trying to write an complex mySQL query in which there are 2 tables action and revenue what I need is:
From auction table take out location, postal code on the basis of user, cat_id, cat and spent and join with revenue table which has revenue column so as that given cat_id, cat and date I can figure out the returns that each 'postal' is generating.
Complexities:
User is unique key here
In auction table has column 'spent' but its populates only when 'event' column has 'show' but it has 'cat' entry. And 'cat_id' starts populating at any event except show. So need to map cat_id from 'cat' for event 'show' to get the spent for that cat_id.
The date has to be setup such that while joining the tables the timestamp should be compared for plus minus 10 mins. Right now in my query I have 24 hrs duration
Aggregating on postal in desc order to postal giving highest returns
**Auction Table**
dt user cat_id cat location postal event spent
2020-11-01 22:12:25 1 0 A US X12 Show 2
2020-11-01 22:12:25 1 0 A US X12 Show 2 (duplicate also in table)
2020-11-01 22:12:25 1 6 A US X12 Mid null
2020-11-01 22:13:20 2 0 B UK L23 Show 2
2020-11-01 22:15:24 2 3 B UK L23 End null
**Revenue table**
dt user cat_id revenue
2020-11-01 22:14:45 1 6 null
2020-11-01 22:13:20 2 3 3
Want to create final table(by aggregating on revenue for each 'postal' area):
location postal spend revenue returns
UK X12 2 0 0
US L23 2 3 3/2=1.5
I have written a query but unable to figure out solution for above mention 3 complexities:
Select s.location, s.postal, s.spend, e.revenue
From revenue e JOIN
auction s
on e.user = s.user
where s.event in ('Mid','End','Show') and
TO_DATE(CAST(UNIX_TIMESTAMP(e.dt, 'y-M-d') AS TIMESTAMP)) = TO_DATE(CAST(UNIX_TIMESTAMP(s.dt, 'y-M-d') AS TIMESTAMP)) and
s.cat_id in ('3') and
s.cat = 'B'
Any suggestion will be helpful
This answers the question for MySQL, which is the original tag on the question as well as mentioned in the question.
If I understand correctly, your issue is "joining" within a time frame. You can do what you want using a correlated subquery. Then the rest is aggregation, which I think is:
select location, postal, max(spend), max(revenue)
from (select a.*,
(select sum(r.revenue)
from revenue r
where r.user = a.user and
r.dte >= s.dt - interval 10 minute and
r.dte <= s.dte + interval 10 minute
) as revenue
from auction a
where s.event in ('Mid', 'End', 'Show') and
s.cat_id in (3) and
s.cat = 'B'
) a
group by location, postal;
Let's say you have a user table that has at least the date the user signed up and an id.
Now let's say you have a separate table that tracks an action like a payment that can happen at any point in the user's lifetime. (Say like an in-app purchase.) In that table we track the userId, payment date, and an id for the payment.
So we have something that looks like this to get our schema set up:
CREATE TABLE users (
UserId INT,
AddedDate DATETIME
);
CREATE TABLE payments (
PaymentId INT,
UserId INT,
PaymentDate Datetime
);
Now you want a table that shows weekly cohorts. A table that looks something like this:
Week size w1 w2 w3 w4 w5 w6 w7
2017-08-28 1 0 0 0 1 0 0 0
2017-09-04 3 1 0 2 0 1 1 2
2017-09-11 2 0 0 1 0 0 0 1
2017-09-18 6 3 1 4 3 1 1 2
2017-09-25 2 1 1 1 0 1 2 0
2017-10-02 7 5 2 3 4 3 1 0
2017-10-09 7 4 5 1 2 5 0 0
2017-10-16 2 1 2 1 1 0 0 0
2017-10-23 7 5 4 4 3 0 0 0
2017-10-30 8 8 7 0 0 0 0 0
2017-11-06 5 5 2 0 0 0 0 0
So the first column has the week, the second has number of people that signed up that week. Say we look at week 2017-09-18. 6 people signed up that week. The 3 under the w1 column means that 3 people out of that 6 made a purchase the week they signed up. The 1 under w2 means 1 person out of that 6 made a purchase the second week they were signed up, and so on.
What query would I use to get a table that looks like that?
This query is modified from the one I wrote here: Cohort analysis in SQL
Here's the final query:
SELECT
STR_TO_DATE(CONCAT(tb.cohort, ' Monday'), '%X-%V %W') as date,
size,
w1,
w2,
w3,
w4,
w5,
w6,
w7
FROM (
SELECT u.cohort,
IFNULL(SUM(s.Offset = 0), 0) w1,
IFNULL(SUM(s.Offset = 1), 0) w2,
IFNULL(SUM(s.Offset = 2), 0) w3,
IFNULL(SUM(s.Offset = 3), 0) w4,
IFNULL(SUM(s.Offset = 4), 0) w5,
IFNULL(SUM(s.Offset = 5), 0) w6,
IFNULL(SUM(s.Offset = 6), 0) w7
FROM (
SELECT
UserId,
DATE_FORMAT(AddedDate, "%Y-%u") AS cohort
FROM users
) as u
LEFT JOIN (
SELECT DISTINCT
payments.UserId,
FLOOR(DATEDIFF(payments.PaymentDate, users.AddedDate)/7) AS Offset
FROM payments
LEFT JOIN users ON (users.UserId = payments.UserId)
) as s ON s.UserId = u.UserId
GROUP BY u.cohort
) as tb
LEFT JOIN (
SELECT DATE_FORMAT(AddedDate, "%Y-%u") dt, COUNT(*) size FROM users GROUP BY dt
) size ON tb.cohort = size.dt
So the core of this is we grab the users and the date they signed up and format the date by year-week number, since we are doing a weekly cohort.
SELECT
UserId,
DATE_FORMAT(AddedDate, "%Y-%u") AS cohort
FROM users
Since we want to group by the cohort we have to put this in a subquery in the FROM part of the query.
Then we want join the payment information on the users.
SELECT DISTINCT
payments.UserId,
FLOOR(DATEDIFF(payments.PaymentDate, users.AddedDate)/7) AS Offset
FROM payments
LEFT JOIN users ON (users.UserId = payments.UserId)
This will get unique weekly payment events per user by the numbers of weeks they have been a user. We use distinct because if a user made 2 purchase in one week, we don't want to count that as two users.
We don't just use the payments table, because some users may sign up and not have payments. So we select from the users table and join on the payments table.
You then group by the week - u.cohort. Then you aggregate on the week numbers to find out how many people made payments the weeks after they signed up.
The version of mysql I used had sql_mode set to only_full_group_by. So to get the cohort size I put the bulk of the query in subquery so I could join on the users to get the size of the cohort.
Further considerations:
Filter by weeks is simple. tb.cohort > start date and tb.cohort < end date where start and end date are formatted with "%Y-%u". To make the query more efficient you'll probably want to filter out payment events that don't fall within the date range as well so you're not joining on data you don't need.
You may want to consider using a calender table to cover cases where there are no user sign ups during the week.
Here's a fiddle with everything working: http://sqlfiddle.com/#!9/172dbe/1
To sort by months, you need to transfer the month to Offset
MONTH(payments.PaymentDate) AS Offset
Also to add a date selection with months
DATE_FORMAT(AddedDate, "%Y-%m") AS cohort_month
And add
ORDER BY tb.cohort_month ASC
I have qualified risks with description and creation date, who are attached to subcategory of risks this last ones are attached to category of risks, each risk has a name like 'Risk_1' , my aim is to count the number of risks by month and risk category including zero.
I have this request :
SELECT DISTINCT risk_names.type as risk_name, MONTH(risk.creation_date) as month, count(risk.id) as number FROM risk As risk , risk_category
JOIN (
SELECT risk_category.name as type
FROM
risk_category
) as risk_names on risk_names.type = risk_category.name
where risk.creation_date >= (NOW()-INTERVAL 3 MONTH) GROUP BY MONTH(risk.creation_date), risk_names.type;
Who return this result :
Risk_name month number
---------------------------------
Risk_1 1 10 ---> instead 8
Risk_2 1 10 ---> instead 1
Risk_3 1 10 ---> instead 1
Risk_1 2 12 ......
Risk_2 2 12
Risk_3 2 12
Risk_1 12 4
Risk_2 12 4
Risk_3 12 4
As you can see the number returned is the total for each month , but my aim is to get total for each distinct risk.
Can you help me . thanks
The comma in your FROM is doing a CROSS JOIN. A Cartesian product is unnecessary and throws all the counts off.
I suspect you want something like this:
SELECT rc.type as risk_name, MONTH(r.creation_date) as month,
count(r.id) as number
FROM risk_category rc LEFT JOIN
risk r
ON r.?? = rc.??
where risk.creation_date >= (NOW()-INTERVAL 3 MONTH)
GROUP BY rc.type, MONTH(r.creation_date);
I don't know what the JOIN criterion is between risk and risk_category.
Then try using distinct keuword with count() like count(distinct risk.id) as number instead
In MySQL I'm tasked with a big dataset, with data from 1970 to 2010.
I want to check for consistency: check if each instance occurs minimum one time per year. I took a snippet from 1970-1972 as example to demonstrate my problem.
input:
id year counts
-- ---- ---------
1 1970 1
1 1971 1
2 1970 3
2 1971 8
2 1972 1
3 1970 4
expected:
id 1970-1972
-- ----------
1 no
2 yes
3 no
I though about counting within the date range and then taking those out who had 3 counts: 1970, 1971, 1972. The following query doesn't force the check on each point in the range though.
select id, count(*)
from table1
WHERE (year BETWEEN '1970' AND '1972') AND `no_counts` >= 1
group by id
What to do?
You can use GROUP BY with CASE / inline if.
Using CASE. SQL Fiddle
select id,CASE WHEN COUNT(distinct year) = 3 THEN 'yes'ELSE 'No' END "1970-72"
from abc
WHERE year between 1970 and 1972
GROUP BY id
Using inline IF. SQL Fiddle
select id,IF( COUNT(distinct year) = 3,'yes','No') "1970-72"
from abc
WHERE year between 1970 and 1972
GROUP BY id
You can use a having clause with distinct count:
select `id`
from `table1`
where `year` between '1970' and '1972'
group by id
having count(distinct `year`) = 3
Do you expect this?
select id, count(*)
from table1
WHERE (year BETWEEN '1970' AND '1972')
group by id
having count(distinct year) = 3
I have a big view called: how_many_per_month
name_of_product | how_many_bought | year | month
p1 20 2012 1
p2 7 2012 1
p1 10 2012 2
p2 5 2012 2
p1 3 2012 3
p2 20 2012 3
p3 66 2012 3
How to write MySQL query in order to get only first few occurences of product p1, p2, p3 at once?
To get it one by one for first 3 months I can write:
SELECT name_of_product , sum(how_many_bought) FROM
(SELECT name_of_product, how_many_bought FROM `how_many_per_month`
WHERE name_of_product= 'p1' LIMIT 3) t
How to do it to all possible products at once so my result for taking only first month is like:
p1 20
p2 7
p3 66
For two months:
p1 30
p2 12
p3 66
The problem is that some products are published in different months and I have to make statistic how many of total of them are sold in first month, first 3 months, 6 months, 1 year divided by total.
Example using union
select
name_of_product,
sum(how_many_bought) as bought,
"first month" as period
from how_many_per_month
where month = 1
group by name_of_product
union
select
name_of_product,
sum(how_many_bought) as bought,
"first 2 month" as period
from how_many_per_month
where month <= 2
group by name_of_product
union
select
name_of_product,
sum(how_many_bought) as bought,
"first 6 month" as period
from how_many_per_month
where month <= 6
group by name_of_product
union
select
name_of_product,
sum(how_many_bought) as bought,
"first 12 month" as period
from how_many_per_month
where month <= 12
group by name_of_product
Demo: http://www.sqlfiddle.com/#!2/788ea/11
Results are different a little bit from your expectation. Are you sure that you write them properly? If you need to gain more speed in query time you can use group by case as I've already said.
I'm not quite sure what you're trying to achieve as the description of your question is a bit unclear. From what I've read so far, I understand you want to show the total of how many ITEM_X, ITEM_Y, ITEM_Z were sold for the past 1,3,6 months.
Based on the data you've provided, I've created this sqlfiddle that sums all results and groups them by item. This is the query:
SELECT
name_of_product,
sum(how_many_bought) as how_many_bought
FROM how_many_per_month
WHERE year = 2012
AND month BETWEEN 1 AND 3
GROUP BY name_of_product
-- NOTE: Not specifying an year will result in including all "months"
which are between the values 1 and 3 for all years. Remove it
in case you need that effect.
In the example above the database will sum all sold items between months 1 and 3 (including) for 2012. When you execute this query in your application just change the range in the BETWEEN X AND X and you'll be good to go.
Additional tip:
Avoid using sub-queries or try using the as a last resort method (in case there's simply no other way to do it). They are significantly slower than normal and even join queries. Usually sub-queries can be transformed into a join query.
SELECT
hmpm.name_of_product , SUM(hmpm.how_many_bought)
FROM (
SELECT name_of_product
FROM how_many_per_month
/* WHERE ... */
/* ORDER BY ... */
) sub
INNER JOIN how_many_per_month hmpm
ON hmpm.name_of_product = sub.name_of_product
GROUP BY hmpm.name_of_product
/* LIMIT ... */
MySQL not support LIMIT in subquery, but you need ordering and condition. And why not have id_of_product field?