SQL Formula for mysql Table - mysql

Hello – I have a DB table (MySQL ver 5.6.41-84.1-log) that has about 92,000 entries, with columns for:
id (incremental unique ID)
post_type (not important)
post_id (not important, but shows relation to another table)
user_id (not important)
vote (not important)
ip (IP Address, ie. 123.123.123.123)
voted (Datestamp in GMT, ie. 2018-12-03 04:50:05)
I recently ran a contest and we had a rule that no single IP could vote more than 60 times per day. So now I need to run a custom SQL formula that applies the following rule:
For each IP address, for each day, if there are > 60 rows, delete those additional rows.
Thank you for your help!

This is a complicated one, and I think it is hard to provide a 100% sure answer without actual table and data to play with.
However let me try to describe the logic, and build the query step by step so you can paly around with it and possibly fix lurking erros.
1) We start with selecting all ip adresses that posted more than 60 votes on a given day. For this we use a group by on the voting day and on the ip adress, combined with a having clause
select date(voted), ip_adress
from table
group by date(voted), ip_adress
having count(*) > 60
2) From then, we go back to the table and select the first 60 ids corresponding to each voting day / ip adress couple. id is an autoincremented field so we just sort using this field and the use the mysql limit instruction
select id, ip_adress, date(voted) as day_voted
from table
where ip_adress, date(voted) in (
select date(voted), ip_adress
from table
group by date(voted), ip_adress
having count(*) > 60
)
order by id
limit 60
3) Finally, we go back once again to the table and search for the all ids whose ip adress and day of vote belong to the above list, but whose id is greater than the max id of the list. This is achieved with a join and requires a group by clause.
select t1.id
from
table t1
join (
select id, ip_adress, date(voted) as day_voted
from table
where ip_adress, date(voted) in (
select date(voted), ip_adress
from table
group by date(voted), ip_adress
having count(*) > 60
)
order by id
limit 60
) t2
on t1.ip_adress = t2.ip_adress
and date(t1.voted) = t2.day_voted and t1.id > max(t2.id)
group by t1.id
That should return the list of all ids that we need to delete. Test if before you go further.
4) The very last step is to delete those ids. There are limitations in mysql that make a delete with subquery condition quite uneasy to achieve. See the following SO question for more information on the technical background. You can either use a temporary table to store the selected ids, or try to outsmart mysql by wrapping the subquery and aliasing it. Let us try with the second option :
delete t.* from table t where id in ( select id from (
select t1.id
from
table t1
join (
select id, ip_adress, date(voted) as day_voted
from table
where ip_adress, date(voted) in (
select date(voted), ip_adress
from table
group by date(voted), ip_adress
having count(*) > 60
)
order by id
limit 60
) t2
on t1.ip_adress = t2.ip_adress
and date(t1.voted) = t2.day_voted
and t1.id > max(t2.id)
group by t1.id
) x );
Hope this helps !

You could approach this by vastly simplifying your sample data and using row number simulation for mysql version prior to 8.0 or window function for versions 8.0 or above. I assume you are not on version 8 or above in the following example
drop table if exists t;
create table t(id int auto_increment primary key,ip varchar(2));
insert into t (ip) values
(1),(1),(3),(3),
(2),
(3),(3),(1),(2);
delete t1 from t t1 join
(
select id,rownumber from
(
select t.*,
if(ip <> #p,#r:=1,#r:=#r+1) rownumber,
#p:=ip p
from t
cross join (select #r:=0,#p:=0) r
order by ip,id
)s
where rownumber > 2
) a on a.id = t1.id;
Working in to out the sub query s allocates a row number per ip, sub query a then picks row numbers > 2 and the outer multi-table delete deletes from t joined to a to give
+----+------+
| id | ip |
+----+------+
| 1 | 1 |
| 2 | 1 |
| 3 | 3 |
| 4 | 3 |
| 5 | 2 |
| 9 | 2 |
+----+------+
6 rows in set (0.00 sec)

I had someone help me write the following query, which addressed my question.
SET SQL_SAFE_UPDATES = 0;
create table temp( SELECT id, ip, voted
FROM
(SELECT id, ip, voted,
#ip_rank := IF(#current_ip = ip, #ip_rank + 1, 1) AS ip_rank,
#current_ip := ip
FROM `table_name` where ip in (SELECT ip from `table_name` group by date(voted),ip having count(*) >60)
ORDER BY ip, voted desc
) ranked
WHERE ip_rank <= 2);
DELETE FROM `table_name`
WHERE id not in (select id from temp) and ip in (select ip from temp);
drop table temp;

Related

count of individual column with group by on multiple columns

I have two columns account_number and customer_id. A single customer can have multiple account but a single account can't have multiple customer.
I have dumped a file containing account_num and its corresponding customer_id to db through LOAD DATA INFILE command. Now I am trying to validate through query does any account which has come multiple times in a file has same customer_id or different customer_id in two different rows.
REQUIREMENT : i want to return those accounts which has come multiple times but having diferent customer ids
I tried with group by , but didn't get desired result.
This is my query which is not giving the desired result
SELECT ACCOUNT_NUM,UNIQUE_CUSTOMER_ID,COUNT(UNIQUE_CUSTOMER_ID)
FROM LINKAGE_FILE
GROUP BY ACCOUNT_NUM, UNIQUE_CUSTOMER_ID
HAVING COUNT(ACCOUNT_NUM) > 1 AND COUNT(UNIQUE_CUSTOMER_ID) = 1;
Hope I am clear.
You can simply get the count of unique customer ids using COUNT(DISTINCT..) for every account_num and filter out those cases where count is more than 1, inside the HAVING clause:
SELECT
ACCOUNT_NUM,
COUNT(DISTINCT CUSTOMER_ID) AS unique_customer_count
FROM LINKAGE_FILE
GROUP BY ACCOUNT_NUM
HAVING unique_customer_count > 1
Drop the customer check into a join query like so
DROP TABLE if exists t;
create table t(accountid int,cid int);
insert into t values
(1,1),(1,2).(1,1),(2,3),(3,4),(3,4);
select distinct t.accountid,t.cid
from t
join
(
select accountid,count(distinct cid) cids
from t
group by accountid having cids > 1
) s on s.accountid = t.accountid;
+-----------+------+
| accountid | cid |
+-----------+------+
| 1 | 1 |
| 1 | 2 |
+-----------+------+
2 rows in set (0.00 sec)
You can use EXISTS :
SELECT lf.*
FROM LINKAGE_FILE lf
WHERE EXISTS (SELECT 1 FROM LINKAGE_FILE lf1 WHERE lf1.ACCOUNT_NUM = lf.ACCOUNT_NUM AND lf1.UNIQUE_CUSTOMER_ID <> lf.UNIQUE_CUSTOMER_ID);
However, you can also aggregation with your query :
SELECT ACCOUNT_NUM, COUNT(DISTINCT UNIQUE_CUSTOMER_ID)
FROM LINKAGE_FILE
GROUP BY ACCOUNT_NUM
HAVING COUNT(DISTINCT UNIQUE_CUSTOMER_ID) > 1;
By this, you can get only ACCOUNT_NUMs which have two or more CUSTOMER_IDs.

Is there a simpler way to find MODE(S) of some values in MySQL

MODE is the value that occurs the MOST times in the data, there can be ONE MODE or MANY MODES
here's some values in two tables (sqlFiddle)
create table t100(id int auto_increment primary key, value int);
create table t200(id int auto_increment primary key, value int);
insert into t100(value) values (1),
(2),(2),(2),
(3),(3),
(4);
insert into t200(value) values (1),
(2),(2),(2),
(3),(3),
(4),(4),(4);
right now, to get the MODE(S) returned as comma separated list, I run the below query for table t100
SELECT GROUP_CONCAT(value) as modes,occurs
FROM
(SELECT value,occurs FROM
(SELECT value,count(*) as occurs
FROM
T100
GROUP BY value)T1,
(SELECT max(occurs) as maxoccurs FROM
(SELECT value,count(*) as occurs
FROM
T100
GROUP BY value)T2
)T3
WHERE T1.occurs = T3.maxoccurs)T4
GROUP BY occurs;
and the below query for table t200 (same query just with table name changed) I have 2 tables in this example because to show that it works for cases where there's 1 MODE and where there are multiple MODES.
SELECT GROUP_CONCAT(value) as modes,occurs
FROM
(SELECT value,occurs FROM
(SELECT value,count(*) as occurs
FROM
T200
GROUP BY value)T1,
(SELECT max(occurs) as maxoccurs FROM
(SELECT value,count(*) as occurs
FROM
T200
GROUP BY value)T2
)T3
WHERE T1.occurs = T3.maxoccurs)T4
GROUP BY occurs;
My question is "Is there a simpler way?"
I was thinking like using HAVING count(*) = max(count(*)) or something similar to get rid of the extra join but couldn't get HAVING to return the result i wanted.
UPDATED:
as suggested by #zneak, I can simplify T3 like below:
SELECT GROUP_CONCAT(value) as modes,occurs
FROM
(SELECT value,occurs FROM
(SELECT value,count(*) as occurs
FROM
T200
GROUP BY value)T1,
(SELECT count(*) as maxoccurs
FROM
T200
GROUP BY value
ORDER BY count(*) DESC
LIMIT 1
)T3
WHERE T1.occurs = T3.maxoccurs)T4
GROUP BY occurs;
Now is there a way to get ride of T3 altogether?
I tried this but it returns no rows for some reason
SELECT value,occurs FROM
(SELECT value,count(*) as occurs
FROM t200
GROUP BY `value`)T1
HAVING occurs=max(occurs)
basically I am wondering if there's a way to do it such that I only need to specify t100 or t200 once.
UPDATED: i found a way to specify t100 or t200 only once by adding a variable to set my own maxoccurs like below
SELECT GROUP_CONCAT(CASE WHEN occurs=#maxoccurs THEN value ELSE NULL END) as modes
FROM
(SELECT value,occurs,#maxoccurs:=GREATEST(#maxoccurs,occurs) as maxoccurs
FROM (SELECT value,count(*) as occurs
FROM t200
GROUP BY `value`)T1,(SELECT #maxoccurs:=0)mo
)T2
You are very close with the last query. The following finds one mode:
SELECT value, occurs
FROM (SELECT value,count(*) as occurs
FROM t200
GROUP BY `value`
LIMIT 1
) T1
I think your question was about multiple modes, though:
SELECT value, occurs
FROM (SELECT value, count(*) as occurs
FROM t200
GROUP BY `value`
) T1
WHERE occurs = (select max(occurs)
from (select `value`, count(*) as occurs
from t200
group by `value`
) t
);
EDIT:
This is much easier in almost any other database. MySQL supports neither with nor window/analytic functions.
Your query (shown below) does not do what you think it is doing:
SELECT value, occurs
FROM (SELECT value, count(*) as occurs
FROM t200
GROUP BY `value`
) T1
HAVING occurs = max(occurs) ;
The final having clause refers to the variable occurs but does use max(occurs). Because of the use of max(occurs) this is an aggregation query that returns one row, summarizing all rows from the subquery.
The variable occurs is not using for grouping. So, what value does MySQL use? It uses an arbitrary value from one of the rows in the subquery. This arbitrary value might match, or it might not. But, the value only comes from one row. There is no iteration over it.
I realize this is a very old question but in looking for the best way to find the MODE in a MySQL table, I came up with this:
SELECT [column name], count(*) as [ccount] FROM [table] WHERE [field] = [item] GROUP BY [column name] ORDER BY [ccount] DESC LIMIT 1 ;
In my actual situation, I had a log with recorded events in it. I wanted to know during which period (1, 2 or 3 as recorded in my log) the specific event occurred the most number of times. (Eg, the MODE of "period" column of the table for that specific event
My table looked like this (abridged):
EVENT_TYPE | PERIOD
-------------------------
1 | 3
1 | 3
1 | 3
1 | 2
2 | 1
2 | 1
2 | 1
2 | 3
Using the query:
SELECT event_type, period, count(*) as pcount FROM proto_log WHERE event_type = 1 GROUP BY period ORDER BY pcount DESC LIMIT 1 ;
I get the result:
> EVENT_TYPE | PERIOD | PCOUNT
> --------------------------------------
1 | 3 | 3
Using this result, the period column ($result['period'] for example) should contain the MODE for that query and of course pcount contains the actual count.
If you wanted to get multiple modes, I suppse you could keep adding other criteria to your WHERE clause using ORs:
SELECT event_type, period, count(*) as pcount FROM proto_log WHERE event_type = 1 ***OR event_type = 2*** GROUP BY period ORDER BY pcount DESC LIMIT 2 ;
The multiple ORs should give you the additional results and the LIMIT increase will add the additional MODES to the results. (Otherwise it will still only show the top 1 result)
Results:
EVENT_TYPE | PERIOD | PCOUNT
--------------------------------------
1 | 3 | 3
2 | 1 | 3
I am not 100% sure this is doing exactly what I think it is doing, or if it will work in all situations, so please let me know if I am on or off track here.

select 2nd row of every ID in mysql

I have a table :
ID | time
1 | 300
1 | 100
1 | 200
2 | 200
2 | 500
I want to get 2nd row for every ID
I know that I can get 1st row as
select ID,time from T group by ID;
But I don't know about how to get 2nd row for every ID.
I know about limit and offset clause in mysql, but can't figure out how to use them here.
How can I do it ?
EDIT : Actually, time is not ordered. I forgot to specify that. I have made an edit in the table.
i have just an idee how to make it but i couldnt fix it , maybe you can fix it. any suggest is appreciated to correct my query
first this to select the first row of each id.
SELECT min(id) id
FROM TableName t2
group by id
then select the min(id) which are not in the first query to select to min(id) (which is second row)
like that
SELECT min(id) id ,time
FROM TableName
WHERE id NOT IN (
SELECT min(id) id
FROM TableName
GROUP BY id
)
GROUP BY id
** as i said its just suggest . it returns me 0 values.if u fix it let me edit my post to be helpful
here a demo
SELECT ID, MAX(time) time
FROM
(
select ID, Time
from TableName a
where
(
select count(*)
from TableName as f
where f.ID = a.ID and f.time <= a.time
) <= 2
) s
GROUP BY ID
SQLFiddle Demo
SELECT x.*
FROM test x
JOIN test y
ON y.id = x.id
AND y.time >= x.time
GROUP
BY id,time
HAVING COUNT(*) = n;
Note that any entries with less than n results will be omitted
You cannot do this with the tables that you have. You could make a valiant attempt with:
select id, time
from (select id, time
from t
group by t
) t
where not exists (select 1 from t t2 where t2.id = t.id and t2.time = t.time)
group by id
That is, attempt to filter out the first row.
The reason this is not possible is because tables are inherently unordered, so there is not real definition of "second" in your tables. This gives the SQL engine the opportunity to rearrange the rows as it sees fit during processing -- which can result in great performance gains.
Even the construct that you are using:
select id, time
from t
group by id
is not guaranteed to return time from the first row. This is a (mis)feature of MySQL called Hidden Columns. It is really only intended for the case where all the values are the same. I will admit that in practice it seems to get the value from the first row, but you cannot guarantee that.
Probably your best solution is to select the data into a new table that has an auto-incrementing column:
create table newtable (
autoid int auto_increment,
id int,
time int
);
insert into newtable(id, time)
select id, time from t;
In practice, this will probably keep the same order as the original table, and you can then use the autoid to get the second row. I want to emphasize, though, the "in practice". There is no guarantee that the values are in the correct order, but they probably will be.

MySQL sorting by date with GROUP BY

My table titles looks like this
id |group|date |title
---+-----+--------------------+--------
1 |1 |2012-07-26 18:59:30 | Title 1
2 |1 |2012-07-26 19:01:20 | Title 2
3 |2 |2012-07-26 19:18:15 | Title 3
4 |2 |2012-07-26 20:09:28 | Title 4
5 |2 |2012-07-26 23:59:52 | Title 5
I need latest result from each group ordered by date in descending order. Something like this
id |group|date |title
---+-----+--------------------+--------
5 |2 |2012-07-26 23:59:52 | Title 5
2 |1 |2012-07-26 19:01:20 | Title 2
I tried
SELECT *
FROM `titles`
GROUP BY `group`
ORDER BY MAX( `date` ) DESC
but I'm geting first results from groups. Like this
id |group|date |title
---+-----+--------------------+--------
3 |2 |2012-07-26 18:59:30 | Title 3
1 |1 |2012-07-26 19:18:15 | Title 1
What am I doing wrong?
Is this query going to be more complicated if I use LEFT JOIN?
This page was very helpful to me; it taught me how to use self-joins to get the max/min/something-n rows per group.
In your situation, it can be applied to the effect you want like so:
SELECT * FROM
(SELECT group, MAX(date) AS date FROM titles GROUP BY group)
AS x JOIN titles USING (group, date);
I found this topic via Google, looked like I had the same issue.
Here's my own solution if, like me, you don't like subqueries :
-- Create a temporary table like the output
CREATE TEMPORARY TABLE titles_tmp LIKE titles;
-- Add a unique key on where you want to GROUP BY
ALTER TABLE titles_tmp ADD UNIQUE KEY `group` (`group`);
-- Read the result into the tmp_table. Duplicates won't be inserted.
INSERT IGNORE INTO titles_tmp
SELECT *
FROM `titles`
ORDER BY `date` DESC;
-- Read the temporary table as output
SELECT *
FROM titles_tmp
ORDER BY `group`;
It has a way better performance. Here's how to increase speed if the date_column has the same order as the auto_increment_one (you then don't need an ORDER BY statement) :
-- Create a temporary table like the output
CREATE TEMPORARY TABLE titles_tmp LIKE titles;
-- Add a unique key on where you want to GROUP BY
ALTER TABLE titles_tmp ADD UNIQUE KEY `group` (`group`);
-- Read the result into the tmp_table, in the natural order. Duplicates will update the temporary table with the freshest information.
INSERT INTO titles_tmp
SELECT *
FROM `titles`
ON DUPLICATE KEY
UPDATE `id` = VALUES(`id`),
`date` = VALUES(`date`),
`title` = VALUES(`title`);
-- Read the temporary table as output
SELECT *
FROM titles_tmp
ORDER BY `group`;
Result :
+----+-------+---------------------+---------+
| id | group | date | title |
+----+-------+---------------------+---------+
| 2 | 1 | 2012-07-26 19:01:20 | Title 2 |
| 5 | 2 | 2012-07-26 23:59:52 | Title 5 |
+----+-------+---------------------+---------+
On large tables this method makes a significant point in terms of performance.
Well, if dates are unique in a group this would work (if not, you'll see several rows that match the max date in a group). (Also, bad naming of columns, 'group', 'date' might give you syntax errors and such specially 'group')
select t1.* from titles t1, (select group, max(date) date from titles group by group) t2
where t2.date = t1.date
and t1.group = t2.group
order by date desc
Another approach is to make use of MySQL user variables to identify a "control break" in the group values.
If you can live with an extra column being returned, something like this will work:
SELECT IF(s.group = #prev_group,0,1) AS latest_in_group
, s.id
, #prev_group := s.group AS `group`
, s.date
, s.title
FROM (SELECT t.id,t.group,t.date,t.title
FROM titles t
ORDER BY t.group DESC, t.date DESC, t.id DESC
) s
JOIN (SELECT #prev_group := NULL) p
HAVING latest_in_group = 1
ORDER BY s.group DESC
What this is doing is ordering all the rows by group and by date in descending order. (We specify DESC on all the columns in the ORDER BY, in case there is an index on (group,date,id) that MySQL can do a "reverse scan" on. The inclusion of the id column gets us deterministic (repeatable) behavior, in the case when there are more than one row with the latest date value.) That's the inline view aliased as s.
The "trick" we use is to compare the group value to the group value from the previous row. Whenever we have a different value, we know that we are starting a "new" group, and that this row is the "latest" row (we have the IF function return a 1). Otherwise (when the group values match), it's not the latest row (and we have the IF function returns a 0).
Then, we filter out all the rows that don't have that latest_in_group set as a 1.
It's possible to remove that extra column by wrapping that query (as an inline view) in another query:
SELECT r.id
, r.group
, r.date
, r.title
FROM ( SELECT IF(s.group = #prev_group,0,1) AS latest_in_group
, s.id
, #prev_group := s.group AS `group`
, s.date
, s.title
FROM (SELECT t.id,t.group,t.date,t.title
FROM titles t
ORDER BY t.group DESC, t.date DESC, t.id DESC
) s
JOIN (SELECT #prev_group := NULL) p
HAVING latest_in_group = 1
) r
ORDER BY r.group DESC
If your id field is an auto-incrementing field, and it's safe to say that the highest value of the id field is also the highest value for the date of any group, then this is a simple solution:
SELECT b.*
FROM (SELECT MAX(id) AS maxid FROM titles GROUP BY group) a
JOIN titles b ON a.maxid = b.id
ORDER BY b.date DESC
Use the below mysql query to get latest updated/inserted record from table.
SELECT * FROM
(
select * from `titles` order by `date` desc
) as tmp_table
group by `group`
order by `date` desc
Use the following query to get the most recent record from each group
SELECT
T1.* FROM
(SELECT
MAX(ID) AS maxID
FROM
T2
GROUP BY Type) AS aux
INNER JOIN
T2 AS T2 ON T1.ID = aux.maxID ;
Where ID is your auto increment field and Type is the type of records, you wanted to group by.
MySQL uses an dumb extension of GROUP BY which is not reliable if you want to get such results therefore, you could use
select id, group, date, title from titles as t where id =
(select id from titles where group = a.group order by date desc limit 1);
In this query, each time the table is scanned full for each group so it can find the most recent date. I could not find any better alternate for this. Hope this will help someone.

Find duplicates or more in Mysql, delete them except the first one input

I have a table with rows like id, length, time and some of them are duplicates, where length and time is the same in some rows. I want to delete all copies of the first row submitted.
id | length | time
01 | 255232 | 1242
02 | 255232 | 1242 <- Delete that one
I have this to show all duplicates in table.
SELECT idgarmin_track, length , time
FROM `80dage_garmin_track`
WHERE length in
( SELECT length
FROM `80dage_garmin_track`
GROUP
BY length
HAVING count(*) > 1 )
ORDER BY idgarmin_track, length, time LIMIT 0,500
DELETE FROM `80dage_garmin_track` t1
WHERE EXISTS (SELECT 1 from `80dage_garmin_track` t2
WHERE t1.Length = t2.Length
AND t1.Time = t2.Time
AND t1.idgarmin_track > t2.idgarmin_track)
If you can take your table offline for a period, then the simplest way is to build a new table containing the data you want and then drop the original table:
create table `80dage_garmin_track_un` like `80dage_garmin_track`;
insert into `80dage_garmin_track_un`
select min(idgarmin_track), length, time
group by length, time;
rename table `80dage_garmin_track` to old, `80dage_garmin_track_un` to `80dage_garmin_track`;
drop table old;
i have the same problem Holsteinkaa, i just use it like this:
delete from table where id in ( select * from (
SELECT id FROM table t1
WHERE EXISTS (SELECT 1 from table t2
WHERE t1.field = t2.field
AND t1.id > t2.id
)
) as tmp )
i was trying to put this like a comment to Michael Pakhantsov answer but i cant :/ sorry