I'm trying to get a total count of all the rows in a table which have duplicates.
Here is the query I am using right now:
SELECT count( `id_lead` )
FROM `lead_history`
GROUP BY `id_lead`
HAVING count( * ) > 1
The problem is that this isn't counting the total number of rows, it is counting the total amount of times each row contains a duplicate and returning that.
So right now, it is returning like this:
2
4
6
2
Given those results, I actually want my query to return 4, since there are 4 rows which have duplicates. The amount of duplicates each row has does not matter to me.
How would I write this query without using subqueries?
If you were using something like Oracle or SQL Server, you could take advantage of analytic functions. But MySQL does not have these, so using a subquery might the best way to get your answer.
SELECT COUNT(*)
FROM `lead_history`
WHERE `id_lead` IN (SELECT `id_lead` FROM `lead_history` GROUP BY `id_lead` HAVING COUNT(*) > 1)
Note that the subquery is not correlated, so the query optimizer should be able to make this run reasonably efficiently.
Try this way
SELECT COUNT(*)
FROM `lead_history` as b
join (SELECT `id_lead` FROM `lead_history` GROUP BY `id_lead` HAVING COUNT(*) > 1) a on b.id_lead=a.id_lead
Try This
SELECT count( * )
FROM `lead_history`
GROUP BY `id_lead`
HAVING count( * ) > 1
You're almost there, just put your query in a derived table and select the count from that. No additional in or join needed:
select count(*) from (
select `id_lead`
from `lead_history`
group by `id_lead`
having count(*) > 1
) t
This following query take 1-2 seconds for querying.
SELECT updated, COUNT( * ) count
FROM v2_subscription
WHERE ss_id IN (SELECT MAX(ss_id) ss_id FROM v2_subscription GROUP BY uid, card_id)
while the subquery do take only few milliseconds.
SELECT MAX(ss_id) ss_id FROM v2_subscription GROUP BY uid, card_id
I do have index on uid, card_id and both uid, card_id
It's my sql and i have no idea how to optimize this.
Please advise,
Try this, May be it would help, let me know, if it does.
SELECT a.updated, COUNT( * ) count
FROM v2_subscription a
inner join v2_subscription b
on a.ss_id = max(b.ss_id)
GROUP BY b.uid, b.card_id
Or perhaps this
SELECT a.updated, COUNT( * ) count
FROM v2_subscription a
inner join v2_subscription b
on a.ss_id = (SELECT MAX(b.ss_id) b.ss_id FROM v2_subscription b GROUP BY b.uid, b.card_id)
Finally i have found the solution beside #arkumar above answer.
Adding "ORDER BY ss_id" inside the subquery also do the trick
Since without order by, the result of subquery do not have index.
I have the following query working
SELECT newTable.Score, COUNT(1) AS Total, COUNT(1) / t.count * 100 AS `Frequency`
FROM mytable newTable
CROSS JOIN (SELECT COUNT(1) AS count FROM mytable) t
GROUP BY newTable.Score
ORDER BY Frequency DESC
However, two things I don't understand from the MySQL docs:
1) I don't understand why there isn't a comma, or a join type, specified in the from clause.
Reading the MySQL docs, this seems necessary.
2) What does the 't' represent in the CROSS JOIN clause?
Any advice appreciated.
The t is the same as the newTable - it is an alias name for the table and the temporary table that the subquery builds.
It is easier to read when the optional as keyword is used
SELECT newTable.Score, COUNT(1) AS Total, COUNT(1) / t.count * 100 AS `Frequency`
FROM mytable as newTable
CROSS JOIN (SELECT COUNT(1) AS count FROM mytable) as t
GROUP BY newTable.Score
ORDER BY Frequency DESC
An alias name replaces the original name of the table with a new one to be used in your query. And you need to give subqueries a name to refer to them in your query too.
I'm creating custom views that show totals for different things in a database, and I'd like to also show the differences.
For example;
SELECT
(SELECT COUNT(*) FROM `documents`) AS `doc_count`,
(SELECT COUNT(*) FROM `contacts`) AS `user_count`,
(`doc_count` - `user_count`) AS `difference`;
I get an error using the aliases this way. Is there a way to write this query without repeating select count(*) queries?
You could wrap both queries with an additional query:
SELECT doc_count, user_count, doc_count - user_count AS difference
FROM ((SELECT COUNT(*) FROM `documents`) AS doc_count,
(SELECT COUNT(*) FROM `contacts`) AS user_count) t
No you can't use the aliases at same level of query you have to use the whole expression or use sub select
SELECT
(SELECT COUNT(*) FROM `documents`) AS `doc_count`,
(SELECT COUNT(*) FROM `contacts`) AS `user_count`,
((SELECT COUNT(*) FROM `documents`) - (SELECT COUNT(*) FROM `contacts`)) AS `difference`;
Here is a "workaround" to get the result you're looking for:
SELECT C.doc_count
,C.user_count
,C.doc_count - C.user_count AS `difference`
FROM (SELECT
(SELECT COUNT(*) FROM `documents`) AS `doc_count`
,(SELECT COUNT(*) FROM `contacts`) AS `user_count`) C
But i'm not sure about the performance of such kind of query...
Hope this will help you
I would move these to the from clause and use cross join:
SELECT d.doc_count, u.user_count, (d.doc_count - u.user_count) as difference
FROM (SELECT COUNT(*) as doc_count FROM `documents`) d CROSS JOIN
(SELECT COUNT(*) as user_count FROM `contacts`) u;
I've got a couple of duplicates in a database that I want to inspect, so what I did to see which are duplicates, I did this:
SELECT relevant_field
FROM some_table
GROUP BY relevant_field
HAVING COUNT(*) > 1
This way, I will get all rows with relevant_field occuring more than once. This query takes milliseconds to execute.
Now, I wanted to inspect each of the duplicates, so I thought I could SELECT each row in some_table with a relevant_field in the above query, so I did like this:
SELECT *
FROM some_table
WHERE relevant_field IN
(
SELECT relevant_field
FROM some_table
GROUP BY relevant_field
HAVING COUNT(*) > 1
)
This turns out to be extreeeemely slow for some reason (it takes minutes). What exactly is going on here to make it that slow? relevant_field is indexed.
Eventually I tried creating a view "temp_view" from the first query (SELECT relevant_field FROM some_table GROUP BY relevant_field HAVING COUNT(*) > 1), and then making my second query like this instead:
SELECT *
FROM some_table
WHERE relevant_field IN
(
SELECT relevant_field
FROM temp_view
)
And that works just fine. MySQL does this in some milliseconds.
Any SQL experts here who can explain what's going on?
The subquery is being run for each row because it is a correlated query. One can make a correlated query into a non-correlated query by selecting everything from the subquery, like so:
SELECT * FROM
(
SELECT relevant_field
FROM some_table
GROUP BY relevant_field
HAVING COUNT(*) > 1
) AS subquery
The final query would look like this:
SELECT *
FROM some_table
WHERE relevant_field IN
(
SELECT * FROM
(
SELECT relevant_field
FROM some_table
GROUP BY relevant_field
HAVING COUNT(*) > 1
) AS subquery
)
Rewrite the query into this
SELECT st1.*, st2.relevant_field FROM sometable st1
INNER JOIN sometable st2 ON (st1.relevant_field = st2.relevant_field)
GROUP BY st1.id /* list a unique sometable field here*/
HAVING COUNT(*) > 1
I think st2.relevant_field must be in the select, because otherwise the having clause will give an error, but I'm not 100% sure
Never use IN with a subquery; this is notoriously slow.
Only ever use IN with a fixed list of values.
More tips
If you want to make queries faster,
don't do a SELECT * only select
the fields that you really need.
Make sure you have an index on relevant_field to speed up the equi-join.
Make sure to group by on the primary key.
If you are on InnoDB and you only select indexed fields (and things are not too complex) than MySQL will resolve your query using only the indexes, speeding things way up.
General solution for 90% of your IN (select queries
Use this code
SELECT * FROM sometable a WHERE EXISTS (
SELECT 1 FROM sometable b
WHERE a.relevant_field = b.relevant_field
GROUP BY b.relevant_field
HAVING count(*) > 1)
SELECT st1.*
FROM some_table st1
inner join
(
SELECT relevant_field
FROM some_table
GROUP BY relevant_field
HAVING COUNT(*) > 1
)st2 on st2.relevant_field = st1.relevant_field;
I've tried your query on one of my databases, and also tried it rewritten as a join to a sub-query.
This worked a lot faster, try it!
I have reformatted your slow sql query with www.prettysql.net
SELECT *
FROM some_table
WHERE
relevant_field in
(
SELECT relevant_field
FROM some_table
GROUP BY relevant_field
HAVING COUNT ( * ) > 1
);
When using a table in both the query and the subquery, you should always alias both, like this:
SELECT *
FROM some_table as t1
WHERE
t1.relevant_field in
(
SELECT t2.relevant_field
FROM some_table as t2
GROUP BY t2.relevant_field
HAVING COUNT ( t2.relevant_field ) > 1
);
Does that help?
Subqueries vs joins
http://www.scribd.com/doc/2546837/New-Subquery-Optimizations-In-MySQL-6
Try this
SELECT t1.*
FROM
some_table t1,
(SELECT relevant_field
FROM some_table
GROUP BY relevant_field
HAVING COUNT (*) > 1) t2
WHERE
t1.relevant_field = t2.relevant_field;
Firstly you can find duplicate rows and find count of rows is used how many times and order it by number like this;
SELECT q.id,q.name,q.password,q.NID,(select count(*) from UserInfo k where k.NID= q.NID) as Count,
(
CASE q.NID
WHEN #curCode THEN
#curRow := #curRow + 1
ELSE
#curRow := 1
AND #curCode := q.NID
END
) AS No
FROM UserInfo q,
(
SELECT
#curRow := 1,
#curCode := ''
) rt
WHERE q.NID IN
(
SELECT NID
FROM UserInfo
GROUP BY NID
HAVING COUNT(*) > 1
)
after that create a table and insert result to it.
create table CopyTable
SELECT q.id,q.name,q.password,q.NID,(select count(*) from UserInfo k where k.NID= q.NID) as Count,
(
CASE q.NID
WHEN #curCode THEN
#curRow := #curRow + 1
ELSE
#curRow := 1
AND #curCode := q.NID
END
) AS No
FROM UserInfo q,
(
SELECT
#curRow := 1,
#curCode := ''
) rt
WHERE q.NID IN
(
SELECT NID
FROM UserInfo
GROUP BY NID
HAVING COUNT(*) > 1
)
Finally, delete dublicate rows.No is start 0. Except fist number of each group delete all dublicate rows.
delete from CopyTable where No!= 0;
sometimes when data grow bigger mysql WHERE IN's could be pretty slow because of query optimization. Try using STRAIGHT_JOIN to tell mysql to execute query as is, e.g.
SELECT STRAIGHT_JOIN table.field FROM table WHERE table.id IN (...)
but beware: in most cases mysql optimizer works pretty well, so I would recommend to use it only when you have this kind of problem
This is similar to my case, where I have a table named tabel_buku_besar. What I need are
Looking for record that have account_code='101.100' in tabel_buku_besar which have companyarea='20000' and also have IDR as currency
I need to get all record from tabel_buku_besar which have account_code same as step 1 but have transaction_number in step 1 result
while using select ... from...where....transaction_number in (select transaction_number from ....), my query running extremely slow and sometimes causing request time out or make my application not responding...
I try this combination and the result...not bad...
`select DATE_FORMAT(L.TANGGAL_INPUT,'%d-%m-%y') AS TANGGAL,
L.TRANSACTION_NUMBER AS VOUCHER,
L.ACCOUNT_CODE,
C.DESCRIPTION,
L.DEBET,
L.KREDIT
from (select * from tabel_buku_besar A
where A.COMPANYAREA='$COMPANYAREA'
AND A.CURRENCY='$Currency'
AND A.ACCOUNT_CODE!='$ACCOUNT'
AND (A.TANGGAL_INPUT BETWEEN STR_TO_DATE('$StartDate','%d/%m/%Y') AND STR_TO_DATE('$EndDate','%d/%m/%Y'))) L
INNER JOIN (select * from tabel_buku_besar A
where A.COMPANYAREA='$COMPANYAREA'
AND A.CURRENCY='$Currency'
AND A.ACCOUNT_CODE='$ACCOUNT'
AND (A.TANGGAL_INPUT BETWEEN STR_TO_DATE('$StartDate','%d/%m/%Y') AND STR_TO_DATE('$EndDate','%d/%m/%Y'))) R ON R.TRANSACTION_NUMBER=L.TRANSACTION_NUMBER AND R.COMPANYAREA=L.COMPANYAREA
LEFT OUTER JOIN master_account C ON C.ACCOUNT_CODE=L.ACCOUNT_CODE AND C.COMPANYAREA=L.COMPANYAREA
ORDER BY L.TANGGAL_INPUT,L.TRANSACTION_NUMBER`
I find this to be the most efficient for finding if a value exists, logic can easily be inverted to find if a value doesn't exist (ie IS NULL);
SELECT * FROM primary_table st1
LEFT JOIN comparision_table st2 ON (st1.relevant_field = st2.relevant_field)
WHERE st2.primaryKey IS NOT NULL
*Replace relevant_field with the name of the value that you want to check exists in your table
*Replace primaryKey with the name of the primary key column on the comparison table.
It's slow because your sub-query is executed once for every comparison between relevant_field and your IN clause's sub-query. You can avoid that like so:
SELECT *
FROM some_table T1 INNER JOIN
(
SELECT relevant_field
FROM some_table
GROUP BY relevant_field
HAVING COUNT(*) > 1
) T2
USING(relevant_field)
This creates a derived table (in memory unless it's too large to fit) as T2, then INNER JOIN's it with T1. The JOIN happens one time, so the query is executed one time.
I find this particularly handy for optimising cases where a pivot is used to associate a bulk data table with a more specific data table and you want to produce counts of the bulk table based on a subset of the more specific one's related rows. If you can narrow down the bulk rows to <5% then the resulting sparse accesses will generally be faster than a full table scan.
ie you have a Users table (condition), an Orders table (pivot) and LineItems table (bulk) which references counts of Products. You want the sum of Products grouped by User in PostCode '90210'. In this case the JOIN will be orders of magnitude smaller than when using WHERE relevant_field IN( SELECT * FROM (...) T2 ), and therefore much faster, especially if that JOIN is spilling to disk!