suppose that i have the following table in my database called 'central':
id
Name
DB
customer
1
Bob
A1
1
2
Marley
NULL
NULL
3
Irene
A2
2
The customer column references to the 'Customers' table which is inside of another database.
IMPORTANT: the database names A1 and A2 are only examples. There will be hundreds of separate databases containing a Customers table.
The central.DB column is a reference to which database to use when selecting the CustomerName from inside of the select statement that you see below the following example databases.
An example from the Customers table inside of Database 'A1':
id
Customer
CustomerName
1
C001
Asta housing
2
C002
Jack's
An example from the Customers table inside of Database 'A2':
id
Customer
CustomerName
1
D900
Mo's
2
D901
Humpries paints
I can't figure out how to do something like this and maybe it's not possible at all:
SELECT Central.id, Central.Name, (central.DB).Customers.CustomerName
FROM central, (central.DB).Customers.
Or maybe use a join to do this?
Either way, the query result should should be:
id
Name
CustomerName
1
Bob
Asta housing
2
Marley
NULL
3
Irene
Humpries paints
Maybe this isn't possible and the result is that i must do it with php? But i'd rather not do this as central.DB could reference potentially hundreds of databases and that will possibly be rather expensive (a lot of calls to the referenced databases).
(I can't change the setup of using a central database and the other separate databases)
Is it possible to reference to another database(table) when the name of that database is inside of the result set?
If both databases are at the same server(MySQL Instance) you can just prefix the tables with the database name.
SELECT C.id, C.Name, CU.CustomerName FROM database_1.Central AS C LEFT JOIN database_2.Customers AS CU
Make sure that the user logged in has permission to access both databases.
SELECT t0.id,
t0.name,
COALESCE(t1.CustomerName, t2.CustomerName) CustomerName
FROM my_database.central t0
LEFT JOIN A1.Customers t1 ON (t0.customer, t0.DB) = (t1.ID, 'DB1')
LEFT JOIN A2.Customers t2 ON (t0.customer, t0.DB) = (t2.ID, 'DB2')
This question already has an answer here:
How can I use FIND_IN_SET within a JOINED table?
(1 answer)
Closed 5 years ago.
Got 2 tables - contacts and messages:
contact_id | contact_email
1 | some#mail.com
2 | other#mail.com
3 | no#nono.com
message_id | message_recipients
1 | 1,2,3
2 | 3
message_recipients field contains ID(s) of contact(s) message was assigned to. Each message can have one or more IDs assigned, so they are separated by , symbol.
I need to show all contacts, and count of messages are assigned to each contact. Since message_recipients field may contain multiple IDs, I can't run a query like SELECT * FROM contacts, messages WHERE contacts.contact_id=messages.message_recipients because it won't work properly.
If I run SELECT * FROM contacts FULL JOIN messages, it returns many duplicated rows from contacts table. Sure thing, I can run SELECT * FROM contacts FULL JOIN messages GROUP BY contact_id, but this one returns only 1st message from messages table.
I know that in order to count how many messages each contact has assigned to, I will probably need to explode message_recipients field from each row into array and use code like if (in_array($contact_id, $message_recipients_array)) {$total++;} or similar. Now my main concern is how to all I need by writing as simple query as possible.
Fix your table structure. Do not store multiple values in one cell. See Normalization
For now, you can use FIND_IN_SET:
select c.contact_id,
c.contact_email,
count(*) no_of_messages
from messages m
join contacts c on find_in_set(c.contact_id, m.message_recipients) > 0
group by c.contact_id,
c.contact_email
But this will be slow as it can't use any index on the contact_id or message_recipients.
To actually fix the issues, don't include recipient_id in the messages table.
You should have stored single recipient in one row in a separate mapping table with many to many relation with (maybe) the following structure.
messages_recipients (
id int PK,
message_id int FK referring message(message_id),
message_recipient_id int FK referring contacts(contact_id)
)
Then all you had to do was:
select c.contact_id,
c.contact_email,
count(*) no_of_messages
from messages_recipients m
join contacts c on c.contact_id = m.message_recipient_id
group by c.contact_id,
c.contact_email
This query is Sargable and will be faster.
Fix your data structure! Storing ids in strings is a really bad idea. Why?
Numbers should be stored as numbers not strings.
SQL does not offer very good string functions.
Foreign key constraints should be properly expressed.
The query optimizer cannot use indexes or partitions.
SQL has a great method for storing lists: it is called a "table".
Sometimes, we are stuck with other people's really, really bad design decisions. MySQL does offer a method for doing what you want, find_in_set(). This is a hack to get around the short-comings of a bad data layout:
select . . .
from contacts c join
messages m
on find_in_set(c.contact_id, m.message_recipients) > 0
Im having the following problem:
I try to implement an achievementsystem. I have two tables. Table 1 contains the achievement_id and achievement_info. Table 2 contains the link to the user, meaning achievement_id and player_id, so that you can tell which user has achieved certain things.
I'm trying to write a method that returns me all achievements, but additionally a flag that tells me if a certain user has achieved this row or not.
E.g.: getPlayerAchievements(playerid) --> returns a list of Achievements with id, info, and a bool flag whether the user has achieved it.
table 1:
achievement_id|achievement_info
1 |info1
2 |info2
3 |info3
table 2:
achievement_id|player_id;
1 |15
3 |15
the result I need by entering the player_id "15":
achievement_id|achievement_info|(bool)achieved
1 |info1 |true
2 |info2 |false
3 |info3 |true
I already have the achievement class so I just have to fill them with my data.
I could always use two seperate sql queries to achieve that, but I thought maybe there was a way to simplify it, since I use php to get my data and don't want two connections and queries in one php script.
You want to select all records from the achievemets table and show them. That's the easy part :-) For every record you want to show whether player 1234 has attained this achievement. You can do this with an EXISTS clause:
select
achievement_id,
achievement_info,
exists
(
select *
from players p
where p.player_id = 1234
and p.achievement_id = a.achievement_id
) as achieved
from achievements a;
Or even simpler with IN:
select
achievement_id,
achievement_info,
achievement_id in (select achievement_id from players where player_id = 1234) as achieved
from achievements;
You can use left join to get a complete list of achievements and the matching records from the user's achievements table:
select t1.achievement_id, t1.achievement_info, (t2.achievement_id is null) as achieved
from table1 t1
left join table2 t2 on t1.achievement_id=t2.achievement_id and t2.player_id=15
In a large user database with the following format and sample data, we are trying to identify duplicated people:
id first_name last_name email
---------------------------------------------------
1 chris baker
2 chris baker chris#gmail.com
3 chris baker chris#hotmail.com
4 chris baker crayzyguy#crazy.com
5 carl castle castle#npr.org
6 mike rotch fakeuser#sample.com
I am using the following query:
SELECT
GROUP_CONCAT(id) AS "ids",
CONCAT(UPPER(first_name), UPPER(last_name)) AS "name",
COUNT(*) AS "duplicate_count"
FROM
users
GROUP BY
name
HAVING
duplicate_count > 1
This works great; I get a list of duplicates with the id numbers of the involved rows.
We would re-assign any associated data tied to a duplicate to the actual person (set user_id = 2 where user_id = 3), then we delete the duplicating user row.
The trouble comes after we make this report the first time, as we clean up the list after manually verifying that they are indeed duplicates -- some ARE NOT duplicates. There are 2 Chris Bakers that are legitimate users.
We don't want to keep seeing Chris Baker in subsequent duplicate reports until the end of time, so I am looking for a way to flag that user id 1 and user id 4 are NOT duplicates of each other for future reports, but they could be duplicated by new users added later.
What I tried
I added a is_not_duplicate field to the user table, but then if a new duplicate "Chris Baker" gets added to the database, it will cause this situation to not show on the duplicate report; the is_not_duplicate improperly excludes one of the accounts. My HAVING statement would not meet the > 1 threshold until there are -two- duplicates of Chris Baker, plus the "real" one marked is_not_duplicate.
Question Summed Up
How can I build exceptions into the above query without looping results or multiple queries?
Sub-queries are fine, but the size of the dataset makes every query count and I'd like the solution to be as performant as possible.
Try to add the is_not_duplicate boolean field and modify your code as follows:
SELECT
GROUP_CONCAT(id) AS "ids",
CONCAT(UPPER(first_name), UPPER(last_name)) AS "name",
COUNT(*) AS "duplicate_count",
SUM(is_not_duplicate) AS "real_count"
FROM
users
GROUP BY
name
HAVING
duplicate_count > 1
AND
duplicate_count - real_count > 0
Newly added duplicates will have is_not_duplicate=0 so the real_count for that name will be less than duplicate_count and the row will be shown
My brain is too fried to come up with the actual query for this at the moment, but I might be able to give you a nudge in a path that should work :)
What if you did add another column (maybe a table of valid duplicated users instead?...both will accomplish the same thing), and ran a subquery that would count up all of the valid duplicates and then you could compare against the count in your current query. You would exclude any users that have matching counts, and would pull in any with counts that are higher. Hopefully that makes sense; I will create a use case:
Chris Baker with id 1 and 4 are marked as valid_duplicates
There are 4 Chris Baker's in the system
You get a count of valid Chris Baker's
You get a count of all Chris Baker's
valid_count <> total_count, so return Chris Baker
*You probably can even modify the query so that it does not even list the duplicate id's (even if you get a duplicate marking of only 1 id). Rather than having to re-check which are the valids. This would be a little more complicated. Without it, at least you ignore Chris Baker until another enters the system
I have written up the basic query, dealing with excluding specific id's I will try to roll in tonight. But, this at least solves your initial need. If you do not need the more complicated query, do let me know so that I do not waste my time on it :)
SELECT
GROUP_CONCAT(id) AS "ids",
CONCAT(UPPER(first_name), UPPER(last_name)) AS "name",
COUNT(*) AS "duplicate_count"
FROM
users
WHERE NOT EXISTS
(
SELECT 1
FROM
(
SELECT
CONCAT(UPPER(first_name), UPPER(last_name)) AS "name",
COUNT(*) AS "valid_duplicate_count"
FROM
users
WHERE
is_valid_duplicate = 1 --true
GROUP BY
name
HAVING
valid_duplicate_count > 1
) AS duplicate_users
WHERE
duplicate_users.name = users.name
AND valid_duplicate_count = duplicate_count
)
GROUP BY
name
HAVING
duplicate_count > 1
Below is the query that should do the same as above, but the final list will only print the id's that are not in the valid list. This actually ended up being a lot simpler than I thought. And, it is mostly the same as above, but the only reason I kept above is to keep the two options and in case I messed the above up...it does get complicated as it is many nested queries. If CTE's are available to you, or even temp tables. It might make the query more expressive to break it up into temp tables :). Hopefully this helps and is what you are looking for
SELECT GROUP_CONCAT(id) AS "ids",
CONCAT(UPPER(first_name), UPPER(last_name)) AS "name",
COUNT(*) AS "final_duplicate_count"
--This count could actually be 1 due to the nature of the query
FROM
users
--get the list of duplicated user names
WHERE EXISTS
(
SELECT
CONCAT(UPPER(first_name), UPPER(last_name)) AS "name",
COUNT(*) AS "total_duplicate_count"
FROM
users AS total_dup_users
--ignore valid_users whose count still matches
WHERE NOT EXISTS
(
SELECT 1
FROM
(
SELECT
CONCAT(UPPER(first_name), UPPER(last_name)) AS "name",
COUNT(*) AS "valid_duplicate_count"
FROM
users AS valid_users
WHERE
is_valid_duplicate = 1 --true
GROUP BY
name
HAVING
valid_duplicate_count > 1
) AS duplicate_users
WHERE
--join inner table to outer table
duplicate_users.name = total_dup_users.name
--valid count check
AND valid_duplicate_count = total_duplicate_count
)
--join inner table to outer table
AND total_dup_users.Name = users.Name
GROUP BY
name
HAVING
duplicate_count > 1
)
--ignore users that are valid when doing the actual counts
AND NOT EXISTS
(
SELECT 1
FROM users AS valid
WHERE
--join inner table to outer table
users.name =
CONCAT(UPPER(valid.first_name), UPPER(valid.last_name))
--only valid users
AND valid.is_valid_duplicate = 1 --true
)
GROUP BY
FinalDuplicates.Name
Since this is basically a many-to-many relationship I would add a new table not_duplicate with fields user1 and user2.
I would probably add two rows for each not_duplicate relationship such that I have one row for 2 -> 3 and a symmetric row for 3 -> 2 to ease querying, but that may introduce data inconsistencies so make sure you delete both rows at the same time (or have only one row and make the correct query in your script).
well it seems to me that the is_not_duplicate column is not complex enough to hold the information you want to store - from what I understand you want to manually tell your detection that two distinct users are not duplicates of each other. so either you create a column like is_not_duplicate_of=other-user-id or if you want to keep the possibility open that one user can be manually defined not duplicate of more than one users, you need a seperate table with two user-id columns.
the query telling you the non overridden duplicates probably has to be a bit more complex than the one you suggested, I cannot think of one that works with a group by and having logic. The only thing that would come to my mind is something like
SELECT u1.* FROM users u1
INNER JOIN users u2
ON u1.id <> u2.id
AND u2.name = u1.name
WHERE NOT EXISTS (
SELECT *
FROM users_non_dups un
WHERE (un.id1 = u1.id AND un.id2 = u2.id)
OR (un.id1 = u2.id AND un.id2 = u1.id)
)
If you were to correct all duplicates each time you run the report, then a very simple solution might be to modify the query:
SELECT
GROUP_CONCAT(id) AS "ids",
MAX(id) AS "max_id",
CONCAT(UPPER(first_name), UPPER(last_name)) AS "name",
COUNT(*) AS "duplicate_count"
FROM
users
GROUP BY
name
HAVING
duplicate_count > 1
AND
max_id > MAX_ID_LAST_TIME_DUPLICATE_REPORT_WAS_GENERATED;
I would go ahead and make the "confirmed_unique" column, defaulted as "False."
In order to avoid the problems you mentioned,
Then I would select all elements that may look like duplicates and have a "False" entry for "confirmed_unique."
I am not sure if this will work, but could you consider the reverse logic of adding a *is_duplicate_of* column? That way you can mark duplicates by entering the ID of the first record at this column which will be greater than zero. The records that you wish to retain will have a 0 value at this field. You can set the default (unchecked records) to -1 to keep track of the validation status for each record.
Afterwards you can keep executing an SQL that will compare new records only with correct records having is_duplicate_of = 0 .
If you are ok to make a slight change to the format of the report. You could do a self-join like this -
SELECT
CONCAT(u1.id,",", u2.id) AS "ids",
CONCAT(UPPER(u1.first_name), UPPER(u1.last_name)) AS "name"
FROM
users u1, users u2
WHERE
u1.id < u2.id AND
UPPER(u1.first_name) = UPPER(u2.first_name) AND
UPPER(u1.last_name) = UPPER(u2.last_name) AND
CONCAT(u1.id,",", u2.id) NOT IN (SELECT ids from not_dupe)
which reports duplicates as follows:
ids | name
----|--------
1,2 | CHRISBAKER
1,3 | CHRISBAKER
...
And the not_dupe table would have rows like below:
ids
------
1,2
3,4
...
I think it would make sense to create a lookup-table storing the ids of the ones that are not duplicates. Thus confirmed non duplicants are removed and the query will only have to ad a small look up for duplicates actualy found on the lookup table.
for instance in this example we would have
id 1 | id 2
2 4
if crayzyguy#crazy.com and chris#gmail.com are diffrent persons.
If I were you, I will add some geolocalisation tables/fields to my database schema.
The probability two end-users are having the same names AND are living in the same place is very very low - except in very big town - but you can split geolocalization to small areas too - it's about granularity.
Good luck.
I would suggest you to create a couple of things:
A Boolean column to flag confirmed users
A String column to save ids
A trigger that will check if the first name and last name are already there to fill up the flag, and save in the string column all ids to which this one is a possible duplicate.
And then build a report that looks for duplicated true and decode the string field to match the possible duplicated
I gave Justin Pihony +1 as the 1st to suggest comparing the duplicate count with the not duplicate count, and Hrant Khachatrian +1 for being the 1st to show an efficient way of doing that.
Here is a slightly different method, plus some renaming to make everything a bit more self explanatory, plus some extra columns in the query to make it obvious which records need to be compared as potential duplicates.
I would call the new column "CONFIRMED_UNIQUE" instead of "IS_NOT_DUPLICATE". Like Hrant I would make it Boolean (tinyint(1) with 0=FALSE and 1=TRUE).
The "potential_duplicate_count" is the maximum number of records that would have to be deleted.
select
group_concat(case when not confirmed_unique then id end) as potential_duplicate_ids,
group_concat(case when confirmed_unique then id end) as confirmed_unique_ids,
concat(upper(first_name), upper(last_name)) as name,
sum( case when not confirmed_unique then 1 end ) - (not max(confirmed_unique)) as potential_duplicate_count
from
users
group by
name
having
potential_duplicate_count > 0
I see someone else has been voted down for the suggestion of merging, but nothing about your problem statement says the data needs to be inplace. The OP followed up with their solution which happens to be a put SQL one, that doesn't imply that every solution needs to be limited to that.
The issue as I understand is around contacts having multiple, similar, but not necessarily identical records in your database, which has cost and reputational implications so you're looking to deduplicate these records.
I would write a batch job that searches for potential duplicates (this can be as complicated or as simple as you like) and then close the two records that it finds are dupes and create a new record.
To enable that you'd need four new columns:
Status, which would be either Open, Merged, Split
RelatedId, which would hold the value of who the record was merged with
ChainId, the new record Id
DateStatusChanged, obvious enough
Open would be the default status
Merged would be when the record is merged (effectively closed and replaced)
Split would be if the merge was reversed
So, as an example, go through all of the records that, for example, have the same name. Merge them in pairs. So if you have three Chris Bakers, records 1, 2 and 3, merge 1 and 2 to make record 4 and then 3 and 4 to make record 5. Your table would end up something like:
ID NAME STATUS RELATEDID CHAINID DATESTATUSCHANGED [other rows omitted]
1 Chris Baker MERGED 2 4 27-AUG-2012
2 Chris Baker MERGED 1 4 27-AUG-2012
3 Chris Baker MERGED 4 5 28-AUG-2012
4 Chris Baker MERGED 3 5 28-AUG-2012
5 Chris Baker OPEN
This way you have a full record of what has happened to your data can reverse any changes by unmerging, if for example contacts 1 and 2 weren't the same you reverse the merge of 3 and 4, reverse the merge of 1 and 2, you'd end up with this:
ID NAME STATUS RELATEDID CHAINID DATESTATUSCHANGED
1 Chris Baker SPLIT 2 4 29-AUG-2012
2 Chris Baker SPLIT 1 4 29-AUG-2012
3 Chris Baker SPLIT 4 5 29-AUG-2012
4 Chris Baker CLOSED 3 5 29-AUG-2012
5 Chris Baker CLOSED 29-AUG-2012
You could then manually merge, as you'd probably not want your job to automatically remerge split records.
Is there a good reason for not merging duplicate accounts into a single account?
From the comments, it seems like the information is being used mostly for contact information so merging should be relatively painless and low risk. Once you merge users they will no longer appear in your duplicate report. Furthermore, you users table will actually shrink which could help with performance.
Add is_not_duplicate by datatype bit to your table and use below query after set is_not_duplicate data value:
SELECT GROUP_CONCAT(id) AS "ids",
CONCAT(UPPER(first_name), UPPER(last_name)) AS "name"
FROM users
GROUP BY name
HAVING COUNT(*) > SUM(CAST(is_not_duplicate AS INT))
above query compare total duplicate rows by total valid duplicate rows.
Why don't you make the email column to be a unique identifier in this case, and after you cleanse your records once, you do not allow duplicates from there onwards?
I've got 3 tables that are something like this (simplified here ofc):
users
user_id
user_name
info
info_id
user_id
rate
contacts
contact_id
user_id
contact_data
users has a one-to-one relationship with info, although info doesn't always have a related entry.
users has a one-to-many relationship with contacts, although contacts doesn't always have related entries.
I know I can grab the proper 'users' + 'info' with a left join, is there a way to get all the data I want at once?
For example, one returned record might be:
user_id: 5
user_name: tom
info_id: 1
rate: 25.00
contact_id: 7
contact_data: 555-1212
contact_id: 8
contact_data: 555-1315
contact_id: 9
contact_data: 555-5511
Is this possible with a single query? Or must I use multiple?
It is possible to do what you're asking in one query, but you'd either need a variable number of columns which is evil because SQL isn't designed for that, or you'd have to have a fixed number of columns, which is even more evil because there is no sensible fixed number of columns you could choose.
I'd suggest using one of two alternatives:
1. Return one row for each contact data, repeating the data in other columns:
5 tom 1 25.00 7 555-1212
5 tom 1 25.00 8 555-1315
5 tom 1 25.00 9 555-5511
The problem with this of course is that redundant data is normally a bad idea, but if you don't have too much redundant data it will be OK. Use your judgement here.
2. Use two queries. This means a slightly longer turnaround time, but less data to transfer.
In most cases I'd prefer the second solution.
You should try to avoid making a large number of queries inside a loop. This can almost always be rewritten to a single query. But if using two queries is the most natural way to solve your problem, just use two queries. Don't try to cram all the data you need into a single query just for the sake of reducing the number of queries.
Each row of result must have the same columns, so you can't aggregate multiple rows of contact not having the other columns as well.
Hopefully, this query would achieve what you need:
SELECT
u.user_id as user_id,
u.user_name as user_name,
i.info_id as info_id,
i.rate as rate,
c.contact_id as contact_id,
c.contact_data as contact_data
FROM users as u
LEFT JOIN info as i ON i.user_id = u.user_id
LEFT JOIN contacts as c ON c.user_id = u.user_id