Subqueries in SELECT clause - mysql
I need to SELECT from multiple tables to get a result table like the following:
+--------+-------+-------------------+----------------------+
| itemID | level | studyPhraseString | meaningPhraseStrings |
+--------+-------+-------------------+----------------------+
| 1 | 4 | la maison | house |
+--------+-------+-------------------+----------------------+
| 2 | 3 | voler | to fly,to steal |
+--------+-------+-------------------+----------------------+
Note: studyPhraseString and meaningPhraseStrings should be concatenated strings made up of values from the word table.
My tables:
item
CREATE TABLE `item` (
`itemID` bigint(11) unsigned NOT NULL AUTO_INCREMENT,
`groupID` bigint(11) unsigned NOT NULL,
`studyLang` varchar(5) NOT NULL,
`meaningLang` varchar(5) NOT NULL,
`studyPhraseID` bigint(11) unsigned NOT NULL,
PRIMARY KEY (`itemID`)
) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=latin1;
meaning
CREATE TABLE `meaning` (
`meaningID` bigint(11) unsigned NOT NULL AUTO_INCREMENT,
`itemID` bigint(11) unsigned NOT NULL,
`meaningPhraseID` bigint(11) unsigned NOT NULL,
`meaningIndex` int(11) unsigned NOT NULL,
PRIMARY KEY (`meaningID`)
) ENGINE=InnoDB AUTO_INCREMENT=16 DEFAULT CHARSET=latin1;
phrase
CREATE TABLE `phrase` (
`phraseID` bigint(11) unsigned NOT NULL AUTO_INCREMENT,
`phraseLang` varchar(5) NOT NULL DEFAULT '',
PRIMARY KEY (`phraseID`)
) ENGINE=InnoDB AUTO_INCREMENT=29 DEFAULT CHARSET=latin1;
phrase_word
CREATE TABLE `phrase_word` (
`phrase_wordID` bigint(11) unsigned NOT NULL AUTO_INCREMENT,
`phraseID` bigint(11) unsigned NOT NULL,
`wordID` bigint(11) unsigned NOT NULL,
`wordIndex` int(11) unsigned NOT NULL,
PRIMARY KEY (`phrase_wordID`)
) ENGINE=InnoDB AUTO_INCREMENT=29 DEFAULT CHARSET=latin1;
status
CREATE TABLE `status` (
`statusID` bigint(11) unsigned NOT NULL AUTO_INCREMENT,
`itemID` bigint(11) unsigned NOT NULL,
`level` tinyint(1) unsigned NOT NULL DEFAULT '0',
`nextReviewTime` int(11) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`statusID`)
) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=latin1;
word
CREATE TABLE `word` (
`wordID` bigint(11) unsigned NOT NULL AUTO_INCREMENT,
`string` varchar(64) NOT NULL DEFAULT '',
PRIMARY KEY (`wordID`)
) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=latin1;
I have written the following SELECT statement:
SELECT item.itemID, status.level,
(SELECT GROUP_CONCAT(word.string ORDER BY phrase_word.wordIndex SEPARATOR ' ')
FROM word INNER JOIN phrase_word ON word.wordID=phrase_word.wordID
INNER JOIN item AS subItem ON phrase_word.phraseID=subItem.studyPhraseID
WHERE subItem.itemID=item.itemID
GROUP BY subItem.itemID
) AS studyPhraseString
FROM item INNER JOIN status ON item.itemID=status.itemID
WHERE item.groupID=5
ORDER BY status.statusID DESC
This works but does not include the meaningPhraseString. I can’t figure out how to concat the words into phrases AND concat the phrases into one string separated by ,
I have tried nested GROUP_CONCAT clauses with no success (subquery returns more than 1 row):
The question:
How should this statement be written to include meaningPhraseStrings? Thanks in advance.
PS: I'd like this to be a single query
I have tried the following but it fails. Why? It has two levels of correlated queries.
SELECT
item.itemID,
status.level,
(
SELECT GROUP_CONCAT(word.string ORDER BY phrase_word.wordIndex SEPARATOR ' ')
FROM word INNER JOIN phrase_word ON word.wordID=phrase_word.wordID
INNER JOIN item AS subItem ON phrase_word.phraseID=subItem.studyPhraseID
WHERE subItem.itemID=item.itemID
GROUP BY subItem.itemID
) AS studyPhraseString,
(
SELECT GROUP_CONCAT(meaningPhraseString SEPARATOR '.')
FROM (
(
SELECT GROUP_CONCAT(word.string ORDER BY phrase_word.wordIndex SEPARATOR ' ') AS meaningPhraseString
FROM word INNER JOIN phrase_word ON word.wordID=phrase_word.wordID
INNER JOIN meaning ON phrase_word.phraseID=meaning.meaningPhraseID
INNER JOIN item AS subItem ON meaning.itemID=subItem.itemID
WHERE subItem.itemID=item.itemID #This fails
GROUP BY meaning.meaningPhraseID
) AS meaningPhraseString
)
) AS meaningPhraseStrings
FROM item INNER JOIN status ON item.itemID=status.itemID
WHERE item.groupID=5
ORDER BY item.itemID DESC
Sample data:
INSERT INTO `status` VALUES (22,22,0,0),(23,23,0,0),(24,25,0,0),(25,24,0,0),(26,26,0,0);
INSERT INTO `item` VALUES (22,5,'fr','en',49),(23,5,'fr','en',48),(24,5,'fr','en',56),(25,5,'fr','en',50),(26,5,'fr','en',57);
INSERT INTO `meaning` VALUES (27,22,51,0),(28,23,52,0),(29,23,54,1),(30,24,59,0),(31,24,61,1),(32,25,53,0),(33,25,55,1),(34,26,58,0),(35,26,60,1);
INSERT INTO `phrase` VALUES (48,'fr'),(49,'fr'),(50,'fr'),(51,'en'),(52,'en'),(53,'en'),(54,'en'),(55,'en'),(56,'fr'),(57,'fr'),(58,'en'),(59,'en'),(60,'en'),(61,'en');
INSERT INTO `word` VALUES (46,'l\'autobus'),(47,'bus'),(48,'pourquoi'),(49,'comment'),(50,'why'),(51,'ça'),(52,'va?'),(53,'voler'),(54,'incroyable'),(55,'how'),(56,'is'),(57,'to'),(58,'are'),(59,'incredible'),(60,'that?'),(61,'you?'),(62,'fly'),(63,'amazing'),(64,'hi'),(65,'steal');
INSERT INTO `phrase_word` VALUES (86,49,46,0),(87,51,47,0),(88,48,48,0),(89,50,49,0),(90,52,50,0),(91,54,50,0),(92,50,51,1),(93,50,52,2),(94,57,53,0),(95,53,55,0),(96,56,54,0),(97,54,56,1),(98,53,58,1),(99,58,57,0),(100,59,59,0),(101,54,60,2),(102,53,61,2),(103,58,62,1),(104,61,63,0),(105,60,57,0),(106,55,64,0),(107,60,65,1);
Final answer:
SELECT i.itemID,
s.level,
sp.studyPhraseString,
GROUP_CONCAT(mp.meaningPhraseString
SEPARATOR ', ') AS meaningPhraseStrings
FROM item AS i
JOIN meaning AS m ON i.itemID = m.itemID
JOIN status AS s ON i.itemID = s.itemID
JOIN (
SELECT subItem.studyPhraseID,
GROUP_CONCAT(word.string
ORDER BY phrase_word.wordIndex
SEPARATOR ' ') AS studyPhraseString
FROM word
JOIN phrase_word
ON word.wordID=phrase_word.wordID
JOIN item AS subItem
ON phrase_word.phraseID=subItem.studyPhraseID
GROUP BY subItem.studyPhraseID
) AS sp ON i.studyPhraseID = sp.studyPhraseID
JOIN (
SELECT meaning.meaningPhraseID,
GROUP_CONCAT(word.string
ORDER BY phrase_word.wordIndex
SEPARATOR ' ') AS meaningPhraseString
FROM word
JOIN phrase_word ON word.wordID=phrase_word.wordID
JOIN meaning ON phrase_word.phraseID=meaning.meaningPhraseID
JOIN item AS subItem ON meaning.itemID=subItem.itemID
GROUP BY meaning.meaningPhraseID
) AS mp ON m.meaningPhraseID = mp.meaningPhraseID
GROUP BY i.itemID, s.level, sp.studyPhraseString
ORDER BY i.itemID, s.level, sp.studyPhraseString
Your question seems to be this:
how to concat the words into phrases AND concat the phrases into one string
Let's break it down. You need to join together five tables. Three of them are physical tables, namely item, meaning, and status. From those tables you get the references to the result set items you need called itemID and level, and you get the relationship between items and their meanings.
The other two tables you need are virtual tables (that is, subqueries). One of these gives you your French language phrases, and the other gives you your English-language translations.
Let us create the two queries for the virtual tables. Let's put the words into phrases first. A query like this achieves that goal.
SELECT subItem.studyPhraseID,
GROUP_CONCAT(word.string
ORDER BY phrase_word.wordIndex
SEPARATOR ' ') AS studyPhraseString
FROM word
INNER JOIN phrase_word ON word.wordID=phrase_word.wordID
INNER JOIN item AS subItem ON phrase_word.phraseID=subItem.studyPhraseID
GROUP BY subItem.studyPhraseID
This gives you a resultset table of phrase ID numbers and the phrases. Here's a SQL fiddle for this one based on your samples. http://sqlfiddle.com/#!2/11ae2/9/0
Then, create a similar query giving you the meaningPhraseString values.
SELECT meaning.meaningPhraseID,
GROUP_CONCAT(word.string
ORDER BY phrase_word.wordIndex
SEPARATOR ' ') AS meaningPhraseString
FROM word
INNER JOIN phrase_word ON word.wordID=phrase_word.wordID
INNER JOIN meaning ON phrase_word.phraseID=meaning.meaningPhraseID
INNER JOIN item AS subItem ON meaning.itemID=subItem.itemID
GROUP BY meaning.meaningPhraseID
This gives a list of ids and meaning phrases. Here's the fiddle. http://sqlfiddle.com/#!2/11ae2/6/0
So, we're going to need a five-way join (three physical tables and two subqueries) to get to our final result set. In summary, it will look like this:
SELECT i.itemID,
s.level,
sp.studyPhraseString,
mp.meaningPhraseString
FROM item AS i
JOIN meaning AS m ON i.itemID = m.itemID
JOIN status AS s ON i.itemID = s.itemID
JOIN (
/* the studyPhrase subquery */
) AS sp ON i.studyPhraseID = sp.studyPhraseID
JOIN (
/* the meaningPhrase subquery */
) AS mp ON m.meaningPhraseID = mp.meaningPhraseID
The trick here is that you can use a query (or virtual table) and a physical table interchangeably. So when you need to summarize a bunch of tables, you create a query to do that and then paste it into JOIN (/*query*/) AS alias.
Finally, you need to create the comma-joined strings (e.g. to fly, to steal) by adding yet another GROUP_CONCAT() and GROUP BY to your query. The end result is then
SELECT i.itemID,
s.level,
sp.studyPhraseString,
GROUP_CONCAT(mp.meaningPhraseString
SEPARATOR ', ') AS meaningPhraseStrings
FROM item AS i
JOIN meaning AS m ON i.itemID = m.itemID
JOIN status AS s ON i.itemID = s.itemID
JOIN (
/* the studyPhrase subquery */
) AS sp ON i.studyPhraseID = sp.studyPhraseID
JOIN (
/* the meaningPhrase subquery */
) AS mp ON m.meaningPhraseID = mp.meaningPhraseID
GROUP BY i.itemID, s.level, sp.studyPhraseString
ORDER BY i.itemID, s.level, sp.studyPhraseString
And that is your query. http://sqlfiddle.com/#!2/11ae2/16/0 It definitely takes advantage of the Structured in Structured Query Language.
Related
SELECT with INNER JOIN, GROUP BY by and WHERE
I have a 3 MySQL tables about movies. First one is movie, second is actor and last one is movie_actor_mapping. CREATE TABLE `movie` ( `movie_id` int(11) NOT NULL AUTO_INCREMENT, `title` varchar(50) DEFAULT NULL, `year` varchar(9) DEFAULT NULL, `rating` float(2,1) DEFAULT NULL, `runtime` varchar(6) DEFAULT NULL, `plot` varchar(1500) DEFAULT NULL, PRIMARY KEY (`movie_id`), UNIQUE KEY `title` (`title`) ) CREATE TABLE `actor` ( `actor_id` int(10) unsigned NOT NULL AUTO_INCREMENT, `name` varchar(40) DEFAULT NULL, PRIMARY KEY (`actor_id`), UNIQUE KEY `name_en` (`name`) ) CREATE TABLE `movie_actor_mapping` ( `movie_actor_id` int(10) unsigned NOT NULL AUTO_INCREMENT, `movie_id` int(10) unsigned DEFAULT NULL, `actor_id` int(10) unsigned DEFAULT NULL, PRIMARY KEY (`movie_actor_id`), UNIQUE KEY `movie_actor_id` (`movie_actor_id`) ) Some movies has NULL in rating and when I do SELECT I have to select all movies which has rating and order them by rating. My SELECT looks like: SELECT *, group_concat(a.name separator ', ') as actors FROM movie m INNER JOIN movie_actor_mapping ma ON m.movie_id=ma.movie_id INNER JOIN actor a ON a.actor_id=ta.actor_id GROUP BY m.movie_id; Also I wrote a select and order SELECT * FROM movie WHERE rating is not null ORDER BY rating DESC; but I don't understand how to split these queries. I know that if I will not have GROUP BY - I can write my second query after ON but how to do that using GROUP BY? UPD: I add example data from my tables. So, for movie it's: 1 | "American Horror Story" | "2011–" | 8.2 | "60 min" | "Both physical..." A data from actor table: 1 | Evan Rachel Wood From movie_actor_mapping: 1 | 21 | 1
Maybe I don't understand the question, because it seems too simple, but this query: SELECT m.*, group_concat(a.name separator ', ') as actors FROM movie m INNER JOIN movie_actor_mapping ma ON m.movie_id = ma.movie_id INNER JOIN actor a ON a.actor_id = ma.actor_id WHERE m.rating IS NOT NULL GROUP BY m.movie_id ORDER BY m.rating DESC ...is selecting movies with rating, groups actors and orders them by rating
Try: SELECT *, group_concat(a.name order by rating desc separator ', ' ) as actors FROM movie m INNER JOIN movie_actor_mapping ma ON m.movie_id=ma.movie_id INNER JOIN actor a ON a.actor_id=ta.actor_id WHERE rating is not null GROUP BY m.movie_id;
When you use GROUP BY, you need to have some aggregation action on those returned values that are not used for grouping (for instance, COUNT, MAX, MIN, etc.). If what you want is to get the list of movies where all records of a specific movie are together, you can use the ORDER BY clause. In your case, it could be movie_id.
Improve speed of MySQL query with 5 left joins
Working on a support ticketing system with not a lot of tickets (~3,000). To get a summary grid of ticket information, there are five LEFT JOIN statements on custom field table (j25_field_value) containing about 10,000 records. The query runs too long (~10 seconds) and in cases with a WHERE clause, it runs even longer (up to ~30 seconds or more). Any suggestions for improving the query to reduce the time to run? Four tables: j25_support_tickets CREATE TABLE `j25_support_tickets` ( `id` int(11) NOT NULL AUTO_INCREMENT, `category_id` int(11) NOT NULL DEFAULT '0', `user_id` int(11) DEFAULT NULL, `email` varchar(50) DEFAULT NULL, `subject` varchar(255) DEFAULT NULL, `message` text, `modified_date` datetime DEFAULT NULL, `priority_id` tinyint(3) unsigned DEFAULT NULL, `status_id` tinyint(3) unsigned DEFAULT NULL, PRIMARY KEY (`id`), UNIQUE KEY `id` (`id`) ) ENGINE=MyISAM AUTO_INCREMENT=3868 DEFAULT CHARSET=utf8 j25_support_priorities CREATE TABLE `j25_support_priorities` ( `id` int(11) NOT NULL AUTO_INCREMENT, `title` varchar(100) DEFAULT NULL, PRIMARY KEY (`id`), UNIQUE KEY `id` (`id`) ) ENGINE=MyISAM AUTO_INCREMENT=14 DEFAULT CHARSET=utf8 j25_support_statuses CREATE TABLE `j25_support_statuses` ( `id` int(11) NOT NULL AUTO_INCREMENT, `title` varchar(255) DEFAULT NULL, PRIMARY KEY (`id`), UNIQUE KEY `id` (`id`) ) ENGINE=MyISAM AUTO_INCREMENT=7 DEFAULT CHARSET=utf8 j25_field_value (id, ticket_id, field_id, field_value) CREATE TABLE `j25_support_field_value` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `ticket_id` int(11) DEFAULT NULL, `field_id` int(11) DEFAULT NULL, `field_value` tinytext, PRIMARY KEY (`id`) ) ENGINE=MyISAM AUTO_INCREMENT=10889 DEFAULT CHARSET=utf8 Also, ran this: SELECT LENGTH(field_value) len FROM j25_support_field_value ORDER BY len DESC LIMIT 1 note: the result = 38 The query: SELECT DISTINCT t.id as ID , (select p.title from j25_support_priorities p where p.id = t.priority_id) as Priority , (select s.title from j25_support_statuses s where s.id = t.status_id) as Status , t.subject as Subject , t.email as SubmittedByEmail , type.field_value AS IssueType , ver.field_value AS Version , utype.field_value AS UserType , cust.field_value AS Company , refno.field_value AS RefNo , t.modified_date as Modified FROM j25_support_tickets AS t LEFT JOIN j25_support_field_value AS type ON t.id = type.ticket_id AND type.field_id =1 LEFT JOIN j25_support_field_value AS ver ON t.id = ver.ticket_id AND ver.field_id =2 LEFT JOIN j25_support_field_value AS utype ON t.id = utype.ticket_id AND utype.field_id =3 LEFT JOIN j25_support_field_value AS cust ON t.id = cust.ticket_id AND cust.field_id =4 LEFT JOIN j25_support_field_value AS refno ON t.id = refno.ticket_id AND refno.field_id =5
ALTER TABLE j25_support_field_value ADD INDEX (`ticket_id`,`field_id`,`field_value`(50)) This index will work as a covering index for your query. It will allow the joins to use only this index to look up the values. It should perform massively faster than without this index, since currently your query would have to read every row in the table to find what matches each combination of ticket_id and field_id. I would also suggest converting your tables to InnoDB engine, unless you have a very explicit reason for using MyISAM. ALTER TABLE tablename ENGINE=InnoDB
As above - a better index would help. You could probably then simplify your query into something like this too (join to the table only once): SELECT t.id as ID , p.title as Priority , s.title as Status , t.subject as Subject , t.email as SubmittedByEmail , case when v.field_id=1 then v.field_value else null end as IssueType , case when v.field_id=2 then v.field_value else null end as Version , case when v.field_id=3 then v.field_value else null end as UserType , case when v.field_id=4 then v.field_value else null end as Company , case when v.field_id=5 then v.field_value else null end as RefNo , t.modified_date as Modified FROM j25_support_tickets AS t LEFT JOIN j25_support_field_value v ON t.id = v.ticket_id LEFT JOIN j25_support_priorities p ON p.id = t.priority_id LEFT JOIN j25_support_statuses s ON s.id = t.status_id;
You can do away with the subqueries for starters and just get them from another join. You can add an index to j25_support_field_value alter table j25_support_field_value add key(id, field_type); I assume there is an index on id in j25_support_tickets - if not and if they are unique, add a unique index alter table j25_support_tickets add unique key(id); If they're not unique, remove the word unique from that statement. In MySQL, a join usually requires an index on the field(s) that you are using to join on. This will hold up and produce very reasonable results with huge tables (100m+), if you follow that rule, you will not go wrong. are the ids in j25_support_tickets unique? If they are you can do away with the distinct - if not, or if you are getting exact dupicates in each row, still do away with the distinct and add a group by t.id to the end of this: SELECT t.id as ID , p.title as Priority , s.title as Status , t.subject as Subject , t.email as SubmittedByEmail , type.field_value AS IssueType , ver.field_value AS Version , utype.field_value AS UserType , cust.field_value AS Company , refno.field_value AS RefNo , t.modified_date as Modified FROM j25_support_tickets AS t LEFT JOIN j25_support_field_value AS type ON t.id = type.ticket_id AND type.field_id =1 LEFT JOIN j25_support_field_value AS ver ON t.id = ver.ticket_id AND ver.field_id =2 LEFT JOIN j25_support_field_value AS utype ON t.id = utype.ticket_id AND utype.field_id =3 LEFT JOIN j25_support_field_value AS cust ON t.id = cust.ticket_id AND cust.field_id =4 LEFT JOIN j25_support_field_value AS refno ON t.id = refno.ticket_id AND refno.field_id =5 LEFT JOIN j25_support_priorities p ON p.id = t.priority_id LEFT JOIN j25_support_statuses s ON s.id = t.status_id;
Switch to InnoDB. After switching to InnoDB, make the PRIMARY KEY for j25_support_field_value be (ticket_id, field_id) (and get rid if id). (Tacking on field_value(50) will hurt, not help.) A PRIMARY KEY is a UNIQUE KEY, so don't have both. Use VARCHAR(255) instead of the nearly-equivalent TINYTEXT. EAV schema sucks. My ran on EAV.
How to make use of index on LEFT OUTER JOIN
Here is a set of tables describing music composers : CREATE TABLE IF NOT EXISTS `compositors` ( `id` int(11) NOT NULL, `name` varchar(45) NOT NULL COMMENT 'Nom et Prenom', `birth_date` varchar(45) DEFAULT NULL, `death_date` varchar(45) DEFAULT NULL, `birth_place` varchar(45) DEFAULT NULL, `death_place` varchar(45) DEFAULT NULL, `gender` enum('M','F') DEFAULT NULL, `century` varchar(45) DEFAULT NULL, `country` int(11) DEFAULT NULL ) ENGINE=InnoDB AUTO_INCREMENT=28741 DEFAULT CHARSET=latin1; CREATE TABLE IF NOT EXISTS `compositor_biography` ( `index` int(11) NOT NULL, `compositor_id` int(11) NOT NULL, `url` varchar(255) DEFAULT NULL ) ENGINE=InnoDB AUTO_INCREMENT=15325 DEFAULT CHARSET=latin1; CREATE TABLE IF NOT EXISTS `compositor_comments` ( `compositor_id` int(11) NOT NULL, `comment` text NOT NULL, `public` enum('Publique','Privé') NOT NULL ) ENGINE=InnoDB DEFAULT CHARSET=latin1; CREATE TABLE IF NOT EXISTS `compositor_country` ( `compositor_id` int(11) NOT NULL, `country_id` int(11) NOT NULL ) ENGINE=InnoDB DEFAULT CHARSET=latin1; Here are my indexes : -- -- Index pour la table `compositors` -- ALTER TABLE `compositors` ADD PRIMARY KEY (`id`), ADD KEY `countries` (`country`); ALTER TABLE `compositor_biography` ADD PRIMARY KEY (`index`), ADD KEY `index` (`compositor_id`); ALTER TABLE `compositor_comments` ADD KEY `c_compositor_idx` (`compositor_id`); And finally sample data : INSERT INTO `compositors` (`id`, `name`, `birth_date`, `death_date`, `birth_place`, `death_place`, `gender`, `century`, `country`) VALUES (1, 'Dummy Compositor', '1606', '1676', 'Bruxellesss', NULL, 'F', '17', 11); INSERT INTO `compositor_biography` (`index`, `compositor_id`, `url`) VALUES (15322, 1, 'Dummy Link 1'), (15323, 1, 'Dummy Link 2'), (15324, 1, 'Dummy Link 3'); INSERT INTO `compositor_comments` (`compositor_id`, `comment`, `public`) VALUES (1, 'Dummy Comment', 'Privé'); Here is an example query that my PHP script generate : SELECT DISTINCT compositors.id, compositors.name, compositors.birth_date, compositors.death_date, compositors.birth_place, compositors.death_place, compositors.gender, compositors.century, compositors.country, GROUP_CONCAT( compositor_biography.url SEPARATOR ';') AS concat_compositor_biography_url, GROUP_CONCAT( compositor_comments.comment SEPARATOR ';') AS concat_compositor_comments_comment, GROUP_CONCAT( compositor_comments.public + 0 SEPARATOR ';') AS concat_compositor_comments_public FROM compositors LEFT JOIN compositor_biography ON compositors.id = compositor_biography.compositor_id LEFT JOIN compositor_comments ON compositors.id = compositor_comments.compositor_id GROUP BY compositors.id However, this one has a problem, if you execute this query, you can see that in the concat_compositor_comments_comment column, you have that result : Dummy Comment;Dummy Comment;Dummy Comment but there is only one actual comment. I didn't really understand what was the problem there, but it seemed like it was the GROUP BY. It should have one GROUP BY per JOIN - according to the second answer at Multiple GROUP_CONCAT on different fields using MySQL -- so I did it, and it worked, with this query : SELECT DISTINCT compositors.id, compositors.NAME, compositors.birth_date, compositors.death_date, compositors.birth_place, compositors.death_place, compositors.gender, compositors.century, compositors.country, concat_compositor_biography_url, concat_compositor_comments_comment, concat_compositor_comments_public FROM compositors LEFT JOIN ( SELECT compositor_id, GROUP_CONCAT(compositor_biography.url SEPARATOR ';') AS concat_compositor_biography_url FROM compositor_biography GROUP BY compositor_biography.compositor_id ) compositor_biography ON compositors.id = compositor_biography.compositor_id LEFT JOIN ( SELECT compositor_id, GROUP_CONCAT(compositor_comments.comment SEPARATOR ';') AS concat_compositor_comments_comment, GROUP_CONCAT(compositor_comments.PUBLIC + 0 SEPARATOR ';') AS concat_compositor_comments_public FROM compositor_comments GROUP BY compositor_comments.compositor_id ) compositor_comments ON compositors.id = compositor_comments.compositor_id However, this query has huge performance problem, since it doesn't use INDEXES, or at least it seems to scan all the tables, and with 24000 composers, it takes approx 420 second for that query, while the other ( which gives wrong results on GROUP BY ) takes 1 second. How can I change the second query, so it uses correctly the index and doesn't scan all the tables ? Here is a link to a SQL-Fiddle the database schema : http://sqlfiddle.com/#!2/6b0132 UPDATE According to #phil_w, and after further testing, this query seems to work with very good performance : SELECT a.id, a.name, a.concat_compositor_biography_url, b.concat_compositor_aliases_data, GROUP_CONCAT(compositor_comments.comment SEPARATOR ';') as concat_compositor_comments_comment, GROUP_CONCAT(compositor_comments.public + 0 SEPARATOR ';') as concat_compositor_comments_public FROM ( SELECT b.id, b.name, b.concat_compositor_biography_url, GROUP_CONCAT(compositor_aliases.data SEPARATOR ';') as concat_compositor_aliases_data FROM ( SELECT compositors.id, compositors.name, GROUP_CONCAT(compositor_biography.url SEPARATOR ';') AS concat_compositor_biography_url FROM compositors LEFT JOIN compositor_biography ON compositors.id = compositor_biography.compositor_id GROUP BY compositors.id ) b LEFT JOIN compositor_aliases ON b.id = compositor_aliases.compositor_id GROUP BY b.id ) a LEFT JOIN compositor_comments ON a.id = compositor_comments.compositor_id GROUP BY a.id However, how would it be possible to have the same result in a more compact query ? ( by the way, shall I create a new question for that and make this one resolved ? )
This question has nothing to do with "indexes". The problem is that you have two joins and every combination of rows will be returned (ie you have 3 matching rows in the other join to compositor_biography). The fix is simple - just add DISTINCT to the GROUP_CONCAT() function: ... GROUP_CONCAT( DISTINCT compositor_comments.comment SEPARATOR ';') AS concat_compositor_comments_comment, ...
It's normal that you have the entry 3 times, because you have 3 rows in compositor_biography... perhaps you could go step by step, first gathering the bio only: SELECT compositors.id, compositors.name, GROUP_CONCAT( compositor_biography.url SEPARATOR ';') AS concat_compositor_biography_url FROM compositors LEFT JOIN compositor_biography ON compositors.id = compositor_biography.compositor_id GROUP BY compositors.id then join the rest select t.id, t.name,t.concat_compositor_biography_url, GROUP_CONCAT( compositor_comments.comment SEPARATOR ';') AS concat_compositor_comments_comment from ( SELECT compositors.id, compositors.name, GROUP_CONCAT( compositor_biography.url SEPARATOR ';') AS concat_compositor_biography_url FROM compositors LEFT JOIN compositor_biography ON compositors.id = compositor_biography.compositor_id GROUP BY compositors.id ) t LEFT JOIN compositor_comments ON t.id = compositor_comments.compositor_id and so on... I don't see why it would not use the index unless the table is small. Try 'explain select...' to confirm that.
MySQL Select from multiple tables ordered by count of left joined table
I have been trying to select data from multiple tables whereas I sort by the count of id's from an additional table that I have joined with a Left Join. Everything works great except for that the count gets incremented with 600something instead of one for each row. I tried as I saw in a similar problem to put distinct inside the count statement but with the only result of freezing the database. Tables: CREATE TABLE IF NOT EXISTS `places` ( `PlaceId` int(10) NOT NULL AUTO_INCREMENT, `Name` varchar(45) COLLATE utf8_unicode_ci NOT NULL, `AreaId` int(10) NOT NULL DEFAULT '1', PRIMARY KEY (`PlaceId`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=0; CREATE TABLE IF NOT EXISTS `places_descriptions` ( `DescId` int(45) NOT NULL AUTO_INCREMENT, `PlaceId` int(10) NOT NULL, `Description` varchar(1024) COLLATE utf8_unicode_ci DEFAULT NULL, PRIMARY KEY (`DescId`), KEY `PlaceId` (`PlaceId`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=0; CREATE TABLE IF NOT EXISTS `places_hits` ( `HitId` int(45) NOT NULL AUTO_INCREMENT, `PlaceId` int(45) NOT NULL, PRIMARY KEY (`HitId`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=0; Query: SELECT p.PlaceId, p.Name, pd.Description, COUNT(ph.HitId) AS numHits FROM places_descriptions AS pd, places AS p LEFT JOIN places_hits AS ph ON (p.PlaceId = ph.PlaceId) WHERE (p.PlaceId = pd.PlaceId) AND (p.AreaId = 1) GROUP BY p.PlaceId, p.Name, pd.Description ORDER BY numHits DESC, p.PlaceId LIMIT 0, 10 Any ideas? Thank you!
At first glance, your problem may be that you're cross-joining pd and p because there's no ON clause. The placeId=placeId in the where masks this problem by filtering out the rows where the join was inaccurate, but it'd be better to place that condition in the ON. The large number of rows actually existing before filtering might be throwing off the count, though I'm not sure. Try this, to start: SELECT p.PlaceId, p.Name, pd.Description, COUNT(ph.HitId) AS numHits FROM places_descriptions AS pd INNER JOIN places AS p ON p.PlaceId = pd.PlaceID LEFT JOIN places_hits AS ph ON (p.PlaceId = ph.PlaceId) WHERE (p.AreaId = 1) GROUP BY p.PlaceId, p.Name, pd.Description ORDER BY numHits DESC, p.PlaceId
Best way to search two queries and eliminate rows without a relationship
I am working on a property website and have record sets for property and for unit, unit has a one-to-many relationship with property. What I'm trying to figure out is how to best create a search function which will output results based on criteria from both. So if I search for a property with the location Manchester and a unit with a freehold tenure I'd like to eliminate all properties which don't have a unit with the tenure of freehold. A potential solution I've considered is to create a record set for properties which match the property criteria and then create a unit record set for units which match the unit criteria and then finally loop through the property record set in server-side code and eliminate any properties which aren't related to any of the units in the unit record set. Really not sure if this is the best way to do things though so would be keen to hear any suggestions? Thanks EDIT (Added table structure and MySQL): -- -- Table structure for table `property` -- CREATE TABLE IF NOT EXISTS `property` ( `id` int(11) NOT NULL AUTO_INCREMENT, `name` text NOT NULL, `street` text NOT NULL, `town` text NOT NULL, `postcode` text NOT NULL, `description` longtext NOT NULL, `team_member` varchar(255) NOT NULL DEFAULT '', `pdf` text NOT NULL, `default_image_id` int(11) DEFAULT NULL, `virtual_tour_link` text NOT NULL, `date` date NOT NULL DEFAULT '0000-00-00', `archive` int(11) NOT NULL DEFAULT '0', PRIMARY KEY (`id`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='' AUTO_INCREMENT=13 ; -- -- Table structure for table `unit` -- CREATE TABLE IF NOT EXISTS `unit` ( `id` int(11) NOT NULL AUTO_INCREMENT, `name` text NOT NULL, `description` text NOT NULL, `size_sq_ft` int(11) DEFAULT NULL, `size_acres` float DEFAULT NULL, `price` float DEFAULT NULL, `rental_price` float DEFAULT NULL, `on_application` tinyint(1) DEFAULT NULL, UNIQUE KEY `id` (`id`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='Stores data for property units' AUTO_INCREMENT=5; -- -- Table structure for table `property_to_unit` -- CREATE TABLE IF NOT EXISTS `property_to_unit` ( `property_id` int(11) NOT NULL, `unit_id` int(11) NOT NULL ) ENGINE=MyISAM DEFAULT CHARSET=utf8; -- -- MySQL which produces list of properties -- SELECT P.id AS id, P.name AS name, P.street AS street, P.town AS town, P.postcode AS postcode, P.description AS description, P.team_member AS team_member, P.pdf AS pdf, P.virtual_tour_link AS virtual_tour_link, P.date AS date, P.archive AS archive, PI.name as image, P2.image_ids as image_ids, L2.location_ids as location_ids, U2.unit_ids as unit_ids FROM property P -- Get default image and join using property id LEFT JOIN property_image PI ON PI.id = P.default_image_id -- Create a list of image_ids from property_image and -- property_to_property_image tables then join using property_id LEFT JOIN ( SELECT property_id, GROUP_CONCAT(CAST(id AS CHAR)) as image_ids FROM property_to_property_image PTPI LEFT JOIN property_image PI ON PI.id = PTPI.property_image_id GROUP BY property_id ) P2 ON P2.property_id = P.id -- Create a list of locations from property_location table -- and join using property_id LEFT JOIN ( SELECT property_id, property_location_id, GROUP_CONCAT(CAST(property_location.id AS CHAR)) AS location_ids FROM property_to_property_location INNER JOIN property_location ON property_location.id = property_to_property_location.property_location_id GROUP BY property_id ) L2 ON L2.property_id = P.id -- Create a list of units from unit table -- and join using property_id LEFT JOIN ( SELECT property_id, unit_id, GROUP_CONCAT(CAST(unit_id AS CHAR)) AS unit_ids FROM property_to_unit INNER JOIN unit ON unit.id = property_to_unit.unit_id GROUP BY property_id ) U2 ON U2.property_id = P.id -- -- MySQL which produces list of units -- SELECT id, name, description, size_sq_ft, size_acres, price, rental_price, on_application, tenure_ids, tenure_names, type_ids, type_names FROM unit AS U -- join tenure ids and names LEFT JOIN ( SELECT unit_id, GROUP_CONCAT( CAST(UT.id AS CHAR) ) AS tenure_ids, GROUP_CONCAT(UT.name) AS tenure_names FROM unit_to_unit_tenure UTUT INNER JOIN unit_tenure UT ON UT.id = UTUT.unit_tenure_id GROUP BY unit_id ) UT ON UT.unit_id = U.id -- join type ids and names LEFT JOIN ( SELECT unit_id, GROUP_CONCAT( CAST(UTYPE.id AS CHAR) ) AS type_ids, GROUP_CONCAT(UTYPE.name) AS type_names FROM unit_to_unit_type UTUT INNER JOIN unit_type UTYPE ON UTYPE.id = UTUT.unit_type_id GROUP BY unit_id ) UTYPE ON UTYPE.unit_id = U.id WHERE 0=0 I'm currently using a dynamically created WHERE statement appended to each MySQL query to filter the property and unit results.
You're making it a bit more complicated than it is. If I understand correctly, you can easily do this in a single query. This would search properties that have units with a particlar unit tenure id: select * from property p where p.id in ( select pu.property_id from property_to_unit pu inner join unit u ON pu.unit_id = u.id inner join unit_to_unit_tenure uut ON u.id = uut.unit_id where uut.id = <cfqueryparam value="#uutid#"> ) Using two queries and then looping through to cross-check sounds like it could be dog slow.
Your situation requires a posted foreign key in the property table. Store the unit_id in the property table and use a join in your query such as: select * from property p, unit u where p.unit_id = u.id and p.town = .... EDIT: So I just noticed the rest of your SQL. If you require to keep the many-to-many relationship table for the unit -> property relationship then you will need to join unit and property off of that table.