I have a chatting application. I have an api which returns list of users who the user talked. But it takes a long to mysql return a list messages when it reachs 100000 rows of data.
This is my messages table
CREATE TABLE IF NOT EXISTS `messages` (
`_id` int(11) NOT NULL AUTO_INCREMENT,
`fromid` int(11) NOT NULL,
`toid` int(11) NOT NULL,
`message` text NOT NULL,
`attachments` text NOT NULL,
`status` tinyint(1) NOT NULL DEFAULT '0',
`date` datetime NOT NULL,
`delete` varchar(50) NOT NULL,
`uuid_read` varchar(250) NOT NULL,
PRIMARY KEY (`_id`),
KEY `fromid` (`fromid`,`toid`,`status`,`delete`,`uuid_read`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=118561 ;
and this is my users table (simplified)
CREATE TABLE IF NOT EXISTS `users` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`login` varchar(50) DEFAULT '',
`sex` tinyint(1) DEFAULT '0',
`status` varchar(255) DEFAULT '',
`avatar` varchar(30) DEFAULT '0',
`last_active` datetime DEFAULT NULL,
`active` tinyint(1) DEFAULT '1',
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=15523 ;
And here is my query (for user with id 1930)
select SQL_CALC_FOUND_ROWS `u_id`, `id`, `login`, `sex`, `birthdate`, `avatar`, `online_status`, SUM(`count`) as `count`, SUM(`nr_count`) as `nr_count`, `date`, `last_mesg` from
(
(select `m`.`fromid` as `u_id`, `u`.`id`, `u`.`login`, `u`.`sex`, `u`.`birthdate`, `u`.`avatar`, `u`.`last_active` as online_status, COUNT(`m`.`_id`) as `count`, (COUNT(`m`.`_id`)-SUM(`m`.`status`)) as `nr_count`, `tm`.`date` as `date`, `tm`.`message` as `last_mesg` from `messages` as m inner join `messages` as tm on `tm`.`_id`=(select MAX(`_id`) from `messages` as `tmz` where `tmz`.`fromid`=`m`.`fromid`) left join `users` as u on `u`.`id`=`m`.`fromid` where `m`.`toid`=1930 and `m`.`delete` not like '%1930;%' group by `u`.`id`)
UNION
(select `m`.toid as `u_id`, `u`.`id`, `u`.`login`, `u`.`sex`, `u`.`birthdate`, `u`.`avatar`, `u`.`last_active` as online_status, COUNT(`m`.`_id`) as `count`, 0 as `nr_count`, `tm`.`date` as `date`, `tm`.`message` as `last_mesg` from `messages` as m inner join `messages` as tm on `tm`.`_id`=(select MAX(`_id`) from `messages` as `tmz` where `tmz`.`toid`=`m`.`toid`) left join `users` as u on `u`.`id`=`m`.`toid` where `m`.`fromid`=1930 and `m`.`delete` not like '%1930;%' group by `u`.`id`)
order by `date` desc ) as `f` group by `u_id` order by `date` desc limit 0,10
Please help to optimize this query
What I need,
Who user talked to (name, sex, and etc)
What was the last message (from me or to me)
Count of messages (all)
Count of unread messages (only to me)
The query works well, but takes too long.
The output must be like this
You have some design problems on your query and database.
You should avoid keywords as column names, as that delete column or the count column;
You should avoid selecting columns not declared in the group by without an aggregation function... although MySQL allows this, it's not a standard and you don't have any control on what data will be selected;
Your not like construction may cause a bad behavior on your query because '%1930;%' may match 11930; and 11930 is not equal to 1930;
You should avoid like constructions starting and ending with % wildcard, which will cause the text processing to take longer;
You should design a better way to represent a message deletion, probably a better flag and/or another table to save any important data related with the action;
Try to limit your result before the join conditions (with a derived table) to perform less processing;
I tried to rewrite your query the best way I understood it. I've executed my query in a messages table with ~200.000 rows and no indexes and it performed in 0,15 seconds. But, for sure you should create the right indexes to help it perform better when the amount of data increase.
SELECT SQL_CALC_FOUND_ROWS
u.id,
u.login,
u.sex,
u.birthdate,
u.avatar,
u.last_active AS online_status,
g._count,
CASE WHEN m.toid = 1930
THEN g.nr_count
ELSE 0
END AS nr_count,
m.`date`,
m.message AS last_mesg
FROM
(
SELECT
MAX(_id) AS _id,
COUNT(*) AS _count,
COUNT(*) - SUM(m.status) AS nr_count
FROM messages m
WHERE 1=1
AND m.`delete` NOT LIKE '%1930;%'
AND
(0=1
OR m.fromid = 1930
OR m.toid = 1930
)
GROUP BY
CASE WHEN m.fromid = 1930
THEN m.toid
ELSE m.fromid
END
ORDER BY MAX(`date`) DESC
LIMIT 0, 10
) g
INNER JOIN messages AS m ON 1=1
AND m._id = g._id
LEFT JOIN users AS u ON 0=1
OR (m.fromid <> 1930 AND u.id = m.fromid)
OR (m.toid <> 1930 AND u.id = m.toid)
ORDER BY m.`date` DESC
;
I'm having a problem with a query for MySQL, i have to get rows with distinct "c.email" AND distinct "pf.cliente_id", I'm doing this code below, but i'm not getting what i want, the "c.email" don't returns DISTINCT
SELECT DISTINCT c.nome as nome, pf.id_cliente as cliente_id
FROM cliente c
INNER JOIN pessoa_fisica pf
ON c.id_cliente = pf.id_cliente
GROUP BY c.email, pf.id_cliente;
I Also tried:
SELECT c.nome as nome, pf.id_cliente as cliente_id
FROM cliente c
INNER JOIN pessoa_fisica pf
ON c.id_cliente = pf.id_cliente
GROUP BY c.email, pf.id_cliente;
Edit
cliente = email( It's not the primary key, is a foreign key and i have it reapeted )
pessoa_fisica = id_cliente ( It's not the primary key, is a foreign key and i have it reapeted )
this query world work for you may be syntax error there because i have not database.
select SUBSTRING_INDEX(full, ' ', 1) AS email ,SUBSTRING_INDEX(full, ' ', -1) AS cliente_id , nome
from
(SELECT DISTINCT (concat(c.email as email,' ',pf.id_cliente )) as 'full' , c.nome as 'nome'
FROM cliente c
INNER JOIN pessoa_fisica pf
ON c.id_cliente = pf.id_cliente) as temp;
Mysql distinct doesn't allow multiple values, but apply little effort means first concat columns and then apply distinct then break column again. I am setting an example for you.
Create Table: CREATE TABLE `dupli` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`fname` varchar(20) DEFAULT NULL,
`lname` varchar(20) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
insert into dupli (fname,lname) values ('Hitesh', 'mundra'),('Neeraj', 'sharma'),('Kailash','yadav');
insert into dupli (fname,lname) values ('Hitesh', 'mundra'),('Neeraj', 'sharma'),('Kailash','yadav');
select id, SUBSTRING_INDEX(name, ' ', 1) AS fname ,SUBSTRING_INDEX(name, ' ', -1) AS lname from (select distinct( concat(fname," ",lname) ) as Name, id from dupli) as temp;
I have two tables. I'm trying to return the thumbnail_img and fullsize_img column respective values for each article_steps row. I want this to include NULL if there are no matching values for each step, too.
CREATE TABLE IF NOT EXISTS `article_steps` (
`id` int(5) NOT NULL AUTO_INCREMENT,
`article_id` int(10) NOT NULL,
`step_num` int(3) NOT NULL,
`step_title` char(100) NOT NULL,
`step_body` text NOT NULL,
PRIMARY KEY (`id`),
FULLTEXT KEY `step_body` (`step_body`,`step_title`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=9 ;
CREATE TABLE IF NOT EXISTS `article_steps_gallery` (
`id` int(5) NOT NULL AUTO_INCREMENT,
`article_id` int(5) NOT NULL,
`step_num` int(5) NOT NULL,
`thumbnail_img` text NOT NULL,
`fullsize_img` text NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=21 ;
Here's my statement:
SELECT a.id,a.step_num, a.step_title,a.step_body, CONCAT( (case when a.step_num=s.step_num then GROUP_CONCAT(fullsize_img, '|',thumbnail_img SEPARATOR ' ') end) ) AS images
FROM article_steps a, article_steps_gallery s
WHERE a.article_id=s.article_id
AND a.article_id=2
GROUP BY step_num
This is returning:
id step_num step_title step_body images
5 1 faketitle1 fakebody1 NULL
6 2 faketitle2 fakebody2 image.jpg|image_thumbnail.jpg
7 3 faketitle3 fakebody3 NULL
8 4 faketitle4 fakebody4 NULL
The only step_num which should be returning NULL is #1 --- however it's only correctly returning fullsize_img & thumbnail_img for step_num 2.
SELECT a.step_num, a.step_title,a.step_body, GROUP_CONCAT(DISTINCT fullsize_img) AS image, GROUP_CONCAT(DISTINCT thumbnail_img) AS thumbnail
FROM article_steps a, article_steps_gallery s
WHERE a.article_id=s.article_id
AND a.step_num=s.step_num
AND a.article_id=2
GROUP BY step_num
This statement here returns the correct images. But, being that it's GROUP_CONCAT it doesn't return NULL values. So the step_num 1, which doesn't have any images doesn't get returned.
id step_num step_title step_body image thumbnail
6 2 faketitle2 fakebody2 image2.jpg image2_thumbnail.jpg
7 3 faketitle3 fakebody3 image3.jpg image3_thumbnail.jpg
8 4 faketitle4 fakebody4 image4.jpg image4_thumbnail.jpg
Why do you need a group by, you're not doing any aggregates?
SELECT a.id,a.step_num, a.step_title,a.step_body, CONCAT( (case when a.step_num=s.step_num then GROUP_CONCAT(fullsize_img, '|',thumbnail_img SEPARATOR ' ') end) ) AS images
FROM article_steps a, article_steps_gallery s
WHERE a.article_id=s.article_id
AND a.article_id=2
PER DOCS on mySQL Group by: http://dev.mysql.com/doc/refman/5.0/en/group-by-extensions.html
The server is free to choose any value from each group, so unless they are the same, the values chosen are indeterminate.
So the reason you're getting NULL is that one of the records that is being rolled up due to your group by is null and the RDBMS is choosing that one at random.
if you MUST have a group by to eliminate duplicates... (why are there duplicates?)
Then either use distinct or group by every field:
SELECT DISTINCT a.id,a.step_num, a.step_title,a.step_body, CONCAT( (case when a.step_num=s.step_num then GROUP_CONCAT(fullsize_img, '|',thumbnail_img SEPARATOR ' ') end) ) AS images
FROM article_steps a, article_steps_gallery s
WHERE a.article_id=s.article_id
AND a.article_id=2
.
SELECT a.id,a.step_num, a.step_title,a.step_body, CONCAT( (case when a.step_num=s.step_num then GROUP_CONCAT(fullsize_img, '|',thumbnail_img SEPARATOR ' ') end) ) AS images
FROM article_steps a, article_steps_gallery s
WHERE a.article_id=s.article_id
AND a.article_id=2
GROUP BY a.id,a.step_num, a.step_title,a.step_body, CONCAT( (case when a.step_num=s.step_num then GROUP_CONCAT(fullsize_img, '|',thumbnail_img SEPARATOR ' ') end) )
I got it to work with the following code:
SELECT a.id,a.step_num, a.step_title,a.step_body,
(SELECT GROUP_CONCAT(fullsize_img) FROM article_steps_gallery s WHERE a.step_num=s.step_num AND a.article_id=s.article_id) AS image,
(SELECT GROUP_CONCAT(thumbnail_img) FROM article_steps_gallery s WHERE a.step_num=s.step_num AND a.article_id=s.article_id) AS thumbnail
FROM article_steps a
WHERE a.article_id=1
I need to SELECT from multiple tables to get a result table like the following:
+--------+-------+-------------------+----------------------+
| itemID | level | studyPhraseString | meaningPhraseStrings |
+--------+-------+-------------------+----------------------+
| 1 | 4 | la maison | house |
+--------+-------+-------------------+----------------------+
| 2 | 3 | voler | to fly,to steal |
+--------+-------+-------------------+----------------------+
Note: studyPhraseString and meaningPhraseStrings should be concatenated strings made up of values from the word table.
My tables:
item
CREATE TABLE `item` (
`itemID` bigint(11) unsigned NOT NULL AUTO_INCREMENT,
`groupID` bigint(11) unsigned NOT NULL,
`studyLang` varchar(5) NOT NULL,
`meaningLang` varchar(5) NOT NULL,
`studyPhraseID` bigint(11) unsigned NOT NULL,
PRIMARY KEY (`itemID`)
) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=latin1;
meaning
CREATE TABLE `meaning` (
`meaningID` bigint(11) unsigned NOT NULL AUTO_INCREMENT,
`itemID` bigint(11) unsigned NOT NULL,
`meaningPhraseID` bigint(11) unsigned NOT NULL,
`meaningIndex` int(11) unsigned NOT NULL,
PRIMARY KEY (`meaningID`)
) ENGINE=InnoDB AUTO_INCREMENT=16 DEFAULT CHARSET=latin1;
phrase
CREATE TABLE `phrase` (
`phraseID` bigint(11) unsigned NOT NULL AUTO_INCREMENT,
`phraseLang` varchar(5) NOT NULL DEFAULT '',
PRIMARY KEY (`phraseID`)
) ENGINE=InnoDB AUTO_INCREMENT=29 DEFAULT CHARSET=latin1;
phrase_word
CREATE TABLE `phrase_word` (
`phrase_wordID` bigint(11) unsigned NOT NULL AUTO_INCREMENT,
`phraseID` bigint(11) unsigned NOT NULL,
`wordID` bigint(11) unsigned NOT NULL,
`wordIndex` int(11) unsigned NOT NULL,
PRIMARY KEY (`phrase_wordID`)
) ENGINE=InnoDB AUTO_INCREMENT=29 DEFAULT CHARSET=latin1;
status
CREATE TABLE `status` (
`statusID` bigint(11) unsigned NOT NULL AUTO_INCREMENT,
`itemID` bigint(11) unsigned NOT NULL,
`level` tinyint(1) unsigned NOT NULL DEFAULT '0',
`nextReviewTime` int(11) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`statusID`)
) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=latin1;
word
CREATE TABLE `word` (
`wordID` bigint(11) unsigned NOT NULL AUTO_INCREMENT,
`string` varchar(64) NOT NULL DEFAULT '',
PRIMARY KEY (`wordID`)
) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=latin1;
I have written the following SELECT statement:
SELECT item.itemID, status.level,
(SELECT GROUP_CONCAT(word.string ORDER BY phrase_word.wordIndex SEPARATOR ' ')
FROM word INNER JOIN phrase_word ON word.wordID=phrase_word.wordID
INNER JOIN item AS subItem ON phrase_word.phraseID=subItem.studyPhraseID
WHERE subItem.itemID=item.itemID
GROUP BY subItem.itemID
) AS studyPhraseString
FROM item INNER JOIN status ON item.itemID=status.itemID
WHERE item.groupID=5
ORDER BY status.statusID DESC
This works but does not include the meaningPhraseString. I can’t figure out how to concat the words into phrases AND concat the phrases into one string separated by ,
I have tried nested GROUP_CONCAT clauses with no success (subquery returns more than 1 row):
The question:
How should this statement be written to include meaningPhraseStrings? Thanks in advance.
PS: I'd like this to be a single query
I have tried the following but it fails. Why? It has two levels of correlated queries.
SELECT
item.itemID,
status.level,
(
SELECT GROUP_CONCAT(word.string ORDER BY phrase_word.wordIndex SEPARATOR ' ')
FROM word INNER JOIN phrase_word ON word.wordID=phrase_word.wordID
INNER JOIN item AS subItem ON phrase_word.phraseID=subItem.studyPhraseID
WHERE subItem.itemID=item.itemID
GROUP BY subItem.itemID
) AS studyPhraseString,
(
SELECT GROUP_CONCAT(meaningPhraseString SEPARATOR '.')
FROM (
(
SELECT GROUP_CONCAT(word.string ORDER BY phrase_word.wordIndex SEPARATOR ' ') AS meaningPhraseString
FROM word INNER JOIN phrase_word ON word.wordID=phrase_word.wordID
INNER JOIN meaning ON phrase_word.phraseID=meaning.meaningPhraseID
INNER JOIN item AS subItem ON meaning.itemID=subItem.itemID
WHERE subItem.itemID=item.itemID #This fails
GROUP BY meaning.meaningPhraseID
) AS meaningPhraseString
)
) AS meaningPhraseStrings
FROM item INNER JOIN status ON item.itemID=status.itemID
WHERE item.groupID=5
ORDER BY item.itemID DESC
Sample data:
INSERT INTO `status` VALUES (22,22,0,0),(23,23,0,0),(24,25,0,0),(25,24,0,0),(26,26,0,0);
INSERT INTO `item` VALUES (22,5,'fr','en',49),(23,5,'fr','en',48),(24,5,'fr','en',56),(25,5,'fr','en',50),(26,5,'fr','en',57);
INSERT INTO `meaning` VALUES (27,22,51,0),(28,23,52,0),(29,23,54,1),(30,24,59,0),(31,24,61,1),(32,25,53,0),(33,25,55,1),(34,26,58,0),(35,26,60,1);
INSERT INTO `phrase` VALUES (48,'fr'),(49,'fr'),(50,'fr'),(51,'en'),(52,'en'),(53,'en'),(54,'en'),(55,'en'),(56,'fr'),(57,'fr'),(58,'en'),(59,'en'),(60,'en'),(61,'en');
INSERT INTO `word` VALUES (46,'l\'autobus'),(47,'bus'),(48,'pourquoi'),(49,'comment'),(50,'why'),(51,'ça'),(52,'va?'),(53,'voler'),(54,'incroyable'),(55,'how'),(56,'is'),(57,'to'),(58,'are'),(59,'incredible'),(60,'that?'),(61,'you?'),(62,'fly'),(63,'amazing'),(64,'hi'),(65,'steal');
INSERT INTO `phrase_word` VALUES (86,49,46,0),(87,51,47,0),(88,48,48,0),(89,50,49,0),(90,52,50,0),(91,54,50,0),(92,50,51,1),(93,50,52,2),(94,57,53,0),(95,53,55,0),(96,56,54,0),(97,54,56,1),(98,53,58,1),(99,58,57,0),(100,59,59,0),(101,54,60,2),(102,53,61,2),(103,58,62,1),(104,61,63,0),(105,60,57,0),(106,55,64,0),(107,60,65,1);
Final answer:
SELECT i.itemID,
s.level,
sp.studyPhraseString,
GROUP_CONCAT(mp.meaningPhraseString
SEPARATOR ', ') AS meaningPhraseStrings
FROM item AS i
JOIN meaning AS m ON i.itemID = m.itemID
JOIN status AS s ON i.itemID = s.itemID
JOIN (
SELECT subItem.studyPhraseID,
GROUP_CONCAT(word.string
ORDER BY phrase_word.wordIndex
SEPARATOR ' ') AS studyPhraseString
FROM word
JOIN phrase_word
ON word.wordID=phrase_word.wordID
JOIN item AS subItem
ON phrase_word.phraseID=subItem.studyPhraseID
GROUP BY subItem.studyPhraseID
) AS sp ON i.studyPhraseID = sp.studyPhraseID
JOIN (
SELECT meaning.meaningPhraseID,
GROUP_CONCAT(word.string
ORDER BY phrase_word.wordIndex
SEPARATOR ' ') AS meaningPhraseString
FROM word
JOIN phrase_word ON word.wordID=phrase_word.wordID
JOIN meaning ON phrase_word.phraseID=meaning.meaningPhraseID
JOIN item AS subItem ON meaning.itemID=subItem.itemID
GROUP BY meaning.meaningPhraseID
) AS mp ON m.meaningPhraseID = mp.meaningPhraseID
GROUP BY i.itemID, s.level, sp.studyPhraseString
ORDER BY i.itemID, s.level, sp.studyPhraseString
Your question seems to be this:
how to concat the words into phrases AND concat the phrases into one string
Let's break it down. You need to join together five tables. Three of them are physical tables, namely item, meaning, and status. From those tables you get the references to the result set items you need called itemID and level, and you get the relationship between items and their meanings.
The other two tables you need are virtual tables (that is, subqueries). One of these gives you your French language phrases, and the other gives you your English-language translations.
Let us create the two queries for the virtual tables. Let's put the words into phrases first. A query like this achieves that goal.
SELECT subItem.studyPhraseID,
GROUP_CONCAT(word.string
ORDER BY phrase_word.wordIndex
SEPARATOR ' ') AS studyPhraseString
FROM word
INNER JOIN phrase_word ON word.wordID=phrase_word.wordID
INNER JOIN item AS subItem ON phrase_word.phraseID=subItem.studyPhraseID
GROUP BY subItem.studyPhraseID
This gives you a resultset table of phrase ID numbers and the phrases. Here's a SQL fiddle for this one based on your samples. http://sqlfiddle.com/#!2/11ae2/9/0
Then, create a similar query giving you the meaningPhraseString values.
SELECT meaning.meaningPhraseID,
GROUP_CONCAT(word.string
ORDER BY phrase_word.wordIndex
SEPARATOR ' ') AS meaningPhraseString
FROM word
INNER JOIN phrase_word ON word.wordID=phrase_word.wordID
INNER JOIN meaning ON phrase_word.phraseID=meaning.meaningPhraseID
INNER JOIN item AS subItem ON meaning.itemID=subItem.itemID
GROUP BY meaning.meaningPhraseID
This gives a list of ids and meaning phrases. Here's the fiddle. http://sqlfiddle.com/#!2/11ae2/6/0
So, we're going to need a five-way join (three physical tables and two subqueries) to get to our final result set. In summary, it will look like this:
SELECT i.itemID,
s.level,
sp.studyPhraseString,
mp.meaningPhraseString
FROM item AS i
JOIN meaning AS m ON i.itemID = m.itemID
JOIN status AS s ON i.itemID = s.itemID
JOIN (
/* the studyPhrase subquery */
) AS sp ON i.studyPhraseID = sp.studyPhraseID
JOIN (
/* the meaningPhrase subquery */
) AS mp ON m.meaningPhraseID = mp.meaningPhraseID
The trick here is that you can use a query (or virtual table) and a physical table interchangeably. So when you need to summarize a bunch of tables, you create a query to do that and then paste it into JOIN (/*query*/) AS alias.
Finally, you need to create the comma-joined strings (e.g. to fly, to steal) by adding yet another GROUP_CONCAT() and GROUP BY to your query. The end result is then
SELECT i.itemID,
s.level,
sp.studyPhraseString,
GROUP_CONCAT(mp.meaningPhraseString
SEPARATOR ', ') AS meaningPhraseStrings
FROM item AS i
JOIN meaning AS m ON i.itemID = m.itemID
JOIN status AS s ON i.itemID = s.itemID
JOIN (
/* the studyPhrase subquery */
) AS sp ON i.studyPhraseID = sp.studyPhraseID
JOIN (
/* the meaningPhrase subquery */
) AS mp ON m.meaningPhraseID = mp.meaningPhraseID
GROUP BY i.itemID, s.level, sp.studyPhraseString
ORDER BY i.itemID, s.level, sp.studyPhraseString
And that is your query. http://sqlfiddle.com/#!2/11ae2/16/0 It definitely takes advantage of the Structured in Structured Query Language.