I have a little problem with optimizing a query, I have 2 tables, one which records the participation (participation) in a quiz, and the other which records the answer to each question (participation_rep), participation is linked to the campaign table.
SELECT count(DISTINCT p.id) as number_of_participation
FROM participation_rep prep
INNER JOIN participation p
ON p.id = prep.id_participation
AND p.trash <> 1
WHERE prep.id_question IN (780,787,794,801,809)
AND prep.trash <> 1
GROUP BY pp.id_campaign
Explain of the query
And the problem is that this request is very heavy to execute when there is a lot of data which is concerned by the request and I do not know how to optimize it.
This query take 30-50ms to execute.
Structure of table participation :
CREATE TABLE IF NOT EXISTS `participation` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`id_campagne` int(11) NOT NULL,
`id_identifiant` int(11) DEFAULT NULL,
`firstname` varchar(255) DEFAULT NULL,
`surname` varchar(255) DEFAULT NULL,
`email` varchar(255) DEFAULT NULL,
`date_p` date NOT NULL,
`hour_p` time NOT NULL,
`comment` text,
`trash` tinyint(1) DEFAULT '0',
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Structure of table participation_rep :
CREATE TABLE IF NOT EXISTS `participation_rep` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`id_participation` int(11) NOT NULL,
`id_question` int(11) NOT NULL,
`id_rep` int(11) NOT NULL,
`trash` tinyint(1) DEFAULT '0',
PRIMARY KEY (`id`),
UNIQUE KEY `id_participation` (`id_participation`,`id_question`,`id_reponse`),
KEY `id_question` (`id_question`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Related
I am trying to execute following query
SELECT
a.sessionID AS `sessionID`,
firstSeen, birthday, gender,
isAnonymous, LanguageCode
FROM transactions AS trx
INNER JOIN actions AS a ON a.sessionID = trx.SessionID
WHERE a.ActionType = 'PURCHASE'
GROUP BY trx.TransactionNumber
Explain provides the following output
1 SIMPLE trx ALL TransactionNumber,SessionID NULL NULL NULL 225036 Using temporary; Using filesort
1 SIMPLE a ref sessionID sessionID 98 infinitiExport.trx.SessionID 1 Using index
The problem is that I am trying to use one field for join and different field for GROUP BY.
How can I tell MySQL to use different indices for same table?
CREATE TABLE `transactions` (
`SessionID` varchar(32) NOT NULL DEFAULT '',
`date` datetime DEFAULT NULL,
`TransactionNumber` varchar(32) NOT NULL DEFAULT '',
`CustomerECommerceTrackID` int(11) DEFAULT NULL,
`SKU` varchar(45) DEFAULT NULL,
`AmountPaid` double DEFAULT NULL,
`Currency` varchar(10) DEFAULT NULL,
`Quantity` int(11) DEFAULT NULL,
`Name` tinytext NOT NULL,
`Category` varchar(45) NOT NULL DEFAULT '',
`customerInfoXML` text,
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`id`),
KEY `TransactionNumber` (`TransactionNumber`),
KEY `SessionID` (`SessionID`)
) ENGINE=InnoDB AUTO_INCREMENT=212007 DEFAULT CHARSET=utf8;
CREATE TABLE `actions` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`sessionActionDate` datetime DEFAULT NULL,
`actionURL` varchar(255) DEFAULT NULL,
`sessionID` varchar(32) NOT NULL DEFAULT '',
`ActionType` varchar(64) DEFAULT NULL,
`CustomerID` int(11) DEFAULT NULL,
`IPAddressID` int(11) DEFAULT NULL,
`CustomerDeviceID` int(11) DEFAULT NULL,
`customerInfoXML` text,
PRIMARY KEY (`id`),
KEY `ActionType` (`ActionType`),
KEY `CustomerDeviceID` (`CustomerDeviceID`),
KEY `sessionID` (`sessionID`)
) ENGINE=InnoDB AUTO_INCREMENT=15042833 DEFAULT CHARSET=utf8;
Thanks
EDIT 1: My indexes were broken, I had to add (SessionID, TransactionNumber) index to transactions table, however now, when I try to include trx.customerInfoXML table mysql stops using index
EDIT 2 Another answer does not really solved my problem because it's not standard sql syntax and generally not a good idea to force indices.
For ORM users such syntax is a unattainable luxury.
EDIT 3 I updated my indices and it solved the problem, see EDIT 1
We have a host management system (let's call it CMDB), and a DNS system, each using different tables. The former syncs to the latter, but manual changes cause them to get out of sync. I would like to craft a query to find aliases in CMDB that do NOT have a matching entry in DNS (either no entry, or the name/IP is different)
Because of the large size of the tables, and the need for this query to run frequently, optimizing the query is very important.
Here's what the tables look like:
cmdb_record: id, ipaddr
cmdb_alias: record_id, host_alias
dns_entry: name, ipaddr
cmdb_alias.record_id is a foreign key from cmdb_record.id, so that one IP address can have multiple aliases.
So far, here's what I've come up with:
SELECT cmdb_alias.host_alias, cmdb_record.ipaddr
FROM cmdb_record
INNER JOIN cmdb_alias ON cmdb_alias.record_id = cmdb_record.id
LEFT JOIN dns_entry
ON dns_entry.ipaddr = cmdb_record.ipaddr
AND dns_entry.name = cmdb_alias.host_alias
WHERE dns_entry.ipaddr IS NULL OR dns_entry.name IS NULL
This seems to work, but takes a very long time to run. Is there a better way to do this? Thanks!
EDIT: As requested, here are the SHOW CREATE TABLEs. There are lots of extra fields that aren't particularly relevant, but included for completeness.
Create Table: CREATE TABLE `cmdb_record` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`ip_version` int(11) DEFAULT NULL,
`ipaddr` varchar(40) DEFAULT NULL,
`ipaddr_numeric` decimal(40,0) DEFAULT NULL,
`block_id` int(11) NOT NULL,
`record_commented` tinyint(1) NOT NULL,
`mod_time` datetime NOT NULL,
`deleted` tinyint(1) NOT NULL,
`deleted_date` datetime DEFAULT NULL,
`record_owner` varchar(50) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `ipaddr` (`ipaddr`),
KEY `cmdb_record_fe30f0f7` (`ipaddr`),
KEY `cmdb_record_2b8b575` (`ipaddr_numeric`),
KEY `cmdb_record_45897ef2` (`block_id`),
CONSTRAINT `block_id_refs_id_ed6ed320` FOREIGN KEY (`block_id`) REFERENCES `cmdb_block` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=104427 DEFAULT CHARSET=latin1
Create Table: CREATE TABLE `cmdb_alias` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`host_alias` varchar(255) COLLATE latin1_general_cs NOT NULL,
`record_id` int(11) NOT NULL,
`record_order` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `cmdb_alias_fcffc3bb` (`record_id`),
KEY `alias_lookup` (`host_alias`),
CONSTRAINT `record_id_refs_id_8169fc71` FOREIGN KEY (`record_id`) REFERENCES `cmdb_record` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=155433 DEFAULT CHARSET=latin1 COLLATE=latin1_general_cs
Create Table: CREATE TABLE `dns_entry` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`rec_grp_id` varchar(40) NOT NULL,
`parent_id` int(11) NOT NULL,
`domain_id` int(11) DEFAULT NULL,
`name` varchar(255) DEFAULT NULL,
`type` varchar(6) DEFAULT NULL,
`ipaddr` varchar(255) DEFAULT NULL,
`ttl` int(11) DEFAULT NULL,
`prio` int(11) DEFAULT NULL,
`status` varchar(20) NOT NULL,
`op` varchar(20) NOT NULL,
`mod_time` datetime NOT NULL,
`whodunit` varchar(50) NOT NULL,
`comments` longtext NOT NULL,
PRIMARY KEY (`id`),
KEY `dns_entry_a2431ea` (`domain_id`),
KEY `dns_entry_52094d6e` (`name`)
) ENGINE=InnoDB AUTO_INCREMENT=49437 DEFAULT CHARSET=utf8
If you don't have one already, create an index on dns_entry(ipaddr, name). This might be all you need to speed the query.
i have such 3 tables. Im trying to Left Join them
SELECT `t`.`title` AS `category_title`,`t`.`id` AS `category_id`, `st`.`title` AS
`subcategory_title`, `st`.`id` AS `subcategory_id`, `st`.`parent_id` AS
`subcategory_parent`, `n`.`title` AS `news_title`,`n`.`id` AS `news_id` FROM
`t_categories` `t` LEFT JOIN t_categories AS `st` ON `st`.`parent_id`=t.`id` LEFT JOIN
t_newsrelations AS `nr` ON `nr`.`category_id`=st.`id` LEFT JOIN t_news AS `n` ON
`n`.`id`=nr.`news_id` WHERE `t`.`enabled` = 1 AND `n`.`enabled` = 1 AND `n`.`type`!=1 AND
`n`.`type`!=5 ORDER BY `t`.`position`,`st`.`position`,`n`.`position` ASC
here is the structure of tables
CREATE TABLE IF NOT EXISTS `t_categories` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`parent_id` int(11) NOT NULL,
`title` tinytext NOT NULL,
`position` tinyint(4) unsigned NOT NULL DEFAULT '0',
`type` tinyint(1) unsigned NOT NULL,
`enabled` tinyint(1) unsigned NOT NULL DEFAULT '1',
UNIQUE KEY `id` (`id`),
KEY `type` (`type`),
KEY `parent_id` (`parent_id`),
KEY `enabled` (`enabled`),
KEY `id_parent_position_enabled` (`id`,`parent_id`,`position`,`enabled`),
KEY `position` (`position`),
KEY `parent_id_2` (`parent_id`,`enabled`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ;
CREATE TABLE IF NOT EXISTS `t_news` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`title` tinytext NOT NULL,
`m_title` tinytext NOT NULL,
`url` varchar(2000) NOT NULL,
`keywords` text NOT NULL,
`description` text NOT NULL,
`body` longtext NOT NULL,
`position` tinyint(4) unsigned NOT NULL DEFAULT '0',
`type` tinyint(1) unsigned NOT NULL,
`city_id` int(4) NOT NULL,
`quickmenu_enabled` tinyint(1) unsigned NOT NULL DEFAULT '0',
`quickmenu` text NOT NULL,
`enabled` tinyint(1) unsigned NOT NULL DEFAULT '1',
PRIMARY KEY (`id`),
UNIQUE KEY `id` (`id`),
KEY `position` (`position`),
KEY `type` (`type`),
KEY `city_id` (`city_id`),
KEY `url` (`url`(333)),
KEY `quickmenu_enabled` (`quickmenu_enabled`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ;
CREATE TABLE IF NOT EXISTS `t_newsrelations` (
`category_id` int(11) NOT NULL,
`news_id` int(11) unsigned NOT NULL,
KEY `category_id` (`category_id`),
KEY `news_id` (`news_id`),
KEY `category_id_2` (`category_id`,`news_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
And SELECT EXPLAIN shows me
t_newsrelations is intermediate table. Table t_categories contains categories and subcategories linked by parent_id column. Each item from t_news can be a member of more than one subcategories thats why they linked through t_newsrelations
how to optimise a query? Why it shows Using index, Using temporary, Using filesort?
ORDER BY `t`.`position`,`st`.`position`,`n`.`position` ASC
You can't eliminate the temp table and filesort in this query, given the tables you have, because you're sorting on columns from multiple tables. Optimizing sorting means using an index so that the query fetches rows in the order you want them. But there's no way in MySQL to create an index that spans multiple tables.
The only way to fix this is to denormalize until all three columns are in a single table, and create one index over the three columns. But denormalization comes with its own downsides.
Is this the most efficient way of joining these 4 tables? Also is it possible to only have some rows of each tables selected? I tried changing * to a name of a column but only the columns from studentlist are allowed.
SELECT c.classID, c.instrument, c.grade, u.ID, u.firstname, u.lastname, u.lastsongplayed, u.title
FROM studentlist s
INNER JOIN classlist c ON s.listID = c.classID
INNER JOIN (
SELECT *
FROM users u
INNER JOIN library l ON u.lastsongplayed = l.fileID
)
u ON s.studentID = u.ID
WHERE teacherID =3
ORDER BY classID
LIMIT 0 , 30
Database structure:
CREATE TABLE IF NOT EXISTS `classlist` (
`classID` int(11) NOT NULL AUTO_INCREMENT,
`teacherID` int(11) NOT NULL,
`instrument` text,
`grade` int(11) DEFAULT NULL,
PRIMARY KEY (`classID`),
KEY `teacherID_2` (`teacherID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=27 ;
CREATE TABLE IF NOT EXISTS `studentlist` (
`listID` int(11) NOT NULL,
`studentID` int(11) NOT NULL,
KEY `teacherID` (`studentID`),
KEY `studentID` (`studentID`),
KEY `listID` (`listID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE IF NOT EXISTS `users` (
`ID` int(11) NOT NULL AUTO_INCREMENT,
`email` varchar(60) NOT NULL,
`password` varchar(60) NOT NULL,
`firstname` text NOT NULL,
`lastname` text NOT NULL,
`sessionID` varchar(60) DEFAULT NULL,
`lastlogin` time DEFAULT NULL,
`registerdate` date NOT NULL,
`isteacher` tinyint(1) DEFAULT NULL,
`isstudent` tinyint(1) DEFAULT NULL,
`iscomposer` tinyint(1) DEFAULT NULL,
`lastsongplayed` int(11) NOT NULL,
PRIMARY KEY (`ID`),
UNIQUE KEY `ID` (`ID`),
UNIQUE KEY `email` (`email`,`sessionID`),
KEY `ID_2` (`ID`),
KEY `ID_3` (`ID`),
KEY `lastsongplayed` (`lastsongplayed`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=63 ;
CREATE TABLE IF NOT EXISTS `library` (
`fileID` int(11) NOT NULL AUTO_INCREMENT,
`userID` int(11) NOT NULL,
`uploaddate` datetime NOT NULL,
`title` varchar(60) NOT NULL,
`OrigComposer` varchar(60) NOT NULL,
`composer` varchar(60) NOT NULL,
`genre` varchar(60) DEFAULT NULL,
`year` year(4) DEFAULT NULL,
`arrangement` varchar(60) DEFAULT NULL,
PRIMARY KEY (`fileID`),
KEY `userID` (`userID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=77 ;
Is this the most efficient way of joining these 3 tables?
Your JOIN looks correct and you are joining on your keys. So this should be efficient. However, I would encourage you to analyze your query with EXPLAIN to determine additional optimizations.
Is it possible to only have some rows of each tables selected?
Yes. Change * to be the columns from each table you want. I encourage you to explicitly prefix them with the originating table. Depending on the columns you select, this could also make your query more performant.
SELECT studentlist.studentID, users.email FROM ...
Tables:
nodes
data_texts
data_profiles
data_locations
data_profiles
data_media
data_products
data_metas
categories
tags
categories_nodes
tags_nodes
This question is a generalized question and is on the back of another question
Explanation:
Each of the "data" tables has a node_id that refers back to the id of the nodes table (hasMany/belongsTo association).
A "Node" can be anything - a TV Show, a Movie, a Person, an Article...etc (all generated via a CMS, so the user can control what type of "Nodes" they want).
When pulling data, I want to be able to query against certain fields. For example if they do a search, I want to be able to pull nodes that have data_texts.title = '%george%' or order by the datetime field in data_locations.
The problem is, when I do a join on all seven data tables (or more), the query has to hit so many combined rows that it just times out (even with a nearly empty database.... total 200 rows across the entire database).
I realize I can determine IF I need a join depending on what I'm doing - but even with five or six joins (once the database gets to 10k+ records), it's going to be horribly slow, if it works at all. Per this question, the query I'm using just doing a join on these tables times out completely.
Each node can have multiple rows of each data type (for multi-language reasons among others).
I'm completely defeated - I'm at the point where I think I need to restructure the entire thing, but don't have the time for that. I've thought about combining all into one table, but aren't sure how....etc
nodes
CREATE TABLE `nodes` (
`id` CHAR(36) NOT NULL,
`name` VARCHAR(100) NOT NULL,
`slug` VARCHAR(100) NOT NULL,
`node_type_id` CHAR(36) NOT NULL,
`site_id` CHAR(36) NOT NULL,
`created` DATETIME NOT NULL,
`modified` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `nodeTypeId` (`node_type_id`),
INDEX `slug` (`slug`),
INDEX `nodeId` (`id`)
)
COLLATE='latin1_swedish_ci'
ENGINE=MyISAM;
data_texts:
CREATE TABLE `data_texts` (
`id` CHAR(36) NOT NULL,
`title` VARCHAR(250) NULL DEFAULT NULL,
`subtitle` VARCHAR(500) NULL DEFAULT NULL,
`content` LONGTEXT NULL,
`byline` VARCHAR(250) NULL DEFAULT NULL,
`language_id` CHAR(36) NULL DEFAULT NULL,
`foreign_key` CHAR(36) NULL DEFAULT NULL,
`model` VARCHAR(40) NULL DEFAULT NULL,
`node_id` CHAR(36) NULL DEFAULT NULL,
`created` DATETIME NOT NULL,
`modified` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `nodeId` (`node_id`),
INDEX `languageId_nodeId` (`language_id`, `node_id`),
INDEX `foreignKey_model` (`foreign_key`, `model`)
)
COLLATE='utf8_general_ci'
ENGINE=MyISAM;
data_profiles
CREATE TABLE `data_profiles` (
`id` CHAR(36) NOT NULL,
`name` VARCHAR(80) NULL DEFAULT NULL,
`email_personal` VARCHAR(100) NULL DEFAULT NULL,
`email_business` VARCHAR(100) NULL DEFAULT NULL,
`email_other` VARCHAR(100) NULL DEFAULT NULL,
`title` VARCHAR(100) NULL DEFAULT NULL,
`description` LONGTEXT NULL,
`prefix` VARCHAR(40) NULL DEFAULT NULL,
`phone_home` VARCHAR(40) NULL DEFAULT NULL,
`phone_business` VARCHAR(40) NULL DEFAULT NULL,
`phone_mobile` VARCHAR(40) NULL DEFAULT NULL,
`phone_other` VARCHAR(40) NULL DEFAULT NULL,
`foreign_key` CHAR(36) NULL DEFAULT NULL,
`model` VARCHAR(40) NULL DEFAULT NULL,
`node_id` CHAR(36) NULL DEFAULT NULL,
`language_id` CHAR(36) NULL DEFAULT NULL,
`created` DATETIME NOT NULL,
`modified` DATETIME NOT NULL,
`user_id` CHAR(36) NULL DEFAULT NULL,
PRIMARY KEY (`id`),
INDEX `nodeId` (`node_id`),
INDEX `languageId_nodeId` (`node_id`, `language_id`),
INDEX `foreignKey_model` (`foreign_key`, `model`)
)
COLLATE='latin1_swedish_ci'
ENGINE=MyISAM;
categories
CREATE TABLE `categories` (
`id` CHAR(36) NOT NULL,
`name` VARCHAR(100) NOT NULL,
`node_type_id` CHAR(36) NOT NULL,
`site_id` CHAR(36) NOT NULL,
`slug` VARCHAR(100) NOT NULL,
`created` DATETIME NOT NULL,
`modified` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `nodeTypeId` (`node_type_id`),
INDEX `slug` (`slug`)
)
COMMENT='Used to categorize nodes'
COLLATE='utf8_general_ci'
ENGINE=MyISAM;
categories_nodes
CREATE TABLE `categories_nodes` (
`id` CHAR(36) NOT NULL,
`category_id` CHAR(36) NOT NULL,
`node_id` CHAR(36) NOT NULL,
PRIMARY KEY (`id`),
INDEX `categoryId_nodeId` (`category_id`, `node_id`)
)
COLLATE='utf8_general_ci'
ENGINE=MyISAM;
node_tags
CREATE TABLE `node_tags` (
`id` CHAR(36) NOT NULL,
`name` VARCHAR(40) NOT NULL,
`site_id` CHAR(36) NOT NULL,
`created` DATETIME NOT NULL,
`modified` DATETIME NOT NULL,
PRIMARY KEY (`id`),
INDEX `siteId` (`site_id`)
)
COLLATE='utf8_general_ci'
ENGINE=MyISAM;
nodes_node_tags
CREATE TABLE `nodes_node_tags` (
`id` CHAR(36) NOT NULL,
`node_id` CHAR(36) NOT NULL,
`node_tag_id` CHAR(36) NOT NULL,
PRIMARY KEY (`id`),
INDEX `node_id_node_tag_id` (`node_id`, `node_tag_id`)
)
COLLATE='utf8_general_ci'
ENGINE=MyISAM;
MySQL:
SELECT `Node`.`id`, `Node`.`name`, `Node`.`slug`, `Node`.`node_type_id`, `Node`.`site_id`, `Node`.`created`, `Node`.`modified`
FROM `mysite`.`nodes` AS `Node`
LEFT JOIN `mysite`.`data_date_times` AS `DataDateTime` ON (`DataDateTime`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_locations` AS `DataLocation` ON (`DataLocation`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_media` AS `DataMedia` ON (`DataMedia`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_metas` AS `DataMeta` ON (`DataMeta`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_profiles` AS `DataProfile` ON (`DataProfile`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_products` AS `DataProduct` ON (`DataProduct`.`node_id` = `Node`.`id`)
LEFT JOIN `mysite`.`data_texts` AS `DataText` ON (`DataText`.`node_id` = `Node`.`id`)
WHERE 1=1
GROUP BY `Node`.`id`
Firstly, try InnoDB, not MyISAM.
Secondly, remove the group by, see how well it runs then, and how many rows are involved. Shouldn't be that many, but it's interesting.
You don't need the 'nodeId' index on node (as you already have it as a primary key). Again, shouldn't make any difference.
The where clause is irrelevant. You can remove it with no effect one way or another.
Thirdly, well, something is seriously broken.
Have a quick look on how to start profiling (e.g. http://dev.mysql.com/doc/refman/5.0/en/show-profile.html) , and run a profile command to see where all the time is going. Post it here if it doesn't immediately show that something is broken.
I'm unfortunately not in a position where I can do any tests right now. I'll just throw out some ideas. I might be able to do some tests later.
Be suspicious of different collations.
Some of your ids are useless. For example, you should drop the column categories_nodes.id, and put a primary key constraint on {category_id, node_id} instead.
Be suspicious of any design that requires joining all the tables at run time. There are better ways.
Use innodb and foreign key constraints.