MySQL select with where takes a long time - mysql

I have a table with about 700.000 rows:
CREATE TABLE IF NOT EXISTS `ext_log_entries` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`action` varchar(8) NOT NULL,
`logged_at` datetime NOT NULL,
`object_id` varchar(32) DEFAULT NULL,
`object_class` varchar(255) NOT NULL,
`version` int(11) NOT NULL,
`data` longtext COMMENT '(DC2Type:array)',
`username` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `log_date_lookup_idx` (`logged_at`),
KEY `log_user_lookup_idx` (`username`),
KEY `log_class_lookup_idx` (`object_class`),
KEY `log_version_lookup_idx` (`object_id`,`object_class`,`version`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1219777 ;
I try to run the following query:
SELECT n0_.id AS id0, n0_.action AS action1, n0_.logged_at AS logged_at2, n0_.object_id AS object_id3, n0_.object_class AS object_class4, n0_.version AS version5, n0_.data AS data6, n0_.username AS username7
FROM ext_log_entries n0_
WHERE n0_.object_id =275634
AND n0_.object_class = 'My\\MyBundle\\Entity\\Field'
AND n0_.version <=1
ORDER BY n0_.version ASC
Here is the MySQL plan:
id 1
select_type SIMPLE
table n0_
type ref
possible_keys log_class_lookup_idx,log_version_lookup_idx
key log_class_lookup_idx
key_len 767
ref const
rows 641159
Extra Using where; Using filesort
My query need about 37 seconds to be executed for only 1 row in the result...
I tried to run the same query by deleting my indexes and it goes a little bit faster : about 31 seconds...
I don't understand why my query is taking so much time and why my indexes don't help the performance? Do you know how I can do to have good performance on this query?
Thanks in advance for your help !
EDIT
Here are the cardinalties of the indexes
log_date_lookup_idx BTREE logged_at 1221578 A
log_user_lookup_idx BTREE username 40 A YES
log_class_lookup_idx BTREE object_class 1010 A
log_version_lookup_idx BTREE object_id 1221578 A YES
object_class 1221578 A
version 1221578 A

I found a solution, not THE solution, but at least it works for me.
I think it could help anyway all people who are using gedmo loggable and who are lucky (like me) to have objects with only integers IDs.
I changes my column object_id to integer instead of varchar(255). My query now take 0.008 second ! It works for me because i'm sure i'll always have only integers, for people who have varchar, I'm sorry i tried many things but nothing worked....
CREATE TABLE IF NOT EXISTS `ext_log_entries` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`action` varchar(8) NOT NULL,
`logged_at` datetime NOT NULL,
`object_id` int(11) DEFAULT NULL,
`object_class` varchar(255) NOT NULL,
`version` int(11) NOT NULL,
`data` longtext COMMENT '(DC2Type:array)',
`username` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `log_date_lookup_idx` (`logged_at`),
KEY `log_user_lookup_idx` (`username`),
KEY `log_class_lookup_idx` (`object_class`),
KEY `log_version_lookup_idx` (`object_id`,`object_class`,`version`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1219777 ;

Related

Improving the performance of a MYSQL query with a one-to-many relationship

I have a query in my DB that is taking 25 seconds to return results, which is way too long. It seems like it should be pretty simple. Two tables; the main table (document) is a standard table with some data columns, the join table is a mapping table with only two columns (parent_id, division_id). Previously there wasn't an index on the mapping table so I added one and that changed the "explain" to include the index but doesn't seem to have had an impact on the performance.
The query looks like this:
explain SELECT DISTINCT doc.*
FROM document doc
LEFT JOIN multi_division_mapper divisions ON doc.id = divisions.parent_id
WHERE doc.clientId = 'SOME_GUID'
AND (divisions.division_id IS NULL OR divisions.division_id IN ('SOME_GUID'));
and the results of explain are:
Total number of rows in document: 6720
Total number of rows in mapper: 6173
From what I've been able to gather I need to improve either the "type" or the "extra" to make the query faster. What can I do here?
Create table statements:
CREATE TABLE `document` (
`id` varchar(36) NOT NULL,
`addedBy` varchar(255) DEFAULT NULL,
`addedDate` datetime NOT NULL,
`editedBy` varchar(255) DEFAULT NULL,
`editedDate` datetime NOT NULL,
`deleted` bit(1) DEFAULT NULL,
`clientId` varchar(36) NOT NULL,
`departmentId` varchar(36) DEFAULT NULL,
`documentParentId` varchar(36) DEFAULT NULL,
`documentParent` varchar(50) DEFAULT NULL,
`fileId` varchar(255) DEFAULT NULL,
`fileUrl` varchar(600) DEFAULT NULL,
`documentName` varchar(500) NOT NULL,
`displayName` varchar(255) NOT NULL,
`documentId` varchar(45) DEFAULT NULL,
`notes` varchar(1000) DEFAULT NULL,
`visibility` varchar(45) NOT NULL DEFAULT 'PRIVATE',
`documentType` varchar(45) NOT NULL,
`restrictDelete` bit(1) NOT NULL,
`customData` text,
`releaseDate` datetime NOT NULL,
`expirationDate` datetime NOT NULL,
`isApproved` bit(1) NOT NULL DEFAULT b'0',
`userSupplier` varchar(36) DEFAULT NULL,
`complianceCertificateId` varchar(36) DEFAULT NULL,
`Status` varchar(50) DEFAULT 'NEUTRAL',
PRIMARY KEY (`id`),
KEY `idx_client` (`clientId`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `multi_division_mapper` (
`parent_id` varchar(36) NOT NULL,
`division_id` varchar(36) NOT NULL,
PRIMARY KEY (`parent_id`,`division_id`),
KEY `idx_parent` (`parent_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
I was able to get a more favorable EXPLAIN report in a test by creating the following index:
ALTER TABLE multi_division_mapper
DROP INDEX idx_parent,
ADD INDEX (division_id, parent_id);
I also dropped idx_parent because it's redundant; it's a prefix of the primary key.
id
select_type
table
partitions
type
possible_keys
key
key_len
ref
rows
filtered
Extra
1
SIMPLE
doc
NULL
ref
idx_client
idx_client
110
const
1
100.00
Using temporary
1
SIMPLE
divisions
NULL
ref
PRIMARY,division_id
division_id
38
const
1
100.00
Using where; Using index; Distinct
The type: ref is better than type: index.
The query I tested is slightly different, but I believe it returns the same result:
SELECT DISTINCT doc.*
FROM document doc
LEFT JOIN multi_division_mapper divisions
ON doc.id = divisions.parent_id AND divisions.division_id in ('SOME_GUID')
WHERE doc.clientId = 'SOME_GUID'

MYSQL - How to fix slow query that is not using indexes, how to speed up response time?

I have read several other posts and pages and i can't figure a way to speed up and use an index instead of a full table scan
Query in question
select f.*,ci.item_name
from forecastpro.current_item_forecast f
left join cat_items as ci on f.item_number=ci.item_number
Tables Structures
CREATE TABLE `current_item_forecast` (
`Axe3` varchar(128) DEFAULT NULL,
`Axe5` varchar(128) DEFAULT NULL,
`Axe7` varchar(128) DEFAULT NULL,
`Axe11` varchar(128) DEFAULT NULL,
`Item_Number` varchar(40) NOT NULL,
`Item_Name` varchar(128) DEFAULT NULL,
`fc_year` int(4) NOT NULL,
`fc_period` int(2) NOT NULL,
`Qty_Forecast` double NOT NULL DEFAULT '0',
`USD_Forecast` double NOT NULL DEFAULT '0',
`Stock_Start` float NOT NULL DEFAULT '0',
`Stock_Transit` float NOT NULL DEFAULT '0',
PRIMARY KEY (`Item_Number`,`fc_year` DESC,`fc_period`),
KEY `Item` (`Item_Number`),
KEY `Axe3` (`Axe3`),
KEY `Axe5` (`Axe5`),
KEY `Axe7` (`Axe7`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
CREATE TABLE `cat_items` (
`Item_Number` char(32) NOT NULL,
`Item_Name` char(128) DEFAULT NULL,
`Axe3` char(10) DEFAULT NULL,
`Item_Group` varchar(32) DEFAULT NULL,
`Axe5` char(10) DEFAULT NULL,
`Axe7` char(10) DEFAULT NULL,
`Axe11` char(10) DEFAULT NULL,
`Date_Created` datetime DEFAULT NULL,
PRIMARY KEY (`Item_Number`),
KEY `Item` (`Item_Number`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
Explain
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE f ALL 121882 100.00
1 SIMPLE ci ALL 36838 100.00 Using where; Using join buffer (Block Nested Loop)
Other Attempts
**This two produce the same result:**
update forecastpro.current_item_forecast f
left join cupfsa.cat_items ci on f.item_number=ci.item_number
set f.item_name=ci.item_name
update forecastpro.current_item_forecast f
left join cupfsa.cat_items ci FORCE KEY FOR JOIN (PRIMARY) on f.item_number=ci.item_number
set f.item_name=ci.item_name
id select_type table type possible_keys key key_len ref rows Extra
1 UPDATE f ALL 76953 100.00
1 SIMPLE ci ALL 36838 100.00 Using where
Any attempt to run simply runs forever.
What would you recomend?
For your first select query, you may try adding the following index to the cat_items table:
CREATE INDEX idx ON cat_items (item_number, item_name);
This ideally should speed up the left join process, as each record in the current_item_forecast table gets matched to the cat_items table using the item_number as a lookup. Note that we also include the item_name in the index so as to cover the SELECT clause.
(Thank you for supplying the CREATE TABLEs; the root cause in your problem would have been very hard to figure out without them.)
Do not mix CHARACTER SETs, at least not when JOINing by varchars. It leads to a costly table scan.
current_item_forecast:
`Item_Number` varchar(40) NOT NULL ... DEFAULT CHARSET=utf8mb4
cat_items:
`Item_Number` char(32) ... DEFAULT CHARSET=utf8
Be consistent in lengths (40 vs 32).
Since PRIMARY KEY is a unique key, don't redundantly include KEY for the same column:
PRIMARY KEY (`Item_Number`),
KEY `Item` (`Item_Number`)
Use VARCHAR unless the column is truly fixed width.

Why is MySQL showing index_merge on this query?

I have what seems like a fairly simple table structure, however MySQL is defaulting to a less than optimal index_merge on a simple query.
Here's the table structure:
CREATE TABLE IF NOT EXISTS `event_log` (
`event_id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(5) DEFAULT NULL,
`location_id` int(10) DEFAULT NULL,
`object_id` int(5) DEFAULT NULL,
`action_id` int(5) DEFAULT NULL,
`date_event` datetime DEFAULT NULL,
PRIMARY KEY (`event_id`),
KEY `user_id` (`user_id`),
KEY `date_event` (`date_event`),
KEY `action_id` (`action_id`),
KEY `object_id` (`object_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
EXPLAIN on a basic SELECT query
EXPLAIN SELECT date_event
FROM event_log
WHERE user_id =123
AND object_id =456
AND location_id =789
Returns this:
select_type table type possible_keys key key_len ref rows Extra
SIMPLE event_log index_merge user_id,object_id object_id,user_id 5,5 NULL 27 Using intersect(object_id,user_id); Using where
Here's the Extra bit, for easier reading:
Using intersect(object_id,user_id); Using where
Why is MySQL not using standard indexes on this query? Why is it intersecting user_id and object_id?
The most effective index for the query is a composite index that includes all three fields, for example: (object_id, user_id, location_id). Since there is no such index, MySQL does its best to get most of the information from existing indexes.

Why do i HAVE to optimize tables?

I have a pretty big table with contains about 3 million records.
When running a very simple query, joining this table on a few others (all with indexes and/or primary keys), the query will take about 25 seconds to complete!
The value of "Handler_read_next" is about 7 million!
Number of requests to read the next row in key order, incremented if you are querying an index column with a range constraint or if you are doing an index scan.
This problem have only started since this table began to grow big.
Now if I do an "optimize tables" on this table, the query will run in about 0.02 seconds and "Handler_read_next" will have a value of about 1500.
How can the difference be so extreme, and do I really have to setup a scheduled query, optimizing this table once a week or so? Even so, I would like to know the meaning behind this and why mysql behaves like this. Sure, rows are deleted and updated pretty much in this table, but should it get so badly fragmented in only one week that the query goes from 0.02 sec to 25 sec?
Edit: After request, here comes the query in question:
SELECT *
FROM budget_expenses
JOIN budget_categories
ON budget_categories.BudgetAreaId = budget_expenses.BudgetAreaId
AND budget_categories.BudgetCategoryId = budget_expenses.BudgetCategoryId
LEFT JOIN budget_types
ON budget_types.BudgetAreaId = budget_expenses.BudgetAreaId
AND budget_types.BudgetCategoryId = budget_expenses.BudgetCategoryId
AND budget_types.BudgetTypeId = budget_expenses.BudgetTypeId
WHERE budget_expenses.BudgetId = 1
AND budget_expenses.ExpenseDate >= '2012-11-25'
AND budget_expenses.ExpenseDate <= '2012-12-24'
AND budget_expenses.BudgetAreaId = 2
ORDER BY budget_expenses.ExpenseDate DESC,
budget_expenses.ExpenseTime IS NULL ASC,
budget_expenses.ExpenseTime DESC
(BudgetAreaId, BudgetCategoryId) is the primary key in budget_categories and (BudgetAreaId, BudgetCategoryId, BudgetTypeId) is the primary key in budget_types. In budget_expenses these 3 keys are indexes and also ExpenseDate has an index. This query returns about 20 rows.
Show create table:
CREATE TABLE `budget_areas` (
`BudgetAreaId` int(11) NOT NULL,
`Name` varchar(255) DEFAULT NULL,
PRIMARY KEY (`BudgetAreaId`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
CREATE TABLE `budget_categories` (
`BudgetAreaId` int(11) NOT NULL,
`BudgetCategoryId` int(11) NOT NULL AUTO_INCREMENT,
`Name` varchar(255) DEFAULT NULL,
`SortOrder` int(11) DEFAULT NULL,
PRIMARY KEY (`BudgetAreaId`,`BudgetCategoryId`),
KEY `BudgetAreaId` (`BudgetAreaId`,`BudgetCategoryId`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
CREATE TABLE `budget_types` (
`BudgetAreaId` int(11) NOT NULL,
`BudgetCategoryId` int(11) NOT NULL,
`BudgetTypeId` int(11) NOT NULL,
`Name` varchar(255) DEFAULT NULL,
`SortId` int(11) DEFAULT NULL,
PRIMARY KEY (`BudgetAreaId`,`BudgetCategoryId`,`BudgetTypeId`),
KEY `BudgetAreaId` (`BudgetAreaId`,`BudgetCategoryId`,`BudgetTypeId`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
CREATE TABLE `budget_expenses` (
`ExpenseId` int(11) NOT NULL AUTO_INCREMENT,
`BudgetId` int(11) NOT NULL,
`TempId` int(11) DEFAULT NULL,
`BudgetAreaId` int(11) DEFAULT NULL,
`BudgetCategoryId` int(11) DEFAULT NULL,
`BudgetTypeId` int(11) DEFAULT NULL,
`Company` varchar(255) DEFAULT NULL,
`ImportCompany` varchar(255) DEFAULT NULL,
`Sum` double(50,2) DEFAULT NULL,
`ExpenseDate` date DEFAULT NULL,
`ExpenseTime` time DEFAULT NULL,
`Inserted` datetime DEFAULT NULL,
`Changed` datetime DEFAULT NULL,
`InsertType` int(1) DEFAULT NULL,
`AccountId` int(11) DEFAULT NULL,
`BankCardId` int(11) DEFAULT NULL,
PRIMARY KEY (`ExpenseId`),
KEY `BudgetId` (`BudgetId`),
KEY `AccountId` (`AccountId`),
KEY `Company` (`Company`) USING BTREE,
KEY `ExpenseDate` (`ExpenseDate`),
KEY `BudgetAreaId` (`BudgetAreaId`),
KEY `BudgetCategoryId` (`BudgetCategoryId`),
KEY `BudgetTypeId` (`BudgetTypeId`),
CONSTRAINT `budget_expenses_ibfk_1` FOREIGN KEY (`BudgetId`) REFERENCES `budgets` (`BudgetId`)
) ENGINE=InnoDB AUTO_INCREMENT=3604462 DEFAULT CHARSET=latin1
After I copy pasted this I changed from MyIsam to Innodb on the budget_categories table.
Edit: The change from myisam to innodb didn't make any difference. The query is now very slow, just 12 hours after i optimized the budget_expenses table!
Here is the explain for the query which now takes about 9 seconds:
http://jsfiddle.net/dmVPY/1/
Ahhh MyISAM....
Try changing the table type (aka 'storage engine') to InnoDB instead.
If you do this, make sure innodb_buffer_pool_size in your my.cnf is a sensible value - the default is too small.

How to set correct index for this MySql query?

I have this query showing up in MySql slow query log. (It is not slow, but it is not using indexes right). I need some help on how to set up the index right.
SELECT tbladded.amount*SUM(tbladdeditem.amount)
FROM tbladded
INNER JOIN tbladdeditem ON tbladded.addedid = tbladdeditem.addedid AND tbladdeditem.deleted='False'
WHERE tbladded.userid=100
AND tbladded.date='2012-01-01'
AND tbladded.deleted='False'
GROUP BY tbladded.addedid
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE tbladded ref PRIMARY,userid_date userid_date 8 const,const 1 Using where
1 SIMPLE tbladdeditem ref addedid addedid 5 tbladded.addedid 1 Using where
This is how the tables look like:
CREATE TABLE `tbladded` (
`addedid` int(11) NOT NULL AUTO_INCREMENT,
`amount` double DEFAULT NULL,
`date` date DEFAULT NULL,
`userid` mediumint(9) DEFAULT NULL,
`deleted` enum('False','True') CHARACTER SET latin1 DEFAULT 'False',
PRIMARY KEY (`addedid`),
KEY `userid_date` (`userid`,`date`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE `tbladdeditem` (
`addeditemid` int(11) NOT NULL AUTO_INCREMENT,
`amount` double DEFAULT NULL,
`addedid` int(11) DEFAULT NULL,
`userid` mediumint(9) DEFAULT NULL,
`deleted` enum('False','True') CHARACTER SET latin1 DEFAULT 'False',
PRIMARY KEY (`addeditemid`),
KEY `addedid` (`addedid`),
KEY `userid` (`userid`),
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
try this:
ALTER TABLE `tbladded` ADD INDEX
`tbladdedIndex` (`userid`, `date`, `deleted`);