I am trying to optimize the below query it yields me the following result
EXPLAIN UPDATE table1 a
JOIN table2 b
ON a.id=b.ref_id
SET a.value=b.value;
When i try to explain the query, I get the following result.
ID select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE b ALL idx_ref_id 10785
1 SIMPLE a ref idx_id idx_id 4 b.ref_id 24
As far as i understand query is using only one index i.e idx_id and not idx_ref_id.
Edit:
Here is the table structure.
SHOW CREATE TABLE table1;
CREATE TABLE table_1 (
`id` int(11) NOT NULL DEFAULT '0',
`companyname` varchar(100) NOT NULL DEFAULT '',
`value` varchar(100) NOT NULL DEFAULT '',
`sve_value` varchar(100) NOT NULL DEFAULT '',
KEY `idx_id` (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
SHOW CREATE TABLE table2;
CREATE TABLE `table2` (
`value` varchar(255) DEFAULT NULL,
`suggested_value_enrichement` varchar(255) DEFAULT NULL,
`tags` varchar(255) DEFAULT NULL,
`ref_id` int(11) NOT NULL DEFAULT '0',
KEY `idx_ref_id` (`ref_id`),
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Please help me, Also please correct me if am wrong.
Thanks.
Issue with query was there were many process running on the server which choked the resources. Also there were many ideal connection to the database to precise around 150.
So by killing the process and terminating the ideal connection solved the issue.
Query execution finished within a minute.
Related
I have read several other posts and pages and i can't figure a way to speed up and use an index instead of a full table scan
Query in question
select f.*,ci.item_name
from forecastpro.current_item_forecast f
left join cat_items as ci on f.item_number=ci.item_number
Tables Structures
CREATE TABLE `current_item_forecast` (
`Axe3` varchar(128) DEFAULT NULL,
`Axe5` varchar(128) DEFAULT NULL,
`Axe7` varchar(128) DEFAULT NULL,
`Axe11` varchar(128) DEFAULT NULL,
`Item_Number` varchar(40) NOT NULL,
`Item_Name` varchar(128) DEFAULT NULL,
`fc_year` int(4) NOT NULL,
`fc_period` int(2) NOT NULL,
`Qty_Forecast` double NOT NULL DEFAULT '0',
`USD_Forecast` double NOT NULL DEFAULT '0',
`Stock_Start` float NOT NULL DEFAULT '0',
`Stock_Transit` float NOT NULL DEFAULT '0',
PRIMARY KEY (`Item_Number`,`fc_year` DESC,`fc_period`),
KEY `Item` (`Item_Number`),
KEY `Axe3` (`Axe3`),
KEY `Axe5` (`Axe5`),
KEY `Axe7` (`Axe7`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
CREATE TABLE `cat_items` (
`Item_Number` char(32) NOT NULL,
`Item_Name` char(128) DEFAULT NULL,
`Axe3` char(10) DEFAULT NULL,
`Item_Group` varchar(32) DEFAULT NULL,
`Axe5` char(10) DEFAULT NULL,
`Axe7` char(10) DEFAULT NULL,
`Axe11` char(10) DEFAULT NULL,
`Date_Created` datetime DEFAULT NULL,
PRIMARY KEY (`Item_Number`),
KEY `Item` (`Item_Number`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
Explain
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE f ALL 121882 100.00
1 SIMPLE ci ALL 36838 100.00 Using where; Using join buffer (Block Nested Loop)
Other Attempts
**This two produce the same result:**
update forecastpro.current_item_forecast f
left join cupfsa.cat_items ci on f.item_number=ci.item_number
set f.item_name=ci.item_name
update forecastpro.current_item_forecast f
left join cupfsa.cat_items ci FORCE KEY FOR JOIN (PRIMARY) on f.item_number=ci.item_number
set f.item_name=ci.item_name
id select_type table type possible_keys key key_len ref rows Extra
1 UPDATE f ALL 76953 100.00
1 SIMPLE ci ALL 36838 100.00 Using where
Any attempt to run simply runs forever.
What would you recomend?
For your first select query, you may try adding the following index to the cat_items table:
CREATE INDEX idx ON cat_items (item_number, item_name);
This ideally should speed up the left join process, as each record in the current_item_forecast table gets matched to the cat_items table using the item_number as a lookup. Note that we also include the item_name in the index so as to cover the SELECT clause.
(Thank you for supplying the CREATE TABLEs; the root cause in your problem would have been very hard to figure out without them.)
Do not mix CHARACTER SETs, at least not when JOINing by varchars. It leads to a costly table scan.
current_item_forecast:
`Item_Number` varchar(40) NOT NULL ... DEFAULT CHARSET=utf8mb4
cat_items:
`Item_Number` char(32) ... DEFAULT CHARSET=utf8
Be consistent in lengths (40 vs 32).
Since PRIMARY KEY is a unique key, don't redundantly include KEY for the same column:
PRIMARY KEY (`Item_Number`),
KEY `Item` (`Item_Number`)
Use VARCHAR unless the column is truly fixed width.
I have a table with 100k+ rows but my queries are slow (they take about 3 seconds).
I tried making an index like this but this doesn't seem to do anything.
ALTER TABLE pm ADD INDEX (sender,reciever)
This is my query:
SELECT id,message FROM pm WHERE reciever = '28075' OR sender = '28075'
That takes 3 seconds more or less.
Explain of table PM
''
EXPLAIN of the query:
SHOW CREATE TABLE PM:
`CREATE TABLE `pm` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`datetime` int(11) NOT NULL,
`sender` int(11) NOT NULL,
`reciever` int(11) NOT NULL,
`users` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
`readm` int(11) NOT NULL DEFAULT '0',
`forOp` int(11) NOT NULL DEFAULT '0',
`bussy` int(11) NOT NULL DEFAULT '0',
`bericht` longtext CHARACTER SET utf8mb4,
`aantal` int(11) NOT NULL DEFAULT '1',
PRIMARY KEY (`id`),
KEY `users` (`users`(191)),
KEY `sender` (`sender`,`reciever`)
) ENGINE=InnoDB AUTO_INCREMENT=1637118 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci`
The reason the query could not use an index is that it uses OR, and your index can't be used to match the receiver (as a compound index requires that you match the leftmost column before matching the second one)
MySQL 5 added an index_merge which allows using multiple indexes for the same query, so if you have separate indexes on sender and receiver it could pick those.
An alternative would be to rewrite the query to use UNION and again use separate indexes instead of compound one:
SELECT id,message FROM pm WHERE reciever = '28075'
UNION
SELECT id,message FROM pm WHERE sender = '28075'
You can read more at this article
I have a SQL Tabe "STG_S_CUST" which contains a lot of rows (up to 1.5 million) and another table "S_CUST" which contains a lot of rows.
when I'm executing the following Update query, it's very slow, it takes too much time.
UPDATE STG_S_CUST AS STG
INNER JOIN S_CUST AS ST ON STG.SRC_NM=ST.SRC_NM
AND STG.SRC_KEY = ST.SRC_KEY
SET UPDATE_IND = 1,
STG.S_ID = ST.S_ID,
STG.M_ID = ST.M_ID
WHERE STG.PROCESSED_IND = 0
The problem is, that I get a Timeout-Exception unable to execute SQL.
EXPLAIN UPDATE STG_S_CUST AS STG
INNER JOIN S_CUST AS ST ON STG.SRC_NM=ST.SRC_NM
AND STG.SRC_KEY = ST.SRC_KEY
SET UPDATE_IND = 1,
STG.S_ID = ST.S_ID,
STG.M_ID = ST.M_ID
WHERE STG.PROCESSED_IND = 0
Result:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE ST ALL NULL NULL NULL NULL 10479 NULL
1 SIMPLE STG ALL NULL NULL NULL NULL 159334 Using where; Using join buffer (Block Nested Loop)
here's an abbreviated version of the create tables
STG_S_CUST :
CREATE TABLE `STG_S_CUST` (
`STG_ID` int(14) NOT NULL AUTO_INCREMENT,
`STG_DATE` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`SRC_KEY` varchar(100) DEFAULT NULL,
`SRC_NM` varchar(20) DEFAULT NULL,
`M_ID` int(14) DEFAULT NULL,
`S_ID` int(14) DEFAULT NULL,
`PROCESSED_IND` int(1) NOT NULL DEFAULT '0',
`THREAD_ID` int(3) DEFAULT NULL,
`UPDATE_IND` int(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`STG_ID`)
) ENGINE=InnoDB AUTO_INCREMENT=171998 DEFAULT CHARSET=latin1
S_CUST :
CREATE TABLE `S_CUST` (
`S_ID` int(14) NOT NULL AUTO_INCREMENT,
`SRC_KEY` varchar(100) DEFAULT NULL,
`SRC_NM` varchar(20) DEFAULT NULL,
`M_ID` int(14) DEFAULT NULL,
PRIMARY KEY (`S_ID`)
) ENGINE=InnoDB AUTO_INCREMENT=10803 DEFAULT CHARSET=latin1
Does anyone have any ideas why this would be so slow and how to speed it up ?
Could anyone help me here for optimization?
You need some indexing for making the select part faster while doing the join update, start with adding the following indexes
alter table STG_S_CUST add index PROCESSED_IND_idx(PROCESSED_IND);
alter table STG_S_CUST add index SRC_idx(SRC_NM,SRC_KEY);
alter table S_CUST add index SRC_NM_idx(SRC_NM)
Take a backup of the tables first before applying the indexes
I have a table with about 700.000 rows:
CREATE TABLE IF NOT EXISTS `ext_log_entries` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`action` varchar(8) NOT NULL,
`logged_at` datetime NOT NULL,
`object_id` varchar(32) DEFAULT NULL,
`object_class` varchar(255) NOT NULL,
`version` int(11) NOT NULL,
`data` longtext COMMENT '(DC2Type:array)',
`username` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `log_date_lookup_idx` (`logged_at`),
KEY `log_user_lookup_idx` (`username`),
KEY `log_class_lookup_idx` (`object_class`),
KEY `log_version_lookup_idx` (`object_id`,`object_class`,`version`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1219777 ;
I try to run the following query:
SELECT n0_.id AS id0, n0_.action AS action1, n0_.logged_at AS logged_at2, n0_.object_id AS object_id3, n0_.object_class AS object_class4, n0_.version AS version5, n0_.data AS data6, n0_.username AS username7
FROM ext_log_entries n0_
WHERE n0_.object_id =275634
AND n0_.object_class = 'My\\MyBundle\\Entity\\Field'
AND n0_.version <=1
ORDER BY n0_.version ASC
Here is the MySQL plan:
id 1
select_type SIMPLE
table n0_
type ref
possible_keys log_class_lookup_idx,log_version_lookup_idx
key log_class_lookup_idx
key_len 767
ref const
rows 641159
Extra Using where; Using filesort
My query need about 37 seconds to be executed for only 1 row in the result...
I tried to run the same query by deleting my indexes and it goes a little bit faster : about 31 seconds...
I don't understand why my query is taking so much time and why my indexes don't help the performance? Do you know how I can do to have good performance on this query?
Thanks in advance for your help !
EDIT
Here are the cardinalties of the indexes
log_date_lookup_idx BTREE logged_at 1221578 A
log_user_lookup_idx BTREE username 40 A YES
log_class_lookup_idx BTREE object_class 1010 A
log_version_lookup_idx BTREE object_id 1221578 A YES
object_class 1221578 A
version 1221578 A
I found a solution, not THE solution, but at least it works for me.
I think it could help anyway all people who are using gedmo loggable and who are lucky (like me) to have objects with only integers IDs.
I changes my column object_id to integer instead of varchar(255). My query now take 0.008 second ! It works for me because i'm sure i'll always have only integers, for people who have varchar, I'm sorry i tried many things but nothing worked....
CREATE TABLE IF NOT EXISTS `ext_log_entries` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`action` varchar(8) NOT NULL,
`logged_at` datetime NOT NULL,
`object_id` int(11) DEFAULT NULL,
`object_class` varchar(255) NOT NULL,
`version` int(11) NOT NULL,
`data` longtext COMMENT '(DC2Type:array)',
`username` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `log_date_lookup_idx` (`logged_at`),
KEY `log_user_lookup_idx` (`username`),
KEY `log_class_lookup_idx` (`object_class`),
KEY `log_version_lookup_idx` (`object_id`,`object_class`,`version`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1219777 ;
The following query takes 10 seconds to finish when having order by. Without order by it finish in 0.0005 seconds. I am already having an index on field "sku", "vid" AND "timestamp". I have more 200,000 record in this table. Please help, what is wrong with the query when using order by.
SELECT i.pn,i.sku,i.title, fl.f_inserted,fl.f_special, fl.f_notinserted
FROM inventory i
LEFT JOIN inventory_flags fl ON fl.sku = i.sku AND fl.vid = i.vid
WHERE i.qty >=2 ORDER BY i.timestamp LIMIT 0,100;
-- --------------------------------------------------------
--
-- Table structure for table `inventory`
--
CREATE TABLE IF NOT EXISTS `inventory` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`pn` varchar(60) DEFAULT NULL,
`sku` varchar(60) DEFAULT NULL,
`title` varchar(60) DEFAULT NULL,
`qty` int(11) DEFAULT NULL,
`vid` int(11) DEFAULT NULL,
`timestamp` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `vid` (`vid`),
KEY `sku` (`sku`),
KEY `timestamp` (`timestamp`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;
-- --------------------------------------------------------
--
-- Table structure for table `inventory_flags`
--
CREATE TABLE IF NOT EXISTS `inventory_flags` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`f_inserted` tinyint(1) DEFAULT NULL,
`f_notinserted` tinyint(1) DEFAULT NULL,
`f_special` tinyint(1) DEFAULT NULL,
`timestamp` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
`sku` varchar(60) DEFAULT NULL,
`vid` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `vid` (`vid`),
KEY `sku` (`sku`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=1 ;
EXPLANE RESULT:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE fl system vid,sku NULL NULL NULL 0 const row not found
1 SIMPLE i index NULL timestamp 5 NULL 10 Using where
Instead of adding seprate indexes on columns you need to put multicolumn index on tables as you are using more than one columns from same table in joining condition.
after including columns from WHERE clause also include columns used in ORDER BY clause in composite index.
try adding flowing indexes and test them using EXPLAIN:
ALTER TABLE ADD INDEX ix_if inventory_flags(sku, vid);
ALTER TABLE ADD INDEX ix_i inventory(sku, qty, timestamp);
also try to avoid DISTINCT clause in your query, it is equivalent to GROUP BY clause, if you still need it then consider adding covering index.
If sku is unique to each inventory item then define it as UNIQUE - it'll speed things up. (Or the combination of sku and vid - define a composite index in that case.)
Why are you doing SELECT DISTINCT? The vast majority of the time using DISTINCT is a sign that your query or your table structure is wrong.
Since it's DISTINCT, and sku is not UNIQUE it can't use the index on timestamp to speed things up, so it has to sort a table with 200,000 records - it can't even use an index on qty to speed that part up.
PS. Omesh has some good advice as well.
you can use force index(index_key). try it, and you will see in explain query that mysql now will use the key index when 'order by'