I have a SQL Tabe "STG_S_CUST" which contains a lot of rows (up to 1.5 million) and another table "S_CUST" which contains a lot of rows.
when I'm executing the following Update query, it's very slow, it takes too much time.
UPDATE STG_S_CUST AS STG
INNER JOIN S_CUST AS ST ON STG.SRC_NM=ST.SRC_NM
AND STG.SRC_KEY = ST.SRC_KEY
SET UPDATE_IND = 1,
STG.S_ID = ST.S_ID,
STG.M_ID = ST.M_ID
WHERE STG.PROCESSED_IND = 0
The problem is, that I get a Timeout-Exception unable to execute SQL.
EXPLAIN UPDATE STG_S_CUST AS STG
INNER JOIN S_CUST AS ST ON STG.SRC_NM=ST.SRC_NM
AND STG.SRC_KEY = ST.SRC_KEY
SET UPDATE_IND = 1,
STG.S_ID = ST.S_ID,
STG.M_ID = ST.M_ID
WHERE STG.PROCESSED_IND = 0
Result:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE ST ALL NULL NULL NULL NULL 10479 NULL
1 SIMPLE STG ALL NULL NULL NULL NULL 159334 Using where; Using join buffer (Block Nested Loop)
here's an abbreviated version of the create tables
STG_S_CUST :
CREATE TABLE `STG_S_CUST` (
`STG_ID` int(14) NOT NULL AUTO_INCREMENT,
`STG_DATE` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`SRC_KEY` varchar(100) DEFAULT NULL,
`SRC_NM` varchar(20) DEFAULT NULL,
`M_ID` int(14) DEFAULT NULL,
`S_ID` int(14) DEFAULT NULL,
`PROCESSED_IND` int(1) NOT NULL DEFAULT '0',
`THREAD_ID` int(3) DEFAULT NULL,
`UPDATE_IND` int(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`STG_ID`)
) ENGINE=InnoDB AUTO_INCREMENT=171998 DEFAULT CHARSET=latin1
S_CUST :
CREATE TABLE `S_CUST` (
`S_ID` int(14) NOT NULL AUTO_INCREMENT,
`SRC_KEY` varchar(100) DEFAULT NULL,
`SRC_NM` varchar(20) DEFAULT NULL,
`M_ID` int(14) DEFAULT NULL,
PRIMARY KEY (`S_ID`)
) ENGINE=InnoDB AUTO_INCREMENT=10803 DEFAULT CHARSET=latin1
Does anyone have any ideas why this would be so slow and how to speed it up ?
Could anyone help me here for optimization?
You need some indexing for making the select part faster while doing the join update, start with adding the following indexes
alter table STG_S_CUST add index PROCESSED_IND_idx(PROCESSED_IND);
alter table STG_S_CUST add index SRC_idx(SRC_NM,SRC_KEY);
alter table S_CUST add index SRC_NM_idx(SRC_NM)
Take a backup of the tables first before applying the indexes
Related
i have a three almost identical queries executed one by one in my app. Two of queries are executing fairly fast but one is much slower. Here are queries:
update `supremeshop_product_attributes` set `id_step` = 899 where `id_step` = 1 and `id_product` = 32641
540ms
update `supremeshop_product_attributes` set `id_step` = 1 where `id_step` = 0 and `id_product` = 32641
1.71s
update `supremeshop_product_attributes` set `id_step` = 0 where `id_step` = 899 and `id_product` = 32641
9.75ms
Create table query
CREATE TABLE `supremeshop_product_attributes` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`id_product` int(11) NOT NULL,
`id_attribute` int(11) NOT NULL,
`id_attribute_group` int(11) NOT NULL,
`id_step` int(11) NOT NULL,
`id_alias` int(9) NOT NULL,
`price_retail` int(11) NOT NULL DEFAULT '0',
`behaviour` int(1) NOT NULL DEFAULT '0',
`active` int(1) NOT NULL DEFAULT '1',
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `supremeshop_product_attributes_id_product_index` (`id_product`),
KEY `supremeshop_product_attributes_id_attribute_index` (`id_attribute`),
KEY `supremeshop_product_attributes_id_attribute_group_index` (`id_attribute_group`),
KEY `supremeshop_product_attributes_id_step_index` (`id_step`)
) ENGINE=InnoDB AUTO_INCREMENT=3012991 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
Facts:
table has 1500000 rows
each query updates around 650 rows
each searched/updated field has index (id(primary), id_product, id_step, id_alias)
only second query takes much longer time to execute (everytime - even executed as single query not one by one)
each variable used in query is integer
if i execute queries direcly in phpymysql -> sql - i get the same execution time (like in my laravel app so query is somehow slow)
Question is why? And is there a good explanation/ fix for that problem.
For any help many thanks!
Mark.
I have read several other posts and pages and i can't figure a way to speed up and use an index instead of a full table scan
Query in question
select f.*,ci.item_name
from forecastpro.current_item_forecast f
left join cat_items as ci on f.item_number=ci.item_number
Tables Structures
CREATE TABLE `current_item_forecast` (
`Axe3` varchar(128) DEFAULT NULL,
`Axe5` varchar(128) DEFAULT NULL,
`Axe7` varchar(128) DEFAULT NULL,
`Axe11` varchar(128) DEFAULT NULL,
`Item_Number` varchar(40) NOT NULL,
`Item_Name` varchar(128) DEFAULT NULL,
`fc_year` int(4) NOT NULL,
`fc_period` int(2) NOT NULL,
`Qty_Forecast` double NOT NULL DEFAULT '0',
`USD_Forecast` double NOT NULL DEFAULT '0',
`Stock_Start` float NOT NULL DEFAULT '0',
`Stock_Transit` float NOT NULL DEFAULT '0',
PRIMARY KEY (`Item_Number`,`fc_year` DESC,`fc_period`),
KEY `Item` (`Item_Number`),
KEY `Axe3` (`Axe3`),
KEY `Axe5` (`Axe5`),
KEY `Axe7` (`Axe7`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
CREATE TABLE `cat_items` (
`Item_Number` char(32) NOT NULL,
`Item_Name` char(128) DEFAULT NULL,
`Axe3` char(10) DEFAULT NULL,
`Item_Group` varchar(32) DEFAULT NULL,
`Axe5` char(10) DEFAULT NULL,
`Axe7` char(10) DEFAULT NULL,
`Axe11` char(10) DEFAULT NULL,
`Date_Created` datetime DEFAULT NULL,
PRIMARY KEY (`Item_Number`),
KEY `Item` (`Item_Number`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
Explain
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE f ALL 121882 100.00
1 SIMPLE ci ALL 36838 100.00 Using where; Using join buffer (Block Nested Loop)
Other Attempts
**This two produce the same result:**
update forecastpro.current_item_forecast f
left join cupfsa.cat_items ci on f.item_number=ci.item_number
set f.item_name=ci.item_name
update forecastpro.current_item_forecast f
left join cupfsa.cat_items ci FORCE KEY FOR JOIN (PRIMARY) on f.item_number=ci.item_number
set f.item_name=ci.item_name
id select_type table type possible_keys key key_len ref rows Extra
1 UPDATE f ALL 76953 100.00
1 SIMPLE ci ALL 36838 100.00 Using where
Any attempt to run simply runs forever.
What would you recomend?
For your first select query, you may try adding the following index to the cat_items table:
CREATE INDEX idx ON cat_items (item_number, item_name);
This ideally should speed up the left join process, as each record in the current_item_forecast table gets matched to the cat_items table using the item_number as a lookup. Note that we also include the item_name in the index so as to cover the SELECT clause.
(Thank you for supplying the CREATE TABLEs; the root cause in your problem would have been very hard to figure out without them.)
Do not mix CHARACTER SETs, at least not when JOINing by varchars. It leads to a costly table scan.
current_item_forecast:
`Item_Number` varchar(40) NOT NULL ... DEFAULT CHARSET=utf8mb4
cat_items:
`Item_Number` char(32) ... DEFAULT CHARSET=utf8
Be consistent in lengths (40 vs 32).
Since PRIMARY KEY is a unique key, don't redundantly include KEY for the same column:
PRIMARY KEY (`Item_Number`),
KEY `Item` (`Item_Number`)
Use VARCHAR unless the column is truly fixed width.
I'm a web developer and I'm posting for the first time on SO.
Today I'm asking for your help because I already tried all the possibilities with no luck.
I created a SAAS web application that is used by salesman on the ground, it includes an offline version where users don't need to be connected to use it.
As the database is getting bigger, the queries are taking more and more time to be executed.
Today I'm facing a big issue where the query is leading to a timeout when the user tries to display it's result.
So, here's the dump :
SET SQL_MODE = "NO_AUTO_VALUE_ON_ZERO";
SET AUTOCOMMIT = 0;
START TRANSACTION;
SET time_zone = "+00:00";
-- 86 rows
CREATE TABLE t2 (
id_t2 int(11) NOT NULL,
quantite_t2 int(11) NOT NULL,
ca_t2 decimal(10,2) NOT NULL,
date_t2 date NOT NULL,
import_t2 datetime NOT NULL,
id_enseigne int(11) NOT NULL,
id_t3 int(11) NOT NULL,
annee_t2 int(11) NOT NULL,
mois_t2 int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
-- 2012065 rows
CREATE TABLE t1 (
id_t2 int(11) NOT NULL,
id_t0 bigint(20) NOT NULL,
id_t4 bigint(20) NOT NULL,
quantite_t1 int(11) NOT NULL,
ca_t1 decimal(10,2) NOT NULL,
pvc_moyen_t1 float NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
-- 388 rows
CREATE TABLE t4 (
id_t4 int(11) NOT NULL,
lib_t4 text NOT NULL,
libcourt_t4 varchar(255) NOT NULL,
ean_t4 varchar(255) NOT NULL,
pcb_t4 int(11) NOT NULL,
pcb2_t4 int(11) NOT NULL,
fam_t4 varchar(255) NOT NULL,
gam_t4 varchar(255) NOT NULL DEFAULT '0',
stat_t4 int(11) NOT NULL DEFAULT 1,
vmh_t4 decimal(10,2) NOT NULL,
detail_t4 text NOT NULL,
ingr_t4 text NOT NULL,
weight_t4 float NOT NULL,
lifetime_t4 varchar(255) NOT NULL,
pmc1_t4 float NOT NULL,
pmc2_t4 float NOT NULL,
dim_t4 decimal(10,2) NOT NULL,
ordre_t4 int(11) NOT NULL,
created_t4 datetime NOT NULL,
updated_t4 datetime NOT NULL,
updated_img_t4 datetime NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
-- 1 row
CREATE TABLE t3 (
id_t3 int(11) NOT NULL,
nom_t3 text NOT NULL,
stat_t3 int(11) NOT NULL,
created_t3 datetime NOT NULL,
deleted_t3 datetime NOT NULL,
updated_t3 datetime NOT NULL,
ip_create_t3 text NOT NULL,
ip_delete_t3 text NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
ALTER TABLE t2
ADD PRIMARY KEY (id_t2),
ADD KEY annee_t2 (annee_t2,mois_t2,date_t2);
ALTER TABLE t1
ADD PRIMARY KEY (id_t2,id_t0,id_t4);
ALTER TABLE t4
ADD PRIMARY KEY (id_t4,ean_t4),
ADD KEY ean_t4 (ean_t4),
ADD KEY id_t4 (id_t4);
ALTER TABLE t3
ADD PRIMARY KEY (id_t3);
ALTER TABLE t2
MODIFY id_t2 int(11) NOT NULL AUTO_INCREMENT;
ALTER TABLE t4
MODIFY id_t4 int(11) NOT NULL AUTO_INCREMENT;
ALTER TABLE t3
MODIFY id_t3 int(11) NOT NULL AUTO_INCREMENT;
COMMIT;
The executed query below takes like 4 min to execute :
-- execution time 228 seconds
SELECT SUM(t1.ca_t1) AS ca_t4, SUM(t1.quantite_t1) AS qte_t4,
t4.fam_t4, t4.gam_t4, t4.lib_t4, t4.ean_t4, t4.id_t4,
t2.annee_t2, t2.mois_t2, COUNT(t1.id_t0) AS count_mag,
t3.id_t3, t3.nom_t3
FROM t1 t1
INNER JOIN t2 t2 ON t2.id_t2 = t1.id_t2
LEFT JOIN t3 t3 ON t2.id_t3 = t3.id_t3
INNER JOIN t4 t4 ON t1.id_t4 = t4.ean_t4
WHERE t2.date_t2 BETWEEN "2017-05-01" AND "2019-05-01"
GROUP BY t2.annee_t2, t2.mois_t2, t4.id_t4
ORDER BY ca_t4 DESC;
I tried all optimization I know to help me reduce execution time but no success...
The EXPLAIN shows this :
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE t2 ALL PRIMARY NULL NULL NULL 86 Using where; Using temporary; Using filesort
1 SIMPLE t3 eq_ref PRIMARY PRIMARY 4 db.t2.id_t3 1
1 SIMPLE t1 ref PRIMARY,id_t2 PRIMARY 4 db.t2.id_t2 11266
1 SIMPLE t4 ALL ean_t4 NULL NULL NULL 388 Using where; Using join buffer (flat, BNL join)
Thank you for your help guys.
This looks like a fairly complex query, with multiple places that will slow it down. However, the first thing I notice is that a double lookup would have to occur in order to use the annee_2 index, which is probably why it is not using it.
Try adding id_t3 to the end of that index on table t2:
(annee_t2,mois_t2,date_t2,id_t3)
This should permit the optimize to use that index.
Run the query again (twice, to populate the buffer cache, report only the 2nd result) and if it doesn't improve sufficiently, post the new EXPLAIN plan.
The GROUP BY is probably improper since it does not include the t3 columns that are not aggergated.
Do you really want 2 years plus 1 day? Perhaps use this:
t2.date_t2 >= "2017-05-01"
AND t2.date_t2 < "2017-05-01" + INTERVAL 2 YEAR
Do not mix datatypes when JOINing -- ON t1.id_t4 = t4.ean_t4:
ean_t4 varchar(255) NOT NULL,
id_t4 bigint(20) NOT NULL,
(There may be other issues, but these should help.)
I am trying to optimize the below query it yields me the following result
EXPLAIN UPDATE table1 a
JOIN table2 b
ON a.id=b.ref_id
SET a.value=b.value;
When i try to explain the query, I get the following result.
ID select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE b ALL idx_ref_id 10785
1 SIMPLE a ref idx_id idx_id 4 b.ref_id 24
As far as i understand query is using only one index i.e idx_id and not idx_ref_id.
Edit:
Here is the table structure.
SHOW CREATE TABLE table1;
CREATE TABLE table_1 (
`id` int(11) NOT NULL DEFAULT '0',
`companyname` varchar(100) NOT NULL DEFAULT '',
`value` varchar(100) NOT NULL DEFAULT '',
`sve_value` varchar(100) NOT NULL DEFAULT '',
KEY `idx_id` (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
SHOW CREATE TABLE table2;
CREATE TABLE `table2` (
`value` varchar(255) DEFAULT NULL,
`suggested_value_enrichement` varchar(255) DEFAULT NULL,
`tags` varchar(255) DEFAULT NULL,
`ref_id` int(11) NOT NULL DEFAULT '0',
KEY `idx_ref_id` (`ref_id`),
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Please help me, Also please correct me if am wrong.
Thanks.
Issue with query was there were many process running on the server which choked the resources. Also there were many ideal connection to the database to precise around 150.
So by killing the process and terminating the ideal connection solved the issue.
Query execution finished within a minute.
The following query, regardless of environment, takes more than 30 seconds to compute.
SELECT COUNT( r.response_answer )
FROM response r
INNER JOIN (
SELECT G.question_id
FROM question G
INNER JOIN answer_group AG ON G.answer_group_id = AG.answer_group_id
WHERE AG.answer_group_stat = 'statistic'
) AS q ON r.question_id = q.question_id
INNER JOIN org_survey os ON os.org_survey_code = r.org_survey_code
WHERE os.survey_id =42
AND r.response_answer = 5
AND DATEDIFF( NOW( ) , r.added_dt ) <1000000
AND r.uuid IS NOT NULL
When I explain the query,
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 1087
1 PRIMARY r ref question_id,org_survey_code,code_question,uuid,uor question_id 4 q.question_id 1545 Using where
1 PRIMARY os eq_ref org_survey_code,survey_id,org_survey_code_2 org_survey_code 12 survey_2.r.org_survey_code 1 Using where
2 DERIVED G ALL agid NULL NULL NULL 1680
2 DERIVED AG eq_ref PRIMARY PRIMARY 1 survey_2.G.answer_group_id 1 Using where
I have a very basic knowledge of indexing, but I have tried nearly every combination I can think of and cannot seem to improve the speed of this query. The responses table is right around 2 million rows, question is about 1500 rows, answer_group is about 50, and org_survey is about 8,000.
Here is the basic structure for each:
CREATE TABLE `response` (
`response_id` int(10) unsigned NOT NULL auto_increment,
`response_answer` text NOT NULL,
`question_id` int(10) unsigned NOT NULL default '0',
`org_survey_code` varchar(7) NOT NULL,
`uuid` varchar(40) default NULL,
`added_dt` datetime default NULL,
PRIMARY KEY (`response_id`),
KEY `question_id` (`question_id`),
KEY `org_survey_code` (`org_survey_code`),
KEY `code_question` (`org_survey_code`,`question_id`),
KEY `IDX_ADDED_DT` (`added_dt`),
KEY `uuid` (`uuid`),
KEY `response_answer` (`response_answer`(1)),
KEY `response_question` (`response_answer`(1),`question_id`),
) ENGINE=MyISAM AUTO_INCREMENT=2298109 DEFAULT CHARSET=latin1
CREATE TABLE `question` (
`question_id` int(10) unsigned NOT NULL auto_increment,
`question_text` varchar(250) NOT NULL default '',
`question_group` varchar(250) default NULL,
`question_position` tinyint(3) unsigned NOT NULL default '0',
`survey_id` tinyint(3) unsigned NOT NULL default '0',
`answer_group_id` mediumint(8) unsigned NOT NULL default '0',
`seq_id` int(11) NOT NULL default '0',
PRIMARY KEY (`question_id`),
KEY `question_group` (`question_group`(10)),
KEY `survey_id` (`survey_id`),
KEY `agid` (`answer_group_id`)
) ENGINE=MyISAM AUTO_INCREMENT=1860 DEFAULT CHARSET=latin1
CREATE TABLE `org_survey` (
`org_survey_id` int(11) NOT NULL auto_increment,
`org_survey_code` varchar(10) NOT NULL default '',
`org_id` int(11) NOT NULL default '0',
`org_manager_id` int(11) NOT NULL default '0',
`org_url_id` int(11) default '0',
`division_id` int(11) default '0',
`sector_id` int(11) default NULL,
`survey_id` int(11) NOT NULL default '0',
`process_batch` tinyint(4) default '0',
`added_dt` datetime default NULL,
PRIMARY KEY (`org_survey_id`),
UNIQUE KEY `org_survey_code` (`org_survey_code`),
KEY `org_id` (`org_id`),
KEY `survey_id` (`survey_id`),
KEY `org_survey_code_2` (`org_survey_code`,`total_taken`),
KEY `org_manager_id` (`org_manager_id`),
KEY `sector_id` (`sector_id`)
) ENGINE=MyISAM AUTO_INCREMENT=9268 DEFAULT CHARSET=latin1
CREATE TABLE `answer_group` (
`answer_group_id` tinyint(3) unsigned NOT NULL auto_increment,
`answer_group_name` varchar(50) NOT NULL default '',
`answer_group_type` varchar(20) NOT NULL default '',
`answer_group_stat` varchar(20) NOT NULL default 'demographic',
PRIMARY KEY (`answer_group_id`)
) ENGINE=MyISAM AUTO_INCREMENT=53 DEFAULT CHARSET=latin1
I know there are small things I can probably do to improve the efficiency of the database, such as reducing the size of integers where it's unnecessary. However, those are fairly trivial considering the ridiculous time it takes just to produce a result here. How can I properly index these tables, based on what explain has shown me? It seems that I have tried a large variety of combinations to no avail. Also, is there anything else that anyone can see that will optimize the table and reduce the query? I need it to be computed in less than a second. Thanks in advance!
1.If you want the index of r.added_dt to be used, instead of:
DATEDIFF(NOW(), r.added_dt) < 1000000
use:
CURDATE() - INTERVAL 1000000 DAY < r.added_dt
Anyway, the above condition is checking if added_at is a million days old or not. Do you really store so old dates? If not, you can simply remove this condition.
If you want this condition, an index on added_at would help a lot. Your query as it is now, checks all rows for this condition, calling the DATEDIFF() function as many times as the rows of the response table.
2.Since r.response_answer cannot be NULL, instead of:
SELECT COUNT( r.response_answer )
use:
SELECT COUNT( * )
COUNT(*) is faster than COUNT(field).
3.Two of the three fields that you use for joining tables have different datatypes:
ON question . answer_group_id
= answer_group . answer_group_id
CREATE TABLE question (
...
answer_group_id mediumint(8) ..., <--- mediumint
CREATE TABLE answer_group (
answer_group_id` tinyint(3) ..., <--- tinyint
-------------------------------
ON org_survey . org_survey_code
= response . org_survey_code
CREATE TABLE response (
...
org_survey_code varchar(7) NOT NULL, <--- 7
CREATE TABLE org_survey (
...
org_survey_code varchar(10) NOT NULL default '', <--- 10
Datatype mediumint is not the same as tinyint and the same goes for varchar(7) and varchar(10). When they are used for join, MySQL has to lose time doing conversion from one type to another. Convert one of them so they have identical datatypes. This is not the main issue of the query but this change will also help all other queries that use these joins.
And after making this change do a 'Analyze Table ' for the table. It will help mysql making better execution plans.
You have a response_answer = 5 condition, where response_answer is text. It's not an error, but it's better to use response_answer = '5' (the conversion of 5 to '5' will be done by MySQL anyway, if you don't do that).
Real issue is that you don't have a compound index on the 3 fields that are used in the WHERE conditions. Try adding this one:
ALTER TABLE response
ADD INDEX ind_u1_ra1_aa
(uuid(1), response_answer(1), added_at) ;
(this may take a while as your table is not small)
Can you try the following query? I've removed the sub-query from your original one. This may let the optimiser produce a better execution plan.
SELECT COUNT(r.response_answer)
FROM response r
INNER JOIN question q ON r.question_id = q.question_id
INNER JOIN answer_group ag ON q.answer_group_id = ag.answer_group_id
INNER JOIN org_survey os ON os.org_survey_code = r.org_survey_code
WHERE
ag.answer_group_stat = 'statistic'
AND os.survey_id = 42
AND r.response_answer = 5
AND DATEDIFF(NOW(), r.added_dt) < 1000000
AND r.uuid IS NOT NULL