I'm running a statement that's selecting stock market data from three tables. The last part of the statement is running a SELECT max(date) on a table that contains rows of stock data that is dated. I need the last date for a chosen stock from this table (tbl_asxd_extended.date). The problem is the statement just hangs and I can't work out why.
If I separate the statements, up to the final SELECT and run them independently they run fine! They just don't play well together when combined.
I'm not sure how to troubleshoot this one.
SELECT tbl_asxd_extended.close, tbl_asxd_extended.mcapintra, tbl_asxco.industry, tbl_asxco.company, tbl_watchlist.*
FROM tbl_watchlist
INNER JOIN tbl_asxco ON tbl_asxco.symbol = tbl_watchlist.symbol
INNER JOIN tbl_asxd_extended ON tbl_asxd_extended.symbol = tbl_watchlist.symbol
WHERE user_email='testuser#test.com'
AND tbl_asxd_extended.date =
(SELECT max(tbl_asxd_extended.date) FROM tbl_asxd_extended
WHERE tbl_watchlist.symbol = tbl_asxd_extended.symbol)
Here is an 'EXPLAIN' of the statement
id select_type table type possible_keys key key_len ref rows Extra
1 PRIMARY tbl_watchlist ALL NULL NULL NULL NULL 9 Using where
1 PRIMARY tbl_asxco eq_ref symbol_2,symbol symbol_2 32 func 1 Using where
1 PRIMARY tbl_asxd_extended ALL NULL NULL NULL NULL 2195 Using where; Using join buffer
2 DEPENDENT SUBQUERY tbl_asxd_extended ALL NULL NULL NULL NULL 2195 Using where
SELECT tbl_asxd_extended.close, tbl_asxd_extended.mcapintra, tbl_asxco.industry, tbl_asxco.company, tbl_watchlist.*
FROM tbl_watchlist
INNER JOIN tbl_asxco ON tbl_asxco.symbol = tbl_watchlist.symbol
INNER JOIN tbl_asxd_extended ON tbl_asxd_extended.symbol = tbl_watchlist.symbol
WHERE user_email='testuser#test.com'
AND tbl_asxd_extended.date =
(SELECT MAX(tbl_asxd_extended.date) FROM tbl_asxd_extended
WHERE tbl_watchlist.symbol = tbl_asxd_extended.symbol)
this will solve your problem as you're using the max() instead of MAX();
And if possible execute last select query first and store it in any variable say result and just assign result variable to the AND part condition matching
An uncorrelated subquery usually outperforms a correlated one:
SELECT e.close
, e.mcapintra
, s.industry
, s.company
, w.*
FROM tbl_watchlist w
JOIN tbl_asxco a
ON a.symbol = w.symbol
JOIN tbl_asxd_extended e
ON e.symbol = w.symbol
JOIN
( SELECT symbol
, MAX(date) date
FROM tbl_asxd_extended
GROUP
BY symbol
) x
ON x.symbol = e.symbol
AND x.date = e.date
WHERE user_email = 'testuser#test.com'
Further performance improvements may be gained by providing the EXPLAIN for the above together with CREATE TABLE statements for ALL relevant tables.
Related
I need to optimise my code.it works but takes time and sometimes timeouts.
Objective selected columns from table 1 and table 2 must be combined in another table. duplicates are not allowed in the new table. TIA
$modelsc=Customers::find()->select('customer_id')->all();
$modelsp = Product::find()->select('product_no')->all();
foreach($modelsc as $modelc) {
$user = $connection->createCommand(
'SELECT product_no as product_no,:cust_no as fkcustomer_id
FROM product AS p
WHERE NOT EXISTS( SELECT pc.fkproduct_no
FROM
productcustomer AS pc
WHERE
pc.fkproduct_no = p.Product_no AND fkcustomer_id = :cust_no)');
$user->bindValue(':cust_no', $modelc->customer_id);
$modelsx = $user->queryAll();
Yii::$app->db->createCommand()->batchInsert('productcustomer', [ 'fkproduct_no', 'fkcustomer_id'], $modelsx)->execute(); }
looking to your code you could avoid the not exists clause an try using a left join checkn for null on pc.fkproduct_no
SELECT product_no as product_no,
:cust_no as fkcustomer_id
FROM product AS p
LEFT JOIN productcustomer AS pc ON pc.fkproduct_no = p.Product_no
AND fkcustomer_id = :cust_no
WHERE pc.fkproduct_no is null
Anyway be sure you have proper index on the column where of join condition
for table products an index on column Product_no
for table productcustomer a composite index on (fkcustomer_id, fkproduct_no)
Below is MySQl query with which I am able to get disired result
But is there any way i could optimize the query
SELECT users.*,
(SELECT country_name FROM country WHERE country_code = users.country_code)
AS country_name,
(SELECT zone_name FROM timezone WHERE timezone_id = users.timezone_id)
AS zone_name,
(SELECT GROUP_CONCAT(list_name)
FROM list LEFT JOIN user_list ON user_list.list_id = list.list_id
WHERE user_list.user_id = users.user_id AND user_list.status = "active")
AS groups,
(SELECT GROUP_CONCAT(promotion_name)
FROM promotion LEFT JOIN promotion_user ON promotion_user.promotion_id = promotion.promotion_id
WHERE promotion_user.user_id = users.user_id AND promotion_user.status = "active")
AS promotions,
(SELECT GROUP_CONCAT(full_name)
FROM users u LEFT JOIN promotion_user ON promotion_user.promotor_id = u.user_id
WHERE promotion_user.user_id = users.user_id AND promotion_user.status = "active")
AS promotors
FROM users WHERE client_id = '2' AND status != 'deleted'
ORDER BY user_id desc
LIMIT 50 OFFSET 0
The Explain Output is
possible key
id select_type table type _keys key _len ref rows Extra
1 PRIMARY users index NULL PRIMARY 4 NULL 1045612 Using where
6 DEPENDENT SUBQUERY promotion_user ALL NULL NULL NULL NULL 16159 Using where
6 DEPENDENT SUBQUERY u eq_ref PRIMARY PRIMARY 4 [1] 1 NULL
5 DEPENDENT SUBQUERY promotion_user ALL NULL NULL NULL NULL 16895 Using where
5 DEPENDENT SUBQUERY promotion ALL PRIMARY NULL NULL NULL 4 Using where; Using join buffer (Block Nested Loop)
4 DEPENDENT SUBQUERY list ALL PRIMARY NULL NULL NULL 1592 NULL
4 DEPENDENT SUBQUERY user_list ALL NULL NULL NULL NULL 159852 Using where; Using join buffer (Block Nested Loop)
3 DEPENDENT SUBQUERY timezone eq_ref PRIMARY PRIMARY 4 [2] 1 NULL
2 DEPENDENT SUBQUERY country ALL NULL NULL NULL NULL 239 Using where
[1] test.promotion_user.promoter_id
[2] test.promotion_user.promoter_id
I would try using non correlated sub queries. However as you are only bringing back the details for a single user (hence a single row probably) this might not help. Beyond probably eliminating one sub query.
Something like this (untested as no data definitions or data examples)
SELECT `users`.*,
country.country_name,
timezone.zone_name,
sub_groups.groups,
sub_promotors.promotions,
sub_promotors.promotors
FROM `users`
INNER JOIN country
ON country.country_code = users.country_code
INNER JOIN timezone
ON timezone.timezone_id = users.timezone_id
INNER JOIN
(
SELECT promotion_user.user_id, GROUP_CONCAT(full_name) AS promotors, GROUP_CONCAT(promotion_name) AS promotions
FROM users u
LEFT JOIN promotion_user ON promotion_user.promotor_id = u.user_id
WHERE promotion_user.status = "active"
GROUP BY promotion_user.user_id
) AS sub_promotors
ON sub_promotors.user_id = users.user_id
INNER JOIN
(
SELECT user_list.user_id, GROUP_CONCAT(list_name) AS groups
FROM list
LEFT JOIN user_list ON user_list.list_id = list.list_id
WHERE user_list.status = "active"
GROUP BY user_list.user_id
) AS sub_groups
ON sub_groups.user_id = users.user_id
WHERE users.client_id = '2'
AND users.status != 'deleted'
ORDER BY users.user_id
DESC LIMIT 50 OFFSET 0
Correlated sub queries effectively forces MySQL to perform themselves once for each returned row. Changing these to non correlated sub queries which are joined means they can be performed once for all returned rows. Down side is that joining onto a sub query is poorly optimised as far as indexes in MySQL.
You might be able to remove the sub queries if the promotor full name, etc are unique.
I have this query:
SELECT SQL_NO_CACHE
COUNT(*) AS `numrows`
FROM
(`citations`)
LEFT JOIN
`projects` ON `projects`.`project_id` = `citations`.`project_id`
LEFT JOIN
`users` ON `users`.`user_id` = `projects`.`user_id`
WHERE
`users`.`role` = '0'
AND `citations`.`created` BETWEEN 1360213200 AND 1360299599
AND `citations`.`in_card` = '0'
AND `citations`.`citation_id` NOT IN (SELECT
user_stats_citations.citation_id
FROM
user_stats_citations,
user_stats FORCE INDEX (user_stats_type_index)
WHERE
user_stats_citations.user_stat_id = user_stats.id
AND user_stats.type IN (69 , 70, 71, 75, 76));
I have those indexes on user table:
users 0 PRIMARY 1 user_id A 42836 (NULL) (NULL) BTREE
users 1 users_industry_id_index 1 industry_id A 118 (NULL) (NULL) YES BTREE
users 1 users_sponsor_index 1 sponsor A 12 (NULL) (NULL) YES BTREE
This is the output of EXPLAIN EXTENDED
id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY users ALL PRIMARY \N \N \N 42836 100.00 Using where
1 PRIMARY projects ref PRIMARY\,projects_user_id_index projects_user_id_index 4 citelighter.users.user_id 1 100.00 Using where; Using index
1 PRIMARY citations ref citations_project_id_index citations_project_id_index 4 citelighter.projects.project_id 4 100.00 Using index condition; Using where
2 SUBQUERY user_stats range user_stats_type_index user_stats_type_index 2 \N 410768 100.00 Using where; Using index
2 SUBQUERY user_stats_citations ref user_stats_citations_index_user_stat_id\,user_stats_citations_index_citation_id user_stats_citations_index_user_stat_id 8 citelighter.user_stats.id 1 100.00 \N
I tried to add FORCE INDEX on users LEFT JOIN but the index is not used. Can you help me to solve this, because this query is taking like 10 seconds on my local and 1 second on production environment.
The first thing I notice is that this predicate in the where clause: WHERE users.role = '0' turns your LEFT JOINs to INNER JOINs, so you may as well just make them inner joins.
Secondly, MySQL has problems optimising correlated subqueries, and also can perform poorly with derived tables. e.g. In this simple query:
SELECT *
FROM (SELECT * FROM T) T
JOIN (SELECT * FROM T) T2 ON T.ID = T2.ID;
Even though ID is the primary key on T, the primary key is not used for the join as it can't be cascaded out of the derived table. Similarly sometimes when you write:
SELECT *
FROM T
WHERE Afield NOT IN (SELECT Afield FROM T WHERE AnotherField = 1);
MySQL does not necessarily materialise the subquery and use this, it will often rewrite the query as:
SELECT *
FROM T
WHERE NOT EXISTS (SELECT 1
FROM T T2
WHERE T.Afield = T2.Afield
AND T2.AnotherField = 1);
And the subquery is executed for each row in the outer query, so if you have a large number of rows in the outer query executing the subquery for every row becomes very costly. The solution is to avoid subqueries as far as possible. In your case you can rewrite your query as:
SELECT SQL_NO_CACHE
COUNT(*) AS `numrows`
FROM `citations`
INNER JOIN `projects`
ON `projects`.`project_id` = `citations`.`project_id`
INNER JOIN `users`
ON `users`.`user_id` = `projects`.`user_id`
LEFT JOIN (user_stats_citations
INNER JOIN user_stats
ON user_stats_citations.user_stat_id = user_stats.id
AND user_stats.type IN (69 , 70, 71, 75, 76))
ON user_stats_citations.citation_id = `citations`.`citation_id`
WHERE `users`.`role` = '0'
AND `citations`.`created` BETWEEN 1360213200 AND 1360299599
AND `citations`.`in_card` = '0'
AND user_stats_citations.citation_id IS NULL;
With no subqueries there is no derived tables, or row by row execution of subqueries. This should improve execution time.
What does this give you?
SELECT COUNT(*) numrows
FROM citations c
JOIN projects p
ON p.project_id = c.project_id
JOIN users u
ON u.user_id = p.user_id
LEFT
JOIN
( SELECT uc.citation_id
FROM user_stats_citations uc
JOIN user_stats us
ON uc.user_stat_id = us.id
AND us.type IN (69,70,71,75,76)
) x
ON x.citation_id = c.citation_id
WHERE u.role = 0
AND c.created BETWEEN 1360213200 AND 1360299599
AND c.in_card = 0
AND x.citation_id IS NULL
I have a short access/mySQL question. I have a mapping table on the format below.
ID Category_A Category_B Category_C Team
1 a b T1
2 a d T2
I have a second table which also includes Category_A, Category_B, and Category_C. I would like to join the Team value to the my second table based on the mappingtable. My problem is that when there is a blank (e.g. ID=2, Category_B) the mapping should assign the T2 to any row that contains Category_A=a and Category_C=d regardless of the value in Category_B.
Can this type of mapping be done?
Grateful for your help!
In MS Access, I think you would need something on the lines of:
SELECT t.ID, m.Team
FROM Team t
INNER JOIN Mapping m
ON (m.Category_C = t.Category_C)
AND (m.Category_B = t.Category_B)
AND (m.Category_A = t.Category_A)
WHERE m.Category_C Is Not Null
AND m.Category_B Is Not Null
AND m.Category_A Is Not Null
UNION ALL
SELECT t.ID, m.Team
FROM Team t
INNER JOIN Mapping m
ON (m.Category_B = t.Category_B)
AND (m.Category_A = t.Category_A)
WHERE m.Category_C Is Null
AND m.Category_B Is Not Null
AND m.Category_A Is Not Null
UNION ALL
SELECT t.ID, m.Team
FROM Team t
INNER JOIN Mapping m
ON (m.Category_C = t.Category_C)
AND (m.Category_A = t.Category_A)
WHERE m.Category_C Is Not Null
AND m.Category_B Is Null
AND m.Category_A Is Not Null
UNION ALL
SELECT t.ID, m.Team
FROM Team t
INNER JOIN Mapping m
ON (m.Category_C = t.Category_C)
AND (m.Category_B = t.Category_B)
WHERE m.Category_C Is Not Null
AND m.Category_B Is Not Null
AND m.Category_A Is Null
Right now, I have
SELECT gp_id FROM gp.keywords
WHERE keyword_id = 15
AND (SELECT practice_link FROM gp.practices
WHERE practice_link IS NOT NULL
AND id = gp_id)
This does not provide a syntax error, however for values where it should return row(s), it just returns 0 rows.
What I'm trying to do is get the gp_id from gp.keywords where the the keywords table keyword_id column is a specific value and the practice_link is the practices table corresponds to the gp_id that I have, which is stored in the id column of that table.
I'm not even sure that is valid SQL, so I'm surprised it is working at all:
SELECT gp_id
FROM gp.keywords
WHERE keyword_id = 15
AND (SELECT practice_link FROM gp.practices WHERE practice_link IS NOT NULL AND id = gp_id)
How about this instead:
SELECT kw.gp_id, p.practice_link
FROM gp.keywords AS kw
INNER JOIN gp.practices AS p
ON p.id = kw.gp_id
WHERE kw.keyword_id = 15
I would steer clear of implicit joins as in the other examples. It only leads to tears later.
select k.gp_id
from gp.keywords as k,
gp.practices as p
where
keyword_id=15
and practice_link is not null
and p.id=k.gp_id
SELECT k.gp_id
FROM gp.keywords k, gp.practices p
WHERE
p.id = k.gp_id.AND
k.keyword_id = 15 AND
p.practice_link is not null
SELECT g.gp_id, p.practice_link FROM gp.keywords g, gp.practices p
WHERE
g.keyword_id = 15 AND p.practice_link IS NOT NULL AND p.id = g.gp_id