SQL statement hanging up in MySQL database - mysql

I am needing some SQL help. I have a SELECT statement that references several tables and is hanging up in the MySQL database. I would like to know if there is a better way to write this statement so that it runs efficiently and does not hang up the DB? Any help/direction would be appreciated. Thanks.
Here is the code:
Select Max(b.BurID) As BurID
From My.AppTable a,
My.AddressTable c,
My.BurTable b
Where a.AppID = c.AppID
And c.AppID = b.AppID
And (a.Forename = 'Bugs'
And a.Surname = 'Bunny'
And a.DOB = '1936-01-16'
And c.PostcodeAnywhereBuildingNumber = '999'
And c.PostcodeAnywherePostcode = 'SK99 9Q9'
And c.isPrimary = 1
And b.ErrorInd <> 1
And DateDiff(CurDate(), a.ApplicationDate) <= 30)
There is NO mysql error in the log. Sorry.

Pro tip: use explicit JOINs rather than a comma-separated list of tables. It's easier to see the logic you're using to JOIN that way. Rewriting your query to do that gives us this.
select Max(b.BurID) As BurID
From My.AppTable AS a
JOIN My.AddressTable AS c ON a.AppID = c.AppID
JOIN My.BurTable AS b ON c.AppID = b.AppID
WHERE (a.Forename = 'Bugs'
And a.Surname = 'Bunny'
And a.DOB = '1936-01-16'
And c.PostcodeAnywhereBuildingNumber = '999'
And c.PostcodeAnywherePostcode = 'SK99 9Q9'
And c.isPrimary = 1
And b.ErrorInd <> 1
And DateDiff(CurDate(), a.ApplicationDate) <= 30)
Next pro tip: Don't use functions (like DateDiff()) in WHERE clauses, because they defeat using indexes to search. That means you should change the last line of your query to
AND a.ApplicationDate >= CurDate() - INTERVAL 30 DAY
This has the same logic as in your query, but it leaves a naked (and therefore index-searchable) column name in the search expression.
Next, we need to look at your columns to see how you are searching, and cook up appropriate indexes.
Let's start with AppTable. You're screening by specific values of Forename, Surname, and DOB. You're screening by a range of ApplicationDate values. Finally you need AppID to manage your join. So, this compound index should help. Its columns are in the correct order to use a range scan to satisfy your query, and contains the needed results.
CREATE INDEX search1 USING BTREE
ON AppTable
(Forename, Surname, DOB, ApplicationDate, AppID)
Next, we can look at your AddressTable. Similar logic applies. You'll enter this table via the JOINed AppID, and then screen by specific values of three columns. So, try this index
CREATE INDEX search2 USING BTREE
ON AddressTable
(AppID, PostcodeAnywherePostcode, PostcodeAnywhereBuildingNumber, isPrimary)
Finally, we're on to your BurTable. Use similar logic as the other two, and try this index.
CREATE INDEX search3 USING BTREE
ON BurTable
(AppID, ErrorInd, BurID)
This kind of index is called a compound covering index, and can vastly speed up the sort of summary query you have asked about.

Related

Improving SQL Query Select Performance

I have a SQL query that pulls data from multiple tables. The only issue I am having is really the query takes to long and I was wondering if there is anyway I can speed it up. I made some minor improvements by using INNER JOIN rather than LEFT JOIN but the query is to slow.
SELECT
clientlist.CRMContactId,
clientlist.ClientAdviser,
COALESCE(NULLIF(clientlist.FirstName, ""), clientlist.CorporateName) AS FirstName,
clientlist.LastName,
clientlist.ServiceStatusName,
FORMAT(t.totalfum, 2) AS "Funds Under Management",
FORMAT(d.totalfci, 2) AS "Total Income",
(SELECT DueDate
FROM tasks
WHERE ClientRef = clientlist.ClientRef
AND `Status` <> "Complete"
ORDER BY DueDate DESC
LIMIT 1) AS NextDate,
(SELECT CompletedDate
FROM tasks
WHERE ClientRef = clientlist.ClientRef
AND `Status` = "Complete"
ORDER BY DueDate DESC
LIMIT 1) AS LastDate
FROM
clientlist
INNER JOIN
(SELECT
plans.ClientId, SUM(plans.CurrentVal) AS totalfum
FROM
plans
GROUP BY
plans.ClientId) t ON clientlist.CRMContactId = t.ClientId
INNER JOIN
(SELECT
adviserfci.ClientId, SUM(adviserfci.Payable) AS totalfci
FROM
adviserfci
WHERE
IncomeType IN ("Renewal Commission", "Ongoing Fee", "Fund Based Commission")
OR (Incometype = "Payaway Received"
AND UnderlyingIncomeType IN ("Renewal", "Ongoing Fee", "Fund Based"))
GROUP BY
adviserfci.ClientId) d ON clientlist.CRMContactId = d.ClientId
WHERE
d.totalfci IS NOT NULL
I've also read somewhere the explain command will help determine issues however I don't understand the response.
Is there any way I can increase the performance of this query?
Fold the test for d.totalfci IS NOT NULL into the subquery that generates it, even if it needs to be in a HAVING clause.
Add some indexes
tasks: INDEX(ClientRef, `Status`, DueDate)
plans: INDEX(ClientId, CurrentVal)
adviserfci: INDEX(ClientId)
First and foremost do you have indexes created for these tables?
Without knowing your data structured and what kind of query load you're going to be putting in them is hard to tell but on first look I'd say these indexes should improve performance, if you don't have them yet:
CRMContactId on table clientlist
ClientId on table plans
ClientId on adviserfci (include IncomeType and Payable)
If you haven't set up table primary keys yet and from the column names they sound like decent candidate keys, so if that works you can kill two birds with one stone.

mysql Query performance is low

I have a query which is running for around 2 hours in last few days. But
before that it took only 2 to 3 minutes of time. i could not able to find
the reason for its sudden slowness. Can any one help me on this?
Please find the below query explain plan[![enter image description here][1]]
[1]...
select
IFNULL(EMAIL,'') as EMAIL,
IFNULL(SITE_CD,'') as SITE_CD,
IFNULL(OPT_TYPE_CD,'') as OPT_TYPE_CD,
IFNULL(OPT_IN_IND,'') as OPT_IN_IND,
IFNULL(EVENT_TSP,'') as EVENT_TSP,
IFNULL(APPLICATION,'') as APPLICATION
from (
SELECT newsletter_entry.email email,
newsletter.site_cd site_cd,
REPLACE (newsletter.TYPE, 'OPTIN_','') opt_type_cd,
CASE
WHEN newsletter_event_temp.post_status = 'SUBSCRIBED' THEN 'Y'
WHEN newsletter_event_temp.post_status = 'UNSUBSCRIBED' THEN
'N'
ELSE ''
END
opt_in_ind,
newsletter_event_temp.event_date event_tsp,
entry_context.application application
FROM amg_toolkit.newsletter_entry,
amg_toolkit.newsletter,
(select NEWSLETTER_EVENT.* from amg_toolkit.NEWSLETTER_EVENT,
amg_toolkit.entry_context where newsletter_event.EVENT_DATE >= '2017-07-11
00:01:23' AND newsletter_event.EVENT_DATE < '2017-07-11 01:01:23' and
newsletter_event.ENTRY_CONTEXT_ID = entry_context.ENTRY_CONTEXT_ID and
entry_context.APPLICATION != 'feedbackloop') newsletter_event_temp,
amg_toolkit.entry_context
WHERE newsletter_entry.newsletter_id = newsletter.newsletter_id
AND newsletter_entry.newsletter_entry_id =
newsletter_event_temp.newsletter_entry_id
AND newsletter.TYPE IN ('OPTIN_PRIM', 'OPTIN_THRD', 'OPTIN_WRLS')
AND newsletter_event_temp.entry_context_id NOT IN
(select d.ENTRY_CONTEXT_ID from amg_toolkit.sweepstake a,
amg_toolkit.sweepstake_entry b, amg_toolkit.user_entry c,
amg_toolkit.entry_context d where a.exclude_data = 'Y' and
a.sweepstake_id=b.sweepstake_id and b.USER_ENTRY_ID=c.USER_ENTRY_ID and
c.ENTRY_CONTEXT_ID = d.ENTRY_CONTEXT_ID)
AND newsletter_event_temp.entry_context_id =
entry_context.entry_context_id
AND newsletter_event_temp.event_date >= '2017-07-11 00:01:23'
AND newsletter_event_temp.event_date < '2017-07-11 01:01:23') a;`
[1]: https://i.stack.imgur.com/cgsS1.png
dont use .*
select only the columns of data you are using in your query.
Avoid nested sub selects if you dont need them.
I don't see a need for them in this query. You query the data 3 times this way instead of just once.
Slowness can be explained by an inefficient query haveing to deal with tables that have a growing number of records.
"Not in" is resource intensive. Can you do that in a better way avoiding "not in" logic?
JOINs are usually faster than subqueries. NOT IN ( SELECT ... ) can usually be turned into LEFT JOIN ... WHERE id IS NULL.
What is the a in a.exclude_data? Looks like a syntax error.
These indexes are likely to help:
newsletter_event: INDEX(ENTRY_CONTEXT_ID, EVENT_DATE) -- in this order
You also need it for newsetter_event_temp, but since that is not possible, something has to give. What version of MySQL are you running? Perhaps you could actually CREATE TEMPORARY TABLE and ADD INDEX.

Which and what type indexes should i create to optimize these mysql querys?

I'm thinking about creating an hashed index in (1) because it uses equalities and bit map on (2) because the state can only be 'accepted' or 'not accepted'. What else can i use? And also my problem is that i can only try b-tree indexes on mysql oracle..
(1)select​ R.user_id from​ rent as R
inner​ ​join​ supervise S on​
R.adress = S.adress
and​ R.space_id = S.space_id
group​ ​by​ R.user_id
having​ ​count​(​distinct​ S.supervisor_id) = 1
(2) select​ ​distinct​ P.adress, P.code from​ space as P where​ (P.adress, P.code) ​not​ ​in ​(
select​ P.adress, P.code ​from​ space as P
natural​ ​join​ rent as R
natural​ ​join​ state as E ​where​ E.state = ‘accepted’)
Since there is no directly limiting criterias in query #1, it will likely be done using a merge join, and no index will improve that.
For query #2, how selective is the criteria E.state = 'accepted'? If very selective (< 5-15% of query result), then index on E.state, indexes for the joins from E to R and from R to P, and index on P.adress, P.code.
Composite index on each table:
INDEX(space_id, adress)
Don't use WHERE(a,b) IN ... -- it performs very poorly.
Don't use IN ( SELECT ... ) -- it often performs poorly.
Instead, use a JOIN.
For state, have
INDEX(state)
(or is it already the PRIMARY KEY?)
If you need more help after all that, provide SHOW CREATE TABLE and EXPLAIN SELECT ....

How to avoid filesort for that mysql query?

I'm using this kind of queries with different parameters :
EXPLAIN SELECT SQL_NO_CACHE `ilan_genel`.`id` , `ilan_genel`.`durum` , `ilan_genel`.`kategori` , `ilan_genel`.`tip` , `ilan_genel`.`ozellik` , `ilan_genel`.`m2` , `ilan_genel`.`fiyat` , `ilan_genel`.`baslik` , `ilan_genel`.`ilce` , `ilan_genel`.`parabirimi` , `ilan_genel`.`tarih` , `kgsim_mahalleler`.`isim` AS mahalle, `kgsim_ilceler`.`isim` AS ilce, (
SELECT `ilanresimler`.`resimlink`
FROM `ilanresimler`
WHERE `ilanresimler`.`ilanid` = `ilan_genel`.`id`
LIMIT 1
) AS resim
FROM (
`ilan_genel`
)
LEFT JOIN `kgsim_ilceler` ON `kgsim_ilceler`.`id` = `ilan_genel`.`ilce`
LEFT JOIN `kgsim_mahalleler` ON `kgsim_mahalleler`.`id` = `ilan_genel`.`mahalle`
WHERE `ilan_genel`.`ilce` = '703'
AND `ilan_genel`.`durum` = '1'
AND `ilan_genel`.`kategori` = '1'
AND `ilan_genel`.`tip` = '9'
ORDER BY `ilan_genel`.`id` DESC
LIMIT 225 , 15
and this is what i get in explain section:
these are the indexes that i already tried to use:
any help will be deeply appreciated what kind of index will be the best option or should i use another table structure ?
You should first simplify your query to understand your problem better. As it appears your problem is constrained to the ilan_gen1 table, the following query would also show you the same symptoms.:
SELECT * from ilan_gene1 WHERE `ilan_genel`.`ilce` = '703'
AND `ilan_genel`.`durum` = '1'
AND `ilan_genel`.`kategori` = '1'
AND `ilan_genel`.`tip` = '9'
So the first thing to do is check that this is the case. If so, the simpler question is simply why does this query require a file sort on 3661 rows. Now the 'hepsi' index sort order is:
ilce->mahelle->durum->kategori->tip->ozelik
I've written it that way to emphasise that it is first sorted on 'ilce', then 'mahelle', then 'durum', etc. Note that your query does not specify the 'mahelle' value. So the best the index can do is lookup on 'ilce'. Now I don't know the heuristics of your data, but the next logical step in debugging this would be:
SELECT * from ilan_gene1 WHERE `ilan_genel`.`ilce` = '703'`
Does this return 3661 rows?
If so, you should be able to see what is happening. The database is using the hepsi index, to the best of it's ability, getting 3661 rows back then sorting those rows in order to eliminate values according to the other criteria (i.e. 'durum', 'kategori', 'tip').
The key point here is that if data is sorted by A, B, C in that order and B is not specified, then the best logical thing that can be done is: first a look up on A then a filter on the remaining values against C. In this case, that filter is performed via a file sort.
Possible solutions
Supply 'mahelle' (B) in your query.
Add a new index on 'ilan_gene1' that doesn't require 'mahelle', i.e. A->C->D...
Another tip
In case I have misdiagnosed your problem (easy to do when I don't have your system to test against), the important thing here is the approach to solving the problem. In particular, how to break a complicated query into a simpler query that produces the same behaviour, until you get to a very simple SELECT statement that demonstrates the problem. At this point, the answer is usually much clearer.

indexes in mysql SELECT AS or using Views

I'm in over my head with a big mysql query (mysql 5.0), and i'm hoping somebody here can help.
Earlier I asked how to get distinct values from a joined query
mysql count only for distinct values in joined query
The response I got worked (using a subquery with join as)
select *
from media m
inner join
( select uid
from users_tbl
limit 0,30) map
on map.uid = m.uid
inner join users_tbl u
on u.uid = m.uid
unfortunately, my query has grown more unruly, and though I have it running, joining into a derived table is taking too long because there is no indexes available to the derived query.
my query now looks like this
SELECT mdate.bid, mdate.fid, mdate.date, mdate.time, mdate.title, mdate.name,
mdate.address, mdate.rank, mdate.city, mdate.state, mdate.lat, mdate.`long`,
ext.link,
ext.source, ext.pre, meta, mdate.img
FROM ext
RIGHT OUTER JOIN (
SELECT media.bid,
media.date, media.time, media.title, users.name, users.img, users.rank, media.address,
media.city, media.state, media.lat, media.`long`,
GROUP_CONCAT(tags.tagname SEPARATOR ' | ') AS meta
FROM media
JOIN users ON media.bid = users.bid
LEFT JOIN tags ON users.bid=tags.bid
WHERE `long` BETWEEN -122.52224684058 AND -121.79760915942
AND lat BETWEEN 37.07500915942 AND 37.79964684058
AND date = '2009-02-23'
GROUP BY media.bid, media.date
ORDER BY media.date, users.rank DESC
LIMIT 0, 30
) mdate ON (mdate.bid = ext.bid AND mdate.date = ext.date)
phew!
SO, as you can see, if I understand my problem correctly, i have two derivative tables without indexes (and i don't deny that I may have screwed up the Join statements somehow, but I kept messing with different types, is this ended up giving me the result I wanted).
What's the best way to create a query similar to this which will allow me to take advantage of the indexes?
Dare I say, I actually have one more table to add into the mix at a later date.
Currently, my query is taking .8 seconds to complete, but I'm sure if I could take advantage of the indexes, this could be significantly faster.
First, check for indices on ext(bid, date), users(bid) and tags(bid), you should really have them.
It seems, though, that it's LONG and LAT that cause you most problems. You should try keeping your LONG and LAT as a (coordinate POINT), create a SPATIAL INDEX on this column and query like that:
WHERE MBRContains(#MySquare, coordinate)
If you can't change your schema for some reason, you can try creating additional indices that include date as a first field:
CREATE INDEX ix_date_long ON media (date, `long`)
CREATE INDEX ix_date_lat ON media (date, lat)
These indices will be more efficient for you query, as you use exact search on date combined with a ranged search on axes.
Starting fresh:
Question - why are you grouping by both media.bid and media.date? Can a bid have records for more than one date?
Here's a simpler version to try:
SELECT
mdate.bid,
mdate.fid,
mdate.date,
mdate.time,
mdate.title,
mdate.name,
mdate.address,
mdate.rank,
mdate.city,
mdate.state,
mdate.lat,
mdate.`long`,
ext.link,
ext.source,
ext.pre,
meta,
mdate.img,
( SELECT GROUP_CONCAT(tags.tagname SEPARATOR ' | ')
FROM tags
WHERE ext.bid = tags.bid
ORDER BY tags.bid GROUP BY tags.bid
) AS meta
FROM
ext
LEFT JOIN
media ON ext.bid = media.bid AND ext.date = media.date
JOIN
users ON ext.bid = users.bid
WHERE
`long` BETWEEN -122.52224684058 AND -121.79760915942
AND lat BETWEEN 37.07500915942 AND 37.79964684058
AND ext.date = '2009-02-23'
AND users.userid IN
(
SELECT userid FROM users ORDER BY rank DESC LIMIT 30
)
ORDER BY
media.date,
users.rank DESC
LIMIT 0, 30
You might want to compare your perforamnces against using a temp table for each selection, and joining those tables together.
create table #whatever
create table #whatever2
insert into #whatever select...
insert into #whatever2 select...
select from #whatever join #whatever 2
....
drop table #whatever
drop table #whatever2
If your system has enough memory to hold full tables this might work out much faster. It depends on how big your database is.