I have a view : vcompanyendofday
The following query executes in just 0.7 secs
Select * from vcompanyendofday
But a simple where condition to this query takes around 200.0 secs
select * from vcompanyendofday where companyid <= 51;
This is the view definition:
CREATE VIEW `vcompanyendofday` AS
select `c`.`companyid` AS `companyid`,
`c`.`scripcode` AS `scripcode`,
`e`.`eoddate` AS `eoddate`,
`e`.`prevclose` AS `prevclose`,
`e`.`delqty` AS `delqty`
from (
`company` `c`
left join
`endofday` `e`
on ((`c`.`companyid` = `e`.`companyid`)))
where (`e`.`eoddate` =
(
select max(`e2`.`eoddate`) AS `max(eoddate)`
from `endofday` `e2`
where (`e2`.`companyid` = `c`.`companyid`)
)
);
Seems you don't have an index on endofday.companyid
When you add the condition, company becomes leading in the join, and kills all performance.
Create an index on endofday.companyid:
CREATE INDEX ix_endofday_companyid ON endofday(companyid)
By the way, if you want all companies to be returned, you need to put the subquery into the ON clause of the OUTER JOIN, or your missing endofday's will be filtered out:
CREATE VIEW `vcompanyendofday` AS
select `c`.`companyid` AS `companyid`,
`c`.`scripcode` AS `scripcode`,
`e`.`eoddate` AS `eoddate`,
`e`.`prevclose` AS `prevclose`,
`e`.`delqty` AS `delqty`
from (
`company` `c`
left join
`endofday` `e`
on `c`.`companyid` = `e`.`companyid`
AND `e`.`eoddate` =
(
select max(`e2`.`eoddate`) AS `max(eoddate)`
from `endofday` `e2`
where (`e2`.`companyid` = `c`.`companyid`)
)
Have you tried the select used to create the view by itself with the WHERE clause to see what happens?
If the problem happens with that, run EXPLAIN on that query to see what's happening.
At a guess, there's no index on companyid in one of the tables, most likely endofday.
Related
I have a query that selects ~8000 rows. When I execute the query it takes 0.1 sec.
When I copy the query into a view and execute the view it takes about 2 seconds. In the first row of explain it selects ~570K rows, i dont know why.
I dont understand the first Row and why it shows up only in the view explain
1 PRIMARY ALL NULL NULL NULL NULL
This is the query (yes i know im not a mysql pro and the query is not that efficent, but it works ans 0.1 sek would be ok for me. Does anyone know why it is so slow in a view?
MariaDB 10.5.9
select
`xxxxxxx`.`auftraege`.`Zustandigkeit` AS `Zustandigkeit`,
`xxxxxxx`.`auftraege`.`cms` AS `cms`,
`xxxxxxx`.`auftraege`.`auftrag_id` AS `auftrag_id`,
`xxxxxxx`.`angebot`.`angebot_id` AS `angebot_id`,
`xxxxxxx`.`kunden`.`kunde_id` AS `kid`,
`xxxxxxx`.`angebot`.`kunde_id` AS `kunde_id`,
`xxxxxxx`.`kunden`.`firma` AS `firma`,
`xxxxxxx`.`auftraege`.`gekuendigt` AS `gekuendigt`,
`xxxxxxx`.`kunden`.`ansprechpartnerVorname` AS `ansprechpartnerVorname`,
`xxxxxxx`.`kunden`.`ansprechpartner` AS `ansprechpartner`,
`xxxxxxx`.`auftraege`.`ampstatus` AS `ampstatus`,
`xxxxxxx`.`auftraege`.`autoMahnungen` AS `autoMahnungen`,
`xxxxxxx`.`kunden`.`mail` AS `mail`,
`xxxxxxx`.`kunden`.`ansprechpartnerAnrede` AS `ansprechpartnerAnrede`,
case
`xxxxxxx`.`kunden`.`ansprechpartnerAnrede`
when
'm'
then
concat('Herr ', ifnull(`xxxxxxx`.`kunden`.`ansprechpartnerVorname`, ''), ifnull(`xxxxxxx`.`kunden`.`ansprechpartner`, ''))
else
concat('Frau ', ifnull(`xxxxxxx`.`kunden`.`ansprechpartnerVorname`, ''), ifnull(`xxxxxxx`.`kunden`.`ansprechpartner`, ''))
end
AS `ansprechpartnerfullName`, `xxxxxxx`.`kunden`.`website` AS `website`, `xxxxxxx`.`personal`.`name_betrieb` AS `name_betrieb`, `xxxxxxx`.`kunden`.`prioritaet` AS `prioritaet`, `xxxxxxx`.`auftraege`.`infoemail` AS `infoemail`, `xxxxxxx`.`auftraege`.`keywords` AS `keywords`, `xxxxxxx`.`auftraege`.`ftp_h` AS `ftp_h`, `xxxxxxx`.`auftraege`.`ftp_u` AS `ftp_u`, `xxxxxxx`.`auftraege`.`ftp_pw` AS `ftp_pw`, `xxxxxxx`.`auftraege`.`lgi_h` AS `lgi_h`, `xxxxxxx`.`auftraege`.`lgi_u` AS `lgi_u`, `xxxxxxx`.`auftraege`.`lgi_pw` AS `lgi_pw`, `xxxxxxx`.`auftraege`.`autoRemind` AS `autoRemind`, `xxxxxxx`.`kunden`.`telefon` AS `telefon`, `xxxxxxx`.`kunden`.`mobilfunk` AS `mobilfunk`, `xxxxxxx`.`auftraege`.`kommentar` AS `kommentar`, `xxxxxxx`.`auftraege`.`phase` AS `phase`, `xxxxxxx`.`auftraege`.`datum` AS `datum`, `xxxxxxx`.`angebot`.`typ` AS `typ`,
case
`xxxxxxx`.`auftraege`.`gekuendigt`
when
'1'
then
'Ja'
else
'Nein'
end
AS `Gekuendigt ? `,
(
select
count(`xxxxxxx`.`status`.`aenderung`)
from
`xxxxxxx`.`status`
where
`xxxxxxx`.`status`.`auftrag_id` = `xxxxxxx`.`auftraege`.`auftrag_id`
)
AS `aenderungen`,
`xxxxxxx`.`auftraege`.`vertragStart` AS `vertragStart`,
`xxxxxxx`.`auftraege`.`vertragEnde` AS `vertragEnde`,
case
`xxxxxxx`.`auftraege`.`zahlungsart`
when
'U'
then
'Überweisung'
when
'L'
then
'Lastschrift'
else
'Unbekannt'
end
AS `Zahlungsart`, `xxxxxxx`.`kunden`.`yyyyy_piwik` AS `yyyyy_piwik`,
(
select
max(`xxxxxxx`.`status`.`datum`) AS `mxDTst`
from
`xxxxxxx`.`status`
where
`xxxxxxx`.`status`.`auftrag_id` = `xxxxxxx`.`auftraege`.`auftrag_id`
and `xxxxxxx`.`status`.`typ` = 'SEO'
)
AS `mxDTst`,
(
select
case
`xxxxxxx`.`rechnungen`.`beglichen`
when
'YES'
then
'isOk'
else
'isAffe'
end
AS `neuUwe`
from
(
`xxxxxxx`.`zahlungsplanneu`
join
`xxxxxxx`.`rechnungen`
on(`xxxxxxx`.`zahlungsplanneu`.`rechnungsnummer` = `xxxxxxx`.`rechnungen`.`rechnungsnummer`)
)
where
`xxxxxxx`.`zahlungsplanneu`.`auftrag_id` = `xxxxxxx`.`auftraege`.`auftrag_id`
and `xxxxxxx`.`rechnungen`.`beglichen` <> 'STO' limit 1
)
AS `neuer`,
(
select
group_concat(`xxxxxxx`.`kunden_keywords`.`keyword` separator ',')
from
`xxxxxxx`.`kunden_keywords`
where
`xxxxxxx`.`kunden_keywords`.`kunde_id` = `xxxxxxx`.`kunden`.`kunde_id`
)
AS `keyword`,
(
select
case
count(0)
when
0
then
'Cool'
else
'Uncool'
end
AS `AusfallVor`
from
`xxxxxxx`.`rechnungen`
where
`xxxxxxx`.`rechnungen`.`rechnung_tag` < current_timestamp() - interval 15 day
and `xxxxxxx`.`rechnungen`.`kunde_id` = `xxxxxxx`.`kunden`.`kunde_id`
and `xxxxxxx`.`rechnungen`.`beglichen` = 'NO' limit 1
)
AS `Liquidiert`
from
(
((((`xxxxxxx`.`auftraege`
join
`xxxxxxx`.`angebot`
on(`xxxxxxx`.`auftraege`.`angebot_id` = `xxxxxxx`.`angebot`.`angebot_id`))
join
`xxxxxxx`.`kunden`
on(`xxxxxxx`.`angebot`.`kunde_id` = `xxxxxxx`.`kunden`.`kunde_id`))
left join
`xxxxxxx`.`kunden_keywords`
on(`xxxxxxx`.`angebot`.`kunde_id` = `xxxxxxx`.`kunden_keywords`.`kunde_id`))
join
`xxxxxxx`.`personal`
on(`xxxxxxx`.`kunden`.`bearbeiter` = `xxxxxxx`.`personal`.`personal_id`))
left join
`xxxxxxx`.`status`
on(`xxxxxxx`.`auftraege`.`auftrag_id` = `xxxxxxx`.`status`.`auftrag_id`)
)
group by
`xxxxxxx`.`auftraege`.`auftrag_id`
order by
NULL
UPDATE 1
1. The View Itself (Duration 1.83 sec)
1.1 Create the View: This is the View i created, it only contains the query from above.
1.2 Executing the View: It takes 1.83 sek to execute the view
1.3 Analyze the View: This is the explain of the view
2. The view with added where clause (Duration 1.86 sec)
2.1 Analyze the View with added where clause #rick wanted me to add a where clause to the view, if i understood him correctly. This is the explain of the view, where i added a where clause, takes 1.86 sec.
3. The Query, that is the source of the view (Duration: 0.1 sec)
3.1 Execute the query directly This is the query, that is the source of the view, when i execute it directly to the server. It takes ~0.1 - 0.2 seconds.
3.2 Analyze the direct queryAnd this is the explain of the pure query.
Why the view is so much slower, by only cupsuling the query inside of the view?
Update 2
These are the indexes I have set
ALTER TABLE angebot ADD INDEX angebot_idx_angebot_id (angebot_id);
ALTER TABLE auftraege ADD INDEX auftraege_idx_auftrag_id (auftrag_id);
ALTER TABLE kunden ADD INDEX kunden_idx_kunde_id (kunde_id);
ALTER TABLE kunden_keywords ADD INDEX kunden_keywords_idx_kunde_id (kunde_id);
ALTER TABLE personal ADD INDEX personal_idx_personal_id (personal_id);
ALTER TABLE rechnungen ADD INDEX rechnungen_idx_rechnungsnummer_beglichen (rechnungsnummer,beglichen);
ALTER TABLE rechnungen ADD INDEX rechnungen_idx_beglichen_kunde_id_rechnung (beglichen,kunde_id,rechnung_tag);
ALTER TABLE status ADD INDEX status_idx_auftrag_id (auftrag_id);
ALTER TABLE status ADD INDEX status_idx_typ_auftrag_id_datum (typ,auftrag_id,datum);
ALTER TABLE zahlungsplanneu ADD INDEX zahlungsplanneu_idx_auftrag_id (auftrag_id);
Be consistent between tables. kunde_id, for example, seems to be declared differently between tables. This may be preventing some obvious optimizations. (There are 6 JOINs that say func in EXPLAIN`.)
Remove the extra parentheses in JOINs. They may be preventing what the Optimizer is happy to do -- rearrange the tables in a JOIN.
Turn the query inside out. By this, I mean to do the minimum amount of work to do the main JOIN. Collect mostly id(s). Then do the dependent subqueries in an outer select. Something like:
SELECT ... ( SELECT ... ), ...
FROM ( SELECT a1.id
FROM a AS a1
JOIN b ON ..
JOIN c ON .. )
JOIN a AS a2 ON a2.id = a1.id
JOIN d ON ...
The "inside-out" kludge may eliminate the need for the GROUP BY. (Your query is too complex for me to see for sure.) If so, then I call the problem "explode-implode" -- Your query first JOINs, producing a temp table with lots of rows ("explodes"). Then it does a GROUP BY ("implodes").
More
These indexes will probably help:
status: (auftrag_id, typ, datum, aenderung)
rechnungen: (beglichen, kunde_id, rechnung_tag)
rechnungen: (rechnungsnummer, beglichen)
zahlungsplanneu: (auftrag_id, rechnungsnummer)
kunden_keywords: (kunde_id, keyword) -- (unless `kunde_id` is the PK)
(I see from all 3 EXPLAINs that you probably have sufficient indexes on kunden_keywords and status. Show me what indexes you have, so I can see if the existing indexes are as good as my suggestions.) "Using index" == "covering index".
Near the end is this LEFT JOIN, but I did not spot any use for the table; perhaps it can be removed?
left join `kunden_keywords` on(`angebot`.`kunde_id` = `kunden_keywords`.`kunde_id`))
SELECT `id`, `field2`, `field3`, `field4` as Title FROM `articles`
WHERE `category_id` = 'X'
AND `id` NOT IN
(SELECT `articleid` FROM `article-seen` WHERE `userid` = 'Y')
How can I optimize this?
I think double select is bad, but im new to mysql
Try using JOIN that will get you the same result but makes the query looks simpler
The optimization depends (I think) on the version of MySQL you are using.
This is how you write it as a join:
SELECT `id`, `field2`, `field3`, `field4` as Title
FROM `articles` a left outer join
`articles_seen` arts
on a.id = arts.articleid and arts.userid = 'Y'
where a.`category_id` = 'X' and
arts.id is null;
This query, at least, doesn't materialize the subquery, which (I think) your originally query would.
To makes this faster, you want to add indexes. The ones that come to mind are: articles(category_id, id) and articles_seen(userid, articleid). You could also add the fields in the select to the first index, so the entire query can be satisfied by looking at the index, rather than returning to the main table.
I found some strange(for me) behavour in MySQL. I have a simple query:
SELECT CONVERT( `text`.`old_text`
USING utf8 ) AS stext
FROM `text`
WHERE `text`.`old_id` IN
(
SELECT `revision`.`rev_text_id`
FROM `revision`
WHERE `revision`.`rev_id`
IN
(
SELECT `page_latest`
FROM `page`
WHERE `page_id` = 108
)
)
when i run it, phpmyadmin show execution time of 77.0446 seconds.
But then i replace
WHERE `text`.`old_id` IN
by
WHERE `text`.`old_id` =
it's execution time falls to about 0.001 sec. Result of this query
SELECT `revision`.`rev_text_id`
FROM `revision`
WHERE `revision`.`rev_id`
IN
(
SELECT `page_latest`
FROM `page`
WHERE `page_id` = 108
)
is
+------------+
|rev_text_id |
+------------+
|6506 |
+------------+
Can somebody please explain this behavour?
try to add INDEX on the following columns,
ALTER TABLE `text` ADD INDEX idx_text (old_id);
ALTER TABLE `revision` ADD INDEX idx_revision (rev_text_id);
and Execute the following query
SELECT DISTINCT CONVERT(a.`old_text` USING utf8 ) AS stext
FROM `text` a
INNER JOIN `revision` b
ON a.`old_id` = b.`rev_text_id`
INNER JOIN `page` c
ON b.`rev_id` = c.`page_latest`
WHERE c.`page_id` = 108
PS: Can you run also the following query and post their respective results?
DESC `text`;
DESC `revision`;
DESC `page`;
There are two primary ways you can increase your query performance here
Add Indexes (such as Kuya mentioned)
Rid yourself of the subqueries where possible
For Indexes, add an index on the columns you are searching for your matches:
text.old_id, revision.rev_text_id & page.page_id
ALTER TABLE `text` ADD INDEX idx_text (old_id);
ALTER TABLE `revision` ADD INDEX idx_revision (rev_text_id);
ALTER TABLE `page` ADD INDEX idx_page (page_id);
Your next issue is that nested-sub-selects are hell on your query execution plan. Here is a good thread discussing JOIN vs Subquery. Here is an article on how to get execution plan info from mySQL.
First looks at an execution plan can be confusing, but it will be your best friend when you have to concern yourself with query optimization.
Here is an example of your same query with just joins ( you could use inner or left and get pretty much the same result). I don't have your tables or data, so forgive synax issues (there is no way I can verify the code works verbatim in your environment, but it should give you a good starting point).
SELECT
CONVERT( `text`.`old_text` USING utf8 ) AS stext
FROM `text`
-- inner join only returns rows when it can find a
-- matching `revision`.`rev_text_id` row to `text`.`old_id`
INNER JOIN `revision`
ON `text`.`old_id` = `revision`.`rev_text_id`
-- Inner Join only returns rows when it can find a
-- matching `page_latest` row to `page_latest`
INNER JOIN `page`
ON `revision`.`rev_id` = `page`.`page_latest`
WHERE `page`.`page_id` = 108
MySQLDB is looping through each result of the inner query and comparing it with each record in the outer query.
in the second inner query;
WHERE `revision`.`rev_id`
IN
( SELECT `page_latest`
FROM `page`
WHERE `page_id` = 108
you should definitely use '=' instead of IN, since you're selecting a distinct record, there would be no point in looping through a result when you know only one record will be returned each time
What's the difference between these two queries:
SELECT `threads`.`id` AS `threads.id` , `posts`.`id` , `posts`.`content`
FROM `threads`
JOIN `posts` ON `threads`.`id` = `posts`.`thread_id`
And
SELECT `threads`.`id` AS `threads.id` , `posts`.`id` , `posts`.`content`
FROM `threads` , `posts`
WHERE `threads`.`id` = `posts`.`thread_id`
They both return same data.
WHERE clause filtering result set which is returned by JOIN, so this is a difference.
As long as you are using INNER JOIN there is no differences neither performance nor execution plan, in case of any OUTER JOIN query would produce different execution plan.
Also pay attention to what is said in the MySql online doc:
Generally, you should use the ON clause for conditions that specify
how to join tables, and the WHERE clause to restrict which rows you
want in the result set.
One uses ANSI Joins the other is using pre-ansi style join. MOST DB engines will compile them into the same execution plan.
In one word: Readability.
Running the following code:
create table #threads (
id int
)
create table #posts (
id int,
thread_id int,
content varchar(10)
)
insert into #threads values (1)
insert into #threads values (2)
insert into #posts values (1, 1, 'Stack')
insert into #posts values (2, 2, 'OverFlow')
SELECT #threads.id AS 'threads.id' , #posts.id , #posts.content
FROM #threads
JOIN #posts ON #threads.id = #posts.thread_id
SELECT #threads.id AS 'threads.id' , #posts.id , #posts.content
FROM #threads, #posts
WHERE #threads.id = #posts.thread_id
drop table #threads
drop table #posts
in https://data.stackexchange.com/stackoverflow/query/new you'll get the same execution plan :)
The only real difference is that inner join is ANSI and the from #threads, #posts is Transact-SQL syntax.
There might be difference in internal implementation of those queries, but I doubt it.
I'd personally prefer JOIN because it makes execution plan more obvious.
Im trying to build a complex (well...) query with Zend_Db_Table where I will both need to join the original table with an extra table and get some extra info from the original table with zend_db_expr.
However, things go wrong from the start. What I to is this:
$select = $this->getDbTable()->select(Zend_Db_Table::SELECT_WITH_FROM_PART)
->setIntegrityCheck(false);
$select->from( $this->getDbTable() , array(
'*' ,
new Zend_Db_Expr('`end` IS NULL as isnull') ,
new Zend_Db_Expr('`sold` IN (1,2,3) as issold') ,
) );
Zend_Debug::dump( $select->__toString() );exit;
What results in this:
SELECT `items`.*, `items_2`.*, `end` IS NULL as isnull, `sold` IN (1,2,3) as issold FROM `items`
INNER JOIN `items` AS `items_2`
What I need it to be though, at this point before I do the join with the other table, is
SELECT `items`.*, `end` IS NULL as isnull, `sold` IN (1,2,3) as issold FROM `items`
I don't need an inner join with itself, I just need to add those two Zend_Db_Expr to the things that should be selected, after which I'd continue building the query with the join and where's etc. like
$select->joinLeft( ... )
->where(...)
Any ideas? Cheers.
You should not redo a ->from() call, which means yu add a new table in the query.
Instead you should just use ->where()->columns() calls containing you Zend_Db_expr.
edit: sorry for the mistake.