Hi I am trying to run a recusrive query in SQL Server 2008. For specific values, the query seems to run forever. For other values, it is very quick (<5s)
Is there anything I could do in sql to prevent query running for more than say 10s?
WITH tree (FID,TID, level) AS (
SELECT FID, TID, 1 as level
FROM table1
WHERE FID = '202223268'
UNION ALL
SELECT child.FID, child.TID, parent.level + 1
FROM table1 as child
JOIN tree parent on parent.TID = child.FID
)
SELECT FID,TID
FROM tree
option (maxrecursion 0)
I wouldnt say you can limit the execution timeout on query basis (you can on server level), but if you limit the recursion depth (option maxrecursion) you can achieve very similar result.
Related
I have a table in MariaDB with different nodes, each node can have multiple hardware components, and I want to count how many hardware components each node has and store it in a column.
I have therefore created an empty column called "HW_Count". I can do this calculation through the following SELECT statement.
SELECT NodeID,COUNT(NodeId) OVER (PARTITION BY NodeId) FROM Hardware AS HW_Count
This returns a list of the following kind
NodeID HWCount
1 33
1 33
... ...
2 11
2 11
... ...
Now I want to UPDATE the empty column in the table called Hardware with tihs information, but I am not sure how I can write the correct UPDATE statement. I want to do something along the lines
UPDATE Hardware
SET HW_count =
COUNT(NodeId) OVER (PARTITION BY NodeId)
But it returns
"SQL ERROR (4015): Window function is allowed only in SELECT list and ORDER BY clause".
What is the correct way to update my column?
Thanks!
I managed to find the answer
UPDATE Hardware, (SELECT NodeID,COUNT(NodeID) AS `HW_Count` FROM
Hardware GROUP BY NodeID) AS dummyTable
SET Hardware.hw_count = Dummytable.Hw_count
WHERE Hardware.NodeID= dummytable.NodeID
Try this please:
UPDATE H1
SET HW_count = (SELECT COUNT(*) HW_count FROM Hardware WHERE NodeID = H1.NodeID GROUP BY NodeID)
FROM Hardware H1
INNER JOIN Hardware H2 ON H1.NodeID=H2.NodeID
I would recommend doing this using JOIN and GROUP BY:
UPDATE Hardware h JOIN
(SELECT NodeID, COUNT(NodeID) AS new_hw_count
FROM Hardware
GROUP BY NodeID
) hh
ON hh.NodeID = h.NodeID
SET h.hw_count = h.new_hw_count;
Lets assume we have got query1 as follow :
select * from users where status = 1
this will output some results,I can cache these data, now the second query is :
select * from users where status = 1 and point >= 50
as you see the second query is somehow the child of first query, it returns a subset of last query data and has common code as well, is there a way which I can speed up my second query by using first query results and shorten my code using the first query code?
Yes, you use nested queries:
select x.*
from
(
select * from users
where status = 1
) as x
where x.point >= 50;
I have this method that using find_by_sql which is return 10 latest records of each source
def latest_results
Entry.find_by_sql(["
select x.id,x.created_at,x.updated_at,x.source_id,x.`data`,x.`uuid`,x.source_entry_id
from
(select t.*,
(#num:=if(#group = `source_id`, #num +1, if(#group := `source_id`, 1, 1))) row_number
from (
select d.id,d.created_at,d.updated_at,d.source_id,d.`data`,d.`uuid`,d.source_entry_id
from `streams` a
JOIN `stream_filters` b
on b.stream_id=a.id
JOIN `filter_results` c
on c.filter_id=b.id
JOIN `entries` d
on d.id=c.entry_id
where a.id=?
) t
order by `source_id`,created_at desc
) as x
where x.row_number <= 10
ORDER BY x.created_at DESC
",self.id])
end
It's working properly on local environment with limited records.
I have t2.micro which has 2 Gib memory to serving the application. Now this query running out my whole memory and app get frizzing.
any suggestion how can I do it better ? I want to solve this without increasing the size of machine.
I had a similar problem once. The solution with mysql variables seems neat at the first place, though it is hard to optimize. It seems that is doing a full table scan in your case.
I would recommend to fetch the sources you want to display first. And then run a second query with multiple top 10 selects, one per source, all combined with a union.
The union top 10 select will have some repetive statements which you can easily autogenerate with ruby.
# pseudo code
sources = Entry.group(:source).limit(n)
sql = sources.map do |source|
"select * from entries where source = #{source} order by created_at limit 10"
end.join("\nunion all\n")
Entry.find_by_sql(sql)
I am running into a small problem,
This is a demo query
select
A.order_id,
if(
A.production_date != '0000-00-00',
A.production_date,
if(
SOME INNER QUERY != '0000-00-00',
SOME INNER QUERY ,
SOME OTHER INNER QUERY
)
) as production_start_date
from
orders A
So basically, suppose the SOME INNER QUERY is taking 10 seconds to do its calculations, fetching data from 8 different tables, checking past history for same order type etc. and if its result is a date, I fetch that date in first condition. But now it will take 20 seconds as 10 seconds for calculation for if condition, and 10 seconds to re-execute to return the result.
Is there any way I can reduce this?
if any one is interested in looking at actual query http://pastebin.com/zqzbpEei
Assuming your query looks like this (sorry, I gave up trying to locate the actual query):
IF(
(SELECT aField FROM aTable WHERE bigCondition) != '0000-00-00',
SELECT aField FROM aTable WHERE bigCondition,
SELECT anotherField FROM anotherTable
)
You can rewrite it as follows:
SELECT IF (
someField != '0000-00-00',
someField,
SELECT anotherField FROM anotherTable
)
FROM aTable WHERE bigCondition
This way you compute bigCondition only once.
This query is quite ugly indeed.
Your major problem seems to be the misuse (and abuse, big time) of the IF() construct. It should be reserved to simple conditions and operations. The same applies to logical operators. Do not operate on entire queries. For instance, I see this one bit appears a few times in your query:
IF(
(SELECT v1.weekends FROM vendor v1 WHERE v1.vendor_id = A.vendor_id) IS NULL
OR (SELECT v1.weekends FROM vendor v1 WHERE v1.vendor_id = A.vendor_id) = '',
'6', -- by the way, why is this a string?! This is an integer, isn't it?
(SELECT v1.weekends FROM vendor v1 WHERE v1.vendor_id = A.vendor_id)
)
This is Bad. The condition should be moved into the SELECT directly. Rewrite it as below:
SELECT
IF (v1.weekends IS NULL OR v1.weekends = '', 6, v1.weekends)
FROM vendor v1 WHERE v1.vendor_id = A.vendor_id
That's two SELECT saved. Do this for every IF() that contains a query, and I am ready to bet you are going to speed up your query by several orders of magnitude.
There is a lot more to say about your current code. Unfortunately, you will probably need to refactor some parts of your ORM. Add new, more specialised methods to some classes, and make them use new queries that you crafted manually. Then refactor your current operation so that it uses these new methods.
Below two queries are subqueries. Both are the same and both works fine for me. But the problem is Method 1 query takes about 10 secs to execute while Method 2 query takes under 1 sec.
I was able to convert method 1 query to method 2 but I don't understand what's happening in the query. I have been trying to figure it out myself. I would really like to learn what's the difference between below two queries and how does the performance gain happen ? what's the logic behind it ?
I'm new to these advance techniques. I hope someone will help me out here. Given that I read the docs which does not give me a clue.
Method 1 :
SELECT
*
FROM
tracker
WHERE
reservation_id IN (
SELECT
reservation_id
FROM
tracker
GROUP BY
reservation_id
HAVING
(
method = 1
AND type = 0
AND Count(*) > 1
)
OR (
method = 1
AND type = 1
AND Count(*) > 1
)
OR (
method = 2
AND type = 2
AND Count(*) > 0
)
OR (
method = 3
AND type = 0
AND Count(*) > 0
)
OR (
method = 3
AND type = 1
AND Count(*) > 1
)
OR (
method = 3
AND type = 3
AND Count(*) > 0
)
)
Method 2 :
SELECT
*
FROM
`tracker` t
WHERE
EXISTS (
SELECT
reservation_id
FROM
`tracker` t3
WHERE
t3.reservation_id = t.reservation_id
GROUP BY
reservation_id
HAVING
(
METHOD = 1
AND TYPE = 0
AND COUNT(*) > 1
)
OR
(
METHOD = 1
AND TYPE = 1
AND COUNT(*) > 1
)
OR
(
METHOD = 2
AND TYPE = 2
AND COUNT(*) > 0
)
OR
(
METHOD = 3
AND TYPE = 0
AND COUNT(*) > 0
)
OR
(
METHOD = 3
AND TYPE = 1
AND COUNT(*) > 1
)
OR
(
METHOD = 3
AND TYPE = 3
AND COUNT(*) > 0
)
)
An Explain Plan would have shown you why exactly you should use Exists. Usually the question comes Exists vs Count(*). Exists is faster. Why?
With regard to challenges present by NULL: when subquery returns Null, for IN the entire query becomes Null. So you need to handle that as well. But using Exist, it's merely a false. Much easier to cope. Simply IN can't compare anything with Null but Exists can.
e.g. Exists (Select * from yourtable where bla = 'blabla'); you get true/false the moment one hit is found/matched.
In this case IN sort of takes the position of the Count(*) to select ALL matching rows based on the WHERE because it's comparing all values.
But don't forget this either:
EXISTS executes at high speed against IN : when the subquery results is very large.
IN gets ahead of EXISTS : when the subquery results is very small.
Reference to for more details:
subquery using IN.
IN - subquery optimization
Join vs. sub-query.
Method 2 is fast because it is using EXISTS operator, where I MySQL do not load any results.
As mentioned in your docs link as well, that it omits whatever is there in SELECT clause. It only checks for the first value that matches the criteria, once found it sets the condition TRUE and moves for further processing.
On the other side Method 1 has IN operator which loads all possible values and then matches it. Condition is set TRUE only when exact match is found which is time consuming process.
Hence your method 2 is fast.
Hope it helps...
The EXISTS operator is a Boolean operator that returns either true or false. The EXISTS operator is often used the in a subquery to test for an “exist” condition.
SELECT
select_list
FROM
a_table
WHERE
[NOT] EXISTS(subquery);
If the subquery returns any row, the EXISTS operator returns true, otherwise, it returns false.
In addition, the EXISTS operator terminates further processing immediately once it finds a matching row. Because of this characteristic, you can use the EXISTS operator to improve the performance of the query in some cases.
The NOT operator negates the EXISTS operator. In other words, the NOT EXISTS returns true if the subquery returns no row, otherwise it returns false.
You can use SELECT *, SELECT column, SELECT a_constant, or anything in the subquery. The results are the same because MySQL ignores the select_list that appears in the SELECT clause.
The reason is that the EXISTS operator works based on the “at least found” principle. It returns true and stops scanning table once at least one matching row found.
On the other hands, when the IN operator is combined with a subquery, MySQL must process the subquery first and then uses the result of the subquery to process the whole query.
The general rule of thumb is that if the subquery contains a large volume of data, the EXISTS operator provides a better performance.
However, the query that uses the IN operator will perform faster if the result set returned from the subquery is very small.
For detail explanations and examples: MySQL EXISTS - mysqltutorial.org
The second Method is faster because you've got this like there "WHERE t3.reservation_id = t.reservation_id". In the first case your subquery has to do a full scan into the table to verify the information. However at the 2o Method the subquery knows exactly what it is looking for and once it is found is checked the having condition then.
Their Official Documentation.SubQuery Optimization with Exists