I am migrating a database from SQL Server 2000 to SQL Server 2014. For some reason a query with the following HAVING clause does not return any entries after more than an hour's execution time, while it returns all the results in less than a minute from the old server:
HAVING (INOB.OBTAB = 'MARA')
AND (isnumeric(LEFT(RIGHT(INOB.OBJEK, 10), 7)) = 1)
AND (LEN(LEFT(RIGHT(INOB.OBJEK, 10), 7)) = 7)
AND (KLAH.KLART = '023')
AND (NOT (MVKE.LVORM = 'x'))
AND (isnumeric(RIGHT(INOB.OBJEK,3)) = 1)
The culprit seems to be:
AND (LEN(LEFT(RIGHT(INOB.OBJEK, 10), 7)) = 7)
If I leave this condition out, the query returns results in about half a minute from the new server. My question is: Why? I don't see any computational hardship for the server in measuring the length of a string? And why would it differ between the two servers?
Thanks in advance for any pointers you might give me.
/Jens
Related
I have an app that runs a complex join and it returns in a few seconds. No big deal. But when I run that same exact query in Workbench 8.0.12, it takes over a minute. The connection string is identical.
This post doesn't help, seems to be a different issue.
My app is .NET using MySQL.Data dll. I have used Workbench many times in the past, but usually with PHP apps. From years of experience I have always seen Workbench run a query in roughly the same time as the app can run it.
Here is my query:
SELECT
model.Code as ModelCode,
prod.parts_group as PartsGroup,
rules.id AS ID,
rules.Code as ConstraintName,
REPLACE(REPLACE(REPLACE(REPLACE(rules.statement, '''', ''), ' me.', ''), '}and ', '} and '), '}or ', '} or ') as Statement,
ifnull(rules.priority,5) AS Priority,
model.serie_id AS SeriesID
FROM
toy_product_constraint as rules
INNER JOIN toy_product as prod ON rules.product_id = prod.id
INNER JOIN toy_product_model as model ON prod.model_id = model.id
JOIN ModelImport mi ON model.code = mi.ModelCode
WHERE
rules.statement <> 'TRUE'
AND rules.market_id is null
AND rules.statement not like 'hasSDR(true)%'
AND rules.statement like 'SPECOPTION{00%'
The last 3 where clauses seem to be adding 99% of the extra query time:
AND rules.market_id is null
AND rules.statement not like 'hasSDR(true)%'
AND rules.statement like 'SPECOPTION{00%'
Without those last 3, the query runs in 1 second in Workbench. With those last 3 it takes over 60 seconds in Workbench. We could talk about indexes and other things, but the real mystery is why does this same query run (and return records) in less than 3 seconds in my app (I have stepped through the code line-by-line and verified this) and over 60 seconds in workbench??
I found the answer: The built-in Limit to 500 rows, etc, seems to be REALLY slow to figure out which 500 to do. When I open it wide, it runs fast. It's a large dataset.
I am hoping someone can point me in the right direction with this. I am relatively new to using pass-through queries but think I have a grip of the basics, I have come across a stumbling block however when trying to add a prefix to query result
I have a select query which includes a line to convert a date into the financial year i.e 01/01/2018 would return 2017 as the result using the code below:
[Created FY] = (CASE WHEN Month(create_date) IN (1, 2, 3) THEN DatePart(year,create_date)-1 ELSE DatePart(year,create_date) END),
I would like to add a prefix to the result so that it would read FY2017. The pass-through is running on SQL Server 2008. I have researched and so far have not come up with any resolutions.
Any help with this conundrum would be greatly appreciated.
+ concatenates strings, but numbers must be converted first.
As there is a MONTH() function, there is YEAR(), to simplify the code a bit.
So:
[Created FY] = 'FY' + CONVERT(char(4),
(CASE WHEN MONTH(create_date) IN (1, 2, 3)
THEN YEAR(create_date)-1
ELSE YEAR(create_date) END)
),
I have a thermometer which is storing all the reading data.
Example:
http://sqlfiddle.com/#!9/c0aab3/9
The idea is to obtain the time that remained with temperature above 85 fahrenheit.
I have invested everything according to my knowledge and I have not been able to find the solution.
Currently, what I'm doing is getting the time when I went above 85 and then getting the next low value of 85 to calculate the difference in time.
If the temperature is maintained at 85 for 5 consecutive minutes the data may fail.
Please, what would be the way to calculate this?
According to the example of sqlfiddle, the results shown are greater than or equal to 85, but in some cases it was not maintained means that low.
that peak from the beginning to the low must be taken and the time is calculated in seconds, therefore, I must do it successively until the end.
Then add all the seconds and get the time.
Base answer (no modification on the table)
I could find a way around with variables and some IF functions that manipulate them. See if this works for you :
SET #currIndex = 1;
SET #indicator = FALSE;
SET #prevIndex = 0;
SELECT Q2.sen,
MIN(Q2.subTime) as 'From',
MAX(Q2.subTime) AS 'To',
TIMEDIFF(MAX(Q2.subTime), MIN(Q2.subTime)) as diff
FROM (SELECT IF(Q1.temp < 85,
IF(#prevIndex = #currIndex,
(#currIndex := #currIndex +1) -1,
#currIndex),
#prevIndex := #currIndex) AS 'Ind',
Q1.subTime,
Q1.temp,
Q1.sen
FROM (SELECT IF(sen_temp.temp < 85,
(#indicator XOR (#indicator := FALSE)),
#indicator := TRUE) as ind,
s_time AS subTime,
temp,
sen
FROM sen_temp
) AS Q1
WHERE Q1.ind
) AS Q2
GROUP BY Q2.`Ind`
ORDER BY Q2.subTime;
Here's an SQL fiddle based on your own.
The main problem of this query is its performance. Since there is no ID on the table, data had to be carried through the queries.
Performance answer (table modification required)
After a lot of optimization work, I ended up adding an ID to the table. It allowed me to have only one sub query instead of 2 and to carry less data in the sub query.
SET #indicator = FALSE;
SET #currentIndex = 0;
SELECT T1.sen, MIN(T1.s_time) as 'From', MAX(T1.s_time) AS 'To',
TIMEDIFF(MAX(T1.s_time), MIN(T1.s_time)) as diff
FROM (SELECT id, (CASE WHEN (temp >= 85) THEN
#currentIndex + (#indicator := TRUE)
WHEN (#indicator) THEN
#currentIndex := #currentIndex + (#indicator := FALSE) + 1
ELSE
#indicator := FALSE
END) as ind
FROM sen_temp
ORDER BY id, s_date, s_time) AS Q1
INNER JOIN sen_temp AS T1 ON Q1.id = T1.id
WHERE Q1.ind > 0
GROUP BY T1.sen, Q1.ind
Please check this fiddle for this more efficient version.
Performance difference explanation
When creating a MySQL Query, performance is always key. If it is simple, the query will execute efficiently and you should not have any problem unless you get into some syntax error or other optimization problems like filtering or ordering data on a field that's not indexed.
When we create sub-queries, it's harder for the database to handle the query. The reason is quite simple : it potentially uses more RAM. When a query containing sub-queries is executed, the sub-queries are executed first ("obviously!" you might say). When a sub-query is executed, the server needs to keep those values for the upper-query, so it kind of creates a temporary table in the RAM, allowing itself to consult the data in the upper-query if it needs to. Even though RAM is quite fast, it may seem slowed down a lot when handling a ludicrous amount of data. It is even worse when the query makes the database server handle so much data that it won't even fit in the RAM, forcing the server to use the much slower system's storage.
If we limit the amount of data generated in sub-queries to the minimum and then only recover wanted data in the main query, the amount of RAM the server uses for the sub-queries is more negligible and the query runs faster.
Therefore, the smaller the data amount is in sub-queries, the faster the whole query will execute. This much is true for pretty much all "SQL like" databases.
Here's a nice reading explaining how queries are optimized
Here is my logic
I want to get the closest 4 more expensive mobiles to a specific mobile #mobile but under one condition the difference between the release dates of the two mobiles is not more than a year and half
Here is the query
high = Mobile.where("price >= #{#mobile.price} AND id != #{#mobile.id} AND visible = true").where("ABS(release_date - #{#mobile.release_date}) > ?", 18.months).order(price: :ASC).first(4)
The first .where() works perfectly but the second is not working and I get this error
Mysql2::Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '00:00:00 UTC) > 46656000) ORDER BY `mobiles`.`price` ASC LIMIT 4' at line 1: SELECT `mobiles`.* FROM `mobiles` WHERE (price >= 123123.0 AND id != 11 AND visible = true) AND (ABS(release_date - 2016-04-10 00:00:00 UTC) > 46656000) ORDER BY `mobiles`.`price` ASC LIMIT 4
I think now you can get my logic. What is the right syntax to achieve it?
A couple of tips here:
It is a dangerous practice to concatenate variables into your queries using the "#{}" operator. Doing so bypasses query parameterization and could leave your app open to SQL injection. Instead, use "?" in your where clause.
The reason MySQL is giving you an error is because you are concatenating a string into your query without encapsulating it in quotes.
With these two things in mind, I would start by refactoring your query like so:
high = Mobile.where("price >= ?", #mobile.price)
.where.not(id: #mobile.id)
.where(visible: true)
.where("ABS(release_date - ?) > 46656000", #mobile.release_date)
.order(price: :ASC).first(4)
You will note that I replaced 18.months with 46656000. This saves a few clock cycles in the Rails app. Depending on your database schema, the last where clause may not work. The modification below may end up working better.
As a further refinement, you could refactor your last where clause to look for a release date that is between 18 months before #mobile.release_date and 18 months after. The saves your MySql database from having to do the math on each record and may lead to better performance:
.where(release_date: (#mobile.release_date - 18.months)..(#mobile.release_date + 18.months) )
I do not know your database schema, so you may run into date conversion problems with the code above. I recommend you play with it in the Rails console.
Use a Range to query between dates/times:
Mobile.where("price >= ?", #mobile.price)
.where.not(id: #mobile.id)
.where(release_date: 18.months.ago..Time.now)
.order(price: :ASC)
.first(4)
This query (or rather one similar to it) is in a codebase we have already deployed.
var timeblocks = from tb in DB.TimeBlocks
where tb.StartDate.Date.AddDays(1) <= DateTime.Today
select tb.Id;
DB is a datacontext that connects to the database. TimeBlocks is a fairly simple table, StartDate is a DateTime column.
Currently the database is hosted on a Sql Server 2005 Installation, but we are in the process of upgrading to a Sql Server 2008 Installation.
The query currently exectutes without problems.
If we change the connection string to point at a copy of the same database running on Sql Server 2008 (with the compatability level set for 2005) the query fails with the SqlException:
"The datepart millisecond is not supported by date function dateadd for data type date."
This seems to be due to the different sql emitted by Linq to SQL when it connects to the 2008db (I assume this is because it uses Sql2008Provider instead of the Sql2005Provider).
The 2005 Provider produces good SQL:
SELECT [t0].[Id]
FROM [dbo].[TimeBlock] AS [t0]
WHERE DATEADD(ms, (CONVERT(BigInt,#p0 * 86400000)) % 86400000, DATEADD(day, (CONVERT(BigInt,#p0 * 86400000)) /
86400000, DATEADD(HOUR, -DATEPART(HOUR, [t0].[StartDate]), DATEADD(MINUTE, -DATEPART(MINUTE,
[t0].[StartDate]), DATEADD(SECOND, -DATEPART(SECOND, [t0].[StartDate]), DATEADD(MILLISECOND,
-DATEPART(MILLISECOND, [t0].[StartDate]), [t0].[StartDate])))))) <= #p1
which successfully executes the query. However the sql emitted by the Sql2008Provider:
SELECT [t0].[Id]
FROM [dbo].[TimeBlock] AS [t0]
WHERE DATEADD(ms, (CONVERT(BigInt,#p0 * 86400000)) % 86400000, DATEADD(day, (CONVERT(BigInt,#p0 * 86400000)) /
86400000, CONVERT(DATE, [t0].[StartDate]))) <= #p1
Contains the erroneous sql that causes the exception.
Am I right in thinking that the it is the Sql provider that is causing this problem?
Is there a way we can get round this problem by forcing the DataContext to use the Sql2005Provider for this db?
Thanks for any help you can give us!
It seems to me you found a bug in LINQ to SQL. You should report it to Microsoft. You can do that here: http://connect.microsoft.com/