I have table with documents nammed: z_web_dok it has 133,369 rows, the other table with products nammed: z_web_dok_art has 693,930 rows, all fields needed for querying are indexed, and connection between those tables is auto increment field of z_web_dok.oid and z_web_dok_art.oid_id.
What is the problem?
I were made simple sql query for returning old prices (by buyer, place, country). The query is:
SELECT (COALESCE(NULLIF(TNew.cijena_e,''), TNew.cijena)*(TN.kurs_iznos/1))
FROM z_web_dok_art As TNew
INNER JOIN z_web_dok As TN ON TN.oid=TNew.oid_id
WHERE (TN.Drzava='BiH' AND TNew.aid='SOME_PRODUCT_ID' AND TN.vrsta_dok='pri'
AND TN.kup_id='1047' AND TN.mag_id='5' AND TN.oid<>'151967')
ORDER BY TN.dat_zavrsena DESC
LIMIT 1
and it worked, for each product, so i used it inside another query, which gets products their new prices and with the query above (subquery) i tried to pull the old prices (Connection was between tables: TNew.aid=WArt.aid )
SELECT WArt.aid, WArt.nc, WArt.isp, COALESCE(NULLIF(WArt.cijena_e,''), WArt.cijena) As cijena, WArt.rab, WArt.vpc, WArt.mpc, C.EAN, C.Model, C.Naziv, C.JM, C.WebBiH, C.DostupnostBiH,
(
SELECT (COALESCE(NULLIF(TNew.cijena_e,''), TNew.cijena)*(TN.kurs_iznos/1))
FROM z_web_dok_art As TNew
INNER JOIN z_web_dok As TN ON TN.oid=TNew.oid_id
WHERE (TN.Drzava='BiH' AND TNew.aid=WArt.aid AND TN.vrsta_dok='pri'
AND TN.kup_id='1047' AND TN.mag_id='5' AND TN.oid<>'151967')
ORDER BY TN.dat_zavrsena DESC
LIMIT 1
) As s_cijena
FROM
z_web_dok_art As WArt
LEFT JOIN Cjenovnik As C ON C.ID=WArt.aid
WHERE WArt.oid_id='151967'
ORDER BY CASE WHEN WArt.isp='0' THEN 1 ELSE 0 END, WArt.id_dok_art
It worked, for a long period, but today we discovered it returned 6 times higher result for old price for two products (on a order that has 19 products).
So i were query TN.oid with same query to see what does it pulls from database and it was oid that has following fields same: (kup_id, vrsta_dok) but all others are different even if the query asked for Drzava='BiH' AND TN.mag_id='5' it doesn't return results for that.
But the other problem is, if i execute subquery as query alone, with that product (or more them) it returns RIGHT RESULTS. But above one mixed results right and wrong one.
It's not problem in solving this!
I can solve it, but i wanted to know why this QUERY didn't work, i mean, what's the problem with it? Did anyone had problem with this?
This is my first bad experience with similar queries...
I were thinking about, it's happening because of subquery JOINS with another table inside of query (because the subquery works perfect without it's parent query).
Thank you for your time!
Best regards,
Nedžad.
IMPORTANT UPDATE (25.10.2016):
The server mysql version: 5.5.52-cll
Local version mysql: 5.6.17 - MySQL Community Server (GPL)
Localhost returned the proper results, with NO MISTAKE,
while the server mysql still returns wrong results..
Is it bug at mysql 5.5.52-cll or what?
Image of results:
UPDATE: (SOLVED)
And i solved it by using group in subquery:
GROUP BY TNew.oid_id, TNew.aid
(grouping by, documents, and products inside of it), and it returned the right results - the performance was good, 0.1704 (which is good because it's always one document at opening). Once again, thanks everyone which were lead me to right path
Related
I am currently optimising a system with many connected tables.
The part that I am working on right now is displaying table orders. The problem is that in this table there are also many relations (around 10) which I am querying. The problem itself is in querying that many relations. I have been using Eloquent + with() methods for eager loading but the page is already getting slow, even though it has just under 3 000 orders. The problem is that in table I also print data from 1:N relations (for example a order has many loadings and in the table I print the city of the first loading)
So since it is already getting slow with eloquent and eager loading, I have decided to switch to Query Builder. But it has a few flaws for me which I do not know how to resolve.
The query itself is already huge:
$orders = DB::table('orders')
->select([
//around 25 different selects
])
// nearly 10 left joins on other tables
->leftJoin(...)
// a few when-s with nested where-s, orderBy-s and so on
->when(...)
->get();
Firstly, I use joins to join together the tables of data, which is nice in case of (table)1:N(orders) but becomes a problem when connecting (orders)1:N(table), because I am not really trying to join all of those records. For nearly all of those I need just the latest or first record.
The first option I decided to use was subquery in select which I quickly realised was a huge mistake since it executes on every row. Then I found a solution by joining a subquery to my table, however the query takes too long (I only connected one like this and query was 8 seconds and I am supposed to join 6 times)
->leftJoin(DB::raw("(SELECT t1.fileable_id, t1.filename FROM files t1 INNER JOIN (SELECT fileable_id, MAX(created_at) AS max_created_at FROM files WHERE fileable_type = 'App\\Order' GROUP BY fileable_id) t2 ON t2.fileable_id = t1.fileable_id AND t2.max_created_at = t1.created_at) AS contractor_files"), 'contractor_files.fileable_id', '=', 'orders.id')
I wanted to ask whether or not there is any solution. I can post the whole query if needed, but it is about 100 lines long and is pretty much summed up
Something is wrong with the query. But this might help the subquery:
INDEX(fileable_type, fileable_id, created_at)
Also, don't you need to include fileable_type in the ON clause?
I need to find a value that exists in LoyaltyTransactionBasketItemStores table but not in DimProductConsolidate table. I need the item code and its corresponding company. This is my query
SELECT
A.ProductReference, A.CompanyCode
FROM
(SELECT ProductReference, CompanyCode FROM dwhdb.LoyaltyTransactionsBasketItemsStores GROUP BY ProductReference) A
LEFT JOIN
(SELECT LoyaltyVariantArticleCode FROM dwhdb.DimProductConsolidate) B ON B.LoyaltyVariantArticleCode = A.ProductReference
WHERE
B.LoyaltyVariantArticleCode IS NULL
It is a pretty straight forward query. But when I run it, it's taking 1 hour and still not finish. Then I use EXPLAIN and this is the result
But when I remove the CompanyCode from my query, its performance is increasing a lot. This is the EXPLAIN result
I want to know why is this happening and is there any way to get ProductReference and its company with a lot more better performance?
Your current query is rife with syntax and structural errors. I would use exists logic here:
SELECT a.ProductReference, a.CompanyCode
FROM dwhdb.LoyaltyTransactionsBasketItemsStores a
WHERE NOT EXISTS (SELECT 1 FROM dwhdb.DimProductConsolidate b
WHERE b.LoyaltyVariantArticleCode = a.ProductReference);
Your current query is doing a GROUP BY in the first subquery, but you never select aggregates, but rather other non aggregate columns. On most other databases, and even on MySQL in strict mode, this syntax is not allowed. Also, there is no need to have 2 subqueries here. Rather, just select from the basket table and then assert that matching records do not exist in the other table.
I have a query which joins 3 tables together. But I sometimes I get result that is twice what I expect.
For example if I run the query with s.Driver_Id = 15000 a total which is twice the stage time.
But if I run the sane query with s.Driver_Id = 15001 I get the expected result.
Here is an SQL Fiddle link to my tables and SQL query http://sqlfiddle.com/#!2/3204ea/12
Any help would be great. Thanks
It's because you are INNER JOINing to driverevent which has multiple rows per driver - each for a different event, but you're only joining on driver_id. So if that driver_id appears on 3 driverevent records, you end up JOINing all 3 rows even though. What you need to do is JOIN ON driver_id and event_id.
INNER JOIN driverevent de
on d.licence_no = de.driver_id and s.Event_Id = de.Event_Id
SQLFiddle
On closer inspection Alan, it appears to me that the logic behind your query may be wrong. As far as I can tell you don't need to be doing any GROUPing or SUMing at all.
I could be incorrect since I don't know the real world logic of the query, however consider this query with the GROUP BY and SUMs removed - SQLFiddle
You simply have two records with driver_id 15000 in the driverevent table.
I fail at mysql, and could really do with some help. I don't know what it would be called, and all my attempts at using combinations of DISTINCT and GROUP BY are just not working out.
I have a table of server monitoring data with these columns:
nStatusNumber
Bandwidth
Load
Users
ServerNumber
DiskFree
MemFree
TimeStamp
**nStatusNumber** - A unique number increasing for each entry
**ServerNumber** - A unique number for each server
For the top of my dashboard for this, I need to display the most recent report for each unique server.
// How many servers are we monitoring ?
$nNumServers = mysql_numrows(mysql_query("SELECT DISTINCT(ServerNumber) FROM server_status;"));
// Get our list of servers
$strQuery = "SELECT * FROM server_status ORDER BY nStatusNumber DESC limit ".$nNumServers.";";
And then loop through the results until we hit $nNumServers . This worked at first, until servers started going down/up and the report order got jumbled.
Say theres 20 servers, the most recent 20 results aren't necessarily 1 from each server.
I'm trying to figure this out in a hurry, and failing at it. I've tried all sorts of combinations of DISTINCT and GROUP BY with no luck so far and would appreciate any guidance on what's probably an embarrassingly easy problem that I just can't see the answer to.
Thanks!
PS - Here's an example query that I've been trying, showing the problem I'm having. Check the "nStatusNumber" field, these should be showing the most recent results only for each server - http://pastebin.com/raw.php?i=ngXLRhd6
PPS - Setting max(nStatusNumber) doesn't give accurate results. I don't want some average/sum/median figure, I need the most recent ACTUAL figures reported by each server. Heres more example results for the queries:
http://pastebin.com/raw.php?i=eyuPD7vj
For your purpose you need to find the row unique to a nServerNumber and TimeStamp. This is not as simple as just saying MAX(TimeStamp) as you need to find the row corresponding to it.
Although I am not an expert in SQL you can try this and see if it works.
SELECT A.nServerNumber, A.nStatusNumber, A.nVNStatsBandwidth, A.fLoad, A.nUsers,
A.nUsersPaid, A.nServerNumber, A.nFreeDisk, A.nTotalDisk, A.nFreeMemory,
A.nTotalMemory, A.TimeStamp
FROM server_status A
INNER JOIN
(
SELECT nServerNumber, MAX(TimeStamp) as `TimeStamp`
FROM server_status
GROUP BY nServerNumber
) B
ON A.nServerNumber = B.nServerNumber
AND A.TimeStamp = B.TimeStamp
ORDER BY A.nServerNumber ASC;
This query will give you all the servers with their latest info. So if you want the total number of servers just run the mysql_numrows(...) function on this result and if you want the data just iterate through the same result (no need to fire two separate SQL queries).
Try this ::
Select
Select MAX(nStatusNumber) from table,
Bandwidth,
Load,
Users,
ServerNumber,
DiskFree,
MemFree,
MAX(`TimeStamp`)
from your table
group by ServerNumber
Here's the query (the largest table has about 40,000 rows)
SELECT
Course.CourseID,
Course.Description,
UserCourse.UserID,
UserCourse.TimeAllowed,
UserCourse.CreatedOn,
UserCourse.PassedOn,
UserCourse.IssuedOn,
C.LessonCnt
FROM
UserCourse
INNER JOIN
Course
USING(CourseID)
INNER JOIN
(
SELECT CourseID, COUNT(*) AS LessonCnt FROM CourseSection GROUP BY CourseID
) C
USING(CourseID)
WHERE
UserCourse.UserID = 8810
If I run this, it executes very quickly (.05 seconds roughly). It returns 13 rows.
When I add an ORDER BY clause at the end of the query (ordering by any column) the query takes about 10 seconds.
I'm using this database in production now, and everything is working fine. All my other queries are speedy.
Any ideas of what it could be? I ran the query in MySQL's Query Browser, and from the command line. Both places it was dead slow with the ORDER BY.
EDIT: Tolgahan ALBAYRAK solution works, but can anyone explain why it works?
maybe this helps:
SELECT * FROM (
SELECT
Course.CourseID,
Course.Description,
UserCourse.UserID,
UserCourse.TimeAllowed,
UserCourse.CreatedOn,
UserCourse.PassedOn,
UserCourse.IssuedOn,
C.LessonCnt
FROM
UserCourse
INNER JOIN
Course
USING(CourseID)
INNER JOIN
(
SELECT CourseID, COUNT(*) AS LessonCnt FROM CourseSection GROUP BY CourseID
) C
USING(CourseID)
WHERE
UserCourse.UserID = 8810
) ORDER BY CourseID
Is the column you're ordering by indexed?
Indexing drastically speeds up ordering and filtering.
You are selecting from "UserCourse" which I assume is a joining table between courses and users (Many to Many).
You should index the column that you need to order by, in the "UserCourse" table.
Suppose you want to "order by CourseID", then you need to index it on UserCourse table.
Ordering by any other column that is not present in the joining table (i.e. UserCourse) may require further denormalization and indexing on the joining table to be optimized for speed;
In other words, you need to have a copy of that column in the joining table and index it.
P.S.
The answer given by Tolgahan Albayrak, although correct for this question, would not produce the desired result, in cases where one is doing a "LIMIT x" query.
Have you updated the statistics on your database? I ran into something similar on mine where I had 2 identical queries where the only difference was a capital letter and one returned in 1/2 a second and the other took nearly 5 minutes. Updating the statistics resolved the issue
Realise answer is too late, however I have just had a similar problem, adding order by increased the query time from seconds to 5 minutes and having tried most other suggestions for speeding it up, noticed that the /tmp files where getting to be 12G for this query. Changed the query such that a varchar(20000) field being returned was "trim("ed and performance dramatically improved (back to seconds). So I guess its worth checking whether you are returning large varchars as part of your query and if so, process them (maybe substring(x, 1, length(x))?? if you dont want to trim them.
Query was returning 500k rows and the /tmp file indicated that each row was using about 20k of data.
A similar question was asked before here.
It might help you as well. Basically it describes using composite indexes and how order by works.
Today I was running into a same kind of problem. As soon as I was sorting the resultset by a field from a joined table, the whole query was horribly slow and took more than a hundred seconds.
The server was running MySQL 5.0.51a and by chance I noticed that the same query was running as fast as it should have always done on a server with MySQL 5.1. When comparing the explains for that query I saw that obviously the usage and handling of indexes has changed a lot (at least from 5.0 -> 5.1).
So if you encounter such a problem, maybe your resolution is to simply upgrade your MySQL