I have a database with two tables. The one contains accounts, and the other contains over 2 million rows containing addresses and their coordinates. Obviously with such an amount of rows, any time a query runs that doesn't take full advantage of the indexes will take minutes if not hours to complete. Unfortunately that is currently the case with one of my queries:
SELECT
addr.`Linje-ID` as lineid,
addr.`Sluttbruker` as companyname,
addr.`Gate` as street,
addr.`Husnr` as housenr,
addr.`Postnr` as zip,
addr.`Poststed` as location,
loc.`UX_KOORDINAT` as coord_x,
loc.`UY_KOORDINAT` as coord_y,
loc.`ADRESSE_ID` as addr_id
FROM
addresses addr INNER JOIN
locationdata loc ON
loc.`POSTSTED` = addr.`Poststed` AND
loc.`POST_NR` = addr.`Postnr` AND
loc.`GATENAVN` = addr.`Gate` AND
loc.`HUSNUMMER` = addr.`Husnr`
GROUP BY
addr.`Linje-ID`
The locationdata table has a primary index id as well as an index defined as (POSTSTED, POST_NR, GATENAVN, HUSNUMMER). Fetching rows from the table using those columns in that order goes very quickly. The query above, however, had to be cancelled as it was taking too long (>15 minutes).
As my MySQL client (HeidiSQL) freezes while queries are performed, it's getting very tedious to force the application shut and start over for every attempt to fix this problem, so I'm asking for help here.
Just for testing, the table "addresses" only contains one row at the moment.
Can anyone identify why this query 'never' completes?
This is the EXPLAIN results I was asked for
http://pastebin.com/qWdQhdv5
You should copy the content and paste it into a larger container as it linebreaks.
EDIT: I've edited the query to reflect some of your replies. It still uses over 300 seconds where it shouldn't need 1.
First of all I'd remove the upper from loc.GATENAVN = UPPER(addr.Gate) since this clause is already searching in a case insensitive mode.
I went with subqueries instead of joins
Related
I'm trying to do what I think is a set of simple set operations on a database table: several intersections and one union. But I don't seem to be able to express that in a simple way.
I have a MySQL table called Moment, which has many millions of rows. (It happens to be a time-series table but that doesn't impact on my problem here; however, these data have a column 'source' and a column 'time', both indexed.) Queries to pull data out of this table are created dynamically (coming in from an API), and ultimately boil down to a small pile of temporary tables indicating which 'source's we care about, and maybe the 'time' ranges we care about.
Let's say we're looking for
(source in Temp1) AND (
((source in Temp2) AND (time > '2017-01-01')) OR
((source in Temp3) AND (time > '2016-11-15'))
)
Just for excitement, let's say Temp2 is empty --- that part of the API request was valid but happened to include 'no actual sources'.
If I then do
SELECT m.* from Moment as m,Temp1,Temp2,Temp3
WHERE (m.source = Temp1.source) AND (
((m.source = Temp2.source) AND (m.time > '2017-01-01')) OR
((m.source = Temp3.source) AND (m.time > '2016-11'15'))
)
... I get a heaping mound of nothing, because the empty Temp2 gives an empty Cartesian product before we get to the WHERE clause.
Okay, I can do
SELECT m.* from Moment as m
LEFT JOIN Temp1 on m.source=Temp1.source
LEFT JOIN Temp2 on m.source=Temp2.source
LEFT JOIN Temp3 on m.source=Temp3.source
WHERE (m.source = Temp1.source) AND (
((m.source = Temp2.source) AND (m.time > '2017-01-01')) OR
((m.source = Temp3.source) AND (m.time > '2016-11-15'))
)
... but this takes >70ms even on my relatively small development database.
If I manually eliminate the empty table,
SELECT m.* from Moment as m,Temp1,Temp3
WHERE (m.source = Temp1.source) AND (
((m.source = Temp3.source) AND (m.time > '2016-11-15'))
)
... it finishes in 10ms. That's the kind of time I'd expect.
I've also tried putting a single unmatchable row in the empty table and doing SELECT DISTINCT, and it splits the difference at ~40ms. Seems an odd solution though.
This really feels like I'm just conceptualizing the query wrong, that I'm asking the database to do more work than it needs to. What is the Right Way to ask the database this question?
Thanks!
--UPDATE--
I did some actual benchmarks on my actual database, and came up with some really unexpected results.
For the scenario above, all tables indexed on the columns being compared, with an empty table,
doing it with left joins took 3.5 minutes (!!!)
doing it without joins (just 'FROM...WHERE') and adding a null row to the empty table, took 3.5 seconds
even more striking, when there wasn't an empty table, but rather ~1000 rows in each of the temporary tables,
doing the whole thing in one query took 28 minutes (!!!!!), but,
doing each of the three AND clauses separately and then doing the final combination in the code took less than a second.
I still feel I'm expressing the query in some foolish way, since again, all I'm trying to do is one set union (OR) and a few set intersections. It really seems like the DB is making this gigantic Cartesian product when it seriously doesn't need to. All in all, as pointed out in the answer below, keeping some of the intelligence up in the code seems to be the better approach here.
There are various ways to tackle the problem. Needless to say it depends on
how many queries are sent to the database,
the amount of data you are processing in a time interval,
how the database backend is configured to manage it.
For your use case, a little more information would be helpful. The optimization of your query by using CASE/COUNT(*) or CASE/LIMIT combinations in queries to sort out empty tables would be one option. However, if-like queries cost more time.
You could split the SQL code to downgrade the scaling of the problem from 1*N^x to y*N^z, where z should be smaller than x.
You said that an API is involved, maybe you are able handle the temporary "no data" tables differently or even don't store them?
Another option would be to enable query caching:
https://dev.mysql.com/doc/refman/5.5/en/query-cache-configuration.html
A simplified version of my MySQL db looks like this:
Table books (ENGINE=MyISAM)
id <- KEY
publisher <- LONGTEXT
publisher_id <- INT <- This is a new field that is currently null for all records
Table publishers (ENGINE=MyISAM)
id <- KEY
name <- LONGTEXT
Currently books.publisher holds values that keep getting repeated, but that the publishers.name holds uniquely.
I want to get rid of books.publisher and instead populate the books.publisher_id field.
The straightforward SQL code that describes what I want done, is as follows:
UPDATE books
JOIN publishers ON books.publisher = publishers.name
SET books.publisher_id = publishers.id;
The problem is that I have a big number of records, and even though it works, it's taking forever.
Is there a faster solution than using something like this in advance?:
CREATE INDEX publisher ON books (publisher(20));
Your question title says ".. optimize ... query without using an index?"
What have you got against using an index?
You should always examine the execution plan if a query is running slowly. I would guess it's having to scan the publishers table for each row in order to find a match. It would make sense to have an index on publishers.name to speed the lookup of an id.
You can drop the index later but it wouldn't harm to leave it in, since you say the process will have to run for a while until other changes are made. I imagine the publishers table doesn't get update very frequently so performance of INSERT and UPDATE on the table should not be an issue.
There are a few problems here that might be helped by optimization.
First of all, a few thousand rows doesn't count as "big" ... that's "medium."
Second, in MySQL saying "I want to do this without indexes" is like saying "I want to drive my car to New York City, but my tires are flat and I don't want to pump them up. What's the best route to New York if I'm driving on my rims?"
Third, you're using a LONGTEXT item for your publisher. Is there some reason not to use a fully indexable datatype like VARCHAR(200)? If you do that your WHERE statement will run faster, index or none. Large scale library catalog systems limit the length of the publisher field, so your system can too.
Fourth, from one of your comments this looks like a routine data maintenance update, not a one time conversion. So you need to figure out how to avoid repeating the whole deal over and over. I am guessing here, but it looks like newly inserted rows in your books table have a publisher_id of zero, and your query updates that column to a valid value.
So here's what to do. First, put an index on tables.publisher_id.
Second, run this variant of your maintenance query:
UPDATE books
JOIN publishers ON books.publisher = publishers.name
SET books.publisher_id = publishers.id
WHERE books.publisher_id = 0
LIMIT 100;
This will limit your update to rows that haven't yet been updated. It will also update 100 rows at a time. In your weekly data-maintenance job, re-issue this query until MySQL announces that your query affected zero rows (look at mysqli::rows_affected or the equivalent in your php-to-mysql interface). That's a great way to monitor database update progress and keep your update operations from getting out of hand.
Your update query has invalid syntax but you can fix that later. The way to get it to run faster is to add a where clause so that you are only updating the necessary records.
This may be a little difficult to answer given that I'm still learning to write queries and I'm not able to view the database at the moment, but I'll give it a shot.
The database I'm trying to acquire information from contains a large table (TransactionLineItems) that essentially functions as a store transaction log. This table currently contains about 5 million rows and several columns describing products which are included in each transaction (TLI_ReceiptAlias, TLI_ScanCode, TLI_Quantity and TLI_UnitPrice). This table has a foreign key which is paired with a primary key in another table (Transactions), and this table contains transaction numbers (TRN_ReceiptNumber). When I join these two tables, the query returns one row for every item we've ever sold, and each row has a receipt number. 16 rows might have the same receipt number, meaning that all of these items were sold in a single transaction. Below that might be 12 more rows, each sharing another receipt number. All transactions are broken down into multiple rows like this.
I'm attempting to build a query which returns all rows sharing a single receipt number where at least one row with that receipt number meets certain criteria in another column. For example, three separate types of gift cards all have values in the TLI_ScanCode column that begin with "740000." I want the query to return rows with values beginning with these six digits in the TLI_ScanCode column, but I would also like to return all rows which share a receipt number with any of the rows which meet the given scan code criteria. Essentially, I need the query to return all rows for every receipt number which is also paired in at least one row with a gift card-related scan code.
I attempted to use a subquery to return a column of all receipt numbers paired with gift card scan codes, using "WHERE A.TRN_ReceiptAlias IN (subquery..." to return only those rows with a receipt number which matched one of the receipt numbers returned by the subquery. This appeared to run without issue for five minutes before the server ground to a halt for another twenty while it processed the query. The query appeared to complete successfully, but given that I was working with IT to restore normal store operations during this time I failed to obtain the results of the query (apart from the associated shame and embarrassment).
I'd like to know if there is a way to write a query to obtain this information without causing the server to hang. I'm assuming that either: a) it wasn't very smart to use a subquery in this manner on such a large table, or b) I don't know enough about SQL to obtain the information I need. I'm assuming the answer is both A and B, but I'd very much like to learn how to do this the right way. Any help would be greatly appreciated. Thanks!
SELECT *
FROM a as a1
JOIN b
ON b.id = a.id
JOIN a as a2
ON a2.id = b.id
WHERE b.some_criteria = 'something';
Include an index on (b.id,b.some_criteria)
You aren't the first person, nor will you be the last to bring down your system with an inefficient query.
The most important lesson is that "Decision Support" and "Analytics" really don't co-exist with a transaction system. You really want to pull the data into a datamart or datawarehouse or some other database that isn't your transaction database, so that you don't take the business offline.
In terms of understanding why your initial query was so inefficient, you want to familiarize yourself with the EXPLAIN EXTENDED syntax that returns you plan information that should help you debug your query and work on making it perform acceptably. If you update your question with the actual explain plan output for it, that would be helpful in determining what the issue is.
Just from the outline you provided, it does sound like a self join would make sense rather than the subquery.
At the moment, I have a table in mysql that records transactions. These transactions may be updated by users - sometimes never, sometimes often. However, I need to track changes to every field in this table. So, what I have at the moment is a TINYINT(1) field in the table called 'is_deleted', and when a transaction is 'updated' by the user, it simply updates the is_deleted field to 1 for the old record and inserts a brand new record.
This all works well, because I simply have to run the following sql statement to get all current records:
SELECT id, foo, bar, something FROM trans WHERE is_deleted = 0;
However, I am concerned that this table will grow unnecessarily large over time, and I am therefor thinking of actually deleting the old record and 'archiving' it off to another table (trans_deleted) instead. That means that the trans table will only contain 'live' records, therefor making SELECT queries that little bit faster.
It does mean, however, the updating records will take slightly longer, as it will be running 3 queries:
1. INSERT INTO trans_deleted old record;
2. DELETE from trans WHERE id = 5;
3. INSERT INTO trans new records
So, updating records will take a bit more work, but reading will be faster.
Any thoughts on this?
I would suggest a table trans and a table_revision
Where trans has the fields id and current_revision and revision has the fields id, transid, foo and bar.
To get all current items then:
SELECT r.foo, r.bar FROM trans AS t
LEFT JOIN revision AS r ON t.id = r.trans_id
WHERE t.current_revision = r.id
If you now put indexes on r.id and r.trans_id archiving woun't make it much faster for you.
Well typically, you read much more often than you write (and you additionally say that some records may be never changed). So that's one reason to go for the archive table.
There's also another one: You also have to account for programmer time, not only processor time :) If you keep the archived rows in the same table with the live ones, you'll have to remember and take care of that in every single query you perform on that table. Not to speak of future programmers who may have to deal with the table... I'd recommend the archiving approach based on this factor alone, even if there wasn't any speed improvement!
So I'm having serious speed problems using a left join to count ticket comments from another table. I've tried using a sub-select in the count field and had precisely the same performance.
With the count, the query takes about 1 second on maybe 30 tickets, and 5 seconds for 19000 tickets (I have both a production and a development server so times are skewed a tad). I'm trying to optimize this as four variations of the query need to be run each time per page refresh.
Without the count, I see execution time fall from 1 second to 0.03 seconds, so certainly this count is being run across all tickets and not just the ones which are selected.
Here's a trimmed down version of the query in question:
SELECT tickets.ticket_id,
ticket_severity,
ticket_short_description,
ticket_full_description,
count(*) as commentCount,
FROM tickets (LEFT JOIN tickets_comment on ticket_id = tickets_comment.ticket_id)
WHERE ticket_status='Open'
and ticket_owner_id='133783475'
GROUP BY
everything,
under,
the,
sun
Now, not all tickets have comments, so I can't just do a right or standard join. When doing that the speed is fairly good (1/10th the current), but any tickets without comments aren't included.
I see three fixes for this, and would like any and all advice you have.
Create a new column comment_count and use a trigger/update query on new comment
Work with the UI and grab comments on the fly (not really wanted)
Hope stackoverflow folks have a more elegant solution :รพ
Ideas?
A co-worker has come to the rescue. The query was just using join improperly.
What must be done here is create a second table with a query like:
select count(*) from tickets_comment group by ticket_id where (clause matches other)
which will create a table with counts for each ticket id. Then join that table with the ticket table where the ticket ids match. It's not as wicked fast as creating a new column, but it's at least 1/10th the speed it was, so I'm pleased as punch.
Last step is converting nulls (on tickets where there were no comments) into zeros
Is by far the fastest solution and you'll see it done in Rails all the time because it really is that fast.
count(*) is really only used when you aren't selecting any other attributes. Try count(ticket_id) and see if that helps. I can't run explain so I can't test it myself but if your analysis is correct it should help.
Try running explain on the query to make sure the correct indexes are being used. If there are no indexes being used, create another one