I apologize in advance if I'm not doing this right, I'll try to follow all the rules but this is my first post on here.
The problem I'm having seems very simple.
I have a Microsoft Access 2007 database with a table that has approximately 300,000 records. I am attempting to query one of the fields in the table with a "Like" condition...
SELECT * FROM Table1 WHERE Field1 LIKE "*_TEST_*"
However, I'm finding this query has significant performance issues, taking 4-5 minutes to run compared to a few seconds if I take that particular "Like" criteria off. I've placed an index on [Field1] to try and alleviate the issue (and maybe I'm just a bit ignorant to the finer details of indexing) but it doesn't seem to have helped at all.
A colleague of mine informed me that these "Like" or "Contains" conditions would not see an improvement because of indexing, but did not really have any other ideas.
Could anyone please explain what it is about this "Like" criteria and indexing that don't play well together? And is there anything I can do to make this seemingly simple query work better?
Thanks so much!
Related
I see this error in phpmyadmin:
The number of joins that do not use indexes
but I never use join in my code.
Now I want get a list of query that do not use indexes.
How can I get this list?
I tried enabling slow query log, but I can not understand which query is not use indexes.
Can someone guide me?
There is no list of "joins not using indexes".
Certain admin queries use VIEWs that have JOINs; possibly that is where they came from.
There are valid cases in which not using an index is OK for a JOIN. A simple example is joining two tables where one of the tables has so few rows that an index would not help with performance.
The slowlog provides something more important -- a list of the "worst" queries (from a performance point of view). Any "slow" query with JOIN that needs an index will show up, even without that setting turned on. (I prefer to turn off that option, since it clutters the slowlog with unexciting queries.)
I'll briefly mention that this isn't an error, it's not even really a warning. The Advisor tab is meant to make broad and generic performance suggestions that are meant to guide you towards optimizing your system. Having some of these suggestions that aren't fixable or don't apply to your situation is pretty normal. In fact, my test system gives the same advice about the join without an index.
As Rick James alluded to, these queries might not come directly from code that you write...some administrative tasks may be triggering it (your operating system might run some housekeeping queries, MySQL runs some queries against itself, etc).
The best way to look at the queries is to log them, which is answered in Log all queries in mysql. You could use the "Process" sub tab of the "Status" area in phpMyAdmin (very similar to how you get to the Advisor tab) to look at active queries, but that's not going to give you a complete picture over time. Once you have that list of queries, you can analyze them to determine if there are improvements you can make to the code.
I apologize if this question is too general, I can post and example code, but it may not be reproducible because there is no access to the actual database.
Suppose I have a big MySQL query with a lot of joins and unions, as well as functions like concat, date, various time and date conversion functions. And it uses a lot of tables, nested select queries etc. Lets suppose it is a select query.
My questions is, where would one start, if they need to optimize this script to run faster? Currently its taking hours to complete. Furthermore, If I run stand-alone chunks of code from it (including some nested queries, etc.) they run much faster. Therefore there are one or a few bottlenecks. Perhaps certain tables are not indexed properly.
I am aware of profiling and bench-marking as well as explain functionality in MySQL they all help us to understand what MySQL does behind the scenes, but all provide the summary for the entire script overall. What would be the best way to identify these bottlenecks without profiling each each portion of the script separately? Are there best practices when faced with such problem?
Again, I apologize for asking a question that may be too broad. I can post and example code, but it may not be reproducible because there is no access to the actual database.
After using EXPLAIN and making sure I make use of proper indexing, I would run it on a subset of your data so I can get it in seconds (easier when tweaking the query).
I would run each subquery individually first and take a note how long they perform. Then run the query that calls that subquery/derived and see how long it performs. Comment out some subqueries and see how it performs. Soon you will get the picture which parts are your bottleneck.
Then I would start experimenting with different techniques. Perhaps using a temporary table first, or maybe I need to run daily cron jobs that summarizes the data for me.
Therefore there are one or a few bottlenecks. Perhaps certain tables are not indexed properly.
This sounds like something you could solve using EXPLAIN?
I don't use MySQL but this is sort of a software agnostic problem. Assuming that you are already doing some of the "right" things such as filtering on indexed fields, etc, there are two steps that might help.
First - move the code to a Stored Procedure. The benefit of this is that the code needs to be compile only once. If your current query is not run frequently, it has to be compiled every time it runs, which takes time.
Second - use temporary tables. While it's not intuitive, I have found that this often improves execution time dramatically.
There is this rather simple query that I have to run on a livesystem, in order to get a count. The problem is that the table and database are rather inefficiently designed and since it is a livesystem altering it is not an option at this point.
So I have to figure out a query that runs fast and won't slow down the system too much, because for the time of the query execution the system basically stops which is not really what I would like a livesystem to do, so I need to streamline my query in order to make it perform in an acceptable time.
SELECT id1, count(id2) AS count FROM table GROUP BY id1 ORDER BY count
DESC;
So here is the query, unfortunately it is so simple that I am out of ideas on how to further improve it, maybe someone else has an idea ... ?
Application Get "good enough" results via application changes:
If you have access to the application, but not the database, then there are possibilities:
Periodically run that slow query and capture the results. Then use the cached results.
Do you need all
What is the goal? Find a few of the most common id1's? Rank all of them?
Back to the query
COUNT(id2) checks for id2 being not null; this us usually unnecessary, so COUNT(*) is better. However the speedup is insignificant.
ORDER BY NULL is irrelevant if you are picking off the rows with the highest COUNT -- the sort needs to be done somewhere. Moving it to the application does not help; at least not much.
Adding LIMIT 10 would only help because of cutting down on the time to send the data back to the client.
INDEX(id1) is the best index for the query (after changing to COUNT(*)). But the operation still requires
full index scan to do the COUNT and GROUP BY
sort the grouped results -- for the ORDER BY
Zero or near-zero downtime
Do you have replication established? Galera Clustering?
Look into pt-online-schema-change and gh-ost.
What is the real goal?
We cannot fix the query as written. What things can we change? Better yet, what is the ultimate goal -- perhaps there is an approach that does not involve any query that looks the least like the one you are trying to speed up.
Now I have just dumped the table and imported it into a MySQL-Docker, ran the query there, took ages and I actually had to move my entire Docker because the dump was so huge, but in the end I got my results and now I know how many id2s are associated with specific id1s (apostrophe to form a plural? You may want to double-check that ;) ).
As it was already pointed out, there wasn't much room for improvement on the query anymore.
FYI suddenly the care about stopping the system was gone and now we are indexing the table, so far it took 6 hours, no end in sight :D
Anyways, thanks for the help everyone.
I have a web service which returns results to a jquery auto-complete.
For the query I must use four full blown like clauses
LIKE('%SOME_TERM%')
The reason being that the users need to be able to return results from sub strings as well as proper words. I have also tried full-text indexes with their many options in this case in both Natural and Boolean mode but they just does not work as well and their results leave a lot to be desired in this case.
The database is highly optimized with indexes and even with the LIKE clauses returning results from a query with multiple joins and with one of the tables having 200,000 rows takes ~ 0.2/0.3 seconds on first run. Once its cached by the server then obviously the time taken is miniscule.
I was wondering if there is anything else that would be worth trying here. I had looked at some standalone search providers but I'm a little tight time-wise on this project(nearly done and ready to launch) so can't afford any large setups or large-scale refactoring time and funding wise.
Its possible that it's as good as it gets but no harm in letting SO have its say is my attitude.
I think apache solr is they way to go for you. For full text searches.
http://lucene.apache.org/solr/
But like you said, if you don't have much time left, and you are sure your queries are performing at their best. I don't see much you can do.
Other than running it to completion...
Is there some sort of progress you can monitor to see what operations are happening as your query is being processed that would give you a sense of how long it's going to take, or at least what step it is on, what steps have happened, and which remain?
If yes, would this same tool help you identify the part of your query that is taking the longest?
I'm trying to get a better sense for what makes some queries take longer than others.
It's called profiling : http://dev.mysql.com/tech-resources/articles/using-new-query-profiler.html
MySQL has one built-in for you. : )
You can use explain to ask MySQL to show you why a function takes as longs as it does.
http://dev.mysql.com/doc/refman/5.0/en/explain.html
The idea is that it'll show you things like which indexes it uses, etc, which will help you to then optimise either the query or the table to make it quicker.
That's not a direct answer to your question, because it won't tell you real-time progress of the query, which is what you're asking for, but it does directly answer the last sentence of your question, and so it may actually be more useful to you for what you really want to know than any real-time progress report would be.