I've seen this several times but I could be misinterpreting the EXPLAIN query plan.
Suppose I have a table(col1, col2).
I want to join it with another table on both col1 and col2.
So I create an index(col1, col2).
Sometimes, the EXPLAIN shows that the index is not being used. Perhaps some other inefficient index is used or none at all.
But if I create another index(col1), then the first index(col1, col2) is used.
Has anyone ever had this happen to them before? Do you have any idea why this might happen?
My theory is that the unused index provides some more accurate statistics about the table that hints to the query plan to use the first index. But I'm not familiar enough with the inner workings of mysql to know if this is true or how to prove it.
The documentation of MySQL for ALTER TABLE states that it may be required to run ANALYZE TABLE on it to refresh the index cardinality, which I believe to be a factor in the behaviour you're seeing. Also, the query optimiser usually handles empty (or near) empty tables quite different from populated tables, and it'll often do a full table scan instead of using an index when there are only a few rows. For my own development at $work I can't rely on the EXPLAIN output of my dev database because of that.
Related
I am trying to speed up a simple SELECT query on a table that has around 2 million entries, in a MariaDB MySQL database. It took over 1.5s until I created an index for the columns that I need, and running it through PhpMyAdmin showed a significant boost in speed (now takes around 0.09s).
The problem is, when I run it through my PHP server (mysqli), the execution time does not change at all. I'm logging my execution time by running microtime() before and after the query, and it takes ~1.5s to run it, regardless of having the index or not (tried removing/readding it to see the difference).
Query example:
SELECT `pair`, `price`, `time` FROM `live_prices` FORCE INDEX
(pairPriceTime) WHERE `time` = '2022-08-07 03:01:59';
Index created:
ALTER TABLE `live_prices` ADD INDEX pairPriceTime (pair, price, time);
Any thoughts on this? Does PHP PDO ignore indexes? Do I need to restart the server in order for it to "acknowledge" that there is a new index? (Which is a problem since I'm using a shared hosting service...)
If that is really the query, then it needs an INDEX starting with the value tested in the WHERE:
INDEX(time)
Or, to make a "covering index":
INDEX(time, pair, price)
However, I suspect that most of your accesses involve pair? If so, then other queries may need
INDEX(pair, time)
especially if you as for a range of times.
To discuss various options further, please provide EXPLAIN SELECT ...
PDO, mysqli, phpmyadmin -- These all work the same way. (A possible exception deals with an implicit LIMIT on phpmyadmin.)
Try hard to avoid the use of FORCE INDEX -- what helps on today's query and dataset may hurt on tomorrow's.
When you see puzzling anomalies in timings, run the query twice. Caching may be the explanation.
The mysql documenation says
The FORCE INDEX hint acts like USE INDEX (index_list), with the addition that a table scan is assumed to be very expensive. In other words, a table scan is used only if there is no way to use one of the named indexes to find rows in the table.
MariaDB documentation Force Index here says this
FORCE INDEX works by only considering the given indexes (like with USE_INDEX) but in addition, it tells the optimizer to regard a table scan as something very expensive. However, if none of the 'forced' indexes can be used, then a table scan will be used anyway.
Use of the index is not mandatory. Since you have only specified one condition - the time, it can choose to use some other index for the fetch. I would suggest that you use another condition for the select in the where clause or add an order by
order by pair, price, time
I ended up creating another index (just for the time column) and it did the trick, running at ~0.002s now. Setting the LIMIT clause had no effect since I was always getting 423 rows (for 423 coin pairs).
Bottom line, I probably needed a more specific index, although the weird part is that the first index worked great on PMA but not through PHP, but the second one now applies to both approaches.
Thank you all for the kind replies :)
It looks like the MySQL EXPLAIN prefix only works in front of certain queries. Is there an equivalent of EXPLAIN that will work in front of an ALTER TABLE query?
I would love to be able to find out how long my planned ALTER TABLE statement is likely to take.
Background: I have a table from someone else that contains 300 columns of data. I know that I'm only going to need to use a few of those columns, and in order to figure out which columns I need, I'm planning to do a full-text search for a few key words. But in order to do that, I need to add a full-text index. And since I'm new to this size of data set, I'm not entirely sure that this is a realistic plan. I'm hoping something like EXPLAIN (or, more likely, a substitute tool from this thread) might help determine that.
EDIT: In answer to a couple questions below, I should mention that this table has about 4 million rows and is on a local testing machine. So I can just run this thing blindly if needed. I just don't prefer to if possible. Thanks for all the good information so far.
Most "Alter table" will trigger the copy to tmp table operation, which it will create temp table with new schema, then lock table, copy data from old table to new table, then rename, drop old table.
So most time consumed is copy to temp table, it's depend on how big of that table if the server have enough memory. Use show table status to check how big of the table (data_length+ index_length), sample on other table to know the transfer speed on your mysql server, then you can estimate how long it will take.
Another way mentioned on mysql doc about explain on DML, but I didn't got result, maybe not finished yet :
http://dev.mysql.com/doc/refman/5.6/en/explain.html
As of MySQL 5.6.3, permitted explainable statements for EXPLAIN are SELECT, DELETE, INSERT, REPLACE, and UPDATE. Before MySQL 5.6.3, SELECT is the only explainable statement.
I have a query that is running very slowly. The table is was querying has about 100k records and no indexes on most of the columns used in the where clause. I just added indexes on those columns but the query hasn't gotten any faster.
I think this is because when a column is indexed, it's value is written in the index at the time of insertion. I just added the indexes now after all those records were added. So is there a way to "re-run the indexes" on the table?
Edit
Here is the query and explain result:
Oddly enough when I copy the query and run in directly in my SQL manager tool it runs quite fast so may bye the problem is in my application code and not in the query itself.
Mysql keeps consistent indexes. It does not matter if the data is added first, the index is added first, or the data is changed at any time. The same final index will result (assuming the same final data and index type).
Your slow query is not caused by adding the index later. There will be some other reason.
This is an extremely common problem.
Use MySQL explain http://dev.mysql.com/doc/refman/5.0/en/using-explain.html
When you precede a SELECT statement with the keyword EXPLAIN, MySQL displays information from the optimizer about the query execution plan. That is, MySQL explains how it would process the statement, including information about how tables are joined and in which order.
Using these results... verify the index you created is functioning the way you expected.
If not, you will want to tweak your index until you have it working as expected.
You might want to create a new table, create indexes, then insert all elements from old table to new while testing this. It's easier than dropping and re-adding indices a million times.
If i use
CREATE INDEX status_index ON eligible_users (status)
or
CREATE INDEX status ON eligible_users (status)
its the same thing no difference?
Also if i create alot of indexes will it actually help with queries or slow down?
Both statements you wrote do the same exact thing, only difference is the name of the index.
As for usefulness, it depends on the database setup and usage. Indexes are useful to speed up queries, but they have to be maintained on every INSERT/UPDATE, so it depends. There are a lot of resources available online about this wide topic.
An index can make or break a query. The execution time for certain queries can go from minutes to fractions of a second just by adding the correct indexes. In case you need to improve a query you can always prepend EXPLAIN to it, to see what MySQL's execution plan is: it will show what indexes the query uses (if any) and will help you troubleshoot some bottlenecks.
As said, an index is useful but is not free. It has to be kept up to date, so every time you insert or modify data in an indexed field, then the index must be updated too.
Generally in cases where you have a lot of reads and (relatively) few writes, indexes help a lot. But unnecessary indexes can degrade performance instead of improving it.
The short syntax for creating a single column index on column col from table tbl is:
CREATE INDEX [index_name] ON tbl (col)
Full details available in the MySQL Manual.
i was wondering, if i add one index for each field in every table of my DB, will that make my queries run faster?
or do i have to analyze my queries and create indexes only when required?
Adding an index on each column will probably make most of your queries faster, but it's not necessarily the best approach. It is better to tune your indexes to your specific queries, using EXPLAIN and performance measurements to guide you in adding the correct indexes.
In particular you need to understand when you shouldn't index a column, and when you need multi-column indexes.
I would advise reading the MySQL manual for optimization of SELECT statements which explains under what conditions indexes can be used.
The more indexes you have, the heavier inserting/updating gets. So it's a tradeoff. The select queries that cannot use an index now will get quicker ofcourse, but if you check what fields you're joining on (or using in a where) you will not trade off that much
(and, ofcourse, there is the disk-space, but most of the time I don't really care bout that: ) )
Another point is that MySql can only use a single index for a query, so if your query is
SELECT * FROM table WHERE status = 1 AND col1='foob' AND col2 = 'bar'
MySql will use 1 of the indexes, and filter out the rest reading the data from the table.
If you have queries like this, its better to create a composite index on (status, col1, col2)
Adding index on every field in every table is not smart.
You should add indexes ONLY on columns that you use in the WHERE clause in select OR on which you sort.
Often, the best results are achieved by using multi-column indexes that are specific to your SQL selects.
There are also a partial indexes with limit on the length of field which can also be used to optimize performance and reduce the index site.
Every unnecessary index will slow down the database during the insert because on every insert, every index has to be updated.
Also the more indexes you have, the more chances you have of data corruption. And lastly, indexes take extra storage space on disk, sometimes a lot of space.
Also MySQL tries to keep indexes in memory. If you have unnecessary indexes, there is a good change MySQL will end up using up the available memory with unnecessary indexes in which case your performance will degrade considerable.
Creating the right kind of indexes is probably the single most important optimization technique. That's why when someone asks something like this I thought it was a joke.
This question can only be asked by someone who have not read a single book on MySQL. Just get a good book and read it, then you will not have to ask questions like this.