Is there any way to force MySQL to push a predicate into a view?
Example:
CREATE TABLE t1(
id INT(11) NOT NULL AUTO_INCREMENT
PRIMARY KEY (id)
);
CREATE VIEW v1
AS SELECT * FROM t1;
The query below will not use PRIMARY KEY index in MySQL:
SELECT *
FROM v1
WHERE id = 1
Instead it will select everything from t1, create a derived table and then filter it out for id = 1.
Is there any way to overcome this?
PS: my real life example is a little bit more complex than the one above, but for simplicity I used the example above
PPS: here's a related Stack Overflow question: How do I get MySQL to use an INDEX for view query?
Yes but you'd have to switch to the fully-compatible MariaDB 10.2.2
This is on by default and can be switched off using optimizer_switch:
SET GLOBAL optimizer_switch='condition_pushdown_for_derived=off'
Related
I have a MySQL 5.7 table with a binary(16) id column used as PRIMARY key.
The engine is InnoDb.
Rows are created with Doctrine using https://github.com/ramsey/uuid-doctrine#innodb-optimised-binary-uuids as per https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/
For instance the uuid 55a54172-f5e4-11e8-aa0d-fe02fd3f406 is stored as HEX(id) = 11E8F5E455A54172AA0DFE02FD3F406 in the database
I'm trying to search for rows with the same string beginning.
Something like this with pseudo code
SELECT *
FROM log
WHERE id LIKE 11E8F5E4
I've tried to query
SELECT HEX(`id`) AS `id`
FROM `log`
WHERE (`id` & UNHEX('11E8F5E4')) = UNHEX('11E8F5E4')
LIMIT 10
which is fast but rows with a different prefix like 11E8E8AD47E47970B1C9525400B61105 are returned
How may I do it while using the Btree index (for perf reason)?
I tried the straightforward approach with LIKE, and it seems to be working fine.
You need to make a binary string which has a % at the end:
SELECT ...
FROM table
WHERE field LIKE CONCAT(UNHEX('11E8F5E4'),'%')
I am trying to create an index in MySQL whereas the query will first check what column is not null. After checking, it will create the index on the column that is not null. However, I am not successful in creating this and it says I have an error, can someone help me? please see my code below
create index IDX_KSE_NO_01 on tb_kse(ifnull(ss_no, id_no);
#lad2025 is correct that MySQL does not support function-based indexes (like PostgreSQL does), but MySQL 5.7 introduced a feature for virtual generated columns based on expressions, and then you can create an index on a virtual column.
ALTER TABLE tb_kse ADD COLUMN either_no VARCHAR(10) AS (IFNULL(ss_no, id_no));
CREATE INDEX IDX_KSE_NO_01 ON tb_kse(either_no);
MySQL does not support function-based index. You should create normal index:
create index IDX_KSE_NO_01 on tb_kse(ss_no);
create index IDX_KSE_NO_02 on tb_kse(id_no);
And rewrite your query (OR-Expansion):
SELECT *
FROM tb_kse WHERE ss_no = ?
UNION
SELECT *
FROM tb_kse
WHERE ss_no IS NULL AND id_no = ?;
DBFiddle Demo
Another way is to create generated column and create index on top of it:
CREATE TABLE tb_kse(id_no INT, ss_no INT,
gen_col INT GENERATED ALWAYS AS (ifnull(ss_no, id_no)) STORED);
create index IDX_KSE_NO_01 on tb_kse(gen_col);
SELECT *
FROM tb_kse
WHERE gen_col = ?;
DBFiddle Demo 2
I have been playing around with indexes on MySQL (5.5.24, WinXP), but I can't find the reason of why the server is not using one index when a LIKE is used.
The example is this:
I have created a test table:
create table testTable (
id varchar(50) primary key,
text1 varchar(50) not null,
startDate varchar(50) not null
) ENGINE = innodb;
Then, I added an index to startDate. (Please, do not ask why the column is a text and not a date time.. this is just a simple test):
create index jeje on testTable(startdate);
analyze table testTable;
After that, I added almost 200,000 rows of that where startDate had 3 possible values. (One third of appearences for each one..near 70,000 times)
So, if I run an EXPLAIN command like this:
explain select * from testTable use index (jeje) where startDate = 'aaaaaaaaa';
The answer is the following:
id = 1
select_type = SIMPLE
type = ref
possible_keys = jeje
key = jeje
rows = 88412
extra = Using where
So, the key is used, and the rows amount is near to 200,000/3 so all is ok.
The poblem is that if I change the query to: (just chaning '=' to 'LIKE'):
explain select * from testTable use index(jeje) where startDate LIKE 'aaaaaaaaa';
In this case, the answer is:
id = 1
select_type = SIMPLE
type = ALL
possible_keys = jeje
key = null
rows = 176824
extra = Using where
So, the index is not being used now(key is null, and rows near to the full table..as the type=all suggests).
MySQL documentation says that LIKE DOES make use of indexes.
So, what am i not seeing here? Where is the problem?
Thanks for your help.
MySql can ignore index if it index incurs access to more than 30% of table rows.
You could try FORCE INDEX [index_name], it will use index in any case.
The value of sysvar_max_seeks_for_key also affects whether the index is used or not:
http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_max_seeks_for_key
Try changing this value to a smaller number.
Search for similar requests on SO.
Based on Ubik comment, and data changes, I found that:
The Index IS used in these cases:
- explain select * from testTable force index jeje where startDate like 'aaaaaaadsfadsfadsfasafsafsasfsadsfa%';
- explain select * from testTable force index jeje where startDate like 'aaaaaaadsfadsfadsfasafsafsasfsadsfa%';
- explain select * from testTable force index jeje where startDate like 'aaa';
But the index is NOT being used when I use this query:
- explain select * from testTable force index jeje where startDate like 'aaaaaaaaa';
Based on the fact that in startDate column all the values have the same length (9 characters), when I use a query using a LIKE command and a 9 characters constant, PERHAPS MySQL prefer to not use the reason because of some performance algorithm, and goes to the table.
My concern was to see if I was making some kind of mistake on my original tests, but now I think that the index and tests are correct, and that MySQL in some cases decides to not use the index... and I will relay on this.
For me, this is a closed task.
If somebody want to add something to the thread, you are welcome.
I've done a lot of reading and Googling on this and I cannot find any satisfactory answer so I'd appreciate any help. Most answers I find come close to my situation but do not address it (and attempting to follow the solutions has not done me any good).
See Edit #2 below for the best example
[This was the original question but is not a great representation of what I'm asking.]
Say I have 2 tables, each with 4 columns:
key (int, auto increment)
c1 (a date)
c2 (a varchar of length 3)
c3 (also a varchar of length 3)
And I want to perform the following query:
SELECT t.c1, t.c2, COUNT(*)
FROM test1 t
LEFT JOIN test2 t2 ON t2.key = t.key
GROUP BY t.c1, t.c2
Both key fields are indexed as primary keys. I want to get the number of rows returned in each grouping of c1, c2.
When I explain this query I get "using temporary; using filesort". The actual table I'm performing this query on is over 500,000 rows, so that means it's a time consuming query.
So my question is (assuming I'm not doing anything wrong in the query): is there a way to index this table to eliminate the temporary/filesort usage?
Thanks in advance for any help.
Edit
Here is the table definition (in this example both tables are identical - in reality they're not but I'm not sure it makes a difference at this point):
CREATE TABLE `test1` (
`key` int(11) NOT NULL auto_increment,
`c1` date NOT NULL,
`c2` varchar(3) NOT NULL,
`c3` varchar(3) NOT NULL,
PRIMARY KEY (`key`),
UNIQUE KEY `c1` (`c1`,`c2`),
UNIQUE KEY `c2_2` (`c2`,`c1`),
KEY `c2` (`c2`,`c3`)
) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=utf8
Full EXPLAIN statement:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE t ALL NULL NULL NULL NULL 2 Using temporary; Using filesort
1 SIMPLE t2 eq_ref PRIMARY PRIMARY 4 tracking.t.key 1 Using index
This is just for my sample tables. In my real tables the rows for t says 500,000+ (every row in the table, though that could be related to something else).
Edit #2
Here is a more concrete example to better explain my situation.
Let's say I have data on Little League baseball games. I have two tables. One holds data on the games:
CREATE TABLE `ex_games` (
`game_id` int(11) NOT NULL auto_increment,
`home_team` int(11) NOT NULL,
`date` date NOT NULL,
PRIMARY KEY (`game_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8
The other holds data on the at bats in each game:
CREATE TABLE `ex_atbats` (
`ab_id` int(11) NOT NULL auto_increment,
`game` int(11) NOT NULL,
`team` int(11) NOT NULL,
`player` int(11) NOT NULL,
`result` tinyint(1) NOT NULL,
PRIMARY KEY (`hit_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8
So I have two questions. Let's start with the simple version: I want to return a list of games with a count of how many at bats are in each game. So I think I would do something like this:
SELECT date, home_team, COUNT(h.ab_id) FROM `ex_atbats` h
LEFT JOIN ex_games g ON g.game_id = h.game
GROUP BY g.game_id
This query uses filesort/temporary. Is there a better way to structure this or to index the tables to get rid of that?
Then, the trickier part: say I now want to not only include a count of the number of at bats, but also include a count of the number of at bats that were preceded by an at bat with the same result by the same team. I assume that would be something like:
SELECT g.date, g.home_team, COUNT(ab.ab_id), COUNT(ab2.ab_id) FROM `ex_atbats` ab
LEFT JOIN ex_games g ON g.game_id = ab.game
LEFT JOIN ex_atbats ab2 ON ab2.ab_id = ab.ab_id - 1 AND ab2.result = ab.result
GROUP BY g.game_id
Is that the correct way to structure that query? This also uses filesort/temporary.
So what is the optimal way to go about accomplishing these tasks?
Thanks again.
Phrases Using temporary/filesort usually are not related to the indexes used in the JOIN operation. There is numerous examples where you can have all indexes set (they show up in key and key_len columns in EXPLAIN) but you still get Using temporary and Using filesort.
Check out what the manual says about Using temporary and Using filesort:
How MySQL Uses Internal Temporary Tables
ORDER BY Optimization
Having a combined index for all columns used in GROUP BY clause may help to get rid of Using filesort in certain circumstances. If you also issue ORDER BY you may need to add more complex indexes.
If you have a huge dataset consider partitioning it using some criteria like date or timestamp by means of actual partitioning or a simple WHERE clause.
First of all, the tables' definitions do matter. It's one thing to join using two primary keys, another to join using a primary key from one side and a non-unique key in the other, etc. It also matters what type of engine the tables use as InnoDB treats Primary Keys differently than MyISAM engine.
What I notice though is that on table test1, the (c1,c2) combination is Unique and the fields are not nullable. This allows your query to be rewritten as:
SELECT t.c1, t.c2, COUNT(*)
FROM test1 t
LEFT JOIN test2 t2 ON t2.key = t.key
GROUP BY t.key
It will give the same results while using the same field for the JOIN and the GROUP BY. Note that MySQL allows you to use in the SELECT list fields that are not in the GROUP BY list, without having aggregate functions on them. This is not allowed in most other systems and is seen as a bug by some. In this situation though it is a very nice feature. Every row can be either identified by (key) or (c1,c2), so it shouldn't matter which of the two is used for the grouping.
Another thing to note is that when you use LEFT JOIN, it's common to use the joining column from the right side for the counting: COUNT(t2.key) and not COUNT(*). Your original query will give 1 in that column for records in test1 that do not mmatch any record in test2 because it counts rows while you probably want to count the related records in test2 - and show 0 in those cases.
So, try this query and post the EXPLAIN:
SELECT t.c1, t.c2, COUNT(t2.key)
FROM test1 t
LEFT JOIN test2 t2 ON t2.key = t.key
GROUP BY t.key
The indexes help with the join, but you still need to do a full sort in order to do the group by. Essentially, it still has to process every record in the set.
Adding a where clause and limiting the set would run faster, of course. It just won't get you the results you want.
There may be other options than doing a group by on the entire table. I notice you're doing a SELECT * - What are you trying to get out of the query?
SELECT DISTINCT c1, c2
FROM test t
LEFT JOIN test2 t2 ON t2.key = t.key
may run faster, for instance. (I realize this was just a sample query, but understand that it's hard to optimize when you don't know what the end goal is!)
EDIT - In doing some reading (http://dev.mysql.com/doc/refman/5.0/en/group-by-optimization.html), I learned that, under the correct circumstances, indexes can help significantly with the group by.
What I'm seeing is that it needs to be a sorted index (like BTREE), not a HASH. Perhaps:
CREATE INDEX c1c2 IN t (c1, c2) USING BTREE;
might help.
For innodb it will work, as the index caries your primary key by default. For myisam you have to have the key as the last column of your index be "key". That will give the optimizers all keys in the same order and he can skip the sort. You cannot do any range queryies on the index prefix theN, puts you right back into filesort. currently struggling with a similiar problem
If I have the following table:
CREATE TABLE `mytable` (
`id` INT NOT NULL AUTO_INCREMENT,
`name` VARCHAR(64) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `name_first_letter` (`name`(1)),
KEY `name_all` (`name`)
)
Will MySQL ever choose to use the name_first_letter index over the name_all index? If so, under what conditions would this happen?
I have done some quick tests and I'm not sure if MySQL will choose the name_first_letter index even when using index hints:
-- This uses name_all
EXPLAIN SELECT name FROM mytable
WHERE SUBSTRING(name FROM 1 FOR 1) = 'T';
-- This uses no index at all
EXPLAIN SELECT name FROM mytable USE INDEX (name_first_letter)
WHERE SUBSTRING(name FROM 1 FOR 1) = 'T';
Can any MySQL gurus shed some light on this? Is there even a point to having name_first_letter on this column?
Edit: Question title wasn't quite right.
It will not make sense to use the index for your query, because you are selecting the full name column. That means that MySQL cannot use the index alone to satisfy the query.
Further, I believe that MySQL cannot understand that the SUBSTRING(name FROM 1 FOR 1) expression is equivalent to the index.
MySQL might, however, use the index if the index alone can satisfy the query. For example:
select count(*)
from mytable
where name like 'T%';
But even that depends on you statistics (hinting should work).
MySQLs partial index feature is intended to save space. It does (usually) not make sense to have both, the partial and the full column index. You would typically drop the shorter one. There might be a rare case where it makes sense, but doesn't make sense in general.