Consider 2 queries:
SELECT * FROM T1 WHERE NAME LIKE '%\%%';
SELECT * FROM T1 WHERE NAME LIKE '%\%\%%';
Assume that T1 has records where NAME is %, %%, or %%%.
I would expect the second query to return fewer results but it is including the record where T1.NAME = '%'! Is there a way to filter out that record using like query? Something like SELECT * FROM T1 WHERE NAME LIKE '%\%\%%' AND NAME <> '%'; is not what I am looking for.
mysql> explain table1;
+-------+------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+------------+------+-----+---------+-------+
| id | int(11) | YES | | NULL | |
| text | varchar(4) | YES | | NULL | |
+-------+------------+------+-----+---------+-------+
2 rows in set (0.00 sec)
mysql> explain select * from table1
-> where text like '%\%%';
+----+-------------+--------+------+---------------+------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------+------+---------------+------+---------+------+------+-------------+
| 1 | SIMPLE | table1 | ALL | NULL | NULL | NULL | NULL | 4 | Using where |
+----+-------------+--------+------+---------------+------+---------+------+------+-------------+
1 row in set (0.00 sec)
mysql> explain select * from table1
-> where text like '%\%\%%';
+----+-------------+--------+------+---------------+------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------+------+---------------+------+---------+------+------+-------------+
| 1 | SIMPLE | table1 | ALL | NULL | NULL | NULL | NULL | 4 | Using where |
+----+-------------+--------+------+---------------+------+---------+------+------+-------------+
1 row in set (0.00 sec)
Here is my config if it helps:
[mysqld]
innodb_buffer_pool_size=402653184
innodb_log_file_size=262144000
innodb_log_buffer_size=8388608
max_allowed_packet=5241856
innodb_additional_mem_pool_size=20971520
I need to investigate further but it appears that the problem has something to do with the way that the database is being created:
create database testdb character set utf8 collate utf8_unicode_ci;
I was able to get the correct results when I create the database in a less specific way:
create database testdb;
Any idea why?
As far as I can tell, this appears to be a bug in mysql... it is sometimes impossible to search for consecutive percent signs using LIKE. (I was experiencing the same issue you described)
Still not sure what conditions are required to trigger the bug.
One workaround, however, would be to use REGEXP:
SELECT * FROM T1 WHERE NAME REGEXP "%%";
Related
how can have my query be able to not differienciate between my lower and upper case column query but without using the lower method of sql?
select * from
User
where
lower(lusername)= 'abdel'
i want to still be able to use my column name with any type of case (Abc, aBc, ABC,aBC..) but without using the lower function in my query sql because the lower disable my index that i put on my column name.
is there a way to do that ? thank you in advance
The default collation used in MySQL is case-insensitive. Therefore you don't need to use LOWER() if you are doing searching.
In other words, the following two queries should return the same results:
SELECT ... FROM mytable WHERE LOWER(name) = 'Abc';
SELECT ... FROM mytable WHERE name = 'Abc'
The latter query is able to optimize the query with an index, but the former can't.
Any expression or function call makes the search not "sargable" which means it can't use an index, because the optimizer can't know if that expression or function outputs have the same sort order as the index on the column.
Based on your comment below, I see you are using a collation utf8_bin, which is case-sensitive. This does improve performance a little bit over using a collation, so if performance is the only thing that's important in this case, I understand why you want to keep using that collation.
Here's a demo of using EXPLAIN to verify that it uses an index when you compare strings directly, but the binary collation does a case-sensitive search:
mysql> create table User (id serial primary key, lusername varchar(75) collate utf8_bin, created datetime, key(lusername));
mysql> explain select * from User where lusername = 'Abc';
+----+-------------+-------+------------+------+---------------+-----------+---------+-------+------+----------+-------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+---------------+-----------+---------+-------+------+----------+-------+
| 1 | SIMPLE | User | NULL | ref | lusername | lusername | 228 | const | 1 | 100.00 | NULL |
+----+-------------+-------+------------+------+---------------+-----------+---------+-------+------+----------+-------+
But when we use a function like LOWER(), it can't use the index, and resorts to a table-scan:
mysql> explain select * from User where LOWER(lusername) = 'Abc';
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-------------+
| 1 | SIMPLE | User | NULL | ALL | NULL | NULL | NULL | NULL | 1 | 100.00 | Using where |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-------------+
However, MySQL 5.7 has a feature to add a virtual column based on the expression, and index the virtual column:
mysql> alter table User add lower_lusername varchar(75) as (LOWER(lusername)), add key (lower_lusername);
mysql> explain select * from User where LOWER(lusername) = 'Abc';
+----+-------------+-------+------------+------+-----------------+-----------------+---------+-------+------+----------+-------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+-----------------+-----------------+---------+-------+------+----------+-------+
| 1 | SIMPLE | User | NULL | ref | lower_lusername | lower_lusername | 303 | const | 1 | 100.00 | NULL |
+----+-------------+-------+------------+------+-----------------+-----------------+---------+-------+------+----------+-------+
Alternatively, in MySQL 8.0, you can now get the index on the expression it without even adding a generated column:
mysql> alter table User add key ((LOWER(lusername)));
mysql> explain select * from User where LOWER(lusername) = 'Abc';
+----+-------------+-------+------------+------+------------------+------------------+---------+-------+------+----------+-------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+------------------+------------------+---------+-------+------+----------+-------+
| 1 | SIMPLE | User | NULL | ref | functional_index | functional_index | 228 | const | 1 | 100.00 | NULL |
+----+-------------+-------+------------+------+------------------+------------------+---------+-------+------+----------+-------+
Background
I made a small table of 10 rows from a previous SELECT already ran (SavedAnimals).
I have a massive table (animals) which I would like to UPDATE using the rows with the same id as each row in my new table.
What I have tried so far
I can quickly SELECT the desired rows from the big table like this:
mysql> EXPLAIN SELECT * FROM animals WHERE ignored=0 and id IN (SELECT animal_id FROM SavedAnimals);
+------+--------------+-------------------------------+--------+---------------+---------+---------+----------------------------------------------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+--------------+-------------------------------+--------+---------------+---------+---------+----------------------------------------------------------+------+-------------+
| 1 | PRIMARY | <subquery2> | ALL | distinct_key | NULL | NULL | NULL | 10 | |
| 1 | PRIMARY | animals | eq_ref | PRIMARY | PRIMARY | 8 | db_staging.SavedAnimals.animal_id | 1 | Using where |
| 2 | MATERIALIZED | SavedAnimals | ALL | NULL | NULL | NULL | NULL | 10 | |
+------+--------------+-------------------------------+--------+---------------+---------+---------+----------------------------------------------------------+------+-------------+
But the "same" command on the UPDATE is not quick:
mysql> EXPLAIN UPDATE animals SET ignored=1, ignored_when=CURRENT_TIMESTAMP WHERE ignored=0 and id IN (SELECT animal_id FROM SavedAnimals);
+------+--------------------+-------------------------------+-------+---------------+---------+---------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+--------------------+-------------------------------+-------+---------------+---------+---------+------+----------+-------------+
| 1 | PRIMARY | animals | index | NULL | PRIMARY | 8 | NULL | 34269464 | Using where |
| 2 | DEPENDENT SUBQUERY | SavedAnimals | ALL | NULL | NULL | NULL | NULL | 10 | Using where |
+------+--------------------+-------------------------------+-------+---------------+---------+---------+------+----------+-------------+
2 rows in set (0.00 sec)
The UPDATE command never finishes if I run it.
QUESTION
How do I make mariaDB run with the Materialized select_type on the UPDATE like it does on the SELECT?
OR
Is there a totally separate way that I should approach this which would be quick?
Notes
Version: 10.3.23-MariaDB-log
Use JOIN rather than WHERE...IN. MySQL tends to optimize them better.
UPDATE animals AS a
JOIN SavedAnimals AS sa ON a.id = sa.animal_id
SET a.ignored=1, a.ignored_when=CURRENT_TIMESTAMP
WHERE a.ignored = 0
You should find an EXISTS clause more efficient than an IN clause. For example:
UPDATE animals a
SET a.ignored = 1,
a.ignored_when = CURRENT_TIMESTAMP
WHERE a.ignored = 0
AND EXISTS (SELECT * FROM SavedAnimals sa WHERE sa.animal_id = a.id)
Is there anything wrong with having one view reference another view? For example, say I have a
users table
CREATE TABLE `users` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`first_name` varchar(255) NOT NULL,
`last_name` varchar(255) NOT NULL,
PRIMARY KEY (`id`)
);
Then for the sake of argument a view that just shows all users
CREATE VIEW all_users AS SELECT * FROM users
And then a view that just returns their first_name and last_name
CREATE VIEW full_names AS SELECT first_name, last_name FROM all_users
Are there performance issue with basing one view off another? Let's also pretend this is the simplest of examples and a real world scenario would be much more complex, but the same general concept of basing one view of another view.
It depends on the ALGORITHM used. TEMPTABLE can be pretty expensive while MERGE should be the same as using the table directly so no loss there.
And for your example is the same:
mysql> explain select * from users;
+----+-------------+-------+--------+---------------+------+---------+------+------+---------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+------+---------+------+------+---------------------+
| 1 | SIMPLE | users | system | NULL | NULL | NULL | NULL | 0 | const row not found |
+----+-------------+-------+--------+---------------+------+---------+------+------+---------------------+
1 row in set (0.01 sec)
mysql> explain select * from all_users;
+----+-------------+-------+--------+---------------+------+---------+------+------+---------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+------+---------+------+------+---------------------+
| 1 | SIMPLE | users | system | NULL | NULL | NULL | NULL | 0 | const row not found |
+----+-------------+-------+--------+---------------+------+---------+------+------+---------------------+
1 row in set (0.00 sec)
mysql> explain select * from full_names ;
+----+-------------+-------+--------+---------------+------+---------+------+------+---------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+------+---------+------+------+---------------------+
| 1 | SIMPLE | users | system | NULL | NULL | NULL | NULL | 0 | const row not found |
+----+-------------+-------+--------+---------------+------+---------+------+------+---------------------+
1 row in set (0.02 sec)
I have very strange problem with mySQL and simple query with simple index. I am trying to use this query:
SELECT * FROM `counter_links` WHERE `link_id`=1544;
as you can see there is also index named link_id:
mysql> show indexes from counter_links;
+---------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+----------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+----------+---------------+
| counter_links | 1 | link_id | 1 | link_id | A | NULL | NULL | NULL | | BTREE | disabled | |
+---------------+------------+-------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+----------+---------------+
1 row in set (0.12 sec)
But look what EXPLAIN returned:
mysql> explain SELECT * FROM `counter_links` WHERE `link_id`=1544;
+----+-------------+---------------+------+---------------+------+---------+------+----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------------+------+---------------+------+---------+------+----------+-------------+
| 1 | SIMPLE | counter_links | ALL | NULL | NULL | NULL | NULL | 22103687 | Using where |
+----+-------------+---------------+------+---------------+------+---------+------+----------+-------------+
1 row in set (0.03 sec)
Why mySQL don't use index there? I see that the same is working good on smaller version of the table, but I am not able to recognize where the problem is. You do?
It is more stranger that it seems, because lower IDs sometimes works with the index.
Thanks you for all comments in advance!
Cheers!
Jakub
The comment field for th eindex says that the index is disabled, not sure why.
ALTER TABLE ... ENABLE KEYS may work
I have a MySQL 5.0 database with a few tables containing over 50M rows. But how do I know this? By running "SELECT COUNT(1) FROM foo", of course. This query on one table containing 58.8M rows took 10 minutes to complete!
mysql> SELECT COUNT(1) FROM large_table;
+----------+
| count(1) |
+----------+
| 58778494 |
+----------+
1 row in set (10 min 23.88 sec)
mysql> EXPLAIN SELECT COUNT(1) FROM large_table;
+----+-------------+-------------------+-------+---------------+----------------------------------------+---------+------+-----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------------------+-------+---------------+----------------------------------------+---------+------+-----------+-------------+
| 1 | SIMPLE | large_table | index | NULL | fk_large_table_other_table_id | 5 | NULL | 167567567 | Using index |
+----+-------------+-------------------+-------+---------------+----------------------------------------+---------+------+-----------+-------------+
1 row in set (0.00 sec)
mysql> DESC large_table;
+-------------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | NULL | auto_increment |
| created_on | datetime | YES | | NULL | |
| updated_on | datetime | YES | | NULL | |
| other_table_id | int(11) | YES | MUL | NULL | |
| parent_id | bigint(20) unsigned | YES | MUL | NULL | |
| name | varchar(255) | YES | | NULL | |
| property_type | varchar(64) | YES | | NULL | |
+-------------------+---------------------+------+-----+---------+----------------+
7 rows in set (0.00 sec)
All of the tables in question are InnoDB.
Any ideas why this is so slow, and how I can speed it up?
Counting all the rows in a table is a very slow operation; you can't really speed it up, unless you are prepared to keep a count somewhere else (and of course, that can become out of sync).
People who are used to MyISAM tend to think that they get count(*) "for free", but it's not really. MyISAM cheats by not having MVCC, which makes it fairly easy.
The query you're showing is doing a full index scan of a not-null index, which is generally the fastest way of counting the rows in an innodb table.
It is difficult to guess from the information you've given, what your application is, but in general, it's ok for users (etc) to see close approximations of the number of rows in large tables.
If you need to have the result instantly and you don't care if it's 58.8M or 51.7M, you can find out the approximate number of rows by calling
show table status like 'large_table';
See the column rows
For more information about the result take a look at the manual at http://dev.mysql.com/doc/refman/5.1/en/show-table-status.html
select count(id) from large_table will surely run faster