I have a very good pc and I was wondering why does it takes so long to make a very simple request to one of my table.
My pc:
i9 10900kf (10 cores)
64 GB ram
2TB NVME SSD
RTX 3090 Ventus
1: My table has 531 732 rows (this is not a lot of rows) with 39 columns
I have the following indexes to my table:
DisplayName
TmiSentTs
Username
Channel
DisplayName_TmiSentTs
Username_TmiSentTs
When I make the following query, it takes 66.016 seconds to get a response:
SELECT from_unixtime(TmiSentTs/1000,'%Y/%m/%d %H:%i:%s'),
DisplayName, message
FROM MY_TABLE
WHERE displayname LIKE '%ab%'
ORDER BY TmiSentTs DESC;
I don't think this is normal.
I tried to:
change my innodb_write_io_threads from 4 to 32
change my innodb_read_io_threads from 4 to 32
But none of this works and I have another table (in another database) with 37 325 332 rows and it takes 2 seconds for a similar query.
EDIT:
After a bit of research, I found that this
SELECT * FROM pepegaclapwr.twitchmessages where instr('aa',username)<>0;
Is faster than
SELECT * FROM pepegaclapwr.twitchmessages where username like '%aa%';
For the same result
An index on the displayname column won't help this query. Any LIKE condition with wildcards at the start of the pattern you are searching for cannot use an index, so it is bound to do a table-scan.
Think about a telephone book. If I ask you to look up last names that start with "T" it's easy because the book is sorted by last name. But if I ask you to look up names where "T" is the 4th letter (or anything after the first letter), the fact that the book is sorted doesn't help. You still have to read the whole book page by page to find the names I asked for.
To optimize the kind of query like the one you show above, you may find it easier to use a fulltext index, but that's only if you are searching for whole words. It looks like you are searching for any string that contains "ab" somewhere in it. This is not a whole word, so a fulltext index won't help either.
In that case, your only solution is to add another column to the table to indicate whether the row contains "ab", and then index that column. In MySQL 5.7 and later, you can can make a virtual column based on an expression.
ALTER TABLE MyTable
ADD COLUMN displayname_ab TINYINT NOT NULL AS (displayname LIKE '%ab%'),
ADD INDEX (displayname_ab);
Then search:
SELECT ... FROM MyTable WHERE displayname_ab = true;
In MySQL 8.0 you don't even need to make a virtual column, you can make an expression index directly from an expression on an existing column:
ALTER TABLE MyTable
ADD INDEX ((displayname LIKE '%ab%'));
Then if you search using the exact same expression, and it will use the index:
SELECT ... FROM MyTable WHERE displayname LIKE '%ab%';
But this fixes the string you need into the virtual column definition. It doesn't help if you need to search for "cd" tomorrow, or any other pattern.
Related
I have a table with data like this:
# id parent sequence mz1_monoisotopic
90 0 SGVNHR 669.34
93 0 IEEIATDLK 1031.56
95 1 MDLILDDR 990.49
100 1 LEVSEELIEK 1188.64
My application treats this as a 'base table' from which new tables are generated by adding some value (defined by the user) to the column mz1_monoisotopic. For example:
SELECT * FROM my_table WHERE (mz1_monoisotopic + 55) BETWEEN 1000 and 1002
Since most of the lookups are done via the mz1_monoisotopic column, and many of my tables are millions upon millions of entries, I really need to increase search speed. Is it possible to leveraging indexing in this situation or am I out of luck? I learned about functional indexes, but the problem is that the added value can change.
Additionally, the index itself takes too long to create on a new table, so it almost seems like my only option is a full table search.
Yes, an ordinary index works fine. You don't even need an expression index.
Behold! The power of algebra!
SELECT * FROM my_table WHERE mz1_monoisotopic BETWEEN 1000 - 55 and 1002 - 55;
The value can change from query to query, and the index will still work fine. As long as the expressions on the right side evaluate to constant values, they can be used for an indexed lookup.
I am using mysql database,its table name is license_csv ,and it has 18 million records, and it is using MYISAM engine when i try to count record without where condition it take less than 1 sec for count, but when i applied where condition it take 2 min for count result, i have applied indexing for that 2 columns lic_state and lic_city, still it take 2 min for the count result, can anyone please help me what i need to do now for that query ? here i also added my query,
SELECT COUNT(*) as total
from license_csv WHERE
lic_state like '%ca%' AND lic_city LIKE '%fresno%'
The problem with your query is that you are using the % wildcard on both sides of the string. This defeats an existing index. Because indexes are sequential, MySQL is not able to use them for this type of condition, since there is nothing to tell him where the start searching.
Possible options :
if possible, switch to equality instead of LIKE; you could then takd advantage of an index on (lic_state, lic_sity)
else, removing the wildcard on the left side (lic_state like 'ca%') could help
switch to FULL TEXT search
store the list of CA codes and cities in another table (which should much smaller than the licences table), and have a column in the licences table referrencing its primary key; then JOIN that table in the search. With this option, the range of records to search when applying the search condition will be mch smaller
I started looking into Index(es) in depth for the first time and started analyzing our db beginning from the users table for the first time. I searched SO to find a similar question but was not able to frame my search well, I guess.
I was going through a particular concept and this first observation left me wondering - The difference in these Explain(s) [Difference : First query is using 'a%' while the second query is using 'ab%']
[Total number of rows in users table = 9193]:
1) explain select * from users where email_address like 'a%';
(Actually matching columns = 1240)
2) explain select * from users where email_address like 'ab%';
(Actually matching columns = 109)
The index looks like this :
My question:
Why is the index totally ignored in the first query? Does mySql think that it is a better idea not to use the index in the case 1? If yes, why?
If the probability, based statistics mysql collects on distribution of the values, is above a certain ratio of the total rows (typically 1/11 of the total), mysql deems it more efficient to simply scan the whole table reading the disks pages in sequentially, rather than use the index jumping around the disk pages in random order.
You could try your luck with this query, which may use the index:
where email_address between 'a' and 'az'
Although doing the full scan may actually be faster.
This is not a direct answer to your question but I still want to point it out (in case you already don't know):
Try:
explain select email_address from users where email_address like 'a%';
explain select email_address from users where email_address like 'ab%';
MySQL would now use indexes in both the queries above since the columns of interest are directly available from the index.
Probably in the case where you do a "select *", index access is more costly since the optmizer has to go through the index records, find the row ids and then go back to the table to retrieve other column values.
But in the query above where you only do a "select email_address", the optmizer knows all the information desired is available right from the index and hence it would use the index irrespective of the 30% rule.
Experts, please correct me if I am wrong.
I am running the following query and however I change it, it still takes almost 5 seconds to run which is completely unacceptable...
The query:
SELECT cat1, cat2, cat3, PRid, title, genre, artist, author, actors, imageURL,
lowprice, highprice, prodcatID, description
from products
where title like '%' AND imageURL <> '' AND cat1 = 'Clothing and accessories'
order by userrating desc
limit 500
I've tried taking out the "like %", taking out the "imageURl <> ''" but still the same. I've tried returning only 1 colum, still the same.
I have indexes on almost every column in the table, certainly all the columns mentioned in the query.
This is basically for a category listing. If I do a fulltext search for something in the title column which has a fulltext index, it takes less than a second.
Should I add another fulltext index to column cat1 and change the query focus to "match against" on that column?
Am I expecting too much?
The table has just short of 3 million rows.
You said you had an index on every column. Do you have an index such as?
alter table products add index (cat1, userrating)
If you don't, give it a try. Run that query and let me know if it run faster.
Also, I assume you're actually setting some kind of filter instead of the % on the title, field, right?
You should rather have the cat1 as a integer, then a string in these 3 million rows. You must also index correctly. If indexing all columns only improved, then it'd be a default thing the system would do.
Apart from that, title LIKE '%' doesn't do anything. I guess you use it to search so it becomes `title LIKE 'search%'
Do you use any sort of framework to fetch this? Getting 500 rows with a lot of columns can exhaust the system if your framework saves this to a large array. It may probably not be the case, but:
Try running a ordinary $query = mysql_query() and while($row = mysql_fetch_object($query)).
I suggest to add an index with the columns queried: title, imageURL and cat1.
Second improvement: use the SQL server cache, it will deadly improve the speed.
Last improvement: if you query is always like that, only the values change, then use prepared statements.
Well, I am quite sure that a % as the first char in a LIKE clause, gives you a full table scan for that column (in your case you won't have that full table scan executed because you already have restricting clauses in the AND clause).
Beside that try to add an index on cat1 column. Also, try to add other criterias to your query, in order to reduce the size of the dataset - your working data set (the number of rows that matches your query, without the LIMIT clause) might be too big also.
I am trying to include in a MYSQL SELECT query a limitation.
My database is structured in a way, that if a record is found in column one then only 5000 max records with the same name can be found after that one.
Example:
mark
..mark repeated 5000 times
john
anna
..other millions of names
So in this table it would be more efficent to find the first Mark, and continue to search maximum 5000 rows down from that one.
Is it possible to do something like this?
Just make a btree index on the name column:
CREATE INDEX name ON your_table(name) USING BTREE
and mysql will silently do exactly what you want each time it looks for a name.
Try with:
SELECT name
FROM table
ORDER BY (name = 'mark') DESC
LIMIT 5000
Basicly you sort mark 1st then the rest follow up and gets limited.
Its actually quite difficult to understand your desired output .... but i think this might be heading in the right direction ?
(SELECT name
FROM table
WHERE name = 'mark'
LIMIT 5000)
UNION
(SELECT name
FROM table
WHERE name != 'mark'
ORDER BY name)
This will first get upto 5000 records with the first name as mark then get the remainder - you can add a limit to the second query if required ... using UNION
For performance you should ensure that the columns used by ORDER BY and WHERE are indexed accordingly
If you make sure that the column is properly indexed, MySQL will take care off optimisation for you.
Edit:
Thinking about it, I figured that this answer is only useful if I specify how to do that. user nobody beat me to the punch: CREATE INDEX name ON your_table(name) USING BTREE
This is exactly what database indexes are designed to do; this is what they are for. MySQL will use the index itself to optimise the search.