I am having trouble with one of my queries which is very slow from time to time.
SELECT * FROM table WHERE primary_key IN (1,2,...)
very slow sometimes even 5s for some 100 entries. Obviously we have index on the primary key. Is there any way to optimize this query. The table is slightly big about 100 million entries.
Can you post an EXPLAIN, perhaps there's something fishy going on.
Also, if this is a MyISAM database table and you have high concurrent INSERTs, the table will be locked during the inserts so your SELECTs may block during this time.
For a 100M records table you can't really do much in terms of query optimization as it is simple enough. What you could look into is MySQL config tuning.
In case your table is InnoDB/XtraDB (and if it's not i'd recommend to use those engines) you might want to look into innodb_buffer_pool_size variable - it's essential that used index fits into memory to achieve best performance for your queries.
Related
I want to change engine of 2 million rows table from MyISAM to InnoDB. I am afraid of this long time operation, so I created similar structure InnoDB table and now I want to copy all data from old one to this new one. What is the fastest way? SELECT INSERT? What about START TRANSACTION? Please, help. I dont want to hang my server.
Do yourself a favor: copy the whole setup to your local machine and try it all out there. You'll have a much better idea of what you are getting into. Just be aware of potential differences in hardware between your production server and your local machine.
The fastest way is probably the most straightforward way:
INSERT INTO table2 SELECT * FROM table1;
I suspect that you cannot do it any faster than what is built into the ALTER. And it does have to copy over all the data and rebuild all the indexes.
Be sure to have innodb_buffer_pool_size raised to prepare for InnoDB. And lower key_buffer_size to allow room. Suggest 35% and 12% of RAM, respectively, for the transition. After all tables are converted, suggest 70% and a mere 20MB.
One slight speedup is to do some select that fetches the entire table and the entire PRIMARY KEY (if it can be cached). This will do some I/O before really starting. Example: SELECT avg(id) FROM tbl where id is the primary key. And SELECT avg(foo) FROM tbl where foo is not indexed but it numeric. These will force a full scan of the PK index and the data, thereby caching the stuff that the ALTER will have to read.
Other tips on converting: http://mysql.rjweb.org/doc.php/myisam2innodb .
I have a Myisam table with composite unique key of 2 columns and 90 million data. Now we are facing memory and load issues and after going through the web I am planning to include partitioning and changing this table to Innodb for better performance. But I have following concerns:
Changing to innodb will have a huge downtime, Is it possible to minimize the downtime?
Most of the select query are on a particular column of the key on which I am planning to have the hash partitioning, how much it will effect the query on another key column?
Will these changes improve the performance to the extent mentioned theoretically? Is there any better solution for such cases. Any suggestion or experience can be helpful.
My queries are simple like
Select * from Table where Col1= "Value"
Select * from Table where Col1="Value" and Col2 IN (V1,V2,V3)
Inserts are very frequently.
InnoDB will probably help some. Conversion to InnoDB comes with some issues, as I state in My conversion blog.
Partitioning, per se, buys no performance gain. My partitioning blog lists 4 cases where you can, with design changes, gain performance.
Regardless of the Engine, your two queries will both benefit from
INDEX(col1, col2)
No form of partitioning will help. HASH partitioning is especially useless.
Conversion to InnoDB will take a lot of downtime, unless pt-online-schema-change will work for your case. Research it.
Also read my answers and comments on
Can i set up Mysql to auto-partition?
for more specifics.
It may be that adding that index is the main performance gain. But you have to do a lengthy ALTER to get it. MyISAM does not have ALGORITHM=INPLACE.
Innodb (about perfomance we are talking now) have sense only when there are alot of inserts and updates to your table, because of row-locking table.
If the most queries on your table are SELECTs then MyIsam will be faster.
Advice: put in my.cnf key_buffer_size equal to 25% of your free RAM.
If inserts on your database are very frequent, you will likely gain performance by switching to innodb, which won't lock down entire tables to insert, allowing other clients to select data concurrently.
Regarding question #1, if you are worried about downtime, I'd suggest you find a parallel dump/load solution for migrating your data to innodb. if you simply run an ALTER statement on your tables, this is a single threaded operation which will be much slower.
Regarding #2, you'd have to post a schema along with your partitioning strategy and the queries you are worried about.
I am having a question about "which storage device to choose" for my database tables. I have a table with 28 million records. I will insert data after creating the table, after that, no insert - update -delete operation will take place. Never. Only select operations.
I have a query like below
SELECT `indexVal`, COUNT(`indexVal`) FROM `key_word` WHERE `hashed_word` IN ('001','01v','0ji','0k9','0vc','0#v','0%d','13#' ,'148' ,'1e1','1sx','1v$','1#c','1?b','1?k','226','2kl','2ue','2*l','2?4','36h','3au','3us','4d~') GROUP BY `indexVal`
This counts how many number of times a particular result appeared in search. In InnoDB, this operation took 5 seconds. This is too much, because my orifginal dataset will be in billions.
To do this kind of work, which MySQL storage you recommend?
More than the storage engine, having the proper index in place seems important.
In your case, CREATE INDEX idx_1 ON key_word (index_val, hashed_word) should help.
And if the data truly never changes, you could even pre-compute and cache some of those results.
For example
CREATE TABLE counts AS SELECT index_val, hashed_word, count(index_val)
FROM key_word
GROUP BY index_val, hashed_word
For SELECT-only queries, ARCHIVE is the fastest storage engine.
As it is MyISAM-based, and the following advice is for MyISAM as well, don't use varchar but fixed-size char columns, and you will get better performance.
Sure, even faster if it's the data is loaded in memory, instead read from disk.
MySQL MyISAM "Table1" having 70% select , 13% update and 0.67% insert statements approximate.
There is one "count_column(int)" which used to increase count with primary key.(Update statements)
Updating of "count_column" make table select queries in "Waiting for table level lock"
So separating "count_column" in other table will reduce "Waiting for table level lock" or not?
I also need separated column in select statements with join.
Thanks, Yogs
AFAIK your locking problem is the COUNT with INSERT, not the UPDATE itself - but you must have a huge bunch of SELECTs. Your question is lacking quite some details...
COUNT is really optimized on MyISAM tables, if you encounter problems with that you maybe should consider a count estimate or memory tables holding this value:-\ But an exact row count is stored for MyISAM that is extremely quick to get by the storage engine, so you maybe even slowed down MySQL with your solution. "Slow" COUNT is valid for engines like InnoDB because of their transactional nature.
One other thing to consider is, that storing a count in a column in the table itself is an additional column for each row and quite bad.
And if you are using triggers to accomplish that you should be aware of http://dev.mysql.com/doc/refman/5.0/en/faqs-triggers.html#qandaitem-B-5-1-12 :)
Moving the frequently updated cells in another table will greatly reduce number of locks on the table and speed up select on it. Converting the table to InnoDB also can help (if you are not using full-text indexes, they are still not supported in MySQL 5.5 InnoDB), since it uses row-level locks instead of table-level. If you have a lot of queries, take a look at this article about implementing efficient counters
i was wondering, if i add one index for each field in every table of my DB, will that make my queries run faster?
or do i have to analyze my queries and create indexes only when required?
Adding an index on each column will probably make most of your queries faster, but it's not necessarily the best approach. It is better to tune your indexes to your specific queries, using EXPLAIN and performance measurements to guide you in adding the correct indexes.
In particular you need to understand when you shouldn't index a column, and when you need multi-column indexes.
I would advise reading the MySQL manual for optimization of SELECT statements which explains under what conditions indexes can be used.
The more indexes you have, the heavier inserting/updating gets. So it's a tradeoff. The select queries that cannot use an index now will get quicker ofcourse, but if you check what fields you're joining on (or using in a where) you will not trade off that much
(and, ofcourse, there is the disk-space, but most of the time I don't really care bout that: ) )
Another point is that MySql can only use a single index for a query, so if your query is
SELECT * FROM table WHERE status = 1 AND col1='foob' AND col2 = 'bar'
MySql will use 1 of the indexes, and filter out the rest reading the data from the table.
If you have queries like this, its better to create a composite index on (status, col1, col2)
Adding index on every field in every table is not smart.
You should add indexes ONLY on columns that you use in the WHERE clause in select OR on which you sort.
Often, the best results are achieved by using multi-column indexes that are specific to your SQL selects.
There are also a partial indexes with limit on the length of field which can also be used to optimize performance and reduce the index site.
Every unnecessary index will slow down the database during the insert because on every insert, every index has to be updated.
Also the more indexes you have, the more chances you have of data corruption. And lastly, indexes take extra storage space on disk, sometimes a lot of space.
Also MySQL tries to keep indexes in memory. If you have unnecessary indexes, there is a good change MySQL will end up using up the available memory with unnecessary indexes in which case your performance will degrade considerable.
Creating the right kind of indexes is probably the single most important optimization technique. That's why when someone asks something like this I thought it was a joke.
This question can only be asked by someone who have not read a single book on MySQL. Just get a good book and read it, then you will not have to ask questions like this.