I am using mysql db for my application.I have one table with 40 million rows data.I want to run one update query in this table.Table contains three fields.Id,TaxiId,date&distance.I want to update distance to zero for particular Taxiid's.I am using one query like this
update abc set distance=0 where TaxiId in(2026,2031,2033,2035,2037,2039);
When using explain I am getting below result
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE abc range abc_trip_fk abc_trip_fk 5 1916397 Using where
Table Declare
CREATE TABLE `testdb`.`abc` (
`id` int(8) NOT NULL AUTO_INCREMENT,
`TaxiId` int(8) DEFAULT NULL,
`distance` double DEFAULT NULL,
`date` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (id),
KEY `trip_fk` (`TaxiId`) USING BTREE,
) ENGINE=InnoDB AUTO_INCREMENT=125127797 DEFAULT CHARSET=latin1;
When I run this query entire application hangs.
Innodb buffer pool option is very important option for mysql performance.
Try to make following settings
innodb_flush_log_at_trx_commit=2
innodb_buffer_pool_size=25GB
Disable indexes while insert/update/delete operation in database.
For more about mysql performance setting find here
Related
I have a query.
SELECT id_id FROM videos_member ORDER BY date_id DESC LIMIT 0,30
Here is the table
CREATE TABLE IF NOT EXISTS `videos` (
`id_id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`date_id` int(11) NOT NULL,
PRIMARY KEY (`id_id`),
KEY `date_id` (`date_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=3 ;
I keep getting this
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE videos ALL NULL NULL NULL NULL 342 Using filesort
Why isn't is using the index?
The table contains (or at least MySQL thinks it contains) 342 rows. This is tiny and likely fits into a single block of physical storage, which means it can be read in a single read operation. Using the index would require at least two read operations. So MySQL might be smart here and realize that reading the whole table at once is just more efficient than reading the index and then using it to access the table.
In other words if you insert more rows into the table the plan might change to using index.
I am performing a very simple select over a simple table, where the column that I am filtering over has an index.
Here is the schema:
CREATE TABLE IF NOT EXISTS `tmp_inventory_items` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`transmission_id` int(11) unsigned NOT NULL,
`inventory_item_id` int(11) unsigned DEFAULT NULL,
`material_id` int(11) unsigned NOT NULL,
PRIMARY KEY (`id`),
KEY `transmission_id` (`transmission_id`)
KEY `inventory_item_id` (`inventory_item_id`),
KEY `material_id` (`material_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=21 ;
Here is the SQL:
SELECT * FROM `tmp_inventory_items` WHERE `transmission_id` = 330
However, when explaining the query, I see that the index is NOT being used, why is that (the table has about 20 rows on my local machine)?
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE tmp_inventory_items... ALL transmission_id NULL NULL NULL 13 Using where
No key is being used even if I hint the mysql with USE INDEX(transmission_id)... this looks very strange to me (MySQL Version 5.5.28)
Because MySQL's algorithms tell it that preparing an index and using it would use more resources than simply performing the query without one.
When you feed query syntax to a DBMS, one of the things it does is attempts to determine the most efficient way to process the query (usually there are at least tens of ways).
If you want to, you can use FORCE INDEX(transmission_id) (documented here) which will inform MySQL that a table scan is assumed to be very expensive, but it's not recommended as to determine for 20 rows, it's just not valuable.
I have the following mysql query
select points_for_user from items where user_id = '38415';
explain on the query returns this
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE items index NULL points_for_user_index 2 NULL 1000511 Using index
The problem is, shouldn't the number of rows be FAR less then the number of rows in the table because of the index?
user_id is the primary index, so I tried creating an index on just points_for_user and that still look through every row. An index on user_id AND points_for_user still searches every row.
What am I missing?
Thanks!
CREATE TABLE IF NOT EXISTS `items` (
`capture_id` int(11) NOT NULL AUTO_INCREMENT,
`id` int(11) NOT NULL,
`creator_user_id` bigint(20) NOT NULL DEFAULT '0',
`user_id` int(11) NOT NULL,
`accuracy` int(11) NOT NULL,
`captured_at` timestamp NOT NULL DEFAULT '2011-01-01 06:00:00',
`ip` varchar(30) NOT NULL,
`capture_type_id` smallint(6) NOT NULL DEFAULT '0',
`points` smallint(6) NOT NULL DEFAULT '5',
`points_for_user` smallint(6) NOT NULL DEFAULT '3',
PRIMARY KEY (`capture_id`),
KEY `capture_user` (`capture_id`,`id`,`user_id`),
KEY `user_id` (`user_id`,`id`),
KEY `id` (`id`),
KEY `capture_creator_index` (`capture_id`,`creator_user_id`),
KEY `points_capture_index` (`points_for_user`,`creator_user_id`),
KEY `points_for_user_index` (`points_for_user`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=1008992 ;
select count(*) from items where user_id = '38415'
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE captures ref user_munzee_id user_munzee_id 4 const 81 Using index
the mysql optimizer try to use the best possible index during the query.
In your first query the optimizer is considering points_for_user_index the best choice, in fact the Extra column show the "Using index" status, this means a "Covering index".
The "Covering index" occurs when all fields required for a query (in your case select points_for_user from ... ) are contained in an index, this avoid the access to the full mysql data (.MYD) in favour of the direct index access (.MYI)
First of all you can try to rebuild the index tree analyzing table
ANALYZE TABLE itemes;
Note for very large tables:
ANALYZE TABLE analyzes and stores the key distribution for a table.
During the analysis, the table is locked with a read lock for InnoDB
and MyISAM. This statement works with InnoDB, NDB, and MyISAM tables.
For MyISAM tables, this statement is equivalent to using myisamchk
--analyze.
If "the problem" persist and you want to bypass the optimizer choice you can explicit try to force the usage of an index
EXPLAIN SELECT points_for_user FROM items USE INDEX ( user_id ) WHERE user_id = '38415'
More details: http://dev.mysql.com/doc/refman/5.5/en/index-hints.html
Cristian
I'm working with an application that has a MySQL database at Amazon RDS. The table in questions is set up as such:
CREATE TABLE `log` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`timestamp` datetime NOT NULL,
`username` varchar(45) NOT NULL,
.. snip some varchar and int fields ..
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
This system has been in beta for a while and already the dataset is quite huge and the queries are starting to be rather slow.
SELECT COUNT(*) FROM log --> 16307224 (takes 105 seconds to complete)
This table is pretty much only used to build one report off a query like this
SELECT timestamp, username, [a few more] FROM log
WHERE timestamp BETWEEN '2012-03-30 08:00:00' AND '2012-03-30 16:00:00'
AND username='XX'
Which typically will give something between 1000 and 6000 rows taking around 100-180 sec to complete, meaning the web application will often time out and leave an empty report (I will look in to the timeout as well, but this question is for the root cause).
I'm not very good with databases, but my guess is that it's the BETWEEN that's killing me here. What I'm thinking is that I should perhaps somehow use the timestamp as index. Timestamp togethere with username should still provide uniqueness (I don't use the id field for anything).
If there's anyone out there with suggestions for optimizations I'm all ears.
UPDATE:
Table is now altered to the following
CREATE TABLE `log` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`timestamp` datetime NOT NULL,
`username` varchar(45) NOT NULL,
.. snip ..
`task_id` int(10) unsigned DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `index_un_ts` (`timestamp`,`username`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
EXPLAIN of the SELECT statement returns the following
id => 1
select_type => SIMPLE
table => log
type => range
possible_keys => index_un_ts
key => index_un_ts
key_len => 55
ref =>
rows => 52258
Extra => Using where; Using index
Well a index on the timestamp column and userid would be helpful. You need to be able to read the output of a EXPLAIN Statement.
Go to MySQL and do the Following:
EXPLAIN SELECT timestamp, username, [a few more] FROM log
WHERE timestamp BETWEEN '2012-03-30 08:00:00' AND '2012-03-30 16:00:00'
AND username='XX'
This show you the plan MySQL uses to execute the query. There will be column called key. This indicates what index MySQL is using in the query. I suspect you will see ALL there which means MySQL is scanning the table from top to bottom matching every row against your where clause. Now create a index on the timestamp and userid columns. Run the EXPLAIN statement again. You should see the index that you created in the key column.
If MySQL uses the index then your query should be considerably faster. Just remember not to over index. Indexes make inserts, updates and deletes slower. When you insert a new row into a table and there is three indexes on the table the new row has to write 3 values to the three different indexes. So it is a double edged sword.
I have the following table
CREATE TABLE `Config` (
`id` mediumint(9) NOT NULL AUTO_INCREMENT,
`type_id` mediumint(9) DEFAULT NULL,
`content_id` mediumint(9) DEFAULT NULL,
`menu_id` int(11) DEFAULT NULL,
`field` varchar(50) NOT NULL DEFAULT '',
`value` text NOT NULL,
PRIMARY KEY (`id`),
KEY `menu_id` (`menu_id`) USING BTREE,
KEY `type_id` (`type_id`,`content_id`,`menu_id`) USING BTREE
) ENGINE=MyISAM AUTO_INCREMENT=1;
It's filled with about 800k rows of test data. Whenever I run the following query it takes about 0.4 seconds to complete:
SELECT id, content_id, menu_id, field, `value`
FROM Config
WHERE type_id = ?
AND content_id = ?
An explain tells me, MySQL is doing a full tablescan instead of using an index:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE Config ALL 792674 Using where
Can someone please explain what I am doing wrong here? How has the index to be like so it's used here? Sometimes the query has the extra condition AND menu_id = ?, which should benefit from it, too.
I had a problem once with a query where it doesn't use the index that I specified. It turned out, MySQL won't use your index if the result (of your query) exceeds certain rows. For an example, if the result itself is taking a lot of your total rows, it won't use your index. However, I don't have the specific percentage. You could try adjusting the query to return smaller result to test this theory.
My question about the problem: MySQL datetime index is not working
0.4s isn't bad for 800,000 rows. The MySQL optimiser may determine it doesn't need your indexes.
You could try using "hints" to see if you can change performance outcomes:
http://dev.mysql.com/doc/refman/5.1/en/index-hints.html
The accepted answer is actually right, but if you want your MySQL to use the Index regardless the matches rows, you can specify the FORCE INDEX (index_name) command.