Database performance checklist - mysql

Question

In a database I have a table with order items. The table holds roughly 36 million of records.
Running a query like this takes about 3 minutes:
SELECT COUNT(DISTINCT DATE(created_on), product_id) FROM order_items;
Running a query like this takes about 13 seconds:
SELECT COUNT(1) FROM order_items;
Something tells me that 36 million of records is not that much, and that both queries are running rather slowly.
What would be the checklist to start looking into the performance issue here?
We are using MySQL (in fact, a Clustrix version of it, MySQL 5.0.45-clustrix-6.0.1).
Edit. Adding more info:
/* SHOW CREATE TABLE order_items; */
CREATE TABLE `order_items` (
`id` int(10) unsigned not null AUTO_INCREMENT,
`state` enum('pending','sold_out','approved','declined','cancelled','processing','completed','expired') CHARACTER SET utf8 not null default 'pending',
`order_id` int(10) unsigned not null,
`product_id` int(10) unsigned not null,
`quantity` smallint(5) unsigned not null,
`price` decimal(10,2) unsigned not null,
`total` decimal(10,2) unsigned not null,
`created_on` datetime not null,
`updated_on` datetime not null,
`employee_id` int(11),
`customer_id` int(11) unsigned not null,
PRIMARY KEY (`id`) /*$ DISTRIBUTE=1 */,
KEY `updated_on` (`updated_on`) /*$ DISTRIBUTE=1 */,
KEY `state` (`state`,`quantity`) /*$ DISTRIBUTE=3 */,
KEY `product_id` (`product_id`,`state`) /*$ DISTRIBUTE=2 */,
KEY `product` (`product_id`) /*$ DISTRIBUTE=1 */,
KEY `order_items_quantity` (`quantity`) /*$ DISTRIBUTE=2 */,
KEY `order_id` (`order_id`,`state`,`created_on`) /*$ DISTRIBUTE=3 */,
KEY `order` (`order_id`) /*$ DISTRIBUTE=1 */,
KEY `index_order_items_on_employee_id` (`employee_id`) /*$ DISTRIBUTE=2 */,
KEY `customer_id` (`customer_id`) /*$ DISTRIBUTE=2 */,
KEY `created_at` (`created_on`) /*$ DISTRIBUTE=1 */,
) AUTO_INCREMENT=36943352 CHARACTER SET utf8 ENGINE=InnoDB /*$ REPLICAS=2 SLICES=12 */
And:
/* SHOW VARIABLES LIKE '%buffer%'; */
+----------------------------------------+-------+
| Variable_name | Value |
+----------------------------------------+-------+
| backup_compression_buffer_size_bytes | 8192 |
| backup_read_buffer_size_bytes | 8192 |
| backup_write_buffer_size_bytes | 8192 |
| mysql_master_trx_buffer_kb | 256 |
| mysql_slave_session_buffer_size_events | 100 |
| net_buffer_length | 16384 |
| replication_master_buffer_kb | 65536 |
+----------------------------------------+-------+
Edit 2. Here's EXPLAIN statements for both queries:
mysql> EXPLAIN SELECT COUNT(1) FROM order_items;
+----------------------------------------------------------+-------------+-------------+
| Operation | Est. Cost | Est. Rows |
+----------------------------------------------------------+-------------+-------------+
| row_count "expr1" | 29740566.81 | 1.00 |
| stream_combine | 26444732.70 | 32958341.10 |
| compute expr0 := param(0) | 1929074.80 | 2746528.43 |
| filter isnotnull(param(0)) | 1915342.16 | 2746528.43 |
| index_scan 1 := order_items.order_items_quantity | 1854308.19 | 3051698.25 |
+----------------------------------------------------------+-------------+-------------+
5 rows in set (0.13 sec)
And:
mysql> EXPLAIN SELECT COUNT(DISTINCT DATE(created_on), product_id) FROM order_items;
+----------------------------------------------------------------------------------+-------------+------------+
| Operation | Est. Cost | Est. Rows |
+----------------------------------------------------------------------------------+-------------+------------+
| hash_aggregate_combine expr1 := count(DISTINCT (0 . "expr0"),(1 . "product_id")) | 10115923.36 | 4577547.38 |
| hash_aggregate_partial GROUPBY((0 . "expr0"), (1 . "product_id")) | 3707357.04 | 4577547.38 |
| compute expr0 := cast(1.created_on, date) | 2166388.20 | 3051698.25 |
| index_scan 1 := order_items.__idx_order_items__PRIMARY | 2151129.71 | 3051698.25 |
+----------------------------------------------------------------------------------+-------------+------------+
4 rows in set (0.24 sec)

Answer 1

The first query must walk the entire database, checking every row in the table. An index on created_on and product_id would probably speed it up significantly. If you don't know about indexes, http://use-the-index-luke.com is a great place to start.
The second query seems to me that it should be instant, because it only has to check table metadata and doesn't need to check any rows.

Answer 2

You should publish the query plan but I suspect that to process the query MySQL must walk through the product_id and the created_on indexes. For the created_on field it must also aggregate the values (the field is datetime but you want to group by date).If you need speed I would add and additional field created_on_date with only the date and I would create an index on the product_id and the created_on_date. It should make your query much faster.
Of course the count(1) query is faster because it doesn't read the table at all and it can use the indexes metadata.

Answer 3

Some things to note:
If you add INDEX(product_id, created_on), the first query should run faster because it would be a "covering index". (The fields can be in the opposite order.)
Running those two queries in the order given could have caused info to be cached, thereby making the second query run faster.
SELECT COUNT(*) FROM tbl will use the smallest index. (In InnoDB.)
If you have enough RAM, and if innodb_buffer_pool_size is bigger than the table, then one or other of the operations may been performed entirely in RAM. RAM is a lot faster than disk.
Please provide SHOW CREATE TABLE order_items; I am having to guess too much.
Please provide SHOW VARIABLES LIKE '%buffer%';. How much RAM do you have?
Edit
Since it is Clustrix, there could be radically different things going on. Here's a guess:
SELECT COUNT(1) ... can probably be distributed to the nodes; each node would get a subtotal; then the subtotals could (very rapidly) be added.
SELECT COUNT(DISTINCT ...)... really has to look at all the rows, one way or another. That is, the effort cannot be distributed. Perhaps what happens is that all the rows are shoveled to one node for processing. I would guess it is a couple GB of stuff.
Is there some way in Clustrix to get EXPLAIN? I would be interested to see what it says about each of the SELECTs. (And whether it backs up my guess.)
I would expect GROUP BY and DISTINCT to be inefficient in a 'sharded' system (such as Clustrix).

Answer 4

COUNT(1)
In Plan, stream_combine was used. It has read only the index (order_items_quantity (quantity))
COUNT(DISTINCT DATE(created_on), product_id)
In general, COUNT(DISTINCT...) may be inefficient in RDB, NewSQL Scale-Out RDB even more, it's because of difficulty in reducing inter-nodes traffic (lots of data should be forwarded to GTM node in many cases). So Clustrix needs 'dist_stream_aggregate' and the right index (the columns and the column orders)
In plan, hash_aggregate_partial was shown. It has scanned FULL TABLE (__idx_order_items__PRIMARY) and took lots of time (much bigger size)
For parallelism, it may not be enough number for the all cpus available. (i.e. SLICES=12). I wonder how many nodes and cpus per nodes (?)
Because of DATE(created_on), the index created_at (created_on) would not work. The optimizer (Plan) thought FULL TABLE SCAN is more efficient than both looking up INDEX(created_at) and then accessing TABLE (__idx_order_items__PRIMARY).
For this case, I recommend to test as below.
Add column create_on_date_type
create index new_index on order_items(create_on_date_type, productid)
regarding distribute=? & slices=?, test should be done for your dataset.(the number of slices might impacts on how much cpu parallelism works)
You have to make sure the plan has dist_stream_aggregate.
dist_stream_aggregate can work efficiently only with the 'new_index' columns for your query.
I believe you would be able to get better performance.