Optimizing Slow, Indexed Select MySql Query - mysql

Question

I am trying to execute a simple select query using a table indexed on src_ip like so:
SELECT * FROM netflow_nov2 WHERE src_IP=3111950672;
However this is not completed after even 4 or 5 hours. I need the response to be in the range of a few seconds. I am wondering how I can optimize it so this is the case.
Also note that source ip’s were converted to integers using the built in SQL command.
Other information about the table:
The table contains netflow data parsed from nfdump. I am using the table to get information about specific IP addresses. In other words, basically only queries like the above will be used.
Here is the relevant info as given by SHOW TABLE STATUS for this table:
Rows: 4,205,602,143 (4 billion)
Data Length: 426,564,911,104 (426 GB)
Index Length: 57,283,706,880 (57 GB)
Information about the system:
Hard disk: ~2TB, using close to maximum
RAM: 64GB
my.cnf file:
see gist: https://gist.github.com/ashtonwebster/e0af038101e1b42ca7e3
Table structure:
mysql> DESCRIBE netflow_nov2;
+-----------+------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+------------------+------+-----+---------+-------+
| date | datetime | YES | MUL | NULL | |
| duration | float | YES | | NULL | |
| protocol | varchar(16) | YES | | NULL | |
| src_IP | int(10) unsigned | YES | MUL | NULL | |
| src_port | int(2) | YES | | NULL | |
| dest_IP | int(10) unsigned | YES | MUL | NULL | |
| dest_port | int(2) | YES | | NULL | |
| flags | varchar(8) | YES | | NULL | |
| Tos | int(4) | YES | | NULL | |
| packets | int(8) | YES | | NULL | |
| bytes | int(8) | YES | | NULL | |
| pps | int(8) | YES | | NULL | |
| bps | int(8) | YES | | NULL | |
| Bpp | int(8) | YES | | NULL | |
| Flows | int(8) | YES | | NULL | |
+-----------+------------------+------+-----+---------+-------+
15 rows in set (0.02 sec)
I have additional info about the indexes and the results of explain, but briefly:
-The indexes are b-trees, and there are indexes for date, src_ip, and dest_ip, but only src_ip will really be used
-Based on the output of EXPLAIN, the src_ip index is being used for that particular query mentioned at the top
And the output of mysqltuner:
see gist: https://gist.github.com/ashtonwebster/cbfd98ee1799a7f6b323
SHOW CREATE TABLE output:
| netflow_nov2 | CREATE TABLE `netflow_nov2` (
`date` datetime DEFAULT NULL,
`duration` float DEFAULT NULL,
`protocol` varchar(16) DEFAULT NULL,
`src_IP` int(10) unsigned DEFAULT NULL,
`src_port` int(2) DEFAULT NULL,
`dest_IP` int(10) unsigned DEFAULT NULL,
`dest_port` int(2) DEFAULT NULL,
`flags` varchar(8) DEFAULT NULL,
`Tos` int(4) DEFAULT NULL,
`packets` int(8) DEFAULT NULL,
`bytes` int(8) DEFAULT NULL,
`pps` int(8) DEFAULT NULL,
`bps` int(8) DEFAULT NULL,
`Bpp` int(8) DEFAULT NULL,
`Flows` int(8) DEFAULT NULL,
KEY `src_IP` (`src_IP`),
KEY `dest_IP` (`dest_IP`),
KEY `date` (`date`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
Thanks in advance

Answer 1

Your current table structure is optimized for random writes: records are placed on disk in the order of writes.
Unfortunately the only read pattern that is well supported by such a structure is a full-table scan.
Usage of non-covering secondary indices still results in a lot of random disk seeks which are killing performance.
The best reading performance is obtained when data is read in the same order as it is located on disk, which for InnoDB means in the primary key order.
A materialized view (another InnoDB table that has an appropriate primary key) could be a possible solution. In this case a primary key that starts with src_IP is required.
upd: The idea is to achieve data locality and avoid random disk IO, aiming for sequential reading. This means that your materialized view will look like this:
CREATE TABLE `netflow_nov2_view` (
`row_id` bigint not null, -- see below
`date` datetime DEFAULT NULL,
`duration` float DEFAULT NULL,
`protocol` varchar(16) DEFAULT NULL,
`src_IP` int(10) unsigned DEFAULT NULL,
`src_port` int(2) DEFAULT NULL,
`dest_IP` int(10) unsigned DEFAULT NULL,
`dest_port` int(2) DEFAULT NULL,
`flags` varchar(8) DEFAULT NULL,
`Tos` int(4) DEFAULT NULL,
`packets` int(8) DEFAULT NULL,
`bytes` int(8) DEFAULT NULL,
`pps` int(8) DEFAULT NULL,
`bps` int(8) DEFAULT NULL,
`Bpp` int(8) DEFAULT NULL,
`Flows` int(8) DEFAULT NULL,
PRIMARY KEY (`src_IP`, `row_id`) -- you won't need other keys
) ENGINE=InnoDB DEFAULT CHARSET=latin1
where row_id has to be maintained by your materializing logic, since you don't have it in the original table (or you can introduce an explicit auto_increment field to your original table, it's how InnoDB handles it anyway).
The crucial difference is that now all data on the disk is placed in the primary key order, which means that once you locate the first record with a given 'src_IP' all other records can be obtained as sequentially as possible.
Depending on the way your data is written and adjacent application logic it can be accomplished either via triggers or by some custom external process.
If it is possible to sacrifice current write performance (or use some async queue as a buffer) then probably having a single table optimized for reading would suffice.
More on InnoDB indexing:
http://dev.mysql.com/doc/refman/5.6/en/innodb-index-types.html

Answer 2

I would think that reading the table without an index would take less than 5 hours. But you do have a big table. There are two "environmental" possibilities that would kill the performance:
The table is locked by another process.
The result set is huge (tens of millions of rows) and the network latency/processing time for returning the result set is causing the problem.
My first guess, though, is that the query is not using the index. I missed this at first, but you have one multi-part index. The only index this query can take advantage of is one where the first key is src_IP. So, if you index is either netflow_nov2(src_IP, date, dest_ip) or netflow_nov2(src_IP, dest_ip, date), then you are ok. If either of the other columns is first, then this index will not be used. You can easily see what is happening by putting explain in front the query to see if the index is being used.
If this is a problem, create an index with src_IP as the first (or only) key in the index.