adding an additional item in SQL statement causes slow performance - mysql

Question

I have a sql statement that looks like this:
SELECT colID
FROM tableName
WHERE ColDateStart <='$lowerDate'
AND ColDateStart>='$upperDate'
AND ColVcamID='$id1'
AND ColVlviID='$id2'
AND ColSomeID='$id3';
All the columns in the WHERE statment are indexed columns.
When I run this it takes over a second. However when I run this without other Id3, the performance is considerably improved (0.03 seconds).
When I run explain, with otherId3, it uses an index merge using otherId1 and otherId3. However when I remove the otherId3, it uses the single index of otherId2.
Why does adding otherId3 make an impact on the performance?
Table Structure:
+----------------------+-------------+------+-----+---------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------------+-------------+------+-----+---------------------+----------------+
| ColID | int(11) | NO | PRI | NULL | auto_increment |
| ColCustID | int(11) | NO | MUL | 0 | |
| ColCarrID | int(11) | NO | MUL | NULL | |
| ColTariID | int(11) | NO | MUL | 0 | |
| ColCarrierRef | varchar(30) | NO | MUL | | |
| ColNumbID | int(11) | NO | MUL | 0 | |
| ColVlviID | int(11) | NO | MUL | NULL | |
| ColVcamID | int(11) | NO | MUL | NULL | |
| ColSomeID | int(11) | NO | MUL | NULL | |
| ColVlnsID | int(11) | NO | MUL | NULL | |
| ColNGNumber | varchar(12) | NO | | | |
| ColOrigNumber | varchar(16) | NO | MUL | NULL | |
| ColCLIRestrictedFlag | int(2) | NO | | NULL | |
| ColOrigLocality | varchar(11) | NO | MUL | | |
| ColOrigAreaCode | varchar(11) | NO | MUL | | |
| ColTermNumber | varchar(16) | NO | MUL | NULL | |
| ColBatchNumber | varchar(10) | NO | | | |
| ColDateStart | date | NO | MUL | 0000-00-00 | |
| ColDateClear | date | NO | | 0000-00-00 | |
| ColTimeStart | time | NO | | 00:00:00 | |
| ColTimeClear | time | NO | | 00:00:00 | |
| ColCallLength | time | NO | | 00:00:00 | |
| ColRingLength | time | NO | | 00:00:00 | |
| ColEffectiveFlag | smallint(1) | NO | MUL | NULL | |
| ColUnansweredFlag | smallint(1) | NO | MUL | NULL | |
| ColEngagedFlag | smallint(1) | NO | | NULL | |
| ColRecID | int(11) | NO | MUL | NULL | |
| ColCreatedUserID | int(11) | NO | | 0 | |
| ColCreatedDatetime | datetime | NO | MUL | 0000-00-00 00:00:00 | |
| ColDirection | int(1) | NO | MUL | NULL | |
+----------------------+-------------+------+-----+---------------------+----------------+
Indexes
+-------+------------+-------------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+-------------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+
| tableName | 0 | PRIMARY | 1 | ColID | A | 18031283 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_ColCustID | 1 | ColCustID | A | 1339 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_ColNumbID | 1 | ColNumbID | A | 24366 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colOrigNumber | 1 | colOrigNumber | A | 4507820 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colOrigLocality | 1 | colOrigLocality | A | 36873 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colOrigAreaCode | 1 | colOrigAreaCode | A | 696 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colTermNumber | 1 | colTermNumber | A | 137643 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colDateStart | 1 | colDateStart | A | 3639 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colEffectiveFlag | 1 | colEffectiveFlag | A | 2 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colUnansweredFlag | 1 | colUnansweredFlag | A | 2 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colEngagedFlag | 1 | colUnansweredFlag | A | 2 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colTariID | 1 | colTariID | A | 91 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_CustID_DateStart | 1 | colCustID | A | 1339 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_CustID_DateStart | 2 | colDateStart | A | 693510 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_NumbID_DateStart | 1 | colNumbID | A | 24366 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_NumbID_DateStart | 2 | colDateStart | A | 4507820 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colRecID | 1 | colRecID | A | 214658 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colCarrierRef | 1 | colCarrierRef | A | 6010427 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colCustID_colTermNumber | 1 | colCustID | A | 1339 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colCustID_colTermNumber | 2 | colTermNumber | A | 143105 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colCreatedDatetime | 1 | colCreatedDatetime | A | 474507 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colDirection | 1 | colDirection | A | 2 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colVlviID | 1 | colVlviID | A | 4133 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colSomeID | 1 | colSomeID | A | 10 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colVcamID | 1 | colVcamID | A | 7 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colVlnsID | 1 | colVlnsID | A | 18 | NULL | NULL | | BTREE | |
| tableName | 1 | idx_colCarrID | 1 | colCarrID | A | 4 | NULL | NULL | | BTREE | |
+-------+------------+-------------------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+

Answer 1

First up, you have far too may indexes. Most of those indexes are likely pointless. If you aren't using a specific index for WHERE criteria, JOINs, or ORDERing, then remove it as it just slows things down.
Next up, for your query you specify 3 columns that are always in the query comprising 1x DATE and 2x INT columns. The DATE column should be first as a date range is pretty fast on an index, and then the two INTs. This gives a starting point of the following 3-column index
CREATE INDEX searchIndex
ON tableName (ColDateStart,ColVlviID,ColVcamID)
USING BTREE;
More information on that available here : http://dev.mysql.com/doc/refman/5.0/en/multiple-column-indexes.html
I ordered the columns that way intentionally. ColDateStart for the date range filtering, followed by the other columns in decreasing order of cardinality. I selected ColVlviID as a second column as it has a cardinality of 4133 compared to ColVcamID having a cardinality of 7. This will allow MySQL to more efficiently reduce the matching rows.
Now, assuming colSomeID is the last column, I might instead do the following
CREATE INDEX searchIndex_someID
ON tableName (ColDateStart,ColVlviID,ColVcamID,ColSomeID)
USING BTREE;
This 3-column index will help MySQL get down to the applicable dataset before checking for that last ID. Optioanlly you could add a 4th column to that index at the end, if you are commonly filtering in another particular INT column.
As an aside, you may want to consider the following instead of the dateCol criteria
SELECT colID
FROM tableName
WHERE ColDateStart BETWEEN DATE('$lowerDate') AND DATE('$upperDate')
AND ColVcamID=$id1
AND ColVlviID=$id2
AND ColSomeID=$id3
The above all of course assumes that you are sanitising the variables before executing the query. I have removed the quotes from the $idx variables, as they should be numeric and therefore do not need to be entered as strings.

Answer 2

SELECT id
FROM tableName FORCE INDEX(`idx_otherId2`)
WHERE dateCol <='$lowerDate'
AND dateCol>='$upperDate'
AND otherId1='$id1'
AND otherId2='$id2'
AND otherId3='$id3';