MySQL/memSQL not using index on BETWEEN join condition - mysql

Question

We have two tables:
A dates table that contains one date per day for the last 10 and next 10 years.
A states table that has the following columns: start_date, end_date, state.
The query we run looks like this:
SELECT dates.date, COUNT(*)
FROM dates
JOIN states
ON dates.date BETWEEN states.start_date AND states.end_date
WHERE dates.date BETWEEN '2017-01-01' AND '2017-01-31'
GROUP BY dates.date
ORDER BY dates.date;
According to the query plan, memSQL isn't using an index on the JOIN condition and this makes the query slow. Is there a way we can use an index on the JOIN condition?
We tried memSQL skiplist indexes on dates.date, states.start_date, states.end_date, (states.start_date, states.end_date)
Tables & EXPLAIN:
CREATE TABLE `dates` (
`date` date DEFAULT NULL,
KEY `date_index` (`date`)
)
CREATE TABLE `states` (
`start_date` datetime DEFAULT NULL,
`end_date` datetime DEFAULT NULL,
`state` varchar(256) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,
KEY `start_date` (`start_date`),
KEY `end_date` (`end_date`),
KEY `start_date_end_date` (`start_date`,`end_date`),
)
+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| EXPLAIN |
+-----------------------------------------------------------------------------------------------------------------------------------------------------+
| GatherMerge [remote_0.date] partitions:all est_rows:96 alias:remote_0 |
| Project [r2.date, CAST(COALESCE($0,0) AS SIGNED) AS `COUNT(*)`] est_rows:96 |
| Sort [r2.date] |
| HashGroupBy [SUM(r2.`COUNT(*)`) AS $0] groups:[r2.date] |
| TableScan r2 storage:list stream:no |
| Repartition [r1.date, `COUNT(*)`] AS r2 shard_key:[date] est_rows:96 est_select_cost:26764032 |
| HashGroupBy [COUNT(*) AS `COUNT(*)`] groups:[r1.date] |
| Filter [r1.date <= states.end_date] |
| NestedLoopJoin |
| |---IndexRangeScan drstates_test.states, KEY start_date (start_date) scan:[start_date <= r1.date] est_table_rows:123904 est_filtered:123904 |
| TableScan r1 storage:list stream:no |
| Broadcast [dates.date] AS r1 distribution:tree est_rows:96 |
| IndexRangeScan drstates_test.dates, KEY date_index (date) scan:[date >= '2017-01-01' AND date <= '2017-01-31'] est_table_rows:18628 est_filtered:96 |
+-----------------------------------------------------------------------------------------------------------------------------------------------------+

Answer 1

ON dates.date BETWEEN states.start_date
AND states.end_date
is essentially un-optimizable. The only practical way to perform this test is to tediously test every row.
If you are using MySQL and don't need the dates table, consider starting with
SELECT *
FROM states
WHERE start_date >= '2017-01-01'
AND end_date < '2017-01-01' + INTERVAL 1 MONTH
Note that this works for any combination of DATE and DATETIME datatypes.
Since I am unclear on the ultimate goal, I am unclear on what to do next.