MySQL some queries never exectue - mysql

Question

I am in the middle of my first data science project and I am having some trouble with extremely slow queries using MySQL Workbench.
These are my 3 tables (each of which comes from data sets from various websites that have been cleaned up and inserted into MySQL):
CREATE TABLE IF NOT EXISTS `starbucks` (
`STORE_NUMBER` varchar(20) NOT NULL,
`CITY` varchar(50) NOT NULL,
`STATE` char(2) NOT NULL,
`ZIPCODE` char(5) NOT NULL,
`LONG` varchar(10) NOT NULL,
`LAT` varchar(10) NOT NULL,
PRIMARY KEY (`STORE_NUMBER`)
)ENGINE=InnoDB")
CREATE TABLE IF NOT EXISTS `income`(
`STATEFIPS` char(2) NOT NULL,
`STATE` char(2) NOT NULL,
`ZIPCODE` char(5) NOT NULL,
`AGI_STUB` tinyint NOT NULL,
`NUM_RETURNS` float(15,4) NOT NULL,
`TOTAL_INCOME` float(15,4) NOT NULL,
PRIMARY KEY (`STATE`, `ZIPCODE`, `AGI_STUB`)
)ENGINE=InnoDB")
CREATE TABLE IF NOT EXISTS `diversity`(
`COUNTY` varchar(50) NOT NULL,
`STATE` char(2) NOT NULL,
`INDEX` float(7,6) NOT NULL,
`1` float(3,1) NOT NULL,
`2` float(3,1) NOT NULL,
`3` float(3,1) NOT NULL,
`4` float(3,1) NOT NULL,
`5` float(3,1) NOT NULL,
`6` float(3,1) NOT NULL,
`7` float(3,1) NOT NULL,
PRIMARY KEY (`COUNTY`, `STATE`)
)ENGINE=InnoDB")
starbucks has 13,608 records,
income has 166,740 records,
diversity has 3,143 records.
The query I am trying to run:
SELECT i.TOTAL_INCOME,
CASE
WHEN s.STORE_NUMBER IS NOT NULL THEN 1
ELSE 0
END AS has_starbucks
FROM income as i
LEFT OUTER JOIN starbucks as s
ON i.ZIPCODE = s.ZIPCODE
If I limit the result to 1,000 rows it will run fast, however I need to get ALL of the records (no row limit) and this is resulting in the query never returning, and eventually timing out and disconnecting me from the MySQL server.
In the past when working for companies on their databases with MILLIONS of records in them I have never had this much trouble.
What table optimizations do I need to do to fix this? What MySQL settings do I need to change? Any other suggestions welcome.
EDIT
It appears that the 'Duration' of the query never exceeds 0.500 seconds, it is the 'Fetch' part that lasts > 120 seconds. I am not sure if this is useful information.

Answer 1

The first issue is create a proper index on join columns
CREATE INDEX idx1 ON starbucks (ZIPCODE );
CREATE INDEX idx2 ON income (ZIPCODE );
or a verbous index adding the column you select
CREATE INDEX idx2 ON income (ZIPCODE , TOTAL_INCOME);
and check the behavior using explain plan

Answer 2

This won't do much to fix your performance issue but will fix your duplicate row issue -
SELECT i.TOTAL_INCOME, 1 AS has_starbucks
FROM income as i
WHERE i.Zipcode in (Select zipcode from Starbucks)
UNION
SELECT i.TOTAL_INCOME, 0 AS has_starbucks
FROM income as i
WHERE i.Zipcode not in (Select zipcode from Starbucks)
EXISTS is sometimes more efficient than IN
SELECT i.TOTAL_INCOME, 1 AS has_starbucks
FROM income as i
WHERE EXISTS
( SELECT 1
FROM Starbucks s
WHERE s.zipcode = i.Zipcode
)
UNION
SELECT i.TOTAL_INCOME, 0 AS has_starbucks
FROM income as i
WHERE NOT EXISTS
( SELECT 1
FROM Starbucks s
WHERE s.zipcode = i.Zipcode
)