Joining multiple tables makes the query run too long - mysql

Question

I have several tables, containing (a.o.) the following fields:
tweets:
--------------------------
tweet_id ticker created_at
--------------------------
1 1 1298063318
2 1 1298053197
stocks:
---------------------------------
ticker date close volume
---------------------------------
1 1313013600 12.25 40370600
1 1312927200 11.60 37281300
wiki:
-----------------------
ticker date views
-----------------------
1 1296514800 550
1 1296601200 504
I want to compile an overview of # of tweets, close, volume and views per day (for rows identified by ticker = 1). The tweets table is leading, meaning that if there is a date on which there are no tweets, the close, volume and views for that day don't matter. In oter words, I want the output of a query to be like:
-------------------------------------
date tweets close volume views
-------------------------------------
2011-02-13 4533 12.25 40370600 550
2011-02-14 6534 11.60 53543564 340
2011-02-16 5333 13.10 56464333 664
In this example output, there were no tweets on 2011-02-15, so there is no need for the rest of the data of that day. My query thus far goes:
SELECT
DATE_FORMAT(FROM_UNIXTIME(tweets.created_at), '%Y-%m-%d') AS date,
COUNT(tweets.tweet_id) AS tweets,
stocks.close,
stocks.volume,
wiki.views
FROM tweets
LEFT JOIN stocks ON tweets.ticker = stocks.ticker
LEFT JOIN wiki ON tweets.ticker = wiki.ticker
WHERE tweets.ticker = 1
GROUP BY date
ORDER BY date ASC
Could someone verify if this query is correct? It doesn't run into any errors but it freezes my PC. Perhaps I should set an index here or there, possibly on the "ticker" columns?
[edit]
As requested, the table definitions:
CREATE TABLE `stocks` (
`ticker` int(3) NOT NULL,
`date` int(10) NOT NULL,
`open` decimal(8,2) NOT NULL,
`high` decimal(8,2) NOT NULL,
`low` decimal(8,2) NOT NULL,
`close` decimal(8,2) NOT NULL,
`volume` int(8) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE `tweets` (
`tweet_id` int(11) NOT NULL AUTO_INCREMENT,
`ticker` varchar(5) NOT NULL,
`id_str` varchar(18) NOT NULL,
`created_at` int(10) NOT NULL,
`from_user` int(11) NOT NULL,
`text` text NOT NULL,
PRIMARY KEY (`tweet_id`),
KEY `id_str` (`id_str`),
KEY `from_user` (`from_user`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
CREATE TABLE `wiki` (
`ticker` int(3) NOT NULL,
`date` int(11) NOT NULL,
`views` int(6) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
I hope this helps.

Answer 1

You are right about indices, without an index on ticker you would have to do a where-search in all tables and if they are big that's going to take alot of time.
I suggest that you turn on logging of all queries that run without index at least every now and then to find queries that if not already are slow will be slow when the data increases.
Check queries with [EXPLAIN SELECT ...][2] if you find them slow, learn how to interpret the results (not easy but important) to understand where to put new indices.

Answer 2

Do believe you should check the joins between the tables. Your query does not indicate which stocks-rows (or wiki-row) is to be matched to the date for tweets. Based on example data the match is done for all stocks and wiki-rows which have the same ticker_id.
Does stocks and wiki-tables have only one row for certain day for one ticker? Assuming this is the case, more logical query would look like this:
SELECT
DATE_FORMAT(FROM_UNIXTIME(t.created_at), '%Y-%m-%d') AS date,
COUNT(t.tweet_id) AS tweets,
s.close,
s.volume,
w.views
FROM tweets t
LEFT JOIN stocks s ON t.ticker = s.ticker
and FROM_UNIXTIME(t.created_at,'%Y-%m-%d')=FROM_UNIXTIME(s.date,'%Y-%m-%d')
LEFT JOIN wiki w ON t.ticker = w.ticker
and FROM_UNIXTIME(t.created_at,'%Y-%m-%d')=FROM_UNIXTIME(w.date,'%Y-%m-%d')
WHERE tweets.ticker = 1
GROUP BY date, s.close, s.volume, w.views
ORDER BY date ASC
If there are more than one row in stocks/wiki for certain day for one ticker, the you need to apply aggregate function to those columns as well and change the COUNT(t.tweet_id) to COUNT(distinct t.created_at).

Answer 3

I think that one of problems is date calculation
DATE_FORMAT(FROM_UNIXTIME(tweets.created_at), '%Y-%m-%d') date
Try to add this field to the tweets table to avoid CPU consumption
edit:
you can use something like this
CREATE TABLE `stocks` (
`ticker` int(3) NOT NULL,
`date` int(10) NOT NULL,
`open` decimal(8,2) NOT NULL,
`high` decimal(8,2) NOT NULL,
`low` decimal(8,2) NOT NULL,
`close` decimal(8,2) NOT NULL,
`volume` int(8) NOT NULL,
`day_date` varchar(10) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE `tweets` (
`tweet_id` int(11) NOT NULL AUTO_INCREMENT,
`ticker` varchar(5) NOT NULL,
`id_str` varchar(18) NOT NULL,
`created_at` int(10) NOT NULL,
`from_user` int(11) NOT NULL,
`text` text NOT NULL,
`day_date` varchar(10) NOT NULL,
PRIMARY KEY (`tweet_id`),
KEY `id_str` (`id_str`),
KEY `from_user` (`from_user`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
CREATE TABLE `wiki` (
`ticker` int(3) NOT NULL,
`date` int(11) NOT NULL,
`views` int(6) NOT NULL,
`day_date` varchar(10) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
SELECT
tweets.day_date AS date,
COUNT(tweets.tweet_id) AS tweets,
stocks.close as close,
stocks.volume as volume,
wiki.views as views
FROM tweets
LEFT JOIN stocks ON tweets.ticker = stocks.ticker
and tweets.day_date = stocks.day_date
LEFT JOIN wiki ON tweets.ticker = wiki.ticker
and tweets.day_date = wiki.day_date
WHERE tweets.ticker = 1
GROUP BY date, close, volume, views
ORDER BY date ASC