Mysql Group by query is taking long time - mysql

Question

I have a table "Words" in mysql database. This table contains 2 fields. word(VARCHAR(256)) and p_id(INTEGER).
Create table statement for the table:
CREATE TABLE `Words` (
`word` varchar(256) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`p_id` int(11) NOT NULL DEFAULT '0',
KEY `word_i` (`word`(255))
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
Sample entries in the table are:
+------+------+
| word | p_id |
+------+------+
| a | 1 |
| a | 2 |
| b | 1 |
| a | 4 |
+------+------+
This table contains 30+ million entries in it. I am running a group by query and it is taking 90+ minutes for running that query. The group by query I am running is:
SELECT word,group_concat(p_id) FROM Words group by word;
To optimize this problem, I sent all the data in the table into a text file using the following query.
SELECT p_id,word FROM Words INTO OUTFILE "/tmp/word_map.txt";
After that I wrote a Perl script to read all the content in the file and parse that and make a hash out of it. It took very less time compared to the Group by query(<3min).In the end hash has 14million keys(words). It is occupying a lot of memory.So Is there any way to improve the performance of Group BY query so that I don't need to go through all the above mentioned steps?
EDT: I am adding the my.cnf file entries below.
[mysqld]
datadir=/media/data/.mysql_data/mysql
tmpdir=/media/data/.mysql_tmp_data
innodb_log_file_size=5M
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
group_concat_max_len=4M
max_allowed_packet=20M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
tmpdir=/media/data/.mysql_tmp_data/
Thanks,
Vinod

Answer 1

I think the index you want is:
create index words_word_pid on words(word, pid)
This does two things. First, the group by can be handled by an index scan rather than loading the original table and sorting the results.
Secondly, this index also eliminates the need to load the original data.
My guess is that the original data does not fit into memory. So, the processing goes through the index (efficiently), finds the word, and then needs to load the pages with the word on it. Well, eventually memory fills up and the page with the word is not in memory. The page is loaded from disk. And the next page is probably not in memory, and that page is loaded from disk. And so on.
You can fix this problem by increasing the memory size. You can also fix the problem by having an index that covers all the columns used in the query.

Answer 2

The problem is that it is hardly a frequent usecase for a database to output the whole 30M rows table into a file. The advantange of your approach with the Perl script is that you do not need random disk IO. To simulate the bahaviour in MySQL you will need to load everythin into an index (p_id, word) (the whole word, not a prefix), which might turn out an overkill for the database.
You can put only p_id into an index, this will speed up grouping, but will require a lot of random disk IO to fetch words for each row.
By the way, the covering index will take ~(4+4+3*256)*30M bytes, that is more than 23Gb of memory. It seems that the solution with the Perl script is the best you can do.
Another thing you should be aware of is that you will need to get more than 20Gb of result through a MySQL connection, and that those 20 Gb of result shoul be collected into a temporary table (and sorted by p_id if you do not append ORDER BY NULL). If you are going to download if through a MySQL binding to a programming language, you will need to force the binding use streaming (by default bindings usually get the whole resultset)

Answer 3

Index the table on the word column. This will accelerate the grouping substantially as the SQL engine can locate the records for grouping with minimal searching through the table.
CREATE INDEX word_idx ON Words(word);