Indexing table columns - mysql

Question

I am developing an application and I create query strings from the program and pass it to a stored procedure which includes four Prepared statements. After passing the variables, the statements would be as follows,
DROP TABLE IF EXISTS tbl_correlatedData;
CREATE TABLE tbl_correlatedData
SELECT t0.*,t1.counttimeStamplocalIp,t2.countlocalPort,t3.countlocalGeo,t4.countisp,t5.countforeignip,t6.countforeignPort,t7.countforeignGeo,t8.countinfection
FROM tbl_union_threats t0
LEFT JOIN tbl_tsli t1
USING (timeStamp,localIp) LEFT JOIN tbl_tslilp t2 USING (timeStamp,localIp,localPort)
LEFT JOIN tbl_tslilplg t3
USING (timeStamp,localIp,localPort,localGeo)
LEFT JOIN tbl_tslilplgisp t4
USING (timeStamp,localIp,localPort,localGeo,isp)
LEFT JOIN tbl_tslilplgispfi t5
USING (timeStamp,localIp,localPort,localGeo,isp,foreignip)
LEFT JOIN tbl_tslilplgispfifp t6
USING (timeStamp,localIp,localPort,localGeo,isp,foreignip,foreignPort)
LEFT JOIN tbl_tslilplgispfifpfg t7
USING (timeStamp,localIp,localPort,localGeo,isp,foreignip,foreignPort,foreignGeo)
LEFT JOIN tbl_tslilplgispfifpfginf t8 USING (timeStamp,localIp,localPort,localGeo,isp,foreignip,foreignPort,foreignGeo,infection)
GROUP BY timeStamp,localIp;
ALTER TABLE tbl_correlatedData
MODIFY timeStamp VARCHAR(200) NOT NULL,
MODIFY localIp VARCHAR(200) NOT NULL,
MODIFY localPort VARCHAR(200) NOT NULL,
MODIFY localGeo VARCHAR(200) NOT NULL,
MODIFY isp VARCHAR(200) NOT NULL,
MODIFY foreignip VARCHAR(200) NOT NULL,
MODIFY foreignPort VARCHAR(200) NOT NULL,
MODIFY foreignGeo VARCHAR(200) NOT NULL,
MODIFY infection VARCHAR(200) NOT NULL;
CREATE INDEX id_index ON tbl_correlatedData (timeStamp,localIp,localPort,localGeo,isp,foreignIp,foreignPort,foreignGeo,infection);
BUT when the process gets to the indexing query, it gives out an error saying,
Incorrect key file for table 'tbl_correlateddata'; try to repair it
FYI :
I am trying this out on Windows Vista 32 bit with a free space of 19 GB on the drive with the xampp server and the table getting created shows its size as 25Mb on phpMyadmin.
EDIT:
when i try to repair it using REPAIR TABLE tbl_correlateddata gives out the following,
Table | Op | Msg_type | Msg_text
-----------------------------------------------------------------------------------------------------------------
db_threatanalysis.tbl_correlateddata | repair| Error | Table 'db_threatanalysis.tbl_correlateddata' doesn...
db_threatanalysis.tbl_correlateddata | repair| status | Operation failed
Thank you very much for the help..in advance :)

Answer 1

You are trying to create compound (multi-column) index, which is just too long for Innodb.
You have 9 columns, each VARCHAR(200), so total index width is 1800 chars. According to MySQL documentation, Innodb key is limited at 3072 chars. So, you should be ok, but, there is no guarantee that ALTER TABLE ... MODIFY ... was able to reduce all column widths to 200 or less, so even if one remained at something like 4000 chars, you will get this error.
Solution:
Reduce number of fields in your compound index.
Analyze queries which are going to query this generated table, and only create indexes that are really necessary. I would imagine most of them will be one-column indexes.
Also, it is rather strange why do you need VARCHAR(200) to store something as simple as timestamp, ip, port, etc. You can probably easily squeeze it to 10 bytes or less and call it a day.

Answer 2

Your key size might be too long. I tried something similiar on my local MySQL install. Since I dont have your tables I could not run the CREATE TABLE statement. As my database is setup for UNICODE my keys size was over 4000 bytes. MySQL InnoDB can only create indexes with a key size of 3072 bytes.
My code looked like follows:
CREATE TABLE tbl_correlatedData
(
`timeStamp` VARCHAR(200) NOT NULL,
localIp VARCHAR(200) NOT NULL,
localPort VARCHAR(200) NOT NULL,
localGeo VARCHAR(200) NOT NULL,
isp VARCHAR(200) NOT NULL,
foreignip VARCHAR(200) NOT NULL,
foreignPort VARCHAR(200) NOT NULL,
foreignGeo VARCHAR(200) NOT NULL,
infection VARCHAR(200) NOT NULL
);
CREATE INDEX id_index ON tbl_correlatedData
(timeStamp,
localIp,
localPort,
localGeo,
isp,
foreignIp,
foreignPort,
foreignGeo,
infection
);
This resulted in the error:
Error Code: 1071. Specified key was too long; max key length is 3072 bytes
Please read about size limitations here: http://dev.mysql.com/doc/refman/5.5/en/innodb-restrictions.html. I suspect you have this problem.

Answer 3

index key prefixes can be up to 767 bytes for innodb table where it will be approximately 1000 bytes for myisam table
total index length of mysql innodb is 3072
here first you just check the length of the index and if possible reduce the column size varchar(100) for all
if possible create separate indexes ( if it suits your requirement )
see the link
http://dev.mysql.com/doc/refman/5.0/en//create-index.html
http://bugs.mysql.com/bug.php?id=6604