MySql - Better Schema for large table IP pairings? - mysql

Question

I'm trying to manage some internet logs. I'm essentially capturing what IPs are reaching out to what other IPs and making reports on it.
Problem is there's a ton of chatter and I'm not sure if I can make my schema any better.
my table schema:
CREATE TABLE `IpChatter` (
`Id` bigint(20) NOT NULL AUTO_INCREMENT,
`SourceIp` bigint(20) NULL,
`DestinationIp` bigint(20) NULL,
`SourcePort` int(11) NULL,
`DestinationPort` int(11) NULL,
`FKToSomeTableWithExtraMetaDataId` bigint(20) NOT NULL,
CONSTRAINT `PK_IpChatter` PRIMARY KEY (`Id` ASC)
) ENGINE=InnoDB;
CREATE INDEX `IX_IpChatter_FKToSomeTableWithExtraMetaDataId` ON `IpChatter` (`FKToSomeTableWithExtraMetaDataId`) using HASH;
CREATE INDEX `IX_IpChatter_Main_Query_SourceIp` ON `IpChatter` (`SourceIp`);
CREATE INDEX `IX_IpChatter_Main_Query_DestinationIp` ON `IpChatter` (`DestinationIp`);
CREATE INDEX `IX_IpChatter_Main_Query_SourcePort` ON `IpChatter` (`SourcePort`);
CREATE INDEX `IX_IpChatter_Main_Query_DestinationPort` ON `IpChatter` (`DestinationPort`);
ALTER TABLE `IpChatter` ADD CONSTRAINT `FK_IpChatter_FKToSomeTableWithExtraMetaData`
FOREIGN KEY (`FKToSomeTableWithExtraMetaDataId`) REFERENCES `FKToSomeTableWithExtraMetaData` (`Id`)
ON DELETE CASCADE;
Right now I've got 2mill rows of data and pulls back data I need in about 4sec. However this is from using relatively light testing data. I'd imagine the size of the data being 30X larger in the final product. So that 4 sec will surely mean 2mins in the final product. Is there a better way I could normalize this data or have I hit a bottle neck and there isn't much I can do? Also, Are the indexes I picked ok?

Answer 1

Never mind, I figured it out. I guess I just needed to type out the problem to help me think up a solution.
So after looking at my data I've noticed a lot of pairings are repeated but under a different FKToSomeTableWithExtraMetaDataId value.
So tells me I can normalize the data by creating a table with distinct pairings of SourceIp,DestinationIp,SourcePort,DestinationPort`. Then create a lookup table to join up that table with the ToSomeTableWithExtraMetaData table.
This reduces my raw IP data by 1700%! This will give a tremendous increase in performance when searching for a range of IPs and now it has to go though far less rows. Plus with the lookup table I have greater flexibility on how I can query.
CREATE TABLE `IpChatter` (
`Id` bigint(20) NOT NULL AUTO_INCREMENT,
`SourceIp` bigint(20) NULL,
`DestinationIp` bigint(20) NULL,
`SourcePort` int(11) NULL,
`DestinationPort` int(11) NULL,
`FKToSomeLookupTableId` bigint(20) NOT NULL,
CONSTRAINT `PK_IpChatter` PRIMARY KEY (`Id` ASC)
) ENGINE=InnoDB;
CREATE INDEX `IX_IpChatter_FKToSomeLookupTableId` ON `IpChatter` (`FKToSomeLookupTableId`) using HASH;
CREATE INDEX `IX_IpChatter_Main_Query_SourceIp` ON `IpChatter` (`SourceIp`);
CREATE INDEX `IX_IpChatter_Main_Query_DestinationIp` ON `IpChatter` (`DestinationIp`);
CREATE INDEX `IX_IpChatter_Main_Query_SourcePort` ON `IpChatter` (`SourcePort`);
CREATE INDEX `IX_IpChatter_Main_Query_DestinationPort` ON `IpChatter` (`DestinationPort`);
ALTER TABLE `IpChatter` ADD CONSTRAINT `FK_IpChatter_FKToSomeLookupTable`
FOREIGN KEY (`FKToSomeLookupTableId`) REFERENCES `FKToSomeLookupTable` (`Id`)
ON DELETE CASCADE;
CREATE TABLE `FKToSomeLookupTable` (
`FKToSomeTableWithExtraMetaDataId` bigint(20) NOT NULL,
`IpChatterId` bigint(20) NOT NULL,
CONSTRAINT `PK_FKToSomeLookupTable` PRIMARY KEY (`Id` ASC)
) ENGINE=InnoDB;
CREATE INDEX `IX_IpChatter_FKToSomeTableWithExtraMetaDataId` ON `FKToSomeLookupTable` (`FKToSomeTableWithExtraMetaDataId`) using HASH;
CREATE INDEX `IX_IpChatter_IpChatterId` ON `FKToSomeLookupTable` (`IpChatterId`) using HASH;
ALTER TABLE `FKToSomeLookupTable` ADD CONSTRAINT `FK_FKToSomeLookupTable_FKToSomeTableWithExtraMetaData`
FOREIGN KEY (`FKToSomeTableWithExtraMetaDataId`) REFERENCES `FKToSomeTableWithExtraMetaData` (`Id`)
ON DELETE CASCADE;
ALTER TABLE `FKToSomeLookupTable` ADD CONSTRAINT `FK_FKToSomeLookupTable_IpChatter`
FOREIGN KEY (`IpChatterId`) REFERENCES `IpChatter` (`Id`)
ON DELETE CASCADE;

Answer 2

Shrink the table size. Smaller is one way to help (some) with the speed.
IPv4 can be packed into INT UNSIGNED, which is 4 bytes versus your current 8-byte BIGINT. IPv6, on the other hand, needs BINARY(16); what you have will not work.
Port number, I think, will fit in a 2-byte SMALLINT UNSIGNED.
Are you expecting your tables to be bigger than 4 billion rows? If not, use INT UNSIGNED instead of BIGINT for ids.
Get rid of FOREIGN KEYs, they slow down things; meanwhile, the constraints have never triggered an error, have they? Do you really use the overhead of CASCADE?
Don't index every column. Look at your queries and index the columns or combinations of columns that would benefit SELECTs, UPDATEs, and DELETEs.
Please show the queries; without them, we cannot judge the performance.