proper index (or removal) to optimize a large data set table - mysql

Question

We have a 'visitor' tracking schema going on - that when pushed, seems to be causing some strain on the DB server.
VISITORS table identifies unique users by a HASH (current records 310,000). A search is performed on the hash, and if not found, it is added. The ID is needed for the following two tables
CREATE TABLE visitors (
id int(10) UNSIGNED NOT NULL auto_increment,
ip varchar(25) NOT NULL,
hash varchar(64) NOT NULL,
first_visit varchar(32) NOT NULL,
created_at datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
ALTER TABLE visitors ADD UNIQUE INDEX (hash);
ALTER TABLE visitors ADD INDEX (created_at);
VISITOR_VISITS table identifies when a user visited only when we can identify some referral sources (current count 142,000). A search is performed looking for the visitor_id, type and visit_date. If there is nothing found - it is added. The ID is used in the following table.
CREATE TABLE visitor_visits (
id int(10) UNSIGNED NOT NULL auto_increment,
visitor_id int(10) UNSIGNED NOT NULL,
source varchar(64) NULL DEFAULT NULL DEFAULT NULL,
medium varchar(64) NULL DEFAULT NULL,
campaign varchar(256) NULL DEFAULT NULL,
page varchar(32) NULL DEFAULT NULL,
landing varchar(32) NULL DEFAULT NULL,
type enum('fundraiser_view') NULL DEFAULT NULL,
visit_date date NOT NULL default '0000-00-00',
created_at datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
ALTER TABLE visitor_visits ADD UNIQUE INDEX (visitor_id,type,visit_date);
ALTER TABLE visitor_visits ADD CONSTRAINT FK_visits_visitor_id FOREIGN KEY (visitor_id) REFERENCES visitors(id);
PAGE_VIEWS logs individual page views (not all pages, just pages we are tracking). It can be linked to a visitor and can reference a visitor_visit (current count 2.4million -- reason it is higher is we started micro-visitor logging after logging individual pages). An insert/on duplicate query is used to add the record to this based on the view_date for the identified user. Since the ID is not needed, a pure lookup query isnt required
CREATE TABLE page_views (
id int(10) UNSIGNED NOT NULL auto_increment,
page_id int(10) UNSIGNED NOT NULL,
current_donations decimal(10,2) NOT NULL DEFAULT 0,
ip varchar(25) NOT NULL,
hash varchar(32) NOT NULL,
visitor_id int(10) UNSIGNED NULL DEFAULT NULL AFTER,
visitor_visit_id int(10) UNSIGNED NULL DEFAULT NULL AFTER,
page_views int(10) UNSIGNED NOT NULL DEFAULT 0,
widget_views int(10) UNSIGNED NOT NULL DEFAULT 0,
view_date date NOT NULL,
viewed_at datetime NOT NULL default '0000-00-00 00:00:00',
created_at datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
ALTER TABLE page_views ADD UNIQUE INDEX (page_id,view_date,visitor_id,hash);
ALTER TABLE page_views ADD INDEX (visitor_id);
ALTER TABLE page_views ADD INDEX (visitor_visit_id);
ALTER TABLE page_views ADD CONSTRAINT FK_page_views_page_id FOREIGN KEY (page_id) REFERENCES pages(id);
ALTER TABLE page_views ADD CONSTRAINT FK_page_views_visitor_id FOREIGN KEY (visitor_id) REFERENCES visitors(id);
ALTER TABLE page_views ADD CONSTRAINT FK_page_views_visit_id FOREIGN KEY (visitor_visit_id) REFERENCES visitor_visits(id);
Last week, our site got a inflow of people due to a news article, and this visitor identifying rall bottlenecked performance. I am wondering if there is an obvious optimization in there. Could it be the foreign key constraints ? Over indexing? Need for better indexing?

Answer 1

Try this ::
1) Index on varchar doesn't much improve the performance.
2) Try to partition the table, on a date range.

Answer 2

You didn't tell us what is bottlenecking your database, so I just guess it's InnoDB concurrent writes. If it isn't so and the problem is only with SELECTs (which I doubt), you should show us the exact queries. You could try to reduce the write performance hit by creating a staging table and then bulk-moving stuff from in to the main table:
CREATE TABLE page_views_tmp (
id int(10) UNSIGNED NOT NULL auto_increment,
page_id int(10) UNSIGNED NOT NULL,
current_donations decimal(10,2) NOT NULL DEFAULT 0,
ip varchar(25) NOT NULL,
hash varchar(32) NOT NULL,
visitor_id int(10) UNSIGNED NULL DEFAULT NULL AFTER,
visitor_visit_id int(10) UNSIGNED NULL DEFAULT NULL AFTER,
page_views int(10) UNSIGNED NOT NULL DEFAULT 0,
widget_views int(10) UNSIGNED NOT NULL DEFAULT 0,
view_date date NOT NULL,
viewed_at datetime NOT NULL default '0000-00-00 00:00:00',
created_at datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (id)
) ENGINE=MEMORY DEFAULT CHARSET=utf8;
And then, once per a couple of seconds or after this table has a considerable amount of rows in it:
START TRANSACTION;
INSERT INTO page_views SELECT * FROM page_views_tmp;
DELETE FROM page_views_tmp;
COMMIT;