Join on large table getting slower - mysql

Question

I have 2 innoDB tables, One has 15166897 records and an other has 700000 records, sometime when there is load on the server a simple join query based on email address on these tables took much time to execute and some time is executes quickly. However, both the tables are properly indexed.
We have around 800 tables into our database, DB size is 40GB. We are running t2.medium RDS instance in AWS having 4GB of RAM in which 3GB is allocated for INNO DB BUFFER POOL.
I have the following questions, Please suggest on it.
Is it a good idea to break the large table into small tables so that records on the one table become less, Does this will improve the query performance?
Does the large table over the time downgrade the performance of the query?
Is this RDS configuration is enough for DB, OR should need to improve the configuration?
EDIT:
Below is my table schema and index definition:
Table 1: Total Records: 7,00,000
CREATE TABLE IF NOT EXISTS `tbl_contact_master` (
`id` int(11) NOT NULL,
`first_name` varchar(60) NOT NULL,
`last_name` varchar(60) NOT NULL,
`email_address` varchar(250) NOT NULL,
`agency_name` varchar(150) NOT NULL,
`state` varchar(30) NOT NULL,
`zip_code` varchar(20) NOT NULL,
`srch_zip` varchar(20) NOT NULL,
`title` varchar(100) NOT NULL,
`street` varchar(255) NOT NULL,
`street2` varchar(255) NOT NULL,
`city` varchar(30) NOT NULL,
`country` varchar(30) NOT NULL,
`phone` varchar(20) NOT NULL,
`ext` varchar(20) NOT NULL,
`fax` varchar(20) NOT NULL,
`years_of_experience` varchar(50) DEFAULT NULL,
`owner_manager` varchar(100) NOT NULL,
`agency_type` varchar(100) NOT NULL,
`agency_sales` varchar(100) NOT NULL,
`agency_website` varchar(100) NOT NULL,
`agency_host` varchar(100) NOT NULL,
`agency_host_name` varchar(100) NOT NULL,
`agency_affiliation` varchar(100) NOT NULL,
`agent_sales` varchar(100) NOT NULL,
`id_number_type` varchar(16) NOT NULL,
`id_number` varchar(15) NOT NULL,
`destination1` varchar(100) NOT NULL,
`destination2` varchar(100) NOT NULL,
`destination3` varchar(100) NOT NULL,
`travel_type1` varchar(100) NOT NULL,
`travel_type2` varchar(100) NOT NULL,
`travel_type3` varchar(100) NOT NULL,
`travel_type4` varchar(100) NOT NULL,
`update_first_name` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_last_name` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_email_address` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agency_name` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_state` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_zip_code` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_title` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_street` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_street2` datetime NOT NULL,
`update_city` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_country` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_phone` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_ext` datetime NOT NULL,
`update_fax` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_years_of_experience` datetime NOT NULL,
`update_owner_manager` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agency_type` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agency_sales` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agency_website` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agency_host` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agency_host_name` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agency_affiliation` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_agent_sales` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_id_number_type` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_id_number` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_destination1` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_destination2` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_destination3` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_travel_type1` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_travel_type2` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_travel_type3` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`update_travel_type4` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`create_status` tinyint(1) NOT NULL COMMENT '1 for website or 0 for add by csv',
`created_date` datetime NOT NULL,
`modified_date` datetime NOT NULL,
`submited_date` datetime NOT NULL,
`FirstPageDate` datetime NOT NULL,
`is_retired` tinyint(1) NOT NULL COMMENT '0 = No, 1 = Yes',
`update_is_retired` datetime NOT NULL,
`ip_address` varchar(30) NOT NULL,
`is_bademail` tinyint(1) NOT NULL,
`update_is_bademail` datetime NOT NULL,
`is_mice` tinyint(4) NOT NULL DEFAULT '0',
`update_is_mice` datetime NOT NULL,
`mice_company_disc` mediumint(5) unsigned NOT NULL,
`update_mice_company_disc` datetime NOT NULL,
`mice_meeting_region_plan` text NOT NULL,
`update_mice_meeting_region_plan` datetime NOT NULL,
`mice_ida` text NOT NULL,
`update_mice_ida` datetime NOT NULL,
`mice_associations` text NOT NULL,
`update_mice_associations` datetime NOT NULL,
`mice_meeting_regions` text NOT NULL,
`update_mice_meeting_regions` datetime NOT NULL,
`mice_average_attendance` mediumint(5) unsigned NOT NULL,
`update_mice_average_attendance` datetime NOT NULL,
`mice_annual_budget` mediumint(5) unsigned NOT NULL,
`update_mice_annual_budget` datetime NOT NULL,
`mice_facilities` text NOT NULL,
`update_mice_facilities` datetime NOT NULL,
`mice_annual_peak_rooms` mediumint(5) unsigned NOT NULL,
`update_mice_annual_peak_rooms` datetime NOT NULL,
`mice_nof_meetings` mediumint(5) unsigned NOT NULL,
`update_mice_nof_meetings` datetime NOT NULL,
`mice_job_resp_level` mediumint(5) unsigned NOT NULL,
`update_mice_job_resp_level` datetime NOT NULL,
`mice_event_resp_level` mediumint(5) unsigned NOT NULL,
`update_mice_event_resp_level` datetime NOT NULL,
`mice_is_planning` mediumint(5) unsigned NOT NULL,
`update_mice_is_planning` datetime NOT NULL,
`mice_experience` mediumint(5) unsigned NOT NULL,
`update_mice_experience` datetime NOT NULL,
`mice_primary_job` mediumint(5) unsigned NOT NULL,
`update_mice_primary_job` datetime NOT NULL,
`mice_company_size` mediumint(5) unsigned NOT NULL,
`update_mice_company_size` datetime NOT NULL,
`mice_primary_business` mediumint(5) unsigned NOT NULL,
`update_mice_primary_business` datetime NOT NULL,
`mice_primary_business_other` text NOT NULL,
`mice_event_specialty` text NOT NULL,
`update_mice_event_specialty` datetime NOT NULL,
`mice_created_date` datetime NOT NULL,
`mice_modified_date` datetime NOT NULL,
`mice_submitted_date` datetime NOT NULL,
`is_ta` tinyint(4) NOT NULL DEFAULT '0',
`update_is_ta` datetime NOT NULL,
`ta_created_date` datetime NOT NULL,
`ta_modified_date` datetime NOT NULL,
`ta_submitted_date` datetime NOT NULL,
`is_mice_retired` tinyint(4) NOT NULL DEFAULT '0',
`update_is_mice_retired` datetime NOT NULL,
`not_ta_by` tinyint(2) NOT NULL COMMENT '0 for default, 1 for superadmin, 2 for contact',
`update_not_ta_by` datetime NOT NULL,
`not_mice_by` tinyint(2) NOT NULL COMMENT '0 for default, 1 for superadmin, 2 for contact',
`update_not_mice_by` datetime NOT NULL
) ENGINE=InnoDB AUTO_INCREMENT=1096438 DEFAULT CHARSET=latin1;
--
-- Indexes for dumped tables
--
--
-- Indexes for table `tbl_contact_master`
--
ALTER TABLE `tbl_contact_master`
ADD PRIMARY KEY (`id`),
ADD KEY `email_address` (`email_address`),
ADD KEY `is_ta` (`is_ta`),
ADD KEY `mice_created_date` (`mice_created_date`),
ADD KEY `mice_modified_date` (`mice_modified_date`),
ADD KEY `mice_submitted_date` (`mice_submitted_date`),
ADD KEY `ta_created_date` (`ta_created_date`),
ADD KEY `ta_modified_date` (`ta_modified_date`),
ADD KEY `ta_submitted_date` (`ta_submitted_date`),
ADD KEY `is_mice_retired` (`is_mice_retired`),
ADD KEY `is_mice` (`is_mice`),
ADD KEY `is_bademail` (`is_bademail`),
ADD KEY `is_retired` (`is_retired`),
ADD KEY `created_date` (`created_date`),
ADD KEY `modified_date` (`modified_date`),
ADD KEY `submited_date` (`submited_date`),
ADD KEY `srch_zip` (`srch_zip`);
Table 2:
CREATE TABLE IF NOT EXISTS `tbl_master_partition` (
`contact_id` int(10) unsigned NOT NULL,
`inactive_group` varchar(200) DEFAULT NULL COMMENT '1=15,2=30,3=45,4=60',
`inactive_from` datetime DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
--
-- Indexes for dumped tables
--
--
-- Indexes for table `tbl_master_partition`
--
ALTER TABLE `tbl_master_partition`
ADD KEY `contact_id` (`contact_id`), ADD KEY `inactive_group` (`inactive_group`);
Table 3: Total Records: 14400000
CREATE TABLE IF NOT EXISTS `tbl_mandrill_email_mapping` (
`id` bigint(20) NOT NULL,
`log_id` int(11) NOT NULL,
`batch_id` varchar(30) COLLATE utf8_unicode_ci NOT NULL,
`m_id` varchar(50) COLLATE utf8_unicode_ci NOT NULL COMMENT 'unique mandrill message id',
`s_id` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`email` varchar(250) COLLATE utf8_unicode_ci NOT NULL,
`open` int(11) NOT NULL DEFAULT '0',
`click` int(11) NOT NULL DEFAULT '0',
`send` int(11) NOT NULL DEFAULT '0',
`hard_bounce` int(11) NOT NULL DEFAULT '0',
`soft_bounce` int(11) NOT NULL DEFAULT '0',
`reject` int(11) NOT NULL DEFAULT '0',
`spam` int(11) NOT NULL DEFAULT '0',
`unsub` int(11) NOT NULL DEFAULT '0'
) ENGINE=InnoDB AUTO_INCREMENT=15131814 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
--
-- Indexes for dumped tables
--
--
-- Indexes for table `tbl_mandrill_email_mapping`
--
ALTER TABLE `tbl_mandrill_email_mapping`
ADD PRIMARY KEY (`id`),
ADD KEY `m_id` (`m_id`),
ADD KEY `email` (`email`),
ADD KEY `hard_bounce` (`hard_bounce`),
ADD KEY `reject` (`reject`),
ADD KEY `open` (`open`),
ADD KEY `click` (`click`),
ADD KEY `send` (`send`),
ADD KEY `log_id` (`log_id`),
ADD KEY `batch_id` (`batch_id`),
ADD KEY `s_id` (`s_id`);
Missing TABLE added
CREATE TABLE IF NOT EXISTS `tbl_contact_173_230` (
`id` int(11) NOT NULL,
`contact_id` int(11) NOT NULL,
`salutation` varchar(10) NOT NULL,
`first_name` varchar(60) NOT NULL,
`last_name` varchar(60) NOT NULL,
`email_address` varchar(250) NOT NULL,
`secondary_email_address` varchar(250) NOT NULL,
`agency_name` varchar(150) NOT NULL,
`state` varchar(30) NOT NULL,
`zip_code` varchar(20) NOT NULL,
`srch_zip` varchar(20) NOT NULL,
`title` varchar(255) NOT NULL,
`street` varchar(255) NOT NULL,
`street2` varchar(255) NOT NULL,
`city` varchar(30) NOT NULL,
`country` varchar(30) NOT NULL,
`phone` varchar(20) NOT NULL,
`ext` varchar(20) NOT NULL,
`fax` varchar(20) NOT NULL,
`owner_manager` varchar(140) DEFAULT NULL,
`years_of_experience` varchar(40) DEFAULT NULL,
`agency_type` varchar(140) NOT NULL,
`agency_sales` varchar(140) NOT NULL,
`agency_website` varchar(140) NOT NULL,
`agency_host` varchar(140) DEFAULT NULL,
`agency_host_name` varchar(140) NOT NULL,
`agency_affiliation` varchar(140) NOT NULL,
`agent_sales` varchar(140) NOT NULL,
`id_number_type` varchar(16) NOT NULL,
`id_number` varchar(15) NOT NULL,
`destination1` varchar(140) NOT NULL,
`destination2` varchar(140) NOT NULL,
`destination3` varchar(140) NOT NULL,
`travel_type1` varchar(140) NOT NULL,
`travel_type2` varchar(140) NOT NULL,
`travel_type3` varchar(140) NOT NULL,
`travel_type4` varchar(140) NOT NULL,
`update_first_name` datetime NOT NULL,
`update_last_name` datetime NOT NULL,
`update_email_address` datetime NOT NULL,
`update_agency_name` datetime NOT NULL,
`update_state` datetime NOT NULL,
`update_zip_code` datetime NOT NULL,
`update_title` datetime NOT NULL,
`update_street` datetime NOT NULL,
`update_street2` datetime NOT NULL,
`update_city` datetime NOT NULL,
`update_country` datetime NOT NULL,
`update_phone` datetime NOT NULL,
`update_ext` datetime NOT NULL,
`update_fax` datetime NOT NULL,
`update_owner_manager` datetime NOT NULL,
`update_years_of_experience` datetime NOT NULL,
`update_agency_type` datetime NOT NULL,
`update_agency_sales` datetime NOT NULL,
`update_agency_website` datetime NOT NULL,
`update_agency_host` datetime NOT NULL,
`update_agency_host_name` datetime NOT NULL,
`update_agency_affiliation` datetime NOT NULL,
`update_agent_sales` datetime NOT NULL,
`update_id_number_type` datetime NOT NULL,
`update_id_number` datetime NOT NULL,
`update_destination1` datetime NOT NULL,
`update_destination2` datetime NOT NULL,
`update_destination3` datetime NOT NULL,
`update_travel_type1` datetime NOT NULL,
`update_travel_type2` datetime NOT NULL,
`update_travel_type3` datetime NOT NULL,
`update_travel_type4` datetime NOT NULL,
`opt_in_marketing` varchar(10) DEFAULT NULL,
`travel_pro_user` varchar(10) DEFAULT NULL,
`create_status` tinyint(1) NOT NULL,
`crm_created_date` datetime NOT NULL,
`crm_modified_date` datetime NOT NULL,
`crm_submited_date` datetime NOT NULL,
`crm_sync_date` datetime NOT NULL,
`created_date` datetime NOT NULL,
`modified_date` datetime NOT NULL,
`submited_date` datetime NOT NULL,
`offer_entry` datetime NOT NULL,
`sync_date` datetime NOT NULL,
`crm_status` tinyint(1) NOT NULL,
`created_by` int(11) NOT NULL,
`is_retired` tinyint(1) NOT NULL,
`update_is_retired` datetime NOT NULL,
`is_bademail` tinyint(1) NOT NULL,
`update_is_bademail` datetime NOT NULL,
`is_ta` tinyint(4) NOT NULL DEFAULT '0',
`update_is_ta` datetime NOT NULL,
`ta_created_date` datetime NOT NULL,
`ta_modified_date` datetime NOT NULL,
`ta_submitted_date` datetime NOT NULL,
`mice_company_disc` text NOT NULL,
`update_mice_company_disc` datetime NOT NULL,
`mice_meeting_region_plan` text NOT NULL,
`update_mice_meeting_region_plan` datetime NOT NULL,
`mice_ida` text NOT NULL,
`update_mice_ida` datetime NOT NULL,
`mice_associations` text NOT NULL,
`update_mice_associations` datetime NOT NULL,
`mice_meeting_regions` text NOT NULL,
`update_mice_meeting_regions` datetime NOT NULL,
`mice_average_attendance` mediumint(5) unsigned NOT NULL,
`update_mice_average_attendance` datetime NOT NULL,
`mice_annual_budget` mediumint(5) unsigned NOT NULL,
`update_mice_annual_budget` datetime NOT NULL,
`mice_facilities` text NOT NULL,
`update_mice_facilities` datetime NOT NULL,
`mice_annual_peak_rooms` mediumint(5) unsigned NOT NULL,
`update_mice_annual_peak_rooms` datetime NOT NULL,
`mice_nof_meetings` mediumint(5) unsigned NOT NULL,
`update_mice_nof_meetings` datetime NOT NULL,
`mice_job_resp_level` mediumint(5) unsigned NOT NULL,
`update_mice_job_resp_level` datetime NOT NULL,
`mice_event_resp_level` mediumint(5) unsigned NOT NULL,
`update_mice_event_resp_level` datetime NOT NULL,
`mice_is_planning` mediumint(5) unsigned NOT NULL,
`update_mice_is_planning` datetime NOT NULL,
`mice_experience` mediumint(5) unsigned NOT NULL,
`update_mice_experience` datetime NOT NULL,
`mice_primary_job` mediumint(5) unsigned NOT NULL,
`update_mice_primary_job` datetime NOT NULL,
`mice_company_size` mediumint(5) unsigned NOT NULL,
`update_mice_company_size` datetime NOT NULL,
`mice_primary_business` mediumint(5) unsigned NOT NULL,
`mice_primary_business_other` text NOT NULL,
`update_mice_primary_business` datetime NOT NULL,
`mice_event_specialty` text NOT NULL,
`update_mice_event_specialty` datetime NOT NULL,
`is_mice` tinyint(4) NOT NULL DEFAULT '0',
`update_is_mice` datetime NOT NULL,
`mice_created_date` datetime NOT NULL,
`mice_modified_date` datetime NOT NULL,
`mice_submitted_date` datetime NOT NULL,
`is_mice_retired` tinyint(4) NOT NULL DEFAULT '0',
`update_is_mice_retired` datetime NOT NULL,
`recurring_opt_out` tinyint(1) unsigned NOT NULL DEFAULT '0',
`custom_648` varchar(255) NOT NULL,
`custom_649` varchar(255) NOT NULL,
`custom_650` varchar(255) NOT NULL,
`custom_717` varchar(255) NOT NULL,
`custom_718` varchar(255) NOT NULL,
`custom_719` varchar(255) NOT NULL,
`custom_1369` varchar(255) NOT NULL,
`custom_1454` varchar(255) NOT NULL,
`custom_1455` varchar(255) NOT NULL,
`custom_1469` varchar(255) NOT NULL
) ENGINE=InnoDB AUTO_INCREMENT=103603 DEFAULT CHARSET=latin1;
--
-- Indexes for table `tbl_contact_173_230`
--
ALTER TABLE `tbl_contact_173_230`
ADD PRIMARY KEY (`id`),
ADD KEY `email_address` (`email_address`),
ADD KEY `contact_id` (`contact_id`),
ADD KEY `zip_code` (`zip_code`),
ADD KEY `srch_zip` (`srch_zip`),
ADD KEY `ta_submitted_date` (`ta_submitted_date`);
Below are the queries which getting slower or sometime just get stuck for an hrs.
1.
SELECT cm.id AS contact_id, mem.email, mem.open
FROM tbl_mandrill_email_mapping AS mem
JOIN tbl_contact_master AS cm ON cm.email_address = mem.email
WHERE mem.log_id = XXXXX
2.
SELECT COUNT(crm.id) as total_record, SUM(UPPER(opt_in_marketing)='NO') as unsub_email_count,
SUM(is_bademail=1) as invalid_email_count, COUNT(DISTINCT mp.`contact_id`) as inactive_email_count,
SUM(is_retired=1
OR is_mice_retired=1
) as retired_count
FROM (`tbl_contact_173_230` AS crm)
INNER JOIN `tbl_mandrill_email_mapping` as mem
ON `mem`.`email` = `crm`.`email_address`
LEFT JOIN`tbl_master_partition` AS mp
ON `mp`.`contact_id` = `crm`.`contact_id`
AND crm.`ta_submitted_date` < '2016-10-19'
AND FIND_IN_SET(5, mp.`inactive_group`) > 0
WHERE `crm`.`email_address` != '' AND`mem`.`log_id` = 'xxxx'
AND `mem`.`open` = 0
AND `mem`.`hard_bounce` = 0
AND `mem`.`reject` = 0
AND `mem`.`spam` = 0
ORDER BY `crm`.`id` ASC
Edit Note:
I have notice that, it mostly creates the issue when there is very frequent update/write on tbl_mandrill_email_mapping table. It's a log table which handles the email event's(sent,open,click,bounce etc.. log). In my case this table gets very busy once any bulk email sent out(approx 1,00,000 email) and it start handling all above events, sometimes this events are very frequent, In this case all the queries which reference the tbl_mandrill_email_mapping get stuck.
Any suggestion to handle this kind of situation? I am thinking about read replica so that all update/insert will handle by different server and read handle by different server, Will it work? Or is there any batter solution to handle this situation?
I am open for any other suggestion, Please suggest.

Answer 1

No! Do not break big tables into smaller ones. You already may have "too many tables". Instead, focus on indexing. Let's see SHOW CREATE TABLE and some of the important queries.
A table growing in size will slow down queries that fail to make good use of indexes. Not so for good indexes. (There are exceptions; let's see your queries.)
You have only 4GB? And you are shoehorning a 3GB buffer_pool into it? That does not leave much room for other structure, nor for other applications. This can lead to swapping, which is terrible for performance. Drop innodb_buffer_pool_size to 1500M.
Increasing the 'machine' size (and buffer_pool) may or may not help. The buffer_pool is a "cache", so the 40GB is not totally relevant. What we need is the "working set size", for which there is no good metric.
Do you know if you are I/O-bound? Or CPU-bound?
Based on the title, I recommend discussing the query.
More
(Now looking at Schema)
More indexes --> slower loading. Normally, a few good indexes is well worth the extra load time. But the emphasis is on few. Do not index flags; they will almost never be useful. (I assume your is_* columns are flags?)
Does your app really care when each column was updated? I see apps that start with such, but eventually abandon the idea.
Be careful not to mix CHARACTER SETs or COLLATIONs when JOINing. That is, in FROM a JOIN b ON a.x = b.y if x and y are no the same CHARACTER SET and COLLATION, then no index will be used.
Query 1 would benefit from this composite (and covering) index: INDEX(log_id, email, open).
I don't see the CREATE TABLE tbl_contact_173_230??