Mysql fulltext search and integer field query slowing down performance extreme - mysql

Question

My table consist of 600K records. The below queries have been troubling me for a while.
DDL:
CREATE TABLE `cvprofiles` (
`tid` bigint(20) NOT NULL AUTO_INCREMENT,
`partnerId` bigint(20) DEFAULT '0' COMMENT 'resume owner',
`tenant` bigint(20) DEFAULT '0' COMMENT 'sellercompany',
`lngId` int(4) DEFAULT '0',
`firstName` varbinary(150) DEFAULT '',
`lastName` varbinary(150) DEFAULT '',
`profilePicture` varchar(150) COLLATE utf8mb4_unicode_ci DEFAULT '',
`mobileIsd` varchar(20) COLLATE utf8mb4_unicode_ci DEFAULT '',
`mobileNumber` varbinary(80) DEFAULT '',
`emailId` varbinary(250) DEFAULT '',
`altEmailId` varchar(150) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`cvPath` varchar(150) COLLATE utf8mb4_unicode_ci DEFAULT '',
`cvFileName` varchar(250) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`genderCode` int(4) DEFAULT '0',
`dob` date DEFAULT NULL,
`presentLocation` varchar(350) COLLATE utf8mb4_unicode_ci DEFAULT '',
`presentLocationId` bigint(20) DEFAULT NULL,
`countryId` bigint(20) DEFAULT '0' COMMENT 'country to be maintained',
`latitude` decimal(19,15) DEFAULT '0.000000000000000',
`longitude` decimal(19,15) DEFAULT '0.000000000000000',
`totalExp` decimal(5,2) DEFAULT '0.00',
`anyKeywords` varchar(4000) COLLATE utf8mb4_unicode_ci DEFAULT '',
`cvKeywords` mediumtext COLLATE utf8mb4_unicode_ci,
`presentEmployer` varchar(400) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`noticePeriod` int(4) DEFAULT '0',
`presentSalaryCurrId` bigint(20) DEFAULT '0',
`crDate` datetime DEFAULT NULL,
`crUserId` bigint(20) DEFAULT '0',
`luDate` datetime DEFAULT NULL,
`luUserId` bigint(20) DEFAULT '0',
`skillKeywords` text COLLATE utf8mb4_unicode_ci,
`industryKeywords` varchar(2000) COLLATE utf8mb4_unicode_ci DEFAULT '',
`certiKeywords` varchar(2000) COLLATE utf8mb4_unicode_ci DEFAULT '',
`prefLocKeywords` varchar(2000) COLLATE utf8mb4_unicode_ci DEFAULT '',
`educationKeywords` varchar(2000) COLLATE utf8mb4_unicode_ci DEFAULT '',
`roleKeywords` tinytext COLLATE utf8mb4_unicode_ci,
`abilityKeywords` text COLLATE utf8mb4_unicode_ci,
`lngKeywords` text COLLATE utf8mb4_unicode_ci,
`infoKeywords` varchar(600) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'combination of fn,ln,mno,email,present employer designation',
`jobTitleId` bigint(20) DEFAULT NULL,
`source` bigint(6) DEFAULT NULL,
`portalUid` varchar(150) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
`salutationId` int(5) DEFAULT NULL,
PRIMARY KEY (`tid`),
KEY `partnerId` (`partnerId`),
KEY `crDate` (`crDate`),
KEY `idx_cvprofiles_firstName` (`firstName`),
KEY `idx_cvprofiles_mobileIsd` (`mobileIsd`),
KEY `idx_cvprofiles_mobileNumber` (`mobileNumber`),
KEY `idx_cvprofiles_emailId` (`emailId`),
KEY `idx_cvprofiles_dob` (`dob`),
KEY `luDate` (`luDate`),
KEY `idx_s_p_m_e` (`tenant`,`partnerId`,`mobileNumber`,`emailId`),
KEY `idx_s_f_l_c_s` (`tenant`,`firstName`,`lastName`,`crDate`),
KEY `sel` (`tenant`),
KEY `c_c` (`crDate`,`crUserId`),
FULLTEXT KEY `fx_cat` (`skillKeywords`,`anyKeywords`,`cvKeywords`,`infoKeywords`,`abilityKeywords`,`lngKeywords`,`roleKeywords`,`industryKeywords`,`educationKeywords`,`prefLocKeywords`)
) ;
Query1: Below fulltext query takes "0.43 sec" for word "java" :
select count(*)
from cvprofiles
where match(`skillKeywords`,`anyKeywords`,`cvKeywords`,`infoKeywords`,
`abilityKeywords`,`lngKeywords`,`roleKeywords`,
`industryKeywords`,`educationKeywords`,`prefLocKeywords`)
against ('java' in boolean mode);
Result: 168944 records
Explain:
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+
| 1 | SIMPLE | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | Select tables optimized away |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+------------------------------+
Query2:
select count(*) from cvprofiles where tenant=429;
Response time: 0.18 sec , Result : 845 records
Explain:
+----+-------------+-----------------+------------+------+-------------------------------+------+---------+-------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------------+------------+------+-------------------------------+------+---------+-------+------+----------+-------------+
| 1 | SIMPLE | icrd_resumeBank | NULL | ref | idx_s_p_m_e,idx_s_f_l_c_s,sel | sel | 9 | const | 845 | 100.00 | Using index |
Query3: Combine fulltext with integer field takes more then 45 sec+.
select count(*)
from cvprofiles
where match(`skillKeywords`,`anyKeywords`,`cvKeywords`,`infoKeywords`,
`abilityKeywords`,`lngKeywords`,`roleKeywords`,
`industryKeywords`,`educationKeywords`,`prefLocKeywords`)
against ('java' in boolean mode)
and tenant=429;
Response time: 40.12 sec, Result : 452 records
Explain:
+----+-------------+-----------------+------------+----------+--------------------------------------+--------+---------+-------+------+----------+-----------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------------+------------+----------+--------------------------------------+--------+---------+-------+------+----------+-----------------------------------+
| 1 | SIMPLE | icrd_resumeBank | NULL | fulltext | idx_s_p_m_e,idx_s_f_l_c_s,sel,fx_cat | fx_cat | 0 | const | 1 | 5.00 | Using where; Ft_hints: no_ranking |
+----+-------------+-----------------+------------+----------+--------------------------------------+--------+---------+-------+------+----------+-----------------------------------+
None of the combinations of the query are working. How to improve performance of the query of fulltext with integer fields?
Using MySQL version: 5.7
Storage Engine: InnoDB

Answer 1

The Optimizer is caught between a rock and a hard place --
whether to use the FULLTEXT index, then filter out those with the wrong tenant. (This is probably faster, and the formulation below may trick it into using it.)
Or to start by fetching on tenant first, then do the FULLTEXT tests. (Please provide SHOW CREATE TABLE cvprofiles so we can see if this is even viable.)
This kludge may make it run faster:
select count(*)
from cvprofiles
where match(`skillKeywords`,`anyKeywords`,`cvKeywords`,`infoKeywords`,
`abilityKeywords`,`lngKeywords`,`roleKeywords`,
`industryKeywords`,`educationKeywords`,`prefLocKeywords`)
against ('java' in boolean mode)
HAVING tenant=429;
The CREATE TABLE may point out flaws in the schema that led to slugishness.
On second thought, my kludge may not help in this case. It will fetch 168944 rows, then reach into the table to check that many value of tenant; this will be 168944 random lookups. The speed of that depends on disk type (SSD vs HDD) and size of innodb_buffer_pool_size. So, which disk type and what is that setting, plus how much RAM on the server?
Where I am going with the previous paragraph... If cache space is tight, it will be slow.
This looks like a case where "index merge intersect" would be useful, but the EXPLAIN says it did not try that. So,... Consider filing a bug report at bugs.mysql.com .

Answer 2

Hard to say without the EXPLAIN statement and table indexes, however most likely MySQL is making a huge temporary table to store the result of the first match and then tries the second filter.
I would recommend setting the 'tenant=429' part as a subquery instead.
select count(*) from
( SELECT `skillKeywords`,`anyKeywords`,`cvKeywords`,`infoKeywords`,`abilityKeywords`,`lngKeywords`,`roleKeywords`,`industryKeywords`,`educationKeywords`,`prefLocKeywords`
from cvprofiles
where tenant=429
) AS x
WHERE match(`skillKeywords`,`anyKeywords`,`cvKeywords`,`infoKeywords`,`abilityKeywords`,`lngKeywords`,`roleKeywords`,`industryKeywords`,`educationKeywords`,`prefLocKeywords`)
against ('java' in boolean mode) ;
Note that you will most likely need to name the columns in the subquery and access that that way.