Related
I have a new centos7 server with mariadb 10.8.4, 32GB ram, AMD Ryzen 5 PRO 3600 6-Core Processor, 12 cores,
The old server is Centos7 with mariadb 5.5.68
My development machine is 2017 macbook pro
The following query runs in 0.17 seconds on my dev machine, 0.56 seconds on the old server, and 29.8 seconds on the new server!
I've tried all I can google to tune the conf of the new server and nothing improves matters.
I have tried using the conf of the old server on the new one, still no change.
Here is the conf, the commented lines are the new server conf, the others are the old..:
[mysqld]
#max_allowed_packet = 512M
#query_cache_size = 32M
#thread_cache_size = 8
#myisam_sort_buffer_size = 64M
#read_rnd_buffer_size = 8M
#read_buffer_size = 2M
#sort_buffer_size = 8M
#table_open_cache = 1512
#key_buffer_size = 384M
#sql_mode=NO_ENGINE_SUBSTITUTION
query_cache_size = 32M
query_cache_type=1
query_cache_limit=1048576
sort_buffer_size = 8M
table_open_cache = 500
key_buffer_size = 32M
max_connections = 500
big-tables
log_error=/var/log/mariadb/mariadb.err
And the query:
SELECT cornishestabs.id, cornishestabs.name, townid, cornishestabs.postcode,
cornishestabs.latitude, cornishestabs.longitude, cornishestabs.image,
cornishestabs.web, cornishtowns.Name AS Town, cornishtowns.bloglink,
cornishtowns.blogtitle, types.title,
SQRT( POW( 69.1 * ( cornishestabs.latitude - '50.16304' ) , 2 ) + POW( 69.1 * ( '-5.3761' - cornishestabs.longitude ) * COS( cornishestabs.latitude / 57.3 ) , 2 ) ) AS distance
FROM cornishestabs
INNER JOIN cornishtowns ON cornishtowns.ID = cornishestabs.townid
INNER JOIN types ON types.id = cornishestabs.etype
WHERE cornishestabs.display = 1
AND cornishestabs.latitude > 0
AND cornishestabs.id != '60959'
GROUP BY name
ORDER BY distance LIMIT 10;
I am honestly at a loss as to how to proceed, I know there is some math in the query but it has never been slow before!
Thanks for the response, here are results from the 2 servers as requested:
EXPLAIN command for New Server:
+------+-------------+---------------+------+--------------------------------------+--------+---------+--------------------------------+------+------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+---------------+------+--------------------------------------+--------+---------+--------------------------------+------+------------------------------------+
| 1 | SIMPLE | cornishtowns | ALL | PRIMARY | NULL | NULL | NULL | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | types | ALL | PRIMARY | NULL | NULL | NULL | 1 | Using join buffer (flat, BNL join) |
| 1 | SIMPLE | cornishestabs | ref | PRIMARY,display,type,townid,latitude | townid | 2 | cornish_breaks.cornishtowns.ID | 88 | Using index condition; Using where |
+------+-------------+---------------+------+--------------------------------------+--------+---------+--------------------------------+------+------------------------------------+
EXPLAIN command for Old Server:
+------+-------------+---------------+--------+--------------------------------------+---------+---------+--------------------------------------+------+---------------------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+---------------+--------+--------------------------------------+---------+---------+--------------------------------------+------+---------------------------------------------------------------------+
| 1 | SIMPLE | cornishestabs | ref | PRIMARY,display,type,townid,latitude | display | 1 | const | 5083 | Using index condition; Using where; Using temporary; Using filesort |
| 1 | SIMPLE | cornishtowns | eq_ref | PRIMARY | PRIMARY | 2 | stayin_cornwall.cornishestabs.townid | 1 | |
| 1 | SIMPLE | types | eq_ref | PRIMARY | PRIMARY | 4 | stayin_cornwall.cornishestabs.etype | 1 | Using where |
+------+-------------+---------------+--------+--------------------------------------+---------+---------+--------------------------------------+------+---------------------------------------------------------------------+
And here are the SHOW CREATE results from the 2 servers.
New Server:
SHOW CREATE TABLE cornishestabs
| cornishestabs | CREATE TABLE `cornishestabs` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`associd` varchar(16) DEFAULT NULL,
`name` varchar(96) NOT NULL DEFAULT '',
`address` text NOT NULL,
`townid` smallint(4) unsigned NOT NULL,
`postcode` varchar(34) NOT NULL DEFAULT '',
`descr` text NOT NULL,
`ourdescr` text DEFAULT NULL,
`rating` varchar(96) NOT NULL DEFAULT '',
`etype` smallint(5) unsigned NOT NULL DEFAULT 0,
`etype2` smallint(5) unsigned NOT NULL DEFAULT 0,
`etype3` smallint(5) unsigned NOT NULL DEFAULT 0,
`etype4` smallint(5) unsigned NOT NULL,
`tel` varchar(64) NOT NULL DEFAULT '',
`email` varchar(76) NOT NULL DEFAULT '',
`web` varchar(255) NOT NULL,
`online` varchar(255) NOT NULL DEFAULT '',
`assoc` smallint(5) unsigned NOT NULL DEFAULT 0,
`date` date DEFAULT NULL,
`image` varchar(255) NOT NULL,
`moreimgs` text NOT NULL,
`latitude` decimal(18,15) DEFAULT NULL,
`longitude` decimal(18,15) DEFAULT NULL,
`northing` int(11) unsigned DEFAULT NULL,
`easting` int(11) unsigned DEFAULT NULL,
`sleeps` tinyint(2) unsigned NOT NULL DEFAULT 0,
`pets` tinyint(1) unsigned NOT NULL DEFAULT 0,
`price` varchar(16) NOT NULL DEFAULT '0',
`facilities` text NOT NULL,
`brochure` tinyint(1) unsigned NOT NULL DEFAULT 0,
`display` tinyint(1) unsigned NOT NULL DEFAULT 0,
`reviewnumber` tinyint(3) unsigned NOT NULL,
`reviews` text NOT NULL,
PRIMARY KEY (`id`),
KEY `display` (`display`),
KEY `type` (`etype`),
KEY `etype2` (`etype2`),
KEY `etype3` (`etype3`),
KEY `etype4` (`etype4`),
KEY `associd` (`associd`),
KEY `postcode` (`postcode`),
KEY `assoc` (`assoc`),
KEY `townid` (`townid`),
KEY `latitude` (`latitude`),
KEY `longitude` (`longitude`)
) ENGINE=InnoDB AUTO_INCREMENT=73642 DEFAULT CHARSET=latin1
cornishtowns | CREATE TABLE `cornishtowns` (
`ID` smallint(5) unsigned NOT NULL AUTO_INCREMENT,
`Name` varchar(255) DEFAULT NULL,
`Description` text DEFAULT NULL,
`telcode` varchar(64) NOT NULL,
`postcode` varchar(10) NOT NULL,
`population` varchar(8) DEFAULT NULL,
`latitude` decimal(18,15) DEFAULT NULL,
`longitude` decimal(18,15) DEFAULT NULL,
`northing` int(11) unsigned DEFAULT NULL,
`easting` int(11) unsigned DEFAULT NULL,
`NationalPark` tinyint(2) unsigned NOT NULL,
`village` tinyint(1) unsigned NOT NULL DEFAULT 0,
`area` tinyint(1) unsigned NOT NULL,
`bloglink` varchar(96) NOT NULL,
`blogtitle` varchar(48) NOT NULL,
PRIMARY KEY (`ID`),
KEY `Name` (`Name`),
KEY `NationalPark` (`NationalPark`),
KEY `northing` (`northing`),
KEY `easting` (`easting`),
KEY `area` (`area`)
) ENGINE=InnoDB AUTO_INCREMENT=4675 DEFAULT CHARSET=latin1
types | CREATE TABLE `types` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`link` varchar(32) NOT NULL DEFAULT '',
`title` varchar(96) NOT NULL DEFAULT '',
`typedescription` text NOT NULL,
`shortname` varchar(24) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=18 DEFAULT CHARSET=latin1
Old Server:
cornishestabs | CREATE TABLE `cornishestabs` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`associd` varchar(16) DEFAULT NULL,
`name` varchar(96) NOT NULL DEFAULT '',
`address` text NOT NULL,
`townid` smallint(4) unsigned NOT NULL,
`postcode` varchar(34) NOT NULL DEFAULT '',
`descr` text NOT NULL,
`ourdescr` text,
`rating` varchar(96) NOT NULL DEFAULT '',
`etype` smallint(5) unsigned NOT NULL DEFAULT '0',
`etype2` smallint(5) unsigned NOT NULL DEFAULT '0',
`etype3` smallint(5) unsigned NOT NULL DEFAULT '0',
`etype4` smallint(5) unsigned NOT NULL,
`tel` varchar(64) NOT NULL DEFAULT '',
`email` varchar(76) NOT NULL DEFAULT '',
`web` varchar(255) NOT NULL,
`online` varchar(255) NOT NULL DEFAULT '',
`assoc` smallint(5) unsigned NOT NULL DEFAULT '0',
`date` date DEFAULT NULL,
`image` varchar(255) NOT NULL,
`moreimgs` text NOT NULL,
`latitude` decimal(18,15) DEFAULT NULL,
`longitude` decimal(18,15) DEFAULT NULL,
`northing` int(11) unsigned DEFAULT NULL,
`easting` int(11) unsigned DEFAULT NULL,
`sleeps` tinyint(2) unsigned NOT NULL DEFAULT '0',
`pets` tinyint(1) unsigned NOT NULL DEFAULT '0',
`price` varchar(16) NOT NULL DEFAULT '0',
`facilities` text NOT NULL,
`brochure` tinyint(1) unsigned NOT NULL DEFAULT '0',
`display` tinyint(1) unsigned NOT NULL DEFAULT '0',
`reviewnumber` tinyint(3) unsigned NOT NULL,
`reviews` text NOT NULL,
PRIMARY KEY (`id`),
KEY `display` (`display`),
KEY `type` (`etype`),
KEY `etype2` (`etype2`),
KEY `etype3` (`etype3`),
KEY `etype4` (`etype4`),
KEY `associd` (`associd`),
KEY `postcode` (`postcode`),
KEY `assoc` (`assoc`),
KEY `townid` (`townid`),
KEY `latitude` (`latitude`),
KEY `longitude` (`longitude`)
) ENGINE=InnoDB AUTO_INCREMENT=73642 DEFAULT CHARSET=latin1
cornishtowns | CREATE TABLE `cornishtowns` (
`ID` smallint(5) unsigned NOT NULL AUTO_INCREMENT,
`Name` varchar(255) DEFAULT NULL,
`Description` text,
`telcode` varchar(64) NOT NULL,
`postcode` varchar(10) NOT NULL,
`population` varchar(8) DEFAULT NULL,
`latitude` decimal(18,15) DEFAULT NULL,
`longitude` decimal(18,15) DEFAULT NULL,
`northing` int(11) unsigned DEFAULT NULL,
`easting` int(11) unsigned DEFAULT NULL,
`NationalPark` tinyint(2) unsigned NOT NULL,
`village` tinyint(1) unsigned NOT NULL DEFAULT '0',
`area` tinyint(1) unsigned NOT NULL,
`bloglink` varchar(96) NOT NULL,
`blogtitle` varchar(48) NOT NULL,
PRIMARY KEY (`ID`),
KEY `Name` (`Name`),
KEY `NationalPark` (`NationalPark`),
KEY `northing` (`northing`),
KEY `easting` (`easting`),
KEY `area` (`area`)
) ENGINE=InnoDB AUTO_INCREMENT=4675 DEFAULT CHARSET=latin1
types | CREATE TABLE `types` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`link` varchar(32) NOT NULL DEFAULT '',
`title` varchar(96) NOT NULL DEFAULT '',
`typedescription` text NOT NULL,
`shortname` varchar(24) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=18 DEFAULT CHARSET=latin1
I hope this helps!
To answer the "why" of the timing differences, please provide EXPLAIN SELECT ... for each server.
To improve the speed of "find the nearest", see Find Nearest
As for configuration -- One cannot [usually] use configuration to significantly improve performance. In particular, none of the listed items (even the commented-out ones) could cause that level of speed difference. What is the value of `innodb_buffer_pool_size? How much RAM do you have? Usually, it is better to turn off the Query cache.
Nor do the hardware configurations justify such a huge difference.
I have this query and takes 40 seconds, Is there a way to speed up? thank you
SELECT *, last.Date
FROM constant.derogation der
LEFT JOIN variable.last last
ON der.code = last.code
WHERE 1 = 1
AND status != 'removed'
ORDER BY status;
+------+-------------+-------+------+---------------+------+---------+------+--------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+-------+------+---------------+------+---------+------+--------+----------------------------------------------+
| 1 | SIMPLE | der | ALL | NULL | NULL | NULL | NULL | 318 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | last | ALL | NULL | NULL | NULL | NULL | 250950 | Using where |
+------+-------------+-------+------+---------------+------+---------+------+--------+----------------------------------------------+
This is the structure of both tables, both databases are on the same server.
I will only get a value from Last Table
DLL
| derogation | CREATE TABLE `derogation` (
`xxx` char(10) NOT NULL,
`xxx` varchar(100) DEFAULT NULL,
`code` char(17) DEFAULT NULL,
`xxx` varchar(20) DEFAULT NULL,
`xxx` char(6) DEFAULT NULL,
`xxx` varchar(50) DEFAULT NULL,
`xxx` varchar(200) DEFAULT NULL,
`xxxx` varchar(200) DEFAULT NULL,
`xxxx` varchar(100) DEFAULT NULL,
`xxxx` datetime DEFAULT NULL,
`xxx` varchar(100) DEFAULT NULL,
`xxxx` varchar(20) DEFAULT NULL,
`xxx` varchar(50) DEFAULT NULL,
`xxx` varchar(10) DEFAULT NULL,
`xxx` datetime DEFAULT NULL,
`xxx` varchar(10) DEFAULT NULL,
`xxx` datetime DEFAULT NULL,
`xxx` datetime DEFAULT NULL,
`status` varchar(20) DEFAULT NULL,
`xxx` varchar(1000) DEFAULT NULL,
KEY `code_index_derogation` (`code`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
| Last | CREATE TABLE `Last` (
`code` char(17) DEFAULT NULL,
`xxx` decimal(10,2) DEFAULT NULL,
`Date` datetime DEFAULT NULL,
`xxxx` datetime DEFAULT NULL,
`xxxx` datetime DEFAULT NULL,
`xxxx` text DEFAULT NULL,
`xxxxx` datetime DEFAULT NULL,
`xxxx` char(6) DEFAULT NULL,
KEY `idx_Last_Code` (`code`),
KEY `idx_Last_xxx` (`xxx`),
KEY `code_index` (`code`),
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
you can use temporary tables, staging tables CTE or you can add indexes for the columns of the tables.
But using temporary tables, it's always an excelent choice to accelerate the excecution of the the SQL query.
(The problem is subtle, and often overlooked in this forum.)
Don't mix collations. You are joining on code, but the CHARACTER SET and COLLATION are different between the two tables. Suggest using `ALTER TABLE .. CONVERT TO .. to get them to be the same.
Other tips:
Use CHAR only for truly fixed-length columns. Use VARCHAR otherwise.
Have a PRIMARY KEY on every table.
Don't say LEFT JOIN if you really expect JOIN; it adds confusion to the reader.
This query has been reported in slow logs which is clearly taking query_time: 00:00:20 seconds to execute. It is expected to send one row. The number of rows examined by this query is 9702034.
Explaining query does not give much clue on how this query can be optimized. :
SELECT DISTINCT `ecu`.*, `ecuad`.`google_address`, `ecuad`.`town`, `ecuad`.`region`, `ecuad`.`zip`, `ecuad`.`country_id`, `ecuad`.`lat`, `ecuad`.`long`, `bc`.`name` AS `company_name`, GROUP_CONCAT( CONCAT(euad.assign_to_company_id,"_",euad.assigned_to)) AS `assign_company_dm_id`, GROUP_CONCAT( distinct edhp.location_id ) AS `location_id`, GROUP_CONCAT( distinct edhp.job_type ) AS `job_type`, GROUP_CONCAT( distinct edhp2.industry_id ) AS `industry_id`, GROUP_CONCAT( distinct edhp2.id ) AS `hiringpref_id)`, GROUP_CONCAT( distinct edhpskl.skill_id ) AS `skill_id`, GROUP_CONCAT( distinct edhpskl.category_id ) AS `category_id`, `ecli`.`login_date` FROM `es_company_users` AS `ecu`
LEFT JOIN `es_company_user_addresses` AS `ecuad` ON ecuad.dm_id = ecu.company_user_id and ecuad.base_company_id = ecu.base_company_id
LEFT JOIN `base_companies` AS `bc` ON bc.id = ecu.base_company_id
LEFT JOIN `es_user_assigned_dm` AS `euad` ON euad.foreign_id = ecu.company_user_id and euad.base_company_id = ecu.base_company_id
LEFT JOIN `es_dm_hiring_preference` AS `edhp` ON edhp.dm_id = ecu.company_user_id and edhp.base_company_id = ecu.base_company_id AND edhp.location_id > 0
LEFT JOIN `es_dm_hiring_preference` AS `edhp2` ON edhp2.dm_id = ecu.company_user_id and edhp2.base_company_id = ecu.base_company_id
LEFT JOIN `es_dm_hiring_skills` AS `edhpskl` ON edhpskl.hiring_preference_id = edhp2.id AND edhpskl.skill_id > 0
LEFT JOIN `es_edclient_login_info` AS `ecli` ON ecu.company_user_id = ecli.company_user_id AND ecli.base_company_id = ecu.base_company_id AND ecli.`login_date` = (SELECT MAX(t2.login_date) FROM es_edclient_login_info t2 WHERE (t2.company_user_id = 780)) WHERE (ecu.id = '636') AND (ecu.company_user_id = '780')
Can you help me out to improve this query performance please.
Here is the Explain of this query
+----+-------------+---------+-------+--------------------------------------------+-----------------+---------+----------------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+---------+-------+--------------------------------------------+-----------------+---------+----------------------+------+-------------+
| 1 | PRIMARY | ecu | const | PRIMARY,company_user_id | PRIMARY | 4 | const | 1 | NULL |
| 1 | PRIMARY | ecuad | ref | dm_id | dm_id | 10 | const,const | 667 | NULL |
| 1 | PRIMARY | bc | const | PRIMARY | PRIMARY | 4 | const | 1 | NULL |
| 1 | PRIMARY | euad | ref | base_company_id | base_company_id | 10 | const,const | 2 | NULL |
| 1 | PRIMARY | edhp | ref | base_company_id | base_company_id | 10 | const,const | 60 | Using where |
| 1 | PRIMARY | edhp2 | ref | base_company_id | base_company_id | 10 | const,const | 60 | NULL |
| 1 | PRIMARY | edhpskl | ref | base_company_id | base_company_id | 5 | singledb_ed.edhp2.id | 1 | Using where |
| 1 | PRIMARY | ecli | ref | base_company_id,company_user_id,login_date | login_date | 5 | const | 1 | Using where |
| 2 | SUBQUERY | t2 | ref | company_user_id | company_user_id | 4 | const | 2 | NULL |
+----+-------------+---------+-------+--------------------------------------------+-----------------+---------+----------------------+------+-------------+
9 rows in set (0.00 sec)
Here is the create info of this query
CREATE TABLE `es_dm_hiring_skills` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`hiring_preference_id` int(11) DEFAULT NULL,
`category_id` int(11) DEFAULT NULL,
`skill_id` int(11) DEFAULT NULL,
`skill_matching_type` int(5) DEFAULT '1',
`created_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`is_cb_migrated` tinyint(4) DEFAULT '0',
`rs_migrated_id` int(11) DEFAULT NULL,
`is_eb_migrated` tinyint(5) NOT NULL DEFAULT '0',
`eb_reff_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `base_company_id` (`hiring_preference_id`)
) ENGINE=InnoDB AUTO_INCREMENT=460021 DEFAULT CHARSET=utf8;
CREATE TABLE `es_dm_hiring_preference` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`base_company_id` int(11) DEFAULT NULL,
`dm_id` int(11) DEFAULT NULL,
`category_id` int(11) DEFAULT NULL,
`job_role_id` int(11) DEFAULT NULL,
`skill_id` int(11) DEFAULT NULL,
`location_id` int(11) DEFAULT NULL,
`office_id` int(11) DEFAULT NULL COMMENT 'id of company_timezone_setting table',
`job_type` tinyint(1) DEFAULT NULL,
`industry_id` varchar(255) DEFAULT NULL,
`created_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`mod_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`is_updated` tinyint(1) DEFAULT '0' COMMENT '0=not updated, 1=updated, 2=updated by cron',
`is_cb_migrated` tinyint(4) DEFAULT '0',
`is_eb_migrated` tinyint(5) NOT NULL DEFAULT '0',
`merge_account_id` int(11) DEFAULT NULL,
`rs_migrated_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `base_company_id` (`base_company_id`,`dm_id`)
) ENGINE=InnoDB AUTO_INCREMENT=415866 DEFAULT CHARSET=utf8;
CREATE TABLE `es_user_assigned_dm` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`base_company_id` int(11) DEFAULT NULL,
`assign_to_company_id` int(11) NOT NULL,
`foreign_id` int(11) DEFAULT NULL,
`assigned_to` int(11) DEFAULT NULL COMMENT 'base user id, candidate is assigned to which user? ',
`assigned_type` int(2) DEFAULT '2' COMMENT '1= My Network, 2= Auto assignee',
`to_type` tinyint(1) DEFAULT '1' COMMENT '1 = Ed manager, 2= Marketing Consultant',
`assigned_by` int(11) DEFAULT NULL COMMENT 'dm assigned by which user',
`tiering_level` int(11) DEFAULT NULL,
`organisation_id` int(11) DEFAULT NULL,
`assigned_date` datetime DEFAULT NULL,
`modified_by` int(11) DEFAULT NULL,
`modified_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`is_cb_migrated` tinyint(4) DEFAULT '0',
`branch_id` int(11) DEFAULT NULL,
`merge_account_id` int(11) DEFAULT NULL,
`rs_migrated_id` int(11) DEFAULT NULL,
`is_eb_migrated` tinyint(4) DEFAULT '0',
`eb_reff_id` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `base_company_id` (`base_company_id`,`foreign_id`)
) ENGINE=InnoDB AUTO_INCREMENT=493547 DEFAULT CHARSET=utf8;
CREATE TABLE `base_companies` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`industry_id` int(11) DEFAULT NULL,
`country_id` int(11) DEFAULT NULL,
`supercrm_account_id` int(11) DEFAULT NULL,
`name` varchar(250) NOT NULL,
`status` tinyint(1) NOT NULL,
`size` tinyint(1) NOT NULL,
`timezone` varchar(255) DEFAULT NULL,
`logo` varchar(255) DEFAULT NULL,
`theme` varchar(255) NOT NULL,
`namespace` varchar(255) DEFAULT NULL,
`db_user` varchar(255) DEFAULT NULL,
`db_pass` varchar(255) DEFAULT NULL,
`db_host` varchar(255) NOT NULL,
`db_port` int(4) NOT NULL,
`db_socket` varchar(255) NOT NULL,
`cdn_url` varchar(255) DEFAULT NULL,
`cdn_ssl_url` varchar(256) NOT NULL,
`client_portal_flag` tinyint(1) DEFAULT NULL,
`client_company_id` int(11) DEFAULT NULL COMMENT 'Parent company id from which this have activated as client portal',
`mod_date` timestamp NULL DEFAULT NULL,
`created_date` datetime DEFAULT NULL,
`show_ed_manager` tinyint(1) DEFAULT '0' COMMENT 'temp column to flag Companys for Ed Manager',
`is_migrate_rpo` int(4) NOT NULL DEFAULT '0',
`is_sourcechain_account` smallint(2) DEFAULT '0' COMMENT '0=> Normal Account, 1=> Sourcechain account',
`sourcechain_from_name` varchar(255) DEFAULT NULL,
`sourcechain_email` varchar(255) DEFAULT NULL,
`is_cb_migrated` tinyint(4) DEFAULT '0',
`organisation_portal_flag` tinyint(4) DEFAULT '0' COMMENT '1 - CB, 2 - EB, 3 - TR ',
`blocked_organisation_flag` int(4) DEFAULT NULL COMMENT '1 - Blocked for new organisation, 2 - Removed Blocking for new organisation',
`rs_migrated_id` bigint(20) DEFAULT NULL,
`is_eb_migrated` tinyint(4) DEFAULT '0',
`eb_reff_id` bigint(20) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `status` (`status`),
KEY `base_companies` (`name`)
) ENGINE=InnoDB AUTO_INCREMENT=122873 DEFAULT CHARSET=utf8;
CREATE TABLE `es_company_user_addresses` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`dm_id` int(11) DEFAULT NULL,
`base_company_id` int(11) DEFAULT NULL,
`office_id` int(11) DEFAULT NULL COMMENT 'Primary key(id) of company_timezone_setting',
`google_address` varchar(255) DEFAULT NULL,
`type` int(11) DEFAULT '2',
`country_id` int(11) DEFAULT NULL,
`street1` tinytext,
`street2` tinytext,
`street3` tinytext,
`town` varchar(150) DEFAULT NULL,
`region` varchar(150) DEFAULT NULL,
`closest_city` varchar(150) DEFAULT NULL,
`zip` varchar(20) DEFAULT NULL,
`lat` float(10,6) DEFAULT NULL,
`long` float(10,6) DEFAULT NULL,
`last_mod_user_id` int(11) NOT NULL,
`mod_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`created_user_id` int(11) NOT NULL,
`created_date` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`is_cb_migrated` tinyint(4) DEFAULT '0',
`is_eb_migrated` tinyint(5) NOT NULL DEFAULT '0',
`eb_reff_id` int(11) DEFAULT NULL,
`updated_geo` tinyint(2) DEFAULT '0',
`merge_account_id` int(11) DEFAULT NULL,
`rs_migrated_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `dm_id` (`dm_id`,`base_company_id`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=434884 DEFAULT CHARSET=utf8;
CREATE TABLE `es_edclient_login_info` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`base_company_id` int(11) NOT NULL,
`company_user_id` int(11) NOT NULL,
`company_name` varchar(255) NOT NULL,
`user_email` varchar(255) NOT NULL,
`remote_ip` varchar(255) NOT NULL,
`assign_to_id` int(11) DEFAULT NULL,
`aasigned_to_company_id` int(11) NOT NULL,
`assign_to_email` int(11) DEFAULT NULL,
`login_date` datetime NOT NULL,
`merge_account_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `base_company_id` (`base_company_id`),
KEY `company_user_id` (`company_user_id`),
KEY `login_date` (`login_date`)
) ENGINE=InnoDB AUTO_INCREMENT=101621 DEFAULT CHARSET=utf8;
CREATE TABLE `es_company_users` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`base_company_id` int(11) DEFAULT NULL,
`base_user_id` int(11) DEFAULT NULL,
`company_user_id` int(11) DEFAULT NULL,
`es_client_candidate_link_id` int(11) DEFAULT NULL,
`branch_id` int(11) DEFAULT NULL,
`client_portal_id` int(11) DEFAULT NULL COMMENT 'client portal id if user is of client portal',
`crm_people_id` int(11) DEFAULT NULL COMMENT 'client user id-people_id',
`education_level` int(11) DEFAULT NULL,
`division_id` int(11) DEFAULT NULL,
`job_level` int(11) DEFAULT NULL,
`fname` varchar(50) DEFAULT NULL,
`lname` varchar(50) DEFAULT NULL,
`title` tinyint(4) DEFAULT NULL,
`job_title` varchar(100) DEFAULT NULL,
`sex` tinyint(1) DEFAULT NULL,
`dob` datetime DEFAULT NULL,
`country_id` int(4) DEFAULT NULL COMMENT 'no constraint should be made on this',
`picture` varchar(255) DEFAULT NULL,
`timezone` varchar(255) DEFAULT 'Europe/Berlin',
`function_id` int(11) DEFAULT NULL,
`default_module_id` int(11) DEFAULT NULL,
`status` tinyint(1) DEFAULT NULL COMMENT 'verified, activated, deactivated, approved',
`endorsed_user` int(1) DEFAULT '1',
`endorsed_dm_status` int(11) DEFAULT '0',
`is_private_dm` tinyint(1) DEFAULT '0' COMMENT '0 : Not private DM, 1 : Private DM ',
`marketplace_status` tinyint(1) NOT NULL DEFAULT '1' COMMENT '0-Deactive, 1-Active',
`flag_dm_feedback` tinyint(1) DEFAULT '0',
`rs_company_id` int(11) DEFAULT NULL,
`source_id` int(11) DEFAULT NULL COMMENT 'Base Company id',
`source_type` tinyint(1) DEFAULT NULL COMMENT 'We are adding dm from which section. 1 = Endorsed Client, 2 = Endorsed Manager, 3 = Get endorsed from RS',
`key_note` text,
`headhunt_note` text,
`created_user_id` int(11) DEFAULT NULL,
`created_date` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`approved_date` datetime DEFAULT NULL,
`last_mod_user_id` int(11) DEFAULT NULL,
`mod_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`last_contacted_date` timestamp NULL DEFAULT NULL,
`last_contacted_user_id` int(11) DEFAULT NULL,
`last_contacted_company_id` int(11) DEFAULT NULL,
`is_third_rep` tinyint(1) DEFAULT '0',
`is_cb_migrated` tinyint(4) DEFAULT '0',
`is_eb_migrated` tinyint(5) NOT NULL DEFAULT '0',
`eb_reff_id` int(11) DEFAULT NULL,
`is_third_sync` tinyint(4) DEFAULT '0',
`is_third_utm_sync` tinyint(4) DEFAULT '0',
`merge_account_id` int(11) DEFAULT NULL,
`is_linked_cron` int(1) DEFAULT '0',
`is_dm_SC_invite` tinyint(2) NOT NULL DEFAULT '0',
`indexed_error_flag_dm` tinyint(4) DEFAULT '0',
`is_digital` tinyint(4) DEFAULT NULL COMMENT '0=''No'', 1="Yes''',
`flag_missing_name` tinyint(2) DEFAULT '0',
PRIMARY KEY (`id`),
KEY `endorsed_dm_status` (`endorsed_dm_status`),
KEY `status` (`status`),
KEY `mod_date` (`mod_date`),
KEY `base_company_id` (`base_company_id`,`company_user_id`),
KEY `fname` (`fname`),
KEY `company_user_id` (`company_user_id`),
KEY `base_user_id` (`base_user_id`),
KEY `last_contacted_user_id` (`last_contacted_user_id`),
KEY `lname` (`lname`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=555262 DEFAULT CHARSET=utf8 COMMENT='this stores the user or DM of company';
You should check your query first before going into optimization.
The GROUP_CONCAT-function is an aggregate function in SQL. This means that unless the GROUP_CONCAT is the only column you query, you should have the GROUP BY-clause in the query.
The GROUP BY-clause divides the query result into smaller sets into which then the aggregate function (in this case GROUP_CONCAT) is applied to.
Older MySQL versions (and new ones configured the ONLY_FULL_GROUP_BY-mode off) allowed using aggregate function without GROUP BY-clause causing all sorts of errornous results.
To fix the issue, you should list the keys for those tables used in the selected columns (GROUP BY ecu.id, ecuad.id, bc.id).
After you have done this, check that you have indexs on column used in WHERE and
the ones doing the JOINs
How can this query be optimized more?
SELECT COUNT(*) AS releases, SUM(bt.size) AS sum_size, SUM(a.download_count) download_count FROM bb_bt_torrents bt LEFT JOIN bb_attachments_desc a ON (a.attach_id = bt.attach_id) WHERE bt.poster_id = 35979;
This query can run looong time, about 25 seconds.
What are my options?
Here is what explain gives to me:
+----+-------------+-------+--------+---------------+-----------+---------+------------------------+--------+-------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+-----------+---------+------------------------+--------+-------+
| 1 | SIMPLE | bt | ref | poster_id | poster_id | 3 | const | 178770 | |
| 1 | SIMPLE | a | eq_ref | PRIMARY | PRIMARY | 3 | unionpeer.bt.attach_id | 1 | |
+----+-------------+-------+--------+---------------+-----------+---------+------------------------+--------+-------+
And the table definitions:
CREATE TABLE IF NOT EXISTS `bb_attachments_desc` (
`attach_id` mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
`physical_filename` varchar(255) NOT NULL DEFAULT '',
`real_filename` varchar(255) NOT NULL DEFAULT '',
`download_count` mediumint(8) unsigned NOT NULL DEFAULT '0',
`comment` varchar(255) NOT NULL DEFAULT '',
`extension` varchar(100) NOT NULL DEFAULT '',
`mimetype` varchar(100) NOT NULL DEFAULT '',
`filesize` int(20) NOT NULL DEFAULT '0',
`filetime` int(11) NOT NULL DEFAULT '0',
`thumbnail` tinyint(1) NOT NULL DEFAULT '0',
`tracker_status` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`attach_id`),
KEY `filetime` (`filetime`),
KEY `filesize` (`filesize`),
KEY `physical_filename` (`physical_filename`(10))
) ENGINE=InnoDB DEFAULT CHARSET=cp1251;
CREATE TABLE IF NOT EXISTS `bb_bt_torrents` (
`torrent_id` mediumint(8) unsigned NOT NULL AUTO_INCREMENT,
`info_hash` varbinary(20) NOT NULL,
`post_id` mediumint(8) unsigned NOT NULL DEFAULT '0',
`poster_id` mediumint(9) NOT NULL DEFAULT '0',
`topic_id` mediumint(8) unsigned NOT NULL DEFAULT '0',
`forum_id` smallint(5) unsigned NOT NULL DEFAULT '0',
`attach_id` mediumint(8) unsigned NOT NULL DEFAULT '0',
`size` bigint(20) unsigned NOT NULL DEFAULT '0',
`reg_time` int(11) NOT NULL DEFAULT '0',
`call_seed_time` int(11) NOT NULL DEFAULT '0',
`complete_count` mediumint(8) unsigned NOT NULL DEFAULT '0',
`seeder_last_seen` int(11) NOT NULL DEFAULT '0',
`tor_status` tinyint(4) NOT NULL DEFAULT '0',
`tor_status_uid` mediumint(8) NOT NULL DEFAULT '0',
`tor_status_time` int(11) NOT NULL DEFAULT '0',
`tor_type` tinyint(1) NOT NULL DEFAULT '0',
`speed_up` mediumint(8) NOT NULL DEFAULT '0',
`speed_down` mediumint(8) NOT NULL DEFAULT '0',
`last_seeder_uid` mediumint(9) NOT NULL DEFAULT '0',
`donor` tinyint(2) NOT NULL DEFAULT '-1',
`donor_get_time` int(11) NOT NULL DEFAULT '0',
`rutracker_ids` tinyint(1) NOT NULL DEFAULT '0',
`dht` tinyint(1) NOT NULL DEFAULT '2',
PRIMARY KEY (`torrent_id`),
UNIQUE KEY `post_id` (`post_id`),
UNIQUE KEY `topic_id` (`topic_id`),
UNIQUE KEY `attach_id` (`attach_id`),
UNIQUE KEY `info_hash` (`info_hash`),
KEY `reg_time` (`reg_time`),
KEY `forum_id` (`forum_id`),
KEY `poster_id` (`poster_id`),
KEY `size` (`size`),
KEY `tor_status` (`tor_status`),
KEY `tor_status_time` (`tor_status_time`),
KEY `tor_status_2` (`tor_status`,`tor_status_time`)
) ENGINE=InnoDB DEFAULT CHARSET=cp1251;
I thought about rewriting this query, but don't know how.
We have a central login that we use to support multiple websites. To store our users' data we have an accounts table which stores each user account and then users tables for each site for site specific information. We also have a simple connections table which stores the connections between users.
We noticed that one query that is joining the tables on their primary key user_id is executing slowly. I'm hoping that some SQL expert out there can explain why it's using WHERE to search the users_site1 table and suggest how we can optimize it. Here is the slow query & the explain results:
mysql> explain select a.username,a.first_name,a.last_name,a.organization_name,a.organization,a.city,a.state,a.zip,a.country,a.profile_photo,a.facebook_id,a.twitter_id,u.reviews from accounts a join users_site1 u ON a.user_id=u.user_id where a.user_id IN (select cid2 from connections where cid1=10001006 AND type="MM" AND status="A") OR a.user_id IN (select cid1 from connections where cid2=10001006 AND type="MM" AND status="A") order by RAND() LIMIT 4;
+----+--------------------+-------------+--------+-------------------+---------+---------+-----------------------+-------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-------------+--------+-------------------+---------+---------+-----------------------+-------+----------------------------------------------+
| 1 | PRIMARY | u | ALL | PRIMARY | NULL | NULL | NULL | 79783 | Using where; Using temporary; Using filesort |
| 1 | PRIMARY | a | eq_ref | PRIMARY | PRIMARY | 4 | exampledb.u.user_id | 1 | |
| 3 | DEPENDENT SUBQUERY | connections | ref | PRIMARY,cid1,cid2 | cid2 | 6 | const,const | 2 | Using where |
| 2 | DEPENDENT SUBQUERY | connections | ref | PRIMARY,cid1,cid2 | cid1 | 6 | const,const | 1 | Using where |
+----+--------------------+-------------+--------+-------------------+---------+---------+-----------------------+-------+----------------------------------------------+
4 rows in set (0.00 sec)
Here are the definitions for each table:
CREATE TABLE `accounts` (
`user_id` int(9) unsigned NOT NULL AUTO_INCREMENT,
`username` varchar(40) DEFAULT NULL,
`facebook_id` bigint(15) unsigned DEFAULT NULL,
`facebook_username` varchar(30) DEFAULT NULL,
`password` varchar(20) DEFAULT NULL,
`profile_photo` varchar(100) DEFAULT NULL,
`first_name` varchar(40) DEFAULT NULL,
`middle_name` varchar(40) DEFAULT NULL,
`last_name` varchar(40) DEFAULT NULL,
`suffix_name` char(3) DEFAULT NULL,
`organization_name` varchar(100) DEFAULT NULL,
`organization` tinyint(1) unsigned DEFAULT NULL,
`address` varchar(200) DEFAULT NULL,
`city` varchar(40) DEFAULT NULL,
`state` varchar(20) DEFAULT NULL,
`zip` varchar(10) DEFAULT NULL,
`province` varchar(40) DEFAULT NULL,
`country` int(3) DEFAULT NULL,
`latitude` decimal(11,7) DEFAULT NULL,
`longitude` decimal(12,7) DEFAULT NULL,
`phone` varchar(20) DEFAULT NULL,
`sex` char(1) DEFAULT NULL,
`birthday` date DEFAULT NULL,
`about_me` varchar(2000) DEFAULT NULL,
`activities` varchar(300) DEFAULT NULL,
`website` varchar(100) DEFAULT NULL,
`email` varchar(150) DEFAULT NULL,
`referrer` int(4) unsigned DEFAULT NULL,
`referredid` int(9) unsigned DEFAULT NULL,
`verify` int(6) DEFAULT NULL,
`status` char(1) DEFAULT 'R',
`created` datetime DEFAULT NULL,
`verified` datetime DEFAULT NULL,
`activated` datetime DEFAULT NULL,
`network` datetime DEFAULT NULL,
`deleted` datetime DEFAULT NULL,
`logins` int(6) unsigned DEFAULT '0',
`api_logins` int(6) unsigned DEFAULT '0',
`last_login` datetime DEFAULT NULL,
`last_update` datetime DEFAULT NULL,
`private` tinyint(1) unsigned DEFAULT NULL,
`ip` varchar(20) DEFAULT NULL,
PRIMARY KEY (`user_id`),
UNIQUE KEY `username` (`username`),
KEY `facebook_id` (`facebook_id`),
KEY `status` (`status`),
KEY `state` (`state`)
);
CREATE TABLE `users_site1` (
`user_id` int(9) unsigned NOT NULL,
`facebook_id` bigint(15) unsigned DEFAULT NULL,
`facebook_username` varchar(30) DEFAULT NULL,
`facebook_publish` tinyint(1) unsigned DEFAULT NULL,
`facebook_checkin` tinyint(1) unsigned DEFAULT NULL,
`facebook_offline` varchar(300) DEFAULT NULL,
`twitter_id` varchar(60) DEFAULT NULL,
`twitter_secret` varchar(50) DEFAULT NULL,
`twitter_username` varchar(20) DEFAULT NULL,
`type` char(1) DEFAULT 'M',
`referrer` int(4) unsigned DEFAULT NULL,
`referredid` int(9) unsigned DEFAULT NULL,
`session` varchar(60) DEFAULT NULL,
`api_session` varchar(60) DEFAULT NULL,
`status` char(1) DEFAULT 'R',
`created` datetime DEFAULT NULL,
`verified` datetime DEFAULT NULL,
`activated` datetime DEFAULT NULL,
`deleted` datetime DEFAULT NULL,
`logins` int(6) unsigned DEFAULT '0',
`api_logins` int(6) unsigned DEFAULT '0',
`last_login` datetime DEFAULT NULL,
`last_update` datetime DEFAULT NULL,
`ip` varchar(20) DEFAULT NULL,
PRIMARY KEY (`user_id`)
);
CREATE TABLE `connections` (
`cid1` int(9) unsigned NOT NULL DEFAULT '0',
`cid2` int(9) unsigned NOT NULL DEFAULT '0',
`cid3` int(9) unsigned NOT NULL DEFAULT '0',
`type` char(2) NOT NULL,
`status` char(1) NOT NULL,
`created` datetime DEFAULT NULL,
`updated` datetime DEFAULT NULL,
PRIMARY KEY (`cid1`,`cid2`,`type`,`cid3`),
KEY `cid1` (`cid1`,`type`),
KEY `cid2` (`cid2`,`type`)
);
Instead of WHERE a.userid IN( ... ) OR a.userid IN( ... ) you should use another join:
select
a.username,a.first_name,a.last_name,a.organization_name,a.organization,a.city,
a.state,a.zip,a.country,a.profile_photo,a.facebook_id,a.twitter_id,u.reviews
from accounts a
join users_site1 u ON a.user_id=u.user_id
join ( select cid2 as id from connections
where cid1=10001006 AND type="MM" AND status="A"
union
select cid1 as id from connections
where cid2=10001006 AND type="MM" AND status="A" ) c
on a.user_id = c.id
order by RAND() LIMIT 4;
have you tried remove order by RAND() and run again?
my result is below:
+----+--------------------+-------------+----------------+-------------------+---------+---------+------------------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+--------------------+-------------+----------------+-------------------+---------+---------+------------------+------+----------------------------------------------+
| 1 | PRIMARY | a | ALL | PRIMARY | NULL | NULL | NULL | 2 | Using where; Using temporary; Using filesort |
| 1 | PRIMARY | u | ALL | PRIMARY | NULL | NULL | NULL | 2 | Using where; Using join buffer |
| 3 | DEPENDENT SUBQUERY | connections | index_subquery | PRIMARY,cid1,cid2 | PRIMARY | 14 | func,const,const | 1 | Using where |
| 2 | DEPENDENT SUBQUERY | connections | ref | PRIMARY,cid1,cid2 | PRIMARY | 14 | const,func,const | 1 | Using where |
+----+--------------------+-------------+----------------+-------------------+---------+---------+------------------+------+----------------------------------------------+
I am not a MySQL guru by any means but been have involved more than once in optimization of high performance applications, though I was more on the implementation end of the optimisation process versus finding what needed to be optimized.
The firt thing I see is the subqueries seem efficient, but the way the first query is run with this where clause: ... where a.user_id IN (select cid2 ...) or a.user_id IN (select cid1 from ...) is a performance killer in my very humble opinion.
The first thing I would try to optimise performance, consider trying join decomposition , split your request in 2 or even 3 queries. The code is less pretty, but the db will be able to work more efficiently. It is a myth that doing everything in one query is better.
What can this bring you? Caching will be more efficient, if using MyISam tables the locking srategy is more efficient when you have less tables in your query, and you will reduce the redundant row accesses. If you can get your main query ( that would be the last one if you decompose ) from Using where; Using temporary; Using filesort, you will have much faster response.
Profile the different options you try with SHOW SESSION STATUS and FLUSH status, also you can disable caching to get true comparison of different options you try by adding SQL_NO_CACHE in your query, ie SELSECT SQL_NO_CACHE a.username ... etc..
Profiling and measuring the results is the only way you will be able to determine the performance gains. Unfortunately this step is often overlooked.
Good luck!