Related
I have a few problems with MYSQL database that I can't solve.
My query below is taking too much time and making the system hang. I'm trying to use the "JOIN" construct to develop this. But this time my aggregation, which I'm trying to do with "SUM", reduces the query to one line. Is it ok to do this job with "JOIN"? or how should i improve this query.
This database works with a total of 22 client devices in ASP .NET application. As I mentioned above, in cases where the query time is long, when the client devices send a query to the database at the same time, the client device freezes. What I don't understand is why a query in the browser app is making all devices wait. Isn't each query processed as a separate "Thread" in MYSQL? So if a query has a return time of 10 seconds, will all clients wait 10 seconds for the query to be answered in the browser?
SELECT *,
(SELECT MachModel FROM machine WHERE MachCode=workorder.MachCode) AS MachModel,
(SELECT RawMaterialDescription FROM rawmaterials WHERE RawMaterialCode=workorder.ProductRawMaterial) AS RawMaterialDescr,
(SELECT RawMaterialColor FROM rawmaterials WHERE RawMaterialCode=workorder.ProductRawMaterial) AS RawMaterialColor,
(SELECT StaffName FROM staff WHERE AccountName=workorder.AssignStaff) AS AssignStaffName,
(SELECT StaffCode FROM staff WHERE AccountName=workorder.AssignStaff) AS AssignStaffCode,
(SELECT MachStatus FROM machine WHERE MachCode=workorder.MachCode) AS MachStatus,
(SELECT SUM(xStopTime) FROM workorderb WHERE xWoNumber=workorder.WoNumber) AS WoTotalStopTime
FROM workorder
WHERE WoStatus=3
ORDER BY PlanProdStartDate DESC, WoSortNumber, WoNumber LIMIT 100
SELECT workorder.*,machine.MachModel,machine.MachStatus,rawmaterials.RawMaterialDescription,rawmaterials.RawMaterialColor,staff.StaffName,staff.StaffCode,SUM(workorderb.xStopTime)
FROM workorder
LEFT JOIN machine ON machine.MachCode=workorder.MachCode
LEFT JOIN rawmaterials ON rawmaterials.RawMaterialCode=workorder.ProductRawMaterial
LEFT JOIN staff ON staff.AccountName=workorder.AssignStaff
LEFT JOIN workorderb ON workorderb.xWoNumber=workorder.WoNumber
WHERE workorder.WoStatus=3
ORDER BY workorder.PlanProdStartDate DESC, workorder.WoSortNumber, workorder.WoNumber LIMIT 100
CREATE TABLE `workorder` (
`WoNumber` varchar(20) NOT NULL,
`MachCode` varchar(15) NOT NULL,
`PlannedMoldCode` varchar(10) NOT NULL,
`PartyNumber` smallint(6) NOT NULL,
`PlanProdCycleTime` smallint(6) NOT NULL,
`CalAverageCycleTime` float(15,1) unsigned NOT NULL,
`ProductRawMaterial` varchar(30) NOT NULL,
`PlanProdStartDate` date NOT NULL,
`PlanProdFinishDate` int(10) unsigned NOT NULL,
`WoStartDate` datetime DEFAULT NULL,
`WoFinishDate` datetime DEFAULT NULL,
`WoWorkTime` int(10) unsigned NOT NULL,
`WoSystemProductivity` smallint(6) unsigned NOT NULL,
`AssignStaff` varchar(50) DEFAULT '',
`WoStatus` smallint(6) NOT NULL,
`WoSortNumber` int(10) unsigned NOT NULL,
`CycleCount` int(11) unsigned NOT NULL,
`ControlDate` datetime NOT NULL ON UPDATE CURRENT_TIMESTAMP,
`WoProductionStatus` smallint(6) NOT NULL DEFAULT '0',
`Creator` varchar(50) NOT NULL,
`Changer` varchar(50) NOT NULL,
`CreateDate` datetime NOT NULL,
PRIMARY KEY (`WoNumber`) USING BTREE,
KEY `WoNumber` (`WoNumber`) USING BTREE,
KEY `WoNumber_2` (`WoNumber`) USING BTREE,
KEY `WoNumber_3` (`WoNumber`) USING BTREE
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC
CREATE TABLE `machine` (
`MachCode` varchar(15) NOT NULL,
`MachModel` varchar(30) NOT NULL,
`FirstProdDate` date NOT NULL,
`MachCapacity` smallint(6) NOT NULL,
`MachStatus` smallint(6) NOT NULL,
`NowMoldOnMach` varchar(10) NOT NULL DEFAULT '',
`NowMachOperator` varchar(50) NOT NULL DEFAULT '',
`NowWorkOrder` varchar(20) NOT NULL DEFAULT '',
`IPNumber` varchar(15) NOT NULL,
`Creator` varchar(50) NOT NULL,
`Changer` varchar(50) NOT NULL,
`ControlDate` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`OperatorLoginDate` datetime DEFAULT NULL,
`Message` varchar(500) DEFAULT NULL,
`MessageReaded` smallint(6) DEFAULT '0',
`StaffName` varchar(50) DEFAULT 'OSIS',
`StaffImage` varchar(255) DEFAULT '',
`StopDesc` varchar(30) DEFAULT 'OSIS',
`StopTime` datetime DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`MachCode`) USING BTREE
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC
CREATE TABLE `rawmaterials` (
`RawMaterialCode` varchar(15) NOT NULL,
`RawMaterialDescription` varchar(30) NOT NULL,
`RawMaterialColor` varchar(30) NOT NULL,
PRIMARY KEY (`RawMaterialCode`) USING BTREE,
KEY `RawMaterialCode` (`RawMaterialCode`) USING BTREE
) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC
CREATE TABLE `staff` (
`StaffCode` varchar(15) DEFAULT NULL,
`StaffCardCode` varchar(10) DEFAULT NULL,
`StaffName` varchar(50) NOT NULL,
`StaffPassword` varchar(10) NOT NULL,
`StaffStatus` smallint(6) NOT NULL DEFAULT '2',
`StaffDateOfStart` date NOT NULL,
`StaffBirthDay` date DEFAULT NULL,
`StaffGender` varchar(5) DEFAULT NULL,
`StaffRoleA` smallint(6) NOT NULL,
`StaffEmail` varchar(100) NOT NULL,
`StaffImageLink` varchar(255) DEFAULT NULL,
`AccountName` varchar(50) NOT NULL,
`StaffRoleB` smallint(6) NOT NULL,
`StaffRoleD` smallint(6) NOT NULL,
`StaffRoleE` smallint(6) NOT NULL,
`StaffRoleC` smallint(6) NOT NULL,
`StaffRoleF` smallint(6) NOT NULL,
`StaffRoleG` smallint(6) NOT NULL,
`StaffRoleH` smallint(6) NOT NULL,
`StaffRoleI` smallint(6) NOT NULL,
`StaffRoleJ` smallint(6) NOT NULL,
`StaffRoleK` smallint(6) NOT NULL,
`StaffRoleL` smallint(6) NOT NULL,
`StaffRoleM` smallint(6) NOT NULL,
`StaffRoleN` smallint(6) NOT NULL,
`StaffConnection` smallint(6) NOT NULL DEFAULT '2',
`MachineWorked` varchar(15) DEFAULT NULL,
`WorkOrderWorked` varchar(20) DEFAULT NULL,
`StaffGroup` varchar(50) NOT NULL,
`Creator` varchar(50) NOT NULL,
`Changer` varchar(50) DEFAULT NULL,
PRIMARY KEY (`AccountName`) USING BTREE
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC
CREATE TABLE `workorderb` (
`xWoNumber` varchar(20) NOT NULL,
`xMachCode` varchar(15) NOT NULL,
`xPlannedMoldCode` varchar(10) NOT NULL,
`xPartyNumber` smallint(6) NOT NULL,
`xStaffName` varchar(50) NOT NULL,
`xStopCode` smallint(6) NOT NULL,
`xStopStartTime` datetime NOT NULL,
`xStopFinishTime` datetime DEFAULT NULL,
`xStopTime` int(11) DEFAULT NULL,
PRIMARY KEY (`xMachCode`,`xStopStartTime`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC
Your query with the joins was nearly there: it was just missing a GROUP BYclause.
I have replaced workorder.* with workorder.WoNumber in the SELECT and added GROUP BY workorder.WoNumber.
You can add as many columns from workorder in the SELECT as you like but you must list them in the GROUP BY.
SELECT workorder.WoNumber,machine.MachModel,machine.MachStatus,rawmaterials.RawMaterialDescription,rawmaterials.RawMaterialColor,staff.StaffName,staff.StaffCode,SUM(workorderb.xStopTime)
FROM workorder
LEFT JOIN machine ON machine.MachCode=workorder.MachCode
LEFT JOIN rawmaterials ON rawmaterials.RawMaterialCode=workorder.ProductRawMaterial
LEFT JOIN staff ON staff.AccountName=workorder.AssignStaff
LEFT JOIN workorderb ON workorderb.xWoNumber=workorder.WoNumber
WHERE workorder.WoStatus=3
GROUP BY workorder.WoNumber \* <<= ADD OTHER COLUMNS HERE AS NEEDED *\
ORDER BY workorder.PlanProdStartDate DESC, workorder.WoSortNumber, workorder.WoNumber LIMIT 100;
db<>fiddle here
Use InnoDB, not MyISAM. MyISAM locks the entire table when INSERTing; InnoDB can often allow other threads to run when inserting.
Other notes
workorder has 4 identical indexes on wonumber; keep the PK, toss the rest. Note that a PRIMARY KEY is an index. Check the other tables for redundant Keys.
Do you need the mixture of DESC and ASC in ORDER BY PlanProdStartDate DESC, WoSortNumber, WoNumber? If not, there may be an optimization here.
As Kendle suggests, JOINs would be faster since there are cases where the same table is needed twice. If values might be missing, then LEFT might be useful; it won't change the performance.
Needed:
workorderb: INDEX(xWoNumber, xStopTime)
Is xStopTime an elapsed time? Or a time of day?
There are two databases, AWS RDS and Digitalocean $ 5 VPS.
I imported the same .sql file to both databases, the performance difference between them is almost 66000%
select count(special_cargo_id) from special_cargos;
AWS RDS elapsed time = 35.681seconds
DO 5$ VPS on Mysql elapsed time = 0.086seconds
Details:
Table Lenght
1.85GB Rows ~260k
MySQL Version 8
Table Engine InnoDB
RDS OUTPUT
CREATE TABLE `special_cargos` (
`special_cargo_id` int(11) NOT NULL AUTO_INCREMENT,
`barcode` varchar(20) DEFAULT '',
`salesCode` varchar(15) NOT NULL,
`price` varchar(15) NOT NULL,
`para` varchar(3) DEFAULT NULL,
`postaceki` varchar(25) DEFAULT '',
`iban` varchar(30) DEFAULT NULL,
`send_type` tinyint(1) unsigned NOT NULL,
`customer_id` int(11) NOT NULL,
`address_id` int(11) NOT NULL,
`height` varchar(5) DEFAULT '',
`weight` varchar(5) DEFAULT '',
`width` varchar(5) NOT NULL DEFAULT '',
`desi` varchar(5) DEFAULT '',
`length` varchar(255) DEFAULT '',
`customer_firstname` varchar(32) NOT NULL,
`customer_lastname` varchar(32) NOT NULL,
`customer_company` varchar(40) NOT NULL,
`customer_address` varchar(255) NOT NULL,
`customer_city` int(11) NOT NULL DEFAULT '0',
`customer_city_name` varchar(55) DEFAULT '',
`customer_zone_id` int(11) NOT NULL DEFAULT '0',
`customer_zone_name` varchar(55) DEFAULT '',
`customer_postcode` varchar(10) NOT NULL,
`firstname` varchar(32) NOT NULL,
`lastname` varchar(32) NOT NULL,
`company` varchar(40) DEFAULT '',
`address` varchar(255) NOT NULL,
`country_id` int(5) DEFAULT NULL,
`country_name` varchar(64) DEFAULT NULL,
`city_id` int(11) NOT NULL,
`city_name` varchar(55) NOT NULL,
`district_id` int(11) NOT NULL,
`district_name` varchar(64) NOT NULL,
`email` varchar(96) NOT NULL,
`phone` varchar(55) NOT NULL,
`postcode` varchar(10) DEFAULT '',
`product` text,
`sendText` text,
`returnText` text,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`deleted_at` datetime DEFAULT NULL,
`cargo_firm` tinyint(1) NOT NULL DEFAULT '0',
`ups_barcode_link` text,
`status` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`special_cargo_id`),
KEY `customer_id` (`customer_id`),
KEY `special_cargo_id` (`special_cargo_id`),
KEY `barcode` (`barcode`),
KEY `salesCode` (`salesCode`),
KEY `price` (`price`),
KEY `postaceki` (`postaceki`),
KEY `customer_firstname` (`customer_firstname`),
KEY `customer_lastname` (`customer_lastname`),
KEY `firstname` (`firstname`),
KEY `lastname` (`lastname`),
KEY `customer_company` (`customer_company`),
KEY `customer_address` (`customer_address`),
KEY `customer_city_name` (`customer_city_name`),
KEY `customer_zone_name` (`customer_zone_name`),
KEY `company` (`company`),
KEY `address` (`address`),
KEY `city_name` (`city_name`),
KEY `district_name` (`district_name`),
KEY `email` (`email`),
KEY `phone` (`phone`),
KEY `postcode` (`postcode`),
KEY `index_special_cargos` (`special_cargo_id`)
) ENGINE=InnoDB AUTO_INCREMENT=497539 DEFAULT CHARSET=utf8
+-----------------------+----------+
|Variable_name |Value |
+-----------------------+----------+
|innodb_buffer_pool_size|1073741824|
+-----------------------+----------+
+--+-----------+--------------+----------+-----+-------------+-----------+-------+----+------+--------+-----------+
|id|select_type|table |partitions|type |possible_keys|key |key_len|ref |rows |filtered|Extra |
+--+-----------+--------------+----------+-----+-------------+-----------+-------+----+------+--------+-----------+
|1 |SIMPLE |special_cargos|NULL |index|NULL |customer_id|4 |NULL|146973|100 |Using index|
+--+-----------+--------------+----------+-----+-------------+-----------+-------+----+------+--------+-----------+
SIMPLE VPS ON MYSQL OUTPUT
CREATE TABLE `special_cargos` (
`special_cargo_id` int(11) NOT NULL AUTO_INCREMENT,
`barcode` varchar(20) NOT NULL,
`salesCode` varchar(15) NOT NULL,
`price` varchar(15) NOT NULL,
`para` varchar(3) DEFAULT NULL,
`postaceki` varchar(25) NOT NULL,
`iban` varchar(30) DEFAULT NULL,
`send_type` tinyint(1) unsigned NOT NULL,
`customer_id` int(11) NOT NULL,
`address_id` int(11) NOT NULL,
`height` varchar(5) NOT NULL,
`weight` varchar(5) NOT NULL,
`width` varchar(5) NOT NULL,
`desi` varchar(5) NOT NULL,
`length` varchar(255) NOT NULL,
`customer_firstname` varchar(32) NOT NULL,
`customer_lastname` varchar(32) NOT NULL,
`customer_company` varchar(40) NOT NULL,
`customer_address` varchar(255) NOT NULL,
`customer_city` int(11) NOT NULL,
`customer_city_name` varchar(55) NOT NULL,
`customer_zone_id` int(11) NOT NULL,
`customer_zone_name` varchar(55) NOT NULL,
`customer_postcode` varchar(10) NOT NULL,
`firstname` varchar(32) NOT NULL,
`lastname` varchar(32) NOT NULL,
`company` varchar(40) NOT NULL,
`address` varchar(255) NOT NULL,
`country_id` int(5) DEFAULT NULL,
`country_name` varchar(64) DEFAULT NULL,
`city_id` int(11) NOT NULL,
`city_name` varchar(55) NOT NULL,
`district_id` int(11) NOT NULL,
`district_name` varchar(64) NOT NULL,
`email` varchar(96) NOT NULL,
`phone` varchar(55) NOT NULL,
`postcode` varchar(10) NOT NULL,
`product` text,
`sendText` text NOT NULL,
`returnText` text NOT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`deleted_at` datetime DEFAULT NULL,
`cargo_firm` int(1) NOT NULL DEFAULT '0',
`ups_barcode_link` text,
`status` int(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`special_cargo_id`),
KEY `customer_id` (`customer_id`),
KEY `special_cargo_id` (`special_cargo_id`),
KEY `barcode` (`barcode`),
KEY `salesCode` (`salesCode`),
KEY `price` (`price`),
KEY `postaceki` (`postaceki`),
KEY `customer_firstname` (`customer_firstname`),
KEY `customer_lastname` (`customer_lastname`),
KEY `firstname` (`firstname`),
KEY `lastname` (`lastname`),
KEY `customer_company` (`customer_company`),
KEY `customer_address` (`customer_address`),
KEY `customer_city_name` (`customer_city_name`),
KEY `customer_zone_name` (`customer_zone_name`),
KEY `company` (`company`),
KEY `address` (`address`),
KEY `city_name` (`city_name`),
KEY `district_name` (`district_name`),
KEY `email` (`email`),
KEY `phone` (`phone`),
KEY `postcode` (`postcode`)
) ENGINE=InnoDB AUTO_INCREMENT=497037 DEFAULT CHARSET=utf8
+-----------------------+---------+
|Variable_name |Value |
+-----------------------+---------+
|innodb_buffer_pool_size|134217728|
+-----------------------+---------+
+--+-----------+--------------+----------+-----+-------------+-----------+-------+----+------+--------+-----------+
|id|select_type|table |partitions|type |possible_keys|key |key_len|ref |rows |filtered|Extra |
+--+-----------+--------------+----------+-----+-------------+-----------+-------+----+------+--------+-----------+
|1 |SIMPLE |special_cargos|NULL |index|NULL |customer_id|4 |NULL|133967|100 |Using index|
+--+-----------+--------------+----------+-----+-------------+-----------+-------+----+------+--------+-----------+
There are two aspects of the problem here.
Aspect one and more important IMO is the query - it's bad. It does full index scan (yes, PRIMARY is also an index). Its performance scales linearly vs better and desired the B+tree's nlog(n).
Aspect two. Why the difference in the execution time. It can be related to caching (in DO's case the query was served from the buffer pool while in RDS from disk(=EBS)). It can be related to MySQL optimizer (DO and RDS could pick different indexes to serve the query PRIMARY vs smaller secondary index). But all that doesn't matter much due to the problems with the query itself.
Can you provide the output of:
SHOW CREATE TABLE special_cargos;
SHOW GLOBAL VARIABLES LIKE 'innodb_buffer_pool_size';
EXPLAIN select count(special_cargo_id) from special_cargos;
on both RDS and your non-RDS instance?
Assuming for a moment that special_cargo_id is the primary key, and it fits into RAM, and innodb_buffer_pool_size is configured reasonably (on a server with 2GB of RAM, 1GB is a reasonable amount, though RDS should come pre-configured to a sane value), it should be running similarly on both nodes.
Two possible explanations come to mind:
1) On RDS you don't have special_cargo_id defined as the PK.
2) RDS is for some reason choosing to not use the index (comparison of EXPLAIN output will confirm whether this is happening).
Edit:
Actually - are you sure your $5 VPS is using InnoDB? If you are getting an answer back in 0.08s, that sounds like it isn't actually scanning even the index. There is a good chance that on the VPS you are using MyISAM, which updates the number of rows in the table header after every write, specifically to make SELECT COUNT(*) FROM table_name; queries return instantaneously - like you have observed.
Edit 2:
Wow - so VPS is configured with only 128MB of innodb_buffer_pool_size and it is STILL going faster than RDS configured with 1GB of innodb_buffer_pool_size, and the execution plan on both is the same?
OK, try this:
select count(1) from special_cargos FORCE INDEX (PRIMARY);
1GB for the buffer_pool is dangerously high when you have only 2GB of RAM. Probably that system was swapping furiously.
Lower it to 200M and see how fast it runs.
See if they have some way of monitoring swap usage.
I'm getting some strange timing values from Mysql running a "simple" query.
This is the DDL of the table:
CREATE TABLE `frame` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`createdBy` varchar(255) DEFAULT NULL,
`createdDate` datetime(6) NOT NULL,
`lastModifiedBy` varchar(255) DEFAULT NULL,
`lastModifiedDate` datetime(6) DEFAULT NULL,
`sid` varchar(36) NOT NULL,
`version` bigint(20) NOT NULL,
`brand` varchar(255) DEFAULT NULL,
`category` varchar(255) DEFAULT NULL,
`colorCode` varchar(255) DEFAULT NULL,
`colorDescription` varchar(255) DEFAULT NULL,
`description` longtext,
`imageUrl` varchar(255) DEFAULT NULL,
`lastPurchase` datetime(6) DEFAULT NULL,
`lastPurchasePrice` decimal(19,2) DEFAULT NULL,
`lastSell` datetime(6) DEFAULT NULL,
`lastSellPrice` decimal(19,2) DEFAULT NULL,
`line` varchar(255) DEFAULT NULL,
`manufacturer` varchar(255) DEFAULT NULL,
`manufacturerCode` varchar(255) DEFAULT NULL,
`name` varchar(255) NOT NULL,
`preset` bit(1) NOT NULL DEFAULT b'0',
`purchasePrice` decimal(19,2) DEFAULT NULL,
`salesPrice` decimal(19,2) DEFAULT NULL,
`sku` varchar(255) NOT NULL,
`stock` bit(1) NOT NULL DEFAULT b'1',
`thumbUrl` varchar(255) DEFAULT NULL,
`upc` varchar(255) DEFAULT NULL,
`arm` int(11) DEFAULT NULL,
`bridge` int(11) DEFAULT NULL,
`caliber` int(11) DEFAULT NULL,
`gender` varchar(255) DEFAULT NULL,
`lensColor` varchar(255) DEFAULT NULL,
`material` varchar(255) DEFAULT NULL,
`model` varchar(255) NOT NULL,
`sphere` decimal(10,2) DEFAULT NULL,
`type` varchar(255) NOT NULL,
`taxRate_id` bigint(20) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UK_k7s4esovkoacsc264bcjrre13` (`sid`),
UNIQUE KEY `UK_ajh6mr6a6qg6mgy8t9nevdym1` (`sku`),
UNIQUE KEY `UK_boqikmg9o89j8q0o5ujkj33b3` (`upc`),
KEY `idx_manufacturer` (`manufacturer`),
KEY `idx_brand` (`brand`),
KEY `idx_line` (`line`),
KEY `idx_colorcode` (`colorCode`),
KEY `idx_preset` (`preset`),
KEY `idx_manufacturer_model_color_caliber` (`manufacturer`,`model`,`colorCode`,`caliber`),
KEY `FK1nau29fd70s1nq905dgs6ft85` (`taxRate_id`),
CONSTRAINT `FK1nau29fd70s1nq905dgs6ft85` FOREIGN KEY (`taxRate_id`) REFERENCES `taxrate` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=392179 DEFAULT CHARSET=utf8;
The query is created programatically from my application. The "strange" syntax (NULL IS NULL OR condition) is very convenient to me in order to make more compact my code and removing the need to create a different query based on the numbers of parameters.
For who understand how Hibernate HQL and JPA works, this is the query:
This query is generated when the user is not setting any filter, so all parameters in my condition are null and this is how the query comes out.
SELECT SQL_NO_CACHE COUNT(frame0_.`id`) AS col_0_0_ FROM `Frame` frame0_
WHERE (NULL IS NULL OR NULL LIKE CONCAT('%', NULL, '%') OR frame0_.`manufacturer` LIKE CONCAT('%', NULL, '%') OR frame0_.`manufacturerCode`=NULL OR frame0_.`sku`=NULL OR frame0_.`upc`=NULL OR frame0_.`line` LIKE CONCAT('%', NULL, '%') OR frame0_.`model` LIKE CONCAT('%', NULL, '%')) AND (NULL IS NULL OR frame0_.`manufacturer`=NULL) AND (NULL IS NULL OR frame0_.`line`=NULL) AND (NULL IS NULL OR frame0_.`caliber`=NULL) AND (NULL IS NULL OR frame0_.`type`=NULL) AND (NULL IS NULL OR frame0_.`material`=NULL) AND (NULL IS NULL OR frame0_.`model`=NULL) AND (NULL IS NULL OR frame0_.`colorCode`=NULL)
The query takes about 0.105s on a table of 137548 rows.
The EXPLAIN of the previous query returns:
id select_type table partitions type possible_keys key key_len ref rows filtered Extra
1 SIMPLE frame0_ \N ALL \ N \N \N \N 137548 100.00 \N
The previous query is identical to this one:
SELECT SQL_NO_CACHE COUNT(frame0_.`id`) AS col_0_0_ FROM `Frame` frame0_
This query takes just 0.05s for the same result in the same table.
Why for Mysql they are different and the first is taking so much time? Is there a way to improve performance of the first query keeping the syntax "NULL IS NULL or condition"?
I think Ryan has right, so the many or statement made the query so bad.
You should programatically build queries for better performance. So if the user not select by a possible filter, than you shouldn't include to the query!
(HQL)
if(!StringUtils.isEmpty(manufacturer)) {
query.and(m.manufacturer.eq(manufacturer))
}
I need to search records in the table which contains millions of records. I have recently updated Mysql version from 5.1 to 5.6.
I was using like in query which was taking around 15 sec to 30 sec.
Currently I have modified query to use MATCH... AGAINST feature of mysql 5.6 INNODB. My query goes like this.
SELECT JOB.IDJOB, JOB.IDEMPLOYER .....
FROM JOB
WHERE ( ( JOB.ITJOBSTARTTYPE=2 AND JOB.DTPLANNEDEND >= '2015-07-06' )
OR ( JOB.ITJOBSTARTTYPE=1
AND ( ( JOB.ITJOBENDTYPE=2 ) OR ( JOB.DTJOBEND>='2015-07-06') )
)
OR ( JOB.ITJOBSTARTTYPE=3
AND ( (JOB.DTJOBEND >='2015-07-06') OR (JOB.ITJOBENDTYPE=2) )
)
)
AND MATCH(VCJOBTITLE, LVJOBCOMPANYDESCRIPTION, VCCOMPANYNAME,
VCSALARYDESC, VCCITY, VCQUALIFICATIONREQUIRED,VCJOBREFERENCE,
LVJOBKEYWORDS )
AGAINST ('test' IN NATURAL LANGUAGE MODE)
ORDER BY JOB.ITJOBBAND ASC,
JOB.VCRANKING DESC,
JOB.IDJOB DESC,
JOB.FJOBWEIGHT ASC
LIMIT 50;
Which takes 80- 120 seconds (3X slower )
If I remove ORDER BY it will load in 6 to 10 sec.
Is their any setting Or performance tunning we can apply here?
Edit: using show create table JOB;
CREATE TABLE `JOB` (
`idJob` int(11) NOT NULL AUTO_INCREMENT,
`idEmployer` int(10) DEFAULT NULL,
`vcJobTitle` varchar(255) NOT NULL,
`lvJobCompanyDescription` text,
`idBasket` int(10) DEFAULT NULL,
`dtActualgolive` datetime DEFAULT NULL COMMENT 'whenever job status become online',
`boImmediatelygolive` int(11) NOT NULL,
`dtRequestedgolive` date DEFAULT NULL,
`dtActualend` datetime DEFAULT NULL COMMENT 'when job is set to be archieved',
`dtPlannedend` date NOT NULL COMMENT 'when posting a job this is calculated. batch will look at this date. date on which regular jobs will be archievd, or date on which internship jobs will be changed to listing',
`dtRequestedend` date DEFAULT NULL COMMENT 'exact date on which employer wants jobs to be archieved',
`dtApplicationdeadline` date DEFAULT NULL COMMENT 'job will be converted to listing on this date',
`boDurationinweeks` int(11) NOT NULL COMMENT 'whether employer gave duration in weeks or not',
`itDurationweeks` int(2) DEFAULT NULL,
`boIsoncredit` int(11) NOT NULL,
`dtCreditend` date DEFAULT NULL,
`fJobbaseprice` float NOT NULL,
`fJobpriceafteradminitemdiscount` float NOT NULL COMMENT 'price after admin item discount',
`fJobpriceafterpromodiscount` float NOT NULL COMMENT 'price after promo discount',
`vcPromotioncode` varchar(50) DEFAULT NULL,
`vcCompanyname` varchar(100) NOT NULL,
`vcSalarydesc` varchar(255) NOT NULL,
`vcCity` varchar(100) NOT NULL,
`vcCounty` varchar(120) DEFAULT NULL,
`vcQualificationrequired` text,
`boWorkfromhome` int(11) NOT NULL,
`boResidential` int(11) NOT NULL,
`boIndoor` int(11) NOT NULL,
`boOutdoor` int(11) NOT NULL,
`boIndoorandoutdoor` int(11) NOT NULL,
`itJobstarttype` int(11) DEFAULT NULL COMMENT 'date,immidiate,always recruiting',
`dtJobstart` date DEFAULT NULL,
`itJobendtype` int(11) DEFAULT NULL COMMENT 'date,ongoing',
`dtJobend` date DEFAULT NULL,
`dtFeaturedstart` date DEFAULT NULL,
`dtFeaturedend` date DEFAULT NULL,
`itFulltimeparttime` int(11) NOT NULL COMMENT 'fulltime,parttime,both',
`boEveningtime` int(11) NOT NULL,
`boDaytime` int(11) NOT NULL,
`boWeekend` int(11) NOT NULL,
`boNewsletter` int(11) NOT NULL,
`vcApplyemail` varchar(500) DEFAULT NULL,
`vcApplyphone` varchar(200) DEFAULT NULL,
`vcApplyaddress` varchar(100) DEFAULT NULL,
`vcApplyURL` varchar(500) DEFAULT NULL,
`boRequirephone` int(11) DEFAULT NULL,
`boRequireaddress` int(11) DEFAULT NULL,
`boRequirecv` int(11) DEFAULT NULL,
`boFeatured` int(11) NOT NULL,
`blLogo` longblob,
`itJobstatus` int(1) DEFAULT NULL COMMENT 'online,offline',
`itEmailedtofriendcount` int(5) DEFAULT NULL COMMENT 'keeps count of how many times users clicked on link email this to friend',
`itDeadlinetype` int(1) DEFAULT NULL,
`idInternshiptypemaster` int(11) DEFAULT NULL,
`vcLengthofscheme` varchar(100) DEFAULT NULL,
`dtInternshiplistingend` date DEFAULT NULL COMMENT 'date when internship job will archieve',
`itJobband` int(2) DEFAULT NULL,
`boGraduate` int(11) NOT NULL,
`boInternship` int(11) NOT NULL,
`boGaptemp` int(11) NOT NULL,
`boParttimeholiday` int(11) NOT NULL,
`boEntrylevel` int(11) NOT NULL DEFAULT '0',
`boHundredPercentDiscountApplicable` int(11) NOT NULL,
`vcContactdetails` varchar(100) DEFAULT NULL,
`vcJobreference` varchar(100) DEFAULT NULL COMMENT 'for importing totaljobs feeds',
`vcJoburlparam` varchar(100) DEFAULT NULL COMMENT 'for importing totaljobs url params',
`dtPutInCurrentBand` datetime DEFAULT NULL COMMENT 'date when job is online and put in band 3',
`itInitcountycount` int(2) NOT NULL,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`lvJobkeywords` text,
`fJobweight` decimal(10,2) DEFAULT '1.00' COMMENT 'Holds the job weight ranging between 0 to 1 (1 being the highest weight)',
`dtReinstatedOn` datetime DEFAULT NULL,
`boIncludeiniframe` int(11) DEFAULT NULL,
`boIsAdvertFeatured` int(11) DEFAULT NULL,
`itincludexml` int(1) DEFAULT '1',
`boAddtojobalert` int(11) DEFAULT '1',
`vcRanking` int(11) DEFAULT '999999',
`itIncludeUkptj` int(11) DEFAULT '1',
PRIMARY KEY (`idJob`),
KEY `JOB_I_1` (`idEmployer`,`itJobstatus`),
KEY `JOB_I_2` (`itJobstatus`,`boGaptemp`,`dtJobstart`),
KEY `JOB_I_3` (`itJobstatus`,`boGraduate`,`dtJobstart`),
KEY `JOB_I_4` (`itJobstatus`,`boInternship`,`dtJobstart`),
KEY `JOB_I_5` (`itJobstatus`,`boParttimeholiday`,`dtJobstart`),
KEY `FI_JOB_idOrder_ORDER_idOrder` (`idBasket`),
KEY `FI_JOB_vcPromotioncode_PROMOTION_vcPromotioncode` (`vcPromotioncode`),
KEY `FI_JOB_idInternshiptype_ITTM_idInternshiptypemaster` (`idInternshiptypemaster`),
KEY `JOB_I_6` (`created_at`),
KEY `boEntrylevel` (`boEntrylevel`),
KEY `itJobband` (`itJobband`),
FULLTEXT KEY `index_ft_search` (`vcJobTitle`,`lvJobCompanyDescription`,`vcCompanyname`,`vcSalarydesc`,`vcCity`,`vcQualificationrequired`,`vcJobreference`,`lvJobkeywords`),
CONSTRAINT `fk_JOB_idEmployer_EMPLOYER_idEmployer` FOREIGN KEY (`idEmployer`) REFERENCES `EMPLOYER` (`idEmployer`),
CONSTRAINT `fk_JOB_idInternshiptype_ITTM_idInternshiptypemaster` FOREIGN KEY (`idInternshiptypemaster`) REFERENCES `INTERNSHIPTYPEMASTER` (`idInternshiptypemaster`),
CONSTRAINT `fk_JOB_idOrder_ORDER_idOrder` FOREIGN KEY (`idBasket`) REFERENCES `BASKET` (`idBasket`),
CONSTRAINT `fk_JOB_vcPromotioncode_PROMOTION_vcPromotioncode` FOREIGN KEY (`vcPromotioncode`) REFERENCES `PROMOTION` (`vcPromotioncode`)
) ENGINE=InnoDB AUTO_INCREMENT=1739324 DEFAULT CHARSET=latin1 |
Edit:
Without sort:
With Sort:
I'm not sure why this query is taking 4 minutes to complete:
SELECT
su.sid,u.uid,u.display_name,u.locale
FROM user u
LEFT JOIN subscription_user su ON su.uid = u.uid
ORDER BY u.display_name DESC
LIMIT 0,25;
Well, I know it's due to the order, remove it and it's very fast. If I change to using INNER JOIN instead it's fast but the issue is not all users may be in the subscription_user table.
CREATE TABLE `user` (
`uid` int(11) NOT NULL AUTO_INCREMENT,
`password` varchar(100) DEFAULT NULL,
`user_type` varchar(10) NOT NULL DEFAULT 'user',
`display_name` varchar(50) NOT NULL,
`email` varchar(100) NOT NULL,
`locale` varchar(8) DEFAULT 'en',
`last_login` datetime DEFAULT NULL,
`auth_type` varchar(10) DEFAULT NULL,
`auth_data` varchar(500) DEFAULT NULL,
`inactive` tinyint(4) NOT NULL DEFAULT '0',
`receive_email` tinyint(4) NOT NULL DEFAULT '1',
`stateid` int(10) DEFAULT NULL,
`owner_group_id` int(11) DEFAULT NULL,
`signature` varchar(500) DEFAULT NULL,
`raw_signature` varchar(500) DEFAULT NULL,
`round_robin` smallint(5) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`uid`),
UNIQUE KEY `email` (`email`),
KEY `stateid` (`stateid`) USING BTREE,
KEY `user_type` (`user_type`) USING BTREE,
KEY `name` (`display_name`)
) ENGINE=InnoDB AUTO_INCREMENT=28343 DEFAULT CHARSET=latin1;
CREATE TABLE `subscription_user` (
`sid` varchar(50) NOT NULL,
`uid` int(11) NOT NULL,
`deleted` tinyint(4) NOT NULL DEFAULT '0',
`forum_user` varchar(50) NOT NULL,
PRIMARY KEY (`sid`,`uid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
When you have an SQL query, the index can only really help you if the first column in the index is part of the query.
Your query joins su.uid = u.uid and the optimizer will not be able to use that to reference the first column in the subscription primary key index.
You should either reverse the order of the columns in the primary key, or alternatively, you should add a foreign key index, or an independent index on the uid