I have an event which fires periodically to 'abort' some abandoned games (a simple matching server).
This update is proving very (VERY) slow and I'm looking for advice on doing this better.
Problematic Update:
UPDATE user SET skill=skill+
(SELECT count(participant_1) * 25 FROM matches
WHERE score_2 IS NULL
AND score_2_time IS NOT NULL
AND participant_1=user.id
AND score_2_time < (NOW() - INTERVAL 1 HOUR)
AND status=0);
Matches table:
matches CREATE TABLE `matches` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`match_hash` varchar(64) DEFAULT NULL,
`skill` int(10) unsigned DEFAULT NULL,
`status` int(10) unsigned DEFAULT NULL,
`participant_1` int(10) unsigned DEFAULT NULL,
`score_1` int(10) unsigned DEFAULT NULL,
`score_1_time` timestamp NULL DEFAULT NULL,
`participant_1_rematched` tinyint(4) DEFAULT NULL,
`participant_2` int(10) unsigned DEFAULT NULL,
`score_2` int(10) unsigned DEFAULT NULL,
`score_2_time` timestamp NULL DEFAULT NULL,
`participant_2_rematched` tinyint(4) DEFAULT NULL,
`created_at` timestamp NOT NULL DEFAULT current_timestamp(),
`finished_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=40667 DEFAULT CHARSET=latin1
User table:
user CREATE TABLE `user` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(32) COLLATE utf8_unicode_ci DEFAULT NULL,
`skill` int(10) unsigned DEFAULT NULL,
`created` timestamp NOT NULL DEFAULT current_timestamp(),
PRIMARY KEY (`id`),
UNIQUE KEY `name` (`name`)
) ENGINE=InnoDB AUTO_INCREMENT=1876 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
Any guidance is greatly appreciated.
You need more indexes on the matches table, certainly at least participant_1, and whatever else is mentioned in the WHERE clause that helps. Probably participant_1 should be a foreign key into user.id for integrity reasons.
Try this query:
update user a join (SELECT participant_1,count(participant_1) * 25 as count FROM matches
WHERE score_2 IS NULL
AND score_2_time IS NOT NULL
AND score_2_time < (NOW() - INTERVAL 1 HOUR)
AND status=0 group by participant_1) b on a.id=b.participant_1 SET a.skill=a.skill+b.count
Related
Is there a way I can speed this up? Right now it's taking an unbelievably insane amount of time to query.
SELECT trades.*, trader1.user_name as trader1_name,
trader2.user_name as trader2_name FROM trades
LEFT JOIN logs_players trader1 ON trader1.user_id = trader1_account_id
LEFT JOIN logs_players trader2 ON trader2.user_id = trader2_account_id
ORDER BY time_added
LIMIT 20 OFFSET 0;
I've done as much as I could in terms of searching online for a solution. Or even just trying to get some more information why it's taking so long to execute.
The query takes about 45 seconds or so to complete.
Create statements:
CREATE TABLE `trades` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`trader1_account_id` int(11) DEFAULT NULL,
`trader2_account_id` int(11) DEFAULT NULL,
`trader1_value` bigint(20) DEFAULT NULL,
`trader2_value` bigint(20) DEFAULT NULL,
`trader1_ip` varchar(16) DEFAULT NULL,
`trader2_ip` varchar(16) DEFAULT NULL,
`world` int(11) DEFAULT NULL,
`x` int(11) DEFAULT NULL,
`z` int(11) DEFAULT NULL,
`level` int(11) DEFAULT NULL,
`trader1_user` varchar(12) DEFAULT NULL,
`trader2_user` varchar(12) DEFAULT NULL,
`time_added` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8
CREATE TABLE `logs_players` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) DEFAULT NULL,
`user_name` varchar(20) DEFAULT NULL,
`world_stage` varchar(20) DEFAULT NULL,
`world_type` varchar(20) DEFAULT NULL,
`bank` longtext,
`inventory` longtext,
`equipment` longtext,
`total_wealth` mediumtext,
`total_play_time` mediumtext,
`rights` int(11) DEFAULT NULL,
`icon` int(11) DEFAULT NULL,
`ironmode` int(11) DEFAULT NULL,
`x` int(11) DEFAULT NULL,
`z` int(11) DEFAULT NULL,
`level` int(11) DEFAULT NULL,
`last_ip` varchar(16) DEFAULT NULL,
`last_online` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`muted_until` timestamp NULL DEFAULT NULL,
`banned_until` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8
I filled a sample database with 10k rows each, and found that a few indexes were what you needed:
ALTER TABLE `logs_players` ADD INDEX(`user_id`);
ALTER TABLE `trades` ADD INDEX(`time_added`);
The main index we need is an index on user_id. Changing that we went from a query time of 20.1390 seconds, to 0.0130 seconds:
We can even get that down further, by adding an index on time_added to make sorting a lot faster, now we ended up with an impressive query time:
Do some research on indexes! A simple EXPLAIN query would show you that you're using filesort (Which is rather bad!):
After indexes, this looks a lot better:
I have a fairly simple query in MySQL but it is taking around 170 minutes to execute.
Can anyone help me here? I am tired of applying indexes on various keys but no benefit.
Update
H20_AUDIENCE_ADDRESS_LOG L
Join
TEMP_V_3064446579 T
Using
( ZS_AUDIENCE_ID, ZS_SOURCE_OBJECT_ID, ZS_ADDRESS_TYPE_ID )
Set
ZS_ACTIVE_PERIOD_END_DT = '2015-08-14 15:05:48',
ZS_IS_ACTIVE_PERIOD = False
Where
ZS_IS_ACTIVE_PERIOD = True
And
L.ZS_ADDRESS_ID <> T.ZS_ADDRESS_ID
And
T.ZS_SOURCE_TIMESTAMP > L.ZS_SOURCE_TIMESTAMP;
Creates:
CREATE TABLE `H20_AUDIENCE_ADDRESS_LOG` (
`ZS_AUDIENCE_ADDRESS_LOG_ID` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`ZS_AUDIENCE_ID` bigint(20) unsigned NOT NULL,
`ZS_SOURCE_OBJECT_ID` int(10) unsigned NOT NULL,
`ZS_INSERT_DT` datetime NOT NULL,
`ZS_ADDRESS_TYPE_ID` tinyint(3) unsigned NOT NULL,
`ZS_ADDRESS_ID` bigint(20) unsigned NOT NULL,
`ZS_SOURCE_TIMESTAMP` datetime NOT NULL,
`ZS_ACTIVE_PERIOD_START_DT` datetime DEFAULT NULL,
`ZS_ACTIVE_PERIOD_END_DT` datetime DEFAULT NULL,
`ZS_IS_ACTIVE_PERIOD` bit(1) DEFAULT NULL,
`ZS_ACTIVE_PRIORITY_PERIOD_START_DT` datetime DEFAULT NULL,
`ZS_ACTIVE_PRIORITY_PERIOD_END_DT` datetime DEFAULT NULL,
`ZS_IS_ACTIVE_PRIORITY_PERIOD` bit(1) DEFAULT NULL,
PRIMARY KEY (`ZS_AUDIENCE_ADDRESS_LOG_ID`),
KEY `IX_H20_AUDIENCE_ADDRESS_LOG` (`ZS_AUDIENCE_ID`,`ZS_SOURCE_OBJECT_ID`,`ZS_ADDRESS_TYPE_ID`,`ZS_ADDRESS_ID`),
KEY `IX_ADDRESS_ID` (`ZS_ADDRESS_ID`,`ZS_IS_ACTIVE_PERIOD`)
) ENGINE=InnoDB AUTO_INCREMENT=22920801 DEFAULT CHARSET=utf8;
CREATE TABLE `TEMP_V_3064446579` (
`ZS_AUDIENCE_ID` bigint(20) unsigned NOT NULL,
`ZS_SOURCE_OBJECT_ID` int(10) unsigned NOT NULL,
`ZS_ADDRESS_TYPE_ID` tinyint(3) unsigned NOT NULL,
`ZS_ADDRESS_ID` bigint(20) unsigned NOT NULL,
`ZS_SOURCE_TIMESTAMP` datetime NOT NULL,
UNIQUE KEY `IX_TEMP_V_3064446579` (`ZS_AUDIENCE_ID`,`ZS_SOURCE_OBJECT_ID`,`ZS_ADDRESS_TYPE_ID`,`ZS_ADDRESS_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Both tables circa 3m records
Something like this should work:
UPDATE
`H20_AUDIENCE_ADDRESS_LOG` `L`
SET
`ZS_ACTIVE_PERIOD_END_DT` = '2015-08-14 15:05:48',
`ZS_IS_ACTIVE_PERIOD` = False
WHERE
`ZS_IS_ACTIVE_PERIOD` = True AND
EXISTS (
SELECT
1
FROM
`TEMP_V_3064446579` `T`
WHERE
`L`.`ZS_ADDRESS_ID` <> `T`.`ZS_ADDRESS_ID` AND
`T`.`ZS_SOURCE_TIMESTAMP` > `L`.`ZS_SOURCE_TIMESTAMP`
LIMIT 1
);
(The ZS_ makes the SQL hard to read; suggest removing it.)
In TEMP_V_3064446579, change UNIQUE to PRIMARY.
Change
KEY `IX_H20_AUDIENCE_ADDRESS_LOG` (`ZS_AUDIENCE_ID`,`ZS_SOURCE_OBJECT_ID`,
`ZS_ADDRESS_TYPE_ID`,`ZS_ADDRESS_ID`)
to
KEY `IX_H20_AUDIENCE_ADDRESS_LOG` (`ZS_AUDIENCE_ID`,`ZS_SOURCE_OBJECT_ID`,
`ZS_ADDRESS_TYPE_ID`,`ZS_ADDRESS_ID`,
`ZS_SOURCE_TIMESTAMP`)
If you have a new enough version, please provide EXPLAIN UPDATE .... If not, please provide EXPLAIN SELECT ... where the SELECT is derived from the UPDATE, but without the SET.
I need to search records in the table which contains millions of records. I have recently updated Mysql version from 5.1 to 5.6.
I was using like in query which was taking around 15 sec to 30 sec.
Currently I have modified query to use MATCH... AGAINST feature of mysql 5.6 INNODB. My query goes like this.
SELECT JOB.IDJOB, JOB.IDEMPLOYER .....
FROM JOB
WHERE ( ( JOB.ITJOBSTARTTYPE=2 AND JOB.DTPLANNEDEND >= '2015-07-06' )
OR ( JOB.ITJOBSTARTTYPE=1
AND ( ( JOB.ITJOBENDTYPE=2 ) OR ( JOB.DTJOBEND>='2015-07-06') )
)
OR ( JOB.ITJOBSTARTTYPE=3
AND ( (JOB.DTJOBEND >='2015-07-06') OR (JOB.ITJOBENDTYPE=2) )
)
)
AND MATCH(VCJOBTITLE, LVJOBCOMPANYDESCRIPTION, VCCOMPANYNAME,
VCSALARYDESC, VCCITY, VCQUALIFICATIONREQUIRED,VCJOBREFERENCE,
LVJOBKEYWORDS )
AGAINST ('test' IN NATURAL LANGUAGE MODE)
ORDER BY JOB.ITJOBBAND ASC,
JOB.VCRANKING DESC,
JOB.IDJOB DESC,
JOB.FJOBWEIGHT ASC
LIMIT 50;
Which takes 80- 120 seconds (3X slower )
If I remove ORDER BY it will load in 6 to 10 sec.
Is their any setting Or performance tunning we can apply here?
Edit: using show create table JOB;
CREATE TABLE `JOB` (
`idJob` int(11) NOT NULL AUTO_INCREMENT,
`idEmployer` int(10) DEFAULT NULL,
`vcJobTitle` varchar(255) NOT NULL,
`lvJobCompanyDescription` text,
`idBasket` int(10) DEFAULT NULL,
`dtActualgolive` datetime DEFAULT NULL COMMENT 'whenever job status become online',
`boImmediatelygolive` int(11) NOT NULL,
`dtRequestedgolive` date DEFAULT NULL,
`dtActualend` datetime DEFAULT NULL COMMENT 'when job is set to be archieved',
`dtPlannedend` date NOT NULL COMMENT 'when posting a job this is calculated. batch will look at this date. date on which regular jobs will be archievd, or date on which internship jobs will be changed to listing',
`dtRequestedend` date DEFAULT NULL COMMENT 'exact date on which employer wants jobs to be archieved',
`dtApplicationdeadline` date DEFAULT NULL COMMENT 'job will be converted to listing on this date',
`boDurationinweeks` int(11) NOT NULL COMMENT 'whether employer gave duration in weeks or not',
`itDurationweeks` int(2) DEFAULT NULL,
`boIsoncredit` int(11) NOT NULL,
`dtCreditend` date DEFAULT NULL,
`fJobbaseprice` float NOT NULL,
`fJobpriceafteradminitemdiscount` float NOT NULL COMMENT 'price after admin item discount',
`fJobpriceafterpromodiscount` float NOT NULL COMMENT 'price after promo discount',
`vcPromotioncode` varchar(50) DEFAULT NULL,
`vcCompanyname` varchar(100) NOT NULL,
`vcSalarydesc` varchar(255) NOT NULL,
`vcCity` varchar(100) NOT NULL,
`vcCounty` varchar(120) DEFAULT NULL,
`vcQualificationrequired` text,
`boWorkfromhome` int(11) NOT NULL,
`boResidential` int(11) NOT NULL,
`boIndoor` int(11) NOT NULL,
`boOutdoor` int(11) NOT NULL,
`boIndoorandoutdoor` int(11) NOT NULL,
`itJobstarttype` int(11) DEFAULT NULL COMMENT 'date,immidiate,always recruiting',
`dtJobstart` date DEFAULT NULL,
`itJobendtype` int(11) DEFAULT NULL COMMENT 'date,ongoing',
`dtJobend` date DEFAULT NULL,
`dtFeaturedstart` date DEFAULT NULL,
`dtFeaturedend` date DEFAULT NULL,
`itFulltimeparttime` int(11) NOT NULL COMMENT 'fulltime,parttime,both',
`boEveningtime` int(11) NOT NULL,
`boDaytime` int(11) NOT NULL,
`boWeekend` int(11) NOT NULL,
`boNewsletter` int(11) NOT NULL,
`vcApplyemail` varchar(500) DEFAULT NULL,
`vcApplyphone` varchar(200) DEFAULT NULL,
`vcApplyaddress` varchar(100) DEFAULT NULL,
`vcApplyURL` varchar(500) DEFAULT NULL,
`boRequirephone` int(11) DEFAULT NULL,
`boRequireaddress` int(11) DEFAULT NULL,
`boRequirecv` int(11) DEFAULT NULL,
`boFeatured` int(11) NOT NULL,
`blLogo` longblob,
`itJobstatus` int(1) DEFAULT NULL COMMENT 'online,offline',
`itEmailedtofriendcount` int(5) DEFAULT NULL COMMENT 'keeps count of how many times users clicked on link email this to friend',
`itDeadlinetype` int(1) DEFAULT NULL,
`idInternshiptypemaster` int(11) DEFAULT NULL,
`vcLengthofscheme` varchar(100) DEFAULT NULL,
`dtInternshiplistingend` date DEFAULT NULL COMMENT 'date when internship job will archieve',
`itJobband` int(2) DEFAULT NULL,
`boGraduate` int(11) NOT NULL,
`boInternship` int(11) NOT NULL,
`boGaptemp` int(11) NOT NULL,
`boParttimeholiday` int(11) NOT NULL,
`boEntrylevel` int(11) NOT NULL DEFAULT '0',
`boHundredPercentDiscountApplicable` int(11) NOT NULL,
`vcContactdetails` varchar(100) DEFAULT NULL,
`vcJobreference` varchar(100) DEFAULT NULL COMMENT 'for importing totaljobs feeds',
`vcJoburlparam` varchar(100) DEFAULT NULL COMMENT 'for importing totaljobs url params',
`dtPutInCurrentBand` datetime DEFAULT NULL COMMENT 'date when job is online and put in band 3',
`itInitcountycount` int(2) NOT NULL,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`lvJobkeywords` text,
`fJobweight` decimal(10,2) DEFAULT '1.00' COMMENT 'Holds the job weight ranging between 0 to 1 (1 being the highest weight)',
`dtReinstatedOn` datetime DEFAULT NULL,
`boIncludeiniframe` int(11) DEFAULT NULL,
`boIsAdvertFeatured` int(11) DEFAULT NULL,
`itincludexml` int(1) DEFAULT '1',
`boAddtojobalert` int(11) DEFAULT '1',
`vcRanking` int(11) DEFAULT '999999',
`itIncludeUkptj` int(11) DEFAULT '1',
PRIMARY KEY (`idJob`),
KEY `JOB_I_1` (`idEmployer`,`itJobstatus`),
KEY `JOB_I_2` (`itJobstatus`,`boGaptemp`,`dtJobstart`),
KEY `JOB_I_3` (`itJobstatus`,`boGraduate`,`dtJobstart`),
KEY `JOB_I_4` (`itJobstatus`,`boInternship`,`dtJobstart`),
KEY `JOB_I_5` (`itJobstatus`,`boParttimeholiday`,`dtJobstart`),
KEY `FI_JOB_idOrder_ORDER_idOrder` (`idBasket`),
KEY `FI_JOB_vcPromotioncode_PROMOTION_vcPromotioncode` (`vcPromotioncode`),
KEY `FI_JOB_idInternshiptype_ITTM_idInternshiptypemaster` (`idInternshiptypemaster`),
KEY `JOB_I_6` (`created_at`),
KEY `boEntrylevel` (`boEntrylevel`),
KEY `itJobband` (`itJobband`),
FULLTEXT KEY `index_ft_search` (`vcJobTitle`,`lvJobCompanyDescription`,`vcCompanyname`,`vcSalarydesc`,`vcCity`,`vcQualificationrequired`,`vcJobreference`,`lvJobkeywords`),
CONSTRAINT `fk_JOB_idEmployer_EMPLOYER_idEmployer` FOREIGN KEY (`idEmployer`) REFERENCES `EMPLOYER` (`idEmployer`),
CONSTRAINT `fk_JOB_idInternshiptype_ITTM_idInternshiptypemaster` FOREIGN KEY (`idInternshiptypemaster`) REFERENCES `INTERNSHIPTYPEMASTER` (`idInternshiptypemaster`),
CONSTRAINT `fk_JOB_idOrder_ORDER_idOrder` FOREIGN KEY (`idBasket`) REFERENCES `BASKET` (`idBasket`),
CONSTRAINT `fk_JOB_vcPromotioncode_PROMOTION_vcPromotioncode` FOREIGN KEY (`vcPromotioncode`) REFERENCES `PROMOTION` (`vcPromotioncode`)
) ENGINE=InnoDB AUTO_INCREMENT=1739324 DEFAULT CHARSET=latin1 |
Edit:
Without sort:
With Sort:
My database contains around 20 tables that holds a user's information. For example it has
Personal : hold user personal info
Documents : uploaded files
Activities :
Etc..
Every table contains a user_Id column for wiring them together ( one to many relationship), along with different table specific columns and constraints.
My question is how should I fetch all data for a single user from all these tables ?
Currently when ever I load user , application need to do
Select * from table1 where user_Id = x;
Select * from table2 where user_Id = x;
Select * from table3 where user_Id = x;
..etc
Since I'm using php (oop) its not a bad thing as every table has its own model that retrieve it. Yet I'm worried about performance as I currently run over 20 queries every time I load page. And since these data are very dynamically updated. Caching isn't helping much.
So what is the best methodology to fix this ?
example of table structures
CREATE TABLE IF NOT EXISTS `documents` (
`id` int(10) unsigned NOT NULL,
`user_id` int(10) unsigned NOT NULL,
`collection` text NOT NULL,
`photo_date` timestamp NULL DEFAULT NULL,
`gallery` varchar(50) NOT NULL,
`cover` int(1) DEFAULT NULL,
`upload_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=8 ;
CREATE TABLE IF NOT EXISTS `problemlist` (
`id` int(10) unsigned NOT NULL,
`user_id` int(10) unsigned NOT NULL,
`visit_id` int(10) unsigned NOT NULL,
`pt_id` int(10) unsigned NOT NULL,
`problem` varchar(200) NOT NULL,
`severity` int(2) unsigned NOT NULL,
`note` text,
`solved` int(1) unsigned NOT NULL,
`datetime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=6 ;
CREATE TABLE IF NOT EXISTS `visits` (
`id` int(10) unsigned NOT NULL,
`pt_id` int(10) unsigned NOT NULL,
`user_id` int(10) unsigned NOT NULL,
`visit_date` timestamp NULL DEFAULT NULL,
`visit_end` timestamp NULL DEFAULT NULL,
`complain` varchar(250) DEFAULT NULL,
`dx` varchar(200) DEFAULT NULL,
`note` text,
`stats` enum('booked','waitting','finished','noshow','canceled','inroom') NOT NULL,
`deleted` int(1) DEFAULT NULL,
`booked_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`arrived_at` timestamp NULL DEFAULT NULL,
`started_at` timestamp NULL DEFAULT NULL,
`checkout_at` timestamp NULL DEFAULT NULL,
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=212 ;
I have a table. (Code taken from table generation code, I did not write this)
DROP TABLE IF EXISTS `CatalogueBasket`;
CREATE TABLE `CatalogueBasket` (
`ID` int(11) NOT NULL auto_increment,
`Shopper` char(35) NOT NULL default '',
`ItemLink` int(11) NOT NULL default '0',
`Quantity` int(11) NOT NULL default '0',
`Created` datetime NOT NULL default '0000-00-00 00:00:00',
`ExpectedDelivery1` datetime default NULL,
`ExpectedDelivery2` datetime default NULL,
`Comments` char(255) default NULL,
`Status` int(10) unsigned default NULL,
`QuantityShipped` int(10) unsigned default NULL,
`HarmonyNumber` int(10) unsigned default NULL,
`StartDate` datetime default NULL,
KEY `ID` (`ID`),
KEY `Shopper` (`Shopper`),
KEY `ItemLink` (`ItemLink`),
KEY `Quantity` (`Quantity`),
KEY `Created` (`Created`)
) TYPE=MyISAM;
When trying to insert a new Row at the end of this table I am getting the following message.
Duplicate entry '116604' for key 1
The insert statement is:
INSERT INTO CatalogueBasket (Shopper,ItemLink,Quantity,Created, Status, StartDate)
VALUES ('0.80916300 1338507348',58825,1,'2012-06-01 09:58:23', 0, '0-0-0')
I'm assuming it is talking about the ID column.
If I run the following query I get 116603 as the last key
SELECT * FROM `CatalogueBasket` order by ID desc limit 1
Any insight / help into this is appreciated.