mysql NOT EXISTS, very slow - How to improve? - mysql

I've got a small question about trying to ompimise this command I'm running on a table in my database. The table has over 934,836 rows and grows daily by about 12,000.
It holds snapshots of tanks taken daily. What I'm trying to achieve is to see the difference in the snapshots. i.e. see if the player has purchased any new tanks.
The actual snapshot data per account_id is only 100 to 250 rows.
Table wot_snapshots:
CREATE TABLE `wot_snapshots` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`snapshot` varchar(30) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=95 DEFAULT CHARSET=latin1
Table wot_tanks_all:
CREATE TABLE `wot_tanks_all` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`snapshot` int(11) NOT NULL,
`account_id` int(11) NOT NULL,
`tank_id` int(6) NOT NULL,
`wn8` float DEFAULT NULL,
`spotted` int(11) NOT NULL,
`avg_damage_blocked` decimal(6,2) NOT NULL,
`capture_points` int(11) NOT NULL,
`explosion_hits` int(11) NOT NULL,
`piercings` int(11) NOT NULL,
`xp` int(11) NOT NULL,
`survived_battles` int(11) NOT NULL,
`dropped_capture_points` int(11) NOT NULL,
`damage_dealt` int(11) NOT NULL,
`hits_percents` int(11) NOT NULL,
`draws` int(11) NOT NULL,
`battles` int(11) NOT NULL,
`damage_received` int(11) NOT NULL,
`frags` int(11) NOT NULL,
`direct_hits_received` int(11) NOT NULL,
`hits` int(11) NOT NULL,
`battle_avg_xp` int(11) NOT NULL,
`wins` int(11) NOT NULL,
`losses` int(11) NOT NULL,
`piercings_received` int(11) NOT NULL,
`no_damage_direct_hits_received` int(11) NOT NULL,
`shots` int(11) NOT NULL,
`explosion_hits_received` int(11) NOT NULL,
`tanking_factor` decimal(2,2) NOT NULL,
`mark_of_mastery` int(1) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=934837 DEFAULT CHARSET=utf8
Query:
SELECT t1.tank_id, wot_tanks.short_name_i18n FROM wot_tanks_all t1
JOIN wot_tanks ON t1.tank_id = wot_tanks.tank_id
WHERE NOT EXISTS (SELECT tank_id
FROM wot_tanks_all t2
WHERE t1.tank_id = t2.tank_id AND account_id = 527080765 AND snapshot = 60)
AND account_id = 527080765 AND snapshot = 93
Output:
tank_id short_name_i18n
8465 Panther II
53505 T-127
54865 Light VIC
Its currently taking around 30 seconds to run. Am I best to do this all in mysql or offload some of it to PHP?
Any advice and help is greatly appreciated
Thanks,
Jason
Edit: This has just been put together from Google. Still learning!

There is no index on tank id so joining on it will be very slow,I`ve rewritten it to eliminate the self join
SELECT t1.tank_id, wot_tanks.short_name_i18n FROM wot_tanks_all t1
JOIN wot_tanks
ON t1.tank_id = wot_tanks.tank_id
GROUP BY t1.tank_id,wot_tanks.short_name_i18n
HAVING SUM(account_id = 527080765 AND snapshot = 60)=0
AND SUM(account_id = 527080765 AND snapshot = 93)>0
These indexes will speed things up
ALTER TABLE wot_tanks_all ADD KEY(account_id ,snapshot )
ALTER TABLE wot_tanks_all ADD KEY(tank_id )

Related

MySql Using filesort when i using a group by

I have a little problem with optimizing a query, I have 2 tables, one which records the participation (participation) in a quiz, and the other which records the answer to each question (participation_rep), participation is linked to the campaign table.
SELECT count(DISTINCT p.id) as number_of_participation
FROM participation_rep prep
INNER JOIN participation p
ON p.id = prep.id_participation
AND p.trash <> 1
WHERE prep.id_question IN (780,787,794,801,809)
AND prep.trash <> 1
GROUP BY pp.id_campaign
Explain of the query
And the problem is that this request is very heavy to execute when there is a lot of data which is concerned by the request and I do not know how to optimize it.
This query take 30-50ms to execute.
Structure of table participation :
CREATE TABLE IF NOT EXISTS `participation` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`id_campagne` int(11) NOT NULL,
`id_identifiant` int(11) DEFAULT NULL,
`firstname` varchar(255) DEFAULT NULL,
`surname` varchar(255) DEFAULT NULL,
`email` varchar(255) DEFAULT NULL,
`date_p` date NOT NULL,
`hour_p` time NOT NULL,
`comment` text,
`trash` tinyint(1) DEFAULT '0',
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Structure of table participation_rep :
CREATE TABLE IF NOT EXISTS `participation_rep` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`id_participation` int(11) NOT NULL,
`id_question` int(11) NOT NULL,
`id_rep` int(11) NOT NULL,
`trash` tinyint(1) DEFAULT '0',
PRIMARY KEY (`id`),
UNIQUE KEY `id_participation` (`id_participation`,`id_question`,`id_reponse`),
KEY `id_question` (`id_question`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

MYSQL: Left Join is very slow

I'm using MySQl for my database and I have three tables where I want to join them using left join but the performance are very slow.
Below are the tables:
CREATE TABLE IF NOT EXISTS `register_doctor` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`doc_title` int(11) NOT NULL,
`first_name` varchar(35) NOT NULL,
`last_name` varchar(35) DEFAULT NULL,
`gender` int(11) NOT NULL,
`city_id` int(11) NOT NULL,
`province_id` int(11) NOT NULL,
`specialty_id` int(11) NOT NULL,
`status` int(11) NOT NULL COMMENT '0 = Pending; 1 = Verified, 2 = Not Reg Yet, 3 = Pending Approval',
`str_number` char(6) DEFAULT NULL,
`editted_by` int(11) DEFAULT NULL,
`editted_date` bigint(20) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `city_id` (`city_id`),
KEY `specialty_id` (`specialty_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=10267 ;
CREATE TABLE IF NOT EXISTS `ref_doctor_practice_place` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`doctor_id` int(11) NOT NULL,
`practice_place_id` int(11) NOT NULL,
`is_primary` int(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `doctor_id_2` (`doctor_id`,`practice_place_id`),
KEY `doctor_id` (`doctor_id`),
KEY `practice_place_id` (`practice_place_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=23677 ;
CREATE TABLE IF NOT EXISTS `practice_place` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(75) NOT NULL,
`statement` text,
`address` varchar(200) NOT NULL,
`phone` varchar(15) NOT NULL,
`fax` varchar(15) NOT NULL,
`email` varchar(50) NOT NULL,
`village_id` varchar(50) NOT NULL,
`sub_district_id` varchar(50) NOT NULL,
`province_id` varchar(50) NOT NULL,
`zipcode` varchar(10) NOT NULL,
`website` varchar(50) NOT NULL,
`latitude` double NOT NULL,
`longitude` double NOT NULL,
`type` int(11) NOT NULL,
`managed_by` int(11) DEFAULT '0',
`doctor_group_id` int(11) NOT NULL,
`category` varchar(50) NOT NULL,
`photo_file` char(36) NOT NULL,
`is_branch` int(11) NOT NULL,
`parent_id` int(11) NOT NULL,
`editted_by` int(11) NOT NULL,
`editted_date` bigint(20) NOT NULL,
`status` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `village_id` (`village_id`),
KEY `doctor_group_id` (`doctor_group_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=24182 ;
My query is like this:
SELECT SQL_CALC_FOUND_ROWS RD.id as rd_id
, RD.first_name
, RD.last_name
, RD.gender
, RD.str_number
, GROUP_CONCAT(DISTINCT PP.type SEPARATOR '|') as pp_type
FROM register_doctor RD
LEFT
JOIN ref_doctor_practice_place RDPP
ON RDPP.doctor_id = RD.id
LEFT
JOIN practice_place PP
ON PP.id = RDPP.practice_place_id
GROUP
BY RD.id
ORDER
BY RD.id DESC
LIMIT 0,25
Can anyone help me about this? Many thanks.
As requested by Strawberry, here I put the result of using EXPLAIN:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE RD index PRIMARY,city_id PRIMARY 4 NULL 15 NULL
1 SIMPLE RDPP ref doctor_id doctor_id 4 k6064619_lokadok.RD.id 1 NULL
1 SIMPLE PP eq_ref PRIMARY,id PRIMARY 4 k6064619_lokadok.RDPP.practice_place_id 1 NULL
I'm sorry guys. I should have posted the real query. The left join acutally is like this:
LEFT JOIN ref_doctor_practice_place RDPP ON **ABS(RDPP.doctor_id) = RD.id**
I thought the ABS didn't really matter so I erase it to make it more straight forward. But actually this is the culprit.
Nothing wrong with my query. So case close. Thanks for any attempt to help me. Appreciate it.

mysql best practice to concentrate data from multiple tables with different table design

My database contains around 20 tables that holds a user's information. For example it has
Personal : hold user personal info
Documents : uploaded files
Activities :
Etc..
Every table contains a user_Id column for wiring them together ( one to many relationship), along with different table specific columns and constraints.
My question is how should I fetch all data for a single user from all these tables ?
Currently when ever I load user , application need to do
Select * from table1 where user_Id = x;
Select * from table2 where user_Id = x;
Select * from table3 where user_Id = x;
..etc
Since I'm using php (oop) its not a bad thing as every table has its own model that retrieve it. Yet I'm worried about performance as I currently run over 20 queries every time I load page. And since these data are very dynamically updated. Caching isn't helping much.
So what is the best methodology to fix this ?
example of table structures
CREATE TABLE IF NOT EXISTS `documents` (
`id` int(10) unsigned NOT NULL,
`user_id` int(10) unsigned NOT NULL,
`collection` text NOT NULL,
`photo_date` timestamp NULL DEFAULT NULL,
`gallery` varchar(50) NOT NULL,
`cover` int(1) DEFAULT NULL,
`upload_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=8 ;
CREATE TABLE IF NOT EXISTS `problemlist` (
`id` int(10) unsigned NOT NULL,
`user_id` int(10) unsigned NOT NULL,
`visit_id` int(10) unsigned NOT NULL,
`pt_id` int(10) unsigned NOT NULL,
`problem` varchar(200) NOT NULL,
`severity` int(2) unsigned NOT NULL,
`note` text,
`solved` int(1) unsigned NOT NULL,
`datetime` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=6 ;
CREATE TABLE IF NOT EXISTS `visits` (
`id` int(10) unsigned NOT NULL,
`pt_id` int(10) unsigned NOT NULL,
`user_id` int(10) unsigned NOT NULL,
`visit_date` timestamp NULL DEFAULT NULL,
`visit_end` timestamp NULL DEFAULT NULL,
`complain` varchar(250) DEFAULT NULL,
`dx` varchar(200) DEFAULT NULL,
`note` text,
`stats` enum('booked','waitting','finished','noshow','canceled','inroom') NOT NULL,
`deleted` int(1) DEFAULT NULL,
`booked_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`arrived_at` timestamp NULL DEFAULT NULL,
`started_at` timestamp NULL DEFAULT NULL,
`checkout_at` timestamp NULL DEFAULT NULL,
) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=212 ;

How can I optimize this MySQL query to execute fastar?

this is my query
SELECT * FROM ".TB_PREFIX."wdata
left JOIN ".TB_PREFIX."vdata
ON ".TB_PREFIX."vdata.wref = ".TB_PREFIX."wdata.id
where ".TB_PREFIX."wdata.id = $id
and this is the result of slow query:
Query_time: 0.005000 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 21184
How can I optimize this code to execute faster?
EDIT:
tables structure:
CREATE TABLE IF NOT EXISTS `tss1_wdata` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`fieldtype` tinyint(2) unsigned NOT NULL,
`oasistype` tinyint(2) unsigned NOT NULL,
`x` smallint(3) NOT NULL,
`y` smallint(3) NOT NULL,
`occupied` tinyint(1) NOT NULL,
`image` char(3) COLLATE utf8_persian_ci NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `id` (`id`),
KEY `id_2` (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=40402 ;
CREATE TABLE IF NOT EXISTS `tss1_vdata` (
`wref` int(11) unsigned NOT NULL,
`owner` int(11) unsigned NOT NULL,
`name` char(100) COLLATE utf8_persian_ci NOT NULL,
`capital` tinyint(1) unsigned NOT NULL,
`pop` mediumint(11) unsigned NOT NULL,
`cp` mediumint(11) unsigned NOT NULL,
`celebration` int(11) NOT NULL DEFAULT '0',
`type` int(11) NOT NULL DEFAULT '0',
`wood` float(12,2) NOT NULL,
`clay` float(12,2) NOT NULL,
`iron` float(12,2) NOT NULL,
`maxstore` int(11) unsigned NOT NULL,
`crop` float(12,2) NOT NULL,
`maxcrop` int(11) unsigned NOT NULL,
`lastupdate` int(11) unsigned NOT NULL,
`loyalty` tinyint(3) NOT NULL DEFAULT '100',
`exp1` int(11) NOT NULL,
`exp2` int(11) NOT NULL,
`exp3` int(11) NOT NULL,
`created` int(11) NOT NULL,
`natar` tinyint(1) unsigned NOT NULL,
`starv` int(11) unsigned NOT NULL,
`starvupdate` int(11) unsigned NOT NULL,
PRIMARY KEY (`wref`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
You are looking for a specific ID, so you can limit the result to 1 row. Also, you can replace wdata.id with the actual id in the ON statement.
SELECT * FROM ".TB_PREFIX."wdata
LEFT JOIN ".TB_PREFIX."vdata
ON ".TB_PREFIX."vdata.wref = '$id'
WHERE ".TB_PREFIX."wdata.id = '$id'
LIMIT 1
You have the right indices, so there's nothing to be done there.
It looks like you have all necessary indexes and query itself is built just right, i.e. there is nothing to improve in SQL. Actually, I would not consider 5ms query as a slow one
If it is critical query executed thousands times a second, you still can cache result on application level or check NoSQL solution. I personally would try to cache it first.
Try to avoid JOIN table, use short and simple query (single) and store results in php array or mysql memory, make sure all the conditions in mysql fields are indexed properly.
May refer mysqlperformanceblog.com for more mysql related articles.

Need suggestion on optimization of MYSQL query

SELECT `tb1`.`id`
FROM `table1` as tb1
INNER JOIN `table2` as tb2 ON tb1.id = tb2.id
INNER JOIN `table3` as tb3 ON tb1.id = tb3.id
WHERE (tb1.item_id = '1')
AND (tb1.user_id = '20')
AND (tb1.type IN ('UPDATE1','UPDATE2','UPDATE3'))
AND (tb1.status = 'DELIVERED')
ORDER BY tb1.date DESC
LIMIT 100
CREATE TABLE `table1` (
`id` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT,
`user_id` INT(11) UNSIGNED NOT NULL,
`item_id` INT(11) UNSIGNED NULL DEFAULT NULL,
`source` ENUM('CPAS','UNIQUE_KEY','BILLING_PARTNER','GAME','MERCURY') NOT NULL,
`date` DATETIME NOT NULL,
`status` ENUM('PENDING','DELIVERED','FAILED','REFUNDED') NOT NULL,
`source_transaction_id` VARCHAR(127) NULL DEFAULT NULL,
`type` ENUM('UPDATE1','UPDATE2','UPDATE3','UPDATE4') NULL DEFAULT NULL,
PRIMARY KEY (`id`)
);
CREATE TABLE `table2` (
`id_p` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT,
`id` INT(11) UNSIGNED NOT NULL,
`amount` DECIMAL(18,2) NOT NULL,
`old_balance` DECIMAL(18,2) NULL DEFAULT NULL,
`description` VARCHAR(255) NULL DEFAULT NULL,
`date` DATETIME NULL DEFAULT NULL,
`wallet_currency_id` INT(11) NULL DEFAULT NULL,
`wallet_currency_code` VARCHAR(50) NULL DEFAULT NULL,
`wallet_currency_name` VARCHAR(100) NULL DEFAULT NULL,
`type` ENUM('GAIN','SPENT') NULL DEFAULT NULL,
PRIMARY KEY (`id_p`),
INDEX `id` (`id`)
)
CREATE TABLE `table3` (
`id_p` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT,
`id` INT(11) UNSIGNED NOT NULL,
`amount` DECIMAL(18,2) NOT NULL,
`old_balance` DECIMAL(18,2) NULL DEFAULT NULL,
`description` VARCHAR(255) NULL DEFAULT NULL,
`date` DATETIME NULL DEFAULT NULL,
`wallet_currency_id` INT(11) NULL DEFAULT NULL,
`wallet_currency_code` VARCHAR(50) NULL DEFAULT NULL,
`wallet_currency_name` VARCHAR(100) NULL DEFAULT NULL,
`type` ENUM('GAIN','SPENT') NULL DEFAULT NULL,
PRIMARY KEY (`id_p`),
INDEX `id` (`id`)
)
What optimization possible on the above query.
table1 contains more than 500000 rows, table2 and table3 can also have more than 100000 rows.
As per query for particular player and game table1 can have more than 100000 rows.
Is the above query is ok for large large tables or should I split the query in multiple queries.
NDB Engine used.
Please suggest me possible optimization.
Thanks,
Shiv
See comments above, but, at a guess, an index on (item_id,user_id,type,status) might help.