Related
Is there a way I can speed this up? Right now it's taking an unbelievably insane amount of time to query.
SELECT trades.*, trader1.user_name as trader1_name,
trader2.user_name as trader2_name FROM trades
LEFT JOIN logs_players trader1 ON trader1.user_id = trader1_account_id
LEFT JOIN logs_players trader2 ON trader2.user_id = trader2_account_id
ORDER BY time_added
LIMIT 20 OFFSET 0;
I've done as much as I could in terms of searching online for a solution. Or even just trying to get some more information why it's taking so long to execute.
The query takes about 45 seconds or so to complete.
Create statements:
CREATE TABLE `trades` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`trader1_account_id` int(11) DEFAULT NULL,
`trader2_account_id` int(11) DEFAULT NULL,
`trader1_value` bigint(20) DEFAULT NULL,
`trader2_value` bigint(20) DEFAULT NULL,
`trader1_ip` varchar(16) DEFAULT NULL,
`trader2_ip` varchar(16) DEFAULT NULL,
`world` int(11) DEFAULT NULL,
`x` int(11) DEFAULT NULL,
`z` int(11) DEFAULT NULL,
`level` int(11) DEFAULT NULL,
`trader1_user` varchar(12) DEFAULT NULL,
`trader2_user` varchar(12) DEFAULT NULL,
`time_added` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8
CREATE TABLE `logs_players` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) DEFAULT NULL,
`user_name` varchar(20) DEFAULT NULL,
`world_stage` varchar(20) DEFAULT NULL,
`world_type` varchar(20) DEFAULT NULL,
`bank` longtext,
`inventory` longtext,
`equipment` longtext,
`total_wealth` mediumtext,
`total_play_time` mediumtext,
`rights` int(11) DEFAULT NULL,
`icon` int(11) DEFAULT NULL,
`ironmode` int(11) DEFAULT NULL,
`x` int(11) DEFAULT NULL,
`z` int(11) DEFAULT NULL,
`level` int(11) DEFAULT NULL,
`last_ip` varchar(16) DEFAULT NULL,
`last_online` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`muted_until` timestamp NULL DEFAULT NULL,
`banned_until` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=0 DEFAULT CHARSET=utf8
I filled a sample database with 10k rows each, and found that a few indexes were what you needed:
ALTER TABLE `logs_players` ADD INDEX(`user_id`);
ALTER TABLE `trades` ADD INDEX(`time_added`);
The main index we need is an index on user_id. Changing that we went from a query time of 20.1390 seconds, to 0.0130 seconds:
We can even get that down further, by adding an index on time_added to make sorting a lot faster, now we ended up with an impressive query time:
Do some research on indexes! A simple EXPLAIN query would show you that you're using filesort (Which is rather bad!):
After indexes, this looks a lot better:
query is simple, as below:
select count(1) from ec_account a join ec_card b on a.id = b.AccountId
there are 2.5 million rows in either ec_account and ec_card.(InnoDB)
here is the execution plan:
execution plan
as you see,
it already added index and used it, but the query still costed almost 60 seconds, is there any way could optimize it except changing database(mariadb has no such choke point as far as i know).
here is table DDL,ec_ccount:
CREATE TABLE `ec_account` (
`Id` varchar(64) NOT NULL,
`AccountType` varchar(32) NOT NULL,
`Name` varchar(32) NOT NULL,
`Status` tinyint(3) unsigned NOT NULL,
`IDCardType` varchar(32) DEFAULT NULL,
`IDCardNo` varchar(64) DEFAULT NULL,
`Password` varchar(256) DEFAULT NULL,
`PasswordHalt` varchar(128) DEFAULT NULL,
`Sex` varchar(8) DEFAULT NULL,
`BirthDay` datetime NOT NULL,
`Mobile` varchar(16) DEFAULT NULL,
`Address` varchar(64) DEFAULT NULL,
`Linkman` varchar(32) DEFAULT NULL,
`LinkmanRelation` varchar(16) DEFAULT NULL,
`LinkmanTel` varchar(16) DEFAULT NULL,
`Remark` varchar(128) DEFAULT NULL,
`Nationality` varchar(32) DEFAULT NULL,
`Nation` varchar(32) DEFAULT NULL,
`MaritalStatus` varchar(8) DEFAULT NULL,
`NativePlace` varchar(64) DEFAULT NULL,
`Occupation` varchar(32) DEFAULT NULL,
`BloodType` varchar(8) DEFAULT NULL,
`Education` varchar(8) DEFAULT NULL,
`LinkmanAddress` varchar(64) DEFAULT NULL,
`HomeAddress` varchar(128) DEFAULT NULL,
`Email` varchar(64) DEFAULT NULL,
`CompanyName` varchar(64) DEFAULT NULL,
`CompanyAddress` varchar(128) DEFAULT NULL,
`CompanyTel` varchar(16) DEFAULT NULL,
`Creator` char(36) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`CreateTime` datetime NOT NULL,
`LastModifier` char(36) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`LastModifyTime` datetime DEFAULT NULL,
`Avatar` longblob,
PRIMARY KEY (`Id`),
KEY `IX_Name` (`Name`) USING HASH,
KEY `Idx_IDCard_Account` (`IDCardType`,`IDCardNo`) USING HASH,
KEY `Idx_Mobile` (`Mobile`) USING HASH,
KEY `Idx_CreateTime` (`CreateTime`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
and ec_card :
CREATE TABLE `ec_card` (
`Id` char(36) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '',
`AccountId` varchar(64) NOT NULL,
`CardType` varchar(32) NOT NULL,
`CardNo` varchar(32) NOT NULL,
`Status` tinyint(3) unsigned NOT NULL,
`IsPasswordAuth` tinyint(1) NOT NULL,
PRIMARY KEY (`Id`),
UNIQUE KEY `Idx_Unique_AccountId_CardType` (`AccountId`,`CardType`) USING HASH,
UNIQUE KEY `Idx_Unique_CardType_CardNo` (`CardType`,`CardNo`) USING HASH,
KEY `Idx_Uniques_AccountId` (`AccountId`) USING BTREE,
CONSTRAINT `FK_ec_card_ec_account_AccountId` FOREIGN KEY (`AccountId`) REFERENCES `ec_account` (`Id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Not without fundamentally changing the query.
There are no conditions on your query! It selects all 2.5 million rows from ec_card, as well as every matching row from ec_account. Reading all this data from disk and sending it over the network is the bottleneck; there is no way to change that without changing what the query does.
Here is a workaround for you. I think it would run much faster, and get the same result.
Calculate the total count of ec_account:
SELECT count(1) AS total_count FROM ec_account;
Calculate the amount of records those existed in ec_account but not existed in ec_card:
SELECT count(1) AS missing_count
FROM ec_account a LEFT JOIN ec_card b on a.id = b.AccountId
WHERE b.AccountId IS NULL;
Matched count = total_count - missing_count
The core problem here is that you combined two large table together, it requires a lot of memory and it apparently needs a lot of time to finish.
try it using correlated subquery. This might help:
select count(1) from ec_account a where exists (select * from ec_card b
where b.AccountId=a.id)
Also, other than indexing following strategies generally help:
- Denormalization
- Caching results
- Using a NoSQL database
I have three tables in MySQL:
CREATE TABLE `mlm`.`facturacion_2012_drm_base` ( `custid`
varchar(20) NOT NULL, `fecha` date NOT NULL, `docid` varchar(20)
NOT NULL, `billid` varchar(20) NOT NULL, `movimiento` varchar(20)
DEFAULT NULL, `movid` varchar(20) DEFAULT NULL, `medio_pago`
varchar(40) DEFAULT NULL, `digitos` varchar(20) DEFAULT NULL,
`monto_facturado` decimal(20,2) NOT NULL, `monto_pagado`
decimal(20,2) NOT NULL, `monto_usado` decimal(20,2) NOT NULL,
`documento` varchar(2) NOT NULL, `codigo_pago` varchar(5) DEFAULT
NULL, `desc_pago` varchar(100) DEFAULT NULL, `sociedad`
varchar(45) DEFAULT NULL, `sociedad_bonif` varchar(45) DEFAULT NULL,
KEY `billid` (`billid`), KEY `motors_no_fact`
(`custid`,`billid`,`fecha`,`documento`) USING BTREE, KEY
`facturacion` (`custid`,`fecha`,`documento`) USING BTREE )
ENGINE=MyISAM DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC;
CREATE TABLE `mlm`.`facturacion_2012_drm_cortes` ( `id` bigint(20)
NOT NULL AUTO_INCREMENT, `fecha_inicial` date NOT NULL,
`fecha_final` date NOT NULL, PRIMARY KEY (`id`) ) ENGINE=MyISAM
AUTO_INCREMENT=433 DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC;
CREATE TABLE `mlm`.`facturacion_2012_drm_emitidas` ( `id`
bigint(20) NOT NULL AUTO_INCREMENT, `custid` varchar(20) NOT NULL,
`fecha_emision` date NOT NULL, `id_fechas` bigint(20) NOT NULL,
`monto` decimal(20,2) NOT NULL, `iva` decimal(20,2) NOT NULL,
`total` decimal(20,2) NOT NULL, `medio_pago` varchar(2000) NOT NULL,
`digitos` varchar(100) NOT NULL, `operaciones` int(10) NOT NULL,
`activa` varchar(2) NOT NULL, `movimiento` varchar(45) NOT NULL,
`parcialidades` varchar(100) NOT NULL, `monto_bruto` decimal(20,2)
NOT NULL, `billid` varchar(45) NOT NULL, `serie` varchar(2)
DEFAULT NULL, `folio` int(10) DEFAULT NULL, `uuid` varchar(45)
DEFAULT NULL, PRIMARY KEY (`id`), KEY `motors`
(`billid`,`id_fechas`,`activa`) ) ENGINE=MyISAM AUTO_INCREMENT=511483
DEFAULT CHARSET=latin1;
I changed the MySQL server from 5.1 (32 Bits) to 5.6 (64 Bits) and restored all my tables but I´m having problems with this query:
SELECT a.custid,
a.monto_facturado,
a.billid,
a.fecha,
b.id,
b.fecha_inicial
FROM facturacion_2012_drm_base a,
facturacion_2012_drm_cortes b
WHERE a.custid = ANY (SELECT custid
FROM facturacion_motors_pendientes
WHERE situacion = 'no facturado')
AND a.billid <> ALL (SELECT billid
FROM facturacion_2012_drm_emitidas
WHERE activa = 'SI')
AND a.fecha BETWEEN b.fecha_inicial AND b.fecha_final
AND a.documento = 'FA'
AND Year(a.fecha) = Year(Curdate())
GROUP BY a.billid
Since the server change, the query never finish, showing the message "Query is being executed..."
Anybody knows why this is happening?
At the very least you have indexing problems:
On facturacion_2012_drm_emitidas you filter by activa but do not have an index that can be used for this
On facturacion_2012_drm_base you don't have an index for fecha or documento that can be used
On facturacion_2012_drm_cortes you don't have an index for fecha_inicial or fecha_final
I don't know if you have index problems on facturacion_motors_pendientes because you did not include DDL for it.
Make sure you have the proper indexes for your query and then see what happens. What is happening now is that you have a complex join with subqueries and basically none of it is utilizing indexes, requiring you to execute full table scans on all tables involved.
You also might consider converting these tables to InnoDB, as that is the preferred table type in 5.6. This is of course unless you need certain MyISAM functionality on these tables (like full text search).
I have 3 tables in which I'm trying to preform joins on, and inserting the resulting data into another table. The query is taking anywhere between 15-30 mins depending on the dataset. The tables I'm selecting from and joining on are at least 25k records each but will quickly grow to be 500k+.
I tried adding indexes on the fields but still isn't helping that much. Are there any other things I can try or are joins on this scale just going to take this long?
Here is the query I'm trying to perform:
INSERT INTO audience.topitem
(runs_id, total_training_count, item, standard_index_value, significance, seed_count, nonseed_count, prod, model_type, level_1, level_2, level_3, level_4, level_5)
SELECT 5, seed_count + nonseed_count AS total_training_count,
ii.item, standard_index_value, NULL, seed_count, nonseed_count,
standard_index_value * seed_count AS prod, 'site', topic_L1, topic_L2, topic_L3, topic_L4, topic_L5
FROM audience.item_indexes ii
LEFT JOIN audience.usercounts uc ON ii.item = uc.item AND ii.runs_id = uc.runs_id
LEFT JOIN categorization.categorization at on ii.item = at.url
WHERE ii.runs_id = 5
Table: audience.item_indexes
CREATE TABLE `item_indexes` (
`item` varchar(1024) DEFAULT NULL,
`standard_index_value` float DEFAULT NULL,
`runs_id` int(11) DEFAULT NULL,
`model_type` enum('site','term','combo') DEFAULT NULL,
KEY `item_idx` (`item`(333))
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
Table: audience.usercounts
CREATE TABLE `usercounts` (
`item` varchar(1024) DEFAULT NULL,
`seed_count` int(11) DEFAULT NULL,
`nonseed_count` int(11) DEFAULT NULL,
`significance` float(19,6) DEFAULT NULL,
`runs_id` int(11) DEFAULT NULL,
`model_type` enum('site','term','combo') DEFAULT NULL,
KEY `item_idx` (`item`(333))
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
Table: audience.topitem
CREATE TABLE `topitem` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`total_training_count` int(11) DEFAULT NULL,
`item` varchar(1024) DEFAULT NULL,
`standard_index_value` float(19,6) DEFAULT NULL,
`significance` float(19,6) DEFAULT NULL,
`seed_count` int(11) DEFAULT NULL,
`nonseed_count` int(11) DEFAULT NULL,
`prod` float(19,6) DEFAULT NULL,
`cat_type` varchar(32) DEFAULT NULL,
`cat_level` int(11) DEFAULT NULL,
`conf` decimal(19,9) DEFAULT NULL,
`level_1` varchar(64) DEFAULT NULL,
`level_2` varchar(64) DEFAULT NULL,
`level_3` varchar(64) DEFAULT NULL,
`level_4` varchar(64) DEFAULT NULL,
`level_5` varchar(64) DEFAULT NULL,
`runs_id` int(11) DEFAULT NULL,
`model_type` enum('site','term','combo') DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=825 DEFAULT CHARSET=utf8;
Table: categorization.categorization
CREATE TABLE `AT_categorization` (
`url` varchar(760) NOT NULL ,
`language` varchar(10) DEFAULT NULL,
`category` text,
`entity` text,
`source` varchar(255) DEFAULT NULL,
`topic_L1` varchar(45) NOT NULL DEFAULT '',
`topic_L2` varchar(45) NOT NULL DEFAULT '',
`topic_L3` varchar(45) NOT NULL DEFAULT '',
`topic_L4` varchar(45) NOT NULL DEFAULT '',
`topic_L5` varchar(45) NOT NULL DEFAULT '',
`last_refreshed` datetime DEFAULT NULL,
PRIMARY KEY (`url`,`topic_L1`,`topic_L2`,`topic_L3`,`topic_L4`,`topic_L5`),
UNIQUE KEY `inx_url` (`url`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
If you add the following indexes, your query will run faster:
CREATE INDEX runs_idx ON audience.item_indexes (runs_id);
ALTER TABLE audience.usercounts
DROP INDEX item_idx,
ADD INDEX item_idx (runs_id, item(333));
Also, item_indexes is utf8, but AT_categorization is latin1, which keeps any indexes from being used. To address this issue, change AT_categorization to utf8:
ALTER TABLE AT_categorization CHARSET=utf8;
Lastly, for the AT_categorization table, the two indexes
PRIMARY KEY (`url`,`topic_L1`,`topic_L2`,`topic_L3`,`topic_L4`,`topic_L5`),
UNIQUE KEY `inx_url` (`url`)
are redundant. So you could DROP these, and simply have the url field be the primary key:
ALTER TABLE AT_categorization
DROP PRIMARY KEY,
DROP KEY `inx_url`,
ADD PRIMARY KEY (url);
The following query is timing out after 600 seconds.
update placed p
,Results r
set p.position = r.position
where p.competitor = r.competitor
AND p.date = r.date
AND REPLACE(p.time,":","") = r.time;
The structure is as follows:
'CREATE TABLE `placed` (
`idplaced` varchar(50) DEFAULT NULL,
`date` decimal(8,0) DEFAULT NULL,
`time` varchar(45) DEFAULT NULL,
`field1` varchar(45) DEFAULT NULL,
`competitor` varchar(45) DEFAULT NULL,
`field2` int(2) DEFAULT NULL,
`field3` varchar(45) DEFAULT NULL,
`field4` varchar(45) DEFAULT NULL,
`field5` decimal(6,2) DEFAULT NULL,
`field6` decimal(10,2) DEFAULT NULL,
`field7` decimal(6,2) DEFAULT NULL,
`field8` char(1) DEFAULT NULL,
`field9` varchar(45) DEFAULT NULL,
`position` char(4) DEFAULT NULL,
`field10` decimal(6,2) DEFAULT NULL,
`field11` char(1) DEFAULT NULL,
`field12` char(1) DEFAULT NULL,
`field13` decimal(6,2) DEFAULT NULL,
`field14` decimal(6,2) DEFAULT NULL,
`field15` decimal(6,2) DEFAULT NULL,
`field16` decimal(6,2) DEFAULT NULL,
`field17` decimal(6,2) DEFAULT NULL,
`field18` char(1) DEFAULT NULL,
`field19` char(20) DEFAULT NULL,
`field20` char(1) DEFAULT NULL,
`field21` char(5) DEFAULT NULL,
`field22` char(5) DEFAULT NULL,
`field23` int(11) DEFAULT NULL
PRIMARY KEY (`idplaced`),
UNIQUE KEY `date_time_competitor_field18_combo` (`date`,`time`,`competitor`,`field18`)
) ENGINE=InnoDB AUTO_INCREMENT=100688607 DEFAULT CHARSET=latin1;
CREATE TABLE `results` (
`idresults` int(11) NOT NULL AUTO_INCREMENT,
`date` char(8) DEFAULT NULL,
`time` char(4) DEFAULT NULL,
`field1` varchar(45) DEFAULT NULL,
`competitor` varchar(45) DEFAULT NULL,
`position` char(4) DEFAULT NULL,
`field2` varchar(45) DEFAULT NULL,
`field3` decimal(2,0) DEFAULT NULL,
PRIMARY KEY (`idresults`)
) ENGINE=InnoDB AUTO_INCREMENT=6644 DEFAULT CHARSET=latin1;
The PLACED table has 65,000 records, the RESULTS table has 9,000 records.
I am assuming the solution involves a JOIN statement of some descript, and I have tried taking several suggestions from this site, but am simply not finding the answer I am looking for. Simply put, I would be grateful for suggestions on this. I can put up example tables / create table code if requried.
The index cannot be used efficiently to perform the join because of your REPLACE operation.
I'd suggest creating an index with the columns in the following slightly different order:
(date, competitor, time, position)
It may also help to add this index on both tables.
It would be even better if you could modify the data in the database so that the data in the time column was stored in the same format in both tables.
First of all, you'd better send us your full tables description, using
show create table
Second, you'd better use join syntax :
update placed p
join Results r on r.competitor = p.competitor
set p.position = r.position
where p.date = r.date
AND REPLACE(p.time,":","") = r.time;
Hope this will help.