AWS RDS performance is very very low - mysql

There are two databases, AWS RDS and Digitalocean $ 5 VPS.
I imported the same .sql file to both databases, the performance difference between them is almost 66000%
select count(special_cargo_id) from special_cargos;
AWS RDS elapsed time = 35.681seconds
DO 5$ VPS on Mysql elapsed time = 0.086seconds
Details:
Table Lenght
1.85GB Rows ~260k
MySQL Version 8
Table Engine InnoDB
RDS OUTPUT
CREATE TABLE `special_cargos` (
`special_cargo_id` int(11) NOT NULL AUTO_INCREMENT,
`barcode` varchar(20) DEFAULT '',
`salesCode` varchar(15) NOT NULL,
`price` varchar(15) NOT NULL,
`para` varchar(3) DEFAULT NULL,
`postaceki` varchar(25) DEFAULT '',
`iban` varchar(30) DEFAULT NULL,
`send_type` tinyint(1) unsigned NOT NULL,
`customer_id` int(11) NOT NULL,
`address_id` int(11) NOT NULL,
`height` varchar(5) DEFAULT '',
`weight` varchar(5) DEFAULT '',
`width` varchar(5) NOT NULL DEFAULT '',
`desi` varchar(5) DEFAULT '',
`length` varchar(255) DEFAULT '',
`customer_firstname` varchar(32) NOT NULL,
`customer_lastname` varchar(32) NOT NULL,
`customer_company` varchar(40) NOT NULL,
`customer_address` varchar(255) NOT NULL,
`customer_city` int(11) NOT NULL DEFAULT '0',
`customer_city_name` varchar(55) DEFAULT '',
`customer_zone_id` int(11) NOT NULL DEFAULT '0',
`customer_zone_name` varchar(55) DEFAULT '',
`customer_postcode` varchar(10) NOT NULL,
`firstname` varchar(32) NOT NULL,
`lastname` varchar(32) NOT NULL,
`company` varchar(40) DEFAULT '',
`address` varchar(255) NOT NULL,
`country_id` int(5) DEFAULT NULL,
`country_name` varchar(64) DEFAULT NULL,
`city_id` int(11) NOT NULL,
`city_name` varchar(55) NOT NULL,
`district_id` int(11) NOT NULL,
`district_name` varchar(64) NOT NULL,
`email` varchar(96) NOT NULL,
`phone` varchar(55) NOT NULL,
`postcode` varchar(10) DEFAULT '',
`product` text,
`sendText` text,
`returnText` text,
`created_at` datetime DEFAULT NULL,
`updated_at` datetime DEFAULT NULL,
`deleted_at` datetime DEFAULT NULL,
`cargo_firm` tinyint(1) NOT NULL DEFAULT '0',
`ups_barcode_link` text,
`status` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`special_cargo_id`),
KEY `customer_id` (`customer_id`),
KEY `special_cargo_id` (`special_cargo_id`),
KEY `barcode` (`barcode`),
KEY `salesCode` (`salesCode`),
KEY `price` (`price`),
KEY `postaceki` (`postaceki`),
KEY `customer_firstname` (`customer_firstname`),
KEY `customer_lastname` (`customer_lastname`),
KEY `firstname` (`firstname`),
KEY `lastname` (`lastname`),
KEY `customer_company` (`customer_company`),
KEY `customer_address` (`customer_address`),
KEY `customer_city_name` (`customer_city_name`),
KEY `customer_zone_name` (`customer_zone_name`),
KEY `company` (`company`),
KEY `address` (`address`),
KEY `city_name` (`city_name`),
KEY `district_name` (`district_name`),
KEY `email` (`email`),
KEY `phone` (`phone`),
KEY `postcode` (`postcode`),
KEY `index_special_cargos` (`special_cargo_id`)
) ENGINE=InnoDB AUTO_INCREMENT=497539 DEFAULT CHARSET=utf8
+-----------------------+----------+
|Variable_name |Value |
+-----------------------+----------+
|innodb_buffer_pool_size|1073741824|
+-----------------------+----------+
+--+-----------+--------------+----------+-----+-------------+-----------+-------+----+------+--------+-----------+
|id|select_type|table |partitions|type |possible_keys|key |key_len|ref |rows |filtered|Extra |
+--+-----------+--------------+----------+-----+-------------+-----------+-------+----+------+--------+-----------+
|1 |SIMPLE |special_cargos|NULL |index|NULL |customer_id|4 |NULL|146973|100 |Using index|
+--+-----------+--------------+----------+-----+-------------+-----------+-------+----+------+--------+-----------+
SIMPLE VPS ON MYSQL OUTPUT
CREATE TABLE `special_cargos` (
`special_cargo_id` int(11) NOT NULL AUTO_INCREMENT,
`barcode` varchar(20) NOT NULL,
`salesCode` varchar(15) NOT NULL,
`price` varchar(15) NOT NULL,
`para` varchar(3) DEFAULT NULL,
`postaceki` varchar(25) NOT NULL,
`iban` varchar(30) DEFAULT NULL,
`send_type` tinyint(1) unsigned NOT NULL,
`customer_id` int(11) NOT NULL,
`address_id` int(11) NOT NULL,
`height` varchar(5) NOT NULL,
`weight` varchar(5) NOT NULL,
`width` varchar(5) NOT NULL,
`desi` varchar(5) NOT NULL,
`length` varchar(255) NOT NULL,
`customer_firstname` varchar(32) NOT NULL,
`customer_lastname` varchar(32) NOT NULL,
`customer_company` varchar(40) NOT NULL,
`customer_address` varchar(255) NOT NULL,
`customer_city` int(11) NOT NULL,
`customer_city_name` varchar(55) NOT NULL,
`customer_zone_id` int(11) NOT NULL,
`customer_zone_name` varchar(55) NOT NULL,
`customer_postcode` varchar(10) NOT NULL,
`firstname` varchar(32) NOT NULL,
`lastname` varchar(32) NOT NULL,
`company` varchar(40) NOT NULL,
`address` varchar(255) NOT NULL,
`country_id` int(5) DEFAULT NULL,
`country_name` varchar(64) DEFAULT NULL,
`city_id` int(11) NOT NULL,
`city_name` varchar(55) NOT NULL,
`district_id` int(11) NOT NULL,
`district_name` varchar(64) NOT NULL,
`email` varchar(96) NOT NULL,
`phone` varchar(55) NOT NULL,
`postcode` varchar(10) NOT NULL,
`product` text,
`sendText` text NOT NULL,
`returnText` text NOT NULL,
`created_at` datetime NOT NULL,
`updated_at` datetime NOT NULL,
`deleted_at` datetime DEFAULT NULL,
`cargo_firm` int(1) NOT NULL DEFAULT '0',
`ups_barcode_link` text,
`status` int(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`special_cargo_id`),
KEY `customer_id` (`customer_id`),
KEY `special_cargo_id` (`special_cargo_id`),
KEY `barcode` (`barcode`),
KEY `salesCode` (`salesCode`),
KEY `price` (`price`),
KEY `postaceki` (`postaceki`),
KEY `customer_firstname` (`customer_firstname`),
KEY `customer_lastname` (`customer_lastname`),
KEY `firstname` (`firstname`),
KEY `lastname` (`lastname`),
KEY `customer_company` (`customer_company`),
KEY `customer_address` (`customer_address`),
KEY `customer_city_name` (`customer_city_name`),
KEY `customer_zone_name` (`customer_zone_name`),
KEY `company` (`company`),
KEY `address` (`address`),
KEY `city_name` (`city_name`),
KEY `district_name` (`district_name`),
KEY `email` (`email`),
KEY `phone` (`phone`),
KEY `postcode` (`postcode`)
) ENGINE=InnoDB AUTO_INCREMENT=497037 DEFAULT CHARSET=utf8
+-----------------------+---------+
|Variable_name |Value |
+-----------------------+---------+
|innodb_buffer_pool_size|134217728|
+-----------------------+---------+
+--+-----------+--------------+----------+-----+-------------+-----------+-------+----+------+--------+-----------+
|id|select_type|table |partitions|type |possible_keys|key |key_len|ref |rows |filtered|Extra |
+--+-----------+--------------+----------+-----+-------------+-----------+-------+----+------+--------+-----------+
|1 |SIMPLE |special_cargos|NULL |index|NULL |customer_id|4 |NULL|133967|100 |Using index|
+--+-----------+--------------+----------+-----+-------------+-----------+-------+----+------+--------+-----------+

There are two aspects of the problem here.
Aspect one and more important IMO is the query - it's bad. It does full index scan (yes, PRIMARY is also an index). Its performance scales linearly vs better and desired the B+tree's nlog(n).
Aspect two. Why the difference in the execution time. It can be related to caching (in DO's case the query was served from the buffer pool while in RDS from disk(=EBS)). It can be related to MySQL optimizer (DO and RDS could pick different indexes to serve the query PRIMARY vs smaller secondary index). But all that doesn't matter much due to the problems with the query itself.

Can you provide the output of:
SHOW CREATE TABLE special_cargos;
SHOW GLOBAL VARIABLES LIKE 'innodb_buffer_pool_size';
EXPLAIN select count(special_cargo_id) from special_cargos;
on both RDS and your non-RDS instance?
Assuming for a moment that special_cargo_id is the primary key, and it fits into RAM, and innodb_buffer_pool_size is configured reasonably (on a server with 2GB of RAM, 1GB is a reasonable amount, though RDS should come pre-configured to a sane value), it should be running similarly on both nodes.
Two possible explanations come to mind:
1) On RDS you don't have special_cargo_id defined as the PK.
2) RDS is for some reason choosing to not use the index (comparison of EXPLAIN output will confirm whether this is happening).
Edit:
Actually - are you sure your $5 VPS is using InnoDB? If you are getting an answer back in 0.08s, that sounds like it isn't actually scanning even the index. There is a good chance that on the VPS you are using MyISAM, which updates the number of rows in the table header after every write, specifically to make SELECT COUNT(*) FROM table_name; queries return instantaneously - like you have observed.
Edit 2:
Wow - so VPS is configured with only 128MB of innodb_buffer_pool_size and it is STILL going faster than RDS configured with 1GB of innodb_buffer_pool_size, and the execution plan on both is the same?
OK, try this:
select count(1) from special_cargos FORCE INDEX (PRIMARY);

1GB for the buffer_pool is dangerously high when you have only 2GB of RAM. Probably that system was swapping furiously.
Lower it to 200M and see how fast it runs.
See if they have some way of monitoring swap usage.

Related

Transferring table from MySQL 5.7 to 8.0, showing wrong data on a blob column

I am transferring a single table from MySQL 5.7 to MySQL 8.
First, I used mysqldump db_name table_name > table_name.sql in my current server(MYSQL 5.7); taken from here
Then, in my VM with MySQL 8, I used mysql -u username -p db_name < /path/to/table_name.sql.
The columns with the 'blob' data type are not correct, the other columns are okay.
the 'blob' columns went from "International Master (1973); Grandmaster (1975); FIDE Senior Trainer (2004).Alexander Genrikhovich Beliavsky was..." to "0x3C703E496E7465726E6174696F6E616C204D6173746572202831393733293B204772616E646D617374657220...".
Both databases have character-set = latin1 and collation = latin1_swedish_ci.
How can I properly transfer the 'blob' columns from MySQL 5.7 to MySQL 8?
Output when running SHOW CREATE TABLE table_name:
OLD:
CREATE TABLE `Player` (
`pid` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(34) DEFAULT NULL,
`longname` varchar(64) DEFAULT NULL,
`rating` int(10) unsigned NOT NULL DEFAULT '0',
`born` varchar(10) DEFAULT NULL,
`died` varchar(10) DEFAULT NULL,
`country` varchar(10) DEFAULT NULL,
`nationality` varchar(10) DEFAULT NULL,
`gender` enum('M','F') DEFAULT NULL,
`bio` blob,
`fiderating` int(10) unsigned NOT NULL DEFAULT '0',
`rawbio` blob,
`fidenumber` int(10) unsigned DEFAULT NULL,
`fideblitz` int(10) unsigned DEFAULT '0',
`fiderapid` int(10) unsigned DEFAULT '0',
PRIMARY KEY (`pid`),
KEY `name` (`name`),
KEY `rating` (`rating`),
KEY `country` (`country`),
KEY `nationality` (`nationality`),
KEY `gender` (`gender`),
KEY `fidenumber` (`fidenumber`),
FULLTEXT KEY `longname` (`longname`)
) ENGINE=MyISAM AUTO_INCREMENT=168045 DEFAULT CHARSET=latin1
NEW:
| Player | CREATE TABLE `Player` (
`pid` int unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(34) DEFAULT NULL,
`longname` varchar(64) DEFAULT NULL,
`rating` int unsigned NOT NULL DEFAULT '0',
`born` varchar(10) DEFAULT NULL,
`died` varchar(10) DEFAULT NULL,
`country` varchar(10) DEFAULT NULL,
`nationality` varchar(10) DEFAULT NULL,
`gender` enum('M','F') DEFAULT NULL,
`bio` blob,
`fiderating` int unsigned NOT NULL DEFAULT '0',
`rawbio` blob,
`fidenumber` int unsigned DEFAULT NULL,
`fideblitz` int unsigned DEFAULT '0',
`fiderapid` int unsigned DEFAULT '0',
PRIMARY KEY (`pid`),
KEY `name` (`name`),
KEY `rating` (`rating`),
KEY `country` (`country`),
KEY `nationality` (`nationality`),
KEY `gender` (`gender`),
KEY `fidenumber` (`fidenumber`),
FULLTEXT KEY `longname` (`longname`)
) ENGINE=MyISAM AUTO_INCREMENT=168045 DEFAULT CHARSET=latin1 |

How to optimize mysql query even it already used index

query is simple, as below:
select count(1) from ec_account a join ec_card b on a.id = b.AccountId
there are 2.5 million rows in either ec_account and ec_card.(InnoDB)
here is the execution plan:
execution plan
as you see,
it already added index and used it, but the query still costed almost 60 seconds, is there any way could optimize it except changing database(mariadb has no such choke point as far as i know).
here is table DDL,ec_ccount:
CREATE TABLE `ec_account` (
`Id` varchar(64) NOT NULL,
`AccountType` varchar(32) NOT NULL,
`Name` varchar(32) NOT NULL,
`Status` tinyint(3) unsigned NOT NULL,
`IDCardType` varchar(32) DEFAULT NULL,
`IDCardNo` varchar(64) DEFAULT NULL,
`Password` varchar(256) DEFAULT NULL,
`PasswordHalt` varchar(128) DEFAULT NULL,
`Sex` varchar(8) DEFAULT NULL,
`BirthDay` datetime NOT NULL,
`Mobile` varchar(16) DEFAULT NULL,
`Address` varchar(64) DEFAULT NULL,
`Linkman` varchar(32) DEFAULT NULL,
`LinkmanRelation` varchar(16) DEFAULT NULL,
`LinkmanTel` varchar(16) DEFAULT NULL,
`Remark` varchar(128) DEFAULT NULL,
`Nationality` varchar(32) DEFAULT NULL,
`Nation` varchar(32) DEFAULT NULL,
`MaritalStatus` varchar(8) DEFAULT NULL,
`NativePlace` varchar(64) DEFAULT NULL,
`Occupation` varchar(32) DEFAULT NULL,
`BloodType` varchar(8) DEFAULT NULL,
`Education` varchar(8) DEFAULT NULL,
`LinkmanAddress` varchar(64) DEFAULT NULL,
`HomeAddress` varchar(128) DEFAULT NULL,
`Email` varchar(64) DEFAULT NULL,
`CompanyName` varchar(64) DEFAULT NULL,
`CompanyAddress` varchar(128) DEFAULT NULL,
`CompanyTel` varchar(16) DEFAULT NULL,
`Creator` char(36) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`CreateTime` datetime NOT NULL,
`LastModifier` char(36) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,
`LastModifyTime` datetime DEFAULT NULL,
`Avatar` longblob,
PRIMARY KEY (`Id`),
KEY `IX_Name` (`Name`) USING HASH,
KEY `Idx_IDCard_Account` (`IDCardType`,`IDCardNo`) USING HASH,
KEY `Idx_Mobile` (`Mobile`) USING HASH,
KEY `Idx_CreateTime` (`CreateTime`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
and ec_card :
CREATE TABLE `ec_card` (
`Id` char(36) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '',
`AccountId` varchar(64) NOT NULL,
`CardType` varchar(32) NOT NULL,
`CardNo` varchar(32) NOT NULL,
`Status` tinyint(3) unsigned NOT NULL,
`IsPasswordAuth` tinyint(1) NOT NULL,
PRIMARY KEY (`Id`),
UNIQUE KEY `Idx_Unique_AccountId_CardType` (`AccountId`,`CardType`) USING HASH,
UNIQUE KEY `Idx_Unique_CardType_CardNo` (`CardType`,`CardNo`) USING HASH,
KEY `Idx_Uniques_AccountId` (`AccountId`) USING BTREE,
CONSTRAINT `FK_ec_card_ec_account_AccountId` FOREIGN KEY (`AccountId`) REFERENCES `ec_account` (`Id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Not without fundamentally changing the query.
There are no conditions on your query! It selects all 2.5 million rows from ec_card, as well as every matching row from ec_account. Reading all this data from disk and sending it over the network is the bottleneck; there is no way to change that without changing what the query does.
Here is a workaround for you. I think it would run much faster, and get the same result.
Calculate the total count of ec_account:
SELECT count(1) AS total_count FROM ec_account;
Calculate the amount of records those existed in ec_account but not existed in ec_card:
SELECT count(1) AS missing_count
FROM ec_account a LEFT JOIN ec_card b on a.id = b.AccountId
WHERE b.AccountId IS NULL;
Matched count = total_count - missing_count
The core problem here is that you combined two large table together, it requires a lot of memory and it apparently needs a lot of time to finish.
try it using correlated subquery. This might help:
select count(1) from ec_account a where exists (select * from ec_card b
where b.AccountId=a.id)
Also, other than indexing following strategies generally help:
- Denormalization
- Caching results
- Using a NoSQL database

Two MySql tables have correct indexes yet JOIN takes 9 seconds on small tables

mysql Ver 14.14 Distrib 5.1.73, for redhat-linux-gnu (x86_64) using readline 5.1
I am taking over a project. It is very old and the original programmer is long gone. No one has any idea why certain decisions were made.
The following query runs (on my Mac) in 9.5 seconds but if I remove the last JOIN then it drops to 2.5 seconds. What is wrong with that last JOIN?
select `ttl`.`id` AS `id`,
`ttl`.`name` AS `name`,
`ttl`.`updated_at` AS `last_update_on`,
`ttl`.`user_id` AS `list_creator`,
`ttl`.`retailer_nomination_list` AS `nomination_list`,
`ttl`.`created_at` AS `created_on`,
count(distinct `tlb`.`title_id`) AS `title_count`
from `haha_title_lists` `ttl`
left join `haha_title_list_to_users` `tltu` on((`ttl`.`id` = `tltu`.`title_list_id`))
left join `users` `u` on((`tltu`.`user_id` = `u`.`id`))
left join `users` `u2` on((`tltu`.`user_id` = `u2`.`id`))
left join `haha_title_list_to_venues` `tlv` on((`ttl`.`id` = `tlv`.`title_list`))
left join `haha_venue_properties` `tvp` on((`tlv`.`venue_id` = `tvp`.`id`))
join `haha_title_list_to_books` `tlb` on((`ttl`.`id` = `tlb`.`title_list_id`))
join `wawa_title` `ot` on((`tlb`.`title_id` = `ot`.`title_id`))
group by `ttl`.`id`;
The tables:
CREATE TABLE `haha_title_list_to_books` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`title_id` int(11) NOT NULL,
`title_list_id` int(11) NOT NULL,
`sdk` varchar(15) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
`created_at` datetime NOT NULL,
`promo_start_date` date DEFAULT NULL,
`promo_end_date` date DEFAULT NULL,
`promo_price` float DEFAULT NULL,
`confirmations` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`nominations` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`title_note` text COLLATE utf8_unicode_ci,
`executed` int(11) DEFAULT NULL,
`event_created` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `idx_promo_start_date` (`promo_start_date`),
KEY `idx_promo_end_date` (`promo_end_date`),
KEY `idx_title_list_to_books_title_id` (`title_id`),
KEY `idx_title_list_to_books_title_list_id` (`title_list_id`)
) ENGINE=MyISAM AUTO_INCREMENT=21847 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
and:
CREATE TABLE `wawa_title` (
`title_id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(200) DEFAULT NULL,
`title_alpha` varchar(25) NOT NULL,
`display_title` varchar(200) NOT NULL,
`subtitle` text NOT NULL,
`sdk10` varchar(13) DEFAULT '',
`sdk13` varchar(15) DEFAULT NULL,
`primary_sdk13` varchar(15) DEFAULT NULL,
`asin` varchar(10) DEFAULT NULL,
`pub_season` varchar(15) NOT NULL,
`pub_year` varchar(15) NOT NULL,
`bisac1` varchar(15) NOT NULL,
`bisac2` varchar(15) NOT NULL,
`bisac3` varchar(15) NOT NULL,
`barcode` varchar(30) DEFAULT NULL,
`dewey_decimal` varchar(15) NOT NULL,
`lib_of_congress` varchar(15) NOT NULL,
`spanish_language` tinyint(4) NOT NULL,
`target_audience` tinyint(3) unsigned DEFAULT NULL,
`language` varchar(20) DEFAULT NULL,
`edition` varchar(45) DEFAULT NULL,
`pages` int(11) DEFAULT NULL,
`number_in_series` int(11) DEFAULT NULL,
`trimsize` varchar(10) DEFAULT NULL,
`filesize` varchar(10) DEFAULT NULL,
`duration_hours` int(11) DEFAULT NULL,
`duration_minutes` int(11) DEFAULT NULL,
`discs` int(11) DEFAULT NULL,
`download` date DEFAULT NULL,
`size_unit` varchar(15) NOT NULL DEFAULT '',
`digitization_date` date NOT NULL,
`us_on_sale_date` date NOT NULL,
`aus_on_sale_date` date NOT NULL,
`can_on_sale_date` date NOT NULL,
`uk_on_sale_date` date NOT NULL,
`us_list_price` varchar(10) NOT NULL,
`aus_list_price` varchar(10) NOT NULL,
`can_list_price` varchar(10) NOT NULL,
`uk_list_price` varchar(10) NOT NULL,
`isPrimary` varchar(1) DEFAULT NULL,
`modified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`modifier` int(11) NOT NULL,
`activated` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`active` varchar(3) NOT NULL DEFAULT 'N',
`flagged_string` text,
`created` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`assets_id` varchar(20) DEFAULT NULL,
`book_details` text,
`book_keynote` text,
`exclude_goodreads` char(1) NOT NULL DEFAULT 'N',
`series_description` text,
`review_quote1` text,
`territory_id` int(11) DEFAULT '27',
`featured_newsletter_id` tinyint(3) unsigned DEFAULT '0',
`retailer_discovery_check` datetime DEFAULT NULL,
`suppress_retailer_approval` tinyint(1) DEFAULT '0',
`suppress_retailer_approval_reason` varchar(255) DEFAULT NULL,
`ebb_description` text CHARACTER SET utf8 COLLATE utf8_unicode_ci,
`slug` varchar(150) DEFAULT NULL,
`legacy_slug` varchar(150) DEFAULT NULL,
`us_agency_price` varchar(10) DEFAULT NULL,
`firebrand_title_id` int(11) DEFAULT NULL,
`ebb_label` varchar(200) DEFAULT NULL,
`ebb_end_sale_date` date DEFAULT NULL,
`ebb_downprice` decimal(10,2) DEFAULT NULL,
`book_club` varchar(1) DEFAULT NULL,
`best_seller` varchar(1) DEFAULT NULL,
`award_winner` varchar(1) DEFAULT NULL,
`discovery` char(1) NOT NULL DEFAULT 'Y',
`narrator_id` int(11) DEFAULT NULL,
`suppress_series_data` varchar(255) DEFAULT NULL,
PRIMARY KEY (`title_id`),
KEY `active_index` (`active`),
KEY `fk_title_series_id_idx` (`series_id`),
KEY `series_id` (`series_id`),
KEY `idx_title_sdk13` (`sdk13`),
KEY `idx_title_active_isprimary` (`active`,`isPrimary`),
KEY `bisac1` (`bisac1`),
KEY `bisac2` (`bisac2`),
KEY `bisac3` (`bisac3`),
KEY `idx_primary_sdk13` (`primary_sdk13`),
KEY `idx_territory_id` (`territory_id`),
CONSTRAINT `fk_title_series_id` FOREIGN KEY (`series_id`) REFERENCES `wawa_series` (`series_id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=19700 DEFAULT CHARSET=utf8 |
If I remove this line:
join `wawa_title` `ot` on((`tlb`.`title_id` = `ot`.`title_id`))
The query speed drops from 9.5 seconds to 2.5 seconds. Not great, but a huge improvement.
And yet, both tables have indexes on table_id, so why would that line be a problem?
I notice that one table is InnoDB and the other is MyISAM. Would that have an effect?
Do not JOIN to tables that you don't use.
A JOIN often "explodes" the number of rows, then a GROUP BY like you have reels in the number of rows. To see this, leave all the JOINs there, but remove the GROUP BY. See how many rows you get.
To avoid part of that explosion, change
count(distinct `tlb`.`title_id`) AS `title_count`
to
( SELECT count(distinct `title_id`)
FROM `haha_title_list_to_books`
WHERE `ttl`.`id` = `title_list_id`
) AS `title_count`
and remove the current JOIN to tlb.
Mixing MyISAM and InnoDB should not have any direct impact on this SELECT. However, you should consider moving all of your tables to InnoDB.

Long time to exec a simple query in mysql

I am with a performance problem in my MySQL database, and do not know if it's the server configuration issue or disk space, or simply not riding the query correctly.
a simple query that summarizes the total number of rows in a table by a distinct field, it takes more than 3 minutes. The status is "Sending data" for a long time.
the query is as follows:
SELECT COUNT (DISTINCT (ca.nm_slug)) the qtd_total
FROM consulta_atual ca
WHERE ca.cd_categoria_site = 436
below the description of the table (has a 4 million records)
CREATE TABLE IF NOT EXISTS `consulta_atual` (
`cd_categoria` int(11) NOT NULL DEFAULT '0',
`cd_categoria_site` int(11) DEFAULT NULL,
`ds_categoria` varchar(100) NOT NULL,
`ds_subcategoria` varchar(200) NOT NULL,
`cd_produto_price` bigint(11) NOT NULL,
`cd_seq` bigint(20) NOT NULL DEFAULT '0',
`tp_fornecedor` int(11) NOT NULL DEFAULT '0',
`name` varchar(300) NOT NULL,
`nm_slug` varchar(200) DEFAULT NULL,
`fornecedor` int(11) NOT NULL,
`url_img_fornecedor` varchar(200) NOT NULL,
`url_raiz_fornecedor` varchar(200) NOT NULL,
`url_imagem` varchar(500) NOT NULL,
`url_produto` varchar(500) NOT NULL,
`vlr_produto` varchar(49) NOT NULL DEFAULT '',
`pnt_produto` varchar(53) NOT NULL DEFAULT '',
`vlr_produto_original` float(10,2) NOT NULL,
`menor_valor` float DEFAULT NULL,
`maior_valor` float DEFAULT NULL,
`qtd_lojas` int(11) DEFAULT NULL,
PRIMARY KEY (`cd_produto_price`,`fornecedor`),
KEY `nm_slug_2` (`nm_slug`),
KEY `ds_categoria` (`ds_categoria`),
KEY `vlr_produto_original` (`vlr_produto_original`),
KEY `cd_categoria_site` (`cd_categoria_site`),
KEY `fornecedor` (`fornecedor`),
KEY `tp_fornecedor` (`tp_fornecedor`),
FULLTEXT KEY `name` (`name`),
FULLTEXT KEY `ds_categoria_2` (`ds_categoria`),
FULLTEXT KEY `ds_categoria_3` (`ds_categoria`,`name`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
How can I improve execution performance of this simple query?

Problems with a MySQL server change

I have three tables in MySQL:
CREATE TABLE `mlm`.`facturacion_2012_drm_base` ( `custid`
varchar(20) NOT NULL, `fecha` date NOT NULL, `docid` varchar(20)
NOT NULL, `billid` varchar(20) NOT NULL, `movimiento` varchar(20)
DEFAULT NULL, `movid` varchar(20) DEFAULT NULL, `medio_pago`
varchar(40) DEFAULT NULL, `digitos` varchar(20) DEFAULT NULL,
`monto_facturado` decimal(20,2) NOT NULL, `monto_pagado`
decimal(20,2) NOT NULL, `monto_usado` decimal(20,2) NOT NULL,
`documento` varchar(2) NOT NULL, `codigo_pago` varchar(5) DEFAULT
NULL, `desc_pago` varchar(100) DEFAULT NULL, `sociedad`
varchar(45) DEFAULT NULL, `sociedad_bonif` varchar(45) DEFAULT NULL,
KEY `billid` (`billid`), KEY `motors_no_fact`
(`custid`,`billid`,`fecha`,`documento`) USING BTREE, KEY
`facturacion` (`custid`,`fecha`,`documento`) USING BTREE )
ENGINE=MyISAM DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC;
CREATE TABLE `mlm`.`facturacion_2012_drm_cortes` ( `id` bigint(20)
NOT NULL AUTO_INCREMENT, `fecha_inicial` date NOT NULL,
`fecha_final` date NOT NULL, PRIMARY KEY (`id`) ) ENGINE=MyISAM
AUTO_INCREMENT=433 DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC;
CREATE TABLE `mlm`.`facturacion_2012_drm_emitidas` ( `id`
bigint(20) NOT NULL AUTO_INCREMENT, `custid` varchar(20) NOT NULL,
`fecha_emision` date NOT NULL, `id_fechas` bigint(20) NOT NULL,
`monto` decimal(20,2) NOT NULL, `iva` decimal(20,2) NOT NULL,
`total` decimal(20,2) NOT NULL, `medio_pago` varchar(2000) NOT NULL,
`digitos` varchar(100) NOT NULL, `operaciones` int(10) NOT NULL,
`activa` varchar(2) NOT NULL, `movimiento` varchar(45) NOT NULL,
`parcialidades` varchar(100) NOT NULL, `monto_bruto` decimal(20,2)
NOT NULL, `billid` varchar(45) NOT NULL, `serie` varchar(2)
DEFAULT NULL, `folio` int(10) DEFAULT NULL, `uuid` varchar(45)
DEFAULT NULL, PRIMARY KEY (`id`), KEY `motors`
(`billid`,`id_fechas`,`activa`) ) ENGINE=MyISAM AUTO_INCREMENT=511483
DEFAULT CHARSET=latin1;
I changed the MySQL server from 5.1 (32 Bits) to 5.6 (64 Bits) and restored all my tables but I´m having problems with this query:
SELECT a.custid,
a.monto_facturado,
a.billid,
a.fecha,
b.id,
b.fecha_inicial
FROM facturacion_2012_drm_base a,
facturacion_2012_drm_cortes b
WHERE a.custid = ANY (SELECT custid
FROM facturacion_motors_pendientes
WHERE situacion = 'no facturado')
AND a.billid <> ALL (SELECT billid
FROM facturacion_2012_drm_emitidas
WHERE activa = 'SI')
AND a.fecha BETWEEN b.fecha_inicial AND b.fecha_final
AND a.documento = 'FA'
AND Year(a.fecha) = Year(Curdate())
GROUP BY a.billid
Since the server change, the query never finish, showing the message "Query is being executed..."
Anybody knows why this is happening?
At the very least you have indexing problems:
On facturacion_2012_drm_emitidas you filter by activa but do not have an index that can be used for this
On facturacion_2012_drm_base you don't have an index for fecha or documento that can be used
On facturacion_2012_drm_cortes you don't have an index for fecha_inicial or fecha_final
I don't know if you have index problems on facturacion_motors_pendientes because you did not include DDL for it.
Make sure you have the proper indexes for your query and then see what happens. What is happening now is that you have a complex join with subqueries and basically none of it is utilizing indexes, requiring you to execute full table scans on all tables involved.
You also might consider converting these tables to InnoDB, as that is the preferred table type in 5.6. This is of course unless you need certain MyISAM functionality on these tables (like full text search).