Very slow join of the small table to the big one - mysql

I needed to add one more column to already existing one (~6M rows).
Unfortunately, I cannot modify the table.
So, I created another one, which contains only two columns: the first (DI_key) corresponds to the primary key of the large table (ID_key) and the second (int(11)) corresponds to the property I want to keep in the database.
For the moment the second table contains only ~250 rows, but when I LEFT JOIN it ON (DI_key = ID_key) my requests to the bigger table are ~ 1000 times slower!
Adding indexes on the small table doesn't solve the problem.
Any suggestions would be welcome...
Unfortunately, I'm working on the very old infrastructure - MySQL 5.1
Technical infos:
--
-- Structure de la table `line`
--
CREATE TABLE IF NOT EXISTS `line` (
`ID_line` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`LISOCI` char(2) DEFAULT '02',
`LICLIE` int(11) DEFAULT '0',
`LIADLIV` tinyint(4) NOT NULL DEFAULT '0',
`LICMD` int(11) NOT NULL DEFAULT '0',
`LICMDM` varchar(32) NOT NULL,
`LILIG` int(11) NOT NULL DEFAULT '0',
`LIARTI` varchar(8) NOT NULL,
`LIDES1` varchar(40) NOT NULL,
`LIDES2` varchar(40) NOT NULL,
`LIQTE1` float DEFAULT '0',
`LIQTE2` float DEFAULT '0',
`LIQTE3` float DEFAULT '0',
`LIQTE4` float DEFAULT '0',
`LIQTE5` float DEFAULT '0',
`LIQTE6` float DEFAULT '0',
`LIQTE7` float DEFAULT '0',
`LIQTE8` float DEFAULT '0',
`LIQTE9` float DEFAULT '0',
`LIQTE10` float DEFAULT '0',
`LIQTE11` float DEFAULT '0',
`LIQTE12` float DEFAULT '0',
`LICOMMENT` varchar(80) NOT NULL,
`LICMT1` varchar(80) NOT NULL,
`LICMT2` varchar(80) NOT NULL,
`LICMT3` varchar(80) NOT NULL,
`LICMT4` varchar(80) NOT NULL,
`LICMT5` varchar(80) NOT NULL,
`LICMT6` varchar(80) NOT NULL,
`LICMT7` varchar(80) NOT NULL,
`LICMT8` varchar(80) NOT NULL,
`LICMT9` varchar(80) NOT NULL,
`LICMT10` varchar(80) NOT NULL,
`LICMT11` varchar(80) NOT NULL,
`LICMT12` varchar(80) NOT NULL,
`LIUNIV` char(3) NOT NULL,
`LILIBQTE` varchar(10) DEFAULT NULL,
`LIQNIF` float NOT NULL DEFAULT '0',
`LIDATE` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`LIPRIX` float NOT NULL DEFAULT '0',
`LIPVCONSO` double NOT NULL DEFAULT '0',
`LITOTAL` float NOT NULL DEFAULT '0',
`LIPRV` double NOT NULL DEFAULT '0',
`LILOGON` varchar(12) NOT NULL,
`LIADRIP` varchar(16) NOT NULL,
`LISESSION` varchar(32) NOT NULL,
`LIGRATUIT` varchar(1) NOT NULL DEFAULT 'N',
PRIMARY KEY (`ID_line`),
KEY `I1` (`LISOCI`,`LICLIE`,`LICMD`,`LILIG`),
KEY `I2` (`LISOCI`,`LICLIE`,`LICMD`,`LISESSION`,`LILIG`),
KEY `L12` (`LICMDM`,`LILIG`),
KEY `LICMD` (`LICMD`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=7185054 ;
--
-- Structure de la table `line_info`
--
CREATE TABLE IF NOT EXISTS `line_info` (
`DI_line` bigint(20) unsigned NOT NULL,
`LICLIES` int(11) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`DI_line`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
SHOW TABLE STATUS WHERE name LIKE "line":
+------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+----------------+---------+
| line | MyISAM | 10 | Dynamic | 6339101 | 179 | 1135581592 | 281474976710655 | 429571072 | 3356 | 7185054 | 2023-01-27 09:31:29 | 2023-02-06 14:49:08 | 2023-01-27 09:34:49 | latin1_swedish_ci | NULL | | |
+------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+----------------+---------+
1 row in set (0.00 sec)
SHOW TABLE STATUS WHERE name LIKE "line_info":
+-----------+--------+---------+------------+------+----------------+-------------+------------------+--------------+-----------+----------------+---------------------+---------------------+------------+-------------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+-----------+--------+---------+------------+------+----------------+-------------+------------------+--------------+-----------+----------------+---------------------+---------------------+------------+-------------------+----------+----------------+---------+
| line_info | MyISAM | 10 | Fixed | 239 | 13 | 3107 | 3659174697238527 | 6144 | 0 | NULL | 2023-02-07 09:50:45 | 2023-02-07 09:50:45 | NULL | latin1_swedish_ci | NULL | | |
+-----------+--------+---------+------------+------+----------------+-------------+------------------+--------------+-----------+----------------+---------------------+---------------------+------------+-------------------+----------+----------------+---------+
1 row in set (0.01 sec)
EXPLAIN SELECT:
EXPLAIN SELECT SQL_NO_CACHE `line`.*, `line_info`.`LICLIES`
FROM `line`
LEFT JOIN `line_info` ON DI_line = ID_line
WHERE (LISOCI LIKE '02')
AND (LICLIE IN ('44', '55') OR LICLIES IN ('44', '55'))
AND LICMD = 0
AND LICMDM = ''
AND LILOGON = 'WWW'
AND LIDATE >= '2022-12-02 16:53:56'
Result:
+----+-------------+-----------+--------+-----------------+---------+---------+--------------+-------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-----------+--------+-----------------+---------+---------+--------------+-------+-------------+
| 1 | SIMPLE | line | ref | I1,I2,L12,LICMD | L12 | 34 | const | 56594 | Using where |
| 1 | SIMPLE | line_info | eq_ref | PRIMARY | PRIMARY | 8 | line.ID_line | 1 | Using where |
+----+-------------+-----------+--------+-----------------+---------+---------+--------------+-------+-------------+
2 rows in set (0.00 sec)

Related

Mysql wrong index after InnoDB compression

I have 2 mysql servers (master and slave). I've enabled InnoDB compression on Slave, after that mysql sometimes chooses wrong index on query.
Explain on Master:
+----+-------------+-------+--------+---------------+---------+---------+-----------------------------+-----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+---------+---------+-----------------------------+-----------+-------------+
| 1 | SIMPLE | p | range | PRIMARY | PRIMARY | 8 | NULL | 112017572 | Using where |
| 1 | SIMPLE | l | eq_ref | PRIMARY | PRIMARY | 8 | p.loan_ID | 1 | NULL |
| 1 | SIMPLE | af | eq_ref | PRIMARY | PRIMARY | 8 | p.fromAccount_ID | 1 | Using where |
| 1 | SIMPLE | at | eq_ref | PRIMARY | PRIMARY | 8 | p.toAccount_ID | 1 | Using where |
+----+-------------+-------+--------+---------------+---------+---------+-----------------------------+-----------+-------------+
Explain on Slave:
+----+-------------+-------+--------+-------------------------------------------------------------------------------+--------------------+---------+-----------------------------+--------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-------------------------------------------------------------------------------+--------------------+---------+-----------------------------+--------+----------------------------------------------+
| 1 | SIMPLE | l | index | PRIMARY | FK243910AAD869E6 | 9 | NULL | 804876 | Using index; Using temporary; Using filesort |
| 1 | SIMPLE | p | ref | PRIMARY,FK4BE7532292C5D482,FK4BE75322AE503A13,FK4BE75322382D11BC,POSTING_DATE | FK4BE75322382D11BC | 9 | l.ID | 101 | Using index condition; Using where |
| 1 | SIMPLE | af | eq_ref | PRIMARY | PRIMARY | 8 | p.fromAccount_ID | 1 | Using where |
| 1 | SIMPLE | at | eq_ref | PRIMARY | PRIMARY | 8 | p.toAccount_ID | 1 | Using where |
+----+-------------+-------+--------+-------------------------------------------------------------------------------+--------------------+---------+-----------------------------+--------+----------------------------------------------+
SELECT
p.ID AS 'payment_id',
p.loan_ID AS 'loan_id',
l.client_ID AS 'client_ID',
p.amount AS 'amount',
p.postingDate AS 'payment_date',
CASE
WHEN af.acc_type = 'POLCH' THEN 'wallet'
WHEN af.acc_type = 'PLTCH' THEN 'wallet'
WHEN af.acc_type = 'CNTT' THEN 'bank'
WHEN af.acc_type = 'CNT2' THEN 'bank'
WHEN af.acc_type = 'KONCH' THEN 'bank'
WHEN af.acc_type = 'KRDTM' THEN 'cash'
WHEN af.acc_type = 'LDRCH' THEN 'bank'
ELSE concat('UNKNOWN_',af.acc_type)
END AS 'payment_system_type',
af.description AS 'payment_system'
FROM Posting AS p
INNER JOIN Account AS af ON p.fromAccount_ID = af.ID
INNER JOIN Account AS at ON p.toAccount_ID = at.ID
INNER JOIN Loan AS l ON p.loan_id = l.ID
WHERE (
af.acc_type = 'KONCH'
OR af.acc_type = 'PLTCH'
OR af.acc_type = 'POLCH'
OR af.acc_type = 'KRDTM'
OR af.acc_type = 'LDRCH'
OR af.acc_type = 'CNT2'
OR af.acc_type = 'CNTT')
AND at.acc_type = 'ABON'
AND p.postingDate < DATE(now())
AND p.ID > 0
ORDER BY p.ID LIMIT 10000;
Loan - l
Posting - P
Master:
| Loan | CREATE TABLE `Loan` (
`ID` bigint(20) NOT NULL AUTO_INCREMENT,
`amount` decimal(19,4) DEFAULT NULL,
`amountToReturn` decimal(19,4) DEFAULT NULL,
`isGivenOut` bit(1) DEFAULT b'0',
`isPaid` bit(1) DEFAULT NULL,
`issueDate` datetime DEFAULT NULL,
`loanPeriod` int(11) DEFAULT NULL,
`productType` varchar(255) DEFAULT NULL,
`realPayDate` datetime DEFAULT NULL,
`client_ID` bigint(20) DEFAULT NULL,
`product_ID` bigint(20) DEFAULT NULL,
`givenOutDate` datetime DEFAULT NULL,
`isPaidByBank` bit(1) DEFAULT NULL,
`accountNumberNBKI` varchar(255) DEFAULT NULL,
`needManualProcessing` bit(1) DEFAULT NULL,
`isReverted` bit(1) DEFAULT b'0',
`showInNBCHReport` bit(1) DEFAULT b'1',
`stake` decimal(19,5) DEFAULT NULL,
`ignoreProlongation` bit(1) DEFAULT b'0',
`stakeAfter21` decimal(19,5) DEFAULT NULL,
`discount_id` bigint(20) DEFAULT NULL,
`showInEquifaxReport` bit(1) DEFAULT b'1',
`ignoreNbch` bit(1) DEFAULT b'0',
PRIMARY KEY (`ID`),
KEY `FK2439106EC0BA18` (`product_ID`),
KEY `ISPAID_INDEX` (`isPaid`) USING BTREE,
KEY `ISP_ISGOUT_INDEX` (`isPaid`,`isGivenOut`),
KEY `ISSUEDATE_INDEX` (`issueDate`),
KEY `FK243910735827C6` (`discount_id`),
KEY `idx_Loan_realPayDate` (`realPayDate`),
KEY `idx_Loan_givenOutDate` (`givenOutDate`),
KEY `FK243910AAD869E6` (`client_ID`),
CONSTRAINT `_FK243910735827C6` FOREIGN KEY (`discount_id`) REFERENCES `Discount` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2000623399 DEFAULT CHARSET=utf8
Posting | CREATE TABLE `Posting` (
`ID` bigint(20) NOT NULL AUTO_INCREMENT,
`amount` decimal(19,4) DEFAULT NULL,
`postingDate` datetime DEFAULT NULL,
`fromAccount_ID` bigint(20) DEFAULT NULL,
`loan_ID` bigint(20) DEFAULT NULL,
`toAccount_ID` bigint(20) DEFAULT NULL,
`sourceType` varchar(255) DEFAULT NULL,
PRIMARY KEY (`ID`),
KEY `FK4BE7532292C5D482` (`fromAccount_ID`),
KEY `FK4BE75322AE503A13` (`toAccount_ID`),
KEY `FK4BE75322382D11BC` (`loan_ID`),
KEY `POSTING_DATE` (`postingDate`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=230996702 DEFAULT CHARSET=utf8
Slave:
| Loan | CREATE TABLE `Loan` (
`ID` bigint(20) NOT NULL AUTO_INCREMENT,
`amount` decimal(19,4) DEFAULT NULL,
`amountToReturn` decimal(19,4) DEFAULT NULL,
`isGivenOut` bit(1) DEFAULT b'0',
`isPaid` bit(1) DEFAULT NULL,
`issueDate` datetime DEFAULT NULL,
`loanPeriod` int(11) DEFAULT NULL,
`productType` varchar(255) DEFAULT NULL,
`realPayDate` datetime DEFAULT NULL,
`client_ID` bigint(20) DEFAULT NULL,
`product_ID` bigint(20) DEFAULT NULL,
`givenOutDate` datetime DEFAULT NULL,
`isPaidByBank` bit(1) DEFAULT NULL,
`accountNumberNBKI` varchar(255) DEFAULT NULL,
`needManualProcessing` bit(1) DEFAULT NULL,
`isReverted` bit(1) DEFAULT b'0',
`showInNBCHReport` bit(1) DEFAULT b'1',
`stake` decimal(19,5) DEFAULT NULL,
`ignoreProlongation` bit(1) DEFAULT b'0',
`stakeAfter21` decimal(19,5) DEFAULT NULL,
`discount_id` bigint(20) DEFAULT NULL,
`showInEquifaxReport` bit(1) DEFAULT b'1',
`ignoreNbch` bit(1) DEFAULT b'0',
PRIMARY KEY (`ID`),
KEY `FK2439106EC0BA18` (`product_ID`),
KEY `ISPAID_INDEX` (`isPaid`) USING BTREE,
KEY `ISP_ISGOUT_INDEX` (`isPaid`,`isGivenOut`),
KEY `ISSUEDATE_INDEX` (`issueDate`),
KEY `FK243910735827C6` (`discount_id`),
KEY `idx_Loan_realPayDate` (`realPayDate`),
KEY `idx_Loan_givenOutDate` (`givenOutDate`),
KEY `FK243910AAD869E6` (`client_ID`),
CONSTRAINT `_FK243910735827C6` FOREIGN KEY (`discount_id`) REFERENCES `Discount` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2000623399 DEFAULT CHARSET=utf8
ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=4
Posting | CREATE TABLE `Posting` (
`ID` bigint(20) NOT NULL AUTO_INCREMENT,
`amount` decimal(19,4) DEFAULT NULL,
`postingDate` datetime DEFAULT NULL,
`fromAccount_ID` bigint(20) DEFAULT NULL,
`loan_ID` bigint(20) DEFAULT NULL,
`toAccount_ID` bigint(20) DEFAULT NULL,
`sourceType` varchar(255) DEFAULT NULL,
PRIMARY KEY (`ID`),
KEY `FK4BE7532292C5D482` (`fromAccount_ID`),
KEY `FK4BE75322AE503A13` (`toAccount_ID`),
KEY `FK4BE75322382D11BC` (`loan_ID`),
KEY `POSTING_DATE` (`postingDate`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=230996702 DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=4
If remove table Loan from Query
+----+-------------+-------+--------+------------------------------------------------------------+---------+---------+-----------------------------+-----------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+------------------------------------------------------------+---------+---------+-----------------------------+-----------+-------------+
| 1 | SIMPLE | p | range | PRIMARY,FK4BE7532292C5D482,FK4BE75322AE503A13,POSTING_DATE | PRIMARY | 8 | NULL | 107736559 | Using where |
| 1 | SIMPLE | af | eq_ref | PRIMARY | PRIMARY | 8 | smsfinance.p.fromAccount_ID | 1 | Using where |
| 1 | SIMPLE | at | eq_ref | PRIMARY | PRIMARY | 8 | smsfinance.p.toAccount_ID | 1 | Using where |
+----+-------------+-------+--------+------------------------------------------------------------+---------+---------+-----------------------------+-----------+-------------+
If I add
create index acc on Account(acc_type);
Plan:
+----+-------------+-------+--------+-------------------------------------------------------------------------------+--------------------+---------+---------------------------+------+--------------------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+-------------------------------------------------------------------------------+--------------------+---------+---------------------------+------+--------------------------------------------------------+
| 1 | SIMPLE | af | range | PRIMARY,acc | acc | 21 | NULL | 4192 | Using index condition; Using temporary; Using filesort |
| 1 | SIMPLE | p | ref | PRIMARY,FK4BE7532292C5D482,FK4BE75322AE503A13,FK4BE75322382D11BC,POSTING_DATE | FK4BE7532292C5D482 | 9 | smsfinance.af.ID | 54 | Using index condition; Using where |
| 1 | SIMPLE | l | eq_ref | PRIMARY | PRIMARY | 8 | smsfinance.p.loan_ID | 1 | NULL |
| 1 | SIMPLE | at | eq_ref | PRIMARY,acc | PRIMARY | 8 | smsfinance.p.toAccount_ID | 1 | Using where |
+----+-------------+-------+--------+-------------------------------------------------------------------------------+--------------------+---------+---------------------------+------+--------------------------------------------------------+
Query execute long time.
I suspect compression leads to different statistics, which can lead to different execution plans.
I see no use for the table Loan in this query. Removing it from the query may force the explain plans to be the same.
You have 200M rows in each table? Another speedup would be to shrink the tables.
Change BIGINT (8 bytes each) to INT UNSIGNED (4 bytes, range 0..4 billion) wherever possible.
Normalize the _type columns, replacing with SMALLINT UNSIGNED (2 bytes, range 0..64K) or other suitable integer type. Or turn the column into an ENUM if there are a finite, small number, of "types".
Does Account have INDEX(acc_type)? (Or at least starting with acc_type.)
Remove INNER JOIN Loan AS l ON p.loan_id = l.ID and replace l.client_ID AS 'client_ID', with
( SELECT client_ID FROM Loans WHERE ID = p.loan_id ) AS 'client_ID',
I think this will force a different query plan, perhaps a good one.

Why adding duplicate indexes to MySQL table caused longer query execution time?

Maybe the index wasn't relevant but I am experiencing a strange issue.
This is my select query:
SELECT DISTINCT completeAddress FROM DB_M3_Medium.AvailableAddressesV3 where postNr = 1050 ORDER BY completeAddress ASC;
My indexes:
create index postNrAndAddress_idx on DB_M3_Medium.AvailableAddressesV3 (completeAddress);
create index postNr_idx on DB_M3_Medium.AvailableAddressesV3 (completeAddress);
create index completeAddress_idx on DB_M3_Medium.AvailableAddressesV3 (completeAddress);
And besides that I've got a PK on an autoincrement id (idIndex).
The execution time of the select query before any of the manually created indexes were present was 2.4s.
Then I have created indexes (one by one):
1st index - select statement execution time - 2.1s
2nd index - select
statement execution time - 2.8s
3rd index - select statement
execution time - 12.7s
What's just happened?
EDIT:
Thank you guys for your comments. My explain statement result:
+----+-------------+----------------------+-------+-----------------------------------------------------+---------------------+---------+-----+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------------------+-------+-----------------------------------------------------+---------------------+---------+-----+---------+-------------+
| 1 | SIMPLE | AvailableAddressesV3 | index | postNrAndAddress_idx,postNr_idx,completeAddress_idx | completeAddress_idx | 363 | | 3526406 | Using where |
+----+-------------+----------------------+-------+-----------------------------------------------------+---------------------+---------+-----+---------+-------------+
Table structure:
+------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------------+--------------+------+-----+---------+----------------+
| vej_Navn | varchar(70) | YES | | | |
| husNr | varchar(20) | YES | | | |
| husbogstav | varchar(50) | YES | | | |
| etage | varchar(30) | YES | | | |
| side_DoerNr | varchar(20) | YES | | | |
| stedNavn | varchar(50) | YES | | | |
| postNr | varchar(15) | YES | MUL | | |
| postDistrikt | varchar(50) | YES | | | |
| lev_Adresse_UUID | varchar(50) | YES | | | |
| fiberstatus | varchar(15) | YES | | | |
| kommune_nr | varchar(35) | YES | | | |
| vej_Kode | varchar(35) | YES | | | |
| completeAddress | varchar(120) | YES | MUL | | |
| randomSalt | varchar(5) | YES | | | |
| id | int(11) | NO | PRI | | auto_increment |
+------------------+--------------+------+-----+---------+----------------+
Create table query:
CREATE TABLE `AvailableAddressesV3` (
`vej_Navn` varchar(70) COLLATE utf8_unicode_ci DEFAULT NULL,
`husNr` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`husbogstav` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`etage` varchar(30) COLLATE utf8_unicode_ci DEFAULT NULL,
`side_DoerNr` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`stedNavn` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`postNr` varchar(15) CHARACTER SET utf8 DEFAULT NULL,
`postDistrikt` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`lev_Adresse_UUID` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`fiberstatus` varchar(15) COLLATE utf8_unicode_ci DEFAULT NULL,
`kommune_nr` varchar(35) COLLATE utf8_unicode_ci DEFAULT NULL,
`vej_Kode` varchar(35) COLLATE utf8_unicode_ci DEFAULT NULL,
`completeAddress` varchar(120) COLLATE utf8_unicode_ci DEFAULT NULL,
`randomSalt` varchar(5) COLLATE utf8_unicode_ci DEFAULT NULL,
`id` int(11) NOT NULL AUTO_INCREMENT,
UNIQUE KEY `idIndex` (`id`),
KEY `postNrAndAddress_idx` (`postNr`,`completeAddress`),
KEY `postNr_idx` (`postNr`),
KEY `completeAddress_idx` (`completeAddress`)
) ENGINE=InnoDB AUTO_INCREMENT=3552718 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Based on your EXPLAIN output, the query is using the completeAddress_idx, probably because of the sort/distinct, but I'm guessing there are very few rows with the postNr = 1050 (in Copenhagen, right?) so it should be more efficient to use postNr_idx or postNrAndAddress_idx (sorting/distinct on a couple of hundred rows should be almost instant). Something is making the query execution planner miss the optimal query.
I have never tried this myself, but you could try the ANALYZE TABLE statement which updates table statistics, for example key cardinality, that could change how the optimizer works.
Either that, or I'm missing something simple - which seems likely :)
Edit
While debugging, it can be useful to force MySQL to use a specific index. Try the FORCE/USE INDEX hint.
I would never expect that this could be an issue or at least that I would get notified by WorkBench or JDBC with an error or at least a warning.
My select query should look like this:
SELECT DISTINCT completeAddress FROM DB_M3_Medium.AvailableAddressesV3 where postNr = '4000' ORDER BY completeAddress ASC;
The difference is the datatype of the postNr. Before I didn't have it wrapped in '.
That improved the select crazily, and then when I removed ORDER BY the execution time dropped down to 0.07s.
So basically what was happening, was that the SELECT query wasn't using any index because none of the indexes was suitable. When I did the EXPLAIN I was receiving NULL my Key column. I was trying to FORCE it, but it made no difference.
Then I have discovered this: Why isn't MySQL using any of these possible keys?
Where in the second answer he has mentioned it.

Cannot Add Foreign Key Constraint - I Checked almost everything

okay, so im having a weird issue with this foreign key constraint.
The data type is Int(11) and all tables and databases are using utf8_general_ci but I'm still getting an issue! the only table that inherited the Medicine_ID is the First Aid table.
Here's my schema:
tbl_Medicine
+--------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-------------+------+-----+---------+-------+
| Medicine_ID | int(11) | NO | PRI | NULL | |
| Brand_Name | varchar(45) | YES | | NULL | |
| Generic_Name | varchar(45) | YES | | NULL | |
| Countable | int(11) | YES | | NULL | |
+--------------+-------------+------+-----+---------+-------+
tbl_batch
+---------------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+---------+------+-----+---------+-------+
| Batch_ID | int(11) | NO | PRI | NULL | |
| Purchase_Date | date | YES | | NULL | |
+---------------+---------+------+-----+---------+-------+
tbl_Stock
+-----------------+---------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+---------+------+-----+---------+-------+
| Stock_ID | int(11) | NO | PRI | NULL | |
| Medicine_ID | int(11) | NO | PRI | NULL | |
| Batch_ID | int(11) | NO | PRI | NULL | |
| Expiration_Date | date | YES | | NULL | |
| Quantity | int(45) | YES | | NULL | |
| Unit_Price | int(11) | YES | | NULL | |
+-----------------+---------+------+-----+---------+-------+
tbl_inventory
+--------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-------------+------+-----+---------+-------+
| Inventory_ID | int(11) | NO | PRI | NULL | |
| Building | varchar(45) | NO | PRI | NULL | |
| Stock_ID | int(11) | NO | PRI | NULL | |
| Medicine_ID | int(11) | NO | PRI | NULL | |
| Quantity | varchar(45) | YES | | NULL | |
+--------------+-------------+------+-----+---------+-------+
tbl_Activity
+--------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------+-------------+------+-----+---------+-------+
| Activity_ID | varchar(10) | NO | PRI | NULL | |
| Purpose | varchar(50) | YES | | NULL | |
| Organization | varchar(50) | YES | | NULL | |
| Reciever | varchar(50) | YES | | NULL | |
| Recieve_Date | date | YES | | NULL | |
| Act_Date | date | YES | | NULL | |
| Validator | varchar(10) | YES | | NULL | |
+--------------+-------------+------+-----+---------+-------+
Heres the creation query:
CREATE TABLE IF NOT EXISTS `csb_inventory`.`tbl_batch` (
`Batch_ID` INT(11) NOT NULL COMMENT '',
`Purchase_Date` DATE NULL DEFAULT NULL COMMENT '',
PRIMARY KEY (`Batch_ID`) COMMENT '')
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8;
CREATE TABLE IF NOT EXISTS `csb_inventory`.`tbl_medicine` (
`Medicine_ID` INT(11) NOT NULL COMMENT '',
`Brand_Name` VARCHAR(45) NULL DEFAULT NULL COMMENT '',
`Generic_Name` VARCHAR(45) NULL DEFAULT NULL COMMENT '',
`Countable` INT(11) NULL DEFAULT NULL COMMENT '',
PRIMARY KEY (`Medicine_ID`) COMMENT '')
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8;
CREATE TABLE IF NOT EXISTS `csb_inventory`.`tbl_stock` (
`Stock_ID` INT(11) NOT NULL COMMENT '',
`Medicine_ID` INT(11) NOT NULL COMMENT '',
`Batch_ID` INT(11) NOT NULL COMMENT '',
`Expiration_Date` DATE NULL DEFAULT NULL COMMENT '',
`Quantity` INT(45) NULL DEFAULT NULL COMMENT '',
`Unit_Price` INT(11) NULL DEFAULT NULL COMMENT '',
PRIMARY KEY (`Stock_ID`, `Medicine_ID`, `Batch_ID`) COMMENT '',
INDEX `fk_tbl_Stock_tbl_Medicine_idx` (`Medicine_ID` ASC) COMMENT '',
INDEX `fk_tbl_Stock_tbl_Batch1_idx` (`Batch_ID` ASC) COMMENT '',
CONSTRAINT `batch`
FOREIGN KEY (`Batch_ID`)
REFERENCES `csb_inventory`.`tbl_batch` (`Batch_ID`)
ON DELETE NO ACTION
ON UPDATE NO ACTION,
CONSTRAINT `medicine`
FOREIGN KEY (`Medicine_ID`)
REFERENCES `csb_inventory`.`tbl_medicine` (`Medicine_ID`)
ON DELETE NO ACTION
ON UPDATE NO ACTION)
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8;
CREATE TABLE IF NOT EXISTS `csb_inventory`.`tbl_inventory` (
`Inventory_ID` INT(11) NOT NULL COMMENT '',
`Building` VARCHAR(45) NOT NULL COMMENT '',
`Stock_ID` INT(11) NOT NULL COMMENT '',
`Medicine_ID` INT(11) NOT NULL COMMENT '',
`Quantity` VARCHAR(45) NULL DEFAULT NULL COMMENT '',
PRIMARY KEY (`Inventory_ID`, `Building`, `Stock_ID`, `Medicine_ID`) COMMENT '',
INDEX `fk_tbl_Inventory_tbl_Building1_idx` (`Building` ASC) COMMENT '',
INDEX `fk_tbl_Inventory_tbl_Stock1_idx` (`Stock_ID` ASC, `Medicine_ID` ASC) COMMENT '',
CONSTRAINT `building`
FOREIGN KEY (`Building`)
REFERENCES `csb_patient`.`tbl_building` (`Building`),
CONSTRAINT `stockitem`
FOREIGN KEY (`Stock_ID` , `Medicine_ID`)
REFERENCES `csb_inventory`.`tbl_stock` (`Stock_ID` , `Medicine_ID`)
ON DELETE NO ACTION
ON UPDATE NO ACTION)
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8;
CREATE TABLE IF NOT EXISTS `csb_inventory`.`tbl_firstaid` (
`FirstAid_ID` VARCHAR(10) NOT NULL COMMENT '',
`Activity_ID` VARCHAR(10) NOT NULL COMMENT '',
`Quantity` SMALLINT(6) NOT NULL COMMENT '',
`Inventory_ID` INT(11) NOT NULL COMMENT '',
`Medicine_ID` INT(11) NOT NULL COMMENT '',
PRIMARY KEY (`FirstAid_ID`, `Activity_ID`, `Inventory_ID`, `Medicine_ID`) COMMENT '',
INDEX `activity_id` (`Activity_ID` ASC) COMMENT '',
INDEX `fk_tbl_firstaid_tbl_inventory1_idx` (`Inventory_ID` ASC, `Medicine_ID` ASC) COMMENT '',
CONSTRAINT `activity_id`
FOREIGN KEY (`Activity_ID`)
REFERENCES `csb_inventory`.`tbl_activity` (`Activity_ID`)
ON DELETE NO ACTION
ON UPDATE NO ACTION,
CONSTRAINT `fk_tbl_firstaid_tbl_inventory1`
FOREIGN KEY (`Inventory_ID` , `Medicine_ID`)
REFERENCES `csb_inventory`.`tbl_inventory` (`Inventory_ID` , `Medicine_ID`)
ON DELETE NO ACTION
ON UPDATE NO ACTION)
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8;
I have read alot of threads regarding the same issue and tested them out.
I made sure that im using the same table_type, collation, and data_type. I even checked all other tables that are related that might have been causing the issue but no good. I was able to successfully add foreign keys to firstAid but only the Medicine_ID is the one failing.
help? xD
Make sure that you have tbl_building on before create tbl_inventory.

Changing field collation to utf-8 causes duplicate key error

Why does simply changing the collation cause a duplicate key error?
mysql> describe phppos_items;
+-----------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------------+--------------+------+-----+---------+----------------+
| name | varchar(255) | NO | MUL | NULL | |
| category | varchar(255) | NO | MUL | NULL | |
| supplier_id | int(11) | YES | MUL | NULL | |
| item_number | varchar(255) | YES | UNI | NULL | |
| description | varchar(255) | NO | | NULL | |
| cost_price | double(15,2) | NO | | NULL | |
| unit_price | double(15,2) | NO | | NULL | |
| quantity | double(15,2) | NO | | 0.00 | |
| reorder_level | double(15,2) | NO | | 0.00 | |
| location | varchar(255) | NO | | NULL | |
| item_id | int(10) | NO | PRI | NULL | auto_increment |
| allow_alt_description | tinyint(1) | NO | | NULL | |
| is_serialized | tinyint(1) | NO | | NULL | |
| deleted | int(1) | NO | MUL | 0 | |
+-----------------------+--------------+------+-----+---------+----------------+
14 rows in set (0.01 sec)
mysql> ALTER TABLE `phppos_items` CHANGE `name` `name` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL , CHANGE `category` `category` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL , CHANGE `item_number` `item_number` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NULL DEFAULT NULL , CHANGE `description` `description` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL , CHANGE `location` `location` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL;
ERROR 1062 (23000): Duplicate entry ' ' for key 'item_number'
Create table:
| phppos_items | CREATE TABLE `phppos_items` (
`name` varchar(255) CHARACTER SET latin1 NOT NULL,
`category` varchar(255) CHARACTER SET latin1 NOT NULL,
`supplier_id` int(11) DEFAULT NULL,
`item_number` varchar(255) CHARACTER SET latin1 DEFAULT NULL,
`description` varchar(255) CHARACTER SET latin1 NOT NULL,
`cost_price` double(15,2) NOT NULL,
`unit_price` double(15,2) NOT NULL,
`quantity` double(15,2) NOT NULL DEFAULT '0.00',
`reorder_level` double(15,2) NOT NULL DEFAULT '0.00',
`location` varchar(255) CHARACTER SET latin1 NOT NULL,
`item_id` int(10) NOT NULL AUTO_INCREMENT,
`allow_alt_description` tinyint(1) NOT NULL,
`is_serialized` tinyint(1) NOT NULL,
`deleted` int(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`item_id`),
UNIQUE KEY `item_number` (`item_number`),
KEY `phppos_items_ibfk_1` (`supplier_id`),
KEY `name` (`name`),
KEY `category` (`category`),
KEY `deleted` (`deleted`),
CONSTRAINT `phppos_items_ibfk_1` FOREIGN KEY (`supplier_id`) REFERENCES `phppos_suppliers` (`person_id`)
) ENGINE=InnoDB AUTO_INCREMENT=1560 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci |
When modifying the field collation, perhaps MySQL is as well trying to convert the data from latin1 to utf8. It might be that after this conversion, some of your data in the item_number column is containing duplicates, a space character perhaps it seems from the error.
I think these steps might help to identify which are the conflicting rows:
select item_id, name and item_number columns for all rows and keep the entries in some XLS file perhaps
remove the unique index on item_number column
run the ALTER TABLE statement that should now run successfully
redo step 1 and compare the output with that from step 1 and find out for which of the items is item_number bearing the same values
Hope this helps!
To fix that, you need to know which rows have duplicate utf-8 values.
You can do that by running this query:
SELECT * FROM (
SELECT CONVERT(YOUR_COLUMN USING utf8mb4) AS c FROM YOUR_TABLE
) AS A GROUP BY c HAVING COUNT(*) > 1

how to optimize this query in MYSQL?

mysql> explain select a.id,a.title from users c
-> straight_join iask a on c.id=a.uid
-> straight_join ianswer b on a.id=b.iaskid
->
-> where (c.last_check is null or b.created>c.last_check) and c.id in (1,2)
-> group by a.id;
+----+-------------+-------+-------+---------------------------+------------+---------+----------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------------------+------------+---------+----------+------+----------------------------------------------+
| 1 | SIMPLE | c | range | PRIMARY,i_users_lastcheck | PRIMARY | 4 | NULL | 2 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | a | ref | PRIMARY,i_iask_uid | i_iask_uid | 4 | bbs.c.id | 2 | |
| 1 | SIMPLE | b | ALL | i_ianswer_iaskid | NULL | NULL | NULL | 17 | Using where; Using join buffer |
+----+-------------+-------+-------+---------------------------+------------+---------+----------+------+----------------------------------------------+
3 rows in set (0.00 sec)
change the above "in (1,2)" into "=1" will make it all using index:
mysql> explain select a.id,a.title from iask a
-> join ianswer b on a.id=b.iaskid
-> join users c on c.id=a.uid
-> where (c.last_check is null or b.created>c.last_check) and c.id=1
-> group by a.id;
+----+-------------+-------+-------+---------------------------+------------------+---------+----------+------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+-------+---------------------------+------------------+---------+----------+------+---------------------------------+
| 1 | SIMPLE | c | const | PRIMARY,i_users_lastcheck | PRIMARY | 4 | const | 1 | Using temporary; Using filesort |
| 1 | SIMPLE | a | ref | PRIMARY,i_iask_uid | i_iask_uid | 4 | const | 1 | Using where |
| 1 | SIMPLE | b | ref | i_ianswer_iaskid | i_ianswer_iaskid | 4 | bbs.a.id | 8 | Using where |
+----+-------------+-------+-------+---------------------------+------------------+---------+----------+------+---------------------------------+
3 rows in set (0.00 sec)
table structure as follows:
mysql> show create table users;
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Table | Create Table
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| users | CREATE TABLE `users` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`email` varchar(128) NOT NULL,
`password` varchar(32) NOT NULL,
`screen_name` varchar(64) DEFAULT NULL,
`reputation` int(10) unsigned NOT NULL DEFAULT '0',
`imtype` varchar(1) DEFAULT '0' COMMENT '0--email,1--gtalk,2--msn',
`last_check` datetime DEFAULT NULL,
`robotno` int(10) unsigned DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `u_users_email` (`email`),
UNIQUE KEY `u_users_screen_name` (`screen_name`),
KEY `i_users_lastcheck` (`last_check`),
KEY `i_users_imtype_robotno` (`imtype`,`robotno`)
) ENGINE=InnoDB AUTO_INCREMENT=31 DEFAULT CHARSET=utf8 |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 row in set (0.02 sec)
mysql> show create table iask;
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| Table | Create Table
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| iask | CREATE TABLE `iask` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`uid` int(10) unsigned NOT NULL,
`title` varchar(250) NOT NULL,
`body` text,
`tags` varchar(100) NOT NULL,
`views` int(10) unsigned DEFAULT '0',
`votes` int(10) unsigned DEFAULT '0',
`answer_id` int(10) unsigned DEFAULT NULL,
`created` datetime NOT NULL,
`keywords` text,
PRIMARY KEY (`id`),
KEY `i_iask_uid` (`uid`),
FULLTEXT KEY `keywords` (`keywords`)
) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 |
+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 row in set (0.00 sec)
mysql> show create table ianswer;
+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ianswer | CREATE TABLE `ianswer` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`iaskid` int(10) unsigned NOT NULL,
`uid` int(10) unsigned DEFAULT NULL,
`body` text,
`votes` int(10) unsigned DEFAULT '0',
`created` datetime NOT NULL,
`anonymous` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`),
KEY `i_ianswer_iaskid` (`iaskid`)
) ENGINE=InnoDB AUTO_INCREMENT=20 DEFAULT CHARSET=utf8 |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
mysql>
If you are asking how to make the first query use an index like the second query does, I suspect it is because there aren't enough rows in your table to justify two index lookups. The row count for a full table scan is 17 rows, so it thinks it is faster just to spin across the rows looking for two ids. If you are concerned, you could try populating the table to see if it still chooses a full table scan.