Why adding duplicate indexes to MySQL table caused longer query execution time? - mysql

Maybe the index wasn't relevant but I am experiencing a strange issue.
This is my select query:
SELECT DISTINCT completeAddress FROM DB_M3_Medium.AvailableAddressesV3 where postNr = 1050 ORDER BY completeAddress ASC;
My indexes:
create index postNrAndAddress_idx on DB_M3_Medium.AvailableAddressesV3 (completeAddress);
create index postNr_idx on DB_M3_Medium.AvailableAddressesV3 (completeAddress);
create index completeAddress_idx on DB_M3_Medium.AvailableAddressesV3 (completeAddress);
And besides that I've got a PK on an autoincrement id (idIndex).
The execution time of the select query before any of the manually created indexes were present was 2.4s.
Then I have created indexes (one by one):
1st index - select statement execution time - 2.1s
2nd index - select
statement execution time - 2.8s
3rd index - select statement
execution time - 12.7s
What's just happened?
EDIT:
Thank you guys for your comments. My explain statement result:
+----+-------------+----------------------+-------+-----------------------------------------------------+---------------------+---------+-----+---------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------------------+-------+-----------------------------------------------------+---------------------+---------+-----+---------+-------------+
| 1 | SIMPLE | AvailableAddressesV3 | index | postNrAndAddress_idx,postNr_idx,completeAddress_idx | completeAddress_idx | 363 | | 3526406 | Using where |
+----+-------------+----------------------+-------+-----------------------------------------------------+---------------------+---------+-----+---------+-------------+
Table structure:
+------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------------+--------------+------+-----+---------+----------------+
| vej_Navn | varchar(70) | YES | | | |
| husNr | varchar(20) | YES | | | |
| husbogstav | varchar(50) | YES | | | |
| etage | varchar(30) | YES | | | |
| side_DoerNr | varchar(20) | YES | | | |
| stedNavn | varchar(50) | YES | | | |
| postNr | varchar(15) | YES | MUL | | |
| postDistrikt | varchar(50) | YES | | | |
| lev_Adresse_UUID | varchar(50) | YES | | | |
| fiberstatus | varchar(15) | YES | | | |
| kommune_nr | varchar(35) | YES | | | |
| vej_Kode | varchar(35) | YES | | | |
| completeAddress | varchar(120) | YES | MUL | | |
| randomSalt | varchar(5) | YES | | | |
| id | int(11) | NO | PRI | | auto_increment |
+------------------+--------------+------+-----+---------+----------------+
Create table query:
CREATE TABLE `AvailableAddressesV3` (
`vej_Navn` varchar(70) COLLATE utf8_unicode_ci DEFAULT NULL,
`husNr` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`husbogstav` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`etage` varchar(30) COLLATE utf8_unicode_ci DEFAULT NULL,
`side_DoerNr` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`stedNavn` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`postNr` varchar(15) CHARACTER SET utf8 DEFAULT NULL,
`postDistrikt` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`lev_Adresse_UUID` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`fiberstatus` varchar(15) COLLATE utf8_unicode_ci DEFAULT NULL,
`kommune_nr` varchar(35) COLLATE utf8_unicode_ci DEFAULT NULL,
`vej_Kode` varchar(35) COLLATE utf8_unicode_ci DEFAULT NULL,
`completeAddress` varchar(120) COLLATE utf8_unicode_ci DEFAULT NULL,
`randomSalt` varchar(5) COLLATE utf8_unicode_ci DEFAULT NULL,
`id` int(11) NOT NULL AUTO_INCREMENT,
UNIQUE KEY `idIndex` (`id`),
KEY `postNrAndAddress_idx` (`postNr`,`completeAddress`),
KEY `postNr_idx` (`postNr`),
KEY `completeAddress_idx` (`completeAddress`)
) ENGINE=InnoDB AUTO_INCREMENT=3552718 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

Based on your EXPLAIN output, the query is using the completeAddress_idx, probably because of the sort/distinct, but I'm guessing there are very few rows with the postNr = 1050 (in Copenhagen, right?) so it should be more efficient to use postNr_idx or postNrAndAddress_idx (sorting/distinct on a couple of hundred rows should be almost instant). Something is making the query execution planner miss the optimal query.
I have never tried this myself, but you could try the ANALYZE TABLE statement which updates table statistics, for example key cardinality, that could change how the optimizer works.
Either that, or I'm missing something simple - which seems likely :)
Edit
While debugging, it can be useful to force MySQL to use a specific index. Try the FORCE/USE INDEX hint.

I would never expect that this could be an issue or at least that I would get notified by WorkBench or JDBC with an error or at least a warning.
My select query should look like this:
SELECT DISTINCT completeAddress FROM DB_M3_Medium.AvailableAddressesV3 where postNr = '4000' ORDER BY completeAddress ASC;
The difference is the datatype of the postNr. Before I didn't have it wrapped in '.
That improved the select crazily, and then when I removed ORDER BY the execution time dropped down to 0.07s.
So basically what was happening, was that the SELECT query wasn't using any index because none of the indexes was suitable. When I did the EXPLAIN I was receiving NULL my Key column. I was trying to FORCE it, but it made no difference.
Then I have discovered this: Why isn't MySQL using any of these possible keys?
Where in the second answer he has mentioned it.

Related

How to improve execution time of a Laravel Query Builder generated SQL query

I have three tables that are concerned by this query
CREATE TABLE `tags` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`latName` varchar(191) NOT NULL,
`araName` varchar(191) NOT NULL,
`active` tinyint(1) NOT NULL DEFAULT 0,
`img_name` varchar(191) DEFAULT NULL,
`icon` varchar(191) DEFAULT NULL,
`rgba_color` varchar(191) DEFAULT NULL,
`color` varchar(191) DEFAULT NULL,
`overlay` varchar(191) DEFAULT NULL,
`position` int(11) NOT NULL,
`mdi_icon` varchar(191) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `tags_latname_unique` (`latName`),
UNIQUE KEY `tags_araname_unique` (`araName`)
) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=utf8mb3
CREATE TABLE `newspapers` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`latName` varchar(191) NOT NULL,
`araName` varchar(191) NOT NULL,
`img_name` varchar(191) DEFAULT NULL,
`active` tinyint(1) NOT NULL DEFAULT 0,
PRIMARY KEY (`id`),
UNIQUE KEY `newspapers_latname_unique` (`latName`),
UNIQUE KEY `newspapers_araname_unique` (`araName`)
) ENGINE=InnoDB AUTO_INCREMENT=21 DEFAULT CHARSET=utf8mb3
CREATE TABLE `articles` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`newspaper_id` bigint(20) unsigned NOT NULL,
`tag_id` bigint(20) unsigned NOT NULL,
`seen` int(10) unsigned NOT NULL,
`link` varchar(1000) NOT NULL,
`title` varchar(191) NOT NULL,
`img_name` varchar(191) NOT NULL,
`date` datetime NOT NULL,
`paragraph` text NOT NULL,
`read_time` int(11) DEFAULT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `articles_link_unique` (`link`),
UNIQUE KEY `articles_img_name_unique` (`img_name`),
KEY `articles_newspaper_id_foreign` (`newspaper_id`),
KEY `articles_tag_id_foreign` (`tag_id`),
CONSTRAINT `articles_newspaper_id_foreign` FOREIGN KEY (`newspaper_id`) REFERENCES `newspapers` (`id`),
CONSTRAINT `articles_tag_id_foreign` FOREIGN KEY (`tag_id`) REFERENCES `tags` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=47421 DEFAULT CHARSET=utf8mb3
Basically, I want to load the latest 5 articles (ordered by date) that have an active newspaper and active tag.
Right now articles table contains about 40k entries.
This is the query generated by Laravel's query builder
SELECT `articles`.*
FROM `articles`
INNER JOIN `tags` ON `tags`.`id` = `articles`.`tag_id`
AND `tags`.`active` = 1
INNER JOIN `newspapers` ON `newspapers`.`id` = `articles`.`newspaper_id`
AND `newspapers`.`active` = 1
ORDER BY `date` DESC
LIMIT 5;
It takes Mysql about 6sec to run the query, when I remove the ORDER BY clause, the query becomes very fast (0.001sec).
Here is the query explanation:
+------+-------------+------------+--------+-------------------------------------------------------+-------------------------------+---------+------------------------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+------------+--------+-------------------------------------------------------+-------------------------------+---------+------------------------+------+----------------------------------------------+
| 1 | SIMPLE | newspapers | ALL | PRIMARY | NULL | NULL | NULL | 18 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | articles | ref | articles_newspaper_id_foreign,articles_tag_id_foreign | articles_newspaper_id_foreign | 8 | mouhim.newspapers.id | 1127 | |
| 1 | SIMPLE | tags | eq_ref | PRIMARY | PRIMARY | 8 | mouhim.articles.tag_id | 1 | Using where |
+------+-------------+------------+--------+-------------------------------------------------------+-------------------------------+---------+------------------------+------+----------------------------------------------+
I tried creating an index on the date attribute but it didn't help.
for convenience, this is how I am using Query Builder for this query:
Article::select("articles.*")
->join("tags", function ($join) {
$join->on("tags.id", "articles.tag_id")
->where("tags.active", 1);
})
->join("newspapers", function ($join) {
$join->on("newspapers.id", "articles.newspaper_id")
->where("newspapers.active", 1);
})
->orderBy("date", "desc")
->paginate(5)
At first, I was using Eloquent (whereHas) but Eloquent was generating non optimized query using (where exists), so I had to go the joins way.
What can I do to improve execution time of this query?
Result of SHOW INDEXES FROM articles;
+----------+------------+-------------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | Ignored |
+----------+------------+-------------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+
| articles | 0 | PRIMARY | 1 | id | A | 36072 | NULL | NULL | | BTREE | | | NO |
| articles | 0 | articles_link_unique | 1 | link | A | 36072 | NULL | NULL | | BTREE | | | NO |
| articles | 0 | articles_img_name_unique | 1 | img_name | A | 36072 | NULL | NULL | | BTREE | | | NO |
| articles | 1 | articles_newspaper_id_foreign | 1 | newspaper_id | A | 32 | NULL | NULL | | BTREE | | | NO |
| articles | 1 | articles_tag_id_foreign | 1 | tag_id | A | 12 | NULL | NULL | | BTREE | | | NO |
| articles | 1 | data | 1 | date | A | 36072 | NULL | NULL | | BTREE | | | NO |
+----------+------------+-------------------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+
This query was suggested by Rick James as a solution
SELECT `articles`.*
FROM `articles`
WHERE EXISTS ( SELECT 1 FROM tags WHERE id = `articles`.`tag_id` and active = 1)
AND EXISTS ( SELECT 1 FROM newspapers WHERE id = `articles`.`newspaper_id` and active = 1)
ORDER BY `date` DESC
LIMIT 5;
Running EXPLAIN on this query yields the following result
+------+-------------+------------+--------+-------------------------------------------------------+-------------------------------+---------+------------------------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+------------+--------+-------------------------------------------------------+-------------------------------+---------+------------------------+------+----------------------------------------------+
| 1 | PRIMARY | newspapers | ALL | PRIMARY | NULL | NULL | NULL | 18 | Using where; Using temporary; Using filesort |
| 1 | PRIMARY | articles | ref | articles_newspaper_id_foreign,articles_tag_id_foreign | articles_newspaper_id_foreign | 8 | mouhim.newspapers.id | 1127 | |
| 1 | PRIMARY | tags | eq_ref | PRIMARY | PRIMARY | 8 | mouhim.articles.tag_id | 1 | Using where |
+------+-------------+------------+--------+-------------------------------------------------------+-------------------------------+---------+------------------------+------+----------------------------------------------+
Assuming you don't want dups, change to this; it is likely to be much faster:
SELECT `articles`.*
FROM `articles`
WHERE EXISTS ( SELECT 1 FROM tags
WHERE id = `articles`.`tag_id` )
AND EXISTS ( SELECT 1 FROM newspapers
WHERE id = `articles`.`newspaper_id` )
ORDER BY `date` DESC
LIMIT 5;
Also, have this index on articles:
INDEX(date)
(This is a rare use case for starting index with a column that will be used in a 'range'.)
(Sorry, I don't speak 'Laravel'; maybe someone else can help with that part.)
PS. Having 3 UNIQUE keys on a table is highly unusual. It often indicates a problem with the schema design.
each article has one and only one Tag associated with it
Can multiple articles have the same Tag?
when I remove the ORDER BY clause, the query becomes very fast (0.001sec).
That is because you get whatever 5 rows are easy to return to you. Clearly the ORDER BY is part of the requirement. "Using temporary; Using filesort" says there was at least a sort. It will actually be a "file" sort -- because SELECT * includes a TEXT column. (There is a technique to avoid "file", but I don't think it is needed here.)
I am not sure if the two queries are supposed to be same, but they are not.
Anyway for the second query I think this should be better
Article::leftJoin('tags', 'articles.tag_id', '=', 'tags.id)
->where('tags.latName', $tag)
->orderBy("articles.date", "desc")
->select(['articles.*'])
->paginate(5);
The problem is probably, that the subquery you created in whereIn is slowing it down and whereIn itself may as well slow your query. This may be eased by using join and where.
As for the first query, can you show how you did the index for date? :)

How can I delete the repeated data in MYSQL with over ten million data in it?

I crawed lots of data, and saved it into mysql table, but there're some data duplicated, and I want to delete them in a effective way.
table (ads_info)
+------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| ad_id | varchar(64) | YES | MUL | NULL | |
| adset_id | varchar(64) | YES | MUL | NULL | |
| campaign_id | varchar(64) | YES | | NULL | |
| account_id | varchar(64) | YES | MUL | NULL | |
| conversion_specs | text | YES | | NULL | |
| creative | text | YES | | NULL | |
| effective_status | varchar(32) | YES | | NULL | |
| status | varchar(32) | YES | | NULL | |
| name | varchar(255) | YES | | NULL | |
| tracking_specs | text | YES | | NULL | |
| object_store_url | varchar(255) | YES | | NULL | |
| link | varchar(255) | YES | | NULL | |
| object_type | varchar(32) | YES | | NULL | |
| updated_time | timestamp | YES | | NULL | |
| created_time | timestamp | YES | | NULL | |
+------------------+--------------+------+-----+---------+----------------+
show create table ads_info
CREATE TABLE `ads_info` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`ad_id` varchar(64) DEFAULT NULL,
`adset_id` varchar(64) DEFAULT NULL,
`campaign_id` varchar(64) DEFAULT NULL,
`account_id` varchar(64) DEFAULT NULL,
`conversion_specs` text,
`creative` text,
`effective_status` varchar(32) DEFAULT NULL,
`status` varchar(32) DEFAULT NULL,
`name` varchar(255) DEFAULT NULL,
`tracking_specs` text,
`object_store_url` varchar(255) DEFAULT NULL,
`link` varchar(255) DEFAULT NULL,
`object_type` varchar(32) DEFAULT NULL,
`updated_time` timestamp NULL DEFAULT NULL,
`created_time` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `ad_id` (`ad_id`),
KEY `adset_id` (`adset_id`),
KEY `account_id` (`account_id`)
) ENGINE=InnoDB AUTO_INCREMENT=18827534 DEFAULT CHARSET=utf8mb4
There're over ten million ad info in the table, and about 40 thusand repeated. And I want delete all those repeated data.
Here's my poor trial
1)select all repeated ad_id
select ad_id from ads_info group by ad_id having count(id) > 1;
#42387 rows in set (12.42 sec)
The query cost 12s, but I don't know how to do an optimization.
2) use subquery to delete all these repeated data.
delete from ads_info where ad_id in ( select ad_id from (select ad_id from ads_info group by ad_id having count(id) > 1) t);
But I failed to get response from mysql with this trial, it seemed to be hanged with the query.
How can I delete these repeated data?
You needed a UNIQUE key in the first place. This will add it and dedup:
ALTER IGNORE TABLE ads_info
ADD UNIQUE KEY(ad_id);
If you want delete all the occurence then
Instead of IN clause you could try using a join
delete ads_info
from ads_info
INNER JOIN (
select ad_id
from ads_info
group by ad_id
having count(*) > 1
) T ON T.ad_id = ads_info.ad_id
be sure you have and index on ads_info.ad_id
if you have index .. but the query optimizer don't use and you are sure is a valid index you could try using USE or FORCE
delete ads_info
from ads_info
INNER JOIN (
select ad_id
from ads_info
group by ad_id
having count(*) > 1
) T FORCE INDEX FOR JOIN (`ad_id`) ON T.ad_id = ads_info.ad_id

MySQL query is slow on big table

I dont have other ideas on how to optimize this query, runs about 2,5 sec with goodsXML table over 2 million rows. It looks like order by is slowing a lot, but i cant remove it. Also, it depends a lot on how many items selected here gx.categoryID IN(892), because later another tables joins this items set. I cant make joins after this option, because joined tables perform in where clauses.
SELECT MD5(CONCAT(gx.id,598)) citySort,gx.dateCreated lastModifiedSince,IF(DATE(gx.dateModified)>=(IF((EXTRACT(HOUR FROM NOW()) BETWEEN 0 AND 6),DATE(NOW() -INTERVAL 6 HOUR),DATE(NOW()))) OR DATE(gx.dateModified)>=DATE(NOW()),1,0) isActual,
gx.id,p.producerName,gx.categoryID,gx.name,CONCAT('::',gxi.imageName) images,IF(CONCAT('::',gxi.imageName)!='',1,0) imExist,gx.price,gx.oldPrice,gx.oldPricePt,gx.sourceUrl,IF(s.offerPostingType='XML',IF(s.alternateName!='',s.alternateName,s.name),CONCAT(u.lastName,' ',u.name)) shopName,s.logoName,
s.id shopID,s.active shopActive,s.offerPostingType,c.titleAdd,'Москва' cityName,
IF((s.cityID='598' AND s.deliveryByCity=1) OR (sa.cityID='598' AND sa.deliveryByCity=1) OR (s.deliveryByMRCities LIKE '%^598^%' AND s.deliveryByMR=1),1,0) deliveryInYourCity,
IF(s.deliveryByCityAll=1 OR (s.cityID='598' AND s.deliveryByCity=1) OR (sa.cityID='598' AND sa.deliveryByCity=1) OR (s.deliveryByMRCities LIKE '%^598^%' AND s.deliveryByMR=1),1,0) deliveryByCity,
IF(s.deliveryByMail=1,1,0) deliveryByMail,
IF(s.deliveryBySelfAll=1 OR (s.cityID='598' AND s.deliveryBySelf=1) OR (sa.cityID='598' AND sa.deliveryBySelf=1),1,0) deliveryBySelf
FROM goodsXML gx
JOIN category c ON c.id=gx.categoryID
LEFT JOIN producer p ON p.id=gx.producerID
JOIN shop s ON s.id=gx.shopID
LEFT JOIN shopAddress sa ON sa.shopID=s.id
LEFT JOIN users u ON u.id=s.userID
LEFT JOIN goodsXMLImages gxi ON gxi.goodsXMLID=gx.id AND gxi.isMain = 1
WHERE 1=1 AND (s.cityID='598' OR s.deliveryByCityAll=1 OR s.deliveryBySelfAll=1 OR s.deliveryByMail=1 OR sa.cityID='598' OR (s.deliveryByMR=1 AND s.deliveryByMRCities LIKE '%^598^%')) AND (s.isPaying=0 OR u.balance>0) AND gx.categoryID IN(892)
GROUP BY gx.id
ORDER BY isActual DESC,imExist DESC,gx.PPC DESC,gx.payPrior ASC,citySort DESC
LIMIT 0,40
Explain is following:
+----+-------------+-------+--------+--------------------------------+-----------------+----------------+------------------------+--------------------+--------------------------------+
| ID | SELECT_TYPE | TABLE | TYPE | POSSIBLE_KEYS | KEY | KEY_LEN | REF | ROWS | EXTRA |
+----+-------------+-------+--------+--------------------------------+-----------------+----------------+------------------------+--------------------+--------------------------------+
| | | | | | | | | | |
| 1 | SIMPLE | c | const | PRIMARY | PRIMARY | 4 | const | 1 | Using temporary; Using filesor |
| | | | | | | | | | |
| 1 | SIMPLE | gx | ref | ixGroupNameCategoryIDShopIDPro | ixCategoryID... | ixCategoryIDid | 4 | const | 82005 |
| | | | | ducerID | | | | | |
| | | | | | | | | | |
| 1 | SIMPLE | s | eq_ref | PRIMARY | deliveryByMR | PRIMARY | 4 | vsesrazu.gx.shopID | 1 |
| | | | | | | | | | |
| 1 | SIMPLE | sa | ref | shopKey | shopKey | 5 | vsesrazu.s.id | 2 | Using where |
| | | | | | | | | | |
| 1 | SIMPLE | u | eq_ref | PRIMARY | PRIMARY | 4 | vsesrazu.s.userID | 1 | Using where |
| | | | | | | | | | |
| 1 | SIMPLE | p | eq_ref | PRIMARY | PRIMARY | 4 | vsesrazu.gx.producerID | 1 | |
| | | | | | | | | | |
| 1 | SIMPLE | gxi | ref | over | over | 4 | vsesrazu.gx.id | 1 | |
+----+-------------+-------+--------+--------------------------------+-----------------+----------------+------------------------+--------------------+--------------------------------+
Show create table for goodsXML:
CREATE TABLE goodsXML (
id bigint(20) unsigned NOT NULL AUTO_INCREMENT,
localID char(255) NOT NULL,
groupID char(255) DEFAULT NULL,
dateCreated datetime NOT NULL,
dateModified datetime NOT NULL,
dateModifiedPrice datetime NOT NULL,
name char(255) DEFAULT NULL,
nameHash char(32) NOT NULL,
groupName char(255) DEFAULT NULL,
newGroupName char(255) NOT NULL,
url char(255) NOT NULL,
sourceUrl char(255) NOT NULL,
categoryID int(6) unsigned NOT NULL,
producerID int(6) DEFAULT NULL,
authorID int(6) DEFAULT NULL,
shopID int(6) NOT NULL,
XMLUrlOrder tinyint(2) NOT NULL,
price float(12,2) NOT NULL,
oldPrice float(12,2) NOT NULL,
oldPricePt smallint(3) NOT NULL,
description text,
descriptionHash char(32) NOT NULL,
descriptionForGroup text NOT NULL,
imExist tinyint(1) NOT NULL DEFAULT '0',
imagesForGroup tinyint(1) NOT NULL DEFAULT '0',
videoHighlight text NOT NULL,
videoSiteUrl char(255) NOT NULL,
videoChannelUrl char(255) NOT NULL,
plusesMinuses text NOT NULL,
toIndex tinyint(1) NOT NULL DEFAULT '0',
isRST tinyint(1) NOT NULL DEFAULT '0',
isReplica tinyint(1) NOT NULL DEFAULT '0',
status tinyint(1) NOT NULL DEFAULT '0',
comment char(255) NOT NULL,
daysLeft tinyint(2) NOT NULL,
PPC float(5,2) DEFAULT '0.00',
payPrior tinyint(1) NOT NULL DEFAULT '4',
PRIMARY KEY (id),
UNIQUE KEY ixGroupNameCategoryIDShopIDProducerID (shopID,localID),
KEY ixGroupNameCategoryID (groupName,categoryID),
KEY ixStatusShopID (status,shopID),
KEY ixCategoryID (categoryID),
KEY authorID (authorID),
KEY ixDateModified (dateModified,imExist),
KEY daysLeft (daysLeft),
KEY sourceUrl (sourceUrl),
KEY ixCategoryIDid (categoryID,id)
) ENGINE=MyISAM AUTO_INCREMENT=4218880 DEFAULT CHARSET=utf8
The EXPLAIN shows that it needs to scan about 82K rows of gx. There are apparently about that many rows with categoryID = 892, correct? Most of the rest is straightforward JOINs.
Don't use MyISAM, use InnoDB.
INT(6) -- the (6) means nothing. Perhaps you meant MEDIUMINT UNSIGNED? INT is 4 bytes; MEDIUMINT is 3.
Are you 'always' searching by category? If so, make use of InnoDB's "clustered" PK by switching to PRIMARY KEY (categoryID, id), INDEX(id) and chuck the two existing indexes starting with categoryID.
Don't use CHAR unless the column is truly fixed length; use VARCHAR.
Don't use FLOAT(m,n), it can lead to subtle rounding errors. For Money, use DECIMAL(m,n); for Scientific values, use FLOAT.
ORs defeat optimizations. See if you can redesign the schema to avoid some of them.
What is LIKE '%^598^%'? Do you have a list of numbers in that column?
After switching to InnoDB, decrease key_buffer_size to only 30M and increase innodb_buffer_pool_size to 70% of available RAM`.

mariadb losing primary key on update

One of my tables is losing the primary key every time I make an update on that particular table.
Describe zizi_card_household
gives me this result after the update.
+-------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | | NULL | auto_increment |
| householdnumber | varchar(45) | NO | | | |
| cardnumber | varchar(45) | YES | MUL | NULL | |
| startdate | datetime | YES | | NULL | |
| enddate | datetime | YES | | NULL | |
| assignedby | int(11) | YES | | NULL | |
| assigneddate | datetime | YES | | NULL | |
Where it should normally be
+-------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| householdnumber | varchar(45) | NO | MUL | | |
| cardnumber | varchar(45) | YES | MUL | NULL | |
| startdate | datetime | YES | | NULL | |
| enddate | datetime | YES | | NULL | |
| assignedby | int(11) | YES | | NULL | |
SHOW CREATE TABLE
will give this statement which shows a primary key was defined. It's possible that entry is updated whenever the table is altered so it shows the table as it was last altered or created.
CREATE TABLE `zizi_card_household` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`householdnumber` varchar(45) NOT NULL DEFAULT '',
`cardnumber` varchar(45) DEFAULT NULL,
`startdate` datetime DEFAULT NULL,
`enddate` datetime DEFAULT NULL,
`assignedby` int(11) DEFAULT NULL,
`assigneddate` datetime DEFAULT NULL,
`deassignedby` int(11) DEFAULT NULL,
.
.
.
.
`modifiedby` int(11) DEFAULT NULL,
`reprintstatus` tinyint(4) DEFAULT NULL,
`printeddate` datetime DEFAULT NULL,
`printedby` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `idx_householdnumber` (`householdnumber`),
KEY `idx_cardnumber` (`cardnumber`),
KEY `idx_deassignedby` (`deassignedby`),
.
.
.
.
KEY `idx_reprintstatus` (`reprintstatus`)
) ENGINE=InnoDB AUTO_INCREMENT=860137 DEFAULT CHARSET=latin1
Attempting to add a primary key
ALTER TABLE `labour`.`zizi_card_household`
CHANGE COLUMN `id` `id` INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY
at this point gives me
ERROR 1068 (42000): Multiple primary key defined
Repair table
repair table zizi_card_household;
Gives me this but it does solve the problem. The primary key returns but will get ruined the next time the table is updated.
+----------------------------+--------+----------+---------------------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+----------------------------+--------+----------+---------------------------------------------------------+
| labour.zizi_card_household | repair | note | The storage engine for the table doesn't support repair |
+----------------------------+--------+----------+---------------------------------------------------------+
The table and other tables have a trigger that updates a summary table. It deletes related rows and reconstructs them. It does something like
FOR EACH ROW
begin
delete from zizi_card_summary where householdnumber=new.householdnumber;
insert into zizi_card_summary(
cardnumber,cardhouseholdid,householdnumber,startdate,enddate,assignedby
.
.
.
)
select ch.cardnumber,ch.id,b.householdnumber,
ch.startdate,ch.enddate,ch.assignedby,
.
.
.
.
where b.householdnumber=new.householdnumber and b.beneficiary_type=0;
end
All other tables work just fine. Thanks. Been playing around with this for a week. The application is a Zend framework 1.12.
The db version is 10.1.11-MariaDB-log. The operating system is Linux 2.6.32-573.8.1.el6.x86_64 x86_64.
I can confirm the bug and the workaround as described by Timothy : removing the trigger keeps the Primary key after a record update.
Bug resolution progress is on https://jira.mariadb.org/browse/MDEV-9558

How can I optimize the searching of two tables?

I have two tables:
1st:
CREATE TABLE IF NOT EXISTS `tags` (
`i` int(10) NOT NULL AUTO_INCREMENT,
`id` int(10) NOT NULL,
`k` varchar(32) COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`i`),
KEY `k` (`k`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin AUTO_INCREMENT=1505426 ;
and 2nd:
CREATE TABLE IF NOT EXISTS `w` (
`id` int(7) NOT NULL,
`title` varchar(255) COLLATE utf8_bin NOT NULL,
`k1` varchar(24) COLLATE utf8_bin NOT NULL,
`k2` varchar(24) COLLATE utf8_bin NOT NULL,
`k3` varchar(24) COLLATE utf8_bin NOT NULL,
`k4` varchar(24) COLLATE utf8_bin NOT NULL,
`k5` varchar(24) COLLATE utf8_bin NOT NULL,
`k6` varchar(24) COLLATE utf8_bin NOT NULL,
`k7` varchar(24) COLLATE utf8_bin NOT NULL,
`k8` varchar(24) COLLATE utf8_bin NOT NULL,
`w` int(5) NOT NULL,
`h` int(5) NOT NULL,
`s` varchar(32) COLLATE utf8_bin NOT NULL,
`r` varchar(11) COLLATE utf8_bin NOT NULL,
`v` int(7) NOT NULL,
`c` varchar(32) COLLATE utf8_bin NOT NULL,
`c1` varchar(6) COLLATE utf8_bin NOT NULL,
`c2` varchar(6) COLLATE utf8_bin NOT NULL,
`c3` varchar(6) COLLATE utf8_bin NOT NULL,
`c4` varchar(6) COLLATE utf8_bin NOT NULL,
`c5` varchar(6) COLLATE utf8_bin NOT NULL,
`c6` varchar(6) COLLATE utf8_bin NOT NULL,
`m` varchar(4) COLLATE utf8_bin NOT NULL,
`t` int(10) NOT NULL,
`i` int(6) NOT NULL,
`o` int(6) NOT NULL,
`f` varchar(255) COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`id`),
KEY `keywords` (`k1`,`k2`,`k3`,`k4`,`k5`,`k6`,`k7`,`k8`),
KEY `category` (`c`),
KEY `color1` (`c1`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin ROW_FORMAT=DYNAMIC;
I'm trying to search 2 words ($search1 + $search2) from table 'tags' and find id's at 'w' table.
query:
SELECT w.id,w.title,w.k1,w.k2,w.k3,w.k4,w.k5,w.k6,w.k7,w.w,w.h,w.s,w.c1,w.c2,w.c3,w.c4,w.c5,w.c6,w.m,w.f
FROM tags,tags as tags2,w
WHERE tags.k ='$search1' AND tags2.k = '$search2'
and tags.id = tags2.id and tags2.id = w.id
ORDER BY `w`.`id`desc limit 24
The problem is that EXPLAIN function shows "Using where; Using temporary; Using filesort"
Also I believe that there is another way to optimize this my type search system.
Actually all data is in 'w' table, I just have no idea how exactly to search it properly.
I would be grateful for any help.
Edit:a bit more information about this
table 'tags' 1505425 records InnoDB = 65,9 MiB
table 'w' 398900 rercords, InnoDB = 140,3 MiB
after this (updated) query:
select `w`.`id` AS `id`,`w`.`title` AS `title`,`w`.`k1` AS `k1`,`w`.`k2` AS `k2`,`w`.`k3` AS `k3`,`w`.`k4` AS `k4`,`w`.`k5` AS `k5`,`w`.`k6` AS `k6`,`w`.`k7` AS `k7`,`w`.`k8` AS `k8`,`w`.`w` AS `w`,`w`.`h` AS `h`,`w`.`s` AS `s`,`w`.`c1` AS `c1`,`w`.`c2` AS `c2`,`w`.`c3` AS `c3`,`w`.`c4` AS `c4`,`w`.`c5` AS `c5`,`w`.`c6` AS `c6`,`w`.`m` AS `m`,`w`.`f` AS `f` from `tags` join `tags` `tags2` join `w` where ((`tags`.`k` = '$search1') and (`tags2`.`id` = `tags`.`id`) and (`w`.`id` = `tags`.`id`) and (`tags2`.`k` = '$search2')) order by `tags`.`i` desc limit 0,24;
mysql still shows:
+----+-------------+------------+--------+---------------+---------+---------+-------------+------+----------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+--------+---------------+---------+---------+-------------+------+----------+-----------------------------+
| 1 | SIMPLE | tags | ref | k,id | k | 98 | const | 902 | 100.00 | Using where; Using filesort |
| 1 | SIMPLE | tags2 | ref | k,id | id | 4 | db.tags.id | 4 | 100.00 | Using where |
| 1 | SIMPLE | w | eq_ref | PRIMARY | PRIMARY | 4 | db.tags2.id | 1 | 100.00 | Using where |
+----+-------------+------------+--------+---------------+---------+---------+-------------+------+----------+-----------------------------+
3 rows in set, 1 warning (0.00 sec)
One warning, also still everything remains the same, is there any solution for better performance? At the moment it works fine, but later everything will goes slower and slower I think.
You're mixing storage engines; mySQL has a hard time joining tables across storage engines.
Before:
mysql> explain SELECT w.id,w.title,w.k1,w.k2,w.k3,w.k4,w.k5,w.k6,w.k7,w.w,w.h,w.s,w.c1,w.c2,w.c3,w.c4,w.c5,w.c6,w.m,w.f FROM tags,tags as tags2,w WHERE tags.k ='$search1' AND tags2.k = 'search2' and tags.id = tags2.id and tags2.id = w.id ORDER BY `w`.`id`desc limit 24;
+----+-------------+-------+--------+---------------+---------+---------+-------------+------+----------------------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+---------+---------+-------------+------+----------------------------------------------+
| 1 | SIMPLE | tags | ref | k | k | 98 | const | 1 | Using where; Using temporary; Using filesort |
| 1 | SIMPLE | tags2 | ref | k | k | 98 | const | 1 | Using where |
| 1 | SIMPLE | w | eq_ref | PRIMARY | PRIMARY | 4 | tmp.tags.id | 1 | |
+----+-------------+-------+--------+---------------+---------+---------+-------------+------+----------------------------------------------+
Change w to MyISAM:
mysql> alter table w engine MyISAM;
Query OK, 1 row affected (0.01 sec)
Records: 1 Duplicates: 0 Warnings: 0
After:
mysql> explain SELECT w.id,w.title,w.k1,w.k2,w.k3,w.k4,w.k5,w.k6,w.k7,w.w,w.h,w.s,w.c1,w.c2,w.c3,w.c4,w.c5,w.c6,w.m,w.f FROM tags,tags as tags2,w WHERE tags.k ='$search1' AND tags2.k = 'search2' and tags.id = tags2.id and tags2.id = w.id ORDER BY `w`.`id`desc limit 24;
+----+-------------+-------+--------+---------------+------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+------+---------+-------+------+-------------+
| 1 | SIMPLE | w | system | PRIMARY | NULL | NULL | NULL | 1 | |
| 1 | SIMPLE | tags | ref | k | k | 98 | const | 1 | Using where |
| 1 | SIMPLE | tags2 | ref | k | k | 98 | const | 1 | Using where |
+----+-------------+-------+--------+---------------+------+---------+-------+------+-------------+
3 rows in set (0.00 sec)
Vagiz, your updated query seems OK:
select `w`.`id` AS `id`,`w`.`title` AS `title`,`w`.`k1` AS `k1`,`w`.`k2` AS `k2`,
`w`.`k3` AS `k3`,`w`.`k4` AS `k4`,`w`.`k5` AS `k5`,`w`.`k6` AS `k6`,`w`.
`k7` AS `k7`,`w`.`k8` AS `k8`,`w`.`w` AS `w`,`w`.`h` AS `h`,`w`.`s` AS `s`,
`w`.`c1` AS `c1`,`w`.`c2` AS `c2`,`w`.`c3` AS `c3`,`w`.`c4` AS `c4`,
`w`.`c5` AS `c5`,`w`.`c6` AS `c6`,`w`.`m` AS `m`,`w`.`f` AS `f`
from `tags` join `tags` `tags2` join `w` where ((`tags`.`k` = '$search1')
and (`tags2`.`id` = `tags`.`id`) and (`w`.`id` = `tags`.`id`)
and (`tags2`.`k` = '$search2')) order by `tags`.`i` desc limit 0,24;
If it's producing following explain result:
+----+-------------+------------+--------+---------------+---------+---------+-------------+------+----------+-----------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+--------+---------------+---------+---------+-------------+------+----------+-----------------------------+
| 1 | SIMPLE | tags | ref | k,id | k | 98 | const | 902 | 100.00 | Using where; Using filesort |
| 1 | SIMPLE | tags2 | ref | k,id | id | 4 | db.tags.id | 4 | 100.00 | Using where |
| 1 | SIMPLE | w | eq_ref | PRIMARY | PRIMARY | 4 | db.tags2.id | 1 | 100.00 | Using where |
+----+-------------+------------+--------+---------------+---------+---------+-------------+------+----------+-----------------------------+
3 rows in set, 1 warning (0.00 sec)
Don't worry about filesort and performance. From the Mysql Performance Blog:
The truth is, filesort is badly named. Anytime a sort can’t be
performed from an index, it’s a filesort. It has nothing to do with
files. Filesort should be called “sort.” It is quicksort at heart.