mysql speed up update statement for copying values between tables - mysql

I have 2 tables where I am trying to copy values from migrate table to regular table. It seems to take awhile. (10 minutes for 43k records)
UPDATE phppos_sales_items,phppos_sales_items_migrate
SET
phppos_sales_items.subtotal = phppos_sales_items_migrate.subtotal,
phppos_sales_items.tax = phppos_sales_items_migrate.tax,
phppos_sales_items.total = phppos_sales_items_migrate.total,
phppos_sales_items.profit = phppos_sales_items_migrate.profit
WHERE
phppos_sales_items.sale_id = phppos_sales_items_migrate.sale_id and
phppos_sales_items.line = phppos_sales_items_migrate.line and
phppos_sales_items.item_id = phppos_sales_items_migrate.item_id;
Migrate Table
CREATE TABLE `phppos_sales_items_migrate` (
`sale_id` int(10) NOT NULL DEFAULT '0',
`item_id` int(10) NOT NULL DEFAULT '0',
`line` int(11) NOT NULL DEFAULT '0',
`subtotal` decimal(23,10) DEFAULT NULL,
`total` decimal(23,10) DEFAULT NULL,
`tax` decimal(23,10) DEFAULT NULL,
`profit` decimal(23,10) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
sales_items
CREATE TABLE `phppos_sales_items` (
`sale_id` int(10) NOT NULL DEFAULT '0',
`item_id` int(10) NOT NULL DEFAULT '0',
`rule_id` int(10) DEFAULT NULL,
`rule_discount` decimal(23,10) DEFAULT NULL,
`description` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`serialnumber` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`line` int(11) NOT NULL DEFAULT '0',
`quantity_purchased` decimal(23,10) NOT NULL DEFAULT '0.0000000000',
`item_cost_price` decimal(23,10) NOT NULL,
`item_unit_price` decimal(23,10) NOT NULL,
`regular_item_unit_price_at_time_of_sale` decimal(23,10) DEFAULT NULL,
`discount_percent` decimal(15,3) NOT NULL DEFAULT '0.000',
`commission` decimal(23,10) NOT NULL DEFAULT '0.0000000000',
`subtotal` decimal(23,10) NOT NULL DEFAULT '0.0000000000',
`tax` decimal(23,10) NOT NULL DEFAULT '0.0000000000',
`total` decimal(23,10) NOT NULL DEFAULT '0.0000000000',
`profit` decimal(23,10) NOT NULL DEFAULT '0.0000000000',
PRIMARY KEY (`sale_id`,`item_id`,`line`),
KEY `item_id` (`item_id`),
KEY `phppos_sales_items_ibfk_3` (`rule_id`),
CONSTRAINT `phppos_sales_items_ibfk_1` FOREIGN KEY (`item_id`) REFERENCES `phppos_items` (`item_id`),
CONSTRAINT `phppos_sales_items_ibfk_2` FOREIGN KEY (`sale_id`) REFERENCES `phppos_sales` (`sale_id`),
CONSTRAINT `phppos_sales_items_ibfk_3` FOREIGN KEY (`rule_id`) REFERENCES `phppos_price_rules` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Is there any way to speed this up?

decimal(23,10) is excessive, especially for integral values, such as quantity_purchased. Shrinking the datatypes will help speed some.
Use a "multi-table Update" with JOIN ... ON ... syntax.
Let's see the output from
EXPLAIN
SELECT * FROM phppos_sales_items AS i
JOIN phppos_sales_items_migrate AS m
ON i.sale_id = m.sale_id
and i.line = m.line
and i.item_id = m.item_id;
If you have a new enough version of MySQL, also provide EXPLAIN UPDATE ...
Any chance there are new rows in migrate? If so, won't you need INSERT ... ON DUPLICATE KEY UPDATE ...?
43K rows in which table? How many in the other table?

In my opinion, adding Index to the Target Table are worth to try.
Since two tables are comparing the three columns for merging it, you might better to adding index to the phppos_sales_items_migrate table for (sale_id,item_id,line) columns.
Good Luck
PS. FYI, adding indexes to the table also take costs for updating and inserting thus minimize to the comparing columns.

Related

MySQL use separate indices for JOIN and GROUP BY

I am trying to execute following query
SELECT
a.sessionID AS `sessionID`,
firstSeen, birthday, gender,
isAnonymous, LanguageCode
FROM transactions AS trx
INNER JOIN actions AS a ON a.sessionID = trx.SessionID
WHERE a.ActionType = 'PURCHASE'
GROUP BY trx.TransactionNumber
Explain provides the following output
1 SIMPLE trx ALL TransactionNumber,SessionID NULL NULL NULL 225036 Using temporary; Using filesort
1 SIMPLE a ref sessionID sessionID 98 infinitiExport.trx.SessionID 1 Using index
The problem is that I am trying to use one field for join and different field for GROUP BY.
How can I tell MySQL to use different indices for same table?
CREATE TABLE `transactions` (
`SessionID` varchar(32) NOT NULL DEFAULT '',
`date` datetime DEFAULT NULL,
`TransactionNumber` varchar(32) NOT NULL DEFAULT '',
`CustomerECommerceTrackID` int(11) DEFAULT NULL,
`SKU` varchar(45) DEFAULT NULL,
`AmountPaid` double DEFAULT NULL,
`Currency` varchar(10) DEFAULT NULL,
`Quantity` int(11) DEFAULT NULL,
`Name` tinytext NOT NULL,
`Category` varchar(45) NOT NULL DEFAULT '',
`customerInfoXML` text,
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`id`),
KEY `TransactionNumber` (`TransactionNumber`),
KEY `SessionID` (`SessionID`)
) ENGINE=InnoDB AUTO_INCREMENT=212007 DEFAULT CHARSET=utf8;
CREATE TABLE `actions` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`sessionActionDate` datetime DEFAULT NULL,
`actionURL` varchar(255) DEFAULT NULL,
`sessionID` varchar(32) NOT NULL DEFAULT '',
`ActionType` varchar(64) DEFAULT NULL,
`CustomerID` int(11) DEFAULT NULL,
`IPAddressID` int(11) DEFAULT NULL,
`CustomerDeviceID` int(11) DEFAULT NULL,
`customerInfoXML` text,
PRIMARY KEY (`id`),
KEY `ActionType` (`ActionType`),
KEY `CustomerDeviceID` (`CustomerDeviceID`),
KEY `sessionID` (`sessionID`)
) ENGINE=InnoDB AUTO_INCREMENT=15042833 DEFAULT CHARSET=utf8;
Thanks
EDIT 1: My indexes were broken, I had to add (SessionID, TransactionNumber) index to transactions table, however now, when I try to include trx.customerInfoXML table mysql stops using index
EDIT 2 Another answer does not really solved my problem because it's not standard sql syntax and generally not a good idea to force indices.
For ORM users such syntax is a unattainable luxury.
EDIT 3 I updated my indices and it solved the problem, see EDIT 1

Foreign ID in SQL tables to link one table to another

This is my SQL code which links users to items based on tutorials:
CREATE TABLE IF NOT EXISTS `users` (
`user_id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`user_name` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
`user_password_hash` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`user_email` varchar(254) COLLATE utf8_unicode_ci NOT NULL,
`user_access_level` tinyint(3) unsigned NOT NULL DEFAULT '0',
`user_active` tinyint(1) NOT NULL DEFAULT '0',
`user_activation_hash` varchar(40) COLLATE utf8_unicode_ci DEFAULT NULL,
`user_password_reset_hash` char(40) COLLATE utf8_unicode_ci DEFAULT NULL,
`user_password_reset_timestamp` bigint(20) DEFAULT NULL,
`user_failed_logins` tinyint(1) NOT NULL DEFAULT '0',
`user_last_failed_login` int(10) DEFAULT NULL,
`user_registration_datetime` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`user_registration_ip` varchar(39) COLLATE utf8_unicode_ci NOT NULL DEFAULT '0.0.0.0',
PRIMARY KEY (`user_id`),
UNIQUE KEY `user_name` (`user_name`),
UNIQUE KEY `user_email` (`user_email`)
) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE IF NOT EXISTS `item` (
`item_id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`item_title` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
`item_location` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
`item_description` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
`item_datetime` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`item_status` int(1) unsigned NOT NULL,
PRIMARY KEY (`item_id`)
) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
http://sqlfiddle.com/#!9/fd011
I'm getting a bit confused about how to link the items to the user. It seems like I need something called a foreign key on the items, a bit like this in my item table:
FOREIGN KEY (user_id) REFERENCES user(ID)
I can't seem to get it to compile and query successfully. Can anyone please show me the right way to associate the items with the user.
You need to add the user_id column to the items table along with the constraint:
CREATE TABLE IF NOT EXISTS `users` (
`user_id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`user_name` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
`user_password_hash` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`user_email` varchar(254) COLLATE utf8_unicode_ci NOT NULL,
`user_access_level` tinyint(3) unsigned NOT NULL DEFAULT '0',
`user_active` tinyint(1) NOT NULL DEFAULT '0',
`user_activation_hash` varchar(40) COLLATE utf8_unicode_ci DEFAULT NULL,
`user_password_reset_hash` char(40) COLLATE utf8_unicode_ci DEFAULT NULL,
`user_password_reset_timestamp` bigint(20) DEFAULT NULL,
`user_failed_logins` tinyint(1) NOT NULL DEFAULT '0',
`user_last_failed_login` int(10) DEFAULT NULL,
`user_registration_datetime` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`user_registration_ip` varchar(39) COLLATE utf8_unicode_ci NOT NULL DEFAULT '0.0.0.0',
PRIMARY KEY (`user_id`),
UNIQUE KEY `user_name` (`user_name`),
UNIQUE KEY `user_email` (`user_email`)
) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
CREATE TABLE IF NOT EXISTS `item` (
`item_id` int(11) unsigned NOT NULL AUTO_INCREMENT,
user_id int unsigned,
`item_title` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
`item_location` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
`item_description` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
`item_datetime` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`item_status` int(1) unsigned NOT NULL,
PRIMARY KEY (`item_id`),
FOREIGN KEY (user_id) REFERENCES users(user_id)
) ENGINE=MyISAM AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
The SQL Fiddle is here.
I should point out a few other things:
Pay attention to the database engine you are using. MyISAM doesn't actually enforce the relationship.
Having weird dates as the default value is probably less useful than just using NULL.
I'm not sure if there is a value to having explicit collations for every character definition, unless your database is going to be supporting a wide variety of collations.
Don't use single quotes for numeric constants. So, if a value is declared as a tinyint, set the default ot 0 not '0' (this doesn't affect performance in a CREATE TABLE statement; it is just misleading).
#GordonLindoff's solution is one method, but that assumes that each item belongs to exactly one user, and an item cannot be referenced by multiple users. If you have a many-to-many relationship, where a user can have multiple items and an item can be referenced by multiple users, then you need a third table that links them together:
CREATE TABLE IF NOT EXISTS `user_item` (
`user_id` int(11) unsigned NOT NULL,
`item_id` int(11) unsigned NOT NULL,
PRIMARY KEY (`user_item`,`item_id`),
FOREIGN KEY (user_id) REFERENCES users(user_id),
FOREIGN KEY (item_id) REFERENCES items(item_id)
)ENGINE=Innodb DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
The Foreign Key constraints enforce that for every row in user_item that the user_id exists in users, and the item_id exists in items. And as was mentioned in a previous comment, that you will need Innodb to have the foreign key constraints enforced.

MySQL Long First Query Time

I have a website that has two main queries to the database which are pretty slow the first time they are run, after a bit of testing it appears to be an issue with MySQL. If I run the query directly in Sequal Pro when I run a query the first time it can take up to 4 seconds to run but running the same query again takes ~60ms, the query time is about the same locally as on our server, which make me think its not a server issue.
Not totally sure that increasing the buffer pool size will help too much as the number of potential query combinations is probably around 800K.
The tables in the database are innodb, both queries access the same table that has 52K records, most of the information I need has been grouped together into a 'searchfield' field which is indexed.
Only fields used in queries or are primary/foreign keys are being indexed.
I have tried changing the inner joins to a select statement in the "where" of the main query but this doesn't make the query any faster.
The queries are
Query 1
SELECT
`item_attribute`.`attribute_id` AS `attribute_id`,
`attribute_value_id` AS `attribute_value_id`,
`collection_attribute`.`title` AS `ca_title`,
`collection_attribute`.`type` AS `ca_type`,
`collection_attribute`.`is_collapsible` AS `ca_is_collapsible`,
`collection_attribute`.`orderindex` AS `ca_orderindex`,
`collection_attribute`.`multi_select` AS `ca_multi_select`,
`item_attribute`.`item_id` AS `item_id`,
`product`.`id` AS `product_id`
FROM `item_attribute`
INNER JOIN `item` ON item.id = item_attribute.item_id
INNER JOIN `product` ON product.id = item.product_id
INNER JOIN `collection_attribute` ON item_attribute.attribute_id = collection_attribute.attribute_id
INNER JOIN `attribute_value` ON attribute_value.id = item_attribute.attribute_value_id
WHERE ((`product`.`searchfilter` LIKE '%c:35∆%') AND (`collection_attribute`.`collection_id`='35')) AND (`attribute_value`.`active`=1)
GROUP BY `attribute_value_id`
Query 2
SELECT DISTINCT `item_attribute`.`attribute_id` AS `attribute_id`,
GROUP_CONCAT(item_attribute.attribute_value_id SEPARATOR \"-\") AS `attribute_value`,
GROUP_CONCAT(attribute_value.title SEPARATOR \" - \") AS `title`
FROM `item_attribute`
LEFT JOIN `item` ON item.id = item_attribute.item_id
LEFT JOIN `attribute` ON attribute.id = item_attribute.attribute_id
LEFT JOIN `attribute_value` ON attribute_value.id = attribute_value_id
WHERE (`item`.`product_id`='894') AND (`attribute`.`is_option`=1)
GROUP BY `attribute_id`, `item_id`
Table Structure
CREATE TABLE `product` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(255) NOT NULL,
`slug` varchar(255) NOT NULL,
`sku` varchar(20) NOT NULL,
`active` tinyint(1) DEFAULT '0',
`orderindex` int(2) DEFAULT '-1',
`search` varchar(255) DEFAULT NULL,
`searchfilter` varchar(255) DEFAULT NULL,
`created_at` int(11) DEFAULT NULL,
`updated_at` int(11) DEFAULT NULL,
`protected` tinyint(1) DEFAULT '0',
`description` varchar(512) DEFAULT NULL,
`type_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `product_ibfk_1` (`type_id`),
KEY `searchfilter` (`searchfilter`),
CONSTRAINT `product_ibfk_1` FOREIGN KEY (`type_id`) REFERENCES `attribute_value` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=1882 DEFAULT CHARSET=utf8;
--
CREATE TABLE `collection_attribute` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`collection_id` int(11) DEFAULT NULL,
`attribute_id` int(11) DEFAULT NULL,
`title` varchar(512) NOT NULL,
`slug` varchar(512) NOT NULL,
`created_at` int(11) DEFAULT NULL,
`updated_at` int(11) DEFAULT NULL,
`active` tinyint(1) DEFAULT '0',
`orderindex` int(2) DEFAULT '-1',
`search` varchar(255) DEFAULT NULL,
`searchfilter` varchar(255) DEFAULT NULL,
`protected` tinyint(1) DEFAULT '0',
`is_collapsible` tinyint(1) DEFAULT '0',
`type` enum('icon','checkbox','checkboxIcon','image') DEFAULT NULL,
`multi_select` tinyint(1) DEFAULT '0',
PRIMARY KEY (`id`),
KEY `collection_attribute_ibfk_1` (`collection_id`),
KEY `collection_attribute_ibfk_2` (`attribute_id`),
CONSTRAINT `collection_attribute_ibfk_1` FOREIGN KEY (`collection_id`) REFERENCES `collection` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `collection_attribute_ibfk_2` FOREIGN KEY (`attribute_id`) REFERENCES `attribute` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=140 DEFAULT CHARSET=utf8;
--
CREATE TABLE `item` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`slug` varchar(255) NOT NULL,
`title` varchar(255) NOT NULL,
`pattern_code` varchar(32) NOT NULL,
`tom_code` varchar(32) NOT NULL,
`navision_code` varchar(32) NOT NULL,
`description` varchar(255) DEFAULT NULL,
`active` tinyint(1) DEFAULT '0',
`orderindex` int(2) DEFAULT '-1',
`created_at` int(11) DEFAULT NULL,
`updated_at` int(11) DEFAULT NULL,
`search` varchar(1024) DEFAULT NULL,
`searchfilter` varchar(255) DEFAULT NULL,
`product_id` int(11) DEFAULT NULL,
`protected` tinyint(1) DEFAULT '0',
`pattern_series` varchar(255) DEFAULT NULL,
`pattern_id` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `item_ibfk_1` (`product_id`),
KEY `searchfilter` (`searchfilter`),
KEY `product_id` (`product_id`),
KEY `pattern_id` (`pattern_id`),
CONSTRAINT `item_ibfk_1` FOREIGN KEY (`product_id`) REFERENCES `product` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `item_ibfk_2` FOREIGN KEY (`pattern_id`) REFERENCES `pattern_series` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=50060 DEFAULT CHARSET=utf8;
--
CREATE TABLE `item_attribute` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`item_id` int(11) NOT NULL,
`attribute_id` int(11) NOT NULL,
`attribute_value_id` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `attribute_id` (`attribute_id`),
KEY `attribute_value_id` (`attribute_value_id`),
KEY `item_id` (`item_id`),
CONSTRAINT `item_attribute_ibfk_1` FOREIGN KEY (`attribute_id`) REFERENCES `attribute` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `item_attribute_ibfk_2` FOREIGN KEY (`attribute_value_id`) REFERENCES `attribute_value` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
CONSTRAINT `item_attribute_ibfk_3` FOREIGN KEY (`item_id`) REFERENCES `item` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=857111 DEFAULT CHARSET=utf8;
--
CREATE TABLE `attribute_value` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`slug` varchar(255) NOT NULL,
`code` varchar(255) DEFAULT NULL,
`title` varchar(255) NOT NULL,
`active` tinyint(1) DEFAULT '0',
`orderindex` int(2) DEFAULT '-1',
`created_at` int(11) DEFAULT NULL,
`updated_at` int(11) DEFAULT NULL,
`search` varchar(255) DEFAULT NULL,
`searchfilter` varchar(255) DEFAULT NULL,
`attribute_id` int(11) DEFAULT NULL,
`protected` tinyint(1) DEFAULT '0',
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3471 DEFAULT CHARSET=utf8;
--
CREATE TABLE `attribute` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(255) NOT NULL,
`slug` varchar(255) NOT NULL,
`code` varchar(255) DEFAULT NULL,
`is_option` tinyint(1) DEFAULT '0',
`searches` tinyint(1) DEFAULT '0',
`option_type` enum('dropdown','switch','fingersizes') DEFAULT NULL,
`option_label` varchar(32) DEFAULT NULL,
`active` tinyint(1) DEFAULT '0',
`orderindex` int(2) DEFAULT '-1',
`created_at` int(11) DEFAULT NULL,
`updated_at` int(11) DEFAULT NULL,
`search` varchar(255) DEFAULT NULL,
`searchfilter` varchar(255) DEFAULT NULL,
`protected` tinyint(1) DEFAULT '0',
`option_requires` int(11) DEFAULT NULL,
`option_depends` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `is_option` (`is_option`)
) ENGINE=InnoDB AUTO_INCREMENT=47 DEFAULT CHARSET=utf8;
Any suggestions on how to improve the initial query time would be great
Thanks in advance :)
-- EDIT --
EXPLAIN SELECT query 1
EXPLAIN SELECT query 2
The first time you perform a query after starting the server, nothing is cache, so the query needs to fetch stuff from disk. All subsequent queries that access the same parts of the same tables will be much faster because of caching. This is "normal".
If you have the "Query cache" enable (it is probably enabled by default), then the second time you run exactly the same query, it will instantly find the result from the Query cache. By "exactly" I mean that not so much as a blank space has changed. Nearly all "production" servers are better off turning off the Query cache.
innodb_buffer_pool_size should be about 70% of available RAM. Changing the value won't affect a SELECT against a cold cache, but might help/hurt subsequent runs. This does not seem to be relevant in your case, since the second run was quite fast.
Please provide EXPLAIN SELECT ... so we can see how the optimizer decided to execute them.
LIKE '%c:35∆%' -- cannot use an index because of the leading wild card.
What is item_ids?
item_attribute is an EAV schema pattern. It sucks. Both the queries are ugly, and scalability hurts. It may help some to get rid of the id and make a compound PRIMARY KEY from a suitable combination of the other fields. The hope is to use the PRIMARY KEY which is clustered with the data instead of having to bounce from the secondary key. More discussion of EAV.
Assuming this has low cardinality, the index will probably never be used:
KEY is_option (is_option)

Perf of select mysql query is really bad

I'm not sure why this query is taking 4 minutes to complete:
SELECT
su.sid,u.uid,u.display_name,u.locale
FROM user u
LEFT JOIN subscription_user su ON su.uid = u.uid
ORDER BY u.display_name DESC
LIMIT 0,25;
Well, I know it's due to the order, remove it and it's very fast. If I change to using INNER JOIN instead it's fast but the issue is not all users may be in the subscription_user table.
CREATE TABLE `user` (
`uid` int(11) NOT NULL AUTO_INCREMENT,
`password` varchar(100) DEFAULT NULL,
`user_type` varchar(10) NOT NULL DEFAULT 'user',
`display_name` varchar(50) NOT NULL,
`email` varchar(100) NOT NULL,
`locale` varchar(8) DEFAULT 'en',
`last_login` datetime DEFAULT NULL,
`auth_type` varchar(10) DEFAULT NULL,
`auth_data` varchar(500) DEFAULT NULL,
`inactive` tinyint(4) NOT NULL DEFAULT '0',
`receive_email` tinyint(4) NOT NULL DEFAULT '1',
`stateid` int(10) DEFAULT NULL,
`owner_group_id` int(11) DEFAULT NULL,
`signature` varchar(500) DEFAULT NULL,
`raw_signature` varchar(500) DEFAULT NULL,
`round_robin` smallint(5) unsigned NOT NULL DEFAULT '0',
PRIMARY KEY (`uid`),
UNIQUE KEY `email` (`email`),
KEY `stateid` (`stateid`) USING BTREE,
KEY `user_type` (`user_type`) USING BTREE,
KEY `name` (`display_name`)
) ENGINE=InnoDB AUTO_INCREMENT=28343 DEFAULT CHARSET=latin1;
CREATE TABLE `subscription_user` (
`sid` varchar(50) NOT NULL,
`uid` int(11) NOT NULL,
`deleted` tinyint(4) NOT NULL DEFAULT '0',
`forum_user` varchar(50) NOT NULL,
PRIMARY KEY (`sid`,`uid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
When you have an SQL query, the index can only really help you if the first column in the index is part of the query.
Your query joins su.uid = u.uid and the optimizer will not be able to use that to reference the first column in the subscription primary key index.
You should either reverse the order of the columns in the primary key, or alternatively, you should add a foreign key index, or an independent index on the uid

Speeding up multi-table joins with MySQL

I have 3 tables in which I'm trying to preform joins on, and inserting the resulting data into another table. The query is taking anywhere between 15-30 mins depending on the dataset. The tables I'm selecting from and joining on are at least 25k records each but will quickly grow to be 500k+.
I tried adding indexes on the fields but still isn't helping that much. Are there any other things I can try or are joins on this scale just going to take this long?
Here is the query I'm trying to perform:
INSERT INTO audience.topitem
(runs_id, total_training_count, item, standard_index_value, significance, seed_count, nonseed_count, prod, model_type, level_1, level_2, level_3, level_4, level_5)
SELECT 5, seed_count + nonseed_count AS total_training_count,
ii.item, standard_index_value, NULL, seed_count, nonseed_count,
standard_index_value * seed_count AS prod, 'site', topic_L1, topic_L2, topic_L3, topic_L4, topic_L5
FROM audience.item_indexes ii
LEFT JOIN audience.usercounts uc ON ii.item = uc.item AND ii.runs_id = uc.runs_id
LEFT JOIN categorization.categorization at on ii.item = at.url
WHERE ii.runs_id = 5
Table: audience.item_indexes
CREATE TABLE `item_indexes` (
`item` varchar(1024) DEFAULT NULL,
`standard_index_value` float DEFAULT NULL,
`runs_id` int(11) DEFAULT NULL,
`model_type` enum('site','term','combo') DEFAULT NULL,
KEY `item_idx` (`item`(333))
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
Table: audience.usercounts
CREATE TABLE `usercounts` (
`item` varchar(1024) DEFAULT NULL,
`seed_count` int(11) DEFAULT NULL,
`nonseed_count` int(11) DEFAULT NULL,
`significance` float(19,6) DEFAULT NULL,
`runs_id` int(11) DEFAULT NULL,
`model_type` enum('site','term','combo') DEFAULT NULL,
KEY `item_idx` (`item`(333))
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
Table: audience.topitem
CREATE TABLE `topitem` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`total_training_count` int(11) DEFAULT NULL,
`item` varchar(1024) DEFAULT NULL,
`standard_index_value` float(19,6) DEFAULT NULL,
`significance` float(19,6) DEFAULT NULL,
`seed_count` int(11) DEFAULT NULL,
`nonseed_count` int(11) DEFAULT NULL,
`prod` float(19,6) DEFAULT NULL,
`cat_type` varchar(32) DEFAULT NULL,
`cat_level` int(11) DEFAULT NULL,
`conf` decimal(19,9) DEFAULT NULL,
`level_1` varchar(64) DEFAULT NULL,
`level_2` varchar(64) DEFAULT NULL,
`level_3` varchar(64) DEFAULT NULL,
`level_4` varchar(64) DEFAULT NULL,
`level_5` varchar(64) DEFAULT NULL,
`runs_id` int(11) DEFAULT NULL,
`model_type` enum('site','term','combo') DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=825 DEFAULT CHARSET=utf8;
Table: categorization.categorization
CREATE TABLE `AT_categorization` (
`url` varchar(760) NOT NULL ,
`language` varchar(10) DEFAULT NULL,
`category` text,
`entity` text,
`source` varchar(255) DEFAULT NULL,
`topic_L1` varchar(45) NOT NULL DEFAULT '',
`topic_L2` varchar(45) NOT NULL DEFAULT '',
`topic_L3` varchar(45) NOT NULL DEFAULT '',
`topic_L4` varchar(45) NOT NULL DEFAULT '',
`topic_L5` varchar(45) NOT NULL DEFAULT '',
`last_refreshed` datetime DEFAULT NULL,
PRIMARY KEY (`url`,`topic_L1`,`topic_L2`,`topic_L3`,`topic_L4`,`topic_L5`),
UNIQUE KEY `inx_url` (`url`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
If you add the following indexes, your query will run faster:
CREATE INDEX runs_idx ON audience.item_indexes (runs_id);
ALTER TABLE audience.usercounts
DROP INDEX item_idx,
ADD INDEX item_idx (runs_id, item(333));
Also, item_indexes is utf8, but AT_categorization is latin1, which keeps any indexes from being used. To address this issue, change AT_categorization to utf8:
ALTER TABLE AT_categorization CHARSET=utf8;
Lastly, for the AT_categorization table, the two indexes
PRIMARY KEY (`url`,`topic_L1`,`topic_L2`,`topic_L3`,`topic_L4`,`topic_L5`),
UNIQUE KEY `inx_url` (`url`)
are redundant. So you could DROP these, and simply have the url field be the primary key:
ALTER TABLE AT_categorization
DROP PRIMARY KEY,
DROP KEY `inx_url`,
ADD PRIMARY KEY (url);