MySQL IN Statement Hang - mysql

What I'm attempting to accomplish is return a SELECT statement of ALL duplicates in a table within given filters.
I'm attempting to run the following query but every time I run it my server locks up and it never completes the query. I have no idea what is causing this and some help would be greatly appreciated on either how to fix this or to accomplish my goal in another manner.
EDIT: I've added the EXPLAIN data as requested!
EDIT 2: I've added the CREATE statements as requested!
SELECT *
FROM red_flags
WHERE customer_number IN (SELECT customer_number
FROM red_flags
GROUP BY customer_number
HAVING COUNT(customer_number) > 1);
Execution plan:
1 PRIMARY leads ALL 80708 Using where
1 PRIMARY customers eq_ref PRIMARY PRIMARY 9 apcard_main.leads.customer_number 1
2 DEPENDENT SUBQUERY leads ALL 80708 Using where; Using temporary; Using filesort
2 DEPENDENT SUBQUERY customers eq_ref PRIMARY PRIMARY 9 apcard_main.leads.customer_number 1 Using index
DDL:
Table customers:
CREATE TABLE `customers` (
`customer_number` varchar(7) NOT NULL,
`dealer_id` varchar(32) DEFAULT NULL,
`first_name` varchar(64) DEFAULT NULL,
`middle_name` varchar(64) DEFAULT NULL,
`last_name` varchar(64) DEFAULT NULL,
`address_one` varchar(128) DEFAULT NULL,
`address_two` varchar(128) DEFAULT NULL,
`city` varchar(128) DEFAULT NULL,
`state` varchar(32) DEFAULT NULL,
`zip` char(5) DEFAULT NULL,
`fico` char(3) DEFAULT NULL,
`phone` varchar(13) DEFAULT NULL,
`mail_type` varchar(32) DEFAULT NULL,
`mail_date` date DEFAULT NULL,
`store` varchar(32) DEFAULT NULL,
PRIMARY KEY (`customer_number`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1$$
Table leads:
CREATE TABLE `leads` (
`lead_id` int(11) NOT NULL AUTO_INCREMENT,
`dealer_id` varchar(32) DEFAULT NULL,
`customer_number` varchar(7) DEFAULT NULL,
`date` date DEFAULT NULL,
`time` time DEFAULT NULL,
`source` varchar(32) DEFAULT NULL,
`source_type` varchar(32) DEFAULT NULL,
`home_phone` varchar(13) DEFAULT NULL,
`email` varchar(64) DEFAULT NULL,
`mail_type` varchar(8) DEFAULT NULL,
`store` varchar(16) DEFAULT NULL,
`first_name` varchar(64) DEFAULT NULL,
`last_name` varchar(64) DEFAULT NULL,
`middle_name` varchar(32) DEFAULT NULL,
`city` varchar(128) DEFAULT NULL,
`state` varchar(32) DEFAULT NULL,
`zip` char(5) DEFAULT NULL,
`fico` char(3) DEFAULT NULL,
`mail_date` date DEFAULT NULL,
`work_phone` varchar(13) DEFAULT NULL,
`cell_phone` varchar(13) DEFAULT NULL,
`address_one` varchar(128) DEFAULT NULL,
`address_two` varchar(128) DEFAULT NULL,
`comment` varchar(255) DEFAULT NULL,
`caller_id` varchar(128) DEFAULT NULL,
PRIMARY KEY (`lead_id`)
) ENGINE=MyISAM AUTO_INCREMENT=125587 DEFAULT CHARSET=latin1$$
View red_flags:
CREATE
ALGORITHM=UNDEFINED
DEFINER=`apcard`#`97.83.30.118`
SQL SECURITY DEFINER
VIEW `red_flags` AS
select
`leads`.`dealer_id` AS `dealer_id`,
`customers`.`phone` AS `phone`,
`leads`.`date` AS `date`,
`leads`.`time` AS `time`,
`leads`.`source` AS `source`,
`leads`.`customer_number` AS `customer_number`,
`leads`.`caller_id` AS `caller_id`,
`leads`.`mail_type` AS `mail_type`,
`leads`.`store` AS `store`,
`leads`.`last_name` AS `last_name`,
`leads`.`first_name` AS `first_name`,
`leads`.`city` AS `city`,
`leads`.`state` AS `state`,
`leads`.`zip` AS `zip`,
`leads`.`fico` AS `fico`,
`leads`.`mail_date` AS `mail_date`,
`leads`.`home_phone` AS `home_phone`,
`leads`.`email` AS `email`
from (`customers` join `leads`)
where ((`customers`.`customer_number` = `leads`.`customer_number`)
and (`leads`.`date` >= (now() - interval 30 day)))$$

Have you tried joining rather than using a subquery?
SELECT rf.*
FROM red_flags rf
inner join red_flags rf2
on (rf2.customer_number = rf.customer_number)
group by rf2.customer_number
having count(rf2.customer_number) >1

I figured it out, I had a poorly formed date filter that was causing it to break when querying with an IN for some reason.
leads.date >= now() - INTERVAL 30 DAY

Related

How to use MYSQL JOIN structure? and how to optimize query time

I have a few problems with MYSQL database that I can't solve.
My query below is taking too much time and making the system hang. I'm trying to use the "JOIN" construct to develop this. But this time my aggregation, which I'm trying to do with "SUM", reduces the query to one line. Is it ok to do this job with "JOIN"? or how should i improve this query.
This database works with a total of 22 client devices in ASP .NET application. As I mentioned above, in cases where the query time is long, when the client devices send a query to the database at the same time, the client device freezes. What I don't understand is why a query in the browser app is making all devices wait. Isn't each query processed as a separate "Thread" in MYSQL? So if a query has a return time of 10 seconds, will all clients wait 10 seconds for the query to be answered in the browser?
SELECT *,
(SELECT MachModel FROM machine WHERE MachCode=workorder.MachCode) AS MachModel,
(SELECT RawMaterialDescription FROM rawmaterials WHERE RawMaterialCode=workorder.ProductRawMaterial) AS RawMaterialDescr,
(SELECT RawMaterialColor FROM rawmaterials WHERE RawMaterialCode=workorder.ProductRawMaterial) AS RawMaterialColor,
(SELECT StaffName FROM staff WHERE AccountName=workorder.AssignStaff) AS AssignStaffName,
(SELECT StaffCode FROM staff WHERE AccountName=workorder.AssignStaff) AS AssignStaffCode,
(SELECT MachStatus FROM machine WHERE MachCode=workorder.MachCode) AS MachStatus,
(SELECT SUM(xStopTime) FROM workorderb WHERE xWoNumber=workorder.WoNumber) AS WoTotalStopTime
FROM workorder
WHERE WoStatus=3
ORDER BY PlanProdStartDate DESC, WoSortNumber, WoNumber LIMIT 100
SELECT workorder.*,machine.MachModel,machine.MachStatus,rawmaterials.RawMaterialDescription,rawmaterials.RawMaterialColor,staff.StaffName,staff.StaffCode,SUM(workorderb.xStopTime)
FROM workorder
LEFT JOIN machine ON machine.MachCode=workorder.MachCode
LEFT JOIN rawmaterials ON rawmaterials.RawMaterialCode=workorder.ProductRawMaterial
LEFT JOIN staff ON staff.AccountName=workorder.AssignStaff
LEFT JOIN workorderb ON workorderb.xWoNumber=workorder.WoNumber
WHERE workorder.WoStatus=3
ORDER BY workorder.PlanProdStartDate DESC, workorder.WoSortNumber, workorder.WoNumber LIMIT 100
CREATE TABLE `workorder` (
`WoNumber` varchar(20) NOT NULL,
`MachCode` varchar(15) NOT NULL,
`PlannedMoldCode` varchar(10) NOT NULL,
`PartyNumber` smallint(6) NOT NULL,
`PlanProdCycleTime` smallint(6) NOT NULL,
`CalAverageCycleTime` float(15,1) unsigned NOT NULL,
`ProductRawMaterial` varchar(30) NOT NULL,
`PlanProdStartDate` date NOT NULL,
`PlanProdFinishDate` int(10) unsigned NOT NULL,
`WoStartDate` datetime DEFAULT NULL,
`WoFinishDate` datetime DEFAULT NULL,
`WoWorkTime` int(10) unsigned NOT NULL,
`WoSystemProductivity` smallint(6) unsigned NOT NULL,
`AssignStaff` varchar(50) DEFAULT '',
`WoStatus` smallint(6) NOT NULL,
`WoSortNumber` int(10) unsigned NOT NULL,
`CycleCount` int(11) unsigned NOT NULL,
`ControlDate` datetime NOT NULL ON UPDATE CURRENT_TIMESTAMP,
`WoProductionStatus` smallint(6) NOT NULL DEFAULT '0',
`Creator` varchar(50) NOT NULL,
`Changer` varchar(50) NOT NULL,
`CreateDate` datetime NOT NULL,
PRIMARY KEY (`WoNumber`) USING BTREE,
KEY `WoNumber` (`WoNumber`) USING BTREE,
KEY `WoNumber_2` (`WoNumber`) USING BTREE,
KEY `WoNumber_3` (`WoNumber`) USING BTREE
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC
CREATE TABLE `machine` (
`MachCode` varchar(15) NOT NULL,
`MachModel` varchar(30) NOT NULL,
`FirstProdDate` date NOT NULL,
`MachCapacity` smallint(6) NOT NULL,
`MachStatus` smallint(6) NOT NULL,
`NowMoldOnMach` varchar(10) NOT NULL DEFAULT '',
`NowMachOperator` varchar(50) NOT NULL DEFAULT '',
`NowWorkOrder` varchar(20) NOT NULL DEFAULT '',
`IPNumber` varchar(15) NOT NULL,
`Creator` varchar(50) NOT NULL,
`Changer` varchar(50) NOT NULL,
`ControlDate` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`OperatorLoginDate` datetime DEFAULT NULL,
`Message` varchar(500) DEFAULT NULL,
`MessageReaded` smallint(6) DEFAULT '0',
`StaffName` varchar(50) DEFAULT 'OSIS',
`StaffImage` varchar(255) DEFAULT '',
`StopDesc` varchar(30) DEFAULT 'OSIS',
`StopTime` datetime DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`MachCode`) USING BTREE
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC
CREATE TABLE `rawmaterials` (
`RawMaterialCode` varchar(15) NOT NULL,
`RawMaterialDescription` varchar(30) NOT NULL,
`RawMaterialColor` varchar(30) NOT NULL,
PRIMARY KEY (`RawMaterialCode`) USING BTREE,
KEY `RawMaterialCode` (`RawMaterialCode`) USING BTREE
) ENGINE=MyISAM AUTO_INCREMENT=3 DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC
CREATE TABLE `staff` (
`StaffCode` varchar(15) DEFAULT NULL,
`StaffCardCode` varchar(10) DEFAULT NULL,
`StaffName` varchar(50) NOT NULL,
`StaffPassword` varchar(10) NOT NULL,
`StaffStatus` smallint(6) NOT NULL DEFAULT '2',
`StaffDateOfStart` date NOT NULL,
`StaffBirthDay` date DEFAULT NULL,
`StaffGender` varchar(5) DEFAULT NULL,
`StaffRoleA` smallint(6) NOT NULL,
`StaffEmail` varchar(100) NOT NULL,
`StaffImageLink` varchar(255) DEFAULT NULL,
`AccountName` varchar(50) NOT NULL,
`StaffRoleB` smallint(6) NOT NULL,
`StaffRoleD` smallint(6) NOT NULL,
`StaffRoleE` smallint(6) NOT NULL,
`StaffRoleC` smallint(6) NOT NULL,
`StaffRoleF` smallint(6) NOT NULL,
`StaffRoleG` smallint(6) NOT NULL,
`StaffRoleH` smallint(6) NOT NULL,
`StaffRoleI` smallint(6) NOT NULL,
`StaffRoleJ` smallint(6) NOT NULL,
`StaffRoleK` smallint(6) NOT NULL,
`StaffRoleL` smallint(6) NOT NULL,
`StaffRoleM` smallint(6) NOT NULL,
`StaffRoleN` smallint(6) NOT NULL,
`StaffConnection` smallint(6) NOT NULL DEFAULT '2',
`MachineWorked` varchar(15) DEFAULT NULL,
`WorkOrderWorked` varchar(20) DEFAULT NULL,
`StaffGroup` varchar(50) NOT NULL,
`Creator` varchar(50) NOT NULL,
`Changer` varchar(50) DEFAULT NULL,
PRIMARY KEY (`AccountName`) USING BTREE
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC
CREATE TABLE `workorderb` (
`xWoNumber` varchar(20) NOT NULL,
`xMachCode` varchar(15) NOT NULL,
`xPlannedMoldCode` varchar(10) NOT NULL,
`xPartyNumber` smallint(6) NOT NULL,
`xStaffName` varchar(50) NOT NULL,
`xStopCode` smallint(6) NOT NULL,
`xStopStartTime` datetime NOT NULL,
`xStopFinishTime` datetime DEFAULT NULL,
`xStopTime` int(11) DEFAULT NULL,
PRIMARY KEY (`xMachCode`,`xStopStartTime`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC
Your query with the joins was nearly there: it was just missing a GROUP BYclause.
I have replaced workorder.* with workorder.WoNumber in the SELECT and added GROUP BY workorder.WoNumber.
You can add as many columns from workorder in the SELECT as you like but you must list them in the GROUP BY.
SELECT workorder.WoNumber,machine.MachModel,machine.MachStatus,rawmaterials.RawMaterialDescription,rawmaterials.RawMaterialColor,staff.StaffName,staff.StaffCode,SUM(workorderb.xStopTime)
FROM workorder
LEFT JOIN machine ON machine.MachCode=workorder.MachCode
LEFT JOIN rawmaterials ON rawmaterials.RawMaterialCode=workorder.ProductRawMaterial
LEFT JOIN staff ON staff.AccountName=workorder.AssignStaff
LEFT JOIN workorderb ON workorderb.xWoNumber=workorder.WoNumber
WHERE workorder.WoStatus=3
GROUP BY workorder.WoNumber \* <<= ADD OTHER COLUMNS HERE AS NEEDED *\
ORDER BY workorder.PlanProdStartDate DESC, workorder.WoSortNumber, workorder.WoNumber LIMIT 100;
db<>fiddle here
Use InnoDB, not MyISAM. MyISAM locks the entire table when INSERTing; InnoDB can often allow other threads to run when inserting.
Other notes
workorder has 4 identical indexes on wonumber; keep the PK, toss the rest. Note that a PRIMARY KEY is an index. Check the other tables for redundant Keys.
Do you need the mixture of DESC and ASC in ORDER BY PlanProdStartDate DESC, WoSortNumber, WoNumber? If not, there may be an optimization here.
As Kendle suggests, JOINs would be faster since there are cases where the same table is needed twice. If values might be missing, then LEFT might be useful; it won't change the performance.
Needed:
workorderb: INDEX(xWoNumber, xStopTime)
Is xStopTime an elapsed time? Or a time of day?

Simple subquery to slow on huge table

so I got a mysql database with 2 tables, one (sd_clients) with about 24k entries:
CREATE TABLE `sd_clients` (
`ms_id` varchar(10) NOT NULL,
`ms_share_id` varchar(10) NOT NULL,
`short_name` varchar(25) DEFAULT NULL,
`standard_name` varchar(75) DEFAULT NULL,
`legal_name` varchar(150) DEFAULT NULL,
`country` varchar(4) DEFAULT NULL,
`status` tinyint(1) DEFAULT NULL COMMENT '1=Paid Client | 2=Non-Paid Client',
`user_id` int(11) DEFAULT NULL,
`summary` text,
`sector` int(4) DEFAULT NULL,
`sub_sector` int(4) DEFAULT NULL,
`business_country` char(3) DEFAULT NULL,
`created_at` date DEFAULT NULL,
`is_paid` int(1) NOT NULL DEFAULT '0' COMMENT '0 = Non-Paid Client | 1=Paid Client',
`description_en` text,
`description_zh-hans` text,
`description_zh-hant` text,
`highlights_en` text,
`highlights_zh-hans` text,
`highlights_zh-hant` text,
`logo` varchar(255) DEFAULT NULL,
`summary_subsection_title_en` varchar(500) DEFAULT NULL,
`summary_subsection_title_zh-hans` varchar(500) DEFAULT NULL,
`summary_subsection_title_zh-hant` varchar(500) DEFAULT NULL,
`summary_subsection_text_en` text,
`summary_subsection_text_zh-hans` text,
`summary_subsection_text_zh-hant` text,
`summary_short_en` varchar(2000) DEFAULT NULL,
`summary_short_zh-hans` varchar(2000) DEFAULT NULL,
`summary_short_zh-hant` varchar(2000) DEFAULT NULL,
`other_information_en` text,
`other_information_zh-hans` text,
`other_information_zh-hant` text,
`change_percentage` decimal(10,3) DEFAULT NULL,
`id_sector` bigint(3) DEFAULT NULL,
`id_subsector` bigint(3) DEFAULT NULL,
`background_info_en` text,
`background_info_zh-hans` text,
`background_info_zh-hant` text,
`share_id_displayed` varchar(10) DEFAULT NULL,
PRIMARY KEY (`ms_id`) KEY_BLOCK_SIZE=1024,
UNIQUE KEY `ms_id` (`ms_id`) KEY_BLOCK_SIZE=1024,
KEY `share_id_displayed` (`share_id_displayed`) KEY_BLOCK_SIZE=1024
) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8;
SET FOREIGN_KEY_CHECKS=1;
And another called sd_clients_daily_stocks, with about 50 million entries:
CREATE TABLE `sd_clients_daily_stocks` (
`ms_id` varchar(10) NOT NULL,
`ms_share_id` varchar(10) DEFAULT NULL,
`created_at` date DEFAULT NULL,
`symbol` varchar(32) DEFAULT NULL,
`exchange_id` char(5) DEFAULT NULL,
`volume` bigint(18) DEFAULT NULL,
`day_low` decimal(19,6) DEFAULT NULL,
`day_high` decimal(19,6) DEFAULT NULL,
`market_cap` bigint(18) DEFAULT NULL,
`open_price` decimal(19,6) DEFAULT NULL,
`close_price` decimal(19,6) DEFAULT NULL,
`enterprise_value` bigint(18) DEFAULT NULL,
`currency_id` char(3) DEFAULT NULL,
`valoren` varchar(20) DEFAULT NULL,
`cusip` char(9) DEFAULT NULL,
`isin` varchar(12) DEFAULT NULL,
`sedol` varchar(7) DEFAULT NULL,
`ipo_date` date DEFAULT NULL,
`is_depositary_receipt` tinyint(1) DEFAULT NULL,
`depositary_receipt_ratio` decimal(9,4) DEFAULT NULL,
`security_type` char(10) DEFAULT NULL,
`share_class_description` varchar(1000) DEFAULT NULL,
`share_class_status` char(1) DEFAULT NULL,
`is_primary_share` tinyint(1) DEFAULT NULL,
`is_dividend_reinvest` tinyint(1) DEFAULT NULL,
`is_direct_invest` tinyint(1) DEFAULT NULL,
`investment_id` char(10) DEFAULT NULL,
`ipo_offer_price` decimal(19,6) DEFAULT NULL,
`delisting_date` date DEFAULT NULL,
`delisting_reason` varchar(100) DEFAULT NULL,
`mic` char(10) DEFAULT NULL,
`common_share_sub_type` varchar(32) DEFAULT NULL,
`ipo_offer_price_range` varchar(32) DEFAULT NULL,
`exchange_sub_market_global_id` char(10) DEFAULT NULL,
`conversion_ratio` decimal(19,9) DEFAULT NULL,
KEY `ms_id` (`ms_id`) USING HASH,
KEY `ms_share_id` (`ms_share_id`) USING HASH,
KEY `symbol` (`symbol`),
KEY `exchange_id` (`exchange_id`),
KEY `created_at` (`created_at`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
SET FOREIGN_KEY_CHECKS=1;
I'm trying to run a fairly simple query:
SELECT DISTINCT
sd_clients.ms_id,
sd_clients.standard_name,
sd_clients.is_paid,
sd_clients.logo,
sd_clients.change_percentage,
(
SELECT
CONCAT(
`exchange_id`, '|--|',
`symbol`, '|--|',
`close_price`, '|--|',
`day_low`, '|--|',
`day_high`
) as items
FROM sd_clients_daily_stocks
WHERE ms_share_id = sd_clients.share_id_displayed
ORDER BY created_at DESC
LIMIT 1
) as company_data
FROM sd_clients
GROUP BY ms_id
ORDER BY sd_clients.standard_name ASC
LIMIT 10
But for some reason, it's taking way too long (like over 1 minute), to get any results, any idea why?
BTW, it works just fine if I remove the subquery, but I need it because the rest of the data, is in another table. Also, I know I could get the results without the subquery first, but I have other queries where the subquery must be there.
I also noticed that it gets blazing fast if I use a string instead of "sd_clients.share_id_displayed" on the subquery.
You should try an index on sd_clients_daily_stocks(ms_share_id, created_at).
You can add the additional columns from the select if you want a covering index.
You would probably be better off joining on a non-correlated subquery based off your current subquery; using this kind of technique here to find the most recent rows for each in the new subquery.
Edit: I was thinking more something like this:
SELECT DISTINCT sd_clients.ms_id, sd_clients.standard_name, sd_clients.is_paid, sd_clients.logo, sd_clients.change_percentage
, scdsB.items
FROM sd_clients
INNER JOIN (
SELECT scdsA.ms_share_id
, CONCAT(
scdsA.`exchange_id`, '|--|', scdsA.`symbol`, '|--|',
scdsA.`close_price`, '|--|', scdsA.`day_low`, '|--|', scdsA.`day_high`
) as items
FROM sd_clients_daily_stocks AS scdsA
INNER JOIN (
SELECT ms_share_id, MAX(created_at)
FROM sd_clients_daily_stocks
GROUP BY ms_share_id
) AS lasts
ON scdsA.ms_share_id = lasts.ms_share_id
AND scdsA.created_at = lasts.created_at
) scdsB
ON sd_clients.share_id_displayed = scdsB.ms_share_id
GROUP BY sd_clients.ms_id
ORDER BY sd_clients.standard_name ASC
LIMIT 10;
... But even this likely won't reduce the speed much more, if any, from the 12 seconds. At that point you are better off looking into indexes that could help. For example, the lasts grouping subquery that finds the most recent(max) created_at value for each ms_share_id value would benefit from an index on sd_clients_daily_stocks (ms_shared_id, created_at) as would the the JOIN it is used in.

Two MySql tables have correct indexes yet JOIN takes 9 seconds on small tables

mysql Ver 14.14 Distrib 5.1.73, for redhat-linux-gnu (x86_64) using readline 5.1
I am taking over a project. It is very old and the original programmer is long gone. No one has any idea why certain decisions were made.
The following query runs (on my Mac) in 9.5 seconds but if I remove the last JOIN then it drops to 2.5 seconds. What is wrong with that last JOIN?
select `ttl`.`id` AS `id`,
`ttl`.`name` AS `name`,
`ttl`.`updated_at` AS `last_update_on`,
`ttl`.`user_id` AS `list_creator`,
`ttl`.`retailer_nomination_list` AS `nomination_list`,
`ttl`.`created_at` AS `created_on`,
count(distinct `tlb`.`title_id`) AS `title_count`
from `haha_title_lists` `ttl`
left join `haha_title_list_to_users` `tltu` on((`ttl`.`id` = `tltu`.`title_list_id`))
left join `users` `u` on((`tltu`.`user_id` = `u`.`id`))
left join `users` `u2` on((`tltu`.`user_id` = `u2`.`id`))
left join `haha_title_list_to_venues` `tlv` on((`ttl`.`id` = `tlv`.`title_list`))
left join `haha_venue_properties` `tvp` on((`tlv`.`venue_id` = `tvp`.`id`))
join `haha_title_list_to_books` `tlb` on((`ttl`.`id` = `tlb`.`title_list_id`))
join `wawa_title` `ot` on((`tlb`.`title_id` = `ot`.`title_id`))
group by `ttl`.`id`;
The tables:
CREATE TABLE `haha_title_list_to_books` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`title_id` int(11) NOT NULL,
`title_list_id` int(11) NOT NULL,
`sdk` varchar(15) COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
`created_at` datetime NOT NULL,
`promo_start_date` date DEFAULT NULL,
`promo_end_date` date DEFAULT NULL,
`promo_price` float DEFAULT NULL,
`confirmations` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`nominations` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`title_note` text COLLATE utf8_unicode_ci,
`executed` int(11) DEFAULT NULL,
`event_created` int(11) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `idx_promo_start_date` (`promo_start_date`),
KEY `idx_promo_end_date` (`promo_end_date`),
KEY `idx_title_list_to_books_title_id` (`title_id`),
KEY `idx_title_list_to_books_title_list_id` (`title_list_id`)
) ENGINE=MyISAM AUTO_INCREMENT=21847 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
and:
CREATE TABLE `wawa_title` (
`title_id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(200) DEFAULT NULL,
`title_alpha` varchar(25) NOT NULL,
`display_title` varchar(200) NOT NULL,
`subtitle` text NOT NULL,
`sdk10` varchar(13) DEFAULT '',
`sdk13` varchar(15) DEFAULT NULL,
`primary_sdk13` varchar(15) DEFAULT NULL,
`asin` varchar(10) DEFAULT NULL,
`pub_season` varchar(15) NOT NULL,
`pub_year` varchar(15) NOT NULL,
`bisac1` varchar(15) NOT NULL,
`bisac2` varchar(15) NOT NULL,
`bisac3` varchar(15) NOT NULL,
`barcode` varchar(30) DEFAULT NULL,
`dewey_decimal` varchar(15) NOT NULL,
`lib_of_congress` varchar(15) NOT NULL,
`spanish_language` tinyint(4) NOT NULL,
`target_audience` tinyint(3) unsigned DEFAULT NULL,
`language` varchar(20) DEFAULT NULL,
`edition` varchar(45) DEFAULT NULL,
`pages` int(11) DEFAULT NULL,
`number_in_series` int(11) DEFAULT NULL,
`trimsize` varchar(10) DEFAULT NULL,
`filesize` varchar(10) DEFAULT NULL,
`duration_hours` int(11) DEFAULT NULL,
`duration_minutes` int(11) DEFAULT NULL,
`discs` int(11) DEFAULT NULL,
`download` date DEFAULT NULL,
`size_unit` varchar(15) NOT NULL DEFAULT '',
`digitization_date` date NOT NULL,
`us_on_sale_date` date NOT NULL,
`aus_on_sale_date` date NOT NULL,
`can_on_sale_date` date NOT NULL,
`uk_on_sale_date` date NOT NULL,
`us_list_price` varchar(10) NOT NULL,
`aus_list_price` varchar(10) NOT NULL,
`can_list_price` varchar(10) NOT NULL,
`uk_list_price` varchar(10) NOT NULL,
`isPrimary` varchar(1) DEFAULT NULL,
`modified` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`modifier` int(11) NOT NULL,
`activated` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`active` varchar(3) NOT NULL DEFAULT 'N',
`flagged_string` text,
`created` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`assets_id` varchar(20) DEFAULT NULL,
`book_details` text,
`book_keynote` text,
`exclude_goodreads` char(1) NOT NULL DEFAULT 'N',
`series_description` text,
`review_quote1` text,
`territory_id` int(11) DEFAULT '27',
`featured_newsletter_id` tinyint(3) unsigned DEFAULT '0',
`retailer_discovery_check` datetime DEFAULT NULL,
`suppress_retailer_approval` tinyint(1) DEFAULT '0',
`suppress_retailer_approval_reason` varchar(255) DEFAULT NULL,
`ebb_description` text CHARACTER SET utf8 COLLATE utf8_unicode_ci,
`slug` varchar(150) DEFAULT NULL,
`legacy_slug` varchar(150) DEFAULT NULL,
`us_agency_price` varchar(10) DEFAULT NULL,
`firebrand_title_id` int(11) DEFAULT NULL,
`ebb_label` varchar(200) DEFAULT NULL,
`ebb_end_sale_date` date DEFAULT NULL,
`ebb_downprice` decimal(10,2) DEFAULT NULL,
`book_club` varchar(1) DEFAULT NULL,
`best_seller` varchar(1) DEFAULT NULL,
`award_winner` varchar(1) DEFAULT NULL,
`discovery` char(1) NOT NULL DEFAULT 'Y',
`narrator_id` int(11) DEFAULT NULL,
`suppress_series_data` varchar(255) DEFAULT NULL,
PRIMARY KEY (`title_id`),
KEY `active_index` (`active`),
KEY `fk_title_series_id_idx` (`series_id`),
KEY `series_id` (`series_id`),
KEY `idx_title_sdk13` (`sdk13`),
KEY `idx_title_active_isprimary` (`active`,`isPrimary`),
KEY `bisac1` (`bisac1`),
KEY `bisac2` (`bisac2`),
KEY `bisac3` (`bisac3`),
KEY `idx_primary_sdk13` (`primary_sdk13`),
KEY `idx_territory_id` (`territory_id`),
CONSTRAINT `fk_title_series_id` FOREIGN KEY (`series_id`) REFERENCES `wawa_series` (`series_id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=19700 DEFAULT CHARSET=utf8 |
If I remove this line:
join `wawa_title` `ot` on((`tlb`.`title_id` = `ot`.`title_id`))
The query speed drops from 9.5 seconds to 2.5 seconds. Not great, but a huge improvement.
And yet, both tables have indexes on table_id, so why would that line be a problem?
I notice that one table is InnoDB and the other is MyISAM. Would that have an effect?
Do not JOIN to tables that you don't use.
A JOIN often "explodes" the number of rows, then a GROUP BY like you have reels in the number of rows. To see this, leave all the JOINs there, but remove the GROUP BY. See how many rows you get.
To avoid part of that explosion, change
count(distinct `tlb`.`title_id`) AS `title_count`
to
( SELECT count(distinct `title_id`)
FROM `haha_title_list_to_books`
WHERE `ttl`.`id` = `title_list_id`
) AS `title_count`
and remove the current JOIN to tlb.
Mixing MyISAM and InnoDB should not have any direct impact on this SELECT. However, you should consider moving all of your tables to InnoDB.

Slow update of one table when comparing multiple fields across two tables

The following query is timing out after 600 seconds.
update placed p
,Results r
set p.position = r.position
where p.competitor = r.competitor
AND p.date = r.date
AND REPLACE(p.time,":","") = r.time;
The structure is as follows:
'CREATE TABLE `placed` (
`idplaced` varchar(50) DEFAULT NULL,
`date` decimal(8,0) DEFAULT NULL,
`time` varchar(45) DEFAULT NULL,
`field1` varchar(45) DEFAULT NULL,
`competitor` varchar(45) DEFAULT NULL,
`field2` int(2) DEFAULT NULL,
`field3` varchar(45) DEFAULT NULL,
`field4` varchar(45) DEFAULT NULL,
`field5` decimal(6,2) DEFAULT NULL,
`field6` decimal(10,2) DEFAULT NULL,
`field7` decimal(6,2) DEFAULT NULL,
`field8` char(1) DEFAULT NULL,
`field9` varchar(45) DEFAULT NULL,
`position` char(4) DEFAULT NULL,
`field10` decimal(6,2) DEFAULT NULL,
`field11` char(1) DEFAULT NULL,
`field12` char(1) DEFAULT NULL,
`field13` decimal(6,2) DEFAULT NULL,
`field14` decimal(6,2) DEFAULT NULL,
`field15` decimal(6,2) DEFAULT NULL,
`field16` decimal(6,2) DEFAULT NULL,
`field17` decimal(6,2) DEFAULT NULL,
`field18` char(1) DEFAULT NULL,
`field19` char(20) DEFAULT NULL,
`field20` char(1) DEFAULT NULL,
`field21` char(5) DEFAULT NULL,
`field22` char(5) DEFAULT NULL,
`field23` int(11) DEFAULT NULL
PRIMARY KEY (`idplaced`),
UNIQUE KEY `date_time_competitor_field18_combo` (`date`,`time`,`competitor`,`field18`)
) ENGINE=InnoDB AUTO_INCREMENT=100688607 DEFAULT CHARSET=latin1;
CREATE TABLE `results` (
`idresults` int(11) NOT NULL AUTO_INCREMENT,
`date` char(8) DEFAULT NULL,
`time` char(4) DEFAULT NULL,
`field1` varchar(45) DEFAULT NULL,
`competitor` varchar(45) DEFAULT NULL,
`position` char(4) DEFAULT NULL,
`field2` varchar(45) DEFAULT NULL,
`field3` decimal(2,0) DEFAULT NULL,
PRIMARY KEY (`idresults`)
) ENGINE=InnoDB AUTO_INCREMENT=6644 DEFAULT CHARSET=latin1;
The PLACED table has 65,000 records, the RESULTS table has 9,000 records.
I am assuming the solution involves a JOIN statement of some descript, and I have tried taking several suggestions from this site, but am simply not finding the answer I am looking for. Simply put, I would be grateful for suggestions on this. I can put up example tables / create table code if requried.
The index cannot be used efficiently to perform the join because of your REPLACE operation.
I'd suggest creating an index with the columns in the following slightly different order:
(date, competitor, time, position)
It may also help to add this index on both tables.
It would be even better if you could modify the data in the database so that the data in the time column was stored in the same format in both tables.
First of all, you'd better send us your full tables description, using
show create table
Second, you'd better use join syntax :
update placed p
join Results r on r.competitor = p.competitor
set p.position = r.position
where p.date = r.date
AND REPLACE(p.time,":","") = r.time;
Hope this will help.

MySQL - Counting a count?

I'm trying to aggregate some data I've pulled from my MySQL db.
Query I'm using is:
SELECT `PUID`,`DROID_V`,`SIG_V`,`SPEED`,
COUNT(distinct IF(sourcelist.hasExtension=1,NAME,NULL)) as Ext,
COUNT(distinct IF(sourcelist.hasExtension=0,NAME,NULL)) as NoExt,
COUNT(distinct NAME) as `All`
FROM sourcelist, main_small
WHERE sourcelist.SourcePUID = 'My_Variable' AND main_small.NAME = sourcelist.SourceFileName
GROUP BY `PUID`,`DROID_V`,`SIG_V`,`SPEED` ORDER BY `DROID_V` ASC, `SIG_V` ASC, `SPEED`
And I wondered if there was a way of counting this result, so I can make a new table that would show me something like:
Every distinct PUID, (count of distinct DROID_V), (count of distinct Sig_V), (SUM of total hits for NAME) WHERE sourcelist.SourcePUID = 'My_Variable' AND main_small.NAME = sourcelist.SourceFileName
As you can see, I'm really not very good at SQL!
Source table:
CREATE TABLE `t1` (
`DROID_V` int(1) DEFAULT NULL,
`Sig_V` varchar(7) DEFAULT NULL,
`SPEED` varchar(4) DEFAULT NULL,
`ID` varchar(7) DEFAULT NULL,
`PARENT_ID` varchar(10) DEFAULT NULL,
`URI` varchar(10) DEFAULT NULL,
`FILE_PATH` varchar(68) DEFAULT NULL,
`NAME` varchar(17) DEFAULT NULL,
`METHOD` varchar(10) DEFAULT NULL,
`STATUS` varchar(14) DEFAULT NULL,
`SIZE` int(10) DEFAULT NULL,
`TYPE` varchar(10) DEFAULT NULL,
`EXT` varchar(4) DEFAULT NULL,
`LAST_MODIFIED` varchar(10) DEFAULT NULL,
`EXTENSION_MISMATCH` varchar(32) DEFAULT NULL,
`MD5_HASH` varchar(10) DEFAULT NULL,
`FORMAT_COUNT` varchar(10) DEFAULT NULL,
`PUID` varchar(15) DEFAULT NULL,
`MIME_TYPE` varchar(24) DEFAULT NULL,
`FORMAT_NAME` varchar(10) DEFAULT NULL,
`FORMAT_VERSION` varchar(10) DEFAULT NULL,
`INDEX` int(11) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`INDEX`)
) ENGINE=MyISAM AUTO_INCREMENT=960831 DEFAULT CHARSET=utf8
example records:
5;"v37";"slow";"10266";;"file:";"V1-FL425817.tif";"V1-FL425817.tif";"BINARY_SIG";"MultipleIdenti";"20603284";"FILE";"tif";"2008-11-03";;;;"fmt/7";"image/tiff";"Tagged Ima";"3";"191977"
5;"v37";"slow";"10268";;"file:";"V1-FL425817.tif";"V1-FL425817.tif";"BINARY_SIG";"MultipleIdenti";"20603284";"FILE";"tif";"2008-11-03";;;;"fmt/8";"image/tiff";"Tagged Ima";"4";"191978"
5;"v37";"slow";"10269";;"file:";"V1-FL425817.tif";"V1-FL425817.tif";"BINARY_SIG";"MultipleIdenti";"20603284";"FILE";"tif";"2008-11-03";;;;"fmt/9";"image/tiff";"Tagged Ima";"5";"191979"
5;"v37";"slow";"10270";;"file:";"V1-FL425817.tif";"V1-FL425817.tif";"BINARY_SIG";"MultipleIdenti";"20603284";"FILE";"tif";"2008-11-03";;;;"fmt/10";"image/tiff";"Tagged Ima";"6";"191980"
Sure just do something like
Select Count(*) From (Select * From SomeTable) adummynamesoSqlParserDoesntgetupset
so put your query in parentheses after FROM give it a unique name, and you can treat it like a table or a view.