I have a table in my database that contain around 10 millions rows.
The problem occur when I execute this query : It takes a very long time to execute (12.418s) and I wonder how can I speed this query up.
SELECT *
FROM cadastre_test
WHERE latitude >= 45.1269166 AND latitude <= 45.127816 AND longitude >= -0.6631578 AND longitude <= -0.6618832
LIMIT 100 ;
I tried the BETWEEN instead of <= and >= but it was slower of 4 seconds.
Here is the CREATE TABLE query :
CREATE TABLE `cadastre_test` (
`id_mutation` varchar(255) NOT NULL,
`date_mutation` date DEFAULT NULL,
`numero_disposition` varchar(8) DEFAULT NULL,
`nature_mutation` varchar(255) DEFAULT NULL,
`valeur_fonciere` float DEFAULT NULL,
`adresse_numero` smallint(6) DEFAULT NULL,
`adresse_suffixe` varchar(128) DEFAULT NULL,
`adresse_nom_voie` varchar(128) DEFAULT NULL,
`adresse_code_voie` varchar(8) DEFAULT NULL,
`code_postal` varchar(10) DEFAULT NULL,
`code_commune` varchar(10) DEFAULT NULL,
`nom_commune` varchar(255) DEFAULT NULL,
`code_departement` varchar(10) DEFAULT NULL,
`ancien_code_commune` varchar(10) DEFAULT NULL,
`ancien_nom_commune` varchar(255) DEFAULT NULL,
`id_parcelle` varchar(20) DEFAULT NULL,
`ancien_id_parcelle` varchar(20) DEFAULT NULL,
`numero_volume` varchar(15) DEFAULT NULL,
`lot1_numero` varchar(6) DEFAULT NULL,
`lot1_surface_carrez` float DEFAULT NULL,
`lot2_numero` varchar(6) DEFAULT NULL,
`lot2_surface_carrez` float DEFAULT NULL,
`lot3_numero` varchar(6) DEFAULT NULL,
`lot3_surface_carrez` float DEFAULT NULL,
`lot4_numero` varchar(6) DEFAULT NULL,
`lot4_surface_carrez` float DEFAULT NULL,
`lot5_numero` varchar(6) DEFAULT NULL,
`lot5_surface_carrez` float DEFAULT NULL,
`nombre_lots` smallint(6) DEFAULT NULL,
`code_type_local` smallint(6) DEFAULT NULL,
`type_local` varchar(255) DEFAULT NULL,
`surface_reelle_bati` float DEFAULT NULL,
`nombre_pieces_principales` smallint(6) DEFAULT NULL,
`code_nature_culture` varchar(10) DEFAULT NULL,
`nature_culture` varchar(255) DEFAULT NULL,
`code_nature_culture_speciale` varchar(255) DEFAULT NULL,
`nature_culture_speciale` varchar(255) DEFAULT NULL,
`surface_terrain` float DEFAULT NULL,
`longitude` decimal(11,7) DEFAULT NULL,
`latitude` decimal(11,7) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
add an index on your table for latitude , longitude:
create index inx_lang_long on cadastre_test (latitude , longitude) ;
Related
I have a data logging application for which i have created a table as shown below -
CREATE TABLE IF NOT EXISTS `waterheater`.`heatertest550001_data`
` (
`Index_Records` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
`Time_Stamp` DATETIME DEFAULT NULL,
`Water_Temperature` FLOAT DEFAULT NULL,
`Body_Temperature` FLOAT DEFAULT NULL,
`Body_Shell_Top_Temperature` FLOAT DEFAULT NULL,
`Body_Shell_Bottom_Temperature` FLOAT DEFAULT NULL,
`Ambient_Temperature` FLOAT DEFAULT NULL,
`Voltage` FLOAT DEFAULT NULL,
`Current` FLOAT DEFAULT NULL,
`Wattage` FLOAT DEFAULT NULL,
`Energy` FLOAT DEFAULT NULL,
`Power_Factor` FLOAT DEFAULT NULL,
`Notice` VARCHAR(45) DEFAULT NULL,
`LVTIMESTAMP` DOUBLE DEFAULT NULL,
PRIMARY KEY (`Index_Records`),
UNIQUE INDEX `INDEX_UNIQUE` (`Index_Records`)
) ENGINE=INNODB DEFAULT CHARSET=UTF8;
And to attach information for the same table I'm creating another table as follows -
CREATE TABLE IF NOT EXISTS `waterheater`.`heatertest550001_info` (
`Index_Records` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT,
`ReportNumber` VARCHAR(45) DEFAULT NULL,
`ProjectNumber` VARCHAR(45) DEFAULT NULL,
`TestDiscipline` VARCHAR(45) DEFAULT 'Electrical',
`Customer` VARCHAR(45) DEFAULT NULL,
`Manufacturer` VARCHAR(45) DEFAULT NULL,
`Program` VARCHAR(45) DEFAULT 'NABL',
`TestLabLocation` VARCHAR(45) DEFAULT '(b)UL Manesar',
`ItemUnderTest` VARCHAR(45) DEFAULT 'Storage Electrical Water Heater',
`Model` VARCHAR(45) DEFAULT NULL,
`NumberOfSamples` VARCHAR(45) DEFAULT NULL,
`SampleIdentification` VARCHAR(45) DEFAULT NULL,
`ManufacturerSerialNumber` VARCHAR(45) DEFAULT NULL,
`IUTCondition` VARCHAR(45) DEFAULT NULL,
`DateOfReceipt` DATETIME DEFAULT NULL,
`ApplicableStandard` VARCHAR(45) DEFAULT 'IS2082:1993 & IS 302-2-21:1992',
`DateOfTestingStartDate` DATETIME DEFAULT NULL,
`DateOfTestingStopDate` DATETIME DEFAULT NULL,
`DurationSecondsOfTesting` FLOAT DEFAULT NULL,
`DurationTestText` VARCHAR(45) DEFAULT NULL,
`LabAmbientTemperature` FLOAT DEFAULT '23',
`LabAmbientHumidity` FLOAT DEFAULT '70',
`DateOfReporting` DATETIME DEFAULT NULL,
`TestIncharge` VARCHAR(45) DEFAULT NULL,
`GeneralRemarks` VARCHAR(45) DEFAULT NULL,
`IUTRatedVoltage` FLOAT DEFAULT NULL,
`IUTRatedCurrent` FLOAT DEFAULT NULL,
`IUTRatedFrequency` FLOAT DEFAULT NULL,
`IUTRatedPower` FLOAT DEFAULT NULL,
`IUTRatedCapacity` FLOAT DEFAULT NULL,
`IUTApplianceSort` VARCHAR(45) DEFAULT 'Class 1',
`IUTRatedWaterProofDegree` VARCHAR(45) DEFAULT 'IP X4',
`IUTRatedPressure` FLOAT DEFAULT '0.80',
`IUTApplianceType` VARCHAR(45) DEFAULT NULL,
`IUTRatedStandingLoss` FLOAT DEFAULT NULL,
`CalcWaterTemperature` FLOAT DEFAULT NULL,
`CalcAmbientTemperature` FLOAT DEFAULT NULL,
`CalcVoltage` FLOAT DEFAULT NULL,
`CalcCurrent` FLOAT DEFAULT NULL,
`CalcFrequency` FLOAT DEFAULT NULL,
`CalcPower` FLOAT DEFAULT NULL,
`CalcCapacity` FLOAT DEFAULT NULL,
`CalcApplianceSort` VARCHAR(45) DEFAULT 'Class 1',
`CalcRatedWaterProofDegree` VARCHAR(45) DEFAULT 'IP X4',
`CalcPressure` FLOAT DEFAULT '0.80',
`CalcApplianceType` VARCHAR(45) DEFAULT NULL,
`CalcStandingLoss` FLOAT DEFAULT NULL,
`CalcStandingLossRate` FLOAT DEFAULT NULL,
`CalcStandingLoss24h` FLOAT DEFAULT NULL,
`EnergyConsumptionTest` FLOAT DEFAULT NULL,
`EnergyConsumption24h` FLOAT DEFAULT NULL,
`MaxRecords` FLOAT DEFAULT NULL,
PRIMARY KEY (`Index_Records`),
UNIQUE KEY `Index_Records_UNIQUE` (`Index_Records`)
) ENGINE=INNODB DEFAULT CHARSET=UTF8;
I'm sure there must be a better way to do this. Can somebody help! and thanks.
I am running the following query in MySQL Workbench:
UPDATE directorylistings INNER JOIN postcodelatlng ON PostalCode = postcode
set directoryListings.Latitude = postcodelatlng.latitude, directoryListings.Longitude = postcodelatlng.longitude
WHERE PostalCode = postcode;
Table directorylistings has 9000 records, postcodelatlng has nearly 2 million records.
directorylistings has a postcode (zip code) field and I'm looking this up in postcodelatlng to retrieve and update the latitude and longitude fields in directorylistings.
Ater timeout issues I found I should change DBMS connection read time out, I set this to 36000 (10 hours). The query has been running of 5 hours so far.
My question is, is this time reasonable? Should I be indexing something? Could the query be optimised?
I did make a 2 record version of the postcodelatlng and ran the query, 16 rows were correctly changed in 0.04 seconds, so I know the query is at least workable.
I have looked at a number of Q/A's on indexing and optimising, but it's taken me over a day to get this far and if it fails again I would like to know what approach to follow. I can't keep trying things and waiting hours for the failure! I bet this query should really take seconds. I'm using a MacBook Pro with a 2.5GHz i7, 16 GB RAM and ssd drive.
Many Thanks.
cpu is at 99%. I aborted after 12 hours! to run the show create tables:
CREATE TABLE `postcodelatlng` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`postcode` varchar(8) NOT NULL,
`latitude` decimal(18,15) NOT NULL,
`longitude` decimal(18,15) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=1734490 DEFAULT CHARSET=latin1
CREATE TABLE `directorylistings` (
`ID` int(11) NOT NULL AUTO_INCREMENT,
`Active` tinyint(1) DEFAULT NULL,
`Tag11` tinyint(1) DEFAULT NULL,
`Minyan` tinyint(1) DEFAULT NULL,
`Latitude` double DEFAULT NULL,
`Longitude` double DEFAULT NULL,
`AffiliatedDate` double DEFAULT NULL,
`RegisteredDate` datetime DEFAULT NULL,
`AccountID` varchar(10) DEFAULT NULL,
`ShowListing` tinyint(1) DEFAULT NULL,
`Bloguser` tinyint(1) DEFAULT NULL,
`Trusted` tinyint(1) DEFAULT NULL,
`ClassifiedUser` tinyint(1) DEFAULT NULL,
`HostingID` int(11) DEFAULT NULL,
`EndDate` double DEFAULT NULL,
`EndPromoDate` datetime DEFAULT NULL,
`BaseID` int(11) DEFAULT NULL,
`Banner` varchar(50) DEFAULT NULL,
`PrivateListing` tinyint(1) DEFAULT NULL,
`Reserved1` tinyint(1) DEFAULT NULL,
`Reserved2` tinyint(1) DEFAULT NULL,
`Reserved3` tinyint(1) DEFAULT NULL,
`Reserved4` tinyint(1) DEFAULT NULL,
`Reserved5` tinyint(1) DEFAULT NULL,
`CommunityOrg` tinyint(1) DEFAULT NULL,
`Tag1` tinyint(1) DEFAULT NULL,
`Tag2` tinyint(1) DEFAULT NULL,
`Tag3` tinyint(1) DEFAULT NULL,
`Tag4` tinyint(1) DEFAULT NULL,
`Tag5` tinyint(1) DEFAULT NULL,
`Tag6` tinyint(1) DEFAULT NULL,
`Tag7` tinyint(1) DEFAULT NULL,
`Tag8` tinyint(1) DEFAULT NULL,
`Tag9` tinyint(1) DEFAULT NULL,
`Tag10` tinyint(1) DEFAULT NULL,
`Tag12` tinyint(1) DEFAULT NULL,
`Tag13` tinyint(1) DEFAULT NULL,
`Tag14` tinyint(1) DEFAULT NULL,
`ShomerShabbos` tinyint(1) DEFAULT NULL,
`CategoryID1` int(11) DEFAULT NULL,
`CategoryID2` int(11) DEFAULT NULL,
`CategoryID3` int(11) DEFAULT NULL,
`CategoryID4` int(11) DEFAULT NULL,
`CategoryID5` int(11) DEFAULT NULL,
`CategoryID6` int(11) DEFAULT NULL,
`HiddenCategory` int(11) DEFAULT NULL,
`Company` varchar(100) DEFAULT NULL,
`Slogan` varchar(250) DEFAULT NULL,
`Description` varchar(2000) DEFAULT NULL,
`CharShown` int(11) DEFAULT NULL,
`Address` varchar(100) DEFAULT NULL,
`Unit` varchar(50) DEFAULT NULL,
`Address2` varchar(150) DEFAULT NULL,
`Streetno` varchar(50) DEFAULT NULL,
`Building` varchar(50) DEFAULT NULL,
`Parade` varchar(50) DEFAULT NULL,
`Locality` varchar(50) DEFAULT NULL,
`City` varchar(50) DEFAULT NULL,
`Province` varchar(50) DEFAULT NULL,
`PostalCode` varchar(10) DEFAULT NULL,
`Country` varchar(25) DEFAULT NULL,
`PhoneNumber` varchar(70) DEFAULT NULL,
`Ext` varchar(25) DEFAULT NULL,
`PhoneNumber2` varchar(70) DEFAULT NULL,
`FaxNumber` varchar(30) DEFAULT NULL,
`CellNumber` varchar(30) DEFAULT NULL,
`WebUrl` varchar(100) DEFAULT NULL,
`Email` varchar(75) DEFAULT NULL,
`Password` varchar(30) DEFAULT NULL,
`ContactName` varchar(100) DEFAULT NULL,
`Keywords` varchar(355) DEFAULT NULL,
`Dcount` int(11) DEFAULT NULL,
`LocationID` int(11) DEFAULT NULL,
`Comments` varchar(1500) DEFAULT NULL,
`ApplicantName` varchar(200) DEFAULT NULL,
`ApplicantPhone` varchar(200) DEFAULT NULL,
`ApplicantEmail` varchar(200) DEFAULT NULL,
`Photo1` varchar(250) DEFAULT NULL,
`pRating` int(11) DEFAULT NULL,
`sRating` int(11) DEFAULT NULL,
`vRating` int(11) DEFAULT NULL,
`Private` tinyint(1) NOT NULL DEFAULT '0',
`VeryPrivate` tinyint(1) NOT NULL DEFAULT '0',
`SemiSecure` tinyint(1) NOT NULL DEFAULT '0',
`Secure` tinyint(1) NOT NULL DEFAULT '0',
`Secret` tinyint(1) NOT NULL DEFAULT '0',
`TrustHigh` tinyint(1) NOT NULL DEFAULT '0',
`FL` tinyint(1) NOT NULL DEFAULT '1',
`GJ` tinyint(1) NOT NULL DEFAULT '1',
`Recommended` tinyint(1) NOT NULL DEFAULT '0',
`NonGeo` tinyint(1) NOT NULL DEFAULT '0',
`Edited` datetime DEFAULT NULL,
`Editor` varchar(40) DEFAULT NULL,
PRIMARY KEY (`ID`),
UNIQUE KEY `ID_UNIQUE` (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=27167 DEFAULT CHARSET=utf8
I am running a update statement:
UPDATE ACTION a
INNER JOIN subscriberinfo s ON a.subscriberId=s.id
SET a.exceedusage = (SELECT FORMAT(((SUM(dataVolumeDownLink + dataVolumeUpLink))/1048576),2)
FROM cdr c
WHERE c.msisdn=s.msisdn
AND c.eventDate>=a.createdon
AND c.eventDate <= a.actionTakenOn)
WHERE a.remark='Reason : Data limit crossed'
AND a.exceedusage IS NULL;
But its taking too much time because of CDR table(millions of row). Is there any way I can rewrite this query so it can work fast?
EDIT:
Table structure of Action:
CREATE TABLE `action` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`actionTakenOn` datetime DEFAULT NULL,
`actionType` varchar(255) DEFAULT NULL,
`cdrCreatedOn` datetime DEFAULT NULL,
`cdrEventDate` datetime DEFAULT NULL,
`createdOn` datetime DEFAULT NULL,
`errorDescription` longtext,
`params` longtext,
`remark` longtext,
`requestedOn` datetime DEFAULT NULL,
`status` varchar(255) DEFAULT NULL,
`subscriberDetails` longtext,
`takenBy` varchar(255) DEFAULT NULL,
`subscriberId` bigint(20) DEFAULT NULL,
`ticketId` bigint(20) DEFAULT NULL,
`dataPlanEndTime` datetime DEFAULT NULL,
`dataPlanStartTime` datetime DEFAULT NULL,
`dataUsage` bigint(20) DEFAULT NULL,
`dataplanName` varchar(255) DEFAULT NULL,
`exceedUsage` bigint(20) DEFAULT NULL,
`isNotified` bit(1) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `FKAB2F7E36E90F678D` (`subscriberId`),
KEY `FKAB2F7E3664633B07` (`ticketId`)
) ENGINE=MyISAM AUTO_INCREMENT=81534 DEFAULT CHARSET=latin1;
for Subscriberinfo:
CREATE TABLE `subscriberinfo` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`imsi` varchar(255) DEFAULT NULL,
`simType` varchar(45) DEFAULT NULL,
`dataPlanStartTime` datetime DEFAULT NULL,
`dataPlanEndTime` datetime DEFAULT NULL,
`dataplan` varchar(255) DEFAULT NULL,
`status` varchar(255) DEFAULT NULL,
`validDays` varchar(255) DEFAULT NULL,
`deviceName` varchar(500) DEFAULT NULL,
`lastDataPlanUpdatedOn` datetime DEFAULT NULL,
`lastDeviceUpdatedOn` datetime DEFAULT NULL,
`createdOn` datetime DEFAULT NULL,
`dataplanType` varchar(255) DEFAULT NULL,
`msisdn` bigint(20) DEFAULT NULL,
`dataLeft` bigint(20) DEFAULT NULL,
`billingSysDataPlanEndTime` datetime DEFAULT NULL,
`billingSysDataPlanStartTime` datetime DEFAULT NULL,
`billingSysValidDays` int(11) DEFAULT NULL,
`dataUsage` bigint(20) DEFAULT NULL,
`planDetail` varchar(255) DEFAULT NULL,
`currentSpeedLimit` varchar(255) DEFAULT NULL,
`lastBillingSysUpdatedOn` datetime DEFAULT NULL,
`lastUpdatedOn` datetime DEFAULT NULL,
`deviceType` varchar(255) DEFAULT NULL,
`lastImsiUpdatedOn` datetime DEFAULT NULL,
`skipCheck` tinyint(1) NOT NULL,
`active` tinyint(1) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `msisdn_UNIQUE` (`msisdn`)
) ENGINE=InnoDB AUTO_INCREMENT=49032 DEFAULT CHARSET=latin1;
for CDR:
CREATE TABLE `cdr` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`dataPacketDownLink` bigint(20) DEFAULT NULL,
`dataPacketUpLink` bigint(20) DEFAULT NULL,
`dataPlanEndTime` datetime DEFAULT NULL,
`dataPlanStartTime` datetime DEFAULT NULL,
`dataVolumeDownLink` bigint(20) DEFAULT NULL,
`dataVolumeUpLink` bigint(20) DEFAULT NULL,
`dataplan` varchar(255) DEFAULT NULL,
`dataplanType` varchar(255) DEFAULT NULL,
`createdOn` datetime DEFAULT NULL,
`deviceName` varchar(500) DEFAULT NULL,
`duration` int(11) NOT NULL,
`effectiveDuration` int(11) NOT NULL,
`hour` int(11) DEFAULT NULL,
`eventDate` datetime DEFAULT NULL,
`msisdn` bigint(20) DEFAULT NULL,
`quarter` int(11) DEFAULT NULL,
`validDays` int(11) DEFAULT NULL,
`dataLeft` bigint(20) DEFAULT NULL,
`completedOn` datetime DEFAULT NULL,
`causeForRecClosing` bigint(20) DEFAULT NULL,
`roaming` tinyint(1) DEFAULT NULL,
`servedBSAddress` varchar(255) DEFAULT NULL,
`simType` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `idx_msisdn` (`msisdn`)
) ENGINE=MyISAM AUTO_INCREMENT=2580713 DEFAULT CHARSET=latin1;
UPDATE ACTION a
INNER JOIN subscriberinfo s ON a.subscriberId=s.id
INNER JOIN (SELECT FORMAT(((SUM(dataVolumeDownLink + dataVolumeUpLink))/1048576),2) as val,
msisdn
FROM cdr c
WHERE
c.eventDate>=a.createdon
AND c.eventDate <= a.actionTakenOn) sub on sub.msisdn=s.msisdn
SET a.exceedusage = sub.val
WHERE a.remark='Reason : Data limit crossed'
AND a.exceedusage IS NULL;
I would move the subquery into FROM (actually FROM in select and UPDATE section for the query) to let it be executed just once.
Possibly change the whole thing to a series of JOIN.
This is very much a guess (I am not sure on how your tables hang together, hence not sure on the GROUP BY), but maybe something like this:-
UPDATE ACTION a
INNER JOIN subscriberinfo s ON a.subscriberId=s.id
INNER JOIN cdr c ON c.msisdn=s.msisdn AND c.eventDate BETWEEN a.createdon AND a.actionTakenOn
SET a.exceedusage = FORMAT(((SUM(c.dataVolumeDownLink + c.dataVolumeUpLink))/1048576),2)
WHERE a.remark='Reason : Data limit crossed'
AND a.exceedusage IS NULL
GROUP BY a.subscriberId;
I have a table that was inherited from a different system and one of the fields has the UUID function enabled so no matter what ID I generate and try to insert, the table creates a completely different one automatically.
I would like to use a PHP function to create the ID instead but I can't workout how to remove the UUID function from the field.
I am not sure what information will be needed to help you so please feel free to ask.
The field in question is
id, char(36)
Table definaition is..
CREATE TABLE `users` (
`id` char(36) NOT NULL,
`user_name` varchar(60) default NULL,
`user_hash` varchar(32) default NULL,
`diary_weekly_view` varchar(500) NOT NULL,
`week_start` date NOT NULL,
`diary_monthly_view` varchar(300) NOT NULL,
`diary_view` int(1) NOT NULL default '1',
`account_search` varchar(1000) NOT NULL,
`cases_search` varchar(500) NOT NULL,
`serials_search` varchar(500) NOT NULL,
`type` varchar(8) NOT NULL,
`email` varchar(255) NOT NULL,
`notes` varchar(3000) NOT NULL,
`authenticate_id` varchar(100) default NULL,
`sugar_login` tinyint(1) default '1',
`first_name` varchar(30) default NULL,
`last_name` varchar(30) default NULL,
`reports_to_id` char(36) default NULL,
`is_admin` tinyint(1) default '0',
`receive_notifications` tinyint(1) default '1',
`date_entered` datetime NOT NULL,
`date_modified` datetime NOT NULL,
`modified_user_id` char(36) default NULL,
`created_by` char(36) default NULL,
`phone_office` varchar(50) default NULL,
`phone_mobile` varchar(50) default NULL,
`status` varchar(25) default NULL,
`address_street` varchar(150) default NULL,
`address_city` varchar(100) default NULL,
`address_state` varchar(100) default NULL,
`address_country` varchar(25) default NULL,
`address_postalcode` varchar(9) default NULL,
`user_preferences` text,
`deleted` tinyint(1) NOT NULL default '0',
`portal_only` tinyint(1) default '0',
`employee_status` varchar(25) default NULL,
`is_group` tinyint(1) default '0',
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf
OK, sussed it, I removed the index and then recreated the index with INDEX
I have the following query:
SELECT contact_purl, contact_firstName, contact_lastName, MIN( contact_id ) AS MinID
FROM contacts
WHERE contact_client_id = 1
GROUP BY contact_purl
HAVING COUNT( contact_id ) > 1
The purpose is to find any contacts with a duplicate "contact_purl," and return the first entry.
I'm running into a very strange problem... If the table has less than 8,000 rows, the query will render in less than 1 second. HOWEVER, if the table has more than 8,000 rows, the query will take consistently 338 seconds on average.
Here is the query plan for the table with ~5000 rows:
And for ~8000 rows:
The table...
CREATE TABLE IF NOT EXISTS `contacts` (
`contact_id` int(11) NOT NULL AUTO_INCREMENT,
`contact_client_id` int(11) DEFAULT NULL,
`contact_sales_id` int(11) DEFAULT NULL,
`contact_campaign_id` int(11) DEFAULT NULL,
`contact_purl` varchar(100) NOT NULL,
`contact_purl1` varchar(50) DEFAULT NULL,
`contact_purl2` varchar(50) DEFAULT NULL,
`contact_firstName` varchar(50) NOT NULL,
`contact_lastName` varchar(50) NOT NULL,
`contact_organization` varchar(100) DEFAULT NULL,
`contact_url_organization` varchar(200) DEFAULT NULL,
`contact_position` varchar(50) DEFAULT NULL,
`contact_email` varchar(100) DEFAULT NULL,
`contact_phone` varchar(20) DEFAULT NULL,
`contact_fax` varchar(20) NOT NULL,
`contact_address1` varchar(100) DEFAULT NULL,
`contact_address2` varchar(100) DEFAULT NULL,
`contact_city` varchar(100) DEFAULT NULL,
`contact_state` varchar(20) DEFAULT NULL,
`contact_zip` varchar(10) DEFAULT NULL,
`contact_IP` varchar(50) DEFAULT NULL,
`contact_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`contact_pw` varchar(200) NOT NULL,
`contact_subscribed` varchar(1) NOT NULL DEFAULT 'Y',
`contact_import` varchar(200) DEFAULT NULL,
`contacts_c_1` varchar(500) DEFAULT NULL,
`contacts_c_2` varchar(500) DEFAULT NULL,
`contacts_c_3` varchar(500) DEFAULT NULL,
`contacts_c_4` varchar(500) DEFAULT NULL,
`contacts_c_5` varchar(500) DEFAULT NULL,
`contacts_c_6` varchar(500) DEFAULT NULL,
`contacts_c_7` varchar(500) DEFAULT NULL,
`contacts_c_8` varchar(500) DEFAULT NULL,
`contacts_c_9` varchar(500) DEFAULT NULL,
`contacts_c_10` varchar(500) DEFAULT NULL,
`contacts_c_11` varchar(500) DEFAULT NULL,
`contacts_c_12` varchar(500) DEFAULT NULL,
`contacts_c_13` varchar(500) DEFAULT NULL,
`contacts_c_14` varchar(500) DEFAULT NULL,
`contacts_c_15` varchar(500) DEFAULT NULL,
`contacts_c_16` varchar(500) DEFAULT NULL,
`contacts_c_17` varchar(500) DEFAULT NULL,
`contacts_c_18` varchar(500) DEFAULT NULL,
`contacts_c_19` varchar(500) DEFAULT NULL,
`contacts_c_20` varchar(500) DEFAULT NULL,
`contacts_c_21` varchar(500) DEFAULT NULL,
`contacts_c_22` varchar(500) DEFAULT NULL,
`contacts_c_23` varchar(500) DEFAULT NULL,
`contacts_c_24` varchar(500) DEFAULT NULL,
`contacts_c_25` varchar(500) DEFAULT NULL,
`contacts_c_26` varchar(500) DEFAULT NULL,
`contacts_c_27` varchar(500) DEFAULT NULL,
`contacts_c_28` varchar(500) DEFAULT NULL,
`contacts_c_29` varchar(500) DEFAULT NULL,
`contacts_c_30` varchar(500) DEFAULT NULL,
`contacts_c_31` varchar(500) DEFAULT NULL,
`contacts_c_32` varchar(500) DEFAULT NULL,
`contacts_c_33` varchar(500) DEFAULT NULL,
`contacts_c_34` varchar(500) DEFAULT NULL,
`contacts_c_35` varchar(500) DEFAULT NULL,
`contacts_c_36` varchar(500) DEFAULT NULL,
`contacts_c_37` varchar(500) DEFAULT NULL,
`contacts_c_38` varchar(500) DEFAULT NULL,
`contacts_c_39` varchar(500) DEFAULT NULL,
`contacts_c_40` varchar(500) DEFAULT NULL,
`contacts_c_41` varchar(500) DEFAULT NULL,
`contacts_c_42` varchar(500) DEFAULT NULL,
`contacts_c_43` varchar(500) DEFAULT NULL,
`contacts_c_44` varchar(500) DEFAULT NULL,
`contacts_c_45` varchar(500) DEFAULT NULL,
`contacts_c_46` varchar(500) DEFAULT NULL,
`contacts_c_47` varchar(500) DEFAULT NULL,
`contacts_c_48` varchar(500) DEFAULT NULL,
`contacts_c_49` varchar(500) DEFAULT NULL,
`contacts_c_50` varchar(500) DEFAULT NULL,
`contacts_i_1` varchar(100) DEFAULT NULL,
`contacts_i_2` varchar(100) DEFAULT NULL,
`contacts_i_3` varchar(100) DEFAULT NULL,
`contacts_i_4` varchar(100) DEFAULT NULL,
`contacts_i_5` varchar(100) DEFAULT NULL,
`contacts_i_6` varchar(100) DEFAULT NULL,
`contacts_i_7` varchar(100) DEFAULT NULL,
`contacts_i_8` varchar(100) DEFAULT NULL,
`contacts_i_9` varchar(100) DEFAULT NULL,
`contacts_i_10` varchar(100) DEFAULT NULL,
`contacts_i_11` varchar(100) DEFAULT NULL,
`contacts_i_12` varchar(100) DEFAULT NULL,
`contacts_i_13` varchar(100) DEFAULT NULL,
`contacts_i_14` varchar(100) DEFAULT NULL,
`contacts_i_15` varchar(100) DEFAULT NULL,
PRIMARY KEY (`contact_id`),
KEY `contact_campaign_id` (`contact_campaign_id`),
KEY `contact_client_id` (`contact_client_id`),
KEY `contact_purl2` (`contact_purl2`),
KEY `contact_purl1` (`contact_purl1`),
KEY `contact_purl` (`contact_purl`)
)
I have recently Optimized and Defragmented the table as well.
Any ideas on what would be causing this?
First off, thank you for posting your table structure, query, and EXPLAIN output in your question. I think you're crossing the memory / disk temporary table size boundary, thus the large performance change. If you put a unique index on the contact_purl column, MySQL won't allow duplicates to be inserted. This would make your query unnecessary. Otherwise, I'd create an index on (contact_client_id, contact_purl) so MySQL can figure out what rows you want from the indexes directly. You could also try separating the search for the columns and retrieving them by using a subquery. Something like this maybe:
SELECT contact_purl, contact_firstName, contact_lastName, contact_id
FROM contacts, (SELECT MIN(contact_id) AS MinID
FROM contacts
WHERE contact_client_id = 1
GROUP BY contact_purl
HAVING COUNT( contact_id ) > 1) nodups WHERE nodups.MinID = contacts.contact_id