MySQL: Optimizing JOINs to find non-matching records - mysql

We have a host management system (let's call it CMDB), and a DNS system, each using different tables. The former syncs to the latter, but manual changes cause them to get out of sync. I would like to craft a query to find aliases in CMDB that do NOT have a matching entry in DNS (either no entry, or the name/IP is different)
Because of the large size of the tables, and the need for this query to run frequently, optimizing the query is very important.
Here's what the tables look like:
cmdb_record: id, ipaddr
cmdb_alias: record_id, host_alias
dns_entry: name, ipaddr
cmdb_alias.record_id is a foreign key from cmdb_record.id, so that one IP address can have multiple aliases.
So far, here's what I've come up with:
SELECT cmdb_alias.host_alias, cmdb_record.ipaddr
FROM cmdb_record
INNER JOIN cmdb_alias ON cmdb_alias.record_id = cmdb_record.id
LEFT JOIN dns_entry
ON dns_entry.ipaddr = cmdb_record.ipaddr
AND dns_entry.name = cmdb_alias.host_alias
WHERE dns_entry.ipaddr IS NULL OR dns_entry.name IS NULL
This seems to work, but takes a very long time to run. Is there a better way to do this? Thanks!
EDIT: As requested, here are the SHOW CREATE TABLEs. There are lots of extra fields that aren't particularly relevant, but included for completeness.
Create Table: CREATE TABLE `cmdb_record` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`ip_version` int(11) DEFAULT NULL,
`ipaddr` varchar(40) DEFAULT NULL,
`ipaddr_numeric` decimal(40,0) DEFAULT NULL,
`block_id` int(11) NOT NULL,
`record_commented` tinyint(1) NOT NULL,
`mod_time` datetime NOT NULL,
`deleted` tinyint(1) NOT NULL,
`deleted_date` datetime DEFAULT NULL,
`record_owner` varchar(50) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `ipaddr` (`ipaddr`),
KEY `cmdb_record_fe30f0f7` (`ipaddr`),
KEY `cmdb_record_2b8b575` (`ipaddr_numeric`),
KEY `cmdb_record_45897ef2` (`block_id`),
CONSTRAINT `block_id_refs_id_ed6ed320` FOREIGN KEY (`block_id`) REFERENCES `cmdb_block` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=104427 DEFAULT CHARSET=latin1
Create Table: CREATE TABLE `cmdb_alias` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`host_alias` varchar(255) COLLATE latin1_general_cs NOT NULL,
`record_id` int(11) NOT NULL,
`record_order` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `cmdb_alias_fcffc3bb` (`record_id`),
KEY `alias_lookup` (`host_alias`),
CONSTRAINT `record_id_refs_id_8169fc71` FOREIGN KEY (`record_id`) REFERENCES `cmdb_record` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=155433 DEFAULT CHARSET=latin1 COLLATE=latin1_general_cs
Create Table: CREATE TABLE `dns_entry` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`rec_grp_id` varchar(40) NOT NULL,
`parent_id` int(11) NOT NULL,
`domain_id` int(11) DEFAULT NULL,
`name` varchar(255) DEFAULT NULL,
`type` varchar(6) DEFAULT NULL,
`ipaddr` varchar(255) DEFAULT NULL,
`ttl` int(11) DEFAULT NULL,
`prio` int(11) DEFAULT NULL,
`status` varchar(20) NOT NULL,
`op` varchar(20) NOT NULL,
`mod_time` datetime NOT NULL,
`whodunit` varchar(50) NOT NULL,
`comments` longtext NOT NULL,
PRIMARY KEY (`id`),
KEY `dns_entry_a2431ea` (`domain_id`),
KEY `dns_entry_52094d6e` (`name`)
) ENGINE=InnoDB AUTO_INCREMENT=49437 DEFAULT CHARSET=utf8

If you don't have one already, create an index on dns_entry(ipaddr, name). This might be all you need to speed the query.

Related

MYSQL INNER JOIN is slow with index

this is my simple inner join:
SELECT
SUM(ASSNZ.assenzeDidattiche) AS TotaleAssenze,
SUM(ASSNZ.ore) AS totale_parziale,
FLOOR(((SUM(ASSNZ.assenzeDidattiche) / SUM(ASSNZ.ore)) * 100)) AS andamento,
MAX(ASSNZ.dataLezione) AS ultima_lezione,
ASSNZ.idServizio,
ASSNZ.idUtente
FROM
ciac_corsi_assenze AS ASSNZ
INNER JOIN
ciac_serviziAcquistati_ITA AS ACQ
ON ACQ.idContatto = ASSNZ.idUtente
AND ACQ.idServizio = ASSNZ.idServizio
AND ACQ.stato_allievo <> 'ritirato'
GROUP BY
ASSNZ.idServizio,
ASSNZ.idUtente
table "ASSNZ" has 213886 rows with index "idUtente", "idServizio"
table "ACQ" has 8950 rows with index "idContatto", "idServizio"
ASSNZ table:
CREATE TABLE `ciac_corsi_assenze` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`idUtente` int(11) DEFAULT NULL,
`idServizio` int(11) DEFAULT NULL,
`idCorso` int(11) DEFAULT NULL,
`idCalendario` int(11) DEFAULT NULL,
`modalita` varchar(255) DEFAULT NULL,
`ore` int(11) DEFAULT NULL,
`assenzeDidattiche` float DEFAULT NULL,
`assenzeAmministrative` float DEFAULT NULL,
`dataLezione` date DEFAULT NULL,
`ora_inizio` varchar(8) DEFAULT NULL,
`ora_fine` varchar(8) DEFAULT NULL,
`dataFineStage` date DEFAULT NULL,
`giustificata` varchar(128) DEFAULT NULL,
`motivazione` longtext,
`grouped` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `idUtente` (`idUtente`) USING BTREE,
KEY `idServizio` (`idServizio`) USING BTREE,
KEY `dataLezione` (`dataLezione`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=574582 DEFAULT CHARSET=utf8;
ACQ table:
CREATE TABLE `ciac_serviziacquistati_ita` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`idServizio` int(11) NOT NULL,
`idContatto` int(11) NOT NULL,
`idAzienda` int(11) NOT NULL,
`idSede` int(11) NOT NULL,
`tipoPersona` int(11) NOT NULL,
`num_registro` int(11) NOT NULL,
`codice` varchar(255) CHARACTER SET latin1 DEFAULT NULL,
`dal` date NOT NULL,
`al` date NOT NULL,
`ore` int(11) NOT NULL,
`costoOrario` decimal(10,0) NOT NULL,
`annoFormativo` varchar(128) CHARACTER SET latin1 NOT NULL,
`stato_attuale` int(11) NOT NULL,
`datore_attuale` int(11) NOT NULL,
`stato_allievo` varchar(64) CHARACTER SET latin1 NOT NULL DEFAULT 'corsista',
`data_ritiro` date DEFAULT NULL,
`crediti_formativi` int(11) NOT NULL,
`note` longtext CHARACTER SET latin1 NOT NULL,
`valore_economico` decimal(10,2) NOT NULL,
`dataInserimento` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `idServizio` (`idServizio`) USING BTREE,
KEY `idAzienda` (`idAzienda`) USING BTREE,
KEY `idContatto` (`idContatto`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=9542 DEFAULT CHARSET=utf8;
this is my EXPLAIN of the select
Now because the query is slow, during 1.5s / 2.0s??
Something wrong?
UPDATE
added new index (with the John Bollinger's answer) to the table ciac_corsi_assenze:
PRIMARY KEY (`id`),
KEY `dataLezione` (`dataLezione`) USING BTREE,
KEY `test` (`idUtente`,`idServizio`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=574582 DEFAULT CHARSET=utf8;
added new index to the table ciac_serviziAcquistati_ITA:
PRIMARY KEY (`id`),
KEY `idAzienda` (`idAzienda`) USING BTREE,
KEY `test2` (`idContatto`,`idServizio`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=9542 DEFAULT CHARSET=utf8;
New EXPLAIN:
But it's always slow :(
Your tables have separate indexes on various columns of interest, but MySQL will use at most one index per table to perform your query. This particular query would probably be sped by table ciac_corsi_assenze having an index on (idUtente, idServizio) (and such an index would supersede the existing one on (idUtente) alone). That should allow MySQL to avoid sorting the result rows to perform the grouping, and it will help more in performing the join than any of the existing indexes do.
The query would probably be sped further by table ciac_serviziAcquistati_ITA having an index on (idContatto, idServizio), or even on (idContatto, idServizio, ritirato). Either of those would supersede the existing index on just (idContatto).
John went the right direction. However the order of columns in the composite index needs changing.
For the GROUP BY, this order is needed (on ASSNZ):
INDEX(idServizio, idUtente)
(and that should replace KEY(idServizio), but not KEY(idUtente))
Then ACQ needs, in this order:
INDEX(idContatto, idServizio, stato_allievo)
replacing only KEY(idContatto).

How to add foreign key in table using existing column, without losing data?

I have a table in which I need to add foreign key on an existing column. Following is create table:
CREATE TABLE `itemtx` (
`itemTxid` int(11) NOT NULL AUTO_INCREMENT,
`itemcode` varchar(1) NOT NULL,
`weight` decimal(7,3) DEFAULT NULL,
`txtype` varchar(10) DEFAULT 'Pickup',
`tripstopid` int(11) NOT NULL,
`barcode` varchar(25) DEFAULT NULL,
`bagcount` int(11) DEFAULT '1',
PRIMARY KEY (`itemTxid`)
) ENGINE=InnoDB AUTO_INCREMENT=33524 DEFAULT CHARSET=latin1
I need to add foreign key on tripstopid column. I cannot drop or empty table as it contains data. Following is the referenced table:
CREATE TABLE `tripstop` (
`tripstopid` int(11) NOT NULL AUTO_INCREMENT,
`tripid` int(11) DEFAULT NULL,
`locationName` varchar(100) DEFAULT NULL,
`userName` varchar(100) DEFAULT NULL,
`locationid` int(11) DEFAULT NULL,
`datetime` varchar(20) DEFAULT NULL,
`createts` varchar(20) DEFAULT NULL,
`latitude` double DEFAULT NULL,
`longitude` double DEFAULT NULL,
`userid` int(11) DEFAULT NULL,
`tid` int(11) DEFAULT NULL,
PRIMARY KEY (`tripstopid`)
) ENGINE=InnoDB AUTO_INCREMENT=4691 DEFAULT CHARSET=latin1
How can I do this without losing my data?
You can achieve this by following:
ALTER TABLE itemtx
ADD FOREIGN KEY (tripstopid) REFERENCES tripstop(tripstopid);
Verified by creating tables, inserting data in them and then updating table for foreign key, previously entered data is not lost.

MySQL Error Number 150 when creating Table with Foreign Key

I am having an issue creating a new table in my database. I've seen that the error code it is returning is to do with Foreign Key constraints.
I checked to ensure that the data type of the foreign key in the new table matched the data type of the primary key in the other table. They are both int(11).
However I am still getting an error. Am I missing something? This is my SQL script for creating the new table:
CREATE TABLE `regular_features` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`name` VARCHAR(200) DEFAULT NULL,
`day` VARCHAR(200) DEFAULT NULL,
`description` TEXT DEFAULT NULL,
`programme_id` INT(11) NOT NULL,
PRIMARY KEY (`id`),
FOREIGN KEY (`programme_id`) REFERENCES directoryprogramme(id)
) ENGINE=INNODB DEFAULT CHARSET=utf8;
This is the original table containing the primary key:
CREATE TABLE `directoryprogramme` (
`id` INT(11) NOT NULL AUTO_INCREMENT,
`name` VARCHAR(250) NOT NULL,
`broadcast_time` VARCHAR(100) NOT NULL,
`description` TEXT NOT NULL,
`days` VARCHAR(150) NOT NULL,
`contributors` VARCHAR(250) NOT NULL,
`directorycompany_id` INT(11) NOT NULL,
`directorycontact_id` VARCHAR(250) NOT NULL,
`facebook_link` VARCHAR(250) DEFAULT NULL,
`twitter_link` VARCHAR(250) DEFAULT NULL,
`wikipedia_link` VARCHAR(250) DEFAULT NULL,
`web` VARCHAR(250) DEFAULT NULL,
`imageextension` VARCHAR(10) DEFAULT NULL,
`type` VARCHAR(20) NOT NULL DEFAULT 'other',
PRIMARY KEY (`id`)
) ENGINE=MYISAM AUTO_INCREMENT=1161 DEFAULT CHARSET=utf8;
The Foreign Key will be the id of directoryprogramme
The problem is the last line of your create statement:
ENGINE=INNODB DEFAULT CHARSET=utf8;
You mix MYISAM in ald table with INNODB in your new table.
That doesn't work.
Chnage the engine in your new table to MYISAM and it works.

cannot add second FK : got error can't create table'.jobstatus\#sql-32c_12f2f.frm' (errno:150)

CREATE TABLE `job` (
`jobId` int(11) NOT NULL auto_increment,
`jobcode` varchar(25) default NULL,
`jobname` varchar(255) default NULL,
`location` varchar(255) default NULL,
`budget` int(10) unsigned default NULL,
`year_type` varchar(100) default NULL,
`worklineId` int(11) default NULL,
PRIMARY KEY (`jobId`),
KEY `NewIndex` (`worklineId`),
FOREIGN KEY (`worklineId`) REFERENCES `workline` (`worklineId`)
) TYPE=InnoDB;
CREATE TABLE `subjob` (
`subjobId` int(11) NOT NULL auto_increment,
`subjobcode` varchar(25) default NULL,
`subjobname` varchar(255) default NULL,
`subjobbudget` int(11) unsigned default NULL,
`jobgoal_date` date default '0000-00-00',
`jobId` int(11) default NULL,
PRIMARY KEY (`subjobId`),
KEY `NewIndex` (`jobId`),
FOREIGN KEY (`jobId`) REFERENCES `job` (`jobId`)
) TYPE=InnoDB;
CREATE TABLE `contract` (
`contractId` int(11) NOT NULL auto_increment,
`contractcode` varchar(25) default NULL,
`price` int(11) unsigned default NULL,
`contractprice` int(11) unsigned default NULL,
`company` varchar(50) default NULL,
`signdate` date default '0000-00-00',
`begindate` date default '0000-00-00',
`enddateplan` date default '0000-00-00',
`note` text,
PRIMARY KEY (`contractId`)
) TYPE=InnoDB;
CREATE TABLE `subjob_contract` (
`subjobcontractId` int(11) NOT NULL auto_increment,
`status` varchar(11) default NULL,
`contractId` int(11) default NULL,
`subjobId` int(11) default NULL,
PRIMARY KEY (`subjobcontractId`),
KEY `NewIndex` (`contractId`),
KEY `NewIndex2` (`subjobId`),
FOREIGN KEY (`contractId`) REFERENCES `contract` (`contractId`)
) TYPE=InnoDB
I m using mysql front 3.2 to manage database,I can add first fk but when i add second fk i got an error following this :
sql execution error #1005. response from the database: can't create table'.jobstatus#sql-32c_12f2f.frm' (errno:150). i already define the new index for fk subjobId reference to subjob table what could be the possibility of this error? thank you
Check the datatype and size of the subjobId column on primary table and referenced table. both must be same than it will allow you to create foreign key.
Answer is: You can not refer that column/table which is not created yet. Try to execute tables having foreign keys after the referenced tables.
Obviously you should have consistency in datatypes of foreign key and referenced column as well
Correct Execution Demo. Also You should use Engine=InnoDB instead of Type=InnoDB

Mysql Table Key

Can anyone tell me what is the meaning of KEY fk_pickup_method (pickup_method_id),
KEY fk_deliv_method (delivery_method_id) lines. As pickup_method and deliv_method are not the tables. So what is the use of these lines.
CREATE TABLE `test` (
`idTest` int(11) NOT NULL AUTO_INCREMENT,
`Name` mediumtext NOT NULL,
`email` varchar(45) NOT NULL,
`password` varchar(45) NOT NULL,
`firstname` varchar(45) NOT NULL,
`lastname` varchar(45) NOT NULL,
`phone` bigint(20) unsigned NOT NULL,
`address_street` varchar(128) NOT NULL,
`address_apt` varchar(45) DEFAULT NULL,
`address_city` varchar(128) NOT NULL,
`address_state` varchar(2) NOT NULL,
`address_zip` int(11) NOT NULL,
`fax` bigint(20) unsigned DEFAULT NULL,
`account_balance` float NOT NULL DEFAULT '0',
`delivery_radius` float DEFAULT NULL,
`pickup_method_id` int(11) NOT NULL DEFAULT '0',
`delivery_method_id` int(11) NOT NULL,
`is_deleted` tinyint(1) NOT NULL DEFAULT '0',
`lat` float NOT NULL DEFAULT '0',
`lng` float NOT NULL DEFAULT '0',
`timezone` varchar(45) NOT NULL,
PRIMARY KEY (`idTest`),
UNIQUE KEY `phone_UNIQUE` (`phone`),
KEY `fk_pickup_method` (`pickup_method_id`),
KEY `fk_deliv_method` (`delivery_method_id`)
)
They're INDEXes on the columns in the ()'s. But the fields are not constrained to being UNIQUE.
Look for {INDEX|KEY} in this MySQL document link.
Line
KEY fk_pickup_method (pickup_method_id)
defines an index named fk_pickup_method on table column pickup_method_id.
When you run EXPLAIN, under possible_keys column you'll see the name of the index.
The usual practice is to call the key as the column it indexes. That's the default behaviour if you don't specify the key name.
These are indexes on the fields pickup_method_id and delivery_method_id.
How mysql uses indexes : http://dev.mysql.com/doc/refman/5.0/en/mysql-indexes.html
The name fk_ points to the intention of the creator to create a foreign key. But this is not a foreign key, and won't care about referential integrity for you.
Foreign keys in mysql (innodb) : http://dev.mysql.com/doc/refman/5.5/en/innodb-foreign-key-constraints.html
My guess is that these were intended to be FOREIGN KEY constraints and reference other tables.
InnoDB engine creates an index (if there isn't one) when a FOREIGN KEY constraint is defined.
MyISAM engine ignores FOREIGN KEY constraints but it still creates the index. Example:
CREATE TABLE test
( test_id int NOT NULL AUTO_INCREMENT,
delivery_method_id int NOT NULL,
PRIMARY KEY (test_id),
FOREIGN KEY fk_deliv_method (delivery_method_id)
REFERENCES delivery_method(delivery_method_id)
) ENGINE=MyISAM
DEFAULT CHARSET=utf8 ;
And then:
SHOW CREATE TABLE test ;
CREATE TABLE `test` (
`test_id` int(11) NOT NULL AUTO_INCREMENT,
`delivery_method_id` int(11) NOT NULL,
PRIMARY KEY (`test_id`),
KEY `fk_deliv_method` (`delivery_method_id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8