Never ending MySQL query during data import - mysql

Question

I'm working on a data import routine from a set of CSV files into my main database and am stuck with this particular set of data. I've used LOAD DATA LOCAL INFILE to dump the CSV data into my table, feed_hcp_leasenote:
CREATE TABLE `feed_hcp_leasenote` (
`BLDGID` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`LEASID` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`NOTEDATE` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`REF1` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`REF2` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`LASTDATE` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`USERID` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`NOTETEXT` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`tempid` int(11) NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`tempid`),
KEY `BLDGID` (`BLDGID`),
KEY `LEASID` (`LEASID`),
KEY `REF1` (`REF1`),
KEY `NOTEDATE` (`NOTEDATE`)
) ENGINE=MyISAM AUTO_INCREMENT=65002 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
I'm trying to import this data into two tables, lease_notes and customfield_data. lease_notes only stores a unique ID value, the note itself, and the lid which links it to the lease table. customfield_data stores a variety of data for system- and user-created fields, with each record linked to another table via the linkid field. Here's the lease_note table:
CREATE TABLE `lease_notes` (
`lnid` int(11) NOT NULL AUTO_INCREMENT,
`notetext` longtext COLLATE utf8_unicode_ci NOT NULL,
`lid` int(11) NOT NULL COMMENT 'Lease ID',
PRIMARY KEY (`lnid`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
And the customfield_data table:
CREATE TABLE `customfield_data` (
`cfdid` int(11) NOT NULL AUTO_INCREMENT,
`data_int` int(11) DEFAULT NULL,
`data_date` datetime DEFAULT NULL,
`data_smtext` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`data_lgtext` longtext COLLATE utf8_unicode_ci,
`data_numeric` decimal(20,2) DEFAULT NULL,
`linkid` int(11) DEFAULT NULL COMMENT 'ID value of specific item',
`cfid` int(11) NOT NULL COMMENT 'Custom field ID',
PRIMARY KEY (`cfdid`),
KEY `data_smtext` (`data_smtext`(333)),
KEY `linkid` (`linkid`),
KEY `cfid` (`cfid`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
The query that is getting stuck is as follows:
SELECT NOTEDATE, REF1, REF2, LASTDATE, USERID, feed_hcp_leasenote.NOTETEXT, leases.lid, lease_notes.lnid
FROM feed_hcp_leasenote
JOIN customfield_data mrileaseid ON feed_hcp_leasenote.LEASID = mrileaseid.data_smtext AND mrileaseid.cfid = ?
JOIN leases ON mrileaseid.linkid = leases.lid
JOIN suites ON leases.sid = suites.sid
JOIN floors ON suites.fid = floors.fid
JOIN customfield_data coid ON floors.bid = coid.linkid AND coid.cfid = ? AND coid.data_smtext = feed_hcp_leasenote.BLDGID
JOIN customfield_data status ON leases.lid = status.linkid AND status.cfid = ? AND status.data_smtext <> ?
LEFT JOIN lease_notes ON leases.lid = lease_notes.lid
LEFT JOIN customfield_data notedate ON lease_notes.lnid = notedate.linkid AND notedate.data_date = feed_hcp_leasenote.NOTEDATE AND notedate.cfid = ?
LEFT JOIN customfield_data ref1 ON lease_notes.lnid = ref1.linkid AND ref1.data_smtext = feed_hcp_leasenote.REF1 AND ref1.cfid = ?
My goal with this is to return all records in feed_hcp_leasenote and, depending on whether or not lease_notes.lnid is null, insert or update the records as needed (nulls would be inserts, not nulls would be updates.) The problem is that the provided data uses a combination of 4 fields to determine uniqueness: BLDGID, LEASID, NOTEDATE, and REF1. A note will not exist without a proper BLDGID and LEASID (translated in my query to a valid lid. It can match an existing record with a valid lid, NOTEDATE, and REF1, but if those don't match then I can assume it's a new record.
If I chop off all of the LEFT JOINs and the lease_notes.lnid from the SELECT, it executes properly and gives me all records. Since I couldn't get my original query to work I played with the idea of cycling all results and performing another SELECT to see if the notedate and ref1 matched. If not, I INSERTed, otherwise UPDATE. While this approach works it can only process about 20 records per second which is a problem when I'm dealing with 30,000 at a crack.
Since I got asked about it in a previous question, here's an EXPLAIN of my query:
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE status ref data_smtext,linkid,cfid cfid 4 const 934 Using where
1 SIMPLE mrileaseid ref data_smtext,linkid,cfid linkid 5 rl_hpsi.status.linkid 19 Using where
1 SIMPLE leases eq_ref PRIMARY,sid PRIMARY 4 rl_hpsi.mrileaseid.linkid 1 Using where
1 SIMPLE suites eq_ref PRIMARY,fid PRIMARY 4 rl_hpsi.leases.sid 1
1 SIMPLE floors eq_ref PRIMARY,bid PRIMARY 4 rl_hpsi.suites.fid 1
1 SIMPLE feed_hcp_leasenote ref BLDGID,LEASID LEASID 153 rl_hpsi.mrileaseid.data_smtext 19 Using where
1 SIMPLE coid ref data_smtext,linkid,cfid data_smtext 1002 rl_hpsi.feed_hcp_leasenote.BLDGID 10 Using where
1 SIMPLE lease_notes ALL NULL NULL NULL NULL 15000
1 SIMPLE notedate ref linkid,cfid linkid 5 rl_hpsi.lease_notes.lnid 24
1 SIMPLE ref1 ref data_smtext,linkid,cfid data_smtext 1002 rl_hpsi.feed_hcp_leasenote.REF1 10
Can anyone point me in the right direction? Thanks!

Answer 1

From our comments:
The answer is to add the columns that make an entry unique to your destination table and create a compound unique key on them. Then when inserting to that table use INSERT ON DUPLICATE KEY UPDATE to prevent duplicate data. When the insert is complete you can drop those columns if they are no longer necessary, to prevent storing data in multiple tables.