Validate fields from one table to another in MySQL - mysql

Question

The problem:
I have 1 table of aprox 5000 rows called imported_cities
I have 1 table of aprox 800 000 rows called postal_codes containing postal codes cities
I need to validate each distinct city from imported_cities against the cities in postal codes table based on city name and its province. See tables structure below.
If they match exactly (yes, exactly. The rest of cities are manually validated) I have to update a column on imported_city and
enter both city from imported_cities and city from postal_codes (side by side) into a third table called imported_cities_equiv
What I have tried:
Adding indexes to tables and make query below. It takes forever... :(
explain SELECT DISTINCT ic.destinationCity, pc.city FROM (imported_cities ic, postalcodes pc)
WHERE LOWER(ic.destinationCity) = LOWER(pc.city)
the result
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE ip index NULL company_city 478 NULL 4221 Using index; Using temporary
1 SIMPLE pc index NULL city_prov 160 NULL 765407 Using where; Using index; Using join buffer (Block...
--
-- Table structure for table postalcodes
CREATE TABLE IF NOT EXISTS `postalcodes` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`code` varchar(11) NOT NULL,
`city` varchar(50) NOT NULL,
`province` varchar(50) NOT NULL,
`provinceISO` varchar(2) NOT NULL,
`latitude` decimal(17,13) NOT NULL,
`longitude` decimal(17,13) NOT NULL,
PRIMARY KEY (`id`),
KEY `code` (`code`),
KEY `city_prov` (`city`,`provinceISO`)
--
-- Table structure for table imported_cities
CREATE TABLE IF NOT EXISTS `imported_cities` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`companyName` varchar(30) CHARACTER SET utf8 NOT NULL,
`destinationCity` varchar(128) CHARACTER SET utf8 NOT NULL,
`destinationProvince` varchar(20) CHARACTER SET utf8 NOT NULL,
`equivCity` varchar(128) CHARACTER SET utf8 DEFAULT NULL,
`minAmount` decimal(6,2) NOT NULL
PRIMARY KEY (`id`),
KEY `company_city` (`companyName`,`destinationCity`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=7933 ;
--
-- Table structure for table imported_cities_equiv
CREATE TABLE IF NOT EXISTS `imported_cities_equiv` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`imported_city` varchar(128) CHARACTER SET utf8 NOT NULL,
`pc_city` varchar(128) CHARACTER SET utf8 NOT NULL,
`province` varchar(20) CHARACTER SET utf8 NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=149 ;
Any help or suggestion is appreciated. Thank you.

Answer 1

The query you want to get your information is:
SELECT ip.*, (pc.city is not null) as exact match
FROM imported_prices ip left join
postalcodes pc
on LOWER(ip.destinationCity) = LOWER(pc.city) and
lower(ip.province) = lower(pc.province);
However, this will have really bad performance. Getting rid of the lower() would help:
SELECT ip.*, (pc.city is not null) as exact match
FROM imported_prices ip left join
postalcodes pc
on(ip.destinationCity) =(pc.city) and
(ip.province) = (pc.province);
Because then you can add an index on postalcodes(city, province).
If you cannot use remove lower(), then alter the table to add new columns and put the lower-case values in those columns. Then build an index on the new columns and use them in the join.

Answer 2

Thank you all for pointing me on the right direction.
Some changes have been made following your advices:
added indexes on imported_cities table on destinationCity and destinationProvince columns
added indexes on postalcodes table on city and provinceISO columns
JOIN clause have only one side upper since the field ic.destinationCity is already in uppercase
limit query by province on WHERE for performance
The final SQL is:
SELECT DISTINCT pc.city, pc.provinceISO
FROM postalcodes pc
LEFT JOIN imported_cities ic
ON upper(pc.city) = ic.destinationCity AND
pc.provinceISO = ic.destinationProvince
WHERE ic.destinationProvince = 'QC';
AND the EXPLAIN
id select_type table type possible_keys key key_len ref rows Extra
1 SIMPLE pc ref province province 8 const 278115 Using index condition; Using temporary
1 SIMPLE ip ref destinationCity,destinationProvince destinationCity 386 func 1 Using index condition; Using where; Distinct
Going forward I can now construct the INSERT query on PHP and make one INSERT query to insert all equivalent cities on the 3rd table. Thank you all.