LOAD DATA FROM S3 - Fastest way for big data and indexes - mysql

Question

I have a table that consists of holiday deals, so just to give you an idea, each row will contain the following bits of data:
Departure airport
Arrival airport
Start date
Duration
Hotel destination
Resort
Hotel name
Hotel rating
A few tiny integer columns for 1s and 0s.
Price
Date time the row was updated
Now, all of these deals get packaged up from 3 tables, they are flights, accommodation and transfers, the packaging up is to find the cheapest deal per variation such as, per departure airport, duration, board basis etc.
The table I am importing into will consist of around about 50 million rows, the import is extremely slow.
I have removed the indexes, that made a massive difference but now when I re-add the indexes back to the table after all data is in there it takes forever to complete.
I would like to know is there a way of bulk loading data quickly or is there a quicker way of adding indexes back to the table after data has been added?
Create Table
```
CREATE TABLE `iv_deals` (
`aid` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'Deal Autonumber PK',
`startdate` DATE NULL DEFAULT NULL COMMENT 'Holiday Start Date',
`startdatet` TINYINT(2) NOT NULL DEFAULT '0',
`depairport` CHAR(3) NULL DEFAULT NULL COMMENT 'Departure Airport IATA Code',
`arrairport` CHAR(3) NULL DEFAULT NULL COMMENT 'Arrival Airport IATA Code',
`destination` VARCHAR(30) NULL DEFAULT NULL COMMENT 'Holiday Destination',
`resort` VARCHAR(30) NULL DEFAULT NULL COMMENT 'Holiday Resort',
`hotel` VARCHAR(50) NULL DEFAULT NULL COMMENT 'Holiday Property Name',
`iv_PropertyID` INT(11) UNSIGNED NOT NULL DEFAULT '0' COMMENT 'Holiday Property ID',
`rating` VARCHAR(2) NULL DEFAULT NULL COMMENT 'Holiday Property Star Rating',
`board` VARCHAR(10) NULL DEFAULT NULL COMMENT 'Holiday Meal Option',
`duration` TINYINT(2) UNSIGNED NULL DEFAULT '0' COMMENT 'Holiday Duration',
`2for1` TINYINT(1) UNSIGNED NULL DEFAULT '0' COMMENT 'Is 2nd Week FREE Offer, 0 = False, 1 = True',
`3for2` TINYINT(1) UNSIGNED NULL DEFAULT '0' COMMENT 'Is 3rd Week FREE Offer, 0 = False, 1 = True',
`3and4` TINYINT(1) UNSIGNED NULL DEFAULT '0' COMMENT 'Is 3rd and 4th Week FREE Offer, 0 = False, 1 = True',
`4for3` TINYINT(1) UNSIGNED NULL DEFAULT '0' COMMENT 'Is 4th Week FREE Offer, 0 = False, 1 = True',
`freebb` VARCHAR(2) NULL DEFAULT NULL COMMENT 'Free Week Meal Option',
`adults` TINYINT(1) UNSIGNED NULL DEFAULT '0' COMMENT 'Number of Adults',
`children` TINYINT(1) UNSIGNED NULL DEFAULT '0' COMMENT 'Number of Children',
`infants` TINYINT(1) UNSIGNED NULL DEFAULT '0' COMMENT 'Number of Infants',
`price` SMALLINT(4) UNSIGNED NULL DEFAULT '9999' COMMENT 'Price',
`carrier` VARCHAR(40) NULL DEFAULT NULL COMMENT 'Flight Carrier IATA Code',
`DateUpdated` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`aid`, `startdatet`),
UNIQUE INDEX `Unique` (`startdate`, `depairport`, `arrairport`, `iv_PropertyID`, `board`, `duration`, `adults`, `children`, `startdatet`),
INDEX `ik_Price` (`price`),
INDEX `ik_Destination` (`destination`),
INDEX `ik_Resort` (`resort`),
INDEX `ik_DepAirport` (`depairport`),
INDEX `ik_Startdate` (`startdate`),
INDEX `ik_Board` (`board`),
INDEX `ik_FILTER_ALL` (`price`, `depairport`, `destination`, `resort`, `board`, `startdate`),
INDEX `iv_PropertyID` (`iv_PropertyID`),
INDEX `ik_Duration` (`duration`),
INDEX `rating` (`rating`),
INDEX `adults` (`adults`),
INDEX `DirectFromPrice` (`iv_PropertyID`, `depairport`, `arrairport`, `board`, `duration`, `adults`, `children`, `startdate`),
INDEX `DirectFromPrice_wo_depairport` (`iv_PropertyID`, `arrairport`, `board`, `duration`, `adults`, `children`),
INDEX `DirectFromPrice_w_pid_dep` (`iv_PropertyID`, `depairport`, `adults`, `children`, `price`),
INDEX `DirectFromPrice_w_pid_night` (`iv_PropertyID`, `duration`, `adults`, `children`),
INDEX `DirectFromPrice_Dur_Board` (`iv_PropertyID`, `duration`, `board`, `adults`, `children`),
INDEX `join_index` (`destination`, `startdate`, `duration`)
)
COLLATE='utf8_general_ci'
AUTO_INCREMENT=1258378560
/*!50100 PARTITION BY LIST (startdatet)
(PARTITION part0 VALUES IN (1) ENGINE = InnoDB,
PARTITION part1 VALUES IN (2) ENGINE = InnoDB,
PARTITION part2 VALUES IN (3) ENGINE = InnoDB,
PARTITION part3 VALUES IN (4) ENGINE = InnoDB,
PARTITION part4 VALUES IN (5) ENGINE = InnoDB,
PARTITION part5 VALUES IN (6) ENGINE = InnoDB,
PARTITION part6 VALUES IN (7) ENGINE = InnoDB,
PARTITION part7 VALUES IN (8) ENGINE = InnoDB,
PARTITION part8 VALUES IN (9) ENGINE = InnoDB,
PARTITION part9 VALUES IN (10) ENGINE = InnoDB,
PARTITION part10 VALUES IN (11) ENGINE = InnoDB,
PARTITION part11 VALUES IN (12) ENGINE = InnoDB,
PARTITION part12 VALUES IN (0) ENGINE = InnoDB) */;
```

Answer 1

If there are 50M rows, but AUTO_INCREMENT=1258378560, let point out another problem that is looming. (It may be related to the slow load.)
`aid` INT(11) UNSIGNED NOT NULL AUTO_INCREMENT
allows only 4 billion; you are already at 1.2 billion. Do a little math to estimate when you will run out of ids. The brute force solution is to change to BIGINT, but let's analyze why the ids are being 'burned'. There are several ways that INSERT/REPLACE/etc can throw away ids. Please describe how the import is working. REPLACE is perhaps the worst -- it burns ids and is effectively DELETE + INSERT. Other techniques are faster.
(I will now ramble in many directions...)
The partitioning by month (which I assume you are doing with (startdatet) probably does not add any performance. What has your experience been? (I usually argue against using PARTITION except for the few use cases where there are benefits. I see no benefit in your case.)
19 indexes means 19 BTrees that must be updated. The 2 uniques ones must be checked before the INSERT is finished; the 17 others can be delayed, but not forever. (The details are discussed under "change buffer".)
How much RAM? What is the setting of innodb_buffer_pool_size? It should be about 70% of RAM. The Change buffer is a portion of that.
I see at least 4 indexes that can be dropped, since other indexes handle their need. In general, if you have INDEX(a, b), you don't also need INDEX(a). (Shrinking from 19 indexes to 15 will help some.)
Flags and other things of low cardinality are virtually useless by them selves as indexes. The Optimizer will decide that it is cheaper to scan the table than to bounce between the index's BTree and the data BTree. I'm thinking of INDEX(rating).
Any SELECT that does not have startdatet in the WHERE is likely to be slower than without partitioning. This is because the query must check all 13 partitions. Even with AND startdatet = 4, performance won't be any better than if there had been an index that included startdatet.
Let me discuss any index starting with a column (perhaps price, rating, startdate) that is queried as a "range" (eg, WHERE price BETWEEN ...). The processing cannot use any columns after that column. I suspect ik_FILTER_ALL will scan a big chunk of the index, since it only filtered on price. Rearrange the columns. Based on the name, I am guessing this is a "covering" index. That is, a common query references only those 6 columns? Note: SELECT * ... references more than just those 6, so the index is not "covering". (Show us the query; I can discuss it more.)
The 5 "DirectFromPrice" indexes are probably each 'perfect' for some query. But they are awfully long (lots of columns). I would guess that 2 shorter lists would come close to handling the 5 cases "well enough". (Remember, decreasing the number of indexes will help in the goal of insert time.)
What version of MySQL/MariaDB are you using?
The main action item at this point: Show us the import. (I will discuss sorting the input after seeing the method being used.)