Mysql import of non-csv data with field identifiers - mysql

Question

I'm trying to import a non-csv data file into MySQL.
1) The data fields are newline delimited, and the field identifier is at the start of each line.
2) Some fields have multiple entries
3) Not every record has every field populated
4) Some blank lines exist inside fields and need to be filtered out
5) Records are generally delimited by blank lines, but also by "Number X"
Here's an example of the file showing an example of three records as they appear
Number 1
ARTIST BOOM JEFF=SINGER
BACKING MUSICIANS=BAND
COMP BOOM JEFF
DATE 1980
TIME 3.23
FIELD3 FRONT ROW
NOTE LIVE RECORDING
Number 2
ARTIST JOHN LEE=VOCAL
COMP JOHN LEE
TIME 4.20
ID 000000230682
PUBLISHER BLAHBLAH
FIELD3 DAY I RODE THE TRAIN
Number 3
ARTIST BURT DAN=NARRATOR
JOHNS RY=DRUMS
STUDIO BAND=ORCHESTRA
FREE DAN=DIRECTOR
COMP JOHNS RY
DATE 1934
DUR 2.32
ID 000055332
PUBLISHER WEEWAH
SHELF 86000002
FIELD3 EVE OF THE WAR
NOTE FROM HE NARRATION "NO MORE THAT IN
THE FIRST YEARS OF THE SEVENTEENTH CENTURY .."
What's the best way to import this data into MySQL?
Can LOAD DATA INFILE be used to read it in? Or should I write a script to strip the field identifiers and convert it to csv format which can then be read in using LOAD DATA INFILE?

Answer 1

I would rather use sed to convert those to INSERT .. SET ... statements like:
INSERT INTO RECORDS SET
ARTIST="BOOM JEFF=SINGER~BACKING MUSICIANS=BAND" ,
COMP="BOOM JEFF" ,
DATE="1980" ,
TIME="3.23" ,
FIELD3="FRONT ROW" ,
NOTE="LIVE RECORDING"
replacing in-record newlines with ~ for example and after that analyse data with the help of SQL.

Answer 2

From what I see, your best bet is a script that would parse the data line by line with a script similar to the following (using php):
$lines=explode("\n",file_get_contents('file.name'));
$record=null;
//go through all the lines
foreach($lines as $line) {
//if the line is not empty, add the field to the record
if(trim($line)) {
//I am only processing the field name-you'll have to do the same for equal signs
$pos = strpos($line, ' ');
$fieldName=substr($line,0,$pos;
$fieldValue=substr($line,$pos+1);
$record[$fieldName]=$fieldValue;
}
//if it is a blank line and we have a record, save it
else if ($record) {
//insert the record into the database
insertRecord($record);
//empty the record as the next line is a new record
$record=null;
}
}
function insertRecord($record) {
//to do implement an sql insert statement
}