MYSQL - parsing XML and running queries - mysql

Question

I'm struggling parsing XML into MySQL. The other option to parse XML to csv is not feasible as it looks that xmlstarlet is not available for Aix 7.1.0.0.
Investigating MySql Reference Manual I realized that the XML I'm dealing with is not fully supported. I have four different types of XML files. Let's take one as example.
<MovementReport version="0100">
<ControlArea>
<Sender>
<Division>WCS1</Division>
<Confirmation>2</Confirmation>
</Sender>
<CreationDateTime>2018-04-17T15:39:32Z</CreationDateTime>
<RefId>
<Id>6897731</Id>
</RefId>
</ControlArea>
<DataArea>
<RequestId>080030603</RequestId>
<FromLocation>
<MHA>ID1</MHA>
<Rack></Rack>
<X></X>
<Y></Y>
</FromLocation>
<StUnit>
<StUnitId>M1813236 </StUnitId>
</StUnit>
<ToLocation>
<MHA>A</MHA>
<Rack>011</Rack>
<X>065</X>
<Y>019</Y>
</ToLocation>
<ReasonCode>00</ReasonCode>
<StandAloneFlag>W</StandAloneFlag>
<Information>No info!</Information>
</DataArea>
</MovementReport>
I have to use ROWS IDENTIFIED BY in order to have some columns populated. I tried almost all tags in the above command and I came up with the following sql
USE xml_lcs; TRUNCATE TEST01;
LOAD XML LOCAL INFILE '33770626.xml'
INTO TABLE TEST01 ROWS IDENTIFIED BY '<DataArea>'
SET N_ID='A';
LOAD XML LOCAL INFILE '33770626.xml'
INTO TABLE TEST01 ROWS IDENTIFIED BY '<ToLocation>'
(#MHA, #Rack, #X, #Y)
SET t_MHA=#MHA, t_Rack=#Rack,t_X=#X, t_Y=#Y;
LOAD XML LOCAL INFILE '33770626.xml'
INTO TABLE TEST01 ROWS IDENTIFIED BY '<StUnit>'
SET N_ID='A';
LOAD XML LOCAL INFILE '33770626.xml'
INTO TABLE TEST01 ROWS IDENTIFIED BY '<FromLocation>'
(#MHA, #Rack, #X, #Y)
SET f_MHA=#MHA, f_Rack=#Rack,f_X=#X, f_Y=#Y;
LOAD XML LOCAL INFILE '33770626.xml'
INTO TABLE TEST01 ROWS IDENTIFIED BY '<Sender>'
SET N_ID='A';
LOAD XML LOCAL INFILE '33770626.xml'
INTO TABLE TEST01 ROWS IDENTIFIED BY '<RefId>'
SET N_ID='A';
LOAD XML LOCAL INFILE '33770626.xml'
INTO TABLE TEST01 ROWS IDENTIFIED BY '<ReasonCode>'
SET N_ID='A';
The above sql code result in the following table
All columns are VARCHAR.
I would like to have one row for each file, so at the end the above XML file would result in..
Any idea how to achieve that ?
Thanks a lot for your time and help.
Ema

Answer 1

You should use another language to parse the file and insert the data. But if you want a MySQL only solution, you can use your query to store the data into a temprary table and then use an aggregation query to combine the data into one row and copy it into the real table.
First create the temporary table:
CREATE TEMPORARY TABLE TEMP01 LIKE TEST01;
Use your code to load the data from XML to the temporary table. Change TEST01 to TEMP01.
After that copy the data with:
INSERT INTO TEST01 (N_ID, Division, RequestId, ... , StUnitId)
SELECT N_ID
, MIN(Division)
, MIN(RequestId)
...
, MIN(StUnitId)
FROM TEMP01
GROUP BY N_ID;
Since you have only one distinct value per column, it doesn't matter if you use MIN or MAX here. In MySQL 5.7 you can also use ANY_VALUE instead.

Answer 2

You also have the option to use functions like LOAD_FILE() and ExtractValue(). Be aware of the necessary privileges.
Example:
SELECT LOAD_FILE('/path/to/file/33770626.xml') INTO #`xml`;
INSERT INTO `TEST01`
SELECT
NULLIF(TRIM(ExtractValue(#`xml`, 'MovementReport/DataArea/ToLocation/MHA')), ''),
NULLIF(TRIM(ExtractValue(#`xml`, 'MovementReport/ControlArea/Sender/Division')), ''),
NULLIF(TRIM(ExtractValue(#`xml`, 'MovementReport/ControlArea/RefId/Id')), ''),
NULLIF(TRIM(ExtractValue(#`xml`, 'MovementReport/DataArea/FromLocation/Rack')), ''),
.
.
.
NULLIF(ExtractValue(#`xml`, 'MovementReport/DataArea/StUnit/StUnitId'), '');