Load XML file to MySQL database using python with column mapping - mysql

Question

I have written a code to load data to MySQL table using pymysql library. I'm loading data to mysql table in following manner:
import pymysql
con = pymysql.connect(host=host,user=user,password=passwd,db=db,port=int(port),autocommit=True,local_infile=1)
sql = "LOAD XML INFILE '" + path + "' INTO TABLE "+ ds_name +"."+table_name +" SET dataset="+ds_name+", factor_date="+factor_date+","+column_map+ " ROWS IDENTIFIED BY '<LoanInfo>'"
cursor.execute(sql)
cursos.commit()
ds_name and factor_date are not comping from xml file so I'm writing them as static across all the rows.
I have a CSV/excel file which contains mapping between XML file columns and MySQL table column name for 100+ columns. I read somewhere that reference column mapping can be added to SQL query as 'SET ABC_AGE = #Age,UNIQUE_ID= #ID, BALANCE=#Money'. I am creating a list of mapping in following manner:
ls = []
for value in zip(map_df['XML Columns'],map_df['SQL Columns']):
ls.append(value[0]+"=#"+value[1])
column_map = ",".join(ls)
My Question is, Is there a better way to approach to load XML file to MySQL using python with mapping?

Answer 1

I found a way to transform xml file into a pandas dataframe and then load it with executemany to mysql databse. Here is a piece of code to convert xml to dataframe:
#reading mapping file and converting mapping to dictionary
import os
import pandas as pd
map_path = 'Mapping.xlsx'
if os.path.isfile(map_path):
map_df = pd.read_excel(map_path,worksheet='Mapping')
mapping_dict = pd.Series(map_df['XML Columns'].values,index=map_df['SQL columns']).to_dict()
#Reading XML file
import xml.etree.ElementTree as ET
xml_path = 'test.xml'
if os.path.isfile(xml_path):
root = ET.parse(xml_path).getroot()
#Reading xml elements one by one and storing attributes in a dictionary.
missing_col = set()
xmldf_dict = {"df_dicts":[]}
for elem in root:
df_dict = {}
for k,v in mapping_dict.items():
if k in [list of columns to skip]:
continue
try:
df_dict[k] = elem.attrib[v]
except KeyError:
missing_col.add(k)
xmldf_dict["df_dicts"].append(df_dict)
#Merging missing columns dataframe with xml dataframe
missing_col_df = pd.DataFrame(columns=missing_col)
xml_df = pd.DataFrame(xmldf_dict["df_dicts"])
final_df = pd.concat([xml_df,missing_col_df],axis=1)