replace values in non standard Json files in pyspark - json

Question

I need help to solve an issue in Databricks preferably with pyspark code.
I have a troublesome json format, that creates issues when importing the file. The issues lie in the data element, that has been enumerated with 58251 and 58252. The desired state would be to remove this key/numerator. The same issue goes for the "lines" elements 40000, 40001 etc that also would be better with a blank. See example code/json below.
jsonfile= """[ {
"success":true,
"numRows":2,
"data":{
"58251":{
"invoiceno":"58251",
"name":"invoice1",
"companyId":"1000",
"departmentId":"1",
"lines":{
"40000":{
"invoiceline":"40000",
"productid":"1",
"amount":"10000",
"quantity":"7"
},
"40001":{
"invoiceline":"40001",
"productid":"2",
"amount":"9000",
"quantity":"7"
}
}
},
"58252":{
"invoiceno":"58252",
"name":"invoice34",
"companyId":"1001",
"departmentId":"2",
"lines":{
"40002":{
"invoiceline":"40002",
"productid":"3",
"amount":"7000",
"quantity":"6"
},
"40003":{
"invoiceline":"40003",
"productid":"2",
"amount":"9000",
"quantity":"7"
},
"40004":{
"invoiceline":"40004",
"productid":"2",
"amount":"9000",
"quantity":"7"
} } } } }]"""
import pandas as pd
df = pd.read_json(jsonfile)
display(df)
Is it possible to change the json file to the format below with pyspark code?
The desired json format below:
jsonfile= """[ {
"success":true,
"numRows":2,
"data":[
{
"invoiceno":"58251",
"name":"invoice1",
"companyId":"1000",
"departmentId":"1",
"lines":[
{
"invoiceline":"40000",
"productid":"1",
"amount":"10000",
"quantity":"7"
},
{
"invoiceline":"40001",
"productid":"2",
"amount":"9000",
"quantity":"7"
}
]
},
{
"invoiceno":"58252",
"name":"invoice34",
"companyId":"1001",
"departmentId":"2",
"lines":[
{
"invoiceline":"40002",
"productid":"3",
"amount":"7000",
"quantity":"6"
},
{
"invoiceline":"40003",
"productid":"2",
"amount":"9000",
"quantity":"7"
},
{
"invoiceline":"40004",
"productid":"2",
"amount":"9000",
"quantity":"7"
}
]
} ] }]"""
import pandas as pd
df = pd.read_json(jsonfile)
display(df)

Answer 1

This is quite an interesting problem. The main idea of my solution is looping through your schema to build the map between the company's ID and its lines, then select and reconstruct another JSON from there.
Build mapping
# my `a.json` file has exact content as your sample json
df = spark.read.json('a.json', multiLine=True)
mapping = {d.name: d.dataType[3].dataType.fieldNames() for d in df.schema['data'].dataType.fields}
# {'58251': ['40000', '40001'], '58252': ['40002', '40003', '40004']}
Reconstruct JSON
You might want to break this down and debug
cols = []
for company, lines in mapping.items():
cols.append(F.struct(
F.col(f'data.{company}.invoiceno'),
F.col(f'data.{company}.name'),
F.col(f'data.{company}.companyId'),
F.col(f'data.{company}.departmentId'),
F.array([f'data.{company}.lines.{line}' for line in lines]).alias('lines')
))
(df
.select(
F.col('success'),
F.col('numRows'),
F.array(cols).alias('data')
)
.printSchema()
)
root
|-- success: boolean (nullable = true)
|-- numRows: long (nullable = true)
|-- data: array (nullable = false)
| |-- element: struct (containsNull = false)
| | |-- invoiceno: string (nullable = true)
| | |-- name: string (nullable = true)
| | |-- companyId: string (nullable = true)
| | |-- departmentId: string (nullable = true)
| | |-- lines: array (nullable = false)
| | | |-- element: struct (containsNull = true)
| | | | |-- amount: string (nullable = true)
| | | | |-- invoiceline: string (nullable = true)
| | | | |-- productid: string (nullable = true)
| | | | |-- quantity: string (nullable = true)