read large file excel with multiple worksheet to json with python - json

Question

I have a large excel file and have multiple worksheets its 100 MB
sheet A
id | name | address
1 | joe | A
2 | gis | B
3 | leo | C
work_1
id| call
1 | 10
1 | 8
2 | 1
3 | 3
work_2
id| call
2 | 4
3 | 8
3 | 7
desired json for each id
data = { id: 1,
address: A,
name: Joe,
log : [{call:10}, {call:8 }]
}
data= { id: 2,
address: B,
name: Gis,
log : [{call:1}, {call:4}]
}
data= { id: 3,
address: C,
name: Leo,
log : [{call:3}, {call:8}, {call:7}]
}
i've tried with pandas but it takes 5 minutes to run it and it only read_excel without any processing. is there any solution to make it faster and how to get desired json?
maybe divide the process into chunk(but pandas removed chunksize for read_excel) and add some threading to make interval so the procees could be print each batch.

Answer 1

You can do:
works=pd.concat([work1,work2],ignore_index=True)
mapper_works=works.groupby('id')[['call']].apply(lambda x: x.to_dict('records'))
dfa['log']=dfa['id'].map(mapper_works)
data=dfa.reindex(columns=['id','address','name','log']).to_dict('records')
print(data)
The output is a list of dict for each id:
[{'id': 1, 'address': 'A', 'name': 'joe', 'log': [{'call': 10}, {'call': 8}]},
{'id': 2, 'address': 'B', 'name': 'gis', 'log': [{'call': 1}, {'call': 4}]},
{'id': 3, 'address': 'C', 'name': 'leo', 'log': [{'call': 3}, {'call': 8}, {'call': 7}]}
]
If you want you can assign to a column:
dfa['dicts']=data
print(dfa)
id name address log \
0 1 joe A [{'call': 10}, {'call': 8}]
1 2 gis B [{'call': 1}, {'call': 4}]
2 3 leo C [{'call': 3}, {'call': 8}, {'call': 7}]
dicts
0 {'id': 1, 'address': 'A', 'name': 'joe', 'log'...
1 {'id': 2, 'address': 'B', 'name': 'gis', 'log'...
2 {'id': 3, 'address': 'C', 'name': 'leo', 'log'...