Convert nested json to multiple dataframes - dynamically - json

i have a question:
Is it possible read a json file and convert to dataframe dynamically?
My example is the next code:
Having this json file, i need 3 table dataframes:
{
"date_time": "01-03-2022, 15:18:32",
"regions": {
"Home Region": "Madrid",
"Primary Region": "Barcelona",
"Secondary Region": "Rio"
},
"customers": [
{
"name": "campo santo",
"address": "rua trebal 1",
"phone": 987456321,
"parking": true
},
{
"name": "santo da silva",
"address": "rua sama 6",
"phone": 654321987,
"parking": false
},
{
"name": "roger campos",
"address": "av casal 10",
"phone": 684426654,
"parking": true
}
],
"office": [
{
"location": "madrid",
"co_working_spaces": 25,
"kitchen": false,
"food_track": 2,
"internal_staff": [
{
"id": 123,
"name": "pablo"
},
{
"id": 874,
"name": "saul"
},
{
"id": 741,
"name": "maria"
}
]
},
{
"location": "rio",
"co_working_spaces": 55,
"kitchen": true,
"food_track": 4,
"internal_staff": [
{
"id": 784,
"name": "raquel"
},
{
"id": 874,
"name": "pedro"
},
{
"id": 145,
"name": "maria"
},
{
"id": 365,
"name": "rocio"
}
]
},
{
"location": "barcelona",
"co_working_spaces": 5,
"kitchen": false,
"food_track": 1,
"internal_staff": [
]
},
{
"location": "la",
"co_working_spaces": 5,
"kitchen": true,
"food_track": 4,
"internal_staff": [
{
"id": 852,
"name": "maria"
},
{
"id": 748,
"name": "sara"
}
]
}
]
}
this is my python code:
import pandas as pd
# from pandas.io.json import json_normalize
import json
with open('offices.json') as f:
dt = json.load(f)
# df = pd.json_normalize(dt)
df1 = pd.json_normalize(dt, 'customers', 'date_time')[['name', 'address', 'phone', 'parking', 'date_time']]
print(df1)
df2 = pd.json_normalize(dt, 'office', 'date_time')[['location', 'co_working_spaces', 'kitchen', 'food_track']]
print(df2)
df3 = pd.json_normalize(dt['office'], 'internal_staff', 'location')
print(df3)
With this code, i got my 3 table dataframes. But i have to know the json structure to create the dataframes.
So is it possible to do it dynamically ?
Regards

Related

Getting JSON data in Groovy

I need to get some datas from JSON, I could manage to transform it into String. For example, I need to get the amount value if team role id is 4.( The last scope in the JSON.) When I run the code below, the "result" output is
{id=1, effectiveDate=2003-01-01, currencyCode=USD, rates=[{id=1, rateTable={id=1, effectiveDate=2003-01-01, currencyCode=USD, name=Tempo Default Price Table, defaultTable=false}, amount=0.0, link={type=DEFAULT_RATE}}], name=Tempo Default Price Table, defaultTable=true}
How can I get the whole data?
Thanks.
http.request(Method.GET) {
response.success = { resp, json ->
arrayDen = JsonOutput.toJson(json).substring(1, JsonOutput.toJson(json).length()-1)
}
}
def slurper = new groovy.json.JsonSlurper()
def result = slurper.parseText(arrayDen)
log.warn(result)
[
{
"id": 1,
"rateTable": {
"id": 1,
"effectiveDate": "2003-01-01",
"currencyCode": "USD",
"name": "Tempo Default Price Table",
"defaultTable": false
},
"amount": 0.0,
"link": {
"type": "DEFAULT_RATE"
}
},
{
"id": 2,
"rateTable": {
"id": 3,
"effectiveDate": "2022-03-21",
"currencyCode": "USD",
"name": "Rate",
"defaultTable": false
},
"amount": 0.0,
"link": {
"type": "DEFAULT_RATE"
}
},
{
"id": 3,
"rateTable": {
"id": 3,
"effectiveDate": "2022-03-21",
"currencyCode": "USD",
"name": "Rate",
"defaultTable": false
},
"amount": 200.0,
"link": {
"type": "TEAM_ROLE",
"id": 8
}
},
{
"id": 4,
"rateTable": {
"id": 3,
"effectiveDate": "2022-03-21",
"currencyCode": "USD",
"name": "Rate",
"defaultTable": false
},
"amount": 500.0,
"link": {
"type": "TEAM_ROLE",
"id": 5
}
},
{
"id": 5,
"rateTable": {
"id": 3,
"effectiveDate": "2022-03-21",
"currencyCode": "USD",
"name": "Rate",
"defaultTable": false
},
"amount": 1000.0,
"link": {
"type": "TEAM_ROLE",
"id": 4
}
}
]
`
Hmmm. The following worked fine for me:
def arrayDen = '<YOUR_URL_GOES_HERE>'.toURL().text
def slurper = new groovy.json.JsonSlurper()
def result = slurper.parseText(arrayDen)
def desiredData = result.find { it.id == 4 }
println desiredData.amount
You might give it a try.

TypeScript convert json structure

I cant figure out how to make this conversion iterating a json.
I have this pojo in my backend:
class Part{
Long id;
String name;
Set<Part> parts = new HashSet<>();
}
Every part can have parts and this part more parts and so on.
I get this parts from httpclient in angular and get this json:
[{
"id": 1,
"name": "Parts A and B",
"parts": [{
"id": 2,
"name": "A",
"parts": [{
"id": 4,
"name": "A1",
"parts": []
}]
},
{
"id": 3,
"name": "B",
"parts": []
}
]
},
{
"id": 2,
"name": "A",
"parts": []
},
{
"id": 3,
"name": "B",
"parts": []
},
{
"id": 4,
"name": "A1",
"parts": []
}
]
And need to convert to this to populate a PrimeNG TreeTable:
{
"data": [{
"data": {
"name": "Parts A and B",
"id": "1"
},
"children": [{
"data": {
"name": "Part A",
"id": "2"
},
"children": [{
"data": {
"name": "A1",
"id": "4"
}
}]
},
{
"data": {
"name": "Part B",
"id": "3"
},
"children": []
}
]
},
{
"data": {
"name": "Part A",
"id": "2"
},
"children": []
},
{
"data": {
"name": "Part B",
"id": "3"
},
"children": []
},
{
"data": {
"name": "A1",
"id": "4"
},
"children": []
}
]
}
How can I do that?
In angular I get this in an array parts: Part[] and need partsTree: TreeNode[]
Thanks!!!
Its just a simple conversion by a map;
interface PartAPI{
id: number;
name: string;
parts : PartAPI[];
}
interface Data {
id: number;
name: string;
}
interface Part {
data : Data;
children : Part[];
}
console.log('a')
let convert = (inputArr: PartAPI[] = []) : Part[] => {
return inputArr.map(partApi => ({ data : { id : partApi.id , name : partApi.name }, children: convert(partApi.parts) }) as Part)
}
let data : PartAPI[] = [{
"id": 1,
"name": "Parts A and B",
"parts": [{
"id": 2,
"name": "A",
"parts": [{
"id": 4,
"name": "A1",
"parts": []
}]
},
{
"id": 3,
"name": "B",
"parts": []
}
]
},
{
"id": 2,
"name": "A",
"parts": []
},
{
"id": 3,
"name": "B",
"parts": []
},
{
"id": 4,
"name": "A1",
"parts": []
}
]
console.log(convert(data));

How to modify json using JRS223 preprocessor in jmeter

Json body looks like this:
{
"access_key": "",
"erid": "",
"ch_sms": {
"messages": [
{
"urlsh": false,
"sp": "123",
"vp": 20,
"heid": 1,
"teid": ,
"peid": ,
"tmid": 4,
"msg": {
"txt": "hello"
},
"da": [
{
"number": "97278",
"cc": "IN",
"uid": "uid1",
"tags": [
"key",
"Value"
]
}
]
}
],
"metadata": {
"chver": "1.0",
"cburl": "",
"heid": 1,
"teid": ,
"peid": ,
"tmid": 4,
"Oa": "",
"flash": false,
"tags": [
"key",
"tag1"
]
}
}
}
We need to add multiple "da" sections based on user input. For example if I give input from a CSV file as 3 then "da" section will repeat 3 times and it will look like as follow:
{
"access_key": "",
"erid": "",
"ch_sms": {
"messages": [
{
"urlsh": false,
"sp": "123",
"vp": 20,
"heid": 1,
"teid": ,
"peid": ,
"tmid": 4,
"msg": {
"txt": "hello"
},
"da": [
{
"number": "97278",
"cc": "IN",
"uid": "uid1",
"tags": [
"key",
"Value"
]
},
{
"number": "97278",
"cc": "IN",
"uid": "uid1",
"tags": [
"key",
"Value"
]
},
{
"number": "97278",
"cc": "IN",
"uid": "uid1",
"tags": [
"key",
"Value"
]
}
]
}
],
"metadata": {
"chver": "1.0",
"cburl": "",
"heid": 1,
"teid": ,
"peid": ,
"tmid": 4,
"Oa": "",
"flash": false,
"tags": [
"key",
"tag1"
]
}
}
}
You can try this,
import org.json.simple.JSONObject
JSONObject data = new JSONObject()
data.put("number", "97278")
data.put("cc", "IN")
data.put("uid", "uid1")
data.put("tags", ["key","Value"])
int da = Integer.parseInt(vars.get("da"))
ArrayList<Object> listData = new ArrayList<Object>()
for(int i=0; i < da; i++)
{
listData.add(data.toString())
}
log.info("da: " + listData )
Result log:
da: [{"cc":"IN","number":"97278","uid":"uid1","tags":["key","Value"]}, {"cc":"IN","number":"97278","uid":"uid1","tags":["key","Value"]}, {"cc":"IN","number":"97278","uid":"uid1","tags":["key","Value"]}]

Nested json - store values in csv

I am trying to convert a nested json file into csv. It's data from a darts API and the structure is always the same. Nevertheless I got some problems flattening and storing the values in a csv because of the nested structure.
json:
{
"summaries": [{
"sport_event": {
"id": "sr:sport_event:12967512",
"start_time": "2017-11-11T13:15:00+00:00",
"start_time_confirmed": true,
"sport_event_context": {
"sport": {
"id": "sr:sport:22",
"name": "Darts"
},
"category": {
"id": "sr:category:104",
"name": "International"
},
"competition": {
"id": "sr:competition:597",
"name": "Grand Slam of Darts"
},
"season": {
"id": "sr:season:47332",
"name": "Grand Slam of Darts 2017",
"start_date": "2017-11-11",
"end_date": "2017-11-20",
"year": "2017",
"competition_id": "sr:competition:597"
},
"stage": {
"order": 1,
"type": "league",
"phase": "stage_1",
"start_date": "2017-11-11",
"end_date": "2017-11-15",
"year": "2017"
},
"round": {
"number": 1
},
"groups": [{
"id": "sr:league:29766",
"name": "Grand Slam of Darts 2017, Group G",
"group_name": "G"
}]
},
"coverage": {
"live": true
},
"competitors": [{
"id": "sr:competitor:35936",
"name": "Smith, Michael",
"abbreviation": "SMI",
"qualifier": "home"
}, {
"id": "sr:competitor:83895",
"name": "Wilson, James",
"abbreviation": "WIL",
"qualifier": "away"
}]
},
"sport_event_status": {
"status": "closed",
"match_status": "ended",
"home_score": 5,
"away_score": 3,
"winner_id": "sr:competitor:35936"
}
}, {
"sport_event": {
"id": "sr:sport_event:12967508",
"start_time": "2017-11-11T13:40:00+00:00",
"start_time_confirmed": true,
"sport_event_context": {
"sport": {
"id": "sr:sport:22",
"name": "Darts"
},
"category": {
"id": "sr:category:104",
"name": "International"
},
"competition": {
"id": "sr:competition:597",
"name": "Grand Slam of Darts"
},
"season": {
"id": "sr:season:47332",
"name": "Grand Slam of Darts 2017",
"start_date": "2017-11-11",
"end_date": "2017-11-20",
"year": "2017",
"competition_id": "sr:competition:597"
},
"stage": {
"order": 1,
"type": "league",
"phase": "stage_1",
"start_date": "2017-11-11",
"end_date": "2017-11-15",
"year": "2017"
},
"round": {
"number": 1
},
"groups": [{
"id": "sr:league:29764",
"name": "Grand Slam of Darts 2017, Group F",
"group_name": "F"
}]
},
"coverage": {
"live": true
},
"competitors": [{
"id": "sr:competitor:70916",
"name": "Bunting, Stephen",
"abbreviation": "BUN",
"qualifier": "home"
}, {
"id": "sr:competitor:191262",
"name": "de Zwaan, Jeffrey",
"abbreviation": "DEZ",
"qualifier": "away"
}]
},
"sport_event_status": {
"status": "closed",
"match_status": "ended",
"home_score": 5,
"away_score": 4,
"winner_id": "sr:competitor:70916"
}
}
So for each sport_event I would like to store the variables:
"start_time"
from "season" the variable "name"
from "competitors" both "id" and "name"
from "sport_event_status" the "winner_id"
I have already tried to flatten the json file with this code:
import json
f = open(r'path of file.json')
data = json.load(f)
def flatten(data):
for key,value in data.items():
print (str(key)+'->'+str(value))
if type(value) == type(dict()):
flatten(value)
elif type(value) == type(list()):
for val in value:
if type(val) == type(str()):
pass
elif type(val) == type(list()):
pass
else:
flatten(val)
flatten(data)
print(data)
This actually prints out the following:
id->sr:season:47332
name->Grand Slam of Darts 2017
start_date->2017-11-11
end_date->2017-11-20
year->2017
competition_id->sr:competition:597
Now my question is how to store the values I mentioned above in a csv file.
Thanks in advance for your support.
Using jq, you basically just have to transcribe your specification, adding a bit of context and taking care of an embedded array:
.summaries[]
| .sport_event # Your specification:
| [.start_time, # start_time
.sport_event_context.season.name] # from "season" the variable "name"
+ [.competitors[] | .id, .name] # from "competitors" both "id" and "name"
+ [.sport_event_status.winner_id] # from "sport_event_status" the "winner_id"
| #csv
Invocation
E.g.
jq -rf program.jq my.json

Convert nested json to csv to sheets json api

I'm want to make my json to csv so that i can upload it on google sheets and make it as json api. Whenever i have change data i will just change it on google sheets. But I'm having problems on converting my json file to csv because it changes the variables whenever i convert it. I'm using https://toolslick.com/csv-to-json-converter to convert my json file to csv.
What is the best way to convert json nested to csv ?
JSON
{
"options": [
{
"id": "1",
"value": "Jumbo",
"shortcut": "J",
"textColor": "#FFFFFF",
"backgroundColor": "#00000"
},
{
"id": "2",
"value": "Hot",
"shortcut": "D",
"textColor": "#FFFFFF",
"backgroundColor": "#FFFFFF"
}
],
"categories": [
{
"id": "1",
"order": 1,
"name": "First Category",
"active": true
},
{
"id": "2",
"order": 2,
"name": "Second Category",
"shortcut": "MT",
"active": true
}
],
"products": [
{
"id": "03c6787c-fc2a-4aa8-93a3-5e0f0f98cfb2",
"categoryId": "1",
"name": "First Product",
"shortcut": "First",
"options": [
{
"optionId": "1",
"price": 23
},
{
"optionId": "2",
"price": 45
}
],
"active": true
},
{
"id": "e8669cea-4c9c-431c-84ba-0b014f0f9bc2",
"categoryId": "2",
"name": "Second Product",
"shortcut": "Second",
"options": [
{
"optionId": "1",
"price": 11
},
{
"optionId": "2",
"price": 20
}
],
"active": true
}
],
"discounts": [
{
"id": "1",
"name": "S",
"type": 1,
"amount": 20,
"active": true
},
{
"id": "2",
"name": "P",
"type": 1,
"amount": 20,
"active": true
},
{
"id": "3",
"name": "G",
"type": 2,
"amount": 5,
"active": true
}
]
}
Using python, this can be easily done or almost done. Maybe this code will help you in some way to understand that.
import json,csv
data = []
with open('your_json_file_here.json') as file:
for line in file:
data.append(json.loads(line))
length = len(data)
with open('create_new_file.csv','w') as f:
writer = csv.writer(f)
writers = csv.DictWriter(f, fieldnames=['header1','header2'])
writers.writeheader()
for iter in range(length):
writer.writerow((data[iter]['specific_col_name1'],data[iter]['specific_col_name2']))
f.close()