Convert nested json to csv to sheets json api - json

I'm want to make my json to csv so that i can upload it on google sheets and make it as json api. Whenever i have change data i will just change it on google sheets. But I'm having problems on converting my json file to csv because it changes the variables whenever i convert it. I'm using https://toolslick.com/csv-to-json-converter to convert my json file to csv.
What is the best way to convert json nested to csv ?
JSON
{
"options": [
{
"id": "1",
"value": "Jumbo",
"shortcut": "J",
"textColor": "#FFFFFF",
"backgroundColor": "#00000"
},
{
"id": "2",
"value": "Hot",
"shortcut": "D",
"textColor": "#FFFFFF",
"backgroundColor": "#FFFFFF"
}
],
"categories": [
{
"id": "1",
"order": 1,
"name": "First Category",
"active": true
},
{
"id": "2",
"order": 2,
"name": "Second Category",
"shortcut": "MT",
"active": true
}
],
"products": [
{
"id": "03c6787c-fc2a-4aa8-93a3-5e0f0f98cfb2",
"categoryId": "1",
"name": "First Product",
"shortcut": "First",
"options": [
{
"optionId": "1",
"price": 23
},
{
"optionId": "2",
"price": 45
}
],
"active": true
},
{
"id": "e8669cea-4c9c-431c-84ba-0b014f0f9bc2",
"categoryId": "2",
"name": "Second Product",
"shortcut": "Second",
"options": [
{
"optionId": "1",
"price": 11
},
{
"optionId": "2",
"price": 20
}
],
"active": true
}
],
"discounts": [
{
"id": "1",
"name": "S",
"type": 1,
"amount": 20,
"active": true
},
{
"id": "2",
"name": "P",
"type": 1,
"amount": 20,
"active": true
},
{
"id": "3",
"name": "G",
"type": 2,
"amount": 5,
"active": true
}
]
}

Using python, this can be easily done or almost done. Maybe this code will help you in some way to understand that.
import json,csv
data = []
with open('your_json_file_here.json') as file:
for line in file:
data.append(json.loads(line))
length = len(data)
with open('create_new_file.csv','w') as f:
writer = csv.writer(f)
writers = csv.DictWriter(f, fieldnames=['header1','header2'])
writers.writeheader()
for iter in range(length):
writer.writerow((data[iter]['specific_col_name1'],data[iter]['specific_col_name2']))
f.close()

Related

JSON Parsing a Plain Text File

I have a lot of data I need to parse though.
I need to pull all pid's and price's.
`
[
{
"id": 159817,
"price": "10.69",
"stocked": true,
"store": {
"id": 809,
"nsn": "22036-0",
"pricingSource": "manual",
"lastUpdated": "2022-12-05T15:24:33.908Z"
},
"sharedFields": {
"type": "PRODUCT",
"id": 24549,
"pid": "12079",
"labels": [
{
"type": "default",
"value": "Chicken Sandwich",
"locale": "en"
},
{
"type": "fresh",
"value": "Chicken",
"locale": "en"
},
{
"type": "product_json",
"value": "Chicken",
"locale": "en"
}
],
"calMin": 600,
"calMax": 600,
"lastUpdated": "2021-12-31T13:49:22.794Z"
}
},
{
"id": 159818,
"price": "9.29",
"stocked": true,
"store": {
"id": 809,
"nsn": "22036-0",
"pricingSource": "manual",
"lastUpdated": "2022-12-05T15:24:33.908Z"
},
"sharedFields": {
"type": "PRODUCT",
"id": 25,
"pid": "1",
"labels": [
{
"type": "default",
"value": "Ham Sandwich",
"locale": "en"
},
{
"type": "fresh",
"value": "Ham",
"locale": "en"
}
],
"calMin": 540,
"calMax": 540,
"lastUpdated": "2021-07-09T19:30:00.326Z"
}
}
]
`
and I need to place them into a string like this, but on a scale of 150 products. I'd also need to change "pid" to "productId"
[{ "productId": "46238", "price": 6.09 }, { "productId": "40240", "price": 1.49 }]
I need to add a string before this data, but I'm pretty confident I can figure that part out.
I am pretty open to the easiest suggestion, whether that be VBS, Excel macro, etc.

Trying to recursively populate a ttk.treeview with json/dictionary in python

I am new to python, but have java experience. Pretty different animals. I have a method that creates a json/dictionary based on walking through a directory structure and I have it where it creates a json like the one below.
I am trying to get another method to populate a treeview based on it. I have seen several examples here on stackoverflow and have attempted to follow them. Below is what I have come up with, but it always errs out after going through the first directory, like it lost track of where it was. the following errors are returned:
Traceback (most recent call last):
File "temp.py", line 147, in <module>
load_treeview_from_project_dictionary(myData, tree_view)
File "temp.py", line 134, in load_treeview_from_project_dictionary
load_treeview_from_project_dictionary(c, tree_view, int(c['index']))
File "temp.py", line 136, in load_treeview_from_project_dictionary
tree_view.insert(parent_id, "end", text=c['name'], values=("", ""))
File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\tkinter\ttk.py", line 1364, in insert
res = self.tk.call(self._w, "insert", parent, index, *opts)
_tkinter.TclError: Item 1 not found
I worked several hours trying to refactor and figure this out, but have been unsuccessful. I am now reaching out to this great community to point out the error of my ways!
Please help.
import json
import tkinter
from tkinter import ttk
from tkinter.ttk import Label
myData = {
"name": "MyRoot",
"type": "directory",
"index": 0,
"children": [
{
"name": "SubDir1",
"type": "directory",
"index": 1,
"children": [
{
"name": "apic.png",
"type": "file"
}
]
},
{
"name": "SubDir2",
"type": "directory",
"index": 2,
"children": [
{
"name": "somefile.txt",
"type": "file"
},
{
"name": "anotherfile.txt",
"type": "file"
}
]
},
{
"name": "SubDir3",
"type": "directory",
"index": 3,
"children": []
},
{
"name": "SubDir4",
"type": "directory",
"index": 4,
"children": [
{
"name": "asomefile.txt",
"type": "file"
},
{
"name": "bsomefile.txt",
"type": "file"
},
{
"name": "csomefile.txt",
"type": "file"
},
{
"name": "dsomefile.txt",
"type": "file"
},
{
"name": "esomefile.txt",
"type": "file"
}
]
},
{
"name": "SubDir5",
"type": "directory",
"index": 5,
"children": [
{
"name": "NesterDir1",
"type": "directory",
"index": 6,
"children": []
}
]
},
{
"name": "SubDir6",
"type": "directory",
"index": 7,
"children": []
},
{
"name": "SubDir7",
"type": "directory",
"index": 8,
"children": []
},
{
"name": "SubDir8",
"type": "directory",
"index": 9,
"children": []
},
{
"name": "SubDir9",
"type": "directory",
"index": 10,
"children": []
},
{
"name": "SubDir10",
"type": "directory",
"index": 11,
"children": []
},
{
"name": "SubDir11",
"type": "directory",
"index": 12,
"children": []
}
]
}
def load_treeview_from_project_dictionary(data, my_tree_view, parent_id=None):
print("this:" + data['name'] + " called function!")
if parent_id is None:
my_tree_view.insert("", "0", text=data['name'], values=("", "")) # applies to first iteration only
if data['children'] is not None:
for c in data['children']:
print("child: " + c['name'])
if c['type'] == "directory":
my_tree_view.insert('', int(c['index']), text=c['name'], values=("", ""))
load_treeview_from_project_dictionary(c, my_tree_view, int(c['index']))
else:
my_tree_view.insert(parent_id, "end", text=c['name'], values=("", ""))
load_treeview_from_project_dictionary(c, my_tree_view, parent_id)
root = tkinter.Tk()
main_label = Label(root, text="Directory Tree")
tree_view = ttk.Treeview(root, height=23)
tree_view.heading("#0", text="Directory Structure")
load_treeview_from_project_dictionary(myData, tree_view)
main_label.pack()
tree_view.pack()
root.mainloop()
Thanks in Advance!
So. After reviewing the tutorial link posted by D.L and then combing through my code and debugging over and over, I came to the conclusion that there was too much recursion going on. Watching the flow I found that the method always stopped after the first file was added to the tree. Taking the recursive call after the file insert fixed a large part of the issue. I scrutinized the insertion process and found that I could use the indices in the json for the treeview's iid's. I then decided that it would be more efficient to use the treeview.move to place the entries where I wanted them as they are being inserted. Below is what I came up with and it works great. I am posting this here for anyone else that runs into the same issue. After the code, there is a screenshot of the resulting treeview (or a link to it due to my rank- I will try to fix that later)
import json
import tkinter
from tkinter import ttk
from tkinter.ttk import Label
myData = {
"name": "MyRoot",
"type": "directory",
"index": 0,
"children": [
{
"name": "SubDir1",
"type": "directory",
"index": 1,
"children": [
{
"name": "apic.png",
"type": "file"
}
]
},
{
"name": "SubDir2",
"type": "directory",
"index": 2,
"children": [
{
"name": "somefile.txt",
"type": "file"
},
{
"name": "anotherfile.txt",
"type": "file"
}
]
},
{
"name": "SubDir3",
"type": "directory",
"index": 3,
"children": []
},
{
"name": "SubDir4",
"type": "directory",
"index": 4,
"children": [
{
"name": "asomefile.txt",
"type": "file"
},
{
"name": "bsomefile.txt",
"type": "file"
},
{
"name": "csomefile.txt",
"type": "file"
},
{
"name": "dsomefile.txt",
"type": "file"
},
{
"name": "esomefile.txt",
"type": "file"
}
]
},
{
"name": "SubDir5",
"type": "directory",
"index": 5,
"children": [
{
"name": "NestedDir1",
"type": "directory",
"index": 6,
"children": [
{
"name": "yetAnotherfile.txt",
"type": "file"
}
]
}
]
},
{
"name": "SubDir6",
"type": "directory",
"index": 7,
"children": []
},
{
"name": "SubDir7",
"type": "directory",
"index": 8,
"children": []
},
{
"name": "SubDir8",
"type": "directory",
"index": 9,
"children": []
},
{
"name": "SubDir9",
"type": "directory",
"index": 10,
"children": []
},
{
"name": "SubDir10",
"type": "directory",
"index": 11,
"children": []
},
{
"name": "SubDir11",
"type": "directory",
"index": 12,
"children": []
}
]
}
def load_treeview_from_project_dictionary(data, my_tree_view, parent_id=None):
print('this:' + data['name'] + ' called function!')
if parent_id is None:
my_tree_view.insert('', '0', text=data['name'], iid=0) # applies to first iteration only
for c in data['children']:
print('child: ' + c['name'])
if c['type'] == 'directory':
my_tree_view.insert('', 'end', text=c['name'], iid=c['index'])
my_tree_view.move(c['index'], data['index'], 'end')
load_treeview_from_project_dictionary(c, my_tree_view, data['index'])
else:
file_index = my_tree_view.insert('', 'end', text=c['name'])
my_tree_view.move(file_index, data['index'], 'end')
root = tkinter.Tk()
main_label = Label(root, text='Directory Tree')
tree_view = ttk.Treeview(root, height=23)
tree_view.heading('#0', text='Directory Structure')
load_treeview_from_project_dictionary(myData, tree_view)
main_label.pack()
tree_view.pack()
root.mainloop()
Treeview screenshot

Nested json - store values in csv

I am trying to convert a nested json file into csv. It's data from a darts API and the structure is always the same. Nevertheless I got some problems flattening and storing the values in a csv because of the nested structure.
json:
{
"summaries": [{
"sport_event": {
"id": "sr:sport_event:12967512",
"start_time": "2017-11-11T13:15:00+00:00",
"start_time_confirmed": true,
"sport_event_context": {
"sport": {
"id": "sr:sport:22",
"name": "Darts"
},
"category": {
"id": "sr:category:104",
"name": "International"
},
"competition": {
"id": "sr:competition:597",
"name": "Grand Slam of Darts"
},
"season": {
"id": "sr:season:47332",
"name": "Grand Slam of Darts 2017",
"start_date": "2017-11-11",
"end_date": "2017-11-20",
"year": "2017",
"competition_id": "sr:competition:597"
},
"stage": {
"order": 1,
"type": "league",
"phase": "stage_1",
"start_date": "2017-11-11",
"end_date": "2017-11-15",
"year": "2017"
},
"round": {
"number": 1
},
"groups": [{
"id": "sr:league:29766",
"name": "Grand Slam of Darts 2017, Group G",
"group_name": "G"
}]
},
"coverage": {
"live": true
},
"competitors": [{
"id": "sr:competitor:35936",
"name": "Smith, Michael",
"abbreviation": "SMI",
"qualifier": "home"
}, {
"id": "sr:competitor:83895",
"name": "Wilson, James",
"abbreviation": "WIL",
"qualifier": "away"
}]
},
"sport_event_status": {
"status": "closed",
"match_status": "ended",
"home_score": 5,
"away_score": 3,
"winner_id": "sr:competitor:35936"
}
}, {
"sport_event": {
"id": "sr:sport_event:12967508",
"start_time": "2017-11-11T13:40:00+00:00",
"start_time_confirmed": true,
"sport_event_context": {
"sport": {
"id": "sr:sport:22",
"name": "Darts"
},
"category": {
"id": "sr:category:104",
"name": "International"
},
"competition": {
"id": "sr:competition:597",
"name": "Grand Slam of Darts"
},
"season": {
"id": "sr:season:47332",
"name": "Grand Slam of Darts 2017",
"start_date": "2017-11-11",
"end_date": "2017-11-20",
"year": "2017",
"competition_id": "sr:competition:597"
},
"stage": {
"order": 1,
"type": "league",
"phase": "stage_1",
"start_date": "2017-11-11",
"end_date": "2017-11-15",
"year": "2017"
},
"round": {
"number": 1
},
"groups": [{
"id": "sr:league:29764",
"name": "Grand Slam of Darts 2017, Group F",
"group_name": "F"
}]
},
"coverage": {
"live": true
},
"competitors": [{
"id": "sr:competitor:70916",
"name": "Bunting, Stephen",
"abbreviation": "BUN",
"qualifier": "home"
}, {
"id": "sr:competitor:191262",
"name": "de Zwaan, Jeffrey",
"abbreviation": "DEZ",
"qualifier": "away"
}]
},
"sport_event_status": {
"status": "closed",
"match_status": "ended",
"home_score": 5,
"away_score": 4,
"winner_id": "sr:competitor:70916"
}
}
So for each sport_event I would like to store the variables:
"start_time"
from "season" the variable "name"
from "competitors" both "id" and "name"
from "sport_event_status" the "winner_id"
I have already tried to flatten the json file with this code:
import json
f = open(r'path of file.json')
data = json.load(f)
def flatten(data):
for key,value in data.items():
print (str(key)+'->'+str(value))
if type(value) == type(dict()):
flatten(value)
elif type(value) == type(list()):
for val in value:
if type(val) == type(str()):
pass
elif type(val) == type(list()):
pass
else:
flatten(val)
flatten(data)
print(data)
This actually prints out the following:
id->sr:season:47332
name->Grand Slam of Darts 2017
start_date->2017-11-11
end_date->2017-11-20
year->2017
competition_id->sr:competition:597
Now my question is how to store the values I mentioned above in a csv file.
Thanks in advance for your support.
Using jq, you basically just have to transcribe your specification, adding a bit of context and taking care of an embedded array:
.summaries[]
| .sport_event # Your specification:
| [.start_time, # start_time
.sport_event_context.season.name] # from "season" the variable "name"
+ [.competitors[] | .id, .name] # from "competitors" both "id" and "name"
+ [.sport_event_status.winner_id] # from "sport_event_status" the "winner_id"
| #csv
Invocation
E.g.
jq -rf program.jq my.json

How to avoid generating all combinations of selected data while constructing an object?

My original JSON is given below.
[
{
"id": "1",
"name": "AA_1",
"total": "100002",
"files": [
{
"filename": "8665b987ab48511eda9e458046fbc42e.csv",
"filename_original": "some.csv",
"status": "3",
"total": "100002",
"time": "2020-08-24 23:25:49"
}
],
"status": "3",
"created": "2020-08-24 23:25:49",
"filenames": "8665b987ab48511eda9e458046fbc42e.csv",
"is_append": "0",
"is_deleted": "0",
"comment": null
},
{
"id": "4",
"name": "AA_2",
"total": "43806503",
"files": [
{
"filename": "1b4812fe634938928953dd40db1f70b2.csv",
"filename_original": "other.csv",
"status": "3",
"total": "21903252",
"time": "2020-08-24 23:33:43"
},
{
"filename": "63ab85fef2412ce80ae8bd018497d8bf.csv",
"filename_original": "some.csv",
"status": "2",
"total": 0,
"time": "2020-08-24 23:29:30"
}
],
"status": "2",
"created": "2020-08-24 23:35:51",
"filenames": "1b4812fe634938928953dd40db1f70b2.csv&&63ab85fef2412ce80ae8bd018497d8bf.csv",
"is_append": "0",
"is_deleted": "0",
"comment": null
}
]
From this JSON I want to create new objects by combining fields from objects which have status: 2 and their files which also have the same pair, status: 2.
So, I am expecting a JSON array as below.
[
{
"id": "4",
"name": "AA_2",
"file_filename": "63ab85fef2412ce80ae8bd018497d8bf.csv",
"file_status": 2
}
]
So far I tried with this JQ filter:
.[]|select(.status=="2")|[{id:.id,file_filename:.files[].filename,file_status:.files[].status}]
But this produces some invalid data.
[
{
"id": "4", # want to remove this as file.status != 2
"file_filename": "1b4812fe634938928953dd40db1f70b2.csv",
"file_status": "3"
},
{
"id": "4",
"file_filename": "1b4812fe634938928953dd40db1f70b2.csv",
"file_status": "2"
},
{
"id": "4", # Repeat
"file_filename": "63ab85fef2412ce80ae8bd018497d8bf.csv",
"file_status": "3"
},
{
"id": "4", # Repeat
"file_filename": "63ab85fef2412ce80ae8bd018497d8bf.csv",
"file_status": "2"
}
]
How do I filter the new JSON using JQ and remove these duplicate objects?
By applying [] operator to files twice, you're running into a combinatorial explosion. That needs to be avoided, for example:
[ .[] | select(.status == "2") | {id, name} + (.files[] | select(.status == "2") | {file_filename: .filename, file_status: .status}) ]
Online demo

Convert json to include name and children values to feed D3

I am trying to get json converted from:
{
"Devices": [
{
"Udid": "7a2b0e6c928f2321a75e423ba23ae93d",
"SerialNumber": "RF1D232ZLEE",
"MacAddress": "40F232726FC8",
"Imei": "3576342323280150",
"EasId": "SEC1BC252327E92B",
"AssetNumber": "7a2b0e23223928f2321a75e423ba23ae93d",
"DeviceFriendlyName": "gel1 Android Android 5.0.1 ZLEE ",
"LocationGroupId": {
"Id": {
"Value": 19529
},
"Name": "Group Express"
},
"LocationGroupName": "Group Express",
"UserId": {
"Name": ""
},
"UserName": "",
"UserEmailAddress": "",
"Ownership": "S",
"PlatformId": {
"Id": {
"Value": 5
},
"Name": "Android"
},
"Platform": "Android",
"ModelId": {
"Id": {
"Value": 5
},
"Name": "samsung GT-I9505"
},
"Model": "samsung GT-I9505",
"OperatingSystem": "5.0.1",
"PhoneNumber": "+447881867010",
"LastSeen": "2016-07-06T14:01:03.590",
"EnrollmentStatus": "Unenrolled",
"ComplianceStatus": "NotAvailable",
"CompromisedStatus": false,
"LastEnrolledOn": "2016-06-15T16:01:38.763",
"LastComplianceCheckOn": "0001-01-01T00:00:00.000",
"LastCompromisedCheckOn": "2016-07-06T13:58:26.183",
"IsSupervised": false,
"DeviceMCC": {
"SIMMCC": "234",
"CurrentMCC": "234"
},
"AcLineStatus": 0,
"VirtualMemory": 0,
"Id": {
"Value": 23459
}
},
{
"Udid": "c5f94db71d406dae7f881d3edf059e",
"SerialNumber": "",
"MacAddress": "000C300F9108",
"Imei": "",
"EasId": "D80DB85EC411C8E9B28BC292A603F05C2C0EEEC8",
"AssetNumber": "c592f93db71d406dae7f881d3edf059e",
"DeviceFriendlyName": "user Windows 10 WinRT 10.0.10240 ",
"LocationGroupId": {
"Id": {
"Value": 18498
},
"Name": "Business Solutions"
},
"LocationGroupName": "Business Solutions",
"UserId": {
"Name": ""
},
"UserName": "",
"UserEmailAddress": "",
"Ownership": "C",
"PlatformId": {
"Id": {
"Value": 12
},
"Name": "WinRT"
},
"Platform": "WinRT",
"ModelId": {
"Id": {
"Value": 50
},
"Name": "Windows 10"
},
"Model": "Windows 10",
"OperatingSystem": "10.0.10240",
"PhoneNumber": "",
"LastSeen": "2016-05-03T10:54:07.650",
"EnrollmentStatus": "Unenrolled",
"ComplianceStatus": "NotAvailable",
"CompromisedStatus": false,
"LastEnrolledOn": "2016-01-29T16:41:57.760",
"LastComplianceCheckOn": "0001-01-01T00:00:00.000",
"LastCompromisedCheckOn": "0001-01-01T00:00:00.000",
"IsSupervised": false,
"DeviceMCC": {
"SIMMCC": "",
"CurrentMCC": ""
},
"AcLineStatus": 0,
"VirtualMemory": 0,
"Id": {
"Value": 23545
}
}
],
"Page": 0,
"PageSize": 500,
"Total": 13}
To something like:
{"name": "Devices",
"children": [
{"name":"Udid", "size":"7f0dsda63274692ea4f0b66fec67a020158"},
{"name":"SerialNumber", "size":"P988KJSPQF938"},
{"name":"MacAddress", "size":"1HJUSUD031C4"},
{"name":"Imei", "size":""},
{"name":"EasId", "size":"ApKJSPQF193"},
{"name":"AssetNumber", "size":"7f0cda636b3305fea4f0b66fec9997267a020158"},
{"name":"DeviceFriendlyName", "size":"TMcKenz iPad iOS 7.1.4 F193 "},
{"name":"LocationGroupId",
"children": [
{"name":"Id","size":7488},
{"name":"Name","size":"MCM"}
]
},
{"name":"UserId",
"children": [
{"name":"Id","size":6418},
{"name":"Name","size":"Tom McKenz"}
]
},
{"name":"UserName", "size":"TMcKenz"},
{"name":"UserEmailAddress", "size":"TMcKenz#awaw.com"}
]
}
Not sure what is the best practice here: is it possible to use D3.nest or do you need to iterate through all the nodes and change to 'name' and 'children' accordingly.