Does TOML support nested arrays of objects/tables? - json

I want to generate JSON from TOML files. The JSON structure should be something like this, with arrays of objects within arrays of objects:
{
"things": [
{
"a": "thing1",
"b": "fdsa",
"multiline": "Some sample text."
},
{
"a": "Something else",
"b": "zxcv",
"multiline": "Multiline string",
"objs": [ // LOOK HERE
{ "x": 1},
{ "x": 4 },
{ "x": 3 }
]
},
{
"a": "3",
"b": "asdf",
"multiline": "thing 3.\nanother line"
}
]
}
I have some TOML that looks like the example below, but it doesn't seem to work with the objs section.
name = "A Test of the TOML Parser"
[[things]]
a = "thing1"
b = "fdsa"
multiLine = """
Some sample text."""
[[things]]
a = "Something else"
b = "zxcv"
multiLine = """
Multiline string"""
[[things.objs]] # MY QUESTION IS ABOUT THIS PART
x = 1
[[things.objs]]
x = 4
[[things.objs]]
x = 3
[[things]]
a = "3"
b = "asdf"
multiLine = """
thing 3.
another line"""
Is there a way to do it in TOML? JSON to TOML converters don't seem to work with my example. And does it work with deeper nesting of arrays of arrays/tables?

As per the PR that merged this feature in the main TOML repository, this is the correct syntax for arrays of objects:
[[products]]
name = "Hammer"
sku = 738594937
[[products]]
[[products]]
name = "Nail"
sku = 284758393
color = "gray"
Which would produce the following equivalent JSON:
{
"products": [
{ "name": "Hammer", "sku": 738594937 },
{ },
{ "name": "Nail", "sku": 284758393, "color": "gray" }
]
}

I'm not sure why it wasn't working before, but this seems to work:
name = "A Test of the TOML Parser"
[[things]]
a = "thing1"
b = "fdsa"
multiLine = """
Some sample text."""
[[things]]
a = "Something else"
b = "zxcv"
multiLine = """
Multiline string"""
[[things.objs]]
x = 1
[[things.objs]]
x = 4
[[things.objs]]
x = 7
[[things.objs.morethings]]
y = [
2,
3,
4
]
[[things.objs.morethings]]
y = 9
[[things]]
a = "3"
b = "asdf"
multiLine = """
thing 3.
another line"""
JSON output:
{
"name": "A Test of the TOML Parser",
"things": [{
"a": "thing1",
"b": "fdsa",
"multiLine": "Some sample text."
}, {
"a": "Something else",
"b": "zxcv",
"multiLine": "Multiline string",
"objs": [{
"x": 1
}, {
"x": 4
}, {
"x": 7,
"morethings": [{
"y": [2, 3, 4]
}, {
"y": 9
}]
}]
}, {
"a": "3",
"b": "asdf",
"multiLine": "thing 3.\\nanother line"
}]
}

Related

pandas column to list for a json file

from a Dataframe, I want to have a JSON output file with one key having a list:
Expected output:
[
{
"model": "xx",
"id": 1,
"name": "xyz",
"categories": [1,2],
},
{
...
},
]
What I have:
[
{
"model": "xx",
"id": 1,
"name": "xyz",
"categories": "1,2",
},
{
...
},
]
The actual code is :
df = pd.read_excel('data_threated.xlsx')
result = df.reset_index(drop=True).to_json("output_json.json", orient='records')
parsed = json.dumps(result)
jsonfile = open("output_json.json", 'r')
data = json.load(jsonfile)
How can I achive this easily?
EDIT:
print(df['categories'].unique().tolist())
['1,2,3', 1, nan, '1,2,3,6', 9, 8, 11, 4, 5, 2, '1,2,3,4,5,6,7,8,9']
You can use:
df = pd.read_excel('data_threated.xlsx').reset_index(drop=True)
df['categories'] = df['categories'].apply(lambda x: [int(i) for i in x.split(',')] if isinstance(x, str) else '')
df.to_json('output.json', orient='records', indent=4)
Content of output.json
[
{
"model":"xx",
"id":1,
"name":"xyz",
"categories":[
1,
2
]
}
]
Note you can also use:
df['categories'] = pd.eval(df['categories'])

merge lists of dictionaries in terraform v0.12

I would like to do the following using terraform:
I have 2 JSONs:
1.json:
[
{
"description": "description1",
"url": "url1",
"data": "data1"
},
{
"description": "description2",
"url": "url2",
"data": "data2",
"action": "action2"
},
{
"description": "description3",
"url": "url3",
"data": "data3"
}
]
2.json:
[
{
"description": "description1",
"url": "url1",
"data": "data1"
},
{
"description": "description2_new",
"url": "url2",
"data": "data2_new"
},
{
"description": "description4",
"url": "url4",
"data": "data4"
}
]
and I want to merge them into one. Dictionaries from the second JSON should override dictionaries from the first one if url key is the same. I.e. combined JSON should look like:
[
{
"description": "description1",
"url": "url1",
"data": "data1"
},
{
"description": "description2_new",
"url": "url2",
"data": "data2_new"
},
{
"description": "description3",
"url": "url3",
"data": "data3"
},
{
"description": "description4",
"url": "url4",
"data": "data4"
}
]
Using python I can easily do it:
import json
with open('1.json') as f:
json1 = json.load(f)
with open('2.json') as f:
json2 = json.load(f)
def list_to_dict(json_list):
res_dict = {}
for d in json_list:
res_dict[d['url']] = d
return res_dict
def merge_json(json1, json2):
j1 = list_to_dict(json1)
j2 = list_to_dict(json2)
j1.update(j2)
res_list = []
for key in j1.keys():
res_list.append(j1[key])
return res_list
print(json.dumps(merge_json(json1, json2), indent=4))
How can I do that using terraform?
Using terraform 0.12.x
$ cat main.tf
locals {
# read from files and turn into json
list1 = jsondecode(file("1.json"))
list2 = jsondecode(file("2.json"))
# iterate over lists and turn url into a unique key
dict1 = { for item in local.list1 : item.url => item }
dict2 = { for item in local.list2 : item.url => item }
# combine both dictionaries so values converge
# only take its values
merged = values(merge(local.dict1, local.dict2))
}
output "this" {
value = local.merged
}
$ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
this = [
{
"data" = "data1"
"description" = "description1"
"url" = "url1"
},
{
"data" = "data2_new"
"description" = "description2_new"
"url" = "url2"
},
{
"data" = "data3"
"description" = "description3"
"url" = "url3"
},
{
"data" = "data4"
"description" = "description4"
"url" = "url4"
},
]
Terraform supports expanding a list into function parameters using the ... operator. This will allow an arbitrary number of documents to be read.
(I'm not sure, but I believe this feature was added in v0.15)
For this example, I added a new file 3.json with the contents:
[
{
"description": "description4_new",
"url": "url4",
"data": "data4_new"
}
]
For main.tf, I'm using the same logic as #someguyonacomputer's answer:
$ cat main.tf
locals {
jsondocs = [
for filename in fileset(path.module, "*.json") : jsondecode(file(filename))
]
as_dicts = [
for arr in local.jsondocs : {
for obj in arr : obj.url => obj
}
]
# This is where the '...' operator is used
merged = merge(local.as_dicts...)
}
output "as_list" {
value = values(local.merged)
}
Result:
Changes to Outputs:
+ as_list = [
+ {
+ data = "data1"
+ description = "description1"
+ url = "url1"
},
+ {
+ data = "data2_new"
+ description = "description2_new"
+ url = "url2"
},
+ {
+ data = "data3"
+ description = "description3"
+ url = "url3"
},
+ {
+ data = "data4_new"
+ description = "description4_new"
+ url = "url4"
},
]
References:
Terraform Docs -- Function Calls # Expanding Function Arguments

How to loop different types of nested JSON objects multiple times in the same message

Python noob here, again. I'm trying to create a python script to auto-generate a JSON with multiple item but records multiple times using a for loop to generate them, the JSON message is structured and cardinality are as follows:
messageHeader[1]
-item [1-*]
--itemAttributesA [0-1]
--itemAttributesB [0-1]
--itemAttributesC [0-1]
--itemLocaton [1]
--itemRelationships [0-1]
I've had some really good help before for looping through the same object but for one record for example just the itemRelationships record. However as soon as I try to create one message with many items (i.e. 5) and a single instance of an itemAttribute, itemLocation and itemRelationships it does not work as I keep getting a key error. I've tried to define what a keyError is in relation to what I am trying to do but cannot link what I am doing wrong to the examples else where.
Here's my code as it stands:
import json
import random
data = {'messageID': random.randint(0, 2147483647), 'messageType': 'messageType'}
data['item'] = list()
itemAttributeType = input("Please selct what type of Attribute item has, either 'A', 'B' or 'C' :")
for x in range(0, 5):
data['item'].append({
'itemId': "I",
'itemType': "T"})
if itemAttributeType == "A":
data['item'][0]['itemAttributesA']
data['item'][0]['itemAttributesA'].append({
'attributeA': "ITA"})
elif itemAttributeType == "B":
data['item'][0]['itemAttributesB']
data['item'][0]['itemAttributesB'].append({
'attributeC': "ITB"})
else:
data['item'][0]['itemAttributesC']
data['item'][0]['itemAttributesC'].append({
'attributeC': "ITC"})
pass
data['item'][0]['itemLocation'] = {
'itemDetail': "ITC"}
itemRelation = input("Does the item have a relation: ")
if itemRelation > '':
data['item'][0]['itemRelations'] = {
'itemDetail': "relation"}
else:
pass
print(json.dumps(data, indent=4))
I have tried also tried this code which gives me better results:
import json
import random
data = {'messageID': random.randint(0, 2147483647), 'messageType': 'messageType'}
data['item'] = list()
itemAttributeType = input("Please selct what type of Attribute item has, either 'A', 'B' or 'C' :")
for x in range(0, 5):
data['item'].append({
'itemId': "I",
'itemType': "T"})
if itemAttributeType == "A":
data['item'][0]['itemAttributesA'] = {
'attributeA': "ITA"}
elif itemAttributeType == "B":
data['item'][0]['itemAttributesB'] = {
'attributeB': "ITB"}
else:
data['item'][0]['itemAttributesC'] = {
'attributeC': "ITC"}
pass
data['item'][0]['itemLocation'] = {
'itemDetail': "ITC"}
itemRelation = input("Does the item have a relation: ")
if itemRelation > '':
data['item'][0]['itemRelations'] = {
'itemDetail': "relation"}
else:
pass
print(json.dumps(data, indent=4))
This actually gives me a result but gives me messageHeader, item, itemAttributeA, itemLocation, itemRelations, and then four items records at the end as follows:
{
"messageID": 1926708779,
"messageType": "messageType",
"item": [
{
"itemId": "I",
"itemType": "T",
"itemAttributesA": {
"itemLocationType": "ITA"
},
"itemLocation": {
"itemDetail": "location"
},
"itemRelations": {
"itemDetail": "relation"
}
},
{
"itemId": "I",
"itemType": "T"
},
{
"itemId": "I",
"itemType": "T"
},
{
"itemId": "I",
"itemType": "T"
},
{
"itemId": "I",
"itemType": "T"
}
]
}
What I am trying to achieve is this output:
{
"messageID": 2018369867,
"messageType": "messageType",
"item": [{
"itemId": "I",
"itemType": "T",
"itemAttributesA": {
"attributeA": "ITA"
},
"itemLocation": {
"itemDetail": "Location"
},
"itemRelation": [{
"itemDetail": "D"
}]
}, {
"item": [{
"itemId": "I",
"itemType": "T",
"itemAttributesB": {
"attributeA": "ITB"
},
"itemLocation": {
"itemDetail": "Location"
},
"itemRelation": [{
"itemDetail": "D"
}]
}, {
"item": [{
"itemId": "I",
"itemType": "T",
"itemAttributesC": {
"attributeA": "ITC"
},
"itemLocation": {
"itemDetail": "Location"
},
"itemRelation": [{
"itemDetail": "D"
}]
}, {
"item": [{
"itemId": "I",
"itemType": "T",
"itemAttributesA": {
"attributeA": "ITA"
},
"itemLocation": {
"itemDetail": "Location"
},
"itemRelation": [{
"itemDetail": "D"
}]
},
{
"item": [{
"itemId": "I",
"itemType": "T",
"itemAttributesB": {
"attributeA": "ITB"
},
"itemLocation": {
"itemDetail": "Location"
},
"itemRelation": [{
"itemDetail": "D"
}]
}]
}
]
}]
}]
}]
}
I've been at this for the best part of a whole day trying to get it to work, butchering away at code, where am I going wrong, any help would be greatly appreciated
Your close. I think the part your are missing is adding the dict to your current dict and indentation with your for loop.
import json
import random
data = {'messageID': random.randint(0, 2147483647), 'messageType': 'messageType'}
data['item'] = list()
itemAttributeType = input("Please selct what type of Attribute item has, either 'A', 'B' or 'C' :")
for x in range(0, 5):
data['item'].append({
'itemId': "I",
'itemType': "T"})
if itemAttributeType == "A":
# First you need to add `itemAttributesA` to your dict:
data['item'][x]['itemAttributesA'] = dict()
# You could also do data['item'][x] = {'itemAttributesA': = dict()}
data['item'][x]['itemAttributesA']['attributeA'] = "ITA"
elif itemAttributeType == "B":
data['item'][x]['itemAttributesB'] = dict()
data['item'][x]['itemAttributesB']['attributeC'] = "ITB"
else:
data['item'][x]['itemAttributesC'] = dict()
data['item'][x]['itemAttributesC']['attributeC'] = "ITC"
data['item'][x]['itemLocation'] = {'itemDetail': "ITC"}
itemRelation = input("Does the item have a relation: ")
if itemRelation > '':
data['item'][x]['itemRelations'] = {'itemDetail': "relation"}
else:
pass
print(json.dumps(data, indent=4))
This code can also be shortened considerably if your example is close to what you truly desire:
import json
import random
data = {'messageID': random.randint(0, 2147483647), 'messageType': 'messageType'}
data['item'] = list()
itemAttributeType = input("Please selct what type of Attribute item has, either 'A', 'B' or 'C' :")
for x in range(0, 5):
new_item = {
'itemId': "I",
'itemType': "T",
'itemAttributes' + str(itemAttributeType): {
'attribute' + str(itemAttributeType): "IT" + str(itemAttributeType)
},
'itemLocation': {'itemDetail': "ITC"}
}
itemRelation = input("Does the item have a relation: ")
if itemRelation > '':
new_item['itemRelations'] = {'itemDetail': itemRelation}
data['item'].append(new_item)
print(json.dumps(data, indent=4))
Another note: If you want messageID to be truly unique than you should probably look into a UUID; otherwise you may have message ids that match.
import uuid
unique_id = str(uuid.uuid4())
print(unique_id)

Julia | DataFrame conversion to JSON

I have a dataframe in Julia like df = DataFrame(A = 1:4, B = ["M", "F", "F", "M"]). I have to convert it into a JSON like
{
"nodes": [
{
"A": "1",
"B": "M"
},
{
"A": "2",
"B": "F"
},
{
"A": "3",
"B": "F"
},
{
"A": "4",
"B": "M"
}
]
}
Please help me in this.
There isn't a method in DataFrames to do this. In a github issue where the following snippet, using JSON.jl, is offered as a method to write json:
using JSON
using DataFrames
function df2json(df::DataFrame)
len = length(df[:,1])
indices = names(df)
jsonarray = [Dict([string(index) => (isna(df[index][i])? nothing : df[index][i])
for index in indices])
for i in 1:len]
return JSON.json(jsonarray)
end
function writejson(path::String,df::DataFrame)
open(path,"w") do f
write(f,df2json(df))
end
end
JSONTables package provides JSON conversion to/from Tables.jl-compatible sources like DataFrame.
using DataFrames
using JSONTables
df = DataFrame(A = 1:4, B = ["M", "F", "F", "M"])
jsonstr = objecttable(df)

Json format - scala

I need to build a Json format in the following way in scala. How to implement the same ?
{
"name": "protocols",
"children": [
{
"name": "tcp", "children": [
{
"name": "source 1",
"children": [
{
"name": "destination 1",
"children": [
{
"name": "packet 1"
},
{
"name": "packet 4"
}
]
},
{
"name": "destination 2","children": [
{
"name": "packet 1"
},
{
"name": "packet 4"
}
]
},
I need a tree structure like this to be wriiten to a file .
If you are using play, your json structure can be represented with single case class
Here is a sample, where this case class is called Node
import play.api.libs.json.Json
case class Node(name: String, children: List[Node] = Nil)
implicit val format = Json.format[Node]
val childSource1 = Node("destination 1", List(Node("packet 1"), Node("packet 4")))
val childSource2 = Node("destination 2", List(Node("packet 1"), Node("packet 4")))
val source1 = Node("source 1", List(childSource1, childSource2))
val example = Node("protocols", List(Node("tcp", List(source1))))
Json.prettyPrint(Json.toJson(example))