How to convert csv to JSON nested structure - json

I have a json file to store my data and I convert it to CSV to edit my data. But when i convert it to json again it all goes unconstructed. How can i convert my csv to same structure as my old json.
JSON
{
"product": [
{
"id": "item0001",
"category": "12",
"name": "Name1",
"tag": "tag1",
"more": [
{
"id": "1",
"name": "AL"
},
{
"id": "1",
"name": "BS"
}
],
"active": true
},
{
"id": "item0002",
"categoryId": "13",
"name": "Name2",
"tag": "tag2",
"size": "2",
"more": [
{
"id": "2",
"name": "DL"
},
{
"id": "2",
"name": "AS"
}
],
"active": true
}
]
}
CSV
id,categoryId,name,shortcut,more/0/optionId,more/0/price,more/1/optionId,more/1/price,active,more/2/optionId,more/2/price,spanSize
item0001,ab92d2c6-010e-4182-844d-65050e746617,Name1,Shortcut1,1,60,1,70,TRUE,,,
item0002,ab92d2c6-010e-4182-844d-65050e746617,Name2,Shortcut2,2,60,2,70,TRUE,2,2,4

You can use Miller (mlr) to convert you file both ways
https://miller.readthedocs.io/en/latest/flatten-unflatten/
first from JSON to CSV
mlr --ijson --ocsv cat test.json > test.csv
then edit CSV (Visidata is a very nice command line tool for the job)
then convert it back to CSV
mlr --icsv --ojson cat test.csv > test_v2.json
If you want to have some JSON lines structure instead, use --ojonl

Related

How can I clean up empty fields when converting CSV to JSON with Miller?

I have several CSV files of item data for a game I'm messing around with that I need to convert to JSON for consumption. The data can be quite irregular with several empty fields per record, which makes for sort of ugly JSON output.
Example with dummy values:
Id,Name,Value,Type,Properties/1,Properties/2,Properties/3,Properties/4
01:Foo:13,Foo,13,ACME,CanExplode,IsRocket,,
02:Bar:42,Bar,42,,IsRocket,,,
03:Baz:37,Baz,37,BlackMesa,CanExplode,IsAlive,IsHungry,
Converted output:
[
{
"Id": "01:Foo:13",
"Name": "Foo",
"Value": 13,
"Type": "ACME",
"Properties": ["CanExplode", "IsRocket", ""]
},
{
"Id": "02:Bar:42",
"Name": "Bar",
"Value": 42,
"Type": "",
"Properties": ["IsRocket", "", ""]
},
{
"Id": "03:Baz:37",
"Name": "Baz",
"Value": 37,
"Type": "BlackMesa",
"Properties": ["CanExplode", "IsAlive", "IsHungry"]
}
]
So far I've been quite successful with using Miller. I've managed to remove completely empty columns from the CSV as well as aggregate the Properties/X columns into a single array.
But now I'd like to do two more things to improve the output format to make consuming the JSON easier:
remove empty strings "" from the Properties array
replace the other empty strings "" (e.g. Type of the second record) with null
Desired output:
[
{
"Id": "01:Foo:13",
"Name": "Foo",
"Value": 13,
"Type": "ACME",
"Properties": ["CanExplode", "IsRocket"]
},
{
"Id": "02:Bar:42",
"Name": "Bar",
"Value": 42,
"Type": null,
"Properties": ["IsRocket"]
},
{
"Id": "03:Baz:37",
"Name": "Baz",
"Value": 37,
"Type": "BlackMesa",
"Properties": ["CanExplode", "IsAlive", "IsHungry"]
}
]
Is there a way to achieve that with Miller?
My current commands are:
mlr -I --csv remove-empty-columns file.csv to clean up the columns
mlr --icsv --ojson --jflatsep '/' --jlistwrap cat file.csv > file.json for the conversion
It's not probably the way you want to do it. I use also jq.
Running
mlr --c2j --jflatsep '/' --jlistwrap remove-empty-columns then cat input.csv | \
jq '.[].Properties|=map(select(length > 0))' | \
jq '.[].Type|=(if . == "" then null else . end)'
you will have
[
{
"Id": "01:Foo:13",
"Name": "Foo",
"Value": 13,
"Type": "ACME",
"Properties": [
"CanExplode",
"IsRocket"
]
},
{
"Id": "02:Bar:42",
"Name": "Bar",
"Value": 42,
"Type": null,
"Properties": [
"IsRocket"
]
},
{
"Id": "03:Baz:37",
"Name": "Baz",
"Value": 37,
"Type": "BlackMesa",
"Properties": [
"CanExplode",
"IsAlive",
"IsHungry"
]
}
]
Using Miller, you can "filter out" the empty fields from each record with:
mlr --c2j --jflatsep '/' --jlistwrap put '
$* = select($*, func(k,v) {return v != ""})
' file.csv
remark: actually, we're building a new record containing the non-empty fields instead of deleting the empty fields from the record; the final result is equivalent though:
[
{
"Id": "01:Foo:13",
"Name": "Foo",
"Value": 13,
"Type": "ACME",
"Properties": ["CanExplode", "IsRocket"]
},
{
"Id": "02:Bar:42",
"Name": "Bar",
"Value": 42,
"Properties": ["IsRocket"]
},
{
"Id": "03:Baz:37",
"Name": "Baz",
"Value": 37,
"Type": "BlackMesa",
"Properties": ["CanExplode", "IsAlive", "IsHungry"]
}
]

How can I build a json payload from a csv file that has string data separated by line using Python?

In an overview, let's say I have a CSV file that has 5 entries of data (I will have a large number of entries in the CSV file) that I need to use dynamically while building the JSON payload using python (in Databricks).
test.csv
1a2b3c
2n3m6g
333b4c
2m345j
123abc
payload.json
{
"records": {
"id": "37c8323c",
"names": [
{
"age": "1",
"identity": "Dan",
"powers": {
"key": "plus",
"value": "1a2b3c"
}
},
{
"age": "2",
"identity": "Jones",
"powers": {
"key": "minus",
"value": "2n3m6g"
}
},
{
"age": "3",
"identity": "Kayle",
"powers": {
"key": "multiply",
"value": "333b4c"
}
},
{
"age": "4",
"identity": "Donnis",
"powers": {
"key": "divide",
"value": "2m345j"
}
},
{
"age": "5",
"identity": "Layla",
"powers": {
"key": "power",
"value": "123abc"
}
}
]
}
}
The above payload that I need to construct as a result of multiple names objects in the array and I also would like the value property to read dynamically from the CSV file.
I basically need to append the below JSON object to the existing names array considering the value for the power object from the CSV file.
{
"age": "1",
"identity": "Dan",
"powers": {
"key": "plus",
"value": "1a2b3c"
}
}
Since I'm a newbie in Python, any guides would be appreciated. Thanks to the StackOverflow team in advance.

Find the value of key from JSON with Shell Script without using jq

I am looking to read a JSON file using shell script and getting Id based on the name attribute and store it in a variable without using "jq". Please suggest how it can be done. The json looks like:
{
"elements": [
{
"id": "1",
"internalId": "AA",
"name": "SampleService",
},
{
"id": "2",
"internalId": "BB",
"name": "Loan_Evaluation",
},
{
"id": "3",
"internalId": "CC",
"name": "Miniloan Service",
}
],`
}

Convert json to xls and after get back data to json

I have a JSON example to i would like to transform to Excell file to be able to modify all fields and after that, be able to export Excell file to retrieve the new JSON file updated.
I tryed some online tool like (http://www.convertcsv.com/csv-to-json.htm) but the result is not good : I am able to create a csv file, but not able to convert csv file to json.
Do you know a tool with which i will be able to convert to csv / convert to json ?
JSON example :
[
{
"key": "keyExample",
"type": "typeExample",
"ref": "refExample",
"items": [
{
"itemRef": "aaa",
"count": 1,
"desc": "aaaaaaaaa"
},
{
"itemRef": "bbb",
"count": 2,
"desc": "bbbbbbb"
},
{
"itemRef": "ccc",
"count": 2,
"desc": "ccccccc"
}
]
},
{
"key": "keyExample2",
"type": "typeExample2",
"ref": "refExample2",
"items": [
{
"itemRef": "aaa",
"count": 1,
"desc": "aaaaaaaaa"
},
{
"itemRef": "bbb",
"count": 2,
"desc": "bbbbbbb"
},
{
"itemRef": "ccc",
"count": 2,
"desc": "ccccccc"
}
]
}
]

PowerShell- Merging JSON files

How to copy the objects of one array of a JSON file to an array of another JSON file using PowerShell? For Example I have one JSON file like:
"type": "Employee",
"Properties": [
{
"Name": "Raj",
"Id": "18111",
"email": "emp1#company.com",
"Position": "Manager",
"DateOfJoining": "16.10.14",
}
],
"Description": "Employee details"
and another JSON file as:
"type": "Employee",
"Properties": [
{
"Name": "Ram",
"Id": "44000",
"email": "emp2#company.com",
"Position": "Admin",
"DateOfJoining": "10.12.14",
},
{
"Name": "Paul",
"Id": "44002",
"email": "emp3#company.com",
"Position": "Programmer",
"DateOfJoining": "10.9.14",
},
],
"Description": "Employee details"
I want to copy the arrays from 1st JSON file to the 2nd JSON file.
You can try something like this:
$c1 = Convert-FromJson (gc file1.json -raw)
$c2 = Convert-FromJson (gc file2.json -raw)
$c3 = $c1.Properties + $c2.Properties
$c3 | ConvertTo-Json