I have several CSV files of item data for a game I'm messing around with that I need to convert to JSON for consumption. The data can be quite irregular with several empty fields per record, which makes for sort of ugly JSON output.
Example with dummy values:
Id,Name,Value,Type,Properties/1,Properties/2,Properties/3,Properties/4
01:Foo:13,Foo,13,ACME,CanExplode,IsRocket,,
02:Bar:42,Bar,42,,IsRocket,,,
03:Baz:37,Baz,37,BlackMesa,CanExplode,IsAlive,IsHungry,
Converted output:
[
{
"Id": "01:Foo:13",
"Name": "Foo",
"Value": 13,
"Type": "ACME",
"Properties": ["CanExplode", "IsRocket", ""]
},
{
"Id": "02:Bar:42",
"Name": "Bar",
"Value": 42,
"Type": "",
"Properties": ["IsRocket", "", ""]
},
{
"Id": "03:Baz:37",
"Name": "Baz",
"Value": 37,
"Type": "BlackMesa",
"Properties": ["CanExplode", "IsAlive", "IsHungry"]
}
]
So far I've been quite successful with using Miller. I've managed to remove completely empty columns from the CSV as well as aggregate the Properties/X columns into a single array.
But now I'd like to do two more things to improve the output format to make consuming the JSON easier:
remove empty strings "" from the Properties array
replace the other empty strings "" (e.g. Type of the second record) with null
Desired output:
[
{
"Id": "01:Foo:13",
"Name": "Foo",
"Value": 13,
"Type": "ACME",
"Properties": ["CanExplode", "IsRocket"]
},
{
"Id": "02:Bar:42",
"Name": "Bar",
"Value": 42,
"Type": null,
"Properties": ["IsRocket"]
},
{
"Id": "03:Baz:37",
"Name": "Baz",
"Value": 37,
"Type": "BlackMesa",
"Properties": ["CanExplode", "IsAlive", "IsHungry"]
}
]
Is there a way to achieve that with Miller?
My current commands are:
mlr -I --csv remove-empty-columns file.csv to clean up the columns
mlr --icsv --ojson --jflatsep '/' --jlistwrap cat file.csv > file.json for the conversion
It's not probably the way you want to do it. I use also jq.
Running
mlr --c2j --jflatsep '/' --jlistwrap remove-empty-columns then cat input.csv | \
jq '.[].Properties|=map(select(length > 0))' | \
jq '.[].Type|=(if . == "" then null else . end)'
you will have
[
{
"Id": "01:Foo:13",
"Name": "Foo",
"Value": 13,
"Type": "ACME",
"Properties": [
"CanExplode",
"IsRocket"
]
},
{
"Id": "02:Bar:42",
"Name": "Bar",
"Value": 42,
"Type": null,
"Properties": [
"IsRocket"
]
},
{
"Id": "03:Baz:37",
"Name": "Baz",
"Value": 37,
"Type": "BlackMesa",
"Properties": [
"CanExplode",
"IsAlive",
"IsHungry"
]
}
]
Using Miller, you can "filter out" the empty fields from each record with:
mlr --c2j --jflatsep '/' --jlistwrap put '
$* = select($*, func(k,v) {return v != ""})
' file.csv
remark: actually, we're building a new record containing the non-empty fields instead of deleting the empty fields from the record; the final result is equivalent though:
[
{
"Id": "01:Foo:13",
"Name": "Foo",
"Value": 13,
"Type": "ACME",
"Properties": ["CanExplode", "IsRocket"]
},
{
"Id": "02:Bar:42",
"Name": "Bar",
"Value": 42,
"Properties": ["IsRocket"]
},
{
"Id": "03:Baz:37",
"Name": "Baz",
"Value": 37,
"Type": "BlackMesa",
"Properties": ["CanExplode", "IsAlive", "IsHungry"]
}
]
In an overview, let's say I have a CSV file that has 5 entries of data (I will have a large number of entries in the CSV file) that I need to use dynamically while building the JSON payload using python (in Databricks).
test.csv
1a2b3c
2n3m6g
333b4c
2m345j
123abc
payload.json
{
"records": {
"id": "37c8323c",
"names": [
{
"age": "1",
"identity": "Dan",
"powers": {
"key": "plus",
"value": "1a2b3c"
}
},
{
"age": "2",
"identity": "Jones",
"powers": {
"key": "minus",
"value": "2n3m6g"
}
},
{
"age": "3",
"identity": "Kayle",
"powers": {
"key": "multiply",
"value": "333b4c"
}
},
{
"age": "4",
"identity": "Donnis",
"powers": {
"key": "divide",
"value": "2m345j"
}
},
{
"age": "5",
"identity": "Layla",
"powers": {
"key": "power",
"value": "123abc"
}
}
]
}
}
The above payload that I need to construct as a result of multiple names objects in the array and I also would like the value property to read dynamically from the CSV file.
I basically need to append the below JSON object to the existing names array considering the value for the power object from the CSV file.
{
"age": "1",
"identity": "Dan",
"powers": {
"key": "plus",
"value": "1a2b3c"
}
}
Since I'm a newbie in Python, any guides would be appreciated. Thanks to the StackOverflow team in advance.
I am looking to read a JSON file using shell script and getting Id based on the name attribute and store it in a variable without using "jq". Please suggest how it can be done. The json looks like:
{
"elements": [
{
"id": "1",
"internalId": "AA",
"name": "SampleService",
},
{
"id": "2",
"internalId": "BB",
"name": "Loan_Evaluation",
},
{
"id": "3",
"internalId": "CC",
"name": "Miniloan Service",
}
],`
}
I have a JSON example to i would like to transform to Excell file to be able to modify all fields and after that, be able to export Excell file to retrieve the new JSON file updated.
I tryed some online tool like (http://www.convertcsv.com/csv-to-json.htm) but the result is not good : I am able to create a csv file, but not able to convert csv file to json.
Do you know a tool with which i will be able to convert to csv / convert to json ?
JSON example :
[
{
"key": "keyExample",
"type": "typeExample",
"ref": "refExample",
"items": [
{
"itemRef": "aaa",
"count": 1,
"desc": "aaaaaaaaa"
},
{
"itemRef": "bbb",
"count": 2,
"desc": "bbbbbbb"
},
{
"itemRef": "ccc",
"count": 2,
"desc": "ccccccc"
}
]
},
{
"key": "keyExample2",
"type": "typeExample2",
"ref": "refExample2",
"items": [
{
"itemRef": "aaa",
"count": 1,
"desc": "aaaaaaaaa"
},
{
"itemRef": "bbb",
"count": 2,
"desc": "bbbbbbb"
},
{
"itemRef": "ccc",
"count": 2,
"desc": "ccccccc"
}
]
}
]
How to copy the objects of one array of a JSON file to an array of another JSON file using PowerShell? For Example I have one JSON file like:
"type": "Employee",
"Properties": [
{
"Name": "Raj",
"Id": "18111",
"email": "emp1#company.com",
"Position": "Manager",
"DateOfJoining": "16.10.14",
}
],
"Description": "Employee details"
and another JSON file as:
"type": "Employee",
"Properties": [
{
"Name": "Ram",
"Id": "44000",
"email": "emp2#company.com",
"Position": "Admin",
"DateOfJoining": "10.12.14",
},
{
"Name": "Paul",
"Id": "44002",
"email": "emp3#company.com",
"Position": "Programmer",
"DateOfJoining": "10.9.14",
},
],
"Description": "Employee details"
I want to copy the arrays from 1st JSON file to the 2nd JSON file.
You can try something like this:
$c1 = Convert-FromJson (gc file1.json -raw)
$c2 = Convert-FromJson (gc file2.json -raw)
$c3 = $c1.Properties + $c2.Properties
$c3 | ConvertTo-Json