I have this kind of JSON file:
[{
"name": "model1",
"reference": 0.1,
"code": [{
"name": "foo",
"version": "a",
"value": 0.2
},{
"name": "foo",
"version": "b",
"value": 0.2
},{
"name": "bar",
"version": "a",
"value": 0.15
}]
},{
"name": "model2",
"reference": 0.12,
"code": [{
"name": "bar",
"version": "a",
"value": 0.09
},{
"name": "baz",
"version": "a",
"value": 0.15
}]
}]
What I want is to get all the top level objects for a certain code.name, let’s say foo. So I wrote the following query:
.[] | select( .code[].name == "foo" )
The problem is that since there is two sub-objects with foo as name, the top level object is returned twice. I’d like to have it only once, is that possible?
You're looking for any/1.
.[] | select(.code | any(.name == "foo"))
Online demo
Related
I have the following 'SetOfSignals' in KQL (using mv-expand):
"SetOfSignals": {
"name": "CompanyName",
"signals": [
{
"name": "AmbientAirTemperature",
"unit": "C",
"dataType": "Float32",
"values": [
"11.5"
]
},
{
"name": "AverageEnergyConsumption",
"unit": "W",
"dataType": "Float32",
"values": [
"780.0"
]
}
}
and now I want to project the signal names with corresponding values.
I want it to look like this:
...
AmbientAirTemperature
AverageEnergyConsumption
...
11.5
780.0
but using something like | extend AmbientAirTemperature = signals.name doesn't works since there are multiple strings within "signals" with the name "name".
Thanks.
datatable(SetOfSignals:dynamic)
[
dynamic
(
{
"name": "CompanyName",
"signals": [
{
"name": "AmbientAirTemperature",
"unit": "C",
"dataType": "Float32",
"values": [
"11.5"
]
},
{
"name": "AverageEnergyConsumption",
"unit": "W",
"dataType": "Float32",
"values": [
"780.0"
]
}
]
}
)
]
| mv-apply signal = SetOfSignals.signals on
(
summarize make_bag(bag_pack(tostring(signal.name), signal.values[0]))
)
| project-away SetOfSignals
| evaluate bag_unpack(bag_)
AmbientAirTemperature
AverageEnergyConsumption
11.5
780.0
Fiddle
I have several CSV files of item data for a game I'm messing around with that I need to convert to JSON for consumption. The data can be quite irregular with several empty fields per record, which makes for sort of ugly JSON output.
Example with dummy values:
Id,Name,Value,Type,Properties/1,Properties/2,Properties/3,Properties/4
01:Foo:13,Foo,13,ACME,CanExplode,IsRocket,,
02:Bar:42,Bar,42,,IsRocket,,,
03:Baz:37,Baz,37,BlackMesa,CanExplode,IsAlive,IsHungry,
Converted output:
[
{
"Id": "01:Foo:13",
"Name": "Foo",
"Value": 13,
"Type": "ACME",
"Properties": ["CanExplode", "IsRocket", ""]
},
{
"Id": "02:Bar:42",
"Name": "Bar",
"Value": 42,
"Type": "",
"Properties": ["IsRocket", "", ""]
},
{
"Id": "03:Baz:37",
"Name": "Baz",
"Value": 37,
"Type": "BlackMesa",
"Properties": ["CanExplode", "IsAlive", "IsHungry"]
}
]
So far I've been quite successful with using Miller. I've managed to remove completely empty columns from the CSV as well as aggregate the Properties/X columns into a single array.
But now I'd like to do two more things to improve the output format to make consuming the JSON easier:
remove empty strings "" from the Properties array
replace the other empty strings "" (e.g. Type of the second record) with null
Desired output:
[
{
"Id": "01:Foo:13",
"Name": "Foo",
"Value": 13,
"Type": "ACME",
"Properties": ["CanExplode", "IsRocket"]
},
{
"Id": "02:Bar:42",
"Name": "Bar",
"Value": 42,
"Type": null,
"Properties": ["IsRocket"]
},
{
"Id": "03:Baz:37",
"Name": "Baz",
"Value": 37,
"Type": "BlackMesa",
"Properties": ["CanExplode", "IsAlive", "IsHungry"]
}
]
Is there a way to achieve that with Miller?
My current commands are:
mlr -I --csv remove-empty-columns file.csv to clean up the columns
mlr --icsv --ojson --jflatsep '/' --jlistwrap cat file.csv > file.json for the conversion
It's not probably the way you want to do it. I use also jq.
Running
mlr --c2j --jflatsep '/' --jlistwrap remove-empty-columns then cat input.csv | \
jq '.[].Properties|=map(select(length > 0))' | \
jq '.[].Type|=(if . == "" then null else . end)'
you will have
[
{
"Id": "01:Foo:13",
"Name": "Foo",
"Value": 13,
"Type": "ACME",
"Properties": [
"CanExplode",
"IsRocket"
]
},
{
"Id": "02:Bar:42",
"Name": "Bar",
"Value": 42,
"Type": null,
"Properties": [
"IsRocket"
]
},
{
"Id": "03:Baz:37",
"Name": "Baz",
"Value": 37,
"Type": "BlackMesa",
"Properties": [
"CanExplode",
"IsAlive",
"IsHungry"
]
}
]
Using Miller, you can "filter out" the empty fields from each record with:
mlr --c2j --jflatsep '/' --jlistwrap put '
$* = select($*, func(k,v) {return v != ""})
' file.csv
remark: actually, we're building a new record containing the non-empty fields instead of deleting the empty fields from the record; the final result is equivalent though:
[
{
"Id": "01:Foo:13",
"Name": "Foo",
"Value": 13,
"Type": "ACME",
"Properties": ["CanExplode", "IsRocket"]
},
{
"Id": "02:Bar:42",
"Name": "Bar",
"Value": 42,
"Properties": ["IsRocket"]
},
{
"Id": "03:Baz:37",
"Name": "Baz",
"Value": 37,
"Type": "BlackMesa",
"Properties": ["CanExplode", "IsAlive", "IsHungry"]
}
]
I have been playing around with jq to format a json file but I am having some issues trying to solve a particular transformation. Given a test.json file in this format:
[
{
"name": "A", // This would be the first key
"number": 1,
"type": "apple",
"city": "NYC" // This would be the second key
},
{
"name": "A",
"number": "5",
"type": "apple",
"city": "LA"
},
{
"name": "A",
"number": 2,
"type": "apple",
"city": "NYC"
},
{
"name": "B",
"number": 3,
"type": "apple",
"city": "NYC"
}
]
I was wondering, how can I format it this way using jq?
[
{
"key": "A",
"values": [
{
"key": "NYC",
"values": [
{
"number": 1,
"type": "a"
},
{
"number": 2,
"type": "b"
}
]
},
{
"key": "LA",
"values": [
{
"number": 5,
"type": "b"
}
]
}
]
},
{
"key": "B",
"values": [
{
"key": "NYC",
"values": [
{
"number": 3,
"type": "apple"
}
]
}
]
}
]
I have followed this thread Using jq, convert array of name/value pairs to object with named keys and tried to group the json using this expression
jq '. | group_by(.name) | group_by(.city) ' ./test.json
but I have not been able to add the keys in the output.
You'll want to group the items at the different levels and building out your result objects as you want.
group_by(.name) | map({
key: .[0].name,
values: (group_by(.city) | map({
key: .[0].city,
values: map({number,type})
}))
})
Just keep in mind that group_by/1 yields groups in a sorted order. You'll probably want an implementation that preserves that order.
def group_by_unsorted(key_selector):
reduce .[] as $i ({};
.["\($i|key_selector)"] += [$i]
)|[.[]];
I have the following db:
{
"a": [{
"name": "foo",
"thing": [{
"name": "bar",
"lyrics": ["1", "2", "3"]
}]
}, {
"name": "abc",
"thing": [{
"name": "123",
"list": ["one", "two"]
}]
}]
}
I can't seem to query it correctly. These two queries return the same thing, the entire db:
db.test.find({"a.name":"abc"})
db.test.find({"a.name":"foo"})
How do I find one collection instead of the whole db?
I would expect the first query to return:
{
"name": "abc",
"thing": [{
"name": "123",
"list": ["one", "two"]
}]
}
The two queries return the same document, because both queries match it.
This is one document
[{
"a": [{
"name": "foo",
"thing": [{
"name": "bar",
"lyrics": ["1", "2", "3"]
}]
}, {
"name": "abc",
"thing": [{
"name": "123",
"list": ["one", "two"]
}]
}]
}]
This is two documents
[{
"a": [{
"name": "foo",
"thing": [{
"name": "bar",
"lyrics": ["1", "2", "3"]
}]
}]
},
{
"a": [{
"name": "abc",
"thing": [{
"name": "123",
"list": ["one", "two"]
}]
}]
}]
You can get stats on of a collection like so
db.test.stats()
"count" will tell you how many documents are there.
Edit: To add to this, in your collection "test" a document has 1 field, which is "a" and is of type array that holds objects (documents). It has 2 array elements
First
{
"name": "foo",
"thing": [{
"name": "bar",
"lyrics": ["1", "2", "3"]
}]
}
Second
{
"name": "abc",
"thing": [{
"name": "123",
"list": ["one", "two"]
}]
}
Everything inside curly braces {..}, including the braces themselves, is a one single document, i.e. the whole your database contains only one document that you receive for any matching query. To receive the desired result, you have to re-write your JSON document as an array of documents inside square braces [..].
A friend wrote a program in VBA, which generates a json data. I am trying to visualize that data via the pack-layout. We extracted the rules by what the json data is being created from the json data here: http://bl.ocks.org/mbostock/7607535
I went through the data many times myself, I just can't seem to find the problem why it is not being visualized. The browser console claims a problem in line 33 with the token "]" but in my eyes the parenthesis are right and I can't seem to find another mistake.
The visualization works properly with the data from where we extracted the rules.
The question now is, which mistake in the json file prevents the code from being visualized?
Would be amazing if somebody can see this, since we cannot see it. Thanks in advance!
The generated json data looks like this:
{
"name": "While",
"children": [
{"name": "While", "size": 27},
{
"name": "If",
"children": [
{"name": "If", "size": 22},
{
"name": "If",
"children": [
{"name": "If", "size": 3}
]
},
{
"name": "If",
"children": [
{"name": "If", "size": 3}
]
},
{
"name": "If",
"children": [
{"name": "If", "size": 3}
]
},
{
"name": "If",
"children": [
{"name": "If", "size": 3}
]
},
]
},
]
}
You have two commas(,) at the end of some arrays within that JSON of yours - that makes it invalid and prone to errors.
Just edit it and it will work. Use https://jsonformatter.curiousconcept.com/ to check.
The error lies with the script that generates it :)
Here's the fixed version of your JSON:
{
"name": "While",
"children": [{
"name": "While",
"size": 27
}, {
"name": "If",
"children": [{
"name": "If",
"size": 22
}, {
"name": "If",
"children": [{
"name": "If",
"size": 3
}]
}, {
"name": "If",
"children": [{
"name": "If",
"size": 3
}]
}, {
"name": "If",
"children": [{
"name": "If",
"size": 3
}]
}, {
"name": "If",
"children": [{
"name": "If",
"size": 3
}]
}]
}]
}