I have the following JSON;
[
{
"id": 1,
"start": "2022-06-20",
"result": 24
},
{
"id": 2,
"start": "2022-06-21",
"result": 56
},
{
"id": 3,
"start": "2022-06-21",
"result": 78
}
]
I'm wanting to add 2 new values to each array above using JQ, dimension and date, but date needs to be a copy of the existing key value start. The expected output is as below;
[
{
"id": 1,
"start": "2022-06-20",
"result": 24,
"date": "2022-06-20",
"dimension": "new"
},
{
"id": 2,
"start": "2022-06-21",
"result": 56,
"date": "2022-06-21",
"dimension": "new"
},
{
"id": 3,
"start": "2022-06-21",
"result": 78,
"date": "2022-06-21",
"dimension": "new"
}
]
The jq I have at present can add the new key dimension, but I can't figure out how to copy start -> date
jq '.[] += {"dimension": "new"}' input.json
Thanks for any help
Just create the new key / value pairs.
jq 'map(.date = .start | .dimension = "new")' input.json
You might have tried the following:
.[] += { date: .start, dimension: "new" } // WRONG
But it's not quite right since . is the array, not the element of the array. You can use |= as a topicalizer.
.[] |= ( . += { date: .start, dimension: "new" } )
But I'd use map instead.
map( . += { date: .start, dimension: "new" } )
Alternatively,
. += { date: .start, dimension: "new" }
can also also be achieved using
.date = .start | .dimension: "new"
So you could use
.[] |= ( .date = .start | .dimension: "new" )
or
map( .date = .start | .dimension = "new" )
Related
Please help me, guys.
I have the following:
{
"id": 88888,
"name": "TESTING",
"online": 1,
"test_list": [
{
"id": 0,
"type": 34,
"name": "Blabla",
"used": 9870,
"rate": 31768897891,
"temp": 0,
"percent": 100,
"enabled": 0,
"reset": 0,
"temp": 2
},
{
"id": 1,
"type": 23,
"name": "AlbAlb",
"used": 0865,
"rate": 317681675432345678951,
"temp": 1,
"percent": 99,
"enabled": 0,
"reset": 0,
"target": 2
},
{
"id": 2,
"type": 98,
"name": "TztS",
"used": 65,
"rate": 6324,
"temp": 0,
"percent": 80,
"enabled": 0,
"reset": 0,
"target": 2
}
],
"info": {
"version": "1.0",
"no": "123",
"sw": "321",
"version_detail": {
"date": "2021-03-30",
"build": "11",
"cot": "dfgdfgd"
}
}
}
And i need to do the following:
[
{
"name": "TESTING",
"online": 1
},
{
"id0": 0,
"type0": 34,
"name0": "Blabla",
"rate0": 31768897891,
"temp0": 0,
"percent0": 100
"id1": 1,
"type1": 23,
"name1": "AlbAlb",
"rate1": 317681675432345670000,
"temp1": 1,
"percent1": 99
"id2": 2,
"type2": 98,
"name2": "TztS",
"rate2": 6324,
"temp2": 0,
"percent2": 80
}
]
I want to join the information, what I need into blocks and then set the key numbering by id or iterate.
Will it be possible to somehow realised it using jq and man in one line?
After a long time, I managed to do only the following:
jq '[. | {name,online}] + [.test_list[] | {id,type,name,rate,temp,percent} | with_entries(.key |= . + "0")]'
I tried using map tool, but it doesn't work = (( Help Please...
Here's a reduce-free approach: it uses with_entries on an array to do the counting:
[ {name, online},
(.test_list
| with_entries(.key |= tostring
| .key as $k
| .value |= with_entries(.key += $k))
| add
)
]
The key is with_entries. One possibility would be to use reduce:
[ {name, online},
(.test_list
| . as $in
| reduce range(0; length) as $i ({}; . + ($in[$i] | with_entries(.key = "\(.key)\($i)")))
)
]
Please note that your example output is not quite correct as JSON, and that with your example input, the above jq program produces slightly different results, but if you want to remove certain key-value pairs, you can easily do that, e.g. as a preliminary step in the pipeline.
I am very beginner into jq and I want to reshape my JSON file.
I have ve got JSON structured like this:
{
"a": [1, 2, 3, 4 ...],
"b": [
{
"x": 1000,
"value": 1
},
{
"x": 1000,
"value": 2
},
{
"x": 1000,
"value": 3
}
...
]
}
I am wondering how I can achieve result like this with jq:
[
{
"value": 1,
"from": "a",
},
{
"value": 2,
"from": "a"
},
...
{
"value": 1,
"from": "b"
},
{
"value": 2,
"from": "b"
}
...
]
Here is a very slightly generic, but hardly robust, solution:
map_values( if type == "array"
then map(if type == "object" then .value else . end)
else . end)
| [ keys_unsorted[] as $k
| .[$k][] as $v
| { value: $v, from: $k } ]
Create two lists, one from .a and one from .b, and merge them with +.
In the first list, create objects with value: set to the original content and add from: "a"; in the second list, remove .x from elements of .b and add the from again.
jq '[.a[] | {value:(.), from: "a"}] + [.b[] | del(.x) + {from: "b"}]
I have two JSON files.
file1.json:
{
"Fruits": [
{
"name": "Apple",
"something_else": 123,
"id": 652090
},
{
"name": "Orange",
"something_else": 456,
"id": 28748
}
]}
file2.json:
{
"Fruits": [
{
"weight": 5,
"id": 652090
},
{
"weight": 7,
"id": 28748
}
]}
I want to combine objects from both files if they have a common key 'id', but to extract only 'name' property from file1. How do I do that using jq?
This is what I want to get:
{
"Fruits": [
{
"name": "Apple",
"weight": 5,
"id": 652090
},
{
"name": "Orange",
"weight": 7,
"id": 28748
},
]}
Combine Fruits arrays, group it by id, select groups with 2 elements because we want fruits present in both files. For each selected group; add name field from first group element to second, and collect results in an array.
jq -n '[inputs.Fruits[]]
| reduce (group_by(.id)[] | select(length==2)) as $f
([]; . + [$f[1] + ($f[0] | {name})])' file1.json file2.json
Note that the order files are given on the command line is important, the file with names should be given before the other.
Combining objects with same id and extracting a subset of fields is way much easier though:
jq -n '[inputs.Fruits[]]
| group_by(.id)
| map(select(length==2) | add | {name, id, weight})
' file1.json file2.json
There's plenty of ways this could be constructed. Here's another way:
$ jq '.Fruits |= (. + input.Fruits | [group_by(.id)[] | add | {name,weight,id}])' \
file1.json file2.json
{
"Fruits": [
{
"name": "Orange",
"weight": 7,
"id": 28748
},
{
"name": "Apple",
"weight": 5,
"id": 652090
}
]
}
I have a json file that I need to convert to a csv file, but I am a little wary of trusting a json-to-csv converter site as the outputted data seems to be incorrect... so I was hoping to get some help here!
I have the following json file structure:
{
"GroupName": "GrpName13",
"Number": 3,
"Notes": "Test Group ",
"Units": [
{
"UnitNumber": "TestUnit13",
"DataSource": "Factory",
"ContractNumber": "TestContract13",
"CarNumber": "2",
"ControllerTypeMessageId" : 4,
"NumberOfLandings": 4,
"CreatedBy": "user1",
"CommissionModeMessageId": 2,
"Details": [
{
"DetailName": "TestFloor13",
"DetailNumber": "5"
}
],
"UnitDevices": [
{
"DeviceTypeMessageId": 1,
"CreatedBy": "user1"
}
]
}
]
}
The issue I think Im seeing is that the converters seem to not be able to comprehend the many nested data values. And the reason I think the converters are wrong is because when I try to convert back to json using them, I dont receive the same structure.
Does anyone know how to manually format this json into csv format, or know of a reliable converter than can handle nested values?
Try
www.json-buddy.com/convert-json-csv-xml.htm
if not working for you then you can try this tool
http://download.cnet.com/JSON-to-CSV/3000-2383_4-76680683.html
should be helpful!
I have tried your json on this for url:
http://www.convertcsv.com/json-to-csv.htm
As a result:
UnitNumber,DataSource,ContractNumber,CarNumber,ControllerTypeMessageId,NumberOfLandings,CreatedBy,CommissionModeMessageId,Details/0/DetailName,Details/0/DetailNumber,UnitDevices/0/DeviceTypeMessageId,UnitDevices/0/CreatedBy
TestUnit13,Factory,TestContract13,2,4,4,user1,2,TestFloor13,5,1,user1
Because it could save the path of the key,like the 'DeviceTypeMessageId' in list 'UnitDevices': it will named the columns name with 'UnitDevices/0/DeviceTypeMessageId', this could avoid the same name mistake, so you can get the columns name by its converter rules.
Hope helpful.
Here is a solution using jq
If the file filter.jq contains
def denormalize:
def headers($p):
keys_unsorted[] as $k
| if .[$k]|type == "array" then (.[$k]|first|headers("\($p)\($k)_"))
else "\($p)\($k)"
end
;
def setup:
[
keys_unsorted[] as $k
| if .[$k]|type == "array" then [ .[$k][]| setup ]
else .[$k]
end
]
;
def iter:
if length == 0 then []
elif .[0]|type != "array" then
[.[0]] + (.[1:] | iter)
else
(.[0][] | iter) as $x
| (.[1:] | iter) as $y
| [$x[]] + $y
end
;
[ headers("") ], (setup | iter)
;
denormalize | #csv
and data.json contains (note extra samples added)
{
"GroupName": "GrpName13",
"Notes": "Test Group ",
"Number": 3,
"Units": [
{
"CarNumber": "2",
"CommissionModeMessageId": 2,
"ContractNumber": "TestContract13",
"ControllerTypeMessageId": 4,
"CreatedBy": "user1",
"DataSource": "Factory",
"Details": [
{
"DetailName": "TestFloor13",
"DetailNumber": "5"
}
],
"NumberOfLandings": 4,
"UnitDevices": [
{
"CreatedBy": "user1",
"DeviceTypeMessageId": 1
},
{
"CreatedBy": "user10",
"DeviceTypeMessageId": 10
}
],
"UnitNumber": "TestUnit13"
},
{
"CarNumber": "99",
"CommissionModeMessageId": 99,
"ContractNumber": "Contract99",
"ControllerTypeMessageId": 99,
"CreatedBy": "user99",
"DataSource": "Another Factory",
"Details": [
{
"DetailName": "TestFloor99",
"DetailNumber": "99"
}
],
"NumberOfLandings": 99,
"UnitDevices": [
{
"CreatedBy": "user99",
"DeviceTypeMessageId": 99
}
],
"UnitNumber": "Unit99"
}
]
}
then the command
jq -M -r -f filter.jq data.json
will produce
"GroupName","Notes","Number","Units_CarNumber","Units_CommissionModeMessageId","Units_ContractNumber","Units_ControllerTypeMessageId","Units_CreatedBy","Units_DataSource","Units_Details_DetailName","Units_Details_DetailNumber","Units_NumberOfLandings","Units_UnitDevices_CreatedBy","Units_UnitDevices_DeviceTypeMessageId","Units_UnitNumber"
"GrpName13","Test Group ",3,"2",2,"TestContract13",4,"user1","Factory","TestFloor13","5",4,"user1",1,"TestUnit13"
"GrpName13","Test Group ",3,"2",2,"TestContract13",4,"user1","Factory","TestFloor13","5",4,"user10",10,"TestUnit13"
"GrpName13","Test Group ",3,"99",99,"Contract99",99,"user99","Another Factory","TestFloor99","99",99,"user99",99,"Unit99"
I have a json of this structure:
{
"nodes": {
"60e327ee58a0": {
"nodeinfo": {
"network": {
"mesh": {
"bat0": {
"interfaces": {
"wireless": [
"<mac-address-removed>"
],
"tunnel": [
"<mac-address-removed>"
]
}
}
},
"mac": "<mac removed>",
"addresses": [
"<ipv6 removed>",
"<ipv6 removed>"
]
},
"hardware": {
"model": "TP-Link TL-WR841N/ND v10",
"nproc": 1
},
"software": {
"batman-adv": {
"compat": 15,
"version": "2015.1"
},
"autoupdater": {
"branch": "stable",
"enabled": true
},
"firmware": {
"release": "v2016.1+1.0.1",
"base": "gluon-v2016.1"
},
"status-page": {
"api": 1
},
"fastd": {
"enabled": true,
"version": "v17"
}
},
"hostname": "Antoniusweg12",
"system": {
"site_code": "ffmsd03"
},
"node_id": "60e327ee58a0"
},
"lastseen": "2016-04-14T12:39:04",
"flags": {
"gateway": false,
"online": true
},
"firstseen": "2016-03-16T15:14:04",
"statistics": {
"clients": 1,
"gateway": "de:ad:be:ef:43:02",
"rootfs_usage": 0.6041666666666667,
"loadavg": 0.09,
"uptime": 1822037.41,
"memory_usage": 0.8124737210932025,
"traffic": {
"rx": {
"packets": 50393821,
"bytes": 5061895206
},
"forward": {
"packets": 173,
"bytes": 17417
},
"mgmt_rx": {
"packets": 47453745,
"bytes": 6623785282
},
"tx": {
"packets": 1205695,
"bytes": 173509528,
"dropped": 5683
},
"mgmt_tx": {
"packets": 37906725,
"bytes": 11475209742
}
}
}
},
"30b5c2b042f4": {
<next block...>
And I want to query it with jq for the hostname, the mac or the IPv6.
cat nodes.json |jq -c '.nodes[] | select(.nodes[]| contains("Antoniusweg12"))'
Most examples do not fit this kind of json structure as the objects have an index
Thanks for help in advance.
If you're going to filter, you need to drill down to the property that you want to check for and see if it matches your criteria. You can't expect to just give a name and you'll magically be presented with the results you want.
Searching by hostname, it is found on the .nodeinfo.hostname property of each node:
$ jq -c --arg hostname "Antoniusweg12" \
'.nodes[] | select(.nodeinfo.hostname == $hostname)' nodes.json
Similarly for the mac address, it's found on the .nodeinfo.network.mac property:
$ jq -c --arg mac "aa:bb:cc:dd:ee:ff" \
'.nodes[] | select(.nodeinfo.network.mac == $mac)' nodes.json
For the ip addresses, there's an array of them but it's not that much different in the query. They're found on the .nodeinfo.network.addresses property:
$ jq -c --arg ip "aaaa:bbbb:cccc:dddd::1" \
'.nodes[] | select(.nodeinfo.network.addresses[] == $ip)' nodes.json
Here's another take on the question. Suppose you want to find all occurrences of the key "hostname" for which the value is "Antoniusweg12",
no matter where the key/value combination occurs.
The following will reveal the path to the key/value combination of interest:
paths as $p
| select ( $p[-1] == "hostname" and getpath($p) == "Antoniusweg12" )
| $p
The result for the given input JSON:
[
"nodes",
"60e327ee58a0",
"nodeinfo",
"hostname"
]
If you wanted the path to the containing object, then replace the final $p with $p[0:-1]; and if you want the containing object itself: getpath($p[0:-1])
Here is a solution which searches for nodes where the specified $needle is present in any of the addresses, mac or hostname fields.
"<ipv6 removed>" as $needle # set to whatever you like
| foreach (.nodes|keys[]) as $k (
.
; .
; ( .nodes[$k].nodeinfo.network.addresses?
+ [ .nodes[$k].nodeinfo.network.mac?
, .nodes[$k].nodeinfo.hostname?
]
) as $haystack
| if $haystack | index($needle)
then {($k): .nodes[$k]}
else empty
end
)
EDIT: I now realize a filter of the form foreach E as $X (.; .; R) can almost always be rewritten as E as $X | R so the above is really just
"<ipv6 removed>" as $needle
| (.nodes|keys[]) as $k
| ( .nodes[$k].nodeinfo.network.addresses?
+ [ .nodes[$k].nodeinfo.network.mac?
, .nodes[$k].nodeinfo.hostname?
]
) as $haystack
| if $haystack | index($needle)
then {($k): .nodes[$k]}
else empty
end