Removing and adding key-value from JSON file using jq - json

I have an json file as follows below-
{
"key1": [
"value1"
],
"key2": [
"value2"
],
"key3": [
"value3"
],
"key4": {
"name": "value4"
},
"key5": [
{
"field1": "abc",
"field2": "xyz"
}
]
}
I want to remove field 2 from key5 array and add another field say field3 in key5 array using jq
I tried various way but couldn't figure it how to do this in single command.
Can you please help . Thanks in advance

.key5[] |= ( ... ) allows you to modify each element of the array found at .key5.
Therein, we can use the usual commands for deleting and adding fields.
jq '.key5[] |= ( del( .field2 ) | .field3 = "foo" )'
Demo in jqplay
Here's a variation where the value to add is provided as a command-line argument:
jq --arg val foo '.key5[] |= ( del( .field2 ) | .field3 = $val )'

This assumes that the insert and deletion keys (field2 and field3) are also dynamic, not just the inserted value.
Use --arg and --argjson to initialize variables with values from outside the filter, here field2 as a string and {"field3": "new"} as a JSON object
Use the update operator |= on the element you want to change, here .key5[].
Note: .key5 is an array, not an object, thus it has no fields. But the array does contain objects as its elements. Using .key5[] to access the array elements will update all objects in the array. To update just one of them, say, the first one, use .key[0] instead.
Use delpaths to delete a given path, here the top-level field [$delkey].
Use simple addition + to add a new key/value pair as object, here $add.
jq --arg delkey 'field2' --argjson addobj '{"field3": "new"}' '
.key5[] |= delpaths([[$delkey]]) + $addobj
' input.json
{
"key1": [
"value1"
],
"key2": [
"value2"
],
"key3": [
"value3"
],
"key4": {
"name": "value4"
},
"key5": [
{
"field1": "abc",
"field3": "new"
}
]
}
Demo
If you want to provide the new object's key and value separately, i.e. as strings, not as a pre-composed JSON, you need a third input variable
jq --arg delkey 'field2' --arg addkey 'field3' --arg addval 'new' '
.key5[] |= (delpaths([[$delkey]]) | .[$addkey] = $addval)
' input.json
Demo

Try filter:
del(.key5[0].field2) | .key5[0] |= .+{field3:"foo"}
Output:
{
"key1": [
"value1"
],
"key2": [
"value2"
],
"key3": [
"value3"
],
"key4": {
"name": "value4"
},
"key5": [
{
"field1": "abc",
"field3": "foo"
}
]
}
Demo
https://jqplay.org/s/EB8MpWRxU2

Related

jq - get dict element based on key regex

I'm working with a JSON object having the following structure:
{
"key-foo-1.0": [
{
"key1": "foo",
"key2": "bar",
"id": "01"
},
{
"key1": "foo",
"key2": "bar",
"id": "23"
}
],
"key-bar-1.0": [
{
"key1": "foo",
"key2": "bar",
"id": "45"
},
{
"key1": "foo",
"key2": "bar",
"id": "67"
}
],
"key-baz-1.0": [
{
"key1": "foo",
"key2": "bar",
"id": "89"
}
]
}
I want to get all the id values where the "parent" key name matches the pattern .*foo.* or .*bar.*.
So in my example something like this:
cat json | jq <some filter>
01
23
45
67
Based on https://unix.stackexchange.com/questions/443884/match-keys-with-regex-in-jq I tried:
$ cat json | jq 'with_entries(if (.key|test(".*foo.*$")) then ( {key: .key, value: .value } ) else empty end )'
{
"key-foo-1.0": [
{
"key1": "foo",
"key2": "bar",
"id": "01"
},
{
"key1": "foo",
"key2": "bar",
"id": "23"
}
]
}
But I don't really know how to continue.
I also think there is a better/simpler solution.
You could go with:
jq -r '.[keys_unsorted[] | select(test(".*foo.*|.bar.."))][].id'
01
23
45
67
This gathers all keys using keys_unsorted, then selects those matching the regular expression in test. The wrapping .[…] descends into them, the following [] iterates over the children, and .id outputs the values as raw text using the -r flag.
you can use the following JQ expression:
jq 'to_entries[] | select(.key | test(".*foo.*|.*bar.*")) | .value[] | .id'
JQ playground example

Transforming json file using jq so data ends up under a common object

I have some json data i get from an API that i need to transform using jq into another json format for later use with ansible as part of an inventory script.
What i have is something like:
{
"results": [
{
"name": "hostname1",
"key1": "somevalue1",
"key2": "somevalue2",
"key3": "somevalue3"
},
{
"name": "hostname2",
"key1": "somevalue12",
"key2": "somevalue22",
"key3": "somevalue32"
},
{
"name": "hostname3",
"key1": "somevalue13",
"key2": "somevalue23",
"key3": "somevalue33"
}
]
}
and i need to transform this to look like this:
{
"_meta": {
"hostvars": {
"hostname1": {
"name": "hostname1",
"key1": "somevalue1",
"key2": "somevalue2",
"key3": "somevalue3"
},
"hostname2": {
"name": "hostname2",
"key1": "somevalue12",
"key2": "somevalue22",
"key3": "somevalue32"
},
"hostname3": {
"name": "hostname3",
"key1": "somevalue13",
"key2": "somevalue23",
"key3": "somevalue33"
}
}
}
}
Things i have not been able to figure out.
If i use something like:
cat input.json | jq '{_meta: { hostvars: { (.results[].name): (.results[]) } }}'
Then _meta and hostvars is repeated for each object in the input and that is not at all what i want, i need a common "header" and then the data under there.
Ideally i would like to also exclude the "name" part in the output since it is already used and duplicated, but this is just a bonus.
Advice on how to do this? or is the filter in jq always run against one object at a time?
I experimented a bit with --slurp but didn't get anywhere
The crucial part is: Take an array of objects and transform it into one (outer) object where the attribute names of that outer object are given by some attribute value of the inner objects. That's the job description for INDEX(filter).
So:
{ _meta : { hostvars: ( .results | INDEX(.name) ) } }
Using .results[] twice will iterate over the same list twice, giving you the cartesian product of each host with each of the other objects ( 3 x 3 = 9 ). You need to reference it once!
You can do something like below. The key to the logic is forming an array of objects, firstly by making the key name as .name and the value as the whole sub-object inside.
Once you have the array, you can un-wrap into a comma-separated list of objects using add.
{ _meta : { hostvars: ( ( .results | map( { ( .name ) : . } ) | add ) ) } }
Demo - jqplay
Yet another approach using from_entries could be:
jq '.results | map({key: .name, value: .}) | {_meta: {hostvars: from_entries}}'
{
"_meta": {
"hostvars": {
"hostname1": {
"name": "hostname1",
"key1": "somevalue1",
"key2": "somevalue2",
"key3": "somevalue3"
},
"hostname2": {
"name": "hostname2",
"key1": "somevalue12",
"key2": "somevalue22",
"key3": "somevalue32"
},
"hostname3": {
"name": "hostname3",
"key1": "somevalue13",
"key2": "somevalue23",
"key3": "somevalue33"
}
}
}
}
Demo
You'll probably want to take advantage of jq pipes.
cat input.json | jq '{ _meta : { hostvars : (.results | map({key : .name, value : del(. | .name) }) | from_entries) }}'
Map these results as if they were entries to a new set of entries where the key is .name, also removing the .name from .value as you do so:
.results | map({key : .name, value : del(. | .name) })
Then, form an object from the entries:
... | from_entries
NOTE: See Inian's answer for why the array syntax used by the OP does not work.

Use JQ Select to format a JSON data

Use JQ Select to format a JSON data
JSON Data
{
"header": [
"key1: value1",
"key2: value2",
"key3: value3"
]
}
Need to convert the header array into object:
{
"header": {
"key1": "value1",
"key2": "value2",
"key3": "value3"
}
}
I have tried using
{header: [.header[]]}
But this converts into an array
{
"header": [
"key1: value1",
"key2: value2",
"key3: value3"
]
}
.header |= (map(split(": ") | { (.[0]): .[1] }) | add)
Will produce:
{
"header": {
"key1": "value1",
"key2": "value2",
"key3": "value3"
}
}
Try it online
Update the header key .header |=
map() over the array items
split() on : to get the key and value
Create an object with first index as key, second as value { (.[0]): .[1] }
add them together to create a single object

Use jq to parse key path for each each leaf

(I’m not sure the technical terms to use but can update the question if someone can clarify the terminology I’m lacking for what I'm trying to do. It might help someone find this answer in the future.)
Given the input JSON, how would I use jq to produce the expected output?
Input:
{
"items": {
"item1": {
"part1": {
"a": {
"key1": "value",
"key2": "value"
},
"b": {
"key1": "value",
"key2": "value"
}
},
"part2": {
"c": {
"key1": "value",
"key2": "value"
},
"d": {
"key1": "value",
"key2": "value"
}
}
},
"item2": {
"part3": {
"e": {
"key1": "value",
"key2": "value"
},
"f": {
"key1": "value",
"key2": "value"
}
},
"part4": {
"g": {
"key1": "value",
"key2": "value"
},
"h": {
"key1": "value",
"key2": "value"
}
}
}
}
}
Expected output:
{
"item1": [
"part1.a",
"part1.b",
"part2.c",
"part2.d"
]
"item2": [
"part3.e",
"part3.f"
"part4.g",
"part4.h"
]
}
Try this:
.items | map_values([path(.[][]) | join(".")])
Online demo
Each output path will contain as many path components as the number of []s in the .[][] part; in other words, if you change .[][] to .[][][], for example, you'll see part1.a.key1, part1.a.key2, etc.
This would do it:
# Output: a stream
def keyKey:
keys_unsorted[] as $k | $k + "." + (.[$k] | keys_unsorted[]);
.items | map_values([keyKey])
Some aspects are underspecified. For instance, you don't specify how deep the aggregation should go for the array items. Is it always two levels deep, or is it the whole tree but the last level?
Here's one way how you would go two levels deep with the keys sorted alphabetically:
jq '.items | .[] |= [keys[] as $k | $k + "." + (.[$k] | keys[])]'
Demo
Here's another way how to go down until the second-to-last level:
jq '.items | .[] |= ([path(.. | scalars)[:-1] | join(".")] | unique)'
Demo
Output:
{
"item1": [
"part1.a",
"part1.b",
"part2.c",
"part2.d"
],
"item2": [
"part3.e",
"part3.f",
"part4.g",
"part4.h"
]
}
the unique sequence of jq paths of 'keys' to each and every leaf
is returned from json2jqpath.jq
json2jqpath.jq dat.json
.
.items
.items|.item1
.items|.item1|.part1
.items|.item1|.part1|.a
.items|.item1|.part1|.a|.key1
.items|.item1|.part1|.a|.key2
.items|.item1|.part1|.b
.items|.item1|.part1|.b|.key1
.items|.item1|.part1|.b|.key2
.items|.item1|.part2
.items|.item1|.part2|.c
.items|.item1|.part2|.c|.key1
.items|.item1|.part2|.c|.key2
.items|.item1|.part2|.d
.items|.item1|.part2|.d|.key1
.items|.item1|.part2|.d|.key2
.items|.item2
.items|.item2|.part3
.items|.item2|.part3|.e
.items|.item2|.part3|.e|.key1
.items|.item2|.part3|.e|.key2
.items|.item2|.part3|.f
.items|.item2|.part3|.f|.key1
.items|.item2|.part3|.f|.key2
.items|.item2|.part4
.items|.item2|.part4|.g
.items|.item2|.part4|.g|.key1
.items|.item2|.part4|.g|.key2
.items|.item2|.part4|.h
.items|.item2|.part4|.h|.key1
.items|.item2|.part4|.h|.key2
It is not the output you asked for but as another noted, your question may be somewhat under specified. starting from a preprocessed structure such as this has the advantage of reducing every json file to its set of paths to start fiddling with.
json2jqpath

jq map object key value to array of objects containing both

I would like to put an object parent key inside the object itself and convert each key value pair to an array
Given:
{
"field1": {
"key1": 11,
"key2": 10
},
"field2": {
"key1": 11,
"key2": 10
}
}
Desired output
[
{"name": "field1", "key1": 11, "key2": 10},
{"name": "field2", "key1": 11, "key2": 10}
]
I know that jq keys would give me ["field1", "field2"] and jq '[.[]]' would give
[
{ "key1": 11, "key2": 10 },
{ "key1": 11, "key2": 10 }
]
I cannot figure out a way to combine them, how should I go about it?
Generate an object in {"name": <key>} form for each key, and merge that with the key's value.
to_entries | map({name: .key} + .value)
or:
[keys_unsorted[] as $k | {name: $k} + .[$k]]
Something like below. Get the list of keys in the JSON using keys[] and add the new field name by indexing key on each object.
jq '[ keys[] as $k | { name: $k } + .[$k] ]'
If you want the ordering of keys maintained, use keys_unsorted[].