Non destructive assignation with jq - json

I got the following data:
{
"things": [
{
"name": "lkj",
"something": [
"hike"
],
"more_data": "important",
"other_stuff": "very important"
},
{
"name": "iou",
"different_more_data": "very important too",
"more_different_data": [
"even more"
]
}
]
}
Each of things has an id called "name", with jq I can edit it like:
jq '(.things[]) |= {name,something:["changed"]}'
{
"things": [
{
"name": "lkj",
"something": [
"changed"
]
},
...
Unfortunately I lose everything not declared in the right hand of the assignation operation.
Is there a way to make assignations without losing data? So that the result is like this:
{
"things": [
{
"name": "lkj",
"something": [
"changed"
],
"more_data": "important",
"other_stuff": "very important"
},
{
"name": "iou",
"something": [
"changed"
],
"different_more_data": "very important too",
"more_different_data": [
"even more"
]
}
]
}

You can simply modify your query so that it looks like:
.things[] |= (.something = ["changed"])
You can also use |= (or one of its siblings, such as +=) instead of = in the RHS expression, e.g.
.things[] |= (.something += ["changed"])
If you want to update some, but not all, items, you can still use the above forms. A straightforward approach is to use if ... then ... else ... end, for example:
.things[] |= (if .name == "lkj" then .something = ["changed"] else . end)
Using select on the LHS of |=
jq (or at least jq since version 1.4) does support the use of select on the LHS of |=, e.g.
(.things[] | select(.name=="lkj")) |= (.something += ["changed"])

With jq's map function:
jq '.things |= map(.something = ["changed"])' jsonfile
map(x) - apply specified filter x for each item of the input array
.something = ["changed"] - set key something to an object with array ["changed"] as a value
The output:
{
"things": [
{
"name": "lkj",
"something": [
"changed"
],
"more_data": "important",
"other_stuff": "very important"
},
{
"name": "iou",
"different_more_data": "very important too",
"more_different_data": [
"even more"
],
"something": [
"changed"
]
}
]
}

Related

How to search within sections of a JSON File?S

So, lets say I had a JSON File like this:
{
"content": [
{
"word": "cat",
"adjectives": [
{
"type": "textile",
"adjective": "fluffy"
},
{
"type": "visual",
"adjective": "small"
}
]
},
{
"word": "dog",
"adjectives": [
{
"type": "textile",
"adjective": "fluffy"
},
{
"type": "visual",
"adjective": "big"
}
]
},
{
"word": "chocolate",
"adjectives": [
{
"type": "visual",
"adjective": "small"
},
{
"type": "gustatory",
"adjective": "sweet"
}
]
}
]
}
Now, say I wanted to search for two words. For example, "Fluffy" and "Small." The problem with this is that both words' adjectives contain small, and so I would have to manually search for which one contains fluffy. So, how would I do this in a quicker manner?
In other words, how would I find the word(s) with both "fluffy" and "small"
EDIT: Sorry, new asker. Anything that words in a terminal is fair game. jq is a really great JSON searcher, and so this is preferred, and sorry for the confusion. I also fixed the JSON
A command-line solution would be to use jq:
jq -r '.content[] | select(.adjectives[].adjective == "fluffy") | .word' /pathToJsonFile.json
Output:
cat
Are you looking for something like this? Do you need a solution that uses other programming languages?
(P.S. your JSON example appears to be invalid)
Since jq is now fair game (this was only clarified later in the comments), here is one solution using jq.
First, fix the JSON to be actually valid:
{
"content": [
{
"word": "cat",
"adjectives": [
{
"type": "textile",
"adjective": "fluffy"
},
{
"type": "visual",
"adjective": "small"
}
]
},
{
"word": "dog",
"adjectives": [
{
"type": "textile",
"adjective": "fluffy"
},
{
"type": "visual",
"adjective": "big"
}
]
},
{
"word": "chocolate",
"adjectives": [
{
"type": "visual",
"adjective": "small"
},
{
"type": "gustatory",
"adjective": "sweet"
}
]
}
]
}
Then, the following jq filter returns an array containing the words which contain both adjectives:
.content
| map(
select(
.adjectives as $adj
| all("small","fluffy"; IN($adj[].adjective))
)
| .word
)
If a non-array output is required, and only one word per line, use .[] instead of map (either after content or as a final filter), e.g.:
jq -r '.content[]
| select(
.adjectives as $adj
| all("small","fluffy"; IN($adj[].adjective))
)
| .word'

how to group json using jq and then convert to yaml

I have this json that i want to convert.
[
{
"externalGroup": "another group admins",
"groupId": "da2e42c8-6423-4d32-99b5-5fc58f9f80b8"
},
{
"externalGroup": "another group users",
"groupId": "7c69cac1-4a70-4170-8251-cde3762fe498"
},
{
"externalGroup": "my group admin",
"groupId": "e08a1d9d-f108-4e87-bdb3-ee4f10c6752a"
},
{
"externalGroup": "my group users",
"groupId": "8370821e-edfa-4615-ac2e-47815b740f40"
},
{
"externalGroup": "some group",
"groupId": "e08a1d9d-f108-4e87-bdb3-ee4f10c6752a"
},
{
"externalGroup": "some group",
"groupId": "8370821e-edfa-4615-ac2e-47815b740f40"
},
{
"externalGroup": "some group",
"groupId": "7c69cac1-4a70-4170-8251-cde3762fe498"
}
]
I have tried this, which is pretty close:
jq '. | group_by(.externalGroup)[] | {(.[0].externalGroup): map(.groupId)}'
I get this:
{
"another group admins": [
"da2e42c8-6423-4d32-99b5-5fc58f9f80b8"
]
}
{
"another group users": [
"7c69cac1-4a70-4170-8251-cde3762fe498"
]
}
{
"my group admin": [
"e08a1d9d-f108-4e87-bdb3-ee4f10c6752a"
]
}
{
"my group users": [
"8370821e-edfa-4615-ac2e-47815b740f40"
]
}
{
"some group": [
"e08a1d9d-f108-4e87-bdb3-ee4f10c6752a",
"8370821e-edfa-4615-ac2e-47815b740f40",
"7c69cac1-4a70-4170-8251-cde3762fe498"
]
}
But this doesn't convert properly with yq. It would need to look something like this instead:
{
"another group admins": [
"da2e42c8-6423-4d32-99b5-5fc58f9f80b8"
],
"another group users": [
"7c69cac1-4a70-4170-8251-cde3762fe498"
],
"my group admin": [
"e08a1d9d-f108-4e87-bdb3-ee4f10c6752a"
],
"my group users": [
"8370821e-edfa-4615-ac2e-47815b740f40"
],
"some group": [
"e08a1d9d-f108-4e87-bdb3-ee4f10c6752a",
"8370821e-edfa-4615-ac2e-47815b740f40",
"7c69cac1-4a70-4170-8251-cde3762fe498"
]
}
In order to get something like:
"another group admins":
- "da2e42c8-6423-4d32-99b5-5fc58f9f80b8"
"another group users":
- "7c69cac1-4a70-4170-8251-cde3762fe498"
"my group admin":
- "e08a1d9d-f108-4e87-bdb3-ee4f10c6752a"
"my group users":
- "8370821e-edfa-4615-ac2e-47815b740f40"
"some group":
- "e08a1d9d-f108-4e87-bdb3-ee4f10c6752a",
- "8370821e-edfa-4615-ac2e-47815b740f40",
- "7c69cac1-4a70-4170-8251-cde3762fe498"
The piece you are missing is from_entries which can build a JSON object from an array of keys and values.
Instead of:
jq '. | group_by(.externalGroup)[] | {(.[0].externalGroup): map(.groupId)}'
Try:
jq 'group_by(.externalGroup) | map({key:.[0].externalGroup, value:map(.groupId)}) | from_entries'
{
"another group admins": [
"da2e42c8-6423-4d32-99b5-5fc58f9f80b8"
],
"another group users": [
"7c69cac1-4a70-4170-8251-cde3762fe498"
],
"my group admin": [
"e08a1d9d-f108-4e87-bdb3-ee4f10c6752a"
],
"my group users": [
"8370821e-edfa-4615-ac2e-47815b740f40"
],
"some group": [
"e08a1d9d-f108-4e87-bdb3-ee4f10c6752a",
"8370821e-edfa-4615-ac2e-47815b740f40",
"7c69cac1-4a70-4170-8251-cde3762fe498"
]
}
I made the following changes:
Removed the . | at the beginning because it doesn't change anything.
Removed the [] and used map(...) instead, because we want to keep things in an array to feed to from_entries.
Instead of assembling a one-entry object, we create {key:..., value:...} pairs to feed to from_entries.
Actually, I just checked and was slightly surprised to discover that add is actually a bit faster than from_entries even for very long lists. If you use add you need to change even less of your solution.
jq 'group_by(.externalGroup) | map({(.[0].externalGroup):map(.groupId)}) | add'
Adding together objects combines their contents together. I tested with a 250,000 element list and it was slightly faster than from_entries. Given that it's also shorter and in my opinion pretty much just as clear, I think it's worthy of consideration.
An alternative worth considering for producing yaml is gojq, the Go implementation of jq, e.g.
gojq --yaml-output '
group_by(.externalGroup)
| map({(.[0].externalGroup):map(.groupId)}) | add'
To avoid the overhead of map, you could use the following generic stream-oriented add that works for objects or arrays just as well as for numbers:
gojq --yaml-output '
def add(s): reduce s as $x (null; . + $x);
add( group_by(.externalGroup)[]
| {(.[0].externalGroup):map(.groupId)})'

jq update json document to alter an array element

I know this has to be simple, but for some reason it's eluding me how to find an element given a condition and modify one of its fields. The doc should be fully output (sed style) with the edit made.
{
"state": "wait",
"steps": {
"step1": [
{ "name":"Foo", "state":"wait" },
{ "name":"Bar", "state":"wait" }
],
"step2": [
{ "name":"Foo", "state":"wait" },
{ "name":"Zoinks", "state":"ready" }
],
"step3": [
{ "name":"Foo", "state":"cancel" }
]
}
}
I'm expecting something like this should be workable.
jq '. | (select(.steps[][].name=="Foo" and .steps[][].state=="wait") |= . + {.state:"Ready"}'
or
jq '. | (select(.steps[][]) | if (.name=="Foo" and .state=="wait") then (.state="Ready") else . end)
The desired output, of course, would be
{
"state": "wait",
"steps": {
"step1": [
{ "name":"Foo", "state":"ready" },
{ "name":"Bar", "state":"wait" }
],
"step2": [
{ "name":"Foo", "state":"ready" },
{ "name":"Zoinks", "state":"ready" }
],
"step3": [
{ "name":"Foo", "state":"cancel" }
]
}
}
Instead, when I'm not getting cryptic errors, I'm either modifying a top-level field in the document or modifying the field for all the elements or repeated the entire doc multiple times.
Any insights greatly appreciated.
Thanks.
p.s. is there a better syntax than [] to wildcard the named-elements under steps? Or after the pipe to identify the indices discovered by the select?
Pipe the output of .steps[][] into a select call that chooses the objects with the desired name and state values, then set the state value on the result.
$ jq '(.steps[][] | select(.name == "Foo" and .state == "wait")).state = "ready"' tmp.json
{
"state": "wait",
"steps": {
"step1": [
{
"name": "Foo",
"state": "ready"
},
{
"name": "Bar",
"state": "wait"
}
],
"step2": [
{
"name": "Foo",
"state": "ready"
},
{
"name": "Zoinks",
"state": "ready"
}
],
"step3": [
{
"name": "Foo",
"state": "cancel"
}
]
}
}
You can help confirm this using diff (the first jq just normalizes the formatting so that only the changes made by the second one show up in the diff):
$ diff <(jq . tmp.json) <(jq '...' tmp.json)
7c7
< "state": "wait"
---
> "state": "ready"
17c17
< "state": "wait"
---
> "state": "ready"

Modifying array of key value in JSON jq

In case, I have an original json look like the following:
{
"taskDefinition": {
"containerDefinitions": [
{
"name": "web",
"image": "my-image",
"environment": [
{
"name": "DB_HOST",
"value": "localhost"
},
{
"name": "DB_USERNAME",
"value": "user"
}
]
}
]
}
}
And I would like to inplace modify the value for the matched key like so:
jq '.taskDefinition.containerDefinitions[0].environment[] | select(.name=="DB_USERNAME") | .value="new"' json
I got the output
{
"name": "DB_USERNAME",
"value": "new"
}
But I want more like in-place modify or the whole json from the original with new value modified, like this:
{
"taskDefinition": {
"containerDefinitions": [
{
"name": "web",
"image": "my-image",
"environment": [
{
"name": "DB_HOST",
"value": "localhost"
},
{
"name": "DB_USERNAME",
"value": "new"
}
]
}
]
}
}
Is it possible to do with jq or any known workaround?
Thank you.
Updated
For anyone looking for editing multi-values,
here is the approach I use
JQ=""
for e in DB_HOST=rds DB_USERNAME=xxx; do
k=${e%=*}
v=${e##*=}
JQ+="(.taskDefinition.containerDefinitions[0].environment[] | select(.name==\"$k\") | .value) |= \"$v\" | "
done
jq '${JQ%??}' json
I think there should be more concise way, but this seems working fine.
It is enough to assign to the path, if you are using |=, e.g.
jq '
(.taskDefinition.containerDefinitions[0].environment[] |
select(.name=="DB_USERNAME") | .value) |= "new"
' infile.json
Output:
{
"taskDefinition": {
"containerDefinitions": [
{
"name": "web",
"image": "my-image",
"environment": [
{
"name": "DB_HOST",
"value": "localhost"
},
{
"name": "DB_USERNAME",
"value": "new"
}
]
}
]
}
}
Here is a select-free solution using |=:
.taskDefinition.containerDefinitions[0].environment |=
map(if .name=="DB_USERNAME" then .value = "new"
else . end)
Avoiding select within the expression on the LHS of |= makes the solution more robust w.r.t. the version of jq being used.
You might like to consider this alternative to using |=:
walk( if type=="object" and .name=="DB_USERNAME"
then .value="new" else . end)

Create one JSON from another, with extra data va JQ

I have this file:
[
"smoke-tests",
"push-apps-manager"
]
I'd like to get this output using JQ:
{
"errands": [
{"name": "smoke-tests", "post_deploy": true},
{"name": "push-apps-manager", "post_deploy": true}
]
}
It seems so simple, yet, I have so much difficulty here...
It's a little tricky, since you need to embed the input into the list bound to the errands key. Start by creating the sequence of name/post_deploy objects:
% jq '.[] | {name: ., post_deploy: true}' names.json
{
"name": "smoke-tests",
"post_deploy": true
}
{
"name": "push-apps-manager",
"post_deploy": true
}
Then wrap that in the list in the outer object:
% jq '{errands: [.[] | {name: ., post_deploy: true}]}' names.json
{
"errands": [
{
"name": "smoke-tests",
"post_deploy": true
},
{
"name": "push-apps-manager",
"post_deploy": true
}
]
}
You can also use the map function (which I rarely remember how to use correctly, but it turns out is pretty simple here):
% jq '{errands: map({name:., post_deploy: true})}' names.json
Here is another approach. If you are new to jq it may be easiest to work towards the goal in small steps.
1) Start with the identity filter
.
which produces as expected
[
"smoke-tests",
"push-apps-manager"
]
2) next add the outer object with the "errands" key:
{ "errands": . }
which produces
{
"errands": [
"smoke-tests",
"push-apps-manager"
]
}
3) next move the data into an array
{ "errands": [ . ] }
which produces
{
"errands": [
[
"smoke-tests",
"push-apps-manager"
]
]
}
4) add the inner object with the "name" and "post_deploy" keys
{ "errands": [ { "name": ., "post_deploy": true } ] }
which produces
{
"errands": [
{
"name": [
"smoke-tests",
"push-apps-manager"
],
"post_deploy": true
}
]
}
5) Now we're really close. All we need to do is take advantage of jq's Object Construction behavior when an expression produces multiple results :
{ "errands": [ { "name": .[], "post_deploy": true } ] }
which gives us the desired result
{
"errands": [
{
"name": "smoke-tests",
"post_deploy": true
},
{
"name": "push-apps-manager",
"post_deploy": true
}
]
}