I have got a configuration with content to be exchanged with snippets from separate file. How would I neatly achieve this?
Config file might look like:
# config file
{
"keep": "whatever type of value",
"manipulate": [
{
"foo": "bar",
"cat": {
"color": "grey"
},
"type": "keep",
"detail": "keep whole array element"
},
{
"stuff": "obsolete",
"more_stuff": "obsolete",
"type": "replace",
"detail": "replace whole array element with content from separate file"
},
{
"foz": "baz",
"dog": {
"color": "brown"
},
"type": "keep",
"detail": "keep whole array element"
},
],
"also_keep": "whatever type of value"
}
The content (coming from separate file) to be inserted as a replacement of obsolete array element:
# replacement
{
"stuff": "i want that",
"fancy": "very",
"type": "new"
}
The desired result should look like:
# result
{
"keep": "whatever kind of value",
"manipulate": [
{
"foo": "bar",
"cat": {
"color": "grey"
},
"type": "keep",
"detail": "keep whole array element"
},
{
"stuff": "i want that",
"fancy": "very",
"type": "new"
},
{
"foz": "baz",
"dog": {
"color": "brown"
},
"type": "keep",
"detail": "keep whole array element"
},
],
"also_keep": "whatever kind of value",
}
Requirements:
Content replacement needs to be done based on type key's value.
Preferably use jq and common bash/linux tools.
Array element ordering should stay same
jq solution:
jq --slurpfile repl repl.json '.manipulate=[.manipulate[]
| if .type=="replace" then .=$repl[0] else . end]' config.json
repl.json - json file containing replacement JSON data
--slurpfile repl repl.json - reads all the JSON texts in the named file and binds an array of the parsed JSON values to the given global variable
The output:
{
"keep": "whatever type of value",
"manipulate": [
{
"foo": "bar",
"cat": {
"color": "grey"
},
"type": "keep",
"detail": "keep whole array element"
},
{
"stuff": "i want that",
"fancy": "very",
"type": "new"
},
{
"foz": "baz",
"dog": {
"color": "brown"
},
"type": "keep",
"detail": "keep whole array element"
}
],
"also_keep": "whatever type of value"
}
jq --slurpfile repl repl.json '.manipulate |=
map(if .type=="replace" then $repl[0] else . end)' config.json
Related
I'm relatively new to JQ and this might be a simple question.
I have a big, pretty nested JSON file with thousands of assets. Each asset always contains one array with two objects (one mother and one child).
{
"data": [
{
"assets": {
"array": [
{
"id": "7978918",
"labels": {
"text": "mother",
"value": true
},
"properties": {
"context_ids": [],
"parent": null
}
},
{
"id": "caa17b2a-4582-4d13-b891-2607e4ba33c6",
"labels": {
"text": "child",
"value": true
},
"properties": {
"context_ids": [],
"parent": null
}
}
]
}
}
]
}
What I want to do is copying the id from the mother object into the parent field of the child object.
{
"data": [
{
"assets": {
"array": [
{
"id": "7978918",
"labels": {
"text": "mother",
"value": true
},
"properties": {
"context_ids": [],
"parent": null
}
},
{
"id": "caa17b2a-4582-4d13-b891-2607e4ba33c6",
"labels": {
"text": "child",
"value": true
},
"properties": {
"context_ids": [],
"parent": "7978918"
}
}
]
}
}
]
}
So far I got
cat test.json | jq '.data[].assets.array[]|=(.properties.parent=.id|select(.labels.text=="mother"))' |less
but this, of course, only copies the mother id into the mother object's parent field and deletes the child object entirely.
I also tried
cat test.json | jq '.data[].assets.array[].properties += {"parent": .data[].assets.array[0].id} |less
On first sight this looked to be correct but seemingly has created a strange loop which never stops and creates a bigger and bigger JSON.
Assuming the first element is always the mother, this should work just fine:
.data[].assets.array |= (.[1].properties.parent = .[0].id)
Online demo
Otherwise sort them by labels.text first so that they are always at certain indices.
.data[].assets.array |= (sort_by(.labels.text) | .[0].properties.parent = .[1].id)
jq '.data |= map(.assets.array | .[1].properties.parent = .[0].id)' test.json
I'm trying to prune nodes deeply within a JSON structure and I'm puzzled why empty behaves seemingly different from a normal value here.
Input
[
{
"name": "foo",
"children": [{
"name": "foo.0",
"color": "red"
}]
},
{
"name": "bar",
"children": [{
"name": "bar.0",
"color": "green"
},
{
"name": "bar.1"
}]
},
{
"name": "baz",
"children": [{
"name": "baz.0"
},
{
"name": "baz.1"
}]
}
]
Program
jq '(.[].children|.[])|=if has("color") then . else empty end' foo.json
Actual output
[
{
"name": "foo",
"children": [
{
"name": "foo.0",
"color": "red"
}
]
},
{
"name": "bar",
"children": [
{
"name": "bar.0",
"color": "green"
}
]
},
{
"name": "baz",
"children": [
{
"name": "baz.1"
}
]
}
]
Expected output
The output I get, except without the baz.1 child, as that one doesn't have a color.
Question
Apart from the right solution, I'm also curious why replacing empty in the script by a regular value like 42 would replace the children without colors with 42 as expected, but when replacing with empty, it looks like the else branch doesn't get executed?
.[].children |= map(select(.color))
Will remove children that does not has an color so the output becomes:
[
{
"name": "foo",
"children": [
{
"name": "foo.0",
"color": "red"
}
]
},
{
"name": "bar",
"children": [
{
"name": "bar.0",
"color": "green"
}
]
},
{
"name": "baz",
"children": []
}
]
Online demo
Regarding why your filter does not seem to like empty;
This git issue seems to be the cause, multiple elements with empty will fail.
There must be a bug with assigning empty to multiple paths.
In this case you can use del instead:
del(.[].children[] | select(has("color") | not))
Online demo
I am working with a JSON file which has contains lot of data that can be removed before sending to an API.
Found that JQ can be used to achieve this but not sure on how to map to get the desired results.
Input JSON
{
"name": "Sample name",
"id": "123",
"userStory": {
"id": "234",
"storyName": "Story Name",
"narrative": "Narrative",
"type": "feature"
},
"testSteps": [
{
"number": 1,
"description": "Step 1",
"level": 0,
"children": [
{
"number": 2,
"description": "Description",
"children": [
{
"number": 3,
"description": "Description"
}
]
},
{
"number": 4,
"anotherfield": "another field"
}
]
}
]
}
Desired Output
{
"name": "Sample name",
"userStory": {
"storyName": "Story Name"
},
"testSteps": [
{
"description": "Step 1",
"children": [
{
"description": "Description",
"children": [
{
"description": "Description"
}
]
},
{
"anotherfield": "anotherfield"
}
]
}
]
}
Tried to do it with the following jq command
map_values(..|{name, id, userStory})
but not sure how to filter only the userStory.storyName.
Thanks in advance.
Note: The actual JSON has different child elements that are repeated in some cases.
To delete .id from root object:
del(.id)
To leave only .storyName in .userStory:
.userStory |= {storyName}
To delete .number and .level from every object on any level in .testSteps:
.testSteps |= walk(if type == "object" then del(.number, .level) else . end)
Putting it all together:
del(.id) | (.userStory |= {storyName}) | (.testSteps |=
walk(if type == "object" then del(.number, .level) else . end))
Online demo
I'm in a dire need of help for a script to basically convert JSON text to CSV text in an attempt to copy users from one AWS Cognito userpool to another.
The export JSON looks like this:
{
"Users": [
{
"Username": "user.name",
"Attributes": [
{
"Name": "sub",
"Value": "some-value"
},
{
"Name": "email_verified",
"Value": "true"
},
{
"Name": "custom:jobtitle",
"Value": Director"
},
{
"Name": "custom:user_id",
"Value": "38"
},
{
"Name": "email",
"Value": "foo.bar#email.com"
}
],
"UserCreateDate": some-value,
"UserLastModifiedDate": some-value,
"Enabled": some-value,
"UserStatus": "some-value"
}
[more lines down here]...
] }
Then the CSV file would contain these lines:
,,,,,,,,,foo.bar#email.com,TRUE,,,,,,FALSE,,,Director,,38,FALSE,foo.bar
[more lines down here]...
So, the variables would be like this for JSON:
{
"Users": [
{
"Username": "%USERNAME%",
"Attributes": [
{
"Name": "sub",
"Value": "some-value"
},
{
"Name": "email_verified",
"Value": "true"
},
{
"Name": "custom:jobtitle",
"Value": %JOB_TITLE%"
},
{
"Name": "custom:user_id",
"Value": "%USER_ID%"
},
{
"Name": "email",
"Value": %EMAIL%"
}
],
"UserCreateDate": some-value,
"UserLastModifiedDate": some-value,
"Enabled": some-value,
"UserStatus": "some-value"
}
...
]
}
And like this for CSV:
,,,,,,,,,%EMAIL%,TRUE,,,,,,FALSE,,,%JOB_TITLE%,,%USER_ID%,FALSE,%USERNAME%
where %EMAIL%, %JOB_TITLE%, %USER_ID%, and %USERNAME% are variables, everything else should be just string.
Appreciate your help in advanced guys.
Consider first this filter:
.Users[].Attributes
| map(select(.Name | . == "custom:jobtitle" or . == "custom:user_id" or . == "email") )
| from_entries
| [ .email, .["custom:jobtitle"], .["custom:user_id"] ]
| #csv
The trick used here is the use of from_entries to convert the array of Name/Value pairs to an object with the Names as keys.
Assuming valid JSON input along the lines shown in the Q, invoking jq with the -r option would yield:
"foo.bar#email.com","Director","38"
Unfortunately the precise requirements are not so clear to me, but you should be able to adapt the above in accordance with your needs.
Given input like this:
{
"type": "collection",
"foo": "bar",
"children": [
{
"properties": {
"country": "GB"
},
"data": "..."
},
{
"properties": {
"country": "PL"
},
"data": "..."
}
]
}
How can I use jq to retain all of the JSON structure, but filter out some of the children using select(). For instance, If I wanted to return only children with country GB, I would expect the following output:
{
"type": "collection",
"foo": "bar",
"children": [
{
"properties": {
"country": "GB"
},
"data": "..."
}
]
}
If I only want the children, this is easy with .children[] | select(.properties.country == "GB"), but does not retain the rest of the JSON.
The key is to use |=. In the present case, you could use the following pattern:
.children |= map(select(...))