jq: easiest way to recursively remove objects based on object value condition - json

I would like to use jq to remove all dictionaries within a JSON "object" (I used that term generally to refer to either an Array or a Dictionary) that
a) contain a key named "delete_me", AND
b) where the key "delete_me" meets some predetermined condition (null, non-zero, true, etc)
Basically, the logic I want to implement is: walk the input, and at each node, if that node is not an Array or an Object, then keep it and move on, otherwise, keep it but remove from it any children that are dictionaries for which either condition a) or b) fail.
Any suggestions?
Sample input:
{
"a": { "foo": "bar" },
"b": {
"i": {
"A": {
"i": [
{
"foo": {},
"bar": {
"delete_if_this_is_null": false,
"an_array": [],
"another_array": [
{
"delete_if_this_is_null": null,
"foo": "bar"
}
],
"etc": ""
},
"foo2": "s"
},
{
"foo": {
"an_array": [
{
"delete_if_this_is_null": "ok",
"foo":"bar",
"another_object": { "a":1 }
},
{
"delete_if_this_is_null": null,
"foo2":"bar2",
"another_object": { "a":1 },
"name": null
}
],
"an_object": {
"delete_if_this_is_null":null,
"foo3":"bar3"
}
},
"zero": 0,
"b": "b"
}
]
}
}
}
}
should yield, if the "delete_me" key is delete_if_this_is_null and the predetermined condition is delete_if_this_is_null == null:
{
"a": { "foo": "bar" },
"b": {
"i": {
"A": {
"i": [
{
"foo": {},
"bar": {
"delete_if_this_is_null": false,
"an_array": [],
"another_array": [],
"etc": ""
},
"foo2": "s"
},
{
"foo": {
"an_array": [
{
"delete_if_this_is_null": "ok",
"foo":"bar",
"another_object": { "a":1 }
}
]
},
"zero": 0,
"b": "b"
}
]
}
}
}
}
UPDATE: Here's the solution: Assume the input is in a file 'input.json':
jq 'def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
def mapper(f):
if type == "array" then map(f)
elif type == "object" then
. as $in
| reduce keys[] as $key
({};
[$in[$key] | f ] as $value
| if $value | length == 0 then .
else . + {($key): $value[0]} end)
else .
end;
walk( mapper(select((type == "object" and .delete_if_this_is_null == null) | not)) )' < input.json

Jeff's solution may zap too much. For example, using:
def data: [1,2, {"hello": {"delete_me": true, "a":3 }, "there": 4} ]; ];
Jeff's solution yields empty (i.e. nothing).
The following may therefore be closer to what you're looking for:
walk(if (type == "object" and .delete_me) then del(.) else . end )
For data, this yields:
[1,2,{"hello":null,"there":4}]
Alternative Solution
If a solution that eliminates the "hello":null in the above example is required, then a variant of jq's map_values/1 is needed. Here's one approach:
def mapper(f):
if type == "array" then map(f)
elif type == "object" then
. as $in
| reduce keys[] as $key
({};
[$in[$key] | f ] as $value
| if $value | length == 0 then .
else . + {($key): $value[0]} end)
else .
end;
data | walk( mapper(select((type == "object" and .delete_me) | not)) )
The result is:
[1,2,{"there":4}]

Here is a solution which uses a recursive function:
def clean(condition):
if type == "object" then
if condition
then empty
else
with_entries(
if (.value|type) == "object" and (.value|condition)
then empty
else .value |= clean(condition)
end
)
end
elif type == "array" then
map(
if type == "object" and condition
then empty
else clean(condition)
end
)
else .
end
;
clean(
has("delete_if_this_is_null") and (.delete_if_this_is_null == null)
)

I'm not sure what exactly you're trying to accomplish in your question but I'm assuming you want to recursively search through a json response and remove json objects that satisfy some condition.
You can do this rather easily with the help of the walk filter that will be coming up in a future version of jq, see the implementation in the source.
# Apply f to composite entities recursively, and to atoms
def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
With that, you can filter them out like so:
def filter_objects(predicate): # removes objects that satisfies some predicate
walk(
if (type == "object") and (predicate) then
empty
else
.
end
)
;
filter_objects(.delete_me) # remove objects that has a truthy property "delete_me"

Related

how to replace only one key/value pair by using jq in json file?

My json template is as of this:
{
"interface_settings": [
{
"name": "lan",
"status": "$status",
...
},
{
"name": "lte1",
"status": "$status",
...
},
{
...
}
],
...
}
And my jq command:
jq '.interface_settings[].status="up"' <my_json_template file>
will update both status values within the interface_settings section. How may I just have one changed ?
let's say I want to update the status where the name is "lan"
One way to update all such objects would be:
.interface_settings |= map( if .name == "lan" then .status = "up" else . end)
Just the first such
.interface_settings |= (reduce .[] as $x (null;
if .done
then .ans += [$x]
elif $x.name == "lan"
then .ans += [$x | .status = "up"] | .done = true
else .ans += [$x]
end) | .ans)

Replace a character with another in object keys

I am learning how to use jq to manipulate json data,
 I'm having a little trouble with it.
This is my input JSON
{
"user":{
"advertisingID":"617a68"
},
"deviceTs":1575387020137,
"activies":[
{
"ts":1575617868326,
"appsUsage":{
"isFull":true,
"data":[
{
"com.orange.phone":44009
}
],
"startTs":1575617281541
}
},
{
"ts":1575618968326,
"appsUsage":{
"isFull":true,
"data":[
{
"uk.green.launcher2":4354
},
{
"com.black.phone":1232
}
],
"startTs":1575617281541
}
}
]
}
I want to replace all keys containing "dots" by "dashes"
and expected output:
{
"user":{
"advertisingID":"617a68"
},
"deviceTs":1575387020137,
"activies":[
{
"ts":1575617868326,
"appsUsage":{
"isFull":true,
"data":[
{
"com-orange-phone":44009 <----
}
],
"startTs":1575617281541
}
},
{
"ts":1575618968326,
"appsUsage":{
"isFull":true,
"data":[
{
"uk-green-launcher2":4354 <----
},
{
"com-black-phone":1232 <----
}
],
"startTs":1575617281541
}
}
]
}
I have tried with
.activies |= map( with_entries(if .key == "appsUsage" then ... else . end) )
...
(split(".")|join("-")) but without success,
Thanks in advance.
I don't think you need regex for this; I believe a conjuction of split and join builtins would be more effective and clean.
.activies |= walk(
if type == "object" then
reduce (keys_unsorted[] | select(index("."))) as $k (.;
(.[$k | split(".") | join("-")] = .[$k])
| del(.[$k])
)
else . end
)
Online demo
Upon peak's recommendation, a more readable solution:
.activies |= walk(
if type == "object" then
with_entries(
.key |= gsub("\\.";"-")
)
else . end
)
Online demo

How to substitute strings in JSON with jq based on the input

Given the input in this form
[
{
"DIR" : "/foo/bar/a/b/c",
"OUT" : "/foo/bar/x/y/z",
"ARG" : [ "aaa", "bbb", "/foo/bar/a", "BASE=/foo/bar" ]
},
{
"DIR" : "/foo/baz/d/e/f",
"OUT" : "/foo/baz/x/y/z",
"ARG" : [ "ccc", "ddd", "/foo/baz/b", "BASE=/foo/baz" ]
},
{
"foo" : "bar"
}
]
I'm trying to find out how to make jq transform that into this:
[
{
"DIR" : "BASE/a/b/c",
"OUT" : "BASE/x/y/z",
"ARG" : [ "aaa", "bbb", "BASE/a", "BASE=/foo/bar" ]
},
{
"DIR" : "BASE/d/e/f",
"OUT" : "BASE/x/y/z",
"ARG" : [ "ccc", "ddd", "BASE/b", "BASE=/foo/baz" ]
},
{
"foo" : "bar"
}
]
In other words, objects having an "ARG" array, containing a string that starts with "BASE=" should use the string after "BASE=", e.g. "/foo" to substitute other string values that start with "/foo" (except the "BASE=/foo" which should remain unchanged")
I'm not even close to finding a solution myself, and at this point I'm unsure that jq alone will do the job.
With jq:
#!/usr/bin/jq -f
# fix-base.jq
def fix_base:
(.ARG[] | select(startswith("BASE=")) | split("=")[1]) as $base
| .DIR?|="BASE"+ltrimstr($base)
| .OUT?|="BASE"+ltrimstr($base)
| .ARG|=map(if startswith($base) then "BASE"+ltrimstr($base) else . end)
;
map(if .ARG? then fix_base else . end)
You can run it like this:
jq -f fix-base.jq input.json
or make it an executable like this:
chmod +x fix-base.jq
./fix-base.jq input.json
Don't worry, jq alone will do the job:
jq 'def sub_base($base): if (startswith("BASE") | not) then sub($base; "BASE") else . end;
map(if .["ARG"] then ((.ARG[] | select(startswith("BASE=")) | split("=")[1]) as $base
| to_entries
| map(if (.value | type == "string") then .value |= sub_base($base)
else .value |= map(sub_base($base)) end)
| from_entries)
else . end)' input.json
The output:
[
{
"DIR": "BASE/a/b/c",
"OUT": "BASE/x/y/z",
"ARG": [
"aaa",
"bbb",
"BASE/a",
"BASE=/foo/bar"
]
},
{
"DIR": "BASE/d/e/f",
"OUT": "BASE/x/y/z",
"ARG": [
"ccc",
"ddd",
"BASE/b",
"BASE=/foo/baz"
]
},
{
"foo": "bar"
}
]
Some helper functions make the going much easier. The first is generic and worthy perhaps of your standard library:
# Returns the integer index, $i, corresponding to the first element
# at which f is truthy, else null
def indexof(f):
label $out
| foreach .[] as $x (null; .+1;
if ($x|f) then (.-1, break $out) else empty end) // null;
# Change the string $base to BASE using gsub
def munge($base):
if type == "string" and (test("^BASE=")|not) then gsub($base; "BASE")
elif type=="array" then map(munge($base))
elif type=="object" then map_values(munge($base))
else .
end;
And now the easy part:
map(if has("ARG")
then (.ARG|indexof(test("^BASE="))) as $ix
| if $ix
then (.ARG[$ix]|sub("^BASE=";"")) as $base | munge($base)
else . end
else . end )
Some points to note:
You may wish to use sub rather than gsub in munge;
The above solution assumes you want to make the change in all keys, not just "DIR", "OUT", and "ARG"
The above solution allows specifications of BASE that include one or more occurrences of "=".

jq: how to add an object, key/value in a nested json tree with arrays

I am pretty new to JQ ... so sorry if it seems obvious..
The bare problem first. I have this JSON file:
Link: https://github.com/mariotti/technical_interview_questions/blob/master/QUESTIONS.json
Extract.
cat QUESTIONS.json | jq '.TechQuestions.category[0,1].question[0,1]'
output:
{
ID: Q1,
categoryname: General,
idC: C1,
idCQ: C1Q1,
idQ: Q1,
title: Find the most frequent integer in an array
}
{
ID: Q21,
categoryname: Strings,
idC: C2,
idCQ: C2Q1,
idQ: Q1,
title: Find the first non-repeated character in a String
}
{
ID: Q2,
categoryname: General,
idC: C1,
idCQ: C1Q2,
idQ: Q2,
title: Find pairs in an integer array whose sum is equal to 10 (bonus; do it in linear time)
}
{
ID: Q22,
categoryname: Strings,
idC: C2,
idCQ: C2Q2,
idQ: Q2,
title: Reverse a String iteratively and recursively
}
As you can see, this is "deep" into:
{
"TechQuestions": {
"category": [
{
"catname": "General",
"idC": "C1",
"question": [
{
"ID": "Q1",
"categoryname": "General",
"idC": "C1",
"idCQ": "C1Q1",
"idQ": "Q1",
"title": "Find the most frequent integer in an array"
},
I want to add the key/field:
"codefile" : "a string to be defined"
within the question[] items to get something like:
{
"ID": "Q1",
"categoryname": "General",
"idC": "C1",
"idCQ": "C1Q1",
"idQ": "Q1",
"title": "Find the most frequent integer in an array",
"codefile" : "not present"
},
And I want to do it programmatically as I might need to develop a bit further...
From other sources (Transforming the name of key deeper in the JSON structure with jq) I could for example rename a key with this:
cat QUESTIONS.json | jq '.' | jq '
# Apply f to composite entities recursively, and to atoms
def walk(f):
. as $in
| if type == "object" then
reduce keys[] as $key
( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
elif type == "array" then map( walk(f) ) | f
else f
end;
(. |= walk(
if type == "object"
then with_entries( if .key == "name" then .key |= sub("name";"title") else . end)
else .
end))'
I was trying to modify this bit without success. It seems I am unable to simply add a key/value!
I will avoid to overload you with odd references and a further list of attempts.
But maybe I give you an example of a try:
(. |= walk(
if type == "object"
then with_entries(
if .key == "question"
then . = ( . + {"freshly": "added"})
else .
end)
else .
end))'
The solution doesn't have to match my attempts. Actually if there is a more straight full way it is very appreciated.
What's wrong with:
.TechQuestions.category[0,1].question[] += {"codefile" : "a string to be defined"}
Using walk/1, you could consider:
walk( if type == "object" and has("question")
then .question[] += {"codefile" : "a string to be defined"}
else .
end)

Modify object properties conditionally with jq

I have this sample of JSON:
[
{
"name": "val1",
"expire": { "$value": 10 }
},
{
"name": "val2",
"expire": 20
},
{
"name": "val3"
}
]
And I want to transform it to this form with jq:
[
{
"name": "val1",
"expire": 10
},
{
"name": "val2",
"expire": 20
},
{
"name": "val3",
"expire": null
}
]
All that I've found it's if-then-else, but it looks like I have no clue how to build right expression.
Condition based on type check, it looks like a right way, but just return "compile"-time error, I don't know how to fix it:
.[] | { name, expire: (if .expire then (if type(.expire) == "number" then .expire else .expire."$value" end) else null end) }
Condition based on "$value" check, somehow filter out second object:
.[] | { name, expire: (if .expire then (if .expire."$value"? then .expire."$value" else .expire end) else null end) }
As I understand, the problem here in internal if, where second object checked with .expire."$value"?, error was thrown and object removed from result because of error.
Try this filter:
map( {name,
"expire": (.expire | if type == "object" then .["$value"] elif type == "number" then . else null end) } )
or (with significantly different semantics in edge cases):
map(.expire |= if type == "object" then .["$value"]
elif type == "number" then . else null end)
You can think of it another way as updating each object's expire property with the "$value" if present, or the current value. If the value doesn't exist, it's just simply null.
.[].expire |= (."$value"? // .)