I have an input JSON document with roughly the following form (actual data has additional keys, which should be passed through unmodified; the whitespace is adjusted for human readability, and there's no expectation that it be maintained):
{
"Rules": [
{"Filter": { "Prefix": "to_me/" }, "Status": "Enabled" },
{"Filter": { "Prefix": "from_me/" }, "Status": "Enabled" },
{"Filter": { "Prefix": "__bg/" }, "Status": "Enabled" }
]
}
I need to match .Rules[].Filter.Prefix=="to_me/" and then change the associated "Status": "Enabled" to "Disabled". Since only the first rule above has a prefix of to_me/, status of that rule would be changed to Disabled, making correct output look like the following:
{
"Rules": [
{"Filter": { "Prefix": "to_me/" }, "Status": "Disabled" },
{"Filter": { "Prefix": "from_me/" }, "Status": "Enabled" },
{"Filter": { "Prefix": "__bg/" }, "Status": "Enabled" }
]
}
I've tried several different combinations but can't seem to get it right.
Anyone have ideas?
I prefer the idiom ARRAY |= map(...) over ARRAY[] |= ..., mainly because the former can be used reliably whether or not any of the substitutions evaluate to empty:
jq '.Rules |= map(if .Filter.Prefix == "to_me/"
then .Status="Disabled" else . end)'
To overwrite the input file, you might like to consider sponge from moremutils.
Doing in-place updates can be done with |=, and deciding whether to modify content in-place can be done with if/then/else. Thus:
jq '.Rules[] |= (if .Filter.Prefix == "to_me/" then .Status="Disabled" else . end)'
Related
I want to describe a network graph of vertices and edges with JSON Schema.
An example JSON could look like this:
{
"V": [
"1",
"2",
"3"
],
"E": [
{
"v1": "1",
"v2": "2"
},
{
"v1": "2",
"v2": "3"
}
]
}
I have a set of 3 vertices and 2 edges to connect them. I want all vertices to have an arbitrary string identifier, so it could also be "node1" or "panda". However, is there a way to validate that the endpoints of my edges only point to existing vertices?
I.e.: Should NOT pass:
{
"V": [
"n1",
"n2",
"n3"
],
"E": [
{
"v1": "n1",
"v2": "IdThatDoesNotExistAbove"
}
]
}
I looked at ENUMs, however, I struggle to have them point at data from a JSON that I want to validate rather than to the specification itself.
With jq this task can be solved.
jq -r '([.E[] | to_entries[].value] | unique) - .V |
if length == 0
then "all vertices defined"
else "undefined vertices: \(.)\n" | halt_error(1)
end
' "$FILE"
echo "exit code: $?"
Output valid file
all vertices defined
exit code: 0
Output invalid file
undefined vertices: ["IdThatDoesNotExistAbove"]
exit code: 1
If you are not interested which vertices are undefined you can use a shorter version
jq -e '([.E[] | to_entries[].value]) - .V | length == 0' "$FILE"
echo "exit code: $?"
Output valid file
true
exit code: 0
Output invalid file
false
exit code: 1
JSON Schema doesn't define a way to reference data like this, but it does have extension vocabularies, which allow the definition of custom keywords. I have created a data vocabulary that does precisely what you're looking to do.
{
"$schema": "https://json-everything.net/meta/data-2022",
"type": "object",
"$defs": {
"user-defined-vertex": {
"data": {
"enum": "/V"
}
}
},
"properties": {
"V": {
"type": "array",
"items": {"type": "string"}
},
"E": {
"type": "array",
"items": {
"type": "object",
"properties":{
"v1": { "$ref": "#/$defs/user-defined-vertex" },
"v2": { "$ref": "#/$defs/user-defined-vertex" }
},
"required": ["v1", "v2"],
"additionalProperties": false
}
}
},
"additionalProperties": false
}
The key part of this is the data keyword in #/$defs.
data takes an object with schema keywords as keys and JSON Pointers or URIs as values. If you want to extract values from the instance data, you'll use JSON Pointers. For anything else, you'll use a URI.
So for this case, I have
{
"data": {
"enum": "/V"
}
}
which says to take the value from /V in the instance data and use that as the value for the enum keyword.
In #/properties/V you define that /V must be an array with string values.
However, to my knowledge, this vocabulary is only implemented for my library, JsonSchema.Net and you'll need the extension package JsonSchema.Net.Data.
I have a json formatted overview of backups, generated using pgbackrest. For simplicity I removed a lot of clutter so the main structures remain. The list can contain multiple backup structures, I reduced here to just 1 for simplicity.
[
{
"backup": [
{
"archive": {
"start": "000000090000000200000075",
"stop": "000000090000000200000075"
},
"info": {
"size": 1200934840
},
"label": "20220103-122051F",
"type": "full"
},
{
"archive": {
"start": "00000009000000020000007D",
"stop": "00000009000000020000007D"
},
"info": {
"size": 1168586300
},
"label": "20220103-153304F_20220104-081304I",
"type": "incr"
}
],
"name": "dbname1"
}
]
Using jq I tried to generate a simpeler format out of this, until now without any luck.
What I would like to see is the backup.archive, backup.info, backup.label, backup.type, name combined in one simple structure, without getting into a cartesian product. I would be very happy to get the following output:
[
{
"backup": [
{
"archive": {
"start": "000000090000000200000075",
"stop": "000000090000000200000075"
},
"name": "dbname1",
"info": {
"size": 1200934840
},
"label": "20220103-122051F",
"type": "full"
},
{
"archive": {
"start": "00000009000000020000007D",
"stop": "00000009000000020000007D"
},
"name": "dbname1",
"info": {
"size": 1168586300
},
"label": "20220103-153304F_20220104-081304I",
"type": "incr"
}
]
}
]
where name is redundantly added to the list. How can I use jq to convert the shown input to the requested output? In the end I just want to generate a simple csv from the data. Even with the simplified structure using
'.[].backup[].name + ":" + .[].backup[].type'
I get a cartesian product:
"dbname1:full"
"dbname1:full"
"dbname1:incr"
"dbname1:incr"
how to solve that?
So, for each object in the top-level array you want to pull in .name into each of its .backup array's elements, right? Then try
jq 'map(.backup[] += {name} | del(.name))'
Demo
Then, generating a CSV output using jq is easy: There is a builtin called #csv which transforms an array into a string of its values with quotes (if they are stringy) and separated by commas. So, all you need to do is to iteratively compose your desired values into arrays. At this point, removing .name is not necessary anymore as we are piecing together the array for CSV output anyway. And we're giving the -r flag to jq in order to make the output raw text rather than JSON.
jq -r '.[]
| .backup[] + {name}
| [(.archive | .start, .stop), .name, .info.size, .label, .type]
| #csv
'
Demo
First navigate to backup and only then “print” the stuff you’re interested.
.[].backup[] | .name + ":" + .type
I have a defaults.json and a current.json.
defaults.json gets copied to current.json, and current.json is used as the main configuration file.
defaults would looke something like this:
{
"AttributeName":"setting1"
"Value": [
{
"ValueName": "Disabled",
"ValueDisplayName": "Disabled"
},
{
"ValueName": "Enabled",
"ValueDisplayName": "Enabled"
}
],
"DefaultValue": "Enabled"
}
and current.json would look like this:
{
"AttributeName":"setting1"
"Value": [
{
"ValueName": "Disabled",
"ValueDisplayName": "Disabled"
},
{
"ValueName": "Enabled",
"ValueDisplayName": "Enabled"
}
],
"DefaultValue": "Enabled",
"CurrentValue": "Enabled"
}
Now when I add a new "setting2" (which has the same keys, but values can be different) to defaults.json, I would like to update current.json with that setting, without overwriting the "currentvalue" field. How can I do this using jq?
I tried things like jq -rs 'add' defaults.json current.json but this just prints current.json.
I've tried looking at some of the other questions regarding jq, but they all cater to a very specific situation, uncomparable to mine.
The following jq program first finds the "new" paths in default.json by subtracting the paths in current.json from the paths in default.json, and then updates the "current" JSON by adding all the "new" paths and their associated values:
jq --argfile default default.json '
. as $current
| ([$default|paths] - [$current|paths]) as $new
| reduce $new[] as $p ($current;
setpath($p; $default|getpath($p)))
' current.json
Caveat
As of this writing, the --argfile option is officially "deprecated", so you might like to use one of the many other ways to pass in the contents of default.json.
I'm trying to change # to %23 in every context value, but I'm having problem with null values.
The shortened JSON is:
{
"stats": {
"suites": 1
},
"results": [
{
"uuid": "676-a46b-47a1-a49f-4da4e46c1120",
"title": "",
"suites": [
{
"uuid": "gghjh-56a9-4713-b139-0d5b36bc7fbc",
"title": "Login process",
"tests": [
{
"pass": false,
"fail": true,
"pending": false,
"context": "\"screenshots/login.spec.js/Login process -- should login #11 (failed).png\""
},
{
"pass": false,
"fail": false,
"pending": true,
"context": null
}
]
}
]
}
]
}
And the JQ command I think it's closest to correct is:
jq '.results[].suites[].tests[].context | strings | sub("#";"%23")'
But the problem is that I need to get in return full edited file. How could I achieve that?
You were close. To retain the original structure, you need to use the update operator (|=) instead of pipe. Enclosing the entire expression to the left of it in parentheses is also necessary, otherwise the original input will be invisible to |=.
(.results[].suites[].tests[].context | strings) |= sub("#"; "%23")
Online demo
change # to %23 in every context value
You might wish to consider:
walk( if type=="object" and (.context|type)=="string"
then .context |= gsub("#"; "%23")
else . end )
I'm trying to replace objects in a complex JSON object. It seemed that the tool jq could be offering the perfect solution, but I'm really struggling with the right choice / chain of filters.
I have a complete configuration JSON object which looks like this (has some more keys in it, shortened it for illustration):
{
"some-array": [
{
"name": "foo",
"attr": "value"
},
{
"name": "foo bar",
"attr": "value"
},
{
"name": "foo bar baz",
"attr": "value"
}
],
"some-other-array": []
}
Now I have another object containing an array with updated objects which I need to merge with the full configuration in some way. I need to find the nested objects by name, add it if it does not exist yet and replace it if it does exist.
{
"some-array": [
{
"name": "foo",
"attr": "new-value",
"new-attrib": "new-value"
},
{
"name": "foo bar",
"attr": "new-value"
}
]
}
So, with the above example, my expected result would be:
{
"some-array": [
{
"name": "foo",
"attr": "new-value",
"new-attrib": "new-value"
},
{
"name": "foo bar",
"attr": "new-value"
},
{
"name": "foo bar baz",
"attr": "value"
}
],
"some-other-array": []
}
I already tried select(."some-array"[].name == "foo") to begin with and a few other things as a jq filter, but I'm struggling to move forward here and would really appreciate some inspiration / an actual solution.
Can anyone tell me if what I'm trying to achieve is actually possible with jq or do I have to find another solution?
Here is a solution to the updated problem. This solution assumes that the names are string-valued. It relies on two helper functions:
# array-to-hash
def a2h(f): reduce .[] as $x ({}; . + {($x | f): $x});
# hash-to-array
def h2a: . as $in | reduce keys_unsorted[] as $k ([]; . + [$in[$k]]);
The first of these creates a "hash" based on an input array, and the second implements the inverse operation.
With these helper functions, the solution can be written:
.["some-array"] |= (a2h(.name) + ($update|.["some-array"] | a2h(.name)) | h2a)
where $update is the "new" value. This solution relies on the "right-dominance" of object-addition.
Output
For the given example, the output is:
{
"some-array": [
{
"name": "foo",
"attr": "new-value",
"new-attrib": "new-value"
},
{
"name": "foo bar",
"attr": "new-value"
},
{
"name": "foo bar baz",
"attr": "value"
}
],
"some-other-array": []
}
Yes, it's possible, and in fact quite easy under various interpretations of the problem as originally stated.
The following solves the the problem as it was originally stated, with "it" being interpreted as .["some-array"] rather than its constituents.
Assuming $update holds the object with the updated information as shown, the update could be performed using this filter:
.["some-array"] = ($update | .["some-array"])
There are many ways to endow $update with the desired value.