Update deeply nested field with value from higher-level object in JQ - json

Given the following input JSON:
{
"version": 2,
"models": [
{
"name": "first_table",
"tests": [
{
"dbt_utils.equal_rowcount": {
"compare_model": null
}
}
]
},
{
"name": "second_table",
"tests": [
{
"dbt_utils.equal_rowcount": {
"compare_model": null
}
}
]
}
]
}
How would I, using jq, replace the null (i.e., the value of "compare_model") with the value from the "name" key? Note that the key-value pairs in question here are not at the same level in the hierarchy: the former is nested in an object in an array, and it is this array that is at the same level as the latter.
For example, the output file should read:
{
"version": 2,
"models": [
{
"name": "first_table",
"tests": [
{
"dbt_utils.equal_rowcount": {
"compare_model": "first_table"
}
}
]
},
{
"name": "second_table",
"tests": [
{
"dbt_utils.equal_rowcount": {
"compare_model": "second_table"
}
}
]
}
]
}
FWIW, this is an intermediate step in some YAML (via yq, the Python wrapper variety of jq as opposed to the go variant) wrangling I'm doing on DBT config files.
(Bonus points if you can wrap the replacement text with parentheses and/or prefix it without breaking out of jq. :D If not, no worries -- this step I can do with another program.)
Needless to say, but your help is very much appreciated!

The key to a simple solution is to use |=, e.g.
.models |=
map(.name as $name
| (.tests[]."dbt_utils.equal_rowcount".compare_model =
$name))
To wrap the replacement value in parentheses, just add them:
.models |=
map("(\(.name))" as $name
| (.tests[]."dbt_utils.equal_rowcount".compare_model =
$name))
If you want the replacement to be conditional on the existing value being null, you could perhaps (depending on the exact requirements) use //=.
Using //= and walk
Here's another take on the problem:
.models
|= map("(\(.name))" as $name
| walk(if type=="object" and has("compare_model")
then .compare_model //= $name
else . end))

That the fields are not at the same level doesn't really matter here.
.models[] |= (.tests[]."dbt_utils.equal_rowcount".compare_model = "(\(.name))")
Online demo

Related

Using JQ to copy one field of an object into another

In my JSON are some objects with a Description property. How do I copy this value over to the GMNotes property of the same object using JQ?
In other words, how does one go from
{
"ObjectStates": [
{
"Description": "",
"GMNotes": ""
},
{
"Description": "foo",
"GMNotes": ""
}
]
}
to
{
"ObjectStates": [
{
"Description": "",
"GMNotes": ""
},
{
"Description": "foo",
"GMNotes": "foo"
}
]
}
.ObjectStates[] | .GMNotes = .Description only returns the modified objects, as shown in the sandbox.
(I could easily do this in Perl. The point is using jq.)
You can use map() in combination with the update assignment operator |=:
jq '(.ObjectStates)|=map(.GMNotes=.Description)' file.json
https://jqplay.org/s/vFV_H4brlH
PS: instead of using map you could also just use the following command, the key is using |=.
jq '.ObjectStates[]|=(.GMNotes=.Description)' file.json
thanks chepner!
https://jqplay.org/s/NCGezXPjLE

jq variable seems to hid original input

I'm trying to parse through some json and put certain sections into variables. I think I'm not understanding something about how variables work though.
Json:
{
"resources": [
{
"type": "Microsoft.ApiManagement/service/apis"
},
{
"type": "Microsoft.ApiManagement/service/apis/schemas"
}
]
}
Then using this jq:
.resources[] | select(.type == "Microsoft.ApiManagement/service/apis") as $apis | { types: [.type], apis: $apis}
I get this:
{
"types": [
"Microsoft.ApiManagement/service/apis"
],
"apis": {
"type": "Microsoft.ApiManagement/service/apis"
}
}
When I expected this:
{
"types": [
"Microsoft.ApiManagement/service/apis",
"Microsoft.ApiManagement/service/apis/schemas"
],
"apis": {
"type": "Microsoft.ApiManagement/service/apis"
}
}
https://jqplay.org/s/4aeBOY9x6q
According to the variables section of the jq manual
The expression exp as $x | ... means: for each value of expression
exp, run the rest of the pipeline with the entire original input, and
with $x set to that value. Thus as functions as something of a foreach
loop.
Which makes me think that .type should return from the original set not the filtered result I stored in $apis. Where is the disconnect?
It's the select that is "hiding" some of the input.
To produce the output you expect, the simplest is not to use variables at all. You could, for example, simply write:
.resources[].type

substitute certain characters in strings found in an object

I have a list of objects and want to replace all occurrences of . with : when the key is Name using jq
input:
{
"Parameters": [
{
"Name": "TEST.AB.SOMETHING",
"Value": "hvfuycsgvfiwbiwbibibewfiwbcfwifcbwibcibc"
},
{
"Name": "TEST_GF_USER",
"Value": "ssssecret"
}
]
}
expected output:
{
"Parameters": [
{
"Name": "TEST:AB:SOMETHING",
"Value": "hvfuycsgvfiwbiwbibibewfiwbcfwifcbwibcibc"
},
{
"Name": "TEST_GF_USER",
"Value": "ssssecret"
}
]
}
You may split by . and join by :
jq '(.Parameters[].Name)|=(split(".")|join(":"))' file.json
The assignment is done using the update operator.
The trick is to use .Name |= gsub("\\.";":"). In your case (a flat list), it's simple. If you want to modify the keys of all objects in an arbitrary JSON text, the simplest would be to use walk/1:
walk( if type == "object" and has("Name") then .Name |= gsub("\\.";":")) else . end )
(If your jq does not have walk/1, then its jq definition can readily be found by googling.)

jq select attribute if any array element satisfies a condition

With the help of jq i would like to select all addresses of nodes that have at least one required=true in their attribute list. The result list should have unique items.
Here is my Json:
{
"nodes": [
{
"address":"127.0.0.1",
"attributes": [
{
"id":"abc",
"required":true
},
{
"id":"def",
"required":true
},
{
"id":"ghi",
"required":false
}
]
},
{
"address":"127.0.0.2",
"attributes": [
{
"id":"abc",
"required":false
},
{
"id":"def",
"required":false
}
]
}
]
}
I first tried with:
jq '.nodes[] | select(.attributes[].required == true) | .address'
This produces:
"127.0.0.1"
"127.0.0.1"
So it gets the address for every required=true field it finds in the attributes section. How to make the result list unique? There is also a unique keyword in jq, but I couldn't figure out how this could help me.
Using unique is safe but it does require sorting, which may not be necessary. In your particular case, for example, the repetition is an artifact of the jq query. Consider using any instead (or as well), as it more precisely captures the intention ("at least one"), as well as having "short-circuit" semantics (i.e., it stops searching once the condition is true):
$ jq '.nodes[]
| select( any(.attributes[]; .required == true))
| .address' input.json
Output:
"127.0.0.1"
You can always add unique if necessary.

Using jq to list keys in a JSON object

I have a hierarchically deep JSON object created by a scientific instrument, so the file is somewhat large (1.3MB) and not readily readable by people. I would like to get a list of keys, up to a certain depth, for the JSON object. For example, given an input object like this
{
"acquisition_parameters": {
"laser": {
"wavelength": {
"value": 632,
"units": "nm"
}
},
"date": "02/03/2525",
"camera": {}
},
"software": {
"repo": "github.com/username/repo",
"commit": "a7642f",
"branch": "develop"
},
"data": [{},{},{}]
}
I would like an output like such.
{
"acquisition_parameters": [
"laser",
"date",
"camera"
],
"software": [
"repo",
"commit",
"branch"
]
}
This is mainly for the purpose of being able to enumerate what is in a JSON object. After processing the JSON objects from the instrument begin to diverge: for example, some may have a field like .frame.cross_section.stats.fwhm, while others may have .sample.species, so it would be convenient to be able to interrogate the JSON object on the command line.
The following should do exactly what you want
jq '[(keys - ["data"])[] as $key | { ($key): .[$key] | keys }] | add'
This will give the following output, using the input you described above:
{
"acquisition_parameters": [
"camera",
"date",
"laser"
],
"software": [
"branch",
"commit",
"repo"
]
}
Given your purpose you might have an easier time using the paths builtin to list all the paths in the input and then truncate at the desired depth:
$ echo '{"a":{"b":{"c":{"d":true}}}}' | jq -c '[paths|.[0:2]]|unique'
[["a"],["a","b"]]
Here is another variation uing reduce and setpath which assumes you have a specific set of top-level keys you want to examine:
. as $v
| reduce ("acquisition_parameters", "software") as $k (
{}; setpath([$k]; $v[$k] | keys)
)