I'm trying to figure out how to remove an array element from some JSON using jq.
Below is the input and desired output.
jq .Array[0]
outputs the array element I want.
{
"blah1": [
"key1:val1"
],
"foobar0": "barfoo0",
"foobar1": "barfoo1"
}
But how do I re-wrap this with:
{
"blah0": "zeroblah",
"Array": [
and
]
}
Input:
{
"blah0": "zeroblah",
"Array": [
{
"blah1": [
"key1:val1"
],
"foobar0": "barfoo0",
"foobar1": "barfoo1"
},
{
"blah2": [
"key2:val2"
],
"foobar2": "barfoo2",
"foobar3": "barfoo3"
}
]
}
Desired output:
{
"blah0": "zeroblah",
"Array": [
{
"blah1": [
"key1:val1"
],
"foobar0": "barfoo0",
"foobar1": "barfoo1"
}
]
}
Regarding the second part of Paul Ericson's question
But more generically, I'm trying to understand how jq would allow for selective array element control. Maybe next time I want to delete array elements 1,3,5 and 11.
To delete elements 1,3,5 and 11 just use
del(
.Array[1,3,5,11]
)
but in general you can use a more sophisticated filter as the argument to del. For example, this filter removes the elements within .Array whose .foobar2 key is "barfoo2":
del(
.Array[]
| select(.foobar2 == "barfoo2")
)
producing in this example
{
"blah0": "zeroblah",
"Array": [
{
"blah1": [
"key1:val1"
],
"foobar0": "barfoo0",
"foobar1": "barfoo1"
}
]
}
In this particular case, the simplest would be:
del(.Array[1])
More generally, if you wanted to delete all items in the array except for the first:
.Array |= [.[0]]
Related
I am new to jq. I have a json file that looks like this
[
{
"k1":"a",
"k2":"aa",
"k3":["sk1":"a","sk2":"cc","sk3":"cc"],
"k4":["sk6":"zs","sk8":"we",...],
...
},
{
"k1":"b",
"k2":"ba",
"k3":["sk1":"a","sk3":"cc",...],
"k4":["sk6":"zs","sk8":"we",...],
...
},
{
"k1":"b",
"k2":"ba",
"k4":["sk6":"zs","sk8":"we",...],
...
}
...
]
I would like to get all the entries in the array such that the key 3 ("k3") doesnt have the subkey "sk2". Note that some of the elements of the array dont have "k3" (so I would like to remove those) and then those that have "k3" sometimes dont have "sk2" (thats the ones I want).
How to accomplish this in jq?
You can use select to filter, and has to check the keys:
jq 'map(select(has("k3") and (.k3 | has("sk2") | not)))' file.json
[
{
"k1": "b",
"k2": "ba",
"k3": {
"sk1": "a",
"sk3": "cc"
},
"k4": {
"sk6": "zs",
"sk8": "we"
}
}
]
Demo
When I run the jq command to parse a json document from the amazon cli I have the following problem.
I’m parsing through the IP address and a tag called "Enviroment". The enviroment tag in the instance does not exist therefore it does not throw me any result.
Here's an example of the relevant output returned by the AWS CLI
{
"Reservations": [
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.1",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
},
{
"Key": "Environment",
"Value": "alpha"
}
]
}
]
},
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.2",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
}
]
}
]
}
]
}
I’m running the following command
aws ec2 describe-instances --filters "Name=tag:Name,Values=Balance-OTA-SS_a" | jq -c '.Reservations[].Instances[] | ({IP: .PrivateIpAddress, Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)})'
## output
empty
How do I show the IP address in the output of the command even if the enviroment tag does not exist?
Regards,
Let's assume this input:
{
"Reservations": [
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.1",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
},
{
"Key": "Environment",
"Value": "alpha"
}
]
}
]
},
{
"Instances": [
{
"PrivateIpAddress": "10.0.0.2",
"Tags": [
{
"Key": "Name",
"Value": "Balance-OTA-SS_a"
}
]
}
]
}
]
}
This is the format returned by describe-instances, but with all the irrelevant fields removed.
Note that tags is always a list of objects, each of which has a Key and a Value. This format is perfect for from_entries, which can transform this list of tags into a convenient mapping object. Try this:
.Reservations[].Instances[] |
{
IP: .PrivateIpAddress,
Ambiente: (.Tags|from_entries.Environment)
}
{"IP":"10.0.0.1","Ambiente":"alpha"}
{"IP":"10.0.0.2","Ambiente":null}
That answers how to do it. But you probably want to understand why your approach didn't work.
.Reservations[].Instances[] |
{
IP: .PrivateIpAddress,
Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)
}
The .[] filter you're using on the tags can return zero or multiple results. Similarly, the select filter can eliminate some or all items. When you apply this inside an object constructor (the expression from { to }), you're causing that whole object to be created a variable number of times. You need to be very careful where you use these filters, because often that's not what you want at all. Often you instead want to do one of the following:
Wrap the expression that returns multiple results in an array constructor [ ... ]. That way instead of outputting the parent object potentially zero or multiple times, you output it once containing an array that potentially has zero or multiple items. E.g.
[.Tags[]|select(.Key=="Environment")]
Apply map to the array to keep it an array but process its contents, e.g.
.Tags|map(select(.Key=="Environment"))
Apply first(expr) to capture only the first value emitted by the expression. If the expression might emit zero items, you can use the comma operator to provide a default, e.g.
first((.Tags[]|select(.Key=="Environment")),null)
Apply some other array-level function, such as from_entries.
.Tags|from_entries.Environment
You can either use an if ... then ... else ... end construct, or //. For example:
.Reservations[].Instances[]
| {IP: .PrivateIpAddress} +
({Ambiente: (.Tags[]|select(.Key=="Environment")|.Value)}
// null)
Given the following input JSON:
{
"version": 2,
"models": [
{
"name": "first_table",
"tests": [
{
"dbt_utils.equal_rowcount": {
"compare_model": null
}
}
]
},
{
"name": "second_table",
"tests": [
{
"dbt_utils.equal_rowcount": {
"compare_model": null
}
}
]
}
]
}
How would I, using jq, replace the null (i.e., the value of "compare_model") with the value from the "name" key? Note that the key-value pairs in question here are not at the same level in the hierarchy: the former is nested in an object in an array, and it is this array that is at the same level as the latter.
For example, the output file should read:
{
"version": 2,
"models": [
{
"name": "first_table",
"tests": [
{
"dbt_utils.equal_rowcount": {
"compare_model": "first_table"
}
}
]
},
{
"name": "second_table",
"tests": [
{
"dbt_utils.equal_rowcount": {
"compare_model": "second_table"
}
}
]
}
]
}
FWIW, this is an intermediate step in some YAML (via yq, the Python wrapper variety of jq as opposed to the go variant) wrangling I'm doing on DBT config files.
(Bonus points if you can wrap the replacement text with parentheses and/or prefix it without breaking out of jq. :D If not, no worries -- this step I can do with another program.)
Needless to say, but your help is very much appreciated!
The key to a simple solution is to use |=, e.g.
.models |=
map(.name as $name
| (.tests[]."dbt_utils.equal_rowcount".compare_model =
$name))
To wrap the replacement value in parentheses, just add them:
.models |=
map("(\(.name))" as $name
| (.tests[]."dbt_utils.equal_rowcount".compare_model =
$name))
If you want the replacement to be conditional on the existing value being null, you could perhaps (depending on the exact requirements) use //=.
Using //= and walk
Here's another take on the problem:
.models
|= map("(\(.name))" as $name
| walk(if type=="object" and has("compare_model")
then .compare_model //= $name
else . end))
That the fields are not at the same level doesn't really matter here.
.models[] |= (.tests[]."dbt_utils.equal_rowcount".compare_model = "(\(.name))")
Online demo
I'm trying to parse through some json and put certain sections into variables. I think I'm not understanding something about how variables work though.
Json:
{
"resources": [
{
"type": "Microsoft.ApiManagement/service/apis"
},
{
"type": "Microsoft.ApiManagement/service/apis/schemas"
}
]
}
Then using this jq:
.resources[] | select(.type == "Microsoft.ApiManagement/service/apis") as $apis | { types: [.type], apis: $apis}
I get this:
{
"types": [
"Microsoft.ApiManagement/service/apis"
],
"apis": {
"type": "Microsoft.ApiManagement/service/apis"
}
}
When I expected this:
{
"types": [
"Microsoft.ApiManagement/service/apis",
"Microsoft.ApiManagement/service/apis/schemas"
],
"apis": {
"type": "Microsoft.ApiManagement/service/apis"
}
}
https://jqplay.org/s/4aeBOY9x6q
According to the variables section of the jq manual
The expression exp as $x | ... means: for each value of expression
exp, run the rest of the pipeline with the entire original input, and
with $x set to that value. Thus as functions as something of a foreach
loop.
Which makes me think that .type should return from the original set not the filtered result I stored in $apis. Where is the disconnect?
It's the select that is "hiding" some of the input.
To produce the output you expect, the simplest is not to use variables at all. You could, for example, simply write:
.resources[].type
My jq command returns objects in brackets but without comma separators. But I would like to create a new json string from it.
This call finds all elements of arr that have a FooItem in them and then returns texts from the nested array at index 3:
jq '.arr[] | select(index("FooItem")) | .[3].texts'
on this json (The original has more elements ):
{
"arr": [
[
"create",
"w199",
"FooItem",
{
"index": 0,
"texts": [
"aBarfoo",
"avalue"
]
}
],
[
"create",
"w200",
"NoItem",
{
"index": 1,
"val": 5,
"hearts": 5
}
],
[
"create",
"w200",
"FooItem",
{
"index": 1,
"texts": [
"mybarfoo",
"bValue"
]
}
]
]
}
returns this output:
[
"aBarfoo",
"avalue"
]
[
"mybarfoo",
"bValue"
]
But I'd like to create a new json from these objects that looks like this:
{
"arr": [
[
"aBarfoo",
"avalue"
],
[
"mybarfoo",
"bValue"
]
]
}
Can jq do this?
EDIT
One more addition: Considering that texts also has strings of zero length, how would you delete those/not have them in the result?
"texts": ["",
"mybarfoo",
"bValue",
""
]
You can always embed a stream of (zero or more) JSON entities within some other JSON structure by decorating the stream, that is, in the present case, by wrapping the STREAM as follows:
{ arr: [ STREAM ] }
In the present case, however, we can also take the view that we are simply editing the original document, and accordingly use a variation of the map(select(...)) idiom:
.arr |= map( select(index("FooItem")) | .[3].texts)
This latter approach ensures that the context of the "arr" key is preserved.
Addendum
To filter out the empty strings, simply add another map(select(...)):
.arr |= map( select(index("FooItem"))
| .[3].texts | map(select(length>0)))