I have a list of objects and want to replace all occurrences of . with : when the key is Name using jq
input:
{
"Parameters": [
{
"Name": "TEST.AB.SOMETHING",
"Value": "hvfuycsgvfiwbiwbibibewfiwbcfwifcbwibcibc"
},
{
"Name": "TEST_GF_USER",
"Value": "ssssecret"
}
]
}
expected output:
{
"Parameters": [
{
"Name": "TEST:AB:SOMETHING",
"Value": "hvfuycsgvfiwbiwbibibewfiwbcfwifcbwibcibc"
},
{
"Name": "TEST_GF_USER",
"Value": "ssssecret"
}
]
}
You may split by . and join by :
jq '(.Parameters[].Name)|=(split(".")|join(":"))' file.json
The assignment is done using the update operator.
The trick is to use .Name |= gsub("\\.";":"). In your case (a flat list), it's simple. If you want to modify the keys of all objects in an arbitrary JSON text, the simplest would be to use walk/1:
walk( if type == "object" and has("Name") then .Name |= gsub("\\.";":")) else . end )
(If your jq does not have walk/1, then its jq definition can readily be found by googling.)
Related
I have following JSON. I want to get key-value pair objects based on their CC role. In this example there are 3 roles(Presenter, Approver, Customer). Presenter is of type TO. Other 2 are of type CC. I want to get of type CC. There can be more as it is dynamic.
JSON
{
"Presenter_TO_Email": "roney#domain.com",
"Approver_CC_Email": "tim#domain.com",
"Customer_CC_Email": "alex#domain.com",
"Invoice": "001",
"Date": "2022-02-14"
}
Output
{
"Approver": {
"email_address": "tim#domain.com",
"role": "Approver"
},
"Customer": {
"email_address": "alex#domain.com",
"role": "Customer"
}
}
I can do using INDEX using this example but as I am using older version of jq, it throws error jq: error: INDEX/2 is not defined at <top-level>, line 1:
Use with_entries to make changes based on keys and values:
jq '
with_entries(
(.key / "_CC_") as $key | select($key[1])
| {key: $key[0], value: {email_address: .value, role: $key[0]}}
)
'
{
"Approver": {
"email_address": "tim#domain.com",
"role": "Approver"
},
"Customer": {
"email_address": "alex#domain.com",
"role": "Customer"
}
}
Demo
In versions of jq which include INDEX/2, it is defined as a simple jq function, so if your jq does not include it, you can simply include its definition yourself:
def INDEX(stream; idx_expr):
reduce stream as $row ({};
.[$row|idx_expr|
if type != "string" then tojson
else .
end] |= $row);
Given the following input JSON:
{
"version": 2,
"models": [
{
"name": "first_table",
"tests": [
{
"dbt_utils.equal_rowcount": {
"compare_model": null
}
}
]
},
{
"name": "second_table",
"tests": [
{
"dbt_utils.equal_rowcount": {
"compare_model": null
}
}
]
}
]
}
How would I, using jq, replace the null (i.e., the value of "compare_model") with the value from the "name" key? Note that the key-value pairs in question here are not at the same level in the hierarchy: the former is nested in an object in an array, and it is this array that is at the same level as the latter.
For example, the output file should read:
{
"version": 2,
"models": [
{
"name": "first_table",
"tests": [
{
"dbt_utils.equal_rowcount": {
"compare_model": "first_table"
}
}
]
},
{
"name": "second_table",
"tests": [
{
"dbt_utils.equal_rowcount": {
"compare_model": "second_table"
}
}
]
}
]
}
FWIW, this is an intermediate step in some YAML (via yq, the Python wrapper variety of jq as opposed to the go variant) wrangling I'm doing on DBT config files.
(Bonus points if you can wrap the replacement text with parentheses and/or prefix it without breaking out of jq. :D If not, no worries -- this step I can do with another program.)
Needless to say, but your help is very much appreciated!
The key to a simple solution is to use |=, e.g.
.models |=
map(.name as $name
| (.tests[]."dbt_utils.equal_rowcount".compare_model =
$name))
To wrap the replacement value in parentheses, just add them:
.models |=
map("(\(.name))" as $name
| (.tests[]."dbt_utils.equal_rowcount".compare_model =
$name))
If you want the replacement to be conditional on the existing value being null, you could perhaps (depending on the exact requirements) use //=.
Using //= and walk
Here's another take on the problem:
.models
|= map("(\(.name))" as $name
| walk(if type=="object" and has("compare_model")
then .compare_model //= $name
else . end))
That the fields are not at the same level doesn't really matter here.
.models[] |= (.tests[]."dbt_utils.equal_rowcount".compare_model = "(\(.name))")
Online demo
I would like to aggregate two or more files into a single json, and aggregate arrays under a same key.
file1.json
{
"shapes": [
{
"id": "1",
"name": "circle"
},
{
"id": "2",
"name": "square"
}
]
}
file2.json
{
"shapes": [
{
"id": "3",
"name": "triangle"
}
]
}
Expected result :
{
"shapes": [
{
"id": "1",
"name": "circle"
},
{
"id": "2",
"name": "square"
},
{
"id": "3",
"name": "triangle"
}
]
}
I can do this with the following jq command :
jq -s '{shapes: map(.shapes)|add }' file*.json
But this requires me to know the shapes attribute and hardcode it. Is there a simple way I can get the same result without ever using the key name explicitly?
Here is a solution that’s suitable when each top-level object has only one key, and that is both efficient and conceptually simple. It assumes jq is invoked with the -n option.
reduce inputs as $in (null;
($in|keys_unsorted[0]) as $k | { ($k): (.[$k] + $in[$k]) })
or slightly more compactly:
reduce inputs as $in (null; ($in|keys_unsorted[0]) as $k | .[$k] += $in[$k] )
Here is a solution that also solves a more general problem: first, it handles arbitrarily many input files; and second, it forms the "sum" by key, for every key, on the assumption that every top-level key is array-valued.
The generic function:
# the values at each key are assumed to be arrays
def aggregate(stream):
reduce stream as $o ({};
reduce ($o|keys_unsorted[]) as $k (.;
.[$k] += $o[$k] ));
To avoid "slurping", we will use inputs:
aggregate(inputs)
The invocation must therefore use the -n command-line option:
jq -n -f program.jq *.json
Try the following code. This can handle any number of files. All inputs are assumed to be json objects with all values inside as arrays. All such arrays are aggregated after grouping by keys. It outputs an object which has keys associated with corresponding aggregated arrays.
jq -s 'map(to_entries)|add|group_by(.key)|
map( { "key": (.[0].key), "value": (map(.value)|add)})|
from_entries' file1.json file2.json
For your sample input this gives:
{
"shapes": [
{
"id": "1",
"name": "circle"
},
{
"id": "2",
"name": "square"
},
{
"id": "3",
"name": "triangle"
}
]
}
Example JSON file:
{
"u": "stuff",
"x": [1,2,3],
"y": {
"field": "value"
},
"z": {
"zz": {
"name": "change me",
"more": "stuff"
},
"randomKey": {
"name": "change me",
"random": "more stuff"
}
}
}
How can I update all the name fields to "something", maintaining the rest of the JSON file the same?
{
"u": "stuff",
"x": [1,2,3],
"y": {
"field": "value"
},
"z": {
"zz": {
"name": "something",
"more": "stuff"
},
"randomKey": {
"name": "something",
"random": "more stuff"
}
}
}
With a direct path, this would be easy, but the parent keys (z and randomKey in these case) varies.
I tried something like:
jq '.z | .. | .name? |= "something"' file.json
And it's updating the names, but putting also all the recursive stuff..
If it is acceptable to change the "name" field wherever it occurs, you could use walk/1:
walk(if type == "object" and has("name") then .name = "something" else . end)
Please note that walk/1 was only included with jq after jq 1.5 was released. If your jq does not have it, then you can find its definition on the jq FAQ, for example.
If you only want to modify the "name" field in the "z" context, then consider:
.z |= with_entries(if .value.name?
then .value.name = "something"
else . end)
Assuming every value within z has a name property, you could do this:
$ jq --arg newname 'something' '.z[].name = $newname' input.json
Using [] on an object will yield all the values contained in that object. And for each of those values, we were simply setting the name to the new name.
If you needed to be more selective with what gets updated, you'll have to add more conditions to what objects to update. In general, I'd use peak's approach, but here's another way it could be achieved using a structure similar to the first approach, assuming we only want to update objects that already have a name property:
$ jq --arg newname 'something' '(.z[] | select(has("name")).name) = $newname' input.json
It's important to wrap the LHS of the assignment in parentheses, we don't want to change the context prior to the assignment, otherwise we won't see the rest of the results.
I want to update a value in a dict, which I can only identify by another value in the dict. That is, given this input:
[
{
"format": "geojson",
"id": "foo"
},
{
"format": "geojson",
"id": "bar"
},
{
"format": "zip",
"id": "baz"
}
]
I want to change baz's accompanying format to 'csv':
[
{
"format": "geojson",
"id": "foo"
},
{
"format": "geojson",
"id": "bar"
},
{
"format": "csv",
"id": "baz"
}
]
I have found that this works:
jq 'map(if .id=="baz" then .format="csv" else . end)' my.json
But this seems rather verbose, so I wonder if there is a more elegant way to express this. jq seems to be missing some kind of expression selector, the equivalent of might be [#id='baz'] in xpath.
(When I started this question, I had [.[] |...], then I discovered map, so it's not quite as bad as I thought.)
A complex assignment is what you're looking for:
jq '(.[] | select(.id == "baz") | .format) |= "csv"' my.json
Perhaps not shorter but it is more elegant, as requested. See the last section of the docs at: http://stedolan.github.io/jq/manual/#Assignment
Edit: using map:
jq 'map((select(.id == "baz") | .format) |= "csv")' my.json