increment a numerical value in a json with jq - json

I have a json which looks like this:-
{
"name": "abc",
"version": "20.02.01",
"tag": "24",
"abc_version": "20.02",
"registry": "registry.abc.com",
"vendor": "greenplanet",
"apps": [
{
"name": "abc-app",
"version": "20.02.01-16",
"volumes": [
"/dev/log:/dev/log"
],
"max_instances": "1"
},
{
"name": "xyz-app",
"version": "2.0.0-2",
"volumes": [
"/dev/log:/dev/log"
],
"max_instances": "1"
}
]
}
based on a condition I need to increment the abc-app's or xyz-app's version. At present its at "20.02.01-16" for abc-app and I need to change it to "20.02.01-17". Also I need to increment the tag of the parent app which is "abc" to "25"
I am able to increment the version with sed but that is not working for me:-
./jq -r ".apps" version.json | ./jq -r ".[] | .version" | grep -v '1.0.2' |sed -r 's/([0-9]+.[0-9]+.[0-9]+)(.*)([0-9]+)/echo "\1\2$((\3+1))"/e'
I need to increment all the above conditions in-place in json or maybe into a temporary file which I can move to original.
Thank you in advance.

First, let's define a helper function to perform the incrementation:
# return a string
def inc:
(capture("(?<pre>.*)-(?<post>[0-9]+$)") | "\(.pre)-\( (.post|tonumber) + 1 )")
// "\(tonumber+1)" ;
The remainder of the solution can now be written in two lines, one for each value to be incremented:
.tag |= inc
| .apps |= map(if .name == "abc-app" then .version |= inc else . end)
In-place editing
You could use sponge, or a temporary file, or ... (see e.g.
"How can "in-place" editing of a JSON file be accomplished?"
on jq's FAQ).

Related

jq merge json via dynamic sub keys

I think I'm a step off from figuring out how to jq reduce via filter a key to another objects sub-key.
I'm trying to combine files (simplified from Elasticsearch's ILM Explain & ILM Policy API responses):
$ echo '{".siem-signals-default": {"modified_date": "siem", "version": 1 }, "kibana-event-log-policy": {"modified_date": "kibana", "version": 1 } }' > ip1.json
$ echo '{"indices": {".siem-signals-default-000001": {"action": "complete", "index": ".siem-signals-default-000001", "policy" : ".siem-signals-default"} } }' > ie1.json
Such that the resulting JSON is:
{
".siem-signals-default-000001": {
"modified_date": "siem",
"version": 1
"action": "complete",
"index": ".siem-signals-default-000001",
"policy": ".siem-signals-default"
}
}
Where ie1 is base JSON and for a child-object, its sub-element policy should line up to ip1's key and copy its sub-elements into itself. I've been trying to build off this, this, and this (from StackOverflow, also this, this, this from external sources). I'll list various rabbit hole attempts building off these, but they're all insufficient:
$ ((cat ie1.json | jq '.indices') && cat ip1.json) | jq -s 'map(to_entries)|flatten|from_entries' | jq '. as $v| reduce keys[] as $k({}; if true then .[$k] += $v[$k] else . end)'
{
".siem-signals-default": {
"modified_date": "siem",
"version": 1
},
".siem-signals-default-000001": {
"action": "complete",
"index": ".siem-signals-default-000001",
"policy": ".siem-signals-default"
},
"kibana-event-log-policy": {
"modified_date": "kibana",
"version": 1
}
}
$ jq --slurpfile ip1 ip1.json '.indices as $ie1|$ie1+{ilm: $ip1 }' ie1.json
{
".siem-signals-default-000001": {
"action": "complete",
"index": ".siem-signals-default-000001",
"policy": ".siem-signals-default"
},
"ilm": [
{
".siem-signals-default": {
"modified_date": "siem",
"version": 1
},
"kibana-event-log-policy": {
"modified_date": "kibana",
"version": 1
}
}
]
}
I also expected something like this to work, but it compile errors
$ jq -s ip1 ip1.json '. as $ie1|$ie1 + {ilm:(keys[] as $k; $ip1 | select(.policy == $ie1[$k]) | $ie1[$k] )}' ie1.json
jq: error: ip1/0 is not defined at <top-level>, line 1:
ip1
jq: 1 compile error
From this you can see, I've determined various ways to join the separate files, but though I have code I thought would play into filtering, it's not correct / taking effect. Does anyone have an idea how to get the filter part working? TIA
This assumes you are trying to combine the .indices object stored in ie1.json with an object within the object stored in ip1.json. As the keys upon to match are different, I further assumed that you want to match the field name from the .indices object, reduced by cutting off everything that comes after the last dash -, to the same key in the object from ip1.json.
To this end, ip1.json is read in from input as $ip (alternatively you can use jq --argfile ip ip1.json for that), then the .indices object is taken from the first input ie1.json and to the inner object accessed via with_entries(.value …) is added the result of a lookup within $ip at the matching and accordingly reduced .key.
jq '
input as $ip | .indices | with_entries(.value += $ip[.key | sub("-[^-]*$";"")])
' ie1.json ip1.json
{
".siem-signals-default-000001": {
"action": "complete",
"index": ".siem-signals-default-000001",
"policy": ".siem-signals-default",
"modified_date": "siem",
"version": 1
}
}
Demo
If instead of the .indices object's inner field nane you want to have the content of field .index as reference (which in your sample data has the same value), you can go with map_values instead of with_entries as you don't need the field's name anymore.
jq '
input as $ip | .indices | map_values(. += $ip[.index | sub("-[^-]*$";"")])
'ie1.json ip1.json
Demo
Note: I used sub with a regex to manipulate the key name, which you can easily adjust to your liking if in reality it is more complicated. If, however, the pattern is infact as simple as cutting off after the last dash, then using .[:rindex("-")] instead will also get the job done.
I also received offline feedback of a simple "workable for my use case" but not exact answer:
$ jq '.indices | map(. * input[.policy])' ie1.json ip1.json
[
{
"action": "complete",
"index": ".siem-signals-default-000001",
"policy": ".siem-signals-default",
"modified_date": "siem",
"version": 1
}
]
Posting in case someone runs into similar, but other answer's better.

jq: convert array to object indexed by filename?

Using jq how can I convert an array into object indexed by filename, or read multiple files into one object indexed by their filename?
e.g.
jq -s 'map(select(.roles[]? | contains ("mysql")))' -C dir/file1.json dir/file2.json
This gives me the data I want, but I need to know which file they came from.
So instead of
[
{ "roles": ["mysql"] },
{ "roles": ["mysql", "php"] }
]
for output, I want:
{
"file1": { "roles": ["mysql"] },
"file2": { "roles": ["mysql", "php"] }
}
I do want the ".json" file extension stripped too if possible, and just the basename (dir excluded).
Example
file1.json
{ "roles": ["mysql"] }
file2.json
{ "roles": ["mysql", "php"] }
file3.json
{ }
My real files obviously have other stuff in them too, but that should be enough for this example. file3 is simply to demonstrate "roles" is sometimes missing.
In other words: I'm trying to find files that contain "mysql" in their list of "roles". I need the filename and contents combined into one JSON object.
To simplify the problem further:
jq 'input_filename' f1 f2
Gives me all the filenames like I want, but I don't know how to combine them into one object or array.
Whereas,
jq -s 'map(input_filename)' f1 f2
Gives me the same filename repeated once for each file. e.g. [ "f1", "f1" ] instead of [ "f1", "f2" ]
If your jq has inputs (as does jq 1.5) then the task can be accomplished with just one invocation of jq.
Also, it might be more efficient to use any than iterating over all the elements of .roles.
The trick is to invoke jq with the -n option, e.g.
jq -n '
[inputs
| select(.roles and any(.roles[]; contains("mysql")))
| {(input_filename | gsub(".*/|\\.json$";"")): .}]
| add' file*.json
jq approach:
jq 'if (.roles[] | contains("mysql")) then {(input_filename | gsub(".*/|\\.json$";"")): .}
else empty end' ./file1.json ./file2.json | jq -s 'add'
The expected output:
{
"file1": {
"roles": [
"mysql"
]
},
"file2": {
"roles": [
"mysql",
"php"
]
}
}

Transform json to ini files using jq + bash

I'm trying to transform json (array of objects) to ini files:
[
{
"connection": {
"id": "br0",
"uuid": "ab1dd903-4786-4c7e-a4b4-3339b144d6c7",
"stable-id": "",
"type": "bridge",
"interface-name": "br0",
"autoconnect": "no",
"autoconnect-priority": "0",
"autoconnect-retries": "-1",
"auth-retries": "-1",
"timestamp": "44444",
"read-only": "no",
"permissions": "",
"zone": "WAN",
"master": "",
"slave-type": "",
"autoconnect-slaves": "1",
"secondaries": "",
"gateway-ping-timeout": "0",
"metered": "unknown",
"lldp": "default"
},
"ipv4": {
"method": "manual",
"dns": "192.168.1.1,192.168.2.1",
"dns-search": "",
"dns-options": " ",
"dns-priority": "0",
"addresses": "192.168.1.3/24",
"gateway": "",
"routes": "192.168.10.0/24 192.168.1.1",
"route-metric": "-1",
"route-table": "0",
"ignore-auto-routes": "no",
"ignore-auto-dns": "no",
"dhcp-client-id": "",
"dhcp-timeout": "0",
"dhcp-send-hostname": "yes",
"dhcp-hostname": "",
"dhcp-fqdn": "",
"never-default": "no",
"may-fail": "yes",
"dad-timeout": "-1"
}
},
{
"connection": {
...
},
}
]
OR
{
"connection": {
...
},
}
What i tried to do is:
1. Transform json to strings
data=$(jq -r 'def keyValue: to_entries[] | "[\(.key)]\\n\(.value | to_entries|map("\(.key)=\(.value)" ) | join("\\n") )\\n"; if type == "array" then keys[] as $k | "\(.[$k] | .connection.id)=\(.[$k] | keyValue)" elif type == "object" then "\(.connection.id)=\(. | keyValue)" else keys end' /tmp/json)
Provides:
br1=[connection]\nid=br1\nuuid=ab1dd903-4786-4c7e-a4b4-3339b144d6c7\nstable-id=\ntype=fff\ninterface-name=br0\nautoconnect=no\nautoconnect-priority=0\nautoconnect-retries=-1\nauth-retries=-1\ntimestamp=1525494904\nread-only=no\npermissions=\nzone=WAN\nmaster=\nslave-type=\nautoconnect-slaves=1\nsecondaries=\ngateway-ping-timeout=0\nmetered=unknown\nlldp=default\n
br1=[802-3-ethernet]\nport=\nspeed=0\nduplex=\nauto-negotiate=no\nmac-address=\ncloned-mac-address=\ngenerate-mac-address-mask=\nmac-address-blacklist=\nmtu=1500\ns390-subchannels=\ns390-nettype=\ns390-options=\nwake-on-lan=default\nwake-on-lan-password=\n....
2. Walk over strings in bash
while IFS="=" read -r key value; do [ "$oldkey" = "$key" ] && echo -en "$value" >> "/tmp/ini/$key" || echo -en "$value" > "/tmp/ini/$key" ; oldkey="$key"; done <<< "$data"
Gives:
[connection]
id=br1
uuid=ab1dd903-4786-4c7e-a4b4-3339b144d6c7
stable-id=
type=fff
interface-name=br0
autoconnect=no
autoconnect-priority=0
autoconnect-retries=-1
auth-retries=-1
timestamp=1525494904
read-only=no
permissions=
zone=WAN
master=
slave-type=
autoconnect-slaves=1
secondaries=
gateway-ping-timeout=0
metered=unknown
lldp=default
[ipv4]
method=manual
dns=192.168.1.1,192.168.2.1
dns-search=
dns-options=
dns-priority=0
addresses=192.168.1.3/24
gateway=
routes=192.168.10.0/24 192.168.1.1
route-metric=-1
route-table=0
ignore-auto-routes=no
ignore-auto-dns=no
dhcp-client-id=
dhcp-timeout=0
dhcp-send-hostname=yes
dhcp-hostname=
dhcp-fqdn=
never-default=no
may-fail=yes
dad-timeout=-1
I'm almost there! But is there possible to do it more "elegantly" and more performance way, avoiding pipes, external calls, etc.
Note: Moreover, mostly it should be done with jq + bash, because other processing tools like sed, awk is slower than i've done, but i do not reject them completely =)
P.S. - Main purpose of this transforming is the fast "bulk operation" to write ini files
To convert an array as shown to the ".ini" format could be accomplished by simply using this jq program:
def kv: to_entries[] | "\(.key)=\(.value)";
.[]
| to_entries[]
| "[\(.key)]", (.value|kv)
Putting this in a file, say program.jq, then assuming the JSON shown in the question (minus the "..." part) is in input.json, the following invocation:
jq -rf program.jq input.json
yields the corresponding ".ini" file.
If you want to ensure that the program will also handle the case when there is no enclosing array, you could modify the first line in the main program above to test whether the input is an array, so you'd have:
if type == "array" then .[] else . end
| to_entries[]
| "[\(.key)]", (.value|kv)
If the ultimate goal is to produce several .ini files, then we can reuse def kv as defined in the other answer:
def kv: to_entries[] | "\(.key)=\(.value)";
The driver program would however now be:
.[]
| [to_entries[] | "[\(.key)]", (.value|kv)]
| join("\n")
Running jq with this program then yields one JSON string for each item in the array. Using the example, the first such string would be:
"[connection]\nid=br0\nuuid=ab1dd903-4786-4c7e-a4b4-3339b144d6c7\nstable-id=\ntype=bridge\ninterface-name=br0\nautoconnect=no\nautoconnect-priority=0\nautoconnect-retries=-1\nauth-retries=-1\ntimestamp=44444\nread-only=no\npermissions=\nzone=WAN\nmaster=\nslave-type=\nautoconnect-slaves=1\nsecondaries=\ngateway-ping-timeout=0\nmetered=unknown\nlldp=default\n[ipv4]\nmethod=manual\ndns=192.168.1.1,192.168.2.1\ndns-search=\ndns-options= \ndns-priority=0\naddresses=192.168.1.3/24\ngateway=\nroutes=192.168.10.0/24 192.168.1.1\nroute-metric=-1\nroute-table=0\nignore-auto-routes=no\nignore-auto-dns=no\ndhcp-client-id=\ndhcp-timeout=0\ndhcp-send-hostname=yes\ndhcp-hostname=\ndhcp-fqdn=\nnever-default=no\nmay-fail=yes\ndad-timeout=-1"
You can then iterate over these newline-delimited strings.

jq update contents of one file to another as key value

I am trying to update branches.json.branch2 values from branch2.json.Employes values
Using jq, How can I merge content of one file to another file
Below are the files
I have tried this but it did work, it just prints the original data without updating the details
#!/bin/sh
#call file with branch name for example ./update.sh branch2
set -xe
branchName=$1
fullPath=`pwd`/$1".json"
list=$(cat ${fullPath})
branchDetails=$(echo ${list} | /usr/local/bin/jq -r '.Employes')
newJson=$(cat branches.json |
jq --arg updateKey "$1" --arg updateValue "$branchDetails" 'to_entries |
map(if .key == "$updateKey"
then . + {"value":"$updateValue"}
else .
end) |
from_entries')
echo $newJson &> results.json
branch1.json
{
"Employes": [
{
"Name": "Ikon",
"age": "30"
},
{
"Name": "Lenon",
"age": "35"
}
]
}
branch2.json
{
"Employes": [
{
"Name": "Ken",
"age": "40"
},
{
"Name": "Frank",
"age": "23"
}
]
}
brances.json / results.json fromat
{
"branch1": [
{
"Name": "Ikon",
"age": "30"
},
{
"Name": "Lenon",
"age": "35"
}
],
"branch2": [
{
"Name": "Ken",
"age": "40"
},
{
"Name": "Frank",
"age": "23"
}
]
}
Note: I dont have the list of all the branch files at any given point, so script is responsible only to update the that branch details.
If the file name is the name of the property you want to update, you could utilize input_filename to select the files. No testing needed, just pass in the files you want to update. Just be aware of the order you pass in the input files.
Merge the contents of the file as you see fit. To simply replace, just do a plain assignment.
$ jq 'reduce inputs as $i (.;
.[input_filename|rtrimstr(".json")] = $i.Employes
)' branches.json branch{1,2}.json
Your script would just need to be:
#!/bin/sh
#call file with branch name for example ./update.sh branch2
set -xe
branchName=$1
newJson=$(jq 'reduce inputs as $i (.; .[input_filename|rtrimstr(".json")] = $i.Employees)' branches.json "$branchName.json")

How to update a subitem in a json file using jq?

Using jq I tried to update this json document:
{
"git_defaults": {
"branch": "master",
"email": "jenkins#host",
"user": "Jenkins"
},
"git_namespaces": [
{
"name": "NamespaceX",
"modules": [
"moduleA",
"moduleB",
"moduleC",
"moduleD"
]
},
{
"name": "NamespaceY",
"modules": [
"moduleE"
]
}
]
}
with adding moduleF to NamespaceY. I need to write the file back again to the original source file.
I came close (but no cigar) with:
jq '. | .git_namespaces[] | select(.name=="namespaceY").modules |= (.+ ["moduleF"])' config.json
and
jq '. | select(.git_namespaces[].name=="namespaceY").modules |= (.+ ["moduleF"])' config.json
The following filter should perform the update you want:
(.git_namespaces[] | select(.name=="NamespaceY").modules) += ["moduleF"]
Note that the initial '.|' in your attempt is not needed; that "NamespaceY" is capitalized in config.json; that the parens as shown are the keys to success; and that += can be used here.
One way to write back to the original file would perhaps be to use 'sponge'; other possibilities are discussed on the jq FAQ https://github.com/stedolan/jq/wiki/FAQ