Is there a command to use jq to add onto this type of map?
append an array of maps using keys (ie, vm1, vm2, vm3)
Note: I have an existing vm_map {} in a json file and i want to add to the vm_map
this is my new_json.json file
{
"gcs_config": [
{
"bucket_name": "somebucket",
"bucket_readers": [],
"bucket_writers": []
}
],
"label_application": "someapp",
"label_environment": "dev",
"lits_vm_zone": "somezone",
"project_id": "someproject",
"region": "someregion",
"storage_bucket_required": true,
"vm_map" : {}
}
expected: using jq to add onto vm_maps map. I will have an empty vm_map and each time it runs, i will add a x amount of new entries.
{
"gcs_config": [
{
"bucket_name": "somebucket",
"bucket_readers": [],
"bucket_writers": []
}
],
"label_application": "someapp",
"label_environment": "dev",
"lits_vm_zone": "zone-a",
"project_id": "someproject",
"region": "someregion",
"storage_bucket_required": true,
"vm_map": {
"vm1": {
"host": "vm1",
"network": "10.1.1.1",
"name": "vm1"
},
"vm2": {
"host": "123",
"network": "10.1.12",
"name": "vm2"
}
}
}
The file you describe is not valid JSON. I'm assuming you mean
{
"vm_map": {
"vm1": {
"host": "vm1",
"network": "xxxxx",
"name": "xxxxxxx"
},
"vm2": {
"host": "vm2",
"network": "xxxxx",
"name": "xxxxxxx"
}
}
}
You can use this:
jq \
--arg VMHOST "$VMHOST" \
--arg NETWORK_IP "$NETWORK_IP" \
--arg VM_NAME "$VM_NAME" \
'
.vm_map[ $VMHOST ] = {
host: $VMHOST,
network: $NETWORK_IP,
name: $VM_NAME
}
'
Related
I have been able to get the list of the value for a particular key which is array in the provided json to be in the shell script :
The "echo ${list.jvm.pega['jenkins-name']}" output these values :
["ent-giem-sasw02","ent-giem-sasw03","ent-giem-sasw04"]
How would I be able to loop each of the list in the in the array and pass it to the node under stage. Expected solution is to loop each of the list as
stage('delete_pegarules_schema_for_each_node') {
///loop 3 times based on the ${list.jvm.pega['jenkins-name'] output list
node("${list.jvm.pega['jenkins-name']") {
sh """
echo -e "-------------------------------System Information of current node running ----------------------------"
echo -e "Hostname:\t\t"`hostname`
"""
}
}
The script is :
#!/usr/bin/env groovy
node{
properties([
parameters([
choice(
name: 'environment',
choices: ['','upgrade', 'BV' ],
description: 'environment to choose'
),
])
])
deleteDir()
dir('dir-switch') {
stage('Checkout') {
// git branch: 'test-upgrade-json', url: 'https://gitlab.xxxxxxx/pipeline.git'
// stash includes: '**', name: 'upgrade-pega'
checkout([$class: 'GitSCM', branches: [[name: '*/test-upgrade-json']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'jenkins-user-github', url: 'https://gitlab.xxxxx/pipeline.git']]])
}
stage('Get Environment') {
sh """
ls -lart ./*
ls ${env.WORKSPACE}
cp -R ./upgrade-pega/environment/env.json /${env.WORKSPACE}/dir-switch
ls /${env.WORKSPACE}/dir-switch
"""
}
def obj = readJSON file: './env.json'
def list = obj[params.environment];
println list
list.each { println (it) }
stage('JVM check content') {
sh "echo ${list.jvm.pega['jenkins-name']}"
}
stage('delete_pegarules_schema_for_each_node') {
///loop
node("Expecting the each of the loop in the list of ${list.jvm.pega['jenkins-name']}") {
sh """
echo -e "-------------------------------System Information of current node running ----------------------------"
echo -e "Hostname:\t\t"`hostname`
"""
}
}
}
}
The json file :
{
"upgrade": {
"level": 1,
"domain": "develop.autosample.co.uk",
"resources": {
"db-schemas": {
"rule": {
"schema": "pegarules",
"database": "sas_develop",
"jvm": ["primary", "secondary"]
}
}
},
"jvm": {
"load-balancer": null,
"pega": [{
"jenkins-name": "ent-giem-sasw02",
"host": "ent-giem-sasw02",
"ip": "x.x.x.x"
},
{
"jenkins-name": "ent-giem-sasw03",
"host": "ent-giem-sasw03",
"ip": "x.x.x.x"
},
{
"jenkins-name": "ent-giem-sasw04",
"host": "ent-giem-sasw04",
"ip": "x.x.x.x"
}
],
"db": {
"primary": {
"jenkins-name": "ent-giem-sasrd26",
"host": "ent-giem-sasrd26",
"ip": "x.x.x.x",
"port": 5432
},
"secondary": {
"jenkins-name": "ent-giem-sasrd98",
"host": "ent-giem-pgrd98",
"ip": "x.x.x.x",
"port": 5432
}
}
}
}
}
I added this :
list.jvm.pega['jenkins-name'].each { elemupdate ->
echo "Item: ${elemupdate}"
stage('delete_pegarules_schema') {
node("$elemupdate") {
......
}
}
I have below JSON where in I need to add key and values at a specific place from env variable. Moreover would try to avoid any hardcoding in JSON if possible
Sample JSON:
{
"abc": {
"admin": {
"username": "admin",
"password": "pa55word",
"tenant_name": "t1"
},
"users": [
{
"username": "n1",
"password": "password",
"user_domain_name": "Default",
"project_name": "p1",
"project_domain_name": "Default"
},
{
"username": "n2",
"password": "password",
"user_domain_name": "Default",
"project_name": "p2",
"project_domain_name": "Default"
}
]
}
}
Output json needs to be like below where in $AUTH and $REG are env variables.
{
"abc": {
"auth_url": "$AUTH",
"region_name": "$REG",
"endpoint_type": "PRIVATE",
"admin": {
"username": "admin",
"password": "pa55word",
"tenant_name": "t1"
},
"users": [
{
"username": "n1",
"password": "password",
"user_domain_name": "Default",
"project_name": "p1",
"project_domain_name": "Default"
},
{
"username": "n2",
"password": "password",
"user_domain_name": "Default",
"project_name": "p2",
"project_domain_name": "Default"
}
]
}
}
Looked into jq but it seemed confusing. How this can be achieved using jq?
Grab AUTH and REG environment variables from $ENV preset variable and create fields using them, then add the resulting object to the first member of the input. E.g:
first(.[]) |= ( $ENV | {
auth_url: .AUTH,
region_name: .REG,
endpoint_type: "PRIVATE"
}
) + .
The simplest approach would be to take advantage of the fact that + on objects respects the ordering of keys:
.abc |= {auth_url: env.AUTH, region_name: env.REG, endpoint_type: "PRIVATE"} + .
This assumes that AUTH and REG are environment variables. If they are not, then you could use the --arg and/or --argjson command-line options, e.g.
jq --arg AUTH "$AUTH" --arg REG "$REG" '
.abc |= {auth_url: $AUTH, region_name: $REG, endpoint_type: "PRIVATE"} + .
' sample.json
I have a JSON like this (sample.json):
{
"sheet1": [
{
"hostname": "sv001",
"role": "web",
"ip1": "172.17.0.3"
},
{
"hostname": "sv002",
"role": "web",
"ip1": "172.17.0.4"
},
{
"hostname": "sv003",
"role": "db",
"ip1": "172.17.0.5",
"ip2": "172.18.0.5"
}
],
"sheet2": [
{
"hostname": "sv004",
"role": "web",
"ip1": "172.17.0.6"
},
{
"hostname": "sv005",
"role": "db",
"ip1": "172.17.0.7"
},
{
"hostname": "vsv006",
"role": "db",
"ip1": "172.17.0.8"
}
],
"sheet3": []
}
I want to extract data like this:
sheet1
jq '(something command)' sample.json
{
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
},
"db": {
"hosts": [
"172.17.0.5"
]
}
}
Is it possible to perform the reconstruction with jq map?
(I will reuse the result for ansible inventory.)
Here's a short, straight-forward and efficient solution -- efficient in part because it avoids group_by by courtesy of the following generic helper function:
def add_by(f;g): reduce .[] as $x ({}; .[$x|f] += [$x|g]);
.sheet1
| add_by(.role; .ip1)
| map_values( {hosts: .} )
Output
This produces the required output:
{
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
},
"db": {
"hosts": [
"172.17.0.5"
]
}
}
If the goal is to regroup the ips by their roles within each sheet you could do this:
map_values(
reduce group_by(.role)[] as $g ({};
.[$g[0].role].hosts = [$g[] | del(.hostname, .role)[]]
)
)
Which produces something like this:
{
"sheet1": {
"db": {
"hosts": [
"172.17.0.5",
"172.18.0.5"
]
},
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4"
]
}
},
"sheet2": {
"db": {
"hosts": [
"172.17.0.7",
"172.17.0.8"
]
},
"web": {
"hosts": [
"172.17.0.6"
]
}
},
"sheet3": {}
}
https://jqplay.org/s/3VpRc5l4_m
If you want to flatten all to a single object keeping only unique ips, you can keep everything mostly the same, you'll just need to flatten the inputs prior to grouping and remove the map_values/1 call.
$ jq -n '
reduce ([inputs[][]] | group_by(.role)[]) as $g ({};
.[$g[0].role].hosts = ([$g[] | del(.hostname, .role)[]] | unique)
)
'
{
"db": {
"hosts": [
"172.17.0.5",
"172.17.0.7",
"172.17.0.8",
"172.18.0.5"
]
},
"web": {
"hosts": [
"172.17.0.3",
"172.17.0.4",
"172.17.0.6"
]
}
}
https://jqplay.org/s/ZGj1wC8hU3
I have json file which is constructed in simmilar way:
[
{
"_id":"1234",
"org":"org1",
"int":
{"url":"http://url.com.uk:1234"}},
{
"_id":"4321",
"org":"org2",
"int":
{"url":"http://url.com.us:4321"}},
...
]
Now im "jumping" from one entry to another and checking if under URL application is working properly. After check i want to add/update field "status". But i can't update whole file, im just getting:
$ jq --arg mod "GOOD" '.[0].int + {stat: $mod}' tmp.json
{
"url": "http://url.com.uk:1234",
"stat": "GOOD"
}
How can i with jq command get new updated whole file, not just only part of it?
If you put your data in data.json and the changes you want to make to
each record into a separate arg.json argument file like
{
"1234": { "int": { "stat": "GOOD" } },
"4321": { "int": { "stat": "BAD", "xxx": "yyy" } }
}
and run jq as
$ jq -M --argfile arg arg.json 'map(. + $arg[._id])' data.json
then it will output the updated data, e.g.
[
{
"_id": "1234",
"org": "org1",
"int": {
"stat": "GOOD"
}
},
{
"_id": "4321",
"org": "org2",
"int": {
"stat": "BAD",
"xxx": "yyy"
}
}
]
Note that the + replaces keys. If you want to merge keys you can use * e.g.
$ jq -M --argfile arg arg.json 'map(. * $arg[._id])' data.json
which generates
[
{
"_id": "1234",
"org": "org1",
"int": {
"url": "http://url.com.uk:1234",
"stat": "GOOD"
}
},
{
"_id": "4321",
"org": "org2",
"int": {
"url": "http://url.com.us:4321",
"stat": "BAD",
"xxx": "yyy"
}
}
]
If you want to update the data in place you could use sponge
as described in the answer Manipulate JSON with jq
e.g.
$ jq -M --argfile arg arg.json 'map(. * $arg[._id])' data.json | sponge data.json
You can map to array and resign the int by operation, like:
jq --arg mod "GOOD" '.[] | .int=.int + {stat: $mod}' tmp.json
{
"_id": "1234",
"org": "org1",
"int": {
"url": "http://url.com.uk:1234",
"stat": "GOOD"
}
}
{
"_id": "4321",
"org": "org2",
"int": {
"url": "http://url.com.us:4321",
"stat": "GOOD"
}
}
I a have a JSON file in which a JSON object needs to be deleted.
Is there any way to delete it by using the sed command only?
My sample file is given below.
{
"name": "ABC",
"description": "XYZ",
"versions": {},
"json_class": "Chef::Environment",
"chef_type": "environment",
"default_attributes": {},
"override_attributes": {
"company": {
"xyz": {
"mailer": {
"smtp_host": "Some IP",
"imap_host": "Some Host",
"imap_user": "Some UserId",
"reply_to": "Some User Id",
"outbound_user": "",
"imap_mail_domain": "compute.oraclecloud.com",
"imap_password": ""
},
"logicaldomainname": ""
}
}
}
}
I want to delete the mailer object from the file using the sed command.
Please take a look and let me know if it is possible using the sed command.
My main motive of this deletion is to replace this object with new
mailer object so that the next time when another mailer object will be posted then we will append it in xyz JSON object.
Expected output:
{
"name": "ABC",
"description": "",
"version": {},
"json_class": "Chef::Environment",
"chef_type": "environment",
"default_attributes": {},
"override_attributes": {
"Company": {
"XYZ": {
"logicaldomainname": ""
}
}
}
}
Using sed:
sed '/\"mailer\"/,/}/ d; /^$/d' 1.a
"Delete from mailer to the first }, then delete all blank lines (you input has one)".
To overwrite the file ("in-place")
sed -i ...
Note: works for multiline files (as yours).
You should strongly consider resolving the licensing issue that prohibits jq:
jq 'del(.override_attributes.company.xyz.mailer) | .description=""' input.json
$ awk -v RS='^$' -v ORS= '{sub(/"mailer":[^}]+},[[:space:]]+/,"")}1' file
{
"name": "ABC",
"description": "",
"versions": {
},
"json_class": "Chef::Environment",
"chef_type": "environment",
"default_attributes": {
},
"override_attributes": {
"Company": {
"XYZ": {
"logicaldomainname": ""
}
}
}
}
The above uses GNU awk for multi-char RS, with other awks you'd build up the record one line at a time and handle it in the END section:
$ awk -v ORS= '{rec=rec $0 RS} END{sub(/"mailer":[^}]+},[[:space:]]+/,"",rec); print rec}' file
{
"name": "ABC",
"description": "",
"versions": {
},
"json_class": "Chef::Environment",
"chef_type": "environment",
"default_attributes": {
},
"override_attributes": {
"Company": {
"XYZ": {
"logicaldomainname": ""
}
}
}
}