Generate fields using environment variables and insert them into object - json

I have below JSON where in I need to add key and values at a specific place from env variable. Moreover would try to avoid any hardcoding in JSON if possible
Sample JSON:
{
"abc": {
"admin": {
"username": "admin",
"password": "pa55word",
"tenant_name": "t1"
},
"users": [
{
"username": "n1",
"password": "password",
"user_domain_name": "Default",
"project_name": "p1",
"project_domain_name": "Default"
},
{
"username": "n2",
"password": "password",
"user_domain_name": "Default",
"project_name": "p2",
"project_domain_name": "Default"
}
]
}
}
Output json needs to be like below where in $AUTH and $REG are env variables.
{
"abc": {
"auth_url": "$AUTH",
"region_name": "$REG",
"endpoint_type": "PRIVATE",
"admin": {
"username": "admin",
"password": "pa55word",
"tenant_name": "t1"
},
"users": [
{
"username": "n1",
"password": "password",
"user_domain_name": "Default",
"project_name": "p1",
"project_domain_name": "Default"
},
{
"username": "n2",
"password": "password",
"user_domain_name": "Default",
"project_name": "p2",
"project_domain_name": "Default"
}
]
}
}
Looked into jq but it seemed confusing. How this can be achieved using jq?

Grab AUTH and REG environment variables from $ENV preset variable and create fields using them, then add the resulting object to the first member of the input. E.g:
first(.[]) |= ( $ENV | {
auth_url: .AUTH,
region_name: .REG,
endpoint_type: "PRIVATE"
}
) + .

The simplest approach would be to take advantage of the fact that + on objects respects the ordering of keys:
.abc |= {auth_url: env.AUTH, region_name: env.REG, endpoint_type: "PRIVATE"} + .
This assumes that AUTH and REG are environment variables. If they are not, then you could use the --arg and/or --argjson command-line options, e.g.
jq --arg AUTH "$AUTH" --arg REG "$REG" '
.abc |= {auth_url: $AUTH, region_name: $REG, endpoint_type: "PRIVATE"} + .
' sample.json

Related

jq command to add onto a map

Is there a command to use jq to add onto this type of map?
append an array of maps using keys (ie, vm1, vm2, vm3)
Note: I have an existing vm_map {} in a json file and i want to add to the vm_map
this is my new_json.json file
{
"gcs_config": [
{
"bucket_name": "somebucket",
"bucket_readers": [],
"bucket_writers": []
}
],
"label_application": "someapp",
"label_environment": "dev",
"lits_vm_zone": "somezone",
"project_id": "someproject",
"region": "someregion",
"storage_bucket_required": true,
"vm_map" : {}
}
expected: using jq to add onto vm_maps map. I will have an empty vm_map and each time it runs, i will add a x amount of new entries.
{
"gcs_config": [
{
"bucket_name": "somebucket",
"bucket_readers": [],
"bucket_writers": []
}
],
"label_application": "someapp",
"label_environment": "dev",
"lits_vm_zone": "zone-a",
"project_id": "someproject",
"region": "someregion",
"storage_bucket_required": true,
"vm_map": {
"vm1": {
"host": "vm1",
"network": "10.1.1.1",
"name": "vm1"
},
"vm2": {
"host": "123",
"network": "10.1.12",
"name": "vm2"
}
}
}
The file you describe is not valid JSON. I'm assuming you mean
{
"vm_map": {
"vm1": {
"host": "vm1",
"network": "xxxxx",
"name": "xxxxxxx"
},
"vm2": {
"host": "vm2",
"network": "xxxxx",
"name": "xxxxxxx"
}
}
}
You can use this:
jq \
--arg VMHOST "$VMHOST" \
--arg NETWORK_IP "$NETWORK_IP" \
--arg VM_NAME "$VM_NAME" \
'
.vm_map[ $VMHOST ] = {
host: $VMHOST,
network: $NETWORK_IP,
name: $VM_NAME
}
'

Using jq to fetch and show key value with quotes

I have a file that looks as below:
{
"Job": {
"Name": "sample_job",
"Description": "",
"Role": "arn:aws:iam::00000000000:role/sample_role",
"CreatedOn": "2021-10-21T23:35:23.660000-03:00",
"LastModifiedOn": "2021-10-21T23:45:41.771000-03:00",
"ExecutionProperty": {
"MaxConcurrentRuns": 1
},
"Command": {
"Name": "glueetl",
"ScriptLocation": "s3://aws-sample-s3/scripts/sample.py",
"PythonVersion": "3"
},
"DefaultArguments": {
"--TempDir": "s3://aws-sample-s3/temporary/",
"--class": "GlueApp",
"--enable-continuous-cloudwatch-log": "true",
"--enable-glue-datacatalog": "true",
"--enable-metrics": "true",
"--enable-spark-ui": "true",
"--job-bookmark-option": "job-bookmark-enable",
"--job-insights-byo-rules": "",
"--job-language": "python",
"--spark-event-logs-path": "s3://aws-sample-s3/logs"
},
"MaxRetries": 0,
"AllocatedCapacity": 100,
"Timeout": 2880,
"MaxCapacity": 100.0,
"WorkerType": "G.1X",
"NumberOfWorkers": 100,
"GlueVersion": "2.0"
}
}
I want to get key/value from "Name", "--enable-continuous-cloudwatch-log": "" and "--enable-metrics": "". So, I need to show the info like this:
"Name" "sample_job"
"--enable-continuous-cloudwatch-log" ""
"--enable-metrics" ""
UPDATE
Follow the tips from #Inian and #0stone0 I came close to it:
jq -r '(.Job ) + (.Job.DefaultArguments | { "--enable-continuous-cloudwatch-log", "--enable-metrics"}) | to_entries[] | "\"\(.key)\" \"\(.value)\""'
This extract the values I need but show all another key/values.
Since you're JSON isn't valid, I've converted it into:
{
"Job": {
"Name": "sample_job",
"Role": "sample_role_job"
},
"DefaultArguments": {
"--enable-continuous-cloudwatch-log": "test_1",
"--enable-metrics": ""
},
"Timeout": 2880,
"NumberOfWorkers": 10
}
Using the following filter:
"Name \(.Job.Name)\n--enable-continuous-cloudwatch-log \(.DefaultArguments."--enable-continuous-cloudwatch-log")\n--enable-metrics \(.DefaultArguments."--enable-metrics")"
We use string interpolation to show the desired output:
Name sample_job
--enable-continuous-cloudwatch-log test_1
--enable-metrics
jq --raw-output '"Name \(.Job.Name)\n--enable-continuous-cloudwatch-log \(.DefaultArguments."--enable-continuous-cloudwatch-log")\n--enable-metrics \(.DefaultArguments."--enable-metrics")"'
Online Demo

Using jq find key/value pair based on another key/value pair

I'm pasting here a JSON example data which would require some manipulation to get a desired output which is mentioned in the next section to be read after this piece of JSON code.
I want to use jq for parsing my desired data.
{
"MetricAlarms": [
{
"EvaluationPeriods": 3,
"ComparisonOperator": "GreaterThanOrEqualToThreshold",
"AlarmActions": [
"Unimportant:Random:alarm:ELK2[10.1.1.2]-Root-Disk-Alert"
],
"AlarmName": "Unimportant:Random:alarm:ELK1[10.1.1.0]-Root-Alert",
"Dimensions": [
{
"Name": "path",
"Value": "/"
},
{
"Name": "InstanceType",
"Value": "m5.2xlarge"
},
{
"Name": "fstype",
"Value": "ext4"
}
],
"DatapointsToAlarm": 3,
"MetricName": "disk_used_percent"
},
{
"EvaluationPeriods": 3,
"ComparisonOperator": "GreaterThanOrEqualToThreshold",
"AlarmActions": [
"Unimportant:Random:alarm:ELK2[10.1.1.2]"
],
"AlarmName": "Unimportant:Random:alarm:ELK2[10.1.1.2]",
"Dimensions": [
{
"Name": "path",
"Value": "/"
},
{
"Name": "InstanceType",
"Value": "r5.2xlarge"
},
{
"Name": "fstype",
"Value": "ext4"
}
],
"DatapointsToAlarm": 3,
"MetricName": "disk_used_percent"
}
]
}
So when I Pass some Key1 & value1 as a parameter "Name": "InstanceType", to the JQ probably using cat | jq and output expected should be as below
m5.2xlarge
r5.2xlarge
A generic approach to search for a key-value pair (sk-sv) in input recursively and extract another key's value (pv) from objects found:
jq -r --arg sk Name \
--arg sv InstanceType \
--arg pv Value \
'.. | objects | select(contains({($sk): $sv})) | .[$pv]' file

Use jq to get parent key for a selected object

I asked a question about getting a key and value from a JSON structure here: Using jq to get key and value from JSON
This is the JSON:
{
"63": {
"state": {
"on": false,
"alert": "select",
"mode": "automation",
"reachable": true
},
"swupdate": {
"state": "notupdatable",
"lastinstall": "2019-09-15T11:19:15"
},
"type": "plug",
"name": "Tree",
"modelid": "XXX",
"manufacturername": "XXX",
"productname": "plug",
"capabilities": {
"certified": false,
"control": {},
"streaming": {
"renderer": false,
"proxy": false
}
},
"config": {
"archetype": "plug",
"function": "functional",
"direction": "omnidirectional"
},
"uniqueid": "00:0d:6f:ff:fe:da:c9:dc-01",
"swversion": "2.0.022"
}
}
I want to emit only the parent key for an object based on the contents of .name: where .name == "Tree", return "63".
I can emit the whole object with:
jq -r '.[] | select(.name == "Tree")'
or a list of key and name with:
jq -r 'map_values(.name)'
You can format the JSON in a key/value format using to_entries() and get the key corresponding to a certain value
jq --raw-output 'to_entries[] | select(.value.name == "Tree").key'

Add or Update a field in one JSON file from another JSON file based on matching field

I have two JSON files a.json and b.json. The contents in a.json file is a JSON object and inside b.json its an array.I want to add/update status field in each mappings in a.json by retrieving the value from b.json file.
a.json:
{
"title": 25886,
"data": {
"request": {
"c": 46369,
"t1": 1562050127.376641
},
},
"rs": {
"mappings": {
"12345": {
"id": "12345",
"name": "test",
"customer_id": "11228",
},
"45678": {
"id": "45678",
"name": "abc",
"customer_id": "11206",
}
}
}}
b.json:
[
{
"status": "pending",
"extra": {
"name": "test"
},
"enabled": true,
"id": "12345"
},
{
"status": "not_started",
"extra": {
"name": "abc"
},
"enabled": true,
"id": "45678"
}
]
Below is my expected output:
{
"title": 25886,
"data": {
"request": {
"c": 46369,
"t1": 1562050127.376641
},
},
"rs": {
"mappings": {
"12345": {
"id": "12345",
"name": "test",
"customer_id": "11228",
"status":"pending"
},
"45678": {
"id": "45678",
"name": "abc",
"customer_id": "11206",
"status":"not_started"
}
}
}}
In this expected JSON file we have status field whose value is retrieved from b.json file based on a matching id value. How to do this using jq ?
For the purposes of this problem, b.json essentially defines a dictionary, so for simplicity, efficiency and perhaps elegance,
it make sense to start by using the builtin function INDEX to create the relevant dictionary:
INDEX( $b[] | {id, status}; .id )
This assumes an invocation of jq along the lines of:
jq --argfile b b.json -f update.jq a.json
(Yes, I know --argfile has been deprecated. Feel free to choose another way to set $b to the contents of b.json.)
Now, to perform the update, it will be simplest to use the "update" operator, |=, in conjunction with map_values. (Feel free to check the jq manual :-)
Putting everything together:
INDEX( $b[] | {id, status}; .id ) as $dict
| .rs.mappings |= map_values( .status = $dict[.id].status )