I have a file that looks as below:
{
"Job": {
"Name": "sample_job",
"Description": "",
"Role": "arn:aws:iam::00000000000:role/sample_role",
"CreatedOn": "2021-10-21T23:35:23.660000-03:00",
"LastModifiedOn": "2021-10-21T23:45:41.771000-03:00",
"ExecutionProperty": {
"MaxConcurrentRuns": 1
},
"Command": {
"Name": "glueetl",
"ScriptLocation": "s3://aws-sample-s3/scripts/sample.py",
"PythonVersion": "3"
},
"DefaultArguments": {
"--TempDir": "s3://aws-sample-s3/temporary/",
"--class": "GlueApp",
"--enable-continuous-cloudwatch-log": "true",
"--enable-glue-datacatalog": "true",
"--enable-metrics": "true",
"--enable-spark-ui": "true",
"--job-bookmark-option": "job-bookmark-enable",
"--job-insights-byo-rules": "",
"--job-language": "python",
"--spark-event-logs-path": "s3://aws-sample-s3/logs"
},
"MaxRetries": 0,
"AllocatedCapacity": 100,
"Timeout": 2880,
"MaxCapacity": 100.0,
"WorkerType": "G.1X",
"NumberOfWorkers": 100,
"GlueVersion": "2.0"
}
}
I want to get key/value from "Name", "--enable-continuous-cloudwatch-log": "" and "--enable-metrics": "". So, I need to show the info like this:
"Name" "sample_job"
"--enable-continuous-cloudwatch-log" ""
"--enable-metrics" ""
UPDATE
Follow the tips from #Inian and #0stone0 I came close to it:
jq -r '(.Job ) + (.Job.DefaultArguments | { "--enable-continuous-cloudwatch-log", "--enable-metrics"}) | to_entries[] | "\"\(.key)\" \"\(.value)\""'
This extract the values I need but show all another key/values.
Since you're JSON isn't valid, I've converted it into:
{
"Job": {
"Name": "sample_job",
"Role": "sample_role_job"
},
"DefaultArguments": {
"--enable-continuous-cloudwatch-log": "test_1",
"--enable-metrics": ""
},
"Timeout": 2880,
"NumberOfWorkers": 10
}
Using the following filter:
"Name \(.Job.Name)\n--enable-continuous-cloudwatch-log \(.DefaultArguments."--enable-continuous-cloudwatch-log")\n--enable-metrics \(.DefaultArguments."--enable-metrics")"
We use string interpolation to show the desired output:
Name sample_job
--enable-continuous-cloudwatch-log test_1
--enable-metrics
jq --raw-output '"Name \(.Job.Name)\n--enable-continuous-cloudwatch-log \(.DefaultArguments."--enable-continuous-cloudwatch-log")\n--enable-metrics \(.DefaultArguments."--enable-metrics")"'
Online Demo
Related
I have the following JSON input:
{
"paging": {
"count": 0,
"total": 0,
"offset": 0,
"max": 0
},
"executions": [
{
"id": 5,
"href": "https://localhost.com.br",
"permalink": "https://localhost.com.br",
"status": "succeeded",
"project": "PROJETO",
"executionType": "scheduled",
"date-started": {
"unixtime": 1660793400012,
"date": "2022-08-18T03:30:00Z"
},
"date-ended": {
"unixtime": 1660793409694,
"date": "2022-08-18T03:30:09Z"
},
"job": {
"id": "cdkwednweoi-8745bjdf-kcjkjr8745",
"averageDuration": 0,
"name": "routine",
"group": "",
"project": "PROJECT",
"description": "",
"href": "https://localhost.com.br",
"permalink": "https://localhost.com.br"
},
"description": "runner",
"argstring": null,
"serverUUID": "jdnsdnasldnaje382nf5ubv",
"successfulNodes": [
"84jsk937nf"
]
}
]
}
First I want to select an array by a property name. And then I want to select an object of the array by the value of the propertyes.
Example of the desired informations on output:
"href"
"status"
"project"
"date-started":
"unixtime": 48298437239847,
"date": "2022-07-17"
"date-ended":
"unixtime": 48298437239847,
"date": "2022-07-17"
"job":
"name": "cleaner"
I knew how to get the firts values:
jq -r '.executions[] | [.href, .status, .project']
But the other ones I don't know how to do, I've tried with:
jq '.executions[] | with_entries( select(.value | has("date-started") ) )'
But it doesn't works.
Your first query produces a JSON array, so in this response, I'll assume it will suffice to produce an array of the eight values of interest in the order you've specified.
With your input, the following invocation produces the eight values as shown below:
jq '.executions[]
| [.href, .status, .project,
(."date-started" | (.unixtime, .date)),
(."date-ended" | (.unixtime, .date)),
.job.name]'
Output:
[
"https://localhost.com.br/rundeck/api/40/execution/2340",
"succeeded",
"PROJETO",
1660793400012,
"2022-08-18T03:30:00Z",
1660793409694,
"2022-08-18T03:30:09Z",
"proc_limpeza_saft"
]
I'm pasting here a JSON example data which would require some manipulation to get a desired output which is mentioned in the next section to be read after this piece of JSON code.
I want to use jq for parsing my desired data.
{
"MetricAlarms": [
{
"EvaluationPeriods": 3,
"ComparisonOperator": "GreaterThanOrEqualToThreshold",
"AlarmActions": [
"Unimportant:Random:alarm:ELK2[10.1.1.2]-Root-Disk-Alert"
],
"AlarmName": "Unimportant:Random:alarm:ELK1[10.1.1.0]-Root-Alert",
"Dimensions": [
{
"Name": "path",
"Value": "/"
},
{
"Name": "InstanceType",
"Value": "m5.2xlarge"
},
{
"Name": "fstype",
"Value": "ext4"
}
],
"DatapointsToAlarm": 3,
"MetricName": "disk_used_percent"
},
{
"EvaluationPeriods": 3,
"ComparisonOperator": "GreaterThanOrEqualToThreshold",
"AlarmActions": [
"Unimportant:Random:alarm:ELK2[10.1.1.2]"
],
"AlarmName": "Unimportant:Random:alarm:ELK2[10.1.1.2]",
"Dimensions": [
{
"Name": "path",
"Value": "/"
},
{
"Name": "InstanceType",
"Value": "r5.2xlarge"
},
{
"Name": "fstype",
"Value": "ext4"
}
],
"DatapointsToAlarm": 3,
"MetricName": "disk_used_percent"
}
]
}
So when I Pass some Key1 & value1 as a parameter "Name": "InstanceType", to the JQ probably using cat | jq and output expected should be as below
m5.2xlarge
r5.2xlarge
A generic approach to search for a key-value pair (sk-sv) in input recursively and extract another key's value (pv) from objects found:
jq -r --arg sk Name \
--arg sv InstanceType \
--arg pv Value \
'.. | objects | select(contains({($sk): $sv})) | .[$pv]' file
I have two JSON files a.json and b.json. The contents in a.json file is a JSON object and inside b.json its an array.I want to add/update status field in each mappings in a.json by retrieving the value from b.json file.
a.json:
{
"title": 25886,
"data": {
"request": {
"c": 46369,
"t1": 1562050127.376641
},
},
"rs": {
"mappings": {
"12345": {
"id": "12345",
"name": "test",
"customer_id": "11228",
},
"45678": {
"id": "45678",
"name": "abc",
"customer_id": "11206",
}
}
}}
b.json:
[
{
"status": "pending",
"extra": {
"name": "test"
},
"enabled": true,
"id": "12345"
},
{
"status": "not_started",
"extra": {
"name": "abc"
},
"enabled": true,
"id": "45678"
}
]
Below is my expected output:
{
"title": 25886,
"data": {
"request": {
"c": 46369,
"t1": 1562050127.376641
},
},
"rs": {
"mappings": {
"12345": {
"id": "12345",
"name": "test",
"customer_id": "11228",
"status":"pending"
},
"45678": {
"id": "45678",
"name": "abc",
"customer_id": "11206",
"status":"not_started"
}
}
}}
In this expected JSON file we have status field whose value is retrieved from b.json file based on a matching id value. How to do this using jq ?
For the purposes of this problem, b.json essentially defines a dictionary, so for simplicity, efficiency and perhaps elegance,
it make sense to start by using the builtin function INDEX to create the relevant dictionary:
INDEX( $b[] | {id, status}; .id )
This assumes an invocation of jq along the lines of:
jq --argfile b b.json -f update.jq a.json
(Yes, I know --argfile has been deprecated. Feel free to choose another way to set $b to the contents of b.json.)
Now, to perform the update, it will be simplest to use the "update" operator, |=, in conjunction with map_values. (Feel free to check the jq manual :-)
Putting everything together:
INDEX( $b[] | {id, status}; .id ) as $dict
| .rs.mappings |= map_values( .status = $dict[.id].status )
I have read properties with jq from a json object and have stored them to variables.
I want to now read these variables and essentially find and replace a word inside the string with a global shell variable.
I've set my json ID's from my JSON file
# Set Json ID's
TARGET_ID=$(jq '.DefaultCacheBehavior.TargetOriginId' distconfig.json)
DOMAIN_NAME=$(jq '.Origins.Items[0].DomainName' distconfig.json)
ORIGIN_ID=$(jq '.Origins.Items[0].Id' distconfig.json)
echo "$TARGET_ID"
echo "$DOMAIN_NAME"
echo "$ORIGIN_ID"
This returns
"S3-Website-stag4.example.io.s3-website.us-east-2.amazonaws.com"
"stag4.example.io.s3-website.us-east-2.amazonaws.com"
"S3-Website-stag4.example.io.s3-website.us-east-2.amazonaws.com"
I have my location id variable and would like to write it to find and replace all stag4 references in those 3 ID's.
Then I would like to write those 3 ID's to the initial json object, or create a temp version of it.
Example, if:
$DOMAIN_NAME is"stag4.example.io.s3-website.us-east-2.amazonaws.com"
I would like to essentially have it set to:
$LOCATION_NAME="stag6"
DOMAIN_LOCATION="example.io"
"$DOMAIN_NAME=S3-Website-\$LOCATION_NAME\.example.io.s3-website.us-east-2.amazonaws.com"
"$TARGET_ID=\$LOCATION_NAME\.example.io.s3-website.us-east-2.amazonaws.com"
"$ORIGIN_ID=S3-Website-\$LOCATION_NAME\.example.io.s3-website.us-east-2.amazonaws.com"
Then write those 3 to the temp or new json file so I can run my cloudformation command:
aws cloudfront create-distribution --distribution-config file://disttemp.json
I have now built out the proper variables from the initial json file like so:
$LOCATION_NAME="stag6"
DOMAIN_LOCATION="example.io"
echo "Build New IDs"
TARGET_ID_BUILT="S3-Website-$LOCATION_NAME.$DOMAIN_LOCATION.s3-website.us-east-2.amazonaws.com"
DOMAIN_NAME_BUILT="$LOCATION_NAME.$DOMAIN_LOCATION.s3-website.us-east-2.amazonaws.com"
ORIGIN_ID_BUILT="S3-Website-$LOCATION_NAME.$DOMAIN_LOCATION.s3-website.us-east-2.amazonaws.com"
echo "$TARGET_ID_BUILT"
echo "$DOMAIN_NAME_BUILT"
echo "$ORIGIN_ID_BUILT"
How do I write these variables to the json file with jq?
EDIT: Sample of distconfig.json requested – domain/creds swapped to example
{
"CallerReference": "my-test-distribution-2",
"Comment": "",
"CacheBehaviors": {
"Quantity": 0
},
"IsIPV6Enabled": true,
"Logging": {
"Bucket": "",
"Prefix": "",
"Enabled": false,
"IncludeCookies": false
},
"WebACLId": "",
"Origins": {
"Items": [
{
"OriginPath": "",
"CustomOriginConfig": {
"OriginSslProtocols": {
"Items": [
"TLSv1",
"TLSv1.1",
"TLSv1.2"
],
"Quantity": 3
},
"OriginProtocolPolicy": "http-only",
"OriginReadTimeout": 30,
"HTTPPort": 80,
"HTTPSPort": 443,
"OriginKeepaliveTimeout": 5
},
"CustomHeaders": {
"Quantity": 0
},
"Id": "S3-Website-stag4.example.io.s3-website.us-east-2.amazonaws.com",
"DomainName": "stag4.example.io.s3-website.us-east-2.amazonaws.com"
}
],
"Quantity": 1
},
}
"DefaultRootObject": "",
"PriceClass": "PriceClass_All",
"Enabled": true,
"DefaultCacheBehavior": {
"TrustedSigners": {
"Enabled": false,
"Quantity": 0
},
"LambdaFunctionAssociations": {
"Quantity": 0
},
"TargetOriginId": "S3-Website-stag4.example.io.s3-website.us-east-2.amazonaws.com",
"ViewerProtocolPolicy": "redirect-to-https",
"ForwardedValues": {
"Headers": {
"Quantity": 0
},
"Cookies": {
"Forward": "none"
},
"QueryStringCacheKeys": {
"Quantity": 0
},
"QueryString": false
},
"MaxTTL": 31536000,
"SmoothStreaming": false,
"DefaultTTL": 86400,
"AllowedMethods": {
"Items": [
"HEAD",
"GET"
],
"CachedMethods": {
"Items": [
"HEAD",
"GET"
],
"Quantity": 2
},
"Quantity": 2
},
"MinTTL": 0,
"Compress": true
},
"ViewerCertificate": {
"SSLSupportMethod": "sni-only",
"ACMCertificateArn": "xxxx",
"MinimumProtocolVersion": "TLSv1.1_2016",
"Certificate": "xxxx",
"CertificateSource": "acm"
},
"CustomErrorResponses": {
"Quantity": 0
},
"HttpVersion": "http2",
"Restrictions": {
"GeoRestriction": {
"RestrictionType": "none",
"Quantity": 0
}
},
"Aliases": {
"Quantity": 0
}
}
You should use sed to do the substitution and then inject the value back into the JSON.
echo $TARGET_ID | sed 's/stag4/stag5/g'
Outputs
S3-Website-stag5.example.io.s3-website.us-east-2.amazonaws.com
Next we'll put the value back into the original JSON, this will technically output a new JSON and does not edit the file, however, you can easily solve for this on the output by temporarily saving to a tmp file.
We will use the --arg flag to reference our bash variable and set the new value for our field
cat distconfig.json | jq --arg TARGET_ID $TARGET_ID '.DefaultCacheBehavior.TargetOriginId = $TARGET_ID' > tmp.json && mv tmp.json distconfig.json
I have this json output
{
"FileSystems": [
{
"CreationToken": "CreationToken1",
"OwnerId": "OwnerId1",
"SizeInBytes": {
"Timestamp": 1552377599.0,
"Value": 1550721024
},
"Name": "Name1",
"NumberOfMountTargets": 3,
"FileSystemId": "fs-1",
"LifeCycleState": "available",
"CreationTime": 1550506468.0,
"PerformanceMode": "generalPurpose"
},
{
"CreationToken": "CreationToken2",
"OwnerId": "OwnerId2",
"SizeInBytes": {
"Timestamp": 1552377599.0,
"Value": 2390339584
},
"Name": "Name2",
"NumberOfMountTargets": 3,
"FileSystemId": "fs-2",
"LifeCycleState": "available",
"CreationTime": 1547663741.0,
"PerformanceMode": "generalPurpose"
}
]
}
By using this command
aws efs describe-file-systems | jq -r ".FileSystems[] | .SizeInBytes.Value"
I can recieve the NFS sized values:
1550725120
2390339584
But I need to get an output with the filesystem Name, so it should be like that:
Name1: 1550725120
Name2: 2390339584
How can I do it?
You can use string interpolation, like this:
jq -r ".FileSystems[] | \"\(.Name) \(.SizeInBytes.Value)\""