Print result of timestamp function one line before string - json

At the bottom is some example json data which I'd like to add timestamps to. i.e "time": "1544785866.176123088" The timestamps need to be unique on each iteration and need to occur on the line above "bladesetName". Unfortunately, what I have so far is not working. After printing the time to the file in the needed places I was going to add 1ns to each entry. Is there a better way of doing this?
timestamp() {
date +%s.%N
}
awk "/bladesetName/{ print '"time" "$(timestamp)"' }1" volumes3.json
This throws an error
awk: /bladesetName/{ print 'time 1544786158.644385726' }1
awk: ^ invalid char ''' in expression
.
{
"pasxml": {
"#version": "6.0.0",
"system": {
"name": "example1",
"IPV4": "0.0.0.0",
"alertLevel": "critical",
"state": "online"
},
"volumes": {
"volume": [
{
"#id": "1",
"name": "/",
"bladesetName": {
"#id": "1",
"#text": "Set-1"
},
"state": "Online",
"raid": "Object RAID6+",
"director": "Shelf-001,1",
"volservice": "0x0400000000000004(FM)",
"objectId": "I-xD0200000000000004-xG7ee84b0c-xU00004a75726a656e",
"recoveryPriority": "1",
"efsaMode": "retry",
"spaceUsedGB": "0",
"spaceAvailableGB": "693903.38",
"hardQuotaGB": "0.52",
"softQuotaGB": "0.52",
"userQuotaPolicy": {
"#inherit": "0",
"#text": "disabled"
},
"stats": null
},
{
"#id": "8",
"name": "/datacentre/archvol/pan101",
"bladesetName": {
"#id": "1",
"#text": "Set-1"
},
"alertLevel": "critical",
"state": "Online",
"raid": "Object RAID6+",
"director": "Shelf-008,1",
"volservice": "0x04000000000000ec(FM)",
"objectId": "I-xD02000000000000ec-xG5c7aef6f-xU00004a75726a656e",
"recoveryPriority": "50",
"efsaMode": "retry",
"spaceUsedGB": "117000.09",
"spaceAvailableGB": "693903.38",
"hardQuotaGB": "117000.00",
"softQuotaGB": "90000.00",
"userQuotaPolicy": {
"#inherit": "1",
"#text": "disabled"
},
"stats": null
},

Here’s one approach, which uses jq:
def addnow:
. as $in
| to_entries
| (map(.key)|index("bladesetName")) as $i
| if $i
then .[0:$i]+[{key:"time",value:now}]+.[$i:] | from_entries
else $in
end ;
walk(if type=="object" then addnow else . end)
If you really want the time as a string, simply add tostring, as in (now|tostring)
If your jq does not have walk, then now would be a good time to upgrade; otherwise simply copy and paste its def, which can easily be found on the web, e.g. by googling: jq “def walk”

Related

How to filter an array of json with jq in linux?

I have the following JSON input:
{
"paging": {
"count": 0,
"total": 0,
"offset": 0,
"max": 0
},
"executions": [
{
"id": 5,
"href": "https://localhost.com.br",
"permalink": "https://localhost.com.br",
"status": "succeeded",
"project": "PROJETO",
"executionType": "scheduled",
"date-started": {
"unixtime": 1660793400012,
"date": "2022-08-18T03:30:00Z"
},
"date-ended": {
"unixtime": 1660793409694,
"date": "2022-08-18T03:30:09Z"
},
"job": {
"id": "cdkwednweoi-8745bjdf-kcjkjr8745",
"averageDuration": 0,
"name": "routine",
"group": "",
"project": "PROJECT",
"description": "",
"href": "https://localhost.com.br",
"permalink": "https://localhost.com.br"
},
"description": "runner",
"argstring": null,
"serverUUID": "jdnsdnasldnaje382nf5ubv",
"successfulNodes": [
"84jsk937nf"
]
}
]
}
First I want to select an array by a property name. And then I want to select an object of the array by the value of the propertyes.
Example of the desired informations on output:
"href"
"status"
"project"
"date-started":
"unixtime": 48298437239847,
"date": "2022-07-17"
"date-ended":
"unixtime": 48298437239847,
"date": "2022-07-17"
"job":
"name": "cleaner"
I knew how to get the firts values:
jq -r '.executions[] | [.href, .status, .project']
But the other ones I don't know how to do, I've tried with:
jq '.executions[] | with_entries( select(.value | has("date-started") ) )'
But it doesn't works.
Your first query produces a JSON array, so in this response, I'll assume it will suffice to produce an array of the eight values of interest in the order you've specified.
With your input, the following invocation produces the eight values as shown below:
jq '.executions[]
| [.href, .status, .project,
(."date-started" | (.unixtime, .date)),
(."date-ended" | (.unixtime, .date)),
.job.name]'
Output:
[
"https://localhost.com.br/rundeck/api/40/execution/2340",
"succeeded",
"PROJETO",
1660793400012,
"2022-08-18T03:30:00Z",
1660793409694,
"2022-08-18T03:30:09Z",
"proc_limpeza_saft"
]

Using jq to fetch and show key value with quotes

I have a file that looks as below:
{
"Job": {
"Name": "sample_job",
"Description": "",
"Role": "arn:aws:iam::00000000000:role/sample_role",
"CreatedOn": "2021-10-21T23:35:23.660000-03:00",
"LastModifiedOn": "2021-10-21T23:45:41.771000-03:00",
"ExecutionProperty": {
"MaxConcurrentRuns": 1
},
"Command": {
"Name": "glueetl",
"ScriptLocation": "s3://aws-sample-s3/scripts/sample.py",
"PythonVersion": "3"
},
"DefaultArguments": {
"--TempDir": "s3://aws-sample-s3/temporary/",
"--class": "GlueApp",
"--enable-continuous-cloudwatch-log": "true",
"--enable-glue-datacatalog": "true",
"--enable-metrics": "true",
"--enable-spark-ui": "true",
"--job-bookmark-option": "job-bookmark-enable",
"--job-insights-byo-rules": "",
"--job-language": "python",
"--spark-event-logs-path": "s3://aws-sample-s3/logs"
},
"MaxRetries": 0,
"AllocatedCapacity": 100,
"Timeout": 2880,
"MaxCapacity": 100.0,
"WorkerType": "G.1X",
"NumberOfWorkers": 100,
"GlueVersion": "2.0"
}
}
I want to get key/value from "Name", "--enable-continuous-cloudwatch-log": "" and "--enable-metrics": "". So, I need to show the info like this:
"Name" "sample_job"
"--enable-continuous-cloudwatch-log" ""
"--enable-metrics" ""
UPDATE
Follow the tips from #Inian and #0stone0 I came close to it:
jq -r '(.Job ) + (.Job.DefaultArguments | { "--enable-continuous-cloudwatch-log", "--enable-metrics"}) | to_entries[] | "\"\(.key)\" \"\(.value)\""'
This extract the values I need but show all another key/values.
Since you're JSON isn't valid, I've converted it into:
{
"Job": {
"Name": "sample_job",
"Role": "sample_role_job"
},
"DefaultArguments": {
"--enable-continuous-cloudwatch-log": "test_1",
"--enable-metrics": ""
},
"Timeout": 2880,
"NumberOfWorkers": 10
}
Using the following filter:
"Name \(.Job.Name)\n--enable-continuous-cloudwatch-log \(.DefaultArguments."--enable-continuous-cloudwatch-log")\n--enable-metrics \(.DefaultArguments."--enable-metrics")"
We use string interpolation to show the desired output:
Name sample_job
--enable-continuous-cloudwatch-log test_1
--enable-metrics
jq --raw-output '"Name \(.Job.Name)\n--enable-continuous-cloudwatch-log \(.DefaultArguments."--enable-continuous-cloudwatch-log")\n--enable-metrics \(.DefaultArguments."--enable-metrics")"'
Online Demo

group json output by array input value

Input data: /tmp/h1.tabs, tab delimited:
INPUT A1
Flavor Controller
Comment Disabled
State DiskOne
INPUT B2
Flavor Controller
Comment Not Applicable
State Not Applicable
ConnectorCount 12
Alarm Alarm Not present
INPUT C3
Flavor Controller
Comment Not Applicable
Media Not Applicable
ConnectorCount 0
State Alarm Not present
Desired Output:
{
"A1": {
"Comment": "Disabled",
"Flavor": "Controller",
"State": "DiskOne"
},
"B2": {
"Alarm": "Alarm Not present",
"Comment": "Not Applicable",
"ConnectorCount": "12",
"Flavor": "Controller",
"State": "Not Applicable"
},
"C3": {
"Comment": "Not Applicable",
"ConnectorCount": "0",
"Flavor": "Controller",
"Media": "Not Applicable",
"State": "Alarm Not present"
}
}
Each INPUT dictionary key could also be an array instead of another dictionary.
{
"A1": [
{ "Comment": "Disabled" },
{ "Flavor": "Controller" },
{ "State": "DiskOne" }
],
About as close as I am able to get is something like this:
jq -Rsn '[inputs|. / "\n"|.[] / "\t"|select(length > 0)|. as $input|(if $input[0] == "INPUT" then $input[1] else { ($input[0]): $input[1] } end)]' /tmp/h1.tabs
[
"A1",
{
"Flavor": "Controller"
},
{
"Comment": "Disabled"
},
{
"State": "DiskOne"
},
"B2",
{
"Flavor": "Controller"
},
I've tried expressions like if $input[0] == "INPUT" then $block = $input[1], but I am not having any luck with an assignment, so I can't use the assignment in the output. Really what I think I need is a variable set to whatever INPUTs value is every time I pass it. Then I can format the output as needed. I am just missing some key magic. I've been banging on this for a while, here is more that doesn't work...
# vim:ft=ansible:tabstop=8 expandtab shiftwidth=2 softtabstop=2
"unknown" as $block
|[
inputs
|. / "\n"
|
(
.[]
| select(length > 0)
|.
)
]
|(.[] / "\t")
|select(length > 0)
|. as $input
|
(
if $input[0] == "INPUT" then $block = $input[1] else empty end
|({($block): [($input[0]):($input[1])]})
) | add
Still learning :-)
reduce is your friend..
reduce (inputs / "\t") as [$k, $v] ([];
if $k == "INPUT" then
.[0] = $v
else
.[1][.[0]] += {($k): $v}
end
) | .[1]
Note that you need to specify -n and -R options on the command line for this to work
Using GNU awk, gawkextlib and gawk-json:
$ gawk '
#load "json"
BEGIN{
FS="\t"
}
{
if($1=="INPUT")
tl=$2
else
data[tl][$1]=$2
}
END {
print json_toJSON(data)
}' file # | jq '.' # for eye-friendly formating
Output (jq assisted):
{
"C3": {
"Comment": "Not Applicable",
"State": "Alarm Not present",
"ConnectorCount": "0",
"Flavor": "Controller",
"Media": "Not Applicable"
},
"A1": {
"Comment": "Disabled",
"State": "DiskOne",
"Flavor": "Controller"
},
"B2": {
"Comment": "Not Applicable",
"State": "Not Applicable",
"Alarm": "Alarm Not present",
"ConnectorCount": "12",
"Flavor": "Controller"
}
}

jq - Combine data from different keys

I have this json output
{
"FileSystems": [
{
"CreationToken": "CreationToken1",
"OwnerId": "OwnerId1",
"SizeInBytes": {
"Timestamp": 1552377599.0,
"Value": 1550721024
},
"Name": "Name1",
"NumberOfMountTargets": 3,
"FileSystemId": "fs-1",
"LifeCycleState": "available",
"CreationTime": 1550506468.0,
"PerformanceMode": "generalPurpose"
},
{
"CreationToken": "CreationToken2",
"OwnerId": "OwnerId2",
"SizeInBytes": {
"Timestamp": 1552377599.0,
"Value": 2390339584
},
"Name": "Name2",
"NumberOfMountTargets": 3,
"FileSystemId": "fs-2",
"LifeCycleState": "available",
"CreationTime": 1547663741.0,
"PerformanceMode": "generalPurpose"
}
]
}
By using this command
aws efs describe-file-systems | jq -r ".FileSystems[] | .SizeInBytes.Value"
I can recieve the NFS sized values:
1550725120
2390339584
But I need to get an output with the filesystem Name, so it should be like that:
Name1: 1550725120
Name2: 2390339584
How can I do it?
You can use string interpolation, like this:
jq -r ".FileSystems[] | \"\(.Name) \(.SizeInBytes.Value)\""

Adding Elements to AWS Route 53 JSON

I've been attempting to programmatically update the AWS Route 53 DNS records, so I've been using jq to update the following JSON file;
{
"Comment": "Update 'A' record for drivepoc.biz zone file",
"Changes": [
{
"Action": "UPSERT",
"ResourceRecordSet": {
"Name": "www.domain.biz.",
"Type": "A",
"TTL": 60,
"ResourceRecords": [
{
"Value": "123.123.123.123"
}
]
}
}
]
}
So, the existing entry "Value": "123.123.123.123" needs to remain, but needs to have additional entry of "Value": "456.456.456.456". The nearest I got to do this was:
cat a_record.json | jq '.Changes[0].ResourceRecordSet.ResourceRecords |= .+ ["Value: 456.456.456.456"]'
but this puts it outside the braces and the quotes are wrong;
"ResourceRecords": [
{
"Value": "52.18.219.57"
},
"Value": "456.456.456.456"
]
Instead of what is required;
"ResourceRecords": [
{
"Value": "52.18.219.57"
},
{
"Value": "456.456.456.456"
}
]
Can anyone give me any tips please?
You're adding an object to that array, not a string. Create an object to be inserted.
.Changes[].ResourceRecordSet.ResourceRecords += [{Value:"456.456.456.456"}]