how to pass variables to jq - json

I have a problem with passing variables into jq. Here is example file:
[
{
"version": 24,
"file": "branding/24"
}
]
and here is my script:
jq --arg value "25" '. += [{"version": $value|tonumber, "file": "branding/$value|tonumber"}]' versions.json >tmp.json && mv tmp.json versions.json
The result is:
[
{
"version": 24,
"file": "branding/24"
},
{
"version": 25,
"file": "branding/$value|tonumber"
}
]
Expected result:
[
{
"version": 24,
"file": "branding/24"
},
{
"version": 25,
"file": "branding/25"
}
]


You need to use interpolation to have the filter evaluated.
{"version": $value|tonumber, "file": "branding/\($value|tonumber)"}

Related

Merge multiple objects under a single object and add sibling object in one command

So I have three files:
cats.json
{
"cats": [
{
"name": "fluffles",
"age": 10,
"color": "white"
}
]
}
dogs.json
{
"dogs": [
{
"name": "sam",
"age": 5,
"color": "black and white"
},
{
"name": "rover",
"age": 2,
"color": "brown and white"
}
]
}
snakes.json
{
"snakes": [
{
"name": "noodles",
"age": 10,
"color": "green"
}
]
}
I wanted to merge these together, under an "animals" object. I've found that will merge the files:
jq -s '{"animals": .} ' cats.json dogs.json snakes.json > animals.json
{
"animals": [
{
"cats": [
{
"name": "fluffles",
"age": 10,
"color": "white"
}
]
},
{
"dogs": [
{
"name": "sam",
"age": 5,
"color": "black and white"
},
{
"name": "rover",
"age": 2,
"color": "brown and white"
}
]
},
{
"snakes": [
{
"name": "noodles",
"age": 10,
"color": "green"
}
]
}
]
}
Now I have an additional object:
owners.json
{
"owners": [
"peter",
"william",
"sally"
]
}
which I want to merge into the same file using
jq -s '.[0] + .[1]' animals.json owners.json
Can I do both of these operations with just one jq command?
jq -s '{"animals": .} ' cats.json dogs.json snakes.json > animals.json
jq -s '.[0] + .[1]' animals.json owners.json
The result would look like this:
{
"animals": [
{
"cats": [
{
"name": "fluffles",
"age": 10,
"color": "white"
}
]
},
{
"dogs": [
{
"name": "sam",
"age": 5,
"color": "black and white"
},
{
"name": "rover",
"age": 2,
"color": "brown and white"
}
]
},
{
"snakes": [
{
"name": "noodles",
"age": 10,
"color": "green"
}
]
}
],
"owners": [
"peter",
"william",
"sally"
]
}

Suppose you had an (a priori) indeterminate or large numbers of types of animals, and just one owners file. In such cases, it would be better (to save memory) not to use the -s option, and it would be easier to invoke jq with the owners file as the first data file, e.g. along the lines of:
jq -n -f program.jq owners.json $(ls *.json | grep -v owners.json)
where program.jq contains a program such as:
input as $owners | {$owners, animals: [inputs]}
(Notice how {"owners": $owners} can be abbreviated.)

Not sure if this is the way-to-go, but it gets the desired output by:
Using --slurp:
Catching the first 3 files as a single array variable
[ .[0] * .[1] * .[2] ] as $all
Catching owners object as a single variable
.[3].owners as $owners
Creating the object as desired
{ "animals": $all, "owners": $owners }
jq \
--slurp \
'[ .[0] * .[1] * .[2] ] as $all | .[3].owners as $owners | { "animals": $all, "owners": $owners }' cats.json dogs.json snakes.json owners.json
Will produce:
{
"animals": [
{
"cats": [
{
"name": "fluffles",
"age": 10,
"color": "white"
}
],
"dogs": [
{
"name": "sam",
"age": 5,
"color": "black and white"
},
{
"name": "rover",
"age": 2,
"color": "brown and white"
}
],
"snakes": [
{
"name": "noodles",
"age": 10,
"color": "green"
}
]
}
],
"owners": [
"peter",
"william",
"sally"
]
}

How to get a value from json based on key in bash

I'm trying to extract value of a branch from the below json/test.json file using jq
{
"pipeline": {
"name": "test",
"roleArn": "arn:aws:iam::1234:role/service-role/AWSCodePipelineServiceRole-us-west-2-test",
"artifactStore": {
"type": "S3"
},
"stages": [{
"name": "Source",
"actions": [{
"name": "Source",
"actionTypeId": {
"category": "Source",
"version": "1"
},
"runOrder": 1,
"configuration": {
"Branch": "experiment"
}
}]
}],
"version": 1
}
}
Below is jq command I'm using jq -r '.pipeline.stages.actions.configuration.Branch' test.jsonwhich returns jq: error (at test.json:76): Cannot index array with string "actions". I'm I missing something here

When you're doing queries, and something doesn't work at first, then try something simple and keep adding to it. So in your case, start with
$ jq -r '.pipelines' test.json
null
Aha, it's "pipeline" not "pipelines", so go from there:
$ jq -r '.pipeline' test.json
{
"name": "test",
"roleArn": "arn:aws:iam::1234:role/service-role/AWSCodePipelineServiceRole-us-west-2-test",
"artifactStore": {
"type": "S3"
},
"stages": [
{
"name": "Source",
"actions": [
{
"name": "Source",
"actionTypeId": {
"category": "Source",
"version": "1"
},
"runOrder": 1,
"configuration": {
"Branch": "experiment"
}
}
]
}
],
"version": 1
}
So that works. Now we want to get to "stages" so we do this
$ jq -r '.pipeline.stages' test.json
[
{
"name": "Source",
"actions": [
{
"name": "Source",
"actionTypeId": {
"category": "Source",
"version": "1"
},
"runOrder": 1,
"configuration": {
"Branch": "experiment"
}
}
]
}
]
Note that "stages" gives you an array, not just a hash, so you have to refer to [0], the zeroth element.
$ jq -r '.pipeline.stages[0]' test.json
{
"name": "Source",
"actions": [
{
"name": "Source",
"actionTypeId": {
"category": "Source",
"version": "1"
},
"runOrder": 1,
"configuration": {
"Branch": "experiment"
}
}
]
}
Now you can get to "actions"
$ jq -r '.pipeline.stages[0].actions' test.json
[
{
"name": "Source",
"actionTypeId": {
"category": "Source",
"version": "1"
},
"runOrder": 1,
"configuration": {
"Branch": "experiment"
}
}
]
and then the zeroth one
$ jq -r '.pipeline.stages[0].actions[0]' test.json
{
"name": "Source",
"actionTypeId": {
"category": "Source",
"version": "1"
},
"runOrder": 1,
"configuration": {
"Branch": "experiment"
}
}
and finally to configuration and Branch
$ jq -r '.pipeline.stages[0].actions[0].configuration.Branch' test.json
experiment

Your json key address should be .pipeline.stages[0].actions[0].configuration.Branch so that would make your command look like this:
jq -r '.pipeline.stages[0].actions[0].configuration.Branch' test.json
That's because both stages and actions are arrays with both only a single item in them.

storing json output in bash from cloudfromation

I am using aws ecs query to get list of properties being used by the current running task.
command -
cft = "aws ecs describe-tasks --cluster arn:aws:ecs:us-west-2:4984314772:cluster/secrets --tasks arn:aws:ecs:us-west-2:4984314772:task/secrets/86855757eec4487f9d4475a1f7c4cb0b
I am storing this in an output variable
output= $( eval $cft)
Output:
"tasks": [
{
"attachments": [
{
"id": "da8a1312-8278-46d5-8e3b-6b6a1d96f820",
"type": "ElasticNetworkInterface",
"status": "ATTACHED",
"details": [
{
"name": "subnetId",
"value": "subnet-0a151f2eb959ad4"
},
{
"name": "networkInterfaceId",
"value": "eni-081948e3666253f"
},
{
"name": "macAddress",
"value": "02:2a:9i:5c:4a:77"
},
{
"name": "privateDnsName",
"value": "ip-172-56-17-177.us-west-2.compute.internal"
},
{
"name": "privateIPv4Address",
"value": "172.56.17.177"
}
]
}
],
"availabilityZone": "us-west-2a",
"clusterArn": "arn:aws:ecs:us-west-2:4984314772:cluster/secrets",
"containers": [
{
"taskArn": "arn:aws:ecs:us-west-2:4984314772:task/secrets/86855757eec4487f9d4475a1f7c4cb0b",
"name": "nginx",
"image": "nginx",
"lastStatus": "PENDING",
"networkInterfaces": [
{
"attachmentId": "da8a1312-8278-46d5-6b6a1d96f820",
"privateIpv4Address": "172.31.17.176"
}
],
"healthStatus": "UNKNOWN",
"cpu": "0"
}
],
"cpu": "256",
"createdAt": "2020-12-10T18:00:16.320000+05:30",
"desiredStatus": "RUNNING",
"group": "family:nginx",
"healthStatus": "UNKNOWN",
"lastStatus": "PENDING",
"launchType": "FARGATE",
"memory": "512",
"overrides": {
"containerOverrides": [
{
"name": "nginx"
}
],
"inferenceAcceleratorOverrides": []
},
"platformVersion": "1.4.0",
"tags": [],
"taskArn": "arn:aws:ecs:us-west-2:4984314772:task/secrets/86855757eec4487f9d4475a1f7c4cb0b",
"taskDefinitionArn": "arn:aws:ecs:us-west-2:4984314772:task-definition/nginx:17",
"version": 2
}
],
"failures": []
}
now if do an echo of $output.tasks[0].containers[0] nothing happens it prints the entire thing again, i want to store the result in output variable and refer different parameter like we do in json format.

You will need to use a json parser such as jq and so:
eval $cft | jq '.tasks[].containers[]'
To avoid using eval you could simple pipe the aws command into jq and so:
aws ecs describe-tasks --cluster arn:aws:ecs:us-west-2:4984314772:cluster/secrets --tasks arn:aws:ecs:us-west-2:4984314772:task/secrets/86855757eec4487f9d4475a1f7c4cb0b | jq '.tasks[].containers[]'
or:
cft=$(aws ecs describe-tasks --cluster arn:aws:ecs:us-west-2:4984314772:cluster/secrets --tasks arn:aws:ecs:us-west-2:4984314772:task/secrets/86855757eec4487f9d4475a1f7c4cb0b | jq '.tasks[].containers[]')
echo $cft | jq '.tasks[].containers[]'

How to filter missing inner key by using jq

Got a json input like this:
[
{
"dimensions": "helloworld",
"metrics": "sum(is_error)",
"values": {
"timestamp": 1558322460000,
"value": "0.0"
}
},
{
"dimensions": "helloworld",
"metrics": "sum(is_error)",
"values": {
"timestamp": 1558322160000,
"value": "0.0"
}
},
{
"dimensions": "helloworld",
"metrics": "sum(is_error)",
"values": "3423.25"
}
]
The third object doesnot have a timestamp on it. How could I return all the object only have a timestamp on it. Like the following:
[
{
"dimensions": "helloworld",
"metrics": "sum(is_error)",
"values": {
"timestamp": 1558322460000,
"value": "0.0"
}
},
{
"dimensions": "helloworld",
"metrics": "sum(is_error)",
"values": {
"timestamp": 1558322160000,
"value": "0.0"
}
}
]
Many thanks in advance.
Cheers,
Vincent

map( select ( .values | has("timestamp")? ))

and here's an alternative solution, using a walk-path unix tool for JSON: jtc:
bash $ <file.json jtc -w'<timestamp>l:[-2]' -j
[
{
"dimensions": "helloworld",
"metrics": "sum(is_error)",
"values": {
"timestamp": 1558322460000,
"value": "0.0"
}
},
{
"dimensions": "helloworld",
"metrics": "sum(is_error)",
"values": {
"timestamp": 1558322160000,
"value": "0.0"
}
}
]
bash $
it finds each (all) label timestamp, then goes 2 levels up from the found json entry and prints found Json element. -j wraps all printed walks back into array.
PS> Disclosure: I'm the creator of the jtc tool

Working example:
[ .[] | select (.values | has("timestamp")?) ]
https://jqplay.org/s/n5jsRsPMhW
Or alternative:
[ .[] | select (.values.timestamp?) ]
https://jqplay.org/s/HRWV44YgUp
P.S. This one was incorrect because of after .[] you are working with each item separately, not with array. So 'map' function is unnecessary.

Nest array into new object with JQ

I know this one should be easy but it has me stumped. I am looking to take the following json example:
[{
"Name": "Test1",
"Version": "5.0.1",
"source": "source"
},
{
"Name": "Test2",
"Version": "2.0.11",
"source": "source"
},
{
"Name": "Test3",
"Version": "2.1.2",
"source": "source"
}]
and convert it to:
{
"packages": [
{
"Name": "Test1",
"Version": "5.0.1",
"source": "source"
},
{
"Name": "Test2",
"Version": "2.0.11",
"source": "source"
},
{
"Name": "Test3",
"Version": "2.1.2",
"source": "source"
}
]
}
I've tried numerous different ways, the closest I got is using something similar to: jq '.packages += [input]'
Basically it's just moving the original JSON to be nested. Any help would be appreciated.

just do this
jq '{ packages : . }' input.json

Even things you think of as literals are really filters in jq. The filter you need in this case is simply {packages: .}:
$ echo '[{}, {}]' | jq '{packages: .}'
{
"packages": [
{},
{}
]
}