#!/bin/bash
DESCRIBE_VPC=$(aws ec2 describe-vpcs --region us-west-2)
The Json value retrieved from aws ec2 describe-vpcs --region us-west-2 stores in DESCRIBE_VPC which comes to the output format below.
> echo $DESCRIBE_VPC
{
"Vpcs": [
{
"VpcId": "vpc-12345678910",
"InstanceTenancy": "default",
"Tags": [
{
"Value": "arn:aws:cloudformation:us-west-2:12345678910:stack/vpc/0123456-vpcid",
"Key": "stack-id"
},
{
"Value": "vpc-type",
"Key": "Name"
},
],
"CidrBlockAssociationSet": [
{
"AssociationId": "vpc-cidr-123456",
"CidrBlock": "123.456.89.10",
"CidrBlockState": {
"State": "associated"
}
}
],
"State": "available",
"DhcpOptionsId": "dpt-01234567",
"OwnerId": "12345678910",
"CidrBlock": "123.456.789.10",
"IsDefault": false
}
]
}
[root#ip bin]# jq '.Vpcs' $DESCRIBE_VPC
jq: error: Could not open file {: No such file or directory
jq: error: Could not open file "Vpcs":: No such file or directory
parse error: Invalid numeric literal at line 1, column 57
Any suggestions here how to parse the json values stored in variable?
Actually you have to use the (json) content of a shell variable in place of a command's (jq) input. This input should be a file or a stream, but you have it in a shell variable. There are many ways to do this, this is a simple one:
echo "$DESCRIBE_VPC" | jq '.Vpcs'
or
printf "%s" "$DESCRIBE_VPC" | jq '.Vpcs'
or this (for bash shell):
jq '.Vpcs' <<< "$DESCRIBE_VPC"
Also jq can accept variables and json variables. For example you could do this:
jq -n --argjson x "$DESCRIBE_VPC" '$x.Vpcs'
But the first one is usually better.
Related
i have a details.json file with a lot of entries and a shops.txt file like below. I like to have a little script which compares two values and just return the matching json entries.
[
{
"userName": "Anne",
"email": "anne#stack.com",
"company": {
"name": "Stack GmbH",
},
"details": {
"key": "EFHJKI-KJEFT-DHMNEB",
"prod": "Car",
},
"store": {
"id": "05611a7f-a679-12ad-a3u2-0745e3650a03",
"storeName": "shop-a57ca0a3-120c-1a73-153b-fa4231cab768",
}
},
{
"userName": "Tom",
"email": "tom#stack.com",
"company": {
"name": "Stack GmbH",
},
"details": {
"key": "DFSGSE-FGEAR-GWRTGW",
"prod": "Bike",
},
"store": null
},
]
This is the other file "shops.txt" (can be a lot more of shops inside)
shop-a57ca0a3-120c-1a73-153b-fa4231cab768
The script is looping through the shops, for every shop it loops through the json and should compare the currentShop with the store.shop from json and then echo the user and the shop.
But I can not access the specific parameters inside the json. How can I do this?
#!/bin/bash
shops="shops.txt"
while IFS= read -r line
do
currentShop="$line"
jq -c '.[.userName, .store.storeName]' details.json | while read i; do
if [[ $i.store.storeName == *$currentShop* ]]; then
echo $i.userName
echo $currentShop
fi
done
done < "$shops"
First of all, you might want to 'clean' your json, remove any trailing ,'s etc.
After looping through each line in the file we just need one select() to get the matching object.
The script could look something like:
#!/bin/bash
while read shop; do
echo "Check: $shop"
jq -r --arg storeName "$shop" '.[] | select(.store.storeName == "\($storeName)") | "\(.userName) - \(.store.storeName)"' details.json
done < "shops.txt"
Which will produce
Check: shop-a57ca0a3-120c-1a73-153b-fa4231cab768
Anne - shop-a57ca0a3-120c-1a73-153b-fa4231cab768
I guess this could be combined into a single jq call, but it seems like you want to loop over each entry found
You can test this jq selector on this online JqPlay Demo.
I was able to access the values with the following command:
echo $i | jq -r '.userName'
I'm new to jq, I have the following JSON & I need to extract FOUR values i.e. SAUCE_KEY, sauceKey, SAUCE_VALUE, sauceValue etc. And I need to covert these bash variables as i.e.
SAUCE_KEY=sauceKey
SAUCE_VALUE=sauceValue
If I will echo it, it should print it's value ie. echo $SAUCE_KEY
I have used the code as:
jq -r '.env_vars[] | with_entries(select([.key] | inside([".env_vars[].name", ".env_vars[].value"])))' | jq -r "to_entries|map(\"\(.value)=\(.value|tostring)\")|.[]"
By doing so, i was able to get values as
name=SAUCE_KEY, value=sauceKey and so on.
{
"#type": "env_vars",
"env_vars": [
{
"#type": "env_var",
"#href": "/repo/xxxxx/env_var/xxxxx",
"#representation": "standard",
"#permissions": {
"read": true,
"write": true
},
"name": "SAUCE_KEY",
"value": "sauceKey"
},
{
"#type": "env_var",
"#href": "/repo/xxxxx/env_var/xxxxx",
"#representation": "standard",
"#permissions": {
"read": true,
"write": true
},
"name": "SAUCE_VALUE",
"value": "sauceValue"
}
]
}
If instead of trying to extract variable names from the JSON, you populate an associate array with the names as keys, it's pretty straightforward.
#!/usr/bin/env bash
declare -A sauces
while IFS=$'\001' read -r name value; do
sauces[$name]=$value
done < <(jq -r '.env_vars[] | "\(.name)\u0001\(.value)"' saucy.json)
for name in "${!sauces[#]}"; do
echo "$name is ${sauces[$name]}"
done
prints out
SAUCE_VALUE is sauceValue
SAUCE_KEY is sauceKey
Using jq how can I convert an array into object indexed by filename, or read multiple files into one object indexed by their filename?
e.g.
jq -s 'map(select(.roles[]? | contains ("mysql")))' -C dir/file1.json dir/file2.json
This gives me the data I want, but I need to know which file they came from.
So instead of
[
{ "roles": ["mysql"] },
{ "roles": ["mysql", "php"] }
]
for output, I want:
{
"file1": { "roles": ["mysql"] },
"file2": { "roles": ["mysql", "php"] }
}
I do want the ".json" file extension stripped too if possible, and just the basename (dir excluded).
Example
file1.json
{ "roles": ["mysql"] }
file2.json
{ "roles": ["mysql", "php"] }
file3.json
{ }
My real files obviously have other stuff in them too, but that should be enough for this example. file3 is simply to demonstrate "roles" is sometimes missing.
In other words: I'm trying to find files that contain "mysql" in their list of "roles". I need the filename and contents combined into one JSON object.
To simplify the problem further:
jq 'input_filename' f1 f2
Gives me all the filenames like I want, but I don't know how to combine them into one object or array.
Whereas,
jq -s 'map(input_filename)' f1 f2
Gives me the same filename repeated once for each file. e.g. [ "f1", "f1" ] instead of [ "f1", "f2" ]
If your jq has inputs (as does jq 1.5) then the task can be accomplished with just one invocation of jq.
Also, it might be more efficient to use any than iterating over all the elements of .roles.
The trick is to invoke jq with the -n option, e.g.
jq -n '
[inputs
| select(.roles and any(.roles[]; contains("mysql")))
| {(input_filename | gsub(".*/|\\.json$";"")): .}]
| add' file*.json
jq approach:
jq 'if (.roles[] | contains("mysql")) then {(input_filename | gsub(".*/|\\.json$";"")): .}
else empty end' ./file1.json ./file2.json | jq -s 'add'
The expected output:
{
"file1": {
"roles": [
"mysql"
]
},
"file2": {
"roles": [
"mysql",
"php"
]
}
}
My json looks like this :
{
"20160522201409-jobsv1-1": {
"vmStateDisplayName": "Ready",
"servers": {
"20160522201409 jobs_v1 1": {
"serverStateDisplayName": "Ready",
"creationDate": "2016-05-22T20:14:22.000+0000",
"state": "READY",
"provisionStatus": "PENDING",
"serverRole": "ROLE",
"serverType": "SERVER",
"serverName": "20160522201409 jobs_v1 1",
"serverId": 2902
}
},
"isAdminNode": true,
"creationDate": "2016-05-22T20:14:23.000+0000",
"totalStorage": 15360,
"shapeId": "ot1",
"state": "READY",
"vmId": 4353,
"hostName": "20160522201409-jobsv1-1",
"label": "20160522201409 jobs_v1 ADMIN_SERVER 1",
"ipAddress": "10.252.159.39",
"publicIpAddress": "10.252.159.39",
"usageType": "ADMIN_SERVER",
"role": "ADMIN_SERVER",
"componentType": "jobs_v1"
}
}
My key keeps changing from time to time. So for example 20160522201409-jobsv1-1 may be something else tomorrow. Also I may more than one such entry in the json payload.
I want to echo $KEYS and I am trying to do it using jq.
Things I have tried :
| jq .KEYS is the command i use frequently.
Is there a jq command to display all the primary keys in the json?
I only care about the hostname field. And I would like to extract that out. I know how to do it using grep but it is NOT a clean approach.
You can simply use: keys:
% jq 'keys' my.json
[
"20160522201409-jobsv1-1"
]
And to get the first:
% jq -r 'keys[0]' my.json
20160522201409-jobsv1-1
-r is for raw output:
--raw-output / -r: With this option, if the filter’s result is a string then it will be written directly to standard output rather than being formatted as a JSON string with quotes. This can be useful for making jq filters talk to non-JSON-based systems.
Source
If you want a known value below an unknown property, eg xxx.hostName:
% jq -r '.[].hostName' my.json
20160522201409-jobsv1-1
Could somebody help me to deal with jq command line utility to update JSON object's inner value?
I want to alter object interpreterSettings.2B263G4Z1.properties by adding several key-values, like "spark.executor.instances": "16".
So far I only managed to fully replace this object, not add new properties with command:
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties |= { \"spark.executor.instances\": \"16\" }"
This is input JSON:
{
"interpreterSettings": {
"2B263G4Z1": {
"id": "2B263G4Z1",
"name": "sh",
"group": "sh",
"properties": {}
},
"2B188AQ5T": {
"id": "2B188AQ5T",
"name": "spark",
"group": "spark",
"properties": {
"spark.cores.max": "",
"spark.yarn.jar": "",
"master": "yarn-client",
"zeppelin.spark.maxResult": "1000",
"zeppelin.dep.localrepo": "local-repo",
"spark.app.name": "Zeppelin",
"spark.executor.memory": "2560M",
"zeppelin.spark.useHiveContext": "true",
"spark.home": "/usr/lib/spark",
"zeppelin.spark.concurrentSQL": "false",
"args": "",
"zeppelin.pyspark.python": "python"
}
}
},
"interpreterBindings": {
"2AXUMXYK4": [
"2B188AQ5T",
"2AY8SDMRU"
]
}
}
I also tried the following but this only prints contents of interpreterSettings.2B263G4Z1.properties, not full object.
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties + { \"spark.executor.instances\": \"16\" }"
The following works using jq 1.4 or jq 1.5 with a Mac/Linux shell:
jq '.interpreterSettings."2B188AQ5T".properties."spark.executor.instances" = "16" ' test.json
If you have trouble adapting the above for Windows, I'd suggest putting the jq program in a file, say my.jq, and invoking it like so:
jq -f my.jq test.json
Notice that there is no need to use "cat" in this case.
p.s. You were on the right track - try replacing |= with +=