I have done some research on this and feel as if I'm about 80% there but struggling to adjust the jq output as required due to splitting one of the strings.
I'm trying to convert the JSON output from AWS SSM to environment variables.
AWS command
aws ssm get-parameters-by-path \
--path /qa/es \
--with-decryption \
--query 'Parameters[*].{Name:Name,Value:Value}' \
Output
[
{
"Name": "/qa/es/AWS_ACCESS_KEY_ID",
"Value": "ABC123"
},
{
"Name": "/qa/es/AWS_SECRET_ACCESS_KEY",
"Value": "abcdefghijkl"
},
{
"Name": "/qa/es/ENDPOINT",
"Value": "https://amazonaws.com"
}
]
My required output from jq, note I'm only after the environment variable AFTER the last /. There may be cases where this could be /qa/es/something/nested/ENV_VAR
AWS_ACCESS_KEY_ID=ABC123
AWS_SECRET_ACCESS_KEY=abcdefghijkl
ENDPOINT=https://amazonaws.com
Once I have this I can utilise the answer here to set the environment variables. Exporting JSON to environment variables
The closest I have got is
jq -r "map(\"\(try(.Name |= split(\"/\")))=\(.Value|tostring)\")|.[]" params.json
Which gives me
{"Name":["","qa","es","AWS_ACCESS_KEY_ID"],"Value":"ABC123"}=ABC123
{"Name":["","qa","es","AWS_SECRET_ACCESS_KEY"],"Value":"abcdefghijkl"}=abcdefghijkl
{"Name":["","qa","es","ENDPOINT"],"Value":"https://amazonaws.com"}=https://amazonaws.com
Close, but not close enough! Can anyone point me in the right direction here?
With the -r command-line option,
.[]
| "\(.Name|split("/")|.[-1])=\(.Value)"
yields:
AWS_ACCESS_KEY_ID=ABC123
AWS_SECRET_ACCESS_KEY=abcdefghijkl
ENDPOINT=https://amazonaws.com
This seems to correspond to what you've asked for, but this approach has the potential disadvantage that it assumes something about "=", so please be careful!
As the previous comment but use '#sh' to escape the value
| "\(.Name|split("/")|.[-1])=\(.Value | #sh)"
Related
Using jq, is it possible to replace the value of each parameter in the sample JSON with the value of the variable that is the initial value?
In my scenario, Azure DevOps does not carryout any kind of variable substitution on the JSON file, so I need to do it manually. So for example, say $SUBSCRIPTION_ID is set to abc-123, I'd like to use jq to update the JSON file.
I can pull out the values using .parameters[].value, but I can't seem to find a way of setting each individual value.
The main challenge here is that the solution should be reusable, and different JSON files will have different parameters, so I don't think I can use --argjson.
Example
Original JSON
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/parametersTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"subscriptionId": {
"value": "$SUBSCRIPTION_ID"
},
"topicName": {
"value": "$TOPIC_NAME"
}
}
}
Variables
SUBSCRIPTION_ID="abc-123"
TOPIC_NAME="SomeTopic"
Desired JSON
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/parametersTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"subscriptionId": {
"value": "abc-123"
},
"topicName": {
"value": "SomeTopic"
}
}
}
Export those variables so that you can access them from within jq.
export SUBSCRIPTION_ID TOPIC_NAME
jq '.parameters[].value |= (env[.[1:]] // .)' file
//. part is for leaving variables absent in the environment as is, you can drop it if not necessary
Use --argjson; essentially, you are just going to ignore the attempt at parameterizing the JSON and simply replace the values unconditionally.
jq --argjson x "$SUBSCRIPTION_ID" \
--argjson y "$TOPIC_NAME" \
'.parameters.subscriptionId.value = $x; .parameters.topicName.value = $y' \
config.json
Here is a "data-driven" approach based on the contents of the schema and the available environment variables:
export SUBSCRIPTION_ID="abc-123"
export TOPIC_NAME="SomeTopic"
< schema.json jq '.parameters
|= map_values(if .value | (startswith("$") and env[.[1:]])
then .value |= env[.[1:]] else . end)'
Notice that none of the template names appear in the jq program.
If your shell supports it, you could avoid the "export" commands by prefacing the jq command with the variable assignments along the lines of:
SUBSCRIPTION_ID="abc-123" TOPIC_NAME="SomeTopic" jq -f program.jq schema.json
Caveat
Using environment variables to pass in the parameter values may not be such a great idea. Two alternatives would be to provide the name-value pairs in a text file or as a JSON object. See also Using jq as a template engine
I have a JSON file very similar to the following:
[
{
"uuid": "832390ed-58ed-4338-bf97-eb42f123d9f3",
"name": "Nacho"
},
{
"uuid": "5b55ea5e-96f4-48d3-a258-75e152d8236a",
"name": "Taco"
},
{
"uuid": "a68f5249-828c-4265-9317-fc902b0d65b9",
"name": "Burrito"
}
]
I am trying to figure out how to use the JQ command line processor to first find the UUID that I input and based on that output the name of the associated item. So for example, if I input UUID a68f5249-828c-4265-9317-fc902b0d65b9 it should search the JSON file, find the matching UUID and then return the name Burrito. I am doing this in Bash. I realize it may require some outside logic in addition to JQ. I will keep thinking about it and put an update here in a bit. I know I could do it in an overly complicated way, but I know there is probably a really simple JQ method of doing this in one or two lines. Please help me.
https://shapeshed.com/jq-json/#how-to-find-a-key-and-value
You can use select:
jq -r --arg query Burrito '.[] | select( .name == $query ) | .uuid ' tst.json
I'm trying to retrieve the last value inserted into a table in influxdb. What I need to do is then post it to another system via HTTP.
I'd like to do all this in a bash script, but I'm open to Python also.
$ curl -sG 'https://influx.server:8086/query' --data-urlencode "db=iotaWatt" --data-urlencode "q=SELECT LAST(\"value\") FROM \"grid\" ORDER BY time DESC" | jq -r
{
"results": [
{
"statement_id": 0,
"series": [
{
"name": "grid",
"columns": [
"time",
"last"
],
"values": [
[
"2018-01-17T04:15:30Z",
690.1
]
]
}
]
}
]
}
What I'm struggling with is getting this value into a clean format I can use. I don't really want to use sed, and I've tried jq but it complains the data is a string and not an index:
jq: error (at <stdin>:1): Cannot index array with string "series"
Anyone have a good suggestion?
Pipe that curl to the jq below
$ your_curl_stuff_here | jq '.results[].series[]|.name,.values[0][]'
"grid"
"2018-01-17T04:15:30Z"
690.1
The results could be stored into a bash array and used later.
$ results=( $(your_curl_stuff_here | jq '.results[].series[]|.name,.values[0][]') )
$ echo "${results[#]}"
"grid" "2018-01-17T04:15:30Z" 690.1
# Individual values could be accessed using "${results[0]}" and so, mind quotes
All good :-)
Given the JSON shown, the jq query:
.results[].series[].values[]
produces:
[
"2018-01-17T04:15:30Z",
690.1
]
This seems to be the output you want, but from the point of view of someone who is not familiar with influxdb, the requirements seem very opaque, so you might want to consider a variant, such as:
.results[-1].series[-1].values[-1]
which in this case produces the same result, as it happens.
If you just want the atomic values, you could simply append [] to either of the queries above.
I'm writing a Bash function to get a portion of a JSON object. The API for the function is:
GetSubobject()
{
local Filter="$1" # Filter is of the form .<key>.<key> ... .<key>
local File="$2" # File is the JSON to get the subobject
# Code to get subobject using jq
# ...
}
To illustrate what I mean by a subobject, consider the Bash function call:
GetSubobject .b.x.y example.json
where the file example.json contains:
{
"a": { "p": 1, "q": 2 },
"b":
{
"x":
{
"y": { "j": true, "k": [1,2,3] },
"z": [4,5,6]
}
}
}
The result from the function call would be emitted to stdout:
{
"y": {
"j": true,
"k": [
1,
2,
3
]
}
}
Note that the code jq -r "$Filter" "$File" would not give the desired answer. It would give:
{ "j": true, "k": [1,2,3] }
Please note that the answer I'm looking for needs to be something I can use in the Bash function API above. So, the answer should use the Filter and File variables as show above and not be specific to the example above.
I have come up with a solution; however, it relies on Bash to do part of the job. I am hoping that the solution can be pure jq without reliance on Bash processing.
#!/bin/bash
GetSubobject()
{
local Filter="$1"
local File="$2"
# General case: separate:
# .<key1>.<key2> ... .<keyN-1>.<keyN>
# into:
# Prefix=.<key1>.<key2> ... .<keyN-1>
# Suffix=<keyN>
local Suffix="${Filter##*.}"
local Prefix="${Filter%.$Suffix}"
# Edge case: where Filter = .<key>
# Set:
# Prefix=.
# Suffix=<key>
if [[ -z $Prefix ]]; then
Prefix='.'
Suffix="${Filter#.}"
fi
jq -r "$Prefix|to_entries|map(select(.key==\"$Suffix\"))|from_entries" "$File"
}
GetSubobject "$#"
How would I complete the above Bash function using jq to obtain the desired result, hopefully in a less brute-force way that takes advantage of jq's capabilities without having to do pre-processing in Bash?
Somewhat further simplifying the jq part but with the same general constraints as JawguyChooser's answer, how about the much simpler Bash function
GetSubject () {
local newroot=${1##*.}
jq -r "{$newroot: $1}" "$2"
}
I may be overlooking some nuances of your more-complex Bash processing, but this seems to work for the example you provided.
If I understand what you're trying to do correctly, it doesn't seem possible to me to do it "pure jq" having read the docs (and being a regular jq user myself). The closest I could come to helping here was to simplify the jq part itself:
jq -r "$Prefix| { $Suffix }" "$File"
This has the same behavior as your example (on this limited set of cases):
GetSubobject '.b.x.y' example.json
{
"y": {
"j": true,
"k": [
1,
2,
3
]
}
}
This is really a case of metaprogramming, you want to programmatically operate on a jq program. Well, it makes sense (to me) that jq takes its program as input but doesn't allow you to alter the program itself. bash seems like an appropriate choice for doing the metaprogramming here: to convert a jq program into another one and then run jq using that.
If the goal is to do as little as possible in bash, then maybe the following bash function will fill the bill:
function GetSubobject {
local Filter="$1" # Filter is of the form .<key>.<key> ... .<key>
local File="$2" # File is the JSON to get the subobject
jq '(null|path('"$Filter"')) as $path
| {($path[-1]): '"$Filter"'}' "$File"
}
An alternative would be to pass $Filter in as a string (e.g. --arg filter "$Filter"), have jq do the parsing, and then use getpath.
It would of course be simplest if GetSubobject could be called with the path separated from the field of interest, like this:
GetSubobject .b.x y filename
This is seemingly a very basic question. I am new to JSON, so I am prepared for the incoming facepalm.
I have the following JSON file (test-app-1.json):
"application.name": "test-app-1",
"environments": {
"development": [
"server1"
],
"stage": [
"server2",
"server3"
],
"production": [
"server4",
"server5"
]
}
The intent is to use this as a configuration file and reference for input validation.
I am using bash 3.2 and will be using jq 1.4 (not the latest) to read the JSON.
The problem:
I need to return all values in the specified JSON array based on an argument.
Example: (how the documentation and other resources show that it should work)
APPLICATION_ENVIRONMENT="developement"
jq --arg appenv "$APPLICATION_ENVIRONMENT" '.environments."$env[]"' test-app-1.json
If executed, this returns null. This should return server1.
Obviously, if I specify text explicitly matching the JSON arrays under environment, it works just fine:
jq 'environments.development[]' test-app-1.json returns: server1.
Limitations: I am stuck to jq 1.4 for this project. I have tried the same actions in 1.5 on a different machine with the same null results.
What am I doing wrong here?
jq documentation: https://stedolan.github.io/jq/manual/
You have three issues - two typos and one jq filter usage issue:
set APPLICATION_ENVIRONMENT to development instead of developement
use variable name consistently: if you define appenv, use $appenv, not $env
address with .environments[$appenv]
When fixed, looks like this:
$ APPLICATION_ENVIRONMENT="development"
$ jq --arg appenv "$APPLICATION_ENVIRONMENT" '.environments[$appenv][]' test-app-1.json
"server1"