passing json array with spaces to bash script for az cli tagging - json

I have a bash script that looks like the following and correctly passes tags to the az cli that includes spaces.
RESOURCE_GROUP_NAME=$1
LOCATION=$2
TAGS_INPUT_ARGUMENT=$3 # This TAGS_INPUT_ARGUMENT needs to finally look like the TAGS below.
echo 'TAGS_INPUT_ARGUMENT:' $TAGS_INPUT_ARGUMENT
# HARD CODED TAGS that Work
TAGS=("owner=Firstname Lastname" "application=cool-name")
echo 'TAGS:' "${TAGS[#]}"
az group create \
--name $RESOURCE_GROUP_NAME \
--location $LOCATION \
--tags "${TAGS[#]}"
I am having problems passing the TAGS into the script.
i.e.
export TAGS='["owner=Firstname Lastname","application=cool-name"]'
bash ./entrypoint.sh rg-lionking eastus2 "${TAGS}"
Output:
TAGS_INPUT_ARGUMENT: ["owner=Firstname Lastname","application=cool-name"]
TAGS: owner=Firstname Lastname application=cool-name
{
"id": "/subscriptions/**REDACTED**/resourceGroups/rg-lionking",
"location": "eastus2",
"managedBy": null,
"name": "rg-lionking",
"properties": {
"provisioningState": "Succeeded"
},
"tags": {
"application": "cool-name",
"owner": "Firstname Lastname"
},
"type": "Microsoft.Resources/resourceGroups"
}
Given that I can get TAGS in a good state, what do I need to do to pass in the same thing in json format and get it working as the final argument to the az group create cli?
My jp knowledge is practically zero, so I am guessing that the answer lies in there somewhere.

The equivalent to expanding an array is on the command line is passing multiple separate arguments
./entrypoint rg-lionking eastus2 "owner=Firstname Lastname" "application=cool-name"
...and then, in entrypoint:
#!/usr/bin/env bash
resource_group_name=$1; shift
location=$1; shift
tags=( "$#" )
az group create \
--name "$resource_group_name" \
--location "$location" \
--tags "${tags[#]}"

Related

Non variable recognition on Syspass API Bash Script

When i input a variable on the curl with the json indexed, it takes the string value of the variable, if i scaped the double-quotes, it returns me a sintax error because of in a json request u have to input the data with double-quotes.
Example:
#!/bin/bash
token="x"
tokenPass="x"
name="prueba"
url="https://prueba.prueba.prueba"
user="prueba"
pass="prueba"
notes="prueba"
curl -k -H "Content-Type:Application/json" -d '{"jsonrpc": "2.0", "method": "account/create", "params": { "authToken": "$token", "tokenPass": "$tokenPass", "name": "$name" , "categoryId": "7", "clientId": "9","login":"$user","url": "$url" ,"pass": "$pass","notes": "$notes"}, "id": 1}' -o- https://syspass.prueba.es/api.php
The curl json request works if i input the data manually, but with the variables, when i create the account, the account is named as the string value of the variable, i mean, if the variable is named $name , the account created is name $name. Any help please?
Also i tried to input the variable: "${variable}" and either works
Try this:
curl -k -H "Content-Type:Application/json" -d '{"jsonrpc": "2.0", "method": "account/create", "params": { "authToken": "'${token}'", "tokenPass": "'${tokenPass}'", "name": "'${name}'" , "categoryId": "7", "clientId": "9","login":"'${user}'","url": "'${url}'" ,"pass": "'${pass}'","notes": "'${notes}'"}, "id": 1}' -o- https://syspass.prueba.es/api.php
A bit of explanation:
Text that is between single quotes ' will not be interpreted by bash and that is why the variables will not be inserted. You have to break the string up...
An other suggestion is that you should use curly braces around variables: When do we need curly braces around shell variables?
I explain it here for the variable token. The other variables can be treated in the same way.
Assuming that the variable token contains the string foo bar baz, then you want curl to receive as its 5th argument the following string:
{"jsonrpc": "2.0", "method": "account/create", "params": { "authToken": "foo bar baz", "tokenPass":... rest omitted for brevity}
Since this has to become a single argument, you have to quote the hole string somehow, and because this string contains many double-quotes, it is easier to use single quotes for wrapping. However, this would prevent parameter expansion. To make this happen, the variables must be outside the single quotes. However, they must also be inside double quotes, lest you become a victim of word splitting, if the variables contain spaces. This yields the following solution:
curl .... '{"jsonrpc": "2.0", "method": "account/create", "params": { "authToken": "'"$token"'", "tokenPass": ... rest omitted for brevity}'
Note the "'" in front of $token: The first double quote is the one which will be part of the expanded "foo bar baz". It must be literal, because curl wants to see it. The next single quote terminates the initial single quote. The next double quote tells the shell to expand $token, but without attempting word splitting.
If there are many parameters to expand, the resulting expression is difficult to debug and maintain. An alternative is to set up a HERE-document:
# Set up Parameter 5 for curl
read p5 <<END
{"jsonrpc": "2.0", "method": "account/create", "params": { "authToken": "$token", "tokenPass":... rest omitted for brevity}
END
and then use it as
curl .... -d "$p5"
This requires the separated step of setting up the argument, but in the end is much more readable, and you don't have to worry about word splitting either.
I'd use jq to generate the JSON string (mostly, to handle the quoting for you, JSON strings are not trivial), and for readability (and for the sanity of the next maintainer) add some newlines and indentation:
data=$(
jq -n \
--arg token "$token" \
--arg tokenPass "$tokenPass" \
--arg name "$name" \
--arg url "$url" \
--arg user "$user" \
--arg pass "$pass" \
--arg notes "$notes" \
'{
jsonrpc: "2.0",
method: "account/create",
params: {
authToken: $token,
tokenPass: $tokenPass,
name: $name,
categoryId: "7",
clientId: "9",
login: $user,
url: $url ,
pass: $pass,
notes: $notes
},
id: 1
}'
)
curl -k \
-H "Content-Type:Application/json" \
-d "$data" \
-o- \
https://syspass.prueba.es/api.php

how to add an object to existing json file using jq

I have an empty output.json and I want to populate it with {key, value} pairs where key is a string and value is a Json array read from file. I need to go through this for multiple files to populate the output.json. So far, value is being populated successfuly.
$ jq --argjson cves "$(cat my-scan-result-N.json)" '.+={"TODO": $cves}' output.json
{
"TODO": [
{
"cvePK": "CVE-2020-11656",
"summary": "In SQLite through 3.31.1, the ALTER TABLE implementation has a use-after-free, as demonstrated by an ORDER BY clause that belongs to a compound SELECT statement.",
"cvss": 7.5,
"notes": ""
},
{
"cvePK": "CVE-2019-19646",
"summary": "pragma.c in SQLite through 3.30.1 mishandles NOT NULL in an integrity_check PRAGMA command in certain cases of generated columns.",
"cvss": 7.5,
"notes": ""
}
]
}
However, when I add another --argjson to populate the key ("TODO") with desired value $FQDN, it fails with an error.
$ FQIN="example.com/foo/bar:7.0.3" # Tried \""example.com/foo/bar:7.0.3"\" as well but doesn't work.
$ jq --argjson cves "$(cat my-scan-result.json)" --argjson fqin="FQIN" '.+={$fqin: $cves}' output.json
C:\ProgramData\chocolatey\lib\jq\tools\jq.exe: invalid JSON text passed to --argjson
Use C:\ProgramData\chocolatey\lib\jq\tools\jq.exe --help for help with command-line options,
or see the jq manpage, or online docs at https://stedolan.github.io/jq
So my goal is to have something like below, but above error message is not helpful enough. Any help would be appreciated.
{
"example.com/foo/bar:7.0.3": [
{
"cvePK": "CVE-2020-11656",
"summary": "In SQLite through 3.31.1, the ALTER TABLE implementation has a use-after-free, as demonstrated by an ORDER BY clause that belongs to a compound SELECT statement.",
"cvss": 7.5,
"notes": ""
},
{
"cvePK": "CVE-2019-19646",
"summary": "pragma.c in SQLite through 3.30.1 mishandles NOT NULL in an integrity_check PRAGMA command in certain cases of generated columns.",
"cvss": 7.5,
"notes": ""
}
]
}
The line:
jq --argjson cves "$(cat my-scan-result.json)" --argjson fqin="FQIN" '.+={$fqin: $cves}' output.json
has several errors:
The phrase --argjson fqin="FQIN" is incorrect. Please see the jq manual for details. Suffice it to say here that you could achieve the desired effect by writing --arg fqin "$FQIN".
The jq expression {$fqin: $cves} is incorrect. When a key name is specified using a variable, the variable must be enclosed in parentheses: {($fqin): $cves}. (Indeed, whenever the key name is specified indirectly, the specifying expression must be enclosed in parentheses.)
You can do
FQIN="example.com/foo/bar:7.0." jq -n --argjson cves "$(cat my-scan-result.json)" '.+={(env.FQIN): $cves}'

execute in bash command stored in variable [duplicate]

This question already has answers here:
Bash script: Use string variable in curl JSON Post data
(3 answers)
Closed 2 years ago.
I'm trying to script a mailing using a curl api (this is the base API, in mine the html part is changed with "xmessage":
curl -s \
-X POST \
--user "$MJ_APIKEY_PUBLIC:$MJ_APIKEY_PRIVATE" \
https://api.mailjet.com/v3.1/send \
-H 'Content-Type: application/json' \
-d '{
"Messages":[
{
"From": {
"Email": "pilot#mailjet.com",
"Name": "Mailjet Pilot"
},
"To": [
{
"Email": "passenger1#mailjet.com",
"Name": "passenger 1"
}
],
"Subject": "Your email flight plan!",
"TextPart": "Dear passenger 1, welcome to Mailjet! May the delivery force be with you!",
"HTMLPart": "<h3>Dear passenger 1, welcome to Mailjet!</h3><br />May the delivery force be with you!",
"CustomCampaign": "SendAPI_campaign",
"DeduplicateCampaign": true
}
]
}'
My script look like this :
...
message=$(cat ./message.txt)
message=${message//"xdate"/$courseDate}
message=${message//"xcoursecode"/$courseCode}
message=${message//"xsubtitle"/$subtitle}
message=${message//"\r"/""}
message=${message//"\r\n"/""}
message=${message//"\n"/""}
message=${message//"\""/"\\\""}
message=${message//"'"/"&apos;"}
mailJet=$(cat ./mailjet.txt) # containing my API as described as above
mailJet=${mailJet//"xmessage"/$message}
echo $mailJet
eval $mailJet
the command "eval $mailJet" does not works but if I do a copy paste in the terminal of the "echo $mailJet" output my command works
The eval $mailJet give the following error :
{"ErrorIdentifier":"5cce36c5-373c-48ca-90b8-2b6bfc5df526","ErrorCode":"mj-0031","StatusCode":400,"ErrorMessage":"Request payload contains not valid UTF-8 encoded characters"}
Something that partially worked, was to put directly the mailJet.txt content in the script
but I'm struggling to find the syntax to replace the xmessage by what's in $message.
Like this it did not worked :
...
message=$(cat ./message.txt)
message=${message//"xdate"/$courseDate}
message=${message//"xcoursecode"/$courseCode}
message=${message//"xsubtitle"/$subtitle}
message=${message//"\r"/""}
message=${message//"\r\n"/""}
message=${message//"\n"/""}
message=${message//"\""/"\\\""}
message=${message//"'"/"&apos;"}
curl -s \
-X POST \
--user "$MJ_APIKEY_PUBLIC:$MJ_APIKEY_PRIVATE" \
https://api.mailjet.com/v3.1/send \
-H 'Content-Type: application/json' \
-d '{
"Messages":[
{
"From": {
"Email": "pilot#mailjet.com",
"Name": "Mailjet Pilot"
},
"To": [
{
"Email": "passenger1#mailjet.com",
"Name": "passenger 1"
}
],
"Subject": "Your email flight plan!",
"TextPart": "Dear passenger 1, welcome to Mailjet! May the delivery force be with you!",
"HTMLPart": "$message", ## neither like this : "HTMLPart": "'$message'",
"CustomCampaign": "SendAPI_campaign",
"DeduplicateCampaign": true
}
]
}'
Whereas if I put any html stuff instead of $message in the curl api, the script run without any issue.
I'm stuck (and not a great bash coder or even coder at all)
Many thanks by advance for your help.
I think your problems start at
-d '{
This use of a single quote means that nothing in this section is interpreted by bash. "$message" later on will be treated as the text $ and message.
If this is the issue, then what you need to do is unquote around the variable names, but also double-quote arround them, so you write '"$message"' Or, if you need the double-quotes to appear inside the curl command "'"$message"'".
Note you can't have the ## comment either, but I assume you put that in for our benefit.

Filter on nested array with jmespath (using az cli)

I would like to know if a value in a TXT record is already available in the DNS:
az network dns record-set txt show -g myresourcegroup -z 'mydomain' -n 'mytxtvalues'
This is the part of the json result where it's all about:
"txtRecords": [
{
"value": [
"abc"
]
},
{
"value": [
"def"
]
}
]
I tried many queries. These are 2 which I expected to work.
az network dns record-set txt show -g myresourcegroup -z 'mydomain.com' -n 'mytxtvalues' --query txtRecords[?value[?starts_with(#, 'abc')]]
az network dns record-set txt show -g myresourcegroup -z 'mydomain.com' -n 'mytxtvalues' --query txtRecords[*].value[?starts_with(#, 'abc')]]
The result is:
At line:1 char:123
+ ... 'mytxtvalues' --query txtRecords[?value[?starts_with(#, 'abc') ...
+ ~ Unrecognized token in source text. At line:1 char:124
+ ... 'mytxtvalues' --query txtRecords[?value[?starts_with(#, 'abc')] ...
+ ~ Missing argument in parameter list.
It looks like the # used to filter the array is not recognized. But I don't know how to query otherwise.
What would be a correct query to know if value abc is already in the list?
You need to use the command like this:
az network dns record-set txt show -g myresourcegroup -z 'mydomain.com' -n 'mytxtvalues' --query "txtRecords[*].value[?starts_with(#, 'abc')]"
And if you just need to output the string you can append the parameter -o tsv. Hope this will help you.

Convert JSON from AWS SSM to environment variables using jq

I have done some research on this and feel as if I'm about 80% there but struggling to adjust the jq output as required due to splitting one of the strings.
I'm trying to convert the JSON output from AWS SSM to environment variables.
AWS command
aws ssm get-parameters-by-path \
--path /qa/es \
--with-decryption \
--query 'Parameters[*].{Name:Name,Value:Value}' \
Output
[
{
"Name": "/qa/es/AWS_ACCESS_KEY_ID",
"Value": "ABC123"
},
{
"Name": "/qa/es/AWS_SECRET_ACCESS_KEY",
"Value": "abcdefghijkl"
},
{
"Name": "/qa/es/ENDPOINT",
"Value": "https://amazonaws.com"
}
]
My required output from jq, note I'm only after the environment variable AFTER the last /. There may be cases where this could be /qa/es/something/nested/ENV_VAR
AWS_ACCESS_KEY_ID=ABC123
AWS_SECRET_ACCESS_KEY=abcdefghijkl
ENDPOINT=https://amazonaws.com
Once I have this I can utilise the answer here to set the environment variables. Exporting JSON to environment variables
The closest I have got is
jq -r "map(\"\(try(.Name |= split(\"/\")))=\(.Value|tostring)\")|.[]" params.json
Which gives me
{"Name":["","qa","es","AWS_ACCESS_KEY_ID"],"Value":"ABC123"}=ABC123
{"Name":["","qa","es","AWS_SECRET_ACCESS_KEY"],"Value":"abcdefghijkl"}=abcdefghijkl
{"Name":["","qa","es","ENDPOINT"],"Value":"https://amazonaws.com"}=https://amazonaws.com
Close, but not close enough! Can anyone point me in the right direction here?
With the -r command-line option,
.[]
| "\(.Name|split("/")|.[-1])=\(.Value)"
yields:
AWS_ACCESS_KEY_ID=ABC123
AWS_SECRET_ACCESS_KEY=abcdefghijkl
ENDPOINT=https://amazonaws.com
This seems to correspond to what you've asked for, but this approach has the potential disadvantage that it assumes something about "=", so please be careful!
As the previous comment but use '#sh' to escape the value
| "\(.Name|split("/")|.[-1])=\(.Value | #sh)"