Parsing variables in curl with bash script - json

Hey I am using conduit curl method to create tasks from post. It work fine when I run from terminal with hardcoded values. But when I try to execute it with variables it throws an error:
Script:
#!/bin/bash
echo "$1"
echo "$2"
echo "$3"
echo "$4"
echo "$5"
echo '{
"transactions": [
{
"type": "title",
"value": "$1"
},
{
"type": "description",
"value": "$2"
},
{
"type": "status",
"value": "$3"
},
{
"type": "priority",
"value": "$4"
},
{
"type": "owner",
"value": "$5"
}
]
}' | arc call-conduit --conduit-uri https://mydomain.phacility.com/ --conduit-token mytoken maniphest.edit
execution:
./test.sh "test003 ticket from api post" "for testing" "open" "high" "ahsan"
Output:
test003 ticket from api post
for testing
open
high
ahsan
{"error":"ERR-CONDUIT-CORE","errorMessage":"ERR-CONDUIT-CORE: Validation errors:\n - User \"$5\" is not a valid user.\n - Task priority \"$4\" is not a valid task priority. Use a priority keyword to choose a task priority: unbreak, very, high, kinda, triage, normal, low, wish.","response":null}
As you can see in error its reading $4 and $5 as values not variables. And I am failing to understand how to use $variables as input in these arguments.

You're using single quotes around the last echo to so that you can use double-quotes inside the JSON, but that causes echo to print the string without expanding anything. You need to use double quotes for the string, so you'll have to escape the double quotes inside of it.
Replace the last echo with this:
echo "{
\"transactions\": [
{
\"type\": \"title\",
\"value\": \"$1\"
},
{
\"type\": \"description\",
\"value\": \"$2\"
},
{
\"type\": \"status\",
\"value\": \"$3\"
},
{
\"type\": \"priority\",
\"value\": \"$4\"
},
{
\"type\": \"owner\",
\"value\": \"$5\"
}
]
}"
and it'll work. To avoid issues like this you can check http://wiki.bash-hackers.org and http://mywiki.wooledge.org/BashGuide for some general tips for bash newbies. Also, you can use shellcheck with a lot of text editors, which will spot errors like this automatically.

Related

Bash. How to access parameters while looping through json with jq?

i have a details.json file with a lot of entries and a shops.txt file like below. I like to have a little script which compares two values and just return the matching json entries.
[
{
"userName": "Anne",
"email": "anne#stack.com",
"company": {
"name": "Stack GmbH",
},
"details": {
"key": "EFHJKI-KJEFT-DHMNEB",
"prod": "Car",
},
"store": {
"id": "05611a7f-a679-12ad-a3u2-0745e3650a03",
"storeName": "shop-a57ca0a3-120c-1a73-153b-fa4231cab768",
}
},
{
"userName": "Tom",
"email": "tom#stack.com",
"company": {
"name": "Stack GmbH",
},
"details": {
"key": "DFSGSE-FGEAR-GWRTGW",
"prod": "Bike",
},
"store": null
},
]
This is the other file "shops.txt" (can be a lot more of shops inside)
shop-a57ca0a3-120c-1a73-153b-fa4231cab768
The script is looping through the shops, for every shop it loops through the json and should compare the currentShop with the store.shop from json and then echo the user and the shop.
But I can not access the specific parameters inside the json. How can I do this?
#!/bin/bash
shops="shops.txt"
while IFS= read -r line
do
currentShop="$line"
jq -c '.[.userName, .store.storeName]' details.json | while read i; do
if [[ $i.store.storeName == *$currentShop* ]]; then
echo $i.userName
echo $currentShop
fi
done
done < "$shops"
First of all, you might want to 'clean' your json, remove any trailing ,'s etc.
After looping through each line in the file we just need one select() to get the matching object.
The script could look something like:
#!/bin/bash
while read shop; do
echo "Check: $shop"
jq -r --arg storeName "$shop" '.[] | select(.store.storeName == "\($storeName)") | "\(.userName) - \(.store.storeName)"' details.json
done < "shops.txt"
Which will produce
Check: shop-a57ca0a3-120c-1a73-153b-fa4231cab768
Anne - shop-a57ca0a3-120c-1a73-153b-fa4231cab768
I guess this could be combined into a single jq call, but it seems like you want to loop over each entry found
You can test this jq selector on this online JqPlay Demo.
I was able to access the values with the following command:
echo $i | jq -r '.userName'

Use newline with jq

I've seen a number of posts on this but can't figure out what I need exactly. I've tried -r and argjson among other things.
I need the newlines to remain as \n and not be escaped to \\n.
I'd also like to be able to use ``` for code blocks but it ignores that section of the string.
FALLBACK_MESSAGE="TEST MESSAGE - $HOSTNAME"
MARKDOWN_MESSAGE="TEST MESSAGE - $HOSTNAME \(0x0a) \(\n) Hi <#U12345789>\n```Can we do a\nmultiline code block```"
JSON_STRING=$( jq -nr \
--arg fallbackMessage "$FALLBACK_MESSAGE" \
--arg sectionType "section" \
--arg markdownType "mrkdwn" \
--arg textMessage "$MARKDOWN_MESSAGE" \
'{
text: $fallbackMessage,
blocks: [
{
type: $sectionType,
text: {
type: $markdownType,
text: $textMessage
}
}
]
}')
echo $JSON_STRING
Outputs:
{ "text": "TEST MESSAGE - devDebug", "blocks": [ { "type": "section", "text": { "type": "mrkdwn", "text": "TEST MESSAGE - devDebug \\(0x0a) \\(\\n) Hi <#U12345789>\\n" } } ] }
Make sure your shell variables contain actual newlines, not \n sequences.
If you want bash to convert escape sequences in a string into the characters they refer to, printf %b can be used for this purpose.
#!/usr/bin/env bash
fallback_message="TEST MESSAGE - $HOSTNAME"
markdown_message="TEST MESSAGE - $HOSTNAME \(0x0a) \(\n) Hi <#U12345789>\n\`\`\`Can we do a\nmultiline code block\`\`\`"
# create markdown_message_unescaped with an unescaped version of markdown_message
printf -v markdown_message_unescaped %b "$markdown_message"
jq -n \
--arg textMessage "$markdown_message_unescaped" \
--arg fallbackMessage "$fallback_message" \
--arg sectionType section --arg markdownType markdown '
{
text: $fallbackMessage,
blocks: [
{
type: $sectionType,
text: {
type: $markdownType,
text: $textMessage
}
}
]
}'
...properly emits as output:
{
"text": "TEST MESSAGE - YOUR_HOSTNAME",
"blocks": [
{
"type": "section",
"text": {
"type": "markdown",
"text": "TEST MESSAGE - YOUR_HOSTNAME (0x0a) (\n)\nHi <#U12345789>\n```\nCan we do a multiline code block\n```"
}
}
]
}

How do I get the following value out of this JSON response?

Basically, I have a retrieved values from a DynamoDB using Powershell and have gotten a row in JSON format like the following
function gettagsfromdynamo() {
$table_name = "table_name"
$dbkey = '{"ReleaseId":{"AttributeValueList":[ {"N":"1"} ],"ComparisonOperator": "EQ"}}' | ConvertTo-Json -Compress
$dbvalue = aws dynamodb query --table-name $table_name --key-conditions $dbkey --region $region
$latest_tags = $dbvalue
$latest_tags
}
$db_tags = gettagsfromdynamo
This is what db_tags looks like
{
"Items": [
{
"Comment": {
"S": "The first PC release."
},
"Product": {
"S": "PC"
},
"ReleaseId": {
"N": "1"
},
"CreatedOn": {
"S": "12/14/2020 15:23:32"
},
"Tags": {
"S": "{\n \"software1\": \"software1.zip\",\n \"software2\": \"software2.zip\",\n \"software3\":\n [\n \"software3.zip\",\n \"software4.zip\",\n \"software5.zip\"\n ],\n \" data1 \": \"2020_NA\",\n \" 2020_EU \": \"20201_EU\",\n \" 2020_WW \": \"2021_WW\",\n \" dataversions\":\n [\n \"2020\",\n \"2019\",\n \"2018\",\n \"2017"\n ],\n \" products \": \" \"\n}"
}
}
],
"Count": 1,
"ScannedCount": 1,
"ConsumedCapacity": null
}
The task I want to achieve is to be able to get the "dataversions" value which is --> [Items][Tags][Dataversions] and write that value to a JSON file available locally. I have tried various things including using the Convert-ToJson and ConvertFrom-Json.
The tags json value looks like this without the escaped spaces (/n)
{
"software1": "software1.zip",
"software2": "software2.zip",
"software3":
[
"software3.zip",
"software4.zip",
"software5.zip"
],
" data1": "2020_NA",
" 2020_eu ": "2020_EU",
" 2020_ww": "2021_WW",
" dataversions":
[
"2020",
"2019",
"2018",
"2017"
],
" products ": " "
}
How do I retrieve the value of 'dataversions', which is a list of strings. Right now, I can only get it like this after using various tries:
{"S":"{\n \"software1\": \"software1.zip\",\n \"software2\": \"software2.zip\",\n \"software3\":\n [\n \"software3.zip\",\n \"software4.zip\",\n \"software5.zip\"\n ],\n \" data1\": \"2020_NA\",\n \" 2020_EU\": \"20201_EU\",\n \" 2020_WW\": \"2021_WW\",\n \" dataversions\":\n [\n \"2020\",\n \"2019\",\n \"2018\",\n \"2017\"\n ],\n \" products\": \" \"\n}"}
I want to be able to get the value of dataversions inorder to overwrite another 'dataversions' which is inside example.json file. How do I get to the dataversions value and also clean up the \n?
In your JSON file, the property name " dataversions" contains a leading space. When the JSON string is converted to a custom object (via ConvertFrom-Json), the space will be included in the property name. Therefore it must be considered when using member access (object.property) syntax:
($db_tags | ConvertFrom-Json).' dataversions'

select specific values from json & convert into bash variables

I'm new to jq, I have the following JSON & I need to extract FOUR values i.e. SAUCE_KEY, sauceKey, SAUCE_VALUE, sauceValue etc. And I need to covert these bash variables as i.e.
SAUCE_KEY=sauceKey
SAUCE_VALUE=sauceValue
If I will echo it, it should print it's value ie. echo $SAUCE_KEY
I have used the code as:
jq -r '.env_vars[] | with_entries(select([.key] | inside([".env_vars[].name", ".env_vars[].value"])))' | jq -r "to_entries|map(\"\(.value)=\(.value|tostring)\")|.[]"
By doing so, i was able to get values as
name=SAUCE_KEY, value=sauceKey and so on.
{
"#type": "env_vars",
"env_vars": [
{
"#type": "env_var",
"#href": "/repo/xxxxx/env_var/xxxxx",
"#representation": "standard",
"#permissions": {
"read": true,
"write": true
},
"name": "SAUCE_KEY",
"value": "sauceKey"
},
{
"#type": "env_var",
"#href": "/repo/xxxxx/env_var/xxxxx",
"#representation": "standard",
"#permissions": {
"read": true,
"write": true
},
"name": "SAUCE_VALUE",
"value": "sauceValue"
}
]
}
If instead of trying to extract variable names from the JSON, you populate an associate array with the names as keys, it's pretty straightforward.
#!/usr/bin/env bash
declare -A sauces
while IFS=$'\001' read -r name value; do
sauces[$name]=$value
done < <(jq -r '.env_vars[] | "\(.name)\u0001\(.value)"' saucy.json)
for name in "${!sauces[#]}"; do
echo "$name is ${sauces[$name]}"
done
prints out
SAUCE_VALUE is sauceValue
SAUCE_KEY is sauceKey

Convert a txt file into JSON

I need to convert a simple txt list into a specific json format.
My list looks like this :
server1
server2
server3
server4
I need to have a JSON output that would look like this :
{ "data": [
{ "{SERVER}":"server1" },
{ "{SERVER}":"server2" },
{ "{SERVER}":"server3" },
{ "{SERVER}":"server4" }
]}
I was able to generate this with a bash script but I don't know how to remove the comma for the last line. The list is dynamic and can have a different amount of servers every time the script is run.
Any tip please ?
EDIT : my current code :
echo "{ "data": [" > /tmp/json_output
for srv in `cat /tmp/list`; do
echo "{ \"{SERVER}\":\"$srv\" }," >> /tmp/json_output
done
echo "]}" >> /tmp/json_output
I'm very new at this, sorry if I sound noobish.
This is very easy for Xidel:
xidel -s input.txt -e '{"data":[x:lines($raw) ! {"{SERVER}":.}]}'
{
"data": [
{
"{SERVER}": "server1"
},
{
"{SERVER}": "server2"
},
{
"{SERVER}": "server3"
},
{
"{SERVER}": "server4"
}
]
}
x:lines($raw) is a shorthand for tokenize($raw,'\r\n?|\n'). It creates an array of every new line.
In human terms you can see x:lines($raw) ! {"{SERVER}":.} as "create a JSON object for every new line".
See also this xidelcgi demo.
I would use jq for this.
$ jq -nR 'reduce inputs as $i ([]; .+[$i]) | map ({"{SERVER}": .}) | {data: .}' tmp.txt
{
"data": [
{
"{SERVER}": "server1"
},
{
"{SERVER}": "server2"
},
{
"{SERVER}": "server3"
},
{
"{SERVER}": "server4"
}
]
}
(It seems to me there should be an easier way to produce the array ["server1", "server2", "server3", "server4"] to feed to the map filter, but this is functional.)
Breaking this down piece by piece, we first create an array of your server names.
$ jq -nR 'reduce inputs as $item ([]; .+[$item])' tmp.txt
[
"server1",
"server2",
"server3",
"server4"
]
This array is fed to a map filter that creates an array of objects with the {SERVER} key:
$ jq -nR 'reduce inputs as $item ([]; .+[$item]) | map ({"{SERVER}": .})' tmp.txt
[
{
"{SERVER}": "server1"
},
{
"{SERVER}": "server2"
},
{
"{SERVER}": "server3"
},
{
"{SERVER}": "server4"
}
]
And finally, this is used to create an object that maps the key data to the array of objects, as shown at the top.
As others said a tool like jq should better be used.
Alternatively, you could use an array and determine whether the element you process is the last or not, for example your code could be :
declare -a servers
servers=($(cat /tmp/list))
pos=$(( ${#servers[*]} - 1 ))
last=${servers[$pos]}
echo "{ "data": [" > /tmp/json_output
for srv in "${servers[#]}"; do
if [[ $srv == $last ]]
then
echo "{ \"{SERVER}\":\"$srv\" }" >> /tmp/json_output
else
echo "{ \"{SERVER}\":\"$srv\" }," >> /tmp/json_output
fi
done
echo "]}" >> /tmp/json_output
You can do this using Python 3 :
#!/usr/local/bin/python3
import os
import json
d = []
with open('listofservers.txt', 'r', encoding='utf-8') as f:
for server in f:
d.append({'{Server}' : server.rstrip("\n")})
print(json.dumps({'data': d}, indent=" "))
Which will print :
{
"data": [
{
"{Server}": "server1"
},
{
"{Server}": "server2"
},
{
"{Server}": "server3"
},
{
"{Server}": "server4"
}
]
}