Populating variables from another file reading user input in bash/shell script - json

I am giving the user the possibility to answers some questions so that I can perform the according cURL requests and create what is needed accordingly.
In this way:
~functions.sh
echo "Question number 1?"
read question1
echo "Question number 2?"
read question2
The script is organised in a couple of files:
functions.sh, where I store all the functions
flow.sh, where the questions are asked and the previous functions are called in order
credentials.sh, where I store the api key credentials, api url etc.
env.sh, where I am hoping to store the env configuration that needs to be dynamically generated...
Unfortunately I am pretty much new to that as I come from a Frontend Background so I Am not aware or can't figure out/find on here, how to achieve the following:
I would need to send a cURL request based on one of the users answers, and that's easily done just by calling ie {$question1} inside one of the functions of functions.sh that is later called in the main flow.sh after putting the file as a source...
Unfortunately, when it comes to populating a multiline environment configuration in a separate file to be later be sent as a json in the cURL it gets out of my comfort zone...
Id need so send something along the lines of
curl -X POST "https//api.url"
-H "headers"
--data '{"content":"'{$env}'"}'
Where $env is the env.sh file, json valid, and populated with the variables read before from user input accordingly...
To solve the multi line json problem maybe I could use the following:
$( awk '{printf "%s\\n", $0}' ./env.sh )
Still I am clueless on how to populate such file form user read and echo it inside that json part in the cURL...

You could use awk to directly save correct JSON data to env.sh (partly taken from Dr.K's answer to this other question):
echo "Question number 1?"
read question1
echo "Question number 2?"
read question2
echo "Question number 3?"
read question3
awk ' BEGIN { print "[" ; } \
{ print " {" \
print " \"Question1\" : \"" $question1 "\"," \
print " \"Question2\" : \"" $question2 "\"," \
print " \"Question3\" : \"" $question3 "\"" \
print " }" \
} \
END { print "]" } ' env.sh
And then this should work correctly as JSON:
curl -X POST "https//api.url"
-H "headers"
--data '{"content":"{$env}"}'
and the data payload should be like this:
{
content: [
{
"Question1" : "User answer 1",
"Question2" : "User answer 2",
"Question3" : "User answer 3"
}
]
}
Is this what you had in mind?

For anyone who ends up here what I did was:
Creating a separate file where I set a variable and I cat the EOF inside of that.
env=$( cat <<EOF
multi
line
content
goes
here
EOF
)
export env
Then I generate a valid json through jq
generate_json() {
newenv="$( jq -nc --arg str "$env" '{"content": $str}' )"
}
and finally I use $newenv in the curl straight away:
example_curl() {
curl -X PUT "https://apiendpoint.co.uk" \
-H "Authorization: Bearer 123123" \
--data "${newenv}"
}

Related

cURL doesn't print whole JSON output

i'm trying to build some script that i will need to check json via curl and print the output, but somehow it's not working. My other API JSON are working fine using this script
Script :
#!/bin/bash
OUT=$(curl -X POST -s -w "|%{time_total}|%{http_code}"\
-H "Content-Type: application/json" \
--ciphers 'AES256-GCM-SHA384' \
-d '{
"Username": "van_user1test",
"Password" : "123MM4566"}' \
https://127.0.0.1/api/call/login)
echo "THIS IS MASTER OUT" $OUT
out2="$(echo $OUT | cut -d\| -f 1)"
out_response="$(echo ${OUT} | cut -d\| -f 2)"
out_httpcode="$(echo ${OUT} | cut -d\| -f 3)"
echo "This is out2" $out2
echo "this is response" $out_response
echo "this is httpcode" $out_httpcode
And i'm getting as result :
}|0.437357|200ccess at 07/03/2022 11:23:11"
}Message": "Success at 07/03/2022 11:23:11"
this is response 0.437357
this is httpcode 200
But when i run curl command(above) manually, i get fine response :
{
"Successmessage": true,
"Message_status": "Success at 07/03/2022 11:27:28"
}|0.483469|200
I guess the problem can be in the tags somewhere because the first line SUccessmessage is missing and i'm worried why, also the format is not displayed correctly
I want to get fully response from json output and printing all the lines
Update : HOW to display in one line the latest result ?
{
"Successmessage": true,
"Message_status": "Success at 07/03/2022 11:27:28"
}|0.483469|200
Thank you!

cURL: unmatched brace in URL position when using bash script

I am trying to write a script to allow for easy writing to a firebase database using the REST API. Here is the part of the script causing issues:
#!/bin/bash
echo "patient username:"
read USER
echo "name:"
read NAME
echo "birthday:"
read BDAY
echo "insurance provider:"
read INSPROV
echo "insurance number:"
read INSNUMB
echo "group number:"
read GRPNUMB
form0="'"
form10='{\n "'
form1=${form0}${form10}
form2='": {\n "'
form3='name": "'
form4='",\n "birthday": "'
form5='",\n "insurance provider": "'
form6='",\n "insurance number": "'
form7='",\n "group number": "'
form8='"\n }\n}'
form9="' "
link="'https://billify-51384-default-rtdb.firebaseio.com/userinfo.json'"
#adding strings together
send1=${form1}${USER}
send2=${form2}${form3}
send3=${send2}${NAME}
send4=${send3}${form4}
send5=${send4}${BDAY}
send6=${send5}${form5}
send7=${send6}${INSPROV}
send8=${send7}${form6}
send9=${send8}${INSNUMB}
send10=${send9}${form7}
send11=${send10}${GRPNUMB}
send12=${send11}${form8}
send13=${send1}${send12}
send14=${send13}${form9}
URL=${send14}${link}
printf "$URL" >> tempfile.txt
URLFX=$(<tempfile.txt)
echo
echo "PRINTING DATA---------------------------------"
printf "$URLFX"
echo
curl -X PUT -d $URLFX
rm tempfile.txt
Sorry for all of the string combining, bash is terrible at making it efficient!
I put the JSON data into a text file then make the script read it to fix the newline issues (which it seems to do). When I run the script I get this output:
PRINTING DATA---------------------------------
'{
"testuser": {
"name": "Johnny Smith",
"birthday": "January 3, 1903",
"insurance provider": "CompanyName",
"insurance number": "101010",
"group number": "1010"
}
}' 'https://billify-51384-default-rtdb.firebaseio.com/userinfo.json'
curl: (3) URL using bad/illegal format or missing URL
curl: (3) unmatched brace in URL position 1:
{
^
It seems to be formatting everything correctly, as when I add the printed data to the cURL command manually, it works and the data is added. What is cURL's issue with the script? Thank you!
The immediate problem is the lack of quoting around $URLFX (which is kind of funny, seeing as you quote it correctly when you print it) and the addition of single quotes around the JSON string, which you probably put in in a misdirected effort to fix the quoting problem.
There is a difference between syntactic quotes around values in code, and literal quotes inside those values. Only the former affect how the value is parsed by the shell. Observe the difference between echo ' (a syntax error) and echo "'" (a string consisting of one single quote, in syntactic double quotes to protect the string from being interpreted as syntax).
With that out of the way, try this refactoring.
#!/bin/bash
read -r -p "patient username: " User
read -r -p "name: " Name
read -r -p "birthday: " Bday
read -r -p "insurance provider: " Insprov
read -r -p "insurance number: " Insnumb
read -r -p "group number: " Grpnumb
curl -X PUT -d "\
{
\"$User\":
{
\"name\": \"$Name\",
\"birthday\": \"$Bday\",
\"insurance provider\": \"$Insprov\",
\"insurance number\": \"$Insnumb\",
\"group number\": \"$Grpnumb\"
}
}" 'https://billify-51384-default-rtdb.firebaseio.com/userinfo.json'
It is not customary to break up the JSON you send to a URL over multiple lines like this, but Bash can cope just fine. Inside syntactic double quotes, we need to backslash-escape any literal double quotes to preserve them, which is slightly unattractive.
Notice also Correct Bash and shell script variable capitalization and probably read When to wrap quotes around a shell variable?
If you are already storing the string to the file tempfile.txt, why not just upload the file itself.
In addition, you should also set the appropriate headers.
curl -X "POST" "$URL" --data #tempfile.txt -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: MyScript/0.1"

How to iterate Json object in shell script [duplicate]

This question already has answers here:
Parsing JSON with Unix tools
(45 answers)
Closed 5 years ago.
I am writing a shell script to run some api's. It return response fine but i need some specific parameter to grep from the response and want to save in file.
My script look like
#!/bin/sh
response=$(curl 'https://example.com' -H 'Content-Type: application/json' )
echo "$response"
reponse is something like
{
status:"success",
response:{
"target":"",
"content":"test content"
}
}
Response is fine and i am able to write whole response in file but My requirement is to save only "content" inside "response" object using the script. which i need for another api.
Note: I cannot change api responses as I am working third party api's;
Thank you
If the output is proper JSON:
$ cat proper.json
{
"status": "success",
"response": {
"target": "",
"content": "test content"
}
}
$ response=$(cat proper.json)
You could use jq:
$ echo $response | jq -r '.response.content'
test content
You can grep for the content and then use awk to split by : and take only the value, not the key
grep "\"content\":" | awk -F":" '{ print $2}'
Will print "test content"
You can do this to get the value of contents into a variable ($content).
content=$(echo "$response" | cut -d'"' -f 7)
Explanation - Split the $response using " (double quote) as the delimiter and use the 7th field of the output (i.e the value (test content) of content in the json response)
Here is an excerpt from the description and usage of the cut command
if you like to extract a whole field, you can combine option -f and -d. The option -f specifies which field you want to extract, and the option -d specifies what is the field delimiter that is used in the input file.

jq add file contents into json and update

I'm trying to use take a JSON object from a REST API GET call, then add the contents of a raw file (.md file in this case) into it using jq, before updating the same object using a PUT call.
I'm using the following to GET the file and write it locally:
curl -u USERNAME:PASSWORD 'https://example.com/myfile.json' | cat > ./test.json
The format of the JSON file is as follows:
{
"content" : "My JSON content",
"owner":...
}
I'd like to add the content of the raw file into content, so the result is as follows:
{
"content" : "My markdown file contents.\n\nMy JSON content",
"owner":...
}
and then update the original JSON using a PUT call:
curl -u USERNAME:PASSWORD -d #./test.json -H "Content-Type: application/json" -X PUT 'https://example.com/myfile.json'
I'm wondering how I can add the file content into my JSON file like that using jq, if it is possible?
The key to a simple jq solution here is to read the raw text file using 'jq -R -s', and to read the JSON using one of the options in the --arg family. Here, I'll use --argfile for simplicity and robustness across jq versions, but please note that the documentation says it is deprecated.
With the following jq program in a file, say program.jq:
. as $file
| $json
| (.content = $file + "\n" + .content)
and the following text in the file contents.txt:
Line 1 of contents.txt;
line 2.
and the following JSON in curl.json:
{
"content": "My JSON content",
"owner": "etc"
}
the invocation:
jq -R -s --argfile json curl.json -f program.jq contents.txt
produces:
{
"content": "Line 1 of contents.txt;\nline 2.\n\nMy JSON content",
"owner": "etc"
}
If using bash, instead of putting the curl output into a file, you could use: --argfile json <(curl ....)

payload is not valid JSON

I'm using curl to send JSON to an API endpoint. However, somewhere in the bash chain it is getting messed up.
Is there something special to know about encoding with curl?
If I construct the payload like this:
PAYLOAD='payload={"channel": "github", "username": "webhookbot", "icon_emoji": ":ghost:", "text": "'
PAYLOAD+=$1
PAYLOAD+=' " }'
echo $PAYLOAD
curl -X POST --data-urlencode "$PAYLOAD" $SLACKPOSTURL
echo "sent"
I'll get back an error
Payload was not valid JSONsent
however if i just hardwire to assign a variable with the output
PAYLOAD='payload={"channel": "github", "username": "webhookbot", "icon_emoji": ":ghost:", "text": "LAST_COMMIT Merge pull request #558 from dcsan/boteditor Boteditor " }'
then it will go through fine.
Is there something that a simple assignment is doing differently vs. concatenating strings? In the console the output looks identical.
FWIW some messages go through but content like this:
LAST_COMMIT Merge pull request #558 from dcsan/boteditor Boteditor
will only go through if hardcoded in. so its not the other end afaican see, its something to do with the way messages are built.
I guess you want to concatenate values into your variable. But += is not the way to do so.
To concatenate strings in a variable you need to say:
PAYLOAD="$PAYLOAD $1"
All together it would be something like the following. Note the need to use " so that the variable $PAYLOAD is expanded and the usage of \" to store a literal double quote:
PAYLOAD='payload={"channel": "github", "username": "webhookbot", "icon_emoji": ":ghost:", "text": "'
PAYLOAD="$PAYLOAD $1 \" }"
echo "$PAYLOAD"
curl -X POST --data-urlencode "$PAYLOAD" $SLACKPOSTURL
echo "sent"
This is what worked from me from a bash script:
curl -X POST --data-urlencode "payload={\"text\": \"$2\"}" https://hooks.slack.com/services/$KEY
Notice the inner quotes are escaped, but the outer quotes are not.
FYI, adding:
set -x
at the beginning of a bash script will show you the actual commands being executed, and save a lot of guesswork.