How to iterate Json object in shell script [duplicate] - json

This question already has answers here:
Parsing JSON with Unix tools
(45 answers)
Closed 5 years ago.
I am writing a shell script to run some api's. It return response fine but i need some specific parameter to grep from the response and want to save in file.
My script look like
#!/bin/sh
response=$(curl 'https://example.com' -H 'Content-Type: application/json' )
echo "$response"
reponse is something like
{
status:"success",
response:{
"target":"",
"content":"test content"
}
}
Response is fine and i am able to write whole response in file but My requirement is to save only "content" inside "response" object using the script. which i need for another api.
Note: I cannot change api responses as I am working third party api's;
Thank you

If the output is proper JSON:
$ cat proper.json
{
"status": "success",
"response": {
"target": "",
"content": "test content"
}
}
$ response=$(cat proper.json)
You could use jq:
$ echo $response | jq -r '.response.content'
test content

You can grep for the content and then use awk to split by : and take only the value, not the key
grep "\"content\":" | awk -F":" '{ print $2}'
Will print "test content"

You can do this to get the value of contents into a variable ($content).
content=$(echo "$response" | cut -d'"' -f 7)
Explanation - Split the $response using " (double quote) as the delimiter and use the 7th field of the output (i.e the value (test content) of content in the json response)
Here is an excerpt from the description and usage of the cut command
if you like to extract a whole field, you can combine option -f and -d. The option -f specifies which field you want to extract, and the option -d specifies what is the field delimiter that is used in the input file.

Related

How to add commas in between JSON objects using Linux Shell and SnowSQL?

While there are several posts about this topic on Stack Overflow, none match my exact use case. I am using a Linux shell script to run SnowSQL to generate a json file.
========================
My json file needs to have a comma between json objects.
This:
{
"CAMPAIGN": "Welcome_New",
"UUID": "fe881781-bdc2-41b2-95f2-e0e8c19dc597"
}
{
"CAMPAIGN": "Welcome_Existing",
"UUID": "77a41c02-beb9-48bf-ada4-b2074c1a78cb"
}
...needs to look this:
{
"CAMPAIGN": "Welcome_New",
"UUID": "fe881781-bdc2-41b2-95f2-e0e8c19dc597"
},
{
"CAMPAIGN": "Welcome_Existing",
"UUID": "77a41c02-beb9-48bf-ada4-b2074c1a78cb"
}
Here is my complete ksh script:
#!/usr/bin/ksh
. /appl/.snf_logon
export SNOW_PKEY_FILE=$(mktemp ./pkey-XXXXXX)
trap "rm -f ${SNOW_PKEY_FILE}" EXIT
LibGetSnowCred
{
outFile=JSON_FILE_TYPE_TEST.json
inDir=/testing
outFileNm=#my_db.my_schema.my_file_stage/${outFile}
snowsql \
--private-key-path $SNOW_PKEY_FILE \
-o exit_on_error=true \
-o friendly=false \
-o timing=false \
-o log_level=ERROR \
-o echo=true <<!
COPY INTO ${outFileNm}
FROM (SELECT object_construct(
'UUID',UUID
,'CAMPAIGN',CAMPAIGN)
FROM my_db.my_schema.JSON_Test_Table
LIMIT 2)
FILE_FORMAT=(
TYPE=JSON
COMPRESSION=NONE
)
OVERWRITE=True
HEADER=False
SINGLE=True
MAX_FILE_SIZE=4900000000
;
get ${outFileNm} file://${inDir}/;
rm ${outFileNm};
!
if [ $? -eq 0 ]; then
echo "Export successful"
else
echo "ERROR in export"
fi
}
Is the best practice to add the comma during the SELECT or after the file is generated and how?
With or without that comma, the text is still not JSON but just a random text that looks like JSON. You export several rows, each row as an independent object. You need to gather all these objects into an array to produce a valid JSON.
A JSON that encodes an array of rows looks like this:
[
{
"CAMPAIGN": "Welcome_New",
"UUID": "fe881781-bdc2-41b2-95f2-e0e8c19dc597"
},
{
"CAMPAIGN": "Welcome_Existing",
"UUID": "77a41c02-beb9-48bf-ada4-b2074c1a78cb"
}
]
The easiest way to produce this output would be to ask the database, if it supports this option (to wrap all the records into a list before generating the JSON, to not export each record in a separate JSON).
If this is not possible then you have a file that contains multiple JSONs. You can use jq to convert these individual JSONs into a JSON similar to the one described above (encoding an array of objects).
It is as simple as that:
jq --slurp '.' input_file > output_file
The option --slurp tells jq to read all the JSONs from the file input_file in memory, to parse them and to put them into an array. That is the program input.
'.' is the jq program. It says "dump the current object". It does not do any processing to the input data. The current object is the array.
After it executes the program (which, in this case doesn't do anything), jq dumps the modified value (as JSON, of course) to the standard output (by default, on screen).
The > output_file part redirects this output to a file (named output_file) instead of showing it on screen.
You can see how it works on the jq playground.

How to use shell script pipe json to curl to sent slack message [duplicate]

This question already has an answer here:
Passing JSON arguments (not simple strings) to jq
(1 answer)
Closed 3 years ago.
I have a monitor API (for example: www.example.com/monitor ) well return system status in JSON format, for example:
{
"foo" : 1000,
"bar" : 100
}
I want to use curl to get system status from this API and pipe to Slack
The following is my current script.
status=$(curl "http://www.xample.com/monitor/status" )
slackWebHook="https://hooks.slack.com/services/xxxxx/xxxxx/xxxxx"
function sentSlack () {
json_template='{
channel: $channel,
username: $username,
text: $text,
icon_emoji: $icon_emoji,
attachments: [$attachments]
}'
jq -n --arg channel "#unicorn_log" \
--arg username "Kuasa Search Report" \
--arg text "${msg}"\
--arg icon_emoji "chart_with_upwards_trend" \
--arg attachments "${status}" "$json_template" |
curl -i -X POST --data-urlencode "$(</dev/stdin)" ${slackWebHook}
}
sentSlack
I always get invalid_payload , because the status be escaped.
{
\"foo\" : 1000,
\"bar\" : 100
}
How can I pass JSON to Slack payload correctly?
--arg key value treats value as a string. Since a string containing literal "s needs to have escapes added to be valid in JSON, that's what jq does.
--argjson key value, by contrast, parses value as JSON text; which appears to be what you want to do here.
Change --arg attachments "$status" to --argjson attachments "$status".

Populating variables from another file reading user input in bash/shell script

I am giving the user the possibility to answers some questions so that I can perform the according cURL requests and create what is needed accordingly.
In this way:
~functions.sh
echo "Question number 1?"
read question1
echo "Question number 2?"
read question2
The script is organised in a couple of files:
functions.sh, where I store all the functions
flow.sh, where the questions are asked and the previous functions are called in order
credentials.sh, where I store the api key credentials, api url etc.
env.sh, where I am hoping to store the env configuration that needs to be dynamically generated...
Unfortunately I am pretty much new to that as I come from a Frontend Background so I Am not aware or can't figure out/find on here, how to achieve the following:
I would need to send a cURL request based on one of the users answers, and that's easily done just by calling ie {$question1} inside one of the functions of functions.sh that is later called in the main flow.sh after putting the file as a source...
Unfortunately, when it comes to populating a multiline environment configuration in a separate file to be later be sent as a json in the cURL it gets out of my comfort zone...
Id need so send something along the lines of
curl -X POST "https//api.url"
-H "headers"
--data '{"content":"'{$env}'"}'
Where $env is the env.sh file, json valid, and populated with the variables read before from user input accordingly...
To solve the multi line json problem maybe I could use the following:
$( awk '{printf "%s\\n", $0}' ./env.sh )
Still I am clueless on how to populate such file form user read and echo it inside that json part in the cURL...
You could use awk to directly save correct JSON data to env.sh (partly taken from Dr.K's answer to this other question):
echo "Question number 1?"
read question1
echo "Question number 2?"
read question2
echo "Question number 3?"
read question3
awk ' BEGIN { print "[" ; } \
{ print " {" \
print " \"Question1\" : \"" $question1 "\"," \
print " \"Question2\" : \"" $question2 "\"," \
print " \"Question3\" : \"" $question3 "\"" \
print " }" \
} \
END { print "]" } ' env.sh
And then this should work correctly as JSON:
curl -X POST "https//api.url"
-H "headers"
--data '{"content":"{$env}"}'
and the data payload should be like this:
{
content: [
{
"Question1" : "User answer 1",
"Question2" : "User answer 2",
"Question3" : "User answer 3"
}
]
}
Is this what you had in mind?
For anyone who ends up here what I did was:
Creating a separate file where I set a variable and I cat the EOF inside of that.
env=$( cat <<EOF
multi
line
content
goes
here
EOF
)
export env
Then I generate a valid json through jq
generate_json() {
newenv="$( jq -nc --arg str "$env" '{"content": $str}' )"
}
and finally I use $newenv in the curl straight away:
example_curl() {
curl -X PUT "https://apiendpoint.co.uk" \
-H "Authorization: Bearer 123123" \
--data "${newenv}"
}

jq add file contents into json and update

I'm trying to use take a JSON object from a REST API GET call, then add the contents of a raw file (.md file in this case) into it using jq, before updating the same object using a PUT call.
I'm using the following to GET the file and write it locally:
curl -u USERNAME:PASSWORD 'https://example.com/myfile.json' | cat > ./test.json
The format of the JSON file is as follows:
{
"content" : "My JSON content",
"owner":...
}
I'd like to add the content of the raw file into content, so the result is as follows:
{
"content" : "My markdown file contents.\n\nMy JSON content",
"owner":...
}
and then update the original JSON using a PUT call:
curl -u USERNAME:PASSWORD -d #./test.json -H "Content-Type: application/json" -X PUT 'https://example.com/myfile.json'
I'm wondering how I can add the file content into my JSON file like that using jq, if it is possible?
The key to a simple jq solution here is to read the raw text file using 'jq -R -s', and to read the JSON using one of the options in the --arg family. Here, I'll use --argfile for simplicity and robustness across jq versions, but please note that the documentation says it is deprecated.
With the following jq program in a file, say program.jq:
. as $file
| $json
| (.content = $file + "\n" + .content)
and the following text in the file contents.txt:
Line 1 of contents.txt;
line 2.
and the following JSON in curl.json:
{
"content": "My JSON content",
"owner": "etc"
}
the invocation:
jq -R -s --argfile json curl.json -f program.jq contents.txt
produces:
{
"content": "Line 1 of contents.txt;\nline 2.\n\nMy JSON content",
"owner": "etc"
}
If using bash, instead of putting the curl output into a file, you could use: --argfile json <(curl ....)

REST API JSON value from CURL command into table format in UNIX

I have a shell script with curl -s http://ifconfig.me/all.json command which prints below output in the terminal window.
{
"version" : {
"ip_addr": "201.73.103.12",
"lang": "java",
"remote_host": "OpenSSL/0.9.8w zlib/1.2.3 libidn/1.23 libssh2/1.2.2",",
"user_agent": "curl/7.23.1 (i386-sun-solaris2.11) libcurl/7.23.1
"charset": "",
"port": "63713"}
}
I need to display the JSON value in table format.
Someone, please help me with this to implement in UNIX shell script. Thanks!
Look into the cool command-line JSON query parser / formatter tool:
http://stedolan.github.io/jq/
You can pipe your JSON input into this tool and extract out / format the keys you need.
Or, just run multiple curl -s http://ifconfigme/ calls for each "row/column" you need and output it in some sane format, E.g.:
#!/bin/bash
ip=`curl -s http://ifconfig.me/ip`
host=`curl -s http://ifconfig.me/host`
echo -e "ip\thost"
echo -e "${ip}\t${host}"