Add JSON element in bash script - json

within my bash script I have a line that looks like this:
JOB_RESULTS=$(curl --fail -k -H "Content-Type: application/xml" --write-out HTTP_CODE='%{http_code}' "${request_url}")
This line outputs the following:
{"name":"callCLF010Job","id":11693,"status":"STARTING","message":"The job has started.","exitCode":"exitCode=UNKNOWN;exitDescription="}HTTP_CODE=200
The problem is, I want to add the HTTP_CODE into the JSON message.
Is there a way to make the return look like {"name":"callCLF010Job","id":11693,"status":"STARTING","message":"The job has started.","exitCode":"exitCode=UNKNOWN;exitDescription=","HTTP_CODE":"200"}
EDIT:
With confettis changes my code looks like this:
http_code="${JOB_RESULTS:${#JOB_RESULTS}-17}"
http_body="${JOB_RESULTS:0:${#JOB_RESULTS}-17}"
http_code_json=", ${http_code}}"
my_result="${http_body/%\}/$http_code_json}"
when I run echo $my_result my output looks like this:
{"name":"callCLF010Job","id":11702,"status":"STARTING","message":"The job has started.","exitCode":"exitCode=UNKNOWN;exitDescriptio

One option is to use sed to format the output the way you want:
echo $JOB_RESULTS | sed 's/}HTTP_CODE=\([0-9]\+\)$/,"HTTP_CODE":"\1"}/'
Or inside your command:
JOB_RESULTS=$(curl --fail -k -H "Content-Type: application/xml" --write-out HTTP_CODE='%{http_code}' "${request_url}" | sed 's/}HTTP_CODE=\([0-9]\+\)$/,"HTTP_CODE":"\1"}/')

This is using shell parameter expansion to get the job done. (Pure bash, besides curl.)
curlout=$(curl -s --fail -k -H "Content-Type: application/xml" --write-out '"HTTP_CODE":"%{http_code}"' "http://example.com")
http_code="${curlout:${#curlout}-17}"
http_body="${curlout:0:${#curlout}-17}"
http_code_json=", ${http_code}}"
my_result="${http_body/%\}/$http_code_json}"
Replace http://example.com with the URL or variable of your choice.
In order to prevent errors in case of a wrong URL or (no) output, you should put the last four lines within an if-construct. E.g. if [[ $http_code != '"HTTP_CODE":"000"' ]].

Related

Pass json value to curl variable via CLI within Bash script

Using GET I need to pass a json value to a URL via the command line within a bash script.
This works:
curl -i "http://MYURL:8080/admin/rest_api/api?api=trigger_dag&dag_id=spark_submit&conf=\{\"filename\":\"myfile.csv\"\}"
If I want to expand on the json value, I would prefer to pass a variable via the URL parameter for readability. Somethig like ... but this doesn't appear to work correctly.
generate_post_data =
{
"filename": "myfile.csv"
}
curl -i "http://MYURL:8080/admin/rest_api/api?api=trigger_dag&dag_id=spark_submit&conf=${generate_post_data}"
You need to properly set the variable and you should url encode it using the --data-urlencode option.
#!/bin/bash
generate_post_data="filename=myfile.csv"
curl -G "http://MYURL:8080/admin/rest_api/api?api=trigger_dag&dag_id=spark_submit" --data-urlencode $generate_post_data
From the manpage:
--data-urlencode <data>
(HTTP) This posts data, similar to the other -d, --data options with
the exception that this performs URL-encoding.
To be CGI-compliant, the part should begin with a name followed
by a separator and a content specification. The part can be
passed to curl using one of the following syntaxes:
For more info you can use man curl and then /data-urlencode to jump to the section on it.

How to get JSON data value in to separate variable using shell script

I am getting response from the below command in JSON format in .sh file (using shell script):
**curl https://api.flipkart.net/sellers/skus/$col1/listings -H "Content-Type: application/json" -H "Authorization: Bearer $value"**
Here is the response I am getting:
{"listingId":"LSMSW","skuId":"M7489","fsn":"ACCCQD","attributeValues":{"actual_stock_count":"1","mrp":"199","seller_listing_state":"current","procurement_sla":"2","zonal_shipping_charge":"0","stock_count":"10","local_shipping_charge":"0","listing_status":"ACTIVE","max_order_quantity_allowed":"3","fulfilled_by":"seller","fk_release_date":"2016-08-29 10:00:08","selling_price":"529","inventory_count":"10","national_shipping_charge":"0","sku_id":"M7489"},"listingValidations":null}
Now I want to extract all the value in a separate variable for reuse it
like:
LISTINGID='listing id value'
SKUID='skuId value'
and many more.
If anyone know the answer please comment with explanation.
Here is a solution using jq. If data.json contains the sample data then the following bash script
#!/bin/bash
jq -M -r '
.listingId
, .skuId
, .attributeValues.mrp
' data.json | \
while read -r listingId
read -r skuId
read -r mrp; do
echo "$listingId $skuId $mrp"
done
produces
LSMSW M7489 199
The jq command in the script writes out the specified attributes on separate lines which are read into bash variables by the successive read -r commands. It should be clear how to extend or modify this to meet your requirements – the primary constraint is that for each line written by jq there should be a corresponding read -r.
I am not aware of any bash json parsers. I would go with Python or Php and parse the Json string in one of these languages and export values from there. In Php there is the putenv() method.
Good luck!
Edit:
Check here
Parsing JSON with Unix tools

How to use bash and curl with escaped json?

The two commands below work perfectly outside of a script if I execute them one after the other, however, if I put them inside a bash script. They fail.
DATA="{\"size\":500,\"sort\":{\"#timestamp\":\"desc\"},\"query\":{\"filtered\":{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"tx.traceId: AIP1447283489-6 AND event: published_notification AND attempt: 1\"}},\"filter\":{\"bool\":{\"must\":[{\"range\":{\"#timestamp\":{\"gte\":1420099200000,\"lte\":1451635199999}}}],\"must_not\":[]}}}},\"highlight\":{\"pre_tags\":[\"#kibana-highlighted-field#\"],\"post_tags\":[\"#\/kibana-highlighted-field#\"],\"fields\":{\"*\":{}}},\"aggs\":{\"2\":{\"date_histogram\":{\"field\":\"#timestamp\",\"interval\":\"1w\",\"pre_zone\":\"-08:00\",\"pre_zone_adjust_large_interval\":true,\"min_doc_count\":0,\"extended_bounds\":{\"min\":1420099200000,\"max\":1451635199999}}}},\"fields\":[\"*\",\"_source\"],\"script_fields\":{},\"fielddata_fields\":[\"log.timestamp\",\"#timestamp\",\"val\"]}"
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d $DATA | jq '.hits.hits[0].fields."log.timestamp"[0]'
By using set -x I've found that the final curl command that is executed is:
curl -s -XPOST 'http://username:password#url/elasticsearch/index-*/_search' -d '{"size":500,"sort":{"#timestamp":"desc"},"query":{"filtered":{"query":{"query_string":{"analyze_wildcard":true,"query":"tx.traceId:' AIP1447283489-6 AND event: published_notification AND attempt: '1"}},"filter":{"bool":{"must":[{"range":{"#timestamp":{"gte":1420099200000,"lte":1451635199999}}}],"must_not":[]}}}},"highlight":{"pre_tags":["#kibana-highlighted-field#"],"post_tags":["#\/kibana-highlighted-field#"],"fields":{"*":{}}},"aggs":{"2":{"date_histogram":{"field":"#timestamp","interval":"1w","pre_zone":"-08:00","pre_zone_adjust_large_interval":true,"min_doc_count":0,"extended_bounds":{"min":1420099200000,"max":1451635199999}}}},"fields":["*","_source"],"script_fields":{},"fielddata_fields":["log.timestamp","#timestamp","val"]}'
And if you notice there are extra single quotes like so: ' added around the value of "query". As you can see here:
"query":"tx.traceId:' AIP1447283489-6 AND event: published_notification AND attempt: '1"
What the heck is going on and how do I use these two commands with this json in a script?
It's much easier to read from a here document than to ensure you've quoted all the quotation marks properly.
url='http://username:password#url/elasticsearch/index-*/_search'
curl -s -X POST "$url" -d#- <<EOF | jq '.hits.hits[0].fields."log.timestamp"[0]'
{ "size":500,
"sort": {
...
}
EOF
You didn't quote $DATA in your command
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d $DATA
so $DATA is subject to word splitting (and more) after parameter expansion is performed. Your $DATA contains whitespace in the value of query, so it's split there and broken into multiple arguments. What you want is
curl -s -XPOST http://username:password#url/elasticsearch/index-*/_search -d "$DATA"
Also, my personal advice is to quote JSON string with single quotes, and use '\'' when you need a literal single quote. Quoting JSON with double quotes in shell just makes the result impossible to read...

How do you POST an entire json string from a bash variable with curl?

I'm trying to POST data from a bash variable using curl, however, I'm unable to get this to work. This is the command that I'm using:
escape() { printf "%q" "$1"; }
curl -d "$(escape "$client")" -X POST -v https://$server/clients
The client variable look like this:
{"roles":["test"],"softwareName":"Some Soft","passwordSalt":"aaa","clientID":"full-client-2","contactPerson":"Test","contactPersonEmail":"a#b.org","description":"test","name":"Full Client-2","organization":"Some Org","passwordAlgorithm":"sha512","passwordHash":"bbb"}
And on the server I'm receiving the following:
{ '{"roles":': { '"test"': { '\"test\"\': '' } } }
I think its a problem with the escaping but I can't figure this out.
I've had a look at a number of other questions about this on here, but it seems most people need to insert variable into a literal that they are then trying to post. My problem is around using an entire variable as the json body. I've tried to use their answers to help me out but I haven't had any luck so far.
Don't try to quote it; use a here document:
curl -d#- -X POST -v https://"$server"/clients <<JSON
{"roles":["test"],"softwareName":"Some Soft","passwordSalt":"aaa","clientID":"full-client-2","contactPerson":"Test","contactPersonEmail":"a#b.org","description":"test","name":"Full Client-2","organization":"Some Org","passwordAlgorithm":"sha512","passwordHash":"bob"}
JSON
#- tells the -d option to look in standard input for the data, rather than using a hard-coded string.
If the text is in a variable, nothing more needs to be done; just quote the variable to prevent the shell from processing it:
curl -d "$client" -X POST -v https://"$server"/clients

Output bash variable with multiple lines to curl json

I'm trying to create a script that will use the Github API to post a comment containing the output of a command. This output has multiple lines.
Here's what I'm trying to do:
curl -H "Authorization: token oauthtoken" \
-H "Content-Type: application/json" \
-X POST -d#- \
https://api.github.com/repos/company/repo/issues/14/comments <<EOF
{
"body":"$OUTPUT"
}
EOF
How can I output the variable in such a way that it respects the multiple lines contained within? Now when I run that command, all of the newlines get squished on to one line.
I don't think that the basic cause of the problem are the newlines, the issues is that the value of $text is not properly formatted json.
Follow this simple example:
test="
Hello
World
"
curl -X POST -d '{"body": "'"$test"'"}' http://server.com/...
to see new lines working.
To make it possible to send the result of arbitrary commands using json, you need to json-encode the text before.