i am writing personal parser for json on bash. So, i need to make grep of line. For example, part of my json-file.
{
"host": "127.0.0.1",
"hhost": "127.0.0.2",
}
Of course, after trying cat json.txt | grep host, i received:
"host": "127.0.0.1",
"hhost": "127.0.0.1",
I found, how to find only host without hhost. I used
cat json.txt | grep '"host"'
Everything is good
"host": "127.0.0.1",
But i want to use it in bash script:
#!/bin/bash
#in a future, i want to read variable var from reading from console
var=host
search='"$var"'
echo $search
In a result, i have:
"$var"
What i did wrong ? Can you advice me please ?
You could use jq, to parse the JSON file for this you will need first a proper valid json
{
"host": "127.0.0.1",
"hhost": "127.0.0.2"
}
Then you could just do something like:
#/bin/sh
HOST=$(jq '.host' data.json)
echo $HOST
...
if you put a variable inside single quotes in bash, I does not get evaluated, you should just use:
search="$var"
if you want the variable to be enclosed by quotes, just escape them like this:
search="\"$var\""
Related
I have a json file test.json with the content:
[
{
"name": "Akshay",
"id": "234"
},
{
"name": "Amit",
"id": "28"
}
]
I have a shell script with content:
#!/bin/bash
function display
{
echo "name is $1 and id is $2"
}
cat test.json | jq '.[].name,.[].id' | while read line; do display $line; done
I want name and id of a single item to be passed together as arguments to the function display but the output is something like this :
name is "Akshay" and id is
name is "Amit" and id is
name is "234" and id is
name is "28" and id is
What should be the correct way to implement the code?
PS: I specifically want to use jq so please base the answer in terms of jq
Two major issues, and some additional items that may not matter for your current example use case but can be important when you're dealing with real-world data from untrusted sources:
Your current code iterates over all names before writing any ids.
Your current code uses newline separators, but doesn't make any effort to read multiple lines into each while loop iteration.
Your code uses newline separators, but newlines can be present inside strings; consequently, this is constraining the input domain.
When you pipe into a while loop, that loop is run in a subshell; when the pipeline exits, the subshell does too, so any variables set by the loop are lost.
Starting up a copy of /bin/cat and making jq read a pipe from its output is silly and inefficient compared to letting jq read from test.json directly.
We can fix all of those:
To write names and ids in pairs, you'd want something more like jq '.[] | (.name, .id)'
To read both a name and an id for each element of the loop, you'd want while IFS= read -r name && IFS= read -r id; do ... to iterate over those pairs.
To switch from newlines to NULs (the NUL being the only character that can't exist in a C string, or thus a bash string), you'd want to use the -j argument to jq, and then add explicit "\u0000" elements to the content being written. To read this NUL-delimited content on the bash side, you'd need to add the -d '' argument to each read.
To move the while read loop out of the subshell, we can use process substitution, as described in BashFAQ #24.
To let jq read directly from test.json, use either <test.json to have the shell connect the file directly to jq's stdin, or pass the filename on jq's command line.
Doing everything described above in a manner robust against input data containing JSON-encoded NULs would look like the following:
#!/bin/bash
display() {
echo "name is $1 and id is $2"
}
cat >test.json <<'EOF'
[
{ "name": "Akshay", "id": "234" },
{ "name": "Amit", "id": "28" }
]
EOF
while IFS= read -r -d '' name && IFS= read -r -d '' id; do
display "$name" "$id"
done < <(jq -j '
def stripnuls: sub("\u0000"; "<NUL>");
.[] | ((.name | stripnuls), "\u0000", (.id | stripnuls), "\u0000")
' <test.json)
You can see the above running at https://replit.com/#CharlesDuffy2/BelovedForestgreenUnits#main.sh
You can use string interpolation.
jq '.[] | "The name is \(.name) and id \(.id)"'
Result:
"The name is Akshay and id 234"
"The name is Amit and id 28"
"The name is hi and id 28"
If you want to get rid of the double-quotes from each object, then:
jq --raw-output '.[] | "The name is \(.name) and is \(.id)"'
https://jqplay.org/s/-lkpHROTBk0
This question already has answers here:
Parsing JSON with Unix tools
(45 answers)
Closed 5 years ago.
I am writing a shell script to run some api's. It return response fine but i need some specific parameter to grep from the response and want to save in file.
My script look like
#!/bin/sh
response=$(curl 'https://example.com' -H 'Content-Type: application/json' )
echo "$response"
reponse is something like
{
status:"success",
response:{
"target":"",
"content":"test content"
}
}
Response is fine and i am able to write whole response in file but My requirement is to save only "content" inside "response" object using the script. which i need for another api.
Note: I cannot change api responses as I am working third party api's;
Thank you
If the output is proper JSON:
$ cat proper.json
{
"status": "success",
"response": {
"target": "",
"content": "test content"
}
}
$ response=$(cat proper.json)
You could use jq:
$ echo $response | jq -r '.response.content'
test content
You can grep for the content and then use awk to split by : and take only the value, not the key
grep "\"content\":" | awk -F":" '{ print $2}'
Will print "test content"
You can do this to get the value of contents into a variable ($content).
content=$(echo "$response" | cut -d'"' -f 7)
Explanation - Split the $response using " (double quote) as the delimiter and use the 7th field of the output (i.e the value (test content) of content in the json response)
Here is an excerpt from the description and usage of the cut command
if you like to extract a whole field, you can combine option -f and -d. The option -f specifies which field you want to extract, and the option -d specifies what is the field delimiter that is used in the input file.
I'm using curl to send JSON to an API endpoint. However, somewhere in the bash chain it is getting messed up.
Is there something special to know about encoding with curl?
If I construct the payload like this:
PAYLOAD='payload={"channel": "github", "username": "webhookbot", "icon_emoji": ":ghost:", "text": "'
PAYLOAD+=$1
PAYLOAD+=' " }'
echo $PAYLOAD
curl -X POST --data-urlencode "$PAYLOAD" $SLACKPOSTURL
echo "sent"
I'll get back an error
Payload was not valid JSONsent
however if i just hardwire to assign a variable with the output
PAYLOAD='payload={"channel": "github", "username": "webhookbot", "icon_emoji": ":ghost:", "text": "LAST_COMMIT Merge pull request #558 from dcsan/boteditor Boteditor " }'
then it will go through fine.
Is there something that a simple assignment is doing differently vs. concatenating strings? In the console the output looks identical.
FWIW some messages go through but content like this:
LAST_COMMIT Merge pull request #558 from dcsan/boteditor Boteditor
will only go through if hardcoded in. so its not the other end afaican see, its something to do with the way messages are built.
I guess you want to concatenate values into your variable. But += is not the way to do so.
To concatenate strings in a variable you need to say:
PAYLOAD="$PAYLOAD $1"
All together it would be something like the following. Note the need to use " so that the variable $PAYLOAD is expanded and the usage of \" to store a literal double quote:
PAYLOAD='payload={"channel": "github", "username": "webhookbot", "icon_emoji": ":ghost:", "text": "'
PAYLOAD="$PAYLOAD $1 \" }"
echo "$PAYLOAD"
curl -X POST --data-urlencode "$PAYLOAD" $SLACKPOSTURL
echo "sent"
This is what worked from me from a bash script:
curl -X POST --data-urlencode "payload={\"text\": \"$2\"}" https://hooks.slack.com/services/$KEY
Notice the inner quotes are escaped, but the outer quotes are not.
FYI, adding:
set -x
at the beginning of a bash script will show you the actual commands being executed, and save a lot of guesswork.
I have a build tool which is creating a versions.json file injected with a json format string.
Initially I was thinking of just injecting the json via an echo, something like below.
json = {"commit_id": "b8f2b8b", "environment": "test", "tags_at_commit": "sometags", "project": "someproject", "current_date": "09/10/2014", "version": "someversion"}
echo -e json > versions.jso
However the echo seems to escape out all of the quote marks so my file will end up something like this:
{commit_id: b8f2b8b, environment: test, tags_at_commit: somereleasetags, project: someproject, current_date: 09/10/2014, version: someproject}
This unfortunately is not valid JSON.
To preserve double quotes you need to surround your variable in single quotes, like so:
json='{"commit_id": "b8f2b8b", "environment": "test", "tags_at_commit": "sometags", "project": "someproject", "current_date": "09/10/2014", "version": "someversion"}'
echo "$json" > versions.json
Take into account that this method will not display variables correctly, but instead print the literal $variable.
If you need to print variables, use the cat << EOF construct, which utilizes the Here Document redirection built into Bash. See man bash and search for "here document" for more information.
Example:
commit="b8f2b8b"
environment="test"
...etc
cat << EOF > /versions.json
{"commit_id": "$commit", "environment": "$environment", "tags_at_commit": "$tags", "project": "$project", "current_date": "$date", "version": "$version"}
EOF
If you're looking for a more advanced json processing tool that works very well with bash, I'd recommend jq
If you want variables in between you can quote does this '"'$variable'"'. Below is the date example.
echo {'"date"' : '"'$(date +"%d_%m_%Y")'"'} > cron_checkpoint.json
I have a shell script with curl -s http://ifconfig.me/all.json command which prints below output in the terminal window.
{
"version" : {
"ip_addr": "201.73.103.12",
"lang": "java",
"remote_host": "OpenSSL/0.9.8w zlib/1.2.3 libidn/1.23 libssh2/1.2.2",",
"user_agent": "curl/7.23.1 (i386-sun-solaris2.11) libcurl/7.23.1
"charset": "",
"port": "63713"}
}
I need to display the JSON value in table format.
Someone, please help me with this to implement in UNIX shell script. Thanks!
Look into the cool command-line JSON query parser / formatter tool:
http://stedolan.github.io/jq/
You can pipe your JSON input into this tool and extract out / format the keys you need.
Or, just run multiple curl -s http://ifconfigme/ calls for each "row/column" you need and output it in some sane format, E.g.:
#!/bin/bash
ip=`curl -s http://ifconfig.me/ip`
host=`curl -s http://ifconfig.me/host`
echo -e "ip\thost"
echo -e "${ip}\t${host}"