I have the json as below, i need to get only the mail from the above json in bash script
value={"count":5,"users":[{"username":"asa","name":"asa
Tran","mail":"asa#xyz.com"},{"username":"qq","name":"qq
Morris","mail":"qq#xyz.com"},{"username":"qwe","name":"qwe
Org","mail":"qwe#xyz.com"}]}
Output can be as
mail=asa#xyz.com,qq#xyz.com,qwe#xyz.com
All the above need to be done in the bash script (.sh)
I have already tried with the array iteration as but of no use
for key in "${!value[#]}"
do
#echo "key = $key"
echo "value = ${value[$key]}"
done
Even i have tried with the array conversion as
alias json-decode="php -r
'print_r(json_decode(file_get_contents(\"php://stdin\"),1));'"
value=$(curl --user $credentials -k $endPoint | json-decode)
Still i was not able to get the specific output.
jq is the tool to iterate through a json. In your case:
while read user; do
jq -r '.mail' <<< $user
done <<< $(jq -c '.users[]' users.json)
would give:
asa#xyz.com
qq#xyz.com
qwe#xyz.com
NOTE: I removed "value=" because that is not valid json. Users.json contains:
{"count":5,"users":[{"username":"asa","name":"asa Tran","mail":"asa#xyz.com"},{"username":"qq","name":"qq Morris","mail":"qq#xyz.com"},{"username":"qwe","name":"qwe Org","mail":"qwe#xyz.com"}]}
If this is valid json and the email field is the only one containing a # character, you can do something like this:
echo $value | tr '"' '\n' | grep #
It replaces double-quotes by new line character and only keeps lines containing #. It is really not json parsing, but it works.
You can store the result in a bash array
emails=($(echo $value | tr '"' '\n' | grep #))
and iterate on them
for email in ${emails[#]}
do
echo $email
done
You should use json_pp tool (in debian, it is part of the libjson-pp-perl package)
One would use it like this :
cat file.json | json_pp
And get a pretty print for your json.
So in your case, you could do :
#!/bin/bash
MAILS=""
LINES=`cat test.json | json_pp | grep '"mail"' | sed 's/.* : "\(.*\)".*/\1/'`
for LINE in $LINES ; do
MAILS="$LINE,$MAILS"
done
echo $MAILS | sed 's/.$//'
Output :
qwe#xyz.com,qq#xyz.com,asa#xyz.com
Using standard unix toolbox : sed command
cat so.json | sed "s/},/\n/g" | sed 's/.*"mail":"\([^"]*\)".*/\1/'
With R you could do this as follows:
$ value={"count":5,"users":[{"username":"asa","name":"asa Tran","mail":"asa#xyz.com"},{"username":"qq","name":"qq Morris","mail":"qq#xyz.com"},{"username":"qwe","name":"qwe Org","mail":"qwe#xyz.com"}]}
$ echo $value | R path users | R map path mail
["asa#xyz.com", "qq#xyz.com", "qwe#gyz.com"]
Related
I have a long list of JSON data, with repeats of contents similar to followings.
Due to the original JSON file is too long, I will just shared the hyperlinks here. This is a result generated from a database called RegulomeDB.
Direct link to the JSON file
I would like to extract specific data (eQTLs) from "method": "eQTLs" and "value": "xxxx", and put them into 2 columns (tab delimited) exactly like below.
Note: "value":"xxxx" is extracted right after "method": "eQTLs"is detected.
eQTLs firstResult, secondResult, thirdResult, ...
In this example, the desired output is:
eQTLs EIF3S8, EIF3CL
I've tried using a python script but was unsuccessful.
import json
with open('file.json') as f:
f_json = json.load(f)
print 'f_json[0]['"method": "eQTLs"'] + "\t" + f_json[0]["value"]
Thank you for your kind help.
Maybe you'll find the JSON-parser xidel useful. It can open urls and can manipulate strings any way you want:
$ xidel -s "https://regulomedb.org/regulome-search/?regions=chr16:28539847-28539848&genome=GRCh37&format=json" \
-e '"eQTLs "||join($json("#graph")()[method="eQTLs"]/value,", ")'
eQTLs EIF3S8, EIF3CL
Or with the XPath/XQuery 3.1 syntax:
-e '"eQTLs "||join($json?"#graph"?*[method="eQTLs"]?value,", ")'
Try this:
cat file.json | grep -iE '"method":\s*"eQTLs"[^}]*' -o | cut -d ',' -f 1,5 | sed -r 's/"|:|method|value//gi' | sed 's/\s*eqtls,\s*//gi' | tr '\n' ',' | sed 's/,$/\n/g' | sed 's/,/, /g' | xargs echo -e 'eQTLs\x09'
I have a file with this kind of text with pattern
[{"foo":"bar:baz:foo*","bar*":"baz*","etc":"etc"},
{"foo2":"bar2:baz2:foo2*","bar2*":"baz2*","etc":"etc"},
{"foo3":"bar3:baz3:foo3*","bar3*":"baz3*","etc":"etc"},
{"foo4":"bar4:baz4:foo4*","bar4*":"baz4*","etc":"etc"}]
I need to take every string like this
{"foo":"bar:baz:foo*","bar*":"baz*","etc":"etc"} and send each of them to some url via curl
for i in text.txt
do (awk,sed,grep etc)
then curl $string
I can't figure out how to get the desired lines properly from the file without unnecessary symbols
I suggest that you can use jq to process your json file. jq is capable of reading json, and formatting output. Here's an example jq script to process your json file (which I unimaginatively call 'jsonfile'):
jq -r '.[] | "curl -d '\'' \(.) '\'' http://restful.com/api " ' jsonfile
Here's the output:
curl -d ' {"foo":"bar:baz:foo*","bar*":"baz*","etc":"etc"} ' http://restful.com/api
curl -d ' {"foo2":"bar2:baz2:foo2*","bar2*":"baz2*","etc":"etc"} ' http://restful.com/api
curl -d ' {"foo3":"bar3:baz3:foo3*","bar3*":"baz3*","etc":"etc"} ' http://restful.com/api
curl -d ' {"foo4":"bar4:baz4:foo4*","bar4*":"baz4*","etc":"etc"} ' http://restful.com/api
Here's what's going on:
We pass three arguments to the jq program: jq -r <script> <inputfile>.
The -r tells jq to output the results in raw format (that is, please don't escape quotes and stuff).
The script looks like this:
.[] | "some string \(.)"
The first . means take the whole json structure and the [] means iterate through each array element in the structure. The | is a filter that processes each element in the array. The filter is to output a string. We are using \(.) to interpolate the whole element passed into the | filter.
Wow... I've never really explained a jq script before (and it shows). But the crux of it is, we are using jq to find each element in the json array and insert it into a string. Our string is this:
curl -d '<the json dictionary array element>' http://restful.com/api
Ok. And you see the output. It works. But wait a second, we only have output. Let's tell the shell to run each line like this:
jq -r '.[] | "curl -d '\'' \(.) '\'' http://restful.com/api " ' jsonfile | bash
By piping the output to bash, we execute each line that we output. Essentially, we are writing a bash script with jq to curl http://restful.com/api passing the json element as the -d data parameter to POST the json element.
Revisiting for single quote issue
#oguz ismail pointed out that bash will explode if there is a single quote in the json input file. This is true. We can avoid the quote by escaping, but we gain more complexity - making this a non-ideal approach.
Here's the problem input (I just inserted a single quote):
[{"foo":"bar:'baz:foo*","bar*":"baz*","etc":"etc"},
{"foo2":"bar2:baz2:foo2*","bar2*":"baz2*","etc":"etc"},
{"foo3":"bar3:baz3:foo3*","bar3*":"baz3*","etc":"etc"},
{"foo4":"bar4:baz4:foo4*","bar4*":"baz4*","etc":"etc"}]
Notice above that baz is now 'baz. The problem is that a single single quote makes the bash shell complain about unmatched quotes:
$ jq -r '.[] | "curl -d '\'' \(.) '\'' http://restful.com/api " ' jsonfile | bash
bash: line 4: unexpected EOF while looking for matching `"'
bash: line 5: syntax error: unexpected end of file
Here's the solution:
$ jq -r $'.[] | "\(.)" | gsub( "\'" ; "\\\\\'" ) | "echo $\'\(.)\'" ' jsonfile | bash
{"foo":"bar'baz:foo*","bar*":"baz*","etc":"etc"}
{"foo2":"bar2:baz2:foo2*","bar2*":"baz2*","etc":"etc"}
{"foo3":"bar3:baz3:foo3*","bar3*":"baz3*","etc":"etc"}
{"foo4":"bar4:baz4:foo4*","bar4*":"baz4*","etc":"etc"}
Above I am using $'' to quote the jq script. This allows me to escape single quotes using '. I've also changed the curl command to echo so I can test the bash script without bothering the folks at http://restful.com/api.
The 'trick' is to make sure that the bash script we generate also escapes all single quotes with a backslash . So, we have to change ' to \'. That's what gsub is doing.
gsub( "\'" ; "\\\\\'" )
After making that substitution ( ' --> \' ) we pipe the entire string to this:
"echo $\'\(.)\'"
which surrounds the output of gsub with echo $''. Now we are using $' again so the \' is properly understood by bash.
So we wind up with this when we put the curl back in:
jq -r $'.[] | "\(.)" | gsub( "\'" ; "\\\\\'" ) | "curl -d $\'\(.)\' http://restful.com/api " ' jsonfile | bash
Use jq command. This is just example parsing.
for k in $(jq -c '.[]' a.txt); do
echo "hello-" $k
done
Output:
hello- {"foo":"bar:baz:foo*","bar*":"baz*","etc":"etc"}
hello- {"foo2":"bar2:baz2:foo2*","bar2*":"baz2*","etc":"etc"}
hello- {"foo3":"bar3:baz3:foo3*","bar3*":"baz3*","etc":"etc"}
hello- {"foo4":"bar4:baz4:foo4*","bar4*":"baz4*","etc":"etc"}
You can use the $k anywhere inside the loop you want.
for k in $(jq -c '.[]' a.txt); do
curl -d "$k" <url>
done
I have a response trace file containing below response:
#RESPONSE BODY
#--------------------
{"totalItems":1,"member":[{"name":"name","title":"PatchedT","description":"My des_","id":"70EA96FB313349279EB089BA9DE2EC3B","type":"Product","modified":"2019 Jul 23 10:22:15","created":"2019 Jul 23 10:21:54",}]}
I need to fetch the value of the "id" key in a variable which I can put in my further code.
Expected result is
echo $id - should give me 70EA96FB313349279EB089BA9DE2EC3B value
With valid JSON (remove first to second row with sed and parse with jq):
id=$(sed '1,2d' file | jq -r '.member[]|.id')
Output to variable id:
70EA96FB313349279EB089BA9DE2EC3B
I would strongly suggest using jq to parse json.
But given that json is mostly compatible with python dictionaries and arrays, this HACK would work too:
$ cat resp
#RESPONSE BODY
#--------------------
{"totalItems":1,"member":[{"name":"name","title":"PatchedT","description":"My des_","id":"70EA96FB313349279EB089BA9DE2EC3B","type":"Product","modified":"2019 Jul 23 10:22:15","created":"2019 Jul 23 10:21:54",}]}
$ awk 'NR==3{print "a="$0;print "print a[\"member\"][0][\"id\"]"}' resp | python
70EA96FB313349279EB089BA9DE2EC3B
$ sed -n '3s|.*|a=\0\nprint a["member"][0]["id"]|p' resp | python
70EA96FB313349279EB089BA9DE2EC3B
Note that this code is
1. dirty hack, because your system does not have the right tool - jq
2. susceptible to shell injection attacks. Hence use it ONLY IF you trust the response received from your service.
Quick and dirty (don't use eval):
eval $(cat response_file | tail -1 | awk -F , '{ print $5 }' | sed -e 's/"//g' -e 's/:/=/')
It is based on the exact structure you gave, and hoping there is no , in any value before "id".
Or assign it yourself:
id=$(cat response_file | tail -1 | awk -F , '{ print $5 }' | cut -d: -f2 | sed -e 's/"//g')
Note that you can't access the name field with that trick, as it is the first item of the member array and will be "swallowed" by the { print $2 }. You can use an even-uglier hack to retrieve it though:
id=$(cat response_file | tail -1 | sed -e 's/:\[/,/g' -e 's/}\]//g' | awk -F , '{ print $5 }' | cut -d: -f2 | sed -e 's/"//g')
But, if you can, jq is the right tool for that work instead of ugly hacks like that (but if it works...).
When you can't use jq, you can consider
id=$(grep -Eo "[0-9A-F]{32}" file)
This is only working when the file looks like what I expect, so you might need to add extra checks like
id=$(grep "My des_" file | grep -Eo "[0-9A-F]{32}" | head -1)
I have a sample json response as shown below which i am trying to parse using jq in shell script.[{"id":1,"notes":"Demo1\nDemo2"}]
This is the command through which I am trying to access notes in the shell script.
value=($(curl $URL | jq -r '.[].notes'))
When I echo "$value" I only get Demo1. How to get the exact value: Demo1\nDemo2 ?
To clarify, there is no backslash or n in the notes field. \n is JSON's way of encoding a literal linefeed, so the value you should be expecting is:
Demo1
Demo2
The issue you're seeing is because you have split the value on whitespace and created an array. Each value can be accessed by index:
$ cat myscript
data='[{"id":1,"notes":"Demo1\nDemo2"}]'
value=($(printf '%s' "$data" | jq -r '.[].notes'))
echo "The first value was ${value[0]} and the second ${value[1]}"
$ bash myscript
The first value was Demo1 and the second Demo2
To instead get it as a simple string, remove the parens from value=(..):
$ cat myscript2
data='[{"id":1,"notes":"Demo1\nDemo2"}]'
value=$(printf '%s' "$data" | jq -r '.[].notes')
echo "$value"
$ bash myscript2
Demo1
Demo2
I have a script that uses jq for parsing a json string MESSAGE (that is read from another application). Meanwhile the json has changed and a field is split in 2 fields: file_path is now split into folder and file. The script was reading the file_path, now the folder may not be present, so for creating the path of the file I have to verify if the field is there. I have search for a while on the internet, and manage to do:
echo $(echo $MESSAGE | jq .folder -r)$'/'$(echo $MESSAGE | jq .file -r)
if [ $MESSAGE | jq 'has(".folder")' -r ]
then
echo $(echo $MESSAGE | jq .folder -r)$'/'$(echo $MESSAGE | jq .file -r)
else
echo $(echo $MESSAGE | jq .file -r)
fi
where MESSAGE='{"folder":"FLDR","file":"fl"}' or MESSAGE='{"file":"fl"}'
The first line is printing FLDR/fl or null/fl if the folder field is not present. So I have thought to create an if that is verifying if the folder field is present or not, but it seems that I am doing it wrong and cannot figure out what is wrong. The output is
bash: [: missing `]'
jq: ]: No such file or directory
null/fl
I'd do the whole thing in a jq filter:
echo "$MESSAGE" | jq -r '[ .folder, .file ] | join("/")'
In the event that you want to do it with bash (or to learn how to do this sort of thing in bash), two points:
Shell variables should almost always be quoted when they are used (i.e., "$MESSAGE" instead of $MESSAGE). You will run into funny problems if one of the strings in your JSON ever contains a shell metacharacter (such as *) and you forgot to do that; the string will be subject to shell expansion (and that * will be expanded into a list of files in the current working directory).
A shell if accepts as condition a command, and the decision where to branch is made depending on the exit status of that command (true if the exit status is 0, false otherwise). The [ you attempted to use is just a command (an alias for test, see man test) and not special in any way.
So, the goal is to construct a command that exits with 0 if the JSON object has a folder property, non-zero otherwise. jq has a -e option that makes it return 0 if the last output value was not false or null and non-zero otherwise, so we can write
if echo "$MESSAGE" | jq -e 'has("folder")' > /dev/null; then
echo "$MESSAGE" | jq -r '.folder + "/" + .file'
else
echo "$MESSAGE" | jq -r .file
fi
The > /dev/null bit redirects the output from jq to /dev/null (where it is ignored) so that we don't see it on the console.