I am trying to create a JSON compatible output in bash that can be read by nodejs & python:
{"link":XX,"signal":YY,"noise":ZZ}
here's the unfiltered result:
iwconfig wlan0
wlan0 IEEE 802.11bg ESSID:"wifi#someplace" Nickname:"<WIFI#REALTEK>"
Mode:Managed Frequency:2.452 GHz Access Point: C8:4C:75:20:B4:8E
Bit Rate:54 Mb/s Sensitivity:0/0
Retry:off RTS thr:off Fragment thr:off
Encryption key:A022-1191-3A Security mode:open
Power Management:off
Link Quality=100/100 Signal level=67/100 Noise level=0/100
Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
Tx excessive retries:0 Invalid misc:0 Missed beacon:0
But after applying my filters:
iwconfig wlan0 | grep Link | tr -d '/100' | tr '=' ' ' | xargs | awk '{printf "{\"link\":"$3",\"signal\":"$6",\"noise\":"$9"}"}'
I am getting erratic and incomplete results:
{"link":98,"signal":6,"noise":}
{"link":Signal,"signal":Noise,"noise":}
The "noise" value is never captured, and sometimes printf returns the wrong chunk.
Is there a more 'reliable' way of doing this ?
The problem with the code in your question is here:
tr -d '/100'
What that command does it simply delete all the characters: '/', '1', '0'.
From the manual for tr,
-d, --delete
delete characters in SET1, do not translate
Thats not something you'd want. What you want is to replace the entire string /100 with "".
Use: sed 's/\/100//g' instead.
... | grep Link | sed 's/\/100//g' | tr '=' ' ' | awk '{printf "{\"link\":"$3",\"signal\":"$6",\"noise\":"$9"}"}'
You could restructure the output using perl, by piping the output through the following command:
perl -n -E 'if ($_ =~ qr{Link Quality=(\d+)/100.*?Signal level=(\d+)/100.*?Noise level=(\d+)/100}) { print qq({"link":$1,"signal":$2,"noise":$3}); }'
Using awk it is quite simple:
awk -F '[ =/]+' '$2=="Link"{printf "{\"%s\":%s,\"%s\":%s,\"%s\":%s}\n",
$2, $5, $6, $8, $10, $12}'
{"Link":100,"Signal":67,"Noise":0}
Related
so, I have the example code here:
#!/bin/bash
clear
curl -s https://www.cnbcindonesia.com/market-data/currencies/IDR=/USD-IDR |
html2text |
sed -n '/USD\/IDR/,$p' |
sed -n '/Last updated/q;p' |
tail -n-1 |
head -c+6 && printf "\n"
exit 0
this should print out some number range 14000~15000
lets start from the very basic one, what I have to do in order to print result + 1 ? so if the printout is 14000 and increment it to 1 become 14001. I suppose the result of the html2text is not calculatable since it should be something like string output not integer.
the more advance thing i want to know is how to calculate the result of 2 curl results?
What I would do, bash + xidel:
$ num=$(xidel -se '//div[#class="mark_val"]/span[1]/text()' 'https://url')
$ num=$((${num//,/}+1)) # num was 14050
$ echo $num
Output
14051
Explanations
$((...))
is an arithmetic substitution. After doing the arithmetic, the whole thing is replaced by the value of the expression. See http://mywiki.wooledge.org/ArithmeticExpression
Command Substitution: "$(cmd "foo bar")" causes the command 'cmd' to be executed with the argument 'foo bar' and "$(..)" will be replaced by the output. See http://mywiki.wooledge.org/BashFAQ/002 and http://mywiki.wooledge.org/CommandSubstitution
Bonus
You can compute directly in xidel, thanks Reino using xquery syntax :
$ xidel -s <url> e 'replace(//div[#class="mark_val"]/span[1],",","") + 1'
And to do addition arithmetic of 2 values :
$ xidel -s <url> -e '
let $num:=replace(//div[#class="mark_val"]/span[1],",","")
return $num + $num
'
I have a response trace file containing below response:
#RESPONSE BODY
#--------------------
{"totalItems":1,"member":[{"name":"name","title":"PatchedT","description":"My des_","id":"70EA96FB313349279EB089BA9DE2EC3B","type":"Product","modified":"2019 Jul 23 10:22:15","created":"2019 Jul 23 10:21:54",}]}
I need to fetch the value of the "id" key in a variable which I can put in my further code.
Expected result is
echo $id - should give me 70EA96FB313349279EB089BA9DE2EC3B value
With valid JSON (remove first to second row with sed and parse with jq):
id=$(sed '1,2d' file | jq -r '.member[]|.id')
Output to variable id:
70EA96FB313349279EB089BA9DE2EC3B
I would strongly suggest using jq to parse json.
But given that json is mostly compatible with python dictionaries and arrays, this HACK would work too:
$ cat resp
#RESPONSE BODY
#--------------------
{"totalItems":1,"member":[{"name":"name","title":"PatchedT","description":"My des_","id":"70EA96FB313349279EB089BA9DE2EC3B","type":"Product","modified":"2019 Jul 23 10:22:15","created":"2019 Jul 23 10:21:54",}]}
$ awk 'NR==3{print "a="$0;print "print a[\"member\"][0][\"id\"]"}' resp | python
70EA96FB313349279EB089BA9DE2EC3B
$ sed -n '3s|.*|a=\0\nprint a["member"][0]["id"]|p' resp | python
70EA96FB313349279EB089BA9DE2EC3B
Note that this code is
1. dirty hack, because your system does not have the right tool - jq
2. susceptible to shell injection attacks. Hence use it ONLY IF you trust the response received from your service.
Quick and dirty (don't use eval):
eval $(cat response_file | tail -1 | awk -F , '{ print $5 }' | sed -e 's/"//g' -e 's/:/=/')
It is based on the exact structure you gave, and hoping there is no , in any value before "id".
Or assign it yourself:
id=$(cat response_file | tail -1 | awk -F , '{ print $5 }' | cut -d: -f2 | sed -e 's/"//g')
Note that you can't access the name field with that trick, as it is the first item of the member array and will be "swallowed" by the { print $2 }. You can use an even-uglier hack to retrieve it though:
id=$(cat response_file | tail -1 | sed -e 's/:\[/,/g' -e 's/}\]//g' | awk -F , '{ print $5 }' | cut -d: -f2 | sed -e 's/"//g')
But, if you can, jq is the right tool for that work instead of ugly hacks like that (but if it works...).
When you can't use jq, you can consider
id=$(grep -Eo "[0-9A-F]{32}" file)
This is only working when the file looks like what I expect, so you might need to add extra checks like
id=$(grep "My des_" file | grep -Eo "[0-9A-F]{32}" | head -1)
I have a JSON file and content like below:
[
{
"id":"54545-f919-4b0f-930c-0117d6e6c987",
"name":"Inventory_Groups",
"path":"/Groups",
"subGroups":[
{
"id":"343534-394b-429a-834e-f8774240d736",
"name":"UserGroup",
"path":"/Groups/UserGroup",
"subGroups":[
]
}
]
}
]
Now I want to grep value of key id from the subGroups area. How to achive this, if id key not duplicate then it can be achieved by:
grep -o '"id": "[^"]*' Group.json | grep -o '[^"]*$'
But in my case how can I get the value of id as it appears two times?
A valid question to ask your employer is why you're in a position to use the shell but not to use appropriate linux packages. Compare:
awk -F '[":,]+' '$2=="subGroups" {f=1} f && $2=="id" {print $3; exit}' file
(Brittle solution, will fail if the structure of your JSON changes)
To:
jq '.[].subGroups[].id' file
Which can handle compact JSON in addition to numerous other realistic complications.
Using just standard UNIX tools and assuming your sed can tolerate input without a terminating newline (otherwise we can swap out the tr for an awk command that keeps the last newline):
$ tr -d '\n' < file | sed 's/.*"subGroups":[^]}]*"id":"\([^"]*\)\".*/\1\n/'
343534-394b-429a-834e-f8774240d736
Alternatively with just a call to any awk:
$ awk '
{ rec = (NR>1 ? rec ORS : "") $0 }
END {
gsub(/.*"subGroups":[^]}]*"id":"|".*/,"",rec)
print rec
}
' file
343534-394b-429a-834e-f8774240d736
I run the curl command $(curl -i -o - --silent -X GET --cert "${CERT}" --key "${KEY}" "$some_url") and save the response in the variable response. ${response} is as shown below
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 34
Connection: keep-alive
Keep-Alive: timeout=5
X-XSS-Protection: 1;
{"status":"running","details":"0"}
I want to parse the JSON {"status":"running","details":"0"} and assign 'running' and 'details' to two different variables where I can print status and details both. Also if the status is equal to error, the script should exit. I am doing the following to achieve the task -
status1=$(echo "${response}" | awk '/^{.*}$/' | jq -r '.status')
details1=$(echo "${response}" | awk '/^{.*}$/' | jq -r '.details')
echo "Status: ${status1}"
echo "Details: ${details1}"
if [[ $status1 == 'error' ]]; then
exit 1
fi
Instead of parsing the JSON twice, I want to do it only once. Hence I want to combine the following lines but still assign the status and details to two separate variables -
status1=$(echo "${response}" | awk '/^{.*}$/' | jq -r '.status')
details1=$(echo "${response}" | awk '/^{.*}$/' | jq -r '.details')
First, stop using the -i argument to curl. That takes away the need for awk (or any other pruning of the header after-the-fact).
Second:
{
IFS= read -r -d '' status1
IFS= read -r -d '' details1
} < <(jq -r '.status + "\u0000" + .details + "\u0000"' <<<"$response")
The advantage of using a NUL as a delimiter is that it's the sole character that can't be present in the value of a C-style string (which is how shell variables' values are stored).
You can use a construction like:
read status1 details1 < <(jq -r '.status + " " + .details' <<< "${response}")
You use read to assign the different inputs to two variables (or an array, if you want), and use jq to print the data you need separated by whitespace.
As Benjamin already suggested, only retrieving the json is a better way to go. Poshi's solution is solid.
However, if you're looking for the most compact to do this, no need to save the response as a variable if the only thing your're going to do with it is extract other variables from it on a one time basis. Just pipe curl directly into:
curl "whatever" | jq -r '[.status, .details] |#tsv'
or
curl "whatever" | jq -r '[.status, .details] |join("\t")'
and you'll get your values fielded for you.
hi i am writings a small shell script. there i use curl command to call to api. what it return is a status of a scan.
{"status":"14"}
i want to get this status and check if it is less than 100; this is what i have done so far
a=0
while [ $a -lt 100 ]
do
curlout=$(curl "http://localhost:9090/JSON/spider/view/status/?zapapiformat=JSON&scanId=0");
echo "$curlout";
a=`expr $a + 1`
done
what i want to do is assign that status to $a; how to get read this json to get the value in shell script
If you need to work with JSON, you should obtain jq:
$ echo '{"status": "14"}' | jq '.status|tonumber'
14
or, less rigorously:
$ echo '{"status": "14"}' | jq -r '.status'
14
If you're sure about the format of the curl output, then it's very simple.
echo "$curlout" | tr -cd '[:digit:]'
From manpage of tr,
-c, -C, --complement
use the complement of SET1
-d, --delete
delete characters in SET1, do not translate
[:digit:]
all digits
So this command removes all characters other than digits.