I need to get the "snapshot" value at the top of the file from this url: https://s3.amazonaws.com/Minecraft.Download/versions/versions.json
So I should get a variable that contains "14w08a" when I run the command to parse the json.
This will do the trick
$ curl -s "$url" | grep -Pom 1 '"snapshot": "\K[^"]*'
14w08a
best thing to do is use a tool with a JSON parser. For example:
value=$(
curl -s "$url" |
ruby -rjson -e 'data = JSON.parse(STDIN.read); puts data["latest"]["snapshot"]'
)
Related
I am performing a curl test using shell script where I need to save curl server response into JSON along with CURL Information like time_connect, http_code, etc. I am trying the following code to write output to a JSON
HOST_ADDR="http://$mynds/"
do_request(){
echo $(curl $HOST_ADDR --silent -w "\n{\n
\"HttpCode\": %{http_code},\n
\"NumRedirects\":%{num_redirects},\n
\"NumConnects\":%{num_connects},\n
\"SizeDownloadInBytes\":%{size_download},\n
\"SizeHeaderInBytes\":%{size_header},\n
\"SizeRequestInBytes\":%{size_request},\n
\"SizeUploadInBytes\":%{size_upload},\n
\"SpeedUploadBPS\":%{speed_upload},\n
\"SpeedDownloadBPS\":%{speed_download},\n
\"TimeAppConnectSec\":%{time_appconnect},\n
\"TimeConnectSec\":%{time_connect},\n
\"TimeNamelookupSec\":%{time_namelookup},\n
\"TimePreTransferSec\":%{time_pretransfer},\n
\"TimeRedirectSec\":%{time_redirect},\n
\"TimeStartTransferSec\":%{time_starttransfer},\n
\"TimeTotalSec\":%{time_total},\n
\"UrlEffective\":\"%{url_effective}\"
}" -s)
}
do_request
The simple output I am getting:
{"hostIPAddr":"0.0.0.0","hostname":"vm01","text":"Hello World from
vm01"}` and `{ "HttpCode": 200, "NumRedirects":0, "NumConnects":1,
"SizeDownloadInBytes":85, "SizeHeaderInBytes":263,
"SizeRequestInBytes":99, "SizeUploadInBytes":0,
"SpeedUploadBPS":0.000, "SpeedDownloadBPS":14.000,
"TimeAppConnectSec":0.000000, "TimeConnectSec":5.553587,
"TimeNamelookupSec":5.097553, "TimePreTransferSec":5.553868,
"TimeRedirectSec":0.000000, "TimeStartTransferSec":5.827584,
"TimeTotalSec":5.827704, "UrlEffective":"http://dns" }
I am getting two JSON outputs one for curl information and one from the server. How do I combine these two outputs into a single JSON variable? Please help.
guid_id=$(uuidgen)
file_1="curlJsonRes_$guid_id.json"
file_2="curlMetaRes_$guid_id.json"
do_request(){
echo $(curl $HOST_ADDR --silent --output $file_1 -w "\n{\n
\"HttpCode\": %{http_code},\n
\"NumRedirects\":%{num_redirects},\n
\"NumConnects\":%{num_connects},\n
}")
} > $file_2
do_request
#Merging file outputs
echo $(jq -s '.[0] * .[1]' $file_1 $file_2) > $RESULTS_FILE
rm $file_1
rm $file_2
Am trying below curl request:
curl -s https://www.instagram.com/p/B_Xv0kgJUNq/?igshid=2zc9h1kyhkyv | grep commentCount
How to get the value number of "commentCount":"26508" in the curl response.
I've tried above command it getting me the full JSON <script type="application/ld+json"> .. while I just need the count of "commentCount" value.
An HTML and JSON parser like xidel would fit the job:
$ xidel -s "https://www.instagram.com/p/B_Xv0kgJUNq/?igshid=2zc9h1kyhkyv" -e '
parse-json(
//script[#type="application/ld+json"]
)/commentCount
'
with Linux grep you can
$ curl -s https://www.instagram.com/p/B_Xv0kgJUNq/?igshid=2zc9h1kyhkyv | grep -Po 'commentCount":"\K\d*'
99795
I am trying to get a list of URL after redirection using bash scripting. Say, google.com gets redirected to http://www.google.com with 301 status.
What I have tried is:
json='[{"url":"google.com"},{"url":"microsoft.com"}]'
echo "$json" | jq -r '.[].url' | while read line; do
curl -LSs -o /dev/null -w %{url_effective} $line 2>/dev/null
done
So, is it possible for us to use commands like curl inside jq for processing JSON objects.
I want to add the resulting URL to existing JSON structure like:
[
{
"url": "google.com",
"redirection": "http://www.google.com"
},
{
"url": "microsoft.com",
"redirection": "https://www.microsoft.com"
}
]
Thank you in advance..!
curl is capable of making multiple transfers in a single process, and it can also read command line arguments from a file or stdin, so, you don't need a loop at all, just put that JSON into a file and run this:
jq -r '"-o /dev/null\nurl = \(.[].url)"' file |
curl -sSLK- -w'%{url_effective}\n' |
jq -R 'fromjson | map(. + {redirection: input})' file -
This way only 3 processes will be spawned for the whole task, instead of n + 2 where n is the number of URLs.
I would generate a dictionary with jq per url and slurp those dictionaries into the final list with jq -s:
json='[{"url":"google.com"},{"url":"microsoft.com"}]'
echo "$json" | jq -r '.[].url' | while read url; do
redirect=$(curl -LSs \
-o /dev/null \
-w '%{url_effective}' \
"${url}" 2>/dev/null)
jq --null-input --arg url "${url}" --arg redirect "${redirect}" \
'{url:$url, redirect: $redirect}'
done | jq -s
Alternative (first) solution:
You can output the url and the effective_url as tab separated data and create the output json with jq:
json='[{"url":"google.com"},{"url":"microsoft.com"}]'
echo "$json" | jq -r '.[].url' | while read line; do
prefix="${line}\t"
curl -LSs -o /dev/null -w "${prefix}"'%{url_effective}'"\n" "$line" 2>/dev/null
done | jq -r --raw-input 'split("\t")|{"url":.[0],"redirection":.[1]}'
Both solutions will generate valid json, independently of whatever characters the url/effective_url might contain.
Trying to keep this in JSON all the way is pretty cumbersome. I would simply try to make Bash construct a new valid JSON fragment inside the loop.
So in other words, if $url is the URL and $redirect is where it redirects to, you can do something like
printf '{"url": "%s", "redirection": "%s"}\n' "$url" "$redirect"
to produce JSON output from these strings. So tying it all together
jq -r '.[].url' <<<"$json" |
while read -r url; do
printf '{"url:" "%s", "redirection": "%s"}\n' \
"$url" "$(curl -LSs -o /dev/null -w '%{url_effective}' "$url")"
done |
jq -s
This is still pretty brittle; in particular, if either of the printf input strings could contain a literal double quote, that should properly be escaped.
As a newbee to bash and jq, I was trying to download several urls from a json file using jq command in bash scripts.
My items.json file looks like this :
［
{"title" : [bob], "link" :[a.b.c]},
{"title" : [alice], "link" :[d.e.f]},
{"title" : [carol], "link" :[]}
]
what I was initially doing was just filter the non-empty link and put them in an array and then download the array:
#!/bin/bash
lnk=( $(jq -r '.[].link[0] | select (.!=null)' items.json) )
for element in ${lnk[#]}
do
wget $element
done
But the problem of this approach is that all the files downloaded use the link as the file names.
I wish to filter json file but still keeps the title name with the link so that i can rename the file in the wget command. But I dont have any idea on what structure should I use here. So how can i keep the title to in the filter and use it after?
You can use this:
IFS=$'\n' read -d '' -a titles < <(jq -r '.[] | select (.link[0]!=null) | .title[0]' items.json);
IFS=$'\n' read -d '' -a links < <(jq -r '.[] | select (.link[0]!=null) | .link[0]' items.json);
Then you can iterate over arrays "${title[#]}" & ${links[#]}...
for i in ${!titles[#]}; do
wget -O "${titles[i]}" "${links[#]}"
done
EDIT: Easier & safer approach:
jq -r '.[] | select (.link[0]!=null) | #sh "wget -O \(.title[0]) \(.link[0])"' items.json | bash
Here is a bash script demonstrating reading the result of a jq filter into bash variables.
#!/bin/bash
jq -M -r '
.[]
| select(.link[0]!=null)
| .title[0], .link[0]
' items.json | \
while read -r title; read -r url; do
echo "$title: $url" # replace with wget command
done
I was wondering how to parse the CURL JSON output from the server into variables.
Currently, I have -
curl -X POST -H "Content: agent-type: application/x-www-form-urlencoded" https://www.toontownrewritten.com/api/login?format=json -d username="$USERNAME" -d password="$PASSWORD" | python -m json.tool
But it only outputs the JSON from the server and then have it parsed, like so:
{
"eta": "0",
"position": "0",
"queueToken": "6bee9e85-343f-41c7-a4d3-156f901da615",
"success": "delayed"
}
But how do I put - for example the success value above returned from the server into a variable $SUCCESS and have the value as delayed & have queueToken as a variable $queueToken and 6bee9e85-343f-41c7-a4d3-156f901da615 as a value?
Then when I use-
echo "$SUCCESS"
it shows this as the output -
delayed
And when I use
echo "$queueToken"
and the output as
6bee9e85-343f-41c7-a4d3-156f901da615
Thanks!
Find and install jq (https://stedolan.github.io/jq/). jq is a JSON parser. JSON is not reliably parsed by line-oriented tools like sed because, like XML, JSON is not a line-oriented data format.
In terms of your question:
source <(
curl -X POST -H "$content_type" "$url" -d username="$USERNAME" -d password="$PASSWORD" |
jq -r '. as $h | keys | map(. + "=\"" + $h[.] + "\"") | .[]'
)
The jq syntax is a bit weird, I'm still working on it. It's basically a series of filters, each pipe taking the previous input and transforming it. In this case, the end result is some lines that look like variable="value"
This answer uses bash's "process substitution" to take the results of the jq command, treat it like a file, and source it into the current shell. The variables will then be available to use.
Here's an example of Extract a JSON value from a BASH script
#!/bin/bash
function jsonval {
temp=`echo $json | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w $prop`
echo ${temp##*|}
}
json=`curl -s -X GET http://twitter.com/users/show/$1.json`
prop='profile_image_url'
picurl=`jsonval`
`curl -s -X GET $picurl -o $1.png`
A bash script which demonstrates parsing a JSON string to extract a
property value. The script contains a jsonval function which operates
on two variables, json and prop. When the script is passed the name of
a twitter user it attempts to download the user's profile picture.
You could use perl module on command line:
1st, ensure they is installed, under debian based, you could
sudo apt-get install libjson-xs-perl
But for other OS, you could install perl modules via CPAN (the Comprehensive Perl Archive Network):
cpan App::cpanminus
cpan JSON::XS
Note: You may have to run this with superuser privileges.
then:
curlopts=(-X POST -H
"Content: apent-type: application/x-www-form-urlencoded"
-d username="$USERNAME" -d password="$PASSWORD")
curlurl=https://www.toontownrewritten.com/api/login?format=json
. <(
perl -MJSON::XS -e '
$/=undef;my $a=JSON::XS::decode_json <> ;
printf "declare -A Json=\047(%s)\047\n", join " ",map {
"[".$_."]=\"".$a->{$_}."\""
} qw|queueToken success eta position|;
' < <(
curl "${curlopts[#]}" $curlurl
)
)
The line qw|...| let you precise which variables you want to be driven... This could be replaced by keys $a, but could have to be debugged as some characters is forbiden is associative arrays values names.
echo ${Json[queueToken]}
6bee9e85-343f-41c7-a4d3-156f901da615
echo ${Json[eta]}
0