Am trying below curl request:
curl -s https://www.instagram.com/p/B_Xv0kgJUNq/?igshid=2zc9h1kyhkyv | grep commentCount
How to get the value number of "commentCount":"26508" in the curl response.
I've tried above command it getting me the full JSON <script type="application/ld+json"> .. while I just need the count of "commentCount" value.
An HTML and JSON parser like xidel would fit the job:
$ xidel -s "https://www.instagram.com/p/B_Xv0kgJUNq/?igshid=2zc9h1kyhkyv" -e '
parse-json(
//script[#type="application/ld+json"]
)/commentCount
'
with Linux grep you can
$ curl -s https://www.instagram.com/p/B_Xv0kgJUNq/?igshid=2zc9h1kyhkyv | grep -Po 'commentCount":"\K\d*'
99795
Related
I am performing a curl test using shell script where I need to save curl server response into JSON along with CURL Information like time_connect, http_code, etc. I am trying the following code to write output to a JSON
HOST_ADDR="http://$mynds/"
do_request(){
echo $(curl $HOST_ADDR --silent -w "\n{\n
\"HttpCode\": %{http_code},\n
\"NumRedirects\":%{num_redirects},\n
\"NumConnects\":%{num_connects},\n
\"SizeDownloadInBytes\":%{size_download},\n
\"SizeHeaderInBytes\":%{size_header},\n
\"SizeRequestInBytes\":%{size_request},\n
\"SizeUploadInBytes\":%{size_upload},\n
\"SpeedUploadBPS\":%{speed_upload},\n
\"SpeedDownloadBPS\":%{speed_download},\n
\"TimeAppConnectSec\":%{time_appconnect},\n
\"TimeConnectSec\":%{time_connect},\n
\"TimeNamelookupSec\":%{time_namelookup},\n
\"TimePreTransferSec\":%{time_pretransfer},\n
\"TimeRedirectSec\":%{time_redirect},\n
\"TimeStartTransferSec\":%{time_starttransfer},\n
\"TimeTotalSec\":%{time_total},\n
\"UrlEffective\":\"%{url_effective}\"
}" -s)
}
do_request
The simple output I am getting:
{"hostIPAddr":"0.0.0.0","hostname":"vm01","text":"Hello World from
vm01"}` and `{ "HttpCode": 200, "NumRedirects":0, "NumConnects":1,
"SizeDownloadInBytes":85, "SizeHeaderInBytes":263,
"SizeRequestInBytes":99, "SizeUploadInBytes":0,
"SpeedUploadBPS":0.000, "SpeedDownloadBPS":14.000,
"TimeAppConnectSec":0.000000, "TimeConnectSec":5.553587,
"TimeNamelookupSec":5.097553, "TimePreTransferSec":5.553868,
"TimeRedirectSec":0.000000, "TimeStartTransferSec":5.827584,
"TimeTotalSec":5.827704, "UrlEffective":"http://dns" }
I am getting two JSON outputs one for curl information and one from the server. How do I combine these two outputs into a single JSON variable? Please help.
guid_id=$(uuidgen)
file_1="curlJsonRes_$guid_id.json"
file_2="curlMetaRes_$guid_id.json"
do_request(){
echo $(curl $HOST_ADDR --silent --output $file_1 -w "\n{\n
\"HttpCode\": %{http_code},\n
\"NumRedirects\":%{num_redirects},\n
\"NumConnects\":%{num_connects},\n
}")
} > $file_2
do_request
#Merging file outputs
echo $(jq -s '.[0] * .[1]' $file_1 $file_2) > $RESULTS_FILE
rm $file_1
rm $file_2
I am trying to get a list of URL after redirection using bash scripting. Say, google.com gets redirected to http://www.google.com with 301 status.
What I have tried is:
json='[{"url":"google.com"},{"url":"microsoft.com"}]'
echo "$json" | jq -r '.[].url' | while read line; do
curl -LSs -o /dev/null -w %{url_effective} $line 2>/dev/null
done
So, is it possible for us to use commands like curl inside jq for processing JSON objects.
I want to add the resulting URL to existing JSON structure like:
[
{
"url": "google.com",
"redirection": "http://www.google.com"
},
{
"url": "microsoft.com",
"redirection": "https://www.microsoft.com"
}
]
Thank you in advance..!
curl is capable of making multiple transfers in a single process, and it can also read command line arguments from a file or stdin, so, you don't need a loop at all, just put that JSON into a file and run this:
jq -r '"-o /dev/null\nurl = \(.[].url)"' file |
curl -sSLK- -w'%{url_effective}\n' |
jq -R 'fromjson | map(. + {redirection: input})' file -
This way only 3 processes will be spawned for the whole task, instead of n + 2 where n is the number of URLs.
I would generate a dictionary with jq per url and slurp those dictionaries into the final list with jq -s:
json='[{"url":"google.com"},{"url":"microsoft.com"}]'
echo "$json" | jq -r '.[].url' | while read url; do
redirect=$(curl -LSs \
-o /dev/null \
-w '%{url_effective}' \
"${url}" 2>/dev/null)
jq --null-input --arg url "${url}" --arg redirect "${redirect}" \
'{url:$url, redirect: $redirect}'
done | jq -s
Alternative (first) solution:
You can output the url and the effective_url as tab separated data and create the output json with jq:
json='[{"url":"google.com"},{"url":"microsoft.com"}]'
echo "$json" | jq -r '.[].url' | while read line; do
prefix="${line}\t"
curl -LSs -o /dev/null -w "${prefix}"'%{url_effective}'"\n" "$line" 2>/dev/null
done | jq -r --raw-input 'split("\t")|{"url":.[0],"redirection":.[1]}'
Both solutions will generate valid json, independently of whatever characters the url/effective_url might contain.
Trying to keep this in JSON all the way is pretty cumbersome. I would simply try to make Bash construct a new valid JSON fragment inside the loop.
So in other words, if $url is the URL and $redirect is where it redirects to, you can do something like
printf '{"url": "%s", "redirection": "%s"}\n' "$url" "$redirect"
to produce JSON output from these strings. So tying it all together
jq -r '.[].url' <<<"$json" |
while read -r url; do
printf '{"url:" "%s", "redirection": "%s"}\n' \
"$url" "$(curl -LSs -o /dev/null -w '%{url_effective}' "$url")"
done |
jq -s
This is still pretty brittle; in particular, if either of the printf input strings could contain a literal double quote, that should properly be escaped.
I am trying to create a pull request comment automatically whenever CI is run. The output of a given command is written to a file (could also just be stored inside an environment variable though). The problem is, I usually get the following response:
curl -XPOST -d "{'body':'$RESULT'}" https://api.github.com/repo/name/issues/number/comment
{
"message": "Problems parsing JSON",
"documentation_url": "https://developer.github.com/v3/issues/comments/#create-a-comment"
}
This is usually due to unescpaed characters, like \n, \t, " etc.
Is there any easy way to achieve this on the command line or in bash, sh, with jq or Python? Using the Octokit.rb library is works straight away, but I don't want to install Ruby in the build environment.
You can use jq to create your JSON object. Supposing you have your comment content in RESULT variable, the full request would be :
DATA=$(echo '{}' | jq --arg val "$RESULT" '.| {"body": $val}')
curl -s -H 'Content-Type: application/json' \
-H 'Authorization: token YOUR_TOKEN' \
-d "$DATA" \
"https://api.github.com/repos/:owner/:repo/issues/:number/comments"
The post "Using curl POST with variables defined in bash script functions" proposes multiple techniques for passing a varialbe like $RESULT in a curl POST parameter.
generate_post_data()
{
cat <<EOF
{
"body": "$RESULT"
}
EOF
}
Then, following "A curl tutorial using GitHub's API ":
curl -X POST \
-H "authToken: <yourToken" \
-H "Content-Type: application/json" \
--data "$(generate_post_data)" https://api.github.com/repo/name/issues/number/comment
I was wondering how to parse the CURL JSON output from the server into variables.
Currently, I have -
curl -X POST -H "Content: agent-type: application/x-www-form-urlencoded" https://www.toontownrewritten.com/api/login?format=json -d username="$USERNAME" -d password="$PASSWORD" | python -m json.tool
But it only outputs the JSON from the server and then have it parsed, like so:
{
"eta": "0",
"position": "0",
"queueToken": "6bee9e85-343f-41c7-a4d3-156f901da615",
"success": "delayed"
}
But how do I put - for example the success value above returned from the server into a variable $SUCCESS and have the value as delayed & have queueToken as a variable $queueToken and 6bee9e85-343f-41c7-a4d3-156f901da615 as a value?
Then when I use-
echo "$SUCCESS"
it shows this as the output -
delayed
And when I use
echo "$queueToken"
and the output as
6bee9e85-343f-41c7-a4d3-156f901da615
Thanks!
Find and install jq (https://stedolan.github.io/jq/). jq is a JSON parser. JSON is not reliably parsed by line-oriented tools like sed because, like XML, JSON is not a line-oriented data format.
In terms of your question:
source <(
curl -X POST -H "$content_type" "$url" -d username="$USERNAME" -d password="$PASSWORD" |
jq -r '. as $h | keys | map(. + "=\"" + $h[.] + "\"") | .[]'
)
The jq syntax is a bit weird, I'm still working on it. It's basically a series of filters, each pipe taking the previous input and transforming it. In this case, the end result is some lines that look like variable="value"
This answer uses bash's "process substitution" to take the results of the jq command, treat it like a file, and source it into the current shell. The variables will then be available to use.
Here's an example of Extract a JSON value from a BASH script
#!/bin/bash
function jsonval {
temp=`echo $json | sed 's/\\\\\//\//g' | sed 's/[{}]//g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | sed 's/\"\:\"/\|/g' | sed 's/[\,]/ /g' | sed 's/\"//g' | grep -w $prop`
echo ${temp##*|}
}
json=`curl -s -X GET http://twitter.com/users/show/$1.json`
prop='profile_image_url'
picurl=`jsonval`
`curl -s -X GET $picurl -o $1.png`
A bash script which demonstrates parsing a JSON string to extract a
property value. The script contains a jsonval function which operates
on two variables, json and prop. When the script is passed the name of
a twitter user it attempts to download the user's profile picture.
You could use perl module on command line:
1st, ensure they is installed, under debian based, you could
sudo apt-get install libjson-xs-perl
But for other OS, you could install perl modules via CPAN (the Comprehensive Perl Archive Network):
cpan App::cpanminus
cpan JSON::XS
Note: You may have to run this with superuser privileges.
then:
curlopts=(-X POST -H
"Content: apent-type: application/x-www-form-urlencoded"
-d username="$USERNAME" -d password="$PASSWORD")
curlurl=https://www.toontownrewritten.com/api/login?format=json
. <(
perl -MJSON::XS -e '
$/=undef;my $a=JSON::XS::decode_json <> ;
printf "declare -A Json=\047(%s)\047\n", join " ",map {
"[".$_."]=\"".$a->{$_}."\""
} qw|queueToken success eta position|;
' < <(
curl "${curlopts[#]}" $curlurl
)
)
The line qw|...| let you precise which variables you want to be driven... This could be replaced by keys $a, but could have to be debugged as some characters is forbiden is associative arrays values names.
echo ${Json[queueToken]}
6bee9e85-343f-41c7-a4d3-156f901da615
echo ${Json[eta]}
0
I need to get the "snapshot" value at the top of the file from this url: https://s3.amazonaws.com/Minecraft.Download/versions/versions.json
So I should get a variable that contains "14w08a" when I run the command to parse the json.
This will do the trick
$ curl -s "$url" | grep -Pom 1 '"snapshot": "\K[^"]*'
14w08a
best thing to do is use a tool with a JSON parser. For example:
value=$(
curl -s "$url" |
ruby -rjson -e 'data = JSON.parse(STDIN.read); puts data["latest"]["snapshot"]'
)