Bash to get JSON doc from Github - json

I am trying to do a curl to basically clone a package.json in my local directory, by using:
curl -U "<email>":"<pass>" -L "https://github.com/flowrepo/blob/master/package.json"
I managed to get pass the redirect by using the -L flag, but I cannot get a valid JSON doc as the command above is returning the entire Github page. Any thoughts?

Access the "raw" variant by changing the URL to:
https://github.com/flowrepo/raw/master/package.json"

[Solved]
Thanks Hans Z. for your answer. This is the final call that works:
curl -U "<email>":"<pass>" -L "https://github.com/flowrepo/raw/master/package.json?token<retrieved_token> -o output.txt

Related

How to use shell variable in MQTT

I am new to shell scripting and MQTT.
I need to publish a JSON file using MQTT. We can do it by storing the JSON contents in a shell variable. But it is not working for me.
my shell script:
#!/bin/sh
var1='{"apiVersion":"2.1","data":{"id":"4TSJhIZmL0A","uploaded":"2008-07-15T18:11:59.000Z","updated":"2013-05-01T21:01:49.000Z","uploader":"burloandbardsey","category":"News","title":"bbc news start up theme","description":"bbc","thumbnail":{"sqDefault":"http://i.ytimg.com/vi/4TSJhIZmL0A/default.jpg","hqDefault":"http://i.ytimg.com/vi/4TSJhIZmL0A/hqdefault.jpg"},"player":{"default":"http://www.youtube.com/watch?v=4TSJhIZmL0A&feature=youtube_gdata_player","mobile":"http://m.youtube.com/details?v=4TSJhIZmL0A"},"content":{"5":"http://www.youtube.com/v/4TSJhIZmL0A?version=3&f=videos&app=youtube_gdata","1":"rtsp://v5.cache7.c.youtube.com/CiILENy73wIaGQlAL2aGhIk04RMYDSANFEgGUgZ2aWRlb3MM/0/0/0/video.3gp","6":"rtsp://v5.cache7.c.youtube.com/CiILENy73wIaGQlAL2aGhIk04RMYESARFEgGUgZ2aWRlb3MM/0/0/0/video.3gp"},"duration":15,"aspectRatio":"widescreen","rating":4.6683936,"likeCount":"354","ratingCount":386,"viewCount":341066,"favoriteCount":0,"commentCount":155,"accessControl":{"comment":"allowed","commentVote":"allowed","videoRespond":"allowed","rate":"allowed","embed":"allowed","list":"allowed","autoPlay":"allowed","syndicate":"allowed"}}}'
mosquitto_pub -h localhost -t test -m "$var1"
echo "$var1"
my Mosquitto commands:
Publisher: `mosquitto_pub -h localhost -t "test" -m "{"Contents":$var1}"
Subscriber: mosquitto_sub -h localhost -t "test"
Output I got:
{"Contents":}
Expected Output:
{"Contents":{"name":"Harini", "age":24, "city":"NewYork", "message":"Hello world"}}
I can get the output only at the terminal because of echo. But I want to publish and subscribe to the contents of the shell variable(var1)
Please help me out to get the output. Whether I need to add some more code in the shell script. I don't know how to proceed. Or can you suggest any other method.
The following works just fine, it's all about which quotes you use where:
#!/bin/sh
var1='{"name":"Harini", "age":24, "city":"NewYork","message":"Hello world"}'
echo $var1
mosquitto_pub -t test -m "{\"Content\": $var1}"
You need to wrap the -m argument in quotes because it contains spaces, which in turn means you need to escape the " round Content.
Wrapping the content of var1 in single quotes means you don't need to escape the double quotes in it.

Jenkins Build Time Trend API does not yield output using curl API

I got this link to get the Build Time Trend along with other Data in jenkins
https://jenkins:8080/view/<view-name>/job/<job-name>/<buildnumber>/api/json
This works well in a web browser but this does not seem to work with curl, does not give any result when I run along with curl command
This is what I tried
curl -u user:api_token -s -k "https://jenkins:8080/view/<view-name>/job/<job-name>/<buildnumber>/api/json"
This syntax worked with other API's.
Not sure what is wrong here.
curl -u userid:api_token -s -k "https://jenkins:8080/view/<view-name>/job/<job-name>/<buildnumber>/api/json" | jq.'causes[]|{result}'
jq.causes[]|{result}: command not found
You need a space between jq and its arguments (and probably not a period).
... | jq 'causes[]|{result}'
^
space here

cURL download files issue

When i give the URL (http://192.168.150.41:8080/filereport/31779/json/) in browser, It automatically downloads the file as 31779_report.json.
Now using i'm trying to download the file using curl but i get the following error.
$ curl -O http://192.168.150.41:8080/filereport/31779/json/
curl: Remote file name has no length!
curl: try 'curl --help' or 'curl --manual' for more information
When using the '-L' switch , I get the JSON content displayed but the file is not saved.
$curl -L http://192.168.150.41:8080/filereport/31779/json/
{
.....
.....
}
How to download the exact file "31779_report.json" using cURL / wget ?
I don't want the contents to be redirected (>) manually to a file (31779_report.json).
Any suggestions please ?
The -O flag of curl tries to use the remote name of the file, but because your URL schema does not end with a filename, it can not do this. The -o flag (lower-case o) can be used to specify a file name manually without redirecting STDOUT like so:
curl <address> -o filename.json
You can manually construct the filename format you want using awk. For example:
URL=http://192.168.150.41:8080/filereport/31779/json/
file_number=$(echo $URL | awk -F/ '{print $(NF-2)}')
file_name="${file_number}_report.json"
curl -L "$URL" -o "$file_name"
Hope this is more helpful.
wget --content-disposition did the trick for me (https://askubuntu.com/a/77713/18665)
$ wget --content-disposition https://www.archlinux.org/packages/core/x86_64/lib32-glibc/download/
...
Saving to: 'lib32-glibc-2.33-4-x86_64.pkg.tar.zst'
Compare to curl:
$ curl -LO https://www.archlinux.org/packages/core/x86_64/lib32-glibc/download/
curl: Remote file name has no length!
curl: (23) Failed writing received data to disk/application
And wget without --content-disposition:
$ wget https://www.archlinux.org/packages/core/x86_64/lib32-glibc/download/
...
Saving to: 'index.html'

Docker API can’t apply json filters

According to the https://docs.docker.com/engine/reference/api/docker_remote_api_v1.24/#/list-tasks, filter can be only used to get running containers with a particular service name. For some reason, I am getting a full list of all tasks regardless of their names or desired states. I can't find any proper examples of using curl with JSON requests with Docker API.
I'm using the following command:
A)
curl -X GET -H "Content-Type: application/json" -d '{"filters":[{ "service":"demo", "desired-state":"running" }]}' https://HOSTNAME:2376/tasks --cert ~/.docker/cert.pem --key ~/.docker/key.pem --cacert ~/.docker/ca.pem
Returns everything
B)
trying to get something working from Docker Remote API Filter Exited
curl https://HOSTNAME:2376/containers/json?all=1&filters={%22status%22:[%22exited%22]} --cert ~/.docker/cert.pem --key ~/.docker/key.pem --cacert ~/.docker/ca.pem
This one returns "curl: (60) Peer's Certificate issuer is not recognized.", so I guess that curl request is malformed.
I have asked on Docker forums and they helped a little. I'm amazed that there are no proper documentation anywhere on the internet on how to use Docker API with curl or is it so obvious and I don't understand something?
I should prefix this with the fact that I have never seen curl erroneously report a certificate error when in fact there was some sort of other issue in play, but I will trust your assertion that this is in fact not a certificate problem.
I thought at first that your argument to filters was incorrect, because
according to the API reference, the filters parameter is...
a JSON encoded value of the filters (a map[string][]string) to process on the containers list.
I wasn't exactly sure how to interpret map[string][]string, so I set up a logging proxy between my Docker client and server and ran docker ps -f status=exited, which produced the following request:
GET /v1.24/containers/json?filters=%7B%22status%22%3A%7B%22exited%22%3Atrue%7D%7D HTTP/1.1\r
If we decode the argument to filters, we see that it is:
{"status":{"exited":true}}
Whereas you are passing:
{"status":["exited"]}
So that's different, obviously, and I was assuming that was the source of the problem...but when trying to verify that, I ran into a curious problem. I can't even run your curl command line as written, because curl tries to perform some globbing behavior due to the braces:
$ curl http://localhost:2376/containers/json'?filters={%22status%22:[%22exited%22]}'
curl: (3) [globbing] nested brace in column 67
If I correctly quote your arguments to filter:
$ python -c 'import urllib; print urllib.quote("""{"status":["exited"]}""")'
%7B%22status%22%3A%5B%22exited%22%5D%7D
It seems to work just fine:
$ curl http://localhost:2376/containers/json'?filters=%7B%22status%22%3A%5B%22exited%22%5D%7D'
[{"Id":...
I can get the same behavior if I use your original expression and pass -g (aka --globoff) to disable the brace expansion:
$ curl -g http://localhost:2376/containers/json'?filters={%22status%22:[%22exited%22]}'
[{"Id":...
One thing I would like to emphasize is the utility of sticking a proxy between the docker client and server. If you ever find yourself asking, "how do I use this API?", an excellent answer is to see exactly what the Docker client is doing in the same situation.
You can create a logging proxy using socat. Here is an example.
docker run -v /var/run/docker.sock:/var/run/docker.sock -p 127.0.0.1:1234:1234 bobrik/socat -v TCP-LISTEN:1234,fork UNIX-CONNECT:/var/run/docker.sock
Then run a command like so in another window.
docker -H localhost:1234 run --rm -p 2222:2222 hello-world
This example uses docker on ubuntu.
A docker REST proxy can be simple like this:
https://github.com/laoshanxi/app-mesh/blob/main/src/sdk/docker/docker-rest.go
Then you can curl like this:
curl -g http://127.0.0.1:6058/containers/json'?filters={%22name%22:[%22jenkins%22]}'

Curl commands not executed properly when escuted through shell script

I have a curl which sends application/json data .when i type this url directly through unix console,it works fine.But when i take this url and store it in a csv file and through shell script try accessing this file and read each curl and esecute it through backticks i am facing two problems
1) it is not allowing spaces in the json data being posted
2) the content type is not being set
Please find below the same url
curl -i -X PUT -H 'content-type:application/json' -H "Accept:application/json" -d '{"startTime":1426172400000,"endTime":1426173300000,"attributes":{"title":"X X X","link":"https://someurl.com}}' http://10.10.7.90:9084/myapp/rest/app/706128.api`
I found the issue .It was in my shell script .Insead of using the back tick '`' to execute the curl ,i used eval curl .which solved the issue