I am trying to do a curl to basically clone a package.json in my local directory, by using:
curl -U "<email>":"<pass>" -L "https://github.com/flowrepo/blob/master/package.json"
I managed to get pass the redirect by using the -L flag, but I cannot get a valid JSON doc as the command above is returning the entire Github page. Any thoughts?
Access the "raw" variant by changing the URL to:
https://github.com/flowrepo/raw/master/package.json"
[Solved]
Thanks Hans Z. for your answer. This is the final call that works:
curl -U "<email>":"<pass>" -L "https://github.com/flowrepo/raw/master/package.json?token<retrieved_token> -o output.txt
Related
I am new to shell scripting and MQTT.
I need to publish a JSON file using MQTT. We can do it by storing the JSON contents in a shell variable. But it is not working for me.
my shell script:
#!/bin/sh
var1='{"apiVersion":"2.1","data":{"id":"4TSJhIZmL0A","uploaded":"2008-07-15T18:11:59.000Z","updated":"2013-05-01T21:01:49.000Z","uploader":"burloandbardsey","category":"News","title":"bbc news start up theme","description":"bbc","thumbnail":{"sqDefault":"http://i.ytimg.com/vi/4TSJhIZmL0A/default.jpg","hqDefault":"http://i.ytimg.com/vi/4TSJhIZmL0A/hqdefault.jpg"},"player":{"default":"http://www.youtube.com/watch?v=4TSJhIZmL0A&feature=youtube_gdata_player","mobile":"http://m.youtube.com/details?v=4TSJhIZmL0A"},"content":{"5":"http://www.youtube.com/v/4TSJhIZmL0A?version=3&f=videos&app=youtube_gdata","1":"rtsp://v5.cache7.c.youtube.com/CiILENy73wIaGQlAL2aGhIk04RMYDSANFEgGUgZ2aWRlb3MM/0/0/0/video.3gp","6":"rtsp://v5.cache7.c.youtube.com/CiILENy73wIaGQlAL2aGhIk04RMYESARFEgGUgZ2aWRlb3MM/0/0/0/video.3gp"},"duration":15,"aspectRatio":"widescreen","rating":4.6683936,"likeCount":"354","ratingCount":386,"viewCount":341066,"favoriteCount":0,"commentCount":155,"accessControl":{"comment":"allowed","commentVote":"allowed","videoRespond":"allowed","rate":"allowed","embed":"allowed","list":"allowed","autoPlay":"allowed","syndicate":"allowed"}}}'
mosquitto_pub -h localhost -t test -m "$var1"
echo "$var1"
my Mosquitto commands:
Publisher: `mosquitto_pub -h localhost -t "test" -m "{"Contents":$var1}"
Subscriber: mosquitto_sub -h localhost -t "test"
Output I got:
{"Contents":}
Expected Output:
{"Contents":{"name":"Harini", "age":24, "city":"NewYork", "message":"Hello world"}}
I can get the output only at the terminal because of echo. But I want to publish and subscribe to the contents of the shell variable(var1)
Please help me out to get the output. Whether I need to add some more code in the shell script. I don't know how to proceed. Or can you suggest any other method.
The following works just fine, it's all about which quotes you use where:
#!/bin/sh
var1='{"name":"Harini", "age":24, "city":"NewYork","message":"Hello world"}'
echo $var1
mosquitto_pub -t test -m "{\"Content\": $var1}"
You need to wrap the -m argument in quotes because it contains spaces, which in turn means you need to escape the " round Content.
Wrapping the content of var1 in single quotes means you don't need to escape the double quotes in it.
I got this link to get the Build Time Trend along with other Data in jenkins
https://jenkins:8080/view/<view-name>/job/<job-name>/<buildnumber>/api/json
This works well in a web browser but this does not seem to work with curl, does not give any result when I run along with curl command
This is what I tried
curl -u user:api_token -s -k "https://jenkins:8080/view/<view-name>/job/<job-name>/<buildnumber>/api/json"
This syntax worked with other API's.
Not sure what is wrong here.
curl -u userid:api_token -s -k "https://jenkins:8080/view/<view-name>/job/<job-name>/<buildnumber>/api/json" | jq.'causes[]|{result}'
jq.causes[]|{result}: command not found
You need a space between jq and its arguments (and probably not a period).
... | jq 'causes[]|{result}'
^
space here
When i give the URL (http://192.168.150.41:8080/filereport/31779/json/) in browser, It automatically downloads the file as 31779_report.json.
Now using i'm trying to download the file using curl but i get the following error.
$ curl -O http://192.168.150.41:8080/filereport/31779/json/
curl: Remote file name has no length!
curl: try 'curl --help' or 'curl --manual' for more information
When using the '-L' switch , I get the JSON content displayed but the file is not saved.
$curl -L http://192.168.150.41:8080/filereport/31779/json/
{
.....
.....
}
How to download the exact file "31779_report.json" using cURL / wget ?
I don't want the contents to be redirected (>) manually to a file (31779_report.json).
Any suggestions please ?
The -O flag of curl tries to use the remote name of the file, but because your URL schema does not end with a filename, it can not do this. The -o flag (lower-case o) can be used to specify a file name manually without redirecting STDOUT like so:
curl <address> -o filename.json
You can manually construct the filename format you want using awk. For example:
URL=http://192.168.150.41:8080/filereport/31779/json/
file_number=$(echo $URL | awk -F/ '{print $(NF-2)}')
file_name="${file_number}_report.json"
curl -L "$URL" -o "$file_name"
Hope this is more helpful.
wget --content-disposition did the trick for me (https://askubuntu.com/a/77713/18665)
$ wget --content-disposition https://www.archlinux.org/packages/core/x86_64/lib32-glibc/download/
...
Saving to: 'lib32-glibc-2.33-4-x86_64.pkg.tar.zst'
Compare to curl:
$ curl -LO https://www.archlinux.org/packages/core/x86_64/lib32-glibc/download/
curl: Remote file name has no length!
curl: (23) Failed writing received data to disk/application
And wget without --content-disposition:
$ wget https://www.archlinux.org/packages/core/x86_64/lib32-glibc/download/
...
Saving to: 'index.html'
According to the https://docs.docker.com/engine/reference/api/docker_remote_api_v1.24/#/list-tasks, filter can be only used to get running containers with a particular service name. For some reason, I am getting a full list of all tasks regardless of their names or desired states. I can't find any proper examples of using curl with JSON requests with Docker API.
I'm using the following command:
A)
curl -X GET -H "Content-Type: application/json" -d '{"filters":[{ "service":"demo", "desired-state":"running" }]}' https://HOSTNAME:2376/tasks --cert ~/.docker/cert.pem --key ~/.docker/key.pem --cacert ~/.docker/ca.pem
Returns everything
B)
trying to get something working from Docker Remote API Filter Exited
curl https://HOSTNAME:2376/containers/json?all=1&filters={%22status%22:[%22exited%22]} --cert ~/.docker/cert.pem --key ~/.docker/key.pem --cacert ~/.docker/ca.pem
This one returns "curl: (60) Peer's Certificate issuer is not recognized.", so I guess that curl request is malformed.
I have asked on Docker forums and they helped a little. I'm amazed that there are no proper documentation anywhere on the internet on how to use Docker API with curl or is it so obvious and I don't understand something?
I should prefix this with the fact that I have never seen curl erroneously report a certificate error when in fact there was some sort of other issue in play, but I will trust your assertion that this is in fact not a certificate problem.
I thought at first that your argument to filters was incorrect, because
according to the API reference, the filters parameter is...
a JSON encoded value of the filters (a map[string][]string) to process on the containers list.
I wasn't exactly sure how to interpret map[string][]string, so I set up a logging proxy between my Docker client and server and ran docker ps -f status=exited, which produced the following request:
GET /v1.24/containers/json?filters=%7B%22status%22%3A%7B%22exited%22%3Atrue%7D%7D HTTP/1.1\r
If we decode the argument to filters, we see that it is:
{"status":{"exited":true}}
Whereas you are passing:
{"status":["exited"]}
So that's different, obviously, and I was assuming that was the source of the problem...but when trying to verify that, I ran into a curious problem. I can't even run your curl command line as written, because curl tries to perform some globbing behavior due to the braces:
$ curl http://localhost:2376/containers/json'?filters={%22status%22:[%22exited%22]}'
curl: (3) [globbing] nested brace in column 67
If I correctly quote your arguments to filter:
$ python -c 'import urllib; print urllib.quote("""{"status":["exited"]}""")'
%7B%22status%22%3A%5B%22exited%22%5D%7D
It seems to work just fine:
$ curl http://localhost:2376/containers/json'?filters=%7B%22status%22%3A%5B%22exited%22%5D%7D'
[{"Id":...
I can get the same behavior if I use your original expression and pass -g (aka --globoff) to disable the brace expansion:
$ curl -g http://localhost:2376/containers/json'?filters={%22status%22:[%22exited%22]}'
[{"Id":...
One thing I would like to emphasize is the utility of sticking a proxy between the docker client and server. If you ever find yourself asking, "how do I use this API?", an excellent answer is to see exactly what the Docker client is doing in the same situation.
You can create a logging proxy using socat. Here is an example.
docker run -v /var/run/docker.sock:/var/run/docker.sock -p 127.0.0.1:1234:1234 bobrik/socat -v TCP-LISTEN:1234,fork UNIX-CONNECT:/var/run/docker.sock
Then run a command like so in another window.
docker -H localhost:1234 run --rm -p 2222:2222 hello-world
This example uses docker on ubuntu.
A docker REST proxy can be simple like this:
https://github.com/laoshanxi/app-mesh/blob/main/src/sdk/docker/docker-rest.go
Then you can curl like this:
curl -g http://127.0.0.1:6058/containers/json'?filters={%22name%22:[%22jenkins%22]}'
I have a curl which sends application/json data .when i type this url directly through unix console,it works fine.But when i take this url and store it in a csv file and through shell script try accessing this file and read each curl and esecute it through backticks i am facing two problems
1) it is not allowing spaces in the json data being posted
2) the content type is not being set
Please find below the same url
curl -i -X PUT -H 'content-type:application/json' -H "Accept:application/json" -d '{"startTime":1426172400000,"endTime":1426173300000,"attributes":{"title":"X X X","link":"https://someurl.com}}' http://10.10.7.90:9084/myapp/rest/app/706128.api`
I found the issue .It was in my shell script .Insead of using the back tick '`' to execute the curl ,i used eval curl .which solved the issue