curl command unable to pass bash parameter - json

so I am new to curl and am trying to write a bash script that will I can run that downloads a file. So I start off by authentication then make a POST to request a download. I am given a Foo_ID in which I parse using bash and set to a parameter. I then try to use GET the certain Foo data via a download URL. The issue I am having is that whenever I pass in the parameter I parsed from the POST response I get nothing. Here is an example of what I am doing.
#!/bin/bash
curl -b cookies -c cookies -X POST #login_info -d "https://foo.bar.com/auth"
curl -b cookies -c cookies -X POST #Foo_info -d "https://foo.bar.com/foos" > ./tmp/stuff.t
myFooID=`cat ./tmp/stuff.t |grep -Po '"foo_id:.*?",'|cut -d '"' -f 4`
curl -b cookies -c cookies "http://foo.bar.com/foo-download?id=${myFooID}" > ./myFoos/Foo1.data
I have echo'd myFooID to make sure it is correct and it is. I have also echo'd "https://foo.bar.com/foo-download?id=${myFooID}" and it is properly showing the URL I need. Could anyone help me with this like I said I am new to using curl and a little rusty on using bash commands.

So I have solved the issue. The problem was after doing my post for the Foo I didn't give enough time for my foo to be created before trying to download it. I added a sleep command between both the last two curl commands and now it works perfectly. I would like to thank Dennis Williamson for helping me clean up my code wich led me to understanding my issue. I have created a shrine for him on my desk.

Related

Is There A Better Way To Scrape Data Off An API Curl Request?

I’m trying to scrape some data off an API and export as a JSON txt file. I have about 10,000 separate requests I would like to do and unfortunately the sequencing is not sequential and each request has a separate number I need to insert into the URL.
I’ve been doing them manually off a CURL command in terminal (macOS) and that seems to be working fine although somewhat time consuming. An example is shown below…
Request 1
curl --compressed -o 182969088.txt 'https://example.com/example/example/182969088/example' \
-X 'GET' \
-H 'x-api-key: i74lIf1J3CFa49sCZYmizr4oMtUS0t2U49m7YRNeF'
Request 2
curl --compressed -o 182962045.txt 'https://example.com/example/example/182962045/example' \
-X 'GET' \
-H 'x-api-key: i74lIf1J3CFa49sCZYmizr4oMtUS0t2U49m7YRNeF'
Does anyone know of a better way? All the separate 10,000 numbers are stored in an excel sheet. I was hoping there would be a way just to create a template and have the numbers copied in automatically and then I can just copy each individual request to the terminal instead of having to copy in the number twice and then go terminal.

Unable to download website content with wget

I am failing to download the below website using wget in bash:
wget --wait 1 -x -H -mk https://bittrex.com/api/v1.1/public/getorderbook?market=usdt-btc&type=sell
I have found some similar issues in other questions, however, their solution was to use -mk, which makes no difference here. The prompt just freeze after the command and nothing happens. If I try to open the same website in a browser, it will open normally. I would be grateful for any help here.
As already pointed out in a comment, you need to quote your URL.
The & in the URL is putting wget --wait 1 -x -H -mk https://bittrex.com/api/v1.1/public/getorderbook?market=usdt-btc into the background and causes everything after the & to be interpreted as a command, which can be a great risk depending on the URL!
If your URL contains a $ you should use single-quotes (') to pass the string literally without variable expansion, otherwise double quotes (") are fine.

How to translate my cURL command into Chrome command?

I want to fire a POST request in command line, to post my image to a image searching site. At first, I tried cURL and get this command which works:
curl -i -X POST -F file=#search.png http://saucenao.com/search.php
It will post a file in FORM to the searching site and returns a HTML page result full with JavaScript which makes it hard to read in terminal. And it's also hard to preview online image in terminal.
Then I remember that I can open Chrome with arguments in command line, which I think may solve my problem. After some digging, I found Chrome switches, but seams it's just about Chrome starting flags (I'm not sure is this right, but I didn't find how to fire a post request like cURL do.)
So, can I use Chrome in command line to start it with a POST request just like my cURL command above?
There are a couple of things you could do.
You could write a script in JavaScript that will send the POST request and display the results inside the <body> element or the like;
You could keep the cURL command and use the -o (or --output) to save the resulting HTML in a file (but lose the -i switch, to avoid having the headers in the file), then open the file in Chrome or whichever browser you prefer. You could combine the two commands as a one-liner in any operating system. If you use Ubuntu, for example:
$ curl -o search.html -X POST -F file=#search.png http://saucenao.com/search.php && google-chrome search.html && rm search.html
According to this answer you could use bcat in order to avoid using a temporary file. Install it by apt-get install ruby-bcat and then just run
$ curl -X POST -F file=#search.png http://saucenao.com/search.php | bcat
I think the easier option is #2, but whichever you prefer.

Send json HTTP post with bash

I would like to send some data to a web service which accept only a json object. I will use curl to make the HTTP POST but I am wondering it there is a library to create a json object in bash.
Another requirement is to avoid installation of other packages (rpm/apt-get) but only other bash files as libraries.
This is an example taken form BigQuery API
curl -H "Authorization: GoogleLogin auth=<<YOUR_TOKEN>>" \
-X POST \
-H "Content-type: application/json" \
-d '{"params":{"q":"select count(*) from [bigquery/samples/shakespeare];"},"method":"bigquery.query"}' \
'https://www.googleapis.com/rpc'
Checkout TickTick.
It's a true Bash JSON parser.
A stringifier should be in the works shortly, (but it wouldn't be difficult to create your own just using bash's foreach).
#!/bin/bash
. /path/to/ticktick.sh
# File
DATA=`cat data.json`
# cURL
#DATA=`curl http://foobar3000.com/echo/request.json`
tickParse "$DATA"
echo ``pathname``
echo ``headers["user-agent"]``
I recommand Jshon.
http://kmkeen.com/jshon/
It is designed to be as usable as possible from within the shell and replaces fragile adhoc parsers made from grep/sed/awk as well as heavyweight one-line parsers made from perl/python.
Requires Jansson,But you can build a static link version.

Curl giving "Invalid UTF-8 JSON" error from CouchDb although JSON is fine? Any ideas?

This is slightly different from the question titled "Error about ‘invalid JSON’ with couchDB view but the json’s fine": I am not trying to upload a file only enter a simple document.
The example I am trying to use is actually from O'Reilly's book "CouchDB: The Definitive Guide" and I am pretty sure that I have tried it before and got it to work. Here's the command:
curl -X PUT http://username:password#127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af -d '{"title":"There is Nothing Left to Lose","artist":"Foo Fighters"}'
The database albums exists and the username and password are correct. I have checked this with JSONLint and the JSON is valid and I am at a loss ... presumably there is an issue with the CouchDB server itself but it appears to be running correctly ... any ideas? This is driving me nuts!
Thanks
Thanks guys. Turns out it's a problem with quote escaping. Here's the answer I got from David on the CouchDB user mailing list:
This is a windows thing regarding
quoting - a real PITA. Unfortunately
cmd.exe shell on windows doesn't parse
this correctly. The rules for when
escaping with a "" or a ^" or a \" are
a bit vague but this works:
C:\tmp>curl -X PUT
http://username:password#127.0.0.1:5984/albums/6e1295ed6c29495e54cc05947f18c8af
-d "{\"title\":\"There is Nothing Left to Lose\",\"artist\":\"Foo
Fighters\"}"
{"ok":true,"id":"6e1295ed6c29495e54cc05947f18c8af","rev":"1-4b39c2971c9ad54cb37e08fa02fec636"}
C:\tmp>
"basically you need to \"escape\" all
\"quotes\" within your JSON"
This fixes my problem
I also met same problem, after searching a while, I found about Git Bash in other question. Git bash has curl command inside. The git bash is included in Git installer.
You have to put the escap characterstic ** and also do not forgot to put after -d all things in **"".
I.e : curl -X PUT http://127.0.0.1:5984/my_database/"001" -d "{\"Name\":\"Suresh\",\"age\":\"32\",\"Designation\":\"Associates Manager\"}"