Guys I'm currently working with cURL for the first time.
What I am trying to do is pull data from a website usimg cURL
an put them into excel using the following commmand. I have to use an API-Key
to get the data.
curl -H "X-API-Key: API_KEY_NUMBER" http://example.com/api/exports/model/62f0d0dc24757f6e5bb0b723 -o "text.xlsx"
This works fine so far, the problem is that if want to open it in Excel it tells me that the file can not be opened because the file format or the file extension is invalid.
If i change the file extension to
curl -H "X-API-Key: API_KEY_NUMBER" http://example.com/api/exports/model/62f0d0dc24757f6e5bb0b723 -o "text.txt"
it opens in a text file but with all the data that i need. Now I am looking for a way to solve this.
Related
I’m trying to scrape some data off an API and export as a JSON txt file. I have about 10,000 separate requests I would like to do and unfortunately the sequencing is not sequential and each request has a separate number I need to insert into the URL.
I’ve been doing them manually off a CURL command in terminal (macOS) and that seems to be working fine although somewhat time consuming. An example is shown below…
Request 1
curl --compressed -o 182969088.txt 'https://example.com/example/example/182969088/example' \
-X 'GET' \
-H 'x-api-key: i74lIf1J3CFa49sCZYmizr4oMtUS0t2U49m7YRNeF'
Request 2
curl --compressed -o 182962045.txt 'https://example.com/example/example/182962045/example' \
-X 'GET' \
-H 'x-api-key: i74lIf1J3CFa49sCZYmizr4oMtUS0t2U49m7YRNeF'
Does anyone know of a better way? All the separate 10,000 numbers are stored in an excel sheet. I was hoping there would be a way just to create a template and have the numbers copied in automatically and then I can just copy each individual request to the terminal instead of having to copy in the number twice and then go terminal.
I have multiple JSON files within a folder and I would like to post them all at once, in a single command line using curl. Is there a way to do this?
I have these files within a folder in my directory..
20190116_101859_WifiSensor(1).json
20190116_101859_WifiSensor(2).json
20190116_101859_WifiSensor(3).json
20190116_101859_WifiSensor(4).json
20190116_101859_WifiSensor(5).json
20190116_101859_WifiSensor(6).json
20190116_101859_WifiSensor(7).json
20190116_101859_WifiSensor(8).json
... plus more
I'd like to post all of the files from the folder in one go.
I know how to post one file using
curl -d "#20190116_101859_WifiSensor(1).json" http://iconsvr:8005/data
I need a way of posting them in one go, without having to write out each file name, if possible.
You can use a foreach loop to iterate over all files in your current directory which contains WifiSensor in the filename.
In Linux (Bash) you could use
for f in *WifiSensor*.json; do curl -d $f http://iconsvr:8005/data; done
In Windows (CMD)
for /r %f in (*WifiSensor*.json) do curl -d %f http://iconsvr:8005/data
Don't forget if you using the Windows Snippet above in a Batch file, you need to double the % signs.
I am trying to view an HTML page that's saved as an attachment over the browser. When I navigate to the link, I get a prompt asking me to download the file. I want to open it directly in the browser. How do I do this?
Here's how I saved it:
curl -vX PUT http://localhost:5984/database/app/index.html -d index.html -H "Content-Type: application/html"
I want to fire a POST request in command line, to post my image to a image searching site. At first, I tried cURL and get this command which works:
curl -i -X POST -F file=#search.png http://saucenao.com/search.php
It will post a file in FORM to the searching site and returns a HTML page result full with JavaScript which makes it hard to read in terminal. And it's also hard to preview online image in terminal.
Then I remember that I can open Chrome with arguments in command line, which I think may solve my problem. After some digging, I found Chrome switches, but seams it's just about Chrome starting flags (I'm not sure is this right, but I didn't find how to fire a post request like cURL do.)
So, can I use Chrome in command line to start it with a POST request just like my cURL command above?
There are a couple of things you could do.
You could write a script in JavaScript that will send the POST request and display the results inside the <body> element or the like;
You could keep the cURL command and use the -o (or --output) to save the resulting HTML in a file (but lose the -i switch, to avoid having the headers in the file), then open the file in Chrome or whichever browser you prefer. You could combine the two commands as a one-liner in any operating system. If you use Ubuntu, for example:
$ curl -o search.html -X POST -F file=#search.png http://saucenao.com/search.php && google-chrome search.html && rm search.html
According to this answer you could use bcat in order to avoid using a temporary file. Install it by apt-get install ruby-bcat and then just run
$ curl -X POST -F file=#search.png http://saucenao.com/search.php | bcat
I think the easier option is #2, but whichever you prefer.
so I am new to curl and am trying to write a bash script that will I can run that downloads a file. So I start off by authentication then make a POST to request a download. I am given a Foo_ID in which I parse using bash and set to a parameter. I then try to use GET the certain Foo data via a download URL. The issue I am having is that whenever I pass in the parameter I parsed from the POST response I get nothing. Here is an example of what I am doing.
#!/bin/bash
curl -b cookies -c cookies -X POST #login_info -d "https://foo.bar.com/auth"
curl -b cookies -c cookies -X POST #Foo_info -d "https://foo.bar.com/foos" > ./tmp/stuff.t
myFooID=`cat ./tmp/stuff.t |grep -Po '"foo_id:.*?",'|cut -d '"' -f 4`
curl -b cookies -c cookies "http://foo.bar.com/foo-download?id=${myFooID}" > ./myFoos/Foo1.data
I have echo'd myFooID to make sure it is correct and it is. I have also echo'd "https://foo.bar.com/foo-download?id=${myFooID}" and it is properly showing the URL I need. Could anyone help me with this like I said I am new to using curl and a little rusty on using bash commands.
So I have solved the issue. The problem was after doing my post for the Foo I didn't give enough time for my foo to be created before trying to download it. I added a sleep command between both the last two curl commands and now it works perfectly. I would like to thank Dennis Williamson for helping me clean up my code wich led me to understanding my issue. I have created a shrine for him on my desk.