Send json HTTP post with bash - json

I would like to send some data to a web service which accept only a json object. I will use curl to make the HTTP POST but I am wondering it there is a library to create a json object in bash.
Another requirement is to avoid installation of other packages (rpm/apt-get) but only other bash files as libraries.


This is an example taken form BigQuery API
curl -H "Authorization: GoogleLogin auth=<<YOUR_TOKEN>>" \
-X POST \
-H "Content-type: application/json" \
-d '{"params":{"q":"select count(*) from [bigquery/samples/shakespeare];"},"method":"bigquery.query"}' \
'https://www.googleapis.com/rpc'


Checkout TickTick.
It's a true Bash JSON parser.
A stringifier should be in the works shortly, (but it wouldn't be difficult to create your own just using bash's foreach).
#!/bin/bash
. /path/to/ticktick.sh
# File
DATA=`cat data.json`
# cURL
#DATA=`curl http://foobar3000.com/echo/request.json`
tickParse "$DATA"
echo ``pathname``
echo ``headers["user-agent"]``


I recommand Jshon.
http://kmkeen.com/jshon/
It is designed to be as usable as possible from within the shell and replaces fragile adhoc parsers made from grep/sed/awk as well as heavyweight one-line parsers made from perl/python.
Requires Jansson,But you can build a static link version.

Related

Is There A Better Way To Scrape Data Off An API Curl Request?

I’m trying to scrape some data off an API and export as a JSON txt file. I have about 10,000 separate requests I would like to do and unfortunately the sequencing is not sequential and each request has a separate number I need to insert into the URL.
I’ve been doing them manually off a CURL command in terminal (macOS) and that seems to be working fine although somewhat time consuming. An example is shown below…
Request 1
curl --compressed -o 182969088.txt 'https://example.com/example/example/182969088/example' \
-X 'GET' \
-H 'x-api-key: i74lIf1J3CFa49sCZYmizr4oMtUS0t2U49m7YRNeF'
Request 2
curl --compressed -o 182962045.txt 'https://example.com/example/example/182962045/example' \
-X 'GET' \
-H 'x-api-key: i74lIf1J3CFa49sCZYmizr4oMtUS0t2U49m7YRNeF'
Does anyone know of a better way? All the separate 10,000 numbers are stored in an excel sheet. I was hoping there would be a way just to create a template and have the numbers copied in automatically and then I can just copy each individual request to the terminal instead of having to copy in the number twice and then go terminal.

Is there a way to send very complex queries in loopback using curl?

I succeed when making simple queries in loopback, using curl, with fairly innocuous use of "or" and "and":
curl.exe -s -o %TEMP%\output.json -X GET --header "Content-Type: application/json" --header "Accept: application/json" --globoff "https://somesite.com/data?filter={\"where\":{\"or\":[{\"id\":123},{\"id\":456}]}}&access_token=abcdefg123456"
But my need actually is to send a much larger number of possibilities for my ors and ands. I get to a point where the filter inevitably reaches some hard limit, that I don't have access to change at the server end, and I get a 414 Request-URI Too Large error.
Is there a way to specify larger filter queries via a file or any other way, in either Windows cmd or Git bash?
Thanks for your time,
Rip

How to pass data from file when calling gcloud function from CLI?

I would like to pass data to my cloud function from a local JSON file by using the command gcloud functions call <MY-FUNCTION>.
I can successfully pass data following the official docs but I would like to know if there's a way to do such a thing:
gcloud functions call <MY-FUNCTION> --data './path/to/my/file.json'
Copy pasting the data works, but it gets inconvenient as it might contain hundreds of lines worth of information.
Is there a way to do such a thing? Perharps a workaround by using something different from the gcloud CLI?
Thanks!

You can do this:
gcloud beta functions call [[YOUR-FUNCTION]] \
--data="$(cat ./path/to/your/file.json)" \
--region=[[YOUR-REGION]] \
--project=[[YOUR-PROJECT]]
Alternatively, you can do this with curl:
curl \
--data #./path/to/your/file.json \
https://[YOUR-REGION]]-[[YOUR-PROJECT]].cloudfunctions.net/[[YOUR-FUNCTION]]

Converting a Swagger YAML file to JSON from the command line

I'd like to convert a Swagger YAML file to JSON from the command line. The plan is to use this command line during a CI job. I've searched on google and found many solutions, but most of them use Python or Ruby, which I'd prefer not to use. For example: http://www.commandlinefu.com/commands/view/12218/convert-yaml-to-json
I'd like to do this without using Python or Ruby, and
I'd also like to be able to control the leading whitespace when formatting the JSON to match exactly the JSON that is output from Swagger's editor.swagger.io editor, when you choose File -> Download JSON
All this means is that I'd like the whitespace padding to be four spaces, like so:
{
"swagger": "2.0",
"info": {
"title": "API TITLE",
I haven't tried the Python method in the link above, but the Ruby method uses two space whitespace padding. Perhaps there is a way to control that, but I don't want to use Ruby or Python in this solution anyway.
I'm sure that there are many "correct" answers to this question. I am looking for the solution that is most elegant with the least number of dependencies. Ideally, a diff of the resulting JSON file against a JSON file generated by the editor.swagger.io should be empty.

I think that you are looking for the swagger-codegen (now OpenApi-generator) functionality:
Running
swagger-codegen generate -i swagger.yaml -l swagger
will out put a swagger.json in the same location.
Update For CI:
If you can install it on your build machine- good for you.
If you can't - the github page has a link to a docker image with a nodejs server is available (to convert using a curl command as suggested in a different answer).
Update For Docker:
If you use Docker, try swaggerapi/swagger-codegen-cli, there is an example for docker-compose that might help a few answers down by Fabian & ckeeney.
Update about OpenApi:
This question is about swagger, and a few years old. If you're just starting to use Swagger you should switch to OpenApi instead, and if you have existing swagger files, i suggest migrating.

Using yamljs:
yaml2json swagger.yaml -p -i4
The output from this command diff'd against the JSON output from editor.swagger.io produces an empty diff.
This is indeed what I'm looking for, but it brings in a huge dependency (node). I'm hoping for something even lighter, yet equally as elegant as this.

swagger-codegen cli interface
As Liel has already pointed out, you can run
swagger-codegen generate -i swagger.yaml -l swagger
Docker
If you use Docker, then I suggest you try swaggerapi/swagger-codegen-cli.
You can generate a json file using docker with the following command:
docker run -v ./docs:/docs swaggerapi/swagger-codegen-cli generate -i /docs/swagger.yaml -l swagger -o /docs
I like to setup a docker-compose.yml to "alias" this command for easy reuse:
version: "2"
services:
gen-swagger:
volumes:
- ./docs:/docs
image: swaggerapi/swagger-codegen-cli
command: generate -i /docs/swagger.yaml -l swagger -o /docs
And now I can just run docker-compose run gen-swagger

For version swagger-codegen 3.0.4
Use
swagger-codegen generate -i my_yaml.yaml -l openapi
to get a .json.

Another possibility to convert a swagger.yml file to swagger.json is a NPM package called swagger-cli.
npm install -g swagger-cli
Then you can convert a yml to json file:
swagger-cli bundle -o api-spec.json api-spec.yml

You can use the online swagger codegen project to do this:
curl -X POST --header "Content-Type: application/json" --header "Accept: application/json" -d "{
\"spec\": {}
}" "https://generator.swagger.io/api/gen/clients/swagger-yaml"
Put the value of your swagger definition in the spec object. You'll get a link to download the converted & validated spec, in yaml format.
For options, take a look here:
http://generator.swagger.io/

I'd use https://openapi-generator.tech/
It's an npm install (I just used it locally npm install #openapitools/openapi-generator-cli) and then
npx #openapitools/openapi-generator-cli generate -i source.yaml -g openapi -o outputdir

For a gradle with Kotlin, i've wrote in my build.gradle.kts:
import com.fasterxml.jackson.databind.JsonNode
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory
import java.nio.file.Path
and then in a some task like compileJavacode for convertation:
val compileJava: Task by tasks.getting {
val openApiDir = "${rootProject.projectDir}/openapi"
val json: JsonNode? = ObjectMapper(YAMLFactory())
.readTree(Path.of("$openApiDir/openapi.yaml").toFile())
ObjectMapper().writerWithDefaultPrettyPrinter()
.writeValue(Path.of("$openApiDir/openapi.json").toFile(), json)
}

curl command unable to pass bash parameter

so I am new to curl and am trying to write a bash script that will I can run that downloads a file. So I start off by authentication then make a POST to request a download. I am given a Foo_ID in which I parse using bash and set to a parameter. I then try to use GET the certain Foo data via a download URL. The issue I am having is that whenever I pass in the parameter I parsed from the POST response I get nothing. Here is an example of what I am doing.
#!/bin/bash
curl -b cookies -c cookies -X POST #login_info -d "https://foo.bar.com/auth"
curl -b cookies -c cookies -X POST #Foo_info -d "https://foo.bar.com/foos" > ./tmp/stuff.t
myFooID=`cat ./tmp/stuff.t |grep -Po '"foo_id:.*?",'|cut -d '"' -f 4`
curl -b cookies -c cookies "http://foo.bar.com/foo-download?id=${myFooID}" > ./myFoos/Foo1.data
I have echo'd myFooID to make sure it is correct and it is. I have also echo'd "https://foo.bar.com/foo-download?id=${myFooID}" and it is properly showing the URL I need. Could anyone help me with this like I said I am new to using curl and a little rusty on using bash commands.

So I have solved the issue. The problem was after doing my post for the Foo I didn't give enough time for my foo to be created before trying to download it. I added a sleep command between both the last two curl commands and now it works perfectly. I would like to thank Dennis Williamson for helping me clean up my code wich led me to understanding my issue. I have created a shrine for him on my desk.