Using Go, I'm attempting to fetch a few JSON responses concurrently from multiple endpoints. I'd like to attach each of these responses to fields in a struct or map and return this struct/map as a JSON object. (Backend for Frontend pattern). So I will make a web request to the Go application with some sort of identifier. It will in turn make several web requests, and compile the data into one large object to return as a response.
I'm using Fiber as my framework but any generic web framework would be similar:
app.Get("/requests/:identifier", func(c *fiber.Ctx) error {
identifier := c.Params("identifier")
timeout := 1600 * time.Millisecond
client := httpclient.NewClient(httpclient.WithHTTPTimeout(timeout))
res, err := client.Get("https://www.example.com/endpoint?id=" + identifier, nil)
if err != nil{
logger.Error("Timout value exceeded")
return c.Status(503).SendString("Socket Timeout")
}
logger.Info("Fetch success: ")
// Heimdall returns the standard *http.Response object
body, err := ioutil.ReadAll(res.Body)
code := 200
response := &main_response{
Service1: body,
}
return c.Status(code).JSON(response)
})
The problem I'm having is, I have no need to unmarshal this data in Go, as I have no use for it (I am simply passing it along). Do I have to unmarshal it just so I can set it as a field in my response struct like this?
type main_response struct {
Service1 []byte `json:"service1"`
Service2 map[string]string `json:"service2"`
Service3 map[string]interface{} `json:"service3"`
}
(I've tried a few different ways to accomplish this. Trying to use a byte array seems to base64 encode the responses)
I will want to marshal that struct to JSON before returning it, so perhaps I have little choice as I can't think of a way to tell Go "only marshal the main struct, everything else is already JSON". It almost feels like I'd be better off building a string at this point.
Use json.RawMessage to copy a []byte containing JSON directly to the response JSON document:
type main_response struct {
Service1 json.RawMessage `json:"service1"`
...
}
response := &main_response{
Service1: body,
}
return c.Status(code).JSON(response)
Related
I have an issue printing the JSON from the POST.
I use gorilla/mux for routing
r := mux.NewRouter()
r.HandleFunc("/test", Point).Methods("POST")
http.ListenAndServe(":80", r)`
and in Point function I have
func Point(w http.ResponseWriter, r *http.Request) {
var callback Decoder
json.NewDecoder(r.Body).Decode(&callback)
}
But I can use this method only when I know the structure and I want to figure out how to log.Print the whole JSON as a string. I tried
func Point(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
log.Println(r.Form)
}
But it prints an empty map. Please help to figure out this.
In the assumption that you are building a standard API endpoint which receives some JSON and you would like to do something with it you should approach it the following way.
Edit:
As mentioned in the comments when you use the ioutil.ReadAll()
function as in this example it will read everything that is sent in the
post request into the application memory. It's a good idea to check
this in a production application (e.g limiting payload size).
1.) Make a struct which will hold the data coming from an API post request in GoLang
2.) Convert your request body into a byte array []byte
3.) Unmarshal your []byte into a single instance of your struct made earlier.
I'll put an example below:
1. Make a struct which you'll put your JSON into.
Let's take an example of a simple blog post.
The JSON object looks like this and has a slug, a title, and description
{
"slug": "test-slug",
"title": "This is the title",
"body": "This is the body of the page"
}
So your struct would look like this:
type Page struct {
Slug string `json:"slug"`
Title string `json:"title"`
Body string `json:"body"`
}
2 - 3. Get the body of your request and convert it to byte[] Then take that string and Unmarshal it into an instance of your struct.
The data of a post request is the request 'Body'.
In Golang the request in almost all cases (unless you're using something fancy outside of the defaults) will be an http.Request object. This is the 'r' which you normally have in your normal code and it holds the 'Body' of our POST request.
import (
"encoding/json"
"github.com/go-chi/chi" // you can remove
"github.com/go-chi/render" // you can remove but be sure to remove where it is used as well below.
"io/ioutil"
"net/http"
)
func GiveMeAPage(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("A page"))
}
So what we are going to do here is convert an io.ReadCloser, which is what the http.Request.Body is, to a []byte as the Unmarshal function takes a []byte type. I've commented inline below for you.
func Create(w http.ResponseWriter, r *http.Request) {
var p Page //Create an instance of our struct
//Read all the data in r.Body from a byte[], convert it to a string, and assign store it in 's'.
s, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err) // This would normally be a normal Error http response but I've put this here so it's easy for you to test.
}
// use the built in Unmarshal function to put the string we got above into the empty page we created at the top. Notice the &p. The & is important, if you don't understand it go and do the 'Tour of Go' again.
err = json.Unmarshal(s, &p)
if err != nil {
panic(err) // This would normally be a normal Error http response but I've put this here so it's easy for you to test.
}
// From here you have a proper Page object which is filled. Do what you want with it.
render.JSON(w, r, p) // This is me using a useful helper function from go-chi which 'Marshals' a struct to a json string and returns it to using the http.ResponseWriter.
}
As a side note. Please don't use Decoder to parse JSON unless you are using JSON streams. You are not here, and it's unlikely you will
for a while. You can read about why that is
here
If you just want the raw JSON data without parsing it, http.Request.Body implements io.Reader, so you can just Read from it. For example with ioutil.ReadAll.
Something like (untested):
func Point(w http.ResponseWriter, r *http.Request) {
data, err := ioutil.ReadAll(r.Body)
// check error
// do whatever you want with the raw data (you can `json.Unmarshal` it too)
}
I have a very simple Go webserver. It's job is to receive an inbound json payload. It then publishes the payload to one or more services that expect a byte array. The payload doesn't need to be checked. Just sent over.
In this case, it receives an inbound job and sends it to Google PubSub. It might be another service - it doesn't really matter. I'm trying to find the most efficient way to convert the object to a byte array without first decoding it.
Why? Seems a bit wasteful to decode and convert to JSON on one server, only to unmarshal it later. Plus, I don't want to maintain two identical structs in two packages.
How is it possible to convert the io.ReadCloser to a byte array so I only need to unmarshal once. I tried something like this answer but don't think that's the most efficient way either:
From io.Reader to string in Go
My http server code looks like this:
func Collect(d DbManager) http.HandlerFunc {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
code := 422
obj := Report{}
response := Response{}
response.Message = "Invalid request"
decoder := json.NewDecoder(r.Body)
decoder.Decode(&obj)
if obj.Device.MachineType != "" {
msg,_ := json.Marshal(obj)
if d.Publish(msg, *Topic) {
code = 200
}
response.Message = "Ok"
}
a, _ := json.Marshal(response)
w.WriteHeader(code)
w.Write(a)
return
})
}
You convert a Reader to bytes, by reading it. There's not really a more efficient way to do it.
body, err := ioutil.ReadAll(r.Body)
If you are unconditionally transferring bytes from an io.Reader to an io.Writer, you can just use io.Copy
I'm trying to decode dynamic/random JSON responses in GO, with nested data
body, _ := ioutil.ReadAll(response.Body)
resp := make(map[string]interface{})
err = json.Unmarshal(body, &resp)
fmt.Printf("BODY: %T<\n", body)
fmt.Printf("BODY: %s<\n", body)
fmt.Printf("RESP: %s<\n", resp)
fmt.Printf("RESP: %T<\n", resp)
fmt.Printf("RESP[results]: %T<\n", resp["results"])
fmt.Printf("RESP[results]: %s<\n", resp["results"])
body is the JSON result from the HTTP server and I unmarshall it and the result looks to be a slice of bytes.
BODY: []uint8
BODY: {"results":[{"code":500.0,"errors":["Configuration file 'c2-web-2.conf' already exists."],"status":"Object could not be created."}]}
So I unmarshall it into resp and that works as expected.
RESP: map[string]interface {}
RESP: map[results:[map[code:%!s(float64=500) errors:[Configuration file 'c2-web-2.conf' already exists.] status:Object could not be created.]]]<
I'm able to access the map with the key results.
RESP[results]: []interface {}
RESP[results]: [map[code:%!s(float64=500) errors:[Configuration file 'conf.d/hosts/c2-web-2.conf' already exists.] status:Object could not be created.]]<
Now what i want to access it the "code", "errors" and "status" which is in resp["results"] This looks like an array or slice and I've tried indexing it but I get the error at compile time
./create_host.go:62: invalid operation: resp["results"][0] (type interface {} does not support indexing)
I've done a lot of googling, tried unmarshalling the data within resp["results"] etc, but after a few days I have not made much progress.
How should I access the map which seems to be a member of an array? The data structure is not guaranteed so I can't create a structure and unmarshall into that.
Thanks
A co-worker provided the code fragement below which made it possible to access the map entries I was looking for.
respBody, _ := ioutil.ReadAll(response.Body)
var rsp interface{}
if err := json.Unmarshal(respBody, &rsp); err != nil {
log.Fatal(err)
}
resultMap := rsp.(map[string]interface{})["results"].([]interface{})[0].(map[string]interface{})
fmt.Printf("test: %s<\n", resultMap["errors"] )
test: [Configuration file 'c2-web-2.conf' already exists.]<
I believe you need to do a type assertion. You have an interface{}, but you need some sort of slice to index into. Try resp["results"].([]interface{})[0]? (Sorry, haven't had a chance to test this myself.)
I am using gin as my http server and sending back an empty array in json as my response:
c.JSON(http.StatusOK, []string{})
The resulting json string I get is "[]\n". The newline is added by the json Encoder object, see here.
Using goconvey, I could test my json like
So(response.Body.String(), ShouldEqual, "[]\n")
But is there a better way to generate the expected json string than just adding a newline to all of them?
You should first unmarshal the body of the response into a struct and compare against the resulting object. Example:
result := []string{}
if err := json.NewDecoder(response.Body).Decode(&result); err != nil {
log.Fatalln(err)
}
So(len(result), ShouldEqual, 0)
You may find jsonassert useful.
It has no dependencies outside the standard library and allows you to verify that JSON strings are semantically equivalent to a JSON string you expect.
In your case:
// white space is ignored, no need for \n
jsonassert.New(t).Assertf(response.Body().String(), "[]")
It can handle any form of JSON, and has very friendly assertion error messages.
Disclaimer: I wrote this package.
Unmarshal the body into a struct and the use Gocheck's DeepEquals
https://godoc.org/launchpad.net/gocheck
I made it this way. Because I don't want to include an extra library.
tc := testCase{
w: httptest.NewRecorder(),
wantResponse: mustJson(t, map[string]string{"message": "unauthorized"}),
}
...
if tc.wantResponse != tc.w.Body.String() {
t.Errorf("want %s, got %s", tt.wantResponse, tt.w.Body.String())
}
...
func mustJson(t *testing.T, v interface{}) string {
t.Helper()
out, err := json.Marshal(v)
if err != nil {
t.Fatal(err)
}
return string(out)
}
I'm using the Beego framework to build a web application, and I'm trying to hand it some JSON encoded data. Roughly, this is what I have:
import (
"github.com/astaxie/beego"
)
type LoginController struct {
beego.Controller
}
func (this *LoginController) Post() {
request := this.Ctx.Request
length := request.ContentLength
p := make([]byte, length)
bytesRead, err := this.Ctx.Request.Body.Read(p)
if err == nil{
//blah
} else {
//tell me the length, bytes read, and error
}
}
Per this tutorial, the above Should Just Work (tm).
My problem is this: bytesRead, err := this.Ctx.Request.Body.Read(p) is returning 0 bytes read and the err.Error() is EOF.
The request.ContentLength, however, is a sane number of bytes (19 or more, depending on what data I type in).
I can't figure out why the request would appear to have some length, but would fail on Read. Any ideas?
If you are trying to reach a JSON payload in Beego, you'll want to call
this.Ctx.Input.RequestBody
That returns a []byte array of the sent payload. You can then pass it to a function like:
var datapoint Datapoint
json.Unmarshal(this.Ctx.Input.RequestBody, &datapoint)
Where datapoint is the struct you are attempting to unmarshall your data into.