I have a very simple Go webserver. It's job is to receive an inbound json payload. It then publishes the payload to one or more services that expect a byte array. The payload doesn't need to be checked. Just sent over.
In this case, it receives an inbound job and sends it to Google PubSub. It might be another service - it doesn't really matter. I'm trying to find the most efficient way to convert the object to a byte array without first decoding it.
Why? Seems a bit wasteful to decode and convert to JSON on one server, only to unmarshal it later. Plus, I don't want to maintain two identical structs in two packages.
How is it possible to convert the io.ReadCloser to a byte array so I only need to unmarshal once. I tried something like this answer but don't think that's the most efficient way either:
From io.Reader to string in Go
My http server code looks like this:
func Collect(d DbManager) http.HandlerFunc {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
code := 422
obj := Report{}
response := Response{}
response.Message = "Invalid request"
decoder := json.NewDecoder(r.Body)
decoder.Decode(&obj)
if obj.Device.MachineType != "" {
msg,_ := json.Marshal(obj)
if d.Publish(msg, *Topic) {
code = 200
}
response.Message = "Ok"
}
a, _ := json.Marshal(response)
w.WriteHeader(code)
w.Write(a)
return
})
}
You convert a Reader to bytes, by reading it. There's not really a more efficient way to do it.
body, err := ioutil.ReadAll(r.Body)
If you are unconditionally transferring bytes from an io.Reader to an io.Writer, you can just use io.Copy
Related
Using Go, I'm attempting to fetch a few JSON responses concurrently from multiple endpoints. I'd like to attach each of these responses to fields in a struct or map and return this struct/map as a JSON object. (Backend for Frontend pattern). So I will make a web request to the Go application with some sort of identifier. It will in turn make several web requests, and compile the data into one large object to return as a response.
I'm using Fiber as my framework but any generic web framework would be similar:
app.Get("/requests/:identifier", func(c *fiber.Ctx) error {
identifier := c.Params("identifier")
timeout := 1600 * time.Millisecond
client := httpclient.NewClient(httpclient.WithHTTPTimeout(timeout))
res, err := client.Get("https://www.example.com/endpoint?id=" + identifier, nil)
if err != nil{
logger.Error("Timout value exceeded")
return c.Status(503).SendString("Socket Timeout")
}
logger.Info("Fetch success: ")
// Heimdall returns the standard *http.Response object
body, err := ioutil.ReadAll(res.Body)
code := 200
response := &main_response{
Service1: body,
}
return c.Status(code).JSON(response)
})
The problem I'm having is, I have no need to unmarshal this data in Go, as I have no use for it (I am simply passing it along). Do I have to unmarshal it just so I can set it as a field in my response struct like this?
type main_response struct {
Service1 []byte `json:"service1"`
Service2 map[string]string `json:"service2"`
Service3 map[string]interface{} `json:"service3"`
}
(I've tried a few different ways to accomplish this. Trying to use a byte array seems to base64 encode the responses)
I will want to marshal that struct to JSON before returning it, so perhaps I have little choice as I can't think of a way to tell Go "only marshal the main struct, everything else is already JSON". It almost feels like I'd be better off building a string at this point.
Use json.RawMessage to copy a []byte containing JSON directly to the response JSON document:
type main_response struct {
Service1 json.RawMessage `json:"service1"`
...
}
response := &main_response{
Service1: body,
}
return c.Status(code).JSON(response)
so I am trying to POST a csv file in JSON format to a website in Golang. I have been successful in POSTing a singular JSON file. However, that is not what I want the purpose of my program to be. Basically, I'm trying to create an account generator for a site. I want to be able to generate multiple accounts at once. I feel the best way to do this is with a csv file.
I've tried using encoding/csv to read the csv file then marshal to JSON. Also, ioutil.ReadFile(). However, the response from the site is, 'first name is mandatory field, last name is a mandatory field' etc etc. So, this obviously means the csv data is not going into JSON format. I'll show my code with the ioutil.ReadFile() below.
func main() {
file, _ := ioutil.ReadFile("accounts.csv")
jsonConv, _ := json.Marshal(file)
client := http.Client{}
req, err := http.NewRequest("POST", "websiteUrlHere", bytes.NewBuffer(jsonConv))
req.Header.Add("cookie", `"really long cookie goes here"`)
req.Header.Set("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36")
req.Header.Set("content-type", "application/json")
resp, err := client.Do(req)
if err != nil {
fmt.Print(err)
}
defer resp.Body.Close()
}
(^ This is just a snippet of the code).
I'm still pretty new to all of this so please understand if the question lacks anything. I've also looked for similar questions but all I find is the use of a struct. I feel this wouldn't be applicable for this as the goal is to create unlimited accounts at once.
Hope the above is sufficient. Thank you.
The issue with your code is actually that you're trying to convert a file to bytes with:
file, _ := ioutil.ReadFile("accounts.csv")
...and then you are AGAIN trying to convert that slice of bytes to JSON bytes with:
jsonConv, _ := json.Marshal(file)
Where the text contents of the file are stored as a slice of bytes in the variable file, and then that slice of bytes (the file contents in bytes form) is then being converted to a JSON array of bytes. So you are basically sending a JSON array of numbers...not good.
The normal flow here would be to take the file bytes and then create a Go struct(s) out of it. Once your Go objects are in place, THEN you would marshal to JSON. That converts the Go objects to a slice of bytes AFTER it has been converted to JSON text form.
So what you are missing is the Go structure middle step but you also should keep in mind that converting a Go struct to JSON bytes with json.Marshal() will only show fields that are exported. Also, usually you should use struct tags to customize exactly how the fields will show up.
Your best bet is just to stick with JSON, forget about the CSV. In your own code example, you are taking a CSV and then trying to convert it to JSON...so, then, just use JSON.
If you want to send multiple accounts, just make your Go structure a slice, which will marshal into a JSON array, which is basically what you are trying to do. The end result will be a JSON array of accounts. Here's a simple example:
package main
import (
"fmt"
"encoding/json"
)
type Account struct {
Username string `json:"username"`
Email string `json:"email"`
}
type AccountsRequest struct {
Accounts []Account `json:"accounts"`
}
func main() {
//gather account info
acct1 := Account{Username: "tom", Email: "tom#example.com"}
acct2 := Account{Username: "dick", Email: "dick#example.com"}
acct3 := Account{Username: "harry", Email: "harry#example.com"}
//create request
acctsReq := AccountsRequest{Accounts: []Account{acct1, acct2, acct3}}
//convert to JSON bytes/data
//jsonData, _ := json.Marshal(acctsReq)
//debug/output
jsonDataPretty, _ := json.MarshalIndent(acctsReq, "", " ")
fmt.Println(string(jsonDataPretty))
//send request with data
//...
}
Runnable here in playground.
The key is that the structs are set up and ready to go and the struct tags determine what the JSON field names will be (i.e. username & email for each account and accounts for the overall array of accounts).
Hope that helps. Drop a comment if you need more specific help.
You need to parse the CSV file first and convert it into the list that you want:
package main
func main() {
file, err := os.Open("file.csv")
if err != nil {
log.Fatal("failed opening file because: %s", err.Error())
}
r := csv.NewReader(file)
records, err := r.ReadAll()
if err != nil {
log.Fatal(err)
}
fmt.Print(records)
}
The above code is parsing the list into a [][]string array. you will now need to iterate over that array and turn it into the json object that the page needs. Then you can send it. You can read more about the csv package here : https://golang.org/pkg/encoding/csv/
A word of advise: never ignore errors, they might give you useful information.
I have an issue printing the JSON from the POST.
I use gorilla/mux for routing
r := mux.NewRouter()
r.HandleFunc("/test", Point).Methods("POST")
http.ListenAndServe(":80", r)`
and in Point function I have
func Point(w http.ResponseWriter, r *http.Request) {
var callback Decoder
json.NewDecoder(r.Body).Decode(&callback)
}
But I can use this method only when I know the structure and I want to figure out how to log.Print the whole JSON as a string. I tried
func Point(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
log.Println(r.Form)
}
But it prints an empty map. Please help to figure out this.
In the assumption that you are building a standard API endpoint which receives some JSON and you would like to do something with it you should approach it the following way.
Edit:
As mentioned in the comments when you use the ioutil.ReadAll()
function as in this example it will read everything that is sent in the
post request into the application memory. It's a good idea to check
this in a production application (e.g limiting payload size).
1.) Make a struct which will hold the data coming from an API post request in GoLang
2.) Convert your request body into a byte array []byte
3.) Unmarshal your []byte into a single instance of your struct made earlier.
I'll put an example below:
1. Make a struct which you'll put your JSON into.
Let's take an example of a simple blog post.
The JSON object looks like this and has a slug, a title, and description
{
"slug": "test-slug",
"title": "This is the title",
"body": "This is the body of the page"
}
So your struct would look like this:
type Page struct {
Slug string `json:"slug"`
Title string `json:"title"`
Body string `json:"body"`
}
2 - 3. Get the body of your request and convert it to byte[] Then take that string and Unmarshal it into an instance of your struct.
The data of a post request is the request 'Body'.
In Golang the request in almost all cases (unless you're using something fancy outside of the defaults) will be an http.Request object. This is the 'r' which you normally have in your normal code and it holds the 'Body' of our POST request.
import (
"encoding/json"
"github.com/go-chi/chi" // you can remove
"github.com/go-chi/render" // you can remove but be sure to remove where it is used as well below.
"io/ioutil"
"net/http"
)
func GiveMeAPage(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("A page"))
}
So what we are going to do here is convert an io.ReadCloser, which is what the http.Request.Body is, to a []byte as the Unmarshal function takes a []byte type. I've commented inline below for you.
func Create(w http.ResponseWriter, r *http.Request) {
var p Page //Create an instance of our struct
//Read all the data in r.Body from a byte[], convert it to a string, and assign store it in 's'.
s, err := ioutil.ReadAll(r.Body)
if err != nil {
panic(err) // This would normally be a normal Error http response but I've put this here so it's easy for you to test.
}
// use the built in Unmarshal function to put the string we got above into the empty page we created at the top. Notice the &p. The & is important, if you don't understand it go and do the 'Tour of Go' again.
err = json.Unmarshal(s, &p)
if err != nil {
panic(err) // This would normally be a normal Error http response but I've put this here so it's easy for you to test.
}
// From here you have a proper Page object which is filled. Do what you want with it.
render.JSON(w, r, p) // This is me using a useful helper function from go-chi which 'Marshals' a struct to a json string and returns it to using the http.ResponseWriter.
}
As a side note. Please don't use Decoder to parse JSON unless you are using JSON streams. You are not here, and it's unlikely you will
for a while. You can read about why that is
here
If you just want the raw JSON data without parsing it, http.Request.Body implements io.Reader, so you can just Read from it. For example with ioutil.ReadAll.
Something like (untested):
func Point(w http.ResponseWriter, r *http.Request) {
data, err := ioutil.ReadAll(r.Body)
// check error
// do whatever you want with the raw data (you can `json.Unmarshal` it too)
}
I'm trying to decode dynamic/random JSON responses in GO, with nested data
body, _ := ioutil.ReadAll(response.Body)
resp := make(map[string]interface{})
err = json.Unmarshal(body, &resp)
fmt.Printf("BODY: %T<\n", body)
fmt.Printf("BODY: %s<\n", body)
fmt.Printf("RESP: %s<\n", resp)
fmt.Printf("RESP: %T<\n", resp)
fmt.Printf("RESP[results]: %T<\n", resp["results"])
fmt.Printf("RESP[results]: %s<\n", resp["results"])
body is the JSON result from the HTTP server and I unmarshall it and the result looks to be a slice of bytes.
BODY: []uint8
BODY: {"results":[{"code":500.0,"errors":["Configuration file 'c2-web-2.conf' already exists."],"status":"Object could not be created."}]}
So I unmarshall it into resp and that works as expected.
RESP: map[string]interface {}
RESP: map[results:[map[code:%!s(float64=500) errors:[Configuration file 'c2-web-2.conf' already exists.] status:Object could not be created.]]]<
I'm able to access the map with the key results.
RESP[results]: []interface {}
RESP[results]: [map[code:%!s(float64=500) errors:[Configuration file 'conf.d/hosts/c2-web-2.conf' already exists.] status:Object could not be created.]]<
Now what i want to access it the "code", "errors" and "status" which is in resp["results"] This looks like an array or slice and I've tried indexing it but I get the error at compile time
./create_host.go:62: invalid operation: resp["results"][0] (type interface {} does not support indexing)
I've done a lot of googling, tried unmarshalling the data within resp["results"] etc, but after a few days I have not made much progress.
How should I access the map which seems to be a member of an array? The data structure is not guaranteed so I can't create a structure and unmarshall into that.
Thanks
A co-worker provided the code fragement below which made it possible to access the map entries I was looking for.
respBody, _ := ioutil.ReadAll(response.Body)
var rsp interface{}
if err := json.Unmarshal(respBody, &rsp); err != nil {
log.Fatal(err)
}
resultMap := rsp.(map[string]interface{})["results"].([]interface{})[0].(map[string]interface{})
fmt.Printf("test: %s<\n", resultMap["errors"] )
test: [Configuration file 'c2-web-2.conf' already exists.]<
I believe you need to do a type assertion. You have an interface{}, but you need some sort of slice to index into. Try resp["results"].([]interface{})[0]? (Sorry, haven't had a chance to test this myself.)
I'm using the Beego framework to build a web application, and I'm trying to hand it some JSON encoded data. Roughly, this is what I have:
import (
"github.com/astaxie/beego"
)
type LoginController struct {
beego.Controller
}
func (this *LoginController) Post() {
request := this.Ctx.Request
length := request.ContentLength
p := make([]byte, length)
bytesRead, err := this.Ctx.Request.Body.Read(p)
if err == nil{
//blah
} else {
//tell me the length, bytes read, and error
}
}
Per this tutorial, the above Should Just Work (tm).
My problem is this: bytesRead, err := this.Ctx.Request.Body.Read(p) is returning 0 bytes read and the err.Error() is EOF.
The request.ContentLength, however, is a sane number of bytes (19 or more, depending on what data I type in).
I can't figure out why the request would appear to have some length, but would fail on Read. Any ideas?
If you are trying to reach a JSON payload in Beego, you'll want to call
this.Ctx.Input.RequestBody
That returns a []byte array of the sent payload. You can then pass it to a function like:
var datapoint Datapoint
json.Unmarshal(this.Ctx.Input.RequestBody, &datapoint)
Where datapoint is the struct you are attempting to unmarshall your data into.