Generic handling of JSON requests and responses - json

Assume a Go program with several handler functions like this:
type FooRequest struct {
FooField string `json:"foofield"`
// ...
}
type FooResponse struct {
BarField string `json:"barfield"`
// ...
}
func handleFoo(w http.ResponseWriter, r *http.Request) {
var req FooRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// do what actually matters:
foo := DoStuff(req)
baz, err := DoSomething(foo)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
resp := DoEvenMoreStuff(baz)
// back to boiler plate:
if err := json.NewEncoder(w).Encode(resp); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
How could this code be refactored to avoid the JSON decoding/encoding boiler plate?
I would probably like to see a generic "handle JSON" func and another func handling the actual foo stuff:
func handleJson(w http.ResponseWriter, r *http.Request) {
var req FooRequest // what about this line?
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
resp, err := handleFooElegantly(req)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if err := json.NewEncoder(w).Encode(resp); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
func handleFoo(req FooRequest) (FooResponse, error) {
var resp FooResponse
foo := DoStuff(req)
baz, err := DoSomething(foo)
if err != nil {
return resp, err
}
resp = DoEvenMoreStuff(baz)
return resp, nil
}
That leaves us with the problem of telling the JSON decoder the type it should try to decode.
What would be the idiomatic Go way of implementing this?

You can use reflection to eliminate the boilerplate. Here's an example that adapts a func(pointer) (pointer, error) to an http.Handler that handles the JSON encoding and decoding.
type jsonHandler struct {
fn interface{}
}
func (jh jsonHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// allocate the input argument
v := reflect.ValueOf(jh.fn)
arg := reflect.New(v.Type().In(0).Elem())
// decode to the argument
if err := json.NewDecoder(r.Body).Decode(arg.Interface()); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// invoke the function
out := v.Call([]reflect.Value{arg})
// check error
if err, ok := out[1].Interface().(error); ok && err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// decode result
if err := json.NewEncoder(w).Encode(out[0].Interface()); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
}
Here's an example of how to use this adapter:
type Input struct {
A, B int
}
type Output struct {
Result int
}
func add(in *Input) (*Output, error) {
return &Output{Result: in.A + in.B}, nil
}
handler := jsonHandler{add} // handler satisfies http.Handler
working playground example
The code will panic if the function does not have a single pointer argument and return a pointer and an error. A more robust and complete implementation should check that the function meets these constraints before returning the handler.
func newHandler(fn interface{)) (http.Handler, error) {
// use reflect to check fn, return error
...
return jsonHandler{fn}, nil
}
The use of reflection in this answer is somewhat similar to an approach used in the standard library.

Related

How to set up web server to perform POST Request in Go?

I want to set up a web server to perform a POST request. How does the post request get executed with the code below since only HandleFunc and ListenAndServe are defined in main function?
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"strings"
)
func post(w http.ResponseWriter, r *http.Request) {
const myurl string = "http://localhost:8000/"
request := strings.NewReader(`
{
"Name":"Tom",
"Age":"20"
}
`)
response, err := http.Post(myurl, "application/json", request)
content, err := ioutil.ReadAll(response.Body)
if err != nil {
panic(err)
}
fmt.Println(string(content))
defer response.Body.Close()
}
func main() {
http.HandleFunc("/", post)
log.Fatal(http.ListenAndServe(":8000", nil))
}
Here is a basic example of how you could go about it. I am using the same program to run both, the server and the client. This is just for demonstration purposes. You can of course make them separate programs.
// use struct to represent the data
// to recieve and send
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
}
// run the example
func main() {
// start the server in a goroutine
go server()
// wait 1 second to give the server time to start
time.Sleep(time.Second)
// make a post request
if err := client(); err != nil {
fmt.Println(err)
}
}
// basic web server to receive a request and
// decode the body into a user struct
func server() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
user := &Person{}
err := json.NewDecoder(r.Body).Decode(user)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
fmt.Println("got user:", user)
w.WriteHeader(http.StatusCreated)
})
if err := http.ListenAndServe(":8080", nil); err != http.ErrServerClosed {
panic(err)
}
}
// a simple client that posts a user to the server
func client() error {
user := &Person{
Name: "John",
Age: 30,
}
b := new(bytes.Buffer)
err := json.NewEncoder(b).Encode(user)
if err != nil {
return err
}
resp, err := http.Post("http://localhost:8080/", "application/json", b)
if err != nil {
return err
}
defer resp.Body.Close()
fmt.Println(resp.Status)
return nil
}
Here is the working example: https://go.dev/play/p/34GT04jy_uA

How to process a request that has multiple inputs and multiple files at the same time

Building a backend go server that can take a form with multiple inputs and 3 of them have multiple file inputs. I searched and it states that if you want to make something like this work you don't want to use the typical
if err := r.ParseMultipartForm(32 << 20); err != nil {
fmt.Println(err)
}
// get a reference to the fileHeaders
files := r.MultipartForm.File["coverArt"]
and instead you should use
mr, err := r.MultipartReader()
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
Standard form-data:
Name
Email
Cover art photos (multiple files)
Profile photos (multiple files)
2 Audio files (2 songs)
2 Videos (personal intro, recording of person in a cappella)
HTML Form
<form method="post" enctype="multipart/form-data" action="/upload">
<input type="text" name="name">
<input type="text" name="email">
<input name="coverArt" type="file" multiple />
<input name="profile" type="file" multiple />
<input type="file" name="songs" multiple />
<input type="file" name="videos" multiple/>
<button type="submit">Upload File</button>
</form>
Go Code:
func FilePOST(w http.ResponseWriter, r *http.Request) error {
fmt.Println("File Upload Endpoint Hit")
mr, err := r.MultipartReader()
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
for {
part, err := mr.NextPart()
// This is OK, no more parts
if err == io.EOF {
break
}
// Some error
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
// CoverArt 'files' part
if part.FormName() == "coverArt" {
name := part.FileName()
outfile, err := os.Create("uploads/" + name)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
// return
}
defer outfile.Close()
_, err = io.Copy(outfile, part)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
// return
}
}
// Profile Pic 'files' part
if part.FormName() == "profile" {
name := part.FileName()
outfile, err := os.Create("uploads/" + name)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
// return
}
defer outfile.Close()
_, err = io.Copy(outfile, part)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
// return
}
}
// Songs 'files' part
if part.FormName() == "songs" {
name := part.FileName()
outfile, err := os.Create("uploads/" + name)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
// return
}
defer outfile.Close()
_, err = io.Copy(outfile, part)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
// return
}
}
// Video 'files' part
if part.FormName() == "videos" {
name := part.FileName()
outfile, err := os.Create("uploads/" + name)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
// return
}
defer outfile.Close()
_, err = io.Copy(outfile, part)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
// return
}
}
}
fmt.Println("done")
return nil
}
Go Server Error:
go run main.go [15:58:21]
now serving at the following location www.localhost:3000
File Upload Endpoint Hit
INFO[0009] POST /upload elapsed="680.422µs" host= method=POST path=/upload query=
2021/07/14 15:58:32 http: panic serving [::1]:62924: runtime error: invalid memory address or nil pointer dereference
It is hard to guess where your code panics. Probably the reason is that your program continue to execute when error occurs. For example if creation of file fails, outfile.Close() will panic as the outfile is nil.
Both approaches support multiple files for single field. The difference is in how they handle memory. The streaming version reads small portions of data from the network and writes it to a file when you call io.Copy. The other variant loads all the data into memory when you call ParseMultiForm(), so it requires as much memory as the size of the files you want to transfer. Below you will find working examples for both variants.
Streaming variant:
func storeFile(part *multipart.Part) error {
name := part.FileName()
outfile, err := os.Create("uploads/" + name)
if err != nil {
return err
}
defer outfile.Close()
_, err = io.Copy(outfile, part)
if err != nil {
return err
}
return nil
}
func filePOST(w http.ResponseWriter, r *http.Request) error {
fmt.Println("File Upload Endpoint Hit")
mr, err := r.MultipartReader()
if err != nil {
return err
}
for {
part, err := mr.NextPart()
// This is OK, no more parts
switch {
case errors.Is(err, io.EOF):
fmt.Println("done")
return nil
case err != nil:
// Some error
return err
default:
switch part.FormName() {
case "coverArt", "profile", "songs", "videos":
if err := storeFile(part); err != nil {
return err
}
}
}
}
}
func main() {
http.HandleFunc("/upload", func(writer http.ResponseWriter, request *http.Request) {
err := filePOST(writer, request)
if err != nil {
http.Error(writer, err.Error(), http.StatusInternalServerError)
log.Println("Error", err)
}
})
if err := http.ListenAndServe(":8080", nil); err != nil {
log.Fatal(err)
}
}
And version with ParseMultipartForm, which reads data to memory.
func storeFile(part *multipart.FileHeader) error {
name := part.Filename
infile, err := part.Open()
if err != nil {
return err
}
defer infile.Close()
outfile, err := os.Create("uploads/" + name)
if err != nil {
return err
}
defer outfile.Close()
_, err = io.Copy(outfile, infile)
if err != nil {
return err
}
return nil
}
func FilePOST(w http.ResponseWriter, r *http.Request) error {
fmt.Println("File Upload Endpoint Hit")
if err := r.ParseMultipartForm(2 << 24); err != nil {
return err
}
for _, fileType := range []string{"coverArt", "profile", "songs", "videos"} {
uploadedFiles, exists := r.MultipartForm.File[fileType]
if !exists {
continue
}
for _, file := range uploadedFiles {
if err := storeFile(file); err != nil {
return err
}
}
}
return nil
}
func main() {
http.HandleFunc("/upload", func(writer http.ResponseWriter, request *http.Request) {
err := FilePOST(writer, request)
if err != nil {
http.Error(writer, err.Error(), http.StatusInternalServerError)
log.Println("Error", err)
}
})
if err := http.ListenAndServe(":8080", nil); err != nil {
log.Fatal(err)
}
}

Golang error: interface conversion: interface {} is bool/float..., not string

I am trying to decode an arbitrary JSON using Golang, so I unmarshal the incoming JSON in a map[string]interface{} as shown in the code below:
func JsonHandler(jsonRequest []byte) {
// Creating the maps for JSON
var m interface{}
// Parsing/Unmarshalling JSON encoding/json
if err := json.Unmarshal([]byte(jsonRequest), &m); err != nil {
panic(err)
}
//Creating an output file for writing
f, err := os.OpenFile("/home/dorrahadrich/Desktop/output.txt", os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
if err != nil {
panic(err)
}
defer f.Close()
ParseJson(m, f, err)
}
func ParseJson(m interface{}, f *os.File, err error) {
switch v := m.(interface{}).(type){
case map[string]interface{}:
ParseMap (m.(map[string]interface{}),f,err)
fmt.Println(v)
case []interface{}:
ParseArray (m.([]interface{}),f,err)
fmt.Println(v)
default:
}
}
func ParseMap(aMap map[string]interface{}, f *os.File, err error) {
for key, val := range aMap {
switch val.(type) {
case map[string]interface{}:
if _, err = f.WriteString(key + "={\n"); err != nil {
panic(err)
}
ParseMap(val.(map[string]interface{}), f, err)
//Close brackets
if _, err = f.WriteString("};\n"); err != nil {
panic(err)
}
case []interface{}:
//Write to file
if _, err = f.WriteString(key + "={\n"); err != nil {
panic(err)
}
ParseArray(val.([]interface{}), f, err)
//Close brackets
if _, err = f.WriteString("};\n"); err != nil {
panic(err)
}
default:
otherValues(key, val.(interface{}), f , err)
}
}
}
func ParseArray(anArray []interface{}, f *os.File, err error) {
for _, val := range anArray {
switch val.(type) {
case map[string]interface{}:
ParseMap(val.(map[string]interface{}), f, err)
case []interface{}:
ParseArray(val.([]interface{}), f, err)
default:
}
}
}
func otherValues(key string, other interface{}, f *os.File, err error) {
if _, err = f.WriteString(key); err != nil {
panic(err)
}
if _, err = f.WriteString("="); err != nil {
panic(err)
}
switch other.(interface{}).(type) {
case string:
if _, err = f.WriteString(other.(string)); err != nil {
panic(err)
}
case float64:
if _, err = f.WriteString(strconv.FormatFloat(other.(float64), 'f', -1, 64)); err != nil {
panic(err)
}
case bool:
if _, err = f.WriteString(strconv.FormatBool(other.(bool))); err != nil {
panic(err)
}
default:
}
}
The problem is that whenever a JSON contains a bool/int/float or any not string value the program panics saying that it fails converting an interface to the given type! Please note that the JSON is arbitrary so I don't have any idea about the keys nor the values, I can't unmrashal into an interface nor access the values giving a path.
The error says it all:
interface conversion: interface{} is bool/float64
when you are unmarshalling json the values for int and bool which are not of interface type. In your switch add case for bool/float64/string too. Since json is arbitrary unmarshal them using interface{}.
func otherValues(other interface{}, f *os.File, err error) {
switch bb := other.(interface{}).(type) {
case string:
fmt.Println("This is a string")
case float64:
fmt.Println("this is a float")
case bool:
fmt.Println("this is a boolean")
default:
fmt.Printf("Default value is of type %v", bb)
}
}
Use file.Write in place of file.WriteString
func (f *File) Write(b []byte) (n int, err error)
Write writes len(b) bytes to the File. It returns the number of bytes
written and an error, if any. Write returns a non-nil error when n !=
len(b).

How to write unit test for failure case of json.Marshall?

I have following code:
func policyDocumentToStr(doc map[string]interface{}) (*string, error) {
policy, err := json.Marshal(doc)
if err != nil {
log.Debugf("Error converting policy document to string. Error %s", err)
return nil, err
}
policyAsString := string(policy)
return &policyAsString, nil
}
I want to write a unit test which would cover the case of json.Marshal(doc) returning an error. Can anybody suggest how can I generate an error? What kind of input to function would result in error at line policy, err := json.Marshal(doc)?
Feed it a value that cannot be represented in JSON. There are many ways to do this, including creating a type with a custom marshaler that always returns an error. But one of the simplest ways is to try to marshal a channel:
x := map[string]interface{}{
"foo": make(chan int),
}
_, err := json.Marshal(x)
fmt.Printf("Marshal error: %s\n", err)
Playground link
Define a type that implements json.Marshaler. Use this type to produce any error you want (including error values from the json package):
type FakeValue struct {
err error
}
func (v FakeValue) MarshalJSON() ([]byte, error) {
if v.err != nil {
return nil, v.err
}
return []byte(`null`), v.err
}
func TestPolicyString(t *testing.T) {
doc := map[string]interface{}{
"fake_error": FakeValue{errors.New("fail!")},
}
_, err := policyDocumentToStr(doc)
if err == nil {
t.Fatal("Got nil, want error")
}
}
Monkey patching is one of the ways to facilitate testing by changing a program at runtime, either by replacing a function or a variable. Here we are swapping the json.Marshal and we can mock it while testing.
var (
jsonMarshal = json.Marshal
)
This is your piece of code
func policyDocumentToStr(doc map[string]interface{}) (*string, error) {
policy, err := jsonMarshal(doc)
if err != nil {
fmt.Printf("Error converting policy document to string. Error %s", err)
return nil, err
}
policyAsString := string(policy)
return &policyAsString, nil
}
The test code would look something like this
import (
"testing"
"errors"
)
func fakemarshal(v interface{}) ([]byte, error) {
return []byte{}, errors.New("Marshalling failed")
}
func restoremarshal(replace func(v interface{}) ([]byte, error)) {
jsonMarshal = replace
}
func TestPolicyDocumentToStr(t * testing.T){
storedMarshal := jsonMarshal
jsonMarshal = fakemarshal
defer restoremarshal(storedMarshal)
input := map[string]interface{} {
"test": "test1",
}
tests := []struct {
name string
arg map[string]interface{}
wantErr string
}{
{
name: "Test if JSON Marshalling fails",
arg: input,
wantErr: "Marshalling failed",
},
}
for _, tt := range tests {
_, gotErr := policyDocumentToStr(tt.arg)
if gotErr != nil && gotErr.Error() != tt.wantErr {
t.Errorf("Expected %s but got %s", tt.wantErr, gotErr.Error())
}
}
}

How to unmarshal JSON into durations?

What is the idiomatic way to unmarshal into time.Duration in Go? How can I make use of time.ParseDuration?
The lack of JSON marshaling and unmarshaling methods on time.Duration was an unfortunate oversight. This should hopefully be resolved in Go2 (see issue #10275).
You can, however, define your own type around time.Duration that supports marshaling to the string representation of the duration and unmarshaling from either the numeric or string representations. Here is an example of such an implementation:
package main
import (
"encoding/json"
"errors"
"fmt"
"time"
)
type Duration struct {
time.Duration
}
func (d Duration) MarshalJSON() ([]byte, error) {
return json.Marshal(d.String())
}
func (d *Duration) UnmarshalJSON(b []byte) error {
var v interface{}
if err := json.Unmarshal(b, &v); err != nil {
return err
}
switch value := v.(type) {
case float64:
d.Duration = time.Duration(value)
return nil
case string:
var err error
d.Duration, err = time.ParseDuration(value)
if err != nil {
return err
}
return nil
default:
return errors.New("invalid duration")
}
}
type Message struct {
Elapsed Duration `json:"elapsed"`
}
func main() {
msgEnc, err := json.Marshal(&Message{
Elapsed: Duration{time.Second * 5},
})
if err != nil {
panic(err)
}
fmt.Printf("%s\n", msgEnc)
var msg Message
if err := json.Unmarshal([]byte(`{"elapsed": "1h"}`), &msg); err != nil {
panic(err)
}
fmt.Printf("%#v\n", msg)
}
https://play.golang.org/p/Zm6hpNR-ZJ2
Just to extend the previous answer. There is another way (very close to Tim's)
type Duration time.Duration
func (d Duration) MarshalJSON() ([]byte, error) {
return json.Marshal(time.Duration(d).String())
}
func (d *Duration) UnmarshalJSON(b []byte) error {
var v interface{}
if err := json.Unmarshal(b, &v); err != nil {
return err
}
switch value := v.(type) {
case float64:
*d = Duration(time.Duration(value))
return nil
case string:
tmp, err := time.ParseDuration(value)
if err != nil {
return err
}
*d = Duration(tmp)
return nil
default:
return errors.New("invalid duration")
}
}