I'm developing a web service and part of that I read the Request.Body and try to unmarshal it.
if err := json.NewDecoder(body).Decode(r); err !=nil{
log.Error(err)
return err
}
The issue is that sometimes the client is sending an empty body and I get a panic runtime error: index out of range
goroutine 7 [running]:
How am I supposed to mitigate this?
I am decomposing your code:
NewDecoder: -
func NewDecoder(r io.Reader) *Decoder
NewDecoder returns a new decoder that reads from r. The decoder
introduces its own buffering and may read data from r beyond the JSON
values requested.
So NewDecoder only reads data from r. it's not care, is r empty...
Decode:-
func (dec *Decoder) Decode(v interface{}) error
Decode reads the next JSON-encoded value from its input and stores it
in the value pointed to by v.
See the documentation for Unmarshal for details about the conversion of JSON into a Go value.
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
If a JSON value is not appropriate for a given target type, or if a
JSON number overflows the target type, Unmarshal skips that field and
completes the unmarshalling as best it can. If no more serious errors
are encountered, Unmarshal returns an UnmarshalTypeError describing
the earliest such error.
The JSON null value unmarshals into an interface, map, pointer, or
slice by setting that Go value to nil. Because null is often used in
JSON to mean “not present,” unmarshaling a JSON null into any other Go
type has no effect on the value and produces no error.
Reading above statement, it's clear that there is not chances, we get run time panic error. I was experimenting with a sample code to reproduce this ERROR. May error coming from inside the JSON package or your own code.
var dummy []byte
dummy = make([]byte, 10)
size, _ := body.Read(dummy)
if size > 0 {
if err := json.NewDecoder(body).Decode(r); err != nil {
log.Error(err)
return err
}
fmt.Fprintf(w, "%s", "Json cannot be empty")// where w is http.ResponseWriter
Related
I am working on a general JSON based message passing protocol in Go. What I would like to do is have a BaseMessage that has general information like Type, timestamp, etc. But at the same time I want to be able to define more specific message structures for certain types of data.
For example:
type Message struct {
Type string `json:type`
Timestamp string `json:timestamp`
}
type EventMessage struct {
Message
EventType string
EventCreator string
EventData interface{}
}
I have a set of handlers and to determine which handler should process the message I decode the JSON to the general Message type first to check the Type field. For this example I would get the handler associated with an "Event" message type.
I run into problems when I then want to assert the EventMessage type onto the structure.
The following code is very rough, but hopefully it displays my general idea of how I am trying to handle the messages.
type Handler func(msg Message) Message
handlers := make(map[string]Handler)
var msg Message
decoder.Decode(&msg)
handler := handlers[msg.Type]
handler(msg)
I have tried to use an interface{} but then the JSON decoder just creates a map which I then cant assert either type on. I have figured out workarounds that make it possible, but its very ugly, probably not efficient, and most likely error prone. I would like to keep things simple and straightforward so this code can be easily maintained.
Is there a method of handling generic JSON objects in Go so that the decoded JSON could be one of many struct formats?
I have also played with the idea of having more specific info in a Data interface{} in the main Message struct, but then I run into the same problem of not being able to assert any types onto the interface. There has to be a better way to handle JSON formats that I am just missing.
One way to handle this is to define a struct for the fixed part of the message with a json.RawMessage field to capture the variant part of the message. Decode the json.RawMessage to a type specific to the variant:
type Message struct {
Type string `json:"type"`
Timestamp string `json:"timestamp"`
Data json.RawMessage
}
type Event struct {
Type string `json:"type"`
Creator string `json:"creator"`
}
var m Message
if err := json.Unmarshal(data, &m); err != nil {
log.Fatal(err)
}
switch m.Type {
case "event":
var e Event
if err := json.Unmarshal([]byte(m.Data), &e); err != nil {
log.Fatal(err)
}
fmt.Println(m.Type, e.Type, e.Creator)
default:
log.Fatal("bad message type")
}
playground example
I have a big.float which I'm encoding into JSON . However the JSON always end up showing the float in scientific notation rater than decimal notation. I can fix this by changing the JSON to be a string rather than a number and using float.Text('f'), however I would really prefer to keep the type as a number.
I was a taking a look at float.Format but I don't believe this is suitable.
A really condensed gist of what I'm doing is below. I do a lot more modification of the value of supply before encoding it to json.
type TokenSupply struct {
TotalSupply *big.Float `json:"totalSupply, omitempty"`
}
supply := Float.NewFloat(1000000)
json.NewEncoder(w).Encode(TokenSupply{supply})
This returns:
{"totalSupply":"1e+06"}
big.Float is marshaled to string when converted to a JSON type
https://golang.org/pkg/encoding/json/#Marshal
Marshal traverses the value v recursively. If an encountered value implements the Marshaler interface and is not a nil pointer, Marshal calls its MarshalJSON method to produce JSON. If no MarshalJSON method is present but the value implements encoding.TextMarshaler instead, Marshal calls its MarshalText method and encodes the result as a JSON string. The nil pointer exception is not strictly necessary but mimics a similar, necessary exception in the behavior of UnmarshalJSON.
https://golang.org/pkg/math/big/#Float.MarshalText
func (x *Float) MarshalText() (text []byte, err error)
What can you do about it?
since your float may be more than 64 bits it won't play well with other languages that have to read the JSON value as a number. I'd suggest you keep the number as a string.
Caveats about encoding numbers that don't fit into 64 bits aside, here is how you could marshal a big.Float as a JSON number by wrapping it in a custom type that implements json.Marshaler. The key is that you can implement theMarshalJSON method anyway you like, as long as it emits valid JSON:
package main
import (
"encoding/json"
"fmt"
"math/big"
)
type BigFloatNumberJSON struct{ *big.Float }
func (bfn BigFloatNumberJSON) MarshalJSON() ([]byte, error) {
// Use big.Float.String() or any other string converter
// that emits a valid JSON number here...
return []byte(bfn.String()), nil
}
func main() {
totalSupply := new(big.Float).SetFloat64(1000000)
obj := map[string]interface{}{
"totalSupply": BigFloatNumberJSON{totalSupply},
}
bytes, err := json.Marshal(&obj)
if err != nil {
panic(err)
}
fmt.Println(string(bytes))
// => {"totalSupply":1000000}
}
I have a JSON file stored on the local machine. I need to read it in a variable and loop through it to fetch the JSON object values. If I use the Marshal command after reading the file using the ioutil.Readfile method, it gives some numbers as an output. These are my few failed attempts,
Attempt 1:
plan, _ := ioutil.ReadFile(filename) // filename is the JSON file to read
var data interface{}
err := json.Unmarshal(plan, data)
if err != nil {
log.Error("Cannot unmarshal the json ", err)
}
fmt.Println(data)
It gave me following error,
time="2016-12-13T22:13:05-08:00" level=error msg="Cannot unmarshal the json json: Unmarshal(nil)"
<nil>
Attempt 2: I tried to store the JSON values in a struct and then using MarshalIndent
generatePlan, _ := json.MarshalIndent(plan, "", " ") // plan is a pointer to a struct
fmt.Println(string(generatePlan))
It give me the output as string. But if I cast the output to string then I won't be able to loop it as JSON object.
How can we read a JSON file as JSON object in golang? Is it possible to do that?
Any help is appreciated. Thanks in advance!
The value to be populated by json.Unmarshal needs to be a pointer.
From GoDoc :
Unmarshal parses the JSON-encoded data and stores the result in the value pointed to by v.
So you need to do the following :
plan, _ := ioutil.ReadFile(filename)
var data interface{}
err := json.Unmarshal(plan, &data)
Your error (Unmarshal(nil)) indicates that there was some problem in reading the file , please check the error returned by ioutil.ReadFile
Also please note that when using an empty interface in unmarshal, you would need to use type assertion to get the underlying values as go primitive types.
To unmarshal JSON into an interface value, Unmarshal stores one of
these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
It is always a much better approach to use a concrete structure to populate your json using Unmarshal.
If you're looking at this in 2022, the ioutil package has been deprecated. You can still use it but you'll get annoying errors.
Instead you can use the os package.
someStruct := SomeStruct{} // Read errors caught by unmarshal
fileBytes, _ := os.ReadFile(filename)
err := json.Unmarshal(fileBytes, spec)
Note, I'm specifically ignoring the error from os.ReadFile since it will also cause an error in json.Unmarshal for the sake of the example.
How do you parse YAML data, in Go, without knowing its structure in advance? All of the examples I've seen assume you want to decode a marshaled map whose keys you already know. What if you don't know the keys? What if it's not a map but a marshaled list, a scalar, or some other common type?
Though I'm principally concerned with YAML, here, it seems like the technique might be generally useful for JSON, etc.. since there's a general pattern for parsing structured data (tagged structs, obviously).
For JSON, unmarshal the data to an interface{} value. Use type assertions to determine what's in the value.
var v interface{}
err := json.Unmarshal(data, &v)
if err != nil {
// handle error
}
switch v := v.(type) {
case string:
fmt.Println("string:", v)
case float64:
fmt.Println("number:", v)
case []interface{}:
fmt.Println("array:", v)
case map[string]interface{}:
fmt.Println("object:", v)
case nil:
fmt.Println("nil")
}
In the case of JSON, the standard library json.Unmarshal function will unmarshal arbitrary JSON if you like, if you pass it a pointer to an uninitialized empty interface. (See this example.)
The official docs even say as much:
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
Edit: Though not documented, the same is true of the yaml package; I tested on my workstation, and passing in a pointer to an uninitialized empty interface results in the initialization of correct arrays, maps, and primitives.
I am working on a general JSON based message passing protocol in Go. What I would like to do is have a BaseMessage that has general information like Type, timestamp, etc. But at the same time I want to be able to define more specific message structures for certain types of data.
For example:
type Message struct {
Type string `json:type`
Timestamp string `json:timestamp`
}
type EventMessage struct {
Message
EventType string
EventCreator string
EventData interface{}
}
I have a set of handlers and to determine which handler should process the message I decode the JSON to the general Message type first to check the Type field. For this example I would get the handler associated with an "Event" message type.
I run into problems when I then want to assert the EventMessage type onto the structure.
The following code is very rough, but hopefully it displays my general idea of how I am trying to handle the messages.
type Handler func(msg Message) Message
handlers := make(map[string]Handler)
var msg Message
decoder.Decode(&msg)
handler := handlers[msg.Type]
handler(msg)
I have tried to use an interface{} but then the JSON decoder just creates a map which I then cant assert either type on. I have figured out workarounds that make it possible, but its very ugly, probably not efficient, and most likely error prone. I would like to keep things simple and straightforward so this code can be easily maintained.
Is there a method of handling generic JSON objects in Go so that the decoded JSON could be one of many struct formats?
I have also played with the idea of having more specific info in a Data interface{} in the main Message struct, but then I run into the same problem of not being able to assert any types onto the interface. There has to be a better way to handle JSON formats that I am just missing.
One way to handle this is to define a struct for the fixed part of the message with a json.RawMessage field to capture the variant part of the message. Decode the json.RawMessage to a type specific to the variant:
type Message struct {
Type string `json:"type"`
Timestamp string `json:"timestamp"`
Data json.RawMessage
}
type Event struct {
Type string `json:"type"`
Creator string `json:"creator"`
}
var m Message
if err := json.Unmarshal(data, &m); err != nil {
log.Fatal(err)
}
switch m.Type {
case "event":
var e Event
if err := json.Unmarshal([]byte(m.Data), &e); err != nil {
log.Fatal(err)
}
fmt.Println(m.Type, e.Type, e.Creator)
default:
log.Fatal("bad message type")
}
playground example