Unmarshal field of nested struct not work - json

Given the following structs
type Foo struct {
Thing time.Duration `json:"thing"`
}
type Bar struct {
Foo
Entry time.Duration `json:"entry"`
}
I want to custom time.Duration format and load Bar value from json string like:
{
"thing": "hour",
"entry": "second"
}
So I override UnmarshalJSON for Foo and Bar (https://play.golang.org/p/6v71eG_Xr98):
package main
import (
"encoding/json"
"fmt"
"time"
)
type Foo struct {
Thing time.Duration `json:"thing"`
}
type Bar struct {
Foo
Entry time.Duration `json:"entry"`
}
func timeConvert(s string) time.Duration {
if s == "hour" {
return time.Hour
}
if s == "second" {
return time.Second
}
return time.Duration(0)
}
func (t *Foo) UnmarshalJSON(data []byte) error {
type Alias Foo
a := struct {
Thing string `json:"thing"`
*Alias
}{Alias: (*Alias)(t)}
err := json.Unmarshal(data, &a)
t.Thing = timeConvert(a.Thing)
fmt.Printf("Foo: %v [%v]\n", *t, err)
return err
}
func (t *Bar) UnmarshalJSON(data []byte) error {
type Alias Bar
a := struct {
Entry string `json:"entry"`
*Alias
}{Alias: (*Alias)(t)}
err := json.Unmarshal(data, &a)
t.Entry = timeConvert(a.Entry)
fmt.Printf("Bar: %v [%v]\n", *t, err)
return err
}
func main() {
data := []byte(`{"entry": "second", "thing": "hour"}`)
json.Unmarshal(data, &Bar{})
}
But it outputs unexpected:
Foo: {1h0m0s} [<nil>]
Bar: {{1h0m0s} 0s} [<nil>]
Why Entry value is incorrect?

Thanks for mkopriva. I found out because 'json.Unmarshal' works on any type, it gives me a hint that all type have implemented 'UnmarshalJSON' function, but it's NOT.
The calling of 'json.Unmarshal' on 'Bar' will actually do the following things:
calling UnmarshalJSON on Bar.
Because anonymous struct in Bar don't implemented UnmarshalJSON, so calling UnmarshalJSON on it's embedding struct Foo instead.
That's why 'Entry' in anonymous struct will not be unmarshal.
mkopriva suggest use a custom type to get the custom parsing, but it's inconvenient when you need pass it as argument on functions in the time package. I figure out another way to do it (just unmarshal twice, see https://play.golang.org/p/2ahDX-0hsRt):
func (t *Bar) UnmarshalJSON(data []byte) error {
type Alias Bar
if err := json.Unmarshal(data, (*Alias)(t)); err != nil {
return err
}
var tmp struct {
Entry string `json:"entry"`
}
err := json.Unmarshal(data, &tmp)
t.Entry = timeConvert(tmp.Entry)
fmt.Printf("Bar: %v [%v]\n", *t, err)
return err
}

Related

Why is variable a (json string) not unmarshalling into struct A?

I am probably making a rookie mistake in the code below. Here is the go playground link: https://play.golang.org/p/WS_pPvBHUTH
I am expecting variable a (json string) to be unmarshalled into struct of type A but somehow it ends up being of type map[string]interface{}
package main
import (
"encoding/json"
"fmt"
)
type A struct {
ParamA string
}
type B struct {
ParamB string
}
var types = map[string]func() interface{}{
"ParamA": func() interface{} { return A{} },
"ParamB": func() interface{} { return B{} },
}
func Unmarshal(data []byte, key string) interface{} {
// if bytes.Contains(data, []byte("ParamA")) {
// var a A
// fmt.Printf("%+T\n", a)
// json.Unmarshal(data, &a)
// return a
// }
// var b B
// fmt.Printf("%+T\n", b)
// json.Unmarshal(data, &b)
// return b
if tmp, ok := types[key]; ok {
a := tmp()
fmt.Printf("%+T\n", a)
fmt.Println(string(data))
if err := json.Unmarshal(data, &a); err != nil {
return err
}
fmt.Printf("%+T\n", a)
return a
}
return nil
}
func main() {
a := []byte(`{"ParamA": "aaa"}`)
b := []byte(`{"ParamB": "bbb"}`)
resultA := Unmarshal(a, "ParamA")
resultB := Unmarshal(b, "ParamC")
fmt.Printf("%+v %+v\n", resultA, resultB)
}
Expected Output:
main.A
{"ParamA": "aaa"}
main.A // <= this is what I want
{ParamA:aaa} <nil> // <= this is what I want
Actual Output:
main.A
{"ParamA": "aaa"}
map[string]interface {} // <= but this is what I am getting
map[ParamA:aaa] <nil> // <= but this is what I am getting
Why is variable a (json string) not unmarshalling into struct A? Really stuck here...
Here:
a := tmp()
a is of type interface{}. Then you json.Unmarshal(data,&a), and overwrite contents of a.
What you really want to do is, first return &A{} or &B{} from your factory functions, then:
a:=tmp()
json.Unmarshal(data,a) // Not &a

Best way to handle interfaces in HTTP response

I'm using an API that formats its responses in this way:
{
"err": 0,
"data": **Other json structure**
}
The way I'm getting a response right now is I'm putting the response in an struct like this:
type Response struct {
Err int `json:"err"`
Data interface{} `json:"data"`
}
and then I'm doing this after getting a response
jsonbytes, _ := json.Marshal(resp.Data)
json.Unmarshal(jsonBytes, &dataStruct)
I'm only ignoring errors for this example.
It seems kinda weird to me that I'm marshaling and unmarshaling when I know what the data is supposed to look like and what type it's supposed to be.
Is there a more simple solution that I'm not seeing or is this a normal thing to do?
Edit: I should probably mention that the Data attribute in the response object can vary depending on what API call I'm doing.
The JSON unmarshaller uses reflection to look at the type it is unmarshalling to. Given an uninitialised interface{} as the destination for the unmarshalled data, a JSON object gets unmarshalled into a map[string]interface{} (example in playground).
Here are some ideas.
Option A
If you know the datatype, you can define a new response struct for each type. Example:
type FooResponse struct {
Err int `json:"err"`
Data Foo `json:"data"`
}
type Foo struct {
FooField string `json:"foofield"`
}
type BarResponse struct {
Err int `json:"err"`
Data Bar `json:"data"`
}
type Bar struct {
BarField string `json:"barfield"`
}
Option B
If you prefer to have a single Response struct instead of one per type, you can tell the JSON unmarshaller to avoid unmarshalling the data field until a later time by using the json.RawMessage data type:
package main
import (
"encoding/json"
"fmt"
"log"
)
type Response struct {
Err int `json:"err"`
Data json.RawMessage `json:"data"`
}
type Foo struct {
FooField string `json:"foofield"`
}
type Bar struct {
BarField string `json:"barfield"`
}
func main() {
fooRespJSON := []byte(`{"data":{"foofield":"foo value"}}`)
barRespJSON := []byte(`{"data":{"barfield":"bar value"}}`)
var (
resp Response
foo Foo
bar Bar
)
// Foo
if err := json.Unmarshal(fooRespJSON, &resp); err != nil {
log.Fatal(err)
}
if err := json.Unmarshal(resp.Data, &foo); err != nil {
log.Fatal(err)
}
fmt.Println("foo:", foo)
// Bar
if err := json.Unmarshal(barRespJSON, &resp); err != nil {
log.Fatal(err)
}
if err := json.Unmarshal(resp.Data, &bar); err != nil {
log.Fatal(err)
}
fmt.Println("bar:", bar)
}
Output:
foo: {foo value}
bar: {bar value}
https://play.golang.org/p/Y7D4uhaC4a8
Option C
A third option, as pointed out by #mkopriva in a comment on the question, is to use interface{} as an intermediary datatype and pre-initialise this to a known datatype.
Emphasis lies on the word intermediary -- of course passing around an interface{} is best avoided (Rob Pike's Go Proverbs). The use-case here is to allow any datatype to be used without the need for multiple different Response types. On way to avoid exposing the interface{} is to wrap the response completely, exposing only the data and the error:
package main
import (
"encoding/json"
"fmt"
"log"
)
type Foo struct {
FooField string `json:"foofield"`
}
type Bar struct {
BarField string `json:"barfield"`
}
type Error struct {
Code int
}
func (e *Error) Error() string {
return fmt.Sprintf("error code %d", e.Code)
}
func unmarshalResponse(data []byte, v interface{}) error {
resp := struct {
Err int `json:"err"`
Data interface{} `json:"data"`
}{Data: v}
if err := json.Unmarshal(data, &resp); err != nil {
return err
}
if resp.Err != 0 {
return &Error{Code: resp.Err}
}
return nil
}
func main() {
fooRespJSON := []byte(`{"data":{"foofield":"foo value"}}`)
barRespJSON := []byte(`{"data":{"barfield":"bar value"}}`)
errRespJSON := []byte(`{"err": 123}`)
// Foo
var foo Foo
if err := unmarshalResponse(fooRespJSON, &foo); err != nil {
log.Fatal(err)
}
fmt.Println("foo:", foo)
// Bar
var bar Bar
if err := unmarshalResponse(barRespJSON, &bar); err != nil {
log.Fatal(err)
}
fmt.Println("bar:", bar)
// Error response
var v interface{}
if err := unmarshalResponse(errRespJSON, &v); err != nil {
log.Fatal(err)
}
}
Output:
foo: {foo value}
bar: {bar value}
2009/11/10 23:00:00 error code 123
https://play.golang.org/p/5SVfQGwS-Wy

Golang - Json decoding, check field exist or not automatically [duplicate]

Is it possible to generate an error if a field was not found while parsing a JSON input using Go?
I could not find it in documentation.
Is there any tag that specifies the field as required?
There is no tag in the encoding/json package that sets a field to "required". You will either have to write your own MarshalJSON() method, or do a post check for missing fields.
To check for missing fields, you will have to use pointers in order to distinguish between missing/null and zero values:
type JsonStruct struct {
String *string
Number *float64
}
Full working example:
package main
import (
"fmt"
"encoding/json"
)
type JsonStruct struct {
String *string
Number *float64
}
var rawJson = []byte(`{
"string":"We do not provide a number"
}`)
func main() {
var s *JsonStruct
err := json.Unmarshal(rawJson, &s)
if err != nil {
panic(err)
}
if s.String == nil {
panic("String is missing or null!")
}
if s.Number == nil {
panic("Number is missing or null!")
}
fmt.Printf("String: %s Number: %f\n", *s.String, *s.Number)
}
Playground
You can also override the unmarshalling for a specific type (so a required field buried in a few json layers) without having to make the field a pointer. UnmarshalJSON is defined by the Unmarshaler interface.
type EnumItem struct {
Named
Value string
}
func (item *EnumItem) UnmarshalJSON(data []byte) (err error) {
required := struct {
Value *string `json:"value"`
}{}
all := struct {
Named
Value string `json:"value"`
}{}
err = json.Unmarshal(data, &required)
if err != nil {
return
} else if required.Value == nil {
err = fmt.Errorf("Required field for EnumItem missing")
} else {
err = json.Unmarshal(data, &all)
item.Named = all.Named
item.Value = all.Value
}
return
}
Here is another way by checking your customized tag
you can create a tag for your struct like:
type Profile struct {
Name string `yourprojectname:"required"`
Age int
}
Use reflect to check if the tag is assigned required value
func (p *Profile) Unmarshal(data []byte) error {
err := json.Unmarshal(data, p)
if err != nil {
return err
}
fields := reflect.ValueOf(p).Elem()
for i := 0; i < fields.NumField(); i++ {
yourpojectTags := fields.Type().Field(i).Tag.Get("yourprojectname")
if strings.Contains(yourpojectTags, "required") && fields.Field(i).IsZero() {
return errors.New("required field is missing")
}
}
return nil
}
And test cases are like:
func main() {
profile1 := `{"Name":"foo", "Age":20}`
profile2 := `{"Name":"", "Age":21}`
var profile Profile
err := profile.Unmarshal([]byte(profile1))
if err != nil {
log.Printf("profile1 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile1 unmarshal: %v\n", profile)
err = profile.Unmarshal([]byte(profile2))
if err != nil {
log.Printf("profile2 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile2 unmarshal: %v\n", profile)
}
Result:
profile1 unmarshal: {foo 20}
2009/11/10 23:00:00 profile2 unmarshal error: required field is missing
You can go to Playground to have a look at the completed code
You can just implement the Unmarshaler interface to customize how your JSON gets unmarshalled.
you can also make use of JSON schema validation.
package main
import (
"encoding/json"
"fmt"
"github.com/alecthomas/jsonschema"
"github.com/xeipuuv/gojsonschema"
)
type Bird struct {
Species string `json:"birdType"`
Description string `json:"what it does" jsonschema:"required"`
}
func main() {
var bird Bird
sc := jsonschema.Reflect(&bird)
b, _ := json.Marshal(sc)
fmt.Println(string(b))
loader := gojsonschema.NewStringLoader(string(b))
documentLoader := gojsonschema.NewStringLoader(`{"birdType": "pigeon"}`)
schema, err := gojsonschema.NewSchema(loader)
if err != nil {
panic("nop")
}
result, err := schema.Validate(documentLoader)
if err != nil {
panic("nop")
}
if result.Valid() {
fmt.Printf("The document is valid\n")
} else {
fmt.Printf("The document is not valid. see errors :\n")
for _, err := range result.Errors() {
// Err implements the ResultError interface
fmt.Printf("- %s\n", err)
}
}
}
Outputs
{"$schema":"http://json-schema.org/draft-04/schema#","$ref":"#/definitions/Bird","definitions":{"Bird":{"required":["birdType","what it does"],"properties":{"birdType":{"type":"string"},"what it does":{"type":"string"}},"additionalProperties":false,"type":"object"}}}
The document is not valid. see errors :
- (root): what it does is required
code example taken from Strict JSON parsing

Golang custom JSON serialization (does something equivalent to gob.register() exist for json?)

Is there a way to serialize custom structs when encoding/decoding with json?
say you have 3 (in my actual code there are 10) different custom structs which are being sent over udp, and you use json for encoding:
type a struct {
Id int
Data msgInfo
}
type b struct {
Id int
Data msgInfo
Other metaInfo
}
type c struct {
Other metaInfo
}
On the recieving end you want to know if the struct recieved was of type a, b or c, so it can for example be passed to a type spesific channel.
type msgtype reflect.Type
.
.
nrOfBytes, err := udpConn.Read(recievedBytes)
if err != nil {...}
var msg interface{}
err = json.Unmarshal(recievedBytes[0:nrOfBytes], &msg)
if err != nil {...}
u := reflect.ValueOf(msg)
msgType := u.Type()
fmt.Printf("msg is of type: %s\n", msgType)
With gob this is easily done by registering the types, but i have to use json seeing as it's communication over udp, so is there anyway to serialize the custom structs? I want the print to be
msg is of type: a
but i'm only getting
msg is of type: map[string]interface {}
One thing you can do is using the json.RawMessage type and a custom Wrapper type.
Then, upon receiving a message, you can do a switch (or use a map of constructors) to get the right struct.
For example (omitting error checking):
package main
import (
"encoding/json"
"fmt"
)
type Message struct {
Type string
Data json.RawMessage
}
func (m Message) Struct() interface{} {
unmarshaller, found := unmarshallers[m.Type]
if !found {
return nil
}
return unmarshaller([]byte(m.Data))
}
type Foo struct {
ID int
}
var unmarshallers = map[string]func([]byte) interface{}{
"foo": func(raw []byte) interface{} {
var f Foo
json.Unmarshal(raw, &f)
return f
},
}
func main() {
var body = []byte(`{"Type":"foo","Data":{"ID":1}}`)
var msg Message
json.Unmarshal(body, &msg)
switch s := msg.Struct().(type) {
case Foo:
fmt.Println(s)
}
}
See this playground example for a live demo http://play.golang.org/p/7FmQqnWPaE
Base on your comment, this maybe a solution:
type Packet {
Type string
Data []byte
}
Encode function:
func packageEncode(i interface{}) ([]byte, error) {
b, err := json.Marshal(i)
if err != nil {
return nil, err
}
p := &Package{Data: b}
switch t.(type) {
case a:
p.Type = "a"
case b:
p.Type = "b"
case c:
p.Type = "c"
}
return json.Marshal(p)
}
Then, when recieved message and decode:
p := &Package{}
err = json.Unmarshal(recievedBytes[0:nrOfBytes], p)
...
if p.Type == "a" {
msg := &a{}
err = json.Unmarshal(p.Data, msg)
}
if p.Type == "b" {
...
}

Unmarshaling json in Go: required field?

Is it possible to generate an error if a field was not found while parsing a JSON input using Go?
I could not find it in documentation.
Is there any tag that specifies the field as required?
There is no tag in the encoding/json package that sets a field to "required". You will either have to write your own MarshalJSON() method, or do a post check for missing fields.
To check for missing fields, you will have to use pointers in order to distinguish between missing/null and zero values:
type JsonStruct struct {
String *string
Number *float64
}
Full working example:
package main
import (
"fmt"
"encoding/json"
)
type JsonStruct struct {
String *string
Number *float64
}
var rawJson = []byte(`{
"string":"We do not provide a number"
}`)
func main() {
var s *JsonStruct
err := json.Unmarshal(rawJson, &s)
if err != nil {
panic(err)
}
if s.String == nil {
panic("String is missing or null!")
}
if s.Number == nil {
panic("Number is missing or null!")
}
fmt.Printf("String: %s Number: %f\n", *s.String, *s.Number)
}
Playground
You can also override the unmarshalling for a specific type (so a required field buried in a few json layers) without having to make the field a pointer. UnmarshalJSON is defined by the Unmarshaler interface.
type EnumItem struct {
Named
Value string
}
func (item *EnumItem) UnmarshalJSON(data []byte) (err error) {
required := struct {
Value *string `json:"value"`
}{}
all := struct {
Named
Value string `json:"value"`
}{}
err = json.Unmarshal(data, &required)
if err != nil {
return
} else if required.Value == nil {
err = fmt.Errorf("Required field for EnumItem missing")
} else {
err = json.Unmarshal(data, &all)
item.Named = all.Named
item.Value = all.Value
}
return
}
Here is another way by checking your customized tag
you can create a tag for your struct like:
type Profile struct {
Name string `yourprojectname:"required"`
Age int
}
Use reflect to check if the tag is assigned required value
func (p *Profile) Unmarshal(data []byte) error {
err := json.Unmarshal(data, p)
if err != nil {
return err
}
fields := reflect.ValueOf(p).Elem()
for i := 0; i < fields.NumField(); i++ {
yourpojectTags := fields.Type().Field(i).Tag.Get("yourprojectname")
if strings.Contains(yourpojectTags, "required") && fields.Field(i).IsZero() {
return errors.New("required field is missing")
}
}
return nil
}
And test cases are like:
func main() {
profile1 := `{"Name":"foo", "Age":20}`
profile2 := `{"Name":"", "Age":21}`
var profile Profile
err := profile.Unmarshal([]byte(profile1))
if err != nil {
log.Printf("profile1 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile1 unmarshal: %v\n", profile)
err = profile.Unmarshal([]byte(profile2))
if err != nil {
log.Printf("profile2 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile2 unmarshal: %v\n", profile)
}
Result:
profile1 unmarshal: {foo 20}
2009/11/10 23:00:00 profile2 unmarshal error: required field is missing
You can go to Playground to have a look at the completed code
You can just implement the Unmarshaler interface to customize how your JSON gets unmarshalled.
you can also make use of JSON schema validation.
package main
import (
"encoding/json"
"fmt"
"github.com/alecthomas/jsonschema"
"github.com/xeipuuv/gojsonschema"
)
type Bird struct {
Species string `json:"birdType"`
Description string `json:"what it does" jsonschema:"required"`
}
func main() {
var bird Bird
sc := jsonschema.Reflect(&bird)
b, _ := json.Marshal(sc)
fmt.Println(string(b))
loader := gojsonschema.NewStringLoader(string(b))
documentLoader := gojsonschema.NewStringLoader(`{"birdType": "pigeon"}`)
schema, err := gojsonschema.NewSchema(loader)
if err != nil {
panic("nop")
}
result, err := schema.Validate(documentLoader)
if err != nil {
panic("nop")
}
if result.Valid() {
fmt.Printf("The document is valid\n")
} else {
fmt.Printf("The document is not valid. see errors :\n")
for _, err := range result.Errors() {
// Err implements the ResultError interface
fmt.Printf("- %s\n", err)
}
}
}
Outputs
{"$schema":"http://json-schema.org/draft-04/schema#","$ref":"#/definitions/Bird","definitions":{"Bird":{"required":["birdType","what it does"],"properties":{"birdType":{"type":"string"},"what it does":{"type":"string"}},"additionalProperties":false,"type":"object"}}}
The document is not valid. see errors :
- (root): what it does is required
code example taken from Strict JSON parsing