I am probably making a rookie mistake in the code below. Here is the go playground link: https://play.golang.org/p/WS_pPvBHUTH
I am expecting variable a (json string) to be unmarshalled into struct of type A but somehow it ends up being of type map[string]interface{}
package main
import (
"encoding/json"
"fmt"
)
type A struct {
ParamA string
}
type B struct {
ParamB string
}
var types = map[string]func() interface{}{
"ParamA": func() interface{} { return A{} },
"ParamB": func() interface{} { return B{} },
}
func Unmarshal(data []byte, key string) interface{} {
// if bytes.Contains(data, []byte("ParamA")) {
// var a A
// fmt.Printf("%+T\n", a)
// json.Unmarshal(data, &a)
// return a
// }
// var b B
// fmt.Printf("%+T\n", b)
// json.Unmarshal(data, &b)
// return b
if tmp, ok := types[key]; ok {
a := tmp()
fmt.Printf("%+T\n", a)
fmt.Println(string(data))
if err := json.Unmarshal(data, &a); err != nil {
return err
}
fmt.Printf("%+T\n", a)
return a
}
return nil
}
func main() {
a := []byte(`{"ParamA": "aaa"}`)
b := []byte(`{"ParamB": "bbb"}`)
resultA := Unmarshal(a, "ParamA")
resultB := Unmarshal(b, "ParamC")
fmt.Printf("%+v %+v\n", resultA, resultB)
}
Expected Output:
main.A
{"ParamA": "aaa"}
main.A // <= this is what I want
{ParamA:aaa} <nil> // <= this is what I want
Actual Output:
main.A
{"ParamA": "aaa"}
map[string]interface {} // <= but this is what I am getting
map[ParamA:aaa] <nil> // <= but this is what I am getting
Why is variable a (json string) not unmarshalling into struct A? Really stuck here...
Here:
a := tmp()
a is of type interface{}. Then you json.Unmarshal(data,&a), and overwrite contents of a.
What you really want to do is, first return &A{} or &B{} from your factory functions, then:
a:=tmp()
json.Unmarshal(data,a) // Not &a
Related
Given the following structs
type Foo struct {
Thing time.Duration `json:"thing"`
}
type Bar struct {
Foo
Entry time.Duration `json:"entry"`
}
I want to custom time.Duration format and load Bar value from json string like:
{
"thing": "hour",
"entry": "second"
}
So I override UnmarshalJSON for Foo and Bar (https://play.golang.org/p/6v71eG_Xr98):
package main
import (
"encoding/json"
"fmt"
"time"
)
type Foo struct {
Thing time.Duration `json:"thing"`
}
type Bar struct {
Foo
Entry time.Duration `json:"entry"`
}
func timeConvert(s string) time.Duration {
if s == "hour" {
return time.Hour
}
if s == "second" {
return time.Second
}
return time.Duration(0)
}
func (t *Foo) UnmarshalJSON(data []byte) error {
type Alias Foo
a := struct {
Thing string `json:"thing"`
*Alias
}{Alias: (*Alias)(t)}
err := json.Unmarshal(data, &a)
t.Thing = timeConvert(a.Thing)
fmt.Printf("Foo: %v [%v]\n", *t, err)
return err
}
func (t *Bar) UnmarshalJSON(data []byte) error {
type Alias Bar
a := struct {
Entry string `json:"entry"`
*Alias
}{Alias: (*Alias)(t)}
err := json.Unmarshal(data, &a)
t.Entry = timeConvert(a.Entry)
fmt.Printf("Bar: %v [%v]\n", *t, err)
return err
}
func main() {
data := []byte(`{"entry": "second", "thing": "hour"}`)
json.Unmarshal(data, &Bar{})
}
But it outputs unexpected:
Foo: {1h0m0s} [<nil>]
Bar: {{1h0m0s} 0s} [<nil>]
Why Entry value is incorrect?
Thanks for mkopriva. I found out because 'json.Unmarshal' works on any type, it gives me a hint that all type have implemented 'UnmarshalJSON' function, but it's NOT.
The calling of 'json.Unmarshal' on 'Bar' will actually do the following things:
calling UnmarshalJSON on Bar.
Because anonymous struct in Bar don't implemented UnmarshalJSON, so calling UnmarshalJSON on it's embedding struct Foo instead.
That's why 'Entry' in anonymous struct will not be unmarshal.
mkopriva suggest use a custom type to get the custom parsing, but it's inconvenient when you need pass it as argument on functions in the time package. I figure out another way to do it (just unmarshal twice, see https://play.golang.org/p/2ahDX-0hsRt):
func (t *Bar) UnmarshalJSON(data []byte) error {
type Alias Bar
if err := json.Unmarshal(data, (*Alias)(t)); err != nil {
return err
}
var tmp struct {
Entry string `json:"entry"`
}
err := json.Unmarshal(data, &tmp)
t.Entry = timeConvert(tmp.Entry)
fmt.Printf("Bar: %v [%v]\n", *t, err)
return err
}
I am trying to unmarshal some pretty ugly json but can't figure out how. I have:
package main
import "fmt"
import "encoding/json"
type PublicKey struct {
ID int `json:"id"`
Key string `json:"key"`
MyData []struct {
ID string `json:"id"`
Value int `json:"value"`
}
}
func main() {
b := `[
{
"id": 1,
"key": "my_key"
},
[
{
"id": "some_id",
"value": 12
},
{
"id": "anorther_id",
"value": 13
}
]
]`
var pk []PublicKey
err := json.Unmarshal([]byte(b), &pk)
if err != nil {
fmt.Println(err)
}
fmt.Println(pk)
}
For the result I am getting:
[{1 my_key []} {0 []}]
The second slice is empty when it shouldn't be.
EDIT:
The error I get is:
json: cannot unmarshal array into Go struct field PublicKey.key of type main.PublicKey
https://play.golang.org/p/cztXOchiiS5
That is some truly hideous JSON! I have two approaches to handling the mixed array elements and I like the 2nd one better. Here's the first approach using interface and a type switch:
package main
import (
"encoding/json"
"errors"
"fmt"
)
type PublicKey struct {
ID int `json:"id"`
Key string `json:"key"`
}
type MyData struct {
ID string `json:"id"`
Value int `json:"value"`
}
type MixedData struct {
Key []PublicKey
MyData [][]MyData
}
func (md *MixedData) UnmarshalJSON(b []byte) error {
md.Key = []PublicKey{}
md.MyData = [][]MyData{}
var obj []interface{}
err := json.Unmarshal([]byte(b), &obj)
if err != nil {
return err
}
for _, o := range obj {
switch o.(type) {
case map[string]interface{}:
m := o.(map[string]interface{})
id, ok := m["id"].(float64)
if !ok {
return errors.New("public key id must be an int")
}
pk := PublicKey{}
pk.ID = int(id)
pk.Key, ok = m["key"].(string)
if !ok {
return errors.New("public key key must be a string")
}
md.Key = append(md.Key, pk)
case []interface{}:
a := o.([]interface{})
myData := make([]MyData, len(a))
for i, x := range a {
m, ok := x.(map[string]interface{})
if !ok {
return errors.New("data array contains unexpected object")
}
val, ok := m["value"].(float64)
if !ok {
return errors.New("data value must be an int")
}
myData[i].Value = int(val)
myData[i].ID, ok = m["id"].(string)
if !ok {
return errors.New("data id must be a string")
}
md.MyData = append(md.MyData, myData)
}
default:
// got something unexpected, handle somehow
}
}
return nil
}
func main() {
b := `[
{
"id": 1,
"key": "my_key"
},
[
{
"id": "some_id",
"value": 12
},
{
"id": "another_id",
"value": 13
}
]
]`
m := MixedData{}
err := json.Unmarshal([]byte(b), &m)
if err != nil {
fmt.Println(err)
}
fmt.Println(m)
}
https://play.golang.org/p/g8d_AsH-pYY
Hopefully there aren't any unexpected other elements, but they can be handled similarly.
Here is the second that relies more on Go's internal JSON parsing with the help of json.RawMessage. It makes the same assumptions about the contents of the array. It assumes that any objects will Unmarshal into PublicKey instances and any arrays consist of only MyData instances. I also added how to marshal back into the target JSON for symmetry:
package main
import (
"encoding/json"
"fmt"
"os"
)
type PublicKey struct {
ID int `json:"id"`
Key string `json:"key"`
}
type MyData struct {
ID string `json:"id"`
Value int `json:"value"`
}
type MixedData struct {
Keys []PublicKey
MyData [][]MyData
}
func (md *MixedData) UnmarshalJSON(b []byte) error {
md.Keys = []PublicKey{}
md.MyData = [][]MyData{}
obj := []json.RawMessage{}
err := json.Unmarshal([]byte(b), &obj)
if err != nil {
return err
}
for _, o := range obj {
switch o[0] {
case '{':
pk := PublicKey{}
err := json.Unmarshal(o, &pk)
if err != nil {
return err
}
md.Keys = append(md.Keys, pk)
case '[':
myData := []MyData{}
err := json.Unmarshal(o, &myData)
if err != nil {
return err
}
md.MyData = append(md.MyData, myData)
default:
// got something unexpected, handle somehow
}
}
return nil
}
func (md *MixedData) MarshalJSON() ([]byte, error) {
out := make([]interface{}, len(md.Keys)+len(md.MyData))
i := 0
for _, x := range md.Keys {
out[i] = x
i++
}
for _, x := range md.MyData {
out[i] = x
i++
}
return json.Marshal(out)
}
func main() {
b := `[
{
"id": 1,
"key": "my_key"
},
[
{
"id": "some_id",
"value": 12
},
{
"id": "another_id",
"value": 13
}
]
]`
m := MixedData{}
err := json.Unmarshal([]byte(b), &m)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
fmt.Println(m)
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
if err := enc.Encode(m); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
https://play.golang.org/p/ryZzaWKNcN0
Here's an approach that combines json.RawMessage with the trick of using the default unmarshaler in a type that implements json.Unmarshaler by creating a new temporary type that aliases the target type.
The idea is that we unmarshal the incoming array into a raw message and ensure that the array length is what we expect. Then we unmarshal the individual array elements into the custom struct types using their JSON tag annotations. The end result is that we can unmarshal the PublicKey type in the usual way and the UnmarshalJSON code is not terribly difficult to follow once you understand the tricks.
For example (Go Playground):
type PublicKey struct {
ID int `json:"id"`
Key string `json:"key"`
Data []MyData
}
type MyData struct {
ID string `json:"id"`
Value int `json:"value"`
}
func (pk *PublicKey) UnmarshalJSON(bs []byte) error {
// Unmarshal into a RawMessage so we can inspect the array length.
var rawMessage []json.RawMessage
err := json.Unmarshal(bs, &rawMessage)
if err != nil {
return err
}
if len(rawMessage) != 2 {
return fmt.Errorf("expected array of length 2, got %d", len(rawMessage))
}
// Parse the first object as PublicKey using the default unmarshaler
// using a temporary type that is an alias for the target type.
type PublicKey2 PublicKey
var pk2 PublicKey2
err = json.Unmarshal(rawMessage[0], &pk2)
if err != nil {
return err
}
// Parse the second object as []MyData in the usual way.
err = json.Unmarshal(rawMessage[1], &pk2.Data)
if err != nil {
return err
}
// Finally, assign the aliased object to the target object.
*pk = PublicKey(pk2)
return nil
}
func main() {
var pk PublicKey
err := json.Unmarshal([]byte(jsonstr), &pk)
if err != nil {
panic(err)
}
fmt.Printf("%#v\n", pk)
// main.PublicKey{ID:1, Key:"my_key", Data:[]main.MyData{main.MyData{ID:"some_id", Value:12}, main.MyData{ID:"anorther_id", Value:13}}}
}
I have the follow example code:
type Num struct {
X uint64 `json:"x,string"`
Y float64 `json:"y,string"`
}
Now, if I run the code
js := []byte(`{"x": "123", "y": "1.23"}`)
var n Num
err := json.Unmarshal(js, &n)
it will parse ok.
But if I change the JSON to
js := []byte(`{"x": 123, "y": 1.23}`)
it returns an error.
The result I can understand.
Now, my problem is how can I make it accept both string and uint64/float64?
You will need to define a custom type that implements the json.Unmarshaler interface in such a way that the value can be either a string or a plain number.
The example below shows how to do it for a single type (uint64); you will need to repeat the pattern for any other numerical types (Go Playground):
type Uint64 uint64
type Num struct {
X Uint64 `json:"x"`
}
func (u *Uint64) UnmarshalJSON(bs []byte) error {
str := string(bs) // Parse plain numbers directly.
if bs[0] == '"' && bs[len(bs)-1] == '"' {
// Unwrap the quotes from string numbers.
str = string(bs[1 : len(bs)-1])
}
x, err := strconv.ParseUint(str, 10, 64)
if err != nil {
return err
}
*u = Uint64(x)
return nil
}
func main() {
ss := []string{`{"x":"123"}`, `{"x":123}`}
var n Num
for _, s := range ss {
err := json.Unmarshal([]byte(s), &n)
fmt.Printf("OK: s=%-11s n=%#v err=%v\n", s, n, err)
}
// OK: s={"x":"123"} n=main.Num{X:0x7b} err=<nil>
// OK: s={"x":123} n=main.Num{X:0x7b} err=<nil>
}
Building on #maerics answer, you can defer both cases to the usual json unmarshaler, which feels a bit more robust:
package main
import (
"encoding/json"
"errors"
"fmt"
)
type Uint64 uint64
type Num struct {
X Uint64 `json:"x"`
}
func (u *Uint64) UnmarshalJSON(bs []byte) error {
var i uint64
if err := json.Unmarshal(bs, &i); err == nil {
*u = Uint64(i)
return nil
}
var s string
if err := json.Unmarshal(bs, &s); err != nil {
return errors.New("expected a string or an integer")
}
if err := json.Unmarshal([]byte(s), &i); err != nil {
return err
}
*u = Uint64(i)
return nil
}
func main() {
ss := []string{`{"x":"123"}`, `{"x":123}`, `{"x":0.12}`}
var n Num
for _, s := range ss {
err := json.Unmarshal([]byte(s), &n)
fmt.Printf("OK: s=%-11s n=%#v err=%v\n", s, n, err)
}
}
which gives
OK: s={"x":"123"} n=main.Num{X:0x7b} err=<nil>
OK: s={"x":123} n=main.Num{X:0x7b} err=<nil>
OK: s={"x":0.12} n=main.Num{X:0x7b} err=expected a string or an integer
I'm using an API that formats its responses in this way:
{
"err": 0,
"data": **Other json structure**
}
The way I'm getting a response right now is I'm putting the response in an struct like this:
type Response struct {
Err int `json:"err"`
Data interface{} `json:"data"`
}
and then I'm doing this after getting a response
jsonbytes, _ := json.Marshal(resp.Data)
json.Unmarshal(jsonBytes, &dataStruct)
I'm only ignoring errors for this example.
It seems kinda weird to me that I'm marshaling and unmarshaling when I know what the data is supposed to look like and what type it's supposed to be.
Is there a more simple solution that I'm not seeing or is this a normal thing to do?
Edit: I should probably mention that the Data attribute in the response object can vary depending on what API call I'm doing.
The JSON unmarshaller uses reflection to look at the type it is unmarshalling to. Given an uninitialised interface{} as the destination for the unmarshalled data, a JSON object gets unmarshalled into a map[string]interface{} (example in playground).
Here are some ideas.
Option A
If you know the datatype, you can define a new response struct for each type. Example:
type FooResponse struct {
Err int `json:"err"`
Data Foo `json:"data"`
}
type Foo struct {
FooField string `json:"foofield"`
}
type BarResponse struct {
Err int `json:"err"`
Data Bar `json:"data"`
}
type Bar struct {
BarField string `json:"barfield"`
}
Option B
If you prefer to have a single Response struct instead of one per type, you can tell the JSON unmarshaller to avoid unmarshalling the data field until a later time by using the json.RawMessage data type:
package main
import (
"encoding/json"
"fmt"
"log"
)
type Response struct {
Err int `json:"err"`
Data json.RawMessage `json:"data"`
}
type Foo struct {
FooField string `json:"foofield"`
}
type Bar struct {
BarField string `json:"barfield"`
}
func main() {
fooRespJSON := []byte(`{"data":{"foofield":"foo value"}}`)
barRespJSON := []byte(`{"data":{"barfield":"bar value"}}`)
var (
resp Response
foo Foo
bar Bar
)
// Foo
if err := json.Unmarshal(fooRespJSON, &resp); err != nil {
log.Fatal(err)
}
if err := json.Unmarshal(resp.Data, &foo); err != nil {
log.Fatal(err)
}
fmt.Println("foo:", foo)
// Bar
if err := json.Unmarshal(barRespJSON, &resp); err != nil {
log.Fatal(err)
}
if err := json.Unmarshal(resp.Data, &bar); err != nil {
log.Fatal(err)
}
fmt.Println("bar:", bar)
}
Output:
foo: {foo value}
bar: {bar value}
https://play.golang.org/p/Y7D4uhaC4a8
Option C
A third option, as pointed out by #mkopriva in a comment on the question, is to use interface{} as an intermediary datatype and pre-initialise this to a known datatype.
Emphasis lies on the word intermediary -- of course passing around an interface{} is best avoided (Rob Pike's Go Proverbs). The use-case here is to allow any datatype to be used without the need for multiple different Response types. On way to avoid exposing the interface{} is to wrap the response completely, exposing only the data and the error:
package main
import (
"encoding/json"
"fmt"
"log"
)
type Foo struct {
FooField string `json:"foofield"`
}
type Bar struct {
BarField string `json:"barfield"`
}
type Error struct {
Code int
}
func (e *Error) Error() string {
return fmt.Sprintf("error code %d", e.Code)
}
func unmarshalResponse(data []byte, v interface{}) error {
resp := struct {
Err int `json:"err"`
Data interface{} `json:"data"`
}{Data: v}
if err := json.Unmarshal(data, &resp); err != nil {
return err
}
if resp.Err != 0 {
return &Error{Code: resp.Err}
}
return nil
}
func main() {
fooRespJSON := []byte(`{"data":{"foofield":"foo value"}}`)
barRespJSON := []byte(`{"data":{"barfield":"bar value"}}`)
errRespJSON := []byte(`{"err": 123}`)
// Foo
var foo Foo
if err := unmarshalResponse(fooRespJSON, &foo); err != nil {
log.Fatal(err)
}
fmt.Println("foo:", foo)
// Bar
var bar Bar
if err := unmarshalResponse(barRespJSON, &bar); err != nil {
log.Fatal(err)
}
fmt.Println("bar:", bar)
// Error response
var v interface{}
if err := unmarshalResponse(errRespJSON, &v); err != nil {
log.Fatal(err)
}
}
Output:
foo: {foo value}
bar: {bar value}
2009/11/10 23:00:00 error code 123
https://play.golang.org/p/5SVfQGwS-Wy
Is there a way to serialize custom structs when encoding/decoding with json?
say you have 3 (in my actual code there are 10) different custom structs which are being sent over udp, and you use json for encoding:
type a struct {
Id int
Data msgInfo
}
type b struct {
Id int
Data msgInfo
Other metaInfo
}
type c struct {
Other metaInfo
}
On the recieving end you want to know if the struct recieved was of type a, b or c, so it can for example be passed to a type spesific channel.
type msgtype reflect.Type
.
.
nrOfBytes, err := udpConn.Read(recievedBytes)
if err != nil {...}
var msg interface{}
err = json.Unmarshal(recievedBytes[0:nrOfBytes], &msg)
if err != nil {...}
u := reflect.ValueOf(msg)
msgType := u.Type()
fmt.Printf("msg is of type: %s\n", msgType)
With gob this is easily done by registering the types, but i have to use json seeing as it's communication over udp, so is there anyway to serialize the custom structs? I want the print to be
msg is of type: a
but i'm only getting
msg is of type: map[string]interface {}
One thing you can do is using the json.RawMessage type and a custom Wrapper type.
Then, upon receiving a message, you can do a switch (or use a map of constructors) to get the right struct.
For example (omitting error checking):
package main
import (
"encoding/json"
"fmt"
)
type Message struct {
Type string
Data json.RawMessage
}
func (m Message) Struct() interface{} {
unmarshaller, found := unmarshallers[m.Type]
if !found {
return nil
}
return unmarshaller([]byte(m.Data))
}
type Foo struct {
ID int
}
var unmarshallers = map[string]func([]byte) interface{}{
"foo": func(raw []byte) interface{} {
var f Foo
json.Unmarshal(raw, &f)
return f
},
}
func main() {
var body = []byte(`{"Type":"foo","Data":{"ID":1}}`)
var msg Message
json.Unmarshal(body, &msg)
switch s := msg.Struct().(type) {
case Foo:
fmt.Println(s)
}
}
See this playground example for a live demo http://play.golang.org/p/7FmQqnWPaE
Base on your comment, this maybe a solution:
type Packet {
Type string
Data []byte
}
Encode function:
func packageEncode(i interface{}) ([]byte, error) {
b, err := json.Marshal(i)
if err != nil {
return nil, err
}
p := &Package{Data: b}
switch t.(type) {
case a:
p.Type = "a"
case b:
p.Type = "b"
case c:
p.Type = "c"
}
return json.Marshal(p)
}
Then, when recieved message and decode:
p := &Package{}
err = json.Unmarshal(recievedBytes[0:nrOfBytes], p)
...
if p.Type == "a" {
msg := &a{}
err = json.Unmarshal(p.Data, msg)
}
if p.Type == "b" {
...
}