Golang interface{} type misunderstanding - json

I got a bug in Go when using an interface{} as function parameter type, when given a non-pointer type, and using json.Unmarshal with it.
Because a piece of code is worth a thousand words, here is an example:
package main
import (
"encoding/json"
"fmt"
)
func test(i interface{}) {
j := []byte(`{ "foo": "bar" }`)
fmt.Printf("%T\n", i)
fmt.Printf("%T\n", &i)
json.Unmarshal(j, &i)
fmt.Printf("%T\n", i)
}
type Test struct {
Foo string
}
func main() {
test(Test{})
}
Which outputs:
main.Test
*interface {}
map[string]interface {}
json.Unmarshal turns my struct to a map[string]interface{} oO...
Little readings later explains some of it, interface{} is a type in itself, not some sort of typeless container, which explains the *interface{}, and the fact that json.Unmarshal could not get the initial type, and returned a map[string]interface{}..
From Unmarshal docs:
To unmarshal JSON into an interface value, Unmarshal stores one of
these in the interface value:
[...]
And if I pass a pointer to the test function like so, it works:
func test(i interface{}) {
j := []byte(`{ "foo": "bar" }`)
fmt.Printf("%T\n", i)
fmt.Printf("%T\n", &i)
json.Unmarshal(j, i)
fmt.Printf("%T\n", i)
fmt.Println(i)
}
func main() {
test(&Test{})
}
Which outputs:
*main.Test
*interface {}
*main.Test
&{bar}
Cool, the data is unmarshalled and all, now in this second snippet I removed the & when calling Unmarshal. Because I have a *Test in i, no use for it.
So in all logic, if I put back the & to i when calling Unmarshal it should mess up with i's type again. But no.
If I run:
func test(i interface{}) {
j := []byte(`{ "foo": "bar" }`)
fmt.Printf("%T\n", i)
fmt.Printf("%T\n", &i)
json.Unmarshal(j, &i)
fmt.Printf("%T\n", i)
fmt.Println(i)
}
func main() {
test(&Test{})
}
Well it still works:
*main.Test
*interface {}
*main.Test
&{bar}
And now I'm out of google search queries.

The right scenario
interface{} is a wrapper for any value and of any type. An interface schematically wraps a (value; type) pair, a concrete value and its type. More details on this: The Laws of Reflection #The representation of an interface.
json.Unmarshal() already takes the value of type interface{}:
func Unmarshal(data []byte, v interface{}) error
So if you already have an interface{} value (the i interface{} parameter of the test() function), don't try to take its address, just pass it along as-is.
Also note that for any package to modify a value stored in an interface{}, you need to pass a pointer to it. So what should be in i is a pointer. So the right scenario is to pass *Test to test(), and inside test() pass i to json.Unmarshal() (without taking its address).
Explanation of other scenarios
When i contains *Test and you pass &i, it will work because the json package will simply dereference the *interface{} pointer, and finds an interface{} value, which wraps a *Test value. It's a pointer, so it's all good: unmarshals the JSON object into the pointed Test value.
When i contains Test and you pass &i, same thing goes as above: *interface{} is dereferenced, so it finds an interface{} which contains a non-pointer: Test. Since the json package can't unmarshal into a non-pointer value, it has to create a new value. And since the passed value to the json.Unmarshal() function is of type *interface{}, it tells the json package to unmarshal the data into a value of type interface{}. This means the json package is free to choose which type to use. And by default the json package unmarshals JSON objects into map[string]interface{} values, so that is what's created and used (and eventually put into the value pointed by the pointer you passed: &i).
All in all
All in all, avoid using pointers to interfaces. Instead "put" pointers into the interfaces (the interface value should wrap the pointer). When you already have an interface{} holding a pointer, just pass it along.

Related

Converting interfaces with json decoding

Consider the following code:
package main
import (
"fmt"
"encoding/json"
)
func testNoPointer() interface{} {
return []int{}
}
func testPointer() interface{} {
return &[]int{}
}
func main() {
jsonString := "[1]"
noPointer := testNoPointer()
fmt.Printf("%T\n", noPointer)
err := json.Unmarshal([]byte(jsonString), &noPointer)
if (err == nil) {
fmt.Printf("%T\n", noPointer)
}
pointer := testPointer()
err2 := json.Unmarshal([]byte(jsonString), &pointer)
if (err2 == nil) {
fmt.Printf("%T\n", pointer)
}
}
It outputs:
[]int
[]interface {}
*[]int
Why does unmarshaling erase the type information for values? As a consequence this decoded := noPointer.([]int) will throw interface conversion: interface {} is []interface {}, not []int. Is it possible to cast it to the correct type?
Your testNoPointer() and testPointer() functions return interface{} values. Understand that once you put (wrap) something in an interface value, you can no longer modify that.
So in the first case you have an interface value wrapping a slice. You can't add elements to the wrapped slice, because that requires modifying the slice header (e.g. its length field), so unmarshaling in the first case cannot use the value you pass for unmarshaling. It has to create a new value. And since you're unmarshaling into an interface{} value (type of noPointer is interface{}), the encoding/json package is free to choose whatever type it sees fit for unmarshaling, and json.Unmarshal() documents that:
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
You're unmarshaling a JSON array, so it chooses []interface{}.
In your second example you are wrapping *int[] in an interface value. It is again true that you cannot modify the value wrapped in the interface, but this doesn't have to happen. The encoding/json package can just modify the pointed value, so the wrapped pointer in the interface can remain unchanged, so the encoding/json package can use the pointer, and hence can retain the element type.
Note that in the second example it's important that the wrapped pointer is not nil, because if it would be nil, the encoding/json package could not use the interface value wrapping the pointer: a nil pointer points to nowhere, so a new, pointer would have to be allocated. But then again we're at a point where encoding/json unmarshals into an interface{} value (type of pointer is interface{}), so the encoding/json package would revert to creating an []interface{} for the JSON array being unmarshaled.
You can test this when a testPointer2() returns a nil pointer wrapped:
func testPointer2() interface{} {
var x *[]int
return x // this will be a nil pointer
}
And testing it:
pointer2 := testPointer2()
err3 := json.Unmarshal([]byte(jsonString), &pointer2)
if err3 == nil {
fmt.Printf("%T\n", pointer2)
}
Which outputs (try it on the Go Playground):
[]interface {}
We again "lost" the type information, because the interface value wrapping a nil pointer could not be used.
See relevant issue / discussion on github: encoding/json: clarify what happens when unmarshaling into a non-empty interface{} #26946

Is Go able to unmarshal to map[string][]interface{}?

Currently, I try to parse JSON to map[string][]interface{}, but unmarshalling returns an error. According to (https://golang.org/pkg/encoding/json/), to unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
-[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
I wonder if golang is able to unmarshal map[string][]interface{}. The following is code snippet. I am new to Golang, thanks for help in advance.
// emailsStr looks like "{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}"
emailsRaw := make(map[string][]*entities.Email)
err := json.Unmarshal([]byte(emailsStr), &emailsRaw)
Error message:
&json.UnmarshalTypeError{Value:"number", Type:(*reflect.rtype)(0x151c7a0), Offset:44, Struct:"", Field:""}
The Go encoding/json package will only unmarshal dynamically to a map[string]interface{}. From there, you will need to use type assertions and casting to pull out the values you want, like so:
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
dynamic := make(map[string]interface{})
json.Unmarshal([]byte(jsonStr), &dynamic)
firstEmail := dynamic["unknown.0"].([]interface{})[0].(map[string]interface{})["email_address"]
fmt.Println(firstEmail)
}
(https://play.golang.org/p/VEUEIwj3CIC)
Each time, Go's .(<type>) operator is used to assert and cast the dynamic value to a specific type. This particular code will panic if anything happens to be the wrong type at runtime, like if the contents of unknown.0 aren't an array of JSON objects.
The more idiomatic (and robust) way to do this in Go is to annotate a couple structs with json:"" tags and have encoding/json unmarshal into them. This avoids all the nasty brittle .([]interface{}) type casting:
type Email struct {
Email string `json:"email_address"`
}
type EmailsList struct {
IsSchemaConforming bool `json:"isSchemaConforming"`
SchemaVersion int `json:"schemaVersion"`
Emails []Email `json:"unknown.0"`
}
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
emails := EmailsList{}
json.Unmarshal([]byte(jsonStr), &emails)
fmt.Printf("%+v\n", emails)
}
(https://play.golang.org/p/iS6e0_87P2J)
A better approach will be to use struct for main schema and then use an slice of email struct for fetching the data for email entities get the values from the same according to requirements. Please find the solution below :-
package main
import (
"fmt"
"encoding/json"
)
type Data struct{
IsSchemaConforming bool `json:"isSchemaConforming"`
SchemaVersion float64 `json:"schemaVersion"`
EmailEntity []Email `json:"unknown.0"`
}
// Email struct
type Email struct{
EmailAddress string `json:"email_address"`
}
func main() {
jsonStr := `{"isSchemaConforming":true,"schemaVersion":0,"unknown.0":[{"email_address":"test1#uber.com"},{"email_address":"test2#uber.com"}]}`
var dynamic Data
json.Unmarshal([]byte(jsonStr), &dynamic)
fmt.Printf("%#v", dynamic)
}

How to enforce float in decimal format when encoding to JSON in Go

I have a big.float which I'm encoding into JSON . However the JSON always end up showing the float in scientific notation rater than decimal notation. I can fix this by changing the JSON to be a string rather than a number and using float.Text('f'), however I would really prefer to keep the type as a number.
I was a taking a look at float.Format but I don't believe this is suitable.
A really condensed gist of what I'm doing is below. I do a lot more modification of the value of supply before encoding it to json.
type TokenSupply struct {
TotalSupply *big.Float `json:"totalSupply, omitempty"`
}
supply := Float.NewFloat(1000000)
json.NewEncoder(w).Encode(TokenSupply{supply})
This returns:
{"totalSupply":"1e+06"}
big.Float is marshaled to string when converted to a JSON type
https://golang.org/pkg/encoding/json/#Marshal
Marshal traverses the value v recursively. If an encountered value implements the Marshaler interface and is not a nil pointer, Marshal calls its MarshalJSON method to produce JSON. If no MarshalJSON method is present but the value implements encoding.TextMarshaler instead, Marshal calls its MarshalText method and encodes the result as a JSON string. The nil pointer exception is not strictly necessary but mimics a similar, necessary exception in the behavior of UnmarshalJSON.
https://golang.org/pkg/math/big/#Float.MarshalText
func (x *Float) MarshalText() (text []byte, err error)
What can you do about it?
since your float may be more than 64 bits it won't play well with other languages that have to read the JSON value as a number. I'd suggest you keep the number as a string.
Caveats about encoding numbers that don't fit into 64 bits aside, here is how you could marshal a big.Float as a JSON number by wrapping it in a custom type that implements json.Marshaler. The key is that you can implement theMarshalJSON method anyway you like, as long as it emits valid JSON:
package main
import (
"encoding/json"
"fmt"
"math/big"
)
type BigFloatNumberJSON struct{ *big.Float }
func (bfn BigFloatNumberJSON) MarshalJSON() ([]byte, error) {
// Use big.Float.String() or any other string converter
// that emits a valid JSON number here...
return []byte(bfn.String()), nil
}
func main() {
totalSupply := new(big.Float).SetFloat64(1000000)
obj := map[string]interface{}{
"totalSupply": BigFloatNumberJSON{totalSupply},
}
bytes, err := json.Marshal(&obj)
if err != nil {
panic(err)
}
fmt.Println(string(bytes))
// => {"totalSupply":1000000}
}

Unmarshalling JSON to interface variable while matching the underlying type in Go

I'm writing a wrapper to map with some additional functionality I need. Some of the most important things is the ability to marshal and unmarshal the data while retaining genericity. I managed to write an marshaller using the encoding/gob encoder, but since it would be nice if the marshalled data was human readable, I decided to code another implementation with JSON.
I got gob to encode from and decode to generic interface variables neatly by passing it a implementation object instance with Register(). (This resource helped me with the details! http://www.funcmain.com/gob_encoding_an_interface)
However, JSON doesn't have Register(). Let's say we have a value of type
type ConcreteImplementation struct {
FieldA string
FieldB string
}
func (c ConcreteImplementation) Key() string {
return c.FieldA // ConcreteImplementation implements genericValue
}
in a variable of type
type genericValue interface {
Key() string
}
When I marshal that, it outputs JSON like this:
{"FieldA":"foo","FieldB":"bar"}
And when I try to unmarshal that again into a variable of type genericValue, it says:
panic: interface conversion: map[string]interface {} is not genericValue: missing method Key EDIT: Oops, actually it says this!
Error with decoding! json: cannot unmarshal object into Go value of genericValue
Quite obviously, it tries to marshal the data like it says here: http://blog.golang.org/json-and-go (See 'Generic JSON with interface{}')
How can I get it to try to fit the data to an specific implementation, like gob decoder would try if the implementation is Register()ed? Register() was godsend, it allowed to marshal and unmarshal generically like it was nothing. How do I get JSON to do the same thing?
What if your types implemented the Unmarshaler?
Here is a small demo.
Or the same code here:
type ConcreteImplementation struct {
FieldA string
FieldB string
}
func (c ConcreteImplementation) Key() string {
return c.FieldA // ConcreteImplementation implements genericValue
}
// implementing json.Unmarshaler
func (c *ConcreteImplementation) UnmarshalJSON(j []byte) error {
m := make(map[string]string)
err := json.Unmarshal(j, &m)
if err != nil {
return err
}
if v, ok := m["FieldA"]; ok {
c.FieldA = v
}
if v, ok := m["FieldB"]; ok {
c.FieldB = v
}
return nil
}
type genericValue interface {
Key() string
json.Unmarshaler
}
func decode(jsonStr []byte, v genericValue) error {
return json.Unmarshal(jsonStr, v)
}
With this you can pass a genericValue to json.Unmarshal.
Allright, got it to work finally. This question provided the answer. Why does json.Unmarshal return a map instead of the expected struct?
"You've passed to json a pointer to an abstract interface. You should simply pass a pointer to Ping as an abstract interface" - This applied to my situation too. (For some reason a pointer TO an abstract interface was enough for gob package. It seems that I have to study Go interfaces and reflection some more to understand why...)
I won't still mark this as solved question, if someone has a better answer.

json decode key types

I want to decode a big set of data from a (static-schema) json file. The file contains exclusively numeric data, and keys are all integers. I know how to decode this json into a struct containing fields of map[string]int or map[string]float32 using json.Unmarshal. But I have no interest in string keys, I'd need to convert them to int somehow.
So what I'd like to know is:
Is there a way to achieve this, .ie getting a struct containing fields of map[int]float32 type directly from decoding,
Otherwise how to achieve this after decoding, in a memory efficient manner ?
Thanks
The JSON format only specifies key/value objects with string keys. Because of this, the encoding/json package only supports string keys as well.
The json/encoding documentation states:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
If you want to use encoding/json package and move it over to a map[int]float64, you can do the following (works with float32 as well):
package main
import (
"fmt"
"strconv"
)
func main() {
a := map[string]float64{"1":1, "2":4, "3":9, "5":25}
b := make(map[int]float64, len(a))
for k,v := range a {
if i, err := strconv.Atoi(k); err == nil {
b[i] = v
} else {
// A non integer key
}
}
fmt.Printf("%#v\n", b)
}
Playground
The encoding/json package includes an interface Unmarshaler which has a single method: UnmarshalJSON(data []byte) error.
If you're feeling brave you could implement that for the following:
type IntToFloat map[int]float32
func (itf *IntToFloat) UnmarshalJSON(data []byte) error {
if itf == nil {
return errors.New("Unmarshalling JSON for a null IntToFload")
}
// MAGIC Goes here.
return nil
}
EDIT
Note: http://golang.org/src/pkg/encoding/json/decode.go?s=2221:2269#L53 is where the std library version of Unmarshal comes from.
http://golang.org/pkg/encoding/json/#Unmarshaler is where the interface referenced above comes from.