Adding Arbitrary fields to json output of an unknown struct - json

In this stackoverflow post it's explained how to add arbitrary fields to a golang struct by using it as an anonymous. This works fine if you are working with known struct types, but I'm wondering how to do the same thing when dealing with an unknown struct or interface.
I wrote the following example to demonstrate:
package main
import (
"os"
"encoding/json"
"fmt"
)
type example interface{}
type Data struct {
Name string
}
func printInterface(val interface{}) {
example1 := struct {
example
Extra string
}{
example: val,
Extra: "text",
}
json.NewEncoder(os.Stdout).Encode(example1)
}
func printStructPointer(val *Data) {
example2 := struct {
*Data
Extra string
}{
Data: val,
Extra: "text",
}
json.NewEncoder(os.Stdout).Encode(example2)
}
func main() {
d := Data{Name:"name"}
fmt.Println("Example 1:")
printInterface(&d)
fmt.Println("Example 2:")
printStructPointer(&d)
}
This prints the following:
Example 1:
{"example":{"Name":"name"},"Extra":"text"}
Example 2:
{"Name":"name","Extra":"text"}
I'm so assuming that I was working within printInterface how do get the JSON output to look like the JSON output of printStructPointer?

There's an important difference between printInterface() and printStructPointer(). The first one embeds an interface type, while the second embeds a struct type (more specifically a pointer to a struct type).
When you embed a struct (or pointer to struct) type, the fields of the embedded type get promoted, so in the 2nd example it will be valid to write example2.Name. When you embed an interface type, an interface does not have fields, so no fields will be promoted. So it doesn't matter if the interface value wraps a struct (or pointer to struct), fields of that struct won't get promoted (they can't be).
Thus, in the printInterface() the interface wrapping a struct won't get "flattened" in the JSON result.
Solving it with generating a dynamic type using reflection
One way to solve this is to generate a dynamic type at runtime, using reflection (reflect package). This new type will be a struct, and it will contain an anonymous struct field being of the type that is wrapped in the passed interface, and will also contain our extra field (of type string).
This is how it could look like:
func printInterface(val interface{}) {
t2 := reflect.StructOf([]reflect.StructField{
reflect.StructField{
Name: "X",
Anonymous: true,
Type: reflect.TypeOf(val),
},
reflect.StructField{
Name: "Extra",
Type: reflect.TypeOf(""),
},
})
v2 := reflect.New(t2).Elem()
v2.Field(0).Set(reflect.ValueOf(val))
v2.FieldByName("Extra").SetString("text")
json.NewEncoder(os.Stdout).Encode(v2.Interface())
}
Output is as expected (try it on the Go Playground):
Example 1:
{"Name":"name","Extra":"text"}
Example 2:
{"Name":"name","Extra":"text"}
Solving it with marshaling twice
Another way would be to marshal the value, unmarshal it into a map, add the extra field and marshal it again:
func printInterface(val interface{}) error {
data, err := json.Marshal(val)
if err != nil {
return err
}
v2 := map[string]interface{}{}
if err := json.Unmarshal(data, &v2); err != nil {
return err
}
v2["Extra"] = "text"
return json.NewEncoder(os.Stdout).Encode(v2)
}
Output is the same. Try it on the Go Playground.
This solution is simpler, easier to follow, but it's slower as it marshals twice. Also note that in this example the fields in the result might be in different order, as iteration order on a map is not specified in Go (for details see Why can't Go iterate maps in insertion order?).

Here's one way:
package main
import (
"encoding/json"
"fmt"
"os"
)
type example interface{}
type Data struct {
Name string
}
func printInterface(val interface{}) {
if d, ok := val.(*Data); ok {
fmt.Println("Asserting type *Data for val is OK")
printStructPointer(d)
} else {
fmt.Println("Asserting type *Data for val is NOT OK")
}
}
func printStructPointer(val *Data) {
example2 := struct {
*Data
Extra string
}{
Data: val,
Extra: "text",
}
json.NewEncoder(os.Stdout).Encode(example2)
}
func main() {
d := Data{Name: "Testing"}
fmt.Println("Example 1:")
printInterface(&d)
fmt.Println("Example 2:")
printStructPointer(&d)
}
Playground: https://play.golang.org/p/OPotpTyUEz
You can also possibly use a type switch for an assertion, esp if you have many types. Hope this helps!

Related

How can I skip a field for json.Marshal & not for json.Unmarshal in go?

type Alpha struct {
Name string `json:"name"`
SkipWhenMarshal string `json:"skipWhenMarshal"`
}
func MarshalJSON(out interface{}){
json.Marshal(out)
}
Is it possible to ignore the SkipWhenMarshal field when I do json.Marshal but not when I do
json.Unmarshal.
It should work for any type who calls MarshalJSON
Field tag modifiers like "omitempty" and "-" apply to both marshaling and unmarshaling, so there's no automatic way.
You can implement a MarshalJSON for your type that ignores whatever fields you need. There's no need to implement a custom unmarshaler, because the default works for you.
E.g. something like this:
type Alpha struct {
Id int32
Name string
SkipWhenMarshal string
}
func (a Alpha) MarshalJSON() ([]byte, error) {
m := map[string]string{
"id": fmt.Sprintf("%d", a.Id),
"name": a.Name,
// SkipWhenMarshal *not* marshaled here
}
return json.Marshal(m)
}
You can also make it simpler by using an alias type:
func (a Alpha) MarshalJSON() ([]byte, error) {
type AA Alpha
aa := AA(a)
aa.SkipWhenMarshal = ""
return json.Marshal(aa)
}
Here SkipWhenMarshal will be output, but its value is zeroed out. The advantage in this approach is that if Alpha has many fields, you don't have to repeat them.
What you want simply cannot be done with encoding/json.
But you can have two types
type AlphaIn struct {
Name string `json:"name"`
Bar string `json:"skipWhenMarshal"`
}
type AlphaOut struct {
Name string `json:"name"`
Bar string `json:"-"`
}
Use AlphaIn to deserialise JSON with encoding/json.Unmarshal and use AlphaOut to serialise a struct with encoding/json.Marshal to JSON.
Now this alone would be absolute painful to work with but: struct tags do not play a role in convertibility between types which lets you convert from AlphaIn to AlphaOut with a simple type conversion:
var a AlphaIn = ...
var b AlphaOut = AlphaOut(a)
(A saner naming scheme would be Alpha and AlphaToJSON or soemthing like this.)
I would write a custom marshal function like so: playground
package main
import (
"encoding/json"
"fmt"
"reflect"
"strings"
)
type Alpha struct {
Name string `json:"name"`
SkipWhenMarshal string `json:"SkipWhenMarshal,skip"`
}
func main() {
var a Alpha
a.Name = "John"
a.SkipWhenMarshal = "Snow"
out, _ := marshal(a)
fmt.Println(string(out))
var b Alpha
json.Unmarshal([]byte(`{"Name":"Samwell","SkipWhenMarshal":"Tarly"}`), &b)
fmt.Println(b)
}
// custom marshaling function for json that accepts an additional tag named skip to ignore the field
func marshal(v Alpha) ([]byte, error) {
m := map[string]interface{}{}
ut := reflect.TypeOf(v)
uv := reflect.ValueOf(v)
for i := 0; i < ut.NumField(); i++ {
field := ut.Field(i)
js, ok := field.Tag.Lookup("json")
if !ok || !strings.Contains(js, "skip") {
intf := uv.Field(i).Interface()
switch val := intf.(type) {
case int, int8, uint8:
m[field.Name] = val
case string:
m[field.Name] = val
}
}
}
return json.Marshal(m)
}
It basically rebuilds the struct as a map without the skipped fields and passes it to the original Marshal function.
Now just add the SKIP tag to any field and it should work.
You just need to improve this part:
switch val := intf.(type) {
Since this only assumes you only have strings or ints as fields. Handling more types would be more ideal.
You can try this i.e. breaking the structure into components -> composition
while unmarshalling use Beta type. It will unmarshall only fields that are defined inside Beta struct
type Beta struct {
Name string `json:"name"`
}
type Alpha struct {
*Beta
SkipWhenMarshal string `json:"skipWhenMarshal"`
}

Why doesn't JSON parsing fail with completely different type passed to Decode()?

I have the following data structures which I'd like to parse from an API:
type OrderBook struct {
Pair string `json:"pair"`
UpdateTime int64 `json:"update_time"`
}
type depthResponse struct {
Result OrderBook `json:"result"`
// doesn't matter here
//Cmd string `json:"-"`
}
and when I parse the following:
data := `{"error":{"code":"3016","msg":"交易对错误"},"cmd":"depth"}`
It doesn't fail. Why?
Full source code (playground)
package main
import (
"encoding/json"
"fmt"
"log"
"strings"
)
type OrderBook struct {
Pair string `json:"pair"`
UpdateTime int64 `json:"update_time"`
}
type depthResponse struct {
Result OrderBook `json:"result"`
}
func main() {
data := `{"error":{"code":"3016","msg":"交易对错误"},"cmd":"depth"}`
r := strings.NewReader(data)
var resp depthResponse
if err := json.NewDecoder(r).Decode(&resp); err != nil {
log.Fatalf("We should end up here: %v", err)
}
fmt.Printf("%+v\n", resp)
}
That's the expected behaviour of Decode (as documented in the Unmarshal function):
https://golang.org/pkg/encoding/json/#Unmarshal
By default, object keys which don't have a corresponding struct field are ignored.
You can however use the DisallowUnknownFields() function (as described in the docs as well) to have it fail if the input JSON has fields not contained in the destination struct.
dec := json.NewDecoder(r)
dec.DisallowUnknownFields()
In that case, you'll get an error as you expect.
Modified playground here: https://play.golang.org/p/A0f6dxTXV34

How to get JSON when decoder fails to unmarshal?

I am using a json.Decoder to decode JSON delivered over a network stream. It works fine, but whenever someone sends data that doesn't fit the schema (e.g. sending a negative integer when the struct's field type is unsigned) it returns an error with a vague message that doesn't pinpoint the offending property. This makes debugging more difficult.
Is there any way to extract the JSON that the decoder was trying to unmarshal when it errored? Here's a small reproducable snippet:
package main
import (
"bytes"
"fmt"
"encoding/json"
)
func main() {
buff := bytes.NewBufferString("{\"bar\": -123}")
decoder := json.NewDecoder(buff)
var foo struct{
Bar uint32
}
if err := decoder.Decode(&foo); err != nil {
fmt.Println(err)
fmt.Println("TODO: how to get JSON that caused this error?")
} else {
fmt.Println(foo.Bar)
}
}
Or on playground: https://play.golang.org/p/-WdYBkYEzJ
Some information is in the error, which is of type *json.UnamrshalTypeError
type UnmarshalTypeError struct {
Value string // description of JSON value - "bool", "array", "number -5"
Type reflect.Type // type of Go value it could not be assigned to
Offset int64 // error occurred after reading Offset bytes
}
You can get the offset in the json string, the reflect.Type of the field being unmarshaled into, and the json description of the Value. This can still pose a problem for types that implement their own unmarshaling logic, which is referenced by issue 11858
As of Go 1.8 this is now possible. The UnmarshalTypeError type now contains Struct and Field values which provide the name of the struct and field which caused a type mismatch.
package main
import (
"bytes"
"fmt"
"encoding/json"
)
func main() {
buff := bytes.NewBufferString("{\"bar\": -123}")
decoder := json.NewDecoder(buff)
var foo struct{
Bar uint32
}
if err := decoder.Decode(&foo); err != nil {
if terr, ok := err.(*json.UnmarshalTypeError); ok {
fmt.Printf("Failed to unmarshal field %s \n", terr.Field)
} else {
fmt.Println(err)
}
} else {
fmt.Println(foo.Bar)
}
}
The error message string also was changed to contain this new information.
Go 1.8:
json: cannot unmarshal number -123 into Go struct field .Bar of type uint32
Go 1.7 and earlier:
json: cannot unmarshal number -123 into Go value of type uint32
You can get at each element, key, value, and even delimiters using decode.Token() such as this in the playground, and below (modified from your example):
package main
import (
"bytes"
"encoding/json"
"fmt"
)
func main() {
buff := bytes.NewBufferString(`{"foo": 123, "bar": -123, "baz": "123"}`)
decoder := json.NewDecoder(buff)
for {
t, err := decoder.Token()
if _, ok := t.(json.Delim); ok {
continue
}
fmt.Printf("type:%11T | value:%5v //", t, t)
switch t.(type) {
case uint32:
fmt.Println("you don't see any uints")
case int:
fmt.Println("you don't see any ints")
case string:
fmt.Println("handle strings as you will")
case float64:
fmt.Println("handle numbers as you will")
}
if !decoder.More() {
break
}
if err != nil {
fmt.Println(err)
}
}
}
This will output
type: string | value: foo //handle strings as you will
type: float64 | value: 123 //handle numbers as you will
type: string | value: bar //handle strings as you will
type: float64 | value: -123 //handle numbers as you will
type: string | value: baz //handle strings as you will
type: string | value: 123 //handle strings as you will
You can switch on the type and handle each one as you wish. I have shown a simple example of that as well, each of the "//comments" in the result are conditional based on the type.
You'll also notice that each numbers' type is float64, although they would fit into an int, or even uint in the case of the "foo" value, and I check for those types in the switch but they never get used. You would have to provide your own logic in order to convert the float64 values to the types you wanted if they could be and handle types that wouldn't convert as special cases, or errors or whatever you wanted.

Accessing Nested Map of Type map[string]interface{} in Golang

So I'm trying to parse a JSON response. It can be multiple levels deep. This is what I did:
var result map[string]interface{}
json.Unmarshal(apiResponse, &result)
Firstly, is this the right way to do it?
Lets say the response was as follows:
{
"args": {
"foo": "bar"
}
}
To access key foo, I saw a playground doing this:
result["args"].(map[string]interface{})["foo"]
Here, what is the .() notation? Is this correct?
The notation x.(T) is called a Type Assertion.
For an expression x of interface type and a type T, the primary expression x.(T) asserts that x is not nil and that the value stored in x is of type T.
Your example:
result["args"].(map[string]interface{})["foo"]
It means that the value of your results map associated with key "args" is of type map[string]interface{} (another map with string keys and any values). And you want to access the element of that map associated with the key "foo".
If you know noting about the input JSON format, then yes, you have to use a generic map[string]interface{} type to process it. If you know the exact structure of the input JSON, you can create a struct to match the expected fields, and doing so you can unmarshal a JSON text into a value of your custom struct type, for example:
type Point struct {
Name string
X, Y int
}
func main() {
in := `{"Name":"center","X":2,"Y":3}`
pt := Point{}
json.Unmarshal([]byte(in), &pt)
fmt.Printf("Result: %+v", pt)
}
Output:
Result: {Name:center X:2 Y:3}
Try it on the Go Playground.
Modeling your input
Your current JSON input could be modelled with this type:
type Data struct {
Args struct {
Foo string
}
}
And accessing Foo (try it on the Go Playground):
d := Data{}
json.Unmarshal([]byte(in), &d)
fmt.Println("Foo:", d.Args.Foo)
struct is the best option, but if you insist, you can add a type declaration for a map, then you can add methods to help
with the type assertions:
package main
import "encoding/json"
type dict map[string]interface{}
func (d dict) d(k string) dict {
return d[k].(map[string]interface{})
}
func (d dict) s(k string) string {
return d[k].(string)
}
func main() {
apiResponse := []byte(`{"args": {"foo": "bar"}}`)
var result dict
json.Unmarshal(apiResponse, &result)
foo := result.d("args").s("foo")
println(foo == "bar")
}
https://golang.org/ref/spec#Type_declarations

How to Unmarshal JSON into an interface in Go

I am trying to simultaneously unmarshal and strip fields from a number of different JSON responses into appropriate Go structs. To do this, I created a Wrappable interface that defines the Unwrap method (which strips the appropriate fields) and pass that interface to the code that unmarshals and unwraps. It looks like the following example (also at http://play.golang.org/p/fUGveHwiz9):
package main
import (
"encoding/json"
"fmt"
)
type Data struct {
A string `json:"a"`
B string `json:"b"`
}
type DataWrapper struct {
Elements []Data `json:"elems"`
}
type Wrapper interface {
Unwrap() []interface{}
}
func (dw DataWrapper) Unwrap() []interface{} {
result := make([]interface{}, len(dw.Elements))
for i := range dw.Elements {
result[i] = dw.Elements[i]
}
return result
}
func unmarshalAndUnwrap(data []byte, wrapper Wrapper) []interface{} {
err := json.Unmarshal(data, &wrapper)
if err != nil {
panic(err)
}
return wrapper.Unwrap()
}
func main() {
data := `{"elems": [{"a": "data", "b": "data"}, {"a": "data", "b": "data"}]}`
res := unmarshalAndUnwrap([]byte(data), DataWrapper{})
fmt.Println(res)
}
However, when I run the code, Go panics with the following error:
panic: json: cannot unmarshal object into Go value of type main.Wrapper
It seems the unmarshaller doesn't want to be passed a pointer to an interface. I am somewhat surprised by this given that I can get at the underlying type and fields using the reflect package within the unmarshalAndUnwrap method. Can anyone provide insight into this problem and how I might work around it?
As you stated, passing a non-pointer fails. Why are you trying to do this anyway?
Replace
res := unmarshalAndUnwrap([]byte(data), DataWrapper{})
by
res := unmarshalAndUnwrap([]byte(data), &DataWrapper{})
It should do the trick and it avoid unnecessary copy.
This error should help you understand: http://play.golang.org/p/jXxCxPQDOw