golang fails to parse json for reflection created object - json

I try to write simple message protocol in go and i've encountered a problem. I have a lot of message types and i want to have a dictionary like this to manipulate with messages:
var dict map[reflect.Type]int = map[reflect.Type]int{
reflect.TypeOf(DataMessage{}): 1000,
reflect.TypeOf(TextMessage{}): 1001,
//....
}
func GetMessageTypeId(value interface{}) int {
if id, ok := dict[reflect.TypeOf(value)]; ok {
return id
} else {
return -1
}
}
func GetValueByTypeId(typeId int) interface{} {
for typeDec, id := range dict {
if id == typeId {
return reflect.Zero(typeDec).Interface()
}
}
fmt.Println("Unknown message type", typeId)
return nil
}
It works fine, but when i instantiate message with GetValueByTypeId and try to unmarshall json into it - i receive map[string]interface instead of my message.
I've made simple example to reproduce the problem:
http://play.golang.org/p/QEyDN9vztr

Please read this article: http://research.swtch.com/interfaces, especially the "Memory Optimizations".
The interface{} by definition consists of two pointers - to method table (e.g. type) and to data it holds. So for
var destination3 interface{} = reflect.Zero(reflect.TypeOf(Message{})).Interface()
it is empty method table (as interface{} has no methods) and reference to Message{}. Taking reference from it returns the reference to this struct so the unmarhal overwrites it with whatever matches interface{}.
If the data interface{} variable holds is a pointer itself, then it is optimized in a way that this pointer is used instead creating interface{} structure. So getting reference to it gives the reference to original variable.
http://play.golang.org/p/KsIS29rUAX
package main
import "fmt"
func main() {
var m1 struct{ Data string }
var m2 interface{}
var m3 interface{}
m2 = &m1
m3 = m1
fmt.Printf("&m1=%p m2=%p &m3=%p\n", &m1, m2, &m3)
}
In your case, using Zero is equivalent to m3 in the example above. Using New is equivalent to m2.

I've found the way how to do what i need
val := reflect.New(reflect.TypeOf(Message{}))
json.Unmarshal(data, val.Interface())
return val.Elem().Interface()
http://play.golang.org/p/8g9FSg3MSj
But was was wrong wit the first version???
It Looks like reflect.Zero(type) should be equivalent to reflect.New(type).Elem() - am i wrong?

Related

Unmarshal remaining JSON after performing custom unmarshalling

I have a JSON object That contains an implementation of an interface within it. I'm attempting to take that JSON and marshal it into a struct whilst creating the implementation of the interface.
I've managed to get it to implement the interface with a custom JSON unmarshal function however I'm struggling to piece together how to then marshal the rest of the fields
I've created an example in the Go playground
https://play.golang.org/p/ztF7H7etdjM
My JSON being passed into my application is
{
"address":"1FYuJ4MsVmpzPoFJ6svJMJfygn91Eubid9",
"nonce":13,
"network_id":"qadre.demo.balance",
"challenge":"f2b19e71876c087e681fc092ea3a34d5680bbfe772e40883563e1d5513bb593f",
"type":"verifying_key",
"verifying_key":{
"verifying_key":"3b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29",
"fqdn":"huski.service.key"
},
"signature":"a3bf8ee202a508d5a5632f50b140b70b7095d8836493dc7ac4159f6f3350280078b3a58b2162a240bc8c7485894554976a9c7b5d279d3f5bf49fec950f024e02",
"fqdn":"huski.service.SingleKeyProof"
}
I've attempted to do a json.Unmarshal and pass in a new struct for the remaining fields however it seems to put me in an infinite loop, my application hangs and then crashes
The best solution I've come up with so far is to marshal the JSON into a `map[string]interface{} and do each field separately, this feels very clunky though
var m map[string]interface{}
if err := json.Unmarshal(data, &m); err != nil {
return err
}
ad, ok := m["address"]
if ok {
s.Address = ad.(string)
}
fqdn, ok := m["fqdn"]
if ok {
s.FQDN = fqdn.(string)
}
n, ok := m["nonce"]
if ok {
s.Nonce = int64(n.(float64))
}
c, ok := m["challenge"]
if ok {
s.Challenge = []byte(c.(string))
}
network, ok := m["network_id"]
if ok {
s.NetworkID = network.(string)
}
sig, ok := m["signature"]
if ok {
s.Signature = []byte(sig.(string))
}
The reason your code gets into an infinite loop when you try to unmarshal the rest of the fields is because, I presume, the implementation of UnmarshalJSON after its done unmarshaling the verifying key, calls json.Unmarshal with the receiver, which in turn calls the UnmarshalJSON method on the receiver and so they invoke each other ad infinitum.
What you can do is to create a temporary type using the existing type as its definition, this will "keep the structure" but "drop the methods", then unmarshal the rest of the fields into an instance of the new type, and, after unmarshal is done, convert the instance to the original type and assign that to the receiver.
While this fixes the infinite loop, it also re-introduces the original problem of json.Unmarshal not being able to unmarshal into a non-empty interface type. To fix that you can embed the new type in another temporary struct that has a field with the same json tag as the problematic field which will cause it to be "overshadowed" while json.Unmarshal is doing its work.
type SingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey PublicKey `json:"verifying_key"`
Signature []byte `json:"signature"`
}
func (s *SingleKey) UnmarshalJSON(data []byte) error {
type _SingleKey SingleKey
var temp struct {
RawKey json.RawMessage `json:"verifying_key"`
_SingleKey
}
if err := json.Unmarshal(data, &temp); err != nil {
return err
}
*s = SingleKey(temp._SingleKey)
switch s.Type {
case "verifying_key":
s.VerifyingKey = &PublicKeyImpl{}
// other cases ...
}
return json.Unmarshal([]byte(temp.RawKey), s.VerifyingKey)
}
https://play.golang.org/p/L3gdQZF47uN
Looking at what you've done in your custom unmarshalling function, you seem to be passing in a map with the name of fields as index, and the reflect.Type you want to unmarshal said value into. That, to me, suggests that the keys might be different for different payloads, but that each key has a distinct type associated with it. You can perfectly handle data like this with a simple wrapper type:
type WrappedSingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey json.RawMessage `json:"verifying_key"`
OtherKey json.RawMessage `json:"other_key"`
Signature []byte `json:"signature"`
}
type SingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey *PublicKey `json:"verifying_key,omitempty"`
OtherType *OtherKey `json:"other_key,omitempty"`
Signature []byte `json:"signature"`
}
So I've changed the type of your VerifyingKey field to a json.RawMessage. That's basically telling json.Unmarshal to leave that as raw JSON input. For every custom/optional field, add a corresponding RawMessage field.
In the unwrapped type, I've changed VerifyingKey to a pointer and added the omitempty bit to the tag. That's just to accomodate mutliple types, and not have to worry about custom marshalling to avoid empty fields, like the included OtherType field I have. To get what you need, then:
func (s *SingleKey) UnmarshalJSON(data []byte) error {
w := WrappedSingleKey{} // create wrapped instance
if err := json.Unmarshal(data, &w); err != nil {
return err
}
switch w.Type {
case "verifying_key":
var pk PublicKey
if err := json.Unmarshal([]byte(w.VerifyingKey), &pk); err != nil {
return err
}
s.VerifyingKey = &pk // assign
case "other_key":
var ok OtherKey
if err := json.Unmarshal([]byte(w.OtherKey), &ok); err != nil {
return err
}
s.OtherKey = &ok
}
// copy over the fields that didn't require anything special
s.FQDN = w.FQDN
s.Address = w.Address
}
This is a fairly simple approach, does away with the reflection, tons of functions, and is quite commonly used. It's something that lends itself quite well to code generation, too. The individual assignment of the fields is a bit tedious, though. You might think that you can solve that by embedding the SingleKey type into the wrapper, but be careful: this will recursively call your custom unmarshaller function.
You could, for example, update all the fields in the WRapped type to be pointers, and have them point to fields on your actual type. That does away with the manual copying of fields... It's up to you, really.
Note
I didn't test this code, just wrote it as I went along. It's something I've used in the past, and I believe what I wrote here should work, but no guarantees (as in: you might need to debug it a bit)

How to type assert a dynamically reflection-generated struct interface

I'm new to Go so please bear with me if this is a trivial problem. I am using a home grown "type registry" to map type names to their type, so as to generate them dynamically based on use cases that point to the various type names (I'm basically trying for a simple solution to polymorphic Aggregation JSON response structures in Elasticsearch, but of course this could apply to many other dynamic/polymorphic situations).
I'm using the solution provided by dolmen in this question: is there a way to create an instance of a struct from a string? :
var typeRegistry = make(map[string]reflect.Type)
func registerType(typedNil interface{}) {
t := reflect.TypeOf(typedNil).Elem()
typeRegistry[t.Name()] = t
}
func init() {
registerType((*playlistIDAggregation)(nil))
registerType((*srcIDAggregation)(nil))
registerType((*assetIDAggregation)(nil))
}
func makeInstance(name string) interface{} {
return reflect.New(typeRegistry[name]).Elem().Interface()
}
I then want to use my dynamically generated struct as the target for the JSON unmarshalling of the Aggregations node in my ES response:
playlistIDAgg := makeInstance("playlistIDAggregation")
err = json.Unmarshal(esResponse.Aggregations, &playlistIDAgg)
This isn't working like I want it to, as the Unmarshal is trying to unmarshall into an empty interface instead of the underlying struct type. it's putting the data under "data" nodes in the playlistIDAgg variable, and those data fields are of course map[string]interface{}. Am I just missing the way to type assert my playlistIDAgg interface or is there a better way to do this?
EDIT---
The questions in the comments made me realize an edit to this question was long overdue.
In my particular case, the structs I defined, to bind to my Bucket aggregations returned by Elasticsearch, have a similar structure and vary only by their root JSON tag, which ES uses to name the aggregation and strongly type it. E.g.
type <name>Aggregation struct {
Agg BucketAggregationWithCamIDCardinality `json:"<name>"`
}
So, rather than a type registry, my particular problem could be solved by dynamically setting the JSON tag on the struct based on the particular use case.
In addition, a heavier but more robust option would be to leverage Oliver Eilhard's Elasticsearch Go client lib, called Elastic, which has built-in support for all the ES aggregation response structures:
https://github.com/olivere/elastic/
I changed makeInstance function by getting address of element and add target structs with exposed fields.
func makeInstance(name string) interface{} {
return reflect.New(typeRegistry[name]).Elem().Addr().Interface()
}
Here is the working code
package main
import (
"encoding/json"
"fmt"
"reflect"
)
type playlistIDAggregation struct {
PlaylistID string
}
type srcIDAggregation struct {
SrcID string
}
type assetIDAggregation struct {
AssetID string
}
var typeRegistry = make(map[string]reflect.Type)
func registerType(typedNil interface{}) {
t := reflect.TypeOf(typedNil).Elem()
typeRegistry[t.Name()] = t
}
func init() {
registerType((*playlistIDAggregation)(nil))
registerType((*srcIDAggregation)(nil))
registerType((*assetIDAggregation)(nil))
}
func makeInstance(name string) interface{} {
return reflect.New(typeRegistry[name]).Elem().Addr().Interface()
}
func main() {
playlistIDAgg := makeInstance("playlistIDAggregation")
fmt.Printf("Type = %[1]T => %#[1]v\n", playlistIDAgg)
err := json.Unmarshal([]byte(`{"PlayListID": "dummy-id"}`), &playlistIDAgg)
if err != nil {
panic(err)
}
fmt.Printf("Type = %[1]T => %#[1]v\n", playlistIDAgg)
}
https://play.golang.org/p/dn19_iG5Xjz

GoLang Redis : Map & Slice

I'm using GoLang to get a data from redis hash and then map into a struct.
type Person struct {
ID string `json:"id"`
FirstName string `json:"firstName"`
LastName string `json:"lastName"`
Filters interface{} `json:"filters"`
Type string `json:"type"`
}
In Redis, a hash field contains a stringified JSON.
HGET hashname fieldname
Above returns a stringified JSON.
Now "filters" key can be array or map based on the type (That's why I defined Filters type as interface in struct).
I marshall the JSON like below:
var p Person
content, err := redis.HGet("hashName", "id").Result()
_ = json.Unmarshal([]byte(content), &p)
Now I have to loop over filters like below but this is giving error that cannot range over p.Filters (type interface {}) (I understand why this error is coming)
for _, filter := range p.Filters {
fmt.Println(filter)
}
Is there any way we can handle this situation?
Thanks,
Shashank
You need to convert Filters type interface{} into the expected slice. If you don't really know what type it will be, you can use map[string]interface{}. Therefore change your Filters type to map[string]interface{}.
Per more information
If Filters can be an array (or not), then you might consider a type switch:
A simple example:
package main
import (
"encoding/json"
"fmt"
)
func main() {
var i interface{}
//json.Unmarshal([]byte(`["hi"]`), &i)
json.Unmarshal([]byte(`{"a":"hi"}`), &i)
switch i.(type) {
case []interface{}:
println("ARRAY")
case map[string]interface{}:
println("NOT ARRAY")
default:
fmt.Printf("%T\n", i)
}
}

Unmarshalling Entities from multiple JSON arrays without using reflect or duplicating code

I'm making an JSON API wrapper client that needs to fetch paginated results, where the URL to the next page is provided by the previous page. To reduce code duplication for the 100+ entities that share the same response format, I would like to have a single client method that fetches and unmarshalls the different entities from all paginated pages.
My current approach in a simplified (pseudo) version (without errors etc):
type ListResponse struct {
Data struct {
Results []interface{} `json:"results"`
Next string `json:"__next"`
} `json:"d"`
}
func (c *Client) ListRequest(uri string) listResponse ListResponse {
// Do a http request to uri and get the body
body := []byte(`{ "d": { "__next": "URL", "results": []}}`)
json.NewDecoder(body).Decode(&listResponse)
}
func (c *Client) ListRequestAll(uri string, v interface{}) {
a := []interface{}
f := c.ListRequest(uri)
a = append(a, f.Data.Results...)
var next = f.Data.Next
for next != "" {
r := c.ListRequest(next)
a = append(a, r.Data.Results...)
next = r.Data.Next
}
b, _ := json.Marshal(a)
json.Unmarshal(b, v)
}
// Then in a method requesting all results for a single entity
var entities []Entity1
client.ListRequestAll("https://foo.bar/entities1.json", &entities)
// and somewehere else
var entities []Entity2
client.ListRequestAll("https://foo.bar/entities2.json", &entities)
The problem however is that this approach is inefficient and uses too much memory etc, ie first Unmarshalling in a general ListResponse with results as []interface{} (to see the next URL and concat the results into a single slice), then marshalling the []interface{} for unmarshalling it directly aftwards in the destination slice of []Entity1.
I might be able to use the reflect package to dynamically make new slices of these entities, directly unmarshal into them and concat/append them afterwards, however if I understand correctly I better not use reflect unless strictly necessary...
Take a look at the RawMessage type in the encoding/json package. It allows you to defer the decoding of json values until later. For example:
Results []json.RawMessage `json:"results"`
or even...
Results json.RawMessage `json:"results"`
Since json.RawMessage is just a slice of bytes this will be much more efficient then the intermediate []interface{} you are unmarshalling to.
As for the second part on how to assemble these into a single slice given multiple page reads you could punt that question to the caller by making the caller use a slice of slices type.
// Then in a method requesting all results for a single entity
var entityPages [][]Entity1
client.ListRequestAll("https://foo.bar/entities1.json", &entityPages)
This still has the unbounded memory consumption problem your general design has, however, since you have to load all of the pages / items at once. You might want to consider changing to an Open/Read abstraction like working with files. You'd have some Open method that returns another type that, like os.File, provides a method for reading a subset of data at a time, while internally requesting pages and buffering as needed.
Perhaps something like this (untested):
type PagedReader struct {
c *Client
buffer []json.RawMessage
next string
}
func (r *PagedReader) getPage() {
f := r.c.ListRequest(r.next)
r.next = f.Data.Next
r.buffer = append(r.buffer, f.Data.Results...)
}
func (r *PagedReader) ReadItems(output []interface{}) int {
for len(output) > len(buffer) && r.next != "" {
r.getPage()
}
n := 0
for i:=0;i<len(output)&&i< len(r.buffer);i++ {
json.Unmarshal(r.buffer[i], output[i] )
n++
}
r.buffer = r.buffer[n:]
return n
}

How to access interface fields on json decode?

I have a json document and I'm using a client which decodes the document in an interface (instead of struct) as below:
var jsonR interface{}
err = json.Unmarshal(res, &jsonR)
How can I access the interface fields? I've read the go doc and blog but my head still can't get it. Their example seem to show only that you can decode the json in an interface but doesn't explain how its fields can be used.
I've tried to use a range loop but it seems the story ends when I reach a map[string]interface. The fields that I need seem to be in the interface.
for k, v := range jsonR {
if k == "topfield" {
fmt.Printf("k is %v, v is %v", k, v)
}
}
The value inside the interface depends on the json structure you're parsing. If you have a json dictionary, the dynamic type of jsonR will be: map[string]interface{}.
Here's an example.
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
a := []byte(`{"topfield": 123}`)
var v interface{}
if err := json.Unmarshal(a, &v); err != nil {
log.Fatalf("unmarshal failed: %s", err)
}
fmt.Printf("value is %v", v.(map[string]interface{})["topfield"])
}
Parsing json like this can be very difficult. The default type of a parse is map[string]interface{}. The Problem arises when you have another complex data structure within the main json(like another list or object). The best way to go about decoding json is defining a struct to hold data. Not only will the values be of the correct type but you can extract the specific data you actually care about.
Your struct can look like this:
type Top struct {
Topfield int `json:"topfield"`
}
which can be decoded like this:
a := []byte(`{"topfield": 123}`)
var data Top
json.Unmarshal(a, &data) //parse the json into data
now you can use regular struct operations to access your data like this:
value := data.Topfield
json which contains more complex data can also be easyli decoded. Perhaps you have a list in your data somewhere, you can use a struct like the following to extract it:
type Data struct {
States []string `json:"states"`
PopulationData []Country `json:"popdata"`
}
type Country struct {
Id int `json:"id"`
LastCensusPop int `json:"lcensuspopulation"`
Gdp float64 `json:"gdp"`
}
such a structure can not only parse list but also parse objects withing fields.