I have a JSON object That contains an implementation of an interface within it. I'm attempting to take that JSON and marshal it into a struct whilst creating the implementation of the interface.
I've managed to get it to implement the interface with a custom JSON unmarshal function however I'm struggling to piece together how to then marshal the rest of the fields
I've created an example in the Go playground
https://play.golang.org/p/ztF7H7etdjM
My JSON being passed into my application is
{
"address":"1FYuJ4MsVmpzPoFJ6svJMJfygn91Eubid9",
"nonce":13,
"network_id":"qadre.demo.balance",
"challenge":"f2b19e71876c087e681fc092ea3a34d5680bbfe772e40883563e1d5513bb593f",
"type":"verifying_key",
"verifying_key":{
"verifying_key":"3b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29",
"fqdn":"huski.service.key"
},
"signature":"a3bf8ee202a508d5a5632f50b140b70b7095d8836493dc7ac4159f6f3350280078b3a58b2162a240bc8c7485894554976a9c7b5d279d3f5bf49fec950f024e02",
"fqdn":"huski.service.SingleKeyProof"
}
I've attempted to do a json.Unmarshal and pass in a new struct for the remaining fields however it seems to put me in an infinite loop, my application hangs and then crashes
The best solution I've come up with so far is to marshal the JSON into a `map[string]interface{} and do each field separately, this feels very clunky though
var m map[string]interface{}
if err := json.Unmarshal(data, &m); err != nil {
return err
}
ad, ok := m["address"]
if ok {
s.Address = ad.(string)
}
fqdn, ok := m["fqdn"]
if ok {
s.FQDN = fqdn.(string)
}
n, ok := m["nonce"]
if ok {
s.Nonce = int64(n.(float64))
}
c, ok := m["challenge"]
if ok {
s.Challenge = []byte(c.(string))
}
network, ok := m["network_id"]
if ok {
s.NetworkID = network.(string)
}
sig, ok := m["signature"]
if ok {
s.Signature = []byte(sig.(string))
}
The reason your code gets into an infinite loop when you try to unmarshal the rest of the fields is because, I presume, the implementation of UnmarshalJSON after its done unmarshaling the verifying key, calls json.Unmarshal with the receiver, which in turn calls the UnmarshalJSON method on the receiver and so they invoke each other ad infinitum.
What you can do is to create a temporary type using the existing type as its definition, this will "keep the structure" but "drop the methods", then unmarshal the rest of the fields into an instance of the new type, and, after unmarshal is done, convert the instance to the original type and assign that to the receiver.
While this fixes the infinite loop, it also re-introduces the original problem of json.Unmarshal not being able to unmarshal into a non-empty interface type. To fix that you can embed the new type in another temporary struct that has a field with the same json tag as the problematic field which will cause it to be "overshadowed" while json.Unmarshal is doing its work.
type SingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey PublicKey `json:"verifying_key"`
Signature []byte `json:"signature"`
}
func (s *SingleKey) UnmarshalJSON(data []byte) error {
type _SingleKey SingleKey
var temp struct {
RawKey json.RawMessage `json:"verifying_key"`
_SingleKey
}
if err := json.Unmarshal(data, &temp); err != nil {
return err
}
*s = SingleKey(temp._SingleKey)
switch s.Type {
case "verifying_key":
s.VerifyingKey = &PublicKeyImpl{}
// other cases ...
}
return json.Unmarshal([]byte(temp.RawKey), s.VerifyingKey)
}
https://play.golang.org/p/L3gdQZF47uN
Looking at what you've done in your custom unmarshalling function, you seem to be passing in a map with the name of fields as index, and the reflect.Type you want to unmarshal said value into. That, to me, suggests that the keys might be different for different payloads, but that each key has a distinct type associated with it. You can perfectly handle data like this with a simple wrapper type:
type WrappedSingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey json.RawMessage `json:"verifying_key"`
OtherKey json.RawMessage `json:"other_key"`
Signature []byte `json:"signature"`
}
type SingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey *PublicKey `json:"verifying_key,omitempty"`
OtherType *OtherKey `json:"other_key,omitempty"`
Signature []byte `json:"signature"`
}
So I've changed the type of your VerifyingKey field to a json.RawMessage. That's basically telling json.Unmarshal to leave that as raw JSON input. For every custom/optional field, add a corresponding RawMessage field.
In the unwrapped type, I've changed VerifyingKey to a pointer and added the omitempty bit to the tag. That's just to accomodate mutliple types, and not have to worry about custom marshalling to avoid empty fields, like the included OtherType field I have. To get what you need, then:
func (s *SingleKey) UnmarshalJSON(data []byte) error {
w := WrappedSingleKey{} // create wrapped instance
if err := json.Unmarshal(data, &w); err != nil {
return err
}
switch w.Type {
case "verifying_key":
var pk PublicKey
if err := json.Unmarshal([]byte(w.VerifyingKey), &pk); err != nil {
return err
}
s.VerifyingKey = &pk // assign
case "other_key":
var ok OtherKey
if err := json.Unmarshal([]byte(w.OtherKey), &ok); err != nil {
return err
}
s.OtherKey = &ok
}
// copy over the fields that didn't require anything special
s.FQDN = w.FQDN
s.Address = w.Address
}
This is a fairly simple approach, does away with the reflection, tons of functions, and is quite commonly used. It's something that lends itself quite well to code generation, too. The individual assignment of the fields is a bit tedious, though. You might think that you can solve that by embedding the SingleKey type into the wrapper, but be careful: this will recursively call your custom unmarshaller function.
You could, for example, update all the fields in the WRapped type to be pointers, and have them point to fields on your actual type. That does away with the manual copying of fields... It's up to you, really.
Note
I didn't test this code, just wrote it as I went along. It's something I've used in the past, and I believe what I wrote here should work, but no guarantees (as in: you might need to debug it a bit)
Related
I'm new to golang generics and have the following setup.
I've gathered loads of different kinds of reports.
Each report has enclosing fields
So I wrapped it in a ReportContainerImpl
I've used a type argument of [T Reportable] where the Reportable is defined as follows
type Reportable interface {
ExportDataPointReport | ImportDataPointReport | MissingDataPointReport | SensorThresoldReport
}
Each of the type in the type constraint is structs that is to be embedded in the container.
type ReportContainerImpl[T Reportable] struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body T `json:"body"`
}
I use a discriminator ReportType to determine the concrete type when Unmarshal.
type ReportType string
const (
ReportTypeExportDataPointReport ReportType = "ExportDataPointReport"
ReportTypeImportDataPointReport ReportType = "ImportDataPointReport"
ReportTypeMissingDataPointReport ReportType = "MissingDataPointReport"
ReportTypeSensorThresoldReport ReportType = "SensorThresoldReport"
)
Since go does not support type assertion for struct (only interfaces) it is not possible to cast the type when Unmarshal. Also go does not support pointer to the "raw" generic type. Hence, I've created a interface that the ReportContainerImpl implements.
type ReportContainer interface {
GetLocationID() string
GetProvider() string
GetReportType() ReportType
GetBody() interface{}
}
The problem I then get is that I cannot do type constrains on the return type in any form or shape and am back at "freetext semantics" on the GetBody() function to allow for type assertion when Unmarshal is done.
container, err := UnmarshalReportContainer(data)
if rep, ok := container.GetBody().(ExportDataPointReport); ok {
// Use the ReportContainerImpl[ExportDataPointReport] here...
}
Maybe I'm getting this wrong? - but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Do you have a better suggestion how to solve this in a type (safer) way?
Cheers,
Mario :)
For completeness I add the UnmarshalReportContainer here
func UnmarshalReportContainer(data []byte) (ReportContainer, error) {
type Temp struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body *json.RawMessage `json:"body"`
}
var temp Temp
err := json.Unmarshal(data, &temp)
if err != nil {
return nil, err
}
switch temp.ReportType {
case ReportTypeExportDataPointReport:
var report ExportDataPointReport
err := json.Unmarshal(*temp.Body, &report)
return &ReportContainerImpl[ExportDataPointReport]{
LocationID: temp.LocationID,
Provider: temp.Provider,
ReportType: temp.ReportType,
Body: report,
}, err
// ...
}
}
but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Precisely.
The concrete types needed to instantiate some generic type or function like ReportContainerImpl or UnmarshalReportContainer must be known at compile time, when you write the code. JSON unmarshalling instead occurs at run-time, when you have the byte slice populated with the actual data.
To unmarshal dynamic JSON based on some discriminatory value, you still need a switch.
Do you have a better suggestion how to solve this in a type (safer) way?
Just forgo parametric polymorphism. It's not a good fit here. Keep the code you have now with json.RawMessage, unmarshal the dynamic data conditionally in the switch and return the concrete structs that implement ReportContainer interface.
As a general solution — if, and only if, you can overcome this chicken-and-egg problem and make type parameters known at compile time, you can write a minimal generic unmarshal function like this:
func unmarshalAny[T any](bytes []byte) (*T, error) {
out := new(T)
if err := json.Unmarshal(bytes, out); err != nil {
return nil, err
}
return out, nil
}
This is only meant to illustrate the principle. Note that json.Unmarshal already accepts any type, so if your generic function actually does nothing except new(T) and return, like in my example, it is no different than "inlining" the entire thing as if unmarshalAny didn't exist.
v, err := unmarshalAny[SomeType](src)
functionally equivalent as
out := &SomeType{}
err := json.Unmarshal(bytes, out)
If you plan to put more logic in unmarshalAny, its usage may be warranted. Your mileage may vary; in general, don't use type parameters when it's not actually necessary.
I am working on a general JSON based message passing protocol in Go. What I would like to do is have a BaseMessage that has general information like Type, timestamp, etc. But at the same time I want to be able to define more specific message structures for certain types of data.
For example:
type Message struct {
Type string `json:type`
Timestamp string `json:timestamp`
}
type EventMessage struct {
Message
EventType string
EventCreator string
EventData interface{}
}
I have a set of handlers and to determine which handler should process the message I decode the JSON to the general Message type first to check the Type field. For this example I would get the handler associated with an "Event" message type.
I run into problems when I then want to assert the EventMessage type onto the structure.
The following code is very rough, but hopefully it displays my general idea of how I am trying to handle the messages.
type Handler func(msg Message) Message
handlers := make(map[string]Handler)
var msg Message
decoder.Decode(&msg)
handler := handlers[msg.Type]
handler(msg)
I have tried to use an interface{} but then the JSON decoder just creates a map which I then cant assert either type on. I have figured out workarounds that make it possible, but its very ugly, probably not efficient, and most likely error prone. I would like to keep things simple and straightforward so this code can be easily maintained.
Is there a method of handling generic JSON objects in Go so that the decoded JSON could be one of many struct formats?
I have also played with the idea of having more specific info in a Data interface{} in the main Message struct, but then I run into the same problem of not being able to assert any types onto the interface. There has to be a better way to handle JSON formats that I am just missing.
One way to handle this is to define a struct for the fixed part of the message with a json.RawMessage field to capture the variant part of the message. Decode the json.RawMessage to a type specific to the variant:
type Message struct {
Type string `json:"type"`
Timestamp string `json:"timestamp"`
Data json.RawMessage
}
type Event struct {
Type string `json:"type"`
Creator string `json:"creator"`
}
var m Message
if err := json.Unmarshal(data, &m); err != nil {
log.Fatal(err)
}
switch m.Type {
case "event":
var e Event
if err := json.Unmarshal([]byte(m.Data), &e); err != nil {
log.Fatal(err)
}
fmt.Println(m.Type, e.Type, e.Creator)
default:
log.Fatal("bad message type")
}
playground example
I am working on a general JSON based message passing protocol in Go. What I would like to do is have a BaseMessage that has general information like Type, timestamp, etc. But at the same time I want to be able to define more specific message structures for certain types of data.
For example:
type Message struct {
Type string `json:type`
Timestamp string `json:timestamp`
}
type EventMessage struct {
Message
EventType string
EventCreator string
EventData interface{}
}
I have a set of handlers and to determine which handler should process the message I decode the JSON to the general Message type first to check the Type field. For this example I would get the handler associated with an "Event" message type.
I run into problems when I then want to assert the EventMessage type onto the structure.
The following code is very rough, but hopefully it displays my general idea of how I am trying to handle the messages.
type Handler func(msg Message) Message
handlers := make(map[string]Handler)
var msg Message
decoder.Decode(&msg)
handler := handlers[msg.Type]
handler(msg)
I have tried to use an interface{} but then the JSON decoder just creates a map which I then cant assert either type on. I have figured out workarounds that make it possible, but its very ugly, probably not efficient, and most likely error prone. I would like to keep things simple and straightforward so this code can be easily maintained.
Is there a method of handling generic JSON objects in Go so that the decoded JSON could be one of many struct formats?
I have also played with the idea of having more specific info in a Data interface{} in the main Message struct, but then I run into the same problem of not being able to assert any types onto the interface. There has to be a better way to handle JSON formats that I am just missing.
One way to handle this is to define a struct for the fixed part of the message with a json.RawMessage field to capture the variant part of the message. Decode the json.RawMessage to a type specific to the variant:
type Message struct {
Type string `json:"type"`
Timestamp string `json:"timestamp"`
Data json.RawMessage
}
type Event struct {
Type string `json:"type"`
Creator string `json:"creator"`
}
var m Message
if err := json.Unmarshal(data, &m); err != nil {
log.Fatal(err)
}
switch m.Type {
case "event":
var e Event
if err := json.Unmarshal([]byte(m.Data), &e); err != nil {
log.Fatal(err)
}
fmt.Println(m.Type, e.Type, e.Creator)
default:
log.Fatal("bad message type")
}
playground example
I would like to UnmarshalJSON a struct containing an interface as follows:
type Filterer interface {
Filter(s string) error
}
type FieldFilter struct {
Key string
Val string
}
func (ff *FieldFilter) Filter(s string) error {
// Do something
}
type Test struct {
Name string
Filters []Filterer
}
My idea was to send a json like so:
{
"Name": "testing",
"Filters": [
{
"FieldFilter": {
"Key": "key",
"Val": "val"
}
}
]
}
However, when sending this json to the unmarshaler, the following exception returns: json: cannot unmarshal object into Go struct field Test.Filters of type Filterer
I understand the problem fully, but do not know how to approach this problem wisely. Looking for advice on an idiomatic way to solving this problem in go.
Following my own question, I researched how one could implement UnmarshalJSON for interface lists. Ultimately this led me to publish a blog post on how to do this properly. Basically there are 2 main solutions:
Parse the required JSON string into a map[string]*json.RawMessage and work your way from there.
Make an alias for the interface list and implement UnmarshalJSON for that alias. However, you'll still need to work with map[string]*json.RawMessage and some manual work. Nothing comes without a price!
I highly suggest taking the seconds approach. While these two solutions may result in the same amount of code lines, taking advantage of type aliasing and being less dependent on json.RawMessage types will make a more easy to manage code, especially when it is required to support multiple interfaces on the UnmarshalJSON implementation
To directly answer the question, start with making a type alias for the interface list:
type Filterers []Filterer
Now continue with implementing the decoding of the JSON:
func (f *Filterers) UnmarshalJSON(b []byte) error {
var FilterFields map[string]*json.RawMessage
if err := json.Unmarshal(b, &FilterFields); err != nil {
return err
}
for LFKey, LFValue := range FilterFields {
if LFKey == "FieldFilter" {
var MyFieldFilters []*json.RawMessage
if err := json.Unmarshal(*LFValue, &MyFieldFilters); err != nil {
return err
}
for _, MyFieldFilter := range MyFieldFilters {
var filter FieldFilter
if err := json.Unmarshal(*MyFieldFilter, &filter); err != nil {
return err
}
*f = append(*f, &filter)
}
}
}
return nil
}
A detailed explanation (with some examples and a full working code snippets) of the second approach is available on my own blog
There is no way for Unmarshal to know what type it should use. The only case where it can just "make something up" is if it's asked to unmarshal into an interface{}, in which case it will use the rules in the documentation. Since none of those types can be put into a []Filterer, it cannot unmarshal that field. If you want to unmarshal into a struct type, you must specify the field to be of that type.
You can always unmarshal into an intermediate struct or map type, and then do your own conversion from that into whatever types you want.
I am working on a general JSON based message passing protocol in Go. What I would like to do is have a BaseMessage that has general information like Type, timestamp, etc. But at the same time I want to be able to define more specific message structures for certain types of data.
For example:
type Message struct {
Type string `json:type`
Timestamp string `json:timestamp`
}
type EventMessage struct {
Message
EventType string
EventCreator string
EventData interface{}
}
I have a set of handlers and to determine which handler should process the message I decode the JSON to the general Message type first to check the Type field. For this example I would get the handler associated with an "Event" message type.
I run into problems when I then want to assert the EventMessage type onto the structure.
The following code is very rough, but hopefully it displays my general idea of how I am trying to handle the messages.
type Handler func(msg Message) Message
handlers := make(map[string]Handler)
var msg Message
decoder.Decode(&msg)
handler := handlers[msg.Type]
handler(msg)
I have tried to use an interface{} but then the JSON decoder just creates a map which I then cant assert either type on. I have figured out workarounds that make it possible, but its very ugly, probably not efficient, and most likely error prone. I would like to keep things simple and straightforward so this code can be easily maintained.
Is there a method of handling generic JSON objects in Go so that the decoded JSON could be one of many struct formats?
I have also played with the idea of having more specific info in a Data interface{} in the main Message struct, but then I run into the same problem of not being able to assert any types onto the interface. There has to be a better way to handle JSON formats that I am just missing.
One way to handle this is to define a struct for the fixed part of the message with a json.RawMessage field to capture the variant part of the message. Decode the json.RawMessage to a type specific to the variant:
type Message struct {
Type string `json:"type"`
Timestamp string `json:"timestamp"`
Data json.RawMessage
}
type Event struct {
Type string `json:"type"`
Creator string `json:"creator"`
}
var m Message
if err := json.Unmarshal(data, &m); err != nil {
log.Fatal(err)
}
switch m.Type {
case "event":
var e Event
if err := json.Unmarshal([]byte(m.Data), &e); err != nil {
log.Fatal(err)
}
fmt.Println(m.Type, e.Type, e.Creator)
default:
log.Fatal("bad message type")
}
playground example