I'm new to golang generics and have the following setup.
I've gathered loads of different kinds of reports.
Each report has enclosing fields
So I wrapped it in a ReportContainerImpl
I've used a type argument of [T Reportable] where the Reportable is defined as follows
type Reportable interface {
ExportDataPointReport | ImportDataPointReport | MissingDataPointReport | SensorThresoldReport
}
Each of the type in the type constraint is structs that is to be embedded in the container.
type ReportContainerImpl[T Reportable] struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body T `json:"body"`
}
I use a discriminator ReportType to determine the concrete type when Unmarshal.
type ReportType string
const (
ReportTypeExportDataPointReport ReportType = "ExportDataPointReport"
ReportTypeImportDataPointReport ReportType = "ImportDataPointReport"
ReportTypeMissingDataPointReport ReportType = "MissingDataPointReport"
ReportTypeSensorThresoldReport ReportType = "SensorThresoldReport"
)
Since go does not support type assertion for struct (only interfaces) it is not possible to cast the type when Unmarshal. Also go does not support pointer to the "raw" generic type. Hence, I've created a interface that the ReportContainerImpl implements.
type ReportContainer interface {
GetLocationID() string
GetProvider() string
GetReportType() ReportType
GetBody() interface{}
}
The problem I then get is that I cannot do type constrains on the return type in any form or shape and am back at "freetext semantics" on the GetBody() function to allow for type assertion when Unmarshal is done.
container, err := UnmarshalReportContainer(data)
if rep, ok := container.GetBody().(ExportDataPointReport); ok {
// Use the ReportContainerImpl[ExportDataPointReport] here...
}
Maybe I'm getting this wrong? - but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Do you have a better suggestion how to solve this in a type (safer) way?
Cheers,
Mario :)
For completeness I add the UnmarshalReportContainer here
func UnmarshalReportContainer(data []byte) (ReportContainer, error) {
type Temp struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body *json.RawMessage `json:"body"`
}
var temp Temp
err := json.Unmarshal(data, &temp)
if err != nil {
return nil, err
}
switch temp.ReportType {
case ReportTypeExportDataPointReport:
var report ExportDataPointReport
err := json.Unmarshal(*temp.Body, &report)
return &ReportContainerImpl[ExportDataPointReport]{
LocationID: temp.LocationID,
Provider: temp.Provider,
ReportType: temp.ReportType,
Body: report,
}, err
// ...
}
}
but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Precisely.
The concrete types needed to instantiate some generic type or function like ReportContainerImpl or UnmarshalReportContainer must be known at compile time, when you write the code. JSON unmarshalling instead occurs at run-time, when you have the byte slice populated with the actual data.
To unmarshal dynamic JSON based on some discriminatory value, you still need a switch.
Do you have a better suggestion how to solve this in a type (safer) way?
Just forgo parametric polymorphism. It's not a good fit here. Keep the code you have now with json.RawMessage, unmarshal the dynamic data conditionally in the switch and return the concrete structs that implement ReportContainer interface.
As a general solution — if, and only if, you can overcome this chicken-and-egg problem and make type parameters known at compile time, you can write a minimal generic unmarshal function like this:
func unmarshalAny[T any](bytes []byte) (*T, error) {
out := new(T)
if err := json.Unmarshal(bytes, out); err != nil {
return nil, err
}
return out, nil
}
This is only meant to illustrate the principle. Note that json.Unmarshal already accepts any type, so if your generic function actually does nothing except new(T) and return, like in my example, it is no different than "inlining" the entire thing as if unmarshalAny didn't exist.
v, err := unmarshalAny[SomeType](src)
functionally equivalent as
out := &SomeType{}
err := json.Unmarshal(bytes, out)
If you plan to put more logic in unmarshalAny, its usage may be warranted. Your mileage may vary; in general, don't use type parameters when it's not actually necessary.
Related
I am working on a general JSON based message passing protocol in Go. What I would like to do is have a BaseMessage that has general information like Type, timestamp, etc. But at the same time I want to be able to define more specific message structures for certain types of data.
For example:
type Message struct {
Type string `json:type`
Timestamp string `json:timestamp`
}
type EventMessage struct {
Message
EventType string
EventCreator string
EventData interface{}
}
I have a set of handlers and to determine which handler should process the message I decode the JSON to the general Message type first to check the Type field. For this example I would get the handler associated with an "Event" message type.
I run into problems when I then want to assert the EventMessage type onto the structure.
The following code is very rough, but hopefully it displays my general idea of how I am trying to handle the messages.
type Handler func(msg Message) Message
handlers := make(map[string]Handler)
var msg Message
decoder.Decode(&msg)
handler := handlers[msg.Type]
handler(msg)
I have tried to use an interface{} but then the JSON decoder just creates a map which I then cant assert either type on. I have figured out workarounds that make it possible, but its very ugly, probably not efficient, and most likely error prone. I would like to keep things simple and straightforward so this code can be easily maintained.
Is there a method of handling generic JSON objects in Go so that the decoded JSON could be one of many struct formats?
I have also played with the idea of having more specific info in a Data interface{} in the main Message struct, but then I run into the same problem of not being able to assert any types onto the interface. There has to be a better way to handle JSON formats that I am just missing.
One way to handle this is to define a struct for the fixed part of the message with a json.RawMessage field to capture the variant part of the message. Decode the json.RawMessage to a type specific to the variant:
type Message struct {
Type string `json:"type"`
Timestamp string `json:"timestamp"`
Data json.RawMessage
}
type Event struct {
Type string `json:"type"`
Creator string `json:"creator"`
}
var m Message
if err := json.Unmarshal(data, &m); err != nil {
log.Fatal(err)
}
switch m.Type {
case "event":
var e Event
if err := json.Unmarshal([]byte(m.Data), &e); err != nil {
log.Fatal(err)
}
fmt.Println(m.Type, e.Type, e.Creator)
default:
log.Fatal("bad message type")
}
playground example
I have a JSON object That contains an implementation of an interface within it. I'm attempting to take that JSON and marshal it into a struct whilst creating the implementation of the interface.
I've managed to get it to implement the interface with a custom JSON unmarshal function however I'm struggling to piece together how to then marshal the rest of the fields
I've created an example in the Go playground
https://play.golang.org/p/ztF7H7etdjM
My JSON being passed into my application is
{
"address":"1FYuJ4MsVmpzPoFJ6svJMJfygn91Eubid9",
"nonce":13,
"network_id":"qadre.demo.balance",
"challenge":"f2b19e71876c087e681fc092ea3a34d5680bbfe772e40883563e1d5513bb593f",
"type":"verifying_key",
"verifying_key":{
"verifying_key":"3b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29",
"fqdn":"huski.service.key"
},
"signature":"a3bf8ee202a508d5a5632f50b140b70b7095d8836493dc7ac4159f6f3350280078b3a58b2162a240bc8c7485894554976a9c7b5d279d3f5bf49fec950f024e02",
"fqdn":"huski.service.SingleKeyProof"
}
I've attempted to do a json.Unmarshal and pass in a new struct for the remaining fields however it seems to put me in an infinite loop, my application hangs and then crashes
The best solution I've come up with so far is to marshal the JSON into a `map[string]interface{} and do each field separately, this feels very clunky though
var m map[string]interface{}
if err := json.Unmarshal(data, &m); err != nil {
return err
}
ad, ok := m["address"]
if ok {
s.Address = ad.(string)
}
fqdn, ok := m["fqdn"]
if ok {
s.FQDN = fqdn.(string)
}
n, ok := m["nonce"]
if ok {
s.Nonce = int64(n.(float64))
}
c, ok := m["challenge"]
if ok {
s.Challenge = []byte(c.(string))
}
network, ok := m["network_id"]
if ok {
s.NetworkID = network.(string)
}
sig, ok := m["signature"]
if ok {
s.Signature = []byte(sig.(string))
}
The reason your code gets into an infinite loop when you try to unmarshal the rest of the fields is because, I presume, the implementation of UnmarshalJSON after its done unmarshaling the verifying key, calls json.Unmarshal with the receiver, which in turn calls the UnmarshalJSON method on the receiver and so they invoke each other ad infinitum.
What you can do is to create a temporary type using the existing type as its definition, this will "keep the structure" but "drop the methods", then unmarshal the rest of the fields into an instance of the new type, and, after unmarshal is done, convert the instance to the original type and assign that to the receiver.
While this fixes the infinite loop, it also re-introduces the original problem of json.Unmarshal not being able to unmarshal into a non-empty interface type. To fix that you can embed the new type in another temporary struct that has a field with the same json tag as the problematic field which will cause it to be "overshadowed" while json.Unmarshal is doing its work.
type SingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey PublicKey `json:"verifying_key"`
Signature []byte `json:"signature"`
}
func (s *SingleKey) UnmarshalJSON(data []byte) error {
type _SingleKey SingleKey
var temp struct {
RawKey json.RawMessage `json:"verifying_key"`
_SingleKey
}
if err := json.Unmarshal(data, &temp); err != nil {
return err
}
*s = SingleKey(temp._SingleKey)
switch s.Type {
case "verifying_key":
s.VerifyingKey = &PublicKeyImpl{}
// other cases ...
}
return json.Unmarshal([]byte(temp.RawKey), s.VerifyingKey)
}
https://play.golang.org/p/L3gdQZF47uN
Looking at what you've done in your custom unmarshalling function, you seem to be passing in a map with the name of fields as index, and the reflect.Type you want to unmarshal said value into. That, to me, suggests that the keys might be different for different payloads, but that each key has a distinct type associated with it. You can perfectly handle data like this with a simple wrapper type:
type WrappedSingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey json.RawMessage `json:"verifying_key"`
OtherKey json.RawMessage `json:"other_key"`
Signature []byte `json:"signature"`
}
type SingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey *PublicKey `json:"verifying_key,omitempty"`
OtherType *OtherKey `json:"other_key,omitempty"`
Signature []byte `json:"signature"`
}
So I've changed the type of your VerifyingKey field to a json.RawMessage. That's basically telling json.Unmarshal to leave that as raw JSON input. For every custom/optional field, add a corresponding RawMessage field.
In the unwrapped type, I've changed VerifyingKey to a pointer and added the omitempty bit to the tag. That's just to accomodate mutliple types, and not have to worry about custom marshalling to avoid empty fields, like the included OtherType field I have. To get what you need, then:
func (s *SingleKey) UnmarshalJSON(data []byte) error {
w := WrappedSingleKey{} // create wrapped instance
if err := json.Unmarshal(data, &w); err != nil {
return err
}
switch w.Type {
case "verifying_key":
var pk PublicKey
if err := json.Unmarshal([]byte(w.VerifyingKey), &pk); err != nil {
return err
}
s.VerifyingKey = &pk // assign
case "other_key":
var ok OtherKey
if err := json.Unmarshal([]byte(w.OtherKey), &ok); err != nil {
return err
}
s.OtherKey = &ok
}
// copy over the fields that didn't require anything special
s.FQDN = w.FQDN
s.Address = w.Address
}
This is a fairly simple approach, does away with the reflection, tons of functions, and is quite commonly used. It's something that lends itself quite well to code generation, too. The individual assignment of the fields is a bit tedious, though. You might think that you can solve that by embedding the SingleKey type into the wrapper, but be careful: this will recursively call your custom unmarshaller function.
You could, for example, update all the fields in the WRapped type to be pointers, and have them point to fields on your actual type. That does away with the manual copying of fields... It's up to you, really.
Note
I didn't test this code, just wrote it as I went along. It's something I've used in the past, and I believe what I wrote here should work, but no guarantees (as in: you might need to debug it a bit)
I'm a beginner with Go, and I'm now writing a function which can call an API. The function receives a part of the url (/user, /account, etc) and the struct to convert the returned json to (the structs User or Account for example) as arguments.
So I now have this:
func (self *RestClient) request(action string, return_type interface{}) interface{} {
res, _ := goreq.Request{Uri:self.url + action}.Do()
var item return_type
res.Body.FromJsonTo(&item)
return item
}
And I try to call this function using (with Index being the struct for the return type):
self.request("/api/v1/public/index", Index)
But this doesn't work. I get the following errors:
return_type is not a type
type Index is not an expression
I guess I understand why this is. I think I have to somehow find out the type of return_type and then convert return_type to that type, before using it as the type for item. I have no idea how though.
About the second error I have no idea. I don't even understand what is meant by it.
Could anybody help me out? How can I make this work? Or should this be done in a completely different way all together? All tips are welcome!
A few hints based on this code:
Don't use self - use a meaningful identifier
Don't use interface{} to avoid dealing with the type system
Don't use reflection
Don't ignore errors returned (as from FromJsonTo or goreq.Request)
Don't use a library like goreq unless you are sure you need it (you don't) - you are pulling in lots of code you don't need and it is teaching you bad habits like attempting to use empty interface and reflection to solve simple problems.
Have a look at the definition of FromJsonTo - if you look through this library you'll see it isn't saving you much effort and is adding lots of complexity. Here is how you could do it without the library:
func (c *RestClient) Request(action string, resource interface{}) error {
res, err := http.Get(c.url + action)
if err != nil {
return err
}
defer res.Body.Close()
return json.NewDecoder(res.Body).Decode(resource)
}
Or use an interface and move the decoding to the resource (which could embed a default decoder):
type Decoder interface {
Decode(r io.Reader) error
}
// RequestDecode fetches a request and feeds it to the decoder
func (c *RestClient) RequestDecode(action string, resource Decoder) error {
res, err := http.Get(c.url + action)
if err != nil {
return err
}
defer res.Body.Close()
return resource.Decode(res.Body)
}
I'll first say that you should always check for errors from any function that possibly returns one.
The error you are seeing is because you are trying to declare a variable item as type return_type and that is the name of a function argument.
The other error is from Index being a type declaration and not a concrete value of the Index type.
I agree with Volker's comment but to put it in code, you could use something like this:
func (self *RestClient) request(action string, return_type interface{}) {
res, err := goreq.Request{Uri:self.url + action}.Do()
if err != nil {
// Do something with error here.
}
res.Body.FromJsonTo(return_type)
}
var index Index
rest_client.request("/some/path", &index)
This allows flexibility but could lead to strange cases if you forget to pass a pointer to the value return_type.
I am working on a general JSON based message passing protocol in Go. What I would like to do is have a BaseMessage that has general information like Type, timestamp, etc. But at the same time I want to be able to define more specific message structures for certain types of data.
For example:
type Message struct {
Type string `json:type`
Timestamp string `json:timestamp`
}
type EventMessage struct {
Message
EventType string
EventCreator string
EventData interface{}
}
I have a set of handlers and to determine which handler should process the message I decode the JSON to the general Message type first to check the Type field. For this example I would get the handler associated with an "Event" message type.
I run into problems when I then want to assert the EventMessage type onto the structure.
The following code is very rough, but hopefully it displays my general idea of how I am trying to handle the messages.
type Handler func(msg Message) Message
handlers := make(map[string]Handler)
var msg Message
decoder.Decode(&msg)
handler := handlers[msg.Type]
handler(msg)
I have tried to use an interface{} but then the JSON decoder just creates a map which I then cant assert either type on. I have figured out workarounds that make it possible, but its very ugly, probably not efficient, and most likely error prone. I would like to keep things simple and straightforward so this code can be easily maintained.
Is there a method of handling generic JSON objects in Go so that the decoded JSON could be one of many struct formats?
I have also played with the idea of having more specific info in a Data interface{} in the main Message struct, but then I run into the same problem of not being able to assert any types onto the interface. There has to be a better way to handle JSON formats that I am just missing.
One way to handle this is to define a struct for the fixed part of the message with a json.RawMessage field to capture the variant part of the message. Decode the json.RawMessage to a type specific to the variant:
type Message struct {
Type string `json:"type"`
Timestamp string `json:"timestamp"`
Data json.RawMessage
}
type Event struct {
Type string `json:"type"`
Creator string `json:"creator"`
}
var m Message
if err := json.Unmarshal(data, &m); err != nil {
log.Fatal(err)
}
switch m.Type {
case "event":
var e Event
if err := json.Unmarshal([]byte(m.Data), &e); err != nil {
log.Fatal(err)
}
fmt.Println(m.Type, e.Type, e.Creator)
default:
log.Fatal("bad message type")
}
playground example
I have a struct like this:
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
But even if the instance of MyStruct is entirely empty (meaning, all values are default), it's being serialized as:
"data":{}
I know that the encoding/json docs specify that "empty" fields are:
false, 0, any nil pointer or interface value, and any array,
slice, map, or string of length zero
but with no consideration for a struct with all empty/default values. All of its fields are also tagged with omitempty, but this has no effect.
How can I get the JSON package to not marshal my field that is an empty struct?
As the docs say, "any nil pointer." -- make the struct a pointer. Pointers have obvious "empty" values: nil.
Fix - define the type with a struct pointer field:
type Result struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
Then a value like this:
result := Result{}
Will marshal as:
{}
Explanation: Notice the *MyStruct in our type definition. JSON serialization doesn't care whether it is a pointer or not -- that's a runtime detail. So making struct fields into pointers only has implications for compiling and runtime).
Just note that if you do change the field type from MyStruct to *MyStruct, you will need pointers to struct values to populate it, like so:
Data: &MyStruct{ /* values */ }
As #chakrit mentioned in a comment, you can't get this to work by implementing json.Marshaler on MyStruct, and implementing a custom JSON marshalling function on every struct that uses it can be a lot more work. It really depends on your use case as to whether it's worth the extra work or whether you're prepared to live with empty structs in your JSON, but here's the pattern I use applied to Result:
type Result struct {
Data MyStruct
Status string
Reason string
}
func (r Result) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}{
Data: &r.Data,
Status: r.Status,
Reason: r.Reason,
})
}
func (r *Result) UnmarshalJSON(b []byte) error {
decoded := new(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
})
err := json.Unmarshal(b, decoded)
if err == nil {
r.Data = decoded.Data
r.Status = decoded.Status
r.Reason = decoded.Reason
}
return err
}
If you have huge structs with many fields this can become tedious, especially changing a struct's implementation later, but short of rewriting the whole json package to suit your needs (not a good idea), this is pretty much the only way I can think of getting this done while still keeping a non-pointer MyStruct in there.
Also, you don't have to use inline structs, you can create named ones. I use LiteIDE with code completion though, so I prefer inline to avoid clutter.
Data is an initialized struct, so it isn't considered empty because encoding/json only looks at the immediate value, not the fields inside the struct.
Unfortunately, returning nil from json.Marshaler doesn't currently work:
func (_ MyStruct) MarshalJSON() ([]byte, error) {
if empty {
return nil, nil // unexpected end of JSON input
}
// ...
}
You could give Result a marshaler as well, but it's not worth the effort.
The only option, as Matt suggests, is to make Data a pointer and set the value to nil.
There is an outstanding Golang proposal for this feature which has been active for over 4 years, so at this point, it is safe to assume that it will not make it into the standard library anytime soon. As #Matt pointed out, the traditional approach is to convert the structs to pointers-to-structs. If this approach is infeasible (or impractical), then an alternative is to use an alternate json encoder which does support omitting zero value structs.
I created a mirror of the Golang json library (clarketm/json) with added support for omitting zero value structs when the omitempty tag is applied. This library detects zeroness in a similar manner to the popular YAML encoder go-yaml by recursively checking the public struct fields.
e.g.
$ go get -u "github.com/clarketm/json"
import (
"fmt"
"github.com/clarketm/json" // drop-in replacement for `encoding/json`
)
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
j, _ := json.Marshal(&Result{
Status: "204",
Reason: "No Content",
})
fmt.Println(string(j))
// Note: `data` is omitted from the resultant json.
{
"status": "204"
"reason": "No Content"
}