UnmarshalJSON on struct containing interface list - json

I would like to UnmarshalJSON a struct containing an interface as follows:
type Filterer interface {
Filter(s string) error
}
type FieldFilter struct {
Key string
Val string
}
func (ff *FieldFilter) Filter(s string) error {
// Do something
}
type Test struct {
Name string
Filters []Filterer
}
My idea was to send a json like so:
{
"Name": "testing",
"Filters": [
{
"FieldFilter": {
"Key": "key",
"Val": "val"
}
}
]
}
However, when sending this json to the unmarshaler, the following exception returns: json: cannot unmarshal object into Go struct field Test.Filters of type Filterer
I understand the problem fully, but do not know how to approach this problem wisely. Looking for advice on an idiomatic way to solving this problem in go.

Following my own question, I researched how one could implement UnmarshalJSON for interface lists. Ultimately this led me to publish a blog post on how to do this properly. Basically there are 2 main solutions:
Parse the required JSON string into a map[string]*json.RawMessage and work your way from there.
Make an alias for the interface list and implement UnmarshalJSON for that alias. However, you'll still need to work with map[string]*json.RawMessage and some manual work. Nothing comes without a price!
I highly suggest taking the seconds approach. While these two solutions may result in the same amount of code lines, taking advantage of type aliasing and being less dependent on json.RawMessage types will make a more easy to manage code, especially when it is required to support multiple interfaces on the UnmarshalJSON implementation
To directly answer the question, start with making a type alias for the interface list:
type Filterers []Filterer
Now continue with implementing the decoding of the JSON:
func (f *Filterers) UnmarshalJSON(b []byte) error {
var FilterFields map[string]*json.RawMessage
if err := json.Unmarshal(b, &FilterFields); err != nil {
return err
}
for LFKey, LFValue := range FilterFields {
if LFKey == "FieldFilter" {
var MyFieldFilters []*json.RawMessage
if err := json.Unmarshal(*LFValue, &MyFieldFilters); err != nil {
return err
}
for _, MyFieldFilter := range MyFieldFilters {
var filter FieldFilter
if err := json.Unmarshal(*MyFieldFilter, &filter); err != nil {
return err
}
*f = append(*f, &filter)
}
}
}
return nil
}
A detailed explanation (with some examples and a full working code snippets) of the second approach is available on my own blog

There is no way for Unmarshal to know what type it should use. The only case where it can just "make something up" is if it's asked to unmarshal into an interface{}, in which case it will use the rules in the documentation. Since none of those types can be put into a []Filterer, it cannot unmarshal that field. If you want to unmarshal into a struct type, you must specify the field to be of that type.
You can always unmarshal into an intermediate struct or map type, and then do your own conversion from that into whatever types you want.

Related

Unmarshall JSON into a Generic Struct [duplicate]

I'm new to golang generics and have the following setup.
I've gathered loads of different kinds of reports.
Each report has enclosing fields
So I wrapped it in a ReportContainerImpl
I've used a type argument of [T Reportable] where the Reportable is defined as follows
type Reportable interface {
ExportDataPointReport | ImportDataPointReport | MissingDataPointReport | SensorThresoldReport
}
Each of the type in the type constraint is structs that is to be embedded in the container.
type ReportContainerImpl[T Reportable] struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body T `json:"body"`
}
I use a discriminator ReportType to determine the concrete type when Unmarshal.
type ReportType string
const (
ReportTypeExportDataPointReport ReportType = "ExportDataPointReport"
ReportTypeImportDataPointReport ReportType = "ImportDataPointReport"
ReportTypeMissingDataPointReport ReportType = "MissingDataPointReport"
ReportTypeSensorThresoldReport ReportType = "SensorThresoldReport"
)
Since go does not support type assertion for struct (only interfaces) it is not possible to cast the type when Unmarshal. Also go does not support pointer to the "raw" generic type. Hence, I've created a interface that the ReportContainerImpl implements.
type ReportContainer interface {
GetLocationID() string
GetProvider() string
GetReportType() ReportType
GetBody() interface{}
}
The problem I then get is that I cannot do type constrains on the return type in any form or shape and am back at "freetext semantics" on the GetBody() function to allow for type assertion when Unmarshal is done.
container, err := UnmarshalReportContainer(data)
if rep, ok := container.GetBody().(ExportDataPointReport); ok {
// Use the ReportContainerImpl[ExportDataPointReport] here...
}
Maybe I'm getting this wrong? - but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Do you have a better suggestion how to solve this in a type (safer) way?
Cheers,
Mario :)
For completeness I add the UnmarshalReportContainer here
func UnmarshalReportContainer(data []byte) (ReportContainer, error) {
type Temp struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body *json.RawMessage `json:"body"`
}
var temp Temp
err := json.Unmarshal(data, &temp)
if err != nil {
return nil, err
}
switch temp.ReportType {
case ReportTypeExportDataPointReport:
var report ExportDataPointReport
err := json.Unmarshal(*temp.Body, &report)
return &ReportContainerImpl[ExportDataPointReport]{
LocationID: temp.LocationID,
Provider: temp.Provider,
ReportType: temp.ReportType,
Body: report,
}, err
// ...
}
}
but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Precisely.
The concrete types needed to instantiate some generic type or function like ReportContainerImpl or UnmarshalReportContainer must be known at compile time, when you write the code. JSON unmarshalling instead occurs at run-time, when you have the byte slice populated with the actual data.
To unmarshal dynamic JSON based on some discriminatory value, you still need a switch.
Do you have a better suggestion how to solve this in a type (safer) way?
Just forgo parametric polymorphism. It's not a good fit here. Keep the code you have now with json.RawMessage, unmarshal the dynamic data conditionally in the switch and return the concrete structs that implement ReportContainer interface.
As a general solution — if, and only if, you can overcome this chicken-and-egg problem and make type parameters known at compile time, you can write a minimal generic unmarshal function like this:
func unmarshalAny[T any](bytes []byte) (*T, error) {
out := new(T)
if err := json.Unmarshal(bytes, out); err != nil {
return nil, err
}
return out, nil
}
This is only meant to illustrate the principle. Note that json.Unmarshal already accepts any type, so if your generic function actually does nothing except new(T) and return, like in my example, it is no different than "inlining" the entire thing as if unmarshalAny didn't exist.
v, err := unmarshalAny[SomeType](src)
functionally equivalent as
out := &SomeType{}
err := json.Unmarshal(bytes, out)
If you plan to put more logic in unmarshalAny, its usage may be warranted. Your mileage may vary; in general, don't use type parameters when it's not actually necessary.

Unmarshal remaining JSON after performing custom unmarshalling

I have a JSON object That contains an implementation of an interface within it. I'm attempting to take that JSON and marshal it into a struct whilst creating the implementation of the interface.
I've managed to get it to implement the interface with a custom JSON unmarshal function however I'm struggling to piece together how to then marshal the rest of the fields
I've created an example in the Go playground
https://play.golang.org/p/ztF7H7etdjM
My JSON being passed into my application is
{
"address":"1FYuJ4MsVmpzPoFJ6svJMJfygn91Eubid9",
"nonce":13,
"network_id":"qadre.demo.balance",
"challenge":"f2b19e71876c087e681fc092ea3a34d5680bbfe772e40883563e1d5513bb593f",
"type":"verifying_key",
"verifying_key":{
"verifying_key":"3b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29",
"fqdn":"huski.service.key"
},
"signature":"a3bf8ee202a508d5a5632f50b140b70b7095d8836493dc7ac4159f6f3350280078b3a58b2162a240bc8c7485894554976a9c7b5d279d3f5bf49fec950f024e02",
"fqdn":"huski.service.SingleKeyProof"
}
I've attempted to do a json.Unmarshal and pass in a new struct for the remaining fields however it seems to put me in an infinite loop, my application hangs and then crashes
The best solution I've come up with so far is to marshal the JSON into a `map[string]interface{} and do each field separately, this feels very clunky though
var m map[string]interface{}
if err := json.Unmarshal(data, &m); err != nil {
return err
}
ad, ok := m["address"]
if ok {
s.Address = ad.(string)
}
fqdn, ok := m["fqdn"]
if ok {
s.FQDN = fqdn.(string)
}
n, ok := m["nonce"]
if ok {
s.Nonce = int64(n.(float64))
}
c, ok := m["challenge"]
if ok {
s.Challenge = []byte(c.(string))
}
network, ok := m["network_id"]
if ok {
s.NetworkID = network.(string)
}
sig, ok := m["signature"]
if ok {
s.Signature = []byte(sig.(string))
}
The reason your code gets into an infinite loop when you try to unmarshal the rest of the fields is because, I presume, the implementation of UnmarshalJSON after its done unmarshaling the verifying key, calls json.Unmarshal with the receiver, which in turn calls the UnmarshalJSON method on the receiver and so they invoke each other ad infinitum.
What you can do is to create a temporary type using the existing type as its definition, this will "keep the structure" but "drop the methods", then unmarshal the rest of the fields into an instance of the new type, and, after unmarshal is done, convert the instance to the original type and assign that to the receiver.
While this fixes the infinite loop, it also re-introduces the original problem of json.Unmarshal not being able to unmarshal into a non-empty interface type. To fix that you can embed the new type in another temporary struct that has a field with the same json tag as the problematic field which will cause it to be "overshadowed" while json.Unmarshal is doing its work.
type SingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey PublicKey `json:"verifying_key"`
Signature []byte `json:"signature"`
}
func (s *SingleKey) UnmarshalJSON(data []byte) error {
type _SingleKey SingleKey
var temp struct {
RawKey json.RawMessage `json:"verifying_key"`
_SingleKey
}
if err := json.Unmarshal(data, &temp); err != nil {
return err
}
*s = SingleKey(temp._SingleKey)
switch s.Type {
case "verifying_key":
s.VerifyingKey = &PublicKeyImpl{}
// other cases ...
}
return json.Unmarshal([]byte(temp.RawKey), s.VerifyingKey)
}
https://play.golang.org/p/L3gdQZF47uN
Looking at what you've done in your custom unmarshalling function, you seem to be passing in a map with the name of fields as index, and the reflect.Type you want to unmarshal said value into. That, to me, suggests that the keys might be different for different payloads, but that each key has a distinct type associated with it. You can perfectly handle data like this with a simple wrapper type:
type WrappedSingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey json.RawMessage `json:"verifying_key"`
OtherKey json.RawMessage `json:"other_key"`
Signature []byte `json:"signature"`
}
type SingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey *PublicKey `json:"verifying_key,omitempty"`
OtherType *OtherKey `json:"other_key,omitempty"`
Signature []byte `json:"signature"`
}
So I've changed the type of your VerifyingKey field to a json.RawMessage. That's basically telling json.Unmarshal to leave that as raw JSON input. For every custom/optional field, add a corresponding RawMessage field.
In the unwrapped type, I've changed VerifyingKey to a pointer and added the omitempty bit to the tag. That's just to accomodate mutliple types, and not have to worry about custom marshalling to avoid empty fields, like the included OtherType field I have. To get what you need, then:
func (s *SingleKey) UnmarshalJSON(data []byte) error {
w := WrappedSingleKey{} // create wrapped instance
if err := json.Unmarshal(data, &w); err != nil {
return err
}
switch w.Type {
case "verifying_key":
var pk PublicKey
if err := json.Unmarshal([]byte(w.VerifyingKey), &pk); err != nil {
return err
}
s.VerifyingKey = &pk // assign
case "other_key":
var ok OtherKey
if err := json.Unmarshal([]byte(w.OtherKey), &ok); err != nil {
return err
}
s.OtherKey = &ok
}
// copy over the fields that didn't require anything special
s.FQDN = w.FQDN
s.Address = w.Address
}
This is a fairly simple approach, does away with the reflection, tons of functions, and is quite commonly used. It's something that lends itself quite well to code generation, too. The individual assignment of the fields is a bit tedious, though. You might think that you can solve that by embedding the SingleKey type into the wrapper, but be careful: this will recursively call your custom unmarshaller function.
You could, for example, update all the fields in the WRapped type to be pointers, and have them point to fields on your actual type. That does away with the manual copying of fields... It's up to you, really.
Note
I didn't test this code, just wrote it as I went along. It's something I've used in the past, and I believe what I wrote here should work, but no guarantees (as in: you might need to debug it a bit)

Unmarshaling nested custom same-type JSON in Go

Given the following JSON
{
"some": "value"
"nested": {
"some": "diffvalue",
"nested": {
"some": "innervalue"
}
}
}
which roughly translates to this struct:
type Envelope struct {
some string `json:"some"`
nested InnerEnvelope `json:"nested"`
}
where InnerEnvelope is: type InnerEnvelope map[string]interface{}
Running a simple json.Unmarshal([]byte value, &target) does not help here, because of the recursive type nature of the original JSON.
I do not know up front how deeply and under which keys the inner maps will exist, so I cannot declare the types upfront.
The idea is, that using map[string]interface{} as the type is not good enough, since I need the values in the InnerEnvelope to be somehow transformed & typed. Details are not important, but image, I need to cast every value inside the NestedEnvelope of a bool type as a string saying "true" or "false" as opposed of having an actual bool type.
I turned to UnmarshalJSON interface to solve this problem. I can easily do it at the top level like so:
func (m *Envelope) UnmarshalJSON(b []byte) error {
var stuff noBoolMap
if err := json.Unmarshal(b, &stuff); err != nil {
return err
}
for key, value := range stuff {
switch value.(type) {
case bool:
stuff[key] = strconv.FormatBool(value.(bool))
}
}
return nil
}
But since the inner json.Unmarshal will already have inner maps parsed as map[string]interface{}, I would need to yet-again to traverse the inner maps, cast them to appropriate type and perform my value transformations.
So my question is: In this case, what is the way this would be approached in Go, and preferably do it in a single-pass?
The expected result of the JSON example above would be:
Envelope {
some: string
nested: InnerEnvelope {
some: string {
nested: InnerEnvelope {
some: string
}
}
}
Given your json, you can do this:
type Envelope struct {
Some string `json:"some"`
Nested json.RawMessage `json:"nested"`
}
json.RawMessage is a rather hidden gem, and more people seem to go for the map[string]interface{}.
Using json.RawMessage will result in the nested json to be represented by this RawMessage, which you then can process again as a normal json (unmarshal it into Envelope).
This is more elegant than the map[string]interface{} approach.

Can I modify json.RawMessage?

This is a follow up to JSON sometimes array sometimes object
In the original question, I asked how to deal with: "I am consuming a json API that might return a string for a variable or might return an array for a variable"
I have a solution but I was wondering, is there a way to modify json.RawMessage?
Rather then if/then looking at the RawMessage for a [ or { char to determine if the object is an array or an string, what if I always took a RawMessage variable string and turned it into an array?
This way, I don't have to code all of the accessors for BOTH strings AND arrays. I could simply deal with arrays.
So my question is: Is there a way to modify the json.RawMessage?
eg:
Turn this:
{
"net": {
"comment": {
"line":
{
"$": "All abuse issues will only be responded to by the Abuse",
"#number": "0"
}
}
}
Into This:
{
"net": {
"comment": {
"line": [
{
"$": "All abuse issues will only be responded to by the Abuse",
"#number": "0"
}
]
}
}
So, that way, when I unmarshal into my struct, there is only 1 type of comment.line, Just line[] vs line[] AND line.
Thanks in advance.
I am a golang neophyte and I'm just getting my head wrapped around the difficulties of unmarshaling into an strongly typed language.
Yes, you can edit json.RawMessage type as it is simply an alias for []byte.
That said, you don't need to keep raw type, just make your own implementation of the array type and in your custom Unmarshal function, make scalars an array.
Here's an example (on Play).
All we do here is see if the bytes for MagicArray starts with '[', if so, just unmarshal as normal. Otherwise, Unmarshal and append to slice.
you will have to implement custom array for each type you want to work like this, but that's still probably better than trying to safely manipulate the json binary to try coerce the scalars into arrays.
Another side benefit to this approach is you can it with the streaming json decoder, as in json.NewDecoder(reader).Decode(&obj)
package main
import "encoding/json"
import "log"
type MagicArray []interface{}
func (ma *MagicArray) UnmarshalJSON(b []byte) error {
if b[0] == '[' {
return json.Unmarshal(b, (*[]interface{})(ma))
} else {
var obj interface{}
if err := json.Unmarshal(b, &obj); err != nil {
return err
}
*ma = append(*ma, obj)
}
return nil
}
func main() {
myStruct := struct {
A MagicArray
B MagicArray
}{}
err := json.Unmarshal(jsonToDecode, &myStruct)
if err != nil {
log.Println("Fail:", err)
} else {
log.Println(myStruct)
}
}
var jsonToDecode = []byte(`
{
"A": "I am not an array",
"B":["I am an array"]
}
`)
I think that David has a good (better) answer, but to answer your question directly: yes, you can modify a json.RawMessage if you're careful. it's declared as type json.RawMessage []byte, meaning it's just another name for []byte under the hood. You can cast it to []byte or string, modify it, and cast it back.
Doing string options on serialized data isn't the kind of thing you should do without thinking about the consequences, but in the case of wrapping [ and ] around a JSON object, it's not too hard to prove that it should be safe. If msg is a json.RawMessage representing an object, then
json.RawMessage("[" + string(msg) + "]")
is what I would consider a readable approach to making a RawMessage representing an array containing that object :)

Unmarshalling JSON to interface variable while matching the underlying type in Go

I'm writing a wrapper to map with some additional functionality I need. Some of the most important things is the ability to marshal and unmarshal the data while retaining genericity. I managed to write an marshaller using the encoding/gob encoder, but since it would be nice if the marshalled data was human readable, I decided to code another implementation with JSON.
I got gob to encode from and decode to generic interface variables neatly by passing it a implementation object instance with Register(). (This resource helped me with the details! http://www.funcmain.com/gob_encoding_an_interface)
However, JSON doesn't have Register(). Let's say we have a value of type
type ConcreteImplementation struct {
FieldA string
FieldB string
}
func (c ConcreteImplementation) Key() string {
return c.FieldA // ConcreteImplementation implements genericValue
}
in a variable of type
type genericValue interface {
Key() string
}
When I marshal that, it outputs JSON like this:
{"FieldA":"foo","FieldB":"bar"}
And when I try to unmarshal that again into a variable of type genericValue, it says:
panic: interface conversion: map[string]interface {} is not genericValue: missing method Key EDIT: Oops, actually it says this!
Error with decoding! json: cannot unmarshal object into Go value of genericValue
Quite obviously, it tries to marshal the data like it says here: http://blog.golang.org/json-and-go (See 'Generic JSON with interface{}')
How can I get it to try to fit the data to an specific implementation, like gob decoder would try if the implementation is Register()ed? Register() was godsend, it allowed to marshal and unmarshal generically like it was nothing. How do I get JSON to do the same thing?
What if your types implemented the Unmarshaler?
Here is a small demo.
Or the same code here:
type ConcreteImplementation struct {
FieldA string
FieldB string
}
func (c ConcreteImplementation) Key() string {
return c.FieldA // ConcreteImplementation implements genericValue
}
// implementing json.Unmarshaler
func (c *ConcreteImplementation) UnmarshalJSON(j []byte) error {
m := make(map[string]string)
err := json.Unmarshal(j, &m)
if err != nil {
return err
}
if v, ok := m["FieldA"]; ok {
c.FieldA = v
}
if v, ok := m["FieldB"]; ok {
c.FieldB = v
}
return nil
}
type genericValue interface {
Key() string
json.Unmarshaler
}
func decode(jsonStr []byte, v genericValue) error {
return json.Unmarshal(jsonStr, v)
}
With this you can pass a genericValue to json.Unmarshal.
Allright, got it to work finally. This question provided the answer. Why does json.Unmarshal return a map instead of the expected struct?
"You've passed to json a pointer to an abstract interface. You should simply pass a pointer to Ping as an abstract interface" - This applied to my situation too. (For some reason a pointer TO an abstract interface was enough for gob package. It seems that I have to study Go interfaces and reflection some more to understand why...)
I won't still mark this as solved question, if someone has a better answer.