Unmarshalling JSON string into a constant - json

In my Go code I have the following situation:
type Operator int
const (
UNKNOWN Operator = iota
EQUALS
CONTAINS
BETWEEN
DISTANCE
)
type Filter struct {
Field string `json:"field"`
Operator Operator `json:"operator"`
Values []string `json:"values"`
}
My expected JSON would look like the following:
{
"operator": "EQUALS",
"field": "name",
"values": [ "John", "Doe" ]
}
Can I create a mapping such that json.Unmarshal will set the right operator constant in the Filter struct?
I am aware of the Unmarshaler interface but I don't think this can really be used on a constant value..
I would really like to keep the constants in my go code since this nicely enforces type checking and consistency when I pass them around.

The encoding/json Unmarshaler interface will work with the Operator type, but it must have a pointer as its receiver:
func (o *Operator) UnmarshalJSON(b []byte) error {
str := strings.Trim(string(b), `"`)
switch {
case str == "CONTAINS":
*o = CONTAINS
default:
*o = UNKNOWN
// or return an error...
}
return nil
}
The JSON decoder will take the address of the Operator field from the Filter struct and invoke the UnmarshalJSON method on it.
Note that you could implement the more generic encoding/TextUnmarshaler instead, by changing UnmarshalJSON to UnmarshalText above.
Here is a playground example: http://play.golang.org/p/szcnC6L86u
Arguably it might be simpler to use a string base type for Operator instead: http://play.golang.org/p/FCCg1NOeYw

You can wrap your object in an Unmarshaler.
ex:
package main
import (
"encoding/json"
"fmt"
)
type Operator int
const (
UNKNOWN Operator = iota
EQUALS
CONTAINS
BETWEEN
DISTANCE
)
type Filter struct {
Field string `json:"field"`
RawOperator string `json:"operator"`
Operator Operator `json:"-"`
Values []string `json:"values"`
}
type FilterUnmarshaler struct {
Filter
}
func (f *FilterUnmarshaler) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &f.Filter); err != nil {
return err
}
switch f.RawOperator {
case "UNKOWN":
f.Operator = UNKNOWN
case "EQUALS":
f.Operator = EQUALS
case "CONTAINS":
f.Operator = CONTAINS
case "BETWEEN":
f.Operator = BETWEEN
case "DISTANCE":
f.Operator = DISTANCE
default:
return fmt.Errorf("Unkown operator %s", f.RawOperator)
}
return nil
}
func main() {
rawJson := []byte(`
{
"operator": "BETWEEN",
"field": "name",
"values": [ "John", "Doe" ]
}
`)
val := &FilterUnmarshaler{}
if err := json.Unmarshal(rawJson, val); err != nil {
panic(err)
}
fmt.Printf("%#v\n", val)
}
Without using a wrapper, I didn't find how not to fall in an infinite recursion.
This example shows how to unmarshal but you most likely want to apply it to Marshal as well in order to have a correct json with Operator strings from your object with integer operator.
You might also want to create a map instead of const. Or both, create a map and manually populate it with your const and their string equivalent.

Related

JSON unmarshal integer field into a string

I am struggling with deserializing a integer into a string struct field.
The struct field is a string and is expected to be assignable from users of my library. That's why I want it to be a string, since for the purpose of writing it to the database I actually don't care about the value inside.
The users can supply text, but some just assign integers.
Consider this struct:
type Test struct {
Foo string
}
Sometimes I end up with a JSON value that is valid but won't deserialize into the struct due to the Foo field being a integer instead of a string:
{ "foo": "1" } // works
{ "foo": 1 } // doesn't
json.Unmarshal will blow up with the following error:
json: cannot unmarshal number into Go struct field test.Foo of type string
See the reproduction: https://play.golang.org/p/4Qau3umaVm
Now in every other JSON library (in other languages) I have worked in so far, if the target field is a string and you get a integer the deserializer will usually just wrap the int in a string and be done with it. Can this be achieved in Go?
Since I can't really control how the data comes in I need to make json.Unmarshal unsensitive to this - the other solution would be to define Foo as interface{} which needlessly complicates my code with type assertions etc..
Any ideas on how to do this? I basically need the inverse of json:",string"
To handle big structs you can use embedding.
Updated to not discard possibly previously set field values.
func (t *T) UnmarshalJSON(d []byte) error {
type T2 T // create new type with same structure as T but without its method set!
x := struct{
T2 // embed
Foo json.Number `json:"foo"`
}{T2: T2(*t)} // don't forget this, if you do and 't' already has some fields set you would lose them
if err := json.Unmarshal(d, &x); err != nil {
return err
}
*t = T(x.T2)
t.Foo = x.Foo.String()
return nil
}
https://play.golang.org/p/BytXCeHMvt
You can customize how the data structure is Unmarshaler by implementing the json.Unmarshaler interface.
The simplest way to handle unknown types is to unnmarshal the JSON into an intermediate structure, and handle the type assertions and validation during deserialization:
type test struct {
Foo string `json:"foo"`
}
func (t *test) UnmarshalJSON(d []byte) error {
tmp := struct {
Foo interface{} `json:"foo"`
}{}
if err := json.Unmarshal(d, &tmp); err != nil {
return err
}
switch v := tmp.Foo.(type) {
case float64:
t.Foo = strconv.Itoa(int(v))
case string:
t.Foo = v
default:
return fmt.Errorf("invalid value for Foo: %v", v)
}
return nil
}
https://play.golang.org/p/t0eI4wCxdB

Is there a way to have json.Unmarshal() select struct type based on "type" property?

I have some JSON of the form:
[{
"type": "car",
"color": "red",
"hp": 85,
"doors": 4
}, {
"type": "plane",
"color": "blue",
"engines": 3
}]
I have types car and plane that satisfy a vehicle interface; I'd like to be able to write:
var v []vehicle
e := json.Unmarshal(myJSON, &v)
... and have JSON fill my slice of vehicles with a car and a plane; instead (and unsurprisingly) I just get "cannot unmarshal object into Go value of type main.vehicle".
For reference, here are suitable definitions of the types involved:
type vehicle interface {
vehicle()
}
type car struct {
Type string
Color string
HP int
Doors int
}
func (car) vehicle() { return }
type plane struct {
Type string
Color string
Engines int
}
func (plane) vehicle() { return }
var _ vehicle = (*car)(nil)
var _ vehicle = (*plane)(nil)
(Note that I'm actually totally uninterested in the t field on car and plane - it could be omitted because this information will, if someone successfully answers this question, be implicit in the dynamic type of the objects in v.)
Is there a way to have the JSON umarhsaller choose which type to use based on some part of the contents (in this case, the type field) of the data being decoded?
(Note that this is not a duplicate of Unmarshal JSON with unknown fields because I want each item in the slice to have a different dynamic type, and from the value of the 'type' property I know exactly what fields to expect—I just don't know how to tell json.Unmarshal how to map 'type' property values onto Go types.)
Taking the answers from the similar question: Unmarshal JSON with unknown fields, we can construct a few ways to unamrshal this JSON object in a []vehicle data structure.
The "Unmarshal with Manual Handling" version can be done by using a generic []map[string]interface{} data structure, then building the correct vehicles from the slice of maps. For brevity, this example does leave out the error checking for missing or incorrectly typed fields which the json package would have done.
https://play.golang.org/p/fAY9JwVp-4
func NewVehicle(m map[string]interface{}) vehicle {
switch m["type"].(string) {
case "car":
return NewCar(m)
case "plane":
return NewPlane(m)
}
return nil
}
func NewCar(m map[string]interface{}) *car {
return &car{
Type: m["type"].(string),
Color: m["color"].(string),
HP: int(m["hp"].(float64)),
Doors: int(m["doors"].(float64)),
}
}
func NewPlane(m map[string]interface{}) *plane {
return &plane{
Type: m["type"].(string),
Color: m["color"].(string),
Engines: int(m["engines"].(float64)),
}
}
func main() {
var vehicles []vehicle
objs := []map[string]interface{}{}
err := json.Unmarshal(js, &objs)
if err != nil {
log.Fatal(err)
}
for _, obj := range objs {
vehicles = append(vehicles, NewVehicle(obj))
}
fmt.Printf("%#v\n", vehicles)
}
We could leverage the json package again to take care of the unmarshaling and type checking of the individual structs by unmarshaling a second time directly into the correct type. This could all be wrapped up into a json.Unmarshaler implementation by defining an UnmarshalJSON method on the []vehicle type to first split up the JSON objects into raw messages.
https://play.golang.org/p/zQyL0JeB3b
type Vehicles []vehicle
func (v *Vehicles) UnmarshalJSON(data []byte) error {
// this just splits up the JSON array into the raw JSON for each object
var raw []json.RawMessage
err := json.Unmarshal(data, &raw)
if err != nil {
return err
}
for _, r := range raw {
// unamrshal into a map to check the "type" field
var obj map[string]interface{}
err := json.Unmarshal(r, &obj)
if err != nil {
return err
}
vehicleType := ""
if t, ok := obj["type"].(string); ok {
vehicleType = t
}
// unmarshal again into the correct type
var actual vehicle
switch vehicleType {
case "car":
actual = &car{}
case "plane":
actual = &plane{}
}
err = json.Unmarshal(r, actual)
if err != nil {
return err
}
*v = append(*v, actual)
}
return nil
}
JSON decoding and encoding in Go is actually surprisingly well at recognizing fields inside embedded structs. E.g. decoding or encoding the following structure works when there is no overlapping fields between type A and type B:
type T struct{
Type string `json:"type"`
*A
*B
}
type A struct{
Baz int `json:"baz"`
}
type B struct{
Bar int `json:"bar"`
}
Be aware that if both "baz" and "bar" are set in the JSON for the example above, both the T.A and T.B properties will be set.
If there is overlapping fields between A and B, or just to be able to better discard invalid combinations of fields and type, you need to implement the json.Unmarshaler interface. To not have to first decode fields into a map, you can extend the trick of using embedded structs.
type TypeSwitch struct {
Type string `json:"type"`
}
type T struct {
TypeSwitch
*A
*B
}
func (t *T) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &t.TypeSwitch); err != nil {
return err
}
switch t.Type {
case "a":
t.A = &A{}
return json.Unmarshal(data, t.A)
case "b":
t.B = &B{}
return json.Unmarshal(data, t.B)
default:
return fmt.Errorf("unrecognized type value %q", t.Type)
}
}
type A struct {
Foo string `json:"bar"`
Baz int `json:"baz"`
}
type B struct {
Foo string `json:"foo"`
Bar int `json:"bar"`
}
For marshaling back, json.Marshaler must also be implemented if there is overlapping fields.
Full example: https://play.golang.org/p/UHAdxlVdFQQ
The two passes approach works fine, but there is also the option of the mapstructure package, that was created to do exactly this.
I was facing the same problem.
I'm using the lib github.com/mitchellh/mapstructure together the encoding/json.
I first, unmarshal the json to a map, and use mapstructure to convert the map to my struct, e.g.:
type (
Foo struct {
Foo string `json:"foo"`
}
Bar struct {
Bar string `json:"bar"`
}
)
func Load(jsonStr string, makeInstance func(typ string) any) (any, error) {
// json to map
m := make(map[string]any)
e := json.Unmarshal([]byte(jsonStr), &m)
if e != nil {
return nil, e
}
data := makeInstance(m["type"].(string))
// decoder to copy map values to my struct using json tags
cfg := &mapstructure.DecoderConfig{
Metadata: nil,
Result: &data,
TagName: "json",
Squash: true,
}
decoder, e := mapstructure.NewDecoder(cfg)
if e != nil {
return nil, e
}
// copy map to struct
e = decoder.Decode(m)
return data, e
}
Using:
f, _ := Load(`{"type": "Foo", "foo": "bar"}`, func(typ string) any {
switch typ {
case "Foo":
return &Foo{}
}
return nil
})
If the property is a string you can use .(string) for casting the property because the origin is an interface.
You can use it the next way:
v["type"].(string)

JSON field set to null vs field not there

Is there a way, in golang, to see if I can differentiate between a json field being set to null vs a json field not being there when unmarshalled into a struct? Because both set the value in the struct to be nil, but I need to know if the field was there to begin with and to see if someone set it to null.
{
"somefield1":"somevalue1",
"somefield2":null
}
VS
{
"somefield1":"somevalue1",
}
Both jsons will be nil when unmarshalled into a struct.
Any useful resources will be very appreciated!
Use json.RawMessage to "delay" the unmarshaling process to determine the raw byte before deciding to do something:
var data = []byte(`{
"somefield1":"somevalue1",
"somefield2": null
}`)
type Data struct {
SomeField1 string
SomeField2 json.RawMessage
}
func main() {
d := &Data{}
_ = json.Unmarshal(data, &d)
fmt.Println(d.SomeField1)
if len(d.SomeField2) > 0 {
if string(d.SomeField2) == "null" {
fmt.Println("somefield2 is there but null")
} else {
fmt.Println("somefield2 is there and not null")
// Do something with the data
}
} else {
fmt.Println("somefield2 doesn't exist")
}
}
See the playground https://play.golang.org/p/Wganpf4sbO
If you're on Go 1.18+ you can use a simple generic struct to know when a JSON value is undefined or null:
type Optional[T any] struct {
Defined bool
Value *T
}
// UnmarshalJSON is implemented by deferring to the wrapped type (T).
// It will be called only if the value is defined in the JSON payload.
func (o *Optional[T]) UnmarshalJSON(data []byte) error {
o.Defined = true
return json.Unmarshal(data, &o.Value)
}
That's all, you can then just use this type in your structs:
type Payload struct {
Field1 Optional[string] `json:"field1"`
Field2 Optional[bool] `json:"field2"`
Field3 Optional[int32] `json:"field3"`
}
And you'll be able to use the Defined field to know if the field was null or undefined.
Check this playground link for a full example: https://go.dev/play/p/JZfZyVVUABz
Original answer (pre-generics)
Another way to do this, with a custom type:
// OptionalString is a struct that represents a JSON string that can be
// undefined (Defined == false), null (Value == nil && Defined == true) or
// defined with a string value
type OptionalString struct {
Defined bool
Value *string
}
// UnmarshalJSON implements the json.Unmarshaler interface.
// When called, it means that the value is defined in the JSON payload.
func (os *OptionalString) UnmarshalJSON(data []byte) error {
// UnmarshalJSON is called only if the key is present
os.Defined = true
return json.Unmarshal(data, &os.Value)
}
// Payload represents the JSON payload that you want to represent.
type Payload struct {
SomeField1 string `json:"somefield1"`
SomeField2 OptionalString `json:"somefield2"`
}
You can then just use the regular json.Unmarshal function to get your values, for example:
var p Payload
_ = json.Unmarshal([]byte(`{
"somefield1":"somevalue1",
"somefield2":null
}`), &p)
fmt.Printf("Should be defined == true and value == nil: \n%+v\n\n", p)
p = Payload{}
_ = json.Unmarshal([]byte(`{"somefield1":"somevalue1"}`), &p)
fmt.Printf("Should be defined == false \n%+v\n\n", p)
p = Payload{}
_ = json.Unmarshal([]byte(`{
"somefield1":"somevalue1",
"somefield2":"somevalue2"
}`), &p)
fmt.Printf("Parsed should be defined == true and value != nil \n%+v\n", p)
if p.SomeField2.Value != nil {
fmt.Printf("SomeField2's value is %s", *p.SomeField2.Value)
}
Should give you this output:
Should be defined == true and value == nil:
{SomeField1:somevalue1 SomeField2:{Defined:true Value:<nil>}}
Should be defined == false
{SomeField1:somevalue1 SomeField2:{Defined:false Value:<nil>}}
Parsed should be defined == true and value != nil
{SomeField1:somevalue1 SomeField2:{Defined:true Value:0xc000010370}}
SomeField2's value is somevalue2
Link to the playground with the full example: https://play.golang.org/p/AUDwPKHBs62
Do note that you will need one struct for each type you want to wrap, so, if you need an optional number you'll need to create an OptionalFloat64 (all JSON numbers can be 64 bit floats) struct with a similar implementation.
If/when generics land in Go, this could be simplified to a single generic struct.
Good question.
I believe you can use https://golang.org/pkg/encoding/json/#RawMessage as:
type MyMessage struct {
somefield1 string
somefield2 json.RawMessage
}
So after unmarshalling you should have []byte("null") in case of null and nil if missing.
Here is a playground code: https://play.golang.org/p/UW8L68K068
If you unmarshall the object into a map[string]interface{} then you can just check if a field is there
type unMarshalledObject map[string]interface{}
json.Unmarshal(input, unMarshalledObject)
_, ok := unMarshalledObject["somefield2"]
Go Playground
If struct field is a pointer, JSON decoder will allocate new variable if the field is present or leave it nil if not. So I suggest to use pointers.
type Data struct {
StrField *string
IntField *int
}
...
if data.StrField != nil {
handle(*data.StrField)
}
By using github.com/golang/protobuf/ptypes/struct and jsonpb github.com/golang/protobuf/jsonpb, you can do like this:
func TestFunTest(t *testing.T) {
p := &pb.KnownTypes{}
e := UnmarshalString(`{"val":null}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{"val":1}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{"val":"string"}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{}`, p)
fmt.Println(e, p)
}
Output:
[ `go test -test.run="^TestFunTest$"` | done: 1.275431416s ]
<nil> val:<null_value:NULL_VALUE >
<nil> val:<number_value:1 >
<nil> val:<string_value:"string" >
<nil>
PASS

How can I implement UnmarshalJSON for a type derived from a scalar in Go?

I have a simple type that implements conversion of subtyped integer consts to strings and vice versa in Go. I want to be able to automatically unmarshal strings in JSON to values of this type. I can't, because UnmarshalJSON doesn't give me a way to return or modify the scalar value. It's expecting a struct, whose members are set by UnmarshalJSON. The ",string" method doesn't work either for other than builtin scalar types. Is there a way to implement UnmarshalJSON correctly for a derived scalar type?
Here's an example of what I'm after. I want it to print "Hello Ralph" four times, but it prints "Hello Bob" four times because the PersonID isn't being changed.
package main
import (
"encoding/json"
"fmt"
)
type PersonID int
const (
Bob PersonID = iota
Jane
Ralph
Nobody = -1
)
var nameMap = map[string]PersonID{
"Bob": Bob,
"Jane": Jane,
"Ralph": Ralph,
"Nobody": Nobody,
}
var idMap = map[PersonID]string{
Bob: "Bob",
Jane: "Jane",
Ralph: "Ralph",
Nobody: "Nobody",
}
func (intValue PersonID) Name() string {
return idMap[intValue]
}
func Lookup(name string) PersonID {
return nameMap[name]
}
func (intValue PersonID) UnmarshalJSON(data []byte) error {
// The following line is not correct
intValue = Lookup(string(data))
return nil
}
type MyType struct {
Person PersonID `json: "person"`
Count int `json: "count"`
Greeting string `json: "greeting"`
}
func main() {
var m MyType
if err := json.Unmarshal([]byte(`{"person": "Ralph", "count": 4, "greeting": "Hello"}`), &m); err != nil {
fmt.Println(err)
} else {
for i := 0; i < m.Count; i++ {
fmt.Println(m.Greeting, m.Person.Name())
}
}
}
Use a pointer receiver for the unmarshal method. If a value receiver is used, changes to the receiver are lost when the method returns.
The argument to the unmarshal method is JSON text. Unmarshal the JSON text to get a plain string with all JSON quoting removed.
func (intValue *PersonID) UnmarshalJSON(data []byte) error {
var s string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
*intValue = Lookup(s)
return nil
}
There's a mismatch between the JSON tags an the example JSON. I changed the JSON to match the tag, but you can change it the other way.
if err := json.Unmarshal([]byte(`{"person": "Ralph", "count": 4, "greeting": "Hello"}`), &m); err != nil {
playground example
Here's an answer based on my comment. I'm not sure this is exactly what you want to do as some of your questions wording confuses me however the basic idea is to separate unmarshalling and transformation into two different steps. First unmarshal raw data into a compatible type, after do a transformation to another type or enrich the type you already have like in the example below. You're welcome to hide this behavior in a custom implementation of UnmarshalJSON if you'd like but I would personally advise against it. Here's my two reasons; 1) it's just not consistent with Go's explicit verbose coding style 2) I despise highly obfuscated packages/libraries/languages that do stuff like this for you because sooner or later it bites you in the ass an costs you a lot more than adding that 1 line of extra code in a few places (like hours trying to debug something that makes no sense to you).
type MyType struct {
Id PersonID
Name string `json: "name"`
Count int `json: "count"`
Greeting string `json: "greeting"`
}
func main() {
var m MyType
if err := json.Unmarshal([]byte(`{"name": "Ralph", "count": 4, "greeting": "Hello"}`), &m); err != nil {
fmt.Println(err)
} else {
m.Id = Lookup(m.Name) // see this isn't unmarshalling
// better to take the data as is and do transformation separate
for i := 0; i < m.Count; i++ {
fmt.Println(m.Greeting, m.Person.Name())
}
}
}

How to specify default values when parsing JSON in Go

I want to parse a JSON object in Go, but want to specify default values for fields that are not given. For example, I have the struct type:
type Test struct {
A string
B string
C string
}
The default values for A, B, and C, are "a", "b", and "c" respectively. This means that when I parse the json:
{"A": "1", "C": 3}
I want to get the struct:
Test{A: "1", B: "b", C: "3"}
Is this possible using the built-in package encoding/json? Otherwise, is there any Go library that has this functionality?
This is possible using encoding/json: when calling json.Unmarshal, you do not need to give it an empty struct, you can give it one with default values.
For your example:
var example []byte = []byte(`{"A": "1", "C": "3"}`)
out := Test{
A: "default a",
B: "default b",
// default for C will be "", the empty value for a string
}
err := json.Unmarshal(example, &out) // <--
if err != nil {
panic(err)
}
fmt.Printf("%+v", out)
Running this example returns {A:1 B:default b C:3}.
As you can see, json.Unmarshal(example, &out) unmarshals the JSON into out, overwriting the values specified in the JSON, but leaving the other fields unchanged.
In case u have a list or map of Test structs the accepted answer is not possible anymore but it can easily be extended with a UnmarshalJSON method:
func (t *Test) UnmarshalJSON(data []byte) error {
type testAlias Test
test := &testAlias{
B: "default B",
}
err := json.Unmarshal(data, test)
if err != nil {
return err
}
*t = Test(*test)
return nil
}
The testAlias is needed to prevent recursive calls to UnmarshalJSON. This works because a new type has no methods defined.
https://play.golang.org/p/qiGyjRbNHg
You can also implement the UnmarshalJSON interface and set the default values in it.
func (test *Test) UnmarshalJSON(data []byte) error {
test.A = "1"
test.B = "2"
test.C = "3"
type tempTest Test
return json.Unmarshal(data, (*tempTest)(test))
}
The type tempTest is needed so that UnmarshalJSON is not called recursively.
The json.Unmarshal with a default value is simple and clean like the answers given by Christian and JW., but it has some downsides.
First, it strongly ties the default values of fields with the parsing logic. It's conceivable that we want to let user code down the line set its defaults; right now, the defaults have to be set before unmarshaling.
The second downside is that it only works in simple cases. If our Options struct has a slice or map of other structs, we can't populate defaults this way.
Another option is Default values with pointer fields
type Test struct {
A *string
B *string
C *string
}
js := []byte(`{"A": "1", "C": "3"}`)
var t Test
if err := json.Unmarshal(js, &t); err != nil {
fmt.Println(err)
}
if t.B == nil {
var defaultB = "B"
t.B = &defaultB
}