JSON field set to null vs field not there - json

Question

Is there a way, in golang, to see if I can differentiate between a json field being set to null vs a json field not being there when unmarshalled into a struct? Because both set the value in the struct to be nil, but I need to know if the field was there to begin with and to see if someone set it to null.
{
"somefield1":"somevalue1",
"somefield2":null
}
VS
{
"somefield1":"somevalue1",
}
Both jsons will be nil when unmarshalled into a struct.
Any useful resources will be very appreciated!

Answer 1

Use json.RawMessage to "delay" the unmarshaling process to determine the raw byte before deciding to do something:
var data = []byte(`{
"somefield1":"somevalue1",
"somefield2": null
}`)
type Data struct {
SomeField1 string
SomeField2 json.RawMessage
}
func main() {
d := &Data{}
_ = json.Unmarshal(data, &d)
fmt.Println(d.SomeField1)
if len(d.SomeField2) > 0 {
if string(d.SomeField2) == "null" {
fmt.Println("somefield2 is there but null")
} else {
fmt.Println("somefield2 is there and not null")
// Do something with the data
}
} else {
fmt.Println("somefield2 doesn't exist")
}
}
See the playground https://play.golang.org/p/Wganpf4sbO

Answer 2

If you're on Go 1.18+ you can use a simple generic struct to know when a JSON value is undefined or null:
type Optional[T any] struct {
Defined bool
Value *T
}
// UnmarshalJSON is implemented by deferring to the wrapped type (T).
// It will be called only if the value is defined in the JSON payload.
func (o *Optional[T]) UnmarshalJSON(data []byte) error {
o.Defined = true
return json.Unmarshal(data, &o.Value)
}
That's all, you can then just use this type in your structs:
type Payload struct {
Field1 Optional[string] `json:"field1"`
Field2 Optional[bool] `json:"field2"`
Field3 Optional[int32] `json:"field3"`
}
And you'll be able to use the Defined field to know if the field was null or undefined.
Check this playground link for a full example: https://go.dev/play/p/JZfZyVVUABz
Original answer (pre-generics)
Another way to do this, with a custom type:
// OptionalString is a struct that represents a JSON string that can be
// undefined (Defined == false), null (Value == nil && Defined == true) or
// defined with a string value
type OptionalString struct {
Defined bool
Value *string
}
// UnmarshalJSON implements the json.Unmarshaler interface.
// When called, it means that the value is defined in the JSON payload.
func (os *OptionalString) UnmarshalJSON(data []byte) error {
// UnmarshalJSON is called only if the key is present
os.Defined = true
return json.Unmarshal(data, &os.Value)
}
// Payload represents the JSON payload that you want to represent.
type Payload struct {
SomeField1 string `json:"somefield1"`
SomeField2 OptionalString `json:"somefield2"`
}
You can then just use the regular json.Unmarshal function to get your values, for example:
var p Payload
_ = json.Unmarshal([]byte(`{
"somefield1":"somevalue1",
"somefield2":null
}`), &p)
fmt.Printf("Should be defined == true and value == nil: \n%+v\n\n", p)
p = Payload{}
_ = json.Unmarshal([]byte(`{"somefield1":"somevalue1"}`), &p)
fmt.Printf("Should be defined == false \n%+v\n\n", p)
p = Payload{}
_ = json.Unmarshal([]byte(`{
"somefield1":"somevalue1",
"somefield2":"somevalue2"
}`), &p)
fmt.Printf("Parsed should be defined == true and value != nil \n%+v\n", p)
if p.SomeField2.Value != nil {
fmt.Printf("SomeField2's value is %s", *p.SomeField2.Value)
}
Should give you this output:
Should be defined == true and value == nil:
{SomeField1:somevalue1 SomeField2:{Defined:true Value:<nil>}}
Should be defined == false
{SomeField1:somevalue1 SomeField2:{Defined:false Value:<nil>}}
Parsed should be defined == true and value != nil
{SomeField1:somevalue1 SomeField2:{Defined:true Value:0xc000010370}}
SomeField2's value is somevalue2
Link to the playground with the full example: https://play.golang.org/p/AUDwPKHBs62
Do note that you will need one struct for each type you want to wrap, so, if you need an optional number you'll need to create an OptionalFloat64 (all JSON numbers can be 64 bit floats) struct with a similar implementation.
If/when generics land in Go, this could be simplified to a single generic struct.

Answer 3

Good question.
I believe you can use https://golang.org/pkg/encoding/json/#RawMessage as:
type MyMessage struct {
somefield1 string
somefield2 json.RawMessage
}
So after unmarshalling you should have []byte("null") in case of null and nil if missing.
Here is a playground code: https://play.golang.org/p/UW8L68K068

Answer 4

If you unmarshall the object into a map[string]interface{} then you can just check if a field is there
type unMarshalledObject map[string]interface{}
json.Unmarshal(input, unMarshalledObject)
_, ok := unMarshalledObject["somefield2"]
Go Playground

Answer 5

If struct field is a pointer, JSON decoder will allocate new variable if the field is present or leave it nil if not. So I suggest to use pointers.
type Data struct {
StrField *string
IntField *int
}
...
if data.StrField != nil {
handle(*data.StrField)
}

Answer 6

By using github.com/golang/protobuf/ptypes/struct and jsonpb github.com/golang/protobuf/jsonpb, you can do like this:
func TestFunTest(t *testing.T) {
p := &pb.KnownTypes{}
e := UnmarshalString(`{"val":null}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{"val":1}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{"val":"string"}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{}`, p)
fmt.Println(e, p)
}
Output:
[ `go test -test.run="^TestFunTest$"` | done: 1.275431416s ]
<nil> val:<null_value:NULL_VALUE >
<nil> val:<number_value:1 >
<nil> val:<string_value:"string" >
<nil>
PASS