I am unable to json.Unmarshal a null value into a *NullString field within a struct. Here is a simplified example of what I mean:
package main
import (
"database/sql"
"encoding/json"
"log"
)
// NullString
type NullString struct {
sql.NullString
}
func (n *NullString) UnmarshalJSON(b []byte) error {
n.Valid = string(b) != "null"
e := json.Unmarshal(b, &n.String)
return e
}
type Person struct {
Name *NullString `json:"name"`
}
func BuildUpdateSQL(jsonString string) string {
p := Person{}
e := json.Unmarshal([]byte(jsonString),&p)
if e != nil {
log.Println(e)
}
if p.Name != nil {
log.Println(p,p.Name)
} else {
log.Println(p)
}
return ""
}
func main() {
// Correctly leaves p.Name unset
BuildUpdateSQL(`{"field_not_exist":"samantha"}`)
// Correctly sets p.Name
BuildUpdateSQL(`{"name":"samantha"}`)
// Incorrectly leaves p.Name as nil when I really want p.Name to have a NullString with .Valid == false
BuildUpdateSQL(`{"name":null}`)
}
As you can see, unmarshalling works for non-null json values. But when I pass in a null json value, the NullString unmarshaller doesn't seem to even fire.
Anyone know what I'm doing wrong?
Background
The reason I'm trying to do this is because I plan to get JSON value from a REST API. Not all fields in the API are required fields. Hence I use pointers for my struct fields to help me build my SQL Update statement because:
field with nil means not populated (do not include a SET name = ?)
non-nil NullString.Valid == false means actual null value (include a SET name = NULL)
and non-nil NullString.Valid == true means a real string value exists (include a SET name = ?)
Yes, this is because of the following unmarshaling rule:
To unmarshal JSON into a pointer, Unmarshal first handles the case of the JSON being the JSON literal null. In that case, Unmarshal sets the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into the value pointed at by the pointer.
(Documentation for encoding/json).
What I suggest to do is to add a Set field which is changed to true when UnmarshalJSON is fired (which if there is any value, is guaranteed to be fired), and then to change the *NullString to a simple NullString, like so:
package main
import (
"database/sql"
"encoding/json"
"log"
)
// NullString
type NullString struct {
Set bool
sql.NullString
}
func (n *NullString) UnmarshalJSON(b []byte) error {
n.Set = true
n.Valid = string(b) != "null"
e := json.Unmarshal(b, &n.String)
return e
}
type Person struct {
Name NullString `json:"name"`
}
func BuildUpdateSQL(jsonString string) string {
p := Person{}
e := json.Unmarshal([]byte(jsonString), &p)
if e != nil {
log.Println(e)
}
log.Printf("%#v", p)
return ""
}
func main() {
BuildUpdateSQL(`{"field_not_exist":"samantha"}`)
BuildUpdateSQL(`{"name":"samantha"}`)
BuildUpdateSQL(`{"name":null}`)
}
Playground
Related
I have a database schema where Age is optional. But when in Go, via my RestFUL interface, I output the data as JSON, I get a very ugly response like {"ID":1,"Name":"John","age":{"Int64":0,"Valid":false}}
for age. I expected a null value to be omitted. What am I missing?
// CREATE TABLE users (
// id INT AUTO_INCREMENT PRIMARY KEY,
// name VARCHAR(50) NOT NULL,
// age INT
// );
// Age is optional
type user struct {
ID int `db:"id"`
Name string `db:"name"`
Age sql.NullInt64 `db:"age" json:"age,omitempty"`
}
func main() {
u := user{ID: 1, Name: "John"}
j, _ := json.Marshal(u)
fmt.Printf("%s", j)
}
// Expected: {"ID":1,"Name":"John"} since age is NULL/empty
https://play.golang.org/p/PS1-4Gw9h5u
sql.NullInt64 exists because SQL's null cannot be represented as Go's int. It's a third state that cannot be represented by any int value.
One solution to that would be to represent such SQL values as *int, but that would require an allocation for cases where the value is not null in the database, and allocations are bad for performance.
Designers of SQL package came up with NullInt64 solution that encodes the third state of null as an additional Valid boolean. It's not a good solution but it's the best we can get.
I'm not sure if it's possible to write JSON marshaller for NullInt64 that would work as you expect.
There's still the "third state" problem when marshalling to JSON. With ,omitempty a 0 int would also be omitted so how can you tell 0 from "not exists"/null?
Either way they didn't write custom marshaller for NullInt64 so it just encodes as the struct that it is.
You can create an alias type for NullInt64, write a JSON marshaller to encode the way you want to JSON (you need an alias because you can't add methods to types from other packages). You would also need to cast between your NullInt64 and sql.NullInt64.
You can use custom type for null int64.
This type works as you expected.
Usage example:
package main
import (
"encoding/json"
"database/sql/driver"
"fmt"
"strconv"
)
type user struct {
ID int `db:"id"`
Name string `db:"name"`
// pointer needed to omitempty work, otherwise output will be "age": null
Age *NullInt64 `db:"age" json:"age,omitempty"`
}
func main() {
u := user{ID: 1, Name: "John"}
j, _ := json.Marshal(u)
fmt.Printf("%s\n", j)
}
// Output:
// {"ID":1,"Name":"John"}
type NullInt64 struct {
Val int64
IsValid bool
}
func NewNullInt64(val interface{}) NullInt64 {
ni := NullInt64{}
ni.Set(val)
return ni
}
func (ni *NullInt64) Scan(value interface{}) error {
ni.Val, ni.IsValid = value.(int64)
return nil
}
func (ni NullInt64) Value() (driver.Value, error) {
if !ni.IsValid {
return nil, nil
}
return ni.Val, nil
}
func (ni *NullInt64) Set(val interface{}) {
ni.Val, ni.IsValid = val.(int64)
}
func (ni NullInt64) MarshalJSON() ([]byte, error) {
if !ni.IsValid {
return []byte(`null`), nil
}
return []byte(strconv.FormatInt(ni.Val, 10)), nil
}
func (ni *NullInt64) UnmarshalJSON(data []byte) error {
if data == nil || string(data) == `null` {
ni.IsValid = false
return nil
}
val, err := strconv.ParseInt(string(data), 10, 64)
if err != nil {
ni.IsValid = false
return err
}
ni.Val = val
ni.IsValid = true
return nil
}
func (ni NullInt64) String() string {
if !ni.IsValid {
return `<nil>`
}
return strconv.FormatInt(ni.Val, 10)
}
I have struct A and B. When a JSON string A is unmarshalled to struct A then it is valid, however if the JSON string A is unmarshalled to struct B it is still successful (which should not).
Is there any way to detect that wrong JSON input has been wrongly converted to a wrong struct type?
Please look at the code below: play
package main
import (
"encoding/json"
"fmt"
)
type A struct {
Name string `json:"name"`
Age int `json:"age"`
}
type B struct {
Alamat string `json:"alamat"`
Umur int `json:"umur"`
}
func main() {
var structA A
var structAA A
valA := "{\"name\":\"budi\",\"age\":10}"
valB := "{\"alamat\":\"jakarta\",\"umur\":120}"
//correct case
err := json.Unmarshal([]byte(valA), &structA)
if err != nil {
fmt.Println("fail unmarshal")
}
fmt.Println(structA.Name)
fmt.Println(structA.Age)
//unmarshalled successfully but with wrong json
err = json.Unmarshal([]byte(valB), &structAA)
if err != nil {
fmt.Println("fail unmarshal")
}
fmt.Println(structAA.Name)
fmt.Println(structAA.Age)
}
Referring the docs, json.Unmarshal(data []byte, v interface{}), by default, does not work in the way you suppose:
By default, object keys which don't have a corresponding struct field are ignored (see Decoder.DisallowUnknownFields for an alternative).
So, JSON properties alamat and umur in your valB are ignored as they are not matched in struct A and name and age are set to their type default.
I have some JSON of the form:
[{
"type": "car",
"color": "red",
"hp": 85,
"doors": 4
}, {
"type": "plane",
"color": "blue",
"engines": 3
}]
I have types car and plane that satisfy a vehicle interface; I'd like to be able to write:
var v []vehicle
e := json.Unmarshal(myJSON, &v)
... and have JSON fill my slice of vehicles with a car and a plane; instead (and unsurprisingly) I just get "cannot unmarshal object into Go value of type main.vehicle".
For reference, here are suitable definitions of the types involved:
type vehicle interface {
vehicle()
}
type car struct {
Type string
Color string
HP int
Doors int
}
func (car) vehicle() { return }
type plane struct {
Type string
Color string
Engines int
}
func (plane) vehicle() { return }
var _ vehicle = (*car)(nil)
var _ vehicle = (*plane)(nil)
(Note that I'm actually totally uninterested in the t field on car and plane - it could be omitted because this information will, if someone successfully answers this question, be implicit in the dynamic type of the objects in v.)
Is there a way to have the JSON umarhsaller choose which type to use based on some part of the contents (in this case, the type field) of the data being decoded?
(Note that this is not a duplicate of Unmarshal JSON with unknown fields because I want each item in the slice to have a different dynamic type, and from the value of the 'type' property I know exactly what fields to expect—I just don't know how to tell json.Unmarshal how to map 'type' property values onto Go types.)
Taking the answers from the similar question: Unmarshal JSON with unknown fields, we can construct a few ways to unamrshal this JSON object in a []vehicle data structure.
The "Unmarshal with Manual Handling" version can be done by using a generic []map[string]interface{} data structure, then building the correct vehicles from the slice of maps. For brevity, this example does leave out the error checking for missing or incorrectly typed fields which the json package would have done.
https://play.golang.org/p/fAY9JwVp-4
func NewVehicle(m map[string]interface{}) vehicle {
switch m["type"].(string) {
case "car":
return NewCar(m)
case "plane":
return NewPlane(m)
}
return nil
}
func NewCar(m map[string]interface{}) *car {
return &car{
Type: m["type"].(string),
Color: m["color"].(string),
HP: int(m["hp"].(float64)),
Doors: int(m["doors"].(float64)),
}
}
func NewPlane(m map[string]interface{}) *plane {
return &plane{
Type: m["type"].(string),
Color: m["color"].(string),
Engines: int(m["engines"].(float64)),
}
}
func main() {
var vehicles []vehicle
objs := []map[string]interface{}{}
err := json.Unmarshal(js, &objs)
if err != nil {
log.Fatal(err)
}
for _, obj := range objs {
vehicles = append(vehicles, NewVehicle(obj))
}
fmt.Printf("%#v\n", vehicles)
}
We could leverage the json package again to take care of the unmarshaling and type checking of the individual structs by unmarshaling a second time directly into the correct type. This could all be wrapped up into a json.Unmarshaler implementation by defining an UnmarshalJSON method on the []vehicle type to first split up the JSON objects into raw messages.
https://play.golang.org/p/zQyL0JeB3b
type Vehicles []vehicle
func (v *Vehicles) UnmarshalJSON(data []byte) error {
// this just splits up the JSON array into the raw JSON for each object
var raw []json.RawMessage
err := json.Unmarshal(data, &raw)
if err != nil {
return err
}
for _, r := range raw {
// unamrshal into a map to check the "type" field
var obj map[string]interface{}
err := json.Unmarshal(r, &obj)
if err != nil {
return err
}
vehicleType := ""
if t, ok := obj["type"].(string); ok {
vehicleType = t
}
// unmarshal again into the correct type
var actual vehicle
switch vehicleType {
case "car":
actual = &car{}
case "plane":
actual = &plane{}
}
err = json.Unmarshal(r, actual)
if err != nil {
return err
}
*v = append(*v, actual)
}
return nil
}
JSON decoding and encoding in Go is actually surprisingly well at recognizing fields inside embedded structs. E.g. decoding or encoding the following structure works when there is no overlapping fields between type A and type B:
type T struct{
Type string `json:"type"`
*A
*B
}
type A struct{
Baz int `json:"baz"`
}
type B struct{
Bar int `json:"bar"`
}
Be aware that if both "baz" and "bar" are set in the JSON for the example above, both the T.A and T.B properties will be set.
If there is overlapping fields between A and B, or just to be able to better discard invalid combinations of fields and type, you need to implement the json.Unmarshaler interface. To not have to first decode fields into a map, you can extend the trick of using embedded structs.
type TypeSwitch struct {
Type string `json:"type"`
}
type T struct {
TypeSwitch
*A
*B
}
func (t *T) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &t.TypeSwitch); err != nil {
return err
}
switch t.Type {
case "a":
t.A = &A{}
return json.Unmarshal(data, t.A)
case "b":
t.B = &B{}
return json.Unmarshal(data, t.B)
default:
return fmt.Errorf("unrecognized type value %q", t.Type)
}
}
type A struct {
Foo string `json:"bar"`
Baz int `json:"baz"`
}
type B struct {
Foo string `json:"foo"`
Bar int `json:"bar"`
}
For marshaling back, json.Marshaler must also be implemented if there is overlapping fields.
Full example: https://play.golang.org/p/UHAdxlVdFQQ
The two passes approach works fine, but there is also the option of the mapstructure package, that was created to do exactly this.
I was facing the same problem.
I'm using the lib github.com/mitchellh/mapstructure together the encoding/json.
I first, unmarshal the json to a map, and use mapstructure to convert the map to my struct, e.g.:
type (
Foo struct {
Foo string `json:"foo"`
}
Bar struct {
Bar string `json:"bar"`
}
)
func Load(jsonStr string, makeInstance func(typ string) any) (any, error) {
// json to map
m := make(map[string]any)
e := json.Unmarshal([]byte(jsonStr), &m)
if e != nil {
return nil, e
}
data := makeInstance(m["type"].(string))
// decoder to copy map values to my struct using json tags
cfg := &mapstructure.DecoderConfig{
Metadata: nil,
Result: &data,
TagName: "json",
Squash: true,
}
decoder, e := mapstructure.NewDecoder(cfg)
if e != nil {
return nil, e
}
// copy map to struct
e = decoder.Decode(m)
return data, e
}
Using:
f, _ := Load(`{"type": "Foo", "foo": "bar"}`, func(typ string) any {
switch typ {
case "Foo":
return &Foo{}
}
return nil
})
If the property is a string you can use .(string) for casting the property because the origin is an interface.
You can use it the next way:
v["type"].(string)
Below is a code snippet - I am confused as to how to assign to variables within my nested struct ("myTime") which I am using for JSON decoding. (I have some Unix timestamps in a JSON file and am hoping to learn how to decode them.)
This throws the following error:
main.go:15: cannot use time.Unix(a, 0) (type time.Time) as type *myTime in assignment
main.go:25: t.String undefined (type myTime has no field or method String)
I'm not sure exactly how to go about understanding the issue, so any explanation or a pointer to specific documentation would help greatly!
package main
import (
"encoding/binary"
"encoding/json"
"fmt"
"log"
"time"
)
type myTime time.Time
func (t *myTime) UnmarshalJSON(buf []byte) error {
a, _ := binary.Varint(buf)
t = time.Unix(a, 0)
return nil
}
func main() {
var t myTime
if err := json.Unmarshal([]byte("123123123.123123"), &t); err != nil {
log.Fatal(err)
}
fmt.Printf("result: %f\n", t.String())
}
time.Unix returns a time.Time value (not pointer)
so just do *t = myTime(time.Unix(a,0))
This basically assigns the value returned from time.Unix(a,0) to the value pointed to by t
as for String, you have to make your own:
func (m myTime) String() string { return time.Time(m).String() }
This is because when you alias a type, you do not inherit its method set.
so you must declare any method you want yourself.
Is there a way, in golang, to see if I can differentiate between a json field being set to null vs a json field not being there when unmarshalled into a struct? Because both set the value in the struct to be nil, but I need to know if the field was there to begin with and to see if someone set it to null.
{
"somefield1":"somevalue1",
"somefield2":null
}
VS
{
"somefield1":"somevalue1",
}
Both jsons will be nil when unmarshalled into a struct.
Any useful resources will be very appreciated!
Use json.RawMessage to "delay" the unmarshaling process to determine the raw byte before deciding to do something:
var data = []byte(`{
"somefield1":"somevalue1",
"somefield2": null
}`)
type Data struct {
SomeField1 string
SomeField2 json.RawMessage
}
func main() {
d := &Data{}
_ = json.Unmarshal(data, &d)
fmt.Println(d.SomeField1)
if len(d.SomeField2) > 0 {
if string(d.SomeField2) == "null" {
fmt.Println("somefield2 is there but null")
} else {
fmt.Println("somefield2 is there and not null")
// Do something with the data
}
} else {
fmt.Println("somefield2 doesn't exist")
}
}
See the playground https://play.golang.org/p/Wganpf4sbO
If you're on Go 1.18+ you can use a simple generic struct to know when a JSON value is undefined or null:
type Optional[T any] struct {
Defined bool
Value *T
}
// UnmarshalJSON is implemented by deferring to the wrapped type (T).
// It will be called only if the value is defined in the JSON payload.
func (o *Optional[T]) UnmarshalJSON(data []byte) error {
o.Defined = true
return json.Unmarshal(data, &o.Value)
}
That's all, you can then just use this type in your structs:
type Payload struct {
Field1 Optional[string] `json:"field1"`
Field2 Optional[bool] `json:"field2"`
Field3 Optional[int32] `json:"field3"`
}
And you'll be able to use the Defined field to know if the field was null or undefined.
Check this playground link for a full example: https://go.dev/play/p/JZfZyVVUABz
Original answer (pre-generics)
Another way to do this, with a custom type:
// OptionalString is a struct that represents a JSON string that can be
// undefined (Defined == false), null (Value == nil && Defined == true) or
// defined with a string value
type OptionalString struct {
Defined bool
Value *string
}
// UnmarshalJSON implements the json.Unmarshaler interface.
// When called, it means that the value is defined in the JSON payload.
func (os *OptionalString) UnmarshalJSON(data []byte) error {
// UnmarshalJSON is called only if the key is present
os.Defined = true
return json.Unmarshal(data, &os.Value)
}
// Payload represents the JSON payload that you want to represent.
type Payload struct {
SomeField1 string `json:"somefield1"`
SomeField2 OptionalString `json:"somefield2"`
}
You can then just use the regular json.Unmarshal function to get your values, for example:
var p Payload
_ = json.Unmarshal([]byte(`{
"somefield1":"somevalue1",
"somefield2":null
}`), &p)
fmt.Printf("Should be defined == true and value == nil: \n%+v\n\n", p)
p = Payload{}
_ = json.Unmarshal([]byte(`{"somefield1":"somevalue1"}`), &p)
fmt.Printf("Should be defined == false \n%+v\n\n", p)
p = Payload{}
_ = json.Unmarshal([]byte(`{
"somefield1":"somevalue1",
"somefield2":"somevalue2"
}`), &p)
fmt.Printf("Parsed should be defined == true and value != nil \n%+v\n", p)
if p.SomeField2.Value != nil {
fmt.Printf("SomeField2's value is %s", *p.SomeField2.Value)
}
Should give you this output:
Should be defined == true and value == nil:
{SomeField1:somevalue1 SomeField2:{Defined:true Value:<nil>}}
Should be defined == false
{SomeField1:somevalue1 SomeField2:{Defined:false Value:<nil>}}
Parsed should be defined == true and value != nil
{SomeField1:somevalue1 SomeField2:{Defined:true Value:0xc000010370}}
SomeField2's value is somevalue2
Link to the playground with the full example: https://play.golang.org/p/AUDwPKHBs62
Do note that you will need one struct for each type you want to wrap, so, if you need an optional number you'll need to create an OptionalFloat64 (all JSON numbers can be 64 bit floats) struct with a similar implementation.
If/when generics land in Go, this could be simplified to a single generic struct.
Good question.
I believe you can use https://golang.org/pkg/encoding/json/#RawMessage as:
type MyMessage struct {
somefield1 string
somefield2 json.RawMessage
}
So after unmarshalling you should have []byte("null") in case of null and nil if missing.
Here is a playground code: https://play.golang.org/p/UW8L68K068
If you unmarshall the object into a map[string]interface{} then you can just check if a field is there
type unMarshalledObject map[string]interface{}
json.Unmarshal(input, unMarshalledObject)
_, ok := unMarshalledObject["somefield2"]
Go Playground
If struct field is a pointer, JSON decoder will allocate new variable if the field is present or leave it nil if not. So I suggest to use pointers.
type Data struct {
StrField *string
IntField *int
}
...
if data.StrField != nil {
handle(*data.StrField)
}
By using github.com/golang/protobuf/ptypes/struct and jsonpb github.com/golang/protobuf/jsonpb, you can do like this:
func TestFunTest(t *testing.T) {
p := &pb.KnownTypes{}
e := UnmarshalString(`{"val":null}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{"val":1}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{"val":"string"}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{}`, p)
fmt.Println(e, p)
}
Output:
[ `go test -test.run="^TestFunTest$"` | done: 1.275431416s ]
<nil> val:<null_value:NULL_VALUE >
<nil> val:<number_value:1 >
<nil> val:<string_value:"string" >
<nil>
PASS