Misunderstanding how to use MarshalJSON - json

I have a rather unusual situation. I want MarshalJSON to conditionally omit a struct field. In the example below, the idea is to omit output of the B Bool field if B.Value == B.Undefined.
type Bool struct {
// Current value
Value bool
// Value if undefined
Undefined bool
}
func (b Bool) MarshalJSON() ([]byte, error) {
if b.Value == b.Undefined {
return []byte{}, nil
} else if b.Value {
return ([]byte)("true"), nil
}
return ([]byte)("false"), nil
}
func main() {
var example = struct {
N int `json:"foo"`
B Bool `json:"value,omitempty"`
}
example.B = Bool{true, true}
output, err := json.Marshal(example)
if err != nil {
panic(err)
}
fmt.Println(string(output))
}
Playground link
According to the docs:
The "omitempty" option specifies that the field should be omitted from
the encoding if the field has an empty value, defined as false, 0, a
nil pointer, a nil interface value, and any empty array, slice, map,
or string.
I return an empty byte slice, which results in an error:
panic: json: error calling MarshalJSON for type main.Bool: unexpected end of JSON input
goroutine 1 [running]:
main.main()
/tmp/sandbox933539113/prog.go:32 +0x160
If I set example.B = Bool{false, true}, then it prints the result, but the field still isn't omitted, despite returning "false" from Bool.MarshalJSON. What am I doing wrong? Does a value with a type satisfying the Marshaler interface effectively ignore the omitempty tag entry?

if b.Value == b.Undefined {
return []byte{}, nil
produces invalid JSON: it returns 0 bytes. Your MarshalJSON method must return valid JSON, which means it must return something.
If your goal is to omit that field entirely, when it's not set, use a pointer, and don't set it:
func main() {
var example = struct {
N int `json:"foo"`
B *Bool `json:"value,omitempty"`
}
example.B = Bool{N: true, B: nil}
output, err := json.Marshal(example)
if err != nil {
panic(err)
}
fmt.Println(string(output))
}
If you want to omit that field entirely when the value of Bool is a particular value, then you must define your MarshalJSON method on the containing struct, so that it can properly omit the field when desired.

Related

Unmarshal json null into Pointer of NullString

I am unable to json.Unmarshal a null value into a *NullString field within a struct. Here is a simplified example of what I mean:
package main
import (
"database/sql"
"encoding/json"
"log"
)
// NullString
type NullString struct {
sql.NullString
}
func (n *NullString) UnmarshalJSON(b []byte) error {
n.Valid = string(b) != "null"
e := json.Unmarshal(b, &n.String)
return e
}
type Person struct {
Name *NullString `json:"name"`
}
func BuildUpdateSQL(jsonString string) string {
p := Person{}
e := json.Unmarshal([]byte(jsonString),&p)
if e != nil {
log.Println(e)
}
if p.Name != nil {
log.Println(p,p.Name)
} else {
log.Println(p)
}
return ""
}
func main() {
// Correctly leaves p.Name unset
BuildUpdateSQL(`{"field_not_exist":"samantha"}`)
// Correctly sets p.Name
BuildUpdateSQL(`{"name":"samantha"}`)
// Incorrectly leaves p.Name as nil when I really want p.Name to have a NullString with .Valid == false
BuildUpdateSQL(`{"name":null}`)
}
As you can see, unmarshalling works for non-null json values. But when I pass in a null json value, the NullString unmarshaller doesn't seem to even fire.
Anyone know what I'm doing wrong?
Background
The reason I'm trying to do this is because I plan to get JSON value from a REST API. Not all fields in the API are required fields. Hence I use pointers for my struct fields to help me build my SQL Update statement because:
field with nil means not populated (do not include a SET name = ?)
non-nil NullString.Valid == false means actual null value (include a SET name = NULL)
and non-nil NullString.Valid == true means a real string value exists (include a SET name = ?)
Yes, this is because of the following unmarshaling rule:
To unmarshal JSON into a pointer, Unmarshal first handles the case of the JSON being the JSON literal null. In that case, Unmarshal sets the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into the value pointed at by the pointer.
(Documentation for encoding/json).
What I suggest to do is to add a Set field which is changed to true when UnmarshalJSON is fired (which if there is any value, is guaranteed to be fired), and then to change the *NullString to a simple NullString, like so:
package main
import (
"database/sql"
"encoding/json"
"log"
)
// NullString
type NullString struct {
Set bool
sql.NullString
}
func (n *NullString) UnmarshalJSON(b []byte) error {
n.Set = true
n.Valid = string(b) != "null"
e := json.Unmarshal(b, &n.String)
return e
}
type Person struct {
Name NullString `json:"name"`
}
func BuildUpdateSQL(jsonString string) string {
p := Person{}
e := json.Unmarshal([]byte(jsonString), &p)
if e != nil {
log.Println(e)
}
log.Printf("%#v", p)
return ""
}
func main() {
BuildUpdateSQL(`{"field_not_exist":"samantha"}`)
BuildUpdateSQL(`{"name":"samantha"}`)
BuildUpdateSQL(`{"name":null}`)
}
Playground

Stop json.Marshal() from stripping trailing zero from floating point number

I got the following problem:
My golang program converts some information into JSON.
For example it results in the following json:
{
"value":40,
"unit":"some_string"
}
The problem is the "input" for value is 40.0 and the marshalling strips the trailing zero. It would be no problem if the EPL which reads the JSON would be able to read 40 as float without the .0
So the JSON output should look like:
{
"value":40.0,
"unit":"some_string"
}
Is there a possibility to "stop" json.Marshal() from removing the zero?
Edit: Value must be a Float
#icza provided a good answer, but just to offer another option, you can define your own float type and define your own serialization for it. Like this
type KeepZero float64
func (f KeepZero) MarshalJSON() ([]byte, error) {
if float64(f) == float64(int(f)) {
return []byte(strconv.FormatFloat(float64(f), 'f', 1, 32)), nil
}
return []byte(strconv.FormatFloat(float64(f), 'f', -1, 32)), nil
}
type Pt struct {
Value KeepZero
Unit string
}
func main() {
data, err := json.Marshal(Pt{40.0, "some_string"})
fmt.Println(string(data), err)
}
This results in {"Value":40.0,"Unit":"some_string"} <nil>. Check it out in playground.
By default floating point numbers are rendered without a decimal point and fractions if its value is an integer value. The representation is shorter, and it means the same number.
If you want control over how a number appears in the JSON representation, use the json.Number type.
Example:
type Pt struct {
Value json.Number
Unit string
}
func main() {
data, err := json.Marshal(Pt{json.Number("40.0"), "some_string"})
fmt.Println(string(data), err)
}
Output (try it on the Go Playground):
{"Value":40.0,"Unit":"some_string"} <nil>
If you have a number as a float64 value, you may convert it to json.Number like this:
func toNumber(f float64) json.Number {
var s string
if f == float64(int64(f)) {
s = fmt.Sprintf("%.1f", f) // 1 decimal if integer
} else {
s = fmt.Sprint(f)
}
return json.Number(s)
}
Testing it:
f := 40.0
data, err := json.Marshal(Pt{toNumber(f), "some_string"})
fmt.Println(string(data), err)
f = 40.123
data, err = json.Marshal(Pt{toNumber(f), "some_string"})
fmt.Println(string(data), err)
Output (try it on the Go Playground):
{"Value":40.0,"Unit":"some_string"} <nil>
{"Value":40.123,"Unit":"some_string"} <nil>
The other direction, if you want the float64 value of a json.Number, simply call its Number.Float64() method.
I had a similar issue where I wanted to marshal a map[string]interface{} with float values f.x 1.0 to JSON as 1.0. I solved it by adding a custom Marshal function for a custom float type and then replace the floats in the map with the custom type:
type customFloat float64
func (f customFloat) MarshalJSON() ([]byte, error) {
if float64(f) == math.Trunc(float64(f)) {
return []byte(fmt.Sprintf("%.1f", f)), nil
}
return json.Marshal(float64(f))
}
func replaceFloat(value map[string]interface{}) {
for k, v := range value {
switch val := v.(type) {
case map[string]interface{}:
replaceFloat(val)
case float64:
value[k] = customFloat(val)
}
}
}
Then replace all float64 nodes:
replaceFloat(myValue)
bytes, err := json.Marshal(myValue)
This will print the floats like 1.0
type MyFloat float64
func (mf MyFloat) MarshalJSON() ([]byte, error) {
return []byte(fmt.Sprintf("%.1f", float64(mf))), nil
}
used like this
type PricePoint struct {
Price MyFloat `json:"price"`
From time.Time `json:"valid_from"`
To time.Time `json:"valid_to"`
}
Replace the 1 in "%.1f" with what ever precision you need
Save the value as a string and cast it back if you need it.

Decode JSON value which can be either string or number

When I make an HTTP call to a REST API I may get the JSON value count back as a Number or String. I'ld like to marshal it to be an integer in either case. How can I deal with this in Go?.
Use the "string" field tag option to specify that strings should be converted to numbers. The documentation for the option is:
The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types. This extra level of encoding is sometimes used when communicating with JavaScript programs:
Here's an example use:
type S struct {
Count int `json:"count,string"`
}
playground example
If the JSON value can be number or string, then unmarshal to interface{} and convert to int after unmarshaling:
Count interface{} `json:"count,string"`
Use this function to convert the interface{} value to an int:
func getInt(v interface{}) (int, error) {
switch v := v.(type) {
case float64:
return int(v), nil
case string:
c, err := strconv.Atoi(v)
if err != nil {
return 0, err
}
return c, nil
default:
return 0, fmt.Errorf("conversion to int from %T not supported", v)
}
}
// Format of your expected request
type request struct {
ACTIVE string `json:"active"`
CATEGORY string `json:"category"`
}
// struct to read JSON input
var myReq request
// Decode the received JSON request to struct
decoder := json.NewDecoder(r.Body)
err := decoder.Decode(&myReq)
if err != nil {
log.Println( err)
// Handler for invalid JSON received or if you want to decode the request using another struct with int.
return
}
defer r.Body.Close()
// Convert string to int
numActive, err = strconv.Atoi(myReq.ACTIVE)
if err != nil {
log.Println(err)
// Handler for invalid int received
return
}
// Convert string to int
numCategory, err = strconv.Atoi(myReq.CATEGORY)
if err != nil {
log.Println(err)
// Handler for invalid int received
return
}
I had the same problem with a list of values where the values were string or struct. The solution I'm using is to create a helper struct with fields of expected types and parse value into the correct field.
type Flag struct {
ID string `json:"id"`
Type string `json:"type"`
}
type FlagOrString struct {
Flag *Flag
String *string
}
func (f *FlagOrString) UnmarshalJSON(b []byte) error {
start := []byte("\"")
for idx := range start {
if b[idx] != start[idx] {
return json.Unmarshal(b, &f.Flag)
}
}
return json.Unmarshal(b, &f.String)
}
var MainStruct struct {
Vals []FlagOrString
}
Custom Unmarshaller simplifies a code. Personally I prefer this over interface{} as it explicitly states what a developer expects.

JSON field set to null vs field not there

Is there a way, in golang, to see if I can differentiate between a json field being set to null vs a json field not being there when unmarshalled into a struct? Because both set the value in the struct to be nil, but I need to know if the field was there to begin with and to see if someone set it to null.
{
"somefield1":"somevalue1",
"somefield2":null
}
VS
{
"somefield1":"somevalue1",
}
Both jsons will be nil when unmarshalled into a struct.
Any useful resources will be very appreciated!
Use json.RawMessage to "delay" the unmarshaling process to determine the raw byte before deciding to do something:
var data = []byte(`{
"somefield1":"somevalue1",
"somefield2": null
}`)
type Data struct {
SomeField1 string
SomeField2 json.RawMessage
}
func main() {
d := &Data{}
_ = json.Unmarshal(data, &d)
fmt.Println(d.SomeField1)
if len(d.SomeField2) > 0 {
if string(d.SomeField2) == "null" {
fmt.Println("somefield2 is there but null")
} else {
fmt.Println("somefield2 is there and not null")
// Do something with the data
}
} else {
fmt.Println("somefield2 doesn't exist")
}
}
See the playground https://play.golang.org/p/Wganpf4sbO
If you're on Go 1.18+ you can use a simple generic struct to know when a JSON value is undefined or null:
type Optional[T any] struct {
Defined bool
Value *T
}
// UnmarshalJSON is implemented by deferring to the wrapped type (T).
// It will be called only if the value is defined in the JSON payload.
func (o *Optional[T]) UnmarshalJSON(data []byte) error {
o.Defined = true
return json.Unmarshal(data, &o.Value)
}
That's all, you can then just use this type in your structs:
type Payload struct {
Field1 Optional[string] `json:"field1"`
Field2 Optional[bool] `json:"field2"`
Field3 Optional[int32] `json:"field3"`
}
And you'll be able to use the Defined field to know if the field was null or undefined.
Check this playground link for a full example: https://go.dev/play/p/JZfZyVVUABz
Original answer (pre-generics)
Another way to do this, with a custom type:
// OptionalString is a struct that represents a JSON string that can be
// undefined (Defined == false), null (Value == nil && Defined == true) or
// defined with a string value
type OptionalString struct {
Defined bool
Value *string
}
// UnmarshalJSON implements the json.Unmarshaler interface.
// When called, it means that the value is defined in the JSON payload.
func (os *OptionalString) UnmarshalJSON(data []byte) error {
// UnmarshalJSON is called only if the key is present
os.Defined = true
return json.Unmarshal(data, &os.Value)
}
// Payload represents the JSON payload that you want to represent.
type Payload struct {
SomeField1 string `json:"somefield1"`
SomeField2 OptionalString `json:"somefield2"`
}
You can then just use the regular json.Unmarshal function to get your values, for example:
var p Payload
_ = json.Unmarshal([]byte(`{
"somefield1":"somevalue1",
"somefield2":null
}`), &p)
fmt.Printf("Should be defined == true and value == nil: \n%+v\n\n", p)
p = Payload{}
_ = json.Unmarshal([]byte(`{"somefield1":"somevalue1"}`), &p)
fmt.Printf("Should be defined == false \n%+v\n\n", p)
p = Payload{}
_ = json.Unmarshal([]byte(`{
"somefield1":"somevalue1",
"somefield2":"somevalue2"
}`), &p)
fmt.Printf("Parsed should be defined == true and value != nil \n%+v\n", p)
if p.SomeField2.Value != nil {
fmt.Printf("SomeField2's value is %s", *p.SomeField2.Value)
}
Should give you this output:
Should be defined == true and value == nil:
{SomeField1:somevalue1 SomeField2:{Defined:true Value:<nil>}}
Should be defined == false
{SomeField1:somevalue1 SomeField2:{Defined:false Value:<nil>}}
Parsed should be defined == true and value != nil
{SomeField1:somevalue1 SomeField2:{Defined:true Value:0xc000010370}}
SomeField2's value is somevalue2
Link to the playground with the full example: https://play.golang.org/p/AUDwPKHBs62
Do note that you will need one struct for each type you want to wrap, so, if you need an optional number you'll need to create an OptionalFloat64 (all JSON numbers can be 64 bit floats) struct with a similar implementation.
If/when generics land in Go, this could be simplified to a single generic struct.
Good question.
I believe you can use https://golang.org/pkg/encoding/json/#RawMessage as:
type MyMessage struct {
somefield1 string
somefield2 json.RawMessage
}
So after unmarshalling you should have []byte("null") in case of null and nil if missing.
Here is a playground code: https://play.golang.org/p/UW8L68K068
If you unmarshall the object into a map[string]interface{} then you can just check if a field is there
type unMarshalledObject map[string]interface{}
json.Unmarshal(input, unMarshalledObject)
_, ok := unMarshalledObject["somefield2"]
Go Playground
If struct field is a pointer, JSON decoder will allocate new variable if the field is present or leave it nil if not. So I suggest to use pointers.
type Data struct {
StrField *string
IntField *int
}
...
if data.StrField != nil {
handle(*data.StrField)
}
By using github.com/golang/protobuf/ptypes/struct and jsonpb github.com/golang/protobuf/jsonpb, you can do like this:
func TestFunTest(t *testing.T) {
p := &pb.KnownTypes{}
e := UnmarshalString(`{"val":null}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{"val":1}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{"val":"string"}`, p)
fmt.Println(e, p)
p = &pb.KnownTypes{}
e = UnmarshalString(`{}`, p)
fmt.Println(e, p)
}
Output:
[ `go test -test.run="^TestFunTest$"` | done: 1.275431416s ]
<nil> val:<null_value:NULL_VALUE >
<nil> val:<number_value:1 >
<nil> val:<string_value:"string" >
<nil>
PASS

In Go, why is JSON null sometimes not passed to UnmarshalJSON for decoding?

Go provides the encoding/json.Unmarshaler interface so types can control the way they are decoded from JSON. In almost all cases, the encoded JSON value is passed directly to the UnmarshalJSON method, but not if the Unmarshaler is a pointer and the JSON value is null. In that case, the pointer is set to nil without calling UnmarshalJSON at all. Here's an example:
package main
import (
"encoding/json"
"fmt"
)
type T string
func (v *T) UnmarshalJSON(b []byte) error {
if b[0] == 'n' {
*v = "null"
} else {
*v = "not null"
}
return nil
}
func main() {
var a struct {
T T
PT1 *T
PT2 *T
}
a.PT1 = nil // just to be explicit
a.PT2 = new(T)
err := json.Unmarshal([]byte(`{"T":null,"PT1":"foo","PT2":null}`), &a)
if err != nil {
panic(err)
}
fmt.Printf("a.T is %#v\n", a.T)
if a.PT1 == nil {
fmt.Println("a.PT1 is nil")
} else {
fmt.Printf("a.PT1 points to %#v\n", *a.PT1)
}
if a.PT2 == nil {
fmt.Println("a.PT2 is nil")
} else {
fmt.Printf("a.PT2 points to %#v\n", *a.PT2)
}
}
I expected this to print
a.T is "null"
a.PT1 points to "not null"
a.PT2 points to "null"
Instead, it prints
a.T is "null"
a.PT1 points to "not null"
a.PT2 is nil
So json.Unmarshal allocates a new T for a.PT1, which is initially nil. But it sets a.PT2 to nil without calling UnmarshalJSON, even though a.PT2 was not nil. Why?
This is because setting the pointer to nil is the most common way to handle a JSON null, and there is no way for an UnmarshalJSON method of *T to do that on its own. If UnmarshalJSON were called in this case, you would have to define (**T).UnmarshalJSON to set the *T to nil. This would make the most common case very awkward.
If you don't want JSON null to become Go nil, don't use a pointer.