Go How to deal with float infinity before converting to JSON - json

I've come across a situation where I have some float64 fields that could be infinity/NaN and trying to marshal to JSON would result in an error regarding +Inf type isn't supported.
type Something interface {
Id string `firestore:"id"`
NumberA float64 `firestore:"numberA"`
NumberB float64 `firestore:"numberB"`
NumberC float64 `firestore:"numberC"`
}
This struct gets initially populated via another library (Google Firestore).
In reality this struct is much larger with a lot more fields that are floats.
I think I could use something like this loop below using reflect to to find them all, though I wonder if there is a cleaner way or more idiomatic approach.
v := reflect.ValueOf(structVar)
typeOfS := v.Type()
for i := 0; i< v.NumField(); i++ {
if typeOfS.Field(i).Type.Kind() == reflect.Float64 && math.IsInf(v.Field(i).Interface().(float64), 1) {
// ... some logic I'll put here
}
}
I don't understand how to implement custom marshalling so maybe that could be an option to handle +Inf?

Custom handling of values can be done through custom types that implement the Marshaler interface. Your Something type, though, is malformed. It's defined as type Something interface{}, whereas that really ought the be a type Something struct:
type Something struct {
Id string `firestore:"id"`
NumberA JSONFloat `firestore:"numberA"`
NumberB JSONFloat `firestore:"numberB"`
NumberC JSONFloat `firestore:"numberC"`
}
type JSONFloat float64
func (j JSONFloat) MarshalJSON() ([]byte, error) {
v := float64(j)
if math.IsInf(j, 0) {
// handle infinity, assign desired value to v
// or say +/- indicates infinity
s := "+"
if math.IsInf(v, -1) {
s = "-"
}
return []byte(s), nil
}
return json.Marshal(v) // marshal result as standard float64
}
func (j *JSONFloat) UnsmarshalJSON(v []byte) error {
if s := string(v); s == "+" || s == "-" {
// if +/- indiciates infinity
if s == "+" {
*j = JSONFloat(math.Inf(1))
return nil
}
*j = JSONFloat(math.Inf(-1))
return nil
}
// just a regular float value
var fv float64
if err := json.Unmarshal(v, &fv); err != nil {\
return err
}
*j = JSONFloat(fv)
return nil
}
That should do it

I created xhhuango/json to support NaN, +Inf, and -Inf.
type T struct {
N float64
IP float64
IN float64
}
func TestMarshalNaNAndInf(t *testing.T) {
s := T{
N: math.NaN(),
IP: math.Inf(1),
IN: math.Inf(-1),
}
got, err := Marshal(s)
if err != nil {
t.Errorf("Marshal() error: %v", err)
}
want := `{"N":NaN,"IP":+Inf,"IN":-Inf}`
if string(got) != want {
t.Errorf("Marshal() = %s, want %s", got, want)
}
}
func TestUnmarshalNaNAndInf(t *testing.T) {
data := []byte(`{"N":NaN,"IP":+Inf,"IN":-Inf}`)
var s T
err := Unmarshal(data, &s)
if err != nil {
t.Fatalf("Unmarshal: %v", err)
}
if !math.IsNaN(s.N) || !math.IsInf(s.IP, 1) || !math.IsInf(s.IN, -1) {
t.Fatalf("after Unmarshal, s.N=%f, s.IP=%f, s.IN=%f, want NaN, +Inf, -Inf", s.N, s.IP, s.IN)
}
}

Related

how to unmarshal json object if object is returning as empty string instead of empty struct

I'm receiving some data as JSON, but if a object is empty, it does not return a empty struct but a empty
string instead, and when unmarshaling, it returns an error.
So instead of data being {"key":{}} is {"key":""}} , it does not work even using omitempty field
Example: https://play.golang.org/p/N1iuWBxuo1C
type Store struct {
Title string `json:"title,omitempty"`
Item item `json:"item,omitempty"`
}
type item struct {
Price float32 `json:"price,omitempty"`
Kind string `json:"kind,omitempty"`
}
func main() {
var data1 Store
json1 := []byte(`{"title":"hello world","item":{"price":45.2,"kind":"fruit"}}`)
if err := json.Unmarshal(json1, &data1); err != nil {
log.Println("1, err: ", err)
return
}
log.Printf("data1: %+v\n", data1)
var data2 Store
json2 := []byte(`{"title":"hello world","item":{}}`)
if err := json.Unmarshal(json2, &data2); err != nil {
log.Println("2, err: ", err)
return
}
log.Printf("data2: %+v\n", data2)
var data3 Store
json3 := []byte(`{"title":"hello world","item":""}`)
if err := json.Unmarshal(json3, &data3); err != nil {
log.Println("3, err: ", err)
return
}
log.Printf("data3: %+v\n", data3)
}
You can have your item type implement the json.Unmarshaler interface.
func (i *item) UnmarshalJSON(data []byte) error {
if string(data) == `""` {
return nil
}
type tmp item
return json.Unmarshal(data, (*tmp)(i))
}
https://play.golang.org/p/1TrD57XULo9
This might be a matter of taste, but "" is a string with zero length. Not an empty object. JSON uses null to describe something empty. This works:
json3 := []byte(`{"title":"hello world","item":null}`)
if err := json.Unmarshal(json3, &data3); err != nil {
log.Println("3, err: ", err)
return
}
As far as the documentation goes, omitempty is an encoding option:
The "omitempty" option specifies that the field should be omitted from
the encoding if the field has an empty value, defined as false, 0, a
nil pointer, a nil interface value, and any empty array, slice, map,
or string.
json.Unmarshal does not specify any use of the omitempty tag.
If you don't have control over the input, use an interface type, a type switch and type assertion:
type Store struct {
Title string `json:"title,omitempty"`
Item item `json:"item,omitempty"`
}
type item struct {
Price float32 `json:"price,omitempty"`
Kind string `json:"kind,omitempty"`
}
func unmarshal(js []byte) (*Store, error) {
var data = struct { // Intermediate var for unmarshal
Title string
Item interface{}
}{}
if err := json.Unmarshal(js, &data); err != nil {
return nil, err
}
s := &Store{Title: data.Title}
switch item := data.Item.(type) { // type switch
case string, nil:
return s, nil // Item remains empty
case map[string]interface{}:
p, ok := item["price"].(float64) // assertion
if ok {
s.Item.Price = float32(p)
}
s.Item.Kind, _ = item["kind"].(string) // _ prevents panic
return s, nil
default:
return nil, errors.New("Unknown type")
}
}
func main() {
jsons := [][]byte{
[]byte(`{"title":"hello world","item":{"price":45.2,"kind":"fruit"}}`),
[]byte(`{"title":"hello world","item":{}}`),
[]byte(`{"title":"hello world","item":""}`),
[]byte(`{"title":"hello world","item":null}`),
}
for i, js := range jsons {
data, err := unmarshal(js)
if err != nil {
log.Println("1, err: ", err)
return
}
log.Printf("data %d: %+v\n", i, data)
}
}
https://play.golang.org/p/Dnq1ZVfGPE7
Create a type like type ItemOrEmptyString item
And implement Unmarshal interface for it to handle your custom case.
func(ies *ItemOrEmptyString)UnmarshalJSON(d []byte) error{
var i item
if string(d) == `""` {
return nil
}
err := json.Unmarshal(d, &i)
*ies = ItemOrEmptyString(i)
return err
}
Full code here

1 interface, 2 packages, same struct variable names but different json naming conventions

I am wondering if it is possible in the below scenario to be able to remove the two "Balance" structs in each package and just use the interface based "Balance" struct somehow for unmarshalling. The issue that I have is that the json structure that comes back from the individual api's for Balance is different so as of right now I am just converting the local Balance structs into the global interface based Balance struct and returning that. This method is not ideal especially for more complex but very similar functions that I have. I am sure there is a better way to do this but could not find anything similar. Thanks in advance.
//******************** API Inteface *****************************
//API Interface path: /global/interfaces/apiinterface/apiinterface.go && structs.go
// /global/interfaces/apiinterface/apiinterface.go
type APIInterface interface {
//.... other interface func(s)
GetBalance(account string) (balance Balance, err error)
}
// /global/interfaces/apiinterface/structs.go
type Balance struct{
Available float64
Unconfirmed float64
}
//******************** Library 1 *****************************
//Library 1 path: /library1/library1.go
type jsonResponse struct {
Success bool `json:"success"`
Message string `json:"message"`
Result json.RawMessage `json:"result"`
}
//Library 1 Balance struct /library1/structs.go
type Balance struct {
Available float64 `json:"available"` //json name difference <----------
Unconfirmed float64 `json:"unconfirmed"` //json name difference <----------
}
//Library 1 GetBalance /library1/library1.go
func (c *Library1) GetBalance(account string) (balance apiinterface.Balance, err error) {
r, err := c.client.do("getbalance", []interface{}{account})
if err != nil {
return
}
var response jsonResponse
if err = json.Unmarshal(r, &response); err != nil {
return
}
var tmpBalance Balance //library 1 Balance struct
err = json.Unmarshal(response.Result, &tmpBalance)
if err != nil{
return
}
//convert to global apiinterface struct
balance = apiinterface.Balance{
Available:tmpBalance.Available,
Unconfirmed:tmpBalance.Unconfirmed,
}
return
}
//******************** Library 2 *****************************
//Library 2 path: /library2/library2.go
type jsonResponse struct {
Success bool `json:"success"`
Message string `json:"message"`
Result json.RawMessage `json:"result"`
}
//Library 2 Balance struct
type Balance struct {
Available float64 `json:"total"` //json name difference <----------
Unconfirmed float64 `json:"pending"` //json name difference <----------
}
//Library 2 GetBalance
func (c *Library2) GetBalance(account string) (balance apiinterface.Balance, err error) {
r, err := c.client.do("getbalance", []interface{}{account})
if err != nil {
return
}
var response jsonResponse
if err = json.Unmarshal(r, &response); err != nil {
return
}
var tmpBalance Balance //library 2 Balance struct
err = json.Unmarshal(response.Result, &tmpBalance)
if err != nil{
return
}
//convert to global apiinterface struct
balance = apiinterface.Balance{
Available:tmpBalance.Available,
Unconfirmed:tmpBalance.Unconfirmed,
}
return
}
See if something like this does what you need:
package main
import (
"encoding/json"
"fmt"
)
type jsonResponse struct {
Success bool `json:"success"`
Message string `json:"message"`
Result Balance `json:"result"`
}
type Balance struct {
Available float64
Unconfirmed float64
}
func (b *Balance) UnmarshalJSON(data []byte) error {
m := map[string]float64{}
if err := json.Unmarshal(data, &m); err != nil {
return err
}
for k, v := range m {
if k == "available" || k == "total" {
b.Available = v
} else if k == "unconfirmed" || k == "pending" {
b.Unconfirmed = v
}
}
return nil
}
func main() {
tests := []string{
`{"success":true,"message":"first","result":{"available":42.42,"unconfirmed":0.88}}`,
`{"success":true,"message":"second","result":{"total":3.14,"pending":0.42}}`,
}
for _, t := range tests {
resp := jsonResponse{}
if err := json.Unmarshal([]byte(t), &resp); err != nil {
panic(err)
}
fmt.Printf("%+v\n", resp)
}
}
playground
you can try somthing like this:
type Balance struct {
*Balance1
*Balance2
}
and then check them with nil, then cast the not nil one to your interface

Golang - Json decoding, check field exist or not automatically [duplicate]

Is it possible to generate an error if a field was not found while parsing a JSON input using Go?
I could not find it in documentation.
Is there any tag that specifies the field as required?
There is no tag in the encoding/json package that sets a field to "required". You will either have to write your own MarshalJSON() method, or do a post check for missing fields.
To check for missing fields, you will have to use pointers in order to distinguish between missing/null and zero values:
type JsonStruct struct {
String *string
Number *float64
}
Full working example:
package main
import (
"fmt"
"encoding/json"
)
type JsonStruct struct {
String *string
Number *float64
}
var rawJson = []byte(`{
"string":"We do not provide a number"
}`)
func main() {
var s *JsonStruct
err := json.Unmarshal(rawJson, &s)
if err != nil {
panic(err)
}
if s.String == nil {
panic("String is missing or null!")
}
if s.Number == nil {
panic("Number is missing or null!")
}
fmt.Printf("String: %s Number: %f\n", *s.String, *s.Number)
}
Playground
You can also override the unmarshalling for a specific type (so a required field buried in a few json layers) without having to make the field a pointer. UnmarshalJSON is defined by the Unmarshaler interface.
type EnumItem struct {
Named
Value string
}
func (item *EnumItem) UnmarshalJSON(data []byte) (err error) {
required := struct {
Value *string `json:"value"`
}{}
all := struct {
Named
Value string `json:"value"`
}{}
err = json.Unmarshal(data, &required)
if err != nil {
return
} else if required.Value == nil {
err = fmt.Errorf("Required field for EnumItem missing")
} else {
err = json.Unmarshal(data, &all)
item.Named = all.Named
item.Value = all.Value
}
return
}
Here is another way by checking your customized tag
you can create a tag for your struct like:
type Profile struct {
Name string `yourprojectname:"required"`
Age int
}
Use reflect to check if the tag is assigned required value
func (p *Profile) Unmarshal(data []byte) error {
err := json.Unmarshal(data, p)
if err != nil {
return err
}
fields := reflect.ValueOf(p).Elem()
for i := 0; i < fields.NumField(); i++ {
yourpojectTags := fields.Type().Field(i).Tag.Get("yourprojectname")
if strings.Contains(yourpojectTags, "required") && fields.Field(i).IsZero() {
return errors.New("required field is missing")
}
}
return nil
}
And test cases are like:
func main() {
profile1 := `{"Name":"foo", "Age":20}`
profile2 := `{"Name":"", "Age":21}`
var profile Profile
err := profile.Unmarshal([]byte(profile1))
if err != nil {
log.Printf("profile1 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile1 unmarshal: %v\n", profile)
err = profile.Unmarshal([]byte(profile2))
if err != nil {
log.Printf("profile2 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile2 unmarshal: %v\n", profile)
}
Result:
profile1 unmarshal: {foo 20}
2009/11/10 23:00:00 profile2 unmarshal error: required field is missing
You can go to Playground to have a look at the completed code
You can just implement the Unmarshaler interface to customize how your JSON gets unmarshalled.
you can also make use of JSON schema validation.
package main
import (
"encoding/json"
"fmt"
"github.com/alecthomas/jsonschema"
"github.com/xeipuuv/gojsonschema"
)
type Bird struct {
Species string `json:"birdType"`
Description string `json:"what it does" jsonschema:"required"`
}
func main() {
var bird Bird
sc := jsonschema.Reflect(&bird)
b, _ := json.Marshal(sc)
fmt.Println(string(b))
loader := gojsonschema.NewStringLoader(string(b))
documentLoader := gojsonschema.NewStringLoader(`{"birdType": "pigeon"}`)
schema, err := gojsonschema.NewSchema(loader)
if err != nil {
panic("nop")
}
result, err := schema.Validate(documentLoader)
if err != nil {
panic("nop")
}
if result.Valid() {
fmt.Printf("The document is valid\n")
} else {
fmt.Printf("The document is not valid. see errors :\n")
for _, err := range result.Errors() {
// Err implements the ResultError interface
fmt.Printf("- %s\n", err)
}
}
}
Outputs
{"$schema":"http://json-schema.org/draft-04/schema#","$ref":"#/definitions/Bird","definitions":{"Bird":{"required":["birdType","what it does"],"properties":{"birdType":{"type":"string"},"what it does":{"type":"string"}},"additionalProperties":false,"type":"object"}}}
The document is not valid. see errors :
- (root): what it does is required
code example taken from Strict JSON parsing

How to parse both plain and enquoted JSON numbers in Go?

I'm dealing with a third-party JSON-based API. Usually it wraps all numbers in quotes, but sometimes it doesn't. Nothing I can do about it.
I'm trying to come up with a solution, which parses the numbers regardless of whether they're enquoted or not. Standard library provides a ,string field tag, which allows to map numeric fields to enquoted values, but, unfortunately, it then fails to process the value, if it's not in quotes.
func test(s string) {
err := json.Unmarshal([]byte(s), &struct {
F1 float64
F2 float64 `json:",string"`
}{})
if err != nil {
fmt.Println(err)
return
}
fmt.Println("success")
}
func main() {
test(`{"f1": 1.23}`) // success
test(`{"f1": "1.23"}`) // cannot unmarshal string into Go value of type float64
test(`{"f2": 1.23}`) // invalid use of ,string struct tag, trying to unmarshal unquoted value into ...
test(`{"f2": "1.23"}`) // success
}
The Go Playground
Is there a way around this?
The proper solution is to "clone" float64 and define custom MarshalJSON and UnmarshalJSON for it:
type jsonFloat64 float64
func (f jsonFloat64) MarshalJSON() ([]byte, error) {
return json.Marshal(float64(f))
}
func (f *jsonFloat64) UnmarshalJSON(data []byte) error {
if len(data) >= 2 && data[0] == '"' && data[len(data)-1] == '"' {
data = data[1 : len(data)-1]
}
var tmp float64
err := json.Unmarshal(data, &tmp)
if err != nil {
return err
}
*f = jsonFloat64(tmp)
return nil
}
Then you'd be able to do something like this:
func test(s string) {
err := json.Unmarshal([]byte(s), &struct {
F jsonFloat64
}{})
if err != nil {
fmt.Println(err)
return
}
fmt.Println("success")
}
func main() {
test(`{"f": 1.23}`) // success
test(`{"f": "1.23"}`) // success
}
The Go Playground
Feel free to adjust UnmarshalJSON to your needs, mine is pretty strict about the spacing. Credit goes to Dave C, this was heavily inspired by his comment on another question (which also features more variations on the solution above).
Alternatively, you could pre-process the JSON data with a regular expression, but don't do this if the solution above is a viable option, it's much faster.
re := regexp.MustCompile(`(":\s*)([\d\.]+)(\s*[,}])`)
rawJsonByteArray = re.ReplaceAll(rawJsonByteArray, []byte(`$1"$2"$3`))

Unmarshaling json in Go: required field?

Is it possible to generate an error if a field was not found while parsing a JSON input using Go?
I could not find it in documentation.
Is there any tag that specifies the field as required?
There is no tag in the encoding/json package that sets a field to "required". You will either have to write your own MarshalJSON() method, or do a post check for missing fields.
To check for missing fields, you will have to use pointers in order to distinguish between missing/null and zero values:
type JsonStruct struct {
String *string
Number *float64
}
Full working example:
package main
import (
"fmt"
"encoding/json"
)
type JsonStruct struct {
String *string
Number *float64
}
var rawJson = []byte(`{
"string":"We do not provide a number"
}`)
func main() {
var s *JsonStruct
err := json.Unmarshal(rawJson, &s)
if err != nil {
panic(err)
}
if s.String == nil {
panic("String is missing or null!")
}
if s.Number == nil {
panic("Number is missing or null!")
}
fmt.Printf("String: %s Number: %f\n", *s.String, *s.Number)
}
Playground
You can also override the unmarshalling for a specific type (so a required field buried in a few json layers) without having to make the field a pointer. UnmarshalJSON is defined by the Unmarshaler interface.
type EnumItem struct {
Named
Value string
}
func (item *EnumItem) UnmarshalJSON(data []byte) (err error) {
required := struct {
Value *string `json:"value"`
}{}
all := struct {
Named
Value string `json:"value"`
}{}
err = json.Unmarshal(data, &required)
if err != nil {
return
} else if required.Value == nil {
err = fmt.Errorf("Required field for EnumItem missing")
} else {
err = json.Unmarshal(data, &all)
item.Named = all.Named
item.Value = all.Value
}
return
}
Here is another way by checking your customized tag
you can create a tag for your struct like:
type Profile struct {
Name string `yourprojectname:"required"`
Age int
}
Use reflect to check if the tag is assigned required value
func (p *Profile) Unmarshal(data []byte) error {
err := json.Unmarshal(data, p)
if err != nil {
return err
}
fields := reflect.ValueOf(p).Elem()
for i := 0; i < fields.NumField(); i++ {
yourpojectTags := fields.Type().Field(i).Tag.Get("yourprojectname")
if strings.Contains(yourpojectTags, "required") && fields.Field(i).IsZero() {
return errors.New("required field is missing")
}
}
return nil
}
And test cases are like:
func main() {
profile1 := `{"Name":"foo", "Age":20}`
profile2 := `{"Name":"", "Age":21}`
var profile Profile
err := profile.Unmarshal([]byte(profile1))
if err != nil {
log.Printf("profile1 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile1 unmarshal: %v\n", profile)
err = profile.Unmarshal([]byte(profile2))
if err != nil {
log.Printf("profile2 unmarshal error: %s\n", err.Error())
return
}
fmt.Printf("profile2 unmarshal: %v\n", profile)
}
Result:
profile1 unmarshal: {foo 20}
2009/11/10 23:00:00 profile2 unmarshal error: required field is missing
You can go to Playground to have a look at the completed code
You can just implement the Unmarshaler interface to customize how your JSON gets unmarshalled.
you can also make use of JSON schema validation.
package main
import (
"encoding/json"
"fmt"
"github.com/alecthomas/jsonschema"
"github.com/xeipuuv/gojsonschema"
)
type Bird struct {
Species string `json:"birdType"`
Description string `json:"what it does" jsonschema:"required"`
}
func main() {
var bird Bird
sc := jsonschema.Reflect(&bird)
b, _ := json.Marshal(sc)
fmt.Println(string(b))
loader := gojsonschema.NewStringLoader(string(b))
documentLoader := gojsonschema.NewStringLoader(`{"birdType": "pigeon"}`)
schema, err := gojsonschema.NewSchema(loader)
if err != nil {
panic("nop")
}
result, err := schema.Validate(documentLoader)
if err != nil {
panic("nop")
}
if result.Valid() {
fmt.Printf("The document is valid\n")
} else {
fmt.Printf("The document is not valid. see errors :\n")
for _, err := range result.Errors() {
// Err implements the ResultError interface
fmt.Printf("- %s\n", err)
}
}
}
Outputs
{"$schema":"http://json-schema.org/draft-04/schema#","$ref":"#/definitions/Bird","definitions":{"Bird":{"required":["birdType","what it does"],"properties":{"birdType":{"type":"string"},"what it does":{"type":"string"}},"additionalProperties":false,"type":"object"}}}
The document is not valid. see errors :
- (root): what it does is required
code example taken from Strict JSON parsing