Go wrapping json.Number with sql.NullFloat64 - json

I'm working with an API that has json values that can be a number or string and can sometimes be an empty string. For example:
[
{
"Description": "Doorknob",
"Amount": 3.25
},
{
"Description": "Light bulb",
"Amount": "4.70"
},
{
"Description": "Screwdriver",
"Amount": ""
}
]
I've learned I can use json.Number to deal with cases where the value can be a number or a string. But how do I deal with the cases where it's an empty string?
I need to insert these values into a postgres database so I'd like cases where it's an empty string to show as null and I'm thinking sql.NullFloat64 would be best suited to handle it.
Here's a function I'm using that works on the first 2 cases (Doorknob and Light bulb), but fails to unmarshal the last ("Screwdriver").
Here is an unmarshaling function I'm trying to use, but I can't figure out why it's not working:
type NullNumber struct{ sql.NullFloat64 }
func (nn *NullNumber) UnmarshalJSON(data []byte) error {
var x *json.Number
if err := json.Unmarshal(data, &x); err != nil {
return err
}
if len(*x) == 0 {
nn.Valid = false // This doesn't seem to be working. Why?
}
this, err := x.Float64()
if err != nil {
return err
}
nn.Valid = true
nn.Float64 = this
return nil
}
Is this a case where I should be using an open interface for the amount?
Any help is very appreciated.
playground: https://play.golang.org/p/QYQRq94OtV3

You're almost there! json.Unmarshal call errors when a json.Number is given an empty string.
Add this check beforehand:
// edge case: json.Number is given an empty string
if bytes.Equal(data, []byte(`""`)) {
nn.Valid = false
return nil
}
https://play.golang.org/p/ILxC8tjYI_G

Related

Building a struct with a field name which has a comma

I'm trying to create a struct based on a response I get.
I have no control over the response, and in it's structure there are field names which use comma as part of the filed name itself
JSON Example:
"date": "2022-09-09 00:00:00 UTC",
"Sum": {
"Change, %": "0.10",
"Price": "254",
"Value, $": "455.26",
}
When trying to create a struct the "regular" way, I get an error since once I use the comma character, reflect.StructTag.Get expects something specific and not the rest of the name.
Struct Example:
Date string `json:"date"`
Sum struct {
Change string `json:"Change, %"`
Price string `json:"Price"`
Value string `json:"Value, $"`
} `json:"Sum"`
The error I receive is:
"struct field tag json:"Value, $" not compatible with reflect.StructTag.Get: suspicious space in struct tag value"
The goal is to later manipulate and use this data (hopefully using my simple names), so "just" printing the JSON or moving it to an array or similar option cannot be the final step.
The 'Price' field is working as expected, but the problem is with the fields using the comma.
I couldn't find a way around it or a similar question.
Would love to understand how to deal with this issue.
Thanks!
You may unmarshal the Sum JSON object into a Go map, and assign the values from the map to the struct fields. The restriction only applies to struct tags (that they may not contain arbitrary characters like comma), but maps may hold any keys.
For example:
type Sum struct {
Change string
Price string
Value string
}
func (s *Sum) UnmarshalJSON(data []byte) error {
var m map[string]string
if err := json.Unmarshal(data, &m); err != nil {
return err
}
s.Change = m["Change, %"]
s.Price = m["Price"]
s.Value = m["Value, $"]
return nil
}
type model struct {
Date string `json:"date"`
Sum Sum `json:"Sum"`
}
func main() {
var m model
if err := json.Unmarshal([]byte(src), &m); err != nil {
panic(err)
}
fmt.Printf("%+v\n", m)
}
const src = `{
"date": "2022-09-09 00:00:00 UTC",
"Sum": {
"Change, %": "0.10",
"Price": "254",
"Value, $": "455.26"
}
}`
This will output (try it on the Go Playground):
{Date:2022-09-09 00:00:00 UTC Sum:{Change:0.10 Price:254 Value:455.26}}

How to check if a json matches a struct / struct fields

Is there an easy way to check if each field of myStruct was mapped by using json.Unmarshal(jsonData, &myStruct).
The only way I could image is to define each field of a struct as pointer, otherwise you will always get back an initialized struct.
So every jsonString that is an object (even an empty one {}) will return an initialized struct and you cannot tell if the json represented your struct.
The only solution I could think of is quite uncomfortable:
package main
import (
"encoding/json"
"fmt"
)
type Person struct {
Name *string `json:name`
Age *int `json:age`
Male *bool `json:male`
}
func main() {
var p *Person
err := json.Unmarshal([]byte("{}"), &p)
// handle parse error
if err != nil {
return
}
// handle json did not match error
if p.Name == nil || p.Age == nil || p.Male == nil {
return
}
// now use the fields with dereferencing and hope you did not forget a nil check
fmt.Println("Hello " + *p.Name)
}
Maybe one could use a library like govalidator and use SetFieldsRequiredByDefault. But then you still have to execute the validation and still you are left with the whole pointer dereferencing for value retrieval and the risk of nil pointer.
What I would like is a function that returns my unmarshaled json as a struct or an error if the fields did not match. The only thing the golang json library offers is an option to fail on unknown fields but not to fail on missing fields.
Any idea?
Another way would be to implement your own json.Unmarshaler which uses reflection (similar to the default json unmarshaler):
There are a few points to consider:
if speed is of great importance to you then you should write a benchmark to see how big the impact of the extra reflection is. I suspect its negligible but it can't hurt to write a small go benchmark to get some numbers.
the stdlib will unmarshal all numbers in your json input into floats. So if you use reflection to set integer fields then you need to provide the corresponding conversion yourself (see TODO in example below)
the json.Decoder.DisallowUnknownFields function will not work as expected with your type. You need to implement this yourself (see example below)
if you decide to take this approach you will make your code more complex and thus harder to understand and maintain. Are you actually sure you must know if fields are omitted? Maybe you can refactor your fields to make good usage of the zero values?
Here a fully executable test of this approach:
package sandbox
import (
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
)
type Person struct {
Name string
City string
}
func (p *Person) UnmarshalJSON(data []byte) error {
var m map[string]interface{}
err := json.Unmarshal(data, &m)
if err != nil {
return err
}
v := reflect.ValueOf(p).Elem()
t := v.Type()
var missing []string
for i := 0; i < t.NumField(); i++ {
field := t.Field(i)
val, ok := m[field.Name]
delete(m, field.Name)
if !ok {
missing = append(missing, field.Name)
continue
}
switch field.Type.Kind() {
// TODO: if the field is an integer you need to transform the val from float
default:
v.Field(i).Set(reflect.ValueOf(val))
}
}
if len(missing) > 0 {
return errors.New("missing fields: " + strings.Join(missing, ", "))
}
if len(m) > 0 {
extra := make([]string, 0, len(m))
for field := range m {
extra = append(extra, field)
}
// TODO: consider sorting the output to get deterministic errors:
// sort.Strings(extra)
return errors.New("unknown fields: " + strings.Join(extra, ", "))
}
return nil
}
func TestJSONDecoder(t *testing.T) {
cases := map[string]struct {
in string
err string
expected Person
}{
"Empty object": {
in: `{}`,
err: "missing fields: Name, City",
expected: Person{},
},
"Name missing": {
in: `{"City": "Berlin"}`,
err: "missing fields: Name",
expected: Person{City: "Berlin"},
},
"Age missing": {
in: `{"Name": "Friedrich"}`,
err: "missing fields: City",
expected: Person{Name: "Friedrich"},
},
"Unknown field": {
in: `{"Name": "Friedrich", "City": "Berlin", "Test": true}`,
err: "unknown fields: Test",
expected: Person{Name: "Friedrich", City: "Berlin"},
},
"OK": {
in: `{"Name": "Friedrich", "City": "Berlin"}`,
expected: Person{Name: "Friedrich", City: "Berlin"},
},
}
for name, c := range cases {
t.Run(name, func(t *testing.T) {
var actual Person
r := strings.NewReader(c.in)
err := json.NewDecoder(r).Decode(&actual)
switch {
case err != nil && c.err == "":
t.Errorf("Expected no error but go %v", err)
case err == nil && c.err != "":
t.Errorf("Did not return expected error %v", c.err)
case err != nil && err.Error() != c.err:
t.Errorf("Expected error %q but got %v", c.err, err)
}
if !reflect.DeepEqual(c.expected, actual) {
t.Errorf("\nWant: %+v\nGot: %+v", c.expected, actual)
}
})
}
}
You could compare p with a empty struct, instead of comparing each field with nil.
// handle json did not match error
if p == Person{} {
return
}
Since Person{} will initialize with the 0 value of each field, this will result in each property that is pointers to be nil, strings will be "", ints will be 0, and so on.

How to decode complicated unnamed JSON in Golang

I am currently trying to decode the following JSON structure:
[
{
"2015-08-14 19:29:48-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1a",
"UTC_time": "2015-08-14 23:29:48",
"error": "0"
}
},
{
"2015-08-14 19:37:07-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1b",
"UTC_time": "2015-08-14 23:37:07",
"error": "0"
}
},
{
"2015-08-14 19:44:16-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1b",
"UTC_time": "2015-08-14 23:44:16",
"error": "0"
}
}
]
This is to eventually have a slice of Reading structs, formatted as the following:
type reading struct {
Value string `json:"value"`
MTID string `json:"measurement_tag_id"`
UTCTime string `json:"UTC_time"`
Error string `json:"error"`
}
I would then like to add this into an existing structure nested as:
type site struct {
Name string
ID string
Tags []tag
}
type tag struct {
ID string
Readings []reading
}
I've currently been able to create the base structure for sites and tags from a more typical JSON payload with appropriate keys. I have been unsuccessful though in figuring out how to decode the reading JSON. So far the closest I have gotten is via map[string]interface{} chaining, but this feels incredibly clunky and verbose.
Solution so far for reference:
var readingData []interface{}
if err := json.Unmarshal(file, &readingData); err != nil {
panic(err)
}
readings := readingData[0].(map[string]interface{})
firstReading := readings["2015-08-14 19:29:48-04:00"].(map[string]interface{})
fmt.Println(firstReading)
value := firstReading["value"].(string)
error := firstReading["error"].(string)
MTID := firstReading["measurement_tag_id"].(string)
UTCTime := firstReading["UTC_time"].(string)
fmt.Println(value, error, MTID, UTCTime)
While I am not sure if its necessary yet, I would also like to hold on to the arbitrary date keys. My first thought was to create a function that returned a map[string]reading but I am not sure how feasible this is.
Thanks for the help in advance!
You can have your reading type implement the json.Unmarshaler interface.
func (r *reading) UnmarshalJSON(data []byte) error {
type _r reading // same structure, but no methods, avoids infinite calls to this method
m := map[string]_r{}
if err := json.Unmarshal(data, &m); err != nil {
return err
}
for _, v := range m {
*r = reading(v)
}
return nil
}
https://play.golang.org/p/7X1oB77XL4
Another way is to use a slice of maps to parse, then copy to slice of readings, eg:
var readingMaps []map[string]reading //slice of maps of string key to reading value
if err := json.Unmarshal([]byte(data), &readingMaps); err != nil {
panic(err)
}
readings := []reading{}
for _, m := range readingMaps {
for _, r := range m {
readings = append(readings, r)
}
}
play.golang.org/p/jXTdmaZz7s

How to tell the client they need to send an integer instead of a string, from a Go server?

Let's say I have the following Go struct on the server
type account struct {
Name string
Balance int
}
I want to call json.Decode on the incoming request to parse it into an account.
var ac account
err := json.NewDecoder(r.Body).Decode(&ac)
If the client sends the following request:
{
"name": "test#example.com",
"balance": "3"
}
Decode() will return the following error:
json: cannot unmarshal string into Go value of type int
Now it's possible to parse that back into "you sent a string for Balance, but you really should have sent an integer", but it's tricky, because you don't know the field name. It also gets a lot trickier if you have a lot of fields in the request - you don't know which one failed to parse.
What's the best way to take that incoming request, in Go, and return the error message, "Balance must be a string", for any arbitrary number of integer fields on a request?
You can use a custom type with custom unmarshaling algorythm for your "Balance" field.
Now there are two possibilities:
Handle both types:
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type Int int
type account struct {
Name string
Balance Int
}
func (i *Int) UnmarshalJSON(b []byte) (err error) {
var s string
err = json.Unmarshal(b, &s)
if err == nil {
var n int
n, err = strconv.Atoi(s)
if err != nil {
return
}
*i = Int(n)
return
}
var n int
err = json.Unmarshal(b, &n)
if err == nil {
*i = Int(n)
}
return
}
func main() {
for _, in := range [...]string{
`{"Name": "foo", "Balance": 42}`,
`{"Name": "foo", "Balance": "111"}`,
} {
var a account
err := json.Unmarshal([]byte(in), &a)
if err != nil {
fmt.Printf("Error decoding JSON: %v\n", err)
} else {
fmt.Printf("Decoded OK: %v\n", a)
}
}
}
Playground link.
Handle only a numeric type, and fail anything else with a sensible error:
package main
import (
"encoding/json"
"fmt"
)
type Int int
type account struct {
Name string
Balance Int
}
type FormatError struct {
Want string
Got string
Offset int64
}
func (fe *FormatError) Error() string {
return fmt.Sprintf("Invalid data format at %d: want: %s, got: %s",
fe.Offset, fe.Want, fe.Got)
}
func (i *Int) UnmarshalJSON(b []byte) (err error) {
var n int
err = json.Unmarshal(b, &n)
if err == nil {
*i = Int(n)
return
}
if ute, ok := err.(*json.UnmarshalTypeError); ok {
err = &FormatError{
Want: "number",
Got: ute.Value,
Offset: ute.Offset,
}
}
return
}
func main() {
for _, in := range [...]string{
`{"Name": "foo", "Balance": 42}`,
`{"Name": "foo", "Balance": "111"}`,
} {
var a account
err := json.Unmarshal([]byte(in), &a)
if err != nil {
fmt.Printf("Error decoding JSON: %#v\n", err)
fmt.Printf("Error decoding JSON: %v\n", err)
} else {
fmt.Printf("Decoded OK: %v\n", a)
}
}
}
Playground link.
There is a third possibility: write custom unmarshaler for the whole account type, but it requires more involved code because you'd need to actually iterate over the input JSON data using the methods of the
encoding/json.Decoder type.
After reading your
What's the best way to take that incoming request, in Go, and return the error message, "Balance must be a string", for any arbitrary number of integer fields on a request?
more carefully, I admit having a custom parser for the whole type is the only sensible possibility unless you are OK with a 3rd-party package implementing a parser supporting validation via JSON schema (I think I'd look at this first as juju is a quite established product).
A solution for this could be to use a type assertion by using a map to unmarshal the JSON data into:
type account struct {
Name string
Balance int
}
var str = `{
"name": "test#example.com",
"balance": "3"
}`
func main() {
var testing = map[string]interface{}{}
err := json.Unmarshal([]byte(str), &testing)
if err != nil {
fmt.Println(err)
}
val, ok := testing["balance"]
if !ok {
fmt.Println("missing field balance")
return
}
nv, ok := val.(float64)
if !ok {
fmt.Println("balance should be a number")
return
}
fmt.Printf("%+v\n", nv)
}
See http://play.golang.org/p/iV7Qa1RrQZ
The type assertion here is done using float64 because it is the default number type supported by Go's JSON decoder.
It should be noted that this use of interface{} is probably not worth the trouble.
The UnmarshalTypeError (https://golang.org/pkg/encoding/json/#UnmarshalFieldError) contains an Offset field that could allow retrieving the contents of the JSON data that triggered the error.
You could for example return a message of the sort:
cannot unmarshal string into Go value of type int near `"balance": "3"`
It would seem that here provides an implementation to work around this issue in Go only.
type account struct {
Name string
Balance int `json:",string"`
}
In my estimation the more correct and sustainable approach is for you to create a client library in something like JavaScript and publish it into the NPM registry for others to use (private repository would work the same way). By providing this library you can tailor the API for the consumers in a meaningful way and prevent errors creeping into your main program.

Serialize a map using a specific order

I have a map that uses string for both key and value. I have an array of keys that specifies the order of the values of the map.
I want to serialize that map to a JSON, but keeping the order defined on the array.
There is a sample code here: http://play.golang.org/p/A52GTDY6Wx
I want to serialize it as:
{
"name": "John",
"age": "20"
}
But if I serialize the map directly, the keys are ordered alphabetically:
{
"age": "20",
"name": "John"
}
I can serialize it as an array of maps, thus keeping the order, however that generates a lot of undesired characters:
[
{
"name": "John"
},
{
"age": "20"
}
]
In my real code I need to serialize the results of a database query which is specified in a text file, and I need to maintain the column order. I cannot use structs because the columns are not known at compile time.
EDIT: I don't need to read the JSON later in the specified order. The generated JSON is meant to be read by people, so I just want it to be as humanly readable as possible.
I could use a custom format but JSON suits me perfectly for this.
Thanks!
You need to implement the json.Marshaler interface on a custom type. This has the advantage of playing well within other struct types.
Sorry, you're always going to have to write a little bit of JSON encoding code.
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
)
type KeyVal struct {
Key string
Val interface{}
}
// Define an ordered map
type OrderedMap []KeyVal
// Implement the json.Marshaler interface
func (omap OrderedMap) MarshalJSON() ([]byte, error) {
var buf bytes.Buffer
buf.WriteString("{")
for i, kv := range omap {
if i != 0 {
buf.WriteString(",")
}
// marshal key
key, err := json.Marshal(kv.Key)
if err != nil {
return nil, err
}
buf.Write(key)
buf.WriteString(":")
// marshal value
val, err := json.Marshal(kv.Val)
if err != nil {
return nil, err
}
buf.Write(val)
}
buf.WriteString("}")
return buf.Bytes(), nil
}
func main() {
dict := map[string]interface{}{
"orderedMap": OrderedMap{
{"name", "John"},
{"age", 20},
},
}
dump, err := json.Marshal(dict)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", dump)
}
Outputs
{"orderedMap":{"name":"John","age":20}}
For that specific requirement you really don't need to use json.Marshal at all, you can simply implement your own function like this:
type OrderedMap map[string]string
func (om OrderedMap) ToJson(order ...string) string {
buf := &bytes.Buffer{}
buf.Write([]byte{'{', '\n'})
l := len(order)
for i, k := range order {
fmt.Fprintf(buf, "\t\"%s\": \"%v\"", k, om[k])
if i < l-1 {
buf.WriteByte(',')
}
buf.WriteByte('\n')
}
buf.Write([]byte{'}', '\n'})
return buf.String()
}
func main() {
om := OrderedMap{
"age": "20",
"name": "John",
}
fmt.Println(om.ToJson("name", "age"))
}
Probably the easiest solution: https://github.com/iancoleman/orderedmap
Although it might be slow as it's mentioned here
Here is a MapSlice implementation similar to what YAML v2 offers. It can do both Marshal and Unmarshal.
https://github.com/mickep76/mapslice-json