How to specify default values when parsing JSON in Go - json

I want to parse a JSON object in Go, but want to specify default values for fields that are not given. For example, I have the struct type:
type Test struct {
A string
B string
C string
}
The default values for A, B, and C, are "a", "b", and "c" respectively. This means that when I parse the json:
{"A": "1", "C": 3}
I want to get the struct:
Test{A: "1", B: "b", C: "3"}
Is this possible using the built-in package encoding/json? Otherwise, is there any Go library that has this functionality?

This is possible using encoding/json: when calling json.Unmarshal, you do not need to give it an empty struct, you can give it one with default values.
For your example:
var example []byte = []byte(`{"A": "1", "C": "3"}`)
out := Test{
A: "default a",
B: "default b",
// default for C will be "", the empty value for a string
}
err := json.Unmarshal(example, &out) // <--
if err != nil {
panic(err)
}
fmt.Printf("%+v", out)
Running this example returns {A:1 B:default b C:3}.
As you can see, json.Unmarshal(example, &out) unmarshals the JSON into out, overwriting the values specified in the JSON, but leaving the other fields unchanged.

In case u have a list or map of Test structs the accepted answer is not possible anymore but it can easily be extended with a UnmarshalJSON method:
func (t *Test) UnmarshalJSON(data []byte) error {
type testAlias Test
test := &testAlias{
B: "default B",
}
err := json.Unmarshal(data, test)
if err != nil {
return err
}
*t = Test(*test)
return nil
}
The testAlias is needed to prevent recursive calls to UnmarshalJSON. This works because a new type has no methods defined.
https://play.golang.org/p/qiGyjRbNHg

You can also implement the UnmarshalJSON interface and set the default values in it.
func (test *Test) UnmarshalJSON(data []byte) error {
test.A = "1"
test.B = "2"
test.C = "3"
type tempTest Test
return json.Unmarshal(data, (*tempTest)(test))
}
The type tempTest is needed so that UnmarshalJSON is not called recursively.

The json.Unmarshal with a default value is simple and clean like the answers given by Christian and JW., but it has some downsides.
First, it strongly ties the default values of fields with the parsing logic. It's conceivable that we want to let user code down the line set its defaults; right now, the defaults have to be set before unmarshaling.
The second downside is that it only works in simple cases. If our Options struct has a slice or map of other structs, we can't populate defaults this way.
Another option is Default values with pointer fields
type Test struct {
A *string
B *string
C *string
}
js := []byte(`{"A": "1", "C": "3"}`)
var t Test
if err := json.Unmarshal(js, &t); err != nil {
fmt.Println(err)
}
if t.B == nil {
var defaultB = "B"
t.B = &defaultB
}

Related

Difference between missing and empty fields on JSON unmarshalling

So I have this struct in Go:
type Car struct {
Name string `json:"name"`
Speed int `json:"speed"`
}
And I have two JSON samples that I unmarshal:
str := `{"name": "", "speed": 0}`
strTwo := `{}`
I do the unmarshaling in this way:
car := Car{}
_ = json.Unmarshal([]byte(str), &car)
carTwo := Car{}
_ = json.Unmarshal([]byte(strTwo), &carTwo)
Now because of the way Go deals with default value types, when I try to print the structure, I get the same result:
car - { 0}
carTwo - { 0}
So I can't see the difference between a missing value in JSON and when a default value is passed. How can I solve this problem?
One way is to use pointers in struct:
type Car struct {
Name *string `json:"name"`
Speed *int `json:"speed"`
}
But I get a very ugly code when using this values, I have to do pointer dereferencing everywhere
Go's primitive data types are not suitable to handle the "all valid values" and an additional "is present" information.
If you do need this, one way is to use pointers, where the nil pointer value corresponds to the "missing" state.
If it is uncomfortable to work with pointers afterwards, do a "post processing": convert your structs with pointer fields to a struct value with non-pointer fields, so you can work with that later on.
You may do this "manually", or write a custom unmarshaler to make this happen automatically.
Here's an example how to do it:
type PCar struct {
Name *string `json:"name"`
Speed *int `json:"speed"`
}
type Car struct {
Name string `json:"-"`
Speed int `json:"-"`
PCar
}
func (c *Car) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &c.PCar); err != nil {
return err
}
if c.PCar.Name != nil {
c.Name = *c.PCar.Name
}
if c.PCar.Speed != nil {
c.Speed = *c.PCar.Speed
}
return nil
}
Example using it:
sources := []string{
`{"name": "", "speed": 0}`,
`{}`,
`{"name": "Bob", "speed": 21}`,
}
for i, src := range sources {
var c Car
if err := json.Unmarshal([]byte(src), &c); err != nil {
panic(err)
}
fmt.Println("car", i, c)
}
Output (try it on the Go Playground):
car 0 { 0 {0x40c200 0x4140ac}}
car 1 { 0 {<nil> <nil>}}
car 2 {Bob 21 {0x40c218 0x41410c}}
As you can see, car 1 contains 2 non-nil pointers, because the respective fields were present in the input JSON, while car 2 contains 2 nil pointers, because those fields were missing in its input. You may use Car.Name and Car.Speed fields as non-pointers (because they are not pointers). To tell if they were present in the input, you may check the corresponding pointers Car.PCar.Name and Car.PCar.Speed if they are nil.

Is there a way to have json.Unmarshal() select struct type based on "type" property?

I have some JSON of the form:
[{
"type": "car",
"color": "red",
"hp": 85,
"doors": 4
}, {
"type": "plane",
"color": "blue",
"engines": 3
}]
I have types car and plane that satisfy a vehicle interface; I'd like to be able to write:
var v []vehicle
e := json.Unmarshal(myJSON, &v)
... and have JSON fill my slice of vehicles with a car and a plane; instead (and unsurprisingly) I just get "cannot unmarshal object into Go value of type main.vehicle".
For reference, here are suitable definitions of the types involved:
type vehicle interface {
vehicle()
}
type car struct {
Type string
Color string
HP int
Doors int
}
func (car) vehicle() { return }
type plane struct {
Type string
Color string
Engines int
}
func (plane) vehicle() { return }
var _ vehicle = (*car)(nil)
var _ vehicle = (*plane)(nil)
(Note that I'm actually totally uninterested in the t field on car and plane - it could be omitted because this information will, if someone successfully answers this question, be implicit in the dynamic type of the objects in v.)
Is there a way to have the JSON umarhsaller choose which type to use based on some part of the contents (in this case, the type field) of the data being decoded?
(Note that this is not a duplicate of Unmarshal JSON with unknown fields because I want each item in the slice to have a different dynamic type, and from the value of the 'type' property I know exactly what fields to expect—I just don't know how to tell json.Unmarshal how to map 'type' property values onto Go types.)
Taking the answers from the similar question: Unmarshal JSON with unknown fields, we can construct a few ways to unamrshal this JSON object in a []vehicle data structure.
The "Unmarshal with Manual Handling" version can be done by using a generic []map[string]interface{} data structure, then building the correct vehicles from the slice of maps. For brevity, this example does leave out the error checking for missing or incorrectly typed fields which the json package would have done.
https://play.golang.org/p/fAY9JwVp-4
func NewVehicle(m map[string]interface{}) vehicle {
switch m["type"].(string) {
case "car":
return NewCar(m)
case "plane":
return NewPlane(m)
}
return nil
}
func NewCar(m map[string]interface{}) *car {
return &car{
Type: m["type"].(string),
Color: m["color"].(string),
HP: int(m["hp"].(float64)),
Doors: int(m["doors"].(float64)),
}
}
func NewPlane(m map[string]interface{}) *plane {
return &plane{
Type: m["type"].(string),
Color: m["color"].(string),
Engines: int(m["engines"].(float64)),
}
}
func main() {
var vehicles []vehicle
objs := []map[string]interface{}{}
err := json.Unmarshal(js, &objs)
if err != nil {
log.Fatal(err)
}
for _, obj := range objs {
vehicles = append(vehicles, NewVehicle(obj))
}
fmt.Printf("%#v\n", vehicles)
}
We could leverage the json package again to take care of the unmarshaling and type checking of the individual structs by unmarshaling a second time directly into the correct type. This could all be wrapped up into a json.Unmarshaler implementation by defining an UnmarshalJSON method on the []vehicle type to first split up the JSON objects into raw messages.
https://play.golang.org/p/zQyL0JeB3b
type Vehicles []vehicle
func (v *Vehicles) UnmarshalJSON(data []byte) error {
// this just splits up the JSON array into the raw JSON for each object
var raw []json.RawMessage
err := json.Unmarshal(data, &raw)
if err != nil {
return err
}
for _, r := range raw {
// unamrshal into a map to check the "type" field
var obj map[string]interface{}
err := json.Unmarshal(r, &obj)
if err != nil {
return err
}
vehicleType := ""
if t, ok := obj["type"].(string); ok {
vehicleType = t
}
// unmarshal again into the correct type
var actual vehicle
switch vehicleType {
case "car":
actual = &car{}
case "plane":
actual = &plane{}
}
err = json.Unmarshal(r, actual)
if err != nil {
return err
}
*v = append(*v, actual)
}
return nil
}
JSON decoding and encoding in Go is actually surprisingly well at recognizing fields inside embedded structs. E.g. decoding or encoding the following structure works when there is no overlapping fields between type A and type B:
type T struct{
Type string `json:"type"`
*A
*B
}
type A struct{
Baz int `json:"baz"`
}
type B struct{
Bar int `json:"bar"`
}
Be aware that if both "baz" and "bar" are set in the JSON for the example above, both the T.A and T.B properties will be set.
If there is overlapping fields between A and B, or just to be able to better discard invalid combinations of fields and type, you need to implement the json.Unmarshaler interface. To not have to first decode fields into a map, you can extend the trick of using embedded structs.
type TypeSwitch struct {
Type string `json:"type"`
}
type T struct {
TypeSwitch
*A
*B
}
func (t *T) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &t.TypeSwitch); err != nil {
return err
}
switch t.Type {
case "a":
t.A = &A{}
return json.Unmarshal(data, t.A)
case "b":
t.B = &B{}
return json.Unmarshal(data, t.B)
default:
return fmt.Errorf("unrecognized type value %q", t.Type)
}
}
type A struct {
Foo string `json:"bar"`
Baz int `json:"baz"`
}
type B struct {
Foo string `json:"foo"`
Bar int `json:"bar"`
}
For marshaling back, json.Marshaler must also be implemented if there is overlapping fields.
Full example: https://play.golang.org/p/UHAdxlVdFQQ
The two passes approach works fine, but there is also the option of the mapstructure package, that was created to do exactly this.
I was facing the same problem.
I'm using the lib github.com/mitchellh/mapstructure together the encoding/json.
I first, unmarshal the json to a map, and use mapstructure to convert the map to my struct, e.g.:
type (
Foo struct {
Foo string `json:"foo"`
}
Bar struct {
Bar string `json:"bar"`
}
)
func Load(jsonStr string, makeInstance func(typ string) any) (any, error) {
// json to map
m := make(map[string]any)
e := json.Unmarshal([]byte(jsonStr), &m)
if e != nil {
return nil, e
}
data := makeInstance(m["type"].(string))
// decoder to copy map values to my struct using json tags
cfg := &mapstructure.DecoderConfig{
Metadata: nil,
Result: &data,
TagName: "json",
Squash: true,
}
decoder, e := mapstructure.NewDecoder(cfg)
if e != nil {
return nil, e
}
// copy map to struct
e = decoder.Decode(m)
return data, e
}
Using:
f, _ := Load(`{"type": "Foo", "foo": "bar"}`, func(typ string) any {
switch typ {
case "Foo":
return &Foo{}
}
return nil
})
If the property is a string you can use .(string) for casting the property because the origin is an interface.
You can use it the next way:
v["type"].(string)

Extracting Elements from JSON

I have a function that returns JSON, and the return type is this:
map[string]interface{}
In my particular case, I know that the returned JSON looks like this:
{
"someNumber": 1,
"someString": "ee",
"someArray": [
"stringInArray",
{
"anotherNumber": 100,
"anotherString": "asdf",
"yetAnotherString": "qwer"
}
]
}
I want to get the value of "stringInArray" and also "anotherString". I've searched for solutions for example from Go by Example and blog.golang.org but I've not been successful.
For example, given that res is the json returned from the function call, I tried this (from the go blog):
var f interface{}
b := []byte(`res` )
err2 := json.Unmarshal(b, &f)
if err2!=nil{
log.Fatalf("Err unmarshalling: %v", err2)
}
m := f.( map[string]interface{} )
for k, v := range m {
switch vv := v.(type) {
case string:
fmt.Println(k, "is string", vv)
case int:
fmt.Println(k, "is int", vv)
case []interface{}:
fmt.Println(k, "is an array:")
for i, u := range vv {
fmt.Println(i, u)
}
default:
fmt.Println(k, "is of a type I don't know how to handle")
}
}
(I know the above would not do exactly what I want but it is a start.)
But when code execution hits b := []byte(res ), I get this error:
Err unmarshalling: invalid character 'r' looking for beginning of value
What is the most efficient method / best practice for obtaining the values? I'm open to any solution, not necessarily the approach above.
#sydnash Here is the code I promised in my response to your comment:
type LoginResponse2 struct {
id float64
jsonrpc string
result struct {
token string
details struct {
a float64
b string
c float64
}
}
}
var resStruct2 LoginResponse2
// Convert result to json
b, _ :=json.Marshal(res)
fmt.Println( string(b) )
// results of Println:
{"id":1,"jsonrpc":"2.0","result":[ "myToken",{"a":someNumber,"b":"some string","c":someOtherNumber} ] }
// Unmarshall json into struct
err2 := json.Unmarshal(b, &resStruct2)
if err2 != nil {
log.Fatalf("Error unmarshalling json: %v", err)
}
fmt.Println("Here is json unmarshalled into a struct")
fmt.Println( resStruct2 )
// results of Println
{0 { {0 0}}}
I think you should use b := []byte(res) instead b :=[]byte[res] and res should be a json string or []byte. res is not a legal json format.
this information maybe help you:
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
you can see there is no int but float64 for JSON numbers.
There is an example of how to unmarshal json in the docs on golang packages encoding/json. Essentially your problem is that the un-marshal expects to be putting the json into type. So your code must declare a type that represents the json you are converting from.
Example of the json and the declared type from the linked example:
var jsonBlob = []byte(`[
{"Name": "Platypus", "Order": "Monotremata"},
{"Name": "Quoll", "Order": "Dasyuromorphia"}
]`)
type Animal struct {
Name string
Order string
}
Once you define the type it should all fall into place.

How can I implement UnmarshalJSON for a type derived from a scalar in Go?

I have a simple type that implements conversion of subtyped integer consts to strings and vice versa in Go. I want to be able to automatically unmarshal strings in JSON to values of this type. I can't, because UnmarshalJSON doesn't give me a way to return or modify the scalar value. It's expecting a struct, whose members are set by UnmarshalJSON. The ",string" method doesn't work either for other than builtin scalar types. Is there a way to implement UnmarshalJSON correctly for a derived scalar type?
Here's an example of what I'm after. I want it to print "Hello Ralph" four times, but it prints "Hello Bob" four times because the PersonID isn't being changed.
package main
import (
"encoding/json"
"fmt"
)
type PersonID int
const (
Bob PersonID = iota
Jane
Ralph
Nobody = -1
)
var nameMap = map[string]PersonID{
"Bob": Bob,
"Jane": Jane,
"Ralph": Ralph,
"Nobody": Nobody,
}
var idMap = map[PersonID]string{
Bob: "Bob",
Jane: "Jane",
Ralph: "Ralph",
Nobody: "Nobody",
}
func (intValue PersonID) Name() string {
return idMap[intValue]
}
func Lookup(name string) PersonID {
return nameMap[name]
}
func (intValue PersonID) UnmarshalJSON(data []byte) error {
// The following line is not correct
intValue = Lookup(string(data))
return nil
}
type MyType struct {
Person PersonID `json: "person"`
Count int `json: "count"`
Greeting string `json: "greeting"`
}
func main() {
var m MyType
if err := json.Unmarshal([]byte(`{"person": "Ralph", "count": 4, "greeting": "Hello"}`), &m); err != nil {
fmt.Println(err)
} else {
for i := 0; i < m.Count; i++ {
fmt.Println(m.Greeting, m.Person.Name())
}
}
}
Use a pointer receiver for the unmarshal method. If a value receiver is used, changes to the receiver are lost when the method returns.
The argument to the unmarshal method is JSON text. Unmarshal the JSON text to get a plain string with all JSON quoting removed.
func (intValue *PersonID) UnmarshalJSON(data []byte) error {
var s string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
*intValue = Lookup(s)
return nil
}
There's a mismatch between the JSON tags an the example JSON. I changed the JSON to match the tag, but you can change it the other way.
if err := json.Unmarshal([]byte(`{"person": "Ralph", "count": 4, "greeting": "Hello"}`), &m); err != nil {
playground example
Here's an answer based on my comment. I'm not sure this is exactly what you want to do as some of your questions wording confuses me however the basic idea is to separate unmarshalling and transformation into two different steps. First unmarshal raw data into a compatible type, after do a transformation to another type or enrich the type you already have like in the example below. You're welcome to hide this behavior in a custom implementation of UnmarshalJSON if you'd like but I would personally advise against it. Here's my two reasons; 1) it's just not consistent with Go's explicit verbose coding style 2) I despise highly obfuscated packages/libraries/languages that do stuff like this for you because sooner or later it bites you in the ass an costs you a lot more than adding that 1 line of extra code in a few places (like hours trying to debug something that makes no sense to you).
type MyType struct {
Id PersonID
Name string `json: "name"`
Count int `json: "count"`
Greeting string `json: "greeting"`
}
func main() {
var m MyType
if err := json.Unmarshal([]byte(`{"name": "Ralph", "count": 4, "greeting": "Hello"}`), &m); err != nil {
fmt.Println(err)
} else {
m.Id = Lookup(m.Name) // see this isn't unmarshalling
// better to take the data as is and do transformation separate
for i := 0; i < m.Count; i++ {
fmt.Println(m.Greeting, m.Person.Name())
}
}
}

Unmarshalling JSON string into a constant

In my Go code I have the following situation:
type Operator int
const (
UNKNOWN Operator = iota
EQUALS
CONTAINS
BETWEEN
DISTANCE
)
type Filter struct {
Field string `json:"field"`
Operator Operator `json:"operator"`
Values []string `json:"values"`
}
My expected JSON would look like the following:
{
"operator": "EQUALS",
"field": "name",
"values": [ "John", "Doe" ]
}
Can I create a mapping such that json.Unmarshal will set the right operator constant in the Filter struct?
I am aware of the Unmarshaler interface but I don't think this can really be used on a constant value..
I would really like to keep the constants in my go code since this nicely enforces type checking and consistency when I pass them around.
The encoding/json Unmarshaler interface will work with the Operator type, but it must have a pointer as its receiver:
func (o *Operator) UnmarshalJSON(b []byte) error {
str := strings.Trim(string(b), `"`)
switch {
case str == "CONTAINS":
*o = CONTAINS
default:
*o = UNKNOWN
// or return an error...
}
return nil
}
The JSON decoder will take the address of the Operator field from the Filter struct and invoke the UnmarshalJSON method on it.
Note that you could implement the more generic encoding/TextUnmarshaler instead, by changing UnmarshalJSON to UnmarshalText above.
Here is a playground example: http://play.golang.org/p/szcnC6L86u
Arguably it might be simpler to use a string base type for Operator instead: http://play.golang.org/p/FCCg1NOeYw
You can wrap your object in an Unmarshaler.
ex:
package main
import (
"encoding/json"
"fmt"
)
type Operator int
const (
UNKNOWN Operator = iota
EQUALS
CONTAINS
BETWEEN
DISTANCE
)
type Filter struct {
Field string `json:"field"`
RawOperator string `json:"operator"`
Operator Operator `json:"-"`
Values []string `json:"values"`
}
type FilterUnmarshaler struct {
Filter
}
func (f *FilterUnmarshaler) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &f.Filter); err != nil {
return err
}
switch f.RawOperator {
case "UNKOWN":
f.Operator = UNKNOWN
case "EQUALS":
f.Operator = EQUALS
case "CONTAINS":
f.Operator = CONTAINS
case "BETWEEN":
f.Operator = BETWEEN
case "DISTANCE":
f.Operator = DISTANCE
default:
return fmt.Errorf("Unkown operator %s", f.RawOperator)
}
return nil
}
func main() {
rawJson := []byte(`
{
"operator": "BETWEEN",
"field": "name",
"values": [ "John", "Doe" ]
}
`)
val := &FilterUnmarshaler{}
if err := json.Unmarshal(rawJson, val); err != nil {
panic(err)
}
fmt.Printf("%#v\n", val)
}
Without using a wrapper, I didn't find how not to fall in an infinite recursion.
This example shows how to unmarshal but you most likely want to apply it to Marshal as well in order to have a correct json with Operator strings from your object with integer operator.
You might also want to create a map instead of const. Or both, create a map and manually populate it with your const and their string equivalent.