I wish to be able to use .Decode() on a response body to populate a struct without first having to attempt to figure out which type of struct I should decode to.
I have a generic struct Match to hold information about a game that was played e.g. a match in Fortnite. Within this struct, I use MatchData to hold the entirety of the game's match data.
When decoding into the MatchData struct, I'm finding the underlying embedded type is initialised, but with all default values, and not the values from the respose.
type Match struct {
MatchID int `json:"match_id"`
GameType int `json:"game_type"`
MatchData *MatchData `json:"match_data"`
}
type MatchData struct {
MatchGame1
MatchGame2
}
type MatchGame1 struct {
X int `json:"x"`
Y int `json:"y"`
}
type MatchGame2 struct {
X int `json:"x"`
Y int `json:"y"`
}
func populateData(m *Match) (Match, error) {
response, err := http.Get("game1.com/path")
if err != nil {
return nil, err
}
// Here, m.MatchData is set with X and Y equal to 0
// when response contains values > 0
err = json.NewDecoder(response.Body).Decode(&m.MatchData)
if err != nil {
return nil, err
}
return m, nil
}
Edit
Example expected JSON payload.
{
"x": 10,
"y": 20
}
I can solve the issue by checking m.GameType, creating a struct that corresponds and then assigning it to m.MatchData, but if I wanted to add another 100 game APIs, I'd prefer if the function could be agnostic of it.
I'm not sure if this is even possible, but thanks in advance.
The approach in the question will not work because the embedded structs share field names. Try this approach.
Declare a map that associates game type identifiers with the associated Go types. This is only code related to decoding that knows about the hundreds of game types.
var gameTypes = map[int]reflect.Type{
1: reflect.TypeOf(&MatchGame1{}),
2: reflect.TypeOf(&MatchGame2{}),
}
Decode the match data to a raw message. Use the game type to create a match data value and decode to that value.
func decodeMatch(r io.Reader) (*Match, error) {
// Start with match data set to a raw messae.
var raw json.RawMessage
m := &Match{MatchData: &raw}
err := json.NewDecoder(r).Decode(m)
if err != nil {
return nil, err
}
m.MatchData = nil
// We are done if there's no raw message.
if len(raw) == 0 {
return m, nil
}
// Create the required match data value.
t := gameTypes[m.GameType]
if t == nil {
return nil, errors.New("unknown game type")
}
m.MatchData = reflect.New(t.Elem()).Interface()
// Decode the raw message to the match data.
return m, json.Unmarshal(raw, m.MatchData)
}
Run it on the playground.
Related
I thought I understood unmarshalling by now, but I guess not. I'm having a little bit of trouble unmarshalling a map in go. Here is the code that I have so far
type OHLC_RESS struct {
Pair map[string][]Candles
Last int64 `json:"last"`
}
type Candles struct {
Time uint64
Open string
High string
Low string
Close string
VWAP string
Volume string
Count int
}
func (c *Candles) UnmarshalJSON(d []byte) error {
tmp := []interface{}{&c.Time, &c.Open, &c.High, &c.Low, &c.Close, &c.VWAP, &c.Volume, &c.Count}
length := len(tmp)
err := json.Unmarshal(d, &tmp)
if err != nil {
return err
}
g := len(tmp)
if g != length {
return fmt.Errorf("Lengths don't match: %d != %d", g, length)
}
return nil
}
func main() {
response := []byte(`{"XXBTZUSD":[[1616662740,"52591.9","52599.9","52591.8","52599.9","52599.1","0.11091626",5],[1616662740,"52591.9","52599.9","52591.8","52599.9","52599.1","0.11091626",5]],"last":15}`)
var resp OHLC_RESS
err := json.Unmarshal(response, &resp)
fmt.Println("resp: ", resp)
}
after running the code, the last field will unmarshal fine, but for whatever reason, the map is left without any value. Any help?
The expedient solution, for the specific example JSON, would be to NOT use a map at all but instead change the structure of OHLC_RESS so that it matches the structure of the JSON, i.e.
type OHLC_RESS struct {
Pair []Candles `json:"XXBTZUSD"`
Last int64 `json:"last"`
}
https://go.dev/play/p/Z9PhJt3wX33
However it's safe to assume, I think, that the reason you've opted to use a map is because the JSON object's key(s) that hold the "pairs" can vary and so hardcoding them into the field's tag is out of the question.
To understand why your code doesn't produce the desired result, you have to realize two things. First, the order of a struct's fields has no bearing on how the keys of a JSON object will be decoded. Second, the name Pair holds no special meaning for the unmarshaler. Therefore, by default, the unmarshaler has no way of knowing that your wish is to decode the "XXBTZUSD": [ ... ] element into the Pair map.
So, to get your desired result, you can have the OHLC_RESS implement the json.Unmarshaler interface and do the following:
func (r *OHLC_RESS) UnmarshalJSON(d []byte) error {
// first, decode just the object's keys and leave
// the values as raw, non-decoded JSON
var obj map[string]json.RawMessage
if err := json.Unmarshal(d, &obj); err != nil {
return err
}
// next, look up the "last" element's raw, non-decoded value
// and, if it is present, then decode it into the Last field
if last, ok := obj["last"]; ok {
if err := json.Unmarshal(last, &r.Last); err != nil {
return err
}
// remove the element so it's not in
// the way when decoding the rest below
delete(obj, "last")
}
// finally, decode the rest of the element values
// in the object and store them in the Pair field
r.Pair = make(map[string][]Candles, len(obj))
for key, val := range obj {
cc := []Candles{}
if err := json.Unmarshal(val, &cc); err != nil {
return err
}
r.Pair[key] = cc
}
return nil
}
https://go.dev/play/p/Lj8a8Gx9fWH
This question already has answers here:
How to specify default values when parsing JSON in Go
(4 answers)
Closed 3 years ago.
I want to set a default value to a field when unmarshal from a json string.
I know i can set the value i want before unmarshaling, I think it's not a beautiful way.
Is there any way else, like using a "default" tag?
func main() {
in := "{}"
myStruct := StructTest{}
json.Unmarshal([]byte(in), &myStruct)
fmt.Println(myStruct)
}
type StructTest struct {
V int64 `default:1`
}
What you can do is define a custom unmarshal function and decide there if you want to use the default value or not. In case you have other fields in StructTest you will want to create an alias for StructTest in UnmarshalJSON so that other fields will still be treated the same, while V will be overridden.
The snippet below shows one way to do it, also check out this working Go playground example.
type StructTest struct {
V int64
Other string // this field should be unmarshaled the regular way
}
func (st *StructTest) UnmarshalJSON(data []byte) error {
// create alias to prevent endless loop
type Alias StructTest
tmp := struct {
*Alias
V *int64
}{
Alias: (*Alias)(st),
}
// unmarshal into temporary struct
err := json.Unmarshal(data, &tmp)
if err != nil {
return err
}
// check if V was supplied in JSON and set default value if it wasn't
if tmp.V == nil {
st.V = 1 // default
} else {
st.V = *tmp.V
}
return nil
}
EDIT:
Actually for this simple example it can be done even simpler:
func (st *StructTest) UnmarshalJSON(data []byte) error {
st.V = 1 // set default value before unmarshaling
type Alias StructTest // create alias to prevent endless loop
tmp := (*Alias)(st)
return json.Unmarshal(data, tmp)
}
The short answer is no, and for cases like this:
type T struct {
Field1 int
Field2 int
}
...
func foo(data []byte) error {
var x T
if err := json.Unmarshal(data, &x); err != nil {
return err
}
// now set Field1 and/or Field2 to default values if needed
// ... but this is hard to do ...
use(x)
return nil
}
it's simple and easy to do it the other way, i.e.:
func foo(data []byte) error {
x := T{Field1: 99 /* red ballons */, Field2: 42 /* The Answer */ }
if err := json.Unmarshal(data, &x); err != nil {
return err
}
// now Field1 and/or Field2 are already set to default values if needed
use(x)
return nil
}
But what if the default is hard to compute? For instance:
type T2 struct {
Answer int
Question string
}
and function foo should have a default Answer of 42 as before, but the default Question should be the one that the mice were trying to compute, and obviously we don't want to spend a few millenia computing it if we don't have to. So we can't pre-initialize x, and we need to know if a question was provided.1
Yet another alternative, of course, is to decode into a struct with a pointer, then convert that nullable thing to the struct that doesn't have a pointer; we know whether the nullable variant's field was filled in or not because it's non-nil if it was filled-in. This produces code of this sort:
type T_ struct {
Answer int
Question *string
}
and fill in one variable x_ of type T_:
func foo(data []byte) error {
var x T2
x_ := T_{Answer: 42}
if err := json.Unmarshal(data, &x_); err != nil {
return err
}
x.Answer = x_.Answer
if x_.Question = nil {
x.Question = computeTheQuestion(x_.Answer)
} else {
x.Question = *x_.Question
}
use(x)
return nil
}
However, once again I lament (at least slightly) the ability to unmarshal json data with code like this hypothetical interface:
func foo(data []byte) error {
var objectish map[string]interface{}
if err := json.Unmarshal(data, &objectish); err != nil {
return err
}
x := T2{Answer: 42}
if err := json.ReUnmarshal(objectish, &x); err != nil {
return err
}
// We now know that the object-ish decoded JSON has the right
// "shape" for a T, and variable x is filled in with a default
// Answer. Its Question is the empty string if there was an
// empty Question, or if there was *no* Question at all, so
// now let's find out if we need to compute the right Question.
if _, ok := objectish["Question"]; !ok {
x.Question = computeTheQuestion(x.Answer)
}
use(x) // pass x to a hoopy frood
return nil
}
This hypothetical ReUnmarshal—which could actually just be Unmarshal itself, really—would, if given an interface{}, treat its value as resulting from an earlier Unmarshal and just re-type the result. If given a map[string]interface{} it would re-type the object. If given a map[string]map[string]interface{} it would do the obvious thing here as well, and so on. The only change the user sees is that Unmarshal (or ReUnmarshal, if changing the type signature is too rude) now takes:
[]byte, or
string, probably, just for convenience, or
json.RawMessage, because that already does almost the right thing, or
map[string]T for T being any of the types it accepts (including map recursively)
and it does the obvious thing with each of these.
Note that this is pretty similar to using json.RawMessage. However, when using json.RawMessage, we're right back to writing out a variant of the original struct type, with the same field names. See this example in the Go Playground, where we have to declare x_ (rather than objectish) with the type that uses the json.RawMessage argument.
(Alternatively, you can create a type with its own unmarshal function, as in lmazgon's answer. But again you must invent a type, rather than just decoding right into the already-supplied target type.)
1In this case, we could use Question *string, or assume that null strings are not allowed, or something, but this example is pretty reductionist. Suppose instead that we're supposed to compute a bignum with an accurate pi to some number of places, or any other hard but realistic computation. The point here is that there can be situations where pre-loading the defaults is relatively expensive, so we'd like to avoid that.
When I make an HTTP call to a REST API I may get the JSON value count back as a Number or String. I'ld like to marshal it to be an integer in either case. How can I deal with this in Go?.
Use the "string" field tag option to specify that strings should be converted to numbers. The documentation for the option is:
The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types. This extra level of encoding is sometimes used when communicating with JavaScript programs:
Here's an example use:
type S struct {
Count int `json:"count,string"`
}
playground example
If the JSON value can be number or string, then unmarshal to interface{} and convert to int after unmarshaling:
Count interface{} `json:"count,string"`
Use this function to convert the interface{} value to an int:
func getInt(v interface{}) (int, error) {
switch v := v.(type) {
case float64:
return int(v), nil
case string:
c, err := strconv.Atoi(v)
if err != nil {
return 0, err
}
return c, nil
default:
return 0, fmt.Errorf("conversion to int from %T not supported", v)
}
}
// Format of your expected request
type request struct {
ACTIVE string `json:"active"`
CATEGORY string `json:"category"`
}
// struct to read JSON input
var myReq request
// Decode the received JSON request to struct
decoder := json.NewDecoder(r.Body)
err := decoder.Decode(&myReq)
if err != nil {
log.Println( err)
// Handler for invalid JSON received or if you want to decode the request using another struct with int.
return
}
defer r.Body.Close()
// Convert string to int
numActive, err = strconv.Atoi(myReq.ACTIVE)
if err != nil {
log.Println(err)
// Handler for invalid int received
return
}
// Convert string to int
numCategory, err = strconv.Atoi(myReq.CATEGORY)
if err != nil {
log.Println(err)
// Handler for invalid int received
return
}
I had the same problem with a list of values where the values were string or struct. The solution I'm using is to create a helper struct with fields of expected types and parse value into the correct field.
type Flag struct {
ID string `json:"id"`
Type string `json:"type"`
}
type FlagOrString struct {
Flag *Flag
String *string
}
func (f *FlagOrString) UnmarshalJSON(b []byte) error {
start := []byte("\"")
for idx := range start {
if b[idx] != start[idx] {
return json.Unmarshal(b, &f.Flag)
}
}
return json.Unmarshal(b, &f.String)
}
var MainStruct struct {
Vals []FlagOrString
}
Custom Unmarshaller simplifies a code. Personally I prefer this over interface{} as it explicitly states what a developer expects.
I have some JSON of the form:
[{
"type": "car",
"color": "red",
"hp": 85,
"doors": 4
}, {
"type": "plane",
"color": "blue",
"engines": 3
}]
I have types car and plane that satisfy a vehicle interface; I'd like to be able to write:
var v []vehicle
e := json.Unmarshal(myJSON, &v)
... and have JSON fill my slice of vehicles with a car and a plane; instead (and unsurprisingly) I just get "cannot unmarshal object into Go value of type main.vehicle".
For reference, here are suitable definitions of the types involved:
type vehicle interface {
vehicle()
}
type car struct {
Type string
Color string
HP int
Doors int
}
func (car) vehicle() { return }
type plane struct {
Type string
Color string
Engines int
}
func (plane) vehicle() { return }
var _ vehicle = (*car)(nil)
var _ vehicle = (*plane)(nil)
(Note that I'm actually totally uninterested in the t field on car and plane - it could be omitted because this information will, if someone successfully answers this question, be implicit in the dynamic type of the objects in v.)
Is there a way to have the JSON umarhsaller choose which type to use based on some part of the contents (in this case, the type field) of the data being decoded?
(Note that this is not a duplicate of Unmarshal JSON with unknown fields because I want each item in the slice to have a different dynamic type, and from the value of the 'type' property I know exactly what fields to expect—I just don't know how to tell json.Unmarshal how to map 'type' property values onto Go types.)
Taking the answers from the similar question: Unmarshal JSON with unknown fields, we can construct a few ways to unamrshal this JSON object in a []vehicle data structure.
The "Unmarshal with Manual Handling" version can be done by using a generic []map[string]interface{} data structure, then building the correct vehicles from the slice of maps. For brevity, this example does leave out the error checking for missing or incorrectly typed fields which the json package would have done.
https://play.golang.org/p/fAY9JwVp-4
func NewVehicle(m map[string]interface{}) vehicle {
switch m["type"].(string) {
case "car":
return NewCar(m)
case "plane":
return NewPlane(m)
}
return nil
}
func NewCar(m map[string]interface{}) *car {
return &car{
Type: m["type"].(string),
Color: m["color"].(string),
HP: int(m["hp"].(float64)),
Doors: int(m["doors"].(float64)),
}
}
func NewPlane(m map[string]interface{}) *plane {
return &plane{
Type: m["type"].(string),
Color: m["color"].(string),
Engines: int(m["engines"].(float64)),
}
}
func main() {
var vehicles []vehicle
objs := []map[string]interface{}{}
err := json.Unmarshal(js, &objs)
if err != nil {
log.Fatal(err)
}
for _, obj := range objs {
vehicles = append(vehicles, NewVehicle(obj))
}
fmt.Printf("%#v\n", vehicles)
}
We could leverage the json package again to take care of the unmarshaling and type checking of the individual structs by unmarshaling a second time directly into the correct type. This could all be wrapped up into a json.Unmarshaler implementation by defining an UnmarshalJSON method on the []vehicle type to first split up the JSON objects into raw messages.
https://play.golang.org/p/zQyL0JeB3b
type Vehicles []vehicle
func (v *Vehicles) UnmarshalJSON(data []byte) error {
// this just splits up the JSON array into the raw JSON for each object
var raw []json.RawMessage
err := json.Unmarshal(data, &raw)
if err != nil {
return err
}
for _, r := range raw {
// unamrshal into a map to check the "type" field
var obj map[string]interface{}
err := json.Unmarshal(r, &obj)
if err != nil {
return err
}
vehicleType := ""
if t, ok := obj["type"].(string); ok {
vehicleType = t
}
// unmarshal again into the correct type
var actual vehicle
switch vehicleType {
case "car":
actual = &car{}
case "plane":
actual = &plane{}
}
err = json.Unmarshal(r, actual)
if err != nil {
return err
}
*v = append(*v, actual)
}
return nil
}
JSON decoding and encoding in Go is actually surprisingly well at recognizing fields inside embedded structs. E.g. decoding or encoding the following structure works when there is no overlapping fields between type A and type B:
type T struct{
Type string `json:"type"`
*A
*B
}
type A struct{
Baz int `json:"baz"`
}
type B struct{
Bar int `json:"bar"`
}
Be aware that if both "baz" and "bar" are set in the JSON for the example above, both the T.A and T.B properties will be set.
If there is overlapping fields between A and B, or just to be able to better discard invalid combinations of fields and type, you need to implement the json.Unmarshaler interface. To not have to first decode fields into a map, you can extend the trick of using embedded structs.
type TypeSwitch struct {
Type string `json:"type"`
}
type T struct {
TypeSwitch
*A
*B
}
func (t *T) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &t.TypeSwitch); err != nil {
return err
}
switch t.Type {
case "a":
t.A = &A{}
return json.Unmarshal(data, t.A)
case "b":
t.B = &B{}
return json.Unmarshal(data, t.B)
default:
return fmt.Errorf("unrecognized type value %q", t.Type)
}
}
type A struct {
Foo string `json:"bar"`
Baz int `json:"baz"`
}
type B struct {
Foo string `json:"foo"`
Bar int `json:"bar"`
}
For marshaling back, json.Marshaler must also be implemented if there is overlapping fields.
Full example: https://play.golang.org/p/UHAdxlVdFQQ
The two passes approach works fine, but there is also the option of the mapstructure package, that was created to do exactly this.
I was facing the same problem.
I'm using the lib github.com/mitchellh/mapstructure together the encoding/json.
I first, unmarshal the json to a map, and use mapstructure to convert the map to my struct, e.g.:
type (
Foo struct {
Foo string `json:"foo"`
}
Bar struct {
Bar string `json:"bar"`
}
)
func Load(jsonStr string, makeInstance func(typ string) any) (any, error) {
// json to map
m := make(map[string]any)
e := json.Unmarshal([]byte(jsonStr), &m)
if e != nil {
return nil, e
}
data := makeInstance(m["type"].(string))
// decoder to copy map values to my struct using json tags
cfg := &mapstructure.DecoderConfig{
Metadata: nil,
Result: &data,
TagName: "json",
Squash: true,
}
decoder, e := mapstructure.NewDecoder(cfg)
if e != nil {
return nil, e
}
// copy map to struct
e = decoder.Decode(m)
return data, e
}
Using:
f, _ := Load(`{"type": "Foo", "foo": "bar"}`, func(typ string) any {
switch typ {
case "Foo":
return &Foo{}
}
return nil
})
If the property is a string you can use .(string) for casting the property because the origin is an interface.
You can use it the next way:
v["type"].(string)
The problem I'm currently having is after saving a struct to a json file and then opening the struct from the json file, somehow the properties of the struct have changed slightly.
In the struct N, sometimes A and B can point to the same J. However, after encoding then decoding they point to different Js of the value.
before encoding this returns true (expected). After decoding it, it returns false (not expected)
fmt.Println("is same pointer", n.A[0] == n.B[0])
Is this supposed to happen? Is there a way around this. Thanks.
type N struct {
A []*J
B []*J
C []*J
}
func (n *N) Save(name string) {
name = "radacted.json"
err := os.Remove(name)
file, err := os.Create(name)
defer file.Close()
if err != nil {
fmt.Println(err)
}
bytes, err := json.Marshal(n)
file.Write(bytes)
}
func Open(name string) *N {
bytes, err := ioutil.ReadFile("redacted.json")
if err != nil {
log.Fatal("decode error:", err)
}
var n NeuralNetwork
json.Unmarshal(bytes, &n)
return &n
}
It's expected and documented behaviour
Pointer values encode as the value pointed to.
You can assert values equality
*n.A[0] == *n.B[0] //should stay
fmt.Println("is same pointer", n.A[0] == n.B[0])
you are comparing the address value here so it will not be the same. Let me give you an example
suppose you have struct like this :
type Test struct {
ValueA *int
ValueB *int
}
and on your main function you add the same value but with different address in this case with different variable :
func main() {
hello := 12
hello2 := 12
testObject := Test{ValueA: &hello, ValueB: &hello2}
if *testObject.ValueA == *testObject.ValueB {
fmt.Println("Equal Value")
} else {
fmt.Println("Different Value")
}
}
Notice that the *testObject.ValueA and *testObject.ValueBis getting the exact value not the value address. If you are not using * the result would be different.
so as uvelichitel said you just need to use * when comparing your struct value.