Unmarshal Inconsistent JSON in Go - json

I'm working with JSON that returns three different object types 'items','categories' and 'modifiers'. An example of the JSON can be viewed here. I created models for the three types of objects. But when I unmarshal I have selected one of the types to unmarshal the entire JSON to.(I know this cant be the correct way...) I then try to parse out the different items depending on what their type is identified as in the json field 'Type' and then append that object to a slice of the proper type. I am having errors because I don't know how to unmarshal JSON that has different types in it that have different fields.
What is the proper method to unmarshal JSON that contains different objects, each with their own respective fields?
Is the solution to create a "super model" which contains all possible fields and then unmarshal to that?
I'm still fairly new and would appreciate any advice. Thanks!

If you implement json.Unmarshaler, you can define a struct that parses each item type into it's relevant struct.
Example:
// Dynamic represents an item of any type.
type Dynamic struct {
Value interface{}
}
// UnmarshalJSON is called by the json package when we ask it to
// parse something into Dynamic.
func (d *Dynamic) UnmarshalJSON(data []byte) error {
// Parse only the "type" field first.
var meta struct {
Type string
}
if err := json.Unmarshal(data, &meta); err != nil {
return err
}
// Determine which struct to unmarshal into according to "type".
switch meta.Type {
case "product":
d.Value = &Product{}
case "post":
d.Value = &Post{}
default:
return fmt.Errorf("%q is an invalid item type", meta.Type)
}
return json.Unmarshal(data, d.Value)
}
// Product and Post are structs representing two different item types.
type Product struct {
Name string
Price int
}
type Post struct {
Title string
Content string
}
Usage:
func main() {
// Parse a JSON item into Dynamic.
input := `{
"type": "product",
"name": "iPhone",
"price": 1000
}`
var dynamic Dynamic
if err := json.Unmarshal([]byte(input), &dynamic); err != nil {
log.Fatal(err)
}
// Type switch on dynamic.Value to get the parsed struct.
// See https://tour.golang.org/methods/16
switch dynamic.Value.(type) {
case *Product:
log.Println("got a product:", dynamic.Value)
case *Post:
log.Println("got a product:", dynamic.Value)
}
}
Output:
2009/11/10 23:00:00 got a product: &{iPhone 1000}
Try it in the Go Playground.
Tip: if you have a list of dynamic objects, just parse into a slice of Dynamic:
var items []Dynamic
json.Unmarshal(`[{...}, {...}]`, &items)
Example output:
[&{iPhone 1000} &{A Good Post Lorem ipsum...}]

I think https://github.com/mitchellh/mapstructure also fits into your use case.

Related

GOLANG unmarshal dynamic JSON

I am very new to GOLANG.
I have been trying for quite some time now to unmarshal an ethereum RPC JSON which has a dynamic structure. No GOLANG struct and map setup I did worked and I am able to get the stateDiff entries (3) but all lower structs seem not to be filled ith any data. So I am able to loop through all the 3 entries but then don't know how to access the values below and when dumping the unmarshal result, I see that GOLANG is not delivering the data anyway into StateDiff
JSON FILE:
{
"jsonrpc":"2.0",
"id":1,
"result":{
"output":"0x0000000000000000000000000000000000000000000000000000000000000001",
"stateDiff":{
"0x0000000000000000000000000000000000000000":{
"balance":{
"*":{
"from":"0x45acecdfadb71366cf",
"to":"0x45aced3909536ccacf"
}
},
"code":"=",
"nonce":"=",
"storage":{
}
},
"0x07865c6e87b9f70255377e024ace6630c1eaa37f":{
"balance":"=",
"code":"=",
"nonce":"=",
"storage":{
"0x86a60af761556602732bbdeaef13ba6e2481f83362d3489389f51353d86a6ac3":{
"*":{
"from":"0x0000000000000000000000000000000000000000000000000000000000000000",
"to":"0x0000000000000000000000000000000000000000000000000000000000002710"
}
},
"0xb0cf6f3c0836765b9dee3d1537458f10fe99447508adc172c1f633ac7352aaa8":{
"*":{
"from":"0x00000000000000000000000000000000000000000000000000092f379a04d2b0",
"to":"0x00000000000000000000000000000000000000000000000000092f379a04aba0"
}
}
}
},
"0x6dbe810e3314546009bd6e1b29f9031211cda5d2":{
"balance":{
"*":{
"from":"0x41c41fc2c0247860",
"to":"0x41c3c66723c4155c"
}
},
"code":"=",
"nonce":{
"*":{
"from":"0x741",
"to":"0x742"
}
},
"storage":{
}
}
},
"trace":[
],
"vmTrace":null
}
}
I have tried to unmarshal the JSON into the following structure (among many) and i can't get the values such as result>stateDiff>0x0000000000000000000000000000000000000000>balance>*>from
Struct below is just one of many i tried. I can't get anything below the entry 0x0000000000000000000000000000000000000000
type structChange struct {
Changes map[string]string `json:"*"`
}
type structStateDiff struct {
Balance *structChange `json:"balance"`
Code string `json:"code"`
Nonce string `json:"nonce"`
Storage map[string]*structChange `json:"storage"`
}
type res_trace_replayTransaction struct {
Jsonrpc string `json:"jsonrpc"`
ID int `json:"id"`
Result struct {
Output string `json:"output"`
StateDiff map[string]*structStateDiff `json:"stateDiff"`
Trace []interface{} `json:"trace"`
VMTrace interface{} `json:"vmTrace"`
} `json:"result"`
}
EDIT:
Code for umarshal
retObj := rpcCall(jstring)
var callResponse res_trace_replayTransaction
err := json.Unmarshal(retObj, &callResponse)
Note that by default the encoding/json package can unmarshal a JSON string into a Go string, and it can unmarshal a JSON object into a Go map or a Go struct. Additionally it can unmarshal any JSON value into an empty interface{}.
Also note that Go is a statically typed language, if you specify a value to be of type T1 then, at runtime, you cannot change it's type to T2. There is just no way to do it, no way to change a value's type.
So if you define a field to be of some struct type, you cannot, by default, unmarshal a JSON string into it. And equally if you define a field to be of type string, you cannot, by default, unmarshal a JSON object into it.
But because JSON itself allows for a dynamic structure the encoding/json package provides two interfaces that give you the ability to customize how the JSON is marshaled and unmarshaled.
So if you have a JSON property (e.g. "balance" or "nonce") that can be either "=" (a string), or { ... } (an object), you will need to declare a custom type that implements the json.Marshaler and json.Unmarshaler interfaces that know how to properly marshal and unmarshal the JSON value.
For example:
type structChange struct {
Changes map[string]string `json:"*"`
}
func (s structChange) MarshalJSON() ([]byte, error) {
// if empty retrun `"="`
if len(s.Changes) == 0 {
return []byte(`"="`), nil
}
// otherwise marshal as is
type T structChange
return json.Marshal(T(s))
}
func (s *structChange) UnmarshalJSON(data []byte) error {
// if `"="`, ignore
if string(data) == `"="` {
return nil
}
// otherwise assume it's a valid object
type T structChange
return json.Unmarshal(data, (*T)(s))
}
NOTE: the temporary type T above is used to avoid a stack overflow caused by an infinite recursive call to the MarshalJSON and UnmarshalJSON methods.
https://go.dev/play/p/yfsTrMozZ2Z

Unmarshaling nested custom same-type JSON in Go

Given the following JSON
{
"some": "value"
"nested": {
"some": "diffvalue",
"nested": {
"some": "innervalue"
}
}
}
which roughly translates to this struct:
type Envelope struct {
some string `json:"some"`
nested InnerEnvelope `json:"nested"`
}
where InnerEnvelope is: type InnerEnvelope map[string]interface{}
Running a simple json.Unmarshal([]byte value, &target) does not help here, because of the recursive type nature of the original JSON.
I do not know up front how deeply and under which keys the inner maps will exist, so I cannot declare the types upfront.
The idea is, that using map[string]interface{} as the type is not good enough, since I need the values in the InnerEnvelope to be somehow transformed & typed. Details are not important, but image, I need to cast every value inside the NestedEnvelope of a bool type as a string saying "true" or "false" as opposed of having an actual bool type.
I turned to UnmarshalJSON interface to solve this problem. I can easily do it at the top level like so:
func (m *Envelope) UnmarshalJSON(b []byte) error {
var stuff noBoolMap
if err := json.Unmarshal(b, &stuff); err != nil {
return err
}
for key, value := range stuff {
switch value.(type) {
case bool:
stuff[key] = strconv.FormatBool(value.(bool))
}
}
return nil
}
But since the inner json.Unmarshal will already have inner maps parsed as map[string]interface{}, I would need to yet-again to traverse the inner maps, cast them to appropriate type and perform my value transformations.
So my question is: In this case, what is the way this would be approached in Go, and preferably do it in a single-pass?
The expected result of the JSON example above would be:
Envelope {
some: string
nested: InnerEnvelope {
some: string {
nested: InnerEnvelope {
some: string
}
}
}
Given your json, you can do this:
type Envelope struct {
Some string `json:"some"`
Nested json.RawMessage `json:"nested"`
}
json.RawMessage is a rather hidden gem, and more people seem to go for the map[string]interface{}.
Using json.RawMessage will result in the nested json to be represented by this RawMessage, which you then can process again as a normal json (unmarshal it into Envelope).
This is more elegant than the map[string]interface{} approach.

UnmarshalJSON on struct containing interface list

I would like to UnmarshalJSON a struct containing an interface as follows:
type Filterer interface {
Filter(s string) error
}
type FieldFilter struct {
Key string
Val string
}
func (ff *FieldFilter) Filter(s string) error {
// Do something
}
type Test struct {
Name string
Filters []Filterer
}
My idea was to send a json like so:
{
"Name": "testing",
"Filters": [
{
"FieldFilter": {
"Key": "key",
"Val": "val"
}
}
]
}
However, when sending this json to the unmarshaler, the following exception returns: json: cannot unmarshal object into Go struct field Test.Filters of type Filterer
I understand the problem fully, but do not know how to approach this problem wisely. Looking for advice on an idiomatic way to solving this problem in go.
Following my own question, I researched how one could implement UnmarshalJSON for interface lists. Ultimately this led me to publish a blog post on how to do this properly. Basically there are 2 main solutions:
Parse the required JSON string into a map[string]*json.RawMessage and work your way from there.
Make an alias for the interface list and implement UnmarshalJSON for that alias. However, you'll still need to work with map[string]*json.RawMessage and some manual work. Nothing comes without a price!
I highly suggest taking the seconds approach. While these two solutions may result in the same amount of code lines, taking advantage of type aliasing and being less dependent on json.RawMessage types will make a more easy to manage code, especially when it is required to support multiple interfaces on the UnmarshalJSON implementation
To directly answer the question, start with making a type alias for the interface list:
type Filterers []Filterer
Now continue with implementing the decoding of the JSON:
func (f *Filterers) UnmarshalJSON(b []byte) error {
var FilterFields map[string]*json.RawMessage
if err := json.Unmarshal(b, &FilterFields); err != nil {
return err
}
for LFKey, LFValue := range FilterFields {
if LFKey == "FieldFilter" {
var MyFieldFilters []*json.RawMessage
if err := json.Unmarshal(*LFValue, &MyFieldFilters); err != nil {
return err
}
for _, MyFieldFilter := range MyFieldFilters {
var filter FieldFilter
if err := json.Unmarshal(*MyFieldFilter, &filter); err != nil {
return err
}
*f = append(*f, &filter)
}
}
}
return nil
}
A detailed explanation (with some examples and a full working code snippets) of the second approach is available on my own blog
There is no way for Unmarshal to know what type it should use. The only case where it can just "make something up" is if it's asked to unmarshal into an interface{}, in which case it will use the rules in the documentation. Since none of those types can be put into a []Filterer, it cannot unmarshal that field. If you want to unmarshal into a struct type, you must specify the field to be of that type.
You can always unmarshal into an intermediate struct or map type, and then do your own conversion from that into whatever types you want.

How to access fields of a JSON in GO

Hi everyone I'm trying to see what the proper way of accessing fields of a json object from a http.get request in go.
I first do an http.get call get the JSON and then print it (which works) but is there a way to access just a field?
for example:
response, err:= http.Get("URL")
//Error checking is done between
contents, err:=ioutil.Readall(response.Body)
//Now at this point I have a json that looks like
{"id": "someID",
"name": "someName",
"test": [{"Name":"Test1",
"Result": "Success"},
{"Name":"Test2",
"Result": "Success"},
{...},
]}
Is there a way to only print the "test" of the Json? What is the proper way of accessing that field?
Use encoding/json package to Unmarshal data into struct, like following.
type Result struct {
ID string `json:"id"`
Name string `json:"name"`
Test []interface{} `json:"test"`
}
var result Result
json.Unmarshal(contents, &result)
fmt.Println(result.Test)
You can also parse Test to specific struct.
Same as the previous answer, use encoding/json package to Unmarshal data. But if you don't want to specify the structure, you could use map[string]interface/bson.M{} to receive the data, and get the field, then cast into types your want.
m := make(map[string]interface{})
err := json.Unmarshal(data, &m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m["id"])
You may want to try gabs container, if you are not sure how depth JSON hierarchy can be. Have a look at below resources
https://github.com/Jeffail/gabs
https://godoc.org/github.com/Jeffail/gabs
If you just want to access one field then you can use the jsonq module https://godoc.org/github.com/jmoiron/jsonq
For your example you could get the test object with code similar to
jq.Object("test")
Where jq is a jsonq query object constructed from your JSON above (see the godoc page for instructions on how to create a query object from a JSON stream or string).
You can also use this library for retrieving specific String, Integer, Float and Bool values at an arbitrary depth inside a JSON object.
Since you are starting with a URL, Decode is a better option than Unmarshal:
package main
import (
"encoding/json"
"net/http"
)
func main() {
r, e := http.Get("https://github.com/manifest.json")
if e != nil {
panic(e)
}
defer r.Body.Close()
var s struct { Name string }
json.NewDecoder(r.Body).Decode(&s)
println(s.Name == "GitHub")
}
https://golang.org/pkg/encoding/json#Decoder.Decode
You may check this https://github.com/valyala/fastjson
s := []byte(`{"foo": [123, "bar"]}`)
fmt.Printf("foo.0=%d\n", fastjson.GetInt(s, "foo", "0"))
// Output:
// foo.0=123

Golang Null Types and json.Decode()

I have not been able to find a way around this issue currently. If I have a structure i would like to populate with json from a http.Request I have no way to tell for instance what value was actually passed in for some values. For instance if I pass in an empty json object and run json.Decode on a structure that looks like this...
var Test struct {
Number int `json:"number"`
}
I now have a json object that supposedly was passed with a key of number and a value of zero when in fact I would rather have this return nothing at all. Does go provide another method that would actually allow me to see what JSON has been passed in or not.
Sorry for the rambling I have been trying to figure out how to to this for a few days now and it's driving me nuts.
Thanks for any help.
Edit:
I made this to depict exactly what I am talking about http://play.golang.org/p/aPFKSvuxC9
You could use pointers, for example:
func main() {
var jsonBlob = []byte(`[
{"Name": "Platypus"},
{"Name": "Quoll", "Order": 100}
]`)
type Animal struct {
Name string
Order *int
}
var animals []Animal
err := json.Unmarshal(jsonBlob, &animals)
if err != nil {
fmt.Println("error:", err)
}
for _, a := range animals {
if a.Order != nil {
fmt.Printf("got order, %s : %d\n", a.Name, *a.Order)
}
}
}
I don't see how you could do this by giving a struct to the Unmarshal function. With the following structure for instance:
type A struct {
Hello string
Foo int
Baz string
}
var a A
json.Unmarshal(data, &a)
Even by doing another implementation of Unmarshal, there would be only two (simple) possibilities:
If baz is not in the json data, set a.Baz to a default value, compatible with its type: the empty string (or 0 if it's an integer). This is the current implementation.
If baz is not in the json data, return an error. That would be very inconvenient if the absence of baz is a normal behaviour.
Another possibility would be to use pointers, and use the default value nil in the same spirit than the default value I talked about, but there would still be issue if your json file could be filled with null values: you would not be able to distinguish values that were in the json file, but set as null, and values that were not in the json, and unmarshalled with nil as their default value.
However, this solution might suit you: instead of using a struct, why not using a map[string]interface{} ? The Unmarshall function would not have to add a default value to non-present fields, and it would be able to retrieve any type of data from the json file.
var b = []byte(`[{"Name": "Platypus"}, {"Name": "Quoll", "Order": 100}]`)
var m []map[string]interface{}
err := json.Unmarshal(b, &m)
fmt.Println(m)
// [map[Name:Platypus] map[Name:Quoll Order:100]]