I am very new to GOLANG.
I have been trying for quite some time now to unmarshal an ethereum RPC JSON which has a dynamic structure. No GOLANG struct and map setup I did worked and I am able to get the stateDiff entries (3) but all lower structs seem not to be filled ith any data. So I am able to loop through all the 3 entries but then don't know how to access the values below and when dumping the unmarshal result, I see that GOLANG is not delivering the data anyway into StateDiff
JSON FILE:
{
"jsonrpc":"2.0",
"id":1,
"result":{
"output":"0x0000000000000000000000000000000000000000000000000000000000000001",
"stateDiff":{
"0x0000000000000000000000000000000000000000":{
"balance":{
"*":{
"from":"0x45acecdfadb71366cf",
"to":"0x45aced3909536ccacf"
}
},
"code":"=",
"nonce":"=",
"storage":{
}
},
"0x07865c6e87b9f70255377e024ace6630c1eaa37f":{
"balance":"=",
"code":"=",
"nonce":"=",
"storage":{
"0x86a60af761556602732bbdeaef13ba6e2481f83362d3489389f51353d86a6ac3":{
"*":{
"from":"0x0000000000000000000000000000000000000000000000000000000000000000",
"to":"0x0000000000000000000000000000000000000000000000000000000000002710"
}
},
"0xb0cf6f3c0836765b9dee3d1537458f10fe99447508adc172c1f633ac7352aaa8":{
"*":{
"from":"0x00000000000000000000000000000000000000000000000000092f379a04d2b0",
"to":"0x00000000000000000000000000000000000000000000000000092f379a04aba0"
}
}
}
},
"0x6dbe810e3314546009bd6e1b29f9031211cda5d2":{
"balance":{
"*":{
"from":"0x41c41fc2c0247860",
"to":"0x41c3c66723c4155c"
}
},
"code":"=",
"nonce":{
"*":{
"from":"0x741",
"to":"0x742"
}
},
"storage":{
}
}
},
"trace":[
],
"vmTrace":null
}
}
I have tried to unmarshal the JSON into the following structure (among many) and i can't get the values such as result>stateDiff>0x0000000000000000000000000000000000000000>balance>*>from
Struct below is just one of many i tried. I can't get anything below the entry 0x0000000000000000000000000000000000000000
type structChange struct {
Changes map[string]string `json:"*"`
}
type structStateDiff struct {
Balance *structChange `json:"balance"`
Code string `json:"code"`
Nonce string `json:"nonce"`
Storage map[string]*structChange `json:"storage"`
}
type res_trace_replayTransaction struct {
Jsonrpc string `json:"jsonrpc"`
ID int `json:"id"`
Result struct {
Output string `json:"output"`
StateDiff map[string]*structStateDiff `json:"stateDiff"`
Trace []interface{} `json:"trace"`
VMTrace interface{} `json:"vmTrace"`
} `json:"result"`
}
EDIT:
Code for umarshal
retObj := rpcCall(jstring)
var callResponse res_trace_replayTransaction
err := json.Unmarshal(retObj, &callResponse)
Note that by default the encoding/json package can unmarshal a JSON string into a Go string, and it can unmarshal a JSON object into a Go map or a Go struct. Additionally it can unmarshal any JSON value into an empty interface{}.
Also note that Go is a statically typed language, if you specify a value to be of type T1 then, at runtime, you cannot change it's type to T2. There is just no way to do it, no way to change a value's type.
So if you define a field to be of some struct type, you cannot, by default, unmarshal a JSON string into it. And equally if you define a field to be of type string, you cannot, by default, unmarshal a JSON object into it.
But because JSON itself allows for a dynamic structure the encoding/json package provides two interfaces that give you the ability to customize how the JSON is marshaled and unmarshaled.
So if you have a JSON property (e.g. "balance" or "nonce") that can be either "=" (a string), or { ... } (an object), you will need to declare a custom type that implements the json.Marshaler and json.Unmarshaler interfaces that know how to properly marshal and unmarshal the JSON value.
For example:
type structChange struct {
Changes map[string]string `json:"*"`
}
func (s structChange) MarshalJSON() ([]byte, error) {
// if empty retrun `"="`
if len(s.Changes) == 0 {
return []byte(`"="`), nil
}
// otherwise marshal as is
type T structChange
return json.Marshal(T(s))
}
func (s *structChange) UnmarshalJSON(data []byte) error {
// if `"="`, ignore
if string(data) == `"="` {
return nil
}
// otherwise assume it's a valid object
type T structChange
return json.Unmarshal(data, (*T)(s))
}
NOTE: the temporary type T above is used to avoid a stack overflow caused by an infinite recursive call to the MarshalJSON and UnmarshalJSON methods.
https://go.dev/play/p/yfsTrMozZ2Z
Related
I'm trying to understand the behavior of JSON decoding and unmarshalling in Go when an error is encountered. In particular, in the following struct, the title field is a different format than is provided by the returned JSON and an error provided. Is the behavior to continue with other fields despite the error, or will processing of the additional fields stop? I haven't found any mention of how this is handled in the standard library.
type Response struct {
Title struct {
Text string `json:"text"`
Accessibility struct {
Label string `json:"label"`
Identifier string `json:"identifier"`
} `json:"accessibility"`
} `json:"title,omitempty"`
Type string `json:"type"`
}
{
"title": "test",
"type": "type"
}
A quick test reveals it:
func main() {
var r Response
if err := json.Unmarshal([]byte(src), &r); err != nil {
fmt.Println(err)
}
fmt.Printf("%+v", r)
}
const src = `{
"title": "test",
"type": "type"
}
Result (try it on the Go Playground):
json: cannot unmarshal string into Go struct field Response.title of type struct { Text string "json:\"text\""; Accessibility struct { Label string "json:\"label\""; Identifier string "json:\"identifier\"" } "json:\"accessibility\"" }
{Title:{Text: Accessibility:{Label: Identifier:}} Type:type}
A non-nil error is returned while the subsequent Type field gets unmarshaled properly.
In general: An error is returned, while unmarshaling continues without any guarantees.
This is documented at json.Unmarshal():
If a JSON value is not appropriate for a given target type, or if a JSON number overflows the target type, Unmarshal skips that field and completes the unmarshaling as best it can. If no more serious errors are encountered, Unmarshal returns an UnmarshalTypeError describing the earliest such error. In any case, it's not guaranteed that all the remaining fields following the problematic one will be unmarshaled into the target object.
I'm working with JSON that returns three different object types 'items','categories' and 'modifiers'. An example of the JSON can be viewed here. I created models for the three types of objects. But when I unmarshal I have selected one of the types to unmarshal the entire JSON to.(I know this cant be the correct way...) I then try to parse out the different items depending on what their type is identified as in the json field 'Type' and then append that object to a slice of the proper type. I am having errors because I don't know how to unmarshal JSON that has different types in it that have different fields.
What is the proper method to unmarshal JSON that contains different objects, each with their own respective fields?
Is the solution to create a "super model" which contains all possible fields and then unmarshal to that?
I'm still fairly new and would appreciate any advice. Thanks!
If you implement json.Unmarshaler, you can define a struct that parses each item type into it's relevant struct.
Example:
// Dynamic represents an item of any type.
type Dynamic struct {
Value interface{}
}
// UnmarshalJSON is called by the json package when we ask it to
// parse something into Dynamic.
func (d *Dynamic) UnmarshalJSON(data []byte) error {
// Parse only the "type" field first.
var meta struct {
Type string
}
if err := json.Unmarshal(data, &meta); err != nil {
return err
}
// Determine which struct to unmarshal into according to "type".
switch meta.Type {
case "product":
d.Value = &Product{}
case "post":
d.Value = &Post{}
default:
return fmt.Errorf("%q is an invalid item type", meta.Type)
}
return json.Unmarshal(data, d.Value)
}
// Product and Post are structs representing two different item types.
type Product struct {
Name string
Price int
}
type Post struct {
Title string
Content string
}
Usage:
func main() {
// Parse a JSON item into Dynamic.
input := `{
"type": "product",
"name": "iPhone",
"price": 1000
}`
var dynamic Dynamic
if err := json.Unmarshal([]byte(input), &dynamic); err != nil {
log.Fatal(err)
}
// Type switch on dynamic.Value to get the parsed struct.
// See https://tour.golang.org/methods/16
switch dynamic.Value.(type) {
case *Product:
log.Println("got a product:", dynamic.Value)
case *Post:
log.Println("got a product:", dynamic.Value)
}
}
Output:
2009/11/10 23:00:00 got a product: &{iPhone 1000}
Try it in the Go Playground.
Tip: if you have a list of dynamic objects, just parse into a slice of Dynamic:
var items []Dynamic
json.Unmarshal(`[{...}, {...}]`, &items)
Example output:
[&{iPhone 1000} &{A Good Post Lorem ipsum...}]
I think https://github.com/mitchellh/mapstructure also fits into your use case.
Given the following JSON
{
"some": "value"
"nested": {
"some": "diffvalue",
"nested": {
"some": "innervalue"
}
}
}
which roughly translates to this struct:
type Envelope struct {
some string `json:"some"`
nested InnerEnvelope `json:"nested"`
}
where InnerEnvelope is: type InnerEnvelope map[string]interface{}
Running a simple json.Unmarshal([]byte value, &target) does not help here, because of the recursive type nature of the original JSON.
I do not know up front how deeply and under which keys the inner maps will exist, so I cannot declare the types upfront.
The idea is, that using map[string]interface{} as the type is not good enough, since I need the values in the InnerEnvelope to be somehow transformed & typed. Details are not important, but image, I need to cast every value inside the NestedEnvelope of a bool type as a string saying "true" or "false" as opposed of having an actual bool type.
I turned to UnmarshalJSON interface to solve this problem. I can easily do it at the top level like so:
func (m *Envelope) UnmarshalJSON(b []byte) error {
var stuff noBoolMap
if err := json.Unmarshal(b, &stuff); err != nil {
return err
}
for key, value := range stuff {
switch value.(type) {
case bool:
stuff[key] = strconv.FormatBool(value.(bool))
}
}
return nil
}
But since the inner json.Unmarshal will already have inner maps parsed as map[string]interface{}, I would need to yet-again to traverse the inner maps, cast them to appropriate type and perform my value transformations.
So my question is: In this case, what is the way this would be approached in Go, and preferably do it in a single-pass?
The expected result of the JSON example above would be:
Envelope {
some: string
nested: InnerEnvelope {
some: string {
nested: InnerEnvelope {
some: string
}
}
}
Given your json, you can do this:
type Envelope struct {
Some string `json:"some"`
Nested json.RawMessage `json:"nested"`
}
json.RawMessage is a rather hidden gem, and more people seem to go for the map[string]interface{}.
Using json.RawMessage will result in the nested json to be represented by this RawMessage, which you then can process again as a normal json (unmarshal it into Envelope).
This is more elegant than the map[string]interface{} approach.
I learned from the gin doc that you can bind json to a struct like
type Login struct {
User string `form:"user" json:"user" binding:"required"`
Password string `form:"password" json:"password" binding:"required"`
}
func main() {
router := gin.Default()
// Example for binding JSON ({"user": "manu", "password": "123"})
router.POST("/loginJSON", func(c *gin.Context) {
var json Login
if c.BindJSON(&json) == nil {
if json.User == "manu" && json.Password == "123" {
c.JSON(http.StatusOK, gin.H{"status": "you are logged in"})
} else {
c.JSON(http.StatusUnauthorized, gin.H{"status": "unauthorized"})
}
}
})
}
You always have to build a struct to bind JSON.
But if there is a very complex JSON data, I only what to get part of it, to create a complex struct is a big burden. Can avoid id and parse it directly?
You don't need to create a struct for all the fields present in the JSON response, only the fields you are interested in. Any other fields present in the response will be ignored when unmarshaling.
You can also unmarshal to a generic map[string]interface{} but this is only really useful for truly dynamic data. If you know the format of the response ahead of time you will nearly always be best to create a custom struct to get type safety and avoid continual nil checks when accessing the map. Additionally, a targeted struct avoids storing unnecessary values when JSON in unmarshalled.
You can use the JSON to Go tool to help quickly create a struct definition from a JSON response. You could then easily strip out all the fields you don't need.
I have a struct like this:
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
But even if the instance of MyStruct is entirely empty (meaning, all values are default), it's being serialized as:
"data":{}
I know that the encoding/json docs specify that "empty" fields are:
false, 0, any nil pointer or interface value, and any array,
slice, map, or string of length zero
but with no consideration for a struct with all empty/default values. All of its fields are also tagged with omitempty, but this has no effect.
How can I get the JSON package to not marshal my field that is an empty struct?
As the docs say, "any nil pointer." -- make the struct a pointer. Pointers have obvious "empty" values: nil.
Fix - define the type with a struct pointer field:
type Result struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
Then a value like this:
result := Result{}
Will marshal as:
{}
Explanation: Notice the *MyStruct in our type definition. JSON serialization doesn't care whether it is a pointer or not -- that's a runtime detail. So making struct fields into pointers only has implications for compiling and runtime).
Just note that if you do change the field type from MyStruct to *MyStruct, you will need pointers to struct values to populate it, like so:
Data: &MyStruct{ /* values */ }
As #chakrit mentioned in a comment, you can't get this to work by implementing json.Marshaler on MyStruct, and implementing a custom JSON marshalling function on every struct that uses it can be a lot more work. It really depends on your use case as to whether it's worth the extra work or whether you're prepared to live with empty structs in your JSON, but here's the pattern I use applied to Result:
type Result struct {
Data MyStruct
Status string
Reason string
}
func (r Result) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}{
Data: &r.Data,
Status: r.Status,
Reason: r.Reason,
})
}
func (r *Result) UnmarshalJSON(b []byte) error {
decoded := new(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
})
err := json.Unmarshal(b, decoded)
if err == nil {
r.Data = decoded.Data
r.Status = decoded.Status
r.Reason = decoded.Reason
}
return err
}
If you have huge structs with many fields this can become tedious, especially changing a struct's implementation later, but short of rewriting the whole json package to suit your needs (not a good idea), this is pretty much the only way I can think of getting this done while still keeping a non-pointer MyStruct in there.
Also, you don't have to use inline structs, you can create named ones. I use LiteIDE with code completion though, so I prefer inline to avoid clutter.
Data is an initialized struct, so it isn't considered empty because encoding/json only looks at the immediate value, not the fields inside the struct.
Unfortunately, returning nil from json.Marshaler doesn't currently work:
func (_ MyStruct) MarshalJSON() ([]byte, error) {
if empty {
return nil, nil // unexpected end of JSON input
}
// ...
}
You could give Result a marshaler as well, but it's not worth the effort.
The only option, as Matt suggests, is to make Data a pointer and set the value to nil.
There is an outstanding Golang proposal for this feature which has been active for over 4 years, so at this point, it is safe to assume that it will not make it into the standard library anytime soon. As #Matt pointed out, the traditional approach is to convert the structs to pointers-to-structs. If this approach is infeasible (or impractical), then an alternative is to use an alternate json encoder which does support omitting zero value structs.
I created a mirror of the Golang json library (clarketm/json) with added support for omitting zero value structs when the omitempty tag is applied. This library detects zeroness in a similar manner to the popular YAML encoder go-yaml by recursively checking the public struct fields.
e.g.
$ go get -u "github.com/clarketm/json"
import (
"fmt"
"github.com/clarketm/json" // drop-in replacement for `encoding/json`
)
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
j, _ := json.Marshal(&Result{
Status: "204",
Reason: "No Content",
})
fmt.Println(string(j))
// Note: `data` is omitted from the resultant json.
{
"status": "204"
"reason": "No Content"
}