I'm trying to understand the behavior of JSON decoding and unmarshalling in Go when an error is encountered. In particular, in the following struct, the title field is a different format than is provided by the returned JSON and an error provided. Is the behavior to continue with other fields despite the error, or will processing of the additional fields stop? I haven't found any mention of how this is handled in the standard library.
type Response struct {
Title struct {
Text string `json:"text"`
Accessibility struct {
Label string `json:"label"`
Identifier string `json:"identifier"`
} `json:"accessibility"`
} `json:"title,omitempty"`
Type string `json:"type"`
}
{
"title": "test",
"type": "type"
}
A quick test reveals it:
func main() {
var r Response
if err := json.Unmarshal([]byte(src), &r); err != nil {
fmt.Println(err)
}
fmt.Printf("%+v", r)
}
const src = `{
"title": "test",
"type": "type"
}
Result (try it on the Go Playground):
json: cannot unmarshal string into Go struct field Response.title of type struct { Text string "json:\"text\""; Accessibility struct { Label string "json:\"label\""; Identifier string "json:\"identifier\"" } "json:\"accessibility\"" }
{Title:{Text: Accessibility:{Label: Identifier:}} Type:type}
A non-nil error is returned while the subsequent Type field gets unmarshaled properly.
In general: An error is returned, while unmarshaling continues without any guarantees.
This is documented at json.Unmarshal():
If a JSON value is not appropriate for a given target type, or if a JSON number overflows the target type, Unmarshal skips that field and completes the unmarshaling as best it can. If no more serious errors are encountered, Unmarshal returns an UnmarshalTypeError describing the earliest such error. In any case, it's not guaranteed that all the remaining fields following the problematic one will be unmarshaled into the target object.
Related
I am very new to GOLANG.
I have been trying for quite some time now to unmarshal an ethereum RPC JSON which has a dynamic structure. No GOLANG struct and map setup I did worked and I am able to get the stateDiff entries (3) but all lower structs seem not to be filled ith any data. So I am able to loop through all the 3 entries but then don't know how to access the values below and when dumping the unmarshal result, I see that GOLANG is not delivering the data anyway into StateDiff
JSON FILE:
{
"jsonrpc":"2.0",
"id":1,
"result":{
"output":"0x0000000000000000000000000000000000000000000000000000000000000001",
"stateDiff":{
"0x0000000000000000000000000000000000000000":{
"balance":{
"*":{
"from":"0x45acecdfadb71366cf",
"to":"0x45aced3909536ccacf"
}
},
"code":"=",
"nonce":"=",
"storage":{
}
},
"0x07865c6e87b9f70255377e024ace6630c1eaa37f":{
"balance":"=",
"code":"=",
"nonce":"=",
"storage":{
"0x86a60af761556602732bbdeaef13ba6e2481f83362d3489389f51353d86a6ac3":{
"*":{
"from":"0x0000000000000000000000000000000000000000000000000000000000000000",
"to":"0x0000000000000000000000000000000000000000000000000000000000002710"
}
},
"0xb0cf6f3c0836765b9dee3d1537458f10fe99447508adc172c1f633ac7352aaa8":{
"*":{
"from":"0x00000000000000000000000000000000000000000000000000092f379a04d2b0",
"to":"0x00000000000000000000000000000000000000000000000000092f379a04aba0"
}
}
}
},
"0x6dbe810e3314546009bd6e1b29f9031211cda5d2":{
"balance":{
"*":{
"from":"0x41c41fc2c0247860",
"to":"0x41c3c66723c4155c"
}
},
"code":"=",
"nonce":{
"*":{
"from":"0x741",
"to":"0x742"
}
},
"storage":{
}
}
},
"trace":[
],
"vmTrace":null
}
}
I have tried to unmarshal the JSON into the following structure (among many) and i can't get the values such as result>stateDiff>0x0000000000000000000000000000000000000000>balance>*>from
Struct below is just one of many i tried. I can't get anything below the entry 0x0000000000000000000000000000000000000000
type structChange struct {
Changes map[string]string `json:"*"`
}
type structStateDiff struct {
Balance *structChange `json:"balance"`
Code string `json:"code"`
Nonce string `json:"nonce"`
Storage map[string]*structChange `json:"storage"`
}
type res_trace_replayTransaction struct {
Jsonrpc string `json:"jsonrpc"`
ID int `json:"id"`
Result struct {
Output string `json:"output"`
StateDiff map[string]*structStateDiff `json:"stateDiff"`
Trace []interface{} `json:"trace"`
VMTrace interface{} `json:"vmTrace"`
} `json:"result"`
}
EDIT:
Code for umarshal
retObj := rpcCall(jstring)
var callResponse res_trace_replayTransaction
err := json.Unmarshal(retObj, &callResponse)
Note that by default the encoding/json package can unmarshal a JSON string into a Go string, and it can unmarshal a JSON object into a Go map or a Go struct. Additionally it can unmarshal any JSON value into an empty interface{}.
Also note that Go is a statically typed language, if you specify a value to be of type T1 then, at runtime, you cannot change it's type to T2. There is just no way to do it, no way to change a value's type.
So if you define a field to be of some struct type, you cannot, by default, unmarshal a JSON string into it. And equally if you define a field to be of type string, you cannot, by default, unmarshal a JSON object into it.
But because JSON itself allows for a dynamic structure the encoding/json package provides two interfaces that give you the ability to customize how the JSON is marshaled and unmarshaled.
So if you have a JSON property (e.g. "balance" or "nonce") that can be either "=" (a string), or { ... } (an object), you will need to declare a custom type that implements the json.Marshaler and json.Unmarshaler interfaces that know how to properly marshal and unmarshal the JSON value.
For example:
type structChange struct {
Changes map[string]string `json:"*"`
}
func (s structChange) MarshalJSON() ([]byte, error) {
// if empty retrun `"="`
if len(s.Changes) == 0 {
return []byte(`"="`), nil
}
// otherwise marshal as is
type T structChange
return json.Marshal(T(s))
}
func (s *structChange) UnmarshalJSON(data []byte) error {
// if `"="`, ignore
if string(data) == `"="` {
return nil
}
// otherwise assume it's a valid object
type T structChange
return json.Unmarshal(data, (*T)(s))
}
NOTE: the temporary type T above is used to avoid a stack overflow caused by an infinite recursive call to the MarshalJSON and UnmarshalJSON methods.
https://go.dev/play/p/yfsTrMozZ2Z
I am trying to retrieve information using Reddit's API. Here is some documentation on their json response, however, I got most of my information by just viewing the link in the browser and pretty-printing the response here.
The following code behaves as intended when the "Replies" field is commented out, but fails when it's not.
[edit] getData() is a function I wrote that uses Go's http Client to get a site response in bytes.
type redditThing struct {
Data struct {
Children []struct {
Data struct {
Permalink string
Subreddit string
Title string
Body string
Replies redditThing
}
}
}
}
func visitLink(link string) {
println("visiting:", link)
var comments []redditThing
if err := json.Unmarshal(getData(link+".json?raw_json=1"), &comments); err != nil {
logError.Println(err)
return
}
}
This throws the following error
json: cannot unmarshal string into Go struct field .Data.Children.Data.Replies.Data.Children.Data.Replies.Data.Children.Data.Replies of type main.redditThing
Any help would be greatly appreciated. Thank you all in advance!
[edit] here a link to some data causing the program to fail
The replies field can be the empty string or a redditThing. Fix by adding an Unmarshal function to handle the empty string:
func (rt *redditThing) UnmarshalJSON(data []byte) error {
// Do nothing if data is the empty string.
if bytes.Equal(data, []byte(`""`)) {
return nil
}
// Prevent recursion by declaring type x with
// same underlying type as redditThing, but
// with no methods.
type x redditThing
return json.Unmarshal(data, (*x)(rt))
}
The x type is used to prevent indefinite recursion. If the final line of the method is json.Unmarshal(data, rt), then json.Unmarshal function will call redditThing.UnmarshalJSON method which calls json.Unmarshal function and so on. Boom!
The statement type x redditThing declares a new type named x with the same underlying type as redditThing. The underlying type is a anonymous struct type. The underlying type has no methods, and crucially, the underlying type does not have the UnmarshalJSON method. This prevents recursion.
I'm working with JSON that returns three different object types 'items','categories' and 'modifiers'. An example of the JSON can be viewed here. I created models for the three types of objects. But when I unmarshal I have selected one of the types to unmarshal the entire JSON to.(I know this cant be the correct way...) I then try to parse out the different items depending on what their type is identified as in the json field 'Type' and then append that object to a slice of the proper type. I am having errors because I don't know how to unmarshal JSON that has different types in it that have different fields.
What is the proper method to unmarshal JSON that contains different objects, each with their own respective fields?
Is the solution to create a "super model" which contains all possible fields and then unmarshal to that?
I'm still fairly new and would appreciate any advice. Thanks!
If you implement json.Unmarshaler, you can define a struct that parses each item type into it's relevant struct.
Example:
// Dynamic represents an item of any type.
type Dynamic struct {
Value interface{}
}
// UnmarshalJSON is called by the json package when we ask it to
// parse something into Dynamic.
func (d *Dynamic) UnmarshalJSON(data []byte) error {
// Parse only the "type" field first.
var meta struct {
Type string
}
if err := json.Unmarshal(data, &meta); err != nil {
return err
}
// Determine which struct to unmarshal into according to "type".
switch meta.Type {
case "product":
d.Value = &Product{}
case "post":
d.Value = &Post{}
default:
return fmt.Errorf("%q is an invalid item type", meta.Type)
}
return json.Unmarshal(data, d.Value)
}
// Product and Post are structs representing two different item types.
type Product struct {
Name string
Price int
}
type Post struct {
Title string
Content string
}
Usage:
func main() {
// Parse a JSON item into Dynamic.
input := `{
"type": "product",
"name": "iPhone",
"price": 1000
}`
var dynamic Dynamic
if err := json.Unmarshal([]byte(input), &dynamic); err != nil {
log.Fatal(err)
}
// Type switch on dynamic.Value to get the parsed struct.
// See https://tour.golang.org/methods/16
switch dynamic.Value.(type) {
case *Product:
log.Println("got a product:", dynamic.Value)
case *Post:
log.Println("got a product:", dynamic.Value)
}
}
Output:
2009/11/10 23:00:00 got a product: &{iPhone 1000}
Try it in the Go Playground.
Tip: if you have a list of dynamic objects, just parse into a slice of Dynamic:
var items []Dynamic
json.Unmarshal(`[{...}, {...}]`, &items)
Example output:
[&{iPhone 1000} &{A Good Post Lorem ipsum...}]
I think https://github.com/mitchellh/mapstructure also fits into your use case.
Given the following JSON
{
"some": "value"
"nested": {
"some": "diffvalue",
"nested": {
"some": "innervalue"
}
}
}
which roughly translates to this struct:
type Envelope struct {
some string `json:"some"`
nested InnerEnvelope `json:"nested"`
}
where InnerEnvelope is: type InnerEnvelope map[string]interface{}
Running a simple json.Unmarshal([]byte value, &target) does not help here, because of the recursive type nature of the original JSON.
I do not know up front how deeply and under which keys the inner maps will exist, so I cannot declare the types upfront.
The idea is, that using map[string]interface{} as the type is not good enough, since I need the values in the InnerEnvelope to be somehow transformed & typed. Details are not important, but image, I need to cast every value inside the NestedEnvelope of a bool type as a string saying "true" or "false" as opposed of having an actual bool type.
I turned to UnmarshalJSON interface to solve this problem. I can easily do it at the top level like so:
func (m *Envelope) UnmarshalJSON(b []byte) error {
var stuff noBoolMap
if err := json.Unmarshal(b, &stuff); err != nil {
return err
}
for key, value := range stuff {
switch value.(type) {
case bool:
stuff[key] = strconv.FormatBool(value.(bool))
}
}
return nil
}
But since the inner json.Unmarshal will already have inner maps parsed as map[string]interface{}, I would need to yet-again to traverse the inner maps, cast them to appropriate type and perform my value transformations.
So my question is: In this case, what is the way this would be approached in Go, and preferably do it in a single-pass?
The expected result of the JSON example above would be:
Envelope {
some: string
nested: InnerEnvelope {
some: string {
nested: InnerEnvelope {
some: string
}
}
}
Given your json, you can do this:
type Envelope struct {
Some string `json:"some"`
Nested json.RawMessage `json:"nested"`
}
json.RawMessage is a rather hidden gem, and more people seem to go for the map[string]interface{}.
Using json.RawMessage will result in the nested json to be represented by this RawMessage, which you then can process again as a normal json (unmarshal it into Envelope).
This is more elegant than the map[string]interface{} approach.
I'm testing out how to unmarshal a json response from an API I'm using.
The json looks something like the following;
body := []byte(`[
{"name":"Name1", "value":100.00},
{"name":"Name2", "value":200.00}
]`)
I've searched around for various ways to do this, but fail to get this to work. The unmarshal returns zero values. I also get an error "unexpected end of JSON input" (I've removed the error handling in the example).
Full code example - https://play.golang.org/p/VMdWuAm6HS
Reference:
https://godoc.org/encoding/json#RawMessage
Golang json Unmarshal "unexpected end of JSON input"
How to unmarshal json into interface{} in golang?
Your input JSON can be modeled with a simple []Obj where Obj is your type:
type Obj struct {
Name string `json:"name"`
Value float32 `json:"value"`
}
Nothing else is required, really:
body := []byte(`[
{"name":"Name1", "value":100.00},
{"name":"Name2", "value":200.00}]`)
var res []Obj
err := json.Unmarshal(body, &res)
fmt.Printf("%#v\n%v\n", res, err)
Output contains the data from the input JSON (try it on the Go Playground):
[]main.Obj{main.Obj{Name:"Name1", Value:100}, main.Obj{Name:"Name2", Value:200}}
<nil>
Back to your code:
Where you're going wrong is that you use this model:
type Obj struct {
Name string `json:"name"`
Value float32 `json:"value"`
}
type Result struct {
Data json.RawMessage
}
var res []Result
But this res variable would model the following JSON:
[
{"Data":{"name":"Name1", "value":100.00}},
{"Data":{"name":"Name2", "value":200.00}}
]
I think you can see the difference: the elements of the array here are JSON Objects with a "Data" field, which then are modeled with your Obj. This input JSON is then properly parsed with your original parsing code, you can try it on the Go Playground:
[{{"name":"Name1", "value":100.00}} {{"name":"Name2", "value":200.00}}]
&main.Obj{Name:"Name1", Value:100}
&main.Obj{Name:"Name2", Value:200}