I learned from the gin doc that you can bind json to a struct like
type Login struct {
User string `form:"user" json:"user" binding:"required"`
Password string `form:"password" json:"password" binding:"required"`
}
func main() {
router := gin.Default()
// Example for binding JSON ({"user": "manu", "password": "123"})
router.POST("/loginJSON", func(c *gin.Context) {
var json Login
if c.BindJSON(&json) == nil {
if json.User == "manu" && json.Password == "123" {
c.JSON(http.StatusOK, gin.H{"status": "you are logged in"})
} else {
c.JSON(http.StatusUnauthorized, gin.H{"status": "unauthorized"})
}
}
})
}
You always have to build a struct to bind JSON.
But if there is a very complex JSON data, I only what to get part of it, to create a complex struct is a big burden. Can avoid id and parse it directly?
You don't need to create a struct for all the fields present in the JSON response, only the fields you are interested in. Any other fields present in the response will be ignored when unmarshaling.
You can also unmarshal to a generic map[string]interface{} but this is only really useful for truly dynamic data. If you know the format of the response ahead of time you will nearly always be best to create a custom struct to get type safety and avoid continual nil checks when accessing the map. Additionally, a targeted struct avoids storing unnecessary values when JSON in unmarshalled.
You can use the JSON to Go tool to help quickly create a struct definition from a JSON response. You could then easily strip out all the fields you don't need.
Related
I am very new to GOLANG.
I have been trying for quite some time now to unmarshal an ethereum RPC JSON which has a dynamic structure. No GOLANG struct and map setup I did worked and I am able to get the stateDiff entries (3) but all lower structs seem not to be filled ith any data. So I am able to loop through all the 3 entries but then don't know how to access the values below and when dumping the unmarshal result, I see that GOLANG is not delivering the data anyway into StateDiff
JSON FILE:
{
"jsonrpc":"2.0",
"id":1,
"result":{
"output":"0x0000000000000000000000000000000000000000000000000000000000000001",
"stateDiff":{
"0x0000000000000000000000000000000000000000":{
"balance":{
"*":{
"from":"0x45acecdfadb71366cf",
"to":"0x45aced3909536ccacf"
}
},
"code":"=",
"nonce":"=",
"storage":{
}
},
"0x07865c6e87b9f70255377e024ace6630c1eaa37f":{
"balance":"=",
"code":"=",
"nonce":"=",
"storage":{
"0x86a60af761556602732bbdeaef13ba6e2481f83362d3489389f51353d86a6ac3":{
"*":{
"from":"0x0000000000000000000000000000000000000000000000000000000000000000",
"to":"0x0000000000000000000000000000000000000000000000000000000000002710"
}
},
"0xb0cf6f3c0836765b9dee3d1537458f10fe99447508adc172c1f633ac7352aaa8":{
"*":{
"from":"0x00000000000000000000000000000000000000000000000000092f379a04d2b0",
"to":"0x00000000000000000000000000000000000000000000000000092f379a04aba0"
}
}
}
},
"0x6dbe810e3314546009bd6e1b29f9031211cda5d2":{
"balance":{
"*":{
"from":"0x41c41fc2c0247860",
"to":"0x41c3c66723c4155c"
}
},
"code":"=",
"nonce":{
"*":{
"from":"0x741",
"to":"0x742"
}
},
"storage":{
}
}
},
"trace":[
],
"vmTrace":null
}
}
I have tried to unmarshal the JSON into the following structure (among many) and i can't get the values such as result>stateDiff>0x0000000000000000000000000000000000000000>balance>*>from
Struct below is just one of many i tried. I can't get anything below the entry 0x0000000000000000000000000000000000000000
type structChange struct {
Changes map[string]string `json:"*"`
}
type structStateDiff struct {
Balance *structChange `json:"balance"`
Code string `json:"code"`
Nonce string `json:"nonce"`
Storage map[string]*structChange `json:"storage"`
}
type res_trace_replayTransaction struct {
Jsonrpc string `json:"jsonrpc"`
ID int `json:"id"`
Result struct {
Output string `json:"output"`
StateDiff map[string]*structStateDiff `json:"stateDiff"`
Trace []interface{} `json:"trace"`
VMTrace interface{} `json:"vmTrace"`
} `json:"result"`
}
EDIT:
Code for umarshal
retObj := rpcCall(jstring)
var callResponse res_trace_replayTransaction
err := json.Unmarshal(retObj, &callResponse)
Note that by default the encoding/json package can unmarshal a JSON string into a Go string, and it can unmarshal a JSON object into a Go map or a Go struct. Additionally it can unmarshal any JSON value into an empty interface{}.
Also note that Go is a statically typed language, if you specify a value to be of type T1 then, at runtime, you cannot change it's type to T2. There is just no way to do it, no way to change a value's type.
So if you define a field to be of some struct type, you cannot, by default, unmarshal a JSON string into it. And equally if you define a field to be of type string, you cannot, by default, unmarshal a JSON object into it.
But because JSON itself allows for a dynamic structure the encoding/json package provides two interfaces that give you the ability to customize how the JSON is marshaled and unmarshaled.
So if you have a JSON property (e.g. "balance" or "nonce") that can be either "=" (a string), or { ... } (an object), you will need to declare a custom type that implements the json.Marshaler and json.Unmarshaler interfaces that know how to properly marshal and unmarshal the JSON value.
For example:
type structChange struct {
Changes map[string]string `json:"*"`
}
func (s structChange) MarshalJSON() ([]byte, error) {
// if empty retrun `"="`
if len(s.Changes) == 0 {
return []byte(`"="`), nil
}
// otherwise marshal as is
type T structChange
return json.Marshal(T(s))
}
func (s *structChange) UnmarshalJSON(data []byte) error {
// if `"="`, ignore
if string(data) == `"="` {
return nil
}
// otherwise assume it's a valid object
type T structChange
return json.Unmarshal(data, (*T)(s))
}
NOTE: the temporary type T above is used to avoid a stack overflow caused by an infinite recursive call to the MarshalJSON and UnmarshalJSON methods.
https://go.dev/play/p/yfsTrMozZ2Z
I'm new to Go so please bear with me if this is a trivial problem. I am using a home grown "type registry" to map type names to their type, so as to generate them dynamically based on use cases that point to the various type names (I'm basically trying for a simple solution to polymorphic Aggregation JSON response structures in Elasticsearch, but of course this could apply to many other dynamic/polymorphic situations).
I'm using the solution provided by dolmen in this question: is there a way to create an instance of a struct from a string? :
var typeRegistry = make(map[string]reflect.Type)
func registerType(typedNil interface{}) {
t := reflect.TypeOf(typedNil).Elem()
typeRegistry[t.Name()] = t
}
func init() {
registerType((*playlistIDAggregation)(nil))
registerType((*srcIDAggregation)(nil))
registerType((*assetIDAggregation)(nil))
}
func makeInstance(name string) interface{} {
return reflect.New(typeRegistry[name]).Elem().Interface()
}
I then want to use my dynamically generated struct as the target for the JSON unmarshalling of the Aggregations node in my ES response:
playlistIDAgg := makeInstance("playlistIDAggregation")
err = json.Unmarshal(esResponse.Aggregations, &playlistIDAgg)
This isn't working like I want it to, as the Unmarshal is trying to unmarshall into an empty interface instead of the underlying struct type. it's putting the data under "data" nodes in the playlistIDAgg variable, and those data fields are of course map[string]interface{}. Am I just missing the way to type assert my playlistIDAgg interface or is there a better way to do this?
EDIT---
The questions in the comments made me realize an edit to this question was long overdue.
In my particular case, the structs I defined, to bind to my Bucket aggregations returned by Elasticsearch, have a similar structure and vary only by their root JSON tag, which ES uses to name the aggregation and strongly type it. E.g.
type <name>Aggregation struct {
Agg BucketAggregationWithCamIDCardinality `json:"<name>"`
}
So, rather than a type registry, my particular problem could be solved by dynamically setting the JSON tag on the struct based on the particular use case.
In addition, a heavier but more robust option would be to leverage Oliver Eilhard's Elasticsearch Go client lib, called Elastic, which has built-in support for all the ES aggregation response structures:
https://github.com/olivere/elastic/
I changed makeInstance function by getting address of element and add target structs with exposed fields.
func makeInstance(name string) interface{} {
return reflect.New(typeRegistry[name]).Elem().Addr().Interface()
}
Here is the working code
package main
import (
"encoding/json"
"fmt"
"reflect"
)
type playlistIDAggregation struct {
PlaylistID string
}
type srcIDAggregation struct {
SrcID string
}
type assetIDAggregation struct {
AssetID string
}
var typeRegistry = make(map[string]reflect.Type)
func registerType(typedNil interface{}) {
t := reflect.TypeOf(typedNil).Elem()
typeRegistry[t.Name()] = t
}
func init() {
registerType((*playlistIDAggregation)(nil))
registerType((*srcIDAggregation)(nil))
registerType((*assetIDAggregation)(nil))
}
func makeInstance(name string) interface{} {
return reflect.New(typeRegistry[name]).Elem().Addr().Interface()
}
func main() {
playlistIDAgg := makeInstance("playlistIDAggregation")
fmt.Printf("Type = %[1]T => %#[1]v\n", playlistIDAgg)
err := json.Unmarshal([]byte(`{"PlayListID": "dummy-id"}`), &playlistIDAgg)
if err != nil {
panic(err)
}
fmt.Printf("Type = %[1]T => %#[1]v\n", playlistIDAgg)
}
https://play.golang.org/p/dn19_iG5Xjz
I'm using Beego framework to build a web application in Go. I have to validate the incoming JSON in an API request.
I can unmarshal the JSON into a struct which works fine, but I want to validate the data as well. For example, if the type doesn't match with the type in struct json.Unmarshal will reutrn an error on the first occurence. I want to validate and get all the errors at once for JSON.
I've tried https://github.com/thedevsaddam/govalidator but the package needs a reference to request object which is not available in the controller of Beego. There are other validators which can validate a struct but i want the json validation as well.
EDIT:
A reference to the request object in beego can be found from the controller's context object as:
func (this *MainController) Post() {
fmt.Println(this.Ctx.Request)
}
But the problem remains same with json unmarshal. If there's any slight mismatch in the type, json.unmarshal will immediately panic. I want to be able to validate the type as well.
Since Golang v1.9 json.Valid([]byte) has been a valid method available from the "encoding/json" package.
Reference:
https://golang.org/pkg/encoding/json/#Valid
We've used go-playground/validator.v8 for a similar purpose. You can define the data validations with the tags that come out of the box (basic stuff like equality, min/max and even has somthing of an expression lang). On top of that you can add your custom validations. It's all in the docs, hope it helps.
There is a method checkValid (in encoding/json package) which can do that. This method, however, is not exposed. So, you can use Unmarshal (in same package) to check if JSON is valid.
I had to do this today so I will share my solution. I created a function using unmarshal that returns true or false based on whether the text returns valid json:
// isJSON checks that the string value of the field can unmarshall into valid json
func isJSON(s string) bool {
var j map[string]interface{}
if err := json.Unmarshal([]byte(s), &j); err != nil {
return false
}
return true
}
Then I used that logic with the validator package to make and register a custom validation tag that I can use in any struct to validate that a field contains json. You can use go playground to check the full solution and experiment:
// isJSON checks that the string value of the field can unmarshall into valid json
func isJSON(fl validator.FieldLevel) bool {
var j map[string]interface{}
if err := json.Unmarshal([]byte(fl.Field().String()), &j); err != nil {
return false
}
return true
}
// register the function with json tag:
validate.RegisterValidation("json", isJSON)
I'm working with JSON that returns three different object types 'items','categories' and 'modifiers'. An example of the JSON can be viewed here. I created models for the three types of objects. But when I unmarshal I have selected one of the types to unmarshal the entire JSON to.(I know this cant be the correct way...) I then try to parse out the different items depending on what their type is identified as in the json field 'Type' and then append that object to a slice of the proper type. I am having errors because I don't know how to unmarshal JSON that has different types in it that have different fields.
What is the proper method to unmarshal JSON that contains different objects, each with their own respective fields?
Is the solution to create a "super model" which contains all possible fields and then unmarshal to that?
I'm still fairly new and would appreciate any advice. Thanks!
If you implement json.Unmarshaler, you can define a struct that parses each item type into it's relevant struct.
Example:
// Dynamic represents an item of any type.
type Dynamic struct {
Value interface{}
}
// UnmarshalJSON is called by the json package when we ask it to
// parse something into Dynamic.
func (d *Dynamic) UnmarshalJSON(data []byte) error {
// Parse only the "type" field first.
var meta struct {
Type string
}
if err := json.Unmarshal(data, &meta); err != nil {
return err
}
// Determine which struct to unmarshal into according to "type".
switch meta.Type {
case "product":
d.Value = &Product{}
case "post":
d.Value = &Post{}
default:
return fmt.Errorf("%q is an invalid item type", meta.Type)
}
return json.Unmarshal(data, d.Value)
}
// Product and Post are structs representing two different item types.
type Product struct {
Name string
Price int
}
type Post struct {
Title string
Content string
}
Usage:
func main() {
// Parse a JSON item into Dynamic.
input := `{
"type": "product",
"name": "iPhone",
"price": 1000
}`
var dynamic Dynamic
if err := json.Unmarshal([]byte(input), &dynamic); err != nil {
log.Fatal(err)
}
// Type switch on dynamic.Value to get the parsed struct.
// See https://tour.golang.org/methods/16
switch dynamic.Value.(type) {
case *Product:
log.Println("got a product:", dynamic.Value)
case *Post:
log.Println("got a product:", dynamic.Value)
}
}
Output:
2009/11/10 23:00:00 got a product: &{iPhone 1000}
Try it in the Go Playground.
Tip: if you have a list of dynamic objects, just parse into a slice of Dynamic:
var items []Dynamic
json.Unmarshal(`[{...}, {...}]`, &items)
Example output:
[&{iPhone 1000} &{A Good Post Lorem ipsum...}]
I think https://github.com/mitchellh/mapstructure also fits into your use case.
I have a struct like this:
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
But even if the instance of MyStruct is entirely empty (meaning, all values are default), it's being serialized as:
"data":{}
I know that the encoding/json docs specify that "empty" fields are:
false, 0, any nil pointer or interface value, and any array,
slice, map, or string of length zero
but with no consideration for a struct with all empty/default values. All of its fields are also tagged with omitempty, but this has no effect.
How can I get the JSON package to not marshal my field that is an empty struct?
As the docs say, "any nil pointer." -- make the struct a pointer. Pointers have obvious "empty" values: nil.
Fix - define the type with a struct pointer field:
type Result struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
Then a value like this:
result := Result{}
Will marshal as:
{}
Explanation: Notice the *MyStruct in our type definition. JSON serialization doesn't care whether it is a pointer or not -- that's a runtime detail. So making struct fields into pointers only has implications for compiling and runtime).
Just note that if you do change the field type from MyStruct to *MyStruct, you will need pointers to struct values to populate it, like so:
Data: &MyStruct{ /* values */ }
As #chakrit mentioned in a comment, you can't get this to work by implementing json.Marshaler on MyStruct, and implementing a custom JSON marshalling function on every struct that uses it can be a lot more work. It really depends on your use case as to whether it's worth the extra work or whether you're prepared to live with empty structs in your JSON, but here's the pattern I use applied to Result:
type Result struct {
Data MyStruct
Status string
Reason string
}
func (r Result) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}{
Data: &r.Data,
Status: r.Status,
Reason: r.Reason,
})
}
func (r *Result) UnmarshalJSON(b []byte) error {
decoded := new(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
})
err := json.Unmarshal(b, decoded)
if err == nil {
r.Data = decoded.Data
r.Status = decoded.Status
r.Reason = decoded.Reason
}
return err
}
If you have huge structs with many fields this can become tedious, especially changing a struct's implementation later, but short of rewriting the whole json package to suit your needs (not a good idea), this is pretty much the only way I can think of getting this done while still keeping a non-pointer MyStruct in there.
Also, you don't have to use inline structs, you can create named ones. I use LiteIDE with code completion though, so I prefer inline to avoid clutter.
Data is an initialized struct, so it isn't considered empty because encoding/json only looks at the immediate value, not the fields inside the struct.
Unfortunately, returning nil from json.Marshaler doesn't currently work:
func (_ MyStruct) MarshalJSON() ([]byte, error) {
if empty {
return nil, nil // unexpected end of JSON input
}
// ...
}
You could give Result a marshaler as well, but it's not worth the effort.
The only option, as Matt suggests, is to make Data a pointer and set the value to nil.
There is an outstanding Golang proposal for this feature which has been active for over 4 years, so at this point, it is safe to assume that it will not make it into the standard library anytime soon. As #Matt pointed out, the traditional approach is to convert the structs to pointers-to-structs. If this approach is infeasible (or impractical), then an alternative is to use an alternate json encoder which does support omitting zero value structs.
I created a mirror of the Golang json library (clarketm/json) with added support for omitting zero value structs when the omitempty tag is applied. This library detects zeroness in a similar manner to the popular YAML encoder go-yaml by recursively checking the public struct fields.
e.g.
$ go get -u "github.com/clarketm/json"
import (
"fmt"
"github.com/clarketm/json" // drop-in replacement for `encoding/json`
)
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
j, _ := json.Marshal(&Result{
Status: "204",
Reason: "No Content",
})
fmt.Println(string(j))
// Note: `data` is omitted from the resultant json.
{
"status": "204"
"reason": "No Content"
}