I'm using Beego framework to build a web application in Go. I have to validate the incoming JSON in an API request.
I can unmarshal the JSON into a struct which works fine, but I want to validate the data as well. For example, if the type doesn't match with the type in struct json.Unmarshal will reutrn an error on the first occurence. I want to validate and get all the errors at once for JSON.
I've tried https://github.com/thedevsaddam/govalidator but the package needs a reference to request object which is not available in the controller of Beego. There are other validators which can validate a struct but i want the json validation as well.
EDIT:
A reference to the request object in beego can be found from the controller's context object as:
func (this *MainController) Post() {
fmt.Println(this.Ctx.Request)
}
But the problem remains same with json unmarshal. If there's any slight mismatch in the type, json.unmarshal will immediately panic. I want to be able to validate the type as well.
Since Golang v1.9 json.Valid([]byte) has been a valid method available from the "encoding/json" package.
Reference:
https://golang.org/pkg/encoding/json/#Valid
We've used go-playground/validator.v8 for a similar purpose. You can define the data validations with the tags that come out of the box (basic stuff like equality, min/max and even has somthing of an expression lang). On top of that you can add your custom validations. It's all in the docs, hope it helps.
There is a method checkValid (in encoding/json package) which can do that. This method, however, is not exposed. So, you can use Unmarshal (in same package) to check if JSON is valid.
I had to do this today so I will share my solution. I created a function using unmarshal that returns true or false based on whether the text returns valid json:
// isJSON checks that the string value of the field can unmarshall into valid json
func isJSON(s string) bool {
var j map[string]interface{}
if err := json.Unmarshal([]byte(s), &j); err != nil {
return false
}
return true
}
Then I used that logic with the validator package to make and register a custom validation tag that I can use in any struct to validate that a field contains json. You can use go playground to check the full solution and experiment:
// isJSON checks that the string value of the field can unmarshall into valid json
func isJSON(fl validator.FieldLevel) bool {
var j map[string]interface{}
if err := json.Unmarshal([]byte(fl.Field().String()), &j); err != nil {
return false
}
return true
}
// register the function with json tag:
validate.RegisterValidation("json", isJSON)
Related
Please forgive my question, I'm new to Golang and possibly have the wrong approach.
I'm currently implementing a Terraform provider for an internal service.
As probably expected, that requires unmarshalling JSON data in to pre-defined Struct Types, e.g:
type SomeTypeIveDefined struct {
ID string `json:"id"`
Name String `json:"name"`
}
I've got myself in to a situation where I have a lot of duplicate code that looks like this
res := r.(*http.Response)
var tempThing SomeTypeIveDefined
dec := json.NewDecoder(res.Body)
err := dec.Decode(&tempThing)
In an effort to reduce duplication, I decided what I wanted to do was create a function which does the JSON unmarshalling, but takes in the Struct Type as a parameter.
I've trawled through several StackOverflow articles and Google Groups trying to make sense of some of the answers around using the reflect package, but I've not had much success in using it.
My latest attempt was using reflect.StructOf and passing in a set of StructFields, but that still seems to require using myReflectedStruct.Field(0) rather than myReflectedStruct.ID.
I suspect there may be no way until something like Generics are widely available in Golang.
I considered perhaps an interface for the structs which requires implementing an unmarshal method, then I could pass the interface to the function and call the unmarshal method. But then I'm still implementing unmarshal on all the structs anyway.
I'm just wondering what suggestions there may be for achieving what I'm after, please?
Create a helper function with the repeated code. Pass the destination value as a pointer.
func decode(r *http.Repsonse, v interface{}) error {
return json.NewDecoder(res.Body).Decode(v)
}
Call the helper function with a pointer to your thing:
var tempThing SomeTypeIveDefined
err := deocde(r, &tempThing)
You can do this with interfaces:
func decodeResponse(r *http.Response, dest interface{}) error {
dec := json.NewDecoder(r.Body)
return dec.Decode(dest)
}
func handler(...) {
res := r.(*http.Response)
var tempThing SomeTypeIveDefined
if err:=decodeResponse(res,&tempThing); err!=nil {
// handle err
}
...
}
You don't need to implement an unmarshal for the structs, because the stdlib decoder will use reflection to set the struct fields.
I am trying to retrieve information using Reddit's API. Here is some documentation on their json response, however, I got most of my information by just viewing the link in the browser and pretty-printing the response here.
The following code behaves as intended when the "Replies" field is commented out, but fails when it's not.
[edit] getData() is a function I wrote that uses Go's http Client to get a site response in bytes.
type redditThing struct {
Data struct {
Children []struct {
Data struct {
Permalink string
Subreddit string
Title string
Body string
Replies redditThing
}
}
}
}
func visitLink(link string) {
println("visiting:", link)
var comments []redditThing
if err := json.Unmarshal(getData(link+".json?raw_json=1"), &comments); err != nil {
logError.Println(err)
return
}
}
This throws the following error
json: cannot unmarshal string into Go struct field .Data.Children.Data.Replies.Data.Children.Data.Replies.Data.Children.Data.Replies of type main.redditThing
Any help would be greatly appreciated. Thank you all in advance!
[edit] here a link to some data causing the program to fail
The replies field can be the empty string or a redditThing. Fix by adding an Unmarshal function to handle the empty string:
func (rt *redditThing) UnmarshalJSON(data []byte) error {
// Do nothing if data is the empty string.
if bytes.Equal(data, []byte(`""`)) {
return nil
}
// Prevent recursion by declaring type x with
// same underlying type as redditThing, but
// with no methods.
type x redditThing
return json.Unmarshal(data, (*x)(rt))
}
The x type is used to prevent indefinite recursion. If the final line of the method is json.Unmarshal(data, rt), then json.Unmarshal function will call redditThing.UnmarshalJSON method which calls json.Unmarshal function and so on. Boom!
The statement type x redditThing declares a new type named x with the same underlying type as redditThing. The underlying type is a anonymous struct type. The underlying type has no methods, and crucially, the underlying type does not have the UnmarshalJSON method. This prevents recursion.
I learned from the gin doc that you can bind json to a struct like
type Login struct {
User string `form:"user" json:"user" binding:"required"`
Password string `form:"password" json:"password" binding:"required"`
}
func main() {
router := gin.Default()
// Example for binding JSON ({"user": "manu", "password": "123"})
router.POST("/loginJSON", func(c *gin.Context) {
var json Login
if c.BindJSON(&json) == nil {
if json.User == "manu" && json.Password == "123" {
c.JSON(http.StatusOK, gin.H{"status": "you are logged in"})
} else {
c.JSON(http.StatusUnauthorized, gin.H{"status": "unauthorized"})
}
}
})
}
You always have to build a struct to bind JSON.
But if there is a very complex JSON data, I only what to get part of it, to create a complex struct is a big burden. Can avoid id and parse it directly?
You don't need to create a struct for all the fields present in the JSON response, only the fields you are interested in. Any other fields present in the response will be ignored when unmarshaling.
You can also unmarshal to a generic map[string]interface{} but this is only really useful for truly dynamic data. If you know the format of the response ahead of time you will nearly always be best to create a custom struct to get type safety and avoid continual nil checks when accessing the map. Additionally, a targeted struct avoids storing unnecessary values when JSON in unmarshalled.
You can use the JSON to Go tool to help quickly create a struct definition from a JSON response. You could then easily strip out all the fields you don't need.
I have a struct VideoInfo that has a key in it called embedCode. The API I am querying returns the embed code as embed_code. During unmarshalling the response how do I ensure embed_code goes into embedCode?
Also is there an easy way to take a large json string and automatically turn it into a struct, or can one only use a map?
With respect to remapping the field names use the corresponding annotation in the structure declaration:
type VideoInfo struct {
EmbedCode string `json:"embed_code"`
}
The marshaller/un-marshaller will only process public field, so you need to capitalise the field name.
With respect to converting the whole structure, yes it is easy. Declare an instance to un-marshal into and pass a reference to the json.Unmarshal method (from a test):
data, _ := json.Marshal(request)
var resp response.VideoInfo
if err := json.Unmarshal(data, &resp); err != nil {
t.Errorf("unexpected error, %v", err)
}
At first, struct's field must be start from capital letter to be public. So you need something like that:
type VideoInfo struct {
EmbedCode string `json:"embed_code"`
}
And look at documentation for more info.
I apologise in advance for a very long question. Hopefully you'll bear with me.
I'm working with the goweb library, and experimenting with the example web app.
I've been trying to modify the RESTful example code, which defines a Thing as:
type Thing struct {
Id string
Text string
}
A Thing is created by sending an HTTP Post request, with an appropriate JSON body, to http://localhost:9090/things. This is handled in the example code in the Create function, specifically the lines:
dataMap := data.(map[string]interface{})
thing := new(Thing)
thing.Id = dataMap["Id"].(string)
thing.Text = dataMap["Text"].(string)
This is all well and good, and I can run the example server (which listens on http://localhost:9090/) and the server operates as expected.
For example:
curl -X POST -H "Content-Type: application/json" -d '{"Id":"TestId","Text":"TestText"}' http://localhost:9090/things
returns with no error, and then I GET that Thing with
curl http://localhost:9090/things/TestId
and it returns
{"d":{"Id":"TestId","Text":"TestText"},"s":200}
So far, so good.
Now, I would like to modify the Thing type, and add a custom ThingText type, like so:
type ThingText struct {
Title string
Body string
}
type Thing struct {
Id string
Text ThingText
}
This in itself isn't an issue, and I can modify the Create function like so:
thing := new(Thing)
thing.Id = dataMap["Id"].(string)
thing.Text.Title = dataMap["Title"].(string)
thing.Text.Body = dataMap["Body"].(string)
and the run the previous curl POST request with the JSON set to:
{"Id":"TestId","Title":"TestTitle","Title":"TestBody"}
and it returns with no error.
Once again I can GET the Thing URL and it returns:
{"d":{"Id":"TestId","Text":{"Title":"TestTitle","Body":"TestBody"}},"s":200}
Again, so far, so good.
Now, my question:
how do I modify the Create function to allow me to POST complex JSON to it?
For example, that last returned JSON string above includes {"Id":"TestId","Text":{"Title":"TestTitle","Body":"TestBody"}}. I'd like to be able to POST that exact JSON to the endpoint and have the Thing created.
I've followed the code back, and it seems that the data variable is of type Context.RequestData() from https://github.com/stretchr/goweb/context, and the internal Map seems to be of type Object.Map from https://github.com/stretchr/stew/, described as "a map[string]interface{} with additional helpful functionality." in particular, I noticed "Supports dot syntax to set deep values."
I can't work out how I can set up the thing.Text.Title = dataMap... statement so that the correct JSON field is parsed into it. I can't seem to use anything other than string types in the dataMap, and if I try that JSON it gives an error similar to:
http: panic serving 127.0.0.1:59113: interface conversion: interface is nil, not string
Once again, sorry for the ridiculously long question. I really appreciate you reading, and any help you may have to offer. Thanks!
As the JSON package documentation and JSON and Go introduction describe, JSON data can be parsed either generically through interface{} / string maps or unmarshalling directly to the struct types.
The example code you linked to and you based your changes on seems to use the generic string-map approach, dataMap := data.(map[string]interface{}).
As your desired JSON data is an object in an object, it’s simply a map within a map.
So you should be able to
dataMap := data.(map[string]interface{})
subthingMap := dataMap["Text"].(map[string]interface{})
thing.Text.Title = subthingMap["Title"].(string)
thing.Text.Body = subthingMap["Body"].(string)
I’m not sure why that code uses casts and generic types over type-safe unmarshalling directly from JSON to struct types (abstraction I guess). Using the json packages unmarshalling to struct types would go something like
type ThingText struct {
Title string
Body string
}
type Thing struct {
Id string
Text ThingText
}
…
decoder := json.NewDecoder(body)
var thingobj Thing
for {
if err := decoder.Decode(&thingobj); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
fmt.Println(thingobj)
}
where body is a io.Reader - in simple/most cases from http.Response.Body.