func deserialize(request *http.Request,typ reflect.Type) (interface{}, *httpNet.HandlerError){
data,e:=ioutil.ReadAll(request.Body)
fmt.Println(string(data))
if e !=nil{
return nil,&httpNet.HandlerError{e,"could not read request",http.StatusBadRequest}
}
v:=typ.Elem()
payload:=reflect.New(v).Elem().Interface()
eaa:= json.NewDecoder(request.Body).Decode(payload)
if e!=nil{
fmt.Println(eaa.Error())
}
fmt.Println(payload)
fmt.Println(reflect.ValueOf(payload)
)
return payload,nil
}
to call it:
r,_:= deserialize(request,reflect.TypeOf(&testData{}))
It does not throw errors and looks completely valid operation to me , but the result is an empty structure of expecting type.
Whats the problem with that?
The problem is that you are passing a non pointer instance of the type:
payload:=reflect.New(v).Elem().Interface()
Means "allocate a new pointer to the type, then take the value of it, and extract it as interface{}.
You should just keep it at:
payload:=reflect.New(v).Interface()
BTW It's also redundant that you are passing the type of a pointer, extracting its Elem(), then allocating a pointer. Do something like this:
if type.Kind() == reflect.Ptr {
typ = typ.Elem()
}
payload := reflect.New(typ).Interface()
then you can pass both pointers and non pointers to the function.
Edit: Working playground example: http://play.golang.org/p/TPafxcpIU5
Related
Please forgive my question, I'm new to Golang and possibly have the wrong approach.
I'm currently implementing a Terraform provider for an internal service.
As probably expected, that requires unmarshalling JSON data in to pre-defined Struct Types, e.g:
type SomeTypeIveDefined struct {
ID string `json:"id"`
Name String `json:"name"`
}
I've got myself in to a situation where I have a lot of duplicate code that looks like this
res := r.(*http.Response)
var tempThing SomeTypeIveDefined
dec := json.NewDecoder(res.Body)
err := dec.Decode(&tempThing)
In an effort to reduce duplication, I decided what I wanted to do was create a function which does the JSON unmarshalling, but takes in the Struct Type as a parameter.
I've trawled through several StackOverflow articles and Google Groups trying to make sense of some of the answers around using the reflect package, but I've not had much success in using it.
My latest attempt was using reflect.StructOf and passing in a set of StructFields, but that still seems to require using myReflectedStruct.Field(0) rather than myReflectedStruct.ID.
I suspect there may be no way until something like Generics are widely available in Golang.
I considered perhaps an interface for the structs which requires implementing an unmarshal method, then I could pass the interface to the function and call the unmarshal method. But then I'm still implementing unmarshal on all the structs anyway.
I'm just wondering what suggestions there may be for achieving what I'm after, please?
Create a helper function with the repeated code. Pass the destination value as a pointer.
func decode(r *http.Repsonse, v interface{}) error {
return json.NewDecoder(res.Body).Decode(v)
}
Call the helper function with a pointer to your thing:
var tempThing SomeTypeIveDefined
err := deocde(r, &tempThing)
You can do this with interfaces:
func decodeResponse(r *http.Response, dest interface{}) error {
dec := json.NewDecoder(r.Body)
return dec.Decode(dest)
}
func handler(...) {
res := r.(*http.Response)
var tempThing SomeTypeIveDefined
if err:=decodeResponse(res,&tempThing); err!=nil {
// handle err
}
...
}
You don't need to implement an unmarshal for the structs, because the stdlib decoder will use reflection to set the struct fields.
I am trying to retrieve information using Reddit's API. Here is some documentation on their json response, however, I got most of my information by just viewing the link in the browser and pretty-printing the response here.
The following code behaves as intended when the "Replies" field is commented out, but fails when it's not.
[edit] getData() is a function I wrote that uses Go's http Client to get a site response in bytes.
type redditThing struct {
Data struct {
Children []struct {
Data struct {
Permalink string
Subreddit string
Title string
Body string
Replies redditThing
}
}
}
}
func visitLink(link string) {
println("visiting:", link)
var comments []redditThing
if err := json.Unmarshal(getData(link+".json?raw_json=1"), &comments); err != nil {
logError.Println(err)
return
}
}
This throws the following error
json: cannot unmarshal string into Go struct field .Data.Children.Data.Replies.Data.Children.Data.Replies.Data.Children.Data.Replies of type main.redditThing
Any help would be greatly appreciated. Thank you all in advance!
[edit] here a link to some data causing the program to fail
The replies field can be the empty string or a redditThing. Fix by adding an Unmarshal function to handle the empty string:
func (rt *redditThing) UnmarshalJSON(data []byte) error {
// Do nothing if data is the empty string.
if bytes.Equal(data, []byte(`""`)) {
return nil
}
// Prevent recursion by declaring type x with
// same underlying type as redditThing, but
// with no methods.
type x redditThing
return json.Unmarshal(data, (*x)(rt))
}
The x type is used to prevent indefinite recursion. If the final line of the method is json.Unmarshal(data, rt), then json.Unmarshal function will call redditThing.UnmarshalJSON method which calls json.Unmarshal function and so on. Boom!
The statement type x redditThing declares a new type named x with the same underlying type as redditThing. The underlying type is a anonymous struct type. The underlying type has no methods, and crucially, the underlying type does not have the UnmarshalJSON method. This prevents recursion.
I'm using Beego framework to build a web application in Go. I have to validate the incoming JSON in an API request.
I can unmarshal the JSON into a struct which works fine, but I want to validate the data as well. For example, if the type doesn't match with the type in struct json.Unmarshal will reutrn an error on the first occurence. I want to validate and get all the errors at once for JSON.
I've tried https://github.com/thedevsaddam/govalidator but the package needs a reference to request object which is not available in the controller of Beego. There are other validators which can validate a struct but i want the json validation as well.
EDIT:
A reference to the request object in beego can be found from the controller's context object as:
func (this *MainController) Post() {
fmt.Println(this.Ctx.Request)
}
But the problem remains same with json unmarshal. If there's any slight mismatch in the type, json.unmarshal will immediately panic. I want to be able to validate the type as well.
Since Golang v1.9 json.Valid([]byte) has been a valid method available from the "encoding/json" package.
Reference:
https://golang.org/pkg/encoding/json/#Valid
We've used go-playground/validator.v8 for a similar purpose. You can define the data validations with the tags that come out of the box (basic stuff like equality, min/max and even has somthing of an expression lang). On top of that you can add your custom validations. It's all in the docs, hope it helps.
There is a method checkValid (in encoding/json package) which can do that. This method, however, is not exposed. So, you can use Unmarshal (in same package) to check if JSON is valid.
I had to do this today so I will share my solution. I created a function using unmarshal that returns true or false based on whether the text returns valid json:
// isJSON checks that the string value of the field can unmarshall into valid json
func isJSON(s string) bool {
var j map[string]interface{}
if err := json.Unmarshal([]byte(s), &j); err != nil {
return false
}
return true
}
Then I used that logic with the validator package to make and register a custom validation tag that I can use in any struct to validate that a field contains json. You can use go playground to check the full solution and experiment:
// isJSON checks that the string value of the field can unmarshall into valid json
func isJSON(fl validator.FieldLevel) bool {
var j map[string]interface{}
if err := json.Unmarshal([]byte(fl.Field().String()), &j); err != nil {
return false
}
return true
}
// register the function with json tag:
validate.RegisterValidation("json", isJSON)
I'm writing a wrapper to map with some additional functionality I need. Some of the most important things is the ability to marshal and unmarshal the data while retaining genericity. I managed to write an marshaller using the encoding/gob encoder, but since it would be nice if the marshalled data was human readable, I decided to code another implementation with JSON.
I got gob to encode from and decode to generic interface variables neatly by passing it a implementation object instance with Register(). (This resource helped me with the details! http://www.funcmain.com/gob_encoding_an_interface)
However, JSON doesn't have Register(). Let's say we have a value of type
type ConcreteImplementation struct {
FieldA string
FieldB string
}
func (c ConcreteImplementation) Key() string {
return c.FieldA // ConcreteImplementation implements genericValue
}
in a variable of type
type genericValue interface {
Key() string
}
When I marshal that, it outputs JSON like this:
{"FieldA":"foo","FieldB":"bar"}
And when I try to unmarshal that again into a variable of type genericValue, it says:
panic: interface conversion: map[string]interface {} is not genericValue: missing method Key EDIT: Oops, actually it says this!
Error with decoding! json: cannot unmarshal object into Go value of genericValue
Quite obviously, it tries to marshal the data like it says here: http://blog.golang.org/json-and-go (See 'Generic JSON with interface{}')
How can I get it to try to fit the data to an specific implementation, like gob decoder would try if the implementation is Register()ed? Register() was godsend, it allowed to marshal and unmarshal generically like it was nothing. How do I get JSON to do the same thing?
What if your types implemented the Unmarshaler?
Here is a small demo.
Or the same code here:
type ConcreteImplementation struct {
FieldA string
FieldB string
}
func (c ConcreteImplementation) Key() string {
return c.FieldA // ConcreteImplementation implements genericValue
}
// implementing json.Unmarshaler
func (c *ConcreteImplementation) UnmarshalJSON(j []byte) error {
m := make(map[string]string)
err := json.Unmarshal(j, &m)
if err != nil {
return err
}
if v, ok := m["FieldA"]; ok {
c.FieldA = v
}
if v, ok := m["FieldB"]; ok {
c.FieldB = v
}
return nil
}
type genericValue interface {
Key() string
json.Unmarshaler
}
func decode(jsonStr []byte, v genericValue) error {
return json.Unmarshal(jsonStr, v)
}
With this you can pass a genericValue to json.Unmarshal.
Allright, got it to work finally. This question provided the answer. Why does json.Unmarshal return a map instead of the expected struct?
"You've passed to json a pointer to an abstract interface. You should simply pass a pointer to Ping as an abstract interface" - This applied to my situation too. (For some reason a pointer TO an abstract interface was enough for gob package. It seems that I have to study Go interfaces and reflection some more to understand why...)
I won't still mark this as solved question, if someone has a better answer.
This example from the json.Unmarshal docs (slightly modified for simplicity to use Animal instead of []Animal) works, no errors:
Playground link of working example
// ...
var animals Animal
err := json.Unmarshal(jsonBlob, &animals)
// ...
But this slightly modified example doesn't:
Playground link of non-working example
// ...
var animals *Animal
err := json.Unmarshal(jsonBlob, animals)
// ...
It displays this obscure error that really isn't helpful (looks more like a function call than an error IMO):
json: Unmarshal(nil *main.Animal)
This appears to be because animals is an uninitialized pointer. But the docs say (emphasis mine):
Unmarshal unmarshals the JSON into the value pointed at by the pointer. If the pointer is nil, Unmarshal allocates a new value for it to point to.
So why does unmarshaling fail in the second example and show that obscure error?
(Also, is it "unmarshalling" or "unmarshaling" (one L)? The docs use both.)
You've encountered an InvalidUnmarshalError (see lines 109 and 110 in decode.go).
// An InvalidUnmarshalError describes an invalid argument passed to Unmarshal.
// (The argument to Unmarshal must be a non-nil pointer.)
It seems the docs could do with some clarification as the quote above and the comment below from the Unmarshal source seem to contradict each other.
If the pointer is nil, Unmarshal allocates a new value for it to point to.
Because your pointer is nil.
If you initialize it it works: http://play.golang.org/p/zprmV0O1fG
var animals *Animal = &Animal{}
Also, it can be spelled either way (consistency in a single doc would be nice, though): http://en.wikipedia.org/wiki/Marshalling_(computer_science)
I believe the issue is that, while you can pass a pointer to nil to Unmarshal(), you can't pass a nil pointer value.
A pointer to nil would be like:
var v interface{}
json.Unmarshal(text, &v)
The value of v is nil, but the pointer to v is a non-zero pointer address. It's a non-zero pointer, which is pointing to a nil interface{} (which itself is a pointer type). Unmarshal doesn't return an error in this case.
A nil pointer would be like:
var v *interface{}
json.Unmarshal(text, v)
In this case, the type of v is pointer to an interface{}, but as with any declaration of a var in golang, the initial value of v is the type's zero-value. So v is a zero-value pointer, which means it isn't pointing to any valid place in memory.
As mentioned in the https://stackoverflow.com/a/20478917/387176, json.Unmarshal() needs a valid pointer to something, so it can change the something (be it a zero value struct, or a pointer) in place.
I had a similiar condition before but in a different case.
It is related with concept of interface in Go.
If a function declares a interface as argument or return value, the caller have to pass or return the reference
In your case, json.Unmarshal accept interface as second argument