Parsing complex JSON into goweb REST Create() function - json

I apologise in advance for a very long question. Hopefully you'll bear with me.
I'm working with the goweb library, and experimenting with the example web app.
I've been trying to modify the RESTful example code, which defines a Thing as:
type Thing struct {
Id string
Text string
}
A Thing is created by sending an HTTP Post request, with an appropriate JSON body, to http://localhost:9090/things. This is handled in the example code in the Create function, specifically the lines:
dataMap := data.(map[string]interface{})
thing := new(Thing)
thing.Id = dataMap["Id"].(string)
thing.Text = dataMap["Text"].(string)
This is all well and good, and I can run the example server (which listens on http://localhost:9090/) and the server operates as expected.
For example:
curl -X POST -H "Content-Type: application/json" -d '{"Id":"TestId","Text":"TestText"}' http://localhost:9090/things
returns with no error, and then I GET that Thing with
curl http://localhost:9090/things/TestId
and it returns
{"d":{"Id":"TestId","Text":"TestText"},"s":200}
So far, so good.
Now, I would like to modify the Thing type, and add a custom ThingText type, like so:
type ThingText struct {
Title string
Body string
}
type Thing struct {
Id string
Text ThingText
}
This in itself isn't an issue, and I can modify the Create function like so:
thing := new(Thing)
thing.Id = dataMap["Id"].(string)
thing.Text.Title = dataMap["Title"].(string)
thing.Text.Body = dataMap["Body"].(string)
and the run the previous curl POST request with the JSON set to:
{"Id":"TestId","Title":"TestTitle","Title":"TestBody"}
and it returns with no error.
Once again I can GET the Thing URL and it returns:
{"d":{"Id":"TestId","Text":{"Title":"TestTitle","Body":"TestBody"}},"s":200}
Again, so far, so good.
Now, my question:
how do I modify the Create function to allow me to POST complex JSON to it?
For example, that last returned JSON string above includes {"Id":"TestId","Text":{"Title":"TestTitle","Body":"TestBody"}}. I'd like to be able to POST that exact JSON to the endpoint and have the Thing created.
I've followed the code back, and it seems that the data variable is of type Context.RequestData() from https://github.com/stretchr/goweb/context, and the internal Map seems to be of type Object.Map from https://github.com/stretchr/stew/, described as "a map[string]interface{} with additional helpful functionality." in particular, I noticed "Supports dot syntax to set deep values."
I can't work out how I can set up the thing.Text.Title = dataMap... statement so that the correct JSON field is parsed into it. I can't seem to use anything other than string types in the dataMap, and if I try that JSON it gives an error similar to:
http: panic serving 127.0.0.1:59113: interface conversion: interface is nil, not string
Once again, sorry for the ridiculously long question. I really appreciate you reading, and any help you may have to offer. Thanks!

As the JSON package documentation and JSON and Go introduction describe, JSON data can be parsed either generically through interface{} / string maps or unmarshalling directly to the struct types.
The example code you linked to and you based your changes on seems to use the generic string-map approach, dataMap := data.(map[string]interface{}).
As your desired JSON data is an object in an object, it’s simply a map within a map.
So you should be able to
dataMap := data.(map[string]interface{})
subthingMap := dataMap["Text"].(map[string]interface{})
thing.Text.Title = subthingMap["Title"].(string)
thing.Text.Body = subthingMap["Body"].(string)
I’m not sure why that code uses casts and generic types over type-safe unmarshalling directly from JSON to struct types (abstraction I guess). Using the json packages unmarshalling to struct types would go something like
type ThingText struct {
Title string
Body string
}
type Thing struct {
Id string
Text ThingText
}
…
decoder := json.NewDecoder(body)
var thingobj Thing
for {
if err := decoder.Decode(&thingobj); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
fmt.Println(thingobj)
}
where body is a io.Reader - in simple/most cases from http.Response.Body.

Related

Passing a struct Type in Golang?

Please forgive my question, I'm new to Golang and possibly have the wrong approach.
I'm currently implementing a Terraform provider for an internal service.
As probably expected, that requires unmarshalling JSON data in to pre-defined Struct Types, e.g:
type SomeTypeIveDefined struct {
ID string `json:"id"`
Name String `json:"name"`
}
I've got myself in to a situation where I have a lot of duplicate code that looks like this
res := r.(*http.Response)
var tempThing SomeTypeIveDefined
dec := json.NewDecoder(res.Body)
err := dec.Decode(&tempThing)
In an effort to reduce duplication, I decided what I wanted to do was create a function which does the JSON unmarshalling, but takes in the Struct Type as a parameter.
I've trawled through several StackOverflow articles and Google Groups trying to make sense of some of the answers around using the reflect package, but I've not had much success in using it.
My latest attempt was using reflect.StructOf and passing in a set of StructFields, but that still seems to require using myReflectedStruct.Field(0) rather than myReflectedStruct.ID.
I suspect there may be no way until something like Generics are widely available in Golang.
I considered perhaps an interface for the structs which requires implementing an unmarshal method, then I could pass the interface to the function and call the unmarshal method. But then I'm still implementing unmarshal on all the structs anyway.
I'm just wondering what suggestions there may be for achieving what I'm after, please?
Create a helper function with the repeated code. Pass the destination value as a pointer.
func decode(r *http.Repsonse, v interface{}) error {
return json.NewDecoder(res.Body).Decode(v)
}
Call the helper function with a pointer to your thing:
var tempThing SomeTypeIveDefined
err := deocde(r, &tempThing)
You can do this with interfaces:
func decodeResponse(r *http.Response, dest interface{}) error {
dec := json.NewDecoder(r.Body)
return dec.Decode(dest)
}
func handler(...) {
res := r.(*http.Response)
var tempThing SomeTypeIveDefined
if err:=decodeResponse(res,&tempThing); err!=nil {
// handle err
}
...
}
You don't need to implement an unmarshal for the structs, because the stdlib decoder will use reflection to set the struct fields.

How to validate JSON input using Golang?

I'm using Beego framework to build a web application in Go. I have to validate the incoming JSON in an API request.
I can unmarshal the JSON into a struct which works fine, but I want to validate the data as well. For example, if the type doesn't match with the type in struct json.Unmarshal will reutrn an error on the first occurence. I want to validate and get all the errors at once for JSON.
I've tried https://github.com/thedevsaddam/govalidator but the package needs a reference to request object which is not available in the controller of Beego. There are other validators which can validate a struct but i want the json validation as well.
EDIT:
A reference to the request object in beego can be found from the controller's context object as:
func (this *MainController) Post() {
fmt.Println(this.Ctx.Request)
}
But the problem remains same with json unmarshal. If there's any slight mismatch in the type, json.unmarshal will immediately panic. I want to be able to validate the type as well.
Since Golang v1.9 json.Valid([]byte) has been a valid method available from the "encoding/json" package.
Reference:
https://golang.org/pkg/encoding/json/#Valid
We've used go-playground/validator.v8 for a similar purpose. You can define the data validations with the tags that come out of the box (basic stuff like equality, min/max and even has somthing of an expression lang). On top of that you can add your custom validations. It's all in the docs, hope it helps.
There is a method checkValid (in encoding/json package) which can do that. This method, however, is not exposed. So, you can use Unmarshal (in same package) to check if JSON is valid.
I had to do this today so I will share my solution. I created a function using unmarshal that returns true or false based on whether the text returns valid json:
// isJSON checks that the string value of the field can unmarshall into valid json
func isJSON(s string) bool {
var j map[string]interface{}
if err := json.Unmarshal([]byte(s), &j); err != nil {
return false
}
return true
}
Then I used that logic with the validator package to make and register a custom validation tag that I can use in any struct to validate that a field contains json. You can use go playground to check the full solution and experiment:
// isJSON checks that the string value of the field can unmarshall into valid json
func isJSON(fl validator.FieldLevel) bool {
var j map[string]interface{}
if err := json.Unmarshal([]byte(fl.Field().String()), &j); err != nil {
return false
}
return true
}
// register the function with json tag:
validate.RegisterValidation("json", isJSON)

How to deal with Struct having different json key than json response

I have a struct VideoInfo that has a key in it called embedCode. The API I am querying returns the embed code as embed_code. During unmarshalling the response how do I ensure embed_code goes into embedCode?
Also is there an easy way to take a large json string and automatically turn it into a struct, or can one only use a map?
With respect to remapping the field names use the corresponding annotation in the structure declaration:
type VideoInfo struct {
EmbedCode string `json:"embed_code"`
}
The marshaller/un-marshaller will only process public field, so you need to capitalise the field name.
With respect to converting the whole structure, yes it is easy. Declare an instance to un-marshal into and pass a reference to the json.Unmarshal method (from a test):
data, _ := json.Marshal(request)
var resp response.VideoInfo
if err := json.Unmarshal(data, &resp); err != nil {
t.Errorf("unexpected error, %v", err)
}
At first, struct's field must be start from capital letter to be public. So you need something like that:
type VideoInfo struct {
EmbedCode string `json:"embed_code"`
}
And look at documentation for more info.

How to insert array of objects into MongoDB using Go

I have JSON from an API that I want to save to MongoDB using the mgo package. My JSON looks like this:
{
"something": "value"
"collection": [
{ "obj1": "value" }
// ... (Variable number of objects here)
]
}
To save this data I've created a new type in my Go application that looks like this:
type MyData struct {
Something string
Collection []string // This actually contains more than strings but I'll be happy if I can just get strings saved
}
mongoSess, err := mgo.Dial("localhost:27017")
if err != nil {
panic(err)
}
defer mongoSess.Close()
c := mongoSess.DB("mydatabase").C("mycollection")
insertErr := c.Insert(&MyData{something, collection})
This code works but the problem is that it isn't saving anything in my collection field which should be an array of JSON objects. Nothing at all. I get the keys in my database and they are the right type but they have no data. Here's what the Mongo output is:
{ "_id" : ObjectId("5520c535a236d8a9a215d096"), "something" : "value", "collection" : [ ] }
Can anyone spot what it is I'm doing wrong? I'm obviously new to Go and having trouble with types.
Solution
The answers here really helped me a lot. I'm at fault for not properly explaining things as the answers sent me on the right track but didn't solve the issue directly. Here's what the actual solution is.
package main
import (
"encoding/json"
"github.com/bitly/go-simplejson"
"gopkg.in/mgo.v2"
//"gopkg.in/mgo.v2/bson"
// Other packages are used as well
)
type MyData struct {
Something int
Collection []interface{}
}
func main() {
// I'm using SimpleJson for parsing JSON
collection, cerr := jsonFromExternalApi.Get("collection").Array()
if cerr != nil {
logger.Debug.Fatalln(cerr)
}
// Save response to Mongo (leaving out all the connection code)
c := mongoSess.DB("mydb").C("mycollection")
insertErr := c.Insert(&Producer{npn, licenses })
}
The issue was that SimpleJSON was returning the array of objects from my API call as a []interface{}. I was not aware I could simply declare part of a struct to be an interface so instead of just correcting what Go's compiler was telling me was wrong I was making it way harder than it should have been.
Coming from loosely typed scripting languages, stuff like this really trips me up and sometimes its hard to see the benefit but hopefully this helps someone out one day.
Looks you have the wrong data structure.
insertErr := c.Insert(&MyData{something, collection})
something => string and collection => slice
You code should be like this:
insertErr := c.Insert(&MyData{"something", []string{"obj1", "value"}})
Here is the working code.
[{Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]}]
For further reference, mgo has great documentation and you can find sample code and details about running mongodb queries here.
Here you are not defining proper type to Collection in your MyData struct.
Collection []string //This is what you are defining.
But from Json you are not getting string array,you are getting map[string]interface{}
This is the reason you are not filling Mydata struct properly
Correct MyData struct will be
type MyData struct {
Something string
Collection map[string]string // This actually contains more than strings but I'll be happy if I can just get strings saved
}

Golang parse JSON returns 0

The JSON that I am at tempting to parse is very basic and looks like this.
{"id": 3, "title":"Test"}
The following is the code that I am attempting to use for creating and parsing the JSON.
package main
import (
"fmt"
"encoding/json"
)
type Config struct{
id int
title string
}
func main() {
var jsonStr = []byte(`{"id": 3, "title":"Test"}`)
var conf Config
err := json.Unmarshal(jsonStr, &conf)
if err!=nil{
fmt.Print("Error:",err)
}
fmt.Println(conf)
fmt.Println(string(jsonStr))
}
Been looking over a lot of different code examples and can't see what I'm doing wrong. When I attempt to run this, this is what I get as a return.
{0 }
{"id": 3, "title":"Test"}
I have verified that the JSON is valid, but continue to get an empty return when attempting to use json.Unmarshal. Any ideas on what I am missing so that I can get this JSON parsed?
EDIT: Looks like I can get this to work if I capitalize the titles (Id, Title). Unfortunately the return I am testing for is a return from an API which returns everything in lowercase. I need to be able to parse this JSON with lowercase titles as listed above.
Your Config struct's fields need to be exported (upper-case), but the keys in your JSON object may remain lower-case.
See here: http://play.golang.org/p/0A5tkCkSO5
Please consult the JSON package documentation, it is worth the read. While Amit already addressed the export issue I will address the following:
EDIT: Looks like I can get this to work if I capitalize the titles (Id, Title). Unfortunately the return I am testing for is a return from an API which returns everything in lowercase. I need to be able to parse this JSON with lowercase titles as listed above.
As you might imagine, the authors of encoding/json have thought of that, so again I encourage you to consult the documentation next time. The solution is this (Example on playground):
type Config struct {
Id int `json:"id"`
Title string `json:"title"`
}