Custom JSON serialization and deserialization for interfaces in Go - json

I'm currently developing a JSON API for a blog in golang, and I've run into a roadblock trying to handle the serialization and deserialization of blog posts. I want my posts to contain an array of Post Sections that could be a number of things (such as normal paragraphs, images, quotes, etc.). I'm using Mongo for storage (with the amazing mgo library) and I want to save the posts like this:
{
"title": "Blog post",
"sections": [
{
"type": "text",
"content": { "en": "English content", "de": "Deutscher Inhalt" }
},
{
"type": "image",
"content": "https://dummyimage.com/100x100"
},
...more sections
],
...other fields
}
I've tried several solutions to implement this in go and none have really seemed like the "right way" to do it:
Not caring about the content
This seemed like the obvious solution, just using a simple struct:
type PostSection struct{
Type string
Content interface{}
}
This way, I can pass through whatever the frontend POSTS and save it. However, manipulating the data or validating it becomes impossible, so it's not a good solution.
Using custom interface serialization
I found this article about serializing interfaces in golang. This seemed great at first, because I could have an interface like this:
type PostSection interface{
Type() string
Content() interface{}
}
and then implement every type like this:
type PostImage string
func (p *PostImage) Type() string {
return "image"
}
func (p *PostImage) Content() interface{} {
return p
}
Optimally, that would've been it, and after implementing MarshalJSON and UnmarshalJSON for all my types, it was working fine when using json.Marshal directly on a PostSection object.
However, when serializing or deserializing an entire Post object containing an array of PostSections, my custom code was just being ignored and the PostSections would just be treated as the underlying objects (string or map[string]string in the examples) when serializing, or result in empty objects when deserializing.
Writing custom serialization for the entire Post struct
So, the solution I'm currently using but would like to change is custom serialization for the entire Post object. This leads to super ugly code, as I only really need custom code for a single field and so I'm passing through the rest, making the deserialization look similar to this:
p.ID = decoded.ID
p.Author = decoded.Author
p.Title = decoded.Title
p.Intro = decoded.Intro
p.Slug = decoded.Slug
p.TitleImage = decoded.TitleImage
p.Images = decoded.Images
...more fields...
and then, decoding the sections like this:
sections := make([]PostSection, len(decoded.Sections))
for i, s := range decoded.Sections {
if s["type"] == "text" {
content := s["content"].(map[string]interface{})
langs := make(PostText, len(content))
for lang, langContent := range content {
langString := langContent.(string)
langs[lang] = langString
}
sections[i] = &langs
} else if s["type"] == "image" {
content := s["content"].(string)
contentString := PostImage(content)
sections[i] = &contentString
}
}
p.Sections = sections
This is a whole lot of code I'll have to use every time I wanna include PostSections in another form somewhere else (for example in a Newsletter) and it doesn't feel like idiomatic go code by a long shot. Also, there is no error handling for malformed sections - They just cause a panic like this.
Is there a clean solution to this problem?

To avoid writing UnmarshalJSON for the whole Post you can wrap your PostSection in a concrete type and have it implement the Unmarshaler interface.
type Post struct {
ID int
Author string
Title string
Intro string
Slug string
TitleImage string
Images []string
Sections []*PostSection
}
type SectionContent interface {
Type() string
Content() interface{}
}
type PostSection struct {
Content SectionContent
}
func (s *PostSection) UnmarshalJSON(data []byte) error {
// ...
return nil
}

Related

Golang nested structs not getting omitted

I want to omit certain structs nested in a JSON request. I've created a rest API on golang which reads a message body from an http request, decodes it into the struct defined in the code and inserts it into Mongo DB
My structs are as follows. Note that for the nested structure C, I use a pointer in order to be able to omit it.
type A struct {
Title string `json:"title"`
Text string `json:"text"`
Data B `json:"data"`
}
type B struct {
Product *C `json:"product,omitempty"`
ExternalLink string `json:"external_link,omitempty"`
}
type C struct {
Name string `json:"name"`
Id int `json:"id"`
}
Here is how I decode it (Didn't go for Json.Unmarshall as I read that for http bodies, decoding should be used over unmarshall)
func NewMessage(req *http.Request) *A {
var newMessage *A
json.NewDecoder(req.Body).Decode(&newMessage)
messageInData := newMessage
return newMessage
}
The "newMessage" upon return is inserted into Mongo directly. However, Even if the request payload contains no such object as the struct C for instance like below
{
"title": "First message from GoLang",
"text": "Hello Dastgyr",
"data": {
"external_link": "some link here"
//no product object (C struct) here
}
}
The object inserted into Mongo still contains the struct C as having a null value as shown below
{
"title": "First message from GoLang",
"text": "Hello Dastgyr",
"data": {
"product": null,
"external_link": "some link here"
}
}
I've also tried using B as a pointer in Struct A but to no avail
type A struct {
Title string `json:"title"`
Text string `json:"text"`
Data *B `json:"data,omitempty"`
}
I want to be able to omit certain nested structs. Despite using pointers, the struct I want is still not omitting. What mistake am I making in defining structs?
Still new to golang so a nudge in the right direction would help
You are using json tags for json un-marshaling, it seems to be un-marshaling correctly (concluding this as you didn't mention any errors, and moved on to MongoDB)
How you add data to MongoDB is a totally different matter, and has nothing to do with your JSON tags. It uses bson tags, and you will need to add them, if you wish to use this same struct as a mongo DB model representation. Something like this:
type A struct {
Title string `json:"title" bson:"title"`
Text string `json:"text" bson:"text"`
Data *B `json:"data,omitempty" bson:"data,omitempty"`
}
Remember that tags in golang, are just some metadata being added with a structure, which some code actually reads and acts on. json library identifies & processes json:"" tag, while official go mongodb library that you might be using, will process the bson:"" tags.

How to unmarshal dynamic json objects with Golang

let me start by telling that I'm pretty recent in the Go world.
What I'm trying to do is to read the json I get from a JSON API (I don't control). Everything is working fine, I can show the received ID and Tags too. But the fields field is a little bit different, because its a dynamic array.
I can receive from the api this:
{
"id":"M7DHM98AD2-32E3223F",
"tags": [
{
"id":"9M23X2Z0",
"name":"History"
},
{
"id":"123123123",
"name":"Theory"
}
],
"fields": {
"title":"Title of the item",
"description":"Description of the item"
}
}
Or instead of title and description I could receive only description, or receive another random object like long_title. The objects return may differ completly and can be an infinite possibility of objects. But it always returns objects with a key and a string content like in the example.
This is my code so far:
type Item struct {
ID string `json:"id"`
Tags []Tag `json:"tags"`
//Fields []Field `json:"fields"`
}
// Tag data from the call
type Tag struct {
ID string `json:"id"`
Name string `json:"name"`
}
// AllEntries gets all entries from the session
func AllEntries() {
resp, _ := client.Get(APIURL)
body, _ := ioutil.ReadAll(resp.Body)
item := new(Item)
_ = json.Unmarshal(body, &item)
fmt.Println(i, "->", item.ID)
}
So the Item.Fields is dynamic, there is no way to predict what will be the key names, and therefore as far I can tell, there is no way to create a struct for it. But again, I'm pretty newbie with Go, could someone give me any tips? Thanks
If the data in "fields" is always going to be a flat-dict then you can use map[string]string as type for the Fields.
For arbitrary data specify Fields as a RawMessage type and parse it later on based on its content. Example from docs: https://play.golang.org/p/IR1_O87SHv
If the fields are way too unpredictable then you can keep this field as is([]byte) or if there are fields that are always going to common then you can parse those and leave the rest(but this would result in loss of data present in other fields).

Unmarshalling neo4j results to a nested struct in Golang

I'm using the neoism library (https://github.com/jmcvetta/neois) to talk to a local neo4j database - I'm fairly new to go and am new to neo4j so the gap in my understanding could be on either side of the problem.
I have a simple database, a single "page" node which is related to a single "template" node. I was hoping to be able to have struct representing each node and nest them inside one another but I'm strugging to get this to work.
Creating the simple DB:
template, err := ioutil.ReadFile(viewPath + "templates/default.mustache")
if err != nil{
panic(err)
}
defaultTemplate, _ := db.CreateNode(neoism.Props{
"name": "default",
"content": string(template),
})
defaultTemplate.AddLabel("Template")
n0, _ := db.CreateNode(neoism.Props{
"name": "Home",
"slug": "home",
"title": "Home Page",
"content" : "here I am",
})
n0.AddLabel("Page")
n0.Relate("TEMPLATE", template.Id(), neoism.Props{})
Now to the business of trying to get the data back out...
Here is my query which works just fine:
type PageStruct struct{
Name string `json:"p.name"`
Slug string `json:"p.slug"`
Title string `json:"p.title"`
Content string `json:"p.content"`
TemplateName string `json:"t.name"`
TemplateContent string `json:"t.content"`
}
res := []PageStruct{}
cq := neoism.CypherQuery{
Statement: `
MATCH (p:Page)-[r:TEMPLATE]->(t:Template)
WHERE p.slug = {slug}
RETURN p.name, p.slug, p.title, p.content, t.name, t.content
`,
Parameters: neoism.Props{"slug": pageSlug},
Result: &res,
}
db.Cypher(&cq)
page := res[0]
But ideally what I want is to unmarshall into a nested struct, something like this:
type PageStruct struct{
Name string `json:"p.name"`
Slug string `json:"p.slug"`
Title string `json:"p.title"`
Content string `json:"p.content"`
Template struct {
Name string `json:"t.name"`
Content string `json:"t.content"`
} `json:"t"`
}
I've been trying various things with no success, could anyone give me any advice on how to achieve this abitious feat of computer engineering...
Additionally I'm slightly unclear as to how to handle relationships in neo4j in the sense of having no enforcement (that I'm aware of) as to the type of relationship (e.g one-to-one, one-to-many) so how to handle this - do we always just assume an array of data?
Any help/advice is greatly appreciated.
Use embedding.
type Page struct {
Name string `json:"p.name"`
// ...
Template
}
type Template struct {
Name string `json:"t.name"`
// ...
}
Playground: http://play.golang.org/p/B3ro3wgsGS.

How to insert array of objects into MongoDB using Go

I have JSON from an API that I want to save to MongoDB using the mgo package. My JSON looks like this:
{
"something": "value"
"collection": [
{ "obj1": "value" }
// ... (Variable number of objects here)
]
}
To save this data I've created a new type in my Go application that looks like this:
type MyData struct {
Something string
Collection []string // This actually contains more than strings but I'll be happy if I can just get strings saved
}
mongoSess, err := mgo.Dial("localhost:27017")
if err != nil {
panic(err)
}
defer mongoSess.Close()
c := mongoSess.DB("mydatabase").C("mycollection")
insertErr := c.Insert(&MyData{something, collection})
This code works but the problem is that it isn't saving anything in my collection field which should be an array of JSON objects. Nothing at all. I get the keys in my database and they are the right type but they have no data. Here's what the Mongo output is:
{ "_id" : ObjectId("5520c535a236d8a9a215d096"), "something" : "value", "collection" : [ ] }
Can anyone spot what it is I'm doing wrong? I'm obviously new to Go and having trouble with types.
Solution
The answers here really helped me a lot. I'm at fault for not properly explaining things as the answers sent me on the right track but didn't solve the issue directly. Here's what the actual solution is.
package main
import (
"encoding/json"
"github.com/bitly/go-simplejson"
"gopkg.in/mgo.v2"
//"gopkg.in/mgo.v2/bson"
// Other packages are used as well
)
type MyData struct {
Something int
Collection []interface{}
}
func main() {
// I'm using SimpleJson for parsing JSON
collection, cerr := jsonFromExternalApi.Get("collection").Array()
if cerr != nil {
logger.Debug.Fatalln(cerr)
}
// Save response to Mongo (leaving out all the connection code)
c := mongoSess.DB("mydb").C("mycollection")
insertErr := c.Insert(&Producer{npn, licenses })
}
The issue was that SimpleJSON was returning the array of objects from my API call as a []interface{}. I was not aware I could simply declare part of a struct to be an interface so instead of just correcting what Go's compiler was telling me was wrong I was making it way harder than it should have been.
Coming from loosely typed scripting languages, stuff like this really trips me up and sometimes its hard to see the benefit but hopefully this helps someone out one day.
Looks you have the wrong data structure.
insertErr := c.Insert(&MyData{something, collection})
something => string and collection => slice
You code should be like this:
insertErr := c.Insert(&MyData{"something", []string{"obj1", "value"}})
Here is the working code.
[{Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]}]
For further reference, mgo has great documentation and you can find sample code and details about running mongodb queries here.
Here you are not defining proper type to Collection in your MyData struct.
Collection []string //This is what you are defining.
But from Json you are not getting string array,you are getting map[string]interface{}
This is the reason you are not filling Mydata struct properly
Correct MyData struct will be
type MyData struct {
Something string
Collection map[string]string // This actually contains more than strings but I'll be happy if I can just get strings saved
}

Parsing complex JSON into goweb REST Create() function

I apologise in advance for a very long question. Hopefully you'll bear with me.
I'm working with the goweb library, and experimenting with the example web app.
I've been trying to modify the RESTful example code, which defines a Thing as:
type Thing struct {
Id string
Text string
}
A Thing is created by sending an HTTP Post request, with an appropriate JSON body, to http://localhost:9090/things. This is handled in the example code in the Create function, specifically the lines:
dataMap := data.(map[string]interface{})
thing := new(Thing)
thing.Id = dataMap["Id"].(string)
thing.Text = dataMap["Text"].(string)
This is all well and good, and I can run the example server (which listens on http://localhost:9090/) and the server operates as expected.
For example:
curl -X POST -H "Content-Type: application/json" -d '{"Id":"TestId","Text":"TestText"}' http://localhost:9090/things
returns with no error, and then I GET that Thing with
curl http://localhost:9090/things/TestId
and it returns
{"d":{"Id":"TestId","Text":"TestText"},"s":200}
So far, so good.
Now, I would like to modify the Thing type, and add a custom ThingText type, like so:
type ThingText struct {
Title string
Body string
}
type Thing struct {
Id string
Text ThingText
}
This in itself isn't an issue, and I can modify the Create function like so:
thing := new(Thing)
thing.Id = dataMap["Id"].(string)
thing.Text.Title = dataMap["Title"].(string)
thing.Text.Body = dataMap["Body"].(string)
and the run the previous curl POST request with the JSON set to:
{"Id":"TestId","Title":"TestTitle","Title":"TestBody"}
and it returns with no error.
Once again I can GET the Thing URL and it returns:
{"d":{"Id":"TestId","Text":{"Title":"TestTitle","Body":"TestBody"}},"s":200}
Again, so far, so good.
Now, my question:
how do I modify the Create function to allow me to POST complex JSON to it?
For example, that last returned JSON string above includes {"Id":"TestId","Text":{"Title":"TestTitle","Body":"TestBody"}}. I'd like to be able to POST that exact JSON to the endpoint and have the Thing created.
I've followed the code back, and it seems that the data variable is of type Context.RequestData() from https://github.com/stretchr/goweb/context, and the internal Map seems to be of type Object.Map from https://github.com/stretchr/stew/, described as "a map[string]interface{} with additional helpful functionality." in particular, I noticed "Supports dot syntax to set deep values."
I can't work out how I can set up the thing.Text.Title = dataMap... statement so that the correct JSON field is parsed into it. I can't seem to use anything other than string types in the dataMap, and if I try that JSON it gives an error similar to:
http: panic serving 127.0.0.1:59113: interface conversion: interface is nil, not string
Once again, sorry for the ridiculously long question. I really appreciate you reading, and any help you may have to offer. Thanks!
As the JSON package documentation and JSON and Go introduction describe, JSON data can be parsed either generically through interface{} / string maps or unmarshalling directly to the struct types.
The example code you linked to and you based your changes on seems to use the generic string-map approach, dataMap := data.(map[string]interface{}).
As your desired JSON data is an object in an object, it’s simply a map within a map.
So you should be able to
dataMap := data.(map[string]interface{})
subthingMap := dataMap["Text"].(map[string]interface{})
thing.Text.Title = subthingMap["Title"].(string)
thing.Text.Body = subthingMap["Body"].(string)
I’m not sure why that code uses casts and generic types over type-safe unmarshalling directly from JSON to struct types (abstraction I guess). Using the json packages unmarshalling to struct types would go something like
type ThingText struct {
Title string
Body string
}
type Thing struct {
Id string
Text ThingText
}
…
decoder := json.NewDecoder(body)
var thingobj Thing
for {
if err := decoder.Decode(&thingobj); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
fmt.Println(thingobj)
}
where body is a io.Reader - in simple/most cases from http.Response.Body.