Unmarshalling neo4j results to a nested struct in Golang - json

I'm using the neoism library (https://github.com/jmcvetta/neois) to talk to a local neo4j database - I'm fairly new to go and am new to neo4j so the gap in my understanding could be on either side of the problem.
I have a simple database, a single "page" node which is related to a single "template" node. I was hoping to be able to have struct representing each node and nest them inside one another but I'm strugging to get this to work.
Creating the simple DB:
template, err := ioutil.ReadFile(viewPath + "templates/default.mustache")
if err != nil{
panic(err)
}
defaultTemplate, _ := db.CreateNode(neoism.Props{
"name": "default",
"content": string(template),
})
defaultTemplate.AddLabel("Template")
n0, _ := db.CreateNode(neoism.Props{
"name": "Home",
"slug": "home",
"title": "Home Page",
"content" : "here I am",
})
n0.AddLabel("Page")
n0.Relate("TEMPLATE", template.Id(), neoism.Props{})
Now to the business of trying to get the data back out...
Here is my query which works just fine:
type PageStruct struct{
Name string `json:"p.name"`
Slug string `json:"p.slug"`
Title string `json:"p.title"`
Content string `json:"p.content"`
TemplateName string `json:"t.name"`
TemplateContent string `json:"t.content"`
}
res := []PageStruct{}
cq := neoism.CypherQuery{
Statement: `
MATCH (p:Page)-[r:TEMPLATE]->(t:Template)
WHERE p.slug = {slug}
RETURN p.name, p.slug, p.title, p.content, t.name, t.content
`,
Parameters: neoism.Props{"slug": pageSlug},
Result: &res,
}
db.Cypher(&cq)
page := res[0]
But ideally what I want is to unmarshall into a nested struct, something like this:
type PageStruct struct{
Name string `json:"p.name"`
Slug string `json:"p.slug"`
Title string `json:"p.title"`
Content string `json:"p.content"`
Template struct {
Name string `json:"t.name"`
Content string `json:"t.content"`
} `json:"t"`
}
I've been trying various things with no success, could anyone give me any advice on how to achieve this abitious feat of computer engineering...
Additionally I'm slightly unclear as to how to handle relationships in neo4j in the sense of having no enforcement (that I'm aware of) as to the type of relationship (e.g one-to-one, one-to-many) so how to handle this - do we always just assume an array of data?
Any help/advice is greatly appreciated.

Use embedding.
type Page struct {
Name string `json:"p.name"`
// ...
Template
}
type Template struct {
Name string `json:"t.name"`
// ...
}
Playground: http://play.golang.org/p/B3ro3wgsGS.

Related

Custom JSON serialization and deserialization for interfaces in Go

I'm currently developing a JSON API for a blog in golang, and I've run into a roadblock trying to handle the serialization and deserialization of blog posts. I want my posts to contain an array of Post Sections that could be a number of things (such as normal paragraphs, images, quotes, etc.). I'm using Mongo for storage (with the amazing mgo library) and I want to save the posts like this:
{
"title": "Blog post",
"sections": [
{
"type": "text",
"content": { "en": "English content", "de": "Deutscher Inhalt" }
},
{
"type": "image",
"content": "https://dummyimage.com/100x100"
},
...more sections
],
...other fields
}
I've tried several solutions to implement this in go and none have really seemed like the "right way" to do it:
Not caring about the content
This seemed like the obvious solution, just using a simple struct:
type PostSection struct{
Type string
Content interface{}
}
This way, I can pass through whatever the frontend POSTS and save it. However, manipulating the data or validating it becomes impossible, so it's not a good solution.
Using custom interface serialization
I found this article about serializing interfaces in golang. This seemed great at first, because I could have an interface like this:
type PostSection interface{
Type() string
Content() interface{}
}
and then implement every type like this:
type PostImage string
func (p *PostImage) Type() string {
return "image"
}
func (p *PostImage) Content() interface{} {
return p
}
Optimally, that would've been it, and after implementing MarshalJSON and UnmarshalJSON for all my types, it was working fine when using json.Marshal directly on a PostSection object.
However, when serializing or deserializing an entire Post object containing an array of PostSections, my custom code was just being ignored and the PostSections would just be treated as the underlying objects (string or map[string]string in the examples) when serializing, or result in empty objects when deserializing.
Writing custom serialization for the entire Post struct
So, the solution I'm currently using but would like to change is custom serialization for the entire Post object. This leads to super ugly code, as I only really need custom code for a single field and so I'm passing through the rest, making the deserialization look similar to this:
p.ID = decoded.ID
p.Author = decoded.Author
p.Title = decoded.Title
p.Intro = decoded.Intro
p.Slug = decoded.Slug
p.TitleImage = decoded.TitleImage
p.Images = decoded.Images
...more fields...
and then, decoding the sections like this:
sections := make([]PostSection, len(decoded.Sections))
for i, s := range decoded.Sections {
if s["type"] == "text" {
content := s["content"].(map[string]interface{})
langs := make(PostText, len(content))
for lang, langContent := range content {
langString := langContent.(string)
langs[lang] = langString
}
sections[i] = &langs
} else if s["type"] == "image" {
content := s["content"].(string)
contentString := PostImage(content)
sections[i] = &contentString
}
}
p.Sections = sections
This is a whole lot of code I'll have to use every time I wanna include PostSections in another form somewhere else (for example in a Newsletter) and it doesn't feel like idiomatic go code by a long shot. Also, there is no error handling for malformed sections - They just cause a panic like this.
Is there a clean solution to this problem?
To avoid writing UnmarshalJSON for the whole Post you can wrap your PostSection in a concrete type and have it implement the Unmarshaler interface.
type Post struct {
ID int
Author string
Title string
Intro string
Slug string
TitleImage string
Images []string
Sections []*PostSection
}
type SectionContent interface {
Type() string
Content() interface{}
}
type PostSection struct {
Content SectionContent
}
func (s *PostSection) UnmarshalJSON(data []byte) error {
// ...
return nil
}

How to unmarshal dynamic json objects with Golang

let me start by telling that I'm pretty recent in the Go world.
What I'm trying to do is to read the json I get from a JSON API (I don't control). Everything is working fine, I can show the received ID and Tags too. But the fields field is a little bit different, because its a dynamic array.
I can receive from the api this:
{
"id":"M7DHM98AD2-32E3223F",
"tags": [
{
"id":"9M23X2Z0",
"name":"History"
},
{
"id":"123123123",
"name":"Theory"
}
],
"fields": {
"title":"Title of the item",
"description":"Description of the item"
}
}
Or instead of title and description I could receive only description, or receive another random object like long_title. The objects return may differ completly and can be an infinite possibility of objects. But it always returns objects with a key and a string content like in the example.
This is my code so far:
type Item struct {
ID string `json:"id"`
Tags []Tag `json:"tags"`
//Fields []Field `json:"fields"`
}
// Tag data from the call
type Tag struct {
ID string `json:"id"`
Name string `json:"name"`
}
// AllEntries gets all entries from the session
func AllEntries() {
resp, _ := client.Get(APIURL)
body, _ := ioutil.ReadAll(resp.Body)
item := new(Item)
_ = json.Unmarshal(body, &item)
fmt.Println(i, "->", item.ID)
}
So the Item.Fields is dynamic, there is no way to predict what will be the key names, and therefore as far I can tell, there is no way to create a struct for it. But again, I'm pretty newbie with Go, could someone give me any tips? Thanks
If the data in "fields" is always going to be a flat-dict then you can use map[string]string as type for the Fields.
For arbitrary data specify Fields as a RawMessage type and parse it later on based on its content. Example from docs: https://play.golang.org/p/IR1_O87SHv
If the fields are way too unpredictable then you can keep this field as is([]byte) or if there are fields that are always going to common then you can parse those and leave the rest(but this would result in loss of data present in other fields).

XML into JSON Multiple Nesting

I'm attempting to write code to translate XML to JSON. The XML I'm trying to translate is as follows...
(Just a snippet)
`<version>0.1</version>
<termsofService>http://www.wunderground.com/weather/api/d/terms.html</termsofService>
<features>
<feature>conditions</feature>
</features>
<current_observation>
<image>
<url>http://icons.wxug.com/graphics/wu2/logo_130x80.png</url>
<title>Weather Underground</title>
<link>http://www.wunderground.com</link>
</image>
<display_location>
<full>Kearney, MO</full>
<city>Kearney</city>
<state>MO</state>
<state_name>Missouri</state_name>`
Current Code:
`package main
import (
"fmt"
"net/url"
"encoding/xml"
"net/http"
"log"
"io/ioutil"
"encoding/json"
)
type reportType struct{
Version xml.CharData `xml:"version"`
TermsOfService xml.CharData `xml:"termsofService"
`
Features xml.CharData `xml:"features>feature"`
Full xml.CharData `xml:"current_observation>display_location>full"`
StateName xml.CharData `xml:"current_observation>display_location>state_name"`
WindGust xml.CharData `xml:"current_observation>observation_location>full"`
Problem myErrorType `xml:"error"`
}
type myErrorType struct{
TypeOfError xml.CharData `xml:"type"`
Desciption xml.CharData `xml:"description"`
}
type reportTypeJson struct{
Version string `json:"version"`;
TermsOfService string `json:"termsofService"`;
Features map[string]string `json:"features"`;
Full map[string]string `json:"display_location"`;
WindGust map[string]string `json:"observation_location"`
}
func main() {
fmt.Println("data is from WeatherUnderground.")
fmt.Println("https://www.wunderground.com/")
var state, city string
str1 := "What is your state?"
str2 := "What is your city?"
fmt.Println(str1)
fmt.Scanf("%s", &state)
fmt.Println(str2)
fmt.Scanf("%s", &city)
baseURL := "http://api.wunderground.com/api/";
apiKey := "3hunna"
var query string
//set up the query
query = baseURL+apiKey +
"/conditions/q/"+
url.QueryEscape(state)+ "/"+
url.QueryEscape(city)+ ".xml"
fmt.Println("The escaped query: "+query)
response, err := http.Get(query)
doErr(err, "After the GET")
var body []byte
body, err = ioutil.ReadAll(response.Body)
doErr(err, "After Readall")
fmt.Println(body);
fmt.Printf("The body: %s\n",body)
//Unmarshalling
var report reportType
xml.Unmarshal(body, &report)
fmt.Printf("The Report: %s\n", report)
fmt.Printf("The description is [%s]\n",report.Problem.Desciption)
//Now marshal the data out in JSON
var data []byte
var output reportTypeJson
output.Version = string(report.Version);
output.TermsOfService = string(report.TermsOfService)
output.Features= map[string]string{"feature":string(report.Features)} // allocate a map, add the 'features' value to it and assign it to output.Features
output.Full=map[string]string{"full":string(report.Full),"state_name":string(report.StateName)}
output.WindGust=map[string]string{"full":string(report.WindGust)}
data,err = json.MarshalIndent(output,""," ")
doErr(err, "From marshalIndent")
fmt.Printf("JSON output nicely formatted: \n%s\n",data)
}
func doErr( err error, message string){
if err != nil{
log.Panicf("ERROR: %s %s \n", message, err.Error())
}
}
As you can see, I'm using maps to map one level nesting such as in the features case. But for in two level nesting cases such as xml:"current_observation>display_location>state_name", I can't figure out how to create the very first level, in this case current_observations. Would there be a way to somehow create a map of maps of sorts? Any and all ideas are much appreciated because I am very confused at the moment, Thanks for your time!
And the Output:
JSON output nicely formatted:
{
"version": "0.1",
"termsofService": "http://www.wunderground.com/weather/api/d/terms.html",
"features": {
"feature": "conditions"
},
"display_location": {
"full": "Kearney, MO",
"state_name": "Missouri"
},
"observation_location": {
"full": "Stonecrest, Kearney, Missouri"
}
}
You could use either structs or a map of maps. I'll give some examples of both, starting with the map of maps. The type would be declares as;
CurrentObservation map[string]map[string]string `json:"current_observation"`
In this case you have a map with strings as the keys and the value is another map that has string for both key and value. As a result when you marshal your json you will end up with something like;
"current_observation" {
"image": { // first level where the value is a map[string]string
"title":"Weather Underground" // inner map, string to string
}
}
If say you wanted to just print the title, you would do;
fmt.Println(reportJson.CurrentObservation["image"]["title"])
Since the data there looks fairly static you could also use structs instead. In which case you'd use something like this;
CurrentObservation CurrentObservation `json:"current_observation"`
type CurrentObservation struct {
Image Image `json:"image"`
}
type Image struct {
Url string `json:"url"`
Title string `json:"title"`
Link string `json:"link"`
}
Both options produce the same output though they could behave differently for different inputs. For example, if another version of current_observation were received as input that for example has another nested item in it call it... previous_observation then the map option would automatically unmarhsal this data as well where the struct options would exclude it since there would be no mapping to any object/type in Go.
Personally I prefer the struct route when possible but it varies from case to case. For your application the map is likely better since you're not working with the input (it comes in as xml) and you just want to print it, you don't really have to deal with the details of current_observation, if it has 3 objects inside it, they'll all be output as expected, if it were 5 it would be the same. With the structs you have to explicitly define every single field which isn't really necessary if you're just transforming the input. The advantage of the struct is more for use later on where you have type safety although in this case, I would say they're still fairly equivalent because for example anytime you want to access something in image, like CurrentObservation.Image.Title you'll have to perform a check to ensure Image is not nil, like;
if CurrentObservation.Image != nil {
fmt.Println(CurrentObservation.Image.Title)
}
With the map you basically have the same overhead, only you're checking for the existence of a key rather than checking if one of the inner structs is nil or not.
EDIT: example of initializing the map of maps using composite literal syntax;
reportJson.CurrentObservation := map[string]map[string]string {
"display_location": map[string]string {
"full": report.Full,
"state_name": report.StateName,
},
}

How to insert array of objects into MongoDB using Go

I have JSON from an API that I want to save to MongoDB using the mgo package. My JSON looks like this:
{
"something": "value"
"collection": [
{ "obj1": "value" }
// ... (Variable number of objects here)
]
}
To save this data I've created a new type in my Go application that looks like this:
type MyData struct {
Something string
Collection []string // This actually contains more than strings but I'll be happy if I can just get strings saved
}
mongoSess, err := mgo.Dial("localhost:27017")
if err != nil {
panic(err)
}
defer mongoSess.Close()
c := mongoSess.DB("mydatabase").C("mycollection")
insertErr := c.Insert(&MyData{something, collection})
This code works but the problem is that it isn't saving anything in my collection field which should be an array of JSON objects. Nothing at all. I get the keys in my database and they are the right type but they have no data. Here's what the Mongo output is:
{ "_id" : ObjectId("5520c535a236d8a9a215d096"), "something" : "value", "collection" : [ ] }
Can anyone spot what it is I'm doing wrong? I'm obviously new to Go and having trouble with types.
Solution
The answers here really helped me a lot. I'm at fault for not properly explaining things as the answers sent me on the right track but didn't solve the issue directly. Here's what the actual solution is.
package main
import (
"encoding/json"
"github.com/bitly/go-simplejson"
"gopkg.in/mgo.v2"
//"gopkg.in/mgo.v2/bson"
// Other packages are used as well
)
type MyData struct {
Something int
Collection []interface{}
}
func main() {
// I'm using SimpleJson for parsing JSON
collection, cerr := jsonFromExternalApi.Get("collection").Array()
if cerr != nil {
logger.Debug.Fatalln(cerr)
}
// Save response to Mongo (leaving out all the connection code)
c := mongoSess.DB("mydb").C("mycollection")
insertErr := c.Insert(&Producer{npn, licenses })
}
The issue was that SimpleJSON was returning the array of objects from my API call as a []interface{}. I was not aware I could simply declare part of a struct to be an interface so instead of just correcting what Go's compiler was telling me was wrong I was making it way harder than it should have been.
Coming from loosely typed scripting languages, stuff like this really trips me up and sometimes its hard to see the benefit but hopefully this helps someone out one day.
Looks you have the wrong data structure.
insertErr := c.Insert(&MyData{something, collection})
something => string and collection => slice
You code should be like this:
insertErr := c.Insert(&MyData{"something", []string{"obj1", "value"}})
Here is the working code.
[{Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]} {Something:something Collection:[obj1 value]}]
For further reference, mgo has great documentation and you can find sample code and details about running mongodb queries here.
Here you are not defining proper type to Collection in your MyData struct.
Collection []string //This is what you are defining.
But from Json you are not getting string array,you are getting map[string]interface{}
This is the reason you are not filling Mydata struct properly
Correct MyData struct will be
type MyData struct {
Something string
Collection map[string]string // This actually contains more than strings but I'll be happy if I can just get strings saved
}

When serializing to JSON, how to omit certain fields based on a run-time condition?

In a web service implemented in Go, I want to be able to restrict fields returned in a JSON response based on a user's role.
For example I may have a currently logged in user who has a role of guest and another with the role of admin
For an admin I want json to have all the keys eg
{
id: 1,
name: "John",
role: "admin"
}
and for a guest to not have the role key eg
{
id: 1,
name: "John"
}
I can currently marshal the json and it returns all fields. I need to be able to restrict it.
You can go by the suggestion #Volker made and clear struct fields for which the user has no permissions. This is probably the easiest to implement.
A second option in a similar vein is to create a custom JSON encoder. One which encodes fields only if a role struct tag matches the current user's role. Here is some pseudo code to illustrate:
type T struct {
currentRole Role `json:"-"`
FieldA string `json:"field_a,omitempty", role:"guest"`
FieldB string `json:"field_b,omitempty", role:"guest"`
FieldC int `json:"field_c,omitempty", role:"admin"`
}
// Have T implement the encoding/json.Marshaler interface.
func (t *T) MarshalJSON() ([]byte, error) {
var buf bytes.Buffer
// Use some reflection magic to iterate over struct fields.
for _, field := range getStructFields(t) {
// More reflection magic to extract field tag data.
role := getFieldTag(field, "role")
// If the field tag's role matches our current role,
// we are good to go. otherwise, skip this field.
if !matchingRole(role, t.currentRole) {
continue // skip this field
}
data, err := json.Marshal(fieldValue(field))
...
_, err = buf.Write(data)
...
}
return buf.Bytes(), nil
}
This is going to be a pain to maintain if you need new roles though. So this would not be something I would lightly consider doing.
Security concerns
I am not entirely sure that what you are looking for is the right solution to your problem. This depends on the context in which you use your code, which is not clear from your question. But if this concerns a website where a user's abilities on the website are defined solely by the value of the role JSON field, then you are looking at a security hole. They can simply go into a browser debugger and change the value of this JSON object to include the "role: "admin" field. And presto! Instant administrative powers. Whether or not to render certain parts of a page, based on user role, should really be handled by the server, during template processing. Just like any and all data posted to the server should be checked and checked again to ensure it came from a trusted source.
If none of this is applicable to you, then by all means, disregard this paragraph.
This question seems old, but I recently wanted to do the same thing. Maybe this will help someone in the future. Here is another method: you can define your own Marshal interface and use anonymous structs.
//User holds all variables
//even private ones
type User struct {
ID int64
Name string
Role string
}
//MarshalJSON gives back json user
//but only the public fields!
func (u *User) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
ID string `json:"id"`
Name string `json:"name"`
}{u.ID, u.Name})
}
it would be pretty easy to put an if u.Role == "admin" statement in the block to decide whether to marshal the rest.
Another option that also works to define the set of fields in the output for a list of struct that comes from an appengine datastore query.
// Setting different JSON output field for the same struct, using anonymous
// fields (inspired by inspired by http://choly.ca/post/go-json-marshalling/)
// This alternative could be used to load a resultset from an appengine datastore
// query and returned a custom field combination for the list items.
package main
import (
"encoding/json"
"fmt"
)
type User struct {
ID string `json:"id"`
Name string `json:"name"`
Role string `json:"-"`
LaunchCode string `json:"-"`
}
type AdminOutputUser User
func (user *AdminOutputUser) MarshalJSON() ([]byte, error) {
type Alias AdminOutputUser
return json.Marshal(&struct {
*Alias
Role string `json:"role"`
}{
(*Alias)(user),
user.Role,
})
}
type SuperadminOutputUser User
func (user *SuperadminOutputUser) MarshalJSON() ([]byte, error) {
type Alias SuperadminOutputUser
return json.Marshal(&struct {
*Alias
Role string `json:"role"`
LaunchCode string `json:"code"`
}{
(*Alias)(user),
user.Role,
user.LaunchCode,
})
}
func main() {
user := User{"007", "James Bond", "admin", "12345678"}
adminOutput := AdminOutputUser(user)
superadminOutput := SuperadminOutputUser(user)
b, _ := json.Marshal(&user)
fmt.Printf("%s\n\n", string(b))
// {"id":"007","name":"James Bond"}
b, _ = json.Marshal(&adminOutput)
fmt.Printf("%s\n\n", string(b))
// {"id":"007","name":"James Bond","role":"admin"}
b, _ = json.Marshal(&superadminOutput)
fmt.Printf("%s\n\n", string(b))
// {"id":"007","name":"James Bond","role":"admin","code":"12345678"}
}
// for appengine could do something like
// ...
// var users []AdminOutputUser // or User or SuperadminOutputUser
// q := datastore.NewQuery("User")
// keys, err := q.GetAll(ctx, &users)
// ...
https://play.golang.org/p/ignIz0hP0z
You might just define your struct like this
type User struct {
ID int64 `json:"id"`
Name string `json:"name"`
Role string `json:"role,omitempty"`
}
And then set them like this
normalUser := User{ID: "boring", Name: "Rubber"}
adminUser := User{ID: "powers", Name: "Ruler", Role: "admin"}
Then json.Marshal() or json.NewEncoder().Encode() as usual
Found in How To Use Struct Tags in Go
Note: I know that omitempty was mentioned in a comment and is even
part of #jimt's code example and mentioned as first option, albeit without a simple example. To me being still pretty new to
Go wasn't clear that that would just work as expected. So I figured it might help others as well 🤓