Replacing ObjectId with embedded json when marshaling struct - json

I'm building a RESTful API with Go and MongoDB, and I'm running into some difficulty with embedding the JSON for one document inside the JSON for another. Here's a toy example of what I'm trying to accomplish. I have the following schemas:
type Post struct {
ID bson.ObjectId `json:"id,omitempty"`
Title string `json:"title,omitempty"`
Owner bson.ObjectId `json:"owner,omitempty"` // references a User
}
type User struct {
ID bson.ObjectId `json:"id,omitempty"`
Name string `json:"name,omitempty"`
}
When creating the JSON for a post, I'd like to first look up the owner of the post in MongoDB and embed the resulting user inside said post's JSON (in-place of the original ObjectId), like so:
{
"id": "...",
"title": "My awesome post",
"owner": {
"id": "...",
"name": "Cody"
}
}
I'm not quite sure how to accomplish this, other than manually constructing the JSON using map[string]interface{}, like so:
post := LookupPost(...)
user := LookupUser(post.Owner)
m := map[string]interface{}{
"id": post.ID,
"title": post.Title,
"owner": map[string]interface{}{
"id": user.ID,
"name": user.Name,
},
}
b, _ := json.Marshal(m)
Obviously this doesn't scale very well isn't very DRY -- ideally, I'd be able to utilize the json tags in each struct definition and have the fields inserted automatically.
Am I missing something, or is what I'm trying to do impossible? Or am I simply not approaching MongoDB/JSON in Go correctly? To put things in perspective, I'm coming from a Node.js background, where this sort of functionality is trivial.
Edit
To clarify things, here's some incorrect Go code that shows what I'd like to do
func getPostJSON() []byte {
p := LookupPost(...)
u := LookupUser(p.Owner, ...)
uj, _ := json.Marshal(u)
p.Owner = uj // can't do this in Go
pj, _ := json.Marshal(p)
return pj
}

I'm not familar with MongoDB or bson.ObjectId, but can you substitute your own type for your User field and have MongoDB easily fill that in for you from a user's bson.ObjectId?
If so you can just wrap user object id's into their own type that implements the json.Marshaler interface. E.g.:
// Embedded (instead of `type x bson.ObjectId`) so that we
// get all the methods and satisfy all the interfaces that
// bson.ObjectId does. Hopefully that's engough to allow MongoDB
// to fill in fields of this type from a database??
type ownerObjID struct{ bson.ObjectId }
// Here we marshal the results of looking up the user from the id
// rather than just the ID itself.
func (oid ownerObjID) MarshalJSON() ([]byte, error) {
user, err := LookupUser(oid.ObjectId)
if err != nil {
return nil, err
}
return json.Marshal(user)
}
type Post struct {
ID bson.ObjectId `json:"id,omitempty"`
Title string `json:"title,omitempty"`
Owner ownerObjID `json:"owner,omitempty"` // <-- is this type wrapping doable/easy with MongoDB?
}
type User struct {
ID bson.ObjectId `json:"id,omitempty"`
Name string `json:"name,omitempty"`
}
func main() {
post := LookupPost()
b, err := json.MarshalIndent(post, "", " ")
if err != nil {
log.Fatal(err)
}
fmt.Printf("JSON:\n%s\n", b)
}
// Some stubs for demo:
func LookupPost() Post {
return Post{
ID: "postID001",
Title: "Ima Test",
Owner: ownerObjID{"ownerID002"},
}
}
func LookupUser(id bson.ObjectId) (User, error) {
return User{
ID: id,
Name: "name for " + string(id),
}, nil
}
Playground
Gives me:
JSON:
{
"id": "postID001",
"title": "Ima Test",
"owner": {
"id": "ownerID002",
"name": "name for ownerID002"
}
}

So I actually discovered a much cleaner solution to this problem:
type Post struct {
ID bson.ObjectId `bson:"_id,omitempty" json:"id,omitempty"`
Title string `bson:"title,omitempty" json:"title,omitempty"`
Owner UserRef `bson:"owner,omitempty" json:"owner,omitempty"`
}
type User struct {
ID bson.ObjectId `json:"id,omitempty"`
Name string `json:"name,omitempty"`
}
type UserRef bson.ObjectId
func (ref UserRef) GetBSON() (interface{}, error) {
return bson.ObjectId(ref), nil
}
func (ref UserRef) MarshalJSON() ([]byte, error) {
u := LookupUserInMongoDB(ref)
return json.Marshal(u)
}
Here's how it works -- mgo can't store UserRef as an ObjectId when converting a Post to bson, so we can implement the GetBSON method for UserRef to return the underlying ObjectId. This allows us to store Owner as an ObjectId in the database. And, like in #DaveC's answer, we implement the MarshalJSON method for UserRef so that when converting a Post to json, we can replace the ObjectId with an actual embedded user.

Related

Golang: Validate Struct Fields in Slice Items

I'm new to Golang.
golang version: 1.17.8
validator: "github.com/go-playground/validator/v10"
I want to validate an incoming JSON payload after loaded into nested struct data structure.
Here's my incoming JSON payload,
{
"name": "Yomiko",
"address": {
"city": "Tokyo",
"street": "Shibaura St"
},
"children":[
{
"lastName": "Takayashi"
}
],
"isEmployed": false
}
Here's my user.go file,
package main
type User struct {
Name string
Address *Address `validate:"required"`
Children []*Child
IsEmployed *bool `validate:"required"`
}
type Address struct {
City string `validate:"required"`
Street string `validate:"required"`
}
type Child struct {
Title string `validate:"required"`
FirstName string
LastName string `validate:"required"`
}
Here's my test function,
func TestUserPayload(t *testing.T) {
actualUserPayload := NewUserPayloadFromFile("userpayload.json")
validate := validator.New()
err := validate.Struct(actualUserPayload)
if err != nil {
t.Error("Validation Error: ", err)
}
}
This test passes. However, I expected it to fail as Child.Title is marked as required. I expected the following error,
Validation Error: Key: 'Child.Title' Error:Field validation for 'Title' failed on the 'required' tag
However, when I loop through the children slice and validate each child struct as follows the test fails as expected,
func TestUserPayload(t *testing.T) {
actualUserPayload := NewUserPayloadFromFile("userpayload.json")
validate := validator.New()
err := validate.Struct(actualUserPayload)
if err != nil {
t.Error("Validation Error: ", err)
}
children := actualUserPayload.Children
for _, child := range children {
err := validate.Struct(child)
if err != nil {
t.Error("Validation Error: ", err)
}
}
}
Is there a straightforward way to do this validation of the items in a slice of structs?
According to the documentation of the validator package, you can use dive in your struct tag to get this behavior. This causes the validator to also validate the nested struct/slice/etc.
So you would need to update your User struct to this:
type User struct {
Name string
Address *Address `validate:"required"`
Children []*Child `validate:"dive"`
IsEmployed *bool `validate:"required"`
}
Here it is working in Go Playground

How to handle missing fields in a JSON response dynamically in Go

I'm working on a Go wrapper for an API and I noticed that two of the JSON fields stay empty when they don't have any data.
Basically the API returns a set of information on a given url, and if it was visited at least once, everything is okay and I get a full json that I then Unmarshal into a struct:
{
"stats":{
"status":1,
"date":"09.07.2019",
"title":"Test",
"devices":{
"dev":[
{
"tag":"Desktop"
}
],
"sys":[
{
"tag":"GNU/Linux "
},
{
"tag":"Windows 10"
}
],
"bro":[
{
"tag":"Firefox 67.0"
},
{
"tag":"Chrome 62.0"
}
]
},
"refs":[
{
"link":"www.google.com"
}
]
}
}
This is the struct I'm using:
type Stats struct {
Stats struct {
Status int `json:"status"`
Date string `json:"date"`
Title string `json:"title"`
Devices struct {
Dev []struct {
Tag string `json:"tag"`
} `json:"dev"`
Sys []struct {
Tag string `json:"tag"`
} `json:"sys"`
Bro []struct {
Tag string `json:"tag"`
} `json:"bro"`
} `json:"devices"`
Refs []struct {
Link string `json:"link"`
} `json:"refs"`
} `json:"stats"`
}
When a new url is given, then things become a little bit weird:
{
"stats": {
"status": 1,
"date": "09.07.2019",
"title": "Test2",
"devices": [
],
"refs": [
]
}
}
As you can see, the fields "dev", "sys" and "bro" just disappear because they're not used and when I try to Unmarshal the JSON into the same struct I get json: cannot unmarshal array into Go struct field Stats.device of type [...]
I tried to use two different structs to handle both the responses but I'm sure that there's a way to handle them gracefully with just one.
Any help would be appreciated, thanks!
I finally managed to make it work with an ugly workaround.
I changed my struct to
type Stats struct {
Status int `json:"status"`
Date string `json:"date"`
Title string `json:"title"`
Devices interface{} `json:"devices"`
Refs interface{} `json:"refs"`
}
Then I can finally Unmarshal the JSON in both cases, but I get a map[string]interface{} when an object is passed and an empty interface{} when an empty array is passed. In order to fix this inconsistency, I simply check for the data type and force the use of a JSON intermediate conversion in order to unpack the map[string]interface{} value inside a custom Devices struct:
// Devices contains devices information
type Devices struct {
Dev []struct {
Tag string `json:"tag"`
Clicks string `json:"clicks"`
} `json:"dev"`
Sys []struct {
Tag string `json:"tag"`
Clicks string `json:"clicks"`
} `json:"sys"`
Bro []struct {
Tag string `json:"tag"`
Clicks string `json:"clicks"`
} `json:"bro"`
}
The algorithms I use are the following:
//ForceDevicesToRightType uses a json conversion as intermediary for filling the Stats.Devices
// struct with map[string]interface{} values
func ForceDevicesToRightType(dev interface{}) (Devices, error) {
temp, err := json.Marshal(dev)
if err != nil {
return Devices{}, err
}
// Use a temporary variable of the right type
var devices Devices
err = json.Unmarshal(temp, &devices)
if err != nil {
return Devices{}, err
}
return devices, nil
}
// ForceRefsToRightType uses a json conversion as intermediary for filling the Stats.Refs
// struct with map[string]interface{} values
func ForceRefsToRightType(refs interface{}) (Refs, error) {
temp, err := json.Marshal(refs)
if err != nil {
return Refs{}, err
}
// Use a temporary variable of the right type
var references Refs
err = json.Unmarshal(temp, &references)
if err != nil {
return Refs{}, err
}
return references, nil
}
Since the compiler knows that both Devices and Refs fields are interface{} I cannot simply access any methods after the conversion, so I simply make a cast of the right type and everything works fine.
For example, if I wanted to access the Dev sub-struct, this is the proper way:
y, _ := GetStats()
fmt.Println(y.Devices.(Devices).Dev)
It's ugly, but it works.
Thank you very much for your help, I hope that this method will save you an headache!

How to create multiple validation methods for one endpoint?

I want to make a validation api in order to validate a set of json requests regarding specific set of rules. To do that I want to use just one endpoint and call functions that correspond to the specific json struct. I know that there is no method overloading in go so I am kind of stumped.
...
type requestBodyA struct {
SomeField string `json:"someField"`
SomeOtherField string `json:"someOtherField"`
}
type requestBodyB struct {
SomeDifferentField string `json:"someDifferentField"`
SomeOtherDifferentField string `json:"someOtherDifferentField"`
}
type ValidationService interface {
ValidateRequest(ctx context.Context, s string) (err error)
}
type basicValidationService struct{}
...
So in order to validate lots of different json requests, is it better to create structs for each and every json request? Or should I create these dynamically? How can I know what kind of request is sent if I only have one endpoint?
If you have a single endpoint/rpc that has to accept different JSON types, you'll need to tell it how to distinguish between them, somehow. One option is to have something like:
type request struct {
bodyA *requestBodyA
bodyB *requestBodyB
}
Then, populate these fields in a container JSON object appropriately. The json module will only populate bodyA if a bodyA key is present, otherwise leaving it a nil, and so on.
Here's a more complete example:
type RequestBodyFoo struct {
Name string
Balance float64
}
type RequestBodyBar struct {
Id int
Ref int
}
type Request struct {
Foo *RequestBodyFoo
Bar *RequestBodyBar
}
func (r *Request) Show() {
if r.Foo != nil {
fmt.Println("Request has Foo:", *r.Foo)
}
if r.Bar != nil {
fmt.Println("Request has Bar:", *r.Bar)
}
}
func main() {
bb := []byte(`
{
"Foo": {"Name": "joe", "balance": 4591.25}
}
`)
var req Request
if err := json.Unmarshal(bb, &req); err != nil {
panic(err)
}
req.Show()
var req2 Request
bb = []byte(`
{
"Bar": {"Id": 128992, "Ref": 801472}
}
`)
if err := json.Unmarshal(bb, &req2); err != nil {
panic(err)
}
req2.Show()
}
Another option is to do it more dynamically with maps, but it's likely that the method above will be sufficient.

How can I parse JSON in GoLang if nested content uses dynamic keys?

I received JSON below from a client API but i am struggling to get nested JSON content. How can I parse it if the inner keys are dynamic?
const jsonStream = `
{
"items": {
"bvu62fu6dq": {
"name": "john",
"age": 23,
"xyz": "weu33s"
},
"iaxdw23fq": {
"name": "kelly",
"age": 21,
"xyz": "weu33s"
}
}
}`
This is what i have tried below by looping the map to extract the value of name and age from above JSON string; but it returns map with nil value as a result.
goplaygound
type Person struct {
Name string `json:"name"`
Age int `json:"age"`
}
type Item struct {
Contact struct {
Info map[string]Person
} `json:"items"`
}
func main() {
dec := json.NewDecoder(strings.NewReader(jsonStream))
for {
var item Item
if err := dec.Decode(&item); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", item.Contact.Info["bvu62fu6dq"].Name)
}
}
Try this instead, looks like you just have your structure set up incorrectly:
http://play.golang.org/p/VRKbv-GVQB
You need to parse the entire json string, which is an object that contains a single element named items. items then contains a map of string -> Person objects.
If you only want to extract name and age from each person, you do it by grabbing data.Items["bvu62fu6dq"].Name.
If you want dynamic keys inside the Person, you'll need to do map[string]interface{} instead of Person in order to capture the dynamic keys again. It would look something like:
type Data struct {
Items map[string]map[string]interface{} `json:"items"`
}
...
fmt.Printf("%v\n", data.Items["bvu62fu6dq"]["name"]
fmt.Printf("%v\n", data.Items["bvu62fu6dq"]["age"]
fmt.Printf("%v\n", data.Items["bvu62fu6dq"]["xyz"]

How to unmarshal two json with same internal structure into one single golang struct?

I have two json files with following structure
{
"cast": [
{
"url": "carey-mulligan",
"name": "Carey Mulligan",
"role": "Actress"
},
{
"url": "leonardo-dicaprio",
"name": "Leonardo DiCaprio",
"role": "Actor"
},
.
.
.
]
}
and
{
"movie": [
{
"url": "carey-mulligan",
"name": "Carey Mulligan",
"role": "Actress"
},
{
"url": "leonardo-dicaprio",
"name": "Leonardo DiCaprio",
"role": "Actor"
},
.
.
.
]
}
as you can see internal structure of the json is same for cast and movie. I want to unmarshel these json file into the same golang structure. But i am not able to give two name tags (cast and movie) for same struct element. I want something like
type Detail struct {
Name string `json:"name"`
Url string `json:"url"`
Role string `json:"role"`
}
type Info struct {
Detail []Detail `json:"cast or movie"`
}
In which case Detail could parse both cast and movie.
Here is my current code
// RIMAGE project main.go
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
const (
website = "https://data.moviebuff.com/"
)
func main() {
fmt.Println("Hello World!")
content, err := ioutil.ReadFile("data/great-getsby")
if err != nil {
panic(err)
}
var info Info
err = json.Unmarshal(content, &info)
if err != nil {
panic(err)
}
fmt.Println(info.Detail)
}
type Detail struct {
Name string `json:"name"`
Url string `json:"url"`
Role string `json:"role"`
}
type Info struct {
Detail []Detail `json:"cast" json:"movie"
}
but it only works for first tag "cast" and gives nill in case json contain the movie.
Thanks in advance.
You can use type Info map[string][]Detail instead of your struct.
Try it on the Go playground
Or you can use both types in your structure, and make method Details() which will return right one:
type Info struct {
CastDetails []Detail `json:"cast"`
MovieDetails []Detail `json:"movie"`
}
func (i Info) Details() []Detail {
if i.CastDetails == nil {
return i.MovieDetails
}
return i.CastDetails
}
Try it on the Go playground
Try anonymous field in struct:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
)
type Detail struct {
Name string `json:"name"`
Url string `json:"url"`
Role string `json:"role"`
}
type Cast struct {
Detail []Detail `json:"cast"`
}
type Movie struct {
Detail []Detail `json:"movie"`
}
type Info struct {
Cast
Movie
}
func (i *Info) getDetails() []Detail {
if len(i.Cast.Detail) > 0 {
return i.Cast.Detail
}
return i.Movie.Detail
}
func main() {
cast, _ := ioutil.ReadFile("./cast.json")
movie, _ := ioutil.ReadFile("./movie.json")
var cInfo Info
err := json.Unmarshal(cast, &cInfo)
fmt.Printf("cast: %+v\n", &cInfo)
fmt.Printf("err: %v\n", err)
fmt.Printf("details: %v\n", cInfo.getDetails())
var mInfo Info
err = json.Unmarshal(movie, &mInfo)
fmt.Printf("movie: %+v\n", &mInfo)
fmt.Printf("err: %v\n", err)
fmt.Printf("details: %v\n", mInfo.getDetails())
}
Things to note:
One more level of indirection: to access 'Details' field, you need to access either 'Cast' or 'Movie' field first in Info first.
Better provide an access function for 'Details' ('getDetail' in this example)
If you dig far enough down into encoding/json you'll get to https://github.com/golang/go/blob/master/src/encoding/json/encode.go and the following:
tag := sf.Tag.Get("json")
if tag == "-" {
continue
}
So it gets one json tag and keeps on going.
You could always go with RoninDev's solution and just copy it over when done.