Unmarshalling Complex Json - json

I just recently started working with apis and http requests and I'm trying to build an application that uses the Reddit API to pull posts on a specific subreddit.
This is the the page with json and search parameters that I'm practicing on:
https://www.reddit.com/r/hiphopheads.json?limit=1
Looking at the standard library of the JSON module for Golang, I still don't understand how to use json.Unmarshal for this complex JSON. From what I gather, I have to define a struct that resembles the JSON structure to actually hold the data
I posted the link into this website to get a feel for what the JSON is actually strucutred like: https://jsonformatter.curiousconcept.com/
Right now the main thing I'm after is the title which is under Data->Children->Data->Title. If I want to unmarshal the JSON into an object, do I define a nested struct object? Or is there a simpler way to do this so that I don't have to figure out all the attributes of the JSON and define them myself??
Any help that can get me on the right track is greatly appreciated. Thanks!

You don't have to define fields you don't need in your struct. Unmarshal will only decode the values that are present in your struct. But with nested JSONs you unfortunately have to define all the parent fields also (unlike in xml package in which you can define paths in tags). So your struct could look like this:
type Foo struct {
Data struct {
Children []struct {
Data struct {
Title string
}
}
}
}
See here for a working example: https://play.golang.org/p/UeUYfWBONL

It seems like the JSON you are trying to unmarshal is over complex, so your struct has to be complex, but that's the way it goes.
There are a few tools that generate struct definitions from a JSON, and that saves a ton of work. Using the JSON you posted and such an online tool, I generated the following struct:
package main
type MyJsonName struct {
Data struct {
After string `json:"after"`
Before interface{} `json:"before"`
Children []struct {
Data struct {
ApprovedBy interface{} `json:"approved_by"`
Archived bool `json:"archived"`
Author string `json:"author"`
AuthorFlairCSSClass string `json:"author_flair_css_class"`
AuthorFlairText interface{} `json:"author_flair_text"`
BannedBy interface{} `json:"banned_by"`
Clicked bool `json:"clicked"`
ContestMode bool `json:"contest_mode"`
Created int `json:"created"`
CreatedUtc int `json:"created_utc"`
Distinguished string `json:"distinguished"`
Domain string `json:"domain"`
Downs int `json:"downs"`
Edited bool `json:"edited"`
Gilded int `json:"gilded"`
Hidden bool `json:"hidden"`
HideScore bool `json:"hide_score"`
ID string `json:"id"`
IsSelf bool `json:"is_self"`
Likes interface{} `json:"likes"`
LinkFlairCSSClass string `json:"link_flair_css_class"`
LinkFlairText string `json:"link_flair_text"`
Locked bool `json:"locked"`
Media interface{} `json:"media"`
MediaEmbed struct{} `json:"media_embed"`
ModReports []interface{} `json:"mod_reports"`
Name string `json:"name"`
NumComments int `json:"num_comments"`
NumReports interface{} `json:"num_reports"`
Over18 bool `json:"over_18"`
Permalink string `json:"permalink"`
Quarantine bool `json:"quarantine"`
RemovalReason interface{} `json:"removal_reason"`
ReportReasons interface{} `json:"report_reasons"`
Saved bool `json:"saved"`
Score int `json:"score"`
SecureMedia interface{} `json:"secure_media"`
SecureMediaEmbed struct{} `json:"secure_media_embed"`
Selftext string `json:"selftext"`
SelftextHTML string `json:"selftext_html"`
Stickied bool `json:"stickied"`
Subreddit string `json:"subreddit"`
SubredditID string `json:"subreddit_id"`
SuggestedSort interface{} `json:"suggested_sort"`
Thumbnail string `json:"thumbnail"`
Title string `json:"title"`
Ups int `json:"ups"`
URL string `json:"url"`
UserReports []interface{} `json:"user_reports"`
Visited bool `json:"visited"`
} `json:"data"`
Kind string `json:"kind"`
} `json:"children"`
Modhash string `json:"modhash"`
} `json:"data"`
Kind string `json:"kind"`
}
Usually, the output of these tools still needs to manual tweaking to work properly. for example:
MediaEmbed struct{} `json:"media_embed"`
I'm pretty sure that's not what's needed here. But it does go a long way in showing the basic idea and figuring out most of the stuff correctly. There are other similar tools you can try.

Related

Use Validator/Dive to enter an array and validate nested structs in go

type myType struct {
value int `json:"value"`
Name string `json:"name" validate:"required"`
URL string `json:"URL" validate:"required"`
args []otherType `json:"args" validate:"dive", "required"`
}
type otherType struct {
name string `validate:"required"`
origin string `validate:"required"`
}
err := paramsValidator.Validate(someInstantiationOfThisStruct)
Hello there! I'm a tad bit stumped on using validator's dive feature. This specific combination of validation scheme isn't present in the documentation for the validator, and I was unable to get it working with a little bit of tweaking.
I would like to simply enter the args array in the primary struct, and validate each of two sets of otherType. However I don't quite understand how this is supposed to transpire.
I understand dive incorrectly and it's not working of course, as the validator is unable to determine incorrect validations using Validate().
Is there any particular thing I'm doing wrong? In general how should I approach evaluating and validating args that are in an array?
I was able to figure it out. I am so sorry for even posting! I was stumped for thirty minutes, but the solution was not that particularly bad.
type myType struct {
value int `json:"value"`
Name string `json:"name" validate:"required"`
URL string `json:"URL" validate:"required"`
args []otherType `json:"args" validate:"dive", "required"`
}
type otherType struct {
name string `validate:"required"`
origin string `validate:"required"`
}
is the updated code. There was a missing , between "dive" and "required", and I had posted code that read
`validate: "dive, required"
dyslexia sorry! :(
I came searching for the answer here but the solution didn't work for me.
In order to validate nested struct using go-playground/validator add dive.
So add below code to the nested struct at top level
`validate:"required,dive,required"`
Note: add without spaces, also make sure the fields are exposed (use PascalCase) to package incase u importing the struct
type myType struct {
value int `json:"value"`
Name string `json:"name" validate:"required"`
URL string `json:"URL" validate:"required"`
Args []OtherType `json:"args" validate:"required,dive,required"`
}
type OtherType struct {
Name string `validate:"required"`
Origin string `validate:"required"`
}
Note: This validation is as per my use case where i want Args to be required and also want it to be exposed to other packages. Just trying to help other who come searching for the same issue as "Dive" is not documented properly in go/playground documentation

How to Unmarshall data to JSON with unknown field types

I have these 'structures'
type Results struct {
Gender string `json:"gender"`
Name struct {
First string `json:"first"`
Last string `json:"last"`
} `json:"name"`
Location struct {
Postcode int `json:"postcode"`
}
Registered struct {
Date string `json:"date"`
} `json:"registered"`
}
type Info struct {
Seed string `json:"seed"`
Results int64 `json:"results"`
Page int64 `json:"page"`
Version string `json:"version"`
}
type Response struct {
Results []Results `json:"results"`
Info Info `json:"info"`
}
I' making a request to an external API and converting data to a JSON view.
I know in advance the types of all fields, but the problem occurs with the 'postcode' field. I'm getting different types of values, which is why I get a JSON decoding error.
In this case, 'postcode' can be one of three variants:
string ~ "13353"
int ~ 13353
string ~ "13353postcode"
Changing the postcode type from string to json.Number solved the problem.
But this solution does not satisfy the third "option".
I know I can try to create a custom type and implement an interface on it. Seems to me the best solution using json.RawMessage. It’s the first I've faced this problem, So I'm still looking for an implementation of a solution to this and reading the documentation.
What's the best way solution in this case?
Thanks in advance.
Declare a custom string type and have it implement the json.Unmarshaler interface.
For example, you could do this:
type PostCodeString string
// UnmarshalJSON implements the json.Unmarshaler interface.
func (s *PostCodeString) UnmarshalJSON(data []byte) error {
if data[0] != '"' { // not string, so probably int, make it a string by wrapping it in double quotes
data = []byte(`"`+string(data)+`"`)
}
// unmarshal as plain string
return json.Unmarshal(data, (*string)(s))
}
https://play.golang.org/p/pp-zNNWY38M

model structs for many to many relationship using std

I'm trying to learn go and I want to create a many to many relationship between a post and a tag. A tag can belong to many posts and a post can have many tags. I am using the standard library using the mysql drive (github.com/go-sql-driver/mysql)
Here's my code:
post.go
package main
type post struct {
ID int `json:"id"`
Title string `json:"title"`
Content string `json:"content"`
Tags: Tag `json:"tags"`
}
tag.go
package main
type tag struct {
ID int `json:"id"`
Name string `json:"name"`
Posts: Post `json:"posts"`
}
Is this the correct way to structure this many to many relationship?
Cheers
Here are a couple of modifications to consider:
Keep your type names consistent (post vs. Post for example)
I think you were going for a slice of Tags and a slice of Posts, so I updated the syntax for that.
One significant question you'll need to answer; do you need/want your structures to be stored in memory with single structs for each item? If so you'll probably want to consider using slices of pointers to Tag and Post instead ([]*Post and []*Tag for example), but popular ORM libraries don't strictly need that (for example gorm: http://jinzhu.me/gorm/associations.html#many-to-many).
Example code:
type Post struct {
ID int `json:"id"`
Title string `json:"title"`
Content string `json:"content"`
Tags []Tag `json:"tags"`
}
type Tag struct {
ID int `json:"id"`
Name string `json:"name"`
Posts []Post `json:"posts"`
}

Override struct tags in go

I have an entity in my project that is viewable by public and by admin. Not all fields should be accessible by public.
For example
type Foo struct {
Id bson.ObjectId `json:"id" bson:"_id"`
DateAdded time.Time `json:"date_added" bson:"date_added"`
Bar string `json:"bar" bson:"bar"`
AdminOnly string `json:"admin_only" bson:"admin_only"`
}
AdminOnly field should be only visible to admins.
For now, when requests comes from public, I call separate method that copies every needed field to new struct
type FooPublic struct {
Id bson.ObjectId `json:"id" bson:"_id"`
DateAdded time.Time `json:"date_added" bson:"date_added"`
Bar string `json:"bar" bson:"bar"`
}
func (f *Foo) Public() (res FooPublic) {
res = FooPublic{
Id: f.Id,
DateAdded: f.DateAdded,
Bar:f.Bar,
}
return
}
But if I need to add new field to my entity, I need to add it in 3 places. In struct itself, in PublicFoo and in Public method.
This seems to be agains DRY principle. What is correct, idiomatic solution here? Can I define FooPublic so it overrides tags of needed fields? Or probably there is at least good way to copy corresponding fields from one struct to another, so I don't need to do this manually in Public method?
In general this repetition can be avoided by using embedding. Your Foo type should embed FooPublic:
type FooPublic struct {
Id bson.ObjectId `json:"id" bson:"_id"`
DateAdded time.Time `json:"date_added" bson:"date_added"`
Bar string `json:"bar" bson:"bar"`
}
type Foo struct {
FooPublic
AdminOnly string `json:"admin_only" bson:"admin_only"`
}
func (f *Foo) Public() FooPublic {
return f.FooPublic
}
But if someone is able to call the Foo.Public(), that someone already has the Foo or *Foo value (and so can access the exported AdminOnly field), so what's the point?
A better solution would be to use an interface, and not expose the Foo struct.

One struct with multiple json representations

The problem I'm trying to solve is that I have a model of a community that looks like this
type Community struct {
Name string
Description string
Sources []Source
Popularity int
FavoriteCount int
Moderators []string
Children []Community
Tracks []Track
}
Communities hold a lot of information and there are scenarios when I want to return only part of the description such as if I'm returning a list of trending communities. In this case I'd want to return only
type Community struct {
Name string
Description string
Popularity int
FavoriteCount int
}
The only way I can think of doing this is to create a new type containing only those fields and write a convenience method that takes a community and returns that type, but essentially creating a new object and copying those fields by value, is there a better way to do this?
I'm aware of the json:"-" syntax, but I'm not sure of how you could do this on a case by case basis as I still need to sometimes return the full object, perhaps a different type that is converted to?
[This](http://attilaolah.eu/2014/09/10/json-and-struct-composition-in-go/
) is a cool approach, which involves creating a sort of Masking struct.
Here's the example in the article:
type User struct {
Email string `json:"email"`
Password string `json:"password"`
// many more fields…
}
type omit *struct{}
type PublicUser struct {
*User
Password omit `json:"password,omitempty"`
}
// when you want to encode your user:
json.Marshal(PublicUser{
User: user,
})
I developed a library which can help you in this regard: Sheriff
You can annotate your struct fields with special tags and call Sheriff to transform the given struct into a subset of it. After that you can call json.Marshal() or whatever else you want to marshal into.
Your example would become as simple as:
type Community struct {
Name string `json:"name" groups:"trending,detail"`
Description string `json:"description" groups:"trending,detail"`
Sources []Source `json:"sources" groups:"detail"`
Popularity int `json:"popularity" groups:"trending,detail"`
FavoriteCount int `json:"favorite_count" groups:"trending,detail"`
Moderators []string `json:"moderators" groups:"detail"`
Children []Community `json:"children" groups:"detail"`
Tracks []Track `json:"tracks" groups:"detail"`
}
communities := []Community{
// communities
}
o := sheriff.Options{
Groups: []string{"trending"},
}
d, err := sheriff.Marshal(&o, communities)
if err != nil {
panic(err)
}
out, _ := json.Marshal(d)
Yep that is the only way as far as I know using the default marshaler. The only other option is if you create your own json.Marshaler .
type Community struct {
}
type CommunityShort Community
func (key *Community) MarshalJSON() ([]byte, os.Error) {
...
}
func (key *Community) UnmarshalJSON(data []byte) os.Error {
...
}
func (key *CommunityShort) MarshalJSON() ([]byte, os.Error) {
...
}
func (key *CommunityShort) UnmarshalJSON(data []byte) os.Error {
...
}
I'll present you another approach that I've developed. I think it's much more clean. The only downside is slightly complicated object initialization, but in usage it's very streamlined.
The main point is that you're not basing your JSON-view-object on the original object and then hiding elements in it, but the other way around, making it a part of the original object:
type CommunityBase struct {
Name string
Description string
}
type Community struct {
CommunityBase
FavoriteCount int
Moderators []string
}
var comm = Community{CommunityBase{"Name", "Descr"}, 20, []string{"Mod1","Mod2"}}
json.Marshal(comm)
//{"Name":"Name","Description":"Descr","FavoriteCount":20,"Moderators":["Mod1","Mod2"]}
json.Marshal(comm.CommunityBase)
//{"Name":"Name","Description":"Descr"}
And that's all if you need only one view, or if your views are gradually expanded.
But if your views can't be inherited, you'll have to resort to a kind of mixins, so you can make a combined view from them:
type ThingBaseMixin struct {
Name string
}
type ThingVisualMixin struct {
Color string
IsRound bool
}
type ThingTactileMixin struct {
IsSoft bool
}
type Thing struct {
ThingBaseMixin
ThingVisualMixin
ThingTactileMixin
Condition string
visualView *ThingVisualView
tactileView *ThingTactileView
}
type ThingVisualView struct {
*ThingBaseMixin
*ThingVisualMixin
}
type ThingTactileView struct {
*ThingBaseMixin
*ThingTactileMixin
}
func main() {
obj := Thing {
ThingBaseMixin: ThingBaseMixin{"Bouncy Ball"},
ThingVisualMixin: ThingVisualMixin{"blue", true},
ThingTactileMixin: ThingTactileMixin{false},
Condition: "Good",
}
obj.visualView = &ThingVisualView{&obj.ThingBaseMixin, &obj.ThingVisualMixin}
obj.tactileView = &ThingTactileView{&obj.ThingBaseMixin, &obj.ThingTactileMixin}
b, _ := json.Marshal(obj)
fmt.Println(string(b))
//{"Name":"Bouncy Ball","Color":"blue","IsRound":true,"IsSoft":false,"Condition":"Good"}
b, _ = json.Marshal(obj.ThingVisualMixin)
fmt.Println(string(b))
//{"Color":"blue","IsRound":true}
b, _ = json.Marshal(obj.visualView)
fmt.Println(string(b))
//{"Name":"Bouncy Ball","Color":"blue","IsRound":true}
b, _ = json.Marshal(obj.tactileView)
fmt.Println(string(b))
//{"Name":"Bouncy Ball","IsSoft":false}
}
Here I've added a view into the object, but if you like, you can create it just when calling Marshal:
json.Marshal(ThingVisualView{&obj.ThingBaseMixin, &obj.ThingVisualMixin})
Or even without a preliminary type declaration:
json.Marshal(struct{*ThingBaseMixin;*ThingVisualMixin}{&obj.ThingBaseMixin,&obj.ThingVisualMixin})
Not sure why this isn't the preferred method, maybe due to the age of the post, but as far as I know, this is the 'best practice' way to handle this, with 'omitempty' tags for those which don't have to exist in the JSON object.
type Community struct {
Name string `json:"name"`
Description string `json:"description"`
Sources *[]Source `json:"sources,omitempty"`
Popularity int `json:"popularity"`
FavoriteCount int `json:"favorite-count"`
Moderators *[]string `json:"moderators,omitempty"`
Children *[]Community `json:"children,omitempty"`
Tracks *[]Track `json:"tracks,omitempty"`
}