Best way to store data in BoltDB - json

I am new to BoltDB and Golang, and trying to get your help.
So, I understand that I can only save byte array ([]byte) for key and value of BoltDB. If I have a struct of user as below, and key will be the username, what would be the best choice to store the data into BoltDB where it expects array of bytes?
Serializing it or JSON? Or better way?
type User struct {
name string
age int
location string
password string
address string
}
Thank you so much, have a good evening

Yes, I would recommend marshaling the User struct to JSON and then use a unique key []byte slice. Don't forget that marshaling to JSON only includes the exported struct fields, so you'll need to change your struct as shown below.
For another example, see the BoltDB GitHub page.
type User struct {
Name string
Age int
Location string
Password string
Address string
}
func (user *User) save(db *bolt.DB) error {
// Store the user model in the user bucket using the username as the key.
err := db.Update(func(tx *bolt.Tx) error {
b, err := tx.CreateBucketIfNotExists(usersBucket)
if err != nil {
return err
}
encoded, err := json.Marshal(user)
if err != nil {
return err
}
return b.Put([]byte(user.Name), encoded)
})
return err
}

A good option is the Storm package, which allows for exactly what you are wanting to do:
package main
import (
"fmt"
"github.com/asdine/storm/v3"
)
type user struct {
ID int `storm:"increment"`
address string
age int
}
func main() {
db, e := storm.Open("storm.db")
if e != nil {
panic(e)
}
defer db.Close()
u := user{address: "123 Main St", age: 18}
db.Save(&u)
fmt.Printf("%+v\n", u) // {ID:1 address:123 Main St age:18}
}
As you can see, you don't have to worry about marshalling, it takes care of it for you. By default it uses JSON, but you can configure it to use GOB or others as well:
https://github.com/asdine/storm

Related

Changing JSON tags in struct with custom MarshalJSON

We get some JSON input, unmarshal, perform some work, then marshal and ship off somewhere else. The JSON we get may have a field named "user". When we marshal back to JSON we need to have that field "user" changed to "username". We can do this by creating a new struct with all the same fields, but different JSON tags, but that seemed a bit cumbersome. I thought a custom marshaller would work here, but I'm a bit stuck. Consider the following code.
package main
import (
"encoding/json"
"fmt"
)
type StructA struct {
Username string `json:"user"`
Process string `json:"process"`
}
func main() {
var test1 StructA
err := json.Unmarshal([]byte(`{"user": "user123", "process": "something"}`), &test1)
if err != nil {
fmt.Println(err)
}
// do some work with test1
jsonByte, err := json.Marshal(&test1)
if err != nil {
fmt.Println(err)
}
fmt.Println(string(jsonByte))
}
func (u *StructA) MarshalJSON() ([]byte, error) {
type Alias StructA
return json.Marshal(&struct {
Username string `json:"username"`
*Alias
}{
Username: u.Username,
Alias: (*Alias)(u),
})
}
https://play.golang.org/p/_w0rlQrcgrW
Ideally this would allow me to change the JSON tag on that field from "user" to "username". However, I get both "username" and "user".
{"username":"user123","user":"user123","process":"something"}
I certainly could create an entirely new struct that mirrors StructA with the tags I want, but I don't have to have to copy every single field and worry about keeping those two structs in sync. It's not the end of the world, but it doesn't seem like a good approach.
To be clear, the end result I'm looking for is the following:
{"username":"user123","process":"something"}
I'm sure I'm missing something trivial here, but it's been a long week and any assistance would be appreciated. Thanks!
One option could be to have one struct with the non-changing values and than 2 alternative structs which both include that struct and have only the changing values. You then use one for unmarshaling and the second one for marshaling.
type StructA struct {
Process string `json:"process"`
...
}
type WithUser struct {
StructA
Username `json:"user"`
}
type WithUsername struct {
StructA
Username `json:"username"`
}
This would require multiple structs but no duplication in each one and can be quite flexible in what you include, instead of hard coding what you want to change into a custom marshal function.
use reflect to create struct and change it's tag
package main
import (
"encoding/json"
"fmt"
"reflect"
)
type StructA struct {
Username string `json:"user"`
Process string `json:"process"`
}
func main() {
var test1 StructA
err := json.Unmarshal([]byte(`{"user": "user123", "process": "something"}`), &test1)
if err != nil {
fmt.Println(err)
}
// do some work with test1
jsonByte, err := json.Marshal(&test1)
if err != nil {
fmt.Println(err)
}
fmt.Println(string(jsonByte))
}
func (u *StructA) MarshalJSON() ([]byte, error) {
// get old struct fields
uType := reflect.TypeOf(u).Elem()
userNameField, _ := uType.FieldByName("Username")
// set username field tag
userNameField.Tag = `json:"username"`
processField, _ := uType.FieldByName("Process")
newType := reflect.StructOf([]reflect.StructField{userNameField, processField})
// set new value field
oldValue := reflect.ValueOf(u).Elem()
newtValue := reflect.New(newType).Elem()
for i := 0; i < oldValue.NumField(); i++ {
newtValue.Field(i).Set(oldValue.Field(i))
}
return json.Marshal(newtValue.Interface())
}

How do I unmarshal a string value from a JSON document into a variable of type int?

My struct:
type User struct {
FirstName string `json:"firstname, omitempty" validate:"required"`
LastName string `json:"lastname, omitempty" validate:"required"`
NumberofDays int `json:"numberofdays, string" validate:"min=0,max=100"`
}
Value for NumberofDays is passed as string from the server but I want to check if it is within range and store as int.
Ex: user := &User{"Michael","Msk","3"}
I'm getting 'cannot unmarshal string into Go value of type int'.
I'm not sure how to typecast to int and do the validation
Remove the space after the comma in the struct tags, e.g. json:"numberofdays, string" should be json:"numberofdays,string". With the space, the json package ignores the string part, hence the error message you're getting. Without the space, the error goes away and behavior is as expected.
Demo comparing the two here: https://play.golang.org/p/AUImnw_PIS
Documentation of json package struct tags: https://golang.org/pkg/encoding/json/#Marshal
You can use a custom type to unmarshal a string
to an integer value using whatever set of parsing rules
you want this mapping to use:
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type User struct {
FirstName string `json:"firstname,omitempty" validate:"required"`
LastName string `json:"lastname,omitempty" validate:"required"`
NumberofDays StrInt `json:"numberofdays" validate:"min=0,max=100"`
}
type StrInt int
func (si *StrInt) UnmarshalJSON(b []byte) error {
var s string
err := json.Unmarshal(b, &s)
if err != nil {
return err
}
n, err := strconv.ParseInt(s, 10, 0)
if err != nil {
return err
}
*si = StrInt(n)
return nil
}
const data = `{
"FirstName": "John",
"LastName": "Doe",
"NumberOfDays": "42"
}`
func main() {
var u User
err := json.Unmarshal([]byte(data), &u)
if err != nil {
panic(err)
}
fmt.Println(u)
}
Playground.
The idea is as follows:
Have a custom type which is mostly int but allows defining custom methods on it.
Have that type satisfy the encoding/json.Unmarshaler interface by implementing the UnmarshalJSON() method on a pointer to that type.
In the method, first unmarshal the value as its native type—string—and
then parse it as an integer.
This approach might appear to be weird, but if you'll think of it
a bit more, there's no single "true" string representation of an integer:
there are different bases and different conventions at representing
integers even in a particular base.
In your unmarshaling method you implement the policy of the text representation
of your integers.
The json package defines a custom type json.Number for these cases.

Golang decode 2 JSON items into 1 struct

i'm trying to decode 2 JSON items into the same struct, because the second JSON complet the first one, but it doesn't work (do nothing) have you got some ideas ?
func getUserClip(this *LibraryController, id string) (*Clip){
//Test Api
//Send Request to azure search
Data := Clip{}
if req := GetClipById("b373400a-bd7e-452a-af68-36992b0323a5"); req == nil {
return nil
} else {
str, err := req.String()
if err != nil {
beego.Debug("Error Json req.String: ", err)
}
//Uncode Json to string
if err := json.Unmarshal([]byte(str), &Data); err != nil {
beego.Debug("Error json", err)
}
for i := range Data.Value {
if req = GetCliRedis(Data.Value[i].Id); err != nil {
return nil
} else {
str, err := req.String()
beego.Debug("JSON REDIS DEBUG: ", str)
if err != nil {
beego.Debug("Error Json req.String: ", err)
}
if err := json.Unmarshal([]byte(str), &Data); err != nil {
beego.Debug("Error json", err)
}
}
i++
}
}
return &Data
}
and the struct
type Clip struct {
Value []InfoClip `json:value`
}
type InfoClip struct {
Id string `json:id`
CreatedAt time.Time `json:createdAt`
StartTimeCode int `json:startTimeCode`
EndTimeCode int `json:endTimeCode`
Metas metas `json:metas`
Tags []string `json:tags`
Categories []string `json:categories`
UserId string `json:userId`
SourceId string `json:sourceId`
ProviderName string `json:providerName`
ProviderReference string `json:providerReference`
PublicationStatus string `json:publicationStatus`
Name string `json:name`
FacebookPage string `json:facebookPage`
TwitterHandle string `json:twitterHandle`
PermaLinkUrl string `json:permalinkUrl`
Logo string `json:logo`
Link string `json:link`
Views int `json:views`
}
type metas struct {
Title string `json:title`
Tags []string `json:tags`
Categories []string `json:categories`
PermaLink string `json:permalink`
}
The JSON I receive is:
{
"clipId":"9b2ea9bb-e54b-4291-ba16-9211fa3c755f",
"streamUrl":"https://<edited out>/asset-32e43a5d-1500-80c3-cc6b-f1e4fe2b5c44\/6c53fbf5-dbe9-4617-9692-78e8d76a7b6e_H264_500kbps_AAC_und_ch2_128kbps.mp4?sv=2012-02-12&sr=c&si=17ed71e8-5176-4432-8092-ee64928a55f6&sig=KHyToRlqvwQxWZXVvRYOkBOBOF0SuBLVmKiGp4joBpw%3D&st=2015-05-18T13%3A32%3A41Z&se=2057-05-07T13%3A32%3A41Z",
"startTimecode":"6",
"endTimecode":"16",
"createdAt":"2015-05-19 13:31:32",
"metas":"{\"title\":\"Zapping : Obama, Marine Le Pen et Michael Jackson\",\"tags\":[\"actualite\"],\"categories\":[\"actualite\"],\"permalink\":\"http:\/\/videos.lexpress.fr\/actualite\/zapping-obama-marine-le-pen-et-michael-jackson_910357.html\"}",
"sourceId":"6c53fbf5-dbe9-4617-9692-78e8d76a7b6e",
"providerName":"dailymotion",
"providerReference":"x1xmnxq",
"publicationStatus":"1",
"userId":"b373400a-bd7e-452a-af68-36992b0323a5",
"name":"LEXPRESS.fr",
"facebookPage":"https:\/\/www.facebook.com\/LExpress",
"twitterHandle":"https:\/\/twitter.com\/lexpress",
"permalinkBaseURL":"https:\/\/tym.net\/fr\/{CLIP_ID}",
"logo":"lexpress-120px.png",
"link":"http:\/\/videos.lexpress.fr\/"
}
The Redis complet the azure search missing information :
here the struct :
type Clip struct {
Value []SearchClip `json:value`
}
type SearchClip struct {
Id string `json:id`
CreatedAt string`json:createdAt`
Tags []string `json:tags`
Categories []string `json:categories`
UserId string `json:userId`
SourceId string `json:sourceId`
Views int `json:views`
}
this is the basic information and redis complet this
I don't want to merge 2 struct into a third one i think it's not the better process, i will do it if it's the last solution.
For lack of activity I'm just gonna post the embedding option as a solution. It's probably the simplest way to do what you want.
type ClipInfoAndMeta struct {
Metas
InfoClip
}
Note I upper cased the name on metas not sure it's necessary but I believe it will be. The language feature being used here is called 'embedding' and it works a lot like composition except that the fields/methods for embedded types are more or less 'hoisted' to the containing types scope. IE with an instance of ClipInfoAndMeta you can directly access any exported field that is defined on InfoClip.
One oddity of your set up is that you'll have collisions on field names between the two types. Not sure how that would play out. With all this being said, it would be helpful to see the json string you're trying to Unmarshal from. As I've been writing this I realized that metas is just a subset of InfoClip. Which has confused me about what you are actually trying to do? I mean, if the data coming back is all in one object, it would mean InfoClip is sufficient for storing all of it. If that is the case you have no reason for the other object... And if you want to trim down the fields which get passed to the display layer of your app you should just define a method on the InfoClip type like func (i *InfoClip) GetMetas() Metas { return &Metas{ ... } } then you can just deal with the one type everywhere and hand off the Metas to the display layer when it's needed.
After a lot of trial and error, I present to you this fully functional solution:
package main
import (
"encoding/json"
"fmt"
"log"
"time"
)
type Clip struct {
Value []InfoClip `json:value`
}
type customTime struct {
time.Time
}
const ctLayout = "2006-01-02 15:04:05"
func (ct *customTime) UnmarshalJSON(b []byte) (err error) {
if b[0] == '"' && b[len(b)-1] == '"' {
b = b[1 : len(b)-1]
}
ct.Time, err = time.Parse(ctLayout, string(b))
return
}
type InfoClip struct {
Id string `json:"clipId"`
CreatedAt customTime `json:"createdAt"`
StartTimeCode string `json:"startTimeCode"` //if you want ints here, you'll have to decode manually, or fix the json beforehand
EndTimeCode string `json:"endTimeCode"` //same for this one
Metas metas `json:"-"`
MetasString string `json:"metas"`
Tags []string `json:"tags"`
Categories []string `json:"categories"`
UserId string `json:"userId"`
SourceId string `json:"sourceId"`
ProviderName string `json:"providerName"`
ProviderReference string `json:"providerReference"`
PublicationStatus string `json:"publicationStatus"`
Name string `json:"name"`
FacebookPage string `json:"facebookPage"`
TwitterHandle string `json:"twitterHandle"`
PermaLinkUrl string `json:"permalinkBaseURL"`
Logo string `json:"logo"`
Link string `json:"link"`
Views int `json:"views"`
}
type metas struct {
Title string `json:"title"`
Tags []string `json:"tags"`
Categories []string `json:"categories"`
PermaLink string `json:"permalink"`
}
var jsonString = `{
"clipId":"9b2ea9bb-e54b-4291-ba16-9211fa3c755f",
"streamUrl":"https://<edited out>/asset-32e43a5d-1500-80c3-cc6b-f1e4fe2b5c44\/6c53fbf5-dbe9-4617-9692-78e8d76a7b6e_H264_500kbps_AAC_und_ch2_128kbps.mp4?sv=2012-02-12&sr=c&si=17ed71e8-5176-4432-8092-ee64928a55f6&sig=KHyToRlqvwQxWZXVvRYOkBOBOF0SuBLVmKiGp4joBpw%3D&st=2015-05-18T13%3A32%3A41Z&se=2057-05-07T13%3A32%3A41Z",
"startTimecode":"6",
"endTimecode":"16",
"createdAt":"2015-05-19 13:31:32",
"metas":"{\"title\":\"Zapping : Obama, Marine Le Pen et Michael Jackson\",\"tags\":[\"actualite\"],\"categories\":[\"actualite\"],\"permalink\":\"http:\/\/videos.lexpress.fr\/actualite\/zapping-obama-marine-le-pen-et-michael-jackson_910357.html\"}",
"sourceId":"6c53fbf5-dbe9-4617-9692-78e8d76a7b6e",
"providerName":"dailymotion",
"providerReference":"x1xmnxq",
"publicationStatus":"1",
"userId":"b373400a-bd7e-452a-af68-36992b0323a5",
"name":"LEXPRESS.fr",
"facebookPage":"https:\/\/www.facebook.com\/LExpress",
"twitterHandle":"https:\/\/twitter.com\/lexpress",
"permalinkBaseURL":"https:\/\/tym.net\/fr\/{CLIP_ID}",
"logo":"lexpress-120px.png",
"link":"http:\/\/videos.lexpress.fr\/"
}`
func main() {
res := parseJson(jsonString)
fmt.Printf("%+v\n",res)
}
func parseJson(theJson string) InfoClip {
toParseInto := struct {
InfoClip
MetasString string `json:"metas"`
}{
InfoClip: InfoClip{},
MetasString: ""}
err := json.Unmarshal([]byte(jsonString), &toParseInto)
if err != nil {
log.Panic(err)
}
err = json.Unmarshal([]byte(toParseInto.MetasString), &toParseInto.InfoClip.Metas)
if err != nil {
log.Panic(err)
}
return toParseInto.InfoClip
}
What are we doing in the parseJson function?
We create a new struct and assign that to the toParseInto variable. We design the struct in a way that it contains all of the fields from InfoClip via embedding, and We add a field to temporarily hold the JSON string metas.
We then unmarshal into that struct, which, after fixing the issues listed below, works fine.
After that, we unmarshal that inner JSON into the correct field in the embedded InfoClip.
We can now easily return that embedded InfoClip to get what we really wanted.
Now, all the issues I have identified in your original solution:
The time format in your JSON is not the standard time format to be used in JSON. That is defined in some RFC, but anyways: because of that, we have to use our own type customTime to parse that. It handles just like a normal time.Time, because that is embedded within.
All your json tags were wrong. All of them had missing quotes, and some were just not even correct.
startTimeCode and endTimeCode are strings in the JSON, not ints
Left to you to improve:
Error handling: Don't just panic in the parseJson function, but rather return the error somehow
If you want startTimecode and endTimecode to be available as ints, parse them manually. You can employ a "hack" similar to the one I used to parse the inner JSON.
One final note, not related to this answer but rather to your question: If you had provided both your code and the JSON with your original question, you would have had an answer in probably less than an hour. Please, please don't make this harder than it needs to be.
EDIT: I forgot to provide my sources, I used this question to parse your time format.

Golang slices of struct or newbie trouble building REST

and need your help.
Wanted to build simple api and stuck with some problem.
I've choose gin and database/sql with postgres driver
package main
import (
"database/sql"
"fmt"
"github.com/gin-gonic/gin"
_ "github.com/lib/pq"
)
func main() {
router := gin.Default()
router.GET("/search/:text", SearchWord)
router.Run(":8080")
}
I need to make query to DB and make json out of this request.
func checkErr(err error) {
if err != nil {
panic(err)
}
}
type Message struct {
ticket_id int `json:"ticket_id"`
event string `json:"event"`
}
func SearchWord(c *gin.Context) {
word := c.Params.ByName("text")
db, err := sql.Open("postgres", "host=8.8.8.8 user= password= dbname=sample")
defer db.Close()
checkErr(err)
rows, err2 := db.Query("SELECT ticket_id,event FROM ....$1, word)
checkErr(err)
for rows.Next() {
var ticket_id int
var event string
err = rows.Scan(&ticket_id, &event)
checkErr(err)
fmt.Printf("%d | %s \n\n", ticket_id, event)
}
}
This coda working nice, but when i need to make json.
I need to make struct of a row
type Message struct {
ticket_id int `json:"ticket_id"`
event string `json:"event"`
}
an then i need to create slice , and append every rows.Next() loop an than answer to browser with Json...
c.JSON(200, messages)
But how to do that...don't know :(
disclaimer: I am brand new to go
Since you Scanned your column data into your variables, you should be able to initialize a structure with their values:
m := &Message{ticket_id: ticket_id, event: event}
You could initialize a slice with
s := make([]*Message, 0)
And then append each of your message structs after instantiation:
s = append(s, m)
Because I'm not too familiar with go there are a couple things i'm not sure about:
after copying data from query to your vars using rows.Scan does initializing the Message struct copy the current iterations values as expected??
If there is a way to get the total number of rows from your query it might be slighlty more performant to initialize a static length array, instead of a slice?
I think #inf deleted answer about marshalling your Message to json down the line might need to be addressed, and Message field's might need to be capitalized
copied from #inf:
The names of the members of your struct need be capitalized so that
they get exported and can be accessed.
type Message struct {
Ticket_id int `json:"ticket_id"`
Event string `json:"event"` }
I'm going to cheat a little here and fix a few things along the way:
First: open your database connection pool once at program start-up (and not on every request).
Second: we'll use sqlx to make it easier to marshal our database rows into our struct.
package main
var db *sqlx.DB
func main() {
var err error
// sqlx.Connect also checks that the connection works.
// sql.Open only "establishes" a pool, but doesn't ping the DB.
db, err = sqlx.Connect("postgres", "postgres:///...")
if err != nil {
log.Fatal(err)
}
router := gin.Default()
router.GET("/search/:text", SearchWord)
router.Run(":8080")
}
// in_another_file.go
type Message struct {
TicketID int `json:"ticket_id" db:"ticket_id"`
Event string `json:"event" db:"event"`
}
func SearchWord(c *gin.Context) {
word := c.Params.ByName("text")
// We create a slice of structs to marshal our rows into
var messages []*Message{}
// Our DB connection pool is safe to use concurrently from here
err := db.Select(&messages, "SELECT ticket_id,event FROM ....$1, word)
if err != nil {
http.Error(c.Writer, err.Error(), 500)
return
}
// Write it out using gin-gonic's JSON writer.
c.JSON(200, messages)
}
I hope that's clear. sqlx also takes care of calling rows.Close() for you, which will otherwise leave connections hanging.

Removing fields from struct or hiding them in JSON Response

I've created an API in Go that, upon being called, performs a query, creates an instance of a struct, and then encodes that struct as JSON before sending back to the caller. I'd now like to allow the caller to be able to select the specific fields they would like returned by passing in a "fields" GET parameter.
This means depending on the fields value(s), my struct would change. Is there any way to remove fields from a struct? Or at least hide them in the JSON response dynamically? (Note: Sometimes I have empty values so the JSON omitEmpty tag will not work here) If neither of these are possible, is there a suggestion on a better way to handle this?
A smaller version of the structs I'm using are below:
type SearchResult struct {
Date string `json:"date"`
IdCompany int `json:"idCompany"`
Company string `json:"company"`
IdIndustry interface{} `json:"idIndustry"`
Industry string `json:"industry"`
IdContinent interface{} `json:"idContinent"`
Continent string `json:"continent"`
IdCountry interface{} `json:"idCountry"`
Country string `json:"country"`
IdState interface{} `json:"idState"`
State string `json:"state"`
IdCity interface{} `json:"idCity"`
City string `json:"city"`
} //SearchResult
type SearchResults struct {
NumberResults int `json:"numberResults"`
Results []SearchResult `json:"results"`
} //type SearchResults
I then encode and output the response like so:
err := json.NewEncoder(c.ResponseWriter).Encode(&msg)
The question is asking for fields to be dynamically selected based on the caller-provided list of fields. This isn't possible to be done with the statically-defined json struct tag.
If what you want is to always skip a field to json-encode, then of course use json:"-" to ignore the field. (Note also that this is not required if your field is unexported; those fields are always ignored by the json encoder.) This isn't what the question asks.
To quote the comment on the json:"-" answer:
This [the json:"-" answer] is the answer most people ending up here from searching would want, but it's not the answer to the question.
I'd use a map[string]interface{} instead of a struct in this case. You can easily remove fields by calling the delete built-in on the map for the fields to remove.
That is, if you can't query only for the requested fields in the first place.
use `json:"-"`
// Field is ignored by this package.
Field int `json:"-"`
// Field appears in JSON as key "myName".
Field int `json:"myName"`
// Field appears in JSON as key "myName" and
// the field is omitted from the object if its value is empty,
// as defined above.
Field int `json:"myName,omitempty"`
// Field appears in JSON as key "Field" (the default), but
// the field is skipped if empty.
// Note the leading comma.
Field int `json:",omitempty"`
doc : http://golang.org/pkg/encoding/json/#Marshal
Another way to do this is to have a struct of pointers with the ,omitempty tag. If the pointers are nil, the fields won't be Marshalled.
This method will not require additional reflection or inefficient use of maps.
Same example as jorelli using this method: http://play.golang.org/p/JJNa0m2_nw
You can use the reflect package to select the fields that you want by reflecting on the field tags and selecting the json tag values. Define a method on your SearchResults type that selects the fields you want and returns them as a map[string]interface{}, and then marshal that instead of the SearchResults struct itself. Here's an example of how you might define that method:
func fieldSet(fields ...string) map[string]bool {
set := make(map[string]bool, len(fields))
for _, s := range fields {
set[s] = true
}
return set
}
func (s *SearchResult) SelectFields(fields ...string) map[string]interface{} {
fs := fieldSet(fields...)
rt, rv := reflect.TypeOf(*s), reflect.ValueOf(*s)
out := make(map[string]interface{}, rt.NumField())
for i := 0; i < rt.NumField(); i++ {
field := rt.Field(i)
jsonKey := field.Tag.Get("json")
if fs[jsonKey] {
out[jsonKey] = rv.Field(i).Interface()
}
}
return out
}
and here's a runnable solution that shows how you would call this method and marshal your selection: http://play.golang.org/p/1K9xjQRnO8
I just published sheriff, which transforms structs to a map based on tags annotated on the struct fields. You can then marshal (JSON or others) the generated map. It probably doesn't allow you to only serialize the set of fields the caller requested, but I imagine using a set of groups would allow you to cover most cases. Using groups instead of the fields directly would most likely also increase cache-ability.
Example:
package main
import (
"encoding/json"
"fmt"
"log"
"github.com/hashicorp/go-version"
"github.com/liip/sheriff"
)
type User struct {
Username string `json:"username" groups:"api"`
Email string `json:"email" groups:"personal"`
Name string `json:"name" groups:"api"`
Roles []string `json:"roles" groups:"api" since:"2"`
}
func main() {
user := User{
Username: "alice",
Email: "alice#example.org",
Name: "Alice",
Roles: []string{"user", "admin"},
}
v2, err := version.NewVersion("2.0.0")
if err != nil {
log.Panic(err)
}
o := &sheriff.Options{
Groups: []string{"api"},
ApiVersion: v2,
}
data, err := sheriff.Marshal(o, user)
if err != nil {
log.Panic(err)
}
output, err := json.MarshalIndent(data, "", " ")
if err != nil {
log.Panic(err)
}
fmt.Printf("%s", output)
}
Take three ingredients:
The reflect package to loop over all the fields of a struct.
An if statement to pick up the fields you want to Marshal, and
The encoding/json package to Marshal the fields of your liking.
Preparation:
Blend them in a good proportion. Use reflect.TypeOf(your_struct).Field(i).Name() to get a name of the ith field of your_struct.
Use reflect.ValueOf(your_struct).Field(i) to get a type Value representation of an ith field of your_struct.
Use fieldValue.Interface() to retrieve the actual value (upcasted to type interface{}) of the fieldValue of type Value (note the bracket use - the Interface() method produces interface{}
If you luckily manage not to burn any transistors or circuit-breakers in the process you should get something like this:
func MarshalOnlyFields(structa interface{},
includeFields map[string]bool) (jsona []byte, status error) {
value := reflect.ValueOf(structa)
typa := reflect.TypeOf(structa)
size := value.NumField()
jsona = append(jsona, '{')
for i := 0; i < size; i++ {
structValue := value.Field(i)
var fieldName string = typa.Field(i).Name
if marshalledField, marshalStatus := json.Marshal((structValue).Interface()); marshalStatus != nil {
return []byte{}, marshalStatus
} else {
if includeFields[fieldName] {
jsona = append(jsona, '"')
jsona = append(jsona, []byte(fieldName)...)
jsona = append(jsona, '"')
jsona = append(jsona, ':')
jsona = append(jsona, (marshalledField)...)
if i+1 != len(includeFields) {
jsona = append(jsona, ',')
}
}
}
}
jsona = append(jsona, '}')
return
}
Serving:
serve with an arbitrary struct and a map[string]bool of fields you want to include, for example
type magic struct {
Magic1 int
Magic2 string
Magic3 [2]int
}
func main() {
var magic = magic{0, "tusia", [2]int{0, 1}}
if json, status := MarshalOnlyFields(magic, map[string]bool{"Magic1": true}); status != nil {
println("error")
} else {
fmt.Println(string(json))
}
}
Bon Appetit!
I created this function to convert struct to JSON string by ignoring some fields. Hope it will help.
func GetJSONString(obj interface{}, ignoreFields ...string) (string, error) {
toJson, err := json.Marshal(obj)
if err != nil {
return "", err
}
if len(ignoreFields) == 0 {
return string(toJson), nil
}
toMap := map[string]interface{}{}
json.Unmarshal([]byte(string(toJson)), &toMap)
for _, field := range ignoreFields {
delete(toMap, field)
}
toJson, err = json.Marshal(toMap)
if err != nil {
return "", err
}
return string(toJson), nil
}
Example: https://play.golang.org/p/nmq7MFF47Gp
You can use tagging attribute "omitifempty" or make optional fields pointers and leave those you want skipped uninitialized.
Here is how I defined my structure.
type User struct {
Username string `json:"username" bson:"username"`
Email string `json:"email" bson:"email"`
Password *string `json:"password,omitempty" bson:"password"`
FullName string `json:"fullname" bson:"fullname"`
}
And inside my function set user.Password = nil for not to be Marshalled.
I didn't have the same problem but similar. Below code solves your problem too, of course if you don't mind performance issue. Before implement that kind of solution to your system I recommend you to redesign your structure if you can. Sending variable structure response is over-engineering. I believe a response structure represents a contract between a request and resource and it should't be depend requests.(you can make un-wanted fields null, I do). In some cases we have to implement this design, if you believe you are in that cases here is the play link and code I use.
type User2 struct {
ID int `groups:"id" json:"id,omitempty"`
Username string `groups:"username" json:"username,omitempty"`
Nickname string `groups:"nickname" json:"nickname,omitempty"`
}
type User struct {
ID int `groups:"private,public" json:"id,omitempty"`
Username string `groups:"private" json:"username,omitempty"`
Nickname string `groups:"public" json:"nickname,omitempty"`
}
var (
tagName = "groups"
)
//OmitFields sets fields nil by checking their tag group value and access control tags(acTags)
func OmitFields(obj interface{}, acTags []string) {
//nilV := reflect.Value{}
sv := reflect.ValueOf(obj).Elem()
st := sv.Type()
if sv.Kind() == reflect.Struct {
for i := 0; i < st.NumField(); i++ {
fieldVal := sv.Field(i)
if fieldVal.CanSet() {
tagStr := st.Field(i).Tag.Get(tagName)
if len(tagStr) == 0 {
continue
}
tagList := strings.Split(strings.Replace(tagStr, " ", "", -1), ",")
//fmt.Println(tagList)
// ContainsCommonItem checks whether there is at least one common item in arrays
if !ContainsCommonItem(tagList, acTags) {
fieldVal.Set(reflect.Zero(fieldVal.Type()))
}
}
}
}
}
//ContainsCommonItem checks if arrays have at least one equal item
func ContainsCommonItem(arr1 []string, arr2 []string) bool {
for i := 0; i < len(arr1); i++ {
for j := 0; j < len(arr2); j++ {
if arr1[i] == arr2[j] {
return true
}
}
}
return false
}
func main() {
u := User{ID: 1, Username: "very secret", Nickname: "hinzir"}
//assume authenticated user doesn't has permission to access private fields
OmitFields(&u, []string{"public"})
bytes, _ := json.Marshal(&u)
fmt.Println(string(bytes))
u2 := User2{ID: 1, Username: "very secret", Nickname: "hinzir"}
//you want to filter fields by field names
OmitFields(&u2, []string{"id", "nickname"})
bytes, _ = json.Marshal(&u2)
fmt.Println(string(bytes))
}
I also faced this problem, at first I just wanted to specialize the responses in my http handler. My first approach was creating a package that copies the information of a struct to another struct and then marshal that second struct. I did that package using reflection, so, never liked that approach and also I wasn't dynamically.
So I decided to modify the encoding/json package to do this. The functions Marshal, MarshalIndent and (Encoder) Encode additionally receives a
type F map[string]F
I wanted to simulate a JSON of the fields that are needed to marshal, so it only marshals the fields that are in the map.
https://github.com/jtorz/jsont
package main
import (
"fmt"
"log"
"net/http"
"github.com/jtorz/jsont/v2"
)
type SearchResult struct {
Date string `json:"date"`
IdCompany int `json:"idCompany"`
Company string `json:"company"`
IdIndustry interface{} `json:"idIndustry"`
Industry string `json:"industry"`
IdContinent interface{} `json:"idContinent"`
Continent string `json:"continent"`
IdCountry interface{} `json:"idCountry"`
Country string `json:"country"`
IdState interface{} `json:"idState"`
State string `json:"state"`
IdCity interface{} `json:"idCity"`
City string `json:"city"`
} //SearchResult
type SearchResults struct {
NumberResults int `json:"numberResults"`
Results []SearchResult `json:"results"`
} //type SearchResults
func main() {
msg := SearchResults{
NumberResults: 2,
Results: []SearchResult{
{
Date: "12-12-12",
IdCompany: 1,
Company: "alfa",
IdIndustry: 1,
Industry: "IT",
IdContinent: 1,
Continent: "america",
IdCountry: 1,
Country: "México",
IdState: 1,
State: "CDMX",
IdCity: 1,
City: "Atz",
},
{
Date: "12-12-12",
IdCompany: 2,
Company: "beta",
IdIndustry: 1,
Industry: "IT",
IdContinent: 1,
Continent: "america",
IdCountry: 2,
Country: "USA",
IdState: 2,
State: "TX",
IdCity: 2,
City: "XYZ",
},
},
}
fmt.Println(msg)
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
//{"numberResults":2,"results":[{"date":"12-12-12","idCompany":1,"idIndustry":1,"country":"México"},{"date":"12-12-12","idCompany":2,"idIndustry":1,"country":"USA"}]}
err := jsont.NewEncoder(w).Encode(msg, jsont.F{
"numberResults": nil,
"results": jsont.F{
"date": nil,
"idCompany": nil,
"idIndustry": nil,
"country": nil,
},
})
if err != nil {
log.Fatal(err)
}
})
http.ListenAndServe(":3009", nil)
}
The question is now a bit old, but I came across the same issue a little while ago, and as I found no easy way to do this, I built a library fulfilling this purpose.
It allows to easily generate a map[string]interface{} from a static struct.
https://github.com/tuvistavie/structomap
To extend chhaileng answer, here is the version that remove all occurrences of a field with recursion
// GetJSONWithOutFields - Description: return a string representation of an interface with specified fields removed
func GetJSONWithOutFields(obj interface{}, ignoreFields ...string) (string, error) {
toJson, err := json.Marshal(obj)
if err != nil {
return "", err
}
if len(ignoreFields) == 0 {
return string(toJson), nil
}
toMap := map[string]interface{}{}
err = json.Unmarshal(toJson, &toMap)
if err != nil {
return "", err
}
for _, field := range ignoreFields {
DeleteField(toMap, field)
}
toJson, err = json.Marshal(toMap)
if err != nil {
return "", err
}
return string(toJson), nil
}
// DeleteField - Description: recursively delete field
func DeleteField(toMap map[string]interface{}, field string) {
delete(toMap, field)
for _, v := range toMap {
if m, isMap := v.(map[string]interface{}); isMap {
DeleteField(m, field)
}
}
}