I am using thrift in my project, thrift will generate code as follow:
type CvJdRelationInfo struct {
JdId string `thrift:"jdId,1" json:"jdId"`
CvId string `thrift:"cvId,2" json:"cvId"`
Status int16 `thrift:"status,3" json:"status"`
AcceptTimestamp int64 `thrift:"acceptTimestamp,4" json:"acceptTimestamp"`
}
as you see thrift already generate json tags(but no bson tags), when I use mgo save record, mgo will auto convert:
JdId -> jdid
CvId -> cvid
Status -> status
AcceptTimeStamp -> accepttimestamp
what I needed is:
type CvJdRelationInfo struct {
JdId string `thrift:"jdId,1" json:"jdId" bson:"jdId"`
CvId string `thrift:"cvId,2" json:"cvId" bson:"cvId"`
Status int16 `thrift:"status,3" json:"status" bson:"status"`
AcceptTimestamp int64 `thrift:"acceptTimestamp,4" json:"acceptTimestamp" bson:"acceptTimestamp"`
}
as your can see, bson tags is the same as json tags. Can I use json tags as bson tags?
MongoDB actually stores the data as binary JSON (bson), which is distinct from JSON. It's slightly confusing because if you access the database using the mongo shell, you get back raw JSON, but it's actually a conversion, it's not the storage format. So in storing the data to the database, the "mgo" driver serializes as bson.
This serializing ignores the json export keys, and chooses the appropriate name by defaulting to the lowercase version of your struct field. (See the bson.Marshal go doc.) If you specify a bson export key, it will then ignore the struct field name and go with whatever you have specified as the bson export key.
For example,
type User struct {
Name string
UserAge int `bson:"age"`
Phone string `json:"phoneNumber"`
}
will result in the following structure in MongoDB:
{
"name": "",
"age": 0,
"phone": ""
}
So it seems like your struct fields should handle most things for you.
The one 'gotcha' that you might not see until it bites you is, if you don't specify the bson export keys, you don't have the ability to do bson:",omitempty" for leaving out blank fields, or bson:",inline" for marshaling embedded (or nested) structs.
For instance, this is how you would handle embedded structs:
type Employee struct {
User `bson:",inline"`
JobTitle string
EmployeeId string
Salary int
}
These sorts of things are specified in that link I provided on bson.Marshal. Hope that helps!
you can change the default encoder/decoder in the following way:
mongo_addr := "mongodb://localhost:27017/"
structcodec, _ := bsoncodec.NewStructCodec(bsoncodec.JSONFallbackStructTagParser)
rb := bson.NewRegistryBuilder()
// register struct codec
rb.RegisterDefaultEncoder(reflect.Struct, structcodec)
rb.RegisterDefaultDecoder(reflect.Struct, structcodec)
clientOptions := options.Client().SetRegistry(rb.Build()).ApplyURI(mongo_addr)
mgclient, err := mongo.Connect(ctx, clientOptions)
if err != nil {
panic(err)
}
Now during marshaling/unmarshalling json tag is used instead of bson.
This trick is a particularly useful when you use tags that both bson and json belong to.
By the way, those clientOptions can be set on collection and database levels too
You can use the following (From thrift test file git.apache.org/thrift.git/lib/go/test/GoTagTest.thrift)
struct tagged {
1: string string_thing,
2: i64 int_thing (go.tag = "json:\"int_thing,string\""),
3: optional i64 optional_int_thing
}
Related
I am new to Go and I feel as though I grappling with something fundamental around typing whilst trying to build a JSON payload.
My http web app (a kubernetes mutating webhook) is returning a JSON []byte(string) in a field admissionResponse.Patch
The JSON returned is an array of JSONPatch and looks something like this:
[{
"op": "add",
"path": "/spec/containers",
"value": <SPEC-JSON>
}]
The problem I have is trying to build the different JSONPatch objects in a generic way.
I am Unmarshalling my <SPEC-JSON> from yaml files using various k8s v1 types (E.g. v1.Volume, v1.Container) which is ok, but then because each <SPEC-JSON> is of different type for different JSONPatch objects, I am having to create different struct{} types like so;
type ContainerPatch struct {
Op string `json:"op"`
Path string `json:"path"`
Value []v1.Container `json:"value"`
}
type VolumePatch struct {
Op string `json:"op"`
Path string `json:"path"`
Value []v1.Volume `json:"value"`
}
Now, when I want to put these into an array ready to Marshal back to a JSON string, they are of different types so I am ending up doing:
myPatch := []interface{}{
VolumePatch(PatchVolumes(pod)),
ContainerPatch(PatchContainers(pod)),
ContainerPatch(PatchInitContainers(pod)),
}
data, _ := json.Marshal(myPatch)
which works, but seems wrong? Essentially these are all of type JSONPatch where the json:"value" can be different k8s objects.
How should we do these things in Go?
---update--- (thanks to mkopriva)
Thank you. Reading around a bit, it seems that any is essentially an (empty) interface{}(?). So I tried with both;
type Patch struct {
Op string `json:"op"`
Path string `json:"path"`
Value interface{} `json:"value"`
}
and
type Patch struct {
Op string `json:"op"`
Path string `json:"path"`
Value any `json:"value"`
}
... and both work ok. I like this way as it allows me to have a single patching func() that returns a common type.
I want to omit certain structs nested in a JSON request. I've created a rest API on golang which reads a message body from an http request, decodes it into the struct defined in the code and inserts it into Mongo DB
My structs are as follows. Note that for the nested structure C, I use a pointer in order to be able to omit it.
type A struct {
Title string `json:"title"`
Text string `json:"text"`
Data B `json:"data"`
}
type B struct {
Product *C `json:"product,omitempty"`
ExternalLink string `json:"external_link,omitempty"`
}
type C struct {
Name string `json:"name"`
Id int `json:"id"`
}
Here is how I decode it (Didn't go for Json.Unmarshall as I read that for http bodies, decoding should be used over unmarshall)
func NewMessage(req *http.Request) *A {
var newMessage *A
json.NewDecoder(req.Body).Decode(&newMessage)
messageInData := newMessage
return newMessage
}
The "newMessage" upon return is inserted into Mongo directly. However, Even if the request payload contains no such object as the struct C for instance like below
{
"title": "First message from GoLang",
"text": "Hello Dastgyr",
"data": {
"external_link": "some link here"
//no product object (C struct) here
}
}
The object inserted into Mongo still contains the struct C as having a null value as shown below
{
"title": "First message from GoLang",
"text": "Hello Dastgyr",
"data": {
"product": null,
"external_link": "some link here"
}
}
I've also tried using B as a pointer in Struct A but to no avail
type A struct {
Title string `json:"title"`
Text string `json:"text"`
Data *B `json:"data,omitempty"`
}
I want to be able to omit certain nested structs. Despite using pointers, the struct I want is still not omitting. What mistake am I making in defining structs?
Still new to golang so a nudge in the right direction would help
You are using json tags for json un-marshaling, it seems to be un-marshaling correctly (concluding this as you didn't mention any errors, and moved on to MongoDB)
How you add data to MongoDB is a totally different matter, and has nothing to do with your JSON tags. It uses bson tags, and you will need to add them, if you wish to use this same struct as a mongo DB model representation. Something like this:
type A struct {
Title string `json:"title" bson:"title"`
Text string `json:"text" bson:"text"`
Data *B `json:"data,omitempty" bson:"data,omitempty"`
}
Remember that tags in golang, are just some metadata being added with a structure, which some code actually reads and acts on. json library identifies & processes json:"" tag, while official go mongodb library that you might be using, will process the bson:"" tags.
I use AWS Lambda with DynamoDB using golang.
My DynamoDB table use lowercase attribute names such as id or name.
In Go, if I want to be able to marshal a struct correctly, I have to name fields starting with a capital letter.
type Item struct {
ID string
Name string
}
To put an item into my DynamoDB table, I have to marshal it into a map[string]*dynamodb.AttributeValue, using dynamodbattribute.MarshalMap function.
item := Item{
ID: "xxxx",
Name: "yyyy"
}
av, _ := dynamodbattribute.MarshalMap(item)
Of course, this will create a map using names written as ID and Name, which are incompatible with id and name from the dynamodb table.
Reading the documentation, I found that you can use a custom encoder, and enable json tags.
type Item struct {
ID string `json="id"`
Name string `json="name"`
}
func setOpts(encoder *dynamodbattribute.Encoder) {
// Not sure which one I sould set
// So I set them both :)
encoder.SupportJSONTags = true
encoder.MarshalOptions.SupportJSONTags = true
}
func main() {
encoder := dynamodbattribute.NewEncoder(setOpts)
encoder.Encode(...)
}
But here the encoder.Encode() method is only used to create a dynamodb.AttributeValue, and not a map[string]*dynamodb.AttributeValue.
Is there a way to use a custom encoder with MarshalMap? Or am I using it in a wrong way?
EDIT:
Okay so as Zak pointed out, there is a dynamodbav tag that can be used.
I also found out that I was using json tags in a wrong way. I should use the syntax json:"id" instead of json="id".
Indeed, DynamoDB SDK uses the json tag if available, and this one can be overrided by the dynamodbav.
So all I had to do was to make my structure looks like this and it worked
type Item struct {
ID string `json:"id"`
Name string `json:"name"`
}
Dynamo's built in marshalling, from MarshalMap(...) can support struct tags, similar to json.
You can add them to the type that you are marshalling, like so:
type Item struct {
ID string `dynamodbav:"id"`
Name string `dynamodbav:"name"`
}
See the docs here
I'd like to marshal part of this struct:
type ValueSet struct {
Id string `json:"id" bson:"_id"`
Url string `bson:"url,omitempty" json:"url,omitempty"`
Identifier *Identifier `bson:"identifier,omitempty" json:"identifier,omitempty"`
Version string `bson:"version,omitempty" json:"version,omitempty"`
Name string `bson:"name,omitempty" json:"name,omitempty"`
Status string `bson:"status,omitempty" json:"status,omitempty"`
Experimental *bool `bson:"experimental,omitempty" json:"experimental,omitempty"`
Publisher string `bson:"publisher,omitempty" json:"publisher,omitempty"`
Contact []ValueSetContactComponent `bson:"contact,omitempty" json:"contact,omitempty"`
Date *FHIRDateTime `bson:"date,omitempty" json:"date,omitempty"`
LockedDate *FHIRDateTime `bson:"lockedDate,omitempty" json:"lockedDate,omitempty"`
Description string `bson:"description,omitempty" json:"description,omitempty"`
UseContext []CodeableConcept `bson:"useContext,omitempty" json:"useContext,omitempty"`
Immutable *bool `bson:"immutable,omitempty" json:"immutable,omitempty"`
Requirements string `bson:"requirements,omitempty" json:"requirements,omitempty"`
Copyright string `bson:"copyright,omitempty" json:"copyright,omitempty"`
Extensible *bool `bson:"extensible,omitempty" json:"extensible,omitempty"`
CodeSystem *ValueSetCodeSystemComponent `bson:"codeSystem,omitempty" json:"codeSystem,omitempty"`
Compose *ValueSetComposeComponent `bson:"compose,omitempty" json:"compose,omitempty"`
Expansion *ValueSetExpansionComponent `bson:"expansion,omitempty" json:"expansion,omitempty"`
}
which is part of a Go implementation of HL7 FHIR, including only the metadata fields, and omitting the three content three fields (codeSystem, compose and expansion). I can't (and shouldn't) change the JSON tags in the original source code, since other code strongly depends on it working the way it's written. How can I tell json.Marshal to override the existing JSON tags on these struct elements?
You can't change it, but you don't have to.
Easiest solution is to create your own struct, define your own json tags (how you want them to appear in the output), copy the fields, and marshal a value of your own struct.
E.g. let's say you want to marshal the Id and Urlfields, then:
type MyValueSet struct {
Id string `json:"MyId"`
Url string `json:"MyUrl"`
}
var vs ValueSet = ... // Comes from somewhere
mvs := MyValueSet {
Id: vs.Id,
Url: vs.Url,
}
data, err := json.Marshal(&mvs)
// Check err
I have a JSON object that looks like this:
{"API version":"1.2.3"}
And I want to convert it to an object it using the json.Unmarshal() function in go. According to this blog post:
How does Unmarshal identify the fields in which to store the decoded data? For a given JSON key "Foo", Unmarshal will look through the destination struct's fields to find (in order of preference):
An exported field with a tag of "Foo" (see the Go spec for more on struct tags),
An exported field named "Foo", or
An exported field named "FOO" or "FoO" or some other case-insensitive match of "Foo".
This is confirmed by the unmarshal documentation.
Since "API version" has a space in it, which is not a valid go identifier, I used a tag on the field:
type ApiVersion struct {
Api_version string "API version"
}
And I try to unmarshal it like so:
func GetVersion() (ver ApiVersion, err error) {
// Snip code that gets the JSON
log.Println("Json:",string(data))
err = json.Unmarshal(data,&ver)
log.Println("Unmarshalled:",ver);
}
The output is:
2014/01/06 16:47:38 Json: {"API version":"1.2.3"}
2014/01/06 16:47:38 Unmarshalled: {}
As you can see, the JSON isn't being marshalled into ver. What am I missing?
The encoding/json module requires that the struct tags be namespaced. So you instead want something like:
type ApiVersion struct {
Api_version string `json:"API version"`
}
This is done so that the json struct tags can co-exist with tags from other libraries (such as the XML encoder).