I'm trying to parse part of the Mailgun notify webhook.
It's a POST request with a x-www-form-urlencoded body.
Here's part of the body:
sender: some#email.com
attachments: [{"url": "https://storage.eu.mailgun.net/v3/domains/beep.boop/messages/randomstring/attachments/0", "content-type": "application/pdf", "name": "example.pdf", "size": 345}]"]
The attachments value is a json encoded array
I would like to decode this string from JSON to a StoredAttachment nested struct as I am decoding the response as x-www-form-urlencoded but I don't know how to do it. The target structs are as follow:
type NotifiedMessage struct {
Sender string `schema:"sender"`
Subject string `schema:"subject"`
Attachments []StoredAttachment `schema:"attachments"`
MessageUrl string `schema:"message-url"`
}
// StoredAttachment structures contain information on an attachment associated with a stored message.
type StoredAttachment struct {
Size int `json:"size"`
Url string `json:"url"`
Name string `json:"name"`
ContentType string `json:"content-type"`
}
Here is my non-working code so far: https://play.golang.org/p/Ofbw2VAYV28
You can implement the TextUnmarshaler interface, the schema package will use that interface instead of executing the default procedure and this allows for custom unmarshaling.
1. Declare a named type and use that as the type of the Attachments field. []StoredAttachment is unnamed. So, for example:
type AttachmentList []StoredAttachment
Why? Because methods can be declared only on named types.
2. Implement TextUnmarhsaler interface and do the json unmashaling there.
func (ls *AttachmentList) UnmarshalText(text []byte) (err error) {
return json.Unmarshal(text, (*[]StoredAttachment)(ls))
}
That's it.
https://play.golang.org/p/t65mI7JRFfS
Related
I want to omit certain structs nested in a JSON request. I've created a rest API on golang which reads a message body from an http request, decodes it into the struct defined in the code and inserts it into Mongo DB
My structs are as follows. Note that for the nested structure C, I use a pointer in order to be able to omit it.
type A struct {
Title string `json:"title"`
Text string `json:"text"`
Data B `json:"data"`
}
type B struct {
Product *C `json:"product,omitempty"`
ExternalLink string `json:"external_link,omitempty"`
}
type C struct {
Name string `json:"name"`
Id int `json:"id"`
}
Here is how I decode it (Didn't go for Json.Unmarshall as I read that for http bodies, decoding should be used over unmarshall)
func NewMessage(req *http.Request) *A {
var newMessage *A
json.NewDecoder(req.Body).Decode(&newMessage)
messageInData := newMessage
return newMessage
}
The "newMessage" upon return is inserted into Mongo directly. However, Even if the request payload contains no such object as the struct C for instance like below
{
"title": "First message from GoLang",
"text": "Hello Dastgyr",
"data": {
"external_link": "some link here"
//no product object (C struct) here
}
}
The object inserted into Mongo still contains the struct C as having a null value as shown below
{
"title": "First message from GoLang",
"text": "Hello Dastgyr",
"data": {
"product": null,
"external_link": "some link here"
}
}
I've also tried using B as a pointer in Struct A but to no avail
type A struct {
Title string `json:"title"`
Text string `json:"text"`
Data *B `json:"data,omitempty"`
}
I want to be able to omit certain nested structs. Despite using pointers, the struct I want is still not omitting. What mistake am I making in defining structs?
Still new to golang so a nudge in the right direction would help
You are using json tags for json un-marshaling, it seems to be un-marshaling correctly (concluding this as you didn't mention any errors, and moved on to MongoDB)
How you add data to MongoDB is a totally different matter, and has nothing to do with your JSON tags. It uses bson tags, and you will need to add them, if you wish to use this same struct as a mongo DB model representation. Something like this:
type A struct {
Title string `json:"title" bson:"title"`
Text string `json:"text" bson:"text"`
Data *B `json:"data,omitempty" bson:"data,omitempty"`
}
Remember that tags in golang, are just some metadata being added with a structure, which some code actually reads and acts on. json library identifies & processes json:"" tag, while official go mongodb library that you might be using, will process the bson:"" tags.
I am taking my first steps with GoLang, and currently setting up an API Server, which is able to read JSON file from POST Request and save that to Memory.
I have a JSON File as Following:
[
{
"id": 0,
"name": "kubernetes",
"uri": "https://github.com/kubernetes/kubernetes"
},
{
"id": 1,
"name": "jenkins",
"uri": "https://github.com/jenkinsci/jenkins"
}
]
Which I am POST:ing to the API Server running on local port.
Here is my setupRoutes() - function:
func setupRoutes() {
// Initialize Router
router := gin.Default()
// Initialize Routes
router.GET("/api/projects", getProjects)
router.GET("/api/projects/:id", getProjectByIdentifier)
router.POST("/api/projects", uploadProjects)
// Start the Router
router.Run("localhost:8080")
}
and here is my uploadProjects() - function:
// Reads file from POST request, and saves that to Memory.
func uploadProjects(c *gin.Context) {
// Initialize Object
var obj []Project
// Bind JSON Data to Object
c.BindJSON(&obj)
fmt.Println(obj) // For Testing: What is binded.
// Save Data to Memory
proj = obj
}
and here is the Project Struct:
type Project struct {
Identifier int64 `json: id`
Name string `json: name`
Uri string `json: uri`
}
After executing this - I can print that data out right away, what is being binded or I can fetch that with my GET /api/projects - call, and the result is always:
[{0 kubernetes https://github.com/kubernetes/kubernetes} {0 jenkins https://github.com/jenkinsci/jenkins}]
What I've tried:
I've tried to swap between string, int and int64 types of Identifier Field in my struct.
Googled a Bunch
This is probably something very simple, but I don't really know where to look at this point, so any help is appreciated.
The id field does not match the field name Identifier. Fix by using properly formatted JSON field tags. The field tags used in the question are not recognized by the JSON codec.
type Project struct {
Identifier int64 `json:"id"`
Name string `json:"name"`
Uri string `json:"uri"`
}
Background
I'm working with JSON data that must be non-repudiable.
The API that grants me this data also has a service to verify that the data originally came from them.
As best as I can tell, they do this by requiring that the complete JSON they originally sent needs to be supplied to them inside another JSON request, with no byte changes.
Issue
I can't seem to preserve the original JSON!
Because I cannot modify the original JSON, I have carefully preserved it as a json.RawMessage when unmarshalling:
// struct I unmarshal my original data into
type SignedResult struct {
Raw json.RawMessage `json:"random"`
Signature string `json:"signature"`
...
}
// struct I marshal my data back into
type VerifiedSignatureReq {
Raw json.RawMessage `json:"random"`
Signature string `json:"signature"`
}
// ... getData is placeholder for function that gets my data
response := SignedResult{}
x, _ := json.Unmarshal(getData(), &response)
// do some post-processing with SignedResult that does not alter `Raw` or `Signature`
// trouble begins here - x.Raw started off as json.RawMessage...
y := json.Marshal(VerifiedSignatureReq{Raw: x.Raw, Signature: x.Signature}
// but now y.Raw is base64-encoded.
The problem is that []bytes / RawMessages are base64-encoded when marshaled. So I can't use this method, because it completely alters the string.
I'm unsure how to ensure this string is correctly preserved. I had assumed that the json.RawMessage specification in my struct would survive the perils of marshaling an already marshaled instance because it implements the Marshaler interface, but I appear mistaken.
Things I've Tried
My next attempt was to try:
// struct I unmarshal my original data into
type SignedResult struct {
Raw json.RawMessage `json:"random"`
Signature string `json:"signature"`
...
}
// struct I marshal my data back into
type VerifiedSignatureReq {
Raw map[string]interface{} `json:"random"`
Signature string `json:"signature"`
}
// ... getData is placeholder for function that gets my data
response := SignedResult{}
x, _ := json.Unmarshal(getData(), &response)
// do some post-processing with SignedResult that does not alter `Raw` or `Signature`
var object map[string]interface{}
json.Unmarshal(x.Raw, &object)
// now correctly generates the JSON structure.
y := json.Marshal(VerifiedSignatureReq{Raw: object, Signature: x.Signature}
// but now this is not the same JSON string as received!
The issue with this approach is that there are minor byte-wise differences in the spacing between the data. It no longer looks exactly the same when catted to a file.
I cannot use string(x.Raw) either because it escapes certain characters when marshaled with \.
You will need a custom type with its own marshaler, in place of json.RawMessage for your VerifiedSignatureReq struct to use. Example:
type VerifiedSignatureReq {
Raw RawMessage `json:"random"`
Signature string `json:"signature"`
}
type RawMessage []byte
func (m RawMessage) MarshalJSON() ([]byte, error) {
return []byte(m), nil
}
I learned from the gin doc that you can bind json to a struct like
type Login struct {
User string `form:"user" json:"user" binding:"required"`
Password string `form:"password" json:"password" binding:"required"`
}
func main() {
router := gin.Default()
// Example for binding JSON ({"user": "manu", "password": "123"})
router.POST("/loginJSON", func(c *gin.Context) {
var json Login
if c.BindJSON(&json) == nil {
if json.User == "manu" && json.Password == "123" {
c.JSON(http.StatusOK, gin.H{"status": "you are logged in"})
} else {
c.JSON(http.StatusUnauthorized, gin.H{"status": "unauthorized"})
}
}
})
}
You always have to build a struct to bind JSON.
But if there is a very complex JSON data, I only what to get part of it, to create a complex struct is a big burden. Can avoid id and parse it directly?
You don't need to create a struct for all the fields present in the JSON response, only the fields you are interested in. Any other fields present in the response will be ignored when unmarshaling.
You can also unmarshal to a generic map[string]interface{} but this is only really useful for truly dynamic data. If you know the format of the response ahead of time you will nearly always be best to create a custom struct to get type safety and avoid continual nil checks when accessing the map. Additionally, a targeted struct avoids storing unnecessary values when JSON in unmarshalled.
You can use the JSON to Go tool to help quickly create a struct definition from a JSON response. You could then easily strip out all the fields you don't need.
I'm starting to go crazy trying to get Go to decode this json request body. Here's a sample request:
curl -X POST -d "{\"username\":\"foo\", \"password\":\"bar\"}" http://localhost:3000/users
And here's my handler:
mux.HandleFunc("/users", func(rw http.ResponseWriter, req *http.Request) {
var body struct {
username string
password string
}
// buf := make([]byte, req.ContentLength)
// req.Body.Read(buf)
// fmt.Println(string(buf))
//
// The above commented out code will correctly print:
// {"username":"foo", "password":"bar"}
err := json.NewDecoder(req.Body).Decode(&body)
if err != nil {
rw.WriteHeader(http.StatusNotAcceptable)
return
}
fmt.Printf("%+v\n", body)
// prints -> {username: password:}
})
Like the comment suggests, I can verify that req.Body is indeed correct -- but for whatever reason, json.NewDecoder(req.Body).Decode(&body) never fills out the fields of body.
Any help would be greatly appreciated!
The problem is that the json decoder does not deal with private struct fields. The fields in your body structs are private.
Rewrite it like so and it will work:
var body struct {
Username string `json:"username"`
Password string `json:"password"`
}
basically the json:"username" is a way to tell the json decoder how to map the json name of the object to the struct name. In this instance, for decoding only, it is not necessary - the json decoder is smart enough to make the translation of the upper/lower case.
But if you use the object to encode json as well, you need it, or you'll have upper case field names in the resulting json.
You can use the json struct tags for a few more useful things, like omitting empty field from encoded json, or skipping fields entirely.
You can read more about the JSON struct tags in the documentation for json.Marshal: http://golang.org/pkg/encoding/json/#Marshal