Unmarshall PubSub Request Data []bytes with Go - json

I have an end-point that receives data from a Google PubSub request. As per this repo, the object is as so:
type pushRequest struct {
Message struct {
Attributes map[string]string
Data []byte
ID string `json:"message_id"`
}
Subscription string
}
The Data field is consistently formatted as so:
type Data struct {
Key string `json:"key"`
Body string `json:"body"`
Meta map[string]interface{} `json:"meta"`
}
I can obviously unmarshal the JSON request with something like this:
f := &pushRequest{}
json.Unmarshal(msg, &f)
That leaves with the the []bytes field. Which I can do something like this to convert to a string, as per the docs
messages = append(messages, string(f.Message.Data))
Which doesn't help, since I need it as a struct.
I can Unmarshal the array again:
var m Data
json.Unmarshal(f.Message.Data, &m)
Have tried changing the field type in the pushRequest struct to Data without success. Blank...
Is there a way I can unpack things in a single pass? Doing it twice seems ridiculous.
If it's obvious, I just can't see it!

decoder := json.NewDecoder(r.Body)
psmsg := &PushRequest{}
decoderErr := decoder.Decode(&psmsg)
if decoderErr != nil {
// Error...
return
}
data := Data{}
unmarshalErr := json.Unmarshal([]byte(string(psmsg.Message.Data)), &data)
if unmarshalErr != nil {
// Error...
return
}

Here is a snippet from my cloud function, which serves as a pub/sub push endpoint. The key is that you first have to decode the body using the PushRequest struct. Next, you can transform the message data into a struct. According to the documentation, the Data field within Message is a base-64 encoded string, therefore you have to decode it first.
type PushRequest struct {
Message pubsub.PubsubMessage `json:"message"`
Subscription string `json:"subscription"`
}
type Example struct {
ID string `json:"id" firestore:"id"`
}
func HTTPEndpoint(w http.ResponseWriter, r *http.Request) {
var pr common.PushRequest
if err := json.NewDecoder(r.Body).Decode(&pr); err != nil {
log.Fatalf("Could not decode body: %v", err)
return
}
data, err := base64.StdEncoding.DecodeString(pr.Message.Data)
if err != nil {
log.Fatalf("Base64: %v", err)
return
}
var example Example
if err := json.Unmarshal(data, &example); err != nil {
log.Fatalf("Json: %v", err)
return
}
// Do something useful with the struct
}

Related

Error while converting json to struct from Go

func MakeMap(w http.ResponseWriter, r *http.Request) {
// userInfo := context.Get(r, "userInfo").(model.User)
type _getData struct {
Title string `json:"title"`
Tag []string `json:"tag"`
}
var getData _getData
err := json.NewDecoder(r.Body).Decode(&getData)
if err != nil {
panic(err.Error())
}
fmt.Print(getData)
}
When I run the above code, I get the following error
2021/08/24 13:56:54 http: panic serving 127.0.0.1:50619: runtime error: invalid memory address or nil pointer dereference
goroutine 23 [running]:
net/http.(*conn).serve.func1(0x140001e9180)
/usr/local/go/src/net/http/server.go:1824 +0x108
panic(0x10505b860, 0x10522f240)
/usr/local/go/src/runtime/panic.go:971 +0x3f4
traveling/controller/mapController.MakeMap(0x1050b5630, 0x140001f40e0, 0x1400018aa00)
/Users/choeyunseog/traveling/traveling/controller/mapController/mapController.go:20 +0x3c
I've just started studying, I'm not sure why I'm having this problem, please help
err := json.NewDecoder(r.Body).Decode(&getData)
I get the following error when i change code line 20 like above
2021/08/24 14:16:44 http: panic serving 127.0.0.1:51396: invalid character '-' in numeric literal
goroutine 23 [running]:
net/http.(*conn).serve.func1(0x140001e9360)
/usr/local/go/src/net/http/server.go:1824 +0x108
panic(0x100d85d00, 0x14000206070)
/usr/local/go/src/runtime/panic.go:971 +0x3f4
traveling/controller/mapController.MakeMap(0x100df1630, 0x140001f40e0, 0x1400018aa00)
/Users/choeyunseog/traveling/traveling/controller/mapController/mapController.go:24 +0x194
net/http.HandlerFunc.ServeHTTP(0x100de75d8, 0x100df1630, 0x140001f40e0, 0x1400018aa00)
/usr/local/go/src/net/http/server.go:2069 +0x40
To get the multipart form data from a POST/PUT/PATCH request's body you can use the ParseMultipartForm method to parse the body and then access the data through the PostForm field. Or you can use FormValue to get just the first value associated with the form's field.
maxMemory := 32<<20
if err := r.ParseMultipartForm(maxMemory); err != nil {
panic(err)
}
fmt.Println(_getData{
Title: r.FormValue("title"), // FormValue returns string
Tag: r.PostForm["tag[]"], // PostForm is a map of []string
})
You can use to parse form data into json like annotated struct using package github.com/senpathi/paramex. Struct fields must be annotated with param keyword and tag name is key of the form data.
your struct should be as below.
type _getData struct {
Title string `param:"title"`
Tag []string `param:"tag[]"`
}
This is the updated MakeMap handler function for your postman request mentioned in the question
func MakeMap(w http.ResponseWriter, r *http.Request) {
// userInfo := context.Get(r, "userInfo").(model.User)
type _getData struct {
Title string `param:"title"`
Tag []string `param:"tag[]"`
}
// this needed because u send data from Postman as multipart/form-data
maxMemory := 32<<20
if err := r.ParseMultipartForm(int64(maxMemory)); err != nil {
panic(err)
}
var getData _getData
extractor := paramex.NewParamExtractor()
err := extractor.ExtractForms(&getData, r)
if err != nil {
panic(err.Error())
}
fmt.Print(getData)
//Output: {defaultMap [travelling travelling2]}
}

Parse JSON having sibling dynamic keys alongside with static in Go

I need to parse this json
{
"version": "1.1.29-snapshot",
"linux-amd64": {
"url": "https://origin/path",
"size": 7794688,
"sha256": "14b3c3ad05e3a98d30ee7e774646aec7ffa8825a1f6f4d9c01e08bf2d8a08646"
},
"windows-amd64": {
"url": "https://origin/path",
"size": 8102400,
"sha256": "01b8b927388f774bdda4b5394e381beb592d8ef0ceed69324d1d42f6605ab56d"
}
}
Keys like linux-amd64 are dynamic and theirs amount is arbitrary. I tried something like that to describe it and unmarshal. Obviously it doesn't work. Items is always empty.
type FileInfo struct {
Url string `json:"url"`
Size int64 `json:"size"`
Sha256 string `json:"sha256"`
}
type UpdateInfo struct {
Version string `json:"version"`
Items map[string]FileInfo
}
It's similar to this use case, but has no parent key items. I suppose I can use 3rd party library or map[string]interface{} approach, but I'm interested in knowing how to achieve this with explicitly declared types.
The rest of the parsing code is:
func parseUpdateJson(jsonStr []byte) (UpdateInfo, error) {
var allInfo = UpdateInfo{Items: make(map[string]FileInfo)}
var err = json.Unmarshal(jsonStr, &allInfo)
return allInfo, err
}
Look at the link I attached and you will realize that is not that simple as you think. Also I pointed that I interested in typed approach. Ok, how to declare this map[string]FileInfo to get parsed?
You can create a json.Unmarshaller to decode the json into a map, then apply those values to your struct: https://play.golang.org/p/j1JXMpc4Q9u
type FileInfo struct {
Url string `json:"url"`
Size int64 `json:"size"`
Sha256 string `json:"sha256"`
}
type UpdateInfo struct {
Version string `json:"version"`
Items map[string]FileInfo
}
func (i *UpdateInfo) UnmarshalJSON(d []byte) error {
tmp := map[string]json.RawMessage{}
err := json.Unmarshal(d, &tmp)
if err != nil {
return err
}
err = json.Unmarshal(tmp["version"], &i.Version)
if err != nil {
return err
}
delete(tmp, "version")
i.Items = map[string]FileInfo{}
for k, v := range tmp {
var item FileInfo
err := json.Unmarshal(v, &item)
if err != nil {
return err
}
i.Items[k] = item
}
return nil
}
This answer is adapted from this recipe in my YouTube video on advanced JSON handling in Go.
func (u *UpdateInfo) UnmarshalJSON(d []byte) error {
var x struct {
UpdateInfo
UnmarshalJSON struct{}
}
if err := json.Unmarshal(d, &x); err != nil {
return err
}
var y map[string]json.RawMessage{}
if err := json.Unsmarshal(d, &y); err != nil {
return err
}
delete(y, "version"_ // We don't need this in the map
*u = x.UpdateInfo
u.Items = make(map[string]FileInfo, len(y))
for k, v := range y {
var info FileInfo
if err := json.Unmarshal(v, &info); err != nil {
return err
}
u.Items[k] = info
}
return nil
}
It:
Unmarshals the JSON into the struct directly, to get the struct fields.
It re-unmarshals into a map of map[string]json.RawMessage to get the arbitrary keys. This is necessary since the value of version is not of type FileInfo, and trying to unmarshal directly into map[string]FileInfo will thus error.
It deletes the keys we know we already got in the struct fields.
It then iterates through the map of string to json.RawMessage, and finally unmarshals each value into the FileInfo type, and stores it in the final object.
If you really don't want to unmarshal multiple times, your next best option is to iterate over the JSON tokens in your input by using the json.Decoder type. I've done this in a couple of performance-sensitive bits of code, but it makes your code INCREDIBLY hard to read, and in almost all cases is not worth the effort.

How to decode JSON in Go which returns multiple elements as array of type and individual elements as type

I am working with an API that sends JSON data. The problem is that an array of a single element shows up as a single value. For example, consider the following JSON:
{ "names": ["Alice","Bob"] }
The API sends this as an array. But when the names field has a single element, the API sends this:
{ "names": "Alice" }
This is how I would normally decode this of response in Go:
type Response struct {
Names []string `json:"names"`
}
// later
d := &Response{}
_ = json.NewDecoder(resp).Decode(d) // resp would be a http.Response.Body containing the problematic JSON
Go decodes the first JSON correctly. However, after decoding the second JSON, the object contains an empty array.
I don't have any control over the API, so I have to get around this problem. How could I decode this JSON correctly in Go, so that the Names slice contains a single element? Thank you for your help.
You could implement the json.Unmarshaler interface and have it check the 0th raw byte for [ or " to find out whether it's an array or a string respectivelly:
type StringSlice []string
func (ss *StringSlice) UnmarshalJSON(data []byte) error {
if data[0] == '[' {
return json.Unmarshal(data, (*[]string)(ss))
} else if data[0] == '"' {
var s string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
*ss = append(*ss, s)
}
return nil
}
https://play.golang.com/p/2GEJsS2YOLJ
You'll have to decode this into an interface{} and then use a type assertion to check whether the underlying type is a slice or just a string.
type Response struct {
Names interface{} `json:"names"`
}
Then after decoding into d, you'd do something like:
slice, ok := d.Names.([]interface{})
if ok {
// it was a slice. use it.
} else {
// it wasn't a slice - so expect it to be a string
// and use that, etc.
}
Use json.RawMessage as type. RawMessage simply delays the decoding of part of a message, so we can do it ourselves later.
type Response struct {
NamesRaw json.RawMessage `json:"names"`
Names []string
}
First, decode the response and then using json.Unmarshal decode json.RawMessage
x := &Response{}
_ = json.NewDecoder(resp).Decode(x);
x.Names = DecodeName(x.NamesRaw)
DecodeName used for decoding NameRaw data
func DecodeName(nameRaw json.RawMessage) (data []string) {
var s string
if err := json.Unmarshal(nameRaw, &s); err == nil {
v := []string{s}
return v
}
var sn []string
if err := json.Unmarshal(nameRaw, &sn); err == nil {
return sn
}
return
}

How can I parse Firestore get() snapshot JSON content in Go?

Firestore returns map[string]interface{} while getting data. How can I render "details" values?
user:{
fname:"john",
lname:"con",
detail:{
address:"Delhi, India",
mob:"0000000009"
}
}
sn := snap.Data()
var bt []byte
for _, val := range sn {
for _, v := range val {
log.Println("value ", v)
}
}
Use json.Unmarshal to convert your JSON content to a map.
jsonString := `{"user":true,"lname":"con","detail":{"address":"Delhi, India","mob":"0000000009"}}`
aMap := make(map[string]interface{})
err := json.Unmarshal([]byte(jsonString), &aMap)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v\n", aMap)
fmt.Printf("Address := %s\n",aMap["detail"].(map[string]interface{})["address"])
https://play.golang.org/p/3133C_sKDf4
First things first. Your JSON appears to be invalid. It looks like a few
quotes are missing.
Try to validate your sample data here and you'll see what's wrong with it.
Alternatively to decode your JSON to a map, you can also unmarshal it into a struct as long as you know its structure in advance.
type User struct {
FirstName string `json:"fname"`
LastName string `json:"lname"`
Detail Detail `json:"detail"`
}
type Detail struct {
Address string `json:"address"`
Mobile string `json:"mob"`
}
if err := json.NewDecoder(strings.NewReader(out)).Decode(&u); err != nil {
log.Fatal(err)
}
Full working example

How to unmarshal json

I am using http.Get in go to a url which results in the following
{"name":"cassandra","tags":["2.2.6","latest"]} that means it behaves like map[string]string for the name field but in the tags it behaves like map[string][]string so how can I unmarshal this in Go?
I tried using map[string][]string but it did not work
map_image_tags := make(map[string][]string)
res2, err := http.Get(fmt.Sprintf("%s/v2/%s/tags/lists", sconf.RegistryConf.url, Images[i]))
if err != nil {
w.WriteHeader(500)
log.Errorf("could not get tags: %s", err)
return
}
log.Debugf("OK")
js2, err := ioutil.ReadAll(res2.Body)
if err != nil {
w.WriteHeader(500)
log.Errorf("could not read body: %s", err)
return
}
log.Debugf("OK")
err = json.Unmarshal(js2, map_image_tags)
if err != nil {
w.WriteHeader(500)
log.Errorf("could not unmarshal json: %s", err)
return
}
I am getting this log error: could not unmarshal json: invalid character 'p' after top-level value
To read a json value like {"name":"cassandra", "tags":["2.2.6","latest"], you can use a struct defined as:
type mapImageTags struct {
Name string `json:"name"`
Tags []string `json:"tags"` // tags is a slice (array) of strings
}
To unmarshal JSON data,
m := mapImageTags{}
err = json.Unmarshal(js2, &m)
A simple map[string]string wont help in this case.
If the structure of json data is dynamic you can unmarshal tags into map[string]interface{}:
var encodedTags map[string]interface{}
result := json.Unmarshal([]byte(image_tags), &encodedTags)
Then you can use type assertion to unmarshal the content of tags:
var tags []interface{}
result = json.Unmarshal([]byte(encodedTags["tags"].(string)), &tags)
And here is a full working example on Go Playground.
Try map[string]interface{}, note that this method forces any numbers to float and in general not recommended when your json is complex. abhink's answer is the recommended way.