How to parse wrapped json object - json

some API endpoint may return successful result or error like this:
// ok
{
"status": "ok",
"payload": {
"id": 10,
"title": "Sample"
},
"request_id": "lx-VHr4OLm"
}
// error
{
"status": "error",
"payload": {
"message": "internal error"
},
"trace_id": "lx-VHr4OLm"
}
I'm trying to find elegant way to parse with Go, something like this
.... some code
if status == "ok" {
struct := AppStruct{} // AppStruct contains 2 fields: id and title
_ := json.Unmarshall(payloadBody, &struct)
return struct
} else {
errorStruct := ErrorStruct{} // contains 1 field for message.
_ := json.Unmarshall(payloadBody, &errorStruct)
return nil, errors.New(errorStruct.Message)
}
My current code not works for success payload:
var result map[string]interface{}
jsonErr := json.Unmarshal(body, &result)
if jsonErr != nil {
return nil, jsonErr
}
if result["status"] == "error" {
errorPayload := result["payload"].(map[string]string)
return nil, errors.New(errorPayload["message"])
} else if result["status"] == "ok" {
apiResponse := AppInfo{}
jsonErr := json.Unmarshal([]byte(result["payload"].(string)), &apiResponse)
if jsonErr != nil {
return nil, jsonErr
}
return &apiResponse, nil
}
And I got runtime error on line json.Unmarshal([]byte(result["payload"].(string)), &apiResponse)
http: panic serving [::1]:51091: interface conversion: interface {} is
map[string]interface {}, not string
Surely, I may have 2 structs: for success response and for error one but I assume it's too complicated way to solve my problem.
How to parse this JSON in elegant way?

I'm really not sure what the problem is. The standard encoding/json doesn't require the struct to match all fields in the JSON data. It's quite easy to handle this with a single, simple type:
type Payload struct {
ID int `json:"id"`
Title string `json:"title"`
Message string `json:"message"`
}
type Response struct {
Status string `json:"status"`
ID string `json:"request_id"`
TraceID string `json:"trace_id"`
Payload Payload `json:"payload"`
}
Then just unmarshal the response in the Response struct:
var resp Response
if err := json.Unmarshal(body, &resp); err != nil {
return err
}
Then you can simply check the Status field, and work out what to do next. For example:
if resp.Status == "error" {
return fmt.Errorf("invalid response: %s - %s", resp.TraceID, resp.Payload.Message)
}
// handle resp.Payload.ID and resp.Payload.Title fields
return nil
You could move those checks for the status field to receiver functions on the response object, depending on the complexity and your specific needs.
Perhaps it's worth using pointer fields for those fields that aren't set in the normal response, and tag them with the omitempty option:
type Payload struct {
ID int `json:"id"`
Title string `json:"title"`
Message *string `json:"message,omitempty"`
}
type Response struct {
Status string `json:"status"`
ID string `json:"request_id"`
TraceID *string `json:"trace_id,omitempty"`
Payload Payload `json:"payload"`
}
With types like this, no longer have to rely on hard-coded string constants to check for errors. Instead, you can easily implement a more generic check like so:
func (r Response) IsError() bool {
return (r.TraceID == nil) // will be set in case of an error response
}
Update
As you pointed out in the comments, the response body is actually substantially larger than the 2 fields in the example. Of course, copy-pasting the struct definitions, or writing mapping functions to map the Payload onto the type you already have is a bit pointless.
The answer here is: composition.
type Payload struct {
AppStruct // embedded the AppStruct type
Message *string `json:"message"`
}
The Response type stays as it is. If the response is successful, you can get the AppStruct directly from the response like so:
appStruct := resp.Payload.AppStruct
This works because the type is embedded. Note that there aren't any json tags there. The embedded struct, at least as far as the unmarshalling is concerned, is a part of the Payload struct. Therefore, all the exported fields in that type will be unmarshalled directly into the struct.

Thanks https://stackoverflow.com/users/965900/mkopriva for idea to use json.RawMessage
My final solution:
func parsePayload(response []byte, successPayload interface{}) error {
var result map[string]json.RawMessage
jsonErr := json.Unmarshal(response, &result)
if jsonErr != nil {
return jsonErr
}
var status string
jsonErr = json.Unmarshal(result["status"], &status)
if jsonErr != nil {
return jsonErr
}
if status == "ok" {
jsonErr = json.Unmarshal(result["payload"], &successPayload)
if jsonErr != nil {
return jsonErr
}
return nil
} else if status == "error" {
errorPayload := ErrorPayload{}
jsonErr = json.Unmarshal(result["payload"], &errorPayload)
if jsonErr != nil {
return jsonErr
}
return errors.New(errorPayload.Message)
}
log.Printf("Unknown http result status: %s", status)
return errors.New("internal error")
}
type ErrorPayload struct {
Message string `json:"message"`
}
//usage
type AppInfo struct {
Id int `json:"app_id"`
Title string `json:"app_title"`
}
body := ... // read body
appInfo := AppInfo{}
parseErr := parsePayload(body, &appInfo)
if parseErr != nil {
return nil, parseErr
}
log.Printf("Parsed app %v", appInfo)
return &appInfo, nil

Related

How to unmarshall different json format

I read lots of similar questions but not able to find the right solution yet. Hope someone can give me better idea.
My error response could be one of these format:
var errProxy = `{"errors":{"id": "1", "message": "failed to resolve the ip", "status": "failed"}}`
var errServer = `{"errors": "failed to ping the dns server."}`
I am trying to solve this by using two structs and trying to unmarshall one after another until err is nil and return:
// ErrorServer and ErrorProxy is my two struct to represent my above two error response.
type ErrorServer struct {
Error string `json:"errors"`
}
type ErrorResponse struct {
ID string `json:"id"`
Message string `json:"message"`
Status string `json:"status"`
}
type ErrorProxy struct {
ErrorResponse
}
This is the function I used to parse it:
func parseErrorResponse(body io.ReadCloser) (string, error) {
var errServer ErrorServer
var errProxy ErrorProxy
err := json.NewDecoder(body).Decode(&errServer)
if err != nil {
err = json.NewDecoder(body).Decode(&errProxy)
if err != nil {
return "", err
}
return errProxy.Error, nil
}
return errServer.Errors, nil
}
I think a custom unmarshal function is good for this case
type ErrorServer struct {
Error string
}
type ErrorResponse struct {
ID string
Message string
Status string
}
type ErrorProxy struct {
Err ErrorResponse
}
func parseErrorResponse(body io.Reader) (interface{}, error) {
data := make(map[string]interface{})
if err := json.NewDecoder(body).Decode(&data); err != nil {
return nil, err
}
// Check type of errors field
switch errors := data["errors"].(type) {
case string:
// Unmarshal for ErrorServer
errServer := &ErrorServer{
Error: errors,
}
return errServer, nil
case map[string]interface{}:
// Unmarshal for ErrorProxy
errProxy := &ErrorProxy{
Err: ErrorResponse{
ID: errors["id"].(string),
Message: errors["message"].(string),
Status: errors["status"].(string),
},
}
return errProxy, nil
default:
return nil, fmt.Errorf(`failed to parse "errors" field`)
}
}
func main() {
body := bytes.NewReader([]byte(`{"errors": "failed to ping the dns server."}`))
//body := bytes.NewReader([]byte(`{"errors":{"id": "1", "message": "failed to resolve the ip", "status": "failed"}}`))
parsedErr, _ := parseErrorResponse(body)
switch err := parsedErr.(type) {
case *ErrorServer:
fmt.Printf("err server: %+v \n", err)
case *ErrorProxy:
fmt.Printf("err response: %+v \n", err)
}
}
Another way to do that is using type assertion. You can unmarshal that json string into map[string]interface{} and check whether that value interface{} is a string or map[string]interface{}. Depends on its type, you know which kind of error it is and construct a struct from that.
Funny enough, I just gave the answer to this problem here: Use a customized UnmarshalJSON function. How do you modify this struct in Golang to accept two different results?
Applied to your situation:
package main
import (
"encoding/json"
"fmt"
"log"
)
type Answer struct {
Errors Error `json:"errors"`
}
type ErrorResponse struct {
ID string `json:"id"`
Message string `json:"message"`
Status string `json:"status"`
}
type Error struct {
Error string
ErrorResponse ErrorResponse
}
func (s *Error) UnmarshalJSON(b []byte) error {
if len(b) == 0 {
// no data, nothing to do
return nil
}
if b[0] == '{' {
// is object
return json.Unmarshal(b, &s.ErrorResponse)
}
return json.Unmarshal(b, &s.Error)
}
func main() {
var errProxy = []byte(`{"errors":{"id": "1", "message": "failed to resolve the ip", "status": "failed"}}`)
var errServer = []byte(`{"errors": "failed to ping the dns server."}`)
var answ Answer
if err := json.Unmarshal(errProxy, &answ); err != nil {
log.Fatal(err)
}
fmt.Println(answ)
answ = Answer{}
if err := json.Unmarshal(errServer, &answ); err != nil {
log.Fatal(err)
}
fmt.Println(answ)
}
Go PlayGround
Key in the above example is the type that contains both variations. We can also simplify that as both errors could be contained within the ErrorResponse type:
type Answer struct {
Errors ErrorResponse `json:"errors"`
}
type ErrorResponse struct {
ID string
Message string
Status string
}
func (s *ErrorResponse) UnmarshalJSON(b []byte) error {
if len(b) == 0 {
// no data, nothing to do
return nil
}
if b[0] == '{' {
// is object
var tmp struct {
ID string `json:"id"`
Message string `json:"message"`
Status string `json:"status"`
}
if err := json.Unmarshal(b, &tmp); err != nil {
return err
}
s.ID = tmp.ID
s.Message = tmp.Message
s.Status = tmp.Status
return nil
}
return json.Unmarshal(b, &s.Message)
}
The Error struct is not needed here.

JSON Unmarshal Expected Response and Error Message

I'm making a JSON request to a 3rd party API. If my auth token has expired, I receive an {"error": "message"}. If everything checks out, I receive a valid response.
Right now, I call json.Unmarshal twice to parse the response: once to check for the error and once to parse the real data.
Is there any way to avoid calling Unmarshal twice?
type Unknown map[string]interface{}
type Response struct {
Status string `json:"status"`
Strategy string `json:"strategy"`
Items []Item `json:"items"`
}
unknown := Unknown{}
json.Unmarshal(jsonData, &unknown)
if val, ok := unknown["error"]; ok {
fmt.Println(val)
return
}
response := Response{}
err := json.Unmarshal(jsonData, &response)
if err != nil {
fmt.Println("There was an error")
fmt.Println(err)
return
}
You can use embedding to decode everything at once:
type Response struct {
Status string `json:"status"`
Strategy string `json:"strategy"`
Items []Item `json:"items"`
}
var dest struct {
Error string `json:"error"`
Response // embedded Response
}
if err := json.Unmarshal(jsonData, &dest); err != nil {
fmt.Println(err)
return
} else if len(dest.Error) > 0 {
fmt.Println(dest.Error)
return
}
response := dest.Response
// ...
See an example on playground: https://play.golang.com/p/eAhFt99n07k

Unmarshal JSON dynamically into structs containing only top level fields based on a type field

Say I have the following event structs that I cannot modify (not even the tags):
type openAudioStream struct {
Type string `json:"type"`
ID string `json:"id"`
Address string `json:"address"`
Channels int `json:"channels"`
}
type openVideoStream struct {
Type string `json:"type"`
ID string `json:"id"`
Address string `json:"address"`
AspectRatio string `json:"aspectRatio"`
}
And I have an API endpoint which emits JSON strings (that I also cannot modify) containing events that map to either one of those two structs and I cannot tell in advance which one it is, so I need to, somehow, extract the type field to figure out which struct to instantiate and then unmarshal the rest of the JSON into the instantiated event object.
The first approach that came to my mind was to call json.Unmarshal twice like so:
func jsonUnmarshal(payload []byte) (Event, error) {
eventType := struct {
Type string
}{}
err := json.Unmarshal(payload, &eventType)
if err != nil {
return nil, fmt.Errorf("failed to unmarshal JSON: %v", err)
}
var event Event
switch eventType.Type {
case "audio":
event = &openAudioStream{}
err = json.Unmarshal(payload, event)
case "video":
event = &openVideoStream{}
err = json.Unmarshal(payload, event)
default:
err = fmt.Errorf("unrecognised event type: %s", eventType.Type)
}
if err != nil {
return nil, fmt.Errorf("failed to unmarshal JSON: %v", err)
}
return event, nil
}
While this works well, it's inefficient to traverse the JSON string twice, so I thought that, maybe, I can create a union type and use that to unmarshal the JSON like so:
func notWorking(payload []byte) (Event, error) {
eventUnion := struct {
Type string `json:"type"`
openAudioStream
openVideoStream
}{}
err := json.Unmarshal(payload, &eventUnion)
if err != nil {
return nil, fmt.Errorf("failed to unmarshal JSON: %v", err)
}
var event Event
switch eventUnion.Type {
case "audio":
event = &extractor.openAudioStream
case "video":
event = &extractor.openVideoStream
default:
return nil, fmt.Errorf("unrecognised event type: %s", eventUnion.Type)
}
return event, nil
}
Besides being an ugly hack, this approach doesn't work if the embedded structs contain conflicting fields. The json unmarshaler simply ignores them without producing any error.
Finally, I recalled that mapstructure might help and, indeed, I am able to use it like so:
func mapstructureDecode(payload []byte) (Event, error) {
var unmarshaledPayload map[string]interface{}
err := json.Unmarshal(payload, &unmarshaledPayload)
if err != nil {
return nil, fmt.Errorf("failed to unmarshal JSON: %v", err)
}
var ok bool
var val interface{}
var eventType string
if val, ok = unmarshaledPayload["type"]; ok {
eventType, ok = val.(string)
}
if !ok {
return nil, fmt.Errorf("failed to determine event type: %v", err)
}
var event Event
switch eventType {
case "audio":
event = &openAudioStream{}
err = mapstructure.Decode(unmarshaledPayload, &event)
case "video":
event = &openVideoStream{}
err = mapstructure.Decode(unmarshaledPayload, event)
default:
err = fmt.Errorf("unrecognised event type: %s", eventType)
}
if err != nil {
return nil, fmt.Errorf("failed to unmarshal JSON: %v", err)
}
return event, nil
}
However, using this library seems a bit overkill for this task and it requires adding mapstructure tags to the struct fields if the input JSON doesn't follow the standard naming conventions, which is an issue if I have, for example aspect_ratio instead of aspectRatio.
The full code for the above experiments can be found here: https://play.golang.org/p/qTGoV6i8m5P
I'm curious if there is any other way to tackle this problem using existing libraries. I was thinking that maybe some creative use of json.RawMessage and a custom UnmarshalJSON method might help, but this doesn't seem to be of use if I have only top level fields in the event structs.
You can unmarshal once in combined struct and then create new struct depends on type.
func jsonUnmarshal(payload []byte) (Event, error) {
data := struct {
Type string
ID string
Address string
Channels int
AspectRatio string
}{}
// Unmarshal in combined stuct
err := json.Unmarshal(payload, &data)
if err != nil {
return nil, fmt.Errorf("failed to unmarshal JSON: %v", err)
}
var event Event
switch data.Type {
case "audio":
// Creating new stuct
event = &openAudioStream{Type: data.Type, ID: data.ID, Address: data.Address, Channels: data.Channels}
case "video":
event = &openVideoStream{Type: data.Type, ID: data.ID, Address: data.Address, AspectRatio: data.AspectRatio}
default:
err = fmt.Errorf("unrecognised event type: %s", data.Type)
}
if err != nil {
return nil, fmt.Errorf("failed to unmarshal JSON: %v", err)
}
return event, nil
}

Unmarshall PubSub Request Data []bytes with Go

I have an end-point that receives data from a Google PubSub request. As per this repo, the object is as so:
type pushRequest struct {
Message struct {
Attributes map[string]string
Data []byte
ID string `json:"message_id"`
}
Subscription string
}
The Data field is consistently formatted as so:
type Data struct {
Key string `json:"key"`
Body string `json:"body"`
Meta map[string]interface{} `json:"meta"`
}
I can obviously unmarshal the JSON request with something like this:
f := &pushRequest{}
json.Unmarshal(msg, &f)
That leaves with the the []bytes field. Which I can do something like this to convert to a string, as per the docs
messages = append(messages, string(f.Message.Data))
Which doesn't help, since I need it as a struct.
I can Unmarshal the array again:
var m Data
json.Unmarshal(f.Message.Data, &m)
Have tried changing the field type in the pushRequest struct to Data without success. Blank...
Is there a way I can unpack things in a single pass? Doing it twice seems ridiculous.
If it's obvious, I just can't see it!
decoder := json.NewDecoder(r.Body)
psmsg := &PushRequest{}
decoderErr := decoder.Decode(&psmsg)
if decoderErr != nil {
// Error...
return
}
data := Data{}
unmarshalErr := json.Unmarshal([]byte(string(psmsg.Message.Data)), &data)
if unmarshalErr != nil {
// Error...
return
}
Here is a snippet from my cloud function, which serves as a pub/sub push endpoint. The key is that you first have to decode the body using the PushRequest struct. Next, you can transform the message data into a struct. According to the documentation, the Data field within Message is a base-64 encoded string, therefore you have to decode it first.
type PushRequest struct {
Message pubsub.PubsubMessage `json:"message"`
Subscription string `json:"subscription"`
}
type Example struct {
ID string `json:"id" firestore:"id"`
}
func HTTPEndpoint(w http.ResponseWriter, r *http.Request) {
var pr common.PushRequest
if err := json.NewDecoder(r.Body).Decode(&pr); err != nil {
log.Fatalf("Could not decode body: %v", err)
return
}
data, err := base64.StdEncoding.DecodeString(pr.Message.Data)
if err != nil {
log.Fatalf("Base64: %v", err)
return
}
var example Example
if err := json.Unmarshal(data, &example); err != nil {
log.Fatalf("Json: %v", err)
return
}
// Do something useful with the struct
}

Golang check if interface type is nil

When unmarshalling a string of json, using Golang's json.Unmarshal() function, I always unmarshal the string into a map[string]interface{} - I'm not sure weather there is a more optimal way of decoding json.
To get to the point:
Sometimes the json's unmarshalled type is nil, not string (or int etc.). This always throws a panic, saying:
interface conversion: interface is nil, not int
How can I avoid this panic or check if the interface's type is nil or not?
Example:
Here is an example of my problem in action: https://play.golang.org/p/0ATzXBbdoS
Check if the key exist instead of letting it panic.
func keyExists(decoded map[string]interface{}, key string) {
val, ok := decoded[key]
return ok && val != nil
}
func main() {
jsonText := `{
"name": "Jimmy",
"age": 23
}`
var decoded map[string]interface{}
if err := json.Unmarshal([]byte(jsonText), &decoded); err != nil {
fmt.Println(err)
os.Exit(0)
}
if keyExists(decoded, "name") {
fmt.Println(decoded["name"].(string))
}
if keyExists(decoded, "age") {
fmt.Println(decoded["age"].(float64))
}
if keyExists(decoded, "gender") {
fmt.Println(decoded["gender"].(int))
}
}
Also, this is far from being optimal if you know what your json will look like. As specified in the documentation, you can unmarshal it directly into a struct:
type Human struct {
Name string
Age int
Gender int
}
func main() {
jsonText := `{
"name": "Jimmy",
"age": 23
}`
decoded := Human{}
if err := json.Unmarshal([]byte(jsonText), &decoded); err != nil {
fmt.Println(err)
os.Exit(0)
}
fmt.Println(decoded.Name)
fmt.Println(decoded.Age)
fmt.Println(decoded.Gender)
}