Decoding variable-schema JSON in Go - json

I'm asking this about Go's encoding/json, but I guess it also applies to any other JSON libraries that map JSON blobs to objects in whatever language.
Here's an example. If you want to a shorten a URL using the goo.gl URL shortener API, you get back either a successful response:
{
"kind": "urlshortener#url",
"id": "http://goo.gl/fbsS",
"longUrl": "http://www.google.com/"
}
Or an error response:
{
"error": {
"errors": [
{
"domain": "global",
"reason": "required",
"message": "Required",
"locationType": "parameter",
"location": "resource.longUrl"
}
],
"code": 400,
"message": "Required"
}
}
Is there an idiomatic way of dealing with this -- a response that could adhere to two completely different schemas?
Normally I deal with JSON using maps/lists; I know that's possible in Go. I could unmarshal to a map[string]interface{} and then check if the map has "error" as a key. But then I'd have to decode again into a proper struct, I think. (Am I wrong?)
I'm doing something like this. I have one type for each kind of response:
type successResponse struct {
Kind string
Id string
LongUrl string
}
type errorResponse struct {
Error struct {
Errors []struct {
Domain string
Reason string
Message string
LocationType string
Location string
}
Code int
Message string
}
}
And decoding looks like this:
s := new(successResponse)
err := json.Unmarshal(blob, s)
if err == nil {
// handle success
} else {
e := new(errorResponse)
err = json.Unmarshal(blob, e)
if err == nil {
// handle error response
} else {
// handle actual error
}
}
But that seems kind of ugly. How should I approach this?

Since the fields in your json responses are distinct from each other you can just create one struct with the union of all the fields. The json decoder will ignore fields that are not present in the json string and you can test the existence of the fields to know which type of response you are getting back.

I was confused about this, too, and thought I had to decode it again. You don't, though. You just have to typecast the interface{} data into the appropriate structure.
For example if the json package has put the value into a generic interface{}, you can typecast it into ErrorType with error := val.(ErrorType).
You can use foo.(type) in a switch statement to "do the right thing", if you are parsing based on what type the value is.
I've only been learning Go this week so it's not the prettiest code, but there are some examples in the geodns JSON configuration parsing.

Have you tried Go-SimpleJSON? I think this might solve your issue.

type Response struct {
Kind string
Id string
LongUrl string
Error struct {
Errors []struct {
Domain string
Reason string
Message string
LocationType string
Location string
}
Code int
Message string
}
}
s := Response{}
if err := json.Unmarshal(blob, &s); err == nil {
if s.Error == nil {
// success
} else {
// error
}
} else {
// something went wrong
}

Related

Wrap API Json response in Go

I am sorry if this is a silly question because i am very new in Go.
I am calling couple of apis on the base of business logic, different types of response coming like json array, nest json and single json object.
i need to wrap a api response that called according to business logic in a common format like:
{
"data":"api response here",
"statusCode":200
}
i tried some but its not expect output
type Model[T any] struct {
Data T
StatusCode int
}
model := Model[string]{Data: apiResponse, StatusCode: 200}
out, err := json.Marshal(model)
out put of this code is
{
"Data": "[{\"name\":\"Harry Potter\",\"city\":\"London\"},{\"name\":\"Don Quixote\",\"city\":\"Madrid\"},{\"name\":\"Joan of Arc\",\"city\":\"Paris\"},{\"name\":\"Rosa Park\",\"city\":\"Alabama\"}]",
"StatusCode": 200
}
this i made these changes
var result []map[string]interface{}
json.Unmarshal([]byte(body), &result)
out, err := json.Marshal(result)
output was as expected, above api response was in proper json when use []map[string]interface
problem is, its only for those api that return array of json. those apis returning single json object then to make it work i need to do this
map[string]interface`
means remove the array map.
i need to make it generic so that any kind of json response map into it.
Use type of field Data as an interface{}
type APIResponse struct {
Data interface{} `json:"data"`
StatusCode int `json:"statusCode"`
}
And then you can assign any API Response type to the Data field and marshal it.
func main() {
r := []Person{
{
Name: "Harry Porter",
City: "London",
},
{
Name: "Don Quixote",
City: "Madrid",
},
}
res := APIResponse{
Data: r,
StatusCode: 200,
}
resByt, err := json.Marshal(res)
if err != nil {
panic(err)
}
fmt.Println(string(resByt))
}
Output
{"data":[{"name":"Harry Porter","city":"London"},{"name":"Don Quixote","city":"Madrid"}],"statusCode":200}
Run the full code here in Playground.
You can simply do:
result:=map[string]interface{} {
"data": apiResponse,
"statusCode": 200,
}
out, err:=json.Marshal(result)

How to handle missing fields in a JSON response dynamically in Go

I'm working on a Go wrapper for an API and I noticed that two of the JSON fields stay empty when they don't have any data.
Basically the API returns a set of information on a given url, and if it was visited at least once, everything is okay and I get a full json that I then Unmarshal into a struct:
{
"stats":{
"status":1,
"date":"09.07.2019",
"title":"Test",
"devices":{
"dev":[
{
"tag":"Desktop"
}
],
"sys":[
{
"tag":"GNU/Linux "
},
{
"tag":"Windows 10"
}
],
"bro":[
{
"tag":"Firefox 67.0"
},
{
"tag":"Chrome 62.0"
}
]
},
"refs":[
{
"link":"www.google.com"
}
]
}
}
This is the struct I'm using:
type Stats struct {
Stats struct {
Status int `json:"status"`
Date string `json:"date"`
Title string `json:"title"`
Devices struct {
Dev []struct {
Tag string `json:"tag"`
} `json:"dev"`
Sys []struct {
Tag string `json:"tag"`
} `json:"sys"`
Bro []struct {
Tag string `json:"tag"`
} `json:"bro"`
} `json:"devices"`
Refs []struct {
Link string `json:"link"`
} `json:"refs"`
} `json:"stats"`
}
When a new url is given, then things become a little bit weird:
{
"stats": {
"status": 1,
"date": "09.07.2019",
"title": "Test2",
"devices": [
],
"refs": [
]
}
}
As you can see, the fields "dev", "sys" and "bro" just disappear because they're not used and when I try to Unmarshal the JSON into the same struct I get json: cannot unmarshal array into Go struct field Stats.device of type [...]
I tried to use two different structs to handle both the responses but I'm sure that there's a way to handle them gracefully with just one.
Any help would be appreciated, thanks!
I finally managed to make it work with an ugly workaround.
I changed my struct to
type Stats struct {
Status int `json:"status"`
Date string `json:"date"`
Title string `json:"title"`
Devices interface{} `json:"devices"`
Refs interface{} `json:"refs"`
}
Then I can finally Unmarshal the JSON in both cases, but I get a map[string]interface{} when an object is passed and an empty interface{} when an empty array is passed. In order to fix this inconsistency, I simply check for the data type and force the use of a JSON intermediate conversion in order to unpack the map[string]interface{} value inside a custom Devices struct:
// Devices contains devices information
type Devices struct {
Dev []struct {
Tag string `json:"tag"`
Clicks string `json:"clicks"`
} `json:"dev"`
Sys []struct {
Tag string `json:"tag"`
Clicks string `json:"clicks"`
} `json:"sys"`
Bro []struct {
Tag string `json:"tag"`
Clicks string `json:"clicks"`
} `json:"bro"`
}
The algorithms I use are the following:
//ForceDevicesToRightType uses a json conversion as intermediary for filling the Stats.Devices
// struct with map[string]interface{} values
func ForceDevicesToRightType(dev interface{}) (Devices, error) {
temp, err := json.Marshal(dev)
if err != nil {
return Devices{}, err
}
// Use a temporary variable of the right type
var devices Devices
err = json.Unmarshal(temp, &devices)
if err != nil {
return Devices{}, err
}
return devices, nil
}
// ForceRefsToRightType uses a json conversion as intermediary for filling the Stats.Refs
// struct with map[string]interface{} values
func ForceRefsToRightType(refs interface{}) (Refs, error) {
temp, err := json.Marshal(refs)
if err != nil {
return Refs{}, err
}
// Use a temporary variable of the right type
var references Refs
err = json.Unmarshal(temp, &references)
if err != nil {
return Refs{}, err
}
return references, nil
}
Since the compiler knows that both Devices and Refs fields are interface{} I cannot simply access any methods after the conversion, so I simply make a cast of the right type and everything works fine.
For example, if I wanted to access the Dev sub-struct, this is the proper way:
y, _ := GetStats()
fmt.Println(y.Devices.(Devices).Dev)
It's ugly, but it works.
Thank you very much for your help, I hope that this method will save you an headache!

Parsing JSON with GoLang in AWS Lamda

As part of an application we are building, one of the process steps is an AWS Lamda that captures a post request does some work with it, and then moves one. It has an API Gateway Request as a trigger and the body of this request would be a JSON String. I am having trouble parsing the JSON String to GoLang Object. Here is what I have:
The function that catches request:
func HandleRequest(ctx context.Context, event events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
log.Print(fmt.Sprintf("body:[%s] ", event.Body))
parseResponseStringToTypedObject(event.Body)
return events.APIGatewayProxyResponse{
StatusCode: http.StatusOK,
Body: "OK",
}, nil
}
Then the parseResponseStringToTypedObject function :
func parseResponseStringToTypedObject(responseString string) {
b := []byte(responseString)
var resp SimpleType
err := json.Unmarshal(b, &resp)
if err == nil {
log.Print(fmt.Sprintf("Account Name: [%s]", resp.accountName))
} else {
log.Print(fmt.Sprintf("Could not unmarshall JSON string: [%s]", err.Error()))
}
}
Here is the SimpleType struct:
type SimpleType struct {
accountName string `json:accountName`
amount int `json:amount`
}
I then, as a test, posted this JSON Body via Postman:
I opened up the CloudWatch Logs (where my lamda logs to) and see that the body is present in the event.Body property, and then logging out a field in the unmarshalled object (resp.accountName) I note that the field is blank. Why is this? Here is log output for the request:
Your SimpleType struct needs 2 things here...
1) The properties need to be "public" or "exported". Meaning they need to start with an upper cased character.
AND
2) The properties need proper tags for the serialization and de serialization of JSON. e.g. each JSON tag surrounded by "
type SimpleType struct {
AccountName string `json:"accountName"`
Amount int `json:"amount"`
}
Hope this helps!

Clean way to conditionally unmarshal JSON to struct

I'm sending requests to a JSON API, and it either returns an error...
{
"error": {
"code": 404,
"message": "Document not found.",
"status": "NOT_FOUND"
}
}
or the data.
{
"name": "projectname",
"fields": {
"userId": {
"stringValue": "erw9384rjidfge"
}
},
"createTime": "2018-06-28T00:52:25.638791Z",
"updateTime": "2018-06-28T00:52:25.638791Z"
}
Here are the corresponding structs
type HttpError struct {
Code int `json:"code"`
Message string `json:"message"`
Status string `json:"status"`
}
type Document struct {
Name string `json:"name"`
Fields struct {
UserID struct {
StringValue string `json:"stringValue"`
} `json:"userId"`
} `json:"fields"`
CreateTime time.Time `json:"createTime"`
UpdateTime time.Time `json:"updateTime"`
}
Once I get the response, how do I cleanly/concisely unmarshal to the correct struct? I've seen a lot of ugly solutions (maybe Go's fault instead of the writers).
func getDocument() {
resp, _ := httpClient.Get("example.com")
defer resp.Body.Close()
bodyBytes, _ := ioutil.ReadAll(resp.Body)
var data map[string]interface{}
// How to unmarshal to either HttpError or Document??
err = json.Unmarshal([]byte(bodyBytes), &data)
}
By the way, I can't use the Go Firestore client library because reasons.
You can use an struct type inside your unmarshal method; with pointers to establish what's been unmarshalled.
Note: This code assumes there is no overlap of top level json keys... error / name / fields / etc.
type outer struct {
*HttpError `json:"error"`
*Document
}
var out outer
if err := json.Unmarshal(bodyBytes, &out); err != nil {
// error handling
}
if out.HttpErr != nil {
// handle error json case
}
// Here you can use out.Document, probably worth check if it is nil first.
Runnable example

Marshall JSON Slice to valid JSON

I'm building a REST API using Golang but I'm having some troubles trying to correctly Marshalling a Json Slice. I've been scratching my head for a while now, even after looking to several questions and answer and on the web.
Essentially, I have a Redis client that called after a call -X GET /todo/ spits up a slice of todos
[{"content":"test6","id":"46"} {"content":"test5","id":"45"}] //[]string
Now, I want to return a given Response based on the fact that I found todos or not, so I have a Struct like
type Response struct {
Status string
Data []string
}
Then, If I found some todos I just Marshal a json with
if(len(todos) > 0){
res := SliceResponse{"Ok", todos}
response, _ = json.Marshal(res)
}
And, In order to remove unnecessary \ inside the response, I use bytes.Replace like
response = bytes.Replace(response, []byte("\\"), []byte(""), -1)
Finally, getting
{
"Status" : "Ok",
"Data" : [
"{"content":"test6","id":"46"}",
"{"content":"test5","id":"45"}"
]
}
As you can see each " before each { and after each }, excluding the first and the last ones, are clearly wrong.
While the correct JSON would be
{
"Status": "Ok",
"Data": [{
"content ": "test6",
"id ": "46"
}, {
"content ": "test5",
"id ": "45"
}]
}
I successfully managed to get them off by finding their index and trim them off and
also with regex but I was wondering.
Is there a clean and better way to achieve that?
Whenever possible you should marshal from go objects that match your desired json. I'd recommend parsing the json from redis:
type Response struct {
Status string
Data []*Info
}
type Info struct {
Content string `json:"content"`
ID string `json:"id"`
}
func main() {
r := &Response{Status: "OK"}
for _, d := range data {
info := &Info{}
json.Unmarshal([]byte(d), info)
//handle error
r.Data = append(r.Data, info)
}
dat, _ := json.Marshal(r)
fmt.Println(string(dat))
}
Playground Link