Parsing JSON with GoLang in AWS Lamda - json

As part of an application we are building, one of the process steps is an AWS Lamda that captures a post request does some work with it, and then moves one. It has an API Gateway Request as a trigger and the body of this request would be a JSON String. I am having trouble parsing the JSON String to GoLang Object. Here is what I have:
The function that catches request:
func HandleRequest(ctx context.Context, event events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
log.Print(fmt.Sprintf("body:[%s] ", event.Body))
parseResponseStringToTypedObject(event.Body)
return events.APIGatewayProxyResponse{
StatusCode: http.StatusOK,
Body: "OK",
}, nil
}
Then the parseResponseStringToTypedObject function :
func parseResponseStringToTypedObject(responseString string) {
b := []byte(responseString)
var resp SimpleType
err := json.Unmarshal(b, &resp)
if err == nil {
log.Print(fmt.Sprintf("Account Name: [%s]", resp.accountName))
} else {
log.Print(fmt.Sprintf("Could not unmarshall JSON string: [%s]", err.Error()))
}
}
Here is the SimpleType struct:
type SimpleType struct {
accountName string `json:accountName`
amount int `json:amount`
}
I then, as a test, posted this JSON Body via Postman:
I opened up the CloudWatch Logs (where my lamda logs to) and see that the body is present in the event.Body property, and then logging out a field in the unmarshalled object (resp.accountName) I note that the field is blank. Why is this? Here is log output for the request:

Your SimpleType struct needs 2 things here...
1) The properties need to be "public" or "exported". Meaning they need to start with an upper cased character.
AND
2) The properties need proper tags for the serialization and de serialization of JSON. e.g. each JSON tag surrounded by "
type SimpleType struct {
AccountName string `json:"accountName"`
Amount int `json:"amount"`
}
Hope this helps!

Related

Go Fiber not able to parse body in unit test

I am officially crying uncle to the benevolent samaritans of Stack Overflow.
I am trying to unit test my GORM (Postgres) + Fiber API using a mock DB. I have a Card model and a CreateCardReqBody model for the POST request body. To setup the test, I create a random CreateCardReqBody instance, marshal it into JSON, then pass it into an *httptest.Request. The handler uses Fiber's (*fiber.Ctx).BodyParser function to "unmarshal" the request body into an empty Card struct. However, when I run the test that is supposed to pass, Fiber throws an "Unprocessable Entity" error.
Below are the relevant parts of my code; the test file is a combination of this tutorial and Fiber's documentation on the (*App).Test method. (I realize the code could be cleaned up; I'm just trying to get a proof of life then focus on revising :)
I've done a few things to debug this: I've made a Postman POST request with the same values as the test and it works. Within the test itself, I marshal then unmarshal the CreateCardReqBody struct and that works. I've triple checked the spelling of the JSON fields match, that the struct fields are exported, etc. I've also run the VSCode debugger and the body field within Fiber.Ctx's also looks correct to me.
I'm starting to wonder if it's something with how Fiber parses the body from a test request vs. a real request. I would greatly appreciate any insight one could share on this!
Model Definition
type Card struct {
gorm.Model
// Implicit Gorm foreign key to fleet ID
FleetID uint `gorm:"index" json:"fleet_id" validate:"required,min=1"`
// Card provider's account number
ProviderAccountNumber string `json:"provider_account_number"`
// Card provider's external card identifier
CardIdentifier string `gorm:"index" json:"card_identifier" validate:"min=1"`
// Implicit Gorm foreign key to driver ID. Driver association is optional.
DriverID uint `json:"associated_driver_id" validate:"min=1"`
// Implicit Gorm foreign key to vehicle ID.
VehicleID uint `json:"associated_vehicle_id" validate:"required,min=1"`
// User-inputted start date, formatted "2020-01-26T22:38:25.000Z" in UTC
StartDate pq.NullTime
}
Test file
// Adapted from tutorial
type testCase struct {
name string
body CreateCardReqBody
setupAuth func(t *testing.T, request *http.Request)
buildStubs func(db *mockDB.MockDBInterface)
checkResponse func(response *http.Response, outputErr error)
}
type CreateCardReqBody struct {
FleetID int `json:"fleet_id"`
ProviderAccountNumber string `json:"provider_account_number"`
CardIdentifier string `json:"card_identifier"`
StartDate string `json:"start_date"`
AssociatedDriverID int `json:"associated_driver_id"`
AssociatedVehicleID int `json:"associated_vehicle_id"`
}
func TestCreateCard(t *testing.T) {
user := randomUser(t)
vehicle := randomVehicle()
driver := randomDriver(vehicle.FleetID)
okReqCard := randomCard(vehicle.FleetID)
finalOutputCard := okReqCard
finalOutputCard.ID = 1
testCases := []testCase{
{
name: "Ok",
body: CreateCardReqBody{
FleetID: int(okReqCard.FleetID),
ProviderAccountNumber: okReqCard.ProviderAccountNumber,
CardIdentifier: okReqCard.CardIdentifier,
StartDate: okReqCard.StartDate.Time.Format("2006-01-02T15:04:05.999Z"),
AssociatedDriverID: int(okReqCard.DriverID),
AssociatedVehicleID: int(okReqCard.VehicleID),
},
setupAuth: func(t *testing.T, request *http.Request) {
addAuthorization(t, request, user)
},
// Tell mock database what calls to expect and what values to return
buildStubs: func(db *mockDB.MockDBInterface) {
db.EXPECT().
UserExist(gomock.Eq(fmt.Sprint(vehicle.FleetID))).
Times(1).Return(user, true, user.ID)
db.EXPECT().
SearchTSP(gomock.Eq(fmt.Sprint(vehicle.FleetID))).
Times(1)
db.EXPECT().
SearchVehicle(gomock.Eq(fmt.Sprint(okReqCard.VehicleID))).
Times(1).
Return(vehicle, nil)
db.EXPECT().
SearchDriver(gomock.Eq(fmt.Sprint(driver.ID))).
Times(1).
Return(driver, nil)
db.EXPECT().
CardCreate(gomock.Eq(okReqCard)).
Times(1).
Return(finalOutputCard, nil)
},
checkResponse: func(res *http.Response, outputErr error) {
require.NoError(t, outputErr)
// Internal helper func, excluded for brevity
requireBodyMatchCard(t, finalOutputCard, res.Body)
},
},
}
for _, test := range testCases {
t.Run(test.name, func(t *testing.T) {
ctrl := gomock.NewController(t)
defer ctrl.Finish()
mockDB := mockDB.NewMockDBInterface(ctrl)
test.buildStubs(mockDB)
jsonBytes, err := json.Marshal(test.body)
require.NoError(t, err)
jsonBody := bytes.NewReader(jsonBytes)
// Debug check: am I able to unmarshal it back? YES.
errUnmarsh := json.Unmarshal(jsonBytes, &CreateCardReqBody{})
require.NoError(t, errUnmarsh)
endpoint := "/v1/transactions/card"
request := httptest.NewRequest("POST", endpoint, jsonBody)
// setupAuth is helper function (not shown in this post) that adds authorization to httptest request
test.setupAuth(t, request)
app := Init("test", mockDB)
res, err := app.Test(request)
test.checkResponse(res, err)
})
}
}
Route handler being tested
func (server *Server) CreateCard(c *fiber.Ctx) error {
var card models.Card
var err error
// 1) Parse POST data
if err = c.BodyParser(&card); err != nil {
return c.Status(http.StatusUnprocessableEntity).SendString(err.Error())
}
...
}
Debugger Output
Json body when defined in test
Body inside Fiber context
facepalm
I forgot to request.Header.Set("Content-Type", "application/json")! Posting this in case it's helpful for anyone else :)

Wrap API Json response in Go

I am sorry if this is a silly question because i am very new in Go.
I am calling couple of apis on the base of business logic, different types of response coming like json array, nest json and single json object.
i need to wrap a api response that called according to business logic in a common format like:
{
"data":"api response here",
"statusCode":200
}
i tried some but its not expect output
type Model[T any] struct {
Data T
StatusCode int
}
model := Model[string]{Data: apiResponse, StatusCode: 200}
out, err := json.Marshal(model)
out put of this code is
{
"Data": "[{\"name\":\"Harry Potter\",\"city\":\"London\"},{\"name\":\"Don Quixote\",\"city\":\"Madrid\"},{\"name\":\"Joan of Arc\",\"city\":\"Paris\"},{\"name\":\"Rosa Park\",\"city\":\"Alabama\"}]",
"StatusCode": 200
}
this i made these changes
var result []map[string]interface{}
json.Unmarshal([]byte(body), &result)
out, err := json.Marshal(result)
output was as expected, above api response was in proper json when use []map[string]interface
problem is, its only for those api that return array of json. those apis returning single json object then to make it work i need to do this
map[string]interface`
means remove the array map.
i need to make it generic so that any kind of json response map into it.
Use type of field Data as an interface{}
type APIResponse struct {
Data interface{} `json:"data"`
StatusCode int `json:"statusCode"`
}
And then you can assign any API Response type to the Data field and marshal it.
func main() {
r := []Person{
{
Name: "Harry Porter",
City: "London",
},
{
Name: "Don Quixote",
City: "Madrid",
},
}
res := APIResponse{
Data: r,
StatusCode: 200,
}
resByt, err := json.Marshal(res)
if err != nil {
panic(err)
}
fmt.Println(string(resByt))
}
Output
{"data":[{"name":"Harry Porter","city":"London"},{"name":"Don Quixote","city":"Madrid"}],"statusCode":200}
Run the full code here in Playground.
You can simply do:
result:=map[string]interface{} {
"data": apiResponse,
"statusCode": 200,
}
out, err:=json.Marshal(result)

GraphQL JSON to Go struct

I have a GraphQL query that looks like this:
{
actor {
entitySearch(query: "name LIKE 'SOME_NAME'") {
results {
entities {
guid
}
}
}
}
}
I can't figure out how to create the Go struct to hold the returned data. The only thing I care about is the guid field that gets returned.
This clearly doesn't work:
type graphQlResponse struct {
guid string
}
Any help? Or is there a way I can simply get the guid and store it in a string without a struct?
Here is the whole code. I don't get an error, but guid is an empty string:
package main
import (
"context"
"fmt"
"log"
"github.com/machinebox/graphql"
)
func main() {
type graphQlResponse struct {
guid string
}
// create a client (safe to share across requests)
client := graphql.NewClient("GraphQL EndPoint")
// make a request
req := graphql.NewRequest(`
{
actor {
entitySearch(query: "name LIKE 'SOME_NAME'") {
results {
entities {
guid
}
}
}
}
}
`)
// set any variables
//req.Var("key", "value")
// set header fields
//req.Header.Set("Cache-Control", "no-cache")
req.Header.Set("Content-Type", "application/json")
req.Header.Set("API-Key", "KEY_HERE")
// define a Context for the request
ctx := context.Background()
// run it and capture the response
var respData graphQlResponse
if err := client.Run(ctx, req, &respData); err != nil {
log.Fatal(err)
}
fmt.Println(respData.guid)
}
In GraphQL, the shape of the returned JSON will match the shape of the GraphQL query: you will have a "data" field, which will have an "actor" child, which will contain "entitySearch", and so on. The library you're calling is in fact pretty minimal. Given the conventional HTTP transport format, it uses ordinary encoding/json decoding to decode the response. Whatever structure you provide needs to be able to unmarshal the "data" field.
This means that you need to create a nested set of structures that mirrors the JSON format, which in turn mirrors your GraphQL query:
type Entity struct {
Guid string `json:"guid"`
}
type Result struct {
Entities Entity `json:"entities"`
}
type EntitySearch struct {
Results Result `json:"results"`
}
type Actor struct {
EntitySearch EntitySearch `json:"entitySearch"`
}
type Response struct {
Actor Actor `json:"actor"`
}
fmt.Println(resp.Actor.EntitySearch.Results.Entities.Guid)
https://play.golang.org/p/ENCIjtfAJif has an example using this structure and an artificial JSON body, though not the library you mention as such.
I suggest using a map and the json package.
I am unfamiliar with graphQL so I'll make a regular HTTP request and hope you can use it to make sense of your own issue:
response, err := http.Get("https://example.com")
// error checking code omitted
defer response.Body.Close()
// now we want to read the body, easiest way is with the ioutil package,
// this should work with the graphQL response body assuming it satisfies
// the io.Reader interface. This gets us the response body as a byte slice
body, err := ioutil.ReadAll(response.Body)
// next make a destination map, interface can be used in place of string or int
// if you need multiple types
jsonResult := map[string]string{"uuid": ""}
// finally, we use json.Unmarshal to write our byte slice into the map
err = json.Unmarshal(body, &jsonResult)
// now you can access your UUID
fmt.Println(jsonResult["uuid"])
I assume that a REST response and a graphQL response are similar, if that's not the case let me know what type the request body is and I can help you figure out a better fitting solution.

go-json-rest: JSON payload is empty

I'm trying to use go-json-rest to create RESTful APIs
I have this model struct:
type Todo struct {
id int
name string
}
And I am trying to do a POST request to create an object of Todo type:
func CreateTodo(w rest.ResponseWriter, r *rest.Request) {
body, _ := ioutil.ReadAll(r.Body)
log.Println("body content is: ", string(body)) // here I can see {"name": "test1"}
var todo Todo = Todo{}
err := r.DecodeJsonPayload(&todo) // here the error shows JSON payload is empty
defer r.Body.Close()
if err != nil {
log.Println("error in parsing json")
rest.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if todo.name == "" {
rest.Error(w, "todo name is required", 400)
return
}
lock.Lock()
var nextId int = len(todos)
// todos[nextId] = todo
todo.id = nextId // set its id
todos = append(todos, todo)
log.Println("num of todos: ", len(todos))
lock.Unlock()
w.WriteJson(&todo)
}
However, in the console of the POSTMAN client, the error shows:
{
"Error": "JSON payload is empty"
}
I was wondering where might go wrong. Thanks
Edit:
This should not be considered a duplicate question, because I am not even trying to use the json package to do marshalling/unmarshalling of JSON object. Instead I am using the rest.Request object (built in go-json-rest) to decode the json parameters as posted from client.
After much digging and search on this problem I found this answer below resolved my issue:
body, _ := ioutil.ReadAll(r.Body) will consume everything from the
request body. So after removing this line, the json parsing works!
I was just being silly doing body, _ := ioutil.ReadAll(r.Body) before decoding the JSON parameters without really understanding what ioutil.ReadAll() does to the request body.
As I quoted above in the edited post, ioutil.ReadAll() consumes everything in the request body, leaving nothing for the json decoder to parse. After removing the line body, _ := ioutil.ReadAll(r.Body), the json parsing works as expected.

Decoding variable-schema JSON in Go

I'm asking this about Go's encoding/json, but I guess it also applies to any other JSON libraries that map JSON blobs to objects in whatever language.
Here's an example. If you want to a shorten a URL using the goo.gl URL shortener API, you get back either a successful response:
{
"kind": "urlshortener#url",
"id": "http://goo.gl/fbsS",
"longUrl": "http://www.google.com/"
}
Or an error response:
{
"error": {
"errors": [
{
"domain": "global",
"reason": "required",
"message": "Required",
"locationType": "parameter",
"location": "resource.longUrl"
}
],
"code": 400,
"message": "Required"
}
}
Is there an idiomatic way of dealing with this -- a response that could adhere to two completely different schemas?
Normally I deal with JSON using maps/lists; I know that's possible in Go. I could unmarshal to a map[string]interface{} and then check if the map has "error" as a key. But then I'd have to decode again into a proper struct, I think. (Am I wrong?)
I'm doing something like this. I have one type for each kind of response:
type successResponse struct {
Kind string
Id string
LongUrl string
}
type errorResponse struct {
Error struct {
Errors []struct {
Domain string
Reason string
Message string
LocationType string
Location string
}
Code int
Message string
}
}
And decoding looks like this:
s := new(successResponse)
err := json.Unmarshal(blob, s)
if err == nil {
// handle success
} else {
e := new(errorResponse)
err = json.Unmarshal(blob, e)
if err == nil {
// handle error response
} else {
// handle actual error
}
}
But that seems kind of ugly. How should I approach this?
Since the fields in your json responses are distinct from each other you can just create one struct with the union of all the fields. The json decoder will ignore fields that are not present in the json string and you can test the existence of the fields to know which type of response you are getting back.
I was confused about this, too, and thought I had to decode it again. You don't, though. You just have to typecast the interface{} data into the appropriate structure.
For example if the json package has put the value into a generic interface{}, you can typecast it into ErrorType with error := val.(ErrorType).
You can use foo.(type) in a switch statement to "do the right thing", if you are parsing based on what type the value is.
I've only been learning Go this week so it's not the prettiest code, but there are some examples in the geodns JSON configuration parsing.
Have you tried Go-SimpleJSON? I think this might solve your issue.
type Response struct {
Kind string
Id string
LongUrl string
Error struct {
Errors []struct {
Domain string
Reason string
Message string
LocationType string
Location string
}
Code int
Message string
}
}
s := Response{}
if err := json.Unmarshal(blob, &s); err == nil {
if s.Error == nil {
// success
} else {
// error
}
} else {
// something went wrong
}