Golang - Can't get array of objects after fetching JSON - json

After I make a request to a server - I get JSON like this:
{
"actions": [
{
"class": "...",
"parameters": [
{ ... }
{ ... }
]
}
]
...
}
The type of the variable I've put this data in, is map[string]interface{}.
I want to access the parameters array in the first object of the actions array. I can successfully get the class property - data.(map[string]interface{})["class"].
However, if I try the same for the parameters property - I get nil ...
I tried data.(map[string][]interface{})["parameters"] - but I get error panic: interface conversion: interface {} is map[string]interface {}, not map[string][]interface {}.
Any idea what I'm missing here?
EDIT:
The Golang code for this is this:
func main() {
var jsonResult map[string]interface{}
errorFromJsonFetching := utils.GetJSON(theUrl, &jsonResult)
if errorFromJsonFetching != nil {
fmt.Printf("Error from checking deploy build: %#v\n", errorFromJsonFetching)
}
// get the top level "actions" prop ..
actionZero := jsonResult["actions"].([]interface{})[0]
fmt.Printf("Class prop: /%v %T\n", actionZero.(map[string]interface{})["class"], actionZero)
fmt.Printf("Parameters prop: /%v %T\n", actionZero.(map[string]interface{})["parameters"], actionZero)
}
The GetJSON function is from another file in the project:
func GetJSON (url string, result interface{}) error {
fmt.Printf("Getting JSON from %v\n", url)
resp, err := http.Get(url)
if err != nil {
return fmt.Errorf("Cannot fetch URL %q: %v\n", url, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("Bad resposne status code: %s\n", resp.Status)
}
// attempt to put the JSON in the `result` ..
err = json.NewDecoder(resp.Body).Decode(&result)
if err != nil {
return fmt.Errorf("Could not decode the JSON from response, err: %v\n", err)
}
return nil;
}
EDIT2:
Thanks to the ideas of #larsks in the comments - I created a minimal code sample with JSON in a string, directly in main.go file - and all worked fine.
Then I went ahead to the browser again and tried to fetch the data again - from directly hitting the URL or with $.getJSON from a page - and both returned one and the same JSON data.
However, in my Go code, when I dump the JSON data - I see this for the first member of actions:
map[_class:hudson.model.ParametersDefinitionProperty parameterDefinitions:[map[...
So when I try to get the parameters array by the key parameterDefinitions - then I get the array of object ... :O
Sooooo ... I don't know what's happening ... either Go itself modifies the JSON data when it gets it from the backend, or the backend itself returns different things, depending on how the data is being fetched.
(The backend is the Jenkins API by the way ... so I don't know why I get parameterDefinitions instead of parameters in Go ... :( ...)

Thanks for updating your question; having the correct data makes it easier to answer.
It's best if the code you include in your question is something we can just grab and run -- that means it compiles and runs, and when given the sample data in your question, it produces the behavior you're asking about.
I've modified your code so that I can run it locally, primarily by replacing the GetJSON method with something that reads from a file named data.json instead of hitting an API:
package main
import (
"encoding/json"
"fmt"
"os"
)
func GetJSON(result interface{}) error {
datafile, err := os.Open("data.json")
if err != nil {
panic(err)
}
return json.NewDecoder(datafile).Decode(&result)
}
func main() {
var jsonResult map[string]interface{}
errorFromJsonFetching := GetJSON(&jsonResult)
if errorFromJsonFetching != nil {
fmt.Printf("Error from checking deploy build: %#v\n", errorFromJsonFetching)
}
// get the top level "actions" prop ..
actionZero := jsonResult["actions"].([]interface{})[0]
fmt.Printf("Class prop: /%v %T\n", actionZero.(map[string]interface{})["class"], actionZero)
fmt.Printf("Parameters prop: /%v %T\n", actionZero.(map[string]interface{})["parameters"], actionZero)
}
If I feed it this sample data, which I believe matches the structure of what you show in your question:
{
"actions": [
{
"class": "Name of class",
"parameters": [
{
"name": "alice",
"color": "blue"
},
{
"name": "bob",
"count": 10,
},
{
"name": "mallory",
"valid": false,
}
]
}
]
}
It produces the following output:
Class prop: /Name of class map[string]interface {}
Parameters prop: /[map[color:blue name:alice] map[count:10 name:bob] map[name:mallory valid:false]] map[string]interface {}
This doesn't produce a nil value for the parameters key. In order to more fully answer your question, can you update it so that it has sample data and code that reproduces the behavior you're asking baout?

Related

What could be a unit test case for the s3/sqs event when unmarshalling json to struck in Go Lang?

What could be a unit test case for the s3/sqs event when unmarshalling json to struck in Go?
One option that comes to mind is if the "Body" is empty, but not sure what else could it be.
Code example:
type Event struct {
Message string `json:"Message"`
}
func unmarshal(message events.SQSMessage) (s3Event, error) {
var e Event
err := json.Unmarshal([]byte(message.Body), &e)
if err != nil {
return s3Event, err
}
var s3Event events.S3Event
err = json.Unmarshal([]byte(e.Message), &sqsEvent)
if err != nil {
return s3Event, err
}
return s3Event, nil
}
Would appreciate any advice.
Premise: I'm not sure if I've fully understood your question. Furthermore, the code has some issues that cause me hard-time to figure out what's going on. With that being said, I'll try to help you by providing a piece of working software (it's not very useful) together with a piece of software written just to test it. Maybe, this can help you in figuring out your issue.
unmarshal.go file
package unmarshaltest
import (
"encoding/json"
"fmt"
"github.com/aws/aws-lambda-go/events"
)
type Event struct {
Message string `json:"body"`
}
func Unmarshal(message events.SQSMessage) (string, error) {
rawBytes, _ := json.Marshal(message)
var e Event
json.Unmarshal(rawBytes, &e)
if e.Message == "" {
return "", fmt.Errorf("err while unmarshaling")
}
return e.Message, nil
}
One thing I fixed and you should pay attention to is how you name your functions. If you call your function unmarshal with lowercase u it means that this function should not be exported from this package (it's private). So I changed the name with the U as I supposed you wanna test this function.
unmarshal_test.go file
package unmarshaltest
import (
"encoding/json"
"testing"
"github.com/aws/aws-lambda-go/events"
"github.com/stretchr/testify/assert"
)
func TestUnmarshal(t *testing.T) {
t.Run("Unmarshal_NotEmpty_Message", func(t *testing.T) {
var sqsMsg events.SQSMessage
sqsEventRaw := `{ "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78", "receiptHandle": "MessageReceiptHandle", "body": "My own event payload!", "attributes": { "ApproximateReceiveCount": "1", "SentTimestamp": "1523232000000", "SenderId": "123456789012", "ApproximateFirstReceiveTimestamp": "1523232000001" }, "messageAttributes": {}, "md5OfBody": "4d1d0024b51659ad8c3725f9ba7e2471", "eventSource": "aws:sqs", "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue", "awsRegion": "us-east-1" }`
json.Unmarshal([]byte(sqsEventRaw), &sqsMsg)
res, err := Unmarshal(sqsMsg)
assert.Equal(t, "My own event payload!", res)
assert.Nil(t, err)
})
t.Run("Unmarshal_Empty_Message", func(t *testing.T) {
res, err := Unmarshal(events.SQSMessage{})
assert.Equal(t, "", res)
assert.NotNil(t, err)
})
}
Here you can find a simple piece of code that is used to test the two possible execution flows that our code might take.
You should also pay attention to how you write the code. It's very hard to swap the invocation to the json.Unmarshal function with a mock as it's hard-coded into the code. If your wish is to be able to mock out this function, you should rely on interfaces in your production code and replace them with mocks during tests.
Let me know if this clarifies something and if you need something else!

Wrap API Json response in Go

I am sorry if this is a silly question because i am very new in Go.
I am calling couple of apis on the base of business logic, different types of response coming like json array, nest json and single json object.
i need to wrap a api response that called according to business logic in a common format like:
{
"data":"api response here",
"statusCode":200
}
i tried some but its not expect output
type Model[T any] struct {
Data T
StatusCode int
}
model := Model[string]{Data: apiResponse, StatusCode: 200}
out, err := json.Marshal(model)
out put of this code is
{
"Data": "[{\"name\":\"Harry Potter\",\"city\":\"London\"},{\"name\":\"Don Quixote\",\"city\":\"Madrid\"},{\"name\":\"Joan of Arc\",\"city\":\"Paris\"},{\"name\":\"Rosa Park\",\"city\":\"Alabama\"}]",
"StatusCode": 200
}
this i made these changes
var result []map[string]interface{}
json.Unmarshal([]byte(body), &result)
out, err := json.Marshal(result)
output was as expected, above api response was in proper json when use []map[string]interface
problem is, its only for those api that return array of json. those apis returning single json object then to make it work i need to do this
map[string]interface`
means remove the array map.
i need to make it generic so that any kind of json response map into it.
Use type of field Data as an interface{}
type APIResponse struct {
Data interface{} `json:"data"`
StatusCode int `json:"statusCode"`
}
And then you can assign any API Response type to the Data field and marshal it.
func main() {
r := []Person{
{
Name: "Harry Porter",
City: "London",
},
{
Name: "Don Quixote",
City: "Madrid",
},
}
res := APIResponse{
Data: r,
StatusCode: 200,
}
resByt, err := json.Marshal(res)
if err != nil {
panic(err)
}
fmt.Println(string(resByt))
}
Output
{"data":[{"name":"Harry Porter","city":"London"},{"name":"Don Quixote","city":"Madrid"}],"statusCode":200}
Run the full code here in Playground.
You can simply do:
result:=map[string]interface{} {
"data": apiResponse,
"statusCode": 200,
}
out, err:=json.Marshal(result)

Proper json unmarshaling in Go with the empty interface

I'm currently learning golang and (probably as many others before me) I'm trying to properly understand the empty interface.
As an exercise, I'm reading a big json file produced by Postman and trying to access just one field (out of the many available).
Here is a simple representation of the json without the unnecessary fields I don't want to read (but that are still there):
{
"results": [
{
"times": [
1,
2,
3,
4
]
}
]
}
Since the json object is big, I opted out of unmarshaling it with a custom struct, and rather decided to use the empty interface interface{}
After some time, I managed to get some working code, but I'm quite sure this isn't the correct way of doing it.
byteValue, _ := ioutil.ReadAll(jsonFile)
var result map[string]interface{}
err = json.Unmarshal(byteValue, &result)
if err != nil {
log.Fatalln(err)
}
// ESPECIALLY UGLY
r := result["results"].([]interface{})
r1 := r[0].(map[string]interface{})
r2 := r1["times"].([]interface{})
times := make([]float64, len(r2))
for i := range r2 {
times[i] = r2[i].(float64)
}
Is there a better way to navigate through my json object without having to instantiate new variables every time i move deeper and deeper into the object?
Even if the JSON is large, you only have to define the fields you actually care about
You only need to use JSON tags if the keys aren't valid Go
identifiers (keys are idents in this case), even then you can sometimes avoid it by using a map[string]something
Unless you need the sub structs for some function or whatnot, you don't need to define them
Unless you need to reuse the type, you don't even have to define that, you can just define the struct at declaration time
Example:
package main
import (
"encoding/json"
"fmt"
)
const s = `
{
"results": [
{
"times": [1, 2, 3, 4]
}
]
}
`
func main() {
var t struct {
Results []struct {
Times []int
}
}
json.Unmarshal([]byte(s), &t)
fmt.Printf("%+v\n", t) // {Results:[{Times:[1 2 3 4]}]}
}
[...] trying to access just one field (out of the many available).
For this concrete use case I would use a library to query and access to a single value in a known path like:
https://github.com/jmespath/go-jmespath
In the other hand, if you're practicing how to access nested values in a JSON, I would recommend you to give a try to write a recursive function that follows a path in an unknown structure the same way (but simple) like go-jmespath does.
Ok, I challenged myself and spent an hour writing this. It works. Not sure about performance or bugs and it's really limited :)
https://play.golang.org/p/dlIsmG6Lk-p
package main
import (
"encoding/json"
"errors"
"fmt"
"strings"
)
func main() {
// I Just added a bit more of data to the structure to be able to test different paths
fileContent := []byte(`
{"results": [
{"times": [
1,
2,
3,
4
]},
{"times2": [
5,
6,
7,
8
]},
{"username": "rosadabril"},
{"age": 42},
{"location": [41.5933262, 1.8376757]}
],
"more_results": {
"nested_1": {
"nested_2":{
"foo": "bar"
}
}
}
}`)
var content map[string]interface{}
if err := json.Unmarshal(fileContent, &content); err != nil {
panic(err)
}
// some paths to test
valuePaths := []string{
"results.times",
"results.times2",
"results.username",
"results.age",
"results.doesnotexist",
"more_results.nested_1.nested_2.foo",
}
for _, p := range valuePaths {
breadcrumbs := strings.Split(p, ".")
value, err := search(breadcrumbs, content)
if err != nil {
fmt.Printf("\nerror searching '%s': %s\n", p, err)
continue
}
fmt.Printf("\nFOUND A VALUE IN: %s\n", p)
fmt.Printf("Type: %T\nValue: %#v\n", value, value)
}
}
// search is our fantastic recursive function! The idea is to search in the structure in a very basic way, for complex querying use jmespath
func search(breadcrumbs []string, content map[string]interface{}) (interface{}, error) {
// we should never hit this point, but better safe than sorry and we could incurr in an out of range error (check line 82)
if len(breadcrumbs) == 0 {
return nil, errors.New("ran out of breadcrumbs :'(")
}
// flag that indicates if we are at the end of our trip and whe should return the value without more checks
lastBreadcrumb := len(breadcrumbs) == 1
// current breadcrumb is always the first element.
currentBreadcrumb := breadcrumbs[0]
if value, found := content[currentBreadcrumb]; found {
if lastBreadcrumb {
return value, nil
}
// if the value is a map[string]interface{}, go down the rabbit hole, recursion!
if aMap, isAMap := value.(map[string]interface{}); isAMap {
// we are calling ourselves popping the first breadcrumb and passing the current map
return search(breadcrumbs[1:], aMap)
}
// if it's an array of interfaces the thing gets complicated :(
if anArray, isArray := value.([]interface{}); isArray {
for _, something := range anArray {
if aMap, isAMap := something.(map[string]interface{}); isAMap && len(breadcrumbs) > 1 {
if v, err := search(breadcrumbs[1:], aMap); err == nil {
return v, nil
}
}
}
}
}
return nil, errors.New("woops, nothing here")
}

Custom JSON mapping function in Go

So I'm making a Go service that makes a call to a restful API, I have no control over the API I'm calling.
I know that Go has a nice built in deserializer in NewDecoder->Decode, but it only works for struct fields that start with capital letters (aka public fields). Which poses a problem because the JSON I'm trying to consume looks like this:
{
"_next": "someValue",
"data": [{/*a collection of objects*/}],
"message": "success"
}
How the heck would I map "_next"?
Use tags to specify the field name in JSON. The JSON object you posted above can be modeled like this:
type Something struct {
Next string `json:"_next"`
Data []interface{} `json:"data"`
Message string `json:"message"`
}
Testing it:
func main() {
var sg Something
if err := json.Unmarshal([]byte(s), &sg); err != nil {
panic(err)
}
fmt.Printf("%+v", sg)
}
const s = `{
"_next": "someValue",
"data": ["one", 2],
"message": "success"
}`
Output (try it on the Go Playground):
{Next:someValue Data:[one 2] Message:success}
Also note that you may also unmarshal into maps or interface{} values, so you don't even have to create structs, but it won't be as convenient using it as the structs:
func main() {
var m map[string]interface{}
if err := json.Unmarshal([]byte(s), &m); err != nil {
panic(err)
}
fmt.Printf("%+v", m)
}
const s = `{
"_next": "someValue",
"data": ["one", 2],
"message": "success"
}`
Output (try it on the Go Playground):
map[_next:someValue data:[one 2] message:success]
Tags will solve your problem.
Hoping it may help others who come here, you can make use of https://mholt.github.io/json-to-go/ to generate Go structs. Paste a JSON structure on the left and the equivalent Go type will be generated to the right, which you can paste into your program.

Golang json Unmarshal "unexpected end of JSON input"

I am working on some code to parse the JSON data from an HTTP response. The code I have looks something like this:
type ResultStruct struct {
result []map[string]string
}
var jsonData ResultStruct
err = json.Unmarshal(respBytes, &jsonData)
The json in the respBytes variable looks like this:
{
"result": [
{
"id": "ID 1"
},
{
"id": "ID 2"
}
]
}
However, err is not nil. When I print it out it says unexpected end of JSON input. What is causing this? The JSON seems to valid. Does this error have something to do with my custom struct?
Thanks in advance!
The unexpected end of JSON input is the result of a syntax error in the JSON input (likely a missing ", }, or ]). The error does not depend on the type of the value that you are decoding to.
I ran the code with the example JSON input on the playground. It runs without error.
The code does not decode anything because the result field is not exported. If you export the result field:
type ResultStruct struct {
Result []map[string]string
}
then the input is decoded as shown in this playground example.
I suspect that you are not reading the entire response body in your application. I suggest decoding the JSON input using:
err := json.NewDecoder(resp.Body).Decode(&jsonData)
The decoder reads directly from the response body.
You can also get this error if you're using json.RawMessage in an unexported field. For example, the following code produces the same error:
package main
import (
"encoding/json"
"fmt"
)
type MyJson struct {
Foo bool `json:"foo"`
bar json.RawMessage `json:"bar"`
}
type Bar struct {
X int `json:"x"`
}
var respBytes = []byte(`
{
"foo": true,
"bar": { "x": 10 }
}`)
func main() {
var myJson MyJson
err := json.Unmarshal(respBytes, &myJson)
if err != nil {
fmt.Println(err)
return
}
myBar := new(Bar)
err = json.Unmarshal(myJson.bar, myBar)
fmt.Println(err)
}
If you export "MyJson.bar" field (e.g. -> "MyJson.Bar", then the code works.
it is not the case here; but if you are getting this error loading json from a file it Will occur if the byte slice for the buffer is not initialized the the byte size of the file. [when you're new like me that happens! ] Since this is the first search result I got it still took some digging to figure out. In this use case the error is a bit misleading.
type GenesisResultStruct []GenesisField
fileinfo, _ := genesis.Stat()
bs := make([]byte, fileinfo.Size())
//bs := []byte {} // wrong!!
_, error := genesis.Read(bs)
if error != nil {
fmt.Println("genesis read error: ", error)
os.Exit(1)
}
var jsonData GenesisResultStruct
eGen = json.Unmarshal(bs, &jsonData)
if eGen != nil {
fmt.Println("genesis unmarshal error: ", eGen)
os.Exit(1)
}
Faced the same issue today.
You can also get this error if the respBytes is nil or there are no brackets [] if you are unmarshalling it to a slice.
In that case, you need to explicitly set respBytes.
As you are unmarshalling it to a slice, brackets [] are expected in the byte-slice
if src == nil {
src = []byte("[]")
}