I have multiple APIs that follow a similar structure on the high level response. It always gives back an answer in that form:
{"data": {"feed":[{...}]}, "success": true}
However, the structure in Feed varies, depending on the concrete API.
I would now like to build an abstract function to process the various APIs. I have the following objects:
type SourceDTO struct { // top level object
Success bool `json:"success"`
Data Feed `json:"data"`
}
type Feed struct {
FeedData []<???> `json:"Feed"`
}
(The real object is more complex, but this shows the idea)
How would be a good way in go to parse this for the different APIs, ut having some common code with some logic based on the high level data (e.g. success)?
EDIT:
I am extending this, to explain more the extend of my question about the "pattern" I am looking for.
I want to create this package that parses the group of APIs. The DTO objects then have to be transferred into some other objects. These 'final' objects are defined in a different package (the entity package) and have then to be persisted.
I am now wondering, how to bring all this together: The 'finaly' entity objects, the transformation functions from DTO to entity, the parsing of the different APIs and their common and different result components.
Where do the transformation functions belong to (package wise)?
EDIT2: Specified FeedData to a slice after digging into the problem (see comments)
You can embed your SourceDTO struct into another struct, like this:
type SourceDTO struct { // top level object
Success bool `json:"success"`
}
type FeedResponse struct {
FeedData YourCustomFeedStruct `json:"feed"`
// Embedded Struct
SourceDTO
}
Now you can access the Success bool from the FeedResponse struct. Also any methods defined on the SourceDTO struct can be accessed from the FeedResponse.
Thanks to #mkopriva for the input for this solution.
In order to have some abstraction in your json unmarshalling it is possible to use interface{} for many use cases.
package main
import (
"encoding/json"
"fmt"
)
type UniversalDTO struct {
Success bool `json:"success"`
Data interface{} `json:"data"`
}
type ConcreteData struct {
Source string `json:"source"`
Site string `json:"site"`
}
func main() {
jsondata := []byte(`{"sucess":"true","data":[{"source":"foo","site":"bar"}]}`)
data := make([]ConcreteData, 0, 10)
dtoToSend := UniversalDTO{Data: &data}
describe(dtoToSend)
describe(dtoToSend.Data)
json.Unmarshal(jsondata, &dtoToSend)
describe(dtoToSend)
describe(dtoToSend.Data)
}
func describe(i interface{}) {
fmt.Printf("(%v, %T)\n", i, i)
}
Test here: https://play.golang.org/p/SSSp_zptMVN
json.Unmarshal expects an object into which the json is being put into. Thus, first we always need an object. Depending on the concrete instance of the target object, the interface{} can be overriden with a concrete struct object (which of course has to be created separately). An important learning here is, that a go interface can also be overridden with a slice. In this way, it is also possible to unmarshal an array into a go object. However, a slice of a struct has to be defined as a slice of pointers to that type.
Related
Please forgive my question, I'm new to Golang and possibly have the wrong approach.
I'm currently implementing a Terraform provider for an internal service.
As probably expected, that requires unmarshalling JSON data in to pre-defined Struct Types, e.g:
type SomeTypeIveDefined struct {
ID string `json:"id"`
Name String `json:"name"`
}
I've got myself in to a situation where I have a lot of duplicate code that looks like this
res := r.(*http.Response)
var tempThing SomeTypeIveDefined
dec := json.NewDecoder(res.Body)
err := dec.Decode(&tempThing)
In an effort to reduce duplication, I decided what I wanted to do was create a function which does the JSON unmarshalling, but takes in the Struct Type as a parameter.
I've trawled through several StackOverflow articles and Google Groups trying to make sense of some of the answers around using the reflect package, but I've not had much success in using it.
My latest attempt was using reflect.StructOf and passing in a set of StructFields, but that still seems to require using myReflectedStruct.Field(0) rather than myReflectedStruct.ID.
I suspect there may be no way until something like Generics are widely available in Golang.
I considered perhaps an interface for the structs which requires implementing an unmarshal method, then I could pass the interface to the function and call the unmarshal method. But then I'm still implementing unmarshal on all the structs anyway.
I'm just wondering what suggestions there may be for achieving what I'm after, please?
Create a helper function with the repeated code. Pass the destination value as a pointer.
func decode(r *http.Repsonse, v interface{}) error {
return json.NewDecoder(res.Body).Decode(v)
}
Call the helper function with a pointer to your thing:
var tempThing SomeTypeIveDefined
err := deocde(r, &tempThing)
You can do this with interfaces:
func decodeResponse(r *http.Response, dest interface{}) error {
dec := json.NewDecoder(r.Body)
return dec.Decode(dest)
}
func handler(...) {
res := r.(*http.Response)
var tempThing SomeTypeIveDefined
if err:=decodeResponse(res,&tempThing); err!=nil {
// handle err
}
...
}
You don't need to implement an unmarshal for the structs, because the stdlib decoder will use reflection to set the struct fields.
How can we store array of different structs into some file and retrieve it back in the same format without losing its properties(methods it provides).
For example: I have data struct A and struct B, both implementing a common interface X {} with some methods.
One options is to write both save and retrieve method to accept the interface X slice.
However the problem is how to unmarshal it back in some generic way which is not tied to my Data struct. i.e., every time I add a new data struct I need not to change my save or retrieve functions to retrieve back the slice of interface X so that its methods can be used independent of data struct.
Example where Unmarshaling throws error :
Go PlayGround Link with a small Example
However the problem is how to unmarshal it back in some generic way which is not tied to my Data struct.
Yes, this is undoable. Redesign.
Intersting question, like Volkel says, undoable as you want it. But as you redesign then there are possibilities. Normally, try to avoid using reflection. It’s not idiomatic, but it’s very powerful in particular cases and it maybe just what you are looking for, especially, often there are not many candidates for structs which are applicable, and you can keep inside your application the unmarshall method generic in its parameter, because the type-discovery procedure is inside the function, and not visible in the function call.
So sometimes, you can solve it by using it inside the UnMarshall function.
if (f.Type() == reflect.TypeOf(Entity{})) {
//reflected type is of type "Entity", you can now Unmarshal
}
It is easier if you think only of the data ...
Your X should not be an interface but a struct so that you can marshall it.
To make the process generic, you can consider X holds a choice
type A struct {
A int64
}
type B struct {
S string
}
type Choice int
const (
XisA Choice = iota
XisB
)
type X struct {
Choice
A
B
}
Before marshalling, you just need to set the choice for each item of your array
a := A{
A: 1,
}
b := B{
S: "2",
}
x1 := X{
Choice: XisA,
A: a,
}
x2 := X{
Choice: XisB,
B: b,
}
x := [2]X{x1, x2}
After unmarshalling, you just need to retrieve the choice you made for each item of the array
for _, item := range decoded {
switch {
case item.Choice == XisA:
println(item.A.GetKey())
case item.Choice == XisB:
println(item.B.GetKey())
}
}
Here is an example: https://play.golang.org/p/RtzF6DmNlKL
I'm new to Golang and using a Go library for processing some webhook events from Github.
I've access to a Deployment's Payload struct defined here:
https://github.com/go-playground/webhooks/blob/v3/github/payload.go#L384
The library parses the webhook JSON payload and constructs that. This is a custom field, i.e. it's a hashmap/dictionary whose fields can be custom set by the client.
So I think it's being defined as an empty struct by the library. How do I extract a specific field called "foo" from this struct?
Now there are certain limitations on what you are able to achieve, but by using the reflect package you can easily check to see if your object is empty or not:
package main
import (
"fmt"
"reflect"
"strconv"
)
type emptiness struct {}
type thing struct {
stuff string
}
func main() {
e := emptiness{}
t := thing{
stuff: "present",
}
fmt.Println(t.stuff)
v := reflect.ValueOf(t)
fmt.Println(strconv.Itoa(v.NumField()))
v = reflect.ValueOf(e)
fmt.Println(strconv.Itoa(v.NumField()))
if v.NumField() == 0 {
// handle your empty object accordingly
}
}
Edit: Forgot to add the runnable example's link. you can play with it and get more info using reflect as well but if you just want to check if it's empty, this will work.
Are there any Go libraries which can tidy up Json output before it is sent to users?
We could unmarshall into a struct and do this manually, but we would like to know if there are any libraries which can make it easier to extract keys into the struct, which we could Marshall and send to the user?
The short answer is not really, due to the way Go handles JSON marshalling and un-marshalling. The common pattern for dealing with your use case is just to define a Response struct.
A classic example would be something like the following:
type User struct {
// fields
}
// Response type used when the user is asking about their own fields
type PrivateUserResponse struct {
// fields with struct tags
}
func (u *User) ToPrivateUserResponse() *PrivateUserResponse { ... }
// Response type used when the user is being listed in a public directory
type PublicUserResponse struct {
// fields with struct tags
}
func (u *User) ToPublicUserResponse() *PublicUserResponse { ... }
Because JSON key configuration is handled by struct tags, a library would be ill-suited to handle the unique business logic cases that arise in dealing with this problem. You might be able to find a code generator that solves this in a more generic way, but I'd recommend just writing the structs yourself - Go favors explicit and clear behavior.
I have a question regarding the golang unmarshalling . I was trying to unmarshal Json array but it is giving nil result for one decoding while it is successful in the other. I don't understand the reason behind it. Is it a mistake in the code or expected?
package main
import "fmt"
import "encoding/json"
type PublicKey struct {
Id int
Key string
}
type KeysResponse struct {
Collection []PublicKey
}
func main() {
keysBody := []byte(`[{"id": 1,"key": "-"},{"id": 2,"key": "-"},{"id": 3,"key": "-"}]`)
keys := make([]PublicKey,0)
json.Unmarshal(keysBody, &keys)//This works
fmt.Printf("%#v\n", keys)
response := KeysResponse{}
json.Unmarshal(keysBody, &response)//This doesn't work
fmt.Printf("%#v\n", response)
}
http://play.golang.org/p/L9xDG26M8-
That's not expected to work. What you have in the json is an array of type PublicKey. The KeysResponse type would be used for json looking like this;
{
"Collection": [{"id": 1,"key": "-"},{"id": 2,"key": "-"},{"id": 3,"key": "-"}]
}
Which is not what you have. If you want the data to be stored in that type I'd recommend the following; response := KeysResponse{keys} on the line after you unmarshal into keys.
To elaborate on that distinction. In the working case the json is just an array with objects inside of it. The json I wrote above is an object which has a single property named Collection which is of type array and the objects in the array are represented by the PublicKey type (objects with an int called id and a string called key). When working on code to unmarshal json, it's helpful to describe the structure using plain English like this, it tells you precisely what types/structures you need in Go to hold the data.