Store and retrieve interfaces in golang - json

How can we store array of different structs into some file and retrieve it back in the same format without losing its properties(methods it provides).
For example: I have data struct A and struct B, both implementing a common interface X {} with some methods.
One options is to write both save and retrieve method to accept the interface X slice.
However the problem is how to unmarshal it back in some generic way which is not tied to my Data struct. i.e., every time I add a new data struct I need not to change my save or retrieve functions to retrieve back the slice of interface X so that its methods can be used independent of data struct.
Example where Unmarshaling throws error :
Go PlayGround Link with a small Example

However the problem is how to unmarshal it back in some generic way which is not tied to my Data struct.
Yes, this is undoable. Redesign.

Intersting question, like Volkel says, undoable as you want it. But as you redesign then there are possibilities. Normally, try to avoid using reflection. It’s not idiomatic, but it’s very powerful in particular cases and it maybe just what you are looking for, especially, often there are not many candidates for structs which are applicable, and you can keep inside your application the unmarshall method generic in its parameter, because the type-discovery procedure is inside the function, and not visible in the function call.
So sometimes, you can solve it by using it inside the UnMarshall function.
if (f.Type() == reflect.TypeOf(Entity{})) {
//reflected type is of type "Entity", you can now Unmarshal
}

It is easier if you think only of the data ...
Your X should not be an interface but a struct so that you can marshall it.
To make the process generic, you can consider X holds a choice
type A struct {
A int64
}
type B struct {
S string
}
type Choice int
const (
XisA Choice = iota
XisB
)
type X struct {
Choice
A
B
}
Before marshalling, you just need to set the choice for each item of your array
a := A{
A: 1,
}
b := B{
S: "2",
}
x1 := X{
Choice: XisA,
A: a,
}
x2 := X{
Choice: XisB,
B: b,
}
x := [2]X{x1, x2}
After unmarshalling, you just need to retrieve the choice you made for each item of the array
for _, item := range decoded {
switch {
case item.Choice == XisA:
println(item.A.GetKey())
case item.Choice == XisB:
println(item.B.GetKey())
}
}
Here is an example: https://play.golang.org/p/RtzF6DmNlKL

Related

Passing a struct Type in Golang?

Please forgive my question, I'm new to Golang and possibly have the wrong approach.
I'm currently implementing a Terraform provider for an internal service.
As probably expected, that requires unmarshalling JSON data in to pre-defined Struct Types, e.g:
type SomeTypeIveDefined struct {
ID string `json:"id"`
Name String `json:"name"`
}
I've got myself in to a situation where I have a lot of duplicate code that looks like this
res := r.(*http.Response)
var tempThing SomeTypeIveDefined
dec := json.NewDecoder(res.Body)
err := dec.Decode(&tempThing)
In an effort to reduce duplication, I decided what I wanted to do was create a function which does the JSON unmarshalling, but takes in the Struct Type as a parameter.
I've trawled through several StackOverflow articles and Google Groups trying to make sense of some of the answers around using the reflect package, but I've not had much success in using it.
My latest attempt was using reflect.StructOf and passing in a set of StructFields, but that still seems to require using myReflectedStruct.Field(0) rather than myReflectedStruct.ID.
I suspect there may be no way until something like Generics are widely available in Golang.
I considered perhaps an interface for the structs which requires implementing an unmarshal method, then I could pass the interface to the function and call the unmarshal method. But then I'm still implementing unmarshal on all the structs anyway.
I'm just wondering what suggestions there may be for achieving what I'm after, please?
Create a helper function with the repeated code. Pass the destination value as a pointer.
func decode(r *http.Repsonse, v interface{}) error {
return json.NewDecoder(res.Body).Decode(v)
}
Call the helper function with a pointer to your thing:
var tempThing SomeTypeIveDefined
err := deocde(r, &tempThing)
You can do this with interfaces:
func decodeResponse(r *http.Response, dest interface{}) error {
dec := json.NewDecoder(r.Body)
return dec.Decode(dest)
}
func handler(...) {
res := r.(*http.Response)
var tempThing SomeTypeIveDefined
if err:=decodeResponse(res,&tempThing); err!=nil {
// handle err
}
...
}
You don't need to implement an unmarshal for the structs, because the stdlib decoder will use reflection to set the struct fields.

Getting all map keys for different content map[string]

I have a general enough function for going through a map[string] and getting all keys:
i := 0
keys := make([]string, len(input))
for k := range input {
keys[i] = k
i++
}
return keys
My problem is I have two different inputs I want to throw in here, a map[string]MyStruct and map[string][][]float64. Whenever I've tried having the input to the func as map[string]interface{}, go resists all my attempts to try to cast the map[string]MyStruct as a map[string]interface{}. Is there a way I can do this without needing to have two functions, one with map[string]MyStruct as input, and one with map[string][][]float64? The contents of the map[string] don't matter at this point, because I'm just trying to get all the keys of them for use later in the code. This needs to be a function that's called; We're using Sonar, and it's set to refuse code duplication, so I can't have this code snippet duplicated.
Until next Go version brings us generics there're several ways to cope with it.
Duplicate code
Use code generation - design some template and then go on build will fill it for you.
Use interface{} as an input type of a function and then use reflection to guess which type was given to a function.
I'm pretty sure in this case general code will be more complicated than 2 separate functions.
func getKeys(input interface{}) []string {
switch inp := input.(type) {
case map[string]MyStruct:
keys := make([]string, 0, len(inp))
for k := range inp {
keys = append(keys, k)
}
return keys
case map[string][]float64:
...
default:
fmt.Printf("I don't know about type %T!\n", v)
}
You cannot use input type map[string]interface{} to place map[string]string or map[string][]string go won't copy each value to new type. You may take the most general type interface{} and then cast it.

How to build an abstract json unmarshaller in go

I have multiple APIs that follow a similar structure on the high level response. It always gives back an answer in that form:
{"data": {"feed":[{...}]}, "success": true}
However, the structure in Feed varies, depending on the concrete API.
I would now like to build an abstract function to process the various APIs. I have the following objects:
type SourceDTO struct { // top level object
Success bool `json:"success"`
Data Feed `json:"data"`
}
type Feed struct {
FeedData []<???> `json:"Feed"`
}
(The real object is more complex, but this shows the idea)
How would be a good way in go to parse this for the different APIs, ut having some common code with some logic based on the high level data (e.g. success)?
EDIT:
I am extending this, to explain more the extend of my question about the "pattern" I am looking for.
I want to create this package that parses the group of APIs. The DTO objects then have to be transferred into some other objects. These 'final' objects are defined in a different package (the entity package) and have then to be persisted.
I am now wondering, how to bring all this together: The 'finaly' entity objects, the transformation functions from DTO to entity, the parsing of the different APIs and their common and different result components.
Where do the transformation functions belong to (package wise)?
EDIT2: Specified FeedData to a slice after digging into the problem (see comments)
You can embed your SourceDTO struct into another struct, like this:
type SourceDTO struct { // top level object
Success bool `json:"success"`
}
type FeedResponse struct {
FeedData YourCustomFeedStruct `json:"feed"`
// Embedded Struct
SourceDTO
}
Now you can access the Success bool from the FeedResponse struct. Also any methods defined on the SourceDTO struct can be accessed from the FeedResponse.
Thanks to #mkopriva for the input for this solution.
In order to have some abstraction in your json unmarshalling it is possible to use interface{} for many use cases.
package main
import (
"encoding/json"
"fmt"
)
type UniversalDTO struct {
Success bool `json:"success"`
Data interface{} `json:"data"`
}
type ConcreteData struct {
Source string `json:"source"`
Site string `json:"site"`
}
func main() {
jsondata := []byte(`{"sucess":"true","data":[{"source":"foo","site":"bar"}]}`)
data := make([]ConcreteData, 0, 10)
dtoToSend := UniversalDTO{Data: &data}
describe(dtoToSend)
describe(dtoToSend.Data)
json.Unmarshal(jsondata, &dtoToSend)
describe(dtoToSend)
describe(dtoToSend.Data)
}
func describe(i interface{}) {
fmt.Printf("(%v, %T)\n", i, i)
}
Test here: https://play.golang.org/p/SSSp_zptMVN
json.Unmarshal expects an object into which the json is being put into. Thus, first we always need an object. Depending on the concrete instance of the target object, the interface{} can be overriden with a concrete struct object (which of course has to be created separately). An important learning here is, that a go interface can also be overridden with a slice. In this way, it is also possible to unmarshal an array into a go object. However, a slice of a struct has to be defined as a slice of pointers to that type.

Unmarshal inconsistent JSON

I have JSON (that I cannot control) like this:
{
"foo1":{
"a":{
"up":10,
"down":5
}
},
"foo2":{
"a":{
"up":1,
"down":1
}
},
"bar":{
"up":11,
"down":6
}
}
"foo1" and "foo2" are dynamic.
How can I properly unmarshal this structure in go?
It would be okay if I could just tell go to not try to deserialize "bar" (the inconsistent property).
Go will by default ignore fields unspecified in the struct you unmarshal into.
In this case, your structure would be set up like this:
type NestedProp2 struct {
Up int
Down int
}
type NestedProp struct {
A NestedProp2
}
type Prop struct {
Foo1 NestedProp
Foo2 NestedProp
}
When you call the the json.Unmarshal function, the extra property will not be deserialized:
var prop Prop
err := json.Unmarshal(jsonBlob, &prop)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", prop)
So you get the following output:
{Foo1:{A:{Up:10 Down:5}} Foo2:{A:{Up:1 Down:1}}}
You can see it in action here.
You said:
I have JSON (that I cannot control)
So to what extent you could control? Here I could provide you with some scenario, and hope some of them match your purpose :)
Remember the general rule first:
In Golang, if a JSON key failed to find a matched field in struct, it will not be unmarshalled.
This means, for a key name in a JSON struct, when unmarshalling, it will look for a field in a golang struct at the same level with the same name case-insensitively. If this search failed, this key won't be unmarshalled.
For example, a key named foo1 will look for a field name foo1 in a golang struct at the same indent level. However it also matches with Foo1 or FoO1, since this matching is case-insensitive.
Remember, you could use field tag to specify the field name as well. Please take a look at the official page.
The value of some of the JSON fields are not consistent, and they could be ignored.
This is the case #gnalck solved in his answer. According to the general rule, if those inconsistent field failed to find a match, they will not be unmarshalled. Therefore, just don't put those inconsistent fields in the struct and you will be fine.
The value of some of the JSON fields are not consistent, but they could not be ignored.
In this case, #gnalck failed since those fields could not be ignored. Now a better way is to unmarshal bar into a json.RawMessage, so that you could unmarshal later.
The keys of the JSON object is undetermined, and their value is undetermined as well.
In this case, we could unmarshal the whole JSON object into a map[string]json.RawMessage, and unmarshal each fields later. When unmarshalling to a map, you could iterate through the map to get all the fields, and unmarshal them into a proper struct later.

Is it possible to merge/merge multiple structs into one dynamically (using reflect)?

I need to build a function which merges responses from multiple other functions into one (struct). Currently I'm thinking that the only way would be to create a map[string]interface{} and then range over the fields of structs I need to merge and assign field name -> value as key, values in the map. Is there any other way? I basically just need to embed two structs into one.
I'm not sure if this answers your question but you can definitely embed structs into one struct and access the properties directly. You wouldn't need to use reflection and might I add that its often the case that reflection is frowned upon since its more preferable to be explicit; reflection is often implicit.
Here is some code, that hopefully addresses what you wanted:
type Foo struct {
Bar
Baz
}
type Bar struct {
BarValue string
}
type Baz struct {
BazValue string
}
func main() {
test := Foo{Bar: Bar{BarValue: "bar"}, Baz: Baz{BazValue: "baz"}}
fmt.Println(test.BarValue)
fmt.Println(test.BazValue)
}