What is an idiomatic golang way to dump the struct into a csv file provided? I am inside a func where my struct is passed as interface{}:
func decode_and_csv(my_response *http.Response, my_struct interface{})
Why interface{}? - reading data from JSON and there could be a few different structs returned, so trying to write a generic enough function.
an example of my types:
type Location []struct {
Name string `json: "Name"`
Region string `json: "Region"`
Type string `json: "Type"`
}
It would be a lot easier if you used a concrete type. You'll probably want to use the encoding/csv package, here is a relevant example; https://golang.org/pkg/encoding/csv/#example_Writer
As you can see, the Write method is expecting a []string so in order to generate this, you'll have to either 1) provide a helper method or 2) reflect my_struct. Personally, I prefer the first method but it depends on your needs. If you want to go route two you can get all the fields on the struct an use them as the column headers, then iterate the fields getting the value for each, use append in that loop to add them to a []string and then pass it to Write out side of the loop.
For the first option, I would define a ToSlice or something on each type and then I would make an interface call it CsvAble that requires the ToSlice method. Change the type in your method my_struct CsvAble instead of using the empty interface and then you can just call ToSlice on my_struct and pass the return value into Write. You could have that return the column headers as well (meaning you would get back a [][]string and need to iterate the outer dimension passing each []string into Write) or you could require another method to satisfy the interface like GetHeaders that returns a []string which is the column headers. If that were the case your code would look something like;
w := csv.NewWriter(os.Stdout)
headers := my_struct.GetHeaders()
values := my_struct.ToSlice()
if err := w.Write(headers); err != nil {
//write failed do something
}
if err := w.Write(values); err != nil {
//write failed do something
}
If that doesn't make sense let me know and I can follow up with a code sample for either of the two approaches.
Related
There is a nested struct that I want to create from a flat JSON that includes another struct:
type Todo struct {
Todo_id int `json:"todo_id" db:"todo_id"`
Todo_name string `json:"todo_name" db:"todo_name"`
User_id int `json:"user_id" db:"user_id"`
Subs []Sub `json:"subs" db:"subs"`
Times Parsed_Time `json:"times" db:"times"`
}
When I unmarshal a JSON I receive a "missing destination name deadline" error because Deadline is inside a Parsed_Time struct. Is there a way to custom unmarshal JSON so that parts of the JSON would be omitted without error? I would like to separately create a Todo struct with an empty Times, then run Unmarshal again to extract deadline and the rest of timestamps separately into another struct. This is to avoid making two separate Get requests to a database.
Yes - you probably already know that if a type implements json.Unmarshaler, it will be used when json.Unmarshal() is called with that type as the second parameter. The problem that often occurs is the need to unmarshal the receiver's type as part of the custom unmarshal code. This can be overcome in several ways, the most common of which is to use a local type to do the unmarshaling. You can save a lot of duplicate code with the judicious use of a type alias though.
I've updated your code above to stub out the types represented by Todo's fields as follows:
type Sub int
type ParsedTime struct {
Deadline time.Time
Created time.Time
}
type Todo struct {
ID int `json:"todo_id" db:"todo_id"`
Name string `json:"todo_name" db:"todo_name"`
UserID int `json:"user_id" db:"user_id"`
Subs []Sub `json:"subs" db:"subs"`
Times ParsedTime `json:"-" db:"times"`
}
Note that the only change of relevance is to ignore the Times field when `json.Unmarshal is called. I only changed the field names to get my IDE's linter to shut up! With these types, we can define a custom unmarshaler as follows:
func (t *Todo) UnmarshalJSON(data []byte) error {
type TodoJSON Todo
todo := struct {
*TodoJSON
Deadline string `json:"deadline"`
}{
TodoJSON: (*TodoJSON)(t),
}
if err := json.Unmarshal(data, &todo); err != nil {
return err
}
deadline, err := time.Parse(time.RFC3339, todo.Deadline)
if err != nil {
return err
}
t.Times.Deadline = deadline
return nil
}
There are two key techniques used in this code. First, using a type alias eliminates the infinite recursion that would occur if Todo was used directly. Second, creating a local type that embeds *Todo eliminates the need to completely retype the Todo type's fields - only the desired Deadline field needs to be added. I also assumed that Deadline was a time.Time to show that this code also allows the field to be processed before it's assigned (time.Parse()).
I'm new to golang generics and have the following setup.
I've gathered loads of different kinds of reports.
Each report has enclosing fields
So I wrapped it in a ReportContainerImpl
I've used a type argument of [T Reportable] where the Reportable is defined as follows
type Reportable interface {
ExportDataPointReport | ImportDataPointReport | MissingDataPointReport | SensorThresoldReport
}
Each of the type in the type constraint is structs that is to be embedded in the container.
type ReportContainerImpl[T Reportable] struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body T `json:"body"`
}
I use a discriminator ReportType to determine the concrete type when Unmarshal.
type ReportType string
const (
ReportTypeExportDataPointReport ReportType = "ExportDataPointReport"
ReportTypeImportDataPointReport ReportType = "ImportDataPointReport"
ReportTypeMissingDataPointReport ReportType = "MissingDataPointReport"
ReportTypeSensorThresoldReport ReportType = "SensorThresoldReport"
)
Since go does not support type assertion for struct (only interfaces) it is not possible to cast the type when Unmarshal. Also go does not support pointer to the "raw" generic type. Hence, I've created a interface that the ReportContainerImpl implements.
type ReportContainer interface {
GetLocationID() string
GetProvider() string
GetReportType() ReportType
GetBody() interface{}
}
The problem I then get is that I cannot do type constrains on the return type in any form or shape and am back at "freetext semantics" on the GetBody() function to allow for type assertion when Unmarshal is done.
container, err := UnmarshalReportContainer(data)
if rep, ok := container.GetBody().(ExportDataPointReport); ok {
// Use the ReportContainerImpl[ExportDataPointReport] here...
}
Maybe I'm getting this wrong? - but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Do you have a better suggestion how to solve this in a type (safer) way?
Cheers,
Mario :)
For completeness I add the UnmarshalReportContainer here
func UnmarshalReportContainer(data []byte) (ReportContainer, error) {
type Temp struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body *json.RawMessage `json:"body"`
}
var temp Temp
err := json.Unmarshal(data, &temp)
if err != nil {
return nil, err
}
switch temp.ReportType {
case ReportTypeExportDataPointReport:
var report ExportDataPointReport
err := json.Unmarshal(*temp.Body, &report)
return &ReportContainerImpl[ExportDataPointReport]{
LocationID: temp.LocationID,
Provider: temp.Provider,
ReportType: temp.ReportType,
Body: report,
}, err
// ...
}
}
but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Precisely.
The concrete types needed to instantiate some generic type or function like ReportContainerImpl or UnmarshalReportContainer must be known at compile time, when you write the code. JSON unmarshalling instead occurs at run-time, when you have the byte slice populated with the actual data.
To unmarshal dynamic JSON based on some discriminatory value, you still need a switch.
Do you have a better suggestion how to solve this in a type (safer) way?
Just forgo parametric polymorphism. It's not a good fit here. Keep the code you have now with json.RawMessage, unmarshal the dynamic data conditionally in the switch and return the concrete structs that implement ReportContainer interface.
As a general solution — if, and only if, you can overcome this chicken-and-egg problem and make type parameters known at compile time, you can write a minimal generic unmarshal function like this:
func unmarshalAny[T any](bytes []byte) (*T, error) {
out := new(T)
if err := json.Unmarshal(bytes, out); err != nil {
return nil, err
}
return out, nil
}
This is only meant to illustrate the principle. Note that json.Unmarshal already accepts any type, so if your generic function actually does nothing except new(T) and return, like in my example, it is no different than "inlining" the entire thing as if unmarshalAny didn't exist.
v, err := unmarshalAny[SomeType](src)
functionally equivalent as
out := &SomeType{}
err := json.Unmarshal(bytes, out)
If you plan to put more logic in unmarshalAny, its usage may be warranted. Your mileage may vary; in general, don't use type parameters when it's not actually necessary.
Please forgive my question, I'm new to Golang and possibly have the wrong approach.
I'm currently implementing a Terraform provider for an internal service.
As probably expected, that requires unmarshalling JSON data in to pre-defined Struct Types, e.g:
type SomeTypeIveDefined struct {
ID string `json:"id"`
Name String `json:"name"`
}
I've got myself in to a situation where I have a lot of duplicate code that looks like this
res := r.(*http.Response)
var tempThing SomeTypeIveDefined
dec := json.NewDecoder(res.Body)
err := dec.Decode(&tempThing)
In an effort to reduce duplication, I decided what I wanted to do was create a function which does the JSON unmarshalling, but takes in the Struct Type as a parameter.
I've trawled through several StackOverflow articles and Google Groups trying to make sense of some of the answers around using the reflect package, but I've not had much success in using it.
My latest attempt was using reflect.StructOf and passing in a set of StructFields, but that still seems to require using myReflectedStruct.Field(0) rather than myReflectedStruct.ID.
I suspect there may be no way until something like Generics are widely available in Golang.
I considered perhaps an interface for the structs which requires implementing an unmarshal method, then I could pass the interface to the function and call the unmarshal method. But then I'm still implementing unmarshal on all the structs anyway.
I'm just wondering what suggestions there may be for achieving what I'm after, please?
Create a helper function with the repeated code. Pass the destination value as a pointer.
func decode(r *http.Repsonse, v interface{}) error {
return json.NewDecoder(res.Body).Decode(v)
}
Call the helper function with a pointer to your thing:
var tempThing SomeTypeIveDefined
err := deocde(r, &tempThing)
You can do this with interfaces:
func decodeResponse(r *http.Response, dest interface{}) error {
dec := json.NewDecoder(r.Body)
return dec.Decode(dest)
}
func handler(...) {
res := r.(*http.Response)
var tempThing SomeTypeIveDefined
if err:=decodeResponse(res,&tempThing); err!=nil {
// handle err
}
...
}
You don't need to implement an unmarshal for the structs, because the stdlib decoder will use reflection to set the struct fields.
I have a struct VideoInfo that has a key in it called embedCode. The API I am querying returns the embed code as embed_code. During unmarshalling the response how do I ensure embed_code goes into embedCode?
Also is there an easy way to take a large json string and automatically turn it into a struct, or can one only use a map?
With respect to remapping the field names use the corresponding annotation in the structure declaration:
type VideoInfo struct {
EmbedCode string `json:"embed_code"`
}
The marshaller/un-marshaller will only process public field, so you need to capitalise the field name.
With respect to converting the whole structure, yes it is easy. Declare an instance to un-marshal into and pass a reference to the json.Unmarshal method (from a test):
data, _ := json.Marshal(request)
var resp response.VideoInfo
if err := json.Unmarshal(data, &resp); err != nil {
t.Errorf("unexpected error, %v", err)
}
At first, struct's field must be start from capital letter to be public. So you need something like that:
type VideoInfo struct {
EmbedCode string `json:"embed_code"`
}
And look at documentation for more info.
I have a struct like this:
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
But even if the instance of MyStruct is entirely empty (meaning, all values are default), it's being serialized as:
"data":{}
I know that the encoding/json docs specify that "empty" fields are:
false, 0, any nil pointer or interface value, and any array,
slice, map, or string of length zero
but with no consideration for a struct with all empty/default values. All of its fields are also tagged with omitempty, but this has no effect.
How can I get the JSON package to not marshal my field that is an empty struct?
As the docs say, "any nil pointer." -- make the struct a pointer. Pointers have obvious "empty" values: nil.
Fix - define the type with a struct pointer field:
type Result struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
Then a value like this:
result := Result{}
Will marshal as:
{}
Explanation: Notice the *MyStruct in our type definition. JSON serialization doesn't care whether it is a pointer or not -- that's a runtime detail. So making struct fields into pointers only has implications for compiling and runtime).
Just note that if you do change the field type from MyStruct to *MyStruct, you will need pointers to struct values to populate it, like so:
Data: &MyStruct{ /* values */ }
As #chakrit mentioned in a comment, you can't get this to work by implementing json.Marshaler on MyStruct, and implementing a custom JSON marshalling function on every struct that uses it can be a lot more work. It really depends on your use case as to whether it's worth the extra work or whether you're prepared to live with empty structs in your JSON, but here's the pattern I use applied to Result:
type Result struct {
Data MyStruct
Status string
Reason string
}
func (r Result) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}{
Data: &r.Data,
Status: r.Status,
Reason: r.Reason,
})
}
func (r *Result) UnmarshalJSON(b []byte) error {
decoded := new(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
})
err := json.Unmarshal(b, decoded)
if err == nil {
r.Data = decoded.Data
r.Status = decoded.Status
r.Reason = decoded.Reason
}
return err
}
If you have huge structs with many fields this can become tedious, especially changing a struct's implementation later, but short of rewriting the whole json package to suit your needs (not a good idea), this is pretty much the only way I can think of getting this done while still keeping a non-pointer MyStruct in there.
Also, you don't have to use inline structs, you can create named ones. I use LiteIDE with code completion though, so I prefer inline to avoid clutter.
Data is an initialized struct, so it isn't considered empty because encoding/json only looks at the immediate value, not the fields inside the struct.
Unfortunately, returning nil from json.Marshaler doesn't currently work:
func (_ MyStruct) MarshalJSON() ([]byte, error) {
if empty {
return nil, nil // unexpected end of JSON input
}
// ...
}
You could give Result a marshaler as well, but it's not worth the effort.
The only option, as Matt suggests, is to make Data a pointer and set the value to nil.
There is an outstanding Golang proposal for this feature which has been active for over 4 years, so at this point, it is safe to assume that it will not make it into the standard library anytime soon. As #Matt pointed out, the traditional approach is to convert the structs to pointers-to-structs. If this approach is infeasible (or impractical), then an alternative is to use an alternate json encoder which does support omitting zero value structs.
I created a mirror of the Golang json library (clarketm/json) with added support for omitting zero value structs when the omitempty tag is applied. This library detects zeroness in a similar manner to the popular YAML encoder go-yaml by recursively checking the public struct fields.
e.g.
$ go get -u "github.com/clarketm/json"
import (
"fmt"
"github.com/clarketm/json" // drop-in replacement for `encoding/json`
)
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
j, _ := json.Marshal(&Result{
Status: "204",
Reason: "No Content",
})
fmt.Println(string(j))
// Note: `data` is omitted from the resultant json.
{
"status": "204"
"reason": "No Content"
}