I am trying to use the libvirt-go api binding and following the example snippet here, have so far populated the domcfg variable thus:
domcfg := &libvirtxml.Domain{
Type: "kvm",
Name: "demo",
Memory: &libvirtxml.DomainMemory{
Value: 4096,
Unit: "MB",
DumpCore: "on"},
VCPU: &libvirtxml.DomainVCPU{
Value: 1},
CPU: &libvirtxml.DomainCPU{
Mode: "host-model"},
Devices: &libvirtxml.DomainDeviceList{
Disks: []libvirtxml.DomainDisk{
Source: &libvirtxml.DomainDiskSource{
File: &DomainDiskSourceFile{
File: "./vm.qcow2"}},
BackingStore: &DomainDiskBackingStore{
BackingStore: &DomainDiskBackingStore{
}
}}}
I am not sure I am completely on the right path so, is this the right way to define the variable before marshalling to xml? Also I find the struct defined at https://godoc.org/github.com/libvirt/libvirt-go-xml#DomainDiskBackingStore specifically the BackingStore field which seems to be a pointer to itself, is it possible to embed a field in a struct which is the struct definition itself
Yes, that is the right approach to defining a XML config inline in Go. To define the domain you can then do
xml, err := domcfg.Marshal()
if err != nil {
...
}
dom, err := conn.DomainDefineXML(xml)
if err != nil {
...
}
The disk backing store is a way to provide information about the full disk chain. Generally you can leave this set to nil because libvirt will automatically detect the chain for you. There can be multiple levels of backing chain, hence why the struct contains a field with its own type again.
Related
I'm new to golang generics and have the following setup.
I've gathered loads of different kinds of reports.
Each report has enclosing fields
So I wrapped it in a ReportContainerImpl
I've used a type argument of [T Reportable] where the Reportable is defined as follows
type Reportable interface {
ExportDataPointReport | ImportDataPointReport | MissingDataPointReport | SensorThresoldReport
}
Each of the type in the type constraint is structs that is to be embedded in the container.
type ReportContainerImpl[T Reportable] struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body T `json:"body"`
}
I use a discriminator ReportType to determine the concrete type when Unmarshal.
type ReportType string
const (
ReportTypeExportDataPointReport ReportType = "ExportDataPointReport"
ReportTypeImportDataPointReport ReportType = "ImportDataPointReport"
ReportTypeMissingDataPointReport ReportType = "MissingDataPointReport"
ReportTypeSensorThresoldReport ReportType = "SensorThresoldReport"
)
Since go does not support type assertion for struct (only interfaces) it is not possible to cast the type when Unmarshal. Also go does not support pointer to the "raw" generic type. Hence, I've created a interface that the ReportContainerImpl implements.
type ReportContainer interface {
GetLocationID() string
GetProvider() string
GetReportType() ReportType
GetBody() interface{}
}
The problem I then get is that I cannot do type constrains on the return type in any form or shape and am back at "freetext semantics" on the GetBody() function to allow for type assertion when Unmarshal is done.
container, err := UnmarshalReportContainer(data)
if rep, ok := container.GetBody().(ExportDataPointReport); ok {
// Use the ReportContainerImpl[ExportDataPointReport] here...
}
Maybe I'm getting this wrong? - but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Do you have a better suggestion how to solve this in a type (safer) way?
Cheers,
Mario :)
For completeness I add the UnmarshalReportContainer here
func UnmarshalReportContainer(data []byte) (ReportContainer, error) {
type Temp struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body *json.RawMessage `json:"body"`
}
var temp Temp
err := json.Unmarshal(data, &temp)
if err != nil {
return nil, err
}
switch temp.ReportType {
case ReportTypeExportDataPointReport:
var report ExportDataPointReport
err := json.Unmarshal(*temp.Body, &report)
return &ReportContainerImpl[ExportDataPointReport]{
LocationID: temp.LocationID,
Provider: temp.Provider,
ReportType: temp.ReportType,
Body: report,
}, err
// ...
}
}
but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Precisely.
The concrete types needed to instantiate some generic type or function like ReportContainerImpl or UnmarshalReportContainer must be known at compile time, when you write the code. JSON unmarshalling instead occurs at run-time, when you have the byte slice populated with the actual data.
To unmarshal dynamic JSON based on some discriminatory value, you still need a switch.
Do you have a better suggestion how to solve this in a type (safer) way?
Just forgo parametric polymorphism. It's not a good fit here. Keep the code you have now with json.RawMessage, unmarshal the dynamic data conditionally in the switch and return the concrete structs that implement ReportContainer interface.
As a general solution — if, and only if, you can overcome this chicken-and-egg problem and make type parameters known at compile time, you can write a minimal generic unmarshal function like this:
func unmarshalAny[T any](bytes []byte) (*T, error) {
out := new(T)
if err := json.Unmarshal(bytes, out); err != nil {
return nil, err
}
return out, nil
}
This is only meant to illustrate the principle. Note that json.Unmarshal already accepts any type, so if your generic function actually does nothing except new(T) and return, like in my example, it is no different than "inlining" the entire thing as if unmarshalAny didn't exist.
v, err := unmarshalAny[SomeType](src)
functionally equivalent as
out := &SomeType{}
err := json.Unmarshal(bytes, out)
If you plan to put more logic in unmarshalAny, its usage may be warranted. Your mileage may vary; in general, don't use type parameters when it's not actually necessary.
Scenario:
Consider the following is the JSON :
{
"Bangalore_City": "35_Temperature",
"NewYork_City": "31_Temperature",
"Copenhagen_City": "29_Temperature"
}
If you notice, the data is structured in such a way that there is no hard-coded keys mentioning City/Temperature its basically just values.
Issue: I am not able to parse any JSON which is dynamic.
Question: Could anyone have found solution for this kind of JSON parsing? I tried go-simplejson, gabs & default encoding/json but no luck.
Note:
The above JSON is just for sample. And there are lot of applications which are using the current API, So I do not want to change how the data is structured. I mean I can't change to something as follows:
[{
"City_Name":"Bangalore",
"Temperature": "35"
},...]
Then I can define struct
type TempData struct {
City_Name string
Temperature string
}
You can unmarshal into a map[string]string for example:
m := map[string]string{}
err := json.Unmarshal([]byte(input), &m)
if err != nil {
panic(err)
}
fmt.Println(m)
Output (wrapped):
map[Bangalore_City:35_Temperature NewYork_City:31_Temperature
Copenhagen_City:29_Temperature]
Try it on the Go Playground.
This way no matter what the keys or values are, you will have all pairs in a map which you can print or loop over.
Also note that although your example contained only string values, but if the value type is varying (e.g. string, numbers etc.), you may use interface{} for the value type, in which case your map would be of type map[string]interface{}.
Also note that I created a library to easily work with such dynamic objects which may be a great help in these cases: github.com/icza/dyno.
Standard encoding/json is good for the majority of use cases, but it may be quite slow comparing to alternative solutions. If you need performance, try using fastjson. It parses arbitrary JSONs without the need for creating structs or maps matching the JSON schema.
See the example code below. It iterates over all the (key, value) pairs of the JSON object:
var p fastjson.Parser
v, err := p.Parse(input)
if err != nil {
log.Fatal(err)
}
// Visit all the items in the top object
v.GetObject().Visit(func(k []byte, v *fastjson.Value) {
fmt.Printf("key=%s, value=%s\n", k, v)
// for nested objects call Visit again
if string(k) == "nested" {
v.GetObject().Visit(func(k []byte, v *fastjson.Value) {
fmt.Printf("nested key=%s, value=%s\n", k, v)
})
}
})
Just to add a general answer to how any valid JSON can be parsed; var m interface{} works for all types. That includes map (which OP asked for) arrays, strings, numbers and nested structures.
var m interface{}
err := json.Unmarshal([]byte(input), &m)
if err != nil {
panic(err)
}
fmt.Println(m)
Also consider using gabs package https://github.com/Jeffail/gabs
"Gabs is a small utility for dealing with dynamic or unknown JSON structures in Go"
I'm new in golang development and have some question regarding something related to this question.
As a learning exercise, I'm trying to create a simple library to handle json based configuration file. As a configuration file to be used for more then one app, it should be able to handle different parameters. Then I have created a type struct Configuration that has the filename and a data interface. Each app will have a struct based on its configuration needs.
In the code bellow, I put all together (lib and "main code") and the "TestData struct" is the "app parameters".
If it doesn't exists, it will set a default values and create the file, and it is working. But when I try to read the file. I try to decode the json and put it back into the data interface. But it is giving me an error and I couldn't figure out how to solve this. Can someone help on this?
[updated] I didn't put the targeted code before, because I though that it would be easier to read in in all as a single program. Bellow is the 'targeted code' for better view of the issue.
As I will not be able to use the TestData struct inside the library, since it will change from program to program, the only way to handle this was using interface. Is there a better way?
library config
package config
import (
"encoding/json"
"fmt"
"os"
)
// Base configuration struct
type Configuration struct {
Filename string
Data interface{}
}
func (c *Configuration) Create(cData *Configuration) bool {
cFile, err := os.Open(cData.Filename)
defer cFile.Close()
if err == nil {
fmt.Println("Error(1) trying to create a configuration file. File '", cData.Filename, "' may already exist...")
return false
}
cFile, err = os.Create(cData.Filename)
if err != nil {
fmt.Println("Error(2) trying to create a configuration file. File '", cData.Filename, "' may already exist...")
return false
}
buffer, _ := json.MarshalIndent(cData.Data, "", "")
cFile.Write(buffer)
return true
}
func (c *Configuration) Read(cData *Configuration) bool {
cFile, err := os.Open(cData.Filename)
defer cFile.Close()
if err != nil {
fmt.Println("Error(1) trying to read a configuration file. File '", cData.Filename, "' may not already exist...")
return false
}
jConfig := json.NewDecoder(cFile)
jerr := jConfig.Decode(&cData.Data)
if jerr != nil {
panic(jerr)
}
return true
}
program using library config
package main
import (
"fmt"
"./config"
)
// struct basic para configuração
type TestData struct {
URL string
Port string
}
func main() {
var Config config.Configuration
Config.Filename = "config.json"
if !Config.Read(&Config) {
Config.Data = TestData{"http", "8080"}
Config.Create(&Config)
}
fmt.Println(Config.Data)
TestData1 := &TestData{}
TestData1 = Config.Data.(*TestData) // error, why?
fmt.Println(TestData1.URL)
}
NEW UPDATE:
I have made some changes after JimB comment about I'm not clear about some concepts and I tried to review it. Sure many things aren't clear for me yet unfortunately. The "big" understanding I believe I got, but what mess my mind up is the "ins" and "outs" of values and formats and pointers, mainly when it goes to other libraries. I'm not able yet to follow the "full path" of it.
Yet, I believe I had some improvement on my code.
I think that I have corrected some points, but still have some big questions:
I stopped sending "Configuration" as a parameter as all "data" were already there as they are "thenselfs" in the instance. Right?
Why do I have use reference in the line 58 (Config.Data = &TestData{})
Why to I have to use pointer in the line 64 (tmp := Config.Data.(*TestData)
Why I CANNOT use reference in line 69 (Config.Data = tmp)
Thanks
The reason you are running into an error is because you are trying to decode into an interface{} type. When dealing with JSON objects, they are decoded by the encoding/json package into map[string]interface{} types by default. This is causing the type assertion to fail since the memory structure for a map[string]interface{} is much different than that of a struct.
The better way to do this is to make your TestData struct the expected data format for your Configuration struct:
// Base configuration struct
type Configuration struct {
Filename string
Data *TestData
}
Then when Decoding the file data, the package will unmarshal the data into the fields that match the closest with the data it finds.
If you need more control over the data unmarshaling process, you can dictate which JSON fields get decoded into which struct members by using struct tags. You can read more about the json struct tags available here: https://golang.org/pkg/encoding/json/#Marshal
You are trying to assert that Config.Data is of type *TestData, but you're assigning it to TestData{"http", "8080"} above. You can take the address of a composite literal to create a pointer:
Config.Data = &TestData{"http", "8080"}
If your config already exsits, your Read method is going to fill in the Data field with the a default json data type, probably a map[string]interface{}. If you assign a pointer of the correct type to Data first, it will decode into the expected type.
Config.Data = &TestData{}
Ans since Data is an interface{}, you do not want to ever use a pointer to that value, so don't use the & operator when marshaling and unmarshaling.
Considering the following input:
"object": {
"array": [
{
"element_field_1": some_value_1,
"element_field_2": some_value_1,
... // More unknown fields
},
...
],
"object_field": foo,
... // Some more unknown fields
}
I need to iterate over every element of the array, modify fields 1 and 2 and then output the JSON object. Here's what I have for now, but it is far from being valid Go code:
func handler(w http.ResponseWriter, r *http.Request) {
// Transform the request's body to an interface
j := getDecodedJSON(r)
// Iterate over every element of the array
for i := 0; i < len(j["object"]["array"]); i++ {
rewrite(j["object"]["array"][i])
}
// Encoding back to JSON shouldn't be a problem
}
func getDecodedJSON(r *http.Request) map[string]interface{} {
dec := json.NewDecoder(r.Body)
var j map[string]interface{}
if err := dec.Decode(&j); err != nil {
log.Fatal(err)
}
return j
}
func rewrite(element *map[string]interface{}) {
element["element_field_1"], element["element_field_2"] = lookupValues(element)
}
Basically the error is:
invalid operation: j["object"]["array"] \
(type interface {} does not support indexing)
but of course there's a more conceptual mistake on my approach.
Writing a struct that details the content of the input isn't really an option, since I don't know the JSON keys beforehand.
How can I do this "the Go way"?
EDIT: This is the actual use case:
I have two web services that need a "translator" between them.
Service 1 makes a request to the translator, where a couple of fields are modified, everything else is left intact.
Then the translator takes the modified JSON and replicates the request to service 2.
In other words, this translator acts like a man in the middle for both services. Go seems to be a good option for this given its fast startup times and fast JSON handling.
I don't think it makes sense to detail every JSON field in a Go struct, since I only need to change a few fields. I'm willing to make a tradeoff in efficiency because of reflection (parsing to a map[string]interface{} should be slower than using a full-blown struct), in exchange of making the code more generic to variations of the JSON input.
Change
j["object"]["array"][i]
to
j["object"].(map[string]interface{})["array"].([]interface{})[i]
I have a struct like this:
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
But even if the instance of MyStruct is entirely empty (meaning, all values are default), it's being serialized as:
"data":{}
I know that the encoding/json docs specify that "empty" fields are:
false, 0, any nil pointer or interface value, and any array,
slice, map, or string of length zero
but with no consideration for a struct with all empty/default values. All of its fields are also tagged with omitempty, but this has no effect.
How can I get the JSON package to not marshal my field that is an empty struct?
As the docs say, "any nil pointer." -- make the struct a pointer. Pointers have obvious "empty" values: nil.
Fix - define the type with a struct pointer field:
type Result struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
Then a value like this:
result := Result{}
Will marshal as:
{}
Explanation: Notice the *MyStruct in our type definition. JSON serialization doesn't care whether it is a pointer or not -- that's a runtime detail. So making struct fields into pointers only has implications for compiling and runtime).
Just note that if you do change the field type from MyStruct to *MyStruct, you will need pointers to struct values to populate it, like so:
Data: &MyStruct{ /* values */ }
As #chakrit mentioned in a comment, you can't get this to work by implementing json.Marshaler on MyStruct, and implementing a custom JSON marshalling function on every struct that uses it can be a lot more work. It really depends on your use case as to whether it's worth the extra work or whether you're prepared to live with empty structs in your JSON, but here's the pattern I use applied to Result:
type Result struct {
Data MyStruct
Status string
Reason string
}
func (r Result) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}{
Data: &r.Data,
Status: r.Status,
Reason: r.Reason,
})
}
func (r *Result) UnmarshalJSON(b []byte) error {
decoded := new(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
})
err := json.Unmarshal(b, decoded)
if err == nil {
r.Data = decoded.Data
r.Status = decoded.Status
r.Reason = decoded.Reason
}
return err
}
If you have huge structs with many fields this can become tedious, especially changing a struct's implementation later, but short of rewriting the whole json package to suit your needs (not a good idea), this is pretty much the only way I can think of getting this done while still keeping a non-pointer MyStruct in there.
Also, you don't have to use inline structs, you can create named ones. I use LiteIDE with code completion though, so I prefer inline to avoid clutter.
Data is an initialized struct, so it isn't considered empty because encoding/json only looks at the immediate value, not the fields inside the struct.
Unfortunately, returning nil from json.Marshaler doesn't currently work:
func (_ MyStruct) MarshalJSON() ([]byte, error) {
if empty {
return nil, nil // unexpected end of JSON input
}
// ...
}
You could give Result a marshaler as well, but it's not worth the effort.
The only option, as Matt suggests, is to make Data a pointer and set the value to nil.
There is an outstanding Golang proposal for this feature which has been active for over 4 years, so at this point, it is safe to assume that it will not make it into the standard library anytime soon. As #Matt pointed out, the traditional approach is to convert the structs to pointers-to-structs. If this approach is infeasible (or impractical), then an alternative is to use an alternate json encoder which does support omitting zero value structs.
I created a mirror of the Golang json library (clarketm/json) with added support for omitting zero value structs when the omitempty tag is applied. This library detects zeroness in a similar manner to the popular YAML encoder go-yaml by recursively checking the public struct fields.
e.g.
$ go get -u "github.com/clarketm/json"
import (
"fmt"
"github.com/clarketm/json" // drop-in replacement for `encoding/json`
)
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
j, _ := json.Marshal(&Result{
Status: "204",
Reason: "No Content",
})
fmt.Println(string(j))
// Note: `data` is omitted from the resultant json.
{
"status": "204"
"reason": "No Content"
}