So I would like to fill any struct via the StructScan method and so read any data I get from the db into the regarding struct I feed the test function.
This script doesn't give any compile error (if you implement the other stuff like a db connection and so on) but still the StructScan method returns an error and tells me that it expects a slice of structs.
How do I create a slice of structs that I don't know the type of?
Thanks for any advice.
package main
import (
"database/sql"
"github.com/jmoiron/sqlx"
)
var db *sql.DB
type A struct {
Name string `db:"name"`
}
type B struct {
Name string `db:"name"
}
func main() {
testA := []A{}
testB := []B{}
test(testA, "StructA")
test(testB, "StructB")
}
func test(dataStruct interface{}, name string) {
rows, err := db.Query("SELECT * FROM table WHERE name =", name)
if err != nil {
panic(err)
}
for rows.Next() {
err := sqlx.StructScan(rows, &dataStruct)
if err != nil {
panic(err)
}
}
}
Super late to the party, but ran into this question while researching another issue. For others that stumble upon it, the problem is that you're passing a pointer to dataStruct into StructScan(). dataStruct is an interface, and pointers to interfaces are almost always an error in Go (in fact, they removed the automatic dereferencing of interface pointers a few versions back). You're also passing in your destination by value.
So, you are passing a pointer to an interface that holds a copy of your destination slice, when what you want instead is to pass the interface directly, and that interface to hold a pointer to your destination slice.
Instead of:
test(testA, "StructA")
test(testB, "StructB")
// ...
err := sqlx.StructScan(rows, &dataStruct)
Use:
test(&testA, "StructA")
test(&testB, "StructB")
// ...
err := sqlx.StructScan(rows, dataStruct)
If you have no idea what the destination struct type is, use sqlx.MapScan or sqlx.SliceScan. They don't map to a struct, but both return all the columns from the query result.
See http://jmoiron.github.io/sqlx/#altScanning
Related
I'm new to golang generics and have the following setup.
I've gathered loads of different kinds of reports.
Each report has enclosing fields
So I wrapped it in a ReportContainerImpl
I've used a type argument of [T Reportable] where the Reportable is defined as follows
type Reportable interface {
ExportDataPointReport | ImportDataPointReport | MissingDataPointReport | SensorThresoldReport
}
Each of the type in the type constraint is structs that is to be embedded in the container.
type ReportContainerImpl[T Reportable] struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body T `json:"body"`
}
I use a discriminator ReportType to determine the concrete type when Unmarshal.
type ReportType string
const (
ReportTypeExportDataPointReport ReportType = "ExportDataPointReport"
ReportTypeImportDataPointReport ReportType = "ImportDataPointReport"
ReportTypeMissingDataPointReport ReportType = "MissingDataPointReport"
ReportTypeSensorThresoldReport ReportType = "SensorThresoldReport"
)
Since go does not support type assertion for struct (only interfaces) it is not possible to cast the type when Unmarshal. Also go does not support pointer to the "raw" generic type. Hence, I've created a interface that the ReportContainerImpl implements.
type ReportContainer interface {
GetLocationID() string
GetProvider() string
GetReportType() ReportType
GetBody() interface{}
}
The problem I then get is that I cannot do type constrains on the return type in any form or shape and am back at "freetext semantics" on the GetBody() function to allow for type assertion when Unmarshal is done.
container, err := UnmarshalReportContainer(data)
if rep, ok := container.GetBody().(ExportDataPointReport); ok {
// Use the ReportContainerImpl[ExportDataPointReport] here...
}
Maybe I'm getting this wrong? - but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Do you have a better suggestion how to solve this in a type (safer) way?
Cheers,
Mario :)
For completeness I add the UnmarshalReportContainer here
func UnmarshalReportContainer(data []byte) (ReportContainer, error) {
type Temp struct {
LocationID string `json:"lid"`
Provider string `json:"pn"`
ReportType ReportType `json:"m"`
Body *json.RawMessage `json:"body"`
}
var temp Temp
err := json.Unmarshal(data, &temp)
if err != nil {
return nil, err
}
switch temp.ReportType {
case ReportTypeExportDataPointReport:
var report ExportDataPointReport
err := json.Unmarshal(*temp.Body, &report)
return &ReportContainerImpl[ExportDataPointReport]{
LocationID: temp.LocationID,
Provider: temp.Provider,
ReportType: temp.ReportType,
Body: report,
}, err
// ...
}
}
but however I do this, I always end up with somewhere needs a interface{} or to know the exact type before Unmarshal
Precisely.
The concrete types needed to instantiate some generic type or function like ReportContainerImpl or UnmarshalReportContainer must be known at compile time, when you write the code. JSON unmarshalling instead occurs at run-time, when you have the byte slice populated with the actual data.
To unmarshal dynamic JSON based on some discriminatory value, you still need a switch.
Do you have a better suggestion how to solve this in a type (safer) way?
Just forgo parametric polymorphism. It's not a good fit here. Keep the code you have now with json.RawMessage, unmarshal the dynamic data conditionally in the switch and return the concrete structs that implement ReportContainer interface.
As a general solution — if, and only if, you can overcome this chicken-and-egg problem and make type parameters known at compile time, you can write a minimal generic unmarshal function like this:
func unmarshalAny[T any](bytes []byte) (*T, error) {
out := new(T)
if err := json.Unmarshal(bytes, out); err != nil {
return nil, err
}
return out, nil
}
This is only meant to illustrate the principle. Note that json.Unmarshal already accepts any type, so if your generic function actually does nothing except new(T) and return, like in my example, it is no different than "inlining" the entire thing as if unmarshalAny didn't exist.
v, err := unmarshalAny[SomeType](src)
functionally equivalent as
out := &SomeType{}
err := json.Unmarshal(bytes, out)
If you plan to put more logic in unmarshalAny, its usage may be warranted. Your mileage may vary; in general, don't use type parameters when it's not actually necessary.
I am tring to get League of Legends champion informations from LOL static database. Link is given below.
Get info for specific hero
The problem is that i can only make request by hero names and all JSON responses are different from each other by only one field which is a "main" field; hero name. You can find problematic field as highlighted below:
Also tree respresentation:
My goal is to get all hero informations with iteration by range of known hero names as slice. You can check the code. I need only a couple of fields but the main tag is varies with every new request.
func GetHeroInfo(heroName string) *LolHeroInfo {
getUrl := fmt.Sprintf("http://ddragon.leagueoflegends.com/cdn/12.2.1/data/en_US/champion/%s.json", heroName)
fmt.Println(getUrl)
resp, err := http.Get(getUrl)
if err != nil {
fmt.Println(err)
return nil
}
defer resp.Body.Close()
read, err := io.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return nil
}
heroGoverted := LolHeroInfo{}
err = json.Unmarshal(read, &heroGoverted)
if err != nil {
fmt.Println("unmarshall failed:", err)
return nil
}
return &heroGoverted }
Struct type LolHeroInfo is structifyed from this link: mholt's JSON to struct
You can check JSON response for another hero eg: JSON response for Annie .
Is there any way to make an agnostic struct field/tag for a JSON field. I believe this will be very hard because encoding/json package needs to check for field for particular tag in that JSON but maybe you encountered this kind of problem before. Creating separate struct for each hero is impossible so i will drop this case if i can't find a solution.
Thanks in advance for help.
Since you know it's just a single unknown key, you could just decode into a map[string]LolHeroInfo for the Data field, then do
heroGoverted := LolHeroInfo{}
for _, v := range decoded.Data {
heroGoverted = v
}
To solve problem, I used #dave 's solution.
I breake main struct into two separate struct. This way varying JSON field eliminated:
type LolHeroInfo struct {
Type string `json:"type"`
Format string `json:"format"`
Version string `json:"version"`
Data map[string]HeroInfoStruct `json:"data"`
}
heroInfo := lib.GetHeroInfo(lolHeroes[i])
for _, v := range heroInfo.Data { //unmarshalled to first struct
a := lib.HeroInfoStruct{} //data field; second struct
a = v
fmt.Println(a.Lore)// now i can reach to every field that i need
}
I have a JSON object That contains an implementation of an interface within it. I'm attempting to take that JSON and marshal it into a struct whilst creating the implementation of the interface.
I've managed to get it to implement the interface with a custom JSON unmarshal function however I'm struggling to piece together how to then marshal the rest of the fields
I've created an example in the Go playground
https://play.golang.org/p/ztF7H7etdjM
My JSON being passed into my application is
{
"address":"1FYuJ4MsVmpzPoFJ6svJMJfygn91Eubid9",
"nonce":13,
"network_id":"qadre.demo.balance",
"challenge":"f2b19e71876c087e681fc092ea3a34d5680bbfe772e40883563e1d5513bb593f",
"type":"verifying_key",
"verifying_key":{
"verifying_key":"3b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29",
"fqdn":"huski.service.key"
},
"signature":"a3bf8ee202a508d5a5632f50b140b70b7095d8836493dc7ac4159f6f3350280078b3a58b2162a240bc8c7485894554976a9c7b5d279d3f5bf49fec950f024e02",
"fqdn":"huski.service.SingleKeyProof"
}
I've attempted to do a json.Unmarshal and pass in a new struct for the remaining fields however it seems to put me in an infinite loop, my application hangs and then crashes
The best solution I've come up with so far is to marshal the JSON into a `map[string]interface{} and do each field separately, this feels very clunky though
var m map[string]interface{}
if err := json.Unmarshal(data, &m); err != nil {
return err
}
ad, ok := m["address"]
if ok {
s.Address = ad.(string)
}
fqdn, ok := m["fqdn"]
if ok {
s.FQDN = fqdn.(string)
}
n, ok := m["nonce"]
if ok {
s.Nonce = int64(n.(float64))
}
c, ok := m["challenge"]
if ok {
s.Challenge = []byte(c.(string))
}
network, ok := m["network_id"]
if ok {
s.NetworkID = network.(string)
}
sig, ok := m["signature"]
if ok {
s.Signature = []byte(sig.(string))
}
The reason your code gets into an infinite loop when you try to unmarshal the rest of the fields is because, I presume, the implementation of UnmarshalJSON after its done unmarshaling the verifying key, calls json.Unmarshal with the receiver, which in turn calls the UnmarshalJSON method on the receiver and so they invoke each other ad infinitum.
What you can do is to create a temporary type using the existing type as its definition, this will "keep the structure" but "drop the methods", then unmarshal the rest of the fields into an instance of the new type, and, after unmarshal is done, convert the instance to the original type and assign that to the receiver.
While this fixes the infinite loop, it also re-introduces the original problem of json.Unmarshal not being able to unmarshal into a non-empty interface type. To fix that you can embed the new type in another temporary struct that has a field with the same json tag as the problematic field which will cause it to be "overshadowed" while json.Unmarshal is doing its work.
type SingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey PublicKey `json:"verifying_key"`
Signature []byte `json:"signature"`
}
func (s *SingleKey) UnmarshalJSON(data []byte) error {
type _SingleKey SingleKey
var temp struct {
RawKey json.RawMessage `json:"verifying_key"`
_SingleKey
}
if err := json.Unmarshal(data, &temp); err != nil {
return err
}
*s = SingleKey(temp._SingleKey)
switch s.Type {
case "verifying_key":
s.VerifyingKey = &PublicKeyImpl{}
// other cases ...
}
return json.Unmarshal([]byte(temp.RawKey), s.VerifyingKey)
}
https://play.golang.org/p/L3gdQZF47uN
Looking at what you've done in your custom unmarshalling function, you seem to be passing in a map with the name of fields as index, and the reflect.Type you want to unmarshal said value into. That, to me, suggests that the keys might be different for different payloads, but that each key has a distinct type associated with it. You can perfectly handle data like this with a simple wrapper type:
type WrappedSingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey json.RawMessage `json:"verifying_key"`
OtherKey json.RawMessage `json:"other_key"`
Signature []byte `json:"signature"`
}
type SingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey *PublicKey `json:"verifying_key,omitempty"`
OtherType *OtherKey `json:"other_key,omitempty"`
Signature []byte `json:"signature"`
}
So I've changed the type of your VerifyingKey field to a json.RawMessage. That's basically telling json.Unmarshal to leave that as raw JSON input. For every custom/optional field, add a corresponding RawMessage field.
In the unwrapped type, I've changed VerifyingKey to a pointer and added the omitempty bit to the tag. That's just to accomodate mutliple types, and not have to worry about custom marshalling to avoid empty fields, like the included OtherType field I have. To get what you need, then:
func (s *SingleKey) UnmarshalJSON(data []byte) error {
w := WrappedSingleKey{} // create wrapped instance
if err := json.Unmarshal(data, &w); err != nil {
return err
}
switch w.Type {
case "verifying_key":
var pk PublicKey
if err := json.Unmarshal([]byte(w.VerifyingKey), &pk); err != nil {
return err
}
s.VerifyingKey = &pk // assign
case "other_key":
var ok OtherKey
if err := json.Unmarshal([]byte(w.OtherKey), &ok); err != nil {
return err
}
s.OtherKey = &ok
}
// copy over the fields that didn't require anything special
s.FQDN = w.FQDN
s.Address = w.Address
}
This is a fairly simple approach, does away with the reflection, tons of functions, and is quite commonly used. It's something that lends itself quite well to code generation, too. The individual assignment of the fields is a bit tedious, though. You might think that you can solve that by embedding the SingleKey type into the wrapper, but be careful: this will recursively call your custom unmarshaller function.
You could, for example, update all the fields in the WRapped type to be pointers, and have them point to fields on your actual type. That does away with the manual copying of fields... It's up to you, really.
Note
I didn't test this code, just wrote it as I went along. It's something I've used in the past, and I believe what I wrote here should work, but no guarantees (as in: you might need to debug it a bit)
I'm using the mgo driver for MongoDB, with the Gin framework.
type Users struct {
User_id *string `json:"id user" bson:"id user"`
Images []string `json:"images" bson:"images"`
}
I have this function which tries to convert the slice into JSON.
The slice here is UsersTotal
func GetUsersApi(c *gin.Context) {
UsersTotal, err := GetUsers()
if err != nil {
fmt.Println("error:", err)
}
c.JSON(http.StatusOK, gin.H{
"Count Users": len(UsersTotal),
"Users Found ": UsersTotal,
})
session, err := mgo.Dial(URL)
if err == nil {
fmt.Println("Connection to mongodb established ok!!")
cc := session.DB("UsersDB").C("results")
err22 := cc.Insert(&UsersTotal)
if err22 != nil {
fmt.Println("error insertion ", err22)
}
}
session.Close()
}
Running it I get the following error:
error insertion Wrong type for documents[0]. Expected a object, got a array.
Inserting multiple documents is the same as inserting a single one because the Collection.Insert() method has a variadic parameter:
func (c *Collection) Insert(docs ...interface{}) error
One thing you should note is that it expects interface{} values. Value of any type qualifies "to be" an interface{}. Another thing you should note is that only the slice type []interface{} qualifies to be []interface{}, a user slice []User does not. For details, see Type converting slices of interfaces in go
So simply create a copy of your users slice where the copy has a type of []interface{}, and that you can directly pass to Collection.Insert():
docs := make([]interface{}, len(UsersTotal))
for i, u := range UsersTotal {
docs[i] = u
}
err := cc.Insert(docs...)
// Handle error
Also please do not connect to MongodB in your handler. Do it once, on app startup, store the global connection / session, and clone / copy it when needed. For details see mgo - query performance seems consistently slow (500-650ms); and too many open files in mgo go server.
I'm a beginner with Go, and I'm now writing a function which can call an API. The function receives a part of the url (/user, /account, etc) and the struct to convert the returned json to (the structs User or Account for example) as arguments.
So I now have this:
func (self *RestClient) request(action string, return_type interface{}) interface{} {
res, _ := goreq.Request{Uri:self.url + action}.Do()
var item return_type
res.Body.FromJsonTo(&item)
return item
}
And I try to call this function using (with Index being the struct for the return type):
self.request("/api/v1/public/index", Index)
But this doesn't work. I get the following errors:
return_type is not a type
type Index is not an expression
I guess I understand why this is. I think I have to somehow find out the type of return_type and then convert return_type to that type, before using it as the type for item. I have no idea how though.
About the second error I have no idea. I don't even understand what is meant by it.
Could anybody help me out? How can I make this work? Or should this be done in a completely different way all together? All tips are welcome!
A few hints based on this code:
Don't use self - use a meaningful identifier
Don't use interface{} to avoid dealing with the type system
Don't use reflection
Don't ignore errors returned (as from FromJsonTo or goreq.Request)
Don't use a library like goreq unless you are sure you need it (you don't) - you are pulling in lots of code you don't need and it is teaching you bad habits like attempting to use empty interface and reflection to solve simple problems.
Have a look at the definition of FromJsonTo - if you look through this library you'll see it isn't saving you much effort and is adding lots of complexity. Here is how you could do it without the library:
func (c *RestClient) Request(action string, resource interface{}) error {
res, err := http.Get(c.url + action)
if err != nil {
return err
}
defer res.Body.Close()
return json.NewDecoder(res.Body).Decode(resource)
}
Or use an interface and move the decoding to the resource (which could embed a default decoder):
type Decoder interface {
Decode(r io.Reader) error
}
// RequestDecode fetches a request and feeds it to the decoder
func (c *RestClient) RequestDecode(action string, resource Decoder) error {
res, err := http.Get(c.url + action)
if err != nil {
return err
}
defer res.Body.Close()
return resource.Decode(res.Body)
}
I'll first say that you should always check for errors from any function that possibly returns one.
The error you are seeing is because you are trying to declare a variable item as type return_type and that is the name of a function argument.
The other error is from Index being a type declaration and not a concrete value of the Index type.
I agree with Volker's comment but to put it in code, you could use something like this:
func (self *RestClient) request(action string, return_type interface{}) {
res, err := goreq.Request{Uri:self.url + action}.Do()
if err != nil {
// Do something with error here.
}
res.Body.FromJsonTo(return_type)
}
var index Index
rest_client.request("/some/path", &index)
This allows flexibility but could lead to strange cases if you forget to pass a pointer to the value return_type.