I have the following structures to export onto json:
type ExportedIncident struct {
Title string `json:"title"`
Host string `json:"host"`
Status string `json:"status"`
Date string `json:"date"`
Notes []ExportedNote `json:"notes"`
LogEntries []ExportedLogEntry `json:"log_entries"`
}
And I want underscore cased fields, so I had to define each field just for that as described in this answer: https://stackoverflow.com/a/11694255/1731473
But it's really cumbersome and I believe there is a simpler solution in Go, but I can't find it.
How can I set default letter-case (underscore, snake, camel...) for JSON export?
Unfortunately there are no opportunities to export your fields into snake_case so you have to maintain tags by yourself.
Technically you can use method MarshalJSON and perform all manipulations inside this method, but it isn't easier way...
As mentioned by #Vladimir Kovpak, you can't do this with the standard library, at least at the moment.
Though, inspired by this, you can achieve something close to what you want to do. Check out MarshalIndentSnakeCase:
func MarshalIndentSnakeCase(v interface{}, prefix, indent string) ([]byte, error) {
b, err := json.MarshalIndent(v, prefix, indent)
if err != nil {
return nil, err
}
x := convertKeys(b) // Here convert all keys from CamelCase to snake_case
buf := &bytes.Buffer{}
err = json.Indent(buf, []byte(x), prefix, indent)
if err != nil {
return nil, err
}
return buf.Bytes(), nil
}
Down Sides:
You have to do the same in the opposite direction for Unmashalling to work.
The order of elements is lost due to using a map inside convertKeys().
Try it on Go Playground.
Related
I am tring to get League of Legends champion informations from LOL static database. Link is given below.
Get info for specific hero
The problem is that i can only make request by hero names and all JSON responses are different from each other by only one field which is a "main" field; hero name. You can find problematic field as highlighted below:
Also tree respresentation:
My goal is to get all hero informations with iteration by range of known hero names as slice. You can check the code. I need only a couple of fields but the main tag is varies with every new request.
func GetHeroInfo(heroName string) *LolHeroInfo {
getUrl := fmt.Sprintf("http://ddragon.leagueoflegends.com/cdn/12.2.1/data/en_US/champion/%s.json", heroName)
fmt.Println(getUrl)
resp, err := http.Get(getUrl)
if err != nil {
fmt.Println(err)
return nil
}
defer resp.Body.Close()
read, err := io.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return nil
}
heroGoverted := LolHeroInfo{}
err = json.Unmarshal(read, &heroGoverted)
if err != nil {
fmt.Println("unmarshall failed:", err)
return nil
}
return &heroGoverted }
Struct type LolHeroInfo is structifyed from this link: mholt's JSON to struct
You can check JSON response for another hero eg: JSON response for Annie .
Is there any way to make an agnostic struct field/tag for a JSON field. I believe this will be very hard because encoding/json package needs to check for field for particular tag in that JSON but maybe you encountered this kind of problem before. Creating separate struct for each hero is impossible so i will drop this case if i can't find a solution.
Thanks in advance for help.
Since you know it's just a single unknown key, you could just decode into a map[string]LolHeroInfo for the Data field, then do
heroGoverted := LolHeroInfo{}
for _, v := range decoded.Data {
heroGoverted = v
}
To solve problem, I used #dave 's solution.
I breake main struct into two separate struct. This way varying JSON field eliminated:
type LolHeroInfo struct {
Type string `json:"type"`
Format string `json:"format"`
Version string `json:"version"`
Data map[string]HeroInfoStruct `json:"data"`
}
heroInfo := lib.GetHeroInfo(lolHeroes[i])
for _, v := range heroInfo.Data { //unmarshalled to first struct
a := lib.HeroInfoStruct{} //data field; second struct
a = v
fmt.Println(a.Lore)// now i can reach to every field that i need
}
I have a JSON object That contains an implementation of an interface within it. I'm attempting to take that JSON and marshal it into a struct whilst creating the implementation of the interface.
I've managed to get it to implement the interface with a custom JSON unmarshal function however I'm struggling to piece together how to then marshal the rest of the fields
I've created an example in the Go playground
https://play.golang.org/p/ztF7H7etdjM
My JSON being passed into my application is
{
"address":"1FYuJ4MsVmpzPoFJ6svJMJfygn91Eubid9",
"nonce":13,
"network_id":"qadre.demo.balance",
"challenge":"f2b19e71876c087e681fc092ea3a34d5680bbfe772e40883563e1d5513bb593f",
"type":"verifying_key",
"verifying_key":{
"verifying_key":"3b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29",
"fqdn":"huski.service.key"
},
"signature":"a3bf8ee202a508d5a5632f50b140b70b7095d8836493dc7ac4159f6f3350280078b3a58b2162a240bc8c7485894554976a9c7b5d279d3f5bf49fec950f024e02",
"fqdn":"huski.service.SingleKeyProof"
}
I've attempted to do a json.Unmarshal and pass in a new struct for the remaining fields however it seems to put me in an infinite loop, my application hangs and then crashes
The best solution I've come up with so far is to marshal the JSON into a `map[string]interface{} and do each field separately, this feels very clunky though
var m map[string]interface{}
if err := json.Unmarshal(data, &m); err != nil {
return err
}
ad, ok := m["address"]
if ok {
s.Address = ad.(string)
}
fqdn, ok := m["fqdn"]
if ok {
s.FQDN = fqdn.(string)
}
n, ok := m["nonce"]
if ok {
s.Nonce = int64(n.(float64))
}
c, ok := m["challenge"]
if ok {
s.Challenge = []byte(c.(string))
}
network, ok := m["network_id"]
if ok {
s.NetworkID = network.(string)
}
sig, ok := m["signature"]
if ok {
s.Signature = []byte(sig.(string))
}
The reason your code gets into an infinite loop when you try to unmarshal the rest of the fields is because, I presume, the implementation of UnmarshalJSON after its done unmarshaling the verifying key, calls json.Unmarshal with the receiver, which in turn calls the UnmarshalJSON method on the receiver and so they invoke each other ad infinitum.
What you can do is to create a temporary type using the existing type as its definition, this will "keep the structure" but "drop the methods", then unmarshal the rest of the fields into an instance of the new type, and, after unmarshal is done, convert the instance to the original type and assign that to the receiver.
While this fixes the infinite loop, it also re-introduces the original problem of json.Unmarshal not being able to unmarshal into a non-empty interface type. To fix that you can embed the new type in another temporary struct that has a field with the same json tag as the problematic field which will cause it to be "overshadowed" while json.Unmarshal is doing its work.
type SingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey PublicKey `json:"verifying_key"`
Signature []byte `json:"signature"`
}
func (s *SingleKey) UnmarshalJSON(data []byte) error {
type _SingleKey SingleKey
var temp struct {
RawKey json.RawMessage `json:"verifying_key"`
_SingleKey
}
if err := json.Unmarshal(data, &temp); err != nil {
return err
}
*s = SingleKey(temp._SingleKey)
switch s.Type {
case "verifying_key":
s.VerifyingKey = &PublicKeyImpl{}
// other cases ...
}
return json.Unmarshal([]byte(temp.RawKey), s.VerifyingKey)
}
https://play.golang.org/p/L3gdQZF47uN
Looking at what you've done in your custom unmarshalling function, you seem to be passing in a map with the name of fields as index, and the reflect.Type you want to unmarshal said value into. That, to me, suggests that the keys might be different for different payloads, but that each key has a distinct type associated with it. You can perfectly handle data like this with a simple wrapper type:
type WrappedSingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey json.RawMessage `json:"verifying_key"`
OtherKey json.RawMessage `json:"other_key"`
Signature []byte `json:"signature"`
}
type SingleKey struct {
FQDN string `json:"fqdn"`
Address string `json:"address"`
Nonce int64 `json:"nonce"`
Challenge []byte `json:"challenge"`
NetworkID string `json:"network_id"`
Type string `json:"type"`
VerifyingKey *PublicKey `json:"verifying_key,omitempty"`
OtherType *OtherKey `json:"other_key,omitempty"`
Signature []byte `json:"signature"`
}
So I've changed the type of your VerifyingKey field to a json.RawMessage. That's basically telling json.Unmarshal to leave that as raw JSON input. For every custom/optional field, add a corresponding RawMessage field.
In the unwrapped type, I've changed VerifyingKey to a pointer and added the omitempty bit to the tag. That's just to accomodate mutliple types, and not have to worry about custom marshalling to avoid empty fields, like the included OtherType field I have. To get what you need, then:
func (s *SingleKey) UnmarshalJSON(data []byte) error {
w := WrappedSingleKey{} // create wrapped instance
if err := json.Unmarshal(data, &w); err != nil {
return err
}
switch w.Type {
case "verifying_key":
var pk PublicKey
if err := json.Unmarshal([]byte(w.VerifyingKey), &pk); err != nil {
return err
}
s.VerifyingKey = &pk // assign
case "other_key":
var ok OtherKey
if err := json.Unmarshal([]byte(w.OtherKey), &ok); err != nil {
return err
}
s.OtherKey = &ok
}
// copy over the fields that didn't require anything special
s.FQDN = w.FQDN
s.Address = w.Address
}
This is a fairly simple approach, does away with the reflection, tons of functions, and is quite commonly used. It's something that lends itself quite well to code generation, too. The individual assignment of the fields is a bit tedious, though. You might think that you can solve that by embedding the SingleKey type into the wrapper, but be careful: this will recursively call your custom unmarshaller function.
You could, for example, update all the fields in the WRapped type to be pointers, and have them point to fields on your actual type. That does away with the manual copying of fields... It's up to you, really.
Note
I didn't test this code, just wrote it as I went along. It's something I've used in the past, and I believe what I wrote here should work, but no guarantees (as in: you might need to debug it a bit)
I'm a beginner with Go, and I'm now writing a function which can call an API. The function receives a part of the url (/user, /account, etc) and the struct to convert the returned json to (the structs User or Account for example) as arguments.
So I now have this:
func (self *RestClient) request(action string, return_type interface{}) interface{} {
res, _ := goreq.Request{Uri:self.url + action}.Do()
var item return_type
res.Body.FromJsonTo(&item)
return item
}
And I try to call this function using (with Index being the struct for the return type):
self.request("/api/v1/public/index", Index)
But this doesn't work. I get the following errors:
return_type is not a type
type Index is not an expression
I guess I understand why this is. I think I have to somehow find out the type of return_type and then convert return_type to that type, before using it as the type for item. I have no idea how though.
About the second error I have no idea. I don't even understand what is meant by it.
Could anybody help me out? How can I make this work? Or should this be done in a completely different way all together? All tips are welcome!
A few hints based on this code:
Don't use self - use a meaningful identifier
Don't use interface{} to avoid dealing with the type system
Don't use reflection
Don't ignore errors returned (as from FromJsonTo or goreq.Request)
Don't use a library like goreq unless you are sure you need it (you don't) - you are pulling in lots of code you don't need and it is teaching you bad habits like attempting to use empty interface and reflection to solve simple problems.
Have a look at the definition of FromJsonTo - if you look through this library you'll see it isn't saving you much effort and is adding lots of complexity. Here is how you could do it without the library:
func (c *RestClient) Request(action string, resource interface{}) error {
res, err := http.Get(c.url + action)
if err != nil {
return err
}
defer res.Body.Close()
return json.NewDecoder(res.Body).Decode(resource)
}
Or use an interface and move the decoding to the resource (which could embed a default decoder):
type Decoder interface {
Decode(r io.Reader) error
}
// RequestDecode fetches a request and feeds it to the decoder
func (c *RestClient) RequestDecode(action string, resource Decoder) error {
res, err := http.Get(c.url + action)
if err != nil {
return err
}
defer res.Body.Close()
return resource.Decode(res.Body)
}
I'll first say that you should always check for errors from any function that possibly returns one.
The error you are seeing is because you are trying to declare a variable item as type return_type and that is the name of a function argument.
The other error is from Index being a type declaration and not a concrete value of the Index type.
I agree with Volker's comment but to put it in code, you could use something like this:
func (self *RestClient) request(action string, return_type interface{}) {
res, err := goreq.Request{Uri:self.url + action}.Do()
if err != nil {
// Do something with error here.
}
res.Body.FromJsonTo(return_type)
}
var index Index
rest_client.request("/some/path", &index)
This allows flexibility but could lead to strange cases if you forget to pass a pointer to the value return_type.
I have an application that consumes data from a third-party api. I need to decode the json into a struct, which requires the struct to have json tags of the "incoming" json fields. The outgoing json fields have a different naming convention, so I need different json tags for the encoding.
I will have to do this with many different structs, and each struct might have many fields.
What is the best way to accomplish this without repeating a lot of code?
Example Structs:
// incoming "schema" field names
type AccountIn struct {
OpenDate string `json:"accountStartDate"`
CloseDate string `json:"cancelDate"`
}
// outgoing "schema" field names
type AccountOut struct {
OpenDate string `json:"openDate"`
CloseDate string `json:"closeDate"`
}
Maybe the coming change on Go 1.8 would help you, it will allow to 'cast' types even if its JSON tags definition is different: This https://play.golang.org/p/Xbsoa8SsEk works as expected on 1.8beta, I guess this would simplify your current solution
A bit an uncommon but probably quite well working method would be to use a intermediate format so u can use different readers and writers and therefore different tags. For example https://github.com/mitchellh/mapstructure which allows to convert a nested map structure into struct
types. Pretty similar like json unmarshal, just from a map.
// incoming "schema" field names
type AccountIn struct {
OpenDate string `mapstructure:"accountStartDate" json:"openDate"`
CloseDate string `mapstructure:"cancelDate" json:"closeDate"`
}
// from json to map with no name changes
temporaryMap := map[string]interface{}{}
err := json.Unmarshal(jsonBlob, &temporaryMap)
// from map to structs using mapstructure tags
accountIn := &AccountIn{}
mapstructure.Decode(temporaryMap, accountIn)
Later when writing (or reading) u will use directly the json functions which will then use the json tags.
If it's acceptable to take another round trip through json.Unmarshal and json.Marshal, and you don't have any ambiguous field names within your various types, you could translate all the json keys in one pass by unmarshaling into the generic structures used by the json package:
// map incoming to outgoing json identifiers
var translation = map[string]string{
"accountStartDate": "openDate",
"cancelDate": "closeDate",
}
func translateJS(js []byte) ([]byte, error) {
var m map[string]interface{}
if err := json.Unmarshal(js, &m); err != nil {
return nil, err
}
translateKeys(m)
return json.MarshalIndent(m, "", " ")
}
func translateKeys(m map[string]interface{}) {
for _, v := range m {
if v, ok := v.(map[string]interface{}); ok {
translateKeys(v)
}
}
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
for _, k := range keys {
if newKey, ok := translation[k]; ok {
m[newKey] = m[k]
delete(m, k)
}
}
}
https://play.golang.org/p/nXmWlj7qH9
This might be a Naive Approach but is fairly easy to implement:-
func ConvertAccountInToAccountOut(AccountIn incoming) (AccountOut outcoming){
var outcoming AccountOut
outcoming.OpenDate = incoming.OpenDate
outcoming.CloseDate = incoming.CloseDate
return outcoming
}
var IncomingJSONData AccountIn
resp := getJSONDataFromSource() // Some method that gives you the Input JSON
err1 := json.UnMarshall(resp,&IncomingJSONData)
OutGoingJSONData := ConvertAccountInToAccountOut(IncomingJSONData)
if err1 != nil {
fmt.Println("Error in UnMarshalling JSON ",err1)
}
fmt.Println("Outgoing JSON Data: ",OutGoingJSONData)
Scenario:
Consider the following is the JSON :
{
"Bangalore_City": "35_Temperature",
"NewYork_City": "31_Temperature",
"Copenhagen_City": "29_Temperature"
}
If you notice, the data is structured in such a way that there is no hard-coded keys mentioning City/Temperature its basically just values.
Issue: I am not able to parse any JSON which is dynamic.
Question: Could anyone have found solution for this kind of JSON parsing? I tried go-simplejson, gabs & default encoding/json but no luck.
Note:
The above JSON is just for sample. And there are lot of applications which are using the current API, So I do not want to change how the data is structured. I mean I can't change to something as follows:
[{
"City_Name":"Bangalore",
"Temperature": "35"
},...]
Then I can define struct
type TempData struct {
City_Name string
Temperature string
}
You can unmarshal into a map[string]string for example:
m := map[string]string{}
err := json.Unmarshal([]byte(input), &m)
if err != nil {
panic(err)
}
fmt.Println(m)
Output (wrapped):
map[Bangalore_City:35_Temperature NewYork_City:31_Temperature
Copenhagen_City:29_Temperature]
Try it on the Go Playground.
This way no matter what the keys or values are, you will have all pairs in a map which you can print or loop over.
Also note that although your example contained only string values, but if the value type is varying (e.g. string, numbers etc.), you may use interface{} for the value type, in which case your map would be of type map[string]interface{}.
Also note that I created a library to easily work with such dynamic objects which may be a great help in these cases: github.com/icza/dyno.
Standard encoding/json is good for the majority of use cases, but it may be quite slow comparing to alternative solutions. If you need performance, try using fastjson. It parses arbitrary JSONs without the need for creating structs or maps matching the JSON schema.
See the example code below. It iterates over all the (key, value) pairs of the JSON object:
var p fastjson.Parser
v, err := p.Parse(input)
if err != nil {
log.Fatal(err)
}
// Visit all the items in the top object
v.GetObject().Visit(func(k []byte, v *fastjson.Value) {
fmt.Printf("key=%s, value=%s\n", k, v)
// for nested objects call Visit again
if string(k) == "nested" {
v.GetObject().Visit(func(k []byte, v *fastjson.Value) {
fmt.Printf("nested key=%s, value=%s\n", k, v)
})
}
})
Just to add a general answer to how any valid JSON can be parsed; var m interface{} works for all types. That includes map (which OP asked for) arrays, strings, numbers and nested structures.
var m interface{}
err := json.Unmarshal([]byte(input), &m)
if err != nil {
panic(err)
}
fmt.Println(m)
Also consider using gabs package https://github.com/Jeffail/gabs
"Gabs is a small utility for dealing with dynamic or unknown JSON structures in Go"