I am having issue with JSON unmarshal to a Go structure. Visibly the source of the issue is understable - a string is coming for the int64, except that I am not finding ways to solve it without changing the datatype of Count to map[string]string. I saw similar discussions on SO, but they didn't help much. Here is a representative code and the issue. Please suggest how I can get through.
type Product struct {
Count map[string]int64 `json:",string"`
}
func main() {
s := `{"count":{"FAN":"58"}}`
var pro Product
err := json.Unmarshal([]byte(s), &pro)
if err == nil {
fmt.Printf("%+v\n", pro)
} else {
fmt.Println(err)
fmt.Printf("%+v\n", pro)
}
}
Output:
json: cannot unmarshal string into Go struct field Product.Count of type int64
{Count:map[FAN:0]}
As you've noted yourself, this is happening because you are not passing in the JSON integer 58, but a JSON string "58".
If you modify your example input to
s := `{"count":{"FAN":58}}`
then your code works as expected.
You could write a custom unmarshaller, but fixing the issue while unmarshalling might not be what you want; rather find a way to modify your data pipeline and ensure that the JSON object is sanitized properly. Or, if that's impossible, do as you suggest yourself and use a map[string]string instead, then fix up the data types after unmashalling.
Related
I am tring to get League of Legends champion informations from LOL static database. Link is given below.
Get info for specific hero
The problem is that i can only make request by hero names and all JSON responses are different from each other by only one field which is a "main" field; hero name. You can find problematic field as highlighted below:
Also tree respresentation:
My goal is to get all hero informations with iteration by range of known hero names as slice. You can check the code. I need only a couple of fields but the main tag is varies with every new request.
func GetHeroInfo(heroName string) *LolHeroInfo {
getUrl := fmt.Sprintf("http://ddragon.leagueoflegends.com/cdn/12.2.1/data/en_US/champion/%s.json", heroName)
fmt.Println(getUrl)
resp, err := http.Get(getUrl)
if err != nil {
fmt.Println(err)
return nil
}
defer resp.Body.Close()
read, err := io.ReadAll(resp.Body)
if err != nil {
fmt.Println(err)
return nil
}
heroGoverted := LolHeroInfo{}
err = json.Unmarshal(read, &heroGoverted)
if err != nil {
fmt.Println("unmarshall failed:", err)
return nil
}
return &heroGoverted }
Struct type LolHeroInfo is structifyed from this link: mholt's JSON to struct
You can check JSON response for another hero eg: JSON response for Annie .
Is there any way to make an agnostic struct field/tag for a JSON field. I believe this will be very hard because encoding/json package needs to check for field for particular tag in that JSON but maybe you encountered this kind of problem before. Creating separate struct for each hero is impossible so i will drop this case if i can't find a solution.
Thanks in advance for help.
Since you know it's just a single unknown key, you could just decode into a map[string]LolHeroInfo for the Data field, then do
heroGoverted := LolHeroInfo{}
for _, v := range decoded.Data {
heroGoverted = v
}
To solve problem, I used #dave 's solution.
I breake main struct into two separate struct. This way varying JSON field eliminated:
type LolHeroInfo struct {
Type string `json:"type"`
Format string `json:"format"`
Version string `json:"version"`
Data map[string]HeroInfoStruct `json:"data"`
}
heroInfo := lib.GetHeroInfo(lolHeroes[i])
for _, v := range heroInfo.Data { //unmarshalled to first struct
a := lib.HeroInfoStruct{} //data field; second struct
a = v
fmt.Println(a.Lore)// now i can reach to every field that i need
}
Please forgive my question, I'm new to Golang and possibly have the wrong approach.
I'm currently implementing a Terraform provider for an internal service.
As probably expected, that requires unmarshalling JSON data in to pre-defined Struct Types, e.g:
type SomeTypeIveDefined struct {
ID string `json:"id"`
Name String `json:"name"`
}
I've got myself in to a situation where I have a lot of duplicate code that looks like this
res := r.(*http.Response)
var tempThing SomeTypeIveDefined
dec := json.NewDecoder(res.Body)
err := dec.Decode(&tempThing)
In an effort to reduce duplication, I decided what I wanted to do was create a function which does the JSON unmarshalling, but takes in the Struct Type as a parameter.
I've trawled through several StackOverflow articles and Google Groups trying to make sense of some of the answers around using the reflect package, but I've not had much success in using it.
My latest attempt was using reflect.StructOf and passing in a set of StructFields, but that still seems to require using myReflectedStruct.Field(0) rather than myReflectedStruct.ID.
I suspect there may be no way until something like Generics are widely available in Golang.
I considered perhaps an interface for the structs which requires implementing an unmarshal method, then I could pass the interface to the function and call the unmarshal method. But then I'm still implementing unmarshal on all the structs anyway.
I'm just wondering what suggestions there may be for achieving what I'm after, please?
Create a helper function with the repeated code. Pass the destination value as a pointer.
func decode(r *http.Repsonse, v interface{}) error {
return json.NewDecoder(res.Body).Decode(v)
}
Call the helper function with a pointer to your thing:
var tempThing SomeTypeIveDefined
err := deocde(r, &tempThing)
You can do this with interfaces:
func decodeResponse(r *http.Response, dest interface{}) error {
dec := json.NewDecoder(r.Body)
return dec.Decode(dest)
}
func handler(...) {
res := r.(*http.Response)
var tempThing SomeTypeIveDefined
if err:=decodeResponse(res,&tempThing); err!=nil {
// handle err
}
...
}
You don't need to implement an unmarshal for the structs, because the stdlib decoder will use reflection to set the struct fields.
I would like to UnmarshalJSON a struct containing an interface as follows:
type Filterer interface {
Filter(s string) error
}
type FieldFilter struct {
Key string
Val string
}
func (ff *FieldFilter) Filter(s string) error {
// Do something
}
type Test struct {
Name string
Filters []Filterer
}
My idea was to send a json like so:
{
"Name": "testing",
"Filters": [
{
"FieldFilter": {
"Key": "key",
"Val": "val"
}
}
]
}
However, when sending this json to the unmarshaler, the following exception returns: json: cannot unmarshal object into Go struct field Test.Filters of type Filterer
I understand the problem fully, but do not know how to approach this problem wisely. Looking for advice on an idiomatic way to solving this problem in go.
Following my own question, I researched how one could implement UnmarshalJSON for interface lists. Ultimately this led me to publish a blog post on how to do this properly. Basically there are 2 main solutions:
Parse the required JSON string into a map[string]*json.RawMessage and work your way from there.
Make an alias for the interface list and implement UnmarshalJSON for that alias. However, you'll still need to work with map[string]*json.RawMessage and some manual work. Nothing comes without a price!
I highly suggest taking the seconds approach. While these two solutions may result in the same amount of code lines, taking advantage of type aliasing and being less dependent on json.RawMessage types will make a more easy to manage code, especially when it is required to support multiple interfaces on the UnmarshalJSON implementation
To directly answer the question, start with making a type alias for the interface list:
type Filterers []Filterer
Now continue with implementing the decoding of the JSON:
func (f *Filterers) UnmarshalJSON(b []byte) error {
var FilterFields map[string]*json.RawMessage
if err := json.Unmarshal(b, &FilterFields); err != nil {
return err
}
for LFKey, LFValue := range FilterFields {
if LFKey == "FieldFilter" {
var MyFieldFilters []*json.RawMessage
if err := json.Unmarshal(*LFValue, &MyFieldFilters); err != nil {
return err
}
for _, MyFieldFilter := range MyFieldFilters {
var filter FieldFilter
if err := json.Unmarshal(*MyFieldFilter, &filter); err != nil {
return err
}
*f = append(*f, &filter)
}
}
}
return nil
}
A detailed explanation (with some examples and a full working code snippets) of the second approach is available on my own blog
There is no way for Unmarshal to know what type it should use. The only case where it can just "make something up" is if it's asked to unmarshal into an interface{}, in which case it will use the rules in the documentation. Since none of those types can be put into a []Filterer, it cannot unmarshal that field. If you want to unmarshal into a struct type, you must specify the field to be of that type.
You can always unmarshal into an intermediate struct or map type, and then do your own conversion from that into whatever types you want.
I've got a problem with my json parsing in golang.
I used some code to parse the json into a map[string]interface{}{} but when I try to iterate through a nested field, the error (type interface {} does not support indexing) is triggered.
I'd like to get the following informations :
iterate through each response->blogs and then get the url of original_size photo that lays in response->posts->blog_n->photos->original_size
meta->status
response->blog->total_posts and response->blog->name
Here's a link to a playground
Thank you for your help !
Is there a reason you want to use a map? To do the indexing you're talking about, with maps, I think you would need nested maps as well.
Have you considered using nested structs, as described here, Go Unmarshal nested JSON structure and Unmarshaling nested JSON objects in Golang ?
For your JSON data, here is a sample -- working but limited -- struct. Go will ignore the fields you don't use.
func main() {
//Creating the maps for JSON
data := &Header{}
//Parsing/Unmarshalling JSON encoding/json
err := json.Unmarshal([]byte(input), &data)
if err != nil {
panic(err)
}
fmt.Printf("%+v\n", m)
}
type Header struct {
Response struct {
Blog struct {
Title string `json:"title"`
} `json:"blog"`
Posts []struct {
Id int64 `json:"id"`
} `json:"posts"`
} `json:"response"`
}
To get JSON with Go to do what you want, you have to brush up on JSON and it's relation to Go types.
Notice posts:
slices of structs are, in JSON, arrays of objects, [{..},{..}]
In Go, only exported fields will be filled.
this error appear because your map[key-type]val-type, and you to try get value as nested map.
you can use Type Assertion to get value.
result := m["response"].(map[string]interface{})
fmt.Printf("%+v\n", result["blog"])
Scenario:
Consider the following is the JSON :
{
"Bangalore_City": "35_Temperature",
"NewYork_City": "31_Temperature",
"Copenhagen_City": "29_Temperature"
}
If you notice, the data is structured in such a way that there is no hard-coded keys mentioning City/Temperature its basically just values.
Issue: I am not able to parse any JSON which is dynamic.
Question: Could anyone have found solution for this kind of JSON parsing? I tried go-simplejson, gabs & default encoding/json but no luck.
Note:
The above JSON is just for sample. And there are lot of applications which are using the current API, So I do not want to change how the data is structured. I mean I can't change to something as follows:
[{
"City_Name":"Bangalore",
"Temperature": "35"
},...]
Then I can define struct
type TempData struct {
City_Name string
Temperature string
}
You can unmarshal into a map[string]string for example:
m := map[string]string{}
err := json.Unmarshal([]byte(input), &m)
if err != nil {
panic(err)
}
fmt.Println(m)
Output (wrapped):
map[Bangalore_City:35_Temperature NewYork_City:31_Temperature
Copenhagen_City:29_Temperature]
Try it on the Go Playground.
This way no matter what the keys or values are, you will have all pairs in a map which you can print or loop over.
Also note that although your example contained only string values, but if the value type is varying (e.g. string, numbers etc.), you may use interface{} for the value type, in which case your map would be of type map[string]interface{}.
Also note that I created a library to easily work with such dynamic objects which may be a great help in these cases: github.com/icza/dyno.
Standard encoding/json is good for the majority of use cases, but it may be quite slow comparing to alternative solutions. If you need performance, try using fastjson. It parses arbitrary JSONs without the need for creating structs or maps matching the JSON schema.
See the example code below. It iterates over all the (key, value) pairs of the JSON object:
var p fastjson.Parser
v, err := p.Parse(input)
if err != nil {
log.Fatal(err)
}
// Visit all the items in the top object
v.GetObject().Visit(func(k []byte, v *fastjson.Value) {
fmt.Printf("key=%s, value=%s\n", k, v)
// for nested objects call Visit again
if string(k) == "nested" {
v.GetObject().Visit(func(k []byte, v *fastjson.Value) {
fmt.Printf("nested key=%s, value=%s\n", k, v)
})
}
})
Just to add a general answer to how any valid JSON can be parsed; var m interface{} works for all types. That includes map (which OP asked for) arrays, strings, numbers and nested structures.
var m interface{}
err := json.Unmarshal([]byte(input), &m)
if err != nil {
panic(err)
}
fmt.Println(m)
Also consider using gabs package https://github.com/Jeffail/gabs
"Gabs is a small utility for dealing with dynamic or unknown JSON structures in Go"