how to convert map[string]interface{} data to struct - json

I do not know how to ask, so i will ask with an example.
I have some data like that
{
..
"velocityStatEntries": {
"8753": {
"estimated": {"value": 23.0,"text": "23.0"},
"completed": {"value": 27.0,"text": "27.0"}
},
"8673": {
"estimated": {"value": 54.5,"text": "54.5"},
"completed": {"value": 58.5,"text": "58.5"}
},
.
.
.
}
..
}
I want to declare a type that takes map key to its "KEY" or any property that is given by me.
Is it possible without using map iteration?
Expected output:
{...
"velocityStatEntries": {
{
"key": "8753",
"estimated": {"value": 54.5,"text": "54.5"},
"completed": {"value": 58.5,"text": "58.5"}
},
{
"key": "8673",
"estimated": {"value": 54.5,"text": "54.5"},
"completed": {"value": 58.5,"text": "58.5"}
},
}
...
}
This is what i have done
type VelocityStatEntry struct {
Key string
Estimated struct {
Value float64 `json:"value"`
Text string `json:"text"`
} `json:"estimated"`
Completed struct {
Value float64 `json:"value"`
Text string `json:"text"`
} `json:"completed"`
}
type RapidChartResponse struct {
...
VelocityStatEntries map[string]VelocityStatEntry `json:"velocityStatEntries"`
..
}
But it is not working. I want to take that string map key to KEY property.

If the data originates from JSON then you should skip the map[string]interface{} and instead use a custom unmarshaler implemented by your desired struct that does what you want. Perhaps by utilizing a map[string]json.RawMessage. But map[string]interface{} to struct conversion is a pain, avoid it if possible.
For example:
type VelocityStatEntryList []*VelocityStatEntry
func (ls *VelocityStatEntryList) UnmarshalJSON(data []byte) error {
var m map[string]json.RawMessage
if err := json.Unmarshal(data, &m); err != nil {
return err
}
for k, v := range m {
e := &VelocityStatEntry{Key: k}
if err := json.Unmarshal([]byte(v), e); err != nil {
return err
}
*ls = append(*ls, e)
}
return nil
}
https://go.dev/play/p/VcaW_BWXRVr

Related

Trouble mapping json to golang struct

I have a json stream as follows ...
[
{
"page": 1,
"pages": 7,
"per_page": "2000",
"total": 13200
},
[
{
"indicator": {
"id": "SP.POP.TOTL",
"value": "Population, total"
},
"country": {
"id": "1A",
"value": "Arab World"
},
"value": null,
"decimal": "0",
"date": "2019"
},
{
"indicator": {
"id": "SP.POP.TOTL",
"value": "Population, total"
},
"country": {
"id": "1A",
"value": "Arab World"
},
"value": "419790588",
"decimal": "0",
"date": "2018"
},
...
]
]
And I'm trying to decode it ... so I have the following struct ... but I keep getting
"cannot unmarshal array into Go value of type struct { P struct ... "
type Message []struct {
P struct {
Page int
}
V []struct {
Indicator struct {
Id string
Value string
}
Country struct {
Value string
}
Value string
Decimal string
Date string
}
}
My struct looks to match the json ... but obviously not! Any ideas?
Since your JSON array have two different types first unmarshal them into a slice of json.RawMessage which is []byte as underlying type so that we can unmarshal again JSON array data.
So unmarshal data for P and V struct type using index directly (predict) or detect if object(starting with '{') then unmarshal into P and array(starting with '[') then unmarshal into V. Now prepare your Message using those data.
type Message struct {
PageData P
ValData []V
}
type P struct {
Page int
}
type V struct {
Indicator struct {
Id string
Value string
}
Country struct {
Value string
}
Value string
Decimal string
Date string
}
func main() {
var rawdata []json.RawMessage
json.Unmarshal([]byte(jsonData), &rawdata)
var pageData P
json.Unmarshal(rawdata[0], &pageData)
var valData []V
json.Unmarshal(rawdata[1], &valData)
res := Message{pageData, valData}
fmt.Println(res)
}
var jsonData = `[...]` //your json data
Full code in Go Playground
As poWar said, the JSON you actually have is a list of objects whose types do not conform to each other. You must therefore unmarshal into something capable of holding different object types, such as interface{} or—since there is an outer array—[]interface{}.
You can also, if you like, decode into a []json.RawMessage. The underlying json.RawMessage itself has underlying type []byte so that it's basically the undecoded "inner" JSON. In at least some cases this is going to be more work than just decoding directly to []interface{} and checking each resulting interface, but you can, if you wish, decode to struct once you have the JSON separated out. For instance:
func main() {
var x []json.RawMessage
err := json.Unmarshal(input, &x)
if err != nil {
fmt.Printf("err = %v\n", err)
return
}
if len(x) != 2 {
fmt.Println("unexpected input")
return
}
var page struct {
Page int
}
err = json.Unmarshal(x[0], &page)
if err != nil {
fmt.Printf("unable to unmarshal page part: %v\n", err)
return
}
fmt.Printf("page = %d\n", page.Page)
// ...
}
Here on the Go Playground is a more complete example. See also Eklavya's answer.
Looking at your struct, your corresponding JSON should look something like this.
[
{
"P": {"page": 1},
"V": [
{
"Indicator": {"Id": ...},
"Country": {"Value":""},
"Value": "",
...
}
]
},
...
]
The JSON structure you are trying to Unmarshal looks like a list of objects where each object is not of the same type. You can start unmarshalling them into interfaces and defining each interface based on the object being unmarhsalled.
package main
import (
"encoding/json"
"log"
)
type Message []interface{}
func main() {
data := `[{"page":1,"pages":7,"per_page":"2000","total":13200},[{"indicator":{"id":"SP.POP.TOTL","value":"Population, total"},"country":{"id":"1A","value":"Arab World"},"value":null,"decimal":"0","date":"2019"},{"indicator":{"id":"SP.POP.TOTL","value":"Population, total"},"country":{"id":"1A","value":"Arab World"},"value":"419790588","decimal":"0","date":"2018"}]]`
var m Message
if err := json.Unmarshal([]byte(data), &m); err != nil {
log.Fatalf("could not unmarshal")
}
log.Printf("message: %v", m)
}
Output:
message: [map[page:1 pages:7 per_page:2000 total:13200] [map[country:map[id:1A value:Arab World] date:2019 decimal:0 indicator:map[id:SP.POP.TOTL value:Population, total] value:<nil>] map[country:map[id:1A value:Arab World] date:2018 decimal:0 indicator:map[id:SP.POP.TOTL value:Population, total] value:419790588]]]
[Edit]: Ideally you should change your JSON to be structured better for unmarshalling. If you do not have control on it, then your corresponding Go structure is just embedded maps of string to interfaces, which you will have to manually type cast and access.

Unmarshalling complex json in Go

So I am trying to fetch the analytics of an app by pinging and endpoint. I make the GET request which is successfull (no errors there) but I am unable to decode the JSON
I need to to decode the following json into structs
{
"noResultSearches": {
"results": [
{
"count": 1,
"key": "\"note 9\""
},
{
"count": 1,
"key": "nokia"
}
]
},
"popularSearches": {
"results": [
{
"count": 4,
"key": "6"
},
{
"count": 2,
"key": "\"note 9\""
},
{
"count": 1,
"key": "nokia"
}
]
},
"searchVolume": {
"results": [
{
"count": 7,
"key": 1537401600000,
"key_as_string": "2018/09/20 00:00:00"
}
]
}
}
For which I am using the following structs
type analyticsResults struct {
Count int `json:"count"`
Key string `json:"key"`
}
type analyticsVolumeResults struct {
Count int `json:"count"`
Key int64 `json:"key"`
DateAsStr string `json:"key_as_string"`
}
type analyticsPopularSearches struct {
Results []analyticsResults `json:"results"`
}
type analyticsNoResultSearches struct {
Results []analyticsResults `json:"results"`
}
type analyticsSearchVolume struct {
Results []analyticsVolumeResults `json:"results"`
}
type overviewAnalyticsBody struct {
NoResultSearches analyticsNoResultSearches `json:"noResultSearches"`
PopularSearches analyticsPopularSearches `json:"popularSearches"`
SearchVolume analyticsSearchVolume `json:"searchVolume"`
}
I make a GET request to an endpoint and then use the response body to decode the json but I get an error. Following is a part of the code that stays in my ShowAnalytics function
func ShowAppAnalytics(app string) error {
spinner.StartText("Fetching app analytics")
defer spinner.Stop()
fmt.Println()
req, err := http.NewRequest("GET", "<some-endpoint>", nil)
if err != nil {
return err
}
resp, err := session.SendRequest(req)
if err != nil {
return err
}
spinner.Stop()
var res overviewAnalyticsBody
dec := json.NewDecoder(resp.Body)
err = dec.Decode(&res)
if err != nil {
return err
}
fmt.Println(res)
return nil
}
json: cannot unmarshal array into Go struct field
overviewAnalyticsBody.noResultSearches of type
app.analyticsNoResultSearches
What am I doing wrong here? Why do I get this error?
EDIT: After you edited, your current code works as-is. Check it out here: Go Playground.
Original answer follows.
There is some inconsistency between the code you posted and the error you get.
I tried it on the Go Playground (here's your version), and I get the following error:
json: cannot unmarshal number into Go struct field analyticsVolumeResults.key of type string
We get this error because in the JSON searchVolume.results.key is a number:
"key": 1537401600000,
And you used string in the Go model:
Key string `json:"key"`
If we change it to int64:
Key int64 `json:"key"`
It works, and prints (try it on the Go Playground):
{{[{1 "note 9"} {1 nokia}]} {[{4 6} {2 "note 9"} {1 nokia}]} {[{7 1537401600000 2018/09/20 00:00:00}]}}
If that key may sometimes be a number and sometimes a string, you may also use json.Number in the Go model:
Key json.Number `json:"key"`

Unmarshalling dynamic JSON, ignoring known fields

I am trying to unmarshal JSON of the following format:
{
"fixedString": {
"uselessStuff": {},
"alsoUseless": {},
"dynamicField": [
{ "name": "jack" }
],
"dynamicToo": [
{ "name": "jill" }
]
}
}
I would like to drop the fields "uselessStuff" and "alsoUseless", and get everything else. The other keys are user-defined and can take any value.
I can remove the fields I don't want using a custom UnmarshalJSON (based on this answer), but I have a feeling that this is unnecessarily complicated:
package main
import (
"encoding/json"
"fmt"
)
type Response struct {
Things map[string]interface{} `json:"fixedString"`
}
type _R Response
func (f *Response) UnmarshalJSON(bs []byte) (err error) {
foo := _R{}
if err = json.Unmarshal(bs, &foo); err == nil {
*f = Response(foo)
}
delete(f.Things, "uselessStuff")
delete(f.Things, "alsoUseless")
return err
}
func main() {
j := []byte(`{ "fixedString": { "uselessStuff": {}, "alsoUseless": {}, "dynamicField": [ { "name": "jack" } ], "dynamicToo": [ { "name": "jill" } ] } }`)
var r Response
err := json.Unmarshal(j, &r)
if err != nil {
panic(err.Error())
}
for x, y := range r.Things {
fmt.Println(x, y)
}
}
Is there a way to ignore those two keys using annotations, rather than deleting them in a custom UnmarshalJSON function (and having to add the extra type alias _R to avoid a stack overflow)?
You could remove your "uselessStuff" and "alsoUseless" from the map and use them as unexported (lowercase) fields in your struct. Most likely not interface{}
json package ignores unexported fields
type Response struct {
Things map[string]interface{} `json:"fixedString"`
uselessStuff interface{}
alsoUseless interface{}
}

Unmarshalling json arrays as json objects

I have to unmarshal a series of Json objects, but one of the objects contain a json array which is not really structured in a good way.
"labels": [
{
"key": "owner",
"value": "harry"
},
{
"key": "group",
"value": "student"
}
]
I am unmarshalling it using this struct -
type StudentDetails struct {
Id string `json:"id"`
Name string `json:"name"`
Labels []Label `json:"labels,omitempty"`
}
type Label struct {
Key string `json:"key"`
Value string `json:"value"`
}
And I have to access it using x.Labels[0].key == "owner" inside a for loop which is very annoying.
I want to be able to do x.Labels.Owner == "harry" instead. How do I go about achieving this? The rest of JSON is unmarshalled fine using the default unmarshal function, so I don't think writing custom function will be good option.
With the constraints you have here, this is about as close as you will get (run in playground):
package main
import (
"encoding/json"
"fmt"
)
func main() {
j := `
{
"id": "42",
"name": "Marvin",
"labels": [
{
"key": "owner",
"value": "harry"
},
{
"key": "group",
"value": "student"
}
]
}`
d := StudentDetails{}
err := json.Unmarshal([]byte(j), &d)
if err != nil {
panic(err)
}
fmt.Println(d.Labels["owner"])
fmt.Println(d.Labels["group"])
}
type StudentDetails struct {
Id string `json:"id"`
Name string `json:"name"`
Labels Labels `json:"labels"`
}
type Labels map[string]string
func (l *Labels) UnmarshalJSON(b []byte) error {
a := []map[string]string{}
err := json.Unmarshal(b, &a)
if err != nil {
return err
}
t := map[string]string{}
for _, m := range a {
t[m["key"]] = m["value"]
}
*l = t
return nil
}
How about to define custom []Label type and add function on it.
For instance
type Labels []Label
func (l Labels) Owner() string {
if len(l) > 1 {
return l[0].Value
}
return ""
}

Custom config loading in Go

I'm using JSON files to store/load my config. Let's say I have the following:
type X interface
// implements interface X
type Y struct {
Value string
}
// implements interface X
type Z struct {
Value string
}
type Config struct {
interfaceInstance X `json:"X"`
}
Config file example:
{
"config1": {
"X": {
"type": "Z",
"Value": "value_1"
}
},
"config2": {
"X": {
"type": "Y",
"Value": "value_2"
}
}
}
I want to be able to define config files something like this example, and be able to dynamically load the JSON as either struct Y or struct Z. Any suggestions on how to accomplish this? I'm using a simple json.Decoder to load the JSON as a struct.
decoder := json.NewDecoder(file)
err = decoder.Decode(&config)
One possible strategy would be to implement json.Unmarshaler for the Config type in such a way that you first unmarshal into a generic object and inspect the "type" attribute then, switching on the type string, unmarshal the same byte array into the known type and assign to the "interfaceInstance" member of the config.
For example (Go Playground):
// Note the slightly different JSON here...
var jsonstr = `{
"config1": {
"type": "Z",
"Value": "value_1"
},
"config2": {
"type": "Y",
"Value": "value_2"
}
}`
func main() {
config := map[string]Config{}
err := json.Unmarshal([]byte(jsonstr), &config)
if err != nil {
panic(err)
}
fmt.Printf("OK: %#v\n", config)
// OK: map[string]main.Config{
// "config1": main.Config{interfaceInstance:main.Z{Value:"value_1"}},
// "config2": main.Config{interfaceInstance:main.Y{Value:"value_2"}},
// }
}
func (c *Config) UnmarshalJSON(bs []byte) error {
// Unmarshal into an object to inspect the type.
var obj map[string]interface{}
err := json.Unmarshal(bs, &obj)
if err != nil {
return err
}
// Unmarshal again into the target type.
configType := obj["type"].(string)
switch configType {
case "Y":
var y Y
if err = json.Unmarshal(bs, &y); err == nil {
c.interfaceInstance = y
}
case "Z":
var z Z
if err = json.Unmarshal(bs, &z); err == nil {
c.interfaceInstance = z
}
default:
return fmt.Errorf("unexpected type %q", configType)
}
return err
}