Let's say I have below json raw message, I unmarshal as component struct, taking json value out,v.id and v.name. Passed to some dfs function and return me with good things. I then assign them back to id and name.
I realize that within the for loop, it has been updated to good things, however, when I exit for loop, it doesn't update the original JSON raw message. Is there any way to update original JSON message instead of just within the loop?
if err := json.Unmarshal(inputs.Components, &component); err != nil {
panic(err)
}
for _, v := range component {
if _, ok := map[v.id+"-"+v.Name]; ok {
var c string
var m string
raw_info1, raw_info2 := dfs(v.id, v.Name, map, &c, &m)
v.id = raw_info1
v.name = raw_info2
}
}
I assume component in this case is a map and not a struct, else you cannot iterate like that.
When you range over components, you get the key and value "by value". As such, what you modify is not an element of components, but their copy.
To modify the components you should rewrite you logic like:
component[key] = v
I actually marshall back so that it will modify the value inside unmarshall raw json object
Related
I'm trying to consume a rest endpoint in my golang project. The JSON structure is pretty large and is semi-structured, so I'm using reflection to iterate over it and grab the information that I am interested in.
Here is how I'm unmarshal-ing the response:
var m map[string]interface{}
json.Unmarshal(bytes, &m)
But the part I'm stuck at is - I'm iterating through a slice of maps (i think) but I'm unable to get the keys or values of the map. Here is the code in question.
if val, ok := m["messages"]; ok {
s := reflect.ValueOf(val)
if s.Kind() == reflect.Slice {
for i := 0; i < s.Len(); i++ {
item := s.Index(i)
fmt.Println("VALUE = ", item.Elem())
fmt.Println("KIND = ", item.Kind())
}
}
return
}
When I run the code the value that is displayed looks like a map:
map[decoration_stats:<nil> highlight_ranges:map[] index:graylog_7 message:map[_id:49272694-1834-11ea-8928-0242ac120004 docker:{"container_id":"0f9d97722c25240c6f99487b247b2416177a749de47d661cd661334514e0e74f"} facility:fluentd gl2_message_id:01DVDSM9VSDQ5PF81T4C31NSH6....
And the kind is:
KIND = interface
I tried various things like:
for _, e := range val.MapKeys() {
v := val.MapIndex(e)
fmt.Println(v)
}
But the code panics with:
panic: reflect: call of reflect.Value.MapKeys on interface Value
Sorry, I'm somewhat new to golang but have used other static typed language, mainly Java, when it comes to any reflection type programming.
My question is how to can I convert this interface to a map or some concrete type so that I can use it. Any help would be greatly appreciated.
Using reflection is an inefficient way to do this. JSON unmarshal, when used with an interface (and map[string]interface{}) produces a limited set of types, and you can use type assertions or a type-switch to deal with it:
if val, ok := m["messages"]; ok {
switch v:=val.(type) {
case map[string]interface{}: // JSON object
for key, value:=range v {
}
case []interface{}: // JSON array
for i,node:=range v {
}
case string: // string value
case float64: // numeric value
case bool: // boolean value
case json.Number: // If you use json.Decoder with UseNumber()
}
}
I got log data source looks like:
{"LogtypeA":{"content":"listen_logs":[{"useful_key1":val1,"useful_key2":val2},{"useful_key1":val1}]}}
Then I use simplejson to parse them. The value of listen_logs is a slice that contain at least one map.
The code is:
for _, v := range js.Get("LogTypeA").Get("content").Get("listen_logs").MustArray() {
_obj := reflect.ValueOf(v)
fmt.Println(_obj.Kind())
}
If I replace MustArray() with Array(), it will report incorrect variable and constant declarations.
Using reflect module I will find that the _obj.Kind() is map, but I can't use any indexing method to get values from v such as:
_val1 := v["useful_key1"]
or
for i, v := range v {...}
Because type interface{} does not support indexing.
So how should I extract those useful keyX from those logs?
You need to perform a type assertion on the resulting map:
new_v := v.(map[string]interface{})
_val1 := new_v["useful_key1"]
Or, if you're not sure it will be a map:
new_v, ok := v.(map[string]interface{})
if ok {
// it was a map from strings to something
_val1 := new_v["useful_key1"]
} else {
print("Could not interpret v as a map.")
}
Since it's initially an empty interface, i.e. interface{}, that means it can be any type. So, you need to specify that you want to treat it as a map before trying to access the items.
I'm new to Golang and I am trying to parse large json like the ones you get from an API which have lots of data. The documentation explains how to do this with any json:
b := []byte(`{"Name":"Wednesday","Age":6,"Parents":["Gomez","Morticia"]}`)
var f interface{}
err := json.Unmarshal(b, &f)
m := f.(map[string]interface{})
This works fine, but when I use a json that I get from the Twitter API, like the one at the end of the reference on the Twitter dev site I get this error:
interface conversion: interface {} is []interface {}, not map[string]interface {}
I know that there are similar questions but I couldn't find the answer. Can someone recommend me the better way to solve this?
My go version go1.6.2 linux/amd64.
Thanks !
In this case you are not unmarshalling a single JSON object, but an array of JSON objects which is why you're having an issue parsing the API response. The error your seeing this is telling you the actual type of f. The other example worked because it is a single JSON object, which can be mapped as a map[string]interface{} Take a look at this:
var f []interface{}
err := json.Unmarshal(str, &f)
if err != nil {
fmt.Println(err)
return
}
for _, v := range f {
z := v.(map[string]interface{})
for k2, v2 := range z {
fmt.Println("Key:", k2, "Value:", v2)
}
}
f should be of type []interface{} an array of interfaces. Depending on how you plan on parsing the API response, you can do something like I'm doing in the for loop to present each key, value pair.
I have an application that consumes data from a third-party api. I need to decode the json into a struct, which requires the struct to have json tags of the "incoming" json fields. The outgoing json fields have a different naming convention, so I need different json tags for the encoding.
I will have to do this with many different structs, and each struct might have many fields.
What is the best way to accomplish this without repeating a lot of code?
Example Structs:
// incoming "schema" field names
type AccountIn struct {
OpenDate string `json:"accountStartDate"`
CloseDate string `json:"cancelDate"`
}
// outgoing "schema" field names
type AccountOut struct {
OpenDate string `json:"openDate"`
CloseDate string `json:"closeDate"`
}
Maybe the coming change on Go 1.8 would help you, it will allow to 'cast' types even if its JSON tags definition is different: This https://play.golang.org/p/Xbsoa8SsEk works as expected on 1.8beta, I guess this would simplify your current solution
A bit an uncommon but probably quite well working method would be to use a intermediate format so u can use different readers and writers and therefore different tags. For example https://github.com/mitchellh/mapstructure which allows to convert a nested map structure into struct
types. Pretty similar like json unmarshal, just from a map.
// incoming "schema" field names
type AccountIn struct {
OpenDate string `mapstructure:"accountStartDate" json:"openDate"`
CloseDate string `mapstructure:"cancelDate" json:"closeDate"`
}
// from json to map with no name changes
temporaryMap := map[string]interface{}{}
err := json.Unmarshal(jsonBlob, &temporaryMap)
// from map to structs using mapstructure tags
accountIn := &AccountIn{}
mapstructure.Decode(temporaryMap, accountIn)
Later when writing (or reading) u will use directly the json functions which will then use the json tags.
If it's acceptable to take another round trip through json.Unmarshal and json.Marshal, and you don't have any ambiguous field names within your various types, you could translate all the json keys in one pass by unmarshaling into the generic structures used by the json package:
// map incoming to outgoing json identifiers
var translation = map[string]string{
"accountStartDate": "openDate",
"cancelDate": "closeDate",
}
func translateJS(js []byte) ([]byte, error) {
var m map[string]interface{}
if err := json.Unmarshal(js, &m); err != nil {
return nil, err
}
translateKeys(m)
return json.MarshalIndent(m, "", " ")
}
func translateKeys(m map[string]interface{}) {
for _, v := range m {
if v, ok := v.(map[string]interface{}); ok {
translateKeys(v)
}
}
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
for _, k := range keys {
if newKey, ok := translation[k]; ok {
m[newKey] = m[k]
delete(m, k)
}
}
}
https://play.golang.org/p/nXmWlj7qH9
This might be a Naive Approach but is fairly easy to implement:-
func ConvertAccountInToAccountOut(AccountIn incoming) (AccountOut outcoming){
var outcoming AccountOut
outcoming.OpenDate = incoming.OpenDate
outcoming.CloseDate = incoming.CloseDate
return outcoming
}
var IncomingJSONData AccountIn
resp := getJSONDataFromSource() // Some method that gives you the Input JSON
err1 := json.UnMarshall(resp,&IncomingJSONData)
OutGoingJSONData := ConvertAccountInToAccountOut(IncomingJSONData)
if err1 != nil {
fmt.Println("Error in UnMarshalling JSON ",err1)
}
fmt.Println("Outgoing JSON Data: ",OutGoingJSONData)
Scenario:
Consider the following is the JSON :
{
"Bangalore_City": "35_Temperature",
"NewYork_City": "31_Temperature",
"Copenhagen_City": "29_Temperature"
}
If you notice, the data is structured in such a way that there is no hard-coded keys mentioning City/Temperature its basically just values.
Issue: I am not able to parse any JSON which is dynamic.
Question: Could anyone have found solution for this kind of JSON parsing? I tried go-simplejson, gabs & default encoding/json but no luck.
Note:
The above JSON is just for sample. And there are lot of applications which are using the current API, So I do not want to change how the data is structured. I mean I can't change to something as follows:
[{
"City_Name":"Bangalore",
"Temperature": "35"
},...]
Then I can define struct
type TempData struct {
City_Name string
Temperature string
}
You can unmarshal into a map[string]string for example:
m := map[string]string{}
err := json.Unmarshal([]byte(input), &m)
if err != nil {
panic(err)
}
fmt.Println(m)
Output (wrapped):
map[Bangalore_City:35_Temperature NewYork_City:31_Temperature
Copenhagen_City:29_Temperature]
Try it on the Go Playground.
This way no matter what the keys or values are, you will have all pairs in a map which you can print or loop over.
Also note that although your example contained only string values, but if the value type is varying (e.g. string, numbers etc.), you may use interface{} for the value type, in which case your map would be of type map[string]interface{}.
Also note that I created a library to easily work with such dynamic objects which may be a great help in these cases: github.com/icza/dyno.
Standard encoding/json is good for the majority of use cases, but it may be quite slow comparing to alternative solutions. If you need performance, try using fastjson. It parses arbitrary JSONs without the need for creating structs or maps matching the JSON schema.
See the example code below. It iterates over all the (key, value) pairs of the JSON object:
var p fastjson.Parser
v, err := p.Parse(input)
if err != nil {
log.Fatal(err)
}
// Visit all the items in the top object
v.GetObject().Visit(func(k []byte, v *fastjson.Value) {
fmt.Printf("key=%s, value=%s\n", k, v)
// for nested objects call Visit again
if string(k) == "nested" {
v.GetObject().Visit(func(k []byte, v *fastjson.Value) {
fmt.Printf("nested key=%s, value=%s\n", k, v)
})
}
})
Just to add a general answer to how any valid JSON can be parsed; var m interface{} works for all types. That includes map (which OP asked for) arrays, strings, numbers and nested structures.
var m interface{}
err := json.Unmarshal([]byte(input), &m)
if err != nil {
panic(err)
}
fmt.Println(m)
Also consider using gabs package https://github.com/Jeffail/gabs
"Gabs is a small utility for dealing with dynamic or unknown JSON structures in Go"