I got log data source looks like:
{"LogtypeA":{"content":"listen_logs":[{"useful_key1":val1,"useful_key2":val2},{"useful_key1":val1}]}}
Then I use simplejson to parse them. The value of listen_logs is a slice that contain at least one map.
The code is:
for _, v := range js.Get("LogTypeA").Get("content").Get("listen_logs").MustArray() {
_obj := reflect.ValueOf(v)
fmt.Println(_obj.Kind())
}
If I replace MustArray() with Array(), it will report incorrect variable and constant declarations.
Using reflect module I will find that the _obj.Kind() is map, but I can't use any indexing method to get values from v such as:
_val1 := v["useful_key1"]
or
for i, v := range v {...}
Because type interface{} does not support indexing.
So how should I extract those useful keyX from those logs?
You need to perform a type assertion on the resulting map:
new_v := v.(map[string]interface{})
_val1 := new_v["useful_key1"]
Or, if you're not sure it will be a map:
new_v, ok := v.(map[string]interface{})
if ok {
// it was a map from strings to something
_val1 := new_v["useful_key1"]
} else {
print("Could not interpret v as a map.")
}
Since it's initially an empty interface, i.e. interface{}, that means it can be any type. So, you need to specify that you want to treat it as a map before trying to access the items.
Related
Let's say I have below json raw message, I unmarshal as component struct, taking json value out,v.id and v.name. Passed to some dfs function and return me with good things. I then assign them back to id and name.
I realize that within the for loop, it has been updated to good things, however, when I exit for loop, it doesn't update the original JSON raw message. Is there any way to update original JSON message instead of just within the loop?
if err := json.Unmarshal(inputs.Components, &component); err != nil {
panic(err)
}
for _, v := range component {
if _, ok := map[v.id+"-"+v.Name]; ok {
var c string
var m string
raw_info1, raw_info2 := dfs(v.id, v.Name, map, &c, &m)
v.id = raw_info1
v.name = raw_info2
}
}
I assume component in this case is a map and not a struct, else you cannot iterate like that.
When you range over components, you get the key and value "by value". As such, what you modify is not an element of components, but their copy.
To modify the components you should rewrite you logic like:
component[key] = v
I actually marshall back so that it will modify the value inside unmarshall raw json object
I have a general enough function for going through a map[string] and getting all keys:
i := 0
keys := make([]string, len(input))
for k := range input {
keys[i] = k
i++
}
return keys
My problem is I have two different inputs I want to throw in here, a map[string]MyStruct and map[string][][]float64. Whenever I've tried having the input to the func as map[string]interface{}, go resists all my attempts to try to cast the map[string]MyStruct as a map[string]interface{}. Is there a way I can do this without needing to have two functions, one with map[string]MyStruct as input, and one with map[string][][]float64? The contents of the map[string] don't matter at this point, because I'm just trying to get all the keys of them for use later in the code. This needs to be a function that's called; We're using Sonar, and it's set to refuse code duplication, so I can't have this code snippet duplicated.
Until next Go version brings us generics there're several ways to cope with it.
Duplicate code
Use code generation - design some template and then go on build will fill it for you.
Use interface{} as an input type of a function and then use reflection to guess which type was given to a function.
I'm pretty sure in this case general code will be more complicated than 2 separate functions.
func getKeys(input interface{}) []string {
switch inp := input.(type) {
case map[string]MyStruct:
keys := make([]string, 0, len(inp))
for k := range inp {
keys = append(keys, k)
}
return keys
case map[string][]float64:
...
default:
fmt.Printf("I don't know about type %T!\n", v)
}
You cannot use input type map[string]interface{} to place map[string]string or map[string][]string go won't copy each value to new type. You may take the most general type interface{} and then cast it.
I'm trying to consume a rest endpoint in my golang project. The JSON structure is pretty large and is semi-structured, so I'm using reflection to iterate over it and grab the information that I am interested in.
Here is how I'm unmarshal-ing the response:
var m map[string]interface{}
json.Unmarshal(bytes, &m)
But the part I'm stuck at is - I'm iterating through a slice of maps (i think) but I'm unable to get the keys or values of the map. Here is the code in question.
if val, ok := m["messages"]; ok {
s := reflect.ValueOf(val)
if s.Kind() == reflect.Slice {
for i := 0; i < s.Len(); i++ {
item := s.Index(i)
fmt.Println("VALUE = ", item.Elem())
fmt.Println("KIND = ", item.Kind())
}
}
return
}
When I run the code the value that is displayed looks like a map:
map[decoration_stats:<nil> highlight_ranges:map[] index:graylog_7 message:map[_id:49272694-1834-11ea-8928-0242ac120004 docker:{"container_id":"0f9d97722c25240c6f99487b247b2416177a749de47d661cd661334514e0e74f"} facility:fluentd gl2_message_id:01DVDSM9VSDQ5PF81T4C31NSH6....
And the kind is:
KIND = interface
I tried various things like:
for _, e := range val.MapKeys() {
v := val.MapIndex(e)
fmt.Println(v)
}
But the code panics with:
panic: reflect: call of reflect.Value.MapKeys on interface Value
Sorry, I'm somewhat new to golang but have used other static typed language, mainly Java, when it comes to any reflection type programming.
My question is how to can I convert this interface to a map or some concrete type so that I can use it. Any help would be greatly appreciated.
Using reflection is an inefficient way to do this. JSON unmarshal, when used with an interface (and map[string]interface{}) produces a limited set of types, and you can use type assertions or a type-switch to deal with it:
if val, ok := m["messages"]; ok {
switch v:=val.(type) {
case map[string]interface{}: // JSON object
for key, value:=range v {
}
case []interface{}: // JSON array
for i,node:=range v {
}
case string: // string value
case float64: // numeric value
case bool: // boolean value
case json.Number: // If you use json.Decoder with UseNumber()
}
}
I was working with json.Unmarshal and came across the following quirk. When running the below code, I get the error json: Unmarshal(non-pointer map[string]string)
func main() {
m := make(map[string]string)
data := `{"foo": "bar"}`
err := json.Unmarshal([]byte(data), m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m)
}
Playground
Looking at the documentation for json.Unmarshal, there is seemingly no indication that a pointer is required. The closest I can find is the following line
Unmarshal parses the JSON-encoded data and stores the result in the value pointed to by v.
The lines regarding the protocol Unmarshal follows for maps are similarly unclear, as it makes no reference to pointers.
To unmarshal a JSON object into a map, Unmarshal first establishes a map to use. If the map is nil, Unmarshal allocates a new map. Otherwise Unmarshal reuses the existing map, keeping existing entries. Unmarshal then stores key-value pairs from the JSON object into the map. The map's key type must either be a string, an integer, or implement encoding.TextUnmarshaler.
Why must I pass a pointer to json.Unmarshal, especially if maps are already reference types? I know that if I pass a map to a function, and add data to the map, the underlying data of the map will be changed (see the following playground example), which means that it shouldn't matter if I pass a pointer to a map. Can someone clear this up?
As stated in the documentation:
Unmarshal uses the inverse of the encodings that Marshal uses, allocating maps, slices, and pointers as necessary, with ...
Unmarshal may allocates the variable(map, slice, etc.). If we pass a map instead of pointer to a map, then the newly allocated map won't be visible to the caller. The following examples (Go Playground) demonstrates this:
package main
import (
"fmt"
)
func mapFunc(m map[string]interface{}) {
m = make(map[string]interface{})
m["abc"] = "123"
}
func mapPtrFunc(mp *map[string]interface{}) {
m := make(map[string]interface{})
m["abc"] = "123"
*mp = m
}
func main() {
var m1, m2 map[string]interface{}
mapFunc(m1)
mapPtrFunc(&m2)
fmt.Printf("%+v, %+v\n", m1, m2)
}
in which the output is:
map[], map[abc:123]
If the requirement says that a function/method may allocate a variable when necessary and the newly allocated variable need to be visible to the caller, the solution will be: (a) the variable must be in function's return statement or (b) the variable can be assigned to the function/method argument. Since in go everything is pass by value, in case of (b), the argument must be a pointer. The following diagram illustrates what happen in the above example:
At first, both map m1 and m2 point to nil.
Calling mapFunc will copy the value pointed by m1 to m resulting m will also point to nil map.
If in (1) the map already allocated, then in (2) the address of underlying map data structure pointed by m1 (not the address of m1) will be copied to m. In this case both m1 and m point to the same map data structure, thus modifying map items through m1 will also be visible to m.
In the mapFunc function, new map is allocated and assigned to m. There is no way to assign it to m1.
In case of pointer:
When calling mapPtrFunc, the address of m2 will be copied to mp.
In the mapPtrFunc, new map is allocated and assigned to *mp (not mp). Since mp is pointer to m2, assigning the new map to *mp will change the value pointed by m2. Note that the value of mp is unchanged, i.e. the address of m2.
The other key part of the documentation is this:
To unmarshal JSON into a pointer, Unmarshal first handles the case of
the JSON being the JSON literal null. In that case, Unmarshal sets the
pointer to nil. Otherwise, Unmarshal unmarshals the JSON into the
value pointed at by the pointer. If the pointer is nil, Unmarshal
allocates a new value for it to point to.
If Unmarshall accepted a map, it would have to leave the map in the same state whether the JSON were null or {}. But by using pointers, there's now a difference between the pointer being set to nil and it pointing to an empty map.
Note that in order for Unmarshall to be able to "set the pointer to nil", you actually need to pass in a pointer to your map pointer:
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
var m *map[string]string
data := `{}`
err := json.Unmarshal([]byte(data), &m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m)
data = `null`
err = json.Unmarshal([]byte(data), &m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m)
data = `{"foo": "bar"}`
err = json.Unmarshal([]byte(data), &m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m)
}
This outputs:
&map[]
<nil>
&map[foo:bar]
Your viewpoint is no different than saying "a slice is nothing but a pointer". Slices (and maps) use pointers to make them lightweight, yes, but there are still more things that make them work. A slice contains info about its length and capacity for example.
As for why this happens, from a code perspective, the last line of json.Unmarshal calls d.unmarshal(), which executes the code in lines 176-179 of decode.go. It basically says "if the value isn't a pointer, or is nil, return an InvalidUnmarshalError."
The docs could probably be clearer about things, but consider a couple of things:
How would the JSON null value be assigned to the map as nil if you don't pass a pointer to the map? If you require the ability to modify the map itself (rather than the items in the map), then it makes sense to pass a pointer to the item that needs modified. In this case, it's the map.
Alternately, suppose you passed a nil map to json.Unmarshal. Values will be unmarshaled as necessary after the code json.Unmarshal uses eventually calls the equivalent of make(map[string]string). However, you still have a nil map in your function because your map pointed to nothing. There's no way to fix this other than to pass a pointer to the map.
However, let's say there was no need to pass the address of your map because "it's already a pointer", and you've already initialized the map, so it's not nil. What happens then? Well, if I bypass the test in the lines I linked earlier by changing line 176 to read if rv.Kind() != reflect.Map && rv.Kind() != reflect.Ptr || rv.IsNil() {, then this can happen:
`{"foo":"bar"}`: false map[foo:bar]
`{}`: false map[]
`null`: panic: reflect: reflect.Value.Set using unaddressable value [recovered]
panic: interface conversion: string is not error: missing method Error
goroutine 1 [running]:
json.(*decodeState).unmarshal.func1(0xc420039e70)
/home/kit/jstest/src/json/decode.go:172 +0x99
panic(0x4b0a00, 0xc42000e410)
/usr/lib/go/src/runtime/panic.go:489 +0x2cf
reflect.flag.mustBeAssignable(0x15)
/usr/lib/go/src/reflect/value.go:228 +0xf9
reflect.Value.Set(0x4b8b00, 0xc420012300, 0x15, 0x4b8b00, 0x0, 0x15)
/usr/lib/go/src/reflect/value.go:1345 +0x2f
json.(*decodeState).literalStore(0xc420084360, 0xc42000e3f8, 0x4, 0x8, 0x4b8b00, 0xc420012300, 0x15, 0xc420000100)
/home/kit/jstest/src/json/decode.go:883 +0x2797
json.(*decodeState).literal(0xc420084360, 0x4b8b00, 0xc420012300, 0x15)
/home/kit/jstest/src/json/decode.go:799 +0xdf
json.(*decodeState).value(0xc420084360, 0x4b8b00, 0xc420012300, 0x15)
/home/kit/jstest/src/json/decode.go:405 +0x32e
json.(*decodeState).unmarshal(0xc420084360, 0x4b8b00, 0xc420012300, 0x0, 0x0)
/home/kit/jstest/src/json/decode.go:184 +0x224
json.Unmarshal(0xc42000e3f8, 0x4, 0x8, 0x4b8b00, 0xc420012300, 0x8, 0x0)
/home/kit/jstest/src/json/decode.go:104 +0x148
main.main()
/home/kit/jstest/src/jstest/main.go:16 +0x1af
Code leading to that output:
package main
// Note "json" is the local copy of the "encoding/json" source that I modified.
import (
"fmt"
"json"
)
func main() {
for _, data := range []string{
`{"foo":"bar"}`,
`{}`,
`null`,
} {
m := make(map[string]string)
fmt.Printf("%#q: ", data)
if err := json.Unmarshal([]byte(data), m); err != nil {
fmt.Println(err)
} else {
fmt.Println(m == nil, m)
}
}
}
The key is this bit here:
reflect.Value.Set using unaddressable value
Because you passed a copy of the map, it's unaddressable (i.e. it has a temporary address or even no address from the low-level machine perspective). I know of one way around this (x := new(Type) followed by *x = value, except using the reflect package), but it doesn't actually solve the problem; you're creating a local pointer that can't be returned to the caller and using it instead of your original storage location!
So now try a pointer:
if err := json.Unmarshal([]byte(data), m); err != nil {
fmt.Println(err)
} else {
fmt.Println(m == nil, m)
}
Output:
`{"foo":"bar"}`: false map[foo:bar]
`{}`: false map[]
`null`: true map[]
Now it works. Bottom line: use pointers if the object itself may be modified (and the docs say it might be, e.g. if null is used where an object or array (map or slice) is expected.
Scenario:
Consider the following is the JSON :
{
"Bangalore_City": "35_Temperature",
"NewYork_City": "31_Temperature",
"Copenhagen_City": "29_Temperature"
}
If you notice, the data is structured in such a way that there is no hard-coded keys mentioning City/Temperature its basically just values.
Issue: I am not able to parse any JSON which is dynamic.
Question: Could anyone have found solution for this kind of JSON parsing? I tried go-simplejson, gabs & default encoding/json but no luck.
Note:
The above JSON is just for sample. And there are lot of applications which are using the current API, So I do not want to change how the data is structured. I mean I can't change to something as follows:
[{
"City_Name":"Bangalore",
"Temperature": "35"
},...]
Then I can define struct
type TempData struct {
City_Name string
Temperature string
}
You can unmarshal into a map[string]string for example:
m := map[string]string{}
err := json.Unmarshal([]byte(input), &m)
if err != nil {
panic(err)
}
fmt.Println(m)
Output (wrapped):
map[Bangalore_City:35_Temperature NewYork_City:31_Temperature
Copenhagen_City:29_Temperature]
Try it on the Go Playground.
This way no matter what the keys or values are, you will have all pairs in a map which you can print or loop over.
Also note that although your example contained only string values, but if the value type is varying (e.g. string, numbers etc.), you may use interface{} for the value type, in which case your map would be of type map[string]interface{}.
Also note that I created a library to easily work with such dynamic objects which may be a great help in these cases: github.com/icza/dyno.
Standard encoding/json is good for the majority of use cases, but it may be quite slow comparing to alternative solutions. If you need performance, try using fastjson. It parses arbitrary JSONs without the need for creating structs or maps matching the JSON schema.
See the example code below. It iterates over all the (key, value) pairs of the JSON object:
var p fastjson.Parser
v, err := p.Parse(input)
if err != nil {
log.Fatal(err)
}
// Visit all the items in the top object
v.GetObject().Visit(func(k []byte, v *fastjson.Value) {
fmt.Printf("key=%s, value=%s\n", k, v)
// for nested objects call Visit again
if string(k) == "nested" {
v.GetObject().Visit(func(k []byte, v *fastjson.Value) {
fmt.Printf("nested key=%s, value=%s\n", k, v)
})
}
})
Just to add a general answer to how any valid JSON can be parsed; var m interface{} works for all types. That includes map (which OP asked for) arrays, strings, numbers and nested structures.
var m interface{}
err := json.Unmarshal([]byte(input), &m)
if err != nil {
panic(err)
}
fmt.Println(m)
Also consider using gabs package https://github.com/Jeffail/gabs
"Gabs is a small utility for dealing with dynamic or unknown JSON structures in Go"