I have a general enough function for going through a map[string] and getting all keys:
i := 0
keys := make([]string, len(input))
for k := range input {
keys[i] = k
i++
}
return keys
My problem is I have two different inputs I want to throw in here, a map[string]MyStruct and map[string][][]float64. Whenever I've tried having the input to the func as map[string]interface{}, go resists all my attempts to try to cast the map[string]MyStruct as a map[string]interface{}. Is there a way I can do this without needing to have two functions, one with map[string]MyStruct as input, and one with map[string][][]float64? The contents of the map[string] don't matter at this point, because I'm just trying to get all the keys of them for use later in the code. This needs to be a function that's called; We're using Sonar, and it's set to refuse code duplication, so I can't have this code snippet duplicated.
Until next Go version brings us generics there're several ways to cope with it.
Duplicate code
Use code generation - design some template and then go on build will fill it for you.
Use interface{} as an input type of a function and then use reflection to guess which type was given to a function.
I'm pretty sure in this case general code will be more complicated than 2 separate functions.
func getKeys(input interface{}) []string {
switch inp := input.(type) {
case map[string]MyStruct:
keys := make([]string, 0, len(inp))
for k := range inp {
keys = append(keys, k)
}
return keys
case map[string][]float64:
...
default:
fmt.Printf("I don't know about type %T!\n", v)
}
You cannot use input type map[string]interface{} to place map[string]string or map[string][]string go won't copy each value to new type. You may take the most general type interface{} and then cast it.
I was working with json.Unmarshal and came across the following quirk. When running the below code, I get the error json: Unmarshal(non-pointer map[string]string)
func main() {
m := make(map[string]string)
data := `{"foo": "bar"}`
err := json.Unmarshal([]byte(data), m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m)
}
Playground
Looking at the documentation for json.Unmarshal, there is seemingly no indication that a pointer is required. The closest I can find is the following line
Unmarshal parses the JSON-encoded data and stores the result in the value pointed to by v.
The lines regarding the protocol Unmarshal follows for maps are similarly unclear, as it makes no reference to pointers.
To unmarshal a JSON object into a map, Unmarshal first establishes a map to use. If the map is nil, Unmarshal allocates a new map. Otherwise Unmarshal reuses the existing map, keeping existing entries. Unmarshal then stores key-value pairs from the JSON object into the map. The map's key type must either be a string, an integer, or implement encoding.TextUnmarshaler.
Why must I pass a pointer to json.Unmarshal, especially if maps are already reference types? I know that if I pass a map to a function, and add data to the map, the underlying data of the map will be changed (see the following playground example), which means that it shouldn't matter if I pass a pointer to a map. Can someone clear this up?
As stated in the documentation:
Unmarshal uses the inverse of the encodings that Marshal uses, allocating maps, slices, and pointers as necessary, with ...
Unmarshal may allocates the variable(map, slice, etc.). If we pass a map instead of pointer to a map, then the newly allocated map won't be visible to the caller. The following examples (Go Playground) demonstrates this:
package main
import (
"fmt"
)
func mapFunc(m map[string]interface{}) {
m = make(map[string]interface{})
m["abc"] = "123"
}
func mapPtrFunc(mp *map[string]interface{}) {
m := make(map[string]interface{})
m["abc"] = "123"
*mp = m
}
func main() {
var m1, m2 map[string]interface{}
mapFunc(m1)
mapPtrFunc(&m2)
fmt.Printf("%+v, %+v\n", m1, m2)
}
in which the output is:
map[], map[abc:123]
If the requirement says that a function/method may allocate a variable when necessary and the newly allocated variable need to be visible to the caller, the solution will be: (a) the variable must be in function's return statement or (b) the variable can be assigned to the function/method argument. Since in go everything is pass by value, in case of (b), the argument must be a pointer. The following diagram illustrates what happen in the above example:
At first, both map m1 and m2 point to nil.
Calling mapFunc will copy the value pointed by m1 to m resulting m will also point to nil map.
If in (1) the map already allocated, then in (2) the address of underlying map data structure pointed by m1 (not the address of m1) will be copied to m. In this case both m1 and m point to the same map data structure, thus modifying map items through m1 will also be visible to m.
In the mapFunc function, new map is allocated and assigned to m. There is no way to assign it to m1.
In case of pointer:
When calling mapPtrFunc, the address of m2 will be copied to mp.
In the mapPtrFunc, new map is allocated and assigned to *mp (not mp). Since mp is pointer to m2, assigning the new map to *mp will change the value pointed by m2. Note that the value of mp is unchanged, i.e. the address of m2.
The other key part of the documentation is this:
To unmarshal JSON into a pointer, Unmarshal first handles the case of
the JSON being the JSON literal null. In that case, Unmarshal sets the
pointer to nil. Otherwise, Unmarshal unmarshals the JSON into the
value pointed at by the pointer. If the pointer is nil, Unmarshal
allocates a new value for it to point to.
If Unmarshall accepted a map, it would have to leave the map in the same state whether the JSON were null or {}. But by using pointers, there's now a difference between the pointer being set to nil and it pointing to an empty map.
Note that in order for Unmarshall to be able to "set the pointer to nil", you actually need to pass in a pointer to your map pointer:
package main
import (
"encoding/json"
"fmt"
"log"
)
func main() {
var m *map[string]string
data := `{}`
err := json.Unmarshal([]byte(data), &m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m)
data = `null`
err = json.Unmarshal([]byte(data), &m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m)
data = `{"foo": "bar"}`
err = json.Unmarshal([]byte(data), &m)
if err != nil {
log.Fatal(err)
}
fmt.Println(m)
}
This outputs:
&map[]
<nil>
&map[foo:bar]
Your viewpoint is no different than saying "a slice is nothing but a pointer". Slices (and maps) use pointers to make them lightweight, yes, but there are still more things that make them work. A slice contains info about its length and capacity for example.
As for why this happens, from a code perspective, the last line of json.Unmarshal calls d.unmarshal(), which executes the code in lines 176-179 of decode.go. It basically says "if the value isn't a pointer, or is nil, return an InvalidUnmarshalError."
The docs could probably be clearer about things, but consider a couple of things:
How would the JSON null value be assigned to the map as nil if you don't pass a pointer to the map? If you require the ability to modify the map itself (rather than the items in the map), then it makes sense to pass a pointer to the item that needs modified. In this case, it's the map.
Alternately, suppose you passed a nil map to json.Unmarshal. Values will be unmarshaled as necessary after the code json.Unmarshal uses eventually calls the equivalent of make(map[string]string). However, you still have a nil map in your function because your map pointed to nothing. There's no way to fix this other than to pass a pointer to the map.
However, let's say there was no need to pass the address of your map because "it's already a pointer", and you've already initialized the map, so it's not nil. What happens then? Well, if I bypass the test in the lines I linked earlier by changing line 176 to read if rv.Kind() != reflect.Map && rv.Kind() != reflect.Ptr || rv.IsNil() {, then this can happen:
`{"foo":"bar"}`: false map[foo:bar]
`{}`: false map[]
`null`: panic: reflect: reflect.Value.Set using unaddressable value [recovered]
panic: interface conversion: string is not error: missing method Error
goroutine 1 [running]:
json.(*decodeState).unmarshal.func1(0xc420039e70)
/home/kit/jstest/src/json/decode.go:172 +0x99
panic(0x4b0a00, 0xc42000e410)
/usr/lib/go/src/runtime/panic.go:489 +0x2cf
reflect.flag.mustBeAssignable(0x15)
/usr/lib/go/src/reflect/value.go:228 +0xf9
reflect.Value.Set(0x4b8b00, 0xc420012300, 0x15, 0x4b8b00, 0x0, 0x15)
/usr/lib/go/src/reflect/value.go:1345 +0x2f
json.(*decodeState).literalStore(0xc420084360, 0xc42000e3f8, 0x4, 0x8, 0x4b8b00, 0xc420012300, 0x15, 0xc420000100)
/home/kit/jstest/src/json/decode.go:883 +0x2797
json.(*decodeState).literal(0xc420084360, 0x4b8b00, 0xc420012300, 0x15)
/home/kit/jstest/src/json/decode.go:799 +0xdf
json.(*decodeState).value(0xc420084360, 0x4b8b00, 0xc420012300, 0x15)
/home/kit/jstest/src/json/decode.go:405 +0x32e
json.(*decodeState).unmarshal(0xc420084360, 0x4b8b00, 0xc420012300, 0x0, 0x0)
/home/kit/jstest/src/json/decode.go:184 +0x224
json.Unmarshal(0xc42000e3f8, 0x4, 0x8, 0x4b8b00, 0xc420012300, 0x8, 0x0)
/home/kit/jstest/src/json/decode.go:104 +0x148
main.main()
/home/kit/jstest/src/jstest/main.go:16 +0x1af
Code leading to that output:
package main
// Note "json" is the local copy of the "encoding/json" source that I modified.
import (
"fmt"
"json"
)
func main() {
for _, data := range []string{
`{"foo":"bar"}`,
`{}`,
`null`,
} {
m := make(map[string]string)
fmt.Printf("%#q: ", data)
if err := json.Unmarshal([]byte(data), m); err != nil {
fmt.Println(err)
} else {
fmt.Println(m == nil, m)
}
}
}
The key is this bit here:
reflect.Value.Set using unaddressable value
Because you passed a copy of the map, it's unaddressable (i.e. it has a temporary address or even no address from the low-level machine perspective). I know of one way around this (x := new(Type) followed by *x = value, except using the reflect package), but it doesn't actually solve the problem; you're creating a local pointer that can't be returned to the caller and using it instead of your original storage location!
So now try a pointer:
if err := json.Unmarshal([]byte(data), m); err != nil {
fmt.Println(err)
} else {
fmt.Println(m == nil, m)
}
Output:
`{"foo":"bar"}`: false map[foo:bar]
`{}`: false map[]
`null`: true map[]
Now it works. Bottom line: use pointers if the object itself may be modified (and the docs say it might be, e.g. if null is used where an object or array (map or slice) is expected.
I expected below code to print an object of type struct J, however it prints a map object of type map[string]interface{}. I can feel why it acts like that, however when I run, reflect.ValueOf(i).Kind(), it returns Struct, so it kinda gives me the impression that Unmarshal method should return type J instead of a map. Could anyone enlighten me ?
type J struct {
Text string
}
func main() {
j := J{}
var i interface{} = j
js := "{\"Text\": \"lala\"}"
json.Unmarshal([]byte(js), &i)
fmt.Printf("%#v", i)
}
The type you're passing into Unmarshal is not *J, you're passing in an *interface{}.
When the json package reflects what the type is of the pointer it received, it sees interface{}, so it then uses the default types of the package to unmarshal into, which are
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
There is almost never a reason to use a pointer to an interface. If you find yourself using a pointer to an interface, and you don't know exactly why, then it's probably a mistake. If you want to unmarshal into J, then pass that in directly. If you need to assign that to an intermediary interface, make sure you use a pointer to the original value, not a pointer to its interface.
http://play.golang.org/p/uJDFKfSIxN
j := J{}
var i interface{} = &j
js := "{\"Text\": \"lala\"}"
json.Unmarshal([]byte(js), i)
fmt.Printf("%#v", i)
This is expected behavior: instead of giving json.Unmarshal a pointer to a properly typed place in memory you give it a pointer to a place in memory with type interface{}. It can essentially store anything in there under the type the JSON defines, so it does just that.
See it like this:
Unmarshal gets a place to store the data v with type interface{}
Unmarshal detects a map encoded as JSON
Unmarshal sees that the target type is of type interface{}, creates a go map from it and stores it in v
If you would have given it a different type than interface{} the process would have looked like this:
Unmarshal gets a place to store the data v with type struct main.J
Unmarshal detects a map encoded as JSON
Unmarshal sees that the target type is struct main.J and begins to recursively fit the data to the type
The main point is, that the initial assignment
var i interface{} = j
is completely ignored by json.Unmarshal.
func deserialize(request *http.Request,typ reflect.Type) (interface{}, *httpNet.HandlerError){
data,e:=ioutil.ReadAll(request.Body)
fmt.Println(string(data))
if e !=nil{
return nil,&httpNet.HandlerError{e,"could not read request",http.StatusBadRequest}
}
v:=typ.Elem()
payload:=reflect.New(v).Elem().Interface()
eaa:= json.NewDecoder(request.Body).Decode(payload)
if e!=nil{
fmt.Println(eaa.Error())
}
fmt.Println(payload)
fmt.Println(reflect.ValueOf(payload)
)
return payload,nil
}
to call it:
r,_:= deserialize(request,reflect.TypeOf(&testData{}))
It does not throw errors and looks completely valid operation to me , but the result is an empty structure of expecting type.
Whats the problem with that?
The problem is that you are passing a non pointer instance of the type:
payload:=reflect.New(v).Elem().Interface()
Means "allocate a new pointer to the type, then take the value of it, and extract it as interface{}.
You should just keep it at:
payload:=reflect.New(v).Interface()
BTW It's also redundant that you are passing the type of a pointer, extracting its Elem(), then allocating a pointer. Do something like this:
if type.Kind() == reflect.Ptr {
typ = typ.Elem()
}
payload := reflect.New(typ).Interface()
then you can pass both pointers and non pointers to the function.
Edit: Working playground example: http://play.golang.org/p/TPafxcpIU5
Been having one last issue with my code which involves the .Call function in the reflect package.
So I'm making a call such as this:
params := "some map[string][]string"
in := make([]reflect.Value,0)
return_values := reflect.ValueOf(&controller_ref).MethodByName(action_name).Call(in)
where the method I'm making the .Call to is as follows:
func (c *Controller) Root(params map[string][]string) map[string] string{}
What I don't quite understand is how to manipulate the "in" variable in order to properly pass the map I need to into the function. I see that the second parameter in the make() is the length of the parameter? But I don't quite understand how to format the vars in order to properly pass in my parameter. I am recursively running into the error message:
reflect: Call with too few input arguments
Any help would be much appreciated!
From the Value.Call documentation:
Call calls the function v with the input arguments in. For example, if len(in) == 3, v.Call(in) represents the Go call v(in[0], in[1], in[2]).
So if you want to call a function with one parameter, in must contain one reflect.Value of the
right type, in your case map[string][]string.
The expression
in := make([]reflect.Value,0)
creates a slice with length 0. Passing this to Value.Call will result in the panic you receive as you
need 1 parameter, not zero.
The correct call would be:
m := map[string][]string{"foo": []string{"bar"}}
in := []reflect.Value{reflect.ValueOf(m)}
myMethod.Call(in)
The call is trying to pass zero parameters to a controller that expects one param (in is an empty slice). You need to do something more like in := []reflect.Value{reflect.ValueOf(params)}.
You could also call .Interface() once you've found the method, then use type assertion to get a func you can call directly:
// get a reflect.Value for the method
methodVal := reflect.ValueOf(&controller_ref).MethodByName(action_name)
// turn that into an interface{}
methodIface := methodVal.Interface()
// turn that into a function that has the expected signature
method := methodIface.(func(map[string][]string) map[string]string)
// call the method directly
res := method(params)
(Then you could even cache method in a map keyed by method name, so you wouldn't have to do reflect operations next call. But you don't have to do that for it to work.)