Go: json encoding struct with pointers is slower than with copies? - json

I have the following test code:
package main
import (
"fmt"
"testing"
"encoding/json"
)
type Coll1 struct {
A string
B string
C string
}
type Coll2 struct {
A *string
B *string
C *string
}
var as = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
var bs = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
var cs = "ccccccccccccccccccccccccccccccccc"
func testBM1(b *testing.B) {
for i := 0; i<b.N; i++ {
json.Marshal(Coll1{as,bs,cs})
}
}
func testBM2(b *testing.B) {
for i := 0; i<b.N; i++ {
json.Marshal(Coll2{&as,&bs,&cs})
}
}
func main() {
fmt.Println(testing.Benchmark(testBM1))
fmt.Println(testing.Benchmark(testBM2))
}
I would expect the second case to run faster since it is using pointers and therefore doesn't have to copy the strings, but in fact it runs at about 4250 ns/op where the first runs near 2800 ns/op. Can anyone shed any light on why this might be?
Edit: Darshan Computing suggested that this may hold true for embedded structs even. A simple test confirms this:
package main
import (
"fmt"
"testing"
"encoding/json"
)
type Coll1 struct {
A,B,C string
}
type Coll1Outer struct {
A,B,C Coll1
}
type Coll2Outer struct {
A,B,C *Coll2
}
type Coll2 struct {
A,B,C *string
}
var as = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
var bs = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
var cs = "ccccccccccccccccccccccccccccccccc"
func testBM1(b *testing.B) {
for i := 0; i<b.N; i++ {
c := Coll1Outer{ Coll1{as,bs,cs},
Coll1{as,bs,cs},
Coll1{as,bs,cs} }
json.Marshal(c)
}
}
func testBM2(b *testing.B) {
for i := 0; i<b.N; i++ {
c := Coll2Outer{ &Coll2{&as,&bs,&cs},
&Coll2{&as,&bs,&cs},
&Coll2{&as,&bs,&cs} }
json.Marshal(c)
}
}
func main() {
fmt.Println(testing.Benchmark(testBM1))
fmt.Println(testing.Benchmark(testBM2))
}
For me this shows the non-pointer struct taking about 12ms/op, while the one with pointers takes 13ms/op. Not a huge difference, but it's interesting that the property still holds.

I notice the biggest percentage difference in ns/op when I set as, bs, and cs to "a", "b", and "c", respectively. As I increase the length of the strings, they approach each other. They seem to always be about 1000 ns/op different.
So I believe all that's going on is that it takes 1000 ns on my machine (1450 on yours) to reflect and follow the pointers. Passing a smaller struct up front doesn't seem to counteract this effect because once the pointers are followed, Marshal still passes the data around internally in the process of generating and returning the equivalent JSON.

Related

JSON tag to decode into a struct in Golang

I have a JSON like this:
{
"add":[{"id": "1234ABCD"}, {"id": "5678EFGH"}]
}
And I have a struct like this:
type ExampleStruct struct {
Added []string
}
I am wondering what JSON tag I should put in my struct so that after I do the JSON decoding (code not shown here) and then call exampleStruct := &ExampleStruct followed by exampleStruct.Added, how can I get ["1234ABCD", "5678EFGH"]?
I tried doing this:
type ExampleStruct struct {
Added []string `json:"add"`
}
But it didn't work.
Use a slice of maps instead of strings, as you have key-value pairs of strings.
type ExampleStruct struct {
Added []map[string]string `json:"add"`
}
Here is a full example:
package main
import (
"bytes"
"encoding/json"
"fmt"
)
func main() {
const code = `{
"add":[{"id": "1234ABCD"}]
}`
type ExampleStruct struct {
Added []map[string]string `json:"add"`
}
var data ExampleStruct
json.NewDecoder(bytes.NewReader([]byte(code))).Decode(&data)
fmt.Println(data)
}
EDIT
Since you want to have only the values of the maps, here is a complete example where Added is a function that can be called on the ExampleStruct. It assumes that each map only contains two strings (id and value):
package main
import (
"bytes"
"encoding/json"
"fmt"
)
func main() {
const code = `{
"add":[{"id": "1234ABCD"}, {"id": "5678EFGH"}]
}`
var data ExampleStruct
json.NewDecoder(bytes.NewReader([]byte(code))).Decode(&data)
fmt.Println(data)
fmt.Println(data.Added())
}
type ExampleStruct struct {
Add []map[string]string `json:"add"`
}
func (e ExampleStruct) Added() []string {
values := make([]string, len(e.Add))
for i := range e.Add {
for _, v := range e.Add[i] {
values[i] = v
}
}
return values
}
have you tried to obtain its key by adding 'id', like this
type ExampleStruct struct {
Added []string json:"add.id"
}
You need to arrange your json such a way that elements of the struct should be accessible directly (without over-engineering it).
package main
import (
"bytes"
"encoding/json"
"fmt"
)
type ExampleStruct struct {
Add []struct {
ID string `json:"id"`
} `json:"add"`
}
func main() {
const code = `{
"add":[{"id": "1234ABCD"}, {"id": "5678EFGH"}]
}`
var data ExampleStruct
json.NewDecoder(bytes.NewReader([]byte(code))).Decode(&data)
fmt.Println(data) //Get all ids: {[{1234ABCD} {5678EFGH}]}
fmt.Println(data.Add[0].ID) //Get 1st ID : 1234ABCD
fmt.Println(data.Add[1].ID) //Get 2nd ID ... and so on.: 5678EFGH
}
You can find the code here https://play.golang.org/p/7tD4fLBewp .
If you have many ids in an array then you can also write a function to loop over the array i.e data.Addand get ids from that.

Golang Function Call in Map

I am a newbie at GO Programming. Here is scenario :-
There exists a JSON file that looks like this :-
{
"template": "linuxbase1",
"checkname": ["check_disk"],
"checkmethod": ["check_disk"]
}
I am Unmarshalling this data into a structure :-
package func1
import (
"io/ioutil"
"os"
"encoding/json"
"fmt"
)
type pluginfunc func() string
type Plugindata struct {
Template string `json:"template"`
Checkname []string `json:"checkname"`
Checkmethod []pluginfunc `json:"checkmethod"`
}
var (
Templatepath = "json_sample1.json"
Templateitems Plugindata
)
func Gettemplatedata() {
tdata, err := ioutil.ReadFile(Templatepath)
if err != nil {
fmt.Printf("Unable to read file %s. Error - %v\n",Templatepath, err.Error())
os.Exit(3)
}
json.Unmarshal(tdata, &Templateitems)
}
The "check_disk" function is here :-
package func1
func check_disk() string {
return "Called check_disk"
}
This is the program with main() :-
package main
import (
"fmt"
"checksexpt/func1"
)
func main() {
func1.Gettemplatedata()
fmt.Printf("Templateitems in Main() => %v\n",func1.Templateitems)
for index,funcname := range func1.Templateitems.Checkmethod {
fmt.Printf("%d = %s\n",index,funcname())
}
}
As expected, when I run main(); I see the error :-
Templateitems in Main() => {linuxbase1 [check_cpu check_disk] [<nil> <nil>]}
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x40115e]
goroutine 1 [running]:
panic(0x50e980, 0xc82000a100)
/opt/go/src/runtime/panic.go:481 +0x3e6
So, I am trying to grab a string from the JSON file and treat it as a function call. That obviously fails ! But, the primary constraint here is that the function names have to be picked from the JSON file. How can I do this ? I know that I can create a static map as follows :-
type checkfunc func() string
var (
Templateitems = map[string]map[string]checkfunc {
"linuxbase1": {
"check_disk": check_disk,
},
}
)
So, A call like - Templateitems["linuxbase1"]["check_disk"]() would work just fine. But, I dont want to create any such static map as the elements in that map needs to keep growing. Any ideas on this?
There is no direct way to parse a function directly from a JSON value. Also, you cannot use string values to refer to variables. So a string check_cpu would not be able to refer to the function with the same name directly.
What you can do instead is parse the json string as is and have a global map for functions. That way, you can call your functions like so:
var funcMap = map[string]pluginfunc{
"check_disk": check_disk,
"check_cpu": check_cpu
}
In your main loop:
for index, funcname := range func1.Templateitems.Checkmethod {
fmt.Printf("%d = %s\n", index, funcMap[funcname]())
}
If however, you really need to put the value in your structure, you can try implementing UnmarshalJSON from the json.Unmarshaler interface. A simple example would be:
type pf map[string]pluginfunc
type Plugindata struct {
Template string `json:"template"`
Checkname []string `json:"checkname"`
Checkmethod pf `json:"checkmethod"`
}
func (p *pf) UnmarshalJSON(data []byte) error {
d := []string{}
if err := json.Unmarshal(data, &d); err != nil {
return err
}
*p = make(pf)
for _, s := range d {
(*p)[s] = funcMap[s]
}
return nil
}
var funcMap = pf{
"check_disk": check_disk,
"check_cpu": check_cpu
}
func main() {
json.Unmarshal(tdata, &Templateitems)
for k, f := range Templateitems.Checkmethod {
fmt.Printf("%s -- %s\n", k, f())
}
}
Working code
Note that this way is not as readable or simple as the first method and it still relies on a function map.
You can read more about json.Unmarshaler here.

Is there a convenient way to convert a JSON-like map[string]interface{} to struct , and vice versa, with tags in Go? [duplicate]

I want to convert a struct to map in Golang. It would also be nice if I could use the JSON tags as keys in the created map (otherwise defaulting to field name).
Edit Dec 14, 2020
Since structs repo was archived, you can use mapstructure instead.
Edit TL;DR version, Jun 15, 2015
If you want the fast solution for converting a structure to map, see the accepted answer, upvote it and use that package.
Happy coding! :)
Original Post
So far I have this function, I am using the reflect package but I don't understand well how to use the package, please bear with me.
func ConvertToMap(model interface{}) bson.M {
ret := bson.M{}
modelReflect := reflect.ValueOf(model)
if modelReflect.Kind() == reflect.Ptr {
modelReflect = modelReflect.Elem()
}
modelRefType := modelReflect.Type()
fieldsCount := modelReflect.NumField()
var fieldData interface{}
for i := 0; i < fieldsCount; i++ {
field := modelReflect.Field(i)
switch field.Kind() {
case reflect.Struct:
fallthrough
case reflect.Ptr:
fieldData = ConvertToMap(field.Interface())
default:
fieldData = field.Interface()
}
ret[modelRefType.Field(i).Name] = fieldData
}
return ret
}
Also I looked at JSON package source code, because it should contain my needed implementation (or parts of it) but don't understand too much.
I also had need for something like this. I was using an internal package which was converting a struct to a map. I decided to open source it with other struct based high level functions. Have a look:
https://github.com/fatih/structs
It has support for:
Convert struct to a map
Extract the fields of a struct to a []string
Extract the values of a struct to a []values
Check if a struct is initialized or not
Check if a passed interface is a struct or a pointer to struct
You can see some examples here: http://godoc.org/github.com/fatih/structs#pkg-examples
For example converting a struct to a map is a simple:
type Server struct {
Name string
ID int32
Enabled bool
}
s := &Server{
Name: "gopher",
ID: 123456,
Enabled: true,
}
// => {"Name":"gopher", "ID":123456, "Enabled":true}
m := structs.Map(s)
The structs package has support for anonymous (embedded) fields and nested structs. The package provides to filter certain fields via field tags.
From struct to map[string]interface{}
package main
import (
"fmt"
"encoding/json"
)
type MyData struct {
One int
Two string
Three int
}
func main() {
in := &MyData{One: 1, Two: "second"}
var inInterface map[string]interface{}
inrec, _ := json.Marshal(in)
json.Unmarshal(inrec, &inInterface)
// iterate through inrecs
for field, val := range inInterface {
fmt.Println("KV Pair: ", field, val)
}
}
go playground here
Here is a function I've written in the past to convert a struct to a map, using tags as keys
// ToMap converts a struct to a map using the struct's tags.
//
// ToMap uses tags on struct fields to decide which fields to add to the
// returned map.
func ToMap(in interface{}, tag string) (map[string]interface{}, error){
out := make(map[string]interface{})
v := reflect.ValueOf(in)
if v.Kind() == reflect.Ptr {
v = v.Elem()
}
// we only accept structs
if v.Kind() != reflect.Struct {
return nil, fmt.Errorf("ToMap only accepts structs; got %T", v)
}
typ := v.Type()
for i := 0; i < v.NumField(); i++ {
// gets us a StructField
fi := typ.Field(i)
if tagv := fi.Tag.Get(tag); tagv != "" {
// set key of map to value in struct field
out[tagv] = v.Field(i).Interface()
}
}
return out, nil
}
Runnable example here.
Note, if you have multiple fields with the same tag value, then you will obviously not be able to store them all within a map. It might be prudent to return an error if that happens.
I like the importable package for the accepted answer, but it does not translate my json aliases. Most of my projects have a helper function/class that I import.
Here is a function that solves my specific problem.
// Converts a struct to a map while maintaining the json alias as keys
func StructToMap(obj interface{}) (newMap map[string]interface{}, err error) {
data, err := json.Marshal(obj) // Convert to a json string
if err != nil {
return
}
err = json.Unmarshal(data, &newMap) // Convert to a map
return
}
And in the main, this is how it would be called...
package main
import (
"fmt"
"encoding/json"
"github.com/fatih/structs"
)
type MyStructObject struct {
Email string `json:"email_address"`
}
func main() {
obj := &MyStructObject{Email: "test#test.com"}
// My solution
fmt.Println(StructToMap(obj)) // prints {"email_address": "test#test.com"}
// The currently accepted solution
fmt.Println(structs.Map(obj)) // prints {"Email": "test#test.com"}
}
package main
import (
"fmt"
"reflect"
)
type bill struct {
N1 int
N2 string
n3 string
}
func main() {
a := bill{4, "dhfthf", "fdgdf"}
v := reflect.ValueOf(a)
values := make(map[string]interface{}, v.NumField())
for i := 0; i < v.NumField(); i++ {
if v.Field(i).CanInterface() {
values[v.Type().Field(i).Name] = v.Field(i).Interface()
} else {
fmt.Printf("sorry you have a unexported field (lower case) value you are trying to sneak past. I will not allow it: %v\n", v.Type().Field(i).Name)
}
}
fmt.Println(values)
passObject(&values)
}
func passObject(v1 *map[string]interface{}) {
fmt.Println("yoyo")
}
I'm a bit late but I needed this kind of feature so I wrote this. Can resolve nested structs. By default, uses field names but can also use custom tags. A side effect is that if you set the tagTitle const to json, you could use the json tags you already have.
package main
import (
"fmt"
"reflect"
)
func StructToMap(val interface{}) map[string]interface{} {
//The name of the tag you will use for fields of struct
const tagTitle = "kelvin"
var data map[string]interface{} = make(map[string]interface{})
varType := reflect.TypeOf(val)
if varType.Kind() != reflect.Struct {
// Provided value is not an interface, do what you will with that here
fmt.Println("Not a struct")
return nil
}
value := reflect.ValueOf(val)
for i := 0; i < varType.NumField(); i++ {
if !value.Field(i).CanInterface() {
//Skip unexported fields
continue
}
tag, ok := varType.Field(i).Tag.Lookup(tagTitle)
var fieldName string
if ok && len(tag) > 0 {
fieldName = tag
} else {
fieldName = varType.Field(i).Name
}
if varType.Field(i).Type.Kind() != reflect.Struct {
data[fieldName] = value.Field(i).Interface()
} else {
data[fieldName] = StructToMap(value.Field(i).Interface())
}
}
return data
}
map := Structpb.AsMap()
// map is the map[string]interface{}

Array of pointers to JSON

In golang I have two dimensional slice of pointers to struct, as shown in code below.
type point struct {
x int
y int
}
type cell struct {
point point
visited bool
walls walls
}
type walls struct {
n bool
e bool
s bool
w bool
}
type maze struct {
cells [][]*cell
solutionStack Stack
}
I would like to serialize cells slice into JSON. But as all the elements are pointers calling encode will give empty JSON. What would be the best way to serialize this slice.
One solution that comes to my mid is to create a local copy of this 2D slice ad replace all pointers with actual struct. It'll work but it is no
I'm not sure if I'm answering your question because the built in JSON package will do the reflection of the pointers automatically. It should "just work". I did notice that you are not exporting the properties in your struct, maybe that is the issue you have? When using reflection, you cannot inspect unexported values.
http://play.golang.org/p/zTuMLBgGWk
package main
import (
"encoding/json"
"fmt"
)
type point struct {
X int
Y int
}
type cell struct {
Point point
Visited bool
Walls walls
}
type walls struct {
N bool
E bool
S bool
W bool
}
type maze struct {
Cells [][]*cell
}
func main() {
m := maze{}
var row1 []*cell
var row2 []*cell
row1 = append(row1, &cell{
Point: point{1, 2},
Walls: walls{N: true},
})
row2 = append(row2, &cell{
Point: point{3, 4},
Walls: walls{E: true},
})
m.Cells = append(m.Cells, row1, row2)
mazeJson, _ := json.MarshalIndent(m, "", " ")
fmt.Println(string(mazeJson))
}

Dynamically create variables of certain type based on string in Go

Simple version
How can you create a variable of a certain type based upon the value of a string?
type ta struct { a int }
type tb struct { b float }
type tc struct { c string }
t := "tb"
v := MagicVarFunc(t) // Returns a new allocated var of type interface{}
v.(tb).b = 8.3
The true example
In my, surprisingly enough, working example below, I am dynamically creating variables based on a string. This is done by registering each struct type in a map with the string being the key and a nil-pointer of the type being the value.
Each type implements an interface with the method New() which returns a new variable of that specific type.
The example below is very close to what I wish to do, where each action has a set of JSON encoded data which will populate the corresponding struct. The way I've structured it is also because I wish to be able to create new stand alone actions that I register to the map.
I am not sure if am abusing the language now.
May anyone give me any pointers if I am completely out of my mind? Is there an obviously easier way?
package main
import (
"fmt"
"encoding/json"
)
// All I require of an action is that it may be executed
type ActionHandler interface {
Exec()
New() ActionHandler
}
// My list of actions
var mActions = make(map[string]ActionHandler)
// Action Exit (leaving the program)
type aExit struct {}
func (s *aExit) Exec() { fmt.Println("Good bye") }
func (s *aExit) New() ActionHandler { return new(aExit) }
func init() {
var a *aExit
mActions[`exit`] = a
}
// Action Say (say a message to someone)
type aSay struct {
To string
Msg string
}
func (s *aSay) Exec() { fmt.Println(`You say, "` + s.Msg + `" to ` + s.To) }
func (s *aSay) New() ActionHandler { return new(aSay) }
func init() {
var a *aSay
mActions[`say`] = a
}
func inHandler(action string, data []byte) {
a := mActions[action].New()
json.Unmarshal(data, &a)
a.Exec()
}
func main(){
inHandler(`say`, []byte(`{"to":"Sonia","msg":"Please help me!"}`))
inHandler(`exit`, []byte(`{}`))
}
You can use reflection to get the zero value of, or to allocate a new value (like new) of a type using reflection, if you can get the Type value at runtime. However, I don't think there is a way to get the Type from a string. You would need to have a value of that type to get the type itself.
I adopted your idea, of using a map. I map the string to the type itself, which you can get using reflect.TypeOf, which gets the type out of an interface value. Then I used reflect.Zero to get the zero value of that type (a convenient value that exists for every type). Then I got the value out as an interface.
package main
import "reflect"
type ta struct { a int }
type tb struct { b float64 }
type tc struct { c string }
var mActions map[string]reflect.Type = make(map[string]reflect.Type)
func init() {
var a ta
mActions[`ta`] = reflect.TypeOf(a)
var b tb
mActions[`tb`] = reflect.TypeOf(b)
var c ta
mActions[`tc`] = reflect.TypeOf(c)
}
func MagicVarFunc(action string) interface{} {
return reflect.Zero(mActions[action]).Interface()
}
func main() {
t := "tb"
v := MagicVarFunc(t) // Returns a new allocated var of type interface{}
x := v.(tb)
x.b = 8.3
}
jorelli's answer is very good. I'm just going to show a few options. Your "true example" looks essentially like command dispatch, with command parameters specified with JSON. To start with simple code that does this,
package main
import (
"encoding/json"
"fmt"
)
func inHandler(action string, data []byte) {
arg := make(map[string]interface{})
json.Unmarshal(data, &arg)
switch action {
case "say":
fmt.Printf("You say, %q to %s\n", arg["msg"], arg["to"])
case "exit":
fmt.Println("Good bye")
}
}
func main() {
inHandler(`say`, []byte(`{"to":"Sonia","msg":"Please help me!"}`))
inHandler(`exit`, []byte(`{}`))
}
Your register new commands by adding cases to the switch statement. Yeah, didn't think you'd like that. So, adding your map and init() idea,
package main
import (
"encoding/json"
"fmt"
)
type jmap map[string]interface{}
var mActions = map[string]func(jmap){}
func init() {
mActions["say"] = func(arg jmap) {
fmt.Printf("You say, %q to %s\n", arg["msg"], arg["to"])
}
}
func init() {
mActions["exit"] = func(jmap) { fmt.Println("Good bye") }
}
func inHandler(action string, data []byte) {
args := make(jmap)
json.Unmarshal(data, &args)
mActions[action](args)
}
func main() {
inHandler(`say`, []byte(`{"to":"Sonia","msg":"Please help me!"}`))
inHandler(`exit`, []byte(`{}`))
}
Now if you wanted, you could put each of those init functions in a separate source file and new commands could be registered by creating a new source file with a new init function.
The rest of the program is simplified with some assumptions that the commands have flat argument lists that the JSON will always encode as an object. This allows you to dispense with separate Go struct definitions for each command. inHandler just creates the same type of object (a map) for all commands, unmarshals into it, and passes it to the command. If you wanted to handle a little more arbitrary JSON, you could unmarshal into an empty interface, and the functions would have to do some extra work to dig out the arguments. If that was too much work and you really wanted to unmarshal directly into a struct, then you arrive near jorelli's solution of making each command function unmarshal its own JSON.
start by defining a function type that does the thing you want:
type Producer func([]byte) interface{}
make a few of them:
func FooProducer(raw []byte) interface{} {
foo := new(Foo)
... // do something to foo
return foo
}
func BarProducter(raw []byte) interface{} {
bar := new(Bar)
... // do something to bar
return bar
}
stick them in a map:
likeThis := map[string]Producer{
"foo": FooProducer,
"bar": BarProducer,
}
and then just do one of these:
myVal := likeThis[someString](raw)
but you probably want to define some interface and make your producer something more like:
type Producer func([]byte) MyAwesomeInterface
since there's probably some common stuff you want to do with those things you're decoding. You also probably want to handle the case of a bad string input, like-a-this:
f, ok := likeThis[someString]
if !ok {
// return, break, panic... something, just get the hell away from here.
}
myVal := f(raw)
The whole notion of inspecting types is kinda cumbersome in Go. It's generally less work to just add new types than it is to try to do reflection gymnastics with the type system.