handle unstructured JSON data using the standard Go unmarshaller - json

Question

Some context, I'm designing a backend that will receive JSON post data, but the nature of the data is that it has fields that are unstructured. My general research tells me this is a static language vs unstructured data problem.
Normally if you can create a struct for it if the data is well known and just unmarshal into the struct. I have create custom unmarshaling functions for nested objects.
The issue now is that one of the fields could contain an object with an arbitrary number of keys. To provide some code context:
properties: {
"k1": "v1",
"k2": "v2",
"k3": "v3",
...
}
type Device struct {
id: string,
name: string,
status: int,
properties: <what would i put here?>
}
So its hard to code an explicit unmarshaling function for it. Should put a type of map[string]string{}? How would it work if the values were not all strings then? And what if that object itself had nested values/objects as well?

Answer 1

You can make the Properties field as map[string]interface{} so that it can accommodate different types of values.I created a small code for your scenario as follows:
package main
import (
"encoding/json"
"fmt"
)
type Device struct {
Id string
Name string
Status int
Properties map[string]interface{}
}
func main() {
devObj := Device{}
data := []byte(`{"Id":"101","Name":"Harold","Status":1,"properties":{"key1":"val1"}}`)
if err := json.Unmarshal(data, &devObj); err != nil {
panic(err)
}
fmt.Println(devObj)
devObj2 := Device{}
data2 := []byte(`{"Id":"102","Name":"Thanor","Status":1,"properties":{"k1":25,"k2":"someData"}}`)
if err := json.Unmarshal(data2, &devObj2); err != nil {
panic(err)
}
fmt.Println(devObj2)
devObj3 := Device{}
data3 := []byte(`{"Id":"101","Name":"GreyBeard","Status":1,"properties":{"k1":25,"k2":["data1","data2"]}}`)
if err := json.Unmarshal(data3, &devObj3); err != nil {
panic(err)
}
fmt.Println(devObj3)
}
Output:
{101 Harold 1 map[key1:val1]}
{102 Thanor 1 map[k1:25 k2:someData]}
{101 GreyBeard 1 map[k1:25 k2:[data1 data2]]}

Answer 2

I would use one of the popular Go JSON parsers that don't require parsing to a pre-defined struct. An added benefit, and the primary reason they were created, is that they are much faster than encoding/json because they don't use reflection, interface{} or certain other approaches.
Here are two:
https://github.com/buger/jsonparser - 4.1k GitHub stars
https://github.com/valyala/fastjson - 1.3k GitHub stars
Using github.com/buger/jsonparser, a property could be retrieved using the GetString function:
func GetString(data []byte, keys ...string) (val string, err error)
Here's a full example:
package main
import (
"fmt"
"strconv"
"github.com/buger/jsonparser"
)
func main() {
jsondata := []byte(`
{
"properties": {
"k1": "v1",
"k2": "v2",
"k3": "v3"
}
}`)
for i := 1; i > 0; i++ {
key := "k" + strconv.Itoa(i)
val, err := jsonparser.GetString(jsondata, "properties", key)
if err == jsonparser.KeyPathNotFoundError {
break
} else if err != nil {
panic(err)
}
fmt.Printf("found: key [%s] val [%s]\n", key, val)
}
}
See it run on Go Playground: https://play.golang.org/p/ykAM4gac8zT