Serialize a map using a specific order - json

I have a map that uses string for both key and value. I have an array of keys that specifies the order of the values of the map.
I want to serialize that map to a JSON, but keeping the order defined on the array.
There is a sample code here: http://play.golang.org/p/A52GTDY6Wx
I want to serialize it as:
{
"name": "John",
"age": "20"
}
But if I serialize the map directly, the keys are ordered alphabetically:
{
"age": "20",
"name": "John"
}
I can serialize it as an array of maps, thus keeping the order, however that generates a lot of undesired characters:
[
{
"name": "John"
},
{
"age": "20"
}
]
In my real code I need to serialize the results of a database query which is specified in a text file, and I need to maintain the column order. I cannot use structs because the columns are not known at compile time.
EDIT: I don't need to read the JSON later in the specified order. The generated JSON is meant to be read by people, so I just want it to be as humanly readable as possible.
I could use a custom format but JSON suits me perfectly for this.
Thanks!

You need to implement the json.Marshaler interface on a custom type. This has the advantage of playing well within other struct types.
Sorry, you're always going to have to write a little bit of JSON encoding code.
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
)
type KeyVal struct {
Key string
Val interface{}
}
// Define an ordered map
type OrderedMap []KeyVal
// Implement the json.Marshaler interface
func (omap OrderedMap) MarshalJSON() ([]byte, error) {
var buf bytes.Buffer
buf.WriteString("{")
for i, kv := range omap {
if i != 0 {
buf.WriteString(",")
}
// marshal key
key, err := json.Marshal(kv.Key)
if err != nil {
return nil, err
}
buf.Write(key)
buf.WriteString(":")
// marshal value
val, err := json.Marshal(kv.Val)
if err != nil {
return nil, err
}
buf.Write(val)
}
buf.WriteString("}")
return buf.Bytes(), nil
}
func main() {
dict := map[string]interface{}{
"orderedMap": OrderedMap{
{"name", "John"},
{"age", 20},
},
}
dump, err := json.Marshal(dict)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", dump)
}
Outputs
{"orderedMap":{"name":"John","age":20}}

For that specific requirement you really don't need to use json.Marshal at all, you can simply implement your own function like this:
type OrderedMap map[string]string
func (om OrderedMap) ToJson(order ...string) string {
buf := &bytes.Buffer{}
buf.Write([]byte{'{', '\n'})
l := len(order)
for i, k := range order {
fmt.Fprintf(buf, "\t\"%s\": \"%v\"", k, om[k])
if i < l-1 {
buf.WriteByte(',')
}
buf.WriteByte('\n')
}
buf.Write([]byte{'}', '\n'})
return buf.String()
}
func main() {
om := OrderedMap{
"age": "20",
"name": "John",
}
fmt.Println(om.ToJson("name", "age"))
}

Probably the easiest solution: https://github.com/iancoleman/orderedmap
Although it might be slow as it's mentioned here

Here is a MapSlice implementation similar to what YAML v2 offers. It can do both Marshal and Unmarshal.
https://github.com/mickep76/mapslice-json

Related

handle unstructured JSON data using the standard Go unmarshaller

Some context, I'm designing a backend that will receive JSON post data, but the nature of the data is that it has fields that are unstructured. My general research tells me this is a static language vs unstructured data problem.
Normally if you can create a struct for it if the data is well known and just unmarshal into the struct. I have create custom unmarshaling functions for nested objects.
The issue now is that one of the fields could contain an object with an arbitrary number of keys. To provide some code context:
properties: {
"k1": "v1",
"k2": "v2",
"k3": "v3",
...
}
type Device struct {
id: string,
name: string,
status: int,
properties: <what would i put here?>
}
So its hard to code an explicit unmarshaling function for it. Should put a type of map[string]string{}? How would it work if the values were not all strings then? And what if that object itself had nested values/objects as well?
You can make the Properties field as map[string]interface{} so that it can accommodate different types of values.I created a small code for your scenario as follows:
package main
import (
"encoding/json"
"fmt"
)
type Device struct {
Id string
Name string
Status int
Properties map[string]interface{}
}
func main() {
devObj := Device{}
data := []byte(`{"Id":"101","Name":"Harold","Status":1,"properties":{"key1":"val1"}}`)
if err := json.Unmarshal(data, &devObj); err != nil {
panic(err)
}
fmt.Println(devObj)
devObj2 := Device{}
data2 := []byte(`{"Id":"102","Name":"Thanor","Status":1,"properties":{"k1":25,"k2":"someData"}}`)
if err := json.Unmarshal(data2, &devObj2); err != nil {
panic(err)
}
fmt.Println(devObj2)
devObj3 := Device{}
data3 := []byte(`{"Id":"101","Name":"GreyBeard","Status":1,"properties":{"k1":25,"k2":["data1","data2"]}}`)
if err := json.Unmarshal(data3, &devObj3); err != nil {
panic(err)
}
fmt.Println(devObj3)
}
Output:
{101 Harold 1 map[key1:val1]}
{102 Thanor 1 map[k1:25 k2:someData]}
{101 GreyBeard 1 map[k1:25 k2:[data1 data2]]}
I would use one of the popular Go JSON parsers that don't require parsing to a pre-defined struct. An added benefit, and the primary reason they were created, is that they are much faster than encoding/json because they don't use reflection, interface{} or certain other approaches.
Here are two:
https://github.com/buger/jsonparser - 4.1k GitHub stars
https://github.com/valyala/fastjson - 1.3k GitHub stars
Using github.com/buger/jsonparser, a property could be retrieved using the GetString function:
func GetString(data []byte, keys ...string) (val string, err error)
Here's a full example:
package main
import (
"fmt"
"strconv"
"github.com/buger/jsonparser"
)
func main() {
jsondata := []byte(`
{
"properties": {
"k1": "v1",
"k2": "v2",
"k3": "v3"
}
}`)
for i := 1; i > 0; i++ {
key := "k" + strconv.Itoa(i)
val, err := jsonparser.GetString(jsondata, "properties", key)
if err == jsonparser.KeyPathNotFoundError {
break
} else if err != nil {
panic(err)
}
fmt.Printf("found: key [%s] val [%s]\n", key, val)
}
}
See it run on Go Playground: https://play.golang.org/p/ykAM4gac8zT

Proper json unmarshaling in Go with the empty interface

I'm currently learning golang and (probably as many others before me) I'm trying to properly understand the empty interface.
As an exercise, I'm reading a big json file produced by Postman and trying to access just one field (out of the many available).
Here is a simple representation of the json without the unnecessary fields I don't want to read (but that are still there):
{
"results": [
{
"times": [
1,
2,
3,
4
]
}
]
}
Since the json object is big, I opted out of unmarshaling it with a custom struct, and rather decided to use the empty interface interface{}
After some time, I managed to get some working code, but I'm quite sure this isn't the correct way of doing it.
byteValue, _ := ioutil.ReadAll(jsonFile)
var result map[string]interface{}
err = json.Unmarshal(byteValue, &result)
if err != nil {
log.Fatalln(err)
}
// ESPECIALLY UGLY
r := result["results"].([]interface{})
r1 := r[0].(map[string]interface{})
r2 := r1["times"].([]interface{})
times := make([]float64, len(r2))
for i := range r2 {
times[i] = r2[i].(float64)
}
Is there a better way to navigate through my json object without having to instantiate new variables every time i move deeper and deeper into the object?
Even if the JSON is large, you only have to define the fields you actually care about
You only need to use JSON tags if the keys aren't valid Go
identifiers (keys are idents in this case), even then you can sometimes avoid it by using a map[string]something
Unless you need the sub structs for some function or whatnot, you don't need to define them
Unless you need to reuse the type, you don't even have to define that, you can just define the struct at declaration time
Example:
package main
import (
"encoding/json"
"fmt"
)
const s = `
{
"results": [
{
"times": [1, 2, 3, 4]
}
]
}
`
func main() {
var t struct {
Results []struct {
Times []int
}
}
json.Unmarshal([]byte(s), &t)
fmt.Printf("%+v\n", t) // {Results:[{Times:[1 2 3 4]}]}
}
[...] trying to access just one field (out of the many available).
For this concrete use case I would use a library to query and access to a single value in a known path like:
https://github.com/jmespath/go-jmespath
In the other hand, if you're practicing how to access nested values in a JSON, I would recommend you to give a try to write a recursive function that follows a path in an unknown structure the same way (but simple) like go-jmespath does.
Ok, I challenged myself and spent an hour writing this. It works. Not sure about performance or bugs and it's really limited :)
https://play.golang.org/p/dlIsmG6Lk-p
package main
import (
"encoding/json"
"errors"
"fmt"
"strings"
)
func main() {
// I Just added a bit more of data to the structure to be able to test different paths
fileContent := []byte(`
{"results": [
{"times": [
1,
2,
3,
4
]},
{"times2": [
5,
6,
7,
8
]},
{"username": "rosadabril"},
{"age": 42},
{"location": [41.5933262, 1.8376757]}
],
"more_results": {
"nested_1": {
"nested_2":{
"foo": "bar"
}
}
}
}`)
var content map[string]interface{}
if err := json.Unmarshal(fileContent, &content); err != nil {
panic(err)
}
// some paths to test
valuePaths := []string{
"results.times",
"results.times2",
"results.username",
"results.age",
"results.doesnotexist",
"more_results.nested_1.nested_2.foo",
}
for _, p := range valuePaths {
breadcrumbs := strings.Split(p, ".")
value, err := search(breadcrumbs, content)
if err != nil {
fmt.Printf("\nerror searching '%s': %s\n", p, err)
continue
}
fmt.Printf("\nFOUND A VALUE IN: %s\n", p)
fmt.Printf("Type: %T\nValue: %#v\n", value, value)
}
}
// search is our fantastic recursive function! The idea is to search in the structure in a very basic way, for complex querying use jmespath
func search(breadcrumbs []string, content map[string]interface{}) (interface{}, error) {
// we should never hit this point, but better safe than sorry and we could incurr in an out of range error (check line 82)
if len(breadcrumbs) == 0 {
return nil, errors.New("ran out of breadcrumbs :'(")
}
// flag that indicates if we are at the end of our trip and whe should return the value without more checks
lastBreadcrumb := len(breadcrumbs) == 1
// current breadcrumb is always the first element.
currentBreadcrumb := breadcrumbs[0]
if value, found := content[currentBreadcrumb]; found {
if lastBreadcrumb {
return value, nil
}
// if the value is a map[string]interface{}, go down the rabbit hole, recursion!
if aMap, isAMap := value.(map[string]interface{}); isAMap {
// we are calling ourselves popping the first breadcrumb and passing the current map
return search(breadcrumbs[1:], aMap)
}
// if it's an array of interfaces the thing gets complicated :(
if anArray, isArray := value.([]interface{}); isArray {
for _, something := range anArray {
if aMap, isAMap := something.(map[string]interface{}); isAMap && len(breadcrumbs) > 1 {
if v, err := search(breadcrumbs[1:], aMap); err == nil {
return v, nil
}
}
}
}
}
return nil, errors.New("woops, nothing here")
}

How can mapstructure use a discriminator to decode concrete type

The docs (https://github.com/mitchellh/mapstructure) mention the benefits of mapstructure for decoding
{
"type": "person",
"name": "Mitchell"
}
I'm looking for an example that show the decoding process, e.g. to unmarshal either a Person or Pet class
That library only maps maps to structures. It does not do type-specific decoding. Quoting from its doc:
Perhaps we can't populate a specific structure without first reading the "type" field from the JSON. We could always do two passes over the decoding of the JSON (reading the "type" first, and the rest later). However, it is much simpler to just decode this into a map[string]interface{} structure, read the "type" key, then use something like this library to decode it into the proper structure.
All it offers is that you only have to parse the JSON source once, into a map, then you can decide yourself what type you need to map or unmarshal into. Then you can use the already constructed map to fill the type you need.
See this example:
type Person struct {
Name string `json:"name"`
}
type Credentials struct {
User string `json:"user"`
Password string `json:"password"`
}
func main() {
for _, src := range []string{srcPerson, srcCredentials} {
var m map[string]interface{}
if err := json.Unmarshal([]byte(src), &m); err != nil {
panic(err)
}
switch m["type"] {
case "person":
var p Person
if err := mapstructure.Decode(m, &p); err != nil {
panic(err)
}
fmt.Printf("%T %+v\n", p, p)
case "credentials":
var c Credentials
if err := mapstructure.Decode(m, &c); err != nil {
panic(err)
}
fmt.Printf("%T %+v\n", c, c)
}
}
}
const srcPerson = `{
"type": "person",
"name": "Mitchell"
}`
const srcCredentials = `{
"type": "credentials",
"user": "bob",
"password": "secret"
}`
Output:
main.Person {Name:Mitchell}
main.Credentials {User:bob Password:secret}

How to decode complicated unnamed JSON in Golang

I am currently trying to decode the following JSON structure:
[
{
"2015-08-14 19:29:48-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1a",
"UTC_time": "2015-08-14 23:29:48",
"error": "0"
}
},
{
"2015-08-14 19:37:07-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1b",
"UTC_time": "2015-08-14 23:37:07",
"error": "0"
}
},
{
"2015-08-14 19:44:16-04:00": {
"value": "0.1",
"measurement_tag_id": "0.1.1b",
"UTC_time": "2015-08-14 23:44:16",
"error": "0"
}
}
]
This is to eventually have a slice of Reading structs, formatted as the following:
type reading struct {
Value string `json:"value"`
MTID string `json:"measurement_tag_id"`
UTCTime string `json:"UTC_time"`
Error string `json:"error"`
}
I would then like to add this into an existing structure nested as:
type site struct {
Name string
ID string
Tags []tag
}
type tag struct {
ID string
Readings []reading
}
I've currently been able to create the base structure for sites and tags from a more typical JSON payload with appropriate keys. I have been unsuccessful though in figuring out how to decode the reading JSON. So far the closest I have gotten is via map[string]interface{} chaining, but this feels incredibly clunky and verbose.
Solution so far for reference:
var readingData []interface{}
if err := json.Unmarshal(file, &readingData); err != nil {
panic(err)
}
readings := readingData[0].(map[string]interface{})
firstReading := readings["2015-08-14 19:29:48-04:00"].(map[string]interface{})
fmt.Println(firstReading)
value := firstReading["value"].(string)
error := firstReading["error"].(string)
MTID := firstReading["measurement_tag_id"].(string)
UTCTime := firstReading["UTC_time"].(string)
fmt.Println(value, error, MTID, UTCTime)
While I am not sure if its necessary yet, I would also like to hold on to the arbitrary date keys. My first thought was to create a function that returned a map[string]reading but I am not sure how feasible this is.
Thanks for the help in advance!
You can have your reading type implement the json.Unmarshaler interface.
func (r *reading) UnmarshalJSON(data []byte) error {
type _r reading // same structure, but no methods, avoids infinite calls to this method
m := map[string]_r{}
if err := json.Unmarshal(data, &m); err != nil {
return err
}
for _, v := range m {
*r = reading(v)
}
return nil
}
https://play.golang.org/p/7X1oB77XL4
Another way is to use a slice of maps to parse, then copy to slice of readings, eg:
var readingMaps []map[string]reading //slice of maps of string key to reading value
if err := json.Unmarshal([]byte(data), &readingMaps); err != nil {
panic(err)
}
readings := []reading{}
for _, m := range readingMaps {
for _, r := range m {
readings = append(readings, r)
}
}
play.golang.org/p/jXTdmaZz7s

Golang issue with accessing Nested JSON Array after Unmarshalling

I'm still in the learning process of Go but am hitting a wall when it comes to JSON response arrays. Whenever I try to access a nested element of the "objects" array, Go throws (type interface {} does not support indexing)
What is going wrong and how can I avoid making this mistake in the future?
package main
import (
"encoding/json"
"fmt"
)
func main() {
payload := []byte(`{"query": "QEACOR139GID","count": 1,"objects": [{"ITEM_ID": "QEACOR139GID","PROD_CLASS_ID": "BMXCPGRIPS","AVAILABLE": 19}]}`)
var result map[string]interface{}
if err := json.Unmarshal(payload, &result); err != nil {
panic(err)
}
fmt.Println(result["objects"]["ITEM_ID"])
}
http://play.golang.org/p/duW-meEABJ
edit: Fixed link
As the error says, interface variables do not support indexing. You will need to use a type assertion to convert to the underlying type.
When decoding into an interface{} variable, the JSON module represents arrays as []interface{} slices and dictionaries as map[string]interface{} maps.
Without error checking, you could dig down into this JSON with something like:
objects := result["objects"].([]interface{})
first := objects[0].(map[string]interface{})
fmt.Println(first["ITEM_ID"])
These type assertions will panic if the types do not match. You can use the two-return form, you can check for this error. For example:
objects, ok := result["objects"].([]interface{})
if !ok {
// Handle error here
}
If the JSON follows a known format though, a better solution would be to decode into a structure. Given the data in your example, the following might do:
type Result struct {
Query string `json:"query"`
Count int `json:"count"`
Objects []struct {
ItemId string `json:"ITEM_ID"`
ProdClassId string `json:"PROD_CLASS_ID"`
Available int `json:"AVAILABLE"`
} `json:"objects"`
}
If you decode into this type, you can access the item ID as result.Objects[0].ItemId.
For who those might looking for similar solution like me, https://github.com/Jeffail/gabs provides better solution.
I provide the example here.
package main
import (
"encoding/json"
"fmt"
"github.com/Jeffail/gabs"
)
func main() {
payload := []byte(`{
"query": "QEACOR139GID",
"count": 1,
"objects": [{
"ITEM_ID": "QEACOR139GID",
"PROD_CLASS_ID": "BMXCPGRIPS",
"AVAILABLE": 19,
"Messages": [ {
"first": {
"text": "sth, 1st"
}
},
{
"second": {
"text": "sth, 2nd"
}
}
]
}]
}`)
fmt.Println("Use gabs:")
jsonParsed, _ := gabs.ParseJSON(payload)
data := jsonParsed.Path("objects").Data()
fmt.Println(" Fetch Data: ")
fmt.Println(" ", data)
children, _ := jsonParsed.Path("objects").Children()
fmt.Println(" Children Array from \"Objects\": ")
for key, child := range children {
fmt.Println(" ", key, ": ", child)
children2, _ := child.Path("Messages").Children()
fmt.Println(" Children Array from \"Messages\": ")
for key2, child2 := range children2 {
fmt.Println(" ", key2, ": ", child2)
}
}
}