Golang issue with accessing Nested JSON Array after Unmarshalling - json

I'm still in the learning process of Go but am hitting a wall when it comes to JSON response arrays. Whenever I try to access a nested element of the "objects" array, Go throws (type interface {} does not support indexing)
What is going wrong and how can I avoid making this mistake in the future?
package main
import (
"encoding/json"
"fmt"
)
func main() {
payload := []byte(`{"query": "QEACOR139GID","count": 1,"objects": [{"ITEM_ID": "QEACOR139GID","PROD_CLASS_ID": "BMXCPGRIPS","AVAILABLE": 19}]}`)
var result map[string]interface{}
if err := json.Unmarshal(payload, &result); err != nil {
panic(err)
}
fmt.Println(result["objects"]["ITEM_ID"])
}
http://play.golang.org/p/duW-meEABJ
edit: Fixed link

As the error says, interface variables do not support indexing. You will need to use a type assertion to convert to the underlying type.
When decoding into an interface{} variable, the JSON module represents arrays as []interface{} slices and dictionaries as map[string]interface{} maps.
Without error checking, you could dig down into this JSON with something like:
objects := result["objects"].([]interface{})
first := objects[0].(map[string]interface{})
fmt.Println(first["ITEM_ID"])
These type assertions will panic if the types do not match. You can use the two-return form, you can check for this error. For example:
objects, ok := result["objects"].([]interface{})
if !ok {
// Handle error here
}
If the JSON follows a known format though, a better solution would be to decode into a structure. Given the data in your example, the following might do:
type Result struct {
Query string `json:"query"`
Count int `json:"count"`
Objects []struct {
ItemId string `json:"ITEM_ID"`
ProdClassId string `json:"PROD_CLASS_ID"`
Available int `json:"AVAILABLE"`
} `json:"objects"`
}
If you decode into this type, you can access the item ID as result.Objects[0].ItemId.

For who those might looking for similar solution like me, https://github.com/Jeffail/gabs provides better solution.
I provide the example here.
package main
import (
"encoding/json"
"fmt"
"github.com/Jeffail/gabs"
)
func main() {
payload := []byte(`{
"query": "QEACOR139GID",
"count": 1,
"objects": [{
"ITEM_ID": "QEACOR139GID",
"PROD_CLASS_ID": "BMXCPGRIPS",
"AVAILABLE": 19,
"Messages": [ {
"first": {
"text": "sth, 1st"
}
},
{
"second": {
"text": "sth, 2nd"
}
}
]
}]
}`)
fmt.Println("Use gabs:")
jsonParsed, _ := gabs.ParseJSON(payload)
data := jsonParsed.Path("objects").Data()
fmt.Println(" Fetch Data: ")
fmt.Println(" ", data)
children, _ := jsonParsed.Path("objects").Children()
fmt.Println(" Children Array from \"Objects\": ")
for key, child := range children {
fmt.Println(" ", key, ": ", child)
children2, _ := child.Path("Messages").Children()
fmt.Println(" Children Array from \"Messages\": ")
for key2, child2 := range children2 {
fmt.Println(" ", key2, ": ", child2)
}
}
}

Related

handle unstructured JSON data using the standard Go unmarshaller

Some context, I'm designing a backend that will receive JSON post data, but the nature of the data is that it has fields that are unstructured. My general research tells me this is a static language vs unstructured data problem.
Normally if you can create a struct for it if the data is well known and just unmarshal into the struct. I have create custom unmarshaling functions for nested objects.
The issue now is that one of the fields could contain an object with an arbitrary number of keys. To provide some code context:
properties: {
"k1": "v1",
"k2": "v2",
"k3": "v3",
...
}
type Device struct {
id: string,
name: string,
status: int,
properties: <what would i put here?>
}
So its hard to code an explicit unmarshaling function for it. Should put a type of map[string]string{}? How would it work if the values were not all strings then? And what if that object itself had nested values/objects as well?
You can make the Properties field as map[string]interface{} so that it can accommodate different types of values.I created a small code for your scenario as follows:
package main
import (
"encoding/json"
"fmt"
)
type Device struct {
Id string
Name string
Status int
Properties map[string]interface{}
}
func main() {
devObj := Device{}
data := []byte(`{"Id":"101","Name":"Harold","Status":1,"properties":{"key1":"val1"}}`)
if err := json.Unmarshal(data, &devObj); err != nil {
panic(err)
}
fmt.Println(devObj)
devObj2 := Device{}
data2 := []byte(`{"Id":"102","Name":"Thanor","Status":1,"properties":{"k1":25,"k2":"someData"}}`)
if err := json.Unmarshal(data2, &devObj2); err != nil {
panic(err)
}
fmt.Println(devObj2)
devObj3 := Device{}
data3 := []byte(`{"Id":"101","Name":"GreyBeard","Status":1,"properties":{"k1":25,"k2":["data1","data2"]}}`)
if err := json.Unmarshal(data3, &devObj3); err != nil {
panic(err)
}
fmt.Println(devObj3)
}
Output:
{101 Harold 1 map[key1:val1]}
{102 Thanor 1 map[k1:25 k2:someData]}
{101 GreyBeard 1 map[k1:25 k2:[data1 data2]]}
I would use one of the popular Go JSON parsers that don't require parsing to a pre-defined struct. An added benefit, and the primary reason they were created, is that they are much faster than encoding/json because they don't use reflection, interface{} or certain other approaches.
Here are two:
https://github.com/buger/jsonparser - 4.1k GitHub stars
https://github.com/valyala/fastjson - 1.3k GitHub stars
Using github.com/buger/jsonparser, a property could be retrieved using the GetString function:
func GetString(data []byte, keys ...string) (val string, err error)
Here's a full example:
package main
import (
"fmt"
"strconv"
"github.com/buger/jsonparser"
)
func main() {
jsondata := []byte(`
{
"properties": {
"k1": "v1",
"k2": "v2",
"k3": "v3"
}
}`)
for i := 1; i > 0; i++ {
key := "k" + strconv.Itoa(i)
val, err := jsonparser.GetString(jsondata, "properties", key)
if err == jsonparser.KeyPathNotFoundError {
break
} else if err != nil {
panic(err)
}
fmt.Printf("found: key [%s] val [%s]\n", key, val)
}
}
See it run on Go Playground: https://play.golang.org/p/ykAM4gac8zT

Converting all snake_case keys in a json to camelCase keys

In Go, how can we convert snake_case keys in a JSON to camelCase ones recursively?
I am writing one http api in Go. This api fetches data from datastore, does some computation and returns the response as JSON.
The situation is that the JSON document in the datastore (ElasticSearch) is present with snake_case keys while the API response should be camelCase-based (this is just to align with other api standards within the project). The source which inserts into ES can't be modified. So it's only at the api level that the key conversion has to take place.
I have written a struct which is reading JSON from datastore nicely. But how can I convert the keys to camelCase in Go?
The JSON can be nested and all keys have to be converted.
The JSON is arbitrarily large; i.e. some keys are just mapped to type interface{}
I am also using Go's echo framework for writing the api.
Ex.
{
"is_modified" : true,
{ "attribute":
{
"legacy_id" : 12345
}
}
}
TO
{
"isModified" : true,
{ "attribute":
{
"legacyId" : 12345
}
}
}
Any pointers on how to do this in Go?
Struct:
type data_in_es struct {
IsModified bool `json:"is_modified,omitempty"`
Attribute *attribute `json:"attribute,omitempty"`
}
type attribute struct {
LegacyId int `json:"legacy_id,omitempty"`
}
Since Go 1.8 you can define two structs that only differ in their tags and trivially convert between the two:
package main
import (
"encoding/json"
"fmt"
)
type ESModel struct {
AB string `json:"a_b"`
}
type APIModel struct {
AB string `json:"aB"`
}
func main() {
b := []byte(`{
"a_b": "c"
}`)
var x ESModel
json.Unmarshal(b, &x)
b, _ = json.MarshalIndent(APIModel(x), "", " ")
fmt.Println(string(b))
}
https://play.golang.org/p/dcBkkX9zQR
To do this in general, attempt to unmarshal the JSON document into a map. If it succeeds, fix all the keys and recursively call your function for each value in the map. The example below shows how to convert all keys to upper case. Replace fixKey with the snake_case conversion function.
package main
import (
"bytes"
"encoding/json"
"fmt"
"strings"
)
func main() {
// Document source as returned by Elasticsearch
b := json.RawMessage(`{
"a_b": "c",
"d_e": ["d"],
"e_f": {
"g_h": {
"i_j": "k",
"l_m": {}
}
}
}`)
x := convertKeys(b)
buf := &bytes.Buffer{}
json.Indent(buf, []byte(x), "", " ")
fmt.Println(buf.String())
}
func convertKeys(j json.RawMessage) json.RawMessage {
m := make(map[string]json.RawMessage)
if err := json.Unmarshal([]byte(j), &m); err != nil {
// Not a JSON object
return j
}
for k, v := range m {
fixed := fixKey(k)
delete(m, k)
m[fixed] = convertKeys(v)
}
b, err := json.Marshal(m)
if err != nil {
return j
}
return json.RawMessage(b)
}
func fixKey(key string) string {
return strings.ToUpper(key)
}
https://play.golang.org/p/QQZPMGKrlg

Convert json with variable types to strings

I am reading in json from an API response and I ran into an issue in that there are multiple data types inside the json values (strings, null, bool). In addition, some keys have values which can be either a string or null which makes reading the data into types more difficult. I want to convert everything to strings for ease of handling. I created a type switch based on googling other examples. I am wondering if this is the easiest way to do this or if I am missing a simpler approach.
package main
import (
"encoding/json"
"fmt"
"strconv"
)
func main() {
json_byte := []byte(`{"response":[{"t_int":1, "t_bool": true, "t_null_or_string": null}, {"t_int":2, "t_bool": false, "t_null_or_string": "string1"}]}`)
//unmarshal the json to data structure using interface for variable data types
data_json := make(map[string][]map[string]interface{}) //create a structure to hold unmarshalled json
if err := json.Unmarshal(json_byte, &data_json); err != nil {
panic(err)
}
fmt.Println("json_data: ", data_json)
//Iterate over data structure and convert Bool, Int, and Null types to string
var v_conv string // temporary holding for converted string values
data_map := make(map[string]string) // temporary holding for converted maps
data_final := make([]map[string]string, 0, 100) // final holding for data converted to strings
for _, v := range data_json { //v is the value of the "response": key which is a slice of maps
for _, v2 := range v { //v2 is one of the maps in the slice of maps
for k3, v3 := range v2 { //k3 and v3 are the keys and values inside the map
fmt.Println("k3: ", k3, "v3: ", v3)
switch v_type := v3.(type) {
case nil:
v_conv = ""
case bool:
v_conv = strconv.FormatBool(v3.(bool))
case int:
v_conv = strconv.Itoa(v3.(int))
case string:
v_conv = v3.(string)
case float64:
v_conv = strconv.FormatFloat(v3.(float64), 'f', 0, 64)
default:
fmt.Println("vtype unknown: ", v_type) //have to use v_type since it is declared
v_conv = ""
}
data_map[k3] = v_conv //append a new map key/value pair both as strings
fmt.Println("data_map: ", data_map)
}
data_final = append(data_final, data_map) // after each cycle through the loop append the map to the new list
fmt.Println("data_final: ", data_final)
}
}
}
Final Format Desired a Slice of Maps
[{
"t_int": "1",
"t_bool": "true",
"t_null_string": ""
},
{
"t_int": "2",
"t_bool": "false",
"t_null_string": "string1"
}]
For this answer I'm assuming that JSON in your example is an example of (part of) your JSON input.
In this case, your JSON has a specific structure: you know which attributes are coming with a known data type and also you know which attributes a dynamic.
For example, you could unmarshal your JSON into smth like ResponseObj below:
package main
import (
"encoding/json"
"fmt"
)
type ResponseObj struct {
Response []Item `json:"response"`
}
type Item struct {
TInt int `json:"t_int"`
TBool bool `json:"t_bool"`
TMixed interface{} `json:"t_null_or_string"`
}
func main() {
json_byte := []byte(`{"response":[{"t_int":1, "t_bool": true, "t_null_or_string": null}, {"t_int":2, "t_bool": false, "t_null_or_string": "string1"}]}`)
data_json := ResponseObj{}
if err := json.Unmarshal(json_byte, &data_json); err != nil {
panic(err)
}
fmt.Printf("%+v\n", data_json)
}
Your data will look like:
{
Response:
[
{
TInt:1
TBool:true
TMixed:<nil>
}
{
TInt:2
TBool:false
TMixed:string1
}
]
}
And yes, for an attribute with a mixed type you'll run a type assertion (or comparison with nil as in your case or both).
Unlikely your JSON is a total chaos of unpredictable types. Most likely, you can single out a core structure and use interface{} for remaining, mixed types.
Hope this helps.

Serialize a map using a specific order

I have a map that uses string for both key and value. I have an array of keys that specifies the order of the values of the map.
I want to serialize that map to a JSON, but keeping the order defined on the array.
There is a sample code here: http://play.golang.org/p/A52GTDY6Wx
I want to serialize it as:
{
"name": "John",
"age": "20"
}
But if I serialize the map directly, the keys are ordered alphabetically:
{
"age": "20",
"name": "John"
}
I can serialize it as an array of maps, thus keeping the order, however that generates a lot of undesired characters:
[
{
"name": "John"
},
{
"age": "20"
}
]
In my real code I need to serialize the results of a database query which is specified in a text file, and I need to maintain the column order. I cannot use structs because the columns are not known at compile time.
EDIT: I don't need to read the JSON later in the specified order. The generated JSON is meant to be read by people, so I just want it to be as humanly readable as possible.
I could use a custom format but JSON suits me perfectly for this.
Thanks!
You need to implement the json.Marshaler interface on a custom type. This has the advantage of playing well within other struct types.
Sorry, you're always going to have to write a little bit of JSON encoding code.
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
)
type KeyVal struct {
Key string
Val interface{}
}
// Define an ordered map
type OrderedMap []KeyVal
// Implement the json.Marshaler interface
func (omap OrderedMap) MarshalJSON() ([]byte, error) {
var buf bytes.Buffer
buf.WriteString("{")
for i, kv := range omap {
if i != 0 {
buf.WriteString(",")
}
// marshal key
key, err := json.Marshal(kv.Key)
if err != nil {
return nil, err
}
buf.Write(key)
buf.WriteString(":")
// marshal value
val, err := json.Marshal(kv.Val)
if err != nil {
return nil, err
}
buf.Write(val)
}
buf.WriteString("}")
return buf.Bytes(), nil
}
func main() {
dict := map[string]interface{}{
"orderedMap": OrderedMap{
{"name", "John"},
{"age", 20},
},
}
dump, err := json.Marshal(dict)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", dump)
}
Outputs
{"orderedMap":{"name":"John","age":20}}
For that specific requirement you really don't need to use json.Marshal at all, you can simply implement your own function like this:
type OrderedMap map[string]string
func (om OrderedMap) ToJson(order ...string) string {
buf := &bytes.Buffer{}
buf.Write([]byte{'{', '\n'})
l := len(order)
for i, k := range order {
fmt.Fprintf(buf, "\t\"%s\": \"%v\"", k, om[k])
if i < l-1 {
buf.WriteByte(',')
}
buf.WriteByte('\n')
}
buf.Write([]byte{'}', '\n'})
return buf.String()
}
func main() {
om := OrderedMap{
"age": "20",
"name": "John",
}
fmt.Println(om.ToJson("name", "age"))
}
Probably the easiest solution: https://github.com/iancoleman/orderedmap
Although it might be slow as it's mentioned here
Here is a MapSlice implementation similar to what YAML v2 offers. It can do both Marshal and Unmarshal.
https://github.com/mickep76/mapslice-json

How to parse an inner field in a nested JSON object

I have a JSON object similar to this one:
{
"name": "Cain",
"parents": {
"mother" : "Eve",
"father" : "Adam"
}
}
Now I want to parse "name" and "mother" into this struct:
struct {
Name String
Mother String `json:"???"`
}
I want to specify the JSON field name with the json:... struct tag, however I don't know what to use as tag, because it is not the top object I am interested in. I found nothing about this in the encoding/json package docs nor in the popular blog post JSON and Go. I also tested mother, parents/mother and parents.mother.
You could use structs so long as your incoming data isn't too dynamic.
http://play.golang.org/p/bUZ8l6WgvL
package main
import (
"fmt"
"encoding/json"
)
type User struct {
Name string
Parents struct {
Mother string
Father string
}
}
func main() {
encoded := `{
"name": "Cain",
"parents": {
"mother": "Eve",
"father": "Adam"
}
}`
// Decode the json object
u := &User{}
err := json.Unmarshal([]byte(encoded), &u)
if err != nil {
panic(err)
}
// Print out mother and father
fmt.Printf("Mother: %s\n", u.Parents.Mother)
fmt.Printf("Father: %s\n", u.Parents.Father)
}
Unfortunately, unlike encoding/xml, the json package doesn't provide a way to access nested values. You'll want to either create a separate Parents struct or assign the type to be map[string]string. For example:
type Person struct {
Name string
Parents map[string]string
}
You could then provide a getter for mother as so:
func (p *Person) Mother() string {
return p.Parents["mother"]
}
This may or may not play into your current codebase and if refactoring the Mother field to a method call is not on the menu, then you may want to create a separate method for decoding and conforming to your current struct.
Here's some code I baked up real quick in the Go Playground
http://play.golang.org/p/PiWwpUbBqt
package main
import (
"fmt"
"encoding/json"
)
func main() {
encoded := `{
"name": "Cain",
"parents": {
"mother": "Eve"
"father": "Adam"
}
}`
// Decode the json object
var j map[string]interface{}
err := json.Unmarshal([]byte(encoded), &j)
if err != nil {
panic(err)
}
// pull out the parents object
parents := j["parents"].(map[string]interface{})
// Print out mother and father
fmt.Printf("Mother: %s\n", parents["mother"].(string))
fmt.Printf("Father: %s\n", parents["father"].(string))
}
There might be a better way. I'm looking forward to seeing the other answers. :-)
More recently, gjson supports selection of nested JSON properties.
name := gjson.Get(json, "name")
mother := gjson.Get(json, "parents.mother")
How about using an intermediary struct as the one suggested above for parsing, and then putting the relevant values in your "real" struct?
import (
"fmt"
"encoding/json"
)
type MyObject struct{
Name string
Mother string
}
type MyParseObj struct{
Name string
Parents struct {
Mother string
Father string
}
}
func main() {
encoded := `{
"name": "Cain",
"parents": {
"mother": "Eve",
"father": "Adam"
}
}`
pj := &MyParseObj{}
if err := json.Unmarshal([]byte(encoded), pj); err != nil {
return
}
final := &MyObject{Name: pj.Name, Mother: pj.Parents.Mother}
fmt.Println(final)
}