How do you read nested Json in Go? - json

I have the following JSON to be decoded, its structure can vary therefore I do not want to use structs:
{ "cabinet": "A", "shelve": {"box": "10", "color": "red"} }
Following the Golang blog (https://blog.golang.org/json) I have prepared this program to parse it:
import (
"fmt"
"encoding/json"
)
func main() {
fmt.Println("Hello, playground")
respString := `{ "cabinet": "A", "shelve": {"box": "10", "color": "red"} }`
respBytes := []byte(respString)
var f interface{}
json.Unmarshal(respBytes, &f)
m := f.(map[string]interface{})
for k, v := range m {
switch vv := v.(type) {
case string:
fmt.Println(k, "is string", vv)
case float64:
fmt.Println(k, "is float64", vv)
case []interface{}:
fmt.Println(k, "is an array:")
for i, u := range vv {
fmt.Println(i, u)
}
default:
fmt.Println(k, "is of a type I don't know how to handle")
}
}
}
However, I am wondering how can I access the nested JSON embbedded as value in "shelve".
So far this is the output:
cabinet is string A
shelve is of a type I don't know how to handle
What shall be done to access inner key/values at shelve? Which strategy is appropriate in Go?
A complete executable code can be found in https://play.golang.org/p/AVMMVQVjY__B

shelve in JSON is an object, so a Go map will be created to model it:
case map[string]interface{}:
fmt.Println(k, "is a map:")
for k, v := range vv {
fmt.Println("\t", k, "=", v)
}
With this change output is (try it on the Go Playground):
cabinet is string A
shelve is a map:
color = red
box = 10
See related question:
Accessing Nested Map of Type map[string]interface{} in Golang
Is there any convenient way to get JSON element without type assertion?
Taking a JSON string, unmarshaling it into a map[string]interface{}, editing, and marshaling it into a []byte seems more complicated then it should be

Related

Extracting Elements from JSON

I have a function that returns JSON, and the return type is this:
map[string]interface{}
In my particular case, I know that the returned JSON looks like this:
{
"someNumber": 1,
"someString": "ee",
"someArray": [
"stringInArray",
{
"anotherNumber": 100,
"anotherString": "asdf",
"yetAnotherString": "qwer"
}
]
}
I want to get the value of "stringInArray" and also "anotherString". I've searched for solutions for example from Go by Example and blog.golang.org but I've not been successful.
For example, given that res is the json returned from the function call, I tried this (from the go blog):
var f interface{}
b := []byte(`res` )
err2 := json.Unmarshal(b, &f)
if err2!=nil{
log.Fatalf("Err unmarshalling: %v", err2)
}
m := f.( map[string]interface{} )
for k, v := range m {
switch vv := v.(type) {
case string:
fmt.Println(k, "is string", vv)
case int:
fmt.Println(k, "is int", vv)
case []interface{}:
fmt.Println(k, "is an array:")
for i, u := range vv {
fmt.Println(i, u)
}
default:
fmt.Println(k, "is of a type I don't know how to handle")
}
}
(I know the above would not do exactly what I want but it is a start.)
But when code execution hits b := []byte(res ), I get this error:
Err unmarshalling: invalid character 'r' looking for beginning of value
What is the most efficient method / best practice for obtaining the values? I'm open to any solution, not necessarily the approach above.
#sydnash Here is the code I promised in my response to your comment:
type LoginResponse2 struct {
id float64
jsonrpc string
result struct {
token string
details struct {
a float64
b string
c float64
}
}
}
var resStruct2 LoginResponse2
// Convert result to json
b, _ :=json.Marshal(res)
fmt.Println( string(b) )
// results of Println:
{"id":1,"jsonrpc":"2.0","result":[ "myToken",{"a":someNumber,"b":"some string","c":someOtherNumber} ] }
// Unmarshall json into struct
err2 := json.Unmarshal(b, &resStruct2)
if err2 != nil {
log.Fatalf("Error unmarshalling json: %v", err)
}
fmt.Println("Here is json unmarshalled into a struct")
fmt.Println( resStruct2 )
// results of Println
{0 { {0 0}}}
I think you should use b := []byte(res) instead b :=[]byte[res] and res should be a json string or []byte. res is not a legal json format.
this information maybe help you:
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
you can see there is no int but float64 for JSON numbers.
There is an example of how to unmarshal json in the docs on golang packages encoding/json. Essentially your problem is that the un-marshal expects to be putting the json into type. So your code must declare a type that represents the json you are converting from.
Example of the json and the declared type from the linked example:
var jsonBlob = []byte(`[
{"Name": "Platypus", "Order": "Monotremata"},
{"Name": "Quoll", "Order": "Dasyuromorphia"}
]`)
type Animal struct {
Name string
Order string
}
Once you define the type it should all fall into place.

Convert json with variable types to strings

I am reading in json from an API response and I ran into an issue in that there are multiple data types inside the json values (strings, null, bool). In addition, some keys have values which can be either a string or null which makes reading the data into types more difficult. I want to convert everything to strings for ease of handling. I created a type switch based on googling other examples. I am wondering if this is the easiest way to do this or if I am missing a simpler approach.
package main
import (
"encoding/json"
"fmt"
"strconv"
)
func main() {
json_byte := []byte(`{"response":[{"t_int":1, "t_bool": true, "t_null_or_string": null}, {"t_int":2, "t_bool": false, "t_null_or_string": "string1"}]}`)
//unmarshal the json to data structure using interface for variable data types
data_json := make(map[string][]map[string]interface{}) //create a structure to hold unmarshalled json
if err := json.Unmarshal(json_byte, &data_json); err != nil {
panic(err)
}
fmt.Println("json_data: ", data_json)
//Iterate over data structure and convert Bool, Int, and Null types to string
var v_conv string // temporary holding for converted string values
data_map := make(map[string]string) // temporary holding for converted maps
data_final := make([]map[string]string, 0, 100) // final holding for data converted to strings
for _, v := range data_json { //v is the value of the "response": key which is a slice of maps
for _, v2 := range v { //v2 is one of the maps in the slice of maps
for k3, v3 := range v2 { //k3 and v3 are the keys and values inside the map
fmt.Println("k3: ", k3, "v3: ", v3)
switch v_type := v3.(type) {
case nil:
v_conv = ""
case bool:
v_conv = strconv.FormatBool(v3.(bool))
case int:
v_conv = strconv.Itoa(v3.(int))
case string:
v_conv = v3.(string)
case float64:
v_conv = strconv.FormatFloat(v3.(float64), 'f', 0, 64)
default:
fmt.Println("vtype unknown: ", v_type) //have to use v_type since it is declared
v_conv = ""
}
data_map[k3] = v_conv //append a new map key/value pair both as strings
fmt.Println("data_map: ", data_map)
}
data_final = append(data_final, data_map) // after each cycle through the loop append the map to the new list
fmt.Println("data_final: ", data_final)
}
}
}
Final Format Desired a Slice of Maps
[{
"t_int": "1",
"t_bool": "true",
"t_null_string": ""
},
{
"t_int": "2",
"t_bool": "false",
"t_null_string": "string1"
}]
For this answer I'm assuming that JSON in your example is an example of (part of) your JSON input.
In this case, your JSON has a specific structure: you know which attributes are coming with a known data type and also you know which attributes a dynamic.
For example, you could unmarshal your JSON into smth like ResponseObj below:
package main
import (
"encoding/json"
"fmt"
)
type ResponseObj struct {
Response []Item `json:"response"`
}
type Item struct {
TInt int `json:"t_int"`
TBool bool `json:"t_bool"`
TMixed interface{} `json:"t_null_or_string"`
}
func main() {
json_byte := []byte(`{"response":[{"t_int":1, "t_bool": true, "t_null_or_string": null}, {"t_int":2, "t_bool": false, "t_null_or_string": "string1"}]}`)
data_json := ResponseObj{}
if err := json.Unmarshal(json_byte, &data_json); err != nil {
panic(err)
}
fmt.Printf("%+v\n", data_json)
}
Your data will look like:
{
Response:
[
{
TInt:1
TBool:true
TMixed:<nil>
}
{
TInt:2
TBool:false
TMixed:string1
}
]
}
And yes, for an attribute with a mixed type you'll run a type assertion (or comparison with nil as in your case or both).
Unlikely your JSON is a total chaos of unpredictable types. Most likely, you can single out a core structure and use interface{} for remaining, mixed types.
Hope this helps.

How to specify default values when parsing JSON in Go

I want to parse a JSON object in Go, but want to specify default values for fields that are not given. For example, I have the struct type:
type Test struct {
A string
B string
C string
}
The default values for A, B, and C, are "a", "b", and "c" respectively. This means that when I parse the json:
{"A": "1", "C": 3}
I want to get the struct:
Test{A: "1", B: "b", C: "3"}
Is this possible using the built-in package encoding/json? Otherwise, is there any Go library that has this functionality?
This is possible using encoding/json: when calling json.Unmarshal, you do not need to give it an empty struct, you can give it one with default values.
For your example:
var example []byte = []byte(`{"A": "1", "C": "3"}`)
out := Test{
A: "default a",
B: "default b",
// default for C will be "", the empty value for a string
}
err := json.Unmarshal(example, &out) // <--
if err != nil {
panic(err)
}
fmt.Printf("%+v", out)
Running this example returns {A:1 B:default b C:3}.
As you can see, json.Unmarshal(example, &out) unmarshals the JSON into out, overwriting the values specified in the JSON, but leaving the other fields unchanged.
In case u have a list or map of Test structs the accepted answer is not possible anymore but it can easily be extended with a UnmarshalJSON method:
func (t *Test) UnmarshalJSON(data []byte) error {
type testAlias Test
test := &testAlias{
B: "default B",
}
err := json.Unmarshal(data, test)
if err != nil {
return err
}
*t = Test(*test)
return nil
}
The testAlias is needed to prevent recursive calls to UnmarshalJSON. This works because a new type has no methods defined.
https://play.golang.org/p/qiGyjRbNHg
You can also implement the UnmarshalJSON interface and set the default values in it.
func (test *Test) UnmarshalJSON(data []byte) error {
test.A = "1"
test.B = "2"
test.C = "3"
type tempTest Test
return json.Unmarshal(data, (*tempTest)(test))
}
The type tempTest is needed so that UnmarshalJSON is not called recursively.
The json.Unmarshal with a default value is simple and clean like the answers given by Christian and JW., but it has some downsides.
First, it strongly ties the default values of fields with the parsing logic. It's conceivable that we want to let user code down the line set its defaults; right now, the defaults have to be set before unmarshaling.
The second downside is that it only works in simple cases. If our Options struct has a slice or map of other structs, we can't populate defaults this way.
Another option is Default values with pointer fields
type Test struct {
A *string
B *string
C *string
}
js := []byte(`{"A": "1", "C": "3"}`)
var t Test
if err := json.Unmarshal(js, &t); err != nil {
fmt.Println(err)
}
if t.B == nil {
var defaultB = "B"
t.B = &defaultB
}

Looping/iterate over the second level nested JSON in go lang

Consider the following code:
package main
import (
"encoding/json"
"fmt"
"reflect"
)
func main() {
//Creating the maps for JSON
m := map[string]interface{}{}
//Parsing/Unmarshalling JSON encoding/json
err := json.Unmarshal([]byte(input), &m)
fmt.Println("\nReflect type of Parsing/Unmarshalling Error Object:\n",reflect.TypeOf(err))
fmt.Println("\nParsing/Unmarshalling Error Object:\n",err)
if err != nil {
panic(err)
}
fmt.Println("\nParsed JSON is as follows:\n",m)
fmt.Println("\nReflect type of parsed json object:\n", reflect.TypeOf(m))
for firstLvlkey, firstLvlValue := range m {
fmt.Println("First Level Key:", firstLvlkey)
fmt.Println("First Level Key reflect type of :", reflect.TypeOf(firstLvlkey))
fmt.Println("First Level Value:", firstLvlValue)
fmt.Println("First Level Value reflect type of :", reflect.TypeOf(firstLvlValue))
// <===============================>
//Here I want to iterate/loop over innerJSON1, InnerJSON2 then reach to level InnerInnerJSONArray - fld1 and fld2
// <===============================>
}
}
const input = `
{
"outterJSON":{
"innerJSON1":{
"value1":10,
"value2":22
,
"InnerInnerArray": [ "test1" , "test2"],
"InnerInnerJSONArray": [ {"fld1" : "val1"} , {"fld2" : "val2"} ]
},
"InnerJSON2":"NoneValue"
}
}
`
I have some requirement like I want to read/get all the Key and value in String type for some processing adn I can't define the struct because I will be getting dynamic JSON input (e.g InnerInnerArray as a string then second level loop will give me index of array and process each JSON having key fld1 and val1).
I wish to iterate over every key/value pair contained within it, what is the most efficient way of going through the map?
Note: I am Newbie for Go-lang, your suggestion/improvement on question is also most welcome.
See this blog entry which thoroughly covers this subject, specifically the section Decoding arbitrary data. Using that you can do something like this:
(playground example)
package main
import (
"encoding/json"
"fmt"
)
func main() {
// Creating the maps for JSON
m := map[string]interface{}{}
// Parsing/Unmarshalling JSON encoding/json
err := json.Unmarshal([]byte(input), &m)
if err != nil {
panic(err)
}
parseMap(m)
}
func parseMap(aMap map[string]interface{}) {
for key, val := range aMap {
switch concreteVal := val.(type) {
case map[string]interface{}:
fmt.Println(key)
parseMap(val.(map[string]interface{}))
case []interface{}:
fmt.Println(key)
parseArray(val.([]interface{}))
default:
fmt.Println(key, ":", concreteVal)
}
}
}
func parseArray(anArray []interface{}) {
for i, val := range anArray {
switch concreteVal := val.(type) {
case map[string]interface{}:
fmt.Println("Index:", i)
parseMap(val.(map[string]interface{}))
case []interface{}:
fmt.Println("Index:", i)
parseArray(val.([]interface{}))
default:
fmt.Println("Index", i, ":", concreteVal)
}
}
}
const input = `
{
"outterJSON": {
"innerJSON1": {
"value1": 10,
"value2": 22,
"InnerInnerArray": [ "test1" , "test2"],
"InnerInnerJSONArray": [{"fld1" : "val1"} , {"fld2" : "val2"}]
},
"InnerJSON2":"NoneValue"
}
}
`
This will print:
//outterJSON
//innerJSON1
//InnerInnerJSONArray
//Index: 0
//fld1 : val1
//Index: 1
//fld2 : val2
//value1 : 10
//value2 : 22
//InnerInnerArray
//Index 0 : test1
//Index 1 : test2
//InnerJSON2 : NoneValue
The key thing is that you have to use type assertion when working with interface types. The type switch makes it easy to determine the type as needed. The code will recursively range through any nested array or map so you can add as many levels as you wish and get all your values.
There are related questions here and here (and possibly others).
There are some more sophisticated JSON parsing APIs that make your job easier. An example is stretchr/objx.
An example of using objx:
document, err := objx.FromJSON(json)
// TODO handle err
document.Get("path.to.field[0].you.want").Str()
This works when you really don't know what the JSON structure will be. However, if you do know the structure of your JSON input ahead of time, the preferred way is to describe it with structs and use the standard API for marshalling.
You'll need to parse the JSON and then recurse through the structure inspecting the types of the contained values and handling them in some way.
The example function below takes an *interface{} (pointer to any type) and a handler function of string, int, and object pointers to which it yields the items it discovers:
func eachJsonValue(obj *interface{}, handler func(*string, *int, *interface{})) {
if obj == nil {
return
}
// Yield all key/value pairs for objects.
o, isObject := (*obj).(map[string]interface{})
if isObject {
for k, v := range o {
handler(&k, nil, &v)
eachJsonValue(&v, handler)
}
}
// Yield each index/value for arrays.
a, isArray := (*obj).([]interface{})
if isArray {
for i, x := range a {
handler(nil, &i, &x)
eachJsonValue(&x, handler)
}
}
// Do nothing for primitives since the handler got them.
}
Calling it as demonstrated below will print the listed results. Your handler function could, of course, do something special with known key/values such as "fld1":
func main() {
// Parse the JSON.
var obj interface{}
json.Unmarshal([]byte(input), &obj) // XXX: check the error value.
// Handle object key/value pairs and array index/items.
eachJsonValue(&obj, func(key *string, index *int, value *interface{}) {
if key != nil { // It's an object key/value pair...
fmt.Printf("OBJ: key=%q, value=%#v\n", *key, *value)
} else { // It's an array item...
fmt.Printf("ARR: index=%d, value=%#v\n", *index, *value)
}
})
}
// OBJ: key="outterJSON", value=map[string]interface {}{...}
// OBJ: key="innerJSON1", value=map[string]interface {}{...}
// OBJ: key="value1", value=10
// OBJ: key="value2", value=22
// OBJ: key="InnerInnerArray", value=[]interface {}{...}
// ARR: index=0, value="test1"
// ARR: index=1, value="test2"
// OBJ: key="InnerInnerJSONArray", value=[]interface {}{...}
// ARR: index=0, value=map[string]interface {}{...}
// OBJ: key="fld1", value="val1"
// ARR: index=1, value=map[string]interface {}{...}
// OBJ: key="fld2", value="val2"
// OBJ: key="InnerJSON2", value="NoneValue"
I have implemented something very similar to https://stackoverflow.com/users/1078890/iamnan solution. The body of parseMap and parseArray is combined into one and it looks something like this.
func printJson(res1 map[string]interface{}, res2 []interface{}) {
for k, v := range res1 {
switch vv := v.(type) {
case float64, int, string:
fmt.Println(k, ":", vv)
case []interface{}:
fmt.Println(k, ":")
printJson(nil, vv)
case map[string]interface{}:
fmt.Println(k, ":")
printJson(vv, nil)
default:
fmt.Println(k, ":", vv)
}
}
for k, v := range res2 {
switch vv := v.(type) {
case float64, int, string:
fmt.Println(k, ":", vv)
case []interface{}:
fmt.Println(k, ":")
printJson(nil, vv)
case map[string]interface{}:
fmt.Println(k, ":")
printJson(vv, nil)
default:
fmt.Println(k, ":", vv)
}
}
}
Code can be found here https://gist.github.com/sandeep-sarkar/78a0e96461b4dec727386a96404d29b0

Serialize a map using a specific order

I have a map that uses string for both key and value. I have an array of keys that specifies the order of the values of the map.
I want to serialize that map to a JSON, but keeping the order defined on the array.
There is a sample code here: http://play.golang.org/p/A52GTDY6Wx
I want to serialize it as:
{
"name": "John",
"age": "20"
}
But if I serialize the map directly, the keys are ordered alphabetically:
{
"age": "20",
"name": "John"
}
I can serialize it as an array of maps, thus keeping the order, however that generates a lot of undesired characters:
[
{
"name": "John"
},
{
"age": "20"
}
]
In my real code I need to serialize the results of a database query which is specified in a text file, and I need to maintain the column order. I cannot use structs because the columns are not known at compile time.
EDIT: I don't need to read the JSON later in the specified order. The generated JSON is meant to be read by people, so I just want it to be as humanly readable as possible.
I could use a custom format but JSON suits me perfectly for this.
Thanks!
You need to implement the json.Marshaler interface on a custom type. This has the advantage of playing well within other struct types.
Sorry, you're always going to have to write a little bit of JSON encoding code.
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
)
type KeyVal struct {
Key string
Val interface{}
}
// Define an ordered map
type OrderedMap []KeyVal
// Implement the json.Marshaler interface
func (omap OrderedMap) MarshalJSON() ([]byte, error) {
var buf bytes.Buffer
buf.WriteString("{")
for i, kv := range omap {
if i != 0 {
buf.WriteString(",")
}
// marshal key
key, err := json.Marshal(kv.Key)
if err != nil {
return nil, err
}
buf.Write(key)
buf.WriteString(":")
// marshal value
val, err := json.Marshal(kv.Val)
if err != nil {
return nil, err
}
buf.Write(val)
}
buf.WriteString("}")
return buf.Bytes(), nil
}
func main() {
dict := map[string]interface{}{
"orderedMap": OrderedMap{
{"name", "John"},
{"age", 20},
},
}
dump, err := json.Marshal(dict)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", dump)
}
Outputs
{"orderedMap":{"name":"John","age":20}}
For that specific requirement you really don't need to use json.Marshal at all, you can simply implement your own function like this:
type OrderedMap map[string]string
func (om OrderedMap) ToJson(order ...string) string {
buf := &bytes.Buffer{}
buf.Write([]byte{'{', '\n'})
l := len(order)
for i, k := range order {
fmt.Fprintf(buf, "\t\"%s\": \"%v\"", k, om[k])
if i < l-1 {
buf.WriteByte(',')
}
buf.WriteByte('\n')
}
buf.Write([]byte{'}', '\n'})
return buf.String()
}
func main() {
om := OrderedMap{
"age": "20",
"name": "John",
}
fmt.Println(om.ToJson("name", "age"))
}
Probably the easiest solution: https://github.com/iancoleman/orderedmap
Although it might be slow as it's mentioned here
Here is a MapSlice implementation similar to what YAML v2 offers. It can do both Marshal and Unmarshal.
https://github.com/mickep76/mapslice-json