Please read the question before flagging it.
{
"out_key": {
"some_uri": "url",
"more_than_one_key_here": {
"Ip": "127.0.0.1",
"port": "80",
}
}
}
out_key is dynamically created. There is no guessing the name of it.
Similarly, more_than_one_key_here is also dynamically created. some_uri will remain constant under out_key. In such a scenario, how do I create a struct for decoding the JSON?
In Go field names of a struct have to be known at compile time. So, in your case a struct type is not appropriate. Alternatively, you can use a map.
Let's say, that you are only interested in the IP and port values. Then you can ignore the keys of the JSON object altogether.
package main
import (
"encoding/json"
"fmt"
"log"
)
const data = `{
"out_key": {
"some_uri": "url",
"more_than_one_key_here": {
"Ip": "127.0.0.1",
"port": "80"
}
}
}`
func main() {
m := make(map[string]map[string]interface{})
err := json.Unmarshal([]byte(data), &m)
if err != nil {
log.Fatalf("failed to parse JSON: %v", err)
}
for _, value := range m {
for _, value := range value {
m, ok := value.(map[string]interface{})
if ok {
fmt.Printf("IP: %s\n", m["Ip"])
fmt.Printf("Port: %s\n", m["port"])
}
}
}
}
You could create a struct for the Ip and Port and a separate one for one of the keys.
type data struct{
Ip string
port string
}
type DynamicKey map[string]data
Then once you decode the json and have the dynamic key, let's say for example it is (rando_key) you can do this
fmt.Println(DynamicKey[rando_key].Ip, DynamicKey[rando_key].port)
This is off course for only one of the keys but it is possible to repeat the process.
Related
Some context, I'm designing a backend that will receive JSON post data, but the nature of the data is that it has fields that are unstructured. My general research tells me this is a static language vs unstructured data problem.
Normally if you can create a struct for it if the data is well known and just unmarshal into the struct. I have create custom unmarshaling functions for nested objects.
The issue now is that one of the fields could contain an object with an arbitrary number of keys. To provide some code context:
properties: {
"k1": "v1",
"k2": "v2",
"k3": "v3",
...
}
type Device struct {
id: string,
name: string,
status: int,
properties: <what would i put here?>
}
So its hard to code an explicit unmarshaling function for it. Should put a type of map[string]string{}? How would it work if the values were not all strings then? And what if that object itself had nested values/objects as well?
You can make the Properties field as map[string]interface{} so that it can accommodate different types of values.I created a small code for your scenario as follows:
package main
import (
"encoding/json"
"fmt"
)
type Device struct {
Id string
Name string
Status int
Properties map[string]interface{}
}
func main() {
devObj := Device{}
data := []byte(`{"Id":"101","Name":"Harold","Status":1,"properties":{"key1":"val1"}}`)
if err := json.Unmarshal(data, &devObj); err != nil {
panic(err)
}
fmt.Println(devObj)
devObj2 := Device{}
data2 := []byte(`{"Id":"102","Name":"Thanor","Status":1,"properties":{"k1":25,"k2":"someData"}}`)
if err := json.Unmarshal(data2, &devObj2); err != nil {
panic(err)
}
fmt.Println(devObj2)
devObj3 := Device{}
data3 := []byte(`{"Id":"101","Name":"GreyBeard","Status":1,"properties":{"k1":25,"k2":["data1","data2"]}}`)
if err := json.Unmarshal(data3, &devObj3); err != nil {
panic(err)
}
fmt.Println(devObj3)
}
Output:
{101 Harold 1 map[key1:val1]}
{102 Thanor 1 map[k1:25 k2:someData]}
{101 GreyBeard 1 map[k1:25 k2:[data1 data2]]}
I would use one of the popular Go JSON parsers that don't require parsing to a pre-defined struct. An added benefit, and the primary reason they were created, is that they are much faster than encoding/json because they don't use reflection, interface{} or certain other approaches.
Here are two:
https://github.com/buger/jsonparser - 4.1k GitHub stars
https://github.com/valyala/fastjson - 1.3k GitHub stars
Using github.com/buger/jsonparser, a property could be retrieved using the GetString function:
func GetString(data []byte, keys ...string) (val string, err error)
Here's a full example:
package main
import (
"fmt"
"strconv"
"github.com/buger/jsonparser"
)
func main() {
jsondata := []byte(`
{
"properties": {
"k1": "v1",
"k2": "v2",
"k3": "v3"
}
}`)
for i := 1; i > 0; i++ {
key := "k" + strconv.Itoa(i)
val, err := jsonparser.GetString(jsondata, "properties", key)
if err == jsonparser.KeyPathNotFoundError {
break
} else if err != nil {
panic(err)
}
fmt.Printf("found: key [%s] val [%s]\n", key, val)
}
}
See it run on Go Playground: https://play.golang.org/p/ykAM4gac8zT
I'm trying to parse JSON files by using dynamically created structs, but apparently I'm doing something wrong. Can somebody please tell we what am I doing wrong here:
structured := make(map[string][]reflect.StructField)
structured["Amqp1"] = []reflect.StructField{
reflect.StructField{
Name: "Test",
Type: reflect.TypeOf(""),
Tag: reflect.StructTag(`json:"test"`),
},
reflect.StructField{
Name: "Float",
Type: reflect.TypeOf(5.5),
Tag: reflect.StructTag(`json:"float"`),
},
reflect.StructField{
Name: "Connections",
Type: reflect.TypeOf([]Connection{}),
Tag: reflect.StructTag(`json:"connections"`),
},
}
sections := []reflect.StructField{}
for sect, params := range structured {
sections = append(sections,
reflect.StructField{
Name: sect,
Type: reflect.StructOf(params),
},
)
}
parsed := reflect.New(reflect.StructOf(sections)).Elem()
if err := json.Unmarshal([]byte(JSONConfigContent), &parsed); err != nil {
fmt.Printf("unable to parse data from provided configuration file: %s\n", err)
os.Exit(1)
}
https://play.golang.org/p/C2I4Pduduyg
Thanks in advance.
You want to use .Interface() to return the actual, underlying value, which should be a pointer to the concrete anonymous struct.
Note that the reflect.New function returns a reflect.Value representing a pointer to a new zero value for the specified type. The Interface method, in this case, returns that pointer as interface{} which is all you need for json.Unmarshal.
If, after unmarshaling, you need a non-pointer of the struct you can turn to reflect again and use reflect.ValueOf(parsed).Elem().Interface() to effectively dereference the pointer.
parsed := reflect.New(reflect.StructOf(sections)).Interface()
if err := json.Unmarshal([]byte(JSONConfigContent), parsed); err != nil {
fmt.Printf("unable to parse data from provided configuration file: %s\n", err)
os.Exit(1)
}
https://play.golang.org/p/Bzu1hUyKjvM
reflect.New returns a value representing a pointer.
Change line 69:
fmt.Printf(">>> %v", &parsed)
Result:
>>> <struct { Amqp1 struct { Test string "json:\"test\""; Float float64 "json:\"float\""; Connections []main.Connection "json:\"connections\"" } } Value>
I would recommend something very different. I generally avoid reflection where possible. You can accomplish what you are trying to do by simply providing structs for each type of config you expect and then do an initial "pre-unmarshalling" to determine what type of config you should actually use by name (which, in this case is a key of your JSON object):
package main
import (
"encoding/json"
"fmt"
"os"
)
//Amqp1 config struct
type Amqp1 struct {
Config struct {
Test string `json:"test"`
Float float64 `json:"float"`
Connections []Connection `json:"connections"`
} `json:"Amqp1"`
}
//Connection struct
type Connection struct {
Type string `json:"type"`
URL string `json:"url"`
}
//JSONConfigContent json
const JSONConfigContent = `{
"Amqp1": {
"test": "woobalooba",
"float": 5.5,
"connections": [
{"type": "test1", "url": "booyaka"},
{"type": "test2", "url": "foobar"}
]
}
}`
func main() {
configMap := make(map[string]interface{})
if err := json.Unmarshal([]byte(JSONConfigContent), &configMap); err != nil {
fmt.Printf("unable to parse data from provided configuration file: %s\n", err)
os.Exit(1)
}
//get config name
var configName string
for cfg := range configMap {
configName = cfg
break
}
//unmarshal appropriately
switch configName {
case "Amqp1":
var amqp1 Amqp1
if err := json.Unmarshal([]byte(JSONConfigContent), &amqp1); err != nil {
fmt.Printf("unable to parse data from provided configuration file: %s\n", err)
os.Exit(1)
}
fmt.Printf("%s >>\n", configName)
fmt.Printf("Test: %s\n", amqp1.Config.Test)
fmt.Printf("Float: %v\n", amqp1.Config.Float)
fmt.Printf("Connections: %#v\n", amqp1.Config.Connections)
default:
fmt.Printf("unknown config encountered: %s\n", configName)
os.Exit(1)
}
}
The initial "pre-unmarshalling" happens on the 2nd line of main(), to a plain map where the key is a string and the value is interface{} because you don't care. You're just doing that to get the actual type of config that it is, which is the key of the first nested object.
You can run this on playground as well.
Let's say I have the following Go struct on the server
type account struct {
Name string
Balance int
}
I want to call json.Decode on the incoming request to parse it into an account.
var ac account
err := json.NewDecoder(r.Body).Decode(&ac)
If the client sends the following request:
{
"name": "test#example.com",
"balance": "3"
}
Decode() will return the following error:
json: cannot unmarshal string into Go value of type int
Now it's possible to parse that back into "you sent a string for Balance, but you really should have sent an integer", but it's tricky, because you don't know the field name. It also gets a lot trickier if you have a lot of fields in the request - you don't know which one failed to parse.
What's the best way to take that incoming request, in Go, and return the error message, "Balance must be a string", for any arbitrary number of integer fields on a request?
You can use a custom type with custom unmarshaling algorythm for your "Balance" field.
Now there are two possibilities:
Handle both types:
package main
import (
"encoding/json"
"fmt"
"strconv"
)
type Int int
type account struct {
Name string
Balance Int
}
func (i *Int) UnmarshalJSON(b []byte) (err error) {
var s string
err = json.Unmarshal(b, &s)
if err == nil {
var n int
n, err = strconv.Atoi(s)
if err != nil {
return
}
*i = Int(n)
return
}
var n int
err = json.Unmarshal(b, &n)
if err == nil {
*i = Int(n)
}
return
}
func main() {
for _, in := range [...]string{
`{"Name": "foo", "Balance": 42}`,
`{"Name": "foo", "Balance": "111"}`,
} {
var a account
err := json.Unmarshal([]byte(in), &a)
if err != nil {
fmt.Printf("Error decoding JSON: %v\n", err)
} else {
fmt.Printf("Decoded OK: %v\n", a)
}
}
}
Playground link.
Handle only a numeric type, and fail anything else with a sensible error:
package main
import (
"encoding/json"
"fmt"
)
type Int int
type account struct {
Name string
Balance Int
}
type FormatError struct {
Want string
Got string
Offset int64
}
func (fe *FormatError) Error() string {
return fmt.Sprintf("Invalid data format at %d: want: %s, got: %s",
fe.Offset, fe.Want, fe.Got)
}
func (i *Int) UnmarshalJSON(b []byte) (err error) {
var n int
err = json.Unmarshal(b, &n)
if err == nil {
*i = Int(n)
return
}
if ute, ok := err.(*json.UnmarshalTypeError); ok {
err = &FormatError{
Want: "number",
Got: ute.Value,
Offset: ute.Offset,
}
}
return
}
func main() {
for _, in := range [...]string{
`{"Name": "foo", "Balance": 42}`,
`{"Name": "foo", "Balance": "111"}`,
} {
var a account
err := json.Unmarshal([]byte(in), &a)
if err != nil {
fmt.Printf("Error decoding JSON: %#v\n", err)
fmt.Printf("Error decoding JSON: %v\n", err)
} else {
fmt.Printf("Decoded OK: %v\n", a)
}
}
}
Playground link.
There is a third possibility: write custom unmarshaler for the whole account type, but it requires more involved code because you'd need to actually iterate over the input JSON data using the methods of the
encoding/json.Decoder type.
After reading your
What's the best way to take that incoming request, in Go, and return the error message, "Balance must be a string", for any arbitrary number of integer fields on a request?
more carefully, I admit having a custom parser for the whole type is the only sensible possibility unless you are OK with a 3rd-party package implementing a parser supporting validation via JSON schema (I think I'd look at this first as juju is a quite established product).
A solution for this could be to use a type assertion by using a map to unmarshal the JSON data into:
type account struct {
Name string
Balance int
}
var str = `{
"name": "test#example.com",
"balance": "3"
}`
func main() {
var testing = map[string]interface{}{}
err := json.Unmarshal([]byte(str), &testing)
if err != nil {
fmt.Println(err)
}
val, ok := testing["balance"]
if !ok {
fmt.Println("missing field balance")
return
}
nv, ok := val.(float64)
if !ok {
fmt.Println("balance should be a number")
return
}
fmt.Printf("%+v\n", nv)
}
See http://play.golang.org/p/iV7Qa1RrQZ
The type assertion here is done using float64 because it is the default number type supported by Go's JSON decoder.
It should be noted that this use of interface{} is probably not worth the trouble.
The UnmarshalTypeError (https://golang.org/pkg/encoding/json/#UnmarshalFieldError) contains an Offset field that could allow retrieving the contents of the JSON data that triggered the error.
You could for example return a message of the sort:
cannot unmarshal string into Go value of type int near `"balance": "3"`
It would seem that here provides an implementation to work around this issue in Go only.
type account struct {
Name string
Balance int `json:",string"`
}
In my estimation the more correct and sustainable approach is for you to create a client library in something like JavaScript and publish it into the NPM registry for others to use (private repository would work the same way). By providing this library you can tailor the API for the consumers in a meaningful way and prevent errors creeping into your main program.
I have a map that uses string for both key and value. I have an array of keys that specifies the order of the values of the map.
I want to serialize that map to a JSON, but keeping the order defined on the array.
There is a sample code here: http://play.golang.org/p/A52GTDY6Wx
I want to serialize it as:
{
"name": "John",
"age": "20"
}
But if I serialize the map directly, the keys are ordered alphabetically:
{
"age": "20",
"name": "John"
}
I can serialize it as an array of maps, thus keeping the order, however that generates a lot of undesired characters:
[
{
"name": "John"
},
{
"age": "20"
}
]
In my real code I need to serialize the results of a database query which is specified in a text file, and I need to maintain the column order. I cannot use structs because the columns are not known at compile time.
EDIT: I don't need to read the JSON later in the specified order. The generated JSON is meant to be read by people, so I just want it to be as humanly readable as possible.
I could use a custom format but JSON suits me perfectly for this.
Thanks!
You need to implement the json.Marshaler interface on a custom type. This has the advantage of playing well within other struct types.
Sorry, you're always going to have to write a little bit of JSON encoding code.
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
)
type KeyVal struct {
Key string
Val interface{}
}
// Define an ordered map
type OrderedMap []KeyVal
// Implement the json.Marshaler interface
func (omap OrderedMap) MarshalJSON() ([]byte, error) {
var buf bytes.Buffer
buf.WriteString("{")
for i, kv := range omap {
if i != 0 {
buf.WriteString(",")
}
// marshal key
key, err := json.Marshal(kv.Key)
if err != nil {
return nil, err
}
buf.Write(key)
buf.WriteString(":")
// marshal value
val, err := json.Marshal(kv.Val)
if err != nil {
return nil, err
}
buf.Write(val)
}
buf.WriteString("}")
return buf.Bytes(), nil
}
func main() {
dict := map[string]interface{}{
"orderedMap": OrderedMap{
{"name", "John"},
{"age", 20},
},
}
dump, err := json.Marshal(dict)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", dump)
}
Outputs
{"orderedMap":{"name":"John","age":20}}
For that specific requirement you really don't need to use json.Marshal at all, you can simply implement your own function like this:
type OrderedMap map[string]string
func (om OrderedMap) ToJson(order ...string) string {
buf := &bytes.Buffer{}
buf.Write([]byte{'{', '\n'})
l := len(order)
for i, k := range order {
fmt.Fprintf(buf, "\t\"%s\": \"%v\"", k, om[k])
if i < l-1 {
buf.WriteByte(',')
}
buf.WriteByte('\n')
}
buf.Write([]byte{'}', '\n'})
return buf.String()
}
func main() {
om := OrderedMap{
"age": "20",
"name": "John",
}
fmt.Println(om.ToJson("name", "age"))
}
Probably the easiest solution: https://github.com/iancoleman/orderedmap
Although it might be slow as it's mentioned here
Here is a MapSlice implementation similar to what YAML v2 offers. It can do both Marshal and Unmarshal.
https://github.com/mickep76/mapslice-json
I have a JSON object similar to this one:
{
"name": "Cain",
"parents": {
"mother" : "Eve",
"father" : "Adam"
}
}
Now I want to parse "name" and "mother" into this struct:
struct {
Name String
Mother String `json:"???"`
}
I want to specify the JSON field name with the json:... struct tag, however I don't know what to use as tag, because it is not the top object I am interested in. I found nothing about this in the encoding/json package docs nor in the popular blog post JSON and Go. I also tested mother, parents/mother and parents.mother.
You could use structs so long as your incoming data isn't too dynamic.
http://play.golang.org/p/bUZ8l6WgvL
package main
import (
"fmt"
"encoding/json"
)
type User struct {
Name string
Parents struct {
Mother string
Father string
}
}
func main() {
encoded := `{
"name": "Cain",
"parents": {
"mother": "Eve",
"father": "Adam"
}
}`
// Decode the json object
u := &User{}
err := json.Unmarshal([]byte(encoded), &u)
if err != nil {
panic(err)
}
// Print out mother and father
fmt.Printf("Mother: %s\n", u.Parents.Mother)
fmt.Printf("Father: %s\n", u.Parents.Father)
}
Unfortunately, unlike encoding/xml, the json package doesn't provide a way to access nested values. You'll want to either create a separate Parents struct or assign the type to be map[string]string. For example:
type Person struct {
Name string
Parents map[string]string
}
You could then provide a getter for mother as so:
func (p *Person) Mother() string {
return p.Parents["mother"]
}
This may or may not play into your current codebase and if refactoring the Mother field to a method call is not on the menu, then you may want to create a separate method for decoding and conforming to your current struct.
Here's some code I baked up real quick in the Go Playground
http://play.golang.org/p/PiWwpUbBqt
package main
import (
"fmt"
"encoding/json"
)
func main() {
encoded := `{
"name": "Cain",
"parents": {
"mother": "Eve"
"father": "Adam"
}
}`
// Decode the json object
var j map[string]interface{}
err := json.Unmarshal([]byte(encoded), &j)
if err != nil {
panic(err)
}
// pull out the parents object
parents := j["parents"].(map[string]interface{})
// Print out mother and father
fmt.Printf("Mother: %s\n", parents["mother"].(string))
fmt.Printf("Father: %s\n", parents["father"].(string))
}
There might be a better way. I'm looking forward to seeing the other answers. :-)
More recently, gjson supports selection of nested JSON properties.
name := gjson.Get(json, "name")
mother := gjson.Get(json, "parents.mother")
How about using an intermediary struct as the one suggested above for parsing, and then putting the relevant values in your "real" struct?
import (
"fmt"
"encoding/json"
)
type MyObject struct{
Name string
Mother string
}
type MyParseObj struct{
Name string
Parents struct {
Mother string
Father string
}
}
func main() {
encoded := `{
"name": "Cain",
"parents": {
"mother": "Eve",
"father": "Adam"
}
}`
pj := &MyParseObj{}
if err := json.Unmarshal([]byte(encoded), pj); err != nil {
return
}
final := &MyObject{Name: pj.Name, Mother: pj.Parents.Mother}
fmt.Println(final)
}