Encoding nested JSON in Go - json

I've had a lot of trouble finding an example of this. Most of the information on the internet is about decoding JSON.
I'd like to serialize some data into nested JSON, like for example:
{
"item": {
"title": "Items",
"properties": [
{
"num": 1,
"name": "Item 1"
},
{
"num": 2,
"name": "Item 2"
}
]
}
}
I know how to marshal data with a flat struct, but how do I put data into a struct that can be serialized with nesting?
http://play.golang.org/p/nDKmv1myTD
I found this tool that generates a struct from a JSON schema, but I don't understand how to get data into the sub structs.
http://mholt.github.io/json-to-go/
type Example struct {
Item struct {
Title string `json:"title"`
Properties []struct {
Num int `json:"num"`
Name string `json:"name"`
} `json:"properties"`
} `json:"item"`
}

This tool you found is nice, but I would not use it. It makes it difficult to initialize the structs.
Init example with your snippet: (http://play.golang.org/p/_Qw3Qp8XZh)
package main
import (
"encoding/json"
"fmt"
)
type Example struct {
Item struct {
Title string `json:"title"`
Properties []struct {
Num int `json:"num"`
Name string `json:"name"`
} `json:"properties"`
} `json:"item"`
}
func main() {
info := &Example{
Item: struct {
Title string `json:"title"`
Properties []struct {
Num int `json:"num"`
Name string `json:"name"`
} `json:"properties"`
}{
Title: "title",
Properties: []struct {
Num int `json:"num"`
Name string `json:"name"`
}{
{Num: 0, Name: "name0"},
{Num: 1, Name: "name1"},
},
},
}
b, err := json.Marshal(info)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(b))
}
Result (pretty printed):
{
"item": {
"title": "title",
"properties": [
{
"num": 0,
"name": "name0"
},
{
"num": 1,
"name": "name1"
}
]
}
}
I think it is better to use named struct vs anonymous nested ones.
Same example with named structs: http://play.golang.org/p/xm7BXxEGTC
package main
import (
"encoding/json"
"fmt"
)
type Example struct {
Item Item `json:"item"`
}
type Item struct {
Title string `json:"title"`
Properties []Property `json:"properties"`
}
type Property struct {
Num int `json:"num"`
Name string `json:"name"`
}
func main() {
info := &Example{
Item: Item{
Title: "title",
Properties: []Property{
{Num: 0, Name: "name0"},
{Num: 1, Name: "name1"},
},
},
}
b, err := json.Marshal(info)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(b))
}
It is the exact same thing, but I find it more clear and easy to use.

Related

Nested Go Structs for JSON marshaling with optional structs

I'm trying to initialize a nested struct to then marshal into json for an API response. The challenge I'm hitting is one of the components (a slice of structs) can have n number of members but of one of two possible types (text, image).
The JSON I want to create looks like this:
{
"messages": [
{
"message_parts": [
{
"text": {
"content": "dfdffd"
}
},
{
"image": {
"url": "https://image.jpg"
}
}
],
"actor_id": "44444444",
"actor_type": "agent"
}
],
"channel_id": "44444444",
"users": [
{
"id": "44444444"
}
]
}
In the message_parts slice, that can contain at least one of text or image but possibly one of each.
My structs look like this currently:
Type messagePayload struct {
Messages []Messages `json:"messages"`
Status string `json:"status,omitempty"`
ChannelID string `json:"channel_id"`
Users []Users `json:"users"`
}
type Messages struct {
MessageParts []MessageParts `json:"message_parts"`
ActorID string `json:"actor_id"`
ActorType string `json:"actor_type"`
}
type Users struct {
ID string `json:"id"`
}
type Text struct {
Content string `json:"content,omitempty"`
}
type MessageParts struct {
Text *Text `json:"text,omitempty"`
Image *Image `json:"image,omitempty"`
}
type Image struct {
URL string `json:"url,omitempty"`
}
I'm really struggling to initialize this in a way that not show up in the json if they're not present.
here's where I'm at but it obviously doesn't work:
payload := &messagePayload{
Messages: []Messages{
{
MessageParts: []MessageParts{
{
&Text{
Content: text,
},
},
{
&Image{
URL: mediaurl,
},
},
},
ActorID: agentID,
ActorType: "agent",
}},
ChannelID: channelid,
Users: []Users{
{
ID: user,
},
},
}
EDIT:
Thanks to hint below and a few other findings, I've found the best way is to initalize the payload and then add the slices for text and images as needed:
https://play.golang.org/p/Pmmv00spcI6
As noted above, I was able to find a solution- you need to initialize the payload without the text or image data, then append them to the MessageParts slice:
package main
import (
"encoding/json"
"fmt"
)
type messagePayload struct {
Messages []Messages `json:"messages"`
Status string `json:"status,omitempty"`
ChannelID string `json:"channel_id"`
Users []Users `json:"users"`
}
type Messages struct {
MessageParts []MessageParts `json:"message_parts"`
ActorID string `json:"actor_id"`
ActorType string `json:"actor_type"`
}
type Users struct {
ID string `json:"id"`
}
type Text struct {
Content string `json:"content,omitempty"`
}
type MessageParts struct {
Text *Text `json:"text,omitempty"`
Image *Image `json:"image,omitempty"`
}
type Image struct {
URL string `json:"url,omitempty"`
}
func main() {
payload := &messagePayload{
Messages: []Messages{
{
MessageParts: []MessageParts{
},
ActorID: "id",
ActorType: "agent",
}},
ChannelID: "cid",
Users: []Users{
{
ID: "user1",
},
},
}
var text= new(MessageParts)
text.Text = &Text{Content: "LOL"}
var image = new(MessageParts)
image.Image= &Image{URL: "https://"}
payload.Messages[0].MessageParts = append(payload.Messages[0].MessageParts, *text)
payload.Messages[0].MessageParts = append(payload.Messages[0].MessageParts, *image)
m, err := json.Marshal(payload)
if err != nil {
fmt.Println("Error, ", err)
return
}
fmt.Println(string(m))
}

Why can this data not be properly unmarshalled into my object model?

I have a (non)working example here: https://play.golang.org/p/qaYhKvJ65J3
I'm not sure why the following data:
alertData := `{
"Id": 0,
"Version": 0,
"OrgId": 1,
"DashboardId": 61,
"PanelId": 84,
"Name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"Message": "",
"Severity": "",
"State": "",
"Handler": 1,
"Silenced": false,
"ExecutionError": "",
"Frequency": 10,
"EvalData": null,
"NewStateDate": "0001-01-01T00:00:00Z",
"PrevStateDate": "0001-01-01T00:00:00Z",
"StateChanges": 0,
"Created": "0001-01-01T00:00:00Z",
"Updated": "0001-01-01T00:00:00Z",
"Settings": {
"conditions": [
{
"evaluator": {
"params": [
10000
],
"type": "gt"
},
"operator": {
"type": "and"
},
"query": {
"datasourceId": 2,
"model": {
"hide": true,
"refCount": 0,
"refId": "C",
"textEditor": false
},
"params": [
"C",
"5m",
"now"
]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"executionErrorState": "keep_state",
"frequency": "10s",
"handler": 1,
"name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"noDataState": "keep_state",
"notifications": []
}
}`
Can't be unmarshalled into the following object model:
type Condition struct {
Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
} `json:"evaluator"`
Operator struct {
Type string `json:"type"`
} `json:"operator"`
Query struct {
Params []string `json:"params"`
} `json:"query"`
Reducer struct {
Params []interface{} `json:"params"`
Type string `json:"type"`
} `json:"reducer"`
Type string `json:"type"`
}
When I do the following:
condition := Condition{}
err := json.Unmarshal([]byte(alertData), &condition)
if err != nil {
panic(err)
}
fmt.Printf("\n\n json object:::: %+v", condition)
I just get: json object:::: {Evaluator:{Params:[] Type:} Operator:{Type:} Query:{Params:[]} Reducer:{Params:[] Type:} Type:}
Ideally I'd be able to parse it into something like type Conditions []struct{ } but I'm not sure if you can define models as lists?
It looks like you are trying to access the "conditions" property nested under the root "Settings" property. As such, you need to define that root-level type and enough fields to tell the unmarshaler how to find your target property. As such, you would just need to create a new "AlertData" type with the necessary "Settings/conditions" fields.
For example (Go Playground):
type AlertData struct {
Settings struct {
Conditions []Condition `json:"conditions"`
}
}
func main() {
alert := AlertData{}
err := json.Unmarshal([]byte(alertData), &alert)
if err != nil {
panic(err)
}
fmt.Printf("OK: conditions=%#v\n", alert.Settings.Conditions)
// OK: conditions=[]main.Condition{main.Condition{Evaluator:struct { Params []int "json:\"params\""; Type string "json:\"type\"" }{Params:[]int{10000}, Type:"gt"}, Operator:struct { Type string "json:\"type\"" }{Type:"and"}, Query:struct { Params []string "json:\"params\"" }{Params:[]string{"C", "5m", "now"}}, Reducer:struct { Params []interface {} "json:\"params\""; Type string "json:\"type\"" }{Params:[]interface {}{}, Type:"avg"}, Type:"query"}}
}
Note that the printed listing includes so much type information because the "Condition" type uses anonymous structs as field types. If you were to extract them into named structs it will be easier to work with the data, e.g.:
type Condition struct {
Evaluator Evaluator `json:"evaluator"`
Operator Operator `json:"operator"`
// ...
}
type Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
}
type Operator struct {
Type string `json:"type"`
}
//...
// OK: conditions=[]main.Condition{
// main.Condition{
// Evaluator:main.Evaluator{Params:[]int{10000}, Type:"gt"},
// Operator:main.Operator{Type:"and"},
// Query:main.Query{Params:[]string{"C", "5m", "now"}},
// Reducer:main.Reducer{Params:[]interface {}{}, Type:"avg"},
// Type:"query",
// },
// }
Go Playground example here...
Maerics explanation is correct, here is an alternative approach which wraps access around struct methods, the data structure is also fully defined. If you're new to Go it's good to get handle on creating the data structures yourself, but here is a handy utility for helping create structs from valid JSON
https://mholt.github.io/json-to-go/
package main
import (
"encoding/json"
"fmt"
"log"
"time"
)
type Data struct {
ID int `json:"Id"`
Version int `json:"Version"`
OrgID int `json:"OrgId"`
DashboardID int `json:"DashboardId"`
PanelID int `json:"PanelId"`
Name string `json:"Name"`
Message string `json:"Message"`
Severity string `json:"Severity"`
State string `json:"State"`
Handler int `json:"Handler"`
Silenced bool `json:"Silenced"`
ExecutionError string `json:"ExecutionError"`
Frequency int `json:"Frequency"`
EvalData interface{} `json:"EvalData"`
NewStateDate time.Time `json:"NewStateDate"`
PrevStateDate time.Time `json:"PrevStateDate"`
StateChanges int `json:"StateChanges"`
Created time.Time `json:"Created"`
Updated time.Time `json:"Updated"`
Settings struct {
Conditions []Condition `json:"conditions"`
ExecutionErrorState string `json:"executionErrorState"`
Frequency string `json:"frequency"`
Handler int `json:"handler"`
Name string `json:"name"`
NoDataState string `json:"noDataState"`
Notifications []interface{} `json:"notifications"`
} `json:"Settings"`
}
type Condition struct {
Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
} `json:"evaluator"`
Operator struct {
Type string `json:"type"`
} `json:"operator"`
Query struct {
DatasourceID int `json:"datasourceId"`
Model struct {
Hide bool `json:"hide"`
RefCount int `json:"refCount"`
RefID string `json:"refId"`
TextEditor bool `json:"textEditor"`
} `json:"model"`
Params []string `json:"params"`
} `json:"query"`
Reducer struct {
Params []interface{} `json:"params"`
Type string `json:"type"`
} `json:"reducer"`
Type string `json:"type"`
}
func (d Data) GetFirstCondition() (Condition, error) {
if len(d.Settings.Conditions) > 0 {
return d.Settings.Conditions[0], nil
}
return Condition{}, fmt.Errorf("no conditions found")
}
func (d Data) GetConditionByIndex(index uint) (Condition, error) {
if len(d.Settings.Conditions) == 0 {
return Condition{}, fmt.Errorf("no conditions found")
}
if int(index) > len(d.Settings.Conditions)-1 {
return Condition{}, fmt.Errorf("index out of bounds")
}
return d.Settings.Conditions[index], nil
}
var alertData = `{
"Id": 0,
"Version": 0,
"OrgId": 1,
"DashboardId": 61,
"PanelId": 84,
"Name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"Message": "",
"Severity": "",
"State": "",
"Handler": 1,
"Silenced": false,
"ExecutionError": "",
"Frequency": 10,
"EvalData": null,
"NewStateDate": "0001-01-01T00:00:00Z",
"PrevStateDate": "0001-01-01T00:00:00Z",
"StateChanges": 0,
"Created": "0001-01-01T00:00:00Z",
"Updated": "0001-01-01T00:00:00Z",
"Settings": {
"conditions": [
{
"evaluator": {
"params": [
10000
],
"type": "gt"
},
"operator": {
"type": "and"
},
"query": {
"datasourceId": 2,
"model": {
"hide": true,
"refCount": 0,
"refId": "C",
"textEditor": false
},
"params": [
"C",
"5m",
"now"
]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"executionErrorState": "keep_state",
"frequency": "10s",
"handler": 1,
"name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"noDataState": "keep_state",
"notifications": []
}
}`
func main() {
var res Data
err := json.Unmarshal([]byte(alertData), &res)
if err != nil {
log.Fatal(err)
}
fmt.Println(res.GetFirstCondition())
fmt.Println(res.GetConditionByIndex(0))
// should fail :-)
fmt.Println(res.GetConditionByIndex(1))
}

Dynamic JSON struct in Golang doesn't behave as expected

I'm trying to create a struct with some basic fields which are always present and some optional fields which are structs by themselves.
I'm wondering why the following code:
package main
import (
"encoding/json"
"fmt"
"time"
)
type DataManagement struct {
DataManagement struct {
Type string
Asset struct {
LocalAssetUID string
Type string
}
*ContentProductionOrder
*ContentItem
TimeStamp time.Time
Hash string
}
}
type ContentProductionOrder struct {
ProductionOrderNo int
ItemNo int
StartDate time.Time
FinishDate time.Time
StatusID int
StatusDate time.Time
SourceTypeID int
TrackingID int
}
type ContentItem struct {
ItemNo int
ItemText string
Quantity int
}
func main() {
var contentItem ContentItem
contentItem.ItemNo = 123
contentItem.ItemText = "aaaaaaaa"
contentItem.Quantity = 3
var dataManagement DataManagement
dataManagement.DataManagement.Type = "asd"
dataManagement.DataManagement.Asset.LocalAssetUID = "asd"
dataManagement.DataManagement.Asset.Type = "asd"
dataManagement.DataManagement.ContentItem = &contentItem
dataManagement.DataManagement.TimeStamp = time.Now().UTC()
dataManagement.DataManagement.Hash = "123"
xy, _ := json.MarshalIndent(dataManagement, "", " ")
fmt.Println(string(xy))
xy, _ = json.MarshalIndent(contentItem, "", " ")
fmt.Println(string(xy))
}
outputs to the following:
{
"DataManagement": {
"Type": "asd",
"Asset": {
"LocalAssetUID": "asd",
"Type": "asd"
},
"ItemText": "aaaaaaaa",
"Quantity": 3,
"TimeStamp": "2009-11-10T23:00:00Z",
"Hash": "123"
}
}
{
"ItemNo": 123,
"ItemText": "aaaaaaaa",
"Quantity": 3
}
and not to:
{
"DataManagement": {
"Type": "asd",
"Asset": {
"LocalAssetUID": "asd",
"Type": "asd"
},
"ContentItem": {
"ItemNo": 123,
"ItemText": "aaaaaaaa",
"Quantity": 3
},
"TimeStamp": "2009-11-10T23:00:00Z",
"Hash": "123"
}
}
{
"ItemNo": 123,
"ItemText": "aaaaaaaa",
"Quantity": 3
}
Any ideas? It's probably pretty easy to explain; I'm not that experienced in Golang.
Here's a Playground link: https://play.golang.org/p/iRDcaRIZ_ZU
The output you are not getting which you want is because you have used embedded struct for ContentItem in DataManagement rather than field name to add to the struct.
A field declared with a type but no explicit field name is called an
embedded field. An embedded field must be specified as a type name T
or as a pointer to a non-interface type name *T, and T itself may not
be a pointer type. The unqualified type name acts as the field name.
A field declaration will sove your issue. You should change the struct DataManagement to:
type DataManagement struct {
DataManagement struct {
Type string
Asset struct {
LocalAssetUID string
Type string
}
*ContentProductionOrder // this is embedded struct
ContentItem *ContentItem
TimeStamp time.Time
Hash string
}
}
Working Code on Go Playground
For more information, Have a look at Golang Spec for Struct Types

How to unmarshal JSON in to array of interface and use

I am having difficulty understanding how to correctly unmarshal some JSON data that goes in to an array of type inteface and then use it. I tried to make this example code as simple as possible to illustrate the problem I am having. The code can be found in the playground here: https://play.golang.org/p/U85J_lBJ7Zr
The output looks like:
[map[ObjectType:chair ID:1234 Brand:Blue Inc.] map[ID:5678
Location:Kitchen ObjectType:table]] { } false { } false
Code
package main
import (
"fmt"
"encoding/json"
)
type Chair struct {
ObjectType string
ID string
Brand string
}
type Table struct {
ObjectType string
ID string
Location string
}
type House struct {
Name string
Objects []interface{}
}
func main() {
var h House
data := returnJSONBlob()
err := json.Unmarshal(data, &h)
if err != nil {
fmt.Println(err)
}
fmt.Println(h.Objects)
s1, ok := h.Objects[0].(Table)
fmt.Println(s1, ok)
s2, ok := h.Objects[0].(Chair)
fmt.Println(s2, ok)
}
func returnJSONBlob() []byte {
s := []byte(`
{
"Name": "house1",
"Objects": [
{
"ObjectType": "chair",
"ID": "1234",
"Brand": "Blue Inc."
},
{
"ObjectType": "table",
"ID": "5678",
"Location": "Kitchen"
}
]
}
`)
return s
}
I'm not sure if this is practical, since this is a simplified version of your scenario. However, one way to do this is combine the two object types to a new one, Object, and then unmarshal them directly to Object instead of using interface{}:
package main
import (
"encoding/json"
"fmt"
)
type Object struct {
ObjectType string
ID string
Brand string
Location string
}
type House struct {
Name string
Objects []Object
}
func returnJSONBlob() []byte {
s := []byte(`
{
"Name": "house1",
"Objects": [
{
"ObjectType": "chair",
"ID": "1234",
"Brand": "Blue Inc."
},
{
"ObjectType": "table",
"ID": "5678",
"Location": "Kitchen"
}
]
}
`)
return s
}
func main() {
var h House
data := returnJSONBlob()
err := json.Unmarshal(data, &h)
if err != nil {
fmt.Println(err)
}
fmt.Println(h.Objects[0].Brand)
fmt.Println(h.Objects[1].Location)
}
Prints:
Blue Inc.
Kitchen
Example here: https://play.golang.org/p/91F4UrQlSjJ

How to decode json into structs

I'm trying to decode some json in Go but some fields don't get decoded.
See the code running in browser here:
What am I doing wrong?
I need only the MX records so I didn't define the other fields. As I understand from the godoc you don't need to define the fields you don't use/need.
// You can edit this code!
// Click here and start typing.
package main
import "fmt"
import "encoding/json"
func main() {
body := `
{"response": {
"status": "SUCCESS",
"data": {
"mxRecords": [
{
"value": "us2.mx3.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
},
{
"value": "us2.mx1.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
},
{
"value": "us2.mx2.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
}
],
"cnameRecords": [
{
"aliasHost": "pop.a.co.uk.",
"canonicalHost": "us2.pop.mailhostbox.com."
},
{
"aliasHost": "webmail.a.co.uk.",
"canonicalHost": "us2.webmail.mailhostbox.com."
},
{
"aliasHost": "smtp.a.co.uk.",
"canonicalHost": "us2.smtp.mailhostbox.com."
},
{
"aliasHost": "imap.a.co.uk.",
"canonicalHost": "us2.imap.mailhostbox.com."
}
],
"dkimTxtRecord": {
"domainname": "20a19._domainkey.a.co.uk",
"value": "\"v=DKIM1; g=*; k=rsa; p=DkfbhO8Oyy0E1WyUWwIDAQAB\"",
"ttl": 1
},
"spfTxtRecord": {
"domainname": "a.co.uk",
"value": "\"v=spf1 redirect=_spf.mailhostbox.com\"",
"ttl": 1
},
"loginUrl": "us2.cp.mailhostbox.com"
}
}}`
type MxRecords struct {
value string
ttl int
priority int
hostName string
}
type Data struct {
mxRecords []MxRecords
}
type Response struct {
Status string `json:"status"`
Data Data `json:"data"`
}
type apiR struct {
Response Response
}
var r apiR
err := json.Unmarshal([]byte(body), &r)
if err != nil {
fmt.Printf("err was %v", err)
}
fmt.Printf("decoded is %v", r)
}
As per the go documentaiton about json.Unmarshal, you can only decode toward exported fields, the main reason being that external packages (such as encoding/json) cannot acces unexported fields.
If your json doesn't follow the go convention for names, you can use the json tag in your fields to change the matching between json key and struct field.
Exemple:
package main
import (
"fmt"
"encoding/json"
)
type T struct {
Foo string `json:"foo"`
priv string `json:"priv"`
}
func main() {
text := []byte(`{"foo":"bar", "priv":"nothing"}`)
var t T
err := json.Unmarshal(text, &t)
if err != nil {
panic(err)
}
fmt.Println(t.Foo) // prints "bar"
fmt.Println(t.priv) // prints "", priv is not exported
}
You must Uppercase struct fields:
type MxRecords struct {
Value string `json:"value"`
Ttl int `json:"ttl"`
Priority int `json:"priority"`
HostName string `json:"hostName"`
}
type Data struct {
MxRecords []MxRecords `json:"mxRecords"`
}
http://play.golang.org/p/EEyiISdoaE
The encoding/json package can only decode into exported struct fields. Your Data.mxRecords member is not exported, so it is ignored when decoding. If you rename it to use a capital letter, the JSON package will notice it.
You will need to do the same thing for all the members of your MxRecords type.