Parsing JSON in GoLang into struct - json

So, I'm having some trouble parsing this data in golang:
{
"gateways": [
{
"token": "my_token_here",
"gateway_type": "test",
"description": null,
"payment_methods": [
"credit_card",
"sprel",
"third_party_token",
"bank_account",
"apple_pay"
],
"state": "retained",
"created_at": "2016-03-12T18:52:37Z",
"updated_at": "2016-03-12T18:52:37Z",
"name": "Spreedly Test",
"characteristics": [
"purchase",
"authorize",
"capture",
"credit",
"general_credit",
"void",
"verify",
"reference_purchase",
"purchase_via_preauthorization",
"offsite_purchase",
"offsite_authorize",
"3dsecure_purchase",
"3dsecure_authorize",
"store",
"remove",
"disburse",
"reference_authorization"
],
"credentials": [],
"gateway_specific_fields": [],
"redacted": false
}
]
}
When using this struct I can get it to output pretty easily.
type gateways struct {
Gateways []struct {
Characteristics []string `json:"characteristics"`
CreatedAt string `json:"created_at"`
Credentials []interface{} `json:"credentials"`
Description interface{} `json:"description"`
GatewaySpecificFields []interface{} `json:"gateway_specific_fields"`
GatewayType string `json:"gateway_type"`
Name string `json:"name"`
PaymentMethods []string `json:"payment_methods"`
Redacted bool `json:"redacted"`
State string `json:"state"`
Token string `json:"token"`
UpdatedAt string `json:"updated_at"`
} `json:"gateways"`
}
But as soon as I seperate the "Gateways []struct" into its own struct then it returns an empty array...
Full source.
type gateway struct {
Characteristics []string `json:"characteristics"`
CreatedAt string `json:"created_at"`
Credentials []interface{} `json:"credentials"`
Description interface{} `json:"description"`
GatewaySpecificFields []interface{} `json:"gateway_specific_fields"`
GatewayType string `json:"gateway_type"`
Name string `json:"name"`
PaymentMethods []string `json:"payment_methods"`
Redacted bool `json:"redacted"`
State string `json:"state"`
Token string `json:"token"`
UpdatedAt string `json:"updated_at"`
}
type gateways struct {
Gateways []gateway `json:"gateways"`
}
func ParseResponse() {
var parsed gateways
json.Unmarshal(json, &parsed)
}

There's a problem with your ParseResponse function, you're calling json.Unmarshal passing as first parameter json, that's a packge name: that's ambiguous.
As you can see, your code works well changing the ParseResponse function.
package main
import (
"encoding/json"
"fmt"
)
type gateway struct {
Characteristics []string `json:"characteristics"`
CreatedAt string `json:"created_at"`
Credentials []interface{} `json:"credentials"`
Description interface{} `json:"description"`
GatewaySpecificFields []interface{} `json:"gateway_specific_fields"`
GatewayType string `json:"gateway_type"`
Name string `json:"name"`
PaymentMethods []string `json:"payment_methods"`
Redacted bool `json:"redacted"`
State string `json:"state"`
Token string `json:"token"`
UpdatedAt string `json:"updated_at"`
}
type gateways struct {
Gateways []gateway `json:"gateways"`
}
func ParseResponse(js []byte) {
var parsed gateways
json.Unmarshal(js, &parsed)
fmt.Println(parsed)
}
func main() {
var js []byte = []byte(`{
"gateways": [
{
"token": "my_token_here",
"gateway_type": "test",
"description": null,
"payment_methods": [
"credit_card",
"sprel",
"third_party_token",
"bank_account",
"apple_pay"
],
"state": "retained",
"created_at": "2016-03-12T18:52:37Z",
"updated_at": "2016-03-12T18:52:37Z",
"name": "Spreedly Test",
"characteristics": [
"purchase",
"authorize",
"capture",
"credit",
"general_credit",
"void",
"verify",
"reference_purchase",
"purchase_via_preauthorization",
"offsite_purchase",
"offsite_authorize",
"3dsecure_purchase",
"3dsecure_authorize",
"store",
"remove",
"disburse",
"reference_authorization"
],
"credentials": [],
"gateway_specific_fields": [],
"redacted": false
}
]
}`)
/*
var parsed gateways
e := json.Unmarshal(js, &parsed)
if e != nil {
fmt.Println(e.Error())
} else {
fmt.Println(parsed)
}
*/
ParseResponse(js)
}
Outputs:
{[{[purchase authorize capture credit general_credit void verify reference_purchase purchase_via_preauthorization offsite_purchase offsite_authorize 3dsecure_purchase 3dsecure_authorize store remove disburse reference_authorization] 2016-03-12T18:52:37Z [] <nil> [] test Spreedly Test [credit_card sprel third_party_token bank_account apple_pay] false retained my_token_here 2016-03-12T18:52:37Z}]}

Take a look at http://play.golang.org/p/3xJHBmhuei - unmarshalling into your second definition of gateways completes successfully, so it must be something little that you're missing.
Check if the json.Unmarshal call returns an error.
P.S. It probably isn't the problem if you're unmarshalling successfully into the first version of gateways, but the JSON string that you've given above is missing a closing "}" bracket.

Related

Why can this data not be properly unmarshalled into my object model?

I have a (non)working example here: https://play.golang.org/p/qaYhKvJ65J3
I'm not sure why the following data:
alertData := `{
"Id": 0,
"Version": 0,
"OrgId": 1,
"DashboardId": 61,
"PanelId": 84,
"Name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"Message": "",
"Severity": "",
"State": "",
"Handler": 1,
"Silenced": false,
"ExecutionError": "",
"Frequency": 10,
"EvalData": null,
"NewStateDate": "0001-01-01T00:00:00Z",
"PrevStateDate": "0001-01-01T00:00:00Z",
"StateChanges": 0,
"Created": "0001-01-01T00:00:00Z",
"Updated": "0001-01-01T00:00:00Z",
"Settings": {
"conditions": [
{
"evaluator": {
"params": [
10000
],
"type": "gt"
},
"operator": {
"type": "and"
},
"query": {
"datasourceId": 2,
"model": {
"hide": true,
"refCount": 0,
"refId": "C",
"textEditor": false
},
"params": [
"C",
"5m",
"now"
]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"executionErrorState": "keep_state",
"frequency": "10s",
"handler": 1,
"name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"noDataState": "keep_state",
"notifications": []
}
}`
Can't be unmarshalled into the following object model:
type Condition struct {
Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
} `json:"evaluator"`
Operator struct {
Type string `json:"type"`
} `json:"operator"`
Query struct {
Params []string `json:"params"`
} `json:"query"`
Reducer struct {
Params []interface{} `json:"params"`
Type string `json:"type"`
} `json:"reducer"`
Type string `json:"type"`
}
When I do the following:
condition := Condition{}
err := json.Unmarshal([]byte(alertData), &condition)
if err != nil {
panic(err)
}
fmt.Printf("\n\n json object:::: %+v", condition)
I just get: json object:::: {Evaluator:{Params:[] Type:} Operator:{Type:} Query:{Params:[]} Reducer:{Params:[] Type:} Type:}
Ideally I'd be able to parse it into something like type Conditions []struct{ } but I'm not sure if you can define models as lists?
It looks like you are trying to access the "conditions" property nested under the root "Settings" property. As such, you need to define that root-level type and enough fields to tell the unmarshaler how to find your target property. As such, you would just need to create a new "AlertData" type with the necessary "Settings/conditions" fields.
For example (Go Playground):
type AlertData struct {
Settings struct {
Conditions []Condition `json:"conditions"`
}
}
func main() {
alert := AlertData{}
err := json.Unmarshal([]byte(alertData), &alert)
if err != nil {
panic(err)
}
fmt.Printf("OK: conditions=%#v\n", alert.Settings.Conditions)
// OK: conditions=[]main.Condition{main.Condition{Evaluator:struct { Params []int "json:\"params\""; Type string "json:\"type\"" }{Params:[]int{10000}, Type:"gt"}, Operator:struct { Type string "json:\"type\"" }{Type:"and"}, Query:struct { Params []string "json:\"params\"" }{Params:[]string{"C", "5m", "now"}}, Reducer:struct { Params []interface {} "json:\"params\""; Type string "json:\"type\"" }{Params:[]interface {}{}, Type:"avg"}, Type:"query"}}
}
Note that the printed listing includes so much type information because the "Condition" type uses anonymous structs as field types. If you were to extract them into named structs it will be easier to work with the data, e.g.:
type Condition struct {
Evaluator Evaluator `json:"evaluator"`
Operator Operator `json:"operator"`
// ...
}
type Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
}
type Operator struct {
Type string `json:"type"`
}
//...
// OK: conditions=[]main.Condition{
// main.Condition{
// Evaluator:main.Evaluator{Params:[]int{10000}, Type:"gt"},
// Operator:main.Operator{Type:"and"},
// Query:main.Query{Params:[]string{"C", "5m", "now"}},
// Reducer:main.Reducer{Params:[]interface {}{}, Type:"avg"},
// Type:"query",
// },
// }
Go Playground example here...
Maerics explanation is correct, here is an alternative approach which wraps access around struct methods, the data structure is also fully defined. If you're new to Go it's good to get handle on creating the data structures yourself, but here is a handy utility for helping create structs from valid JSON
https://mholt.github.io/json-to-go/
package main
import (
"encoding/json"
"fmt"
"log"
"time"
)
type Data struct {
ID int `json:"Id"`
Version int `json:"Version"`
OrgID int `json:"OrgId"`
DashboardID int `json:"DashboardId"`
PanelID int `json:"PanelId"`
Name string `json:"Name"`
Message string `json:"Message"`
Severity string `json:"Severity"`
State string `json:"State"`
Handler int `json:"Handler"`
Silenced bool `json:"Silenced"`
ExecutionError string `json:"ExecutionError"`
Frequency int `json:"Frequency"`
EvalData interface{} `json:"EvalData"`
NewStateDate time.Time `json:"NewStateDate"`
PrevStateDate time.Time `json:"PrevStateDate"`
StateChanges int `json:"StateChanges"`
Created time.Time `json:"Created"`
Updated time.Time `json:"Updated"`
Settings struct {
Conditions []Condition `json:"conditions"`
ExecutionErrorState string `json:"executionErrorState"`
Frequency string `json:"frequency"`
Handler int `json:"handler"`
Name string `json:"name"`
NoDataState string `json:"noDataState"`
Notifications []interface{} `json:"notifications"`
} `json:"Settings"`
}
type Condition struct {
Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
} `json:"evaluator"`
Operator struct {
Type string `json:"type"`
} `json:"operator"`
Query struct {
DatasourceID int `json:"datasourceId"`
Model struct {
Hide bool `json:"hide"`
RefCount int `json:"refCount"`
RefID string `json:"refId"`
TextEditor bool `json:"textEditor"`
} `json:"model"`
Params []string `json:"params"`
} `json:"query"`
Reducer struct {
Params []interface{} `json:"params"`
Type string `json:"type"`
} `json:"reducer"`
Type string `json:"type"`
}
func (d Data) GetFirstCondition() (Condition, error) {
if len(d.Settings.Conditions) > 0 {
return d.Settings.Conditions[0], nil
}
return Condition{}, fmt.Errorf("no conditions found")
}
func (d Data) GetConditionByIndex(index uint) (Condition, error) {
if len(d.Settings.Conditions) == 0 {
return Condition{}, fmt.Errorf("no conditions found")
}
if int(index) > len(d.Settings.Conditions)-1 {
return Condition{}, fmt.Errorf("index out of bounds")
}
return d.Settings.Conditions[index], nil
}
var alertData = `{
"Id": 0,
"Version": 0,
"OrgId": 1,
"DashboardId": 61,
"PanelId": 84,
"Name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"Message": "",
"Severity": "",
"State": "",
"Handler": 1,
"Silenced": false,
"ExecutionError": "",
"Frequency": 10,
"EvalData": null,
"NewStateDate": "0001-01-01T00:00:00Z",
"PrevStateDate": "0001-01-01T00:00:00Z",
"StateChanges": 0,
"Created": "0001-01-01T00:00:00Z",
"Updated": "0001-01-01T00:00:00Z",
"Settings": {
"conditions": [
{
"evaluator": {
"params": [
10000
],
"type": "gt"
},
"operator": {
"type": "and"
},
"query": {
"datasourceId": 2,
"model": {
"hide": true,
"refCount": 0,
"refId": "C",
"textEditor": false
},
"params": [
"C",
"5m",
"now"
]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"executionErrorState": "keep_state",
"frequency": "10s",
"handler": 1,
"name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"noDataState": "keep_state",
"notifications": []
}
}`
func main() {
var res Data
err := json.Unmarshal([]byte(alertData), &res)
if err != nil {
log.Fatal(err)
}
fmt.Println(res.GetFirstCondition())
fmt.Println(res.GetConditionByIndex(0))
// should fail :-)
fmt.Println(res.GetConditionByIndex(1))
}

Dynamic JSON struct in Golang doesn't behave as expected

I'm trying to create a struct with some basic fields which are always present and some optional fields which are structs by themselves.
I'm wondering why the following code:
package main
import (
"encoding/json"
"fmt"
"time"
)
type DataManagement struct {
DataManagement struct {
Type string
Asset struct {
LocalAssetUID string
Type string
}
*ContentProductionOrder
*ContentItem
TimeStamp time.Time
Hash string
}
}
type ContentProductionOrder struct {
ProductionOrderNo int
ItemNo int
StartDate time.Time
FinishDate time.Time
StatusID int
StatusDate time.Time
SourceTypeID int
TrackingID int
}
type ContentItem struct {
ItemNo int
ItemText string
Quantity int
}
func main() {
var contentItem ContentItem
contentItem.ItemNo = 123
contentItem.ItemText = "aaaaaaaa"
contentItem.Quantity = 3
var dataManagement DataManagement
dataManagement.DataManagement.Type = "asd"
dataManagement.DataManagement.Asset.LocalAssetUID = "asd"
dataManagement.DataManagement.Asset.Type = "asd"
dataManagement.DataManagement.ContentItem = &contentItem
dataManagement.DataManagement.TimeStamp = time.Now().UTC()
dataManagement.DataManagement.Hash = "123"
xy, _ := json.MarshalIndent(dataManagement, "", " ")
fmt.Println(string(xy))
xy, _ = json.MarshalIndent(contentItem, "", " ")
fmt.Println(string(xy))
}
outputs to the following:
{
"DataManagement": {
"Type": "asd",
"Asset": {
"LocalAssetUID": "asd",
"Type": "asd"
},
"ItemText": "aaaaaaaa",
"Quantity": 3,
"TimeStamp": "2009-11-10T23:00:00Z",
"Hash": "123"
}
}
{
"ItemNo": 123,
"ItemText": "aaaaaaaa",
"Quantity": 3
}
and not to:
{
"DataManagement": {
"Type": "asd",
"Asset": {
"LocalAssetUID": "asd",
"Type": "asd"
},
"ContentItem": {
"ItemNo": 123,
"ItemText": "aaaaaaaa",
"Quantity": 3
},
"TimeStamp": "2009-11-10T23:00:00Z",
"Hash": "123"
}
}
{
"ItemNo": 123,
"ItemText": "aaaaaaaa",
"Quantity": 3
}
Any ideas? It's probably pretty easy to explain; I'm not that experienced in Golang.
Here's a Playground link: https://play.golang.org/p/iRDcaRIZ_ZU
The output you are not getting which you want is because you have used embedded struct for ContentItem in DataManagement rather than field name to add to the struct.
A field declared with a type but no explicit field name is called an
embedded field. An embedded field must be specified as a type name T
or as a pointer to a non-interface type name *T, and T itself may not
be a pointer type. The unqualified type name acts as the field name.
A field declaration will sove your issue. You should change the struct DataManagement to:
type DataManagement struct {
DataManagement struct {
Type string
Asset struct {
LocalAssetUID string
Type string
}
*ContentProductionOrder // this is embedded struct
ContentItem *ContentItem
TimeStamp time.Time
Hash string
}
}
Working Code on Go Playground
For more information, Have a look at Golang Spec for Struct Types

Golang Json marshalling

type Status struct {
slug string `json:"slug"`
}
type Instance struct {
Slug string `json:"slug"`
Stat map[string]*Status `json:"status"`
}
This Json response is recieved for API call:
[ {
"status": {
"stop": false,
"traffic": true,
"label": null,
"dead": true,
"slug": "up",
},
"slug": "instances_raw",
"started_at": "15120736198",
"replacement": null
},
{
"status": {
"stop": false,
"traffic": true,
"label": null,
"dead": true,
"slug": "down",
},
"slug": "instance_raw2",
"started_at": "1512073194",
"replacement": null
}
]
I am trying to marshall json into above struct but running into issue:
instances := make([]Instance, 0)
res := api call return above json
body, _ := ioutil.ReadAll(res.Body)
json.Unmarshal(body, &instances)
fmt.Println("I am struct %s",instances)
It is marshalling into:
I am struct %s [{ map[stop:0xc42018e1b0 dead:0xc42018e1e0 label:<nil> running:0xc42018e220 down:0xc42018e150 traffic:0xc42018e180]}]
Can someone help me figure out why it is not marshalling as I am expecting?
Expected marshalling:
[{instances_raw map[slug:up]} {instances_raw2 map[slug:down]}]
The structs do not match the structure of the data. Perhaps you wanted this:
type Status struct {
Slug string `json:"slug"`
}
type Instance struct {
Slug string `json:"slug"`
Stat Status `json:"status"`
}
Output: [{instances_raw {up}} {instance_raw2 {down}}]
run it on the plaground
or this:
type Instance struct {
Slug string `json:"slug"`
Stat map[string]interface{} `json:"status"`
}
Output: [{instances_raw map[label: dead:true slug:up stop:false traffic:true]} {instance_raw2 map[slug:down stop:false traffic:true label: dead:true]}]
run it on the playground
Always check errors. The example JSON above is not valid and the json.Unmarshal function reports this error.

JSON string array without Keys into a GoLang struct

I am trying to parse JSON that is submitted from a POST request into a struct in a web service to send email. The following JSON is submitted in the body of the request.
{
"body": {
"template": "abctemplate.html",
"params": {
"name": "Chase",
"email": "1234#gmail.com"
}
},
"to": [
"abc#gmail.com",
"xyz#gmail.com"
],
"cc": [
"xxx#example.com",
"yyy#example.com"
],
"replyTo": {
"email": "aaa#gmail.com",
"name": "Jack"
},
"bcc": "ccc#gmail.com",
"subject": "Hello, world!"
}
This is mapped and read into the following struct
type emailData struct {
Body struct {
Template string `json:"template"`
Params map[string]string `json:"params"`
} `json:"body"`
To map[string]string `json:"To"` // TODO This is wrong
CC string `json:"cc"` // TODO this is wrong
ReplyTo struct {
Email string `json:"email"`
Name string `json:"name"`
}
BCC string `json:"bcc"`
Subject string `json:"subject"`
}
Both the 'to' and 'cc' JSON fields are string arrays of unknown length and do not have keys. Is there a way to map the string arrays into the struct fields? I've tried the two different ways where there are // TODO tags with no luck. Thanks!
Both cc and to are json arrays which you can unmarshal into Go slices without worrying about the length.
type emailData struct {
Body struct {
Template string `json:"template"`
Params map[string]string `json:"params"`
} `json:"body"`
To []string `json:"to"`
CC []string `json:"cc"`
ReplyTo struct {
Email string `json:"email"`
Name string `json:"name"`
}
BCC string `json:"bcc"`
Subject string `json:"subject"`
}
https://play.golang.org/p/Pi_5aSs922
Use the below to convert the json to struct in golang:
Json-goStruct
Take care of the maps, which might be go struct and vice-versa.

Encoding nested JSON in Go

I've had a lot of trouble finding an example of this. Most of the information on the internet is about decoding JSON.
I'd like to serialize some data into nested JSON, like for example:
{
"item": {
"title": "Items",
"properties": [
{
"num": 1,
"name": "Item 1"
},
{
"num": 2,
"name": "Item 2"
}
]
}
}
I know how to marshal data with a flat struct, but how do I put data into a struct that can be serialized with nesting?
http://play.golang.org/p/nDKmv1myTD
I found this tool that generates a struct from a JSON schema, but I don't understand how to get data into the sub structs.
http://mholt.github.io/json-to-go/
type Example struct {
Item struct {
Title string `json:"title"`
Properties []struct {
Num int `json:"num"`
Name string `json:"name"`
} `json:"properties"`
} `json:"item"`
}
This tool you found is nice, but I would not use it. It makes it difficult to initialize the structs.
Init example with your snippet: (http://play.golang.org/p/_Qw3Qp8XZh)
package main
import (
"encoding/json"
"fmt"
)
type Example struct {
Item struct {
Title string `json:"title"`
Properties []struct {
Num int `json:"num"`
Name string `json:"name"`
} `json:"properties"`
} `json:"item"`
}
func main() {
info := &Example{
Item: struct {
Title string `json:"title"`
Properties []struct {
Num int `json:"num"`
Name string `json:"name"`
} `json:"properties"`
}{
Title: "title",
Properties: []struct {
Num int `json:"num"`
Name string `json:"name"`
}{
{Num: 0, Name: "name0"},
{Num: 1, Name: "name1"},
},
},
}
b, err := json.Marshal(info)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(b))
}
Result (pretty printed):
{
"item": {
"title": "title",
"properties": [
{
"num": 0,
"name": "name0"
},
{
"num": 1,
"name": "name1"
}
]
}
}
I think it is better to use named struct vs anonymous nested ones.
Same example with named structs: http://play.golang.org/p/xm7BXxEGTC
package main
import (
"encoding/json"
"fmt"
)
type Example struct {
Item Item `json:"item"`
}
type Item struct {
Title string `json:"title"`
Properties []Property `json:"properties"`
}
type Property struct {
Num int `json:"num"`
Name string `json:"name"`
}
func main() {
info := &Example{
Item: Item{
Title: "title",
Properties: []Property{
{Num: 0, Name: "name0"},
{Num: 1, Name: "name1"},
},
},
}
b, err := json.Marshal(info)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(b))
}
It is the exact same thing, but I find it more clear and easy to use.