Dynamic JSON struct in Golang doesn't behave as expected - json

I'm trying to create a struct with some basic fields which are always present and some optional fields which are structs by themselves.
I'm wondering why the following code:
package main
import (
"encoding/json"
"fmt"
"time"
)
type DataManagement struct {
DataManagement struct {
Type string
Asset struct {
LocalAssetUID string
Type string
}
*ContentProductionOrder
*ContentItem
TimeStamp time.Time
Hash string
}
}
type ContentProductionOrder struct {
ProductionOrderNo int
ItemNo int
StartDate time.Time
FinishDate time.Time
StatusID int
StatusDate time.Time
SourceTypeID int
TrackingID int
}
type ContentItem struct {
ItemNo int
ItemText string
Quantity int
}
func main() {
var contentItem ContentItem
contentItem.ItemNo = 123
contentItem.ItemText = "aaaaaaaa"
contentItem.Quantity = 3
var dataManagement DataManagement
dataManagement.DataManagement.Type = "asd"
dataManagement.DataManagement.Asset.LocalAssetUID = "asd"
dataManagement.DataManagement.Asset.Type = "asd"
dataManagement.DataManagement.ContentItem = &contentItem
dataManagement.DataManagement.TimeStamp = time.Now().UTC()
dataManagement.DataManagement.Hash = "123"
xy, _ := json.MarshalIndent(dataManagement, "", " ")
fmt.Println(string(xy))
xy, _ = json.MarshalIndent(contentItem, "", " ")
fmt.Println(string(xy))
}
outputs to the following:
{
"DataManagement": {
"Type": "asd",
"Asset": {
"LocalAssetUID": "asd",
"Type": "asd"
},
"ItemText": "aaaaaaaa",
"Quantity": 3,
"TimeStamp": "2009-11-10T23:00:00Z",
"Hash": "123"
}
}
{
"ItemNo": 123,
"ItemText": "aaaaaaaa",
"Quantity": 3
}
and not to:
{
"DataManagement": {
"Type": "asd",
"Asset": {
"LocalAssetUID": "asd",
"Type": "asd"
},
"ContentItem": {
"ItemNo": 123,
"ItemText": "aaaaaaaa",
"Quantity": 3
},
"TimeStamp": "2009-11-10T23:00:00Z",
"Hash": "123"
}
}
{
"ItemNo": 123,
"ItemText": "aaaaaaaa",
"Quantity": 3
}
Any ideas? It's probably pretty easy to explain; I'm not that experienced in Golang.
Here's a Playground link: https://play.golang.org/p/iRDcaRIZ_ZU

The output you are not getting which you want is because you have used embedded struct for ContentItem in DataManagement rather than field name to add to the struct.
A field declared with a type but no explicit field name is called an
embedded field. An embedded field must be specified as a type name T
or as a pointer to a non-interface type name *T, and T itself may not
be a pointer type. The unqualified type name acts as the field name.
A field declaration will sove your issue. You should change the struct DataManagement to:
type DataManagement struct {
DataManagement struct {
Type string
Asset struct {
LocalAssetUID string
Type string
}
*ContentProductionOrder // this is embedded struct
ContentItem *ContentItem
TimeStamp time.Time
Hash string
}
}
Working Code on Go Playground
For more information, Have a look at Golang Spec for Struct Types

Related

Convert string having json to json or struct

I am getting this type of response from an API:
{
"ok": true,
"response": "[
{
"Id": 163,
"Name": "Availability",
"Path": "Performance|Tier1",
"frequency": "ONE_MIN",
"Values": [
{
"startTimeInMillis": 1571314200000,
"occurrences": 1,
"current": 1,
"min": 0,
"max": 0,
"useRange": false,
"count": 1,
"sum": 1,
"value": 1,
"standardDeviation": 0
},
{
"startTimeInMillis": 1571314260000,
"occurrences": 1,
"current": 1,
"min": 0,
"max": 0,
"useRange": false,
"count": 1,
"sum": 1,
"value": 1,
"standardDeviation": 0
},
}
]
}
]
}
I want to convert this into time series format. To do so first I am trying to unmarshal the response to this struct:
type App struct{
ID string `json:"metric_id"`
Name string `json:"metric_name"`
Path string `json:"metric_path"`
Frequency string `json:"frequency"`
Values []string `json:"metric_values"`
}
I am doing this:
apprsp := App{}
fmt.Println(json.Unmarshal([]byte(ame.Response), &apprsp))
But I am getting error while json.Unmarshal.
What I am trying to do is to generate a json of format:
{'time':'value','time1':'value2'}
Where time/time1 and value/value2 is startTimeInMillis and value from values array.
What I am doing wrong while json unmarshal? What should be done to unmarshal the above data?
Your App structure is not even closely related to the json document you're trying to unmarshal. To unmarshal a json document, you have to have a Go structure that somewhat matches the structure of the underlying document.
type ResponseValue struct {
StartTime int64 `json:"startTimeMillis"`
// other elements of Values here, if you're interested in them
}
type Response struct {
Id int `json:"Id"`
Name string `json:"Name"`
Path string `json:"Path"`
Frequency string `json:"frequency"`
Values []ResponseValue `json:"Values"`
}
type Body struct {
Response []Response `json:"response"`
}
var data Body
json.Unmarshal([]byte(ame.Response),&data)
Then, you can extract the time series from data.
As the one above me just said your mapping from your json to the struct is wrong.
An easy way to find the write mapping is to use this tool https://mholt.github.io/json-to-go/
It enables you to map json to an auto generated struct.
here is your response struct
type Response struct {
ID int `json:"Id"`
Name string `json:"Name"`
Path string `json:"Path"`
Frequency string `json:"frequency"`
Values []struct {
StartTimeInMillis int64 `json:"startTimeInMillis"`
Occurrences int `json:"occurrences"`
Current int `json:"current"`
Min int `json:"min"`
Max int `json:"max"`
UseRange bool `json:"useRange"`
Count int `json:"count"`
Sum int `json:"sum"`
Value int `json:"value"`
StandardDeviation int `json:"standardDeviation"`
} `json:"Values"`
}

Why can this data not be properly unmarshalled into my object model?

I have a (non)working example here: https://play.golang.org/p/qaYhKvJ65J3
I'm not sure why the following data:
alertData := `{
"Id": 0,
"Version": 0,
"OrgId": 1,
"DashboardId": 61,
"PanelId": 84,
"Name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"Message": "",
"Severity": "",
"State": "",
"Handler": 1,
"Silenced": false,
"ExecutionError": "",
"Frequency": 10,
"EvalData": null,
"NewStateDate": "0001-01-01T00:00:00Z",
"PrevStateDate": "0001-01-01T00:00:00Z",
"StateChanges": 0,
"Created": "0001-01-01T00:00:00Z",
"Updated": "0001-01-01T00:00:00Z",
"Settings": {
"conditions": [
{
"evaluator": {
"params": [
10000
],
"type": "gt"
},
"operator": {
"type": "and"
},
"query": {
"datasourceId": 2,
"model": {
"hide": true,
"refCount": 0,
"refId": "C",
"textEditor": false
},
"params": [
"C",
"5m",
"now"
]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"executionErrorState": "keep_state",
"frequency": "10s",
"handler": 1,
"name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"noDataState": "keep_state",
"notifications": []
}
}`
Can't be unmarshalled into the following object model:
type Condition struct {
Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
} `json:"evaluator"`
Operator struct {
Type string `json:"type"`
} `json:"operator"`
Query struct {
Params []string `json:"params"`
} `json:"query"`
Reducer struct {
Params []interface{} `json:"params"`
Type string `json:"type"`
} `json:"reducer"`
Type string `json:"type"`
}
When I do the following:
condition := Condition{}
err := json.Unmarshal([]byte(alertData), &condition)
if err != nil {
panic(err)
}
fmt.Printf("\n\n json object:::: %+v", condition)
I just get: json object:::: {Evaluator:{Params:[] Type:} Operator:{Type:} Query:{Params:[]} Reducer:{Params:[] Type:} Type:}
Ideally I'd be able to parse it into something like type Conditions []struct{ } but I'm not sure if you can define models as lists?
It looks like you are trying to access the "conditions" property nested under the root "Settings" property. As such, you need to define that root-level type and enough fields to tell the unmarshaler how to find your target property. As such, you would just need to create a new "AlertData" type with the necessary "Settings/conditions" fields.
For example (Go Playground):
type AlertData struct {
Settings struct {
Conditions []Condition `json:"conditions"`
}
}
func main() {
alert := AlertData{}
err := json.Unmarshal([]byte(alertData), &alert)
if err != nil {
panic(err)
}
fmt.Printf("OK: conditions=%#v\n", alert.Settings.Conditions)
// OK: conditions=[]main.Condition{main.Condition{Evaluator:struct { Params []int "json:\"params\""; Type string "json:\"type\"" }{Params:[]int{10000}, Type:"gt"}, Operator:struct { Type string "json:\"type\"" }{Type:"and"}, Query:struct { Params []string "json:\"params\"" }{Params:[]string{"C", "5m", "now"}}, Reducer:struct { Params []interface {} "json:\"params\""; Type string "json:\"type\"" }{Params:[]interface {}{}, Type:"avg"}, Type:"query"}}
}
Note that the printed listing includes so much type information because the "Condition" type uses anonymous structs as field types. If you were to extract them into named structs it will be easier to work with the data, e.g.:
type Condition struct {
Evaluator Evaluator `json:"evaluator"`
Operator Operator `json:"operator"`
// ...
}
type Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
}
type Operator struct {
Type string `json:"type"`
}
//...
// OK: conditions=[]main.Condition{
// main.Condition{
// Evaluator:main.Evaluator{Params:[]int{10000}, Type:"gt"},
// Operator:main.Operator{Type:"and"},
// Query:main.Query{Params:[]string{"C", "5m", "now"}},
// Reducer:main.Reducer{Params:[]interface {}{}, Type:"avg"},
// Type:"query",
// },
// }
Go Playground example here...
Maerics explanation is correct, here is an alternative approach which wraps access around struct methods, the data structure is also fully defined. If you're new to Go it's good to get handle on creating the data structures yourself, but here is a handy utility for helping create structs from valid JSON
https://mholt.github.io/json-to-go/
package main
import (
"encoding/json"
"fmt"
"log"
"time"
)
type Data struct {
ID int `json:"Id"`
Version int `json:"Version"`
OrgID int `json:"OrgId"`
DashboardID int `json:"DashboardId"`
PanelID int `json:"PanelId"`
Name string `json:"Name"`
Message string `json:"Message"`
Severity string `json:"Severity"`
State string `json:"State"`
Handler int `json:"Handler"`
Silenced bool `json:"Silenced"`
ExecutionError string `json:"ExecutionError"`
Frequency int `json:"Frequency"`
EvalData interface{} `json:"EvalData"`
NewStateDate time.Time `json:"NewStateDate"`
PrevStateDate time.Time `json:"PrevStateDate"`
StateChanges int `json:"StateChanges"`
Created time.Time `json:"Created"`
Updated time.Time `json:"Updated"`
Settings struct {
Conditions []Condition `json:"conditions"`
ExecutionErrorState string `json:"executionErrorState"`
Frequency string `json:"frequency"`
Handler int `json:"handler"`
Name string `json:"name"`
NoDataState string `json:"noDataState"`
Notifications []interface{} `json:"notifications"`
} `json:"Settings"`
}
type Condition struct {
Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
} `json:"evaluator"`
Operator struct {
Type string `json:"type"`
} `json:"operator"`
Query struct {
DatasourceID int `json:"datasourceId"`
Model struct {
Hide bool `json:"hide"`
RefCount int `json:"refCount"`
RefID string `json:"refId"`
TextEditor bool `json:"textEditor"`
} `json:"model"`
Params []string `json:"params"`
} `json:"query"`
Reducer struct {
Params []interface{} `json:"params"`
Type string `json:"type"`
} `json:"reducer"`
Type string `json:"type"`
}
func (d Data) GetFirstCondition() (Condition, error) {
if len(d.Settings.Conditions) > 0 {
return d.Settings.Conditions[0], nil
}
return Condition{}, fmt.Errorf("no conditions found")
}
func (d Data) GetConditionByIndex(index uint) (Condition, error) {
if len(d.Settings.Conditions) == 0 {
return Condition{}, fmt.Errorf("no conditions found")
}
if int(index) > len(d.Settings.Conditions)-1 {
return Condition{}, fmt.Errorf("index out of bounds")
}
return d.Settings.Conditions[index], nil
}
var alertData = `{
"Id": 0,
"Version": 0,
"OrgId": 1,
"DashboardId": 61,
"PanelId": 84,
"Name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"Message": "",
"Severity": "",
"State": "",
"Handler": 1,
"Silenced": false,
"ExecutionError": "",
"Frequency": 10,
"EvalData": null,
"NewStateDate": "0001-01-01T00:00:00Z",
"PrevStateDate": "0001-01-01T00:00:00Z",
"StateChanges": 0,
"Created": "0001-01-01T00:00:00Z",
"Updated": "0001-01-01T00:00:00Z",
"Settings": {
"conditions": [
{
"evaluator": {
"params": [
10000
],
"type": "gt"
},
"operator": {
"type": "and"
},
"query": {
"datasourceId": 2,
"model": {
"hide": true,
"refCount": 0,
"refId": "C",
"textEditor": false
},
"params": [
"C",
"5m",
"now"
]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"executionErrorState": "keep_state",
"frequency": "10s",
"handler": 1,
"name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"noDataState": "keep_state",
"notifications": []
}
}`
func main() {
var res Data
err := json.Unmarshal([]byte(alertData), &res)
if err != nil {
log.Fatal(err)
}
fmt.Println(res.GetFirstCondition())
fmt.Println(res.GetConditionByIndex(0))
// should fail :-)
fmt.Println(res.GetConditionByIndex(1))
}

JSON to struct conversion

This is the struct I have:
type Resource struct {
Name string `json:"name"`
Ranges struct {
Range []struct {
Begin int `json:"begin"`
End int `json:"end"`
} `json:"range"`
} `json:"ranges,omitempty"`
Role string `json:"role,omitempty"`
Type string `json:"type"`
Scalar Scalar `json:"scalar,omitempty"`
}
I don't know how to make fields in the JSON not null. For example, struct Range like that:
{
"name": "cpus",
"ranges": {
"range": null
},
"type": "SCALAR",
"scalar": {
"value": 1
}
}, {
"name": "mem",
"ranges": {
"range": null
}
a way to do that is assign range as *string and then you should compare it with nil or not, if not nil convert it to string and marshall it again
That changes struct is resolved my problem:
type Resource struct {
Name string `json:"name"`
Ranges *Ranges `json:"ranges,omitempty"`
Role string `json:"role,omitempty"`
Type string `json:"type"`
Scalar *Scalar `json:"scalar,omitempty"`
}
assuming that you want to marshal the structs in your question and get a json output that looks like this:
{
"name": "cpus",
"ranges": {
"range": []
},
"type": "SCALAR",
"scalar": {
"value": 1
}
},
{
"name": "mem",
"ranges": {
"range": []
}
}
In golang slices [] are a reference type, that are backed by an array.
You can read up on the internals of slices here: https://blog.golang.org/go-slices-usage-and-internals
Basically the reason that you are getting null in the output is because you have not instantiated the slice, the slice is essentially a pointer, and that pointer is nil.
Create a new empty slice, like []Range{} and assign that to the field in Resource that is currently null in the json, and instead of a nil pointer, you will have an empty slice that will be marshalled as [] and not null.

Parsing JSON in GoLang into struct

So, I'm having some trouble parsing this data in golang:
{
"gateways": [
{
"token": "my_token_here",
"gateway_type": "test",
"description": null,
"payment_methods": [
"credit_card",
"sprel",
"third_party_token",
"bank_account",
"apple_pay"
],
"state": "retained",
"created_at": "2016-03-12T18:52:37Z",
"updated_at": "2016-03-12T18:52:37Z",
"name": "Spreedly Test",
"characteristics": [
"purchase",
"authorize",
"capture",
"credit",
"general_credit",
"void",
"verify",
"reference_purchase",
"purchase_via_preauthorization",
"offsite_purchase",
"offsite_authorize",
"3dsecure_purchase",
"3dsecure_authorize",
"store",
"remove",
"disburse",
"reference_authorization"
],
"credentials": [],
"gateway_specific_fields": [],
"redacted": false
}
]
}
When using this struct I can get it to output pretty easily.
type gateways struct {
Gateways []struct {
Characteristics []string `json:"characteristics"`
CreatedAt string `json:"created_at"`
Credentials []interface{} `json:"credentials"`
Description interface{} `json:"description"`
GatewaySpecificFields []interface{} `json:"gateway_specific_fields"`
GatewayType string `json:"gateway_type"`
Name string `json:"name"`
PaymentMethods []string `json:"payment_methods"`
Redacted bool `json:"redacted"`
State string `json:"state"`
Token string `json:"token"`
UpdatedAt string `json:"updated_at"`
} `json:"gateways"`
}
But as soon as I seperate the "Gateways []struct" into its own struct then it returns an empty array...
Full source.
type gateway struct {
Characteristics []string `json:"characteristics"`
CreatedAt string `json:"created_at"`
Credentials []interface{} `json:"credentials"`
Description interface{} `json:"description"`
GatewaySpecificFields []interface{} `json:"gateway_specific_fields"`
GatewayType string `json:"gateway_type"`
Name string `json:"name"`
PaymentMethods []string `json:"payment_methods"`
Redacted bool `json:"redacted"`
State string `json:"state"`
Token string `json:"token"`
UpdatedAt string `json:"updated_at"`
}
type gateways struct {
Gateways []gateway `json:"gateways"`
}
func ParseResponse() {
var parsed gateways
json.Unmarshal(json, &parsed)
}
There's a problem with your ParseResponse function, you're calling json.Unmarshal passing as first parameter json, that's a packge name: that's ambiguous.
As you can see, your code works well changing the ParseResponse function.
package main
import (
"encoding/json"
"fmt"
)
type gateway struct {
Characteristics []string `json:"characteristics"`
CreatedAt string `json:"created_at"`
Credentials []interface{} `json:"credentials"`
Description interface{} `json:"description"`
GatewaySpecificFields []interface{} `json:"gateway_specific_fields"`
GatewayType string `json:"gateway_type"`
Name string `json:"name"`
PaymentMethods []string `json:"payment_methods"`
Redacted bool `json:"redacted"`
State string `json:"state"`
Token string `json:"token"`
UpdatedAt string `json:"updated_at"`
}
type gateways struct {
Gateways []gateway `json:"gateways"`
}
func ParseResponse(js []byte) {
var parsed gateways
json.Unmarshal(js, &parsed)
fmt.Println(parsed)
}
func main() {
var js []byte = []byte(`{
"gateways": [
{
"token": "my_token_here",
"gateway_type": "test",
"description": null,
"payment_methods": [
"credit_card",
"sprel",
"third_party_token",
"bank_account",
"apple_pay"
],
"state": "retained",
"created_at": "2016-03-12T18:52:37Z",
"updated_at": "2016-03-12T18:52:37Z",
"name": "Spreedly Test",
"characteristics": [
"purchase",
"authorize",
"capture",
"credit",
"general_credit",
"void",
"verify",
"reference_purchase",
"purchase_via_preauthorization",
"offsite_purchase",
"offsite_authorize",
"3dsecure_purchase",
"3dsecure_authorize",
"store",
"remove",
"disburse",
"reference_authorization"
],
"credentials": [],
"gateway_specific_fields": [],
"redacted": false
}
]
}`)
/*
var parsed gateways
e := json.Unmarshal(js, &parsed)
if e != nil {
fmt.Println(e.Error())
} else {
fmt.Println(parsed)
}
*/
ParseResponse(js)
}
Outputs:
{[{[purchase authorize capture credit general_credit void verify reference_purchase purchase_via_preauthorization offsite_purchase offsite_authorize 3dsecure_purchase 3dsecure_authorize store remove disburse reference_authorization] 2016-03-12T18:52:37Z [] <nil> [] test Spreedly Test [credit_card sprel third_party_token bank_account apple_pay] false retained my_token_here 2016-03-12T18:52:37Z}]}
Take a look at http://play.golang.org/p/3xJHBmhuei - unmarshalling into your second definition of gateways completes successfully, so it must be something little that you're missing.
Check if the json.Unmarshal call returns an error.
P.S. It probably isn't the problem if you're unmarshalling successfully into the first version of gateways, but the JSON string that you've given above is missing a closing "}" bracket.

Encoding nested JSON in Go

I've had a lot of trouble finding an example of this. Most of the information on the internet is about decoding JSON.
I'd like to serialize some data into nested JSON, like for example:
{
"item": {
"title": "Items",
"properties": [
{
"num": 1,
"name": "Item 1"
},
{
"num": 2,
"name": "Item 2"
}
]
}
}
I know how to marshal data with a flat struct, but how do I put data into a struct that can be serialized with nesting?
http://play.golang.org/p/nDKmv1myTD
I found this tool that generates a struct from a JSON schema, but I don't understand how to get data into the sub structs.
http://mholt.github.io/json-to-go/
type Example struct {
Item struct {
Title string `json:"title"`
Properties []struct {
Num int `json:"num"`
Name string `json:"name"`
} `json:"properties"`
} `json:"item"`
}
This tool you found is nice, but I would not use it. It makes it difficult to initialize the structs.
Init example with your snippet: (http://play.golang.org/p/_Qw3Qp8XZh)
package main
import (
"encoding/json"
"fmt"
)
type Example struct {
Item struct {
Title string `json:"title"`
Properties []struct {
Num int `json:"num"`
Name string `json:"name"`
} `json:"properties"`
} `json:"item"`
}
func main() {
info := &Example{
Item: struct {
Title string `json:"title"`
Properties []struct {
Num int `json:"num"`
Name string `json:"name"`
} `json:"properties"`
}{
Title: "title",
Properties: []struct {
Num int `json:"num"`
Name string `json:"name"`
}{
{Num: 0, Name: "name0"},
{Num: 1, Name: "name1"},
},
},
}
b, err := json.Marshal(info)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(b))
}
Result (pretty printed):
{
"item": {
"title": "title",
"properties": [
{
"num": 0,
"name": "name0"
},
{
"num": 1,
"name": "name1"
}
]
}
}
I think it is better to use named struct vs anonymous nested ones.
Same example with named structs: http://play.golang.org/p/xm7BXxEGTC
package main
import (
"encoding/json"
"fmt"
)
type Example struct {
Item Item `json:"item"`
}
type Item struct {
Title string `json:"title"`
Properties []Property `json:"properties"`
}
type Property struct {
Num int `json:"num"`
Name string `json:"name"`
}
func main() {
info := &Example{
Item: Item{
Title: "title",
Properties: []Property{
{Num: 0, Name: "name0"},
{Num: 1, Name: "name1"},
},
},
}
b, err := json.Marshal(info)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(b))
}
It is the exact same thing, but I find it more clear and easy to use.