How to make complex json data with go struct? [closed] - json

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I want to build a json like
{
"data": {
"posts": [
{"title": "first", "body": "first body"},
{"title": "second", "body": "second body"}
],
"categories": [
{"name": "red"},
{"name": "white"}
]
}
}
It includes two parts of data: post and category. They are different structure. But want to send them together one time.
With this code
package main
import (
"net/http"
"github.com/labstack/echo/v4"
)
type Article struct {
Data struct {
Post []struct {
Title string `json:"title"`
Body string `json:"body"`
} `json:"posts"`
Category []struct {
Name string `json:"name"`
} `json:"categories"`
} `json:"data"`
}
func main() {
e := echo.New()
e.GET("/", func(c echo.Context) error {
article := &Article{}
data := article.Data
post := data.Post
post[0].Title = "first"
post[0].Body = "first body"
post[1].Title = "second"
post[1].Body = "second body"
category := data.Category
category[0].Name = "red"
category[1].Name = "white"
return c.JSON(http.StatusOK, article)
})
e.Logger.Fatal(e.Start(":8008"))
}
I tried to make a root struct Article, then define its children, then set data to tree one by one.
When run the server and call / endpoint, got error
echo: http: panic serving [::1]:49501: runtime error: index out of range [0] with length 0
goroutine 35 [running]:
net/http.(*conn).serve.func1(0xc00012afa0)
/Users/user/.goenv/versions/1.15.7/src/net/http/server.go:1801 +0x147
panic(0x1314380, 0xc00001a200)
/Users/user/.goenv/versions/1.15.7/src/runtime/panic.go:975 +0x47a
main.main.func1(0x139b160, 0xc000208000, 0x0, 0x0)
/Users/user/test/testecho/server.go:28 +0x13b
github.com/labstack/echo/v4.(*Echo).add.func1(0x139b160, 0xc000208000, 0x0, 0x0)
/Users/user/go/1.15.7/pkg/mod/github.com/labstack/echo/v4#v4.2.0/echo.go:536 +0x62
github.com/labstack/echo/v4.(*Echo).ServeHTTP(0xc000180000, 0x1397680, 0xc000200000, 0xc000072000)
/Users/user/go/1.15.7/pkg/mod/github.com/labstack/echo/v4#v4.2.0/echo.go:646 +0x187
Then tried this
e.GET("/", func(c echo.Context) error {
article := &Article{
Data: struct{
"Post": {
{
"Title": "first",
"Body": "first body",
},
{
"Title": "second",
"Body": "second body",
},
},
"Category": {
{
"Name": "red",
},
{
"Name": "white",
},
},
}
a, err := json.Marshal(article)
if err != nil {
fmt.Println(err)
}
return c.JSON(http.StatusOK, a)
})
But the Data: struct{ line got expected expression.

You can simplify the code required by declaring a named type for each object in the output:
type Post struct {
Title string `json:"title"`
Body string `json:"body"`
}
type Category struct {
Name string `json:"name"`
}
type Data struct {
Posts []Post `json:"posts"`
Categories []Category `json:"categories"`
}
type Article struct {
Data Data
}
With this change, it's easy to write the composite literal:
article := &Article{
Data: Data{Posts: []Post{
{
Title: "first",
Body: "first body",
},
{
Title: "second",
Body: "second body",
},
},
Categories: []Category{
{
Name: "red",
},
{
Name: "white",
},
},
},
}

Related

recursively iterate struct array in Go

I need to iterate over all the answer options from a json file:
{"questions": [
{"id": 1,
"question": "What is your marital status?",
"answer":[ {
"text": "Single",
"next_question": 2
},
{
"text": "Married",
"next_question": 3
}]
},
{"id":2,
"question": "Are you planning on getting married next year?",
"answer":[{
"text": "Yes",
"next_question": 3
},
{
"text": "No",
"next_question": 5
}]},
{"id":3,
"question": "How long have you been married?",
"answer": [{
"text": "Less than a year",
"next_question": 6
},
{
"text": "More than a year",
"next_question": 4
}]},
{"id":4,
"question":"Have you celebrated your one year anniversary?",
"answer": [{
"text": "Yes",
"next_question": 7
},
{
"text": "No",
"next_question": 8
}]}
]}
and write the paths and number, like:
{"paths": {"number": 3, "list": [
[{"What is your marital status?": "Single"},
{"Are you planning on getting married next year?": "Yes/No"}],
[{"What is your marital status?": "Married"},
{"How long have you been married?": "Less than a year"}],
[{"What is your marital status?": "Married"},
{"How long have you been married?": "More than a year"},
{"Have you celebrated your one year anniversary?": "Yes/No"}]
]}}
You can change JSON structure like you want, but the main thing is to display information about the number of all possible poll paths (paths.number), and all possible paths by a sequence of questions with answers (paths.list)
So, I parse JSON into this structs:
type (
Answer struct {
Text string `json:"text"`
NextQuestion int `json:"next_question"`
}
Question struct {
Id int `json:"id"`
Question string `json:"question"`
Answer []Answer `json:"answer"`
}
Input struct {
Questions []Question `json:"questions"`
}
)
And try to iterate:
func (input Input) Script(itterQuestion []Question, element Question) []Question {
itterQuestion = append(itterQuestion, element)
for i, item := range input.Questions {
if item.Id != itterQuestion[i].Id {
itterQuestion = append(itterQuestion, item)
} else {
return input.Script(itterQuestion, item)
}
}
return itterQuestion
}
But I don't understand how to correctly write recursive func and output struct for json.
Since you want to create multiple paths there is gotta be [][]Question.
Also you have to append result from recursive function instead just returning.
Here's working example:
func (input Input) Script (id int) (out [][]Question) {
for _, q := range input.Questions {
if q.Id == id {
added := false // avoid add last multiple times
for _, answer := range q.Answer {
paths := input.Script(answer.NextQuestion)
if len(paths) == 0 && !added {
// answer has no next question | question not found in input
out = append(out, []Question{q})
added = true
}
for _, path := range paths {
// prepend question to every path from recursive function
path = append([]Question{q}, path...)
out = append(out, path)
}
}
return out
}
}
return out
}

How to filter out duplicates while unmarshalling json into struct?

I have this json which I am trying to unmarshall to my struct.
{
"clientMetrics": [
{
"clientId": 951231,
"customerData": {
"Process": [
"ABC"
],
"Mat": [
"KKK"
]
},
"legCustomer": [
8773
]
},
{
"clientId": 1234,
"legCustomer": [
8789
]
},
{
"clientId": 3435,
"otherIds": [
4,
32,
19
],
"legCustomer": [
10005
]
},
{
"clientId": 9981,
"catId": 8,
"legCustomer": [
13769
]
},
{
"clientId": 12124,
"otherIds": [
33,
29
],
"legCustomer": [
12815
]
},
{
"clientId": 8712,
"customerData": {
"Process": [
"College"
]
},
"legCustomer": [
951
]
},
{
"clientId": 23214,
"legCustomer": [
12724,
12727
]
},
{
"clientId": 119812,
"catId": 8,
"legCustomer": [
14519
]
},
{
"clientId": 22315,
"otherIds": [
32
],
"legCustomer": [
12725,
13993
]
},
{
"clientId": 765121,
"catId": 8,
"legCustomer": [
14523
]
}
]
}
I used this tool to generate struct as shown below -
type AutoGenerated struct {
ClientMetrics []ClientMetrics `json:"clientMetrics"`
}
type CustomerData struct {
Process []string `json:"Process"`
Mat []string `json:"Mat"`
}
type ClientMetrics struct {
ClientID int `json:"clientId"`
CustomerData CustomerData `json:"customerData,omitempty"`
LegCustomer []int `json:"legCustomer"`
OtherIds []int `json:"otherIds,omitempty"`
CatID int `json:"catId,omitempty"`
CustomerData CustomerData `json:"customerData,omitempty"`
}
Now my confusion is, I have lot of string or int array so how can I filter out duplicates? I believe there is no set data type in golang so how can I achieve same thing here? Basically when I unmarshall json into my struct I need to make sure there are no duplicates present at all. Is there any way to achieve this? If yes, can someone provide an example how to achieve this for my above json and how should I design my struct for that.
Update
So basically just use like this and change my struct definitions and that's all? Internally it will call UnmarshalJSON and take care of duplicates? I will pass json string and structure to JSONStringToStructure method.
func JSONStringToStructure(jsonString string, structure interface{}) error {
jsonBytes := []byte(jsonString)
return json.Unmarshal(jsonBytes, structure)
}
type UniqueStrings []string
func (u *UniqueStrings) UnmarshalJSON(in []byte) error {
var arr []string
if err := json.Unmarshal(in, arr); err != nil {
return err
}
*u = UniqueStrings(dedupStr(arr))
return nil
}
func dedupStr(in []string) []string {
seen:=make(map[string]struct{})
w:=0
for i:=range in {
if _,s:=seen[in[i]]; !s {
seen[in[i]]=struct{}{}
in[w]=in[i]
w++
}
}
return in[:w]
}
Ideally, you should post-process these arrays to remove duplicates. However, you can achieve this during unmarshaling using a custom type with an unmarshaler:
type UniqueStrings []string
func (u *UniqueStrings) UnmarshalJSON(in []byte) error {
var arr []string
if err:=json.Unmarshal(in,arr); err!=nil {
return err
}
*u=UniqueStrings(dedupStr(arr))
return nil
}
where
func dedupStr(in []string) []string {
seen:=make(map[string]struct{})
w:=0
for i:=range in {
if _,s:=seen[in[i]]; !s {
seen[in[i]]=struct{}{}
in[w]=in[i]
w++
}
}
return in[:w]
}
You may use a similar approach for []ints.
You use the custom types in your structs:
type CustomerData struct {
Process UniqueStrings `json:"Process"`
Mat UniqueStrings `json:"Mat"`
}

Creating dynamic json

I need to create dynamic json i.e whose key value varies, below mentioned is the json
[{"email":"xxx#gmail.com","location":{"set":"Redmond"},"fname":{"set":"xxxxx"},"clicked_time":{"set":"zz"},"domain":{"add":"ttt"}},{"email":"zzz#gmail.com","location":{"set":"Greece"},"fname":{"set":"zzzzz"},"clicked_time":{"set":"zzz"},"domain":{"add":"zxxxx"}}]
I tried using below code:
rows := []map[string]string{}
if i > 0 {
row := make(map[string]string)
for j:=0;j<len(record);j++ {
key := header[j]
value := record[j]
row[key] = value
}
rows = append(rows, row)
}
How may I add set to location and add to domain to create a nested structure as map can have only one type string or nested structure?
Perhaps I have missed the point a little here, but I am not seeing why this is so dynamic in a way that can't be handled by a struct and the json unmarshal method.
Please see the following for an example
https://play.golang.org/p/8nrO36HQGhy
package main
import (
"encoding/json"
"fmt"
)
type (
Details struct {
Email string `json:"email"`
Location Entry `json:"location"`
FName Entry `json:"fname"`
ClickedTime Entry `json:"clicked_time"`
Domain Entry `json:"domain"`
}
Entry struct {
Set string `json:"set"`
Add string `json:"add"`
}
)
func main() {
d := []byte(`[{
"email": "xxx#gmail.com",
"location": {
"set": "Redmond"
},
"fname": {
"set": "xxxxx"
},
"clicked_time": {
"set": "zz"
},
"domain": {
"add": "ttt"
}
}, {
"email": "zzz#gmail.com",
"location": {
"set": "Greece"
},
"fname": {
"set": "zzzzz"
},
"clicked_time": {
"set": "zzz"
},
"domain": {
"add": "zxxxx"
}
}]`)
x := []Details{}
_ = json.Unmarshal(d, &x)
fmt.Printf("%+v\n", x)
}

Why can this data not be properly unmarshalled into my object model?

I have a (non)working example here: https://play.golang.org/p/qaYhKvJ65J3
I'm not sure why the following data:
alertData := `{
"Id": 0,
"Version": 0,
"OrgId": 1,
"DashboardId": 61,
"PanelId": 84,
"Name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"Message": "",
"Severity": "",
"State": "",
"Handler": 1,
"Silenced": false,
"ExecutionError": "",
"Frequency": 10,
"EvalData": null,
"NewStateDate": "0001-01-01T00:00:00Z",
"PrevStateDate": "0001-01-01T00:00:00Z",
"StateChanges": 0,
"Created": "0001-01-01T00:00:00Z",
"Updated": "0001-01-01T00:00:00Z",
"Settings": {
"conditions": [
{
"evaluator": {
"params": [
10000
],
"type": "gt"
},
"operator": {
"type": "and"
},
"query": {
"datasourceId": 2,
"model": {
"hide": true,
"refCount": 0,
"refId": "C",
"textEditor": false
},
"params": [
"C",
"5m",
"now"
]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"executionErrorState": "keep_state",
"frequency": "10s",
"handler": 1,
"name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"noDataState": "keep_state",
"notifications": []
}
}`
Can't be unmarshalled into the following object model:
type Condition struct {
Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
} `json:"evaluator"`
Operator struct {
Type string `json:"type"`
} `json:"operator"`
Query struct {
Params []string `json:"params"`
} `json:"query"`
Reducer struct {
Params []interface{} `json:"params"`
Type string `json:"type"`
} `json:"reducer"`
Type string `json:"type"`
}
When I do the following:
condition := Condition{}
err := json.Unmarshal([]byte(alertData), &condition)
if err != nil {
panic(err)
}
fmt.Printf("\n\n json object:::: %+v", condition)
I just get: json object:::: {Evaluator:{Params:[] Type:} Operator:{Type:} Query:{Params:[]} Reducer:{Params:[] Type:} Type:}
Ideally I'd be able to parse it into something like type Conditions []struct{ } but I'm not sure if you can define models as lists?
It looks like you are trying to access the "conditions" property nested under the root "Settings" property. As such, you need to define that root-level type and enough fields to tell the unmarshaler how to find your target property. As such, you would just need to create a new "AlertData" type with the necessary "Settings/conditions" fields.
For example (Go Playground):
type AlertData struct {
Settings struct {
Conditions []Condition `json:"conditions"`
}
}
func main() {
alert := AlertData{}
err := json.Unmarshal([]byte(alertData), &alert)
if err != nil {
panic(err)
}
fmt.Printf("OK: conditions=%#v\n", alert.Settings.Conditions)
// OK: conditions=[]main.Condition{main.Condition{Evaluator:struct { Params []int "json:\"params\""; Type string "json:\"type\"" }{Params:[]int{10000}, Type:"gt"}, Operator:struct { Type string "json:\"type\"" }{Type:"and"}, Query:struct { Params []string "json:\"params\"" }{Params:[]string{"C", "5m", "now"}}, Reducer:struct { Params []interface {} "json:\"params\""; Type string "json:\"type\"" }{Params:[]interface {}{}, Type:"avg"}, Type:"query"}}
}
Note that the printed listing includes so much type information because the "Condition" type uses anonymous structs as field types. If you were to extract them into named structs it will be easier to work with the data, e.g.:
type Condition struct {
Evaluator Evaluator `json:"evaluator"`
Operator Operator `json:"operator"`
// ...
}
type Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
}
type Operator struct {
Type string `json:"type"`
}
//...
// OK: conditions=[]main.Condition{
// main.Condition{
// Evaluator:main.Evaluator{Params:[]int{10000}, Type:"gt"},
// Operator:main.Operator{Type:"and"},
// Query:main.Query{Params:[]string{"C", "5m", "now"}},
// Reducer:main.Reducer{Params:[]interface {}{}, Type:"avg"},
// Type:"query",
// },
// }
Go Playground example here...
Maerics explanation is correct, here is an alternative approach which wraps access around struct methods, the data structure is also fully defined. If you're new to Go it's good to get handle on creating the data structures yourself, but here is a handy utility for helping create structs from valid JSON
https://mholt.github.io/json-to-go/
package main
import (
"encoding/json"
"fmt"
"log"
"time"
)
type Data struct {
ID int `json:"Id"`
Version int `json:"Version"`
OrgID int `json:"OrgId"`
DashboardID int `json:"DashboardId"`
PanelID int `json:"PanelId"`
Name string `json:"Name"`
Message string `json:"Message"`
Severity string `json:"Severity"`
State string `json:"State"`
Handler int `json:"Handler"`
Silenced bool `json:"Silenced"`
ExecutionError string `json:"ExecutionError"`
Frequency int `json:"Frequency"`
EvalData interface{} `json:"EvalData"`
NewStateDate time.Time `json:"NewStateDate"`
PrevStateDate time.Time `json:"PrevStateDate"`
StateChanges int `json:"StateChanges"`
Created time.Time `json:"Created"`
Updated time.Time `json:"Updated"`
Settings struct {
Conditions []Condition `json:"conditions"`
ExecutionErrorState string `json:"executionErrorState"`
Frequency string `json:"frequency"`
Handler int `json:"handler"`
Name string `json:"name"`
NoDataState string `json:"noDataState"`
Notifications []interface{} `json:"notifications"`
} `json:"Settings"`
}
type Condition struct {
Evaluator struct {
Params []int `json:"params"`
Type string `json:"type"`
} `json:"evaluator"`
Operator struct {
Type string `json:"type"`
} `json:"operator"`
Query struct {
DatasourceID int `json:"datasourceId"`
Model struct {
Hide bool `json:"hide"`
RefCount int `json:"refCount"`
RefID string `json:"refId"`
TextEditor bool `json:"textEditor"`
} `json:"model"`
Params []string `json:"params"`
} `json:"query"`
Reducer struct {
Params []interface{} `json:"params"`
Type string `json:"type"`
} `json:"reducer"`
Type string `json:"type"`
}
func (d Data) GetFirstCondition() (Condition, error) {
if len(d.Settings.Conditions) > 0 {
return d.Settings.Conditions[0], nil
}
return Condition{}, fmt.Errorf("no conditions found")
}
func (d Data) GetConditionByIndex(index uint) (Condition, error) {
if len(d.Settings.Conditions) == 0 {
return Condition{}, fmt.Errorf("no conditions found")
}
if int(index) > len(d.Settings.Conditions)-1 {
return Condition{}, fmt.Errorf("index out of bounds")
}
return d.Settings.Conditions[index], nil
}
var alertData = `{
"Id": 0,
"Version": 0,
"OrgId": 1,
"DashboardId": 61,
"PanelId": 84,
"Name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"Message": "",
"Severity": "",
"State": "",
"Handler": 1,
"Silenced": false,
"ExecutionError": "",
"Frequency": 10,
"EvalData": null,
"NewStateDate": "0001-01-01T00:00:00Z",
"PrevStateDate": "0001-01-01T00:00:00Z",
"StateChanges": 0,
"Created": "0001-01-01T00:00:00Z",
"Updated": "0001-01-01T00:00:00Z",
"Settings": {
"conditions": [
{
"evaluator": {
"params": [
10000
],
"type": "gt"
},
"operator": {
"type": "and"
},
"query": {
"datasourceId": 2,
"model": {
"hide": true,
"refCount": 0,
"refId": "C",
"textEditor": false
},
"params": [
"C",
"5m",
"now"
]
},
"reducer": {
"params": [],
"type": "avg"
},
"type": "query"
}
],
"executionErrorState": "keep_state",
"frequency": "10s",
"handler": 1,
"name": "{qa-dev}{stats-pipeline} Topology Message Age (aggregator) alert",
"noDataState": "keep_state",
"notifications": []
}
}`
func main() {
var res Data
err := json.Unmarshal([]byte(alertData), &res)
if err != nil {
log.Fatal(err)
}
fmt.Println(res.GetFirstCondition())
fmt.Println(res.GetConditionByIndex(0))
// should fail :-)
fmt.Println(res.GetConditionByIndex(1))
}

How to decode json into structs

I'm trying to decode some json in Go but some fields don't get decoded.
See the code running in browser here:
What am I doing wrong?
I need only the MX records so I didn't define the other fields. As I understand from the godoc you don't need to define the fields you don't use/need.
// You can edit this code!
// Click here and start typing.
package main
import "fmt"
import "encoding/json"
func main() {
body := `
{"response": {
"status": "SUCCESS",
"data": {
"mxRecords": [
{
"value": "us2.mx3.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
},
{
"value": "us2.mx1.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
},
{
"value": "us2.mx2.mailhostbox.com.",
"ttl": 1,
"priority": 100,
"hostName": "#"
}
],
"cnameRecords": [
{
"aliasHost": "pop.a.co.uk.",
"canonicalHost": "us2.pop.mailhostbox.com."
},
{
"aliasHost": "webmail.a.co.uk.",
"canonicalHost": "us2.webmail.mailhostbox.com."
},
{
"aliasHost": "smtp.a.co.uk.",
"canonicalHost": "us2.smtp.mailhostbox.com."
},
{
"aliasHost": "imap.a.co.uk.",
"canonicalHost": "us2.imap.mailhostbox.com."
}
],
"dkimTxtRecord": {
"domainname": "20a19._domainkey.a.co.uk",
"value": "\"v=DKIM1; g=*; k=rsa; p=DkfbhO8Oyy0E1WyUWwIDAQAB\"",
"ttl": 1
},
"spfTxtRecord": {
"domainname": "a.co.uk",
"value": "\"v=spf1 redirect=_spf.mailhostbox.com\"",
"ttl": 1
},
"loginUrl": "us2.cp.mailhostbox.com"
}
}}`
type MxRecords struct {
value string
ttl int
priority int
hostName string
}
type Data struct {
mxRecords []MxRecords
}
type Response struct {
Status string `json:"status"`
Data Data `json:"data"`
}
type apiR struct {
Response Response
}
var r apiR
err := json.Unmarshal([]byte(body), &r)
if err != nil {
fmt.Printf("err was %v", err)
}
fmt.Printf("decoded is %v", r)
}
As per the go documentaiton about json.Unmarshal, you can only decode toward exported fields, the main reason being that external packages (such as encoding/json) cannot acces unexported fields.
If your json doesn't follow the go convention for names, you can use the json tag in your fields to change the matching between json key and struct field.
Exemple:
package main
import (
"fmt"
"encoding/json"
)
type T struct {
Foo string `json:"foo"`
priv string `json:"priv"`
}
func main() {
text := []byte(`{"foo":"bar", "priv":"nothing"}`)
var t T
err := json.Unmarshal(text, &t)
if err != nil {
panic(err)
}
fmt.Println(t.Foo) // prints "bar"
fmt.Println(t.priv) // prints "", priv is not exported
}
You must Uppercase struct fields:
type MxRecords struct {
Value string `json:"value"`
Ttl int `json:"ttl"`
Priority int `json:"priority"`
HostName string `json:"hostName"`
}
type Data struct {
MxRecords []MxRecords `json:"mxRecords"`
}
http://play.golang.org/p/EEyiISdoaE
The encoding/json package can only decode into exported struct fields. Your Data.mxRecords member is not exported, so it is ignored when decoding. If you rename it to use a capital letter, the JSON package will notice it.
You will need to do the same thing for all the members of your MxRecords type.