recursively iterate struct array in Go - json

I need to iterate over all the answer options from a json file:
{"questions": [
{"id": 1,
"question": "What is your marital status?",
"answer":[ {
"text": "Single",
"next_question": 2
},
{
"text": "Married",
"next_question": 3
}]
},
{"id":2,
"question": "Are you planning on getting married next year?",
"answer":[{
"text": "Yes",
"next_question": 3
},
{
"text": "No",
"next_question": 5
}]},
{"id":3,
"question": "How long have you been married?",
"answer": [{
"text": "Less than a year",
"next_question": 6
},
{
"text": "More than a year",
"next_question": 4
}]},
{"id":4,
"question":"Have you celebrated your one year anniversary?",
"answer": [{
"text": "Yes",
"next_question": 7
},
{
"text": "No",
"next_question": 8
}]}
]}
and write the paths and number, like:
{"paths": {"number": 3, "list": [
[{"What is your marital status?": "Single"},
{"Are you planning on getting married next year?": "Yes/No"}],
[{"What is your marital status?": "Married"},
{"How long have you been married?": "Less than a year"}],
[{"What is your marital status?": "Married"},
{"How long have you been married?": "More than a year"},
{"Have you celebrated your one year anniversary?": "Yes/No"}]
]}}
You can change JSON structure like you want, but the main thing is to display information about the number of all possible poll paths (paths.number), and all possible paths by a sequence of questions with answers (paths.list)
So, I parse JSON into this structs:
type (
Answer struct {
Text string `json:"text"`
NextQuestion int `json:"next_question"`
}
Question struct {
Id int `json:"id"`
Question string `json:"question"`
Answer []Answer `json:"answer"`
}
Input struct {
Questions []Question `json:"questions"`
}
)
And try to iterate:
func (input Input) Script(itterQuestion []Question, element Question) []Question {
itterQuestion = append(itterQuestion, element)
for i, item := range input.Questions {
if item.Id != itterQuestion[i].Id {
itterQuestion = append(itterQuestion, item)
} else {
return input.Script(itterQuestion, item)
}
}
return itterQuestion
}
But I don't understand how to correctly write recursive func and output struct for json.

Since you want to create multiple paths there is gotta be [][]Question.
Also you have to append result from recursive function instead just returning.
Here's working example:
func (input Input) Script (id int) (out [][]Question) {
for _, q := range input.Questions {
if q.Id == id {
added := false // avoid add last multiple times
for _, answer := range q.Answer {
paths := input.Script(answer.NextQuestion)
if len(paths) == 0 && !added {
// answer has no next question | question not found in input
out = append(out, []Question{q})
added = true
}
for _, path := range paths {
// prepend question to every path from recursive function
path = append([]Question{q}, path...)
out = append(out, path)
}
}
return out
}
}
return out
}

Related

How to make complex json data with go struct? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I want to build a json like
{
"data": {
"posts": [
{"title": "first", "body": "first body"},
{"title": "second", "body": "second body"}
],
"categories": [
{"name": "red"},
{"name": "white"}
]
}
}
It includes two parts of data: post and category. They are different structure. But want to send them together one time.
With this code
package main
import (
"net/http"
"github.com/labstack/echo/v4"
)
type Article struct {
Data struct {
Post []struct {
Title string `json:"title"`
Body string `json:"body"`
} `json:"posts"`
Category []struct {
Name string `json:"name"`
} `json:"categories"`
} `json:"data"`
}
func main() {
e := echo.New()
e.GET("/", func(c echo.Context) error {
article := &Article{}
data := article.Data
post := data.Post
post[0].Title = "first"
post[0].Body = "first body"
post[1].Title = "second"
post[1].Body = "second body"
category := data.Category
category[0].Name = "red"
category[1].Name = "white"
return c.JSON(http.StatusOK, article)
})
e.Logger.Fatal(e.Start(":8008"))
}
I tried to make a root struct Article, then define its children, then set data to tree one by one.
When run the server and call / endpoint, got error
echo: http: panic serving [::1]:49501: runtime error: index out of range [0] with length 0
goroutine 35 [running]:
net/http.(*conn).serve.func1(0xc00012afa0)
/Users/user/.goenv/versions/1.15.7/src/net/http/server.go:1801 +0x147
panic(0x1314380, 0xc00001a200)
/Users/user/.goenv/versions/1.15.7/src/runtime/panic.go:975 +0x47a
main.main.func1(0x139b160, 0xc000208000, 0x0, 0x0)
/Users/user/test/testecho/server.go:28 +0x13b
github.com/labstack/echo/v4.(*Echo).add.func1(0x139b160, 0xc000208000, 0x0, 0x0)
/Users/user/go/1.15.7/pkg/mod/github.com/labstack/echo/v4#v4.2.0/echo.go:536 +0x62
github.com/labstack/echo/v4.(*Echo).ServeHTTP(0xc000180000, 0x1397680, 0xc000200000, 0xc000072000)
/Users/user/go/1.15.7/pkg/mod/github.com/labstack/echo/v4#v4.2.0/echo.go:646 +0x187
Then tried this
e.GET("/", func(c echo.Context) error {
article := &Article{
Data: struct{
"Post": {
{
"Title": "first",
"Body": "first body",
},
{
"Title": "second",
"Body": "second body",
},
},
"Category": {
{
"Name": "red",
},
{
"Name": "white",
},
},
}
a, err := json.Marshal(article)
if err != nil {
fmt.Println(err)
}
return c.JSON(http.StatusOK, a)
})
But the Data: struct{ line got expected expression.
You can simplify the code required by declaring a named type for each object in the output:
type Post struct {
Title string `json:"title"`
Body string `json:"body"`
}
type Category struct {
Name string `json:"name"`
}
type Data struct {
Posts []Post `json:"posts"`
Categories []Category `json:"categories"`
}
type Article struct {
Data Data
}
With this change, it's easy to write the composite literal:
article := &Article{
Data: Data{Posts: []Post{
{
Title: "first",
Body: "first body",
},
{
Title: "second",
Body: "second body",
},
},
Categories: []Category{
{
Name: "red",
},
{
Name: "white",
},
},
},
}

Protobuf custom options not showing in JSON made by protojson library

I'm trying to extract Protobuf custom options from a FileDescriptorSet generated by the protoc compiler. I'm unable to do so using protoreflect. So, I tried to do so using the protojson library.
PS : Importing the Go-generated code is not an option for my use case.
Here's the Protobuf Message I'm testing with :
syntax = "proto3";
option go_package = "./protoze";
import "google/protobuf/descriptor.proto";
extend google.protobuf.FieldOptions {
string Meta = 50000;
}
extend google.protobuf.FileOptions {
string Food = 50001;
}
option (Food) = "cheese";
message X {
int64 num = 1;
}
message P {
string Fname = 1 [json_name = "FNAME"];
string Lname = 2 [json_name = "0123", (Meta) = "Yo"];
string Designation = 3;
repeated string Email = 4;
string UserID = 5;
string EmpID = 6;
repeated X z = 7;
}
// protoc --go_out=. filename.proto
Here's how far I got :
package main
import (
"fmt"
"io/ioutil"
"os/exec"
"google.golang.org/protobuf/encoding/protojson"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/descriptorpb"
)
func main() {
exec.Command("protoc", "-oBinaryFile", "1.proto").Run()
Fset := descriptorpb.FileDescriptorSet{}
byts, _ := ioutil.ReadFile("File")
proto.Unmarshal(byts, &Fset)
byts, _ = protojson.Marshal(Fset.File[0])
fmt.Println(string(byts))
}
And here's the output JSON
{
"name": "1.proto",
"dependency": [
"google/protobuf/descriptor.proto"
],
"messageType": [
{
"name": "X",
"field": [
{
"name": "num",
"number": 1,
"label": "LABEL_OPTIONAL",
"type": "TYPE_INT64",
"jsonName": "num"
}
]
},
{
"name": "P",
"field": [
{
"name": "Fname",
"number": 1,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "FNAME"
},
{
"name": "Lname",
"number": 2,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "0123",
"options": {}
},
{
"name": "Designation",
"number": 3,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "Designation"
},
{
"name": "Email",
"number": 4,
"label": "LABEL_REPEATED",
"type": "TYPE_STRING",
"jsonName": "Email"
},
{
"name": "UserID",
"number": 5,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "UserID"
},
{
"name": "EmpID",
"number": 6,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"jsonName": "EmpID"
},
{
"name": "z",
"number": 7,
"label": "LABEL_REPEATED",
"type": "TYPE_MESSAGE",
"typeName": ".X",
"jsonName": "z"
}
]
}
],
"extension": [
{
"name": "Meta",
"number": 50000,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"extendee": ".google.protobuf.FieldOptions",
"jsonName": "Meta"
},
{
"name": "Food",
"number": 50001,
"label": "LABEL_OPTIONAL",
"type": "TYPE_STRING",
"extendee": ".google.protobuf.FileOptions",
"jsonName": "Food"
}
],
"options": {
"goPackage": "./protoze"
},
"syntax": "proto3"
}
So, data about my custom options showed up in the extensions. But what I really wanted was the value of those Custom Options in the "options" as well. (Which in my case was (Food) = "Cheese" and I want Cheese)
Can someone tell me how I can extract my custom options from the FileDescriptorSet using Protoreflect or by using Protojson.
I tried a lot to try and extract it using Protoreflect but failed !
Although not specifically an answer to how to get the custom options in a generated JSON, I believe I have an answer to what sounds like your underlying question: how to access the custom options without loading the generated Go code. This is thanks to dsnet's answer to my question on the golang issues board. Needless to say all the credit for this tricky solution goes to him. The punchline is to Marshal and then Unmarshal the options using a runtime-populated protoregistry.Types that actually knows about the custom options.
I made a complete demonstration of this approach working in this repo, and the key section (all the guts of which come from dsnet's example) is here:
func main() {
protogen.Options{
}.Run(func(gen *protogen.Plugin) error {
gen.SupportedFeatures = uint64(pluginpb.CodeGeneratorResponse_FEATURE_PROTO3_OPTIONAL)
// The type information for all extensions is in the source files,
// so we need to extract them into a dynamically created protoregistry.Types.
extTypes := new(protoregistry.Types)
for _, file := range gen.Files {
if err := registerAllExtensions(extTypes, file.Desc); err != nil {
panic(err)
}
}
// run through the files again, extracting and printing the Message options
for _, sourceFile := range gen.Files {
if !sourceFile.Generate {
continue
}
// setup output file
outputfile := gen.NewGeneratedFile("./out.txt", sourceFile.GoImportPath)
for _, message := range sourceFile.Messages {
outputfile.P(fmt.Sprintf("\nMessage %s:", message.Desc.Name()))
// The MessageOptions as provided by protoc does not know about
// dynamically created extensions, so they are left as unknown fields.
// We round-trip marshal and unmarshal the options with
// a dynamically created resolver that does know about extensions at runtime.
options := message.Desc.Options().(*descriptorpb.MessageOptions)
b, err := proto.Marshal(options)
if err != nil {
panic(err)
}
options.Reset()
err = proto.UnmarshalOptions{Resolver: extTypes}.Unmarshal(b, options)
if err != nil {
panic(err)
}
// Use protobuf reflection to iterate over all the extension fields,
// looking for the ones that we are interested in.
options.ProtoReflect().Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool {
if !fd.IsExtension() {
return true
}
outputfile.P(fmt.Sprintf("Value of option %s is %s",fd.Name(), v.String()))
// Make use of fd and v based on their reflective properties.
return true
})
}
}
return nil
})
}
// Recursively register all extensions into the provided protoregistry.Types,
// starting with the protoreflect.FileDescriptor and recursing into its MessageDescriptors,
// their nested MessageDescriptors, and so on.
//
// This leverages the fact that both protoreflect.FileDescriptor and protoreflect.MessageDescriptor
// have identical Messages() and Extensions() functions in order to recurse through a single function
func registerAllExtensions(extTypes *protoregistry.Types, descs interface {
Messages() protoreflect.MessageDescriptors
Extensions() protoreflect.ExtensionDescriptors
}) error {
mds := descs.Messages()
for i := 0; i < mds.Len(); i++ {
registerAllExtensions(extTypes, mds.Get(i))
}
xds := descs.Extensions()
for i := 0; i < xds.Len(); i++ {
if err := extTypes.RegisterExtension(dynamicpb.NewExtensionType(xds.Get(i))); err != nil {
return err
}
}
return nil
}

Creating dynamic json

I need to create dynamic json i.e whose key value varies, below mentioned is the json
[{"email":"xxx#gmail.com","location":{"set":"Redmond"},"fname":{"set":"xxxxx"},"clicked_time":{"set":"zz"},"domain":{"add":"ttt"}},{"email":"zzz#gmail.com","location":{"set":"Greece"},"fname":{"set":"zzzzz"},"clicked_time":{"set":"zzz"},"domain":{"add":"zxxxx"}}]
I tried using below code:
rows := []map[string]string{}
if i > 0 {
row := make(map[string]string)
for j:=0;j<len(record);j++ {
key := header[j]
value := record[j]
row[key] = value
}
rows = append(rows, row)
}
How may I add set to location and add to domain to create a nested structure as map can have only one type string or nested structure?
Perhaps I have missed the point a little here, but I am not seeing why this is so dynamic in a way that can't be handled by a struct and the json unmarshal method.
Please see the following for an example
https://play.golang.org/p/8nrO36HQGhy
package main
import (
"encoding/json"
"fmt"
)
type (
Details struct {
Email string `json:"email"`
Location Entry `json:"location"`
FName Entry `json:"fname"`
ClickedTime Entry `json:"clicked_time"`
Domain Entry `json:"domain"`
}
Entry struct {
Set string `json:"set"`
Add string `json:"add"`
}
)
func main() {
d := []byte(`[{
"email": "xxx#gmail.com",
"location": {
"set": "Redmond"
},
"fname": {
"set": "xxxxx"
},
"clicked_time": {
"set": "zz"
},
"domain": {
"add": "ttt"
}
}, {
"email": "zzz#gmail.com",
"location": {
"set": "Greece"
},
"fname": {
"set": "zzzzz"
},
"clicked_time": {
"set": "zzz"
},
"domain": {
"add": "zxxxx"
}
}]`)
x := []Details{}
_ = json.Unmarshal(d, &x)
fmt.Printf("%+v\n", x)
}

Unmarshalling complex JSON into struct and accessing data (slice of slice of slices)

I've been bashing my head about this for a while now. I have a JSON file that must be in the following format that I need to iterate through and use IF statements on in Go:
[
[
{
"configName": "customer"
},
{
"config": [
{
"emailSubject": "New customer added"
},
{
"text": "Hi test 2"
},
{
"text": "added 2"
}
]
}
]
[
{
"configName": "customerAndUser"
},
{
"config": [
{
"emailSubject": "New customer added"
},
{
"text": "Hi, test 1"
},
{
"text": "added 1"
}
]
}
]
]
And I want to put it into a struct, like this:
type Config [][]struct {
configName string `json: configName`
config []struct {
Text string `json: text`
EmailSubject string `json: emailSubject`
} `json: config`
}
I can unmarshal the data fine, like this:
configData, err := ioutil.ReadFile("testing-config.json")
if err != nil {
fmt.Println(err)
}
var configDataUnmarshalled Config
json.Unmarshal([]byte(configData), &configDataUnmarshalled)
And then the data prints, sort of okay, but here is where things get a little strange: the print statement is returning blanks for the items that I do not specify to print. Here is a sample of what's printed when I print the unmarshalled data:
Print output from unmarshalled data:
[[{customer []} { [{ New customer added} {hi test 2 } {added 2 }]}] [{customerAndUser []} { [{ New customer added} {hi test 1 } {added 1 }]}]]
But then I can't seem to use IF statements or loop over the elements in the config key!
IF statement being ignored in the for loop (see output below code)
for _, configs := range configDataUnmarshalled {
for _, configurations := range configs {
fmt.Println("These are the top level elements in my struct: ", configurations.ConfigName)
if configurations.ConfigName == "customerAndUser" {
for _, config := range configurations.Config {
fmt.Println(config)
}
}
}
}
This is what is printed:
These are the top level elements in my struct: customer
These are the top level elements in my struct:
These are the top level elements in my struct: customerAndUser
These are the top level elements in my struct:
From the FOR loop you can see that I want to access the data when a config is of a certain name, in this case "customerAndUser"
Here the IF statement is being ignored completely
I have two things I want to understand/solve:
How can I access the data following an IF statement
Why is the program printing blanks?
Desired output would be printing out the emailSubject, and the two Text element's data to the console for the config with name customerAndUser
What should be printed:
New customer added
hi test 1
added 1
Thanks for your help
json config is very smell. The struct contain configName and config is two separately structs in a slice. configName have value so config is empty and backwards. This will work when json like this.
{
"configName": "customerAndUser",
"config": [
{
"emailSubject": "New customer added"
},
{
"text": "Hi, test 1"
},
{
"text": "added 1"
}
]
}
So if you can't change json config format. This is solution
endUser := false
for _, configs := range configDataUnmarshalled {
endUser = false
for _, configurations := range configs {
if configurations.ConfigName == "customerAndUser" {
endUser = true
continue
}
if !endUser || len(configurations.Config) == 0 {
continue
}
for _, config := range configurations.Config {
fmt.Println(config)
}
}
}

What is the most efficient way of "filtering" out JSON objects from a key-value pair?

I am reading in a .json file. It's an array of objects in valid JSON format, example:
[
{
"Id": 13,
"Location": "Australia",
"Content": "Another string"
},
{
"Id": 145,
"Location": "England",
"Content": "SomeString"
},
{
"Id": 12,
"Location": "England",
"Content": "SomeString"
},
{
"Id": 12331,
"Location": "Sweden",
"Content": "SomeString"
},
{
"Id": 213123,
"Location": "England",
"Content": "SomeString"
}
]
I want to filter these objects out - say, removing anything where "Location"doesn't equal "England".
What I've tried so far is creating a custom UnmarshalJSON function. It does unmarshal it, but the objects it produces are empty - and as many as the input.
Sample code:
type languageStruct struct {
ID int `json:"Id"`
Location string `json:"Location"`
Content string `json:"Content"`
}
func filterJSON(file []byte) ([]byte, error) {
var x []*languageStruct
err := json.Unmarshal(file, &x)
check(err)
return json.MarshalIndent(x, "", " ")
}
func (s *languageStruct) UnmarshalJSON(p []byte) error {
var result struct {
ID int `json:"Id"`
Location string `json:"Location"`
Content string `json:"Content"`
}
err := json.Unmarshal(p, &result)
check(err)
// slice of locations we'd like to filter the objects on
locations := []string{"England"} // Can be more
if sliceContains(s.Location, locations) {
s.ID = result.ID
s.Location= result.Location
s.Content = result.Content
}
return nil
}
// helper func to check if a given string, f.e. a value of a key-value pair in a json object, is in a provided list
func sliceContains(a string, list []string) bool {
for _, b := range list {
if b == a {
fmt.Println("it's a match!")
return true
}
}
return false
}
While this runs - the output is wrong. It creates as many objects as comes in - however, the new ones are empty, f.e.:
// ...
[
{
"Id": 0,
"Location": "",
"Content": ""
},
{
"Id": 0,
"Location": "",
"Content": ""
}
]
//...
Whereas my desired output, from the first given input, would be:
[
{
"Id": 145,
"Location": "England",
"Content": "SomeString"
},
{
"Id": 12,
"Location": "England",
"Content": "SomeString"
},
{
"Id": 213123,
"Location": "England",
"Content": "SomeString"
}
]
When languageStruct.UnmarshalJSON() is called, there is already a languageStruct prepared that will be appended to the slice, no matter if you fill its content (fields) or not.
The easiest and my suggested solution is to just unmarshal normally, and post-process the slice: remove elements according to your requirements. This results in clean code, which you can easily adjust / alter in the future. Although it could be implemented as custom marshaling logic on a custom slice type []languageStruct, I would still not create custom marshaling logic for this but implement it as a separate filtering logic.
Here's a simple code unmarshaling, filtering and marshaling it again (note: no custom marshaling is defined / used for this):
var x []*languageStruct
err := json.Unmarshal(file, &x)
if err != nil {
panic(err)
}
var x2 []*languageStruct
for _, v := range x {
if v.Location == "England" {
x2 = append(x2, v)
}
}
data, err := json.MarshalIndent(x2, "", " ")
fmt.Println(string(data), err)
This will result in your desired output. Try it on the Go Playground.
The fastest and most complex solution would be to use event-driven parsing and building a state machine, but the complexity would increase by large. The idea would be to process the JSON by tokens, track where you're at currently in the object tree, and when an object is detected that must be excluded, don't process / add it to your slice. For details and ideas how this can be written, check out this anwser: Go - Decode JSON as it is still streaming in via net/http