This question already has answers here:
How can we read a json file as json object in golang
(2 answers)
Closed 7 months ago.
What I want to achieve
I'm parsing IAM Policies in Go. In IAM Policies, most of the fields can be either a string or an array of strings. It's hard to think these decision trees in my head what I want is kind of exhaustive pattern matching.
What I did
I loeded a policy with json.Unmarshal
type Policy struct {
Version string `json:"Version"`
Id string `json:"ID,omitempty"`
Statements []Statement `json:"Statement"`
}
type Statement struct {
// ...
Action interface{} `json:"Action"` // string or array
// ...
}
And iterating over statements.
switch ele := st.Action.(type) {
case string:
action, _ := expandAction(ele, data)
actions = append(actions, action...)
setter(i, actions)
case []string:
for _, a := range ele {
if strings.Contains(a, "*") {
exps, _ := expandAction(a, data)
actions = append(actions, exps...)
} else {
actions = append(actions, a)
}
setter(i, actions)
fmt.Println(actions)
}
default:
// interface{}
}
The Problem
It always goes to the default case.
Can use reflection, but don't think I really should, since runtime could know types when json.Unnarshal is called.
As you can see from the official document the type for JSON array is []interface. If you update []string to []interface then you can run the related case block. However, if you have to sure that is array of string, reflection can provide it.
Related
I'm trying to consume a rest endpoint in my golang project. The JSON structure is pretty large and is semi-structured, so I'm using reflection to iterate over it and grab the information that I am interested in.
Here is how I'm unmarshal-ing the response:
var m map[string]interface{}
json.Unmarshal(bytes, &m)
But the part I'm stuck at is - I'm iterating through a slice of maps (i think) but I'm unable to get the keys or values of the map. Here is the code in question.
if val, ok := m["messages"]; ok {
s := reflect.ValueOf(val)
if s.Kind() == reflect.Slice {
for i := 0; i < s.Len(); i++ {
item := s.Index(i)
fmt.Println("VALUE = ", item.Elem())
fmt.Println("KIND = ", item.Kind())
}
}
return
}
When I run the code the value that is displayed looks like a map:
map[decoration_stats:<nil> highlight_ranges:map[] index:graylog_7 message:map[_id:49272694-1834-11ea-8928-0242ac120004 docker:{"container_id":"0f9d97722c25240c6f99487b247b2416177a749de47d661cd661334514e0e74f"} facility:fluentd gl2_message_id:01DVDSM9VSDQ5PF81T4C31NSH6....
And the kind is:
KIND = interface
I tried various things like:
for _, e := range val.MapKeys() {
v := val.MapIndex(e)
fmt.Println(v)
}
But the code panics with:
panic: reflect: call of reflect.Value.MapKeys on interface Value
Sorry, I'm somewhat new to golang but have used other static typed language, mainly Java, when it comes to any reflection type programming.
My question is how to can I convert this interface to a map or some concrete type so that I can use it. Any help would be greatly appreciated.
Using reflection is an inefficient way to do this. JSON unmarshal, when used with an interface (and map[string]interface{}) produces a limited set of types, and you can use type assertions or a type-switch to deal with it:
if val, ok := m["messages"]; ok {
switch v:=val.(type) {
case map[string]interface{}: // JSON object
for key, value:=range v {
}
case []interface{}: // JSON array
for i,node:=range v {
}
case string: // string value
case float64: // numeric value
case bool: // boolean value
case json.Number: // If you use json.Decoder with UseNumber()
}
}
I'm making an JSON API wrapper client that needs to fetch paginated results, where the URL to the next page is provided by the previous page. To reduce code duplication for the 100+ entities that share the same response format, I would like to have a single client method that fetches and unmarshalls the different entities from all paginated pages.
My current approach in a simplified (pseudo) version (without errors etc):
type ListResponse struct {
Data struct {
Results []interface{} `json:"results"`
Next string `json:"__next"`
} `json:"d"`
}
func (c *Client) ListRequest(uri string) listResponse ListResponse {
// Do a http request to uri and get the body
body := []byte(`{ "d": { "__next": "URL", "results": []}}`)
json.NewDecoder(body).Decode(&listResponse)
}
func (c *Client) ListRequestAll(uri string, v interface{}) {
a := []interface{}
f := c.ListRequest(uri)
a = append(a, f.Data.Results...)
var next = f.Data.Next
for next != "" {
r := c.ListRequest(next)
a = append(a, r.Data.Results...)
next = r.Data.Next
}
b, _ := json.Marshal(a)
json.Unmarshal(b, v)
}
// Then in a method requesting all results for a single entity
var entities []Entity1
client.ListRequestAll("https://foo.bar/entities1.json", &entities)
// and somewehere else
var entities []Entity2
client.ListRequestAll("https://foo.bar/entities2.json", &entities)
The problem however is that this approach is inefficient and uses too much memory etc, ie first Unmarshalling in a general ListResponse with results as []interface{} (to see the next URL and concat the results into a single slice), then marshalling the []interface{} for unmarshalling it directly aftwards in the destination slice of []Entity1.
I might be able to use the reflect package to dynamically make new slices of these entities, directly unmarshal into them and concat/append them afterwards, however if I understand correctly I better not use reflect unless strictly necessary...
Take a look at the RawMessage type in the encoding/json package. It allows you to defer the decoding of json values until later. For example:
Results []json.RawMessage `json:"results"`
or even...
Results json.RawMessage `json:"results"`
Since json.RawMessage is just a slice of bytes this will be much more efficient then the intermediate []interface{} you are unmarshalling to.
As for the second part on how to assemble these into a single slice given multiple page reads you could punt that question to the caller by making the caller use a slice of slices type.
// Then in a method requesting all results for a single entity
var entityPages [][]Entity1
client.ListRequestAll("https://foo.bar/entities1.json", &entityPages)
This still has the unbounded memory consumption problem your general design has, however, since you have to load all of the pages / items at once. You might want to consider changing to an Open/Read abstraction like working with files. You'd have some Open method that returns another type that, like os.File, provides a method for reading a subset of data at a time, while internally requesting pages and buffering as needed.
Perhaps something like this (untested):
type PagedReader struct {
c *Client
buffer []json.RawMessage
next string
}
func (r *PagedReader) getPage() {
f := r.c.ListRequest(r.next)
r.next = f.Data.Next
r.buffer = append(r.buffer, f.Data.Results...)
}
func (r *PagedReader) ReadItems(output []interface{}) int {
for len(output) > len(buffer) && r.next != "" {
r.getPage()
}
n := 0
for i:=0;i<len(output)&&i< len(r.buffer);i++ {
json.Unmarshal(r.buffer[i], output[i] )
n++
}
r.buffer = r.buffer[n:]
return n
}
This question already has answers here:
Lowercase JSON key names with JSON Marshal in Go
(4 answers)
Closed 5 years ago.
What I am trying to do
I am parsing a JSON HTTP response based on this answer to a similar question. My code is able to parse the JSON without any error but is unable to read the values and store them in the provided variable.
This has been puzzling me for the last 2 hours and it might be due to a trivial reason that I am overlooking here.
CODE
type ImporterResponse struct {
results []packagemeta `json:"results"`
}
type packagemeta struct {
path string `json:"path"`
synopsis string `json:"synopsis,omitempty"`
count int `json:"import_count,omitempty`
}
func main() {
res := []byte(`{"results":[{"path":"4d63.com/randstr/lib/randstr","import_count":0,"synopsis":"Package randstr generates random strings (e.g."},{"path":"bitbucket.org/pcas/tool/mathutil","import_count":0}]}`)
fmt.Println("Decoding the JSON")
r := bytes.NewReader(res)
decoder := json.NewDecoder(r)
packageimporters := &ImporterResponse{}
err := decoder.Decode(packageimporters)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Packageimporters: %+v", packageimporters)
fmt.Println(len(packageimporters.results))
}
Link to Playground: https://play.golang.org/p/NzLl7Ujo2IJ
What I want:
How to fix this?
Why is no error message raised if JSON is not parsed properly?
P.S: I understand that this question has been asked before and there are possible solutions available but none of them work for me. Hence, I have made this post.
You need to make your struct fields exported, otherwise the json package cannot access them.
Please read JSON and go for more details, specifically this paragraph:
The json package only accesses the exported fields of struct types
(those that begin with an uppercase letter). Therefore only the the
exported fields of a struct will be present in the JSON output.
And this one for more details:
How does Unmarshal identify the fields in which to store the decoded
data? For a given JSON key "Foo", Unmarshal will look through the
destination struct's fields to find (in order of preference):
An exported field with a tag of "Foo" (see the Go spec for more on
struct tags),
An exported field named "Foo", or
An exported field
named "FOO" or "FoO" or some other case-insensitive match of "Foo".
So your struct should really be:
type Packagemeta struct {
Path string `json:"path"`
Synopsis string `json:"synopsis,omitempty"`
Count int `json:"import_count,omitempty`
}
This question already has answers here:
JSON and dealing with unexported fields
(2 answers)
(un)marshalling json golang not working
(3 answers)
json.Marshal(struct) returns "{}"
(3 answers)
Printing Empty Json as a result [duplicate]
(1 answer)
Parsing JSON in Golang doesn't Populate Object [duplicate]
(1 answer)
Closed 10 months ago.
I'm a new Go programmer (From Java) and I would like to reproduce a generic way which is esay to use in Java.
I want to create some function which allow me to do an Unmarshal on a JSON string in order to avoid code duplicity.
This is my current code which is not working :
type myStruct1 struct {
id string
name string
}
func (obj myStruct1) toString() string {
var result bytes.Buffer
result.WriteString("id : ")
result.WriteString(obj.id)
result.WriteString("\n")
result.WriteString("name : ")
result.WriteString(obj.name)
return result.String()
}
func main() {
content := `{id:"id1",name="myName"}`
object := myStruct1{}
parseJSON(content, object)
fmt.Println(object.toString())
}
func parseJSON(content string, object interface{}) {
var parsed interface{}
json.Unmarshal([]byte(content), &parsed)
}
This code, on run, returns me this :
id :
name :
Do you have any idea ?
Thanks
The issue is you want to write to a generic type? You probably want a string map. This works with BSON anyways:
var anyJson map[string]interface{}
json.Unmarshal(bytes, &anyJson)
You'll be able to access the fields like so:
anyJson["id"].(string)
Don't forget to type assert your values, and they must be the correct type or they'll panic. (You can read more about type assertions on the golang site)
To parse "generic JSON" when you have no idea what schema it has:
var parsed any
err := json.Unmarshal(jsonText, &parsed)
The returned any in parsed will be a map[string]any or []any or nil or single values float64, bool, string.
You can test the type and react accordingly.
import (
"encoding/json"
"fmt"
)
func test(jsonText []byte) {
// parsing
var parsed any
err := json.Unmarshal(jsonText, &parsed)
if err != nil {
panic(err) // malformed input
}
// type-specific logic
switch val := parsed.(type) {
case nil:
fmt.Println("json specifies null")
case map[string]any:
fmt.Printf("id:%s name:%s\n", val["id"], val["name"])
case []any:
fmt.Printf("list of %d items\n", len(val))
case float64:
fmt.Printf("single number %f\n", val)
case bool:
fmt.Printf("single bool %v\n", val)
case string:
fmt.Printf("single string %s\n", val)
default:
panic(fmt.Errorf("type %T unexpected", parsed))
}
}
Unmarshal will only set exported fields of the struct.
Which means you need to modify the json struct to use capital case letters:
type myStruct1 struct {
Id string
Name string
}
The reason behind this is that the json library does not have the ability to view fields using reflect unless they are exported.
You have to export your fields:
type myStruct1 struct {
Id string
Name string
}
See Exported Identifiers from documentation.
There are a few changes you need to make in your code to make it work:
The function json.Unmarshal can only set variables inside your struct which are exported, that is, which start with capital letters. Use something like ID and Name for your variable names inside myStruct1.
Your content is invalid JSON. What you actually want is {"ID":"id1","Name":"myName"}.
You're passing object to parseJSON but you're using parsed instead, not object. Make parseJSON receive a *myStruct (instead of an interface{}), and use that variable instead of parsed when unmarshalling the string. Also, always handle the error returns, like err := json.Unmarshal(content, object), and check err.
I'd suggest you to do the Golang Tour ;)
You can also set the file as an Object with dynamic properties inside another struct. This will let you add metadata and you read it the same way.
type MyFile struct {
Version string
Data map[string]interface{}
}
I have a struct like this:
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
But even if the instance of MyStruct is entirely empty (meaning, all values are default), it's being serialized as:
"data":{}
I know that the encoding/json docs specify that "empty" fields are:
false, 0, any nil pointer or interface value, and any array,
slice, map, or string of length zero
but with no consideration for a struct with all empty/default values. All of its fields are also tagged with omitempty, but this has no effect.
How can I get the JSON package to not marshal my field that is an empty struct?
As the docs say, "any nil pointer." -- make the struct a pointer. Pointers have obvious "empty" values: nil.
Fix - define the type with a struct pointer field:
type Result struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
Then a value like this:
result := Result{}
Will marshal as:
{}
Explanation: Notice the *MyStruct in our type definition. JSON serialization doesn't care whether it is a pointer or not -- that's a runtime detail. So making struct fields into pointers only has implications for compiling and runtime).
Just note that if you do change the field type from MyStruct to *MyStruct, you will need pointers to struct values to populate it, like so:
Data: &MyStruct{ /* values */ }
As #chakrit mentioned in a comment, you can't get this to work by implementing json.Marshaler on MyStruct, and implementing a custom JSON marshalling function on every struct that uses it can be a lot more work. It really depends on your use case as to whether it's worth the extra work or whether you're prepared to live with empty structs in your JSON, but here's the pattern I use applied to Result:
type Result struct {
Data MyStruct
Status string
Reason string
}
func (r Result) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}{
Data: &r.Data,
Status: r.Status,
Reason: r.Reason,
})
}
func (r *Result) UnmarshalJSON(b []byte) error {
decoded := new(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
})
err := json.Unmarshal(b, decoded)
if err == nil {
r.Data = decoded.Data
r.Status = decoded.Status
r.Reason = decoded.Reason
}
return err
}
If you have huge structs with many fields this can become tedious, especially changing a struct's implementation later, but short of rewriting the whole json package to suit your needs (not a good idea), this is pretty much the only way I can think of getting this done while still keeping a non-pointer MyStruct in there.
Also, you don't have to use inline structs, you can create named ones. I use LiteIDE with code completion though, so I prefer inline to avoid clutter.
Data is an initialized struct, so it isn't considered empty because encoding/json only looks at the immediate value, not the fields inside the struct.
Unfortunately, returning nil from json.Marshaler doesn't currently work:
func (_ MyStruct) MarshalJSON() ([]byte, error) {
if empty {
return nil, nil // unexpected end of JSON input
}
// ...
}
You could give Result a marshaler as well, but it's not worth the effort.
The only option, as Matt suggests, is to make Data a pointer and set the value to nil.
There is an outstanding Golang proposal for this feature which has been active for over 4 years, so at this point, it is safe to assume that it will not make it into the standard library anytime soon. As #Matt pointed out, the traditional approach is to convert the structs to pointers-to-structs. If this approach is infeasible (or impractical), then an alternative is to use an alternate json encoder which does support omitting zero value structs.
I created a mirror of the Golang json library (clarketm/json) with added support for omitting zero value structs when the omitempty tag is applied. This library detects zeroness in a similar manner to the popular YAML encoder go-yaml by recursively checking the public struct fields.
e.g.
$ go get -u "github.com/clarketm/json"
import (
"fmt"
"github.com/clarketm/json" // drop-in replacement for `encoding/json`
)
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
j, _ := json.Marshal(&Result{
Status: "204",
Reason: "No Content",
})
fmt.Println(string(j))
// Note: `data` is omitted from the resultant json.
{
"status": "204"
"reason": "No Content"
}