Unmarshal inconsistent JSON - json

I have JSON (that I cannot control) like this:
{
"foo1":{
"a":{
"up":10,
"down":5
}
},
"foo2":{
"a":{
"up":1,
"down":1
}
},
"bar":{
"up":11,
"down":6
}
}
"foo1" and "foo2" are dynamic.
How can I properly unmarshal this structure in go?
It would be okay if I could just tell go to not try to deserialize "bar" (the inconsistent property).

Go will by default ignore fields unspecified in the struct you unmarshal into.
In this case, your structure would be set up like this:
type NestedProp2 struct {
Up int
Down int
}
type NestedProp struct {
A NestedProp2
}
type Prop struct {
Foo1 NestedProp
Foo2 NestedProp
}
When you call the the json.Unmarshal function, the extra property will not be deserialized:
var prop Prop
err := json.Unmarshal(jsonBlob, &prop)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", prop)
So you get the following output:
{Foo1:{A:{Up:10 Down:5}} Foo2:{A:{Up:1 Down:1}}}
You can see it in action here.

You said:
I have JSON (that I cannot control)
So to what extent you could control? Here I could provide you with some scenario, and hope some of them match your purpose :)
Remember the general rule first:
In Golang, if a JSON key failed to find a matched field in struct, it will not be unmarshalled.
This means, for a key name in a JSON struct, when unmarshalling, it will look for a field in a golang struct at the same level with the same name case-insensitively. If this search failed, this key won't be unmarshalled.
For example, a key named foo1 will look for a field name foo1 in a golang struct at the same indent level. However it also matches with Foo1 or FoO1, since this matching is case-insensitive.
Remember, you could use field tag to specify the field name as well. Please take a look at the official page.
The value of some of the JSON fields are not consistent, and they could be ignored.
This is the case #gnalck solved in his answer. According to the general rule, if those inconsistent field failed to find a match, they will not be unmarshalled. Therefore, just don't put those inconsistent fields in the struct and you will be fine.
The value of some of the JSON fields are not consistent, but they could not be ignored.
In this case, #gnalck failed since those fields could not be ignored. Now a better way is to unmarshal bar into a json.RawMessage, so that you could unmarshal later.
The keys of the JSON object is undetermined, and their value is undetermined as well.
In this case, we could unmarshal the whole JSON object into a map[string]json.RawMessage, and unmarshal each fields later. When unmarshalling to a map, you could iterate through the map to get all the fields, and unmarshal them into a proper struct later.

Related

Why is Golang json.Unmarshall not working with "e" and "E" properties?

Suppose we want to unmarshal the JSON string {"e": "foo", "E": 1}.
Unmarshalling using the type messageUppercaseE works like expected. When using the type message though, the error json: cannot unmarshal number into Go struct field message.e of type string is returned.
Why are we not able to unmarshal the JSON, if only the "e" struct tag is present?
How would I be able to unmarshal the JSON? (I know that I am able to do this via Jeffail/gabs, but would like to stick to the type based approach.)
type message struct {
EventType string `json:"e"`
}
type messageUppercaseE struct {
EventType string `json:"e"`
UppercaseE uint64 `json:"E"`
}
Try it yourself at https://play.golang.org/p/T6KMJRLy7TN
Quoting the docs for unmarshal:
To unmarshal JSON into a struct, Unmarshal matches incoming object keys to the keys used by Marshal (either the struct field name or its tag), preferring an exact match but also accepting a case-insensitive match.
In this case, it is the case-insensitive match that causes the trouble.

How to get data as is from MongoDB and send it to an API as JSON in Golang

I am writing a Golang API at work which when called, gets data from two different MongoDB Collections and appends it into a struct, converts it to JSON, and stringified and sends to an API (Amazon SQS)
The problem is, defining the struct of the data receiving from MongoDB, while some of the fields are defined properly, some are varying
// IncentiveRule struct defines the structure of Incentive rule from Mongo
type IncentiveRule struct {
... Other vars
Rule Rule `bson:"rule" json:"rule"`
... Other vars
}
// Rule defines the struct for Rule Object inside an incentive rule
type Rule struct {
...
Rules interface{} `bson:"rules" json:"rules"`
RuleFilter RuleFilter `bson:"rule_filter" bson:"rule_filter"`
...
}
// RuleFilter ...
type RuleFilter struct {
Condition string `bson:"condition" json:"condition"`
Rules []interface{} `bson:"rules" json:"rules"`
}
While this works, the interface{} defined inside Rule struct is varying and while getting as BSON and decoding and re-encoding to JSON, instead of encoding as "fookey":"barvalue" in JSON, it is encoded as "Key":"fookey","Value":"barvalue", how to avoid this behavior and have it as "fookey":"barvalue"
If you use interface{}, the mongo-go driver is free to choose whatever implementation it sees fits for representing the results. Often it will choose bson.D to represent documents which is an ordered list of key-value pairs where a pair is a struct having a field for Key and a field for Value , so the Go value can preserve the field order.
If field order is not required / important, you may explicitly use bson.M instead of interface{} and []bson.M instead of []interface{}. bson.M is an unordered map, but it represents fields in the form of fieldName: fieldValue, which is exactly what you want.

How to unmarshal dynamic json objects with Golang

let me start by telling that I'm pretty recent in the Go world.
What I'm trying to do is to read the json I get from a JSON API (I don't control). Everything is working fine, I can show the received ID and Tags too. But the fields field is a little bit different, because its a dynamic array.
I can receive from the api this:
{
"id":"M7DHM98AD2-32E3223F",
"tags": [
{
"id":"9M23X2Z0",
"name":"History"
},
{
"id":"123123123",
"name":"Theory"
}
],
"fields": {
"title":"Title of the item",
"description":"Description of the item"
}
}
Or instead of title and description I could receive only description, or receive another random object like long_title. The objects return may differ completly and can be an infinite possibility of objects. But it always returns objects with a key and a string content like in the example.
This is my code so far:
type Item struct {
ID string `json:"id"`
Tags []Tag `json:"tags"`
//Fields []Field `json:"fields"`
}
// Tag data from the call
type Tag struct {
ID string `json:"id"`
Name string `json:"name"`
}
// AllEntries gets all entries from the session
func AllEntries() {
resp, _ := client.Get(APIURL)
body, _ := ioutil.ReadAll(resp.Body)
item := new(Item)
_ = json.Unmarshal(body, &item)
fmt.Println(i, "->", item.ID)
}
So the Item.Fields is dynamic, there is no way to predict what will be the key names, and therefore as far I can tell, there is no way to create a struct for it. But again, I'm pretty newbie with Go, could someone give me any tips? Thanks
If the data in "fields" is always going to be a flat-dict then you can use map[string]string as type for the Fields.
For arbitrary data specify Fields as a RawMessage type and parse it later on based on its content. Example from docs: https://play.golang.org/p/IR1_O87SHv
If the fields are way too unpredictable then you can keep this field as is([]byte) or if there are fields that are always going to common then you can parse those and leave the rest(but this would result in loss of data present in other fields).

json.Unmarshal doesn't seem to pay attention to struct tags

I have a JSON object that looks like this:
{"API version":"1.2.3"}
And I want to convert it to an object it using the json.Unmarshal() function in go. According to this blog post:
How does Unmarshal identify the fields in which to store the decoded data? For a given JSON key "Foo", Unmarshal will look through the destination struct's fields to find (in order of preference):
An exported field with a tag of "Foo" (see the Go spec for more on struct tags),
An exported field named "Foo", or
An exported field named "FOO" or "FoO" or some other case-insensitive match of "Foo".
This is confirmed by the unmarshal documentation.
Since "API version" has a space in it, which is not a valid go identifier, I used a tag on the field:
type ApiVersion struct {
Api_version string "API version"
}
And I try to unmarshal it like so:
func GetVersion() (ver ApiVersion, err error) {
// Snip code that gets the JSON
log.Println("Json:",string(data))
err = json.Unmarshal(data,&ver)
log.Println("Unmarshalled:",ver);
}
The output is:
2014/01/06 16:47:38 Json: {"API version":"1.2.3"}
2014/01/06 16:47:38 Unmarshalled: {}
As you can see, the JSON isn't being marshalled into ver. What am I missing?
The encoding/json module requires that the struct tags be namespaced. So you instead want something like:
type ApiVersion struct {
Api_version string `json:"API version"`
}
This is done so that the json struct tags can co-exist with tags from other libraries (such as the XML encoder).

How to not marshal an empty struct into JSON with Go?

I have a struct like this:
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
But even if the instance of MyStruct is entirely empty (meaning, all values are default), it's being serialized as:
"data":{}
I know that the encoding/json docs specify that "empty" fields are:
false, 0, any nil pointer or interface value, and any array,
slice, map, or string of length zero
but with no consideration for a struct with all empty/default values. All of its fields are also tagged with omitempty, but this has no effect.
How can I get the JSON package to not marshal my field that is an empty struct?
As the docs say, "any nil pointer." -- make the struct a pointer. Pointers have obvious "empty" values: nil.
Fix - define the type with a struct pointer field:
type Result struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
Then a value like this:
result := Result{}
Will marshal as:
{}
Explanation: Notice the *MyStruct in our type definition. JSON serialization doesn't care whether it is a pointer or not -- that's a runtime detail. So making struct fields into pointers only has implications for compiling and runtime).
Just note that if you do change the field type from MyStruct to *MyStruct, you will need pointers to struct values to populate it, like so:
Data: &MyStruct{ /* values */ }
As #chakrit mentioned in a comment, you can't get this to work by implementing json.Marshaler on MyStruct, and implementing a custom JSON marshalling function on every struct that uses it can be a lot more work. It really depends on your use case as to whether it's worth the extra work or whether you're prepared to live with empty structs in your JSON, but here's the pattern I use applied to Result:
type Result struct {
Data MyStruct
Status string
Reason string
}
func (r Result) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}{
Data: &r.Data,
Status: r.Status,
Reason: r.Reason,
})
}
func (r *Result) UnmarshalJSON(b []byte) error {
decoded := new(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
})
err := json.Unmarshal(b, decoded)
if err == nil {
r.Data = decoded.Data
r.Status = decoded.Status
r.Reason = decoded.Reason
}
return err
}
If you have huge structs with many fields this can become tedious, especially changing a struct's implementation later, but short of rewriting the whole json package to suit your needs (not a good idea), this is pretty much the only way I can think of getting this done while still keeping a non-pointer MyStruct in there.
Also, you don't have to use inline structs, you can create named ones. I use LiteIDE with code completion though, so I prefer inline to avoid clutter.
Data is an initialized struct, so it isn't considered empty because encoding/json only looks at the immediate value, not the fields inside the struct.
Unfortunately, returning nil from json.Marshaler doesn't currently work:
func (_ MyStruct) MarshalJSON() ([]byte, error) {
if empty {
return nil, nil // unexpected end of JSON input
}
// ...
}
You could give Result a marshaler as well, but it's not worth the effort.
The only option, as Matt suggests, is to make Data a pointer and set the value to nil.
There is an outstanding Golang proposal for this feature which has been active for over 4 years, so at this point, it is safe to assume that it will not make it into the standard library anytime soon. As #Matt pointed out, the traditional approach is to convert the structs to pointers-to-structs. If this approach is infeasible (or impractical), then an alternative is to use an alternate json encoder which does support omitting zero value structs.
I created a mirror of the Golang json library (clarketm/json) with added support for omitting zero value structs when the omitempty tag is applied. This library detects zeroness in a similar manner to the popular YAML encoder go-yaml by recursively checking the public struct fields.
e.g.
$ go get -u "github.com/clarketm/json"
import (
"fmt"
"github.com/clarketm/json" // drop-in replacement for `encoding/json`
)
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
j, _ := json.Marshal(&Result{
Status: "204",
Reason: "No Content",
})
fmt.Println(string(j))
// Note: `data` is omitted from the resultant json.
{
"status": "204"
"reason": "No Content"
}