Here is a Go Playground demonstrating my problem: http://play.golang.org/p/2fq3Fg7rPg
Essentially, I am trying to JSON marshal a struct containing a custom type wrapping json.RawMessage. When using CustomType.MarshalJSON() I get the expected results, but just calling json.Marshal on my full struct does not work as expected. See the playground link for a concrete example.
What is causing this difference?
Is there a way to have json.Marshal work as I expect it to?
Your code works fine, you just have one little bug.
// MarshalJSON returns the *j as the JSON encoding of j.
func (j JsonText) MarshalJSON() ([]byte, error) {
return j, nil
} // note i modified this so the receiver isn't a pointer
Your code didn't work because this is your definition of your datatype that wraps JsonText;
// Example struct I want to marshal and unmarshal
type TestData struct {
Field1 JsonText `json:"field_1"`
}
But only the *JsonText type implements the marshaler interface in your code. So you can change the types in either place ( I did in the MarshalJSON() ) but they need to be consistent.
In the playground; http://play.golang.org/p/NI_z3bQx7a
Related
I am trying to retrieve information using Reddit's API. Here is some documentation on their json response, however, I got most of my information by just viewing the link in the browser and pretty-printing the response here.
The following code behaves as intended when the "Replies" field is commented out, but fails when it's not.
[edit] getData() is a function I wrote that uses Go's http Client to get a site response in bytes.
type redditThing struct {
Data struct {
Children []struct {
Data struct {
Permalink string
Subreddit string
Title string
Body string
Replies redditThing
}
}
}
}
func visitLink(link string) {
println("visiting:", link)
var comments []redditThing
if err := json.Unmarshal(getData(link+".json?raw_json=1"), &comments); err != nil {
logError.Println(err)
return
}
}
This throws the following error
json: cannot unmarshal string into Go struct field .Data.Children.Data.Replies.Data.Children.Data.Replies.Data.Children.Data.Replies of type main.redditThing
Any help would be greatly appreciated. Thank you all in advance!
[edit] here a link to some data causing the program to fail
The replies field can be the empty string or a redditThing. Fix by adding an Unmarshal function to handle the empty string:
func (rt *redditThing) UnmarshalJSON(data []byte) error {
// Do nothing if data is the empty string.
if bytes.Equal(data, []byte(`""`)) {
return nil
}
// Prevent recursion by declaring type x with
// same underlying type as redditThing, but
// with no methods.
type x redditThing
return json.Unmarshal(data, (*x)(rt))
}
The x type is used to prevent indefinite recursion. If the final line of the method is json.Unmarshal(data, rt), then json.Unmarshal function will call redditThing.UnmarshalJSON method which calls json.Unmarshal function and so on. Boom!
The statement type x redditThing declares a new type named x with the same underlying type as redditThing. The underlying type is a anonymous struct type. The underlying type has no methods, and crucially, the underlying type does not have the UnmarshalJSON method. This prevents recursion.
I have a json that contains a field with a bigint
{"NETWORK_ID": 6000370005980500000071}
The format I have before marshaling is map[string]interface{}
When I marshal it and print to console everything seems to be fine but this field actually creates problems due to its size in other mediums so I want to serialize it as a string.
UseNumber() seems to be for this purpose but it's only for decoding I think.
Is there any way that I can detect this kind of bigint number and make them serialize as strings?
You'll need to create a custom type that implements the json.Marshaler interface, and marshals to a string. Example:
type MyBigInt big.Int
func (i MyBigInt) MarshalJSON() ([]byte, error) {
i2 := big.Int(i)
return []byte(fmt.Sprintf(`"%s"`, i2.String()), nil
}
This will always marshal your custom type as a quoted decimal number.
I have a big.float which I'm encoding into JSON . However the JSON always end up showing the float in scientific notation rater than decimal notation. I can fix this by changing the JSON to be a string rather than a number and using float.Text('f'), however I would really prefer to keep the type as a number.
I was a taking a look at float.Format but I don't believe this is suitable.
A really condensed gist of what I'm doing is below. I do a lot more modification of the value of supply before encoding it to json.
type TokenSupply struct {
TotalSupply *big.Float `json:"totalSupply, omitempty"`
}
supply := Float.NewFloat(1000000)
json.NewEncoder(w).Encode(TokenSupply{supply})
This returns:
{"totalSupply":"1e+06"}
big.Float is marshaled to string when converted to a JSON type
https://golang.org/pkg/encoding/json/#Marshal
Marshal traverses the value v recursively. If an encountered value implements the Marshaler interface and is not a nil pointer, Marshal calls its MarshalJSON method to produce JSON. If no MarshalJSON method is present but the value implements encoding.TextMarshaler instead, Marshal calls its MarshalText method and encodes the result as a JSON string. The nil pointer exception is not strictly necessary but mimics a similar, necessary exception in the behavior of UnmarshalJSON.
https://golang.org/pkg/math/big/#Float.MarshalText
func (x *Float) MarshalText() (text []byte, err error)
What can you do about it?
since your float may be more than 64 bits it won't play well with other languages that have to read the JSON value as a number. I'd suggest you keep the number as a string.
Caveats about encoding numbers that don't fit into 64 bits aside, here is how you could marshal a big.Float as a JSON number by wrapping it in a custom type that implements json.Marshaler. The key is that you can implement theMarshalJSON method anyway you like, as long as it emits valid JSON:
package main
import (
"encoding/json"
"fmt"
"math/big"
)
type BigFloatNumberJSON struct{ *big.Float }
func (bfn BigFloatNumberJSON) MarshalJSON() ([]byte, error) {
// Use big.Float.String() or any other string converter
// that emits a valid JSON number here...
return []byte(bfn.String()), nil
}
func main() {
totalSupply := new(big.Float).SetFloat64(1000000)
obj := map[string]interface{}{
"totalSupply": BigFloatNumberJSON{totalSupply},
}
bytes, err := json.Marshal(&obj)
if err != nil {
panic(err)
}
fmt.Println(string(bytes))
// => {"totalSupply":1000000}
}
I'm writing a wrapper to map with some additional functionality I need. Some of the most important things is the ability to marshal and unmarshal the data while retaining genericity. I managed to write an marshaller using the encoding/gob encoder, but since it would be nice if the marshalled data was human readable, I decided to code another implementation with JSON.
I got gob to encode from and decode to generic interface variables neatly by passing it a implementation object instance with Register(). (This resource helped me with the details! http://www.funcmain.com/gob_encoding_an_interface)
However, JSON doesn't have Register(). Let's say we have a value of type
type ConcreteImplementation struct {
FieldA string
FieldB string
}
func (c ConcreteImplementation) Key() string {
return c.FieldA // ConcreteImplementation implements genericValue
}
in a variable of type
type genericValue interface {
Key() string
}
When I marshal that, it outputs JSON like this:
{"FieldA":"foo","FieldB":"bar"}
And when I try to unmarshal that again into a variable of type genericValue, it says:
panic: interface conversion: map[string]interface {} is not genericValue: missing method Key EDIT: Oops, actually it says this!
Error with decoding! json: cannot unmarshal object into Go value of genericValue
Quite obviously, it tries to marshal the data like it says here: http://blog.golang.org/json-and-go (See 'Generic JSON with interface{}')
How can I get it to try to fit the data to an specific implementation, like gob decoder would try if the implementation is Register()ed? Register() was godsend, it allowed to marshal and unmarshal generically like it was nothing. How do I get JSON to do the same thing?
What if your types implemented the Unmarshaler?
Here is a small demo.
Or the same code here:
type ConcreteImplementation struct {
FieldA string
FieldB string
}
func (c ConcreteImplementation) Key() string {
return c.FieldA // ConcreteImplementation implements genericValue
}
// implementing json.Unmarshaler
func (c *ConcreteImplementation) UnmarshalJSON(j []byte) error {
m := make(map[string]string)
err := json.Unmarshal(j, &m)
if err != nil {
return err
}
if v, ok := m["FieldA"]; ok {
c.FieldA = v
}
if v, ok := m["FieldB"]; ok {
c.FieldB = v
}
return nil
}
type genericValue interface {
Key() string
json.Unmarshaler
}
func decode(jsonStr []byte, v genericValue) error {
return json.Unmarshal(jsonStr, v)
}
With this you can pass a genericValue to json.Unmarshal.
Allright, got it to work finally. This question provided the answer. Why does json.Unmarshal return a map instead of the expected struct?
"You've passed to json a pointer to an abstract interface. You should simply pass a pointer to Ping as an abstract interface" - This applied to my situation too. (For some reason a pointer TO an abstract interface was enough for gob package. It seems that I have to study Go interfaces and reflection some more to understand why...)
I won't still mark this as solved question, if someone has a better answer.
I have a struct like this:
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
But even if the instance of MyStruct is entirely empty (meaning, all values are default), it's being serialized as:
"data":{}
I know that the encoding/json docs specify that "empty" fields are:
false, 0, any nil pointer or interface value, and any array,
slice, map, or string of length zero
but with no consideration for a struct with all empty/default values. All of its fields are also tagged with omitempty, but this has no effect.
How can I get the JSON package to not marshal my field that is an empty struct?
As the docs say, "any nil pointer." -- make the struct a pointer. Pointers have obvious "empty" values: nil.
Fix - define the type with a struct pointer field:
type Result struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
Then a value like this:
result := Result{}
Will marshal as:
{}
Explanation: Notice the *MyStruct in our type definition. JSON serialization doesn't care whether it is a pointer or not -- that's a runtime detail. So making struct fields into pointers only has implications for compiling and runtime).
Just note that if you do change the field type from MyStruct to *MyStruct, you will need pointers to struct values to populate it, like so:
Data: &MyStruct{ /* values */ }
As #chakrit mentioned in a comment, you can't get this to work by implementing json.Marshaler on MyStruct, and implementing a custom JSON marshalling function on every struct that uses it can be a lot more work. It really depends on your use case as to whether it's worth the extra work or whether you're prepared to live with empty structs in your JSON, but here's the pattern I use applied to Result:
type Result struct {
Data MyStruct
Status string
Reason string
}
func (r Result) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}{
Data: &r.Data,
Status: r.Status,
Reason: r.Reason,
})
}
func (r *Result) UnmarshalJSON(b []byte) error {
decoded := new(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
})
err := json.Unmarshal(b, decoded)
if err == nil {
r.Data = decoded.Data
r.Status = decoded.Status
r.Reason = decoded.Reason
}
return err
}
If you have huge structs with many fields this can become tedious, especially changing a struct's implementation later, but short of rewriting the whole json package to suit your needs (not a good idea), this is pretty much the only way I can think of getting this done while still keeping a non-pointer MyStruct in there.
Also, you don't have to use inline structs, you can create named ones. I use LiteIDE with code completion though, so I prefer inline to avoid clutter.
Data is an initialized struct, so it isn't considered empty because encoding/json only looks at the immediate value, not the fields inside the struct.
Unfortunately, returning nil from json.Marshaler doesn't currently work:
func (_ MyStruct) MarshalJSON() ([]byte, error) {
if empty {
return nil, nil // unexpected end of JSON input
}
// ...
}
You could give Result a marshaler as well, but it's not worth the effort.
The only option, as Matt suggests, is to make Data a pointer and set the value to nil.
There is an outstanding Golang proposal for this feature which has been active for over 4 years, so at this point, it is safe to assume that it will not make it into the standard library anytime soon. As #Matt pointed out, the traditional approach is to convert the structs to pointers-to-structs. If this approach is infeasible (or impractical), then an alternative is to use an alternate json encoder which does support omitting zero value structs.
I created a mirror of the Golang json library (clarketm/json) with added support for omitting zero value structs when the omitempty tag is applied. This library detects zeroness in a similar manner to the popular YAML encoder go-yaml by recursively checking the public struct fields.
e.g.
$ go get -u "github.com/clarketm/json"
import (
"fmt"
"github.com/clarketm/json" // drop-in replacement for `encoding/json`
)
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
j, _ := json.Marshal(&Result{
Status: "204",
Reason: "No Content",
})
fmt.Println(string(j))
// Note: `data` is omitted from the resultant json.
{
"status": "204"
"reason": "No Content"
}