I have a json that contains a field with a bigint
{"NETWORK_ID": 6000370005980500000071}
The format I have before marshaling is map[string]interface{}
When I marshal it and print to console everything seems to be fine but this field actually creates problems due to its size in other mediums so I want to serialize it as a string.
UseNumber() seems to be for this purpose but it's only for decoding I think.
Is there any way that I can detect this kind of bigint number and make them serialize as strings?
You'll need to create a custom type that implements the json.Marshaler interface, and marshals to a string. Example:
type MyBigInt big.Int
func (i MyBigInt) MarshalJSON() ([]byte, error) {
i2 := big.Int(i)
return []byte(fmt.Sprintf(`"%s"`, i2.String()), nil
}
This will always marshal your custom type as a quoted decimal number.
Related
The question How to marshal JSON with bigints? is about marshaling big.Int values into strings in JSON. This question asks, how does one marshal and unmarshal big.Int values natively as numbers in JSON?
Passing around large values marshaled in this manner may be incompatible with other implementations of JSON, particularly JavaScript and jq, as RFC 7159 notes:
Note that when such software is used, numbers that are integers and
are in the range [-(2**53)+1, (2**53)-1] are interoperable in the
sense that implementations will agree exactly on their numeric
values.
Create a custom type BigInt which implements json.Marshaler and json.Unmarshaler like the following:
import (
"fmt"
"math/big"
)
type BigInt struct {
big.Int
}
func (b BigInt) MarshalJSON() ([]byte, error) {
return []byte(b.String()), nil
}
func (b *BigInt) UnmarshalJSON(p []byte) error {
if string(p) == "null" {
return nil
}
var z big.Int
_, ok := z.SetString(string(p), 10)
if !ok {
return fmt.Errorf("not a valid big integer: %s", p)
}
b.Int = z
return nil
}
Rationale:
Implemented as a struct type embedding big.Int instead of as a subtype of big.Int so that the methods (Add, SetString, etc.) are kept
MarshalJSON takes a value receiver so that marshaled values using BigInt don't have to use pointers
UnmarshalJSON takes a pointer receiver because it modifies the receiver; however, types using BigInt still don't have to use pointers
UnmarshalJSON works on a temporary value because the SetString method has undefined behavior on its receiver in the case of errors
As with big.Int the zero value is useful and is equal to the number 0.
I have a big.float which I'm encoding into JSON . However the JSON always end up showing the float in scientific notation rater than decimal notation. I can fix this by changing the JSON to be a string rather than a number and using float.Text('f'), however I would really prefer to keep the type as a number.
I was a taking a look at float.Format but I don't believe this is suitable.
A really condensed gist of what I'm doing is below. I do a lot more modification of the value of supply before encoding it to json.
type TokenSupply struct {
TotalSupply *big.Float `json:"totalSupply, omitempty"`
}
supply := Float.NewFloat(1000000)
json.NewEncoder(w).Encode(TokenSupply{supply})
This returns:
{"totalSupply":"1e+06"}
big.Float is marshaled to string when converted to a JSON type
https://golang.org/pkg/encoding/json/#Marshal
Marshal traverses the value v recursively. If an encountered value implements the Marshaler interface and is not a nil pointer, Marshal calls its MarshalJSON method to produce JSON. If no MarshalJSON method is present but the value implements encoding.TextMarshaler instead, Marshal calls its MarshalText method and encodes the result as a JSON string. The nil pointer exception is not strictly necessary but mimics a similar, necessary exception in the behavior of UnmarshalJSON.
https://golang.org/pkg/math/big/#Float.MarshalText
func (x *Float) MarshalText() (text []byte, err error)
What can you do about it?
since your float may be more than 64 bits it won't play well with other languages that have to read the JSON value as a number. I'd suggest you keep the number as a string.
Caveats about encoding numbers that don't fit into 64 bits aside, here is how you could marshal a big.Float as a JSON number by wrapping it in a custom type that implements json.Marshaler. The key is that you can implement theMarshalJSON method anyway you like, as long as it emits valid JSON:
package main
import (
"encoding/json"
"fmt"
"math/big"
)
type BigFloatNumberJSON struct{ *big.Float }
func (bfn BigFloatNumberJSON) MarshalJSON() ([]byte, error) {
// Use big.Float.String() or any other string converter
// that emits a valid JSON number here...
return []byte(bfn.String()), nil
}
func main() {
totalSupply := new(big.Float).SetFloat64(1000000)
obj := map[string]interface{}{
"totalSupply": BigFloatNumberJSON{totalSupply},
}
bytes, err := json.Marshal(&obj)
if err != nil {
panic(err)
}
fmt.Println(string(bytes))
// => {"totalSupply":1000000}
}
I expected below code to print an object of type struct J, however it prints a map object of type map[string]interface{}. I can feel why it acts like that, however when I run, reflect.ValueOf(i).Kind(), it returns Struct, so it kinda gives me the impression that Unmarshal method should return type J instead of a map. Could anyone enlighten me ?
type J struct {
Text string
}
func main() {
j := J{}
var i interface{} = j
js := "{\"Text\": \"lala\"}"
json.Unmarshal([]byte(js), &i)
fmt.Printf("%#v", i)
}
The type you're passing into Unmarshal is not *J, you're passing in an *interface{}.
When the json package reflects what the type is of the pointer it received, it sees interface{}, so it then uses the default types of the package to unmarshal into, which are
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
There is almost never a reason to use a pointer to an interface. If you find yourself using a pointer to an interface, and you don't know exactly why, then it's probably a mistake. If you want to unmarshal into J, then pass that in directly. If you need to assign that to an intermediary interface, make sure you use a pointer to the original value, not a pointer to its interface.
http://play.golang.org/p/uJDFKfSIxN
j := J{}
var i interface{} = &j
js := "{\"Text\": \"lala\"}"
json.Unmarshal([]byte(js), i)
fmt.Printf("%#v", i)
This is expected behavior: instead of giving json.Unmarshal a pointer to a properly typed place in memory you give it a pointer to a place in memory with type interface{}. It can essentially store anything in there under the type the JSON defines, so it does just that.
See it like this:
Unmarshal gets a place to store the data v with type interface{}
Unmarshal detects a map encoded as JSON
Unmarshal sees that the target type is of type interface{}, creates a go map from it and stores it in v
If you would have given it a different type than interface{} the process would have looked like this:
Unmarshal gets a place to store the data v with type struct main.J
Unmarshal detects a map encoded as JSON
Unmarshal sees that the target type is struct main.J and begins to recursively fit the data to the type
The main point is, that the initial assignment
var i interface{} = j
is completely ignored by json.Unmarshal.
Here is a Go Playground demonstrating my problem: http://play.golang.org/p/2fq3Fg7rPg
Essentially, I am trying to JSON marshal a struct containing a custom type wrapping json.RawMessage. When using CustomType.MarshalJSON() I get the expected results, but just calling json.Marshal on my full struct does not work as expected. See the playground link for a concrete example.
What is causing this difference?
Is there a way to have json.Marshal work as I expect it to?
Your code works fine, you just have one little bug.
// MarshalJSON returns the *j as the JSON encoding of j.
func (j JsonText) MarshalJSON() ([]byte, error) {
return j, nil
} // note i modified this so the receiver isn't a pointer
Your code didn't work because this is your definition of your datatype that wraps JsonText;
// Example struct I want to marshal and unmarshal
type TestData struct {
Field1 JsonText `json:"field_1"`
}
But only the *JsonText type implements the marshaler interface in your code. So you can change the types in either place ( I did in the MarshalJSON() ) but they need to be consistent.
In the playground; http://play.golang.org/p/NI_z3bQx7a
I have a struct like this:
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
But even if the instance of MyStruct is entirely empty (meaning, all values are default), it's being serialized as:
"data":{}
I know that the encoding/json docs specify that "empty" fields are:
false, 0, any nil pointer or interface value, and any array,
slice, map, or string of length zero
but with no consideration for a struct with all empty/default values. All of its fields are also tagged with omitempty, but this has no effect.
How can I get the JSON package to not marshal my field that is an empty struct?
As the docs say, "any nil pointer." -- make the struct a pointer. Pointers have obvious "empty" values: nil.
Fix - define the type with a struct pointer field:
type Result struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
Then a value like this:
result := Result{}
Will marshal as:
{}
Explanation: Notice the *MyStruct in our type definition. JSON serialization doesn't care whether it is a pointer or not -- that's a runtime detail. So making struct fields into pointers only has implications for compiling and runtime).
Just note that if you do change the field type from MyStruct to *MyStruct, you will need pointers to struct values to populate it, like so:
Data: &MyStruct{ /* values */ }
As #chakrit mentioned in a comment, you can't get this to work by implementing json.Marshaler on MyStruct, and implementing a custom JSON marshalling function on every struct that uses it can be a lot more work. It really depends on your use case as to whether it's worth the extra work or whether you're prepared to live with empty structs in your JSON, but here's the pattern I use applied to Result:
type Result struct {
Data MyStruct
Status string
Reason string
}
func (r Result) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}{
Data: &r.Data,
Status: r.Status,
Reason: r.Reason,
})
}
func (r *Result) UnmarshalJSON(b []byte) error {
decoded := new(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
})
err := json.Unmarshal(b, decoded)
if err == nil {
r.Data = decoded.Data
r.Status = decoded.Status
r.Reason = decoded.Reason
}
return err
}
If you have huge structs with many fields this can become tedious, especially changing a struct's implementation later, but short of rewriting the whole json package to suit your needs (not a good idea), this is pretty much the only way I can think of getting this done while still keeping a non-pointer MyStruct in there.
Also, you don't have to use inline structs, you can create named ones. I use LiteIDE with code completion though, so I prefer inline to avoid clutter.
Data is an initialized struct, so it isn't considered empty because encoding/json only looks at the immediate value, not the fields inside the struct.
Unfortunately, returning nil from json.Marshaler doesn't currently work:
func (_ MyStruct) MarshalJSON() ([]byte, error) {
if empty {
return nil, nil // unexpected end of JSON input
}
// ...
}
You could give Result a marshaler as well, but it's not worth the effort.
The only option, as Matt suggests, is to make Data a pointer and set the value to nil.
There is an outstanding Golang proposal for this feature which has been active for over 4 years, so at this point, it is safe to assume that it will not make it into the standard library anytime soon. As #Matt pointed out, the traditional approach is to convert the structs to pointers-to-structs. If this approach is infeasible (or impractical), then an alternative is to use an alternate json encoder which does support omitting zero value structs.
I created a mirror of the Golang json library (clarketm/json) with added support for omitting zero value structs when the omitempty tag is applied. This library detects zeroness in a similar manner to the popular YAML encoder go-yaml by recursively checking the public struct fields.
e.g.
$ go get -u "github.com/clarketm/json"
import (
"fmt"
"github.com/clarketm/json" // drop-in replacement for `encoding/json`
)
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
j, _ := json.Marshal(&Result{
Status: "204",
Reason: "No Content",
})
fmt.Println(string(j))
// Note: `data` is omitted from the resultant json.
{
"status": "204"
"reason": "No Content"
}