The question How to marshal JSON with bigints? is about marshaling big.Int values into strings in JSON. This question asks, how does one marshal and unmarshal big.Int values natively as numbers in JSON?
Passing around large values marshaled in this manner may be incompatible with other implementations of JSON, particularly JavaScript and jq, as RFC 7159 notes:
Note that when such software is used, numbers that are integers and
are in the range [-(2**53)+1, (2**53)-1] are interoperable in the
sense that implementations will agree exactly on their numeric
values.
Create a custom type BigInt which implements json.Marshaler and json.Unmarshaler like the following:
import (
"fmt"
"math/big"
)
type BigInt struct {
big.Int
}
func (b BigInt) MarshalJSON() ([]byte, error) {
return []byte(b.String()), nil
}
func (b *BigInt) UnmarshalJSON(p []byte) error {
if string(p) == "null" {
return nil
}
var z big.Int
_, ok := z.SetString(string(p), 10)
if !ok {
return fmt.Errorf("not a valid big integer: %s", p)
}
b.Int = z
return nil
}
Rationale:
Implemented as a struct type embedding big.Int instead of as a subtype of big.Int so that the methods (Add, SetString, etc.) are kept
MarshalJSON takes a value receiver so that marshaled values using BigInt don't have to use pointers
UnmarshalJSON takes a pointer receiver because it modifies the receiver; however, types using BigInt still don't have to use pointers
UnmarshalJSON works on a temporary value because the SetString method has undefined behavior on its receiver in the case of errors
As with big.Int the zero value is useful and is equal to the number 0.
Related
I have a json that contains a field with a bigint
{"NETWORK_ID": 6000370005980500000071}
The format I have before marshaling is map[string]interface{}
When I marshal it and print to console everything seems to be fine but this field actually creates problems due to its size in other mediums so I want to serialize it as a string.
UseNumber() seems to be for this purpose but it's only for decoding I think.
Is there any way that I can detect this kind of bigint number and make them serialize as strings?
You'll need to create a custom type that implements the json.Marshaler interface, and marshals to a string. Example:
type MyBigInt big.Int
func (i MyBigInt) MarshalJSON() ([]byte, error) {
i2 := big.Int(i)
return []byte(fmt.Sprintf(`"%s"`, i2.String()), nil
}
This will always marshal your custom type as a quoted decimal number.
I have a big.float which I'm encoding into JSON . However the JSON always end up showing the float in scientific notation rater than decimal notation. I can fix this by changing the JSON to be a string rather than a number and using float.Text('f'), however I would really prefer to keep the type as a number.
I was a taking a look at float.Format but I don't believe this is suitable.
A really condensed gist of what I'm doing is below. I do a lot more modification of the value of supply before encoding it to json.
type TokenSupply struct {
TotalSupply *big.Float `json:"totalSupply, omitempty"`
}
supply := Float.NewFloat(1000000)
json.NewEncoder(w).Encode(TokenSupply{supply})
This returns:
{"totalSupply":"1e+06"}
big.Float is marshaled to string when converted to a JSON type
https://golang.org/pkg/encoding/json/#Marshal
Marshal traverses the value v recursively. If an encountered value implements the Marshaler interface and is not a nil pointer, Marshal calls its MarshalJSON method to produce JSON. If no MarshalJSON method is present but the value implements encoding.TextMarshaler instead, Marshal calls its MarshalText method and encodes the result as a JSON string. The nil pointer exception is not strictly necessary but mimics a similar, necessary exception in the behavior of UnmarshalJSON.
https://golang.org/pkg/math/big/#Float.MarshalText
func (x *Float) MarshalText() (text []byte, err error)
What can you do about it?
since your float may be more than 64 bits it won't play well with other languages that have to read the JSON value as a number. I'd suggest you keep the number as a string.
Caveats about encoding numbers that don't fit into 64 bits aside, here is how you could marshal a big.Float as a JSON number by wrapping it in a custom type that implements json.Marshaler. The key is that you can implement theMarshalJSON method anyway you like, as long as it emits valid JSON:
package main
import (
"encoding/json"
"fmt"
"math/big"
)
type BigFloatNumberJSON struct{ *big.Float }
func (bfn BigFloatNumberJSON) MarshalJSON() ([]byte, error) {
// Use big.Float.String() or any other string converter
// that emits a valid JSON number here...
return []byte(bfn.String()), nil
}
func main() {
totalSupply := new(big.Float).SetFloat64(1000000)
obj := map[string]interface{}{
"totalSupply": BigFloatNumberJSON{totalSupply},
}
bytes, err := json.Marshal(&obj)
if err != nil {
panic(err)
}
fmt.Println(string(bytes))
// => {"totalSupply":1000000}
}
I got a bug in Go when using an interface{} as function parameter type, when given a non-pointer type, and using json.Unmarshal with it.
Because a piece of code is worth a thousand words, here is an example:
package main
import (
"encoding/json"
"fmt"
)
func test(i interface{}) {
j := []byte(`{ "foo": "bar" }`)
fmt.Printf("%T\n", i)
fmt.Printf("%T\n", &i)
json.Unmarshal(j, &i)
fmt.Printf("%T\n", i)
}
type Test struct {
Foo string
}
func main() {
test(Test{})
}
Which outputs:
main.Test
*interface {}
map[string]interface {}
json.Unmarshal turns my struct to a map[string]interface{} oO...
Little readings later explains some of it, interface{} is a type in itself, not some sort of typeless container, which explains the *interface{}, and the fact that json.Unmarshal could not get the initial type, and returned a map[string]interface{}..
From Unmarshal docs:
To unmarshal JSON into an interface value, Unmarshal stores one of
these in the interface value:
[...]
And if I pass a pointer to the test function like so, it works:
func test(i interface{}) {
j := []byte(`{ "foo": "bar" }`)
fmt.Printf("%T\n", i)
fmt.Printf("%T\n", &i)
json.Unmarshal(j, i)
fmt.Printf("%T\n", i)
fmt.Println(i)
}
func main() {
test(&Test{})
}
Which outputs:
*main.Test
*interface {}
*main.Test
&{bar}
Cool, the data is unmarshalled and all, now in this second snippet I removed the & when calling Unmarshal. Because I have a *Test in i, no use for it.
So in all logic, if I put back the & to i when calling Unmarshal it should mess up with i's type again. But no.
If I run:
func test(i interface{}) {
j := []byte(`{ "foo": "bar" }`)
fmt.Printf("%T\n", i)
fmt.Printf("%T\n", &i)
json.Unmarshal(j, &i)
fmt.Printf("%T\n", i)
fmt.Println(i)
}
func main() {
test(&Test{})
}
Well it still works:
*main.Test
*interface {}
*main.Test
&{bar}
And now I'm out of google search queries.
The right scenario
interface{} is a wrapper for any value and of any type. An interface schematically wraps a (value; type) pair, a concrete value and its type. More details on this: The Laws of Reflection #The representation of an interface.
json.Unmarshal() already takes the value of type interface{}:
func Unmarshal(data []byte, v interface{}) error
So if you already have an interface{} value (the i interface{} parameter of the test() function), don't try to take its address, just pass it along as-is.
Also note that for any package to modify a value stored in an interface{}, you need to pass a pointer to it. So what should be in i is a pointer. So the right scenario is to pass *Test to test(), and inside test() pass i to json.Unmarshal() (without taking its address).
Explanation of other scenarios
When i contains *Test and you pass &i, it will work because the json package will simply dereference the *interface{} pointer, and finds an interface{} value, which wraps a *Test value. It's a pointer, so it's all good: unmarshals the JSON object into the pointed Test value.
When i contains Test and you pass &i, same thing goes as above: *interface{} is dereferenced, so it finds an interface{} which contains a non-pointer: Test. Since the json package can't unmarshal into a non-pointer value, it has to create a new value. And since the passed value to the json.Unmarshal() function is of type *interface{}, it tells the json package to unmarshal the data into a value of type interface{}. This means the json package is free to choose which type to use. And by default the json package unmarshals JSON objects into map[string]interface{} values, so that is what's created and used (and eventually put into the value pointed by the pointer you passed: &i).
All in all
All in all, avoid using pointers to interfaces. Instead "put" pointers into the interfaces (the interface value should wrap the pointer). When you already have an interface{} holding a pointer, just pass it along.
I want to decode a big set of data from a (static-schema) json file. The file contains exclusively numeric data, and keys are all integers. I know how to decode this json into a struct containing fields of map[string]int or map[string]float32 using json.Unmarshal. But I have no interest in string keys, I'd need to convert them to int somehow.
So what I'd like to know is:
Is there a way to achieve this, .ie getting a struct containing fields of map[int]float32 type directly from decoding,
Otherwise how to achieve this after decoding, in a memory efficient manner ?
Thanks
The JSON format only specifies key/value objects with string keys. Because of this, the encoding/json package only supports string keys as well.
The json/encoding documentation states:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
If you want to use encoding/json package and move it over to a map[int]float64, you can do the following (works with float32 as well):
package main
import (
"fmt"
"strconv"
)
func main() {
a := map[string]float64{"1":1, "2":4, "3":9, "5":25}
b := make(map[int]float64, len(a))
for k,v := range a {
if i, err := strconv.Atoi(k); err == nil {
b[i] = v
} else {
// A non integer key
}
}
fmt.Printf("%#v\n", b)
}
Playground
The encoding/json package includes an interface Unmarshaler which has a single method: UnmarshalJSON(data []byte) error.
If you're feeling brave you could implement that for the following:
type IntToFloat map[int]float32
func (itf *IntToFloat) UnmarshalJSON(data []byte) error {
if itf == nil {
return errors.New("Unmarshalling JSON for a null IntToFload")
}
// MAGIC Goes here.
return nil
}
EDIT
Note: http://golang.org/src/pkg/encoding/json/decode.go?s=2221:2269#L53 is where the std library version of Unmarshal comes from.
http://golang.org/pkg/encoding/json/#Unmarshaler is where the interface referenced above comes from.
I have a struct like this:
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
But even if the instance of MyStruct is entirely empty (meaning, all values are default), it's being serialized as:
"data":{}
I know that the encoding/json docs specify that "empty" fields are:
false, 0, any nil pointer or interface value, and any array,
slice, map, or string of length zero
but with no consideration for a struct with all empty/default values. All of its fields are also tagged with omitempty, but this has no effect.
How can I get the JSON package to not marshal my field that is an empty struct?
As the docs say, "any nil pointer." -- make the struct a pointer. Pointers have obvious "empty" values: nil.
Fix - define the type with a struct pointer field:
type Result struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
Then a value like this:
result := Result{}
Will marshal as:
{}
Explanation: Notice the *MyStruct in our type definition. JSON serialization doesn't care whether it is a pointer or not -- that's a runtime detail. So making struct fields into pointers only has implications for compiling and runtime).
Just note that if you do change the field type from MyStruct to *MyStruct, you will need pointers to struct values to populate it, like so:
Data: &MyStruct{ /* values */ }
As #chakrit mentioned in a comment, you can't get this to work by implementing json.Marshaler on MyStruct, and implementing a custom JSON marshalling function on every struct that uses it can be a lot more work. It really depends on your use case as to whether it's worth the extra work or whether you're prepared to live with empty structs in your JSON, but here's the pattern I use applied to Result:
type Result struct {
Data MyStruct
Status string
Reason string
}
func (r Result) MarshalJSON() ([]byte, error) {
return json.Marshal(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}{
Data: &r.Data,
Status: r.Status,
Reason: r.Reason,
})
}
func (r *Result) UnmarshalJSON(b []byte) error {
decoded := new(struct {
Data *MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
})
err := json.Unmarshal(b, decoded)
if err == nil {
r.Data = decoded.Data
r.Status = decoded.Status
r.Reason = decoded.Reason
}
return err
}
If you have huge structs with many fields this can become tedious, especially changing a struct's implementation later, but short of rewriting the whole json package to suit your needs (not a good idea), this is pretty much the only way I can think of getting this done while still keeping a non-pointer MyStruct in there.
Also, you don't have to use inline structs, you can create named ones. I use LiteIDE with code completion though, so I prefer inline to avoid clutter.
Data is an initialized struct, so it isn't considered empty because encoding/json only looks at the immediate value, not the fields inside the struct.
Unfortunately, returning nil from json.Marshaler doesn't currently work:
func (_ MyStruct) MarshalJSON() ([]byte, error) {
if empty {
return nil, nil // unexpected end of JSON input
}
// ...
}
You could give Result a marshaler as well, but it's not worth the effort.
The only option, as Matt suggests, is to make Data a pointer and set the value to nil.
There is an outstanding Golang proposal for this feature which has been active for over 4 years, so at this point, it is safe to assume that it will not make it into the standard library anytime soon. As #Matt pointed out, the traditional approach is to convert the structs to pointers-to-structs. If this approach is infeasible (or impractical), then an alternative is to use an alternate json encoder which does support omitting zero value structs.
I created a mirror of the Golang json library (clarketm/json) with added support for omitting zero value structs when the omitempty tag is applied. This library detects zeroness in a similar manner to the popular YAML encoder go-yaml by recursively checking the public struct fields.
e.g.
$ go get -u "github.com/clarketm/json"
import (
"fmt"
"github.com/clarketm/json" // drop-in replacement for `encoding/json`
)
type Result struct {
Data MyStruct `json:"data,omitempty"`
Status string `json:"status,omitempty"`
Reason string `json:"reason,omitempty"`
}
j, _ := json.Marshal(&Result{
Status: "204",
Reason: "No Content",
})
fmt.Println(string(j))
// Note: `data` is omitted from the resultant json.
{
"status": "204"
"reason": "No Content"
}