I want to convert bson in mongo-go-driver to json effectively.
I should take care to handle NaN, because json.Marshal fail if NaN exists in data.
For instance, I want to convert below bson data to json.
b, _ := bson.Marshal(bson.M{"a": []interface{}{math.NaN(), 0, 1}})
// How to convert b to json?
The below fails.
// decode
var decodedBson bson.M
bson.Unmarshal(b, &decodedBson)
_, err := json.Marshal(decodedBson)
if err != nil {
panic(err) // it will be invoked
// panic: json: unsupported value: NaN
}
If you know the structure of your BSON, you can create a custom type that implements the json.Marshaler and json.Unmarshaler interfaces, and handles NaN as you wish. For example:
type maybeNaN struct{
isNan bool
number float64
}
func (n maybeNaN) MarshalJSON() ([]byte, error) {
if n.isNan {
return []byte("null"), nil // Or whatever you want here
}
return json.Marshal(n.number)
}
func (n *maybeNan) UnmarshalJSON(p []byte) error {
if string(p) == "NaN" {
n.isNan = true
return nil
}
return json.Unmarshal(p, &n.number)
}
type myStruct struct {
someNumber maybeNaN `json:"someNumber" bson:"someNumber"`
/* ... */
}
If you have an arbitrary structure of your BSON, your only option is to traverse the structure, using reflection, and convert any occurrences of NaN into a type (possibly a custom type as described above)
Related
Welcome all.
Tell me how to get a flat array in GO.
That is, conditionally, I have a structure without keys of the form:
type DashboardHeatMapStruct struct {
float64
string
}
Next, I give it in response to rest in the form of JSON and get output of the form:
[[0,"#AEAEAE"],[0.01,"#0e00ff"],[0.65,"#00ffcf"],[0.7,"#00ffcf"],[0.75,"#00ff9c"],[0.8,"#00ff0a"],[0.85,"#b3ff00"],[0.9,"#ffdc00"],[0.95,"#ff6d00"],[1,"#c60000"]]
Declare a struct type to represent the elements of the JSON array.
type DashboardHeatMapStruct struct {
t float64
c string
}
Implement the json.Unmarshaler interface on that type:
func (d *DashboardHeatMapStruct) UnmarshalJSON(p []byte) error {
// p is expected to be JSON array with float and
// string values. Create slice to match.
v := []interface{}{&d.t, &d.c}
// Unmarshal to JSON array to the slice. The JSON decoder
// follows the pointers in the slice to set the struct members.
return json.Unmarshal(p, &v)
}
Implement the json.Marshler interface to encode back to JSON.
func (d DashboardHeatMapStruct) MarshalJSON() ([]byte, error) {
v := []interface{}{d.t, d.c}
return json.Marshal(v)
}
Unmarshal to a slice of DashboardHeatMapStruct:
var result []DashboardHeatMapStruct
if err := json.Unmarshal(data, &result); err != nil {
// handle error
}
Run it on the playground.
I got the following problem:
My golang program converts some information into JSON.
For example it results in the following json:
{
"value":40,
"unit":"some_string"
}
The problem is the "input" for value is 40.0 and the marshalling strips the trailing zero. It would be no problem if the EPL which reads the JSON would be able to read 40 as float without the .0
So the JSON output should look like:
{
"value":40.0,
"unit":"some_string"
}
Is there a possibility to "stop" json.Marshal() from removing the zero?
Edit: Value must be a Float
#icza provided a good answer, but just to offer another option, you can define your own float type and define your own serialization for it. Like this
type KeepZero float64
func (f KeepZero) MarshalJSON() ([]byte, error) {
if float64(f) == float64(int(f)) {
return []byte(strconv.FormatFloat(float64(f), 'f', 1, 32)), nil
}
return []byte(strconv.FormatFloat(float64(f), 'f', -1, 32)), nil
}
type Pt struct {
Value KeepZero
Unit string
}
func main() {
data, err := json.Marshal(Pt{40.0, "some_string"})
fmt.Println(string(data), err)
}
This results in {"Value":40.0,"Unit":"some_string"} <nil>. Check it out in playground.
By default floating point numbers are rendered without a decimal point and fractions if its value is an integer value. The representation is shorter, and it means the same number.
If you want control over how a number appears in the JSON representation, use the json.Number type.
Example:
type Pt struct {
Value json.Number
Unit string
}
func main() {
data, err := json.Marshal(Pt{json.Number("40.0"), "some_string"})
fmt.Println(string(data), err)
}
Output (try it on the Go Playground):
{"Value":40.0,"Unit":"some_string"} <nil>
If you have a number as a float64 value, you may convert it to json.Number like this:
func toNumber(f float64) json.Number {
var s string
if f == float64(int64(f)) {
s = fmt.Sprintf("%.1f", f) // 1 decimal if integer
} else {
s = fmt.Sprint(f)
}
return json.Number(s)
}
Testing it:
f := 40.0
data, err := json.Marshal(Pt{toNumber(f), "some_string"})
fmt.Println(string(data), err)
f = 40.123
data, err = json.Marshal(Pt{toNumber(f), "some_string"})
fmt.Println(string(data), err)
Output (try it on the Go Playground):
{"Value":40.0,"Unit":"some_string"} <nil>
{"Value":40.123,"Unit":"some_string"} <nil>
The other direction, if you want the float64 value of a json.Number, simply call its Number.Float64() method.
I had a similar issue where I wanted to marshal a map[string]interface{} with float values f.x 1.0 to JSON as 1.0. I solved it by adding a custom Marshal function for a custom float type and then replace the floats in the map with the custom type:
type customFloat float64
func (f customFloat) MarshalJSON() ([]byte, error) {
if float64(f) == math.Trunc(float64(f)) {
return []byte(fmt.Sprintf("%.1f", f)), nil
}
return json.Marshal(float64(f))
}
func replaceFloat(value map[string]interface{}) {
for k, v := range value {
switch val := v.(type) {
case map[string]interface{}:
replaceFloat(val)
case float64:
value[k] = customFloat(val)
}
}
}
Then replace all float64 nodes:
replaceFloat(myValue)
bytes, err := json.Marshal(myValue)
This will print the floats like 1.0
type MyFloat float64
func (mf MyFloat) MarshalJSON() ([]byte, error) {
return []byte(fmt.Sprintf("%.1f", float64(mf))), nil
}
used like this
type PricePoint struct {
Price MyFloat `json:"price"`
From time.Time `json:"valid_from"`
To time.Time `json:"valid_to"`
}
Replace the 1 in "%.1f" with what ever precision you need
Save the value as a string and cast it back if you need it.
I can't understand a strange behavior of custom marshal int to string.
Here is an example:
package main
import (
"encoding/json"
"fmt"
)
type Int int
func (a Int) MarshalJSON() ([]byte, error) {
test := a / 10
return json.Marshal(fmt.Sprintf("%d-%d", a, test))
}
func main() {
array := []Int{100, 200}
arrayJson, _ := json.Marshal(array)
fmt.Println("array", string(arrayJson))
maps := map[Int]bool{
100: true,
200: true,
}
mapsJson, _ := json.Marshal(maps)
fmt.Println("map wtf?", string(mapsJson))
fmt.Println("map must be:", `{"100-10":true, "200-20":true}`)
}
The output is:
array ["100-10","200-20"]
map wtf? {"100":true,"200":true}
map must be: {"100-10":true, "200-20":true}
https://play.golang.org/p/iiUyL2Hc5h_P
What am I missing?
This is the expected outcome, which is documented at json.Marshal():
Map values encode as JSON objects. The map's key type must either be a string, an integer type, or implement encoding.TextMarshaler. The map keys are sorted and used as JSON object keys by applying the following rules, subject to the UTF-8 coercion described for string values above:
- string keys are used directly
- encoding.TextMarshalers are marshaled
- integer keys are converted to strings
Note that map keys are handled differently than values of properties because map keys in JSON are the property names which are always string values (while property values may be JSON text, number and boolean values).
As per the doc, if you want it to work for map keys as well, implement encoding.TextMarshaler:
func (a Int) MarshalText() (text []byte, err error) {
test := a / 10
return []byte(fmt.Sprintf("%d-%d", a, test)), nil
}
(Note that MarshalText() is ought to return "just" simple text, not JSON text, hence we omit JSON marshaling in it!)
With this, output will be (try it on the Go Playground):
array ["100-10","200-20"] <nil>
map wtf? {"100-10":true,"200-20":true} <nil>
map must be: {"100-10":true, "200-20":true}
Note that encoding.TextMarshaler is enough as that is also checked when marsaling as values, not just for map keys. So you don't have to implement both encoding.TextMarshaler and json.Marshaler.
If you do implement both, you can have different output when the value is marshaled as a "simple" value and as a map key because json.Marshaler takes precedence when generating a value:
func (a Int) MarshalJSON() ([]byte, error) {
test := a / 100
return json.Marshal(fmt.Sprintf("%d-%d", a, test))
}
func (a Int) MarshalText() (text []byte, err error) {
test := a / 10
return []byte(fmt.Sprintf("%d-%d", a, test)), nil
}
This time the output will be (try it on the Go Playground):
array ["100-1","200-2"] <nil>
map wtf? {"100-10":true,"200-20":true} <nil>
map must be: {"100-10":true, "200-20":true}
The accepted answer is great but I've had to re-search for this enough times that I wanted to put a complete answer regarding marshal/unmarshal with examples so next time I can just copy paste as a starting point :)
The things I often search for include:
encode custom type to sql database
json encode enum int as string
json encode map key but not value
In this example I make a custom Weekday type which matches time.Weekday int values but allows for the string value in request/response json and in the database
This same thing can be done with any int enum using iota to have the human readable value in json and database
Playground example with tests: https://go.dev/play/p/aUxxIJ6tY9K
The important bit is here though:
var (
// read/write from/to json values
_ json.Marshaler = (*Weekday)(nil)
_ json.Unmarshaler = (*Weekday)(nil)
// read/write from/to json keys
_ encoding.TextMarshaler = (*Weekday)(nil)
_ encoding.TextUnmarshaler = (*Weekday)(nil)
// read/write from/to sql
_ sql.Scanner = (*Weekday)(nil)
_ driver.Valuer = (*Weekday)(nil)
)
// MarshalJSON marshals the enum as a quoted json string
func (w Weekday) MarshalJSON() ([]byte, error) {
return []byte(`"` + w.String() + `"`), nil
}
func (w Weekday) MarshalText() (text []byte, err error) {
return []byte(w.String()), nil
}
func (w *Weekday) UnmarshalJSON(b []byte) error {
return w.UnmarshalText(b)
}
func (w *Weekday) UnmarshalText(b []byte) error {
var dayName string
if err := json.Unmarshal(b, &dayName); err != nil {
return err
}
d, err := ParseWeekday(dayName)
if err != nil {
return err
}
*w = d
return nil
}
// Value is used for sql exec to persist this type as a string
func (w Weekday) Value() (driver.Value, error) {
return w.String(), nil
}
// Scan implements sql.Scanner so that Scan will be scanned correctly from storage
func (w *Weekday) Scan(src interface{}) error {
switch t := src.(type) {
case int:
*w = Weekday(t)
case int64:
*w = Weekday(int(t))
case string:
d, err := ParseWeekday(t)
if err != nil {
return err
}
*w = d
case []byte:
d, err := ParseWeekday(string(t))
if err != nil {
return err
}
*w = d
default:
return errors.New("Weekday.Scan requires a string or byte array")
}
return nil
}
Note that the var block simply forces you to implement the methods correctly else it won't compile.
Also note that if you exclude MarshalJSON then go will use MarshalText if it is there, so if you want only the key to have a custom marshal but to have the default behavior for the value then you should not use these methods on your main type, but instead have a wrapper type that you only use for map keys
type MyType struct{}
type MyTypeKey MyType
var (
// read/write from/to json keys
_ encoding.TextMarshaler = (*MyTypeKey)(nil)
_ encoding.TextUnmarshaler = (*MyTypeKey)(nil)
)
func (w MyTypeKey) MarshalText() (text []byte, err error) {
return []byte(w.String()), nil
}
func (w *MyTypeKey) UnmarshalText(b []byte) error {
*w = MyTypeKey(ParseMyType(string(b)))
return nil
}
Feel free to improve this answer, I hope others find it helpful and I hope I can find it again next time I want this and search for it again myself :)
Golang encoding/json package lets you use ,string struct tag in order to marshal/unmarshal string values (like "309230") into int64 field. Example:
Int64String int64 `json:",string"`
However, this doesn't work for slices, ie. []int64:
Int64Slice []int64 `json:",string"` // Doesn't work.
Is there any way to marshal/unmarshal JSON string arrays into []int64 field?
Quote from https://golang.org/pkg/encoding/json:
The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types. This extra level of encoding is sometimes used when communicating with JavaScript programs:
For anyone interested, I found a solution using a custom type having MarshalJSON() and UnmarshalJSON() methods defined.
type Int64StringSlice []int64
func (slice Int64StringSlice) MarshalJSON() ([]byte, error) {
values := make([]string, len(slice))
for i, value := range []int64(slice) {
values[i] = fmt.Sprintf(`"%v"`, value)
}
return []byte(fmt.Sprintf("[%v]", strings.Join(values, ","))), nil
}
func (slice *Int64StringSlice) UnmarshalJSON(b []byte) error {
// Try array of strings first.
var values []string
err := json.Unmarshal(b, &values)
if err != nil {
// Fall back to array of integers:
var values []int64
if err := json.Unmarshal(b, &values); err != nil {
return err
}
*slice = values
return nil
}
*slice = make([]int64, len(values))
for i, value := range values {
value, err := strconv.ParseInt(value, 10, 64)
if err != nil {
return err
}
(*slice)[i] = value
}
return nil
}
The above solution marshals []int64 into JSON string array. Unmarshaling works from both JSON string and integer arrays, ie.:
{"bars": ["1729382256910270462", "309286902808622", "23"]}
{"bars": [1729382256910270462, 309286902808622, 23]}
See example at https://play.golang.org/p/BOqUBGR3DXm
As you quoted from json.Marshal(), the ,string option only applies to specific types, namely:
The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types.
You want it to work with a slice, but that is not supported by the json package.
If you still want this functionality, you have to write your custom marshaling / unmarshaling logic.
What you presented works, but it is unnecessarily complex. This is because you created your custom logic on slices, but you only want this functionality on individual elements of the slices (arrays). You don't want to change how an array / slice (as a sequence of elements) is rendered or parsed.
So a much simpler solution is to only create a custom "number" type producing this behavior, and elements of slices of this custom type will behave the same.
Our custom number type and the marshaling / unmarshaling logic:
type Int64Str int64
func (i Int64Str) MarshalJSON() ([]byte, error) {
return json.Marshal(strconv.FormatInt(int64(i), 10))
}
func (i *Int64Str) UnmarshalJSON(b []byte) error {
// Try string first
var s string
if err := json.Unmarshal(b, &s); err == nil {
value, err := strconv.ParseInt(s, 10, 64)
if err != nil {
return err
}
*i = Int64Str(value)
return nil
}
// Fallback to number
return json.Unmarshal(b, (*int64)(i))
}
And that's all!
The type using it:
type Foo struct {
Bars []Int64Str `json:"bars"`
}
Testing it the same way as you did yields the same result. Try it on the Go Playground.
When I make an HTTP call to a REST API I may get the JSON value count back as a Number or String. I'ld like to marshal it to be an integer in either case. How can I deal with this in Go?.
Use the "string" field tag option to specify that strings should be converted to numbers. The documentation for the option is:
The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types. This extra level of encoding is sometimes used when communicating with JavaScript programs:
Here's an example use:
type S struct {
Count int `json:"count,string"`
}
playground example
If the JSON value can be number or string, then unmarshal to interface{} and convert to int after unmarshaling:
Count interface{} `json:"count,string"`
Use this function to convert the interface{} value to an int:
func getInt(v interface{}) (int, error) {
switch v := v.(type) {
case float64:
return int(v), nil
case string:
c, err := strconv.Atoi(v)
if err != nil {
return 0, err
}
return c, nil
default:
return 0, fmt.Errorf("conversion to int from %T not supported", v)
}
}
// Format of your expected request
type request struct {
ACTIVE string `json:"active"`
CATEGORY string `json:"category"`
}
// struct to read JSON input
var myReq request
// Decode the received JSON request to struct
decoder := json.NewDecoder(r.Body)
err := decoder.Decode(&myReq)
if err != nil {
log.Println( err)
// Handler for invalid JSON received or if you want to decode the request using another struct with int.
return
}
defer r.Body.Close()
// Convert string to int
numActive, err = strconv.Atoi(myReq.ACTIVE)
if err != nil {
log.Println(err)
// Handler for invalid int received
return
}
// Convert string to int
numCategory, err = strconv.Atoi(myReq.CATEGORY)
if err != nil {
log.Println(err)
// Handler for invalid int received
return
}
I had the same problem with a list of values where the values were string or struct. The solution I'm using is to create a helper struct with fields of expected types and parse value into the correct field.
type Flag struct {
ID string `json:"id"`
Type string `json:"type"`
}
type FlagOrString struct {
Flag *Flag
String *string
}
func (f *FlagOrString) UnmarshalJSON(b []byte) error {
start := []byte("\"")
for idx := range start {
if b[idx] != start[idx] {
return json.Unmarshal(b, &f.Flag)
}
}
return json.Unmarshal(b, &f.String)
}
var MainStruct struct {
Vals []FlagOrString
}
Custom Unmarshaller simplifies a code. Personally I prefer this over interface{} as it explicitly states what a developer expects.