I am writing a time alias to JSON Marshal some time into a Unix format (left some of my experimenting test code in there
type NotifyTime time.Time
// MarshalJSON implements marshaller interface. Marshals time.Time into unix time in bytes
func (t NotifyTime) MarshalJSON() ([]byte, error) {
// unixTime := time.Time(t).Unix()
unixTime := 1626132059 // unixTime := time.Now().Unix()
buffer := make([]byte, 8)
binary.PutVarint(buffer, int64(unixTime))
// binary.LittleEndian.PutUint64(buffer, uint64(unixTime))
// try to convert back
fmt.Println(string(buffer))
unixIntValue := int64(binary.LittleEndian.Uint64(buffer))
fmt.Println(unixIntValue)
return buffer, nil
}
When I run json.Marshal on an object with NotifyTime struct, it errs, with the following,
json: error calling MarshalJSON for type notify.NotifyTime: invalid character '¶' looking for beginning of value
type TestMe struct {
Time NotifyTime `json:"time"`
}
testJSON := TestMe{
Time: NotifyTime(time.Now()),
}
marshalled, err := json.Marshal(testJSON)
I have switched to marshalling it as a string, but still curious as to why this happens. When stepping through the code it seems to be because
on function compact on go/1.16.2/libexec/src/encoding/json/indent.go:17
it is looping over the marshalled bytes of the JSON
and the first (0th) byte fails the checks in
go/1.16.2/libexec/src/encoding/json/scanner.go:247
Let's put aside the aspect of encoding a time.Time and lets focus on how the int64 is being turned into JSON.
binary.PutVarint uses an encoding that is appropriate for low level wire or file formats. For the constant 1626132059, this writes into buffer [182 185 230 142 12 0 0 0]. The first character is 182 PILCROW SIGN in UTF-8. This is where '¶' comes from. You are getting an error like:
json: error calling MarshalJSON for type main.NotifyTime: invalid character '¶' looking for beginning of value
This is not the beginning of a valid JSON value. You will need to find an encoding of int64 that is a JSON value, such as a decimal number 1626132059 or a string of hexadecimal digits "60ecce5b".
In general you need to be careful putting binary string values into JSON as these can contain special characters that need to be escaped.
Related
I am sending this json:
{"origin":{"lat":23.589367061768648,"lng":58.42860314995051},"destination":null,"vehicle":"WATER_TANKER_600GL","scheduled_time":"2022-07-27T14:16:00Z04:00"}
Golang Gin handler function:
func (config *Config) CreateOrder(c *gin.Context) {
userID := auth.GetToken(c).ID()
data := &struct {
Origin models.Origin `json:"origin" binding:"required"`
Vehicle VehicleType `json:"vehicle" binding:"required"`
Destination *models.Destination `json:"destination" `
ScheduledTime *time.Time `json:"scheduled_time"`
}{}
if err := c.ShouldBindWith(data, binding.JSON); err != nil {
fmt.Println(err) // -> prints the below error
res.UnprocessableEntity(c, fmt.Sprintf("%s", err))
return
}
// rest of the code
}
I get this error:
parsing time "\"2022-07-27T14:16:00Z04:00\"" as "\"2006-01-02T15:04:05Z07:00\"": cannot parse "04:00\"" as "\""
Note that JSON does not define a date or timestamp data type. Transferring a timestamp as a JSON string is possible, but how that timestamp string is interpreted–or should be interpreted–is not written in stone. The standard lib (the encoding/json package) supports unmarshaling time.Time values from JSON strings, but it does so by requiring them to adhere the RFC 3339 standard, else the unmarshaling will fail.
time.Time implements json.Unmarshaler, and Time.UnmarshalJSON() states that the JSON string must be in RFC 3339 layout which is:
RFC3339 = "2006-01-02T15:04:05Z07:00"
The Z in the layout means:
Numeric time zone offsets format as follows:
"-0700" ±hhmm
"-07:00" ±hh:mm
"-07" ±hh
Replacing the sign in the format with a Z triggers the ISO 8601 behavior of printing Z instead of an offset for the UTC zone. Thus:
"Z0700" Z or ±hhmm
"Z07:00" Z or ±hh:mm
"Z07" Z or ±hh
Which means if the time has UTC offset, it should be
"2022-07-27T14:16:00Z"
If the time has +04:00 zone, it should be
"2022-07-27T14:16:00+04:00"
The input "2022-07-27T14:16:00Z04:00" is invalid for RFC 3339. Either change your source to provide RFC 3339 compliant timestamp strings, or if you can't or don't want to, you may create your custom type that implements json.Unmarshaler, and you handle this non-RFC 3339 format yourself.
Background
I'm working with JSON data that must be non-repudiable.
The API that grants me this data also has a service to verify that the data originally came from them.
As best as I can tell, they do this by requiring that the complete JSON they originally sent needs to be supplied to them inside another JSON request, with no byte changes.
Issue
I can't seem to preserve the original JSON!
Because I cannot modify the original JSON, I have carefully preserved it as a json.RawMessage when unmarshalling:
// struct I unmarshal my original data into
type SignedResult struct {
Raw json.RawMessage `json:"random"`
Signature string `json:"signature"`
...
}
// struct I marshal my data back into
type VerifiedSignatureReq {
Raw json.RawMessage `json:"random"`
Signature string `json:"signature"`
}
// ... getData is placeholder for function that gets my data
response := SignedResult{}
x, _ := json.Unmarshal(getData(), &response)
// do some post-processing with SignedResult that does not alter `Raw` or `Signature`
// trouble begins here - x.Raw started off as json.RawMessage...
y := json.Marshal(VerifiedSignatureReq{Raw: x.Raw, Signature: x.Signature}
// but now y.Raw is base64-encoded.
The problem is that []bytes / RawMessages are base64-encoded when marshaled. So I can't use this method, because it completely alters the string.
I'm unsure how to ensure this string is correctly preserved. I had assumed that the json.RawMessage specification in my struct would survive the perils of marshaling an already marshaled instance because it implements the Marshaler interface, but I appear mistaken.
Things I've Tried
My next attempt was to try:
// struct I unmarshal my original data into
type SignedResult struct {
Raw json.RawMessage `json:"random"`
Signature string `json:"signature"`
...
}
// struct I marshal my data back into
type VerifiedSignatureReq {
Raw map[string]interface{} `json:"random"`
Signature string `json:"signature"`
}
// ... getData is placeholder for function that gets my data
response := SignedResult{}
x, _ := json.Unmarshal(getData(), &response)
// do some post-processing with SignedResult that does not alter `Raw` or `Signature`
var object map[string]interface{}
json.Unmarshal(x.Raw, &object)
// now correctly generates the JSON structure.
y := json.Marshal(VerifiedSignatureReq{Raw: object, Signature: x.Signature}
// but now this is not the same JSON string as received!
The issue with this approach is that there are minor byte-wise differences in the spacing between the data. It no longer looks exactly the same when catted to a file.
I cannot use string(x.Raw) either because it escapes certain characters when marshaled with \.
You will need a custom type with its own marshaler, in place of json.RawMessage for your VerifiedSignatureReq struct to use. Example:
type VerifiedSignatureReq {
Raw RawMessage `json:"random"`
Signature string `json:"signature"`
}
type RawMessage []byte
func (m RawMessage) MarshalJSON() ([]byte, error) {
return []byte(m), nil
}
I have a big.float which I'm encoding into JSON . However the JSON always end up showing the float in scientific notation rater than decimal notation. I can fix this by changing the JSON to be a string rather than a number and using float.Text('f'), however I would really prefer to keep the type as a number.
I was a taking a look at float.Format but I don't believe this is suitable.
A really condensed gist of what I'm doing is below. I do a lot more modification of the value of supply before encoding it to json.
type TokenSupply struct {
TotalSupply *big.Float `json:"totalSupply, omitempty"`
}
supply := Float.NewFloat(1000000)
json.NewEncoder(w).Encode(TokenSupply{supply})
This returns:
{"totalSupply":"1e+06"}
big.Float is marshaled to string when converted to a JSON type
https://golang.org/pkg/encoding/json/#Marshal
Marshal traverses the value v recursively. If an encountered value implements the Marshaler interface and is not a nil pointer, Marshal calls its MarshalJSON method to produce JSON. If no MarshalJSON method is present but the value implements encoding.TextMarshaler instead, Marshal calls its MarshalText method and encodes the result as a JSON string. The nil pointer exception is not strictly necessary but mimics a similar, necessary exception in the behavior of UnmarshalJSON.
https://golang.org/pkg/math/big/#Float.MarshalText
func (x *Float) MarshalText() (text []byte, err error)
What can you do about it?
since your float may be more than 64 bits it won't play well with other languages that have to read the JSON value as a number. I'd suggest you keep the number as a string.
Caveats about encoding numbers that don't fit into 64 bits aside, here is how you could marshal a big.Float as a JSON number by wrapping it in a custom type that implements json.Marshaler. The key is that you can implement theMarshalJSON method anyway you like, as long as it emits valid JSON:
package main
import (
"encoding/json"
"fmt"
"math/big"
)
type BigFloatNumberJSON struct{ *big.Float }
func (bfn BigFloatNumberJSON) MarshalJSON() ([]byte, error) {
// Use big.Float.String() or any other string converter
// that emits a valid JSON number here...
return []byte(bfn.String()), nil
}
func main() {
totalSupply := new(big.Float).SetFloat64(1000000)
obj := map[string]interface{}{
"totalSupply": BigFloatNumberJSON{totalSupply},
}
bytes, err := json.Marshal(&obj)
if err != nil {
panic(err)
}
fmt.Println(string(bytes))
// => {"totalSupply":1000000}
}
When trying to parse a json with a float value for distance to the following struct
type CreateBookingRequest struct {
Distance float64 `json:"distance,string"`
DistanceSource string `json:"distanceSource"`
}
I get the following error
json: invalid use of ,string struct tag, trying to unmarshal unquoted
value into [34 100 105 115 116 97 110 99 101 34]%!(EXTRA
*reflect.rtype=dto.CreateBookingRequest)
Is there a way for me to avoid the error/get a better error message?
Edit:
I am actually expecting the users of the API to pass in a string value but if they for some reason pass in a non-string value, I would like to be able to tell them clearly, instead of this hard to read error message.
I had to work with an API which sometimes quotes numbers and sometimes doesn't. The owners of the service weren't likely to fix it, so I came up with a simple workaround:
re := regexp.MustCompile(`(":\s*)([\d\.]+)(\s*[,}])`)
rawJsonByteArray = re.ReplaceAll(rawJsonByteArray, []byte(`$1"$2"$3`))
Regular expressions are somewhat inefficient, but I don't believe I'd be able to implement something substantially faster.
func unmarshal state:
To unmarshal JSON into a struct, Unmarshal matches incoming object
keys to the keys used by Marshal (either the struct field name or its
tag), preferring an exact match but also accepting a case-insensitive
match.
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
So, unmarshal expecting Distance should be float64 by default. But as per tag, you are requesting unmarshal to except Distance as string. Here is data type missing matches.
So you have two options, either you change distance tag with float64 or marshal distance as string.
This error happens when the "distance" JSON value is encoded as a number instead of a string (per the "string" tag on the "Distance") field:
str := []byte(`{"distance":1.23,"distanceSource":"foo"}`)
// Note JSON number -------^
var cbr CreateBookingRequest
err := json.Unmarshal(str, &cbr)
// err => json: invalid use of ,string struct tag, trying to unmarshal unquoted value into [34 100 105 115 116 97 110 99 101 34]%!(EXTRA *reflect.rtype=main.CreateBookingRequest)
If you change the type of the distance value to a string (per the tag) then it works fine:
str := []byte(`{"distance":"1.23","distanceSource":"foo"}`)
// Note JSON string -------^
You could change the error message by identifying that specific error somehow and provide a different message. You might also consider changing the tag for the Distance type to simply accept a number instead of a string:
type CreateBookingRequest struct {
Distance float64 `json:"distance"`
...
}
...
str := []byte(`{"distance":1.23,"distanceSource":"foo"}`)
The error is simply saying you designated Distance as a string with your json annotations but in the json string you're trying to deserialize the value is not quoted (therefor not a string).
The solution is simple, either change this json:"distance,string" to json:"distance" or get json that matches your definition (meaning it has distince in quotes like "Distance":"10.4")
Given, the error and the fact that your native Go type is a float64 I would advise getting rid of the string annotation.
Another way to deal with this issue is to use json.Number. It will parse all numeric data into json.Number type, which is a string alias. Then you have to cast it:
package main
import (
"encoding/json"
"fmt"
)
type x struct {
Num json.Number `json:"price"`
}
func castToFloat64(num json.Number) (float64, error) {
return num.Float64()
}
func main() {
var resultHolder x
data := `{"price":"5"}`
jsonErr := json.Unmarshal([]byte(data), &resultHolder)
if jsonErr != nil {
fmt.Println(jsonErr)
}
convertedNum, convertErr := castToFloat64(resultHolder.Num)
if convertErr != nil {
fmt.Println(convertErr)
}
fmt.Println(convertedNum*2, resultHolder.Num+"extraString")
}
PlayGround
I'm developing a web service and part of that I read the Request.Body and try to unmarshal it.
if err := json.NewDecoder(body).Decode(r); err !=nil{
log.Error(err)
return err
}
The issue is that sometimes the client is sending an empty body and I get a panic runtime error: index out of range
goroutine 7 [running]:
How am I supposed to mitigate this?
I am decomposing your code:
NewDecoder: -
func NewDecoder(r io.Reader) *Decoder
NewDecoder returns a new decoder that reads from r. The decoder
introduces its own buffering and may read data from r beyond the JSON
values requested.
So NewDecoder only reads data from r. it's not care, is r empty...
Decode:-
func (dec *Decoder) Decode(v interface{}) error
Decode reads the next JSON-encoded value from its input and stores it
in the value pointed to by v.
See the documentation for Unmarshal for details about the conversion of JSON into a Go value.
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
If a JSON value is not appropriate for a given target type, or if a
JSON number overflows the target type, Unmarshal skips that field and
completes the unmarshalling as best it can. If no more serious errors
are encountered, Unmarshal returns an UnmarshalTypeError describing
the earliest such error.
The JSON null value unmarshals into an interface, map, pointer, or
slice by setting that Go value to nil. Because null is often used in
JSON to mean “not present,” unmarshaling a JSON null into any other Go
type has no effect on the value and produces no error.
Reading above statement, it's clear that there is not chances, we get run time panic error. I was experimenting with a sample code to reproduce this ERROR. May error coming from inside the JSON package or your own code.
var dummy []byte
dummy = make([]byte, 10)
size, _ := body.Read(dummy)
if size > 0 {
if err := json.NewDecoder(body).Decode(r); err != nil {
log.Error(err)
return err
}
fmt.Fprintf(w, "%s", "Json cannot be empty")// where w is http.ResponseWriter