enc := json.NewEncoder(w)
err := enc.Encode(struct {
Method string `json:"method"`
Results []interface{} `json:"results"`
CacheTime int `json:"cache_time"`
}{Method: answerInlineQueryMethod, Results: results, CacheTime: 0})
if err != nil {
log.Printf("failed to answer to inline query: %s", err)
}
How can I distinguish between JSON errors, which should cause a panic and errors caused by sending the response, which should be logged?
The encoding/json package defines the error types it will return. For encoding you have MarshalerError,
UnsupportedTypeError,
and UnsupportedValueError.
You can inspect if the error type returned by Encode is one of these 3.
If the responses are not huge and don't rely on sending multiple json values per the json.Encoder protocol, you can simply use json.Marshal and write the response separately.
Related
Here is my minimal .proto file:
syntax = "proto3";
message getDhtParams {}
message DhtContents {
string dht_contents=1;
}
service MyApp {
rpc getDhtContent(getDhtParams) returns (DhtContents) {}
}
Two things to note related to the above proto file:
It is a minimal file. There is a lot more to it.
The server is already generated and running. The server is implemented in Python.
I am writing client in Go. And this is the fetching code I have come up with:
func fetchDht() (*pb.DhtContents, error) {
// Set up a connection to the server.
address := "localhost:9998"
conn, err := grpc.Dial(address, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
client := pb.NewMyAppClient(conn)
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
r, err := client.GetDhtContent(ctx, &pb.GetDhtParams{})
if err != nil {
return nil, errors.New("could not get dht contents")
}
return r, nil
}
For sake of simplicity, I have tripped down the output, but the output looks something like this:
dht_contents:"{'node_ids': ['dgxydhlqoopevxv'], 'peer_addrs': [['192.168.1.154', '41457']], 'peer_meta': [{'peer_id': {'nodeID': 'dgxydhlqoopevxv', 'key': 'kdlvjdictuvgxspwkdizqryr', 'mid': 'isocvavbtzkxeigkkrubzkcx', 'public_key': 'uhapwxnfeqqmnojsaijghhic', '_address': 'xklqlebqngpkxb'}, 'ip_addrs': ['192.168.1.154', '41457'], 'services': [{'service_input': '', 'service_output': '', 'price': 0}], 'timestamp': 1661319968}]}"
A few things to note about this response:
It starts with dht_contents: which I know is a field of DhtContents message.
This could be an issue from the server side; in that case I will inform the service developer. But the json enclosed is not a valid JSON as it uses single quotes.
My questions:
What is an elegant way to deal with that dht_contents? There must be the protobuf/grpc way. I aim to get the contents between double quotes.
How do I convert the content to JSON? I have already created the struct to unmarshal.
It would be enough if I am also able to convert the response which is of type *pb.DhtContents to []byte, from there I can convert it to JSON.
The generated code should have a method which will get rid of dht_contents:" from the start and " from the end.
In your case, that method should be called GetDhtContents().
You can modify your fetchDht function to something like this:
func fetchDht() (string, error) {
address := "localhost:9998"
// ...
if err != nil {
return nil, errors.New("could not get dht contents")
}
return r.GetDhtContents(), nil
}
From there on, you can work on making it a valid JSON by replacing single quotes to double quotes. Or it may be handled on the service end.
there is the methods generated by proto file to get the content from the result(the "r"), then use r.Get..., you could get the content.
convert string to the type you want.
suggest:
change proto type to bytes
then json.Unmarshal([r.Get...],[dst])
I am parsing rather simple json where the slash in the date format string is escaped when it arrives in response.
however, when you try to unmarshall the string, it fails on "invalid syntax" error.
So I googled and we should use strconv.Unquote to replace the escaped characters first. Did that, and now the unquote function fails on the "unknown escape" error. However JSON RFC 8259 shows it is a valid case. So why valid JSON is causing failures in Go unmarshaller?
This is the simple JSON
{
"created_at": "6\/30\/2022 21:51:49"
}
var dst string
err := json.Unmarshal([]byte("6\/30\/2022 21:51:49"), dst) // this fails on ivnalid quote
fmt.Println(dst, err)
s, err := strconv.Unquote(`"6\/30\/2022 21:51:49"`) // this fails on ivnalid syntax
fmt.Println(s, err)
How does one proceed with such cases when the JSON is valid, but none of the built-in functions of Go actually parse it? What is the right way, apart from writing simple find and replace pattern in json []byte manually?
EDIT:
ok I think I should have asked a better question or provided a specific example from the scenario rather than just a portion of it. Garbage in garbage out, my fault.
Here is the thing. Let's say we have the specific, non standard time format like in the above json example.
{
"created_at": "6\/30\/2022 21:51:49"
}
I need to parse it into the struct with go time type. Because the standard parser for time would fail for that format I create a custom type.
type Request struct {
CreatedAt Time `json:"created_at"`
}
type Time time.Time
unc (d *Time) UnmarshalJSON(b []byte) error {
s := strings.Trim(string(b), "\"")
parse, err := time.Parse("1/2/2006 15:04:05", s) // this fails because of the escaped backslashes
if err != nil {
return err
}
*d = Time(parse)
return nil
}
Here is an example in Playground: https://go.dev/play/p/bE3AWQeV-ug
panic: parsing time "6\\/30\\/2022 21:51:49" as "1/2/2006 15:04:05": cannot parse "\\/30\\/2022 21:51:49" as "/"
This is the reason I am trying to put the Unquote into the custom unmarshal function, but that is not the right approach as it was already pointed out. How does the default string json unmarshaller for example removes those escaped slashes? Oh and I have no control about the side that writes the json, it comes in with single escaped slashes like that in []byte from *http.Response
Go has a great built-in function to convert a JSON string to a Go string: json.Unmarshal. Here is how you can integrate it with a custom UnmarshalJSON method:
func (d *Time) UnmarshalJSON(b []byte) error {
var s string
if err := json.Unmarshal(b, &s); err != nil {
return err
}
// Remove this line: s := strings.Trim(string(b), "\"")
// The rest of the code is unchanged
Unquote is designed for cases when you have a double quoted string like "\"My name\"", you don't really need that in this case.
The problem with that code is that you're trying to unmarshal a byte slice that isn't JSON. You're also unmarshaling into an string which isn't going to work. You need to unmarshal into a struct that has json tags or unmarshal into a map.
var dest map[string]string
someJsonString := `{
"created_at": "6\/30\/2022 21:51:49"
}`
err := json.Unmarshal([]byte(someJsonString), &dest)
if err != nil {
panic(err)
}
fmt.Println(dest)
if you wanted to do Unmarshal into a struct, you can do it like so
type SomeStruct struct {
CreatedAt string `json:"created_at"`
}
someJsonString := `{
"created_at": "6\/30\/2022 21:51:49"
}`
var data SomeStruct
err := json.Unmarshal([]byte(someJsonString), &data)
if err != nil {
panic(err)
}
fmt.Println(data)
I have been writing in Go for a long time and recently while rewriting the code I came across a strange thing. I did a couple of tests and if request == nil check never worked. Previously, I was always afraid of getting a nil pointer exception, and so I inserted checks everywhere. But in this case, the json decode error handler seems to cover all cases.
var (
request *models.Owner
err error
)
err = json.NewDecoder(r.Body).Decode(&request)
if err != nil {
render.Render(w, r, response.ErrInvalidRequest(err))
return
}
if request == nil {
render.Render(w, r, response.ErrInvalidRequest("request is nil"))
return
}
if request == nil is it possible to catch this? Perhaps this check is unnecessary, and if I remove this check in my project, the code will become cleaner.
It is possible that nil error will be returned and request will still be nil, but only if the input JSON is the JSON null value.
For example:
type Owners struct {
Name string
}
var request *Owners
if err := json.Unmarshal([]byte("null"), &request); err != nil {
panic(err)
}
fmt.Println(request == nil)
fmt.Println(request)
This will output (try it on the Go Playground):
true
<nil>
This is documented at json.Unmarshal():
To unmarshal JSON into a pointer, Unmarshal first handles the case of the JSON being the JSON literal null. In that case, Unmarshal sets the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into the value pointed at by the pointer. If the pointer is nil, Unmarshal allocates a new value for it to point to.
I am using gin as my http server and sending back an empty array in json as my response:
c.JSON(http.StatusOK, []string{})
The resulting json string I get is "[]\n". The newline is added by the json Encoder object, see here.
Using goconvey, I could test my json like
So(response.Body.String(), ShouldEqual, "[]\n")
But is there a better way to generate the expected json string than just adding a newline to all of them?
You should first unmarshal the body of the response into a struct and compare against the resulting object. Example:
result := []string{}
if err := json.NewDecoder(response.Body).Decode(&result); err != nil {
log.Fatalln(err)
}
So(len(result), ShouldEqual, 0)
You may find jsonassert useful.
It has no dependencies outside the standard library and allows you to verify that JSON strings are semantically equivalent to a JSON string you expect.
In your case:
// white space is ignored, no need for \n
jsonassert.New(t).Assertf(response.Body().String(), "[]")
It can handle any form of JSON, and has very friendly assertion error messages.
Disclaimer: I wrote this package.
Unmarshal the body into a struct and the use Gocheck's DeepEquals
https://godoc.org/launchpad.net/gocheck
I made it this way. Because I don't want to include an extra library.
tc := testCase{
w: httptest.NewRecorder(),
wantResponse: mustJson(t, map[string]string{"message": "unauthorized"}),
}
...
if tc.wantResponse != tc.w.Body.String() {
t.Errorf("want %s, got %s", tt.wantResponse, tt.w.Body.String())
}
...
func mustJson(t *testing.T, v interface{}) string {
t.Helper()
out, err := json.Marshal(v)
if err != nil {
t.Fatal(err)
}
return string(out)
}
I'm developing a web service and part of that I read the Request.Body and try to unmarshal it.
if err := json.NewDecoder(body).Decode(r); err !=nil{
log.Error(err)
return err
}
The issue is that sometimes the client is sending an empty body and I get a panic runtime error: index out of range
goroutine 7 [running]:
How am I supposed to mitigate this?
I am decomposing your code:
NewDecoder: -
func NewDecoder(r io.Reader) *Decoder
NewDecoder returns a new decoder that reads from r. The decoder
introduces its own buffering and may read data from r beyond the JSON
values requested.
So NewDecoder only reads data from r. it's not care, is r empty...
Decode:-
func (dec *Decoder) Decode(v interface{}) error
Decode reads the next JSON-encoded value from its input and stores it
in the value pointed to by v.
See the documentation for Unmarshal for details about the conversion of JSON into a Go value.
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
If a JSON value is not appropriate for a given target type, or if a
JSON number overflows the target type, Unmarshal skips that field and
completes the unmarshalling as best it can. If no more serious errors
are encountered, Unmarshal returns an UnmarshalTypeError describing
the earliest such error.
The JSON null value unmarshals into an interface, map, pointer, or
slice by setting that Go value to nil. Because null is often used in
JSON to mean “not present,” unmarshaling a JSON null into any other Go
type has no effect on the value and produces no error.
Reading above statement, it's clear that there is not chances, we get run time panic error. I was experimenting with a sample code to reproduce this ERROR. May error coming from inside the JSON package or your own code.
var dummy []byte
dummy = make([]byte, 10)
size, _ := body.Read(dummy)
if size > 0 {
if err := json.NewDecoder(body).Decode(r); err != nil {
log.Error(err)
return err
}
fmt.Fprintf(w, "%s", "Json cannot be empty")// where w is http.ResponseWriter