I am using gin as my http server and sending back an empty array in json as my response:
c.JSON(http.StatusOK, []string{})
The resulting json string I get is "[]\n". The newline is added by the json Encoder object, see here.
Using goconvey, I could test my json like
So(response.Body.String(), ShouldEqual, "[]\n")
But is there a better way to generate the expected json string than just adding a newline to all of them?
You should first unmarshal the body of the response into a struct and compare against the resulting object. Example:
result := []string{}
if err := json.NewDecoder(response.Body).Decode(&result); err != nil {
log.Fatalln(err)
}
So(len(result), ShouldEqual, 0)
You may find jsonassert useful.
It has no dependencies outside the standard library and allows you to verify that JSON strings are semantically equivalent to a JSON string you expect.
In your case:
// white space is ignored, no need for \n
jsonassert.New(t).Assertf(response.Body().String(), "[]")
It can handle any form of JSON, and has very friendly assertion error messages.
Disclaimer: I wrote this package.
Unmarshal the body into a struct and the use Gocheck's DeepEquals
https://godoc.org/launchpad.net/gocheck
I made it this way. Because I don't want to include an extra library.
tc := testCase{
w: httptest.NewRecorder(),
wantResponse: mustJson(t, map[string]string{"message": "unauthorized"}),
}
...
if tc.wantResponse != tc.w.Body.String() {
t.Errorf("want %s, got %s", tt.wantResponse, tt.w.Body.String())
}
...
func mustJson(t *testing.T, v interface{}) string {
t.Helper()
out, err := json.Marshal(v)
if err != nil {
t.Fatal(err)
}
return string(out)
}
Related
Here is my minimal .proto file:
syntax = "proto3";
message getDhtParams {}
message DhtContents {
string dht_contents=1;
}
service MyApp {
rpc getDhtContent(getDhtParams) returns (DhtContents) {}
}
Two things to note related to the above proto file:
It is a minimal file. There is a lot more to it.
The server is already generated and running. The server is implemented in Python.
I am writing client in Go. And this is the fetching code I have come up with:
func fetchDht() (*pb.DhtContents, error) {
// Set up a connection to the server.
address := "localhost:9998"
conn, err := grpc.Dial(address, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
client := pb.NewMyAppClient(conn)
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
r, err := client.GetDhtContent(ctx, &pb.GetDhtParams{})
if err != nil {
return nil, errors.New("could not get dht contents")
}
return r, nil
}
For sake of simplicity, I have tripped down the output, but the output looks something like this:
dht_contents:"{'node_ids': ['dgxydhlqoopevxv'], 'peer_addrs': [['192.168.1.154', '41457']], 'peer_meta': [{'peer_id': {'nodeID': 'dgxydhlqoopevxv', 'key': 'kdlvjdictuvgxspwkdizqryr', 'mid': 'isocvavbtzkxeigkkrubzkcx', 'public_key': 'uhapwxnfeqqmnojsaijghhic', '_address': 'xklqlebqngpkxb'}, 'ip_addrs': ['192.168.1.154', '41457'], 'services': [{'service_input': '', 'service_output': '', 'price': 0}], 'timestamp': 1661319968}]}"
A few things to note about this response:
It starts with dht_contents: which I know is a field of DhtContents message.
This could be an issue from the server side; in that case I will inform the service developer. But the json enclosed is not a valid JSON as it uses single quotes.
My questions:
What is an elegant way to deal with that dht_contents? There must be the protobuf/grpc way. I aim to get the contents between double quotes.
How do I convert the content to JSON? I have already created the struct to unmarshal.
It would be enough if I am also able to convert the response which is of type *pb.DhtContents to []byte, from there I can convert it to JSON.
The generated code should have a method which will get rid of dht_contents:" from the start and " from the end.
In your case, that method should be called GetDhtContents().
You can modify your fetchDht function to something like this:
func fetchDht() (string, error) {
address := "localhost:9998"
// ...
if err != nil {
return nil, errors.New("could not get dht contents")
}
return r.GetDhtContents(), nil
}
From there on, you can work on making it a valid JSON by replacing single quotes to double quotes. Or it may be handled on the service end.
there is the methods generated by proto file to get the content from the result(the "r"), then use r.Get..., you could get the content.
convert string to the type you want.
suggest:
change proto type to bytes
then json.Unmarshal([r.Get...],[dst])
I am parsing rather simple json where the slash in the date format string is escaped when it arrives in response.
however, when you try to unmarshall the string, it fails on "invalid syntax" error.
So I googled and we should use strconv.Unquote to replace the escaped characters first. Did that, and now the unquote function fails on the "unknown escape" error. However JSON RFC 8259 shows it is a valid case. So why valid JSON is causing failures in Go unmarshaller?
This is the simple JSON
{
"created_at": "6\/30\/2022 21:51:49"
}
var dst string
err := json.Unmarshal([]byte("6\/30\/2022 21:51:49"), dst) // this fails on ivnalid quote
fmt.Println(dst, err)
s, err := strconv.Unquote(`"6\/30\/2022 21:51:49"`) // this fails on ivnalid syntax
fmt.Println(s, err)
How does one proceed with such cases when the JSON is valid, but none of the built-in functions of Go actually parse it? What is the right way, apart from writing simple find and replace pattern in json []byte manually?
EDIT:
ok I think I should have asked a better question or provided a specific example from the scenario rather than just a portion of it. Garbage in garbage out, my fault.
Here is the thing. Let's say we have the specific, non standard time format like in the above json example.
{
"created_at": "6\/30\/2022 21:51:49"
}
I need to parse it into the struct with go time type. Because the standard parser for time would fail for that format I create a custom type.
type Request struct {
CreatedAt Time `json:"created_at"`
}
type Time time.Time
unc (d *Time) UnmarshalJSON(b []byte) error {
s := strings.Trim(string(b), "\"")
parse, err := time.Parse("1/2/2006 15:04:05", s) // this fails because of the escaped backslashes
if err != nil {
return err
}
*d = Time(parse)
return nil
}
Here is an example in Playground: https://go.dev/play/p/bE3AWQeV-ug
panic: parsing time "6\\/30\\/2022 21:51:49" as "1/2/2006 15:04:05": cannot parse "\\/30\\/2022 21:51:49" as "/"
This is the reason I am trying to put the Unquote into the custom unmarshal function, but that is not the right approach as it was already pointed out. How does the default string json unmarshaller for example removes those escaped slashes? Oh and I have no control about the side that writes the json, it comes in with single escaped slashes like that in []byte from *http.Response
Go has a great built-in function to convert a JSON string to a Go string: json.Unmarshal. Here is how you can integrate it with a custom UnmarshalJSON method:
func (d *Time) UnmarshalJSON(b []byte) error {
var s string
if err := json.Unmarshal(b, &s); err != nil {
return err
}
// Remove this line: s := strings.Trim(string(b), "\"")
// The rest of the code is unchanged
Unquote is designed for cases when you have a double quoted string like "\"My name\"", you don't really need that in this case.
The problem with that code is that you're trying to unmarshal a byte slice that isn't JSON. You're also unmarshaling into an string which isn't going to work. You need to unmarshal into a struct that has json tags or unmarshal into a map.
var dest map[string]string
someJsonString := `{
"created_at": "6\/30\/2022 21:51:49"
}`
err := json.Unmarshal([]byte(someJsonString), &dest)
if err != nil {
panic(err)
}
fmt.Println(dest)
if you wanted to do Unmarshal into a struct, you can do it like so
type SomeStruct struct {
CreatedAt string `json:"created_at"`
}
someJsonString := `{
"created_at": "6\/30\/2022 21:51:49"
}`
var data SomeStruct
err := json.Unmarshal([]byte(someJsonString), &data)
if err != nil {
panic(err)
}
fmt.Println(data)
Using Go, I'm attempting to fetch a few JSON responses concurrently from multiple endpoints. I'd like to attach each of these responses to fields in a struct or map and return this struct/map as a JSON object. (Backend for Frontend pattern). So I will make a web request to the Go application with some sort of identifier. It will in turn make several web requests, and compile the data into one large object to return as a response.
I'm using Fiber as my framework but any generic web framework would be similar:
app.Get("/requests/:identifier", func(c *fiber.Ctx) error {
identifier := c.Params("identifier")
timeout := 1600 * time.Millisecond
client := httpclient.NewClient(httpclient.WithHTTPTimeout(timeout))
res, err := client.Get("https://www.example.com/endpoint?id=" + identifier, nil)
if err != nil{
logger.Error("Timout value exceeded")
return c.Status(503).SendString("Socket Timeout")
}
logger.Info("Fetch success: ")
// Heimdall returns the standard *http.Response object
body, err := ioutil.ReadAll(res.Body)
code := 200
response := &main_response{
Service1: body,
}
return c.Status(code).JSON(response)
})
The problem I'm having is, I have no need to unmarshal this data in Go, as I have no use for it (I am simply passing it along). Do I have to unmarshal it just so I can set it as a field in my response struct like this?
type main_response struct {
Service1 []byte `json:"service1"`
Service2 map[string]string `json:"service2"`
Service3 map[string]interface{} `json:"service3"`
}
(I've tried a few different ways to accomplish this. Trying to use a byte array seems to base64 encode the responses)
I will want to marshal that struct to JSON before returning it, so perhaps I have little choice as I can't think of a way to tell Go "only marshal the main struct, everything else is already JSON". It almost feels like I'd be better off building a string at this point.
Use json.RawMessage to copy a []byte containing JSON directly to the response JSON document:
type main_response struct {
Service1 json.RawMessage `json:"service1"`
...
}
response := &main_response{
Service1: body,
}
return c.Status(code).JSON(response)
I am using Google's go-jsonnet library to evaluate some jsonnet files.
I have a function, like so, which renders a Jsonnet document:
// Takes a list of jsonnet files and imports each one and mixes them with "+"
func renderJsonnet(files []string, param string, prune bool) string {
// empty slice
jsonnetPaths := files[:0]
// range through the files
for _, s := range files {
jsonnetPaths = append(jsonnetPaths, fmt.Sprintf("(import '%s')", s))
}
// Create a JSonnet VM
vm := jsonnet.MakeVM()
// Join the slices into a jsonnet compat string
jsonnetImport := strings.Join(jsonnetPaths, "+")
if param != "" {
jsonnetImport = "(" + jsonnetImport + ")" + param
}
if prune {
// wrap in std.prune, to remove nulls, empty arrays and hashes
jsonnetImport = "std.prune(" + jsonnetImport + ")"
}
// render the jsonnet
out, err := vm.EvaluateSnippet("file", jsonnetImport)
if err != nil {
log.Panic("Error evaluating jsonnet snippet: ", err)
}
return out
}
This function currently returns a string, because the jsonnet EvaluateSnippet function returns a string.
What I now want to do is render that result JSON using the go-prettyjson library. However, because the JSON i'm piping in is a string, it's not rendering correctly.
So, some questions:
Can I convert the returned JSON string to a JSON type, without knowing beforehand what struct to marshal it into
if not, can I render the json in a pretty manner some other way?
Is there an option, function or method I'm missing here to make this easier?
Can I convert the returned JSON string to a JSON type, without knowing beforehand what struct to marshal it into
Yes. It's very easy:
var jsonOut interface{}
err := json.Unmarshal([]byte(out), &jsonOut)
if err != nil {
log.Panic("Invalid json returned by jsonnet: ", err)
}
formatted, err := prettyjson.Marshal([]byte(jsonOut))
if err != nil {
log.Panic("Failed to format jsonnet output: ", err)
}
More info here: https://blog.golang.org/json-and-go#TOC_5.
Is there an option, function or method I'm missing here to make this easier?
Yes. The go-prettyjson library has a Format function which does the unmarshalling for you:
formatted, err := prettyjson.Format([]byte(out))
if err != nil {
log.Panic("Failed to format jsonnet output: ", err)
}
can I render the json in a pretty manner some other way?
Depends on your definition of pretty. Jsonnet normally outputs every field of an object and every array element on a separate line. This is usually considered pretty printing (as opposed to putting everything on the same line with minimal whitespace to save a few bytes). I suppose this is not good enough for you. You can write your own manifester in jsonnet which formats it to your liking (see std.manifestJson as an example).
enc := json.NewEncoder(w)
err := enc.Encode(struct {
Method string `json:"method"`
Results []interface{} `json:"results"`
CacheTime int `json:"cache_time"`
}{Method: answerInlineQueryMethod, Results: results, CacheTime: 0})
if err != nil {
log.Printf("failed to answer to inline query: %s", err)
}
How can I distinguish between JSON errors, which should cause a panic and errors caused by sending the response, which should be logged?
The encoding/json package defines the error types it will return. For encoding you have MarshalerError,
UnsupportedTypeError,
and UnsupportedValueError.
You can inspect if the error type returned by Encode is one of these 3.
If the responses are not huge and don't rely on sending multiple json values per the json.Encoder protocol, you can simply use json.Marshal and write the response separately.