Getting Elasticsearch response - json

I am using gin framework to built an Api to get data from Elastic.
Issue is that when I run the application, I get response successfully in the first request but after that in any subsequent request I am getting error:
Error parsing the response body: EOF
Elastic configuration:
var cfg = elasticsearch.Config{
Password: GetConnectConfig().esPassword,
Username: GetConnectConfig().esUserName,
Addresses: GetConnectConfig().esHost,
Logger: &estransport.ColorLogger{
Output: os.Stdout,
}
My request handler function looks like:
func Search() gin.HandlerFunc {
client, err := elasticsearch.NewClient(cfg)
if err != nil {
log.Fatalf("elastic configuration failed %s", err)
}
res, err := client.Search(
client.Search.WithIndex(Index_Name),
client.Search.WithSize(10),
client.Search.WithPretty(),
)
if err != nil {
log.Fatalf("elastic failed to respond %s", err)
}
return func(c *gin.Context) {
r := map[string]interface{}{}
if err := json.NewDecoder(res.Body).Decode(&r); err != nil {
c.JSON(http.StatusInternalServerError, gin.H{"error": err})
log.Fatalf("Error parsing the response body: %s", err) // Error
} else {
for _, hit := range r["hits"].(map[string]interface{})["hits"].([]interface{}) {
log.Printf(" * ID=%s, %s", hit.(map[string]interface{})["_id"], hit.(map[string]interface{})["_source"])
}
c.JSON(http.StatusOK, "success")
}
}
}
As I told, I am able to get the response only in the very first request each time I run the application.
I am not sure what is the cause of error here.
I also tried with closing the response body after search by adding:
defer res.Body.Close()
but now I am getting following error:
Error parsing the response body: http2: response body closed

defer res.Body.Close() should be inside the func(c *gin.Context){} yo. can move your search initialization inside the func and do a close inside there something like and client can be injected as a dependency here.
return func(c *gin.Context) {
res, err := client.Search(
client.Search.WithIndex(Index_Name),
client.Search.WithSize(10),
client.Search.WithPretty(),
)
defer res.Body.Close()
}
}

Related

Go json.NewDecoder().Decode() doesn't seem to respect context deadline

I have a Golang program with a context deadline set. I am sending an HTTP request, and expected to see a deadline exceeded error when Im reading the body.
It seems that when I read the response body with ioutil.ReadAll then that read method will get interrupted (?) and return the appropriate error (context.DeadlineExceeded).
However if I read the response body with json.NewDecoder(resp.Body).Decode then the error returned is nil (instead of context.DeadlineExceeded). My full code is below. Is this a bug in json.NewDecoder(resp.Body).Decode?
package main
import (
"context"
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"time"
)
var url string = "http://ip.jsontest.com/"
func main() {
readDoesntFail()
readFails()
}
type IpResponse struct {
Ip string
}
func readDoesntFail() {
ctx, _ := context.WithTimeout(context.Background(), time.Second*5)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
panic(err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
ipResponse := new(IpResponse)
time.Sleep(time.Second * 6)
fmt.Println("before reading response body, context error is:", ctx.Err())
err = json.NewDecoder(resp.Body).Decode(ipResponse)
if err != nil {
panic(err)
}
fmt.Println("Expected panic but there was none")
}
func readFails() {
ctx, _ := context.WithTimeout(context.Background(), time.Second*5)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
panic(err)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
time.Sleep(time.Second * 6)
fmt.Println("before reading response body, context error is:", ctx.Err())
_, err = ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Println("received expected error", err)
}
}
The net/http package may use buffers to process requests. This means the incoming response body may be read and buffered partly or entirely before you read it, so an expiring context may not prevent you to finish reading the body. And this is exactly what happens.
Let's modify your example to fire up a test HTTP server which deliberately delays the response (partly):
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
s := []byte(`{"ip":"12.34.56.78"}`)
w.Write(s[:10])
if f, ok := w.(http.Flusher); ok {
f.Flush()
}
time.Sleep(time.Second * 6)
w.Write(s[10:])
}))
defer ts.Close()
url = ts.URL
readDoesntFail()
readFails()
This test server sends a similar JSON object to that of ip.jsontest.com's response. But it only sends 10 bytes body, then flushes it, then sleeps 6 seconds on purpose before sending the rest, "allowing" the client to time out.
Now let's see what happens if we call readDoesntFail():
before reading response body, context error is: context deadline exceeded
panic: Get "http://127.0.0.1:38230": context deadline exceeded
goroutine 1 [running]:
main.readDoesntFail()
/tmp/sandbox721114198/prog.go:46 +0x2b4
main.main()
/tmp/sandbox721114198/prog.go:28 +0x93
Try it on the Go Playground.
In your example json.Decoder.Decode() reads already buffered data, so the expired context plays no role here. In my example json.Decoder.Decode() tries to read from the connection because the data isn't yet buffered (it can't be as it hasn't been sent yet), so once the context expires, further reading from the connection returns a deadline exceeded error.

How can I send request payload data using a JSON file in GO?

I'm really new to coding and Golang itself.
I would like to know how can I send request Payload data using a JSON file in GO?
I mean, I have a post request and the JSON file and I would like to put it into the request body but I am coming across some errors.
The request is working when I use an alternative HTTP client.
Depending on the nature of the HTTP request, you may be able to use an existing client package. Eg, JSON RPC.
Here is an example if you would like to understand how to make a request using the standard library. This example also demonstrates using context to set timeouts for client requests:
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"time"
)
func main() {
ctx := context.Background()
var client http.Client
reqCtx, cancel := context.WithTimeout(ctx, time.Minute)
defer cancel()
err := deleteEntry(reqCtx, &client, 42)
fmt.Println(err)
}
func deleteEntry(ctx context.Context, client *http.Client, entryID int) error {
payload := &struct {
EntryID int `json:"entry_id"`
Method string `json:"method"`
}{
EntryID: entryID,
Method: "delete",
}
buf, err := json.Marshal(payload)
if err != nil {
return err
}
req, err := http.NewRequestWithContext(ctx, "POST", "http://localhost/example", bytes.NewReader(buf))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return err
}
// Note: Response body must always be closed.
// Response body data (if any) should be consumed before closure, otherwise the
// the client connection may not be reused.
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("request failed with %s", resp.Status)
}
return nil
}
I'd recommend reading through the net/http documentation to gain a better understanding. In particular:
http.Request
http.Response

Golang json marshal and encoding give weird output

I am trying to customise error message for my db query . Following is what I am doing first I create struct Errormessage . Next if there is error in db.query I do this marshaling then encoding and return. But I end up getting this output "e30=" on my postman testing. What could be wrong I check and followed few examples are showing this mechanism ?
error1 := Errormessage{"Error in select"}
error1_enc,errEn := json.Marshal(error1)
if errEn != nil {
// if error is not nil
// print error
fmt.Println(errEn)
}
json.NewEncoder(w).Encode(error1_enc)
return
/
/ declaring a struct
type Errormessage struct{
// defining struct variables
errormessage string
}
func checkExistUser(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
fmt.Println("File Name :", r.FormValue("email"))
result, err := db.Query("SELECT * from userDetailsss")
if err != nil {
//http.Error(w, err, 500)
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(400)
fmt.Println(err)
error1 := Errormessage{"Error in select"}
error1_enc,errEn := json.Marshal(error1)
if errEn != nil {
// if error is not nil
// print error
fmt.Println(errEn)
}
json.NewEncoder(w).Encode(error1_enc)
return
//panic(err.Error())
}
// This part is how my db is defined and opened
var db *sql.DB
var err error
func main() {
db, err = sql.Open("mysql", "******##tcp(127.0.0.1:3306)/****")
if err != nil {
panic(err.Error())
}
defer db.Close()
router := mux.NewRouter()
router.HandleFunc("/", DoHealthCheck).Methods("POST")
router.HandleFunc("/checkExistUser", checkExistUser).Methods("POST")
log.Fatal(http.ListenAndServe(":8080", router))
}
There are two issues with your code:
You are json encoding the already json encoded error. This means that you are json encoding raw json bytes, which is the reason for the weird output.
Your Errormessage struct's field is unexported. Unexported fields will not be encoded by the encoding/json package.
To fix #1 you can do:
func checkExistUser(w http.ResponseWriter, r *http.Request) {
r.ParseForm()
fmt.Println("File Name :", r.FormValue("email"))
result, err := db.Query("SELECT * from userDetailsss")
if err != nil {
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("X-Content-Type-Options", "nosniff")
w.WriteHeader(400)
// use only Encode, no need to call json.Marshal
if err := json.NewEncoder(w).Encode(Errormessage{"Error in select"}); err != nil {
log.Println("failed to send reposnse:", err)
}
return
}
// ...
}
To fix #2 you can do:
type Errormessage struct {
// export the field, i.e. change it to start with an upper case letter
Errormessage string `json:"errormessage"`
}

How to unmarshall JSON into a value created with reflection?

package controllers
import (
"encoding/json"
"errors"
"io"
"io/ioutil"
"reflect"
)
func GetTypeFromReq(c *App, ty interface{}) (interface{}, error) {
//get the type we are going to marshall into
item := reflect.ValueOf(ty)
//define and set the error that we will be returning to null
var retErr error
retErr = nil
//extract the body from the request and defer closing of the body
body, err := ioutil.ReadAll(io.LimitReader(c.Request.Body, 1048576))
defer c.Request.Body.Close()
//handle errors and unmarshal our data
if err != nil {
retErr = errors.New("Failed to Read body: " + err.Error())
} else if err = json.Unmarshal(body, &item); err != nil {
retErr = errors.New("Unmarshal Failed: " + err.Error())
}
return item, retErr
}
I am trying to pass a type and a request into a function, then inside that function unMarshall the request into a variable and return it.
I assume my approach is wrong because when i try to do this:
inter, err := GetTypeFromReq(&c, models.User{})
if err != nil {
revel.ERROR.Println(err.Error())
}
user := inter.(models.User)
I get the error "interface conversion: interface {} is reflect.Value, not models.User"
any tips on how to approach this?
Here's how to modify the the function to make it work as expected:
func GetTypeFromReq(c *App, ty interface{}) (interface{}, error) {
// Allocate new value with same type as ty
v := reflect.New(reflect.TypeOf(ty))
//define and set the error that we will be returning to null
var retErr error
retErr = nil
//extract the body from the request and defer closing of the body
body, err := ioutil.ReadAll(io.LimitReader(c.Request.Body, 1048576))
defer c.Request.Body.Close()
//handle errors and unmarshal our data
if err != nil {
retErr = errors.New("Failed to Read body: " + err.Error())
} else if err = json.Unmarshal(body, v.Interface()); err != nil {
retErr = errors.New("Unmarshal Failed: " + err.Error())
}
// v holds a pointer, call Elem() to get the value.
return v.Elem().Interface(), retErr
}
Note the calls to Interface() to get a reflect.Value's current value.
Here's an approach that avoids reflection and type assertions:
func GetFromReq(c *App, item interface{}) error {
//extract the body from the request and defer closing of the body
body, err := ioutil.ReadAll(io.LimitReader(c.Request.Body, 1048576))
defer c.Request.Body.Close()
//handle errors and unmarshal our data
if err != nil {
retErr = errors.New("Failed to Read body: " + err.Error())
} else if err = json.Unmarshal(body, item); err != nil {
retErr = errors.New("Unmarshal Failed: " + err.Error())
}
return retErr
}
Use it like this:
var user models.User
err := GetFromReq(&c, &user)
if err != nil {
revel.ERROR.Println(err.Error())
}
Use a JSON decoder to simplify the code:
func GetFromReq(c *App, item interface{}) error {
defer c.Request.Body.Close()
return json.NewDecoder(io.LimitReader(c.Request.Body, 1048576)).Deocode(item)
}
If c.Request is a *http.Request and c.Response is an http.ResponseWriter, then write the function as:
func GetFromReq(c *App, item interface{}) error {
return json.NewDecoder(http.MaxBytesReaer(c.Response, c.Request.Body, 1048576)).Deocode(item)
}
There's no need to close the request body in the net/http server. Use MaxBytesReader instead of io.LimitReader to prevents clients from accidentally or maliciously sending a large request and wasting server resources.
Modify code of the last line: change user := inter.(models.User) to user := inter.Interface().(models.User),have a try!
"interface conversion: interface {} is reflect.Value, not models.User"
pretty straight forward about the message error. That your item is reflect.Value it is not models.User.
so I think in your code you can change the item to models.User.
But I assume that your are tying to create the function that will work with all type of your models, in this case models.User{}.
Your approach is expensive since it is using interface. you could convert the incoming request directly like this:
func GetTypeFromReq(c *App, ty models.User) (models.User, error) {
//get the type we are going to marshall into
var item models.User
//define and set the error that we will be returning to nil
var retErr error // this var if the value not define then it is nil. Because error is interface
//extract the body from the request and defer closing of the body
body, err := ioutil.ReadAll(io.LimitReader(c.Request.Body, 1048576))
defer c.Request.Body.Close()
//handle errors and unmarshal our data
if err != nil {
retErr = errors.New("Failed to Read body: " + err.Error())
} else if err = json.Unmarshal(body, &item); err != nil {
retErr = errors.New("Unmarshal Failed: " + err.Error())
}
return item, retErr
}
if your body has the same structure as your model it will give you the value, if not then it is error.
Note that you need to be careful when using interface. you can see some guideline in this article. Use an interface:
When users of the API need to provide an implementation detail.
When API’s have multiple implementations they need to maintain internally.
When parts of the API that can change have been identified and require decoupling.
Your function convert the value of your models.User to interface, and then return the interface value. that's why it's expensive.

How to get JSON object by calling a url in Go Language?

I'm starting to learn Golang and I would like to know how to get a json response by calling an url, if you could give me an example it would be great in order to guide myself.
Here's a simple example to get you started. Instead of a map[string]interface{} you should consider making a struct to hold the result of your request.
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
)
func main() {
resp, err := http.Get("http://api.geonames.org/citiesJSON?north=44.1&south=-9.9&east=-22.4&west=55.2&lang=de&username=demo")
if err != nil {
log.Fatal(err)
}
var generic map[string]interface{}
err = json.NewDecoder(resp.Body).Decode(&generic)
if err != nil {
log.Fatal(err)
}
fmt.Println(generic)
}
I'd write a little helper function to do it:
// getJSON fetches the contents of the given URL
// and decodes it as JSON into the given result,
// which should be a pointer to the expected data.
func getJSON(url string, result interface{}) error {
resp, err := http.Get(url)
if err != nil {
return fmt.Errorf("cannot fetch URL %q: %v", url, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("unexpected http GET status: %s", resp.Status)
}
// We could check the resulting content type
// here if desired.
err := json.NewDecoder(resp.Body).Decode(result)
if err != nil {
return fmt.Errorf("cannot decode JSON: %v", err)
}
return nil
}
A full working example can be found here: http://play.golang.org/p/b1WJb7MbQV
Note that it is important to check the status code as well as the Get error, and the response body must be closed explicitly (see the documentation here: http://golang.org/pkg/net/http/#Get)