I read this solution for resolve body data from a proxy.
Golang: how to read response body of ReverseProxy?
But I cannot read the body as a plain string, maybe the encoding is not right or other cryption.
My question is how to encode or transform the body to readable html string?
Currently I get:
n8�������♠�♠�A��b:J���g>-��ˤ���[���.....
Code example:
reverseProxy := httputil.NewSingleHostReverseProxy(url)
reverseProxy.ModifyResponse = func (resp *http.Response) (err error) {
var contentType = resp.Header.Get("Content-Type")
if strings.HasPrefix(contentType, "text/html") {
b, err := ioutil.ReadAll(resp.Body) //Read html
if err != nil {
return err
}
err = resp.Body.Close()
if err != nil {
return err
}
body := ioutil.NopCloser(bytes.NewReader(b))
resp.Body = body
resp.ContentLength = int64(len(b))
log.Printf(string(b))
}
return nil
}
Related
I wrote some code which hits one public API and saves the JSON output in a file. But the data is storing line by line into the file instead of a single JSON format.
For eg.
Current Output:
{"ip":"1.1.1.1", "Country":"US"}
{"ip":"8.8.8.8", "Country":"IN"}
Desired Output:
[
{"ip":"1.1.1.1", "Country":"US"},
{"ip":"8.8.8.8", "Country":"IN"}
]
I know this should be pretty simple and i am missing out something.
My Current Code is:
To read IP from file and hit the API one by one on each IP.
func readIPfromFile(filename string, outFile string, timeout int) {
data := jsonIn{}
//open input file
jsonFile, err := os.Open(filename) //open input file
...
...
jsonData := bufio.NewScanner(jsonFile)
for jsonData.Scan() {
// marshal json data & check for logs
if err := json.Unmarshal(jsonData.Bytes(), &data); err != nil {
log.Fatal(err)
}
//save to file
url := fmt.Sprintf("http://ipinfo.io/%s", data.Host)
GetGeoIP(url, outFile, timeout)
}
}
To make HTTP Request with custom request header and call write to file function.
func GetGeoIP(url string, outFile string, timeout int) {
geoClient := http.Client{
Timeout: time.Second * time.Duration(timeout), // Timeout after 5 seconds
}
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
log.Fatal(err)
}
req.Header.Set("accept", "application/json")
res, getErr := geoClient.Do(req)
if getErr != nil {
log.Fatal(getErr)
}
if res.Body != nil {
defer res.Body.Close()
}
body, readErr := ioutil.ReadAll(res.Body)
if readErr != nil {
log.Fatal(readErr)
}
jsonout := jsonOut{}
jsonErr := json.Unmarshal(body, &jsonout)
if jsonErr != nil {
log.Fatal(jsonErr)
}
file, _ := json.Marshal(jsonout)
write2file(outFile, file)
}
To Write data to file:
func write2file(outFile string, file []byte) {
f, err := os.OpenFile(outFile, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
if err != nil {
log.Fatal(err)
}
defer f.Close()
if _, err = f.WriteString(string(file)); err != nil {
log.Fatal(err)
}
if _, err = f.WriteString("\n"); err != nil {
log.Fatal(err)
}
I know, i can edit f.WriteString("\n"); to f.WriteString(","); to add comma but still adding [] in the file is challenging for me.
First, please do not invent a new way of json marshaling, just use golang built-in encoding/json or other library on github.
Second, if you want to create a json string that represents an array of object, you need to create the array of objects in golang and marshal it into string (or more precisely, into array of bytes)
I create a simple as below, but please DIY if possible.
https://go.dev/play/p/RR_ok-fUTb_4
I've seen some go code that looks like this :
type userForm struct {
Name string `json:"name" validate:"min=2"`
Surname string `json:"surname" validate:"min=2"`
Phone string `json:"phone" validate:"min=10"`
Email string `json:"email,omitempty" validate:"omitempty,email"`
}
type emailForm struct {
Email string `json:"email" validate:"email"`
}
func decodeUserForm(r *http.Request) (userForm, error) {
var body userForm
d := json.NewDecoder(r.Body)
if err := d.Decode(&body); err != nil {
return body, NewHTTPError(err, 400, "unable to decode user form")
}
return body, validateStruct(body)
}
func decodeEmailForm(r *http.Request) (emailForm, error) {
var body emailForm
d := json.NewDecoder(r.Body)
if err := d.Decode(&body); err != nil {
return body, NewHTTPError(err, 400, "unable to decode email form")
}
return body, validateStruct(body)
}
I find two functions redundant. Is there a simpler way to merge those two into a more generic function? Is it good practice in Go?
func decodeForm(r *http.Request, dst interface{}) error {
if err := json.NewDecoder(r.Body).Decode(dst); err != nil {
return NewHTTPError(err, 400, "unable to decode email form")
}
return validateStruct(dst)
}
Then use it as so:
var body emailForm
if err := decodeForm(r, &body); err != nil {
panic(err)
}
I'm getting JSON resonse from an external API with the following way:
func Request(url string, contentType string) []byte {
resp, err := http.Get(url)
resp.Header.Set("Content-Type", contentType)
if err != nil {
log.Fatal(err)
}
body, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
log.Fatal(err)
}
return body
}
url := fmt.Sprintf("https://example.com/api/category/%s", category)
contentType := "application/json"
body := Request(url, contentType)
res := &JSONRespStruct{}
err := json.Unmarshal([]byte(body), res)
if err != nil {
log.Fatal(err)
}
The problem if I start to benchmark my site with go-wrk, the server crashes with the following error message:
2018/01/02 12:13:35 invalid character '<' looking for beginning of value
I think the code try to parse the JSON response as HTML. How I can force to get the response as a JSON?
You probably want to set the header on the request. Setting the header on the response has no impact.
func Request(url string, contentType string) []byte {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
log.Fatal(err)
}
req.Header.Set("Content-Type", contentType)
resp, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatal(err)
}
return body
}
I'm building microservices app with go and beego. I'm trying to pass JSON response from service A to service B as following:
func (u *ServiceController) GetAll() {
req := httplib.Get("http://localhost/api/1/services")
str, err := req.String()
// str = {"id":1, "name":"some service"}
if err != nil {
fmt.Println(err)
}
u.Data["json"] = str
u.ServeJSON()
}
However, when I send the response I actually double json encoding:
"{\"id\":\"1\",\"name\":\"some service\"}"
Finally, this is the solution I came up with:
func (u *ServiceController) GetAll() {
req := httplib.Get("http://localhost/api/1/services")
str, err := req.String()
if err != nil {
fmt.Println(err)
}
strToByte := []byte(str)
u.Ctx.Output.Header("Content-Type", "application/json")
u.Ctx.Output.Body(strToByte)
}
Try this:
func (u *ServiceController) GetAll() {
req := httplib.Get("http://localhost/api/1/services")
str, err := req.Bytes()
// str = {"id":1, "name":"some service"}
if err != nil {
fmt.Println(err)
}
u.Ctx.Output.Header("Content-Type", "text/plain;charset=UTF-8")
u.Ctx.ResponseWriter.Write(str)
}
If you call req.String(), it will encode the " in the json string. I suggest you use []byte to handle data usually.
In the following http handler, I try to distinguish whether the request body is empty
type Request struct {
A bool `json:"lala"`
B bool `json:"kaka"`
C int32 `json:"cc"`
D int32 `json:"dd"`
}
var (
opts Request
hasOpts bool = true
)
err = json.NewDecoder(r.Body).Decode(&opts)
switch {
case err == io.EOF:
hasOpts = false
case err != nil:
return errors.New("Could not get advanced options: " + err.Error())
}
However, even with r.Body equals '{}', hasOpts is still true. Is this to be expected? In that case, how should I detect empty request body?
Read the body first, to check its content, then unmarshal it:
body, err := ioutil.ReadAll(r.Body)
if err != nil {
return err
}
if len(body) > 0 {
err = json.Unmarshal(body, &opts)
if err != nil {
return fmt.Errorf("Could not get advanced options: %s", err)
}
}