I have been working on a golang script that uses the chrome devtools protocol to:
1) Intercept a request
2) Grab the response body for the intercepted request
3) Make some modifications to the html document
4) Continue the intercepted request
The script works for HTML documents except when Content-Encoding is set to gzip. The step-by-step process looks like this"
1) Intercept Request
s.Debugger.CallbackEvent("Network.requestIntercepted", func(params godet.Params) {
iid := params.String("interceptionId")
rtype := params.String("resourceType")
reason := responses[rtype]
headers := getHeadersString(params["responseHeaders"])
log.Println("[+] Request intercepted for", iid, rtype, params.Map("request")["url"])
if reason != "" {
log.Println(" abort with reason", reason)
}
// Alter HTML in request response
if s.Options.AlterDocument && rtype == "Document" && iid != "" {
res, err := s.Debugger.GetResponseBodyForInterception(iid)
if err != nil {
log.Println("[-] Unable to get intercepted response body!")
}
rawAlteredResponse, err := AlterDocument(res, headers)
if err != nil{
log.Println("[-] Unable to alter HTML")
}
if rawAlteredResponse != "" {
log.Println("[+] Sending modified body")
err := s.Debugger.ContinueInterceptedRequest(iid, godet.ErrorReason(reason), rawAlteredResponse, "", "", "", nil)
if err != nil {
fmt.Println("OH NOES AN ERROR!")
log.Println(err)
}
}
} else {
s.Debugger.ContinueInterceptedRequest(iid, godet.ErrorReason(reason), "", "", "", "", nil)
}
})
2) Alter the response body
Here I am making small changes to the HTML markup in procesHtml() (but the code for that function is not relevant to this issue, so will not post it here). I also grab headers from the request and when necessary update the content-length and date before continue the reponse. Then, I gzip compress the body when calling r := gZipCompress([]byte(alteredBody), which returns a string. The string is then concatenated to the headers so I can craft the rawResponse.
func AlterDocument(debuggerResponse []byte, headers map[string]string) (string, error) {
alteredBody, err := processHtml(debuggerResponse)
if err != nil {
return "", err
}
alteredHeader := ""
for k, v := range headers{
switch strings.ToLower(k) {
case "content-length":
v = strconv.Itoa(len(alteredBody))
fmt.Println("Updating content-length to: " + strconv.Itoa(len(alteredBody)))
break
case "date":
v = fmt.Sprintf("%s", time.Now().Format(time.RFC3339))
break
}
alteredHeader += k + ": " + v + "\r\n"
}
r := gZipCompress([]byte(alteredBody))
rawAlteredResponse :=
base64.StdEncoding.EncodeToString([]byte("HTTP/1.1 200 OK" + "\r\n" + alteredHeader + "\r\n\r\n\r\n" + r))
return rawAlteredResponse, nil
}
Note: I am now gzip compressing the body for all responses. The above is temporary while I figure out how to solve this issue.
The gzip compress function looks like this:
func gZipCompress(dataToWorkWith []byte) string{
var b bytes.Buffer
gz, err := gzip.NewWriterLevel(&b, 5)
if err != nil{
panic(err)
}
if _, err := gz.Write(dataToWorkWith); err != nil {
panic(err)
}
if err := gz.Flush(); err != nil {
panic(err)
}
if err := gz.Close(); err != nil {
panic(err)
}
return b.String()
}
As seen in the first code snippet, the response body and headers are set here:
err := s.Debugger.ContinueInterceptedRequest(iid, godet.ErrorReason(reason), rawAlteredResponse, "", "", "", nil)
The result is a bunch of garbled characters in the browser. This works without the gzip functions for non gzipped requests. I have changed the compression level as well (without success). Am I processing the body in the wrong order (string > []byte > gzip > string > base64)? Should this be done in a different order to work? Any help would be immensely appreciated.
The response looks like this, which Chrome puts inside a <body></body> tag
����rܸ� ��_A��Q%GH��Kʔ��vU�˷c�v�}
or in the response:
I can also tell that it is compressing correctly as, when I remove headers, the request results in a .gz file download with all the correct .html when uncompressed. Additionally, the first few bytes in the object returned in gZipCompress tell me that it is gzipped correctly:
31 139 8
or
0x1f 0x8B 0x08
I ended up using a different library that handles larger responses better and more efficiently.
Now, it appears that the DevTools protocol returns the response body after decompression but before rendering it in the browser when calling Network.GetResponseBodyForInterception. This is an assumption only of course, as I do not see code for that method in https://github.com/ChromeDevTools/devtools-protocol. The assumption is based on the fact that, when calling Network.GetResponseBodyForInterception the response body obtained is NOT compressed (though it may be base64 encoded). Furthermore, the method is marked as experimental and the documentation does not mention anything in regards to compressed responses. Based on that assumption, I will further assume that, at the point that we get the response from Network.GetResponseBodyForInterception it is too late to compress the body ourselves. I confirm that the libraries that I am working with do not bother to compress or uncompress gzipped responses.
I am able to continue working with my code without a need to worry about gzip compressed responses, as I can alter the body without problems.
For reference, I am now using https://github.com/wirepair/gcd, as it is more robust and stable when intercepting larger responses.
Related
I am encountering a problem while using ristretto cache. Indeed, i have a little api that should return me a value stored in my ristretto cache as json.
The problem is that when i call my function, the return is the json encoded in base64 and i just can't find the way to decode it.
Here is the code i have:
Part 1: the code for initializing my ristretto cache:
func InitCache() {
var err error
ristrettoCache, err = ristretto.NewCache(&ristretto.Config{
NumCounters: 3000,
MaxCost: 1e6,
BufferItems: 64,
})
if err != nil {
panic(err)
}
}
Part 2: Putting my values in cache:
for _, t := range listTokensFromDB {
b, err := json.Marshal(t)
if err != nil {
fmt.Println(err)
}
ristrettoCache.Set(t.Symbol, b, 1)
}
Part 3: getting the value from cache
func getTokenInfo(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
key := vars["chain"]+vars["symbol"]
value, found := ristrettoCache.Get(key)
if !found {
return
}
json.NewEncoder(w).Encode(value)
}
The result i have when i make a call to my api is:
"eyJTeW1ib2wiOiJic2NDUllQVE8iLCJBZGRyIjoiMHgyQmNBMUFlM0U1MjQ0NzMyM0IzRWE0NzA4QTNkMTg1ODRDYWY4NWE3IiwiTHBzIjpbeyJTeW1ib2xUb2tlbiI6IkZFRyIsIlRva2VuQWRkciI6IjB4YWNGQzk1NTg1RDgwQWI2MmY2N0ExNEM1NjZDMWI3YTQ5RmU5MTE2NyIsIkxwQWRkciI6IjB4NDU5ZTJlMjQ4NGNlMDU2MWRmNTJiYzFlNjkxMzkyNDA2M2JhZDM5MCJ9LHsiU3ltYm9sVG9rZW4iOiJmQk5CIiwiVG9rZW5BZGRyIjoiMHg4N2IxQWNjRTZhMTk1OEU1MjIyMzNBNzM3MzEzQzA4NjU1MWE1Yzc2IiwiTHBBZGRyIjoiMHg3OGM2NzkzZGMxMDY1OWZlN2U0YWJhMTQwMmI5M2Y2ODljOGY0YzI3In1dfQ=="
But i want the base64 decoded version...
If I change the value b to be string when i insert it in cache like so:
for _, t := range listTokensFromDB {
b, err := json.Marshal(t)
if err != nil {
fmt.Println(err)
}
ristrettoCache.Set(t.Symbol, string(b), 1)
}
When i get the response, i get the stringified json like this:
"{"Symbol":"bscCRYPTO","Addr":"0x2BcA1Ae3E52447323B..."
And i can't find a way to get out of this string :/
Anyone would know how i could get the real json please?
Thank you in advance and i wish u a good day!
From my comments, I meant, in this line, value is most likely of type []byte (or []uint8 - which is the same thing)
value, found := ristrettoCache.Get(key)
JSON encoding a []byte will implicitly base64 the output - since JSON is text-based.
json.NewEncoder(w).Encode(value) // <- value is of type []byte
Inspecting the base64 you posted (https://play.golang.org/p/NAVS4qRfDM2) the underlying binary-bytes are already encoded in JSON - so no extra json.Encode is needed.
Just output the raw-bytes in your handler - and set the content-type to application/json:
func getTokenInfo(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
key := vars["chain"]+vars["symbol"]
value, found := ristrettoCache.Get(key)
if !found {
return
}
// json.NewEncoder(w).Encode(value) // not this
w.Header().Set("Content-Type", "application/json")
if bs, ok := value.([]byte); ok {
_, err := w.Write(bs) //raw bytes (already encoded in JSON)
// check 'err'
} else {
// error unexpected type behind interface{}
}
}
I am using this Go API client on my app https://github.com/heroku/docker-registry-client to interact with a docker registry using Go. The case is that internally that is having some issue when does a PUT request using the package "net/http".
When I run the following code I am getting this as error Put url: http: ContentLength=2821 with Body length 0. So it seems that net/http Client.Do() function is not getting the body I set at some point of the function. But as you can se right on the code below at some point I still have the JSON content that I want to send in a []byte.
body, err := manifest.MarshalJSON()
if err != nil {
return err
}
log.Println(string(body)) // I get the JSON data back here
req, err := http.NewRequest("PUT", url, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Content-Type", manifestV2.MediaTypeManifest)
resp, err := registry.Client.Do(req)
if resp != nil {
defer resp.Body.Close()
}
return err
As far as I have digged into it, the error comes from the net/http Client.do() function (golang.org/src/net/http/client.go line 514), and I'd say the error is triggered from Request.GetBody() function (from line 591 on Client).
So still trying to go deeper and do some tests to find out what is going on here.
Any clue?
In case the error is given by the server I'll have to get something like this, but in the response body and no errors on the net/http Client.Do() call.
Content-Length: <length>
Content-Type: application/json; charset=utf-8
{
"errors:" [
{
"code": <error code>,
"message": "<error message>",
"detail": ...
},
...
]
}
Thank you so much!
Cheers
The issue is modified RoundTripper in https://github.com/heroku/docker-registry-client makes a redirect call to registry after getting authentication token.when it tries to redirect call to registry ,body is empty because http.request body is a buffer and once it is read in first call,buffer becomes empty and there is no body content to send in redirect call.
To fix it:
if req.Method == "PUT" || req.Method == "POST" {
if req.Body != nil {
reUseBody, _ = req.GetBody()
}
}
add this in https://github.com/heroku/docker-registry-client/blob/master/registry/tokentransport.go#L17 and
if req.Method == "PUT" || req.Method == "POST" {
if reUseBody != nil {
req.Body = reUseBody
}
}
in here https://github.com/heroku/docker-registry-client/blob/master/registry/tokentransport.go#L72
and change this function signature https://github.com/heroku/docker-registry-client/blob/master/registry/tokentransport.go#L71
to accept extra argument.
func (t *TokenTransport) retry(req *http.Request, token string, reUseBody io.ReadCloser)
Background story: I'm retrieving details using a GET method. I managed to get the program to parse the output given by the server into a JSON file titled "output.json".
Problem: Every time I make different requests, the output will overwrite any previous content in the output.json file. Is it possible to make a new JSON file for every request?
I am rather new to GoLang and any help will be very useful :)
Note: I am only showing the method that is being used to call the API, if need to, I will show the rest of my code.
Here is my method used:
func render(endpoint string, w http.ResponseWriter, r *http.Request) {
// check if request has cookie set
if cookie, err := r.Cookie(COOKIE_NAME); err != nil {
// else redirect to OAuth Authorization EP
redirectToOAuth(w, r, endpoint)
} else {
session := cookie.Value
accessToken := sessions[session]
// pipe api endpoint
ep := fmt.Sprintf("%s/%s", fidorConfig.FidorApiUrl, endpoint)
if api_req, err := http.NewRequest("GET", ep, nil); err != nil {
w.WriteHeader(500)
w.Write([]byte(err.Error()))
} else {
api_req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", accessToken))
api_req.Header.Set("Accept", "application/vnd.fidor.de; version=1,text/json")
client := &http.Client{}
if api_resp, err := client.Do(api_req); err != nil {
w.WriteHeader(500)
w.Write([]byte(err.Error()))
} else {
if api_resp.StatusCode == 401 { // token probably expired
handleLogout(w, r, endpoint)
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(api_resp.StatusCode)
defer api_resp.Body.Close()
out, err := os.Create("output.json")
if err != nil {
// panic?
}
defer out.Close()
io.Copy(out, api_resp.Body)
}
}
}
}
If you want to append time in your filename (like #Florian suggested), you can do something like this when you are creating file:
out, err := os.Create(fmt.Sprintf("output-%d.json", time.Now().Unix()))
// filename => output-1257894000.json
Here time.Now().Unix() returns the number of seconds elapsed since January 1, 1970 UTC (aka Unix time). So each time it will create different json file.
More info about time: https://golang.org/pkg/time/#Time.Unix
If you don't want your file to be overwritten you can either give the file a different name (for example by appending the current time using time.Now()) or you can append the output to the file, so the output.json contains all json files.
How can I fetch the json response from the POST method? Currently I'm only able to fetch Status - 401 Unauthorized and StatusCode - 401
func postUrl(url string, byt []byte) (*http.Response, error) {
tr := &http.Transport{
DisableCompression: true,
}
client := &http.Client{Transport: tr, Timeout: 10 * time.Second}
req, err := http.NewRequest("POST", url, bytes.NewBuffer(byt))
req.Header.Set("X-Custom-Header", "myvalue")
req.Header.Set("Content-Type", "application/json")
req.Header.Add("Authorization", "Basic "+basicAuth("username", "password"))
resp, err := client.Do(req)
return resp, err
}
Above code produces the output:
{
"errorMessages": [
"You do not have the permission to see the specified issue.",
"Login Required"
],
"errors": {}
}
The way to read the response (if there is one) is the same regardless of what status you get.
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
As Frank said, regardless of the status code in the response you can simply read its body to use whatever content it has.
Particularly for the case of a JSON message, you have two options depending on whether you know the JSON message structure in advance (or want your code to depend on it).
If you know the structure and are ok with hard-coding it (plus you gain some type safety and better client code) you can have:
type ErrorResponse struct {
Messages []string `json:"errorMessages"`
}
And then when you detect an error status code unmarshal the response body as that struct:
if resp.StatusCode % 100 != 2 {
var error ErrorResponse
err := json.Unmarshall(resp.Body, &error)
// check err != nil ...
// user error.ErrorMessages for whatever you want
}
Alternatively if you don't want to depend on the JSON structure (to some degree) you can try to unmarshall it to a map[string]interface{} and try to use that in the generic way you think you can (generally not very useful).
This question is not related to http responses and http methods. Decode json string (wich is http response body in that case) with json decoder.
Simple example
(not directly related to your code snippet)
type Transition struct {
Transition map[string]int
}
func main() {
resp, err := postUrl(url, byt)
if err != nil {
log.Fatal(err)
}
var trans Transition
decoder := json.NewDecoder(resp.Body)
if err := decoder.Decode(&trans); err != nil {
log.Fatal(err)
}
fmt.Printf("%+v\n", trans)
}
As a Go newbie it's difficult for me to pinpoint the problem area, but hopefully giving you some facts will help.
I'm playing with an API which returns its Content-Encoding as gzip. I have written the following to encode my response struct:
reader, err = gzip.NewReader(resp.Body)
defer reader.Close()
// print to standard out
//_, err = io.Copy(os.Stdout, reader)
//if err != nil {
// log.Fatal(err)
//}
// Decode the response into our tescoResponse struct
var response TescoResponse
err := json.NewDecoder(reader).Decode(&response)
I've removed the error handling for brevity, but the point of interest is that if I uncomment the print to stdout, I get the expected result. However, the decode doesn't give me what I expect. Any pointers? Is it that the struct has to map exactly to the response??
Here's the full example:
https://play.golang.org/p/4eCuXxXm3T
From the documenation:
DisableCompression, if true, prevents the Transport from requesting
compression with an "Accept-Encoding: gzip" request header when the
Request contains no existing Accept-Encoding value. If the Transport
requests gzip on its own and gets a gzipped response, it's
transparently decoded in the Response.Body. However, if the user
explicitly requested gzip it is not automatically uncompressed.
Proposed solution:
type gzreadCloser struct {
*gzip.Reader
io.Closer
}
func (gz gzreadCloser) Close() error {
return gz.Closer.Close()
}
// then in your http call ....
if resp.Header.Get("Content-Encoding") == "gzip" {
resp.Header.Del("Content-Length")
zr, err := gzip.NewReader(resp.Body)
if err != nil {
return nil, err
}
resp.Body = gzreadCloser{zr, resp.Body}
}
// then you will be able to decode the json transparently
if err := json.NewDecoder(resp.Body).Decode(&response); err != nil {
}
Adapted solution from your code: https://play.golang.org/p/Vt07y_xgak
As #icza mentioned in the comments, decoding isn't required because the gzip reader automatically decodes when you read using it. Perhaps try:
ubs := make([]byte, len) // check Content-Length header to set len
n, err := reader.Read(ubs)
err := json.Unmarshal(ubs, &response)