Decoding gZip json with Go - json

As a Go newbie it's difficult for me to pinpoint the problem area, but hopefully giving you some facts will help.
I'm playing with an API which returns its Content-Encoding as gzip. I have written the following to encode my response struct:
reader, err = gzip.NewReader(resp.Body)
defer reader.Close()
// print to standard out
//_, err = io.Copy(os.Stdout, reader)
//if err != nil {
// log.Fatal(err)
//}
// Decode the response into our tescoResponse struct
var response TescoResponse
err := json.NewDecoder(reader).Decode(&response)
I've removed the error handling for brevity, but the point of interest is that if I uncomment the print to stdout, I get the expected result. However, the decode doesn't give me what I expect. Any pointers? Is it that the struct has to map exactly to the response??
Here's the full example:
https://play.golang.org/p/4eCuXxXm3T

From the documenation:
DisableCompression, if true, prevents the Transport from requesting
compression with an "Accept-Encoding: gzip" request header when the
Request contains no existing Accept-Encoding value. If the Transport
requests gzip on its own and gets a gzipped response, it's
transparently decoded in the Response.Body. However, if the user
explicitly requested gzip it is not automatically uncompressed.
Proposed solution:
type gzreadCloser struct {
*gzip.Reader
io.Closer
}
func (gz gzreadCloser) Close() error {
return gz.Closer.Close()
}
// then in your http call ....
if resp.Header.Get("Content-Encoding") == "gzip" {
resp.Header.Del("Content-Length")
zr, err := gzip.NewReader(resp.Body)
if err != nil {
return nil, err
}
resp.Body = gzreadCloser{zr, resp.Body}
}
// then you will be able to decode the json transparently
if err := json.NewDecoder(resp.Body).Decode(&response); err != nil {
}
Adapted solution from your code: https://play.golang.org/p/Vt07y_xgak

As #icza mentioned in the comments, decoding isn't required because the gzip reader automatically decodes when you read using it. Perhaps try:
ubs := make([]byte, len) // check Content-Length header to set len
n, err := reader.Read(ubs)
err := json.Unmarshal(ubs, &response)

Related

Golang - Ristretto Cache returning base 64

I am encountering a problem while using ristretto cache. Indeed, i have a little api that should return me a value stored in my ristretto cache as json.
The problem is that when i call my function, the return is the json encoded in base64 and i just can't find the way to decode it.
Here is the code i have:
Part 1: the code for initializing my ristretto cache:
func InitCache() {
var err error
ristrettoCache, err = ristretto.NewCache(&ristretto.Config{
NumCounters: 3000,
MaxCost: 1e6,
BufferItems: 64,
})
if err != nil {
panic(err)
}
}
Part 2: Putting my values in cache:
for _, t := range listTokensFromDB {
b, err := json.Marshal(t)
if err != nil {
fmt.Println(err)
}
ristrettoCache.Set(t.Symbol, b, 1)
}
Part 3: getting the value from cache
func getTokenInfo(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
key := vars["chain"]+vars["symbol"]
value, found := ristrettoCache.Get(key)
if !found {
return
}
json.NewEncoder(w).Encode(value)
}
The result i have when i make a call to my api is:
"eyJTeW1ib2wiOiJic2NDUllQVE8iLCJBZGRyIjoiMHgyQmNBMUFlM0U1MjQ0NzMyM0IzRWE0NzA4QTNkMTg1ODRDYWY4NWE3IiwiTHBzIjpbeyJTeW1ib2xUb2tlbiI6IkZFRyIsIlRva2VuQWRkciI6IjB4YWNGQzk1NTg1RDgwQWI2MmY2N0ExNEM1NjZDMWI3YTQ5RmU5MTE2NyIsIkxwQWRkciI6IjB4NDU5ZTJlMjQ4NGNlMDU2MWRmNTJiYzFlNjkxMzkyNDA2M2JhZDM5MCJ9LHsiU3ltYm9sVG9rZW4iOiJmQk5CIiwiVG9rZW5BZGRyIjoiMHg4N2IxQWNjRTZhMTk1OEU1MjIyMzNBNzM3MzEzQzA4NjU1MWE1Yzc2IiwiTHBBZGRyIjoiMHg3OGM2NzkzZGMxMDY1OWZlN2U0YWJhMTQwMmI5M2Y2ODljOGY0YzI3In1dfQ=="
But i want the base64 decoded version...
If I change the value b to be string when i insert it in cache like so:
for _, t := range listTokensFromDB {
b, err := json.Marshal(t)
if err != nil {
fmt.Println(err)
}
ristrettoCache.Set(t.Symbol, string(b), 1)
}
When i get the response, i get the stringified json like this:
"{"Symbol":"bscCRYPTO","Addr":"0x2BcA1Ae3E52447323B..."
And i can't find a way to get out of this string :/
Anyone would know how i could get the real json please?
Thank you in advance and i wish u a good day!
From my comments, I meant, in this line, value is most likely of type []byte (or []uint8 - which is the same thing)
value, found := ristrettoCache.Get(key)
JSON encoding a []byte will implicitly base64 the output - since JSON is text-based.
json.NewEncoder(w).Encode(value) // <- value is of type []byte
Inspecting the base64 you posted (https://play.golang.org/p/NAVS4qRfDM2) the underlying binary-bytes are already encoded in JSON - so no extra json.Encode is needed.
Just output the raw-bytes in your handler - and set the content-type to application/json:
func getTokenInfo(w http.ResponseWriter, r *http.Request){
vars := mux.Vars(r)
key := vars["chain"]+vars["symbol"]
value, found := ristrettoCache.Get(key)
if !found {
return
}
// json.NewEncoder(w).Encode(value) // not this
w.Header().Set("Content-Type", "application/json")
if bs, ok := value.([]byte); ok {
_, err := w.Write(bs) //raw bytes (already encoded in JSON)
// check 'err'
} else {
// error unexpected type behind interface{}
}
}

Chrome DevTools Protocol - ContinueInterceptedRequest with gzip body in Golang

I have been working on a golang script that uses the chrome devtools protocol to:
1) Intercept a request
2) Grab the response body for the intercepted request
3) Make some modifications to the html document
4) Continue the intercepted request
The script works for HTML documents except when Content-Encoding is set to gzip. The step-by-step process looks like this"
1) Intercept Request
s.Debugger.CallbackEvent("Network.requestIntercepted", func(params godet.Params) {
iid := params.String("interceptionId")
rtype := params.String("resourceType")
reason := responses[rtype]
headers := getHeadersString(params["responseHeaders"])
log.Println("[+] Request intercepted for", iid, rtype, params.Map("request")["url"])
if reason != "" {
log.Println(" abort with reason", reason)
}
// Alter HTML in request response
if s.Options.AlterDocument && rtype == "Document" && iid != "" {
res, err := s.Debugger.GetResponseBodyForInterception(iid)
if err != nil {
log.Println("[-] Unable to get intercepted response body!")
}
rawAlteredResponse, err := AlterDocument(res, headers)
if err != nil{
log.Println("[-] Unable to alter HTML")
}
if rawAlteredResponse != "" {
log.Println("[+] Sending modified body")
err := s.Debugger.ContinueInterceptedRequest(iid, godet.ErrorReason(reason), rawAlteredResponse, "", "", "", nil)
if err != nil {
fmt.Println("OH NOES AN ERROR!")
log.Println(err)
}
}
} else {
s.Debugger.ContinueInterceptedRequest(iid, godet.ErrorReason(reason), "", "", "", "", nil)
}
})
2) Alter the response body
Here I am making small changes to the HTML markup in procesHtml() (but the code for that function is not relevant to this issue, so will not post it here). I also grab headers from the request and when necessary update the content-length and date before continue the reponse. Then, I gzip compress the body when calling r := gZipCompress([]byte(alteredBody), which returns a string. The string is then concatenated to the headers so I can craft the rawResponse.
func AlterDocument(debuggerResponse []byte, headers map[string]string) (string, error) {
alteredBody, err := processHtml(debuggerResponse)
if err != nil {
return "", err
}
alteredHeader := ""
for k, v := range headers{
switch strings.ToLower(k) {
case "content-length":
v = strconv.Itoa(len(alteredBody))
fmt.Println("Updating content-length to: " + strconv.Itoa(len(alteredBody)))
break
case "date":
v = fmt.Sprintf("%s", time.Now().Format(time.RFC3339))
break
}
alteredHeader += k + ": " + v + "\r\n"
}
r := gZipCompress([]byte(alteredBody))
rawAlteredResponse :=
base64.StdEncoding.EncodeToString([]byte("HTTP/1.1 200 OK" + "\r\n" + alteredHeader + "\r\n\r\n\r\n" + r))
return rawAlteredResponse, nil
}
Note: I am now gzip compressing the body for all responses. The above is temporary while I figure out how to solve this issue.
The gzip compress function looks like this:
func gZipCompress(dataToWorkWith []byte) string{
var b bytes.Buffer
gz, err := gzip.NewWriterLevel(&b, 5)
if err != nil{
panic(err)
}
if _, err := gz.Write(dataToWorkWith); err != nil {
panic(err)
}
if err := gz.Flush(); err != nil {
panic(err)
}
if err := gz.Close(); err != nil {
panic(err)
}
return b.String()
}
As seen in the first code snippet, the response body and headers are set here:
err := s.Debugger.ContinueInterceptedRequest(iid, godet.ErrorReason(reason), rawAlteredResponse, "", "", "", nil)
The result is a bunch of garbled characters in the browser. This works without the gzip functions for non gzipped requests. I have changed the compression level as well (without success). Am I processing the body in the wrong order (string > []byte > gzip > string > base64)? Should this be done in a different order to work? Any help would be immensely appreciated.
The response looks like this, which Chrome puts inside a <body></body> tag
����rܸ� ��_A��Q%GH��Kʔ��vU�˷c�v�}
or in the response:
I can also tell that it is compressing correctly as, when I remove headers, the request results in a .gz file download with all the correct .html when uncompressed. Additionally, the first few bytes in the object returned in gZipCompress tell me that it is gzipped correctly:
31 139 8
or
0x1f 0x8B 0x08
I ended up using a different library that handles larger responses better and more efficiently.
Now, it appears that the DevTools protocol returns the response body after decompression but before rendering it in the browser when calling Network.GetResponseBodyForInterception. This is an assumption only of course, as I do not see code for that method in https://github.com/ChromeDevTools/devtools-protocol. The assumption is based on the fact that, when calling Network.GetResponseBodyForInterception the response body obtained is NOT compressed (though it may be base64 encoded). Furthermore, the method is marked as experimental and the documentation does not mention anything in regards to compressed responses. Based on that assumption, I will further assume that, at the point that we get the response from Network.GetResponseBodyForInterception it is too late to compress the body ourselves. I confirm that the libraries that I am working with do not bother to compress or uncompress gzipped responses.
I am able to continue working with my code without a need to worry about gzip compressed responses, as I can alter the body without problems.
For reference, I am now using https://github.com/wirepair/gcd, as it is more robust and stable when intercepting larger responses.

Create a new JSON file for every GET request

Background story: I'm retrieving details using a GET method. I managed to get the program to parse the output given by the server into a JSON file titled "output.json".
Problem: Every time I make different requests, the output will overwrite any previous content in the output.json file. Is it possible to make a new JSON file for every request?
I am rather new to GoLang and any help will be very useful :)
Note: I am only showing the method that is being used to call the API, if need to, I will show the rest of my code.
Here is my method used:
func render(endpoint string, w http.ResponseWriter, r *http.Request) {
// check if request has cookie set
if cookie, err := r.Cookie(COOKIE_NAME); err != nil {
// else redirect to OAuth Authorization EP
redirectToOAuth(w, r, endpoint)
} else {
session := cookie.Value
accessToken := sessions[session]
// pipe api endpoint
ep := fmt.Sprintf("%s/%s", fidorConfig.FidorApiUrl, endpoint)
if api_req, err := http.NewRequest("GET", ep, nil); err != nil {
w.WriteHeader(500)
w.Write([]byte(err.Error()))
} else {
api_req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", accessToken))
api_req.Header.Set("Accept", "application/vnd.fidor.de; version=1,text/json")
client := &http.Client{}
if api_resp, err := client.Do(api_req); err != nil {
w.WriteHeader(500)
w.Write([]byte(err.Error()))
} else {
if api_resp.StatusCode == 401 { // token probably expired
handleLogout(w, r, endpoint)
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(api_resp.StatusCode)
defer api_resp.Body.Close()
out, err := os.Create("output.json")
if err != nil {
// panic?
}
defer out.Close()
io.Copy(out, api_resp.Body)
}
}
}
}
If you want to append time in your filename (like #Florian suggested), you can do something like this when you are creating file:
out, err := os.Create(fmt.Sprintf("output-%d.json", time.Now().Unix()))
// filename => output-1257894000.json
Here time.Now().Unix() returns the number of seconds elapsed since January 1, 1970 UTC (aka Unix time). So each time it will create different json file.
More info about time: https://golang.org/pkg/time/#Time.Unix
If you don't want your file to be overwritten you can either give the file a different name (for example by appending the current time using time.Now()) or you can append the output to the file, so the output.json contains all json files.

How to output golang http json body to website page using html/javascript/jquery

I've got a Golang Website where I want to display 'scores' from my UWP Game using SQLite's Mobile App Quickstart's API called SwaggerUI. I am getting the scores by doing a HTTP GET request. The problem is that the scores output to the Golang console in JSON Format. I want to display the scores onto the actual website. How could I call my golang function in the Frontend in order to do this? The frontend is written in HTML/Javascript/JQuery.
This is my Golang Function that does the HTTP Request to SwaggerUI and outputs to the Golang Console:
func scoresPage(res http.ResponseWriter, req *http.Request) {
//Connecting to SwaggerUI API to get Scores from Azure for UWP Application
req, err := http.NewRequest("GET", os.ExpandEnv("https://brainworksappservice.azurewebsites.net/tables/TodoItem?$select=score"), nil)
if err != nil {
log.Fatal(err)
}
//You have to specify these headers
req.Header.Set("Accept", "application/json")
//If you do not specify what version your API is, you cannot receive the JSON
req.Header.Set("Zumo-Api-Version", "2.0.0")
//Do the request
resp, err := http.DefaultClient.Do(req)
//Error if the request cannot be done
if err != nil {
log.Fatal(err)
}
//You need to close the Body everytime, as if you don't you could leak information
defer resp.Body.Close()
//Read all of the information from the body
body, err := ioutil.ReadAll(resp.Body)
//Error if the info cannot be read
if err != nil {
log.Fatal(err)
}
//Write the JSON to the standard output (the Console)
_, err = os.Stdout.Write(body)
//Error if the info cannot be output to the console
if err != nil {
log.Fatal(err)
}
http.ServeFile(res, req, "Scores.html")
} `
This is the main Function which serves up the website and handles the scores page:
func main() {
http.HandleFunc("/scores", scoresPage)
//serve on the port 8000 forever
http.ListenAndServe(":8000", nil)
}
Assuming, that you don't want to dump the json as is onto you page but instead format it in some way with html and css, then you could first decode the returned body into a slice of structs that mirror the structure of your json. For example like this:
type Score struct {
Id string `json:"id"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
Version string `json:"version"`
Deleted bool `json:"deleted"`
Text string `json:"text"`
Complete bool `json:"complete"`
Score string `json:"score"`
}
scores := []*Score{}
if err := json.Unmarshal(body, &scores); err != nil {
panic(err)
}
fmt.Println(scores[0])
https://play.golang.org/p/m_ySdZulqy
After you've decoded the json you can use Go's template package to loop over the scores and format them as you wish. While you should use the html/template package for rendering html you should check out text/template for the documentation on how to actually program the templates, they have the same interface.
Here's a quick example: https://play.golang.org/p/EYfV-TzoA0
In that example I'm using the template package to parse a string (scoresPage) and output the result to stdout, but you can just as easily parse your Scores.html file with ParseFiles and return the output in the http response by passing the res http.ResponseWriter instead of os.Stdout as the first argument to template.Execute.

Get response from POST type method from restful API

How can I fetch the json response from the POST method? Currently I'm only able to fetch Status - 401 Unauthorized and StatusCode - 401
func postUrl(url string, byt []byte) (*http.Response, error) {
tr := &http.Transport{
DisableCompression: true,
}
client := &http.Client{Transport: tr, Timeout: 10 * time.Second}
req, err := http.NewRequest("POST", url, bytes.NewBuffer(byt))
req.Header.Set("X-Custom-Header", "myvalue")
req.Header.Set("Content-Type", "application/json")
req.Header.Add("Authorization", "Basic "+basicAuth("username", "password"))
resp, err := client.Do(req)
return resp, err
}
Above code produces the output:
{
"errorMessages": [
"You do not have the permission to see the specified issue.",
"Login Required"
],
"errors": {}
}
The way to read the response (if there is one) is the same regardless of what status you get.
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
As Frank said, regardless of the status code in the response you can simply read its body to use whatever content it has.
Particularly for the case of a JSON message, you have two options depending on whether you know the JSON message structure in advance (or want your code to depend on it).
If you know the structure and are ok with hard-coding it (plus you gain some type safety and better client code) you can have:
type ErrorResponse struct {
Messages []string `json:"errorMessages"`
}
And then when you detect an error status code unmarshal the response body as that struct:
if resp.StatusCode % 100 != 2 {
var error ErrorResponse
err := json.Unmarshall(resp.Body, &error)
// check err != nil ...
// user error.ErrorMessages for whatever you want
}
Alternatively if you don't want to depend on the JSON structure (to some degree) you can try to unmarshall it to a map[string]interface{} and try to use that in the generic way you think you can (generally not very useful).
This question is not related to http responses and http methods. Decode json string (wich is http response body in that case) with json decoder.
Simple example
(not directly related to your code snippet)
type Transition struct {
Transition map[string]int
}
func main() {
resp, err := postUrl(url, byt)
if err != nil {
log.Fatal(err)
}
var trans Transition
decoder := json.NewDecoder(resp.Body)
if err := decoder.Decode(&trans); err != nil {
log.Fatal(err)
}
fmt.Printf("%+v\n", trans)
}