I am using this Go API client on my app https://github.com/heroku/docker-registry-client to interact with a docker registry using Go. The case is that internally that is having some issue when does a PUT request using the package "net/http".
When I run the following code I am getting this as error Put url: http: ContentLength=2821 with Body length 0. So it seems that net/http Client.Do() function is not getting the body I set at some point of the function. But as you can se right on the code below at some point I still have the JSON content that I want to send in a []byte.
body, err := manifest.MarshalJSON()
if err != nil {
return err
}
log.Println(string(body)) // I get the JSON data back here
req, err := http.NewRequest("PUT", url, bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Content-Type", manifestV2.MediaTypeManifest)
resp, err := registry.Client.Do(req)
if resp != nil {
defer resp.Body.Close()
}
return err
As far as I have digged into it, the error comes from the net/http Client.do() function (golang.org/src/net/http/client.go line 514), and I'd say the error is triggered from Request.GetBody() function (from line 591 on Client).
So still trying to go deeper and do some tests to find out what is going on here.
Any clue?
In case the error is given by the server I'll have to get something like this, but in the response body and no errors on the net/http Client.Do() call.
Content-Length: <length>
Content-Type: application/json; charset=utf-8
{
"errors:" [
{
"code": <error code>,
"message": "<error message>",
"detail": ...
},
...
]
}
Thank you so much!
Cheers
The issue is modified RoundTripper in https://github.com/heroku/docker-registry-client makes a redirect call to registry after getting authentication token.when it tries to redirect call to registry ,body is empty because http.request body is a buffer and once it is read in first call,buffer becomes empty and there is no body content to send in redirect call.
To fix it:
if req.Method == "PUT" || req.Method == "POST" {
if req.Body != nil {
reUseBody, _ = req.GetBody()
}
}
add this in https://github.com/heroku/docker-registry-client/blob/master/registry/tokentransport.go#L17 and
if req.Method == "PUT" || req.Method == "POST" {
if reUseBody != nil {
req.Body = reUseBody
}
}
in here https://github.com/heroku/docker-registry-client/blob/master/registry/tokentransport.go#L72
and change this function signature https://github.com/heroku/docker-registry-client/blob/master/registry/tokentransport.go#L71
to accept extra argument.
func (t *TokenTransport) retry(req *http.Request, token string, reUseBody io.ReadCloser)
Related
I'm currently experiencing a problem that really frustrates me and where i absolutely can't see anny issues with my code.
What i'm trying to achieve is to send a http POST message to a mockup of an iDrac i wrote (both softwares written in golang) to control the mockup's powerstate, but no matter what i configure for the request, the mockup always receives get requests with an empty body.
The function i create and send the request with:
func (iDrac *IDrac) SetPowerstate(powerstate string) error {
//Create reset type json string
var jsonStr = `{"ResetType":"` + powerstate + `"}`
//Create the request with auth header
req, reqErr := iDrac.buildHttpRequestWithAuthorizationHeader(http.MethodPost, "https://"+iDrac.hostAddress+apiBaseURL+apiChassisURL+apiChassisResetAction, jsonStr)
if reqErr != nil {
return fmt.Errorf("COULD_NOT_CREATE_REQUEST: " + reqErr.Error())
}
req.Header.Set("Content-Type", "application/json; charset=UTF-8")
//Make the request
resp, doErr := http.DefaultClient.Do(req)
if doErr != nil {
return fmt.Errorf("COULD_NOT_SEND_POST_REQUEST_TO_IDRAC_API" + doErr.Error())
}
//Check if the request was successful
if resp.StatusCode != 200 {
return fmt.Errorf("COULD_NOT_CHANGE_SERVER_POWER_STATUS_OVER_IDRAC HTTP:" + resp.Status)
}
return nil
}
The helper function i use to build the request with:
func (iDrac *IDrac) buildHttpRequestWithAuthorizationHeader(method string, url string, content string) (*http.Request, error) {
req, err := http.NewRequest(method, url, bytes.NewBuffer([]byte(content)))
if err != nil {
return nil, err
}
req.Header.Add("Authorization", "Basic "+iDrac.hashedCredentials)
return req, nil
}
And finally the function where the mockup proccesses the request:
func handlePerformReset(w http.ResponseWriter, r *http.Request) {
garbagelog.Log("Handling reset perform request from " + r.RemoteAddr)
if r.Method != http.MethodPost {
garbagelog.Log("Invalid http method! Got " + r.Method + " expected POST")
w.WriteHeader(405)
return
}
if !checkAuthorization(r.BasicAuth()) {
w.WriteHeader(401)
return
}
var resetType nidrac.ResetType
err := json.NewDecoder(r.Body).Decode(&resetType)
if err != nil {
garbagelog.Log("Could not decode reset type: " + err.Error())
w.WriteHeader(422)
return
}
iDracMock.PerformResetAction(resetType.ResetType)
garbagelog.Log(">>>>SEVERSTATE: " + iDracMock.PowerState().PowerState + "<<<<")
w.WriteHeader(200)
}
The type the iDrac mock tries to convert the body to:
type ResetType struct {
ResetType string
}
It works flawlessly when i try to reach the endpoint with postman:
iDrac mockup log confirming the successful request
Postnam request configuration:
Postman request configuration
But it somehow does not work when i try making the request with go code:
iDrac mockup log saying that the http method is invalid (because it's get instead of post)
I spent two hours trying to find a solution but i somehow did not find a post with someone having the same problem that i have.
Edit: Corrected old code. The problem remains even with the silly mistake fixed:
//Create the request with auth header
req, reqErr := iDrac.buildHttpRequestWithAuthorizationHeader(http.MethodPost, "https://"+iDrac.hostAddress+apiBaseURL+apiChassisURL+apiChassisResetAction, jsonStr)
if reqErr != nil {
return fmt.Errorf("COULD_NOT_CREATE_REQUEST: " + reqErr.Error())
}
How i build the routes in the iDrac mockup:
http.Handle("/", http.HandlerFunc(handleDefault))
http.Handle("/reset", http.HandlerFunc(handleReset))
http.Handle("/redfish/v1/Systems/System.Embedded.1", http.HandlerFunc(handlePowerStateGet))
http.Handle("/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset", http.HandlerFunc(handlePerformReset))
garbagelog.Log("Starting webserver")
err := http.ListenAndServeTLS(":443", currentConfig.CertFile, currentConfig.PrivKeyFile, nil)
if err != nil {
garbagelog.Log("Could not serve TLS: " + err.Error())
}
In both requests, the one created in my iDrac comms module and the one received by the iDrac mockup, did confirm that the requested path is:
r.URL.Path = /redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset
I found the problem:
The constants i built the urls with were defined like this:
const (
apiBaseURL = "/redfish/v1/"
apiChassisURL = "/Systems/System.Embedded.1"
apiChassisResetAction = "/Actions/ComputerSystem.Reset"
)
Leading to a url that looks like this:
https://host/redfish/v1//Systems/System.Embedded.1/Actions/ComputerSystem.Reset
(Notice the double // between v1 and Systems)
So i've fixed it:
const (
apiBaseURL = "/redfish/v1"
apiChassisURL = "/Systems/System.Embedded.1"
apiChassisResetAction = "/Actions/ComputerSystem.Reset"
)
And everything works correctly:
Test results showing that every test was successful
I thank everyone for their input for helping me not lose my mind completely.
based on your screenshot, I can see you are using a "POST" request in your postman, but in your code is "GET".
I think nothing wrong with the code but the method only :)
I am making an authentication form on my web app.
First, I send the form data to server, it produces the token and I store it in the localStorage
Then, I want to redirect from form page to the home page. Here is the JavaScript that sends the token to server so that I would be seen as an authorized user on the home page:
const xhr = new XMLHttpRequest()
xhr.open('GET', '/')
xhr.setRequestHeader('Authorization', localStorage.token)
xhr.send()
xhr.onload = () => {
if(xhr.status >= 400) {
console.log("error")
}
}
xhr.onerror = () => {
console.log("error")
}
Then I want to check the token and show the home page. Here is the Golang func for it:
func (h *Handler) Home_page(c *gin.Context) {
header := c.GetHeader("Authorization")
if header != "" {
_, err := h.services.Authorization.ParseToken(header)
if err != nil {
newErrorResponse(c, http.StatusUnauthorized, err.Error())
return
}
c.HTML(
http.StatusOK,
"home_page.gohtml",
gin.H{
"IsAuth": true,
},
)
return
}
}
ParseToken func:
func (s *AuthService) ParseToken(accessToken string) (int, error) {
token, err := jwt.ParseWithClaims(accessToken, &tokenClaims{}, func(token *jwt.Token) (interface{}, error) {
if _, ok := token.Method.(*jwt.SigningMethodHMAC); !ok {
return nil, errors.New("invalid signing method")
}
return []byte(signingKey), nil //signingKey is a string with random elements
})
if err != nil {
return 0, err
}
claims, ok := token.Claims.(*tokenClaims)
if !ok {
return 0, errors.New("token claims are not of type *tokenClaims")
}
return claims.UserId, nil
}
The problem is that I get this error:
ERRO[0001] illegal base64 data at input byte 0
I already checked the token on the jwt.io, it shows that the token is verified, maybe the problem is in the type of string that I am passing as a token.
Another problem is that if I don't check the error returning from token validation, the home page doesn't show with this error:
http: panic serving [::1]:50490: write tcp [::1]:8083->[::1]:50490: write: broken pipe
I am new to Golang, was struggling with this problem, though it seems typical.
I would be very thankful for any help on how to process the token or to redirect from the authentication form!
Edit: I am using the https://github.com/golang-jwt/jwt/v4
Somehow, this issue:
illegal base64 data at input byte 0
Was solved by changing the way I stored the token. I was storing in the local storage like this:
localStorage.setItem('token', data.token)
I changed it to the next line and the error disappeared:
localStorage.token = data.token
(data is the JSON with token that my server returns)
Now if I log the value of token it appears without commas. #Crowman, thank you for answer!
Edit: The second issue with broken pipe occurred, because I was not waiting for the answer on the client side. So now I changed the JS code to wait for the response but still I am struggling with how to show the html page that I receive from server using JavaScript.
I have been working on a golang script that uses the chrome devtools protocol to:
1) Intercept a request
2) Grab the response body for the intercepted request
3) Make some modifications to the html document
4) Continue the intercepted request
The script works for HTML documents except when Content-Encoding is set to gzip. The step-by-step process looks like this"
1) Intercept Request
s.Debugger.CallbackEvent("Network.requestIntercepted", func(params godet.Params) {
iid := params.String("interceptionId")
rtype := params.String("resourceType")
reason := responses[rtype]
headers := getHeadersString(params["responseHeaders"])
log.Println("[+] Request intercepted for", iid, rtype, params.Map("request")["url"])
if reason != "" {
log.Println(" abort with reason", reason)
}
// Alter HTML in request response
if s.Options.AlterDocument && rtype == "Document" && iid != "" {
res, err := s.Debugger.GetResponseBodyForInterception(iid)
if err != nil {
log.Println("[-] Unable to get intercepted response body!")
}
rawAlteredResponse, err := AlterDocument(res, headers)
if err != nil{
log.Println("[-] Unable to alter HTML")
}
if rawAlteredResponse != "" {
log.Println("[+] Sending modified body")
err := s.Debugger.ContinueInterceptedRequest(iid, godet.ErrorReason(reason), rawAlteredResponse, "", "", "", nil)
if err != nil {
fmt.Println("OH NOES AN ERROR!")
log.Println(err)
}
}
} else {
s.Debugger.ContinueInterceptedRequest(iid, godet.ErrorReason(reason), "", "", "", "", nil)
}
})
2) Alter the response body
Here I am making small changes to the HTML markup in procesHtml() (but the code for that function is not relevant to this issue, so will not post it here). I also grab headers from the request and when necessary update the content-length and date before continue the reponse. Then, I gzip compress the body when calling r := gZipCompress([]byte(alteredBody), which returns a string. The string is then concatenated to the headers so I can craft the rawResponse.
func AlterDocument(debuggerResponse []byte, headers map[string]string) (string, error) {
alteredBody, err := processHtml(debuggerResponse)
if err != nil {
return "", err
}
alteredHeader := ""
for k, v := range headers{
switch strings.ToLower(k) {
case "content-length":
v = strconv.Itoa(len(alteredBody))
fmt.Println("Updating content-length to: " + strconv.Itoa(len(alteredBody)))
break
case "date":
v = fmt.Sprintf("%s", time.Now().Format(time.RFC3339))
break
}
alteredHeader += k + ": " + v + "\r\n"
}
r := gZipCompress([]byte(alteredBody))
rawAlteredResponse :=
base64.StdEncoding.EncodeToString([]byte("HTTP/1.1 200 OK" + "\r\n" + alteredHeader + "\r\n\r\n\r\n" + r))
return rawAlteredResponse, nil
}
Note: I am now gzip compressing the body for all responses. The above is temporary while I figure out how to solve this issue.
The gzip compress function looks like this:
func gZipCompress(dataToWorkWith []byte) string{
var b bytes.Buffer
gz, err := gzip.NewWriterLevel(&b, 5)
if err != nil{
panic(err)
}
if _, err := gz.Write(dataToWorkWith); err != nil {
panic(err)
}
if err := gz.Flush(); err != nil {
panic(err)
}
if err := gz.Close(); err != nil {
panic(err)
}
return b.String()
}
As seen in the first code snippet, the response body and headers are set here:
err := s.Debugger.ContinueInterceptedRequest(iid, godet.ErrorReason(reason), rawAlteredResponse, "", "", "", nil)
The result is a bunch of garbled characters in the browser. This works without the gzip functions for non gzipped requests. I have changed the compression level as well (without success). Am I processing the body in the wrong order (string > []byte > gzip > string > base64)? Should this be done in a different order to work? Any help would be immensely appreciated.
The response looks like this, which Chrome puts inside a <body></body> tag
����rܸ� ��_A��Q%GH��Kʔ��vU�˷c�v�}
or in the response:
I can also tell that it is compressing correctly as, when I remove headers, the request results in a .gz file download with all the correct .html when uncompressed. Additionally, the first few bytes in the object returned in gZipCompress tell me that it is gzipped correctly:
31 139 8
or
0x1f 0x8B 0x08
I ended up using a different library that handles larger responses better and more efficiently.
Now, it appears that the DevTools protocol returns the response body after decompression but before rendering it in the browser when calling Network.GetResponseBodyForInterception. This is an assumption only of course, as I do not see code for that method in https://github.com/ChromeDevTools/devtools-protocol. The assumption is based on the fact that, when calling Network.GetResponseBodyForInterception the response body obtained is NOT compressed (though it may be base64 encoded). Furthermore, the method is marked as experimental and the documentation does not mention anything in regards to compressed responses. Based on that assumption, I will further assume that, at the point that we get the response from Network.GetResponseBodyForInterception it is too late to compress the body ourselves. I confirm that the libraries that I am working with do not bother to compress or uncompress gzipped responses.
I am able to continue working with my code without a need to worry about gzip compressed responses, as I can alter the body without problems.
For reference, I am now using https://github.com/wirepair/gcd, as it is more robust and stable when intercepting larger responses.
I am trying to make a Json web token authentication system with Go however I cant seem to get the parsing of the web token working.
The error occurs in the following function.
func RequireTokenAuthentication(rw http.ResponseWriter, req *http.Request, next http.HandlerFunc) {
authBackend := InitJWTAuthenticationBackend()
jwtString := req.Header.Get("Authorization")
token, err := jwt.Parse(jwtString, func(token *jwt.Token) (interface{}, error) {
if _, ok := token.Method.(*jwt.SigningMethodRSA); !ok {
log.Println("Unexpected signing method")
return nil, fmt.Errorf("Unexpected signing method: %v", token.Header["alg"])
} else {
log.Println("The token has been successfully returned")
return authBackend.PublicKey, nil
}
})
log.Println(token)
log.Println(token.Valid)
if err == nil && token.Valid && !authBackend.IsInBlacklist(req.Header.Get("Authorization")) {
next(rw, req)
} else {
rw.WriteHeader(http.StatusUnauthorized)
log.P
rintln("Status unauthorized RequireTokenAuthentication")
}
}
returns the following log
[negroni] Started GET /test/hello
2016/09/13 01:34:46 &{Bearer eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE0NzM5NzQ4OTAsImlhdCI6MTQ3MzcxNTY5MCwic3ViIjoiIn0.mnwEwdR8nuvdLo_4Ie43me7iph2LeSj1uikokgD6VJB7isjFPShN8E7eQr4GKwuIiLTi34_i6iJRpmx9qrPugkzvsoxX44qlFi6M7FDhVySRiYbBQwTCvKCpvhnsK8BHJyEgy813aaxOMK6sKZJoaKs5JYUvnNZdNqmENYj1BM6FdbGP-oLHuR_CJK0Pym1NMhv9zLI1rpJOGu4mfj1t4tHYZAEGirPnzYMamtrK6TyEFE6Xi4voEEadq7hXvWREg6wNSQsYgww8uOaIWLy1yLbhTkPmT8zfRwLLYLqS_UuZ0xIaSWO1mF2plvOzz1WlF3ZEHLS31T1egB1XL4WTNQe <nil> map[] <nil> false}
2016/09/13 01:34:46 false
2016/09/13 01:34:46 Status unauthorized RequireTokenAuthentication
[negroni] Completed 401 Unauthorized in 71.628ms
and here is the cURL that I am using to initiate it
curl -H "Authorization: Bearer eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE0NzM5NzQ4OTAsImlhdCI6MTQ3MzcxNTY5MCwic3ViIjoiIn0.mnwEwdR8nuvdLo_4Ie43me7iph2LeSj1uikokgD6VJB7isjFPShN8E7eQr4GKwuIiLTi34_i6iJRpmx9qrPugkzvsoxX44qlFi6M7FDhVySRiYbBQwTCvKCpvhnsK8BHJyEgy813aaxOMK6sKZJoaKs5JYUvnNZdNqmENYj1BM6FdbGP-oLHuR_CJK0Pym1NMhv9zLI1rpJOGu4mfj1t4tHYZAEGirPnzYMamtrK6TyEFE6Xi4voEEadq7hXvWREg6wNSQsYgww8uOaIWLy1yLbhTkPmT8zfRwLLYLqS_UuZ0xIaSWO1mF2plvOzz1WlF3ZEHLS31T1egB1XL4WTNQe" http://localhost:5000/test/hello
I have also tried curl without Bearer
curl -H "Authorization:eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE0NzM5NzQ4OTAsImlhdCI6MTQ3MzcxNTY5MCwic3ViIjoiIn0.mnwEwdR8nuvdLo_4Ie43me7iph2LeSj1uikokgD6VJB7isjFPShN8E7eQr4GKwuIiLTi34_i6iJRpmx9qrPugkzvsoxX44qlFi6M7FDhVySRiYbBQwTCvKCpvhnsK8BHJyEgy813aaxOMK6sKZJoaKs5JYUvnNZdNqmENYj1BM6FdbGP-oLHuR_CJK0Pym1NMhv9zLI1rpJOGu4mfj1t4tHYZAEGirPnzYMamtrK6TyEFE6Xi4voEEadq7hXvWREg6wNSQsYgww8uOaIWLy1yLbhTkPmT8zfRwLLYLqS_UuZ0xIaSWO1mF2plvOzz1WlF3ZEHLS31T1egB1XL4WTNQe" http://localhost:5000/test/hello
The error is occurring because the token is invalid token.Valid = false I have generated it using the following process.
Here is the router
router.HandleFunc("/token-auth", controllers.Login).Methods("POST")
Here is the login controller
func Login(w http.ResponseWriter, r *http.Request) {
requestUser := new(models.User)
decoder := json.NewDecoder(r.Body)
decoder.Decode(&requestUser)
responseStatus, token := utils.Login(requestUser) //here the util file seen below is used
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(responseStatus)
w.Write(token)
}
This is the util file
func Login(requestUser *models.User) (int, []byte) {
authBackend := authentication.InitJWTAuthenticationBackend()
if authBackend.Authenticate(requestUser) {
token, err := authBackend.GenerateToken(requestUser.UUID)
if err != nil {
return http.StatusInternalServerError, []byte("")
} else {
response, _ := json.Marshal(parameters.TokenAuthentication{token})
return http.StatusOK, response
}
}
return http.StatusUnauthorized, []byte("")
}
and here is the method used to generate the token
func (backend *JWTAuthenticationBackend) GenerateToken(userUUID string) (string, error) {
token := jwt.New(jwt.SigningMethodRS512)
claims := token.Claims.(jwt.MapClaims)
claims["exp"] = time.Now().Add(time.Hour * time.Duration(settings.Get().JWTExpirationDelta)).Unix()
claims["iat"] = time.Now().Unix()
claims["sub"] = userUUID
tokenString, err := token.SignedString(backend.privateKey)
if err != nil {
panic(err)
return "", err
}
return tokenString, nil
}
How do I fix the Token Parsing system so that the token is valid?
If you need any additional information I would be more than happy to make an edit with the respective information.
Thank
The error returned by jwt.Parse() says
tokenstring should not contain 'bearer '
So if you remove "Bearer ":
jwtString = strings.Split(jwtString, "Bearer ")[1]
you get a bit further
The token has been successfully returned
however now there's a new error:
key is of invalid type
Sorry it's not a complete answer!
key is of invalid type
type in this context is referring to the dynamic data-type in Go.
For SigningMethodRSA, the public key must be of type *rsa.PublicKey which can be constructed by calling jwt.ParseRSAPublicKeyFromPEM().
The key value returned to the parser might be created with something like:
keyStruct, _ := jwt.ParseRSAPublicKeyFromPEM(myPublicKeyString)
See:
https://github.com/dgrijalva/jwt-go#signing-methods-and-key-types
https://godoc.org/github.com/dgrijalva/jwt-go#SigningMethodRSA
https://godoc.org/github.com/dgrijalva/jwt-go#ParseRSAPublicKeyFromPEM
Related:
How to generate JWT token always through invalid key type error
As a Go newbie it's difficult for me to pinpoint the problem area, but hopefully giving you some facts will help.
I'm playing with an API which returns its Content-Encoding as gzip. I have written the following to encode my response struct:
reader, err = gzip.NewReader(resp.Body)
defer reader.Close()
// print to standard out
//_, err = io.Copy(os.Stdout, reader)
//if err != nil {
// log.Fatal(err)
//}
// Decode the response into our tescoResponse struct
var response TescoResponse
err := json.NewDecoder(reader).Decode(&response)
I've removed the error handling for brevity, but the point of interest is that if I uncomment the print to stdout, I get the expected result. However, the decode doesn't give me what I expect. Any pointers? Is it that the struct has to map exactly to the response??
Here's the full example:
https://play.golang.org/p/4eCuXxXm3T
From the documenation:
DisableCompression, if true, prevents the Transport from requesting
compression with an "Accept-Encoding: gzip" request header when the
Request contains no existing Accept-Encoding value. If the Transport
requests gzip on its own and gets a gzipped response, it's
transparently decoded in the Response.Body. However, if the user
explicitly requested gzip it is not automatically uncompressed.
Proposed solution:
type gzreadCloser struct {
*gzip.Reader
io.Closer
}
func (gz gzreadCloser) Close() error {
return gz.Closer.Close()
}
// then in your http call ....
if resp.Header.Get("Content-Encoding") == "gzip" {
resp.Header.Del("Content-Length")
zr, err := gzip.NewReader(resp.Body)
if err != nil {
return nil, err
}
resp.Body = gzreadCloser{zr, resp.Body}
}
// then you will be able to decode the json transparently
if err := json.NewDecoder(resp.Body).Decode(&response); err != nil {
}
Adapted solution from your code: https://play.golang.org/p/Vt07y_xgak
As #icza mentioned in the comments, decoding isn't required because the gzip reader automatically decodes when you read using it. Perhaps try:
ubs := make([]byte, len) // check Content-Length header to set len
n, err := reader.Read(ubs)
err := json.Unmarshal(ubs, &response)