Groovy HttpBuilder issues with cookies - exception

Question

I need to write a web client that hits a legacy web app, logs in to it, pulls down some information from a /widget page, and does some work based off of this page's HTML. I'm choosing to go with a Groovy/HttpBuilder solution, for reasons outside the scope of this question.
The only shortcoming (from what I can tell) is that HttpBuilder doesn't support the retaining of cookies between requests. This is a major problem since the (Java) web app uses JSESSIONID cookies to determine whether or not the user is logged in, has permissions, etc.
So first, if my assertion above is incorrect, and HttpBuilder does support the retaining of cookies across requests, please correct me and perhaps the answer here is a solution that shows me how to tap into this part of HttpBuilder. In this case all of my code below is moot.
Assuming I'm correct and this isn't handled by HttpBuilder, I found this excellent solution that I can't get to work for some reason, hence my question.
My adaptation of that code (see link above) is as follows:
TaskAutomator.groovy
====================
package com.me.myapp.tasker
import groovyx.net.http.ContentType
import groovyx.net.http.Method
class TaskAutomator {
static void main(String[] args) {
TaskAutomator tasker = new TaskAutomator()
String result = tasker.doWork("http://myapp.example.com")
println result
}
String doWork(String baseUrl) {
CookieRetainingHttpBuilder cookiedBuilder = new CookieRetainingHttpBuilder(baseUrl)
Map logins = [username: 'user', password: '12345']
// Go to the main page where we will get back the HTML for a login screen.
// We don't really care about the response here, so long as its HTTP 200.
cookiedBuilder.request(Method.GET, ContentType.HTML, "", null)
// Log in to the app, where, on success, we will get back the HTML for a the
// "Main Menu" screen users see when they log in. We don't really care about
// the response here, so long as its HTTP 200.
cookiedBuilder.request(Method.POST, ContentType.HTML, "/auth", logins)
// Finally, now that our JSESSIONID cookies is authenticated, go to the widget page
// which is what we actually care about interacting with.
def response = cookiedBuilder.request(Method.GET, ContentType.HTML, "/widget", null)
// Test to make sure the response is what I think it is.
print response
String result
// TODO: Now actually do work based off the response.
result
}
}
CookieRetainingHttpBuilder
==========================
package com.me.myapp.tasker
import groovyx.net.http.ContentType
import groovyx.net.http.HTTPBuilder
import groovyx.net.http.HttpResponseDecorator
import groovyx.net.http.Method
class CookieRetainingHttpBuilder {
private String baseUrl
private HTTPBuilder httpBuilder
private List<String> cookies
CookieRetainingHttpBuilder(String baseUrl) {
this.baseUrl = baseUrl
this.httpBuilder = initializeHttpBuilder()
this.cookies = []
}
public def request(Method method, ContentType contentType, String url, Map<String, Serializable> params) {
httpBuilder.request(method, contentType) { request ->
uri.path = url
uri.query = params
headers['Cookie'] = cookies.join(';')
}
}
private HTTPBuilder initializeHttpBuilder() {
def httpBuilder = new HTTPBuilder(baseUrl)
httpBuilder.handler.success = { HttpResponseDecorator resp, reader ->
resp.getHeaders('Set-Cookie').each {
String cookie = it.value.split(';')[0]
cookies.add(cookie)
}
reader
}
httpBuilder
}
}
When I run this code I get the following stack trace (I've culled out un-interesting parts as its pretty large):
Exception in thread "main" groovyx.net.http.HttpResponseException: Not Found
at groovyx.net.http.HTTPBuilder.defaultFailureHandler(HTTPBuilder.java:642)
... (lines omitted for brevity)
at groovyx.net.http.HTTPBuilder$1.handleResponse(HTTPBuilder.java:494)
... (lines omitted for brevity)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:506)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:425)
at groovyx.net.http.HTTPBuilder.request(HTTPBuilder.java:374)
at groovyx.net.http.HTTPBuilder$request.call(Unknown Source)
at com.me.myapp.tasker.CookieRetainingHttpBuilder.request(CookieRetainingHttpBuilder.groovy:20)
... (lines omitted for brevity)
at com.me.myapp.tasker.TaskAutomator.doWork(TaskAutomator.groovy:23)
... (lines omitted for brevity)
at com.me.myapp.tasker.TaskAutomator.main(TaskAutomator.groovy:13)
CookieRetainingHttpBuilder:20 is this line from request:
httpBuilder.request(method, contentType) { request ->
Can anyone see why I'm getting this? Additionally, I wanted to confirm my approach/strategy in the TaskAutomater#doWork(...) method. Is my use of CookieRetainingHttpBuilder "correct" in the sense that I'm:
Going to the main/login page
POSTing login creds and logging in
Going to the widget page
Or is there a different way to use HttpBuilder that is better/more efficient here (remember CookieRetainingHttpBuilder is, after all, just a wrapper for HttpBuilder).

Answer 1

I believe that error may be manifesting due to missing imports, or perhaps an older version of HttpBuilder. Looking into HttpBuilder.Class, I see this, which informs my suggestions:
protected java.lang.Object parseResponse(org.apache.http.HttpResponse resp, java.lang.Object contentType) throws groovyx.net.http.HttpResponseException { /* compiled code */ }
I am fairly certain you can just use headers.'Set-Cookiein your httpBuilder setup. The syntax is different from what you have, but the change is small and simple, and this is the base method I use when using HttpBuilder.
#Grab(group = 'org.codehaus.groovy.modules.http-builder', module = 'http-builder', version = '0.7)
import groovyx.net.http.HTTPBuilder
import org.apache.http.HttpException
import static groovyx.net.http.ContentType.TEXT
import static groovyx.net.http.Method.GET
def http = new HTTPBuilder(urlToHit)
http.request(urlToHit, GET, TEXT) { req ->
headers.'User-Agent' = ${userAgent}
headers.'Set-Cookie' = "${myCookie}"
response.success = { resp, reader ->
html = reader.getText()
}
response.failure = { resp, reader ->
System.err.println "Failure response: ${resp.status}"
throw new HttpException()
}
}
Something else to note is that you have no failure handling. I don't know if that will raise an exception, but it could be worth looking into.
EDIT
As suggested, I'm merging my answers (thanks for letting me know...I wasn't sure what proper etiquette was).
Here's what I've come up with. I did my best to reuse the code you posted. I commented as best I could. If you have any questions, let me know.
#Grab(group = 'org.codehaus.groovy.modules.http-builder', module = 'http-builder', version = '0.7')
import static groovyx.net.http.ContentType.HTML
import static groovyx.net.http.Method.POST
import static groovyx.net.http.Method.GET
import groovyx.net.http.ContentType
import groovyx.net.http.HTTPBuilder
import groovyx.net.http.URIBuilder
import groovyx.net.http.Method
import org.apache.http.HttpException
/**
* This class defines the methods used for getting and using cookies
* #param baseUrl The URL we will use to make HTTP requests. In this example, it is https://www.pinterest.com
*/
class CookieRetainingHttpBuilder {
String baseUrl
/**
* This method makes an http request and adds cookies to the array list for later use
* #param method The method used to make the http request. In this example, we use GET and POST
* #param contentType The content type we are requesting. In this example, we are getting HTML
* #param url The URI path for the appropriate page. For example, /login/ is for the login page
* #param params The URI query used for setting parameters. In this example, we are using login credentials
*/
public request (Method method, ContentType contentType, String url, Map<String, Serializable> params) {
List<String> cookies = new ArrayList<>()
def http = new HTTPBuilder(baseUrl)
http.request(baseUrl, method, contentType) { req ->
URIBuilder uriBuilder = new URIBuilder(baseUrl)
uriBuilder.query = params
uriBuilder.path = url
headers.'Accept' = HTML
headers.'User-Agent' = "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36"
headers.'Set-Cookie' = cookies.join(";")
response.success = { resp, reader ->
resp.getHeaders('Set-Cookie').each {
def cookie = it.value.split(";").toString()
cookies.add(cookie)
}
return reader
}
response.failure = { resp, reader ->
System.err.println "Failure response: ${resp.status}"
throw new HttpException()
}
}
}
}
/**
* This class contains the method to make HTTP requests in the proper sequence
* #param base The base URL
* #param user The username of the site being logged in to
* #param pass The password for the username
*/
class TaskAutomator {
private static String base = "http://myapp.example.com"
private static String user = "thisIsMyUser"
private static String pass = "thisIsMyPassword"
/**
* This method contains the functions in proper order to set cookies and login to a site
* #return response Returns the HTML from the final GET request
*/
static String doWork () {
CookieHandler.setDefault(new CookieManager());
CookieRetainingHttpBuilder cookiedBuilder = new CookieRetainingHttpBuilder(baseUrl: base)
Map logins = [username: user, password: pass]
// Go to the main page where we will get back the HTML for a login screen.
// We don't really care about the response here, so long as its HTTP 200.
cookiedBuilder.request(GET, HTML, "", null)
// Log in to the app, where, on success, we will get back the HTML for a the
// "Main Menu" screen users see when they log in. We don't really care about
// the response here, so long as its HTTP 200.
cookiedBuilder.request(POST, HTML, "/login/", logins)
// Finally, now that our JSESSIONID cookies is authenticated, go to the widget page
// which is what we actually care about interacting with.
def response = cookiedBuilder.request(GET, HTML, "/", null)
// Test to make sure the response is what I think it is.
return response
// TODO: Now actually do work based off the response.
}
}
TaskAutomator tasker = new TaskAutomator()
String result = tasker.doWork()
println result