How to count all HTTP requests sent, retries in? - apache-httpclient-4.x

Question

Some use cases require being able to count the requests sent by the Apache API. For example, when massively requesting a web API, which API requires an authentication through an API key, and which TOS limits the requests count in time for each key.
Being more specific on the case, I'm requesting https://domain1/fooNeedNoKey, and depending on its response analyzed data, I request https://domain2/fooNeedKeyWithRequestsCountRestrictions. All sends of those 1-to-2-requests sequences, are performed through a single org.apache.http.impl.client.FutureRequestExecutionService.
As of now, depending on org.apache.httpcomponents:httpclient:4.3.3, I'm using those API elements:
org.apache.http.impl.client.FutureRequestExecutionService, to perform multi-threaded HTTP requests. It offers time metrics (how much time did an HTTP thread took until terminated), but no requests counter metrics
final CloseableHttpClient httpClient = HttpClients.custom()
// the auto-retry feature of the Apache API will retry up to 5
// times on failure, being also allowed to send again requests
// that were already sent if necessary (I don't really understand
// the purpose of the second parameter below)
.setRetryHandler(new StandardHttpRequestRetryHandler(5, true))
// for HTTP 503 'Service unavailable' errors, also retrying up to
// 5 times, waiting 500ms between each retry. Guessed is that those
// 5 retries are part of the previous "global" 5 retries setting.
// The below setting, when used alone, would allow to only enable
// retries for HTTP 503, or to get a greater count of retries for
// this specific error
.setServiceUnavailableRetryStrategy(new DefaultServiceUnavailableRetryStrategy(5, 500))
.build();, which customizes the Apache API retry behavior
Getting back to the topic :
A request counter could be created by extending the Apache API retry-related classes quoted before
Alternatively, an Apache API support unrelated ticket tends to indicate this requests-counter metrics could be available and forwarded out of the API, into Java NIO
Edit 1:
Looks like the Apache API won't permit this to be done.
Quote from the inside of the API, RetryExec not beeing extendable in the API code I/Os:
package org.apache.http.impl.execchain;
public class RetryExec implements ClientExecChain {
..
public CloseableHttpResponse execute(
final HttpRoute route,
final HttpRequestWrapper request,
final HttpClientContext context,
final HttpExecutionAware execAware) throws IOException, HttpException {
..
for (int execCount = 1;; execCount++) {
try {
return this.requestExecutor.execute(route, request, context, execAware);
} catch (final IOException ex) {
..
if (retryHandler.retryRequest(ex, execCount, context)) {
..
}
..
}
}
The 'execCount' variable is the needed info, and it can't be accessed since it's only locally used.
As well, one can extend 'retryHandler', and manually count requests in it, but 'retryHandler.retryRequest(ex, execCount, context)' is not provided with the 'request' variable, making it impossible to know on what we're incrementing a counter (one may only want to increment the counter for requests sent to a specific domain).
I'm out of Java ideas for it. A 3rd party alternative: having the Java process polling a file on disk, managed by a shell script counting the desired requests. Sure it will make a lot of disk read-accesses and will be a hardware killer option.

Answer 1

Ok, the work around was easy, the HttpContext class of the API is intended for this:
// optionnally, in case your HttpCLient is configured for retry
class URIAwareHttpRequestRetryHandler extends StandardHttpRequestRetryHandler {
public URIAwareHttpRequestRetryHandler(final int retryCount, final boolean requestSentRetryEnabled)
{
super(retryCount, requestSentRetryEnabled);
}
#Override
public boolean retryRequest(final IOException exception, final int executionCount, final HttpContext context)
{
final boolean ret = super.retryRequest(exception, executionCount, context);
if (ret) {
doForEachRequestSentOnURI((String) context.getAttribute("requestURI"));
}
return ret;
}
}
// optionnally, in addition to the previous one, in case your HttpClient has specific settings for the 'Service unavailable' errors retries
class URIAwareServiceUnavailableRetryStrategy extends DefaultServiceUnavailableRetryStrategy {
public URIAwareServiceUnavailableRetryStrategy(final int maxRetries, final int retryInterval)
{
super(maxRetries, retryInterval);
}
#Override
public boolean retryRequest(final HttpResponse response, final int executionCount, final HttpContext context)
{
final boolean ret = super.retryRequest(response, executionCount, context);
if (ret) {
doForEachRequestSentOnURI((String) context.getAttribute("requestURI"));
}
return ret;
}
}
// main HTTP querying code: retain the URI in the HttpContext to make it available in the custom retry-handlers code
httpContext.setAttribute("requestURI", httpGET.getURI().toString());
try {
httpContext.setAttribute("requestURI", httpGET.getURI().toString());
httpClient.execute(httpGET, getHTTPResponseHandlerLazy(), httpContext);
// if request got successful with no need of retries, of if it succeeded on the last send: in any cases, this is the last query sent to server and it got successful
doForEachRequestSentOnURI(httpGET.getURI().toString());
} catch (final ClientProtocolException e) {
// if request definitively failed after retries: it's the last query sent to server, and it failed
doForEachRequestSentOnURI(httpGET.getURI().toString());
} catch (final IOException e) {
// if request definitively failed after retries: it's the last query sent to server, and it failed
doForEachRequestSentOnURI(httpGET.getURI().toString());
} finally {
// restoring the context as it was initially
httpContext.removeAttribute("requestURI");
}
Solved.