What can cause HTTP GET to localhost to timeout when the server thinks it responded? - apache-httpclient-4.x

Question

I have a Java application running in AWS Beanstalk (Tomcat 8.5 + Apache httpd).
At one point the app calls a REST endpoint on localhost.
Occasionally I see a failure such as this in the log:
14:55:45 ... SEVERE: url[http://localhost/detail.api?id=200030599] timing=12.010 ...
That indicates that my CustomRestTemplate gave up waiting for a response, after 12 seconds.
However, looking a few lines up in the log, I see a log entry from the service endpoint:
{
"server_ts": "2020-08-19T14:55:33.890Z",
"remote_ip": "127.0.0.1",
"local_ip": "127.0.0.1",
"method": "GET",
"url": "/detail.api",
"query_string": "?id=200030599",
"protocol": "HTTP/1.1",
"http_status": 200,
"referer": null,
"user_agent": "Apache-HttpClient/4.5.2 (Java/1.8.0_252)",
"time_elapsed": 5,
"thread_name": "http-nio-8080-exec-20",
"host": "localhost",
}
That's my custom servlet logger showing a 5 millisecond response. This is logged from the outer wrapping Servlet Filter.
This problem is recurring but rare enough that I can't reproduce it. So I need to take an intellectual approach... develop a series of hypothesis and a test to disprove each one until the correct one is found.
What are some possible causes?
What have I tried so far
I wrote custom loggers so I could capture the timings shown above. Then I basically hit a brick wall as the timeout happens in some hidden dimension between the response being sent by the "server" (localhost endpoint) and the response being read by the client.
I see that the Apache log (from elasticbeanstack) also shows the local request:
127.0.0.1 (-) - - [19/Aug/2020:14:55:33 +0000] "GET /detail.api?id=200030599 HTTP/1.1" 200 4982 "-" "Apache-HttpClient/4.5.2 (Java/1.8.0_252)"

Answer 1

Finally I was able to reproduce the problem in a local test using JMeter.
Turns out the "hidden dimension" is lurking in a Servlet Filter:
#Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
throws IOException, ServletException {
long started = System.currentTimeMillis();
chain.doFilter(request, response);
long elapsed = System.currentTimeMillis() - started;
log(request, response, elapsed);
}
The log() calls contains a synchronous database insert which slows down and eventually fails under load. I assume the servlet engine keeps the connection open until this filter returns. Investigation moves on to finding a fix now.