Making sense of Apache HttpClient wire log from response - apache-httpclient-4.x

Question

My application is using Apache HTTPClient 4.3.5 that sends HTTP request and gets response.
I am trying to figure out what response the app received.
Below is the log snippet -
[Jan 04 2015 05:38:14.109 AM] DEBUG wire:72 - http-outgoing-2 >> "POST /app HTTP/1.1[\r][\n]"
[Jan 04 2015 05:38:14.109 AM] DEBUG wire:72 - http-outgoing-2 >> "User-Agent: Mozilla/5.0[\r][\n]"
[Jan 04 2015 05:38:14.109 AM] DEBUG wire:72 - http-outgoing-2 >> "Content-Length: 47[\r][\n]"
[Jan 04 2015 05:38:14.109 AM] DEBUG wire:72 - http-outgoing-2 >> "Content-Type: application/x-www-form-urlencoded[\r][\n]"
[Jan 04 2015 05:38:14.109 AM] DEBUG wire:72 - http-outgoing-2 >> "Host: test.com[\r][\n]"
[Jan 04 2015 05:38:14.109 AM] DEBUG wire:72 - http-outgoing-2 >> "Connection: Keep-Alive[\r][\n]"
[Jan 04 2015 05:38:14.110 AM] DEBUG wire:72 - http-outgoing-2 >> "Accept-Encoding: gzip,deflate[\r][\n]"
[Jan 04 2015 05:38:14.110 AM] DEBUG wire:72 - http-outgoing-2 >> "[\r][\n]"
[Jan 04 2015 05:38:14.489 AM] DEBUG wire:72 - http-outgoing-2 << "HTTP/1.1 200 OK[\r][\n]"
[Jan 04 2015 05:38:14.489 AM] DEBUG wire:72 - http-outgoing-2 << "Server: nginx[\r][\n]"
[Jan 04 2015 05:38:14.490 AM] DEBUG wire:72 - http-outgoing-2 << "Date: Sun, 04 Jan 2015 10:37:31 GMT[\r][\n]"
[Jan 04 2015 05:38:14.490 AM] DEBUG wire:72 - http-outgoing-2 << "Content-Type: text/html;charset=UTF-8[\r][\n]"
[Jan 04 2015 05:38:14.490 AM] DEBUG wire:72 - http-outgoing-2 << "Transfer-Encoding: chunked[\r][\n]"
[Jan 04 2015 05:38:14.490 AM] DEBUG wire:72 - http-outgoing-2 << "Connection: keep-alive[\r][\n]"
[Jan 04 2015 05:38:14.490 AM] DEBUG wire:72 - http-outgoing-2 << "Vary: Accept-Encoding[\r][\n]"
[Jan 04 2015 05:38:14.490 AM] DEBUG wire:72 - http-outgoing-2 << "Pragma: no-cache[\r][\n]"
[Jan 04 2015 05:38:14.490 AM] DEBUG wire:72 - http-outgoing-2 << "Expires: Thu, 01 Jan 1970 00:00:00 GMT[\r][\n]"
[Jan 04 2015 05:38:14.491 AM] DEBUG wire:72 - http-outgoing-2 << "Cache-Control: no-cache[\r][\n]"
[Jan 04 2015 05:38:14.491 AM] DEBUG wire:72 - http-outgoing-2 << "Cache-Control: no-store[\r][\n]"
[Jan 04 2015 05:38:14.491 AM] DEBUG wire:72 - http-outgoing-2 << "Set-Cookie: JSESSIONID=0C1444D7E0FA6B0F3CCE20AFBA28237E; Path=/; HttpOnly[\r][\n]"
[Jan 04 2015 05:38:14.491 AM] DEBUG wire:72 - http-outgoing-2 << "Front-End-Https: on[\r][\n]"
[Jan 04 2015 05:38:14.491 AM] DEBUG wire:72 - http-outgoing-2 << "Content-Encoding: gzip[\r][\n]"
[Jan 04 2015 05:38:14.491 AM] DEBUG wire:72 - http-outgoing-2 << "[\r][\n]"
[Jan 04 2015 05:38:14.491 AM] DEBUG wire:72 - http-outgoing-2 << "37[\r][\n]"
[Jan 04 2015 05:38:14.492 AM] DEBUG wire:72 - http-outgoing-2 << "[0x1f][0x8b][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x3][0xb3])J-.[0xc8][0xcf]+N[0xb5][0xe3][0xb2][0x81]1[0xe3][0x93][0xf3]S[0x80][0xfc][0xe0][0xd2][0xe4][0xe4][0xd4][0xe2]b.[0x1b]}4[0x9][0x84][0x80][0x1d][0x17][0x0]<Xn]#[0x0][0x0][0x0][\r][\n]"
[Jan 04 2015 05:38:14.492 AM] DEBUG wire:72 - http-outgoing-2 << "0[\r][\n]"
[Jan 04 2015 05:38:14.492 AM] DEBUG wire:72 - http-outgoing-2 << "[\r][\n]"
At first, I thought "[0x1f][0x8b][0x8][0x0][0x0]..." is hex string but there are non-hex string such as ")J-." and "+N." and each debug line is ending with "[\r][\n]". How would I be able to determine what response the app received.
I have spent quite few hours trying to find the answer but was unsuccessful.
Does it have to do anything with gzip compression or chunked transfer encoding?
Your help would be greatly appreciated if you could answer it.
Thanks,
James

Answer 1

HttpClient wire log replaces character 10 with [\n], character 13 with [\r], characters < 32 and > 127 (non-printable) with [0xN] where N is char's hexadecimal representation.
The reason you are seeing a lot of non-printable chars is indeed GZIP content encoding.

Answer 2

This is an old post, but I've just spent too much time looking for code to take such a log entry and decode/unzip it to a readable string. So here's my contribution:
public void parseHexReponse() throws IOException {
// make sure NOT to delete any spaces which are part of the payload!
final String responseHex = "[0x1f]...[0x0]"; // paste your response here
final String[] parts = responseHex.split("[\\[\\]]+");
final byte[] data = new byte[responseHex.length() / 3];
int idx = 0;
for (final String s : parts) {
if (s.startsWith("0x")) {
data[idx] = (byte) Integer.parseInt(s.substring(2), 16);
idx++;
} else if ("\\n".equals(s)) {
data[idx] = 10;
idx++;
} else if ("\\r".equals(s)) {
data[idx] = 13;
idx++;
} else {
for (final byte b : s.getBytes()) {
data[idx] = b;
idx++;
}
}
}
try (GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(Arrays.copyOfRange(data, 0, idx)));
ByteArrayOutputStream out = new ByteArrayOutputStream()) {
final byte[] buffer = new byte[1024];
int len;
while ((len = gis.read(buffer)) > 0) {
out.write(buffer, 0, len);
}
System.out.println(new String(out.toByteArray()));
}
}

Answer 3

http://moznion.github.io/PrettyWireLogViewer/
This tool decodes ugly httpclient format to human-readable.