I'm trying to debug a call made from a Java desktop application to a Java server, which goes via Apache http server. The call is an http json call, I want to log the request and response on Apache. I have enabled the dump io module and can now see the request and response, but the response is formatted as binary:
16:58:10 GMT\r\nX-XSS-Protection: 1; mode=block\r\nX-Content-Type-Options:
nosniff\r\nReferrer-Policy: no-referrer\r\nbreadcrumbId: ID-machine-name-here-33747-
1626280202382-10-673\r\nContent-Type: application/json;charset=UTF-8\r\nSet-Cookie:
JSESSIONID=88F411906EC0D; Path=/; HttpOnly\r\nVary: Accept-Encoding\r\nContent-
Encoding: gzip\r\nSet-Cookie: ROUTEID=.1; path=/app-path\r\nContent-Length:
61\r\nConnection: close\r\n\r\n
[Fri Jul 16 17:58:07 2021] [debug] mod_dumpio.c(74): mod_dumpio: dumpio_out (data-
HEAP): \xabV\xaa\xc8\xcdQ\xb2R\xb2)JMK-
J\xcdKNuI,I\x0cIL\xcaIuIM\xcb\xcc\xcb,\xc9\xcc\xcf\xd3\xb7S\xaa\x05
Why is this showing as binary? Is there a way I can get this logged as text?
Why is this showing as binary?
Everything is binary, but the reason it's not showing as plain text is because this is the actual body that's sent, and the body is gzipped.
Is there a way I can get this logged as text?
You could turn off gzipping.
Related
I'm trying to set up HAProxy to server a static JSON file on 504 errors. To test, we've set up the configuration file to timeout after 10 seconds, and to use the errorfile option:
defaults
log global
mode http
retries 3
timeout client 10s
timeout connect 10s
timeout server 10s
option tcplog
balance roundrobin
frontend https
maxconn 2000
bind 0.0.0.0:9000
errorfile 504 /home/user1/test/error.json
acl employee-api-service path_reg /employee/api.*
use_backend servers-employee-api if employee-api-service
backend servers-employee-api
server www.server.com 127.0.0.1:8000
Effectively, I'm trying to serve JSON instead of HTML on a timeout, so the backend service can fail gracefully. However, on testing, we could not get anything, neither HTML or JSON. On looking at the response, it simply says it failed, with no status code. Is my setup correct for errorfile? Does HAProxy 1.5 support this?
According to the documentation of errorfile:
<file> designates a file containing the full HTTP response. It is
recommended to follow the common practice of appending ".http" to
the filename so that people do not confuse the response with HTML
error pages, and to use absolute paths, since files are read
before any chroot is performed.
So, the file should contain a complete HTTP response but you're trying to serve JSON only.
The documentation further says that:
For better HTTP compliance, it is
recommended that all header lines end with CR-LF and not LF alone.
The example configuration, for example,
errorfile 503 /etc/haproxy/errorfiles/503sorry.http
shows the common practice of .http extension for the error file.
You can find samples of some default error files here.
Sample (504.http):
HTTP/1.0 504 Gateway Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
So, in your scenario, 504.http would be like this:
HTTP/1.0 504 Gateway Time-out
Cache-Control: no-cache
Connection: close
Content-Type: application/json
{
"message": "Gateway Timeout"
}
Also, you need to keep the file size under limit i.e. BUFSIZE (8 or 16 KB) as described in the documentation.
There might be some error logs for not serving your JSON file. You might want to look at HAProxy's logs again thoroughly. Just to be sure.
I'm using angular with i18n translations in json files like de.json and en.json. In my production environment (nginx) I have the problem that these JSON files are cached by the web browser. After an upgrade, Chrome will not download the new version of the current json file even though the date header has changed.
Request Information (Chrome):
Request URL: https://[my-site]/assets/i18n/de.json
Request Method: GET
Status Code: 200 (from disk cache)
Remote Address: X.X.X.X:443
Referrer Policy: no-referrer-when-downgrade
Response Headers:
content-encoding: gzip
content-type: application/json
date: Fri, 15 Feb 2019 09:04:42 GMT
etag: W/"5c62bf4d-2aea"
last-modified: Tue, 12 Feb 2019 12:42:53 GMT
server: nginx/1.14.0 (Ubuntu)
status: 304
Does anyone have experience with this problem and can help me?
Not specifically a fix for Angular/nginx, but a practice I often use is to append a query string parameter to the resource when you load it. For me, this is typically derived from version number of the .js file / application, e.g. using it as a seed for a RNG
So, instead of: <script src="/assets/de.json" />
use
<script src="/assets/de.json?_=12345" />
Bonus points- in your Angular application, you can keep track of what version of the assets that you want to include, meaning that you can release new asset files without having clients immediately update to them if they have them in local cache (although note that new clients will get the new version regardless)
I am sending a GET request for JSON data to my proxy server via lighttpd:
Firefox -> lighttpd -> proxy server
Lighttpd is configured for gzipped json via the following line in lighttpd.conf:
compress.filetype = ( "application/json", "application/javascript", …)
The proxy server gzips the JSON, adds
Content-Type: application/json
Content-Encoding: "gzip"
to the HTTP response header, and sends the response to lighttpd, which forwards it to the client.
When Firefox receives the response, I get the following error:
SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data
So there is something wrong with the compressed JSON (or its interpretation). I checked that the compressed JSON data has the same hash as if I save the uncompressed JSON to file and gzip it on the command line.
Can anyone offer any insight as to why Firefox won't interpret the compressed JSON?
I'm new to this with a very low background so bear with me. I want to debug my issue of prematurely terminating loading the page from the reverse proxied site as it request something like filename.json file. I noticed in the request header it has Content-Type:application/x-www-form-urlencoded, and the response headers are
$WSEP:
Cache-Control:no-cache
Connection:Keep-Alive
Content-Encoding:gzip
Content-Language:en-US
Content-Length:55
Content-Type:text/html;charset=UTF-8
Date:Sun, 12 Jun 2016 04:54:18 GMT
Expires:Thu, 01 Jan 1970 00:00:00 GMT
Keep-Alive:timeout=5, max=97
Server:Apache
Vary:Accept-Encoding
X-Cnection:Close
but I'm wondering why it's text/html content-type when it's json, so I tried manipulating the response with application/json and even trying to instead of downloading the page from the remote server, I made it download locally from my server /var/www/html by creating dummy json file. I was thinking because this is json file it should be application/json not text/html as Content-Type. But the issue still exists. It just stops when it reached requesting that json file.
Now my question is, how can I make sure that requesting to that json is the culprit? Or more appropriately, how can I debug the mod_perl. I tried adding
use warning;
in the code
PerlWarn On
in the apache config but still I don't see any warning/errors that will guide me to debug.
Can you give me advice on how to do find the fault?
By the way, the site of that json file being requested returns 404 but I assume that the proxy server should ignore it and continue downloading other files.
I am trying to use http://manifest-validator.com/ to check my cache manifest file by URI. It has been failing with the message:
Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request POST /validate.
Reason: Error reading from remote server
I have tried removing my robots.txt file in case Disallow: / was causing problems.
Has anyone else experienced this Proxy Error?
Is this something that I can fix or is it an error with manifest-validator.com ?
In short
The reason for your error in this validator is a missing content-type attribute in your http response headers. Add one like text/cache-manifest and it will work.
Explaining the details
First of all, here are the response headers sent by your server. Note the missing content-type:
robert#robert-ubuntu:~$ curl -I -X HEAD http://nucreativetesting.co.uk/cache-test/test003/manifest.appcache
HTTP/1.1 200 OK
Date: Tue, 07 Oct 2014 16:25:59 GMT
Server: Apache/2.4.9 (Unix)
Last-Modified: Mon, 11 Aug 2014 13:45:07 GMT
ETag: "46c-5005ac34a2331"
Accept-Ranges: bytes
Content-Length: 1132
Why the proxy error?
The validator you are using is a NodeJS project, proxied by an apache webserver. The reason you get an error is that the apache waits for NodeJS to generate a response but it's hanging. After some time it gets a timeout and returns your error.
So why is it hanging?
In the current implementation of this validator you find the following code [1]:
try {
contentType = res.headers['content-type'].split(';')[0];
if ( ['text/plain', 'text/cache-manifest'].indexOf( contentType ) === -1 ) {
callback('ERR_MANIFEST_MIMETYPE');
return;
}
} catch(e) {
// TODO: Debug statement in production, had some strange output in the error logs.
// Should be removed again
console.error('manifest.js: Content-Type');
return;
}
We see that res.headers['content-type'] should get split, but if there's no content type in your request header, this will raise an Exception because res.headers['content-type'] is undefined. The Exception itself is just logged to the servers output but isn't handled with their callback() function. Thats the reason the request is hanging till timeout.
Conclusion
It might be a bug or bad error handling on their side or just very unusual to not have a content type in your header. I don't know. But it should work with a content type ;).
References
[1]: https://github.com/fhemberger/manifest-validator/blob/6d82bc4660c4daaa131fee3c19a88ae6e462a44b/app/lib/manifest.js#L63 code producing your error