I have some fairly straightforward code that's making a call to a URL, getting the response, and, if there's an entity, transferring the entity down.
The catch comes when I added a test of the entity content length. I can run the call through a proxy and see that the Content-Length header is getting set properly:
Content-Disposition: attachment; filename="something.zip";
Content-Length: 12390600
Content-Type: application/zip
So in my code I check that:
HttpResponse response = _client.execute(request);
HttpEntity entity = response.getEntity();
if (entity != null && entity.getContentLength() > 0) {
handleEntity(entity);
}
If I look at the response, all of the expected headers are there except for the content length. When I check the entity object, it returns -1 for the getContentLength() call. In the documentation, if your content length exceeds Long.MAX_VALUE it'll return a negative number, but that's not the case here.
So I've just removed this check for now but I'd really like to understand that what means in this context. The docs add a caveat that it will return the content length "if known" but in this case it appears that it SHOULD be known.
Thanks for any insight into what's going on here...
I suspect you are mixing up HTTP message content length and that of individual MIME body parts in a multipart content body
Related
I am automating Adobe InDesign to create documents using JSON data gathered from a web API with a SQL Server backend. I am using the Sockets object to make an HTTP 1.0 call to our server. Sometimes the response received is missing about 1700 characters from various points within the JSON string, yet when I call the same API endpoint using curl or Postman I get a complete and valid response.
The response should be about 150k characters long, and I'm using conn.read(99999999) to read it. In addition, the appearance of the end of the string looks correct, so I don't believe it's any kind of truncation problem.
The problem only seems to occur when I request a UTF-8 encoding. If I request ASCII I get a complete and valid response, but missing various Unicode characters. If I request BINARY I get a complete and valid response but the JavaScript/ExtendScript seems to be handling any multi-byte Unicode characters received as individual bytes, rather than as the Unicode characters we want to display.
Here is an illustration of the behavior I'm seeing, using bogus data...
"Expected" response...
[{"Id":1, "name":"Random Name", "Text":"A bunch of text", "AnotherId": 1}]
"Actual" response...
[{"Id":1, "name":"Random Name", "Text":"A bunc": 1}]
The problem first manifested itself as a JSON2 parsing error, for obvious reasons, but the root of it seems to be the fact that parts of the data are going missing in-transit.
So far we've only seen this problem when making the call using the InDesign Sockets object, and not every response exhibits this behavior.
Any help or insights you could offer would be appreciated.
Here is the function I'm using to call for data...
function httpRequest(url, encoding) {
try {
var response = "";
var hostName = getHostFromUrl(url);
var pathAndQuery = getPathAndQueryFromUrl(url);
var httpGet = "GET ";
httpGet += pathAndQuery;
httpGet += " HTTP/1.0\r\nHost: ";
httpGet += hostName;
httpGet += "\r\n";
var conn = new Socket;
conn.timeout = 30;
//conn.encoding = encoding || "UTF-8";
//conn.charset = "UTF-16";
if (conn.open(hostName + ":80", encoding || "UTF-8")) {
// send a HTTP GET request
conn.writeln(httpGet);
// and read the server's response
response = conn.read(99999999);
conn.close();
}
return parseHttpResponse(response);
}
catch (e) {
$.writeln(e);
$.global.alert("There was a problem making an HTTP Request: " + e);
return null;
}
}
It turns out my handling of the HTTP response was too simplistic and needed extra logic to handle Unicode characters properly.
The solution, in my case, was to use the GetURL method made available by Kris Coppieter here.
We have decided to only allow requests with a Content-Type header "application/json".
So, whenever we receive a request with an alternative or missing Content-Type header, we throw an HttpError. This should return a 400 response containing a JSON ResponseStatus body with relevant info.
However, if a Content-Type text/plain is sent, we throw an HttpError, but the response's Content-Type is text/plain and content-length: 0. I expected ServiceStack's ResponseStatus to be returned. The ResponseStatus is returned fine if I add an Accept application/json header to the request.
I executed the request using Postman. Fiddler4 screen shot:
I am aware that Postman adds the Accept / header. So my question is: How can I ensure that a thrown HttpError always return the ResponseStatus as JSON, no matter the request's Accept header?
The SetConfig:
SetConfig(new HostConfig
{
EnableFeatures = Feature.All.Remove( Feature.Html | Feature.Csv | Feature.Jsv | Feature.Xml | Feature.Markdown | Feature.Razor | Feature.Soap | Feature.Soap11 | Feature.Soap12 | Feature.PredefinedRoutes),
DebugMode = false,
DefaultContentType = MimeTypes.Json
});
As I understand it, the DefaultContentType is only used whenever there isn't an Accept header in the request.
The PreRequestFilter:
PreRequestFilters.Add((request, response) =>
{
if (request.Verb.Equals("OPTIONS"))
response.EndRequest();
if (request.GetHeader("Content-Type") == null || !request.GetHeader("Content-Type").Equals(MimeTypes.Json))
throw new HttpError((int)HttpStatusCode.BadRequest, "Bad request", "Expected a Content-Type header with an application/json value but found none. See http://docsdomain.com/ for any required headers.");
});
The HTTP Accept header is what the client uses to indicate what Response Type should be returned but you can override this to always return JSON by adding a Global Request Filter and explicitly setting the ResponseContentType, e.g:
GlobalRequestFilters.Add((req,res,dto) =>
req.ResponseContentType = MimeTypes.Json);
If the Accept Header doesn't specify a specific Response Type it will default to using the PreferredContentTypes which you can change by:
SetConfig(new HostConfig {
PreferredContentTypes = new []{ MimeTypes.Json }.ToList(),
});
I am trying to pass parameters to a server and extract the report in csv format. So the code i have has PUT/GET/POST in the order. I could get GET and POST work, but when i add PUT there is no error just blank screen.
String output1 = null;
URL url = new URL("http://<servername>/biprws/raylight/v1/documents/12345/parameters");
HttpURLConnection conn1 = (HttpURLConnection) url.openConnection();
conn1.setRequestMethod("PUT");
conn1.setRequestProperty("Accept", "application/json");
conn1.setRequestProperty("Content-Type", "application/json; charset=utf-8");
conn1.setDoInput(true);
conn1.setDoOutput(true);
String body = "<parameters><parameter><id>0</id><answer><values><value>EN</value></values></answer></parameter></parameters>";
int len1 = body.length();
conn1.setRequestProperty("Content-Length", Integer.toString(len1));
conn1.connect();
OutputStreamWriter out1 = new OutputStreamWriter(conn1.getOutputStream());
out1.write(body, 0, len1);
out1.flush();
What i am trying to do is pass parameter EN to the report and refresh it, take the output in csv using GET. POST is used for login to the server. I could make GET and POST work and get the output in CSV but not refreshed one.
Appreciate very much any help here.
Thanks,
Ak
What is the response code from the server when using PUT?
A PUT may not actually return a body to display on the screen; often times a PUT will only return a 200 or 204 response code. 204 would clearly mean that the server took the data and applied it, but is not sending you anything back, 200/201 may include a response, but maybe not. It depends on the folks who implemented the API.
https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html (section 9.6)
Should a RESTful 'PUT' operation return something
I am using Play scala WS to send a REST api call to a web server and sometimes get an exception error. Json is sent to the server and the response from the server could be one of the following.
Server returns a valid Json response.
Server returns "No valid Json found"
Server returns an error web page that triggers the exception error com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
How do I modify the code below to get the contents of the web page without the exception error?
import play.api.libs.ws._
var tempText = Helpers.await(WS.url("localhost:9000/someApi").post(jsonToSend)).body
println(tempText)
tempJson = Json.parse(tempText)
println(tempJson)
Much depends on how "correct" that downstream API server is.
In a perfect world, we could probably assert the following facts:
Success case => HTTP status is 200, HTTP Content-Type header is application/json
"No valid Json found" => HTTP status is 404 or similar non-200, HTTP Content-Type header is application/json
"Error web page" => HTTP status is not 200, Content-Type is text/html
If the above assertions are all true, then we can simply put a little bit of "protection" around our response-handling rather than just jumping in and trying to parse it as JSON:
val futureOptionalResult = WS.url("localhost").post("...").map { response =>
response.status match {
case 200 => {
println(response.body)
println(response.json)
Some(response.json)
}
case _ => {
println(s"Not OK: ${response.status} - body is: ${response.body}")
None
}
}
}
Some notes:
Doing it asynchronously is just as easy as using an await and scales better
response.json returns the same thing as your explicit Json.parse on the body
I'm returning an Option[JsValue] that holds the returned JSON if it worked
If the above assumptions are not true, deeper inspection of the Content-Type header, finer-grained switching on the status value and/or other attributes of the response will probably be needed. Good luck!
I am using webapp2 for development in App Engine. What I would like to do is to send a custom JSON formatted response in case of an error. For example when the request length is larger that a threshold, to respond with HTTP 400 and response body
{'error':'InvalidMessageLength'}
In webapp2, there is the option to assign error handlers for certain exceptions. For example:
app.error_handlers[400] = handle_error_400
Where handle_error_400 is the following:
def handle_error_400(request, response, exception):
response.write(exception)
response.set_status(400)
When webapp2.RequestHandler.abort(400) is executed, the above code is executed.
How is it possible to have different response formats (HTML and JSON) dynamically based on the above setup? That is, how it is possible to call different versions of handle_error_400 function?
Here is a fully working example that demonstrates how to have the same error handler for all kind of errors and if your URL starts with /json then the response will be an application/json (use your imagination on how you could make a good use of the request object to decide what kind of response you should provide):
import webapp2
import json
def handle_error(request, response, exception):
if request.path.startswith('/json'):
response.headers.add_header('Content-Type', 'application/json')
result = {
'status': 'error',
'status_code': exception.code,
'error_message': exception.explanation,
}
response.write(json.dumps(result))
else:
response.write(exception)
response.set_status(exception.code)
app = webapp2.WSGIApplication()
app.error_handlers[404] = handle_error
app.error_handlers[400] = handle_error
In the above example you can easily test the different behaviours by visting the following URLs that will return a 404 which is the easiest error to test:
http://localhost:8080/404
http://localhost:8080/json/404