A "side effect" of using Netty is that you need to handle stuff you never thought about, like sockets closing and connection resets. A recurring theme is having your logs stuffed full of java.lang.IOException: Connection reset by peer.
What I am wondering about is how to handle these "correctly" from a web server perspective. AFAIK, this error simply means the other side has closed its socket (for instance, if reloading the web page or similar) while a request was sent to the server.
This is how we currently handle exceptions happening in our pipeline (I think it does not make full sense):
s, not the handler I have attached to the end of the pipeline.
current setup
pipeline.addLast(
new HttpServerCodec(),
new HttpObjectAggregator(MAX_CONTENT_LENGTH),
new HttpChunkContentCompressor(),
new ChunkedWriteHandler()
// lots of handlers
// ...
new InterruptingExceptionHandler()
);
pipeline.addFirst(new OutboundExceptionRouter());
the handler of exceptions
private class InterruptingExceptionHandler extends ChannelInboundHandlerAdapter {
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
final var id = ctx.channel().id();
// This needs to ge before the next section as the interrupt handler might shutdown the server before
// we are able to notify the client of the error
ctx.writeAndFlush(serverErrorJSON("A server error happened. Examine the logs for channel id " + id));
if (cause instanceof Error) {
logger.error(format("Error caught at end of pipeline in channel %s, interrupting!", id), cause);
ApplicationPipelineInitializer.this.serverInterruptHook.run();
} else {
logger.error(format("Uncaught user land exception in channel %s for request %s: ", id, requestId(ctx)), cause);
}
}
If some exception, like the IOException, is thrown we try and write a response back. In the case of a closed socket, this will then fail, right? So I guess we should try and detect "connection reset by peer" somehow and just ignore the exception silently to avoid triggering a new issue by writing to a closed socket ... If so, how? Should I try and do err instanceof IOException and err.message.equals("Connection reset by peer") or are there more elegant solutions? To me, it seems like this should be handled by some handler further down in the stack, closer to the HTTP handler
If you wonder about the OutboundExceptionRouter:
/**
* This is the first outbound handler invoked in the pipeline. What it does is add a listener to the
* outbound write promise which will execute future.channel().pipeline().fireExceptionCaught(future.cause())
* when the promise fails.
* The fireExceptionCaught method propagates the exception through the pipeline in the INBOUND direction,
* eventually reaching the ExceptionHandler.
*/
private class OutboundExceptionRouter extends ChannelOutboundHandlerAdapter {
#Override
public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception {
promise.addListener(ChannelFutureListener.FIRE_EXCEPTION_ON_FAILURE);
super.write(ctx, msg, promise);
}
}
Related
I'm using Feign from the spring-cloud-starter-feign to send requests to a defined backend. I would like to use Hystrix as a circuit-breaker but for only one type of use-case: If the backend responds with a HTTP 429: Too many requests code, my Feign client should wait exactly one hour until it contacts the real backend again. Until then, a fallback method should be executed.
How would I have to configure my Spring Boot (1.5.10) application in order to accomplish that? I see many configuration possibilities but only few examples which are - in my opinion - unfortunately not resolved around use-cases.
This can be achieved by defining an ErrorDecoder and taking manual control of the Hystrix Circuit Breaker. You can inspect the response codes from the exceptions and provide your own fallback. In addition, if you wish to retry the request, wrap and throw your exception in a RetryException.
To meet your Retry requirement, also register a Retryer bean with the appropriate configuration. Keep in mind that using a Retryer will tie up a thread for the duration. The default implementation of Retryer does use an exponential backoff policy as well.
Here is an example ErrorDecoder taken from the OpenFeign documentation:
public class StashErrorDecoder implements ErrorDecoder {
#Override
public Exception decode(String methodKey, Response response) {
if (response.status() >= 400 && response.status() <= 499) {
return new StashClientException(
response.status(),
response.reason()
);
}
if (response.status() >= 500 && response.status() <= 599) {
return new StashServerException(
response.status(),
response.reason()
);
}
return errorStatus(methodKey, response);
}
}
In your case, you would react to 419 as desired.
You can forcibly open the Circuit Breaker setting this property at runtime
hystrix.command.HystrixCommandKey.circuitBreaker.forceOpen
ConfigurationManager.getConfigInstance()
.setProperty(
"hystrix.command.HystrixCommandKey.circuitBreaker.forceOpen", true);
Replace HystrixCommandKey with your own command. You will need to restore this circuit breaker back to closed after the desired time.
I could solve it with the following adjustments:
Properties in application.yml:
hystrix.command.commandKey:
execution.isolation.thread.timeoutInMilliseconds: 10_000
metrics.rollingStats.timeInMilliseconds: 10_000
circuitBreaker:
errorThresholdPercentage: 1
requestVolumeThreshold: 1
sleepWindowInMilliseconds: 3_600_000
Code in the respective Java class:
#HystrixCommand(fallbackMethod = "fallbackMethod", commandKey = COMMAND_KEY)
public void doCall(String parameter) {
try {
feignClient.doCall(parameter);
} catch (FeignException e) {
if (e.status() == 429) {
throw new TooManyRequestsException(e.getMessage());
}
}
}
My MSMQ is located on a remote machine.
My code is as follows,
private void OnReceiveCompleted(object sender, ReceiveCompletedEventArgs e)
{
System.Messaging.Message msg = _queue.EndReceive(e.AsyncResult);
FireReceiveEvent(msg.Body); // Here msg.Body throws exception
_queue.BeginReceive();
}
I'm running this as a windows service, not sure if that makes a difference. But msg.Body throws a InvaliOperationException.
Infact most of the msg's properties are throwing exceptions. Any idea?
here is a screen shot
Why don't you try casting source parameter as MessageQueue
private void MessageQueueReceiveCompleted(Object source, ReceiveCompletedEventArgs asyncReceive)
{
try
{
//Get a handle to the Message Queue
MessageQueue messageQueue = (MessageQueue)source;
Message message = messageQueue.EndReceive(asyncReceive.AsyncResult);
if (message != null)
{
ProcessMsmqMessage(message.Body);
}
}
catch (Exception e)
{
Exception err = new Exception(String.Format("Error in QueueListener: {0}. Detail: {1}", queueName, e.Message), e);
OnListeningError(err);
}
finally{
messageQueue.BeginReceive();
}
}
OK. So after much work and reading, and banging my head against the wall, I found what the problem was.
REMOTE queues work very differently than local private queues.
You may ask, why? Well... this is probably a deficiency in MS' API.
Remote queues are very icky. They do not support a lot of feature that are available for regular local queues.
For example, in a remote queue unless it's transactional, you cannot do a BeginPeek. You cannot even check message.Body, because it would throw an error.
But that's not all. You cannot even accidentally subscribe to an event like OnPeekCompleted (even if you don't do a BeginPeek). The entire MessageQueue object goes crazy when you do that.
This silly reason was the reason for my headache.
In GWT when RPC fail due to any reason at that time onfailure() method execute at client side.
When onFailure() called at that time actual error is visible in browser's Networks response.
So, My Question is simple how can hide / Modify this actual error with some user friendly Error?
You can override onFailure() method to display what you need, but you cannot modify what a browser shows in the Network tab.
This is an example from my code (LoginException and VersionException are exceptions that my RPC calls throw when necessary):
#Override
public void onFailure(Throwable caught) {
if (caught instanceof LoginException) {
// Redirect a user to login page
Window.Location.assign("/");
} else if (caught instanceof IncompatibleRemoteServiceException ||
caught instanceof VersionException) {
/*
* Here I tell a user that a new version is available,
* so a user needs to refresh the page
*/
} else {
// Here I show a simple message about a connection error
}
}
I have a service method which does some operation inside a transaction.
public User method1() {
// some code...
Vehicle.withTransaction { status ->
// some collection loop
// some other delete
vehicle.delete(failOnError:true)
}
if (checkSomething outside transaction) {
return throw some user defined exception
}
return user
}
If there is a runtime exception we dont have to catch that exception and the transaction will be rolled back automatically. But how to determine that transaction rolled back due to some exception and I want to throw some user friendly error message. delete() call also wont return anything.
If I add try/catch block inside the transaction by catching the Exception (super class) it is not getting into that exception block. But i was expecting it to go into that block and throw user friendly exception.
EDIT 1: Is it a good idea to add try/catch arround withTransaction
Any idea how to solver this?? Thanks in advance.
If I understand you question correctly, you want to know how to catch an exception, determine what the exception is, and return a message to the user. There are a few ways to do this. I will show you how I do it.
Before I get to the code there are a few things I might suggest. First, you don't need to explicitly declare the transaction in a service (I'm using v2.2.5). Services are transactional by default (not a big deal).
Second, the transaction will automatically roll back if any exception occurs while executing the service method.
Third, I would recommend removing failOnError:true from save() (I don't think it works on delete()... I may be wrong?). I find it is easier to run validate() or save() in the service then return the model instance to the controller where the objects errors can be used in a flash message.
The following is a sample of how I like to handle exceptions and saves using a service method and try/catch in the controller:
class FooService {
def saveFoo(Foo fooInstance) {
return fooInstance.save()
}
def anotherSaveFoo(Foo fooInstance) {
if(fooInstance.validate()){
fooInstance.save()
}else{
do something else or
throw new CustomException()
}
return fooInstance
}
}
class FooController {
def save = {
def newFoo = new Foo(params)
try{
returnedFoo = fooService.saveFoo(newFoo)
}catch(CustomException | Exception e){
flash.warning = [message(code: 'foo.validation.error.message',
args: [org.apache.commons.lang.exception.ExceptionUtils.getRootCauseMessage(e)],
default: "The foo changes did not pass validation.<br/>{0}")]
redirect('to where ever you need to go')
return
}
if(returnedFoo.hasErrors()){
def fooErrors = returnedFoo.errors.getAllErrors()
flash.warning = [message(code: 'foo.validation.error.message',
args: [fooErrors],
default: "The foo changes did not pass validation.<br/>${fooErrors}")]
redirect('to where ever you need to go')
return
}else {
flash.success = [message(code: 'foo.saved.successfully.message',
default: "The foo was saved successfully")]
redirect('to where ever you need to go')
}
}
}
Hope this helps, or gets some other input from more experienced Grails developers.
Here are a few other ways I've found to get exception info to pass along to your user:
request.exception.cause
request.exception.cause.message
response.status
A few links to other relevant questions that may help:
Exception handling in Grails controllers
Exception handling in Grails controllers with ExceptionMapper in Grails 2.2.4 best practice
https://commons.apache.org/proper/commons-lang/javadocs/api-2.6/org/apache/commons/lang/exception/ExceptionUtils.html
Some use cases require being able to count the requests sent by the Apache API. For example, when massively requesting a web API, which API requires an authentication through an API key, and which TOS limits the requests count in time for each key.
Being more specific on the case, I'm requesting https://domain1/fooNeedNoKey, and depending on its response analyzed data, I request https://domain2/fooNeedKeyWithRequestsCountRestrictions. All sends of those 1-to-2-requests sequences, are performed through a single org.apache.http.impl.client.FutureRequestExecutionService.
As of now, depending on org.apache.httpcomponents:httpclient:4.3.3, I'm using those API elements:
org.apache.http.impl.client.FutureRequestExecutionService, to perform multi-threaded HTTP requests. It offers time metrics (how much time did an HTTP thread took until terminated), but no requests counter metrics
final CloseableHttpClient httpClient = HttpClients.custom()
// the auto-retry feature of the Apache API will retry up to 5
// times on failure, being also allowed to send again requests
// that were already sent if necessary (I don't really understand
// the purpose of the second parameter below)
.setRetryHandler(new StandardHttpRequestRetryHandler(5, true))
// for HTTP 503 'Service unavailable' errors, also retrying up to
// 5 times, waiting 500ms between each retry. Guessed is that those
// 5 retries are part of the previous "global" 5 retries setting.
// The below setting, when used alone, would allow to only enable
// retries for HTTP 503, or to get a greater count of retries for
// this specific error
.setServiceUnavailableRetryStrategy(new DefaultServiceUnavailableRetryStrategy(5, 500))
.build();, which customizes the Apache API retry behavior
Getting back to the topic :
A request counter could be created by extending the Apache API retry-related classes quoted before
Alternatively, an Apache API support unrelated ticket tends to indicate this requests-counter metrics could be available and forwarded out of the API, into Java NIO
Edit 1:
Looks like the Apache API won't permit this to be done.
Quote from the inside of the API, RetryExec not beeing extendable in the API code I/Os:
package org.apache.http.impl.execchain;
public class RetryExec implements ClientExecChain {
..
public CloseableHttpResponse execute(
final HttpRoute route,
final HttpRequestWrapper request,
final HttpClientContext context,
final HttpExecutionAware execAware) throws IOException, HttpException {
..
for (int execCount = 1;; execCount++) {
try {
return this.requestExecutor.execute(route, request, context, execAware);
} catch (final IOException ex) {
..
if (retryHandler.retryRequest(ex, execCount, context)) {
..
}
..
}
}
The 'execCount' variable is the needed info, and it can't be accessed since it's only locally used.
As well, one can extend 'retryHandler', and manually count requests in it, but 'retryHandler.retryRequest(ex, execCount, context)' is not provided with the 'request' variable, making it impossible to know on what we're incrementing a counter (one may only want to increment the counter for requests sent to a specific domain).
I'm out of Java ideas for it. A 3rd party alternative: having the Java process polling a file on disk, managed by a shell script counting the desired requests. Sure it will make a lot of disk read-accesses and will be a hardware killer option.
Ok, the work around was easy, the HttpContext class of the API is intended for this:
// optionnally, in case your HttpCLient is configured for retry
class URIAwareHttpRequestRetryHandler extends StandardHttpRequestRetryHandler {
public URIAwareHttpRequestRetryHandler(final int retryCount, final boolean requestSentRetryEnabled)
{
super(retryCount, requestSentRetryEnabled);
}
#Override
public boolean retryRequest(final IOException exception, final int executionCount, final HttpContext context)
{
final boolean ret = super.retryRequest(exception, executionCount, context);
if (ret) {
doForEachRequestSentOnURI((String) context.getAttribute("requestURI"));
}
return ret;
}
}
// optionnally, in addition to the previous one, in case your HttpClient has specific settings for the 'Service unavailable' errors retries
class URIAwareServiceUnavailableRetryStrategy extends DefaultServiceUnavailableRetryStrategy {
public URIAwareServiceUnavailableRetryStrategy(final int maxRetries, final int retryInterval)
{
super(maxRetries, retryInterval);
}
#Override
public boolean retryRequest(final HttpResponse response, final int executionCount, final HttpContext context)
{
final boolean ret = super.retryRequest(response, executionCount, context);
if (ret) {
doForEachRequestSentOnURI((String) context.getAttribute("requestURI"));
}
return ret;
}
}
// main HTTP querying code: retain the URI in the HttpContext to make it available in the custom retry-handlers code
httpContext.setAttribute("requestURI", httpGET.getURI().toString());
try {
httpContext.setAttribute("requestURI", httpGET.getURI().toString());
httpClient.execute(httpGET, getHTTPResponseHandlerLazy(), httpContext);
// if request got successful with no need of retries, of if it succeeded on the last send: in any cases, this is the last query sent to server and it got successful
doForEachRequestSentOnURI(httpGET.getURI().toString());
} catch (final ClientProtocolException e) {
// if request definitively failed after retries: it's the last query sent to server, and it failed
doForEachRequestSentOnURI(httpGET.getURI().toString());
} catch (final IOException e) {
// if request definitively failed after retries: it's the last query sent to server, and it failed
doForEachRequestSentOnURI(httpGET.getURI().toString());
} finally {
// restoring the context as it was initially
httpContext.removeAttribute("requestURI");
}
Solved.