Hystrix/Feign to solely react on HTTP status 429 - spring-cloud-netflix

I'm using Feign from the spring-cloud-starter-feign to send requests to a defined backend. I would like to use Hystrix as a circuit-breaker but for only one type of use-case: If the backend responds with a HTTP 429: Too many requests code, my Feign client should wait exactly one hour until it contacts the real backend again. Until then, a fallback method should be executed.
How would I have to configure my Spring Boot (1.5.10) application in order to accomplish that? I see many configuration possibilities but only few examples which are - in my opinion - unfortunately not resolved around use-cases.

This can be achieved by defining an ErrorDecoder and taking manual control of the Hystrix Circuit Breaker. You can inspect the response codes from the exceptions and provide your own fallback. In addition, if you wish to retry the request, wrap and throw your exception in a RetryException.
To meet your Retry requirement, also register a Retryer bean with the appropriate configuration. Keep in mind that using a Retryer will tie up a thread for the duration. The default implementation of Retryer does use an exponential backoff policy as well.
Here is an example ErrorDecoder taken from the OpenFeign documentation:
public class StashErrorDecoder implements ErrorDecoder {
#Override
public Exception decode(String methodKey, Response response) {
if (response.status() >= 400 && response.status() <= 499) {
return new StashClientException(
response.status(),
response.reason()
);
}
if (response.status() >= 500 && response.status() <= 599) {
return new StashServerException(
response.status(),
response.reason()
);
}
return errorStatus(methodKey, response);
}
}
In your case, you would react to 419 as desired.
You can forcibly open the Circuit Breaker setting this property at runtime
hystrix.command.HystrixCommandKey.circuitBreaker.forceOpen
ConfigurationManager.getConfigInstance()
.setProperty(
"hystrix.command.HystrixCommandKey.circuitBreaker.forceOpen", true);
Replace HystrixCommandKey with your own command. You will need to restore this circuit breaker back to closed after the desired time.

I could solve it with the following adjustments:
Properties in application.yml:
hystrix.command.commandKey:
execution.isolation.thread.timeoutInMilliseconds: 10_000
metrics.rollingStats.timeInMilliseconds: 10_000
circuitBreaker:
errorThresholdPercentage: 1
requestVolumeThreshold: 1
sleepWindowInMilliseconds: 3_600_000
Code in the respective Java class:
#HystrixCommand(fallbackMethod = "fallbackMethod", commandKey = COMMAND_KEY)
public void doCall(String parameter) {
try {
feignClient.doCall(parameter);
} catch (FeignException e) {
if (e.status() == 429) {
throw new TooManyRequestsException(e.getMessage());
}
}
}

Related

How to pause message processing for a certain period?

We use Apache Camel in Talend ESB Studio v6.4
In an ESB route, we consume JMS messages, process them then send them to an HTTP server. But that target server is down for maintainance every saturday from 6pm to 10pm.
How can we "pause" message consuming or message processing during that period ? I think quartz only works with file/ftp endpoints.
We could use a Processor component to check in Java if we are in the down period, but what to do after that ?
There are several ways to do this. One camel specific way to do it is through CamelControlBus. It takes in a routeId and performs an action (start/stop/resume etc) on it - Read more here to get an understanding Camel ControlBus
However, there is another approach that you can take. You can create a POJO bean that has 3 methods
shouldRouteStop() : to check the current time and decide if it should stop your route.
startRoute() : Starts a route if it is suspended
stopRoute() : Suspends a route if it is started
A simple implementation can be as follows:
public class ManagementBean {
public boolean shouldRouteStop() {
// Mocking the decision here
return new Random().nextBoolean();
}
public void startRoute(org.apache.camel.CamelContext ctx) throws Exception {
if (ctx.getRouteStatus("GenerateInvoices") == ServiceStatus.Suspended)
// replace the argument with your route Id
ctx.resumeRoute("GenerateInvoices");
}
public void stopRoute(org.apache.camel.CamelContext ctx) throws Exception {
if (ctx.getRouteStatus("GenerateInvoices") == ServiceStatus.Started)
// replace the argument with your route Id
ctx.suspendRoute("GenerateInvoices");
}
}
Make sure that the jms-route that you wish to control has a routeId and add this bean to your base/default CamelContext like this
main.bind("manageRouteBean", new ManagementBean());
Create another timer based route, that checks on each tick, if the route should be stopped or not and then suspends or resumes the route by routeId. This route can be implemented like below:
public class MonitoringRoute extends RouteBuilder {
#Override
public void configure() throws Exception {
onException(Exception.class).log(exceptionMessage().toString());
from("timer:time?period=10000")
.choice()
.when().simple("${bean:manageRouteBean?method=shouldRouteStop}")
.log("Route Should Stop")
.bean(ManagementBean.class, "stopRoute(*)")
.otherwise()
.log("Route Should Start")
.bean(ManagementBean.class, "startRoute(*)")
.end();
}
}
Note that startRoute and stopRoute take the argument as *. This is camel way of automatically binding parameters based on type.
Finally, you can add this route to the main camel context like : main.addRouteBuilder(new MonitoringRoute());
For a complete implementation, have a look at this github repo

Send a specific response, or at least a specific HTTP status code, using an exception

In Django, there are a couple of exceptions that are designed to be intercepted by the framework and turned into specific HTTP response codes, such as 404 Not Found and 403 Forbidden.
This is especially useful for request validation, because it allows you to factor out common validation logic into utility functions and cleanup your controller actions.
Whenever the utility functions decide that the current request must be aborted with a specific HTTP error code, they can do so by throwing the relevant exception, without any support code in the controller action, in the form of a return statement or a try/catch.
For example, given a tree of nested REST resources:
static mappings = {
"/authors" (resources: "author") {
"/sagas" (resources: "saga") {
"/books" (resources: "book") {
}
}
}
Then the URL pattern for the Book resource is /authors/$authorId/sagas/$sagaId/books/$id, which means that any of the show(), delete(), or update() actions in BookController have this signature and must include some boilerplate validation logic:
def actionName(int authorId, int sagaId, Book book) {
// -- common validation logic ----------
// fetch parent objects
def author = Author.get(authorId)
def saga = Saga.get(sagaId)
// check that they exists
if (author == null || saga == null || book == null) {
return render(status: NOT_FOUND)
}
// check consistency
if (book.author != author || book.saga != saga || saga.author != author) {
return render(status: BAD_REQUEST)
}
// -- end of commond code --------------
...
}
What is the Grails way of factoring this out into a common method, while still allowing it to terminate request processing whenever an exceptional condition occurs?
I would think the best way is a NotFoundException, ForbiddenException, BadRequestException, and so on, or maybe a generic exception that accepts a HTTP status code. Is there anything like it in Grails? If not, where is the best place to add it? A filter?
Edit: I see now that the standard method is to add an error controller with a matching URL pattern, such as:
"500" (controller: "error")
The problem with this is that Grails will still log full stacktraces for all exceptions, including those that are not programming errors. This spams log files with all sorts of useless tracebacks.
Is there a solution?
You catch the exception in the beforeInterceptor closure of your controller. I resolved this same problem by examining the exception thrown and then acting accordingly. For example:
class BaseController {
/**
* Define DRA exception handlers. This prevents the default Grails
* behavior of returning an HTTP 500 error for every exception.
*
* Instead the exceptions are intercepted and modified according to
* the exception that was thrown. These exceptions are not logged
* whereas application exceptions are.
*/
def beforeInterceptor = {
request.exceptionHandler = { exception ->
def cause = exception.cause
def exceptionBody = [:]
if(cause.class == BadRequestException) {
response.setStatus(HttpStatus.BAD_REQUEST.value()) // HTTP 400 BAD REQUEST
exceptionBody.httpStatus = HttpStatus.BAD_REQUEST.value()
exceptionBody.error = cause.message
}
// render the exception body, the status code is set above.
render exceptionBody as JSON
return true
}
}
}
In order to get this to work you will have to create an ErrorController or something where all server errors are processed and rendered. For example:
class ErrorController {
def serverError() {
def handler = request.exceptionHandler
if(handler) {
request.exceptionHandler = null
if(handler.call(request.exception)) {
return
}
}
}
I have tested this an it does work. I copied the code from a running project that I have been working on. You can build out the if statement in the beforeInterceptor to catch any type of Exception you wish.

MVVMCross ViewModel construction failure notifications

We've noticed a couple of times in our mobile applications that users have reported the application hanging or seeming to become unresponsive between views / rare crashes when switching between views. We've tracked down these cases to when our view model constructors throw uncaught exceptions.
We want to put a solution in place so that if a view model fails to construct for some reason then we can notify the user and provide some message that will be useful to us when it's logged through support.
I've been taking a look at doing this but haven't found a reliable way to achieve this.
The first thing we tried was at the IMvxViewModelLocator level. We already have a custom implementation of IMvxViewModelLocator so we've modified this. We allow all exceptions to be thrown and then we have an IErrorHandler interface which each platform implements. We then call this to attempt to show a dialog. This has proved to be unreliable and the dialog does not always display. Something along the lines of: (note - here ResolveViewModel will always return true or throw)
public override bool TryLoad(Type viewModelType, IMvxBundle parameterValues, IMvxBundle savedState, out IMvxViewModel viewModel)
{
try
{
return ResolveViewModel(viewModelType, parameterValues, savedState, out viewModel);
}
catch (Exception exception)
{
_errorHandler.HandleViewModelConstructionException(viewModelType, exception);
viewModel = null;
return false;
}
}
What we would ideally like to do is intercept any failure to construct a view model and then re-request an ErrorViewModel. We've tried to do this 2 ways:
1)
We've tried defining a custom IMvxViewDispatcher for each platform and we're trying to intercept failures as below but if an exception in the constructor is thrown we never get back this far:
public class TouchDispatcher : MvxTouchUIThreadDispatcher, IMvxViewDispatcher
{
private readonly IMvxTouchViewPresenter _presenter;
public TouchDispatcher(IMvxTouchViewPresenter presenter)
{
_presenter = presenter;
}
public bool ShowViewModel(MvxViewModelRequest request)
{
Action action = () =>
{
_presenter.Show(request);
};
try
{
bool success = RequestMainThreadAction(action);
return !success ? HandleError() : success;
}
catch (Exception)
{
return HandleError();
}
}
// Other bits
}
2)
We thought we might have more success at the presenter level. We modified our ViewPresenter for each platform and we have overridden void Show(MvxViewModelRequest request). This has not proved to be successful either as exceptions don't propagate back this far.
This leaves me thinking that maybe we are better attempting this at the IMvxViewModelLocator level again.
Has anyone found a way to reliably intercept failures to construct view models and then ideally re-request a different view model / present some dialog to the user?
It seems you've identified that the core of the problem is when: "view model constructors throw uncaught exceptions."
This is going to be slightly problematic as the ViewModel's are generally constructed during View lifecycle overrides like ViewDidLoad, OnCreate or NavigatedTo - which is generally after the Presenter has finished requesting presentation.
As you've already found an easy place to identify when ViewModel construction has failed is in a custom IMvxViewModelLocator - others likeIMvxViewModelLoader are also possible. This is probably the easiest place to catch the error and to trigger the error handling - you can then get hold of the IMvxViewDispatcher (or presenter) there in order to change the display. However, you will still need to make sure your Views can handle null created ViewModels - as the ViewDidLoad, etc calls will still need to complete.

How to count all HTTP requests sent, retries in?

Some use cases require being able to count the requests sent by the Apache API. For example, when massively requesting a web API, which API requires an authentication through an API key, and which TOS limits the requests count in time for each key.
Being more specific on the case, I'm requesting https://domain1/fooNeedNoKey, and depending on its response analyzed data, I request https://domain2/fooNeedKeyWithRequestsCountRestrictions. All sends of those 1-to-2-requests sequences, are performed through a single org.apache.http.impl.client.FutureRequestExecutionService.
As of now, depending on org.apache.httpcomponents:httpclient:4.3.3, I'm using those API elements:
org.apache.http.impl.client.FutureRequestExecutionService, to perform multi-threaded HTTP requests. It offers time metrics (how much time did an HTTP thread took until terminated), but no requests counter metrics
final CloseableHttpClient httpClient = HttpClients.custom()
// the auto-retry feature of the Apache API will retry up to 5
// times on failure, being also allowed to send again requests
// that were already sent if necessary (I don't really understand
// the purpose of the second parameter below)
.setRetryHandler(new StandardHttpRequestRetryHandler(5, true))
// for HTTP 503 'Service unavailable' errors, also retrying up to
// 5 times, waiting 500ms between each retry. Guessed is that those
// 5 retries are part of the previous "global" 5 retries setting.
// The below setting, when used alone, would allow to only enable
// retries for HTTP 503, or to get a greater count of retries for
// this specific error
.setServiceUnavailableRetryStrategy(new DefaultServiceUnavailableRetryStrategy(5, 500))
.build();, which customizes the Apache API retry behavior
Getting back to the topic :
A request counter could be created by extending the Apache API retry-related classes quoted before
Alternatively, an Apache API support unrelated ticket tends to indicate this requests-counter metrics could be available and forwarded out of the API, into Java NIO
Edit 1:
Looks like the Apache API won't permit this to be done.
Quote from the inside of the API, RetryExec not beeing extendable in the API code I/Os:
package org.apache.http.impl.execchain;
public class RetryExec implements ClientExecChain {
..
public CloseableHttpResponse execute(
final HttpRoute route,
final HttpRequestWrapper request,
final HttpClientContext context,
final HttpExecutionAware execAware) throws IOException, HttpException {
..
for (int execCount = 1;; execCount++) {
try {
return this.requestExecutor.execute(route, request, context, execAware);
} catch (final IOException ex) {
..
if (retryHandler.retryRequest(ex, execCount, context)) {
..
}
..
}
}
The 'execCount' variable is the needed info, and it can't be accessed since it's only locally used.
As well, one can extend 'retryHandler', and manually count requests in it, but 'retryHandler.retryRequest(ex, execCount, context)' is not provided with the 'request' variable, making it impossible to know on what we're incrementing a counter (one may only want to increment the counter for requests sent to a specific domain).
I'm out of Java ideas for it. A 3rd party alternative: having the Java process polling a file on disk, managed by a shell script counting the desired requests. Sure it will make a lot of disk read-accesses and will be a hardware killer option.
Ok, the work around was easy, the HttpContext class of the API is intended for this:
// optionnally, in case your HttpCLient is configured for retry
class URIAwareHttpRequestRetryHandler extends StandardHttpRequestRetryHandler {
public URIAwareHttpRequestRetryHandler(final int retryCount, final boolean requestSentRetryEnabled)
{
super(retryCount, requestSentRetryEnabled);
}
#Override
public boolean retryRequest(final IOException exception, final int executionCount, final HttpContext context)
{
final boolean ret = super.retryRequest(exception, executionCount, context);
if (ret) {
doForEachRequestSentOnURI((String) context.getAttribute("requestURI"));
}
return ret;
}
}
// optionnally, in addition to the previous one, in case your HttpClient has specific settings for the 'Service unavailable' errors retries
class URIAwareServiceUnavailableRetryStrategy extends DefaultServiceUnavailableRetryStrategy {
public URIAwareServiceUnavailableRetryStrategy(final int maxRetries, final int retryInterval)
{
super(maxRetries, retryInterval);
}
#Override
public boolean retryRequest(final HttpResponse response, final int executionCount, final HttpContext context)
{
final boolean ret = super.retryRequest(response, executionCount, context);
if (ret) {
doForEachRequestSentOnURI((String) context.getAttribute("requestURI"));
}
return ret;
}
}
// main HTTP querying code: retain the URI in the HttpContext to make it available in the custom retry-handlers code
httpContext.setAttribute("requestURI", httpGET.getURI().toString());
try {
httpContext.setAttribute("requestURI", httpGET.getURI().toString());
httpClient.execute(httpGET, getHTTPResponseHandlerLazy(), httpContext);
// if request got successful with no need of retries, of if it succeeded on the last send: in any cases, this is the last query sent to server and it got successful
doForEachRequestSentOnURI(httpGET.getURI().toString());
} catch (final ClientProtocolException e) {
// if request definitively failed after retries: it's the last query sent to server, and it failed
doForEachRequestSentOnURI(httpGET.getURI().toString());
} catch (final IOException e) {
// if request definitively failed after retries: it's the last query sent to server, and it failed
doForEachRequestSentOnURI(httpGET.getURI().toString());
} finally {
// restoring the context as it was initially
httpContext.removeAttribute("requestURI");
}
Solved.

Handling database connection exceptions with Linq to SQL and Rx

I am trying learn how to best use the Reactive Extensions library and have set up simple test WPF application to view a logging database table. In a ViewModel class I am populating an ObservableCollection with the first 100 log entries from a Linq to Sql DataContext and I'm trying to use Rx to keep the UI responsive.
The following snippet works unless the database is unavailable at which point the app throws an exception and crashes. Where would be the best place to handle database connection exceptions and why are they not handled by the OnError method of the Observer?
ObservableCollection<LogEntry> _logEntries = new ObservableCollection<LogEntry>();
DataContext dataContext = new DataContext( "connection string" );
(from e in dataContext.LogEntries
select e).Take( 100 ).ToObservable()
.SubscribeOn( Scheduler.ThreadPool )
.ObserveOnDispatcher()
.Subscribe( _logEntries.Add, ex => System.Diagnostics.Debug.WriteLine( ex.ToString() ) );
Try this instead of ToObservable:
public static IObservable<T> SafeToObservable(this IEnumerable<T> This)
{
return Observable.Create(subj => {
try {
foreach(var v in This) {
subj.OnNext(v);
}
subj.OnCompleted();
} catch (Exception ex) {
subj.OnError(ex);
}
return Disposable.Empty;
});
}
In general though, this isn't a great use of Rx since the data source isn't very easy to Rx'ify - in fact, the code will execute most of the work on the UI thread, send it out to random worker threads, then send it back (i.e. completely wasted work). Task + Dispatcher.BeginInvoke might suit you better here.