laravel queue (retry jobs 5 times) & mark job as failed "manually" - exception

I've got a queue running some callback requests to endpoint of my users.
Here's the code of my queue.
public function handle()
{
//send webrequest here....
//check the response of user backend
if ($res->getStatusCode() != 200 || $res->getBody()->getContents() != "*received*")
throw new Exception('callback url not reachable');
}
public function failed(Exception $exception)
{
//check tries and try again if needed
//check if job failed for 5 times
//if not ->retry again in 5 minutes, increment the times tried
//if yes ->disable API access, send email
Log::info("user email send, callback disabled!");
}
How to let the job fail (my current exception makes the whole job end), when webrequest answer is != "received" and check if the certain job failed 5 times?
If a job failed it should be tried again in 5 minutes.
I don't understand the doc regarding these points.

I know it's too late but yesterday I had the same problem with the queue and when I throw a new exception the queue run it again and again until my VM hanged up. actually, you did right the only thing you have to add to your code is define a tries variable to the public area of your job class and it should run only for example 5 times and pushed to failed job ;)
class NewCustomer implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public $tries = 5;
.
.
.

Related

How should "Connection reset by peer" be handled in Netty?

A "side effect" of using Netty is that you need to handle stuff you never thought about, like sockets closing and connection resets. A recurring theme is having your logs stuffed full of java.lang.IOException: Connection reset by peer.
What I am wondering about is how to handle these "correctly" from a web server perspective. AFAIK, this error simply means the other side has closed its socket (for instance, if reloading the web page or similar) while a request was sent to the server.
This is how we currently handle exceptions happening in our pipeline (I think it does not make full sense):
s, not the handler I have attached to the end of the pipeline.
current setup
pipeline.addLast(
new HttpServerCodec(),
new HttpObjectAggregator(MAX_CONTENT_LENGTH),
new HttpChunkContentCompressor(),
new ChunkedWriteHandler()
// lots of handlers
// ...
new InterruptingExceptionHandler()
);
pipeline.addFirst(new OutboundExceptionRouter());
the handler of exceptions
private class InterruptingExceptionHandler extends ChannelInboundHandlerAdapter {
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
final var id = ctx.channel().id();
// This needs to ge before the next section as the interrupt handler might shutdown the server before
// we are able to notify the client of the error
ctx.writeAndFlush(serverErrorJSON("A server error happened. Examine the logs for channel id " + id));
if (cause instanceof Error) {
logger.error(format("Error caught at end of pipeline in channel %s, interrupting!", id), cause);
ApplicationPipelineInitializer.this.serverInterruptHook.run();
} else {
logger.error(format("Uncaught user land exception in channel %s for request %s: ", id, requestId(ctx)), cause);
}
}
If some exception, like the IOException, is thrown we try and write a response back. In the case of a closed socket, this will then fail, right? So I guess we should try and detect "connection reset by peer" somehow and just ignore the exception silently to avoid triggering a new issue by writing to a closed socket ... If so, how? Should I try and do err instanceof IOException and err.message.equals("Connection reset by peer") or are there more elegant solutions? To me, it seems like this should be handled by some handler further down in the stack, closer to the HTTP handler
If you wonder about the OutboundExceptionRouter:
/**
* This is the first outbound handler invoked in the pipeline. What it does is add a listener to the
* outbound write promise which will execute future.channel().pipeline().fireExceptionCaught(future.cause())
* when the promise fails.
* The fireExceptionCaught method propagates the exception through the pipeline in the INBOUND direction,
* eventually reaching the ExceptionHandler.
*/
private class OutboundExceptionRouter extends ChannelOutboundHandlerAdapter {
#Override
public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception {
promise.addListener(ChannelFutureListener.FIRE_EXCEPTION_ON_FAILURE);
super.write(ctx, msg, promise);
}
}

How to handle Modbus exception 0x5

I'm writing a Modbus client program using Qt5 and the QModbusTcpClient class. Here the code I'm using for open a connection and read something:
QModbusClient *_modbus;
bool ModbusMaster::open(QString host, int port)
{
// Disconnect and delete any existing instance
if (_modbus)
{
_modbus->disconnectDevice();
delete _modbus;
}
// Create and open the new connection
_modbus = new QModbusTcpClient(this);
_modbus->setConnectionParameter(QModbusDevice::NetworkPortParameter, port);
_modbus->setConnectionParameter(QModbusDevice::NetworkAddressParameter, host);
_modbus->setTimeout(250);
_modbus->setNumberOfRetries(1);
return _modbus->connectDevice();
}
bool ModbusMaster::read(QModbusDataUnit::RegisterType type, int startAddress, quint16 count)
{
if (!_modbus) return false;
if (_modbus->state() != QModbusDevice::ConnectedState) return false;
QModbusDataUnit req(type, startAddress, count);
if (auto *reply = _modbus->sendReadRequest(req, _id))
{
if (!reply->isFinished()) connect(reply, &QModbusReply::finished, this, &ModbusMaster::readReady);
else delete reply;
return true;
}
return false;
}
void ModbusMaster::readReady()
{
auto reply = qobject_cast<QModbusReply *>(sender());
if (!reply) return;
reply->deleteLater();
if (reply->error() == QModbusDevice::NoError)
{
// do something
}
else if (reply->error() == QModbusDevice::ProtocolError)
{
qDebug() << QString("Read response error: %1 (Mobus exception: 0x%2)").
arg(reply->errorString()).
arg(reply->rawResult().exceptionCode(), -1, 16);
} else {
qDebug() << QString("Read response error: %1 (code: 0x%2)").
arg(reply->errorString()).
arg(reply->error(), -1, 16);
}
}
Sometimes when I read something from the remote device it happens the device returns the exception 0x5. Reading the official Modbus documentation, at page 48 I read:
Specialized use in conjunction with programming
commands.
The server has accepted the request and is
processing it, but a long duration of time will be
required to do so. This response is returned to
prevent a timeout error from occurring in the
client. The client can next issue a Poll Program
Complete message to determine if processing is
completed.
[bold is mine]
I cannot find a description of this "Poll Program Complete message" that seems I must use to handle the exception 0x5.
Did I search wrong? Is there another way to handle this exception?
It depends on type of an equipment, you are working with. You just have to follow the logic, described in equipment mans for this particular exception.
In general there is no special 'Program Complete' event. That means, as it is written for 0x5 - "Specialized use in conjunction with programming commands.". So you just have to poll (read) some flag from your device meaning the internal process in device, which caused this exception, is complete.
Just as an example, I've met with such exception in relay protection device, which issued it when it has been in a process of writing a disturbance record. I had just to check for that record readiness in some time.

Hystrix/Feign to solely react on HTTP status 429

I'm using Feign from the spring-cloud-starter-feign to send requests to a defined backend. I would like to use Hystrix as a circuit-breaker but for only one type of use-case: If the backend responds with a HTTP 429: Too many requests code, my Feign client should wait exactly one hour until it contacts the real backend again. Until then, a fallback method should be executed.
How would I have to configure my Spring Boot (1.5.10) application in order to accomplish that? I see many configuration possibilities but only few examples which are - in my opinion - unfortunately not resolved around use-cases.
This can be achieved by defining an ErrorDecoder and taking manual control of the Hystrix Circuit Breaker. You can inspect the response codes from the exceptions and provide your own fallback. In addition, if you wish to retry the request, wrap and throw your exception in a RetryException.
To meet your Retry requirement, also register a Retryer bean with the appropriate configuration. Keep in mind that using a Retryer will tie up a thread for the duration. The default implementation of Retryer does use an exponential backoff policy as well.
Here is an example ErrorDecoder taken from the OpenFeign documentation:
public class StashErrorDecoder implements ErrorDecoder {
#Override
public Exception decode(String methodKey, Response response) {
if (response.status() >= 400 && response.status() <= 499) {
return new StashClientException(
response.status(),
response.reason()
);
}
if (response.status() >= 500 && response.status() <= 599) {
return new StashServerException(
response.status(),
response.reason()
);
}
return errorStatus(methodKey, response);
}
}
In your case, you would react to 419 as desired.
You can forcibly open the Circuit Breaker setting this property at runtime
hystrix.command.HystrixCommandKey.circuitBreaker.forceOpen
ConfigurationManager.getConfigInstance()
.setProperty(
"hystrix.command.HystrixCommandKey.circuitBreaker.forceOpen", true);
Replace HystrixCommandKey with your own command. You will need to restore this circuit breaker back to closed after the desired time.
I could solve it with the following adjustments:
Properties in application.yml:
hystrix.command.commandKey:
execution.isolation.thread.timeoutInMilliseconds: 10_000
metrics.rollingStats.timeInMilliseconds: 10_000
circuitBreaker:
errorThresholdPercentage: 1
requestVolumeThreshold: 1
sleepWindowInMilliseconds: 3_600_000
Code in the respective Java class:
#HystrixCommand(fallbackMethod = "fallbackMethod", commandKey = COMMAND_KEY)
public void doCall(String parameter) {
try {
feignClient.doCall(parameter);
} catch (FeignException e) {
if (e.status() == 429) {
throw new TooManyRequestsException(e.getMessage());
}
}
}

additemeventlistener is not called for published pubsub message

I am trying to implement MUC over pubsub mechanism so for each MUC group, I send invites to multiple people. In the MUC's invitationReceivedListener, I get the pubsub node and addItemEventListener to it. Now, when I send message to group, internally, I publish item to pubsub node , its received at other end (i can tell that from below shown ejabberd log) but ItemEventListener is not called.
why my itemEventListener is not getting called ?
The other thing I noted, below sample ejabberd log, keep on looping everytime receiver logs in. So it seems it keep on sending same message multiple time to receiver.
Code
//Subscribing to node
multiUserChatManager.addInvitationListener(new InvitationListener() {
#Override
public void invitationReceived(XMPPConnection connection_,
MultiUserChat muc, String invitor, String reason, String password,Message msg) {
try {
System.out.println("in invitation Recieved----------------");
//Subscribe to node
LeafNode node=(LeafNode)pubSubMgr.getNode(parseName(muc.getRoom()));
addMessageListenerToPubSubNode(node);
node.subscribe(username+"#abc.mydomain.com");
System.out.println("Subscribing to node "+node.getId());
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
});
//Declaration of event Listener
private void addMessageListenerToPubSubNode(LeafNode node) {
node.addItemEventListener(new ItemEventListener(){
#Override
public void handlePublishedItems(ItemPublishEvent itemEvents) {
List<PayloadItem> item = itemEvents.getItems();
System.out.println("^^^^^^^^^^^^^^Received Item thru pubsub Item="+item.toString());
for(PayloadItem i:item)
{
System.out.println("message in the item on pubsub="+i.toXML());
}
}
}
);
}
Ejabberd Logs
following message I can see in my ejabberd logs however, ItemEventListener is not called on recipient side(+999999 is my receiver and +888888 is my sender).
2015-11-03 20:51:18.795 [debug] <0.2230.0>#ejabberd_c2s:send_text:1892 Send XML on stream =
<<"<message from='pubsub.abc.mydomain.com' to='+999999#abc.mydomain.com' type='headline'>
<event xmlns='http://jabber.org/protocol/pubsub#event'><items type='headline' node='03aa73745fa14e30bd72f66c9861f528'>
<item id='5A689B43B9777'><x xmlns='jabber:x:oob'>
<file1url>1446564035727125000.jpeg</file1url>
<postuuid>5522fbbc-5c4d-46af-bc65-b0a4b5777cdc</postuuid>
</x></item></items>
</event>
<delay xmlns='urn:xmpp:delay' from='+888888#abc.mydomain.com/+888888' stamp='2015-11-03T15:20:35Z'>
</delay>
</message>">>
Actually I found the solution
The issue was , I was not adding listener every time application open ups. I need to addeventlistener every time application is opened. This solved issue of not receiving messages

How to count all HTTP requests sent, retries in?

Some use cases require being able to count the requests sent by the Apache API. For example, when massively requesting a web API, which API requires an authentication through an API key, and which TOS limits the requests count in time for each key.
Being more specific on the case, I'm requesting https://domain1/fooNeedNoKey, and depending on its response analyzed data, I request https://domain2/fooNeedKeyWithRequestsCountRestrictions. All sends of those 1-to-2-requests sequences, are performed through a single org.apache.http.impl.client.FutureRequestExecutionService.
As of now, depending on org.apache.httpcomponents:httpclient:4.3.3, I'm using those API elements:
org.apache.http.impl.client.FutureRequestExecutionService, to perform multi-threaded HTTP requests. It offers time metrics (how much time did an HTTP thread took until terminated), but no requests counter metrics
final CloseableHttpClient httpClient = HttpClients.custom()
// the auto-retry feature of the Apache API will retry up to 5
// times on failure, being also allowed to send again requests
// that were already sent if necessary (I don't really understand
// the purpose of the second parameter below)
.setRetryHandler(new StandardHttpRequestRetryHandler(5, true))
// for HTTP 503 'Service unavailable' errors, also retrying up to
// 5 times, waiting 500ms between each retry. Guessed is that those
// 5 retries are part of the previous "global" 5 retries setting.
// The below setting, when used alone, would allow to only enable
// retries for HTTP 503, or to get a greater count of retries for
// this specific error
.setServiceUnavailableRetryStrategy(new DefaultServiceUnavailableRetryStrategy(5, 500))
.build();, which customizes the Apache API retry behavior
Getting back to the topic :
A request counter could be created by extending the Apache API retry-related classes quoted before
Alternatively, an Apache API support unrelated ticket tends to indicate this requests-counter metrics could be available and forwarded out of the API, into Java NIO
Edit 1:
Looks like the Apache API won't permit this to be done.
Quote from the inside of the API, RetryExec not beeing extendable in the API code I/Os:
package org.apache.http.impl.execchain;
public class RetryExec implements ClientExecChain {
..
public CloseableHttpResponse execute(
final HttpRoute route,
final HttpRequestWrapper request,
final HttpClientContext context,
final HttpExecutionAware execAware) throws IOException, HttpException {
..
for (int execCount = 1;; execCount++) {
try {
return this.requestExecutor.execute(route, request, context, execAware);
} catch (final IOException ex) {
..
if (retryHandler.retryRequest(ex, execCount, context)) {
..
}
..
}
}
The 'execCount' variable is the needed info, and it can't be accessed since it's only locally used.
As well, one can extend 'retryHandler', and manually count requests in it, but 'retryHandler.retryRequest(ex, execCount, context)' is not provided with the 'request' variable, making it impossible to know on what we're incrementing a counter (one may only want to increment the counter for requests sent to a specific domain).
I'm out of Java ideas for it. A 3rd party alternative: having the Java process polling a file on disk, managed by a shell script counting the desired requests. Sure it will make a lot of disk read-accesses and will be a hardware killer option.
Ok, the work around was easy, the HttpContext class of the API is intended for this:
// optionnally, in case your HttpCLient is configured for retry
class URIAwareHttpRequestRetryHandler extends StandardHttpRequestRetryHandler {
public URIAwareHttpRequestRetryHandler(final int retryCount, final boolean requestSentRetryEnabled)
{
super(retryCount, requestSentRetryEnabled);
}
#Override
public boolean retryRequest(final IOException exception, final int executionCount, final HttpContext context)
{
final boolean ret = super.retryRequest(exception, executionCount, context);
if (ret) {
doForEachRequestSentOnURI((String) context.getAttribute("requestURI"));
}
return ret;
}
}
// optionnally, in addition to the previous one, in case your HttpClient has specific settings for the 'Service unavailable' errors retries
class URIAwareServiceUnavailableRetryStrategy extends DefaultServiceUnavailableRetryStrategy {
public URIAwareServiceUnavailableRetryStrategy(final int maxRetries, final int retryInterval)
{
super(maxRetries, retryInterval);
}
#Override
public boolean retryRequest(final HttpResponse response, final int executionCount, final HttpContext context)
{
final boolean ret = super.retryRequest(response, executionCount, context);
if (ret) {
doForEachRequestSentOnURI((String) context.getAttribute("requestURI"));
}
return ret;
}
}
// main HTTP querying code: retain the URI in the HttpContext to make it available in the custom retry-handlers code
httpContext.setAttribute("requestURI", httpGET.getURI().toString());
try {
httpContext.setAttribute("requestURI", httpGET.getURI().toString());
httpClient.execute(httpGET, getHTTPResponseHandlerLazy(), httpContext);
// if request got successful with no need of retries, of if it succeeded on the last send: in any cases, this is the last query sent to server and it got successful
doForEachRequestSentOnURI(httpGET.getURI().toString());
} catch (final ClientProtocolException e) {
// if request definitively failed after retries: it's the last query sent to server, and it failed
doForEachRequestSentOnURI(httpGET.getURI().toString());
} catch (final IOException e) {
// if request definitively failed after retries: it's the last query sent to server, and it failed
doForEachRequestSentOnURI(httpGET.getURI().toString());
} finally {
// restoring the context as it was initially
httpContext.removeAttribute("requestURI");
}
Solved.