File component's [consumer.]bridgeErrorHandler in conjunction with startingDirectoryMustExist - exception

I have a route (with Camel 2.23.1) like:
from("file://not.existing.dir?autoCreate=false&startingDirectoryMustExist=true&consumer.bridgeErrorHandler=true")
.onException(Exception.class)
.handled(true)
.log(LoggingLevel.ERROR, "...exception text...")
.end()
.log(LoggingLevel.INFO, "...process text...")
...
(I tried it with just &bridgeErrorHandler, too, since according to the latest doc the consumer. prefix seems to be not necessary any longer.)
According to the doc of startingDirectoryMustExist:
| startingDirectoryMustExist | [...] Will thrown an exception if the directory doesn’t exist. |
the following exception is thrown:
org.apache.camel.FailedToCreateRouteException: Failed to create route route1:
Route(route1)[[From[file://not.existing.dir?autoCreate=false...
because of Starting directory does not exist: not.existing.dir
...
but, despite of the doc and the description of [consumer.]bridgeErrorHandler it's propagated to the caller, i.e neither "exception text" nor "process text" are printed.
There is a unit test FileConsumerBridgeRouteExceptionHandlerTest that covers consumer.bridgeErrorHandler, so I think this works basically. Can it be that [consumer.]bridgeErrorHandler doesn't work in conjunction with the exception thrown by startingDirectoryMustExist?
Do I have to write my own [consumer.]exceptionHandler as mentioned in this answer to "Camel - Stop route when the consuming directory not exists"?
There also a post on the mailing list from 2014 that reports similar behaviour with startingDirectoryMustExist and consumer.bridgeErrorHandler.
UPDATE
After TRACEing and debugging through the code I found that the exception is propagated as follows:
FileEndpoint.createConsumer()
throw new FileNotFoundException(...);
--> RouteService.warmUp()
throw new FailedToCreateRouteException(...)
--> DefaultCamelContext.doStart()
(re)throw e
--> ServiceSupport.start()
(re)throw e
I couldn't find any point where bridgeErrorHandler comes into play.
Setting breakpoints on BridgeExceptionHandlerToErrorHandler's constructor and all of its handleException() methods doesn't stop at any of them.
Am I still missing something?

You should use the directoryMustExist option instead, then you can have the error during polling, which is where the bridge error handler can be triggered. The startingDirectoryMustExist option is checked during creating the consumer and therefore before the polling and where the bridge error handler operates.
See also the JIRA ticket: https://issues.apache.org/jira/browse/CAMEL-13174

Related

catching classes that do not inherit from BaseException is not allowed

I'm making a custom plugin to query a database for user info to aide customer support. My backend is slack.
Everytime I start the bot command I'm greeted with:
Computer says nooo. See logs for details:
catching classes that do not inherit from BaseException is not allowed
I'm not sure if this is warning me that I'm attempting to catch an exception that isn't a BaseClass in my code or if an unknown exception was raised and caught elsewhere outside of my plugin.
To debug I tried:
try:
do_the_thing()
except (TypeError, ValueError) as e:
return('Something went wrong.')
I also tried:
try:
do_the_thing()
except Exception as e:
return('Something went wrong.')
And I still get the errbot admonition. Note that the command still runs and does the right thing where there is no exception raised by do_the_thing().
It means that:
Somewhere in your code you have an except ... statement where the exception ... (or one of the exceptions in the sequence ...) is not a subclass of BaseException, and
An exception is being thrown that is caught by that except ... statement.
The TypeError can be raised only when an exception is actually thrown because the names you give to except ... must be evaluated for their current values at that time; just because TypeError referenced a particular class at one point in the program's execution doesn't mean it won't be changed later to reference another object (though that would be admittedly perverse).
The Python interpreter should be giving you a full traceback of the exception; the first thing you need to do is find this. It could be occurring in one of two situations. (This is for single-threaded programs; I'm assuming your program is not multithreaded.)
During the execution of your program, in which case the program will be terminated by the exception, or
During finalization of objects (in their __del__(self) functions) in which case the error will be printed to stderr.
In both cases there should be a stack trace, not just the error message; I've confirmed that at least on Python ≥3.4 a stack trace is printed out for case 2.
You then need to follow this stack trace to see where the problem lies. Remember that the names you give to except ... are variables (even things like TypeError) that can be reassigned, so that you could conceivably be dealing with a (perverse) situation like:
TypeError = False
try:
...
except TypeError:
...
But more likely it will be something obvious such as:
class MyException: # Doesn't inherit from Exception
...
try:
...
except MyException:
...
There is one special case you need to be aware of: if you are seeing messages to stderr (case "2. During finalization," above) printed out as your program exits that means that the exception was thrown during cleanup as the interpreter shuts down, where random variables throughout the program may have already been set to None as part of the cleanup process. But in this case your program should still exit successfully.

kafka-python 1.3.3: KafkaProducer.send with explicit key fails to send message to broker

(Possibly a duplicate of Can't send a keyedMessage to brokers with partitioner.class=kafka.producer.DefaultPartitioner, although the OP of that question didn't mention kafka-python. And anyway, it never got an answer.)
I have a Python program that has been successfully (for many months) sending messages to the Kafka broker, using essentially the following logic:
producer = kafka.KafkaProducer(bootstrap_servers=[some_addr],
retries=3)
...
msg = json.dumps(some_message)
res = producer.send(some_topic, value=msg)
Recently, I tried to upgrade it to send messages to different partitions based on a definite key value extracted from the message:
producer = kafka.KafkaProducer(bootstrap_servers=[some_addr],
key_serializer=str.encode,
retries=3)
...
try:
key = some_message[0]
except:
key = None
msg = json.dumps(some_message)
res = producer.send(some_topic, value=msg, key=key)
However, with this code, no messages ever make it out of the program to the broker. I've verified that the key value extracted from some_message is always a valid string. Presumably I don't need to define my own partitioner, since, according to the documentation:
The default partitioner implementation hashes each non-None key using the same murmur2 algorithm as the java client so that messages with the same key are assigned to the same partition.
Furthermore, with the new code, when I try to determine what happened to my send by calling res.get (to obtain a kafka.FutureRecordMetadata), that call throws a TypeError exception with the message descriptor 'encode' requires a 'str' object but received a 'unicode'.
(As a side question, I'm not exactly sure what I'd do with the FutureRecordMetadata if I were actually able to get it. Based on the kafka-python source code, I assume I'd want to call either its succeeded or its failed method, but the documentation is silent on the point. The documentation does say that the return value of send "resolves to" RecordMetadata, but I haven't been able to figure out, from either the documentation or the code, what "resolves to" means in this context.)
Anyway: I can't be the only person using kafka-python 1.3.3 who's ever tried to send messages with a partitioning key, and I have not seen anything on teh Intertubes describing a similar problem (except for the SO question I referenced at the top of this post).
I'm certainly willing to believe that I'm doing something wrong, but I have no idea what that might be. Is there some additional parameter I need to supply to the KafkaProducer constructor?
The fundamental problem turned out to be that my key value was a unicode, even though I was quite convinced that it was a str. Hence the selection of str.encode for my key_serializer was inappropriate, and was what led to the exception from res.get. Omitting the key_serializer and calling key.encode('utf-8') was enough to get my messages published, and partitioned as expected.
A large contributor to the obscurity of this problem (for me) was that the kafka-python 1.3.3 documentation does not go into any detail on what a FutureRecordMetadata really is, nor what one should expect in the way of exceptions its get method can raise. The sole usage example in the documentation:
# Asynchronous by default
future = producer.send('my-topic', b'raw_bytes')
# Block for 'synchronous' sends
try:
record_metadata = future.get(timeout=10)
except KafkaError:
# Decide what to do if produce request failed...
log.exception()
pass
suggests that the only kind of exception it will raise is KafkaError, which is not true. In fact, get can and will (re-)raise any exception that the asynchronous publishing mechanism encountered in trying to get the message out the door.
I also faced the same error. Once I added json.dumps while sending the key, it worked.
producer.send(topic="first_topic", key=json.dumps(key)
.encode('utf-8'), value=json.dumps(msg)
.encode('utf-8'))
.add_callback(on_send_success).add_errback(on_send_error)

How to use JSCallback in Firebreath?

I'd like to add an event handler in my C++ code.
I followed document in firebreath.org (Callback from Scripts):
FB::JSObjectPtr doc = m_host->getDOMDocument()->getJSObject();
doc->Invoke("addEventListener", FB::variant_list_of("load", FB::make_callback(this, &mine::foo)));
but seeing following error:
/home/dq/manager/mine.cpp: In member function ‘void mine::init()’:
/home/dq/manager/mine.cpp:284:119: error: no matching function for call to ‘variant_list_of(const char [5], FB::JSAPIPtr)’
/home/dq/manager/mine.cpp:284:119: note: candidates are:
/usr/include/firebreath/ScriptingCore/variant_list.h:122:5: note: FB::detail::VariantListInserter FB::variant_list_of(FB::variant)
/usr/include/firebreath/ScriptingCore/variant_list.h:122:5: note: candidate expects 1 argument, 2 provided
/usr/include/firebreath/ScriptingCore/variant_list.h:128:5: note: FB::VariantList FB::variant_list_of()
/usr/include/firebreath/ScriptingCore/variant_list.h:128:5: note: candidate expects 0 arguments, 2 provided
In file included from /home/deqing/manager/mine.h:51:0,
from /home/deqing/manager/mine.cpp:37:
/usr/include/firebreath/ScriptingCore/JSCallback.h: In function ‘FB::JSAPIPtr FB::make_callback(const T&, F, bool) [with T = mine*, F = void (mine::*)(), FB::JSAPIPtr = boost::shared_ptr<FB::JSAPI>]’:
/home/dq/manager/mine.cpp:284:118: instantiated from here
/usr/include/firebreath/ScriptingCore/JSCallback.h:47:107: error: request for member ‘get’ in ‘instance’, which is of non-class type ‘mine* const’
/usr/include/firebreath/ScriptingCore/JSCallback.h:49:97: error: request for member ‘get’ in ‘instance’, which is of non-class type ‘mine* const’
make[2]: *** [CMakeFiles/mine.dir/manager/mine.cpp.o] Error 1
make[1]: *** [CMakeFiles/mine.dir/all] Error 2
make: *** [all] Error 2
Looking into the implementation of make_callback(), I tried following instead:
FB::JSObjectPtr doc = m_host->getDOMDocument()->getJSObject();
doc->Invoke("addEventListener", FB::variant_list_of("load")(FB::JSAPIPtr(new FB::JSCallback(FB::make_method(this, &mine::foo)))));
Compile passed this time, but my function - mine::foo() is not called with document.load()
By using "Inspect Element" in chrome, in "Event Listeners" I can see a new listener is added for "load". However, the listenerBody is a <JSAPI-Auto Javascript Object>.
I'm afraid this is why mine::foo() is not called, Javascript don't know how to call it because it is not a function, only an object.
Any one know how to get this done?
Another way that I can think of is:
Register a custom event handler
Fire the custom event on init
I'd like to use something like:
registerEventMethod("myevent", &mine::foo);
so that when myevent is fired, mine::foo() can be called.
The question here is, mine::foo is not a JSObjectPtr, so this snippet of code wouldn't work.
What is the proper way to use registerEventMethod() in this case?
registerEventObject was really never intended to be called manually; it is used by the internal implementations of addEventListener and attachEvent. The purpose is to attach javascript handlers to events in the plugin.
I would recommend you use something like boost's signals and slots to implement your own c++-side event system; FireBreath's stuff was never intended to solve that problem. On the other hand, if you wanted to look at it it would probably be possible to extend FireBreath's functionality to support that and I have it on very good authority that the gatekeeper for FireBreath would probably entertain a pull request to that end. =]

Getting the line number or full stack trace on exceptions in a gremlin-groovy script

I have a gremlin-groovy script that traverses a database which is incredibly noisy. There are lots of cases with missing edges or properties. When I assume an edge or property exists and it doesn't an exception is thrown I get a very simple output like this:
javax.script.ScriptException: java.util.IllegalFormatConversionException: d != java.lang.String
I'd like to make it so when the script encounters a fatal exception, as the one above, it provides a stack dump or at least a line number so I can debug where it happened, similar to how java can print a full stack trace on fatal exceptions.
Any suggestions on how to get a better dump?
I recommend using the Gremlin terminal for this.
gremlin$ ./gremlin.sh
\,,,/
(o o)
-----oOOo-(_)-oOOo-----
gremlin> a bad thing
No such property: bad for class: groovysh_evaluate
Display stack trace? [yN] y
groovy.lang.MissingPropertyException: No such property: bad for class: groovysh_evaluate
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:50)
If you don't have this luxury (e.g. via a REST server), then you may want to place some intelligent 'printlns' in your traversal.
my.long.traversal.sideEffect{println 'I am here'}.is.very.long

BeepBeep and ErlyDB integration issue

Further to my adventures with Erlang and ErlyDB. I am attempting to get ErlyDB working with BeepBeep
My ErlyDB setup works correctly when run outside of the BeepBeep environment (see Debugging ErlyDB and MySQL). I have basically take the working code and attempted to get it running inside BeepBeep.
I have the following code in my controller:
handle_request("index",[]) ->
erlydb:start(mysql,Database),
erlydb:code_gen(["thing.erl"],mysql),
NewThing = thing:new_with([{name, "name"},{value, "value"}]),
thing:save(NewThing),
{render,"home/index.html",[{data,"Hello World!"}]};
When I call the URL, the response outputs "Server Error".
There is no other error or exception information reported.
I have tried wrapping the call in try/catch to see if there is an underlying error - there is definitely an exception at the call to thing:new_with(), but no further information is available.
The stacktrace reports:
{thing,new,[["name","value"]]}
{home_controller,create,1}
{home_controller,handle_request,3}
{beepbeep,process_request,4}
{test_web,loop,1}
{mochiweb_http,headers,4}
{proc_lib,init_p_do_apply,3}
Use pattern matching to assert that things work up to the call to thing:new/1:
ok = erlydb:start(mysql,Database),
ok = erlydb:code_gen(["thing.erl"],mysql),
You include only the stack trace, look at the exception message as well. I suspect that the error is that you get an 'undef' exception. But check that it is so. The first line in the stack trace indicates that it is a problem with calling thing:new/1 with ["name", "value"] as argument.
It is slightly odd that you show one clause of handle_request that is not calling home_controller:create/1 as per {home_controller,create,1} in the stack-trace. What do the other clauses in your handle_request/2 function look like?