I have a django based web application where I am using Kafka to process some orders. Now I use Celery Workers to assign a Kafka Consumer to each topics. Each Kafka Consumer is assigned to a Kafka topic in the form of a Kafka tasks. However after a day or so, when I am submitting a task I am getting the following error :
_mysql.connection.query(self, query)
_mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away')
The above exception was the direct cause of the following exception:
Below is how my tasks.py file looks like :
#shared_task
def init_kafka_consumer(topic):
try:
if topic is None:
raise Exception("Topic is none, unable to initialize kafka consumer")
logger.info("Spawning new task to subscribe to topic")
params = []
params.append(topic)
background_thread = Thread(target=sunscribe_consumer, args=params)
background_thread.start()
except Exception :
logger.exception("An exception occurred while reading message from kafka")
def sunscribe_consumer(topic) :
try:
if topic is None:
raise Exception("Topic is none, unable to initialize kafka consumer")
conf = {'bootstrap.servers': "localhost:9092", 'group.id': 'test', 'session.timeout.ms': 6000,
'auto.offset.reset': 'earliest'}
c = Consumer(conf)
logger.info("Subscribing consumer to topic "+str(topic[0]))
c.subscribe(topic)
# Read messages from Kafka
try:
while True:
msg = c.poll(timeout=1.0)
if msg is None:
continue
if msg.error():
raise KafkaException(msg.error())
else:
try:
objs = serializers.deserialize("json", msg.value())
for obj in objs:
order = obj.object
order = BuyOrder.objects.get(id=order.id) #Getting an error while accessing DB
if order.is_pushed_to_kafka :
return
order.is_pushed_to_kafka = True
order.save()
from web3 import HTTPProvider, Web3, exceptions
w3 = Web3(HTTPProvider(INFURA_MAIN_NET_ETH_URL))
processBuyerPayout(order,w3)
except Exception :
logger.exception("An exception occurred while de-serializing message")
except Exception :
logger.exception("An exception occurred while reading message from kafka")
finally:
c.close()
except Exception :
logger.exception("An exception occurred while reading message from kafka")
Is there anyway that I could check if database connection exists as soon as a task is received and if not, I can re-establish the connection?
According to https://github.com/celery/django-celery-results/issues/58#issuecomment-418413369
and comments above putting this code:
from django.db import close_old_connections
close_old_connections()
which is closing old connection and opening new one inside your task should helps.
I am currently trying to setup some data collections for our app using the full elk stack (Beats - Logstash - ElasticSearch-Kibana). So far everything is working as it should but I have a requirement to capture statistics on the exceptions thrown by the applications (e.g. java.lang.IllegalArgumentException)
I am not really interested in the stack trace itself so I went ahead and added a separate grok filter just for the exception.
Example of Message:
2016-11-15 05:19:28,801 ERROR [App-Initialisation-Thread] appengine.java:520 Failed to initialize external authenticator myapp Support Access || appuser#vm23-13:/mnt/data/install/assembly app-1.4.12#cad85b224cce11eb5defa126030f21fa867b0dad
java.lang.IllegalArgumentException: Could not check if provided root is a directory
at com.myapp.io.AbstractRootPrefixedFileSystem.checkAndGetRoot(AbstractRootPrefixedFileSystem.java:67)
at com.myapp.io.AbstractRootPrefixedFileSystem.<init>(AbstractRootPrefixedFileSystem.java:30)
at com.myapp.io.s3.S3FileSystem.<init>(S3FileSystem.java:32)
at com.myapp.io.s3.S3FileSystemDriver.loadFileSystem(S3FileSystemDriver.java:60)
at com.myapp.io.FileSystems.getFileSystem(FileSystems.java:55)
at com.myapp.authentication.ldap.S3LdapConfigProvider.initializeCloudFS(S3LdapConfigProvider.java:77)
at com.myapp.authentication.ldap.S3LdapConfigProvider.loadS3Config(S3LdapConfigProvider.java:51)
at com.myapp.authentication.ldap.S3LdapConfigProvider.getLdapConfig(S3LdapConfigProvider.java:42)
at com.myapp.authentication.ldap.DelegatingLdapConfigProvider.getLdapConfig(DelegatingLdapConfigProvider.java:45)
at com.myapp.authentication.ldap.LdapExternalAuthenticatorFactory.create(LdapExternalAuthenticatorFactory.java:28)
at com.myapp.authentication.ldap.LdapExternalAuthenticatorFactory.create(LdapExternalAuthenticatorFactory.java:10)
at com.myapp.frob.appengine.getExternalAuthenticators(appengine.java:516)
at com.myapp.frob.appengine.startUp(appengine.java:871)
at com.myapp.frob.appengine.startUp(appengine.java:754)
at com.myapp.jsp.KewServeInitContextListener$1.run(QServerInitContextListener.java:104)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.NoSuchFileException: fh-ldap-config/
at com.upplication.s3fs.util.S3Utils.getS3ObjectSummary(S3Utils.java:55)
at com.upplication.s3fs.util.S3Utils.getS3FileAttributes(S3Utils.java:64)
at com.upplication.s3fs.S3FileSystemProvider.readAttributes(S3FileSystemProvider.java:463)
at com.myapp.io.AbstractRootPrefixedFileSystem.checkAndGetRoot(AbstractRootPrefixedFileSystem.java:61)
Example of grok statement:
grok {
patterns_dir => ["./patterns"]
match => ["message", "%{GREEDYDATA}\n%{JAVAFILE:exception}"]
}
Testing on the grok debugger shows correct results:
{
"GREEDYDATA": [
[
"2016-11-15 05:19:28,801 ERROR [App-Initialisation-Thread] appengine.java:520 Failed to initialize external authenticator myapp Support Access || appuser#vm23-13:/mnt/data/install/assembly app-1.4.12#cad85b224cce11eb5defa126030f21fa867b0dad"
]
],
"exception": [
[
"java.lang.IllegalArgumentException"
]
]
}
Problem
When I add the configuration to logstash it captures the Caused string instead of the exception name, even though the "Caused" string is after another new line character. However it works perfectly for other exception messages such as :
016-11-15 06:17:49,691 WARN [SCReplicationWorkerThread-2] ClientJob.java:207 50345 Error communicating to server `199.181.131.249':`80'. Waiting `10' seconds before retrying... If you see this message rarely, the sc will have recovered gracefully. || appuser#vm55-12:/mnt/deployment/install/app app-3.1.23#cad85b224cce11eb5defa126030f21fa867b0dad
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.net.SocketInputStream.read(SocketInputStream.java:223)
at java.io.DataInputStream.readBoolean(DataInputStream.java:242)
at com.myapp.replication.client.ClientJob.passCheckRevision(ClientJob.java:279)
at com.myapp.replication.client.ClientJob.execute(ClientJob.java:167)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:534)
Any advice would be appreciated.
Thanks
Did you setting the mutiline in the input or filebeat input ,
like this to show the pattern start with ISO8601
I think maybe you mutiline not fetch the whole line
input {
beats {
port => 5044
codec => multiline {
pattern => "^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}[\.,][0-9]{3,7} "
negate => true
what => "previous"
}
}
}
I have a grails project and I want to deploy it on Cloud Foundry, but the the console shows this:
[CONTAINER] n.spring.CloudProfileApplicationContextInitializer INFO Adding 'cloud' to list of active profiles
[CONTAINER] g.CloudPropertySourceApplicationContextInitializer INFO Adding 'cloud' PropertySource to ApplicationContext
[CONTAINER] udAutoReconfigurationApplicationContextInitializer INFO Adding cloud service auto-reconfiguration to ApplicationContext
[CONTAINER] ing.DataSourceCloudServiceBeanFactoryPostProcessor INFO Auto-reconfiguring beans of type javax.sql.DataSource
[CONTAINER] ing.DataSourceCloudServiceBeanFactoryPostProcessor INFO No beans of type javax.sql.DataSource found. Skipping auto-reconfiguration.
[CONTAINER] lina.core.ContainerBase.[Catalina].[localhost].[/] SEVERE Exception sending context initialized event to listener instance of class org.codehaus.groovy.grails.web.context.GrailsContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pluginManager' defined in ServletContext resource [/WEB-INF/applicationContext.xml]: Invocation of init method failed; nested exception is java.lang.NullPointerException: Cannot invoke method getAt() on null object
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Caused by: java.lang.NullPointerException: Cannot invoke method getAt() on null object
at java.lang.Thread.run(Thread.java:745)
[CONTAINER] org.apache.catalina.core.StandardContext SEVERE Error listenerStart
... 5 more
[CONTAINER] org.apache.catalina.util.SessionIdGeneratorBase INFO Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [135] milliseconds.
[CONTAINER] org.apache.catalina.core.StandardContext SEVERE Context [] startup failed due to previous errors
[CONTAINER] org.apache.catalina.loader.WebappClassLoader WARNING The web application [] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
[CONTAINER] org.apache.catalina.loader.WebappClassLoader WARNING The web application [] registered the JDBC driver [org.h2.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
java.lang.Object.wait(Native Method)
[CONTAINER] org.apache.catalina.loader.WebappClassLoader WARNING The web application [] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
I think that it's a problem of DB Conex but I don't know to fix it. I use MySQL in my app and the service plan ClearDB MySQL Database Spark DB.
My Datasource.groovy is:
import org.springframework.cloud.CloudFactory
def cloud
try {
cloud = new CloudFactory().cloud
} catch(e) {}
dataSource {
pooled = true
driverClassName = "com.mysql.jdbc.Driver"
dialect = "org.hibernate.dialect.MySQL5InnoDBDialect"
username = "root"
password = "root"
}
hibernate {
cache.use_second_level_cache = true
cache.use_query_cache = true
cache.region.factory_class = 'net.sf.ehcache.hibernate.EhCacheRegionFactory'
//cache.region.factory_class = 'org.hibernate.cache.EhCacheProvider'
}
environments {
development {
dataSource {
dbCreate = "create-drop"
url = "jdbc:mysql://localhost:8080/bbddSRL"
username="root"
password="root"
}
}
test {
dataSource {
dbCreate = "create-drop"
url = "jdbc:mysql://localhost:8080/bbddSRL"
username="root"
password="root"
}
}
production {
dataSource {
pooled = true
dbCreate = 'update'
driverClassName = 'com.mysql.jdbc.Driver'
if (cloud) {
def dbInfo = cloud.getServiceInfo('mysql-instance') //mysql-instance is the name of the ClearDB's service.
url = dbInfo.jdbcUrl
username = dbInfo.userName
password = dbInfo.password
} else {
url = 'jdbc:mysql://localhost:8080/bbddSRLprod'
username = 'root'
password = 'root'
}
}
}
}
Any anwers or suggestions?
Try eliminating the datasource block where you define dialect. You will have to still define the items in that block but do it in the datasource of each environment. I did this because it seemed like any changes that I made to the datasource within the environment had no effect.
I use Mule 3.x
I have a code that tests MuleClient connectivity.
This test is ok and works proper way:
public void testHello() throws Exception
{
MuleClient client = new MuleClient(muleContext);
MuleMessage result = client.send("http://127.0.0.1:8080/hello", "some data", null);
assertNotNull(result);
assertNull(result.getExceptionPayload());
assertFalse(result.getPayload() instanceof NullPayload);
//TODO Assert the correct data has been received
assertEquals("hello", result.getPayloadAsString());
}
But this tes is not ok - it fail with an connection exceptions:
public void testHello_with_Spring() throws Exception {
MuleClient client = new MuleClient("mule-config-test.xml");
client.getMuleContext().start();
//it fails here
MuleMessage result = client.send("http://127.0.0.1:8080/hello", "some data", null);
assertNotNull(result);
assertNull(result.getExceptionPayload());
assertFalse(result.getPayload() instanceof NullPayload);
//TODO Assert the correct data has been received
assertEquals("hello", result.getPayloadAsString());
}
The 'mule-config-test.xml' is used in both tests, the path for this file is ok, i checked.
This is error message I have in the end:
Exception stack is:
1. Address already in use (java.net.BindException) java.net.PlainSocketImpl:-2 (null)
2. Failed to bind to uri "http://127.0.0.1:8080/hello" (org.mule.transport.ConnectException)
org.mule.transport.tcp.TcpMessageReceiver:81
(http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/transport/ConnectException.html)
-------------------------------------------------------------------------------- Root Exception stack trace: java.net.BindException: Address already in
use at java.net.PlainSocketImpl.socketBind(Native Method) at
java.net.PlainSocketImpl.bind(PlainSocketImpl.java:383) at
java.net.ServerSocket.bind(ServerSocket.java:328)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
[10-05 16:33:37] ERROR HttpConnector [main]:
org.mule.transport.ConnectException: Failed to bind to uri
"http://127.0.0.1:8080/hello" [10-05 16:33:37] ERROR ConnectNotifier
[main]: Failed to connect/reconnect: HttpConnector {
name=connector.http.mule.default lifecycle=stop this=7578a7d9
numberOfConcurrentTransactedReceivers=4
createMultipleTransactedReceivers=true connected=false
supportedProtocols=[http] serviceOverrides= } . Root Exception
was: Address already in use. Type: class java.net.BindException [10-05
16:33:37] ERROR DefaultSystemExceptionStrategy [main]: Failed to bind
to uri "http://127.0.0.1:8080/hello"
org.mule.api.lifecycle.LifecycleException: Cannot process event as
"connector.http.mule.default" is stopped
I think the problem is in what you're not showing: testHello_with_Spring() is probably executing while Mule is already running. The second Mule you're starting in it then port-conflicts with the other one.
Are testHello() and testHello_with_Spring() in the same test suite? If yes, seeing that testHello() relies on an already running Mule, I'd say that would be the cause of port conflict for testHello_with_Spring().
I made a comet chat server with Erlang and Mochiweb. And I run the "./start-dev.sh" to start the server. But after about 1 month I got the following error:
=ERROR REPORT==== 26-Sep-2009::09:21:06 ===
{mochiweb_socket_server,235,
{child_error,
{badmatch,
{error,
[70,97,105,108,101,100,32,115,101,110,100,105,110,103,32,100,
97,116,97,32,111,110,32,115,111,99,107,101,116,32,58,32,
"closed"]}}}}
mysql: fetch "SELECT appKey FROM applications WHERE appID = 1" (id p1)
=CRASH REPORT==== 26-Sep-2009::09:21:10 ===
crasher:
initial call: mochiweb_socket_server:acceptor_loop/1
pid: <0.4271.23>
registered_name: []
exception error: no match of right hand side value
{error,[70,97,105,108,101,100,32,115,101,110,100,105,110,
103,32,100,97,116,97,32,111,110,32,115,111,99,
107,101,116,32,58,32,"closed"]}
in function moonwalker_web:loop/2
in call from mochiweb_http:headers/5
ancestors: [moonwalker_web,moonwalker_sup,<0.52.0>]
messages: []
links: [<0.54.0>,#Port<0.792854>]
dictionary: [{mochiweb_request_body,
<<"appID=1&appKey=keyy&userID=8048943&nickName=bill&buddies=N%3B×tamp=1253928070154">>},
{mochiweb_request_recv,true},
{mochiweb_request_post,
[{"appID","1"},
{"appKey","key"},
{"userID","8048943"},
{"nickName",[143,229,167,144]},
{"buddies","N;"},
{"timestamp","1253928070154"}]},
{mochiweb_request_path,"/online"}]
trap_exit: false
status: running
heap_size: 2584
stack_size: 24
reductions: 1368
neighbours:
=ERROR REPORT==== 26-Sep-2009::09:21:10 ===
{mochiweb_socket_server,235,
{child_error,
{badmatch,
{error,
[70,97,105,108,101,100,32,115,101,110,100,105,110,103,32,100,
97,116,97,32,111,110,32,115,111,99,107,101,116,32,58,32,
"closed"]}}}}
And if turn the following numbers into characters
[70,97,105,108,101,100,32,115,101,110,100,105,110,103,32,100,
97,116,97,32,111,110,32,115,111,99,107,101,116,32,58,32,
"closed"]}}}}
they are
Failed sending data on socket :"closed"
Does that mean I have problems with MySQL connection or socket?
I don't know if this error has something to do with my "./start-dev.sh" or I just had some wrong settings?
And what else information do I have to provide for diagnosing?
Thanks and looking forward to your reply?
It looks like somewhere in the loop/2 function you don't handle an {error,Error} return from a function call. This causes the error which crashes the process. Without the code it is difficult to say what caused the error return.