I want to configure 3 partitions and 3 replications of a topic in distributed environment with three nodes. How can I configure these by java api without shell command?
If I have three nodes: node1, node2 and node3. I want partition1 and replication3 are deployed in node1, partition2 and replication1 are deployed in node2, partition3 and replication2 are deployed in node3.
I've tried spring-kafka's api in single-machine environment, this can create a topic and 1 partition automatically. But it not work in distributed environment.
My maven configuration is:
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
<version>1.1.7.RELEASE</version>
</dependency>
1.1.x is no longer supported; you should be using at least 1.3.9.
1.3.x comes with KafkaAdmin, which can automatically configure any NewTopic beans in the application context.
See Configuring Topics.
If you define a KafkaAdmin bean in your application context, it can automatically add topics to the broker. Simply add a NewTopic #Bean for each topic to the application context.
#Bean
public KafkaAdmin admin() {
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
StringUtils.arrayToCommaDelimitedString(kafkaEmbedded().getBrokerAddresses()));
return new KafkaAdmin(configs);
}
#Bean
public NewTopic topic1() {
return new NewTopic("foo", 10, (short) 2);
}
#Bean
public NewTopic topic2() {
return new NewTopic("bar", 10, (short) 2);
}
I have tomcat memory leak issue when stop/redeploy application. It says The following web applications were stopped (reloaded, undeployed), but their
classes from previous runs are still loaded in memory, thus causing a memory
leak (use a profiler to confirm):/test-1.0-SNAPSHOT
MySQL connector driver located in Tomcat/lib folder.
I can reproduce this issue in both: Tomcat 7/8. Also tried MS SQL database with "net.sourceforge.jtds.*" driver but didn't help.
Please find below project files. Project only creates 1 table in DB.
build.gradle
group 'com.test'
version '1.0-SNAPSHOT'
apply plugin: 'java'
apply plugin: 'war'
sourceCompatibility = 1.8
repositories {
mavenCentral()
}
dependencies {
compile group: 'org.hibernate', name: 'hibernate-entitymanager', version: '5.2.10.Final'
compile group: 'org.springframework.data', name: 'spring-data-jpa', version: '1.11.4.RELEASE'
compile group: 'org.springframework', name: 'spring-webmvc', version: '4.3.9.RELEASE'
providedCompile 'javax.servlet:javax.servlet-api:3.1.0'
providedCompile group: 'mysql', name: 'mysql-connector-java', version: '5.1.6'
compile group: 'commons-dbcp', name: 'commons-dbcp', version: '1.4'
}
ApplicationConfig.java
#Configuration
#Import({JPAConfiguration.class})
#EnableWebMvc
public class ApplicationConfig {}
JPAConfiguration.java
#Configuration
#EnableJpaRepositories("com.test.dao")
#EnableTransactionManagement
public class JPAConfiguration {
#Bean
public EntityManagerFactory entityManagerFactory() {
LocalContainerEntityManagerFactoryBean factory = new LocalContainerEntityManagerFactoryBean();
factory.setJpaVendorAdapter(new HibernateJpaVendorAdapter());
factory.setPackagesToScan("com.test.model");
factory.setDataSource(restDataSource());
factory.setJpaPropertyMap(getPropertyMap());
factory.afterPropertiesSet();
return factory.getObject();
}
#Bean(destroyMethod = "close")
public DataSource restDataSource() {
BasicDataSource dataSource = new BasicDataSource();
dataSource.setDriverClassName("com.mysql.jdbc.Driver");
dataSource.setUrl("jdbc:mysql://localhost:3306/test");
dataSource.setUsername("test");
dataSource.setPassword("test");
return dataSource;
}
private Map<String, String> getPropertyMap() {
Map<String, String> hibernateProperties = new HashMap<>();
hibernateProperties.put("hibernate.hbm2ddl.auto", "update");
hibernateProperties.put("hibernate.show_sql", "true");
hibernateProperties.put("hibernate.format_sql", "true");
hibernateProperties.put("hibernate.dialect", "org.hibernate.dialect.MySQL5InnoDBDialect");
return hibernateProperties;
}
#Bean
public PlatformTransactionManager transactionManager() {
JpaTransactionManager txManager = new JpaTransactionManager();
txManager.setEntityManagerFactory(entityManagerFactory());
return txManager;
}
}
TestRepository.java
#Repository
public interface TestRepository extends JpaRepository<TestEntity, Long> {}
TestEntity.java
#Entity
#Table(name = "ent")
public class TestEntity {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
private String descript;
//equals, hashcode, toString, getters, setters
}
AppInitializer.java
public class AppInitializer extends AbstractAnnotationConfigDispatcherServletInitializer {
private WebApplicationContext rootContext;
#Override
protected Class<?>[] getRootConfigClasses() {
return new Class[]{ApplicationConfig.class};
}
#Override
protected Class<?>[] getServletConfigClasses() {
return null;
}
#Override
protected String[] getServletMappings() {
return new String[]{"/"};
}
}
Command
jmap -histo <tomcat_pid>
shows only 2 items from project structure after tomcat stop:
com.test.config.dao.JPAConfiguration$$EnhancerBySpringCGLIB$$792cb231$$FastClassBySpringCGLIB$$45ff499c
com.test.config.dao.JPAConfiguration$$FastClassBySpringCGLIB$$10104c1e
Anyone have ideas or suggestions to fix this problem?
There are 2 memory leaks in this small project:
The problem with MySQL jdbc driver.
We have to add ContextLoaderListener to deregister jdbc driver:
Listener:
#WebListener
public class ContextListener extends ContextLoaderListener {
private final Logger log = LoggerFactory.getLogger(this.getClass());
#Override
public void contextInitialized(ServletContextEvent sce) {
log.info("-= Context started =-");
}
#Override
public void contextDestroyed(ServletContextEvent sce) {
super.contextDestroyed(sce);
log.info("-= Context destroyed =-");
try {
log.info("Calling MySQL AbandonedConnectionCleanupThread checkedShutdown");
com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.uncheckedShutdown();
} catch (Exception e) {
log.error("Error calling MySQL AbandonedConnectionCleanupThread checkedShutdown {}", e);
}
ClassLoader cl = Thread.currentThread().getContextClassLoader();
Enumeration<Driver> drivers = DriverManager.getDrivers();
while (drivers.hasMoreElements()) {
Driver driver = drivers.nextElement();
if (driver.getClass().getClassLoader() == cl) {
try {
log.info("Deregistering JDBC driver {}", driver);
DriverManager.deregisterDriver(driver);
} catch (SQLException ex) {
log.error("Error deregistering JDBC driver {}", driver, ex);
}
} else {
log.info("Not deregistering JDBC driver {} as it does not belong to this webapp's ClassLoader", driver);
}
}
}
}
or if you had an access to tomcat server you can modify listener in tomcat/conf/server.xml example.
The second problem is known memory leak in jboss-logging library (link).
The memory leak has gone after we exclude this library from hibernate dependency:
build.gradle:
group 'com.test'
version '1.0-SNAPSHOT'
apply plugin: 'java'
apply plugin: 'war'
sourceCompatibility = 1.8
repositories {
mavenCentral()
}
dependencies {
compile(group: 'org.hibernate', name: 'hibernate-entitymanager', version: '5.2.10.Final') {
exclude group: 'org.jboss.logging', module: 'jboss-logging'
}
compile group: 'org.springframework.data', name: 'spring-data-jpa', version: '1.11.4.RELEASE'
compile group: 'org.springframework', name: 'spring-webmvc', version: '4.3.9.RELEASE'
providedCompile 'javax.servlet:javax.servlet-api:3.1.0'
providedCompile group: 'mysql', name: 'mysql-connector-java', version: '8.0.11'
compile group: 'org.slf4j', name: 'slf4j-api', version: '1.7.25'
compile group: 'org.slf4j', name: 'slf4j-simple', version: '1.7.25'
}
then build jar from repo and added to tomcat /lib folder.
The issue with jboss-logging probably fixed in Java 9 (pull request link).
Short answer - hopefully the same problem for you...
Those two com.test.config.dao.JPAConfiguration$$...CGLIB$$... classes were being referenced indirectly by the Abandoned connection cleanup thread in MySQL:
20-Jun-2018 21:25:22.987 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [test-1.0-SNAPSHOT] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
The following answer enabled me to resolve the problem. E.g. in tomcat/conf/server.xml, look for the JreMemoryLeakPreventionListener line and replace it with this:
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
classesToInitialize="com.mysql.jdbc.Driver" />
This forces the MySQL JDBC driver, and its cleanup thread, to be loaded outside the classloader for the web application. This means the cleanup thread won't hold a reference to the webapp classloader as its context class loader.
Expanded answer - how to trace the leak in your environment...
Hopefully the above is all you need - it was enough to reproduce and solve the problem against https://github.com/egotovko/tomcat-leak
However there are many other causes of a leaked reference to a web application that can stop it undeploying. E.g. other threads still running (Tomcat is good at warning about these) or references from outside the web application.
To properly trace the cause, you can chase the reference in a heap dump. If this is not familiar, you can get a heap dump from jmap -dump:file=dump.hprof <pid>, or by directly connecting from such as jvisualvm (also included in the JDK).
With the heap dump open in jvisualvm:
Select the Classes button for the heap dump
Sort the list of classes by name
Look for classes in the web application - e.g. com.test.config.dao.JPAConfiguration$$EnhancerBySpringCGLIB$$ in this example
This should be showing with an instance count of 2 or so
Double click to show these in the Instances View
In the References pane for one of these instances, right click and Show Nearest GC Root
E.g. for that Abandoned connection cleanup thread in MySQL:
Note how the AbandonedConnectionCleanupThread has a contextClassLoader, which is the ParallelWebappClassLoader for the web application. Tomcat needs to be able to release the class loader to undeploy the web application.
Once you've tracked down what's holding the reference, it's then normally a case of investigating how better to configure that library in Tomcat, or perhaps someone else has seen that memory leak. It's also not uncommon to have to repeat the exercise, when there are several references to clear up.
I have REST web service which I test with Jersey Test, Mockito, Junit. When web service method is executed successfully, I get correct response. In case of invalid data, custom exception is thrown which must be handled by ExceptionMapper. It should return the same structure response but with different codes. ExceptionMapper works well in not test environment. However, after test execution logs show:
1 < 500
1 < Connection: close
1 < Content-Length: 1033
1 < Content-Type: text/html;charset=ISO-8859-1
1 < Date: Wed, 30 Mar 2016 11:55:37 GMT
javax.ws.rs.InternalServerErrorException: HTTP 500 Request failed.
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1020)
at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:816)
at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:92)
I use:
<dependency>
<groupId>org.glassfish.jersey.test-framework.providers</groupId>
<artifactId>jersey-test-framework-provider-grizzly2</artifactId>
<version>2.22.2</version>
</dependency>
Shortened version of mocked service which is extented by test classes:
public class MockedService extends JerseyTest {
#Override
public ResourceConfig configure() {
forceSet(TestProperties.CONTAINER_PORT, "8080");
enable(TestProperties.LOG_TRAFFIC);
return new ResourceConfig().register(new Service());
}
}
How to get response from ExceptionMapper?
You still need to register the ExceptionMapper
return new ResourceConfig()
.register(new Service())
.register(new YourMapper());
In your real environment, the mapper is probably scanned for, either with package scanning or classpath scanning.
I am using Azure service-bus queues (AMQP Protocol) with Apache Qpid (0.3) as Java client.
I am also using Spring JmsTemplate to produce messages and DefaultMessageListenerContainer to manage my consumers, spring JMS 4.0.6.
Spring configurations:
#PostConstruct
private void JndiLookup() throws NamingException {
// Configure JNDI environment
Hashtable<String, String> envPrp = new Hashtable<String, String>();
envPrp.put(Context.INITIAL_CONTEXT_FACTORY,
PropertiesFileInitialContextFactory.class.getName());
envPrp.put("connectionfactory.SBCF", "amqps://owner:{parimeryKey}#{namespace}.servicebus.windows.net");
envPrp.put("queue.STORAGE_NEW_QUEUE", "QueueName");
context = new InitialContext(envPrp);
}
#Bean
public ConnectionFactory connectionFactory() throws NamingException {
ConnectionFactory cf = (ConnectionFactory) context.lookup("SBCF");
return cf;
}
#Bean
public DefaultMessageListenerContainer messageListenerContainer() throws NamingException {
DefaultMessageListenerContainer messageListenerContainer = new DefaultMessageListenerContainer();
messageListenerContainer.setConnectionFactory(connectionFactory());
Destination queue = (Destination) context.lookup("QueueName");
messageListenerContainer.setDestination(queue);
messageListenerContainer.setConcurrency("3-10");
MessageListenerAdapter adapter = new MessageListenerAdapter();
adapter.setDelegate(new MessageWorker());
adapter.setDefaultListenerMethod("onMessage");
messageListenerContainer.setMessageListener(adapter);
return messageListenerContainer;
}
#Bean
public JmsTemplate jmsTemplate() throws NamingException {
JmsTemplate jmsTemplate = new JmsTemplate();
jmsTemplate.setConnectionFactory(connectionFactory());
return jmsTemplate;
}
Nothing fancy in the configurations just straight forward.
Running the code and everything seems to be working .. but after few minutes without traffic in the queue it seems like the consumers are losing connection with the queue and not taking messages.
I dont know if it is related but every 5 minutes in am getting the following warning:
Fri Nov 07 15:23:53 +0000 2014, (DefaultMessageListenerContainer.java:842) WARN : Setup of JMS message listener invoker failed for destination 'org.apache.qpid.amqp_1_0.jms.impl.QueueImpl#8fb0427b' - trying to recover. Cause: Force detach the link because the session is remotely ended.
Fri Nov 07 15:23:56 +0000 2014, (DefaultMessageListenerContainer.java:891) INFO : Successfully refreshed JMS Connection
I have messages being in the queue for hours and not being handled by the consumers only when I restart the app the consumers renewing the connection properly and taking the messages.
Is it possible that the problem is with the Spring Listener container properties or qpid connection factory or is it an issue with Azure service bus??
Couldn't find related post to my situation will appreciate the help!!
I have a simple program using JPA entities to write into a Derby DB (the entities were generated from an existing DB tables). I am using Eclipse and there is a working connection between the Derby client and the server via the EclipseLink Data Source Explorer .
Here is the start of my main():
import javax.persistence.*;
import java.sql.Timestamp;
import java.util.*;
import javax.*;
public class start {
/**
* #param args
*/
private static final String PERSISTENCE_UNIT_NAME = "zodiac";
private static EntityManagerFactory factory;
public static void main(String[] args) {
try {
factory = Persistence.createEntityManagerFactory(PERSISTENCE_UNIT_NAME);
EntityManager em = factory.createEntityManager();
System.out.println("after factory gen" );
when I the line with createEntityManager() is executed the following exception is thrown:
[EL Info]: 2012-03-07 22:46:21.892--ServerSession(253038357)--EclipseLink, version: Eclipse Persistence Services - 2.3.2.v20111125-r10461
[EL Severe]: 2012-03-07 22:46:22.064--ServerSession(253038357)--Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: No suitable driver
Error Code: 0
Any idea what is the problem ? thanks
If you're in Eclipse you need to add the driver to your project classpath. Sounds like you already have a datasource so you must have defined the driver library. All you need to do is "Add Library" to your Java Build Path and choose "Connectivity Driver Definition" and then the Derby driver from the drop down list of available driver definitions.
FYI, there's a checkbox in the New JPA Project wizard where you can select "add driver to classpath" to do this when you create a new project.
Of course you can also add the derbyclient.jar to your classpath directly or define a user library that includes it.
--Shaun