Testing neo4j unmanaged extensions locally - junit

I develop some unmanaged extensions for neo4j 2.3.0. Now I want to test the functionality of my code with junit. Is there a way to test my methods locally without an neo4j instance running on my pc?
I want something like this:
- create a temporary instance of neo4j before executing test
- fill instance with data
- call my extension via rest
- check the results
I have created a Neo4JTestServer class (using neo4j-harness):
public final class Neo4jTestServer {
public static final String EXTENSION_MOUNT_POINT = "/v1";
public static final String EXTENSION_RESOURCES = "my.company.neo4j.extension";
private static Neo4jTestServer INSTANCE = null;
public static synchronized Neo4jTestServer getInstance() {
if (INSTANCE == null) {
INSTANCE = new Neo4jTestServer();
}
return INSTANCE;
}
private final ServerControls serverControls;
private Neo4jTestServer() {
serverControls = TestServerBuilders.newInProcessBuilder()
.withExtension(EXTENSION_MOUNT_POINT, EXTENSION_RESOURCES)
.newServer();
}
public ServerControls getServerControls() {
return serverControls;
}
public void shutdown() {
serverControls.close();
}
}
And my test class is looking like this:
public class TestResource {
private Neo4jTestServer server;
#Before
public void prepare(){
this.server = Neo4jTestServer.getInstance();
}
#After
public void endup(){
// Shutdown server
this.server.shutdown();
}
#Test
public void test(){
//TODO fill neo4j with data
HTTP.Response response = HTTP.GET(this.server.getServerControls().httpURI().resolve("/v1/calculation/test").toString());
}
}
Here is a link! on the resource i have used.
I also checked the questions from here! but i don't understand if they are running the tests on a 'real' neo4j instance or not.
Can this only be executed on the server or is there a way to run such tests locally?
EDIT:
I always get a 500 response calling the rest method.
And this is the output when executing the test:
Nov 04, 2015 10:33:17 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFORMATION: Scanning for root resource and provider classes in the packages:my.company.neo4j.extension
Nov 04, 2015 10:33:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFORMATION: Root resource classes found: class my.company.neo4j.extension.Resource
Nov 04, 2015 10:33:17 AM com.sun.jersey.api.core.ScanningResourceConfig init
INFORMATION: No provider classes found.
Nov 04, 2015 10:33:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFORMATION: Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM'
Nov 04, 2015 10:33:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFORMATION: Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM'
Nov 04, 2015 10:33:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFORMATION: Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM'
Nov 04, 2015 10:33:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFORMATION: Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM'
Is this "INFORMATION: No provider classes found." a problem maybe?
EDIT - this is my extension (it is just for testing)
#Path("/calculation")
public class ResourceV1 {
#Path("/test")
#GET
public Response test() throws Exception{
return Response.ok().build();
}
}
I also found the neo4j.log file from the temporary database:
2015-11-04 11:52:39.876+0100 INFO [o.n.s.d.LifecycleManagingDatabase] Successfully started database
2015-11-04 11:52:39.896+0100 INFO [o.n.s.CommunityNeoServer] Starting HTTP on port 7474 (4 threads available)
2015-11-04 11:52:40.087+0100 INFO [o.n.s.m.ThirdPartyJAXRSModule] Mounted unmanaged extension [my.company.neo4j.extension] at [/v1]
2015-11-04 11:52:40.156+0100 INFO [o.n.s.w.Jetty9WebServer] Mounting static content at /webadmin
2015-11-04 11:52:40.233+0100 WARN [o.n.s.w.Jetty9WebServer] No static content available for Neo Server at port 7474, management console may not be available.
2015-11-04 11:52:40.252+0100 INFO [o.n.s.w.Jetty9WebServer] Mounting static content at /browser
2015-11-04 11:52:41.267+0100 INFO [o.n.s.CommunityNeoServer] Remote interface ready and available at http://localhost:7474/

What you are trying is following
- Neo4JTestServer class is a factory for an in-memory Neo4j server. It's basically same as "real" one.
- TestResource class is a simple test for your extension.
Problem could be how do you load your extension into the test. Is your extension in the package name "my.company.neo4j.extension"?
Could you please show us code of your extension?
My suggestions are following
- read http://neo4j.com/docs/stable/server-unmanaged-extensions-testing.html
- look on GraphUnit which provide much better possibilities how to test extensions - https://github.com/graphaware/neo4j-framework/tree/master/tests

After setting up a new project I found the solution for my problem. In my maven dependencies I had both
<dependency>
<groupId>org.neo4j.test</groupId>
<artifactId>neo4j-harness</artifactId>
<version>${neo4j.version}</version>
<scope>test</scope>
</dependency>
and
<dependency>
<groupId>javax.ws.rs</groupId>
<artifactId>javax.ws.rs-api</artifactId>
<version>2.0</version>
<scope>provided</scope>
</dependency>
And in my case, here was the problem - after removing the dependency for javax ws the testcase was successful and I can call the rest methods.
EDIT: After removing the jax rs dependency it was not possible for me to build the extension because of the missing dependencies.
My solution: Adding following dependency instead of javax.ws.rs-api.
<dependency>
<groupId>org.neo4j.3rdparty.javax.ws.rs</groupId>
<artifactId>jsr311-api</artifactId>
<scope>provided</scope>
<version>1.1.2.r612</version>
</dependency>
It seems that it is working fine with neo4j-harness.

Related

How to configure spring-kafka's partition and replication in distributed environment?

I want to configure 3 partitions and 3 replications of a topic in distributed environment with three nodes. How can I configure these by java api without shell command?
If I have three nodes: node1, node2 and node3. I want partition1 and replication3 are deployed in node1, partition2 and replication1 are deployed in node2, partition3 and replication2 are deployed in node3.
I've tried spring-kafka's api in single-machine environment, this can create a topic and 1 partition automatically. But it not work in distributed environment.
My maven configuration is:
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
<version>1.1.7.RELEASE</version>
</dependency>
1.1.x is no longer supported; you should be using at least 1.3.9.
1.3.x comes with KafkaAdmin, which can automatically configure any NewTopic beans in the application context.
See Configuring Topics.
If you define a KafkaAdmin bean in your application context, it can automatically add topics to the broker. Simply add a NewTopic #Bean for each topic to the application context.
#Bean
public KafkaAdmin admin() {
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
StringUtils.arrayToCommaDelimitedString(kafkaEmbedded().getBrokerAddresses()));
return new KafkaAdmin(configs);
}
#Bean
public NewTopic topic1() {
return new NewTopic("foo", 10, (short) 2);
}
#Bean
public NewTopic topic2() {
return new NewTopic("bar", 10, (short) 2);
}

Tomcat 7+ memory leak on stop/redeploy. Spring Data, JPA, Hibernate, MySQL

I have tomcat memory leak issue when stop/redeploy application. It says The following web applications were stopped (reloaded, undeployed), but their
classes from previous runs are still loaded in memory, thus causing a memory
leak (use a profiler to confirm):/test-1.0-SNAPSHOT
MySQL connector driver located in Tomcat/lib folder.
I can reproduce this issue in both: Tomcat 7/8. Also tried MS SQL database with "net.sourceforge.jtds.*" driver but didn't help.
Please find below project files. Project only creates 1 table in DB.
build.gradle
group 'com.test'
version '1.0-SNAPSHOT'
apply plugin: 'java'
apply plugin: 'war'
sourceCompatibility = 1.8
repositories {
mavenCentral()
}
dependencies {
compile group: 'org.hibernate', name: 'hibernate-entitymanager', version: '5.2.10.Final'
compile group: 'org.springframework.data', name: 'spring-data-jpa', version: '1.11.4.RELEASE'
compile group: 'org.springframework', name: 'spring-webmvc', version: '4.3.9.RELEASE'
providedCompile 'javax.servlet:javax.servlet-api:3.1.0'
providedCompile group: 'mysql', name: 'mysql-connector-java', version: '5.1.6'
compile group: 'commons-dbcp', name: 'commons-dbcp', version: '1.4'
}
ApplicationConfig.java
#Configuration
#Import({JPAConfiguration.class})
#EnableWebMvc
public class ApplicationConfig {}
JPAConfiguration.java
#Configuration
#EnableJpaRepositories("com.test.dao")
#EnableTransactionManagement
public class JPAConfiguration {
#Bean
public EntityManagerFactory entityManagerFactory() {
LocalContainerEntityManagerFactoryBean factory = new LocalContainerEntityManagerFactoryBean();
factory.setJpaVendorAdapter(new HibernateJpaVendorAdapter());
factory.setPackagesToScan("com.test.model");
factory.setDataSource(restDataSource());
factory.setJpaPropertyMap(getPropertyMap());
factory.afterPropertiesSet();
return factory.getObject();
}
#Bean(destroyMethod = "close")
public DataSource restDataSource() {
BasicDataSource dataSource = new BasicDataSource();
dataSource.setDriverClassName("com.mysql.jdbc.Driver");
dataSource.setUrl("jdbc:mysql://localhost:3306/test");
dataSource.setUsername("test");
dataSource.setPassword("test");
return dataSource;
}
private Map<String, String> getPropertyMap() {
Map<String, String> hibernateProperties = new HashMap<>();
hibernateProperties.put("hibernate.hbm2ddl.auto", "update");
hibernateProperties.put("hibernate.show_sql", "true");
hibernateProperties.put("hibernate.format_sql", "true");
hibernateProperties.put("hibernate.dialect", "org.hibernate.dialect.MySQL5InnoDBDialect");
return hibernateProperties;
}
#Bean
public PlatformTransactionManager transactionManager() {
JpaTransactionManager txManager = new JpaTransactionManager();
txManager.setEntityManagerFactory(entityManagerFactory());
return txManager;
}
}
TestRepository.java
#Repository
public interface TestRepository extends JpaRepository<TestEntity, Long> {}
TestEntity.java
#Entity
#Table(name = "ent")
public class TestEntity {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
private String descript;
//equals, hashcode, toString, getters, setters
}
AppInitializer.java
public class AppInitializer extends AbstractAnnotationConfigDispatcherServletInitializer {
private WebApplicationContext rootContext;
#Override
protected Class<?>[] getRootConfigClasses() {
return new Class[]{ApplicationConfig.class};
}
#Override
protected Class<?>[] getServletConfigClasses() {
return null;
}
#Override
protected String[] getServletMappings() {
return new String[]{"/"};
}
}
Command
jmap -histo <tomcat_pid>
shows only 2 items from project structure after tomcat stop:
com.test.config.dao.JPAConfiguration$$EnhancerBySpringCGLIB$$792cb231$$FastClassBySpringCGLIB$$45ff499c
com.test.config.dao.JPAConfiguration$$FastClassBySpringCGLIB$$10104c1e
Anyone have ideas or suggestions to fix this problem?
There are 2 memory leaks in this small project:
The problem with MySQL jdbc driver.
We have to add ContextLoaderListener to deregister jdbc driver:
Listener:
#WebListener
public class ContextListener extends ContextLoaderListener {
private final Logger log = LoggerFactory.getLogger(this.getClass());
#Override
public void contextInitialized(ServletContextEvent sce) {
log.info("-= Context started =-");
}
#Override
public void contextDestroyed(ServletContextEvent sce) {
super.contextDestroyed(sce);
log.info("-= Context destroyed =-");
try {
log.info("Calling MySQL AbandonedConnectionCleanupThread checkedShutdown");
com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.uncheckedShutdown();
} catch (Exception e) {
log.error("Error calling MySQL AbandonedConnectionCleanupThread checkedShutdown {}", e);
}
ClassLoader cl = Thread.currentThread().getContextClassLoader();
Enumeration<Driver> drivers = DriverManager.getDrivers();
while (drivers.hasMoreElements()) {
Driver driver = drivers.nextElement();
if (driver.getClass().getClassLoader() == cl) {
try {
log.info("Deregistering JDBC driver {}", driver);
DriverManager.deregisterDriver(driver);
} catch (SQLException ex) {
log.error("Error deregistering JDBC driver {}", driver, ex);
}
} else {
log.info("Not deregistering JDBC driver {} as it does not belong to this webapp's ClassLoader", driver);
}
}
}
}
or if you had an access to tomcat server you can modify listener in tomcat/conf/server.xml example.
The second problem is known memory leak in jboss-logging library (link).
The memory leak has gone after we exclude this library from hibernate dependency:
build.gradle:
group 'com.test'
version '1.0-SNAPSHOT'
apply plugin: 'java'
apply plugin: 'war'
sourceCompatibility = 1.8
repositories {
mavenCentral()
}
dependencies {
compile(group: 'org.hibernate', name: 'hibernate-entitymanager', version: '5.2.10.Final') {
exclude group: 'org.jboss.logging', module: 'jboss-logging'
}
compile group: 'org.springframework.data', name: 'spring-data-jpa', version: '1.11.4.RELEASE'
compile group: 'org.springframework', name: 'spring-webmvc', version: '4.3.9.RELEASE'
providedCompile 'javax.servlet:javax.servlet-api:3.1.0'
providedCompile group: 'mysql', name: 'mysql-connector-java', version: '8.0.11'
compile group: 'org.slf4j', name: 'slf4j-api', version: '1.7.25'
compile group: 'org.slf4j', name: 'slf4j-simple', version: '1.7.25'
}
then build jar from repo and added to tomcat /lib folder.
The issue with jboss-logging probably fixed in Java 9 (pull request link).
Short answer - hopefully the same problem for you...
Those two com.test.config.dao.JPAConfiguration$$...CGLIB$$... classes were being referenced indirectly by the Abandoned connection cleanup thread in MySQL:
20-Jun-2018 21:25:22.987 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [test-1.0-SNAPSHOT] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:43)
The following answer enabled me to resolve the problem. E.g. in tomcat/conf/server.xml, look for the JreMemoryLeakPreventionListener line and replace it with this:
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
classesToInitialize="com.mysql.jdbc.Driver" />
This forces the MySQL JDBC driver, and its cleanup thread, to be loaded outside the classloader for the web application. This means the cleanup thread won't hold a reference to the webapp classloader as its context class loader.
Expanded answer - how to trace the leak in your environment...
Hopefully the above is all you need - it was enough to reproduce and solve the problem against https://github.com/egotovko/tomcat-leak
However there are many other causes of a leaked reference to a web application that can stop it undeploying. E.g. other threads still running (Tomcat is good at warning about these) or references from outside the web application.
To properly trace the cause, you can chase the reference in a heap dump. If this is not familiar, you can get a heap dump from jmap -dump:file=dump.hprof <pid>, or by directly connecting from such as jvisualvm (also included in the JDK).
With the heap dump open in jvisualvm:
Select the Classes button for the heap dump
Sort the list of classes by name
Look for classes in the web application - e.g. com.test.config.dao.JPAConfiguration$$EnhancerBySpringCGLIB$$ in this example
This should be showing with an instance count of 2 or so
Double click to show these in the Instances View
In the References pane for one of these instances, right click and Show Nearest GC Root
E.g. for that Abandoned connection cleanup thread in MySQL:
Note how the AbandonedConnectionCleanupThread has a contextClassLoader, which is the ParallelWebappClassLoader for the web application. Tomcat needs to be able to release the class loader to undeploy the web application.
Once you've tracked down what's holding the reference, it's then normally a case of investigating how better to configure that library in Tomcat, or perhaps someone else has seen that memory leak. It's also not uncommon to have to repeat the exercise, when there are several references to clear up.

Jersey test - ExceptionMapper is not invoked

I have REST web service which I test with Jersey Test, Mockito, Junit. When web service method is executed successfully, I get correct response. In case of invalid data, custom exception is thrown which must be handled by ExceptionMapper. It should return the same structure response but with different codes. ExceptionMapper works well in not test environment. However, after test execution logs show:
1 < 500
1 < Connection: close
1 < Content-Length: 1033
1 < Content-Type: text/html;charset=ISO-8859-1
1 < Date: Wed, 30 Mar 2016 11:55:37 GMT
javax.ws.rs.InternalServerErrorException: HTTP 500 Request failed.
at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1020)
at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:816)
at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:92)
I use:
<dependency>
<groupId>org.glassfish.jersey.test-framework.providers</groupId>
<artifactId>jersey-test-framework-provider-grizzly2</artifactId>
<version>2.22.2</version>
</dependency>
Shortened version of mocked service which is extented by test classes:
public class MockedService extends JerseyTest {
#Override
public ResourceConfig configure() {
forceSet(TestProperties.CONTAINER_PORT, "8080");
enable(TestProperties.LOG_TRAFFIC);
return new ResourceConfig().register(new Service());
}
}
How to get response from ExceptionMapper?
You still need to register the ExceptionMapper
return new ResourceConfig()
.register(new Service())
.register(new YourMapper());
In your real environment, the mapper is probably scanned for, either with package scanning or classpath scanning.

Azure service bus with qpid JMS java client messages not taking from queue

I am using Azure service-bus queues (AMQP Protocol) with Apache Qpid (0.3) as Java client.
I am also using Spring JmsTemplate to produce messages and DefaultMessageListenerContainer to manage my consumers, spring JMS 4.0.6.
Spring configurations:
#PostConstruct
private void JndiLookup() throws NamingException {
// Configure JNDI environment
Hashtable<String, String> envPrp = new Hashtable<String, String>();
envPrp.put(Context.INITIAL_CONTEXT_FACTORY,
PropertiesFileInitialContextFactory.class.getName());
envPrp.put("connectionfactory.SBCF", "amqps://owner:{parimeryKey}#{namespace}.servicebus.windows.net");
envPrp.put("queue.STORAGE_NEW_QUEUE", "QueueName");
context = new InitialContext(envPrp);
}
#Bean
public ConnectionFactory connectionFactory() throws NamingException {
ConnectionFactory cf = (ConnectionFactory) context.lookup("SBCF");
return cf;
}
#Bean
public DefaultMessageListenerContainer messageListenerContainer() throws NamingException {
DefaultMessageListenerContainer messageListenerContainer = new DefaultMessageListenerContainer();
messageListenerContainer.setConnectionFactory(connectionFactory());
Destination queue = (Destination) context.lookup("QueueName");
messageListenerContainer.setDestination(queue);
messageListenerContainer.setConcurrency("3-10");
MessageListenerAdapter adapter = new MessageListenerAdapter();
adapter.setDelegate(new MessageWorker());
adapter.setDefaultListenerMethod("onMessage");
messageListenerContainer.setMessageListener(adapter);
return messageListenerContainer;
}
#Bean
public JmsTemplate jmsTemplate() throws NamingException {
JmsTemplate jmsTemplate = new JmsTemplate();
jmsTemplate.setConnectionFactory(connectionFactory());
return jmsTemplate;
}
Nothing fancy in the configurations just straight forward.
Running the code and everything seems to be working .. but after few minutes without traffic in the queue it seems like the consumers are losing connection with the queue and not taking messages.
I dont know if it is related but every 5 minutes in am getting the following warning:
Fri Nov 07 15:23:53 +0000 2014, (DefaultMessageListenerContainer.java:842) WARN : Setup of JMS message listener invoker failed for destination 'org.apache.qpid.amqp_1_0.jms.impl.QueueImpl#8fb0427b' - trying to recover. Cause: Force detach the link because the session is remotely ended.
Fri Nov 07 15:23:56 +0000 2014, (DefaultMessageListenerContainer.java:891) INFO : Successfully refreshed JMS Connection
I have messages being in the queue for hours and not being handled by the consumers only when I restart the app the consumers renewing the connection properly and taking the messages.
Is it possible that the problem is with the Spring Listener container properties or qpid connection factory or is it an issue with Azure service bus??
Couldn't find related post to my situation will appreciate the help!!

main() throws exception from createEntityManager() when using EclipseLink

I have a simple program using JPA entities to write into a Derby DB (the entities were generated from an existing DB tables). I am using Eclipse and there is a working connection between the Derby client and the server via the EclipseLink Data Source Explorer .
Here is the start of my main():
import javax.persistence.*;
import java.sql.Timestamp;
import java.util.*;
import javax.*;
public class start {
/**
* #param args
*/
private static final String PERSISTENCE_UNIT_NAME = "zodiac";
private static EntityManagerFactory factory;
public static void main(String[] args) {
try {
factory = Persistence.createEntityManagerFactory(PERSISTENCE_UNIT_NAME);
EntityManager em = factory.createEntityManager();
System.out.println("after factory gen" );
when I the line with createEntityManager() is executed the following exception is thrown:
[EL Info]: 2012-03-07 22:46:21.892--ServerSession(253038357)--EclipseLink, version: Eclipse Persistence Services - 2.3.2.v20111125-r10461
[EL Severe]: 2012-03-07 22:46:22.064--ServerSession(253038357)--Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLException: No suitable driver
Error Code: 0
Any idea what is the problem ? thanks
If you're in Eclipse you need to add the driver to your project classpath. Sounds like you already have a datasource so you must have defined the driver library. All you need to do is "Add Library" to your Java Build Path and choose "Connectivity Driver Definition" and then the Derby driver from the drop down list of available driver definitions.
FYI, there's a checkbox in the New JPA Project wizard where you can select "add driver to classpath" to do this when you create a new project.
Of course you can also add the derbyclient.jar to your classpath directly or define a user library that includes it.
--Shaun