I want to configure 3 partitions and 3 replications of a topic in distributed environment with three nodes. How can I configure these by java api without shell command?
If I have three nodes: node1, node2 and node3. I want partition1 and replication3 are deployed in node1, partition2 and replication1 are deployed in node2, partition3 and replication2 are deployed in node3.
I've tried spring-kafka's api in single-machine environment, this can create a topic and 1 partition automatically. But it not work in distributed environment.
My maven configuration is:
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
<version>1.1.7.RELEASE</version>
</dependency>
1.1.x is no longer supported; you should be using at least 1.3.9.
1.3.x comes with KafkaAdmin, which can automatically configure any NewTopic beans in the application context.
See Configuring Topics.
If you define a KafkaAdmin bean in your application context, it can automatically add topics to the broker. Simply add a NewTopic #Bean for each topic to the application context.
#Bean
public KafkaAdmin admin() {
Map<String, Object> configs = new HashMap<>();
configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
StringUtils.arrayToCommaDelimitedString(kafkaEmbedded().getBrokerAddresses()));
return new KafkaAdmin(configs);
}
#Bean
public NewTopic topic1() {
return new NewTopic("foo", 10, (short) 2);
}
#Bean
public NewTopic topic2() {
return new NewTopic("bar", 10, (short) 2);
}
Related
I am using Redis Server for message broker in my spring boot application.
Is there any simple way to Junit my publish and receive API?
e.g :
Publisher :
public String publish(Object domainObj) {
template.convertAndSend(topic.getTopic(), domainObj.toString());
return "Event Published";
}
Receiver :
public class Receiver implements MessageListener {
#Override
public void onMessage(Message message, byte[] bytes) {
System.out.println("Consumed Message {}" + message);
}
}
I am using JedisConnectionFactory and RedisMessageListenerContainer and RedisTemplate for my implementation
#Configuration
#EnableRedisRepositories
public class RedisConfig {
#Bean
public JedisConnectionFactory connectionFactory() {
RedisStandaloneConfiguration configuration = new RedisStandaloneConfiguration();
configuration.setHostName("localhost");
configuration.setPort(6379);
return new JedisConnectionFactory(configuration);
}
#Bean
public RedisTemplate<String, Object> template() {
RedisTemplate<String, Object> template = new RedisTemplate<>();
template.setConnectionFactory(connectionFactory());
template.setKeySerializer(new StringRedisSerializer());
template.setHashKeySerializer(new StringRedisSerializer());
template.setHashKeySerializer(new JdkSerializationRedisSerializer());
template.setValueSerializer(new JdkSerializationRedisSerializer());
template.setEnableTransactionSupport(true);
template.afterPropertiesSet();
return template;
}
#Bean
public ChannelTopic topic() {
return new ChannelTopic("common-channel");
}
#Bean
public MessageListenerAdapter messageListenerAdapter() {
return new MessageListenerAdapter(new Receiver());
}
#Bean
public RedisMessageListenerContainer redisMessageListenerContainer() {
RedisMessageListenerContainer container = new RedisMessageListenerContainer();
container.setConnectionFactory(connectionFactory());
container.addMessageListener(messageListenerAdapter(), topic());
return container;
}
Unit Testing Receiver and Publisher implementation is quite straight.
JUnit 5 coupled with Mockito extension should do the job.
For example for testing that :
public String publish(Object domainObj) {
template.convertAndSend(topic.getTopic(), domainObj.toString());
return "Event Published";
}
I expect that topic and template be fields of the current class.
These fields could be set by constructor.
So you could write something that check that convertAndSend() is eventually executed with the correct parameters :
#Mock
RedisTemplate<String, Object> templateMock;
#Test
void publish(){
Topic topicFixture = new Topic(...);
Object domainObjFixture = new FooBar(...);
Publisher publisher = new Publisher(templateMock, topicFixture);
//when
publisher.publish(domainObjFixture);
// then
Mockito.verify(templateMock)
.convertAndSend(topicFixture.getTopic(), domainObjFixture);
}
But I don't think that the unit test of these two classes be enough because it never tests the final things : the JMS processing performed by Redis backend.
Particularly, the RedisConfig part that you set with specific things as serializers that have important side effects on the processing.
For my part, I try to always write integration or partial integration tests for Redis backend stuffs to ensure a good no regression harness.
The java embedded-redis library is good for that. It allows to start a redis server
on localhost (works on Windows as well as on Linux).
Starting and stopping the redis server is as simple as :
RedisServer redisServer = new RedisServer(6379);
redisServer.start();
// do some work
redisServer.stop();
Move the start() in the #BeforeEach and the stop() in the #AfterEach and the server is ready.
Then it still requires some adjustments to ensure that the redis configuration specified in Spring is well setup during the tests while using your local redis server and not the "real" redis server. Not always simple to set but great when it is done !
The simplest way to unit test this is to use embedded-redis module. What you do is in BeforeAll you can start embedded Redis and stop the embedded Redis in AfterAll method.
You can also PostConstruct PreDestroy annotations to accomplish this.
If you're looking for Junit5 then you can find the code in my repo here
See BootstrapRedis annotation and their usage here
https://github.com/sonus21/rqueue/blob/7ef545c15985ef91ba719f070f7cc80745525047/rqueue-core/src/test/java/com/github/sonus21/rqueue/core/RedisScriptFactoryTest.java#L40
It's there anyway to create a springboot application that whent it runs for the first time, connect to mysql, and creates the database schema if it do not exists?
I'm using this configuration:
#Configuration
public class DataConfiguration {
#Bean
public DataSource dataSource(){
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("com.mysql.jdbc.Driver");
dataSource.setUrl("jdbc:mysql://localhost:3306/dbname");
dataSource.setUsername("root");
dataSource.setPassword("root");
return dataSource;
}
#Bean
public JpaVendorAdapter jpaVendorAdapter(){
HibernateJpaVendorAdapter adapter = new HibernateJpaVendorAdapter();
adapter.setDatabase(Database.MYSQL);
adapter.setGenerateDdl(true);
adapter.setDatabasePlatform("org.hibernate.dialect.MySQLDialect");
adapter.setPrepareConnection(true);
return adapter;
}
Sure,
Spring boot already has an integration with Flyway and Liquidbase.
For example, Flyway allows creating the schema and running a series of database migrations upon the first run.
If the migrations are already done, flyway won't change the database schema.
This tool is really powerful and you can configure it to run when the spring boot application starts.
Check This document for more information about spring boot integration with database related tools
I have tried doing some poc on springfox swagger with spring boot. It does generate swagger ui on the same host and port as my application is running.
http://localhost:8080/swagger-ui.html
My application is composed of multiple microservices deployed on a cloud infrastructure. This way i may end up having multiple swagger hub ui as
http://microservice1:8080/swagger-ui.html
http://microservice2:8081/swagger-ui.html
http://microservice3:8082/swagger-ui.html
How i can host all of my springfox swagger hub application on same host. So that i can have a consolidate webpage to have all my api documentation at single place.
For spring rest doc, i could generate a single html document using asciidoctor for my microservice. Again i had different html docs for different microservices.
Is this feature available with spring rest doc? or in spring cloud where consolidate all my documents in one single web application.
Create a Zuul filter concept. Create a Zuul filter service and add swagger2 dependency in your pom.xml and create a configuration class in this service as mentioned below
#EnableSwagger2
#Configuration
#Component
#Primary
public class SwaggerConfig implements SwaggerResourcesProvider {
#Override
public List<SwaggerResource> get() {
List resources = new ArrayList<>();
resources.add(swaggerResource("microservice1", "/microservice1/v2/api-docs", "2.0"));
resources.add(swaggerResource("microservice2", "/microservice2/v2/api-docs", "2.0"));
resources.add(swaggerResource("microservice3", "/microservice3/v2/api-docs", "2.0"));
return resources;
}
private SwaggerResource swaggerResource(String name, String location, String version) {
SwaggerResource swaggerResource = new SwaggerResource();
swaggerResource.setName(name);
swaggerResource.setLocation(location);
swaggerResource.setSwaggerVersion(version);
return swaggerResource;
}
}
Add below mentioned configuration in the other three microservices(microservice1,microservice2,microservice3..)
#Configuration
#EnableSwagger2
public class SwaggerConfig {
#Bean
public Docket api() {
return new Docket(DocumentationType.SWAGGER_2).select()
.apis(RequestHandlerSelectors.basePackage("com.stackoverflow.login.controller"))
.paths(PathSelectors.regex("/.*")).build().apiInfo(apiEndPointsInfo());
}
private ApiInfo apiEndPointsInfo() {
return new ApiInfoBuilder().title("Spring Boot REST API").description("Employee Management REST API")
.contact(new Contact("stackoverflow", "www.stackoverflow.com", ""))
.license("Apache 2.0").licenseUrl("http://www.apache.org/licenses/LICENSE-2.0.html").version("1.0.0")
.build();
}
}
I am new to Spring boot.I want to add some sql while database is creating like seed data.
#Value("classpath:com/foo/sql/db-test-data.sql")
private Resource dataScript;
#Bean
public DataSourceInitializer dataSourceInitializer(final DataSource dataSource) {
final DataSourceInitializer initializer = new DataSourceInitializer();
initializer.setDataSource(dataSource);
initializer.setDatabasePopulator(databasePopulator());
return initializer;
}
private DatabasePopulator databasePopulator() {
final ResourceDatabasePopulator populator = new ResourceDatabasePopulator();
populator.addScript(dataScript);
return populator;
}
props.put("hibernate.query.substitutions", "true 1, false 0");
props.put("hibernate.hbm2ddl.auto", "create-drop");
props.put("hibernate.show_sql", "false");
props.put("hibernate.format_sql", "true");
I have perform this action.But it not working on spring boot.Can any one help me.
Sometimes spring-boot gets more in the way than it helps; IMHO this is especially so with web applications.
What you can do to get around this is to rename the bean that you define.
#Bean("springBootPleaseStopTellingMeHowYouThinkDataSourceInitializer")
public DataSourceInitializer dataSourceInitializer(DataSource dataSource) {
// build it.
}
Now, to turn off the built in bit that looks for data.sql in application.properties
spring.datasource.initialize=false
There, now boot is booted out of the way.
You can take advantage of Spring Boot database initialization capabilities. The simplest way is to place a "data.sql" file in the root of the classpath. So you just need to:
Change your sql file name to "data.sql".
Place it in "src/main/resources".
Spring Boot will automatically pick up the file and use it to initialize the database on startup.
You can check the documentation if you need to customize the file name, location, etc.
I want to use spring data jpa repository. And I have to connect to 2 databases. I already found so many similar questions. But most of answers are using Entity manager instead of repository.
For example
#persistenceContext(unitname = "example")
Entitymanager em;
But I want to use repository of spring data jpa. How can I configure in applicationContext.xml?
My 2 databases are MySQL and one is local another is remote server.
You can separate those repositories in different packages. Then its possible to create two DB's configurations with different enity manager factory and transaction manager.
For example first config:
#Configuration
#EnableJpaRepositories(basePackages = "com.firstpackage",
entityManagerFactoryRef = "entityManagerFactoryDb1",
transactionManagerRef = "transactionManagerDb1")
public class DB1Config {
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactoryDb1() {
...
}
public JpaTransactionManager transactionManagerDb1() {
...
}
public DataSource dataSourceDb11() {
...
}
Second config would be similar.
You didnt put many details in question, but in case you need to switch databases for same repositories ( for example for different locales ) you can use AbstractRoutingDataSource class which will define determineCurrentLookupKey method.