I'm testing spring-yarn integration API and I'm little confused about what is the best practice of Yarn container customization in terms of:
1) If I want to use spring-boot-yarn combo, what is the correct way of telling the spring boot to pick up my implementation of yarn container instead of DefaultYarnContainer...The only way I figured out was via ImportResource annotation at container project class containing main method, which was pointing to spring application xml with declaration:
<yarn:container container class="myhadoop.yarn.container.custom.MyContainerImplementation"/>
Component scan doesn't work at all...Spring boot was still using DefaultYarnContainer...
2) If I understand Yarn architecture correctly then application master is responsible for launching the container. But If I change DefaultYarnContainer for my implementation then I need to start container manually via run method, nothing was starting it, please what is the correct way?
Thanks a lot in advance for help
If boot is doing auto-configuration for yarn container, there are few ways to define the actual container which defaults to DefaultYarnContainer.
Logic of this can be found from here https://github.com/spring-projects/spring-hadoop/blob/master/spring-yarn/spring-yarn-boot/src/main/java/org/springframework/yarn/boot/YarnContainerAutoConfiguration.java#L107
Use spring.yarn.container.containerClass=foo.jee.MyContainer in yml
Create class as bean with name yarnContainerClass
Create your container impl as bean with name yarnContainerRef
Create bean as name customContainerClass which would be a class as string
Janne, thanks a lot! This way is far more elegant and it works...Here is what I did:
#EnableAutoConfiguration
#Configuration
#ComponentScan
public class ContainerApplication {
#Autowired
private MyContainerImplementation myContainerImplementation;
#Bean(name="yarnContainerClass")
public Class<? extends YarnContainer> getYarnContainerClass() {
return MyContainerImplementation.class;
}
#Bean(name="yarnContainerRef")
public MyContainerImplementation getYarnContainerRef() {
return myContainerImplementation;
}
#Bean(name="customContainerClass")
public String getCustomContainerClass() {
return "myhadoop.yarn.container.custom.MyContainerImplementation";
}
public static void main(String[] args) {
SpringApplication.run(ContainerApplication.class, args);
}
}
I added MyContainerImplementation into yml as you pointed and my container implementation was started by application master without me running the run method manually, because I see the following lines in the hadoop logs:
LifecycleObjectSupport:started
myhadoop.yarn.container.custom.MyContainerImplementation#5e2cd950
.
.
LifecycleObjectSupport: stopped
myhadoop.yarn.container.custom.MyContainerImplementation#5e2cd950
Anyway, I have additional question. I wanted to test low-level yarn things like ContainerStateListener and YarnPublisher, but they're not called at all..:-( Here is my test customized container:
#Component
public class MyContainerImplementation extends AbstractYarnContainer {
private static final Log log = LogFactory.getLog(MyContainerImplementation.class);
public MyContainerImplementation() {
super();
log.info("...Initializing yarn MyContainerImplementation....");
this.setYarnEventPublisher(new DefaultYarnEventPublisher() {
#Override
public void publishContainerAllocated(Object source, Container container) {
super.publishContainerAllocated(source, container);
log.info("Yarn container allocated: "+container.getResource().getMemory());
}
});
this.addContainerStateListener(new ContainerStateListener() {
#Override
public void state(ContainerState state, Object exit) {
switch(state) {
case COMPLETED: {
log.info("...Container started successfully!...");
break;
}
case FAILED: {
log.info("...Starting of container failed!...");
break;
}
default: {
log.info("Unexpected container state...exiting!...");
}
}
}
});
}
public void runInternal() {
log.info("...Running internal method...");
}
}
Do I need to add additional configuration to make ContainerStateListener and YarnPublisher to work?
You don't need to implement all beans for container, just one is
enough.
You're right. I didn't notice of the following method in SpringYarnConfig:
#Override
public void configure(YarnContainerConfigurer container) throws Exception {
if (StringUtils.hasText(sycp.getContainerClass())) {
container
.containerClass(sycp.getContainerClass());
} else if (yarnContainerClass != null){
container
.containerClass(yarnContainerClass);
} else if (yarnContainerRef != null) {
if (yarnContainerRef instanceof YarnContainer) {
container
.containerRef((YarnContainer) yarnContainerRef);
}
} else if (StringUtils.hasText(containerClass)) {
container.containerClass(containerClass);
}
}
where it's clear that one is really enough...:)
Use of event publisher and state listener are really meant as building
blocks what you'd need to call yourself.
Yes, I see that DefaultYarnContainer is handling notifyXXXX methods(fires up container state changes) calling by himself...Alright, I will definitelly play with it more.
Janne, thanks a lot for help. You provided perfect closer insight into Spring-Yarn.
Related
I am trying to use DynamoDbLocal server for unit test cases.
And came up with two options,
Either define a junit class rule which starts local server before class and stops it after class. So essentially it will start and stop server for each unit test class.
public class MyDynamoDbLocalServerRule extends ExternalResource {
#Override
protected void before() throws Throwable {
myInMemoryDynamoDbServer.start();
}
#Override
protected void after() throws Throwable{
inMemoryDynamoDbServer.stop();
}
OR
Singleton instance :
public static DynamoDBProxyServerContainer createInstance(final int portToListenIn) {
if (dynamoLocal == null) {
synchronized (DynamoDBProxyServerContainer.class) {
if (dynamoLocal == null) {
dynamoLocal = new DynamoDBProxyServerContainer(portToListenIn);
}
}
}
return dynamoLocal;
}
private DynamoDBProxyServerContainer(final int portToListenIn) {
this.startServer(portToListenIn);
getRuntime().addShutdownHook(new Thread(() -> stopServer()));
}
Which one would you recommend and do you have better of doing this ? Please note i should be able to use it with Guice dependency injection framework.
I would recommend Singleton approach as creating the database instance for each test case will be a time consuming option. Also, if you have many test cases, the unit testing is likely to take more time to complete. If you have continuous integration, the build and unit test would take more time.
As the unit tests run in sequential manner, you don't need separate instance for each test case.
I'd like to run Surefire in parallel mode (multiple JVMs) where each JVM must run:
SLF4JBridgeHandler.removeHandlersForRootLogger();
SLF4JBridgeHandler.install();
exactly once before the first test. How can this be done?
There are various ways to make some code run at the beginning of a test suite.
Here are 4 (I'm sure there are more):
JUnit via RunWith Suite with Suite.SuiteClasses and BeforeClass (adapted from examples in SuiteTest):
#RunWith(Suite.class)
#SuiteClasses({FirstTest.class, SecondTest.class/*, ...*/, LastTest.class})
public static class AllWithSLF4JBridgeHandler {
#BeforeClass
public static void registerRootLoggerHandlers() {
SLF4JBridgeHandler.removeHandlersForRootLogger();
SLF4JBridgeHandler.install();
}
}
TestNG with BeforeSuite:
/**
* Base class for each test class (i.e. every test class should extend this class).
*/
public abstract class BaseTest {
#BeforeSuite
public void registerRootLoggerHandlers() {
SLF4JBridgeHandler.removeHandlersForRootLogger();
SLF4JBridgeHandler.install();
}
}
TestNG with Guice:
/**
* Test module. Each test class should be annotated with `#Guice(TestModule.class)`.
*/
public class TestModule implements Module {
#Override
public void configure(Binder binder) {
SLF4JBridgeHandler.removeHandlersForRootLogger();
SLF4JBridgeHandler.install();
}
}
Static initialization blocks (test-framework independent):
/**
* Base class for each test class (i.e. every test class should extend this class).
*/
public abstract class BaseTest {
static {
SLF4JBridgeHandler.removeHandlersForRootLogger();
SLF4JBridgeHandler.install();
}
}
I'm not sure how all of these methods work with Surefire's parallel mode. Methods 1 and 2 may not work there but I believe methods 3 and 4 should.
Another option would be to not use the programmatic installation of the SLF4JBridgeHandler but to use a java.util.logging.config file or class (see LogManager):
"java.util.logging.config.file":
logging.properties file:
// register SLF4JBridgeHandler as handler for the j.u.l. root logger
handlers = org.slf4j.bridge.SLF4JBridgeHandler
System property assignment:
java -Djava.util.logging.config.file=/path/to/logging.properties ...
This works well if you know the path to your logging file beforehand.
"java.util.logging.config.class":
Using a file may not be a good option if you're deploying a WAR and don't know where the file will be, etc. so alternatively you can create a logging config class:
public class SLF4JBridgeHandlerInitializer {
public SLF4JBridgeHandlerInitializer() throws IOException {
String loggingConfigurationString = "handlers = " + SLF4JBridgeHandler.class.getName();
InputStream inputStream = new ByteArrayInputStream(loggingConfigurationString.getBytes());
LogManager.getLogManager().readConfiguration(inputStream);
}
}
System property assignment:
java -Djava.util.logging.config.class=package.SLF4JBridgeHandlerInitializer ...
I've done this before and it has worked well for me (SLF4JBridgeHandler.Initializer by mfulton26 · Pull Request #57 · qos-ch/slf4j).
These final two options should initialize each JVM instance as long as the appropriate system property is set.
Does Jodd framework provide mechanism to inject petitebeans references for the objects created by other frameworks.
Below are scenarios
- Domain/Service objects are created by Spring Framework
- Domain objects created are by ORM Frameworks
- These objects need to be injected with Repository/DAO object (Singleton objects registered as PetiteBean via AutomagicPetiteConfigurator)
Below is sample code, after petite container is shutdown, initMethod() is invoked when pc.getBean(Greetings.class).message(null) is invoked and destroyMethod() is not invoked, can you please point me what I am doing wrong?
#PetiteBean("greetings")
public class EnglishGreetings implements Greetings {
#Override
public String message(String message) {
if (message == null) {
return "defaultMessage";
}
return message;
}
#PetiteInitMethod
public void initMethod() {
System.out.println("Entered initMethod");
}
#PetiteDestroyMethod
public void destroyMethod() {
System.out.println("Entered destroyMethod");
}
}
public class GreetingRunner {
final static Logger logger = LoggerFactory.getLogger(GreetingRunner.class);
#PetiteInject
public Greetings greetings;
public static void main(String s[]) {
jodd.log.LoggerFactory.setLoggerFactory(new Slf4jLoggerFactory());
PetiteContainer pc = new PetiteContainer();
AutomagicPetiteConfigurator configurator = new AutomagicPetiteConfigurator();
configurator.setIncludedEntries("com.rans.*");
configurator.configure(pc);
pc.shutdown();
System.out.println(pc.getBean(Greetings.class).message(null));
}
}
Destroy method has not been invoked because of lazy aspect of Petite - if bean has not been used, no destroy method will be called. The same applies to init methods. If bean is not used, Petite simple ignores it.
Now back to the question:
Does Jodd framework provide mechanism to inject petitebeans references for the objects created by other frameworks.
Technically, yes - if you overwrite it :) See PetiteProxettaContainer. You may override getBean and use 3rd party container to fetch the bean. Actually, you may override createBeanDefinitionForRegistration method to register the bean in the different container. To be honest, we might make this more obvious :)
(Sorry for late response)
I want to auto register all interfaces which name ends with "Service" and also doesn't have concrete implementations to be resolved to a generated type/proxy (which off course differs per interface).
So when I want to resolve IContractService I want it to return a proxied object. I got this idea from this article where they implemented it in some way with Castle Windsor.
What would be the structuremap approach for achieving this. I tried all kind of things with custom conventions and all but I can't get my head around it.
I fixed this by using Castle's Dynamic Proxy and a StructureMap convention. BTW. I also renamed some of the classes mentioned in the article.
public class InfraRegistry : Registry
{
public InfraRegistry()
{
For<IClientProviderFactory>().Use<WcfClientProviderProviderFactory>();
Scan(scan =>
{
scan.AssemblyContainingType<MidleWareServiceConvention>();
scan.Convention<MidleWareServiceConvention>();
});
}
}
public class MidleWareServiceConvention : IRegistrationConvention
{
private readonly ProxyGenerator _proxyGen = new ProxyGenerator();
public void Process(Type type, Registry registry)
{
if (type.IsInterface && type.Name.EndsWith("Service", StringComparison.OrdinalIgnoreCase))
{
registry.For(type)
.HybridHttpOrThreadLocalScoped()
.Use(
context =>
_proxyGen.CreateInterfaceProxyWithoutTarget(type,
new WcfInterceptor(
context.GetInstance<IClientProviderFactory>())));
}
}
}
I'm experimenting with interception in Castle Windsor and notice that interceptors seem to be created as decorators of my service interface.
In other words, if I have an interface "ISomethingDoer" and a concrete "ConcreteSomethingDoer", the proxy implements ISomethingDoer but does not inherit from ConcreteSomethingDoer.
This is fine, and no doubt by design, but what I'm wondering is whether I can intercept protected virtual methods in my concrete classes that wouldn't be known by the public interface. I am doing this in order to add logging support, but I might want to log some of the specific internal details of a class.
In my slightly unimaginative test case I have this:
public interface ISomethingDoer
{
void DoSomething(int Count);
}
[Loggable]
public class ConcreteSomethingDoer : ISomethingDoer
{
public void DoSomething(int Count)
{
for (var A = 0; A < Count; A++)
{
DoThisThing(A);
}
}
[Loggable]
protected virtual void DoThisThing(int A)
{
("Doing a thing with " + A.ToString()).Dump();
}
}
So what I want to do is log calls to "DoThisThing" even though it's not part of the interface.
I've managed to get this working in Autofac. (I've created a Linqpad script here: http://share.linqpad.net/frn5a2.linq) but am struggling with Castle Windsor (see http://share.linqpad.net/wn7877.linq)
In both cases my interceptor is the same and looks like this:
public class Logger : IInterceptor
{
public void Intercept(IInvocation Invocation)
{
String.Format("Calling method {0} on type {1} with parameters {2}",
Invocation.Method.Name,
Invocation.InvocationTarget.GetType().Name,
String.Join(", ", Invocation.Arguments.Select(a => (a ?? "*null*").ToString()).ToArray())).Dump();
Invocation.Proceed();
"Done".Dump();
}
}
What I really want to do is say "any classes with a [Loggable] attribute, should use the logging interceptor". In the Autofac example I've specifically attached a logger to the registration, whereas with Castle I'm using an IModelInterceptorsSelector which looks like this:
public class LoggerInterceptorSelector : IModelInterceptorsSelector
{
public bool HasInterceptors(ComponentModel Model)
{
return Model.Implementation.IsDefined(typeof(LoggableAttribute), true);
}
public InterceptorReference[] SelectInterceptors(ComponentModel Model, InterceptorReference[] Interceptors)
{
return new[]
{
InterceptorReference.ForType<Logger>()
};
}
}
Finally, the code to execute all this is:
var Container = new WindsorContainer();
Container.Register(
Component.For<Logger>().LifeStyle.Transient
);
Container.Kernel.ProxyFactory.AddInterceptorSelector(new LoggerInterceptorSelector());
Container.Register(
Component.For<ISomethingDoer>()
.ImplementedBy<ConcreteSomethingDoer>()
.LifeStyle.Transient
);
var Doer = Container.Resolve<ISomethingDoer>();
Doer.DoSomething(5);
When run I would expect to see "Calling method DoThisThing with parameters x" for each time the method is called. Instead I only get the call to DoSomething logged.
I can see why Castle Windsor is doing this, but I'm wondering if there is a way to tweak the behaviour?
(As a side-note I don't want to use Windsor's own interceptor attributes as I don't want to introduce dependencies to Castle outside of my composition root.)
I have tried resolving the ConcreteSomethingDoer specifically and this works, but not if I'm resolving the ISomethingDoer.
Apologies for the long post, and also apologies because I am pretty new to Castle Windsor!
I you could register like:
Container.Register(
Component.For<ISomethingDoer, ConcreteSomethingDoer>()
.ImplementedBy<ConcreteSomethingDoer>()
.LifeStyle.Transient
);
This should create a class proxy by deriving from ConcreteSomethingDoer. However this won't work with dynamic interceptors. However you probably can work around that by creating a facility which registers the interceptor when needed.