Ribbon without Eureka : Ribbon doesn't remember that a server is down via PingURL - spring-cloud-netflix

Question

I am using Ribbon without Eureka.I am using a ConfigurationBasedServerList to provide the list of server instances like so..
customerinfo.ribbon.listOfServers=localhost:9003,localhost:9008
I have configured PingURL with the /health endpoint. I have also configured AvailabilityFilteringRule which needs to filter the sever instances that are not available. like so..
public class RibbonConfig {
#Autowired
IClientConfig ribbonClientConfig;
#Bean
public IPing ribbonPing(IClientConfig config) {
return new PingUrl(true, "/health");
}
#Bean
public IRule ribbonRule(IClientConfig config) {
return new AvailabilityFilteringRule();
}
}
This mostly works well. It doesn't work well in one case. Thats the case when the server instance running on port 9008 is down.
Let me explain with some DEBUG messages.
DEBUG com.netflix.loadbalancer.DynamicServerListLoadBalancer - List of Servers for customerinfo obtained from Discovery client: [localhost:9003, localhost:9008]
DEBUG com.netflix.loadbalancer.DynamicServerListLoadBalancer - Filtered List of Servers for customerinfo obtained from Discovery client: [localhost:9003, localhost:9008]
DEBUG com.netflix.loadbalancer.BaseLoadBalancer - LoadBalancer: clearing server list (SET op)
DEBUG com.netflix.loadbalancer.BaseLoadBalancer - LoadBalancer: addServer [localhost:9003]
DEBUG com.netflix.loadbalancer.BaseLoadBalancer - LoadBalancer: addServer [localhost:9008]
com.netflix.loadbalancer.DynamicServerListLoadBalancer - Setting server list for zones: {unknown=[localhost:9003, localhost:9008]}
DEBUG com.netflix.loadbalancer.BaseLoadBalancer - LoadBalancer: clearing server list (SET op)
DEBUG com.netflix.loadbalancer.BaseLoadBalancer - LoadBalancer: addServer [localhost:9003]
DEBUG com.netflix.loadbalancer.BaseLoadBalancer - LoadBalancer: addServer [localhost:9008]
DEBUG com.netflix.loadbalancer.BaseLoadBalancer - LoadBalancer: forceQuickPing invoked
DEBUG com.netflix.loadbalancer.BaseLoadBalancer - LoadBalancer: PingTask executing [2] servers configured
DEBUG com.netflix.loadbalancer.BaseLoadBalancer - LoadBalancer: Server [localhost:9008] status changed to DEAD
Looking at the DEBUG messages. The process thats being followed looks like this:
1) Clear the server list and add the servers from the config again.
2) Ping them for their status.
3) Update the available server list depending on the ping results.
Every 30 secs the above process seems to be happening which is to maintain the DynamicServerList.
Now, the problem is - from the first log statement to the penultimate log statement, ribbon thinks both the server instances are available. So, if there is load balancing request that comes within that time, then there is a chance that its send to the server localhost:9008 which is DOWN.
From my understanding Ribbon library does not keep PingStatistics. I think the library depends on Service Discovery tools like Eureka to provide the DynamicServerlist which are healthy depending on some health checks.
Now, to fix this problem, I can start using Eureka and this problem might vanish. I don't want to use Eureka as my environment doesnt grow/shrink often...its pretty much static.
Is there a config that i am missing here? How do we solve this issue?
I am using "spring-cloud-starter-ribbon" Version 1.2.6.RELEASE.

Answer 1

All the available IRule implementations do not use reachableServers correctly, we have to implement a new IRule.
#Slf4j
public class LoadBalanceConfig {
#Bean
public IClientConfig ribbonClientConfig() {
DefaultClientConfigImpl config = new DefaultClientConfigImpl();
config.set(IClientConfigKey.Keys.IsSecure, false);
config.set(IClientConfigKey.Keys.ListOfServers, XXXXXXX);
config.set(IClientConfigKey.Keys.ServerListRefreshInterval, 3000);
return config;
}
#Bean
public ServerList<Server> ribbonServerList(IClientConfig clientConfig) {
AbstractServerList<Server> lst = new ConfigurationBasedServerList();
lst.initWithNiwsConfig(clientConfig);
return lst;
}
#Bean
public ServerListFilter<Server> ribbonServerListFilter() {
return new AbstractServerListFilter<Server>() {
#Override
public List<Server> getFilteredListOfServers(List<Server> servers) {
return servers;
}
};
}
// modified from com.netflix.loadbalancer.RoundRobinRule
public static class RoundRobinRule implements IRule {
private ILoadBalancer lb;
private AtomicInteger nextServerCyclicCounter = new AtomicInteger(0);
#Override
public void setLoadBalancer(ILoadBalancer lb) {
this.lb = lb;
}
#Override
public ILoadBalancer getLoadBalancer() {
return lb;
}
#Override
public Server choose(Object key) {
ILoadBalancer lb = getLoadBalancer();
if (lb == null) {
log.warn("no load balancer");
return null;
}
List<Server> reachableServers = lb.getReachableServers();
int upCount = reachableServers.size();
if (upCount == 0) {
log.warn("No up servers available from load balancer: " + lb);
return null;
}
int nextServerIndex = incrementAndGetModulo(upCount);
return reachableServers.get(nextServerIndex);
}
private int incrementAndGetModulo(int modulo) {
for (;;) {
int current = nextServerCyclicCounter.get();
int next = (current + 1) % modulo;
if (nextServerCyclicCounter.compareAndSet(current, next)) {
return next;
}
}
}
}
#Bean
public IRule ribbonRule() {
return new RoundRobinRule();
}
#Bean
public IPing ribbonPing() {
PingUrl ping = new PingUrl(false, "/XXXactive_detect");
ping.setExpectedContent("{\"status\":\"OK\"}");
return ping;
}
#Bean
public ILoadBalancer ribbonLoadBalancer(IClientConfig clientConfig, IRule rule, IPing ping,
ServerList<Server> serverList, ServerListFilter<Server> filter, ServerListUpdater serverListUpdater) {
DynamicServerListLoadBalancer<Server> loadBalancer = new DynamicServerListLoadBalancer<>(clientConfig, rule,
ping, serverList, filter, serverListUpdater);
return loadBalancer;
}
#Bean
public ServerListUpdater ribbonServerListUpdater(IClientConfig clientConfig) {
return new PollingServerListUpdater(clientConfig);
}
}