Why does me use HttpClients.createDefault() as HttpClient singleton instance execute third request always hang - apache-httpclient-4.x

All ,
I create :
public static final HttpClient DEFAULT_HTTPCLIENT = HttpClients
.createDefault();
for(int i=0 ; i<5; i++){
DEFAULT_HTTPCLIENT.execute(requests[i]);
}
But when loop is to i =2 , that means just execute first two request , till third request , the client will hang and seems dead loop .
I refer some materials , I got may be caused by Http Thread Pool configuration limited . But I know what is standard solutions for this issue ? Since I want to send any request any times, but I don't want each time to create new HttpClient . So Do you have any good and standard suggestions for this issue ?
and After I debug this issue , I find it is block on HttpClient below codes : PoolingHttpClientConnectionManager -> leaseConnection ->
entry = future.get(timeout, tunit);
protected HttpClientConnection leaseConnection(
final Future<CPoolEntry> future,
final long timeout,
final TimeUnit tunit) throws InterruptedException, ExecutionException, ConnectionPoolTimeoutException {
final CPoolEntry entry;
try {
entry = future.get(timeout, tunit);
if (entry == null || future.isCancelled()) {
throw new InterruptedException();
}
Asserts.check(entry.getConnection() != null, "Pool entry with no connection");
if (this.log.isDebugEnabled()) {
this.log.debug("Connection leased: " + format(entry) + formatStats(entry.getRoute()));
}
return CPoolProxy.newProxy(entry);
} catch (final TimeoutException ex) {
throw new ConnectionPoolTimeoutException("Timeout waiting for connection from pool");
}
}

That is because your code is leaking connections. By default HttpClient is configured to allow no more than two concurrent connections for the same route, hence it takes only two request executions before the pool is fully exhausted.
http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html#d5e145

Related

How to handle exception in completableFuture in the for loop

I am new to completableFuture, what I am trying to do is I have this logic below in spring boot project which I am trying to convert it to the parallel processing approach using completableFuture.
#Transaction
void serviceMethod{
for(Object obj : objList) //objlist can have 10000 objects and its a multi level composite objects
{
//Get corresponding entity obj from the database, if not found throw user exception
//Process i.e. change some fields
}
}
In the above logic sinnce the method is annotated with #Transaction I am not calling JPA save explicitly to save the entity.
Now, I am trying to do parallel processing with the above logic.
#Transaction
void serviceMethod{
for(Object obj : objList) //objlist can have 10000 objects and its a multi level composite objects
{
//Get corresponding entity obj from the database, if not found throw user exception
CompletableFuture<Optional<Obj>> optionalObjFuture = CompletableFuture.supplyAsync( () -> {//get the object from repository})
CompletableFuture<Obj> objFuture = optionalObjFuture.thenApply( optionalObj -> {
if(obj.isPresent()){
return obj.get();
}
else{
throw user exception;
}
})
////Process i.e. change some fields
}
}
Now the question is
what is the approach I have to follow to break the for loop when there is an exception?
How to handle transaction in this scenario. Is there any way to handle transaction without the need to call saveAll on the processed objects stored in datastructure?
For (1), you could split those 10_000 calls into much smaller batches. For example:
public static void main(String[] args) throws Exception {
int innerIterations = 5;
ExecutorService service = Executors.newFixedThreadPool(innerIterations);
for(int i=0;i<10;++i) {
List<CompletableFuture<String>> list = new ArrayList<>();
int j = 0;
for(;j<innerIterations;++j) {
CompletableFuture<Optional<String>> cf = CompletableFuture.supplyAsync(() -> Optional.of("a"), service);
CompletableFuture<String> cf2 = cf.thenApplyAsync(x -> {
if(x.isPresent()) {
return x.get();
} else {
throw new RuntimeException("test");
}
}, Executors.newFixedThreadPool(2));
list.add(cf2);
}
i += j;
CompletableFuture<Void> all5 = CompletableFuture.allOf(list.toArray(new CompletableFuture[0]));
if(all5.isCompletedExceptionally()) {
// log exception, whatever
break;
} else {
List<String> set = list.stream().map(CompletableFuture::join).collect(Collectors.toList());
System.out.println(set);
}
}
}
This does a couple of things:
splits the initial 10 into two batches of 5.
call CompletableFuture::allOf on those 5 futures. Read the documentation to see that if at least one future fails from those 5, then all5 failed also.
if all5 did not fail (that is all5.isCompletedExceptionally() did not pass), call CompletableFuture::join to get all of the results. There will be no "joining" really, since the previous allOff already waited for them to be completed.
For the second question - you can't. You will need to create a transaction manually, but Spring makes it rather easy.

Vert.x httpclient - 3.5.0 throws exception "Connection was closed" intermittently

I have vert.x app which is consuming api REST over json but intermittently I am seeing exception with reason "Connection was closed". Below are my details -
please share your inputs if anything wrong in the configuration. may be creating scheduler or instantiating httpclient ?
on a different note is it advisable to use same http client to call more than 1 different api's on the same host and port ?
Vert.x Version: 3.5.0
import io.vertx.core.http.HttpClient;
private static Scheduler scheduler =
Schedulers.from(Executors.newFixedThreadPool(8));
// http client instantiated at the time of verticle startup
HttpClient httpclient = vertx.createHttpClient(getHttpClientOptions());
public static HttpClientOptions getHttpClientOptions() {
return new HttpClientOptions()
.setKeepAlive(true)
.setMaxPoolSize(100)
.setPipelining(true)
.setDefaultHost(xxxx.xxxx.com)
.setDefaultPort(8084)
.setSsl(true);
}
// invoke api call
public static Single<Response> invokePOSTServiceAsync(String reqBodyStr, String endpointURI) throws Exception {
try{
return Single.create((SingleEmitter<Response> emitter) -> {
HttpClientRequest request = httpClient.post(endpointURI);
request.putHeader("Content-type","application/json")
request.exceptionHandler(error -> {
LOG.error("ExceptionHandler "+error.getMessage());
emitter.onError(new Throwable(" Failure"));
})
.handler(response -> {
int statusCode = response.statusCode();
if (statusCode == 200) {
response.bodyHandler(body -> {
StringBuilder responseData = new StringBuilder();
responseData.append(body);
emitter.onSuccess(new Response(statusCode,responseData.toString(),"","",null));
});
} else {
emitter.onError(new Throwable(" Failure"));
}
})
.putHeader(HttpHeaders.CONTENT_LENGTH, reqBodyStr.length() + "")
.setTimeout(6000)
.write(reqBodyStr)
.end();
}).subscribeOn(scheduler);
}catch(Exception exe){
exe.printStackTrace();
return null;
}
}
My guess is that this is not related to the client. Either your server is being overloaded, or your network is unreliable. If you're consuming service which doesn't belong to you, you also may get throttled, and that's the reason you're seeing this.
In any case, you need to circumvent those problems, as the network is unreliable anyway. Make your POST requests idempotent and introduce retries.

error when running map reduce in CouchBase Lite

I am trying to use map-reduce on CouchBase Lite. I have documents and they are being channelised. All doucments what I want are coming to Couchbase Lite. But When I Try to run map-reduce on them I am getting the following error
com.couchbase.lite.CouchbaseLiteException: Error when calling map block of view 'calendar', Status: 593 (HTTP 500 Application callback block failed)
Below is my map reduce function
private View createView(Database database){
View calendarView = database.getView("calendar");
calendarView.setMap(new Mapper() {
#Override
public void map(Map<String, Object> document, Emitter emitter) {
emitter.emit((long) document.get("date"),(long) document.get("cost"));
}
},"2");
return calendarView;
}
and Below is the part of main where I am calling the view and querying over it
View calendarView = createView(database);
Query query = database.getView("calendar").createQuery();
query.setStartKey(1472467249448l);
query.setEndKey(1472553649449l);
QueryEnumerator result = null;
try {
result = query.run();
} catch (CouchbaseLiteException e) {
e.printStackTrace();
}
for (Iterator<QueryRow> it = result; it.hasNext(); ) {
QueryRow row = it.next();
Log.d(TAG, row.getValue().toString());
}
As borrrden said in a comment above, Application callback block failed means your map function threw an exception. Use a debugger to find out what it is.
A likely possibility is that one of the documents in your database does not have a date property. In that case your map function would be passing null to the first (key) parameter of emit, which is an invalid argument.

Resubmit web requests on resuming from dormancy

I am using RestSharp (a REST client for .NET) in my Windows Phone 8 app, but I think my question also applies using HttpWebRequest or any other ways of running web requests.
I am trying to find a way to automatically resubmit the requests when app is resumed from dormant state. This is only from dormant and not from the tombstone state.
The idea I had was to create a wrapper object which subscribes to the Deactivated event before starting the request and rerunning the request in case it received the event.
I assume that since the deactivated event was received, the request failed.
public class RestClientEx
{
bool wasDeactivated = false;
public async Task<T> ExecuteTaskAsync<T>(RestClient client, RestRequest request) where T : new()
{
var phoneApplicationService = App.Current.ApplicationLifetimeObjects.OfType<PhoneApplicationService>().First();
phoneApplicationService.Deactivated += phoneApplicationService_Deactivated;
var t = await client.ExecuteTaskAsync<T>(request);
if (this.wasDeactivated)
{
// resubmit request
this.wasDeactivated = false;
t = await this.ExecuteTaskAsync<T>(client, request);
}
return t;
}
void phoneApplicationService_Deactivated(object sender, DeactivatedEventArgs e)
{
(sender as PhoneApplicationService).Deactivated -= phoneApplicationService_Deactivated;
this.wasDeactivated = true;
}
}
My question is, is there another way to achieve this?
Is it OK what I am doing?

What to Return? Error String, Bool with Error String Out, or Void with Exception

I spend most of my time in C# and am trying to figure out which is the best practice for handling an exception and cleanly return an error message from a called method back to the calling method.
For example, here is some ActiveDirectory authentication code. Please imagine this Method as part of a Class (and not just a standalone function.)
bool IsUserAuthenticated(string domain, string user, string pass, out errStr)
{
bool authentic = false;
try
{
// Instantiate Directory Entry object
DirectoryEntry entry = new DirectoryEntry("LDAP://" + domain, user, pass);
// Force connection over network to authenticate
object nativeObject = entry.NativeObject;
// No exception thrown? We must be good, then.
authentic = true;
}
catch (Exception e) { errStr = e.Message().ToString(); }
return authentic;
}
The advantages of doing it this way are a clear YES or NO that you can embed right in your If-Then-Else statement. The downside is that it also requires the person using the method to supply a string to get the Error back (if any.)
I guess I could overload this method with the same parameters minus the "out errStr", but ignoring the error seems like a bad idea since there can be many reasons for such a failure...
Alternatively, I could write a method that returns an Error String (instead of using "out errStr") in which a returned empty string means that the user authenticated fine.
string AuthenticateUser(string domain, string user, string pass)
{
string errStr = "";
try
{
// Instantiate Directory Entry object
DirectoryEntry entry = new DirectoryEntry("LDAP://" + domain, user, pass);
// Force connection over network to authenticate
object nativeObject = entry.NativeObject;
}
catch (Exception e) { errStr = e.Message().ToString(); }
return errStr;
}
But this seems like a "weak" way of doing things.
Or should I just make my method "void" and just not handle the exception so that it gets passed back to the calling function?
void AuthenticateUser(string domain, string user, string pass)
{
// Instantiate Directory Entry object
DirectoryEntry entry = new DirectoryEntry("LDAP://" + domain, user, pass);
// Force connection over network to authenticate
object nativeObject = entry.NativeObject;
}
This seems the most sane to me (for some reason). Yet at the same time, the only real advantage of wrapping those 2 lines over just typing those 2 lines everywhere I need to authenticate is that I don't need to include the "LDAP://" string. The downside with this way of doing it is that the user has to put this method in a try-catch block.
Thoughts?
Is there another way of doing this that I'm not thinking of?
There is no "one size fits all". If you return a flag, that makes it easy to use a method in if() and loops. Exceptions always need a lot of boiler plate code. If you just want a string which you can display to the user (say, in a web UI), returning the error string (or null for "no error") is good, too.
But most of the time, I throw an exception (and in Java a subclass of RuntimeException) because that allows me to return more than a single information about the error (like: Which file caused the error? Which line/column? What was I doing? Which field in a form should be marked as illegal? etc).
In your case, you can't handle the exception in your method, so you shouldn't catch it. Only catch it when you can do something about it.
In this example, I agree, you should let the exception flow through to the consumer. However, as an alternative to the approaches you highlighted, consider this approach.
You can use a response object to hold information coming out of a method run, for example:
public abstract class BaseResponse
{
public bool IsOk { get; protected set;}
public string Message { get; protected set; }
}
public class AuthenticationResponse: BaseResponse
{
public AuthenticationResponse(bool isOk): this(isOk, "") {}
public AuthenticationResponse(bool isOk, string message)
{
IsOk = isOk;
Message = message;
}
}
AuthenticationResponse IsUserAuthenticated(string domain, string user, string pass)
{
bool authentic = false;
string errStr;
try
{
// Instantiate Directory Entry object
DirectoryEntry entry = new DirectoryEntry("LDAP://" + domain, user, pass);
// Force connection over network to authenticate
object nativeObject = entry.NativeObject;
// No exception thrown? We must be good, then.
authentic = true;
}
catch (Exception e) { errStr = e.Message().ToString(); }
return new AuthenticationResponse(authentic, errStr);
}
Then to use it in your if statements:
AuthenticationResponse response;
if((response = IsUserAuthenticated("domain", "user", "pass")).IsOk)
{
// do successful activity
} else {
Console.WriteLine(response.Message)
}
The trick is the return value of an assignment operation is the value that was assigned. So, we can do the assignment and the valid check in the same line. If you didn't need to hold onto the result of the call, you could simply call the method and check the IsOk property.
if(IsUserAuthenticated("domain", "user", "pass").IsOk)
{
// do successful activity
}
Then you can build up your custom response object to return any combination of values from your method as you need.
Don't handle the exception or return a message of any kind. Let the consumer of your method take care of this.