Apache HttpClient 4.3.x has no default host? - apache-httpclient-4.x

HttpClient 4.3.x issue.
There does not seem to be a way to attach a default host on CloseableHttpClient for 4.3.x.
This is frustrating as it requires all of your request builders to know up front all the host info, rather than just building up the request parts specific to the call and letting the client fill in any left out defaults (eg. like a default host, port, etc).
With 4.2.x and previous, you could set the default host on the client and any request just needs a subpath + parameters.
But with 4.3.x you have confusing layers of setRoutePlanner(x) (which could have proxy settings) and setProxy(x) (which could be overridden by route-planner) and I'm confused how they settle with the actual client instance. And debugging it shows that route-planner will not get used for default_host, and the 4.3.2 version actually expects the deprecated ClientPNames.DEFAULT_HOST to be set (for case with null target host) which is maybe a defect.
I am finding apache httpclient to be going off a deep edge with all these changes.
Also the examples do not fully clarify http client usage unfortunately.
As an aside: the new design is such mud, why not just have setDefaultHost(x) ? and clear up the confusion on proxy layering(s).
Unless I'm missing something, how does one set the default host in http client 4.3.x?
Why do you think they changed and decided to make everything up front in the request objects vs. defaults in the client?

This how one can provide a default target host using a custom route planner
HttpRoutePlanner routePlanner = new DefaultRoutePlanner(DefaultSchemePortResolver.INSTANCE) {
#Override
public HttpRoute determineRoute(
final HttpHost target,
final HttpRequest request,
final HttpContext context) throws HttpException {
return super.determineRoute(
target != null ? target : new HttpHost("some.default.host", 80),
request, context);
}
};
CloseableHttpClient client = HttpClients.custom()
.setRoutePlanner(routePlanner)
.build();

Related

Chrome remote debugging in a seleniumgrid

Im running a selenium-grid with several chrome instances. The selenium grid are 2 machines(windows) with several nodes. The tests are executed from another machine which connects to the grid. To be able to use the features of remote debugging, i need to connect from the executing machine(which can read the sessions host and the drivers debugUrl) to the other machines and finally the chrome instances.
But chrome rejects anything else than localhost.
I can only find solutions, where people tunnel or port forwarding, which is perhaps ok, when there is only a single instance. In a grid i don't have static ports or static rules to provide static forwarding.
In my scenario the grid is build up automated and not an ever running system.
Has anybody a hint how to solve this?
Since i found a solution by myself i want to share. I will only post parts of code to give the hints and not the full code since its to much work here, but for an experienced developer this should be enough.
To be able to address the right browser and access its remote-debug websocket i implemented a custom servlet for my nodes.
First the servlet:
public class DebugServlet extends RegistryBasedServlet
being registered through the node.json like
"servlets" :["com.....ui.util.DebugServlet"],
To access the node(on the right machine) i ask the selenium session for it like:
"http://" + hubHost + ":" + hubPort + "/grid/api/testsession?session=" + sessionId
where the "sessionid" can be retrieved from chromedriver.
From the returned json we can extract the node info of the session, here we need the url.
url = JSONUtil.get(response.getBody(), "proxyId")
No we can call the servlet of the correct host and give in the websocket url for the browser and whatever data is needed. In my example to add a default network-header for BasicAuth.
url+ "/extra/DebugServlet"
with the header in java(can also be parameters or other http provided possibilities)
new BasicHeader("BrowserUrl", webSocketDebuggerUrl), new BasicHeader("Name", name),
new BasicHeader("Value", value)
In the servlet we extract now the data and open a websocket to the browser with the given url and make our calls.
In the servlet:
public static final String networkDebugging = "{\"id\": 1,\"method\": \"Network.enable\",\"params\": {\"maxTotalBufferSize\": 10000000,\"maxResourceBufferSize\": 5000000 }}";
public static final String addHeader = "{\"id\": 2,\"method\": \"Network.setExtraHTTPHeaders\",\"params\": { \"headers\": {\"${key}\": \"${value}\"}}}";
ws.connect();
ws.setAutoFlush(true);
ws.sendText(networkDebugging);
String payload = TemplateUtil.replace(addHeader, name, value);
ws.sendText(payload);

Single HttpClientContext for multiple threads

I am using apache HttpClient 4.5 to process http request in java.
According to documentation HttpClient is thread safe so we can use same instance of HttpClient for all the threads but HttpContext should be maintain by each thread of execution.
For authentication (NTLM authentication) we need to set CredentialsProvider to the context, which will authenticate on the server.
Requirement
All the request will hit the same server with same authentication details. I want to authenticate only once when application will initialise or first request to the server, all other request should serve in same session but can be from different threads.
Can I use same context because hitting to the same server with same authentication details, or there is another way to achieve it?
Even though HttpContext instances ought not be shared between threads, there is nothing wrong with sharing thread-safe objects between multiple contexts. For instance, one can easily use the same CredentialsProvider and AuthCache instances with multiple concurrent contexts.
// External dependencies
CloseableHttpClient client;
CredentialsProvider credentialsProvider;
AuthCache authCache;
CookieStore cookieStore;
Principal userPrincipal;
// request execution
HttpClientContext context = HttpClientContext.create();
context.setCredentialsProvider(credentialsProvider);
context.setAuthCache(authCache);
context.setCookieStore(cookieStore);
context.setUserToken(userPrincipal);
HttpGet httpGet = new HttpGet("http://targethost/");
try (CloseableHttpResponse response1 = client.execute(httpGet, context)) {
System.out.println(response1.getStatusLine());
EntityUtils.consume(response1.getEntity());
}
VERY IMPORTANT: NTLM connections are stateful and can be re-used between contexts only if associated with the same user identity. One can either turn off connection state tracking when wiring up HttpClient instance (as below) or manually set up user identity in the execution context (as above).
CloseableHttpClient client = HttpClientBuilder.create()
.disableConnectionState()
.build();

why DeferredResult ends on setResult() on trying to use SSE

i am trying to implement a Server Sent Events (SSE) webpage which is powered by Spring. My test code does the following:
Browser uses EventSource(url) to connect to server. Spring accepts the request with the following controller code:
#RequestMapping(value="myurl", method = RequestMethod.GET, produces = "text/event-stream")
#ResponseBody
public DeferredResult<String> subscribe() throws Exception {
final DeferredResult<String> deferredResult = new DeferredResult<>();
resultList.add(deferredResult);
deferredResult.onCompletion(() -> {
logTimer.info("deferedResult "+deferredResult+" completion");
resultList.remove(deferredResult);
});
return deferredResult;
}
So mainly it puts the DeferredResult in a List and register a completion callback so that i can remove this thing from the List in case of completion.
Now i have a timer method, that will periodically output current timestamp to all registered "browser" via their DeferredResults.
#Scheduled(fixedRate=10000)
public void processQueues() {
Date d = new Date();
log.info("outputting to "+ LoginController.resultList.size()+ " connections");
LoginController.resultList.forEach(deferredResult -> deferredResult.setResult("data: "+d.getTime()+"\n\n"));
}
The data is sent to the browser and the following client code works:
var source = new EventSource('/myurl');
source.addEventListener('message', function (e) {
console.log(e.data);
$("#content").append(e.data).append("<br>");
});
Now the problem:
The completion callback on the DeferredResult is called on every setResult() call in the timer thread. So for some reason the connection is closed after the setResult() call. SSE in the browser reconnects as per spec and then same thing again. So on client side i have a polling behavior, but i want an kept open request where i can push data on the same DeferredResult over and over again.
Do i miss something here? Is DeferredResult not capable of sending multiple results? i put in a 10 seconds delay in the timer thread to see if the request only terminates after setResult(). So in the browser the request is kept open until the timer pushes the data but then its closed.
Thanks for any hint on that. One more note: I added async-supported to all filters/servlets in tomcat.
Indeed DeferredResult can be set only once (notice that setResult returns a boolean). It completes processing with the full range of Spring MVC processing options, i.e. meaning that all you know about what happens during a Spring MVC request remains more or less the same, except for the asynchronously produced return value.
What you need for SSE is something more focused, i.e. write each value to the response using an HttpMessageConverter. I've created a ticket for that https://jira.spring.io/browse/SPR-12212.
Note that Spring's SockJS support does have an SSE transport which takes care of a few extras such as cross-domain requests with cookies (important for IE). It's also used on top of a WebSocket API and WebSocket-style messaging (even if WebSocket is not available on either the client or the server side) which fully abstracts the details of HTTP long polling.
As a workaround you can also write directly to the Servlet response using an HttpMessageConverter.

How to fix cross-site origin policy for server and web-site

I'm using Dropwizard, which I'm hosting, along with a website, on the google cloud (GCE). This means that there are 2 locations currently active:
Some.IP.Address - UI
Some.IP.Address:8080 - Dropwizard server
When the UI tries to call anything from my dropwizard server, I get cross-site origin errors, which is understandable. However, this is posing a problem for me. How do I fix this? It would be great if I could somehow spoof the addresses so that I don't have to fully qualify the resource in the UI.
What I'm looking to do is this:
$.get('/provider/upload/display_information')
Or, if I have to fully qualify
$.get('http://Some.IP.Address:8080/provider/upload/display_information')
I tried setting Origin Filters in Dropwizard per this google groups thread (https://groups.google.com/forum/#!topic/dropwizard-user/ybDOTOxjlLI), but it doesn't seem to work.
In index.html that is served by the server at http://Some.IP.Address you might have a jQuery script that look as follows.
$.get('http://Some.IP.Address:8080/provider/upload/display_information', data, callback);
Of course your browser will not allow accessing http://Some.IP.Address:8080 due to the Same-Origin-Policy (SOP). The protocol (http, https) and the host as well as the port have to be the same.
To achieve Cross-Origin Resource Sharing (CORS) on Dropwizard, you have to add a CrossOriginFilter to the servlet environment. This filter will add some Access-Control-Headers to every response the server is sending. In the run method of your Dropwizard application write:
import org.eclipse.jetty.servlets.CrossOriginFilter;
public class SomeApplication extends Application<SomeConfiguration> {
#Override
public void run(TodoConfiguration config, Environment environment) throws Exception {
FilterRegistration.Dynamic filter = environment.servlets().addFilter("CORS", CrossOriginFilter.class);
filter.addMappingForUrlPatterns(EnumSet.allOf(DispatcherType.class), true, "/*");
filter.setInitParameter("allowedOrigins", "http://Some.IP.Address"); // allowed origins comma separated
filter.setInitParameter("allowedHeaders", "Content-Type,Authorization,X-Requested-With,Content-Length,Accept,Origin");
filter.setInitParameter("allowedMethods", "GET,PUT,POST,DELETE,OPTIONS");
filter.setInitParameter("preflightMaxAge", "5184000"); // 2 months
filter.setInitParameter("allowCredentials", "true");
// ...
}
// ...
}
This solution works for Dropwizard 0.7.0 and can be found on https://groups.google.com/d/msg/dropwizard-user/xl5dc_i8V24/gbspHyl4y5QJ.
This filter will add some Access-Control-Headers to every response. Have a look on http://www.eclipse.org/jetty/documentation/current/cross-origin-filter.html for a detailed description of the initialisation parameters of the CrossOriginFilter.

WebAPI and HTML5 SSE

was trying to encapsulate a partial view to show feedback that i can push back to the client.
This Article shows a method of pushing back data using HTML5 Server-Sent events (SSE).
I noticed that if i opened up several browser tabs and then closed one i got exceptions as the logic didn't remove the respective stream from the ConcurrentQueue. I amended the code as below
private static void TimerCallback(object state)
{
StreamWriter data;
Random randNum = new Random();
// foreach (var data in _streammessage)
for (int x = 0; x < _streammessage.Count; x++)
{
_streammessage.TryDequeue(out data);
data.WriteLine("data:" + randNum.Next(30, 100) + "\n");
try
{
data.Flush();
_streammessage.Enqueue(data);
}
catch (Exception ex)
{
// dont re-add the stream as an error ocurred presumable the client has lost connection
}
}
//To set timer with random interval
_timer.Value.Change(TimeSpan.FromMilliseconds(randNum.Next(1, 3) * 500), TimeSpan.FromMilliseconds(-1));
}
I also had to amend the OnStreamAvailable member as the framework syntax had changed to the second parameter being a HttpContent rather than HttpContentHeaders
public static void OnStreamAvailable(Stream stream, HttpContent headers, TransportContext context)
The problem now is i am still getting inconsistant behaviour if i add or remove clients i.e it times out when trying to initialise a new client. Does anyone have any ideas or more examples of using SSE with WinAPI and the correct "framework of methods" to handle disconnected clients
Cheers
Tim
This article is actually an adaptation of my original article from May - http://www.strathweb.com/2012/05/native-html5-push-notifications-with-asp-net-web-api-and-knockout-js/ (notice even variable names and port numbers are the same :-).
It is a very valid point that you are raising, and detecting a broken connection is something that's not very easy with this setup. The main reason is that while ASP.NET (the host) allows you to check a broken connection, there is no notification mechanism between ASP.NET (host) and Web API informing about that.
That is why in order to detect a broken connection (disconnected client) you should really try writing to the stream, and catch any error - this would mean the client has been disconnected.
I asked the same question to Brad Wilson/Marcin Dobosz/Damien Edwards at aspconf, and Damien suggested using HttpContext.Current.Response.IsClientConnected - so basically bypassing Web API and obtaining the connectivity info from the underlying host directly (however there is still a race condition involved anyway). That is really .NET 4. He also pointed an interesting way in which this problem could be avoided in .NET 4.5 using an async cancellation token. Frankly, I have never got around to test it, but perhaps this is something you should explore.
You can see their response to this problem in this video - http://channel9.msdn.com/Events/aspConf/aspConf/Ask-The-Experts - fast forward to 48:00