I'm trying to determine the proper way to measure the full time taken to obtain the full response from the HTTP request. From the time the HTTP GET is sent, to the time the client gets the very last entity byte from the server.
What would be the best way to do this?
For instance, in this modified version of code borrowed from the httpclient tutorial. Would the following give me the time it took to get the entire MP4 from the server?
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet("http://server_ip/video.mp4");
//Start the timer here
long start_time = System.currentTimeMillis();
CloseableHttpResponse response = httpclient.execute(httpget);
try {
HttpEntity entity = response.getEntity();
if (entity != null) {
InputStream instream = entity.getContent();
try {
final byte[] lBytes = new byte[4096];
do{
//Consume 4KB from the entity inputStream
instream.read(lBytes);
} while(instream.read(lBytes) != -1)
//Stop the timer here
long stop_time = System.currentTimeMillis();
//Get total download time?
long mp4DownloadTime = stop_time - start_time;
}
finally {
instream.close();
}
}
}
finally {
response.close();
}
Does that make any sense? I don't quite understand how the HttpClient deals with the entity content. When running a tcpdump on the interface, I see that I get the full entity regardless of what I consume in the Java code... Any guidance would be appreciated.
Related
I want to read JSON response coming from jenkins API to read last build details. I am using http://jenkins_server/job/job_name/lastBuild/api/json. When I type this URL in browser, I need to sign in to my Jenkins job and after that I get proper json response.
I have written a java code to read JSON response from same Jenkins API. But I get "Server returned HTTP response code: 403" as I have not handled the authentication part in code.
public class GetJSONResponse {
public static void main(String[] args) throws MalformedURLException, IOException, JSONException {
InputStream is = new URL("http://jenkins_server/job/job_name/lastBuild/api/json").openStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is, Charset.forName("UTF-8")));
StringBuilder sb = new StringBuilder();
int cp;
while ((cp = rd.read()) != -1) {
sb.append((char) cp);
}
JSONObject json = new JSONObject(sb.toString());
System.out.println(json.toString());
}
}
I searched alot on how to get JSON response from jenkins API which reqiures authentication , but didn't find anything useful. How do I add authentication part in my code? Can anybody please help me with this?
Thanks in advance.
I resolved my problem after changing the jenkins config.xml "useSecurity" fileld.
Change <useSecurity>true</useSecurity> to <useSecurity>false</useSecurity>
And I got correct JSON response.
I have vert.x app which is consuming api REST over json but intermittently I am seeing exception with reason "Connection was closed". Below are my details -
please share your inputs if anything wrong in the configuration. may be creating scheduler or instantiating httpclient ?
on a different note is it advisable to use same http client to call more than 1 different api's on the same host and port ?
Vert.x Version: 3.5.0
import io.vertx.core.http.HttpClient;
private static Scheduler scheduler =
Schedulers.from(Executors.newFixedThreadPool(8));
// http client instantiated at the time of verticle startup
HttpClient httpclient = vertx.createHttpClient(getHttpClientOptions());
public static HttpClientOptions getHttpClientOptions() {
return new HttpClientOptions()
.setKeepAlive(true)
.setMaxPoolSize(100)
.setPipelining(true)
.setDefaultHost(xxxx.xxxx.com)
.setDefaultPort(8084)
.setSsl(true);
}
// invoke api call
public static Single<Response> invokePOSTServiceAsync(String reqBodyStr, String endpointURI) throws Exception {
try{
return Single.create((SingleEmitter<Response> emitter) -> {
HttpClientRequest request = httpClient.post(endpointURI);
request.putHeader("Content-type","application/json")
request.exceptionHandler(error -> {
LOG.error("ExceptionHandler "+error.getMessage());
emitter.onError(new Throwable(" Failure"));
})
.handler(response -> {
int statusCode = response.statusCode();
if (statusCode == 200) {
response.bodyHandler(body -> {
StringBuilder responseData = new StringBuilder();
responseData.append(body);
emitter.onSuccess(new Response(statusCode,responseData.toString(),"","",null));
});
} else {
emitter.onError(new Throwable(" Failure"));
}
})
.putHeader(HttpHeaders.CONTENT_LENGTH, reqBodyStr.length() + "")
.setTimeout(6000)
.write(reqBodyStr)
.end();
}).subscribeOn(scheduler);
}catch(Exception exe){
exe.printStackTrace();
return null;
}
}
My guess is that this is not related to the client. Either your server is being overloaded, or your network is unreliable. If you're consuming service which doesn't belong to you, you also may get throttled, and that's the reason you're seeing this.
In any case, you need to circumvent those problems, as the network is unreliable anyway. Make your POST requests idempotent and introduce retries.
All ,
I create :
public static final HttpClient DEFAULT_HTTPCLIENT = HttpClients
.createDefault();
for(int i=0 ; i<5; i++){
DEFAULT_HTTPCLIENT.execute(requests[i]);
}
But when loop is to i =2 , that means just execute first two request , till third request , the client will hang and seems dead loop .
I refer some materials , I got may be caused by Http Thread Pool configuration limited . But I know what is standard solutions for this issue ? Since I want to send any request any times, but I don't want each time to create new HttpClient . So Do you have any good and standard suggestions for this issue ?
and After I debug this issue , I find it is block on HttpClient below codes : PoolingHttpClientConnectionManager -> leaseConnection ->
entry = future.get(timeout, tunit);
protected HttpClientConnection leaseConnection(
final Future<CPoolEntry> future,
final long timeout,
final TimeUnit tunit) throws InterruptedException, ExecutionException, ConnectionPoolTimeoutException {
final CPoolEntry entry;
try {
entry = future.get(timeout, tunit);
if (entry == null || future.isCancelled()) {
throw new InterruptedException();
}
Asserts.check(entry.getConnection() != null, "Pool entry with no connection");
if (this.log.isDebugEnabled()) {
this.log.debug("Connection leased: " + format(entry) + formatStats(entry.getRoute()));
}
return CPoolProxy.newProxy(entry);
} catch (final TimeoutException ex) {
throw new ConnectionPoolTimeoutException("Timeout waiting for connection from pool");
}
}
That is because your code is leaking connections. By default HttpClient is configured to allow no more than two concurrent connections for the same route, hence it takes only two request executions before the pool is fully exhausted.
http://hc.apache.org/httpcomponents-client-ga/tutorial/html/fundamentals.html#d5e145
I'm using apache httpcompnonents library for httpclient. I want to use it in a multithreaded application where number of threads are going to be really high and there would be frequent http calls. This is the code I'm using to read the response after execute call.
HttpEntity entity = httpResponse.getEntity();
String response = EntityUtils.toString(entity);
I just want to confirm that is it the most efficient way of reading the response?
Thanks,
Hemant
This in fact represents the most inefficient way of processing an HTTP response.
You most likely want to digest the content of the response into a domain object of a sort. So, what is the point of buffering it in-memory in a form of a string?
The recommended way to deal with response processing is by using a custom ResponseHandler that can process the content by streaming it directly from the underlying connection. The added benefit of using a ResponseHandler is that it completely relieves from dealing with connection release and resource deallocation.
EDIT: modified the sample code to use JSON
Here's an example of it using HttpClient 4.2 and Jackson JSON processor. Stuff is assumed to be your domain object with JSON bindings.
ResponseHandler<Stuff> rh = new ResponseHandler<Stuff>() {
#Override
public Stuff handleResponse(
final HttpResponse response) throws IOException {
StatusLine statusLine = response.getStatusLine();
if (statusLine.getStatusCode() >= 300) {
throw new HttpResponseException(
statusLine.getStatusCode(),
statusLine.getReasonPhrase());
}
HttpEntity entity = response.getEntity();
if (entity == null) {
throw new ClientProtocolException("Response contains no content");
}
JsonFactory jsonf = new JsonFactory();
InputStream instream = entity.getContent();
// try - finally is not strictly necessary here
// but is a good practice
try {
JsonParser jsonParser = jsonf.createParser(instream);
// Use the parser to deserialize the object from the content stream
return stuff;
} finally {
instream.close();
}
}
};
DefaultHttpClient client = new DefaultHttpClient();
Stuff mystuff = client.execute(new HttpGet("http://somehost/stuff"), rh);
I have a SSIS package in which i use a WebService task to call a Spring WS.
The authentication is done by client certificate and username & password.
I have tried to do it like this a simple HttpConnection and a WebService task - Error 504 Gateway Timeout. When i edit the HttpConnection and click on Test Connection i get an error that states:
"The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel."
I have tried doing it with a script task and the same error.
I have even tried with a dummy console application and the same result.
I also have a java written app that actually does the job but i do not have access to it's code-behind. This basically proves that the problem is not from the server itself.
The java application has it's own keystore and the same certificates that i have installed on the server.
I opened a wireshark capture and i saw that when i used either of my apps the host made a DNS request for an address that i did not configure anywhere(it seems like a proxy address from the intranet), while the java app made a DNS request with the correct address.
I am stuck here, and i have no idea what the problem might be or what else i can do so that i would get a proper error.
Please advise!
Edit:
This is the code that calls the WS:
public static void CallWebService()
{
var _url = "https://<IP>/App/soap/DataService";
string action = "getData";
Dictionary<string, string> parameters = new Dictionary<string, string>();
parameters.Add("param1", "0");
parameters.Add("param2", "0");
parameters.Add("param3", "value");
XmlDocument soapEnvelopeXml = CreateSoapEnvelope(action, parameters);
HttpWebRequest webRequest = CreateWebRequest(_url);
InsertSoapEnvelopeIntoWebRequest(soapEnvelopeXml, webRequest);
// begin async call to web request.
IAsyncResult asyncResult = webRequest.BeginGetResponse(null, null);
// suspend this thread until call is complete. You might want to
// do something usefull here like update your UI.
asyncResult.AsyncWaitHandle.WaitOne();
// get the response from the completed web request.
string soapResult;
using (WebResponse webResponse = webRequest.EndGetResponse(asyncResult))
{
using (StreamReader rd = new StreamReader(webResponse.GetResponseStream()))
{
soapResult = rd.ReadToEnd();
}
}
Console.WriteLine(soapResult);
}
private static HttpWebRequest CreateWebRequest(string url)
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.ContentType = "text/xml;charset=\"utf-8\"";
webRequest.Accept = "text/xml";
webRequest.Method = "POST";
string thumbprint = "CERTIFICATE THUMBPRINT";
byte[] thumbprintArray = new byte[thumbprint.Split(new char[]{ ' ' }).Length];
string[] stringArray = thumbprint.Split(new char[] { ' ' });
for (int i = 0; i < thumbprintArray.Length; i++)
{
thumbprintArray[i] = Convert.ToByte(stringArray[i], 16);
}
X509Store localStore = new X509Store("My");
localStore.Open(OpenFlags.ReadOnly);
X509Certificate2Collection certCol = localStore.Certificates.Find(X509FindType.FindByTimeValid, DateTime.Now, true);
foreach (X509Certificate cert in certCol)
{
if (cert.GetCertHashString() == thumbprint)
{
webRequest.ClientCertificates.Add(cert);
break;
}
}
webRequest.UseDefaultCredentials = false;
webRequest.Credentials = new NetworkCredential("USER", "PASSWORD");
return webRequest;
}
private static XmlDocument CreateSoapEnvelope(string action, Dictionary<string, string> parameters)
{
string formatedParameters = string.Empty;
string paramFormat = "<{0}>{1}</{0}>";
foreach (string key in parameters.Keys)
{
formatedParameters += string.Format(paramFormat, key, parameters[key]);
}
XmlDocument soapEnvelop = new XmlDocument();
soapEnvelop.LoadXml(string.Format(#"
<soapenv:Envelope xmlns:soap=""http://custom/soap/"" xmlns:soapenv=""http://schemas.xmlsoap.org/soap/envelope/"">
<soapenv:Header/>
<soapenv:Body>
<soap:{0}>
{1}
</soap:{0}>
</soapenv:Body>
</soapenv:Envelope>", action, formatedParameters));
return soapEnvelop;
}
private static void InsertSoapEnvelopeIntoWebRequest(XmlDocument soapEnvelopeXml, HttpWebRequest webRequest)
{
using (Stream stream = webRequest.GetRequestStream())
{
soapEnvelopeXml.Save(stream);
}
}