I have a routine which checks if a record has been indexed by Solr. I have a deprecated method of creating a HTTPClient which I'm trying to remove:
From
DefaultHttpClient httpClient = new DefaultHttpClient();
To
CloseableHttpClient httpClient = HttpClientBuilder.create().build();
The problem I now have is that after 2 call to the URL, the 3rd attempt seems to hang. I'm not quite sure what I'm missing if anyone can help please?
This is my complete method which I've extracted out into a test:
#Test
public void checkUntilRecordAvailable() {
String output;
String solrSingleJobURL = "http://solr01.prod.efinancialcareers.com:8080/solr/jobSearchCollection/select?q=id%3A7618769%0A&fl=*&wt=json&indent=true";
CloseableHttpClient httpClient = HttpClientBuilder.create().build();
HttpGet httpGet = new HttpGet(solrSingleJobURL);
StringBuilder jobResponseBuilder;
Gson gson = new Gson();
while (true) {
System.out.print("WAITING FOR SOLR PARTIAL TO RUN " + solrSingleJobURL);
jobResponseBuilder = new StringBuilder();
try {
HttpResponse response = httpClient.execute(httpGet);
BufferedReader br = new BufferedReader(new InputStreamReader((response.getEntity().getContent())));
while ((output = br.readLine()) != null) {
System.out.println(output);
jobResponseBuilder.append(output);
}
JobResponse jobResponse = gson.fromJson(jobResponseBuilder.toString(), JobResponse.class);
Long numberOfRecordsFound = jobResponse.getNumberOfRecordsFound();
if (numberOfRecordsFound == 0) {
System.out.println("- PAUSE FOR 10 SECONDS UNTIL NEXT CHECK");
Thread.sleep(5000);
} else {
System.out.println(" RECORD FOUND ");
httpClient.close();
break;
}
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
here is some code using the builders from 4.3 httpclient.. dont know if it helps .. I use skeleton from here. So , i wrap the creation of the httpclient in a runnable and post it to a que-processor for the EXEC. Note the runnable has your 'builder' stuff in it.
RequestConfig config = null;
private HttpClientContext context;
public void create(int method, final String url, final String data) {
this.method = method;// GET, POST, HEAD, DELETE etc
this.url = url;
this.data = data; //entity body of POST
this.config = RequestConfig.custom()
.setConnectionRequestTimeout(60 * 1000)
.setSocketTimeout(60 * 1000)
.build();
this.context = HttpClientContext.create();
ConnectionMgr.getInstance().push(this);
}
//above creates a runnable that can be posted to a generic execution que
//detls on run() include builder asked about
public void run() {
handler.sendMessage(Message.obtain(handler, HttpConnection.DID_START));
CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(YourConnectionMgr.getInstance())
.addInterceptorLast(new HttpRequestInterceptor() {
public void process(
final HttpRequest request,
final HttpContext context) throws HttpException, IOException {
if (request.getRequestLine().getMethod() == "POST"){
request.addHeader("Content-Type", mimeType) ;}
}else if(request.getRequestLine().getMethod() == "GET" && !request.getRequestLine().getUri().toString().contains("ffmpeg")){
request.addHeader("X-Parse-Application-Id", ParseApplication.key_appId);
}
}) .build();
Related
I have this code,
ClassPathResource classPathResource = new ClassPathResource("json/data.json");
try {
byte[] binaryData = FileCopyUtils.copyToByteArray(classPathResource.getInputStream());
strJson = new String(binaryData, StandardCharsets.UTF_8);
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(strJson); //works fine here
return strJson; //return it doesn't display pretty on browser
Any idea how to fix this? I've been trying all the solution here on the internet and especially stackoverflow and none of it works.
If you want clear view, it's from my previous code
I use thymeleaf html again,
#Controller
#RequestMapping("/menu")
public class DataController {
// load json
private List<DataModel> theDatawiz;
private String strJson = null;
#PostConstruct
private void loadData() {
// load json
ClassPathResource classPathResource = new ClassPathResource("json/data.json");
try {
byte[] binaryData = FileCopyUtils.copyToByteArray(classPathResource.getInputStream());
strJson = new String(binaryData, StandardCharsets.UTF_8);
} catch (IOException e) {
e.printStackTrace();
}
// setup array mapper
ObjectMapper objectMapper = new ObjectMapper();
DataModel[] datawiz = null;
try {
datawiz = objectMapper.readValue(strJson, DataModel[].class);
} catch (Exception e) {
e.printStackTrace();
}
// create the list
theDatawiz = new ArrayList<>();
for(int i = 0; i < datawiz.length; i++) {
DataModel dat = new DataModel(datawiz[i].getId(),datawiz[i].getName());
theDatawiz.add(dat);
}
}
// add mapping for "/list"
#GetMapping("/list")
public String listMenu(Model theModel) {
// add to the spring model
theModel.addAttribute("thelist", theDatawiz);
return "menu-list";
}
// add mapping for "/list"
#GetMapping("/jason")
public String printJson(Model theModel) {
// add to the spring model
theModel.addAttribute("result", strJson);
return "jason";
}
}
On the jason.html,
<p th:text="'JSON: ' + ${result}" style="white-space: pre"></p>
```protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main3);
ctx=getApplicationContext();
txtString= (TextView)findViewById(R.id.txtString);
httpClient = new OkHttpClient();
try {
sendGETT();
}
catch (Exception e)
{
e.printStackTrace();
}
}
protected void sendGETT() throws IOException {
httpClient = new OkHttpClient();
OkHttpClient client = new OkHttpClient();
Request request = new Request.Builder()
.url("https://devru-gaana-v1.p.rapidapi.com/featuredAlbums.php")
.get()
.addHeader("x-rapidapi-host", "devru-gaana-v1.p.rapidapi.com")
.addHeader("x-rapidapi-key", "my api key")
.build();
if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.KITKAT) {
httpClient.newCall(request).enqueue(new Callback() {
#Override
public void onFailure(Call call, IOException e) {
e.printStackTrace();
}
#Override
public void onResponse(Call call, Response response) throws IOException {
try (ResponseBody responseBody = response.body()) {
if (!response.isSuccessful())
throw new IOException("Unexpected code " + response.body().string());
Headers responseHeaders = response.headers();
for (int i = 0, size = responseHeaders.size(); i < size; i++) {
System.out.println(responseHeaders.name(i) + ": " +
responseHeaders.value(i));
Main3Activity.txtString.setText(response.header("Server"));
}
Gson gson = new GsonBuilder().setPrettyPrinting().create();
JsonParser jp = new JsonParser();
JsonElement je = jp.parse(response.body().charStream());
final String prettyJsonString = gson.toJson(je);
runOnUiThread(new Runnable() {
#Override
public void run() {
txtString.setText(prettyJsonString);
}
});
}
}
});
}
}```
I'm trying to use okhttpclient with okhttp3, but it return a null value.i tried another url with headers which work fine but when i try this it gives null respone.I tried many solutions from net but I can't figured this out.hope for the help.thanks
This code works fine, for example,
for
Response response = client.newCall(request).execute();
Request request = new Request.Builder()
.url("https://httpbin.org/get")
.addHeader("custom-key", "mkyong") // add request headers
.addHeader("User-Agent", "OkHttp Bot")
.build();
or any other website but I want to get the content of website using rapid api with add headers
```Request request = new Request.Builder()
.url("https://devru-gaana-v1.p.rapidapi.com/featuredAlbums.php")
.get()
.addHeader("x-rapidapi-host", "devru-gaana-v1.p.rapidapi.com")
.addHeader("x-rapidapi-key", "mine api for site")
.build();```
I'm using Apache HTTPClient 4.5.3 to make some HTTP requests, but I am getting a g-zipped response I have tried many things I found online but non of them worked. I still get gibberish when I print the response. Below are the relevant code. What do I need to do to get a human readable response?
static public CloseableHttpClient CreateHttpClient() {
// return
// HttpClients.custom().disableAutomaticRetries().setHttpProcessor(HttpProcessorBuilder.create().build())
// .build();
return HttpClientBuilder.create().disableAutomaticRetries()
.setHttpProcessor(HttpProcessorBuilder.create().build()).build();
}
static public RequestConfig GetConfig() {
return RequestConfig.custom().setSocketTimeout(READTIMEOUT).setConnectTimeout(CONNECTTIMEOUT)
.setConnectionRequestTimeout(REQUESTTIMEOUT).build();
}
static public String updates() {
String result = "";
String url = "https://example.com";
CloseableHttpClient httpClient = CreateHttpClient();
CloseableHttpResponse response = null;
URL urlObj;
RequestConfig config = GetConfig();
try {
urlObj = new URL(url);
HttpPost request = new HttpPost(url);
request.setConfig(config);
StringEntity params = new StringEntity("example");
request.addHeader("Accept-Language", "en");
request.addHeader("Content-Type", "application/json; charset=UTF-8");
request.addHeader("Content-Length", String.valueOf(params.getContentLength()));
request.addHeader("Host", urlObj.getHost());
request.addHeader("Connection", "Keep-Alive");
request.addHeader("Accept-Encoding", "gzip");
request.setEntity(params);
response = httpClient.execute(request);
int responseCode = response.getStatusLine().getStatusCode();
System.out.println("updates response code: " + responseCode);
// BufferedReader rd = new BufferedReader(new
// InputStreamReader(response.getEntity().getContent(), "UTF-8"));
result = EntityUtils.toString(response.getEntity());
// String line = "";
// while ((line = rd.readLine()) != null) {
// result.append(line);
// }
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
if (response != null)
response.close();
httpClient.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return result;
}
You should get the content from the response entity which is an InputStream. Than you could create a String from that InputStream with BufferedReader
HttpEntity entity = response.getEntity();
if (entity != null) {
InputStream instream = entity.getContent();
String result = convertInputStreamToString(instream);
instream.close();
}
Write your own convertInputStreamToString. If you need help for doing that check here:
Read/convert an InputStream to a String
I am doing Http POST request using HTTPClient 4.2.2. I am using .pfx certificate to access the URL mentioned in post request. But I am getting 302, Move temporarily error
//Java Code
public class CertificateAuth {
private static final long TIMEOUT = 500000000L;
//set trust store to be used to trust server certificate
private String tokeApiPostUrl = "http://test.com/l1/rest1/lt/v1/data";
private String tokenPost = "{\"id\": \"Token_15555\",\"type\": \"token\",\"entity_type\": \"Store\",\"entity_id\": \"StoreId\",\"expiration_time\": 1376579410}";
//client is taken as class varibable so that Cookies set by Server persists between
//multiple calls
private HttpClient client = null;
public CertificateAuth() {
}
public String createToken() throws Exception {
// set reasonable timeouts as we seem to wait forever to get a response:
KeyStore keystore = KeyStore.getInstance("pkcs12");
InputStream keystoreInput = new FileInputStream("abc.pfx");
keystore.load(keystoreInput, "password".toCharArray());
SchemeRegistry schemeRegistry = new SchemeRegistry();
SSLSocketFactory lSchemeSocketFactory = new SSLSocketFactory(keystore, "qwerty10");
schemeRegistry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80));
schemeRegistry.register(new Scheme("https", 443, lSchemeSocketFactory));
final HttpParams httpParams = new BasicHttpParams();
client = new DefaultHttpClient(new SingleClientConnManager(httpParams, schemeRegistry), httpParams);
String version = null;
HttpPost httpPost = new HttpPost(tokeApiPostUrl);
// httpPost.getParams().setParameter(ClientPNames.HANDLE_REDIRECTS, Boolean.TRUE);
client.getParams().setParameter(ClientPNames.ALLOW_CIRCULAR_REDIRECTS, true);
try {
Map<String, String> headersParameters = new HashMap<String, String>();
JSONObject jsonObj = new JSONObject(tokenPost);
setParametersJson(httpPost, headersParameters, jsonObj);
HttpResponse resp = client.execute(httpPost);
if(resp.getStatusLine().getStatusCode() == HttpURLConnection.HTTP_OK || resp.getStatusLine().getStatusCode() == HttpURLConnection.HTTP_CREATED) {
System.out.println("Succesfully queried");
}
} finally {
httpPost.releaseConnection();
}
return version;
}
private void setParametersJson(HttpRequestBase httpOperation, Map <String, String> headerParameters, JSONObject jsonObject) {
for (String headerName : headerParameters.keySet()) {
httpOperation.setHeader(headerName, headerParameters.get(headerName));
}
if (jsonObject != null) {
try {
StringEntity stringEntity = new StringEntity(jsonObject.toString());
if (httpOperation instanceof HttpPost) {
((HttpPost) httpOperation).setEntity(stringEntity);
} else if (httpOperation instanceof HttpPut) {
((HttpPut) httpOperation).setEntity(stringEntity);
}
} catch(UnsupportedEncodingException ex) {
ex.printStackTrace();
} catch(Exception ex) {
ex.printStackTrace();
}
}
}
public static void main(String[] args) throws Exception {
CertificateAuth ua = new CertificateAuth();
ua.createToken();
}
}
Add this line to your code.
client.setRedirectStrategy(new LaxRedirectStrategy());
I'm working on an application that needs to get the source of a web page from a link, and then parse the html from that page.
Could you give me some examples, or starting points where to look to start writing such an app?
You can use HttpClient to perform an HTTP GET and retrieve the HTML response, something like this:
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet(url);
HttpResponse response = client.execute(request);
String html = "";
InputStream in = response.getEntity().getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder str = new StringBuilder();
String line = null;
while((line = reader.readLine()) != null)
{
str.append(line);
}
in.close();
html = str.toString();
I would suggest jsoup.
According to their website:
Fetch the Wikipedia homepage, parse it to a DOM, and select the headlines from the In the news section into a list of Elements (online sample):
Document doc = Jsoup.connect("http://en.wikipedia.org/").get();
Elements newsHeadlines = doc.select("#mp-itn b a");
Getting started:
Download the jsoup jar core library
Read the cookbook introduction
This question is a bit old, but I figured I should post my answer now that DefaultHttpClient, HttpGet, etc. are deprecated. This function should get and return HTML, given a URL.
public static String getHtml(String url) throws IOException {
// Build and set timeout values for the request.
URLConnection connection = (new URL(url)).openConnection();
connection.setConnectTimeout(5000);
connection.setReadTimeout(5000);
connection.connect();
// Read and store the result line by line then return the entire string.
InputStream in = connection.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder html = new StringBuilder();
for (String line; (line = reader.readLine()) != null; ) {
html.append(line);
}
in.close();
return html.toString();
}
public class RetrieveSiteData extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String... urls) {
StringBuilder builder = new StringBuilder(100000);
for (String url : urls) {
DefaultHttpClient client = new DefaultHttpClient();
HttpGet httpGet = new HttpGet(url);
try {
HttpResponse execute = client.execute(httpGet);
InputStream content = execute.getEntity().getContent();
BufferedReader buffer = new BufferedReader(new InputStreamReader(content));
String s = "";
while ((s = buffer.readLine()) != null) {
builder.append(s);
}
} catch (Exception e) {
e.printStackTrace();
}
}
return builder.toString();
}
#Override
protected void onPostExecute(String result) {
}
}
Call it like
new RetrieveFeedTask(new OnTaskFinished()
{
#Override
public void onFeedRetrieved(String feeds)
{
//do whatever you want to do with the feeds
}
}).execute("http://enterurlhere.com");
RetrieveFeedTask.class
class RetrieveFeedTask extends AsyncTask<String, Void, String>
{
String HTML_response= "";
OnTaskFinished onOurTaskFinished;
public RetrieveFeedTask(OnTaskFinished onTaskFinished)
{
onOurTaskFinished = onTaskFinished;
}
#Override
protected void onPreExecute()
{
super.onPreExecute();
}
#Override
protected String doInBackground(String... urls)
{
try
{
URL url = new URL(urls[0]); // enter your url here which to download
URLConnection conn = url.openConnection();
// open the stream and put it into BufferedReader
BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
while ((inputLine = br.readLine()) != null)
{
// System.out.println(inputLine);
HTML_response += inputLine;
}
br.close();
System.out.println("Done");
}
catch (MalformedURLException e)
{
e.printStackTrace();
}
catch (IOException e)
{
e.printStackTrace();
}
return HTML_response;
}
#Override
protected void onPostExecute(String feed)
{
onOurTaskFinished.onFeedRetrieved(feed);
}
}
OnTaskFinished.java
public interface OnTaskFinished
{
public void onFeedRetrieved(String feeds);
}
If you have a look here or here, you will see that you can't do that directly with android API, you need an external librairy...
You can choose between the 2 here's hereabove if you need an external librairy.
One of the other SO post answer helped me. This doesn't read line by line; supposingly the html file had a line null in between. As preRequisite add this dependancy from project settings "com.koushikdutta.ion:ion:2.2.1" implement this code in AsyncTASK. If you want the returned -something- to be in UI thread, pass it to a mutual interface.
Ion.with(getApplicationContext()).
load("https://google.com/hashbrowns")
.asString()
.setCallback(new FutureCallback<String>()
{
#Override
public void onCompleted(Exception e, String result) {
//int s = result.lastIndexOf("user_id")+9;
// String st = result.substring(s,s+5);
// Log.e("USERID",st); //something
}
});
public class DownloadTask extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String... urls) {
String result = "";
URL url;
HttpsURLConnection urlConnection = null;
try {
url = new URL(urls[0]);
urlConnection = (HttpsURLConnection) url.openConnection();
BufferedReader br = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String inputLine;
while ((inputLine = br.readLine()) != null)
{
// System.out.println(inputLine);
result += inputLine;
}
br.close();
return result;
} catch (Exception e) {
e.printStackTrace();
return "failed";
}
}
}
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
DownloadTask task = new DownloadTask();
String result = null;
try {
result = task.execute("https://www.example.com").get();
}catch (Exception e){
e.printStackTrace();
}
Log.i("Result", result);
}