Apache HttpClient 4.5 cross host redirect - apache-httpclient-4.x

Here is my scenario
I am calling a url from HOSTA
http://www.hosta.com/something
This is redirecting me to the HOSTB url
http://www.hostb.com/something
This is response code which I gets from url a
http-outgoing-5 << HTTP/1.1 302 Moved Temporarily
http-outgoing-5 << Location: www.hostb.com/something
Redirecting to www.hostb.com/something
http-outgoing-10 >> Host: hosta.com
In the last line I see the host header is set to HOSTA which is wrong.
Here is the code I am using
This is how connection is created
private PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();
HttpClientBuilder builder= HttpClients.custom();
builder.setMaxConnPerRoute(50)
.setConnectionManager(cm)
.setConnectionTimeToLive(timeout, TimeUnit.MILLISECONDS)
.setDefaultCookieStore(cookieStore)
.setRedirectStrategy(new LaxRedirectStrategy())
.setDefaultSocketConfig(socketConfig)
.setServiceUnavailableRetryStrategy(retryStrategy)
.setKeepAliveStrategy(keepAliveStrategy)
.setRetryHandler(myRetryHandler);
Method to call the request
HttpPost post = new HttpPost(url);
post.setHeader("User-Agent", USER_AGENT);
post.setHeader("Cache-Control", "max-age=0");
post.setHeader("Connection", "keep-alive");
post.setHeader("Content-Type", "application/x-www-form-urlencoded");
post.setHeader("Upgrade-Insecure-Requests", "1");
post.setConfig(requestConfig);
if (origin != null) {
post.setHeader("Origin", origin);
}
if (host != null) {
post.setHeader("Host", host);
}
if (referer != null) {
post.setHeader("Referer", referer);
}
//some more code
I am using the automatic redirection (using LaxDirectStrategy) which are working fine. But in this particular case I am getting issue.
is there any way to update the HOST header in case of cross host redirect or automatic redirection can be stopped for one particular request?

I used the RequestConfig to disable/enable the redirection for particular request.
By Default I have redirection enabled.Whenever I wanted it to disable I used
RequestConfig requestConfig = RequestConfig.custom()
.setRedirectsEnabled(false).build();//disable redirect
HttpPost post = new HttpPost(url);
post.setConfig(requestConfig);//poass the request config to request

I faced similar issue with HttpClient 4.5.9 (and 4.5.13 too). It turned out that I was inadvertently writing the Host header.
HttpClient keeps the Host header if it is already defined in a request.
From Oleg Kalnichevski :
One ought not mess with HTTP transport headers such as 'Host' (...) and let HttpClient generate the appropriate header values instead.(...) As far as the 'Host' header is concerned HttpClient simply does not overwrite the existing value assuming that the user knows better.
Oleg
Reference: in case of a redirect a wrong host param is set to http header

Related

How do I make a secure API request from an Arduino ESP32, programmed in the Arduino IDE using ArduinoJson?

I have been hacking away at this for a few days with no luck.
I am trying to make a secure (SSL/HTTPS) API request in an Arduino environment. The controller I am using is an ESP32, which connects through wifi fine, and can retrieve/post data. However I am having no luck connecting to a secure API.
I'm trying to connect to this API https://strike.acinq.co/documentation/api-reference
EXAMPLE CURL REQUEST IN API'S DOCUMENTATION:
$ curl https://api.dev.strike.acinq.co/api/v1/charges \
-u sk_pJDwxFxCVw5fQJhRRMpf29jReUjjN: \
-X POST \
-d amount=42000 \
-d currency="btc" \
-d description="1%20Blockaccino"
Here is my Arduino code, I am using the ArduinoJson.h and WiFi.h libraries:
// Connect to HTTP server
WiFiClient client;
client.setTimeout(10000);
if (!client.connect("api.strike.acinq.co", 80)) {
Serial.println(F("Connection failed"));
return;
}
Serial.println(F("Connected!"));
// Send HTTP request
client.println(F("GET /api/v1/charges?id=MYKEY&amount=4200&currency=btc HTTP/1.0"));
client.println(F("Host: api.strike.acinq.co"));
client.println(F("Content-Type: application/x-www-form-urlencoded"));
client.println(F("Connection: close"));
if (client.println() == 0) {
Serial.println(F("Failed to send request"));
return;
}
// Check HTTP status
char status[32] = {0};
client.readBytesUntil('\r', status, sizeof(status));
if (strcmp(status, "HTTP/1.1 200 OK") != 0) {
Serial.print(F("Unexpected response: "));
Serial.println(status);
return;
}
A 401 "Invalid API Key" Is the closest I have got. I know the API-key works, and that I am just using it wrong. I've tried moving the key to:
client.println(F("id: MYKEY"));
but that didn't work either.
I have tried other libraries and ArduinoJson seems to be the best. I think the issue is the fact its a secure server and the layout of my request. I found many resources for connecting to open API's on Arduino, but nothing on connecting to secure ones. I think I am almost there with the code...
UPDATE
So I have updated my code. I am still trying to use ArduinoJson. I can connect to the API but it keeps spitting out "HTTP/1.1 400 BAD_REQUEST". I don't know weather this is because its over HTTPS or the formatting of my request.
In the API docs -u and -X don't have a field name like "amount=4200", so I am assuming -u would just be added client.print("?="+apiKey);
//open weather map api key
String apiKey= "myapikey";
int status = WL_IDLE_STATUS;
char server[] = "api.strike.acinq.co";
Serial.println("\nStarting connection to server...");
// if you get a connection, report back via serial:
if (client.connect(server, 80)) {
Serial.println("connected to server");
// Make a HTTP request:
client.print("POST /api/v1/charges");
client.print("?="+apiKey);
client.print("&amount=4200");
client.print("&currency='btc'");
client.println("&description='sweets'");
client.println("Host: api.strike.acinq.co");
client.println("Connection: close");
client.println();
}
else {
Serial.println("unable to connect");
}
UPDATE
I figured out the println and print actually mean something and have subsequently organised my request much better. It still comes back with 400 Unauthorized?
String PostData = "&description=\"car\"&amount=1000&currency=\"sweetsandthat\"";
client.println("POST /api/v1/charges HTTP/1.1");
client.println("Host: api.strike.acinq.co");
client.println("Authorization: Basic "+apiKey);
client.print("Content-Length: ");
client.println(PostData.length());
client.println(); // blank line required
client.println(PostData);
Serial.println("POSTED DATA: " + PostData);
// client.stop();
client.println();
} else {
Serial.println("unable to connect");
}
delay(1000);
String line = "";
while (client.connected()) {
line = client.readStringUntil('999');
Serial.println(line);
Serial.println("parsingValues");
//create a json buffer where to store the json data
StaticJsonBuffer<5000> jsonBuffer;
JsonObject& root = jsonBuffer.parseObject(line);
if (!root.success()) {
Serial.println("parseObject() failed");
return;
}
//get the data from the json tree
String nextWeatherTime0 = root["id"][0];
// Print values.
Serial.println(nextWeatherTime0);
}
client.println("Connection: close");
client.stop();
}
Check the response for a BAD request, We usually get it when we deal with a bad URL or URL not found. check whether you are connecting to the same url mentioned in docs.
First connect to the api and after that make queries like providing your api key and feilds
remove this.
client.println("Host: api.strike.acinq.co");
and use GET request to get the response of the data you have in these fields
String PostData = "&description=\"car\"&amount=1000&currency=\"sweetsandthat\""
I have also been struggling to get an https post to work on the esp32. A few things, the wifi.h module, I believe, does not support https. The WiFiClientSecure.h does, and you need to set the port to 443. I have also failed to get a POST to work, but I succeed in a basic GET test connection to howsmysssl.com. Andreas Spiess covers this well in a youtube video. He goes beyond SSL to establishing trust. I just want basic SSL to work, so if you get this figured out, please let me know. Hopefully I got you one step closer. :)

Allow Access-Control-Allow-Origin header using HTML5 fetch API

I am using HTML5 fetch API.
var request = new Request('https://davidwalsh.name/demo/arsenal.json');
fetch(request).then(function(response) {
// Convert to JSON
return response.json();
}).then(function(j) {
// Yay, `j` is a JavaScript object
console.log(JSON.stringify(j));
}).catch(function(error) {
console.log('Request failed', error)
});
I am able to use normal json but unable to fetch the data of above api url.
It throws error:
Fetch API cannot load https://davidwalsh.name/demo/arsenal.json. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost' is therefore not allowed access. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
Like epascarello said, the server that hosts the resource needs to have CORS enabled. What you can do on the client side (and probably what you are thinking of) is set the mode of fetch to CORS (although this is the default setting I believe):
fetch(request, {mode: 'cors'});
However this still requires the server to enable CORS as well, and allow your domain to request the resource.
Check out the CORS documentation, and this awesome Udacity video explaining the Same Origin Policy.
You can also use no-cors mode on the client side, but this will just give you an opaque response (you can't read the body, but the response can still be cached by a service worker or consumed by some API's, like <img>):
fetch(request, {mode: 'no-cors'})
.then(function(response) {
console.log(response);
}).catch(function(error) {
console.log('Request failed', error)
});
This worked for me :
npm install -g local-cors-proxy
API endpoint that we want to request that has CORS issues:
https://www.yourdomain.com/test/list
Start Proxy:
lcp --proxyUrl https://www.yourdomain.com
Proxy Active
Proxy Url: http://www.yourdomain.com:28080
Proxy Partial: proxy
PORT: 8010
Then in your client code, new API endpoint:
http://localhost:8010/proxy/test/list
End result will be a request to https://www.yourdomain.ie/test/list without the CORS issues!
Solution to resolve issue in Local env's
I had my front-end code running in http://localhost:3000 and my API(Backend code) running at http://localhost:5000
Was using fetch API to call the API. Initially, it was throwing "cors" error.
Then added this below code in my Backend API code, allowing origin and header from anywhere.
let allowCrossDomain = function(req, res, next) {
res.header('Access-Control-Allow-Origin', "*");
res.header('Access-Control-Allow-Headers', "*");
next();
}
app.use(allowCrossDomain);
However you must restrict origins in case of other environments like stage, prod.
Strictly NO for higher environments.
I know this is an older post, but I found what worked for me to fix this error was using the IP address of my server instead of using the domain name within my fetch request.
So for example:
#(original) var request = new Request('https://davidwalsh.name/demo/arsenal.json');
#use IP instead
var request = new Request('https://0.0.0.0/demo/arsenal.json');
fetch(request).then(function(response) {
// Convert to JSON
return response.json();
}).then(function(j) {
// Yay, `j` is a JavaScript object
console.log(JSON.stringify(j));
}).catch(function(error) {
console.log('Request failed', error)
});
You need to set cors header on server side where you are requesting data from.
For example if your backend server is in Ruby on rails, use following code before sending back response. Same headers should be set for any backend server.
headers['Access-Control-Allow-Origin'] = '*'
headers['Access-Control-Allow-Methods'] = 'POST, PUT, DELETE, GET, OPTIONS'
headers['Access-Control-Request-Method'] = '*'
headers['Access-Control-Allow-Headers'] = 'Origin, X-Requested-With, Content-Type, Accept, Authorization'
If you are use nginx try this
#Control-Allow-Origin access
# Authorization headers aren't passed in CORS preflight (OPTIONS) calls. Always return a 200 for options.
add_header Access-Control-Allow-Credentials "true" always;
add_header Access-Control-Allow-Origin "https://URL-WHERE-ORIGIN-FROM-HERE " always;
add_header Access-Control-Allow-Methods "GET,OPTIONS" always;
add_header Access-Control-Allow-Headers "x-csrf-token,authorization,content-type,accept,origin,x-requested-with,access-control-allow-origin" always;
if ($request_method = OPTIONS ) {
return 200;
}
Look at https://expressjs.com/en/resources/middleware/cors.html
You have to use cors.
Install:
$ npm install cors
const cors = require('cors');
app.use(cors());
You have to put this code in your node server.

Extracting gzip data from Apache-httpclient without decompression

I'm using apache httpclient 4.3.5 to send a request to an upstream server which returns a gzipped response. I need to pass this response AS-IS to a downstream server without any form of decompression. However, httpclient is far too helpful and insists on decompressing the response and I can't find any way of persuading it to stop.
CloseableHttpClient client = HttpClients.createDefault();
CloseableHttpResponse serverResponse = client.execute(serverRequest);
try {
HttpEntity entity = serverResponse.getEntity();
downstreamResponse.setStatus(serverResponse.getStatusLine().getStatusCode());
for (Header header : serverResponse.getAllHeaders()) {
downstreamResponse.setHeader(header.getName(), header.getValue());
}
entity.writeTo(downstreamResponse.getOutputStream());
downstreamResponse.flushBuffer();
} finally {
serverResponse.close();
}
I'm sure that there is some way of configuring the client using some form of the construct
return HttpClients.custom()
....
.build();
but I can't find it. Can the experts please advise?
CloseableHttpClient client = HttpClients.custom()
.disableContentCompression()
.build();

WinRT use HttpClient to call Web API based URL with token

We are building a WinRT app which gets data from server which is Web API based & so it gives data in json and/or XML format.
When app user logs in for the first time using his credentials(username,password), the response that comes from server is a success bit & a TOKEN, which should be used in successive URL requests.
I am using httpclient for sending requests
using (HttpClient httpClient1 = new HttpClient())
{
string url = "http://example.com/abc/api/process1/GetLatestdata/10962f61-4865-4e7a-a121-3fdd968824b5?employeeid=6";
//The string 10962f61-4865-4e7a-a121-3fdd968824b5 is the token sent by the server
var response = await httpClient1.GetAsync(new Uri(url));
string content = await response.Content.ReadAsStringAsync();
}
Now the response that i get is with status code 401 "unauthorised".
And the xml i get in response is "Unauthorised User".
Is there anything i need to change in appManifest??
I've checked this, but cant we use httpclient without credentials??
Your Capabilities are enough. You don't even need Internet (Client) because it's included in Internet (Client & Server).
You do not have credentials for WinRT HttpClient, in your linked post they referr to System.Net.Http.HttpClientHandler.
Maybe you can use the HttpBaseProtocolFilter to add the credentials?
using (var httpFilter = new HttpBaseProtocolFilter())
{
using (var httpClient = new HttpClient(httpFilter))
{
httpFilter.ServerCredential...
}
}
I don't know your security mechanism, I'm using a HttpClient and my session-key is in a cookie. But I think your client code looks fine.

ASP.NET Web API: How to do a Post back results with redirection

I got a Web API that performs a function and posts a JSON response back to a calling page.
This is standard Web API behaviour and works beautifully.
Now I want to modify the controller so that in addition to the post back the user is redirected back to the page on the calling web site where the result of the Web API call can be displayed (in JSON).
So basically I want to:
(1) Server side post back the results in JSON to a page and redirect to the same page from the Web API
(2) On the caller's site, I want to display the JSON that was posted back.
How do I do this?
I already tried for many hours ...
e.g.:
using (WebClient client = new WebClient())
{
client.Headers.Add("Content-Type", "text/json");
client.Headers.Add("Accept", "text/json");
try
{
ErrorText = client.UploadString(redirectURL, "POST", JsonConvert.SerializeObject(orderresponse));
Response.Redirect(redirectURL);
}
catch (WebException err)
{
ErrorText = err.Message; //Todo - write to logfile
}
}
Instead of doing the redirect on the server, instruct the client to do it by using the appropriate HTTP status code. For example:
public HttpResponseMessage Post(MyModel model)
{
// handle the post
MyResult result = ...;
// redirect
var response = Request.CreateResponse<MyResult>(HttpStatusCode.Moved, result);
response.Headers.Location = new Uri("http://www.yourdomain.com/redirectURI");
return response;
}