What would happen in Varnish if multiple IPs are in an X-Forward-For header which is compared to an ACL using the tilde operator?
Dummy example:
The request has the following HTTP header:
X-Forward-For: 160.12.34.56, 10.10.10.10
The Varnish config looks like this:
acl internal {
"10.10.10.10"
}
if ((std.ip(req.http.X-Forward.For, "0.0.0.0") ~ internal)){
# THIS CODE
}
else {
# OR THIS CODE
}
Which code block is executed?
Also, does the order of the IPs matter in the X-Forward-For header?
Does it change if there are 2 X-Forward-For headers, each with one of the two IPs?
Will it work?
The short answer to your question is no, it won't work.
std.ip() expects to receive a single IP address, not a collection. The conversion will fail, and the fallback value (second argument of the function) will be returned.
Here's a quick test script that illustrates this:
vcl 4.0;
import std;
backend default none;
sub vcl_recv {
set req.http.x-f = "1.2.3.4, 5.6.7.8";
return(synth(200,std.ip(req.http.x-f,"0.0.0.0")));
}
This example will return 0.0.0.0.
Does X-Forwarded-For need multiple IP addresses?
It does make sense to ask the question if your X-Forwarded-For header needs multiple IP addresses.
The idea is to indicate to the origin server what the IP address of the original client was.
In your case there is more than 1 proxy in front of the webserver, so a natural reaction is to chain the IP addresses in the X-Forwarded-For header.
A better solution would be to figure out what the IP address of the original client was, and set that value in X-Forwarded-For.
The best way to get this done is by leveraging the PROXY protocol, which Varnish supports.
Leverage the PROXY protocol
The PROXY protocol has the capability of transporting the HTTP protocol, but additionally keep track of the connection parameters of the original client.
Varnish supports this and allows you to set an extra listening port that listens for PROXY requests.
Here's an example of how you can start varnishd with PROXY support:
varnishd -a :80 -a :8443,PROXY -f /etc/varnish/default.vcl -s malloc,256m
As you can see, port 80 is still available for regular HTTP, but port 8443 was allocated for PROXY support.
If the proxy servers in front of Varnish support PROXY, Varnish will take the value from the original client and automatically set X-Forwarded-For with that value.
This way you always know who the client was, and you can safely perform your ACL check.
Additionally, there's also a PROXY module for Varnish, that can give you information about potential TLS termination that took place in front of Varnish.
Related
My apologies if this is a duplicate. I can find a million results about CORS policy issues, but not about this specific one:
I developed a simple "speed test" site for my users (wfh employees of my company) to access. It tests speeds across the public net to different datacenters we utilize, and via the users' VPN connection to one of our DCs.
There are more complicated elements, but for a basic round-trip "ping" I have an extremely simple PHP script on the server that contains:
<?php
header('Access-Control-Allow-Origin: *');
header('Access-Control-Allow-Headers: *');
if ($_GET['simple'] == '1')
die('{ }');
?>
It is called like this:
$.ajax({
type: 'GET',
url: sURL,
data: { ignore: (pingCounter.start = new Date().getTime()) },
dataType: 'text',
timeout: iTimeout
})
.done(function(ret) {
pingCounter.end = new Date().getTime();
[...] (additional code omitted for brevity)
(I know this has additional overhead other than the raw round-trip network traffic timing, but I don't need sub-ms accuracy. I just need to be able to tell users "the problem is on your end" or "ah yes, the problem is the latency between your house and this particular DC".)
The same server running that PHP code is addressable at the following URLs at the DC wherein our VPN server lies:
http://speedtest-int.mycompany.com/ping.php
http://speedtest-ext.mycompany.com/ping.php
Public DNS resolves like this:
speedtest-int.mycompany.com IN A 1.1.1.1 (Actual public IP redacted)
speedtest-int.mycompany.com IN A 10.1.1.1 (Actual internal IP redacted)
If I access either URL from my browser directly, it loads fine (which is to say it responds with { }).
When loading via the JS snippet above, the call to http://speedtest-ext.mycompany.com/ping.php works fine.
The call to http://speedtest-int.mycompany.com/ping.php fails with "The request client is not a secure context and the resource is in more-private address space 'private'".
Fair enough, the solution is to add Access-Control-Allow-Private-Network: *, right?
EXCEPT that apparently can only be used with SSL:
https://developer.chrome.com/blog/private-network-access-update/
I have a self-signed cert on there, but that obviously fails by policy for that reason.
I could just get a LetsEncrypt cert for multiple subdomains. EXCEPT it will never validate the URL http://speedtest-int.mycompany.com because the LetsEncrypt servers won't be able to reach that to validate ownership, as it's a private IP.
I have no control over most of my users' machines, so I can't necessarily install trusted internal certs or change browser options. Most users use Chrome.
So is my solution to buy a UCC or wildcard cert?
I feel like I'm in a catch-22, and I don't want to spend however-much on a UCC cert for an internal app that will be very very very occasionally used by one of our 25 home-based employees when I want to prove that their home "internet is bad" and not the corp network.
Thanks in advance; I'm sure there's a stupidly obvious solution I'm not seeing.
(I'm considering pushing a /32 route to my VPN users for another real public IP to be used in place of the internal IP. Then I can have the "internal" test run against an otherwise publicly accessible IP which could be validated by LetsEncrypt, but VPN users would hit it via the VPN. Is that silly?)
Edit: If anyone is curious -- or it helps to clarify my goal here -- this is the output when accessing the speedtest page:
http://s.co.tt/wp-content/uploads/2021/12/Internal_Speedtest_Example-Redacted.png
It repeats for 20 cycles (or until stopped) and runs each element a varying number of times per cycle, collecting the average time for each. It ain't pretty, but it work(ed).
i configured an egress router like described here:
https://docs.openshift.com/container-platform/3.3/admin_guide/managing_pods.html#admin-guide-controlling-egress-traffic
But it does not work.
In my understanding, the options will be resolved like this:
name: EGRESS_SOURCE <-- This is the network where the nodes live (in my case the vm where the Containers are running on)
value: 192.168.12.99
name: EGRESS_GATEWAY <-- The gateway over which the destination ip address is routable.
value: 192.168.12.1
name: EGRESS_DESTINATION <--- The destination ip of the application i want to reach. In my case its a mongoDB living in a classical VM.
value: 203.0.113.25
Am i right or do i miss something ?
How would i be able to reach the target ?
Do i need to address the source ip to access the MongoDB or do i simply address the IP of my MongoDB an the traffic gets nat'd the way over my egress router (This is how i understood the traffic flow will be btw.) ?
How can i troubleshoot this kind of problem ?
Best Regards,
Marcus
Ok, it worked by now. I created a service and adressed the ip of this service to reach my destination.
The alternative way is to address the ip of the container.
So from inside the container to reach your original destination don't use the original ip, rather use the egress pod ip or preferred use the ip of the created service.
!!Attention: The destination ip must be outside of the host/node ip range otherwise it would not work. It seems that, if you use a destination ip from your host/node range, the standard gw will get the request and i think it will discard it. !!
And i would suggest to use the egress router image from redhat, instead the origin, which is stated in the official document from redhat...
image: registry.access.redhat.com/openshift3/ose-egress-router
If we have a customer with a cname record, sub1.notourserver.com, pointing to something like abcdefg.ourserver.com, we read the host as sub1.notoursever.com. Is it possible for us to somehow get the value abcdefg.ourserver.com from this request? We have a subdomain route setup, but it is not picking up on it because the host does not match our SERVER_NAME config setting.
HTTPÂ does not provide that information, and so neither can Flask/Werkzeug. You need to use in Flask/Werkzeug the (sub)domain names actually used by clients.
If you really can not do that, you need to hack a WSGI middleware that maintains an explicit mapping (or makes DNS requests) and patches environ['HTTP_HOST'].
I have a domain name abc.mydomain.com
This is a https URL ( http redirects to the https version )
However, I now need to be able to handle www.abc.mydomain.com to redirect to abc.mydomain.com
How can I do this? is it a webserver level redirect or something to be done at the DNS resolution.
I know my URL already has the "abc" as its sub-domain and I dont need a "www", however, we noticed that "www.news.google.com" resolves to "news.google.com" - hence wondering if I can achieve it too
Thank you!
In short, yes.
DNS works on a hierarchy - the DNS server for .com can delegate down to the nameserver for your domain which can delegate further, or just answer the requests, which needs to be your first step.
If you use Bind style zone files, you can do something like (where 123.45.67.89 is your webserver IP address):
* IN A 123.45.67.89
Then, you also need your webserver to resolve that to the right virtual host/redirect as desired.
I have the beginnings of an HTTP Intercepting Proxy written in Ruby:
require 'socket' # Get sockets from stdlib
server = TCPServer.open(8080) # Socket to listen on port 8080
loop { # Servers run forever
Thread.start(server.accept) do |client|
puts "** Got connection!"
#output = ""
#host = ""
#port = 80
while line = client.gets
line.chomp!
if (line =~ /^(GET|CONNECT) .*(\.com|\.net):(.*) (HTTP\/1.1|HTTP\/1.0)$/)
#port = $3
elsif (line =~ /^Host: (.*)$/ && #host == "")
#host = $1
end
print line + "\n"
#output += line + "\n"
# This *may* cause problems with not getting full requests,
# but without this, the loop never returns.
break if line == ""
end
if (#host != "")
puts "** Got host! (#{#host}:#{#port})"
out = TCPSocket.open(#host, #port)
puts "** Got destination!"
out.print(#output)
while line = out.gets
line.chomp!
if (line =~ /^<proxyinfo>.*<\/proxyinfo>$/)
# Logic is done here.
end
print line + "\n"
client.print(line + "\n")
end
out.close
end
client.close
end
}
This simple proxy that I made parses the destination out of the HTTP request, then reads the HTTP response and performs logic based on special HTML tags. The proxy works for the most part, but seems to have trouble dealing with binary data and HTTPS connections.
How can I fix these problems?
First, you would probably be better off building on an existing Ruby HTTP proxy implementation. One such is already available in the Ruby standard library, namely WEBrick::HTTPProxyServer. See for example this related question for an implementation based on that same class: Webrick transparent proxy.
Regarding proxying HTTPS, you can't do much more than just pass the raw bytes. As HTTPS is cryptographically protected, you cannot inspect the contents at the HTTP protocol level. It is just an opaque stream of bytes.
WEBrick is blocking I/O ... This mean it does not able to stream the response. For example if you go on a youtube page to see a video, the stream will not be forwarded to your browser until the proxy have downloaded all the video cotent.
If you want the video be played in your browser during it download, you have to look for a non blocking I/O solution like EventMachine.
For HTTPS the solution is a little bit complicated since you have to develop a man in the middle proxy.
This was an old question, but for the sake of completeness here goes another answer.
I've implemented a HTTP/HTTPS interception proxy in Ruby, the project is hosted in github.
The HTTP case is obvious, HTTPS interception in accomplished via an HTTPS server that acts as a reverse proxy (and handles the TLS handshake). I.e.
Client(e.g. Browser) <--> Proxy1 <--> HTTPS Reverse Proxy <--> Target Server
As Valko mentioned, when a client connects to a HTTPS server through a proxy, you'll see a stream of encrypted bytes (since SSL provides end-to-end encryption). But not everything is encrypted, the proxy needs to know to whom the stream of bytes should be forwarded, so the client issues a CONNECT host:port request (being the body of the request the SSL stream).
The trick here is that the first proxy will forward this request to the HTTPS Reverse Proxy instead of the real target server. This reverse proxy will handle the SSL negotiation with the client, have access to the decrypted requests, and send copies (optionally altered versions) of these requests to the real target server by acting as a normal client. It will get the responses from the target server, (optionally) alter the responses, and send them back to the client.