Setting up docker containers with nat - containers

Question

I am setting up two docker containers
container1 container2
| | |
eth0 eth1 |
| | eth1
docker0 docker1<----------------
|
|
internet
docker0 and docker1 are the bridges.
I have ip forwarding to 1 in both host and in containers.
I have setup
iptables -I POSTROUTING -t nat -o eth0 -j MASQUERADE in container 1
Still i am not able to ping anything from container 2 to internet. I can see that packets are being received at eth1 of container 1.
OS: ubuntu 13.10
docker version: 0.11.1, build fb99f99
Am i missing some configuration?
Steps to reproduce:
SERV=$(docker run --privileged=true -i -d -t -v ~/Projects/code/myproject/build:/build:ro debian:7.4 /bin/bash)
CLI=$(docker run --privileged=true -i -d -t -v ~/Projects/code/myproject/build:/build:ro debian:7.4 /bin/bash)
sudo pipework br1 $SERV 10.1.0.1/8
sudo pipework br1 $CLI 10.1.0.3/8
In $SERV:
iptables -I POSTROUTING -t nat -o eth0 -j MASQUERADE
In $CLI
Disable the interface eth0. Set default route to eth1 interface.
Now ping is happening to 10.1.0.1 from $CLI but not to the internet.

Answer 1

Hm, it should work as you described. Maybe the default route is not configured correctly.
This is what I did:
SERV=$(docker run -i --privileged -d -t debian:7.4 /bin/bash)
CLI=$(docker run --privileged -i -d -t debian:7.4 /bin/bash)
docker exec -ti $CLI ping google.de # Internet up
docker exec -ti $CLI ip link set eth0 down
docker exec -ti $CLI ping google.de # Internet down
pipework br1 $SERV 10.1.0.1/8
pipework br1 $CLI 10.1.0.2/8
docker exec -ti $SERV apt-get install -y iptables
docker exec -ti $SERV iptables -I POSTROUTING -t nat -o eth0 -j MASQUERADE
docker exec -ti $CLI ip route add default via 10.1.0.1 dev eth1
docker exec -ti $CLI ping google.de # Internet up
docker exec -ti $CLI apt-get install -y traceroute
docker exec -ti $CLI traceroute google.de

Answer 2

The only way iptables is changed is when executed from Docker host on a containers run with
--privileged
Here is a script:
iptables along with a couple of tools are installed during the image build (Dcokerfile)
inetutils-traceroute iputils-tracepath iptables
Here I use "phusion-dockerbase", you can use whatever image you want:
#!/bin/bash
### ==> Install & configure iptable during build
#RUN sudo apt-get install -y inetutils-traceroute iputils-tracepath iptables
# Build the image
#sudo docker build -t mybimage -f phusion-dockerbase .
### container1
C1=$(docker run --privileged -i -d -t mybimage /bin/bash)
sudo docker exec -ti $C1 iptables -I POSTROUTING -t nat -o eth0 -j MASQUERADE
sleep 2
sudo pipework br6 -i eth1 $C1 192.168.66.1/24
### container2
lxterminal -e "sudo docker run -ti --name c2name mybimage /bin/bash"
sleep 2
C2="$(sudo docker ps | grep c2name | awk '{ print $1; }')"
sudo pipework br6 -i eth1 $C2 192.168.66.2/24#192.168.66.1
Result:
./lab.sh
From the Container1 (I use lxterminal to open it in a new window):
Note that, as soon as you stop container1, corresponding pipework and iptable modification are lost and even when restart the stopped container, you will need to reissue the commands:
pipework br6 -i eth1 52b95d6052f7 192.168.66.1/24
docker exec 52b95d6052f7 iptables -I POSTROUTING -t nat -o eth0 -j MASQUERADE
for container1 to act again like nat box.
Even commiting the running container1 to a new image and run a new container from it, doesn't help.