What a wicked NAT we weave: detangling the cloud

Cloud has 3 levels of address translation. Shocking I know. But… Believe it or not, this is the chain of events for a stream that arrives at your service in the cloud.

The sequence ends up being:

Client->LoadBalancer->Ingress->Sidecar->Service

and, LoadBalancer does a NAT, Ingress and Sidecar are proxies, so, well, Service never sees the IP of client.

Other people have been working at this problem (e.g. RFC7974), HAProxy ‘Proxy Protocol‘, others.

Today lets look at a practical example, using the HAProxy Proxy Protocol. Specifically, lets look at a tool CloudFlare did that allows adding transparency on the far side. They talk about it more here.

Here’s a recipe for you to try it out at home:

Start a new container (as per first line) and run the following lines

docker run --name mmp --privileged --rm -it -v $PWD:$PWD ubuntu:18.04
apt update && apt install -y iproute2 curl iptables python3 netcat

iptables -t mangle -I PREROUTING -m mark --mark 123 -j CONNMARK --save-mark 
iptables -t mangle -I OUTPUT -m connmark --mark 123 -j CONNMARK --restore-mark
ip6tables -t mangle -I PREROUTING -m mark --mark 123 -j CONNMARK --save-mark 
ip6tables -t mangle -I OUTPUT -m connmark --mark 123 -j CONNMARK --restore-mark 
ip rule add fwmark 123 lookup 100 
ip route add local 0.0.0.0/0 dev lo table 100 
ip -6 rule add fwmark 123 lookup 100 
ip -6 route add local ::/0 dev lo table 100 
echo 1 | tee /proc/sys/net/ipv4/conf/eth0/route_localnet

python3 -m http.server -b 127.0.0.1 8000

Now run this from host:

docker exec -it mmp $PWD/mmproxy -a $PWD/networks.txt \
  -l 0.0.0.0:80 -4 127.0.0.1:8000 -6 '[::1]:8000'

Now run this from host:

echo -en "PROXY TCP4 1.2.3.4 1.2.3.4 11 11\r\nGET / HTTP/1.1\r\n\r\n" | \
 docker exec -i mmp nc -v 127.0.0.1 80

On the first window, you will see something like:

1.2.3.4 - - [18/Jun/2018 14:32:04] "GET / HTTP/1.1" 200 

The 1.2.3.4 indicates the source IP.

What sourcery is this? Is this a tool to undo the magic NAT stuff of the cloud? Or a security nightmare?