Agilicus Connector Air Gap Machine With Containers

Imagine we have a complex set of orchestrated containers. They interoperate with each other. But, we don’t want them reaching the Internet, or the Internet reaching them.

Some or all of these containers have a need to use remote services. Perhaps a cloud provider, perhaps another set in another building.

In this example we show how to:

1. Block all inbound traffic to a Debian 12 host
2. Block all outbound traffic except for Agilicus API + Dataplane
3. Use a service forwarder to allow specific containers to reach specific remote services

For the container runtime we will use docker. The example below is a Debian 12 host with all options default, with docker.io installed.

Debian 12: Default Firewall Rules

Before we start we can inspect the default firewall rules. In a nutshell these:

1. allow all outbound traffic from the host to Internet
2. allow all outbound traffic from containers to Internet
3. allow all outbound traffic from containers to host

# iptables-save
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Apr 18 18:57:40 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Apr 18 18:57:40 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Apr 18 18:57:40 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Thu Apr 18 18:57:40 2024

Setup: Block inbound traffic, block outbound except to Agilicus Cloud, Local DNS/DHCP

Note: you may wish to have console access rather than SSH access in case you make a mistake.

We can see the Agilicus required IP/hostnames. For this example, we will allow (on port 443), these two IP:

• 34.95.12.47 (www.agilicus.com)
• 35.203.36.11 (api.agilicus.com)

iptables -P OUTPUT DROP
iptables -P INPUT DROP
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp -d 34.95.12.47/32 --dport 443 -j ACCEPT
iptables -A OUTPUT -p tcp -d 35.203.36.11/32 --dport 443 -j ACCEPT
iptables -A INPUT -p udp -m multiport --sports 67,68,123,53 -j ACCEPT
iptables -A OUTPUT -p udp -m multiport --dports 67,68,123,53 -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT

At this stage, we can test. First let us confirm that DNS works and that we can reach Agilicus WWW (for system updates):

$ curl -I https://www.agilicus.com/
HTTP/2 200

Now let us check that we cannot reach other websites:

$ curl -I https://www.google.ca/
<timeout>

Now let us demonstrate that a docker container cannot escape:

# docker run --rm -it busybox
Unable to find image 'busybox:latest' locally
... timeout

Not so fast, we have now blocked docker hub. For our scheme to work, we will need to either get all containers on the machine out of band, or allow a docker registry. I will demonstrate the out-of-band technique:

# On a machine with Internet access (not the one we are firewalling):
$ docker pull library/busybox
Using default tag: latest
latest: Pulling from library/busybox
7b2699543f22: Pull complete
Digest: sha256:c3839dd800b9eb7603340509769c43e146a74c63dca3045a8e7dc8ee07e53966
Status: Downloaded newer image for busybox:latest
docker.io/library/busybox:latest
$ docker image save -o /tmp/busybox.tar library/busybox
$ scp /tmp/busybox.tar don@192.168.122.206:/tmp/
busybox.tar                                                                                       100% 4400KB  54.7MB/s   00:00
# 192.168.122.206 is the IP of the firewalled machine which we have allowed SSH to, you can also use a USB flash drive
# On the firewalled machine:
$ docker image load -i /tmp/busybox.tar
$ docker run --rm -it library/busybox
/ # ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=55 time=17.874 ms
^C
### HUH?

OK, why did the container reach the Internet if the host cannot? For this we need to understand FORWARD (e..g routing) versus INPUT/OUTPUT. The Host is routing the docker traffic, we need some rules in the FORWARD path.

$ iptables -P FORWARD DROP
$ iptables -t nat -D POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE

At this stage, lets test again. We have made the default forward policy be to drop, and we have removed the MASQUERADE (NAT) rule.

$ docker run --rm -it library/busybox
/ #
/ # ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
^C
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
/ # wget https://www.agilicus.com/
^C

OK, we now have a set of containers which cannot reach the Internet on anything, and, a host that can reach a very limited set (DNS, DHCP, and Agilicus AnyX Cloud).

Let us now check that the container can still reach the host. For this we will use two windows. In one, we will run ‘netcat’ to listen on a port, in the other we will test from the host to itself. In green, window 1, we run netcat in listen mode:

$ nc -lvp 8888
listening on [any] 8888 ...
connect to [127.0.0.1] from localhost [127.0.0.1] 38276

In black, we run netcat in connect mode, we observe it works.

$ nc -vv localhost 8888
localhost [127.0.0.1] 8888 (?) open
^C sent 0, rcvd 0

Let us now try from within the container

$ docker run --rm -it library/busybox
/ # nc 172.17.0.1 8888
... hang

Why did this hang? We are blocking all INPUT/OUTPUT except loopback (lo) and except for certain IP/port pairs. Let us add docker0 back in:

# iptables -A OUTPUT -o docker0 -j ACCEPT
# iptables -A INPUT -i docker0 -j ACCEPT

OK at this stage we can test again, and it works. The container can reach the host (on all ports), but not the Internet.

Install Agilicus Connector on Host

Let us install the Agilicus connector.

# which curl && (curl -sSL agilicus.com/www/releases/secure-agent/stable/install.sh > /tmp/i.sh) || (wget -O - agilicus.com/www/releases/secure-agent/stable/install.sh > /tmp/i.sh); sh /tmp/i.sh -c RXXXXr -s jxXXXn 
(hang)

Why did this hang? As a shortcut in the command, it is using ‘agilicus.com’ and ‘http’. We can make a minor change:

$ which curl && (curl -sSL https://www.agilicus.com/www/releases/secure-agent/stable/install.sh > /tmp/i.sh) || (wget -O - agilicus.com/www/releases/secure-agent/stable/install.sh > /tmp/i.sh); sh /tmp/i.sh -c RikNorxMq33ZzNyCnpvfir -s jdy29jkn 

This now works, demonstrating that our above firewall rules prevent port 80 access.

Next Steps

You can save/restore your iptables rules with iptables-save and iptables-restore. You can install iptables-persistent package to make these load on reboot (or put the rules in e.g. /etc/rc.local).

You may consider only allowing the containers to reach specific ports, where we would run the Agilicus connector.

The final rule set we ended up with:

# iptables-save
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Apr 18 20:19:46 2024
*filter
:INPUT DROP [87:68971]
:FORWARD DROP [0:0]
:OUTPUT DROP [154:12030]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p udp -m multiport --dports 67,68,123,53 -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i docker0 -j ACCEPT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -d 34.95.12.47/32 -p tcp -m tcp --dport 443 -j ACCEPT
-A OUTPUT -d 35.203.36.11/32 -p tcp -m tcp --dport 443 -j ACCEPT
-A OUTPUT -p udp -m multiport --sports 67,68,123,53 -j ACCEPT
-A OUTPUT -p udp -m multiport --dports 67,68,123,53 -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Thu Apr 18 20:19:46 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Thu Apr 18 20:19:46 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Thu Apr 18 20:19:46 2024