Sample Setup
Kafka Messaging From AWS VPC To Private Network
Kafka is a messaging bus using TCP as its underpinnings. It may be desirable to run a Kafka broker inside a private cloud environment (e.g. an AWS VPC) and connect to that broker from a remote private network. Likewise it can be desirable to run a Kafka broker inside a remote private network and connect as a client from a remote private cloud such as AWS VPC. These instructions work equally for each case.
Overview
Kafka is a messaging bus using TCP as its underpinnings. It may be desirable to run a Kafka broker inside a private cloud environment (e.g. an AWS VPC) and connect to that broker from a remote private network. Likewise it can be desirable to run a Kafka broker inside a remote private network and connect as a client from a remote private cloud such as AWS VPC. These instructions work equally for each case.
![Sample Kafka Messaging From AWS VPC To Private Network 2 c3fde201 image](https://www.agilicus.com/www/c3fde201-image-1024x287.png)
For this example, we will use two hosts. Cube exists inside a private network with a restrictive firewall. Only HTTPS/443 is allowed, and only in the outbound direction. On this host runs a Kafka broker on port 9092.
Office runs inside a Virtual Private Cloud. It two has outbound-only access: no inbound, no public IP.
For the sake of this example, we will use docker-compose to bring up the Broker on Cube. We will then use kafkacat
to send and receive traffic via the broker.
Steps (Cube):
- docker-compose up
- apt-get install kafkacat
- Install Agilicus Connector
- Create a Network Service called
kafka-cube
, on port 9092, localhost (or the IP of the docker container if you prefer). - Run
kafkacat -C -b localhost:9092 -t test
which will sit and listen for messages
Steps (Office):
- Install Agilicus Connector
- Create Service Forwarder, source-connector == Office, Destination-network == kafka-cube, source-ip localhost, source-port 9092
- Assign permissions to Service Forwarder to office connector service-account
- apt-get install kafkacat
- run
echo foo | kafkacat -P -b localhost:9092 -t test
At this stage you should observe the kafkacat on cube
wake up and say foo
. We are done.
You may now wish to try e.g. change the server-forwarder to listen on 127.0.10.1:9092, THen we can run echo foo | kafkacat -P -b 127.0.10.1:9092 -t test
. We can then add an entry to /etc/hosts and call it ‘Cube’ and repeat with a hostname.
docker-compose.yml
version: '2'
services:
zookeeper:
build: ./zookeeper
ports:
- "2181:2181"
kafka:
build: ./kafka
ports:
- "9092:9092"
zookeeper/Dockerfile
FROM ubuntu:22.04
EXPOSE 2181
RUN : \
&& apt-get update \
&& apt-get install -y wget vim net-tools default-jre curl
WORKDIR /opt
RUN : \
&& curl -sSL https://dlcdn.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz > /tmp/apache-zookeeper-3.8.0-bin.tar.gz \
&& tar xvzf /tmp/apache-zookeeper-3.8.0-bin.tar.gz \
&& rm -f /tmp/apa*gz \
&& cp apache-zookeeper-3.8.0-bin/conf/zoo_sample.cfg apache-zookeeper-3.8.0-bin/conf/zoo.cfg
ENTRYPOINT apache-zookeeper-3.8.0-bin/bin/zkServer.sh start-foreground
kafka/Dockerfile
FROM ubuntu:22.04
EXPOSE 9020
RUN : \
&& apt-get update \
&& apt-get install -y wget vim net-tools default-jre curl
WORKDIR /opt
RUN : \
&& curl -sSL https://downloads.apache.org/kafka/3.2.1/kafka_2.12-3.2.1.tgz > /tmp/kafka_2.12-3.2.1.tgz \
&& tar xvzf /tmp/kafka_2.12-3.2.1.tgz \
&& rm -f /tmp/*tgz
COPY server.properties /opt/kafka_2.12-3.2.1/config/server.properties
ENTRYPOINT /opt/kafka_2.12-3.2.1/bin/kafka-server-start.sh /opt/kafka_2.12-3.2.1/config/server.properties
kafka/server.properties:
broker.id=0
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=zookeeper:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
offsets.topic.replication.factor=1
ssl.endpoint.identification.algorithm=
advertised.listeners=PLAINTEXT://localhost:9092
listeners=PLAINTEXT://0.0.0.0:9092