You’ve got a mixed-used Kubernetes cluster. You have a very nifty security logging system that wants to audit everything from certain namespaces, and of course you want general logs from all. How would you go about configuring Fluent-bit to route all logs to one output, and only a single namespace to another, simultaneously? Read on to learn how to perform Fluent-Bit rlog outing by namespace.
First, let’s understand the flow of information. Fluent-bit operates with a set of concepts (Input, Output, Filter, Parser). Inputs consume data from an external source, Parsers modify or enrich the log-message, Filter’s modify or enrich the overall container of the message, and Outputs write the data somewhere.
The typical flow in a Kubernetes Fluent-bit environment is to have an Input of type Tail
, which conceptually does a tail -f
on all your log files. It relies on the fact the files have a magic name (incorporating the pod/namespace/container information).
From the Input we then go through a Filter
of type Kubernetes
. This takes the input parsed from the filename (Pod/Namespace/Container), does a lookup to the Kubernetes API (e.g. to look at annotations). From here we go to the output stage.
In the example use case here, we want to send *all* records to the default Output, and *some* (e.g. all from a namespace) to a secondary Output. Let’s dig in.
[SERVICE]
Flush 1
Daemon Off
Log_Level info
Parsers_File parsers.conf
[INPUT]
Name tail
Path /var/log/containers/*log
Parser docker_no_time
Tag kube...
Tag_Regex (?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-
Refresh_Interval 5
Mem_Buf_Limit 50KB
Skip_Long_Lines On
[FILTER]
Name kubernetes
Match kube.*
Merge_Log Off
Regex_Parser kube-tag
K8S-Logging.Parser On
K8S-Logging.Exclude On
[OUTPUT]
Match *
Name stdout
Format json_lines
JSON_Date_Format iso8601
[OUTPUT]
Name file
Match kube.ns2.*
Path ns2-logs-only.txt
OK, we added two different outputs. One matches all, one only namespace ns2. This Match parameter is how we do Fluent-Bit routing by namespace.
[PARSER]
Name json
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
[PARSER]
Name kube-tag
Format regex
Regex (?[^.]+).(?[^.]+).(?[^.]+).(?[^.]+)
[PARSER]
Name docker_no_time
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep Off
Decode_Field_As escaped log
Decode_Field_As escaped stream
OK what have we achieved? Well, for namespace ns2
we route that output to a file. For all logs (including ns2
) we route that to stdout. We have done Fluent-Bit log routing by namespace in Kubernetes.
Now, replace stdout
and file
with your outputs. And profit!
You may use this tag match routing for other users. I use it with 2 different clusters, each running fluent-bit. 1 of them is configured to forward to the other, and the logs are routed by cluster source.
I also use this technique to route the Node logs to one Elasticsearch Index, and the Pod logs to another.