Logging real remote address with Nginx and Lua

A common pattern in the Cloud world is load-balancers. Your environment might have Kubernetes with Istio feeding a set of Pods that each have a web server in them, often based on Nginx. In this environment, the web server (Nginx) receives a header X-Forwarded-For which is trusted by you since it is your front-end load-balancer. The syntax is

X-Forwarded-For: <client>, <proxy1>, <proxy2>

However, you may have tools that parse your access logs and assume the remote address is Client, when in fact your are logging Proxy2. How can you fix this?

Well, one approach is to use something like Proxy Protocol and make the IP transparent all the way through. Our team contributed this to Envoy. Another approach is to take the real client IP from the X-Forwarded-For and place it in the log. For this you might use the Nginx HTTP RealIP Module, but it has the side affects of changing the client address, not just logging it.

Others may be using OpenResty which does not support the RealIP module. In this post I show you a happy medium: log (in JSON format for easy consumption into FluentBit and ElasticSearch/Kibana) such that the downstream_remote_address is filled in with the real remote IP. This does not have any other side affects other than the access log. If you paste these snippets into your nginx config in your various and sundry containers that run inside your cluster, it will just work.

The key blob is the set_by_lua_block. We introduce a new variable, $real_remote. For every request, if X-Forwarded-For is set, we take the first component of it and use. Else we return the original remote_addr.

Enjoy! If you wish to expose a service from inside your facility, and still want to know the origin IP, this can be for you.

http {
 ...
  log_format json escape=json '{'
    '"time": "$time_iso8601",'
    '"downstream_remote_address": "$real_remote",'
    '"x_forward_for": "$proxy_add_x_forwarded_for",'
    '"request_id": "$request_id",'
    '"remote_user": "$remote_user",'
    '"bytes_sent": $bytes_sent,'
    '"start_time": $request_time,'
    '"response_code": $status,'
    '"authority": "$host",'
    '"protocol": "$server_protocol",'
    '"path": "$uri",'
    '"request_query": "$args",'
    '"request_length": $request_length,'
    '"duration": $request_time,'
    '"method": "$request_method",'
    '"http_referrer": "$http_referer",'
    '"http_user_agent": "$http_user_agent"'
  '}';

 ...
    server {
      listen 5000;
  ...

      set_by_lua_block $real_remote {
        if ngx.var.http_x_forwarded_for then
          for r in ngx.var.http_x_forwarded_for:gmatch('([^,]+)') do
            return r
          end
        end
        return ngx.var.remote_addr
    }
...