Proxy Servers: Part 2 — Nginx

Nginx — an open-source (BSD-licensed) web server and reverse proxy. Since 2019 the project has been owned by F5. Besides the free version there is a commercial Nginx Plus. Nginx serves static files, accepts HTTPS, proxies requests to applications and distributes load between multiple servers. This part covers its operation as a reverse proxy.

What the free nginx can do

• HTTP proxy — forwards requests to applications over HTTP, FastCGI (PHP-FPM), uwsgi, gRPC.
• TLS termination — accepts HTTPS and HTTP/2, and in newer versions HTTP/3.
• Load balancing — algorithms round-robin (default), least_conn, ip_hash, hash, with server weights.
• Passive health checks — if a server fails to respond max_fails times within fail_timeout, nginx temporarily stops sending requests to it.
• Caching of responses on disk (proxy_cache).
• Rate limiting (limit_req) and connection limiting (limit_conn).
• WebSocket — proxied since version 1.3.13.
• TCP and UDP — the stream module, available since 1.9.0. It is included in the free version; when building from source include --with-stream, in official nginx.org packages it is already built in, in Debian and Ubuntu it is provided by the libnginx-mod-stream package.

Nginx Plus adds active health checks (nginx itself periodically queries servers for status), changing the set of servers via an API without reloads, extended statistics and commercial support. Active health checks are not available in the free version — this is one reason to choose HAProxy if load balancing is the primary concern.

Example: site with HTTPS and WebSocket

The application listens on port 3000 on the same server, the certificate is issued by Let’s Encrypt. The configuration is placed in /etc/nginx/conf.d/example.ru.conf:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

upstream app {
    server 127.0.0.1:3000;
}

server {
    listen 80;
    server_name example.ru;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    http2 on;
    server_name example.ru;

    ssl_certificate     /etc/letsencrypt/live/example.ru/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.ru/privkey.pem;

    client_max_body_size 20m;

    location / {
        proxy_pass http://app;
        proxy_http_version 1.1;

        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_set_header Upgrade    $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_read_timeout 1h;
    }
}

What’s important here:

• the map block selects the value of the Connection header: upgrade for WebSocket and close for ordinary requests. The Upgrade and Connection headers are not forwarded to the application automatically — they must be set explicitly;
• the directive http2 on appeared in version 1.25.1; in older versions HTTP/2 is enabled with listen 443 ssl http2;
• proxy_http_version 1.1 is needed for WebSocket in nginx versions prior to 1.29.7, in which HTTP/1.0 is used by default when talking to the application;
• proxy_read_timeout 1h prevents nginx from closing the WebSocket connection: by default it is dropped if the application sends nothing for 60 seconds.

Before applying, check the configuration and then reload without stopping the server:

nginx -t && systemctl reload nginx

Load balancing between multiple servers

upstream app {
    least_conn;
    server 10.0.0.11:3000 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:3000 max_fails=3 fail_timeout=30s;
    server 10.0.0.13:3000 backup;
}

least_conn sends the request to the server with the fewest active connections. A server with three failures within 30 seconds is excluded for the same 30 seconds. A backup server receives requests only when the primary ones are unavailable. If the application stores sessions in process memory, a user should be pinned to a single server (ip_hash or hash $cookie_...), or better move sessions to shared storage, for example Redis.

Rate limiting

# in http context
limit_req_zone $binary_remote_addr zone=login:10m rate=5r/m;

# in server
location /login {
    limit_req zone=login burst=5 nodelay;
    limit_req_status 429;
    proxy_pass http://app;
}

From a single IP address five requests per minute are allowed to the login page with a short additional burst of five. Others receive a 429 response. This rule significantly hinders password brute-forcing.

Caching application responses

If an application returns identical responses to different visitors, for example a catalog or a public API, nginx can store them on disk and avoid contacting the application on every request:

# in http context
proxy_cache_path /var/cache/nginx/app levels=1:2 keys_zone=app_cache:10m
                 max_size=1g inactive=60m use_temp_path=off;

# in server
location /catalog/ {
    proxy_cache app_cache;
    proxy_cache_valid 200 301 10m;
    proxy_cache_valid 404 1m;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    add_header X-Cache-Status $upstream_cache_status;
    proxy_pass http://app;
}

proxy_cache_use_stale allows serving a stale copy while the application is unavailable or updating the response. The X-Cache-Status header shows whether the response came from cache (HIT) or from the application (MISS). By default nginx does not cache responses where the application sets cookies or forbids caching with Cache-Control headers. This protects against a situation where one user’s personal page gets served to another. Disable these checks only if you are completely sure the response contains no personal data.

Minimal protection

• server_tokens off; removes the nginx version number from headers and error pages;
• access to administrative addresses is restricted with allow/deny directives or a password (auth_basic);
• it’s better not to serve unknown domains with your site by default: a separate server block with listen 443 ssl default_server and return 444; closes the connection for requests with foreign or empty names. Such a block still needs a certificate, for example self-signed, or the directive ssl_reject_handshake on; (since 1.19.4).

TCP proxy: the stream module

The stream block is placed at the top level of nginx.conf, alongside the http block, not inside it. Example — access to PostgreSQL with a standby server:

stream {
    upstream postgres {
        server 10.0.0.21:5432;
        server 10.0.0.22:5432 backup;
    }

    server {
        listen 5432;
        proxy_pass postgres;
        proxy_connect_timeout 5s;
        proxy_timeout 10m;
    }
}

UDP is proxied in the same way; add the udp parameter to the listen directive. The ssl_preread module allows in stream mode to read the site name from the TLS handshake and route the connection to the correct server without decrypting it. This way you can serve several services with their own certificates on a single port 443.

Common mistakes

• Host not forwarded. By default nginx sets the Host header to the name from proxy_pass (for example, app), and an application hosting multiple domains responds with the wrong site. You need proxy_set_header Host $host.
• X-Forwarded-Proto not forwarded. The application thinks it was opened over HTTP and endlessly redirects to HTTPS or generates links with http://.
• Trailing slash in proxy_pass. proxy_pass http://app/; in location /api/ strips /api/ from the request path, while proxy_pass http://app; forwards the path intact. This is a frequent cause of 404s after configuration.
• Buffering of streaming responses. By default nginx buffers the application’s response. For streaming (Server-Sent Events, progressive output from neural networks) disable buffering: proxy_buffering off; in the relevant location or by the application sending the header X-Accel-Buffering: no.
• Error 413. By default the request body size is limited to 1 MB and file uploads are interrupted. The limit is set by client_max_body_size.
• Error 504. The application responds longer than 60 seconds — the default proxy_read_timeout. Increase the time for the specific location, not for the entire site.
• Incorrect client address behind a second proxy. If another proxy or CDN sits in front of nginx, restore the real address with the realip module: set_real_ip_from with the addresses of that proxy and real_ip_header X-Forwarded-For.

Forks: freenginx and Angie

• freenginx — a fork founded in February 2024 by Maxim Dunin, one of the main nginx developers, after disagreements with F5 about security policy. The project aims to maintain full compatibility with nginx.
• Angie — a fork created by former nginx developers; the codebase is separated from nginx 1.23.1, BSD licensed. The developer is the Russian company “Web-Server”. Angie adds built-in statistics, active health checks and other features not present in free nginx. The commercial Angie PRO is included in the Russian software registry.

Nginx configurations are in most cases portable to both forks without changes.

When to choose nginx

Nginx is a good default choice when you need to publish a site or application: it simultaneously serves static files, accepts HTTPS and proxies requests, and there are configuration examples for almost any software. If the main task is load balancing between many servers with active health checks, detailed statistics and fine-grained TCP rules, HAProxy is more convenient.