Proxy Servers: Part 1 — Why They're Needed and How They Work

A proxy server is a program that accepts network requests from a client and forwards them to the target server. The response also passes through the proxy. As a result, the client and server do not connect directly, and the proxy gains the ability to inspect, modify, cache, and log traffic.

Forward, reverse, and transparent proxy

Proxies are distinguished by which side they operate on.

Forward proxy sits on the client side, for example at the gateway of an office network to the Internet. Browsers and programs are explicitly configured to use it. Target sites see the proxy’s IP address, not the employees’ addresses. Forward proxies are used for:

• access control: who is allowed to reach which resources;
• authentication: Internet access only with a username and password;
• logging: who accessed what and when;
• filtering: blocking unwanted sites and file types;
• egress from a single fixed IP address, for example so an external service allows only the company’s address.

Reverse proxy sits on the server side and accepts requests from the Internet on their behalf. A client connects to example.ru and does not know how many servers are behind the proxy or where they are located. Reverse proxies are used for:

• load balancing — distributing requests among multiple application instances;
• TLS termination — certificates and encryption are handled in one place, while applications run over plain HTTP on the internal network;
• routing — /api/ goes to one server, the rest to another; shop.example.ru and blog.example.ru are served by different applications on a single IP address;
• caching application responses;
• rate limiting and protection against password-guessing attacks;
• hiding internal infrastructure — only the proxy is exposed externally.

Transparent proxy intercepts traffic without client-side configuration: a router or firewall redirects connections to the proxy, for example with a REDIRECT or TPROXY rule in Linux. For plain HTTP this works transparently for the user. A transparent proxy cannot see HTTPS content without special configuration: it can only access the site’s name from the TLS handshake (SNI). To inspect HTTPS, the proxy must decrypt the traffic, and a root certificate of the organization must be installed on all computers. This is a complex and security-sensitive scheme and should be used only where truly necessary.

What traffic goes through a proxy

HTTP proxy and the CONNECT method

An HTTP proxy understands the HTTP protocol: it sees the address, headers, and request body, can cache responses, and filter by URL. For HTTPS, the browser uses the CONNECT method (RFC 9110): it asks the proxy to open a TCP connection to example.ru:443, after which the proxy simply forwards encrypted bytes in both directions. In this mode the proxy knows which site the connection is to, but does not see which pages are opened or what is transmitted within them.

SOCKS5

SOCKS5 (RFC 1928) operates at a lower level and does not parse the application protocol. The client tells the proxy the destination address and port, and the proxy establishes the connection on its behalf. Therefore any TCP traffic goes through SOCKS5 — mail clients, SSH, databases, messengers — and, with support for the UDP ASSOCIATE command, UDP as well. SOCKS5 supports username/password authentication (RFC 1929). It cannot cache or filter by content: it does not know what is inside the connection.

TCP proxy and TLS: termination or passthrough

A reverse proxy can handle TLS in two ways:

• termination — the proxy decrypts the traffic, sees HTTP requests and can route by path, add headers, cache. The certificate is stored on the proxy;
• passthrough — the proxy forwards the encrypted connection to the server without decrypting it. Routing can only be done by the site’s name from SNI. The certificate remains on the destination server.

Termination is more convenient and provides more capabilities. Passthrough is needed when the certificate key must not leave the application server or when the application itself validates client certificates.

How the server learns the client’s address

Behind a reverse proxy, the application sees all connections coming from the proxy’s IP address. To convey the real client’s address, the following are used:

• the X-Forwarded-For header (and X-Forwarded-Proto for the original scheme — http or https) — for HTTP;
• PROXY protocol — a short header at the start of the TCP connection added by the proxy. It is supported by HAProxy, Nginx, and many applications; it is suitable for non-HTTP traffic as well.

You should trust these values only when they come from your proxy. Otherwise a client can put any address into X-Forwarded-For. Therefore the application or the next proxy specifies a list of trusted addresses from which these headers are accepted.

How to check operation through a proxy

It’s convenient to check a proxy with the curl utility. A request through an HTTP proxy with authentication:

curl -x http://user:password@proxy.example.ru:3128 https://example.ru/

A request through SOCKS5 where the proxy resolves the site name rather than the client:

curl --socks5-hostname user:password@proxy.example.ru:1080 https://example.ru/

The -v option will show how the connection is established: for HTTPS via an HTTP proxy the output shows a CONNECT example.ru:443 request. Many programs on Linux use proxies from the environment variables http_proxy, https_proxy, and no_proxy, but this is a convention, not a rule: some applications ignore them and are configured separately.

Proxy and VPN

Proxies and VPNs solve similar problems but are structured differently:

• proxy works at the level of individual applications: only traffic from programs configured to use it goes through it;
• VPN creates a virtual network interface, and all system traffic or traffic for specified networks goes through the tunnel, including programs that know nothing about a proxy;
• encryption: a VPN encrypts traffic between the client and the VPN server; HTTP proxies and SOCKS5 do not themselves encrypt the channel to the proxy, so proxy credentials and unencrypted traffic are sent in the clear.

VPNs are typically used to give employees access to internal company resources. Proxies are chosen when you need to control Internet egress, publish services, or route traffic of specific applications through a separate address.

Caching today

A caching forward proxy was especially useful while most sites used HTTP. Now almost all traffic is encrypted, and without decryption a forward proxy cannot cache it. Caching is more useful on a reverse proxy: there the traffic is decrypted, and the owner decides which application responses can be stored and for how long.

Which solution to choose

Solutions are often combined: HAProxy accepts traffic and distributes it among several servers running Nginx, and Nginx serves static files and proxies requests to the application.

Limitations and security

• An open proxy is dangerous. A proxy without authentication and address restrictions is quickly discovered by scanners and used to send spam and attacks on your behalf. Access to a forward proxy is always restricted by an address list or password.
• A proxy is a single point of failure. If all traffic goes through it, its failure stops everything. For critical services, run two proxies and switch the address between them, for example with keepalived.
• The proxy sees the traffic. An administrator of a proxy that decrypts TLS gains access to passwords and users’ personal data. Such logs require the same protection as the data itself.
• A proxy is not a firewall. It hides addresses and filters requests at its level, but does not replace packet-filtering rules on the server and the router.