HTTP, HTTPS and TLS: Part 2 — HTTP Headers: types and functions

Headers are the metadata of an HTTP message: lines like Name: value between the start-line and the body. From them the server understands which site is requested, in what format and language to return the response, who is in front of it, and whether the client has a fresh copy. The browser uses response headers to decide how to display content, whether it can be cached, and which security restrictions apply to the page.

Below are headers you encounter frequently, grouped. You can view them for any site with:

curl -sI https://example.ru/          # response headers only (HEAD method)
curl -sv -o /dev/null https://example.ru/ 2>&1 | grep -E '^[<>]'   # request and response

How headers are structured

• Names are case-insensitive. In HTTP/2 and HTTP/3 they are always lowercase, and in logs you will see content-type, not Content-Type.
• A header may appear multiple times; values are then combined with commas. The exception is Set-Cookie: each cookie goes on a separate line.
• The X- prefix once indicated non-standard headers. RFC 6648 (2012) recommends avoiding it, but old names like X-Forwarded-For remain in use.
• The size of headers is limited by the server. In Nginx the buffer for the request line and headers is set by large_client_header_buffers (by default 4 buffers of 8 KB each); if cookies grow too large, the server will respond with 400 Request Header Or Cookie Too Large.

Request headers

Host — the site name and port, if non-standard. Hundreds of sites may live on one IP, and the web server selects the correct one by Host (in Nginx — by server_name). This is the only required header in HTTP/1.1. In HTTP/2 and HTTP/3 its role is played by the pseudo-header :authority.

User-Agent — who makes the request: the browser and its version, curl/8.7.1, python-requests/2.32. Servers use it for statistics and sometimes to block bots. The value is filled in by the client, so anyone can fake it; more reliable client fingerprints are TLS fingerprints — see “What are JA3 and JA4”.

Accept, Accept-Language, Accept-Encoding — content negotiation. The client lists what it can accept, with weights q:

Accept: text/html,application/xhtml+xml,*/*;q=0.8
Accept-Language: ru-RU,ru;q=0.9,en;q=0.8
Accept-Encoding: gzip, deflate, br, zstd

The server chooses a variant and indicates the choice in Content-Type, Content-Language and Content-Encoding. If the response depends on one of these headers, the server should add Vary: Accept-Encoding (or another name), otherwise proxies and CDNs may serve a compressed version to a client that doesn’t understand it.

Authorization — credentials. Basic — username and password in Base64 (this is an encoding, not encryption: without HTTPS the password is readable as-is). Bearer — a token, for example a JWT or an API key.

Cookie — cookies the browser saved for this site: Cookie: session=abc123; lang=ru.

Referer — the address of the page the user came from (the word was misspelled in the very first specification and it stayed that way). How much of it is sent is determined by the response header Referrer-Policy; by default browsers send only the domain to other sites.

Origin — the scheme, host and port of the page that sent the request. The browser adds it to cross-site requests and to POST; CORS and CSRF checks are based on it.

If-None-Match, If-Modified-Since — conditional requests. The browser tells which version it already has, and the server responds with a short 304 Not Modified without a body if nothing changed.

Range — request for part of a file: Range: bytes=1000000-. Resuming downloads and video seeking work on this.

Content-Type, Content-Length — the type and length of the request body. For forms — application/x-www-form-urlencoded or multipart/form-data, for APIs — usually application/json. A common mistake: sending JSON without Content-Type: application/json, so the application doesn’t see the data.

Response headers related to content

Content-Type — what is in the body and in which charset: text/html; charset=utf-8, application/json, image/webp. The browser decides how to render the response based on it. To prevent the browser from trying to guess the type itself (which used to lead to user-uploaded files being executed as scripts), add X-Content-Type-Options: nosniff.

Content-Encoding — how the body is compressed: gzip, br (Brotli), zstd.

Content-Disposition — show the file in the browser (inline) or prompt download (attachment; filename="report.pdf").

Location — redirect address in 3xx responses and the address of the created resource in 201 Created.

Server — the web server software. It’s better not to expose the version: in Nginx — server_tokens off;.

Caching

There is caching in the browser, CDN and proxy. The server sets rules for them.

Cache-Control — the main caching header:

• max-age=3600 — the response is fresh for one hour; during that time the browser takes it from cache without asking the server;
• s-maxage=600 — the same for shared caches (CDN, proxy), this takes precedence over max-age;
• no-cache — storing is allowed, but before each use revalidate with the server (via If-None-Match);
• no-store — do not store at all; for pages with personal data;
• private — only the browser should store it, CDN should not; for pages that depend on the user;
• public — can be stored in shared caches;
• immutable — the resource will never change; set on files with a hash in the name (app.3f9a1c.js) together with a large max-age.

The name no-cache is confusing: it does not forbid caching. no-store forbids storage.

ETag — the resource version identifier (for example, a hash of the content). The browser returns it in If-None-Match. Last-Modified — the modification date, returned in If-Modified-Since.

Age — how many seconds the response sat in a CDN or proxy cache. If Age is present, the response did not come from your server. CDNs often add their own headers: CF-Cache-Status: HIT at Cloudflare, X-Cache: HIT at many others.

Cookies

The server sets cookies with the Set-Cookie header, one per line:

Set-Cookie: session=abc123; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age=86400

• Secure — send only over HTTPS;
• HttpOnly — inaccessible from JavaScript, so an injected script cannot steal it;
• SameSite=Lax — don’t send on cross-site requests, except for regular navigation by link; Strict — never send on cross-site requests; None — always (only together with Secure). Modern browsers treat cookies as Lax if not explicitly specified;
• Max-Age or Expires — lifetime; without them the cookie is deleted when the browser is closed;
• Domain — if set, the cookie is sent to subdomains as well. Without it — only to the host that set it.

For a session cookie, a reasonable minimum is Secure; HttpOnly; SameSite=Lax.

CORS

The browser does not allow JavaScript from https://app.example.ru to read responses from https://api.example.ru unless the API server explicitly allowed it. This is the same-origin policy, and CORS is the permission mechanism:

Access-Control-Allow-Origin: https://app.example.ru
Access-Control-Allow-Credentials: true

For “non-simple” requests — with methods PUT and DELETE, with Content-Type: application/json, or with the Authorization header — the browser first sends a preflight OPTIONS request with the headers Access-Control-Request-Method and Access-Control-Request-Headers. The server replies which methods and headers are allowed (Access-Control-Allow-Methods, Access-Control-Allow-Headers, Access-Control-Max-Age), and only then the real request is sent.

Important details:

• CORS protects the browser user, not the server. curl and any scripts outside the browser do not enforce CORS;
• Access-Control-Allow-Origin: * does not work together with cookies and Authorization — for credentialed requests a specific origin is required;
• if a CORS error is visible in the console, first check the OPTIONS response: often it’s blocked by authentication or the server responds with 404/405.

Security headers

• Strict-Transport-Security (HSTS) — “only use HTTPS for this site”: max-age=31536000; includeSubDomains. More — “Moving to HTTPS: redirects, HSTS, HTTPS-First”.
• Content-Security-Policy — from where the page is allowed to load scripts, styles, images, and where to send forms. The main protection against XSS. Example: default-src 'self'; img-src 'self' data:; frame-ancestors 'none'. It’s convenient to start with Content-Security-Policy-Report-Only: the browser reports violations to the console but doesn’t block anything.
• X-Frame-Options: DENY or the CSP directive frame-ancestors — prevent embedding the site in a frame on foreign pages (protection against clickjacking).
• X-Content-Type-Options: nosniff — do not guess content type.
• Referrer-Policy: strict-origin-when-cross-origin — do not give other sites the full page URL.
• Permissions-Policy — which browser capabilities are available to the page: camera=(), microphone=(), geolocation=().

The header X-XSS-Protection is obsolete: modern browsers ignore it; CSP is used instead.

In Nginx response headers are added with the add_header directive, and it has a trap: if a location block contains even one add_header, headers from the server level do not apply in it. The always parameter is needed so the header is sent in error responses as well:

server {
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
}

Proxy headers

When Nginx, HAProxy or a CDN sits in front of the application, the application sees the connection from the proxy, not the client. The proxy forwards original data in headers:

• X-Forwarded-For: 192.0.2.7, 192.0.2.2 — chain of addresses: the client and all proxies along the way;
• X-Forwarded-Proto: https — which protocol the client used; without it an application behind a proxy that terminates HTTPS may think it’s running on HTTP and build incorrect links or get into an infinite HTTPS redirect;
• X-Forwarded-Host — the original Host;
• X-Real-IP — the client address as a single value (Nginx convention);
• Forwarded: for=192.0.2.7;proto=https — the standard replacement for all of the above (RFC 7239), supported less often.

You can trust these headers only if your proxy set them. A client can send X-Forwarded-For itself, and an application that takes the first address from the list will receive a forgery. Rule: take the address added by the last trusted proxy; in Nginx specify trusted networks explicitly:

set_real_ip_from 10.0.0.0/8;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

Connection-related headers

Connection, Keep-Alive, Transfer-Encoding, Upgrade relate to a single hop — between the client and the nearest proxy — and are not forwarded further. They are forbidden in HTTP/2 and HTTP/3. Therefore, to proxy WebSocket through Nginx, Upgrade and Connection must be forwarded to the application explicitly — example in the article “Moving to HTTPS and the Upgrade header”.

Common mistakes

• Cache-Control not set — browsers and CDNs cache heuristically (often 10% of the age from Last-Modified), and users see old styles after a deploy.
• no-cache instead of no-store on pages with personal data.
• The response depends on cookies or language, but Vary is not set and there’s no private — the CDN serves one user’s page to another.
• Access-Control-Allow-Origin taken from the request’s Origin without checking against a list — it’s the same as allowing everyone, but with cookies.
• The application trusts X-Forwarded-For from any client — IP-based limits are bypassed and logs are forged.