HTTP, HTTPS and TLS: Part 9 — Debugging HTTP with curl: DNS spoofing, redirects, timings and tracing

curl is a command-line HTTP client available on Linux, macOS and Windows 10 and newer. For debugging it is more convenient than a browser: it shows the full request and response, allows you to set any header and method, send a request to a specific IP bypassing DNS, and measure which stage takes time. Below are debugging techniques: from viewing headers to diagnosing a slow site behind a load balancer. If you are just starting with curl, start with the article “curl for beginners”: it covers the main flags, API requests and file uploads.

The curl version on the system may be old; some flags appeared recently (each shows the version). Your version and build features are shown by curl --version.

View response and headers

curl https://example.ru/                 # only response body
curl -i https://example.ru/              # response headers + body
curl -I https://example.ru/              # only headers (HEAD method)
curl -v https://example.ru/              # everything: DNS, TCP, TLS, request, response
curl -sv -o /dev/null https://example.ru/ 2>&1   # everything except the body
curl -s -D - -o /dev/null https://example.ru/    # only headers, but using GET

What the flags mean:

• -v (verbose) — detailed output to stderr: lines starting with * are informational messages (address, TLS, certificate), > — sent request, < — received headers;
• -s — no progress meter; together with -S errors are still shown;
• -o /dev/null — discard the body; -o file — save to file, -O — save using the name from the URL;
• -D - — write response headers to stdout.

Be careful with -I. It sends HEAD, not GET. Some applications respond to HEAD differently — 405, 404 or without needed headers. If the response to -I looks strange, check the same URL with -s -D - -o /dev/null.

Send headers

A header is set with -H, and the flag can be repeated:

curl -H 'Accept: application/json' \
     -H 'Authorization: Bearer eyJhbGciOi...' \
     -H 'X-Request-ID: debug-42' \
     https://api.example.ru/v1/orders

Special forms:

curl -H 'User-Agent:' https://example.ru/      # remove the header that curl adds by default
curl -H 'X-Empty;' https://example.ru/         # send a header with an empty value
curl -H @headers.txt https://example.ru/       # read headers from a file, one per line (7.55+)

There are separate flags for common headers:

curl -A 'Mozilla/5.0 ...' https://example.ru/     # User-Agent
curl -e https://example.ru/ https://example.ru/x  # Referer
curl -u admin:secret https://example.ru/admin/    # Basic authentication
curl --oauth2-bearer TOKEN https://api.example.ru/ # Authorization: Bearer
curl --compressed https://example.ru/             # requests Accept-Encoding and decompresses response

You shouldn’t set -H 'Accept-Encoding: gzip' manually: curl will get the compressed response and print it to the terminal as-is. --compressed both requests compression and decompresses.

Passwords and tokens on the command line are visible in ps and remain in shell history. In scripts pass them from a file: -H @auth-header.txt, or use ~/.netrc with --netrc.

Cookie

curl -b 'session=abc123; lang=ru' https://example.ru/account   # send cookies
curl -c cookies.txt https://example.ru/login                   # save received cookies
curl -b cookies.txt -c cookies.txt https://example.ru/account  # read and update cookie file

A typical scenario — log in via a form and then continue with the received session:

curl -c jar.txt -d 'login=ivanov&password=secret' https://example.ru/login
curl -b jar.txt https://example.ru/account

Methods and request body

curl chooses the method itself: no body — GET, with -d — POST, with -I — HEAD. You can explicitly set the method with -X:

curl -X DELETE https://api.example.ru/v1/orders/42
curl -X PUT -H 'Content-Type: application/json' -d '{"status":"paid"}' https://api.example.ru/v1/orders/42

-X POST together with -d is not necessary — POST will be used anyway. And -X together with -L is dangerous: curl will use the specified method for all requests after redirects, even where the standard would switch to GET.

Form data (application/x-www-form-urlencoded):

curl -d 'name=Ivan&city=Moscow' https://example.ru/form
curl --data-urlencode 'comment=Price 100% & delivery' https://example.ru/form   # with encoding

-d sends data as-is and strips newlines from files (-d @file). For files without changes — --data-binary @file, for strings starting with @ — --data-raw.

JSON:

curl --json '{"name":"Ivan","qty":2}' https://api.example.ru/v1/orders    # curl 7.82+
curl --json @order.json https://api.example.ru/v1/orders

--json sets Content-Type: application/json and Accept: application/json automatically. In older versions the same can be done as:

curl -H 'Content-Type: application/json' -d '{"name":"Ivan"}' https://api.example.ru/v1/orders

File upload (multipart/form-data, like a form with a “file” field):

curl -F 'title=Report' -F 'file=@report.pdf;type=application/pdf' https://example.ru/upload

GET parameters from data: -G turns -d into query string:

curl -G -d 'q=nginx' -d 'page=2' https://example.ru/search   # GET /search?q=nginx&page=2

Redirects

By default curl does not follow redirects and shows the 301/302 response itself. To follow the whole chain:

curl -sIL https://example.ru/old-page | grep -iE '^(HTTP|location)'

HTTP/1.1 301 Moved Permanently
Location: https://example.ru/old-page
HTTP/2 301
location: https://www.example.ru/new-page
HTTP/2 200

• -L — follow redirects;
• --max-redirs 5 — limit the number of redirects (default 50);
• -w '%{num_redirects} %{url_effective}\n' — how many redirects and where it ended.

When redirecting to another host curl does not forward the Authorization header and -u credentials — this protects against leaking them. If you really need this, use --location-trusted.

Request to a specific server bypassing DNS

This is most useful when migrating, configuring a load balancer, or checking one node out of several.

--resolve — substitute an IP address for a name:

curl --resolve example.ru:443:192.0.2.10 https://example.ru/

curl will connect to 192.0.2.10, but Host, SNI and certificate verification will be for example.ru — exactly like a browser after changing DNS. This is used to check the new server before switching the DNS record.

--connect-to — replace the host and port for the connection:

curl --connect-to example.ru:443:backend-2.internal:8443 https://example.ru/

Useful to bypass the load balancer and go directly to one server.

Why not -H 'Host: …'. The command curl -H 'Host: example.ru' https://192.0.2.10/ changes only the Host header, while SNI and certificate verification will be for 192.0.2.10: the server will return the default certificate, and curl will report a name mismatch error. For HTTPS use --resolve or --connect-to. For HTTP -H 'Host: …' works.

Timing

The -w (write-out) flag prints variables after the request. Breakdown by stages:

curl -s -o /dev/null -w '
dns:       %{time_namelookup}
tcp:       %{time_connect}
tls:       %{time_appconnect}
ttfb:      %{time_starttransfer}
total:     %{time_total}
code:      %{http_code}
http:      %{http_version}
ip:        %{remote_ip}
size:      %{size_download}
' https://example.ru/

All times are in seconds from the start of the request, cumulative:

• time_namelookup — DNS; more than 0.1s — problem with the resolver;
• time_connect − time_namelookup — TCP setup, roughly one RTT to the server;
• time_appconnect − time_connect — TLS handshake;
• time_starttransfer − time_appconnect — time while the server prepared the response (TTFB excluding network). If large — the problem is in the application or database, not the network;
• time_total − time_starttransfer — body download.

Keep the template in a file and call -w @curl-format.txt. Since curl 7.70 there is -w '%{json}' — all variables at once in JSON, and since 7.83 — %{header_json} for response headers.

Check stability with a series of requests:

for i in $(seq 1 20); do
  curl -s -o /dev/null -w '%{http_code} %{time_total}\n' https://example.ru/
done

Choose version and protocol

curl --http1.1 https://example.ru/
curl --http2 https://example.ru/
curl --http3 https://example.ru/          # if the build supports HTTP/3
curl -4 https://example.ru/               # IPv4 only
curl -6 https://example.ru/               # IPv6 only

If a site works over IPv4 but fails for some users, check -6: a common cause is an AAAA record pointing to an address where the site is not listening.

Proxy

curl -x http://proxy.example.ru:3128 https://example.ru/
curl -x socks5h://127.0.0.1:1080 https://example.ru/    # h — DNS also goes through the proxy
curl --noproxy '*' https://example.ru/                  # ignore environment variables

curl respects the http_proxy, https_proxy, no_proxy environment variables. If a request works from one environment but not another, check env | grep -i proxy — a proxy might have been picked up unnoticed.

Return codes and scripts

By default curl exits successfully (code 0) and considers only connection failures as errors; responses like 404 or 500 don’t make it fail. For scripts and checks:

curl -fsS https://example.ru/health || echo 'service is unavailable'
curl --fail-with-body -sS https://api.example.ru/v1/check   # 7.76+: error code and response body

• -f (--fail) — on 400+ responses the exit code is 22;
• --retry 3 --retry-delay 2 — retry on transient errors (timeout, 408, 429, 5xx);
• --connect-timeout 5 --max-time 20 — limit connection time and total request time. Without them curl can wait for a hung server for a very long time.

Common curl error codes: 6 — could not resolve host; 7 — could not connect; 28 — timeout; 35 — TLS handshake error; 60 — certificate verification failed.

Full trace

When -v is not enough — for example, you need to see exact bytes of the request body or the order of events:

curl --trace-ascii trace.txt --trace-time https://example.ru/

The file contains each sent and received data block with a timestamp. --trace writes the same with a hex dump.

Copy request from browser

In developer tools (F12) on the Network tab right-click the request → Copy → Copy as cURL. You’ll get a command with all the browser headers and cookies. This is the fastest way to repeat in the console a request that behaves oddly, and then remove headers one by one to find the cause.

Note: such a request will reproduce headers, but not the browser’s TLS fingerprint. If a site behind bot protection returns 200 to the browser but 403 to the same request from curl, the issue is likely the fingerprint.

Typical mistakes

• Using -I to check an application that doesn’t handle HEAD — false 404 and 405.
• Using -H 'Host: …' for HTTPS instead of --resolve — certificate error and wrong default site.
• Using -k in health checks — an expired or wrong certificate goes unnoticed.
• Checking availability without -f — the script treats 502 as success.
• Sending JSON with -d without Content-Type — curl sends application/x-www-form-urlencoded, and the application doesn’t see the data.
• No --max-time in cron jobs — hung curl processes accumulate.