File Transfer Protocols: Part 8 — BitTorrent

BitTorrent — a peer-to-peer file transfer protocol created by Bram Cohen in 2001. Unlike HTTP or FTP, where everyone downloads a file from a single server, in BitTorrent each participant simultaneously downloads and uploads parts of the file to others. The more people get the file, the faster it spreads, and the load on the original server grows very little.

How a torrent is organized

A file (or a set of files) is split into pieces of a fixed size. A hash is computed for each piece, and these hashes together with the list of files are recorded in the torrent metadata — the .torrent file. A peer that has received a piece verifies it by hash and immediately starts uploading it to others. Corrupted or tampered data is discarded automatically.

Participants are called peers. A participant who has the whole file is a seed. A torrent is started by at least one seed, and the data then propagates among peers.

How peers find each other

• Tracker — a server that keeps a list of participants in the torrent and returns it to clients. The files themselves do not go through the tracker.
• DHT — a distributed hash table where the list of participants is stored by the clients themselves. A torrent continues to work even if the tracker is unavailable.
• PEX (peer exchange) — clients tell each other the addresses of other participants.
• Local discovery — clients on the same local network find each other without the internet, which is convenient for sharing inside an office.

Magnet links

Instead of a .torrent file you can provide a magnet link: it contains the torrent metadata hash (info hash). The client finds peers via DHT or the tracker, gets the metadata from them, verifies it by hash, and starts downloading. This is convenient when a torrent needs to be published as a single line.

BitTorrent v2

The BEP 52 specification describes the second version of the protocol. The main changes:

• SHA-256 instead of SHA-1 — the deprecated SHA-1 hash function is replaced with a more secure one;
• hash tree for each file — each file is described by its own Merkle tree of blocks of 16 KiB, so identical files in different torrents can be recognized and individual blocks can be verified;
• hybrid torrents — the .torrent file can contain descriptions in both v1 and v2 formats so that both old and new clients understand the torrent.

The full v2 hash is 32 bytes; for compatibility with trackers a shortened 20-byte version is used. Modern clients support downloading v2 and hybrid torrents — for example, Transmission since version 4.0 — but many torrents are still created in the v1 or hybrid format.

Where the protocol is useful in practice

• OS images. Many Linux distributions publish official torrents alongside direct links: this reduces mirror load and speeds up downloads.
• Distributing updates and large files within a company. When the same multi-gigabyte image or archive is needed on dozens of machines, BitTorrent with local discovery distributes the load among them instead of relying on a single file server.
• Archives and datasets. Scientific and public projects distribute large datasets via BitTorrent that would be expensive to serve from a single server.
• Downloading over unstable connections. Each piece is verified separately, so interruptions do not corrupt the file and downloading can continue from any point.

Security and limitations

• The protocol itself does not encrypt data. Clients support stream encryption (MSE/PE), but it only hides traffic from simple filters and does not make the exchange anonymous: participant addresses are visible to each other and to the tracker.
• Hash verification guarantees that the data matches the metadata, but not that the metadata came from a trusted source. Get .torrent files and magnet links only from official project websites.
• In many corporate networks peer-to-peer traffic is blocked by security policies. Using BitTorrent inside a company should be coordinated and restricted to the internal network.
• Distributing copyrighted material without the rightsholder’s permission is illegal — the protocol is not to blame here, but the person who distributes the material is responsible.