File Transfer Protocols: Part 5 — S3 and Object Storage

S3 (Simple Storage Service) — Amazon’s object storage service, launched in 2006. Its API has over time become an industry standard: today “S3” often refers not to a specific Amazon service but to the protocol supported by dozens of storages, both cloud and on-premises. Backup programs, CMSs, log collection systems and analytics platforms can work with S3 “out of the box”, so object storage has become a convenient common denominator for storing large volumes of data.

Objects instead of files

A classic filesystem is a tree of directories where a file can be opened, appended to, or modified in the middle. Object storage is simpler:

• bucket — a top-level container that holds objects;
• object — the full data plus metadata (type, date, arbitrary user fields);
• key — the name of the object inside the bucket, for example backups/2026/09/db.sql.gz.

There are actually no directories in S3: backups/2026/09/ is just part of the name. Interfaces show “folders” by grouping keys by the / delimiter. An object cannot be modified partially — it is overwritten entirely. Large files are uploaded in parts (multipart upload), and the storage stitches them together on its side.

How the exchange looks over HTTP

Each operation is an ordinary HTTP request:

• PUT /bucket/key — upload an object;
• GET /bucket/key — retrieve an object;
• DELETE /bucket/key — delete;
• GET /bucket?list-type=2&prefix=backups/ — get a list of objects (ListObjectsV2 operation, up to 1000 keys per request, then via a continuation token).

There is no separate “LIST” method in the protocol: listing is a GET to the bucket with parameters.

Each request is signed. The client receives a pair of keys — an identifier (access key) and a secret (secret key) — and computes a signature over the method, path, headers and request time using the Signature Version 4 algorithm. The secret is not sent over the network, and a captured request cannot be replayed later. No one computes the signature manually: SDKs, aws cli, rclone and other clients do it.

Working with S3-compatible storage

Most clients can work not only with Amazon but with any compatible storage — just specify the service address (endpoint).

Using aws cli:

export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret

aws --endpoint-url https://s3.example.ru s3 cp db.sql.gz s3://backups/2026/09/db.sql.gz
aws --endpoint-url https://s3.example.ru s3 ls s3://backups/2026/09/

Using rclone, which is convenient for syncing directories:

rclone config create s3backup s3 provider=Other \
  access_key_id=your-key secret_access_key=your-secret \
  endpoint=https://s3.example.ru

rclone sync /var/backups s3backup:backups/server1 --dry-run

The --dry-run flag shows what will be done without changing anything. After verifying the result, run the command without it.

Where to get S3 in Russia

Amazon S3 and Google Cloud Storage are not available to Russian companies: payments with Russian bank cards don’t go through, and Google Cloud has not accepted new customers from Russia since 2022. Compatible storages are offered by Russian clouds — Yandex Object Storage, VK Cloud, Selectel. Their pricing, storage classes and payment procedures are covered in the cloud backup article: Backup: Part 3 — Cloud Storage for Small Businesses (Backblaze and Russian Services). For storing personal data, a Russian cloud is also convenient from the perspective of Federal Law 152-FZ.

Running your own S3 storage

For a long time the default choice for a self-hosted S3 was MinIO. In 2025–2026 the situation changed: the free-edition repository of MinIO was archived on April 25, 2026; it explicitly states that the project is no longer maintained and prebuilt binaries are not released — only source code under AGPLv3. The developers offer commercial products AIStor. Existing installations continue to function, but there will be no security fixes for them, and it is wiser to build new projects on a different solution.

Open alternatives:

• Garage — a lightweight distributed storage (AGPLv3), designed for small clusters of heterogeneous machines across different sites; 1 GB of memory is enough for a node.
• SeaweedFS — a distributed filesystem with an S3 gateway (Apache 2.0); supports versioning, object locking and lifecycle rules.

Before choosing, check which S3 operations your applications need: not all storages are fully compatible, especially regarding access policies and object locking.

When S3 is suitable and when it is not

S3 is good for backups, archives, website media files, logs and analytics data — anything that is written as a whole and read by key. It is poorly suited for tasks where a file is constantly modified in small parts: databases, collaborative documents, directories with many small files that an application expects to see as a regular filesystem. For those tasks, SMB, NFS or WebDAV are better suited.

Common mistakes

• Public bucket. A bucket with open read access is a common cause of leaks. Access should be closed by default, and public files are better served via a separate bucket or a CDN.
• One key for everything. A key with full permissions embedded on all servers, if leaked, exposes the entire storage. Create separate keys with minimal permissions for each task.
• Backups without protection against deletion. If the server with the keys is compromised, the attacker will delete the backups too. Enable versioning or Object Lock if the storage supports them.
• Unbounded growth. Without lifecycle rules, old versions and incomplete multipart uploads accumulate and increase the bill.