How to cheaply collect logs from remote clients over HTTP

If your application runs on hundreds of client devices or you have a fleet of sensors with telemetry, sooner or later you’ll need to see what’s happening on the ground. Commercial systems like Splunk or Datadog are overkill and expensive for this, and if clients can send HTTP requests, most of the job is already solved: HTTPS passes through almost any firewall, and you only need a server that will accept logs.

Below are three inexpensive ways to organize log collection over HTTP: from a simple script to a serverless option. General principles of centralized logging are discussed in the article “Centralized logging: Part 1 — Why collect logs in one place”.

Why HTTP

• Universality — an HTTP client exists for every programming language and platform.
• Simplicity — a POST request with a JSON body is easy to form and debug.
• Availability — port 443 is open almost everywhere, unlike syslog or GELF ports.

Use only HTTPS. A free Let’s Encrypt certificate can be set up in a few minutes and protects logs from interception; see the article “Beyond Let’s Encrypt” about other free certificate authorities.

Option 1. Script on a VPS

The simplest method, which deploys fastest. Suitable for small projects and a small number of clients.

What you’ll need

• An inexpensive VPS from any provider.
• Nginx and a small application in any language (Python/Flask, Node.js/Express, PHP).

How it works

1. Nginx accepts HTTPS requests and forwards them to the application.

2. The application (about twenty lines of code):

   • accepts POST requests at /log;
   • checks a secret key in the X-API-Key header;
   • appends JSON to a log file.

Example in Python (Flask)

from flask import Flask, request, abort
import os
import json

app = Flask(__name__)

API_KEY = os.environ["LOG_API_KEY"]

@app.route('/log', methods=['POST'])
def receive_log():
    if request.headers.get('X-API-Key') != API_KEY:
        abort(401)

    data = request.get_json(silent=True)
    if not data:
        abort(400)

    try:
        with open("/var/log/my-app/events.log", "a") as f:
            f.write(json.dumps(data) + "\n")
    except Exception as e:
        print(f"Failed to write log: {e}")
        abort(500)

    return "OK", 200

if __name__ == '__main__':
    app.run(host='127.0.0.1', port=5000)

The key is taken from the environment variable LOG_API_KEY with no default: if the variable is not set, the application won’t start rather than accept logs with a default key. The built-in Flask server is fine for testing; in production the application is run via a WSGI server (for example, gunicorn).

Advantages

• Minimal cost — the price of the cheapest VPS.
• Full control over storage.
• Deployment in half an hour.

Disadvantages

• Poor scalability: writing to a single file quickly becomes a bottleneck.
• Maintenance required: log rotation, disk monitoring, updates.
• Analysis is manual, via SSH and grep.

Option 2. Vector collector and Loki storage

An evolution of the first option using ready-made open-source components.

What you’ll need

• A VPS with 1–2 GB of RAM.
• Vector, Loki, and Grafana.

How it works

1. Vector accepts logs over HTTP, processes them, and forwards them.
2. Loki stores logs and indexes only labels, so storage is inexpensive.
3. Grafana is the interface for searching and analyzing logs using LogQL.

Read more about Loki and Grafana in the article “Centralized logging: Part 5 — Loki and Grafana”.

Example Vector configuration (vector.yaml)

The http_server source has two access control methods: basic (login and password) and custom — checks written in VRL. Below the key is checked via the X-API-Key header using custom.

sources:
  http_logs:
    type: "http_server"
    address: "0.0.0.0:8080"
    decoding:
      codec: "json"
    auth:
      strategy: "custom"
      source: |-
        .headers."x-api-key" == "super-secret-key"

transforms:
  my_transform:
    type: "remap"
    inputs: ["http_logs"]
    source: |
      .app = "my_mobile_app"
      if !exists(.level) { .level = "info" }

sinks:
  loki:
    type: "loki"
    inputs: ["my_transform"]
    endpoint: "http://localhost:3100"
    labels:
      app: "{{ app }}"
      level: "{{ level }}"
    encoding:
      codec: "json"

Replace super-secret-key with a long random key. In this example Vector listens on port 8080 over HTTP, so there should be Nginx or another reverse proxy with HTTPS in front of it, and port 8080 should not be accessible from the internet.

Advantages

• High performance.
• Search and filtering in Grafana.
• Vector can send logs not only to Loki but also to S3, ClickHouse, and other storages.

Disadvantages

• More complex setup: three components to run (via Docker or systemd) and keep updated.

Option 3. Serverless

No server required: you pay only for requests and storage.

What you’ll need

• A cloud account. For Russian companies Yandex Cloud is available: Cloud Functions, API Gateway, and Object Storage. Russian companies cannot pay for AWS and Google Cloud services.
• A stack (example below — AWS): API Gateway + Lambda + S3. In Yandex Cloud the scheme is the same: API Gateway calls a Cloud Function, which writes to Object Storage via an S3-compatible API.

How it works

1. The client sends a POST request.
2. API Gateway accepts it at a managed HTTP endpoint.
3. The gateway invokes the function.
4. The function saves the JSON to object storage, organizing files by date.

Example function in Python (writing to S3)

import json
import boto3
import time
import os

s3 = boto3.client('s3')
BUCKET_NAME = os.environ['LOG_BUCKET_NAME']

def lambda_handler(event, context):
    body = event.get('body')
    if not body:
        return {'statusCode': 400, 'body': 'No data'}

    try:
        log_data = json.loads(body)
    except json.JSONDecodeError:
        return {'statusCode': 400, 'body': 'Invalid JSON'}

    now = time.strftime('%Y/%m/%d/%H', time.gmtime())
    file_name = f"{context.aws_request_id}.json"
    s3_key = f"logs/{now}/{file_name}"

    try:
        s3.put_object(
            Bucket=BUCKET_NAME,
            Key=s3_key,
            Body=json.dumps(log_data),
            ContentType='application/json'
        )
        return {'statusCode': 200, 'body': 'OK'}
    except Exception as e:
        print(e)
        return {'statusCode': 500, 'body': 'Error saving log'}

Advantages

• At low log volumes, costs fit within the cloud’s free tiers or are near zero — check provider limits for details.
• Scaling without your involvement.
• No servers to update.

Disadvantages

• Additional services are needed for analysis (for example, Athena, BigQuery, or queries against object storage).
• Lock-in to a specific cloud provider.
• Each request creates a separate object: with a high log rate this is inefficient, so clients should send logs in batches.

Recommendations for the client

1. Asynchronous sending — do not block the application’s main thread.
2. Send in batches — accumulate records and send them, for example, every 30 seconds or in batches of 50 events.
3. Retries — store unsent logs locally (for example, in SQLite) and resend later.
4. Filtering — send only necessary levels (INFO, WARN, ERROR).

Conclusion

Collecting logs from remote clients can be inexpensive — you just need to choose the approach that fits your task.