Centralized Logging: Part 2 — ELK Stack (Elasticsearch, Logstash, Kibana)

ELK Stack — the best-known software stack for centralized logging. The name is formed from the initials of three components from Elastic: Elasticsearch, Logstash, and Kibana. Together they collect logs, parse them into fields, store them with a full-text index, and display them in a web interface.

Components

• Elasticsearch — a distributed search and analytics engine based on Apache Lucene. It stores records in indices, performs full-text search and aggregations, and scales by adding nodes.
• Logstash — a processing pipeline: it accepts data from various sources, parses, filters, enriches, and sends it to Elasticsearch. It works with dozens of input and output plugins.
• Kibana — a web interface for search, dashboards, alerts, and cluster management.
• Beats and Elastic Agent — agents on servers. Filebeat reads log files, Metricbeat collects metrics. Elastic Agent combines their functions into a single agent that is managed centrally via Fleet in Kibana.

Therefore the bundle is often called Elastic Stack: it has more agents than there are letters in the name.

How data flows

1. An agent on the server reads a log and sends records to Logstash or directly to Elasticsearch.
2. Logstash or an ingest pipeline in Elasticsearch parses the lines into fields.
3. Elasticsearch indexes the records.
4. In Kibana you search them, build charts, and configure alerts.

Logstash is needed when processing is complex or there are many sources. For simple cases an agent plus an ingest pipeline is sufficient — one less component.

License

The history of Elastic’s licenses matters for choosing:

• before 2021 Elasticsearch and Kibana were distributed under Apache 2.0;
• in 2021 Elastic moved to a choice of two licenses — SSPL and the Elastic License 2.0. Neither is recognized as open by the OSI; the response to this was the OpenSearch fork;
• on 29 August 2024 Elastic announced the addition of a third option — AGPLv3, approved by the OSI. From that moment Elasticsearch and Kibana can again be called open source software.

For a company that deploys ELK on its own servers for its own logs, none of these licenses creates restrictions. They matter for those who sell Elasticsearch as a cloud service or embed it in their product.

Separately from code licensing there are Elastic subscriptions. The free Basic level includes basic security — users, roles, encrypted connections — and Kibana alerts with a limited set of delivery channels. Machine learning, parts of Elastic Security, and advanced integrations are available in paid subscriptions.

Query language ES|QL

In addition to the classic Query DSL and KQL in Kibana, Elasticsearch has the ES|QL language. A query is built as a chain of steps separated by a vertical bar. ES|QL appeared in version 8.11 as an experimental feature and became generally available in version 8.14.

FROM logs-*
| WHERE log.level == "error"
| STATS errors = COUNT(*) BY service.name
| SORT errors DESC
| LIMIT 10

The query selects errors from log indices, counts them by service, and shows the ten services with the most errors.

How to try it out

For local experimentation Elastic offers the start-local script: it downloads current Elasticsearch and Kibana and runs them in Docker. It’s best to get the command and description from the official documentation — it is updated along with versions. For production there are separate instructions: a multi-node cluster in Docker Compose, installation from DEB and RPM packages, and deployment in Kubernetes via the ECK operator.

When installing on your own it’s important to:

• allocate enough memory to Elasticsearch but don’t give it all the server RAM — some is needed for the OS file cache;
• don’t expose port 9200 to the internet;
• keep authentication and TLS enabled, which in modern versions are configured on first startup.

Retention: ILM

Log indices grow continuously, so they are managed by an Index Lifecycle Management (ILM) policy. It moves indices through phases — hot, warm, cold, frozen, delete — and performs actions in each: create a new index, shrink, move to cheaper nodes, delete.

An example policy from Elastic documentation: a new index is created when the primary shard reaches 25 GB, and the index is deleted after 30 days:

PUT _ilm/policy/logs-30d
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_primary_shard_size": "25GB" }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

Without such a policy the cluster disk will sooner or later fill up, and Elasticsearch will stop accepting writes.

Advantages

• Search. Full-text search and aggregations over large volumes of logs.
• Analytics and visualization. Kibana provides dashboards, data exploration, and alerts.
• Ecosystem. Ready integrations for popular systems, agents for all major platforms, and a large community.
• Scalability. The cluster grows by adding nodes.

Drawbacks

• Resources. Elasticsearch and Logstash are demanding on memory, CPU, and disks; the full-text index increases storage size.
• Operational complexity. You need to monitor shards, indices, retention policies, upgrades, and backups.
• Some features are paid. Machine learning and advanced security features require a subscription.

Common mistakes

• Too many small shards. Creating an index per day with multiple shards quickly yields thousands of shards, and the cluster spends resources tracking them.
• No ILM policy. Disk fills up and writes stop.
• Cluster exposed to the internet. Unprotected Elasticsearch instances regularly become sources of data leaks.
• Single node without replicas and snapshots. A disk failure means losing all logs; backups are made via snapshots to separate storage.

When to choose ELK

ELK is appropriate if you need full-text search across large volumes of logs, advanced analytics, and have people to operate the cluster. For small infrastructures where logs are mainly used for debugging failures it’s overkill: Loki or Graylog will be cheaper in resources and easier to maintain.