Centralized logging: Part 5 — Loki and Grafana

Loki — a log storage system from Grafana Labs. Its main difference from ELK, OpenSearch and Graylog is that Loki does not build a full-text index. Only labels are indexed — a short set of key–value pairs describing the source: server, service, environment. The log lines themselves are stored in compressed blocks, including in S3-compatible storage. Because of this Loki requires significantly fewer resources, and logs are viewed in Grafana alongside Prometheus metrics.

License

Since April 2021 Loki, like Grafana, is distributed under AGPLv3 — an OSI-approved open license. Agents and some libraries remain under Apache 2.0. For a company that runs Loki for its own logs, AGPLv3 does not impose restrictions; the requirement to disclose changes applies to those who modify Loki and provide it as a network service.

Components

• Loki — receives, stores and serves logs. For small installations it runs as a single process; for large ones — as a set of separate services.
• Grafana — the UI for searching (Explore section), dashboards and alerts.
• Agent on servers — reads logs, adds labels and sends them to Loki. Currently this is Grafana Alloy.

Promtail is no longer supported

For a long time the agent for Loki was Promtail, and most older instructions describe it. Since 13 February 2025 Promtail received only fixes, and on 2 March 2026 its support was fully discontinued. Grafana recommends migrating to Grafana Alloy. To convert an existing configuration there is a command:

alloy convert --source-format=promtail --output=config.alloy promtail.yaml

It converts a Promtail configuration to Alloy format; the result should be checked manually.

Example Alloy configuration

An Alloy configuration consists of components connected to each other. To read files from /var/log and send them to Loki you need three components: file discovery, reading and sending.

local.file_match "system" {
  path_targets = [{
    __address__ = "localhost",
    __path__    = "/var/log/*.log",
    job         = "system",
  }]
}

loki.source.file "system" {
  targets    = local.file_match.system.targets
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki.internal:3100/loki/api/v1/push"
  }
  external_labels = {
    host = "web-01",
    env  = "prod",
  }
}

local.file_match finds files by pattern, loki.source.file reads them and forwards the lines, loki.write sends them to Loki and adds the host and env labels to each entry. For the system journal there is the loki.source.journal component, for containers — loki.source.docker.

Labels and cardinality

Labels are the most important part of Loki configuration. Each unique combination of label values forms a separate stream, and each stream has its own index and data blocks. Loki documentation explicitly states that the system is designed for long-lived streams and a small number of label values.

Good labels have a small and stable set of values: env, cluster, namespace, app, job, host.

Bad labels have an unbounded number of values: request IDs, trace IDs, users and orders, IP addresses, timestamps. Documentation even recommends not putting log level and HTTP response code into labels, but filtering by the log line instead. High cardinality leads to a huge index and thousands of small blocks in storage, and Loki starts to perform poorly.

For frequently searched fields with many values, for example a client identifier, Loki has structured metadata: they are stored alongside the entry and do not go into the index.

LogQL

Queries are written in LogQL. A query starts with selecting streams by labels, followed by a chain of filters and processors.

Log lines with errors in nginx on production servers:

{job="nginx", env="prod"} |= "error"

Parse JSON and filter by a field:

{app="payments"} | json | level="error" | order_id != ""

Number of errors per second by servers over five-minute windows — this query can be displayed on a graph and used in an alert:

sum by (host) (rate({job="nginx"} |= " 500 " [5m]))

The syntax is similar to PromQL, so it’s easier for those who work with Prometheus.

Advantages

• Resource savings. A label index is orders of magnitude smaller than a full-text one; lines are stored compressed, including in cheap object storage.
• Logs next to metrics. In Grafana you can jump from a metric graph to the logs of the same service for the same interval.
• Easy start. For a small infrastructure one Loki process and an agent on servers are sufficient.
• Open license AGPLv3.

Limitations

• Content search is slower. Without a full-text index a text search scans the lines of the selected streams. The more precise the label selection and the shorter the time range, the faster the query.
• Less processing than in Logstash or Graylog pipelines: the agent and LogQL queries do most of the parsing work.
• Discipline with labels. One bad label with a large number of values can overload the system.

Common mistakes

• New setups on Promtail. The agent is not supported; use Alloy for new servers.
• Putting request ID or IP address in a label. The number of streams grows exponentially.
• No retention period. In Loki it is set by the retention_period parameter and enabled in the compactor; without it data is stored indefinitely.
• Loki exposed externally without authentication. In single-user mode Loki does not itself verify who is sending or reading logs. Access should be restricted by a firewall or a reverse proxy with authentication.

When to choose Loki

Loki is a good choice if Prometheus already collects metrics and dashboards are built in Grafana, and you need to cheaply add logs alongside them. It works well in Kubernetes and with large volumes of logs where searches are mostly by source and time. If you need full-text search and complex content analytics, OpenSearch or ELK are more suitable.