Centralized Logging: Part 4 — Graylog

Graylog — a platform for centralized log management. Unlike ELK, which is assembled from separate components, Graylog is originally built as a single system: intake, processing, routing, storage, search, alerts and access control are all configured in one web interface.

License and editions

The free edition is called Graylog Open. Since version 4.0 (2021) all its releases are distributed under the Server Side Public License (SSPL). This license is based on the GPL but additionally requires disclosing the service’s source code if the software is offered as a cloud service. The OSI does not recognize it as open source, so it’s more accurate to call Graylog Open a source-available product.

For a company that deploys Graylog for its own logs, SSPL does not create restrictions. Besides Open, there are paid editions with additional features — archives, extended audit, and capabilities for investigating security incidents.

Components

• Graylog Server — message intake, processing, web interface, users, streams, alerts.
• MongoDB — stores configuration, users and metadata, but not the logs themselves. Graylog 7.0 requires MongoDB 7.0 or newer.
• Log storage — the search engine where messages are indexed:
  • Graylog Data Node — the preferred option: a Graylog-managed node based on OpenSearch. Graylog handles its configuration, certificates and updates;
  • self-managed OpenSearch — if a cluster already exists or full control is needed. Support for OpenSearch 1.x in Graylog 7.0 is deprecated and will be removed in 8.0;
  • Elasticsearch — starting with Graylog 7.0, the use of Elasticsearch is deprecated and will be completely removed in Graylog 8.0. It should not be chosen for new installations.
• Graylog Sidecar — a lightweight management agent: from the Graylog interface it configures and runs log shippers on servers, for example Filebeat or NXLog.

How messages flow

1. Inputs (Inputs). Graylog receives data via syslog (UDP/TCP), in GELF format, via Beats, over HTTP and other protocols.
2. Extraction and pipelines. Extractors and processing pipelines (Pipelines) parse messages into fields, drop unnecessary data, and add metadata.
3. Streams (Streams). Rules route messages into streams: web server logs, security logs, logs of a specific application. For each stream you set its index set, retention period and access rights.
4. Storage. Messages are indexed in the Data Node or OpenSearch.
5. Search, dashboards, alerts. Everything — in one interface.

GELF

Graylog Extended Log Format — a JSON message format that removes the limitations of classic syslog: there is no hard length limit, and fields are transmitted explicitly. GELF libraries exist for most languages, and Docker can send container output to GELF using its built-in logging driver. Minimal message:

{
  "version": "1.1",
  "host": "web-01",
  "short_message": "payment failed",
  "level": 3,
  "_service": "payments",
  "_order_id": 1842
}

Additional fields start with an underscore; level is the severity in syslog numbering (3 — error).

Processing pipelines

A pipeline rule is written in a simple language and triggers for messages that match the condition. Example: parse a key=value string and remove the original field:

rule "parse key-value"
when
  has_field("message") && contains(to_string($message.message), "=")
then
  set_fields(key_value(to_string($message.message)));
end

Rules are grouped into stages, stages into a pipeline, and the pipeline is attached to the relevant streams.

Installation

The official way for getting started and small installations is Docker Compose. The Graylog2/docker-compose repository contains ready files for the Open edition: three services — MongoDB, Data Node and Graylog. Before launching you set a secret for password encryption and the admin password hash; separate volumes are needed for each of the three services to store data. For production, the documentation describes package-based installation and a multi-node cluster.

Compatibility of Graylog, MongoDB and OpenSearch versions is described in the compatibility matrix in the documentation. Check it before each upgrade: major Graylog releases regularly raise minimum requirements.

Advantages

• Unified system. Intake, processing, storage, search and alerts are configured in one place.
• Streams and access control. Convenient to split logs by teams and set different access.
• Processing pipelines without a separate Logstash.
• Data Node relieves the administrator of much of the work of maintaining OpenSearch.
• Sidecar — centralized management of shippers on servers.

Drawbacks

• Multiple components. MongoDB, Data Node and Graylog Server need to be upgraded in coordination.
• Visualization is simpler than in Kibana or OpenSearch Dashboards.
• SSPL license for the free edition and some features are available only in paid editions.
• Resource usage. Under the storage there is still a search engine with a full-text index.

Common pitfalls

• New installation on Elasticsearch. Support will be removed in Graylog 8.0, and you’ll have to migrate.
• Outdated MongoDB. After upgrading to Graylog 7.0 the server will fail startup checks if MongoDB is below 7.0.
• Syslog over UDP for important logs. UDP does not guarantee delivery; for security logs prefer TCP with TLS or GELF over TCP.
• Everything in one stream. Without splitting into streams you can’t set different retention periods and access rights.

When to choose Graylog

Graylog is suitable for small and medium teams that need a ready-made logging system with access control, processing rules and alerts, but don’t want to assemble and configure a stack of separate components. If resource efficiency is the top priority and logs need to be colocated with Prometheus metrics, consider Loki.