Centralized Logging: Part 3 — OpenSearch

OpenSearch — an open data search and analytics system most often used for centralized log storage. It is a fork of Elasticsearch and Kibana: in 2021, after Elastic changed the license to SSPL and the Elastic License, Amazon Web Services created OpenSearch based on the latest versions under Apache 2.0 — Elasticsearch 7.10.2 and Kibana 7.10.2.

Since September 16, 2024 the project is governed by the OpenSearch Software Foundation as part of the Linux Foundation. The founders of the foundation are AWS, SAP and Uber; before that the project was developed under the leadership of AWS. The license remains the same — Apache 2.0.

Components

• OpenSearch — storage and search engine: indices, full-text search, aggregations, cluster of multiple nodes.
• OpenSearch Dashboards — web interface for search, visualizations, dashboards and management.
• Plugins included in the standard distribution: security (users, roles, TLS, audit log), alerting (Alerting), anomaly detection, SQL and PPL, index state management (ISM).

The main practical difference from Elastic Stack is that all of the above is available for free, without a separate subscription.

Versions

In April 2025 OpenSearch 3.0 was released — the first major release in three years and the first under the foundation. Since then releases of the 3.x branch have been published. The 1.x branch should no longer be used for new installations: it is being dropped by third-party products, for example Graylog.

How to send logs

Here is the main difference from ELK that’s easy to forget. Recent Elastic agents do not work with OpenSearch: according to OpenSearch documentation, Beats newer than 7.12.x are not supported. Clients of versions 7.x–7.12.x that check the server version need to enable compatibility mode in OpenSearch. Therefore, other tools are used for new installations:

• Data Prepper — the project’s own tool for ingesting, processing and routing data. OpenSearch documentation calls it the preferred way to ingest data. A typical scheme: Fluent Bit on servers sends logs to Data Prepper over HTTP, which parses lines with a grok processor and writes the result to OpenSearch.
• Fluent Bit — a lightweight agent with a built-in opensearch output: sends records in batches via the _bulk API, supports TLS.
• Logstash with the logstash-output-opensearch output plugin — if Logstash is already used and you don’t want to rewrite pipelines. A common pattern when migrating from ELK: Filebeat remains on servers, and Logstash with the OpenSearch plugin acts as a bridge.

Minimal Fluent Bit output in classic configuration format:

[OUTPUT]
    Name        opensearch
    Match       *
    Host        opensearch.internal
    Port        9200
    HTTP_User   fluentbit
    HTTP_Passwd ${OPENSEARCH_PASSWORD}
    tls         On
    Index       logs
    Suppress_Type_Name On

The Suppress_Type_Name parameter is needed for OpenSearch 2.x and newer: in those versions the _type field in requests is no longer used.

Query languages

In addition to the DSL inherited from Elasticsearch, OpenSearch has two languages convenient for working with logs:

• SQL — familiar SELECT … FROM … WHERE queries;
• PPL (Piped Processing Language) — a chain of commands separated by a vertical bar:

source = logs-*
| where level = 'error'
| stats count() by service

The query counts errors by service. PPL is an OpenSearch-specific language; Elasticsearch has its own language with a similar idea — ES|QL.

Retention: ISM

Log indices are managed by Index State Management (ISM). A policy describes index states and transitions between them: an index lives in the hot state, and upon reaching an age or size moves on and eventually is deleted. Example policy from the documentation: the index is deleted after 30 days.

PUT _plugins/_ism/policies/logs-30d
{
  "policy": {
    "description": "Delete logs after 30 days",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [],
        "transitions": [
          { "state_name": "delete", "conditions": { "min_index_age": "30d" } }
        ]
      },
      {
        "name": "delete",
        "actions": [ { "delete": {} } ]
      }
    ]
  }
}

In essence ISM corresponds to ILM in Elasticsearch, but the policy syntax is different: when migrating, policies need to be rewritten.

Installation

Official images are opensearchproject/opensearch and opensearchproject/opensearch-dashboards. Since version 2.12 you must set the administrator password on first startup in the OPENSEARCH_INITIAL_ADMIN_PASSWORD variable; without it the container will not start. For a single node set discovery.type=single-node. For production the documentation describes a multi-node cluster in Docker Compose, installation from packages, and Helm charts for Kubernetes.

Like Elasticsearch, OpenSearch requires increasing the host kernel parameter vm.max_map_count and allocating enough memory for the JVM.

Advantages

• Open Apache 2.0 license and governance by an independent foundation.
• Security and alerting for free — no subscription.
• API compatibility with Elasticsearch 7.10: many tools and queries transfer without changes.
• SQL and PPL — lower barrier to entry than DSL.

Disadvantages

• Resources. Memory, CPU and disk requirements are of the same order as Elasticsearch.
• Divergence from Elastic. Since 2021 the projects have evolved separately: new Elastic features do not appear in OpenSearch, and Elastic integrations and agents do not work with it.
• Cluster maintenance — shards, policies, upgrades, snapshots — is your responsibility.

Common mistakes

• Using the latest Filebeat directly with OpenSearch. The connection doesn’t establish or breaks after agent upgrades. Use Fluent Bit, Data Prepper, or Logstash with the plugin.
• Disabled security plugin. In online examples it’s often turned off for simplicity, and then left disabled in production.
• No ISM policy. Indices accumulate until disk space runs out.
• Single node without snapshots. Backups are made via snapshots to a separate repository, for example an S3-compatible storage.

When to choose OpenSearch

OpenSearch is suitable for those who need ELK-level capabilities — full-text search, dashboards, alerting, access control — under an open license and without a paid subscription. If you already work with Elastic Stack and use Elastic Agent and built-in integrations, migrating will require replacing agents and retention policies; this should be taken into account in advance.