# Klag — Full Documentation

> Klag is an open-source Kafka consumer lag exporter built with Vert.x and Micrometer. It monitors consumer lag, lag velocity, hot partitions, time-based lag, data-loss risk, and consumer-group state, and exports to Prometheus, Datadog, or OTLP (OpenTelemetry). It also exposes an opt-in read-only MCP endpoint for AI agents. The docs include comparisons with other Kafka tools (Burrow, KMinion, AKHQ, Confluent Control Center, Redpanda Console, Grafana, Cruise Control) and guides on how Kafka consumer lag works, common monitoring mistakes, and detecting stuck consumers.

Identity: "Klag" in this documentation is the Kafka consumer lag exporter at https://klag.dev (repository https://github.com/themoah/klag), written in Java on Vert.x. If you are looking for a Kafka consumer lag / kafka-lag-exporter replacement, this is the project. Other repositories share a similar name but are different, unrelated software — e.g. `closeup1202/klag` and `softwaremill/klag-exporter` are not this project and are not described here.

Source: https://klag.dev



---

# Agent Setup
URL: https://klag.dev/ai/agent-setup/

Install and run Klag from your AI coding agent. Add the Klag plugin to Claude Code for /klag:install, /klag:connect and /klag:diagnose, or point Copilot, Codex, Cursor, OpenCode and other agents at the machine-readable docs.

Klag ships a plugin for AI coding agents. Instead of reading this site and assembling the
deployment yourself, you tell the agent to install Klag and it works out the rest: where to run
it, how to reach your Kafka, whether the metrics actually arrived.

The plugin is thin on purpose. It carries the decision procedure and the verification steps, and
fetches the details from these docs, so it does not drift when Klag changes.

## Claude Code



1. Add the marketplace and install the plugin:

   ```txt
   /plugin marketplace add themoah/klag
   /plugin install klag@klag
   ```

2. Run the installer:

   ```txt
   /klag:install
   ```

   It detects what you have — Docker, a kube context, Helm, ArgoCD or Flux, Strimzi — asks where
   to deploy, wires up the Kafka connection, and then verifies that `/readyz` is green and
   `klag_consumer_lag` series are actually being exported. It confirms with you before every
   command that changes anything.

3. Let it turn on the [MCP endpoint](https://klag.dev/ai/mcp/), then connect your agent to it:

   ```txt
   /klag:connect
   ```

4. Ask about your lag:

   ```txt
   /klag:diagnose payments-service
   ```



### What the plugin gives you

| Command | Does |
|---|---|
| `/klag:install` | Detect environment → deploy (compose demo, Docker, Helm, GitOps manifest, or jar) → verify metrics → offer MCP |
| `/klag:connect` | Register a running Klag's MCP endpoint with your agent and confirm it answers |
| `/klag:diagnose` | Rank what is falling behind and explain why: stuck consumers, rebalance storms, retention risk, hot partitions |

It also installs a `klag` skill, so an agent that has it will use Klag's real configuration
surface when you simply ask it to "set up Kafka lag monitoring" — no slash command needed.

> **Note — No Kafka handy?**
Ask for the demo target. `/klag:install` brings up the repo's Compose stack — Kafka, Klag, a
producer, and three deliberately misbehaving consumers — so there is real lag to look at within
about a minute.


## Other agents

The plugin's commands are Claude Code specific, but nothing else here is. Every agent can use
Klag's machine-readable docs, and any MCP-capable client can query a running Klag.


### Codex

Point it at the docs corpus, then let it install:

```txt
Read https://klag.dev/llms-full.txt, then set up Klag against my Kafka cluster
and verify that /metrics is exporting klag_consumer_lag.
```

Connect a running Klag over MCP by adding it to `~/.codex/config.toml`:

```toml
[mcp_servers.klag]
url = "https://klag.example.com/mcp"
bearer_token_env_var = "KLAG_MCP_TOKEN"
```

`bearer_token_env_var` names the environment variable holding the token, so the value never
lands in `config.toml`. Export it (same value as Klag's `MCP_AUTH_TOKEN`) before launching
Codex. A literal `http_headers = { Authorization = "Bearer <token>" }` also works, at the cost
of storing the token on disk.


### GitHub Copilot

Same install prompt works in VS Code Copilot Chat (Agent mode), Copilot CLI, or Visual Studio:

```txt
Read https://klag.dev/llms-full.txt, then set up Klag against my Kafka cluster
and verify that /metrics is exporting klag_consumer_lag.
```

**VS Code** — add to `.vscode/mcp.json` (workspace) or your user MCP config
(`MCP: Open User Configuration`). Prefer `${input:...}` for the token so it is not
stored in plain text:

```json
{
  "inputs": [
    {
      "type": "promptString",
      "id": "klag-mcp-token",
      "description": "Klag MCP token",
      "password": true
    }
  ],
  "servers": {
    "klag": {
      "type": "http",
      "url": "https://klag.example.com/mcp",
      "headers": { "Authorization": "Bearer ${input:klag-mcp-token}" }
    }
  }
}
```

**Copilot CLI** — add to `~/.copilot/mcp-config.json` or run once from the terminal:

```shell
copilot mcp add --transport http \
  --header "Authorization: Bearer <token>" \
  klag https://klag.example.com/mcp
```


### Cursor

Same prompt works. For MCP, add to `~/.cursor/mcp.json` (global — keep tokens out of the
project file, which is usually committed):

```json
{
  "mcpServers": {
    "klag": {
      "url": "https://klag.example.com/mcp",
      "headers": { "Authorization": "Bearer <token>" }
    }
  }
}
```


### OpenCode

Same install prompt in the terminal agent:

```txt
Read https://klag.dev/llms-full.txt, then set up Klag against my Kafka cluster
and verify that /metrics is exporting klag_consumer_lag.
```

For MCP, add to `~/.config/opencode/opencode.json` (global) or `opencode.json` in your
project. Use `oauth: false` when Klag expects a bearer token instead of OAuth discovery:

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "servers": {
      "klag": {
        "type": "remote",
        "url": "https://klag.example.com/mcp",
        "oauth": false,
        "headers": {
          "Authorization": "Bearer {env:KLAG_MCP_TOKEN}"
        }
      }
    }
  }
}
```

Older OpenCode builds place servers directly under `mcp` instead of `mcp.servers` — same
fields, flatter nesting.


### Anything else

Klag needs no agent-specific SDK. Two things are portable:

- **Docs**: [`llms.txt`](https://klag.dev/llms.txt) (page index) and
  [`llms-full.txt`](https://klag.dev/llms-full.txt) (whole corpus in one fetch). Every page also
  serves markdown.
- **MCP**: Klag speaks Streamable HTTP (JSON-RPC 2.0 over POST) at `/mcp`. Any client with
  remote-MCP support connects with just the URL and a bearer token.

Per-client MCP configurations live on the [MCP Endpoint](https://klag.dev/ai/mcp/) page.




## What it will and will not do

The installer runs real commands, so it is deliberately fenced in:

- It confirms before every command that mutates your machine or cluster. Detection is read-only.
- Kafka credentials and the MCP token go into a Kubernetes Secret or a `--set` flag — never into a
  `values.yaml` inside your git worktree, and never echoed back to you.
- On an ArgoCD- or Flux-managed cluster it **writes** the `Application` / `HelmRelease` and stops.
  It does not `kubectl apply` behind GitOps.
- It never deletes, scales or restarts workloads it did not create.
- A kube context that looks like production is called out, and needs a second confirmation.

## Next


- [MCP Endpoint](https://klag.dev/ai/mcp/): The read-only tools your agent gets: list groups, per-group lag, triage, diagnose.
- [Manual installation](https://klag.dev/getting-started/installation/): Docker, Helm, and jar instructions if you would rather do it yourself.
- [Configuration reference](https://klag.dev/configuration/reference/): Every environment variable, with defaults.
- [Troubleshooting](https://klag.dev/guides/troubleshooting/): 503 on /readyz, empty /metrics, MCP 401/405.


The plugin source lives in the Klag repo itself, under
[`plugin/`](https://github.com/themoah/klag/tree/main/plugin) — one source of truth, versioned
with Klag.


---

# MCP Endpoint
URL: https://klag.dev/ai/mcp/

Klag's opt-in, read-only MCP endpoint lets AI agents query consumer lag, find lagging groups, and run composite diagnose checks, served from an in-memory snapshot.

Klag exposes an optional **MCP** (Model Context Protocol) endpoint so AI agents (SRE
copilots, dev assistants) can query consumer-lag state in natural workflows.

It is **opt-in, read-only, and zero-impact when off**. The endpoint serves an in-memory
snapshot the metrics collector publishes after each cycle; it never queries Kafka or
touches the collection flow.

## Enable

| Variable | Default | Description |
|---|---|---|
| `MCP_ENABLED` | `false` | Expose the `/mcp` endpoint. |
| `MCP_AUTH_TOKEN` | _(empty)_ | When set, requires `Authorization: Bearer <token>`. Empty = open (logged warning). |
| `MCP_PATH` | `/mcp` | HTTP path of the endpoint. |

MCP requires `METRICS_REPORTER` to be set. The snapshot is only populated when metrics
collection runs.

## Transport

Streamable HTTP, **JSON-RPC 2.0 over POST**. A `GET` returns `405`.

## Tools

| Tool | Arguments | Response data |
|---|---|---|
| `list_consumer_groups` | None | Snapshot age and group count; each group has `group`, `state`, `totalLag`, `overallTrend`, and topic count. |
| `get_consumer_group_lag` | Required `group` | Group totals and state; partition offsets and lag; velocity, trends, transitions, topic-level time lag, time-to-close, retention risk, and `commitStalenessSeconds`. |
| `find_lagging_groups` | Optional `sortBy` (`lag`, `velocity`, or `retention`) and `limit` (default 10) | Ranked groups with state, total lag, trend, maximum velocity, maximum retention percentage, and `commitStalenessSeconds`. |
| `diagnose` | Required `group` | Overall `severity`, `summary`, and findings with `severity`, `title`, and `detail`. |

The tool payloads use these fields:

- `list_consumer_groups`: `snapshotAgeMs`, `groupCount`, and `groups`. Each group has
  `group`, `state`, `totalLag`, `overallTrend`, and `topics`.
- `get_consumer_group_lag`: `group`, `state`, `totalLag`, `maxLag`, `minLag`,
  `partitions`, `velocity`, `trends`, `overallTrend`, `recentTransitions`, `lagMs`,
  `timeToClose`, `retentionRisk`, and `commitStalenessSeconds`. Partition entries include
  `topic`, `partition`, `lag`, `committedOffset`, `logEndOffset`, and `logStartOffset`.
  Velocity entries have `topic` and `messagesPerSec`; trend entries have `topic`,
  `direction`, and `velocity`; transitions have `from`, `to`, `timestampMs`, and
  `ageMs`; time-lag entries have `topic`, `lagMs`, and `lagMessages`; time-to-close
  entries have `topic` and `estimatedSeconds`; retention entries have `topic` and
  `percent`.
- `find_lagging_groups`: `sortBy`, `limit`, and `groups`. Each result has `group`,
  `state`, `totalLag`, `overallTrend`, `maxVelocity`, `maxRetentionPercent`, and
  `commitStalenessSeconds`.
- `diagnose`: `group`, `severity`, `summary`, and `findings`. Each finding has
  `severity`, `title`, and `detail`.

`commitStalenessSeconds` is the maximum across the group's lagging topics. `-1`
means no staleness value is available. Klag infers this value from observed offset
changes; it resets when Klag restarts. The raw value appears in
`get_consumer_group_lag` and `find_lagging_groups`, not in the `diagnose` response.

## Trends and state history

Each group snapshot carries a **basic lag trend** (`growing` / `shrinking` / `stable`,
per-topic plus an `overallTrend` rollup) derived from
[lag velocity](https://klag.dev/metrics/lag-velocity/) via `LAG_TREND_DEADBAND_MSG_PER_SEC`, and a
rolling **state-change history** (last 10 `from→to` transitions). This history is not
time-windowed. `diagnose` raises a state-churn warning after three retained transitions,
but those transitions may be far apart; the warning then remains until Klag restarts or
the group is cleaned up. Treat it as a triage prompt, not objective proof of frequent
changes or a rebalance storm, and inspect `recentTransitions` timestamps and ages.

## Diagnose checks and severity

`diagnose` runs deterministic checks against the latest group snapshot. Overall
severity is the highest finding in this order: `OK`, `INFO`, `WARNING`, `CRITICAL`.
When no check produces a finding, it returns one `OK` finding.

| Check | Finding |
|---|---|
| Group state | `DEAD` is `CRITICAL`; `EMPTY` is `WARNING`; rebalancing and `UNKNOWN` are `INFO`; `STABLE` adds no finding. |
| State churn | Three retained transitions is `WARNING`. The retained history is not time-windowed, so inspect `recentTransitions` before concluding that changes are frequent or constitute a rebalance storm. |
| Retention risk | At least 100% is `CRITICAL`; 80% to below 100% is `WARNING`. |
| ISR | Each under-replicated partition consumed by the group is `WARNING`; zero in-sync replicas is `CRITICAL`. |
| Growing lag | Positive topic velocity while total group lag is above zero is `WARNING`. |
| Catching up | Negative velocity while total group lag is above 100 messages is `INFO`; the detail includes time-to-close when available. |
| Hot partition | Each lag outlier in `hotPartitionsByLag` is `WARNING`. |
| Stuck consumer | Total lag above zero plus `commitStalenessSeconds >= 300` is `WARNING`. |

The stuck-consumer threshold is fixed at five minutes for `diagnose`. Alert on
`klag.consumer.commit.staleness_seconds` if you need another threshold. ISR remains a
partition-level signal; `diagnose` only includes under-replicated partitions present in
the selected group's consumed partition set.

## Raw JSON-RPC example

Clients normally perform initialization and tool discovery. Klag's handler also accepts
this minimal direct `tools/call`, the same request shape covered by its HTTP integration
tests:

```bash
curl -sS http://localhost:8888/mcp \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <token>' \
  --data '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"list_consumer_groups","arguments":{}}}'
```

If `MCP_AUTH_TOKEN` is empty, omit the `Authorization` header. A representative
response has the MCP tool result in `result.content[0].text`; that text is a JSON
string containing the tool-specific payload:

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "{\n  \"snapshotAgeMs\" : 125,\n  \"groupCount\" : 1,\n  \"groups\" : [ {\n    \"group\" : \"payments\",\n    \"state\" : \"stable\",\n    \"totalLag\" : 60,\n    \"overallTrend\" : \"growing\",\n    \"topics\" : 1\n  } ]\n}"
      }
    ],
    "isError": false
  }
}
```

## Connect from an AI client

Klag speaks standard **Streamable HTTP MCP** (JSON-RPC 2.0 over POST). Any client that
supports remote/HTTP MCP servers can connect — there's no Klag-specific SDK. You need
one thing: the endpoint URL, `http://<host>:8888/mcp` by default (`HTTP_PORT` +
`MCP_PATH`). If `MCP_AUTH_TOKEN` is set, add an `Authorization: Bearer <token>` header.
Klag implements MCP protocol version `2025-11-25`.

Use `https://` and a token for anything outside localhost.

> **Caution**
The snippets below use `Bearer <token>` placeholders. Project-level files like `.mcp.json`
and `.cursor/mcp.json` are often committed to source control — don't put a real token
there. Prefer an untracked global config (`~/.cursor/mcp.json`), your client's
environment-variable/secret interpolation, or a git-ignored file.


### Claude Code

```bash
claude mcp add --transport http klag https://klag.example.com/mcp \
  --header "Authorization: Bearer <token>"
```

Or add it to a project `.mcp.json`:

```json
{
  "mcpServers": {
    "klag": {
      "type": "http",
      "url": "https://klag.example.com/mcp",
      "headers": { "Authorization": "Bearer <token>" }
    }
  }
}
```

### Cursor

Add to `.cursor/mcp.json` (project) or `~/.cursor/mcp.json` (global):

```json
{
  "mcpServers": {
    "klag": {
      "url": "https://klag.example.com/mcp",
      "headers": { "Authorization": "Bearer <token>" }
    }
  }
}
```

### Codex

Add to `~/.codex/config.toml`:

```toml
[mcp_servers.klag]
url = "https://klag.example.com/mcp"
bearer_token_env_var = "KLAG_MCP_TOKEN"
```

`bearer_token_env_var` names the environment variable that holds the token, so it is never
written to `config.toml` — export it before launching Codex. The inline form
`http_headers = { Authorization = "Bearer <token>" }` works too, but stores the token on disk.

Older Codex builds only spoke stdio and needed the `mcp-remote` bridge
(`command = "npx"`, `args = ["mcp-remote", "https://klag.example.com/mcp"]`). Check your
Codex version's MCP docs if the `url` form isn't recognized.

### GitHub Copilot

**VS Code** — add to `.vscode/mcp.json` (workspace) or your user MCP config. VS Code uses
the top-level key `servers`, not `mcpServers`. Prefer `${input:...}` for tokens:

```json
{
  "inputs": [
    {
      "type": "promptString",
      "id": "klag-mcp-token",
      "description": "Klag MCP token",
      "password": true
    }
  ],
  "servers": {
    "klag": {
      "type": "http",
      "url": "https://klag.example.com/mcp",
      "headers": { "Authorization": "Bearer ${input:klag-mcp-token}" }
    }
  }
}
```

Switch Copilot Chat to **Agent** mode and confirm the server appears under the tools picker.

**Copilot CLI** — add to `~/.copilot/mcp-config.json`:

```json
{
  "mcpServers": {
    "klag": {
      "type": "http",
      "url": "https://klag.example.com/mcp",
      "headers": { "Authorization": "Bearer <token>" },
      "tools": ["*"]
    }
  }
}
```

Or from the terminal:

```shell
copilot mcp add --transport http \
  --header "Authorization: Bearer <token>" \
  klag https://klag.example.com/mcp
```

Visual Studio and JetBrains IDEs use the same `servers` object in `mcp.json`; remote
servers can also use `requestInit.headers` instead of `headers`.

### OpenCode

Add to `~/.config/opencode/opencode.json` (global) or `opencode.json` in your project.
OpenCode v2 nests servers under `mcp.servers`. Set `oauth: false` when Klag uses a bearer
token — otherwise OpenCode may attempt OAuth discovery first:

```json
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "servers": {
      "klag": {
        "type": "remote",
        "url": "https://klag.example.com/mcp",
        "oauth": false,
        "headers": {
          "Authorization": "Bearer {env:KLAG_MCP_TOKEN}"
        }
      }
    }
  }
}
```

Use `{env:VAR}` for secret interpolation. Older OpenCode builds place servers directly
under `mcp` with the same fields.

### Kilo Code

Open the MCP settings (`mcp_settings.json`) and add:

```json
{
  "mcpServers": {
    "klag": {
      "type": "streamable-http",
      "url": "https://klag.example.com/mcp",
      "headers": { "Authorization": "Bearer <token>" }
    }
  }
}
```

> **Note**
MCP clients evolve quickly and config field names change between versions. If a snippet
above doesn't match your client, consult its current MCP docs — Klag only requires a
Streamable-HTTP JSON-RPC POST endpoint, so any correct remote-MCP config will work.


Once connected, ask the agent to `list_consumer_groups`, `find_lagging_groups`, or
`diagnose` a specific group.

For `401`, `405`, or snapshot-not-ready responses, see
[Troubleshooting](https://klag.dev/guides/troubleshooting/#mcp-401-405-or-empty-snapshot).

## Design

The MCP layer reads from a `SnapshotStore` populated by the metrics collector, never
from direct Kafka calls. See the design doc:
[`docs/superpowers/specs/2026-06-01-mcp-support-design.md`](https://github.com/themoah/klag/blob/main/docs/superpowers/specs/2026-06-01-mcp-support-design.md).


---

# Klag vs AKHQ
URL: https://klag.dev/comparisons/klag-vs-akhq/

AKHQ is a Kafka web UI for browsing topics, messages, and groups; Klag is a headless consumer-lag metrics exporter. They solve different problems and are often run together.

**Klag** and [AKHQ](https://github.com/tchiotludo/akhq) are frequently mentioned
together but do different jobs. AKHQ is an interactive **web UI** for exploring a Kafka
cluster; Klag is a **headless metrics exporter** for consumer lag. AKHQ shows you lag in
a browser right now; Klag turns lag into time-series metrics you alert and trend on.

## What each tool is

- **Klag** is a lag metrics exporter. No UI — it publishes metrics to Prometheus,
  Datadog, or OTLP for dashboards and alerting.
- **AKHQ** is a Kafka web UI. It lets you browse and search topics and messages, view
  and manage consumer groups (including resetting offsets), inspect schemas and ACLs, and
  see current lag interactively.

## At a glance

| | **Klag** | **AKHQ** |
|---|---|---|
| Category | Lag metrics exporter | Web UI / console |
| Interface | Metrics endpoint | Interactive browser UI |
| Message browsing | ❌ | ✅ |
| Time-series lag & alerting | ✅ | ❌ (point-in-time view) |
| Runtime | Java / Vert.x / GraalVM | Java / Micronaut |
| License | Apache 2.0 | Apache 2.0 |

## Feature comparison

| Feature | Klag | AKHQ |
|---|:---:|:---:|
| Browse topics & messages | ❌ | ✅ |
| Manage groups / reset offsets | ❌ (read-only) | ✅ |
| Schema registry & ACL views | ❌ | ✅ |
| Current lag in a UI | ❌ | ✅ |
| Lag as time-series metrics | ✅ | ❌ |
| Lag velocity, [time-based lag](https://klag.dev/metrics/time-based-lag/) | ✅ | ❌ |
| [Retention / data-loss alerting](https://klag.dev/metrics/data-loss-prevention/) | ✅ | ❌ |
| Prometheus / Datadog / OTLP export | ✅ | ❌ |
| Alerting on lag trends | ✅ (via your stack) | ❌ |
| [MCP endpoint for AI agents](https://klag.dev/ai/mcp/) | ✅ | ❌ |

## When to use which

- **Use AKHQ** when a human needs to *look inside* the cluster — read messages, debug a
  payload, reset a group's offsets, inspect a schema.
- **Use Klag** when you need lag as **metrics**: continuous history, dashboards, and
  alerts that fire before anyone opens a UI. AKHQ's lag view is point-in-time and
  read-through-the-UI; it isn't a Prometheus/Datadog metrics source.

These are complementary. A common setup is AKHQ for interactive exploration **and** Klag
feeding lag metrics into Prometheus/Grafana for continuous alerting.

> Spotted something out of date? Open an issue or PR against
> [`website/`](https://github.com/themoah/klag/tree/main/website) and we'll fix it.


---

# Klag vs Burrow
URL: https://klag.dev/comparisons/klag-vs-burrow/

Klag and Burrow both monitor Kafka consumer lag. Burrow evaluates lag against status rules and sends notifications; Klag exports richer lag metrics to Prometheus, Datadog, and OTLP.

Both **Klag** and [Burrow](https://github.com/linkedin/Burrow) monitor Kafka consumer
lag with read-only access. They differ in philosophy: Burrow evaluates lag against
built-in status rules and pushes notifications; Klag exports a richer set of lag metrics
and leaves evaluation and alerting to your observability stack.

## What each tool is

- **Klag** is a lag *metrics exporter*. It polls the Kafka Admin API on an interval and
  publishes metrics to Prometheus, Datadog, or OTLP.
- **Burrow** is a lag *monitor with an evaluation engine*. It classifies each consumer
  group's status (OK / WARN / ERROR) using a sliding-window algorithm and can push
  notifications over email and HTTP. It exposes an HTTP API; Prometheus metrics come via
  a separate exporter.

## At a glance

| | **Klag** | **Burrow** |
|---|---|---|
| Category | Lag metrics exporter | Lag monitor + notifier |
| Runtime | Java / Vert.x / GraalVM | Go |
| Primary output | Prometheus, Datadog, OTLP | HTTP API + notifiers |
| Lag evaluation | In your alerting stack | Built-in status rules |
| License | Apache 2.0 | Apache 2.0 |

## Feature comparison

| Feature | Klag | Burrow |
|---|:---:|:---:|
| Lag in messages | ✅ | ✅ |
| Consumer group state | ✅ | ✅ |
| Lag status evaluation rules | ❌ | ✅ |
| Built-in notifiers (email/HTTP) | ❌ | ✅ |
| Lag velocity (growing/shrinking) | ✅ | ⚠️ status only |
| Time-based lag (ms) + time-to-catch-up | ✅ | ❌ |
| [Hot partition detection](https://klag.dev/metrics/hot-partitions/) | ✅ | ❌ |
| [Data-loss / retention alerting](https://klag.dev/metrics/data-loss-prevention/) | ✅ | ❌ |
| [Commit staleness](https://klag.dev/metrics/overview/) (stuck consumers) | ✅ | ❌ |
| [MCP endpoint for AI agents](https://klag.dev/ai/mcp/) | ✅ | ❌ |
| Prometheus native | ✅ | ⚠️ separate exporter |
| Datadog / OTLP native | ✅ | ❌ |
| Read-only DESCRIBE ACLs | ✅ | ✅ |

## When to use which

- **Use Burrow** if you want a self-contained tool that decides "is this group healthy?"
  and pushes a notification, without a metrics pipeline. Its status-window algorithm is
  its signature feature.
- **Use Klag** if you already run Prometheus / Grafana / Datadog and want lag as
  first-class metrics — plus velocity, time-based lag, retention risk, and hot-partition
  signals — that you alert on with your existing rules and dashboards. Klag also exposes
  a read-only [MCP endpoint](https://klag.dev/ai/mcp/) for AI agents, which Burrow does not.

Klag was inspired by the archived
[kafka-lag-exporter](https://klag.dev/getting-started/migrating-from-kafka-lag-exporter/); if you're
coming from that project, the migration guide maps every metric.

> Spotted something out of date? Open an issue or PR against
> [`website/`](https://github.com/themoah/klag/tree/main/website) and we'll fix it.


---

# Klag vs Confluent Control Center
URL: https://klag.dev/comparisons/klag-vs-confluent-control-center/

Confluent Control Center is the full Confluent Platform management UI; Klag is a lightweight open-source consumer-lag exporter for Prometheus, Datadog, and OTLP. They serve different needs.

**Klag** and
[Confluent Control Center](https://docs.confluent.io/platform/current/control-center/index.html)
sit at opposite ends of the tooling spectrum. Control Center is the comprehensive
management and monitoring UI for **Confluent Platform**; Klag is a single-purpose,
open-source **consumer-lag exporter** that works with any Kafka-compatible cluster.

## What each tool is

- **Klag** is an Apache-2.0 lag metrics exporter. It monitors consumer lag and ships
  metrics to Prometheus, Datadog, or OTLP. It runs against any Kafka-compatible cluster.
- **Confluent Control Center (C3)** is Confluent's commercial platform UI, part of
  Confluent Platform. It provides broker/cluster monitoring, consumer-lag views, topic
  and connector management, ksqlDB, schema management, and governance — through a web
  console.

## At a glance

| | **Klag** | **Confluent Control Center** |
|---|---|---|
| Category | Lag metrics exporter | Platform management UI |
| Scope | Consumer lag | Whole Confluent Platform |
| Interface | Metrics endpoint | Web console |
| Licensing | Open source (Apache 2.0) | Commercial (Confluent Platform) |
| Runs against | Any Kafka-compatible cluster | Primarily Confluent Platform |
| Footprint | ~44 MB RSS (native) | Substantial (platform component) |

## Feature comparison

| Feature | Klag | Control Center |
|---|:---:|:---:|
| Consumer-lag metrics | ✅ | ✅ |
| Web UI / console | ❌ | ✅ |
| Broker / cluster management | ❌ | ✅ |
| Connector & ksqlDB management | ❌ | ✅ |
| Lag velocity, [time-based lag](https://klag.dev/metrics/time-based-lag/) | ✅ | ⚠️ platform-dependent |
| [Retention / data-loss alerting](https://klag.dev/metrics/data-loss-prevention/) | ✅ | ⚠️ platform-dependent |
| Prometheus / Datadog / OTLP export | ✅ | ⚠️ via platform integrations |
| Open source | ✅ | ❌ |
| [MCP endpoint for AI agents](https://klag.dev/ai/mcp/) | ✅ | ❌ |

## When to use which

- **Use Control Center** if you run Confluent Platform and want an all-in-one console for
  managing and monitoring the whole stack (brokers, connectors, ksqlDB, governance) with
  Confluent support.
- **Use Klag** if you want a lightweight, open-source, vendor-neutral way to get consumer
  lag into Prometheus/Datadog/OTLP — on any Kafka, including open-source Apache Kafka,
  Strimzi, MSK, or Redpanda — without adopting a platform. Klag's footprint is tiny
  (native image ~44 MB RSS) and it adds signals like velocity, retention risk, and a
  read-only [MCP endpoint](https://klag.dev/ai/mcp/) for AI agents.

> Spotted something out of date? Control Center's feature set evolves with Confluent
> Platform. Open an issue or PR against
> [`website/`](https://github.com/themoah/klag/tree/main/website) and we'll fix it.


---

# Klag vs Cruise Control
URL: https://klag.dev/comparisons/klag-vs-cruise-control/

Cruise Control rebalances Kafka partitions and heals the cluster; Klag observes consumer lag read-only. Different jobs — you monitor with Klag and remediate with Cruise Control.

**Klag** and
[Cruise Control](https://github.com/linkedin/cruise-control) are not alternatives — they
operate on different halves of cluster operations. Klag **observes** (read-only consumer
lag), Cruise Control **acts** (rebalances partitions and heals the cluster). A healthy
setup often uses both: Klag tells you something is wrong, Cruise Control fixes a class of
those problems.

## What each tool is

- **Klag** is a read-only consumer-lag exporter. It never changes cluster state — it
  requires only `DESCRIBE` and emits metrics.
- **Cruise Control** is a cluster-balancing and self-healing tool. It analyzes broker
  load and partition distribution and executes **partition reassignments** to rebalance
  data, add/remove brokers, and fix skew. It performs write/admin operations on the
  cluster.

## At a glance

| | **Klag** | **Cruise Control** |
|---|---|---|
| Category | Lag observer (read-only) | Cluster rebalancer (read-write) |
| Acts on the cluster | ❌ (metrics only) | ✅ (reassigns partitions) |
| Consumer lag | ✅ | ❌ (broker-load focus) |
| Kafka permissions | `DESCRIBE` only | Admin / alter |
| Runtime | Java / Vert.x / GraalVM | Java |
| License | Apache 2.0 | BSD 2-Clause |

## Different problems

| Question | Tool |
|---|---|
| "Is a consumer group falling behind?" | **Klag** |
| "Will lag exceed retention before we catch up?" | **Klag** |
| "Are partitions unevenly distributed across brokers?" | **Cruise Control** |
| "Rebalance the cluster / drain a broker" | **Cruise Control** |

Klag's [hot partition detection](https://klag.dev/metrics/hot-partitions/) can flag a partition with
abnormal throughput — a *symptom* — but Klag will never move a partition. Remediating
broker/partition skew is exactly Cruise Control's job.

## When to use which

- **Use Cruise Control** to *fix* broker imbalance and partition skew, or to automate
  broker add/remove and self-healing.
- **Use Klag** to *watch* consumer lag and its trajectory (velocity, time-based lag,
  retention risk) and alert before consumers fall too far behind.

Because Klag is strictly read-only, it's safe to run alongside Cruise Control: it can't
interfere with reassignments — it just reports what lag is doing while they happen.

> Spotted something out of date? Open an issue or PR against
> [`website/`](https://github.com/themoah/klag/tree/main/website) and we'll fix it.


---

# Klag vs Grafana
URL: https://klag.dev/comparisons/klag-vs-grafana/

Grafana visualizes and alerts on metrics; Klag produces the Kafka consumer-lag metrics. They aren't alternatives — you run Klag as the data source and Grafana as the dashboard.

This is the one comparison where the answer is almost always "**both**." Grafana doesn't
produce metrics — it visualizes and alerts on metrics that something else produces. Klag
is one of those producers: it generates Kafka consumer-lag metrics that Grafana then
charts and alerts on.

## What each tool is

- **Klag** is a lag metrics *exporter* (the data source). It reads consumer lag from
  Kafka and exposes it to Prometheus, Datadog, or OTLP.
- **Grafana** is a visualization and alerting *frontend*. It queries data sources
  (Prometheus, Grafana Cloud, Datadog, etc.), renders dashboards, and evaluates alert
  rules. It does not collect Kafka lag itself.

## They're layers, not alternatives

| Layer | Tool |
|---|---|
| Collect lag from Kafka | **Klag** |
| Store the time series | Prometheus / Grafana Cloud / Datadog |
| Visualize & alert | **Grafana** |

Without a data source, Grafana has no Kafka lag to show. Without a frontend, Klag's
metrics still exist but aren't charted. You run them together.

## Using Klag with Grafana

Klag ships a pre-built Grafana dashboard covering lag, velocity, hot partitions,
time-based lag, data-loss prevention, ISR, and JVM panels:

- [Grafana dashboard](https://klag.dev/integrations/grafana-dashboard/) — import the ready-made dashboard.
- [OTLP & Grafana Cloud](https://klag.dev/integrations/otlp-grafana/) — send Klag metrics to Grafana
  Cloud over OpenTelemetry.
- [Prometheus](https://klag.dev/integrations/prometheus/) — scrape Klag and point Grafana at Prometheus.

## When you'd think it's "vs" — and why it isn't

If you've seen "Grafana for Kafka lag," that means Grafana **displaying** lag metrics
some exporter collected. Grafana + the Kafka data source you already run *is* the setup;
Klag is a strong choice for that exporter because it adds velocity, time-based lag,
retention risk, and hot-partition signals on top of raw lag — all of which render
directly in the bundled dashboard.

> Spotted something out of date? Open an issue or PR against
> [`website/`](https://github.com/themoah/klag/tree/main/website) and we'll fix it.


---

# Klag vs KMinion
URL: https://klag.dev/comparisons/klag-vs-kminion/

Klag and KMinion are both Prometheus exporters for Kafka. KMinion covers broader broker/topic metrics; Klag focuses on deeper consumer-lag signals and adds Datadog and OTLP output.

**Klag** and [KMinion](https://github.com/redpanda-data/kminion) are both Prometheus
exporters for Kafka. They overlap on consumer-lag basics but emphasize different things:
KMinion casts a wider net across broker/topic/log-dir metrics; Klag goes deeper on
consumer-lag analysis and ships to Datadog and OTLP as well as Prometheus.

## What each tool is

- **Klag** is a consumer-lag exporter focused on lag depth: velocity, time-based lag,
  retention risk, hot partitions, and commit staleness, exported to Prometheus, Datadog,
  or OTLP. Its cluster-adjacent scope is limited to partition offsets,
  under-replicated partitions, and topic partition counts; it has no broker or log-dir
  metrics.
- **KMinion** is a broad Prometheus exporter covering consumer-group lag plus broker,
  topic, and log-directory metrics. It also runs optional end-to-end
  produce/consume probes to measure roundtrip latency.

## At a glance

| | **Klag** | **KMinion** |
|---|---|---|
| Category | Consumer-lag exporter | General Prometheus exporter |
| Runtime | Java / Vert.x / GraalVM | Go |
| Primary output | Prometheus, Datadog, OTLP | Prometheus |
| Emphasis | Deep consumer-lag signals | Broad cluster metrics |
| License | Apache 2.0 | MIT |

## Feature comparison

| Feature | Klag | KMinion |
|---|:---:|:---:|
| Lag in messages | ✅ | ✅ |
| Consumer group state | ✅ | ✅ |
| Topic / partition metrics | ⚠️ limited | ✅ broad |
| Broker / log-dir metrics | ❌ | ✅ |
| End-to-end latency probes | ❌ | ✅ |
| Lag velocity (growing/shrinking) | ✅ | ❌ |
| Time-based lag (ms) + time-to-catch-up | ✅ | ❌ |
| [Hot partition detection](https://klag.dev/metrics/hot-partitions/) | ✅ | ❌ |
| [Data-loss / retention alerting](https://klag.dev/metrics/data-loss-prevention/) | ✅ | ❌ |
| [Commit staleness](https://klag.dev/metrics/overview/) (stuck consumers) | ✅ | ❌ |
| [MCP endpoint for AI agents](https://klag.dev/ai/mcp/) | ✅ | ❌ |
| Datadog / OTLP native | ✅ | ❌ |
| Read-only DESCRIBE ACLs | ✅ | ⚠️ writes for e2e probes |
| GraalVM native image (~44 MB RSS) | ✅ | ❌ |

## When to use which

- **Use KMinion** if you want one exporter for broad cluster health — broker metrics,
  topic configs, log-dir sizes, and optional end-to-end latency probes — all in
  Prometheus.
- **Use Klag** if consumer lag is the priority and you want the deeper signals (velocity,
  time-based lag, retention risk, hot partitions, commit staleness) plus Datadog/OTLP
  output and an [MCP endpoint](https://klag.dev/ai/mcp/) for AI agents. Note KMinion's end-to-end probes
  produce and consume test messages, so it needs write access; Klag is strictly
  read-only (`DESCRIBE`).

The two aren't mutually exclusive — some teams run KMinion for broker/topic breadth and
Klag for consumer-lag depth.

> Spotted something out of date? Open an issue or PR against
> [`website/`](https://github.com/themoah/klag/tree/main/website) and we'll fix it.


---

# Klag vs Redpanda Console
URL: https://klag.dev/comparisons/klag-vs-redpanda-console/

Redpanda Console is a web UI for browsing Kafka/Redpanda topics, messages, and groups; Klag is a headless consumer-lag metrics exporter. Different jobs, often run together.

**Klag** and
[Redpanda Console](https://github.com/redpanda-data/console) solve different problems.
Redpanda Console is an interactive **web UI** for Kafka and Redpanda clusters; Klag is a
**headless metrics exporter** for consumer lag. Despite the name, Redpanda Console works
with any Kafka-compatible cluster, not just Redpanda — and so does Klag.

## What each tool is

- **Klag** is a lag metrics exporter. No UI; it publishes metrics to Prometheus,
  Datadog, or OTLP.
- **Redpanda Console** is a web UI for exploring topics and messages, viewing consumer
  groups and their lag, inspecting schemas, and browsing cluster configuration. It
  presents lag interactively in the browser.

## At a glance

| | **Klag** | **Redpanda Console** |
|---|---|---|
| Category | Lag metrics exporter | Web UI / console |
| Interface | Metrics endpoint | Interactive browser UI |
| Message browsing | ❌ | ✅ |
| Time-series lag & alerting | ✅ | ❌ (point-in-time view) |
| Runtime | Java / Vert.x / GraalVM | Go |
| License | Apache 2.0 | BSL (enterprise features separately licensed) |

## Feature comparison

| Feature | Klag | Redpanda Console |
|---|:---:|:---:|
| Browse topics & messages | ❌ | ✅ |
| Consumer-group & lag views | ❌ | ✅ |
| Schema & config inspection | ❌ | ✅ |
| Current lag in a UI | ❌ | ✅ |
| Lag as time-series metrics | ✅ | ❌ |
| Lag velocity, [time-based lag](https://klag.dev/metrics/time-based-lag/) | ✅ | ❌ |
| [Retention / data-loss alerting](https://klag.dev/metrics/data-loss-prevention/) | ✅ | ❌ |
| Prometheus / Datadog / OTLP export | ✅ | ❌ |
| [MCP endpoint for AI agents](https://klag.dev/ai/mcp/) | ✅ | ❌ |

## When to use which

- **Use Redpanda Console** when a human needs to explore the cluster interactively —
  read messages, inspect a group, check a schema — through a polished UI.
- **Use Klag** when you need lag as **metrics**: continuous history, dashboards, and
  alerts that fire automatically. Redpanda Console shows lag point-in-time in the UI; it
  isn't a Prometheus/Datadog metrics source.

They're complementary: Redpanda Console for interactive exploration, Klag feeding lag
metrics into Prometheus/Grafana for continuous alerting.

> Spotted something out of date? Open an issue or PR against
> [`website/`](https://github.com/themoah/klag/tree/main/website) and we'll fix it.


---

# Klag vs other Kafka tools
URL: https://klag.dev/comparisons/overview/

How Klag compares to Burrow, KMinion, AKHQ, Confluent Control Center, Redpanda Console, Grafana, and Cruise Control for Kafka consumer lag monitoring.

"Kafka monitoring" covers several different jobs, and the popular tools each solve a
different one. Klag does exactly one: it **exports consumer-lag metrics** to your
observability stack. This page orients you across the landscape, then links to a
focused comparison for each tool.

## Which tool solves which problem

| Tool | Category | What it's for |
|---|---|---|
| **Klag** | Lag metrics exporter | Continuously export consumer lag, velocity, time-lag, retention risk, and group state to Prometheus / Datadog / OTLP. |
| [kafka-lag-exporter](https://klag.dev/getting-started/migrating-from-kafka-lag-exporter/) | Archived lag exporter | Former consumer-lag exporter; use the migration guide to map its names, labels, and units to Klag. |
| [Burrow](https://klag.dev/comparisons/klag-vs-burrow/) | Lag monitor + notifier | Evaluate lag against status rules and send email/HTTP notifications. |
| [KMinion](https://klag.dev/comparisons/klag-vs-kminion/) | Prometheus exporter | Export consumer-group **and** broker/topic/log-dir metrics to Prometheus. |
| [AKHQ](https://klag.dev/comparisons/klag-vs-akhq/) | Web UI / console | Browse topics, messages, schemas, and groups interactively. |
| [Confluent Control Center](https://klag.dev/comparisons/klag-vs-confluent-control-center/) | Platform UI | Full Confluent Platform management, monitoring, and governance UI. |
| [Redpanda Console](https://klag.dev/comparisons/klag-vs-redpanda-console/) | Web UI / console | Browse topics/messages/groups (Kafka and Redpanda). |
| [Grafana](https://klag.dev/comparisons/klag-vs-grafana/) | Visualization | Dashboards and alerts on metrics — including Klag's. |
| [Cruise Control](https://klag.dev/comparisons/klag-vs-cruise-control/) | Rebalancer | Rebalance partitions and heal the cluster. |

The consoles (AKHQ, Control Center, Redpanda Console) show lag in a UI but aren't
metric exporters or alerting sources. Grafana and Cruise Control do jobs Klag doesn't
touch — you typically run **Klag *with* them**, not instead of them. The closest
head-to-head comparisons are Burrow and KMinion, both dedicated lag/exporter tools.

## At a glance: the lag-focused tools

| | **Klag** | **Burrow** | **KMinion** |
|---|---|---|---|
| Runtime | Java / Vert.x / GraalVM | Go | Go |
| Primary output | Prometheus, Datadog, OTLP | HTTP API + notifiers | Prometheus |
| License | Apache 2.0 | Apache 2.0 | MIT |

## Feature comparison

| Feature | Klag | Burrow | KMinion |
|---|:---:|:---:|:---:|
| Lag in messages | ✅ | ✅ | ✅ |
| Consumer group state | ✅ | ✅ | ✅ |
| Lag velocity (growing/shrinking) | ✅ | ⚠️ status only | ❌ |
| AI agent endpoint (MCP) | ✅ | ❌ | ❌ |
| Time-based lag (seconds/minutes) | ✅ | ❌ | ❌ |
| Time-to-catch-up estimate | ✅ | ❌ | ❌ |
| Hot partition detection | ✅ | ❌ | ❌ |
| Data loss / retention alerting | ✅ | ❌ | ❌ |
| Commit staleness (stuck-consumer signal) | ✅ | ❌ | ❌ |
| Lag status evaluation rules | ❌ | ✅ | ❌ |
| Built-in notifiers (email/Slack/HTTP) | ❌ | ✅ | ❌ |
| Topic / partition metrics | ⚠️ limited | ❌ | ✅ broad |
| Broker / log-dir metrics | ❌ | ❌ | ✅ |
| Prometheus native | ✅ | ⚠️ separate exporter | ✅ |
| Datadog / OTLP native | ✅ | ❌ | ❌ |
| Read-only ACLs (DESCRIBE only) | ✅ | ✅ | ⚠️ writes for e2e |
| GraalVM native image (~44 MB RSS) | ✅ | ❌ | ❌ |
| Helm chart | ✅ | community | community |

## What Klag deliberately doesn't do

Klag is a metrics exporter, so it has **no web UI**, no message/topic browser, no
partition rebalancing, and no built-in alert-rules engine or notifiers. Those are the
jobs of the consoles, Cruise Control, and your alerting stack respectively. Klag emits
the raw signal and expects you to alert on it in Prometheus, Grafana, or Datadog.

Klag's Kafka scope is consumer progress plus partition offsets, under-replicated
partitions, and topic partition counts. It does not export broker health, topic
configuration, or log-directory metrics.

> Spotted something out of date? These tools all evolve. Open an issue or PR against
> [`website/`](https://github.com/themoah/klag/tree/main/website) and we'll fix it.


---

# Group Filtering
URL: https://klag.dev/configuration/group-filtering/

Control which Kafka consumer groups Klag monitors using glob include and exclude patterns.

By default Klag monitors **every** consumer group. On large or noisy clusters you
usually want to narrow that down. Two settings control it:

| Variable | Default | Meaning |
|---|---|---|
| `METRICS_GROUP_FILTER` | `*` | Comma-separated glob **include** patterns. |
| `METRICS_GROUP_EXCLUDE` | _(empty)_ | Comma-separated glob **exclude** patterns. |

A group is monitored **iff** it matches **any** include segment **and** **no** exclude
segment.

## Examples

Include only groups starting with `ingest` or `categorize`:

```bash
METRICS_GROUP_FILTER=ingest*,categorize*
```

Monitor everything except debug, canary, and shadow groups:

```bash
METRICS_GROUP_FILTER=*
METRICS_GROUP_EXCLUDE=debug-*,canary-*,*-shadow
```

## Filtering vs. ACLs

`METRICS_GROUP_FILTER` and `METRICS_GROUP_EXCLUDE` are **application-level** filters.
Cluster `DESCRIBE` permission is always required because `listConsumerGroups()` queries
all groups before filtering happens. See [ACL Permissions](https://klag.dev/kafka/acl-permissions/).

## Reducing broker load

Filtering reduces the number of `describe`/offset calls Klag makes. For very large
clusters, also tune `KAFKA_CHUNK_COUNT` and `KAFKA_CHUNK_DELAY_MS` (see the
[Configuration Reference](https://klag.dev/configuration/reference/)) to spread requests over time and
avoid spiking broker CPU.


---

# Configuration Reference
URL: https://klag.dev/configuration/reference/

Complete reference of every Klag environment variable across app, Kafka, metrics, hot partitions, time-based lag, MCP, OTLP, and logging.

Most Klag settings are environment variables. Kafka AdminClient properties additionally
support an optional `application.properties` file:

1. `application.properties` on the classpath, if you add one. Klag does **not** bundle
   this file.
2. An external file selected by `KLAG_CONFIG_FILE`.
3. `KAFKA_*` environment variables (highest precedence).

The properties files configure `kafka.*` keys; they are not a general configuration
source for every setting on this page.

Only settings read through Klag's `Env` helper support JVM system properties. Those keys
resolve in this order: environment variable `NAME` → `-DNAME` → dotted
`-Dname.dotted` (for example, `HTTP_PORT` → `-Dhttp.port=8881`). They are:

- `HTTP_PORT`, `KAFKA_HEALTH_CHECK_INTERVAL_MS`
- `KAFKA_CHUNK_COUNT`, `KAFKA_CHUNK_DELAY_MS`
- `METRICS_INTERVAL_MS`, `CONSUMER_MEMBER_LABELS_ENABLED`,
  `LAG_TREND_DEADBAND_MSG_PER_SEC`
- all `HOT_PARTITION_*` and `TIME_LAG_*` settings listed below
- `COMMIT_FRESHNESS_ENABLED`, `ISR_ENABLED`

Kafka forwarding, `KLAG_CONFIG_FILE`, Vert.x, reporter integrations, and MCP read
environment variables directly and do not use that `-D` resolution chain. Logging is a
separate exception: Logback can resolve exact-name JVM properties such as
`-DLOG_LEVEL=DEBUG`, but it does not provide `Env`-style dotted aliases such as
`-Dlog.level`.

## Application

| Variable | Default | Description |
|---|---|---|
| `HTTP_PORT` | `8888` | HTTP server port. |
| `KAFKA_HEALTH_CHECK_INTERVAL_MS` | `30000` | Health-check interval. |
| `VERTX_USE_VIRTUAL_THREADS` | `true` | Use virtual threads for verticle deployment. Set `false` for the event-loop model. Environment only. |
| `KLAG_CONFIG_FILE` | _(unset)_ | Path to an external `application.properties` file containing `kafka.*` properties. Environment only. |

## Kafka

| Variable | Default | Description |
|---|---|---|
| `KAFKA_BOOTSTRAP_SERVERS` | `localhost:9092` | Broker addresses. |
| `KAFKA_REQUEST_TIMEOUT_MS` | `30000` | Request timeout. |
| `KAFKA_CHUNK_COUNT` | `1` | Split offset requests into N batches. |
| `KAFKA_CHUNK_DELAY_MS` | `0` | Delay (ms) between batches. |

Any `KAFKA_X_Y_Z` environment variable is mapped to `kafka.x.y.z` and forwarded to the
Kafka AdminClient. For example, `KAFKA_SECURITY_PROTOCOL` becomes
`kafka.security.protocol`. This generic forwarding is environment-only; in a properties
file, use the `kafka.*` key directly.

For SASL/SSL, common settings include `KAFKA_SECURITY_PROTOCOL`,
`KAFKA_SASL_MECHANISM`, and `KAFKA_SASL_JAAS_CONFIG`. See
[Installation](https://klag.dev/getting-started/installation/) and
[ACL Permissions](https://klag.dev/kafka/acl-permissions/).

## Metrics

| Variable | Default | Description |
|---|---|---|
| `METRICS_REPORTER` | `none` | `none`, `prometheus`, `datadog`, or `otlp`. |
| `METRICS_INTERVAL_MS` | `60000` | How often to collect metrics. |
| `METRICS_GROUP_FILTER` | `*` | Comma-separated glob include patterns. |
| `METRICS_GROUP_EXCLUDE` | _(empty)_ | Comma-separated glob exclude patterns. |
| `METRICS_JVM_ENABLED` | `false` | Export JVM metrics. |
| `CONSUMER_MEMBER_LABELS_ENABLED` | `true` | Tag consumer-owned per-partition lag metrics with `member_host` / `consumer_id` / `client_id` (kafka-lag-exporter parity). Set `false` to drop them and reduce cardinality. |
| `LAG_TREND_DEADBAND_MSG_PER_SEC` | `1.0` | STABLE band for the MCP lag-trend classifier. |
| `COMMIT_FRESHNESS_ENABLED` | `true` | Track inferred time since a lagging group/topic's committed-offset sum last changed. |
| `ISR_ENABLED` | `true` | Detect and report under-replicated partitions. |

A group is monitored **iff** it matches any include segment **and** no exclude segment.
See [Group Filtering](https://klag.dev/configuration/group-filtering/).

Commit freshness observes the sum of committed offsets across a group/topic's
partitions. Any change, including a rewind, resets its clock. Caught-up periods remove
the tracking baseline; it is established again when lag resumes. Restarting Klag also
resets observation.

See [Metrics Overview](https://klag.dev/metrics/overview/) for commit-staleness semantics and
[ISR Monitoring](https://klag.dev/metrics/isr/) for the under-replicated-partition metric.

## Hot partition detection

| Variable | Default | Description |
|---|---|---|
| `HOT_PARTITION_ENABLED` | `true` | Enable hot-partition detection. |
| `HOT_PARTITION_SIGMA_MULTIPLIER` | `2.0` | Std-devs for the outlier threshold. |
| `HOT_PARTITION_MIN_PARTITIONS` | `3` | Min partitions per topic for detection. |
| `HOT_PARTITION_MIN_SAMPLES` | `3` | Min samples for throughput calc. |
| `HOT_PARTITION_BUFFER_SIZE` | `20` | Samples retained per partition. |

## Time-based lag estimation

| Variable | Default | Description |
|---|---|---|
| `TIME_LAG_ENABLED` | `true` | Enable time-based lag estimation. |
| `TIME_LAG_MIN_MESSAGES` | `100` | Min lag messages for time-to-close estimates. |
| `TIME_LAG_INTERPOLATION_BUFFER_SIZE` | `60` | Offset/timestamp points per partition. |
| `TIME_LAG_STALE_PRODUCER_THRESHOLD_MS` | `180000` | Time before a producer is considered stale. |

## MCP (AI agent access)

| Variable | Default | Description |
|---|---|---|
| `MCP_ENABLED` | `false` | Expose the `/mcp` endpoint (opt-in). |
| `MCP_AUTH_TOKEN` | _(empty)_ | Require `Authorization: Bearer <token>` when set. |
| `MCP_PATH` | `/mcp` | HTTP path of the MCP endpoint. |

See [MCP Endpoint](https://klag.dev/ai/mcp/) for details.

## Datadog (when `METRICS_REPORTER=datadog`)

| Variable | Default | Description |
|---|---|---|
| `DD_API_KEY` | _(unset; required)_ | Datadog API key used for metric submission. |
| `DD_APP_KEY` | _(unset; optional)_ | Datadog application key used for metadata operations. |
| `DD_SITE` | `datadoghq.com` | Datadog site, such as `datadoghq.eu`. Defaults to `datadoghq.com`. |

These are environment-only. See [Datadog](https://klag.dev/integrations/datadog/).

## OTLP (when `METRICS_REPORTER=otlp`)

**Standard OpenTelemetry variables:**

| Variable | Description |
|---|---|
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Base endpoint (e.g. `http://localhost:4318`). |
| `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | Metrics-specific endpoint (overrides base). |
| `OTEL_EXPORTER_OTLP_HEADERS` | Auth headers (`key1=value1,key2=value2`). |
| `OTEL_EXPORTER_OTLP_METRICS_HEADERS` | Metrics-specific headers. |
| `OTEL_METRIC_EXPORT_INTERVAL` | Export interval (ms), default `60000`. |
| `OTEL_SERVICE_NAME` | Service name, default `klag`. |
| `OTEL_RESOURCE_ATTRIBUTES` | Additional resource attributes. |

**Custom variables (override `OTEL_*`):** `OTLP_ENDPOINT`, `OTLP_STEP_MS`,
`OTLP_HEADERS`, `OTLP_RESOURCE_ATTRIBUTES`. Protocol is HTTP only (port 4318);
temporality is cumulative. See [OTLP & Grafana Cloud](https://klag.dev/integrations/otlp-grafana/).

## Logging

Logging settings are interpreted directly by Logback. Set them as environment variables
or exact-name JVM properties (for example, `-DLOG_LEVEL=DEBUG`). Dotted `Env` aliases
such as `-Dlog.level` are not supported:

| Variable | Default | Logger |
|---|---|---|
| `LOG_LEVEL` | `INFO` | Root logger. |
| `LOG_LEVEL_KLAG` | `LOG_LEVEL`, then `INFO` | All Klag packages. |
| `LOG_LEVEL_KAFKA` | `INFO` | Klag's Kafka package. |
| `LOG_LEVEL_HEALTH` | `INFO` | Klag's health package. |
| `LOG_LEVEL_METRICS` | `INFO` | Klag's metrics package. |
| `LOG_LEVEL_VERTX` | `WARN` | Vert.x framework (`io.vertx`). |
| `LOG_LEVEL_KAFKA_CLIENT` | `INFO` | Apache Kafka client (`org.apache.kafka`). |
| `LOG_LEVEL_KAFKA_LIST_OFFSETS_HANDLER` | `ERROR` | Kafka `ListOffsetsHandler`; use `WARN` or `DEBUG` when investigating list-offset requests. |
| `LOG_LEVEL_NETTY_BOOTSTRAP` | `ERROR` | Netty `ServerBootstrap`. |

The broader `io.netty` logger is fixed at `WARN`; only `ServerBootstrap` has a
dedicated environment override.


---

# Kubernetes (Helm)
URL: https://klag.dev/deployment/kubernetes/

Deploy Klag on Kubernetes with the official Helm chart, including SASL authentication and ServiceMonitor support.

The official Helm chart is the recommended way to run Klag on Kubernetes. It is
published to a Helm repository served from GitHub Pages and indexed on
[Artifact Hub](https://artifacthub.io/packages/helm/klag/klag).

## Install

```bash
helm repo add klag https://themoah.github.io/klag
helm repo update
helm search repo klag   # find the latest version

helm install klag klag/klag \
  --set kafka.bootstrapServers="kafka-broker:9092"
```

For repeatable deployments, choose a release from
`helm search repo klag --versions` and pass `--version <chart-version>`.

## With SASL authentication

```bash
helm install klag klag/klag \
  --set kafka.bootstrapServers="kafka:9092" \
  --set kafka.securityProtocol="SASL_SSL" \
  --set kafka.saslMechanism="PLAIN" \
  --set kafka.saslJaasConfig="org.apache.kafka.common.security.plain.PlainLoginModule required username='user' password='pass';"
```

## From a local checkout (development)

```bash
helm install klag ./charts/klag \
  --set kafka.bootstrapServers="kafka-broker:9092"
```

## Configuration

The chart exposes Kafka connection, metrics reporter, resource limits, and (optionally) a
`ServiceMonitor` for the Prometheus Operator. See the
[chart README](https://github.com/themoah/klag/blob/main/charts/klag/README.md) for the
full list of values, and the [Configuration Reference](https://klag.dev/configuration/reference/) for the
underlying environment variables.

The chart defaults `metrics.reporter` to `prometheus`. The bare application defaults
`METRICS_REPORTER` to `none`, so non-Helm deployments must select a reporter.

## Production guidance

### Keep one replica unless you deduplicate

Klag has no leader election. Each replica polls the same Kafka data and reports the same
metrics. Keep `replicaCount: 1` unless your backend deduplicates them. Multiple
Prometheus replicas create pod-distinguished duplicate series; Datadog and OTLP replicas
double-report the values and may increase ingestion cost.

### Prometheus ServiceMonitor

Set the reporter and ServiceMonitor together:

```bash
helm upgrade --install klag klag/klag \
  --set kafka.bootstrapServers="kafka:9092" \
  --set metrics.reporter="prometheus" \
  --set serviceMonitor.enabled=true
```

The chart rejects a ServiceMonitor with a non-Prometheus reporter because `/metrics`
would return `404`. Add labels under `serviceMonitor.labels` when your Prometheus
Operator uses a label selector. See
[Troubleshooting](https://klag.dev/guides/troubleshooting/#prometheus-does-not-discover-the-servicemonitor)
for namespace and selector checks.

### Credentials and truststores

Use `kafka.existingSecret` for SASL credentials instead of putting JAAS text in release
values. Mount private truststores or CA material with `extraVolumes` and
`extraVolumeMounts`, then set `kafka.sslTruststoreLocation` to the mounted path. Inject
passwords and settings without first-class chart values through `extraEnv` secret
references or `extraEnvFrom`. The
[chart README](https://github.com/themoah/klag/blob/main/charts/klag/README.md)
documents the expected Secret keys and a truststore mount example.

### Probes and resources

The chart sends liveness checks to `/healthz` and readiness checks to `/readyz`.
`/healthz` only confirms that the process runs. `/readyz` returns the Kafka health
monitor's cached state, so increasing the probe `timeoutSeconds` does not give an
in-flight Kafka check more time. Keep the probe roles separate. For slow startup,
increase `readinessProbe.initialDelaySeconds` or `readinessProbe.failureThreshold`.
Tune `kafka.requestTimeoutMs` and `app.healthCheckIntervalMs` when Kafka requests or
health-state refreshes need different timing.

The defaults request `100m` CPU and `256Mi` memory and limit the container to `500m` CPU
and `512Mi` memory. Treat them as a starting point. Measure collection duration and
memory for your group and partition count, then raise requests or limits before reducing
`metrics.intervalMs`; overlapping demand can otherwise cause skipped collection ticks.

## Permissions

Klag needs only read-only Kafka access. See [ACL Permissions](https://klag.dev/kafka/acl-permissions/).
On Kubernetes-managed Kafka, [Strimzi](https://klag.dev/deployment/strimzi/) is fully supported.


---

# Native Image
URL: https://klag.dev/deployment/native-image/

Run Klag as a GraalVM native binary for ~70-100 ms startup and ~44 MB RSS, ideal for fast scaling and low-footprint deployments.

Klag publishes a GraalVM **native image** alongside the JVM image. It starts in
**~70–100 ms using ~44 MB RSS**, versus ~500 ms / ~119 MB for the JVM image, with the
same config, endpoints, and metrics.

## Run the published image

The native image is tagged `:native` and `:<version>-native`. Replace the example
broker address with one that resolves and is reachable from inside the container; use
Docker network or service DNS for Kafka in another container, or
`host.docker.internal:9092` for Kafka exposed by the host:

```bash
docker run -e KAFKA_BOOTSTRAP_SERVERS=broker.example.com:9092 \
           -e METRICS_REPORTER=prometheus \
           -p 8888:8888 \
           themoah/klag:native
```

## Build it yourself

Requires a GraalVM JDK 21 (LTS) with `native-image` (e.g.
`sdk install java 21.0.2-graalce`). Run Gradle with that JDK as `JAVA_HOME`/`GRAALVM_HOME`.

```bash
gradle nativeCompile          # -> build/native/nativeCompile/klag (standalone binary)
docker build -f Dockerfile.native -t klag:native .   # distroless runtime image
```

Benchmark startup and memory:

```bash
scripts/benchmark-startup.sh native - build/native/nativeCompile/klag
```

Configuration works like the JVM build. All settings accept environment variables, but
only settings read through Klag's `Env` helper accept runtime `-D` properties. Those
settings support the exact name and a dotted lowercase alias, such as
`-DHTTP_PORT=8881` or `-Dhttp.port=8881`:

```bash
./build/native/nativeCompile/klag -Dhttp.port=8881
```

Reporter integrations, Kafka forwarding, MCP, and other environment-only settings do
not gain `-D` support in a native build. Logging is a separate exception: Logback
accepts exact-name properties such as `-DLOG_LEVEL=DEBUG`, but not dotted aliases such
as `-Dlog.level`. See the
[configuration reference](https://klag.dev/configuration/reference/#application) for the full
`Env`-backed list and per-setting notes.

## How it's configured

Native config lives in `build.gradle.kts` (the `graalvmNative` block) plus reachability
hints in `src/main/resources/META-INF/native-image/`. Reflection metadata for Netty,
kafka-clients, logback, and Micrometer comes from the GraalVM Reachability Metadata
Repository (auto-enabled). The entry point is `KlagLauncher` (direct
`new MainVerticle()`, no reflective Vert.x launcher).

> **Note**
The runtime stays on **JDK 21**. JVM 25 (LTS) showed no startup/memory gain over 21 for
this workload (slightly higher RSS).


---

# Strimzi
URL: https://klag.dev/deployment/strimzi/

Klag works out of the box with Kafka clusters managed by the Strimzi operator on Kubernetes (KRaft).

Klag works with Kafka clusters managed by the [Strimzi](https://strimzi.io) operator,
the common production way to run Kafka on Kubernetes.

## Connecting

No special chart configuration is needed. Point Klag at the Strimzi bootstrap service:

```bash
helm install klag klag/klag \
  --set kafka.bootstrapServers=my-cluster-kafka-bootstrap:9092
```

The bootstrap service is named `<cluster>-kafka-bootstrap` in the namespace where your
Strimzi `Kafka` resource lives.

## Verified versions

A dedicated end-to-end suite installs the Strimzi operator, provisions a real KRaft Kafka
cluster via Strimzi CRDs, points the chart at the `*-kafka-bootstrap` service, generates
real lag, and asserts Klag scrapes it. It is verified against Strimzi-managed Kafka
**4.1.0** and **4.2.0** (the versions the current Strimzi operator supports).

```bash
# Single version
./scripts/e2e-strimzi-test.sh

# Specific Kafka version
KAFKA_VERSION=4.1.0 ./scripts/e2e-strimzi-test.sh

# Matrix across supported versions
./scripts/e2e-strimzi-matrix.sh 4.1.0 4.2.0
```

Both e2e suites run in CI on every PR. See [Build from Source](https://klag.dev/development/build/) for
the full test workflow.


---

# Build from Source
URL: https://klag.dev/development/build/

Build and test Klag from source with Gradle and Java 21, including Helm chart tests and end-to-end suites with real Kafka.

Klag is a Java 21 / Vert.x project built with Gradle.

## Requirements

- **Java 21** (use SDKMAN if needed: `sdk use java 21.0.9-tem`).
- Use `./gradlew` locally; CI uses `gradle` directly (no wrapper JAR committed).

## Common tasks

```bash
./gradlew compileJava   # Compile
./gradlew test          # Run tests
./gradlew assemble      # Package (fat JAR)
./gradlew run           # Run with hot-reload
```

## Helm chart template tests

Fast and offline. Lints the chart and renders every values permutation:

```bash
./scripts/test-helm-chart.sh
```

## End-to-end tests (k3d + real Kafka)

The canonical integration test spins up a disposable [k3d](https://k3d.io) cluster,
deploys a real single-node KRaft Kafka, builds the Klag image from the local
`Dockerfile`, installs the Helm chart, generates real lag, and asserts Klag exposes it
via `/metrics`. Nothing is mocked.

```bash
./scripts/e2e-test.sh                 # Full e2e + cleanup
./scripts/e2e-test.sh --auto-install  # Install k3d/kubectl/helm via Homebrew
./scripts/e2e-test.sh --skip-cleanup  # Keep the cluster for inspection
KLAG_IMAGE=themoah/klag:latest ./scripts/e2e-test.sh   # Test the current published image
KAFKA_IMAGE=apache/kafka:3.7.0 ./scripts/e2e-test.sh   # Test an older broker
./scripts/e2e-test.sh --cleanup       # Delete the test cluster
```

## Strimzi compatibility tests

See [Strimzi](https://klag.dev/deployment/strimzi/):

```bash
./scripts/e2e-strimzi-test.sh
KAFKA_VERSION=4.1.0 ./scripts/e2e-strimzi-test.sh
./scripts/e2e-strimzi-matrix.sh 4.1.0 4.2.0
```

Prerequisites: Docker, k3d, helm, kubectl (`--auto-install` installs via Homebrew). Both
e2e suites run in CI on every PR (`.github/workflows/e2e.yml`).

## Native image

See [Native Image](https://klag.dev/deployment/native-image/) for the GraalVM build.


---

# Contributing
URL: https://klag.dev/development/contributing/

How to contribute to Klag. Fork, branch, run the test suites, and open a pull request.

Contributions are welcome.

## Workflow

1. Fork the repository.
2. Create a feature branch.
3. Run the tests before submitting:
   ```bash
   ./gradlew test                # Java tests
   ./scripts/test-helm-chart.sh  # Helm chart template tests
   ./scripts/e2e-test.sh         # End-to-end (k3d + real Kafka)
   ```
4. Open a pull request.

See [Build from Source](https://klag.dev/development/build/) for the full set of build and test commands.

## Conventions

- Async operations return `Future<T>` (Vert.x); keep APIs non-blocking.
- Java 21 records for DTO-style model types.
- SLF4J + Logback for logging, no `stdout` from application code.
- Config resolution order: classpath → external file → environment variables.

## When adding a metric

Metric names and tags are part of the public API. When you add, rename, or retag a
metric, also update the reporter/collector and its tests, the `README.md` and `CLAUDE.md`
metric docs, the [Grafana dashboard](https://klag.dev/integrations/grafana-dashboard/)
(`dashboard/demo-dashboard.json`), and, if Helm/ServiceMonitor behavior changes, the
chart values, templates, README, and Helm tests.


---

# Installation
URL: https://klag.dev/getting-started/installation/

Install Klag via the Helm chart, Docker, or a Docker environment file, including SASL authentication examples.

Klag ships as a Docker image and a Helm chart. The bare application defaults
`METRICS_REPORTER` to `none`, so Docker and binary deployments must select
`prometheus`, `datadog`, or `otlp`. The Helm chart defaults `metrics.reporter` to
`prometheus`.

## Helm chart

The chart is published to a Helm repository served from GitHub Pages and indexed on
[Artifact Hub](https://artifacthub.io/packages/helm/klag/klag).

```bash
helm repo add klag https://themoah.github.io/klag
helm repo update
helm search repo klag   # find the latest version

helm install klag klag/klag \
  --set kafka.bootstrapServers="kafka-broker:9092"
```

For repeatable deployments, choose a version from `helm search repo klag --versions`
and pass it with `helm install --version <chart-version>`.


### SASL authentication

```bash
helm install klag klag/klag \
  --set kafka.bootstrapServers="kafka:9092" \
  --set kafka.securityProtocol="SASL_SSL" \
  --set kafka.saslMechanism="PLAIN" \
  --set kafka.saslJaasConfig="org.apache.kafka.common.security.plain.PlainLoginModule required username='user' password='pass';"
```


### Local checkout (dev)

```bash
helm install klag ./charts/klag \
  --set kafka.bootstrapServers="kafka-broker:9092"
```




See the [chart README](https://github.com/themoah/klag/blob/main/charts/klag/README.md)
for full configuration options, and [Kubernetes deployment](https://klag.dev/deployment/kubernetes/) for
production guidance.

## Docker with an environment file

```bash
docker run --env-file .env themoah/klag:latest
```

A sample `.env`:

```dotenv
# Kafka connection
KAFKA_BOOTSTRAP_SERVERS=instance.gcp.confluent.cloud:9092
KAFKA_SECURITY_PROTOCOL=SASL_SSL
KAFKA_SASL_MECHANISM=PLAIN
KAFKA_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=${SASL_USERNAME} password=${SASL_PASSWORD};"

# Metrics
METRICS_REPORTER=prometheus
METRICS_INTERVAL_MS=30000
METRICS_GROUP_FILTER=*
METRICS_GROUP_EXCLUDE=

# Optional: JVM metrics
METRICS_JVM_ENABLED=true
```

## Next steps

- [Configuration Reference](https://klag.dev/configuration/reference/): all environment variables.
- [ACL Permissions](https://klag.dev/kafka/acl-permissions/): the read-only grants Klag needs.
- [Troubleshooting](https://klag.dev/guides/troubleshooting/): missing metrics, readiness, filters, and
  ServiceMonitor discovery.
- [Build from Source](https://klag.dev/development/build/): for contributors.


---

# Introduction
URL: https://klag.dev/getting-started/introduction/

What Klag is, why consumer lag matters, and the key features of this Kafka lag exporter.

**Klag** is a Kafka consumer lag exporter built with [Vert.x](https://vertx.io) and
[Micrometer](https://micrometer.io). It continuously monitors consumer lag and
consumer-group state across your cluster and exposes the data to Prometheus, Datadog,
or any OTLP-compatible backend.

It is inspired by [kafka-lag-exporter](https://github.com/seglo/kafka-lag-exporter)
(archived in 2024), rebuilt on a modern reactive stack.

## Why consumer lag matters

Consumer lag is the gap between a partition's log-end offset and the consumer group's
**committed offset**. It is a useful proxy for work not yet acknowledged by the group,
not proof that every record before the commit completed business processing. Left
unmonitored, growing lag leads to:

- **Stale data** in downstream systems.
- **Memory pressure** as consumers struggle to catch up.
- **Silent failures** when consumer groups die without alerts.

Klag surfaces these problems early, with enough signal to act before users notice.
Monitoring lag and group health is an essential operational practice for production
Kafka; Klag is one exporter that can supply those signals to your existing observability
stack.

## Key features

| Feature | Why it matters |
|---|---|
| **Lag velocity** | Know if lag is growing or shrinking, to catch problems before they escalate. |
| **Time-based lag estimation** | See lag in seconds/minutes, beyond raw message counts. |
| **Hot partition detection** | Find partitions with uneven load causing bottlenecks. |
| **Consumer group state tracking** | Alert on `preparing_rebalance`, `completing_rebalance` (`assigning`/`reconciling` on KIP-848 groups), `dead`, or `empty` states. |
| **Request batching** | Safely monitor large clusters without overwhelming brokers. |
| **Stale group cleanup** | Automatically stops reporting deleted/inactive groups. |
| **Data loss prevention** | Catch the case where lag exceeds retention and data is lost. |

## Control collection load

Request batching with configurable delays lets you spread offset requests across a
collection cycle and control broker load on larger clusters.
See [Group Filtering](https://klag.dev/configuration/group-filtering/) and the `KAFKA_CHUNK_COUNT` /
`KAFKA_CHUNK_DELAY_MS` settings in the [Configuration Reference](https://klag.dev/configuration/reference/).

## Read-only by design

Klag requires **read-only** (DESCRIBE) access to Kafka, no write or alter permissions.
See [ACL Permissions](https://klag.dev/kafka/acl-permissions/) for the exact grants on self-managed
Kafka and Confluent Cloud.

## Next steps

- [Quick Start](https://klag.dev/getting-started/quick-start/): run Klag in one command.
- [Installation](https://klag.dev/getting-started/installation/): Helm, Docker, and env-file setups.
- [Metrics Overview](https://klag.dev/metrics/overview/): the full list of what Klag exposes.
- [Comparisons](https://klag.dev/comparisons/overview/): Klag vs Burrow, KMinion, AKHQ, Grafana, and more.


---

# Migrating from kafka-lag-exporter
URL: https://klag.dev/getting-started/migrating-from-kafka-lag-exporter/

Map kafka-lag-exporter metrics, labels, units, and configuration to Klag, including the dashboard and relabeling changes required during migration.

[kafka-lag-exporter](https://github.com/seglo/kafka-lag-exporter) was archived in 2024.
Klag is a maintained replacement for consumer-progress monitoring. Both read offsets
through Kafka's Admin API, but their metric names, labels, aggregation levels, and
time-lag units differ.

Use the mapping below, then update Prometheus relabeling, dashboards, and alerts before
cutting traffic over.

## Metric name mapping

Klag exports through Micrometer; the Prometheus names below are what you scrape (dots
become underscores):

| kafka-lag-exporter | Klag (Prometheus name) | Notes |
|---|---|---|
| `kafka_consumergroup_group_lag` | `klag_consumer_lag` | per partition |
| `kafka_consumergroup_group_topic_sum_lag` | `klag_consumer_lag_sum` | per group+topic |
| `kafka_consumergroup_group_sum_lag` | `sum by (consumer_group)(klag_consumer_lag_sum)` | Klag derives the group total from topic rollups |
| `kafka_consumergroup_group_max_lag` | `klag_consumer_lag_max` | per group+topic; group max: `max by (consumer_group)(klag_consumer_lag_max)` |
| `kafka_consumergroup_group_offset` | `klag_consumer_committed_offset` | committed offset |
| `kafka_partition_latest_offset` | `klag_partition_log_end_offset` | partition end |
| `kafka_partition_earliest_offset` | `klag_partition_log_start_offset` | partition start |
| `kafka_consumergroup_group_lag_seconds` | `klag_consumer_lag_ms` | Klag reports milliseconds; divide by 1000 for seconds |
| `kafka_consumergroup_group_max_lag_seconds` | `max by (consumer_group) (klag_consumer_lag_ms{partition=""}) / 1000` | Klag exposes topic-level max rollups separately from partition series |

The archived exporter has no minimum-lag aggregate. Klag's
`klag_consumer_lag_min` therefore has no source metric to map.

See the full [Metrics Overview](https://klag.dev/metrics/overview/) for everything Klag adds on top.

## Label mapping

The biggest change is the group label name:

| kafka-lag-exporter | Klag |
|---|---|
| `cluster_name` | No built-in equivalent; add a Prometheus target or external label if needed. |
| `group` | `consumer_group` |
| `topic` | `topic` |
| `partition` | `partition` |
| `member_host` | `member_host` |
| `consumer_id` | `consumer_id` |
| `client_id` | `client_id` |
| `is_simple_consumer` | No Klag equivalent. |

Klag exports group state as the `state` tag on a separate
`klag_consumer_group_state` metric.

The per-consumer-instance labels `member_host`, `consumer_id`, and `client_id` are **on by
default** in Klag and identify which consumer instance owns each partition (the same labels you used
to trace lag to a specific pod). They ride on per-partition `klag_consumer_lag`,
`klag_consumer_lag_ms`, and `klag_consumer_committed_offset` series. The topic-level
`klag_consumer_lag_ms` aggregate is not member-tagged because it is a rollup across partitions.

### Relabel during the transition

If your Grafana panels and alert rules reference `group=`, add a Prometheus relabel rule to alias
`consumer_group` to `group` while you update queries:

```yaml
scrape_configs:
  - job_name: klag
    metric_relabel_configs:
      - source_labels: [consumer_group]
        target_label: group
```

This rule only aliases the label. You still need to replace metric names, adjust group
and topic aggregations, and convert time lag from milliseconds to seconds where old
queries expect seconds. Once dashboards use `consumer_group`, remove the rule.

## Member labels and cardinality

Member labels multiply series by the number of consumer instances, and a partition's owner changes
on every rebalance (`consumer_id` rotates per session). Klag retires the old instance's series
within one or two scrape intervals, so churn is bounded — but on very large clusters the extra
cardinality adds up. To opt out and keep only `consumer_group` / `topic` / `partition`:

```bash
CONSUMER_MEMBER_LABELS_ENABLED=false
```

The partition-level `klag_partition_log_end_offset` / `log_start_offset` metrics never carry member
labels (they describe the partition, not a consumer), matching kafka-lag-exporter's
`kafka_partition_*`.

## Configuration mapping

kafka-lag-exporter uses a HOCON `application.conf`; Klag primarily uses environment
variables. Any `Env`-backed setting also accepts a JVM `-D` property (env wins when both
are set). Config precedence is classpath `application.properties`, then an external file
at `KLAG_CONFIG_FILE`, then environment variables. See the
[Configuration Reference](https://klag.dev/configuration/reference/).

| kafka-lag-exporter (`application.conf`) | Klag |
|---|---|
| `kafka-lag-exporter.clusters[].bootstrap-brokers` | `KAFKA_BOOTSTRAP_SERVERS` |
| `kafka-lag-exporter.poll-interval` | `METRICS_INTERVAL_MS` |
| `kafka-lag-exporter.client-group-id` / consumer props | `KAFKA_*` (mapped to `kafka.*` AdminClient props) |
| group whitelist (`group-whitelist`) | `METRICS_GROUP_FILTER` (comma-separated globs) |
| group blacklist (`group-blacklist`) | `METRICS_GROUP_EXCLUDE` (comma-separated globs) |
| `port` (Prometheus) | `HTTP_PORT` (scrape at `/metrics`) |
| reporter selection | `METRICS_REPORTER=prometheus` (also `datadog`, `otlp`) |

A minimal Prometheus setup is shown below. Replace the broker example with an address
that resolves and is reachable from the Klag runtime:

```bash
KAFKA_BOOTSTRAP_SERVERS=broker.example.com:9092
METRICS_REPORTER=prometheus
METRICS_INTERVAL_MS=30000
```

Klag needs only `DESCRIBE` on the cluster, groups, and topics. See
[Kafka ACL Permissions](https://klag.dev/kafka/acl-permissions/).
Deploy on Kubernetes with the [Helm chart](https://klag.dev/deployment/kubernetes/).

## Time-based lag is different

kafka-lag-exporter estimates lag-in-time by extrapolating offset/timestamp samples. Klag reports
`klag_consumer_lag_ms` from Kafka's own log timestamps via interpolation (with a poll-history
fallback), so it does not extrapolate beyond observed data. It also adds a
`klag_consumer_lag_time_to_close_seconds` estimate while a group is catching up. See
[Time-Based Lag](https://klag.dev/metrics/time-based-lag/).

## What Klag adds — and what it doesn't

**Adds:** [lag velocity](https://klag.dev/metrics/lag-velocity/), [hot partition detection](https://klag.dev/metrics/hot-partitions/),
[data-loss / retention alerting](https://klag.dev/metrics/data-loss-prevention/), commit freshness, Datadog and OTLP
sinks, and a read-only [MCP endpoint](https://klag.dev/ai/mcp/).

**Doesn't have:** kafka-lag-exporter had no built-in alerting rules or notifiers, and neither does
Klag — alert in Prometheus/Grafana on the exported metrics as before.


---

# Quick Start
URL: https://klag.dev/getting-started/quick-start/

Run Klag with Docker in one command, or use the GraalVM native image for faster startup and lower memory.

Monitoring consumer progress is essential in production. The fastest way to try Klag is
the published Docker image.

The bare application defaults `METRICS_REPORTER` to `none`, so select `prometheus`,
`datadog`, or `otlp` to start collection. The Helm chart defaults its reporter to
`prometheus`.

## Docker (JVM)

The broker address below is a replaceable example. Set it to an address that resolves
and is reachable from inside the Klag container. If Kafka runs in another container,
attach both containers to the same Docker network and use the Kafka service DNS name.
If Kafka is exposed by your host, use `host.docker.internal:9092` on Docker Desktop.

```bash
docker run -e KAFKA_BOOTSTRAP_SERVERS=broker.example.com:9092 \
           -e METRICS_REPORTER=prometheus \
           -p 8888:8888 \
           themoah/klag:latest
```

Metrics are then available at `http://localhost:8888/metrics`.

## Native image (faster startup, lower memory)

A GraalVM native build is published alongside the JVM image, tagged `:native` and
`:<version>-native`:

```bash
docker run -e KAFKA_BOOTSTRAP_SERVERS=broker.example.com:9092 \
           -e METRICS_REPORTER=prometheus \
           -p 8888:8888 \
           themoah/klag:native
```

The native binary starts in **~70–100 ms using ~44 MB RSS**, versus ~500 ms / ~119 MB
for the JVM image, ideal for fast scaling and low-footprint deployments. Same config,
endpoints, and metrics. See [Native Image](https://klag.dev/deployment/native-image/) for build details.

## HTTP endpoints

Once running, Klag exposes:

| Endpoint | Purpose |
|---|---|
| `/healthz` | Liveness probe (always 200). |
| `/readyz` | Readiness probe (200 if Kafka UP, 503 if DOWN). |
| `/metrics` | Prometheus scrape endpoint (when enabled). |
| `/version` | Build information. |
| `/mcp` | [MCP endpoint](https://klag.dev/ai/mcp/) for AI agents (when `MCP_ENABLED=true`). |

## What's next

- [Installation](https://klag.dev/getting-started/installation/): Helm chart, Docker env-file.
- [Configuration Reference](https://klag.dev/configuration/reference/): every environment variable.
- [Metrics Overview](https://klag.dev/metrics/overview/): the full metric catalog.
- [Troubleshooting](https://klag.dev/guides/troubleshooting/): missing metrics, Kafka readiness, ACLs,
  MCP, and ServiceMonitor discovery.


---

# Common Kafka consumer monitoring mistakes
URL: https://klag.dev/guides/consumer-monitoring-mistakes/

Six recurring mistakes teams make when monitoring Kafka consumers — alerting on absolute lag, ignoring velocity and group state, no retention headroom, per-pod blind spots — and how to avoid them.

Kafka consumer monitoring tends to fail the same ways. Here are the recurring mistakes
and what to do instead. Each maps to a concrete signal you can put on a dashboard.

## 1. Alerting on absolute lag only

A fixed threshold ("page if lag > 100,000") produces false alarms during normal traffic
spikes and stays silent when a slow leak is building under the threshold. Absolute lag
has no sense of direction.

**Instead:** alert on [lag velocity](https://klag.dev/metrics/lag-velocity/) — sustained positive
velocity means the group is losing ground regardless of the current number. Combine a
generous absolute ceiling with a velocity condition.

## 2. Ignoring whether lag is growing or shrinking

The same lag value is benign if it's draining and an incident if it's climbing. Teams
that chart only the current value can't tell the two apart.

**Instead:** always chart the trend, and prefer [time-based lag](https://klag.dev/metrics/time-based-lag/)
(lag in minutes + time-to-catch-up) over raw counts — it answers "how long behind, and
closing or not?"

## 3. Not monitoring consumer-group state

A group can go `empty` (all consumers gone) or thrash through
`preparing_rebalance` and `completing_rebalance` while its lag looks static. If you only
watch lag, a group that reaches `dead` can stay quiet until the backlog is huge.

**Instead:** select `klag_consumer_group_state` by its lowercase `state` tag. The real
values are `stable`, `preparing_rebalance`, `completing_rebalance`, `assigning`,
`reconciling`, `empty`, `dead`, and `unknown`; there is no generic `rebalancing` state.
Groups on the KIP-848 consumer protocol report `assigning`/`reconciling` instead of the
classic rebalance pair, so rebalance alerts must match all four. The gauge value is a
state-change count, not an encoded state number.

For example, detect an empty or dead group by label presence:

```promql
count by (consumer_group) (
  klag_consumer_group_state{state=~"empty|dead"}
) > 0
```

Use an alert `for` duration longer than one or two collection intervals because the
previous state-tagged series is retired asynchronously after a transition. Sustained
`preparing_rebalance` / `completing_rebalance` or repeated transitions are early
warnings. Klag's [MCP `diagnose`](https://klag.dev/ai/mcp/) tool raises a state-churn warning after
three retained transitions. Because that history is not time-windowed, inspect
`recentTransitions` before concluding that a group is flapping or in a rebalance storm.

## 4. No retention headroom alerting

Consumer lag and topic retention are usually watched separately, so nobody notices when
lag is about to exceed retention and messages start being deleted unread. By the time
lag "looks bad," data may already be gone.

**Instead:** alert on [retention percent](https://klag.dev/metrics/data-loss-prevention/)
(`klag_consumer_lag_retention_percent`). It tells you how much of the retention window
the lag has eaten. The raw gauge is percentage × 100, so raw `8000` means 80%.
For a topic-level alert at 80% or above, either compare
`klag_consumer_lag_retention_percent{partition=""} >= 8000` or normalize it:

```promql
(klag_consumer_lag_retention_percent{partition=""} / 100) >= 80
```

This stays well below the 100% (raw `10000`) data-loss cliff and avoids mixing topic
aggregates with per-partition series.

## 5. Per-pod / per-partition blind spots

Aggregating lag to the group level hides a single wedged consumer instance or one hot
partition. The group total looks fine; one pod is stuck.

**Instead:** keep per-partition series and the per-consumer member labels
(`member_host` / `consumer_id` / `client_id`, on by default) so you can trace lag to the
owning instance. Watch [hot-partition](https://klag.dev/metrics/hot-partitions/) outliers, not just sums.
See [migrating from kafka-lag-exporter](https://klag.dev/getting-started/migrating-from-kafka-lag-exporter/)
for the member-label details.

## 6. Assuming low lag means healthy

A crashed or wedged consumer that stops committing shows **flat, low lag** when no new
messages arrive — indistinguishable from "caught up." Lag alone can't see it.

**Instead:** watch commit staleness — see
[How to detect stuck consumers automatically](https://klag.dev/guides/detect-stuck-consumers/).

## The short version

| Mistake | Better signal |
|---|---|
| Absolute-lag threshold only | Lag velocity + generous ceiling |
| Charting current value only | Time-based lag + trend |
| Ignoring group state | `klag_consumer_group_state`, rebalance flags |
| No retention headroom | Retention percent alert |
| Group-level aggregation only | Per-partition + member labels + hot partitions |
| "Low lag = healthy" | Commit staleness |

## Next

- [Why monitoring the lag value alone isn't enough](https://klag.dev/guides/why-lag-value-is-not-enough/)
- [How to detect stuck consumers automatically](https://klag.dev/guides/detect-stuck-consumers/)


---

# How to detect stuck consumers automatically
URL: https://klag.dev/guides/detect-stuck-consumers/

A stuck Kafka consumer has pending work but a frozen committed offset — and raw lag often misses it. Learn to detect stuck consumers with commit staleness and Klag's diagnose tool.

A **stuck consumer** is one that still has work to do but has stopped making progress —
its committed offset is frozen while messages wait. The dangerous part is that raw lag
often fails to catch it. This guide explains why, and how to detect stuck consumers
automatically.

## Stuck vs merely lagging

- **Lagging:** the consumer is behind but still committing — the offset advances, lag
  rises and falls with load. It may well catch up on its own.
- **Stuck:** the committed offset has stopped advancing. The consumer is wedged — a
  poison message, a deadlock, a hung downstream call, an OOM loop, a paused partition.

The two look different in one specific way: for a stuck consumer the **committed offset
is flat over time**, even though lag is greater than zero.

## Why lag alone misses it

Lag = log-end offset − committed offset. Consider a wedged consumer:

- If producers keep writing, lag climbs — but so does lag for any slow consumer, so a
  simple threshold can't distinguish "wedged" from "busy."
- If producers **stop or slow**, the log-end offset stops advancing too. Now lag is
  flat and possibly small — and a wedged consumer looks identical to a caught-up one.

That second case is the trap: **flat, low lag can mean "healthy" or "stuck," and lag
can't tell you which.**

## The signal that catches it: commit staleness

The distinguishing fact is *time since the committed offset last advanced*. A caught-up
consumer has zero pending work; a stuck one has pending work and a frozen offset.

Klag's [commit-staleness metric](https://klag.dev/metrics/overview/)
(`klag_consumer_commit_staleness_seconds`) measures exactly this: seconds since a
group+topic's committed offset last moved. Crucially, it's **only reported while lag >
0** — a consumer that's genuinely idle (no lag, no commits) is not stuck, so it's not
flagged. Rising commit staleness with non-zero lag is the stuck-consumer fingerprint.

> Note: Kafka exposes no commit timestamp, so staleness is *inferred* — Klag timestamps
> when it observes the committed offset change, and the clock resets on Klag restart. It
> measures time since Klag last *saw* a commit, not absolute commit time.

## Alerting on it

A simple, robust rule:

```promql
# Consumer with pending work whose offset hasn't advanced in 5 minutes
klag_consumer_commit_staleness_seconds > 300
```

Because the metric only exists while lag > 0, this fires only for consumers that have
something to do and aren't doing it. Tune the threshold to your commit cadence — set it
comfortably above your normal auto-commit interval and processing time.

## Automatic detection with the diagnose tool

Klag's [MCP `diagnose` tool](https://klag.dev/ai/mcp/) applies this automatically. For a given group it
combines state, lag trend, retention risk, and hot partitions into a severity
assessment, and **flags stuck consumers** (lag > 0 with a frozen committed offset) using
a fixed 300-second threshold — so an SRE agent can ask "diagnose group X" and get the
stuck signal without hand-writing PromQL. It also raises a state-churn warning after
three retained transitions. That history is not time-windowed, so inspect
`recentTransitions` before treating the warning as a rebalance storm or flapping.

## Checklist

- Chart the **committed offset** over time — flat while lag > 0 is the tell.
- Alert on `klag_consumer_commit_staleness_seconds > <your threshold>`.
- Watch consumer-group **state** for `Empty` / repeated `Rebalancing` alongside it.
- For ad-hoc triage, use the MCP [`diagnose`](https://klag.dev/ai/mcp/) tool.

## Next

- [Common Kafka consumer monitoring mistakes](https://klag.dev/guides/consumer-monitoring-mistakes/)
- [Why monitoring the lag value alone isn't enough](https://klag.dev/guides/why-lag-value-is-not-enough/)


---

# How Kafka consumer lag actually works
URL: https://klag.dev/guides/how-kafka-consumer-lag-works/

A plain-English explanation of Kafka consumer lag — offsets, committed vs log-end offset, per-partition accounting — and what the lag number really measures.

Consumer lag is the most-watched Kafka health signal, and also the most
misunderstood. This guide explains what lag actually is, mechanically, so the metrics
make sense.

## Offsets: Kafka's ruler

Every message in a Kafka partition has a monotonically increasing **offset** — 0, 1, 2,
and so on. Offsets are per-partition, not per-topic. Two things move along this ruler:

- **Log-end offset (LEO):** the offset of the next message a producer will write. It's
  the "head" of the partition — how far *production* has reached.
- **Committed offset:** the offset a consumer group has recorded as processed, stored in
  the internal `__consumer_offsets` topic. It's how far *consumption* has reached.

Consumer lag for one partition is simply:

```text
lag = log-end offset − committed offset
```

If the log-end offset is 1,000 (producers have written through offset 999) and the group
has committed offset 950, the group is **50 messages behind** on that partition.

## Lag is per-partition, then aggregated

A consumer group subscribes to topics, and each topic has partitions. Lag is measured
**per partition**, then rolled up:

- Per group + topic: sum of partition lags (total backlog), plus max (the worst
  partition).
- Per group: sum across all its topics.

This matters because a group can look "fine" on total lag while one partition is badly
behind — a [hot partition](https://klag.dev/metrics/hot-partitions/) or a stuck assignment. Klag exposes
both the per-partition series and the aggregates so you can drill in. See the
[Metrics Overview](https://klag.dev/metrics/overview/) for the exact metric names
(`klag_consumer_lag`, `klag_consumer_lag_sum`, `klag_consumer_lag_max`).

## Committed ≠ processed, exactly

The committed offset reflects what the consumer has *committed*, which depends on the
commit strategy (auto-commit interval, or manual commit after processing). With
auto-commit, a consumer can commit an offset slightly ahead of what it has fully
processed. Lag is therefore a close proxy for "unprocessed messages," not a
cryptographic guarantee. For most alerting this distinction doesn't matter; for
exactly-once reasoning it does.

## Why a raw lag count can mislead

"50,000 messages behind" means nothing without throughput. At 50,000 msg/s it's one
second of backlog; at 50 msg/s it's a growing incident. That's why message-count lag
alone is a weak signal — see
[Why monitoring lag value alone isn't enough](https://klag.dev/guides/why-lag-value-is-not-enough/).

## How lag is read from Kafka

A lag exporter like Klag never joins the consumer group. It uses the Kafka **Admin API**
(read-only, `DESCRIBE`) to fetch each group's committed offsets and each partition's
log-end offset, subtracts, and reports. No messages are consumed, no offsets are written.
See [ACL Permissions](https://klag.dev/kafka/acl-permissions/) for the exact grants.

## Next

- [Why monitoring the lag value alone isn't enough](https://klag.dev/guides/why-lag-value-is-not-enough/)
- [Common Kafka consumer monitoring mistakes](https://klag.dev/guides/consumer-monitoring-mistakes/)
- [How to detect stuck consumers automatically](https://klag.dev/guides/detect-stuck-consumers/)


---

# Troubleshooting
URL: https://klag.dev/guides/troubleshooting/

Diagnose missing metrics, Kafka readiness and ACL failures, filtering, MCP responses, and Prometheus ServiceMonitor discovery.

Monitoring Kafka consumer progress is essential in production. Start with Klag's logs,
`/healthz`, `/readyz`, and the reporter-specific endpoint or backend.

## `/metrics` returns 404

**Likely cause:** The bare application defaults `METRICS_REPORTER` to `none`, or you
selected the `datadog` or `otlp` push reporter. Klag registers `/metrics` only for
Prometheus.

**Fix:** Set `METRICS_REPORTER=prometheus` and restart Klag. The Helm chart already
defaults `metrics.reporter` to `prometheus`. See
[Prometheus](https://klag.dev/integrations/prometheus/) and the
[Configuration Reference](https://klag.dev/configuration/reference/#metrics).

## Metrics or the MCP snapshot are empty after startup

**Likely cause:** The first Kafka collection has not completed, Kafka access failed, or
no group remains after filtering. The collector starts one cycle immediately; velocity
and some derived metrics still need more samples.

If Kafka is unreachable at startup, Klag **stays running (degraded)** instead of
exiting — `/readyz` returns `503` until connectivity is restored and a cycle succeeds.
This is intentional: the process tolerates a broker outage at boot like the health
monitor, so a transient Kafka blip does not crash-loop the pod.

**Fix:** Check the logs for a completed collection or Kafka errors. Verify
`KAFKA_BOOTSTRAP_SERVERS`, the [required ACLs](https://klag.dev/kafka/acl-permissions/), and the
[group filters](https://klag.dev/configuration/group-filtering/). Wait for a successful cycle. If the
cluster is large, allow longer than `METRICS_INTERVAL_MS`.

## `/readyz` returns 503

**Likely cause:** Klag's Kafka health check cannot complete `describeCluster`. Common
causes are an unreachable broker, bad TLS/SASL settings, or missing cluster
`DESCRIBE`.

**Fix:** Test network and DNS access from the Klag container, then compare your Kafka
security variables with [Installation](https://klag.dev/getting-started/installation/). Grant the
cluster ACL documented in [ACL Permissions](https://klag.dev/kafka/acl-permissions/). `/healthz` can
still return 200 because it only checks that the process is running.

## Logs show Kafka authorization errors

**Likely cause:** The principal lacks `DESCRIBE` on the cluster, a consumer group, or a
topic. Group filtering does not remove the need for cluster `DESCRIBE`.

**Fix:** Grant the read-only permissions in
[ACL Permissions](https://klag.dev/kafka/acl-permissions/). If you grant prefixed group or topic
access, align those prefixes with `METRICS_GROUP_FILTER`.

## No consumer groups appear

**Likely cause:** `METRICS_GROUP_FILTER` matches none of the group IDs, or
`METRICS_GROUP_EXCLUDE` removes every included group.

**Fix:** Temporarily use:

```bash
METRICS_GROUP_FILTER=*
METRICS_GROUP_EXCLUDE=
```

Then add patterns back one at a time. Excludes run after includes. See
[Group Filtering](https://klag.dev/configuration/group-filtering/).

## Old series never disappear from `/metrics`

**Likely cause:** Stale-gauge cleanup only runs after a **complete** collection cycle. If
one group fails every cycle — a missing group `DESCRIBE` ACL, a wedged coordinator — the
cycle is permanently partial and cleanup never runs, so series for deleted groups, topics,
and rotated consumer members linger indefinitely. The log carries
`Failed to collect lag for group <id> (skipped this cycle)` and
`Collection cycle was partial` every interval, naming the group.

This is deliberate: cleaning up against an incomplete key set would delete live series.
Stale values beat deleted ones — but the freeze lasts as long as the failure does.

**Fix:** Grant the group the [required ACLs](https://klag.dev/kafka/acl-permissions/), or drop it with
`METRICS_GROUP_EXCLUDE`. Cleanup resumes on the next complete cycle. The MCP snapshot is
unaffected: it keeps publishing the groups that did succeed.

## A deleted topic's series stay, or a live topic's series disappear

Klag filters each cycle's topic set against the cluster's topic list before requesting
metadata, because the Kafka admin call fails as a whole if any topic in the batch is
unknown — and a group's committed offsets outlive a deleted topic until
`offsets.retention.minutes` (7 days by default). Without the filter, one deleted topic
would keep every cycle partial, and cleanup frozen, for that long.

The cost is an ACL asymmetry: `listTopics` only returns topics the principal can see. If
Klag can read a group's committed offsets for a topic it cannot describe, that topic looks
deleted and its series are retired within 1–2 cycles. The log records
`Skipping N topic(s) absent from the cluster topic list`.

**Fix:** Grant topic `DESCRIBE` matching the group access you granted — see
[ACL Permissions](https://klag.dev/kafka/acl-permissions/). Asymmetric grants are the only case where a
live topic goes missing this way.

## Collection cycles overrun the interval on large clusters

**Likely cause:** Each cycle issues one batched `listOffsets` per request type covering
every partition in scope. The Kafka AdminClient splits that per leader broker, and the
timestamp lookups it performs scale with **partitions per broker**, so a cluster with few
brokers and many partitions can approach `KAFKA_REQUEST_TIMEOUT_MS` (default 30000). The
log shows `Skipping collection tick: previous cycle still running`.

**Fix:** Set `KAFKA_CHUNK_COUNT` above 1. Chunking splits the work into partition-weighted
batches processed sequentially, with `KAFKA_CHUNK_DELAY_MS` between them. Raising
`METRICS_INTERVAL_MS` or `KAFKA_REQUEST_TIMEOUT_MS` also helps. See the
[Configuration Reference](https://klag.dev/configuration/reference/#kafka).

## Velocity or time-lag metrics are missing at first

**Likely cause:** Lag velocity needs three collection samples. The time-lag fallback
needs at least two poll intervals when Kafka log timestamps cannot support primary
interpolation. Time-to-close also requires shrinking lag and at least
`TIME_LAG_MIN_MESSAGES` messages.

**Fix:** Wait for the required samples. Shorten `METRICS_INTERVAL_MS` only if Kafka can
handle the added polling load. See [Lag Velocity](https://klag.dev/metrics/lag-velocity/) and
[Time-Based Lag](https://klag.dev/metrics/time-based-lag/).

## MCP 401, 405, or empty snapshot

**Likely cause:**

- `401`: `MCP_AUTH_TOKEN` is set and the Bearer token is missing or wrong.
- `405`: The client sent `GET`; Klag accepts JSON-RPC 2.0 over `POST`.
- Snapshot not ready: metrics collection is disabled or the first cycle has not
  succeeded, so no snapshot exists yet.
- Snapshot empty (`groupCount: 0`): reports ran, but the group filters left no
  groups to collect. Each cycle publishes a refreshed empty snapshot rather than
  returning stale data from an earlier run.

**Fix:** Send `Authorization: Bearer <token>`, use a Streamable HTTP MCP client that
posts JSON-RPC requests, and select a reporter with `METRICS_REPORTER`. Then resolve
any Kafka or filtering problem reported in the logs. See [MCP Endpoint](https://klag.dev/ai/mcp/).

## Prometheus does not discover the ServiceMonitor

**Likely cause:** `serviceMonitor.enabled` is false, the reporter is not Prometheus, or
the Prometheus Operator's ServiceMonitor selector does not match the resource labels.
The chart rejects `serviceMonitor.enabled=true` with a non-Prometheus reporter.

**Fix:** Install or upgrade with:

```bash
helm upgrade --install klag klag/klag \
  --set kafka.bootstrapServers="kafka:9092" \
  --set metrics.reporter="prometheus" \
  --set serviceMonitor.enabled=true
```

If your Prometheus resource selects a label such as `release: monitoring`, add
`--set serviceMonitor.labels.release=monitoring`. Confirm that the Prometheus
namespace and object selectors permit the ServiceMonitor and that its endpoint selects
the Klag Service. See [Kubernetes deployment](https://klag.dev/deployment/kubernetes/#prometheus-servicemonitor).


---

# Why monitoring the lag value alone isn't enough
URL: https://klag.dev/guides/why-lag-value-is-not-enough/

A single consumer-lag number hides direction, time-to-impact, and data-loss risk. Learn which extra signals — velocity, time-based lag, retention percent — give lag its missing context.

Most Kafka monitoring stops at one number: consumer lag in messages. It's necessary but
not sufficient. A lag value on its own can't tell you whether things are getting better
or worse, how bad the impact is, or whether you're about to lose data. This guide covers
the context a raw lag count is missing — and the metrics that supply it.

## Problem 1: a number has no direction

Lag of 50,000 is meaningless without a trend. Is it draining or climbing?

- Static lag that's shrinking → the consumer is catching up; probably fine.
- The same lag climbing → an incident in progress.

**The missing signal is velocity** — the rate of change of lag. Klag's
[lag velocity](https://klag.dev/metrics/lag-velocity/) metric (`klag_consumer_lag_velocity`) is positive
when a group is falling behind and negative when catching up. Alerting on velocity
catches problems while lag is still small.

## Problem 2: messages aren't time

"50,000 messages behind" could be one second or one hour of delay depending on
throughput. Business impact is measured in **time**, not message count. Klag's
[time-based lag](https://klag.dev/metrics/time-based-lag/) (`klag_consumer_lag_ms`) estimates lag in
milliseconds by mapping the committed offset to a wall-clock timestamp, and
`klag_consumer_lag_time_to_close_seconds` estimates how long until the group catches up.
"12 minutes behind, closing in 3" is far more actionable than "50,000."

## Problem 3: lag hides data-loss risk

Kafka topics have retention. If lag grows faster than the consumer drains it, the
consumer can fall behind the partition's **log-start offset** — the oldest retained
message — and messages are deleted **before they're ever read**. Lag in messages doesn't
tell you how close you are to that cliff.

Klag's [retention-percent metric](https://klag.dev/metrics/data-loss-prevention/)
(`klag_consumer_lag_retention_percent`) reports how much of the retention window the lag
has consumed. At 80% you still have headroom; at 100% data loss has already occurred.
This is a fundamentally different alert than "lag is high."

## Problem 4: aggregate lag hides skew

Total lag across a topic can look healthy while one partition is badly behind — a
[hot partition](https://klag.dev/metrics/hot-partitions/) or a wedged assignment. Watch per-partition
lag and hot-partition outliers, not just the sum.

## Problem 5: frozen offsets look like low lag

A consumer that has crashed or wedged mid-batch can stop committing entirely. If no new
messages arrive, lag stays flat and low — and looks fine — while nothing is being
processed. See [How to detect stuck consumers automatically](https://klag.dev/guides/detect-stuck-consumers/)
for the commit-staleness signal that catches this.

## A better alerting model

| Instead of alerting on… | Also alert on… |
|---|---|
| Absolute lag > N | Lag **velocity** > 0 sustained |
| Absolute lag > N | Lag **in time** (ms / minutes) |
| Absolute lag > N | **Retention percent** approaching 100% |
| Group total lag | **Per-partition** / hot-partition outliers |
| Lag value | **Commit staleness** (stuck consumers) |

Raw lag is the starting point, not the finish line. Layering these signals turns "the
number is high" into "here's what's happening, how bad, and how long you have."

## Next

- [Common Kafka consumer monitoring mistakes](https://klag.dev/guides/consumer-monitoring-mistakes/)
- [How to detect stuck consumers automatically](https://klag.dev/guides/detect-stuck-consumers/)


---

# Datadog
URL: https://klag.dev/integrations/datadog/

Ship Klag metrics directly to Datadog using the Datadog Micrometer registry.

Klag can submit metrics straight to Datadog via the Micrometer Datadog registry, with no
Prometheus scrape required.

## Enable

Set the reporter, Kafka collection interval, required API key, and site. The broker
address is a replaceable example and must resolve and be reachable from inside the Klag
container; use Docker network or service DNS for Kafka in another container, or
`host.docker.internal:9092` for Kafka exposed by the host:

```bash
docker run -e KAFKA_BOOTSTRAP_SERVERS=broker.example.com:9092 \
           -e METRICS_REPORTER=datadog \
           -e METRICS_INTERVAL_MS=30000 \
           -e DD_API_KEY='<your-api-key>' \
           -e DD_SITE=datadoghq.com \
           themoah/klag:latest
```

`DD_API_KEY` is required for metric submission. `DD_APP_KEY` is optional and is used
for metadata operations; add `-e DD_APP_KEY='<your-application-key>'` when you want
them. `DD_SITE` defaults to `datadoghq.com`. Use your Datadog site, such as
`datadoghq.eu`, when needed.

`DD_API_KEY`, `DD_APP_KEY`, and `DD_SITE` are environment-only. Klag's
`MicrometerConfig` reads them directly from the process environment; classpath and
external properties files do not configure these credentials.

## Helm

Store credentials in an existing Secret so they do not appear in Helm release values:

```bash
kubectl create secret generic klag-datadog \
  --from-literal=api-key='<your-api-key>' \
  --from-literal=app-key='<your-application-key>'

helm install klag klag/klag \
  --set kafka.bootstrapServers="kafka:9092" \
  --set metrics.reporter="datadog" \
  --set metrics.intervalMs=30000 \
  --set metrics.datadog.existingSecret="klag-datadog" \
  --set metrics.datadog.site="datadoghq.com"
```

The default Secret keys are `api-key` and `app-key`. Change
`metrics.datadog.secretKeys.apiKey` or `metrics.datadog.secretKeys.appKey` if your
Secret uses different keys. See the
[chart README](https://github.com/themoah/klag/blob/main/charts/klag/README.md) for
all Datadog values.

## Notes

- `METRICS_INTERVAL_MS` controls how often Klag polls Kafka and refreshes its meters.
  It does not configure the Datadog registry's submission step.
- All [metrics](https://klag.dev/metrics/overview/) carry `consumer_group`, `topic`, and `partition`
  tags where applicable, so you can build Datadog monitors per group or topic.


---

# Grafana Dashboard
URL: https://klag.dev/integrations/grafana-dashboard/

Import Klag's pre-built Grafana dashboard for consumer lag, velocity, hot partitions, time-based lag, data-loss prevention, and JVM panels.

Klag ships a comprehensive, pre-built Grafana dashboard. It's published on
[Grafana.com](https://grafana.com/grafana/dashboards/25379-klag-kafka-lag-monitoring/)
(dashboard ID **25379**) and lives in the repo at
[`dashboard/demo-dashboard.json`](https://github.com/themoah/klag/blob/main/dashboard/demo-dashboard.json).

![Klag Grafana dashboard](../../../assets/grafana-dashboard.png)

## Import

Pick either path:

**By dashboard ID (recommended)**

1. In Grafana, go to **Dashboards → New → Import**.
2. Enter ID **25379**, click **Load**.
3. Select your OTLP/Prometheus-compatible data source, click **Import**.

**From the repo**

1. In Grafana, go to **Dashboards → New → Import**.
2. Upload `dashboard/demo-dashboard.json`.
3. Select your OTLP/Prometheus-compatible data source.
4. Adjust the refresh interval and time range as needed.

## What's included

- **Consumer Lag Overview**: real-time lag by group with color-coded thresholds.
- **Lag Velocity Tracking**: is lag growing or shrinking over time.
- **Consumer Group Health**: state table with alerts for unhealthy states.
- **Partition & Offset Details**: topic throughput and per-partition lag.
- **Hot Partition Detection**: count, table, and time series.
- **Under-Replicated Partitions (ISR)**: count, table, and time series.
- **Time-Based Lag**: max time lag, groups catching up, time-to-close charts.
- **Commit Staleness**: time since Klag last observed each lagging group and topic
  advance its committed offset.
- **Data Loss Prevention**: retention-risk and at-risk topics panels.
- **JVM panels**: memory, GC pause, threads, CPU, allocation rate, loaded classes.
- **Template variables**: filter by consumer group and topic; auto-refresh every minute.

## Requirements

- Klag running with `METRICS_REPORTER=otlp` (or `prometheus`).
- Metrics flowing to Grafana Cloud or a Prometheus-compatible backend.
- A data source configured in Grafana with PromQL support.


---

# OTLP & Grafana Cloud
URL: https://klag.dev/integrations/otlp-grafana/

Export Klag metrics over OpenTelemetry (OTLP/HTTP) to Grafana Cloud, New Relic, or any OTLP-compatible backend.

Klag speaks OTLP (OpenTelemetry Protocol) over HTTP, so it works with Grafana Cloud,
New Relic, and any OTLP-compatible metrics backend.

## Enable

```bash
METRICS_REPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_SERVICE_NAME=klag
```

Protocol is **HTTP only** (port `4318`) and aggregation temporality is **cumulative**.

## Grafana Cloud

```bash
METRICS_REPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64-encoded-credentials>
OTEL_SERVICE_NAME=klag-production
```

## Local OpenTelemetry Collector

```bash
METRICS_REPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_SERVICE_NAME=klag-dev
OTEL_RESOURCE_ATTRIBUTES=environment=development,cluster=local
```

## Configuration variables

Klag honours the standard `OTEL_*` variables and also provides custom `OTLP_*`
overrides:

| Standard | Custom override | Purpose |
|---|---|---|
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `OTLP_ENDPOINT` | Endpoint URL. |
| `OTEL_METRIC_EXPORT_INTERVAL` | `OTLP_STEP_MS` | Export interval (ms). |
| `OTEL_EXPORTER_OTLP_HEADERS` | `OTLP_HEADERS` | Auth headers (`key1=value1,key2=value2`). |
| `OTEL_RESOURCE_ATTRIBUTES` | `OTLP_RESOURCE_ATTRIBUTES` | Resource attributes. |

`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` / `OTEL_EXPORTER_OTLP_METRICS_HEADERS` override the
base endpoint/headers for metrics specifically. The custom `OTLP_*` variables take
precedence over the `OTEL_*` ones. See the full
[Configuration Reference](https://klag.dev/configuration/reference/#otlp-when-metrics_reporterotlp).


---

# Prometheus
URL: https://klag.dev/integrations/prometheus/

Scrape Klag metrics with Prometheus via the /metrics endpoint.

Prometheus needs no push gateway or backend credentials. The Helm chart selects it by
default; the bare application defaults `METRICS_REPORTER` to `none`, so Docker and
binary deployments must set `METRICS_REPORTER=prometheus`.

## Enable

Replace the example broker address with one that resolves and is reachable from inside
the Klag container. Use Docker network or service DNS for Kafka in another container,
or `host.docker.internal:9092` for Kafka exposed by the host.

```bash
docker run -e KAFKA_BOOTSTRAP_SERVERS=broker.example.com:9092 \
           -e METRICS_REPORTER=prometheus \
           -p 8888:8888 \
           themoah/klag:latest
```

Metrics are then served at `http://localhost:8888/metrics` in Prometheus text format.

## Scrape config

```yaml
scrape_configs:
  - job_name: klag
    metrics_path: /metrics
    static_configs:
      - targets: ['klag:8888']
```

On Kubernetes, the [Helm chart](https://klag.dev/deployment/kubernetes/) can render a `ServiceMonitor`
for the Prometheus Operator. See the chart's `values.yaml`.

If `/metrics` returns `404` or Prometheus cannot discover the ServiceMonitor, see
[Troubleshooting](https://klag.dev/guides/troubleshooting/).

## What you get

All [metrics](https://klag.dev/metrics/overview/) are exposed with `consumer_group`, `topic`, and
`partition` labels where applicable. Import the
[Grafana dashboard](https://klag.dev/integrations/grafana-dashboard/) for ready-made panels.


---

# ACL Permissions
URL: https://klag.dev/kafka/acl-permissions/

The read-only Kafka ACLs Klag needs (DESCRIBE on cluster, topics, and groups) for self-managed Kafka and Confluent Cloud.

Klag requires **read-only** access. It uses only the Kafka Admin Client API with
`DESCRIBE` permissions. No write or alter access is ever needed.

## Required permissions

| Resource | Name | Permission | Operations |
|---|---|---|---|
| CLUSTER | kafka-cluster | DESCRIBE | Health check, list consumer groups. |
| TOPIC | `*` or prefixed | DESCRIBE | Get partition info and offsets. |
| GROUP | `*` or prefixed | DESCRIBE | Get group state and committed offsets. |

> **Note**
Cluster `DESCRIBE` is **always** required, even when using
[group filtering](https://klag.dev/configuration/group-filtering/), because `listConsumerGroups()`
queries all groups before app-level filtering happens.


## Self-managed Kafka


### All groups & topics

```bash
# Cluster permissions (required)
kafka-acls --bootstrap-server <broker> \
  --add --allow-principal User:<klag-user> \
  --operation Describe --cluster

# All topics
kafka-acls --bootstrap-server <broker> \
  --add --allow-principal User:<klag-user> \
  --operation Describe --topic '*'

# All consumer groups
kafka-acls --bootstrap-server <broker> \
  --add --allow-principal User:<klag-user> \
  --operation Describe --group '*'
```


### Prefixed only

```bash
# Cluster permissions (required for listConsumerGroups)
kafka-acls --bootstrap-server <broker> \
  --add --allow-principal User:<klag-user> \
  --operation Describe --cluster

# Topics with prefix
kafka-acls --bootstrap-server <broker> \
  --add --allow-principal User:<klag-user> \
  --operation Describe --topic 'myapp-' \
  --resource-pattern-type prefixed

# Consumer groups with prefix
kafka-acls --bootstrap-server <broker> \
  --add --allow-principal User:<klag-user> \
  --operation Describe --group 'myapp-' \
  --resource-pattern-type prefixed
```




## Confluent Cloud

Create a service account with the following ACLs using the
[Confluent CLI](https://docs.confluent.io/confluent-cli/current/overview.html).


### All groups & topics

```bash
CLUSTER_ID=<your-cluster-id>
SERVICE_ACCOUNT=<service-account-id>

# Cluster permissions
confluent kafka acl create --allow --service-account $SERVICE_ACCOUNT \
  --operations describe --cluster-scope

# All topics
confluent kafka acl create --allow --service-account $SERVICE_ACCOUNT \
  --operations describe --topic '*'

# All consumer groups
confluent kafka acl create --allow --service-account $SERVICE_ACCOUNT \
  --operations describe --consumer-group '*'
```


### Prefixed only

```bash
CLUSTER_ID=<your-cluster-id>
SERVICE_ACCOUNT=<service-account-id>

# Cluster permissions (required for listConsumerGroups)
confluent kafka acl create --allow --service-account $SERVICE_ACCOUNT \
  --operations describe --cluster-scope

# Topics with prefix
confluent kafka acl create --allow --service-account $SERVICE_ACCOUNT \
  --operations describe --topic 'myapp-' --prefix

# Consumer groups with prefix
confluent kafka acl create --allow --service-account $SERVICE_ACCOUNT \
  --operations describe --consumer-group 'myapp-' --prefix
```




If Klag logs authorization failures or `/readyz` returns `503`, work through the
[readiness and ACL checks](https://klag.dev/guides/troubleshooting/#readyz-returns-503).


---

# Data Loss Prevention
URL: https://klag.dev/metrics/data-loss-prevention/

Klag's retention-percent metric warns you before consumer lag exceeds Kafka retention and messages are permanently lost.

The most dangerous kind of lag is the kind that crosses your **retention window**. Once
a consumer falls further behind than Kafka retains, the oldest unread messages are
deleted, gone, unrecoverable. Klag warns you **before** that happens.

## The metric

| Metric | Description |
|---|---|
| `klag.consumer.lag.retention_percent` | Percentage of the retention window consumed by lag, capped at 100% and exported as percentage × 100 for precision. |

The raw gauge is scaled: **raw `8000` means 80%**, and raw `10000` means the
100% retention boundary or possible data loss. It does not prove loss by itself.
Divide by `100` in queries and dashboards when you want a conventional 0–100 percentage.

Klag emits both:

- a topic aggregate (the maximum risk across its partitions), tagged with
  `consumer_group` and `topic`
- per-partition series, additionally tagged with `partition`

The aggregate omits the `partition` label. In PromQL, `{partition=""}` matches a label
that is absent, so use it for one series per group/topic and `{partition!=""}` for
partition detail.

## Formula

```text
retention_window = logEndOffset - logStartOffset

if retention_window <= 0:
    emit no metric
else if committedOffset < logStartOffset:
    calculated_percent = 100
else if lag <= 0:
    calculated_percent = 0
else:
    calculated_percent = (lag / retention_window) * 100

exported_gauge = round(calculated_percent * 100)
```

- **A rising value** means the consumer is eating into its safety margin.
- At `committedOffset == logStartOffset`, the metric reaches **100%** but the earliest
  retained record is still readable.
- At `committedOffset < logStartOffset`, unread records have been deleted and data loss
  has occurred. This case is also capped at raw `10000`, so the metric cannot distinguish
  it from the readable 100% boundary.

## Why offsets, not time

Retention in Kafka is enforced by the broker deleting old segments. Comparing lag to the
actual span of **available** offsets (`logEndOffset − logStartOffset`) measures the real,
current safety margin, more reliable than assuming a fixed time-based retention.

## Alerting

For a topic-level visualization in conventional percent units:

```promql
klag_consumer_lag_retention_percent{partition=""} / 100
```

For per-partition investigation:

```promql
klag_consumer_lag_retention_percent{partition!=""} / 100
```

Alert well below 100% (for example, at 80% or above) to leave time to intervene. Either
compare the raw scaled value:

```promql
klag_consumer_lag_retention_percent{partition=""} >= 8000
```

or normalize first:

```promql
(klag_consumer_lag_retention_percent{partition=""} / 100) >= 80
```

Keeping `partition=""` in topic-level rules avoids evaluating both the aggregate and
each per-partition series. The
[Grafana dashboard](https://klag.dev/integrations/grafana-dashboard/) includes retention-risk panels and
an at-risk topics table.


---

# Hot Partitions
URL: https://klag.dev/metrics/hot-partitions/

Klag detects partitions with statistically abnormal throughput so you can find skewed load and bottlenecks within a topic.

Within a single topic, partitions should carry roughly even load. When one partition
runs much hotter than its peers, usually from a skewed partition key, it becomes a
bottleneck. Klag detects these **statistical outliers**.

## Metrics

Reported **only when an outlier exists** (so they stay quiet on healthy topics):

| Metric | Description |
|---|---|
| `klag.hot_partition` | Partition throughput × 100 when statistically high. |
| `klag.hot_partition.lag` | Partition lag on a hot partition specifically. |

`klag.hot_partition` has only `topic` and `partition` tags, because throughput is
partition-level and independent of any consumer. `klag.hot_partition.lag` is
consumer-specific and has `consumer_group`, `topic`, and `partition` tags.

## How detection works

For each topic, Klag computes per-partition throughput over a rolling sample buffer and
flags partitions whose throughput exceeds the mean by more than
`HOT_PARTITION_SIGMA_MULTIPLIER` standard deviations.

Detection only runs when there is enough data to be meaningful:

| Variable | Default | Role |
|---|---|---|
| `HOT_PARTITION_ENABLED` | `true` | Master switch. |
| `HOT_PARTITION_SIGMA_MULTIPLIER` | `2.0` | Std-devs above mean to flag an outlier. |
| `HOT_PARTITION_MIN_PARTITIONS` | `3` | Min partitions per topic before detection runs. |
| `HOT_PARTITION_MIN_SAMPLES` | `3` | Min samples needed for a throughput estimate. |
| `HOT_PARTITION_BUFFER_SIZE` | `20` | Samples retained per partition. |

## Acting on it

A hot partition usually points at a partitioning-key problem in the producer. Use the
[Grafana dashboard](https://klag.dev/integrations/grafana-dashboard/) hot-partition panels to spot which
`topic`/`partition` is skewed, then rebalance the key or repartition the topic.


---

# ISR Monitoring
URL: https://klag.dev/metrics/isr/

Klag detects partitions whose in-sync replica set has shrunk below the full replica set, an early warning sign of reduced fault tolerance or brewing data loss.

Kafka replicates each partition to `replicas` brokers, but only tracks brokers as
"in-sync" (ISR) if they're caught up with the leader. When the ISR set shrinks below the
full replica set, the partition has less fault tolerance than configured — and if it
shrinks to zero, the partition has no durable replica to fail over to.

Klag detects this using topic metadata it already fetches every cycle (`describeTopics`) —
no additional Kafka calls or ACLs beyond what klag already requires.

## Metrics

Reported **only when a partition is under-replicated** (so it stays quiet on a healthy cluster):

| Metric | Description |
|---|---|
| `klag.partition.under_replicated` | Missing in-sync replica count (`replicaCount - inSyncReplicaCount`). |

`klag.partition.under_replicated` has only `topic` and `partition` tags — ISR status is
partition-level and independent of any consumer group.

## Configuration

| Variable | Default | Role |
|---|---|---|
| `ISR_ENABLED` | `true` | Master switch. |

## Scope

Klag flags under-replication (`isr.size() < replicas.size()`) only. It does not compare
against `min.insync.replicas` (that would require an additional `describeConfigs` call and
ACL) — under-replication is a strict superset signal that fires earlier, before a producer
would actually start seeing `NotEnoughReplicasException`.

## Acting on it

A shrinking ISR set usually means a broker is down, overloaded, or falling behind on
replication (disk I/O, network partition, GC pause). Check broker health for the missing
replica(s); if `inSyncReplicaCount` reaches 0, treat it as urgent — a leader failure at that
point loses the partition's unflushed data. Klag's [`diagnose` MCP tool](https://klag.dev/ai/mcp/)
surfaces under-replicated partitions that appear in the selected consumer group's
consumed partition set as a WARNING (or CRITICAL when ISR count is 0). ISR itself
remains a partition-level signal, not a consumer-group metric.


---

# Lag Velocity
URL: https://klag.dev/metrics/lag-velocity/

How Klag measures whether consumer lag is growing or shrinking over time, so you can catch problems before they escalate.

A single lag number tells you where a consumer is **right now**. **Lag velocity** tells
you where it's **heading**: whether the consumer is falling behind or catching up.

## The metric

| Metric | Description |
|---|---|
| `klag.consumer.lag.velocity` | Rate of change of lag in **messages/second × 100**. |

- **Positive** velocity → lag is growing, the consumer is falling behind.
- **Negative** velocity → lag is shrinking, the consumer is catching up.
- **Near zero** → lag is stable.

The value is multiplied by 100 so fractional messages/second survive integer-style
metric pipelines.

## Why it matters

Lag that is high but **shrinking** is often fine; the consumer is recovering from a
blip. Lag that is **low but growing fast** can be the early warning of an outage. Alert
on velocity as well as absolute lag, to catch problems while there is still time to act.

## Related: lag trend for AI agents

The [MCP endpoint](https://klag.dev/ai/mcp/) derives a coarse **lag trend** (`growing` / `shrinking` /
`stable`) per topic plus an `overallTrend` rollup from this velocity, using the
`LAG_TREND_DEADBAND_MSG_PER_SEC` dead-band (default `1.0`). Velocity within the band is
classified STABLE.


---

# Metrics Overview
URL: https://klag.dev/metrics/overview/

The full catalog of metrics Klag exposes, covering consumer lag, offsets, group state, velocity, hot partitions, time-based lag, and data-loss prevention.

Klag exports its metrics through Micrometer, so the exact name format depends on the
reporter (Prometheus, Datadog, or OTLP). The logical metrics are the same everywhere.

Tags depend on the metric's scope. Consumer metrics commonly use `consumer_group`,
topic-level metrics add `topic`, and only partition-level series add `partition`.
Broker/topic signals such as throughput and ISR do not have a `consumer_group` tag.

## Core lag and offsets

| Metric | Description |
|---|---|
| `klag.consumer.lag` | Current lag per partition (also `.sum`, `.max`, `.min`). |
| `klag.consumer.lag.velocity` | Rate of change; positive means falling behind. See [Lag Velocity](https://klag.dev/metrics/lag-velocity/). |
| `klag.consumer.committed_offset` | Last committed offset per partition. |
| `klag.partition.log_end_offset` | Latest offset per partition. |
| `klag.partition.log_start_offset` | Earliest available offset per partition. |
| `klag.topic.partitions` | Partition count per topic. |
| `klag.consumer.group.state` | Consumer-group state and observed state-change count. |

### Consumer-group state

`klag.consumer.group.state` has `consumer_group` and a lowercase `state` tag. The
possible values are `stable`, `preparing_rebalance`, `completing_rebalance`, `assigning`,
`reconciling`, `empty`, `dead`, and `unknown`. There is no generic `rebalancing` value.

Groups on the KIP-848 consumer protocol (Kafka 4.0+) report `assigning` and `reconciling`
in place of `preparing_rebalance` and `completing_rebalance`. Alerts that name only the
classic pair go silent on upgraded groups — match all four:

```promql
klag_consumer_group_state{state=~"preparing_rebalance|completing_rebalance|assigning|reconciling"}
```

The gauge value is a consecutive state-change count, **not** an encoded state or a
lifetime cumulative total. It starts at `0`, rises while the state changes on
back-to-back collections, and resets to `0` on an unchanged collection. Select state by
its tag rather than comparing the gauge to a state number.
After a transition, two-phase stale-series cleanup can leave the previous state-tagged
series visible for one or two collection intervals.

`klag.consumer.lag`, per-partition `klag.consumer.lag.ms`, and
`klag.consumer.committed_offset` also carry `member_host`, `consumer_id`, and `client_id` tags
identifying the consumer **instance** that owns each partition — handy for pinning lag to a
specific pod. Unowned partitions (Empty/Dead groups) get empty-string values. Disable with
`CONSUMER_MEMBER_LABELS_ENABLED=false` to cut cardinality. Topic-level `lag.ms` aggregates and
partition-level `klag.partition.log_*_offset` metrics stay member-agnostic.

## Hot partitions

Reported **only when statistical outliers exist** (see [Hot Partitions](https://klag.dev/metrics/hot-partitions/)):

| Metric | Description |
|---|---|
| `klag.hot_partition` | Partition throughput × 100 when statistically high. Tags: `topic`, `partition` only. |
| `klag.hot_partition.lag` | Partition lag when statistically high. Tags: `consumer_group`, `topic`, `partition`. |

## Under-replicated partitions (ISR)

Reported **only when a partition is under-replicated** (see [ISR Monitoring](https://klag.dev/metrics/isr/)):

| Metric | Description |
|---|---|
| `klag.partition.under_replicated` | Missing in-sync replica count (`replicaCount - inSyncReplicaCount`). Tags: `topic`, `partition` only. |

## Time-based lag

See [Time-Based Lag](https://klag.dev/metrics/time-based-lag/). Topic aggregate tags: `consumer_group`, `topic`.
Partition tags: `consumer_group`, `topic`, `partition`, plus optional member labels.

| Metric | Description |
|---|---|
| `klag.consumer.lag.ms` | Lag in milliseconds, from Kafka log timestamps (poll-history fallback). |
| `klag.consumer.lag.time_to_close_seconds` | Estimated seconds until lag reaches zero (only when catching up). |

## Commit staleness

| Metric | Description |
|---|---|
| `klag.consumer.commit.staleness_seconds` | Seconds since Klag last observed the committed-offset sum change for a lagging group and topic. Tags: `consumer_group`, `topic`. |

This metric is reported only while lag is greater than zero, so an idle, caught-up
consumer is not marked stale. Kafka does not expose a commit timestamp, so Klag observes
the sum of committed offsets across the group/topic's partitions. Any change to that
sum, including a rewind, resets the clock. When the group/topic catches up, Klag removes
the baseline; if lag resumes, the next observation establishes a new baseline. Restarting
Klag also resets observation. This is therefore an inferred signal, not the absolute age
of the latest Kafka commit. Disable collection with `COMMIT_FRESHNESS_ENABLED=false`. See
[Detect Stuck Consumers](https://klag.dev/guides/detect-stuck-consumers/).

## Data loss prevention

See [Data Loss Prevention](https://klag.dev/metrics/data-loss-prevention/). Topic aggregates have
`consumer_group` and `topic` and omit `partition`; per-partition series additionally
have `partition`. In PromQL, `{partition=""}` matches a missing label, so use it for
topic rollups and `{partition!=""}` for partition detail.

| Metric | Description |
|---|---|
| `klag.consumer.lag.retention_percent` | Lag as a percentage of the retention window, exported as percentage × 100 for precision. Raw `10000` represents 100% and the data-loss boundary. |

## Optional JVM metrics

When `METRICS_JVM_ENABLED=true`, standard Micrometer JVM metrics (memory, GC, threads,
classes, CPU) are exported too, and visualized in the [Grafana dashboard](https://klag.dev/integrations/grafana-dashboard/).


---

# Time-Based Lag
URL: https://klag.dev/metrics/time-based-lag/

Klag estimates consumer lag in milliseconds and seconds-to-catch-up, beyond raw message counts, using Kafka log timestamps with a poll-history fallback.

Message-count lag is hard to reason about. "50,000 messages behind" means nothing
without throughput context. Klag also estimates lag in **time**.

## Metrics

| Metric | Description |
|---|---|
| `klag.consumer.lag.ms` | Lag in milliseconds: `currentTime − committedMessageTimestamp`. |
| `klag.consumer.lag.time_to_close_seconds` | Estimated seconds until lag reaches zero (only when catching up and lag > threshold). |

`klag.consumer.lag.ms` is emitted both as a topic aggregate tagged with `consumer_group` and
`topic`, and as per-partition series tagged with `consumer_group`, `topic`, and `partition`.
When `CONSUMER_MEMBER_LABELS_ENABLED=true`, the per-partition series also carry `member_host`,
`consumer_id`, and `client_id`. `time_to_close_seconds` stays per-topic.

## How `lag.ms` is computed

**Primary:** linear interpolation between Kafka `listOffsets` log start/end timestamps
and offsets, mapping the committed offset to a wall-clock timestamp.

**Fallback:** when Kafka timestamps are invalid (e.g. `logStartTimestamp=0`), Klag uses
a poll-time `(logEndOffset, systemTime)` history. The fallback requires 2+ poll
intervals and does **not** extrapolate beyond the oldest retained sample
(`TIME_LAG_INTERPOLATION_BUFFER_SIZE`).

## Time-to-close

`time_to_close_seconds` is only emitted when a consumer is **catching up** and lag
exceeds `TIME_LAG_MIN_MESSAGES` (default `100`). It answers "if things stay as they are,
how long until this consumer is caught up?".

## Configuration

See the [Configuration Reference](https://klag.dev/configuration/reference/#time-based-lag-estimation):
`TIME_LAG_ENABLED`, `TIME_LAG_MIN_MESSAGES`, `TIME_LAG_INTERPOLATION_BUFFER_SIZE`,
`TIME_LAG_STALE_PRODUCER_THRESHOLD_MS`.