Troubleshooting

Monitoring Kafka consumer progress is essential in production. Start with Klag’s logs, /healthz, /readyz, and the reporter-specific endpoint or backend.

/metrics returns 404

Likely cause: The bare application defaults METRICS_REPORTER to none, or you selected the datadog or otlp push reporter. Klag registers /metrics only for Prometheus.

Fix: Set METRICS_REPORTER=prometheus and restart Klag. The Helm chart already defaults metrics.reporter to prometheus. See Prometheus and the Configuration Reference.

Metrics or the MCP snapshot are empty after startup

Likely cause: The first Kafka collection has not completed, Kafka access failed, or no group remains after filtering. The collector starts one cycle immediately; velocity and some derived metrics still need more samples.

If Kafka is unreachable at startup, Klag stays running (degraded) instead of exiting — /readyz returns 503 until at least one cluster is connected. This is intentional: the process tolerates a broker outage at boot like the health monitor, so a transient Kafka blip does not crash-loop the pod.

Fix: Check the logs for a completed collection or Kafka errors. Verify KAFKA_BOOTSTRAP_SERVERS, the required ACLs, and the group filters. Wait for a successful cycle. If the cluster is large, allow longer than METRICS_INTERVAL_MS.

/readyz returns 503

Likely cause: Every configured Kafka health check cannot complete describeCluster. Common causes are an unreachable broker, bad TLS/SASL settings, or missing cluster DESCRIBE. With several clusters, HTTP 200 means at least one is up; inspect clusters[] in the /readyz JSON to see which names are disconnected.

Fix: Test network and DNS access from the Klag container, then compare your Kafka security variables with Installation. Grant the cluster ACL documented in ACL Permissions. /healthz can still return 200 because it only checks that the process is running.

Logs show Kafka authorization errors

Likely cause: The principal lacks DESCRIBE on the cluster, a consumer group, or a topic. Group filtering does not remove the need for cluster DESCRIBE.

Fix: Grant the read-only permissions in ACL Permissions. If you grant prefixed group or topic access, align those prefixes with METRICS_GROUP_FILTER.

No consumer groups appear

Likely cause: METRICS_GROUP_FILTER matches none of the group IDs, or METRICS_GROUP_EXCLUDE removes every included group.

Fix: Temporarily use:

METRICS_GROUP_FILTER=*
METRICS_GROUP_EXCLUDE=

Then add patterns back one at a time. Excludes run after includes. See Group Filtering.

Old series never disappear from /metrics

Likely cause: The series belong to a group that fails every cycle, for example because of a missing group DESCRIBE ACL or a wedged coordinator. Klag holds that group’s last good series instead of deleting them, along with the topic series of topics no other group reports, for as long as the failure lasts. Series of every other group and topic are still cleaned up each cycle. The log carries Failed to collect lag for group <id> (skipped this cycle) and Collection cycle was partial every interval.

This is deliberate: stale values beat deleting the series of a group that may only be failing for a while.

Fix: Grant the group the required ACLs, or drop it with METRICS_GROUP_EXCLUDE. Once the group is collected again or excluded, its series update or retire as usual. The MCP snapshot is unaffected: it keeps publishing the groups that did succeed.

Lag keeps growing for a topic the group no longer consumes

Committed offsets outlive an unsubscribe until offsets.retention.minutes (7 days by default). By default Klag still reports lag for those topics, so dashboards can show a group “falling behind” on a topic it has left.

Fix: Set METRICS_ASSIGNED_TOPICS_ONLY=true. For Stable groups, Klag keeps only topics with at least one partition assigned to a member (from the same describeConsumerGroups call already used for group state). Empty and rebalancing groups are unchanged so a total consumer outage is not hidden. You can also delete the stale offsets with kafka-consumer-groups.sh --bootstrap-server <host:port> --delete-offsets --group <group-id> --topic <abandoned-topic>.

A deleted topic’s series stay, or a live topic’s series disappear

Klag filters each cycle’s topic set against the cluster’s topic list before requesting metadata, because the Kafka admin call fails as a whole if any topic in the batch is unknown — and a group’s committed offsets outlive a deleted topic until offsets.retention.minutes (7 days by default). Without the filter, one deleted topic would fail collection for every group in the same batch for that long, holding their stale series.

The cost is an ACL asymmetry: listTopics only returns topics the principal can see. If Klag can read a group’s committed offsets for a topic it cannot describe, that topic looks deleted and its series are retired within 1–2 cycles. The log records Skipping N topic(s) absent from the cluster topic list.

Fix: Grant topic DESCRIBE matching the group access you granted — see ACL Permissions. Asymmetric grants are the only case where a live topic goes missing this way.

Collection cycles overrun the interval on large clusters

Likely cause: Each cycle issues one batched listOffsets per request type covering every partition in scope. The Kafka AdminClient splits that per leader broker, and the timestamp lookups it performs scale with partitions per broker, so a cluster with few brokers and many partitions can approach KAFKA_REQUEST_TIMEOUT_MS (default 30000). The log shows Skipping collection tick: previous cycle still running.

Fix: Set KAFKA_CHUNK_COUNT above 1. Chunking splits the work into partition-weighted batches processed sequentially, with KAFKA_CHUNK_DELAY_MS between them. Raising METRICS_INTERVAL_MS or KAFKA_REQUEST_TIMEOUT_MS also helps. See the Configuration Reference.

Velocity or time-lag metrics are missing at first

Likely cause: Lag velocity needs three collection samples. The time-lag fallback needs at least two poll intervals when Kafka log timestamps cannot support primary interpolation. Time-to-close also requires shrinking lag and at least TIME_LAG_MIN_MESSAGES messages.

Fix: Wait for the required samples. Shorten METRICS_INTERVAL_MS only if Kafka can handle the added polling load. See Lag Velocity and Time-Based Lag.

MCP 401, 403, 405, 415, or empty snapshot

Likely cause:

• 401: MCP_AUTH_TOKEN is set and the Bearer token is missing or wrong.
• 403: The request has an Origin header that is not in MCP_ALLOWED_ORIGINS. Browsers send Origin on every POST; agents, SDKs, and curl do not.
• 405: The client sent GET; Klag accepts JSON-RPC 2.0 over POST.
• 415: The request did not send Content-Type: application/json.
• Snapshot not ready: metrics collection is disabled or the first cycle has not succeeded, so no snapshot exists yet.
• Snapshot empty (groupCount: 0): reports ran, but the group filters left no groups to collect. Each cycle publishes a refreshed empty snapshot rather than returning stale data from an earlier run.

Fix: Send Authorization: Bearer <token>, use a Streamable HTTP MCP client that posts JSON-RPC requests with Content-Type: application/json, and select a reporter with METRICS_REPORTER. For a browser-based client, add its origin to MCP_ALLOWED_ORIGINS. Then resolve any Kafka or filtering problem reported in the logs. See MCP Endpoint.

Prometheus does not discover the ServiceMonitor

Likely cause: serviceMonitor.enabled is false, the reporter is not Prometheus, or the Prometheus Operator’s ServiceMonitor selector does not match the resource labels. The chart rejects serviceMonitor.enabled=true with a non-Prometheus reporter.

Fix: Install or upgrade with:

helm upgrade --install klag klag/klag \
  --set kafka.bootstrapServers="kafka:9092" \
  --set metrics.reporter="prometheus" \
  --set serviceMonitor.enabled=true

If your Prometheus resource selects a label such as release: monitoring, add --set serviceMonitor.labels.release=monitoring. Confirm that the Prometheus namespace and object selectors permit the ServiceMonitor and that its endpoint selects the Klag Service. See Kubernetes deployment.