Check Topic Lag

Check

Fail the experiment if consumer lag exceeds a threshold for a specific topic

Targets:

Kafka Consumers

Install now

Check Topic Lag

Fail the experiment if consumer lag exceeds a threshold for a specific topic

Check

Targets:

Kafka Consumers

Install now

Check Topic Lag

Check

Fail the experiment if consumer lag exceeds a threshold for a specific topic

Targets:

Kafka Consumers

Install now

Check Topic Lag

Fail the experiment if consumer lag exceeds a threshold for a specific topic

Check

Targets:

Kafka Consumers

Install now

Go back to list

The action "Check Topic Lag" within the experiment editor.

Introduction

Monitors the consumer lag (offset difference between the topic's latest offset and the consumer group's committed offset) for a specific topic. If the lag exceeds the configured threshold at any point during the check, the experiment fails.

This is different from Check Consumer State which monitors the consumer group's coordination state (Stable, Dead, etc.). This action specifically measures how far behind the consumer is in processing messages.

Use Cases

Fail the experiment if consumer lag grows beyond an acceptable threshold during a broker fault injection.
Verify that consumers keep up with message production while injecting network latency.
Combine with Produce Periodically to generate load and verify lag stays within bounds.

Parameters

Parameter	Description	Default
Duration	How long the check runs. Consumer lag is polled continuously for this duration.	10s
Topic to track lag	Required. The name of the Kafka topic to monitor lag for. This is the topic whose offset will be compared against the consumer group's committed offset.
Lag alert threshold	Required. The maximum acceptable offset difference between the topic's latest offset and the consumer group's committed offset. If lag exceeds this value, the experiment fails. Specified as an integer (number of messages).

Useful Templates

See all

Simulate Kafka brokers downtime and see how consumers are handling topic lag

An experiment to block access to brokers for consumers while producing records in the topic. The consumers should get back to consume when the kafka brokers are available again and the accumulated lag must be deal with quickly.

Message Queue

Kafka

Recoverability

Validate consumer's behavior when new leader is elected

Verify that your application handles a change of the leader properly.

Motivation

By testing your system's resilience to Kafka leader changes in a partition, you can identify potential weaknesses and take appropriate measures to improve its performance.

Structure

To conduct this experiment, we will ensure that all Kafka topics and producers are ready and that the consumer receives and processes messages correctly. We will then elect a new leader for one partition in Kafka and expect the system to work fine.

Message Queue

Kafka

Recoverability