An experiment to block access to brokers for consumers while producing records in the topic. The consumers should get back to consume when the kafka brokers are available again and the accumulated lag must be deal with quickly.

Verify that your application handles a change of the leader properly.

Motivation

By testing your system's resilience to Kafka leader changes in a partition, you can identify potential weaknesses and take appropriate measures to improve its performance.

Structure

To conduct this experiment, we will ensure that all Kafka topics and producers are ready and that the consumer receives and processes messages correctly. We will then elect a new leader for one partition in Kafka and expect the system to work fine.

Verify a Kafka's broker re-election when the current controller is isolated

Motivation

Testing your Zookeeper re-election process of a new controller broker helps to ensure high availability in your Kafka cluster. In addition, verify the behavior of your producers and consumers and verify what happens, e.g., to your message throughput while the re-election is happening or whether messages are lost.

Structure

We enforce the re-election of Kafka controller broker by blocking the network traffic to the current controller. By leveraging the 'check brokers'-step, we can validate that the new controller is elected and the old controller is detected as being down.

Intent

Intentionally deny access to the topic for consumers and during this time where consumption is stopped, delete records.

We can check the logs of the consumers to see how they handle the loss of records and also authorization access issues.