Intent

An unavailable RabbitMQ cluster should be handled by your application gracefully and being indicated appropriately. Specifically, we want to ensure that at least one monitor in Datadog is alerting us to the outage.

Motivation

RabbitMQ downtime can lead to degraded system performance, lost data, and potentially long system recovery times. By testing the resilience of your system to RabbitMQ downtime, you can ensure that your system can handle the outage gracefully and continue to deliver its intended functionality. Additionally, you can identify any potential weaknesses in your system and take appropriate measures to improve its performance and resilience.

Structure

We will verify that a load-balanced user-facing endpoint fully works while having all pods ready. As soon as we simulate RabbitMQ downtime, we expect the system to indicate unavailability appropriately and maintain its throughput. To simulate downtime, we can shut down the RabbitMQ instance or cluster. The experiment aims to ensure that your system can handle the outage gracefully and continue to deliver its intended functionality. The performance should return to normal after the RabbitMQ instance or cluster is available again.