Delay Outgoing Traffic
AttackDelay Outgoing Traffic
Inject latency into egress network traffic.AttackDelay Outgoing Traffic
AttackDelay Outgoing Traffic
Inject latency into egress network traffic.Attack
Go back to list
Kubernetes deployment survives Redis latency
Verify that your application handles an increased latency in a Redis cache properly, allowing for increased processing time while maintaining throughput.
Motivation
Latency issues in Redis can lead to degraded system performance, longer response times, and potentially lost or delayed data. By testing your system's resilience to Redis latency, you can ensure that it can handle increased processing time and maintain its throughput during increased latency. Additionally, you can identify any potential bottlenecks or inefficiencies in your system and take appropriate measures to optimize its performance and reliability.
Structure
We will verify that a load-balanced user-facing endpoint fully works while having all pods ready. As soon as we simulate Redis latency, we expect the system to maintain its throughput and indicate unavailability appropriately. We can introduce delays in Redis operations to simulate latency. The experiment aims to ensure that your system can handle increased processing time and maintain its throughput during increased latency. The performance should return to normal after the latency has ended.
Graceful degradation and Datadog alerts when Postgres suffers latency
Your application should continue functioning properly and indicate unavailability appropriately in case of increased connection latency to PostgreSQL. Additionally, this experiment can highlight requests that need optimization of timeouts to prevent dropped requests.
Motivation
Latencies in shared or overloaded databases are common and can significantly impact the performance of your application. By conducting this experiment, you can gain insights into the robustness of your application and identify areas for improvement.
Structure
To conduct this experiment, we will ensure that all pods are ready and that the load-balanced user-facing endpoint is fully functional. We will then simulate a latency attack on the PostgreSQL database by adding a delay of 100 milliseconds to all traffic to the database hostname. During the attack, we will monitor the system's behavior to ensure the service remains operational and can deliver its purpose. We will also analyze the performance metrics to identify any request types most affected by the latency and optimize them accordingly. Finally, we will end the attack and monitor the system's recovery time to ensure it returns to its normal state promptly. By conducting this experiment, we can gain valuable insights into our application's resilience to database latencies and make informed decisions to optimize its performance under stress.
Latency progressively increases for Kubernetes DaemonSet
Latency of a Kubernetes DaemonSet progressively increases to analyse at which point the communication breaks.
Structure
We start by adding a 250ms latency on the Kubernetes DaemonSet's outgoing traffic for 30 seconds. Next, we stepwise increase the latency to 500ms, 750ms, and 1s - each for 30 seconds. In between, we have small wait steps to ease analysis in external observability tools for each phase.
Latency progressively increases for Kubernetes Deployment
Latency of a Kubernetes Deployment progressively increases to analyse at which point the communication breaks.
Structure
We start by adding a 250ms latency on the Kubernetes Deployment's outgoing traffic for 30 seconds. Next, we stepwise increase the latency to 500ms, 750ms, and 1s - each for 30 seconds. In between, we have small wait steps to ease analysis in external observability tools for each phase.
