Create an action suppression "one time" during the desired duration

Collects information about the health-rule state and optionally verifies that the rule has an expected state

Return a static response for a given load balancer listener

Lower the throttle rate and burst limits on an API Gateway stage for the duration of the experiment. Supports REST APIs (v1) and HTTP APIs (v2).

Suspend one or more Auto Scaling processes (Launch, HealthCheck, ReplaceUnhealthy, ...) on an ASG for the duration of the experiment.

Simulates an outage of an entire availability zone

Temporarily lower a PROVISIONED-mode DynamoDB table's read + write capacity (table and all GSIs) to force ProvisionedThroughputExceededException.

Reboot, terminate, stop or hibernate EC2 instances

Fill ephemeral storage for the given duration

Drop inbound or outbound traffic for the specified protocol and port

Inject latency into egress network traffic

Cause packet loss for outgoing network traffic (egress)

Stop a particular process by name in a task

Stresses IO on the ephemeral storage for the given duration

Terminate a percentage of EC2 instances backing an EKS managed node group via the underlying Auto Scaling group.

Triggers Elasticache node group failover by promoting a replica node to primary

Disable an EventBridge rule for the duration of the experiment; re-enable on stop.

Reboot an Amazon MQ broker via the RebootBroker API.

Triggers DB cluster failover by promoting a standby instance to primary

Temporarily change an SQS queue's visibility timeout to force premature redelivery or to stall in-flight messages.

Delete a percentage of nodes from an AKS managed node pool via the AKS API.

Promote the secondary region to write for a multi-region Cosmos DB account, simulating a regional failover.

Disassociate the NAT Gateway from every attached subnet for the duration of the experiment; restore on stop.

Reboot, delete, stop or deallocate azure virtual machines

Set the queue's status to 'Disabled' for the duration of the experiment; restore the original status on stop.

Set the topic's status to 'Disabled' for the duration of the experiment; restore the original status on stop.

Check the state of a Cloud Foundry application over time.

Stop a Cloud Foundry application. The app will be restarted when the action ends.

Reduce the available egress network bandwidth

Blocks network traffic (incoming and outgoing)

Inject DNS errors (NXDOMAIN/SERVFAIL/TIMEOUT) into DNS queries

Inject corrupt packets by introducing single bit error at a random offset into egress network traffic

Injects TCP resets for matching connections (incoming and outgoing)

Pauses the Docker container for the given duration

Generate CPU load for one or more cores. The resources on a host are shared between containers. Running a resource attack for a container is executed in the containers namespace, but affects the host's resources and all containers running on it.

Generate read/write operation on hard disks

Stresses memory by ongoing reallocation, read and write operations

Collects information about the monitor status and optionally verifies that the monitor has an expected status

Creates a maintenance window in Dynatrace

Checks if there are any problems in Dynatrace

Removes the Cloud NAT so VMs in its subnetworks lose internet egress. Restored on stop.

Fails over a REGIONAL Cloud SQL HA instance to its standby. Not reversible.

Recreates a percentage of instances in a GKE node pool via the underlying MIG's RecreateInstances API. Pool size is preserved. Not reversible.

Fails over a STANDARD_HA Memorystore for Redis instance's primary to its replica. Not reversible.

Recreates a percentage of RUNNING instances in a Managed Instance Group. The MIG's targetSize is preserved. Not reversible.

Reset, delete, stop or suspend gcp virtual machines

Collects information about the alert-rule state and optionally verifies that the monitor has an expected state

Changes the CPU frequency limits for all cores

Drops network traffic (incoming and outgoing)

Inject corrupt packets by introducing single bit error at a random offset into ergress network traffic

Triggers a reboot or shutdown of the host

Validates that your Kubernetes resources have a CPU limit configured.

Validates that your Kubernetes resources request reasonable CPU.

Validates that you deploy Kubernetes resources via rolling updates.

Validates that your Kubernetes resources have a ephemeral storage limit configured.

Validates that your Kubernetes resources request reasonable ephemeral storage.

Ensure multiple pods aren't deployed on the same cluster's node to increase availability in case one node becomes unavailable.

Validates usage of explicit image versioning and avoids latest-tags.

Ensure that your containers are always running the identical latest container image.

Validates that your Kubernetes resources have a memory limit configured.

Validates that your Kubernetes resources request reasonable memory.

Validates configuration of liveness and readiness probes.

Validate pod redundancy to have at least 2 pods per deployment

Validate whether, right now, multiple pods of the same workload resource are deployed in the same Availability Zone.

A Steadybit check implementation for data exposed through AppDynamics.

A Steadybit discovery and action implementation to inject faults into various AWS services.

A Steadybit discovery and action implementation to inject faults into Azure services.

A Steadybit extension to discover and conduct chaos engineering attacks on Cloud Foundry applications.

A Steadybit extension to discover container and attack them.

Bi-directional integration of Steadybit and Datadog via the Datadog API.

Bi-directional integration of Steadybit and Dynatrace via the Dynatrace API.

A Steadybit action implementation to integrate Gatling Load Tests into Steadybit experiments.

A Steadybit discovery and action implementation to inject faults into GCP / Google Cloud services.

A Steadybit check implementation for data exposed through Grafana.

A Steadybit extension to discover linux hosts and attack them.

A Steadybit extension to discover Windows hosts and attack them.

A Steadybit extension to execute HTTP checks.

Integration of Steadybit and Instana via the Instana API.

A Steadybit extension adding support for Istio's service mesh fault injection capabilities.

A Steadybit action implementation to integrate Jenkins Build Jobs into Steadybit experiments.

A Steadybit action implementation to integrate JMeter Load Tests into Steadybit experiments.

A Steadybit extension to discover jvm / java applications and attack them.

A Steadybit action implementation to integrate K6 Load Tests into Steadybit experiments.

A Steadybit extension with various actions and check about Kafka.

A Steadybit attack implementation to inject HTTP faults into Kong API gateway.

A Steadybit extension to check the state of the Kubernetes cluster and inject faults.

A Steadybit extension to execute Postman collections via Postman Cloud API.

A Steadybit check implementation to gather and verify the result PromQL queries.

A Steadybit extension with various actions and check about RabbitMQ.

A Steadybit extension for Redis chaos engineering with attacks and checks for instances and databases.

A Steadybit check implementation for data exposed through Splunk Observability Cloud.

A Steadybit integration for the Splunk Cloud Platform and Splunk Enterprise.

A Steadybit check implementation for data exposed through StackState.

Verify that an AppDynamics health violation alerts you when pods are not ready to accept traffic for a certain time.

## Motivation

Kubernetes features a readiness probe to determine whether your pod is ready to accept traffic.
If it isn't becoming ready, Kubernetes tries to solve it by restarting the underlying container and hoping to achieve its readiness eventually.
If this isn't working, Kubernetes will eventually back off to restart the container, and the Kubernetes resource remains non-functional.


## Structure

First, check that the AppDynamics health violation responsible for tracking non-ready containers is in a non-violating state.
As soon as one of the containers is crash looping, caused by the [crash loop](https://hub.steadybit.com/action/com.steadybit.extension_kubernetes.crash_loop_pod) attack, the AppDynamics health violation should notify and escalate it to your on-call team.

## Solution Sketch

- [Kubernetes liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)

AppDynamics alerts when a Kubernetes pod is in crash loop

EC2 is part of the AWS Elastic Compute Cloud, which acquires and releases resources depending on the traffic demand.
Check whether your application is elastic as well by rebooting an EC2 instance.

## Motivation

Depending on your traffic demand, you can use AWS cloud's ability to acquire and release resources automatically.
Some services, such as S3 and SQS, do that automatically, while others, such as EC2, integrate with AWS Auto Scaling.
Once configured, it boils down to fluctuating EC2 instances starting or shutting down frequently.
Even when not using AWS Autoscaling, your EC2 instances may need to be restarted occasionally for maintenance and updating purposes.
Thus, it is best practice to validate your application's behavior.

## Structure

We ensure that a load-balanced user-facing endpoint fully works while having all EC2 instances available.
While restarting an EC2 instance, the HTTP endpoint continues operating but may suffer from degraded performance (e.g., lower success rate or higher response time).
The performance should recover to a 100% success rate once all EC2 instances are back.

## Solution Sketch

- [AWS Well-Architected Framework](https://wa.aws.amazon.com/wat.concept.elasticity.en.html)
- [Kubernetes liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)

Load balancer covers an AWS EC2 restart

Validate your observability to detect a CPU spike in your AWS ECS cluster

## Motivation
When you have New Relic configured to detect CPU spikes in your AWS ECS cluster, you can easily validate your observability strategy with this experiment template.

## Structure
First, we validate whether New Relic has no ongoing incident. After that, we inject the CPU spike for an ECS service and expected that New Relic detect this as an incident within the given time frame of 3 minutes.

New Relic detects an incident for CPU spikes in an ECS task

Verify that your ECS service is scaled up on increased CPU usage.

## Motivation
Important ECS services should be scaled up within a reasonable time for an elastic and resilient cloud infrastructure. Undetected high CPU spikes and long startup times are undesirable in these infrastructures.

## Structure
First, we ensure that all ECS service's tasks are ready to serve traffic. Afterward, we inject high CPU usage into the ECS task and expect that within a reasonable amount of time, ECS increases the number of ECS tasks and they become ready to handle incoming traffic.

AWS ECS Service Is Scaled up Within Reasonable Time

Ensure that you can scale up your ECS service in a reasonable time.

## Motivation
For an elastic and resilient cloud infrastructure, ensure you can scale up your ECS services within a reasonable time. Long startup times are undesirable but sometimes unnoticed and unexpected.

## Structure
Validate that all ECS tasks of an ECS service are running. Once we scale the ECS service up, the newly scheduled task should be ready within a reasonable time.

Scaling up of ECS Service Within Given Time

AWS achieves high availability via redundancy across different Availability Zones.
Ensure that failover works seamlessly by simulating Zone outages.

## Motivation

AWS hosts your deployments and services across multiple locations worldwide.
From a reliability standpoint, AWS regions and Availability Zones are most interesting.
While the former refers to separate geographic areas spread worldwide, the latter refers to an isolated location within a region.
For most use cases, applying deployments across AWS availability zone is sufficient.
Given that failures may happen at this level quite frequently, you should verify that your applications are still working in case of an outage.

## Structure

We leverage the AWS blackhole attack to simulate an AWS availability zone outage.
Before the simulated outage, we ensure that a load-balanced user-facing endpoint works appropriately.
During an AWS availability zone's unavailability, the HTTP endpoint must continue operating but may suffer from degraded performance (e.g., lower success rate or higher response time).
The performance should recover as soon as the zone is back again.

## Solution Sketch

- [Regions and Zones](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html)
- [Kubernetes liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)

Load balancer covers an AWS zone outage

Verify that your application handles an increased latency in a Redis cache properly, allowing for increased processing time while maintaining throughput.

## Motivation

Latency issues in Redis can lead to degraded system performance, longer response times, and potentially lost or delayed data.
By testing your system's resilience to Redis latency, you can ensure that it can handle increased processing time and maintain its throughput during increased latency.
Additionally, you can identify any potential bottlenecks or inefficiencies in your system and take appropriate measures to optimize its performance and reliability.

## Structure

We will verify that a load-balanced user-facing endpoint fully works while having all pods ready.
As soon as we simulate Redis latency, we expect the system to maintain its throughput and indicate unavailability appropriately.
We can introduce delays in Redis operations to simulate latency. The experiment aims to ensure that your system can handle increased processing time and maintain its throughput during increased latency.
The performance should return to normal after the latency has ended.

Kubernetes deployment survives Redis latency

Check that your application gracefully handles a Redis cache downtime and continues to deliver its intended functionality. The cache downtime may be caused by an unavailable Redis instance or a complete cluster.

## Motivation

Redis downtime can lead to degraded system performance, lost data, and potentially long system recovery times.
By testing your system's resilience to Redis downtime, you can ensure that it can handle the outage gracefully and continue to deliver its intended functionality.
Additionally, you can identify any potential weaknesses in your system and take appropriate measures to improve its performance and resilience.

## Structure

We will verify that a load-balanced user-facing endpoint fully works while having all pods ready.
As soon as we simulate Redis downtime, we expect the system to indicate unavailability appropriately and maintain its throughput.
We can block the traffic to the Redis instance to simulate downtime. The experiment aims to ensure that your system can gracefully handle the outage and continue delivering its intended functionality.
The performance should return to normal after the Redis instance is available again.

Kubernetes deployment survives Redis downtime

Turn time forward and check whether your TLS/SSL certificates are valid.

## Motivation

Noticing the TLS/SSL certification expiry too late is one problem you can easily avoid by frequently checking your expiry dates. While observability tools already handle this job nicely, you can't know whether they are working in your environment.
With this experiment, you can turn the time forward to check whether your HTTPS endpoint works at a given date in the future. Additionally, you can configure one of the [observability integrations](https://hub.steadybit.com/extensions?tags=Observability) to validate your observability tool's alerting.

## Structure
First, we validate that the given HTTPS endpoint is working today. Next, we will travel with the Linux host in time to validate that the HTTPS endpoint continues to work on a given date. If the TLS/SSL certificate has already expired at that date, the HTTP check will throw failures.

## Warning
Please be aware that we will manipulate the time for a given Linux host. Applications running at that Linux host may struggle to deal with the change in the clock correctly, and you may experience other side effects.

Certificate TLS/SSL expiry for Linux Hosts

Turn time forward and check whether your TLS/SSL certificates are valid.

## Motivation

Noticing the TLS/SSL certification expiry too late is one problem you can easily avoid by frequently checking your expiry dates. While observability tools already handle this job nicely, you can't know whether they are working in your environment.
With this experiment, you can turn the time forward to check whether your HTTPS endpoint works at a given date in the future. Additionally, you can configure one of the [observability integrations](https://hub.steadybit.com/extensions?tags=Observability) to validate your observability tool's alerting.

## Structure
First, we validate that the given HTTPS endpoint is working today. Next, we will travel with the Windows host in time to validate that the HTTPS endpoint continues to work on a given date. If the TLS/SSL certificate has already expired at that date, the HTTP check will throw failures.

## Warning
Please be aware that we will manipulate the time for a given Windows host. Applications running at that Windows host may struggle to deal with the change in the clock correctly, and you may experience other side effects.

Certificate TLS/SSL expiry for Windows Hosts

Turn time forward and check whether your TLS/SSL certificates are valid.

## Motivation

Noticing the TLS/SSL certification expiry too late is one problem you can easily avoid by frequently checking your expiry dates. While observability tools already handle this job nicely, you can't know whether they are working in your environment.
With this experiment, you can turn the time forward to check whether your HTTPS endpoint works at a given date in the future. Additionally, you can configure one of the [observability integrations](https://hub.steadybit.com/extensions?tags=Observability) to validate your observability tool's alerting.

## Structure
First, we validate that the given HTTPS endpoint is working today. Next, we will travel with the host in time to validate that the HTTPS endpoint continues to work on a given date. If the TLS/SSL certificate has already expired at that date, the HTTP check will throw failures.

## Warning
Please be aware that we will manipulate the time for a given Kubernetes node. Containers running at that host may struggle to deal with the change in the clock correctly, and you may experience other side effects.

Certificate TLS/SSL expiry for Kubernetes deployment

Achieve high availability of your Kubernetes cluster via redundancy across different Availability Zones. Check what happens to your Kubernetes cluster when one of the zones is down.

## Motivation

Cloud providers host your deployments and services across multiple locations worldwide.
From a reliability standpoint, regions and availability zones are most interesting.
While the former refers to separate geographic areas spread worldwide, the latter refers to an isolated location within a region.
For most use cases, applying deployments across availability zones is sufficient.
Given that failures may happen at this level quite frequently, you should verify that your applications are still working in case of an outage.

## Structure

We leverage the block traffic attack to simulate a full network loss in an availability zone. While the zone outage happens, we observe changes in the Kubernetes cluster with Steadybit's built-in visibility. Once the zone outage is over, we expect that all deployments will recover again within a specified time.

## Solution Sketch

- [AWS Regions and Zones](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html)
- [Azure Regions and Zones](https://learn.microsoft.com/en-us/azure/reliability/availability-zones-overview?tabs=azure-cli)
- [GCP Regions and Zones](https://cloud.google.com/compute/docs/regions-zones)
- [Kubernetes liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)

Network outage for Kubernetes nodes in an availability zone

Achieve high availability of your Kubernetes cluster via redundancy across different Availability Zones. Check what happens to your Kubernetes cluster when one of the zones suffers from a network loss.

## Motivation

Cloud provider host your deployments and services across multiple locations worldwide.
From a reliability standpoint, regions and availability zones are most interesting.
While the former refers to separate geographic areas spread worldwide, the latter refers to an isolated location within a region.
For most use cases, applying deployments across availability zone is sufficient.
Given that failures may happen at this level quite frequently, you should verify that your applications are still working in case of an outage.

## Structure

We leverage the drop outgoing traffic to simulate network loss in an availability. If you want to test for a full outage of the zone, configure it to 100% loss.
While the network loss happens, we observe changes of a Kubernetes cluster with Steadybit's built-in visibility. Once the network loss is over, we expect that all deployments will recover again within a specified time.

## Solution Sketch

- [AWS Regions and Zones](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html)
- [Azure Regions and Zones](https://learn.microsoft.com/en-us/azure/reliability/availability-zones-overview?tabs=azure-cli)
- [GCP Regions and Zones](https://cloud.google.com/compute/docs/regions-zones)
- [Kubernetes liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)

Network loss for Kubernetes node's outgoing traffic in an availability zone

Verify that a Datadog monitor alerts you when pods are not ready to accept traffic for a certain time.

## Motivation

Kubernetes features a readiness probe to determine whether your pod is ready to accept traffic.
If it isn't becoming ready, Kubernetes tries to solve it by restarting the underlying container and hoping to achieve its readiness eventually.
If this isn't working, Kubernetes will eventually back off to restart the container, and the Kubernetes resource remains non-functional.


## Structure

First, check that the Datadog monitor responsible for tracking non-ready containers is in an 'okay' state.
As soon as one of the containers is crash looping, caused by the [crash loop](https://hub.steadybit.com/action/com.steadybit.extension_kubernetes.crash_loop_pod) attack, the Datadog monitor should alert and escalate it to your on-call team.

## Solution Sketch

- [Kubernetes liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)

Datadog alerts when a Kubernetes pod is in crash loop

When a host suddenly loses connection to the network and your system, Datadog should alert about this. Eventually, everything should recover once the network is back again.

## Motivation

When you're working in a less volatile system environment, a loss of network can be crucial as there is likely no backup host that will enable faster recovery. Thus, you should check your observability tools to catch this.

## Structure
Before blocking a host from the network, we verify that the Datadog monitor is in an ok state
Afterward, we block all traffic to and from a host and expect Datadog to alert about the isolated host. Eventually, when the host is online again, we expect Datadog to turn into an OK state again.
While experimenting, we create a downtime for the Monitor so that it will not escalate due to the ongoing alert.

Linux host losing network connection is detected by Datadog

When a Linux host is suddenly missing from your system, Datadog should alert you to this. Eventually, everything should recover when only rebooting the host.

## Motivation

When you're working in a less volatile system environment, where you expect hosts always to run, you should validate whether you notice whenever a host is rebooting.

## Structure
Before restarting a host, we verify that the Datadog monitor is in an ok state
Afterward, we trigger the shutdown of a host and expect Datadog to alert about the missing host. Eventually, the host should come back and Datadog turn into an OK state again.
While experimenting, we create a downtime for the Monitor so that it will not escalate due to the ongoing alert.

Linux Host reboot is alerted by Datadog

Windows host losing network connection is detected by Datadog

When a windows host is suddenly missing from your system, Datadog should alert you to this. Eventually, everything should recover when only rebooting the host.

## Motivation

When you're working in a less volatile system environment, where you expect hosts always to run, you should validate whether you notice whenever a host is rebooting.

## Structure
Before restarting a host, we verify that the Datadog monitor is in an ok state
Afterward, we trigger the shutdown of a host and expect Datadog to alert about the missing host. Eventually, the host should come back and Datadog turn into an OK state again.
While experimenting, we create a downtime for the Monitor so that it will not escalate due to the ongoing alert.

Windows host reboot is alerted by Datadog

An unavailable Microsoft SQL Server database might be too severe for suitable fallbacks and requires your system to indicate unavailability appropriately

## Motivation

Depending on your context, an unavailable Microsoft SQL Server database may be considered so severe that there are no suitable fallbacks.
In this case, ensuring that your system indicates an appropriate error message is essential.
After the Microsoft SQL Server database returns, your system should recover automatically.

## Structure

We ensure that a load-balanced user-facing endpoint fully works while having all pods ready.
When we simulate an unavailable Microsoft SQL Server database, we expect the user-facing endpoint to indicate unavailability by responding with a "Service unavailable" status.
To simulate the unavailability, we can block the Microsoft SQL Server database client connection on its hostname so that no incoming or outgoing traffic goes through.
The endpoint should recover automatically once the Microsoft SQL Server database is reachable again.

Graceful degradation when Microsoft SQL Server database can not be reached

An unavailable Oracle database might be too severe for suitable fallbacks and requires your system to indicate unavailability appropriately

## Motivation

Depending on your context, an unavailable Oracle database may be considered so severe that there are no suitable fallbacks.
In this case, ensuring that your system indicates an appropriate error message is essential.
After the Oracle database returns, your system should recover automatically.

## Structure

We ensure that a load-balanced user-facing endpoint fully works while having all pods ready.
When we simulate an unavailable Oracle database, we expect the user-facing endpoint to indicate unavailability by responding with a "Service unavailable" status.
To simulate the unavailability, we can block the Oracle database client connection on its hostname so that no incoming or outgoing traffic goes through.
The endpoint should recover automatically once the Oracle database is reachable again.

Graceful degradation when Oracle database can not be reached

Your application should continue functioning properly and indicate unavailability appropriately in case of increased connection latency to PostgreSQL.
Additionally, this experiment can highlight requests that need optimization of timeouts to prevent dropped requests.

## Motivation

Latencies in shared or overloaded databases are common and can significantly impact the performance of your application.
By conducting this experiment, you can gain insights into the robustness of your application and identify areas for improvement.

## Structure

To conduct this experiment, we will ensure that all pods are ready and that the load-balanced user-facing endpoint is fully functional.
We will then simulate a latency attack on the PostgreSQL database by adding a delay of 100 milliseconds to all traffic to the database hostname.
During the attack, we will monitor the system's behavior to ensure the service remains operational and can deliver its purpose.
We will also analyze the performance metrics to identify any request types most affected by the latency and optimize them accordingly.
Finally, we will end the attack and monitor the system's recovery time to ensure it returns to its normal state promptly.
By conducting this experiment, we can gain valuable insights into our application's resilience to database latencies and make informed decisions to optimize its performance under stress.

Graceful degradation and Datadog alerts when Postgres suffers latency

An unavailable Postgres database might be too severe for suitable fallbacks and requires your system to indicate unavailability appropriately

## Motivation

Depending on your context, an unavailable Postgres database may be considered so severe that there are no suitable fallbacks.
In this case, ensuring that your system indicates an appropriate error message is essential.
After the Postgres database returns, your system should recover automatically.

## Structure

We ensure that a load-balanced user-facing endpoint fully works while having all pods ready.
When we simulate an unavailable Postgres database, we expect the user-facing endpoint to indicate unavailability by responding with a "Service unavailable" status.
To simulate the unavailability, we can block the Postgres database client connection on its hostname so that no incoming or outgoing traffic goes through.
The endpoint should recover automatically once the Postgres database is reachable again.

Graceful degradation when Postgres database can not be reached

An unavailable database should be handled by your application gracefully and indicated appropriately
Specifically, we want to ensure that at least one monitor in Datadog is alerting us to the outage.
You can address a potential impact on your system by implementing, e.g., a failover or caching mechanism.

## Motivation

Database outages can occur for various reasons, including hardware failures, software bugs, network connectivity issues, or even intentional attacks. Such outages can severely affect your application, such as lost revenue, dissatisfied customers, and reputational damage. By testing your application's resilience to a database outage, you can identify areas for improvement and implement measures to minimize the impact of such outages on your system.

## Structure

To conduct this experiment, we will ensure that all pods are ready and that the load-balanced user-facing endpoint is fully functional. We will then simulate an unavailable PostgreSQL database by blocking the PostgreSQL database client connection on a given hostname.
During the outage, we will monitor the system and ensure that the user-facing endpoint indicates unavailability by responding with a "Service unavailable" status. We will also verify that at least one monitor in Datadog is alerting us to the database outage.
Once the database becomes available again, we will verify that the endpoint automatically recovers and resumes its normal operation. We will also analyze the monitoring data to identify any potential weaknesses in the system and take appropriate measures to address them.
By conducting this experiment, we can identify any weaknesses in our system's resilience to database outages and take appropriate measures to minimize their impact.

Graceful degradation and Datadog alerts when Postgres database can not be reached

An unavailable database might be too severe for suitable fallbacks and requires your system to indicate unavailability appropriately

## Motivation

Depending on your context, an unavailable database may be considered so severe that there are no suitable fallbacks.
In this case, ensuring that your system indicates an appropriate error message is essential.
After the database returns, your system should recover automatically.

## Structure

We ensure that a load-balanced user-facing endpoint fully works while having all pods ready.
When we simulate an unavailable database, we expect the user-facing endpoint to indicate unavailability by responding with a "Service unavailable" status.
To simulate the unavailability, we can block the database client connection on its hostname so that no incoming or outgoing traffic goes through.
The endpoint should recover automatically once the database is reachable again.

Graceful degradation when database can not be reached

Verify that Dynatrace alerts you on pods not being ready to accept traffic for a certain amount of time.

## Motivation

Kubernetes features a readiness probe to determine whether your pod is ready to accept traffic.
If it isn't becoming ready, Kubernetes tries to solve it by restarting the underlying container and hoping to achieve its readiness eventually.
If this isn't working, Kubernetes will eventually back off to restart the container, and the Kubernetes resource remains non-functional.


## Structure

First, check that Dynatrace has no problems for an entity and doesn't alert already on non-ready containers.
As soon as one of the containers is crash looping, caused by the Steadybit attack [crash loop](https://hub.steadybit.com/action/com.steadybit.extension_kubernetes.crash_loop_pod), Dynatrace should detect the problem and alert to ensure your on-call team is taking action.

## Solution Sketch

- [Kubernetes liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)

Dynatrace should detect a crash looping as problem

Verify that a Grafana alert rule alerts you when pods are not ready to accept traffic for a certain time.

## Motivation

Kubernetes features a readiness probe to determine whether your pod is ready to accept traffic.
If it isn't becoming ready, Kubernetes tries to solve it by restarting the underlying container and hoping to achieve its readiness eventually.
If this isn't working, Kubernetes will eventually back off to restart the container, and the Kubernetes resource remains non-functional.


## Structure

First, check that the Grafana alert rule responsible for tracking non-ready containers is in an 'okay' state.
As soon as one of the containers is crash looping, caused by the [crash loop](https://hub.steadybit.com/action/com.steadybit.extension_kubernetes.crash_loop_pod) attack, the Grafana alert rule should fire and escalate it to your on-call team.

## Solution Sketch

- [Kubernetes liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)

Grafana alert rule fires when a Kubernetes pod is in crash loop

Fill up the memory of a Host progressively to see at which percentage it becomes unstable. Additionally, you may want to add one of our observability checks.

### Structure
We start by filling `50%` of the host's memory for `30 seconds`. Next, we stepwise fill the memory to `75%`, `90%`, and `100%` - each for `30 seconds`. In between, we have small wait steps to ease analysis in external observability tools for each phase.

Fill Memory progressively of a Linux Host

Latency of a Host progressively increases to analyse at which point the communication breaks. Additionally, you may want to add one of our observability checks.

### Structure
We start by adding a `250ms` latency on the host's outgoing traffic for `30 seconds`. Next, we stepwise increase the latency to `500ms`, `750ms`, and `1s` - each for `30 seconds`. In between, we have small wait steps to ease analysis in external observability tools for each phase.

Latency progressively increases for a Linux Host

Stress the CPU of a host progressively to see at which percentage it becomes unstable. Additionally, you may want to add one of our observability checks.

### Structure
We start by stressing `50%` of the host's CPU for `30 seconds`. Next, we stepwise stress the CPU by `75%`, `90%`, and `100%` - each for `30 seconds`. In between, we have small wait steps to ease analysis in external observability tools for each phase.

Stress CPU progressively of a Linux Host

Fill Memory progressively of a Windows Host

Latency progressively increases for a Windows Host

Stress CPU progressively of a Windows Host

# Intent
Verify that Instana alerts you that pods are not ready to accept traffic for some time.

## Motivation

Kubernetes features a readiness probe to determine whether your pod is ready to accept traffic.
If it isn't becoming ready, Kubernetes tries to solve it by restarting the underlying container and hoping to achieve its readiness eventually.
If this isn't working, Kubernetes will eventually back off to restart the container, and the Kubernetes resource remains non-functional.


## Structure

First, check that Instana has no critical events for an application perspective.
As soon as one of the containers is crash looping, caused by the Steadybit attack [crash loop](https://hub.steadybit.com/action/com.steadybit.extension_kubernetes.crash_loop_pod), Instana should detect this via a critical event to ensure your on-call team is taking action.

## Solution Sketch

- [Kubernetes liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)

Instana should detect a crash looping as incident

Learn how easy you can block traffic of an entire Kubernetes DaemonSet.

Block Traffic of Kubernetes DaemonSet

Fill up the memory of a Kubernetes DaemonSet progressively to see at which percentage it will be killed by Kubernetes.

### Structure
We start by filling `50%` of the Kubernetes DaemonSet's memory for `30 seconds`. Next, we stepwise fill the memory to `75%`, `90%`, and `100%` - each for `30 seconds`. In between, we have small wait steps to ease analysis in external observability tools for each phase.

Fill Memory progressively of Kubernetes DaemonSet

Latency of a Kubernetes DaemonSet progressively increases to analyse at which point the communication breaks.

### Structure
We start by adding a `250ms` latency on the Kubernetes DaemonSet's outgoing traffic for `30 seconds`. Next, we stepwise increase the latency to `500ms`, `750ms`, and `1s` - each for `30 seconds`. In between, we have small wait steps to ease analysis in external observability tools for each phase.

Latency progressively increases for Kubernetes DaemonSet

Learn how easy you can stress CPU of an entire Kubernetes DaemonSet.

Stress CPU of Kubernetes DaemonSet

Stress the CPU of a Kubernetes DaemonSet progressively to see at which percentage it will be killed by Kubernetes.

### Structure
We start by stressing `50%` of the Kubernetes DaemonSet's CPU for `30 seconds`. Next, we stepwise stress the CPU by `75%`, `90%`, and `100%` - each for `30 seconds`. In between, we have small wait steps to ease analysis in external observability tools for each phase.

Stress CPU progressively of Kubernetes DaemonSet

Learn how easy you can stress memory of an entire Kubernetes DaemonSet.

Stress Memory of Kubernetes DaemonSet

Ensure that all containers of Kubernetes deployment resources have proper ephemeral storage limits configured to prevent the instability of other containers.

## Motivation
For an elastic and resilient cloud infrastructure, ensure that the over-usage of ephemeral storage of one container doesn't affect any others.
Furthermore, if one container exceeds its configured limits, Kubernetes should kill it and eventually prepare it within a given timeframe.

## Structure
Verify that a user-visible endpoint responds within the expected success rates while exceeding the ephemeral storage.
As soon as one container exceeds the ephemeral storage, by filling the disk in a `/tmp` directory, Kubernetes should evict the container, decreasing ready pods. Within 60 seconds, the evicted container should run again, and the pod should be ready.

Load-balanced endpoint covers exceeding ephemeral storage of Kubernetes deployment

Check what happens when all pods of a Kubernetes deployment aren't coming up again. 
## Motivation
Typically, Kubernetes tries to keep as many pods running as desired for a Kubernetes deployment. However, some circumstances may prevent Kubernetes from achieving this, like missing resources in the cluster, problems with the deployment's probes, or a CrashLoopBackOff.
You should validate what happens to your upstream services that are using the given deployment as a downstream service.

## Structure
To keep the pods down for a given deployment, we first kill all the pods in the deployment. Simultaneously, we will scale down the Kubernetes deployment to 0 to simulate that these pods can't be scheduled again.
At the of the experiment, we automatically roll back the deployment's scale to the initial value.

Keep Deployment's pods down

If a pod becomes temporarily unavailable, you want to ensure that Kubernetes is properly reacting, excluding that pod from the Service and restarting it.

## Motivation

If configured properly, Kubernetes can detect a non-responding pod and try to fix it by simply restarting the unresponsive pod.
Even so, the exact configuration requires careful consideration to avoid killing your pods too early or flooding your cluster's traffic with liveness probes.

## Structure

Before killing a container of a Kubernetes pod, we verify that a load-balanced user-facing endpoint is working properly and that all Kubernetes deployment's pods are marked as ready.
As soon as one container crashes, Kubernetes should detect the crashed container via a failing liveness probe and mark the related pod as not ready.
Now, Kubernetes is expected to restart the container so the pod becomes ready within a certain time.
The user-facing HTTP endpoint may suffer from degraded performance when being under load (e.g., lower success rate or higher response time).
Even so, this is expected to be within the SLA boundaries.

## Solution Sketch

- [Kubernetes liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)

Load balancer hides a single container failure for end users

Verify that your Prometheus metrics are catching unready pods in a Kubernetes deployment.

### Motivation
Whenever important deployments aren't available your Prometheus metric should catch this in order to alert properly.

### Structure
By blocking the traffic to all containers of the deployment, we expected failing probes and eventually, Kubernetes detecting the pods as being down. After a short time frame, Prometheus should detect this as well in the specified Prometheus metric.

Prometheus detect unhealthy deployments

Kubernetes features a [rolling update strategy](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment) to deploy new releases without downtime.
When being under load, this only works reliably when your load balancer and the Kubernetes readiness probe are configured properly and DNS caches are up-to-date.

## Motivation

The Kubernetes rolling update strategy ensures that a minimum number of pods remain available when a new release is deployed.
This implies that a new pod with a new release is started and needs to be ready before an old pod is evicted.
Even so, this process may result in degraded performance and user-facing errors, e.g., Kubernetes sending requests to pods indicated as ready but not able to respond properly or evicted pods are still retained in the load balancer.

## Structure

Before performing the rolling update all desirable pods of the deployment need to be in the “ready”-state, and a load-balanced user-facing HTTP endpoint is expected to respond successfully while under load.
As soon as the rolling update takes place, the HTTP endpoint under load may suffer from a degraded performance (e.g.
lower success rate or higher response time).
Even so, this should be within the boundaries of your SLA.
After the rolling update, the number of desirable pods matches the actual pods of the deployment and the performance of the user-facing HTTP endpoint is similar to before the update.

## Solution Sketch

- [Kubernetes liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)
- [Kubernetes deployment strategy](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-update-deployment)

Faultless redundancy during rolling update

Ensure that you can scale your deployment in a reasonable time without noticeable errors.

## Motivation
For an elastic and resilient cloud infrastructure, ensure that you can scale your deployments without user-visible errors and within a reasonable amount of time.
Long startup times, hiccups in the load balancer, or resource misallocation are undesirable but sometimes unnoticed and unexpected.

## Structure

For the duration of the experiment and the deployment's upscaling, verify that a user-visible endpoint offered is responding within expected success rates and that no monitors are alerting.
As soon as the deployment is scaled up, the newly scheduled pod should be ready to receive traffic within a reasonable time, e.g., 60 seconds.

Faultless scaling of Kubernetes Deployment

Learn how easy you can block traffic of an entire Kubernetes deployment.

Block Traffic of Kubernetes Deployment

Fill up the memory of a Kubernetes Deployment progressively to see at which percentage it will be killed by Kubernetes.

### Structure
We start by filling `50%` of the Kubernetes Deployment's memory for `30 seconds`. Next, we stepwise fill the memory to `75%`, `90%`, and `100%` - each for `30 seconds`. In between, we have small wait steps to ease analysis in external observability tools for each phase.

Fill Memory progressively of Kubernetes Deployment

Latency of a Kubernetes Deployment progressively increases to analyse at which point the communication breaks.

### Structure
We start by adding a `250ms` latency on the Kubernetes Deployment's outgoing traffic for `30 seconds`. Next, we stepwise increase the latency to `500ms`, `750ms`, and `1s` - each for `30 seconds`. In between, we have small wait steps to ease analysis in external observability tools for each phase.

Latency progressively increases for Kubernetes Deployment

Learn how easy you can stress CPU of an entire Kubernetes deployment.

Stress CPU of Kubernetes Deployment

Stress the CPU of a Kubernetes Deployment progressively to see at which percentage it will be killed by Kubernetes.

### Structure
We start by stressing `50%` of the Kubernetes Deployment's CPU for `30 seconds`. Next, we stepwise stress the CPU by `75%`, `90%`, and `100%` - each for `30 seconds`. In between, we have small wait steps to ease analysis in external observability tools for each phase.

Stress CPU progressively of Kubernetes Deployment

Learn how easy you can stress memory of an entire Kubernetes deployment.

Stress Memory of Kubernetes Deployment

Identify the effect of high latency of a third-party service on your deployment's service's success metrics.

## Motivation

When you provide a synchronous service via HTTP that requires the availability of other downstream third-party services, you absolutely should check how your service behaves in case the third-party service suffers high latency. Also, you want to validate whether your service is working again as soon as the third-party service is working again.

## Structure

We ensure that a load-balanced user-facing endpoint fully works while having all pods ready.
When we simulate the third-party service's high latency, we expect the user-facing endpoint to work within specified HTTP success rates..
To simulate the high latency, we can delay the traffic to the third-party service on the client side using its hostname.
The endpoint should recover automatically once the third-party service is reachable again.

Third-party service suffers high latency for a Kubernetes Deployment

Identify the effect of an unavailable third-party service on your deployment's service's success metrics.

## Motivation

When you provide a synchronous service via HTTP that requires the availability of other downstream third-party services, you absolutely should check how your service behaves in case the third-party service is unavailable. Also, you want to validate whether your service is working again as soon as the third-party service is working again.

## Structure

We ensure that a load-balanced user-facing endpoint fully works while having all pods ready.
When we simulate the third-party service being unavailable, we expect the user-facing endpoint to work within specified HTTP success rates..
To simulate the unavailability, we can block the traffic to the third-party service on the client side using its hostname.
The endpoint should recover automatically once the third-party service is reachable again.

Third-party service is unavailable for a Kubernetes Deployment

Quick startup times are favorable in Cloud environments to enable fast recovery and improve scaling.

## Motivation
In Cloud environments, it is accepted that a pod or container may crash - the more important principle is that it should recover quickly.
A faster startup time is beneficial in that case as it results in a smaller Mean Time To Recover (MTTR) and reduces user-facing downtime.
Also, in case of request peaks, a reasonably short startup time allows scaling the deployment properly.

## Structure

We simply stop a container of one of the pods to measure the time until it is marked as ready again. Therefore, before stopping the container, we ensure that the deployment is ready. If so, we stop the container and expect the number of ready pods to drop.
Within a reasonable time (e.g., 60 seconds), the container should start up again, and all desirable pods should be marked as ready.

## Solution Sketch

- [Kubernetes liveness, readiness, and startup probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)

Reasonable recovery time in case of container failures

When deleting a pod, Kubernetes should bring up a new pod to ensure system stability.

## Motivation
Deleting a pod simulates situations in which, for any reason, a pod stops working properly and needs to be rescheduled. This experiment makes sure that rescheduling works as expected and newly scheduled pods become ready within the expected timeframe.

## Structure

All pods should be ready to accept traffic at the beginning of the experiment. Rescheduling should start as soon as a pod is deleted.
Eventually, after the allotted time, all pods should be ready again.

Reasonable recovery time when losing a pod

Verify that an unavailable downstream service doesn't result in user-visible errors.

## Motivation

When offering a service that is dependent on downstream services, you should ensure that the offered service also works fine whenever one of the downstream services can't be reached. This is especially true when multiple downstream services are involved and the responses of each downstream service are considered optional.

## Structure

First, we validate that the HTTP endpoint of the upstream service is working as expected. Then, we block the traffic of the downstream service and expect that the HTTP endpoint will still work within the expected success rate.



## Read more
This experiment template is used in our quick start on running an experiment and is especially useful for the shopping demo example. To learn more, check out the [quick start in the Steadybit docs](https://docs.steadybit.com/quick-start/run-experiment).

Amazon MQ Brokers

API Gateways

AppDynamics Applications

AppDynamics Health Rules

Application Load Balancers

Argo Rollouts

Auto Scaling Groups

AWS Lambdas

Azure AKS clusters

Azure AKS node pools

Azure API Management services

Azure Application Gateways

Azure Cosmos DB accounts

Azure Event Grid event subscriptions

Azure Event Grid topics

Azure Functions

Azure Load Balancers

Azure Managed Disks

Azure NAT Gateways

Azure Network Security Groups

Azure Scale Set Instances

Azure Service Bus namespaces

Azure Service Bus queues

Azure Service Bus topics

Azure Storage Queues

Azure Virtual Machine Scale Sets

Azure Virtual Machines

Cloud Foundry Apps

Cloud NATs

Cloud Run services

Cloud SQL instances

Container Apps

Containers

Datadog Monitors

DynamoDB Tables

EBS Volumes

EC2 Instances

ECS Services

ECS Tasks

EKS Clusters

EKS Nodegroups

Elasticaches

Envoy Gateway HTTP Routes

EventBridge Rules

FIS Experiment Templates

Gatling Enterprise Simulations

GCP Virtual Machines

GKE clusters

GKE node pools

Grafana Alert Rules

Instana Application Perspectives

Istio Virtual services

Jenkins Jobs

JVM Instances

Kafka Brokers

Kafka Consumers

Kafka Topics

Kong Routes

Kong Services

Kubernetes Cluster

Kubernetes Daemonsets

Kubernetes Deployments

Kubernetes HAProxy Ingresses

Kubernetes NGINX Ingresses

Kubernetes Nodes

Kubernetes Pods

Kubernetes ReplicaSets

Kubernetes Statefulsets

Linux Hosts

Managed Instance Groups

Memorystore for Redis instances

MSK Brokers

NAT Gateways

Network Load Balancers

New Relic Accounts

New Relic Workloads

Persistent Disks

Postman Collections

Prometheus Instances

Pub/Sub subscriptions