Intent

Ensure that you can scale your deployment in a reasonable time without noticeable errors.

Motivation

For an elastic and resilient cloud infrastructure, ensure that you can scale your deployments without user-visible errors and within a reasonable amount of time. Long startup times, hiccups in the load balancer, or misallocation of resources are undesirable but sometimes unnoticed and unexpected.

Structure

For the duration of the experiment and the up-scaling of the deployment, verify that an offered user-visible endpoint is responding within expected success rates and that no monitors are alerting. As soon as the deployment is scaled up, the newly scheduled pod should be ready to receive traffic within a reasonable time, e.g. 60 seconds.

Environment Example

In our example, we are scaling up hot deals and expect the new pod to be ready within 60 seconds. Furthermore, we are checking that our Datadog production monitors are always in an 'okay' state.