Check what happens when all pods of a Kubernetes StatefulSet aren't coming up again.

Motivation

Typically, Kubernetes tries to keep as many pods running as desired for a Kubernetes StatefulSet. However, some circumstances may prevent Kubernetes from achieving this, like missing resources in the cluster, problems with the StatefulSet's probes, or a CrashLoopBackOff. You should validate what happens to your upstream services that are using the given StatefulSet as a downstream service.

Structure

To keep the pods down for a given StatefulSet, we first kill all the pods in the StatefulSet. Simultaneously, we will scale down the Kubernetes StatefulSet to 0 to simulate that these pods can't be scheduled again. At the of the experiment, we automatically roll back the StatefulSet's scale to the initial value.