Steadybit logoResilience Hub

Splunk Detector Fires when a Kubernetes Pod Is in Crash Loop

Use Template

Splunk Detector Fires when a Kubernetes Pod Is in Crash Loop

Use Template

Splunk Detector Fires when a Kubernetes Pod Is in Crash Loop

Use Template

Splunk Detector Fires when a Kubernetes Pod Is in Crash Loop

Use Template
Go back to list
1
GIVEN: Detector have no incidents
10s
2
Wait
10s
WHEN: Kubernetes pod of [[k8sWorkloadOwner]] crash loops
10s
3
Wait
10s
THEN: Detector has an incident
10s
4
Show Kubernetes events from the cluster
10s
Show Kubernetes events from the cluster
10s
5
Show pod count metrics for the cluster
10s
Show pod count metrics for the cluster
10s

Verify that a Splunk detector alerts you when pods are not ready to accept traffic for a certain time.

Motivation

Kubernetes features a readiness probe to determine whether your pod is ready to accept traffic. If it isn't becoming ready, Kubernetes tries to solve it by restarting the underlying container and hoping to achieve its readiness eventually. If this isn't working, Kubernetes will eventually back off to restart the container, and the Kubernetes resource remains non-functional.

Structure

First, check that the Splunk detector responsible for tracking non-ready containers is in an 'okay' state. As soon as one of the containers is crash looping, caused by the crash loop attack, the Splunk alert rule should fire and escalate it to your on-call team.

Solution Sketch


How to use this template?

Import via Hub Connection

Steadybit’s Reliability Hub is already connected to your platform. If you are an admin, you can just easily import templates with just one click.

Import as Experiment

Simply download the template and upload it as an experiment to use it once. Perfect if you are no administrator in the platform and just want to use the template once.

.json (4KB)

>_ boost your chaos journey Connect Your Hub to Steadybit

Maximise Steadybit potential! Connect your own Hub to the platform and smoothly import your own templates: using them it’s never been this easy!