Steadybit logoResilience Hub
Try SteadybitGitHub icon
Steadybit logoResilience Hub

DataDog Alerts when a Kubernetes Pod Is in Crash Loop

When one of your containers has problems starting, it may result in a crash loop and, eventually, Kubernetes backing off to restart this container. Verify that Datadog notices a crash loop and will alert you to take action.
Targets:

Datadog monitors

Kubernetes pods

Download now

DataDog Alerts when a Kubernetes Pod Is in Crash Loop

When one of your containers has problems starting, it may result in a crash loop and, eventually, Kubernetes backing off to restart this container. Verify that Datadog notices a crash loop and will alert you to take action.
Targets:

Datadog monitors

Kubernetes pods

Download now

DataDog Alerts when a Kubernetes Pod Is in Crash Loop

When one of your containers has problems starting, it may result in a crash loop and, eventually, Kubernetes backing off to restart this container. Verify that Datadog notices a crash loop and will alert you to take action.
Targets:

Datadog monitors

Kubernetes pods

Download now

DataDog Alerts when a Kubernetes Pod Is in Crash Loop

When one of your containers has problems starting, it may result in a crash loop and, eventually, Kubernetes backing off to restart this container. Verify that Datadog notices a crash loop and will alert you to take action.
Targets:

Datadog monitors

Kubernetes pods

Download now
Go back to list
The experiment editor showing the visual structure of the experiment.The experiment editor showing the visual structure of the experiment.

Intent

Verify that Datadog monitor alerts you on pods not being ready to accept traffic for a certain amount of time.

Motivation

Kubernetes features a readiness probe to determine whether your pod is ready to accept traffic. If it isn't becoming ready, Kubernetes tries to solve it by restarting the underlying container and hoping to achieve its readiness eventually. If this isn't working, Kubernetes will eventually back off to restart the container, and the Kubernetes resource remains non-functional.

Structure

First, check that Datadog monitor responsible for tracking non-ready container is in an 'okay' state. As soon as one of the containers is crash looping, caused by the Steadybit attack crash loop, the Datadog monitor should alert and escalate it to your on-call team.

Environment Example

The Kubernetes deployment gateway consists of two pods. We are attacking one of the two pods by causing a crash loop and waiting on a specific monitor in Datadog to detect this crashing pod.

Solution Sketch


Download now

.json (4 kB)

It's quick and easy

  1. 1.

    Download .json file
  2. 2.

    Upload it inside Steadybit
  3. 3.

    Start your experiment!
Screenshot showing the Steadybit UI elements to import the experiment.json file into the Steadybit platform.
Tags
Kubernetes
Datadog
Restart
Crash loop
Harden Observability
GitHub
steadybit/reliability-hub-db/tree/main/recipes/datadog.monitor-alerts-on-pod-crash-loop
License
MIT
MaintainerSteadybit

Used Actions

See all
Cause Crash Loop

Causes a crash loop in a pod

Attack

Attack

Kubernetes pods

Start Using Steadybit Today

Get started with Steadybit, and you’ll get access to all of our features to discover the full power of Steadybit. Available for SaaS and on-prem!

Are you unsure where to begin?

No worries, our reliability experts are here to help: book a demo with them!

Steadybit logoResilience Hub
Try Steadybit
HubActionsTargetsExtensionsRecipes
© 2024 Steadybit GmbH. All rights reserved.
Twitter iconLinkedIn iconGitHub icon