Failing upstream service (e.g., message broker, database, or cache) shouldn't cause liveness or readiness probe failures in Kubernetes to avoid cascading restarts.

Motivation

In Kubernetes, liveness and readiness probes indicate whether a container is alive and able to serve incoming requests. These are especially helpful for load balancers. However, it is best practice in Kubernetes not to include upstream services in the probes. Otherwise, as soon as, e.g., a Kubernetes deployment's upstream service has issues, the deployment restarts a well, which may cause a cascade of failures in the Kubernetes cluster.

Structure

While blocking traffic from a deployment's container to a upstream service, we explicitly check the HTTP liveness and readiness probes of the Kubernetes deployment. Following best practices, we expect them not to be affected by the unavailable upstream service.

References

• Kubernetes liveness, readiness, and startup probes
• Readiness and Liveness Probes best practices by kube-score