An unavailable database might be too severe for suitable fallbacks and requires your system to indicate unavailability appropriately

Motivation

Depending on your context, an unavailable database may be considered so severe that there are no suitable fallbacks. In this case, ensuring that your system indicates an appropriate error message is essential. After the database returns, your system should recover automatically.

Structure

We ensure that a load-balanced user-facing endpoint fully works while having all pods ready. When we simulate an unavailable database, we expect the user-facing endpoint to indicate unavailability by responding with a "Service unavailable" status. To simulate the unavailability, we can block the database client connection on its hostname so that no incoming or outgoing traffic goes through. The endpoint should recover automatically once the database is reachable again.