An unavailable Postgres database might be too severe for suitable fallbacks and requires your system to indicate unavailability appropriately

Motivation

Depending on your context, an unavailable Postgres database may be considered so severe that there are no suitable fallbacks. In this case, ensuring that your system indicates an appropriate error message is essential. After the Postgres database returns, your system should recover automatically.

Structure

We ensure that a load-balanced user-facing endpoint fully works while having all pods ready. When we simulate an unavailable Postgres database, we expect the user-facing endpoint to indicate unavailability by responding with a "Service unavailable" status. To simulate the unavailability, we can block the Postgres database client connection on its hostname so that no incoming or outgoing traffic goes through. The endpoint should recover automatically once the Postgres database is reachable again.