Turn time forward and check whether your TLS/SSL certificates are valid.

Motivation

Noticing the TLS/SSL certification expiry too late is one problem you can easily avoid by frequently checking your expiry dates. While observability tools already handle this job nicely, you can't know whether they are working in your environment. With this experiment, you can turn the time forward to check whether your HTTPS endpoint works at a given date in the future. Additionally, you can configure one of the observability integrations to validate your observability tool's alerting.

Structure

First, we validate that the given HTTPS endpoint is working today. Next, we will travel with the Windows host in time to validate that the HTTPS endpoint continues to work on a given date. If the TLS/SSL certificate has already expired at that date, the HTTP check will throw failures.

Warning

Please be aware that we will manipulate the time for a given Windows host. Applications running at that Windows host may struggle to deal with the change in the clock correctly, and you may experience other side effects.

When a host suddenly loses connection to the network and your system, Datadog should alert about this. Eventually, everything should recover once the network is back again.

Motivation

When you're working in a less volatile system environment, a loss of network can be crucial as there is likely no backup host that will enable faster recovery. Thus, you should check your observability tools to catch this.

Structure

Before blocking a host from the network, we verify that the Datadog monitor is in an ok state Afterward, we block all traffic to and from a host and expect Datadog to alert about the isolated host. Eventually, when the host is online again, we expect Datadog to turn into an OK state again. While experimenting, we create a downtime for the Monitor so that it will not escalate due to the ongoing alert.

When a windows host is suddenly missing from your system, Datadog should alert you to this. Eventually, everything should recover when only rebooting the host.

Motivation

When you're working in a less volatile system environment, where you expect hosts always to run, you should validate whether you notice whenever a host is rebooting.

Structure

Before restarting a host, we verify that the Datadog monitor is in an ok state Afterward, we trigger the shutdown of a host and expect Datadog to alert about the missing host. Eventually, the host should come back and Datadog turn into an OK state again. While experimenting, we create a downtime for the Monitor so that it will not escalate due to the ongoing alert.

Fill up the memory of a Host progressively to see at which percentage it becomes unstable. Additionally, you may want to add one of our observability checks.

Structure

We start by filling 50% of the host's memory for 30 seconds. Next, we stepwise fill the memory to 75%, 90%, and 100% - each for 30 seconds. In between, we have small wait steps to ease analysis in external observability tools for each phase.