What to watch
Two endpoints, both already public and both cheap.
Watch the customer facing origin, not the container. A probe against
localhost on the machine tells you the container is up while every customer sees nothing.
The second endpoint reports each dependency separately rather than as one boolean, so a probe that records the body distinguishes a database outage from a stalled pipeline without anyone opening a terminal.
The interval
One minute. Both endpoints touch PostgreSQL and Redis on every request and neither is authenticated, so a five second interval from several regions is a self inflicted load test against the smallest part of the deployment. One minute from two or three regions is enough to tell an outage from a blip, and cheap enough that the probe never shows up in the capacity numbers.Where the alert has to land
A device that is not the mini PC. This is the part that is easy to get wrong. An email alert is worthless if the SMTP relay is configured in the instance that just went down, or if the notification is delivered to a mailbox you read on a laptop that is off at 3am. Route the probe’s alerts to a phone: push, SMS or a chat app that pushes. The probe is a hosted service, so its alerts leave from somewhere unaffected by your outage. That is the entire point of it being outside.Certificates
With the shipped Cloudflare Tunnel, the certificate is Cloudflare’s and renews without you. There is nothing on the machine to expire. If you brought your own reverse proxy, add a certificate expiry check to the same probe. An expired certificate is an outage that every uptime check reports as a connection failure, and every hour spent diagnosing it is an hour not spent on the one line that fixes it.The status page
Host it off the machine. A status page that shares a power supply with the service is decoration: the outage that takes down the service takes down the page that would have explained it. Every hosted status page provider will drive the page from the same probe. Point the probe at the two endpoints above, publish the page, and put the URL in three places:STATUS_PAGE_URLin the web app environment, which links the app’s own error states to it. See Web app configuration.- Support, so somebody looking for help finds the page before they find the contact address.
- The terms, so the availability you promise and the availability you publish are the same document trail.
What to write during an incident
Decide this before the incident, because during one you will not.
The first update costs nothing and is the only one that matters to somebody deciding whether to open a support ticket.
Verify it
Unplug the network on the machine. The probe must alert within its stated interval, to a device that is not on that network. That is the whole test and it takes two minutes. A probe nobody has watched fire is a belief, not a signal.Related
Hosted observability
The alerts asobeast evaluates about itself.
Health checks
What each endpoint reports and why.