Lion Cipher Lab
Set an availability SLO and alert on error-budget burn rate
- Cyber / Security Engineer
- Intermediate
- ~55 min
- Free · no account required
Why it matters
Availability is one of the three classic security properties, and the hardest to manage by instinct. Alert on every failed health check and the on-call engineer gets paged dozens of times a month for blips. They learn to ignore the pager, and the real outage gets ignored too. Alert only on long outages and you find out from customers. Site reliability engineering (SRE) practice, as described in Google's SRE books, replaces instinct with a target and a budget. You choose a service level objective (SLO), such as 99.9 percent of minutes available over 30 days. The remaining 0.1 percent is the…
What you'll learn
- SLIs, SLOs and error budgets
- computing availability and budget consumption from health checks
- burn rate and why it beats 'alert on any failure'
- the low-traffic problem in burn-rate alerting
- comparing alert policies on pages and detection delay
- the trade-off of confirming failures before counting them
- using an error budget to make reliability decisions
Steps in this lab
- SLI, SLO, error budget, burn rate
- Generate 30 days of health checks
- Write the SLO analyzer
- Run it: the budget, and four ways to page
- Fix the signal, then measure what the fix costs
- Use the budget to decide, not just to alert