Lion Cipher Lab
Rotate a database password with zero downtime: build AWS Secrets Manager's four-step rotation offline and test how it fails
- Cloud Security
- Intermediate
- ~75 min
- Free · no account required
Why it matters
Almost every application holds at least one password it uses to log in to its database. That password is a long-lived secret: it sits in config, in the secrets manager, in the memory of every running server, and sometimes in a log line or a backup. If it leaks and never changes, whoever has it can log in forever. Rotation means changing that password on a schedule, so a leaked copy has an expiry date. The hard part is doing it without breaking the app, because for a few seconds the database and the apps disagree about what the password is. AWS Secrets Manager solves this with a four-step…
What you'll learn
- explaining what secret rotation is and what it buys you: a leaked database password stops working on a deadline instead of never
- reading AWS Secrets Manager's version model, where a secret is a set of versions and the staging labels AWSCURRENT, AWSPENDING and AWSPREVIOUS say which one is which
- building the documented four-step Lambda rotation contract (createSecret, setSecret, testSecret, finishSecret) from scratch in standard-library Python, against a SQLite stand-in for database users
- measuring the gap between setSecret and finishSecret: how many app logins fail with the single-user strategy, and why the alternating-users strategy gets that to zero
- proving why createSecret saves the new password in AWSPENDING BEFORE the database changes: a crash becomes a safe retry instead of a lockout
- seeing what an aborted testSecret does and does not protect, and why the answer differs between single-user and alternating-users rotation
- sizing a client's secret cache TTL against the rotation interval, and measuring how alternating users doubles how long a leaked password stays valid
- judgment: when to rotate, when to let the database service manage rotation, and when to have no stored password at all (IAM database authentication, dynamic credentials)
Steps in this lab
- What rotation is, and the one thing it buys you
- Set up: a terminal, Python, and an empty folder
- The contract: a secret is a set of versions, and labels say which is which
- The four steps, and why they come in this order
- Build the machinery: rotlib.py
- Write experiment.py and watch the labels move
- Measure the gap: single user vs alternating users
- Failure 1: the function crashes right after setSecret
- Failure 2: testSecret fails. What did the abort actually protect?
- Size the client cache against the rotation interval
- Watch for rotation failures: a failed rotation is a silent ticking clock
- Better than rotating: let the service rotate it, or have no stored password at all
- What rotation does NOT fix, and how a pro runs it