Lion Cipher Lab
Encrypt Kubernetes Secrets at rest, part 2: rotate the key and catch the silent failures
- Cloud Security
- Advanced
- ~75 min
- Free · no account required
Why it matters
Turning on encryption at rest is a one-time change. Keeping it on is operations, and the failures there are quiet. A provider list with identity at the top still looks enabled while every new Secret goes to etcd in the clear. A rotation done in the wrong order leaves some API servers unable to read what others wrote, or strands Secrets on a key that has been deleted. A migration that missed a few records passes a spot check of one Secret. None of these shows up as an error. You find them by knowing what each change should do to etcd and checking every stored Secret, not a sample. This part…
What you'll learn
- changing a running API server's encryption config with --encryption-provider-config-automatic-reload
- confirming a config reload landed from the API server's own metric
- reproducing the identity-first failure, where encryption is silently off, and repairing it
- rotating a local encryption key in the documented order: stage, promote, re-write, retire
- auditing every Secret in etcd by its k8s:enc: prefix rather than trusting one sample
- removing the identity fallback once no plaintext is left
- stating what encryption at rest defends and what it does not
Steps in this lab
- Where Part 1 left you, and the failures it could not show
- Rebuild Part 1's end state in one pass, with secretbox and automatic reload
- Break it on purpose: put identity first
- Rotate the key, in the order the docs give
- Audit every Secret, then retire key1 and identity
- The judgment: what this defends, and the problem it leaves