Lion Cipher Lab
Stop a runaway log from filling the disk with logrotate (and measure what create vs copytruncate really cost)
- Foundations
- Beginner
- ~60 min
- Free · no account required
Why it matters
A log that grows forever is one of the most common outages there is: an app gets stuck in an error loop, writes the same warning thousands of times a second, and a few hours later the disk is 100% full. Databases stop accepting writes, services crash, and nobody can even log in to fix it. For a security person there is a second, quieter cost: when the disk is full, the security log cannot be written either. In this lab you will watch a failed-login event be thrown away with 'No space left on device' -- the exact moment you most need a record is the moment you lose it. The standard Linux tool…
What you'll learn
- reproducing a disk-full outage safely inside a throwaway Docker container with a small tmpfs 'disk', and measuring it with du and df
- seeing why a full disk is a security problem: the security log itself can no longer be written
- writing a logrotate config (size, rotate, compress, delaycompress, missingok, notifempty, create, postrotate) and testing it with a dry run (-d) before forcing it (-f -v -s)
- the difference between create (rename the file, then tell the app to reopen) and copytruncate (copy, then empty the original in place), and measuring what each one costs
- spotting the classic 'du says small, df says full' symptom of a process still writing to a deleted file, and finding it in /proc/<pid>/fd
- where distributions actually run logrotate (cron.daily or the logrotate.timer systemd unit) and why a daily run cannot save a disk that fills in minutes
- why rotation is not a retention policy: rotate count decides how much evidence survives
Steps in this lab
- Build a throwaway container with a tiny 32 MB 'disk'
- Write a fake app that logs forever, the way a real daemon does
- BEFORE: let it run away, measure the outage, and watch a security event get thrown away
- Write a logrotate config and dry-run it before you trust it
- Force it, and watch create fill the disk anyway (du says small, df says full)
- AFTER: create + postrotate signal + delaycompress, run often enough to matter
- copytruncate: no signal needed, but measure the lines it drops
- Where logrotate really runs, and why a daily run cannot save a fast-filling disk
- The honest negative: rotation is not retention, and logrotate only guards the files it knows
- The judgment: when to reach for which setting