Lion Cipher Lab

Make a build reproducible, and diff it byte for byte

• Cyber / Security Engineer
• Advanced
• ~120 min
• Free · no account required

Why it matters

Every serious conversation about software supply chain security ends up leaning on the same unspoken assumption: that we can tell whether the binary somebody handed us is really the binary that our source code produces. Signing does not establish that -- a signature proves who published the bytes, and a compromised build server publishes perfectly valid signatures over backdoored bytes all day. An SBOM does not establish it either; an SBOM is an inventory of what went in, and an attacker who owns the compiler can hand you a truthful inventory alongside a poisoned artifact. Provenance and SLSA…

What you'll learn

• what a reproducible build actually is (same source + same declared build environment => bit-for-bit identical artifacts) and why it is a SECURITY control, not a tidiness project: it is the only thing that lets an independent rebuilder detect a compromised build server
• enumerating and MEASURING every source of build nondeterminism separately: embedded timestamps, filesystem file order, uid/gid/uname/gname and permission bits, build paths baked into bytecode, timezone, compression level and zlib version, environment leakage, and per-process hash randomization
• the two-copy trap in the zip format: every member's date/time is stored in BOTH the local file header and the central directory, so a normalizer that patches one leaves you with a file that is nondeterministic AND internally inconsistent
• gzip's header carries a timestamp AND the output filename, so 'w:gz' can never be reproducible -- and the exact stdlib incantation (gzip.GzipFile(filename='', mtime=0)) that fixes it
• PEP 552 hash-based .pyc: why timestamp invalidation guarantees a different artifact every time, what the 16-byte pyc header actually contains, and how the absolute build directory leaks into the bytecode through co_filename
• SOURCE_DATE_EPOCH as the spec really defines it -- CLAMP timestamps down to it, never SET them to it -- demonstrated on a file whose mtime is older than the epoch, plus the zip format's hard 1980 floor
• writing a diffoscope-lite from scratch: recursive container descent (gzip -> tar -> zip -> pyc), member-list, per-member metadata and per-member content comparison, and a triage verdict that separates PACKAGING-ONLY noise from CONTENT DIFFERS signal
• the false sense of security in hash-only verification: archives whose members are all byte-identical still fail sha256 if the ORDER moved, which makes every downstream signature, SBOM checksum and SLSA subject digest unstable
• reading the honest limits: what is portable across operating systems and filesystems (measured identical on NTFS/Windows and ext4/Linux) versus what is pinned to your exact toolchain version (measured different between CPython 3.12.10 and 3.12.3)
• the judgment call -- normalize at build time vs post-process the artifact vs record the whole build environment -- when reproducibility is not worth the effort, and the professional pattern of treating the DIFF as the deliverable rather than the hash

Steps in this lab

1. What 'reproducible' means, and why it is a security control
2. Set up (assume nothing -- open a terminal and check Python)
3. Create buildpkg.py -- a small but realistic build, in two versions
4. Create reprodiff.py -- your diffoscope-lite, written from scratch
5. Create harness.py and run the whole thing
6. Section [1]: build it twice, and watch sha256 fail you
7. Section [2a] and [2b]: timestamps -- and the fact that zip stores them TWICE
8. Section [2c]: file order comes from the filesystem, and it is stable HERE
9. Section [2d]: who built it -- uid, gid, uname, gname and permission bits
10. Section [2e]: the .pyc -- a timestamp AND your build directory, baked into bytecode
11. Sections [2f] through [2i]: timezone, compression, environment, and hash randomization
12. Section [3]: SOURCE_DATE_EPOCH -- the spec says CLAMP, not SET
13. Section [4]: the fix, measured -- five builds, five directories, one hash
14. Failure mode 1: 'we normalized the timestamps but forgot the central directory'
15. Failure mode 2: 'we sorted the file list and it STILL differs' -- directory entries
16. Failure mode 3: 'byte-identical on my machine, different on yours'
17. Failure mode 4: the false sense of security -- identical members, unstable hash
18. Section [5e]: the payoff -- can you spot a poisoned build?
19. Judgment: when to do this, when not to, and how a professional does it differently
20. Where to go next (all optional, none required)

Open the full interactive lab →