Lion Cipher Lab

Seed a fuzzer with model-generated inputs and measure the coverage gain against random and hand-written seeds (deterministic STUB model)

• AI Security
• Advanced
• ~2 hours
• Free · no account required

Why it matters

Google now uses large language models to write fuzz targets and inputs for open-source projects (OSS-Fuzz-gen), and research systems such as Fuzz4All and Project Zero's Big Sleep use them to reach code that blind mutation never touches. The pitch is simple: a model has read millions of log lines and file formats, so it can hand a fuzzer inputs that already get past the format checks. This lab measures that pitch instead of believing it. On a toy CEF security-log parser with five planted bugs, random bytes covered 1 arc and random text 13, because neither ever wrote the four bytes CEF: that…

What you'll learn

• building a coverage-guided mutation fuzzer in pure Python that records arc (line-to-line) coverage with sys.settrace and names each planted bug it hits
• comparing seven seed strategies on the same parser and budget - random bytes 1 arc, random text 13, minimal hand seed 71, one real log line 105, STUB model seeds 97, STUB plus the real line 113 - over 5 repeats each
• reading time-to-coverage instead of end-state coverage: STUB plus the real line reached 105 arcs by input 50, which the real line alone needed 1000 inputs to reach
• spotting a format-prior blind spot: model-shaped seeds found the deep start == end bug on the seeds themselves but found the syslog-PRI bug in 0 of 5 runs, because the prior never wrote a syslog wrapper
• testing whether random catches up with patience: random text sat at 13 arcs after 30000 inputs, and an auto-extracted dictionary only crept to 45
• deciding when model-generated seeds are worth it (structured formats behind magic gates, no sample corpus) and when a real sample, a dictionary, or a grammar is the cheaper first move

Steps in this lab

1. The words first: fuzzing, seeds, coverage, magic gates, and the fact that the model here is a STUB
2. Open a terminal and make a scratch folder
3. Write the target: a toy CEF/syslog parser with five planted bugs
4. Write the seed sets, including the STUB model that stands in for an LLM
5. Write the fuzzer: mutation plus arc coverage from sys.settrace
6. Measure: seven seed sets, five repeats, 3000 mutated inputs each
7. Read the result: the gain is real, and most of it is getting past the gate
8. The honest negatives: what the model-shaped seeds did NOT do
9. Does random catch up if you just wait? Run it ten times longer
10. The judgment: when to ask a model for seeds, when not to, and how a pro does it
11. Optional appendix: replace the STUB with a free local model through Ollama

Open the full interactive lab →