Lion Cipher Lab

Offload CIDR and time-window math to code with program-aided reasoning, and measure where the errors move (deterministic stub model)

• AI Security
• Intermediate
• ~90 min
• Free · no account required

Why it matters

Analysts ask language models small exact questions constantly - is this IP in that block, do these rules overlap, how many failures landed in this window - and a model answering in text gets them wrong more often than anyone expects, because it does bit arithmetic and timezone shifts one fallible step at a time. The standard fix, program-aided reasoning, has the model write a few lines of Python and lets the interpreter do the math. This lab measures that fix instead of taking it on faith, on 300 answer-keyed questions and a deterministic stub model whose slip and misread rates are written…

What you'll learn

• building a 300-question answer key for exact SOC questions - CIDR membership, usable-host counts, rule overlap, timezone-shifted window counts, 90-second proximity, and a no-arithmetic control - with every true answer computed by Python's ipaddress and datetime
• measuring chain-of-thought against program-aided answers per question family: overall 0.727 -> 0.820, window 0.400 -> 0.620, near 0.600 -> 0.780, ports unchanged at 0.920, overlap down 0.880 -> 0.860
• separating arithmetic errors from translation errors: text got 82 wrong (34 slips, 33 misreads, 11 both), programs got 54 wrong with zero arithmetic errors and 43 misreads computed exactly wrong
• writing a restricted evaluator for model-written code with ast whitelisting and an opcode budget, and knowing why it is a guard rail and not a sandbox
• demonstrating that executing model-written code is an attack surface: a poisoned log field produced a whitelisted program that returned 0 instead of 4
• replacing free code with unit-checked, literal-argument helpers plus grounding, which moved the result to 265 correct, 15 abstained and 20 wrong
• deciding when to offload - exact, many-step deterministic subproblems - and when not to: knowledge questions, and questions whose hard part is understanding

Steps in this lab

1. The words first: chain-of-thought, program-aided reasoning, CIDR, time windows, and the fact that the model here is a stub
2. Open a terminal and make a scratch folder (Windows, macOS, Linux)
3. Build 300 answer-keyed analyst questions in six families
4. Write the STUB model: one reader, two ways to answer - chain-of-thought text, or a program
5. Build a restricted evaluator for model-written code - a guard rail, not a sandbox
6. Measure it: accuracy per family, before (CoT text) and after (program-aided)
7. Read the result honestly: the errors did not vanish, they moved
8. Executing model-written code is an attack surface - poison one log field and watch
9. The way a pro does it: unit-checked helpers the model CALLS, grounded in the question text
10. Why this, not that - and when NOT to offload
11. Ship it: the policy, what to log, and one experiment to run yourself

Open the full interactive lab →