Lion Cipher Lab

Rebuild an incident timeline across broken clock domains

• SOC Analyst / Blue Team
• Advanced
• ~150 min
• Free · no account required

Why it matters

Every correlation rule you own, every timeline you have ever put in front of a client, and every sentence you have written of the form 'the attacker did A and then B' rests on an assumption nobody in the industry audits: that the timestamps across your log sources mean the same thing. They do not, and it is not close. A domain controller drifts 38 seconds because its NTP client silently gave up months ago. A firewall speaks RFC 3164 syslog, which has no year in it at all and pads the day with a space. A cloud appliance stamps records at INGEST time because it never recorded an event time. A…

What you'll learn

• reading a multi-source log estate as a set of independent CLOCK DOMAINS rather than one timeline, and naming each source's specific pathology: unit error, epoch error, missing offset, missing year, wrong configured zone, drifted clock, monotonic-only counter, or no event time at all
• the arithmetic that makes timestamps portable: epoch seconds vs epoch milliseconds (a 1000x error that lands 2025 in the year 57,807), Windows FILETIME as 100-nanosecond ticks since 1601-01-01 (a 369-year offset if you forget the epoch shift), and detecting the unit from magnitude instead of trusting a field name
• PEP 495 fold semantics, done for real: an ambiguous local timestamp in the repeated DST hour maps to TWO instants 3600 s apart, a local timestamp in the skipped hour maps to NO instant, and neither round-trips -- plus how to resolve a fold from a co-observed event and how to declare it unresolvable when you cannot
• writing a fold-aware tzinfo class from scratch so the lab runs on a machine with no tzdata at all, including the hand-written fromutc() that the default implementation gets wrong on the skipped hour, and MEASURING the fallback against the real tz database instead of hoping
• estimating per-source clock offsets with no NTP, from events provably observed by two sources: median pairwise deltas, a co-observation graph solved breadth-first so a source two hops from the reference still gets an estimate, and why the MEDIAN and not the mean (measured: 153.191 s of error from three bad records in 141)
• the difference between a timeline that is CONSISTENT and one that is TRUE: relative alignment leaves every source agreeing with every other and the whole case off by the reference clock's own error, and only an anchor declared from outside the data fixes that
• turning 'the timeline is wrong' into a number with Kendall tau and an inversion count you compute yourself with a merge sort, per source and overall, so a clock problem becomes something you can put on a slide and re-measure after the fix
• quantifying the SOC cost: the same failed-auth-then-success-then-spawn correlation rule run over raw and corrected timestamps across ten window widths, counting detections missed, false correlations created, and the honest counter-result that a wider window buys back true positives at a measured false-positive price
• a canonical event envelope that carries event time, observed time, offset applied, source precision and an uncertainty half-width -- and a timeline that outputs UNORDERED PAIRS instead of guessing, so an analyst can see exactly where 'A then B' is unsupported
• the named failure modes, each with a measured cost: sorting on ingest time instead of event time, precision collapse producing ties, a year-less RFC 3164 line parsed on 1 January, a leap-second smear that reorders 9.01% of nearby event pairs, and two sources in the same wrong time zone that agree with each other perfectly

Steps in this lab

1. The assumption nobody audits (and which labs this is NOT)
2. The words you need, defined once
3. [0] Open a terminal and check Python (assume nothing)
4. [1] Write tzfallback.py -- a US Eastern time zone, by hand, with no tzdata
5. [2] Write clockgen.py -- one incident, eight clock pathologies
6. [3] Run clockgen.py and read the corpus it built
7. [4] Look at one line from each of the eight sources
8. [5] Write clockparse.py -- two parsers and one canonical envelope
9. [6] Write clockfix.py -- the measurement and correction harness
10. [7] Write probes.py -- five isolated experiments
11. [8] Run the harness -- section [1], what the obvious parser does
12. [9] Section [2] -- format fixed, time still wrong, per source
13. [10] Section [3] -- Kendall tau, or 'the timeline is wrong' as a number
14. [11] Section [4] -- ties, or the events your timeline cannot order at all
15. [12] Section [5] -- the DST census: the repeated hour
16. [13] Section [5], continued -- the skipped hour, and why it is worse
17. [14] Section [6] -- estimating clock offsets with no NTP at all
18. [15] Section [7] STAGE 2 -- a zone bug is not a clock bug, and it took two rounds
19. [16] Section [7] STAGE 3 -- offsets, hops, and the difference between consistent and true
20. [17] Section [7] STAGE 4 -- the switch that only counts uptime, and reboots
21. [18] Section [7] STAGE 5 -- the error you cannot remove, and stages that are not independent
22. [19] Section [7b] -- the same number, after
23. [20] Section [8] -- the whole point: what this costs in detections
24. [21] Section [8] continued -- the counter-results, kept rather than smoothed
25. [22] Section [9] -- carrying uncertainty, and refusing to order what you cannot
26. [23] Section [10] -- sorting on the wrong field, and what the WEC pair really tells you
27. [24] Run probes.py -- [P1] the hand-built zone versus the real database
28. [25] [P2] The year-less syslog line, on the one night of the year it matters
29. [26] [P3] Leap seconds: a step, a smear, and 9% of pairs reordered
30. [27] [P4] Precision is not accuracy
31. [28] [P5] Three standard-library traps that produce a wrong instant with no error
32. [29] Prove it is deterministic, and prove it runs with no tz data
33. [30] The named failure modes, each with its number
34. [31] When NOT to do any of this
35. [32] Why the SIEM's normalization layer is the wrong place for half of this
36. [33] What to write into the detection's own metadata
37. [34] The adversarial cousin: timestomping, and why this lab is not that
38. [35] Take it to your own estate (and what to do first)
39. Where to go next (all optional, none required)

Open the full interactive lab →