Lion Cipher Lab

Detect log anomalies with an autoencoder

• AI Security
• Advanced
• ~70 min
• Free · no account required

Why it matters

A huge amount of defensive work is really one question: is this thing NORMAL, or is it weird enough to look at? You almost never have a clean list of every attack you might face -- attackers invent new ones -- but you usually have a mountain of examples of normal: normal logins, normal network connections, normal process behavior. That asymmetry is the opening. If you can teach a model what normal looks like, then anything it struggles to recognize is, by definition, unusual and worth a human's attention. An AUTOENCODER is a neural network built exactly for that job. You train it only on…

What you'll learn

• what an anomaly-detection problem is when you cannot list the attacks in advance, only describe what normal looks like
• what a neural network is in one honest sentence: layers of simple multiply-add-bend units whose numbers are tuned to fit data
• what an autoencoder is: a network with an encoder that squeezes each row down to a small latent summary and a decoder that rebuilds the row from it
• why training an autoencoder ONLY on normal data is the whole trick: it learns to rebuild normal well and rebuilds anything unfamiliar badly
• what reconstruction error is: how far the rebuilt row is from the original, measured as mean squared error per row
• generating a self-contained set of normal network connections plus a few planted attack shapes so you are never blocked on data
• fitting a StandardScaler on the NORMAL training rows only, so the model's idea of normal is not contaminated by anomalies
• building a tiny CPU-friendly Keras autoencoder (a few Dense layers, a 2-number bottleneck) and training it for a set number of epochs
• choosing a threshold on reconstruction error from the normal training errors, and reading a histogram that shows normal and anomalies separating
• flagging anomalies by high reconstruction error and scoring the result with precision and recall against the planted labels

Steps in this lab

1. The problem: you can describe normal, but you cannot list every attack
2. The words you need: neural network, autoencoder, encoder, decoder, latent space, reconstruction error
3. Install TensorFlow/Keras and the helpers
4. Generate the data: mostly-normal connections plus a few planted attacks
5. Split normal-only, and scale on the normal rows alone
6. Build the tiny autoencoder
7. Train on normal, score every connection, and set a threshold
8. Read the plot and score it: precision and recall
9. The honest limits: mostly-normal data, a tuned threshold, and unusual is not malicious
10. What you can now do that most people can't, and where to go next

Open the full interactive lab →