A SMALL MODEL. AN OPEN NOTEBOOK.
Language, one
character at a time.
A tiny transformer, trained from scratch. Give it a beginning, watch it predict what comes next, and look at the numbers underneath.
THE WHOLE IDEA
Read the context.
Predict one character. Repeat.
01 / TRY THE MODEL
Start a sentence. See what follows.
Its next 48 characters will appear here.
This is a toy language model, not a chatbot. It learns small text patterns; expect repetition, mistakes, and invented words.
Your prompt stays in this browser. No paid API. No pretrained language model.
02 / LOOK INSIDE
Small pieces. Visible mechanics.
A view of one forward pass
through the trained model.
Characters → token IDs
Each character has a number. Those IDs select the vectors the model works with.
Token IDs will appear when the model is ready.
What could come next?
The most likely next characters, before temperature changes the sampling distribution.
Waiting for predictions.
Attention to earlier characters
Rows read from columns. Darker cells mean greater attention weight in this head. Future positions are masked.
The attention map will appear here.
Final layer · head 1. Attention weights show one part of the computation; they are not an explanation of the model’s reasoning.
03 / THE TRAINING RECORD
Learning leaves a trace.
Recorded from a real training run.
These weights are already trained.
Loading the training record…
Next-character loss
Lower means less surprise at the correct next character. Validation uses text held out from training.
No chart until the recorded measurements load.
A falling loss on this small dataset is evidence of learned patterns, not a measure of general intelligence.
Same task, before & after
Actual samples saved with the training run. Imperfect text is part of the record.
Loading recorded sample…
Loading recorded sample…
- Training steps
- —
- Training loop only
- —
- Training device
- —
- Random seed
- —
What did this model learn from?
The dataset description will appear when the training record loads.
04 / CONNECT THE DOTS
One loop makes it a language model.
- 01
Turn text into numbers.
The tokenizer maps each character to an ID. Learned embeddings turn those IDs into vectors.
- 02
Use the available context.
Causal attention combines information from earlier characters. A position can’t read the future.
- 03
Learn from the next character.
Training compares predictions with the actual next character. Gradients adjust the weights to reduce that loss.
- 04
Sample. Append. Repeat.
Generation samples one character, adds it to the context, and runs the model again. Training is already finished.
Keep the scale in mind. This experiment makes a transformer’s mechanics visible. Its tiny dataset and short context limit what it can produce. It does not retrieve facts or verify what it writes.