Transformers, Part 1: Tokenization, Embeddings & Positional Encoding
How transformers turn text into meaningful numbers: tokenization, Byte-Pair Encoding, embeddings, and sinusoidal positional encoding, with worked examples.
This article has an interactive companion
Don't just read it — step through it interactively, one state change at a time.
A transformer never sees the word "Hello." It sees numbers. Every headline about attention skips the unglamorous truth: before a single dot product fires, raw text has to become vectors that carry both meaning and order. That conversion is the subject of this post.
Three stages do the work. Tokenization slices text into a fixed vocabulary of pieces and swaps each for an integer ID. Embeddings map each ID to a dense vector in a space where "king" lands near "queen." Positional encoding stamps each vector with where it sits, because attention on its own can't tell "dog bit man" from "man bit dog." No attention happens until all three are done — this is the plumbing, invisible until it leaks: a bad tokenizer inflates context length, a bad positional scheme silently discards word order.
Why does text have to become numbers first?
Neural networks consume and produce floating-point tensors; text is a string of Unicode characters, so something must bridge the gap. The naive bridge assigns every word an integer, and it fails on real language: vocabularies explode into the millions, every typo becomes an out-of-vocabulary hole, and "run," "runs," "running," and "ran" get four unrelated IDs that share nothing. The tokenizer's job is a middle granularity — a small vocabulary that never chokes on an unseen word yet still exposes shared structure like the -ing suffix.
Characters, words, or subwords — the Goldilocks problem
There are three obvious ways to chop text, and two of them are traps.
| Granularity | Vocabulary size | Sequence length | Out-of-vocab words | Cost |
|---|---|---|---|---|
| Characters | Tiny (~100) | Very long | Never happens | Relearns spelling; long sequences |
| Words | Huge (millions) | Short | Constant problem | Any unseen word becomes [UNK] |
| Subwords | Moderate (~30k–50k) | Medium | Never happens | The practical winner |
The two extremes are the traps: characters spell anything but blow up sequence length ("transformer" becomes eleven tokens), while words keep sequences short yet need millions of entries and still emit [UNK] on any word they never saw. Subword tokenization is the Goldilocks answer. Common words stay whole ("the," "running"), while rare or unseen words split into learned fragments — reader → read·er, slowest → slow·est. Nothing becomes [UNK], and the shared er and est pieces carry morphology the model reuses. That's why every modern LLM uses a subword scheme.
How Byte-Pair Encoding builds a vocabulary
Byte-Pair Encoding (BPE) learns those pieces from data instead of hand-writing them. The rule is embarrassingly simple: start from characters, merge the most frequent adjacent pair into a new symbol, and repeat until you hit the target vocabulary size.
def train_bpe(corpus, num_merges):
# each word starts as its characters plus an end-of-word marker
words = {word: list(word) + ["</w>"] for word in corpus}
merges = []
for _ in range(num_merges):
pairs = count_adjacent_pairs(words) # frequency-weighted
if not pairs:
break
best = max_count_pair(pairs) # most frequent adjacent pair
merge_everywhere(words, best) # replace best with one symbol
merges.append(best)
return merges # the ordered merge tableThe </w> end-of-word marker lets the tokenizer distinguish a suffix at a word's end from the same letters mid-word, so "er at the end of a word" becomes its own piece.
Walkthrough: four words, three merges
Take a tiny corpus with these word counts: low×5, lowest×2, newer×6, wider×3. Split every word into characters plus </w> and count each adjacent pair, weighted by its word's frequency. Ranked, the top of that first pass looks like this:
| Pair | Count | Comes from |
|---|---|---|
| e r | 9 | newer (6) + wider (3) |
| r </w> | 9 | newer (6) + wider (3) |
| w e | 8 | lowest (2) + newer (6) |
| l o | 7 | low (5) + lowest (2) |
| o w | 7 | low (5) + lowest (2) |
| n e | 6 | newer (6) |
e r and r </w> tie at 9. Ties break deterministically toward the lexicographically smaller pair, so e r wins. Merge it everywhere, re-count, and repeat:
| Merge | Winning pair | Count | Why it won |
|---|---|---|---|
| 1 | e + r → er | 9 | most frequent; beats r </w> on the tie-break |
| 2 | er + </w> → er</w> | 9 | after merge 1, "newer"/"wider" both end in er</w> |
| 3 | l + o → lo | 7 | now the top pair; ties with o w, wins lexically |
The pair w e = 8 outranks both count-7 pairs but never wins: merge 1 consumes the e in "newer," dropping its count to 2. After three merges the tokenizer has learned the suffix er</w> — a real morpheme — from frequency alone, no linguist in the loop. Run 30,000 merges over a real corpus and you get a full subword vocabulary; at inference the tokenizer replays the merge table in learned order, so an unseen word still segments into familiar pieces.
Embeddings: turning IDs into meaning
Tokenization hands the model a list of integer IDs — say [8, 412, 9]. Those integers are arbitrary labels; ID 412 isn't "more" than ID 8. The model needs vectors it can do math with — the embedding layer.
An embedding layer is a learned matrix of shape vocab_size × d_model. Token ID i selects row i — a lookup, not a multiplication.
# embedding_matrix: shape (vocab_size, d_model), learned during training
def embed(token_id):
return embedding_matrix[token_id] # returns one dense d_model vectorEach row is a point in a d_model-dimensional space (768 for GPT-2, larger for bigger models), and training pushes rows that appear in similar contexts close together. Closeness is cosine similarity — the angle between two vectors, ignoring length:
cosine(a, b) = (a · b) / (|a| · |b|)A cosine of 1.0 means the vectors point the same way (near-synonyms), 0.0 means unrelated, negative means opposed. So "Hello" and "Hi" score high while "Hello" and "banana" sit near zero — this is where a bare integer ID finally acquires meaning.
Does king − man + woman really equal queen?
The famous party trick of embeddings is vector arithmetic. Because directions in the space line up with concepts — a "gender" direction, a "royalty" direction — you can add and subtract word vectors and land somewhere meaningful. The interactive deep dive computes these live:
| Analogy | Nearest word | Cosine | Verdict |
|---|---|---|---|
| king − man + woman | queen | 0.992 | works |
| boy − man + woman | girl | 0.999 | works |
| paris − france + italy | rome | 0.998 | works |
| london − paris + rome | italy | 0.569 | honestly wrong |
The first three land cleanly: subtract the "male" direction, add the "female" direction, and king slides over to queen. The last row is honest — the same arithmetic that nails capitals sometimes returns a low-confidence miss, because the geometry only approximately encodes these relationships. It's a genuine emergent property, not a guarantee.
Positional encoding: injecting order into a bag of words
Once you have embeddings, self-attention treats its input as an unordered set. Feed it "dog bit man" and "man bit dog" and — without positional information — it sees the identical multiset of three vectors and cannot tell the sentences apart. Word order, which carries most of the meaning, is gone.
The fix from the original transformer paper is sinusoidal positional encoding: a fixed vector for each position, added to the token embedding. Sine goes on even dimensions, cosine on odd ones, each at a geometrically increasing wavelength:
PE(pos, 2i) = sin(pos / 10000^(2i / d))
PE(pos, 2i+1) = cos(pos / 10000^(2i / d))Low dimensions oscillate fast, high ones slowly, and the combined phases give every position a unique fingerprint — with zero learned parameters. In code:
function PE(pos, d, base = 10000) {
const v = [];
for (let i = 0; i < d; i++) {
const angle = pos / base ** (2 * Math.floor(i / 2) / d);
v[i] = i % 2 === 0 ? Math.sin(angle) : Math.cos(angle);
}
return v; // a unique fingerprint per position
}
// inject order: add PE(pos) to each token embedding
const h = tokens.map((tok, pos) => add(embed(tok), PE(pos, d)));Now "dog bit man" and "man bit dog" produce genuinely different inputs: the position-0 slot holds dog + PE(0) in one and man + PE(0) in the other. Permutation invariance is broken, at no extra weight cost.
One more elegant property: the dot product PE(p) · PE(p+k) depends only on the offset k, not the absolute position p. So a head can learn "look two tokens back" once and apply it at every position, even past the training length.
What to remember
- Text becomes numbers in three stages — tokenize → embed → add position — and no attention runs until all three finish.
- Subword tokenization (BPE) keeps the vocabulary moderate, never emits
[UNK], and shares morphemes like-ingacross words, learned by greedily merging the most frequent adjacent pair — simple enough to trace by hand. - Embeddings are a learned lookup table; cosine similarity scores closeness, and analogies like
king − man + woman ≈ queenemerge from it (and occasionally miss). - Sinusoidal positional encoding breaks permutation invariance for free, and its dot product depends only on relative distance.
Keep going
This is the input pipeline; attention is what acts on it. Continue with Transformers, Part 2: Self-Attention & Multi-Head Attention to see how these position-aware vectors talk to each other, then What Changed Since 2017: RMSNorm, SwiGLU, RoPE, KV Cache & GQA for how modern models swapped sinusoidal encoding for rotary embeddings. To build the whole thing, Train Your First Language Model: From Raw Text to Generation walks the full loop, and LLM Fine-Tuning, Part 1: When to Fine-Tune & How LoRA Works covers adapting a trained model.
Prefer to poke at it directly? Open the interactive deep dive to watch BPE self-organize a vocabulary, rank the vocab by real cosine similarity, and slide the positional-encoding waves until the relative-distance property clicks.
FAQ
What is the difference between tokenization and embedding?
They are two separate stages. Tokenization is text processing: it chops a string into pieces from a fixed vocabulary and swaps each for an integer ID. Embedding happens next: each integer ID indexes a row of a learned matrix, turning an arbitrary label like ID 412 into a point in a meaning-carrying space. Tokenization decides how text is chunked; embedding decides what each chunk means.
Why do transformers use Byte-Pair Encoding instead of just splitting on words?
Word splitting has no graceful way to handle a token it never saw in training. Fix a word vocabulary and every future typo, hashtag, or code identifier that isn't in it collapses to a single [UNK] — the model can't even tell two unknown words apart. BPE removes that failure mode: because its symbols bottom out at individual characters, there is always some segmentation for any input, and common strings still merge up into whole units so ordinary text pays no length penalty. You get graceful degradation instead of a hard vocabulary wall.
What does positional encoding actually do?
It hands the model a way to recover where each token sits, which raw attention discards. Before the first attention layer you add a position-dependent vector to every token embedding, so the same word carries a different signature at slot 0 than at slot 7. The sinusoidal version builds those signatures from sine and cosine waves at many frequencies, which buys two things: positions never collide, and slot 100 uses the same formula as slot 2, so it keeps working at lengths longer than anything seen in training. None of it is learned — pure arithmetic layered onto the embeddings.
Does king − man + woman = queen always work?
No — it is a reliable tendency, not a guarantee, and there is a subtlety most demos quietly hide. The nearest-neighbor search has to exclude the three input words, because the closest vector to king − man + woman is usually just king itself — the arithmetic only nudges the point, it doesn't leave the neighborhood. Once the inputs are removed, whether the answer lands depends on how cleanly the relevant contrast was learned: high-frequency, well-attested relationships (pluralization, verb tense, common gender pairs) carve sharp, consistent directions, while sparse or many-sensed words blur the direction until the top neighbor drifts to something plausible but wrong. Read the trick as a diagnostic of how regular a relationship is in the corpus, not as a symbolic reasoning engine.
Make it stick: run this one yourself
Don't just read it — step through it interactively, one state change at a time.