AI
LLM
Machine Learning
Transformers
Deep Learning
Blog AI Assistant
Ask anything about this article
Hi! I've read this article.
What would you like to know?
Ask anything about this article
Hi! I've read this article.
What would you like to know?
@farhan
Everyone uses ChatGPT, but few understand how it works. Let's fix that. No math PhD required.
LLMs don't see words — they see tokens (sub-word pieces):
Input: "Understanding transformers is fascinating"
Tokens: ["Under", "standing", " transform", "ers", " is", " fascin", "ating"]
IDs: [8521, 4975, 7062, 388, 318, 24549, 803]
Each token becomes a vector (list of numbers) in high-dimensional space:
"king" → [0.2, -0.4, 0.8, 0.1, ...] (768 dimensions)
"queen" → [0.3, -0.3, 0.7, 0.2, ...]
"man" → [0.1, -0.5, 0.3, 0.6, ...]
"woman" → [0.2, -0.4, 0.2, 0.7, ...]
The famous equation works in embedding space:
king - man + woman ≈ queen
Self-attention lets each token "look at" every other token to understand context:
Sentence: "The bank by the river was steep"
Without attention: "bank" = financial institution? riverbank?
With attention: "bank" looks at "river" → riverbank! ✅
For each token:
Head 1: Focuses on syntax (subject-verb agreement)
Head 2: Focuses on entities (proper nouns)
Head 3: Focuses on coreference ("he" → "the doctor")
...
Each block has two sub-layers:
Input
↓
[Multi-Head Self-Attention] ← "What's related to what?"
↓ + Residual Connection
[Layer Normalization]
↓
[Feed-Forward Network] ← "Transform the representation"
↓ + Residual Connection
[Layer Normalization]
↓
Output
GPT-4 has ~120 of these blocks stacked together.
LLMs generate one token at a time by predicting probabilities:
Input: "The capital of France is"
Probs: {"Paris": 0.92, "Lyon": 0.03, "the": 0.02, ...}
Output: "Paris"
New input: "The capital of France is Paris"
Probs: {".": 0.75, ",": 0.15, "and": 0.05, ...}
Output: "."
Temperature 0.0: Always picks highest probability (deterministic)
Temperature 0.7: Balanced (default for most tasks)
Temperature 1.5: Very creative/random (might say wild things)
LLMs generate the most probable next token based on patterns in training data. They don't:
They're essentially the world's most sophisticated autocomplete.
| Model | Parameters | Training Data | Training Cost |
|---|---|---|---|
| GPT-3 | 175B | 300B tokens | $4.6M |
| GPT-4 | ~1.7T (MoE) | ~13T tokens | ~$100M |
| Llama 3 (405B) | 405B | 15T tokens | ~$40M |
| Gemini Ultra | ~1T+ | Undisclosed | ~$200M+ |
Understanding how LLMs work helps you use them better, prompt them more effectively, and build better AI applications.
