Transformer Explained: Attention Is All You Need

Transformer Explained: Attention Is All You Need | Nahraf

Introduction

The Transformer model, introduced in the seminal 2017 paper Attention Is All You Need, reshaped how we process sequences. Unlike recurrent or convolutional networks, it relies solely on attention mechanisms, enabling unprecedented parallelism and state‑of‑the‑art performance on NLP tasks.

Background: From RNNs to Attention

Traditional sequence models (RNNs, LSTMs) suffered from sequential bottlenecks and limited long‑range dependency capture. Early attention mechanisms alleviated these issues by letting the decoder focus on relevant encoder states, but the overall architecture remained recurrent. The Transformer removes recurrence entirely and builds a stack of attention‑only layers.

The Transformer Architecture

The model consists of an encoder and a decoder, each built from identical layers:

Multi‑Head Self‑Attention – learns relationships between all tokens.

Position‑wise Feed‑Forward Network (FFN) – a two‑layer MLP applied independently to each position.

Add & Norm – residual connections followed by layer normalization.

Positional Encoding – injects order information since attention itself is order‑agnostic.

Both encoder and decoder stack N such layers (commonly N=6).

Scaled Dot‑Product Attention

At the heart of the Transformer lies the scaled dot‑product attention function:

python
import math, torch

def scaled_dot_product_attention(Q, K, V, mask=None):
    d_k = Q.size(-1)
    # Compute raw scores
    scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
    # Softmax to obtain attention weights
    attn = torch.softmax(scores, dim=-1)
    # Weighted sum of values
    return torch.matmul(attn, V)

Q, K, V are query, key, and value matrices derived from the same input (self‑attention) or from encoder‑decoder cross‑attention.

Scaling by sqrt(d_k) prevents the softmax from entering regions with extremely small gradients.

Optional masking blocks attention to future tokens (decoder) or pads.

Multi‑Head Attention

Instead of a single attention function, the Transformer splits the embedding dimension d_model into h heads, each with dimension d_k = d_v = d_model / h. This allows the model to attend to information from different representation subspaces.

python
class MultiHeadAttention(torch.nn.Module):
    def __init__(self, d_model, num_heads):
        super().__init__()
        assert d_model % num_heads == 0
        self.d_k = d_model // num_heads
        self.num_heads = num_heads
        self.W_q = torch.nn.Linear(d_model, d_model)
        self.W_k = torch.nn.Linear(d_model, d_model)
        self.W_v = torch.nn.Linear(d_model, d_model)
        self.W_o = torch.nn.Linear(d_model, d_model)

    def forward(self, query, key, value, mask=None):
        batch_size = query.size(0)
        # Linear projections
        Q = self.W_q(query).view(batch_size, -1, self.num_heads, self.d_k).transpose(1,2)
        K = self.W_k(key).view(batch_size, -1, self.num_heads, self.d_k).transpose(1,2)
        V = self.W_v(value).view(batch_size, -1, self.num_heads, self.d_k).transpose(1,2)
        # Apply scaled dot‑product attention on each head
        attn_output = scaled_dot_product_attention(Q, K, V, mask)
        # Concatenate heads
        concat = attn_output.transpose(1,2).contiguous().view(batch_size, -1, self.num_heads * self.d_k)
        return self.W_o(concat)

Positional Encoding

Since attention lacks recurrence, we add sinusoidal positional encodings to the input embeddings:


PE_{(pos, 2i)}   = sin(pos / 10000^{2i/d_model})
PE_{(pos, 2i+1)} = cos(pos / 10000^{2i/d_model})

These functions allow the model to learn relative positions and extrapolate to longer sequences.

Feed‑Forward Networks

Each layer contains a fully‑connected feed‑forward network applied position‑wise:


FFN(x) = max(0, xW_1 + b_1)W_2 + b_2

Typical dimensions are d_model = 512 and inner layer size d_ff = 2048.

Training Tricks

Label Smoothing (e.g., ε=0.1) regularizes the classifier.

Adam Optimizer with β₁=0.9, β₂=0.98, and a custom learning‑rate schedule:

lr = d_model^{-0.5} * min(step^{-0.5}, step * warmup_steps^{-1.5})

Dropout (≈0.1) applied after attention and FFN.

Weight Initialization using Xavier uniform.

Applications and Impact

Transformers quickly became the backbone of models such as BERT, GPT, T5, and Vision Transformers. Their ability to scale with data and compute has driven the current AI boom.

Conclusion

Attention Is All You Need introduced a simple yet powerful paradigm: replace recurrence with multi‑head self‑attention, enrich inputs with positional encodings, and stack these blocks with residual connections. Understanding the math and implementation details equips you to build, modify, and innovate on top of the Transformer foundation.

Blog AI Assistant

more_articles()

How to Deliver a Seamless Offer Acceptance (OA) Process in a Tech Company

Demystifying the Transformer: A Deep Dive into Attention Is All You Need

Introduction

Background: From RNNs to Attention

The Transformer Architecture

Scaled Dot‑Product Attention

Multi‑Head Attention

Positional Encoding

Feed‑Forward Networks

Training Tricks

Applications and Impact

Conclusion

discussion(0)

Leave a comment

Spark vs Flink in 2024: How the Lakehouse is Toppling Flink’s Reign

Why WebAssembly is Winning the Edge War in 2024

Why PWAs Are Finally Outpacing Native in 2026 – A Hot Take