First-Principles Mathematical Derivations for Modern Transformers

Part 2: Complex Numbers & Geometric Rotation — High-School Derivation of RoPE

Rotary Position Embedding (RoPE) is celebrated as one of the most brilliant mathematical inventions in modern LLMs. Behind its formidable reputation lies only high-school complex multiplication and trigonometric addition formulas!

2.1 High-School Complex Numbers: The Rotational Geometry of Imaginary $i$

In algebra, to solve $x^2 = -1$, mathematicians defined the imaginary unit $i = \sqrt{-1}$. A complex number has the form $z = a + bi$ ($a$ real, $b$ imaginary).

On the complex Gauss plane:

• The horizontal axis is real, vertical is imaginary. Complex $z = 1$ is point $(1, 0)$;
• Multiply by $i$: $1 \times i = i$, reaching $(0, 1)$—a $90^\circ$ counterclockwise rotation;
• Multiply by $i$ again: $i \times i = i^2 = -1$, reaching $(-1, 0)$—another $90^\circ$ rotation;
• Multiply by $i$ again: $-1 \times i = -i$, reaching $(0, -1)$—another $90^\circ$ rotation;
• Multiply by $i$ once more: $-i \times i = -(-1) = 1$, returning to the start!

Core High-School Mathematical Insight: In complex numbers, multiplication is geometrically identical to rotation!

===================================================================================
                       复数平面上连续乘 i 的 90度 几何旋转
===================================================================================

                              虚数轴 (Im)
                                   ▲
                                   │  (0, 1) = i
                                   │   ▲
                                   │   │ 乘 i (逆时针 90°)
                                   │   │
        (-1, 0) = -1 ◄─────────────┼─────────────► (1, 0) = 1
                乘 i (又转 90°)    │   │  实数轴 (Re)
                                   │   │
                                   │   ▼ 乘 i (再转 90°)
                                   │  (0, -1) = -i
                                   ▼
===================================================================================

2.2 Euler's Formula: Rotating at Arbitrary Angles

What if we wish to rotate by an arbitrary angle $\theta$ rather than fixed $90^\circ$ increments? Master mathematician Leonhard Euler provided the crown jewel theorem of mathematics—Euler's Formula:

Consider a complex number of unit magnitude $e^{i\theta}$:

• When $\theta = 0$: $e^{i0} = \cos 0 + i\sin 0 = 1$ (pointing along the positive real axis);
• When $\theta = \frac{\pi}{2}$ ($90^\circ$): $e^{i\pi/2} = \cos 90^\circ + i\sin 90^\circ = i$ (matching our earlier rotation!);
• When $\theta = \pi$ ($180^\circ$): $e^{i\pi} = \cos 180^\circ + i\sin 180^\circ = -1$, rearranging to Euler's famous identity: $e^{i\pi} + 1 = 0$!

Thus, $e^{i\theta}$ is a pure rotation operator: multiplying any complex number by it preserves exact magnitude while rotating counterclockwise by $\theta$ radians!

2.3 Pen-and-Paper Proof: Why Multiplying Complex Numbers Adds Angles

Using standard high-school trigonometric addition formulas, let us derive Euler multiplication by hand:

• $\cos(\alpha + \beta) = \cos\alpha\cos\beta - \sin\alpha\sin\beta$
• $\sin(\alpha + \beta) = \sin\alpha\cos\beta + \cos\alpha\sin\beta$

Let two rotation operators be $e^{i\alpha}$ and $e^{i\beta}$. Multiplying them as polynomials:

Since $i^2 = -1$, separating real and imaginary components:

Behold the beauty! Under complex multiplication, intricate geometric rotations simplify into elementary addition in the exponent!

2.4 The Climax: How RoPE Naturally Yields Relative Distance $(m - n)$

Now let us derive the mathematical centerpiece of RoPE!

Suppose a sentence contains two tokens:

• Token 1 at position $m$ ($m=2$), with Query vector $q = (x_1, x_2)$, represented as complex $q = x_1 + i x_2$;
• Token 2 at position $n$ ($n=5$), with Key vector $k = (y_1, y_2)$, represented as complex $k = y_1 + i y_2$.

Step 1: Apply Positional Rotation

Rotate each vector according to its positional index:

• Rotate $q$ by $m\theta$: $q_m = q \cdot e^{im\theta}$
• Rotate $k$ by $n\theta$: $k_n = k \cdot e^{in\theta}$

Step 2: Compute Attention Inner Product

In vector geometry, the dot product equals the real part of the complex conjugate product:

(Note: $k^*$ denotes complex conjugate, where $(a + bi)^* = a - bi$, and in polar form $(e^{in\theta})^* = e^{-in\theta}$)

Substitute rotated vectors:

Group real magnitudes and complex exponentials:

Apply common-base exponent addition:

Extract the real part:

Derivation complete! Notice the term in the exponent: the absolute positional indices $m$ and $n$ naturally collapse into the relative distance difference $(m - n)$! This guarantees:

• If $m=2, n=5$, the relative offset is $-3$;
• If the entire sentence shifts right by 100 tokens ($m=102, n=105$), the offset is still $102 - 105 = -3$!;
• Their dot product attention score remains strictly identical! This is the rigorous mathematical proof behind RoPE's shift invariance and length extrapolation!

Part 3: Statistics & Variance — Why Attention Must Divide by $\sqrt{d_k}$

In the standard self-attention equation:

初学者往往对分母上的 $\sqrt{d_k}$ 感到莫名其妙：为什么要平白无故开个根号除一下？

3.1 Foundational High-School Statistics: Expectation and Variance

• In basic statistics, we measure random distributions with two metrics:
• Expectation (Mean) $E(X)$: The average center of mass of the distribution;
• Variance $\text{Var}(X)$: The degree of dispersion / spread around the mean (Standard deviation $\sigma = \sqrt{\text{Var}(X)}$).

Statistical Properties of Two Independent Random Variables:

If two random variables $X$ and $Y$ are independent with zero mean ($E[X]=E[Y]=0$):

• Variance of Product:

• Variance of Sum:

3.2 Calculating Variance: The Catastrophic Avalanche without Dividing by $\sqrt{d_k}$

In our 0.04B model, each attention head has dimension $d_k = 64$.

因为前面的数据都经过了归一化，我们合理假设每一个分量都是标准的均值为 0、方差为 1 的变量：

• Under LayerNorm/RMSNorm, each component $q_i$ and $k_i$ behaves as an independent zero-mean unit-variance variable:
• $E[k_i] = 0, \quad \text{Var}(k_i) = 1$

Let us derive the expectation and variance of the single-dimension product $q_i k_i$:

• 每一对单项乘积的方差：

• 64 项相加后的总方差：

• 总打分结果的标准差：

😱 The Catastrophe Occurs:

The variance of the attention score explodes to $64$, meaning its standard deviation is $\sigma = \sqrt{64} = 8$!

• $e^{20} \approx 485,165,195$ (nearly 500 million!)
• $e^{-15} \approx 0.0000003$
• Outcome: The single maximum token captures 99.9999% of attention probability, while all other tokens collapse to 0.0000%!

Even worse is the gradient: the derivative of Softmax is:

When $p_i \approx 1$, derivative is $1 \times (1 - 1) = \mathbf{0}$; when $p_i \approx 0$, derivative is $0 \times (1 - 0) = \mathbf{0}$! Conclusion: All backpropagated gradients vanish to zero! The network suffers derivative paralysis and stops learning!

3.3 The Antidote: Resetting Variance with Scaling

To prevent gradient death, mathematicians scale the score by $c = \frac{1}{\sqrt{d_k}}$, leveraging $\text{Var}(c \cdot X) = c^2 \cdot \text{Var}(X)$:

Standard deviation is brought back to $\sqrt{1.0} = 1.0$! Scores distribute smoothly between $-3$ and $+3$, Softmax yields healthy probabilities, derivatives stay in active responsive regimes, and parameters update smoothly!

Part 4: Calculus & Limits — RMSNorm Scale Invariance & Causal Masking

4.1 Scale Invariance Derivation of RMSNorm

In Chapter 02 we met RMSNorm:

Why does RMSNorm not require absolute amplitude tracking?

Suppose an preceding layer experiences an unexpected gain spike, amplifying signals by $100\times$ (turning $x$ into $100x$):

• Numerator becomes: $100x$;
• Denominator RMS calculation:

• Divide numerator by denominator:

The arbitrary constant 100 cancels out completely! This is mathematical scale invariance: regardless of input scale drift, RMSNorm outputs remain rock-solid.

4.2 Limit Analysis: Why Causal Masks Drive Future Probabilities to Zero

During self-attention, token 2 must never peek at token 3. In code, future positions are masked with negative infinity ($-\infty$). According to exponential properties ($y = e^x$), as $x \to -\infty$ exponentially:

Substituting into Softmax: the future numerator term $e^{-\infty}$ strictly converges to 0. Regardless of the denominator, future tokens receive exactly $0.00\%$ attention weight—a mathematically airtight guarantee against future leakage.

Part 5: Information Theory & Probability Chains — The Origin of Autoregressive Cross-Entropy

How does training a model to predict single "next tokens" produce fluent, cohesive long-form prose and complex reasoning?

5.1 The High-School Conditional Probability Chain Rule

From high-school probability theory, the conditional probability formula states:

Extended to a sequence of $T$ tokens $W = (w_1, w_2, w_3, \dots, w_T)$, the joint probability of the entire text is the sequential chain product of conditional step probabilities:

5.2 The Magic of Logarithms: Converting Multiplication into Addition

We want the overall probability $P(W)$ to be as large as possible (Maximum Likelihood Estimation, MLE). However, multiplying thousands of tiny probabilities ($0 \sim 1$) collapses numbers to $10^{-500}$, underflowing computer floating-point registers to absolute zero!

The monotonically increasing nature of the natural logarithm saves us:

Taking natural logarithms converts multiplication into clean addition! Adding a negative sign transforms maximizing probability into standard deep learning Loss Minimization:

This is the mathematical core of autoregressive training: shifting the target sequence by 1 position and calculating Negative Log-Likelihood (NLL) at each step is mathematically identical to Cross-Entropy Loss!

Part 6: High-School Math to LLM Core Formula Reference Table

Let us synthesize all derivations into a structured reference matrix connecting high-school mathematics directly to modern Transformer operators:

| High-School Math Concept | Classic Mathematical Formula | Industrial LLM Implementation | Core Engineering & Physical Problem Solved | | Plane Vector Dot Product | $\vec{a} \cdot \vec{b} = |\vec{a}||\vec{b}|\cos\theta$ | Attention Score Matrix $Q \cdot K^T$ | Measures semantic association between tokens in high-dimensional embedding spaces. | | Complex Numbers & Euler Identity | $e^{i\theta} = \cos\theta + i\sin\theta$ | RoPE Rotary Embeddings $\text{Re}(q k^* e^{i(m-n)\theta})$ | Derives relative distance offsets $(m-n)$ through pure geometric rotation, enabling robust sequence extrapolation. | | Independent Variable Variance Addition | $\text{Var}(X+Y) = \text{Var}(X) + \text{Var}(Y)$ | Attention Scaling Factor $\frac{1}{\sqrt{d_k}}$ | Neutralizes 64-dimensional variance explosion, preventing Softmax derivative saturation and vanishing gradients. | | Gaussian Exponential Limits | $\lim_{x \to -\infty} e^x = 0$ | Causal Mask Assigned to Negative Infinity | Airtight shielding of future tokens, preventing information leakage during autoregressive training. | | Root Mean Square (RMS) | $\text{RMS} = \sqrt{\frac{1}{d}\sum x_i^2}$ | Pre-RMSNorm Normalization Layers | Delivers scale invariance while discarding redundant mean computation, stabilizing gradient highways across 12 layers. | | Conditional Probability Chain Rule | $P(A, B) = P(A)P(B \mid A)$ | Autoregressive Causal Cross-Entropy Loss | Rigorously decomposes whole-document generation into single next-token prediction steps. |

💡 Next Step Guidance:

Armed with these mathematical foundations, return with confidence to Chapter 02 | Implementing Modern Transformer Architecture from Scratch and see how these laws guide billions of transistors in code!