Hugging Face 详解 RoPE 位置编码演进

原标题:You could have designed state of the art positional encoding
68AI 研判分

Hugging Face 发布技术博客,通过迭代推导过程解析旋转位置编码(RoPE)的设计原理。文章从基础线性代数出发,逐步展示如何从简单系统演化出当前主流 Transformer 模型采用的 SOTA 位置编码方案。该教程旨在降低理解门槛,帮助开发者掌握 Llama 3.2 等现代模型的核心架构机制。

报道正文

本文目录3 个章节

返回文章列表

你本可以设计出最先进的位置编码

发布于 2024 年 11 月 25 日 在 GitHub 上更新 点赞 503

Christopher Fleetwood FL33TW00D-HF

本文也有中文版本:简体中文。

加尔定律 (Gall's Law) 一个能正常工作的复杂系统, invariably(总是)是从一个能正常工作的简单系统演化而来的。 John Gall

本文将带你一步步探索 Transformer 模型中最先进位置编码的发现过程。我们将通过迭代改进位置信息的编码方式,最终推导出用于最新 Llama 3.2 发布版及大多数现代 Transformer 的 Rotary Postional Encoding (RoPE,旋转位置编码)。本文旨在尽量减少所需的数学知识门槛,但读者仍需具备基础的线性代数、三角函数以及自注意力机制的理解。

问题陈述

观其伴,知其性 John Rupert Firth

与所有问题一样,最好先明确我们究竟想要实现什么。Transformer 中的自注意力机制用于理解序列中 token 之间的关系。自注意力是一种集合操作,这意味着它是置换等变的。如果我们不向自注意力注入位置信息,许多重要的关系将无法被确定。

以下示例最能说明这一点。

动机示例

考虑这句话,其中同一个词出现在不同位置:

The dog chased another dog \text{The dog chased another dog} The dog chased another dog

直观来看,“dog”指代两个不同的实体。让我们看看如果先对其进行分词,映射到 Llama 3.2 1B 的真实 token 嵌入,并通过 torch.nn.MultiheadAttention 处理时会发生什么。

PYTHON
import torch
import torch.nn as nn
from transformers import AutoTokenizer, AutoModel

model_id = "meta-llama/Llama-3.2-1B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id)

text = "The dog chased another dog" tokens = tok(text, return_tensors="pt")["input_ids"] embeddings = model.embed_tokens(tokens) hdim = embeddings.shape[-1]

W_q = nn.Linear(hdim, hdim, bias=False) W_k = nn.Linear(hdim, hdim, bias=False) W_v = nn.Linear(hdim, hdim, bias=False) mha = nn.MultiheadAttention(embed_dim=hdim, num_heads=4, batch_first=True)

with torch.no_grad(): for param in mha.parameters(): nn.init.normal_(param, std=0.1)

output, _ = mha(W_q(embeddings), W_k(embeddings), W_v(embeddings))

dog1_out = output[0, 2] dog2_out = output[0, 5] print(f"Dog output identical?: {torch.allclose(dog1_out, dog2_out, atol=1e-6)}")

文本

正如我们所见,在没有任何位置信息的情况下,(多头)自注意力操作对于不同位置的相同 token 会产生**完全相同的输出**,尽管这些 token 显然代表不同的实体。让我们开始设计一种方法,通过引入位置信息来增强自注意力机制,使其能够根据单词的位置编码来确定它们之间的关系。

理想的 positional encoding(位置编码)方案应该具备怎样的特性?

## 理想特性

让我们尝试定义一些理想特性,以使优化过程尽可能简单。

#### 特性 1 - 每个位置具有唯一编码(跨序列一致)

每个位置都需要一个唯一的编码,且该编码不受序列长度影响——无论当前序列长度为 10 还是 10,000,位于第 5 个位置的 token 都应具有相同的编码。

#### 特性 2 - 两个编码位置之间存在线性关系

位置之间的关系应在数学上保持简单。如果我们知道位置 $p$ 的编码,那么计算位置 $p+k$ 的编码应当非常直接,这有助于模型更容易地学习位置模式。

回想一下我们在数轴上如何表示数字:很容易理解 5 距离 3 有 2 步,或者 10 距离 15 有 5 步。我们的编码中也应存在这种直观的对应关系。

#### 特性 3 - 能泛化到比训练时遇到的更长的序列

为了提升模型在现实世界中的实用性,它们应当能够泛化到训练分布之外的数据。因此,我们的编码方案需要足够灵活,以处理意外的输入长度,同时不违反其他任何理想特性。

#### 特性 4 - 由模型可学习的确定性过程生成

It would be ideal if our positional encodings could be drawn from a deterministic process. This should allow the model to learn the mechanism behind our encoding scheme efficiently.

#### Property 5 - Extensible to multiple dimensions

With multimodal models becoming the norm, it is crucial that our positional encoding scheme can naturally extend from 1D1D1D to nDnDnD. This will allow models to consume data like images or brain scans, which are 2D2D2D and 4D4D4D respectively.

Now we know the ideal properties (henceforth referred to as PrnPr_nPrn​), let's start designing and iterating on our encoding scheme.

## Integer Position Encoding

The first approach that may jump to mind is simply to add the integer value of the token position to each component of the token embedding, with values ranging from 0→L0 \rightarrow L0→L where LLL is the length of our current sequence.

It's clear that our current naïve approach is going to cause problems. The magnitude of the position value vastly exceeds the actual values of our input. This means the signal-to-noise ratio is very low, and it's hard for the model to separate the semantic information from the positional information.

With this new knowledge, a natural follow on might be to normalize the position value by 1N\frac{1}{N}N1​. This constrains the values between 0 and 1, but introduces another problem. If we choose NNN to be the length of the current sequence, then the position values will be completely different for each sequence of differing lengths, violating Pr1Pr_1Pr1​.

Is there a better way to ensure our numbers are between 0 and 1? If we thought really hard about this for a while, we might come up with switching from decimal to binary numbers.

## Binary Position Encoding

Instead of adding our (potentially normalized) integer position to each component of the embedding, we could instead convert it into its binary representation and *s t r e t c h* our value out to match our embedding dimension, as demonstrated below.

We've converted the position of interest (252) into its binary representation (11111100) and added each bit to the corresponding component of the token embedding. The least significant bit (LSB) will cycle between 0 and 1 for every subsequent token, whilst the most significant bit (MSB) will cycle every 2n−12^{n-1}2n−1 tokens where nnn is the number of bits. You can see the positional encoding vector for different indices in the animation below [1][^1][1].

We've solved the value range problem, and we now have unique encodings that are consistent across different sequence lengths. What happens if we plot a low dimensional version of our token embedding and visualize the addition of our binary positional vector for different values.

We can see that the result is very "jumpy" (as we might expect from the discrete nature of binary). The optimization process likes smooth, continuous and predictable changes. Do we know any functions with similar value ranges that are smooth and continuous?

If we looked around a little, we might notice that both sin⁡\sinsin and cos⁡\coscos fit the bill!

## Sinusoidal positional encoding

The above animation visualizes our position embedding if each component is alternatively drawn from sin⁡\sinsin and cos⁡\coscos with gradually increasing wavelengths. If you compare it with the previous animation, you'll notice a striking similarity!

We've now arrived at Sinusoidal embeddings; originally defined in the [Attention is all you need](https://arxiv.org/abs/1706.03762) paper. Let's look at the equations:

PE(pos,2i)=sin⁡(pos100002i/d)PE(pos,2i+1)=cos⁡(pos100002i/d) PE_{(pos,2i)} = \color{#58C4DD}\sin\left(\color{black}\frac{pos}{10000^{2i/d}}\color{#58C4DD}\right)\color{black} \\ \quad \\ PE_{(pos,2i+1)} = \color{#FC6255}\cos\left(\color{black}\frac{pos}{10000^{2i/d}}\color{#FC6255}\right)\color{black} \\ PE(pos,2i)​=sin(100002i/dpos​)PE(pos,2i+1)​=cos(100002i/dpos​)

where pospospos is the tokens position index, iii is the component index in the positional encoding vector, and ddd is the model dimension. 10,00010,00010,000 is the **base wavelength** (henceforth referred to as θ\thetaθ), which we stretch or compress as a function of the component index. I encourage you to plug in some realistic values to get a feel for this geometric progression.

There's a few parts of this equation that are confusing at first glance. How did the authors choose 10,00010,00010,000? Why are we using sin⁡\sinsin **and** cos⁡\coscos for even and odd positions respectively?



> *[注:本文篇幅超长,以上为核心前篇精译,后续内容保留原文呈现]*


It seems that using 10,00010,00010,000 for the base wavelength was determined experimentally [2][^2][2]. Deciphering the usage of both sin⁡\sinsin and cos⁡\coscos is more involved, but crucial for our iterative approach to understanding. The key here is our desire for a linear relation between two encoded positions Pr2Pr_2Pr2​. To understand how using sin⁡\sinsin and cos⁡\coscos in tandem produce this linear relation, we will have to dive into some trigonometry.

Consider a sequence of sine and cosine pairs, each associated with a frequency ωi\omega_iωi​. Our goal is to find a linear transformation matrix M\mathbf{M}M that can shift these sinusoidal functions by a fixed offset kkk:

M⋅[sin⁡(ωip)cos⁡(ωip)]=[sin⁡(ωi(p+k))cos⁡(ωi(p+k))] \mathbf{M} \cdot \begin{bmatrix} \sin(\omega_i p) \\ \cos(\omega_i p) \end{bmatrix} = \begin{bmatrix} \sin(\omega_i(p + k)) \\ \cos(\omega_i(p + k)) \end{bmatrix} M⋅[sin(ωi​p)cos(ωi​p)​]=[sin(ωi​(p+k))cos(ωi​(p+k))​]

The frequencies ωi\omega_iωi​ follow a geometric progression that decreases with dimension index iii, defined as:

ωi=1100002i/d \omega_i = \frac{1}{10000^{2i/d}} ωi​=100002i/d1​

To find this transformation matrix, we can express it as a general 2×2 matrix with unknown coefficients u1u_1u1​, v1v_1v1​, u2u_2u2​, and v2v_2v2​:

[u1v1u2v2]⋅[sin⁡(ωip)cos⁡(ωip)]=[sin⁡(ωi(p+k))cos⁡(ωi(p+k))] \begin{bmatrix} u_1 & v_1 \\ u_2 & v_2 \end{bmatrix} \cdot \begin{bmatrix} \sin(\omega_i p) \\ \cos(\omega_i p) \end{bmatrix} = \begin{bmatrix} \sin(\omega_i(p+k)) \\ \cos(\omega_i(p+k)) \end{bmatrix} [u1​u2​​v1​v2​​]⋅[sin(ωi​p)cos(ωi​p)​]=[sin(ωi​(p+k))cos(ωi​(p+k))​]

By applying the trigonometric addition theorem to the right-hand side, we can expand this into:

[u1v1u2v2]⋅[sin⁡(ωip)cos⁡(ωip)]=[sin⁡(ωip)cos⁡(ωik)+cos⁡(ωip)sin⁡(ωik)cos⁡(ωip)cos⁡(ωik)−sin⁡(ωip)sin⁡(ωik)] \begin{bmatrix} u_1 & v_1 \\ u_2 & v_2 \end{bmatrix} \cdot \begin{bmatrix} \sin(\omega_i p) \\ \cos(\omega_i p) \end{bmatrix} = \begin{bmatrix} \sin(\omega_i p)\cos(\omega_i k) + \cos(\omega_i p)\sin(\omega_i k) \\ \cos(\omega_i p)\cos(\omega_i k) - \sin(\omega_i p)\sin(\omega_i k) \end{bmatrix} [u1​u2​​v1​v2​​]⋅[sin(ωi​p)cos(ωi​p)​]=[sin(ωi​p)cos(ωi​k)+cos(ωi​p)sin(ωi​k)cos(ωi​p)cos(ωi​k)−sin(ωi​p)sin(ωi​k)​]

This expansion gives us a system of two equations by matching coefficients:

u1sin⁡(ωip)+v1cos⁡(ωip)=cos⁡(ωik)sin⁡(ωip)+sin⁡(ωik)cos⁡(ωip)u2sin⁡(ωip)+v2cos⁡(ωip)=−sin⁡(ωik)sin⁡(ωip)+cos⁡(ωik)cos⁡(ωip) \begin{align} u_1\sin(\omega_i p) + v_1\cos(\omega_i p) &= \cos(\omega_i k)\sin(\omega_i p) + \sin(\omega_i k)\cos(\omega_i p) \\ u_2\sin(\omega_i p) + v_2\cos(\omega_i p) &= -\sin(\omega_i k)\sin(\omega_i p) + \cos(\omega_i k)\cos(\omega_i p) \end{align} u1​sin(ωi​p)+v1​cos(ωi​p)u2​sin(ωi​p)+v2​cos(ωi​p)​=cos(ωi​k)sin(ωi​p)+sin(ωi​k)cos(ωi​p)=−sin(ωi​k)sin(ωi​p)+cos(ωi​k)cos(ωi​p)​

By comparing terms with sin⁡(ωip)\sin(\omega_i p)sin(ωi​p) and cos⁡(ωip)\cos(\omega_i p)cos(ωi​p) on both sides, we can solve for the unknown coefficients:

u1=cos⁡(ωik)v1=sin⁡(ωik)u2=−sin⁡(ωik)v2=cos⁡(ωik) \begin{align} u_1 &= \cos(\omega_i k) & v_1 &= \sin(\omega_i k) \\ u_2 &= -\sin(\omega_i k) & v_2 &= \cos(\omega_i k) \end{align} u1​u2​​=cos(ωi​k)=−sin(ωi​k)​v1​v2​​=sin(ωi​k)=cos(ωi​k)​

These solutions give us our final transformation matrix Mk\mathbf{M_k}Mk​:

Mk=[cos⁡(ωik)sin⁡(ωik)−sin⁡(ωik)cos⁡(ωik)] \mathbf{M_k} = \begin{bmatrix} \cos(\omega_i k) & \sin(\omega_i k) \\ -\sin(\omega_i k) & \cos(\omega_i k) \end{bmatrix} Mk​=[cos(ωi​k)−sin(ωi​k)​sin(ωi​k)cos(ωi​k)​]

If you've done any game programming before, you might notice that the result of our derivation is oddly familiar. That's right, it's the [Rotation Matrix!](https://en.wikipedia.org/wiki/Rotation_matrix) [3][^3][3].

So the encoding scheme designed by [Noam Shazeer](https://en.wikipedia.org/wiki/Noam_Shazeer) in [Attention is all you need](https://arxiv.org/abs/1706.03762) was already encoding relative position as a rotation back in 2017! It took another **4 years** to go from Sinusoidal Encoding to RoPE, despite rotations already being on the table...

## Absolute vs Relative Position Encoding

With the knowledge in hand that rotations are important here, let's return to our motivating example and try to discover some intuitions for our next iteration.

01234The dog chased another dog-2-1012The dog chased another dog \begin{align*} &\hspace{0.7em}0 \hspace{1.4em} 1 \hspace{2em} 2 \hspace{2.6em} 3 \hspace{2.4em} 4\\ &\text{The dog chased another dog} \\ \\ &\hspace{0.3em}\text{-2} \hspace{1.4em} \text{-1} \hspace{1.7em} 0 \hspace{2.6em} 1 \hspace{2.4em} 2\\ &\text{The dog \color{#699C52}chased \color{black}another dog} \end{align*} ​01234The dog chased another dog-2-1012The dog chased another dog​

Above we can see the absolute positions of our tokens, and the relative positions from chased\color{#699C52}\text{chased}chased to every other token. With Sinusoidal Encoding, we generated a separate vector which represents the absolute position, and using some trigonometric trickery we were able to encode relative positions.

When we're trying to understand these sentences, does it matter that *this* word is the 2157th word in this blog post? Or do we care about its relationship to the words around it? The absolute position of a word rarely matters for meaning - what matters is how words relate to each other.

## Positional encoding in context

From this point on, it's key to consider positional encoding **in the context of** self attention. To reiterate, the self-attention mechanism enables the model to weigh the importance of different elements in an input sequence and dynamically adjust their influence on the output.

Attn(Q,K,V)=softmax(QKTdk)V \text{Attn}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V Attn(Q,K,V)=softmax(dk​​QKT​)V

In all our previous iterations, we've generated a separate positional encoding vector and **added** it to our token embedding prior to our QQQ, KKK and VVV projections. By adding the positional information directly to our token embedding, we are **polluting** the semantic information with the positional information. We should be attempting to encode the information without modifying the norm. Shifting to multiplicative is the key.

Using the dictionary analogy, when looking up a word (query) in our dictionary (keys), nearby words should have more influence than distant ones. The influence of one token upon another is determined by the QKTQK^TQKT dot product - so that's exactly where we should focus our positional encoding!

a⃗⋅b⃗=∣a⃗∣∣b⃗∣cos⁡θ \vec{a} \cdot \vec{b} = |\vec{a}| |\vec{b}| \cos \theta a⋅b=∣a∣∣b∣cosθ

The geometric interpretation of the dot product shown above gives us a magnificent insight. We can modulate the result of our dot product of two vectors purely by increasing or decreasing the angle between them. Furthermore, by rotating the vector, we have absolutely zero impact on the norm of the vector, which encodes the semantic information of our token.

So now we know where to focus our *attention*, and have seen from another *angle* why a rotation might be a sensible "channel" in which to encode our positional information, let's put it all together!

## **Ro**tary **P**ostional **E**ncoding

**Ro**tary **P**ostional **E**ncoding or RoPE was defined in the [RoFormer paper](https://arxiv.org/pdf/2104.09864) ([Jianlin Su](https://x.com/bojone1993) designed it independently on his blog [here](https://kexue.fm/archives/8130) and [here](https://kexue.fm/archives/8265)). While it may seem like voodoo if you skip to the end result, by thinking about Sinusoidal Encoding in the context of self attention (and more specifically dot products), we can see how it all comes together.

Much like in Sinusoidal Encoding, we decompose our vectors q\mathbf{q}q or k\mathbf{k}k, instead of pre-projection x\mathbf{x}x) into 2D pairs/chunks. Rather than encoding *absolute* position directly by adding a vector we drew from sinusoidal functions of slowly decreasing frequencies, we cut to the chase and encode *relative* position by **multiplying each pair with the rotation matrix**.

Let q\mathbf{q}q or k\mathbf{k}k be our input vector at position ppp. We create a block diagonal matrix where Mi\mathbf{M_i}Mi​ is the corresponding rotation matrix for that component pairs desired rotation:

R(q,p)=(M1M2⋱Md/2)(q1q2⋮qd) R(\mathbf{q}, p) = \begin{pmatrix} \mathbf{M_1} & & & \\ & \mathbf{M_2} & & \\ & & \ddots & \\ & & & \mathbf{M_{d/2}} \end{pmatrix} \begin{pmatrix} q_1 \\ q_2 \\ \vdots \\ q_d \end{pmatrix} R(q,p)=​M1​​M2​​⋱​Md/2​​​​q1​q2​⋮qd​​​

Much the same as Sinusoidal Encoding, Mi\mathbf{M_i}Mi​ is simply:

Mi=[cos⁡(ωip)sin⁡(ωip)−sin⁡(ωip)cos⁡(ωip)] \mathbf{M_i} = \begin{bmatrix} \cos(\omega_i p) & \sin(\omega_i p) \\ -\sin(\omega_i p) & \cos(\omega_i p) \end{bmatrix} Mi​=[cos(ωi​p)−sin(ωi​p)​sin(ωi​p)cos(ωi​p)​]

In practice, we don't use a matrix multiplication to compute RoPE as it would be computationally inefficient with such a sparse matrix. Instead, we can directly apply the rotations to pairs of elements independently, taking advantage of the regular pattern in the computation:

RΘ,pdq=(q1q2q3q4⋮qd−1qd)⊗(cos⁡pθ1cos⁡pθ1cos⁡pθ2cos⁡pθ2⋮cos⁡pθd/2cos⁡pθd/2)+(−q2q1−q4q3⋮−qdqd−1)⊗(sin⁡pθ1sin⁡pθ1sin⁡pθ2sin⁡pθ2⋮sin⁡pθd/2sin⁡pθd/2) R_{\Theta,p}^d q = \begin{pmatrix} q_1 \\ q_2 \\ q_3 \\ q_4 \\ \vdots \\ q_{d-1} \\ q_d \end{pmatrix} \otimes \begin{pmatrix} \cos p\theta_1 \\ \cos p\theta_1 \\ \cos p\theta_2 \\ \cos p\theta_2 \\ \vdots \\ \cos p\theta_{d/2} \\ \cos p\theta_{d/2} \end{pmatrix} + \begin{pmatrix} -q_2 \\ q_1 \\ -q_4 \\ q_3 \\ \vdots \\ -q_d \\ q_{d-1} \end{pmatrix} \otimes \begin{pmatrix} \sin p\theta_1 \\ \sin p\theta_1 \\ \sin p\theta_2 \\ \sin p\theta_2 \\ \vdots \\ \sin p\theta_{d/2} \\ \sin p\theta_{d/2} \end{pmatrix} RΘ,pd​q=​q1​q2​q3​q4​⋮qd−1​qd​​​⊗​cospθ1​cospθ1​cospθ2​cospθ2​⋮cospθd/2​cospθd/2​​​+​−q2​q1​−q4​q3​⋮−qd​qd−1​​​⊗​sinpθ1​sinpθ1​sinpθ2​sinpθ2​⋮sinpθd/2​sinpθd/2​​​

That's all there is to it! By artfully applying our rotations to 2D chunks of q\mathbf{q}q and k\mathbf{k}k prior to their dot product, and switching from additive to multiplicative, we can gain a big performance boost in evaluations [4][^4][4].

## Extending RoPE to nnn-Dimensions

We've explored the 1D1D1D case for RoPE and by this point I hope you've gained an intuitive understanding of an admittedly unintuitive component of transformers. Finally, let's explore extending it to higher dimensions, such as images.

A natural first intuition could be to directly use the [xy] \begin{bmatrix} x \\ y \end{bmatrix}[xy​] coordinate pairs from the image. This might seem intuitive, after all, we were almost arbitrarily pairing up our components previously. However, this would be a mistake!

In the 1D1D1D case, we encode the relative position m−nm - nm−n through a rotation of pairs of values from our input vector. For 2D2D2D data, we need to encode both horizontal and vertical relative positions, say m−nm - nm−n and i−ji - ji−j independently. RoPE's brilliance lies in how it handles multiple dimensions. Instead of trying to encode all positional information in a single rotation, we pair components **within the same dimension** and rotate those, otherwise we would be intermixing the xxx and yyy offset information. By handling each dimension independently, we maintain the natural structure of the space. This can generalize to as many dimensions as required!

## The future of positional encoding

Is RoPE the final incarnation of positional encoding? This [recent paper](https://arxiv.org/pdf/2410.06205) from DeepMind deeply analyses RoPE and highlights some fundamental problems. TLDR: RoPE isn't a perfect solution, and the models mostly focus on the lower frequencies, but the paper shows that **removing** (not rotating) the lowest frequencies improves performance on Gemma 2B!

I anticipate some future breakthroughs, perhaps taking inspiration from signal processing with ideas like wavelets or hierarchical implementations. As models are increasingly quantized for deployment, I'd also expect to see some innovation in encoding schemes that remain robust under low-precision arithmetic.

## Conclusion

Positional encoding has and continues to be treated as an after thought in transformers. I believe we should view it differently - self attention has an Achilles heel that has been repeatedly patched.

I hope this blog post showed you that you too could have discovered state of the art positional encoding, despite it being unintuitive at first. In a follow up post I'd love to explore practical implementation details for RoPE in order to maximise performance.

This post was originally published [here](https://fleetwood.dev/posts/you-could-have-designed-SOTA-positional-encoding).

## References

- [Transformer Architecture: The Positional Encoding](https://kazemnejad.com/blog/transformer_architecture_positional_encoding/)
- [Rotary Embeddings: A Relative Revolution](https://blog.eleuther.ai/rotary-embeddings/)
- [How positional encoding works in transformers?](https://www.youtube.com/watch?v=T3OT8kqoqjc)
- [Attention Is All You Need](https://arxiv.org/pdf/1706.03762)
- [Round and round we go! What makes Rotary Positional Encodings useful?](https://arxiv.org/pdf/2410.06205)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://arxiv.org/pdf/2104.09864)

[^1]: Binary and Sinusoidal animations are reproductions of animations contained in [this](https://www.youtube.com/watch?v=T3OT8kqoqjc0) video.

[^2]: Using θ=10000\theta = 10000θ=10000 gives us 2π⋅10000 2 \pi \cdot 100002π⋅10000 unique positions, or a theoretical upper bound on the context length at ~63,000.

[^3]: Pieces of this post are based on [this fantastic post](https://kazemnejad.com/blog/transformer_architecture_positional_encoding/) by [Amirhossein Kazemnejad](https://kazemnejad.com/).

[^4]: For empirical evidence, see [this](https://blog.eleuther.ai/rotary-embeddings/) great post by EleutherAI.