Most positional encoding schemes add a position-dependent vector to the input embedding before attention even runs. RoPE does something different: it ROTATES each query/key vector by an angle proportional to its position, applied independently to d/2 different 2D "planes" within the vector. The payoff is elegant -- the dot product between a rotated query and a rotated key ends up depending only on the DIFFERENCE between their positions, never their absolute positions. That's exactly the relative-position signal attention needs, produced as a side effect of the rotation itself rather than injected separately.

Implement rotate(x, position, base=10000.0): x is a vector of even length d. Treat it as d/2 pairs: (x[0],x[1]), (x[2],x[3]), .... Pair i gets rotated by angle = theta_i * position, where theta_i = base ** (-2*i / d). A standard 2D rotation by angle a maps:

(x0, x1) -> (x0*cos(a) - x1*sin(a),  x0*sin(a) + x1*cos(a))

Example

rotate([1.0, 2.0, 3.0, 4.0], position=0)  # -> [1.0, 2.0, 3.0, 4.0] (no rotation)

Constraints

• The property that matters: for any query q, key k, and positions p_q, p_k, dot(rotate(q, p_q), rotate(k, p_k)) depends only on p_q - p_k -- shifting both positions by the same amount changes nothing.
• Each pair rotates independently, using its OWN theta_i -- lower-index pairs (smaller i) get a larger theta_i and so rotate faster per unit of position; higher-index pairs rotate much more slowly.

Real-world grounding

• llama.cpp's RoPE implementation: search ggml_rope in ggml-org/llama.cpp
• The original RoPE paper (Su et al., "RoFormer") is what essentially every modern open-weight LLM (Llama, Mistral, Qwen, and others) uses for position encoding today, in place of the sinusoidal or learned absolute encodings earlier transformers used.