stackformers 4.2.0__tar.gz → 4.2.1__tar.gz

{stackformers-4.2.0 → stackformers-4.2.1}/CHANGELOG.md

@@ -6,6 +6,40 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
 adheres to [Semantic Versioning](https://semver.org/): MAJOR for breaking public API changes,
 MINOR for backwards-compatible features, PATCH for bug fixes and internal changes.
 
+## [4.2.1] — 2026-07-14
+
+### Removed
+
+- **`RoPENDConfig.headroom`.** It was redundant with `r_max`: the ladder only ever depended on
+  the product `headroom · r_max`, so the two knobs shared one degree of freedom, and anyone
+  wanting more reach at the slow end could get it by raising `r_max` — the parameter that has a
+  measurement procedure attached. It was also the last *tuned* number in a module whose premise
+  is that the band range is measured, not tuned.
+
+  Its default has a derivation, so it need not be a parameter at all. Attention sees *signed*
+  offsets spanning `[−r_max, +r_max]`, a width of `2·r_max`; asking that the slowest band turn
+  through at most half a circle across that width gives `ω_lo = π / (2·r_max)` directly. That is
+  bit-identical to the old `headroom=4.0` default at every `r_max`, so callers on the default —
+  which is to say every caller, since the field shipped in 4.2.0 — see no change in behaviour.
+
+  Both ends of the ladder are now the same rule, `ω = π / scale`: a half turn over `r_min` at
+  the fast end (Nyquist), and a half turn over `2·r_max` at the slow end (no wrap).
+
+  Treated as a PATCH rather than a MAJOR bump because `headroom` existed for exactly one
+  release, is not known to be set anywhere, and its removal cannot change the behaviour of a
+  caller that did not set it. Note that pydantic's default `extra="ignore"` means a leftover
+  `RoPENDConfig(headroom=...)` is *dropped silently* rather than raising — a caller who had set
+  it to something other than 4.0 will now get different frequencies without being told.
+
+### Fixed
+
+- **`RoPENDConfig` now requires `dim_head >= 4 * coords`** (at least two bands per axis).
+  `dim_head == 2 * coords` divides cleanly and passed validation, but leaves a one-band ladder
+  with nowhere to descend: `torch.linspace(hi, lo, 1)` returns `[hi]`, so the lone band lands on
+  the fast end, `r_max` is discarded entirely, and the encoding becomes periodic with period
+  `2 · r_min` across the whole domain — offsets of `0`, `2·r_min`, `4·r_min` … all produce an
+  identical attention logit. It failed silently; it is now rejected at construction.
+
 ## [4.2.0] — 2026-07-14
 
 ### Added

{stackformers-4.2.0 → stackformers-4.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: stackformers
-Version: 4.2.0
+Version: 4.2.1
 Summary: Typed, composable, SOLID transformer library for PyTorch
 Project-URL: Homepage, https://github.com/Red-Eyed/stackformers
 Project-URL: Repository, https://github.com/Red-Eyed/stackformers

{stackformers-4.2.0 → stackformers-4.2.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "stackformers"
-version = "4.2.0"
+version = "4.2.1"
 description = "Typed, composable, SOLID transformer library for PyTorch"
 readme = "README.md"
 license = { file = "LICENSE" }

{stackformers-4.2.0 → stackformers-4.2.1}/stackformers/positional/README.md

@@ -8,12 +8,14 @@ Positional encodings applied to Q and K tensors inside attention — not to the
 
 RoPE's shortest wavelength is fixed at 2π coordinate units, and `base` only stretches the long end of the band range. So `base` lands the ladder correctly only when tokens sit *one unit apart* — true for text, true for a patch grid, meaningless for scattered continuous coordinates. Positions normalised to `[0, 1]` leave every band longer than the whole domain, so the encoding barely rotates and carries almost no signal; and `base=10000` on a 14×14 patch grid leaves 14 of 16 bands frozen.
 
-`RotaryEmbeddingND` (`RoPENDConfig`) drops `base` entirely and builds the ladder from the two numbers that actually mean something, in any number of dimensions:
+`RotaryEmbeddingND` (`RoPENDConfig`) drops `base` entirely and builds the ladder from the two numbers that actually mean something, in any number of dimensions. Both ends are the same rule — **a half turn over the scale in question, `ω = π / scale`**:
 
-- `r_min` — the finest separation the model must resolve. Sets the shortest wavelength to `2·r_min`, the Nyquist limit. Measure it as a low percentile of the nearest-neighbour distance.
-- `r_max` — the domain diameter. Sets the longest wavelength (times `headroom`, so the slowest band stays monotone rather than wrapping). Measure it as a high percentile of the pairwise distance distribution.
+- `r_min` — the finest separation the model must resolve. Turned through π on the fastest band, so the shortest wavelength is `2·r_min`: the Nyquist limit, past which distinct offsets alias onto the same rotation. Measure it as a low percentile of the nearest-neighbour distance.
+- `r_max` — the domain diameter. Attention sees *signed* offsets, spanning `[−r_max, +r_max]` — a width of `2·r_max`, not `r_max` — so the slowest band is given a half turn over that full width. It cannot wrap, which leaves it monotone in the offset: a coarse ramp the model reads as displacement. Measure it as a high percentile of the pairwise distance distribution.
 
-The resulting ladder depends only on the *dynamic range* `r_max / r_min`, so metres, pixels and millimetres all give the identical encoding — the property `base` never had. `RoPE1DConfig` and `RoPE2DConfig` keep `base` and are unchanged; use them for text and grids.
+There is no third knob, and no constant to tune: two measured distances fix the ladder. The resulting band range depends only on the *dynamic range* `r_max / r_min`, so metres, pixels and millimetres all give the identical encoding — the property `base` never had. `RoPE1DConfig` and `RoPE2DConfig` keep `base` and are unchanged; use them for text and grids.
+
+`dim_head` must be at least `4 · coords`, so that every axis gets at least two bands. At one band per axis the ladder has nowhere to descend: it collapses onto the fast end, `r_max` is never reached, and the encoding silently becomes periodic with period `2·r_min` across the whole domain. `RoPENDConfig` rejects that at construction.
 
 **Centre your coordinates.** Not for invariance — RoPE is translation-invariant exactly, since the query and key rotations cancel into `ω·(pᵢ − pⱼ)`. It is for float32: the angle is `ω·p`, and coordinates far from the origin push it into the hundreds of radians and spend the mantissa before the cosine is taken. A 1e5 offset costs three orders of magnitude of accuracy. This cannot be done inside the module — cross-attention must subtract the *same* constant from query and key positions, and the module sees them separately — so subtract the centroid at the input boundary.
 

{stackformers-4.2.0 → stackformers-4.2.1}/stackformers/positional/config.py

@@ -53,9 +53,10 @@ class RoPENDConfig(BaseModel):
     Takes no ``base``. RoPE's ``base`` ladder pins its fastest band at a wavelength of 2π
     *coordinate units* regardless of the base, so it only lands correctly when tokens sit one
     unit apart — true for text and patch grids, meaningless for continuous coordinates. The
-    band range is set by the data instead: r_min fixes the fast end, r_max the slow end. The
-    ladder then depends only on the ratio r_max / r_min, so the units the coordinates happen
-    to be expressed in stop mattering.
+    band range is set by the data instead: r_min fixes the fast end, r_max the slow end, and
+    each is a half turn over the scale it names (ω = π / scale). Nothing else is free. The
+    ladder's shape then depends only on the ratio r_max / r_min, so the units the coordinates
+    happen to be expressed in stop mattering.
     """
 
     kind: Literal["rope_nd"] = "rope_nd"
@@ -66,29 +67,22 @@ class RoPENDConfig(BaseModel):
     r_min: float = Field(
         gt=0.0,
         description=(
-            "Finest separation between two nodes that the model must tell apart. Sets the"
-            " shortest wavelength to 2*r_min — the Nyquist limit, below which distinct offsets"
-            " alias onto the same rotation. Measure it as a low percentile of the"
-            " nearest-neighbour distance, not the minimum, which is noise."
+            "Finest separation between two nodes that the model must tell apart. Gets a half"
+            " turn on the fastest band, so the shortest wavelength is 2*r_min — the Nyquist"
+            " limit, below which distinct offsets alias onto the same rotation. Measure it as a"
+            " low percentile of the nearest-neighbour distance, not the minimum, which is noise."
         ),
     )
     r_max: float = Field(
         gt=0.0,
         description=(
-            "Diameter of the domain — the largest offset that must stay distinguishable. Sets"
-            " the longest wavelength. Measure it as a high percentile of the pairwise distance"
+            "Diameter of the domain — the largest offset that must stay distinguishable."
+            " Attention sees signed offsets spanning [-r_max, +r_max], so the slowest band is"
+            " given a half turn over that full width of 2*r_max: it never wraps, which leaves it"
+            " monotone in the offset. Measure it as a high percentile of the pairwise distance"
             " distribution, not the maximum, which is an outlier."
         ),
     )
-    headroom: float = Field(
-        default=4.0,
-        gt=1.0,
-        description=(
-            "How far the longest wavelength reaches past r_max. Keeps the slowest band monotone"
-            " across the whole domain instead of wrapping, so it can act as a coarse absolute"
-            " coordinate. Llama's defaults sit at roughly 13x; 2-4x is ample here."
-        ),
-    )
 
     @model_validator(mode="after")
     def _check(self) -> RoPENDConfig:
@@ -99,6 +93,14 @@ class RoPENDConfig(BaseModel):
                 f" dim_head is split into {self.coords} per-axis blocks, and each block is"
                 " rotated in pairs of channels."
             )
+        if self.bands_per_axis < 2:
+            raise ValueError(
+                f"dim_head ({self.dim_head}) leaves {self.bands_per_axis} band per axis; at"
+                f" least 2 bands are needed, so dim_head must be at least {4 * self.coords}."
+                " A lone band lands on the fast end of the ladder and r_max is never reached,"
+                " leaving the encoding periodic with period 2 * r_min across the whole domain:"
+                " every offset an integer number of periods apart becomes indistinguishable."
+            )
         if self.r_max <= self.r_min:
             raise ValueError(f"r_max ({self.r_max}) must exceed r_min ({self.r_min}).")
         return self

{stackformers-4.2.0 → stackformers-4.2.1}/stackformers/positional/rope_nd.py

@@ -18,18 +18,28 @@ def _frequency_ladder(config: RoPENDConfig) -> Tensor:
     at ω=1, i.e. a wavelength of exactly 2π coordinate units, whatever the base — so ``base``
     can only stretch the slow end, and the ladder lands correctly only when tokens happen to
     sit one unit apart. That holds for text and for patch grids; it is meaningless for
-    continuous coordinates, where the useful band range is fixed by the data instead:
+    continuous coordinates, where the band range is fixed by the data instead.
 
-        ω_hi = π / r_min                  → shortest wavelength = 2·r_min  (Nyquist on the
-                                            finest separation the model must resolve)
-        ω_lo = 2π / (headroom · r_max)    → longest wavelength spans the whole domain
+    Both ends are a half turn over the scale they name — ω = π / scale:
 
-    So the ladder is a function of the *dynamic range* r_max / r_min alone, and is invariant
-    to the units the coordinates are expressed in — metres, pixels, or normalised to [0, 1]
-    all give the same encoding. That is precisely the property ``base`` does not have.
+        ω_hi = π / r_min       half-period r_min. The finest separation the model must resolve
+                               is turned through π, the furthest two points can be driven apart
+                               on a circle. Any faster is past Nyquist, and distinct offsets
+                               start aliasing onto the same rotation.
+
+        ω_lo = π / (2·r_max)   half-period 2·r_max. Attention sees *signed* offsets, spanning
+                               [−r_max, +r_max] — a width of 2·r_max, not r_max — so the slowest
+                               band turns through at most π across the entire domain. It cannot
+                               wrap, which leaves it monotone in the offset: a coarse ramp the
+                               model can read directly as displacement.
+
+    Nothing here is tuned, and there is no knob left to set: two measured distances fix the
+    ladder. Its *shape* depends only on the dynamic range r_max / r_min, so the encoding is
+    invariant to the units the coordinates are expressed in — metres, pixels, or normalised to
+    [0, 1] all give the same result. That is precisely the property ``base`` does not have.
     """
     omega_hi = math.pi / config.r_min
-    omega_lo = 2.0 * math.pi / (config.headroom * config.r_max)
+    omega_lo = math.pi / (2.0 * config.r_max)
     steps = torch.linspace(math.log(omega_hi), math.log(omega_lo), config.bands_per_axis)
     return torch.exp(steps)  # fastest band first, matching the 1-D/2-D convention
 

{stackformers-4.2.0 → stackformers-4.2.1}/tests/positional/test_rope_nd.py

@@ -11,7 +11,7 @@ from stackformers.positional.rope_nd import RotaryEmbeddingND
 from tests.conftest import atol
 
 B, N, H = 2, 16, 4
-R_MIN, R_MAX, HEADROOM = 0.5, 100.0, 4.0
+R_MIN, R_MAX = 0.5, 100.0
 
 DIMS = [(1, 64), (2, 64), (3, 96)]  # (coords, dim_head) — dim_head must divide by 2 * coords
 
@@ -19,9 +19,7 @@ DIMS = [(1, 64), (2, 64), (3, 96)]  # (coords, dim_head) — dim_head must divid
 @pytest.fixture(params=DIMS, ids=lambda p: f"c{p[0]}")
 def config(request: pytest.FixtureRequest) -> RoPENDConfig:
     coords, dim_head = request.param
-    return RoPENDConfig(
-        dim_head=dim_head, coords=coords, r_min=R_MIN, r_max=R_MAX, headroom=HEADROOM
-    )
+    return RoPENDConfig(dim_head=dim_head, coords=coords, r_min=R_MIN, r_max=R_MAX)
 
 
 @pytest.fixture
@@ -65,15 +63,32 @@ def test_output_shape(
 
 
 def test_ladder_spans_nyquist_to_domain(config: RoPENDConfig) -> None:
-    """The whole point of dropping `base`: the band range is pinned to the data, not a constant."""
+    """The whole point of dropping `base`: the band range is pinned to the data, not a constant.
+
+    Both ends are a half turn over the scale they name — r_min at the fast end, and the full
+    signed offset width 2*r_max at the slow end — so both are stated here as half-periods.
+    """
     inv_freq: torch.Tensor = RotaryEmbeddingND(config).inv_freq  # type: ignore[assignment]
-    shortest = 2 * math.pi / float(inv_freq.max())
-    longest = 2 * math.pi / float(inv_freq.min())
-    assert shortest == pytest.approx(2 * R_MIN, rel=1e-4)  # Nyquist on the finest separation
-    assert longest == pytest.approx(HEADROOM * R_MAX, rel=1e-4)  # spans the domain, with headroom
+    fastest_half_period = math.pi / float(inv_freq.max())
+    slowest_half_period = math.pi / float(inv_freq.min())
+    assert fastest_half_period == pytest.approx(R_MIN, rel=1e-4)  # Nyquist on the finest gap
+    assert slowest_half_period == pytest.approx(2 * R_MAX, rel=1e-4)  # the whole signed range
     assert inv_freq.shape == (config.bands_per_axis,)
 
 
+def test_slowest_band_never_wraps(config: RoPENDConfig) -> None:
+    """The slow end must stay monotone over every offset attention can see.
+
+    Attention sees signed offsets over [-r_max, +r_max], so the slowest band sweeps
+    omega_lo * 2 * r_max. Let that exceed pi and the band turns back on itself: two different
+    offsets in the domain collide on the same rotation, in the one band whose entire job is to
+    tell the coarse end apart.
+    """
+    inv_freq: torch.Tensor = RotaryEmbeddingND(config).inv_freq  # type: ignore[assignment]
+    sweep = float(inv_freq.min()) * 2 * R_MAX
+    assert sweep == pytest.approx(math.pi, rel=1e-4)
+
+
 def test_units_do_not_matter(config: RoPENDConfig, device: torch.device) -> None:
     """Scale the coordinates and r_min/r_max together — the encoding must not move.
 
@@ -209,6 +224,17 @@ def test_rejects_indivisible_head_dim() -> None:
         RoPENDConfig(dim_head=64, coords=3, r_min=R_MIN, r_max=R_MAX)
 
 
+def test_rejects_single_band_per_axis() -> None:
+    """dim_head == 2 * coords divides cleanly, but leaves a ladder with nowhere to descend.
+
+    The lone band lands on omega_hi and r_max is never reached, so the encoding is periodic with
+    period 2 * r_min across the entire domain — offsets of 0, 2*r_min, 4*r_min ... all give the
+    same rotation. It fails silently, which is why it is rejected at construction.
+    """
+    with pytest.raises(ValidationError, match="at least 2 bands"):
+        RoPENDConfig(dim_head=6, coords=3, r_min=R_MIN, r_max=R_MAX)
+
+
 def test_rejects_inverted_range() -> None:
     with pytest.raises(ValidationError, match="must exceed r_min"):
         RoPENDConfig(dim_head=64, coords=2, r_min=100.0, r_max=1.0)

{stackformers-4.2.0 → stackformers-4.2.1}/uv.lock

@@ -811,7 +811,7 @@ wheels = [
 
 [[package]]
 name = "stackformers"
-version = "4.2.0"
+version = "4.2.1"
 source = { editable = "." }
 dependencies = [
     { name = "beartype" },