hypertopos 0.2.2__tar.gz → 0.3.1__tar.gz

{hypertopos-0.2.2 → hypertopos-0.3.1}/CHANGELOG.md

@@ -5,6 +5,67 @@ All notable changes to hypertopos will be documented in this file.
 Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.1] — 2026-04-12
+
+### Added
+
+- Benjamini-Hochberg FDR control on attract primitives. `fdr_alpha` parameter on `pi5_attract_anomaly`, `pi6_attract_boundary`, `pi7_attract_hub`, and `pi9_attract_drift` applies BH multiple-testing correction to empirical-null p-values derived from delta rank percentile. Entities with `q_value > alpha` are removed before the top-N step. Guarantees `E[FDR] <= alpha`. Each retained entity carries a `q_value` — the minimum alpha at which it would still be retained. Off by default; legacy behavior preserved when omitted.
+- Diverse selection via submodular facility location. `select="diverse"` on the same four primitives replaces top-N ranking with a lazy-greedy maximisation of the facility location objective over pairwise cosine similarity of delta vectors. Achieves `(1 - 1/e)` approximation to the optimal K-subset. Each selected entity reports a `representativeness` count — how many population members are nearest to it among the selected set. Covers the geometric space of the result set instead of clustering around a single extreme region.
+- `engine/fdr.py` module: `benjamini_hochberg`, `empirical_p_values_from_rank`, `q_values_from_p_values` — pure NumPy, no state, no I/O.
+- `engine/selection.py` module: `lazy_greedy_facility_location`, `compute_similarity_matrix` — pure NumPy, Nemhauser-Wolsey-Fisher optimality guarantee.
+
+### Changed
+
+- Build pipeline: derived dimension scatter loops replaced with Arrow `pc.index_in` + numpy fancy indexing. Eliminates per-row Python iteration in both static (`compute_derived_batch`) and temporal (`_precompute_shape_tensor`) build paths.
+- Build pipeline: contagion stats computation (`_build_contagion_stats`) rewritten from Python loops over edge table to Arrow group_by + join. Eliminates O(E+N) Python iterations.
+- Build pipeline: rolling z-score (`_compute_max_rolling_z`) uses Welford online algorithm. Reduces time complexity from O(n_buckets² × N × D) to O(n_buckets × N × D) and memory from O(N × n_buckets × D) to O(N × D).
+- Build pipeline: temporal Arrow assembly uses single numpy reshape instead of per-bucket table construction loop.
+- Build pipeline: graph dims temporal computation uses pre-sorted event table with `searchsorted` for O(1) per-bucket slicing instead of O(E) scan per bucket.
+- Build pipeline: `build_temporal` adaptively chunks entities when the shape tensor would exceed available RAM. Auto-detects memory via platform APIs, falls back to 4 GB budget. Single-pass path (zero overhead) when tensor fits.
+
+### Fixed
+
+- `empirical_p_values_from_rank` used a `1/N` floor where N was the input array length. When the input was a small `top_n` slice (e.g. 5 entities), the floor clipped all p-values to 0.2, causing BH to reject everything at `alpha=0.05`. Now uses a fixed `1e-10` floor — the rank percentile is already computed against the full population, so the floor only needs to prevent exact zero for numerical stability.
+- `π7_attract_hub` and `π9_attract_drift` FDR computed p-values as `(N-i)/N` where N was the `top_n` result count (e.g. 10), making the best possible p-value `1/top_n` — mathematically unable to pass BH at any `alpha < 1/top_n`. Now uses population-level rank percentiles: hubs use the empirical CDF from the full scores array, drift uses the rank position relative to the total pre-truncation population.
+
+## [0.3.0] — 2026-04-12
+
+> **Theme:** Lance 3.x perf upgrade. Aggregate engine moves from a persistent subprocess worker plus an optional external DataFusion package to the Lance scanner's built-in DataFusion executor via `LanceDataset.sql(...)`. The build-time contagion stats table replaces the runtime edge-table replay in the graph contagion scanner. Format 2.2 is the new write default.
+>
+> **Mandatory rebuild for spheres built before this release.** See the Migration section below.
+
+### Changed
+
+- `pylance` minimum bumped to 4.x. New writes target Lance format 2.2 with structural-decode parallelism, BSS float compression, cascading codecs, and block-level RLE on top of the previous baseline. Older spheres written with format 2.0 / 2.1 remain readable transparently.
+- The aggregate engine now pushes GROUP BY computation directly into the Lance scanner. The new `lance_sql_agg` module hosts `aggregate_count`, `aggregate_metric` (sum / avg / min / max), `aggregate_filtered_metric`, `aggregate_pivot`, `aggregate_property`, `aggregate_percentile`, and `find_anomalies`. Each helper streams the columns it needs from the geometry / event-line / property-line Lance datasets through `LanceDataset.sql(...)` and finishes the join + aggregate in pyarrow on the small post-projection result. The full geometry table is never loaded into Python memory and the persistent subprocess worker is no longer involved for any of these paths.
+- Test fixture cloning uses `lance.LanceDataset.shallow_clone` for every Lance dataset directory inside a sphere tree, with a thin `clone_sphere` helper in `tests/conftest.py` that walks the directory and falls back to `shutil.copy2` for non-Lance files. Shallow clone copies only metadata and references — measurably faster on Windows where deep `copytree` over a sphere built from many small Lance fragment files is dominated by per-file NTFS overhead.
+
+### Added
+
+- `GDSBuilder` rejects patterns with zero declared dimensions (no relations, no event dimensions, no derived dimensions, no tracked properties) at validation time. A pattern with width-0 delta vectors had no meaningful geometry to compute even before this release; format 2.0 silently allowed the construct, format 2.1+ refuses to encode the resulting `fixed_size_list[0]` columns at all. The validation surfaces a typed error up front instead of letting the call descend into an opaque write-time panic.
+- Builder precomputes per-entity contagion statistics for every pattern with an edge table and writes them to `_gds_meta/contagion_stats/{pattern_id}.lance` with a BTREE index on `primary_key`. The table holds `(primary_key, neighbor_count, anomalous_neighbor_count, contagion_ratio)` snapshotted at build time against the just-calibrated anchor geometry.
+
+### Fixed
+
+- `_scan_graph` (the graph contagion source feeding `passive_scan` and `composite_risk`) reads the precomputed `_gds_meta/contagion_stats/{pattern_id}.lance` table directly and threshold-filters inline. The legacy edge-table-replay path is gone; spheres built before 0.3.0 must be rebuilt to get graph contagion hits.
+- `contagion_score` anchor pattern resolution now correctly resolves via sibling lines when the anchor pattern is not directly associated with the target entity's line. Previously crashed with `KeyError` on spheres where the anchor pattern entity line differed from the navigated line.
+- Edge table auto-detect excludes metadata columns (`created_at`, `changed_at`, `version`) from the timestamp type-based fallback, preventing incorrect timestamp column selection on entity lines with versioning columns. Amount candidate list widened to cover `fare_amount`, `total_amount`, `amount_received`, `amount_paid`.
+
+### Security
+
+- `lance_sql_agg` validates user-controlled SQL inputs up front. Lance's `LanceDataset.sql(...)` does not expose parameter binding, so values and identifiers are inlined into the query string. Filter values (entity primary keys) are escaped with `_escape_sql_string`, which doubles single quotes and rejects backslash and ASCII control characters. Column-name identifiers (`metric_col`, `pivot_field`, `prop_name`) are validated against `^[A-Za-z_][A-Za-z0-9_]*$` via `_validate_sql_identifier` before being inlined into SQL strings — anything outside that pattern raises immediately. Both helpers fail loud rather than silently produce wrong results or pass injected SQL through to the parser.
+
+### Removed
+
+- `hypertopos.engine.subprocess_agg` (the persistent subprocess worker and its JSON command protocol) is deleted; every entry point it served is now routed through `lance_sql_agg`.
+- `hypertopos.engine.datafusion_agg` (the in-memory external-DataFusion fast path for count / metric / pivot / gbp / percentile) is deleted along with the optional `datafusion` extra in `pyproject.toml`. Lance 4.x's built-in DataFusion executor covers every query the external package was carrying.
+- The matching `tests/test_subprocess_agg.py` and `tests/test_datafusion_agg.py` are deleted. Aggregate semantics are exercised through `tests/test_aggregation_engine.py` and the MCP tool tests.
+
+### Migration
+
+- **Rebuild required.** Spheres built before 0.3.0 are still openable but degrade gracefully on graph contagion — the precomputed `contagion_stats` table is created at build time only, and the runtime scanner returns zero hits without it. Rebuild with `hypertopos build --config sphere.yaml --force`. There are no in-place upgrade paths.
+- The optional `datafusion` package is no longer a dependency of any aggregate path. If your environment installed it via `pip install hypertopos[datafusion]`, that extra now resolves to nothing — you can drop it from your install command.
+
 ## [0.2.2] — 2026-04-11
 
 ### Added

{hypertopos-0.2.2 → hypertopos-0.3.1}/LICENSE.md

@@ -26,7 +26,7 @@ For purposes of this License:
 - **Internal use** within an organization is not a competitive offering. Your
   organization includes entities under common control.
 
-**Change Date:** 2029-04-11
+**Change Date:** 2029-04-12
 
 **Change License:** Apache License, Version 2.0
 

{hypertopos-0.2.2 → hypertopos-0.3.1}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: hypertopos
-Version: 0.2.2
-Summary: Navigate your data like space.
+Version: 0.3.1
+Summary: Understand the structure of your data — without training machine learning models
 Project-URL: Homepage, https://github.com/hypertopos/hypertopos-py
 Project-URL: Documentation, https://github.com/hypertopos/hypertopos-py/tree/main/docs
 Project-URL: Repository, https://github.com/hypertopos/hypertopos-py
@@ -12,10 +12,8 @@ License-File: LICENSE.md
 Requires-Python: >=3.12
 Requires-Dist: numpy>=1.26
 Requires-Dist: pyarrow>=14.0
-Requires-Dist: pylance>=0.20
+Requires-Dist: pylance<5.0,>=4.0
 Requires-Dist: pyyaml>=6.0
-Provides-Extra: datafusion
-Requires-Dist: datafusion>=40.0; extra == 'datafusion'
 Provides-Extra: dev
 Requires-Dist: mypy; extra == 'dev'
 Requires-Dist: pytest; extra == 'dev'

{hypertopos-0.2.2 → hypertopos-0.3.1}/README.md

@@ -6,9 +6,9 @@
 [![License: BSL 1.1](https://img.shields.io/badge/license-BSL%201.1-orange.svg)](LICENSE.md)
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19482069.svg)](https://doi.org/10.5281/zenodo.19482069)
 [![PyArrow](https://img.shields.io/badge/format-PyArrow-red.svg)](https://arrow.apache.org/docs/python/)
-[![Lance](https://img.shields.io/badge/storage-Lance-blueviolet.svg)](https://github.com/lancedb/lancedb)
+[![Lance](https://img.shields.io/badge/storage-Lance-blueviolet.svg)](https://github.com/lance-format/lance)
 [![MCP](https://img.shields.io/badge/MCP-compatible-green.svg)](https://modelcontextprotocol.io)
-[![Version](https://img.shields.io/badge/version-0.2.2-blue.svg)](pyproject.toml)
+[![Version](https://img.shields.io/badge/version-0.3.0-blue.svg)](pyproject.toml)
 
 **hypertopos** transforms relational data into navigable geometric space.
 

{hypertopos-0.2.2 → hypertopos-0.3.1}/ROADMAP.md

@@ -2,7 +2,19 @@
 
 > Planned direction. Priorities may change based on feedback. 
 
-## Current: 0.2.2
+## Current: 0.3.1
+
+FDR control, diverse anomaly summarisation, and build pipeline optimization — Benjamini-Hochberg multiple-testing correction, submodular facility location, vectorized build with adaptive memory chunking. See CHANGELOG.
+
+---
+
+## 0.3.0
+
+Lance 3.x perf upgrade — aggregate engine rewritten around Lance SQL, precomputed contagion stats, format 2.2, contagion anchor resolution fix, edge table auto-detect fix. See CHANGELOG.
+
+---
+
+## 0.2.2
 
 As-of graph reconstruction across edge-table graph primitives + latency fix on `detect_cross_pattern_discrepancy`. See CHANGELOG.
 
@@ -26,33 +38,57 @@ First public release — full GDS stack, π1–π12, builder, MCP server, valida
 
 ---
 
-## Plan: 0.3.0
+## Plan: 0.3.1
 
-**Storage / performance:**
-- Upgrade `pylance 2.0.1 → 3.0.1` — format 2.2, ~300% faster scans, LZ4/zstd compression
-- Replace `shutil.copytree` with `lance.dataset.clone()` in test fixtures (O(1) metadata clone)
-- Builder: incremental rebuild (geometry-only without `--force` wipe)
+Two production-quality improvements on top of the 0.3.0 storage layer:
 
-**Code refactoring:**
-- Break up oversized modules into smaller, domain-focused components
-- Reduce coupling between core layers by replacing private cross-layer access with explicit interfaces where practical
-- Tighten error handling in hot paths so failures are surfaced more consistently
-- Consolidate repeated orchestration logic into shared helpers instead of duplicating it across modules
-- Make the refactor pass incremental: preserve behavior, improve structure, then revisit deeper architectural boundaries
+- **FDR control on anomaly queries** — Benjamini–Hochberg `fdr_alpha` parameter + per-row `q_value` column on `find_anomalies` and friends. Replaces unbounded "top-K by `delta_norm`" with a set whose false-discovery rate is bounded by α.
+- **Diverse-coverage anomaly summarisation** — `select="diverse"` option using lazy-greedy submodular maximisation, plus a `representativeness` column ("this 1 anomaly stands in for 38 cases").
+- **Build pipeline optimization** — vectorized scatter (Arrow `pc.index_in` + numpy), Welford rolling z-score, Arrow-native contagion stats, adaptive memory chunking for temporal build, pre-sorted graph dims bucketing.
 
-**PassiveScanner evolution:**
-- Native temporal source support for direct temporal inputs, without requiring manual dataset plumbing in benchmark scripts
-- Optional weighted scoring mode that uses continuous intensity instead of binary counts
-- **SphereProfiler** — autonomous sphere scanner that profiles all patterns, runs calibration sweeps across source combinations, proposes optimal PassiveScanner composition for Layer 1 surveillance. Core loop: enumerate patterns → single-source anomaly scans → greedy multi-source combination → ranked composition report. No labeled GT needed; optional GT file unlocks supervised calibration.
+---
 
-**Anomaly detection quality:**
-- Confidence scoring, robust estimators, multi-scale resolution — improve anomaly precision and reduce false positives on heavy-tail and multi-modal populations
+## Plan: 0.3.2
+
+TBD.
+
+---
+
+## Plan: 0.4.0
+
+TBD.
 
 ---
 
 ## Future
 
+**Detection quality**
+- Edge-derived dimensions + temporal motif matcher
+- Confidence scoring, robust estimators, multi-scale resolution — improve anomaly precision and reduce false positives on heavy-tail and multi-modal populations
+- Lazy chain geometry — on-demand chain delta vectors via sampled population calibration; supplements the build-time `chain_lines` path
+
+**Builder evolution**
+- Incremental rebuild — geometry-only without `--force` wipe
+- Generalized (d+m+g+t+s) dimension blocks — geographic / metric / semantic dimension support
+
+**PassiveScanner evolution**
+- Native temporal source support for direct temporal inputs, without requiring manual dataset plumbing in benchmark scripts
+- Optional weighted scoring mode that uses continuous intensity instead of binary counts
+- **SphereProfiler** — autonomous sphere scanner that profiles all patterns, runs calibration sweeps across source combinations, proposes optimal PassiveScanner composition for Layer 1 surveillance. Core loop: enumerate patterns → single-source anomaly scans → greedy multi-source combination → ranked composition report. No labeled GT needed; optional GT file unlocks supervised calibration.
+
+**Code refactoring**
+- Break up oversized modules into smaller, domain-focused components
+- Reduce coupling between core layers by replacing private cross-layer access with explicit interfaces where practical
+- Tighten error handling in hot paths so failures are surfaced more consistently
+- Consolidate repeated orchestration logic into shared helpers instead of duplicating it across modules
+- Incremental refactor: preserve behavior, improve structure, then revisit deeper architectural boundaries
+
+**Cross-sphere capabilities**
 - Cross-sphere comparison — dimensionless metrics across independently calibrated coordinate spaces
 - What-if analysis — hypothetical edge changes producing modified coordinate vectors
+
+**Enterprise / governance**
 - Dimension access control — per-agent visibility constraints on delta dimensions
+
+**Tooling**
 - Runtime latency benchmarks in package docs

{hypertopos-0.2.2 → hypertopos-0.3.1}/SECURITY.md

@@ -1,6 +1,6 @@
 # Security
 
-## Current version: 0.2.2
+## Current version: 0.3.0
 
 hypertopos runs locally. No network services, no auth layer, no multi-tenancy.
 

{hypertopos-0.2.2 → hypertopos-0.3.1}/docs/api-reference.md

@@ -103,9 +103,9 @@ with sphere.session("agent-1") as session:
 
 | Primitive | Method | What it does | Returns |
 |-----------|--------|--------------|---------|
-| π5 | `π5_attract_anomaly(pattern_id, radius, top_n)` | Find most anomalous polygons | `(list[Polygon], int, list, dict)` |
-| π6 | `π6_attract_boundary(alias_id, pattern_id, direction, top_n)` | Find entities nearest to alias cutting plane | `list[(Polygon, float)]` |
-| π7 | `π7_attract_hub(pattern_id, top_n, line_id_filter)` | Find entities with highest connectivity | `list[(str, int, float)]` |
+| π5 | `π5_attract_anomaly(pattern_id, radius, top_n, fdr_alpha, fdr_method, select)` | Find most anomalous polygons | `(list[Polygon], int, list, dict)` |
+| π6 | `π6_attract_boundary(alias_id, pattern_id, direction, top_n, fdr_alpha, fdr_method, select)` | Find entities nearest to alias cutting plane | `list[(Polygon, float)]` |
+| π7 | `π7_attract_hub(pattern_id, top_n, line_id_filter, fdr_alpha, fdr_method, select)` | Find entities with highest connectivity | `list[(str, int, float)]` |
 | π7+ | `π7_attract_hub_and_stats(pattern_id, top_n, line_id_filter)` | Hub ranking + population hub score statistics in one scan | `(list, dict)` |
 | π8 | `π8_attract_cluster(pattern_id, n_clusters, top_n, sample_size)` | Discover geometric archetypes via k-means++ | `list[dict]` |
 
@@ -113,7 +113,7 @@ with sphere.session("agent-1") as session:
 
 | Primitive | Method | What it does | Returns |
 |-----------|--------|--------------|---------|
-| π9 | `π9_attract_drift(pattern_id, top_n, sample_size)` | Find entities with highest temporal drift | `list[dict]` |
+| π9 | `π9_attract_drift(pattern_id, top_n, sample_size, fdr_alpha, fdr_method, select)` | Find entities with highest temporal drift | `list[dict]` |
 | π10 | `π10_attract_trajectory(primary_key, pattern_id, top_n)` | Find entities with similar temporal trajectory | `list[dict]` |
 | π11 | `π11_attract_population_compare(pattern_id, window_a_from, window_a_to, window_b_from, window_b_to)` | Compare population geometry between two time windows | `dict` |
 | π12 | `π12_attract_regime_change(pattern_id, timestamp_from, timestamp_to)` | Detect population geometry regime shifts | `list[dict]` |

{hypertopos-0.2.2 → hypertopos-0.3.1}/docs/architecture.md

@@ -24,7 +24,7 @@ The core package is split into a few layers:
 
 - **Model**: sphere, line, pattern, alias, point, edge, polygon, solid, manifest, contract
 - **Storage**: readers, writers, caches, and physical layout handling
-- **Engine**: geometry, anomaly scoring, similarity, drift, forecasting, aggregation, calibration tracking
+- **Engine**: geometry, anomaly scoring, similarity, drift, forecasting, aggregation (Lance SQL pushdown via `lance_sql_agg`), calibration tracking
 - **Navigation**: stateful step-by-step movement through the sphere
 - **PassiveScanner**: multi-source batch screening over navigation primitives
 - **Builder**: transforms source data into a complete sphere

{hypertopos-0.2.2 → hypertopos-0.3.1}/docs/concepts.md

@@ -58,6 +58,16 @@ where μ = E[s] is the population mean, σ = max(std(s), ε) the clamped standar
 - **Cross-sphere comparison** *(planned)* — dimensionless scalar metrics from independently calibrated coordinate spaces enable comparison across separate data systems
 - **What-if analysis** *(planned)* — hypothetical edge changes produce a modified coordinate vector showing the geometric effect
 
+## Statistical Guarantees
+
+Two opt-in post-processing stages can be applied to any attract primitive that scans a population (pi5, pi6, pi7, pi9). Both are off by default — omitting them preserves the legacy behavior exactly.
+
+**FDR control.** When `fdr_alpha` is set, the result set passes through a Benjamini-Hochberg (BH) procedure. Each entity's empirical-null p-value is derived from its population rank percentile: `p = 1 - rank_pct / 100`, where `rank_pct` is the entity's position in the full population sorted by the primitive's ranking metric (delta norm for pi5/pi6, hub score for pi7, displacement for pi9). The BH procedure guarantees `E[FDR] <= alpha` under independence or positive regression dependency (PRDS). Every retained entity carries a q-value — the minimum alpha at which it would still be retained. Entities with `q_value > alpha` are removed before the top-N / selection step.
+
+**Diverse selection.** When `select="diverse"`, the top-N step is replaced by a lazy-greedy facility location algorithm that maximizes a monotone submodular objective over pairwise cosine similarity of delta vectors. The greedy selection achieves a `(1 - 1/e)` approximation to the optimal K-subset. Each selected entity reports a representativeness count — how many population members are closest to it among the selected set. The result is K entities that cover the geometric space of the result set rather than clustering around a single extreme region.
+
+Both stages compose: `fdr_alpha=0.05, select="diverse"` first filters to FDR-controlled discoveries, then picks the K most geometrically diverse among them.
+
 ## Three Scales
 
 ![Three Scales](images/three-scales.svg)

{hypertopos-0.2.2 → hypertopos-0.3.1}/docs/configuration.md

@@ -871,6 +871,7 @@ hypertopos build --config sphere.yaml --force --verbose
 The builder parallelizes where possible:
 - **Points write** — parallel across lines (ThreadPoolExecutor, up to 4 workers)
 - **Geometry build** — parallel across patterns (one thread per pattern)
+- **Temporal build** — parallel across patterns; adaptive memory chunking when the shape tensor exceeds available RAM (auto-detected via platform API, 4 GB fallback)
 
 ### Build timing
 

{hypertopos-0.2.2 → hypertopos-0.3.1}/docs/data-format.md

@@ -17,7 +17,8 @@ gds_{sphere_id}/
 │   ├── geometry_stats/          # precomputed population summaries
 │   ├── trajectory/              # ANN index for trajectory similarity search
 │   ├── temporal_centroids/      # cached population centroids per time window
-│   └── edge_stats/              # per-event-pattern edge table summary cache (row count, unique from/to, ts/amount range)
+│   ├── edge_stats/              # per-event-pattern edge table summary cache (row count, unique from/to, ts/amount range)
+│   └── contagion_stats/         # per-pattern (primary_key, neighbor_count, anomalous_neighbor_count, contagion_ratio) — feeds the graph contagion scanner directly, BTREE-indexed on primary_key
 ├── points/
 │   ├── {line_id}/v={n}/
 │   │   └── data.lance           # entity records for this line version

{hypertopos-0.2.2 → hypertopos-0.3.1}/hypertopos/__init__.py

@@ -16,4 +16,4 @@ __all__ = [
     "GDSBuilder",
     "RelationSpec",
 ]
-__version__ = "0.2.2"
+__version__ = "0.3.0"

hypertopos-0.3.1/hypertopos/builder/_scatter.py

@@ -0,0 +1,72 @@
+# Copyright (C) 2026 Karol Kędzia
+# Licensed under the Business Source License 1.1 (the "License");
+# you may not use this file except in compliance with the License.
+# See LICENSE.md in the repository root for full terms.
+"""Vectorized Arrow→tensor scatter utilities.
+
+Replaces Python-level row iteration with Arrow pc.index_in + numpy
+fancy indexing. Used by both static (compute_derived_batch) and
+temporal (_precompute_shape_tensor) build paths.
+"""
+from __future__ import annotations
+
+import numpy as np
+import pyarrow as pa
+import pyarrow.compute as pc
+from numpy.typing import NDArray
+
+
+def vectorized_scatter(
+    tensor: NDArray[np.float32],
+    dim_idx: int,
+    edge_max: float,
+    anchor_keys_arr: pa.Array,
+    grouped_fk_col: pa.Array,
+    grouped_bucket_col: pa.Array,
+    grouped_values_col: pa.Array,
+) -> None:
+    if len(grouped_fk_col) == 0:
+        return
+
+    entity_idx_arr = pc.index_in(grouped_fk_col, anchor_keys_arr)
+    valid = pc.and_(pc.is_valid(entity_idx_arr), pc.is_valid(grouped_values_col))
+
+    entity_indices = entity_idx_arr.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.intp)
+    bucket_indices = grouped_bucket_col.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.intp)
+    values = grouped_values_col.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.float64)
+
+    nan_ok = ~np.isnan(values)
+    entity_indices = entity_indices[nan_ok]
+    bucket_indices = bucket_indices[nan_ok]
+    values = np.clip(values[nan_ok], 0.0, edge_max) / edge_max
+
+    tensor[entity_indices, bucket_indices, dim_idx] = values.astype(np.float32)
+
+
+def vectorized_scatter_1d(
+    result: NDArray[np.float64],
+    anchor_keys_arr: pa.Array,
+    grouped_fk_col: pa.Array,
+    grouped_values_col: pa.Array,
+) -> None:
+    if len(grouped_fk_col) == 0:
+        return
+
+    entity_idx_arr = pc.index_in(grouped_fk_col, anchor_keys_arr)
+    valid = pc.and_(pc.is_valid(entity_idx_arr), pc.is_valid(grouped_values_col))
+
+    entity_indices = entity_idx_arr.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.intp)
+    values = grouped_values_col.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.float64)
+
+    nan_ok = ~np.isnan(values)
+    result[entity_indices[nan_ok]] = values[nan_ok]

{hypertopos-0.2.2 → hypertopos-0.3.1}/hypertopos/builder/_stats.py

@@ -116,6 +116,7 @@ def compute_stats_grouped(
     shape_vectors: np.ndarray,
     group_ids: np.ndarray,
     anomaly_percentile: float = 95.0,
+    prop_dim_start: int | None = None,
 ) -> tuple[
     dict[str, tuple[np.ndarray, np.ndarray, np.ndarray, int]],
     np.ndarray,
@@ -123,6 +124,10 @@ def compute_stats_grouped(
 ]:
     """Compute per-group population statistics.
 
+    prop_dim_start: if set, apply SIGMA_EPS_PROP floor to sigma[prop_dim_start:]
+    for each group before computing deltas. Eliminates the need for a separate
+    re-run pass in the caller.
+
     Returns:
         group_results: {group_name: (mu, sigma, theta, population_size)}
         deltas: (N, D) float32 — z-scored against per-group mu/sigma
@@ -140,9 +145,12 @@ def compute_stats_grouped(
         group_shape = shape_vectors[mask]
 
         if len(group_shape) < 2:
-            # Too few entities for meaningful stats — use global-like single point
             mu_g = group_shape.mean(axis=0).astype(np.float32)
             sigma_g = np.full(D, SIGMA_EPS, dtype=np.float32)
+            if prop_dim_start is not None:
+                sigma_g[prop_dim_start:] = np.maximum(
+                    sigma_g[prop_dim_start:], SIGMA_EPS_PROP,
+                )
             theta_g = np.zeros(D, dtype=np.float32)
             deltas[mask] = ((group_shape - mu_g) / sigma_g).astype(np.float32)
             _masked = deltas[mask]
@@ -155,6 +163,22 @@ def compute_stats_grouped(
         mu_g, sigma_g, theta_g, group_deltas, group_norms, _ = compute_stats(
             group_shape, anomaly_percentile,
         )
+
+        if prop_dim_start is not None:
+            sigma_g[prop_dim_start:] = np.maximum(
+                sigma_g[prop_dim_start:], SIGMA_EPS_PROP,
+            )
+            group_deltas = ((group_shape - mu_g) / sigma_g).astype(np.float32)
+            group_norms = np.sqrt(
+                np.einsum('ij,ij->i', group_deltas, group_deltas),
+            ).astype(np.float32)
+            g_theta_scalar = (
+                float(np.percentile(group_norms, anomaly_percentile))
+                if len(group_norms) > 1 else 0.0
+            )
+            g_comp = g_theta_scalar / np.sqrt(D) if D > 0 else 0.0
+            theta_g = np.full(D, g_comp, dtype=np.float32)
+
         deltas[mask] = group_deltas
         delta_norms[mask] = group_norms
         group_results[gid_str] = (mu_g, sigma_g, theta_g, int(mask.sum()))