hypertopos 0.3.0__tar.gz → 0.3.1__tar.gz

{hypertopos-0.3.0 → hypertopos-0.3.1}/CHANGELOG.md

@@ -5,6 +5,29 @@ All notable changes to hypertopos will be documented in this file.
 Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.1] — 2026-04-12
+
+### Added
+
+- Benjamini-Hochberg FDR control on attract primitives. `fdr_alpha` parameter on `pi5_attract_anomaly`, `pi6_attract_boundary`, `pi7_attract_hub`, and `pi9_attract_drift` applies BH multiple-testing correction to empirical-null p-values derived from delta rank percentile. Entities with `q_value > alpha` are removed before the top-N step. Guarantees `E[FDR] <= alpha`. Each retained entity carries a `q_value` — the minimum alpha at which it would still be retained. Off by default; legacy behavior preserved when omitted.
+- Diverse selection via submodular facility location. `select="diverse"` on the same four primitives replaces top-N ranking with a lazy-greedy maximisation of the facility location objective over pairwise cosine similarity of delta vectors. Achieves `(1 - 1/e)` approximation to the optimal K-subset. Each selected entity reports a `representativeness` count — how many population members are nearest to it among the selected set. Covers the geometric space of the result set instead of clustering around a single extreme region.
+- `engine/fdr.py` module: `benjamini_hochberg`, `empirical_p_values_from_rank`, `q_values_from_p_values` — pure NumPy, no state, no I/O.
+- `engine/selection.py` module: `lazy_greedy_facility_location`, `compute_similarity_matrix` — pure NumPy, Nemhauser-Wolsey-Fisher optimality guarantee.
+
+### Changed
+
+- Build pipeline: derived dimension scatter loops replaced with Arrow `pc.index_in` + numpy fancy indexing. Eliminates per-row Python iteration in both static (`compute_derived_batch`) and temporal (`_precompute_shape_tensor`) build paths.
+- Build pipeline: contagion stats computation (`_build_contagion_stats`) rewritten from Python loops over edge table to Arrow group_by + join. Eliminates O(E+N) Python iterations.
+- Build pipeline: rolling z-score (`_compute_max_rolling_z`) uses Welford online algorithm. Reduces time complexity from O(n_buckets² × N × D) to O(n_buckets × N × D) and memory from O(N × n_buckets × D) to O(N × D).
+- Build pipeline: temporal Arrow assembly uses single numpy reshape instead of per-bucket table construction loop.
+- Build pipeline: graph dims temporal computation uses pre-sorted event table with `searchsorted` for O(1) per-bucket slicing instead of O(E) scan per bucket.
+- Build pipeline: `build_temporal` adaptively chunks entities when the shape tensor would exceed available RAM. Auto-detects memory via platform APIs, falls back to 4 GB budget. Single-pass path (zero overhead) when tensor fits.
+
+### Fixed
+
+- `empirical_p_values_from_rank` used a `1/N` floor where N was the input array length. When the input was a small `top_n` slice (e.g. 5 entities), the floor clipped all p-values to 0.2, causing BH to reject everything at `alpha=0.05`. Now uses a fixed `1e-10` floor — the rank percentile is already computed against the full population, so the floor only needs to prevent exact zero for numerical stability.
+- `π7_attract_hub` and `π9_attract_drift` FDR computed p-values as `(N-i)/N` where N was the `top_n` result count (e.g. 10), making the best possible p-value `1/top_n` — mathematically unable to pass BH at any `alpha < 1/top_n`. Now uses population-level rank percentiles: hubs use the empirical CDF from the full scores array, drift uses the rank position relative to the total pre-truncation population.
+
 ## [0.3.0] — 2026-04-12
 
 > **Theme:** Lance 3.x perf upgrade. Aggregate engine moves from a persistent subprocess worker plus an optional external DataFusion package to the Lance scanner's built-in DataFusion executor via `LanceDataset.sql(...)`. The build-time contagion stats table replaces the runtime edge-table replay in the graph contagion scanner. Format 2.2 is the new write default.

{hypertopos-0.3.0 → hypertopos-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hypertopos
-Version: 0.3.0
+Version: 0.3.1
 Summary: Understand the structure of your data — without training machine learning models
 Project-URL: Homepage, https://github.com/hypertopos/hypertopos-py
 Project-URL: Documentation, https://github.com/hypertopos/hypertopos-py/tree/main/docs

{hypertopos-0.3.0 → hypertopos-0.3.1}/ROADMAP.md

@@ -2,7 +2,13 @@
 
 > Planned direction. Priorities may change based on feedback. 
 
-## Current: 0.3.0
+## Current: 0.3.1
+
+FDR control, diverse anomaly summarisation, and build pipeline optimization — Benjamini-Hochberg multiple-testing correction, submodular facility location, vectorized build with adaptive memory chunking. See CHANGELOG.
+
+---
+
+## 0.3.0
 
 Lance 3.x perf upgrade — aggregate engine rewritten around Lance SQL, precomputed contagion stats, format 2.2, contagion anchor resolution fix, edge table auto-detect fix. See CHANGELOG.
 
@@ -38,19 +44,19 @@ Two production-quality improvements on top of the 0.3.0 storage layer:
 
 - **FDR control on anomaly queries** — Benjamini–Hochberg `fdr_alpha` parameter + per-row `q_value` column on `find_anomalies` and friends. Replaces unbounded "top-K by `delta_norm`" with a set whose false-discovery rate is bounded by α.
 - **Diverse-coverage anomaly summarisation** — `select="diverse"` option using lazy-greedy submodular maximisation, plus a `representativeness` column ("this 1 anomaly stands in for 38 cases").
-- **Temporal build optimization** — pre-aggregate event counts before the temporal loop, eliminating per-window event table rescans.
+- **Build pipeline optimization** — vectorized scatter (Arrow `pc.index_in` + numpy), Welford rolling z-score, Arrow-native contagion stats, adaptive memory chunking for temporal build, pre-sorted graph dims bucketing.
 
 ---
 
 ## Plan: 0.3.2
 
-README and docs narrative restructure.
+TBD.
 
 ---
 
 ## Plan: 0.4.0
 
-TBD — picked after 0.3.2 lands.
+TBD.
 
 ---
 

{hypertopos-0.3.0 → hypertopos-0.3.1}/docs/api-reference.md

@@ -103,9 +103,9 @@ with sphere.session("agent-1") as session:
 
 | Primitive | Method | What it does | Returns |
 |-----------|--------|--------------|---------|
-| π5 | `π5_attract_anomaly(pattern_id, radius, top_n)` | Find most anomalous polygons | `(list[Polygon], int, list, dict)` |
-| π6 | `π6_attract_boundary(alias_id, pattern_id, direction, top_n)` | Find entities nearest to alias cutting plane | `list[(Polygon, float)]` |
-| π7 | `π7_attract_hub(pattern_id, top_n, line_id_filter)` | Find entities with highest connectivity | `list[(str, int, float)]` |
+| π5 | `π5_attract_anomaly(pattern_id, radius, top_n, fdr_alpha, fdr_method, select)` | Find most anomalous polygons | `(list[Polygon], int, list, dict)` |
+| π6 | `π6_attract_boundary(alias_id, pattern_id, direction, top_n, fdr_alpha, fdr_method, select)` | Find entities nearest to alias cutting plane | `list[(Polygon, float)]` |
+| π7 | `π7_attract_hub(pattern_id, top_n, line_id_filter, fdr_alpha, fdr_method, select)` | Find entities with highest connectivity | `list[(str, int, float)]` |
 | π7+ | `π7_attract_hub_and_stats(pattern_id, top_n, line_id_filter)` | Hub ranking + population hub score statistics in one scan | `(list, dict)` |
 | π8 | `π8_attract_cluster(pattern_id, n_clusters, top_n, sample_size)` | Discover geometric archetypes via k-means++ | `list[dict]` |
 
@@ -113,7 +113,7 @@ with sphere.session("agent-1") as session:
 
 | Primitive | Method | What it does | Returns |
 |-----------|--------|--------------|---------|
-| π9 | `π9_attract_drift(pattern_id, top_n, sample_size)` | Find entities with highest temporal drift | `list[dict]` |
+| π9 | `π9_attract_drift(pattern_id, top_n, sample_size, fdr_alpha, fdr_method, select)` | Find entities with highest temporal drift | `list[dict]` |
 | π10 | `π10_attract_trajectory(primary_key, pattern_id, top_n)` | Find entities with similar temporal trajectory | `list[dict]` |
 | π11 | `π11_attract_population_compare(pattern_id, window_a_from, window_a_to, window_b_from, window_b_to)` | Compare population geometry between two time windows | `dict` |
 | π12 | `π12_attract_regime_change(pattern_id, timestamp_from, timestamp_to)` | Detect population geometry regime shifts | `list[dict]` |

{hypertopos-0.3.0 → hypertopos-0.3.1}/docs/concepts.md

@@ -58,6 +58,16 @@ where μ = E[s] is the population mean, σ = max(std(s), ε) the clamped standar
 - **Cross-sphere comparison** *(planned)* — dimensionless scalar metrics from independently calibrated coordinate spaces enable comparison across separate data systems
 - **What-if analysis** *(planned)* — hypothetical edge changes produce a modified coordinate vector showing the geometric effect
 
+## Statistical Guarantees
+
+Two opt-in post-processing stages can be applied to any attract primitive that scans a population (pi5, pi6, pi7, pi9). Both are off by default — omitting them preserves the legacy behavior exactly.
+
+**FDR control.** When `fdr_alpha` is set, the result set passes through a Benjamini-Hochberg (BH) procedure. Each entity's empirical-null p-value is derived from its population rank percentile: `p = 1 - rank_pct / 100`, where `rank_pct` is the entity's position in the full population sorted by the primitive's ranking metric (delta norm for pi5/pi6, hub score for pi7, displacement for pi9). The BH procedure guarantees `E[FDR] <= alpha` under independence or positive regression dependency (PRDS). Every retained entity carries a q-value — the minimum alpha at which it would still be retained. Entities with `q_value > alpha` are removed before the top-N / selection step.
+
+**Diverse selection.** When `select="diverse"`, the top-N step is replaced by a lazy-greedy facility location algorithm that maximizes a monotone submodular objective over pairwise cosine similarity of delta vectors. The greedy selection achieves a `(1 - 1/e)` approximation to the optimal K-subset. Each selected entity reports a representativeness count — how many population members are closest to it among the selected set. The result is K entities that cover the geometric space of the result set rather than clustering around a single extreme region.
+
+Both stages compose: `fdr_alpha=0.05, select="diverse"` first filters to FDR-controlled discoveries, then picks the K most geometrically diverse among them.
+
 ## Three Scales
 
 ![Three Scales](images/three-scales.svg)

{hypertopos-0.3.0 → hypertopos-0.3.1}/docs/configuration.md

@@ -871,6 +871,7 @@ hypertopos build --config sphere.yaml --force --verbose
 The builder parallelizes where possible:
 - **Points write** — parallel across lines (ThreadPoolExecutor, up to 4 workers)
 - **Geometry build** — parallel across patterns (one thread per pattern)
+- **Temporal build** — parallel across patterns; adaptive memory chunking when the shape tensor exceeds available RAM (auto-detected via platform API, 4 GB fallback)
 
 ### Build timing
 

hypertopos-0.3.1/hypertopos/builder/_scatter.py

@@ -0,0 +1,72 @@
+# Copyright (C) 2026 Karol Kędzia
+# Licensed under the Business Source License 1.1 (the "License");
+# you may not use this file except in compliance with the License.
+# See LICENSE.md in the repository root for full terms.
+"""Vectorized Arrow→tensor scatter utilities.
+
+Replaces Python-level row iteration with Arrow pc.index_in + numpy
+fancy indexing. Used by both static (compute_derived_batch) and
+temporal (_precompute_shape_tensor) build paths.
+"""
+from __future__ import annotations
+
+import numpy as np
+import pyarrow as pa
+import pyarrow.compute as pc
+from numpy.typing import NDArray
+
+
+def vectorized_scatter(
+    tensor: NDArray[np.float32],
+    dim_idx: int,
+    edge_max: float,
+    anchor_keys_arr: pa.Array,
+    grouped_fk_col: pa.Array,
+    grouped_bucket_col: pa.Array,
+    grouped_values_col: pa.Array,
+) -> None:
+    if len(grouped_fk_col) == 0:
+        return
+
+    entity_idx_arr = pc.index_in(grouped_fk_col, anchor_keys_arr)
+    valid = pc.and_(pc.is_valid(entity_idx_arr), pc.is_valid(grouped_values_col))
+
+    entity_indices = entity_idx_arr.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.intp)
+    bucket_indices = grouped_bucket_col.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.intp)
+    values = grouped_values_col.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.float64)
+
+    nan_ok = ~np.isnan(values)
+    entity_indices = entity_indices[nan_ok]
+    bucket_indices = bucket_indices[nan_ok]
+    values = np.clip(values[nan_ok], 0.0, edge_max) / edge_max
+
+    tensor[entity_indices, bucket_indices, dim_idx] = values.astype(np.float32)
+
+
+def vectorized_scatter_1d(
+    result: NDArray[np.float64],
+    anchor_keys_arr: pa.Array,
+    grouped_fk_col: pa.Array,
+    grouped_values_col: pa.Array,
+) -> None:
+    if len(grouped_fk_col) == 0:
+        return
+
+    entity_idx_arr = pc.index_in(grouped_fk_col, anchor_keys_arr)
+    valid = pc.and_(pc.is_valid(entity_idx_arr), pc.is_valid(grouped_values_col))
+
+    entity_indices = entity_idx_arr.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.intp)
+    values = grouped_values_col.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.float64)
+
+    nan_ok = ~np.isnan(values)
+    result[entity_indices[nan_ok]] = values[nan_ok]

{hypertopos-0.3.0 → hypertopos-0.3.1}/hypertopos/builder/_stats.py

@@ -116,6 +116,7 @@ def compute_stats_grouped(
     shape_vectors: np.ndarray,
     group_ids: np.ndarray,
     anomaly_percentile: float = 95.0,
+    prop_dim_start: int | None = None,
 ) -> tuple[
     dict[str, tuple[np.ndarray, np.ndarray, np.ndarray, int]],
     np.ndarray,
@@ -123,6 +124,10 @@ def compute_stats_grouped(
 ]:
     """Compute per-group population statistics.
 
+    prop_dim_start: if set, apply SIGMA_EPS_PROP floor to sigma[prop_dim_start:]
+    for each group before computing deltas. Eliminates the need for a separate
+    re-run pass in the caller.
+
     Returns:
         group_results: {group_name: (mu, sigma, theta, population_size)}
         deltas: (N, D) float32 — z-scored against per-group mu/sigma
@@ -140,9 +145,12 @@ def compute_stats_grouped(
         group_shape = shape_vectors[mask]
 
         if len(group_shape) < 2:
-            # Too few entities for meaningful stats — use global-like single point
             mu_g = group_shape.mean(axis=0).astype(np.float32)
             sigma_g = np.full(D, SIGMA_EPS, dtype=np.float32)
+            if prop_dim_start is not None:
+                sigma_g[prop_dim_start:] = np.maximum(
+                    sigma_g[prop_dim_start:], SIGMA_EPS_PROP,
+                )
             theta_g = np.zeros(D, dtype=np.float32)
             deltas[mask] = ((group_shape - mu_g) / sigma_g).astype(np.float32)
             _masked = deltas[mask]
@@ -155,6 +163,22 @@ def compute_stats_grouped(
         mu_g, sigma_g, theta_g, group_deltas, group_norms, _ = compute_stats(
             group_shape, anomaly_percentile,
         )
+
+        if prop_dim_start is not None:
+            sigma_g[prop_dim_start:] = np.maximum(
+                sigma_g[prop_dim_start:], SIGMA_EPS_PROP,
+            )
+            group_deltas = ((group_shape - mu_g) / sigma_g).astype(np.float32)
+            group_norms = np.sqrt(
+                np.einsum('ij,ij->i', group_deltas, group_deltas),
+            ).astype(np.float32)
+            g_theta_scalar = (
+                float(np.percentile(group_norms, anomaly_percentile))
+                if len(group_norms) > 1 else 0.0
+            )
+            g_comp = g_theta_scalar / np.sqrt(D) if D > 0 else 0.0
+            theta_g = np.full(D, g_comp, dtype=np.float32)
+
         deltas[mask] = group_deltas
         delta_norms[mask] = group_norms
         group_results[gid_str] = (mu_g, sigma_g, theta_g, int(mask.sum()))