hypertopos 0.3.2__tar.gz → 0.4.0__tar.gz

{hypertopos-0.3.2 → hypertopos-0.4.0}/CHANGELOG.md

@@ -7,6 +7,44 @@ Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.4.0] — 2026-04-15
+
+### Added
+- Distribution-aware Bregman divergence with per-dimension kind tags (gaussian/poisson/bernoulli). Auto-detected from dimension type; YAML `kind:` override per dimension.
+- Per-dimension anomaly threshold (hyper-ellipsoid boundary) replacing uniform decomposition (hyper-sphere).
+- `anomaly_confidence` per entity via stratified bootstrap resampling. Configurable `bootstrap_iterations` (default 200, 0 to skip). Skipped for populations > 50K (use `conformal_p`), `group_by_property`, and `use_mahalanobis` patterns.
+- `bregman_divergence` stored alongside `delta_norm` in geometry.
+- `dimension_kinds` per pattern in sphere.json metadata.
+- `find_anomalies` returns edges on polygons (both fast path and in-process path).
+- `find_neighborhood` raises `GDSNavigationError` for continuous-mode patterns instead of silently returning empty results.
+- In-memory adjacency index for graph operations — lazy build on first graph call, cached on session reader.
+- Build pipeline per-pattern parallelism: geometry → temporal as concurrent pipeline (up to 4 threads).
+
+### Changed
+- `explain_anomaly` returns per-dimension Bregman contributions (additive, kind-labeled) instead of absolute z-score deltas when dimension_kinds available.
+- Sphere format bumped to 2.3. Old spheres must be rebuilt.
+- Chain extraction uses temporal bisect for neighbor filtering instead of linear timestamp scan.
+
+### Fixed
+- `find_anomalies` pagination: deterministic ordering on tied `delta_norm` values prevents entity duplication across page boundaries.
+- `hub_score_history` clamps negative `hub_score`/`alive_edges_est` to zero for early temporal slices.
+- `anomalous_edges` deduplicates by `event_key` for self-loops (`from_key == to_key`).
+- `find_clusters`, `find_drifting_entities`, `compare_time_windows`, `find_regime_changes` use `pattern.dim_labels` for dimension names, including event dimensions.
+- `find_geometric_path` default `beam_width` increased from 10 to 50; auto-scales for deep searches.
+- `aggregate` raises explicit error when `group_by_line` matches multiple relations on the same pattern.
+- `search_entities` JSON serialization handles datetime columns.
+- `sphere_overview` mode detection uses `sample_size=200` instead of full geometry table read.
+- Weighted Bregman norms for `metric="bregman"` in `find_anomalies`.
+
+## [0.3.3] — 2026-04-13
+
+### Added
+
+- `dim_mask` parameter on `find_similar_entities` — compute distance only on named dimensions. Focuses similarity search on specific aspects of geometry (e.g., only the dimensions driving an anomaly).
+- `metric` parameter on `find_similar_entities` — `"L2"` (default Euclidean) or `"cosine"` (shape similarity ignoring magnitude). Cosine compares anomaly profile direction, not severity.
+- `metric` parameter on `find_anomalies` (π5) — `"L2"` (default, pre-computed) or `"Linf"` (max single-dimension |delta|, runtime scan). Linf catches single-dimension spikes that L2 dilutes.
+- `push-mirrors.sh --from-ref` mode for incremental mirror pushes between tagged releases, with pre/post author verification guard.
+
 ## [0.3.2] — 2026-04-13
 
 ### Added

{hypertopos-0.3.2 → hypertopos-0.4.0}/LICENSE.md

@@ -26,7 +26,7 @@ For purposes of this License:
 - **Internal use** within an organization is not a competitive offering. Your
   organization includes entities under common control.
 
-**Change Date:** 2029-04-13
+**Change Date:** 2029-04-15
 
 **Change License:** Apache License, Version 2.0
 

{hypertopos-0.3.2 → hypertopos-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hypertopos
-Version: 0.3.2
+Version: 0.4.0
 Summary: Understand the structure of your data — without training machine learning models
 Project-URL: Homepage, https://github.com/hypertopos/hypertopos-py
 Project-URL: Documentation, https://github.com/hypertopos/hypertopos-py/tree/main/docs

hypertopos-0.4.0/README.md

@@ -0,0 +1,103 @@
+# hypertopos
+
+> A behavioral feature layer for graph and temporal data — turning behavior into coordinates, trajectories, and explanations.
+
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+[![License: BSL 1.1](https://img.shields.io/badge/license-BSL%201.1-orange.svg)](LICENSE.md)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19482069.svg)](https://doi.org/10.5281/zenodo.19482069)
+[![PyArrow](https://img.shields.io/badge/format-PyArrow-red.svg)](https://arrow.apache.org/docs/python/)
+[![Lance](https://img.shields.io/badge/storage-Lance-blueviolet.svg)](https://github.com/lance-format/lance)
+[![MCP](https://img.shields.io/badge/MCP-compatible-green.svg)](https://modelcontextprotocol.io)
+[![Version](https://img.shields.io/badge/version-0.4.0-%235500FF.svg)](pyproject.toml)
+
+hypertopos is not a database, and not a machine learning model. It is a layer that turns relational data into a coordinate system where every entity gets a position derived from its relationships and the population around it.
+
+```
+typical:    data → storage → features → feature store → ML → serving → decision
+hypertopos: data → hypertopos (WIP) → ML
+```
+
+```bash
+pip install hypertopos
+```
+
+![hypertopos overview](docs/images/hypertopos-overview.svg)
+
+## How it works
+
+You describe your data in YAML — entity types, sources, relationships. hypertopos computes population statistics and produces a sphere: pre-computed geometry stored in Apache Arrow format.
+
+Agents (or Python code) open the sphere and navigate it using twelve primitives that cover movement, clustering, anomaly detection, population comparison, and temporal analysis. Each step is stateful — where you are determines what you see next.
+
+For the full picture: [Introduction](docs/introduction.md) · [Core Concepts](docs/concepts.md) · [Quick Start](docs/quickstart.md)
+
+## What's different
+
+Each capability below emerges from treating entities as points in a shared, population-calibrated space.
+
+| Capability | What it does | Why it matters | What others do instead | Since |
+|---|---|---|---|---|
+| Population-relative coordinates | Represents each entity as its deviation from the population (in sigma units). `delta = (shape - mu) / sigma`, computed in one aggregation pass. | Anomaly detection, clustering, and comparison all work on the same coordinate — no separate pipelines. | node2vec/GNN: learned embeddings, latent dims, retraining on shift. Feature stores: static columns, no population calibration. | `0.1.0` |
+| Self-calibrating threshold | Anomaly when `\|\|delta\|\| > theta`, where `theta = percentile(\|\|delta\|\|, 95)`. Adapts to any population shape. | Works on banking, logistics, healthcare without parameter tuning or labeled data. | PyOD/Isolation Forest: fit model, choose contamination rate. Sigma rules: assume Gaussian. | `0.1.0` |
+| Named dimension attribution | `explain_anomaly` ranks dimensions by contribution: `loan_count: +3.2 sigma`, `has_email: -1.8 sigma`. Contributions sum to the total anomaly score. | Every anomaly is immediately interpretable — analyst sees which structural property drives it. | SHAP/LIME: explain a model's prediction on latent features. Here there is no model — dimensions ARE the explanation. | `0.1.0` |
+| Temporal deformation log | Append-only log of delta vectors at each structural change. Three drift metrics: displacement (net), path length (cumulative), directionality (displacement/path). | Behavioral change is geometric movement — "how far", "how fast", and "in which direction" are one query. | Time-series DBs store metric values. Here the time series is the entity's trajectory through coordinate space. | `0.1.0` |
+| Stateful navigation | 12 typed primitives (`walk_line`, `jump_polygon`, `dive_solid`, `attract_anomaly`, ...). Position type (None/Point/Polygon/Solid) constrains valid operations at runtime. | AI agents navigate structure with guardrails — invalid operations are rejected by the type system, not by error handling. | SQL/GraphQL: stateless queries. REST APIs: stateless endpoints. | `0.1.0` |
+| Cross-sphere comparison | `\|\|delta\|\|` is dimensionless (units of sigma). 4.2 in a banking sphere = 4.2 in a logistics sphere. No shared schema or feature alignment required. | Compare structural deviation across domains without joint training or manual normalization. | Cross-domain comparison typically requires shared features or joint embeddings. | `0.1.0` |
+| Counterfactual simulation | `simulate_edges(entity, add=[(type, 5)])` recomputes delta against fixed mu/sigma (population baseline unchanged). Reports if entity crosses threshold. | Answers "what if this entity had 5 more transactions?" deterministically, without building a causal model. | Causal inference: requires explicit DAG. What-if in SQL: manual re-aggregation. | `0.1.0` |
+| Graph contagion scoring | `contagion_score` measures mean `\|\|delta\|\|` of connected entities. `propagate_influence` computes Cohen's d between connected and control groups. | Measures how anomaly propagates through relationships — not who is central, but whose behavior spreads. | PageRank/betweenness measure graph topology, not behavioral propagation through edges. | `0.2.0` |
+| Witness cohort | `find_witness_cohort` finds entities with similar deltas that are NOT directly connected. Combines delta cosine, witness Jaccard, trajectory cosine. | Validates an anomaly is a pattern rather than a one-off — "others behave the same way." | No direct equivalent in common tools. Nearest: k-NN on features, but witness cohort explicitly excludes graph neighbors. | `0.2.1` |
+| FDR-controlled detection | `find_anomalies(fdr_alpha=0.05)` applies Benjamini-Hochberg to rank-based p-values of `\|\|delta\|\|`. Returns per-entity q-values. | At alpha=0.05, at most 5% of flagged entities are false discoveries. Statistical guarantee instead of heuristic thresholding. | BH correction exists in biostatistics, but not combined with geometric anomaly detection in one system. | `0.3.1` |
+| Regime change detection | `attract_regime_change` computes per-bucket centroids, flags intervals where centroid shift exceeds `mean + 2*std`. Self-calibrating. | Detects shifts in population structure, not just individual entity drift — systemic changes visible. | Evidently/NannyML monitor model prediction drift, not coordinate-space population structure. | `0.1.0` |
+| Diverse anomaly selection | `find_anomalies(select="diverse")` uses facility location (Nemhauser-Wolsey-Fisher) to cover distinct anomaly regions. | Returns representative anomalies across the anomaly space, not redundant extremes from one cluster. | Top-N by score returns extremes from the same region. | `0.3.1` |
+| Distribution-aware scoring | Each dimension gets a `kind` tag (gaussian/poisson/bernoulli). Anomaly score uses the matching Bregman divergence per dimension — KL for counts, KL for binary, squared z-score for continuous. Contributions are exactly additive (Pythagorean property). | A count dimension with value 1 vs population mean 50 is scored differently than a continuous amount with the same z-score. Attribution sums to 100% — no approximation. | PyOD/sklearn: uniform distance metric across all features. SHAP: approximate, model-dependent attribution. | `0.4.0` |
+| Anomaly confidence | `anomaly_confidence: 0-1` per entity via bootstrap resampling. Measures how stable the anomaly verdict is if the population composition changes. `min_confidence` filter on `find_anomalies`. | Agent distinguishes solid anomalies (confidence=0.95) from borderline cases (confidence=0.3) without manual threshold exploration. | No equivalent in standard anomaly detectors — binary verdict with no stability signal. | `0.4.0` |
+| In-memory graph index | All graph operations use a lazily-built adjacency index with O(1) neighbor lookups. Cached on the session reader after first graph call. | `find_counterparties`, `discover_chains`, `contagion_score`, `find_geometric_path` run without per-call disk reads — interactive latency on million-edge graphs. | Neo4j/NetworkX: separate graph DB or in-memory load. Here the index is part of the sphere session. | `0.4.0` |
+
+### What changes in practice
+
+The same problems look different when graph, time, and statistics are unified:
+
+| Problem | Typical approach | With hypertopos |
+|---|---|---|
+| Detect anomalies | Train model, engineer features, choose contamination rate, retrain on shift | `hypertopos build` from YAML, `find_anomalies()` — threshold auto-calibrated from population |
+| Explain an anomaly | SHAP on trained model — feature importance for latent dimensions | `explain_anomaly(entity)` — ranked real dimensions: `loan_count: +3.2 sigma (42%)` |
+| Compare across domains | Align schemas, build shared features, normalize units | Compare `\|\|delta\|\|` directly — 4.2 sigma means the same in any sphere |
+| Track behavioral drift | Export to time-series DB, build dashboard, set manual thresholds | `attract_drift(window)` — displacement, path length, directionality per entity |
+| Validate anomaly is real | Manual investigation, ask domain expert | `find_witness_cohort(entity)` — similar non-connected entities confirming the pattern |
+| Understand propagation | PageRank, manual path tracing, cross-table joins | `propagate_influence(source)` — Cohen's d between connected vs control group |
+| Trust an anomaly verdict | Re-run with different thresholds, manual sensitivity analysis | `find_anomalies(min_confidence=0.8)` — only entities stable under population perturbation |
+| Understand why anomalous | SHAP on black-box model, approximate feature importance | `explain_anomaly` — per-dimension Bregman contribution with distribution kind, sums to 100% |
+
+## Benchmarks
+
+Validated on three domains with the same engine, zero domain rules, zero labels:
+
+| Domain | Dataset | Key result |
+|--------|---------|------------|
+| Banking | Berka (Czech, real data) | 85.5% recall on loan defaults |
+| AML | IBM AML (synthetic) | 80.4% recall, zero labels |
+| Transport | NYC Yellow Taxi (7.6M trips) | 8/8 anomaly categories detected |
+
+Benchmark scripts and data preparation are included. Results are reproducible. Numbers are from the pre-0.1.0 validation run and have not been re-evaluated against recent releases.
+
+Full results: [Benchmarks](docs/benchmarks.md)
+
+## Documentation
+
+| | |
+|---|---|
+| [Introduction](docs/introduction.md) | The idea and where it stands |
+| [Quick Start](docs/quickstart.md) | Install, build, navigate |
+| [Core Concepts](docs/concepts.md) | Mathematical foundation |
+| [Configuration](docs/configuration.md) | Sphere builder YAML reference |
+| [API Reference](docs/api-reference.md) | Python API |
+| [Data Format](docs/data-format.md) | On-disk storage format |
+| [Architecture](docs/architecture.md) | Package layers and design |
+
+## Status
+
+Research-stage project. Working code, reproducible benchmarks, active development. API may change.
+
+## License
+
+[Business Source License 1.1](LICENSE.md). Free for internal use, development, testing, and research. See LICENSE.md for details.

hypertopos-0.4.0/ROADMAP.md

@@ -0,0 +1,83 @@
+# hypertopos — Roadmap
+
+> Planned direction. Priorities may change based on feedback.
+
+## Plan: 0.5.0
+
+Investigation depth, detection quality, edge-derived geometry, and advanced analytics.
+
+- **Lazy chain geometry** — on-demand chain delta vectors via sampled population calibration; `discover_chains(include_geometry=True)` supplements build-time `chain_lines` for per-entity investigation without rebuild cost.
+- **Multi-hop anomaly explanation chains** — automate the trace-to-root-cause workflow: anomalous entity → top dimension → edge entity → hub check → neighbor check → explanation chain with per-hop evidence. Collapses 4–6 tool calls into one.
+- **Geometric edge potential** — relationship anomaly scoring via endpoint geometric distance; flags edges connecting geometrically distant entities.
+- **Geometric dark matter** — population density gap detection; identifies coordinate-space regions that should have entities but are empty (anomaly by absence).
+- **Gradient alignment on drift** — drift direction signal (normalizing vs deteriorating) on `find_drifting_entities` and `dive_solid`.
+- **Edge-derived dimensions on event patterns** — pair-level and node-derived features computed at build time as geometric dimensions.
+- **`find_motif` — structural pattern matching on edge table** — per-hop predicates for subgraph patterns invisible to flat queries.
+- **Advanced coordinate space analytics** — new methods leveraging properties unique to population-relative coordinate construction.
+
+---
+
+## 0.4.0
+
+Bregman geometry, anomaly confidence, distribution-aware calibration, build pipeline, graph acceleration.
+
+- **0.4.0** — Distribution-aware Bregman divergence with per-dimension kind tags. Per-dimension anomaly threshold (hyper-ellipsoid). Theta calibration fix. Anomaly confidence via stratified bootstrap. Interpretive MCP output (`explain_anomaly` returns Bregman contributions). Sphere format 2.3. Per-pattern pipeline parallelism (geometry → temporal overlap, up to 4 threads). In-memory adjacency index — full graph in dict, O(1) neighbor lookups replacing Lance BTREE scans across all graph operations. Temporal bisect in chain extraction.
+- **0.4.x** — Native graph algorithms as dimensions — PageRank, community detection (label propagation), betweenness centrality, triangle count, connected components computed at build time via adjacency index. Continuous features as derived dimensions, categorical features as group_by_property.
+
+---
+
+## 0.3.0
+
+Lance perf upgrade, FDR control, builder intelligence.
+
+- **0.3.0** — Aggregate engine rewritten around Lance SQL, precomputed contagion stats, format 2.2, edge table auto-detect fix.
+- **0.3.1** — Benjamini-Hochberg FDR control, submodular facility location, vectorized build with adaptive memory chunking.
+- **0.3.2** — NumPy graph features, chunked pre-computation, Lance compact tuning, per-dim index removal. Generalized dimension blocks (g/t/s), geometric heredity (`find_novel_entities`).
+- **0.3.3** — Agent navigation policy (investigation memory, failure guards, decision scoring). Dimension-selective similarity (`dim_mask`, `metric="cosine"`, `metric="Linf"`). Sphere-specific hardcoding removed from 6 skill guidance texts.
+
+---
+
+## 0.2.0
+
+Graph meets geometry — edge table, runtime traversal, contagion/influence.
+
+- **0.2.0** — Edge table, +11 navigator functions, contagion/influence primitives.
+- **0.2.1** — Witness cohort discovery, investigative peer ranking.
+- **0.2.2** — As-of graph reconstruction, `detect_cross_pattern_discrepancy` latency fix.
+
+---
+
+## 0.1.0
+
+First public release — full GDS stack, π1–π12, builder, MCP server, validation suite.
+
+---
+
+## Future
+
+**Detection quality**
+- Edge-derived dimensions + temporal motif matcher
+- Confidence scoring, robust estimators, multi-scale resolution — improve anomaly precision and reduce false positives on heavy-tail and multi-modal populations
+
+**Builder evolution**
+- Incremental rebuild — geometry-only without `--force` wipe
+
+**PassiveScanner evolution**
+- Native temporal source support for direct temporal inputs, without requiring manual dataset plumbing in benchmark scripts
+- Optional weighted scoring mode that uses continuous intensity instead of binary counts
+- **SphereProfiler** — autonomous sphere scanner that profiles all patterns, runs calibration sweeps across source combinations, proposes optimal PassiveScanner composition for Layer 1 surveillance.
+
+**Code refactoring**
+- Break up oversized modules into smaller, domain-focused components
+- Reduce coupling between core layers
+- Consolidate repeated orchestration logic into shared helpers
+
+**Cross-sphere capabilities**
+- Cross-sphere comparison — dimensionless metrics across independently calibrated coordinate spaces
+- What-if analysis — hypothetical edge changes producing modified coordinate vectors
+
+**Enterprise / governance**
+- Dimension access control — per-agent visibility constraints on delta dimensions
+
+**Tooling**
+- Runtime latency benchmarks in package docs

{hypertopos-0.3.2 → hypertopos-0.4.0}/SECURITY.md

@@ -1,6 +1,6 @@
 # Security
 
-## Current version: 0.3.2
+## Current version: 0.4.0
 
 hypertopos runs locally. No network services, no auth layer, no multi-tenancy.
 

{hypertopos-0.3.2 → hypertopos-0.4.0}/docs/api-reference.md

@@ -103,7 +103,7 @@ with sphere.session("agent-1") as session:
 
 | Primitive | Method | What it does | Returns |
 |-----------|--------|--------------|---------|
-| π5 | `π5_attract_anomaly(pattern_id, radius, top_n, fdr_alpha, fdr_method, select)` | Find most anomalous polygons | `(list[Polygon], int, list, dict)` |
+| π5 | `π5_attract_anomaly(pattern_id, radius, top_n, fdr_alpha, fdr_method, select, min_confidence)` | Find most anomalous polygons | `(list[Polygon], int, list, dict)` |
 | π6 | `π6_attract_boundary(alias_id, pattern_id, direction, top_n, fdr_alpha, fdr_method, select)` | Find entities nearest to alias cutting plane | `list[(Polygon, float)]` |
 | π7 | `π7_attract_hub(pattern_id, top_n, line_id_filter, fdr_alpha, fdr_method, select)` | Find entities with highest connectivity | `list[(str, int, float)]` |
 | π7+ | `π7_attract_hub_and_stats(pattern_id, top_n, line_id_filter)` | Hub ranking + population hub score statistics in one scan | `(list, dict)` |
@@ -141,9 +141,9 @@ with sphere.session("agent-1") as session:
 
 | Method | Description |
 |--------|-------------|
-| `explain_anomaly(primary_key, pattern_id)` | Structured investigation: severity, witness set, repair set, conformal p-value, reputation |
+| `explain_anomaly(primary_key, pattern_id)` | Structured investigation: severity, witness set, repair set, conformal p-value, reputation, and Bregman contributions returned under `top_dimensions[]` (fields: `dim`, `kind`, `bregman`, `pct_of_total`) |
 | `explain_anomaly_chain(primary_key, pattern_id, max_hops)` | Trace anomaly propagation through geometric neighbors |
-| `find_similar_entities(primary_key, pattern_id, top_n)` | ANN search for nearest entities in delta-space. Returns `SimilarityResult` |
+| `find_similar_entities(primary_key, pattern_id, top_n, dim_mask, metric)` | ANN search for nearest entities in delta-space. `dim_mask`: list of dimension names to restrict distance. `metric`: `"L2"` or `"cosine"`. Returns `SimilarityResult` |
 | `contrast_populations(pattern_id, group_a, group_b)` | Dimension-by-dimension comparison of two entity groups (Cohen's d) |
 | `composite_risk(primary_key, line_id)` | Fisher's method combination of conformal p-values across patterns |
 | `composite_risk_batch(primary_keys, line_id)` | Batch Fisher combination for multiple entities |
@@ -213,9 +213,9 @@ builder = GDSBuilder(
 | `add_graph_features(anchor_line, event_line, from_col, to_col, features)` | Auto-compute graph structural features (in/out-degree, reciprocity) |
 | `add_chain_line(line_id, chains, features)` | Create anchor line from extracted chain dicts |
 | `add_alias(alias_id, base_pattern_id, cutting_plane_dimension, cutting_plane_threshold)` | Register an alias with a cutting plane for sub-population analysis |
-| `build()` | Validate, compute statistics, write all files. Returns output path |
+| `build(temporal_configs=None)` | Validate, compute statistics, write all files. Pass `temporal_configs` to run geometry→temporal pipeline per pattern. Returns output path |
 | `incremental_update(pattern_id, changed_entities, deleted_keys)` | Update geometry incrementally with drift tracking |
-| `build_temporal(time_col, time_window)` | Generate temporal snapshots from time-windowed event data. Call after `build()` |
+| `build_temporal(time_col, time_window)` | Generate temporal snapshots from time-windowed event data. Call after `build()` when not using pipeline mode |
 
 ### RelationSpec
 
@@ -304,6 +304,8 @@ Key methods: `current_version()`, `has_fts()`.
 | `is_anomaly` | `bool` | Whether delta_norm >= theta_norm |
 | `edges` | `list[Edge]` | All edges in this polygon |
 | `delta_rank_pct` | `float \| None` | Percentile rank within population (0-100) |
+| `bregman_divergence` | `float \| None` | Per-entity Bregman divergence score — distribution-aware anomaly distance computed per dimension using its `kind` tag. `None` on pre-2.3 spheres. |
+| `anomaly_confidence` | `float \| None` | Bootstrap stability score (0–1): fraction of bootstrap samples in which the entity is classified as anomalous. `None` when bootstrap was skipped (N > 50K, `group_by_property`, or `use_mahalanobis`). |
 
 Key methods: `is_event()`, `is_anchor()`, `alive_edges()`, `edges_for_line(line_id)`.
 
@@ -343,6 +345,7 @@ Key methods: `slice_at(timestamp)` -- binary search for the slice active at a gi
 | `version` | `int` | Pattern version |
 | `prop_columns` | `list[str]` | Boolean property columns tracked as dimensions |
 | `dimension_weights` | `np.ndarray \| None` | Per-dimension importance weights |
+| `dimension_kinds` | `list[str] \| None` | Per-dimension distribution family: `"gaussian"`, `"poisson"`, or `"bernoulli"`. Populated at build time; `None` on pre-2.3 spheres. |
 
 Key properties: `theta_norm`, `dim_labels`, `delta_dim()`, `is_continuous`, `max_hub_score`.
 

{hypertopos-0.3.2 → hypertopos-0.4.0}/docs/benchmark/nyc-taxi/sphere/prepare_trips.py

@@ -79,6 +79,16 @@ def _prepare_trips() -> pa.Table:
     )
     raw = raw.append_column("tip_pct", tip_pct)
 
+    # Filter out-of-range timestamps (TLC data contains rare bad dates 2008–2088)
+    _pickup_ts = raw["tpep_pickup_datetime"]
+    _ts_min = pa.scalar(1546300800000000, pa.int64())  # 2019-01-01 00:00 UTC in us
+    _ts_max = pa.scalar(1548979200000000, pa.int64())  # 2019-02-01 00:00 UTC in us
+    _pickup_int = pc.cast(_pickup_ts, pa.int64())
+    raw = raw.filter(pc.and_(
+        pc.greater_equal(_pickup_int, _ts_min),
+        pc.less(_pickup_int, _ts_max),
+    ))
+
     # Cast to timestamp without tz (Windows tzdata workaround)
     pickup_no_tz = raw["tpep_pickup_datetime"].cast(pa.timestamp("us"))
     # Pickup hour and day of week

{hypertopos-0.3.2 → hypertopos-0.4.0}/docs/concepts.md

@@ -68,6 +68,53 @@ Two opt-in post-processing stages can be applied to any attract primitive that s
 
 Both stages compose: `fdr_alpha=0.05, select="diverse"` first filters to FDR-controlled discoveries, then picks the K most geometrically diverse among them.
 
+## Bregman Divergence
+
+Standard anomaly scoring in GDS uses `‖δ‖₂` — a single Euclidean distance that treats every dimension identically. This is fast and works well when dimensions are roughly Gaussian.
+
+**Bregman divergence** generalises the scoring: each dimension contributes an anomaly score computed from the generator function matched to its distribution family (`kind`). For a dimension with kind `gaussian`, the generator is squared loss (identical to L2). For `poisson`, it is the Kullback-Leibler divergence for counts. For `bernoulli`, it is binary cross-entropy.
+
+The per-dimension Bregman scores are summed to produce `bregman_divergence` — a single float stored alongside `delta_norm` in the geometry file. Dimensions with a poor match between their assumed family and actual data shape produce disproportionately large Bregman terms, making them visible in `explain_anomaly` output.
+
+**What this enables:**
+- Binary FK dimensions (bernoulli) no longer inflate the total score when an entity is merely "active" in that relation — the scoring accounts for the 0/1 nature of the edge
+- Count-based derived features (poisson) are scored against their expected arrival rate rather than a z-score
+- `explain_anomaly` returns per-dimension Bregman contributions with `kind` and `pct_of_total`, identifying which distribution mismatch is the primary driver
+
+`bregman_divergence` is stored in sphere format ≥ 2.3. Pre-2.3 spheres return `None` for this field.
+
+### Dimension Kinds
+
+Every dimension in a pattern carries a `kind` tag — the distribution family used for Bregman scoring. The builder auto-detects kinds at build time using the following rules:
+
+| Dimension source | Inferred kind |
+|-----------------|---------------|
+| Binary FK (`edge_max = null`) | `bernoulli` |
+| `count`, `count_distinct`, `count:window=*` | `poisson` |
+| `sum:col`, `avg:col`, `std:col`, `iet_*` | `gaussian` |
+| `max:col`, `min:col` | `gaussian` |
+| `precomputed_dimension` with `edge_max: 1` | `bernoulli` |
+| `precomputed_dimension`, otherwise | `gaussian` |
+| `graph_features` in/out-degree (`edge_max > 1`) | `poisson` |
+| `graph_features` reciprocity, overlap (`edge_max = 1`) | `bernoulli` |
+
+Override via `kind:` on individual dimension entries in `sphere.yaml`. The resulting kind list is stored in `Pattern.dimension_kinds` and surfaced in `sphere_overview` as `dimension_kinds`.
+
+## Anomaly Confidence
+
+Bootstrap confidence measures how stable an entity's anomaly classification is under random resampling of the calibration population.
+
+The builder draws `bootstrap_iterations` bootstrap samples from the population (default 200), recomputes mu/sigma/theta on each sample, and records whether the entity's delta_norm exceeds the resampled theta. `anomaly_confidence` is the fraction of samples in which the entity is classified as anomalous.
+
+**Interpretation:**
+- `1.0` — entity is anomalous under every bootstrap resample; classification is robust
+- `0.5–0.99` — entity sits near the boundary; classification may flip with slightly different calibration data
+- `< 0.5` — entity is nominally anomalous (delta_norm > theta) but unstable; treat with caution
+
+`anomaly_confidence` is skipped when the pattern uses `group_by_property` (bootstrap is per-group, cost grows multiplicatively), `use_mahalanobis` (full covariance bootstrap is expensive), or N > 50K entities. In those cases the field is `None`.
+
+Filter `find_anomalies` by minimum confidence using `min_confidence` — for example `min_confidence=0.8` returns only entities whose anomaly classification is stable across at least 80% of bootstrap samples.
+
 ## Three Scales
 
 ![Three Scales](images/three-scales.svg)
@@ -172,6 +219,9 @@ Chains (both build-time `chain_lines` and runtime `discover_chains`) are sequenc
 | **Delta norm** | L2 magnitude of the deviation -- basis for similarity, clustering, and anomaly scoring |
 | **Theta threshold** | Statistical boundary derived from the population distribution (used for anomaly classification) |
 | **Deformation log** | History of changes that produced the current solid |
+| **Bregman divergence** | Distribution-aware anomaly distance summed across per-dimension Bregman terms |
+| **Anomaly confidence** | Bootstrap stability score (0–1) for anomaly classification under population resampling |
+| **Dimension kind** | Distribution family tag per dimension (`gaussian`, `poisson`, `bernoulli`) driving Bregman scoring |
 
 Every dimension has a meaning tied to a relation or tracked property.
 

{hypertopos-0.3.2 → hypertopos-0.4.0}/docs/configuration.md

@@ -361,6 +361,7 @@ patterns:
 | `graph_features` | dict | `null` | Auto-compute graph structural features from event from/to columns. |
 | `description` | string | `null` | Human-readable description. Stored in sphere.json — visible to agents via `get_sphere_info`. |
 | `edge_table` | dict | `null` | Explicit edge table config (see below). Auto-emitted for event patterns with 2+ FK relations to same anchor line. |
+| `bootstrap_iterations` | int | `200` | Number of bootstrap resamples used to compute `anomaly_confidence` per entity. Skipped automatically for N > 50K, `group_by_property`, or `use_mahalanobis`. |
 
 ### Edge Table
 
@@ -529,6 +530,44 @@ derived_dimensions:
 IET requires a `temporal` config for the pattern (provides `timestamp_col`).
 Temporal column is auto-resolved from the temporal config — no explicit time_col needed.
 
+**Dimension `kind` on derived dimensions:** Each derived feature can carry an explicit `kind` tag to override auto-detection. Add it as an extra property alongside the feature spec:
+
+```yaml
+derived_dimensions:
+  - from_pattern: tx_pattern
+    features:
+      - tx_count: count
+        kind: poisson           # override auto-detection
+      - total_spend: sum:amount
+        kind: gaussian
+      - has_foreign: count_distinct:country
+        kind: bernoulli         # treat as 0/1 presence indicator
+```
+
+When omitted, `kind` is auto-detected from the metric type — see below.
+
+---
+
+### Dimension Kind Auto-Detection
+
+The builder assigns a `kind` tag to every dimension at build time. The tag drives Bregman divergence scoring: each dimension contributes an anomaly score calibrated to its own distribution family rather than a single global metric.
+
+**Auto-detection rules:**
+
+| Dimension source | Inferred kind | Rationale |
+|-----------------|---------------|-----------|
+| Binary FK (`edge_max = null`) | `bernoulli` | 0/1 presence indicator |
+| `count`, `count_distinct`, `count:window=*` | `poisson` | Event counts follow Poisson arrival models |
+| `sum:col`, `avg:col`, `std:col`, `iet_*` | `gaussian` | Continuous magnitudes |
+| `max:col`, `min:col` | `gaussian` | Extremal statistics |
+| `precomputed_dimension` with `edge_max: 1` | `bernoulli` | Binary column indicator |
+| `precomputed_dimension`, otherwise | `gaussian` | Default continuous |
+| `graph_features` (`in_degree`, `out_degree`) | `poisson` | Degree counts (`edge_max > 1`) |
+| `graph_features` (`reciprocity`, `counterpart_overlap`) | `bernoulli` | Binary presence (`edge_max = 1`) |
+| `geo_properties`, `metric_properties` | `gaussian` | Continuous measurements |
+
+**Override via `kind:`** on individual dimension entries (precomputed, derived). Override takes precedence over auto-detection. Event dimensions use auto-detection only.
+
 ---
 
 ## Composite Lines
@@ -595,6 +634,7 @@ patterns:
 | `edge_max` | int or `"auto"` | `"auto"` | Fixed normalization cap, or auto-compute from percentile. |
 | `percentile` | float | `99.0` | Percentile for auto edge_max computation. |
 | `display_name` | string | `null` | Human-readable dimension label. |
+| `kind` | string | auto | Distribution family for Bregman scoring: `"gaussian"`, `"poisson"`, or `"bernoulli"`. Auto-detected when omitted. |
 
 ---
 
@@ -904,8 +944,9 @@ hypertopos build --config sphere.yaml --force --verbose
 
 The builder parallelizes where possible:
 - **Points write** — parallel across lines (ThreadPoolExecutor, up to 4 workers)
-- **Geometry build** — parallel across patterns (one thread per pattern)
-- **Temporal build** — parallel across patterns; adaptive memory chunking when the shape tensor exceeds available RAM (auto-detected via platform API, 4 GB fallback)
+- **Pipeline mode** (when `temporal:` is configured) — each pattern runs geometry → temporal as a sequential pipeline, with up to 4 patterns executing concurrently. Temporal for a pattern starts immediately after its geometry completes, without waiting for other patterns.
+- **Geometry-only mode** (no `temporal:` or `--no-temporal`) — geometry runs in parallel across patterns (up to 4 threads)
+- **Temporal adaptive chunking** — when the shape tensor exceeds available RAM, the temporal phase uses adaptive memory chunking (auto-detected via platform API, 4 GB fallback)
 
 ### Build timing
 
@@ -915,15 +956,11 @@ Source 'transactions': 5,078,345 rows, 18 cols
 Source 'accounts': 515,080 rows, 11 cols
 Sources total: 0.6s
 Extracting chains for 'tx_chains'...
-  Seeds: 109,718
-  Chains cached to chains_tx_chains_ed008d26354b42ee.pkl
-  Chains extracted: 300,000
-Chains total: 336s
-Building geometry...
-Geometry total: 263s
-Building temporal for 'account_pattern' (2d windows)...
-Temporal total: 305s
-Build total: 527s
+  Chains extracted: 290,784
+Chains total: 9.5s
+Building geometry + temporal (pipeline)...
+Geometry + temporal total: 304.9s
+Build total: 315.5s
 ```
 
 ### Skip flags (iterative build)
@@ -945,10 +982,17 @@ hypertopos build --config sphere.yaml --force --verbose
 
 | Sphere | First run | Cached run |
 |--------|-----------|-----------|
-| Berka (4.5K accounts, 1M tx) | 24s | 24s |
-| TPCH SF1 (150K cust, 6M tx) | 2.9 min | 2.9 min |
-| AML HI-Small (515K accts, 5M tx, 300K chains) | ~22 min | ~9.5 min |
-| AML LI-Small (712K accts, 7M tx, 300K chains) | ~13.4 min | ~9.4 min |
+| Berka (4.5K accounts, 1M tx) | ~35s | ~25s |
+| NYC Taxi (265 zones, 7.5M trips) | ~3.2 min | ~2.8 min |
+| AML HI-Small (515K accts, 5M tx, 300K chains) | ~5.3 min | ~6.7 min |
+
+### Benchmark sphere configurations
+
+| Sphere | Edge table | Chains | Temporal | Composites | Key features |
+|--------|-----------|--------|----------|------------|--------------|
+| Berka | none | none | 3 patterns (90d) | account×bank pairs | NB-Split (accounts + accounts_stress + loan_accounts), precomputed dims, group_by_property |
+| NYC Taxi | auto-detected (trip_pattern: pu_zone → do_zone) | none | 2 patterns (7d) | none | NB-Split (zones_pickup + zones_dropoff), 6 event dims, 2 anchor patterns |
+| AML HI-Small | explicit (tx_pattern: from_account → to_account, timestamp, amount) | 300K (seeded BFS) | 3 patterns (2d) | account pairs | graph_features, precomputed ratios, IET metrics, Mahalanobis |
 
 ### Parallel chain extraction
 

{hypertopos-0.3.2 → hypertopos-0.4.0}/docs/data-format.md

@@ -109,6 +109,9 @@ Domain columns vary per line. The `primary_key` column is always present and alw
 | `delta` | fixed_size_list\<float32\> | Z-scored delta vector (deviation from mu) |
 | `delta_norm` | float32 | L2 norm of delta (distance from population center -- used for scoring, clustering, and similarity) |
 | `delta_rank_pct` | float32 | Percentile rank of delta_norm (0--100) |
+| `conformal_p` | float32 | Conformal p-value (null when not computed) |
+| `bregman_divergence` | float32 | Sum of per-dimension Bregman terms (distribution-aware anomaly distance). Null on pre-2.3 spheres. |
+| `anomaly_confidence` | float32 | Bootstrap stability score: fraction of bootstrap resamples in which entity is anomalous. Null when bootstrap was skipped. |
 | `edges` | list\<struct\> | Edge list: line_id, point_key, direction, status |
 
 The `delta` vector length equals the number of dimensions in the pattern. Geometry datasets carry an IVF-PQ ANN index on the `delta` column for trajectory similarity search.