hypertopos 0.3.3__tar.gz → 0.4.1__tar.gz

{hypertopos-0.3.3 → hypertopos-0.4.1}/CHANGELOG.md

@@ -7,6 +7,43 @@ Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.4.1] — 2026-04-16
+
+### Added
+- Native graph algorithms as geometry dimensions: `pagerank`, `connected_component`, `clustering_coefficient`, `community` (label propagation), `betweenness` (sampled Brandes). Computed at build time on the adjacency index — no external dependencies. Add to `graph_features.features` list in sphere YAML.
+
+### Fixed
+- `find_geometric_path` rewritten from beam search to bidirectional BFS for reliable path discovery in sparse graphs.
+
+## [0.4.0] — 2026-04-15
+
+### Added
+- Distribution-aware Bregman divergence with per-dimension kind tags (gaussian/poisson/bernoulli). Auto-detected from dimension type; YAML `kind:` override per dimension.
+- Per-dimension anomaly threshold (hyper-ellipsoid boundary) replacing uniform decomposition (hyper-sphere).
+- `anomaly_confidence` per entity via stratified bootstrap resampling. Configurable `bootstrap_iterations` (default 200, 0 to skip). Skipped for populations > 50K (use `conformal_p`), `group_by_property`, and `use_mahalanobis` patterns.
+- `bregman_divergence` stored alongside `delta_norm` in geometry.
+- `dimension_kinds` per pattern in sphere.json metadata.
+- `find_anomalies` returns edges on polygons (both fast path and in-process path).
+- `find_neighborhood` raises `GDSNavigationError` for continuous-mode patterns instead of silently returning empty results.
+- In-memory adjacency index for graph operations — lazy build on first graph call, cached on session reader.
+- Build pipeline per-pattern parallelism: geometry → temporal as concurrent pipeline (up to 4 threads).
+
+### Changed
+- `explain_anomaly` returns per-dimension Bregman contributions (additive, kind-labeled) instead of absolute z-score deltas when dimension_kinds available.
+- Sphere format bumped to 2.3. Old spheres must be rebuilt.
+- Chain extraction uses temporal bisect for neighbor filtering instead of linear timestamp scan.
+
+### Fixed
+- `find_anomalies` pagination: deterministic ordering on tied `delta_norm` values prevents entity duplication across page boundaries.
+- `hub_score_history` clamps negative `hub_score`/`alive_edges_est` to zero for early temporal slices.
+- `anomalous_edges` deduplicates by `event_key` for self-loops (`from_key == to_key`).
+- `find_clusters`, `find_drifting_entities`, `compare_time_windows`, `find_regime_changes` use `pattern.dim_labels` for dimension names, including event dimensions.
+- `find_geometric_path` default `beam_width` increased from 10 to 50; auto-scales for deep searches.
+- `aggregate` raises explicit error when `group_by_line` matches multiple relations on the same pattern.
+- `search_entities` JSON serialization handles datetime columns.
+- `sphere_overview` mode detection uses `sample_size=200` instead of full geometry table read.
+- Weighted Bregman norms for `metric="bregman"` in `find_anomalies`.
+
 ## [0.3.3] — 2026-04-13
 
 ### Added

{hypertopos-0.3.3 → hypertopos-0.4.1}/LICENSE.md

@@ -26,7 +26,7 @@ For purposes of this License:
 - **Internal use** within an organization is not a competitive offering. Your
   organization includes entities under common control.
 
-**Change Date:** 2029-04-13
+**Change Date:** 2029-04-16
 
 **Change License:** Apache License, Version 2.0
 

{hypertopos-0.3.3 → hypertopos-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hypertopos
-Version: 0.3.3
+Version: 0.4.1
 Summary: Understand the structure of your data — without training machine learning models
 Project-URL: Homepage, https://github.com/hypertopos/hypertopos-py
 Project-URL: Documentation, https://github.com/hypertopos/hypertopos-py/tree/main/docs
@@ -15,10 +15,13 @@ Requires-Dist: pyarrow>=14.0
 Requires-Dist: pylance<5.0,>=4.0
 Requires-Dist: pyyaml>=6.0
 Provides-Extra: dev
+Requires-Dist: igraph>=0.11; extra == 'dev'
 Requires-Dist: mypy; extra == 'dev'
 Requires-Dist: pytest; extra == 'dev'
 Requires-Dist: pytest-cov; extra == 'dev'
 Requires-Dist: ruff; extra == 'dev'
+Provides-Extra: graph
+Requires-Dist: igraph>=0.11; extra == 'graph'
 Provides-Extra: mcp
 Requires-Dist: mcp[cli]>=1.20; extra == 'mcp'
 Requires-Dist: tzdata>=2024.1; extra == 'mcp'

hypertopos-0.4.1/README.md

@@ -0,0 +1,103 @@
+# hypertopos
+
+> A behavioral feature layer for graph and temporal data — turning behavior into coordinates, trajectories, and explanations.
+
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+[![License: BSL 1.1](https://img.shields.io/badge/license-BSL%201.1-orange.svg)](LICENSE.md)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19482069.svg)](https://doi.org/10.5281/zenodo.19482069)
+[![PyArrow](https://img.shields.io/badge/format-PyArrow-red.svg)](https://arrow.apache.org/docs/python/)
+[![Lance](https://img.shields.io/badge/storage-Lance-blueviolet.svg)](https://github.com/lance-format/lance)
+[![MCP](https://img.shields.io/badge/MCP-compatible-green.svg)](https://modelcontextprotocol.io)
+[![Version](https://img.shields.io/badge/version-0.4.0-%235500FF.svg)](pyproject.toml)
+
+hypertopos is not a database, and not a machine learning model. It is a layer that turns relational data into a coordinate system where every entity gets a position derived from its relationships and the population around it.
+
+```
+typical:    data → features → feature store → ML → decision
+hypertopos: data → representation (hypertopos) → ML / decision
+```
+
+```bash
+pip install hypertopos
+```
+
+![hypertopos overview](docs/images/hypertopos-overview.svg)
+
+## How it works
+
+You describe your data in YAML — entity types, sources, relationships. hypertopos computes population statistics and produces a sphere: pre-computed geometry stored in Apache Arrow format.
+
+Agents (or Python code) open the sphere and navigate it using twelve primitives that cover movement, clustering, anomaly detection, population comparison, and temporal analysis. Each step is stateful — where you are determines what you see next.
+
+For the full picture: [Introduction](docs/introduction.md) · [Core Concepts](docs/concepts.md) · [Quick Start](docs/quickstart.md)
+
+## What's different
+
+Each capability below emerges from treating entities as points in a shared, population-calibrated space.
+
+| Capability | What it does | Compared to | Since |
+|---|---|---|---|
+| Population-relative coordinates | `delta = (shape - mu) / sigma` — one coordinate for anomaly, clustering, drift | node2vec/GNN: latent dims, retraining on shift | `0.1.0` |
+| Self-calibrating threshold | `theta = percentile(norms, 95)` — no tuning, no labels | PyOD: choose contamination rate | `0.1.0` |
+| Named dimension attribution | `explain_anomaly` → `loan_count: +3.2σ (42%)`. Sums to 100% | SHAP/LIME: approximate, model-dependent | `0.1.0` |
+| Temporal deformation | Append-only delta log. Displacement, path length, directionality | Time-series DBs: metric values, not trajectories | `0.1.0` |
+| Stateful navigation | 12 typed primitives. Position type constrains valid ops | SQL/GraphQL: stateless queries | `0.1.0` |
+| Cross-sphere comparison | `\|\|delta\|\|` is dimensionless — 4.2σ means the same in any domain | Requires shared features or joint embeddings | `0.1.0` |
+| Counterfactual simulation | `simulate_edges` recomputes delta against fixed baseline | Causal inference: explicit DAG required | `0.1.0` |
+| Regime change detection | Per-bucket centroids, self-calibrating shift threshold | Evidently/NannyML: model prediction drift | `0.1.0` |
+| Graph contagion | Mean `\|\|delta\|\|` of neighbors. Cohen's d vs control group | PageRank: topology, not behavioral propagation | `0.2.0` |
+| Witness cohort | Similar deltas, NOT connected. Validates pattern vs one-off | k-NN on features (includes neighbors) | `0.2.1` |
+| FDR-controlled detection | Benjamini-Hochberg on rank p-values. Per-entity q-values | BH not combined with geometric detection | `0.3.1` |
+| Diverse anomaly selection | Facility location covers distinct anomaly regions | Top-N returns redundant extremes | `0.3.1` |
+| Distribution-aware scoring | Per-dim Bregman divergence (gaussian/poisson/bernoulli). Additive | PyOD/sklearn: uniform metric across all features | `0.4.0` |
+| Anomaly confidence | Bootstrap `anomaly_confidence: 0-1`. `min_confidence` filter | No equivalent — binary verdict, no stability signal | `0.4.0` |
+| Graph algorithm dimensions | PageRank, betweenness, community, clustering as geometry dims | Separate graph DB + manual joins | `0.4.1` |
+
+### What changes in practice
+
+The same problems look different when graph, time, and statistics are unified:
+
+| Problem | Typical approach | With hypertopos |
+|---|---|---|
+| Detect anomalies | Train model, engineer features, choose contamination rate, retrain on shift | `hypertopos build` from YAML, `find_anomalies()` — threshold auto-calibrated from population |
+| Explain an anomaly | SHAP on trained model — feature importance for latent dimensions | `explain_anomaly(entity)` — ranked real dimensions: `loan_count: +3.2 sigma (42%)` |
+| Compare across domains | Align schemas, build shared features, normalize units | Compare `\|\|delta\|\|` directly — 4.2 sigma means the same in any sphere |
+| Track behavioral drift | Export to time-series DB, build dashboard, set manual thresholds | `attract_drift(window)` — displacement, path length, directionality per entity |
+| Validate anomaly is real | Manual investigation, ask domain expert | `find_witness_cohort(entity)` — similar non-connected entities confirming the pattern |
+| Understand propagation | PageRank, manual path tracing, cross-table joins | `propagate_influence(source)` — Cohen's d between connected vs control group |
+| Trust an anomaly verdict | Re-run with different thresholds, manual sensitivity analysis | `find_anomalies(min_confidence=0.8)` — only entities stable under population perturbation |
+| Understand why anomalous | SHAP on black-box model, approximate feature importance | `explain_anomaly` — per-dimension Bregman contribution with distribution kind, sums to 100% |
+
+## Benchmarks
+
+Validated on three domains with the same engine, zero domain rules, zero labels:
+
+| Domain | Dataset | Key result |
+|--------|---------|------------|
+| Banking | Berka (Czech, real data) | 85.5% recall on loan defaults |
+| AML | IBM AML (synthetic) | 80.4% recall, zero labels |
+| Transport | NYC Yellow Taxi (7.6M trips) | 8/8 anomaly categories detected |
+
+Benchmark scripts and data preparation are included. Results are reproducible. Numbers are from the pre-0.1.0 validation run and have not been re-evaluated against recent releases.
+
+Full results: [Benchmarks](docs/benchmarks.md)
+
+## Documentation
+
+| | |
+|---|---|
+| [Introduction](docs/introduction.md) | The idea and where it stands |
+| [Quick Start](docs/quickstart.md) | Install, build, navigate |
+| [Core Concepts](docs/concepts.md) | Mathematical foundation |
+| [Configuration](docs/configuration.md) | Sphere builder YAML reference |
+| [API Reference](docs/api-reference.md) | Python API |
+| [Data Format](docs/data-format.md) | On-disk storage format |
+| [Architecture](docs/architecture.md) | Package layers and design |
+
+## Status
+
+Research-stage project. Working code, reproducible benchmarks, active development. API may change.
+
+## License
+
+[Business Source License 1.1](LICENSE.md). Free for internal use, development, testing, and research. See LICENSE.md for details.

{hypertopos-0.3.3 → hypertopos-0.4.1}/ROADMAP.md

@@ -2,28 +2,28 @@
 
 > Planned direction. Priorities may change based on feedback.
 
-## Plan: 0.4.0
+## Plan: 0.5.0
 
-Investigation depth + detection quality — new geometric primitives, agent trust, calibration.
+Investigation depth, detection quality, edge-derived geometry, and advanced analytics.
 
 - **Lazy chain geometry** — on-demand chain delta vectors via sampled population calibration; `discover_chains(include_geometry=True)` supplements build-time `chain_lines` for per-entity investigation without rebuild cost.
 - **Multi-hop anomaly explanation chains** — automate the trace-to-root-cause workflow: anomalous entity → top dimension → edge entity → hub check → neighbor check → explanation chain with per-hop evidence. Collapses 4–6 tool calls into one.
-- **Theta calibration fix** — builder writes correct theta_norm to sphere.json at build time; eliminates miscalibration on mixed-type patterns.
 - **Geometric edge potential** — relationship anomaly scoring via endpoint geometric distance; flags edges connecting geometrically distant entities.
 - **Geometric dark matter** — population density gap detection; identifies coordinate-space regions that should have entities but are empty (anomaly by absence).
 - **Gradient alignment on drift** — drift direction signal (normalizing vs deteriorating) on `find_drifting_entities` and `dive_solid`.
-- **Interpretive MCP output** — computed interpretation fields on tool responses (`passive_scan`, `find_similar_entities`, `dive_solid`, `find_regime_changes`) so agents stop misreading raw numbers.
-- **Anomaly confidence via bootstrap** — `anomaly_confidence: 0–1` per entity quantifying how stable the anomaly verdict is under population perturbation.
+- **Edge-derived dimensions on event patterns** — pair-level and node-derived features computed at build time as geometric dimensions.
+- **`find_motif` — structural pattern matching on edge table** — per-hop predicates for subgraph patterns invisible to flat queries.
+- **Advanced coordinate space analytics** — new methods leveraging properties unique to population-relative coordinate construction.
 
 ---
 
-## Plan: 0.5.0
+## 0.4.0
 
-Edge-derived geometry + advanced analytics.
+Bregman geometry, anomaly confidence, distribution-aware calibration, build pipeline, graph acceleration.
 
-- **Edge-derived dimensions on event patterns** — pair-level and node-derived features computed at build time as geometric dimensions.
-- **`find_motif` — structural pattern matching on edge table** — per-hop predicates for subgraph patterns invisible to flat queries.
-- **Advanced coordinate space analytics** — new methods leveraging properties unique to population-relative coordinate construction.
+- **0.4.0** — Distribution-aware Bregman divergence with per-dimension kind tags. Per-dimension anomaly threshold (hyper-ellipsoid). Theta calibration fix. Anomaly confidence via stratified bootstrap. Interpretive MCP output (`explain_anomaly` returns Bregman contributions). Sphere format 2.3. Per-pattern pipeline parallelism (geometry → temporal overlap, up to 4 threads). In-memory adjacency index — full graph in dict, O(1) neighbor lookups replacing Lance BTREE scans across all graph operations. Temporal bisect in chain extraction.
+- **0.4.1** — Native graph algorithms as geometry dimensions (igraph C backend): PageRank, betweenness centrality (edge-sampled), community detection (label propagation), clustering coefficient, connected components. Stress test fixes: find_anomalies edges and pagination, hub_score_history clamp, detect_pattern type guard, aggregate ambiguity guard, dim_labels in find_clusters.
+- **0.4.2** — Intrinsic/extrinsic decomposition (entity-caused vs population-caused position changes). Entity influence scoring (hidden influencer detection). Cross-pattern temporal lead-lag discovery. Bidirectional BFS for find_geometric_path (guaranteed path finding). Storey adaptive pi0 estimation for FDR.
 
 ---
 

{hypertopos-0.3.3 → hypertopos-0.4.1}/SECURITY.md

@@ -1,6 +1,6 @@
 # Security
 
-## Current version: 0.3.3
+## Current version: 0.4.1
 
 hypertopos runs locally. No network services, no auth layer, no multi-tenancy.
 

{hypertopos-0.3.3 → hypertopos-0.4.1}/docs/api-reference.md

@@ -103,7 +103,7 @@ with sphere.session("agent-1") as session:
 
 | Primitive | Method | What it does | Returns |
 |-----------|--------|--------------|---------|
-| π5 | `π5_attract_anomaly(pattern_id, radius, top_n, fdr_alpha, fdr_method, select)` | Find most anomalous polygons | `(list[Polygon], int, list, dict)` |
+| π5 | `π5_attract_anomaly(pattern_id, radius, top_n, fdr_alpha, fdr_method, select, min_confidence)` | Find most anomalous polygons | `(list[Polygon], int, list, dict)` |
 | π6 | `π6_attract_boundary(alias_id, pattern_id, direction, top_n, fdr_alpha, fdr_method, select)` | Find entities nearest to alias cutting plane | `list[(Polygon, float)]` |
 | π7 | `π7_attract_hub(pattern_id, top_n, line_id_filter, fdr_alpha, fdr_method, select)` | Find entities with highest connectivity | `list[(str, int, float)]` |
 | π7+ | `π7_attract_hub_and_stats(pattern_id, top_n, line_id_filter)` | Hub ranking + population hub score statistics in one scan | `(list, dict)` |
@@ -122,7 +122,7 @@ with sphere.session("agent-1") as session:
 
 | Method | Description |
 |--------|-------------|
-| `find_geometric_path(from_key, to_key, pattern_id, max_depth=5, beam_width=10, scoring="geometric")` | Beam search for paths between two entities via the edge table, scored by geometric coherence. Scoring modes: `"geometric"` (witness overlap + delta alignment + anomaly preservation), `"amount"` (geometric score modulated by log(transaction amount)), `"anomaly"` (prefer paths through anomalous entities), `"shortest"` (plain BFS, no geometric scoring). Returns best paths with per-hop scores |
+| `find_geometric_path(from_key, to_key, pattern_id, max_depth=5, beam_width=50, scoring="geometric")` | Bidirectional BFS for paths between two entities via the edge table, scored by geometric coherence. Scoring modes: `"geometric"` (witness overlap + delta alignment + anomaly preservation), `"amount"` (geometric score modulated by log(transaction amount)), `"anomaly"` (prefer paths through anomalous entities), `"shortest"` (plain BFS, no geometric scoring). Returns top `beam_width` paths by score |
 | `discover_chains(primary_key, pattern_id, time_window_hours=168, max_hops=10, min_hops=2, max_chains=100, direction="forward")` | Runtime temporal BFS on the edge table to discover entity chains from a starting point. Unlike `find_chains_for_entity()` which queries pre-computed chains, this performs live traversal -- works without build-time chain extraction |
 | `entity_flow(primary_key, pattern_id, top_n=20, *, timestamp_cutoff=None)` | Net flow analysis per counterparty via edge table. Two edge lookups (outgoing + incoming), sum amounts, compute per-counterparty net flow. `timestamp_cutoff` (Unix seconds) restricts to edges with `timestamp <= cutoff`. Returns outgoing/incoming totals, net_flow, flow_direction, counterparties sorted by abs(net_flow) |
 | `contagion_score(primary_key, pattern_id, *, timestamp_cutoff=None)` | Score how many of an entity's counterparties are anomalous via edge table + geometry check. `timestamp_cutoff` enables as-of contagion reconstruction. Returns score (0.0–1.0), total/anomalous counterparty counts |
@@ -141,7 +141,7 @@ with sphere.session("agent-1") as session:
 
 | Method | Description |
 |--------|-------------|
-| `explain_anomaly(primary_key, pattern_id)` | Structured investigation: severity, witness set, repair set, conformal p-value, reputation |
+| `explain_anomaly(primary_key, pattern_id)` | Structured investigation: severity, witness set, repair set, conformal p-value, reputation, and Bregman contributions returned under `top_dimensions[]` (fields: `dim`, `kind`, `bregman`, `pct_of_total`) |
 | `explain_anomaly_chain(primary_key, pattern_id, max_hops)` | Trace anomaly propagation through geometric neighbors |
 | `find_similar_entities(primary_key, pattern_id, top_n, dim_mask, metric)` | ANN search for nearest entities in delta-space. `dim_mask`: list of dimension names to restrict distance. `metric`: `"L2"` or `"cosine"`. Returns `SimilarityResult` |
 | `contrast_populations(pattern_id, group_a, group_b)` | Dimension-by-dimension comparison of two entity groups (Cohen's d) |
@@ -210,12 +210,12 @@ builder = GDSBuilder(
 | `add_derived_dimension(anchor_line, event_line, anchor_fk, metric, metric_col, dimension_name)` | Dimension derived from event aggregation (count, sum, max, std, mean) |
 | `add_composite_line(anchor_line, event_line, anchor_fk, ...)` | Create composite anchor line from event-anchor join |
 | `add_precomputed_dimension(anchor_line, dimension_name, edge_max)` | Dimension from a column already on the entity table |
-| `add_graph_features(anchor_line, event_line, from_col, to_col, features)` | Auto-compute graph structural features (in/out-degree, reciprocity) |
+| `add_graph_features(anchor_line, event_line, from_col, to_col, features)` | Auto-compute graph structural features (degree, reciprocity, pagerank, betweenness, community, clustering, components) |
 | `add_chain_line(line_id, chains, features)` | Create anchor line from extracted chain dicts |
 | `add_alias(alias_id, base_pattern_id, cutting_plane_dimension, cutting_plane_threshold)` | Register an alias with a cutting plane for sub-population analysis |
-| `build()` | Validate, compute statistics, write all files. Returns output path |
+| `build(temporal_configs=None)` | Validate, compute statistics, write all files. Pass `temporal_configs` to run geometry→temporal pipeline per pattern. Returns output path |
 | `incremental_update(pattern_id, changed_entities, deleted_keys)` | Update geometry incrementally with drift tracking |
-| `build_temporal(time_col, time_window)` | Generate temporal snapshots from time-windowed event data. Call after `build()` |
+| `build_temporal(time_col, time_window)` | Generate temporal snapshots from time-windowed event data. Call after `build()` when not using pipeline mode |
 
 ### RelationSpec
 
@@ -304,6 +304,8 @@ Key methods: `current_version()`, `has_fts()`.
 | `is_anomaly` | `bool` | Whether delta_norm >= theta_norm |
 | `edges` | `list[Edge]` | All edges in this polygon |
 | `delta_rank_pct` | `float \| None` | Percentile rank within population (0-100) |
+| `bregman_divergence` | `float \| None` | Per-entity Bregman divergence score — distribution-aware anomaly distance computed per dimension using its `kind` tag. `None` on pre-2.3 spheres. |
+| `anomaly_confidence` | `float \| None` | Bootstrap stability score (0–1): fraction of bootstrap samples in which the entity is classified as anomalous. `None` when bootstrap was skipped (N > 50K, `group_by_property`, or `use_mahalanobis`). |
 
 Key methods: `is_event()`, `is_anchor()`, `alive_edges()`, `edges_for_line(line_id)`.
 
@@ -343,6 +345,7 @@ Key methods: `slice_at(timestamp)` -- binary search for the slice active at a gi
 | `version` | `int` | Pattern version |
 | `prop_columns` | `list[str]` | Boolean property columns tracked as dimensions |
 | `dimension_weights` | `np.ndarray \| None` | Per-dimension importance weights |
+| `dimension_kinds` | `list[str] \| None` | Per-dimension distribution family: `"gaussian"`, `"poisson"`, or `"bernoulli"`. Populated at build time; `None` on pre-2.3 spheres. |
 
 Key properties: `theta_norm`, `dim_labels`, `delta_dim()`, `is_continuous`, `max_hub_score`.
 

{hypertopos-0.3.3 → hypertopos-0.4.1}/docs/benchmark/nyc-taxi/sphere/prepare_trips.py

@@ -79,6 +79,16 @@ def _prepare_trips() -> pa.Table:
     )
     raw = raw.append_column("tip_pct", tip_pct)
 
+    # Filter out-of-range timestamps (TLC data contains rare bad dates 2008–2088)
+    _pickup_ts = raw["tpep_pickup_datetime"]
+    _ts_min = pa.scalar(1546300800000000, pa.int64())  # 2019-01-01 00:00 UTC in us
+    _ts_max = pa.scalar(1548979200000000, pa.int64())  # 2019-02-01 00:00 UTC in us
+    _pickup_int = pc.cast(_pickup_ts, pa.int64())
+    raw = raw.filter(pc.and_(
+        pc.greater_equal(_pickup_int, _ts_min),
+        pc.less(_pickup_int, _ts_max),
+    ))
+
     # Cast to timestamp without tz (Windows tzdata workaround)
     pickup_no_tz = raw["tpep_pickup_datetime"].cast(pa.timestamp("us"))
     # Pickup hour and day of week

{hypertopos-0.3.3 → hypertopos-0.4.1}/docs/concepts.md

@@ -68,6 +68,55 @@ Two opt-in post-processing stages can be applied to any attract primitive that s
 
 Both stages compose: `fdr_alpha=0.05, select="diverse"` first filters to FDR-controlled discoveries, then picks the K most geometrically diverse among them.
 
+## Bregman Divergence
+
+Standard anomaly scoring in GDS uses `‖δ‖₂` — a single Euclidean distance that treats every dimension identically. This is fast and works well when dimensions are roughly Gaussian.
+
+**Bregman divergence** generalises the scoring: each dimension contributes an anomaly score computed from the generator function matched to its distribution family (`kind`). For a dimension with kind `gaussian`, the generator is squared loss (identical to L2). For `poisson`, it is the Kullback-Leibler divergence for counts. For `bernoulli`, it is binary cross-entropy.
+
+The per-dimension Bregman scores are summed to produce `bregman_divergence` — a single float stored alongside `delta_norm` in the geometry file. Dimensions with a poor match between their assumed family and actual data shape produce disproportionately large Bregman terms, making them visible in `explain_anomaly` output.
+
+**What this enables:**
+- Binary FK dimensions (bernoulli) no longer inflate the total score when an entity is merely "active" in that relation — the scoring accounts for the 0/1 nature of the edge
+- Count-based derived features (poisson) are scored against their expected arrival rate rather than a z-score
+- `explain_anomaly` returns per-dimension Bregman contributions with `kind` and `pct_of_total`, identifying which distribution mismatch is the primary driver
+
+`bregman_divergence` is stored in sphere format ≥ 2.3. Pre-2.3 spheres return `None` for this field.
+
+### Dimension Kinds
+
+Every dimension in a pattern carries a `kind` tag — the distribution family used for Bregman scoring. The builder auto-detects kinds at build time using the following rules:
+
+| Dimension source | Inferred kind |
+|-----------------|---------------|
+| Binary FK (`edge_max = null`) | `bernoulli` |
+| `count`, `count_distinct`, `count:window=*` | `poisson` |
+| `sum:col`, `avg:col`, `std:col`, `iet_*` | `gaussian` |
+| `max:col`, `min:col` | `gaussian` |
+| `precomputed_dimension` with `edge_max: 1` | `bernoulli` |
+| `precomputed_dimension`, otherwise | `gaussian` |
+| `graph_features` in/out-degree (`edge_max > 1`) | `poisson` |
+| `graph_features` reciprocity, overlap (`edge_max = 1`) | `bernoulli` |
+| `graph_features` pagerank, betweenness, clustering_coefficient | auto |
+| `graph_features` community, connected_component | auto |
+
+Override via `kind:` on individual dimension entries in `sphere.yaml`. The resulting kind list is stored in `Pattern.dimension_kinds` and surfaced in `sphere_overview` as `dimension_kinds`.
+
+## Anomaly Confidence
+
+Bootstrap confidence measures how stable an entity's anomaly classification is under random resampling of the calibration population.
+
+The builder draws `bootstrap_iterations` bootstrap samples from the population (default 200), recomputes mu/sigma/theta on each sample, and records whether the entity's delta_norm exceeds the resampled theta. `anomaly_confidence` is the fraction of samples in which the entity is classified as anomalous.
+
+**Interpretation:**
+- `1.0` — entity is anomalous under every bootstrap resample; classification is robust
+- `0.5–0.99` — entity sits near the boundary; classification may flip with slightly different calibration data
+- `< 0.5` — entity is nominally anomalous (delta_norm > theta) but unstable; treat with caution
+
+`anomaly_confidence` is skipped when the pattern uses `group_by_property` (bootstrap is per-group, cost grows multiplicatively), `use_mahalanobis` (full covariance bootstrap is expensive), or N > 50K entities. In those cases the field is `None`.
+
+Filter `find_anomalies` by minimum confidence using `min_confidence` — for example `min_confidence=0.8` returns only entities whose anomaly classification is stable across at least 80% of bootstrap samples.
+
 ## Three Scales
 
 ![Three Scales](images/three-scales.svg)
@@ -139,7 +188,7 @@ An edge table is a flat Lance dataset that links anchor entities through an even
 **When it exists:** The builder emits an edge table automatically for event patterns with 2+ FK relations to the same anchor line (e.g. an event pattern with `from_entity` and `to_entity` both pointing to the same anchor line). It can also be configured explicitly in YAML.
 
 **What it enables:**
-- **Runtime graph traversal** -- `find_geometric_path` uses beam search over the edge table, scoring paths by geometric coherence of intermediate entities
+- **Runtime graph traversal** -- `find_geometric_path` uses bidirectional BFS over the edge table, scoring discovered paths by geometric coherence of intermediate entities
 - **Lazy chain discovery** -- `discover_chains` performs temporal BFS on edges without requiring build-time chain extraction
 - **Edge statistics** -- row counts, unique entity counts, timestamp and amount ranges
 - **Witness cohort discovery** -- `find_witness_cohort` ranks entities that share the target's witness signature by combining delta similarity, witness overlap, trajectory alignment, and a graded anomaly bonus, while excluding entities already connected via the edge table. **Investigative peer ranking, not edge forecasting** — surfaces existing peers rather than predicting future edges. Uniquely possible because population-relative geometry, witness sets, temporal solids, and the edge table all live in one storage layer
@@ -172,6 +221,9 @@ Chains (both build-time `chain_lines` and runtime `discover_chains`) are sequenc
 | **Delta norm** | L2 magnitude of the deviation -- basis for similarity, clustering, and anomaly scoring |
 | **Theta threshold** | Statistical boundary derived from the population distribution (used for anomaly classification) |
 | **Deformation log** | History of changes that produced the current solid |
+| **Bregman divergence** | Distribution-aware anomaly distance summed across per-dimension Bregman terms |
+| **Anomaly confidence** | Bootstrap stability score (0–1) for anomaly classification under population resampling |
+| **Dimension kind** | Distribution family tag per dimension (`gaussian`, `poisson`, `bernoulli`) driving Bregman scoring |
 
 Every dimension has a meaning tied to a relation or tracked property.
 

{hypertopos-0.3.3 → hypertopos-0.4.1}/docs/configuration.md

@@ -361,6 +361,7 @@ patterns:
 | `graph_features` | dict | `null` | Auto-compute graph structural features from event from/to columns. |
 | `description` | string | `null` | Human-readable description. Stored in sphere.json — visible to agents via `get_sphere_info`. |
 | `edge_table` | dict | `null` | Explicit edge table config (see below). Auto-emitted for event patterns with 2+ FK relations to same anchor line. |
+| `bootstrap_iterations` | int | `200` | Number of bootstrap resamples used to compute `anomaly_confidence` per entity. Skipped automatically for N > 50K, `group_by_property`, or `use_mahalanobis`. |
 
 ### Edge Table
 
@@ -529,6 +530,46 @@ derived_dimensions:
 IET requires a `temporal` config for the pattern (provides `timestamp_col`).
 Temporal column is auto-resolved from the temporal config — no explicit time_col needed.
 
+**Dimension `kind` on derived dimensions:** Each derived feature can carry an explicit `kind` tag to override auto-detection. Add it as an extra property alongside the feature spec:
+
+```yaml
+derived_dimensions:
+  - from_pattern: tx_pattern
+    features:
+      - tx_count: count
+        kind: poisson           # override auto-detection
+      - total_spend: sum:amount
+        kind: gaussian
+      - has_foreign: count_distinct:country
+        kind: bernoulli         # treat as 0/1 presence indicator
+```
+
+When omitted, `kind` is auto-detected from the metric type — see below.
+
+---
+
+### Dimension Kind Auto-Detection
+
+The builder assigns a `kind` tag to every dimension at build time. The tag drives Bregman divergence scoring: each dimension contributes an anomaly score calibrated to its own distribution family rather than a single global metric.
+
+**Auto-detection rules:**
+
+| Dimension source | Inferred kind | Rationale |
+|-----------------|---------------|-----------|
+| Binary FK (`edge_max = null`) | `bernoulli` | 0/1 presence indicator |
+| `count`, `count_distinct`, `count:window=*` | `poisson` | Event counts follow Poisson arrival models |
+| `sum:col`, `avg:col`, `std:col`, `iet_*` | `gaussian` | Continuous magnitudes |
+| `max:col`, `min:col` | `gaussian` | Extremal statistics |
+| `precomputed_dimension` with `edge_max: 1` | `bernoulli` | Binary column indicator |
+| `precomputed_dimension`, otherwise | `gaussian` | Default continuous |
+| `graph_features` (`in_degree`, `out_degree`) | `poisson` | Degree counts (`edge_max > 1`) |
+| `graph_features` (`reciprocity`, `counterpart_overlap`) | `bernoulli` | Binary presence (`edge_max = 1`) |
+| `graph_features` (`pagerank`, `betweenness`, `clustering_coefficient`) | auto | Continuous scores (igraph C backend) |
+| `graph_features` (`community`, `connected_component`) | auto | Integer IDs (categorical) |
+| `geo_properties`, `metric_properties` | `gaussian` | Continuous measurements |
+
+**Override via `kind:`** on individual dimension entries (precomputed, derived). Override takes precedence over auto-detection. Event dimensions use auto-detection only.
+
 ---
 
 ## Composite Lines
@@ -595,6 +636,7 @@ patterns:
 | `edge_max` | int or `"auto"` | `"auto"` | Fixed normalization cap, or auto-compute from percentile. |
 | `percentile` | float | `99.0` | Percentile for auto edge_max computation. |
 | `display_name` | string | `null` | Human-readable dimension label. |
+| `kind` | string | auto | Distribution family for Bregman scoring: `"gaussian"`, `"poisson"`, or `"bernoulli"`. Auto-detected when omitted. |
 
 ---
 
@@ -609,7 +651,7 @@ patterns:
       event_line: transactions
       from_col: from_account
       to_col: to_account
-      features: [in_degree, out_degree, reciprocity, counterpart_overlap]
+      features: [in_degree, out_degree, reciprocity, counterpart_overlap, pagerank, betweenness, community, clustering_coefficient, connected_component]
 ```
 
 **Supported features:**
@@ -617,13 +659,18 @@ patterns:
 - `out_degree` — number of unique outgoing edges
 - `reciprocity` — fraction of counterparties with bidirectional flow
 - `counterpart_overlap` — Jaccard similarity of in/out counterparty sets
+- `pagerank` — network centrality (igraph C backend)
+- `betweenness` — bridge score between communities (edge-sampled on large graphs)
+- `community` — community ID via label propagation
+- `clustering_coefficient` — local triangle density
+- `connected_component` — isolated subgraph membership
 
 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
 | `event_line` | string | — | **Required.** Event line with from/to columns. |
 | `from_col` | string | — | **Required.** Source entity FK column. |
 | `to_col` | string | — | **Required.** Destination entity FK column. |
-| `features` | list | all four | Which features to compute. |
+| `features` | list | all nine | Which features to compute. Requires `pip install hypertopos[graph]` for pagerank/betweenness/community/clustering_coefficient/connected_component. |
 
 ---
 
@@ -904,8 +951,9 @@ hypertopos build --config sphere.yaml --force --verbose
 
 The builder parallelizes where possible:
 - **Points write** — parallel across lines (ThreadPoolExecutor, up to 4 workers)
-- **Geometry build** — parallel across patterns (one thread per pattern)
-- **Temporal build** — parallel across patterns; adaptive memory chunking when the shape tensor exceeds available RAM (auto-detected via platform API, 4 GB fallback)
+- **Pipeline mode** (when `temporal:` is configured) — each pattern runs geometry → temporal as a sequential pipeline, with up to 4 patterns executing concurrently. Temporal for a pattern starts immediately after its geometry completes, without waiting for other patterns.
+- **Geometry-only mode** (no `temporal:` or `--no-temporal`) — geometry runs in parallel across patterns (up to 4 threads)
+- **Temporal adaptive chunking** — when the shape tensor exceeds available RAM, the temporal phase uses adaptive memory chunking (auto-detected via platform API, 4 GB fallback)
 
 ### Build timing
 
@@ -915,15 +963,11 @@ Source 'transactions': 5,078,345 rows, 18 cols
 Source 'accounts': 515,080 rows, 11 cols
 Sources total: 0.6s
 Extracting chains for 'tx_chains'...
-  Seeds: 109,718
-  Chains cached to chains_tx_chains_ed008d26354b42ee.pkl
-  Chains extracted: 300,000
-Chains total: 336s
-Building geometry...
-Geometry total: 263s
-Building temporal for 'account_pattern' (2d windows)...
-Temporal total: 305s
-Build total: 527s
+  Chains extracted: 290,784
+Chains total: 9.5s
+Building geometry + temporal (pipeline)...
+Geometry + temporal total: 304.9s
+Build total: 315.5s
 ```
 
 ### Skip flags (iterative build)
@@ -945,10 +989,17 @@ hypertopos build --config sphere.yaml --force --verbose
 
 | Sphere | First run | Cached run |
 |--------|-----------|-----------|
-| Berka (4.5K accounts, 1M tx) | 24s | 24s |
-| TPCH SF1 (150K cust, 6M tx) | 2.9 min | 2.9 min |
-| AML HI-Small (515K accts, 5M tx, 300K chains) | ~22 min | ~9.5 min |
-| AML LI-Small (712K accts, 7M tx, 300K chains) | ~13.4 min | ~9.4 min |
+| Berka (4.5K accounts, 1M tx) | ~35s | ~25s |
+| NYC Taxi (265 zones, 7.5M trips) | ~3.2 min | ~2.8 min |
+| AML HI-Small (515K accts, 5M tx, 300K chains) | ~5.3 min | ~6.7 min |
+
+### Benchmark sphere configurations
+
+| Sphere | Edge table | Chains | Temporal | Composites | Key features |
+|--------|-----------|--------|----------|------------|--------------|
+| Berka | none | none | 3 patterns (90d) | account×bank pairs | NB-Split (accounts + accounts_stress + loan_accounts), precomputed dims, group_by_property |
+| NYC Taxi | auto-detected (trip_pattern: pu_zone → do_zone) | none | 2 patterns (7d) | none | NB-Split (zones_pickup + zones_dropoff), 6 event dims, 2 anchor patterns |
+| AML HI-Small | explicit (tx_pattern: from_account → to_account, timestamp, amount) | 300K (seeded BFS) | 3 patterns (2d) | account pairs | graph_features, precomputed ratios, IET metrics, Mahalanobis |
 
 ### Parallel chain extraction
 

{hypertopos-0.3.3 → hypertopos-0.4.1}/docs/data-format.md

@@ -109,6 +109,9 @@ Domain columns vary per line. The `primary_key` column is always present and alw
 | `delta` | fixed_size_list\<float32\> | Z-scored delta vector (deviation from mu) |
 | `delta_norm` | float32 | L2 norm of delta (distance from population center -- used for scoring, clustering, and similarity) |
 | `delta_rank_pct` | float32 | Percentile rank of delta_norm (0--100) |
+| `conformal_p` | float32 | Conformal p-value (null when not computed) |
+| `bregman_divergence` | float32 | Sum of per-dimension Bregman terms (distribution-aware anomaly distance). Null on pre-2.3 spheres. |
+| `anomaly_confidence` | float32 | Bootstrap stability score: fraction of bootstrap resamples in which entity is anomalous. Null when bootstrap was skipped. |
 | `edges` | list\<struct\> | Edge list: line_id, point_key, direction, status |
 
 The `delta` vector length equals the number of dimensions in the pattern. Geometry datasets carry an IVF-PQ ANN index on the `delta` column for trajectory similarity search.

{hypertopos-0.3.3 → hypertopos-0.4.1}/hypertopos/__init__.py

@@ -16,4 +16,4 @@ __all__ = [
     "GDSBuilder",
     "RelationSpec",
 ]
-__version__ = "0.3.0"
+__version__ = "0.4.1"