hypertopos 0.2.1__tar.gz → 0.3.0__tar.gz

{hypertopos-0.2.1 → hypertopos-0.3.0}/CHANGELOG.md

@@ -5,6 +5,61 @@ All notable changes to hypertopos will be documented in this file.
 Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.0] — 2026-04-12
+
+> **Theme:** Lance 3.x perf upgrade. Aggregate engine moves from a persistent subprocess worker plus an optional external DataFusion package to the Lance scanner's built-in DataFusion executor via `LanceDataset.sql(...)`. The build-time contagion stats table replaces the runtime edge-table replay in the graph contagion scanner. Format 2.2 is the new write default.
+>
+> **Mandatory rebuild for spheres built before this release.** See the Migration section below.
+
+### Changed
+
+- `pylance` minimum bumped to 4.x. New writes target Lance format 2.2 with structural-decode parallelism, BSS float compression, cascading codecs, and block-level RLE on top of the previous baseline. Older spheres written with format 2.0 / 2.1 remain readable transparently.
+- The aggregate engine now pushes GROUP BY computation directly into the Lance scanner. The new `lance_sql_agg` module hosts `aggregate_count`, `aggregate_metric` (sum / avg / min / max), `aggregate_filtered_metric`, `aggregate_pivot`, `aggregate_property`, `aggregate_percentile`, and `find_anomalies`. Each helper streams the columns it needs from the geometry / event-line / property-line Lance datasets through `LanceDataset.sql(...)` and finishes the join + aggregate in pyarrow on the small post-projection result. The full geometry table is never loaded into Python memory and the persistent subprocess worker is no longer involved for any of these paths.
+- Test fixture cloning uses `lance.LanceDataset.shallow_clone` for every Lance dataset directory inside a sphere tree, with a thin `clone_sphere` helper in `tests/conftest.py` that walks the directory and falls back to `shutil.copy2` for non-Lance files. Shallow clone copies only metadata and references — measurably faster on Windows where deep `copytree` over a sphere built from many small Lance fragment files is dominated by per-file NTFS overhead.
+
+### Added
+
+- `GDSBuilder` rejects patterns with zero declared dimensions (no relations, no event dimensions, no derived dimensions, no tracked properties) at validation time. A pattern with width-0 delta vectors had no meaningful geometry to compute even before this release; format 2.0 silently allowed the construct, format 2.1+ refuses to encode the resulting `fixed_size_list[0]` columns at all. The validation surfaces a typed error up front instead of letting the call descend into an opaque write-time panic.
+- Builder precomputes per-entity contagion statistics for every pattern with an edge table and writes them to `_gds_meta/contagion_stats/{pattern_id}.lance` with a BTREE index on `primary_key`. The table holds `(primary_key, neighbor_count, anomalous_neighbor_count, contagion_ratio)` snapshotted at build time against the just-calibrated anchor geometry.
+
+### Fixed
+
+- `_scan_graph` (the graph contagion source feeding `passive_scan` and `composite_risk`) reads the precomputed `_gds_meta/contagion_stats/{pattern_id}.lance` table directly and threshold-filters inline. The legacy edge-table-replay path is gone; spheres built before 0.3.0 must be rebuilt to get graph contagion hits.
+- `contagion_score` anchor pattern resolution now correctly resolves via sibling lines when the anchor pattern is not directly associated with the target entity's line. Previously crashed with `KeyError` on spheres where the anchor pattern entity line differed from the navigated line.
+- Edge table auto-detect excludes metadata columns (`created_at`, `changed_at`, `version`) from the timestamp type-based fallback, preventing incorrect timestamp column selection on entity lines with versioning columns. Amount candidate list widened to cover `fare_amount`, `total_amount`, `amount_received`, `amount_paid`.
+
+### Security
+
+- `lance_sql_agg` validates user-controlled SQL inputs up front. Lance's `LanceDataset.sql(...)` does not expose parameter binding, so values and identifiers are inlined into the query string. Filter values (entity primary keys) are escaped with `_escape_sql_string`, which doubles single quotes and rejects backslash and ASCII control characters. Column-name identifiers (`metric_col`, `pivot_field`, `prop_name`) are validated against `^[A-Za-z_][A-Za-z0-9_]*$` via `_validate_sql_identifier` before being inlined into SQL strings — anything outside that pattern raises immediately. Both helpers fail loud rather than silently produce wrong results or pass injected SQL through to the parser.
+
+### Removed
+
+- `hypertopos.engine.subprocess_agg` (the persistent subprocess worker and its JSON command protocol) is deleted; every entry point it served is now routed through `lance_sql_agg`.
+- `hypertopos.engine.datafusion_agg` (the in-memory external-DataFusion fast path for count / metric / pivot / gbp / percentile) is deleted along with the optional `datafusion` extra in `pyproject.toml`. Lance 4.x's built-in DataFusion executor covers every query the external package was carrying.
+- The matching `tests/test_subprocess_agg.py` and `tests/test_datafusion_agg.py` are deleted. Aggregate semantics are exercised through `tests/test_aggregation_engine.py` and the MCP tool tests.
+
+### Migration
+
+- **Rebuild required.** Spheres built before 0.3.0 are still openable but degrade gracefully on graph contagion — the precomputed `contagion_stats` table is created at build time only, and the runtime scanner returns zero hits without it. Rebuild with `hypertopos build --config sphere.yaml --force`. There are no in-place upgrade paths.
+- The optional `datafusion` package is no longer a dependency of any aggregate path. If your environment installed it via `pip install hypertopos[datafusion]`, that extra now resolves to nothing — you can drop it from your install command.
+
+## [0.2.2] — 2026-04-11
+
+### Added
+
+**Temporal as-of reconstruction across edge-table graph primitives**
+- `contagion_score()`, `contagion_score_batch()` — optional keyword-only `timestamp_cutoff: float | None` parameter. When set, only edges with `timestamp <= timestamp_cutoff` are considered. Batch forwards the cutoff to every per-entity call.
+- `entity_flow()` — same `timestamp_cutoff` parameter; both outgoing and incoming edge reads honor the cutoff, so net flow reflects the as-of graph state.
+- `degree_velocity()` — same parameter; buckets derive from the filtered edge set so the last bucket endpoint is naturally `<= timestamp_cutoff`.
+- `propagate_influence()` — same parameter; threaded through the BFS `_expand_neighbors` closure so expansion follows only edges with `timestamp <= cutoff`.
+- `find_counterparties()` — same parameter on the edge-table fast path via `_find_counterparties_via_edges`. The points-scan fallback has no timestamp column and **raises `GDSNavigationError`** when `timestamp_cutoff` is supplied — fail loudly instead of silently returning unfiltered results.
+
+The semantic mirrors the existing `WitnessCohortConfig.timestamp_cutoff` from 0.2.1: edges with `timestamp <= timestamp_cutoff` are included. Enables agents to reconstruct contagion, flow, connection velocity, and influence propagation state at a prior point in time — useful for incident forensics ("what did this neighborhood look like on the day the alert fired?"), retroactive detection validation, and historical-snapshot comparisons.
+
+### Fixed
+
+- `detect_cross_pattern_discrepancy` no longer triggers full edge-table scans through `PassiveScanner.auto_discover`. The scanner gains an `include_graph: bool = True` keyword-only parameter; `detect_cross_pattern_discrepancy` calls it with `include_graph=False` because graph contagion plays no role in the downstream geometry-disagreement check. Eliminates the dominant latency regression on multi-pattern spheres with edge tables — the discrepancy detector previously paid one full edge-table read per event pattern with no signal benefit. Every other `auto_discover` caller (`composite_risk`, explicit scanner use) stays at the graph-enabled default.
+
 ## [0.2.1] — 2026-04-11
 
 ### Added
@@ -15,17 +70,11 @@ Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - Excludes already-connected entities via BTREE edge lookup; bidirectional check by default; `timestamp_cutoff` for as-of evaluation in temporal hold-out
 - Auto-resolves the event pattern with edge table covering an anchor's entity line; explicit `edge_pattern_id` override is validated
 - Trajectory branch decided once per call (not per candidate) — when ref has trajectory, missing candidates get neutral 0.5 instead of mixed renormalization
-- Batch trajectory load via single Lance scan instead of per-candidate (~11× speedup: 3.7s → ~330ms on AML HI-small)
+- Batch trajectory load via single Lance scan instead of per-candidate (~11× speedup over the per-candidate path)
 - `WitnessCohortConfig` and `WitnessCohortWeights` dataclasses group all tunable parameters; navigator API takes a single `config=` keyword
 - `CohortMember` and `WitnessCohortResult` frozen dataclasses with per-component scores, exclusion counts, and reproducibility metadata
 - `GDSEngine.witness_jaccard()`, `GDSEngine.trajectory_cosine()`, `GDSEngine.composite_link_score()` — pure scoring helpers exposed for reuse
 
-**Empirical validation on AML HI-small** (5M edges, 6357 labeled launderers):
-- Co-laundering precision@10: **25.3%** (vs 1.2% base rate = **20.5× lift**)
-- 2.6× improvement over `find_similar_entities + is_anomaly` baseline (which scored 6.5%)
-- Top-10 overlap with that baseline: 15.5% (function returns substantively different entities, not the same ranking)
-- Temporal hold-out recall@10: **0.0%** — confirms the function does NOT forecast future edges; it surfaces existing geometric peers
-
 ## [0.2.0] — 2026-04-10
 
 ### Added

{hypertopos-0.2.1 → hypertopos-0.3.0}/LICENSE.md

@@ -26,7 +26,7 @@ For purposes of this License:
 - **Internal use** within an organization is not a competitive offering. Your
   organization includes entities under common control.
 
-**Change Date:** 2029-04-11
+**Change Date:** 2029-04-12
 
 **Change License:** Apache License, Version 2.0
 

{hypertopos-0.2.1 → hypertopos-0.3.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: hypertopos
-Version: 0.2.1
-Summary: Navigate your data like space.
+Version: 0.3.0
+Summary: Understand the structure of your data — without training machine learning models
 Project-URL: Homepage, https://github.com/hypertopos/hypertopos-py
 Project-URL: Documentation, https://github.com/hypertopos/hypertopos-py/tree/main/docs
 Project-URL: Repository, https://github.com/hypertopos/hypertopos-py
@@ -12,10 +12,8 @@ License-File: LICENSE.md
 Requires-Python: >=3.12
 Requires-Dist: numpy>=1.26
 Requires-Dist: pyarrow>=14.0
-Requires-Dist: pylance>=0.20
+Requires-Dist: pylance<5.0,>=4.0
 Requires-Dist: pyyaml>=6.0
-Provides-Extra: datafusion
-Requires-Dist: datafusion>=40.0; extra == 'datafusion'
 Provides-Extra: dev
 Requires-Dist: mypy; extra == 'dev'
 Requires-Dist: pytest; extra == 'dev'

{hypertopos-0.2.1 → hypertopos-0.3.0}/README.md

@@ -6,9 +6,9 @@
 [![License: BSL 1.1](https://img.shields.io/badge/license-BSL%201.1-orange.svg)](LICENSE.md)
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19482069.svg)](https://doi.org/10.5281/zenodo.19482069)
 [![PyArrow](https://img.shields.io/badge/format-PyArrow-red.svg)](https://arrow.apache.org/docs/python/)
-[![Lance](https://img.shields.io/badge/storage-Lance-blueviolet.svg)](https://github.com/lancedb/lancedb)
+[![Lance](https://img.shields.io/badge/storage-Lance-blueviolet.svg)](https://github.com/lance-format/lance)
 [![MCP](https://img.shields.io/badge/MCP-compatible-green.svg)](https://modelcontextprotocol.io)
-[![Version](https://img.shields.io/badge/version-0.2.1-blue.svg)](pyproject.toml)
+[![Version](https://img.shields.io/badge/version-0.3.0-blue.svg)](pyproject.toml)
 
 **hypertopos** transforms relational data into navigable geometric space.
 
@@ -38,7 +38,7 @@ Navigation primitives turn this space into a stateful, step-by-step workspace, w
 
 - **Geometry, not queries** — entities live in a population-calibrated coordinate space (μ, σ, θ). Position tells you what's typical. Distance reveals what's unusual, what's similar, and where clusters form.
 - **Navigate, don't search** — twelve primitives (π1–π12) let agents walk lines, jump across relationships, cluster populations, compare groups, find hubs, and track drift. Stateful, composable, geometric.
-- **Graph meets geometry** — edge tables give runtime graph traversal with geometric path scoring. Find paths scored by witness overlap and anomaly propagation — not just hop count. Lazy chain discovery without build-time extraction. Surface witness cohorts (`find_witness_cohort`) — entities sharing the target's anomaly signature, ranked by delta similarity, witness overlap, trajectory alignment, and graded anomaly bonus, with already-connected entities filtered out. Investigative peer ranking validated on AML HI-small at **20.5× lift** over random base rate for co-laundering precision@10.
+- **Graph meets geometry** — edge tables give runtime graph traversal with geometric path scoring. Find paths scored by witness overlap and anomaly propagation — not just hop count. Lazy chain discovery without build-time extraction. Surface witness cohorts (`find_witness_cohort`) — entities sharing the target's anomaly signature, ranked by delta similarity, witness overlap, trajectory alignment, and graded anomaly bonus, with already-connected entities filtered out. Investigative peer ranking, not edge forecasting.
 - **Time is built in** — polygons accumulate into solids. Trajectory similarity, regime change detection, and drift analysis are first-class — not bolted on.
 - **Agents first** — designed for MCP tool-calling. Smart detection recipes, investigation workflows. The sphere is the agent's workspace.
 - **Zero training** — no ML models, no labels, no training pipeline. Population statistics and empirical thresholds. Deploy in minutes.
@@ -56,7 +56,7 @@ Navigation primitives turn this space into a stateful, step-by-step workspace, w
 | **Navigation** | 12 stateful primitives | Query-based | KNN search | N/A |
 | **Multi-hop** | Edge table (runtime graph + lazy chains) + pre-computed chain geometry | Native (Cypher, ad-hoc) | No | No |
 | **Anomaly propagation** | Contagion scoring + influence BFS in delta-space | No | No | Local neighborhood only |
-| **Witness cohort discovery** | Witness overlap + delta + trajectory + edge absence (validated 20.5× lift on AML) | Topological only (Adamic-Adar, common neighbors) | KNN, no graph awareness | Learned embeddings (training required) |
+| **Witness cohort discovery** | Witness overlap + delta + trajectory + edge absence | Topological only (Adamic-Adar, common neighbors) | KNN, no graph awareness | Learned embeddings (training required) |
 | **Agent-native (MCP)** | Yes | No | No | No |
 | **Explains why** | Witness sets, dimension ranking | Centrality scores | Nearest neighbors | Anomaly scores |
 

hypertopos-0.3.0/ROADMAP.md

@@ -0,0 +1,88 @@
+# hypertopos — Roadmap
+
+> Planned direction. Priorities may change based on feedback. 
+
+## Current: 0.3.0
+
+Lance 3.x perf upgrade — aggregate engine rewritten around Lance SQL, precomputed contagion stats, format 2.2, contagion anchor resolution fix, edge table auto-detect fix. See CHANGELOG.
+
+---
+
+## 0.2.2
+
+As-of graph reconstruction across edge-table graph primitives + latency fix on `detect_cross_pattern_discrepancy`. See CHANGELOG.
+
+---
+
+## 0.2.1
+
+Witness cohort discovery — investigative peer ranking. See CHANGELOG.
+
+---
+
+## 0.2.0
+
+Graph meets geometry — edge table, runtime traversal, contagion/influence, +11 navigator functions. See CHANGELOG.
+
+---
+
+## 0.1.0
+
+First public release — full GDS stack, π1–π12, builder, MCP server, validation suite. See CHANGELOG.
+
+---
+
+## Plan: 0.3.1
+
+Two production-quality improvements on top of the 0.3.0 storage layer:
+
+- **FDR control on anomaly queries** — Benjamini–Hochberg `fdr_alpha` parameter + per-row `q_value` column on `find_anomalies` and friends. Replaces unbounded "top-K by `delta_norm`" with a set whose false-discovery rate is bounded by α.
+- **Diverse-coverage anomaly summarisation** — `select="diverse"` option using lazy-greedy submodular maximisation, plus a `representativeness` column ("this 1 anomaly stands in for 38 cases").
+- **Temporal build optimization** — pre-aggregate event counts before the temporal loop, eliminating per-window event table rescans.
+
+---
+
+## Plan: 0.3.2
+
+README and docs narrative restructure.
+
+---
+
+## Plan: 0.4.0
+
+TBD — picked after 0.3.2 lands.
+
+---
+
+## Future
+
+**Detection quality**
+- Edge-derived dimensions + temporal motif matcher
+- Confidence scoring, robust estimators, multi-scale resolution — improve anomaly precision and reduce false positives on heavy-tail and multi-modal populations
+- Lazy chain geometry — on-demand chain delta vectors via sampled population calibration; supplements the build-time `chain_lines` path
+
+**Builder evolution**
+- Incremental rebuild — geometry-only without `--force` wipe
+- Generalized (d+m+g+t+s) dimension blocks — geographic / metric / semantic dimension support
+
+**PassiveScanner evolution**
+- Native temporal source support for direct temporal inputs, without requiring manual dataset plumbing in benchmark scripts
+- Optional weighted scoring mode that uses continuous intensity instead of binary counts
+- **SphereProfiler** — autonomous sphere scanner that profiles all patterns, runs calibration sweeps across source combinations, proposes optimal PassiveScanner composition for Layer 1 surveillance. Core loop: enumerate patterns → single-source anomaly scans → greedy multi-source combination → ranked composition report. No labeled GT needed; optional GT file unlocks supervised calibration.
+
+**Code refactoring**
+- Break up oversized modules into smaller, domain-focused components
+- Reduce coupling between core layers by replacing private cross-layer access with explicit interfaces where practical
+- Tighten error handling in hot paths so failures are surfaced more consistently
+- Consolidate repeated orchestration logic into shared helpers instead of duplicating it across modules
+- Incremental refactor: preserve behavior, improve structure, then revisit deeper architectural boundaries
+
+**Cross-sphere capabilities**
+- Cross-sphere comparison — dimensionless metrics across independently calibrated coordinate spaces
+- What-if analysis — hypothetical edge changes producing modified coordinate vectors
+
+**Enterprise / governance**
+- Dimension access control — per-agent visibility constraints on delta dimensions
+
+**Tooling**
+- Runtime latency benchmarks in package docs

{hypertopos-0.2.1 → hypertopos-0.3.0}/SECURITY.md

@@ -1,6 +1,6 @@
 # Security
 
-## Current version: 0.2.x
+## Current version: 0.3.0
 
 hypertopos runs locally. No network services, no auth layer, no multi-tenancy.
 

{hypertopos-0.2.1 → hypertopos-0.3.0}/docs/api-reference.md

@@ -124,15 +124,15 @@ with sphere.session("agent-1") as session:
 |--------|-------------|
 | `find_geometric_path(from_key, to_key, pattern_id, max_depth=5, beam_width=10, scoring="geometric")` | Beam search for paths between two entities via the edge table, scored by geometric coherence. Scoring modes: `"geometric"` (witness overlap + delta alignment + anomaly preservation), `"amount"` (geometric score modulated by log(transaction amount)), `"anomaly"` (prefer paths through anomalous entities), `"shortest"` (plain BFS, no geometric scoring). Returns best paths with per-hop scores |
 | `discover_chains(primary_key, pattern_id, time_window_hours=168, max_hops=10, min_hops=2, max_chains=100, direction="forward")` | Runtime temporal BFS on the edge table to discover entity chains from a starting point. Unlike `find_chains_for_entity()` which queries pre-computed chains, this performs live traversal -- works without build-time chain extraction |
-| `entity_flow(primary_key, pattern_id, top_n=20)` | Net flow analysis per counterparty via edge table. Two edge lookups (outgoing + incoming), sum amounts, compute per-counterparty net flow. Returns outgoing/incoming totals, net_flow, flow_direction, counterparties sorted by abs(net_flow) |
-| `contagion_score(primary_key, pattern_id)` | Score how many of an entity's counterparties are anomalous via edge table + geometry check. Returns score (0.0–1.0), total/anomalous counterparty counts |
-| `contagion_score_batch(primary_keys, pattern_id, max_keys=200)` | Batch contagion scoring for multiple entities. Returns per-entity scores plus summary (mean, max, high_contagion_count) |
-| `degree_velocity(primary_key, pattern_id, n_buckets=4)` | Temporal connection velocity — buckets edges by timestamp, counts unique counterparties per bucket. Velocity = last_bucket_degree / first_bucket_degree. Returns buckets with out/in degree, velocity metrics |
+| `entity_flow(primary_key, pattern_id, top_n=20, *, timestamp_cutoff=None)` | Net flow analysis per counterparty via edge table. Two edge lookups (outgoing + incoming), sum amounts, compute per-counterparty net flow. `timestamp_cutoff` (Unix seconds) restricts to edges with `timestamp <= cutoff`. Returns outgoing/incoming totals, net_flow, flow_direction, counterparties sorted by abs(net_flow) |
+| `contagion_score(primary_key, pattern_id, *, timestamp_cutoff=None)` | Score how many of an entity's counterparties are anomalous via edge table + geometry check. `timestamp_cutoff` enables as-of contagion reconstruction. Returns score (0.0–1.0), total/anomalous counterparty counts |
+| `contagion_score_batch(primary_keys, pattern_id, max_keys=200, *, timestamp_cutoff=None)` | Batch contagion scoring for multiple entities — forwards `timestamp_cutoff` to each per-entity call. Returns per-entity scores plus summary (mean, max, high_contagion_count) |
+| `degree_velocity(primary_key, pattern_id, n_buckets=4, *, timestamp_cutoff=None)` | Temporal connection velocity — buckets edges by timestamp, counts unique counterparties per bucket. Velocity = last_bucket_degree / first_bucket_degree. `timestamp_cutoff` clamps the last bucket endpoint by filtering edges at the read level. Returns buckets with out/in degree, velocity metrics |
 | `investigation_coverage(primary_key, pattern_id, explored_keys)` | Agent guidance: how much of an entity's edge neighborhood has been explored. Splits counterparties into explored/unexplored, batch anomaly check on unexplored |
-| `propagate_influence(seed_keys, pattern_id, max_depth=3, decay=0.7, min_threshold=0.001, max_affected=10_000)` | BFS influence propagation from seed entities with geometric decay and tx_count weighting. At each hop: influence = parent_score * decay * geometric_coherence * tx_weight. Returns affected_entities with tx_count per neighbor |
+| `propagate_influence(seed_keys, pattern_id, max_depth=3, decay=0.7, min_threshold=0.001, max_affected=10_000, *, timestamp_cutoff=None)` | BFS influence propagation from seed entities with geometric decay and tx_count weighting. At each hop: influence = parent_score * decay * geometric_coherence * tx_weight. `timestamp_cutoff` restricts BFS expansion to edges with `timestamp <= cutoff`. Returns affected_entities with tx_count per neighbor |
 | `cluster_bridges(pattern_id, n_clusters=5, top_n_bridges=10)` | Find entities bridging geometric clusters via edge table. Runs π8 clustering then identifies cross-cluster edges and bridge entities |
 | `anomalous_edges(from_key, to_key, pattern_id, top_n=10)` | Find edges between two entities enriched with event-level geometry (delta_norm, is_anomaly). Unlike path tools which score entities (anchor), this scores individual transactions (event geometry) |
-| `find_witness_cohort(primary_key, pattern_id, top_n=10, *, config=None, edge_pattern_id=None)` | Rank entities that share the target's witness signature. Investigative peer ranking — NOT edge forecasting. Combines four signals: `exp(-distance/theta)` delta similarity, witness Jaccard overlap, trajectory cosine alignment (optional), and graded anomaly bonus from `delta_rank_pct`. Excludes entities already connected via BTREE edge lookup — this is the function's main contribution over plain ANN. Configure via `WitnessCohortConfig(weights=WitnessCohortWeights(...), candidate_pool, min_witness_overlap, min_score, use_trajectory, bidirectional_check, timestamp_cutoff)`. Returns `WitnessCohortResult` with ranked `CohortMember` items, per-component scores, exclusion counts, and reproducibility metadata. Validated on AML HI-small: 20.5× lift over random for co-laundering precision@10 (25.3% vs 1.2%), 2.6× over `find_similar_entities + is_anomaly` baseline (6.5%), 15.5% top-10 overlap with that baseline |
+| `find_witness_cohort(primary_key, pattern_id, top_n=10, *, config=None, edge_pattern_id=None)` | Rank entities that share the target's witness signature. Investigative peer ranking — NOT edge forecasting. Combines four signals: `exp(-distance/theta)` delta similarity, witness Jaccard overlap, trajectory cosine alignment (optional), and graded anomaly bonus from `delta_rank_pct`. Excludes entities already connected via BTREE edge lookup — this is the function's main contribution over plain ANN. Configure via `WitnessCohortConfig(weights=WitnessCohortWeights(...), candidate_pool, min_witness_overlap, min_score, use_trajectory, bidirectional_check, timestamp_cutoff)`. Returns `WitnessCohortResult` with ranked `CohortMember` items, per-component scores, exclusion counts, and reproducibility metadata |
 
 ### Analysis and Detection
 
@@ -149,7 +149,7 @@ with sphere.session("agent-1") as session:
 | `cross_pattern_profile(primary_key, line_id)` | Anomaly status from all patterns the entity participates in |
 | `find_chains_for_entity(primary_key, pattern_id, top_n)` | Find chains involving a specific entity |
 | `find_neighborhood(primary_key, pattern_id, max_hops)` | BFS through polygon edges to find reachable entities |
-| `find_counterparties(primary_key, line_id, from_col, to_col, pattern_id, top_n, use_edge_table=True)` | Discover counterparty entities from event data. When pattern_id is given and edge table exists, uses BTREE fast path with amount_sum/amount_max per counterparty |
+| `find_counterparties(primary_key, line_id, from_col, to_col, pattern_id, top_n, use_edge_table=True, *, timestamp_cutoff=None)` | Discover counterparty entities from event data. When pattern_id is given and edge table exists, uses BTREE fast path with amount_sum/amount_max per counterparty. `timestamp_cutoff` applies to the edge-table fast path only; raises `GDSNavigationError` when supplied without an edge-table-eligible configuration |
 | `assess_false_positive(primary_key, pattern_id)` | Evaluate likelihood of false positive anomaly classification |
 
 **Detection recipes (population-level patterns):**
@@ -422,7 +422,7 @@ Or use the navigator shortcut: `nav.passive_scan(home_line_id)`.
 | `add_points_source(name, line_id, rules, combine)` | Register a points-rule source filtering by column thresholds |
 | `add_compound_source(name, geometry_pattern_id, line_id, rules)` | Geometry expansion intersected with points rules |
 | `add_graph_source(name, pattern_id, contagion_threshold=0.3, weight)` | Register a graph contagion source — flags entities whose anomalous counterparty ratio exceeds threshold. Requires event pattern with edge table |
-| `auto_discover(home_line_id, include_borderline)` | Auto-register all patterns related to a line. Also auto-detects graph sources for event patterns with edge tables |
+| `auto_discover(home_line_id, include_borderline, *, include_graph=True)` | Auto-register all patterns related to a line. Also auto-detects graph sources for event patterns with edge tables. Pass `include_graph=False` to skip graph source registration when the downstream scan does not need contagion signal — `detect_cross_pattern_discrepancy` uses this to avoid ~37s-per-event-pattern edge-table reads |
 | `scan(home_line_id, scoring, threshold, top_n)` | Execute batch scan across all registered sources. Returns `ScanResult` |
 
 ### ScanResult

{hypertopos-0.2.1 → hypertopos-0.3.0}/docs/architecture.md

@@ -24,7 +24,7 @@ The core package is split into a few layers:
 
 - **Model**: sphere, line, pattern, alias, point, edge, polygon, solid, manifest, contract
 - **Storage**: readers, writers, caches, and physical layout handling
-- **Engine**: geometry, anomaly scoring, similarity, drift, forecasting, aggregation, calibration tracking
+- **Engine**: geometry, anomaly scoring, similarity, drift, forecasting, aggregation (Lance SQL pushdown via `lance_sql_agg`), calibration tracking
 - **Navigation**: stateful step-by-step movement through the sphere
 - **PassiveScanner**: multi-source batch screening over navigation primitives
 - **Builder**: transforms source data into a complete sphere
@@ -224,6 +224,10 @@ The scanner collects, deduplicates, and ranks results into a unified candidate l
 is the primary entry point for broad screening workflows where the agent does not yet know
 which specific primitive to focus on. `auto_discover()` registers the appropriate sources
 automatically — graph sources are added for every event pattern that has an edge table.
+Callers that do not need graph contagion signal (e.g. `detect_cross_pattern_discrepancy`,
+which measures geometry disagreement between patterns, not neighbor contamination) pass
+`include_graph=False` to skip graph source registration and avoid the per-pattern edge
+table read that would otherwise dominate scan latency on large spheres.
 
 ## Build Flow
 
@@ -278,7 +282,7 @@ The MCP package is a transport and tool layer.
 It does not define the core GDS model. Instead, it:
 
 - opens and closes spheres
-- exposes 66 navigation and analysis tools
+- exposes 67 navigation and analysis tools
 - formats results as JSON
 - manages session state for an agentic client
 

{hypertopos-0.2.1 → hypertopos-0.3.0}/docs/concepts.md

@@ -132,7 +132,7 @@ An edge table is a flat Lance dataset that links anchor entities through an even
 - **Runtime graph traversal** -- `find_geometric_path` uses beam search over the edge table, scoring paths by geometric coherence of intermediate entities
 - **Lazy chain discovery** -- `discover_chains` performs temporal BFS on edges without requiring build-time chain extraction
 - **Edge statistics** -- row counts, unique entity counts, timestamp and amount ranges
-- **Witness cohort discovery** -- `find_witness_cohort` ranks entities that share the target's witness signature by combining delta similarity, witness overlap, trajectory alignment, and a graded anomaly bonus, while excluding entities already connected via the edge table. **Investigative peer ranking, not edge forecasting.** Validated on AML HI-small: 25.3% co-laundering precision@10 vs 1.2% random base rate (20.5× lift), 2.6× improvement over `find_similar_entities + is_anomaly` baseline (6.5%), 15.5% top-10 overlap with that baseline (substantively different results, not ANN with extra steps); temporal hold-out recall@10 is 0%, confirming the function surfaces existing peers rather than predicting future edges. Uniquely possible because population-relative geometry, witness sets, temporal solids, and the edge table all live in one storage layer
+- **Witness cohort discovery** -- `find_witness_cohort` ranks entities that share the target's witness signature by combining delta similarity, witness overlap, trajectory alignment, and a graded anomaly bonus, while excluding entities already connected via the edge table. **Investigative peer ranking, not edge forecasting** — surfaces existing peers rather than predicting future edges. Uniquely possible because population-relative geometry, witness sets, temporal solids, and the edge table all live in one storage layer
 
 **Edge table fields:** `from_key` and `to_key` are anchor entity keys derived from event pattern FK columns. `event_key` links back to the event line for traceability. `timestamp` is epoch seconds from the column specified by `edge_table.timestamp_col` (or auto-detected). `amount` is the numeric value from `edge_table.amount_col` (or auto-detected from columns named `amount`, `value`, `total`, `amt`). The semantic meaning of `amount` depends on the domain (e.g. payment value, fare, order total, shipment weight). When no amount column is found, defaults to 0.0.
 

{hypertopos-0.2.1 → hypertopos-0.3.0}/docs/data-format.md

@@ -17,7 +17,8 @@ gds_{sphere_id}/
 │   ├── geometry_stats/          # precomputed population summaries
 │   ├── trajectory/              # ANN index for trajectory similarity search
 │   ├── temporal_centroids/      # cached population centroids per time window
-│   └── edge_stats/              # per-event-pattern edge table summary cache (row count, unique from/to, ts/amount range)
+│   ├── edge_stats/              # per-event-pattern edge table summary cache (row count, unique from/to, ts/amount range)
+│   └── contagion_stats/         # per-pattern (primary_key, neighbor_count, anomalous_neighbor_count, contagion_ratio) — feeds the graph contagion scanner directly, BTREE-indexed on primary_key
 ├── points/
 │   ├── {line_id}/v={n}/
 │   │   └── data.lance           # entity records for this line version

{hypertopos-0.2.1 → hypertopos-0.3.0}/hypertopos/__init__.py

@@ -16,4 +16,4 @@ __all__ = [
     "GDSBuilder",
     "RelationSpec",
 ]
-__version__ = "0.1.0"
+__version__ = "0.3.0"

{hypertopos-0.2.1 → hypertopos-0.3.0}/hypertopos/builder/builder.py

@@ -1380,16 +1380,24 @@ class GDSBuilder:
         schema_names = set(line_reg.table.schema.names)
 
         ts_candidates = (
-            "timestamp", "ts", "event_time", "created_at", "tx_date", "date",
+            "timestamp", "ts", "event_time", "tx_date", "date",
         )
         amt_candidates = (
             "amount_received", "amount", "amount_paid", "value",
-            "total", "amt",
+            "total", "amt", "fare_amount", "total_amount",
         )
 
         ts_col = next(
             (c for c in ts_candidates if c in schema_names), None,
         )
+        # Type-based fallback: first non-metadata column with timestamp type
+        _META_COLS = {"created_at", "changed_at", "version"}
+        if ts_col is None:
+            import pyarrow as pa
+            for field in line_reg.table.schema:
+                if pa.types.is_timestamp(field.type) and field.name not in _META_COLS:
+                    ts_col = field.name
+                    break
         amt_col = next(
             (c for c in amt_candidates if c in schema_names), None,
         )
@@ -1417,23 +1425,31 @@ class GDSBuilder:
         to_arr = event_table[cfg.to_col]
         event_key_arr = event_table["primary_key"]
 
-        # Timestamp
+        # Timestamp — resolve from config, then name heuristic, then type fallback
         ts_arr: pa.Array
         ts_col = cfg.timestamp_col
         if ts_col and ts_col in schema_names:
             ts_arr = self._to_epoch_seconds(event_table[ts_col])
         else:
-            # Try common column names
             for name in ("timestamp", "ts", "date", "created_at", "tx_date"):
                 if name in schema_names:
                     ts_arr = self._to_epoch_seconds(event_table[name])
                     break
             else:
-                ts_arr = pa.array(
-                    [0.0] * len(event_table), type=pa.float64(),
-                )
+                # Type-based fallback: first non-metadata column with timestamp type
+                _META = {"created_at", "changed_at", "version"}
+                ts_resolved = False
+                for field in event_table.schema:
+                    if pa.types.is_timestamp(field.type) and field.name not in _META:
+                        ts_arr = self._to_epoch_seconds(event_table[field.name])
+                        ts_resolved = True
+                        break
+                if not ts_resolved:
+                    ts_arr = pa.array(
+                        [0.0] * len(event_table), type=pa.float64(),
+                    )
 
-        # Amount
+        # Amount — resolve from config, then name heuristic
         amt_arr: pa.Array
         amt_col = cfg.amount_col
         if amt_col and amt_col in schema_names:
@@ -1441,8 +1457,10 @@ class GDSBuilder:
                 pc.fill_null(event_table[amt_col], 0.0), pa.float64(),
             )
         else:
-            # Try common column names
-            for name in ("amount", "value", "total", "amt"):
+            for name in (
+                "amount", "value", "total", "amt",
+                "fare_amount", "total_amount", "amount_received", "amount_paid",
+            ):
                 if name in schema_names:
                     amt_arr = pc.cast(
                         pc.fill_null(event_table[name], 0.0), pa.float64(),
@@ -2222,6 +2240,12 @@ class GDSBuilder:
                 _, stats = _build_and_write(pat_id, pat)
                 pattern_stats[pat_id] = stats
 
+        # 2.5 Precompute per-entity contagion stats for every pattern with an
+        # edge table. The runtime scanner reads this small table directly
+        # instead of replaying the full edge table on every passive_scan /
+        # composite_risk call.
+        self._build_contagion_stats(_stats_writer)
+
         # 3. Write sphere.json
         sphere_data = self._build_sphere_json(pattern_stats)
         meta_dir = self.output_path / "_gds_meta"
@@ -2241,6 +2265,106 @@ class GDSBuilder:
 
         return str(self.output_path)
 
+    def _resolve_anchor_pattern_for_edge_table(
+        self, pat_id: str, pat: _PatternReg,
+    ) -> str | None:
+        """Find which anchor pattern's geometry holds the is_anomaly bitmap
+        for entities appearing in *pat*'s edge table.
+
+        For non-event patterns the pattern is its own anchor (its edges connect
+        entities of its own line). For event patterns we walk the relations and
+        return the anchor that owns the entity line of the first matching
+        relation. Mirrors PassiveScanner._resolve_anchor_for_event so the
+        precomputed table covers exactly the keys the scanner asks about at
+        runtime.
+        """
+        if pat.pattern_type != "event":
+            return pat_id
+        for other_id, other in self._patterns.items():
+            if other.pattern_type != "anchor" or other_id == pat_id:
+                continue
+            for rel in pat.relations:
+                if rel.line_id == other.entity_line:
+                    return other_id
+        return None
+
+    def _build_contagion_stats(self, stats_writer: Any) -> None:
+        """Second pass: precompute per-entity contagion statistics for every
+        pattern that has an edge table on disk. Writes one Lance table per
+        pattern under ``_gds_meta/contagion_stats/{pattern_id}.lance`` with a
+        BTREE index on ``primary_key``.
+
+        Schema: primary_key, neighbor_count (int32), anomalous_neighbor_count
+        (int32), contagion_ratio (float32). Only entities that appear as
+        at least one edge endpoint are included, so neighbor_count is
+        always >= 1 in practice.
+
+        The runtime scanner reads this small table directly instead of replaying
+        the full edge table on every passive_scan / composite_risk call.
+        """
+        import math
+        from collections import defaultdict
+
+        import lance as _lance
+
+        for pat_id, pat in self._patterns.items():
+            edge_path = self.output_path / "edges" / pat_id / "data.lance"
+            if not edge_path.exists():
+                continue
+            anchor_pat_id = self._resolve_anchor_pattern_for_edge_table(pat_id, pat)
+            if anchor_pat_id is None:
+                continue
+            edges_ds = _lance.dataset(str(edge_path))
+            if edges_ds.count_rows() == 0:
+                continue
+            edges_table = edges_ds.scanner(columns=["from_key", "to_key"]).to_table()
+            from_arr = edges_table["from_key"].to_pylist()
+            to_arr = edges_table["to_key"].to_pylist()
+            neighbors: dict[str, set[str]] = defaultdict(set)
+            for f, t in zip(from_arr, to_arr, strict=False):
+                if f != t:
+                    neighbors[f].add(t)
+                    neighbors[t].add(f)
+            if not neighbors:
+                continue
+
+            geom_path = (
+                self.output_path / "geometry" / anchor_pat_id / "v=1" / "data.lance"
+            )
+            if not geom_path.exists():
+                continue
+            anomalous: set[str] = set()
+            geom_table = _lance.dataset(str(geom_path)).scanner(
+                columns=["primary_key", "is_anomaly"],
+            ).to_table()
+            pk_col = geom_table["primary_key"]
+            anom_col = geom_table["is_anomaly"]
+            for i in range(geom_table.num_rows):
+                if anom_col[i].as_py():
+                    anomalous.add(pk_col[i].as_py())
+
+            primary_keys = sorted(neighbors.keys())
+            nbr_counts: list[int] = []
+            anom_counts: list[int] = []
+            ratios: list[float] = []
+            for k in primary_keys:
+                nbs = neighbors[k]
+                total = len(nbs)
+                anom = len(nbs & anomalous)
+                nbr_counts.append(total)
+                anom_counts.append(anom)
+                ratios.append(anom / total if total > 0 else math.nan)
+
+            contagion_table = pa.table(
+                {
+                    "primary_key": pa.array(primary_keys, type=pa.string()),
+                    "neighbor_count": pa.array(nbr_counts, type=pa.int32()),
+                    "anomalous_neighbor_count": pa.array(anom_counts, type=pa.int32()),
+                    "contagion_ratio": pa.array(ratios, type=pa.float32()),
+                },
+            )
+            stats_writer.write_contagion_stats(pat_id, contagion_table)
+
     def incremental_update(
         self,
         pattern_id: str,
@@ -3706,3 +3830,22 @@ class GDSBuilder:
                         f"Pattern '{pat_id}' event dimension: "
                         f"column '{edim.column}' must be numeric, got '{col_type}'"
                     )
+
+            # A pattern with zero declared dimensions has no geometry to compute —
+            # delta vectors collapse to width 0. Reject up front: silent acceptance
+            # used to be possible under Lance format 2.0, but format 2.1+ panics on
+            # fixed_size_list[0] columns (lance-format/lance#5102), and an anomaly
+            # geometry without dimensions is meaningless regardless of the storage
+            # format. Tracked properties count as dimension sources because they
+            # become prop_columns in the resolved delta width.
+            if (
+                not pat.relations
+                and not pat.event_dimensions
+                and not pat.tracked_properties
+            ):
+                raise ValueError(
+                    f"Pattern '{pat_id}' has no dimensions: declare at least one "
+                    f"relation, event dimension, derived dimension, or "
+                    f"tracked_properties entry. A pattern with zero dimensions "
+                    f"cannot produce a meaningful geometry."
+                )

{hypertopos-0.2.1 → hypertopos-0.3.0}/hypertopos/cli/build.py

@@ -750,6 +750,14 @@ def _add_chain_line(
                 ts_col = col_name
                 needed_cols.append(ts_col)
                 break
+        # Type-based fallback: first column with timestamp type
+        if ts_col is None:
+            import pyarrow as pa
+            for field in event_table.schema:
+                if pa.types.is_timestamp(field.type):
+                    ts_col = field.name
+                    needed_cols.append(ts_col)
+                    break
 
         cat_col = None
         for col in ("receiving_currency", "category", "k_symbol", "type"):
@@ -759,7 +767,7 @@ def _add_chain_line(
                 break
 
         amt_col = None
-        for col in ("amount_received", "amount", "extendedprice"):
+        for col in ("amount_received", "amount", "extendedprice", "fare_amount", "total_amount"):
             if col in event_table.schema.names:
                 amt_col = col
                 needed_cols.append(amt_col)