hypertopos 0.5.2__tar.gz → 0.6.0__tar.gz

{hypertopos-0.5.2 → hypertopos-0.6.0}/CHANGELOG.md

@@ -7,6 +7,234 @@ Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.6.0] — 2026-04-30
+
+### Theme
+
+**Architecture-emergent analytics, properly enabled.** Calibration is
+now versioned (multi-epoch retention, sphere format 2.4) and four
+patent-aligned analytics ride on that versioning:
+`compare_calibrations`, intrinsic / extrinsic drift decomposition,
+hidden-influencer matrix, cross-pattern temporal lead-lag. Companion
+features round out the release: edge-derived build-time dimensions,
+joint density gap detection, and a declarative `HopPredicate` motif
+API as power-user escape hatch from the closed-vocab `find_motif`
+registry.
+
+### Added
+- `GDSNavigator.find_motif_by_hops(pattern_id, hops, *, seed_keys, max_results, score)`
+  — declarative motif API. Caller passes a list of `HopPredicate`s
+  describing per-hop constraints (`amount_min` / `amount_max` /
+  `time_delta_max_hours` / `direction` (`"forward"` / `"reverse"` /
+  `"any"`) / `edge_dim_predicates` referencing the per-edge sidecar from
+  the edge_dimensions YAML block) and the navigator walks the edge table
+  for matching chains of length 1..6. Power-user escape hatch from the
+  closed-vocab `find_motif` registry — composes cleanly with
+  `pair_edge_count`, `position_in_chain`, `time_since_pair_last_edge`,
+  `pair_amount_zscore`, `find_motif_structuring`. New `HopPredicate`
+  frozen dataclass in `hypertopos.model.sphere` (re-exported from the
+  package root). Seeded queries (`seed_keys` provided) build a scoped
+  `AdjacencyIndex` via Lance BTREE-pushdown reads expanding the BFS
+  frontier hop-by-hop — the visited subgraph alone is materialised, not
+  the full edge table. Unseeded full enumeration uses the cached global
+  `AdjacencyIndex` so repeat calls are O(1) per node lookup. Bounded
+  MVP — `amount_ratio_to_prev` and
+  `require_anomalous_entity` predicates plus k>6 land in a follow-up.
+  Scoring path uses the existing `_score_motif_from_edges` infrastructure
+  on anchor patterns; on event patterns the score is silently skipped
+  (anchor-companion scoring is the follow-up wiring); `score` defaults to
+  `False` since only event patterns are accepted today. Direction-aware
+  temporal monotonicity: `direction="forward"` enforces strict-increasing
+  timestamps, `direction="reverse"` enforces strict-decreasing
+  (causal-predecessor chain), `direction="any"` drops the monotonic
+  constraint and treats the time window as `|Δt|`. `hops[0].time_delta_max_hours`
+  must be `None` (rejected at validation — first hop has no previous
+  timestamp); `time_delta_max_hours` on subsequent hops must be positive.
+  Sidecar lookup is materialised lazily as `ek→idx` plus per-dim parallel
+  lists (rather than a 5M-row dict-of-dicts) — avoids the per-event
+  nested-dict allocation when `edge_dim_predicates` are used.
+- `GDSNavigator.find_density_gaps(pattern_id, *, top_n, dim_pairs, bins, alpha, r_min, r_max, sample_size)`
+  — joint density gap detection via probability integral transform plus
+  independence null on dim pairs. For each pattern dim that survives the
+  usability check (≥30 finite values, σ > 0, more than 2 unique levels)
+  the empirical CDF transform produces a uniform marginal on `[0, 1]`.
+  Pairs whose Pearson `|r|` lands inside the configurable
+  `[r_min, r_max]` window become the test set. Per pair the `bins x bins`
+  joint histogram is compared against the uniform-independence
+  expectation `N / bins²` via per-cell chi² residual; only
+  under-populated cells are kept and Benjamini-Hochberg correction is
+  applied across all under-populated cells to control FDR. Each flagged
+  cell maps back to a named delta-space (z-score) range via
+  `ECDFEntry.inverse` — values live on geometry-delta units; raw
+  property unit mapping is deferred follow-up.
+  Returns a dict with `gaps` (sorted by `expected/observed` ratio
+  descending), `excluded_dims` reporting (with reason), and
+  `n_pairs_tested`. Anchor patterns only (event patterns are not the
+  primary use case); patterns with `< 100` entities raise. Geometry is
+  read with column projection narrowed to `["delta"]` and Lance-side
+  random sampling via `sample_size` (default `100,000`; pass `0` to
+  scan the full population). Delta matrix is built via
+  `combine_chunks().values.to_numpy()` reshape rather than
+  `to_pylist()` + `np.array`. New `engine.density_gaps` module exposes
+  `ECDFEntry`, `compute_density_gaps_for_pair`, `is_usable_for_gap`,
+  and `select_pairs_by_corr` as building blocks.
+- Edge-derived dimensions on event patterns:
+  `engine.edge_features` exposes five build-time per-edge dim functions
+  (`pair_edge_count`, `position_in_chain` with default `min_position=5`
+  and parse-time reject below 3, `time_since_pair_last_edge`,
+  `pair_amount_zscore` LOW_VAR pairs only, `find_motif_structuring`)
+  plus an orchestrator `compute_all_edge_dims(edges, config)` that runs
+  every dim listed in a config dict and returns a single Arrow table
+  keyed by `event_key`. `EDGE_DIM_KINDS` table publishes the kind tag
+  (poisson / gaussian / bernoulli) for each dim so builder calibration
+  can pick the right normaliser. New `engine.structuring` module hosts
+  the structuring motif enumerator (single-seed +
+  all-seeds sweep returning the set of `event_key`s in any motif) for
+  the build-time `find_motif_structuring` dim; the navigator runtime
+  `_enumerate_structuring` path is unchanged. New
+  `EDGE_FEATURES_SCHEMA` constant in `storage._schemas` describes the
+  per-edge sidecar Lance dataset layout. New
+  `GDSReader.read_edge_features(pattern_id)` reads
+  `_gds_meta/edge_features/{pid}/data.lance` (empty table when sidecar
+  absent), forward-compatible with the
+  HopPredicate.edge_dim_predicates query API planned in a future
+  release. YAML surface for the new dims: `patterns.<pid>.edge_dimensions`
+  list of either bare dim names or single-key dicts with overrides;
+  parsed into `EdgeDimensionsConfig` and attached to `PatternMapping`.
+  Builder integration baked the per-event dim values into the event
+  polygon `shape_snapshot` (one extra dim per declared entry) and
+  extends `dimension_kinds` so downstream Bregman / theta-norm
+  calibration treats the new dims natively. Sidecar Lance dataset at
+  `_gds_meta/edge_features/{pid}/data.lance` is written alongside,
+  forward-compatible with the planned HopPredicate.edge_dim_predicates
+  query API. Anchor patterns and event patterns without edge_dimensions
+  are byte-identical to builds without the new YAML block.
+- Multi-epoch calibration retention: builder writes per-pattern historical
+  fits to `_gds_meta/calibration_history/{pid}/v={N}.json` on every full
+  build; sphere format bumps to 2.4 with `pattern.calibration_epoch`,
+  `pattern.schema_hash`, and root `calibration_history_policy`. New reader
+  API: `read_calibration_fit`, `list_calibration_versions`,
+  `read_calibration_history_policy`. New types: `CalibrationFit`,
+  `CalibrationNotFoundError`. GC keeps the most-recent K epochs (default 5).
+  Schema drift wipes history; 2.3 spheres open read-only and migrate on
+  first 0.6.0 build.
+- `GDSNavigator.compare_calibrations(pattern_id, v_from, v_to, top_n, verbose)`
+  — per-dimension μ/σ/θ drift between two calibration epochs, with auto-resolve
+  defaults and aggregate RMS scalar. New types: `CalibrationDriftReport`,
+  `DimensionDrift`. Diagnostic for inspecting calibration shifts after a
+  builder rebuild; reads via the multi-epoch retention API added earlier in
+  this release.
+- `GDSNavigator.decompose_drift(entity_key, pattern_id, v_from, v_to, timestamp_from, timestamp_to, top_n, verbose)`
+  — per-entity intrinsic vs extrinsic decomposition of geometric drift between
+  two temporal slices viewed across two calibration epochs. Returns
+  `IntrinsicExtrinsicReport` with aggregate L2 displacements, sum-of-squares
+  `intrinsic_fraction` bounded `[0, 1]`, and ranked per-dimension breakdown.
+  New types: `IntrinsicExtrinsicReport`, `DimensionDecomposition`.
+- `find_drifting_entities` per-entity dict gains 3 additive scalar fields:
+  `intrinsic_displacement`, `extrinsic_displacement`, `intrinsic_fraction`.
+  Auto-defaults to (oldest retained, current) calibration epochs;
+  resolves to `null` per-entity when decomposition isn't computable.
+- `GDSNavigator.find_calibration_influencers(pattern_id, top_n, classify, high_threshold_pct, sample_size, verbose)`
+  — per-entity exact leave-one-out impact on coordinate system calibration with
+  4-cell classification matrix (hidden / distorter / standard_anomaly / normal).
+  Math uses exact leave-one-out via rolling Σs/Σs² (NOT first-order μ-only
+  approximation — that approximation makes the "hidden influencer" cell empty
+  by construction; corrected during brainstorm). New types:
+  `DimensionContribution`, `InfluenceEntry`, `InfluenceReport`. `verbose=True`
+  attaches `cascading_flip_count` per entry — count of OTHER entities
+  flipping is_anomaly after this entity's removal.
+- `GDSNavigator.find_group_influence(pattern_id, groups)` — caller-
+  supplied leave-set-out impact with `reinforcing_factor = total_impact_set
+  / Σ_individuals`. Reinforcing > 1 detects coordinated population-shift
+  attacks (collusion rings, duplicate-record contamination); < 1 detects
+  canceling sets. New type: `GroupInfluenceReport`.
+- `find_anomalies` MCP per-entity dict gains 2 additive scalar fields:
+  `total_impact`, `classification`. Hooked at the MCP-layer polygon→dict
+  conversion via `_attach_influence_fields_to_anomaly_entries`; resolves to
+  `null` per-entity when pattern is event-type, N<2, or storage backend
+  lacks shape-reconstruction prerequisites.
+- `GDSNavigator.find_lead_lag(pattern_a, pattern_b, *, timestamp_from, timestamp_to, cohort, min_epochs, max_lag, fdr_alpha, fdr_method, verbose, entity_key)`
+  — cross-pattern temporal lead-lag in population-relative coordinates.
+  Population-aggregated centroid drift cross-correlation (primary signal,
+  patent-line) plus mean step-magnitude volatility confirmation series with
+  `agreement` label. Per-dim D_A × D_B matrix with BH or Storey FDR over
+  Bonferroni-over-lag-adjusted p-values (`top_dim_pairs` ranked by ascending
+  q-value, full matrix in `per_dim_pairs` when `verbose=True`). Per-entity
+  drill-down via `entity_key` parameter. Time alignment by intersection of
+  pattern timestamp sets, hard floor `min_epochs=8`, default
+  `cohort="fixed"` (panel-clean centroid). Significance: peak Bonferroni
+  adjustment for the population peak (`max_corr_threshold` cut-off,
+  `bartlett_ci_95` reported alongside, `is_significant` boolean,
+  `degenerate_signal` flag set when either centroid drift series has zero
+  variance — agreement forced to `"divergent"`). Reliability label mirrors
+  `engine.forecast.reliability_label` convention (high N-1 ≥ 24, medium
+  ≥ 12, else low). New types: `LeadLagReport`, `DimPairLeadLag`. Two-phase
+  Lance scan: a column-projected meta read picks the cohort and
+  pre-validates a 1 GB tensor budget against `count_geometry_rows` upper
+  bounds, so cross-pattern queries over disjoint entity_lines (e.g. AML
+  accounts vs account_pairs) raise immediately with an actionable error
+  pointing the agent at `cohort='fixed'` or `entity_key=<id>` instead of
+  allocating gigabytes. `read_temporal_batched` gains optional `columns=`
+  projection.
+
+### Fixed
+- `find_anomalies` polygon construction crashed with
+  `TypeError: float() argument must be a string or a real number, not 'NoneType'`
+  on rows whose `delta_rank_pct` / `bregman_divergence` / `anomaly_confidence`
+  column existed but stored a null Arrow value. The reconstruction code
+  used `float(row.get(field, 0.0))` guarded by `if field in row`, but
+  `row.get` returns `None` (not the default) for null cells, so the
+  default never fired. Replaced with explicit
+  `None if row.get(field) is None else float(row[field])` across both
+  `engine.geometry._reconstruct_polygons_from_geometry_table` and the
+  two find_anomalies branches in `navigation.navigator` (full-rank and
+  rank-by-property). No agent-visible API change.
+- `aggregate_anomalies`: negative `ungrouped_anomalies` when
+  `read_points_batch` returns duplicate rows per entity across Lance
+  partitions. Added dedup via `seen_pks` set — each entity is now counted
+  in exactly one group regardless of partition layout. `ungrouped_anomalies`
+  is now guaranteed non-negative.
+- `detect_neighbor_contamination`: replaced row-by-row `as_py()` loop
+  (O(N) PyArrow element access) with vectorised `to_pylist()` for the
+  initial geometry scan. Also vectorised the small batch-read in the
+  unknown-keys path.
+- `detect_segment_shift`: removed internal `π12_attract_regime_change`
+  call used only to populate the non-essential `changepoint_date` output
+  field. Calling full changepoint detection (O(N×buckets)) for a single
+  optional output field dominated the tool's wall-clock time. Field
+  removed from output; callers needing changepoint context should call
+  `find_regime_changes` separately. Per-segment Python loop over the
+  points table also replaced with vectorised PyArrow ops — `pc.is_in`
+  for the anomalous-PK mask, `pc.count_distinct` for the high-
+  cardinality early exit, and
+  `group_by(...).aggregate([("_pk","count"),("is_anomaly","sum")])` for
+  the per-segment counts.
+- `find_high_potential_edges` (`attract_edge_potential`) on event
+  patterns with large edge tables: replaced the
+  `AdjacencyIndex.from_lance()` rebuild — which read all five edge
+  columns and called `to_pylist()` over the full table — with a direct
+  two-column PyArrow `group_by(["from_key","to_key"])` aggregate.
+  Eliminates the per-call full-table materialisation that dominated
+  cost on multi-million-edge event patterns. Added an entity-type ratio
+  guard that fires BEFORE the groupby: read the cached
+  `_gds_meta/edge_stats/<pid>.json` (built at sphere-write time) for
+  `unique_from + unique_to` and divide by `pattern.population_size`
+  (already loaded from `sphere.json`). When the ratio is < 1 % the
+  edge endpoints belong to a different entity type than the host
+  geometry (e.g. zone IDs in a trip-edge pattern whose geometry holds
+  trips); the tool returns an empty ranking without opening either the
+  Lance edge table or the geometry dataset. New
+  `GDSReader.edge_stats_cached(pid)` helper that reads the build-time
+  JSON only — never falls back to the live full-table scan that
+  `edge_table_stats` performs as a recovery path — so primitives that
+  need a cheap guard cannot accidentally trigger a scan when the cache
+  is missing.
+- `detect_trajectory_anomaly`: changed `sample_size` default from
+  `None` (full population scan) to `10,000`. Full-population temporal
+  streaming on large spheres caused multi-minute latency; the default
+  now caps at 10k entities. Pass `sample_size=0` to restore the full
+  scan.
+
 ## [0.5.2] — 2026-04-28
 
 ### Added

{hypertopos-0.5.2 → hypertopos-0.6.0}/LICENSE.md

@@ -26,7 +26,7 @@ For purposes of this License:
 - **Internal use** within an organization is not a competitive offering. Your
   organization includes entities under common control.
 
-**Change Date:** 2029-04-28
+**Change Date:** 2029-04-30
 
 **Change License:** Apache License, Version 2.0
 

{hypertopos-0.5.2 → hypertopos-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hypertopos
-Version: 0.5.2
+Version: 0.6.0
 Summary: Understand the structure of your data — without training machine learning models
 Project-URL: Homepage, https://github.com/hypertopos/hypertopos-py
 Project-URL: Documentation, https://github.com/hypertopos/hypertopos-py/tree/main/docs

{hypertopos-0.5.2 → hypertopos-0.6.0}/README.md

@@ -8,7 +8,7 @@
 [![PyArrow](https://img.shields.io/badge/format-PyArrow-red.svg)](https://arrow.apache.org/docs/python/)
 [![Lance](https://img.shields.io/badge/storage-Lance-blueviolet.svg)](https://github.com/lance-format/lance)
 [![MCP](https://img.shields.io/badge/MCP-compatible-green.svg)](https://modelcontextprotocol.io)
-[![Version](https://img.shields.io/badge/version-0.5.2-%235500FF.svg)](pyproject.toml)
+[![Version](https://img.shields.io/badge/version-0.6.0-%235500FF.svg)](pyproject.toml)
 
 hypertopos is not a database, and not a machine learning model. It is a layer that turns relational data into a coordinate system where every entity gets a position derived from its relationships and the population around it.
 

{hypertopos-0.5.2 → hypertopos-0.6.0}/ROADMAP.md

@@ -2,18 +2,25 @@
 
 > Planned direction. Priorities may change based on feedback.
 
-## Plan: 0.6.0
+## Plan: 0.6.1
 
-Theme: **architecture-emergent analytics, properly enabled.**
+Theme stays **architecture-emergent analytics, properly enabled** — 0.6.1 is a fast-follow patch that finishes the items deferred from the 0.6.0 MVP and lands the natural extensions of S1 / X1 once the 0.6.0 stress test surfaces real workloads.
 
-- Multi-epoch calibration retention in the builder (versioned μ/σ/θ per pattern) — prerequisite infrastructure that gates the three patent-aligned analytics below.
-- Intrinsic/extrinsic decomposition — split an entity's temporal position change into entity-caused vs population-caused components.
-- Entity influence on coordinate system — leave-one-out influence score; hidden-influencer detection.
-- Cross-pattern temporal lead-lag — cross-correlation between per-entity displacement series across independently calibrated patterns.
-- Edge-derived dimensions on event patterns — five build-time edge dimensions empirically validated for AML recall.
-- Joint density gap detection — distribution-free "anomaly by absence" via probability integral transform + independence null.
+- **Anchor-pattern aggregation of edge-derived dimensions** — extension of S1: aggregate the per-edge sidecar (`pair_edge_count`, `position_in_chain`, `time_since_pair_last_edge`, `pair_amount_zscore`, `find_motif_structuring`) up to the entity / pair / chain anchor patterns so account-level recall benefits from the same structural signals that lifted transaction-level recall in 0.6.0.
+- **HopPredicate full PBL vectorisation + k>6** — replace the depth-first Python walk with the full Paranjape-Benson-Leskovec δ-temporal-motif algorithm; bumps the bound on hop count and unlocks the larger temporal windows where structuring chains nest inside each other.
+- **`HopPredicate.amount_ratio_to_prev` and `require_anomalous_entity`** — the two predicate vocab items deferred from the 0.6.0 X1 MVP. `amount_ratio_to_prev` lets a hop assert "≤ 60% of the previous hop's amount" without baking absolute thresholds; `require_anomalous_entity` filters intermediates by their anchor-pattern anomaly score.
+- **`find_motif_by_hops` anchor-companion scoring** — wire the existing `_score_motif_from_edges` infrastructure to event-pattern motif results via the anchor-companion lookup, so callers get a `score` field instead of the silent 0.6.0 skip.
+- **Stress-test-driven fixes** — anything the 0.6.0 multi-sphere stress test surfaces (perf regressions, edge-case crashes, MCP serialisation gaps) lands here.
 
-Any item may slip to 0.7.0 if scope pressure requires. The multi-epoch retention infrastructure is non-negotiable for the theme.
+The 0.6.1 surface stays additive — no sphere format bump, no new tools beyond the deferred predicate fields. If any item turns out to need a format change, it slips to 0.7.0.
+
+---
+
+## 0.6.0
+
+Architecture-emergent analytics, properly enabled — calibration is now **versioned**, and the patent-aligned analytics ride on that versioning.
+
+- **0.6.0** — Multi-epoch calibration retention in the builder (sphere format 2.4, back-compat for 2.3) makes calibration `(pattern_id, version)`-keyed; the previous epoch is retained alongside the active one so all subsequent analytics can compare epochs without rebuilding the sphere. `compare_calibrations(pattern_id, v1, v2)` reports per-dim μ/σ drift between any two retained epochs. Intrinsic / extrinsic drift decomposition splits an entity's temporal position change into entity-caused vs population-caused components and ships through `decompose_drift` plus an inline path on `find_drifting_entities`. Hidden-influencer matrix (`find_group_influence`) computes leave-one-out influence on the coordinate system to surface entities whose presence reshapes the population geometry without themselves being anomalous. Cross-pattern temporal lead-lag (`find_lead_lag`) cross-correlates per-entity displacement series across independently calibrated patterns and reports significant leaders / laggards with FDR control. Edge-derived dimensions on event patterns ship five build-time per-edge dim functions (`pair_edge_count`, `position_in_chain`, `time_since_pair_last_edge`, `pair_amount_zscore` LOW_VAR pairs only, `find_motif_structuring`) baked into the polygon shape via the `edge_dimensions:` YAML block; sidecar Lance dataset at `_gds_meta/edge_features/{pid}/data.lance` enables runtime per-edge predicate filtering. Joint density gap detection (`find_density_gaps`) produces distribution-free "anomaly by absence" via probability integral transform plus independence null on dim pairs, with BH-corrected χ² residuals on a 2-D histogram. Declarative motif API (`find_motif_by_hops`) ships as a power-user escape hatch from the closed-vocab `find_motif` registry — `HopPredicate` per-hop predicates (`amount_min` / `amount_max` / `time_delta_max_hours` / `direction` ∈ {forward, reverse, any} / `edge_dim_predicates`) walk chains of length 1..6 over the cached `AdjacencyIndex`, with direction-aware temporal monotonicity (forward = strict-↑, reverse = strict-↓ causal-predecessor, any = `|Δt|` window) and a lazy ek→idx + per-dim sidecar lookup that avoids the per-event nested-dict allocation.
 
 ---
 

{hypertopos-0.5.2 → hypertopos-0.6.0}/SECURITY.md

@@ -1,6 +1,6 @@
 # Security
 
-## Current version: 0.5.2
+## Current version: 0.6.0
 
 hypertopos runs locally. No network services, no auth layer, no multi-tenancy.
 

{hypertopos-0.5.2 → hypertopos-0.6.0}/docs/api-reference.md

@@ -245,6 +245,7 @@ class RelationSpec:
 | `read_edges(pattern_id, from_keys=None, to_keys=None, timestamp_from=None, timestamp_to=None, columns=None)` | Read edge table with Lance BTREE-indexed push-down filters. Returns `pa.Table` |
 | `has_edge_table(pattern_id)` | Check if an edge table exists for a pattern. Returns `bool` |
 | `edge_table_stats(pattern_id)` | Quick statistics (row count, unique entities, timestamp/amount ranges). Returns `dict` or `None` |
+| `edge_stats_cached(pattern_id)` | Read precomputed edge_stats JSON only — never falls back to a live scan. Returns `dict` or `None` if cache missing |
 
 ### Edge Table (GDSWriter)
 
@@ -254,6 +255,299 @@ class RelationSpec:
 | `append_edges(pattern_id, new_edges)` | Append new edges to an existing Lance dataset (streaming build) |
 | `create_edge_indexes(pattern_id)` | Build BTREE indexes on `from_key` and `to_key` after streaming writes |
 
+### Edge Features Sidecar (GDSReader)
+
+| Method | Description |
+|--------|-------------|
+| `read_edge_features(pattern_id)` | Read the per-edge derived dimension sidecar at `_gds_meta/edge_features/{pattern_id}/data.lance`. Returns an empty `pa.Table` conforming to `EDGE_FEATURES_SCHEMA` when no sidecar exists (the pattern did not declare an `edge_dimensions:` block). Forward-compatibility entry for a future HopPredicate query API; current navigator primitives read the dim values from the polygon `shape_snapshot` directly. |
+
+### `GDSNavigator.find_motif_by_hops`
+
+```python
+nav.find_motif_by_hops(
+    pattern_id: str,
+    hops: list[HopPredicate],
+    *,
+    seed_keys: list[str] | None = None,
+    max_results: int = 100,
+    score: bool = True,
+) -> dict
+```
+
+Declarative motif API — power-user escape hatch from the closed-vocab `find_motif` registry. Caller passes a list of `HopPredicate`s describing per-hop constraints (`amount_min`, `amount_max`, `time_delta_max_hours`, `direction` (`"forward"` / `"reverse"` / `"any"`), `edge_dim_predicates: dict[str, tuple[op, value]]`); the navigator walks the in-memory `AdjacencyIndex` for matching chains. `seed_keys=None` enumerates from all `from_key` nodes (capped at `max_results`). Returns `{pattern_id, n_results, motifs}` with each motif carrying `nodes`, `edges` (event_keys), `timestamps`, `amounts`, `dim_values_per_hop` (only when `edge_dim_predicates` were used), optional `score` (anchor patterns only) and `score_breakdown`. Raises on anchor pattern (event-only), unknown pattern, empty hops, k>6, `max_results<1`. Bounded MVP — `amount_ratio_to_prev` and `require_anomalous_entity` predicates plus k>6 land in a follow-up.
+
+### `HopPredicate` (dataclass, frozen)
+
+Fields: `amount_min`, `amount_max`, `time_delta_max_hours`, `direction` (Literal["forward", "reverse", "any"], default `"forward"`), `edge_dim_predicates` (`dict[str, tuple[str, float]]`, e.g. `{"pair_edge_count": (">=", 20.0)}`). Operators: `<`, `<=`, `>`, `>=`, `==`.
+
+### Density Gaps Engine (`hypertopos.engine.density_gaps`)
+
+| Symbol | Description |
+|--------|-------------|
+| `ECDFEntry(sorted_values)` | Frozen dataclass with `transform(x)` (raw → uniform `[0, 1]`) and `inverse(u)` (uniform → raw). Constructed via `ECDFEntry.from_values(x)` |
+| `is_usable_for_gap(col)` | `(bool, reason)` admissibility check — rejects `too_sparse` (<30 finite), `degenerate` (σ≈0), `bernoulli_like` (≤2 unique) |
+| `select_pairs_by_corr(corr, *, r_min, r_max, top_k)` | Pick top-`top_k` dim pairs with Pearson `|r| ∈ [r_min, r_max]`, sorted by `|r|` desc |
+| `compute_density_gaps_for_pair(u_i, u_j, *, n, bins, alpha)` | Per-cell chi² residuals against uniform-independence expectation, Benjamini-Hochberg correction. Returns under-populated cells with `p_value`, `q_value`, `is_gap` |
+
+### `GDSNavigator.find_density_gaps`
+
+```python
+nav.find_density_gaps(
+    pattern_id: str,
+    *,
+    top_n: int = 10,
+    dim_pairs: list[tuple[str, str]] | None = None,
+    bins: int = 10,
+    alpha: float = 0.05,
+    r_min: float = 0.1,
+    r_max: float = 0.7,
+) -> dict
+```
+
+Returns dict with `pattern_id`, `n_entities`, `gaps` (each sorted by
+`ratio` desc with `dim_i` / `dim_j` / `delta_range_i` / `delta_range_j`
+(z-score space — geometry deltas, not raw property values) / `u_range_i`
+/ `u_range_j` / `observed` / `expected` / `q_value` / `correlation`), `excluded_dims`, and `n_pairs_tested`. Raises
+`GDSNavigationError` for unknown pattern, fewer than 100 entities,
+invalid `alpha` / `bins` / `r_min`/`r_max` / `top_n`, or unknown dim
+names in user-supplied `dim_pairs`.
+
+### Edge Features Engine (`hypertopos.engine.edge_features`)
+
+| Symbol | Description |
+|--------|-------------|
+| `EDGE_DIM_KINDS: dict[str, str]` | Per-dim Bregman kind tag (poisson / gaussian / bernoulli) keyed by dim name |
+| `compute_pair_edge_count(edges)` | edges per `(from_key, to_key)` directed pair |
+| `compute_position_in_chain(edges, *, min_position)` | depth in longest reverse-temporal chain ending at this edge; values below `min_position` zero out |
+| `compute_time_since_pair_last_edge(edges, *, burst_seconds, dormant_seconds)` | seconds since previous edge in same pair; first edge → `dormant_seconds` |
+| `compute_pair_amount_zscore(edges, *, cv_threshold, min_count)` | signed z-score of amount within LOW_VAR pairs |
+| `compute_find_motif_structuring(edges, *, time_window_hours, amt1_min, amt2_max)` | 1.0 if edge participates in any A→B→C→D structuring motif |
+| `compute_all_edge_dims(edges, config)` | orchestrator — runs each dim listed in config, returns Arrow table keyed by `event_key` |
+
+### Structuring Engine (`hypertopos.engine.structuring`)
+
+| Symbol | Description |
+|--------|-------------|
+| `enumerate_structuring_for_seed(seed, edges, *, time_window_sec, amt1_min, amt2_max, max_instances)` | Single-seed enumeration of A→B→C→D motifs anchored at `seed`; returns list of motif dicts |
+| `enumerate_structuring_event_keys(edges, *, time_window_sec, amt1_min, amt2_max)` | All-seeds sweep — returns the set of `event_key`s participating in any motif. Build-time helper for `compute_find_motif_structuring` |
+
+### Edge Dimensions YAML Parser (`hypertopos.builder.mapping`)
+
+| Symbol | Description |
+|--------|-------------|
+| `EdgeDimensionsConfig(dims: dict[str, dict])` | Parsed `edge_dimensions:` block — frozen dataclass attached to `PatternMapping.edge_dimensions` |
+| `parse_edge_dimensions(raw_list, *, pattern_type)` | Parse + validate the YAML list of dim entries (bare strings or single-key dicts). Raises `ValueError` on anchor pattern, `min_position < 3`, `cv_threshold` outside `(0, 1]`, `min_count < 2`, non-positive `amt1_min` / `amt2_max` / `time_window_hours`, negative `burst_seconds`, duplicate or unknown dim names, malformed entries |
+
+### Calibration History (GDSReader)
+
+| Method | Description |
+|--------|-------------|
+| `read_calibration_fit(pattern_id, version=None)` | Load one calibration epoch as a frozen `CalibrationFit` dataclass. `version=None` resolves to the pattern's current `calibration_epoch` from sphere.json. Raises `CalibrationNotFoundError` if the requested version does not exist on disk (trimmed by GC, or schema bump wiped history). For 2.3 spheres, `version=None` and `version=1` both reconstruct a `CalibrationFit` from the inline sphere.json fields; any `version >= 2` raises `CalibrationNotFoundError`. |
+| `list_calibration_versions(pattern_id)` | Return all available calibration epochs for a pattern, ascending. On a 2.3 sphere returns `[1]`. On a 2.4 sphere returns the integers N present in `_gds_meta/calibration_history/{pattern_id}/v={N}.json`. |
+| `read_calibration_history_policy()` | Read the `calibration_history_policy` from sphere.json. Defaults to `{"last_k": 5}` if absent. Raises `ValueError` if `last_k < 1`. |
+
+### `CalibrationFit` (dataclass, frozen)
+
+Fields: `pattern_id`, `calibration_epoch`, `schema_version`, `schema_hash`,
+`mu`, `sigma_diag`, `theta`, `population_size`, `dimension_weights`,
+`dimension_kinds`, `dim_percentiles`, `group_stats`, `gmm_components`,
+`edge_max`, `computed_at`, `last_calibrated_at`.
+
+### `CalibrationNotFoundError` (exception)
+
+Extends `GDSError`. Raised by `read_calibration_fit` when the requested epoch
+is not on disk.
+
+### Calibration drift
+
+#### `GDSNavigator.compare_calibrations(pattern_id, v_from=None, v_to=None, top_n=10, verbose=False) -> CalibrationDriftReport`
+
+Per-dimension μ/σ/θ drift between two calibration epochs of the same pattern.
+Auto-resolves: both `None` → second-to-last vs last; only `v_to=None` →
+explicit `v_from` vs latest. Returns a `CalibrationDriftReport` with an
+aggregate `overall_drift_rms` (RMS in σ units), ranked `top_drifted`
+list, and optional full `per_dimension` breakdown when `verbose=True`.
+
+Raises `ValueError` on `v_from == v_to`, single-epoch auto-resolve, or
+schema_hash mismatch (cross-schema mu vectors are not dimensionally
+comparable). `CalibrationNotFoundError` bubbles from missing versions.
+
+#### `CalibrationDriftReport` (dataclass, frozen)
+
+Fields: `pattern_id`, `v_from`, `v_to`, `schema_hash`,
+`population_size_from`, `population_size_to`, `overall_drift_rms`,
+`top_drifted: list[DimensionDrift]`, `per_dimension: list[DimensionDrift] | None`.
+
+#### `DimensionDrift` (dataclass, frozen)
+
+Fields: `dim_index`, `dim_kind`, `mu_from`, `mu_to`, `mu_delta`,
+`mu_delta_normalized` (z-score: `(mu_to - mu_from) / sigma_from` with
+`sigma_safe` guard for degenerate dims), `sigma_from`, `sigma_to`,
+`sigma_delta`, `theta_from`, `theta_to`, `theta_delta`.
+
+### Drift decomposition
+
+#### `GDSNavigator.decompose_drift(entity_key, pattern_id, v_from=None, v_to=None, timestamp_from=None, timestamp_to=None, top_n=10, verbose=False) -> IntrinsicExtrinsicReport`
+
+Decompose an entity's drift between two temporal slices into intrinsic
+(entity-driven, σ_v1-normalised shape change) and extrinsic (population-
+recalibration-driven, residual) components, viewed across two calibration
+epochs. Auto-resolves: both version args `None` → oldest retained vs current;
+both timestamp args `None` → first vs last temporal slice. Returns an
+`IntrinsicExtrinsicReport` with aggregate L2 displacements, sum-of-squares
+`intrinsic_fraction` in [0, 1], ranked `top_dimensions`, and optional full
+`per_dimension` breakdown when `verbose=True`.
+
+Raises `ValueError` on `<2` retained epochs, `v_from == v_to`, schema_hash
+mismatch, `<2` slices in window, or event pattern. `CalibrationNotFoundError`
+bubbles up from missing versions.
+
+#### `IntrinsicExtrinsicReport` (dataclass, frozen)
+
+Fields: `pattern_id`, `entity_key`, `v_from`, `v_to`, `schema_hash`,
+`timestamp_from`, `timestamp_to`, `intrinsic_displacement`,
+`extrinsic_displacement`, `total_displacement`, `intrinsic_fraction`,
+`top_dimensions: list[DimensionDecomposition]`, `per_dimension: list[DimensionDecomposition] | None`.
+
+#### `DimensionDecomposition` (dataclass, frozen)
+
+Fields: `dim_index`, `dim_kind`, `dim_label`, `total` (delta_b - delta_a),
+`intrinsic` ((s_b - s_a) / σ_v1), `extrinsic` (residual), `intrinsic_fraction`
+(per-dim sum-of-squares ratio in [0, 1]).
+
+### Influence analysis
+
+#### `GDSNavigator.find_calibration_influencers(pattern_id, top_n=10, classify="hidden", high_threshold_pct=90.0, sample_size=None, verbose=False) -> InfluenceReport`
+
+Detect entities with high influence on coordinate system calibration. Classifies
+into the 4-cell influence × anomaly matrix (hidden / distorter /
+standard_anomaly / normal); default `classify="hidden"` returns top_n entities
+with high `total_impact` but low anomaly score (the patent's primary detection
+cell — entities defining what 'normal' means without being detected as anomalous).
+
+Math: exact leave-one-out via rolling Σs/Σs². For each entity E:
+- `μ_without[i] = (Σs[i] - s_E[i]) / (N-1)`
+- `σ²_without[i] = (Σs²[i] - s_E[i]²) / (N-1) - μ_without[i]²`
+- `mu_impact = ‖(μ_full - μ_without) / σ_full_safe‖`
+- `sigma_impact = ‖(σ_full - σ_without) / σ_full_safe‖`
+- `total_impact = sqrt(mu_impact² + sigma_impact²)`
+
+Classification: `high_impact = total_impact ≥ percentile(total_impact, high_threshold_pct)`;
+`high_anomaly = ‖δ(E)‖ ≥ θ_norm`.
+
+`verbose=True` attaches per-entry `cascading_flip_count` — count of
+OTHER entities flipping `is_anomaly` after this entity's removal.
+
+Raises `ValueError` on event pattern, `N<2`, `high_threshold_pct ∉ (0, 100)`,
+invalid `classify`, or `top_n ∉ [1, 50]`.
+
+#### `GDSNavigator.find_group_influence(pattern_id, groups) -> list[GroupInfluenceReport]`
+
+Caller-supplied per-group leave-set-out impact. For each input
+group, computes the set's collective μ/σ shift plus
+`reinforcing_factor = total_impact_set / Σ_individuals`. Reinforcing > 1
+indicates members pull together (coordinated injection or duplicate-record
+contamination); < 1 indicates canceling (members offset each other).
+
+Returns `list[GroupInfluenceReport]` (input order preserved).
+
+Raises `ValueError` on event pattern, `N<3`, empty groups list, group with
+`<2` members, group `≥ N`, missing entity_key, duplicate entity in group, or
+undefined reinforcing factor (sum of individual impacts = 0).
+
+#### Additive surface on `find_anomalies` MCP tool
+
+Each per-entity polygon dict gains 2 scalar fields: `total_impact` (M4
+leave-one-out scalar) and `classification` (`"hidden"` / `"distorter"` /
+`"standard_anomaly"` / `"normal"`). Resolves to `null` per-entry when pattern
+is event-type, `N<2`, or storage backend lacks shape-reconstruction
+prerequisites — keeps batch response intact.
+
+#### `InfluenceReport` (dataclass, frozen)
+
+Fields: `pattern_id`, `pattern_version`, `population_size`, `high_threshold_pct`,
+`total_impact_threshold`, `theta_norm`, `classify_filter`,
+`cell_counts: dict[str, int]`, `entries: list[InfluenceEntry]`.
+
+#### `InfluenceEntry` (dataclass, frozen)
+
+Fields: `entity_key`, `mu_impact`, `sigma_impact`, `total_impact`, `delta_norm`,
+`classification`, `top_dim_contributions: list[DimensionContribution]`,
+`cascading_flip_count: int | None`.
+
+#### `GroupInfluenceReport` (dataclass, frozen)
+
+Fields: `pattern_id`, `pattern_version`, `group_index`, `member_count`,
+`members: list[str]`, `mu_impact_set`, `sigma_impact_set`, `total_impact_set`,
+`sum_individual_impacts`, `reinforcing_factor`,
+`top_dim_contributions: list[DimensionContribution]`.
+
+#### `DimensionContribution` (dataclass, frozen)
+
+Fields: `dim_index`, `dim_kind`, `dim_label`, `mu_shift`, `sigma_shift`,
+`contribution` (sqrt(mu_shift² + sigma_shift²)).
+
+### Cross-pattern temporal lead-lag
+
+#### `Navigator.find_lead_lag`
+
+```python
+nav.find_lead_lag(
+    pattern_a: str,
+    pattern_b: str,
+    *,
+    timestamp_from: str | None = None,
+    timestamp_to: str | None = None,
+    cohort: Literal["fixed", "all"] = "fixed",
+    min_epochs: int = 8,
+    max_lag: int | None = None,
+    fdr_alpha: float = 0.05,
+    fdr_method: Literal["bh", "storey"] = "storey",
+    verbose: bool = False,
+    entity_key: str | None = None,
+) -> LeadLagReport
+```
+
+Cross-pattern temporal lead-lag in population-relative coordinates. Both
+patterns must be `pattern_type="anchor"` and (effectively) over the same
+entity space — `cohort="fixed"` raises empty-cohort otherwise. Time alignment
+uses the intersection of pattern timestamp sets; `min_epochs` is a hard
+floor. Default `max_lag = (N - 1) // 4`.
+
+Three nested answer levels:
+
+1. **Population scalar.** `lag` and `correlation` from the cross-correlation
+   of differenced population centroid drift series. Bonferroni-adjusted peak
+   threshold (`max_corr_threshold` field) is the cut-off used by
+   `is_significant`.
+2. **Per-dim D_A × D_B matrix.** `top_dim_pairs` (top 10 by ascending
+   q-value, ties broken by descending |corr|) with full sorted matrix in
+   `per_dim_pairs` when `verbose=True`. BH or Storey FDR applied to
+   Bonferroni-over-lag-adjusted p-values across all `D_A * D_B` pairs.
+3. **Per-entity drill-down.** Pass `entity_key` to replace the population
+   centroid by that entity's own delta trajectory.
+
+#### `LeadLagReport` (dataclass, frozen)
+
+Fields: `pattern_a`, `pattern_b`, `entity_key: str | None`, `n_epochs_used`,
+`n_dropped_a`, `n_dropped_b`, `cohort_size`, `cohort_dropped: int | None`,
+`timestamp_from: datetime`, `timestamp_to: datetime`, `schema_hash_a`,
+`schema_hash_b`, `lag`, `correlation`,
+`centroid_drift_series_a/b: list[float]`, `lag_volatility`,
+`correlation_volatility`, `volatility_series_a/b: list[float]`, `agreement`
+(`"strong"` / `"weak"` / `"divergent"`), `bartlett_ci_95`,
+`max_corr_threshold`, `is_significant`, `fdr_alpha`, `fdr_method`,
+`n_dim_pairs`, `n_significant_pairs`, `top_dim_pairs: list[DimPairLeadLag]`,
+`per_dim_pairs: list[DimPairLeadLag] | None`, `reliability`
+(`"high"` / `"medium"` / `"low"`), `max_lag`,
+`correlation_by_lag: list[float]`, `coverage_warning`, `degenerate_signal`.
+
+#### `DimPairLeadLag` (dataclass, frozen)
+
+Fields: `dim_index_a`, `dim_index_b`, `dim_label_a: str | None`,
+`dim_label_b: str | None`, `lag`, `correlation`, `p_value`, `q_value`,
+`is_significant`.
+
 ---
 
 ## Model Objects

{hypertopos-0.5.2 → hypertopos-0.6.0}/docs/architecture.md

@@ -113,7 +113,7 @@ For a denser top-down view, this stack is a useful shorthand:
 
 ```text
 ┌─────────────────────────────────────┐
-│  MCP Server (hypertopos-mcp)        │  67 tools, smart detection mode
+│  MCP Server (hypertopos-mcp)        │  80 tools, smart detection mode
 ├─────────────────────────────────────┤
 │  PassiveScanner                     │  multi-source batch screening
 ├─────────────────────────────────────┤
@@ -175,7 +175,7 @@ graph LR
     ST --> E["Engine\ngeometry, delta,\nclustering, DTW"]
     E --> N["Navigator\nπ1–π12, detect_*"]
     N --> PS["PassiveScanner\nmulti-source\nbatch screening"]
-    N --> MCP["MCP Server\n67 tools\nstdio transport"]
+    N --> MCP["MCP Server\n72 tools\nstdio transport"]
     MCP --> Agent["AI Agent"]
     CT["CalibrationTracker\nonline drift\nWelford stats"] -.-> E
     M["Manifest\nMVCC version pin"] -.-> ST