hypertopos 0.3.1__tar.gz → 0.3.3__tar.gz

{hypertopos-0.3.1 → hypertopos-0.3.3}/CHANGELOG.md

@@ -5,6 +5,35 @@ All notable changes to hypertopos will be documented in this file.
 Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [0.3.3] — 2026-04-13
+
+### Added
+
+- `dim_mask` parameter on `find_similar_entities` — compute distance only on named dimensions. Focuses similarity search on specific aspects of geometry (e.g., only the dimensions driving an anomaly).
+- `metric` parameter on `find_similar_entities` — `"L2"` (default Euclidean) or `"cosine"` (shape similarity ignoring magnitude). Cosine compares anomaly profile direction, not severity.
+- `metric` parameter on `find_anomalies` (π5) — `"L2"` (default, pre-computed) or `"Linf"` (max single-dimension |delta|, runtime scan). Linf catches single-dimension spikes that L2 dilutes.
+- `push-mirrors.sh --from-ref` mode for incremental mirror pushes between tagged releases, with pre/post author verification guard.
+
+## [0.3.2] — 2026-04-13
+
+### Added
+
+- Generalized dimension blocks (g/t/s). `geo_properties`, `metric_properties`, and `semantic_dim` optional fields on pattern config. Geographic and metric blocks use empirical mu/sigma normalization; semantic block applies SVD-based PCA dimensionality reduction. Dimension names are prefixed (`g:lat`, `t:balance`, `s:pc0`) for identification in `explain_anomaly` output. All optional — existing configs unaffected.
+- `find_novel_entities(pattern_id)` navigator method. Ranks entities by geometric deviation from neighbor-expected position using edge table adjacency. High novelty_score = entity doesn't behave like its neighborhood. Requires a pattern with an edge table.
+- `engine/heredity.py` module: `compute_expected_delta`, `compute_novelty_score`, `compute_novelty_decomposition` — pure NumPy scoring functions for geometric heredity.
+- `builder/dim_blocks.py` module: `normalize_metric_block`, `normalize_geo_block`, `normalize_semantic_block` — normalization helpers for generalized dimension blocks.
+
+### Changed
+
+- Temporal build: `reciprocity` and `counterpart_overlap` graph features replaced per-window Arrow fallback with NumPy integer-encoded set operations. One-time entity encoding via `pc.index_in`, then per-bucket `np.unique` + `np.intersect1d` + `np.bincount`. Eliminates Arrow table construction and hash joins per temporal window.
+- Temporal build: `in_degree`/`out_degree` use single-pass Arrow `group_by([key, bucket]).aggregate([count_distinct])` across all buckets. Eliminates O(n_buckets) per-window iterations.
+- Temporal build: chunked path pre-computes derived-dim groupby tables and graph feature tensors once before the entity chunk loop. Eliminates redundant full event-table scans per chunk.
+- Geometry build: per-dimension BTREE scalar indices (`delta_dim_0` through `delta_dim_N`) replaced with Lance zone-map filtering. Saves sequential index builds proportional to dimension count.
+- Geometry build: streaming path keeps `is_anomaly` in memory instead of re-reading from Lance after finalization.
+- Lance compaction `target_rows_per_fragment` raised from 1M to 4M for temporal, geometry, and geometry chunk finalization — fewer fragments to merge during compaction.
+
 ## [0.3.1] — 2026-04-12
 
 ### Added

{hypertopos-0.3.1 → hypertopos-0.3.3}/LICENSE.md

@@ -26,7 +26,7 @@ For purposes of this License:
 - **Internal use** within an organization is not a competitive offering. Your
   organization includes entities under common control.
 
-**Change Date:** 2029-04-12
+**Change Date:** 2029-04-13
 
 **Change License:** Apache License, Version 2.0
 

{hypertopos-0.3.1 → hypertopos-0.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hypertopos
-Version: 0.3.1
+Version: 0.3.3
 Summary: Understand the structure of your data — without training machine learning models
 Project-URL: Homepage, https://github.com/hypertopos/hypertopos-py
 Project-URL: Documentation, https://github.com/hypertopos/hypertopos-py/tree/main/docs

hypertopos-0.3.3/README.md

@@ -0,0 +1,69 @@
+# hypertopos
+
+> Understand the structure of your data — without training machine learning models.
+
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+[![License: BSL 1.1](https://img.shields.io/badge/license-BSL%201.1-orange.svg)](LICENSE.md)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19482069.svg)](https://doi.org/10.5281/zenodo.19482069)
+[![PyArrow](https://img.shields.io/badge/format-PyArrow-red.svg)](https://arrow.apache.org/docs/python/)
+[![Lance](https://img.shields.io/badge/storage-Lance-blueviolet.svg)](https://github.com/lance-format/lance)
+[![MCP](https://img.shields.io/badge/MCP-compatible-green.svg)](https://modelcontextprotocol.io)
+[![Version](https://img.shields.io/badge/version-0.3.3-%235500FF.svg)](pyproject.toml)
+
+hypertopos transforms relational data into a geometric space where every entity gets a coordinate derived from its relationships. Distance from the population center reveals anomalies. Proximity between entities reveals similarity. Movement over time reveals drift. No training, no labels — geometry is computed from the data.
+
+```bash
+pip install hypertopos
+```
+
+![hypertopos overview](docs/images/hypertopos-overview.svg)
+
+## What you can do with it
+
+- Detect anomalies without training ML models or labeling data
+- Discover clusters and structural archetypes
+- Track behavioral drift over time
+- Compare populations to find what differentiates them
+- Explore datasets with AI agents that navigate rather than query
+
+## How it works
+
+You describe your data in YAML — entity types, sources, relationships. hypertopos computes population statistics and produces a sphere: pre-computed geometry stored in Apache Arrow format.
+
+Agents (or Python code) open the sphere and navigate it using twelve primitives that cover movement, clustering, anomaly detection, population comparison, and temporal analysis. Each step is stateful — where you are determines what you see next.
+
+For the full picture: [Introduction](docs/introduction.md) · [Core Concepts](docs/concepts.md) · [Quick Start](docs/quickstart.md)
+
+## Benchmarks
+
+Validated on three domains with the same engine, zero domain rules, zero labels:
+
+| Domain | Dataset | Key result |
+|--------|---------|------------|
+| Banking | Berka (Czech, real data) | 85.5% recall on loan defaults |
+| AML | IBM AML (synthetic) | 80.4% recall, zero labels |
+| Transport | NYC Yellow Taxi (7.6M trips) | 8/8 anomaly categories detected |
+
+Benchmark scripts and data preparation are included. Results are reproducible. Numbers are from the pre-0.1.0 validation run and have not been re-evaluated against recent releases.
+
+Full results: [Benchmarks](docs/benchmarks.md)
+
+## Documentation
+
+| | |
+|---|---|
+| [Introduction](docs/introduction.md) | The idea and where it stands |
+| [Quick Start](docs/quickstart.md) | Install, build, navigate |
+| [Core Concepts](docs/concepts.md) | Mathematical foundation |
+| [Configuration](docs/configuration.md) | Sphere builder YAML reference |
+| [API Reference](docs/api-reference.md) | Python API |
+| [Data Format](docs/data-format.md) | On-disk storage format |
+| [Architecture](docs/architecture.md) | Package layers and design |
+
+## Status
+
+Research-stage project. Working code, reproducible benchmarks, active development. API may change.
+
+## License
+
+[Business Source License 1.1](LICENSE.md). Free for internal use, development, testing, and research. See LICENSE.md for details.

hypertopos-0.3.3/ROADMAP.md

@@ -0,0 +1,84 @@
+# hypertopos — Roadmap
+
+> Planned direction. Priorities may change based on feedback.
+
+## Plan: 0.4.0
+
+Investigation depth + detection quality — new geometric primitives, agent trust, calibration.
+
+- **Lazy chain geometry** — on-demand chain delta vectors via sampled population calibration; `discover_chains(include_geometry=True)` supplements build-time `chain_lines` for per-entity investigation without rebuild cost.
+- **Multi-hop anomaly explanation chains** — automate the trace-to-root-cause workflow: anomalous entity → top dimension → edge entity → hub check → neighbor check → explanation chain with per-hop evidence. Collapses 4–6 tool calls into one.
+- **Theta calibration fix** — builder writes correct theta_norm to sphere.json at build time; eliminates miscalibration on mixed-type patterns.
+- **Geometric edge potential** — relationship anomaly scoring via endpoint geometric distance; flags edges connecting geometrically distant entities.
+- **Geometric dark matter** — population density gap detection; identifies coordinate-space regions that should have entities but are empty (anomaly by absence).
+- **Gradient alignment on drift** — drift direction signal (normalizing vs deteriorating) on `find_drifting_entities` and `dive_solid`.
+- **Interpretive MCP output** — computed interpretation fields on tool responses (`passive_scan`, `find_similar_entities`, `dive_solid`, `find_regime_changes`) so agents stop misreading raw numbers.
+- **Anomaly confidence via bootstrap** — `anomaly_confidence: 0–1` per entity quantifying how stable the anomaly verdict is under population perturbation.
+
+---
+
+## Plan: 0.5.0
+
+Edge-derived geometry + advanced analytics.
+
+- **Edge-derived dimensions on event patterns** — pair-level and node-derived features computed at build time as geometric dimensions.
+- **`find_motif` — structural pattern matching on edge table** — per-hop predicates for subgraph patterns invisible to flat queries.
+- **Advanced coordinate space analytics** — new methods leveraging properties unique to population-relative coordinate construction.
+
+---
+
+## 0.3.0
+
+Lance perf upgrade, FDR control, builder intelligence.
+
+- **0.3.0** — Aggregate engine rewritten around Lance SQL, precomputed contagion stats, format 2.2, edge table auto-detect fix.
+- **0.3.1** — Benjamini-Hochberg FDR control, submodular facility location, vectorized build with adaptive memory chunking.
+- **0.3.2** — NumPy graph features, chunked pre-computation, Lance compact tuning, per-dim index removal. Generalized dimension blocks (g/t/s), geometric heredity (`find_novel_entities`).
+- **0.3.3** — Agent navigation policy (investigation memory, failure guards, decision scoring). Dimension-selective similarity (`dim_mask`, `metric="cosine"`, `metric="Linf"`). Sphere-specific hardcoding removed from 6 skill guidance texts.
+
+---
+
+## 0.2.0
+
+Graph meets geometry — edge table, runtime traversal, contagion/influence.
+
+- **0.2.0** — Edge table, +11 navigator functions, contagion/influence primitives.
+- **0.2.1** — Witness cohort discovery, investigative peer ranking.
+- **0.2.2** — As-of graph reconstruction, `detect_cross_pattern_discrepancy` latency fix.
+
+---
+
+## 0.1.0
+
+First public release — full GDS stack, π1–π12, builder, MCP server, validation suite.
+
+---
+
+## Future
+
+**Detection quality**
+- Edge-derived dimensions + temporal motif matcher
+- Confidence scoring, robust estimators, multi-scale resolution — improve anomaly precision and reduce false positives on heavy-tail and multi-modal populations
+
+**Builder evolution**
+- Incremental rebuild — geometry-only without `--force` wipe
+
+**PassiveScanner evolution**
+- Native temporal source support for direct temporal inputs, without requiring manual dataset plumbing in benchmark scripts
+- Optional weighted scoring mode that uses continuous intensity instead of binary counts
+- **SphereProfiler** — autonomous sphere scanner that profiles all patterns, runs calibration sweeps across source combinations, proposes optimal PassiveScanner composition for Layer 1 surveillance.
+
+**Code refactoring**
+- Break up oversized modules into smaller, domain-focused components
+- Reduce coupling between core layers
+- Consolidate repeated orchestration logic into shared helpers
+
+**Cross-sphere capabilities**
+- Cross-sphere comparison — dimensionless metrics across independently calibrated coordinate spaces
+- What-if analysis — hypothetical edge changes producing modified coordinate vectors
+
+**Enterprise / governance**
+- Dimension access control — per-agent visibility constraints on delta dimensions
+
+**Tooling**
+- Runtime latency benchmarks in package docs

{hypertopos-0.3.1 → hypertopos-0.3.3}/SECURITY.md

@@ -1,6 +1,6 @@
 # Security
 
-## Current version: 0.3.0
+## Current version: 0.3.3
 
 hypertopos runs locally. No network services, no auth layer, no multi-tenancy.
 
@@ -9,6 +9,7 @@ hypertopos runs locally. No network services, no auth layer, no multi-tenancy.
 - **Pickle files** — chain cache (`.cache/chains_*.pkl`) uses pickle. Don't load cache files from untrusted sources.
 - **Sphere paths** — `HyperSphere.open(path)` reads from the local filesystem. Don't point it at user-controlled paths without validation.
 - **MCP server** — communicates over stdio. Not designed for network exposure.
+- **SQL injection protection** — `lance_sql_agg` validates user-controlled SQL inputs. Filter values are escaped (doubles single quotes, rejects backslash and ASCII control chars). Column identifiers are validated against `^[A-Za-z_][A-Za-z0-9_]*$`.
 
 ## Reporting
 

hypertopos-0.3.3/docs/README.md

@@ -0,0 +1,22 @@
+# hypertopos Documentation
+
+hypertopos turns relational data into navigable geometric space — a coordinate system where structure, anomalies, and change become directly observable.
+
+**New here?** Read the [Introduction](introduction.md) for the idea, then [Quick Start](quickstart.md) to try it.
+
+**Want depth?** [Core Concepts](concepts.md) covers the mathematical foundation, objects, and vocabulary.
+
+**Building something?** Jump to the reference docs below.
+
+## Documents
+
+| Document | What it covers |
+|----------|----------------|
+| [Introduction](introduction.md) | The idea, how it works, where it stands |
+| [Quick Start](quickstart.md) | Install, build a sphere, connect an agent |
+| [Core Concepts](concepts.md) | Mathematical foundation, objects, population model |
+| [Architecture](architecture.md) | Package layers, runtime flow, design decisions |
+| [Configuration](configuration.md) | Sphere builder YAML reference |
+| [API Reference](api-reference.md) | Python API — classes, methods, errors |
+| [Data Format](data-format.md) | On-disk layout, Arrow schemas, sphere.json |
+| [Benchmarks](benchmarks.md) | Validation results — banking, AML, transportation |

{hypertopos-0.3.1 → hypertopos-0.3.3}/docs/api-reference.md

@@ -133,6 +133,7 @@ with sphere.session("agent-1") as session:
 | `cluster_bridges(pattern_id, n_clusters=5, top_n_bridges=10)` | Find entities bridging geometric clusters via edge table. Runs π8 clustering then identifies cross-cluster edges and bridge entities |
 | `anomalous_edges(from_key, to_key, pattern_id, top_n=10)` | Find edges between two entities enriched with event-level geometry (delta_norm, is_anomaly). Unlike path tools which score entities (anchor), this scores individual transactions (event geometry) |
 | `find_witness_cohort(primary_key, pattern_id, top_n=10, *, config=None, edge_pattern_id=None)` | Rank entities that share the target's witness signature. Investigative peer ranking — NOT edge forecasting. Combines four signals: `exp(-distance/theta)` delta similarity, witness Jaccard overlap, trajectory cosine alignment (optional), and graded anomaly bonus from `delta_rank_pct`. Excludes entities already connected via BTREE edge lookup — this is the function's main contribution over plain ANN. Configure via `WitnessCohortConfig(weights=WitnessCohortWeights(...), candidate_pool, min_witness_overlap, min_score, use_trajectory, bidirectional_check, timestamp_cutoff)`. Returns `WitnessCohortResult` with ranked `CohortMember` items, per-component scores, exclusion counts, and reproducibility metadata |
+| `find_novel_entities(pattern_id, top_n=10, sample_size=5000)` | Rank entities by geometric deviation from neighbor-expected position using edge table adjacency. High `novelty_score` = entity doesn't behave like its neighborhood. Requires a pattern with an edge table. Returns list of dicts with `primary_key`, `novelty_score`, and per-dimension decomposition |
 
 ### Analysis and Detection
 
@@ -142,7 +143,7 @@ with sphere.session("agent-1") as session:
 |--------|-------------|
 | `explain_anomaly(primary_key, pattern_id)` | Structured investigation: severity, witness set, repair set, conformal p-value, reputation |
 | `explain_anomaly_chain(primary_key, pattern_id, max_hops)` | Trace anomaly propagation through geometric neighbors |
-| `find_similar_entities(primary_key, pattern_id, top_n)` | ANN search for nearest entities in delta-space. Returns `SimilarityResult` |
+| `find_similar_entities(primary_key, pattern_id, top_n, dim_mask, metric)` | ANN search for nearest entities in delta-space. `dim_mask`: list of dimension names to restrict distance. `metric`: `"L2"` or `"cosine"`. Returns `SimilarityResult` |
 | `contrast_populations(pattern_id, group_a, group_b)` | Dimension-by-dimension comparison of two entity groups (Cohen's d) |
 | `composite_risk(primary_key, line_id)` | Fisher's method combination of conformal p-values across patterns |
 | `composite_risk_batch(primary_keys, line_id)` | Batch Fisher combination for multiple entities |

{hypertopos-0.3.1 → hypertopos-0.3.3}/docs/configuration.md

@@ -394,6 +394,40 @@ patterns:
 
 To skip edge table emission entirely (e.g. for faster iterative builds), use the `--no-edges` CLI flag.
 
+### Generalized Dimension Blocks
+
+Patterns optionally accept three additional dimension block fields that inject geographic, metric, or semantic columns from the entity line into the delta vector. All three are optional — existing configs are unaffected.
+
+```yaml
+patterns:
+  account_pattern:
+    type: anchor
+    entity_line: accounts
+    # ... existing fields (derived_dimensions, relations, etc.) ...
+
+    # Generalized dimension blocks (optional)
+    geo_properties:          # list of geographic column names from entity line
+      - latitude
+      - longitude
+    metric_properties:       # list of numeric column names from entity line
+      - balance
+      - credit_limit
+    semantic_dim:            # embedding dimensionality reduction
+      columns:
+        - description_embedding
+      n_components: 8        # target PCA dimensions
+```
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `geo_properties` | list | `null` | Geographic columns. Normalized with empirical mu/sigma. Dimensions prefixed `g:` (e.g. `g:lat`). |
+| `metric_properties` | list | `null` | Numeric columns. Normalized with empirical mu/sigma. Dimensions prefixed `t:` (e.g. `t:balance`). |
+| `semantic_dim` | dict | `null` | Embedding columns reduced via SVD-based PCA. Dimensions prefixed `s:` (e.g. `s:pc0`). |
+| `semantic_dim.columns` | list | — | **Required** (within `semantic_dim`). List of embedding column names. |
+| `semantic_dim.n_components` | int | — | **Required** (within `semantic_dim`). Target number of PCA dimensions. |
+
+Dimension prefixes (`g:`, `t:`, `s:`) appear in `explain_anomaly` output, making it clear which block a dimension belongs to.
+
 ### Multiple patterns on the same entity line
 
 You can define multiple patterns for the same anchor line with different calibration:

hypertopos-0.3.3/docs/introduction.md

@@ -0,0 +1,57 @@
+# Introduction
+
+## The problem
+
+Relational databases store entities and their relationships. SQL retrieves specific records that match filters. That works when you know what you're looking for.
+
+But some questions aren't filters:
+
+- Which entities are behaving differently from the rest of the population?
+- How is that difference changing over time?
+- What structural patterns exist that nobody thought to query for?
+
+These are exploration questions. They require context — not just a row, but how that row relates to every other row in the system. SQL doesn't naturally express this.
+
+## The idea
+
+hypertopos maps every entity in a relational dataset into a shared geometric space.
+
+The coordinate of each entity is derived from its typed relationships — who it connects to, through which channels, how often, which properties are filled or missing. This produces a position vector that captures the entity's structural identity relative to the population.
+
+From there, geometric operations become analytical operations:
+
+- **Distance from center** — anomaly detection
+- **Proximity to other entities** — similarity and clustering
+- **Movement over time** — drift tracking
+- **Group differences** — population comparison
+
+No models are trained. No labels are required. The geometry is computed directly from the relational structure of the data.
+
+## How it works
+
+The process has two phases:
+
+**Build.** You describe your data sources and their relationships in a YAML configuration. hypertopos reads the data, computes population statistics (mean, standard deviation, empirical thresholds per dimension), and produces a sphere — a directory of pre-computed geometry stored in Apache Arrow format.
+
+**Navigate.** An AI agent (or Python code) opens the sphere and moves through it. Twelve navigation primitives cover the common exploratory operations: walking along entity collections, jumping across relationships, finding clusters, detecting outliers, comparing groups, and tracking temporal change. Each step is stateful — where you are determines what you see next.
+
+The sphere is designed for on-demand loading. Opening it reads a few kilobytes of metadata. Everything else is loaded as navigation requires it.
+
+## What you can do with it
+
+- Detect anomalies without training ML models or labeling data
+- Discover clusters and structural archetypes in a population
+- Track how entities drift from their historical behavior
+- Compare groups to find which dimensions differentiate them
+- Explore unknown datasets with AI agents that navigate rather than query
+- Build investigation workflows that chain geometric operations
+
+## Where this is now
+
+hypertopos is a research-stage project with working code and reproducible benchmarks.
+
+It has been validated on three domains — banking (Berka), anti-money laundering (IBM AML), and transportation (NYC Taxi) — using the same engine with zero domain-specific rules. Benchmark scripts and data preparation are included in the repository.
+
+The library is under active development. The API may change. Contributions and feedback are welcome.
+
+For the mathematical foundation, see [Core Concepts](concepts.md). To try it, see [Quick Start](quickstart.md).

{hypertopos-0.3.1 → hypertopos-0.3.3}/docs/quickstart.md

@@ -1,6 +1,10 @@
 # Quick Start
 
-> Build a sphere from your data, then let AI agents navigate it.
+> From zero to a navigable geometric space.
+
+This guide walks you through building a sphere from a relational dataset and connecting it to an AI agent. You will define entity types and their relationships in YAML, compute population-relative coordinates for every entity, and produce a sphere that agents can explore.
+
+For the idea behind this, see [Introduction](introduction.md).
 
 ## Install
 

{hypertopos-0.3.1 → hypertopos-0.3.3}/hypertopos/builder/_writer.py

@@ -179,9 +179,9 @@ def finalize_geometry_chunks(
         return
 
     # Compact fragments from multiple appends into fewer files.
-    # target_rows_per_fragment=1M merges small chunks into large fragments
+    # target_rows_per_fragment=4M merges small chunks into large fragments
     # for better sequential scan performance (fewer random I/O ops).
-    ds.optimize.compact_files(target_rows_per_fragment=1_048_576)
+    ds.optimize.compact_files(target_rows_per_fragment=4_194_304)
     ds = _lance.dataset(str(lance_path))  # re-open after compaction
 
     # Detect list_size for index building