hypertopos 0.3.1__tar.gz → 0.3.2__tar.gz

{hypertopos-0.3.1 → hypertopos-0.3.2}/CHANGELOG.md

@@ -5,6 +5,26 @@ All notable changes to hypertopos will be documented in this file.
 Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [0.3.2] — 2026-04-13
+
+### Added
+
+- Generalized dimension blocks (g/t/s). `geo_properties`, `metric_properties`, and `semantic_dim` optional fields on pattern config. Geographic and metric blocks use empirical mu/sigma normalization; semantic block applies SVD-based PCA dimensionality reduction. Dimension names are prefixed (`g:lat`, `t:balance`, `s:pc0`) for identification in `explain_anomaly` output. All optional — existing configs unaffected.
+- `find_novel_entities(pattern_id)` navigator method. Ranks entities by geometric deviation from neighbor-expected position using edge table adjacency. High novelty_score = entity doesn't behave like its neighborhood. Requires a pattern with an edge table.
+- `engine/heredity.py` module: `compute_expected_delta`, `compute_novelty_score`, `compute_novelty_decomposition` — pure NumPy scoring functions for geometric heredity.
+- `builder/dim_blocks.py` module: `normalize_metric_block`, `normalize_geo_block`, `normalize_semantic_block` — normalization helpers for generalized dimension blocks.
+
+### Changed
+
+- Temporal build: `reciprocity` and `counterpart_overlap` graph features replaced per-window Arrow fallback with NumPy integer-encoded set operations. One-time entity encoding via `pc.index_in`, then per-bucket `np.unique` + `np.intersect1d` + `np.bincount`. Eliminates Arrow table construction and hash joins per temporal window.
+- Temporal build: `in_degree`/`out_degree` use single-pass Arrow `group_by([key, bucket]).aggregate([count_distinct])` across all buckets. Eliminates O(n_buckets) per-window iterations.
+- Temporal build: chunked path pre-computes derived-dim groupby tables and graph feature tensors once before the entity chunk loop. Eliminates redundant full event-table scans per chunk.
+- Geometry build: per-dimension BTREE scalar indices (`delta_dim_0` through `delta_dim_N`) replaced with Lance zone-map filtering. Saves sequential index builds proportional to dimension count.
+- Geometry build: streaming path keeps `is_anomaly` in memory instead of re-reading from Lance after finalization.
+- Lance compaction `target_rows_per_fragment` raised from 1M to 4M for temporal, geometry, and geometry chunk finalization — fewer fragments to merge during compaction.
+
 ## [0.3.1] — 2026-04-12
 
 ### Added

{hypertopos-0.3.1 → hypertopos-0.3.2}/LICENSE.md

@@ -26,7 +26,7 @@ For purposes of this License:
 - **Internal use** within an organization is not a competitive offering. Your
   organization includes entities under common control.
 
-**Change Date:** 2029-04-12
+**Change Date:** 2029-04-13
 
 **Change License:** Apache License, Version 2.0
 

{hypertopos-0.3.1 → hypertopos-0.3.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hypertopos
-Version: 0.3.1
+Version: 0.3.2
 Summary: Understand the structure of your data — without training machine learning models
 Project-URL: Homepage, https://github.com/hypertopos/hypertopos-py
 Project-URL: Documentation, https://github.com/hypertopos/hypertopos-py/tree/main/docs

hypertopos-0.3.2/README.md

@@ -0,0 +1,69 @@
+# hypertopos
+
+> Understand the structure of your data — without training machine learning models.
+
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+[![License: BSL 1.1](https://img.shields.io/badge/license-BSL%201.1-orange.svg)](LICENSE.md)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19482069.svg)](https://doi.org/10.5281/zenodo.19482069)
+[![PyArrow](https://img.shields.io/badge/format-PyArrow-red.svg)](https://arrow.apache.org/docs/python/)
+[![Lance](https://img.shields.io/badge/storage-Lance-blueviolet.svg)](https://github.com/lance-format/lance)
+[![MCP](https://img.shields.io/badge/MCP-compatible-green.svg)](https://modelcontextprotocol.io)
+[![Version](https://img.shields.io/badge/version-0.3.1-%235500FF.svg)](pyproject.toml)
+
+hypertopos transforms relational data into a geometric space where every entity gets a coordinate derived from its relationships. Distance from the population center reveals anomalies. Proximity between entities reveals similarity. Movement over time reveals drift. No training, no labels — geometry is computed from the data.
+
+```bash
+pip install hypertopos
+```
+
+![hypertopos overview](docs/images/hypertopos-overview.svg)
+
+## What you can do with it
+
+- Detect anomalies without training ML models or labeling data
+- Discover clusters and structural archetypes
+- Track behavioral drift over time
+- Compare populations to find what differentiates them
+- Explore datasets with AI agents that navigate rather than query
+
+## How it works
+
+You describe your data in YAML — entity types, sources, relationships. hypertopos computes population statistics and produces a sphere: pre-computed geometry stored in Apache Arrow format.
+
+Agents (or Python code) open the sphere and navigate it using twelve primitives that cover movement, clustering, anomaly detection, population comparison, and temporal analysis. Each step is stateful — where you are determines what you see next.
+
+For the full picture: [Introduction](docs/introduction.md) · [Core Concepts](docs/concepts.md) · [Quick Start](docs/quickstart.md)
+
+## Benchmarks
+
+Validated on three domains with the same engine, zero domain rules, zero labels:
+
+| Domain | Dataset | Key result |
+|--------|---------|------------|
+| Banking | Berka (Czech, real data) | 85.5% recall on loan defaults |
+| AML | IBM AML (synthetic) | 80.4% recall, zero labels |
+| Transport | NYC Yellow Taxi (7.6M trips) | 8/8 anomaly categories detected |
+
+Benchmark scripts and data preparation are included. Results are reproducible. Numbers are from the pre-0.1.0 validation run and have not been re-evaluated against recent releases.
+
+Full results: [Benchmarks](docs/benchmarks.md)
+
+## Documentation
+
+| | |
+|---|---|
+| [Introduction](docs/introduction.md) | The idea and where it stands |
+| [Quick Start](docs/quickstart.md) | Install, build, navigate |
+| [Core Concepts](docs/concepts.md) | Mathematical foundation |
+| [Configuration](docs/configuration.md) | Sphere builder YAML reference |
+| [API Reference](docs/api-reference.md) | Python API |
+| [Data Format](docs/data-format.md) | On-disk storage format |
+| [Architecture](docs/architecture.md) | Package layers and design |
+
+## Status
+
+Research-stage project. Working code, reproducible benchmarks, active development. API may change.
+
+## License
+
+[Business Source License 1.1](LICENSE.md). Free for internal use, development, testing, and research. See LICENSE.md for details.

{hypertopos-0.3.1 → hypertopos-0.3.2}/ROADMAP.md

@@ -38,25 +38,35 @@ First public release — full GDS stack, π1–π12, builder, MCP server, valida
 
 ---
 
-## Plan: 0.3.1
+## 0.3.2
 
-Two production-quality improvements on top of the 0.3.0 storage layer:
+Builder intelligence — smarter geometry, faster builds.
 
-- **FDR control on anomaly queries** — Benjamini–Hochberg `fdr_alpha` parameter + per-row `q_value` column on `find_anomalies` and friends. Replaces unbounded "top-K by `delta_norm`" with a set whose false-discovery rate is bounded by α.
-- **Diverse-coverage anomaly summarisation** — `select="diverse"` option using lazy-greedy submodular maximisation, plus a `representativeness` column ("this 1 anomaly stands in for 38 cases").
-- **Build pipeline optimization** — vectorized scatter (Arrow `pc.index_in` + numpy), Welford rolling z-score, Arrow-native contagion stats, adaptive memory chunking for temporal build, pre-sorted graph dims bucketing.
+- **Temporal build performance** — NumPy graph features, chunked pre-computation, Lance compact tuning. Combined speedup across benchmark spheres.
+- **Geometry build performance** — per-dimension BTREE indices replaced with zone-map filtering, streaming re-read eliminated.
+- **Generalized (d+m+g+t+s) dimension blocks** — geographic, metric, and semantic dimension support in the builder beyond the current anchor/event model.
+- **Geometric heredity for cold-start entities** — `find_novel_entities` scores entities by deviation from neighbor-expected geometry using edge table adjacency.
 
 ---
 
-## Plan: 0.3.2
+## Plan: 0.3.3
 
-TBD.
+Agent navigation policy — skill-level changes to make agents effective navigators of geometric space.
+
+- **Graph confirmation flow** — add witness_cohort + contagion confirmation step to investigator and fraud-investigator skills after explain_anomaly.
+- **Investigation memory** — instruct agents to track checked entities, dead ends, and promising leads in conversation state.
+- **Failure mode guards** — depth limits, anomaly strength thresholds, and force-switch after N consecutive calls on same entity.
+- **Decision scoring** — priority queue heuristic (anomaly_strength + graph_support + temporal_signal + novelty) for iterative re-ranking of investigation targets.
 
 ---
 
 ## Plan: 0.4.0
 
-TBD.
+Edge-derived geometry — the edge table becomes a first-class geometric citizen.
+
+- **Edge-derived dimensions on event patterns** — degree, flow, velocity computed at build time as geometric dimensions. Edges feed geometry.
+- **`find_motif` — structural pattern matching on edge table** — per-hop predicates, new navigation primitive for subgraph patterns invisible to flat queries.
+- **Geometric edge potential** — relationship anomaly scoring via endpoint geometric distance. Geometry feeds edge scoring.
 
 ---
 

{hypertopos-0.3.1 → hypertopos-0.3.2}/SECURITY.md

@@ -1,6 +1,6 @@
 # Security
 
-## Current version: 0.3.0
+## Current version: 0.3.2
 
 hypertopos runs locally. No network services, no auth layer, no multi-tenancy.
 
@@ -9,6 +9,7 @@ hypertopos runs locally. No network services, no auth layer, no multi-tenancy.
 - **Pickle files** — chain cache (`.cache/chains_*.pkl`) uses pickle. Don't load cache files from untrusted sources.
 - **Sphere paths** — `HyperSphere.open(path)` reads from the local filesystem. Don't point it at user-controlled paths without validation.
 - **MCP server** — communicates over stdio. Not designed for network exposure.
+- **SQL injection protection** — `lance_sql_agg` validates user-controlled SQL inputs. Filter values are escaped (doubles single quotes, rejects backslash and ASCII control chars). Column identifiers are validated against `^[A-Za-z_][A-Za-z0-9_]*$`.
 
 ## Reporting
 

hypertopos-0.3.2/docs/README.md

@@ -0,0 +1,22 @@
+# hypertopos Documentation
+
+hypertopos turns relational data into navigable geometric space — a coordinate system where structure, anomalies, and change become directly observable.
+
+**New here?** Read the [Introduction](introduction.md) for the idea, then [Quick Start](quickstart.md) to try it.
+
+**Want depth?** [Core Concepts](concepts.md) covers the mathematical foundation, objects, and vocabulary.
+
+**Building something?** Jump to the reference docs below.
+
+## Documents
+
+| Document | What it covers |
+|----------|----------------|
+| [Introduction](introduction.md) | The idea, how it works, where it stands |
+| [Quick Start](quickstart.md) | Install, build a sphere, connect an agent |
+| [Core Concepts](concepts.md) | Mathematical foundation, objects, population model |
+| [Architecture](architecture.md) | Package layers, runtime flow, design decisions |
+| [Configuration](configuration.md) | Sphere builder YAML reference |
+| [API Reference](api-reference.md) | Python API — classes, methods, errors |
+| [Data Format](data-format.md) | On-disk layout, Arrow schemas, sphere.json |
+| [Benchmarks](benchmarks.md) | Validation results — banking, AML, transportation |

{hypertopos-0.3.1 → hypertopos-0.3.2}/docs/api-reference.md

@@ -133,6 +133,7 @@ with sphere.session("agent-1") as session:
 | `cluster_bridges(pattern_id, n_clusters=5, top_n_bridges=10)` | Find entities bridging geometric clusters via edge table. Runs π8 clustering then identifies cross-cluster edges and bridge entities |
 | `anomalous_edges(from_key, to_key, pattern_id, top_n=10)` | Find edges between two entities enriched with event-level geometry (delta_norm, is_anomaly). Unlike path tools which score entities (anchor), this scores individual transactions (event geometry) |
 | `find_witness_cohort(primary_key, pattern_id, top_n=10, *, config=None, edge_pattern_id=None)` | Rank entities that share the target's witness signature. Investigative peer ranking — NOT edge forecasting. Combines four signals: `exp(-distance/theta)` delta similarity, witness Jaccard overlap, trajectory cosine alignment (optional), and graded anomaly bonus from `delta_rank_pct`. Excludes entities already connected via BTREE edge lookup — this is the function's main contribution over plain ANN. Configure via `WitnessCohortConfig(weights=WitnessCohortWeights(...), candidate_pool, min_witness_overlap, min_score, use_trajectory, bidirectional_check, timestamp_cutoff)`. Returns `WitnessCohortResult` with ranked `CohortMember` items, per-component scores, exclusion counts, and reproducibility metadata |
+| `find_novel_entities(pattern_id, top_n=10, sample_size=5000)` | Rank entities by geometric deviation from neighbor-expected position using edge table adjacency. High `novelty_score` = entity doesn't behave like its neighborhood. Requires a pattern with an edge table. Returns list of dicts with `primary_key`, `novelty_score`, and per-dimension decomposition |
 
 ### Analysis and Detection
 

{hypertopos-0.3.1 → hypertopos-0.3.2}/docs/configuration.md

@@ -394,6 +394,40 @@ patterns:
 
 To skip edge table emission entirely (e.g. for faster iterative builds), use the `--no-edges` CLI flag.
 
+### Generalized Dimension Blocks
+
+Patterns optionally accept three additional dimension block fields that inject geographic, metric, or semantic columns from the entity line into the delta vector. All three are optional — existing configs are unaffected.
+
+```yaml
+patterns:
+  account_pattern:
+    type: anchor
+    entity_line: accounts
+    # ... existing fields (derived_dimensions, relations, etc.) ...
+
+    # Generalized dimension blocks (optional)
+    geo_properties:          # list of geographic column names from entity line
+      - latitude
+      - longitude
+    metric_properties:       # list of numeric column names from entity line
+      - balance
+      - credit_limit
+    semantic_dim:            # embedding dimensionality reduction
+      columns:
+        - description_embedding
+      n_components: 8        # target PCA dimensions
+```
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `geo_properties` | list | `null` | Geographic columns. Normalized with empirical mu/sigma. Dimensions prefixed `g:` (e.g. `g:lat`). |
+| `metric_properties` | list | `null` | Numeric columns. Normalized with empirical mu/sigma. Dimensions prefixed `t:` (e.g. `t:balance`). |
+| `semantic_dim` | dict | `null` | Embedding columns reduced via SVD-based PCA. Dimensions prefixed `s:` (e.g. `s:pc0`). |
+| `semantic_dim.columns` | list | — | **Required** (within `semantic_dim`). List of embedding column names. |
+| `semantic_dim.n_components` | int | — | **Required** (within `semantic_dim`). Target number of PCA dimensions. |
+
+Dimension prefixes (`g:`, `t:`, `s:`) appear in `explain_anomaly` output, making it clear which block a dimension belongs to.
+
 ### Multiple patterns on the same entity line
 
 You can define multiple patterns for the same anchor line with different calibration:

hypertopos-0.3.2/docs/introduction.md

@@ -0,0 +1,57 @@
+# Introduction
+
+## The problem
+
+Relational databases store entities and their relationships. SQL retrieves specific records that match filters. That works when you know what you're looking for.
+
+But some questions aren't filters:
+
+- Which entities are behaving differently from the rest of the population?
+- How is that difference changing over time?
+- What structural patterns exist that nobody thought to query for?
+
+These are exploration questions. They require context — not just a row, but how that row relates to every other row in the system. SQL doesn't naturally express this.
+
+## The idea
+
+hypertopos maps every entity in a relational dataset into a shared geometric space.
+
+The coordinate of each entity is derived from its typed relationships — who it connects to, through which channels, how often, which properties are filled or missing. This produces a position vector that captures the entity's structural identity relative to the population.
+
+From there, geometric operations become analytical operations:
+
+- **Distance from center** — anomaly detection
+- **Proximity to other entities** — similarity and clustering
+- **Movement over time** — drift tracking
+- **Group differences** — population comparison
+
+No models are trained. No labels are required. The geometry is computed directly from the relational structure of the data.
+
+## How it works
+
+The process has two phases:
+
+**Build.** You describe your data sources and their relationships in a YAML configuration. hypertopos reads the data, computes population statistics (mean, standard deviation, empirical thresholds per dimension), and produces a sphere — a directory of pre-computed geometry stored in Apache Arrow format.
+
+**Navigate.** An AI agent (or Python code) opens the sphere and moves through it. Twelve navigation primitives cover the common exploratory operations: walking along entity collections, jumping across relationships, finding clusters, detecting outliers, comparing groups, and tracking temporal change. Each step is stateful — where you are determines what you see next.
+
+The sphere is designed for on-demand loading. Opening it reads a few kilobytes of metadata. Everything else is loaded as navigation requires it.
+
+## What you can do with it
+
+- Detect anomalies without training ML models or labeling data
+- Discover clusters and structural archetypes in a population
+- Track how entities drift from their historical behavior
+- Compare groups to find which dimensions differentiate them
+- Explore unknown datasets with AI agents that navigate rather than query
+- Build investigation workflows that chain geometric operations
+
+## Where this is now
+
+hypertopos is a research-stage project with working code and reproducible benchmarks.
+
+It has been validated on three domains — banking (Berka), anti-money laundering (IBM AML), and transportation (NYC Taxi) — using the same engine with zero domain-specific rules. Benchmark scripts and data preparation are included in the repository.
+
+The library is under active development. The API may change. Contributions and feedback are welcome.
+
+For the mathematical foundation, see [Core Concepts](concepts.md). To try it, see [Quick Start](quickstart.md).

{hypertopos-0.3.1 → hypertopos-0.3.2}/docs/quickstart.md

@@ -1,6 +1,10 @@
 # Quick Start
 
-> Build a sphere from your data, then let AI agents navigate it.
+> From zero to a navigable geometric space.
+
+This guide walks you through building a sphere from a relational dataset and connecting it to an AI agent. You will define entity types and their relationships in YAML, compute population-relative coordinates for every entity, and produce a sphere that agents can explore.
+
+For the idea behind this, see [Introduction](introduction.md).
 
 ## Install
 

{hypertopos-0.3.1 → hypertopos-0.3.2}/hypertopos/builder/_writer.py

@@ -179,9 +179,9 @@ def finalize_geometry_chunks(
         return
 
     # Compact fragments from multiple appends into fewer files.
-    # target_rows_per_fragment=1M merges small chunks into large fragments
+    # target_rows_per_fragment=4M merges small chunks into large fragments
     # for better sequential scan performance (fewer random I/O ops).
-    ds.optimize.compact_files(target_rows_per_fragment=1_048_576)
+    ds.optimize.compact_files(target_rows_per_fragment=4_194_304)
     ds = _lance.dataset(str(lance_path))  # re-open after compaction
 
     # Detect list_size for index building