hypertopos 0.3.0__tar.gz → 0.3.2__tar.gz

{hypertopos-0.3.0 → hypertopos-0.3.2}/CHANGELOG.md

@@ -5,6 +5,49 @@ All notable changes to hypertopos will be documented in this file.
 Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [0.3.2] — 2026-04-13
+
+### Added
+
+- Generalized dimension blocks (g/t/s). `geo_properties`, `metric_properties`, and `semantic_dim` optional fields on pattern config. Geographic and metric blocks use empirical mu/sigma normalization; semantic block applies SVD-based PCA dimensionality reduction. Dimension names are prefixed (`g:lat`, `t:balance`, `s:pc0`) for identification in `explain_anomaly` output. All optional — existing configs unaffected.
+- `find_novel_entities(pattern_id)` navigator method. Ranks entities by geometric deviation from neighbor-expected position using edge table adjacency. High novelty_score = entity doesn't behave like its neighborhood. Requires a pattern with an edge table.
+- `engine/heredity.py` module: `compute_expected_delta`, `compute_novelty_score`, `compute_novelty_decomposition` — pure NumPy scoring functions for geometric heredity.
+- `builder/dim_blocks.py` module: `normalize_metric_block`, `normalize_geo_block`, `normalize_semantic_block` — normalization helpers for generalized dimension blocks.
+
+### Changed
+
+- Temporal build: `reciprocity` and `counterpart_overlap` graph features replaced per-window Arrow fallback with NumPy integer-encoded set operations. One-time entity encoding via `pc.index_in`, then per-bucket `np.unique` + `np.intersect1d` + `np.bincount`. Eliminates Arrow table construction and hash joins per temporal window.
+- Temporal build: `in_degree`/`out_degree` use single-pass Arrow `group_by([key, bucket]).aggregate([count_distinct])` across all buckets. Eliminates O(n_buckets) per-window iterations.
+- Temporal build: chunked path pre-computes derived-dim groupby tables and graph feature tensors once before the entity chunk loop. Eliminates redundant full event-table scans per chunk.
+- Geometry build: per-dimension BTREE scalar indices (`delta_dim_0` through `delta_dim_N`) replaced with Lance zone-map filtering. Saves sequential index builds proportional to dimension count.
+- Geometry build: streaming path keeps `is_anomaly` in memory instead of re-reading from Lance after finalization.
+- Lance compaction `target_rows_per_fragment` raised from 1M to 4M for temporal, geometry, and geometry chunk finalization — fewer fragments to merge during compaction.
+
+## [0.3.1] — 2026-04-12
+
+### Added
+
+- Benjamini-Hochberg FDR control on attract primitives. `fdr_alpha` parameter on `pi5_attract_anomaly`, `pi6_attract_boundary`, `pi7_attract_hub`, and `pi9_attract_drift` applies BH multiple-testing correction to empirical-null p-values derived from delta rank percentile. Entities with `q_value > alpha` are removed before the top-N step. Guarantees `E[FDR] <= alpha`. Each retained entity carries a `q_value` — the minimum alpha at which it would still be retained. Off by default; legacy behavior preserved when omitted.
+- Diverse selection via submodular facility location. `select="diverse"` on the same four primitives replaces top-N ranking with a lazy-greedy maximisation of the facility location objective over pairwise cosine similarity of delta vectors. Achieves `(1 - 1/e)` approximation to the optimal K-subset. Each selected entity reports a `representativeness` count — how many population members are nearest to it among the selected set. Covers the geometric space of the result set instead of clustering around a single extreme region.
+- `engine/fdr.py` module: `benjamini_hochberg`, `empirical_p_values_from_rank`, `q_values_from_p_values` — pure NumPy, no state, no I/O.
+- `engine/selection.py` module: `lazy_greedy_facility_location`, `compute_similarity_matrix` — pure NumPy, Nemhauser-Wolsey-Fisher optimality guarantee.
+
+### Changed
+
+- Build pipeline: derived dimension scatter loops replaced with Arrow `pc.index_in` + numpy fancy indexing. Eliminates per-row Python iteration in both static (`compute_derived_batch`) and temporal (`_precompute_shape_tensor`) build paths.
+- Build pipeline: contagion stats computation (`_build_contagion_stats`) rewritten from Python loops over edge table to Arrow group_by + join. Eliminates O(E+N) Python iterations.
+- Build pipeline: rolling z-score (`_compute_max_rolling_z`) uses Welford online algorithm. Reduces time complexity from O(n_buckets² × N × D) to O(n_buckets × N × D) and memory from O(N × n_buckets × D) to O(N × D).
+- Build pipeline: temporal Arrow assembly uses single numpy reshape instead of per-bucket table construction loop.
+- Build pipeline: graph dims temporal computation uses pre-sorted event table with `searchsorted` for O(1) per-bucket slicing instead of O(E) scan per bucket.
+- Build pipeline: `build_temporal` adaptively chunks entities when the shape tensor would exceed available RAM. Auto-detects memory via platform APIs, falls back to 4 GB budget. Single-pass path (zero overhead) when tensor fits.
+
+### Fixed
+
+- `empirical_p_values_from_rank` used a `1/N` floor where N was the input array length. When the input was a small `top_n` slice (e.g. 5 entities), the floor clipped all p-values to 0.2, causing BH to reject everything at `alpha=0.05`. Now uses a fixed `1e-10` floor — the rank percentile is already computed against the full population, so the floor only needs to prevent exact zero for numerical stability.
+- `π7_attract_hub` and `π9_attract_drift` FDR computed p-values as `(N-i)/N` where N was the `top_n` result count (e.g. 10), making the best possible p-value `1/top_n` — mathematically unable to pass BH at any `alpha < 1/top_n`. Now uses population-level rank percentiles: hubs use the empirical CDF from the full scores array, drift uses the rank position relative to the total pre-truncation population.
+
 ## [0.3.0] — 2026-04-12
 
 > **Theme:** Lance 3.x perf upgrade. Aggregate engine moves from a persistent subprocess worker plus an optional external DataFusion package to the Lance scanner's built-in DataFusion executor via `LanceDataset.sql(...)`. The build-time contagion stats table replaces the runtime edge-table replay in the graph contagion scanner. Format 2.2 is the new write default.

{hypertopos-0.3.0 → hypertopos-0.3.2}/LICENSE.md

@@ -26,7 +26,7 @@ For purposes of this License:
 - **Internal use** within an organization is not a competitive offering. Your
   organization includes entities under common control.
 
-**Change Date:** 2029-04-12
+**Change Date:** 2029-04-13
 
 **Change License:** Apache License, Version 2.0
 

{hypertopos-0.3.0 → hypertopos-0.3.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hypertopos
-Version: 0.3.0
+Version: 0.3.2
 Summary: Understand the structure of your data — without training machine learning models
 Project-URL: Homepage, https://github.com/hypertopos/hypertopos-py
 Project-URL: Documentation, https://github.com/hypertopos/hypertopos-py/tree/main/docs

hypertopos-0.3.2/README.md

@@ -0,0 +1,69 @@
+# hypertopos
+
+> Understand the structure of your data — without training machine learning models.
+
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+[![License: BSL 1.1](https://img.shields.io/badge/license-BSL%201.1-orange.svg)](LICENSE.md)
+[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.19482069.svg)](https://doi.org/10.5281/zenodo.19482069)
+[![PyArrow](https://img.shields.io/badge/format-PyArrow-red.svg)](https://arrow.apache.org/docs/python/)
+[![Lance](https://img.shields.io/badge/storage-Lance-blueviolet.svg)](https://github.com/lance-format/lance)
+[![MCP](https://img.shields.io/badge/MCP-compatible-green.svg)](https://modelcontextprotocol.io)
+[![Version](https://img.shields.io/badge/version-0.3.1-%235500FF.svg)](pyproject.toml)
+
+hypertopos transforms relational data into a geometric space where every entity gets a coordinate derived from its relationships. Distance from the population center reveals anomalies. Proximity between entities reveals similarity. Movement over time reveals drift. No training, no labels — geometry is computed from the data.
+
+```bash
+pip install hypertopos
+```
+
+![hypertopos overview](docs/images/hypertopos-overview.svg)
+
+## What you can do with it
+
+- Detect anomalies without training ML models or labeling data
+- Discover clusters and structural archetypes
+- Track behavioral drift over time
+- Compare populations to find what differentiates them
+- Explore datasets with AI agents that navigate rather than query
+
+## How it works
+
+You describe your data in YAML — entity types, sources, relationships. hypertopos computes population statistics and produces a sphere: pre-computed geometry stored in Apache Arrow format.
+
+Agents (or Python code) open the sphere and navigate it using twelve primitives that cover movement, clustering, anomaly detection, population comparison, and temporal analysis. Each step is stateful — where you are determines what you see next.
+
+For the full picture: [Introduction](docs/introduction.md) · [Core Concepts](docs/concepts.md) · [Quick Start](docs/quickstart.md)
+
+## Benchmarks
+
+Validated on three domains with the same engine, zero domain rules, zero labels:
+
+| Domain | Dataset | Key result |
+|--------|---------|------------|
+| Banking | Berka (Czech, real data) | 85.5% recall on loan defaults |
+| AML | IBM AML (synthetic) | 80.4% recall, zero labels |
+| Transport | NYC Yellow Taxi (7.6M trips) | 8/8 anomaly categories detected |
+
+Benchmark scripts and data preparation are included. Results are reproducible. Numbers are from the pre-0.1.0 validation run and have not been re-evaluated against recent releases.
+
+Full results: [Benchmarks](docs/benchmarks.md)
+
+## Documentation
+
+| | |
+|---|---|
+| [Introduction](docs/introduction.md) | The idea and where it stands |
+| [Quick Start](docs/quickstart.md) | Install, build, navigate |
+| [Core Concepts](docs/concepts.md) | Mathematical foundation |
+| [Configuration](docs/configuration.md) | Sphere builder YAML reference |
+| [API Reference](docs/api-reference.md) | Python API |
+| [Data Format](docs/data-format.md) | On-disk storage format |
+| [Architecture](docs/architecture.md) | Package layers and design |
+
+## Status
+
+Research-stage project. Working code, reproducible benchmarks, active development. API may change.
+
+## License
+
+[Business Source License 1.1](LICENSE.md). Free for internal use, development, testing, and research. See LICENSE.md for details.

{hypertopos-0.3.0 → hypertopos-0.3.2}/ROADMAP.md

@@ -2,7 +2,13 @@
 
 > Planned direction. Priorities may change based on feedback. 
 
-## Current: 0.3.0
+## Current: 0.3.1
+
+FDR control, diverse anomaly summarisation, and build pipeline optimization — Benjamini-Hochberg multiple-testing correction, submodular facility location, vectorized build with adaptive memory chunking. See CHANGELOG.
+
+---
+
+## 0.3.0
 
 Lance 3.x perf upgrade — aggregate engine rewritten around Lance SQL, precomputed contagion stats, format 2.2, contagion anchor resolution fix, edge table auto-detect fix. See CHANGELOG.
 
@@ -32,25 +38,35 @@ First public release — full GDS stack, π1–π12, builder, MCP server, valida
 
 ---
 
-## Plan: 0.3.1
+## 0.3.2
 
-Two production-quality improvements on top of the 0.3.0 storage layer:
+Builder intelligence — smarter geometry, faster builds.
 
-- **FDR control on anomaly queries** — Benjamini–Hochberg `fdr_alpha` parameter + per-row `q_value` column on `find_anomalies` and friends. Replaces unbounded "top-K by `delta_norm`" with a set whose false-discovery rate is bounded by α.
-- **Diverse-coverage anomaly summarisation** — `select="diverse"` option using lazy-greedy submodular maximisation, plus a `representativeness` column ("this 1 anomaly stands in for 38 cases").
-- **Temporal build optimization** — pre-aggregate event counts before the temporal loop, eliminating per-window event table rescans.
+- **Temporal build performance** — NumPy graph features, chunked pre-computation, Lance compact tuning. Combined speedup across benchmark spheres.
+- **Geometry build performance** — per-dimension BTREE indices replaced with zone-map filtering, streaming re-read eliminated.
+- **Generalized (d+m+g+t+s) dimension blocks** — geographic, metric, and semantic dimension support in the builder beyond the current anchor/event model.
+- **Geometric heredity for cold-start entities** — `find_novel_entities` scores entities by deviation from neighbor-expected geometry using edge table adjacency.
 
 ---
 
-## Plan: 0.3.2
+## Plan: 0.3.3
+
+Agent navigation policy — skill-level changes to make agents effective navigators of geometric space.
 
-README and docs narrative restructure.
+- **Graph confirmation flow** — add witness_cohort + contagion confirmation step to investigator and fraud-investigator skills after explain_anomaly.
+- **Investigation memory** — instruct agents to track checked entities, dead ends, and promising leads in conversation state.
+- **Failure mode guards** — depth limits, anomaly strength thresholds, and force-switch after N consecutive calls on same entity.
+- **Decision scoring** — priority queue heuristic (anomaly_strength + graph_support + temporal_signal + novelty) for iterative re-ranking of investigation targets.
 
 ---
 
 ## Plan: 0.4.0
 
-TBD — picked after 0.3.2 lands.
+Edge-derived geometry — the edge table becomes a first-class geometric citizen.
+
+- **Edge-derived dimensions on event patterns** — degree, flow, velocity computed at build time as geometric dimensions. Edges feed geometry.
+- **`find_motif` — structural pattern matching on edge table** — per-hop predicates, new navigation primitive for subgraph patterns invisible to flat queries.
+- **Geometric edge potential** — relationship anomaly scoring via endpoint geometric distance. Geometry feeds edge scoring.
 
 ---
 

{hypertopos-0.3.0 → hypertopos-0.3.2}/SECURITY.md

@@ -1,6 +1,6 @@
 # Security
 
-## Current version: 0.3.0
+## Current version: 0.3.2
 
 hypertopos runs locally. No network services, no auth layer, no multi-tenancy.
 
@@ -9,6 +9,7 @@ hypertopos runs locally. No network services, no auth layer, no multi-tenancy.
 - **Pickle files** — chain cache (`.cache/chains_*.pkl`) uses pickle. Don't load cache files from untrusted sources.
 - **Sphere paths** — `HyperSphere.open(path)` reads from the local filesystem. Don't point it at user-controlled paths without validation.
 - **MCP server** — communicates over stdio. Not designed for network exposure.
+- **SQL injection protection** — `lance_sql_agg` validates user-controlled SQL inputs. Filter values are escaped (doubles single quotes, rejects backslash and ASCII control chars). Column identifiers are validated against `^[A-Za-z_][A-Za-z0-9_]*$`.
 
 ## Reporting
 

hypertopos-0.3.2/docs/README.md

@@ -0,0 +1,22 @@
+# hypertopos Documentation
+
+hypertopos turns relational data into navigable geometric space — a coordinate system where structure, anomalies, and change become directly observable.
+
+**New here?** Read the [Introduction](introduction.md) for the idea, then [Quick Start](quickstart.md) to try it.
+
+**Want depth?** [Core Concepts](concepts.md) covers the mathematical foundation, objects, and vocabulary.
+
+**Building something?** Jump to the reference docs below.
+
+## Documents
+
+| Document | What it covers |
+|----------|----------------|
+| [Introduction](introduction.md) | The idea, how it works, where it stands |
+| [Quick Start](quickstart.md) | Install, build a sphere, connect an agent |
+| [Core Concepts](concepts.md) | Mathematical foundation, objects, population model |
+| [Architecture](architecture.md) | Package layers, runtime flow, design decisions |
+| [Configuration](configuration.md) | Sphere builder YAML reference |
+| [API Reference](api-reference.md) | Python API — classes, methods, errors |
+| [Data Format](data-format.md) | On-disk layout, Arrow schemas, sphere.json |
+| [Benchmarks](benchmarks.md) | Validation results — banking, AML, transportation |

{hypertopos-0.3.0 → hypertopos-0.3.2}/docs/api-reference.md

@@ -103,9 +103,9 @@ with sphere.session("agent-1") as session:
 
 | Primitive | Method | What it does | Returns |
 |-----------|--------|--------------|---------|
-| π5 | `π5_attract_anomaly(pattern_id, radius, top_n)` | Find most anomalous polygons | `(list[Polygon], int, list, dict)` |
-| π6 | `π6_attract_boundary(alias_id, pattern_id, direction, top_n)` | Find entities nearest to alias cutting plane | `list[(Polygon, float)]` |
-| π7 | `π7_attract_hub(pattern_id, top_n, line_id_filter)` | Find entities with highest connectivity | `list[(str, int, float)]` |
+| π5 | `π5_attract_anomaly(pattern_id, radius, top_n, fdr_alpha, fdr_method, select)` | Find most anomalous polygons | `(list[Polygon], int, list, dict)` |
+| π6 | `π6_attract_boundary(alias_id, pattern_id, direction, top_n, fdr_alpha, fdr_method, select)` | Find entities nearest to alias cutting plane | `list[(Polygon, float)]` |
+| π7 | `π7_attract_hub(pattern_id, top_n, line_id_filter, fdr_alpha, fdr_method, select)` | Find entities with highest connectivity | `list[(str, int, float)]` |
 | π7+ | `π7_attract_hub_and_stats(pattern_id, top_n, line_id_filter)` | Hub ranking + population hub score statistics in one scan | `(list, dict)` |
 | π8 | `π8_attract_cluster(pattern_id, n_clusters, top_n, sample_size)` | Discover geometric archetypes via k-means++ | `list[dict]` |
 
@@ -113,7 +113,7 @@ with sphere.session("agent-1") as session:
 
 | Primitive | Method | What it does | Returns |
 |-----------|--------|--------------|---------|
-| π9 | `π9_attract_drift(pattern_id, top_n, sample_size)` | Find entities with highest temporal drift | `list[dict]` |
+| π9 | `π9_attract_drift(pattern_id, top_n, sample_size, fdr_alpha, fdr_method, select)` | Find entities with highest temporal drift | `list[dict]` |
 | π10 | `π10_attract_trajectory(primary_key, pattern_id, top_n)` | Find entities with similar temporal trajectory | `list[dict]` |
 | π11 | `π11_attract_population_compare(pattern_id, window_a_from, window_a_to, window_b_from, window_b_to)` | Compare population geometry between two time windows | `dict` |
 | π12 | `π12_attract_regime_change(pattern_id, timestamp_from, timestamp_to)` | Detect population geometry regime shifts | `list[dict]` |
@@ -133,6 +133,7 @@ with sphere.session("agent-1") as session:
 | `cluster_bridges(pattern_id, n_clusters=5, top_n_bridges=10)` | Find entities bridging geometric clusters via edge table. Runs π8 clustering then identifies cross-cluster edges and bridge entities |
 | `anomalous_edges(from_key, to_key, pattern_id, top_n=10)` | Find edges between two entities enriched with event-level geometry (delta_norm, is_anomaly). Unlike path tools which score entities (anchor), this scores individual transactions (event geometry) |
 | `find_witness_cohort(primary_key, pattern_id, top_n=10, *, config=None, edge_pattern_id=None)` | Rank entities that share the target's witness signature. Investigative peer ranking — NOT edge forecasting. Combines four signals: `exp(-distance/theta)` delta similarity, witness Jaccard overlap, trajectory cosine alignment (optional), and graded anomaly bonus from `delta_rank_pct`. Excludes entities already connected via BTREE edge lookup — this is the function's main contribution over plain ANN. Configure via `WitnessCohortConfig(weights=WitnessCohortWeights(...), candidate_pool, min_witness_overlap, min_score, use_trajectory, bidirectional_check, timestamp_cutoff)`. Returns `WitnessCohortResult` with ranked `CohortMember` items, per-component scores, exclusion counts, and reproducibility metadata |
+| `find_novel_entities(pattern_id, top_n=10, sample_size=5000)` | Rank entities by geometric deviation from neighbor-expected position using edge table adjacency. High `novelty_score` = entity doesn't behave like its neighborhood. Requires a pattern with an edge table. Returns list of dicts with `primary_key`, `novelty_score`, and per-dimension decomposition |
 
 ### Analysis and Detection
 

{hypertopos-0.3.0 → hypertopos-0.3.2}/docs/concepts.md

@@ -58,6 +58,16 @@ where μ = E[s] is the population mean, σ = max(std(s), ε) the clamped standar
 - **Cross-sphere comparison** *(planned)* — dimensionless scalar metrics from independently calibrated coordinate spaces enable comparison across separate data systems
 - **What-if analysis** *(planned)* — hypothetical edge changes produce a modified coordinate vector showing the geometric effect
 
+## Statistical Guarantees
+
+Two opt-in post-processing stages can be applied to any attract primitive that scans a population (pi5, pi6, pi7, pi9). Both are off by default — omitting them preserves the legacy behavior exactly.
+
+**FDR control.** When `fdr_alpha` is set, the result set passes through a Benjamini-Hochberg (BH) procedure. Each entity's empirical-null p-value is derived from its population rank percentile: `p = 1 - rank_pct / 100`, where `rank_pct` is the entity's position in the full population sorted by the primitive's ranking metric (delta norm for pi5/pi6, hub score for pi7, displacement for pi9). The BH procedure guarantees `E[FDR] <= alpha` under independence or positive regression dependency (PRDS). Every retained entity carries a q-value — the minimum alpha at which it would still be retained. Entities with `q_value > alpha` are removed before the top-N / selection step.
+
+**Diverse selection.** When `select="diverse"`, the top-N step is replaced by a lazy-greedy facility location algorithm that maximizes a monotone submodular objective over pairwise cosine similarity of delta vectors. The greedy selection achieves a `(1 - 1/e)` approximation to the optimal K-subset. Each selected entity reports a representativeness count — how many population members are closest to it among the selected set. The result is K entities that cover the geometric space of the result set rather than clustering around a single extreme region.
+
+Both stages compose: `fdr_alpha=0.05, select="diverse"` first filters to FDR-controlled discoveries, then picks the K most geometrically diverse among them.
+
 ## Three Scales
 
 ![Three Scales](images/three-scales.svg)

{hypertopos-0.3.0 → hypertopos-0.3.2}/docs/configuration.md

@@ -394,6 +394,40 @@ patterns:
 
 To skip edge table emission entirely (e.g. for faster iterative builds), use the `--no-edges` CLI flag.
 
+### Generalized Dimension Blocks
+
+Patterns optionally accept three additional dimension block fields that inject geographic, metric, or semantic columns from the entity line into the delta vector. All three are optional — existing configs are unaffected.
+
+```yaml
+patterns:
+  account_pattern:
+    type: anchor
+    entity_line: accounts
+    # ... existing fields (derived_dimensions, relations, etc.) ...
+
+    # Generalized dimension blocks (optional)
+    geo_properties:          # list of geographic column names from entity line
+      - latitude
+      - longitude
+    metric_properties:       # list of numeric column names from entity line
+      - balance
+      - credit_limit
+    semantic_dim:            # embedding dimensionality reduction
+      columns:
+        - description_embedding
+      n_components: 8        # target PCA dimensions
+```
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `geo_properties` | list | `null` | Geographic columns. Normalized with empirical mu/sigma. Dimensions prefixed `g:` (e.g. `g:lat`). |
+| `metric_properties` | list | `null` | Numeric columns. Normalized with empirical mu/sigma. Dimensions prefixed `t:` (e.g. `t:balance`). |
+| `semantic_dim` | dict | `null` | Embedding columns reduced via SVD-based PCA. Dimensions prefixed `s:` (e.g. `s:pc0`). |
+| `semantic_dim.columns` | list | — | **Required** (within `semantic_dim`). List of embedding column names. |
+| `semantic_dim.n_components` | int | — | **Required** (within `semantic_dim`). Target number of PCA dimensions. |
+
+Dimension prefixes (`g:`, `t:`, `s:`) appear in `explain_anomaly` output, making it clear which block a dimension belongs to.
+
 ### Multiple patterns on the same entity line
 
 You can define multiple patterns for the same anchor line with different calibration:
@@ -871,6 +905,7 @@ hypertopos build --config sphere.yaml --force --verbose
 The builder parallelizes where possible:
 - **Points write** — parallel across lines (ThreadPoolExecutor, up to 4 workers)
 - **Geometry build** — parallel across patterns (one thread per pattern)
+- **Temporal build** — parallel across patterns; adaptive memory chunking when the shape tensor exceeds available RAM (auto-detected via platform API, 4 GB fallback)
 
 ### Build timing
 

hypertopos-0.3.2/docs/introduction.md

@@ -0,0 +1,57 @@
+# Introduction
+
+## The problem
+
+Relational databases store entities and their relationships. SQL retrieves specific records that match filters. That works when you know what you're looking for.
+
+But some questions aren't filters:
+
+- Which entities are behaving differently from the rest of the population?
+- How is that difference changing over time?
+- What structural patterns exist that nobody thought to query for?
+
+These are exploration questions. They require context — not just a row, but how that row relates to every other row in the system. SQL doesn't naturally express this.
+
+## The idea
+
+hypertopos maps every entity in a relational dataset into a shared geometric space.
+
+The coordinate of each entity is derived from its typed relationships — who it connects to, through which channels, how often, which properties are filled or missing. This produces a position vector that captures the entity's structural identity relative to the population.
+
+From there, geometric operations become analytical operations:
+
+- **Distance from center** — anomaly detection
+- **Proximity to other entities** — similarity and clustering
+- **Movement over time** — drift tracking
+- **Group differences** — population comparison
+
+No models are trained. No labels are required. The geometry is computed directly from the relational structure of the data.
+
+## How it works
+
+The process has two phases:
+
+**Build.** You describe your data sources and their relationships in a YAML configuration. hypertopos reads the data, computes population statistics (mean, standard deviation, empirical thresholds per dimension), and produces a sphere — a directory of pre-computed geometry stored in Apache Arrow format.
+
+**Navigate.** An AI agent (or Python code) opens the sphere and moves through it. Twelve navigation primitives cover the common exploratory operations: walking along entity collections, jumping across relationships, finding clusters, detecting outliers, comparing groups, and tracking temporal change. Each step is stateful — where you are determines what you see next.
+
+The sphere is designed for on-demand loading. Opening it reads a few kilobytes of metadata. Everything else is loaded as navigation requires it.
+
+## What you can do with it
+
+- Detect anomalies without training ML models or labeling data
+- Discover clusters and structural archetypes in a population
+- Track how entities drift from their historical behavior
+- Compare groups to find which dimensions differentiate them
+- Explore unknown datasets with AI agents that navigate rather than query
+- Build investigation workflows that chain geometric operations
+
+## Where this is now
+
+hypertopos is a research-stage project with working code and reproducible benchmarks.
+
+It has been validated on three domains — banking (Berka), anti-money laundering (IBM AML), and transportation (NYC Taxi) — using the same engine with zero domain-specific rules. Benchmark scripts and data preparation are included in the repository.
+
+The library is under active development. The API may change. Contributions and feedback are welcome.
+
+For the mathematical foundation, see [Core Concepts](concepts.md). To try it, see [Quick Start](quickstart.md).

{hypertopos-0.3.0 → hypertopos-0.3.2}/docs/quickstart.md

@@ -1,6 +1,10 @@
 # Quick Start
 
-> Build a sphere from your data, then let AI agents navigate it.
+> From zero to a navigable geometric space.
+
+This guide walks you through building a sphere from a relational dataset and connecting it to an AI agent. You will define entity types and their relationships in YAML, compute population-relative coordinates for every entity, and produce a sphere that agents can explore.
+
+For the idea behind this, see [Introduction](introduction.md).
 
 ## Install
 

hypertopos-0.3.2/hypertopos/builder/_scatter.py

@@ -0,0 +1,72 @@
+# Copyright (C) 2026 Karol Kędzia
+# Licensed under the Business Source License 1.1 (the "License");
+# you may not use this file except in compliance with the License.
+# See LICENSE.md in the repository root for full terms.
+"""Vectorized Arrow→tensor scatter utilities.
+
+Replaces Python-level row iteration with Arrow pc.index_in + numpy
+fancy indexing. Used by both static (compute_derived_batch) and
+temporal (_precompute_shape_tensor) build paths.
+"""
+from __future__ import annotations
+
+import numpy as np
+import pyarrow as pa
+import pyarrow.compute as pc
+from numpy.typing import NDArray
+
+
+def vectorized_scatter(
+    tensor: NDArray[np.float32],
+    dim_idx: int,
+    edge_max: float,
+    anchor_keys_arr: pa.Array,
+    grouped_fk_col: pa.Array,
+    grouped_bucket_col: pa.Array,
+    grouped_values_col: pa.Array,
+) -> None:
+    if len(grouped_fk_col) == 0:
+        return
+
+    entity_idx_arr = pc.index_in(grouped_fk_col, anchor_keys_arr)
+    valid = pc.and_(pc.is_valid(entity_idx_arr), pc.is_valid(grouped_values_col))
+
+    entity_indices = entity_idx_arr.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.intp)
+    bucket_indices = grouped_bucket_col.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.intp)
+    values = grouped_values_col.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.float64)
+
+    nan_ok = ~np.isnan(values)
+    entity_indices = entity_indices[nan_ok]
+    bucket_indices = bucket_indices[nan_ok]
+    values = np.clip(values[nan_ok], 0.0, edge_max) / edge_max
+
+    tensor[entity_indices, bucket_indices, dim_idx] = values.astype(np.float32)
+
+
+def vectorized_scatter_1d(
+    result: NDArray[np.float64],
+    anchor_keys_arr: pa.Array,
+    grouped_fk_col: pa.Array,
+    grouped_values_col: pa.Array,
+) -> None:
+    if len(grouped_fk_col) == 0:
+        return
+
+    entity_idx_arr = pc.index_in(grouped_fk_col, anchor_keys_arr)
+    valid = pc.and_(pc.is_valid(entity_idx_arr), pc.is_valid(grouped_values_col))
+
+    entity_indices = entity_idx_arr.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.intp)
+    values = grouped_values_col.filter(valid).to_numpy(
+        zero_copy_only=False,
+    ).astype(np.float64)
+
+    nan_ok = ~np.isnan(values)
+    result[entity_indices[nan_ok]] = values[nan_ok]

{hypertopos-0.3.0 → hypertopos-0.3.2}/hypertopos/builder/_stats.py

@@ -116,6 +116,7 @@ def compute_stats_grouped(
     shape_vectors: np.ndarray,
     group_ids: np.ndarray,
     anomaly_percentile: float = 95.0,
+    prop_dim_start: int | None = None,
 ) -> tuple[
     dict[str, tuple[np.ndarray, np.ndarray, np.ndarray, int]],
     np.ndarray,
@@ -123,6 +124,10 @@ def compute_stats_grouped(
 ]:
     """Compute per-group population statistics.
 
+    prop_dim_start: if set, apply SIGMA_EPS_PROP floor to sigma[prop_dim_start:]
+    for each group before computing deltas. Eliminates the need for a separate
+    re-run pass in the caller.
+
     Returns:
         group_results: {group_name: (mu, sigma, theta, population_size)}
         deltas: (N, D) float32 — z-scored against per-group mu/sigma
@@ -140,9 +145,12 @@ def compute_stats_grouped(
         group_shape = shape_vectors[mask]
 
         if len(group_shape) < 2:
-            # Too few entities for meaningful stats — use global-like single point
             mu_g = group_shape.mean(axis=0).astype(np.float32)
             sigma_g = np.full(D, SIGMA_EPS, dtype=np.float32)
+            if prop_dim_start is not None:
+                sigma_g[prop_dim_start:] = np.maximum(
+                    sigma_g[prop_dim_start:], SIGMA_EPS_PROP,
+                )
             theta_g = np.zeros(D, dtype=np.float32)
             deltas[mask] = ((group_shape - mu_g) / sigma_g).astype(np.float32)
             _masked = deltas[mask]
@@ -155,6 +163,22 @@ def compute_stats_grouped(
         mu_g, sigma_g, theta_g, group_deltas, group_norms, _ = compute_stats(
             group_shape, anomaly_percentile,
         )
+
+        if prop_dim_start is not None:
+            sigma_g[prop_dim_start:] = np.maximum(
+                sigma_g[prop_dim_start:], SIGMA_EPS_PROP,
+            )
+            group_deltas = ((group_shape - mu_g) / sigma_g).astype(np.float32)
+            group_norms = np.sqrt(
+                np.einsum('ij,ij->i', group_deltas, group_deltas),
+            ).astype(np.float32)
+            g_theta_scalar = (
+                float(np.percentile(group_norms, anomaly_percentile))
+                if len(group_norms) > 1 else 0.0
+            )
+            g_comp = g_theta_scalar / np.sqrt(D) if D > 0 else 0.0
+            theta_g = np.full(D, g_comp, dtype=np.float32)
+
         deltas[mask] = group_deltas
         delta_norms[mask] = group_norms
         group_results[gid_str] = (mu_g, sigma_g, theta_g, int(mask.sum()))

{hypertopos-0.3.0 → hypertopos-0.3.2}/hypertopos/builder/_writer.py

@@ -179,9 +179,9 @@ def finalize_geometry_chunks(
         return
 
     # Compact fragments from multiple appends into fewer files.
-    # target_rows_per_fragment=1M merges small chunks into large fragments
+    # target_rows_per_fragment=4M merges small chunks into large fragments
     # for better sequential scan performance (fewer random I/O ops).
-    ds.optimize.compact_files(target_rows_per_fragment=1_048_576)
+    ds.optimize.compact_files(target_rows_per_fragment=4_194_304)
     ds = _lance.dataset(str(lance_path))  # re-open after compaction
 
     # Detect list_size for index building