hypertopos 0.9.0__tar.gz → 0.9.2__tar.gz

{hypertopos-0.9.0 → hypertopos-0.9.2}/CHANGELOG.md

@@ -7,6 +7,36 @@ Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
+## [0.9.2] — 2026-10-25
+
+### Added
+
+- A Spark quickstart notebook (`docs/examples/spark-quickstart.ipynb`): labelled synthetic payments, `find_anomalies` per segment, a comparison with the labels, `explain`, `score` on a later period and the hand-off to an agent; runs on Databricks or a local Spark session.
+
+### Fixed
+
+- `sphere_of(result).lance_path()`, `mcp_config()` and `serve()` work on Windows: the sphere is copied into place without a directory rename, which Windows refused right after the copy.
+- `find_anomalies` in a Databricks serverless notebook no longer fails with `CONFIG_NOT_AVAILABLE` for `hypertopos.sphere_root`: a Spark conf that the compute refuses to read counts as not set, and the calibration goes to `HYPERTOPOS_SPHERE_ROOT` or the driver's temp directory.
+- The Spark docs and the quickstart notebook say what `No matching distribution found for hypertopos` means on Databricks (a Python older than 3.12) and how to pick a newer environment.
+
+## [0.9.1] — 2026-10-25
+
+### Added
+
+- `hypertopos.spark.find_anomalies(df, entity, counterparty, amount, time)`: one call from a Spark DataFrame of events to a DataFrame with one row per entity - distance from the population, anomaly flag, conformal p-value, rank and the top three reasons. The per-entity aggregation runs on the executors; the driver only calibrates the per-entity table and scores it. Optional per-group populations (`group_by`, with small groups pooled and a `calibration_scope` column), a calibration cache keyed by the input, and shares, spreads and ratios measured on few events pulled toward the population mean so that one or two events cannot make an entity extreme. Runs on classic clusters and on Spark Connect sessions, including Databricks serverless (environment 3 or later, Python 3.12). Install with `pip install "hypertopos[spark]"` (PySpark 3.5 or 4); see the new Spark page in the docs.
+- `hypertopos.scoring.FrozenCalibration`: fit a population calibration on an entity table and score any table against it without recalibrating; plain numpy, picklable.
+- `hypertopos spark find-anomalies` CLI: the same call from the shell, on a local Spark session or as a Databricks Python wheel task; reads a Parquet, CSV or Delta path or a table, and writes Parquet, a table, or prints the top entities (`--show N`).
+- `hypertopos.spark.explain(result, entities=None, top_n=50, dims=None)`: one row per entity and dimension - value, population mean and standard deviation (the group's, with `group_by`), signed deviation, share of the distance and direction - for the most distant entities or named ones. `hypertopos.spark.sphere_of(result)` returns the calibration behind a result: its per-entity geometry as a table, written to a table with `to_delta`, or built into a navigable sphere for the MCP server (`lance_path`, `mcp_config`, `serve`) whose geometry matches the result.
+- `hypertopos.spark.score(df, like=result)`: new events scored against a frozen calibration, aggregated and scored on the executors (no recalibration; `p_value` and `rank_pct` against the original population; unseen `group_by` groups fall back to the global calibration).
+- `hypertopos.datasets.synthetic_aml(n_accounts, n_days, laundering_share, seed)`: a deterministic, labelled synthetic payments dataset (retail and business accounts; fan-out, fan-in, cycle, scatter-gather, structuring and pass-through laundering patterns) for trying the library and comparing detectors, generated on the fly.
+- `kind:` on `precomputed_dimensions` entries in `sphere.yaml` (and `kind=` on `GDSBuilder.add_precomputed_dimension`) sets the Bregman kind of the dimension - `gaussian`, `poisson` or `bernoulli` - instead of the auto-detected one; an unknown value fails the config.
+
+### Fixed
+
+- On Linux the builder's memory budget now counts reclaimable page cache (`MemAvailable`), so a machine with a warm file cache no longer looks out of memory and builds are not split into needlessly small chunks.
+- Patterns with `metric_properties`, `geo_properties` or `semantic_dim` blocks: `Pattern.dim_labels` and `Pattern.delta_dim()` now include the block dimensions (`t:<col>`, `g:<col>`, `s:pc<k>`), so labels and widths match the stored geometry; a pattern above the chunked-build size without `group_by` no longer drops those dimensions; and a pattern made only of dimension blocks is accepted.
+- `explain_anomaly` and other polygon reads on patterns with `metric_properties`, `geo_properties` or `semantic_dim` blocks use the stored geometry instead of failing with a shape mismatch.
+
 ## [0.9.0] — 2026-10-25
 
 ### Added

{hypertopos-0.9.0 → hypertopos-0.9.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: hypertopos
-Version: 0.9.0
+Version: 0.9.2
 Summary: Understand the structure of your data — without training machine learning models
 Project-URL: Homepage, https://github.com/hypertopos/hypertopos-py
 Project-URL: Documentation, https://github.com/hypertopos/hypertopos-py/tree/main/docs
@@ -29,16 +29,21 @@ Provides-Extra: dev
 Requires-Dist: filelock; extra == 'dev'
 Requires-Dist: igraph>=0.11; extra == 'dev'
 Requires-Dist: mypy==1.19.1; extra == 'dev'
+Requires-Dist: pyspark<3.6,>=3.5; extra == 'dev'
 Requires-Dist: pytest; extra == 'dev'
 Requires-Dist: pytest-asyncio; extra == 'dev'
 Requires-Dist: pytest-cov; extra == 'dev'
 Requires-Dist: pytest-xdist; extra == 'dev'
 Requires-Dist: ruff==0.15.2; extra == 'dev'
+Requires-Dist: setuptools; (python_version >= '3.12') and extra == 'dev'
 Provides-Extra: graph
 Requires-Dist: igraph>=0.11; extra == 'graph'
 Provides-Extra: mcp
 Requires-Dist: mcp[cli]<3,>=2.3; extra == 'mcp'
 Requires-Dist: tzdata>=2024.1; extra == 'mcp'
+Provides-Extra: spark
+Requires-Dist: pyspark<5,>=3.5; extra == 'spark'
+Requires-Dist: setuptools; (python_version >= '3.12') and extra == 'spark'
 Provides-Extra: topology
 Requires-Dist: persim>=0.3.5; extra == 'topology'
 Requires-Dist: ripser>=0.6.14; extra == 'topology'
@@ -56,7 +61,7 @@ Description-Content-Type: text/markdown
 [![PyArrow](https://img.shields.io/badge/format-PyArrow-red.svg)](https://arrow.apache.org/docs/python/)
 [![Lance](https://img.shields.io/badge/storage-Lance-blueviolet.svg)](https://github.com/lance-format/lance)
 [![MCP](https://img.shields.io/badge/MCP-compatible-green.svg)](https://modelcontextprotocol.io)
-[![Version](https://img.shields.io/badge/version-0.9.0-%235500FF.svg)](pyproject.toml)
+[![Version](https://img.shields.io/badge/version-0.9.2-%235500FF.svg)](pyproject.toml)
 
 Describe your tables in a YAML file, run `hypertopos build`, and every entity gets a position relative to its own population — so "out of place", "similar", "drifting" and "different between groups" become distances you read instead of queries you write.
 
@@ -162,6 +167,60 @@ my_sphere/
 > [!NOTE]
 > Paths under `sources:` resolve relative to the YAML file, not to your shell. On Windows, `python -m hypertopos.cli.main …` works wherever the `hypertopos` script is not on `PATH`.
 
+## On Spark
+
+Event tables that live in Spark (Databricks, a lakehouse, a cluster) need no YAML: one call goes from a DataFrame of events to a DataFrame of scored entities. The per-entity aggregation runs on the executors, and only one row per entity reaches the driver.
+
+<p align="center">
+  <img src="docs/images/demo-spark.svg" width="900"
+       alt="A data scientist calls find_anomalies on a Spark DataFrame of payment events grouped by segment, shows the three most distant entities, then calls explain on A000335">
+</p>
+<p align="center"><sub><i>Real output on synthetic data. A000335 lands outside its segment's θ ring; <code>explain</code> names the dimension that moved it.</i></sub></p>
+
+```bash
+pip install "hypertopos[spark]"        # on Databricks, plain `%pip install hypertopos`
+```
+
+```python
+from hypertopos.datasets import synthetic_aml
+from hypertopos.spark import explain, find_anomalies
+
+data = synthetic_aml(n_accounts=1_000, seed=1)            # labelled synthetic payments, no download
+tx = spark.createDataFrame(data.transactions.to_pandas())
+accounts = spark.createDataFrame(data.accounts.to_pandas())
+events = tx.join(accounts.select("account", "segment"), tx.from_account == accounts.account)
+
+result = find_anomalies(events, entity="from_account", counterparty="to_account",
+                        amount="amount", time="ts", group_by="segment")
+result.orderBy("delta_norm", ascending=False).show(3)
+explain(result, entities=["A000335"]).show(2)
+```
+
+Real output (local Spark, two cores):
+
+```
++-------+----------+----------+-------+--------+------------------------------------------+
+| entity|delta_norm|is_anomaly|p_value| segment|                               top_reasons|
++-------+----------+----------+-------+--------+------------------------------------------+
+|A000335|    10.616|      true|  0.001|  retail|[{burst_max_daily, 5.31, 25.0}, ...       |
+|A000259|    10.215|      true|  0.002|  retail|[{n_out, 5.62, 30.3}, ...                 |
+|A000179|    10.208|      true|  0.003|business|[{active_days, -4.80, 22.1}, ...          |
++-------+----------+----------+-------+--------+------------------------------------------+
+
++-------+---------------+-----+-----+-----+-----+------------+---------+
+|entity |dim            |value|mu   |sigma|delta|pct_of_total|direction|
++-------+---------------+-----+-----+-----+-----+------------+---------+
+|A000335|burst_max_daily|20.0 |7.683|2.320|5.31 |25.0        |above    |
+|A000335|amount_std_out |7.583|3.771|0.779|4.89 |21.2        |above    |
++-------+---------------+-----+-----+-----+-----+------------+---------+
+```
+
+`A000335` made 20 payments on its busiest day, against about 8 for retail accounts (+5.3 σ), and its payment amounts vary far more than theirs (`amount_std_out` is log-scaled). In the labels of `data.accounts` it is one of the planted fan-out accounts. The other two are not laundering, only unusual, which is what a distance from the population measures: `p_value` 0.001 means one entity in a thousand is as far out. The labels let you measure how a detector does on the same data. `explain` shows where every number comes from. `score(new_events, like=result)` scores the next batch against this calibration on the executors, and `sphere_of(result)` hands the same geometry to an agent through the MCP server.
+> [!TIP]
+> On a laptop, create the session with `.master("local[2]").config("spark.sql.shuffle.partitions", "4")`. The default of 200 shuffle partitions is sized for a cluster.
+
+Everything else, from column roles and the 23 dimensions to time handling, the cache and the CLI, is in [hypertopos on Spark](docs/spark.md). To run the whole flow on labelled synthetic data, open the [quickstart notebook](docs/examples/spark-quickstart.ipynb) on Databricks or locally.
+
 ## See it in action
 
 Real output from the YAML above on a 40-customer, 313-order CSV pair (three orders carry amounts an order of magnitude above the rest):
@@ -280,6 +339,9 @@ hypertopos sphere branch list my_sphere                         # what exists, w
 hypertopos sphere branch create my_sphere trial --purpose "recalibrate without top anomaly"
 hypertopos sphere branch promote my_sphere trial                # replay the branch's recipes on main, then delete it
 hypertopos sphere branch delete my_sphere trial                 # discard it; main is unchanged
+
+hypertopos spark find-anomalies --input tx.parquet --entity from_account \
+    --counterparty to_account --amount amount --time ts --show 20   # events on Spark -> scored entities (hypertopos[spark])
 ```
 
 Every `hypertopos sphere …` subcommand takes `--json` for machine-readable output. See the [full command reference](#full-command-reference) below.
@@ -313,7 +375,7 @@ Every capability below comes from one decision: an entity is a point in a space
 |---|---|---|
 | Population-relative coordinates | `delta = (shape − μ) / σ` per dimension; one vector serves anomaly, clustering, drift and comparison | Learned embeddings with latent dimensions that need retraining when the data shifts |
 | Self-calibrating threshold | θ is a percentile of the population's own `delta_norm` — per group when a pattern declares `group_by_property` | Choosing a contamination rate or a hand-tuned cut-off |
-| Distribution-aware dimensions | Each dimension declares a kind (gaussian, poisson, bernoulli); `sphere_overview` flags dead, sparse, non-normal and dominant dimensions with concrete advice | One metric for every feature, and silent garbage when a feature breaks its assumption |
+| Distribution-aware dimensions | Each dimension declares a kind (gaussian, poisson, bernoulli), detected or set with `kind:` on precomputed dimensions; `sphere_overview` flags dead, sparse, non-normal and dominant dimensions with concrete advice | One metric for every feature, and silent garbage when a feature breaks its assumption |
 | Calibration epochs | Every build and `recalibrate` records an epoch; `compare_calibrations` shows per-dimension μ / σ / θ drift, `theta_sensitivity` shows whether θ sits near a cliff | No record of how the coordinate system itself moved |
 | Hidden influencers | `find_calibration_influencers` measures each entity's leave-one-out impact on the calibration and names the ones that distort it | Explaining predictions while the baseline itself is contaminated |
 
@@ -360,6 +422,9 @@ Every capability below comes from one decision: an entity is a point in a space
 | Incremental ingest | `incremental_update` adds, changes and removes entities without a full rebuild | Rebuilding from source for every batch |
 | Operations CLI | `hypertopos sphere health`, `validate` and `diff` give CI gates and drift between two builds; vector-index health is reported before trajectory tools rely on it | Hand-rolled validation scripts |
 | An agent surface | The MCP server runs over stdio or HTTP with a session per agent and compact, strict-JSON responses sized for a context window | Wrapping a database in ad-hoc tools |
+| On Spark | `hypertopos.spark.find_anomalies(df, entity=…, …)` aggregates events per entity on the executors and returns a DataFrame with distance, flag, p-value and reasons; `explain` breaks it down, `sphere_of` hands it to an agent. Runs on classic clusters and Spark Connect, including Databricks serverless | Exporting events, building features by hand and running a detector outside the cluster |
+| Frozen scoring | `FrozenCalibration` (and `hypertopos.spark.score` on executors) scores new data against a fixed population without recalibrating | Refitting the detector on every batch, so yesterday's and today's scores are not comparable |
+| Data to try it on | `hypertopos.datasets.synthetic_aml` generates a labelled payments dataset with six laundering patterns | Hunting for a public dataset before the first call |
 
 ### What changes in practice
 
@@ -434,6 +499,7 @@ See [Configuration](docs/configuration.md) for the YAML reference and [Container
 | [Data Format](docs/data-format.md) | On-disk storage format, including branches |
 | [External Chains as Anchor Lines](docs/external-chains-as-anchor-line.md) | Ingest chains found by other systems and investigate them with the same primitives |
 | [Container Deployment](docs/container-deployment.md) | Run the CLI as one-shot container jobs |
+| [hypertopos on Spark](docs/spark.md) | From a Spark DataFrame of events to per-entity geometry |
 | [Benchmarks](docs/benchmarks.md) | Validation results — banking, AML, transportation |
 
 ## Status

{hypertopos-0.9.0 → hypertopos-0.9.2}/README.md

@@ -9,7 +9,7 @@
 [![PyArrow](https://img.shields.io/badge/format-PyArrow-red.svg)](https://arrow.apache.org/docs/python/)
 [![Lance](https://img.shields.io/badge/storage-Lance-blueviolet.svg)](https://github.com/lance-format/lance)
 [![MCP](https://img.shields.io/badge/MCP-compatible-green.svg)](https://modelcontextprotocol.io)
-[![Version](https://img.shields.io/badge/version-0.9.0-%235500FF.svg)](pyproject.toml)
+[![Version](https://img.shields.io/badge/version-0.9.2-%235500FF.svg)](pyproject.toml)
 
 Describe your tables in a YAML file, run `hypertopos build`, and every entity gets a position relative to its own population — so "out of place", "similar", "drifting" and "different between groups" become distances you read instead of queries you write.
 
@@ -115,6 +115,60 @@ my_sphere/
 > [!NOTE]
 > Paths under `sources:` resolve relative to the YAML file, not to your shell. On Windows, `python -m hypertopos.cli.main …` works wherever the `hypertopos` script is not on `PATH`.
 
+## On Spark
+
+Event tables that live in Spark (Databricks, a lakehouse, a cluster) need no YAML: one call goes from a DataFrame of events to a DataFrame of scored entities. The per-entity aggregation runs on the executors, and only one row per entity reaches the driver.
+
+<p align="center">
+  <img src="docs/images/demo-spark.svg" width="900"
+       alt="A data scientist calls find_anomalies on a Spark DataFrame of payment events grouped by segment, shows the three most distant entities, then calls explain on A000335">
+</p>
+<p align="center"><sub><i>Real output on synthetic data. A000335 lands outside its segment's θ ring; <code>explain</code> names the dimension that moved it.</i></sub></p>
+
+```bash
+pip install "hypertopos[spark]"        # on Databricks, plain `%pip install hypertopos`
+```
+
+```python
+from hypertopos.datasets import synthetic_aml
+from hypertopos.spark import explain, find_anomalies
+
+data = synthetic_aml(n_accounts=1_000, seed=1)            # labelled synthetic payments, no download
+tx = spark.createDataFrame(data.transactions.to_pandas())
+accounts = spark.createDataFrame(data.accounts.to_pandas())
+events = tx.join(accounts.select("account", "segment"), tx.from_account == accounts.account)
+
+result = find_anomalies(events, entity="from_account", counterparty="to_account",
+                        amount="amount", time="ts", group_by="segment")
+result.orderBy("delta_norm", ascending=False).show(3)
+explain(result, entities=["A000335"]).show(2)
+```
+
+Real output (local Spark, two cores):
+
+```
++-------+----------+----------+-------+--------+------------------------------------------+
+| entity|delta_norm|is_anomaly|p_value| segment|                               top_reasons|
++-------+----------+----------+-------+--------+------------------------------------------+
+|A000335|    10.616|      true|  0.001|  retail|[{burst_max_daily, 5.31, 25.0}, ...       |
+|A000259|    10.215|      true|  0.002|  retail|[{n_out, 5.62, 30.3}, ...                 |
+|A000179|    10.208|      true|  0.003|business|[{active_days, -4.80, 22.1}, ...          |
++-------+----------+----------+-------+--------+------------------------------------------+
+
++-------+---------------+-----+-----+-----+-----+------------+---------+
+|entity |dim            |value|mu   |sigma|delta|pct_of_total|direction|
++-------+---------------+-----+-----+-----+-----+------------+---------+
+|A000335|burst_max_daily|20.0 |7.683|2.320|5.31 |25.0        |above    |
+|A000335|amount_std_out |7.583|3.771|0.779|4.89 |21.2        |above    |
++-------+---------------+-----+-----+-----+-----+------------+---------+
+```
+
+`A000335` made 20 payments on its busiest day, against about 8 for retail accounts (+5.3 σ), and its payment amounts vary far more than theirs (`amount_std_out` is log-scaled). In the labels of `data.accounts` it is one of the planted fan-out accounts. The other two are not laundering, only unusual, which is what a distance from the population measures: `p_value` 0.001 means one entity in a thousand is as far out. The labels let you measure how a detector does on the same data. `explain` shows where every number comes from. `score(new_events, like=result)` scores the next batch against this calibration on the executors, and `sphere_of(result)` hands the same geometry to an agent through the MCP server.
+> [!TIP]
+> On a laptop, create the session with `.master("local[2]").config("spark.sql.shuffle.partitions", "4")`. The default of 200 shuffle partitions is sized for a cluster.
+
+Everything else, from column roles and the 23 dimensions to time handling, the cache and the CLI, is in [hypertopos on Spark](docs/spark.md). To run the whole flow on labelled synthetic data, open the [quickstart notebook](docs/examples/spark-quickstart.ipynb) on Databricks or locally.
+
 ## See it in action
 
 Real output from the YAML above on a 40-customer, 313-order CSV pair (three orders carry amounts an order of magnitude above the rest):
@@ -233,6 +287,9 @@ hypertopos sphere branch list my_sphere                         # what exists, w
 hypertopos sphere branch create my_sphere trial --purpose "recalibrate without top anomaly"
 hypertopos sphere branch promote my_sphere trial                # replay the branch's recipes on main, then delete it
 hypertopos sphere branch delete my_sphere trial                 # discard it; main is unchanged
+
+hypertopos spark find-anomalies --input tx.parquet --entity from_account \
+    --counterparty to_account --amount amount --time ts --show 20   # events on Spark -> scored entities (hypertopos[spark])
 ```
 
 Every `hypertopos sphere …` subcommand takes `--json` for machine-readable output. See the [full command reference](#full-command-reference) below.
@@ -266,7 +323,7 @@ Every capability below comes from one decision: an entity is a point in a space
 |---|---|---|
 | Population-relative coordinates | `delta = (shape − μ) / σ` per dimension; one vector serves anomaly, clustering, drift and comparison | Learned embeddings with latent dimensions that need retraining when the data shifts |
 | Self-calibrating threshold | θ is a percentile of the population's own `delta_norm` — per group when a pattern declares `group_by_property` | Choosing a contamination rate or a hand-tuned cut-off |
-| Distribution-aware dimensions | Each dimension declares a kind (gaussian, poisson, bernoulli); `sphere_overview` flags dead, sparse, non-normal and dominant dimensions with concrete advice | One metric for every feature, and silent garbage when a feature breaks its assumption |
+| Distribution-aware dimensions | Each dimension declares a kind (gaussian, poisson, bernoulli), detected or set with `kind:` on precomputed dimensions; `sphere_overview` flags dead, sparse, non-normal and dominant dimensions with concrete advice | One metric for every feature, and silent garbage when a feature breaks its assumption |
 | Calibration epochs | Every build and `recalibrate` records an epoch; `compare_calibrations` shows per-dimension μ / σ / θ drift, `theta_sensitivity` shows whether θ sits near a cliff | No record of how the coordinate system itself moved |
 | Hidden influencers | `find_calibration_influencers` measures each entity's leave-one-out impact on the calibration and names the ones that distort it | Explaining predictions while the baseline itself is contaminated |
 
@@ -313,6 +370,9 @@ Every capability below comes from one decision: an entity is a point in a space
 | Incremental ingest | `incremental_update` adds, changes and removes entities without a full rebuild | Rebuilding from source for every batch |
 | Operations CLI | `hypertopos sphere health`, `validate` and `diff` give CI gates and drift between two builds; vector-index health is reported before trajectory tools rely on it | Hand-rolled validation scripts |
 | An agent surface | The MCP server runs over stdio or HTTP with a session per agent and compact, strict-JSON responses sized for a context window | Wrapping a database in ad-hoc tools |
+| On Spark | `hypertopos.spark.find_anomalies(df, entity=…, …)` aggregates events per entity on the executors and returns a DataFrame with distance, flag, p-value and reasons; `explain` breaks it down, `sphere_of` hands it to an agent. Runs on classic clusters and Spark Connect, including Databricks serverless | Exporting events, building features by hand and running a detector outside the cluster |
+| Frozen scoring | `FrozenCalibration` (and `hypertopos.spark.score` on executors) scores new data against a fixed population without recalibrating | Refitting the detector on every batch, so yesterday's and today's scores are not comparable |
+| Data to try it on | `hypertopos.datasets.synthetic_aml` generates a labelled payments dataset with six laundering patterns | Hunting for a public dataset before the first call |
 
 ### What changes in practice
 
@@ -387,6 +447,7 @@ See [Configuration](docs/configuration.md) for the YAML reference and [Container
 | [Data Format](docs/data-format.md) | On-disk storage format, including branches |
 | [External Chains as Anchor Lines](docs/external-chains-as-anchor-line.md) | Ingest chains found by other systems and investigate them with the same primitives |
 | [Container Deployment](docs/container-deployment.md) | Run the CLI as one-shot container jobs |
+| [hypertopos on Spark](docs/spark.md) | From a Spark DataFrame of events to per-entity geometry |
 | [Benchmarks](docs/benchmarks.md) | Validation results — banking, AML, transportation |
 
 ## Status

{hypertopos-0.9.0 → hypertopos-0.9.2}/SECURITY.md

@@ -1,6 +1,6 @@
 # Security
 
-## Current version: 0.9.0
+## Current version: 0.9.2
 
 hypertopos runs locally. No network services, no auth layer, no multi-tenancy.
 

{hypertopos-0.9.0 → hypertopos-0.9.2}/docs/README.md

@@ -21,4 +21,5 @@ hypertopos turns relational data into navigable geometric space — a coordinate
 | [Data Format](data-format.md) | On-disk layout, Lance schemas, branches, sphere.json |
 | [External Chains as Anchor Lines](external-chains-as-anchor-line.md) | Ingesting chains found by other systems and navigating them with the same primitives |
 | [Container Deployment](container-deployment.md) | Running the CLI as one-shot container jobs |
+| [hypertopos on Spark](spark.md) | From a Spark DataFrame of events to per-entity geometry: column roles, dimensions, time handling |
 | [Benchmarks](benchmarks.md) | Validation results — banking, AML, transportation |

{hypertopos-0.9.0 → hypertopos-0.9.2}/docs/api-reference.md

@@ -263,7 +263,7 @@ builder = GDSBuilder(
 | `add_event_dimension(pattern_id, column, edge_max)` | Add a continuous dimension to an event pattern (amounts, quantities) |
 | `add_derived_dimension(anchor_line, event_line, anchor_fk, metric, metric_col, dimension_name)` | Dimension derived from event aggregation (count, sum, max, std, mean) |
 | `add_composite_line(anchor_line, event_line, anchor_fk, ...)` | Create composite anchor line from event-anchor join |
-| `add_precomputed_dimension(anchor_line, dimension_name, edge_max)` | Dimension from a column already on the entity table |
+| `add_precomputed_dimension(anchor_line, dimension_name, edge_max="auto", percentile=99.0, display_name=None, kind=None)` | Dimension from a column already on the entity table; `kind` overrides the Bregman kind (`gaussian`, `poisson`, `bernoulli`) |
 | `add_graph_features(anchor_line, event_line, from_col, to_col, features)` | Auto-compute graph structural features (degree, reciprocity, pagerank, betweenness, community, clustering, components) |
 | `add_chain_line(line_id, chains, features)` | Create anchor line from extracted chain dicts |
 | `add_alias(alias_id, base_pattern_id, cutting_plane_dimension, cutting_plane_threshold)` | Register an alias with a cutting plane for sub-population analysis |

{hypertopos-0.9.0 → hypertopos-0.9.2}/docs/concepts.md

@@ -102,7 +102,7 @@ Every dimension in a pattern carries a `kind` tag — the distribution family us
 | `graph_features` pagerank, betweenness, clustering_coefficient | auto |
 | `graph_features` community, connected_component | auto |
 
-Override via `kind:` on individual dimension entries in `sphere.yaml`. The resulting kind list is stored in `Pattern.dimension_kinds` and surfaced in `sphere_overview` as `dimension_kinds`.
+Override via `kind:` on precomputed dimension entries in `sphere.yaml`. The resulting kind list is stored in `Pattern.dimension_kinds` and surfaced in `sphere_overview` as `dimension_kinds`.
 
 ### Dim-quality warnings
 

{hypertopos-0.9.0 → hypertopos-0.9.2}/docs/configuration.md

@@ -728,7 +728,7 @@ The builder assigns a `kind` tag to every dimension at build time. The tag drive
 | `graph_features` (`community`, `connected_component`) | auto | Integer IDs (categorical) |
 | `geo_properties`, `metric_properties` | `gaussian` | Continuous measurements |
 
-**Override via `kind:`** on individual dimension entries (precomputed, derived). Override takes precedence over auto-detection. Event dimensions use auto-detection only.
+**Override via `kind:`** on precomputed dimension entries (`gaussian`, `poisson` or `bernoulli`). The override takes precedence over auto-detection; an unknown value fails the config. Derived and event dimensions use auto-detection only.
 
 ---
 
@@ -788,6 +788,8 @@ patterns:
       - column: structuring_pct
         edge_max: 1
         display_name: structuring        # Optional label.
+      - column: n_counterparties
+        kind: poisson                    # Optional Bregman kind override.
 ```
 
 | Field | Type | Default | Description |
@@ -796,6 +798,7 @@ patterns:
 | `edge_max` | int or `"auto"` | `"auto"` | Fixed normalization cap, or auto-compute from percentile. |
 | `percentile` | float | `99.0` | Percentile for auto edge_max computation. |
 | `display_name` | string | `null` | Human-readable dimension label. |
+| `kind` | string | auto | Bregman kind: `gaussian`, `poisson` or `bernoulli`. Auto gives `bernoulli` for `edge_max: 1` and `gaussian` otherwise. |
 
 ---
 

hypertopos-0.9.2/docs/examples/spark-quickstart.ipynb

@@ -0,0 +1,222 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "cell-00",
+   "metadata": {},
+   "source": [
+    "# hypertopos on Spark: quickstart\n",
+    "\n",
+    "From a table of payments to the accounts that stand out from their population, why they do, and how to keep scoring new data against the same baseline. Runs on Databricks (Runtime 16+ or serverless environment 3+) or on a local Spark session. The data is synthetic and labelled, so you can see which flagged accounts take part in a laundering pattern."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-01",
+   "metadata": {},
+   "source": [
+    "On Databricks PySpark is already installed, so install hypertopos alone. hypertopos needs Python 3.12: on serverless, pick environment version 3 or later in the **Environment** side panel first. Locally, install it with the Spark extra instead: `%pip install \"hypertopos[spark]\"`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-02",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install hypertopos"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-03",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# On Databricks `spark` already exists. Locally, start a small session:\n",
+    "# few shuffle partitions, because the default of 200 is sized for a cluster.\n",
+    "if \"spark\" not in globals():\n",
+    "    from pyspark.sql import SparkSession\n",
+    "\n",
+    "    spark = (\n",
+    "        SparkSession.builder.master(\"local[2]\")\n",
+    "        .config(\"spark.sql.shuffle.partitions\", \"4\")\n",
+    "        .getOrCreate()\n",
+    "    )"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-04",
+   "metadata": {},
+   "source": [
+    "## 1. Data\n",
+    "\n",
+    "`synthetic_aml` generates retail and business accounts and their payments over 60 days. About 1 % of the accounts take part in one of six laundering patterns (fan-out, fan-in, cycle, scatter-gather, structuring, pass-through). Each payment carries the sender's segment, so the accounts can be compared within their segment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-05",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from hypertopos.datasets import synthetic_aml\n",
+    "\n",
+    "data = synthetic_aml(n_accounts=5_000, seed=1)\n",
+    "tx = spark.createDataFrame(data.transactions.to_pandas())\n",
+    "accounts = spark.createDataFrame(data.accounts.to_pandas())\n",
+    "events = tx.join(accounts.select(\"account\", \"segment\"), tx.from_account == accounts.account)\n",
+    "events.select(\"ts\", \"from_account\", \"to_account\", \"amount\", \"segment\").show(5)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-06",
+   "metadata": {},
+   "source": [
+    "## 2. Find anomalies\n",
+    "\n",
+    "Four column roles: who the event belongs to, the other side, the amount and the time. `group_by=\"segment\"` compares business accounts with businesses and retail with retail."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-07",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from hypertopos.spark import find_anomalies\n",
+    "\n",
+    "result = find_anomalies(\n",
+    "    events,\n",
+    "    entity=\"from_account\",\n",
+    "    counterparty=\"to_account\",\n",
+    "    amount=\"amount\",\n",
+    "    time=\"ts\",\n",
+    "    group_by=\"segment\",\n",
+    ")\n",
+    "result.orderBy(\"delta_norm\", ascending=False).show(10, truncate=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-08",
+   "metadata": {},
+   "source": [
+    "`delta_norm` is the distance from the segment's centre in standard deviations, `is_anomaly` marks the accounts beyond the segment's 99th percentile, `p_value` is the share of the population at least as far out, and `top_reasons` names the dimensions that move the account most.\n",
+    "\n",
+    "## 3. Compare with the labels\n",
+    "\n",
+    "The threshold flags about 1 % of every population by construction, so the flagged set always has a size; the labels show what it contains."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-09",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pyspark.sql import functions as F\n",
+    "\n",
+    "labels = accounts.select(F.col(\"account\").alias(\"entity\"), \"is_laundering_account\", \"typology\")\n",
+    "flagged = result.where(\"is_anomaly\").join(labels, \"entity\", \"left\")\n",
+    "flagged.groupBy(\"segment\", \"is_laundering_account\").count().orderBy(\"segment\").show()\n",
+    "flagged.where(\"is_laundering_account\").groupBy(\"typology\").count().show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-10",
+   "metadata": {},
+   "source": [
+    "## 4. Why an account stands out\n",
+    "\n",
+    "`explain` gives one row per account and dimension: the value, the segment's mean and standard deviation, the signed deviation and its share of the distance."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-11",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from hypertopos.spark import explain\n",
+    "\n",
+    "explain(result, top_n=3).show(15, truncate=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-12",
+   "metadata": {},
+   "source": [
+    "## 5. Score the next period against the same baseline\n",
+    "\n",
+    "`score` measures new events against the population `result` was calibrated on, without recalibrating, so scores stay comparable from one period to the next. Here a second synthetic period stands in for next month's payments."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-13",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from hypertopos.spark import score\n",
+    "\n",
+    "later = synthetic_aml(n_accounts=5_000, seed=2)\n",
+    "later_tx = spark.createDataFrame(later.transactions.to_pandas())\n",
+    "later_accounts = spark.createDataFrame(later.accounts.to_pandas())\n",
+    "later_events = later_tx.join(\n",
+    "    later_accounts.select(\"account\", \"segment\"), later_tx.from_account == later_accounts.account\n",
+    ")\n",
+    "scored = score(later_events, like=result)\n",
+    "scored.where(\"is_anomaly\").orderBy(\"delta_norm\", ascending=False).show(10, truncate=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-14",
+   "metadata": {},
+   "source": [
+    "## 6. Hand the result to an agent\n",
+    "\n",
+    "`sphere_of` returns the calibration behind the result: its geometry as a table, or a sphere for the [hypertopos MCP server](https://github.com/hypertopos/hypertopos-mcp) with the same distances and flags, so an agent can investigate the flagged accounts."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-15",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from hypertopos.spark import sphere_of\n",
+    "\n",
+    "sphere = sphere_of(result)\n",
+    "geometry = sphere.geometry()  # pyarrow table: values, deviations and scores per account\n",
+    "print(geometry.num_rows, \"accounts,\", len(geometry.column_names), \"columns\")\n",
+    "sphere.mcp_config()  # put under mcpServers in an MCP client's configuration"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}