sqllocks-shape-simulation 0.9.1__tar.gz

sqllocks_shape_simulation-0.9.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025-2026 SQLLocks (Jonathan Stewart)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sqllocks_shape_simulation-0.9.1/PKG-INFO

@@ -0,0 +1,61 @@
+Metadata-Version: 2.4
+Name: sqllocks-shape-simulation
+Version: 0.9.1
+Summary: Shape plugin: Simulation scenarios
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/sqllocks/shape
+Project-URL: Repository, https://github.com/sqllocks/shape
+Project-URL: Issues, https://github.com/sqllocks/shape/issues
+Project-URL: Documentation, https://github.com/sqllocks/shape/tree/main/plugins/shape-simulation
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Database
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: sqllocks-shape==0.9.1
+Dynamic: license-file
+
+# sqllocks-shape-simulation
+
+Shape plugin: simulation scenarios. Turns generated tables into what an upstream system would
+produce over time, and generates the telemetry of systems that have no table behind them.
+
+| Simulator | Module | `shape simulate` |
+|---|---|---|
+| `FileDropSimulator` | `shape_simulation.file_drop` | `file-drop` |
+| `SCD2FileDropSimulator` | `shape_simulation.scd2_file_drops` | `scd2` |
+| `StreamEmitter` | `shape_simulation.stream_emit` | `stream` |
+| `HybridSimulator` | `shape_simulation.hybrid` | `hybrid` |
+| `WorkflowSimulator` | `shape_simulation.state_machine` | `workflow` |
+| `ClickstreamSimulator` | `shape_simulation.clickstream_patterns` | `clickstream` |
+| `FinancialStreamSimulator` | `shape_simulation.financial_patterns` | `financial` |
+| `IoTTelemetrySimulator` | `shape_simulation.iot_patterns` | `iot` |
+| `OperationalLogSimulator` | `shape_simulation.operational_log_patterns` | `operational-log` |
+| `PulseDemandSimulator` | `shape_simulation.pulse_patterns` | `pulse` |
+
+Each takes a configuration and, where it works on existing data, Arrow tables (or a generation
+result), and is deterministic for a seed. The first five are described in
+`docs/SIMULATION_FILES_EVENTS.md`, the pattern simulators (the last five) in
+`docs/plugins/simulation.md`.
+
+```bash
+pip install 'sqllocks-shape[simulation]'
+shape simulate clickstream --set users=500 -o out/
+```
+
+```python
+from shape_simulation.clickstream_patterns import ClickstreamConfig, ClickstreamSimulator
+
+result = ClickstreamSimulator(ClickstreamConfig(users=500, seed=7)).run()
+result.sessions, result.page_views, result.funnels, result.stats
+```
+
+It registers one plugin command, `simulate` (`shape.commands`); the simulators are importable from
+`shape_simulation` without it. Its version always equals core's (`sqllocks-shape`), and it is
+released together with core. How plugins are written: `docs/plugins/authoring.md` in the repository.

sqllocks_shape_simulation-0.9.1/README.md

@@ -0,0 +1,38 @@
+# sqllocks-shape-simulation
+
+Shape plugin: simulation scenarios. Turns generated tables into what an upstream system would
+produce over time, and generates the telemetry of systems that have no table behind them.
+
+| Simulator | Module | `shape simulate` |
+|---|---|---|
+| `FileDropSimulator` | `shape_simulation.file_drop` | `file-drop` |
+| `SCD2FileDropSimulator` | `shape_simulation.scd2_file_drops` | `scd2` |
+| `StreamEmitter` | `shape_simulation.stream_emit` | `stream` |
+| `HybridSimulator` | `shape_simulation.hybrid` | `hybrid` |
+| `WorkflowSimulator` | `shape_simulation.state_machine` | `workflow` |
+| `ClickstreamSimulator` | `shape_simulation.clickstream_patterns` | `clickstream` |
+| `FinancialStreamSimulator` | `shape_simulation.financial_patterns` | `financial` |
+| `IoTTelemetrySimulator` | `shape_simulation.iot_patterns` | `iot` |
+| `OperationalLogSimulator` | `shape_simulation.operational_log_patterns` | `operational-log` |
+| `PulseDemandSimulator` | `shape_simulation.pulse_patterns` | `pulse` |
+
+Each takes a configuration and, where it works on existing data, Arrow tables (or a generation
+result), and is deterministic for a seed. The first five are described in
+`docs/SIMULATION_FILES_EVENTS.md`, the pattern simulators (the last five) in
+`docs/plugins/simulation.md`.
+
+```bash
+pip install 'sqllocks-shape[simulation]'
+shape simulate clickstream --set users=500 -o out/
+```
+
+```python
+from shape_simulation.clickstream_patterns import ClickstreamConfig, ClickstreamSimulator
+
+result = ClickstreamSimulator(ClickstreamConfig(users=500, seed=7)).run()
+result.sessions, result.page_views, result.funnels, result.stats
+```
+
+It registers one plugin command, `simulate` (`shape.commands`); the simulators are importable from
+`shape_simulation` without it. Its version always equals core's (`sqllocks-shape`), and it is
+released together with core. How plugins are written: `docs/plugins/authoring.md` in the repository.

sqllocks_shape_simulation-0.9.1/pyproject.toml

@@ -0,0 +1,38 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sqllocks-shape-simulation"
+# Lockstep with core (T-09): scripts/check_plugin_skeletons.py fails when this drifts.
+version = "0.9.1"
+description = "Shape plugin: Simulation scenarios"
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+license-files = ["LICENSE"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Operating System :: OS Independent",
+    "Topic :: Database",
+]
+dependencies = ["sqllocks-shape==0.9.1"]
+
+# Entry points (shape.* groups, plugin API v1); see docs/plugins/authoring.md.
+
+[project.urls]
+Homepage = "https://github.com/sqllocks/shape"
+Repository = "https://github.com/sqllocks/shape"
+Issues = "https://github.com/sqllocks/shape/issues"
+Documentation = "https://github.com/sqllocks/shape/tree/main/plugins/shape-simulation"
+
+[project.entry-points."shape.commands"]
+simulate = "shape_simulation.cli:SimulateCommand"
+
+[tool.setuptools.packages.find]
+where = ["src"]

sqllocks_shape_simulation-0.9.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

sqllocks_shape_simulation-0.9.1/src/shape_simulation/__init__.py

@@ -0,0 +1,157 @@
+"""Shape plugin: simulation scenarios (P6-04).
+
+Each simulator lives in its own module and is also importable from here, lazily (importing the
+package loads no simulator, and none of Arrow or NumPy):
+
+* ``file_drop``: :class:`FileDropSimulator`, an upstream source landing files over a date range;
+* ``scd2_file_drops``: :class:`SCD2FileDropSimulator`, a full load and daily versioned deltas;
+* ``stream_emit``: :class:`StreamEmitter`, tables as enveloped events on the emit runtime;
+* ``hybrid``: :class:`HybridSimulator`, a file drop and a stream of the same tables, linked;
+* ``state_machine``: :class:`WorkflowSimulator`, business-process events with dwell times;
+* ``clickstream_patterns``, ``financial_patterns``, ``iot_patterns``,
+  ``operational_log_patterns``, ``pulse_patterns``: the pattern simulators (sessions and funnels,
+  reversals and fraud bursts, sensor drift and alert storms, service logs and traces, rideshare
+  telemetry and marts), which take a configuration and, where they layer on existing data, Arrow
+  tables, and return Arrow tables and summary statistics;
+* ``cli``, ``cli_batch``, ``cli_patterns``: ``shape simulate``, the command that runs them.
+"""
+
+from __future__ import annotations
+
+import importlib
+from typing import TYPE_CHECKING, Any
+
+SHAPE_API = "1.0"
+"""The plugin targets the plugin API major version this Shape provides."""
+
+# name -> module that defines it (one line per name; each simulator adds its own lines)
+_EXPORTS: dict[str, str] = {
+    "ClickstreamConfig": "shape_simulation.clickstream_patterns",
+    "ClickstreamResult": "shape_simulation.clickstream_patterns",
+    "ClickstreamSimulator": "shape_simulation.clickstream_patterns",
+    "FileDropConfig": "shape_simulation.file_drop",
+    "FileDropResult": "shape_simulation.file_drop",
+    "FileDropSimulator": "shape_simulation.file_drop",
+    "FinancialStreamConfig": "shape_simulation.financial_patterns",
+    "FinancialStreamResult": "shape_simulation.financial_patterns",
+    "FinancialStreamSimulator": "shape_simulation.financial_patterns",
+    "HybridConfig": "shape_simulation.hybrid",
+    "HybridResult": "shape_simulation.hybrid",
+    "HybridSimulator": "shape_simulation.hybrid",
+    "IoTTelemetryConfig": "shape_simulation.iot_patterns",
+    "IoTTelemetryResult": "shape_simulation.iot_patterns",
+    "IoTTelemetrySimulator": "shape_simulation.iot_patterns",
+    "OperationalLogConfig": "shape_simulation.operational_log_patterns",
+    "OperationalLogResult": "shape_simulation.operational_log_patterns",
+    "OperationalLogSimulator": "shape_simulation.operational_log_patterns",
+    "PulseDemandConfig": "shape_simulation.pulse_patterns",
+    "PulseDemandSimulator": "shape_simulation.pulse_patterns",
+    "PulseSimResult": "shape_simulation.pulse_patterns",
+    "SCD2FileDropConfig": "shape_simulation.scd2_file_drops",
+    "SCD2FileDropResult": "shape_simulation.scd2_file_drops",
+    "SCD2FileDropSimulator": "shape_simulation.scd2_file_drops",
+    "SimulateCommand": "shape_simulation.cli",
+    "StateDefinition": "shape_simulation.state_machine",
+    "StreamEmitConfig": "shape_simulation.stream_emit",
+    "StreamEmitResult": "shape_simulation.stream_emit",
+    "StreamEmitter": "shape_simulation.stream_emit",
+    "TransitionRule": "shape_simulation.state_machine",
+    "WorkflowConfig": "shape_simulation.state_machine",
+    "WorkflowResult": "shape_simulation.state_machine",
+    "WorkflowSimulator": "shape_simulation.state_machine",
+    "get_preset_workflow": "shape_simulation.state_machine",
+}
+
+__all__ = [
+    "ClickstreamConfig",
+    "ClickstreamResult",
+    "ClickstreamSimulator",
+    "FileDropConfig",
+    "FileDropResult",
+    "FileDropSimulator",
+    "FinancialStreamConfig",
+    "FinancialStreamResult",
+    "FinancialStreamSimulator",
+    "HybridConfig",
+    "HybridResult",
+    "HybridSimulator",
+    "IoTTelemetryConfig",
+    "IoTTelemetryResult",
+    "IoTTelemetrySimulator",
+    "OperationalLogConfig",
+    "OperationalLogResult",
+    "OperationalLogSimulator",
+    "PulseDemandConfig",
+    "PulseDemandSimulator",
+    "PulseSimResult",
+    "SCD2FileDropConfig",
+    "SCD2FileDropResult",
+    "SCD2FileDropSimulator",
+    "SimulateCommand",
+    "StateDefinition",
+    "StreamEmitConfig",
+    "StreamEmitResult",
+    "StreamEmitter",
+    "TransitionRule",
+    "WorkflowConfig",
+    "WorkflowResult",
+    "WorkflowSimulator",
+    "get_preset_workflow",
+]
+
+if TYPE_CHECKING:
+    from shape_simulation.cli import SimulateCommand
+    from shape_simulation.clickstream_patterns import (
+        ClickstreamConfig,
+        ClickstreamResult,
+        ClickstreamSimulator,
+    )
+    from shape_simulation.file_drop import FileDropConfig, FileDropResult, FileDropSimulator
+    from shape_simulation.financial_patterns import (
+        FinancialStreamConfig,
+        FinancialStreamResult,
+        FinancialStreamSimulator,
+    )
+    from shape_simulation.hybrid import HybridConfig, HybridResult, HybridSimulator
+    from shape_simulation.iot_patterns import (
+        IoTTelemetryConfig,
+        IoTTelemetryResult,
+        IoTTelemetrySimulator,
+    )
+    from shape_simulation.operational_log_patterns import (
+        OperationalLogConfig,
+        OperationalLogResult,
+        OperationalLogSimulator,
+    )
+    from shape_simulation.pulse_patterns import (
+        PulseDemandConfig,
+        PulseDemandSimulator,
+        PulseSimResult,
+    )
+    from shape_simulation.scd2_file_drops import (
+        SCD2FileDropConfig,
+        SCD2FileDropResult,
+        SCD2FileDropSimulator,
+    )
+    from shape_simulation.state_machine import (
+        StateDefinition,
+        TransitionRule,
+        WorkflowConfig,
+        WorkflowResult,
+        WorkflowSimulator,
+        get_preset_workflow,
+    )
+    from shape_simulation.stream_emit import StreamEmitConfig, StreamEmitResult, StreamEmitter
+
+
+def __getattr__(name: str) -> Any:
+    module = _EXPORTS.get(name)
+    if module is None:
+        raise AttributeError(f"module 'shape_simulation' has no attribute {name!r}")
+    value = getattr(importlib.import_module(module), name)
+    globals()[name] = value
+    return value
+
+
+def __dir__() -> list[str]:
+    return sorted([*globals(), *_EXPORTS])

sqllocks_shape_simulation-0.9.1/src/shape_simulation/_patterns.py

@@ -0,0 +1,321 @@
+"""Shared helpers of the pattern simulators (clickstream, financial, IoT, operational log, pulse).
+
+The simulators work on Arrow tables (inputs and outputs) with numpy, so the plugin needs nothing
+beyond Shape's own dependencies. Everything random comes from one ``numpy`` generator seeded by
+the configuration, ids included: the same configuration and seed give the same tables.
+"""
+
+from __future__ import annotations
+
+import json
+from collections.abc import Mapping
+from datetime import UTC, datetime
+from pathlib import Path
+from typing import Any, ClassVar
+
+import numpy as np
+import pyarrow as pa  # type: ignore[import-untyped]
+import pyarrow.compute as pc  # type: ignore[import-untyped]
+
+FORMATS = ("parquet", "csv", "jsonl")
+
+
+# ---- ids ------------------------------------------------------------------------------------
+
+
+def uuid_strings(rng: np.random.Generator, n: int) -> list[str]:
+    """``n`` version-4 style UUID strings drawn from ``rng`` (so a seed reproduces them)."""
+    if n <= 0:
+        return []
+    raw = rng.integers(0, 256, size=(n, 16), dtype=np.uint8)
+    raw[:, 6] = (raw[:, 6] & 0x0F) | 0x40
+    raw[:, 8] = (raw[:, 8] & 0x3F) | 0x80
+    hexa = raw.tobytes().hex()
+    return [
+        f"{h[0:8]}-{h[8:12]}-{h[12:16]}-{h[16:20]}-{h[20:32]}"
+        for h in (hexa[i * 32 : (i + 1) * 32] for i in range(n))
+    ]
+
+
+def hex_strings(rng: np.random.Generator, n: int, length: int) -> list[str]:
+    """``n`` random lowercase hex strings of ``length`` characters."""
+    if n <= 0:
+        return []
+    nbytes = (length + 1) // 2
+    raw = rng.integers(0, 256, size=(n, nbytes), dtype=np.uint8).tobytes().hex()
+    width = nbytes * 2
+    return [raw[i * width : i * width + length] for i in range(n)]
+
+
+# ---- weighted draws -------------------------------------------------------------------------
+
+
+def pick(
+    rng: np.random.Generator,
+    values: list[Any] | np.ndarray,
+    size: int,
+    p: list[float] | np.ndarray | None = None,
+) -> np.ndarray:
+    """``size`` draws from ``values`` (uniform, or with probabilities ``p``, normalised)."""
+    if p is not None:
+        weights = np.asarray(p, dtype=np.float64)
+        weights = weights / weights.sum()
+        return np.asarray(values)[rng.choice(len(values), size=size, p=weights)]
+    return np.asarray(values)[rng.integers(0, len(values), size=size)]
+
+
+# ---- Arrow in -------------------------------------------------------------------------------
+
+
+def as_table(obj: Any) -> pa.Table:
+    """An Arrow table from a table, a record batch, a mapping of columns or a pandas frame."""
+    if isinstance(obj, pa.Table):
+        return obj
+    if isinstance(obj, pa.RecordBatch):
+        return pa.Table.from_batches([obj])
+    if isinstance(obj, Mapping):
+        return pa.table(dict(obj))
+    to_arrow = getattr(obj, "to_arrow", None)
+    if callable(to_arrow):
+        out = to_arrow()
+        if isinstance(out, pa.Table):
+            return out
+    if hasattr(obj, "__arrow_c_stream__"):
+        return pa.table(obj)
+    if hasattr(obj, "columns") and hasattr(obj, "dtypes"):  # a pandas frame
+        return pa.Table.from_pandas(obj, preserve_index=False)
+    raise TypeError(f"cannot read {type(obj).__name__} as an Arrow table")
+
+
+def table_mapping(tables: Any) -> dict[str, pa.Table]:
+    """``tables`` (a mapping of tables, or a generation result with ``.tables``) as a dict."""
+    inner = getattr(tables, "tables", tables)
+    if not isinstance(inner, Mapping):
+        raise TypeError("tables must map table names to tables")
+    return {str(k): as_table(v) for k, v in inner.items()}
+
+
+def _is_number(value: Any) -> bool:
+    return not isinstance(value, bool) and isinstance(value, int | float | np.integer | np.floating)
+
+
+def check_settings(
+    config: Any,
+    *,
+    positive: tuple[str, ...] = (),
+    non_negative: tuple[str, ...] = (),
+    probabilities: tuple[str, ...] = (),
+    non_empty: tuple[str, ...] = (),
+) -> None:
+    """Refuse settings a simulator cannot use, with a ``ValueError`` that names the setting.
+
+    Every field whose default is a number must be a number (``--set users=abc`` gives text);
+    ``positive`` ones are above 0 (they divide), ``non_negative`` ones 0 or more,
+    ``probabilities`` in [0, 1] and ``non_empty`` lists hold at least one item. ``None`` is
+    accepted where it is the default (an unset optional)."""
+    import dataclasses
+
+    for f in dataclasses.fields(config):
+        default = f.default
+        if isinstance(default, bool) or not isinstance(default, int | float):
+            continue
+        value = getattr(config, f.name)
+        if not _is_number(value):
+            raise ValueError(f"{f.name} must be a number, got {value!r}")
+    for names, strict, high, what in (
+        (positive, True, None, "above 0"),
+        (non_negative, False, None, "0 or more"),
+        (probabilities, False, 1.0, "between 0 and 1"),
+    ):
+        for name in names:
+            value = getattr(config, name)
+            if value is None:
+                continue
+            if not _is_number(value):
+                raise ValueError(f"{name} must be a number, got {value!r}")
+            number = float(value)
+            # written so that NaN fails every rule
+            inside = number > 0 if strict else number >= 0
+            if high is not None:
+                inside = inside and number <= high
+            if not inside:
+                raise ValueError(f"{name} must be {what}, got {value!r}")
+    for name in non_empty:
+        if not getattr(config, name):
+            raise ValueError(f"{name} must hold at least one item")
+
+
+def timestamp_us(col: pa.ChunkedArray | pa.Array) -> tuple[np.ndarray, np.ndarray, str | None]:
+    """A date, timestamp or ISO-8601 string column as ``(microseconds, valid, timezone)``.
+
+    ``microseconds`` is int64 since the epoch (0 where null), ``valid`` marks the non-null
+    cells and ``timezone`` is the column's time zone, if it had one.
+    """
+    if isinstance(col, pa.ChunkedArray):
+        col = col.combine_chunks() if col.num_chunks else pa.array([], type=col.type)
+    tz: str | None = None
+    if pa.types.is_timestamp(col.type):
+        tz = col.type.tz
+        col = col.cast(pa.timestamp("us", tz))
+    elif pa.types.is_date(col.type):
+        col = col.cast(pa.timestamp("us"))
+    elif pa.types.is_string(col.type) or pa.types.is_large_string(col.type):
+        col = _text_times(col)
+    else:
+        raise TypeError(f"{col.type} is not a date or timestamp type")
+    valid = ~np.asarray(col.is_null().to_numpy(zero_copy_only=False), dtype=bool)
+    ints = col.cast(pa.int64()).fill_null(0).to_numpy(zero_copy_only=False)
+    return ints.astype(np.int64), valid, tz
+
+
+def _text_times(col: pa.Array) -> pa.Array:
+    """ISO-8601 text as naive ``timestamp[us]``: a time with ``Z`` or an offset becomes its UTC
+    wall time, one without is taken as it is. Text that is not a time raises ``ValueError``."""
+    try:
+        return pc.cast(col, pa.timestamp("us"))
+    except pa.ArrowInvalid:
+        pass
+    try:  # every time carries a zone
+        return pc.cast(col, pa.timestamp("us", "UTC")).cast(pa.timestamp("us"))
+    except pa.ArrowInvalid:
+        pass
+    values: list[datetime | None] = []
+    for text in col.to_pylist():  # some with a zone, some without
+        if text is None:
+            values.append(None)
+            continue
+        try:
+            value = datetime.fromisoformat(text)
+        except ValueError:
+            raise ValueError(f"{text!r} is not an ISO-8601 date or time") from None
+        if value.tzinfo is not None:
+            value = value.astimezone(UTC).replace(tzinfo=None)
+        values.append(value)
+    return pa.array(values, pa.timestamp("us"))
+
+
+def timestamps(us: np.ndarray, tz: str | None = None, valid: np.ndarray | None = None) -> pa.Array:
+    """A ``timestamp[us]`` array (``tz`` if given) from microseconds, null where not ``valid``."""
+    mask = None if valid is None else ~np.asarray(valid, dtype=bool)
+    return pa.array(
+        np.asarray(us, dtype=np.int64).astype("datetime64[us]"),
+        type=pa.timestamp("us", tz),
+        mask=mask,
+    )
+
+
+def parse_start(text: str) -> int:
+    """An ISO-8601 instant as microseconds since the epoch (a missing zone means UTC)."""
+    moment = datetime.fromisoformat(text)
+    if moment.tzinfo is None:
+        moment = moment.replace(tzinfo=UTC)
+    return int(moment.timestamp() * 1_000_000)
+
+
+def float_values(col: pa.ChunkedArray | pa.Array) -> tuple[np.ndarray, np.ndarray]:
+    """A numeric column as ``(float64 values, valid)``; null cells are NaN."""
+    arr = pc.cast(col, pa.float64())
+    if isinstance(arr, pa.ChunkedArray):
+        arr = arr.combine_chunks() if arr.num_chunks else pa.array([], type=pa.float64())
+    valid = ~np.asarray(arr.is_null().to_numpy(zero_copy_only=False), dtype=bool)
+    values = arr.fill_null(float("nan")).to_numpy(zero_copy_only=False).astype(np.float64)
+    return values, valid
+
+
+def float_array(values: np.ndarray) -> pa.Array:
+    """A float64 Arrow array from ``values``, null where the value is NaN."""
+    values = np.asarray(values, dtype=np.float64)
+    return pa.array(values, type=pa.float64(), mask=np.isnan(values))
+
+
+def empty_table(schema: pa.Schema) -> pa.Table:
+    return schema.empty_table()
+
+
+def combine(tables: list[pa.Table]) -> pa.Table:
+    """The rows of ``tables`` stacked, with the union of their columns (missing cells are null).
+
+    Where the same column has different types (decimal and float amounts, say) numeric
+    columns become float64 and anything else text.
+    """
+    names: list[str] = []
+    types: dict[str, pa.DataType] = {}
+    for table in tables:
+        for field in table.schema:
+            if field.name not in types:
+                names.append(field.name)
+                types[field.name] = field.type
+            elif types[field.name] != field.type:
+                a, b = types[field.name], field.type
+                numeric = all(
+                    pa.types.is_integer(t) or pa.types.is_floating(t) or pa.types.is_decimal(t)
+                    for t in (a, b)
+                )
+                types[field.name] = pa.float64() if numeric else pa.string()
+    schema = pa.schema([pa.field(n, types[n]) for n in names])
+    parts = []
+    for table in tables:
+        cols = []
+        for name in names:
+            if name in table.column_names:
+                cols.append(pc.cast(table.column(name), types[name]))
+            else:
+                cols.append(pa.nulls(table.num_rows, type=types[name]))
+        parts.append(pa.Table.from_arrays(cols, schema=schema))
+    return pa.concat_tables(parts)
+
+
+def value_counts(values: np.ndarray) -> dict[str, int]:
+    """``{value: count}`` ordered by count (largest first; ties by first appearance)."""
+    if len(values) == 0:
+        return {}
+    uniq, first, counts = np.unique(values, return_index=True, return_counts=True)
+    order = np.lexsort((first, -counts))
+    return {str(uniq[i]): int(counts[i]) for i in order}
+
+
+# ---- results --------------------------------------------------------------------------------
+
+
+class TablesResult:
+    """What every simulator's result offers: its tables, as flat events, and on disk."""
+
+    TABLES: ClassVar[tuple[str, ...]] = ()
+    stats: dict[str, Any]
+
+    def table_map(self) -> dict[str, pa.Table]:
+        """Every output table by name."""
+        return {name: getattr(self, name) for name in self.TABLES}
+
+    def events(self, table: str, *, seq_start: int = 0) -> pa.RecordBatch:
+        """The rows of ``table`` as flat stream events: its columns plus ``_shape_table``,
+        ``_shape_seq`` and (when it has a date or timestamp column) ``_shape_event_time``.
+        Feed them to ``shape.streaming.emit`` sinks or an emitter."""
+        from shape.streaming.emit.formats import with_event_fields
+
+        source = self.table_map().get(table)
+        if source is None:
+            raise KeyError(f"no table {table!r}; the result has {', '.join(self.table_map())}")
+        batch = source.combine_chunks().to_batches()
+        merged = batch[0] if batch else pa.RecordBatch.from_pylist([], schema=source.schema)
+        events: pa.RecordBatch = with_event_fields(merged, table, seq_start)
+        return events
+
+    def write(self, directory: str | Path, fmt: str = "parquet") -> dict[str, Path]:
+        """Write every table to ``directory`` as ``<table>.<fmt>`` (parquet, csv or jsonl)
+        and return the paths."""
+        if fmt not in FORMATS:
+            raise ValueError(f"unknown format {fmt!r}; choose from {', '.join(FORMATS)}")
+        from shape.builtins.sinks.files import CsvSink, JsonlSink, ParquetSink
+
+        sink: Any = {"parquet": ParquetSink, "csv": CsvSink, "jsonl": JsonlSink}[fmt]()
+        out = Path(directory)
+        out.mkdir(parents=True, exist_ok=True)
+        written: dict[str, Path] = {}
+        for name, table in self.table_map().items():
+            sink.write(str(out) + "/", name, table.to_batches(), schema=table.schema)
+            written[name] = out / f"{name}.{fmt}"
+        (out / "stats.json").write_text(
+            json.dumps(self.stats, indent=2, default=str) + "\n", encoding="utf-8", newline="\n"
+        )
+        return written