pirn-data 0.4.0__py3-none-any.whl

pirn_data/AGENTIC_USE.md

@@ -0,0 +1,315 @@
+# AGENTIC_USE — pirn_data
+
+Provides a tiered, engine-agnostic layer for reading, transforming, and writing structured data — it does NOT include API connectors, message-queue consumers, or ML training loops (those are separate domains).
+
+---
+
+## Mental model
+
+pirn is the **orchestrator**; the engines do the work. The data domain is stratified into tiers, each independently opt-in:
+
+| Tier | Label | Engines | When to use |
+|------|-------|---------|-------------|
+| **1** | Dict / DataBatch | Pure Python | Always available. Use for < ~100 k rows, glue logic, or environments where no heavy deps are allowed. Every record is a `dict`; the exchange currency is `DataBatch`. |
+| **2** | Native frames (CPU) | **Polars** (preferred), DuckDB, pandas+PyArrow, DataFusion | Fits comfortably in RAM. Need fast vectorised ops, joins, window functions, pivots. Polars is the default; reach for DuckDB when SQL is more natural or when you need concurrent read-only queries. |
+| **2-GPU** | Native frames (GPU) | cuDF | CUDA cluster only. Drop-in Polars-compatible when `cudf-cu12` is installed. |
+| **2.5** | Out-of-core | Modin | Data exceeds RAM, single machine. Pandas-compatible API chunked on disk. |
+| **3** | Lazy / push-down | **Ibis** (preferred), Spark, Dask, Ray Data | Data doesn't fit on one machine, or lives in a warehouse already. Ibis first — it targets many backends without a Spark cluster. Spark/Dask/Ray when distributed compute is required. |
+| **3-stream** | Streaming dataflow | Pathway, Bytewax | Continuous / low-latency event streams. Requires Python < 3.14 until upstream catches up. |
+| **4** | Specialised | Lance (vector), Eland (Elasticsearch) | Domain-specific columnar layouts; not general-purpose. |
+
+**Tier-1 (`DataBatch`) is always installed with `pirn[data]`.** Tiers 2–4 are independent extras that do not pull each other in.
+
+---
+
+## Install
+
+```bash
+pip install pirn[data]          # Tier 1 only — DataBatch, sources, sinks, transforms (no heavy deps)
+pip install pirn[all-frames]    # Tier 2 single-machine engines (Polars, DuckDB, pandas, PyArrow, DataFusion)
+pip install pirn[all-lazy]      # Tier 3 push-down engines (Ibis, Spark, Dask, Ray Data)
+```
+
+Specialised extras (one at a time as needed):
+
+```bash
+pip install pirn[polars]        # Polars only
+pip install pirn[delta]         # Delta Lake lakehouse adapter
+pip install pirn[iceberg]       # Apache Iceberg adapter
+pip install pirn[health]        # DICOM, HL7, FHIR, NIfTI, EDF, BIDS, etc.
+pip install pirn[genomics]      # FASTA, FASTQ, VCF, BAM, CRAM, SAM
+pip install pirn[oilgas]        # SEG-Y, DLIS, LAS, WITSML
+```
+
+---
+
+## Source map
+
+```
+pirn_data/
+├── data_batch.py               # Tier-1 exchange type: immutable tuple of dicts
+├── data_schema.py              # Optional schema metadata attached to DataBatch
+├── data_profile.py             # Statistical profile of a DataBatch
+├── quality/                    # Quality checks and reports
+│   ├── quality_check.py
+│   └── quality_report.py
+├── sources/
+│   ├── file_source.py          # Single-file source (ObjectStore + FileFormat)
+│   └── directory_source.py     # Prefix glob → one or many DataBatches
+├── sinks/
+│   └── file_sink.py            # Encode + write DataBatch to ObjectStore
+├── transforms/                 # Tier-1 transforms (pure Python, no extras)
+│   ├── filter.py               # Row predicate (Python callable)
+│   ├── rename.py               # Column rename map
+│   ├── cast.py                 # Type coercion
+│   ├── normalize.py            # String cleanup rules
+│   ├── aggregate.py            # Group-by + aggregations
+│   └── deduplicate.py          # Row deduplication
+├── frames/                     # Tier-2 engine-specific knots
+│   ├── polars/                 # PolarsFilter, PolarsJoin, PolarsAggregate, etc.
+│   │   └── bridges/            # DataBatch ↔ PolarsDataBatch conversions
+│   ├── duckdb/                 # DuckDB equivalents
+│   ├── pandas/
+│   ├── pyarrow/
+│   └── datafusion/
+├── lazy/                       # Tier-3 lazy/push-down knots
+│   ├── ibis/
+│   ├── spark/
+│   ├── dask/
+│   └── ray/
+├── lakehouse/                  # Lakehouse table adapters
+│   ├── delta/                  # DeltaTable (full CRUD + merge + time-travel)
+│   ├── iceberg/                # IcebergTable (read/write/time-travel; merge not yet available in pyiceberg)
+│   └── hudi/                   # HudiTable (read-only; writes require Spark writer)
+├── specializations/            # Pre-wired high-level patterns  ← specializations
+│   ├── ingestion/              # AppendOnlyIngest, FullRefreshExtract, WatermarkIncrementalExtract  ← specializations
+│   ├── medallion/              # BronzeRawIngest, SilverCleanTransform, GoldAggregation  ← specializations
+│   ├── scd/                   # ScdType1/2/3/4/5/6/7, CdcDebezium, DebeziumSource  ← specializations
+│   ├── dimensional/            # DateDimGenerator, DimTableLoad, FactTableLoad, BridgeTableBuilder  ← specializations
+│   ├── data_vault/             # DataVaultHubLoader, DataVaultLinkLoader, DataVaultSatelliteLoader, DataVaultPITTableBuilder, DataVaultBridgeTableBuilder  ← specializations
+│   ├── incremental/            # SnapshotTableAppender, DbtStyleSnapshot, MergeUpsert, DeleteSafeSync, PartitionedOverwrite  ← specializations
+│   ├── quality/                # RowCountAnomalyDetector, NullRateMonitor, SchemaEvolutionDetector, FreshnessCheck, ReferentialIntegrityCheck, ReconciliationDiff, StatisticalProfiler  ← specializations
+│   ├── deduplication/          # ExactDeduplicator, WindowedDeduplicator, FuzzyDeduplicator, ProbabilisticLinker  ← specializations
+│   ├── timeseries/             # TimeSeriesResampler, RollingWindowAggregator, SessionizationKnot, FunnelAnalysisKnot, CohortAggregator, LateArrivingEventHandler  ← specializations
+│   ├── feature_engineering/    # DerivedColumnCalculator, ColumnHasher, BinningKnot, StringNormalizer, DatePartExtractor, LookupEnricher, GeoEnricher, TextTokenCounter  ← specializations
+│   ├── analytics_engineering/  # StagingModelKnot, IntermediateModelKnot, MartModelKnot, RefreshMaterializedView, MetricLayerAggregator, ExposureLineageTag  ← specializations
+│   └── schema_migration/       # BackfillRunner, SchemaVersionMigrator, ColumnLineageTracker  ← specializations
+└── specialized/                # Tier-4 specialised adapters (Lance, Eland)
+```
+
+File formats and object stores live under `pirn/domains/connectors/`, not here.
+
+---
+
+## Tier selection guide
+
+| Data size | Latency need | Infrastructure | Recommended tier |
+|-----------|-------------|----------------|-----------------|
+| < 100 k rows | Any | Any | **Tier 1** — zero deps, fast enough |
+| Fits in RAM (up to ~10 GB) | Batch | Single machine | **Tier 2 Polars** |
+| Fits in RAM, SQL-heavy | Batch | Single machine | **Tier 2 DuckDB** |
+| Exceeds RAM, single machine | Batch | Single machine | **Tier 2.5 Modin** |
+| Warehouse-scale, existing backend | Batch | Any (warehouse/cluster) | **Tier 3 Ibis** |
+| Distributed compute required | Batch | Spark / Dask / Ray cluster | **Tier 3 Spark/Dask/Ray** |
+| Continuous / event-driven | Sub-second | Stream processor | **Tier 3-stream** |
+| Vector similarity search | Any | Any | **Tier 4 Lance** |
+
+---
+
+## Canonical patterns
+
+### Tier-1 dict batch pipeline
+
+```python
+from pirn.core.knot_config import KnotConfig
+from pirn.core.knot_factory import knot
+from pirn.core.parameter import Parameter
+from pirn.core.run_request import RunRequest
+from pirn.tapestry import Tapestry
+from pirn_data.sources.file_source import FileSource
+from pirn_data.transforms.filter import Filter
+from pirn_data.transforms.aggregate import Aggregate
+from pirn_data.transforms.aggregate_spec import AggregateSpec
+from pirn_data.sinks.file_sink import FileSink
+from pirn.connectors.file_formats.csv_format import CsvFormat
+from pirn.connectors.file_formats.parquet_format import ParquetFormat
+from pirn.connectors.object_stores.local_object_store import LocalObjectStore
+
+with Tapestry() as t:
+    store = LocalObjectStore(root="/data")
+    source = FileSource(
+        store=store,
+        format=CsvFormat(),
+        key="input/sales.csv",
+        _config=KnotConfig(id="source"),
+    )
+    active = Filter(
+        batch=source,
+        predicate=lambda row: row["status"] == "active",
+        _config=KnotConfig(id="active_only"),
+    )
+    summary = Aggregate(
+        batch=active,
+        by=["region"],
+        aggs={"revenue": AggregateSpec(op="sum", column="revenue")},
+        _config=KnotConfig(id="by_region"),
+    )
+    FileSink(
+        batch=summary,
+        store=store,
+        format=ParquetFormat(compression="zstd"),
+        key="output/region_revenue.parquet",
+        _config=KnotConfig(id="sink"),
+    )
+
+result = await t.run(RunRequest(parameters={}))
+```
+
+### Tier-2 Polars pipeline
+
+```python
+import polars as pl
+from pirn_data.frames.polars.polars_data_batch import PolarsDataBatch
+from pirn_data.frames.polars.polars_filter import PolarsFilter
+from pirn_data.frames.polars.polars_join import PolarsJoin
+from pirn_data.frames.polars.polars_aggregate import PolarsAggregate
+from pirn_data.frames.polars.bridges.data_batch_to_polars import DataBatchToPolars
+from pirn_data.frames.polars.bridges.polars_to_data_batch import PolarsToDataBatch
+
+# Promote a Tier-1 source to a Polars frame
+polars_batch = DataBatchToPolars(
+    batch=source,
+    _config=KnotConfig(id="to_polars"),
+)
+filtered = PolarsFilter(
+    batch=polars_batch,
+    expression=pl.col("region") == "EU",
+    _config=KnotConfig(id="eu_only"),
+)
+joined = PolarsJoin(
+    left=filtered,
+    right=ref_batch,
+    on="region",
+    how="left",
+    _config=KnotConfig(id="join_ref"),
+)
+# Demote back to Tier-1 DataBatch if downstream knots expect it
+result_batch = PolarsToDataBatch(
+    batch=joined,
+    _config=KnotConfig(id="to_data_batch"),
+)
+```
+
+---
+
+## Anti-patterns
+
+### Reaching for Tier 3 when data fits in memory
+
+Ibis/Spark add serialisation overhead and cluster setup cost. If the dataset fits in RAM, use Tier-2 Polars. Reserve Tier-3 for warehouse-resident data or genuinely distributed workloads.
+
+### Mixing tier knots without conversion bridges
+
+Passing a `PolarsDataBatch` directly into a Tier-1 `Filter` (or vice versa) will fail at runtime because the knots expect different input types. Always interpose the bridge knots:
+
+- `DataBatchToPolars` — `DataBatch` → `PolarsDataBatch`
+- `PolarsToDataBatch` — `PolarsDataBatch` → `DataBatch`
+
+Equivalent bridges exist in `frames/duckdb/bridges/`.
+
+### Using Tier-1 `Aggregate` for joins
+
+Tier-1 has no `Join` knot — a Python-level hash-join of two `tuple[dict]` lists would be O(n·m) without indexes. If you need joins, promote to Tier 2 first with a bridge knot, then use `PolarsJoin` or `DuckDbJoin`.
+
+### Calling `IcebergTable.merge()` in production
+
+As of mid-2026, `pyiceberg`'s Python writer does not implement merge. The method raises `NotImplementedError`. Use the Java/Scala Iceberg writer for production upserts.
+
+### Writing to a Hudi table from Python
+
+`HudiTable` is read-only. All write methods raise `NotImplementedError`. Writes require the `hudi-spark-bundle` Spark writer.
+
+### Instantiating format objects inside `process()`
+
+`FileFormat` objects are stateless but they import their vendor library at construction time. Build them once at pipeline-wiring time, not inside knot `process()` methods.
+
+---
+
+## Constraints and gotchas
+
+- **`DataBatch.rows` is a `tuple`** (immutable). Never mutate it. Produce a new batch with `batch.with_rows(new_rows)` — this preserves `schema` and `source_uri`.
+- **`DirectorySource` with `concatenate=True` loses per-file lineage.** The `source_uri` collapses to `{store}://{prefix}*`. Use `concatenate=False` when provenance matters.
+- **`Aggregate` skips `None` values.** Empty groups yield `None` for mean/min/max/first/last and `0` for count/count_distinct.
+- **Tier-3-stream requires Python < 3.14** until Pathway and Bytewax catch up to the new Python release.
+- **`pirn[hudi]` is a no-op marker extra** — there is no stable vendor SDK on PyPI. The read path relies only on `pyarrow` (included with `pirn[data]`).
+- **`CompressedFileFormat` codec availability varies.** `gzip` and `bzip2` use stdlib and are always available. `zstd`, `snappy`, and `lz4` each require their own extra.
+- **`ArchiveFileFormat` is always non-streaming** — the full archive must be buffered. Do not use it for large files where incremental streaming matters.
+- **`FileSink.process()` returns the destination `key` string**, not a `DataBatch`. Downstream knots that expect a `DataBatch` must not follow a `FileSink` directly.
+- **Lakehouse vendor SDKs load lazily.** Import errors for missing extras are raised on first use, not at module import time.
+
+---
+
+## Quick reference
+
+| Task | Tier | Knot / Class |
+|------|------|-------------|
+| Read a single file | 1 | `FileSource` |
+| Read all files under a prefix | 1 | `DirectorySource` |
+| Write a file | 1 | `FileSink` |
+| Filter rows (Python callable) | 1 | `Filter` |
+| Rename columns | 1 | `Rename` |
+| Type coercion | 1 | `Cast` |
+| String normalisation | 1 | `Normalize` |
+| Group-by aggregation | 1 | `Aggregate` |
+| Deduplication | 1 | `Deduplicate` |
+| Vectorised filter (Polars expr) | 2 | `PolarsFilter` |
+| Join two frames | 2 | `PolarsJoin` / `DuckDbJoin` |
+| Window functions | 2 | `PolarsWindowCalc` |
+| Pivot / unpivot | 2 | `PolarsPivot` / `PolarsUnpivot` |
+| DataBatch → Polars | bridge | `DataBatchToPolars` |
+| Polars → DataBatch | bridge | `PolarsToDataBatch` |
+| Delta Lake read/write/merge | lakehouse | `DeltaTable` + `LakehouseTableSource` |
+| Iceberg read/write | lakehouse | `IcebergTable` |
+| Hudi read (only) | lakehouse | `HudiTable` |
+| Bronze raw ingest | specialisation | `BronzeRawIngest` |
+| Silver clean transform | specialisation | `SilverCleanTransform` |
+| Gold aggregation | specialisation | `GoldAggregation` |
+| Incremental watermark ingest | specialisation | `WatermarkIncrementalExtract` |
+| SCD Type 2 history | specialisation | `ScdType2` |
+| SCD Type 3 previous value | specialisation | `ScdType3PreviousValue` |
+| SCD Type 4 mini-dimension | specialisation | `ScdType4MiniDimension` |
+| SCD Type 6 hybrid (1+2+3) | specialisation | `ScdType6Hybrid` |
+| Debezium CDC apply | specialisation | `CdcDebezium` |
+| Date dimension generate | specialisation | `DateDimGenerator` |
+| Dimension table load | specialisation | `DimTableLoad` |
+| Fact table load | specialisation | `FactTableLoad` |
+| Data Vault hub load | specialisation | `DataVaultHubLoader` |
+| Data Vault satellite load | specialisation | `DataVaultSatelliteLoader` |
+| Data Vault PIT table | specialisation | `DataVaultPITTableBuilder` |
+| Snapshot append | specialisation | `SnapshotTableAppender` |
+| dbt-style snapshot | specialisation | `DbtStyleSnapshot` |
+| Merge upsert | specialisation | `MergeUpsert` |
+| Row count anomaly | specialisation | `RowCountAnomalyDetector` |
+| Null rate monitor | specialisation | `NullRateMonitor` |
+| Referential integrity check | specialisation | `ReferentialIntegrityCheck` |
+| Reconciliation diff | specialisation | `ReconciliationDiff` |
+| Exact deduplication | specialisation | `ExactDeduplicator` |
+| Fuzzy deduplication | specialisation | `FuzzyDeduplicator` |
+| Time series resample | specialisation | `TimeSeriesResampler` |
+| Rolling window aggregation | specialisation | `RollingWindowAggregator` |
+| Sessionization | specialisation | `SessionizationKnot` |
+| Derived column calculation | specialisation | `DerivedColumnCalculator` |
+| Binning / bucketing | specialisation | `BinningKnot` |
+| Lookup enrichment | specialisation | `LookupEnricher` |
+| dbt staging model | specialisation | `StagingModelKnot` |
+| dbt mart model | specialisation | `MartModelKnot` |
+| Metric layer aggregation | specialisation | `MetricLayerAggregator` |
+| Schema version migration | specialisation | `SchemaVersionMigrator` |
+| Backfill runner | specialisation | `BackfillRunner` |
+| Column lineage tracking | specialisation | `ColumnLineageTracker` |
+| Compressed format | wrapper | `CompressedFileFormat(inner, codec=...)` |
+| Multi-file archive | wrapper | `ArchiveFileFormat(inner, archive_type=...)` |
+
+---
+
+*See also: [pirn AGENTIC_USE.md](../../AGENTIC_USE.md)*

pirn_data/__init__.py

@@ -0,0 +1,23 @@
+"""Data Engineering / Analytics Engineering knot library.
+
+Install with::
+
+    pip install 'pirn-data[data]'
+
+Note: ``data_schema``, ``data_batch``, ``quality_check``, and
+``quality_report`` are pure-Python contracts and remain importable in
+minimal environments. Modules that touch pandas / pyarrow (sources,
+transforms, sinks) import those dependencies lazily, so the
+missing-dependency error fires only when those modules are imported.
+"""
+
+import warnings
+
+from sweet_tea.registry import Registry
+from sweet_tea.sweet_tea_warning import SweetTeaWarning
+
+with warnings.catch_warnings():
+    warnings.simplefilter("ignore", SweetTeaWarning)
+    Registry.fill_registry(module=__name__, library="pirn")
+
+__all__: list[str] = []

pirn_data/data_batch.py

@@ -0,0 +1,99 @@
+"""A batch of rows flowing through the data pipeline."""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from dataclasses import dataclass, field
+from datetime import UTC, datetime
+from typing import Any
+
+from pirn.core.pirn_opaque_value import PirnOpaqueValue
+
+from pirn_data.data_schema import DataSchema
+
+
+@dataclass(frozen=True)
+class DataBatch(PirnOpaqueValue):
+    """Tabular batch of rows.
+
+    Each row is a ``dict[str, Any]`` keyed by column name. Heavy frames
+    (Pandas / Arrow / Polars) wrap into ``DataBatch`` via thin adapters in
+    the sources / transforms modules.
+
+    Attributes
+    ----------
+    rows:
+        Sequence of row dicts.
+    schema:
+        Schema the rows conform to.
+    source_uri:
+        Where the rows came from (DSN, file path, API endpoint). DSN-style
+        values must be passed through
+        :class:`pirn.connectors.dsn_scrubber.DsnScrubber` before
+        assignment to avoid leaking credentials into lineage records.
+    fetched_at:
+        UTC instant the data was materialised.
+    """
+
+    rows: tuple[Mapping[str, Any], ...] = ()
+    schema: DataSchema = field(default_factory=DataSchema)
+    source_uri: str = ""
+    fetched_at: datetime = field(default_factory=lambda: datetime.now(UTC))
+
+    @property
+    def row_count(self) -> int:
+        return len(self.rows)
+
+    def with_rows(self, rows: tuple[Mapping[str, Any], ...]) -> DataBatch:
+        """Copy with ``rows`` replaced; schema/uri/fetched_at preserved."""
+        return DataBatch(
+            rows=rows,
+            schema=self.schema,
+            source_uri=self.source_uri,
+            fetched_at=self.fetched_at,
+        )
+
+    def with_schema(self, schema: DataSchema) -> DataBatch:
+        """Copy with ``schema`` replaced; rows preserved."""
+        return DataBatch(
+            rows=self.rows,
+            schema=schema,
+            source_uri=self.source_uri,
+            fetched_at=self.fetched_at,
+        )
+
+    def _pirn_audit_dict(self) -> dict[str, Any]:
+        """Flatten to a primitive dict for pydantic serialisation.
+
+        Pirn IO validation only needs ``isinstance(value, DataBatch)``;
+        content-addressing flattens the rows into this stable summary.
+        The contained :class:`DataSchema` (with its
+        ``Mapping[str, type]`` columns) is intentionally omitted —
+        ``DataSchema`` is opaque-serialised separately when needed.
+        """
+        return {
+            "row_count": self.row_count,
+            "source_uri": self.source_uri,
+            "fetched_at": self.fetched_at.isoformat(),
+            "rows": [dict(r) for r in self.rows],
+        }
+
+    def __pirn_canonical__(self) -> dict[str, Any]:
+        """Sanctioned canonical form for :func:`pirn.core.hashing.content_hash`.
+
+        Returned dict is fully JSON-serialisable: ``schema.columns`` is
+        flattened to ``{name: type-name}`` so the otherwise-opaque
+        :class:`type` values do not blow up the hasher. The audit dict
+        deliberately omits the schema (pydantic IO already validates the
+        boundary); content-addressing keeps it so two structurally
+        identical batches with the same column types hash equally.
+        """
+        return {
+            "row_count": self.row_count,
+            "source_uri": self.source_uri,
+            "fetched_at": self.fetched_at.isoformat(),
+            "rows": [dict(r) for r in self.rows],
+            "schema_columns": {
+                name: column_type.__name__ for name, column_type in self.schema.columns.items()
+            },
+        }

pirn_data/data_profile.py

@@ -0,0 +1,46 @@
+"""``DataProfile`` and ``ColumnProfile`` — descriptive statistics for a
+:class:`DataBatch`.
+
+Emitted by :class:`pirn_data.quality.profiler.Profiler`. A profile
+is observation, not policy: every field describes the input batch, no
+field carries a pass/fail verdict. Compose with a downstream knot if you
+want thresholds enforced (or use :class:`NullRateCheck` /
+:class:`RowCountCheck` directly).
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from datetime import UTC, datetime
+from typing import Any
+
+
+@dataclass(frozen=True)
+class ColumnProfile:
+    """Per-column statistics."""
+
+    name: str
+    observed_count: int
+    null_count: int
+    distinct_count: int
+    min_value: Any | None = None
+    max_value: Any | None = None
+    top_value: Any | None = None
+    top_value_count: int = 0
+
+
+@dataclass(frozen=True)
+class DataProfile:
+    """Aggregate profile of a :class:`DataBatch`."""
+
+    row_count: int
+    column_count: int
+    columns: tuple[ColumnProfile, ...] = ()
+    sampled_at: datetime = field(default_factory=lambda: datetime.now(UTC))
+
+    def column(self, name: str) -> ColumnProfile | None:
+        """Return the per-column profile for ``name`` or ``None``."""
+        for c in self.columns:
+            if c.name == name:
+                return c
+        return None

pirn_data/data_schema.py

@@ -0,0 +1,85 @@
+"""Tabular schema declaration used across ``pirn_data``."""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from dataclasses import dataclass, field
+from typing import Any
+
+from pirn.core.pirn_opaque_value import PirnOpaqueValue
+
+
+@dataclass(frozen=True)
+class DataSchema(PirnOpaqueValue):
+    """Declarative schema for a tabular :class:`DataBatch`.
+
+    Attributes
+    ----------
+    columns:
+        Mapping of column name → expected Python type. Insertion order is
+        the canonical column order.
+    primary_keys:
+        Subset of ``columns`` keys; non-empty for any sink that performs
+        upsert or dedup operations.
+    nullable:
+        Subset of ``columns`` keys whose values may be ``None``.
+    """
+
+    columns: Mapping[str, type] = field(default_factory=dict)
+    primary_keys: tuple[str, ...] = ()
+    nullable: tuple[str, ...] = ()
+
+    def __post_init__(self) -> None:
+        unknown_pks = [k for k in self.primary_keys if k not in self.columns]
+        if unknown_pks:
+            raise ValueError(f"primary_keys reference unknown columns: {unknown_pks}")
+        unknown_nullable = [k for k in self.nullable if k not in self.columns]
+        if unknown_nullable:
+            raise ValueError(f"nullable references unknown columns: {unknown_nullable}")
+
+    @property
+    def column_names(self) -> tuple[str, ...]:
+        return tuple(self.columns.keys())
+
+    def is_nullable(self, column: str) -> bool:
+        """Return True if ``column`` is permitted to hold ``None``."""
+        return column in self.nullable
+
+    def with_columns(self, columns: Mapping[str, type]) -> DataSchema:
+        """Return a new schema with the given columns merged in."""
+        merged = dict(self.columns)
+        merged.update(columns)
+        return DataSchema(
+            columns=merged,
+            primary_keys=self.primary_keys,
+            nullable=self.nullable,
+        )
+
+    def _pirn_audit_dict(self) -> dict[str, Any]:
+        """Flatten to a primitive dict for pydantic serialisation.
+
+        ``columns`` holds Python ``type`` objects which pydantic's
+        default JSON serialiser can't dump; here we convert each type
+        to its name. Pirn IO validation just checks
+        ``isinstance(value, DataSchema)``; content-addressing serialises
+        via this stable summary.
+        """
+        return {
+            "columns": {k: t.__name__ for k, t in self.columns.items()},
+            "primary_keys": list(self.primary_keys),
+            "nullable": list(self.nullable),
+        }
+
+    def __pirn_canonical__(self) -> dict[str, Any]:
+        """Sanctioned canonical form for :func:`pirn.core.hashing.content_hash`.
+
+        Mirrors :meth:`_pirn_audit_dict` but is the explicit hook the
+        hasher prefers. Keeping both methods avoids forcing every
+        existing pydantic-serialisation call site through the canonical
+        path (and vice versa).
+        """
+        return {
+            "columns": {name: column_type.__name__ for name, column_type in self.columns.items()},
+            "primary_keys": list(self.primary_keys),
+            "nullable": list(self.nullable),
+        }