equiframe 0.12.4__py3-none-any.whl

equiframe/docs/extending.md

@@ -0,0 +1,158 @@
+# Integrating your own data and implementations
+
+[Handbook](index.md) · [API](api.md) · [Semantics](semantics.md)
+
+Start with the smallest boundary you need. `diff` compares already-materialized
+results without adopting the execution runner. Use an explicit source/adapter or
+compiler only when you need EquiFrame to capture or execute your integration.
+The core does not import the benchmark, a cloud provider or a domain warehouse.
+
+## Reuse a pinned snapshot with core alone
+
+`SnapshotClient`, `SnapshotDataSource`, `SnapshotColumn`, `SnapshotTable`,
+`SnapshotManifest` and `snapshot_fingerprint` are available through `equiframe`
+and `equiframe.snapshot`. These core exports are experimental. The existing
+`equiframe_benchmark` and `equiframe_benchmark.snapshot` imports remain exact
+aliases; no migration of historical manifest format 1.0 is required.
+
+The base installation can construct/validate manifests without Arrow, a native
+engine or the benchmark. Resolving data needs the `arrow` extra. Executing a
+native transformation additionally needs its corresponding engine extra. No
+new base dependency, remote transport or implicit plugin discovery was added.
+
+A client implements two methods:
+
+- `get_snapshot(snapshot_id: str) -> SnapshotManifest`
+- `download_artifact(ref: ArtifactRef, destination: Path) -> None`
+
+The client owns authentication, transport timeouts and service coordination.
+The source fetches the manifest once at construction, refuses `latest`, pins its
+identity/revisions and optionally selects a declared subset of tables. Each
+resolve downloads to a fresh private run directory and verifies bytes, SHA-256,
+row count, names/order and declared physical/logical types before returning a
+`LoadedTable`. It records the complete manifest identity even for a selection.
+A failed client is not retried silently. The provider must supply genuine
+immutable snapshots; the consumer cannot make a mutable service immutable.
+
+The source-distribution example `examples/snapshot_source.py` is self-contained.
+After installing the identified core wheel with the `duckdb` extra and copying
+the example outside the checkout, run:
+
+```bash
+python snapshot_source.py --output fresh-snapshot-example
+```
+
+It uses an injected in-memory client and synthetic Arrow IPC bytes, executes a
+native DuckDB identity transformation, then checks exact Decimal, NULL and
+multiplicity against a literal synthetic table. It needs no benchmark, credentials
+or network service. This is a technical integration example, not acceptance of a
+real service or independent human verification.
+
+The snapshot manifest retains its historical ASCII-escaped canonical JSON and
+fingerprint algorithm. Do not substitute contract canonicalization, rename fields,
+normalize strings or rewrite older evidence to make an integration fit.
+
+## A complete external SQLite producer
+
+The [SQLite snapshot integration](sqlite-snapshots.md) implements this same port
+outside core with a native owned read transaction, explicit Arrow schemas and
+immutable detached bytes. Its executable examples and tests demonstrate a fixed
+multi-table snapshot during a committed WAL update, exact Decimal/NULL, explicit
+differences and failed byte/schema validation. It is a local reference adapter,
+not a claim of portable isolation semantics across arbitrary DB-API drivers.
+
+## Choose an extension port
+
+For CSV/JSON/Parquet, read-only SQLite, API responses, typed Arrow batches and
+composition of multiple explicit providers, see [universal inputs](universal-sources.md).
+External inputs can implement `load(context) -> LoadedTable` and be passed to
+`LocalDataSource`. Existing `DataSource` providers can be bound through
+`CompositeDataSource` without changing core or discovering plugins at import.
+
+| Boundary | Implement or register | Responsibility |
+|---|---|---|
+| Captured input | `DataSource` (`table_names`, `resolve`) | Fixed table identity, faithful `LoadedTable`, explicit failure |
+| Artifact storage | `ArtifactStore` (`put`, `get`) | Declared bytes/format and verified hash/size |
+| Native execution | `EngineAdapter` plus `TransformationRegistry` | Your native implementation, declared capabilities and actual output metadata |
+| Bounded relational migration | `PlanCompiler` plus `CompilerRegistry` | Validate/support the declared IR and compile a genuine native transformation |
+| Additional rules | `RuleRegistry` | Evaluate one execution within its declared applicability and budgets |
+| Local publication | `Destination` | Honor verification, idempotency and publication semantics |
+
+`PlanCompiler.compile` is a lower-level extension operation: a compiled
+transformation assumes its actual inputs match the validated IR contract.
+`migrate` enforces the physical/value admission before running either engine.
+Calling the compiled transformation through `execute` alone does not reproduce
+that preflight. Native casts can change mismatched input data; `completed` only
+records execution/materialization and establishes no equivalence or correctness.
+
+Use explicit instance-owned registries. Registration does not prove support;
+preflight validation and execution still enforce the contract. Do not place
+callback/module names into JSON and expect them to be imported. This library is
+not a sandbox for untrusted extensions.
+
+## Preserve the complete loaded-table boundary
+
+A `LoadedTable` can carry an Arrow table, native column types, an index,
+`meaningful_index`, source reference and notes. Preserve those fields when
+wrapping an existing source. Keeping only `.table` or a data-file reference can
+lose the distinction between native and transport types or silently discard an
+index. Meaningful indexes remain unsupported for the current migration IR and
+must be rejected; do not drop one to make migration pass.
+
+Generic workflows capture each external source once. Independent stages can
+continue after another source fails under `skip_dependents`; their declared
+resource limits remain enforced separately. A shared-source capture uses the
+permitted consumer envelope, followed by each stage's own limits. Missing or
+invalid inputs never become empty successful tables.
+
+## Bounds you still own
+
+Source resolution, callbacks and Arrow decoding can allocate memory before a
+cooperative checkpoint. `RunContext` cancellation, time and byte budgets do not
+preempt a blocked network call or impose a hard process-memory quota. Snapshot
+manifest fetching precedes run budgets. Use bounded clients and an appropriately
+isolated execution service when you need aggregate quotas or hard timeouts.
+
+Data-service publication, writer coordination, retention, backups and domain
+mapping remain external responsibilities. The separate SERCOP/DW scripts in the
+repository are owner-maintained prototypes, excluded from the two distributable
+packages. Neither this port nor those scripts establish production readiness,
+accepted real DW integration or support for every Python platform/data type.
+
+## Arrow/Parquet partitions and external-source boundaries
+
+The current `DataSource.resolve` and `TableInput.load` ports return an eager
+`LoadedTable`; `ArtifactInput` verifies one local Arrow IPC or Parquet file and
+then materializes its table. A directory of Hive partitions or a remote object
+prefix is not an accepted `ArtifactRef`. Do not present those interfaces as a
+streaming dataset scanner.
+
+The existing comparison reader already iterates Arrow record batches and Parquet
+batches under `ComparisonOptions.batch_rows`, and spills comparison partitions
+to private SQLite. New [CSV/JSONL ingestion batches](ingestion.md) emit a normal
+multi-batch Arrow IPC artifact that this reader can consume without a new file
+format. An IPC record batch or Parquet decode may allocate before a checkpoint;
+these are cooperative bounds. `read_table` and execution remain eager.
+
+An application can use the existing explicit source port to capture its own
+partitioned dataset, but must pin the complete fragment inventory, versions,
+schemas, multiplicity and order policy, verify every fragment, and return the
+complete declared table. Comparing partitions separately can miss cross-partition
+keys, duplicates or moves. No new partition-directory connector, implicit
+partition-column coercion, concatenation protocol or distributed executor is
+implemented or needed for the bounded local ingestion change.
+
+SQLite remains the implemented external-producer reference described above:
+a single owned native read transaction, explicit schemas, detached immutable
+Arrow bytes and the existing `SnapshotClient` contract. Its supported schema,
+size limits and consistency tests are specific to that producer. They do not
+establish PostgreSQL isolation or a generic DB-API connector.
+
+No requested source, schema or credentials demonstrate a need for new Excel,
+PostgreSQL or object-storage connectors in this change. Those remain external
+extensions, not installed capabilities. An Excel producer must declare formula
+versus cached values, cell types and textual identifiers; PostgreSQL needs an
+owned consistent multi-query snapshot and exact type mappings; object storage
+needs a fixed inventory/version identity, bounded transport and verified bytes.
+Use existing `DataSource`/`SnapshotClient` ports if such a concrete consumer is
+provided. None of this changes the data-service or domain-warehouse ownership.

equiframe/docs/fast-ci.md

@@ -0,0 +1,52 @@
+# Quick local source checks
+
+[Handbook](index.md) · [Contributing](contributing.md) · [Performance](performance.md)
+
+The quick selection runs locally. Publication is manual; this public repository
+does not configure a GitHub Actions workflow. A quick pass covers the selected
+source checks and does not establish full-suite, installed-package, platform or
+release readiness.
+
+## Run the selection
+
+Start at the repository root with a supported interpreter and `uv`. These
+commands select Python 3.12; choose 3.10, 3.11 or 3.13 for another interpreter.
+The runner uses its actual interpreter for strict typing.
+
+```bash
+cd /path/to/equiframe
+uv sync --locked --all-extras --no-editable --python 3.12
+uv run --locked --all-extras python tools/quick_check.py --package core
+cd packages/equiframe-benchmark
+uv sync --locked --all-extras --no-editable --reinstall-package equiframe --python 3.12
+uv run --locked --all-extras python ../../tools/quick_check.py --package benchmark
+cd ../..
+```
+
+Use `--list` to inspect commands without executing them. `--output /new/local/path`
+retains the receipt, logs and pytest files in that new directory. Without it,
+the runner uses a fresh repository `.cache/quick-<id>` directory. Existing output
+directories are rejected. A failed command stops the selection and retains its
+actual exit code; an interrupted or incomplete run is not a pass. Logs stay local
+and may include ordinary diagnostic paths and synthetic fixture values.
+
+## What is checked
+
+- Whole-package Ruff lint, formatting, strict mypy and generated-asset drift.
+- The core architecture boundary and an explicit selection of contract, semantic,
+  migration, snapshot, compatibility, CLI and product-journey tests.
+- An explicit benchmark selection covering core aliases, grouped splits, training
+  selection, evaluation boundaries, the Verify bridge and CLI diagnostics.
+
+The exact test lists live in `tools/quick_check.py`. The runner deliberately
+imports the current source directories of the core and benchmark. Its receipt
+records the commit, worktree status, lock digest, interpreter, platform, commands,
+exit codes and elapsed time. Dependency installation is outside those timings.
+
+## Before a release
+
+Run the complete checks in [Releasing](releasing.md): full tests, builds, strict
+Twine validation, clean wheel/sdist consumers, exact artifact receipts and the
+affected dependency and platform combinations. Those checks run separately from
+this source-development selection. Local command output does not establish that
+a remote job ran or that a package was published.

equiframe/docs/first-use.md

@@ -0,0 +1,78 @@
+# First use by role / Primer uso por perfil
+
+[Install an identified artifact](installed-private.md) · [Arquitecto](arquitecto.md) ·
+[Científico](cientifico.md) · [API](api.md)
+
+For **0.12.4**, use the identified wheel and its reviewed
+dependencies. Run these commands with the Python of that environment, outside
+the source checkout. On Windows, replace `python` with `.venv\Scripts\python.exe`;
+on Linux use `.venv/bin/python`. Activation is optional.
+
+## Español: instalar, comprobar, ejecutar e interpretar
+
+1. Sigue [la instalación offline](installed-private.md), comprueba el SHA-256
+   recibido por un canal de confianza y ejecuta `python -m pip check`.
+   Para arquitecto instala `[duckdb]`; para científico basta `[arrow]`.
+   No necesitas ML, benchmark, OCI, una cuenta nueva ni un servicio adicional.
+2. Comprueba el intérprete antes de preparar datos:
+
+   ```text
+   python -I -m equiframe doctor --extra duckdb
+   python -I -m equiframe doctor --extra arrow --json
+   ```
+
+   `Metadata ready: True` comprueba identidad de versión y disponibilidad local;
+   no prueba rangos, DLL, ejecución nativa ni corrección de datos. El JSON conserva
+   esa distinción. Una dependencia ausente o metadatos de otra versión devuelven
+   **3** y una acción de recuperación. El comando no instala ni usa la red.
+3. Elige **uno** de estos ejercicios, con una carpeta de salida nueva:
+
+   ```text
+   python -I -m equiframe demo --role architect --output sql-good
+   python -I -m equiframe demo --role scientist --output prep-good
+   ```
+
+   Arquitecto: consulta DuckDB local con valores enlazados y esperado literal.
+   Científico: IDs con ceros iniciales, Decimal exacto en segundos, NULL y
+   timestamps UTC. Las tablas son sintéticas, pequeñas y autocontenidas.
+4. Comprueba que también detecta problemas:
+
+   ```text
+   python -I -O -m equiframe demo --role scientist --case different --output prep-different
+   python -I -O -m equiframe demo --role architect --case not-evaluable --output sql-limit
+   ```
+
+   `equivalent` acepta y devuelve **0**; `different` rechaza y devuelve **1**;
+   `not_evaluable` rechaza y devuelve **3**. Los errores de argumentos devuelven
+   **2**; fallo de escritura, dependencia o carpeta ya existente devuelve **3**.
+   Conserva `$LASTEXITCODE` en PowerShell o `$?` en Linux. No conviertas un rechazo
+   esperado del ejercicio en aceptación de datos de tu pipeline.
+5. Abre el archivo HTML indicado por la salida. `report/report.html` minimiza
+   datos; `synthetic-detail/report.html` muestra la clave, columna y valores del
+   ejercicio sintético. Se guardan antes del gate. Ambas vistas representan el
+   mismo resultado. Las capturas completas permanecen en `private/`: ocultar
+   valores en una vista no borra el original. Repetir la misma salida se rechaza;
+   elige otra carpeta y conserva la anterior si necesitas comparar intentos.
+
+Para sustituir los datos sintéticos por tu pipeline, sigue la guía larga de tu
+perfil y usa `diff` → `write_comparison_report` → `assert_equivalent`. No existe
+una traducción automática de SQL/Python arbitrario ni un servicio desplegado.
+`--help` describe los parámetros reales. Ambos comandos nuevos son experimentales.
+
+## English: a short installed route
+
+Install the identified wheel with `[duckdb]` for the architect or `[arrow]`
+for the scientist. Run `python -m pip check`, then `doctor --extra duckdb` or
+`doctor --extra arrow`. A metadata-ready diagnosis is not native validation.
+
+Run `demo --role architect --output sql-good` or
+`demo --role scientist --output prep-good` through `python -I -m equiframe`.
+Use `--case different` or `--case not-evaluable` and a **new** output directory to
+inspect rejection. Exit meanings are 0 accepted, 1 different, 2 invalid CLI usage,
+3 unavailable/failure. `-O` does not disable the gate. Reports are persisted before
+the verdict and never replace an existing destination. Full synthetic detail is
+separate from the default redacted report; raw evidence remains private.
+
+Use the role guide for integration with your own Python/SQL and local pytest.
+No automated runtime measurement is presented as a five-minute human observation.
+The independent human-adoption protocol remains [pending](adoption.md).

equiframe/docs/index.md

@@ -0,0 +1,142 @@
+# EquiFrame user handbook
+
+[File loading and data conversion](lectura-conversion.md) creates Pandas inputs
+from JSON, Excel, TXT, XML, CSV, Parquet or GeoJSON and offers Polars/DuckDB targets.
+[Optional independent review in R](r_review.md) adds a second exact comparison
+after rules and Isolation Forest without suppressing their alerts.
+
+[Experimental standalone EDA](eda.md): raw quality, explicit conversions, grouped
+statistics and investigation candidates in separate private runs. Diagnostics
+never replace full comparison, Verify, model fitting or publication gates.
+
+
+[Primer uso por perfil / First use](first-use.md)
+
+[Recorrido del arquitecto](arquitecto.md) · [Recorrido del científico](cientifico.md)
+
+[Inicio en español](inicio.md) · [Choose an API](choosing-api.md) · [Eight journeys](journeys.md)
+
+[Lectura JSON, Excel, TXT, XML, CSV, Parquet y GeoJSON con conversión Pandas → Polars/DuckDB](lectura-conversion.md)
+
+**Semantic comparison and reviewable evidence for tabular results in Python.**
+
+EquiFrame lets you declare what a result must preserve, compare materialized
+outputs, and locate differences in values, types, keys, multiplicity and order.
+You can use it inside an existing pandas, Polars, Arrow or CI workflow without
+adopting its execution runner. Native execution, bounded declarative migration,
+operational verification and anomaly-detection interfaces are separate services.
+Their compatibility commitments do not establish scientific detection accuracy.
+
+## Nuevas fuentes y ventanas
+
+El [recorrido de universalidad en cinco pasos](universalidad-cinco-pasos.md)
+ejecuta seis entradas bajo un contrato y tres motores nativos desde el wheel.
+Consulta [fuentes](universal-sources.md), [ventanas](migration-ir.md) y
+[extensiones](extending.md) para sus interfaces y límites.
+
+## Release status
+
+This handbook targets **0.12.4**, preserving the **0.12.0 stable release contract**. Preparation and
+publication are separate: the owner has authorized a manual TestPyPI rehearsal
+followed by PyPI, subject to the [release gates](releasing.md). Until registry
+ownership, uploads and clean consumer installation are actually verified, those
+steps remain pending; a source version or a passing local test is not publication.
+Use the exact wheel/sdist SHA-256 and source identity for each identified artifact.
+Historical private Alpha notes retain their original versions and results.
+
+Project-owned code uses MIT. Included third-party components retain their
+licenses; the core archive expression is `MIT AND Apache-2.0` because it includes
+CycloneDX schemas. See [licensing](licensing.md) and the packaged notices.
+There is no claimed production certification, scientific accuracy, independent
+human adoption, security-response SLA or accepted real data-warehouse integration.
+
+Hernán Troya (`htroya`) is the maintainer and release/security owner, on a
+best-effort basis without a promised backup or response deadline. The public
+[issue channel](https://github.com/htroya/equiframe/issues) and
+[private vulnerability reporting](https://github.com/htroya/equiframe/security/advisories/new)
+are reporting routes of the project's single public repository. For each
+release, the maintainer must enable both routes and verify their availability.
+See [maintenance](maintenance.md) and [security](security.md); never submit
+sensitive details in a public issue.
+
+## Start here
+
+1. [Quick start](quickstart.md): identified artifacts, a clean environment,
+   a complete small comparison and a fail-closed assertion
+2. [Three installed recipes](recipes.md): diagnose a pandas refactor, validate a
+   pandas → DuckDB migration and retain evidence before a pytest/CI failure
+3. [Public API reference](api.md): entry points, models, CLI and safe result states
+4. [Semantics, guarantees and limits](semantics.md): what equality means, what is
+   preserved, what is rejected and what evidence proves
+5. [Compatibility](compatibility.md): stability, dependency extras, measured
+   platforms and the distinction between declared and demonstrated support
+6. [Performance and resource budgets](performance.md): cost boundaries, bounded
+   measurement, missing finalization and practical budget choices
+7. [Upgrading and troubleshooting](upgrading.md): safe acceptance, historical
+   evidence, deprecations and actionable failure diagnosis
+8. [Extension ports and pinned snapshots](extending.md): reuse your own data client
+   without the benchmark, preserve metadata and understand resource ownership
+9. [Contributing](contributing.md): reproducible checks, review and ownership
+10. [Dependency inventory](dependencies.md), [security](security.md) and
+   [release process](releasing.md): artifact identity, supply-chain evidence,
+   disclosure and current release gates
+11. [Adoption](adoption.md) and [maintenance](maintenance.md): consented real-project
+    observations, later-change retention, useful issue reports and critical trust incidents
+
+## Artifact and integration guides
+
+- [Normal pip installation from an identified offline wheelhouse](installed-private.md)
+- [Native relational extensions and per-engine limits](migration-ir.md)
+- [CSV/JSONL batches and exact ingestion receipts](ingestion.md)
+- [Installed option inventory and honest coverage denominators](option-audit.md)
+
+## Reading this handbook without the repository
+
+The source distribution contains `README.md`, this `docs/` directory, examples,
+tests, tools and JSON schemas. The wheel carries the same handbook as package
+resources under `equiframe/docs/`; it does not require a source checkout.
+After installing the identified wheel, locate it with:
+
+```python
+from importlib.resources import files
+
+handbook = files("equiframe").joinpath("docs")
+print(handbook.joinpath("index.md").read_text(encoding="utf-8"))
+```
+
+The first-use kit additionally contains the exact wheel, dependency pins and
+hashes, four recipe/test scripts, an integrity verifier and English guides.
+[Recipes](recipes.md) explains its layout. The package's README is also part of
+its distribution metadata. Relative links within this handbook stay within the
+handbook; no document from another repository is required to follow the core guide.
+
+## Choosing the right service
+
+| Need | Entry point | Important boundary |
+|---|---|---|
+| Compare eager tables or an explicitly supplied DuckDB relation | `diff` | Right-hand reference by default; the relation authorizes consumption of its caller-owned native plan, not inference of arbitrary SQL/connections |
+| Compare existing captured outputs or local files | `compare` | Explicit contract; peer comparison by default |
+| Fail a test on anything short of accepted equivalence | `assert_equivalent` | Never use truth testing of the result |
+| Explain a comparison safely | `comparison_report` | Redacted by default; a view is not the acceptance gate |
+| Capture one trusted DuckDB query | `capture_duckdb` | Explicit execution and ownership; detached snapshot |
+| Execute a registered native transformation | `execute` | `completed` means execution, not semantic correctness |
+| Compile and compare a bounded relational plan | `migrate` | Native implementations; not arbitrary program translation |
+| Verify existing outputs against a reviewed reference | `verify` | Agreement, expected-output checks and review remain distinct |
+| Retain transportable finalized evidence | `export_bundle` | Only the supported closed inventories; hashes are not signatures |
+| Study anomalies or engineering costs | Separate `equiframe_benchmark` | Experimental protocol; synthetic checks are not scientific results |
+
+The [outcome-gate guide](outcome-gates.md) lists the result/decision models that
+reject implicit truth testing and the explicit field appropriate to each.
+
+## The minimum safe interpretation
+
+An `equivalent` result means the **captured outputs satisfy the evaluated
+contract**, including any explicitly authorized tolerance or normalization. It
+does not establish universal program equivalence or business correctness. Two
+engines can share a bug, and a declared expected result can be wrong.
+
+`different` and `not_evaluable` both fail an equivalence gate. A complete scan
+with an invalid expected result is still `not_evaluable`. Missing finalization,
+failed publication, resource exhaustion and an unhandled exception are never a
+pass. See [semantics](semantics.md) before using a result to approve a deployment
+or data publication.

equiframe/docs/ingestion.md

@@ -0,0 +1,145 @@
+# Ingest CSV and JSONL in bounded value batches
+
+[Handbook](index.md) · [API](api.md) · [Extension ports](extending.md)
+
+Use `ingest_file` when converting a local text source under an explicit schema.
+It preserves the original bytes and SHA-256, exact conversion receipts, rejected
+record locations, duplicates and declared Decimal/time precision. It does not
+infer a schema, repair values or establish business correctness.
+
+## A complete small example
+
+After installing an identified artifact with the `arrow` extra, run
+this in a fresh directory. The input is deliberately synthetic.
+
+```python
+from decimal import Decimal
+from pathlib import Path
+import equiframe as ef
+import pyarrow as pa
+
+path = Path("example.csv")
+path.write_text("id,money\n001,1.10\n001,1.10\n002,NULL\n", encoding="utf-8")
+spec = ef.IngestionSpec(
+    format="csv",
+    fields=(
+        ef.FieldMapping(
+            column=ef.ColumnSpec(
+                name="id",
+                logical_type=ef.StringType(),
+                nullable=False,
+                null_policy=ef.NullPolicy(representations=("null",)),
+            ),
+            path=("id",),
+        ),
+        ef.FieldMapping(
+            column=ef.ColumnSpec(
+                name="money",
+                logical_type=ef.DecimalType(precision=10, scale=2),
+                nullable=True,
+                null_policy=ef.NullPolicy(representations=("null",)),
+                numeric_policy=ef.NumericPolicy(comparison="exact", unit="USD"),
+            ),
+            path=("money",),
+            null_tokens=("NULL",),
+        ),
+    ),
+)
+result = ef.ingest_file(path, spec=spec, work_dir=Path("ingestion"), batch_rows=2)
+if result.state != "completed" or result.output is None:
+    raise RuntimeError("Review the ingestion receipt before accepting these data")
+expected = pa.table(
+    {
+        "id": ["001", "001", "002"],
+        "money": pa.array([Decimal("1.10"), Decimal("1.10"), None], type=pa.decimal128(10, 2)),
+    }
+)
+comparison = ef.diff(ef.read_table(result.output), expected, work_dir=Path("comparison"))
+report = ef.write_comparison_report(comparison, Path("report"))
+print(report.to_text())
+ef.assert_equivalent(comparison)
+```
+
+`batch_rows` is an execution knob, default **4096**, accepted by `ingest_file`
+and `FileInput(path, spec, batch_rows=4096)`. It must be a positive Python integer;
+booleans, floats, strings, zero and negative values fail before creating files.
+There is no artificial upper bound: a large batch requests a correspondingly
+large allocation. Record and source budgets still apply. The CLI uses the
+default. The `IngestionSpec`/`IngestionResult` JSON format remains **1.0** and gains
+no field: record the chosen batch size in your application's run configuration.
+Existing calls and historical receipts need no migration.
+
+## What is bounded, and what is not
+
+- CSV and JSONL use incremental text decoding and at most one batch of converted
+  accepted values before an Arrow IPC write. They no longer retain the full
+  decoded file or a list of all JSONL documents
+- An initial bounded decode-validation pass retains the historical behavior for
+  invalid encoding: the entire source is rejected before accepting any row.
+  The preserved raw file is then read again; this costs an extra sequential pass
+- CSV record positions remain `$[index]`. JSONL keeps its historical `line:N`
+  plus nested JSON path; array expansion and root/ancestor scopes are unchanged.
+  These are logical record/line positions, not byte offsets. CSV quoted newlines
+  retain the prior universal-newline decoding behavior, while original bytes do
+  not change
+- Full JSON documents still materialize. A single large JSONL document and its
+  explicit nested expansion can also materialize. `max_source_bytes` bounds the
+  copied source; `max_records` bounds expanded/inspected records, not parser RSS
+- The public result still retains **all** conversion and quarantine receipt
+  objects. Result JSON is now serialized in chunks with exactly the historical
+  canonical bytes, avoiding a second complete receipt dictionary/string copy.
+  Receipt objects and a single large string/value are not bounded by batch size
+- Context row and Arrow-buffer budgets are cumulative across batches. Arrow
+  buffer accounting includes each batch's buffers (including validity bitmaps).
+  Physical IPC bytes and hashes can differ with chunk boundaries; semantic
+  values, multiplicity, order and receipts must not. These are cooperative
+  checks, not hard memory, CPU or disk quotas
+- `FileInput.load`, `read_table` and native execution still return/materialize a
+  complete `LoadedTable`. Batching ingestion does not make a whole workflow
+  streaming. Existing file-based comparison readers have their own batch/spill
+  support; see the [extension boundary](extending.md#arrowparquet-partitions-and-external-source-boundaries)
+
+## CSV headers and surplus cells
+
+Any unique, nonempty declared CSV header is legitimate, including names such as
+`<surplus_fields>` or `null`. Diagnostic labels never replace named source cells.
+Rows with additional cells follow `extra_fields`: `ignore` discards only the
+undeclared surplus cells before scalar conversion; `reject` records
+`new_or_extra_field` and follows the declared `invalid_records` policy.
+
+Scalar conversion and diagnostics use separate records. For non-colliding names,
+overflow diagnostics retain their historical `original_text` bytes, including
+the existing `null` and `<surplus_fields>` labels. If either label is an actual
+header or declared field path, the diagnostic instead uses the unambiguous
+`{"fields": {named source cells}, "surplus_fields": [extra cells]}` shape inside
+`original_text`. This wrapper is a narrow diagnostic difference for collision
+cases, not a new ingestion schema version; historical 1.0 receipts stay readable.
+No diagnostic labels can become the value of a declared missing field.
+
+Original source bytes, SHA-256, source size, logical record positions and
+conversion receipts for named fields are retained. Missing cells still follow
+their declared `missing` policy and never become surplus values.
+
+## Rejection, quarantine and interruption
+
+`rejected` has no output reference, even if valid rows had already been flushed
+to private staging. `quarantined` publishes accepted rows but retains the rejected
+records and incomplete coverage. `FileInput.load` accepts only `completed`.
+A successful parse alone does not imply equivalence; use an independent expected
+output or the other requested verification checks.
+
+Staged Arrow files are removed on parse rejection, budget exhaustion,
+cancellation, timeout or write failure before publication. Resource and I/O
+errors raise rather than become a successful result. Cancellation or a failure
+while writing the final receipt leaves no `ingestion.json`; its unique operation
+folder may still contain original bytes or already completed artifact files.
+Their existence is not a finalized result. Existing operations are never
+replaced. Original sources and unredacted receipts can contain private values;
+keep evidence in an authorized private directory.
+
+The focused tests exercise varying batch sizes, independent exact expected
+values, duplicates, nulls, Unicode, encodings/separators, nested paths, empty
+inputs, cumulative budgets and interruptions. Platform and installed-artifact
+claims require the separate receipt for the exact candidate; source tests and
+historical results are not that receipt. No constant-memory or universal speedup
+claim is made.