mic-evals 0.1.0__tar.gz

mic_evals-0.1.0/.gitignore

@@ -0,0 +1,12 @@
+.venv/
+.cache/
+.artifacts/
+.pytest_cache/
+.ruff_cache/
+__pycache__/
+*.pyc
+.mic/
+dist/
+*.egg-info/
+node_modules/
+.coverage

mic_evals-0.1.0/.python-version

@@ -0,0 +1 @@
+3.12

mic_evals-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ryan Eiger
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mic_evals-0.1.0/PKG-INFO

@@ -0,0 +1,268 @@
+Metadata-Version: 2.5
+Name: mic-evals
+Version: 0.1.0
+Summary: Typed, provider-independent micro-evaluations for Python
+Project-URL: Homepage, https://github.com/rybosome/mic
+Project-URL: Repository, https://github.com/rybosome/mic
+Project-URL: Issues, https://github.com/rybosome/mic/issues
+Author: Ryan Eiger
+License-Expression: MIT
+License-File: LICENSE
+Keywords: evals,evaluation,llm,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Provides-Extra: bigquery
+Requires-Dist: google-cloud-bigquery<4,>=3.30; extra == 'bigquery'
+Provides-Extra: braintrust
+Requires-Dist: braintrust==0.39.0; extra == 'braintrust'
+Provides-Extra: pydantic
+Requires-Dist: pydantic<3,>=2.11; extra == 'pydantic'
+Description-Content-Type: text/markdown
+
+# mic
+
+[![CI](https://github.com/rybosome/mic/actions/workflows/ci.yml/badge.svg)](https://github.com/rybosome/mic/actions/workflows/ci.yml)
+
+**Did that prompt, model, or application change actually improve the answers?**
+
+Mic is a small Python evaluation harness for systems whose answers can vary.
+Give it examples, the code you want to evaluate, and a way to score the results.
+Run the evaluation repeatedly, then inspect what happened, case by case.
+
+Use it to test an LLM classifier, score an agent's work, or evaluate another ML
+application. You keep your application code and choose your own models and scoring
+logic; Mic handles dataset validation, bounded concurrent execution, repeated
+trials, and local evidence. No hosted evaluation platform is required.
+
+## Evaluate a support-ticket classifier
+
+Suppose you're using an LLM to sort support messages into bugs, feature requests,
+and questions. Before changing its prompt or model, give yourself a repeatable check.
+
+With Python 3.12+, install Mic and the SDK used by this example:
+
+```console
+python -m pip install "mic-evals[pydantic]" openai
+```
+
+Mic's core has no third-party runtime dependencies. This example opts into Pydantic
+and the OpenAI SDK to share a typed output contract between Mic and the model call.
+It uses [structured outputs](https://developers.openai.com/api/docs/guides/structured-outputs)
+with [GPT-4.1 mini](https://developers.openai.com/api/docs/models/gpt-4.1-mini);
+replace the classifier function with your own model or application call.
+Set `OPENAI_API_KEY` in your environment using your usual secret-management method.
+Running it sends the example messages to OpenAI and incurs normal API charges.
+
+Save this complete example as `ticket_eval.py`:
+
+```python
+from typing import Literal
+
+from pydantic import BaseModel
+
+import mic
+
+##
+## Define a dataset
+##
+
+
+class Ticket(BaseModel):
+    subject: str
+    body: str
+
+
+class Classification(BaseModel):
+    label: Literal["bug", "feature", "question"]
+
+
+@mic.dataset(name="tickets", schema=mic.case_schema(input=Ticket, expected=Classification))
+def tickets() -> list[mic.RawCase]:
+    return [
+        mic.RawCase(
+            id="upload",
+            input=Ticket(subject="PDF upload", body="The app closes whenever I upload a PDF."),
+            expected=Classification(label="bug"),
+        ),
+        mic.RawCase(
+            id="export",
+            input=Ticket(
+                subject="Invoice export", body="Can you add an option to export invoices?"
+            ),
+            expected=Classification(label="feature"),
+        ),
+        mic.RawCase(
+            id="invoice",
+            input=Ticket(subject="Past invoice", body="Where can I download last month's invoice?"),
+            expected=Classification(label="question"),
+        ),
+    ]
+
+
+##
+## Define scoring
+##
+
+
+@mic.scorer(name="accuracy", requires_expected=True)
+def accuracy(
+    ctx: mic.ScoreContext[Ticket, Classification, Classification, mic.JsonObject],
+) -> float:
+    return float(ctx.output.label == ctx.require_expected().label)
+
+
+##
+## Define the task
+##
+
+
+@mic.eval(name="classify", dataset=tickets, output=Classification, scorers=[accuracy])
+def classify(
+    ctx: mic.TaskContext[Classification, mic.JsonObject], ticket: Ticket
+) -> Classification:
+    from openai import OpenAI
+
+    # Create the client only when the task runs, and close it after the call.
+    with OpenAI(timeout=30, max_retries=0) as client:
+        response = client.responses.parse(
+            model="gpt-4.1-mini",
+            instructions=(
+                "Classify the support ticket as "
+                "bug (broken behavior), feature (new capability), or question (how-to)."
+            ),
+            input=ticket.model_dump_json(),
+            text_format=Classification,
+            store=False,
+        )
+        if response.output_parsed is None:
+            raise ValueError("The model did not return a classification.")
+        return response.output_parsed
+```
+
+`Ticket` gives the task typed inputs; `Classification` defines both the expected
+answer and the model's structured output. The SDK derives its output schema from
+that class and parses the response into it. Mic validates dataset values and task
+outputs against the same types, so the scorer works with objects, not JSON parsing
+or string cleanup.
+
+A valid but wrong label scores `0`; the right label scores `1`. Invalid output or
+no parsed classification (for example, a refusal) is an execution failure, not a
+wrong answer. These three cases are illustrative; a useful evaluation needs a
+larger, representative set of labeled tickets.
+
+Run it from the directory containing `ticket_eval.py`, then open the report:
+
+```console
+mic run ticket_eval:classify --output .mic/tickets-first
+mic report .mic/tickets-first --open
+```
+
+That's three classifier calls. The report shows each message, its expected label,
+the actual response, and its score, alongside aggregate accuracy and execution
+failures. Review low-scoring cases for wrong answers or ambiguous expected labels,
+and execution failures for calls that did not produce a valid classification.
+
+Each run saves its dataset snapshot, per-trial results, and summary as JSON/JSONL,
+plus a self-contained HTML report you can open without a server or network.
+Explicit output directories must be empty; omit `--output` to get a unique directory
+automatically. A successfully executed run can still have poor scores—execution
+success and answer quality are separate.
+
+The same code lives in [examples/ticket_eval.py](examples/ticket_eval.py). To explore
+the report without credentials or API charges, the repository also includes an
+[offline classification demo](examples/triage.py) with deliberately imperfect rules;
+see the [walkthrough](docs/verification.md).
+
+## Repeat, check, improve
+
+### Look for variation
+
+Run each message five times, with at most two tasks executing concurrently:
+
+```console
+mic run ticket_eval:classify --trials 5 --concurrency 2 --output .mic/tickets-repeat
+```
+
+This makes 15 classifier calls. Inspect individual trials as well as the mean:
+one message that fails intermittently deserves attention even if the average looks
+good. Repetition gives you more observations, not proof of statistical significance.
+
+Change the prompt or model in `classify`, run again into a fresh directory,
+and review both reports against the same labeled cases. Keep the evaluation set
+representative rather than tuning only to these three examples.
+
+### Make quality a check
+
+Add a score requirement when you're ready to use the evaluation locally or in CI:
+
+```console
+mic run ticket_eval:classify --trials 5 --require 'accuracy>=0.9'
+```
+
+The command exits nonzero if execution fails or mean accuracy falls below `0.9`.
+That threshold is illustrative; choose one appropriate to your dataset and the cost
+of a wrong answer. A failed quality gate still leaves local results to investigate.
+
+### Grow the dataset without changing the task
+
+Save the same cases as `tickets.jsonl` beside `ticket_eval.py`:
+
+```jsonl
+{"id":"upload","input":{"subject":"PDF upload","body":"The app closes whenever I upload a PDF."},"expected":{"label":"bug"}}
+{"id":"export","input":{"subject":"Invoice export","body":"Can you add an option to export invoices?"},"expected":{"label":"feature"}}
+{"id":"invoice","input":{"subject":"Past invoice","body":"Where can I download last month's invoice?"},"expected":{"label":"question"}}
+```
+
+Replace the `tickets` definition with this, adding the two imports:
+
+```python
+from pathlib import Path
+
+from mic.providers.files import FileHandle
+
+
+@mic.dataset(name="tickets", schema=mic.case_schema(input=Ticket, expected=Classification))
+def tickets() -> FileHandle:
+    return FileHandle(Path(__file__).with_name("tickets.jsonl"))
+```
+
+Mic hydrates the JSON objects into `Ticket` and `Classification` instances and
+rejects invalid cases before calling the model. The task, scorer, and run command
+stay the same. Add cases from real failures as
+you encounter them—for example, a message that sounds like a feature request but
+describes an existing feature that stopped working.
+
+## Bring your own application
+
+The same pattern applies beyond classification: score extracted fields, check an
+agent's result against a rubric, or compute a metric for an ML prediction. Tasks
+and scorers are Python functions, so they can call your existing code. Sync and
+async functions are supported; scripts can use `mic.run()` and notebooks can use
+`await mic.arun()` instead of the CLI.
+
+- **Structured data:** use ordinary dataclasses for inputs, outputs, and expected
+  values; see the [structured example](examples/structured.py) and [schema guide](docs/schemas.md).
+- **Other dataset sources:** load local files, BigQuery queries, or versioned
+  Braintrust datasets with optional integrations; see [providers](docs/providers.md).
+- **Remote reporting:** optionally export results to a Braintrust experiment after
+  saving local evidence. Dataset storage and reporting are independent; see [reporting](docs/reporting.md).
+
+## Before you use it
+
+Mic is an early release, and its public API may change before a stable release.
+It runs finite, bounded datasets locally—not distributed jobs or an application
+hosting service. Synchronous callbacks must finish cooperatively: a timeout cannot
+forcibly stop a running Python thread.
+
+Reports and artifacts contain your actual evaluation data, including inputs and
+outputs. They are **not automatically redacted**. Treat them as sensitive, keep
+credentials out of your cases, and review evidence before sharing or committing it.
+Your model calls and optional cloud integrations have their own costs and data-handling
+policies. See [artifact handling](docs/artifacts.md) for details.
+
+[API reference](docs/api.md) · [Verification](docs/verification.md) ·
+[Contributing](CONTRIBUTING.md) · [Releasing](docs/releasing.md) · [MIT license](LICENSE)

mic_evals-0.1.0/README.md

@@ -0,0 +1,243 @@
+# mic
+
+[![CI](https://github.com/rybosome/mic/actions/workflows/ci.yml/badge.svg)](https://github.com/rybosome/mic/actions/workflows/ci.yml)
+
+**Did that prompt, model, or application change actually improve the answers?**
+
+Mic is a small Python evaluation harness for systems whose answers can vary.
+Give it examples, the code you want to evaluate, and a way to score the results.
+Run the evaluation repeatedly, then inspect what happened, case by case.
+
+Use it to test an LLM classifier, score an agent's work, or evaluate another ML
+application. You keep your application code and choose your own models and scoring
+logic; Mic handles dataset validation, bounded concurrent execution, repeated
+trials, and local evidence. No hosted evaluation platform is required.
+
+## Evaluate a support-ticket classifier
+
+Suppose you're using an LLM to sort support messages into bugs, feature requests,
+and questions. Before changing its prompt or model, give yourself a repeatable check.
+
+With Python 3.12+, install Mic and the SDK used by this example:
+
+```console
+python -m pip install "mic-evals[pydantic]" openai
+```
+
+Mic's core has no third-party runtime dependencies. This example opts into Pydantic
+and the OpenAI SDK to share a typed output contract between Mic and the model call.
+It uses [structured outputs](https://developers.openai.com/api/docs/guides/structured-outputs)
+with [GPT-4.1 mini](https://developers.openai.com/api/docs/models/gpt-4.1-mini);
+replace the classifier function with your own model or application call.
+Set `OPENAI_API_KEY` in your environment using your usual secret-management method.
+Running it sends the example messages to OpenAI and incurs normal API charges.
+
+Save this complete example as `ticket_eval.py`:
+
+```python
+from typing import Literal
+
+from pydantic import BaseModel
+
+import mic
+
+##
+## Define a dataset
+##
+
+
+class Ticket(BaseModel):
+    subject: str
+    body: str
+
+
+class Classification(BaseModel):
+    label: Literal["bug", "feature", "question"]
+
+
+@mic.dataset(name="tickets", schema=mic.case_schema(input=Ticket, expected=Classification))
+def tickets() -> list[mic.RawCase]:
+    return [
+        mic.RawCase(
+            id="upload",
+            input=Ticket(subject="PDF upload", body="The app closes whenever I upload a PDF."),
+            expected=Classification(label="bug"),
+        ),
+        mic.RawCase(
+            id="export",
+            input=Ticket(
+                subject="Invoice export", body="Can you add an option to export invoices?"
+            ),
+            expected=Classification(label="feature"),
+        ),
+        mic.RawCase(
+            id="invoice",
+            input=Ticket(subject="Past invoice", body="Where can I download last month's invoice?"),
+            expected=Classification(label="question"),
+        ),
+    ]
+
+
+##
+## Define scoring
+##
+
+
+@mic.scorer(name="accuracy", requires_expected=True)
+def accuracy(
+    ctx: mic.ScoreContext[Ticket, Classification, Classification, mic.JsonObject],
+) -> float:
+    return float(ctx.output.label == ctx.require_expected().label)
+
+
+##
+## Define the task
+##
+
+
+@mic.eval(name="classify", dataset=tickets, output=Classification, scorers=[accuracy])
+def classify(
+    ctx: mic.TaskContext[Classification, mic.JsonObject], ticket: Ticket
+) -> Classification:
+    from openai import OpenAI
+
+    # Create the client only when the task runs, and close it after the call.
+    with OpenAI(timeout=30, max_retries=0) as client:
+        response = client.responses.parse(
+            model="gpt-4.1-mini",
+            instructions=(
+                "Classify the support ticket as "
+                "bug (broken behavior), feature (new capability), or question (how-to)."
+            ),
+            input=ticket.model_dump_json(),
+            text_format=Classification,
+            store=False,
+        )
+        if response.output_parsed is None:
+            raise ValueError("The model did not return a classification.")
+        return response.output_parsed
+```
+
+`Ticket` gives the task typed inputs; `Classification` defines both the expected
+answer and the model's structured output. The SDK derives its output schema from
+that class and parses the response into it. Mic validates dataset values and task
+outputs against the same types, so the scorer works with objects, not JSON parsing
+or string cleanup.
+
+A valid but wrong label scores `0`; the right label scores `1`. Invalid output or
+no parsed classification (for example, a refusal) is an execution failure, not a
+wrong answer. These three cases are illustrative; a useful evaluation needs a
+larger, representative set of labeled tickets.
+
+Run it from the directory containing `ticket_eval.py`, then open the report:
+
+```console
+mic run ticket_eval:classify --output .mic/tickets-first
+mic report .mic/tickets-first --open
+```
+
+That's three classifier calls. The report shows each message, its expected label,
+the actual response, and its score, alongside aggregate accuracy and execution
+failures. Review low-scoring cases for wrong answers or ambiguous expected labels,
+and execution failures for calls that did not produce a valid classification.
+
+Each run saves its dataset snapshot, per-trial results, and summary as JSON/JSONL,
+plus a self-contained HTML report you can open without a server or network.
+Explicit output directories must be empty; omit `--output` to get a unique directory
+automatically. A successfully executed run can still have poor scores—execution
+success and answer quality are separate.
+
+The same code lives in [examples/ticket_eval.py](examples/ticket_eval.py). To explore
+the report without credentials or API charges, the repository also includes an
+[offline classification demo](examples/triage.py) with deliberately imperfect rules;
+see the [walkthrough](docs/verification.md).
+
+## Repeat, check, improve
+
+### Look for variation
+
+Run each message five times, with at most two tasks executing concurrently:
+
+```console
+mic run ticket_eval:classify --trials 5 --concurrency 2 --output .mic/tickets-repeat
+```
+
+This makes 15 classifier calls. Inspect individual trials as well as the mean:
+one message that fails intermittently deserves attention even if the average looks
+good. Repetition gives you more observations, not proof of statistical significance.
+
+Change the prompt or model in `classify`, run again into a fresh directory,
+and review both reports against the same labeled cases. Keep the evaluation set
+representative rather than tuning only to these three examples.
+
+### Make quality a check
+
+Add a score requirement when you're ready to use the evaluation locally or in CI:
+
+```console
+mic run ticket_eval:classify --trials 5 --require 'accuracy>=0.9'
+```
+
+The command exits nonzero if execution fails or mean accuracy falls below `0.9`.
+That threshold is illustrative; choose one appropriate to your dataset and the cost
+of a wrong answer. A failed quality gate still leaves local results to investigate.
+
+### Grow the dataset without changing the task
+
+Save the same cases as `tickets.jsonl` beside `ticket_eval.py`:
+
+```jsonl
+{"id":"upload","input":{"subject":"PDF upload","body":"The app closes whenever I upload a PDF."},"expected":{"label":"bug"}}
+{"id":"export","input":{"subject":"Invoice export","body":"Can you add an option to export invoices?"},"expected":{"label":"feature"}}
+{"id":"invoice","input":{"subject":"Past invoice","body":"Where can I download last month's invoice?"},"expected":{"label":"question"}}
+```
+
+Replace the `tickets` definition with this, adding the two imports:
+
+```python
+from pathlib import Path
+
+from mic.providers.files import FileHandle
+
+
+@mic.dataset(name="tickets", schema=mic.case_schema(input=Ticket, expected=Classification))
+def tickets() -> FileHandle:
+    return FileHandle(Path(__file__).with_name("tickets.jsonl"))
+```
+
+Mic hydrates the JSON objects into `Ticket` and `Classification` instances and
+rejects invalid cases before calling the model. The task, scorer, and run command
+stay the same. Add cases from real failures as
+you encounter them—for example, a message that sounds like a feature request but
+describes an existing feature that stopped working.
+
+## Bring your own application
+
+The same pattern applies beyond classification: score extracted fields, check an
+agent's result against a rubric, or compute a metric for an ML prediction. Tasks
+and scorers are Python functions, so they can call your existing code. Sync and
+async functions are supported; scripts can use `mic.run()` and notebooks can use
+`await mic.arun()` instead of the CLI.
+
+- **Structured data:** use ordinary dataclasses for inputs, outputs, and expected
+  values; see the [structured example](examples/structured.py) and [schema guide](docs/schemas.md).
+- **Other dataset sources:** load local files, BigQuery queries, or versioned
+  Braintrust datasets with optional integrations; see [providers](docs/providers.md).
+- **Remote reporting:** optionally export results to a Braintrust experiment after
+  saving local evidence. Dataset storage and reporting are independent; see [reporting](docs/reporting.md).
+
+## Before you use it
+
+Mic is an early release, and its public API may change before a stable release.
+It runs finite, bounded datasets locally—not distributed jobs or an application
+hosting service. Synchronous callbacks must finish cooperatively: a timeout cannot
+forcibly stop a running Python thread.
+
+Reports and artifacts contain your actual evaluation data, including inputs and
+outputs. They are **not automatically redacted**. Treat them as sensitive, keep
+credentials out of your cases, and review evidence before sharing or committing it.
+Your model calls and optional cloud integrations have their own costs and data-handling
+policies. See [artifact handling](docs/artifacts.md) for details.
+
+[API reference](docs/api.md) · [Verification](docs/verification.md) ·
+[Contributing](CONTRIBUTING.md) · [Releasing](docs/releasing.md) · [MIT license](LICENSE)

mic_evals-0.1.0/docs/api.md

@@ -0,0 +1,125 @@
+# Public API and execution contract
+
+The top-level `mic` package contains ordinary authoring, execution, schema, result, and
+error APIs. Advanced provider contracts are imported from `mic.providers`; reporter
+contracts are imported from `mic.reporters`. Runtime materialization and artifact
+implementation modules are private. The package contains `py.typed`, and the entire source
+passes strict Pyright.
+
+## Authoring
+
+- `@dataset(name=..., schema=..., map_row=...)` binds a no-argument source factory.
+  It can return a fresh synchronous/async iterable, an awaitable of either, or a
+  provider handle. The factory is invoked once per run or inspection.
+- `case_schema(input=..., expected=..., metadata=..., expected_policy=...)` accepts
+  ordinary Python annotations or `Schema[T]` adapters. Metadata defaults to `JsonObject`.
+  The dependency-free backend hydrates nested dataclasses from JSON mappings and
+  revalidates/clones native instances. Maps require string keys; native schemas are
+  strict and reject `strict=False`. Use explicit mapping for normalization.
+  Optional Pydantic adapters preserve constraints and aliases through strict JSON
+  hydration; canonical snapshots use Python field names. See [schemas](schemas.md).
+- `@scorer(name=..., requires_expected=True)` accepts a synchronous or async
+  function taking `ScoreContext[I,O,E,M]`. Each scorer defines exactly one metric,
+  named by the scorer. Return a finite `float`/`int`, `None` when inapplicable, or
+  `Score(value, metadata)` when the score needs JSON metadata.
+- `@eval(name=..., dataset=..., output=..., scorers=[...], trials=1, concurrency=10)`
+  accepts a task `(TaskContext[E,M], input: I) -> O | TaskResult[O]`, sync or async.
+  `TaskResult` is the only metadata wrapper. Ordinary dictionaries containing
+  `output` and `metadata` keys are not unpacked.
+- Descriptors expose their stable `name`; callback storage is an implementation detail.
+  Their constructors/decorators do not call provider SDKs or authenticate. Python module
+  import is still ordinary code execution, not a sandbox.
+
+## Presence, metadata and stable identity
+
+Input must be present. Expected must be present by default; nullable schemas may
+accept `None`. Set `expected_policy="optional"` for unlabeled data. A required
+scorer rejects missing labels during preflight. `require_expected()` either returns
+the typed expected value or raises `MissingExpectedError`. `Missing` is encoded as
+an omitted field, while `None` is JSON null.
+
+Absent/null metadata becomes absent. Supplied metadata must validate and serialize
+to an object. Task metadata is shallow-merged onto the JSON projection of dataset
+metadata with task keys winning; the result is hydrated through the metadata type
+again. The artifact retains dataset metadata and task metadata separately.
+
+IDs come from mapped IDs or provider record identity. When none exists, the fallback
+combines normalized row content and its position. Explicit IDs must be unique after
+normalization. Physical source information remains in provenance and is excluded
+from the logical dataset digest. Row ordering is part of that digest.
+
+Serializers must preserve a meaningful JSON round trip. Nonfinite numbers fail;
+there is no automatic `repr`, pickle, or silent null substitution. Validation may
+execute user validators, so validators and serializers should be pure. Dataset
+validation runs once before trials; output and merged metadata are validated at
+their own boundaries.
+
+## Execution and errors
+
+Blocking entrypoints use plain names: `run`, `preflight`, and `inspect_dataset`.
+Async hosts use `await arun`, `await apreflight`, and `await ainspect_dataset`.
+The run pair returns `RunResult(manifest, cases, output_dir, exit_code)`. Setup
+failures raise typed `ConfigurationError`/`DatasetError`; when error artifacts are
+successfully saved, the exception has a note pointing to its report. A dataset-snapshot
+write failure returns exit 1 and saves an artifact-phase failure where the remaining
+evidence files are writable. This also applies to filesystem initialization and
+terminal manifest/report failures: computed cases remain in the returned result,
+and persisted files may be incomplete or stale. Configuration/dataset errors and
+cancellation retain their original exception when error persistence also fails;
+secondary failures appear in exception notes. An existing nonempty output directory
+is rejected without writing into it. Uninspectable schema adapters and non-callable scorers
+fail before source access. Library code does not set process exit status. The CLI
+translates results/errors into exit codes.
+
+Configuration precedence is invocation arguments, decorated defaults, then library
+defaults. There is no implicit `.env` loading.
+
+All selected rows are read/validated before tasks start. `ReadLimits` defaults to
+10,000 rows, 64 MiB serialized bytes, 1 MiB per record, and a 60-second source deadline.
+The default maximum is 50,000 row/trial executions. Prefix inspection is an explicit
+selection; caps fail instead of truncating. These are serialized-data limits, not
+an exact heap quota, and output artifact size is not currently capped separately.
+
+Work is admitted lazily to bounded workers. Every trial gets a fresh nested copy;
+every scorer gets an independent copy of pristine input/expected and validated
+output/merged metadata. Scorers run in declaration order. Final API/report ordering
+is row index then one-based trial number; the on-disk case journal records completion
+order. Earlier successful scores remain if a later scorer fails, and the case still
+fails.
+
+Async functions are awaited; synchronous functions run in a dedicated bounded
+thread executor. Trial timeout covers task and scorers. On timeout/cancellation,
+running sync callbacks are drained before releasing their slots; a hanging thread
+cannot be forcibly stopped. Use async callbacks and SDK request deadlines when prompt
+cancellation matters. Source reads and exports also clean up cooperatively.
+Cancellation preserves partial evidence then re-raises `CancelledError`; the CLI
+returns 130. No tasks or scorers are retried automatically.
+
+## Reports and statistics
+
+Every run writes `dataset.jsonl`, `cases.jsonl`, `run.json`, and `report.html`.
+The manifest records input/expected/metadata and output schemas, selection, logical digest, source provenance, options,
+source-module/framework hashes, versions, metrics, failures, and export status.
+It intentionally omits environment dumps. Full evaluated values remain available.
+Code provenance's framework hash covers Python sources throughout the `mic` package.
+The [artifact reference](artifacts.md) specifies field shapes, partial states, and
+machine-readable schemas for the current format.
+Reports embed case data, and exception text and provenance may be sensitive. Read
+[sensitive evidence and persistence failures](reporting.md#sensitive-evidence-and-sharing)
+before sharing reports or enabling remote export.
+
+Scores are finite numbers or `None`; booleans are rejected. Numeric means exclude
+null/unavailable values. Percentiles use TypeScript's nearest-rank convention.
+Numeric, null and unavailable counts are reported for every scorer.
+The CLI gate grammar is `metric >= number` (also `<=`, `==`, `>`, `<`); the argument
+must be shell-quoted. Missing numeric values make a configured gate unevaluable
+and therefore failing. Any execution error fails regardless of the numeric mean.
+
+Optional reporters implement `name`, `async prepare()`, and
+`async report(manifest, cases) -> JsonObject`. Preparation validates configuration;
+reporting runs only after complete local artifacts exist. Reporters receive copies
+so they cannot mutate local evidence. Export failure/cancellation is recorded
+separately and never causes task replay.
+Artifact failures prevent export; failure to save a reporter's outcome prevents
+subsequent reporters from starting. The completed remote write is not undone or
+repeated. In-memory results retain the known outcome when the filesystem cannot.