gigai-core 0.1.0__tar.gz

gigai_core-0.1.0/.gitignore

@@ -0,0 +1,10 @@
+__pycache__/
+*.py[cod]
+.venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+tests/consumer/.mypy_cache/

gigai_core-0.1.0/CHANGELOG.md

@@ -0,0 +1,80 @@
+# Changelog
+
+Each release is one `## <version> - <date>` block; `release.yml` slices the block of the tagged version into the
+GitHub release notes.
+
+## 0.1.0 - 2026-11-13
+
+First release. The library half of GigAI: the evaluation records, graders, statistics and run files that gigai
+0.3.2 kept inside its CLI, moved into a package with one runtime dependency.
+
+### What ships
+
+- `gigai_core.models`: `CoreModel`, the pydantic base of every record (`strict=True`, `frozen=True`,
+  `extra="forbid"`, `allow_inf_nan=False`; keyword-only construction). Files are read with `model_validate_json`
+  / `TypeAdapter.validate_json` and written with `model_dump_json`; a bad line raises an error that names the file
+  and the line (`RunFileError`, `TraceSchemaError`, `LabelFileError`).
+- `gigai_core.canonical`: canonical JSON encoding and sha256 digests (`canonical_json_bytes`,
+  `canonical_json_digest`, `sha256_hex`, `digest_imported_bytes`, `parse_json_bytes`).
+- `gigai_core.providers`: the provider seam. `ModelProvider` (a `Protocol`: `identity`, `capabilities`, `invoke`,
+  `close`), `CallSettings`, `ModelRequest`, `ModelResult`, `Turn`, `ToolCall`, `ToolSpec`, `TokenUsage`,
+  `ProviderIdentity`, `RetryPolicy`, `request_sha256`, `bounded` (a wall-time cap), and `ProviderError` with
+  `AuthenticationRequired`, `UnsupportedRequestField`, `OutputNotJSON`, `CallTimedOut`, `CallCancelled`.
+- `gigai_core.budget`: `Budget` (Protocol), `CallBudget` (`max_usd`, `max_calls`, a price table; `plan` refuses a
+  run that cannot fit before its first call), `Price`, `Reservation`, `budgeted`, `BudgetExceeded`,
+  `CallCapExceeded`.
+- `gigai_core.recorded`: `RecordedProvider`, which answers from stored `ModelSpan`s keyed by
+  `(model, request_sha256)` and never calls out.
+- `gigai_core.trace`: the span trace (`TrialSpan`, `TurnSpan`, `ModelSpan`, `ToolSpan`, `ModelCall`,
+  `ModelExchange`, `CallError`), `TraceSink` with `ListTrace` and `JsonlTrace` (one JSONL file per run, schema
+  `gigai-core-trace:1`), `read_spans`, `TraceSchemaError`.
+- `gigai_core.dataset`: `Dataset`, `DatasetItem`, `DatasetRef`, `SliceRecipe`, `Dataset.of`, `Dataset.from_jsonl`,
+  `items_sha256`, `DatasetDigestMismatch`.
+- `gigai_core.datasets`: ten loaders (`DatasetLoader`) and `load_slice`; six `ci` slices ship in the wheel
+  (`contractnli`, `ifeval`, `jsonschemabench`, `ragtruth`, `tinygsm8k`, `tinymmlu`); `quality`, `rewardbench2`,
+  `judgebench` and `mtbench_human` are download-only (`build_release`). Licences and attributions are in
+  `src/gigai_core/datasets/NOTICES.md`, shipped in the wheel.
+- `gigai_core.sample`, `gigai_core.score`, `gigai_core.records`: `Sample`, `Score`, `ScoreRecord`, `JudgeNote`,
+  `RunRecord`, `Prereg`, `LabelRecord`, `LabelValue`, `JudgeValidationRecord`.
+- `gigai_core.graders`: one `Grader` Protocol (`id`, `version`, `kind`, `grade(sample, expected) -> Score`), `Judge`
+  and `CannotGrade` (for a sample a grader cannot judge), defined in `graders.protocol` and re-exported with every
+  constructor from `gigai_core.graders` (40 names). `builtin`: `last_number`, `answer_letter`, `json_valid`,
+  `schema_valid`, `label_share`, `bool_field`, `ifeval`, `ranks_first`, `verdict_equals`, `constant`,
+  `with_wrong_labels`, `parse_json_text`. `lifted` (the vendored inspect_ai, autoevals and openevals scorers):
+  `includes`, `match`, `pattern`, `model_graded_qa`, `autoevals_judge`, `openevals_judge`. `judge`: `llm_judge`,
+  `pairwise_judge`, the vendored autoevals templates (`load_template`, `render_template`, `swap_blocks`),
+  `JudgeVerdict`. `labels`: `read_labels`, `write_labels`, `labelled`, `LabelFileError`.
+- `gigai_core.run`: `run(dataset, provider, graders, *, settings, budget, ...)` returns a typed `Run`
+  (`values`, `trace`, `ungraded`, `contested`); `save_run` / `open_run` (one JSONL file, schema `gigai-core-run:1`,
+  `RunFileError` with the line number).
+- `gigai_core.stats`: Wilson and cluster-bootstrap intervals, exact McNemar, Wilcoxon signed-rank, Holm and
+  Benjamini-Hochberg corrections, TOST, Cohen's kappa, Krippendorff's alpha, AUC, minimum detectable difference,
+  variance components, pass@k.
+- `gigai_core.comparison`: `compare(a, b, metrics)` pairs two runs by item and reports per-metric differences with
+  intervals and corrected tests (`Comparison`, `MetricComparison`, `ComparisonRefused`).
+- `gigai_core.gate`: `gate(candidate, baseline, rules)` with `MinLowerBound`, `NoWorseThan`, `HeldOutShare`;
+  `GateResult` (fields `status`, `verdicts`, `reason`; properties `passed`, `failed`, `exit_code`).
+- `gigai_core.agreement`: `judge_agreement(run, grader_id, labels)` and `agreement(pairs)` with kappa, swap
+  consistency and the validation thresholds (`MIN_KAPPA`, `MIN_UNITS`, `MIN_SWAP`).
+- `gigai_core.__init__`: the façade, 41 names in `__all__`: `run`, `save_run`, `open_run`, `Run`, `Dataset`,
+  `DatasetItem`, `CallSettings`, `ModelProvider`, `RecordedProvider`, `CallBudget`, `JsonlTrace`, `read_spans`,
+  `compare`, `gate`, `MinLowerBound`, `NoWorseThan`, `cohen_kappa`, `judge_agreement`, `CoreModel`, `read_labels`,
+  `write_labels`, and the 20 grader constructors (the eleven `builtin` ones, the six `lifted` ones, `llm_judge`,
+  `pairwise_judge`, `labelled`); plus `__version__` from the package metadata.
+
+### Not in 0.1.0
+
+- The agent loop, tools and environments (`TurnLimits`, `holds_every_turn`, `state_equals`): 0.2.0.
+- The sqlite ledger, the CLI and the gig layer: they stay in gigai.
+- `RunRecord.prices` is always empty in 0.1.0: the budget exposes no price table for `run` to copy.
+
+### Dependencies and platforms
+
+- Required: `pydantic>=2.10,<3`. Nothing else at import time: a plain `pip install gigai-core` loads none of
+  `click`, `jsonschema`, `pyarrow`, `numpy`, `yaml`, `sqlite3`, `inspect_ai` or `gigai`.
+- Extras: `schema` (`jsonschema>=4.23`, needed by `graders.builtin.schema_valid`), `parquet` (`pyarrow>=18`, needed
+  by `datasets.build_release` for parquet-backed sets), `inspect` (`inspect-ai==0.3.288`,
+  `agent-client-protocol<1`).
+- Python 3.11, 3.12 and 3.13; Linux and macOS.
+- Licence `Apache-2.0 AND MIT`: the package is Apache-2.0; the vendored scorers and templates under
+  `gigai_core._vendor` (autoevals, inspect_ai, openevals) are MIT, each with its own `LICENSE` and `DIFF.md`.

gigai_core-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

gigai_core-0.1.0/PKG-INFO

@@ -0,0 +1,193 @@
+Metadata-Version: 2.5
+Name: gigai-core
+Version: 0.1.0
+Summary: Typed evaluation records, graders, statistics and run files for GigAI and the edgelab books.
+License-Expression: Apache-2.0 AND MIT
+License-File: LICENSE
+License-File: src/gigai_core/_vendor/autoevals/LICENSE
+License-File: src/gigai_core/_vendor/inspect_ai/LICENSE
+License-File: src/gigai_core/_vendor/openevals/LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: pydantic<3,>=2.10
+Provides-Extra: inspect
+Requires-Dist: agent-client-protocol<1; extra == 'inspect'
+Requires-Dist: inspect-ai==0.3.288; extra == 'inspect'
+Provides-Extra: parquet
+Requires-Dist: pyarrow>=18; extra == 'parquet'
+Provides-Extra: schema
+Requires-Dist: jsonschema>=4.23; extra == 'schema'
+Description-Content-Type: text/markdown
+
+# gigai-core
+
+The library half of GigAI: typed evaluation records, a provider seam, a local span trace, graders, paired statistics,
+a release gate and one run file format. Every record is a pydantic model (strict, frozen, `extra="forbid"`), every
+file is JSON Lines read through `model_validate_json` and written through `model_dump_json`, and the only runtime
+dependency is `pydantic`. There is no CLI, no server and no database: `run()` takes a dataset, a provider and graders
+and returns a `Run` value. Import name `gigai_core`.
+
+## Install
+
+```text
+pip install gigai-core                 # pydantic only
+pip install 'gigai-core[schema]'       # + jsonschema, for graders.builtin.schema_valid
+pip install 'gigai-core[parquet]'      # + pyarrow, for datasets.build_release on parquet-backed sets
+pip install 'gigai-core[inspect]'      # + inspect-ai, agent-client-protocol
+```
+
+Python 3.11, 3.12 or 3.13. `schema_valid` without the extra raises
+`ModuleNotFoundError: schema_valid needs the schema extra: pip install 'gigai-core[schema]'`; a parquet build without
+`pyarrow` raises `datasets.ParquetUnavailable`.
+
+## Quickstart
+
+The examples below run in order as one script and need no API key: the provider answers from a list. A real
+provider implements the same four members of `gigai_core.providers.ModelProvider` (`identity`, `capabilities`,
+`invoke`, `close`); gigai binds its adapters to it.
+
+```python
+import threading
+from datetime import UTC, datetime
+
+from gigai_core.providers import Capability, ModelRequest, ModelResult, ProviderIdentity, TokenUsage
+
+NOW = datetime(2026, 1, 1, tzinfo=UTC)
+
+
+class Scripted:
+    """A ModelProvider that answers from a list, in order."""
+
+    identity = ProviderIdentity(provider="scripted", endpoint="memory", adapter="readme")
+    capabilities: frozenset[Capability] = frozenset({"text", "system", "messages"})
+
+    def __init__(self, replies: list[str]) -> None:
+        self._replies = iter(replies)
+
+    def invoke(self, request: ModelRequest, *, cancel: threading.Event | None = None) -> ModelResult:
+        usage = TokenUsage(input_tokens=None, output_tokens=None, total_tokens=None)
+        return ModelResult(text=next(self._replies), finish_reason="stop", resolved_model=request.settings.model,
+                           usage=usage, cost_status="unavailable", cost_usd=None, latency_ms=0,
+                           started_at=NOW, completed_at=NOW)
+
+    def close(self) -> None:
+        pass
+```
+
+Ask one item, grade the answer with the lifted inspect_ai `includes` scorer, and read the run:
+
+```python
+from gigai_core import CallBudget, CallSettings, Dataset, DatasetItem, includes, run
+from gigai_core.budget import Price
+
+dataset = Dataset.of("capitals", (DatasetItem(id="fr", cluster_id="fr", prompt="Capital of France?", expected="Paris"),))
+price = Price(model="some/model", prompt_usd_per_token=1e-6, completion_usd_per_token=1e-6, fetched_at=NOW, source="readme")
+budget = CallBudget(max_usd=0.05, max_calls=20, prices={"some/model": price})
+result = run(dataset, Scripted(["Paris, of course."]), [includes()], settings=CallSettings(model="some/model"), budget=budget)
+print(result.record.spent_usd, result.values("includes"))  # 0.0 {'fr': 1.0}
+```
+
+`budget.plan` runs over every planned request before the first call, so a run that cannot fit its cap never
+starts; a money cap needs a price for the model (nothing is guessed).
+
+### The trace
+
+Every model call (and, in 0.2.0, every tool call) is a span. With `trace_dir` the spans of a run go to one JSONL
+file, `<trace_dir>/<run_id>.jsonl`; without it they stay in memory on the samples.
+
+```python
+from pathlib import Path
+
+settings = CallSettings(model="some/model")
+result = run(dataset, Scripted(["Paris"]), [includes()], settings=settings, budget=CallBudget(max_calls=20), trace_dir=Path("traces"))
+for span in result.trace("fr"):
+    print(span.turn, span.kind, span.started_at)  # 0 trial ..., 0 model_call ...
+assert result.record.trace_path == Path("traces") / f"{result.record.run_id}.jsonl"
+```
+
+### Paired comparison and the gate
+
+Two runs over the same items are compared item by item (per-cluster bootstrap intervals, exact McNemar for 0/1
+metrics, Holm correction across metrics). A gate turns that into a verdict: `MinLowerBound` asks for a floor on the
+candidate's interval, `NoWorseThan` for a bounded regression against the baseline.
+
+```python
+from gigai_core import MinLowerBound, NoWorseThan, compare, gate, open_run, save_run
+
+items = tuple(DatasetItem(id=f"c{n}", cluster_id=f"c{n}", prompt=f"Capital of country {n}?", expected="Paris") for n in range(20))
+twenty = Dataset.of("twenty-capitals", items)
+for name, replies in (("gem", ["Paris"] * 20), ("min", ["Paris"] * 18 + ["Lyon"] * 2)):
+    saved = run(twenty, Scripted(replies), [includes()], settings=settings, budget=CallBudget())
+    save_run(saved, Path(f"{name}.jsonl"), inputs_by_reference=False)
+
+a, b = open_run(Path("gem.jsonl")), open_run(Path("min.jsonl"))
+comparison = compare(a, b, ["includes"])
+verdict = gate(candidate=b, baseline=a, rules=[MinLowerBound(metric="includes", floor=0.6), NoWorseThan(metric="includes", margin=0.05)])
+print(round(comparison.metrics[0].difference, 2), verdict.status, verdict.exit_code)  # -0.1 fail 1
+assert not verdict.passed and verdict.failed[0].rule == NoWorseThan(metric="includes", margin=0.05)
+```
+
+`open_run` raises `RunFileError` with the file and line number on a bad line; `save_run` is the only write path and
+refuses to overwrite.
+
+## Modules
+
+`from gigai_core import ...` is the façade (`gigai_core.__all__`, 41 names): `run`, `save_run`, `open_run`, `Run`,
+`Dataset`, `DatasetItem`, `CallSettings`, `ModelProvider`, `RecordedProvider`, `CallBudget`, `JsonlTrace`,
+`read_spans`, `compare`, `gate`, `MinLowerBound`, `NoWorseThan`, `cohen_kappa`, `judge_agreement`, `CoreModel`,
+`read_labels`, `write_labels` and the 20 grader constructors. Everything else is imported from its module:
+
+- `gigai_core.models`: `CoreModel`, the strict frozen base of every record.
+- `gigai_core.values`: the shared literals (`ParseStatus`, `Outcome`, `Direction`).
+- `gigai_core.canonical`: canonical JSON bytes and sha256 digests.
+- `gigai_core.providers`: the seam (`ModelProvider`, `CallSettings`, `ModelRequest`, `ModelResult`, `ProviderError`).
+- `gigai_core.budget`: `CallBudget`, `Price`, `budgeted`, `BudgetExceeded`.
+- `gigai_core.trace`: the span records, `JsonlTrace`, `ListTrace`, `read_spans`.
+- `gigai_core.dataset`: `Dataset`, `DatasetItem`, `Dataset.of`, `Dataset.from_jsonl`.
+- `gigai_core.datasets`: the ten loaders, `load_slice`, `build_release`, the six shipped slices and `NOTICES.md`.
+- `gigai_core.sample`, `gigai_core.score`, `gigai_core.records`: `Sample`, `Score`, `ScoreRecord`, `RunRecord`, `Prereg`, `LabelRecord`.
+- `gigai_core.recorded`: `RecordedProvider`, replay from stored spans.
+- `gigai_core.graders`: the `Grader` Protocol, `Judge`, `CannotGrade` (defined in `graders.protocol`) and every constructor, re-exported; submodules `builtin`, `lifted`, `judge` (`llm_judge`, `pairwise_judge`), `labels`.
+- `gigai_core.run`: `run`, `Run`, `save_run`, `open_run`.
+- `gigai_core.stats`: intervals, tests, corrections, kappa, alpha, pass@k, power.
+- `gigai_core.comparison`, `gigai_core.gate`, `gigai_core.agreement`: `compare`, `gate`, `judge_agreement`.
+
+## Layering
+
+Imports only go downward, and `tests/test_layers.py` fails the build on a violation: `models`, `values` and
+`canonical` import nothing from the package; `providers` sits above them; `budget`, `trace`, `dataset` and `stats`
+above `providers` (`stats` imports only `models` and `values`); then `sample`, `score`, `records`, `datasets`
+(`dataset` only) and `recorded` (`providers` and `trace` only); then `graders`; then `run`; then `comparison`,
+`agreement` and `gate`. No module imports `gigai`, and nothing imports `click`, `jsonschema`, `pyarrow`, `numpy`,
+`yaml` or `sqlite3` at load time; `jsonschema` and `pyarrow` are imported inside the one function each that needs
+them, behind the `schema` and `parquet` extras.
+
+## Typing
+
+The package ships `py.typed` and passes pyright strict (`typeCheckingMode = "strict"`, Python 3.11) with no `Any`
+on the public surface; sequence fields are tuples, open maps are `Mapping`, timestamps are aware datetimes, and
+models are constructed by keyword.
+
+## Licences and datasets
+
+`Apache-2.0 AND MIT`. The package is Apache-2.0 (`LICENSE`). `gigai_core._vendor` carries scorers and templates
+from inspect_ai, autoevals and openevals under their MIT licences, each with its `LICENSE` and a `DIFF.md` of the
+changes. The six `ci` dataset slices in the wheel (`contractnli`, `ifeval`, `jsonschemabench`, `ragtruth`,
+`tinygsm8k`, `tinymmlu`) and the four download-only sets are listed with their licences and attributions in
+`src/gigai_core/datasets/NOTICES.md`, also shipped in the wheel.
+
+## Not in 0.1.0
+
+- The agent loop, tools, environments and the conversation graders (`TurnLimits`, `holds_every_turn`,
+  `state_equals`): 0.2.0.
+- The sqlite ledger, the CLI, provider adapters and the gig layer: they stay in gigai.
+- Windows is not tested.
+
+See `CHANGELOG.md` for what each release holds.

gigai_core-0.1.0/README.md

@@ -0,0 +1,165 @@
+# gigai-core
+
+The library half of GigAI: typed evaluation records, a provider seam, a local span trace, graders, paired statistics,
+a release gate and one run file format. Every record is a pydantic model (strict, frozen, `extra="forbid"`), every
+file is JSON Lines read through `model_validate_json` and written through `model_dump_json`, and the only runtime
+dependency is `pydantic`. There is no CLI, no server and no database: `run()` takes a dataset, a provider and graders
+and returns a `Run` value. Import name `gigai_core`.
+
+## Install
+
+```text
+pip install gigai-core                 # pydantic only
+pip install 'gigai-core[schema]'       # + jsonschema, for graders.builtin.schema_valid
+pip install 'gigai-core[parquet]'      # + pyarrow, for datasets.build_release on parquet-backed sets
+pip install 'gigai-core[inspect]'      # + inspect-ai, agent-client-protocol
+```
+
+Python 3.11, 3.12 or 3.13. `schema_valid` without the extra raises
+`ModuleNotFoundError: schema_valid needs the schema extra: pip install 'gigai-core[schema]'`; a parquet build without
+`pyarrow` raises `datasets.ParquetUnavailable`.
+
+## Quickstart
+
+The examples below run in order as one script and need no API key: the provider answers from a list. A real
+provider implements the same four members of `gigai_core.providers.ModelProvider` (`identity`, `capabilities`,
+`invoke`, `close`); gigai binds its adapters to it.
+
+```python
+import threading
+from datetime import UTC, datetime
+
+from gigai_core.providers import Capability, ModelRequest, ModelResult, ProviderIdentity, TokenUsage
+
+NOW = datetime(2026, 1, 1, tzinfo=UTC)
+
+
+class Scripted:
+    """A ModelProvider that answers from a list, in order."""
+
+    identity = ProviderIdentity(provider="scripted", endpoint="memory", adapter="readme")
+    capabilities: frozenset[Capability] = frozenset({"text", "system", "messages"})
+
+    def __init__(self, replies: list[str]) -> None:
+        self._replies = iter(replies)
+
+    def invoke(self, request: ModelRequest, *, cancel: threading.Event | None = None) -> ModelResult:
+        usage = TokenUsage(input_tokens=None, output_tokens=None, total_tokens=None)
+        return ModelResult(text=next(self._replies), finish_reason="stop", resolved_model=request.settings.model,
+                           usage=usage, cost_status="unavailable", cost_usd=None, latency_ms=0,
+                           started_at=NOW, completed_at=NOW)
+
+    def close(self) -> None:
+        pass
+```
+
+Ask one item, grade the answer with the lifted inspect_ai `includes` scorer, and read the run:
+
+```python
+from gigai_core import CallBudget, CallSettings, Dataset, DatasetItem, includes, run
+from gigai_core.budget import Price
+
+dataset = Dataset.of("capitals", (DatasetItem(id="fr", cluster_id="fr", prompt="Capital of France?", expected="Paris"),))
+price = Price(model="some/model", prompt_usd_per_token=1e-6, completion_usd_per_token=1e-6, fetched_at=NOW, source="readme")
+budget = CallBudget(max_usd=0.05, max_calls=20, prices={"some/model": price})
+result = run(dataset, Scripted(["Paris, of course."]), [includes()], settings=CallSettings(model="some/model"), budget=budget)
+print(result.record.spent_usd, result.values("includes"))  # 0.0 {'fr': 1.0}
+```
+
+`budget.plan` runs over every planned request before the first call, so a run that cannot fit its cap never
+starts; a money cap needs a price for the model (nothing is guessed).
+
+### The trace
+
+Every model call (and, in 0.2.0, every tool call) is a span. With `trace_dir` the spans of a run go to one JSONL
+file, `<trace_dir>/<run_id>.jsonl`; without it they stay in memory on the samples.
+
+```python
+from pathlib import Path
+
+settings = CallSettings(model="some/model")
+result = run(dataset, Scripted(["Paris"]), [includes()], settings=settings, budget=CallBudget(max_calls=20), trace_dir=Path("traces"))
+for span in result.trace("fr"):
+    print(span.turn, span.kind, span.started_at)  # 0 trial ..., 0 model_call ...
+assert result.record.trace_path == Path("traces") / f"{result.record.run_id}.jsonl"
+```
+
+### Paired comparison and the gate
+
+Two runs over the same items are compared item by item (per-cluster bootstrap intervals, exact McNemar for 0/1
+metrics, Holm correction across metrics). A gate turns that into a verdict: `MinLowerBound` asks for a floor on the
+candidate's interval, `NoWorseThan` for a bounded regression against the baseline.
+
+```python
+from gigai_core import MinLowerBound, NoWorseThan, compare, gate, open_run, save_run
+
+items = tuple(DatasetItem(id=f"c{n}", cluster_id=f"c{n}", prompt=f"Capital of country {n}?", expected="Paris") for n in range(20))
+twenty = Dataset.of("twenty-capitals", items)
+for name, replies in (("gem", ["Paris"] * 20), ("min", ["Paris"] * 18 + ["Lyon"] * 2)):
+    saved = run(twenty, Scripted(replies), [includes()], settings=settings, budget=CallBudget())
+    save_run(saved, Path(f"{name}.jsonl"), inputs_by_reference=False)
+
+a, b = open_run(Path("gem.jsonl")), open_run(Path("min.jsonl"))
+comparison = compare(a, b, ["includes"])
+verdict = gate(candidate=b, baseline=a, rules=[MinLowerBound(metric="includes", floor=0.6), NoWorseThan(metric="includes", margin=0.05)])
+print(round(comparison.metrics[0].difference, 2), verdict.status, verdict.exit_code)  # -0.1 fail 1
+assert not verdict.passed and verdict.failed[0].rule == NoWorseThan(metric="includes", margin=0.05)
+```
+
+`open_run` raises `RunFileError` with the file and line number on a bad line; `save_run` is the only write path and
+refuses to overwrite.
+
+## Modules
+
+`from gigai_core import ...` is the façade (`gigai_core.__all__`, 41 names): `run`, `save_run`, `open_run`, `Run`,
+`Dataset`, `DatasetItem`, `CallSettings`, `ModelProvider`, `RecordedProvider`, `CallBudget`, `JsonlTrace`,
+`read_spans`, `compare`, `gate`, `MinLowerBound`, `NoWorseThan`, `cohen_kappa`, `judge_agreement`, `CoreModel`,
+`read_labels`, `write_labels` and the 20 grader constructors. Everything else is imported from its module:
+
+- `gigai_core.models`: `CoreModel`, the strict frozen base of every record.
+- `gigai_core.values`: the shared literals (`ParseStatus`, `Outcome`, `Direction`).
+- `gigai_core.canonical`: canonical JSON bytes and sha256 digests.
+- `gigai_core.providers`: the seam (`ModelProvider`, `CallSettings`, `ModelRequest`, `ModelResult`, `ProviderError`).
+- `gigai_core.budget`: `CallBudget`, `Price`, `budgeted`, `BudgetExceeded`.
+- `gigai_core.trace`: the span records, `JsonlTrace`, `ListTrace`, `read_spans`.
+- `gigai_core.dataset`: `Dataset`, `DatasetItem`, `Dataset.of`, `Dataset.from_jsonl`.
+- `gigai_core.datasets`: the ten loaders, `load_slice`, `build_release`, the six shipped slices and `NOTICES.md`.
+- `gigai_core.sample`, `gigai_core.score`, `gigai_core.records`: `Sample`, `Score`, `ScoreRecord`, `RunRecord`, `Prereg`, `LabelRecord`.
+- `gigai_core.recorded`: `RecordedProvider`, replay from stored spans.
+- `gigai_core.graders`: the `Grader` Protocol, `Judge`, `CannotGrade` (defined in `graders.protocol`) and every constructor, re-exported; submodules `builtin`, `lifted`, `judge` (`llm_judge`, `pairwise_judge`), `labels`.
+- `gigai_core.run`: `run`, `Run`, `save_run`, `open_run`.
+- `gigai_core.stats`: intervals, tests, corrections, kappa, alpha, pass@k, power.
+- `gigai_core.comparison`, `gigai_core.gate`, `gigai_core.agreement`: `compare`, `gate`, `judge_agreement`.
+
+## Layering
+
+Imports only go downward, and `tests/test_layers.py` fails the build on a violation: `models`, `values` and
+`canonical` import nothing from the package; `providers` sits above them; `budget`, `trace`, `dataset` and `stats`
+above `providers` (`stats` imports only `models` and `values`); then `sample`, `score`, `records`, `datasets`
+(`dataset` only) and `recorded` (`providers` and `trace` only); then `graders`; then `run`; then `comparison`,
+`agreement` and `gate`. No module imports `gigai`, and nothing imports `click`, `jsonschema`, `pyarrow`, `numpy`,
+`yaml` or `sqlite3` at load time; `jsonschema` and `pyarrow` are imported inside the one function each that needs
+them, behind the `schema` and `parquet` extras.
+
+## Typing
+
+The package ships `py.typed` and passes pyright strict (`typeCheckingMode = "strict"`, Python 3.11) with no `Any`
+on the public surface; sequence fields are tuples, open maps are `Mapping`, timestamps are aware datetimes, and
+models are constructed by keyword.
+
+## Licences and datasets
+
+`Apache-2.0 AND MIT`. The package is Apache-2.0 (`LICENSE`). `gigai_core._vendor` carries scorers and templates
+from inspect_ai, autoevals and openevals under their MIT licences, each with its `LICENSE` and a `DIFF.md` of the
+changes. The six `ci` dataset slices in the wheel (`contractnli`, `ifeval`, `jsonschemabench`, `ragtruth`,
+`tinygsm8k`, `tinymmlu`) and the four download-only sets are listed with their licences and attributions in
+`src/gigai_core/datasets/NOTICES.md`, also shipped in the wheel.
+
+## Not in 0.1.0
+
+- The agent loop, tools, environments and the conversation graders (`TurnLimits`, `holds_every_turn`,
+  `state_equals`): 0.2.0.
+- The sqlite ledger, the CLI, provider adapters and the gig layer: they stay in gigai.
+- Windows is not tested.
+
+See `CHANGELOG.md` for what each release holds.