oloproof 0.1.0a1__tar.gz

oloproof-0.1.0a1/.gitignore

@@ -0,0 +1,22 @@
+.DS_Store
+.idea/
+.venv/
+__pycache__/
+.pytest_cache/
+.ruff_cache/
+.pyright/
+*.py[cod]
+node_modules/
+.next/
+next-env.d.ts
+*.tsbuildinfo
+.oloproof/
+dist/
+build/
+*.egg-info/
+
+# Secrets. Nothing here was ignored before slice G step 04, which is the step that starts telling
+# people to write a client secret into a file (deploy/README.md).
+.env
+.env.*
+deploy/.env

oloproof-0.1.0a1/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        https://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       https://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

oloproof-0.1.0a1/NOTICE

@@ -0,0 +1,21 @@
+Oloproof
+Copyright 2026 the Oloproof authors
+
+This repository is licensed in two parts.
+
+Apache License, Version 2.0 (LICENSE at the repository root) covers
+everything a developer installs or runs to evaluate their own systems:
+
+  python/        the engine, the CLI and the SDK, including the runner
+  packages/      the design tokens and UI primitives
+  schemas/, examples/, tests/, benchmarks/, scripts/, deploy/, docs/
+  and every other file not listed below.
+
+Elastic License 2.0 (apps/web/LICENSE) covers the hosted control plane
+and workbench:
+
+  apps/web/
+
+You may use, modify and self-host apps/web, including inside your
+organisation. You may not provide it to third parties as a hosted or
+managed service. The full terms are in apps/web/LICENSE.

oloproof-0.1.0a1/PKG-INFO

@@ -0,0 +1,270 @@
+Metadata-Version: 2.5
+Name: oloproof
+Version: 0.1.0a1
+Summary: Developer-first AI systems evaluation, diagnostics, and decision evidence.
+License-Expression: Apache-2.0
+License-File: LICENSE
+License-File: NOTICE
+Requires-Python: >=3.11
+Requires-Dist: cryptography>=43
+Requires-Dist: httpx>=0.27
+Requires-Dist: jsonschema>=4.22
+Requires-Dist: numpy>=2.0
+Requires-Dist: pydantic>=2.11
+Requires-Dist: pyyaml>=6.0.1
+Requires-Dist: rich>=13.7
+Requires-Dist: scipy>=1.14
+Requires-Dist: typer>=0.15.4
+Provides-Extra: collect
+Requires-Dist: opentelemetry-proto>=1.34; extra == 'collect'
+Provides-Extra: dev
+Requires-Dist: cryptography>=43; extra == 'dev'
+Requires-Dist: hypothesis>=6.100; extra == 'dev'
+Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.34; extra == 'dev'
+Requires-Dist: opentelemetry-proto>=1.34; extra == 'dev'
+Requires-Dist: opentelemetry-sdk>=1.34; extra == 'dev'
+Requires-Dist: psycopg[binary]>=3.2; extra == 'dev'
+Requires-Dist: pytest>=8.2; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Requires-Dist: scikit-learn>=1.6.1; extra == 'dev'
+Provides-Extra: postgres
+Requires-Dist: psycopg[binary]>=3.2; extra == 'postgres'
+Provides-Extra: worker
+Requires-Dist: cryptography>=43; extra == 'worker'
+Description-Content-Type: text/markdown
+
+# Oloproof
+
+Oloproof is a developer-first AI systems evaluation, diagnostics and decision
+platform. It runs an AI system on a versioned suite of cases, measures it with
+deterministic, custom, judge, retrieval, agent and predictive evaluators, puts an
+interval with the right denominator on every number, compares a candidate with a
+baseline, diagnoses failures by intervening on them, and decides a release as
+`PASS`, `FAIL`, `INSUFFICIENT_EVIDENCE` or `MANUAL_REVIEW` from declared rules. The
+same canonical objects are read in the terminal and in the Next.js workbench.
+
+Oloproof is not a toy eval harness. The intended product direction is credible
+beside DeepEval/Confident AI, Braintrust, LangSmith, Arize/Phoenix, Galileo,
+Langfuse and Opik, while differentiating through statistical rigor, evaluator
+reliability, intervention-driven diagnosis and decision-grade evidence.
+
+## Install For Development
+
+From the repository root:
+
+```bash
+python3 -m venv .venv
+.venv/bin/python -m pip install -e '.[dev]'
+pnpm install --frozen-lockfile
+make check
+pnpm --filter @oloproof/web check
+```
+
+The public Python surface is:
+
+```python
+from oloproof import evaluate, system, current_case
+from oloproof.evaluators import ExactMatch, Contains, JsonSchema, Regex, RubricJudge, evaluator
+```
+
+Everything under `oloproof_core` is engine internals.
+
+## Quick Start
+
+Generate a small project:
+
+```bash
+. .venv/bin/activate
+oloproof init /tmp/oloproof-demo
+cd /tmp/oloproof-demo
+oloproof run
+```
+
+### Evaluators
+
+The `type:` values `oloproof.yaml` accepts:
+
+| | |
+|---|---|
+| deterministic | `exact_match`, `contains`, `regex`, `json_schema` |
+| LLM judge | `rubric_judge`, `groundedness_judge`, `citation_support_judge`; `probability_judge`: a yes/no, choice or score question answered by one token's probabilities; `cascade`: a probability judge first, a second judge only where it is unsure |
+| model | `model_classifier`: a trained classifier or NLI model on a TEI server, scored against a threshold |
+| retrieval | `hit_rate`, `recall`, `mrr`, `ndcg`, `citation_validity` |
+| agent | `agent_max_steps`, `agent_tool_called`, `agent_no_tool_loop`, `agent_tool_sequence`, `agent_no_undeclared_tool`, `agent_constraints_satisfied` |
+| multi-agent | `agent_route`, `agent_tool_permissions`, `agent_max_handoffs` |
+| predictive | `predictive_correct`, `predictive_precision`, `predictive_recall`, `predictive_ranking`, `predictive_brier`, `predictive_log_loss`, `predictive_absolute_error` |
+
+A judge names its `provider` (`anthropic`, `openai` or `openai_compatible`), its `model`, and its
+rubric as `rubric_text` or `rubric_file`. One on a provider that speaks the OpenAI API but is not
+OpenAI (Groq, Together, a local server) also needs `base_url` and `api_key_env`:
+
+```yaml
+evaluators:
+  - type: rubric_judge
+    criterion: answer_correct
+    provider: openai_compatible
+    model: openai/gpt-oss-20b
+    base_url: https://api.groq.com/openai/v1
+    api_key_env: GROQ_API_KEY
+    rubric_text: "PASS if the answer conveys the same fact as the reference."
+```
+
+Naming a field an evaluator does not take is answered with the ones it does.
+
+Re-decide an existing run without rerunning the system or evaluators:
+
+```bash
+oloproof gate RUN_ID --policy release.yaml
+```
+
+Inspect failures and one case:
+
+```bash
+oloproof inspect RUN_ID --failures
+oloproof inspect RUN_ID --case refund_00
+```
+
+Export a bundle and open it in the web workbench:
+
+```bash
+oloproof export RUN_ID
+OLOPROOF_BUNDLE_DIR="$PWD/.oloproof/bundles" pnpm --dir /path/to/Oloproof --filter @oloproof/web dev
+```
+
+The bundle path must be absolute, because pnpm starts the app from `apps/web`. The workbench opens
+on its workspace index; a lone `OLOPROOF_BUNDLE_DIR` appears there as the `local/bundles` project.
+To hold several projects on one machine, point `OLOPROOF_WORKBENCH_DIR` at a directory containing a
+`workbench.json` instead (`docs/API_CONTRACTS.md`).
+
+## Examples
+
+Five runnable projects, none of which needs provider credentials. Each README explains what the
+run shows and why it decides what it decides.
+
+| Example | Shows |
+|---|---|
+| `examples/support_bot/` | the first run: a label accuracy that is too thin to decide, inspecting failures, re-deciding with `gate`, adding an LLM judge |
+| `examples/support_rag/` | RAG evaluation: retrieval and generation evaluators, declared slices, `expected.gold_context`, and `oloproof diagnose` intervening on failed cases |
+| `examples/support_agent/` | agent and tool evaluation: an agent that runs its own loop and records a trajectory with `current_case().agent_trajectory(...)`, a tool reference in each case's `expected.tools` |
+| `examples/triage_agents/` | several agents in one trajectory: each step names its agent, hand-offs, a declared route in each case's `expected.route`, a closed tool-permission map and a hand-off bound |
+| `examples/churn_model/` | predictive classification: the `predictive:` block, three denominators, calibration and ranking |
+
+```bash
+cd examples/support_bot
+../../.venv/bin/python -m oloproof_cli run
+```
+
+The run exits 3. Its keyword-rule bot labels 13 of 18 questions correctly,
+which is too few cases to show label accuracy of at least 0.70, so the release
+is blocked with `INSUFFICIENT_EVIDENCE`.
+
+## Benchmark
+
+The Slice A analysis benchmark stores runs of synthetic precomputed evidence and
+times what `oloproof gate` does with them: loading, denominators, cold and
+cached analysis, the clustered bootstrap and decisions. It excludes system and
+model latency.
+
+```bash
+.venv/bin/python benchmarks/analysis_bench.py --sizes 1000,10000,100000
+```
+
+Recorded results are in `benchmarks/README.md`.
+
+## Engine Rules
+
+The evidence chain is:
+
+```text
+scenarios -> execution -> artifacts -> evaluators -> judgments
+           -> statistical inference -> diagnostics/interventions
+           -> decisions -> evidence/provenance
+           -> LLM explanation/hypothesis/next experiment
+```
+
+Decision states are exactly `PASS`, `FAIL`, `INSUFFICIENT_EVIDENCE` and
+`MANUAL_REVIEW`. Execution states such as `RUN_ERROR` and `CANCELLED` are not
+decision states. Release actions are separate: `ALLOW`, `WARN` or `BLOCK`.
+
+For minimum-threshold interval rules, `PASS` requires the lower bound to meet the
+threshold and `FAIL` requires the upper bound to be below it. Otherwise the result
+is `INSUFFICIENT_EVIDENCE`. Unsupported declared structures fail closed.
+
+## Current Scope
+
+Built:
+
+- Python SDK and CLI for local evaluation; suites, systems and evaluators are versioned and
+  every record is content-addressed.
+- Deterministic, custom (`@evaluator`), LLM-judge, retrieval, agent-trajectory and predictive
+  evaluators, and an evaluator registry that measures a judge against human labels on the answers
+  people judged: agreement, bias and hit rates, each with an interval, and a bar on each before a
+  judge may gate. People label from a file (`oloproof labels export` and `import`), the terminal
+  (`oloproof review`) or, in a hosted workspace, a review queue in the browser; `oloproof evaluators
+  try` measures a draft judge before it is adopted. Probability judges can be calibrated against
+  labels, and a cascade sends only uncertain cases to a stronger judge.
+- Judges measured without labels: `oloproof evaluators probe` reports how often a judge's verdict
+  moves when it is asked again or shown padding, and `oloproof prefer` asks a pairwise judge which
+  of two runs answered better, in both orders, with the judge's position bias beside the result.
+- Systems of several agents: each trajectory step names the agent that took it, a hand-off is a
+  step, and deterministic criteria judge the route against a declared one, tool calls against a
+  closed permission map, and the number of hand-offs against a bound. No failure is attributed to
+  an agent.
+- Binary-rate, bounded-mean and quantile metrics with missingness bounds; paired and
+  cluster-aware analysis where declared.
+- Candidate-versus-baseline comparison with superiority, non-inferiority and equivalence rules.
+- Declared slices with multiplicity control; a slice rule gates only inside a declared family with
+  a correction and a minimum support.
+- Intervention-driven diagnosis of RAG failures (`oloproof diagnose`) and sample-size planning
+  for undecided rules (`oloproof plan`).
+- Streaming progress with live provisional estimates, bounded concurrency for systems and
+  judges (`concurrency:`), and cache reuse across execution, judgment, analysis and gate
+  recomputation.
+- CI gating through exit codes, sign-off, usage and cost reporting.
+- Early stopping (`early_stopping: true` in a policy): a run, or a comparison, stops after a
+  seeded batch once every rule has decided PASS or FAIL. Cases that never ran count at their worst
+  value, so a stop is valid whenever it happens, and the run reports the work it avoided.
+- Prediction-powered inference (`ppi_judge_rate@1`): a judge's pass rate corrected by a random,
+  blind sample of human labels. A sample drawn on the labeller's machine is good-faith; a hosted
+  workspace draws the sample itself and verifies the interval with its own engine.
+- Verified evidence in a hosted workspace: runners an Owner registers sign the runs they execute,
+  an Owner pins the evaluation and the release policy, and the workspace gate decides each system
+  version on its first run of that evaluation, counting independent labellers by account.
+- The Next.js workbench, locally over exported bundles or as a hosted workspace that
+  `oloproof push` sends runs to. The hosted workspace is invitation-only.
+- Hosted review queues: an Owner or Admin assigns reviewers, who give pass/fail verdicts, scores
+  or A/B preferences that do not name the runs, in the browser. Content a project keeps on its own machines reaches the
+  reviewer's browser from `oloproof collect`, never through Oloproof. Scores and preferences are
+  stored and enter no statistic yet.
+- Production traces received locally: `oloproof collect` accepts OpenTelemetry (GenAI and
+  OpenInference conventions) and pushes them through the project's egress policy.
+
+Not built yet: production evaluation (hosted trace ingestion, workspace-drawn traffic samples,
+online scoring and monitoring, turning a trace into a test case; `docs/plans/SLICE_S.md` S06 to
+S10), and the rest of the statistical roadmap in Engine §24: conformal risk control,
+hierarchical dependence, causal intervention analysis, IRT and a method for score and preference
+labels. Distribution is not built either: there is no PyPI release, and the hosted stack has not
+yet run on a real host (`docs/roadmap/INITIAL_COMPETITIVE_RELEASE.md`).
+
+## Project Instructions
+
+Coding agents should start with:
+
+- `AGENTS.md` - shared instructions and source-of-truth map.
+- `AGENT_WORKFLOWS.md` - Claude/Codex division of labor and task prompts.
+- `DOCUMENTATION_STATUS.md` - canonical synchronized doc versions and status.
+- `docs/engine/ENGINE_SPECIFICATION.md` - statistical and decision contract.
+- `docs/plans/` - the implementation plans, one per slice, and their accepted decisions.
+
+The top-level local repository folder is `Oloproof/` with a capital O. Python
+packages, npm package names, CLI commands and environment variables remain
+lowercase or uppercase as documented: `oloproof`, `oloproof_core`,
+`@oloproof/...`, `OLOPROOF_*`.
+
+## License
+
+Oloproof is licensed in two parts. The engine, CLI, SDK and runner (`python/`), the UI packages
+(`packages/`) and everything else a developer installs are under the
+[Apache License 2.0](LICENSE). The hosted control plane and workbench (`apps/web/`) are under the
+[Elastic License 2.0](apps/web/LICENSE): you may use, modify and self-host it, and you may not offer
+it to others as a managed service. [NOTICE](NOTICE) says which covers what.