ragops 2.0.0__tar.gz → 2.1.0__tar.gz

{ragops-2.0.0 → ragops-2.1.0}/.github/ISSUE_TEMPLATE/bug.yml

@@ -10,7 +10,7 @@ body:
     id: version
     attributes:
       label: RAGOps version or commit
-      placeholder: v2.0.0 or commit SHA
+      placeholder: v2.1.0 or commit SHA
     validations:
       required: true
   - type: textarea

{ragops-2.0.0 → ragops-2.1.0}/.github/workflows/ragops-gate.yml

@@ -12,7 +12,7 @@ on:
         description: RAGOps Git revision or tag to install
         required: false
         type: string
-        default: "v2.0.0"
+        default: "v2.1.0"
       scenario:
         description: Caller-relative scenario JSON path
         required: true

{ragops-2.0.0 → ragops-2.1.0}/CHANGELOG.md

@@ -4,6 +4,17 @@ All notable changes follow Keep a Changelog. This project uses Semantic Versioni
 
 ## [Unreleased]
 
+## [2.1.0] - 2026-07-13
+
+### Added
+
+- Portable external metric envelope and dependency-free evaluator bridge for
+  per-case scores exported by Ragas, DeepEval, Langfuse, or internal judges.
+- CLI support for external metrics in both absolute evaluation and baseline
+  comparison, with exact coverage/provider/metric compatibility checks.
+- Copyable downstream GitHub PR-comment publisher recipe with a commit-pinned
+  trusted publisher checkout and explicit workflow-name allowlist.
+
 ## [2.0.0] - 2026-07-13
 
 ### Added

{ragops-2.0.0 → ragops-2.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ragops
-Version: 2.0.0
+Version: 2.1.0
 Summary: Evaluation and red-team release gates for RAG and agent systems
 Project-URL: Homepage, https://github.com/thangldw/ragops
 Project-URL: Documentation, https://github.com/thangldw/ragops#readme
@@ -65,6 +65,8 @@ optional.
 - Exports JSON, Markdown, and standalone HTML evidence for review and CI.
 - Supports Python, CLI, an optional FastAPI adapter, and portable JSONL traces.
 - Keeps scenarios, policies, and reports versioned and provider-independent.
+- Gates portable per-case metrics exported by Ragas, DeepEval, Langfuse, or
+  internal judges without adding those frameworks to the core.
 
 ## How the release gate works
 
@@ -92,7 +94,7 @@ ragops demo --output ragops-demo
 For the latest published stable CLI without cloning the repository:
 
 ```bash
-pip install ragops==2.0.0
+pip install ragops==2.1.0
 ragops demo --output ragops-demo
 ```
 
@@ -134,7 +136,7 @@ validate the harness and architecture comparison; they do not establish
 Japanese semantic quality, customer adoption, or production ROI.
 
 The repository currently validates its core, adapters, reference deployment,
-showcase, and demo paths with 126 automated tests.
+showcase, and demo paths with 138 automated tests.
 
 <p align="center">
   <img src="docs/demo/infographics/evidence-stack.svg" alt="RAGOps evidence stack combining quality, safety, operational budgets, and regression comparison into a release decision" width="100%">
@@ -173,6 +175,10 @@ Promote selected evaluator metrics or finding severities into release gates
 with an explicit [evaluation policy](docs/evaluation/evaluator-gates.md). The
 same evaluator and gate options are available on `evaluate` and `compare`.
 
+Already use another evaluator stack? Export per-case results through the
+[portable external metric envelope](docs/engineering/provider-adapters.md#external-evaluator-metrics)
+and gate namespaced scores without installing that framework into RAGOps.
+
 Add a deterministic Unicode code-point budget when response length matters:
 
 ```bash

{ragops-2.0.0 → ragops-2.1.0}/README.md

@@ -34,6 +34,8 @@ optional.
 - Exports JSON, Markdown, and standalone HTML evidence for review and CI.
 - Supports Python, CLI, an optional FastAPI adapter, and portable JSONL traces.
 - Keeps scenarios, policies, and reports versioned and provider-independent.
+- Gates portable per-case metrics exported by Ragas, DeepEval, Langfuse, or
+  internal judges without adding those frameworks to the core.
 
 ## How the release gate works
 
@@ -61,7 +63,7 @@ ragops demo --output ragops-demo
 For the latest published stable CLI without cloning the repository:
 
 ```bash
-pip install ragops==2.0.0
+pip install ragops==2.1.0
 ragops demo --output ragops-demo
 ```
 
@@ -103,7 +105,7 @@ validate the harness and architecture comparison; they do not establish
 Japanese semantic quality, customer adoption, or production ROI.
 
 The repository currently validates its core, adapters, reference deployment,
-showcase, and demo paths with 126 automated tests.
+showcase, and demo paths with 138 automated tests.
 
 <p align="center">
   <img src="docs/demo/infographics/evidence-stack.svg" alt="RAGOps evidence stack combining quality, safety, operational budgets, and regression comparison into a release decision" width="100%">
@@ -142,6 +144,10 @@ Promote selected evaluator metrics or finding severities into release gates
 with an explicit [evaluation policy](docs/evaluation/evaluator-gates.md). The
 same evaluator and gate options are available on `evaluate` and `compare`.
 
+Already use another evaluator stack? Export per-case results through the
+[portable external metric envelope](docs/engineering/provider-adapters.md#external-evaluator-metrics)
+and gate namespaced scores without installing that framework into RAGOps.
+
 Add a deterministic Unicode code-point budget when response length matters:
 
 ```bash

ragops-2.1.0/docs/architecture/adr/0018-portable-external-metric-envelope.md

@@ -0,0 +1,43 @@
+# ADR 0018: Portable external metric envelope
+
+- Status: Accepted for implementation
+- Date: 2026-07-13
+- Owner authorization: continued staged implementation and release requested by
+  Thang on 2026-07-13
+
+## Context
+
+RAGOps should compete as a portable release-policy and evidence layer rather
+than by recreating every evaluator in Ragas, DeepEval, Langfuse, or private
+model-quality stacks. Native export formats are dependency-specific and change
+outside RAGOps release control.
+
+## Decision
+
+Define one versioned JSON envelope containing a provider identifier and one
+finite numeric metric map per scenario case. A dependency-free adapter exposes
+that envelope through the existing `CaseEvaluator` protocol. Aggregate metric
+keys are namespaced as `<provider>.<metric>` and therefore work with the v2.0
+evaluation policy without special gate logic.
+
+Evaluate requires exact scenario coverage. Compare uses distinct baseline and
+candidate envelopes, requires the same provider and per-case metric sets, and
+then reports ordinary aggregate deltas. RAGOps never changes metric direction,
+scale, calibration, or meaning; the policy author owns those decisions.
+
+Framework integrations should export into this stable envelope outside the
+core. Native format parsers may be optional adapters in the future, but cannot
+silently redefine the canonical envelope.
+
+## Compatibility impact
+
+The envelope, adapter, and CLI options are additive. Omitted external metrics
+preserve existing reports and decisions. Report schema already permits numeric
+metric and delta keys.
+
+## Consequences
+
+- Teams can reuse existing evaluator investments while centralizing release
+  governance in RAGOps.
+- Framework upgrades do not force changes to the dependency-free core.
+- Metric comparability and calibration remain explicit owner responsibilities.

{ragops-2.0.0 → ragops-2.1.0}/docs/demo/cv-linkedin-copy.md

@@ -15,7 +15,7 @@ not customer production adoption.
   candidate with 25-point citation coverage/precision regressions. The fixture
   validates harness behavior, not Japanese semantic quality.
 - Published architecture decisions, threat model, rollout plan, executive
-  go/no-go recommendation, reproducible reports, and 126 automated tests.
+  go/no-go recommendation, reproducible reports, and 138 automated tests.
 
 ## LinkedIn project description
 

{ragops-2.0.0 → ragops-2.1.0}/docs/engineering/github-pr-gate.md

@@ -22,7 +22,7 @@ jobs:
   release-gate:
     uses: thangldw/ragops/.github/workflows/ragops-gate.yml@main
     with:
-      ragops-version: v2.0.0
+      ragops-version: v2.1.0
       scenario: evals/scenario.json
       baseline: evals/baseline.json
       candidate: evals/candidate.json
@@ -82,3 +82,30 @@ The publisher has `actions: read`, `contents: read`, and
 Copying the publisher to another repository requires changing
 `RAGOPS_SOURCE_WORKFLOW` and the `workflow_run.workflows` allowlist to the exact
 caller workflow name.
+
+For a downstream repository, copy
+[`docs/examples/github-pr-comment.yml`](../examples/github-pr-comment.yml) to
+`.github/workflows/ragops-pr-comment.yml`. The example assumes the read-only
+caller workflow has the exact top-level name `RAGOps`, matching the caller
+recipe above. If it has another name, change both of these values to the same
+exact string:
+
+```yaml
+on:
+  workflow_run:
+    workflows: ["Your exact caller workflow name"]
+
+env:
+  RAGOPS_SOURCE_WORKFLOW: Your exact caller workflow name
+```
+
+The recipe checks out publisher code from a reviewed RAGOps commit into a
+separate path. It does not checkout the pull-request head, restore its cache,
+execute its artifacts, or use `pull_request_target`. Keep that commit pin under
+dependency review when upgrading.
+
+GitHub must associate the source run with exactly one pull request. A fork PR
+is publishable when GitHub supplies that single association; otherwise the
+publisher fails closed without writing a comment. The source workflow remains
+the required branch-protection check. The publisher only improves visibility
+and never converts a blocked gate into a passing check.

{ragops-2.0.0 → ragops-2.1.0}/docs/engineering/gitlab-ci-gate.md

@@ -19,7 +19,7 @@ Set these repository-relative variables:
 
 ```yaml
 variables:
-  RAGOPS_VERSION: "2.0.0"
+  RAGOPS_VERSION: "2.1.0"
   RAGOPS_SCENARIO: "evaluation/scenario.json"
   RAGOPS_BASELINE: "evaluation/baseline.json"
   RAGOPS_CANDIDATE: "evaluation/candidate.json"

ragops-2.1.0/docs/engineering/provider-adapters.md

@@ -0,0 +1,75 @@
+# Provider adapters
+
+RAGOps core remains provider-neutral and dependency-free. Optional adapters may
+generate application outputs, but evaluation consumes the same portable trace
+contract regardless of provider.
+
+## OpenAI Responses API
+
+`OpenAIResponsesAdapter` sends an explicit model and input to
+`POST /v1/responses`, extracts text output, and records response/model/token
+metadata. The model has no default: production evaluations should pin a chosen
+model or snapshot rather than silently following an alias.
+
+The adapter follows the official Responses API/model guidance:
+https://developers.openai.com/api/docs/models
+
+```python
+from ragops.providers import OpenAIResponsesAdapter
+
+adapter = OpenAIResponsesAdapter(
+    api_key=os.environ["OPENAI_API_KEY"],
+    model=os.environ["RAGOPS_OPENAI_MODEL"],
+)
+result = adapter.generate(
+    instructions="Answer only from approved evidence and cite sources.",
+    input_text=question,
+)
+```
+
+No live provider test runs in public CI. Tests inject a recorded transport so
+contributors do not need credentials and secrets never enter fixtures.
+
+## External evaluator metrics
+
+RAGOps does not require teams to replace Ragas, DeepEval, Langfuse, or an
+internal judge. Export their per-case numeric results into the versioned
+`external-metrics-0.1` envelope:
+
+```json
+{
+  "schema_version": "0.1",
+  "provider": "ragas",
+  "records": [
+    {
+      "case_id": "error-e42",
+      "metrics": {"faithfulness": 0.96, "answer_relevance": 0.94}
+    }
+  ]
+}
+```
+
+RAGOps preserves the supplied scale and meaning. It only validates coverage and
+finite numeric values, namespaces the metrics, aggregates them, and applies the
+reviewed release policy.
+
+```bash
+ragops evaluate \
+  --scenario scenarios/japanese_troubleshooting/scenario.json \
+  --responses scenarios/japanese_troubleshooting/sample_responses.json \
+  --external-metrics scenarios/japanese_troubleshooting/external-metrics-baseline.json \
+  --evaluation-policy scenarios/japanese_troubleshooting/external-metric-policy.toml
+
+ragops compare \
+  --scenario scenarios/japanese_troubleshooting/scenario.json \
+  --baseline scenarios/japanese_troubleshooting/sample_responses.json \
+  --candidate scenarios/japanese_troubleshooting/sample_responses.json \
+  --baseline-external-metrics scenarios/japanese_troubleshooting/external-metrics-baseline.json \
+  --candidate-external-metrics scenarios/japanese_troubleshooting/external-metrics-candidate.json \
+  --evaluation-policy scenarios/japanese_troubleshooting/external-metric-policy.toml
+```
+
+The second command intentionally returns exit code 2 because candidate
+faithfulness is below policy. Native framework export parsing stays outside the
+dependency-free core so vendor upgrades cannot silently change the RAGOps
+contract.

{ragops-2.0.0 → ragops-2.1.0}/docs/engineering/pypi-publishing.md

@@ -9,10 +9,10 @@ PyPI API token and publishes only an existing tag whose `vX.Y.Z` value matches
 
 - Project: [ragops on PyPI](https://pypi.org/project/ragops/)
 - First Trusted Publishing release: `1.6.0`
-- Current verified release: `1.8.0`
+- Current verified release: `2.0.0`
 - GitHub environment: `pypi`
 - Publisher: `thangldw/ragops`, workflow `publish-pypi.yml`
-- Verification: 1.8.0 wheel and sdist present; clean install and
+- Verification: 2.0.0 wheel and sdist present; clean install and
   credential-free proposal-review demo pass
 
 ## One-time owner setup
@@ -32,7 +32,7 @@ GitHub's current OIDC guidance is linked from the
 ## Publish an accepted tag
 
 1. Open **Actions → Publish package to PyPI → Run workflow**.
-2. Enter an existing tag such as `v1.8.0`.
+2. Enter an existing tag such as `v2.0.0`.
 3. Approve the `pypi` environment when protection rules require it.
 4. After success, install in a clean environment and compare the installed
    version and wheel hash with the GitHub Release asset.

ragops-2.1.0/docs/examples/github-pr-comment.yml

@@ -0,0 +1,30 @@
+name: RAGOps PR comment publisher
+
+on:
+  workflow_run:
+    # Must exactly match the top-level name of the read-only caller workflow.
+    workflows: ["RAGOps"]
+    types: [completed]
+
+permissions:
+  actions: read
+  contents: read
+  pull-requests: write
+
+jobs:
+  publish:
+    if: github.event.workflow_run.event == 'pull_request'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout reviewed RAGOps publisher code
+        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        with:
+          repository: thangldw/ragops
+          ref: e3139cc03ffcf522803e2e3e4ef141f37ee4fbf8 # v2.0.0
+          path: .ragops-publisher
+          persist-credentials: false
+      - name: Validate evidence and publish one idempotent comment
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
+          RAGOPS_SOURCE_WORKFLOW: RAGOps
+        run: python .ragops-publisher/apps/github_pr_comment.py

{ragops-2.0.0 → ragops-2.1.0}/docs/examples/gitlab-ci-ragops.yml

@@ -2,7 +2,7 @@ ragops_release_gate:
   stage: test
   image: python:3.12-slim
   variables:
-    RAGOPS_VERSION: "2.0.0"
+    RAGOPS_VERSION: "2.1.0"
     RAGOPS_SCENARIO: "scenarios/japanese_troubleshooting/benchmark-v0.2.json"
     RAGOPS_BASELINE: "scenarios/japanese_troubleshooting/benchmark-baseline.json"
     RAGOPS_CANDIDATE: "scenarios/japanese_troubleshooting/benchmark-regressed.json"

ragops-2.1.0/docs/product/requirements-v2.1.md

@@ -0,0 +1,50 @@
+# Requirements v2.1 — portable external metric evidence
+
+## User stories
+
+1. As a release owner already using Ragas, DeepEval, Langfuse, or an internal
+   judge, I want to import per-case scores without replacing my evaluator stack.
+2. As a CI owner, I want imported scores to pass through the same deterministic
+   policy and comparison engine as built-in metrics.
+3. As a reviewer, I want provider identity and canonical metric names visible in
+   reports so that the source of a release gate is auditable.
+4. As an adopter, I want a copyable PR publisher recipe whose workflow-name
+   allowlist and trust boundary are explicit.
+
+## Public contract impact
+
+- Add external metric envelope schema 0.1 with provider, per-case metric maps,
+  and exact scenario coverage.
+- Add `--external-metrics` to `evaluate` and repeatable baseline/candidate-shared
+  external metric inputs to `compare`.
+- Aggregate imported metrics under `<provider>.<metric>` and expose them to the
+  existing evaluation policy without changing their meaning.
+- Add a downstream-repository publisher recipe and setup guide. The privileged
+  publisher remains isolated from untrusted PR execution.
+
+## Acceptance criteria
+
+1. Envelope provider is one of `ragas`, `deepeval`, `langfuse`, or `custom` and
+   all case IDs are unique.
+2. Imported metric names are non-empty, values are finite numbers, booleans are
+   rejected, and every scenario case has exactly one metric record.
+3. Evaluate can gate an imported metric using the existing minimum/maximum
+   evaluation policy.
+4. Compare accepts distinct baseline and candidate envelopes from the same
+   provider with identical metric names and reports their deltas.
+5. Provider, coverage, missing cases, extra cases, and metric-set mismatch fail
+   closed with actionable contract errors.
+6. Existing CLI and Python behavior is unchanged when no external metrics are
+   supplied.
+7. The downstream publisher recipe documents the exact source workflow name,
+   permissions, artifact contract, marker behavior, and fork limitation.
+8. Ruff, unit/negative/CLI/workflow tests, and the full Python matrix pass.
+
+## Non-goals
+
+- Importing unstable native vendor export formats directly into the core.
+- Installing Ragas, DeepEval, Langfuse, model SDKs, or network clients.
+- Reinterpreting, normalizing, calibrating, or comparing semantically different
+  third-party metrics.
+- Running provider-backed judges inside the release gate.
+- Claiming external framework endorsement or production adoption.

{ragops-2.0.0 → ragops-2.1.0}/docs/product/roadmap.md

@@ -76,6 +76,14 @@ PR comment integration and notifications remain hosted/integration work.
 - Fail-closed API authentication, bounded input, and explicit local dev mode.
 - Synthetic benchmark positioning and evidence-safe workbench error states.
 
+## 2.1 — portable external evaluator evidence
+
+- Versioned per-case external metric envelope for Ragas, DeepEval, Langfuse,
+  and internal judges.
+- Shared absolute gate semantics and baseline/candidate deltas without provider
+  dependencies in the core.
+- Copyable downstream PR-comment publisher recipe with explicit trust boundary.
+
 ## 3.0 — hosted control plane (local alpha boundary implemented)
 
 - Workspace isolation, SSO/RBAC, audit retention, managed ingestion, and

{ragops-2.0.0 → ragops-2.1.0}/docs/project/program-completion-audit.md

@@ -3,8 +3,8 @@
 ## Outcome
 
 The repository now implements the complete locally executable program defined
-in the FDE blueprint: product strategy, open-source evaluation core, Japanese
-enterprise benchmark, red-team contracts, reference GraphRAG-style deployment,
+in the FDE blueprint: product strategy, open-source evaluation core, synthetic
+Japanese-question release-gate fixture, red-team contracts, reference GraphRAG-style deployment,
 portable observability evidence, team review workflow, control-plane alpha, and
 public presentation assets.
 
@@ -22,13 +22,14 @@ public presentation assets.
 | v1.6 pull-request adoption path | Released | `docs/project/v1.6-acceptance.md` |
 | v1.7 broader adoption proof | Released | `docs/project/v1.7-acceptance.md` |
 | v1.8 review visibility and measured adoption | Released | `docs/project/v1.8-acceptance.md` |
-| v2.0 trustworthy extensible release gates | Release candidate | `docs/project/v2.0-acceptance.md` |
+| v2.0 trustworthy extensible release gates | Released | `docs/project/v2.0-acceptance.md` |
+| v2.1 portable external evaluator evidence | Release candidate | `docs/project/v2.1-acceptance.md` |
 
 ## FDE competency coverage
 
 - **Discover:** ambiguous request, users, constraints, KPI hypothesis, scope.
 - **Build:** ACL-first lexical+graph retrieval and controlled workflow agent.
-- **Evaluate:** 30-case benchmark, baseline/regression/adversarial candidates,
+- **Evaluate:** 30-case synthetic fixture, baseline/regression/adversarial candidates,
   citations, claim support, latency, cost, and critical policy gates.
 - **Deploy:** Docker/API/CLI, CI/nightly workflows, portable traces, rollout plan.
 - **Measure:** experiment history, trends, review decisions, release evidence.
@@ -41,7 +42,9 @@ public presentation assets.
 ## Verification baseline
 
 - Python 3.11+ dependency-free core.
-- Ninety-nine automated tests at the latest acceptance run.
+- 126 automated tests on Python 3.11, 3.12, and 3.13 at the latest acceptance run.
+- GitHub/PyPI distributions are byte-identical; GitHub Release includes a
+  reproducible CycloneDX SBOM, SHA-256 manifest, and provenance attestation.
 - Ruff and diff checks pass.
 - Baseline and reference deployment pass; regressed/adversarial builds return
   expected release-blocking exit codes.
@@ -60,6 +63,6 @@ therefore launch activities rather than missing repository implementation:
 
 ## Final recommendation
 
-Use release 1.8.0 for FDE interviews, OSS feedback, and customer discovery.
+Use release 2.0.0 for FDE interviews, OSS feedback, and customer discovery.
 Proceed to a real offline design-partner pilot; do not market the control-plane
 alpha as production SaaS or claim measured ROI until external evidence exists.

{ragops-2.0.0 → ragops-2.1.0}/docs/project/v2.0-acceptance.md

@@ -2,9 +2,9 @@
 
 ## Decision
 
-**GO for release 2.0.0. The owner authorized staged implementation and release
-on 2026-07-13. Public publication remains conditional on release validation and
-green GitHub CI.**
+**RELEASED as 2.0.0. The owner authorized staged implementation and release on
+2026-07-13; release validation, GitHub CI, GitHub publication, and PyPI
+promotion all passed.**
 
 ## Contract impact
 
@@ -37,6 +37,9 @@ this incompatible correction establishes the 2.0 major boundary.
 | Package discovery | Pass | Project URLs, classifiers, and keywords added to package metadata |
 | Core quality | Pass | Actionlint and Ruff clean; 126 tests pass locally in clean Python 3.12 and 3.13 dev/API environments |
 | Reusable release gate | Pass | [Run 29220368893](https://github.com/thangldw/ragops/actions/runs/29220368893) passes against the release PR head revision |
+| GitHub publication | Pass | [v2.0.0](https://github.com/thangldw/ragops/releases/tag/v2.0.0) includes wheel, source, CycloneDX SBOM, and SHA-256 manifest; [run 29220431136](https://github.com/thangldw/ragops/actions/runs/29220431136) attested them |
+| PyPI promotion | Pass | [Run 29220474540](https://github.com/thangldw/ragops/actions/runs/29220474540) verified release checksums and published the exact wheel/sdist; public hashes match GitHub byte-for-byte |
+| Public install | Pass | Clean Python 3.12 install of `ragops[api]==2.0.0` reports 2.0.0 and runs the proposal-review demo |
 
 ## Limitations
 

ragops-2.1.0/docs/project/v2.1-acceptance.md

@@ -0,0 +1,36 @@
+# v2.1 portable external evaluator evidence acceptance
+
+## Decision
+
+**GO for release 2.1.0. The owner authorized continued staged implementation
+and release on 2026-07-13. Public publication remains conditional on release
+validation and green GitHub CI.**
+
+## Contract impact
+
+The external metric envelope, adapter, evaluator-set comparison options, CLI
+flags, schema, and downstream publisher recipe are additive. Existing defaults
+and metric meanings remain unchanged. Imported metric semantics remain owned by
+the producing framework and reviewed policy.
+
+## Acceptance evidence
+
+| Gate | Result | Evidence |
+| --- | --- | --- |
+| Requirement and ADR | Pass | Requirements v2.1 and ADR 0018 define portable evidence without vendor dependencies |
+| External envelope | Pass | Provider enum, unique case IDs, exact coverage, consistent metric names, and finite numbers enforced |
+| Evaluation gate | Pass | Passing baseline scores aggregate under `ragas.*`; configured minimums remain satisfied and exit 0 |
+| Comparison gate | Pass | Candidate faithfulness 0.825 fails minimum 0.90; compare exits 2 and reports delta -0.145 |
+| Fail-closed negatives | Pass | Missing/extra cases, provider mismatch, metric mismatch, booleans, NaN, infinity, strings, and unknown fields rejected |
+| Core boundary | Pass | No Ragas, DeepEval, Langfuse, model SDK, or runtime dependency added |
+| Downstream publisher | Pass | Actionlint clean; commit-pinned trusted checkout, exact allowlist, least privileges, and no PR-code execution |
+| Core quality | Pass | Ruff clean and 138 tests pass locally; feature PR passed Python 3.11, 3.12, and 3.13 |
+| Release supply chain | Pending | v2.1 tag must prove build-once, SBOM, checksums, provenance, and byte-identical PyPI promotion |
+
+## Limitations
+
+- Native framework export conversion remains an upstream adapter responsibility.
+- RAGOps does not establish cross-provider metric equivalence or calibration.
+- No design-partner response or external adoption evidence exists yet.
+- Repository branch protection remains an explicit owner setting.
+- Security assessment remains deferred by owner instruction.

ragops-2.1.0/docs/releases/v2.1.0.md

@@ -0,0 +1,37 @@
+# RAGOps v2.1.0 — portable external evaluator evidence
+
+RAGOps v2.1.0 lets teams keep Ragas, DeepEval, Langfuse, or internal judges as
+metric producers while using RAGOps as the provider-independent release-policy
+and evidence layer.
+
+## Highlights
+
+- Import a versioned per-case external metric envelope from `ragas`, `deepeval`,
+  `langfuse`, or `custom` sources without adding their SDKs to RAGOps.
+- Validate exact scenario coverage, unique case IDs, consistent metric sets,
+  finite numeric values, and provider identity before evaluation.
+- Gate imported aggregate metrics with the same evaluation-policy minimum and
+  maximum thresholds introduced in v2.0.
+- Compare distinct baseline and candidate envelopes and report namespaced
+  external metric deltas alongside built-in evidence.
+- Copy a downstream PR-comment publisher recipe that checks out reviewed
+  publisher code, uses an exact workflow-name allowlist, and preserves the
+  isolated least-privilege trust boundary.
+
+## Compatibility
+
+This is an additive minor release. Existing scenarios, response fixtures,
+traces, evaluator plugins, API behavior, and omitted-external-metric decisions
+remain unchanged. The external metric schema is new at version 0.1.
+
+RAGOps does not reinterpret imported metric meaning, direction, scale, or
+calibration. Policy owners must confirm that baseline and candidate scores are
+comparable and choose appropriate thresholds.
+
+## Known limits
+
+Native vendor export formats are not parsed in the core. Each framework or
+internal pipeline must map its output to the stable RAGOps envelope. Provider
+SDKs, live judges, calibration, privacy controls, and network availability stay
+outside the dependency-free core. No external adoption is claimed, and security
+assessment remains deferred.

{ragops-2.0.0 → ragops-2.1.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "ragops"
-version = "2.0.0"
+version = "2.1.0"
 description = "Evaluation and red-team release gates for RAG and agent systems"
 readme = "README.md"
 requires-python = ">=3.11"

ragops-2.1.0/scenarios/japanese_troubleshooting/external-metric-policy.toml

@@ -0,0 +1,5 @@
+[metrics."ragas.faithfulness"]
+minimum = 0.90
+
+[metrics."ragas.answer_relevance"]
+minimum = 0.90

ragops-2.1.0/scenarios/japanese_troubleshooting/external-metrics-baseline.json

@@ -0,0 +1,8 @@
+{
+  "schema_version": "0.1",
+  "provider": "ragas",
+  "records": [
+    {"case_id": "error-e42", "metrics": {"faithfulness": 0.96, "answer_relevance": 0.94}},
+    {"case_id": "escalation-e42", "metrics": {"faithfulness": 0.98, "answer_relevance": 0.95}}
+  ]
+}

ragops-2.1.0/scenarios/japanese_troubleshooting/external-metrics-candidate.json

@@ -0,0 +1,8 @@
+{
+  "schema_version": "0.1",
+  "provider": "ragas",
+  "records": [
+    {"case_id": "error-e42", "metrics": {"faithfulness": 0.81, "answer_relevance": 0.92}},
+    {"case_id": "escalation-e42", "metrics": {"faithfulness": 0.84, "answer_relevance": 0.91}}
+  ]
+}

ragops-2.1.0/schemas/external-metrics-0.1.schema.json

@@ -0,0 +1,30 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://ragops.dev/schemas/external-metrics-0.1.schema.json",
+  "title": "RAGOps external metric envelope 0.1",
+  "type": "object",
+  "additionalProperties": false,
+  "required": ["schema_version", "provider", "records"],
+  "properties": {
+    "schema_version": {"const": "0.1"},
+    "provider": {"enum": ["ragas", "deepeval", "langfuse", "custom"]},
+    "records": {
+      "type": "array",
+      "minItems": 1,
+      "items": {
+        "type": "object",
+        "additionalProperties": false,
+        "required": ["case_id", "metrics"],
+        "properties": {
+          "case_id": {"type": "string", "minLength": 1},
+          "metrics": {
+            "type": "object",
+            "minProperties": 1,
+            "propertyNames": {"minLength": 1},
+            "additionalProperties": {"type": "number"}
+          }
+        }
+      }
+    }
+  }
+}