ragops 2.0.0__tar.gz → 2.2.0__tar.gz

{ragops-2.0.0 → ragops-2.2.0}/.github/ISSUE_TEMPLATE/bug.yml

@@ -10,7 +10,7 @@ body:
     id: version
     attributes:
       label: RAGOps version or commit
-      placeholder: v2.0.0 or commit SHA
+      placeholder: v2.1.0 or commit SHA
     validations:
       required: true
   - type: textarea

{ragops-2.0.0 → ragops-2.2.0}/.github/workflows/ragops-gate.yml

@@ -12,7 +12,7 @@ on:
         description: RAGOps Git revision or tag to install
         required: false
         type: string
-        default: "v2.0.0"
+        default: "v2.1.0"
       scenario:
         description: Caller-relative scenario JSON path
         required: true

{ragops-2.0.0 → ragops-2.2.0}/CHANGELOG.md

@@ -4,6 +4,25 @@ All notable changes follow Keep a Changelog. This project uses Semantic Versioni
 
 ## [Unreleased]
 
+## [2.2.0] - 2026-07-13
+
+### Added
+
+- Quota-independent local release fallback with full offline validation,
+  build-once artifacts, reproducible SBOM, checksums, GitHub CLI publication,
+  and fail-closed optional PyPI token promotion.
+
+## [2.1.0] - 2026-07-13
+
+### Added
+
+- Portable external metric envelope and dependency-free evaluator bridge for
+  per-case scores exported by Ragas, DeepEval, Langfuse, or internal judges.
+- CLI support for external metrics in both absolute evaluation and baseline
+  comparison, with exact coverage/provider/metric compatibility checks.
+- Copyable downstream GitHub PR-comment publisher recipe with a commit-pinned
+  trusted publisher checkout and explicit workflow-name allowlist.
+
 ## [2.0.0] - 2026-07-13
 
 ### Added

ragops-2.2.0/NEXT_SESSION.md

@@ -0,0 +1,195 @@
+# RAGOps — handoff for 2026-07-14
+
+> Superseded on 2026-07-13 by the staged completion run. RAGOps v1.8.0 and
+> v2.0.0 and v2.1.0 were released and verified on GitHub and PyPI. The live BLOCK
+> publisher path was proven through PR #7, including an idempotent update of the
+> same comment. v2.0.0 adds first-class evaluator gates, unified compare
+> semantics, strict numeric validation, fail-closed API authentication and
+> bounds, synthetic benchmark positioning, Product Design remediation,
+> Python 3.13 CI, build-once promotion, CycloneDX SBOM, checksums, provenance,
+> and complete package metadata. v2.1.0 adds portable Ragas/DeepEval/Langfuse/
+> custom metric evidence plus a downstream publisher recipe. Canonical evidence
+> is in `docs/project/v2.1-acceptance.md`.
+>
+> Remaining items below that require real design partners, account-owner
+> settings, credential rotation, hosted infrastructure, or an independent
+> security assessment remain external owner work. They are not incomplete core
+> implementation. Security testing stays deferred by explicit owner request.
+
+This is a local-only working note. Do not treat unchecked items as shipped or
+as customer evidence.
+
+## Current verified state
+
+- Branch: `main`
+- Latest pushed commit: `deeaf7e` (`docs: record live publisher and partner outreach`)
+- Latest public release: `v1.7.0` on GitHub and PyPI
+- v1.8 scope: complete on `main`, not versioned or released
+- Local quality gate: Ruff clean, 99 tests passing, 10 JSON schemas parsing
+- PR publisher live proof: [PR #4](https://github.com/thangldw/ragops/pull/4)
+  created one PASS comment and updated the same comment for a second head SHA;
+  the PR was closed without merge and the temporary branch was deleted.
+- Design-partner recruitment: [Discussion #5](https://github.com/thangldw/ragops/discussions/5)
+  is public. No reviewed real-partner observations or ROI evidence exist yet.
+- Acceptance record: `docs/project/v1.8-acceptance.md`
+
+## P0 — do before a v1.8 release
+
+### 1. Verify the live BLOCK path
+
+- [ ] Create a temporary PR that changes the reusable smoke candidate from the
+  passing baseline to `benchmark-regressed.json`.
+- [ ] Confirm the release-gate workflow fails with exit `2`.
+- [ ] Confirm the publisher still creates a `BLOCK` comment containing the
+  canonical workflow link and correct head SHA.
+- [ ] Push a second commit and confirm the same bot comment is updated rather
+  than duplicated.
+- [ ] Close without merge and delete the temporary branch.
+- [ ] Record the PR/run/comment links in `docs/project/v1.8-acceptance.md`.
+
+Why: PASS creation/update is proven live; BLOCK publication is covered by unit
+tests but has not yet been exercised end to end on GitHub.
+
+### 2. Decide whether to release v1.8.0
+
+- [ ] Owner explicitly selects or rejects the v1.8.0 release boundary.
+- [ ] If GO, use `skills/ragops-release/SKILL.md`.
+- [ ] Bump `pyproject.toml`, `src/ragops/__init__.py`, workflow examples, tests,
+  README install command, and issue template from 1.7.0 to 1.8.0.
+- [ ] Move Unreleased changelog entries into `1.8.0` and write
+  `docs/releases/v1.8.0.md`.
+- [ ] Run Ruff, full tests on available Python versions, PASS/BLOCK fixtures,
+  pilot CLI, API health/evaluate, all schemas, build, wheel/sdist clean install,
+  and version-consistency checks.
+- [ ] Tag and publish GitHub Release only after owner GO.
+- [ ] Trigger PyPI Trusted Publishing for `v1.8.0`, then verify public wheel,
+  source distribution, clean install, and pilot command.
+- [ ] Update v1.8 acceptance and launch checklist with public artifact evidence.
+
+Important: the package metadata on `main` still says `1.7.0`; do not publish a
+build from `main` until the release version is intentionally bumped.
+
+## P0 — real design-partner work
+
+### 3. Qualify the first design partner
+
+- [ ] Review replies to Discussion #5; do not infer interest from views or stars.
+- [ ] Select a team with an existing RAG/agent pilot, recurring regression pain,
+  a technical owner, and a workflow/business owner.
+- [ ] Hold the 60-minute discovery using
+  `docs/gtm/design-partner-outreach.md`.
+- [ ] Agree on one workflow, eligible cohort, baseline/pilot periods, exclusions,
+  primary targets, critical stop conditions, and decision owner.
+- [ ] Obtain explicit consent for pseudonymous workflow measurement.
+- [ ] Start with synthetic/redacted traces. Do not collect raw customer
+  documents, prompts, names, email addresses, credentials, or secrets.
+
+### 4. Freeze and run the pilot contract
+
+- [ ] Copy the synthetic manifest/observations/economics fixtures into a private,
+  partner-reviewed workspace; do not commit partner data to the public repo.
+- [ ] Replace synthetic identifiers with pseudonymous IDs.
+- [ ] Collect at least two weeks or 50 eligible baseline tasks when feasible.
+- [ ] Freeze targets before pilot exposure.
+- [ ] Collect pilot observations using the same task definition and exclusions.
+- [ ] Run `ragops pilot-report` without economics first.
+- [ ] Add economics only after the partner reviews hourly cost and pilot
+  investment assumptions.
+- [ ] Review SCALE/HOLD/REDESIGN jointly and document owner sign-off.
+- [ ] Publish no adoption or ROI percentage until cohort, period, denominator,
+  exclusions, assumptions, consent, and partner approval are documented.
+
+Runbook: `docs/gtm/design-partner-pilot-runbook.md`
+
+## P1 — repository and adoption cleanup
+
+### 5. Documentation consistency
+
+- [ ] Update `docs/demo/cv-linkedin-copy.md`, which still says 50 automated
+  tests; use 99 only if the statement is meant to describe current `main`.
+- [ ] Review historical v1.7 wording that says PR comments are design-only.
+  Preserve release history, but link readers to the v1.8 implementation where
+  current-state wording could be ambiguous.
+- [ ] Update `docs/project/program-completion-audit.md` after v1.8 is released or
+  a real pilot produces evidence.
+- [ ] Recheck README, Pages, portfolio, screenshots, and social preview after
+  the next version number/test count changes.
+
+### 6. GitHub owner settings
+
+- [ ] Upload `docs/demo/social-preview.png` as the repository social preview if
+  it is still not configured.
+- [ ] Configure branch protection and required checks for `main` under the
+  owner's collaboration policy.
+- [ ] Decide whether the PR comment publisher should be required, optional, or
+  limited to selected paths.
+- [ ] Confirm workflow retention is long enough for audit evidence and short
+  enough for the intended data-retention policy.
+
+### 7. Publisher adopter experience
+
+- [ ] Add a copyable downstream-repository publisher recipe; the current
+  workflow is configured for `RAGOps reusable gate smoke` in this repository.
+- [ ] Document the exact workflow-name allowlist change required by adopters.
+- [ ] Exercise a fork PR. Expected behavior is safe publication when GitHub
+  supplies exactly one PR association, otherwise fail closed without a comment.
+- [ ] Add live or fixture coverage for expired artifacts, more than 100
+  artifacts, more than 1,000 comments, and GitHub API rate-limit responses.
+- [ ] Decide whether artifact/comment retention and deletion need an ADR.
+
+## P1 — pilot measurement improvements after first feedback
+
+- [ ] Confirm whether repeat usage across two ISO weeks is meaningful for the
+  partner's workflow cadence.
+- [ ] Decide whether incomplete tasks belong in success and duration
+  denominators; current success/duration metrics use completed tasks.
+- [ ] Add explicit exclusion reasons if a real pilot needs auditable exclusions.
+- [ ] Consider paired-task or comparison-cohort analysis before making causal
+  claims. Current before/after reporting is observational.
+- [ ] Add uncertainty intervals only when sample size and study design justify
+  them; do not create statistical decoration for a tiny pilot.
+- [ ] Validate willingness-to-pay separately from estimated labor savings.
+
+## P2 — not immediate source-code work
+
+- [ ] Hosted 2.0 requires cloud, SSO/RBAC, encryption, tenant isolation,
+  billing, support, incident response, legal, and operations decisions.
+- [ ] Independent security review is required before processing customer-
+  sensitive traces. The previously requested Codex Security scan was explicitly
+  skipped and should remain deferred unless the owner reauthorizes it.
+- [ ] Provider-backed judges, plugin sandboxing, signed baselines, and production
+  data controls remain future architecture work, not v1.8 release blockers.
+
+## Account hygiene
+
+- [ ] Rotate the PyPI password because it was shared in conversation history.
+- [ ] Regenerate PyPI recovery codes because they were also exposed in the
+  working context; store the replacement set only in a password manager.
+- [ ] Confirm 2FA and the `thangldw/ragops` Trusted Publisher still work after
+  rotation. Do not add a long-lived PyPI token to GitHub secrets.
+
+## Fast start commands
+
+```bash
+cd /Users/thang/Documents/ragops
+git status --short
+git pull --ff-only
+ruff check .
+PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 pytest -q
+
+PYTHONPATH=src python -m ragops.cli pilot-report \
+  --manifest docs/gtm/pilot-fixtures/synthetic-manifest.json \
+  --observations docs/gtm/pilot-fixtures/synthetic-observations.jsonl \
+  --economics docs/gtm/pilot-fixtures/synthetic-economics.json \
+  --output /tmp/ragops-pilot-report.md
+
+diff -u docs/gtm/synthetic-pilot-report.md /tmp/ragops-pilot-report.md
+```
+
+## Recommended order tomorrow
+
+1. Live BLOCK PR publisher test.
+2. Review Discussion #5 and qualify any real respondent.
+3. Decide v1.8.0 release GO/HOLD.
+4. If GO, run the full release workflow.
+5. Finish documentation/settings cleanup while waiting for partner baseline data.

{ragops-2.0.0 → ragops-2.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: ragops
-Version: 2.0.0
+Version: 2.2.0
 Summary: Evaluation and red-team release gates for RAG and agent systems
 Project-URL: Homepage, https://github.com/thangldw/ragops
 Project-URL: Documentation, https://github.com/thangldw/ragops#readme
@@ -65,6 +65,8 @@ optional.
 - Exports JSON, Markdown, and standalone HTML evidence for review and CI.
 - Supports Python, CLI, an optional FastAPI adapter, and portable JSONL traces.
 - Keeps scenarios, policies, and reports versioned and provider-independent.
+- Gates portable per-case metrics exported by Ragas, DeepEval, Langfuse, or
+  internal judges without adding those frameworks to the core.
 
 ## How the release gate works
 
@@ -92,7 +94,7 @@ ragops demo --output ragops-demo
 For the latest published stable CLI without cloning the repository:
 
 ```bash
-pip install ragops==2.0.0
+pip install ragops==2.1.0
 ragops demo --output ragops-demo
 ```
 
@@ -134,7 +136,7 @@ validate the harness and architecture comparison; they do not establish
 Japanese semantic quality, customer adoption, or production ROI.
 
 The repository currently validates its core, adapters, reference deployment,
-showcase, and demo paths with 126 automated tests.
+showcase, and demo paths with 138 automated tests.
 
 <p align="center">
   <img src="docs/demo/infographics/evidence-stack.svg" alt="RAGOps evidence stack combining quality, safety, operational budgets, and regression comparison into a release decision" width="100%">
@@ -173,6 +175,10 @@ Promote selected evaluator metrics or finding severities into release gates
 with an explicit [evaluation policy](docs/evaluation/evaluator-gates.md). The
 same evaluator and gate options are available on `evaluate` and `compare`.
 
+Already use another evaluator stack? Export per-case results through the
+[portable external metric envelope](docs/engineering/provider-adapters.md#external-evaluator-metrics)
+and gate namespaced scores without installing that framework into RAGOps.
+
 Add a deterministic Unicode code-point budget when response length matters:
 
 ```bash

{ragops-2.0.0 → ragops-2.2.0}/README.md

@@ -34,6 +34,8 @@ optional.
 - Exports JSON, Markdown, and standalone HTML evidence for review and CI.
 - Supports Python, CLI, an optional FastAPI adapter, and portable JSONL traces.
 - Keeps scenarios, policies, and reports versioned and provider-independent.
+- Gates portable per-case metrics exported by Ragas, DeepEval, Langfuse, or
+  internal judges without adding those frameworks to the core.
 
 ## How the release gate works
 
@@ -61,7 +63,7 @@ ragops demo --output ragops-demo
 For the latest published stable CLI without cloning the repository:
 
 ```bash
-pip install ragops==2.0.0
+pip install ragops==2.1.0
 ragops demo --output ragops-demo
 ```
 
@@ -103,7 +105,7 @@ validate the harness and architecture comparison; they do not establish
 Japanese semantic quality, customer adoption, or production ROI.
 
 The repository currently validates its core, adapters, reference deployment,
-showcase, and demo paths with 126 automated tests.
+showcase, and demo paths with 138 automated tests.
 
 <p align="center">
   <img src="docs/demo/infographics/evidence-stack.svg" alt="RAGOps evidence stack combining quality, safety, operational budgets, and regression comparison into a release decision" width="100%">
@@ -142,6 +144,10 @@ Promote selected evaluator metrics or finding severities into release gates
 with an explicit [evaluation policy](docs/evaluation/evaluator-gates.md). The
 same evaluator and gate options are available on `evaluate` and `compare`.
 
+Already use another evaluator stack? Export per-case results through the
+[portable external metric envelope](docs/engineering/provider-adapters.md#external-evaluator-metrics)
+and gate namespaced scores without installing that framework into RAGOps.
+
 Add a deterministic Unicode code-point budget when response length matters:
 
 ```bash

ragops-2.2.0/docs/architecture/adr/0018-portable-external-metric-envelope.md

@@ -0,0 +1,43 @@
+# ADR 0018: Portable external metric envelope
+
+- Status: Accepted for implementation
+- Date: 2026-07-13
+- Owner authorization: continued staged implementation and release requested by
+  Thang on 2026-07-13
+
+## Context
+
+RAGOps should compete as a portable release-policy and evidence layer rather
+than by recreating every evaluator in Ragas, DeepEval, Langfuse, or private
+model-quality stacks. Native export formats are dependency-specific and change
+outside RAGOps release control.
+
+## Decision
+
+Define one versioned JSON envelope containing a provider identifier and one
+finite numeric metric map per scenario case. A dependency-free adapter exposes
+that envelope through the existing `CaseEvaluator` protocol. Aggregate metric
+keys are namespaced as `<provider>.<metric>` and therefore work with the v2.0
+evaluation policy without special gate logic.
+
+Evaluate requires exact scenario coverage. Compare uses distinct baseline and
+candidate envelopes, requires the same provider and per-case metric sets, and
+then reports ordinary aggregate deltas. RAGOps never changes metric direction,
+scale, calibration, or meaning; the policy author owns those decisions.
+
+Framework integrations should export into this stable envelope outside the
+core. Native format parsers may be optional adapters in the future, but cannot
+silently redefine the canonical envelope.
+
+## Compatibility impact
+
+The envelope, adapter, and CLI options are additive. Omitted external metrics
+preserve existing reports and decisions. Report schema already permits numeric
+metric and delta keys.
+
+## Consequences
+
+- Teams can reuse existing evaluator investments while centralizing release
+  governance in RAGOps.
+- Framework upgrades do not force changes to the dependency-free core.
+- Metric comparability and calibration remain explicit owner responsibilities.

ragops-2.2.0/docs/architecture/adr/0019-local-release-fallback.md

@@ -0,0 +1,22 @@
+# ADR 0019: Local release fallback
+
+Status: Accepted for owner review
+
+## Decision
+
+Provide `scripts/local_release.py` with three explicit phases: `verify`,
+`publish-github`, and `publish-pypi`. Verification produces immutable package
+artifacts, an SBOM, checksums, and local evidence. GitHub publication requires
+`--yes`; PyPI additionally requires a project-scoped token supplied only via an
+environment variable and promotes the artifacts downloaded back from GitHub.
+
+Workflows may be disabled through the GitHub UI or `gh workflow disable` while
+quota is exhausted. They remain versioned so the owner can restore the preferred
+OIDC/provenance path later.
+
+## Consequences
+
+The fallback avoids Actions minutes and keeps release gates observable. Local
+PyPI uploads do not receive GitHub Trusted Publisher identity or GitHub build
+provenance, so `LOCAL_RELEASE_EVIDENCE.json` records `trusted_publishing: false`.
+The owner must protect and rotate the temporary project token.

{ragops-2.0.0 → ragops-2.2.0}/docs/demo/cv-linkedin-copy.md

@@ -15,7 +15,7 @@ not customer production adoption.
   candidate with 25-point citation coverage/precision regressions. The fixture
   validates harness behavior, not Japanese semantic quality.
 - Published architecture decisions, threat model, rollout plan, executive
-  go/no-go recommendation, reproducible reports, and 126 automated tests.
+  go/no-go recommendation, reproducible reports, and 138 automated tests.
 
 ## LinkedIn project description
 

{ragops-2.0.0 → ragops-2.2.0}/docs/engineering/github-pr-gate.md

@@ -22,7 +22,7 @@ jobs:
   release-gate:
     uses: thangldw/ragops/.github/workflows/ragops-gate.yml@main
     with:
-      ragops-version: v2.0.0
+      ragops-version: v2.1.0
       scenario: evals/scenario.json
       baseline: evals/baseline.json
       candidate: evals/candidate.json
@@ -82,3 +82,30 @@ The publisher has `actions: read`, `contents: read`, and
 Copying the publisher to another repository requires changing
 `RAGOPS_SOURCE_WORKFLOW` and the `workflow_run.workflows` allowlist to the exact
 caller workflow name.
+
+For a downstream repository, copy
+[`docs/examples/github-pr-comment.yml`](../examples/github-pr-comment.yml) to
+`.github/workflows/ragops-pr-comment.yml`. The example assumes the read-only
+caller workflow has the exact top-level name `RAGOps`, matching the caller
+recipe above. If it has another name, change both of these values to the same
+exact string:
+
+```yaml
+on:
+  workflow_run:
+    workflows: ["Your exact caller workflow name"]
+
+env:
+  RAGOPS_SOURCE_WORKFLOW: Your exact caller workflow name
+```
+
+The recipe checks out publisher code from a reviewed RAGOps commit into a
+separate path. It does not checkout the pull-request head, restore its cache,
+execute its artifacts, or use `pull_request_target`. Keep that commit pin under
+dependency review when upgrading.
+
+GitHub must associate the source run with exactly one pull request. A fork PR
+is publishable when GitHub supplies that single association; otherwise the
+publisher fails closed without writing a comment. The source workflow remains
+the required branch-protection check. The publisher only improves visibility
+and never converts a blocked gate into a passing check.

{ragops-2.0.0 → ragops-2.2.0}/docs/engineering/gitlab-ci-gate.md

@@ -19,7 +19,7 @@ Set these repository-relative variables:
 
 ```yaml
 variables:
-  RAGOPS_VERSION: "2.0.0"
+  RAGOPS_VERSION: "2.1.0"
   RAGOPS_SCENARIO: "evaluation/scenario.json"
   RAGOPS_BASELINE: "evaluation/baseline.json"
   RAGOPS_CANDIDATE: "evaluation/candidate.json"

ragops-2.2.0/docs/engineering/local-release.md

@@ -0,0 +1,32 @@
+# Release without GitHub Actions
+
+Use this only while GitHub Actions is unavailable. Run from a clean release
+commit with Python 3.11+ and `gh` authenticated.
+
+```bash
+python -m pip install -e '.[dev]' build cyclonedx-bom==7.3.0 twine
+python scripts/local_release.py verify --tag vX.Y.Z
+python scripts/local_release.py publish-github --tag vX.Y.Z --yes
+```
+
+The first command runs all required local gates and writes `dist/` artifacts,
+`SHA256SUMS`, an SBOM, and `LOCAL_RELEASE_EVIDENCE.json`. The second creates and
+pushes the annotated tag, then creates the GitHub Release directly with `gh`.
+
+PyPI cannot use the repository's GitHub OIDC Trusted Publisher from a local
+machine. If PyPI publication cannot wait, create a project-scoped token, expose
+it only for this command, and rotate/revoke it afterwards:
+
+```bash
+read -s PYPI_API_TOKEN && export PYPI_API_TOKEN
+python scripts/local_release.py publish-pypi --tag vX.Y.Z --yes
+unset PYPI_API_TOKEN
+```
+
+The PyPI phase downloads and verifies the GitHub Release artifacts before
+uploading them; it never rebuilds. Do not paste tokens into chat, shell history,
+configuration files, or repository secrets.
+
+Pause quota-consuming workflows with `gh workflow disable <workflow-file>` and
+restore them later with `gh workflow enable <workflow-file>`. Keep manual notes
+of every disabled workflow.

ragops-2.2.0/docs/engineering/provider-adapters.md

@@ -0,0 +1,75 @@
+# Provider adapters
+
+RAGOps core remains provider-neutral and dependency-free. Optional adapters may
+generate application outputs, but evaluation consumes the same portable trace
+contract regardless of provider.
+
+## OpenAI Responses API
+
+`OpenAIResponsesAdapter` sends an explicit model and input to
+`POST /v1/responses`, extracts text output, and records response/model/token
+metadata. The model has no default: production evaluations should pin a chosen
+model or snapshot rather than silently following an alias.
+
+The adapter follows the official Responses API/model guidance:
+https://developers.openai.com/api/docs/models
+
+```python
+from ragops.providers import OpenAIResponsesAdapter
+
+adapter = OpenAIResponsesAdapter(
+    api_key=os.environ["OPENAI_API_KEY"],
+    model=os.environ["RAGOPS_OPENAI_MODEL"],
+)
+result = adapter.generate(
+    instructions="Answer only from approved evidence and cite sources.",
+    input_text=question,
+)
+```
+
+No live provider test runs in public CI. Tests inject a recorded transport so
+contributors do not need credentials and secrets never enter fixtures.
+
+## External evaluator metrics
+
+RAGOps does not require teams to replace Ragas, DeepEval, Langfuse, or an
+internal judge. Export their per-case numeric results into the versioned
+`external-metrics-0.1` envelope:
+
+```json
+{
+  "schema_version": "0.1",
+  "provider": "ragas",
+  "records": [
+    {
+      "case_id": "error-e42",
+      "metrics": {"faithfulness": 0.96, "answer_relevance": 0.94}
+    }
+  ]
+}
+```
+
+RAGOps preserves the supplied scale and meaning. It only validates coverage and
+finite numeric values, namespaces the metrics, aggregates them, and applies the
+reviewed release policy.
+
+```bash
+ragops evaluate \
+  --scenario scenarios/japanese_troubleshooting/scenario.json \
+  --responses scenarios/japanese_troubleshooting/sample_responses.json \
+  --external-metrics scenarios/japanese_troubleshooting/external-metrics-baseline.json \
+  --evaluation-policy scenarios/japanese_troubleshooting/external-metric-policy.toml
+
+ragops compare \
+  --scenario scenarios/japanese_troubleshooting/scenario.json \
+  --baseline scenarios/japanese_troubleshooting/sample_responses.json \
+  --candidate scenarios/japanese_troubleshooting/sample_responses.json \
+  --baseline-external-metrics scenarios/japanese_troubleshooting/external-metrics-baseline.json \
+  --candidate-external-metrics scenarios/japanese_troubleshooting/external-metrics-candidate.json \
+  --evaluation-policy scenarios/japanese_troubleshooting/external-metric-policy.toml
+```
+
+The second command intentionally returns exit code 2 because candidate
+faithfulness is below policy. Native framework export parsing stays outside the
+dependency-free core so vendor upgrades cannot silently change the RAGOps
+contract.

{ragops-2.0.0 → ragops-2.2.0}/docs/engineering/pypi-publishing.md

@@ -9,10 +9,10 @@ PyPI API token and publishes only an existing tag whose `vX.Y.Z` value matches
 
 - Project: [ragops on PyPI](https://pypi.org/project/ragops/)
 - First Trusted Publishing release: `1.6.0`
-- Current verified release: `1.8.0`
+- Current verified release: `2.1.0`
 - GitHub environment: `pypi`
 - Publisher: `thangldw/ragops`, workflow `publish-pypi.yml`
-- Verification: 1.8.0 wheel and sdist present; clean install and
+- Verification: 2.1.0 wheel and sdist present; clean install and
   credential-free proposal-review demo pass
 
 ## One-time owner setup
@@ -32,7 +32,7 @@ GitHub's current OIDC guidance is linked from the
 ## Publish an accepted tag
 
 1. Open **Actions → Publish package to PyPI → Run workflow**.
-2. Enter an existing tag such as `v1.8.0`.
+2. Enter an existing tag such as `v2.1.0`.
 3. Approve the `pypi` environment when protection rules require it.
 4. After success, install in a clean environment and compare the installed
    version and wheel hash with the GitHub Release asset.

ragops-2.2.0/docs/examples/github-pr-comment.yml

@@ -0,0 +1,30 @@
+name: RAGOps PR comment publisher
+
+on:
+  workflow_run:
+    # Must exactly match the top-level name of the read-only caller workflow.
+    workflows: ["RAGOps"]
+    types: [completed]
+
+permissions:
+  actions: read
+  contents: read
+  pull-requests: write
+
+jobs:
+  publish:
+    if: github.event.workflow_run.event == 'pull_request'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout reviewed RAGOps publisher code
+        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+        with:
+          repository: thangldw/ragops
+          ref: e3139cc03ffcf522803e2e3e4ef141f37ee4fbf8 # v2.0.0
+          path: .ragops-publisher
+          persist-credentials: false
+      - name: Validate evidence and publish one idempotent comment
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
+          RAGOPS_SOURCE_WORKFLOW: RAGOps
+        run: python .ragops-publisher/apps/github_pr_comment.py

{ragops-2.0.0 → ragops-2.2.0}/docs/examples/gitlab-ci-ragops.yml

@@ -2,7 +2,7 @@ ragops_release_gate:
   stage: test
   image: python:3.12-slim
   variables:
-    RAGOPS_VERSION: "2.0.0"
+    RAGOPS_VERSION: "2.1.0"
     RAGOPS_SCENARIO: "scenarios/japanese_troubleshooting/benchmark-v0.2.json"
     RAGOPS_BASELINE: "scenarios/japanese_troubleshooting/benchmark-baseline.json"
     RAGOPS_CANDIDATE: "scenarios/japanese_troubleshooting/benchmark-regressed.json"