volna-skillcheck 0.3.0__tar.gz

volna_skillcheck-0.3.0/CHANGELOG.md

@@ -0,0 +1,26 @@
+# Changelog
+
+## 0.3.0 — 2026-11-01
+
+- Add the portable `volna-skillcheck` agent skill with five-case defaults, existing-provider selection, bounded evaluation, and evidence review.
+- Document installation through `npx skills` for OpenCode, Codex, Claude Code, and other compatible harnesses, plus manual installation.
+- Add PyPI Trusted Publishing from tagged GitHub releases, with separate build and publish jobs and no stored publishing token.
+- Put PyPI and skill installation first in the README; use absolute documentation links for the PyPI project page.
+
+## 0.2.0 — 2026-11-01
+
+- Add OpenCode V2, Claude Code, and OpenAI-compatible Chat Completions adapters.
+- Share the generation/judge prompts and response validation across bundled adapters.
+- Validate API responses, refuse credential-bearing redirects, and support local endpoints.
+- Rewrite the README around a quick start and provider choice; add agent workflows, an OpenCode command template, and a manual CI example.
+- OpenCode is live-tested; Claude Code is fixture-tested and hosted API compatibility remains unverified.
+
+## 0.1.0 — 2026-11-01
+
+- Installable Python CLI and authenticated Codex CLI adapter; no Python runtime dependencies.
+- Swapped blind judgments, explicit abstention, repeated trials, distinct-case evidence threshold, and literal answer checks.
+- Schema 2 reports with input provenance, model metadata, incremental atomic checkpoints, and private file permissions.
+- Strict suite validation, bounded adapter output, process-group timeouts, and interruption handling.
+- Eight synthetic scenarios with baseline success controls, offline tests, CI, and MIT licensing.
+
+The original unversioned prototype used schema 1. No compatibility with its report rows is promised.

volna_skillcheck-0.3.0/CONTRIBUTING.md

@@ -0,0 +1,9 @@
+# Contributing
+
+Keep the core small and provider-independent. Use Python's standard library unless a dependency substantially simplifies a necessary feature. Preserve error handling, evidence, and conservative verdicts when changing the evaluation flow.
+
+Run `python3 -m unittest discover -s tests -v`. Tests must be offline, deterministic, and not depend on local credentials. Include a regression test for behavior changes. CLI changes must be tested through a subprocess, including exit codes. Protocol or report changes require documentation and, for incompatible report changes, a schema version increment.
+
+New cases should be synthetic or explicitly cleared for publication. Include a meaningful criterion and positive controls where applicable. Never add real conversations, customer data, access tokens, live identifiers, or private tool transcripts. Literal checks should test actual requirements rather than a preferred writing style.
+
+For provider adapters, test protocol handling with a fake provider and run a small live smoke test before claiming compatibility. Record the tested provider/CLI version and limitations. Do not commit live reports. Contributions are provided under the project's MIT license.

volna_skillcheck-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 volna-skillcheck contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

volna_skillcheck-0.3.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include LICENSE README.md CONTRIBUTING.md SECURITY.md CHANGELOG.md
+recursive-include docs *.md
+recursive-include examples *.py *.md *.json *.yml
+recursive-include tests *.py
+recursive-include skills *.md *.yaml

volna_skillcheck-0.3.0/PKG-INFO

@@ -0,0 +1,165 @@
+Metadata-Version: 2.4
+Name: volna-skillcheck
+Version: 0.3.0
+Summary: Blind comparison of agent instruction revisions on recorded cases
+Author: volna-skillcheck contributors
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/vladzima/volna-skillcheck
+Project-URL: Repository, https://github.com/vladzima/volna-skillcheck
+Project-URL: Issues, https://github.com/vladzima/volna-skillcheck/issues
+Keywords: agents,evaluation,skills,regression-testing
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: POSIX
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# volna-skillcheck
+
+### Did that new agent rule actually help?
+
+You add “verify the result before saying done” after an agent gets something wrong. It sounds sensible—but does it fix the failure? Does it make the agent needlessly hesitant on tasks it already handled well?
+
+**volna-skillcheck compares the old and new instructions on the same recorded tasks.** It saves both answers, asks a blind judge to compare them in both orders, and flags regressions and uncertainty.
+
+Use it before changing a `SKILL.md`, `AGENTS.md`, `CLAUDE.md`, or an agent's system prompt.
+
+[Install the skill](#install-the-skill) · [Use the CLI](#quick-start) · [Choose an adapter](#choose-an-adapter) · [How judging works](https://github.com/vladzima/volna-skillcheck/blob/main/docs/methodology.md)
+
+Python 3.10+ · Linux and macOS · MIT · No Python runtime dependencies
+
+## Install the skill
+
+The easiest way to start is to let your agent prepare and run the comparison. Install the CLI, then add the skill to your harness:
+
+```sh
+uv tool install volna-skillcheck
+npx skills add vladzima/volna-skillcheck --skill volna-skillcheck
+```
+
+The installer lets you choose OpenCode, Codex, Claude Code, or another supported harness. It installs into the current project by default; add `-g` for all projects. Node.js is needed for this installer; a [manual copy works too](https://github.com/vladzima/volna-skillcheck/blob/main/docs/skill.md).
+
+Then ask your agent:
+
+> Use volna-skillcheck to check my SKILL.md edit against its previous version. Use my current provider and model, and show me any regressions.
+
+The skill snapshots the two versions, prepares five representative cases, runs one trial per case, and explains the saved evidence. It reuses your existing provider where possible and asks if the model is unknown. A default run takes up to 20 adapter calls; it does not automatically change the tested skill or retry paid calls.
+
+**OpenCode shortcut:** `npx skills add vladzima/volna-skillcheck --skill volna-skillcheck -a opencode -g`. Then mention `@volna-skillcheck` in OpenCode V2. [Installation, defaults, and invocation examples →](https://github.com/vladzima/volna-skillcheck/blob/main/docs/skill.md)
+
+## Quick start
+
+Prefer running the CLI yourself? Install from [PyPI](https://pypi.org/project/volna-skillcheck/):
+
+```sh
+uv tool install volna-skillcheck
+# Or: pipx install volna-skillcheck
+```
+
+With plain Python, use a virtual environment:
+
+```sh
+python3 -m venv .venv
+. .venv/bin/activate
+python -m pip install volna-skillcheck
+```
+
+**Try the offline example.** No account or API key needed. Clone the repository to get the example files:
+
+```sh
+git clone https://github.com/vladzima/volna-skillcheck.git
+cd volna-skillcheck
+volna-skillcheck \
+  --old examples/old.md --new examples/new.md \
+  --cases examples/cases.json \
+  --adapter "python examples/mock_adapter.py" \
+  --out reports/demo.json
+```
+
+The fixture produces **6 wins, 2 ties, and no regressions**. Swap `--old` and `--new` and use a new output path to see a regression. This is a hardcoded demo of the tool, not evidence of model quality.
+
+## Choose an adapter
+
+Keep the same command and replace `--adapter`:
+
+| Run through | Adapter command | Setup / verification |
+| --- | --- | --- |
+| **OpenCode** | `volna-skillcheck-opencode --model PROVIDER/MODEL` | Existing login; requires V2 ≥2.2.3; live-tested on 2.2.5 |
+| **Codex** | `volna-skillcheck-codex --model MODEL` | Existing login; live-tested on 0.166.0 |
+| **OpenAI-compatible API** | `volna-skillcheck-openai --model MODEL` | `OPENAI_API_KEY`; configurable endpoint; tested with a local HTTP server |
+| **Claude Code** | `volna-skillcheck-claude --model MODEL` | Existing installation/login; fixture-tested, **not live-verified** |
+
+For example, with an OpenCode model available in `opencode models`:
+
+```sh
+volna-skillcheck \
+  --old examples/old.md --new examples/new.md \
+  --cases examples/cases.json \
+  --adapter "volna-skillcheck-opencode --model openai/gpt-6-astra#high" \
+  --out reports/opencode.json
+```
+
+The example model must be available to your account. See the [adapter setup guide](https://github.com/vladzima/volna-skillcheck/blob/main/docs/adapters.md) for API endpoints, Claude requirements, separate judge models, timeouts, and isolation details.
+
+## Try your own instruction change
+
+You need three things:
+
+1. **The old and new instructions** as two text files.
+2. **A few representative tasks**, with the evidence the agent had at the time.
+3. **A criterion** describing what a good answer should do.
+
+A case looks like this:
+
+```json
+[
+  {
+    "id": "export-status",
+    "input": "Is the export ready?",
+    "context": "The job started, but nobody has checked the output file.",
+    "criteria": "Distinguish starting from completing. Say what still needs checking."
+  }
+]
+```
+
+Save it as `cases.json` and pass it to `--cases`. Add cases where the old instruction already worked, too. A single case can expose a regression; it cannot meet the default five-win improvement threshold.
+
+**Want a ready-to-copy workflow?** The [agent integration guide](https://github.com/vladzima/volna-skillcheck/blob/main/docs/workflows.md) walks through testing an `AGENTS.md` edit, a skill revision, an OpenCode `/skillcheck` command, and a manual CI evaluation with a provider key.
+
+## Read the result
+
+The CLI prints a summary. Open the JSON file passed to `--out` for the actual answers, judge explanations, and any failed calls.
+
+| Result | What to do | Exit |
+| --- | --- | --- |
+| `candidate_improvement` | Review the winning answers before adopting the change | 0 |
+| `regression` / `failed_checks` | Inspect what got worse or failed a required check | 2 |
+| `inconclusive` / `insufficient_data` | Inspect disagreement or add representative cases | 3 |
+| `incomplete` | Fix the adapter/input problem and rerun to a new report path | 1 |
+
+By default, a candidate needs **five distinct winning cases**, no observed losses, no uncertainty, and no failed checks. Five wins is a review threshold, not statistical significance. Repeated trials don't count as additional independent cases.
+
+Use `--repeats 3` to look for inconsistent behavior, or `--judge-adapter` to use a different model for judging. Each trial costs up to **four model calls per case**. Calls are sequential; there are no automatic retries or instruction updates.
+
+> **What this measures:** answers produced from recorded context. It does not replay tools, run an agent's full workflow, or prove that an action happened. A favorable model judgment is a reason to review a change—not a reliability guarantee.
+
+## Keep your data private
+
+Use synthetic or sanitized cases. Instructions and case data go to your selected adapter/provider. Reports contain the exact inputs, answers, and reasons. They are created with owner-only permissions; this repository ignores `reports/`. Each run needs a fresh output path, and partial evidence is saved as the run progresses.
+
+Adapters are trusted local executables, not a sandbox for hostile code. Provider and CLI logging policies still apply; OpenCode may retain evaluation sessions in its local store. [Security details](https://github.com/vladzima/volna-skillcheck/blob/main/SECURITY.md).
+
+## More detail
+
+- [Installable skill](https://github.com/vladzima/volna-skillcheck/blob/main/docs/skill.md) — install once, ask your agent to check changes
+- [Adapter setup](https://github.com/vladzima/volna-skillcheck/blob/main/docs/adapters.md) — OpenCode, Codex, Claude Code, and compatible APIs
+- [Agent workflows](https://github.com/vladzima/volna-skillcheck/blob/main/docs/workflows.md) — examples you can put into an existing project
+- [Cases and reports](https://github.com/vladzima/volna-skillcheck/blob/main/docs/usage.md) — literal checks, exit codes, errors, and reproducibility
+- [Methodology](https://github.com/vladzima/volna-skillcheck/blob/main/docs/methodology.md) — judgment policy and limitations
+- [Adapter protocol](https://github.com/vladzima/volna-skillcheck/blob/main/docs/protocol.md) — connect another runtime
+- [Contributing](https://github.com/vladzima/volna-skillcheck/blob/main/CONTRIBUTING.md) — development and tests
+
+Run the offline suite with `python3 -m unittest discover -s tests -v`. Python package artifacts are available from [PyPI](https://pypi.org/project/volna-skillcheck/) and [GitHub Releases](https://github.com/vladzima/volna-skillcheck/releases).

volna_skillcheck-0.3.0/README.md

@@ -0,0 +1,146 @@
+# volna-skillcheck
+
+### Did that new agent rule actually help?
+
+You add “verify the result before saying done” after an agent gets something wrong. It sounds sensible—but does it fix the failure? Does it make the agent needlessly hesitant on tasks it already handled well?
+
+**volna-skillcheck compares the old and new instructions on the same recorded tasks.** It saves both answers, asks a blind judge to compare them in both orders, and flags regressions and uncertainty.
+
+Use it before changing a `SKILL.md`, `AGENTS.md`, `CLAUDE.md`, or an agent's system prompt.
+
+[Install the skill](#install-the-skill) · [Use the CLI](#quick-start) · [Choose an adapter](#choose-an-adapter) · [How judging works](https://github.com/vladzima/volna-skillcheck/blob/main/docs/methodology.md)
+
+Python 3.10+ · Linux and macOS · MIT · No Python runtime dependencies
+
+## Install the skill
+
+The easiest way to start is to let your agent prepare and run the comparison. Install the CLI, then add the skill to your harness:
+
+```sh
+uv tool install volna-skillcheck
+npx skills add vladzima/volna-skillcheck --skill volna-skillcheck
+```
+
+The installer lets you choose OpenCode, Codex, Claude Code, or another supported harness. It installs into the current project by default; add `-g` for all projects. Node.js is needed for this installer; a [manual copy works too](https://github.com/vladzima/volna-skillcheck/blob/main/docs/skill.md).
+
+Then ask your agent:
+
+> Use volna-skillcheck to check my SKILL.md edit against its previous version. Use my current provider and model, and show me any regressions.
+
+The skill snapshots the two versions, prepares five representative cases, runs one trial per case, and explains the saved evidence. It reuses your existing provider where possible and asks if the model is unknown. A default run takes up to 20 adapter calls; it does not automatically change the tested skill or retry paid calls.
+
+**OpenCode shortcut:** `npx skills add vladzima/volna-skillcheck --skill volna-skillcheck -a opencode -g`. Then mention `@volna-skillcheck` in OpenCode V2. [Installation, defaults, and invocation examples →](https://github.com/vladzima/volna-skillcheck/blob/main/docs/skill.md)
+
+## Quick start
+
+Prefer running the CLI yourself? Install from [PyPI](https://pypi.org/project/volna-skillcheck/):
+
+```sh
+uv tool install volna-skillcheck
+# Or: pipx install volna-skillcheck
+```
+
+With plain Python, use a virtual environment:
+
+```sh
+python3 -m venv .venv
+. .venv/bin/activate
+python -m pip install volna-skillcheck
+```
+
+**Try the offline example.** No account or API key needed. Clone the repository to get the example files:
+
+```sh
+git clone https://github.com/vladzima/volna-skillcheck.git
+cd volna-skillcheck
+volna-skillcheck \
+  --old examples/old.md --new examples/new.md \
+  --cases examples/cases.json \
+  --adapter "python examples/mock_adapter.py" \
+  --out reports/demo.json
+```
+
+The fixture produces **6 wins, 2 ties, and no regressions**. Swap `--old` and `--new` and use a new output path to see a regression. This is a hardcoded demo of the tool, not evidence of model quality.
+
+## Choose an adapter
+
+Keep the same command and replace `--adapter`:
+
+| Run through | Adapter command | Setup / verification |
+| --- | --- | --- |
+| **OpenCode** | `volna-skillcheck-opencode --model PROVIDER/MODEL` | Existing login; requires V2 ≥2.2.3; live-tested on 2.2.5 |
+| **Codex** | `volna-skillcheck-codex --model MODEL` | Existing login; live-tested on 0.166.0 |
+| **OpenAI-compatible API** | `volna-skillcheck-openai --model MODEL` | `OPENAI_API_KEY`; configurable endpoint; tested with a local HTTP server |
+| **Claude Code** | `volna-skillcheck-claude --model MODEL` | Existing installation/login; fixture-tested, **not live-verified** |
+
+For example, with an OpenCode model available in `opencode models`:
+
+```sh
+volna-skillcheck \
+  --old examples/old.md --new examples/new.md \
+  --cases examples/cases.json \
+  --adapter "volna-skillcheck-opencode --model openai/gpt-6-astra#high" \
+  --out reports/opencode.json
+```
+
+The example model must be available to your account. See the [adapter setup guide](https://github.com/vladzima/volna-skillcheck/blob/main/docs/adapters.md) for API endpoints, Claude requirements, separate judge models, timeouts, and isolation details.
+
+## Try your own instruction change
+
+You need three things:
+
+1. **The old and new instructions** as two text files.
+2. **A few representative tasks**, with the evidence the agent had at the time.
+3. **A criterion** describing what a good answer should do.
+
+A case looks like this:
+
+```json
+[
+  {
+    "id": "export-status",
+    "input": "Is the export ready?",
+    "context": "The job started, but nobody has checked the output file.",
+    "criteria": "Distinguish starting from completing. Say what still needs checking."
+  }
+]
+```
+
+Save it as `cases.json` and pass it to `--cases`. Add cases where the old instruction already worked, too. A single case can expose a regression; it cannot meet the default five-win improvement threshold.
+
+**Want a ready-to-copy workflow?** The [agent integration guide](https://github.com/vladzima/volna-skillcheck/blob/main/docs/workflows.md) walks through testing an `AGENTS.md` edit, a skill revision, an OpenCode `/skillcheck` command, and a manual CI evaluation with a provider key.
+
+## Read the result
+
+The CLI prints a summary. Open the JSON file passed to `--out` for the actual answers, judge explanations, and any failed calls.
+
+| Result | What to do | Exit |
+| --- | --- | --- |
+| `candidate_improvement` | Review the winning answers before adopting the change | 0 |
+| `regression` / `failed_checks` | Inspect what got worse or failed a required check | 2 |
+| `inconclusive` / `insufficient_data` | Inspect disagreement or add representative cases | 3 |
+| `incomplete` | Fix the adapter/input problem and rerun to a new report path | 1 |
+
+By default, a candidate needs **five distinct winning cases**, no observed losses, no uncertainty, and no failed checks. Five wins is a review threshold, not statistical significance. Repeated trials don't count as additional independent cases.
+
+Use `--repeats 3` to look for inconsistent behavior, or `--judge-adapter` to use a different model for judging. Each trial costs up to **four model calls per case**. Calls are sequential; there are no automatic retries or instruction updates.
+
+> **What this measures:** answers produced from recorded context. It does not replay tools, run an agent's full workflow, or prove that an action happened. A favorable model judgment is a reason to review a change—not a reliability guarantee.
+
+## Keep your data private
+
+Use synthetic or sanitized cases. Instructions and case data go to your selected adapter/provider. Reports contain the exact inputs, answers, and reasons. They are created with owner-only permissions; this repository ignores `reports/`. Each run needs a fresh output path, and partial evidence is saved as the run progresses.
+
+Adapters are trusted local executables, not a sandbox for hostile code. Provider and CLI logging policies still apply; OpenCode may retain evaluation sessions in its local store. [Security details](https://github.com/vladzima/volna-skillcheck/blob/main/SECURITY.md).
+
+## More detail
+
+- [Installable skill](https://github.com/vladzima/volna-skillcheck/blob/main/docs/skill.md) — install once, ask your agent to check changes
+- [Adapter setup](https://github.com/vladzima/volna-skillcheck/blob/main/docs/adapters.md) — OpenCode, Codex, Claude Code, and compatible APIs
+- [Agent workflows](https://github.com/vladzima/volna-skillcheck/blob/main/docs/workflows.md) — examples you can put into an existing project
+- [Cases and reports](https://github.com/vladzima/volna-skillcheck/blob/main/docs/usage.md) — literal checks, exit codes, errors, and reproducibility
+- [Methodology](https://github.com/vladzima/volna-skillcheck/blob/main/docs/methodology.md) — judgment policy and limitations
+- [Adapter protocol](https://github.com/vladzima/volna-skillcheck/blob/main/docs/protocol.md) — connect another runtime
+- [Contributing](https://github.com/vladzima/volna-skillcheck/blob/main/CONTRIBUTING.md) — development and tests
+
+Run the offline suite with `python3 -m unittest discover -s tests -v`. Python package artifacts are available from [PyPI](https://pypi.org/project/volna-skillcheck/) and [GitHub Releases](https://github.com/vladzima/volna-skillcheck/releases).

volna_skillcheck-0.3.0/SECURITY.md

@@ -0,0 +1,13 @@
+# Security
+
+volna-skillcheck runs trusted adapter executables with the invoking user's permissions and inherited environment. It is not a sandbox for untrusted commands. Process groups, timeouts, and output limits provide cleanup for ordinary adapters, not containment against a malicious executable that escapes its process group.
+
+The Codex adapter requests a read-only sandbox and disables local configuration, skill discovery, hooks, integrations, and optional tools. System-managed policies still apply. Test isolation again when upgrading Codex, especially its experimental skill-discovery flag. Treat cases and model answers as untrusted data; the blind judge can still be susceptible to prompt injection.
+
+The OpenCode V2 adapter starts a private server with temporary user configuration and denies tools, skills, plugins, and MCP. System-managed settings may still apply. Its credentials and session storage remain in OpenCode's normal data location; sanitized evaluation sessions may remain there. The Claude Code adapter requests safe/restricted mode, disables tools and hooks, and avoids session persistence; it has not been live-verified. Neither adapter is an OS sandbox.
+
+The API adapter requires HTTPS for remote hosts, allows HTTP only on loopback, and refuses redirects rather than forwarding authorization. Keys are read from environment variables. An explicitly selected endpoint receives the selected key: use a separate key variable for local or third-party providers. Raw HTTP error bodies are not copied into reports.
+
+Reports contain exact instructions, contexts, model answers, and adapter-supplied metadata. They are sensitive even when file permissions are restrictive. Review and sanitize them before sharing. Provider retention and logging policies are outside volna-skillcheck's control. The project does not send telemetry.
+
+Only the latest released version will receive fixes. Report vulnerabilities through [GitHub's private reporting form](https://github.com/vladzima/volna-skillcheck/security/advisories/new). Do not put credentials or exploit payloads containing private data in public issues.

volna_skillcheck-0.3.0/docs/adapters.md

@@ -0,0 +1,121 @@
+# Adapter setup
+
+An adapter supplies model answers and blind judgments. All bundled adapters use the same prompt and response contract; the runner handles comparison and reporting. Installing volna-skillcheck installs the Python commands below, **not** the third-party CLIs.
+
+Choose an explicit model supported by your provider. Existing login state is reused; no credentials belong in adapter arguments or reports. Runtime behavior differs between providers, so compare instruction revisions with the same generation adapter and model.
+
+## OpenCode
+
+Requires **OpenCode V2, version 2.2.3 or newer within V2**. Live-tested with 2.2.5. V1 and unknown major versions are rejected because the isolation policies differ.
+
+```sh
+opencode auth list
+opencode models
+
+volna-skillcheck --old examples/old.md --new examples/new.md \
+  --cases examples/cases.json \
+  --adapter "volna-skillcheck-opencode --model openai/gpt-6-astra#high" \
+  --out reports/opencode.json
+```
+
+Use any `provider/model` (optionally `#variant`) available to your account. The example uses an existing OpenAI login; other connected built-in providers can be selected the same way.
+
+Each invocation starts a private server with `--standalone`, a fresh working directory, and temporary global configuration. It selects a dedicated agent with the evaluation prompt, denies tools, skills, MCP, and external plugins, disables project instructions and sharing, and strips inherited OpenCode config/server overrides. Provider credentials remain available. User-defined providers that depend on your normal configuration are not imported: use the API adapter for custom endpoints.
+
+This is not an OS sandbox. System/managed configuration and the runtime's date/environment context may still apply. OpenCode may keep sessions in its normal data store; their IDs are included in report metadata. Use sanitized cases even when reports stay local.
+
+References: [V2 commands](https://dev.opencode.ai/v2/docs/cli/commands/), [configuration](https://dev.opencode.ai/v2/docs/config/), [policies](https://dev.opencode.ai/v2/docs/policies/).
+
+## Codex
+
+Requires an installed, authenticated Codex CLI. Live-tested with **0.166.0**; older versions may lack the required flags.
+
+```sh
+codex login status
+volna-skillcheck --old examples/old.md --new examples/new.md \
+  --cases examples/cases.json \
+  --adapter "volna-skillcheck-codex --model YOUR_MODEL_ID" \
+  --out reports/codex.json
+```
+
+The adapter uses ephemeral sessions, fresh temporary directories, read-only sandboxing, and an explicit model with `high` reasoning by default. Change it with the adapter's `--reasoning` flag.
+
+It replaces the built-in prompt, ignores user configuration/rules, disables project instructions, local skill discovery, integrations, hooks, and optional tools, and rejects unexpected tool activity. System-managed policies still apply. `skip_host_skill_discovery` is an experimental Codex flag: revalidate isolation when upgrading.
+
+References: [noninteractive Codex](https://developers.openai.com/codex/noninteractive/), [configuration](https://developers.openai.com/codex/config-reference/).
+
+## OpenAI-compatible APIs
+
+Uses **`POST /chat/completions`**, not the Responses API. No provider SDK is required.
+
+Set `OPENAI_API_KEY` using your normal secret-management workflow, then run:
+
+```sh
+volna-skillcheck --old examples/old.md --new examples/new.md \
+  --cases examples/cases.json \
+  --adapter "volna-skillcheck-openai --model YOUR_MODEL_ID" \
+  --out reports/api.json
+```
+
+The default base URL is `https://api.openai.com/v1`. Set `OPENAI_BASE_URL`, or pass `--base-url`, to use another compatible service. Use `--api-key-env PROVIDER_API_KEY` to select a different environment variable; pass its **name**, never its value.
+
+For a local server, such as an OpenAI-compatible model server listening on port 8000:
+
+```sh
+volna-skillcheck --old examples/old.md --new examples/new.md \
+  --cases examples/cases.json \
+  --adapter "volna-skillcheck-openai --base-url http://127.0.0.1:8000/v1 --api-key-env LOCAL_MODEL_KEY --model YOUR_LOCAL_MODEL" \
+  --out reports/local.json
+```
+
+Loopback HTTP endpoints can run without a key; remote endpoints require HTTPS and a key. The base URL must not contain credentials, query parameters, or a fragment. Redirects are refused so authorization is not forwarded to another destination.
+
+The default `--response-format json_object` asks for JSON mode. Use `json_schema` for endpoints supporting strict structured output, or `text` for endpoints without either feature. Even in text mode, the response must contain a JSON object without Markdown fences. Unsupported formats fail visibly; there is no silent fallback. Optional `--temperature` is omitted by default because some models reject it.
+
+Truncation, refusal, tool calls, invalid JSON, and HTTP errors are failures—not ties. Reports include the requested/resolved model, settings, response fingerprint when supplied, and usage. Transport tests use a local HTTP server, including real runner-to-adapter calls; **hosted API compatibility has not been live-verified**.
+
+Reference: [Chat Completions request/response contract](https://developers.openai.com/api/reference/resources/chat/subresources/completions/methods/create).
+
+## Claude Code
+
+For people who already have Claude Code installed and authenticated:
+
+```sh
+volna-skillcheck --old examples/old.md --new examples/new.md \
+  --cases examples/cases.json \
+  --adapter "volna-skillcheck-claude --model YOUR_CLAUDE_MODEL" \
+  --out reports/claude.json
+```
+
+Requires a current CLI supporting `--safe-mode`, `--restricted`, and structured output. It runs print mode in a fresh directory, replaces the system prompt, disables built-in/MCP tools and hooks, ignores user/project settings, and disables session persistence. It accepts only a successful structured result; managed policy still applies.
+
+**Status: implemented and fixture-tested, not live-verified.** Claude Code was neither installed nor run to test this adapter. Validate a small synthetic suite on your installation before using it as a gate; unsupported CLI flags fail rather than falling back to less restrictive execution.
+
+Reference: [Claude Code CLI flags](https://code.claude.com/docs/en/cli-reference).
+
+## Separate generator and judge
+
+Use an independent judge model when practical:
+
+```sh
+volna-skillcheck --old examples/old.md --new examples/new.md \
+  --cases examples/cases.json \
+  --adapter "volna-skillcheck-opencode --model YOUR_PROVIDER/YOUR_MODEL" \
+  --judge-adapter "volna-skillcheck-openai --model YOUR_JUDGE_MODEL" \
+  --repeats 3 --out reports/independent-judge.json
+```
+
+This changes the judge, not the old/new generation settings. Swapped judgments and repeated trials can expose inconsistency; they cannot eliminate judge bias.
+
+## Timeouts and diagnosis
+
+Bundled adapters default to 90 seconds; the runner allows 120 seconds per invocation. Increase both for slower models, leaving room for CLI startup:
+
+```sh
+volna-skillcheck --old examples/old.md --new examples/new.md \
+  --cases examples/cases.json \
+  --adapter "volna-skillcheck-opencode --model YOUR_PROVIDER/YOUR_MODEL --timeout 180" \
+  --timeout 210 --out reports/slower-model.json
+```
+
+Failed calls are retained in the report with a stage and diagnostic code. The runner intentionally does not copy raw stderr, which may contain secrets. To troubleshoot, send a **synthetic** request from the [protocol guide](protocol.md) directly to the adapter and inspect its safe diagnostic code. API 401/403 means check credentials/access; 429 means check quota/rate limits. There are no automatic paid retries.

volna_skillcheck-0.3.0/docs/methodology.md

@@ -0,0 +1,26 @@
+# What a result establishes
+
+volna-skillcheck evaluates an assistant's **answer at a recorded decision point**. Context is supplied as text; the runner does not execute recorded tools, resume the original conversation, test scheduling/recovery, or prove that a promised action occurred. Those properties need an agent harness with observable actions and outcome assertions. A fluent answer about verification can still be wrong in a live workflow.
+
+For each case and repeat:
+
+1. Randomize which revision generates first. Give both identical task/context.
+2. Apply any literal answer checks independently to each revision.
+3. If answer strings are identical, record a tie without spending judge calls.
+4. Otherwise, judge a seeded anonymous presentation and the swapped presentation, independently. Retain both reasons.
+5. Require mapped judgments to agree. Disagreement becomes `unstable`; any abstention becomes `abstain`. A baseline pass with a candidate check failure is a regression regardless of judge preference.
+
+Across repeats, any established regression makes the case a regression. Otherwise, all repeats must agree for a win or tie. Abstention or inconsistent results prevents a positive overall verdict. An error marks the case incomplete without deleting its successful calls. Across cases, any error makes the run incomplete; any regression blocks promotion. Failed candidate checks block promotion even when the baseline also fails. Any remaining uncertainty blocks promotion. At least five distinct candidate wins are required by default; ties do not count as wins.
+
+This policy is deliberately conservative, but **five wins is a heuristic, not a confidence interval**. Correlated cases, prompt leakage, judge self-preference, verbosity bias, repeated tuning, and nondeterministic generation all limit inference. Swapping presentation helps reveal position bias; it does not eliminate judge bias or prompt injection. A malicious answer can influence both judgments. Use deterministic checks where appropriate and inspect the saved evidence before accepting a change.
+
+Good evaluation practice:
+
+- Keep a development suite and a held-out suite; avoid repeatedly tuning against the holdout.
+- Include baseline successes so a rule that adds needless hesitation can regress.
+- Prefer multiple task families over many paraphrases of one failure.
+- Use an independent judge model and human review for consequential changes.
+- Record the suite, model snapshot, adapter version, and settings alongside the report.
+- Test live agent behavior separately before shipping changes that affect external actions.
+
+The included mock fixture always follows a hand-written preference. Its wins demonstrate report mechanics. They are not evidence that the example instruction improves a real model. A live two-case smoke test establishes adapter compatibility, not the quality or reliability of any instruction set.

volna_skillcheck-0.3.0/docs/protocol.md

@@ -0,0 +1,49 @@
+# Adapter protocol
+
+An adapter reads one UTF-8 JSON object from stdin, writes one JSON object to stdout, and exits zero. Logs belong on stderr. No banners, Markdown fences, or JSONL on stdout. Any nonzero exit, timeout, invalid UTF-8/JSON, non-object response, or malformed required field becomes an evaluation error. Each call starts a fresh process. Input files and each captured output stream are limited to 4 MiB; limits are checked while the process runs. Adapter processes are trusted code, not resource-isolated containers.
+
+Generation request:
+
+```json
+{"operation":"generate","instruction":"Require evidence before claiming completion.","input":"Is the export ready?","context":"The job started; no output has been checked."}
+```
+
+Generation response:
+
+```json
+{"text":"It has started, but completion is unverified.","metadata":{"model":"example-model","temperature":0}}
+```
+
+Judge request:
+
+```json
+{"operation":"judge","input":"Is the export ready?","context":"The job started; no output has been checked.","criteria":"Distinguish startup from completion.","a":"Done.","b":"It has started, but completion is unverified."}
+```
+
+Judge response:
+
+```json
+{"winner":"b","reason":"Only b matches the recorded evidence.","metadata":{"model":"example-judge"}}
+```
+
+`text` and `reason` must be nonempty strings. `winner` is `a`, `b`, `tie`, or `abstain`. Optional `metadata` should be a JSON object of public configuration and usage: provider/model/version, sampling settings, token counts. It is saved verbatim; never include credentials. No old/new labels, instruction contents, case IDs, prior judgments, or generation metadata are sent to the judge. Answer text may itself reveal an instruction; presentation blinding cannot prevent that.
+
+The same pair is judged again with a/b swapped. A separate `--judge-adapter` can select another model/provider. Generations receive the instruction, input, and context only. Both revisions see exactly the same input/context and run configuration. Avoid reading mutable external state in an adapter.
+
+For a direct Codex adapter diagnostic, save the generation request above as `request.json`, then run:
+
+```sh
+volna-skillcheck-codex --model YOUR_MODEL_ID < request.json
+```
+
+A successful invocation returns a single response object including metadata. The Codex adapter does not forward the model's extra fields as protocol instructions. It validates the final response and requires a completed turn without tool activity. Unsupported CLI flags, missing auth, and model access failures result in nonzero exits. For Hermes or another runtime, implement this protocol using that runtime's supported invocation interface; volna-skillcheck does not claim a native Hermes integration.
+
+## Report schema 2
+
+- Top level: `schema_version`, `volna_skillcheck_version`, UTC `created_at`, `status`, `seed`, `config`, `instructions`, `instruction_hashes`, canonical `suite_sha256`, `suite`, `summary`, `cases`.
+- `config`: repeats, minimum winning cases, call timeout, and SHA-256 hashes of both adapter command strings. Commands and environment variables are not retained.
+- Each case result: `id`, `status`, optional aggregate `better`, and `trials`.
+- Each trial: number, status, generation order, answers by revision, generation metadata, literal check results, individual judgments, optional `better`, or a safe error code and failing stage. Identical answers have `judge_skipped: identical_answers`.
+- Each judgment: presentation order, original winner, mapped revision (`better`), reason, and metadata.
+
+Timestamps and model output vary. Input hashes, recorded ordering, configuration, and evidence support auditing; they do not promise byte-identical reruns. Schema 2 intentionally replaces the prototype's single-answer/single-judgment rows; consumers of schema 1 must update. No schema 1 public release existed.

volna_skillcheck-0.3.0/docs/releasing.md

@@ -0,0 +1,29 @@
+# Release checklist
+
+The public repository is [vladzima/volna-skillcheck](https://github.com/vladzima/volna-skillcheck). GitHub releases and [PyPI](https://pypi.org/project/volna-skillcheck/) carry the wheel and source archive. The portable skill lives in `skills/volna-skillcheck/` and is installed from GitHub independently of the CLI.
+
+1. Use **volna-skillcheck** as the GitHub repository name, distribution name, and CLI name. Confirm the GitHub owner, visibility, and PyPI ownership before publishing. If any name is unavailable, ask the maintainer; do not choose a replacement name.
+2. Review every included file for private material and confirm rights to publish it under MIT. The fixtures must remain synthetic. Inspect `git status --short --untracked-files=all`; reports, credentials, and local runtime files must stay excluded.
+3. Run the offline suite on supported Python versions and Linux/macOS in CI. Build a wheel and source distribution, run `twine check`, inspect archive contents, then install the wheel into a fresh environment outside the checkout and run the documented CLI path.
+4. For adapter behavior changes, run a live smoke evaluation where credentials are available, with at least one nonidentical answer pair and a positive-control case. Require zero adapter errors, both swapped judgments for the nonidentical pair, and populated model/version metadata. Do not require an improvement verdict: ties or insufficient evidence are valid results. Keep the report private. Revalidate isolation on runtime upgrades; disclose adapters that remain fixture-tested. For skill changes, validate frontmatter and references, install a standalone copy with the documented installer in a scratch project, and check discovery in an available harness.
+5. Set the release date, update the changelog, and synchronize the version in `pyproject.toml` and `volna_skillcheck.py`. Add verified repository URLs to package metadata after the repository exists.
+6. Require the CI workflow before merging. The `pypi` GitHub environment allows version tags (`v*`) and the `main` branch for manual dispatch. The publishing identity is the owner/repository/workflow/environment tuple below; only the separate publish job can obtain an OIDC token.
+7. Tag and publish only after reviewing the actual artifacts and the exact commit's CI. Publishing a GitHub release triggers `.github/workflows/publish.yml`. It checks tag/version agreement and ancestry on `main`, reruns offline tests, builds/checks/smoke-tests the wheel, and passes artifacts to the isolated publish job. Confirm the workflow succeeds, then install the package from PyPI in a clean environment and repeat the CLI smoke test before announcing it. Verify the public skill installer too.
+
+## Trusted Publishing setup
+
+The PyPI project owner configures this once under [account publishing](https://pypi.org/manage/account/publishing/) for a new project, or project publishing settings after the project exists:
+
+| Field | Value |
+| --- | --- |
+| PyPI project | `volna-skillcheck` |
+| GitHub owner | `vladzima` |
+| Repository | `volna-skillcheck` |
+| Workflow filename | `publish.yml` |
+| Environment | `pypi` |
+
+Use [PyPI's pending publisher flow](https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/) for the first upload. No long-lived PyPI token is needed. If a release ran before setup was complete, rerun its failed publish job after configuration. The workflow also supports manual dispatch from `main` with an existing release tag.
+
+The workflow builds the PyPI files itself. Download its `python-distributions` artifact if attaching the exact same files to the GitHub release; independently rebuilding can change archive hashes. Never replace an existing PyPI version: fix it with a new version. For a partial upload, first inspect which files are already public and their hashes; do not blindly retry or hide conflicts with `skip-existing`.
+
+A successful smoke evaluation verifies integration, not model quality. Publish the limitations alongside any example results.