superagentic-metaharness 0.1.0__tar.gz

superagentic_metaharness-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Shashi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

superagentic_metaharness-0.1.0/PKG-INFO

@@ -0,0 +1,355 @@
+Metadata-Version: 2.4
+Name: superagentic-metaharness
+Version: 0.1.0
+Summary: Filesystem-first harness optimization for coding agents.
+Author-email: Superagentic AI <hello@super-agentic.ai>, Shashi Jagtap <shashikant.jagtap@icloud.com>, Shashi Jagtap <shashi@super-agentic.ai>
+Maintainer-email: Superagentic AI <hello@super-agentic.ai>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/SuperagenticAI/metaharness
+Project-URL: Documentation, https://superagenticai.github.io/metaharness/
+Project-URL: Repository, https://github.com/SuperagenticAI/metaharness
+Project-URL: Issues, https://github.com/SuperagenticAI/metaharness/issues
+Keywords: agents,benchmarking,codex,evaluation,harness,optimization
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: mkdocs<2,>=1.6; extra == "dev"
+Requires-Dist: mkdocs-material<10,>=9.6; extra == "dev"
+Dynamic: license-file
+
+# metaharness
+
+[![CI](https://github.com/SuperagenticAI/metaharness/actions/workflows/ci.yml/badge.svg)](https://github.com/SuperagenticAI/metaharness/actions/workflows/ci.yml)
+[![Docs](https://github.com/SuperagenticAI/metaharness/actions/workflows/pages.yml/badge.svg)](https://github.com/SuperagenticAI/metaharness/actions/workflows/pages.yml)
+[![Documentation](https://img.shields.io/badge/docs-GitHub%20Pages-111111?logo=github&logoColor=white)](https://superagenticai.github.io/metaharness/)
+[![Python](https://img.shields.io/badge/python-3.11%2B-3776AB?logo=python&logoColor=white)](https://github.com/SuperagenticAI/metaharness/blob/main/pyproject.toml)
+[![License](https://img.shields.io/github/license/SuperagenticAI/metaharness)](https://github.com/SuperagenticAI/metaharness/blob/main/LICENSE)
+[![Status](https://img.shields.io/badge/status-alpha-F59E0B)](https://github.com/SuperagenticAI/metaharness)
+[![Paper](https://img.shields.io/badge/paper-Meta%20Harness-B31B1B)](https://arxiv.org/pdf/2603.28052)
+
+`metaharness` is an open source Python library for optimizing executable harnesses around agentic coding systems.
+It is inspired by the [Meta Harness paper](https://arxiv.org/pdf/2603.28052) and is an unofficial open source implementation of the core ideas in that work.
+The current implementation and benchmark evidence in this repository are centered on the Codex CLI path, including hosted Codex and Codex over local Ollama models.
+
+It is built for teams who want to improve the code and files around an agent workflow, not just the prompt.
+That includes instruction files, setup flows, validation scripts, test scripts, routing logic, and other executable support code.
+
+## Why `metaharness`
+
+Many agent failures come from the harness around the model:
+
+- weak repository instructions
+- missing setup steps
+- broken validation logic
+- incomplete test flows
+- poor iteration memory
+- acceptance checks that do not match the real task
+
+`metaharness` turns those artifacts into a repeatable optimization target with stored evidence for every proposal.
+It also captures a compact environment snapshot before each proposal so agents do not waste early turns on basic workspace discovery.
+Projects can also declare an allowed write scope so off-target edits are rejected automatically.
+
+## How It Works
+
+`metaharness` runs an outer optimization loop around a harness:
+
+1. start from a baseline workspace
+2. ask a coding agent to improve it
+3. validate and evaluate the result
+4. keep the best candidate
+5. store all artifacts on disk
+
+The result is a practical, inspectable workflow for improving real harnesses instead of ad hoc prompt tinkering.
+
+## Who It Is For
+
+- developers building agentic coding systems who want to optimize harness code, workflow scripts, retrieval wrappers, routing, and evaluation flows
+- practitioners using coding-agent tools who want to improve `AGENTS.md`, `GEMINI.md`, bootstrap scripts, validation scripts, and acceptance tests
+
+## Quickstart
+
+Install the project:
+
+```bash
+uv sync
+```
+
+Run the fake backend on a real benchmark:
+
+```bash
+uv run metaharness run examples/python_fixture_benchmark --backend fake --budget 1 --run-name quickstart
+```
+
+Inspect the run:
+
+```bash
+uv run metaharness inspect examples/python_fixture_benchmark/runs/quickstart
+```
+
+Export the candidate ledger:
+
+```bash
+uv run metaharness ledger examples/python_fixture_benchmark/runs/quickstart --tsv
+```
+
+Run a saved experiment matrix:
+
+```bash
+uv run metaharness experiment --config examples/experiment_configs/fake-benchmarks.json
+```
+
+## Core Capabilities
+
+- a minimal optimization engine
+- a filesystem-backed run store
+- automatic environment bootstrap snapshots for each proposal
+- optional write-scope enforcement through `allowed_write_paths`
+- a provider-neutral proposer backend interface
+- a real `CodexExecBackend`
+- a deterministic `FakeBackend`
+- a coding-tool integration for instruction files and script-based harnesses
+- explicit per-candidate outcomes: `keep`, `discard`, `crash`, `timeout`, `no-change`, and `scope-violation`
+- reporting commands for `inspect`, `ledger`, `summarize`, and `compare`
+- experiment-matrix execution with JSON and TSV outputs
+- benchmark targets and experiment records
+
+## Current Status
+
+The repository currently includes:
+
+- two real coding-tool benchmark targets
+- a smaller deterministic ticket-router example
+- hosted Codex runs on the real benchmarks
+- local Codex over Ollama runs with `gpt-oss:20b` and `gpt-oss:120b`
+- a docs site published from GitHub Actions
+
+Current documented experiments in this repository show:
+
+- hosted Codex solves both real benchmarks in one proposal iteration
+- local `gpt-oss:120b` solves `python_fixture_benchmark`
+- local `gpt-oss:20b` is useful for smoke checks but timed out on the current real benchmark runs
+
+Detailed experiment records:
+
+- [Benchmark overview](BENCHMARKS.md)
+- [Recorded benchmark results](BENCHMARK_RESULTS.md)
+- [Experiment notes](docs/experiments.md)
+
+## Provider Status
+
+- Codex is the main validated harness path in this repository today
+- hosted Codex is the strongest current path for real runs
+- local Codex over Ollama works and has been exercised with `gpt-oss:20b` and `gpt-oss:120b`
+- Gemini exists as a scaffolded backend and is not yet at parity with Codex
+
+All real provider results currently documented in this repository were produced through the Codex CLI path.
+That includes both hosted Codex runs and local Ollama runs driven through Codex with `gpt-oss` models.
+Other coding-agent evaluations in the wider ecosystem often emphasize Claude Code and Opus, but this repository's current benchmark evidence is Codex-first.
+
+## Documentation
+
+- [Project documentation](https://superagenticai.github.io/metaharness/)
+- [Getting started](https://superagenticai.github.io/metaharness/getting-started/)
+- [Architecture](https://superagenticai.github.io/metaharness/architecture/)
+- [Providers](https://superagenticai.github.io/metaharness/providers/)
+- [Benchmarks](https://superagenticai.github.io/metaharness/benchmarks/)
+- [CLI reference](https://superagenticai.github.io/metaharness/cli-reference/)
+- [Experiments](https://superagenticai.github.io/metaharness/experiments/)
+
+## Installation
+
+Project setup:
+
+```bash
+uv sync
+```
+
+If you want the docs toolchain too:
+
+```bash
+uv sync --group dev
+```
+
+Check the CLI:
+
+```bash
+uv run metaharness --help
+```
+
+Editable install with `pip` also works:
+
+```bash
+pip install -e .
+```
+
+## Hosted Codex
+
+Requirements:
+
+- `codex` CLI installed
+- authenticated Codex session or API key
+- outbound network access
+
+Run a real benchmark with hosted Codex:
+
+```bash
+uv run metaharness run examples/python_fixture_benchmark --backend codex --hosted --budget 1 --run-name hosted-codex
+```
+
+Important:
+
+- use `--hosted` when a project config defaults to local Ollama
+- the library is ready for hosted Codex runs today
+
+## Local Codex Over Ollama
+
+Probe the local setup:
+
+```bash
+uv run metaharness smoke codex examples/python_fixture_benchmark --probe-only --oss --local-provider ollama --model gpt-oss:20b
+```
+
+Run with `gpt-oss:20b`:
+
+```bash
+uv run metaharness run examples/python_fixture_benchmark --backend codex --oss --local-provider ollama --model gpt-oss:20b --proposal-timeout 240 --budget 1 --run-name ollama-20b
+```
+
+Run with `gpt-oss:120b`:
+
+```bash
+uv run metaharness run examples/python_fixture_benchmark --backend codex --oss --local-provider ollama --model gpt-oss:120b --proposal-timeout 420 --budget 1 --run-name ollama-120b
+```
+
+## Benchmarks And Examples
+
+Real benchmarks:
+
+- [examples/python_fixture_benchmark](examples/python_fixture_benchmark)
+- [examples/python_cli_benchmark](examples/python_cli_benchmark)
+
+Smaller deterministic example:
+
+- [examples/ticket_router](examples/ticket_router)
+
+Run the ticket router example:
+
+```bash
+uv run python examples/ticket_router/run.py --backend fake --budget 1
+```
+
+## Scaffold Your Own Project
+
+Create a coding-tool project:
+
+```bash
+uv run metaharness scaffold coding-tool ./my-coding-tool-optimizer
+```
+
+Available profiles:
+
+- `standard`
+- `local-oss-smoke`
+- `local-oss-medium`
+
+Run the scaffold with the fake backend:
+
+```bash
+uv run metaharness run ./my-coding-tool-optimizer --backend fake --budget 1
+```
+
+## CLI Overview
+
+Create a scaffold:
+
+```bash
+uv run metaharness scaffold coding-tool ./my-project
+```
+
+Run a project:
+
+```bash
+uv run metaharness run ./my-project --backend fake --budget 1
+```
+
+Probe Codex:
+
+```bash
+uv run metaharness smoke codex ./my-project --probe-only
+```
+
+Inspect a run:
+
+```bash
+uv run metaharness inspect ./my-project/runs/example
+```
+
+Compare runs:
+
+```bash
+uv run metaharness compare \
+  ./examples/python_fixture_benchmark/runs/hosted-codex-20260401 \
+  ./examples/python_fixture_benchmark/runs/ollama-20b-20260401 \
+  ./examples/python_fixture_benchmark/runs/ollama-120b-20260401
+```
+
+Run an experiment matrix:
+
+```bash
+uv run metaharness experiment --config examples/experiment_configs/fake-benchmarks.json
+```
+
+## Benefits Of The Filesystem Approach
+
+Every run stores:
+
+- prompts
+- candidate workspaces
+- validation results
+- evaluation results
+- proposal metadata
+- workspace diffs
+- per-candidate manifests
+
+That makes the optimization history reviewable, debuggable, and reusable.
+
+## Development
+
+Compile checks:
+
+```bash
+uv run python -m compileall -q src tests examples docs
+```
+
+Unit tests:
+
+```bash
+uv run python -m unittest discover -s tests -v
+```
+
+Docs build:
+
+```bash
+uv run mkdocs build --strict
+```
+
+Fake benchmark smoke runs:
+
+```bash
+uv run metaharness run examples/python_fixture_benchmark --backend fake --budget 1 --run-name ci-fixture-local
+uv run metaharness run examples/python_cli_benchmark --backend fake --budget 1 --run-name ci-cli-local
+uv run python examples/ticket_router/run.py --backend fake --budget 1
+```
+
+## License
+
+MIT. See [LICENSE](LICENSE).

superagentic_metaharness-0.1.0/README.md

@@ -0,0 +1,327 @@
+# metaharness
+
+[![CI](https://github.com/SuperagenticAI/metaharness/actions/workflows/ci.yml/badge.svg)](https://github.com/SuperagenticAI/metaharness/actions/workflows/ci.yml)
+[![Docs](https://github.com/SuperagenticAI/metaharness/actions/workflows/pages.yml/badge.svg)](https://github.com/SuperagenticAI/metaharness/actions/workflows/pages.yml)
+[![Documentation](https://img.shields.io/badge/docs-GitHub%20Pages-111111?logo=github&logoColor=white)](https://superagenticai.github.io/metaharness/)
+[![Python](https://img.shields.io/badge/python-3.11%2B-3776AB?logo=python&logoColor=white)](https://github.com/SuperagenticAI/metaharness/blob/main/pyproject.toml)
+[![License](https://img.shields.io/github/license/SuperagenticAI/metaharness)](https://github.com/SuperagenticAI/metaharness/blob/main/LICENSE)
+[![Status](https://img.shields.io/badge/status-alpha-F59E0B)](https://github.com/SuperagenticAI/metaharness)
+[![Paper](https://img.shields.io/badge/paper-Meta%20Harness-B31B1B)](https://arxiv.org/pdf/2603.28052)
+
+`metaharness` is an open source Python library for optimizing executable harnesses around agentic coding systems.
+It is inspired by the [Meta Harness paper](https://arxiv.org/pdf/2603.28052) and is an unofficial open source implementation of the core ideas in that work.
+The current implementation and benchmark evidence in this repository are centered on the Codex CLI path, including hosted Codex and Codex over local Ollama models.
+
+It is built for teams who want to improve the code and files around an agent workflow, not just the prompt.
+That includes instruction files, setup flows, validation scripts, test scripts, routing logic, and other executable support code.
+
+## Why `metaharness`
+
+Many agent failures come from the harness around the model:
+
+- weak repository instructions
+- missing setup steps
+- broken validation logic
+- incomplete test flows
+- poor iteration memory
+- acceptance checks that do not match the real task
+
+`metaharness` turns those artifacts into a repeatable optimization target with stored evidence for every proposal.
+It also captures a compact environment snapshot before each proposal so agents do not waste early turns on basic workspace discovery.
+Projects can also declare an allowed write scope so off-target edits are rejected automatically.
+
+## How It Works
+
+`metaharness` runs an outer optimization loop around a harness:
+
+1. start from a baseline workspace
+2. ask a coding agent to improve it
+3. validate and evaluate the result
+4. keep the best candidate
+5. store all artifacts on disk
+
+The result is a practical, inspectable workflow for improving real harnesses instead of ad hoc prompt tinkering.
+
+## Who It Is For
+
+- developers building agentic coding systems who want to optimize harness code, workflow scripts, retrieval wrappers, routing, and evaluation flows
+- practitioners using coding-agent tools who want to improve `AGENTS.md`, `GEMINI.md`, bootstrap scripts, validation scripts, and acceptance tests
+
+## Quickstart
+
+Install the project:
+
+```bash
+uv sync
+```
+
+Run the fake backend on a real benchmark:
+
+```bash
+uv run metaharness run examples/python_fixture_benchmark --backend fake --budget 1 --run-name quickstart
+```
+
+Inspect the run:
+
+```bash
+uv run metaharness inspect examples/python_fixture_benchmark/runs/quickstart
+```
+
+Export the candidate ledger:
+
+```bash
+uv run metaharness ledger examples/python_fixture_benchmark/runs/quickstart --tsv
+```
+
+Run a saved experiment matrix:
+
+```bash
+uv run metaharness experiment --config examples/experiment_configs/fake-benchmarks.json
+```
+
+## Core Capabilities
+
+- a minimal optimization engine
+- a filesystem-backed run store
+- automatic environment bootstrap snapshots for each proposal
+- optional write-scope enforcement through `allowed_write_paths`
+- a provider-neutral proposer backend interface
+- a real `CodexExecBackend`
+- a deterministic `FakeBackend`
+- a coding-tool integration for instruction files and script-based harnesses
+- explicit per-candidate outcomes: `keep`, `discard`, `crash`, `timeout`, `no-change`, and `scope-violation`
+- reporting commands for `inspect`, `ledger`, `summarize`, and `compare`
+- experiment-matrix execution with JSON and TSV outputs
+- benchmark targets and experiment records
+
+## Current Status
+
+The repository currently includes:
+
+- two real coding-tool benchmark targets
+- a smaller deterministic ticket-router example
+- hosted Codex runs on the real benchmarks
+- local Codex over Ollama runs with `gpt-oss:20b` and `gpt-oss:120b`
+- a docs site published from GitHub Actions
+
+Current documented experiments in this repository show:
+
+- hosted Codex solves both real benchmarks in one proposal iteration
+- local `gpt-oss:120b` solves `python_fixture_benchmark`
+- local `gpt-oss:20b` is useful for smoke checks but timed out on the current real benchmark runs
+
+Detailed experiment records:
+
+- [Benchmark overview](BENCHMARKS.md)
+- [Recorded benchmark results](BENCHMARK_RESULTS.md)
+- [Experiment notes](docs/experiments.md)
+
+## Provider Status
+
+- Codex is the main validated harness path in this repository today
+- hosted Codex is the strongest current path for real runs
+- local Codex over Ollama works and has been exercised with `gpt-oss:20b` and `gpt-oss:120b`
+- Gemini exists as a scaffolded backend and is not yet at parity with Codex
+
+All real provider results currently documented in this repository were produced through the Codex CLI path.
+That includes both hosted Codex runs and local Ollama runs driven through Codex with `gpt-oss` models.
+Other coding-agent evaluations in the wider ecosystem often emphasize Claude Code and Opus, but this repository's current benchmark evidence is Codex-first.
+
+## Documentation
+
+- [Project documentation](https://superagenticai.github.io/metaharness/)
+- [Getting started](https://superagenticai.github.io/metaharness/getting-started/)
+- [Architecture](https://superagenticai.github.io/metaharness/architecture/)
+- [Providers](https://superagenticai.github.io/metaharness/providers/)
+- [Benchmarks](https://superagenticai.github.io/metaharness/benchmarks/)
+- [CLI reference](https://superagenticai.github.io/metaharness/cli-reference/)
+- [Experiments](https://superagenticai.github.io/metaharness/experiments/)
+
+## Installation
+
+Project setup:
+
+```bash
+uv sync
+```
+
+If you want the docs toolchain too:
+
+```bash
+uv sync --group dev
+```
+
+Check the CLI:
+
+```bash
+uv run metaharness --help
+```
+
+Editable install with `pip` also works:
+
+```bash
+pip install -e .
+```
+
+## Hosted Codex
+
+Requirements:
+
+- `codex` CLI installed
+- authenticated Codex session or API key
+- outbound network access
+
+Run a real benchmark with hosted Codex:
+
+```bash
+uv run metaharness run examples/python_fixture_benchmark --backend codex --hosted --budget 1 --run-name hosted-codex
+```
+
+Important:
+
+- use `--hosted` when a project config defaults to local Ollama
+- the library is ready for hosted Codex runs today
+
+## Local Codex Over Ollama
+
+Probe the local setup:
+
+```bash
+uv run metaharness smoke codex examples/python_fixture_benchmark --probe-only --oss --local-provider ollama --model gpt-oss:20b
+```
+
+Run with `gpt-oss:20b`:
+
+```bash
+uv run metaharness run examples/python_fixture_benchmark --backend codex --oss --local-provider ollama --model gpt-oss:20b --proposal-timeout 240 --budget 1 --run-name ollama-20b
+```
+
+Run with `gpt-oss:120b`:
+
+```bash
+uv run metaharness run examples/python_fixture_benchmark --backend codex --oss --local-provider ollama --model gpt-oss:120b --proposal-timeout 420 --budget 1 --run-name ollama-120b
+```
+
+## Benchmarks And Examples
+
+Real benchmarks:
+
+- [examples/python_fixture_benchmark](examples/python_fixture_benchmark)
+- [examples/python_cli_benchmark](examples/python_cli_benchmark)
+
+Smaller deterministic example:
+
+- [examples/ticket_router](examples/ticket_router)
+
+Run the ticket router example:
+
+```bash
+uv run python examples/ticket_router/run.py --backend fake --budget 1
+```
+
+## Scaffold Your Own Project
+
+Create a coding-tool project:
+
+```bash
+uv run metaharness scaffold coding-tool ./my-coding-tool-optimizer
+```
+
+Available profiles:
+
+- `standard`
+- `local-oss-smoke`
+- `local-oss-medium`
+
+Run the scaffold with the fake backend:
+
+```bash
+uv run metaharness run ./my-coding-tool-optimizer --backend fake --budget 1
+```
+
+## CLI Overview
+
+Create a scaffold:
+
+```bash
+uv run metaharness scaffold coding-tool ./my-project
+```
+
+Run a project:
+
+```bash
+uv run metaharness run ./my-project --backend fake --budget 1
+```
+
+Probe Codex:
+
+```bash
+uv run metaharness smoke codex ./my-project --probe-only
+```
+
+Inspect a run:
+
+```bash
+uv run metaharness inspect ./my-project/runs/example
+```
+
+Compare runs:
+
+```bash
+uv run metaharness compare \
+  ./examples/python_fixture_benchmark/runs/hosted-codex-20260401 \
+  ./examples/python_fixture_benchmark/runs/ollama-20b-20260401 \
+  ./examples/python_fixture_benchmark/runs/ollama-120b-20260401
+```
+
+Run an experiment matrix:
+
+```bash
+uv run metaharness experiment --config examples/experiment_configs/fake-benchmarks.json
+```
+
+## Benefits Of The Filesystem Approach
+
+Every run stores:
+
+- prompts
+- candidate workspaces
+- validation results
+- evaluation results
+- proposal metadata
+- workspace diffs
+- per-candidate manifests
+
+That makes the optimization history reviewable, debuggable, and reusable.
+
+## Development
+
+Compile checks:
+
+```bash
+uv run python -m compileall -q src tests examples docs
+```
+
+Unit tests:
+
+```bash
+uv run python -m unittest discover -s tests -v
+```
+
+Docs build:
+
+```bash
+uv run mkdocs build --strict
+```
+
+Fake benchmark smoke runs:
+
+```bash
+uv run metaharness run examples/python_fixture_benchmark --backend fake --budget 1 --run-name ci-fixture-local
+uv run metaharness run examples/python_cli_benchmark --backend fake --budget 1 --run-name ci-cli-local
+uv run python examples/ticket_router/run.py --backend fake --budget 1
+```
+
+## License
+
+MIT. See [LICENSE](LICENSE).