robotframework-harness 1.0.0__tar.gz

robotframework_harness-1.0.0/.gitignore

@@ -0,0 +1,23 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.venv/
+dist/
+build/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+/site/
+# generated at build time from plugins/rf-harness (hatch_build.py)
+src/rf_harness/_assets/
+# Robot Framework / harness run artefacts
+output.xml
+log.html
+report.html
+results/
+.rf/
+# never commit secrets
+.env
+
+# evidence: release-asset build dir (transcript packs; published as GitHub release assets, never committed)
+evidence/_release_assets/

robotframework_harness-1.0.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

robotframework_harness-1.0.0/PKG-INFO

@@ -0,0 +1,85 @@
+Metadata-Version: 2.4
+Name: robotframework-harness
+Version: 1.0.0
+Summary: Agent harness for Robot Framework: an Agent Skill, agent hooks and a local robotcode-based live session for Claude Code, Codex, GitHub Copilot, VS Code and OpenCode.
+Project-URL: Homepage, https://github.com/manykarim/robotframework-harness
+Project-URL: Documentation, https://github.com/manykarim/robotframework-harness/tree/main/docs
+Project-URL: Issues, https://github.com/manykarim/robotframework-harness/issues
+Author: Many Kasiriha
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: agents,ai,claude-code,codex,copilot,opencode,playwright,robotcode,robotframework,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: Robot Framework
+Classifier: Framework :: Robot Framework :: Tool
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Provides-Extra: quality
+Requires-Dist: robotframework-robocop>=9.1; extra == 'quality'
+Provides-Extra: runtime
+Requires-Dist: robotcode[analyze,repl,runner]<2.8,>=2.7; extra == 'runtime'
+Requires-Dist: robotframework>=7.0; extra == 'runtime'
+Description-Content-Type: text/markdown
+
+# robotframework-harness
+
+**An agent harness for Robot Framework.** It gives coding agents — Claude Code, OpenAI Codex, GitHub Copilot (CLI and
+VS Code) and OpenCode — what they need to write, debug and repair Robot Framework tests reliably:
+
+- **Live execution while the agent edits.** Every write or edit of a `.robot` file runs the new lines in a local Robot
+  Framework session (built on [robotcode](https://github.com/robotcodedev/robotcode)). The agent gets a short answer:
+  passed, or the failing line with a verdict and the next step.
+- **Failures the agent can act on.** Covered elements, missing elements, iframes, timing and wrong values are told
+  apart from evidence. The agent also gets concrete selectors and the page state as files it can grep.
+- **Guard rails, not handcuffs.** Only certainly-wrong edits are refused (broken syntax, deleted tests, hidden
+  failures). Everything else becomes a one-line warning, and a green plain `robot` run is required before the agent
+  finishes.
+
+Everything runs on your machine; no MCP server is needed.
+
+## Quick start
+
+```bash
+cd your-robot-project
+uvx robotframework-harness install     # pick your agents; installs the skill, hooks and the runtime
+uvx robotframework-harness doctor      # proves the hooks fire and the live session works
+```
+
+Then ask your agent for a test. Per-agent guides are in [docs/](docs/).
+
+Plugin marketplace (Claude Code, Copilot CLI, VS Code, Codex):
+
+```bash
+claude plugin marketplace add manykarim/robotframework-harness
+claude plugin install rf-harness@robotframework-harness
+```
+
+## Why: evidence
+
+In same-window A/B runs on live end-to-end journeys (insurance quote, shop checkout, car configurator), the harness
+**passed 83% of sessions versus 69% for rf-mcp** across six agent/model combinations (206 sessions). It also cut wall
+time and cost for most agents. Weaker, cheaper models gained the most (Qwen 3.8 Flash: 0.38 → 0.88). All data,
+graders and reports are in [evidence/](evidence/).
+
+## Repository
+
+| path | what |
+|---|---|
+| `plugins/rf-harness/` | the plugin: Agent Skill, hook definitions, launcher, per-agent variants |
+| `.claude-plugin/marketplace.json` | plugin marketplace for Claude Code, Copilot CLI, VS Code and Codex |
+| `src/rf_harness/installer/` | the `rf-harness` installer (no dependencies) |
+| `src/rf_harness/runtime/` | hook dispatcher and live session (installed into your project environment) |
+| `docs/` | documentation |
+| `evidence/` | all experiment data behind the claims |
+| `eval/` | the A/B evaluation harness used as release gate |
+
+## License
+
+Apache-2.0

robotframework_harness-1.0.0/README.md

@@ -0,0 +1,55 @@
+# robotframework-harness
+
+**An agent harness for Robot Framework.** It gives coding agents — Claude Code, OpenAI Codex, GitHub Copilot (CLI and
+VS Code) and OpenCode — what they need to write, debug and repair Robot Framework tests reliably:
+
+- **Live execution while the agent edits.** Every write or edit of a `.robot` file runs the new lines in a local Robot
+  Framework session (built on [robotcode](https://github.com/robotcodedev/robotcode)). The agent gets a short answer:
+  passed, or the failing line with a verdict and the next step.
+- **Failures the agent can act on.** Covered elements, missing elements, iframes, timing and wrong values are told
+  apart from evidence. The agent also gets concrete selectors and the page state as files it can grep.
+- **Guard rails, not handcuffs.** Only certainly-wrong edits are refused (broken syntax, deleted tests, hidden
+  failures). Everything else becomes a one-line warning, and a green plain `robot` run is required before the agent
+  finishes.
+
+Everything runs on your machine; no MCP server is needed.
+
+## Quick start
+
+```bash
+cd your-robot-project
+uvx robotframework-harness install     # pick your agents; installs the skill, hooks and the runtime
+uvx robotframework-harness doctor      # proves the hooks fire and the live session works
+```
+
+Then ask your agent for a test. Per-agent guides are in [docs/](docs/).
+
+Plugin marketplace (Claude Code, Copilot CLI, VS Code, Codex):
+
+```bash
+claude plugin marketplace add manykarim/robotframework-harness
+claude plugin install rf-harness@robotframework-harness
+```
+
+## Why: evidence
+
+In same-window A/B runs on live end-to-end journeys (insurance quote, shop checkout, car configurator), the harness
+**passed 83% of sessions versus 69% for rf-mcp** across six agent/model combinations (206 sessions). It also cut wall
+time and cost for most agents. Weaker, cheaper models gained the most (Qwen 3.8 Flash: 0.38 → 0.88). All data,
+graders and reports are in [evidence/](evidence/).
+
+## Repository
+
+| path | what |
+|---|---|
+| `plugins/rf-harness/` | the plugin: Agent Skill, hook definitions, launcher, per-agent variants |
+| `.claude-plugin/marketplace.json` | plugin marketplace for Claude Code, Copilot CLI, VS Code and Codex |
+| `src/rf_harness/installer/` | the `rf-harness` installer (no dependencies) |
+| `src/rf_harness/runtime/` | hook dispatcher and live session (installed into your project environment) |
+| `docs/` | documentation |
+| `evidence/` | all experiment data behind the claims |
+| `eval/` | the A/B evaluation harness used as release gate |
+
+## License
+
+Apache-2.0

robotframework_harness-1.0.0/evidence/README.md

@@ -0,0 +1,173 @@
+# Evidence archive
+
+Every number this project publishes comes from the data in this folder. It covers every experiment round of the
+rf-mcp research that led to the harness: raw per-session results, graders, reports, the harness versions under
+test, the evaluation harness, excluded sessions with their reasons, and incident notes. Anyone can check,
+recompute and challenge the claims.
+
+**Source.** rf-mcp branch `research/robotcode-remote-mcp`, commit `7994ae3` (`experiments/robotcode-remote/**`
+and `docs/research/static-analysis-quality-gates.md`), plus files from the research worktree that were never
+committed: result-file backups, runner logs and one fixture. Details in [PROVENANCE.md](PROVENANCE.md).
+
+## Verify in one minute
+
+```bash
+python3 evidence/checksums.py --verify               # archived files match checksums.sha256 (also: sha256sum -c)
+python3 evidence/scrub.py --check evidence --exclude _release_assets   # no keys, tokens, e-mails or home paths
+python3 evidence/check_claims.py --check-sources     # every published number recomputed from the results
+python3 evidence/check_claims.py --reports           # archived analyzer regenerates the archived reports
+```
+
+All four use only the Python standard library (3.10+; `--reports` needs 3.12+, because `sum()` switched to
+compensated float summation in 3.12 and the research ran on 3.12/3.13). CI runs them on every change to `evidence/` and
+`README.md` (`.github/workflows/evidence.yml`).
+
+## Rounds
+
+Rounds are numbered as in the research. The two early A/B rounds ran between rounds 3 and 4 and carry their own
+names in the findings ("A/B round 1", "A/B round 2"), so they are `03a` and `03b`. Round 8 is the v1 release gate.
+
+| folder | dates | what was tested | agents / models | sessions | key result |
+|---|---|---|---|---|---|
+| [01-capability-spikes](rounds/01-capability-spikes/) | 2026-10-01 | robotcode as backend: remote MCP limits, hints, stepwise sessions, static validation, locators | none (scripted) | – | hints portable (32/33), analyze + dryrun catch 32/33 planted errors, 24–81 ms per step |
+| [02-local-runtime](rounds/02-local-runtime/) | 2026-10-01 | embedded helper, stock REPL, local keyword docs, agent shell channel | Claude Code, Codex, OpenCode, Copilot CLI | 34 | repl-server not needed; the helper works in all four agents with `setsid` |
+| [03-lsp-robocop](rounds/03-lsp-robocop/) | 2026-10-01 | robotcode LSP in agents, LSP as validator, Robocop 9.1 | same | 32 | LSP feedback only after a shim; Robocop agent profile catches 26/33 |
+| [03a-ab-file-authoring](rounds/03a-ab-file-authoring/) | 2026-10-01 | A/B 1: rf-mcp vs skill + local package, 4 hard authoring scenarios | Haiku 4.5, gpt-5.6-luna, MiniMax-M2.5 | 180 | mcp 0.97, skillpkg 0.92, neither 0.83 (n.s.) |
+| [03b-ab-live-journeys](rounds/03b-ab-live-journeys/) | 2026-10-02 | A/B 2: same arms on 5 live journeys (explore + debug) | same | 135 | skillpkg 0.60, neither 0.53, mcp 0.44 |
+| [04-hooks-matrix-and-hooks-ab](rounds/04-hooks-matrix-and-hooks-ab/) | 2026-10-08 | hook capability matrix; hooks (hookpkg) vs CLI (skillpkg) | same | 90 | hooks 0.58 vs CLI 0.56; Claude +0.33, OpenCode −0.33 |
+| [05-v2](rounds/05-v2/) | 2026-10-08/09 | REPL/debugger research; hookpkg v1 vs v2 | same | 90 | v2 0.60 vs v1 0.51; fixes OpenCode, costs Claude +48% |
+| [06-v3-main](rounds/06-v3-main/) | 2026-10-09 | diagnosis; same-day rf-mcp vs v2 vs v3 | same | 180 | v2 0.70, v3 0.63, rf-mcp 0.52; v2 > rf-mcp in 8/0 cells (p = 0.008) |
+| [07-v31-v32](rounds/07-v31-v32/) | 2026-10-10 | v3.1 per-model profiles vs v3 vs rf-mcp, OpenRouter models, Haiku 5.5; v3.1 vs v3.2 | Haiku 5.5, gpt-5.6-luna, MiniMax-M2.5, DeepSeek V4.1 / Qwen 3.8 / GLM 5.3 Flash (+ stopped Gemini) | 206 valid + 40 v3.2 + 10 Gemini | **v3.1 0.83 vs rf-mcp 0.69** (206 sessions); Qwen 0.38 → 0.88 |
+| [08-release-gate-v1](rounds/08-release-gate-v1/) | 2026-10-11 | release gate: rf-harness (product 0.9.0, own installer) vs the v3.2 prototype; real-session checks per agent | Haiku 5.5, gpt-5.6-luna, MiniMax-M2.5 (+ Copilot CLI check) | 90 + 5 | **0.82 vs 0.82**, gate PASS; OpenCode 345 s vs 492 s |
+| [static-analysis](rounds/static-analysis/) | 2026-10-02 | Robocop + robotcode analyze as quality gates for rf-mcp | none (scripted) | – | agent-profile Robocop + analyze + dryrun catch 32/33 |
+
+Every A/B round folder has the same layout:
+
+| path | content |
+|---|---|
+| `README.md` | what was tested, design, result, exclusions, files |
+| `findings*.md` | the original findings report (its `harness/<file>` paths now point to `results/` and `reports/`) |
+| `results/` | raw per-session result rows (`results_*.jsonl`, one JSON object per session) |
+| `reports/` | analyzer output (`report_*.txt`, `summary_*.json`) |
+| `scenarios.json` | scenario prompts, grader definitions, fixture and reference-solution hashes |
+| `config.json` | agents, models, arms, scenarios, reps, harness version, dates, recorded agent versions, excluded counts |
+| `excluded/` | `excluded_<reason>.jsonl`: each row with `reason`, `detail`, the backup it came from and whether it is still in the results file |
+| `backups/` | the original result files the excluded rows were removed from |
+| `logs/` | runner logs |
+| `build-verification/` | build checks of the harness version introduced in the round |
+| `checksums.sha256` | sha256 of every file in the folder (`sha256sum -c checksums.sha256`) |
+
+Other folders:
+- [`harness-versions/`](harness-versions/): source snapshots of the research prototypes (`skillpkg`, `hookpkg`
+  v1, `hookpkg2`, `hookpkg3`, `hookpkg31`, `hookpkg32`), code only.
+- [`eval-snapshots/ab-7994ae3/`](eval-snapshots/ab-7994ae3/): the evaluation harness, analyzers, runners and
+  scenario fixtures (with hidden references and negative controls) as committed at the end of the research.
+- [`upstream/`](upstream/): summary of the earlier robotcode CLI study (findings only, by owner decision).
+- [`incidents.md`](incidents.md): expired logins, SIGTERMs, credit exhaustion, the Codex outage, grader
+  artifacts, stopped runs, disk pressure.
+- [`claims.yaml`](claims.yaml): index of published claims.
+- [`transcripts/`](transcripts/): manifests of the transcript release assets.
+- `.gitignore` re-includes `results/`, `output.xml`, `log.html`, `report.html` and `site/`, which the root
+  `.gitignore` hides elsewhere. `.gitattributes` turns off end-of-line conversion, so checksums hold on every
+  platform.
+
+## Reading a result row
+
+One line per session. The fields you need most:
+- identity: `key` = `agent|arm|scenario|rep`, plus `agent`, `arm`, `scenario`, `rep`, `model`;
+- validity: `status` (`ok` or `aborted`) and `contaminated`;
+- outcome: `primary_pass` (all graders passed), `verdicts` (each grader with `passed` and `detail`),
+  `n_checks` / `n_pass`;
+- `metrics`: `wall_s`, `turns`, `tool_calls`, tokens, `cost_usd`, `timed_out`;
+- `stepwise`: live-step counts;
+- per-arm engagement, e.g. `hookpkg31`: hook firings, verdict classes, context characters per edit, profile;
+- `transcript`: path of the session transcript inside the release asset.
+
+The analyzers score rows with `status == "ok"` and keep the last ok row per `key`. Pass rate = mean of
+`primary_pass`. Contaminated rows are kept unless `--drop-contaminated` is given; the published reports keep them
+(see `claims.yaml`, `r7-rfmcp-drop-contaminated`).
+
+## Claims
+
+[`claims.yaml`](claims.yaml) lists every number in the README (and the main findings numbers the docs use). For
+each claim it records the round, the result files, how to recompute it (metric, filter, contamination policy),
+the expected values and tolerances, and the quoted source text. It is written in the JSON-compatible subset of YAML
+(full-line `#` comments, the rest is JSON), so `check_claims.py` needs no YAML library.
+
+```bash
+python3 evidence/check_claims.py --only readme-pooled-harness --show
+# ok   readme-pooled-harness: harness (v3.1) pooled pass rate 0.83 (83%) on live journeys  -> {'value': 0.8286, 'passes': 58, 'n': 70}
+```
+
+When a number in `README.md` or `docs/` changes, change or add its claim in the same commit.
+`check_claims.py --check-sources` fails when a quoted claim text is no longer in its source file.
+
+## Transcripts (release assets)
+
+The session transcripts (about 130 MB gzip in the research tree) are too large for git. They are published as
+GitHub release assets, one per A/B round. Each was scrubbed with the same `scrub.py`, packed decompressed, and
+compressed with zstd:
+
+| asset | transcripts | size |
+|---|---|---|
+| `evidence-transcripts-03a-ab-file-authoring.tar.zst` | 226 | 2.0 MB |
+| `evidence-transcripts-03b-ab-live-journeys.tar.zst` | 214 | 24.7 MB |
+| `evidence-transcripts-04-hooks-matrix-and-hooks-ab.tar.zst` | 93 | 11.7 MB |
+| `evidence-transcripts-05-v2.tar.zst` | 92 | 8.2 MB |
+| `evidence-transcripts-06-v3-main.tar.zst` | 197 | 12.7 MB |
+| `evidence-transcripts-07-v31-v32.tar.zst` | 335 | 25.8 MB |
+
+The committed manifest `transcripts/<round>.manifest.json` lists, for every transcript:
+- its path in the asset, sha256 and size;
+- the transcript path the result row refers to;
+- the results file and row `key`;
+- the row status: `scored`, `not_scored_in_results_file`, `excluded:<reason>`, or `unreferenced` (one
+  re-run smoke cell in 03a);
+- the scrub substitutions.
+
+`transcripts/assets.json` holds each asset's sha256.
+
+```bash
+mkdir -p evidence/_release_assets
+gh release download <tag> --repo manykarim/robotframework-harness --pattern 'evidence-transcripts-*' \
+    --dir evidence/_release_assets            # once the assets are published (task 8.2)
+python3 evidence/verify_assets.py --require    # asset sha256, every member's sha256, every transcript -> one row
+```
+
+`verify_assets.py` needs the `zstd` CLI (or Python 3.14+) for `.tar.zst`. To rebuild the assets from the research
+tree, run `python3 evidence/pack_transcripts.py --transcripts <research ab/harness dir>`. They are written to
+`evidence/_release_assets/` (git-ignored); nothing is uploaded.
+
+## Limits: read before quoting a number
+
+- **Small samples.** 3–5 reps per agent × arm × scenario cell. Most single contrasts are not significant at
+  p < 0.05. The strongest are round 6 (v2 > rf-mcp in 8 of 8 non-tied cells, sign test p = 0.008) and round 7
+  pooled (Fisher p = 0.073). Same-arm run-to-run noise between rounds is about ±0.07–0.13.
+- **Live sites.** The journeys run against public websites that change. Only same-window comparisons are valid,
+  and every round's arms were interleaved in one window ([REPRODUCE.md](REPRODUCE.md)).
+- **Grader artifacts.** The `\$916\.99` regex case hits 4 round-7 cells and 1 round-6 cell, in all arms. It is
+  not corrected; adjusted rates are in `incidents.md`.
+- **Exclusions.** 76 round-7 sessions were removed after the OpenRouter credit exhaustion. The OpenRouter half of
+  round 7 is therefore partial: rep 1 complete, rep 2 partial.
+- **Content confounds.** The prototypes' skills contained better guidance than rf-mcp's in some rounds (noted in
+  the findings). The comparison is "harness as built" vs "rf-mcp 0.36.0", not delivery mechanism alone.
+- **Weak and cheap models by design.** The models were chosen to leave headroom (Haiku, MiniMax, Flash models).
+  Strong models were near the ceiling with either delivery in round 7.
+- **Only the final harness code survives.** Earlier rounds ran earlier harness revisions. The archived reports of
+  the early rounds regenerate identically except for one data-quality line that the later analyzer splits
+  (turn caps vs agent errors).
+- **Uncommitted raw outputs.** About 46 MB of spike outputs, plus about 300 MB of agent run directories from rounds
+  1–5, were never committed upstream and are not in this archive (PROVENANCE.md).
+
+## Scrub
+
+`scrub.py` removes or redacts:
+- provider keys and tokens (OpenRouter, Anthropic, OpenAI, GitHub, Slack, AWS, Google, bearer tokens, JWTs);
+- OAuth fields and private keys;
+- the literal values of secret environment variables;
+- e-mail addresses outside the allowlist (the author's public addresses and fixture or demo-site domains);
+- agent home directories and authentication files.
+
+It also normalizes absolute home paths to `~`. It ran on every file during import and on every transcript during
+packing, and it re-checks the scrubbed transcript content before an asset is built. Import substitutions:
+`scrub_report.json` (counts only). Transcript substitutions: in each manifest.

robotframework_harness-1.0.0/evidence/rounds/01-capability-spikes/README.md

@@ -0,0 +1,27 @@
+# Round 1: capability spikes (2026-10-01)
+
+**Question.** Can robotcode replace the parts of rf-mcp that an agent harness needs: stepwise execution, hints on
+failures, static validation, keyword discovery, result analysis and locator guidance? Can a remote MCP server
+drive local execution at all?
+
+**How.** Six scripted studies, run by research subagents of the coordinating session against robotcode 2.7.0,
+Robot Framework 7.5 (cross-checked on 7.3.2) and rf-mcp 0.36.0. Public demo sites: Sauce Demo, the Flowline
+demoshop, Restful Booker, the-internet. No coding agents were involved and there are no A/B sessions.
+
+| study | folder | key result |
+|---|---|---|
+| deep research: remote MCP + robotcode | `research/findings.md` | A remote MCP server cannot make the client execute anything. robotcode `repl-server` is the only released way to keep RF state across separate shell calls. |
+| rf-mcp host-dependency map | `map/host-dependency-map.md` | Which rf-mcp components need the host (browser, files, RF context), with `file:line` references |
+| hints on robotcode output | `hints/findings.md` | rf-mcp's hint logic is portable: 32/33 identical hints from parsed `robotcode repl -k` output |
+| persistent stepwise sessions | `stepwise/findings.md` | `repl-server` + one-shot client: 24 ms (bash) / 81 ms (Python) median per step |
+| static validation, keyword search, results | `static/findings.md` | `robotcode analyze` + `robot --dryrun` catch 32 of 33 planted errors. Neither alone is enough. |
+| locator intelligence as RF steps | `locators/findings.md` | A guarded step reports 13 failure classes with live re-counted candidate locators, more informative than rf-mcp for Browser |
+
+**Files.** Each study folder holds its `findings.md`, the scripts and the committed outputs (tables, raw JSON,
+corpora). The outputs that were *not* committed upstream (run directories, caches, large logs, about 46 MB) are
+not in this archive; see `../../PROVENANCE.md`.
+
+**Config.** `config.json`. **Checksums.** `checksums.sha256` (run `sha256sum -c checksums.sha256` in this folder).
+
+**Limits.** Single machine, single runs. The latency numbers come from a machine that was running other Browser jobs.
+These are capability findings, not effect sizes.

robotframework_harness-1.0.0/evidence/rounds/02-local-runtime/README.md

@@ -0,0 +1,25 @@
+# Round 2: local runtime (2026-10-01)
+
+**Question.** Round 1 needed a long-lived `repl-server`. Can the same be done locally and more simply? Does a
+helper that agents call from their shell work in every coding agent?
+
+| study | folder | key result |
+|---|---|---|
+| embedded robotcode helper (Python API) | `python_api/findings.md` | Embedding robotcode's REPL engine in a per-project daemon is faster than `repl-server`, shuts down cleanly and runs rf-mcp logic around each keyword in the live RF context (spike 1,526 lines, core about 860). |
+| stock `robotcode repl` instead of repl-server | `repl_only/findings.md` | `repl-server` is not needed: stock `repl --plain` fed from a named pipe plus a listener gives per-step JSON. Shutdown 0.4 s vs 5.5–22 s. |
+| keyword docs via robotcode, locally | `keyword_docs/findings.md` | robotcode returns full keyword docs locally. Only *search* is name-only; BM25 search runs locally in under 2 ms per query. |
+| local helper channel across agents | `agents/findings.md` | Works in all four CLI agents if the helper self-detaches with `setsid` (34 headless sessions) |
+
+**Agents (agents study).** Claude Code 2.1.286 + claude-haiku-4-5, Codex 0.153.4 + gpt-5.6-luna@low (no sandbox:
+bwrap cannot start on this host), OpenCode 1.15.12 + MiniMax-M2.7, Copilot CLI 1.0.54 + gpt-4.1. VS Code Copilot
+was assessed from its docs only.
+
+**Files.** Each study folder holds its `findings.md`, the code (`python_api/rfmcp_local/`, `repl_only/rfrepl.py`,
+`keyword_docs/helper/KwDocs.py`, `agents/common/`), and the committed results (`agents/results_summary.json`,
+`keyword_docs/outputs/`, `repl_only/run/`). The agent run directories (`agents/runs/`), logs and benchmark work
+dirs were not committed upstream and are not archived.
+
+**Config.** `config.json`. **Checksums.** `checksums.sha256`.
+
+**Limits.** One session per agent × scenario. The sandboxes were not tested (Codex ran without sandbox, and Claude's
+sandbox cannot run on this host).

robotframework_harness-1.0.0/evidence/rounds/03-lsp-robocop/README.md

@@ -0,0 +1,25 @@
+# Round 3: robotcode LSP in agents, LSP as validator, Robocop (2026-10-01)
+
+**Question.** Can static feedback reach the agent *while it edits*, without running anything? Which static
+analyzer catches which agent mistakes, and how noisy is it?
+
+| study | folder | key result |
+|---|---|---|
+| robotcode LSP diagnostics inside coding agents | `agent_lsp/findings.md` | Holds only after fixes. As-is, the LSP gives no usable feedback after the first edit in Claude Code and none in OpenCode, because of three client/server mismatches. `shim/rc_lsp_shim.py` fixes all three. Codex has no LSP support, and Copilot CLI never starts the server on edits. |
+| LSP as validator + file-based step execution | `lsp_quality/findings.md` | Same engine as `robotcode analyze`: 28 of 33 planted errors caught, 48/48 diagnostic codes identical. All noise comes from Robocop defaults. |
+| Robocop 9.1 for agent-written RF | `robocop/findings.md` | Default rules catch 8 of 33; the project-rule "agent profile" catches 26. Robocop is the only layer that auto-fixes outdated syntax. |
+
+**Agents (agent_lsp, 32 headless sessions).** Claude Code 2.1.286, OpenCode 1.15.12, Copilot CLI 1.0.54,
+Codex 0.153.4. Every run's final file was checked independently with `robotcode analyze code`.
+
+**Tools.** robotcode 2.7.0, Robocop 9.1.0, Robot Framework 7.5.
+
+**Files.** `agent_lsp/` (orchestrator, analyzer, LSP probe, shim, hook, `results_summary.json`,
+`harness_notes.json`), `lsp_quality/` (experiment scripts, test projects; the `results/` and `logs/` folders were
+not committed upstream), `robocop/` (configs incl. `robocop-agent.toml`, corpora, scripts). Agent run directories
+(`agent_lsp/runs/`, about 260 MB) are not archived.
+
+**Config.** `config.json`. **Checksums.** `checksums.sha256`.
+
+**Limits.** One run per agent × cell. The error corpus is planted (33 errors, 15 valid controls), not sampled from
+real agent output; `robocop/noise_corpus` (36 agent-written suites) covers the noise side.