pyplaykit 1.37.1__tar.gz → 1.37.2__tar.gz

{pyplaykit-1.37.1 → pyplaykit-1.37.2}/CHANGELOG.md

@@ -11,6 +11,26 @@ Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ---
 
+## [1.37.2] — 2026-10-11
+
+`test-creation` can now commit generated tests to a feature branch of your sandbox repository and push that branch for you to validate and raise as a pull request; a hands-on playground with sample CSVs ships with the package (`pyplaykit docs test-creation`); scaffolded projects no longer break without pandas. Agents stay **disabled by default**, and branch mode needs `agents.autonomy.max: branch` plus `--autonomy branch`.
+
+### Behaviour changes — check before upgrading
+
+- **Test runs started by agents no longer receive `PYPLAYKIT_SANDBOX_*` variables** (they hold the sandbox push token). *What to do:* nothing, unless a test reads such a variable; rename it.
+- **`agents.scm.sandbox` is a new config block** (empty `repo` by default). Projects that copy `config/config.yaml` by hand can add it from a fresh `pyplaykit init`; without it the defaults apply.
+
+### Added
+
+- **Test creation playground (`pyplaykit docs test-creation`).** A hands-on guide from a fresh `pip install` + `pyplaykit init` project: skeletons without a model, real tests with a model, sandbox feature branches, CSV format, settings and troubleshooting. Seven sample CSVs ship with the package (UI, API with inferred types, database, device, mixed, one row per step, custom column names); `pyplaykit docs <topic> --save-examples [DIR]` copies them (default `test_data/testcases/`, never overwrites).
+- **Sandbox feature branches for `test-creation` (ADR-0019 amendment 1, US-AG12.3.3).** With `--autonomy branch` and `agents.scm.sandbox.repo` set, `pyplaykit agents run test-creation` clones the sandbox repository, creates `feature/test-creation/<csv>-<UTC time>` from `origin/main`, generates the tests against that repository's layout, validates them (`pytest --collect-only`; device tests in sim mode), commits them with the test-case CSV as the engineer, and pushes only the feature branch. It never opens a pull request or touches `main`: the engineer validates and edits the tests and opens the pull request. The push token comes from `PYPLAYKIT_SANDBOX_GIT_TOKEN` (environment or `.env`) and reaches git only as an HTTP header; test runs never receive it. New `agents.scm.sandbox` settings; one `sandbox_branch` audit record per run.
+
+### Fixed
+
+- **Scaffolded data-comparison demos without pandas.** In a project created by `pyplaykit init` on a plain `pip install pyplaykit`, `test_demo_excel.py` broke test collection (`No module named 'pandas'`) and the file-comparison demos failed. They now skip with `pip install "pyplaykit[data-comparison]"`; `init` names the same extra when it skips the demo workbooks. Existing projects: re-run `pyplaykit init` does not replace existing files; copy the new guard from a fresh scaffold or install the extra.
+
+---
+
 ## [1.37.1] — 2026-10-11
 
 First release on PyPI of the QE agents Phase 1 work: the v1.37.0 tag was not published because its CI gate failed on three agent unit tests under `CI=true`. 1.37.1 contains everything in [1.37.0] below, plus two new built-in agents, `test-review` and `test-creation`. Agents stay **disabled by default**.

{pyplaykit-1.37.1/pyplaykit.egg-info → pyplaykit-1.37.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pyplaykit
-Version: 1.37.1
+Version: 1.37.2
 Summary: Enterprise Python + Playwright automation framework
 Author-email: Shan Konduru <shankonduru@gmail.com>
 License: MIT
@@ -1021,6 +1021,7 @@ Each project config includes:
 - GenAI spans for agent runs, LLM and tool calls (OpenTelemetry GenAI conventions), content capture off by default; `agent.*` events and `pyplaykit_agent_*` Prometheus metrics (tokens, cost, latency, tool calls, budget breaches, proposals); `reports/agents/runs.jsonl` and `usage_summary.json`, agent sections in the persona reports, a dashboard panel, `pyplaykit agents usage` and the MCP `list_agents` / `get_agent_usage` tools
 - v1.37.0: Phase 1 complete — tool layer and Python stack adapter, the test-impact map, and the first built-in agent `change-intelligence` (impact, risk score, regression plan applied with `--pyplaykit-select-from` in PR pipelines); user guide: `pyplaykit docs agents`
 - v1.37.1: first PyPI release of Phase 1 (v1.37.0 was not published), plus built-in agents `test-review` (test code quality) and `test-creation` (UI, API, database and device tests from test-case CSVs)
+- v1.37.2: `test-creation` commits to a feature branch of your sandbox repository and pushes it (never a pull request); hands-on playground with sample CSVs (`pyplaykit docs test-creation --save-examples`); scaffold demos skip without pandas
 - v1.36.0: agent governance and safety — policy through org governance, redaction, audit log, tool tiers, enforcement model
 - v1.35.0: agent observability — GenAI traces, `agent.*` events and `pyplaykit_agent_*` metrics, run log and reports
 - v1.34.0: LLM layer — provider-neutral requests, Anthropic / OpenAI-compatible / Ollama adapters, fake and record/replay clients, model tiers with fallback and retries, budgets, cost and a response cache. No built-in agents ship yet.

{pyplaykit-1.37.1 → pyplaykit-1.37.2}/README.md

@@ -846,6 +846,7 @@ Each project config includes:
 - GenAI spans for agent runs, LLM and tool calls (OpenTelemetry GenAI conventions), content capture off by default; `agent.*` events and `pyplaykit_agent_*` Prometheus metrics (tokens, cost, latency, tool calls, budget breaches, proposals); `reports/agents/runs.jsonl` and `usage_summary.json`, agent sections in the persona reports, a dashboard panel, `pyplaykit agents usage` and the MCP `list_agents` / `get_agent_usage` tools
 - v1.37.0: Phase 1 complete — tool layer and Python stack adapter, the test-impact map, and the first built-in agent `change-intelligence` (impact, risk score, regression plan applied with `--pyplaykit-select-from` in PR pipelines); user guide: `pyplaykit docs agents`
 - v1.37.1: first PyPI release of Phase 1 (v1.37.0 was not published), plus built-in agents `test-review` (test code quality) and `test-creation` (UI, API, database and device tests from test-case CSVs)
+- v1.37.2: `test-creation` commits to a feature branch of your sandbox repository and pushes it (never a pull request); hands-on playground with sample CSVs (`pyplaykit docs test-creation --save-examples`); scaffold demos skip without pandas
 - v1.36.0: agent governance and safety — policy through org governance, redaction, audit log, tool tiers, enforcement model
 - v1.35.0: agent observability — GenAI traces, `agent.*` events and `pyplaykit_agent_*` metrics, run log and reports
 - v1.34.0: LLM layer — provider-neutral requests, Anthropic / OpenAI-compatible / Ollama adapters, fake and record/replay clients, model tiers with fallback and retries, budgets, cost and a response cache. No built-in agents ship yet.

{pyplaykit-1.37.1 → pyplaykit-1.37.2}/config/config.yaml

@@ -311,6 +311,14 @@ agents:
   scm:
     protected_branches: [main, master, "release/*"]
     allow_pr: false
+    sandbox:                            # test-creation with --autonomy branch (and autonomy.max: branch)
+      repo: ""                          # repository for feature branches (URL or path); "" = this repository
+      base: main                        # branches start from origin/<base>; never pushed to
+      branch_prefix: feature/test-creation
+      push: true                        # push the feature branch only; never a pull request
+      token_env: PYPLAYKIT_SANDBOX_GIT_TOKEN  # token variable (environment, else .env); sent as a header only
+      writable: [tests, pages, test_data]
+      workspace: reports/agents/sandbox # clone and worktrees
   telemetry:                            # GenAI spans need telemetry.enabled (and otlp_endpoint for OTLP)
     capture_content: false              # prompts, completions, tool arguments/results; redacted
     otlp_classes: [metadata]            # data classes the OTLP endpoint may receive; captured content

{pyplaykit-1.37.1 → pyplaykit-1.37.2}/docs/QE_AGENTS_DESIGN.md

@@ -953,6 +953,8 @@ Implemented in US-AG17.3.3 (`pyplaykit/agents/builtin/test_review/`): `test-revi
 
 Implemented in US-AG12.3.1 and US-AG12.3.2 ([ADR-0019](adr/0019-test-creation-from-test-cases-on-sandbox-branches.md), `pyplaykit/agents/builtin/test_creation/`): `test-creation` reads a test-case CSV (columns by name and synonyms; one row per case or per step), detects the `pyplaykit init` layout (multi- or single-project) with the new `list_files` read tool, gathers page objects, `BasePage`, utilities, fixtures, registered markers, DUTs and an example test per type, and asks the model per type and batch for test bodies and page-object members. The host assembles every file (imports, traceable docstrings, registered markers, unique module basenames, page-object edits that never overwrite existing code), checks it with the test-review rules and resends problems up to `fix_attempts`, and delivers a patch, report and skeletons when no model is available. `read_source` now reports each file's newline style so patches apply to CRLF and LF checkouts.
 
+Implemented in US-AG12.3.3 ([ADR-0019](adr/0019-test-creation-from-test-cases-on-sandbox-branches.md) amendment 1, `pyplaykit/agents/sandbox.py`): with effective autonomy `branch`, `pyplaykit agents run` clones `agents.scm.sandbox.repo` (or uses the project's repository), adds a worktree on `<branch_prefix>/<name>-<UTC time>` from `origin/<base>`, stages local input files, points the run's read tools at the worktree, applies the agent's patch (only `writable` paths), validates with `pytest --collect-only` and device tests in sim mode, commits as the engineer and pushes the feature branch only (token as an HTTP header in git's environment, masked everywhere). No pull request is opened; one `sandbox_branch` audit record per run.
+
 **Order.** Read-only STLC agents (requirements review, test planning, test data design, test review, environment intelligence, execution optimizer, defect triage, test closure) can follow Phase 2 once their dependencies ship; agents that change files (service virtualization, defect reproduction, baseline and retirement proposals, feature files) need the Phase 4 proposal pipeline. Which STLC agents to build first is decided per client from the same value-and-risk view used for the core agents; each is released only through the agent release gates (§7).
 
 ### Shared capability: knowledge index

{pyplaykit-1.37.1 → pyplaykit-1.37.2}/docs/QE_AGENTS_GUIDE.md

@@ -113,6 +113,9 @@ agents:
 
 ## 3c. test-creation
 
+**Hands-on:** `pyplaykit docs test-creation` walks through it with sample CSVs
+(`pyplaykit docs test-creation --save-examples`).
+
 Turns your test cases (a CSV in the repository) into PyPlayKit tests, in your project's own layout, reusing your
 page objects and the framework's libraries:
 
@@ -142,8 +145,42 @@ credentials only from environment variables. Without `Type` the agent infers it
 **Output:** `test_creation.patch` (apply with `git apply`), `test_creation.md` (each test case to its test, page
 objects changed, locators to verify) and `test_creation.json`. Without a model every test case becomes a skeleton
 with its steps as comments. Generated code is checked with the test-review rules and problems are sent back to
-the model up to `fix_attempts` times. A sandbox branch (`agents/test-creation/<run id>`) follows in the next
-release (ADR-0019).
+the model up to `fix_attempts` times.
+
+**Feature branches in your sandbox repository** (ADR-0019 amendment 1). With `--autonomy branch` (and
+`agents.autonomy.max: branch`) the agent works on a fresh checkout of `agents.scm.sandbox.repo` instead of the
+current project:
+
+```bash
+export PYPLAYKIT_SANDBOX_GIT_TOKEN=...   # or put it in .env (gitignored); Contents read/write on that repo only
+pyplaykit agents run test-creation --autonomy branch \
+  --input '{"test_cases": "cases/checkout.csv", "project": "demo"}'
+```
+
+1. The host clones the repository (kept under `reports/agents/sandbox/`) and creates
+   `feature/test-creation/<csv name>-<UTC time>` from `origin/main`.
+2. Your CSV is copied to `test_data/testcases/` on that branch; the agent reads the sandbox repository's layout,
+   page objects, fixtures and markers and writes tests there (only `tests/`, `pages/`, `test_data/`).
+3. The new tests are collected with `pytest --collect-only` and device tests run in sim mode.
+4. The host commits as you (your git identity) and pushes **only the feature branch**. It never opens a pull
+   request and never touches `main`.
+5. You pull the branch, validate and edit the tests, and open the pull request to `main` when they work; the
+   output prints the GitHub page that opens it.
+
+```yaml
+agents:
+  autonomy: {max: branch}
+  scm:
+    sandbox:
+      repo: https://github.com/ShanKonduru/pyplaykit-sandbox.git
+      base: main
+      branch_prefix: feature/test-creation
+      push: true                     # false: commit in the local worktree only
+      token_env: PYPLAYKIT_SANDBOX_GIT_TOKEN
+```
+
+The exit code is 0 only when every test was generated, the branch was committed (and pushed) and validation
+passed. Every run is recorded in the audit log (`sandbox_branch`), without credentials.
 
 ```yaml
 agents:

pyplaykit-1.37.2/docs/TEST_CREATION_PLAYGROUND.md

@@ -0,0 +1,269 @@
+# Test creation playground: from test-case CSVs to PyPlayKit tests
+
+A hands-on walkthrough of the `test-creation` agent. You start with no model and no git, then add a model, then
+let the agent commit to a feature branch in your sandbox repository. Every command here works on a fresh
+`pip install pyplaykit` + `pyplaykit init` project, using the sample CSVs that ship with the package.
+
+Background and every setting: `pyplaykit docs agents` (section 3c). Design decisions: ADR-0019.
+
+---
+
+## 1. Set up a playground project (2 minutes)
+
+```bash
+python -m venv .venv
+source .venv/bin/activate              # Windows: .venv\Scripts\activate
+pip install "pyplaykit[agents]"        # the agents extra validates agent input and output
+playwright install chromium
+
+mkdir tc-playground && cd tc-playground
+pyplaykit init                         # multi-project layout; the project is "demo"
+git init -q && git add -A && git commit -qm "scaffold"   # optional: lets you apply the patch with git
+```
+
+Turn the agent on in `config/config.yaml` (the `agents:` block is already there, switched off):
+
+```yaml
+agents:
+  enabled: true
+  enabled_agents: [test-creation]
+```
+
+Check: `pyplaykit agents list` shows `test-creation`.
+
+## 2. Get the sample CSVs
+
+```bash
+pyplaykit docs test-creation --save-examples          # into test_data/testcases/ (never overwrites)
+pyplaykit docs test-creation --save-examples cases    # or any folder you like
+```
+
+| File | What it shows |
+|---|---|
+| `ui_login.csv` | UI only: login and cart against the scaffolded `LoginPage` (SauceDemo) |
+| `api_posts.csv` | API only, **no Type column**: the type is inferred from the steps (`GET /posts`, status codes) |
+| `db_employees.csv` | Database: row counts and migrated values in the scaffolded `test_data/demo.db` |
+| `device_thermostat.csv` | Device: the simulated `demo-thermostat` from `config/devices.yaml` |
+| `mixed_checkout.csv` | One file with UI, API, DB and device test cases |
+| `steps_per_row.csv` | One row per step: the test case id is filled only on its first row |
+| `custom_columns.csv` | Your own column names, mapped in config (see section 6) |
+
+## 3. First run: skeletons, no model, no git
+
+```bash
+pyplaykit agents run test-creation \
+  --input '{"test_cases": "test_data/testcases/ui_login.csv", "project": "demo", "summarise": false}'
+```
+
+`"summarise": false` means "do not call a model": every test case becomes a **skeleton**, a test whose docstring
+carries the test case id, objective and steps, with the steps as comments and `pytest.skip(...)` in the body. The
+run reports `partial` and exits 1 because the tests still need bodies. That is expected here.
+
+Windows PowerShell: put the JSON in a file and pass the path, e.g. `--input run.json`.
+
+**Where the output is:** `reports/agents/runs/<run id>/artifacts/`
+
+| File | Read it for |
+|---|---|
+| `test_creation.md` | Each test case → test function and file; page objects changed; locators to verify; notes |
+| `test_creation.patch` | The change itself; apply it with `git apply` |
+| `test_creation.json` | The same report for tools and pipelines (also the full new file contents) |
+
+Find the newest run and apply it:
+
+```bash
+ls -t reports/agents/runs | head -1                    # the run id
+git apply reports/agents/runs/<run id>/artifacts/test_creation.patch
+pytest tests/demo/functional/ui --collect-only -q      # the generated tests are collected
+```
+
+Without git: copy the files from `test_creation.json` (`content`), or use the branch mode in section 5.
+
+## 4. With a model: real test bodies
+
+Add a model to `config/config.yaml`. Any of the three adapters works; a local Ollama keeps everything on your
+machine:
+
+```yaml
+agents:
+  enabled: true
+  enabled_agents: [test-creation]
+  llm:
+    adapter: openai_compat               # anthropic | openai_compat | ollama
+    endpoint: http://localhost:11434/v1  # Ollama's OpenAI-compatible endpoint
+    tiers:
+      strong: { model: "qwen3:14b" }     # test-creation asks for the "strong" tier
+    providers:
+      default: { hosting: self_hosted }
+```
+
+For a hosted model set `api_key: ${SECRET:PYPLAYKIT_LLM_API_KEY}` and allow what may be sent with
+`agents.llm.egress` (see `pyplaykit docs agents`, section 2, and `pyplaykit docs secrets-governance`).
+
+Then drop `"summarise": false`:
+
+```bash
+# UI: reuses LoginPage; members it needs but cannot find are added, new locators are marked UNVERIFIED
+pyplaykit agents run test-creation --input '{"test_cases": "test_data/testcases/ui_login.csv", "project": "demo"}'
+
+# API: ApiClient + ResponseValidator; types inferred (the report says so)
+pyplaykit agents run test-creation --input '{"test_cases": "test_data/testcases/api_posts.csv", "project": "demo"}'
+
+# Database: db_client / DbValidationHelper against the demo database
+pyplaykit agents run test-creation --input '{"test_cases": "test_data/testcases/db_employees.csv", "project": "demo"}'
+
+# Device: dut, requires_dut("demo-thermostat"), sim_clock
+pyplaykit agents run test-creation --input '{"test_cases": "test_data/testcases/device_thermostat.csv", "project": "demo"}'
+
+# Everything at once: one module per type and feature
+pyplaykit agents run test-creation --input '{"test_cases": "test_data/testcases/mixed_checkout.csv", "project": "demo"}'
+```
+
+Narrow a run:
+
+```bash
+# only some test cases
+pyplaykit agents run test-creation --input '{"test_cases": "test_data/testcases/ui_login.csv", "ids": ["TC-UI-001", "TC-UI-004"]}'
+
+# force a type for every selected test case (overrides the Type column and inference)
+pyplaykit agents run test-creation --input '{"test_cases": "test_data/testcases/api_posts.csv", "type": "api"}'
+
+# put every test in one feature folder
+pyplaykit agents run test-creation --input '{"test_cases": "test_data/testcases/mixed_checkout.csv", "feature": "smoke_pack"}'
+```
+
+Generated code is checked with the test-review rules (assertions, no sleeps, no raw selectors in tests, no
+hardcoded URLs or credentials, no print). Problems go back to the model up to `fix_attempts` times; a test case
+that still fails is written as a skeleton and listed in the report.
+
+Run what was generated:
+
+```bash
+git apply reports/agents/runs/<run id>/artifacts/test_creation.patch
+pytest tests/demo/functional/api -v
+pytest tests/demo/functional/device -v                  # simulated thermostat, no hardware
+pytest tests/demo/functional/ui -v --pyplaykit-headed   # export PYPLAYKIT_TEST_USERNAME / _PASSWORD first
+```
+
+## 5. Feature branches in your sandbox repository
+
+Instead of a patch, the agent can commit the tests to a new feature branch of a repository you choose and push
+that branch. It never opens a pull request and never touches `main`: you validate, edit, and open the PR.
+
+```yaml
+agents:
+  enabled: true
+  enabled_agents: [test-creation]
+  autonomy:
+    max: branch                           # allow the branch level
+  scm:
+    protected_branches: [main, master, "release/*"]
+    sandbox:
+      repo: https://github.com/<you>/<your-sandbox>.git   # a pyplaykit init project
+      base: main
+      branch_prefix: feature/test-creation
+      push: true                          # false: commit in a local worktree only
+      token_env: PYPLAYKIT_SANDBOX_GIT_TOKEN
+```
+
+The token: a GitHub fine-grained token with **Contents: read and write** on that repository only. Put it in the
+environment or in `.env` (gitignored) as `PYPLAYKIT_SANDBOX_GIT_TOKEN=...`. Git receives it only as an HTTP
+header; it never appears in URLs, logs, the audit record or test runs.
+
+```bash
+pyplaykit agents run test-creation --autonomy branch --project demo \
+  --input '{"test_cases": "test_data/testcases/ui_login.csv", "project": "demo"}'
+```
+
+```text
+succeeded: 4 test case(s) -> 4 generated test(s), 0 skeleton(s) in 2 file(s); 1 page object(s) changed, ...
+sandbox: pushed (https://github.com/<you>/<your-sandbox>.git, base main)
+  branch      feature/test-creation/ui-login-20261011-155901 @ f2a9a7fead59
+  validation  passed
+  next        validate and edit the tests, then open the pull request: https://github.com/<you>/<your-sandbox>/compare/main...feature/test-creation/ui-login-20261011-155901?expand=1
+```
+
+What happened:
+
+1. The repository was cloned to `reports/agents/sandbox/` and `feature/test-creation/<csv>-<UTC time>` created
+   from `origin/main`.
+2. Your CSV was copied to `test_data/testcases/` on the branch, so the test cases travel with the tests.
+3. The agent read **that** repository's layout, page objects, fixtures and markers, and wrote only under
+   `tests/`, `pages/` and `test_data/`.
+4. The new tests were collected (`pytest --collect-only`), device tests ran in sim mode.
+5. The commit is yours (your git name and email); only the feature branch was pushed.
+
+Then, on your machine:
+
+```bash
+git fetch origin
+git switch feature/test-creation/ui-login-20261011-155901
+pytest tests/demo/functional/ui -v        # validate; edit tests and verify UNVERIFIED locators
+git commit -am "Verify locators" && git push
+# open the pull request to main (the link printed above)
+```
+
+The CSV can also be outside the project: an absolute path or a path relative to the project is copied in.
+
+## 6. Your own CSVs
+
+Columns are found by name; case, spaces, `-` and `_` are ignored.
+
+| Field | Required | Column names it recognises |
+|---|---|---|
+| id | yes | Test Case ID, TC ID, TC No, ID, Test ID, Case ID |
+| name | yes | Test Case Name, Name, Title, Summary, Scenario |
+| steps | yes | Test Steps - Actions, Steps, Actions, Step, Procedure |
+| expected | | Expected Steps, Expected Result(s), Expected, Expected Outcome |
+| description | | Test Case Description, Description, Details |
+| objective | | Test Objective, Objective, Purpose, Goal |
+| type | | Type, Test Type, Category, Layer: `UI`/`Web`, `API`/`REST`, `DB`/`Database`/`SQL`, `Device`/`Embedded`/`IoT` |
+| feature | | Feature, Module, Component, Area (the folder under `functional/<type>/`) |
+| tags | | Tags, Labels, Markers, Suite: offered as markers; only markers registered in `pytest.ini` are kept |
+| preconditions, test data, priority | | Preconditions, Test Data, Priority / Severity: given to the model |
+
+**Steps:** numbered lines in one cell (`1. Open ...` newline `2. Enter ...`) pair with numbered expected
+results; or one row per step with the id only on the first row (`steps_per_row.csv`).
+
+**Other column names:** map them in config. `custom_columns.csv` uses `Why` and `Outcome`:
+
+```yaml
+agents:
+  test_creation:
+    columns: {objective: "Why", expected: "Outcome"}
+```
+
+**Excel:** save the sheet as CSV (UTF-8). Cells with line breaks are fine; Excel quotes them.
+
+**Type inference** (no Type column): `GET /path`, status codes, payloads → API; `SELECT`, tables, row counts →
+DB; device, firmware, console, sensor → device; click, page, button, login → UI. The report marks inferred types;
+set the column or `"type"` when it guesses wrong.
+
+## 7. Settings worth knowing
+
+```yaml
+agents:
+  test_creation:
+    max_test_cases: 25     # per run; a warning says how many were left out
+    batch_size: 5          # test cases per model request
+    fix_attempts: 2        # rounds of review findings sent back to the model
+    columns: {}            # field -> your column name
+```
+
+## 8. Troubleshooting
+
+| You see | Do this |
+|---|---|
+| `agents are disabled` / `not in enabled_agents` | `agents.enabled: true` and `enabled_agents: [test-creation]` |
+| `validating agent input needs 'jsonschema'` | `pip install "pyplaykit[agents]"` |
+| `cannot read '...csv' through read_source` | The path is relative to the project root (patch mode); in branch mode local files are copied in |
+| `The test-case file cannot be used` | A required column (id, name, steps) is missing; rename it or map it in `columns` |
+| Every test is a skeleton, `provider_failure` | No model reachable: check `agents.llm`; or use `"summarise": false` on purpose |
+| `The target project is unclear` | Multi-project layout with several projects: pass `"project": "<name>"` |
+| `sandbox: git clone failed` | Check `scm.sandbox.repo` and the token (`Contents: read and write` on that repository) |
+| `branch ... is protected` | `branch_prefix` matches `protected_branches`; use e.g. `feature/test-creation` |
+| `validation failed` (collect) | The branch is still pushed; fix the import or fixture on the branch and push |
+| Branch mode not used | Needs `--autonomy branch` **and** `agents.autonomy.max: branch` |
+
+Exit codes: `0` every test generated (and, in branch mode, committed, pushed and validated); `1` partial results
+(skeletons, validation or push failed: see the output); `2` the run could not start (config, input, sandbox).

{pyplaykit-1.37.1 → pyplaykit-1.37.2}/docs/adr/0019-test-creation-from-test-cases-on-sandbox-branches.md

@@ -87,3 +87,35 @@
   `recommend` (patch only).
 - 🔧 Plan: EP-AG12 gains "Test creation from test cases (Python)"; EP-AG10's worktree and agent-branch stories are
   delivered with it for `test-creation`; recipes (EP-AG12 F1) later absorb the per-type rules.
+
+## Amendment 1 (2026-10-11): feature branches in a separate sandbox repository
+
+Decided with the team after the first release of the agent. Decision 5 changes as follows; the rest stands.
+
+- **Where.** The branch is created in the repository named by `agents.scm.sandbox.repo` (a URL or a local path;
+  e.g. `https://github.com/ShanKonduru/pyplaykit-sandbox.git`), from `origin/<base>` (`base`, default `main`). The
+  host keeps a clone under `reports/agents/sandbox/` and adds a worktree per run. Without `repo`, the project's own
+  repository is used as before (from `base`, else `HEAD`).
+- **What the agent reads.** The run's read tools point at the sandbox worktree, so the agent learns *that*
+  repository's layout, page objects, fixtures and markers. A local test-case file named in the input is copied to
+  `test_data/testcases/` on the branch and committed with the tests (traceability).
+- **Branch name.** `<branch_prefix>/<test-case file>-<UTC time>` (default prefix `feature/test-creation`); a name
+  matching `agents.scm.protected_branches` or `base` is refused.
+- **Push.** With `push: true` (default) the host pushes the feature branch only: never the base branch, never
+  forced, and never a pull request (`agents.scm.allow_pr` stays false). The engineer pulls the branch, validates
+  and edits the tests offline, and opens the pull request to `base` when they work. With `push: false` the commit
+  stays in the local worktree.
+- **Credentials.** A token in the variable named by `token_env` (default `PYPLAYKIT_SANDBOX_GIT_TOKEN`; from the
+  environment, else the project's `.env`) reaches git only as an HTTP header in git's environment, scoped to the
+  remote's host: never in a URL, argument, file, log or the audit record, and masked in every message. Test runs
+  never receive it (the scrubbed environment refuses `PYPLAYKIT_SANDBOX_*`). Without it git uses its own
+  credential helpers.
+- **Validation.** `pytest --collect-only` on the new test modules, and device tests in `sim` mode (judged by their
+  JUnit results, so a repository's session gates do not mask passing tests). A failed check does not block the
+  commit: the branch is for the engineer to fix, and the result, the commit message and the CLI exit code (1) say
+  so. Collection errors are not sent back to the model in v1 (decision 6's fix loop covers syntax and the
+  test-review rules).
+- **Audit.** One `sandbox_branch` record per run: repository (without credentials), base, branch, commit, pushed,
+  files, validation status.
+- **Writes.** Only the patch's paths under `writable` (default `tests/`, `pages/`, `test_data/`) and the staged
+  input file are committed; a patch with any other path is refused and the branch is discarded.

{pyplaykit-1.37.1 → pyplaykit-1.37.2}/pyplaykit/agents/builtin/test_creation/agent.py

@@ -10,8 +10,9 @@
    objects in the page directory. Only ``tests/``, the page directory and ``test_data/`` may be written (F5b).
 4. Validate statically (syntax, the test-review rules) and send problems back to the model up to
    ``fix_attempts``; a test case that still fails becomes a traceable skeleton.
-5. Deliver ``test_creation.patch``, ``test_creation.md`` and ``test_creation.json``. The sandbox branch
-   (US-AG12.3.3) applies the same files on ``agents/test-creation/<run id>``.
+5. Deliver ``test_creation.patch``, ``test_creation.md`` and ``test_creation.json``. With autonomy ``branch``
+   the host runs the agent on a sandbox worktree and commits the patch to a feature branch
+   (:mod:`pyplaykit.agents.sandbox`, US-AG12.3.3).
 
 Without a model every test case becomes a skeleton (docstring with the steps, ``pytest.skip``) and the run is
 ``partial``.
@@ -449,7 +450,8 @@ class TestCreationAgent:
         tests = report["tests"]
         lines = [f"# Generated tests from `{report['source']}`", "",
                  f"Layout: {report['layout']}-project" + (f", project `{report['project']}`" if report["project"] else ""),
-                 "", "Apply with `git apply test_creation.patch` (or use the sandbox branch), then review.", "",
+                 "", "Apply with `git apply test_creation.patch` (with `--autonomy branch` it is committed to a sandbox "
+                 "feature branch), then review.", "",
                  "| Test case | Type | Test | Notes |", "|---|---|---|---|"]
         for test in tests:
             notes = []

{pyplaykit-1.37.1 → pyplaykit-1.37.2}/pyplaykit/agents/cli.py

@@ -33,7 +33,9 @@ from pyplaykit.agents.output import encode
 from pyplaykit.agents.policy import agent_policy_report, resolve_agent_policy
 from pyplaykit.agents.registry import BUILTIN_DIR, DiscoveredAgent, DiscoveryReport
 from pyplaykit.agents.reports import aggregate
-from pyplaykit.agents.runner import RunRequest, effective_autonomy, mcp_status
+from pyplaykit.agents import sandbox
+from pyplaykit.agents.runner import RunRequest, autonomy_allows, effective_autonomy, mcp_status
+from pyplaykit.agents.stacks.environment import execution_environment
 from pyplaykit.agents.audit import DEFAULT_RETENTION_DAYS as AUDIT_RETENTION_DAYS
 from pyplaykit.agents.audit import chain_head, verify
 from pyplaykit.agents.enforcement import controls_by_class, scan_path
@@ -428,29 +430,86 @@ def _run(report: DiscoveryReport, section: dict[str, Any], args: argparse.Namesp
     except (OSError, ValueError) as exc:
         print(f"error: invalid --input: {exc}", file=out)
         return 2
+    project_root = config_dir.resolve().parent
+    policy = run_policy(section, base_dir=project_root, project=args.project)
+    box = None
+    try:
+        level = effective_autonomy(agent.manifest.autonomy_max, policy.autonomy_max, args.autonomy)
+    except ValueError:
+        level = "analyze"  # the runner reports the invalid level
+    if autonomy_allows(level, "branch"):
+        try:  # US-AG12.3.3: the agent works on a feature branch of the sandbox repository
+            box = sandbox.Sandbox(sandbox.SandboxConfig.from_section(section), project_root)
+            hint = Path(str(payload.get("test_cases") or args.agent)).stem
+            box.prepare(hint)
+            payload = box.stage_inputs(payload, project_root)
+        except (sandbox.SandboxError, OSError) as exc:
+            message = box.mask(str(exc)) if box else str(exc)
+            if box:
+                box.discard(message)
+            print(f"error: sandbox: {encode(message, 'terminal')}", file=out)
+            return 2
     request = RunRequest(agent=agent, project=args.project, input=payload, requested_autonomy=args.autonomy,
                          stack=args.stack, trace_context=trace_links(), trigger="cli", environment=args.env)
     try:
-        runner = agent_runner(section, config_dir, project=args.project)
+        runner = agent_runner(section, config_dir, project=args.project, tools_root=box.worktree if box else None)
     except ValueError as exc:  # agents.llm, agents.telemetry or the agents policy is invalid
+        if box:
+            box.discard(str(exc))
         print(f"error: {exc}", file=out)
         return 2
     try:
-        outcome = runner.run(request, run_policy(section, base_dir=config_dir.resolve().parent, project=args.project))
+        outcome = runner.run(request, policy)
     finally:
         runner.close()
     result = outcome.result
+    branch = None
+    if box:
+        env = execution_environment(section, config_dir=box.worktree / "config", environment=args.env).env
+        timeout = int((section.get("execution") or {}).get("timeout_s") or 900)
+        branch = sandbox.complete(box, result, runner.artifacts_dir or project_root / "reports/agents/runs",
+                                  agent=args.agent, env=env, timeout=timeout)
+        audit_log(config_dir).append("sandbox_branch", agent=args.agent, run_id=result.run_id,
+                                     **{k: v for k, v in branch.to_dict().items() if k not in ("validation",)},
+                                     validation=branch.validation.get("status"))
     if args.json:
-        print(json.dumps({"mcp_status": mcp_status(result), "result": to_dict(result), "events": outcome.events},
-                         indent=2, default=str), file=out)
+        print(json.dumps({"mcp_status": mcp_status(result), "result": to_dict(result), "events": outcome.events,
+                          **({"sandbox": branch.to_dict()} if branch else {})}, indent=2, default=str), file=out)
     else:
         print(f"{result.status}: {encode(result.summary, 'terminal')}", file=out)
         if result.error:
             print(f"  {result.error.category}: {encode(result.error.message, 'terminal')}", file=out)
         for warning in result.warnings:
             print(f"  warning: {encode(warning, 'terminal')}", file=out)
+        if branch:
+            _print_sandbox(branch, out)
+    if branch and (branch.status not in ("committed", "pushed") or branch.error
+                   or branch.validation.get("status") == "failed"):
+        return 1
     return 0 if result.status == "succeeded" else 1
 
 
+def _print_sandbox(branch: sandbox.SandboxOutcome, out: TextIO) -> None:
+    print(f"sandbox: {branch.status} ({branch.repo}, base {branch.base})", file=out)
+    if branch.branch:
+        print(f"  branch      {branch.branch}" + (f" @ {branch.commit[:12]}" if branch.commit else ""), file=out)
+    if branch.validation:
+        print(f"  validation  {branch.validation['status']}", file=out)
+        for name, check in branch.validation.items():
+            if isinstance(check, dict) and not check["passed"]:
+                print(f"    {name} failed (exit {check['exit_code']}):", file=out)
+                for line in check["output"].splitlines()[-10:]:
+                    print(f"      {encode(line, 'terminal')}", file=out)
+    if branch.pull_request_url:
+        print(f"  next        validate and edit the tests, then open the pull request: {branch.pull_request_url}",
+              file=out)
+    elif branch.worktree:
+        print(f"  worktree    {branch.worktree}", file=out)
+    if branch.error:
+        print(f"  error: {encode(branch.error, 'terminal')}", file=out)
+    for warning in branch.warnings:
+        print(f"  warning: {encode(warning, 'terminal')}", file=out)
+
+
 if __name__ == "__main__":  # pragma: no cover - python -m pyplaykit.agents.cli (CI templates)
     sys.exit(main())