lisa-harness-e2e 0.1.0__tar.gz

lisa_harness_e2e-0.1.0/.gitignore

@@ -0,0 +1,33 @@
+# Python
+__pycache__/
+*.pyc
+*.pyo
+*.egg-info/
+dist/
+build/
+
+
+# Virtual environments
+.venv/
+
+# IDE
+.vscode/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Testing
+.import_linter_cache/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.lisa-tmp/
+coverage/
+.coverage
+.coverage.*
+htmlcov/
+
+# Outputs
+artifacts/
+.e2e/

lisa_harness_e2e-0.1.0/PKG-INFO

@@ -0,0 +1,13 @@
+Metadata-Version: 2.5
+Name: lisa-harness-e2e
+Version: 0.1.0
+Summary: End-to-end scenario harness: drives a full Lisa runtime through pack workflows
+Requires-Python: >=3.11
+Requires-Dist: lisa-harness-kernel
+Requires-Dist: lisa-harness-runtime
+Requires-Dist: lisa-harness-sdk
+Requires-Dist: python-dotenv>=1.0
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: dev
+Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
+Requires-Dist: pytest>=8.0.0; extra == 'dev'

lisa_harness_e2e-0.1.0/README.md

@@ -0,0 +1,269 @@
+# e2etest — live end-to-end pack scenarios (`lisa-e2e`)
+
+`e2etest` drives a **real** `HostRuntime` with a **live model** through a
+scripted pack workflow, then judges the outcome from on-disk evidence. It is
+the layer above [`packstest`](../packstest/README.md):
+
+| Layer | Drives | Model | Answers |
+| --- | --- | --- | --- |
+| `packstest` evals | one tool in-process | none | "Does the tool honor its contract?" |
+| `lisa-e2e` scenarios | the whole runtime + `handle_message` | live | "Can Lisa actually complete the workflow?" |
+
+Use `packstest` for fast, deterministic regressions. Use `lisa-e2e` when the
+question is whether routing, handoffs, approvals, policy, memory, sandboxing,
+and the pack's tools work together for a real model.
+
+See the [documentation index](../docs/README.md) and [testing](../docs/TESTING.md).
+The workspace installs `lisa-e2e`; the framework's current publish workflow
+does not publish the `lisa-harness-e2e` distribution.
+
+## How a run works
+
+```mermaid
+flowchart LR
+  S[scenario.yml] --> V[packs venv<br/>editable installs]
+  S --> R[run dir<br/>isolated data dir + fixture repo]
+  V --> D[driver: build_runtime + handle_message]
+  R --> D
+  D -->|turns, /approve, /deny| D
+  D --> O[oracles over evidence]
+  O --> B[verdict.json + transcript.md]
+```
+
+1. **Packs venv** — `.e2e/cache/packs-venv-<hash>` holds editable installs of
+   kernel/sdk/mcp_server plus the scenario's `packs`, sourced from
+   [`packs.yml`](packs.yml). Editable means a pack fix is live on the next run.
+   It is rebuilt only when the pack set or source paths change.
+2. **Run dir** — `.e2e/runs/<scenario-id>/<UTC stamp>/` (gitignored). Contains
+   a fresh `data/` dir used as `LISA_DATA_DIR`, the fixture repo copied to
+   `workspace/<name>` with a git history and a bare `remote.git`, and any
+   `fixture.data_files` tree copied into the storage sandbox (`data/data/files`).
+   Only `config/models.yaml` and `.env` are read from the base data dir
+   (`~/.lisa`, override with `LISA_E2E_BASE_DATA_DIR`).
+3. **Driver** — boots the runtime in-process, sends each scripted turn, then
+   follows up until the turn finishes:
+   - approval pending → `/approve <id>` or `/deny <id>` per the scenario rules
+     (the real approval path, not a policy bypass);
+   - yield → `yield_reply`, only if `max_yield_replies` allows;
+   - pause/stop exit → `continue_message`, only if `max_continues` allows.
+4. **Oracles** — assertions over the thread log, memory DB, code-change
+   artifacts, runtime log, and git. They never trust the model's prose.
+5. **Bundle** — `verdict.json` (first failure first), `transcript.md`
+   (exchanges + per-turn receipts), `outcome.json`, and the whole `data/` dir.
+
+## Running the existing scenarios
+
+From the framework checkout:
+
+```bash
+uv sync --frozen --extra dev            # once
+uv run --frozen lisa-e2e run sdk/packs/recovery/tests/e2e/skip_all.scenario.yml
+```
+
+This bundled scenario needs the configured base roster/provider **and**
+external bash/bwrap/markdown/personas sources named in its `packs` list.
+Inspect the scenario and `packs.yml` before running; a scenario located
+in this checkout is not necessarily self-contained.
+
+| Scenario | Workflow |
+| --- | --- |
+| `coding/review-fix-commit` | knowledge-capturing review → worktree → patch → required check → commit, verified by re-running the tests outside Lisa |
+| `coding/feature-design-implement` | product review → feature design doc → multi-file implementation in a worktree → required check → commit, judged by hidden acceptance tests |
+| `coding/roadmap-chunk-implement` | open-ended product review → roadmap → model-proposed design overridden by operator pushback → terse "start the first chunk" that must recover the agreed design → checked commit, same hidden acceptance tests |
+| `finances/categorize-and-close` | import bank + card exports → survey + proposed categories → user decisions (incl. a new category) → final ledger, verified by re-reading the ledger outside Lisa |
+
+Options:
+
+- `--rebuild-venv` — force a packs-venv rebuild (after editing `packs.yml` or a
+  pack's dependencies; source edits do not need it).
+- `--runs-dir DIR` — write bundles elsewhere.
+
+Exit codes: `0` pass, `1` an oracle failed, `2` setup error (bad scenario,
+missing base `models.yaml`, venv build failure).
+
+A run takes minutes and costs real tokens. Live runs vary between attempts, so
+confirm a fix with more than one passing run before calling it done.
+
+## Debugging a failed run
+
+1. Read `verdict.json` → `first_failure` (oracle, args, detail).
+2. Read `transcript.md` for the exchange sequence, approvals, and the
+   per-turn receipt list.
+3. Go to the raw evidence in `data/`:
+   - `logs/runtime.jsonl` — warnings/errors (operation timeouts, validation
+     failures, truncation retries);
+   - `data/threads/*.current.jsonl` — every turn's transcript, tool calls with
+     exact arguments, `turn_input` requests, provenance;
+   - `state/code-change-artifacts.json` — WorkProduct state, check records;
+   - `data/memory.db` — what memory actually stored.
+4. Reproduce the failing piece **deterministically** before fixing: replay the
+   exact tool arguments from the thread against `git`/the tool, or boot the
+   runtime and call the pool directly (no model, seconds):
+
+   ```python
+   rt = build_runtime(host_kind="e2e", data_dir=..., session_context=..., in_process_mcp=True)
+   await rt.start()
+   await rt.resources.mcp_tool_pool.call_tool("memory.add", {...})
+   ```
+
+5. Fix the root cause (pack, kernel, or runtime), add a regression test at the
+   lowest layer that reproduces it, then rerun the scenario.
+
+Never loosen an oracle to get green. If the harness mis-modelled a contract,
+fix the harness and add a test in [`tests/`](tests).
+
+## Writing a scenario
+
+Scenarios live **in the pack**, next to its tests:
+
+```text
+<pack>/tests/e2e/
+  conftest.py                 # collect_ignore_glob = ["fixtures/*"]
+  <name>.scenario.yml
+  session_context.md          # template; ${...} placeholders are substituted
+  fixtures/<repo>/            # optional seed repo (planted defect, sample data)
+```
+
+`conftest.py` keeps pytest from collecting fixture code (which is deliberately
+broken). Give fixture repos a realistic `.gitignore`.
+
+### Scenario file reference
+
+```yaml
+id: <pack>/<workflow>            # bundle dir name; defaults to the file stem
+session_context: session_context.md
+fixture:
+  repo: fixtures/<repo>          # optional; becomes a git repo with a bare origin
+  data_files: fixtures/<tree>    # optional; copied into <data_dir>/data/files
+packs: [lisa-pack-...]           # keys of e2etest/packs.yml installed into the venv
+model:
+  script: model/<name>.py        # optional; scripted model instead of the live LLM
+policy: policy.yaml              # optional; default is empty `mode: ask`
+env: {NAME: value}               # optional extra process env
+turns:                           # user messages, in order
+  - "..."
+approvals:
+  approve: [tool_id, ...]        # "*" approves everything not denied
+  deny: [tool_id, ...]           # unlisted targets are denied
+  max_decisions: 20
+budgets:
+  max_messages: 16               # every message sent, including /approve
+  max_continues: 0               # replies to pause exits
+  max_yield_replies: 0           # replies to model questions
+  wall_clock_s: 1800
+continue_message: continue
+yield_reply: Proceed with your best judgment.
+oracles:
+  <phase>:                       # free-form grouping shown in the verdict
+    - <oracle_name>: {args}
+```
+
+`session_context.md` placeholders: `${WORKSPACE}` (fixture repo path),
+`${WORKSPACE_PARENT}` (where coding worktrees go), `${RUN_DIR}`. Mirror the
+real session the pack targets, but name it uniquely (`name: e2e-<pack>`) so
+live policy rules keyed on a workspace id do not apply.
+
+### Scripted model (token-free repros)
+
+`model.script` replaces every provider gateway with a Python module, so a
+scenario replays a fixed tool sequence in seconds at zero token cost.
+Everything below the model (routing, policy, approvals, packs, sandbox,
+persistence) is production code. Use it to pin down a bug a live run
+surfaced, not to judge model behavior.
+
+The module defines `respond(turn: ScriptTurn) -> Reply` and, optionally,
+`auxiliary(turn)` for tool-less requests (distillers, classifiers, titles).
+Those otherwise receive `{}` or empty text, which some operations log as
+errors. `ScriptTurn` exposes `tool_names`, `user_text`, `count(name)`, and
+`last(name)`. `last(name).data` is the JSON object inside that tool's
+result, so ids and digests flow from real tool output. Return
+`Reply.call(tool, **args)`, `Reply.calls(...)`, or `Reply.text(...)`; raise
+`ScriptError` when the transcript is not what the script expects. Add
+`"model/*"` to the pack's `conftest.py` `collect_ignore_glob`. See
+`coding/tests/e2e/edit_after_check.scenario.yml` in the coding pack.
+
+### Oracle reference
+
+Conversation hygiene:
+
+| Oracle | Args | Passes when |
+| --- | --- | --- |
+| `not_aborted` | — | no budget was exceeded |
+| `no_turn_errors` | — | no exchange errored |
+| `no_operator_exit` | — | no pause/stop/terminate exit |
+| `no_unanswered_yield` | — | the conversation did not end on a question |
+| `no_runtime_errors` | `ignore: [regex]` | no error-level record in `runtime.jsonl` |
+| `max_exchanges` | `n` | at most `n` messages were sent |
+
+Capability and knowledge:
+
+| Oracle | Args | Passes when |
+| --- | --- | --- |
+| `receipt` | `capability` (id or list), `min` | ≥ `min` effect receipts for those capabilities |
+| `memory_recorded` | `mentions_any`, `min` | ≥ `min` stored memories mention any term |
+
+Code change (coding pack WorkProduct + git; needs a `fixture.repo`):
+
+| Oracle | Args | Passes when |
+| --- | --- | --- |
+| `workproduct_committed` | — | an artifact has a `commit_sha` |
+| `branch_prefix` | `prefix` | the committed branch starts with `prefix` |
+| `main_untouched` | — | local and remote `main` equal the fixture's initial commit |
+| `diff_touches` | `paths`, `exact`, `allow` | the commit changes `paths` (and nothing outside `paths` + `allow` when `exact`) |
+| `checks_current` | — | every required check passed at the artifact revision and committed tree |
+| `worktree_clean` | — | the owned worktree is clean and at the commit |
+| `baseline_fails` | `argv`, `timeout_s` | `argv` fails on the fixture baseline (the defect is real) |
+| `independent_check` | `argv`, `timeout_s`, `restore` | `argv` passes on a fresh checkout of the commit, outside Lisa, with `restore` paths reset to the fixture baseline |
+
+`baseline_fails` and `independent_check` run with cwd at the checkout and
+expand `${SCENARIO_DIR}` and `${RUN_DIR}` in `argv`, so hidden acceptance
+tests can live beside the scenario where the model cannot see or edit them.
+
+Storage sandbox (packs whose work product lives under `data/files`):
+
+| Oracle | Args | Passes when |
+| --- | --- | --- |
+| `data_check` | `argv`, `timeout_s` | `argv` exits 0 with cwd at the storage root; `${SCENARIO_DIR}`, `${FILES}`, `${RUN_DIR}` expand |
+
+An unknown oracle name fails the run, so typos surface immediately.
+
+### Design rules for good scenarios
+
+- **Assert outcomes, not prose.** Prefer receipts, stored records, git state,
+  and an independent check over text matching.
+- **Prove the fixture.** Pair every "fixed" oracle with a baseline oracle that
+  shows the starting state was actually broken.
+- **Allow tests, judge with the originals.** A fix may add regression tests:
+  list test files under `diff_touches.allow` and pair it with an
+  `independent_check` that `restore`s them, so edited tests cannot mask an
+  unfixed defect.
+- **Verify outside Lisa.** End-state checks (`independent_check`, a re-read of
+  the ledger/file) must not reuse the tool path under test.
+- **Exercise approvals for real.** Approve the effectful tools the workflow
+  needs; deny the ones it must not reach (e.g. `coding_push`).
+- **Default to strict budgets.** `max_continues: 0` and `max_yield_replies: 0`
+  make stalls and capability gaps fail loudly instead of being papered over.
+- **Keep fixtures tiny and deterministic.** Stdlib-only test commands, no
+  network, one planted defect per workflow.
+- **Judge features with hidden tests.** For feature work, keep the acceptance
+  suite outside the fixture (`${SCENARIO_DIR}/acceptance/`) and pair a
+  `baseline_fails` with an `independent_check` on it.
+
+### Adding a new pack to the harness
+
+1. Add the pack's source path to [`packs.yml`](packs.yml) (relative to that
+   file). The `packs` list in a scenario may only name keys from it.
+2. If the scenario needs an assertion the oracle table lacks, add it to
+   [`oracles.py`](src/e2etest/oracles.py) with the `@oracle("name")`
+   decorator and a test in [`tests/test_harness.py`](tests/test_harness.py)
+   that covers both pass and fail.
+
+## Gotchas
+
+- The coding pack only runs checks under a bwrap floor with
+  `execution: sandbox` and `persistence: direct-write`, and the worktree parent
+  (`${WORKSPACE_PARENT}`) must be bound `rw`.
+- The base `.env` is loaded without overriding the process env; location
+  overrides (`LISA_PROFILE`, `LISA_THREAD_DIR`, `LISA_SESSION_CONTEXT_PATH`,
+  `LISA_MCP_LOG_PATH`) are cleared so a run can never touch live state.
+- Run one scenario per process; the runtime keeps process-global registries.

lisa_harness_e2e-0.1.0/packs.yml

@@ -0,0 +1,17 @@
+# Pack sources installed (editable) into the e2e packs venv.
+# Paths are relative to this file. Editable installs mean a pack fix is live on
+# the next scenario run without a reinstall.
+framework:
+  - ../kernel
+  - ../sdk
+  - ../mcp_server
+packs:
+  lisa-pack-bash: ../../lisa-packs/bash
+  lisa-pack-bwrap: ../../lisa-packs/bwrap
+  lisa-pack-coding: ../../packs-wip/coding
+  lisa-pack-finances: ../../packs-wip/finances
+  lisa-pack-markdown: ../../lisa-packs/markdown
+  lisa-pack-personas: ../../lisa-packs/personas
+  lisa-pack-recovery: ../sdk/packs/recovery
+  lisa-pack-search: ../../lisa-packs/search
+  lisa-pack-serper: ../../lisa-packs/serper

lisa_harness_e2e-0.1.0/pyproject.toml

@@ -0,0 +1,41 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/e2etest"]
+
+[project]
+name = "lisa-harness-e2e"
+version = "0.1.0"
+description = "End-to-end scenario harness: drives a full Lisa runtime through pack workflows"
+requires-python = ">=3.11"
+
+dependencies = [
+    "lisa-harness-kernel",
+    "lisa-harness-sdk",
+    "lisa-harness-runtime",
+    "python-dotenv>=1.0",
+    "pyyaml>=6.0",
+]
+
+[project.scripts]
+lisa-e2e = "e2etest.cli:main"
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0.0",
+    "pytest-asyncio>=0.23.0",
+]
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+testpaths = ["tests"]
+
+[tool.uv]
+package = true
+
+[tool.uv.sources]
+lisa-harness-kernel = { workspace = true }
+lisa-harness-sdk = { workspace = true }
+lisa-harness-runtime = { workspace = true }

lisa_harness_e2e-0.1.0/src/e2etest/__init__.py

@@ -0,0 +1 @@
+"""End-to-end scenario harness for Lisa packs (drives a full ``HostRuntime``)."""

lisa_harness_e2e-0.1.0/src/e2etest/bundle.py

@@ -0,0 +1,67 @@
+"""Write a run bundle: verdict, raw outcome, and a readable transcript."""
+
+from __future__ import annotations
+
+import json
+from dataclasses import asdict
+from pathlib import Path
+from typing import Any
+
+from e2etest.oracles import Evidence, OracleResult
+from e2etest.scenario import Scenario
+
+_TEXT_LIMIT = 4000
+
+
+def verdict(scenario: Scenario, evidence: Evidence, results: list[OracleResult]) -> dict[str, Any]:
+    failed = [r for r in results if not r.passed]
+    return {
+        "scenario": scenario.scenario_id,
+        "scenario_file": str(scenario.source),
+        "run_dir": str(evidence.paths.run_dir),
+        "passed": not failed,
+        "first_failure": asdict(failed[0]) if failed else None,
+        "exchanges": len(evidence.outcome.exchanges),
+        "aborted": evidence.outcome.aborted,
+        "results": [asdict(r) for r in results],
+    }
+
+
+def _transcript(evidence: Evidence) -> str:
+    lines = ["# Transcript", ""]
+    for ex in evidence.outcome.exchanges:
+        lines.append(f"## #{ex.index} {ex.kind} ({ex.duration_s}s)")
+        lines += ["", "```text", ex.message, "```", ""]
+        flags = []
+        if ex.approval:
+            flags.append(
+                f"approval: {ex.approval.get('target_id')} ({ex.approval.get('request_id')})"
+            )
+        if ex.operator_exit:
+            flags.append(f"operator_exit: {ex.operator_exit}")
+        if ex.yielded_question:
+            flags.append(f"yielded: {ex.yielded_question}")
+        flags += [f"error: {e}" for e in ex.errors]
+        lines += [f"- {f}" for f in flags]
+        text = ex.text if len(ex.text) <= _TEXT_LIMIT else ex.text[:_TEXT_LIMIT] + "\n…[truncated]"
+        lines += ["", text, ""]
+    lines += ["# Thread entries", ""]
+    for entry in evidence.thread_entries:
+        receipts = [r.get("capability_id") for r in entry.get("effect_receipts") or ()]
+        provenance = entry.get("provenance") or {}
+        lines.append(
+            f"- turn {entry.get('turn')} mode={entry.get('mode_id')} "
+            f"handoff={entry.get('handoff_to') or '-'} model={provenance.get('model_id', '-')} "
+            f"exit={provenance.get('operator_exit') or '-'} receipts={receipts}"
+        )
+    return "\n".join(lines) + "\n"
+
+
+def write_bundle(run_dir: Path, payload: dict[str, Any], evidence: Evidence) -> Path:
+    (run_dir / "outcome.json").write_text(
+        json.dumps(evidence.outcome.to_json(), indent=1), encoding="utf-8"
+    )
+    (run_dir / "transcript.md").write_text(_transcript(evidence), encoding="utf-8")
+    path = run_dir / "verdict.json"
+    path.write_text(json.dumps(payload, indent=1), encoding="utf-8")
+    return path

lisa_harness_e2e-0.1.0/src/e2etest/cli.py

@@ -0,0 +1,63 @@
+"""``lisa-e2e``: run one scenario end to end and write a verdict bundle."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from datetime import UTC, datetime
+from pathlib import Path
+
+from e2etest.bundle import verdict, write_bundle
+from e2etest.driver import run_conversation
+from e2etest.environment import prepare_run
+from e2etest.oracles import Evidence, evaluate
+from e2etest.packs_venv import ensure_packs_venv
+from e2etest.scenario import load_scenario
+
+_REPO_ROOT = Path(__file__).resolve().parents[3]
+_EXIT_FAIL = 1
+_EXIT_SETUP = 2
+
+
+def _parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(prog="lisa-e2e", description=__doc__)
+    sub = parser.add_subparsers(dest="command", required=True)
+    run = sub.add_parser("run", help="Run a scenario against a live model.")
+    run.add_argument("scenario", type=Path)
+    run.add_argument("--runs-dir", type=Path, default=_REPO_ROOT / ".e2e" / "runs")
+    run.add_argument("--rebuild-venv", action="store_true")
+    return parser
+
+
+def _run(args: argparse.Namespace) -> int:
+    scenario = load_scenario(args.scenario)
+    venv = ensure_packs_venv(
+        scenario.packs, _REPO_ROOT / ".e2e" / "cache", rebuild=args.rebuild_venv
+    )
+    stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
+    run_dir = (args.runs_dir / scenario.scenario_id.replace("/", "__") / stamp).resolve()
+    paths = prepare_run(scenario, run_dir, venv)
+    print(f"run dir: {run_dir}", flush=True)
+    outcome = run_conversation(scenario, paths, venv)
+    evidence = Evidence(paths, outcome)
+    payload = verdict(scenario, evidence, evaluate(scenario.oracles, evidence))
+    path = write_bundle(run_dir, payload, evidence)
+    for result in payload["results"]:
+        mark = "PASS" if result["passed"] else "FAIL"
+        first = result["detail"].splitlines()[0] if result["detail"] else ""
+        print(f"{mark} {result['phase']}.{result['name']}: {first}")
+    print(f"verdict: {'PASS' if payload['passed'] else 'FAIL'} → {path}")
+    return 0 if payload["passed"] else _EXIT_FAIL
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = _parser().parse_args(argv)
+    try:
+        return _run(args)
+    except (OSError, ValueError, RuntimeError) as exc:
+        print(f"setup error: {type(exc).__name__}: {exc}", file=sys.stderr)
+        return _EXIT_SETUP
+
+
+if __name__ == "__main__":
+    sys.exit(main())