agent-learning-kit 0.1.0__py3-none-any.whl

fi/alk/harness/ENVIRONMENT_CONFORMANCE.md

@@ -0,0 +1,127 @@
+# Environment packaging conformance
+
+The harness supports repository packaging incrementally while preserving one invariant: it may
+package submitted code and provide infrastructure, but it never rewrites or invents agent tools.
+
+## Case 1: repository supplies Compose
+
+ALK adopts the submitted Compose definition and:
+
+- renders it with Compose rather than reimplementing its schema;
+- rejects privileged/host escape features;
+- assigns an isolated project, host ports and volumes;
+- starts only default infrastructure services;
+- identifies opt-in agent/worker services;
+- discovers typed and generic TCP capabilities;
+- injects external/internal endpoints through existing configuration seams;
+- waits for Compose healthchecks, protocol-level readiness and a short startup-stability window;
+- performs lifecycle reset with only that project's volumes; and
+- tears down only that project.
+
+Conformance fixture: `tests/fixtures/harness_agents/voice_analytics_agent`.
+
+The real test starts two simultaneous copies even though the submitted Compose publishes fixed
+ClickHouse and Redis ports. It starts both unchanged worker images, proves each reaches its own
+seeded ClickHouse and Redis services, resets one project, verifies the other remains healthy, and
+cleans up both.
+
+## Case 2: repository supplies Dockerfile but no Compose
+
+ALK uses the submitted Dockerfile unchanged and generates a Compose adapter around it. A
+standalone runtime is valid and gets no invented infrastructure. When the contract declares
+dependencies, the adapter contains only those supported infrastructure services. Current managed
+templates are:
+
+| Engine | Connector | Reset mechanism |
+|---|---|---|
+| Postgres | `DATABASE_URL`/declared seam | project volume recreation + submitted SQL init |
+| ClickHouse | `CLICKHOUSE_URL`/declared seam | project volume recreation + submitted SQL init |
+| Redis | `REDIS_URL`/declared seam | project volume recreation |
+| MongoDB | `MONGODB_URL`/declared seam | project volume recreation |
+| Qdrant | `QDRANT_URL`/declared seam | project volume recreation |
+
+Multiple dependencies are included in one private network and injected into the submitted
+runtime. Generated services use no persisted resolved password, so the environment bundle passes
+secret scanning and content sealing.
+
+Conformance fixtures:
+
+- `voice_analytics_agent` with Compose removed: managed ClickHouse + Redis;
+- `voice_ledger_agent`: managed Postgres.
+
+An additional generated Dockerfile-only conformance agent uses MongoDB and Qdrant together. Its
+unchanged runtime writes both services, ALK destroys and recreates the project volumes, and a
+second runtime proves both stores begin empty. The real integration gate completes in about 41
+seconds on the current Docker Desktop host.
+
+The public `dograh-hq/dograh` root Compose is also exercised unchanged. Its default API, UI,
+pgvector/Postgres, password-protected Redis and MinIO services pass readiness and cleanup while
+profile-gated TURN/init/tunnel services remain dormant. See `ENVIRONMENT_VALIDATION_MATRIX.md`
+for evidence levels and the wider repository batch.
+
+Both real tests build and run the submitted Dockerfile, query submitted seed data from inside the
+runtime, emit application readiness evidence and clean up.
+
+### Official LiveKit agent validation
+
+The opt-in real-agent suite also validates three unmodified agents from the official
+`livekit/agents` examples repository at commit
+`da6af86ac640a3bc54585764e64321d7048c1c16`:
+
+- `drive_thru`: multi-tool ordering agent with in-process business state;
+- `frontdesk`: scheduling agent with an optional external Cal.com integration and source-owned
+  fallback; and
+- `hotel_receptionist`: larger multi-tool booking agent with SQLite-backed local state.
+
+For each repository ALK detects a Dockerfile-only standalone runtime, builds the upstream image,
+seals and verifies a portable environment bundle, starts the actual LiveKit worker with referenced
+credentials, checks runtime health, verifies the source fingerprint did not change, and tears the
+job down. No database or service is synthesized for these agents.
+
+Credential discovery includes requirements consumed internally by a detected connector SDK. For
+LiveKit that means `LIVEKIT_URL`, `LIVEKIT_API_KEY` and `LIVEKIT_API_SECRET` are requested before
+worker startup even when submitted code contains no direct environment read. Resolved values are
+passed through the child process environment and selected by name; they are never placed in
+Docker command arguments or persisted in a bundle.
+
+## Explicit failure behavior
+
+- A missing custom service/tool implementation is not generated.
+- An unsupported engine is not replaced with a different database.
+- External infrastructure with no Compose and no supported managed adapter fails before calls.
+- A required runtime with no Compose and no Dockerfile fails with an actionable packaging error.
+- Infrastructure/readiness failures are not agent evaluation results.
+
+## Running the matrix
+
+Unit and contract checks:
+
+```bash
+.venv/bin/pytest -q tests/test_harness_service_environments.py
+```
+
+Real Docker checks:
+
+```bash
+RUN_INTEGRATION=1 .venv/bin/pytest -q tests/test_harness_service_environments.py
+```
+
+Official LiveKit example build and bundle checks:
+
+```bash
+RUN_INTEGRATION=1 \
+LIVEKIT_EXAMPLES_ROOT=/path/to/livekit-agents/examples \
+.venv/bin/pytest -q tests/test_harness_livekit_examples.py -k builds
+```
+
+Add `LIVEKIT_EXAMPLES_START_WORKERS=1` and provide the three discovered `LIVEKIT_*` values to
+include real worker registration/readiness checks.
+
+The real suite pulls/builds container images, starts services, mutates state, resets and removes
+all test projects. It should run on CI with a dedicated Docker daemon, not a shared production
+host.
+
+Known protocols are checked semantically where possible (`/ping` for ClickHouse and `PING` for
+Redis). This prevents a container or briefly-open socket from being reported ready while database
+initialization is still restarting the service. Later liveness probes are point-in-time checks and
+do not reapply the startup window.

fi/alk/harness/HOW-IT-WORKS.md

@@ -0,0 +1,297 @@
+# How the harness actually works
+
+What happens between you typing a sentence and a graded result appearing. Written to be read
+alongside the code, so every claim below names the file it lives in.
+
+The shape is the same at every stage, and worth holding onto:
+
+> **A stage is a model session with a small set of tools and its instructions in a markdown file.
+> The model decides what to do; the tools do anything that must be exact and refuse anything that
+> must not happen. Nothing reaches disk except through a tool that checked it first.**
+
+There is no pipeline. Each stage is a conversation you can interrupt, correct, and resume.
+
+---
+
+## The pieces
+
+| Piece | Where | What it is |
+|---|---|---|
+| Stage | `session.py` | A live model session, held open across turns, emitting typed events |
+| Instructions | `skills/<stage>/SKILL.md` | How that stage works, in prose. Editable without touching code |
+| Tools | `tools.py`, `world/tools.py`, `scenario_tools.py`, `run/tools.py` | The exact half: they execute, validate, and refuse |
+| Artifacts | `artifacts/sessions/<id>/` | What each stage leaves behind for the next |
+| Conversation | `chat.py` | Holds one agent's journey through the stages |
+| Session | `sessions.py` | One conversation, one folder: chat and artifacts together |
+
+The artifacts, each the input to the next stage:
+
+```
+contract.json → world.sqlite + handlers/ + simulator_prompt.md + sub_goals.json
+             → scenarios/<name>/ (+ scenarios.json as the index) → runs.json
+```
+
+All of it, plus the conversation that produced it, lives in one folder per session. There is
+nothing held in memory that is not also on disk, so closing the page, restarting the server or
+coming back tomorrow all resume the same way: by reading the folder.
+
+---
+
+## 1. Reception — which agent is this?
+
+`reception.py`
+
+A stage with `Read`, `Glob`, `Grep` and one tool, `point_at_agent`. You say where your agent
+lives; it looks, confirms the path exists, picks a short name, and calls that tool.
+
+It looks from the **workspace root** (the directory holding your repos), not from inside
+`agent-learning-kit`, because the agent under test is almost never inside the harness.
+
+`point_at(name, path, kind)` refuses a path that does not exist, and refuses an unknown kind.
+`kind` selects an `AgentSource` from `sources.py` — `repo` (code on disk, gets file tools) or
+`spec` (a prompt and tool schema pasted in, gets no file tools). **A new kind of agent is one
+registration, not a new code path.**
+
+---
+
+## 2. Understand — what is this agent, verifiably?
+
+`understand.py`, `skills/understand-agent/SKILL.md`, gate in `tools.py`
+
+The session gets read-only file tools and one submission tool. It reads the agent's source and
+calls `submit_contract`.
+
+**The contract** (`contract.py`) is the anti-hallucination device for everything downstream:
+
+| Field | Why it matters |
+|---|---|
+| `tools[]` — name, `args`, `arg_types`, `arg_values`, description | The agent's action space. `arg_values` are the real permitted values — the menu, the enum, the lookup |
+| `hard_constraints[]` | Rules the agent must follow. Told to the agent under test, and graded by the judge |
+| `base_environment` | Its real starting data, reproduced row for row |
+| `real_use_cases[]` | What it is actually for |
+| `notes` | Free-form: whatever else the reader judged worth carrying forward |
+| `amendments[]` | Anything **not** read from source — see below |
+
+**How it is written:** `accept_contract` in `tools.py` validates before anything reaches disk. It
+refuses a contract with no tools, no use cases, duplicate tool names, types for arguments that do
+not exist, or — the one that mattered most in practice — *every* tool having no arguments, which
+means the arguments were read and then not recorded. Problems are returned **into the
+conversation**, so the model corrects and resubmits rather than a bad contract landing.
+
+**How it is changed later** (`amend.py`). The contract is not frozen, but every change is
+recorded with a reason in `amendments[]`, so months later you can still tell what came from the
+agent and what came from us:
+
+- `amend_contract` — let an argument accept a value it did not before
+- `add_rule` / `drop_rule` — a hard constraint the source did not state, or one misread
+- `fix_tool` — correct argument names, types, description, or remove a tool that does not exist
+
+Each demands a `why`. A contract that can be rewritten invisibly is no longer evidence.
+
+---
+
+## 3. Build — the world its tools run against
+
+`build.py`, `skills/build-environment/SKILL.md`, tools in `world/tools.py`
+
+**This is the part that makes the whole thing worth doing.** The repository's own services,
+migrations, seed process and tool code run in an isolated environment. A call for something that
+is not there is refused by the submitted implementation, not by a mock or a rewritten handler.
+
+It builds three things, all shared by every scenario: **the world**, **the simulator prompt** for
+a conversational agent, and **the sub-goal catalogue**. The stage has sixteen tools and no file
+access at all:
+
+| Tool | Does |
+|---|---|
+| `create_schema` | Run the CREATE TABLE statements |
+| `seed` | Insert rows — the agent's real catalogue |
+| `change_data` | One UPDATE or DELETE, for fixing a row put in wrong |
+| `adopt_tool` | Bind and smoke-test the agent's own implementation |
+| `write_env_file` / `run_env_command` | Container orchestration only; never agent behavior |
+| `run_tool` | Call a defined tool and see what the world does |
+| `declare_sequence` / `drop_sequence` | A series of calls whose end state must hold |
+| `inspect_world` | Look at what is in the world |
+| `amend_contract`, `add_rule`, `drop_rule`, `fix_tool` | Correct the contract |
+| `check_world` | Run every probe, report without saving |
+| `save_world` | Freeze it — refused unless it holds up |
+
+Bindings are small adapters to the callable the repository ships. There is no generated-handler
+fallback. If the callable cannot be imported or the shipped service cannot start, the build stops
+and reports the missing runtime/configuration seam.
+
+**The distinction the whole design turns on** (`runtime.py`):
+
+- `ToolError` — the world saying *no*. The id does not exist; the item is unavailable. **This is
+  the world working.**
+- Any other exception — our bug.
+
+They are recorded differently and never confused. `_is_refusal` matches the agent's own refusal
+type across the class hierarchy.
+
+### The gate: what `check_world` and `save_world` actually run
+
+`world/probe.py`. Every probe restores the world to a frozen baseline first, so probes cannot
+inherit each other's rows.
+
+| Probe | Asks |
+|---|---|
+| `happy` | A valid call built from the contract's permitted values. A refusal is acceptable; a crash never is |
+| `edge` | A nonexistent id → must refuse, not succeed and not crash. A missing required argument → must refuse |
+| `coverage` | Every contract tool has a handler; no handler for a tool the agent lacks; **and each handler actually reads the arguments the contract says it takes** |
+| `data` | Every identifier the contract permits exists in the world — catches a whole category left unseeded, which otherwise looks exactly like correct strictness |
+| `sequence` | Each declared sequence, run from the frozen world, leaves the state it claims |
+| unknown tool | Calling a tool that does not exist must refuse |
+
+`save_world` refuses on three counts: **score below 0.85**; **no declared sequence** (calls that
+each work alone can still forget what the last one did); and **a dirty world** — rows left over
+from building, which would otherwise appear in every scenario as somebody else's order already
+in the cart.
+
+Then `world/snapshot.py` writes `world.sqlite`, `handlers/*.py`, `world.py` and `manifest.json`.
+**The snapshot is the base state every scenario restores from.**
+
+---
+
+## 4. Scenarios — the conversations worth having
+
+`scenarios.py`, `skills/write-scenarios/SKILL.md`, tools in `scenario_tools.py`
+
+A scenario is a **change on the base environment**, and it owns a folder (`folder.py`):
+
+```
+scenarios/<name>/
+    scenario.json     the instruction, the reference solution, which sub-goals it names
+    setup.py          def setup(world)     what this scenario changes first
+    ready.py          def ready(world)     is the world ready for it
+    checks/<goal>.py  def check(world, calls)   one per deterministic sub-goal
+```
+
+`scenarios.json` is an index over those folders, regenerated from them, so anything wanting the
+whole suite at a glance has it.
+
+Code lives in files, never duplicated into JSON, and the file is the artifact: each check file is
+written with a `__main__` block so it runs standalone against what a run left behind, and a test
+proves the file and the harness give the same answer.
+
+`setup` is **code** rather than a list of rows: what a scenario changes is not necessarily the
+database alone. There is no persona and no opening line. Variability comes from **real
+conditions** that live in `setup`, and the base world stays the shared starting point.
+
+`sub_goals` are **names from the catalogue the environment step defined**, not restated wording.
+That is what makes results roll up: the same sub-goal failing in seven of twelve scenarios is one
+sentence.
+
+`solution` is what a correct agent would do. It is never run against the agent — it exists so the
+scenario can be proved.
+
+The writer can **look** (`inspect_world`) and **rehearse** (`try_calls` — run calls against a
+throwaway copy and see what state they leave), so a solution is written from what was observed.
+
+### The gate: three proofs, no model involved
+
+`prove.py`, called by `submit_scenario` before a scenario is kept:
+
+| | Run | Must |
+|---|---|---|
+| **Ready** | reset → `setup` → `ready` | **hold** |
+| **Solvable** | reset → setup → **the solution** → the checks | **pass** |
+| **Not vacuous** | reset → setup → **nothing** → the checks | **fail** |
+
+If the first fails, the world does not hold what the scenario presumes, and running it would test
+us rather than the agent: the agent would fail for a precondition we got wrong, and it would read
+as a finding about the agent. If the second fails, either the scenario cannot be passed or the
+check is wrong. If the third passes, the checks grade nothing while reporting a result, which is
+the failure that makes a suite quietly green.
+
+Vacuity is judged on *all* checks passing, because one check surviving an empty run ("no
+unavailable item was ordered") is legitimate. A single check that survives is still reported, in
+`Proof.weak`: sub-goals are shared, so a check that cannot fail without calls would roll up as a
+pass for an agent that did nothing at all.
+
+`save_scenarios` additionally refuses a suite where no sub-goal is shared by two scenarios,
+because nothing would roll up.
+
+---
+
+## 5. Run — put someone in front of it
+
+`run/`
+
+The simulation is **not a loop written here**. ALK already owns placing a call, driving the
+synthetic user, and producing a transcript; the harness supplies the world, the instruction, and
+the grading. Against the real hosted agent that is `run/live.py` and `run/call.py`:
+
+```
+world + setup ──► webhook ──► public url ──► the assistant's OWN tools repointed
+                                                       │
+                     ALK's voice case places the call ──┘
+                                                       │
+                       the world afterwards + the calls ──► the sub-goals' checks
+```
+
+1. **Restore** the frozen world and apply this scenario's `setup`. Its own copy, so nothing leaks
+   between scenarios.
+2. **Stand up the webhook** (`run/voice.py`) and bind that world to it. A hosted voice agent
+   executes its tools by calling a webhook, so answering that webhook from `handle_tool_call` is
+   the entire integration.
+3. **Expose it** (`cloudflared`, or `HARNESS_WEBHOOK_URL`), because a hosted agent cannot reach
+   loopback.
+4. **Repoint the assistant.** `pointed_at` copies the agent's **own** tools and changes only
+   `server.url`. Nothing about the agent is redefined — rebuilding its tools would mean testing an
+   agent we wrote.
+5. **Place the call** through ALK's own voice case, with the scenario's filled simulator prompt
+   driving the caller.
+6. **Grade** from the world afterwards plus the recorded calls, through the same checks the gates
+   used. A sub-goal marked `judged` is reported as judged, never silently counted.
+
+`run/alk.py` is the same story for the text path: the world goes in as `environment=` to ALK's
+`ChatEnvironment`, which owns the turn loop.
+
+Running is also a **stage of the conversation**, not only a command (`run/stage.py`,
+`skills/run-scenarios/SKILL.md`, tools in `run/tools.py`: `preflight`, `list_scenarios`,
+`run_scenario`, `read_results`). The stage exists because reading a failure is judgement: it has
+to sort every failure into one of four causes — the agent was wrong, the world wrongly refused,
+the check is wrong, or the simulated caller never asked for the thing — and only the first is a
+finding about the agent. Each run's record lands in `runs.json` with the instruction, the
+per-sub-goal verdicts, every tool call, and the transcript.
+
+This is what the environment was built for. The previous run's known issues — *"mocked tools
+always succeed, including removing an item that was never added"*, *"mock responses do not vary by
+argument"*, *"world state does not change unless a scenario sets `state_updates`"* — are all the
+same defect, and a world that really holds rows and can really refuse answers all three.
+
+---
+
+## What is exact, and what is judgement
+
+The split is deliberate and worth defending:
+
+| Judgement (the model) | Exact (code) |
+|---|---|
+| Reading unfamiliar source | Whether the contract is structurally usable |
+| Designing a schema | Whether a handler crashes or refuses |
+| Choosing what is worth testing | Whether an expectation resolves against real tables |
+| Whether a claim held | Whether the state matches |
+
+**The model never decides whether something passed.** It decides what to try.
+
+---
+
+## Where to extend it
+
+- A new **agent kind** → a class in `sources.py` and one registration
+- A new **world kind** (browser, filesystem, queue) → a class in `world/kinds.py` implementing
+  `values_present` / `mutable_state` / `describe`, and one registration. Browser is registered
+  and stubbed
+- A new **place the agent runs** (Vapi, LiveKit, a hosted endpoint) → a class in `run/targets.py`
+  with `open` / `say` / `close`, whose tool calls reach the same `world.handle_tool_call`
+- A change to **how a stage works** → edit its `SKILL.md`. No code
+
+## What is not built
+
+- Browser worlds: registered, not implemented
+- Snapshots are local files, not S3
+- Judged sub-goals are reported as judged, not actually sent to a judge yet
+- Results do not post to the platform
+- Nothing reports which of the contract's use cases have no scenario

fi/alk/harness/IMPLEMENTATION_AND_VALIDATION_STATUS.md

@@ -0,0 +1,229 @@
+# Harness implementation and validation status
+
+Status date: 22 August 2026
+
+This document records what is implemented and what has actually been exercised. It complements
+`ARCHITECTURE.md` (the target design) and `ENVIRONMENT_CONFORMANCE.md` (the packaging contract).
+
+## Product boundary now implemented
+
+ALK owns the execution plane: repository understanding, environment construction, test-data
+setup, scenario execution, evidence collection, grading and artifact production. The platform is
+the control and data plane: it creates jobs, supplies job-scoped secret references, receives
+events/results and presents history. Local CLI execution and hosted execution use the same
+`HarnessJob`, `EnvironmentBundle`, executor and result contracts.
+
+The central invariant is enforced throughout provisioning: ALK provides the environment in which
+the submitted agent and its existing tools run. It does not rewrite tools, invent proprietary
+service behavior or silently replace an unsupported dependency.
+
+## Completed implementation
+
+### Portable jobs, bundles and runtimes
+
+- Immutable local and hosted job contracts, with local paths rejected for hosted jobs.
+- Content-addressed environment bundles with provenance, per-file hashes and sizes.
+- Bundle verification before execution; symlinks, path traversal and changed content are rejected.
+- A runtime-provider boundary so local Docker/Compose can later be replaced by a hosted sandbox
+  provider without changing the harness pipeline.
+- Structured lifecycle events, cancellation and terminal result delivery for platform-driven jobs.
+- Retry classification that separates transient infrastructure/transport failures from agent
+  failures and avoids rerunning deterministic agent outcomes.
+
+### Repository and environment provisioning
+
+- **Compose repository:** adopt the submitted Compose model, isolate project names, ports, networks
+  and volumes, start dependency services, inject test endpoints and clean up only that run.
+- **Dockerfile-only repository:** keep the submitted runtime unchanged and generate only the
+  declared infrastructure around it. Managed adapters currently cover Postgres, ClickHouse,
+  Redis, MongoDB and Qdrant, including initialization, protocol-aware readiness and reset
+  behavior.
+- **Neither Compose nor Dockerfile:** fail admission with an actionable packaging error when an
+  external runtime is required. Automatic packaging for this third case remains open work.
+- Source fingerprints prevent reuse of stale builds.
+- Protocol-aware readiness avoids treating a merely open socket as a ready database.
+
+### Credentials, GitHub input and isolation
+
+- Static credential discovery from source, Compose, Dockerfile and known SDK connectors.
+- Preflight reports missing credential names before a sandbox is started.
+- Jobs and bundles contain `SecretRef` references, not resolved values; secret values are rejected
+  from persisted payloads and redacted from emitted output.
+- GitHub repository/revision references are represented in the hosted job contract. The platform
+  remains responsible for repository authorization and job-scoped credential resolution.
+- Unsafe Compose features such as privileged execution and host escape configuration are rejected.
+- Resource/lifecycle ownership is scoped to the run so cleanup cannot target another environment.
+
+### Evidence and artifact integrity
+
+- Setup activity is kept separate from agent activity and cannot earn evaluation credit.
+- A missing tool call cannot satisfy a tool-dependent check; dependent checks cannot pass when the
+  prerequisite action is absent.
+- Results distinguish environment, connectivity, simulator, agent, grading and artifact failures.
+- Artifact manifests include hashes and sizes, are verified on ingestion and are delivered with
+  an idempotency identity for safe platform retries.
+- LiveKit recording selection now ignores ambient/background publications and selects the actual
+  conversational speech track.
+
+## Environment conformance tested
+
+The checked-in fixtures are small conformance repositories, not product demo agents:
+
+| Packaging case | Dependencies | What was proven |
+|---|---|---|
+| Compose | ClickHouse + Redis | Two copies can run simultaneously despite fixed submitted ports; each worker reaches its own seeded services; resetting one does not affect the other. |
+| Dockerfile only | ClickHouse + Redis | ALK composes managed dependencies around the unchanged image, injects endpoints and verifies seeded data from the running agent. |
+| Dockerfile only | Postgres | ALK creates the database, applies submitted SQL, verifies application readiness and performs isolated cleanup. |
+| Dockerfile only | MongoDB + Qdrant | ALK creates both services, the runtime writes/reads both, reset recreates clean state and cleanup removes the project. |
+
+The real Docker conformance matrix exercised build, startup, readiness, state mutation, reset and
+teardown. The detailed contract and commands are in `ENVIRONMENT_CONFORMANCE.md`.
+
+## Real voice-agent validation
+
+Three agents from the official `livekit/agents` examples repository were validated at commit
+`da6af86ac640a3bc54585764e64321d7048c1c16`:
+
+- `drive_thru` — multi-tool ordering with in-process state;
+- `frontdesk` — scheduling with an optional external Cal.com connector and source fallback; and
+- `hotel_receptionist` — a larger booking flow with local SQLite state.
+
+For all three, ALK detected the Dockerfile-only runtime, built the upstream image, produced and
+verified a sealed bundle, started the worker, checked readiness and source integrity, and cleaned
+up the execution environment.
+
+Five real WebRTC calls were then made to each agent (15 total):
+
+| Agent | Calls | Completed | Environment failures | Preserved agent failures |
+|---|---:|---:|---:|---:|
+| Drive-through | 5 | 5 | 0 | 0 |
+| Front desk | 5 | 5 | 0 | 0 |
+| Hotel receptionist | 5 | 4 | 0 | 1 |
+| **Total** | **15** | **14** | **0** | **1** |
+
+The hotel failure is a useful deterministic finding rather than a harness failure: the agent
+repeatedly treated a valid 16-digit card number as 15 digits, entered a validation loop and reached
+the seven-minute scenario limit.
+
+Artifact checks passed for every call:
+
+- 15/15 reports and transcripts were created;
+- 15/15 combined recordings and 15/15 stereo recordings were present and larger than 100 KB;
+- all recordings contained measurable audio; and
+- all campaign containers were stopped after execution.
+
+The public example repositories, temporary model-adapted checkouts, recordings and campaign
+artifacts are intentionally not committed to ALK.
+
+### Hosted-provider connector validation
+
+Two non-native target connectors were exercised with real calls using the unchanged configured
+provider agents:
+
+| Connector | Result | Evidence |
+|---|---|---|
+| Vapi WebSocket | Completed; evaluation 0.8401 | 14 committed messages, 1,071 transcript characters, SDK and provider recordings; combined audio peak -3.6 dB. |
+| Retell Web Call | Completed; evaluation 0.8422 | 11 committed messages, 1,432 transcript characters, SDK and provider recordings; combined audio peak -1.9 dB. |
+
+Both ended through `simulator_end_call`, produced no typed call failure and cleaned up their
+FutureAGI LiveKit rooms. Daily and direct OpenAI Realtime remain separate transport gates; model
+substitution must not be represented as transport validation.
+
+## Additional public-repository packaging validation
+
+The environment admission path was also exercised against twelve structurally different public
+voice-agent repositories on 22 August 2026. These checks intentionally used untouched checkouts;
+repository defects remain findings instead of being patched by ALK.
+
+| Repository | Revision | Packaging result | Runtime result |
+|---|---|---|---|
+| Bolna | `0172347b601ea66dac0414cc1c6b14dc0d85422a` | Nested `local_setup/docker-compose.yml` is now discovered. Host-bound AWS credential mounts are reported explicitly. | Not started: its local Compose requires provider/telephony credentials and a local `.env`; ALK correctly stops before build rather than fabricating them. |
+| Pipecat examples (`websocket`) | `696c3541001350262dd11e81ccbf00754a56850c` | Root Dockerfile detected; `env.example` and its placeholder Google key are now recognized. | Rejected before Docker because the published Dockerfile requires an absent `uv.lock`. The same failure previously took about 80 seconds in BuildKit; static admission now reports it in about 1.2 seconds. |
+| TEN Framework (`ai_agents`) | `2e56d9659d8599350962374c0dc24725a03d73ce` | Development Compose and production Dockerfile are distinguished; ALK selects the production Dockerfile and preserves the repository's `linux/amd64` hint. | The upstream release reaches `task install` but Bun exits 132 under Docker Desktop emulation on the ARM test host. This is recorded as host/runtime compatibility, not an environment success or agent evaluation. |
+| Pipecat Cloud Starter | `9167c21ea76a67a02f330bae0c009d0ef5a6ef95` | Its root Dockerfile and sample credential declarations are accepted without modification. | The submitted image builds, but its real entrypoint fails: the archived repository leaves `pipecat-ai` unpinned and current 1.7.0 removed `LLMMessagesFrame`. Build success alone must not be represented as runtime success. A test-only compatibility checkout produced the real-call result below. |
+| Voice Noob | `755552f1b92b3c760799720f2c250e7f0a6bf0bd` | Its Compose-provided Postgres and Redis are selected as infrastructure. Profile-gated pgAdmin and Redis Commander are not mistaken for agent runtimes. | Infrastructure only: the unchanged services started and passed readiness in 6.348 seconds, reset cleanly in 6.238 seconds and were removed. The repository does not package its FastAPI agent runtime in Compose or a Dockerfile; its browser uses direct OpenAI Realtime WebRTC, which the current LiveKit-only call engine cannot drive. This is not an end-to-end agent success. |
+| Vocode Core (`telephony_app`) | `e054c33a72787b6a4920f91eb8598ad0bafb4240` | The component's Dockerfile and Compose are both found; the missing Compose `.env` is reported per affected service. | The unchanged Dockerfile reached dependency installation, then failed because it installs current Poetry while invoking the removed `--no-dev` option. This is an upstream reproducibility failure and is not patched by ALK. |
+| Open Telephony Stack | `a42e5b6c17c772af72825dd40240807c415b220b` | The independently runnable Asterisk, shim and voice-server components are reported as an ambiguous monorepo rather than guessed. | The Asterisk Compose is rejected for hosted execution because it requests host networking and host TLS/system mounts. Its voice-server Dockerfile also requires the repository root as build context, proving the need for explicit component plus build-context selection. |
+| Dograh | `058c540c4d92c55f529d04fabceb17da4901a0cb` | The root Compose runtime is selected; credential discovery is scoped to that runtime instead of every optional provider module. | The unchanged default stack started API, UI, pgvector/Postgres, authenticated Redis and MinIO in 36.391 seconds and cleaned up. This found and fixed authenticated-Redis readiness, empty-body HTTP readiness and dormant-profile port allocation. |
+| Voice Asterisk Agent | `6a5e0533b5293d1727847d123bf2ab8a1fc136de` | Asterisk, AudioSocket, API, agent, STT/LLM/TTS and monitoring components are discovered without mistaking `Dockerfile.dockerignore` for a runtime. | Admission remains blocked by missing external env injection, component selection and forbidden Docker-socket access. These are real hosted-environment requirements, not an agent grade. |
+| Aeyetech local voice agent | `34df18ab3e023bbf99b103bc4b891556999b6787` | Root Compose and local LiveKit/Whisper/llama.cpp/Kokoro/agent components are found. | Not started: a machine-specific absolute model bind mount and large CPU/GPU model requirements need explicit resource and mount admission first. |
+| Salesforce VoiceAgentRAG | `4d653890db18855d564d4ed9ca4d678047799cf4` | Qdrant/FAISS and local/cloud model requirements are visible, but the repository has neither Compose nor a Dockerfile. | Correct case-three packaging finding; ALK does not silently invent a runtime. |
+| LiveKit Node starter | `2045cc4917e40382fcf6b39e46ce24316a97370a` | Dockerfile-only Node runtime detected. Multi-source `COPY` validation now checks every input. | Rejected before Docker because the published Dockerfile requires absent `pnpm-lock.yaml`; the earlier BuildKit failure is now a deterministic admission finding. |
+
+The new preflight is deliberately conservative. It detects missing Docker build inputs, ambiguous
+monorepo component roots, nested Compose files, development-oriented Compose configuration,
+privileged/host namespace requests and host bind mounts before starting a container. Large Docker
+errors are bounded before entering job state so one failed build cannot overwhelm the platform UI.
+
+### Pipecat test-only real-call validation
+
+Because no Daily credential was available, a temporary external checkout replaced Daily with
+Pipecat's officially supported LiveKit transport. The archived starter's prompt and LLM behavior
+were preserved. Deepgram supplied transcription and OpenAI replaced a Cartesia credential that
+returned HTTP 401. These compatibility changes were not made to ALK or committed to the public
+repository.
+
+One real harness-driven WebRTC call then completed with 10 committed messages, 1,060 transcript
+characters and `simulator_end_call`. Combined and stereo recordings were produced at 1,295,634
+and 2,591,224 bytes; measured peak audio was -2.9 dB, proving that the files were not silence. The
+target received and transcribed the caller audio, generated responses through its LLM pipeline and
+returned synthesized speech through LiveKit. Its isolated runtime was removed after the call.
+
+### Test-only compatibility note
+
+The current LiveKit server returned HTTP 401 for the example agents' LiveKit Inference models. To
+exercise real calls, temporary external checkouts used direct Deepgram and OpenAI model providers
+for STT/LLM/TTS. This was a test setup change only: no agent tools, prompts, database behavior or
+harness source was rewritten. The temporary checkouts are not part of this branch.
+
+## Automated release gate
+
+The harness-focused gate completed with:
+
+```text
+508 passed, 11 skipped
+```
+
+The skipped cases are opt-in Docker/real-agent checks and require their documented runtime flags
+and credentials. The WebRTC engine regression suite is included in the 508 passing tests. Ruff and
+`git diff --check` also pass for this change set.
+
+## Findings that remain open
+
+These items should not be represented as complete:
+
+1. Implement the production hosted provider (for example Daytona or a microVM fleet) behind the
+   existing runtime interface, with quotas, egress policy and hard cleanup guarantees.
+2. Add an explicit packaging adapter for repositories with neither Compose nor a Dockerfile.
+3. Expand managed dependency conformance beyond Postgres, ClickHouse, Redis, MongoDB and Qdrant.
+   RabbitMQ/NATS, MinIO/S3 and browser/code-execution services are the next explicit gates;
+   unsupported services must continue to fail explicitly.
+4. Add first-class component/build-context selection for monorepositories. Detection and
+   actionable ambiguity are implemented, but the platform does not yet let the user select one
+   of several independently runnable agents within a repository. Credential discovery now follows
+   an automatically selected Compose runtime and its build context, but must also follow an
+   explicit user-selected component in ambiguous monorepositories.
+   Compose files that require a repository-local `.env` also need a job-scoped external-env
+   adapter; ALK currently reports the missing file rather than mutating the submitted checkout.
+   Internal environment secrets such as a local stack's JWT signing key should be generated as
+   job-scoped secret references rather than requested from the customer.
+5. Add generic target-tool event ingestion for uninstrumented third-party LiveKit agents. Audio and
+   transcripts work today, but source-owned tool calls cannot always be normalized automatically.
+   Add target media adapters for non-LiveKit agents, beginning with direct OpenAI Realtime WebRTC,
+   Daily and generic bidirectional audio WebSocket. Voice Noob cannot be called through the current
+   engine until one of these adapters exists; switching its model does not change its transport.
+6. Productize GitHub authorization (GitHub App), secret storage/resolution and credential UX on the
+   platform. The ALK contracts and preflight are present; the production integrations are not.
+7. Improve platform run UX with granular live stages, logs, actionable waiting states and artifact
+   reconciliation rather than a long generic status.
+8. Continue work on diverse scenario/persona generation and controllable simulator behavior.
+9. Add production object-storage upload/reconciliation, retention policy and load/chaos testing.
+10. Track LiveKit transport instability observed as intermittent signal-resume 502/EOF responses.
+
+## Current conclusion
+
+The local execution architecture and the environment creation path for well-packaged and
+Dockerfile-only voice agents are functioning end to end. Real agents could be built, started and
+called through the generated environment, and complete transcripts and audio artifacts were
+produced. The remaining work is primarily the production hosted sandbox/provider integration,
+broader packaging/service coverage, third-party tool telemetry and platform product experience.