antpath 0.1.1 → 0.2.1

package/references/architecture-decisions.md

@@ -20,9 +20,9 @@ This supersedes the earlier SDK-only MVP boundary.
 - Minimal tenant-scoped dashboard metadata for monitoring run execution.
 - Durable worker recovery after restarts, deploys, crashes, or missed notifications.
 - Horizontal worker scaling without duplicate run ownership.
-- Provider cleanup by default after terminal states, with optional retention when configured per-request or per-deployment.
-- BYO provider key custody with encrypted storage.
-- Private output capture with quota enforcement.
+- Provider cleanup by default after terminal states, with optional retention of Claude-side resources when configured per-request or per-deployment.
+- Per-session BYO provider key custody: keys arrive inline with each run, are encrypted in Supabase Vault for the lifetime of that single run, and are deleted at cleanup. Persistent workspace-level provider key custody is backlog.
+- Unconditional capture of every session-scoped Claude artifact into private Supabase Storage, bounded only by the workspace storage cap.
 - Programmatic SDK access for submitting and observing platform runs.
 - Test-driven implementation across SDK, dashboard, worker, database, storage, and provider adapters.
 
@@ -34,6 +34,8 @@ This supersedes the earlier SDK-only MVP boundary.
 - No direct browser access to provider APIs.
 - No direct browser Supabase data access.
 - No raw prompt, raw model output, raw provider event payload, MCP credential, provider key, or output file content in normal application tables.
+- No persistent workspace-level provider connections (no "saved Anthropic keys"). Every run carries its own secrets bundle; nothing about the user's Anthropic key persists past run cleanup. Persistent BYO custody is backlog.
+- No user-configurable output capture: every artifact Claude exposes on the run's session is captured. There are no globs, no `capture: false`, no per-file or per-run user-visible caps.
 - No provider Agent/Environment caching in MVP.
 - No provider webhooks in MVP.
 - No Supabase Realtime in MVP.
@@ -44,7 +46,7 @@ This supersedes the earlier SDK-only MVP boundary.
 | Area | Decision |
 | --- | --- |
 | Product surface | Platform plus SDK. |
-| Repository | Convert the repository to an npm TypeScript workspace. |
+| Repository | Convert the repository to a pnpm TypeScript workspace. |
 | SDK location | Move the existing SDK package to `packages/sdk`. |
 | SDK role | The SDK submits runs and observes status, metadata, outputs, cancellation, and deletion through the platform API. |
 | Dashboard | Build a first-party minimal dashboard. |
@@ -59,8 +61,8 @@ This supersedes the earlier SDK-only MVP boundary.
 | Service credentials | Supabase service-role credentials are server/worker only. |
 | SDK auth | SDK uses hashed, workspace-scoped API tokens. Dashboard uses Auth.js sessions. |
 | API token attribution | Token-authenticated runs are attributed to the token creator at submission time unless a future service-account model is introduced. |
-| Provider key custody | Workspace BYO Anthropic key is stored encrypted through Supabase Vault. |
-| Secret lifetime | Worker resolves provider keys per claimed lifecycle step, keeps them only in memory for that step, and drops them before lease release. |
+| Provider key custody | BYO Anthropic key arrives inline with every run submission. The BFF stores the secrets bundle in Supabase Vault, attaches the secret id to the run row, and deletes the Vault entry as part of cleanup. Persistent workspace-level provider connections are backlog. |
+| Secret lifetime | Worker resolves the per-run secrets bundle once per claimed lifecycle step, keeps the decoded payload only in memory for that step, and drops it before lease release. The Vault entry itself is destroyed when the run reaches terminal cleanup. |
 | Provider resources | Create provider Agent, Environment, Vault/Credential, Session, and file resources per run in MVP. |
 | Provider resource caching | Backlog; later cache Agent/Environment by Template/config hash. |
 | Worker wakeup | Poll due rows from the runs table. Postgres `NOTIFY` is a latency optimization only. |
@@ -72,12 +74,13 @@ This supersedes the earlier SDK-only MVP boundary.
 | MCP approvals | Disallow approval-required tools. Worker must not approve tools or return custom tool results. |
 | Template boundary | Templates are code-first, secret-free snapshots with stable hashes. |
 | Idempotency | Run submission idempotency is scoped by `(workspace_id, idempotency_key)` and request hash. Same key and same hash returns the existing run; same key and different hash returns conflict. |
-| Quotas | Enforce workspace plan quotas with per-user attribution. |
-| Run caps | Plan-based caps cover duration, concurrency, storage, polling, retries, and output capture. |
+| Quotas | Enforce workspace storage and concurrency quotas. Storage is the only user-visible cap on output capture. |
+| Run caps | Plan-based caps cover duration, concurrency, polling, retries, and workspace storage. There are no user-visible per-file or per-run output caps. |
 | Operational config | Exact cap/tier values are environment-configurable defaults with conservative fallbacks and run-level snapshots. |
-| Output capture | Enabled by default when configured by the run/template and within quota. |
+| Boot-time env validation | Worker and dashboard validate required environment variables at startup through a shared role-scoped helper and exit/abort with an aggregated error when any are missing. No silent defaults for identity, credentials, or required URLs. |
+| Output capture | Unconditional. Every file Claude exposes on the run's session is captured into private Supabase Storage. The workspace storage cap is the only user-visible quota; capture failures (quota exceeded, download failed) are persisted as `output_capture_failures` rows and surfaced in the dashboard alongside successful outputs. |
 | Output access | BFF returns signed links only after workspace authorization. |
-| Cleanup | Clean up Claude provider resources by default after terminal state and output capture; explicit run policy or worker default can retain them for inspection. |
+| Cleanup | Clean up Claude provider resources by default after terminal state and output capture; explicit run policy or worker default can retain Claude-side resources for inspection. The per-run Vault secret is always deleted at cleanup regardless of the Claude-side retention knob. |
 | Delete semantics | User delete is soft/pending while execution, cleanup, or storage deletion is active. Hard purge happens only after cleanup/storage deletion succeeds. |
 | Retention | Run metadata and stored outputs remain until user deletion in MVP. |
 | Realtime | Defer from MVP; use BFF-mediated refresh/polling first. Revisit with custom Supabase JWT/RLS or a server-mediated realtime bridge. |
@@ -125,35 +128,44 @@ packages/
 
 ### Identity and tenancy
 
-- `users`
-  - Auth.js user identity mirror or Auth.js adapter user table reference.
+- Auth.js adapter tables: `users`, `accounts`, `sessions`, and `verification_token`
+  - Auth.js owns OAuth account linking and email magic-link verification persistence in the same Supabase Postgres database.
+  - Active application sessions use Auth.js JWT strategy; the `sessions` table exists for adapter compatibility but is not the active session store.
+  - Auth.js OAuth tokens remain confined to Auth.js adapter tables and never become antpath provider credentials.
+- `app_users`
+  - Antpath-owned user identity row linked to Auth.js `users.id` through `auth_user_id`.
+  - Workspace membership, run attribution, API-token ownership, disable state, and JWT token-version checks use `app_users.id`.
+  - `disabled_at` and `token_version` are checked by privileged BFF routes to reject disabled or stale JWT users.
   - No provider secrets.
 - `workspaces`
   - Tenant boundary.
   - Plan, quota, status, and retention settings.
 - `workspace_memberships`
-  - Workspace, user, role, and membership status.
+  - Workspace, `app_user_id`, role, and membership status.
   - Every dashboard/API operation must prove membership/scope.
 - `api_tokens`
   - Workspace-scoped SDK credentials.
-  - Store hashed token material only, plus scopes, creator, last-used timestamp, and revoked timestamp.
+  - Store hashed token material only, plus scopes, `creator_app_user_id`, last-used timestamp, and revoked timestamp.
   - API-token submissions freeze the attributed user on the run row at creation time.
 
-### Provider connections
+### Per-run secrets
 
-- `provider_connections`
-  - Workspace-owned provider configuration.
-  - Provider type, display name, validation status, rotation/revocation status, and encrypted secret reference.
-  - Secret values live in Supabase Vault.
-  - Only trusted server/worker code can resolve decrypted provider keys.
+There is no `provider_connections` table in MVP. Every run carries its own secrets.
+
+- The submission contract has a top-level `secrets` block: `{ anthropic: { apiKey }, mcpServers?: [{ name, url, headers? }], skills?: [...] }`. The shared parser enforces that **no other field** carries secret-bearing keys; the `secrets` block is the single allowlisted carrier and never serializes into normal metadata, logs, events, audit rows, or stored snapshots.
+- The dashboard BFF stores the JSON-encoded bundle in Supabase Vault at submission time and writes the resulting `vault_secret_id` into `runs.execution_secret_id`. The bundle is the only durable trace of the user's Anthropic key, MCP credentials, and skill references for that run.
+- Worker code reads `runs.execution_secret_id` once per claimed step, decodes the bundle in memory, builds the Claude provider, and drops the decoded values before lease release.
+- At terminal cleanup the worker deletes the Vault entry and clears `runs.execution_secret_id`. Cleanup of the Vault secret happens regardless of the Claude-side retention knob.
+- If the user revokes (rotates) the Anthropic key on Anthropic's side mid-run, antpath has no knowledge of this until the next provider call fails. Active runs surface a `tenant_permanent` failure with a redacted reason; cleanup proceeds with the previously vaulted key while it remains valid and gives up gracefully if it does not.
 
 ### Runs
 
 - `runs`
   - User-visible logical run.
-  - Workspace, creator/attributed user, template snapshot/hash, status, lifecycle phase, plan caps, timestamps.
+  - Workspace, `creator_app_user_id`, template snapshot/hash, status, lifecycle phase, plan caps, timestamps.
   - Submission idempotency key and request hash.
   - Unique constraint: `(workspace_id, idempotency_key)`.
+  - `execution_secret_id` — Supabase Vault secret id for the per-run secrets bundle (Anthropic key + MCP credentials + skill references). Cleared at cleanup.
   - Lease fields: `lease_owner`, `lease_token`, `lease_expires_at`, `attempt_count`, `next_check_at`, `priority`.
   - Cancellation/deletion fields: `cancel_requested_at`, `pending_delete_at`, `deleted_at`.
   - Provider observation cursor/watermark where available.
@@ -174,24 +186,34 @@ packages/
 
 - `output_objects`
   - Workspace/run owner, storage bucket/path, size, checksum if available, content type, provider file id.
+  - One row per artifact successfully captured from the run's session into private Supabase Storage.
   - Used for quota accounting and signed-link generation.
+- `output_capture_failures`
+  - Workspace/run owner, provider file id, filename, byte size as reported by the provider, reason (`workspace_quota_exceeded`, `download_failed`, etc.), redacted error message.
+  - Surfaced in the dashboard alongside `output_objects` so the user can see exactly which session artifacts didn't make it to storage and why.
 - `cleanup_attempts`
   - Per-resource cleanup action, status, retry count, redacted error code/message, timestamps.
 - `usage_ledger`
   - Token/cost/storage attribution by workspace and attributed user.
   - Written transactionally with source event/output rows to avoid drift.
+- `audit_logs`
+  - Safe security audit trail for sign-in user upsert, API-token changes, run cancellation/deletion, signed-link creation, and cleanup failures.
+  - Stores safe identifiers, action names, redacted errors, and non-secret metadata only.
+- `rate_limit_buckets`
+  - Durable token-bucket-style counters for abuse-sensitive BFF/API operations.
+  - Used before side effects for run submission, run cancellation/deletion, signed-link creation, API-token creation/revocation, and email-auth throttling where practical.
 
 ## Run lifecycle
 
-1. SDK/dashboard submits a run to the BFF.
+1. SDK/dashboard submits a run to the BFF. The submission carries a top-level `secrets` bundle (Anthropic key, optional MCP credentials, optional skill references) plus the non-secret template/variables/cleanup payload.
 2. BFF validates Auth.js session or SDK API token.
 3. BFF resolves active workspace membership/scope.
-4. BFF validates Template/request shape, plan limits, and idempotency key/request hash.
-5. BFF writes a `runs` row with execution-affecting cap values snapshotted from plan/env defaults.
+4. BFF validates Template/request shape, plan limits, and idempotency key/request hash. The shared parser rejects every secret-bearing key outside the allowlisted `secrets` block.
+5. BFF stores the JSON-encoded `secrets` bundle in Supabase Vault and writes a `runs` row with execution-affecting cap values snapshotted from plan/env defaults plus `execution_secret_id` pointing at the Vault entry.
 6. BFF emits Postgres `NOTIFY` for fast wakeup.
 7. Worker claims due runs with row locking, `SKIP LOCKED`, lease token, and lease expiry.
-8. Worker resolves the provider key from Supabase Vault for the claimed step.
-9. Worker validates platform constraints: no approval-required tools, no custom antpath-executed tools, known output policy.
+8. Worker resolves the per-run secrets bundle from Supabase Vault for the claimed step.
+9. Worker validates platform constraints: no approval-required tools, no custom antpath-executed tools.
 10. Worker pre-journals intended provider resources.
 11. Worker creates provider resources and records provider IDs immediately after successful calls.
 12. Worker creates provider session and sends the initial user event.
@@ -199,8 +221,8 @@ packages/
 14. Worker later reclaims the run and polls provider session status plus event list.
 15. Worker stores only metadata events and usage, deduped by provider event id.
 16. Before every side effect, worker checks lease token and cancellation/deletion requests.
-17. On terminal provider state or antpath timeout, worker captures configured outputs within per-file, per-run, and workspace quota caps.
-18. Worker cleans up provider resources by default, or records them as retained when the run/deployment policy asks to retain them.
+17. On terminal provider state or antpath timeout, worker lists session-scoped files via the Claude Files API and unconditionally captures every file the workspace storage quota still has room for, persisting `output_capture_failures` rows for any file that would exceed the cap.
+18. Worker cleans up Claude-side provider resources by default, or records them as retained when the run/deployment policy asks to retain them. The per-run Vault entry is deleted regardless.
 19. Worker marks the run terminal and releases the lease.
 20. Dashboard/SDK read tenant-scoped metadata and signed output links through BFF APIs.
 
@@ -259,51 +281,59 @@ BFF/server actions must:
 - return only metadata allowed by the user's role/scope;
 - keep Supabase service-role credentials out of browser bundles.
 
+Auth.js and antpath authorization stay separate in the database. Auth.js owns `users`, `accounts`, `sessions`, and `verification_token`; antpath owns `app_users` and links each row to Auth.js `users.id`. JWT callbacks expose only safe identity fields such as `app_user_id` and token version. BFF routes must load the current `app_users` row and reject disabled users or stale token versions before privileged actions.
+
+All platform and Auth.js tables have RLS enabled and `anon`/`authenticated` table privileges revoked. This is a defense-in-depth guard against accidental Supabase Data API exposure; direct `pg` repository code remains the primary BFF/worker authorization boundary for this phase.
+
 Supabase Realtime is backlog until antpath either mints short-lived Supabase-compatible JWTs with RLS policies or exposes a server-mediated realtime bridge.
 
 ## Secret handling
 
-- Provider keys are workspace BYOK and stored via Supabase Vault.
-- MCP credentials follow the same no-log/no-table-secret rule.
-- Normal app tables store only secret references and validation/rotation status.
-- Worker resolves decrypted provider keys only for a claimed lifecycle step.
-- Secret values should use explicit redacted wrappers in code so they cannot serialize into logs, metrics, errors, events, or fixtures.
+- Provider keys arrive inline with each submission and are stored in Supabase Vault tied to a single run via `runs.execution_secret_id`.
+- MCP credentials and any skill payloads referenced by the run live alongside the provider key inside the same per-run Vault bundle.
+- Normal app tables store only the Vault secret id, not decoded provider keys, MCP credentials, or skill payloads.
+- Worker resolves the decoded bundle only for a claimed lifecycle step and drops the decoded values before lease release.
+- The shared submission parser keeps an allowlist of exactly one carrier (`secrets`) and rejects every other field that looks like a secret-bearing key (`apiKey`, `accessToken`, `password`, `mcpCredentials`, etc.).
+- Secret values use explicit redacted wrappers in code so they cannot serialize into logs, metrics, errors, events, or fixtures.
 - Secret redaction applies to logs, errors, run metadata, provider events, tests, fixtures, and docs.
-- If a provider key is revoked/rotated mid-run, active runs are failed or cancelled with a tenant-permanent error unless cleanup can still authenticate.
-- Revoked keys do not get a hidden grace cache in MVP. If cleanup cannot authenticate after revocation, the run/resource moves to `cleanup_failed` with an actionable tenant error.
+- The per-run Vault entry is deleted as part of run cleanup. Cleanup of the Vault entry happens regardless of the Claude-side retention knob.
+- If a provider key is revoked or rotated on the provider side mid-run, the next provider call fails with a `tenant_permanent` error. The worker fails the affected run, attempts cleanup with whatever credentials are still valid, and surfaces a redacted reason on the dashboard.
+- There is no platform-side rotation/revocation flow because there is no persistent provider connection. To "rotate" the user simply submits the next run with a different key.
 
 ## Output storage and quotas
 
+Output capture is unconditional. Every artifact the user's Claude session exposes is downloaded to private Supabase Storage. There are no user-facing knobs to opt out, no globs, and no per-file or per-run user-visible caps.
+
 Output flow:
 
-1. Worker lists provider session files.
-2. Worker checks provider file metadata, output policy, and remaining quota.
-3. Worker downloads selected files with streaming hard caps.
-4. If provider size is unknown or exceeds caps, worker aborts capture for that object and records a quota/cap warning.
-5. Worker uploads accepted files to private Supabase Storage.
-6. Worker records `output_objects` rows.
-7. Worker cleans up provider resources by default, or retains them when explicitly configured.
-8. Dashboard/SDK request signed links through the BFF.
+1. Worker lists files scoped to the run's Claude session: `GET /v1/files?scope_id=<session_id>`. Each entry includes `id`, `filename`, and `size_bytes`.
+2. For each file, before downloading, the worker checks `(workspace.storage_used_bytes + size_bytes)` against the workspace storage cap.
+3. If the file would exceed the cap, the worker writes an `output_capture_failures` row (`reason = "workspace_quota_exceeded"`, with filename and reported byte size) and moves on without downloading.
+4. Otherwise the worker downloads via `GET /v1/files/{file_id}/content` with a hard streaming safety cap so a malformed listing response cannot OOM the worker, uploads to private Supabase Storage at a workspace-scoped path, and writes an `output_objects` row.
+5. Other download/upload failures (network, storage, unexpected size) are recorded as `output_capture_failures` with the redacted reason.
+6. Dashboard/SDK request signed links through the BFF for any `output_objects` row.
 
 Quota rules:
 
-- Workspace plan quota is the hard enforcement boundary.
-- Per-file and per-run caps protect the worker before workspace quota is consumed.
-- Per-user attribution is frozen from the run row at submission time.
-- Free first X users is a plan/billing rule on top of workspace-level enforcement.
+- Workspace storage cap is the only hard enforcement boundary. Default is generous (configurable via env, see [`environment-variables.md`](./environment-variables.md)).
+- There is no per-user quota. Per-user attribution still freezes on the run row at submission time for billing/audit purposes, but it is not a quota dimension.
+- There are no user-visible per-file or per-run caps. A worker-internal streaming safety cap protects against malformed listings.
+- Capture failures are first-class data: the dashboard renders `<filename> (<size>) — skipped: workspace quota exceeded` next to successful outputs so the user can see exactly what didn't make it.
 
 ## Cleanup and reconciliation
 
-Claude provider-resource cleanup is mandatory by default after terminal provider state and output capture so antpath does not leave behind provider state. Retention is opt-in and policy-driven through `cleanup.claudeSession = "retain"` or `ANTPATH_WORKER_CLAUDE_SESSION_CLEANUP_DEFAULT=retain`. Retained resources are recorded in `provider_resources.cleanup_status = retained` and remain reachable via the provider API.
+Claude provider-resource cleanup is mandatory by default after terminal provider state and output capture so antpath does not leave behind provider state. Retention is opt-in for **Claude-side resources only** through `cleanup.claudeSession = "retain"` or `ANTPATH_WORKER_CLAUDE_SESSION_CLEANUP_DEFAULT=retain`. Retained resources are recorded in `provider_resources.cleanup_status = retained` and remain reachable via the provider API. Antpath-side metadata, output objects, and the per-run Vault entry are not affected by this knob — Vault entries are always deleted at cleanup; metadata and outputs persist until the user explicitly deletes the run.
 
 Explicit cleanup order:
 
-1. Provider credentials/vaults.
-2. Session files/session where supported.
-3. Agent/archive.
-4. Environment/archive or delete where allowed.
-5. Uploaded provider file resources.
-6. Local output metadata/storage only when user deletes a run.
+1. Capture session-scoped files into private Supabase Storage (with quota pre-check; failures persisted).
+2. Provider session files / session where supported.
+3. Agent / archive.
+4. Environment / archive or delete where allowed.
+5. Skills uploaded for this session and any other ephemeral provider resources created for this run.
+6. Provider Vault/Credentials created on Claude's side for MCP wiring.
+7. The antpath per-run Vault entry: `vault.deleteSecret(runs.execution_secret_id)`, then clear the column.
+8. Local output metadata/storage only when the user deletes a run.
 
 Cleanup properties:
 
@@ -319,10 +349,19 @@ Resource leak recovery:
 - A reconciliation sweeper reviews unfinished intended rows, expired leases, and provider-listable resources tagged with antpath metadata.
 - If provider create succeeded but provider id was not persisted before a crash, the sweeper attempts to match by deterministic name/metadata and attach the provider id for cleanup.
 
-Workspace/key deletion:
+Workspace deletion:
 
 - Workspace deletion blocks new runs, requests cancellation for active runs, drains cleanup, deletes stored outputs, then purges metadata.
-- Provider key revocation blocks new runs immediately and may force existing runs to fail/cancel if cleanup can no longer authenticate.
+- There is no separate "provider key revocation" flow because there is no persistent provider key. Submitting the next run with a different key is the rotation mechanism.
+
+## Audit, rate limits, quotas, and deletion
+
+- Abuse-sensitive actions are rate-limited before provider, Vault, Storage, or mutation side effects where practical.
+- Audit logs must never include provider keys, OAuth tokens, MCP credentials, raw prompts, raw outputs, signed URLs, or service-role credentials.
+- Workspace storage quota is enforced before output download/upload when file size is known, with streaming hard caps for unknown-size files.
+- App-user deletion is soft/disable-first; hard deletion must not break historical run, usage, output, or audit references.
+- Workspace deletion is pending-first: block new submissions, mark active runs pending-delete, allow cleanup/storage deletion to drain, then purge only safe metadata.
+- Recovery operations should be driven by explicit cleanup/deletion state and audit logs, not manual DB guessing.
 
 ## Run state model
 
@@ -364,6 +403,8 @@ Error classes:
 - Database: Supabase Postgres.
 - Storage: Supabase Storage private bucket.
 - Secrets: Vercel/Railway environment variables plus Supabase Vault for tenant provider keys.
+- CI: GitHub Actions with pnpm, default lint/test/build/package checks, and a separate local Supabase integration job.
+- Local dev stack: `pnpm dev:stack` starts Supabase without printing local keys, validates local config, checks migrations, builds shared packages, and runs the dashboard and worker together. Missing Auth.js providers do not block local service startup unless `ANTPATH_DEV_STACK_REQUIRE_AUTH_PROVIDER=1` is set.
 
 Required runtime config includes:
 
@@ -374,7 +415,10 @@ Required runtime config includes:
 - Auth.js secret/providers;
 - SDK token hashing secret/pepper;
 - worker identity;
-- provider API base/version settings.
+- provider API base/version settings;
+- dashboard BFF rate-limit defaults;
+- worker polling/lease/metrics settings.
+- dev stack ports for dashboard and worker.
 
 Environment-configurable defaults include:
 
@@ -396,26 +440,28 @@ Environment-configurable defaults include:
 - workspace storage cap;
 - signed URL TTL;
 - free user allowance;
-- metadata retention toggles.
+- metadata retention toggles;
+- audit/rate-limit windows for sensitive BFF actions.
+- live e2e opt-in flags for provider sessions, public MCP endpoints, Anthropic skill IDs, and timeouts.
 
 Missing optional env vars must fall back to conservative low limits. Missing required secret/connectivity env vars must fail service startup.
 
+Live public MCP/skills e2e tests are intentionally opt-in because they depend on external provider and public MCP availability. Deterministic unit/component tests validate request wiring for MCP server declarations, tool policies, skill references, and secret non-leakage; the live e2e validates the real provider accepts and runs that configuration end to end.
+
+No-cost provider resilience and load testing uses sanitized replay snapshots from real Managed Agents sessions. Replay tests inject network disconnects, transient HTTP failures, provider latency, slow SSE chunks, cleanup failures, and concurrent full-session load around those snapshots. Idempotent read/delete-style calls may retry with bounded backoff; provider create/send POST calls are not retried by default because their side effects may already have happened.
+
 ## Open implementation details
 
 These are not architecture blockers, but must be pinned during implementation planning:
 
-- Exact plan tier values for duration, concurrency, storage, polling, and free user allowance.
-- Exact Auth.js providers.
-- Auth.js adapter/session mode and user mirror lifecycle.
-- Supabase Vault access method and SQL privilege wrapper shape.
-- Signed URL TTL and whether storage paths include random unguessable components in addition to workspace/run ids.
-- Deletion audit requirements.
+- Production plan tier values for duration, concurrency, storage, polling, and free user allowance.
 - Provider metadata naming convention.
 - Provider list/search capabilities and reconciliation coverage for each resource type.
 - Exact Claude Managed Agents event pagination/filter semantics and bounded polling fallback.
 
 ## Backlog
 
+- Persistent workspace-level provider connections (BYO Anthropic key custody across runs, with rotation, revocation, and re-use). The MVP submission contract carries the key inline per run instead.
 - Provider webhooks as wakeup/reconciliation accelerator.
 - SSE live event stream for richer UI.
 - Supabase Realtime with explicit Auth.js-to-Supabase authorization design.

package/references/implementation-plan.md

@@ -8,7 +8,7 @@ scope: platform MVP
 
 ## Goal
 
-Convert antpath from an SDK-only package into a TypeScript workspace containing:
+Convert antpath from an SDK-only package into a pnpm TypeScript workspace containing:
 
 - a platform SDK in `packages/sdk`;
 - a dashboard app in `apps/dashboard`;
@@ -21,27 +21,30 @@ Implementation is test-driven. Each behavior starts with the narrowest failing t
 
 ## Acceptance criteria
 
-- The repository is an npm workspace and the current SDK is moved to `packages/sdk`.
+- The repository is a pnpm workspace and the current SDK is moved to `packages/sdk`.
 - Dashboard and worker app folders exist with build/test integration.
 - Auth.js authenticates dashboard users.
 - SDK API tokens authenticate programmatic clients.
 - Workspaces are the tenant boundary.
 - BFF/server actions scope every dashboard/API operation by workspace membership or API-token scope.
 - Supabase service-role credentials are never exposed to browser/client bundles.
-- Supabase Postgres stores durable run, attempt, provider resource, event, output, cleanup, usage, workspace, membership, and provider connection metadata.
-- Provider keys are stored encrypted through Supabase Vault and resolved only by trusted server/worker code.
-- Run submission idempotency is enforced by workspace and request hash.
+- Supabase Postgres stores durable run, attempt, provider resource, event, output, output-capture-failure, cleanup, usage, workspace, and membership metadata. There is no persistent `provider_connections` table in MVP.
+- Per-run secrets bundles (Anthropic key, optional MCP credentials, optional skill references) arrive inline on submission, are encrypted through Supabase Vault for the lifetime of that single run, and are deleted at terminal cleanup. The Vault entry is the only durable trace of the user's provider key.
+- Run submission idempotency is enforced by workspace and request hash. The hash excludes the `secrets` block so re-submitting the same logical run with a new key still matches.
 - Workers claim due runs with leases and `FOR UPDATE SKIP LOCKED`.
 - Multiple workers can run concurrently without duplicate lifecycle ownership.
 - Worker polling recovers from missed `NOTIFY`, worker restart, deploy, and expired leases.
 - Worker creates per-run provider resources and journals them for cleanup.
 - Worker polls provider status and events and stores only redacted metadata.
-- Output capture writes configured files to private Supabase Storage within per-file, per-run, and workspace quotas.
+- Output capture is unconditional: every artifact Claude exposes on the run's session is written to private Supabase Storage, bounded only by the workspace storage cap. Files that would exceed the cap are persisted as `output_capture_failures` rows.
 - BFF returns signed output links only after workspace authorization.
-- Cleanup runs after terminal provider state and output capture.
+- Cleanup runs after terminal provider state and output capture, deletes the per-run Vault entry, and (by default) deletes Claude-side resources. `cleanup.claudeSession: "retain"` only affects Claude-side resources.
 - Reconciliation can recover intended provider resources after partial worker crashes.
 - User deletion is pending/soft until cleanup and storage deletion are complete.
 - Exact tier/cap values are configurable through environment variables with conservative defaults and run-level snapshots.
+- Security-sensitive BFF/API actions are audited and rate-limited.
+- Workspace storage quota and workspace deletion are enforced before provider/storage side effects proceed.
+- CI runs pnpm lint/test/build/package checks plus a local Supabase integration job.
 - Tests follow the accepted taxonomy: deterministic unit tests (including fakes and sanitized recorded snapshots), live external-system integration tests with no skip flags, and top-to-bottom live e2e tests.
 
 ## Phase 1: Workspace foundation
@@ -50,16 +53,16 @@ Create the workspace structure without changing runtime behavior.
 
 Deliverables:
 
-- Root npm workspace configuration.
+- Root pnpm workspace configuration.
 - Current SDK moved to `packages/sdk`.
 - `apps/dashboard` placeholder.
 - `apps/worker` placeholder.
 - Shared TypeScript config/build/test commands.
 - Existing SDK exports preserved or intentionally migrated with compatibility notes.
 - Root validation commands:
-  - `npm run lint`
-  - `npm test`
-  - `npm run build`
+  - `pnpm lint`
+  - `pnpm test`
+  - `pnpm build`
 
 TDD gate:
 
@@ -113,23 +116,25 @@ Deliverables:
 
 - Migration framework.
 - Tables:
-  - `users`;
+  - Auth.js adapter tables: `users`, `accounts`, `sessions`, `verification_token`;
+  - antpath `app_users`;
   - `workspaces`;
   - `workspace_memberships`;
   - `api_tokens`;
-  - `provider_connections`;
-  - `runs`;
+  - `runs` (includes `execution_secret_id` referencing the per-run Vault entry);
   - `run_attempts`;
   - `provider_resources`;
   - `run_events`;
   - `output_objects`;
+  - `output_capture_failures`;
   - `cleanup_attempts`;
   - `usage_ledger`.
 - Constraints:
   - unique `(workspace_id, idempotency_key)` on runs;
   - request-hash conflict handling;
   - unique `(run_attempt_id, provider_event_id)` on events;
-  - foreign keys for workspace and attributed user where practical.
+  - foreign keys for workspace and attributed `app_user_id` where practical;
+  - RLS enabled and `anon`/`authenticated` direct table access revoked for platform/Auth.js tables.
 - DB query helpers for tenant-scoped access.
 
 TDD gate:
@@ -174,10 +179,11 @@ Turn the SDK into the programmatic client for the platform while preserving Temp
 Deliverables:
 
 - SDK client for platform API base URL and API token.
-- Submit run API.
+- `AntpathPlatformClient` constructor accepts a default `secrets` bundle (Anthropic key + optional MCP credentials + optional skill references); every method accepts a per-call override.
+- Submit run API (carries inline `secrets`).
 - Get run status/detail API.
 - List metadata events API.
-- List outputs API.
+- List outputs API (returns successful captures and capture failures).
 - Create signed output link API.
 - Cancel run API.
 - Delete run API.
@@ -187,10 +193,11 @@ Deliverables:
 TDD gate:
 
 - Type/contract tests for public SDK API and runtime tests with fake platform responses.
+- Tests asserting `secrets` never appears in serialized request hashes, error metadata, or retry telemetry.
 
 Validation:
 
-- SDK never accepts or stores provider keys for platform runs.
+- SDK does not persist provider keys: it forwards them to the platform per submission, where they are vaulted for the lifetime of one run and then deleted.
 - SDK handles idempotency conflict, unauthorized, quota, and terminal states deterministically.
 
 ## Phase 6: Worker claim loop and state machine
@@ -203,7 +210,7 @@ Deliverables:
 - Optional Postgres `NOTIFY` listener for fast wakeup.
 - Lease claim/release helpers.
 - Lease-guarded status update helper.
-- Per-workspace/provider-key rate limit hooks.
+- Per-workspace rate limit hooks.
 - Fair due-run ordering across workspaces.
 - Cancellation/delete request checks before side effects.
 - Timeout handling.
@@ -295,58 +302,65 @@ Validation:
 
 ## Phase 10: Output capture and Supabase Storage
 
-Capture configured outputs safely.
+Unconditionally capture every artifact the user's Claude session exposes into private Supabase Storage. The workspace storage cap is the only user-visible quota.
 
 Deliverables:
 
-- Output policy from Template/run request.
-- Provider file metadata inspection.
-- Per-file and per-run caps.
-- Streaming download with hard byte cap and abort.
-- Private Supabase Storage upload.
-- Deterministic plus unguessable storage path policy if required.
-- `output_objects` metadata.
-- Workspace storage quota accounting.
-- Per-user attribution from run row.
-- Signed-link BFF action/API.
+- Worker terminal-state list of session-scoped files via the Claude Files API (`GET /v1/files?scope_id=<sessionId>`), returning `id`, `filename`, and `size_bytes`.
+- Workspace storage usage accounting helper.
+- Pre-download quota check: if `(workspace_used + size_bytes) > workspace cap`, write an `output_capture_failures` row with `reason = "workspace_quota_exceeded"` and continue without downloading.
+- Streaming download via `GET /v1/files/{file_id}/content` with a worker-internal safety cap so malformed listing responses cannot OOM the worker; safety-cap aborts are recorded as `output_capture_failures` with `reason = "download_failed"`.
+- Private Supabase Storage upload with workspace-scoped path policy.
+- `output_objects` metadata insert for each successful capture.
+- Per-user attribution frozen from the run row (audit/billing dimension only; not a quota).
+- Signed-link BFF action/API for both `output_objects` rows; capture failures are read-only.
 
 TDD gate:
 
-- Add failing tests for quota checks before download, unknown-size streaming caps, storage metadata, signed-link authorization, and cross-workspace denial.
+- Add failing tests for: pre-download quota check choosing failure-row over download when over cap; safety-cap abort recording a failure row; happy path writing both an `output_objects` row and an audit entry; signed-link authorization; cross-workspace denial; listing returning zero files producing zero rows; listing returning N files with mixed sizes recording the expected mix of successes and failures.
+- Tests must assert there is no user-facing knob for `capture: boolean`, `globs`, or per-file caps.
 
 Validation:
 
-- Oversized outputs do not OOM the worker.
+- Output capture is unconditional and cannot be opted out by the submitter.
+- Oversized output payloads do not OOM the worker.
 - BFF only creates signed links for authorized workspace users/tokens.
+- Dashboard renders successful captures and capture failures side by side.
 
 ## Phase 11: Cleanup, deletion, and retention
 
-Make cleanup and deletion first-class state machines.
+Make cleanup and deletion first-class state machines. Cleanup must also destroy the per-run Vault secret.
 
 Deliverables:
 
-- Cleanup ordering:
-  - provider credentials/vaults;
-  - session files/session where supported;
-  - agent/archive;
-  - environment/archive/delete;
-  - uploaded provider file resources;
-  - local storage only on user deletion.
-- Cleanup retry/backoff.
+- Cleanup ordering (per the architecture decisions doc):
+  - capture session-scoped files (Phase 10);
+  - provider session files / session where supported;
+  - agent / archive;
+  - environment / archive or delete;
+  - skills uploaded for this session and other ephemeral provider resources;
+  - provider Vault/credentials created on Claude's side for MCP wiring;
+  - the antpath per-run Vault entry: `vault.deleteSecret(runs.execution_secret_id)` and clear the column;
+  - local Supabase Storage and metadata only on user deletion.
+- The Claude-side cleanup steps are skipped when `cleanup.claudeSession === "retain"` (or the worker default is `retain`); the per-run Vault deletion still runs.
+- Cleanup retry/backoff with idempotent calls.
 - `cleanup_attempts` records.
 - Cleanup state separate from user-facing run terminal state.
 - User `pending_delete` flow.
 - Workspace deletion flow.
-- Provider key revocation behavior.
+- Audit logs and rate limits for delete/cancel/signed-link/API-token mutations.
 
 TDD gate:
 
-- Add tests for cleanup after success, failure, timeout, cancellation, partial provider creation, duplicate cleanup calls, key revocation, and pending-delete races.
+- Add tests for cleanup after success, failure, timeout, cancellation, partial provider creation, duplicate cleanup calls, and pending-delete races.
+- Add tests that the per-run Vault entry is deleted exactly once and `runs.execution_secret_id` is cleared whether or not Claude-side resources are retained.
+- Add tests that a key rotated on the provider side mid-run produces a `tenant_permanent` failure on the next provider call and that cleanup still runs (and the Vault entry is still deleted).
 
 Validation:
 
 - Cleanup failures surface actionable redacted errors.
 - Hard deletion only happens after cleanup/storage deletion succeeds.
+- The per-run Vault entry is always destroyed at terminal cleanup, regardless of retention knobs.
 
 ## Phase 12: Minimal dashboard
 
@@ -407,18 +421,26 @@ Deliverables:
 - Cleanup in `finally`.
 - Release/readiness docs.
 - Updated README and examples for platform SDK usage.
+- GitHub Actions pnpm CI and local Supabase integration job.
+- Vercel dashboard and Railway worker environment-variable contracts.
+- `pnpm dev:stack` for local Supabase/dashboard/worker startup with fail-fast config checks.
+- Deterministic MCP/skills request-wiring tests for SDK and worker providers.
+- Separately gated public MCP plus Anthropic skill full-session e2e.
 
 TDD gate:
 
 - Live e2e is not a TDD driver for core logic, but must prove final integration before release.
+- MCP/skills request shape, permission policy, and secret non-leakage must be covered by normal deterministic tests before relying on public live e2e.
 
 Validation:
 
 - Full submit -> provider session -> metadata poll -> output capture -> signed link -> cleanup works.
-- Default `npm run lint`, `npm test`, and `npm run build` pass.
+- Public MCP/skills e2e runs through the explicit `pnpm test:e2e:live` command with live credentials, reaches success, emits SDK/provider events, records safe provider IDs, exercises the default cleanup path, and exercises the `cleanup.claudeSession = "retain"` override path with a follow-up provider API check.
+- Default `pnpm lint`, `pnpm test`, and `pnpm build` pass.
 
 ## Backlog
 
+- Persistent workspace-level provider connections (saved Anthropic keys with rotation/revocation/re-use). The MVP submission contract carries the key inline per run.
 - Provider webhooks as wakeup/reconciliation accelerator.
 - SSE live event stream for richer dashboard UI.
 - Supabase Realtime with explicit Auth.js-to-Supabase authorization design.