antpath 0.1.1 → 0.2.0

package/references/implementation-plan.md

@@ -8,7 +8,7 @@ scope: platform MVP
 
 ## Goal
 
-Convert antpath from an SDK-only package into a TypeScript workspace containing:
+Convert antpath from an SDK-only package into a pnpm TypeScript workspace containing:
 
 - a platform SDK in `packages/sdk`;
 - a dashboard app in `apps/dashboard`;
@@ -21,27 +21,30 @@ Implementation is test-driven. Each behavior starts with the narrowest failing t
 
 ## Acceptance criteria
 
-- The repository is an npm workspace and the current SDK is moved to `packages/sdk`.
+- The repository is a pnpm workspace and the current SDK is moved to `packages/sdk`.
 - Dashboard and worker app folders exist with build/test integration.
 - Auth.js authenticates dashboard users.
 - SDK API tokens authenticate programmatic clients.
 - Workspaces are the tenant boundary.
 - BFF/server actions scope every dashboard/API operation by workspace membership or API-token scope.
 - Supabase service-role credentials are never exposed to browser/client bundles.
-- Supabase Postgres stores durable run, attempt, provider resource, event, output, cleanup, usage, workspace, membership, and provider connection metadata.
-- Provider keys are stored encrypted through Supabase Vault and resolved only by trusted server/worker code.
-- Run submission idempotency is enforced by workspace and request hash.
+- Supabase Postgres stores durable run, attempt, provider resource, event, output, output-capture-failure, cleanup, usage, workspace, and membership metadata. There is no persistent `provider_connections` table in MVP.
+- Per-run secrets bundles (Anthropic key, optional MCP credentials, optional skill references) arrive inline on submission, are encrypted through Supabase Vault for the lifetime of that single run, and are deleted at terminal cleanup. The Vault entry is the only durable trace of the user's provider key.
+- Run submission idempotency is enforced by workspace and request hash. The hash excludes the `secrets` block so re-submitting the same logical run with a new key still matches.
 - Workers claim due runs with leases and `FOR UPDATE SKIP LOCKED`.
 - Multiple workers can run concurrently without duplicate lifecycle ownership.
 - Worker polling recovers from missed `NOTIFY`, worker restart, deploy, and expired leases.
 - Worker creates per-run provider resources and journals them for cleanup.
 - Worker polls provider status and events and stores only redacted metadata.
-- Output capture writes configured files to private Supabase Storage within per-file, per-run, and workspace quotas.
+- Output capture is unconditional: every artifact Claude exposes on the run's session is written to private Supabase Storage, bounded only by the workspace storage cap. Files that would exceed the cap are persisted as `output_capture_failures` rows.
 - BFF returns signed output links only after workspace authorization.
-- Cleanup runs after terminal provider state and output capture.
+- Cleanup runs after terminal provider state and output capture, deletes the per-run Vault entry, and (by default) deletes Claude-side resources. `cleanup.claudeSession: "retain"` only affects Claude-side resources.
 - Reconciliation can recover intended provider resources after partial worker crashes.
 - User deletion is pending/soft until cleanup and storage deletion are complete.
 - Exact tier/cap values are configurable through environment variables with conservative defaults and run-level snapshots.
+- Security-sensitive BFF/API actions are audited and rate-limited.
+- Workspace storage quota and workspace deletion are enforced before provider/storage side effects proceed.
+- CI runs pnpm lint/test/build/package checks plus a local Supabase integration job.
 - Tests follow the accepted taxonomy: deterministic unit tests (including fakes and sanitized recorded snapshots), live external-system integration tests with no skip flags, and top-to-bottom live e2e tests.
 
 ## Phase 1: Workspace foundation
@@ -50,16 +53,16 @@ Create the workspace structure without changing runtime behavior.
 
 Deliverables:
 
-- Root npm workspace configuration.
+- Root pnpm workspace configuration.
 - Current SDK moved to `packages/sdk`.
 - `apps/dashboard` placeholder.
 - `apps/worker` placeholder.
 - Shared TypeScript config/build/test commands.
 - Existing SDK exports preserved or intentionally migrated with compatibility notes.
 - Root validation commands:
-  - `npm run lint`
-  - `npm test`
-  - `npm run build`
+  - `pnpm lint`
+  - `pnpm test`
+  - `pnpm build`
 
 TDD gate:
 
@@ -113,23 +116,25 @@ Deliverables:
 
 - Migration framework.
 - Tables:
-  - `users`;
+  - Auth.js adapter tables: `users`, `accounts`, `sessions`, `verification_token`;
+  - antpath `app_users`;
   - `workspaces`;
   - `workspace_memberships`;
   - `api_tokens`;
-  - `provider_connections`;
-  - `runs`;
+  - `runs` (includes `execution_secret_id` referencing the per-run Vault entry);
   - `run_attempts`;
   - `provider_resources`;
   - `run_events`;
   - `output_objects`;
+  - `output_capture_failures`;
   - `cleanup_attempts`;
   - `usage_ledger`.
 - Constraints:
   - unique `(workspace_id, idempotency_key)` on runs;
   - request-hash conflict handling;
   - unique `(run_attempt_id, provider_event_id)` on events;
-  - foreign keys for workspace and attributed user where practical.
+  - foreign keys for workspace and attributed `app_user_id` where practical;
+  - RLS enabled and `anon`/`authenticated` direct table access revoked for platform/Auth.js tables.
 - DB query helpers for tenant-scoped access.
 
 TDD gate:
@@ -174,10 +179,11 @@ Turn the SDK into the programmatic client for the platform while preserving Temp
 Deliverables:
 
 - SDK client for platform API base URL and API token.
-- Submit run API.
+- `AntpathPlatformClient` constructor accepts a default `secrets` bundle (Anthropic key + optional MCP credentials + optional skill references); every method accepts a per-call override.
+- Submit run API (carries inline `secrets`).
 - Get run status/detail API.
 - List metadata events API.
-- List outputs API.
+- List outputs API (returns successful captures and capture failures).
 - Create signed output link API.
 - Cancel run API.
 - Delete run API.
@@ -187,10 +193,11 @@ Deliverables:
 TDD gate:
 
 - Type/contract tests for public SDK API and runtime tests with fake platform responses.
+- Tests asserting `secrets` never appears in serialized request hashes, error metadata, or retry telemetry.
 
 Validation:
 
-- SDK never accepts or stores provider keys for platform runs.
+- SDK does not persist provider keys: it forwards them to the platform per submission, where they are vaulted for the lifetime of one run and then deleted.
 - SDK handles idempotency conflict, unauthorized, quota, and terminal states deterministically.
 
 ## Phase 6: Worker claim loop and state machine
@@ -203,7 +210,7 @@ Deliverables:
 - Optional Postgres `NOTIFY` listener for fast wakeup.
 - Lease claim/release helpers.
 - Lease-guarded status update helper.
-- Per-workspace/provider-key rate limit hooks.
+- Per-workspace rate limit hooks.
 - Fair due-run ordering across workspaces.
 - Cancellation/delete request checks before side effects.
 - Timeout handling.
@@ -295,58 +302,65 @@ Validation:
 
 ## Phase 10: Output capture and Supabase Storage
 
-Capture configured outputs safely.
+Unconditionally capture every artifact the user's Claude session exposes into private Supabase Storage. The workspace storage cap is the only user-visible quota.
 
 Deliverables:
 
-- Output policy from Template/run request.
-- Provider file metadata inspection.
-- Per-file and per-run caps.
-- Streaming download with hard byte cap and abort.
-- Private Supabase Storage upload.
-- Deterministic plus unguessable storage path policy if required.
-- `output_objects` metadata.
-- Workspace storage quota accounting.
-- Per-user attribution from run row.
-- Signed-link BFF action/API.
+- Worker terminal-state list of session-scoped files via the Claude Files API (`GET /v1/files?scope_id=<sessionId>`), returning `id`, `filename`, and `size_bytes`.
+- Workspace storage usage accounting helper.
+- Pre-download quota check: if `(workspace_used + size_bytes) > workspace cap`, write an `output_capture_failures` row with `reason = "workspace_quota_exceeded"` and continue without downloading.
+- Streaming download via `GET /v1/files/{file_id}/content` with a worker-internal safety cap so malformed listing responses cannot OOM the worker; safety-cap aborts are recorded as `output_capture_failures` with `reason = "download_failed"`.
+- Private Supabase Storage upload with workspace-scoped path policy.
+- `output_objects` metadata insert for each successful capture.
+- Per-user attribution frozen from the run row (audit/billing dimension only; not a quota).
+- Signed-link BFF action/API for both `output_objects` rows; capture failures are read-only.
 
 TDD gate:
 
-- Add failing tests for quota checks before download, unknown-size streaming caps, storage metadata, signed-link authorization, and cross-workspace denial.
+- Add failing tests for: pre-download quota check choosing failure-row over download when over cap; safety-cap abort recording a failure row; happy path writing both an `output_objects` row and an audit entry; signed-link authorization; cross-workspace denial; listing returning zero files producing zero rows; listing returning N files with mixed sizes recording the expected mix of successes and failures.
+- Tests must assert there is no user-facing knob for `capture: boolean`, `globs`, or per-file caps.
 
 Validation:
 
-- Oversized outputs do not OOM the worker.
+- Output capture is unconditional and cannot be opted out by the submitter.
+- Oversized output payloads do not OOM the worker.
 - BFF only creates signed links for authorized workspace users/tokens.
+- Dashboard renders successful captures and capture failures side by side.
 
 ## Phase 11: Cleanup, deletion, and retention
 
-Make cleanup and deletion first-class state machines.
+Make cleanup and deletion first-class state machines. Cleanup must also destroy the per-run Vault secret.
 
 Deliverables:
 
-- Cleanup ordering:
-  - provider credentials/vaults;
-  - session files/session where supported;
-  - agent/archive;
-  - environment/archive/delete;
-  - uploaded provider file resources;
-  - local storage only on user deletion.
-- Cleanup retry/backoff.
+- Cleanup ordering (per the architecture decisions doc):
+  - capture session-scoped files (Phase 10);
+  - provider session files / session where supported;
+  - agent / archive;
+  - environment / archive or delete;
+  - skills uploaded for this session and other ephemeral provider resources;
+  - provider Vault/credentials created on Claude's side for MCP wiring;
+  - the antpath per-run Vault entry: `vault.deleteSecret(runs.execution_secret_id)` and clear the column;
+  - local Supabase Storage and metadata only on user deletion.
+- The Claude-side cleanup steps are skipped when `cleanup.claudeSession === "retain"` (or the worker default is `retain`); the per-run Vault deletion still runs.
+- Cleanup retry/backoff with idempotent calls.
 - `cleanup_attempts` records.
 - Cleanup state separate from user-facing run terminal state.
 - User `pending_delete` flow.
 - Workspace deletion flow.
-- Provider key revocation behavior.
+- Audit logs and rate limits for delete/cancel/signed-link/API-token mutations.
 
 TDD gate:
 
-- Add tests for cleanup after success, failure, timeout, cancellation, partial provider creation, duplicate cleanup calls, key revocation, and pending-delete races.
+- Add tests for cleanup after success, failure, timeout, cancellation, partial provider creation, duplicate cleanup calls, and pending-delete races.
+- Add tests that the per-run Vault entry is deleted exactly once and `runs.execution_secret_id` is cleared whether or not Claude-side resources are retained.
+- Add tests that a key rotated on the provider side mid-run produces a `tenant_permanent` failure on the next provider call and that cleanup still runs (and the Vault entry is still deleted).
 
 Validation:
 
 - Cleanup failures surface actionable redacted errors.
 - Hard deletion only happens after cleanup/storage deletion succeeds.
+- The per-run Vault entry is always destroyed at terminal cleanup, regardless of retention knobs.
 
 ## Phase 12: Minimal dashboard
 
@@ -407,18 +421,26 @@ Deliverables:
 - Cleanup in `finally`.
 - Release/readiness docs.
 - Updated README and examples for platform SDK usage.
+- GitHub Actions pnpm CI and local Supabase integration job.
+- Vercel dashboard and Railway worker environment-variable contracts.
+- `pnpm dev:stack` for local Supabase/dashboard/worker startup with fail-fast config checks.
+- Deterministic MCP/skills request-wiring tests for SDK and worker providers.
+- Separately gated public MCP plus Anthropic skill full-session e2e.
 
 TDD gate:
 
 - Live e2e is not a TDD driver for core logic, but must prove final integration before release.
+- MCP/skills request shape, permission policy, and secret non-leakage must be covered by normal deterministic tests before relying on public live e2e.
 
 Validation:
 
 - Full submit -> provider session -> metadata poll -> output capture -> signed link -> cleanup works.
-- Default `npm run lint`, `npm test`, and `npm run build` pass.
+- Public MCP/skills e2e runs through the explicit `pnpm test:e2e:live` command with live credentials, reaches success, emits SDK/provider events, records safe provider IDs, exercises the default cleanup path, and exercises the `cleanup.claudeSession = "retain"` override path with a follow-up provider API check.
+- Default `pnpm lint`, `pnpm test`, and `pnpm build` pass.
 
 ## Backlog
 
+- Persistent workspace-level provider connections (saved Anthropic keys with rotation/revocation/re-use). The MVP submission contract carries the key inline per run.
 - Provider webhooks as wakeup/reconciliation accelerator.
 - SSE live event stream for richer dashboard UI.
 - Supabase Realtime with explicit Auth.js-to-Supabase authorization design.