vgxness 1.5.1 → 1.6.0

package/docs/roadmap.md

@@ -0,0 +1,88 @@
+# Roadmap
+
+This document tracks planned work that is not yet shipped. The [Architecture](./architecture.md) document describes what is built; this is what is next. Items are grouped by area and ordered roughly by impact and dependency.
+
+## Runtime execution
+
+The lifecycle, policy recording, approval records, and retry policy are all in place. What is still test-only is the actual executor that turns a reserved attempt into a real-world side effect.
+
+- **Real provider/tool executor.** The current `RunService.executeOperation(...)` takes an injected `RunOperationExecutor`; production code still uses fake/deterministic executors in tests. The plan is to ship a sandboxed executor that respects the `ExecutionIsolationPlan` (`workspace`/`git-worktree`/`process-sandbox`) and the SDD phase permission matrix.
+- **CLI/MCP orchestration for `resume-after-approval`.** Once the executor is real, the `vgxness_run_resume_inspect` and `vgxness_run_resume_gate` tools need an `apply` partner that calls the executor only after a human resolves the pending approval.
+- **Sandbox/worktree execution strategies.** `src/runs/sandbox-worktree-planning.ts` produces a plan; the plan needs to materialize into a real worktree or sandbox. Symlink and realpath hardening is already in the policy evaluator; the next step is wiring it to a runner.
+- **Richer verification evidence summaries.** Verification currently records `pass`/`fail`/`skipped`; the next step is to summarize per-task verification into the run, task list, and SDD cockpit so blocked phases have concrete evidence.
+
+## Code runtime
+
+The four modes (`inspect`/`plan`/`craft-preview`/`craft`) and the 19 tools are in. What is left is provider coverage, ergonomics, and the OpenTUI shell promotion.
+
+- **Native Anthropic provider.** The code runtime speaks to any OpenAI-compatible endpoint. A native Anthropic adapter would remove the need to route Claude through an OpenAI-compatible bridge.
+- **More providers.** The adapter contract is small and provider-neutral; additional adapters (Anthropic, Google, local model servers) can be added incrementally.
+- **`vgxcode` OpenTUI shell promotion.** The shell is currently root-owned during development. To promote it, we need: deterministic event contracts, snapshot tests, and a packaging path that does not require `bun src/cli/tui/opentui/code/index.ts`.
+- **Workspace-executor for the `craft` mode.** Workspace mutations (`apply_patch`, `create_file`, `update_file`, `delete_file`) currently use the `WorkspaceToolExecutor` with policy gating. The next step is a real file-system backend with rollback on failure.
+- **Repair loop.** The `--verification repair` option exists in the CLI but is not wired to a loop in the runtime.
+
+## SDD governance
+
+The hard acceptance gate and the cockpit blockers are in. The remaining work is mostly UX and stronger cross-linking.
+
+- **Verification evidence linked to the cockpit.** The cockpit currently returns blockers; surfacing the per-phase verification evidence (pass/fail/skipped counts, last-run timestamps) would make "why is this blocked" answers more useful.
+- **Per-phase model/profile routing in the cockpit.** The manager profile overlay exists. The cockpit should recommend a model for the next phase based on the active overlay.
+- **Migration of `openspec/`-style workflows.** Some users bring artifacts from other tools. A formal `import` path through the artifact portability service would help, but it should not silently convert external artifacts into accepted SDD phases.
+
+## Skills and agents
+
+The registries, version model, payloads, and improvement-proposal lifecycle are in.
+
+- **Skill evaluation harness.** `013_skill_evaluation_scenarios.sql` is the storage; the runtime that runs scenarios against a proposed skill version is not yet built.
+- **Skill improvement suggestion agent.** Skill proposals today are user-driven. A trace-driven candidate detector is planned but explicitly out of scope per the "no silent skill mutation" rule.
+- **Canonical manifest v7.** v6 is in the tree (WIP commit). v7 would address the uncommitted `canonical-agent-manifest.ts` and `canonical-agent-projection.ts` work and ship a stable, validated manifest as the source of truth.
+
+## Storage and portability
+
+SQLite, scopes, migrations, and the run snapshot export are in.
+
+- **`import` path for SDD artifacts.** The `ArtifactPortabilityService` exports; an import path that re-creates acceptance records with explicit human re-acceptance is planned.
+- **Sensitive-data redaction during export.** `src/export/redaction.ts` exists. Wiring it as a default into the export path and a CLI flag (`--redact`) is the next step.
+- **Database upgrade tooling.** Forward-only migrations work; downgrade requires a snapshot. A small CLI helper for "is my DB on the latest migration?" would be useful for support.
+
+## TUI
+
+The main menu and setup screens are in via OpenTUI. The dashboard directory is empty in the current tree.
+
+- **TUI dashboard screen.** Real-time view of active runs, pending approvals, and SDD blockers. Should match the cockpit surface from MCP.
+- **TUI runs screen.** Run list, drill into a run, see events/approvals/attempts, with read-only safe actions.
+- **TUI approvals screen.** Resolve pending approvals directly from the TUI, with the same audit trail as `vgxness_sdd_accept_artifact`.
+
+## Observability and evaluation
+
+95 test files cover the eval targets in v1.5.1.
+
+- **Eval gate.** A `vgxness verify --evals` runner that orchestrates a quality gate beyond `bun run verify`. The architecture document lists 11 eval targets; lifting them into a single runner would close the loop.
+- **Token/cost tracking.** A nice-to-have that the trace model already supports structurally (`ProviderTokenInput`).
+- **Timeline export.** `RunSnapshotPackageV1` exists. A `runs timeline <id> --format html|md|json` CLI would make sharing easier.
+
+## Provider coverage
+
+- **Claude Code runtime validation.** Project-local install planning/apply is supported behind explicit run preflight. Remaining work is optional manual runtime smoke guidance or future live smoke tooling; it must stay out of normal CI.
+- **Antigravity and other custom adapters.** Listed as placeholders; a real adapter would unblock those workflows.
+- **Provider doctor upgrades.** Doctor checks could surface real config drift, not just existence of expected files.
+
+## Out of scope (still)
+
+These remain future expansion areas per the [PRD](./prd.md):
+
+- Cloud sync across machines.
+- Team/shared memory spaces.
+- Hosted web console.
+- Distributed agent workers.
+- Hosted observability and evaluation UI.
+- Skill marketplace or shared skill catalog.
+
+## How work gets into this list
+
+When the architecture drifts from the code, or when a real product gap is identified, the right path is:
+
+1. Open an SDD change with `vgxness sdd status --project <project> --change <id>` (or a fresh change id).
+2. Move the item from here to an `explore` artifact.
+3. Walk the SDD phases: `explore → proposal → spec → design → tasks → apply-progress → verify → archive`.
+4. Once shipped, retire the item from this roadmap and update [Architecture](./architecture.md) and the [CHANGELOG](../../CHANGELOG.md) to reflect the new state.

package/docs/safety.md

@@ -0,0 +1,147 @@
+# Safety model
+
+VGXNESS treats safety as a core domain, not as adapter-specific behavior. This document describes the three layers that make up the safety model: the **policy evaluator** for the control plane, the **per-SDD-phase permission matrix**, and the **code runtime approval flow** with redactors.
+
+If a tool, surface, or new provider wants to take an action, the action must:
+
+1. Be classified into a permission category.
+2. Resolve through the policy evaluator or the per-SDD-phase matrix.
+3. If the decision is `ask`, surface an approval prompt through the configured approval channel.
+4. Record the decision in the run event stream regardless of outcome.
+
+## Permission categories
+
+The category list lives in `src/permissions/schema.ts` (`permissionCategories`). The default policy (`defaultPermissionPolicy` in `src/permissions/policy-evaluator.ts`) defines the baseline decision for each.
+
+| Category | Default | Risky | Notes |
+|---|---|---|---|
+| `read` | `allow` | no | Files, memory, artifacts. Allowed only when path stays inside `workspaceRoot`. |
+| `edit` | `ask` | yes | Write/patch/modify files. |
+| `implementation-edit` | `ask` | yes | Edits tied to a SDD `apply-progress` phase. |
+| `spec-write` | `ask` | yes | Writes to a spec artifact. |
+| `design-write` | `ask` | yes | Writes to a design artifact. |
+| `task-write` | `ask` | yes | Writes to a task artifact. |
+| `shell` | `ask` | yes | Commands, scripts, package managers. |
+| `test-run` | `ask` | yes | Test execution. |
+| `install` | `ask` | yes | Dependency installation mutates local state. |
+| `network` | `ask` | yes | Web fetch, API calls, package downloads. |
+| `git` | `ask` | yes | Status, diff, branch, commit. |
+| `git-write` | `ask` | yes | Push, merge, branch mutation. |
+| `memory` | `ask` | yes | Memory read/write/search. |
+| `memory-write` | `ask` | yes | Memory upsert/update. |
+| `external-directory` | `deny` | yes | Access outside project/user-approved roots. Cannot be relaxed by agent overrides. |
+| `provider-tool` | `ask` | yes | Opaque adapter/provider tool calls. |
+| `secrets` | `deny` | yes | Environment variables, credentials, tokens. |
+
+The risky categories (`isRiskyPermissionCategory`) get a forced `ask` even when an agent override would otherwise allow the category.
+
+## Decisions
+
+Every permission request resolves to one of three decisions:
+
+- `allow` — execute and record `succeeded` or `failed`.
+- `ask` — create a pending approval record, record `pending-approval`, and pause until the approval is resolved.
+- `deny` — record `blocked` and do not invoke the executor.
+
+In addition, the SDD-phase permission matrix uses four `PermissionMode` values:
+
+- `allow` — pass through.
+- `audit` — log and pass through.
+- `require-preflight` — must run preflight and receive an `allow` decision before the operation executes.
+- `deny` — block.
+
+## Per-SDD-phase permission matrix
+
+The matrix (`sddPhasePermissionMatrix`, version `sdd-phase-permissions-v1`) gives every (phase, category) pair a mode. The matrix version is exported as a constant so consumers can compare.
+
+| Phase | Distinctive modes (vs. the planning baseline) |
+|---|---|
+| `explore`, `proposal`, `spec`, `design`, `tasks`, `archive` | Planning baseline. Reads allowed; edits denied; spec/design/task writes, shell, install, network, git, memory all `require-preflight`; provider-tool `audit`; external-directory and secrets `deny`. |
+| `apply-progress` | Edits and `implementation-edit` are `require-preflight` (instead of `deny`). Shell and `test-run` are `require-preflight`. |
+| `verify` | Edits and `implementation-edit` stay `deny`. Shell and `test-run` are `require-preflight`. |
+
+Workspace reads are allowed only when the target path stays inside `workspaceRoot`. The evaluator resolves the real path with `realpathSync` to defeat symlink escapes and refuses to relax workspace boundary denials through agent overrides.
+
+## Code runtime approval flow
+
+The code runtime layers a second, finer-grained decision on top of the policy evaluator. Per-tool definitions declare whether a tool is `read`, `confirm`, or `restricted` (see [Code runtime](./code-runtime.md)). The approval coordinator (`src/code/runtime/approval-coordinator.ts`) wires that to a broker and a gateway.
+
+```text
+tool call
+   │
+   ▼
+PolicyApprovalBroker  ──►  ConservativePermissionGateway.evaluate(tool, mode, context)
+   │                              │
+   │                              ├── allow  → execute, emit CodeRuntimeEvent(succeeded)
+   │                              ├── ask    → ApprovalPrompt
+   │                              │            ├── stdio channel: read line from stdin
+   │                              │            └── auto channel:   injected broker (e.g. MCP client)
+   │                              └── deny   → emit CodeRuntimeEvent(blocked), return error
+   ▼
+CodeRuntimeEventSink  (stream JSONL to consumer; OpenTUI shell, scripts, tests)
+```
+
+The gateway is conservative by default: a non-`read` tool in a non-`apply-progress` phase defaults to `ask`. Even with `--approval-policy allow`, the gateway can re-promote to `ask` for risky categories or workspace boundary issues.
+
+## Redaction
+
+`src/code/reporting/redaction.ts` ships three helpers used by every consumer that emits prompts, reports, checkpoints, transcripts, or memory saves.
+
+| Helper | What it does |
+|---|---|
+| `redactSecrets(value: string): string` | Replaces secret-shaped substrings with `[REDACTED]`. Patterns include `*TOKEN=...`, `*SECRET=...`, `*KEY=...`, `*PASSWORD=...`, `sk_*`/`pk_*` keys, `Bearer ...` headers, and PEM private keys. |
+| `redactJson(value: JsonValue): JsonValue` | Recursively walks JSON, redacts string values, and replaces values whose key matches `(token\|secret\|password\|api.?key\|authorization\|credential)` with the literal `"[REDACTED]"`. |
+| `omitSensitiveCommandOutput<T extends JsonValue>(value: T): JsonValue` | For tool results that include `stdout`/`stderr`, replaces those fields with `"[omitted by default]"` unless the consumer explicitly opts in. |
+
+These helpers are deterministic and are the only place secret-shaped values should be stripped. Tool results that include shell output must use `omitSensitiveCommandOutput` before they are recorded into the run event stream or rendered into a checkpoint.
+
+## Approval lifecycle
+
+Approvals are first-class records linked to a permission-decision event:
+
+```text
+executeOperation(...)
+   │
+   ▼
+evaluate permission
+   │
+   ├── allow  → no approval record, executor runs
+   ├── deny   → no approval record, executor blocked
+   └── ask    → pending approval record + reserved operation attempt
+                    │
+                    ▼
+              human resolves approval (approved | rejected | cancelled)
+                    │
+                    ▼
+              resumeApprovedOperation({ approvalId, executor })
+                    │
+                    ▼
+              reserved attempt → succeeded | failed | abandoned
+```
+
+Retry policy evaluation is separate from execution. `vgxness_run_resume_gate` evaluates the policy and returns an `OperationRetryDecision` (`allowed`/`reasonCode`/`reason`/attempt metadata). The default policy is `never`, so any prior `reserved`/`succeeded`/`failed`/`abandoned` attempt blocks later resume before the executor is invoked.
+
+| Policy | Allows a new attempt after | Always blocks |
+|---|---|---|
+| `never` | No previous attempt only | `reserved`, `succeeded`, `failed`, `abandoned` |
+| `after-abandoned` | latest attempt is `abandoned` | active `reserved`, `succeeded`, `failed` |
+| `after-failure` | latest attempt is `failed` | active `reserved`, `succeeded`, `abandoned` |
+| `after-failure-or-abandoned` | latest attempt is `failed` or `abandoned` | active `reserved`, `succeeded` |
+
+`RunService.abandonReservedOperationAttempt({ attemptId, actor, reason })` is recovery-only: it transitions a stuck attempt to `abandoned` and appends an `operation-execution` audit event. It does not call an executor or retry.
+
+## Safety boundaries (enforced everywhere)
+
+- Read-only/preview commands must stay non-mutating: setup plans, MCP setup previews, OpenCode previews, workflow previews, status views, and the natural-language orchestrator preview never write provider config, call providers, or install global memory.
+- Provider config writes (`opencode.json`, `.opencode/`, `.claude/`, `CLAUDE.md`) require explicit consent (`--yes` or an equivalent confirmed flow) plus backup/rollback behavior.
+- The control plane and the code runtime do not create or write `openspec/`. SDD artifacts are stored through the local SQLite artifact service under canonical topic keys.
+- Human acceptance is distinct from artifact presence: `vgxness_sdd_accept_artifact` records explicit human-only acceptance; saving a draft never implies acceptance.
+- Unrelated user work is preserved. Workspace boundary denials cannot be relaxed by agent or subagent overrides.
+
+## Tests
+
+The policy and approval flow are covered by:
+
+- `test/permissions/policy-evaluator.test.ts` — decisions, defaults, workspace boundary, SDD phase matrix.
+- `test/runs/` — approval records, reserved attempts, retry policy, abandonment, resume inspect/gate.
+- `test/code/` — approval broker, tool definitions, runtime approval flow.

package/docs/storage.md

@@ -0,0 +1,93 @@
+# Storage
+
+VGXNESS keeps all durable state in local SQLite. There is no required cloud account, no remote sync in v1.5.1, and no implicit sharing between scopes. Project data and personal/global data must not be collapsed into one scope.
+
+## Database location
+
+The default is a per-user global database. The path is resolved by `resolveMemoryDatabasePath(...)` in `src/memory/storage-paths.ts` with this precedence:
+
+1. `--db <path>` flag (`source: 'flag'`).
+2. `VGXNESS_DB_PATH` env var (`source: 'environment'`).
+3. Global default (`source: 'global-default'`).
+
+| Platform | Global default |
+|---|---|
+| macOS | `~/Library/Application Support/vgxness/memory.sqlite` |
+| Linux | `${XDG_DATA_HOME:-~/.local/share}/vgxness/memory.sqlite` |
+| Windows | `%LOCALAPPDATA%\\vgxness\\memory.sqlite` when available; otherwise `%APPDATA%\\vgxness\\memory.sqlite` |
+
+If the global default cannot be resolved (for example, `HOME` is not set on Linux), the resolver returns a `store_unavailable` error rather than falling back to the current working directory. The fix is to pass `--db` or set `VGXNESS_DB_PATH`.
+
+The parent directory is created on demand by `prepareMemoryDatabasePath(...)`. Existing project-local databases remain compatible; opt in explicitly when needed:
+
+```bash
+vgxness memory search --project vgxness --db .vgx/memory.sqlite
+```
+
+## Runtime engine
+
+Local SQLite storage is supported through `bun:sqlite`. Node.js is not the storage runtime; it remains development/build/test tooling. The release gate `bun run verify:bun-sqlite` proves SQLite readiness on a clean database by:
+
+- Copying the source migrations into a temporary directory.
+- Applying them against a temporary database.
+- Checking foreign keys, busy timeout, FTS/search, transaction rollback, integrity, and cleanup.
+
+## Migration layout
+
+SQL migrations live in `src/memory/sqlite/migrations/` and are applied in lexicographic order. The package build copies them into `dist/memory/sqlite/migrations` so installed bins can apply them on first use. The full list as of v1.5.1:
+
+| # | File | Adds |
+|---|---|---|
+| 001 | `001_initial.sql` | Base memory observations table. |
+| 002 | `002_observation_revisions.sql` | Observation revision history. |
+| 003 | `003_agent_registry.sql` | Agent and subagent registry. |
+| 004 | `004_run_runtime.sql` | Run records, events, checkpoints. |
+| 005 | `005_run_approvals.sql` | Approval records linked to permission-decision events. |
+| 006 | `006_run_operation_attempts.sql` | Reserved operation attempts. |
+| 007 | `007_abandoned_operation_attempts.sql` | Recovery-only `abandoned` state. |
+| 008 | `008_run_execution_plan_events.sql` | Execution isolation plan events. |
+| 009 | `009_multiple_operation_attempts.sql` | Multiple ordered attempts per approval. |
+| 010 | `010_skill_registry.sql` | Skill identity, versions, source metadata. |
+| 011 | `011_skill_usage_resolution_outcomes.sql` | Skill usage records from resolution. |
+| 012 | `012_skill_improvement_proposals.sql` | Versioned, reviewable skill proposals. |
+| 013 | `013_skill_evaluation_scenarios.sql` | Skill evaluation harness. |
+| 014 | `014_manager_profile_overlays.sql` | Manager profile overlays. |
+| 015 | `015_artifact_metadata.sql` | Artifact metadata for SDD. |
+
+## Scopes
+
+Two scopes share the database:
+
+| Scope | Purpose |
+|---|---|
+| `project` | Repo-specific memory, SDD artifacts, run history, project agents/skills, project manager profile overlay. |
+| `personal` | User preferences, reusable skills, cross-project patterns, personal manager profile overlay. |
+
+Scopes are not stored in separate databases in v1.5.1; they are columns on the relevant tables. Code that wants to read or write a scope must pass it explicitly. Memory writes default to `project`; pass `scope: "personal"` for personal/global observations.
+
+## Tables and lifecycle
+
+The schema follows a small set of conventions:
+
+- **Memory observations** are immutable for content; updates create revisions tracked by `observation_revisions`. `topicKey` is the durable upsert key: re-saving with the same key updates the existing observation rather than creating a duplicate.
+- **SDD artifacts** are stored under canonical topic keys `sdd/{change}/{phase}`. Acceptance is recorded separately on `SddArtifactAcceptanceRecord` and is not a content column.
+- **Runs** are append-only in spirit. `RunRecord.status` transitions through the lifecycle described in [Safety model](./safety.md); `RunEvent` is the audit trail; `RunCheckpoint` is the resumable JSON state. Terminal runs cannot be finalized again, and the final `outcome` must match the terminal `status`.
+- **Operation attempts** are reserved through `RunService.executeOperation(...)` and finalized in one transaction. `reserved` is exclusive; once finalized, a new attempt requires a fresh reservation and policy admission.
+- **Skills** are versioned, with `currentVersionId` pointing at the active version. Skill improvement proposals do not mutate the active skill until `applySkillImprovementProposal(...)` is called on an `approved` proposal.
+- **Manager profile overlays** are a thin layer over the canonical agent manifest; they record per-user/per-project model and prompt-contract preferences without mutating the built-in manifest.
+
+## Backup and recovery
+
+There is no automatic backup of the SQLite database in v1.5.1. The recommended pattern is to treat the database file as a normal file in the user's data directory and use OS-level snapshots if needed.
+
+Provider config (OpenCode) has its own backup/rollback path through `vgxness setup rollback --backup <path>`. That is separate from the database and is documented in the [CLI reference](./cli.md).
+
+## Wiping and migrating
+
+To reset state for a project, point `--db` at a new path; do not delete a database file that another `vgxness` process might still be using. Migrations are forward-only and idempotent within a single database version. If a downgrade is required, restore the previous database file from a snapshot — VGXNESS does not support automatic down-migrations.
+
+## Safety
+
+- The database file holds potentially sensitive content (memory observations, run transcripts, prompt excerpts). Treat it as you would any other local credential-bearing file. The redaction helpers in `src/code/reporting/redaction.ts` are applied at the prompt/report/checkpoint boundary, not at the storage boundary.
+- `--db` accepts any path the calling user can write to. VGXNESS refuses to write through symlinks that escape the parent directory of the resolved path.
+- `*.sqlite*`, `.vgx/`, and `.opencode/` are git-ignored.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "vgxness",
-  "version": "1.5.1",
+  "version": "1.6.0",
   "description": "CLI and MCP control plane for guided AI-agent workflows, SDD, memory, and OpenCode setup.",
   "license": "SEE LICENSE IN LICENSE",
   "repository": {