agentic-orchestrator 0.1.28 → 0.2.0

package/specs/002-scope-aware-lease/quickstart.md

@@ -0,0 +1,150 @@
+# Quickstart: Scope-Aware Run Lease
+
+**Feature Branch**: `002-scope-aware-lease`
+
+---
+
+## Overview
+
+The scope-aware run lease feature allows multiple orchestrator runs to proceed in parallel, as long as their declared codebase scopes do not overlap. This guide covers how to use the feature from the CLI and programmatically.
+
+---
+
+## CLI Usage
+
+### Running with an explicit scope
+
+Pass `--scope <path>` one or more times to declare which parts of the codebase a run will touch:
+
+```bash
+# Run A: targets authentication subsystem
+aop run --spec specs/auth-refresh.md --scope packages/auth/
+
+# Run B: targets payments subsystem (can run in parallel with Run A)
+aop run --spec specs/payment-flow.md --scope packages/payments/
+
+# Run C: targets multiple subsystems
+aop run --spec specs/shared-utils.md \
+  --scope packages/shared/ \
+  --scope packages/utils/
+```
+
+If no `--scope` flag is provided, the run uses **unknown scope** (`null`) — the conservative fallback that blocks against all other active runs (same behavior as before this feature).
+
+### Explicit empty scope
+
+If your run provably touches nothing (e.g., a read-only audit run), declare an empty scope:
+
+```bash
+aop run --spec specs/audit.md --scope-empty
+```
+
+An empty scope (`{ paths: [], plan_section_ids: [] }`) never conflicts with any other scope.
+
+---
+
+## How Conflict Detection Works
+
+When a new run attempts to acquire a lease:
+
+1. The system reads the **Lease Registry** (`.aop/runtime/lease-registry.json`).
+2. Expired entries are pruned (entries where `lease_expires_at < now`).
+3. The incoming scope is compared against each active lease's scope.
+4. If **any overlap** is detected, the acquisition is denied with `LEASE_SCOPE_CONFLICT`.
+5. If **no overlap**, the lease is granted and the registry is updated.
+
+### Overlap rules
+
+| Incoming                     | Active                 | Result                                   |
+| ---------------------------- | ---------------------- | ---------------------------------------- |
+| `null` (unknown)             | anything               | CONFLICT — unknown scope is conservative |
+| anything                     | `null` (unknown)       | CONFLICT                                 |
+| `{ paths: [], ... }` (empty) | anything               | No conflict                              |
+| `packages/auth/`             | `packages/payments/`   | No conflict                              |
+| `packages/auth/`             | `packages/auth/`       | CONFLICT (exact match)                   |
+| `packages/`                  | `packages/auth/utils/` | CONFLICT (parent contains child)         |
+| `src/`                       | `src/config.ts`        | CONFLICT (file within directory scope)   |
+| `src/`                       | `src-old/config.ts`    | No conflict (different prefix)           |
+
+---
+
+## Scope Expansion During a Run
+
+Once planning completes and the actual affected files are known, the orchestrator automatically expands the run's declared scope by reading the plan's `affected_files` list. This narrows the window of conservative blocking.
+
+**Programmatic (supervisor internal)**:
+
+```typescript
+await kernel.expandRunLeaseScope({
+  run_id: state.runId,
+  owner_instance_id: state.ownerInstanceId,
+  additional_paths: planAffectedFiles,
+  additional_plan_section_ids: planSectionIds,
+});
+```
+
+Expansion is **additive** — existing declared paths are never removed.
+
+---
+
+## Observability
+
+### Log output
+
+When a lease is deferred due to scope conflict, a warning is emitted:
+
+```
+[run-lease] scope conflict: run run_123 blocked by run_456 on packages/auth/
+```
+
+### Error response
+
+The structured error response (visible in `aop run` JSON output) includes:
+
+```json
+{
+  "ok": false,
+  "error": {
+    "code": "lease_scope_conflict",
+    "message": "Lease acquisition deferred: scope overlaps with active run run_456",
+    "details": {
+      "incoming_scope": { "paths": ["packages/auth/"], "plan_section_ids": [] },
+      "blocking_run_id": "run_456",
+      "blocking_owner_instance_id": "supervisor:12345:abc",
+      "conflicting_scope_entry": "packages/auth/",
+      "retryable": true,
+      "requires_human": false
+    }
+  }
+}
+```
+
+### Lease Registry inspection
+
+Inspect all active leases:
+
+```bash
+cat .aop/runtime/lease-registry.json | jq '.leases[] | {run_id, scope, lease_expires_at}'
+```
+
+---
+
+## Migration Guide
+
+### Existing orchestrator runs (no scope declared)
+
+All existing callers of `acquireRunLease` that do not pass a `scope` field will receive `null` (unknown scope) by default. This means **the behavior is identical to today** — a run without a declared scope blocks against all other active runs.
+
+To opt in to parallel execution, callers must explicitly declare their scope.
+
+### Existing persisted lease files
+
+Lease files created before this feature (`run-lease.json` without a `scope` field) will deserialize with `scope: undefined`, which is normalized to `null` (conservative). No migration is needed.
+
+---
+
+## Known Limitations (in-scope for this feature)
+
+- Symbolic links and aliased paths are **not** resolved — two paths that refer to the same directory via different names are not treated as overlapping unless they share a literal prefix.
+- Scope expansion is **not retroactive** — a run that started with `null` scope cannot become non-conflicting by expanding later; the initial acquisition is the binding decision.
+- The CLI `--scope` flag is a **declaration**, not a contract enforcement — the system trusts that runs operate within their declared scope (plan-gated patches and worktree isolation provide the actual enforcement).

package/specs/002-scope-aware-lease/research.md

@@ -0,0 +1,135 @@
+# Research: Scope-Aware Run Lease
+
+**Feature Branch**: `002-scope-aware-lease`  
+**Research Phase**: Phase 0
+
+---
+
+## R-001 — Path-Prefix Overlap Semantics
+
+**Decision**: Use trailing-slash normalization before prefix comparison. A path `a` overlaps `b` if `normalize(a)` is a prefix of `normalize(b)` or vice versa.
+
+**Normalization rule**: Strip any trailing `/`, then append exactly one `/`. Empty string after strip becomes `/` (root — overlaps everything).
+
+**Examples**:
+
+- `"packages/auth"` → `"packages/auth/"`
+- `"packages/auth/"` → `"packages/auth/"` (idempotent)
+- `""` → `"/"`
+
+**Overlap examples**:
+
+- `"packages/auth/"` vs `"packages/authentication/"` → `false` (no false positive)
+- `"packages/auth/"` vs `"packages/auth/utils/"` → `true` (child is within parent)
+- `"src/"` vs `"src/config.ts/"` → `true` (file within directory scope)
+- `"src/"` vs `"src-old/config.ts/"` → `false` (similar but not prefix)
+
+**Rationale**: Spec FR-003 mandates "path-prefix containment." The trailing-slash normalization is the canonical technique to avoid false positives on directories that share a common name prefix but are distinct (e.g., `packages/auth` vs `packages/authorization`).
+
+**Alternatives considered**:
+
+- Glob pattern matching — adds significant complexity; spec only requires prefix containment.
+- `path.resolve()` + `path.relative()` — cross-platform edge cases, unnecessary for a Linux-first CLI.
+
+---
+
+## R-002 — Plan Section ID Overlap Semantics
+
+**Decision**: Two plan section IDs overlap iff they are **equal** (case-sensitive string equality, no hierarchical relationship).
+
+**Rationale**: The spec explicitly states: "Plan section identifiers used as scope entries are treated as opaque strings; two identifiers overlap only if they are equal (no hierarchical relationship between plan sections)." This keeps the implementation simple and consistent with spec intent.
+
+---
+
+## R-003 — Empty Scope vs. Unknown (null) Scope
+
+**Decision**:
+| Value | Meaning | Conflict behaviour |
+|---|---|---|
+| `null` | Unknown / undeclared | Conflicts with **all** active leases |
+| `undefined` (absent) | Not provided | Normalized to `null` at service boundary |
+| `{ paths: [], plan_section_ids: [] }` | Explicitly empty — touches nothing | Conflicts with **no** active leases |
+| `{ paths: [...], plan_section_ids: [...] }` | Declared scope | Checked via prefix/equality overlap |
+
+**Rationale**: The spec edge case "Empty scope list vs. no scope declared" explicitly calls for this distinction. Absent `scope` field is treated as `null` by normalization in `RunLeaseService` before any comparison — this matches how existing fields like `provider_config_ref_hash` are normalized.
+
+---
+
+## R-004 — Lease Registry Architecture
+
+**Decision**: A single shared file `.aop/runtime/lease-registry.json` (shape `{ leases: ActiveLeaseRecord[], updated_at: string }`) protected by the **existing index lock** (`.aop/features/.index.lock` via `withIndexLock`). Per-instance lease files at `.aop/runtime/<instanceId>/run-lease.json` are retained for per-instance session metadata.
+
+**Two-file design rationale**:
+
+| File                                       | Purpose                                                                                                        |
+| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------- |
+| `.aop/runtime/<instanceId>/run-lease.json` | Per-instance session metadata: run_id, owner, heartbeat timestamps, session IDs, feature sessions, scope       |
+| `.aop/runtime/lease-registry.json`         | Cross-instance scope conflict detection: all active leases with scopes, for pairwise comparison on acquisition |
+
+**Why the index lock**:
+
+- Already the single serialization point for all lease operations
+- Avoids introducing a second lock primitive that could cause priority inversion
+- Hold time increase is negligible (read + write a tiny JSON file)
+
+**Pruning strategy**: Expired entries (where `lease_expires_at < now`) are pruned on every registry read inside `withIndexLock`. No background GC process needed. This handles the "stale lease expiry" case (FR-008) automatically.
+
+**Alternatives considered**:
+
+- Dedicated lock `.aop/runtime/.lease-registry.lock` — same critical section; second lock adds complexity with no benefit.
+- `active_leases[]` in `index.json` — mixes feature index (SRP violation) and makes the index file larger on every lease change.
+- Per-instance files only — cannot enumerate all active leases for cross-instance conflict detection.
+
+---
+
+## R-005 — Scope Expansion Mid-Run (FR-007)
+
+**Decision**: New `expandRunLeaseScope(runId, ownerInstanceId, { additional_paths, additional_plan_section_ids })` method on `RunLeaseService` (and kernel). Expansion is **additive** — existing paths and section IDs are unioned with the new additions; no paths are removed. Re-evaluation of blocked pending runs is **pull-based** (retry on next acquisition attempt) rather than push-based.
+
+**Why pull-based re-evaluation**: Spec says scope expansion is "an infrequent operation." A push-based notification system (observer pattern, event queue) would add significant complexity and indirection for a rare operation. Blocked runs already retry via `LeaseHeartbeatService`; they naturally re-check on each retry.
+
+**When to call `expandRunLeaseScope`**: After the planning wave populates `plan.json` with an `affected_files` list. The `RunCoordinator` calls `expandScopesFromPlans(activeFeatureIds)` after `planningWaveExecutor.run()`. This narrows the window of conservative blocking from "entire run duration" to "pre-planning only."
+
+---
+
+## R-006 — Stale Lease Takeover (FR-008)
+
+**Decision**: Stale lease takeover is handled by the existing `takeover_stale_run: true` path. Expired registry entries are pruned on every read (R-004). When a stale lease is taken over:
+
+1. The per-instance file check detects staleness (existing behavior).
+2. On registry read inside `withIndexLock`, the expired entry is already pruned.
+3. The new run's entry is added with the new scope.
+4. Net result: old scope replaced by new scope, no `LEASE_SCOPE_CONFLICT` from expired leases.
+
+**No additional code needed** beyond the pruning logic in registry reads.
+
+---
+
+## R-007 — Observable Deferral Record (FR-010)
+
+**Decision**: Two observability mechanisms:
+
+1. **Structured error response** (`LEASE_SCOPE_CONFLICT`): The thrown error's `details` includes `blocking_run_id`, `blocking_owner_instance_id`, `conflicting_scope_entry` (first conflicting path/ID), `incoming_scope`.
+
+2. **`console.warn` log line**: Emitted from `RunLeaseService.acquireRunLease` before throwing. Uses `console.warn` (permitted in `apps/control-plane/src/` per CLAUDE.md). Format: `[run-lease] scope conflict: run <incoming_run_id> blocked by <blocking_run_id> on <conflicting_scope_entry>`.
+
+**Why `console.warn`**: `console.log` is forbidden in source. `console.error` is reserved for actual errors. A scope conflict is a normal, expected condition (not a bug), so `console.warn` is the appropriate level.
+
+---
+
+## R-008 — Simultaneous Acquisition Race (Edge Case)
+
+**Decision**: The `withIndexLock` serializes all acquisition attempts through a single file lock (`.aop/features/.index.lock`). Two runs attempting to acquire simultaneously will be serialized — one will enter the lock first, check the registry, find no conflicts (assuming non-overlapping scopes), write its entry, then release the lock. The second run will then enter, read the updated registry (now including the first run's entry), check for conflicts (none, since scopes don't overlap), and also succeed. Both runs correctly acquire their leases.
+
+**This is correct by construction** — no additional race-condition handling needed.
+
+---
+
+## R-009 — Backward Compatibility
+
+**Decision**: All changes are backward-compatible:
+
+- `scope` field on `AcquireRunLeaseInput` is optional — existing callers (e.g., `RunCoordinator` before updating) get `undefined` → normalized to `null` → conservative fallback → same behavior as today.
+- `scope` on `RuntimeSessionsSnapshot` is `scope?: LeaseScopeDeclaration` (optional) — existing persisted lease files (no `scope` field) deserialize without error; normalized to `null` on read.
+- `LeaseRegistry` file is created on first write; reads return empty registry if file is absent.
+- `LEASE_SCOPE_CONFLICT` is a new error code — existing callers that catch `RUN_ALREADY_ACTIVE` are unaffected; they may need updating to also handle `LEASE_SCOPE_CONFLICT` if they distinguish error codes.

package/specs/002-scope-aware-lease/spec.md

@@ -0,0 +1,128 @@
+# Feature Specification: Scope-Aware Run Lease
+
+**Feature Branch**: `002-scope-aware-lease`  
+**Created**: 2025-07-23  
+**Status**: Draft  
+**Input**: User description: "Update the runLease logic to not globally block executions based on the lease, and only block the executions if the active leases are expected to operate in the same sections of the codebase."
+
+## Overview
+
+Currently, when an orchestrator instance acquires a run lease it creates a global exclusive lock that prevents any other orchestrator from starting — regardless of whether the two would actually touch the same files or plan sections. This serializes all orchestration work even when runs target entirely different parts of the codebase, leaving substantial parallelism on the table.
+
+This feature replaces that global execution barrier with a scope-aware one. When a new execution attempts to start, the system compares its declared codebase scope against the scopes of all currently active leases. Execution is only blocked when an overlap exists. Runs that target different sections of the codebase proceed concurrently.
+
+## User Scenarios & Testing _(mandatory)_
+
+### User Story 1 - Parallel Execution of Non-Overlapping Runs (Priority: P1)
+
+An operator or automation triggers two orchestrator runs simultaneously. One run targets the authentication subsystem; the other targets the payment processing subsystem. Because these runs touch disjoint parts of the codebase, both should be allowed to proceed in parallel rather than one waiting for the other to finish.
+
+**Why this priority**: This is the primary value of the feature. Serializing unrelated runs is the exact inefficiency that motivated this change. Delivering this story alone demonstrates the core capability and provides measurable throughput improvement.
+
+**Independent Test**: Can be fully tested by launching two runs with non-overlapping scope declarations and verifying both reach an active state simultaneously without either entering a waiting state.
+
+**Acceptance Scenarios**:
+
+1. **Given** no active leases exist, **When** Run A declares scope `[packages/auth/]` and acquires a lease, **Then** Run A is granted the lease immediately.
+2. **Given** Run A holds a lease for scope `[packages/auth/]`, **When** Run B attempts to acquire a lease for scope `[packages/payments/]`, **Then** Run B is granted its lease without waiting for Run A to finish.
+3. **Given** Run A and Run B are both active with non-overlapping scopes, **When** both complete their work independently, **Then** each releases its own lease without affecting the other.
+
+---
+
+### User Story 2 - Blocking of Overlapping Runs (Priority: P1)
+
+An operator triggers two orchestrator runs that both target the same directory or plan section. The second run must not be allowed to proceed until the first completes, preserving the existing safety guarantee against concurrent mutation of the same codebase region.
+
+**Why this priority**: Equal in importance to Story 1 — the parallelism gain is only acceptable if the safety invariant is maintained. Without this story the system is correct but unsafe.
+
+**Independent Test**: Can be fully tested by launching two runs with identical or overlapping scope declarations and verifying that the second run is held in a waiting state until the first releases its lease.
+
+**Acceptance Scenarios**:
+
+1. **Given** Run A holds a lease for scope `[packages/auth/]`, **When** Run B attempts to acquire a lease for scope `[packages/auth/]`, **Then** Run B is blocked (lease denied or queued) until Run A releases its lease.
+2. **Given** Run A holds a lease for scope `[packages/]` (a parent directory), **When** Run B attempts to acquire a lease for scope `[packages/auth/]` (a child path), **Then** Run B is blocked because the scopes overlap via path prefix containment.
+3. **Given** Run A holds a lease for scope `[packages/auth/, packages/shared/]`, **When** Run B declares scope `[packages/shared/, packages/payments/]`, **Then** Run B is blocked because `packages/shared/` appears in both scopes.
+
+---
+
+### User Story 3 - Conservative Fallback for Unknown Scope (Priority: P2)
+
+A run is started before a plan has been established — for example, during an initial planning phase where the codebase scope is not yet known. The system must not allow an unknown-scope run to proceed concurrently with any other active run, since its scope might overlap anything.
+
+**Why this priority**: Ensures the system degrades safely when scope information is missing. Without this, runs without scope declarations could corrupt work in progress.
+
+**Independent Test**: Can be fully tested by launching a run without a scope declaration while another run is active and verifying that the scope-less run is blocked.
+
+**Acceptance Scenarios**:
+
+1. **Given** Run A holds a lease for any declared scope, **When** Run B attempts to acquire a lease with no declared scope, **Then** Run B is blocked until Run A's lease is released.
+2. **Given** Run A holds a lease with no declared scope, **When** Run B attempts to acquire a lease for any scope, **Then** Run B is blocked until Run A's lease is released.
+3. **Given** no active leases exist and Run A has no declared scope, **When** Run A acquires the lease, **Then** Run A is granted the lease immediately (absence of conflicts allows acquisition).
+
+---
+
+### User Story 4 - Scope Declaration at Lease Acquisition (Priority: P2)
+
+Orchestrators can supply a set of codebase paths or plan section identifiers when requesting a lease. This declared scope is persisted in the lease record and used for conflict detection against all other active leases.
+
+**Why this priority**: This is the prerequisite data-capture step that enables Stories 1–3. Without persisting scope at acquisition time the comparison cannot happen.
+
+**Independent Test**: Can be fully tested by acquiring a lease with a specific scope, then reading back the lease record and verifying the scope is preserved accurately.
+
+**Acceptance Scenarios**:
+
+1. **Given** an orchestrator supplies a list of directory paths when requesting a lease, **When** the lease is granted, **Then** the lease record contains exactly those paths.
+2. **Given** an orchestrator supplies feature/plan section identifiers when requesting a lease, **When** the lease is granted, **Then** the lease record contains those identifiers alongside any path-based scope.
+3. **Given** a stale lease is taken over by a new orchestrator, **When** the takeover completes, **Then** the new orchestrator's scope replaces the previous lease's scope in the record.
+
+---
+
+### Edge Cases
+
+- **Partially overlapping nested paths**: A scope of `packages/` and a scope of `packages/auth/utils/` — the system must treat prefix-containment as overlap (child path is within parent scope).
+- **Scope update during an active run**: If an execution discovers additional files during its run, it must be able to expand its declared scope; other pending leases must be re-evaluated.
+- **Simultaneous acquisition race**: Two runs with non-overlapping scopes attempt to acquire leases at the exact same moment — both must succeed without one incorrectly blocking the other.
+- **Lease expiry with active scope**: When a stale lease expires and a new run takes it over, the scope conflict table must reflect the new owner's scope, not the expired one's.
+- **Empty scope list vs. no scope declared**: An explicitly empty scope list (meaning "touches nothing") should be treated as non-conflicting with any other scope, distinct from an absent/unknown scope.
+- **Single file overlap within a large scope**: Run A declares `[src/]` and Run B declares `[src/config.ts]` — because `src/config.ts` is within `src/`, this must be treated as a conflict.
+
+## Requirements _(mandatory)_
+
+### Functional Requirements
+
+- **FR-001**: The system MUST allow two orchestrator runs to hold active leases simultaneously when their declared codebase scopes do not overlap.
+- **FR-002**: The system MUST deny or defer lease acquisition when the requesting run's scope overlaps with any currently active lease's scope.
+- **FR-003**: Scope overlap MUST be evaluated using path-prefix containment — a scope entry is considered to overlap another if one is a prefix of (or equal to) the other.
+- **FR-004**: The system MUST treat a run with an undeclared or unknown scope as conflicting with all other active leases, preserving the conservative blocking behavior.
+- **FR-005**: The system MUST persist the declared scope as part of the lease record so it survives across heartbeat renewals and instance restarts.
+- **FR-006**: The system MUST support scope declarations consisting of file-system paths, feature/plan section identifiers, or a combination of both.
+- **FR-007**: An active orchestrator MUST be able to expand its declared scope during a run (e.g., after discovering additional affected files), and the updated scope MUST be used for any subsequent conflict evaluations against pending lease requests.
+- **FR-008**: When a stale lease is taken over, the system MUST replace the expired scope with the new orchestrator's declared scope before evaluating conflicts.
+- **FR-009**: The system MUST continue to use short-duration atomic locks for mutating the lease registry itself, ensuring that scope comparison and lease grant/deny decisions are free from race conditions.
+- **FR-010**: The system MUST produce an observable record (log entry or status field) when a lease acquisition is deferred due to scope conflict, identifying which active lease caused the block.
+
+### Key Entities
+
+- **Run Lease**: Represents an orchestrator's exclusive claim over a set of codebase sections. Key attributes: run identifier, owning instance identifier, declared scope (set of paths and/or plan section IDs), lease validity window (start, last heartbeat, expiry), and current status (active, stale, released).
+- **Codebase Scope**: The declared set of file-system paths and/or plan section identifiers that a run expects to read or modify. Can be empty (no conflicts), absent/unknown (conflicts with everything), or a non-empty list (checked via prefix overlap).
+- **Scope Conflict**: A relationship between two leases where at least one path or section identifier in one lease's scope is a prefix of, or equal to, at least one entry in the other lease's scope. Conflicts are symmetric.
+- **Lease Registry**: The shared, atomically-updated store of all active run leases including their scopes. The registry is the authoritative source for conflict detection and is read in full during each lease acquisition attempt.
+
+## Success Criteria _(mandatory)_
+
+### Measurable Outcomes
+
+- **SC-001**: Two orchestrator runs with non-overlapping scopes both reach an active state within the same elapsed time window as a single run would — demonstrating true parallelism with no artificial serialization delay.
+- **SC-002**: Zero instances of concurrent lease holders with overlapping scopes across any test scenario, confirming the safety invariant is fully preserved.
+- **SC-003**: A run with an unknown or undeclared scope is never permitted to run concurrently with any other active run, maintaining 100% conservative blocking for ambiguous cases.
+- **SC-004**: The declared scope of an active lease is accurately reflected in the lease record immediately after acquisition and after any scope expansion, with no lag or inconsistency observable by other requesting runs.
+- **SC-005**: Scope conflict detection adds no more than a negligible overhead to lease acquisition time compared to the previous global-lock approach when there are zero active competing leases.
+- **SC-006**: The system correctly identifies and blocks a lease request in 100% of cases where path-prefix overlap exists between the incoming scope and any active lease's scope, including nested path relationships.
+
+## Assumptions
+
+- The codebase sections a run will touch are knowable (at least partially) before execution begins — typically derivable from the feature plan or spec file paths. Runs that genuinely cannot declare any scope at start will use the conservative unknown-scope path.
+- Path-prefix matching is sufficient for scope overlap detection; symbolic links and aliased paths are out of scope for this feature.
+- Scope expansion (FR-007) is an infrequent operation. Frequent expansion mid-run is not an expected usage pattern and does not need to be optimized for high throughput.
+- The lease registry remains small in practice (single-digit concurrent runs); exhaustive pairwise comparison is acceptable without an indexing strategy.
+- Plan section identifiers used as scope entries are treated as opaque strings; two identifiers overlap only if they are equal (no hierarchical relationship between plan sections).