loki-mode 11.3.0 → 11.3.2

package/docs/bug-fixes/agent-20-architecture-review.md

@@ -1,331 +0,0 @@
-# Agent 20: Architecture Review Report
-
-**Date:** 2026-03-24
-**Scope:** Systemic architecture issues, component boundaries, security, complexity
-**Files Reviewed:** 40+ files across all major subsystems
-
----
-
-## Executive Summary
-
-Loki Mode's architecture follows a filesystem-as-IPC pattern where all components
-communicate through `.loki/` state files. While pragmatic for a CLI-first tool,
-this design has accumulated several systemic issues as the codebase scaled to
-40,000+ lines across Bash, Python, and TypeScript. The most critical findings
-are: (1) three inconsistent LOKI_DIR resolution mechanisms within the dashboard
-package, (2) a dual event system where events are written to two different
-locations with different formats, (3) non-atomic JSON writes in `set_phase()`
-that can corrupt state under concurrent access, and (4) extreme file sizes
-that hinder maintainability.
-
----
-
-## CRITICAL -- Bugs Found and Fixed
-
-### BUG ARCH-001: Non-atomic JSON write in `set_phase()` (SEVERITY: HIGH)
-
-**File:** `autonomy/run.sh`, line 3254-3261
-
-The `set_phase()` function updates `.loki/state/orchestrator.json` using a
-direct read-then-write pattern without atomicity:
-
-```python
-with open(sys.argv[1], 'r') as f:
-    data = json.load(f)
-data['currentPhase'] = sys.argv[2]
-with open(sys.argv[1], 'w') as f:       # <-- Non-atomic: truncates then writes
-    json.dump(data, f, indent=2)
-```
-
-If the process is killed (Ctrl+C, OOM) between truncation and write completion,
-`orchestrator.json` will be empty or contain partial JSON. The dashboard's
-`_safe_json_read()` handles corrupt reads gracefully, but the state is permanently
-lost until the next `write_dashboard_state()` call re-derives it.
-
-This is the same class of bug already fixed elsewhere (BUG-XC-004, BUG-ST-008)
-using temp-file-plus-rename. The fix pattern is already established in the codebase.
-
-**Impact:** State corruption on crash during phase transition. The dashboard
-and completion council may see stale or missing phase data.
-
-### BUG ARCH-002: Three inconsistent LOKI_DIR resolution mechanisms (SEVERITY: HIGH)
-
-Within the `dashboard/` package, three different files resolve the `.loki/`
-directory path using three different mechanisms:
-
-| File | Env Var | Default | Resolution |
-|------|---------|---------|------------|
-| `dashboard/control.py:30` | `LOKI_DIR` | `.loki` (CWD-relative) | Static at import time |
-| `dashboard/api_v2.py:68` | `LOKI_DATA_DIR` | `~/.loki` (user home) | Static at import time |
-| `dashboard/server.py:1869` | `LOKI_DIR` | `.loki` then `~/.loki` | Dynamic 4-step resolution per call |
-
-This means:
-- `control.py` reads/writes state from CWD's `.loki/`
-- `api_v2.py` reads policies from `~/.loki/`
-- `server.py` dynamically resolves based on env, API override, CWD, or home
-
-The `api_v2.py` one is particularly problematic because it uses `LOKI_DATA_DIR`
-(not `LOKI_DIR`) and resolves at module import time to `~/.loki`, which is the
-global directory, not the project directory. If a user has per-project `.loki/`
-directories (the normal case), the policy endpoints in api_v2 will look in the
-wrong location.
-
-**Impact:** Policy endpoints may read from wrong directory. State written by
-`control.py` may not be visible to `server.py` if CWD differs from project
-directory at dashboard startup time.
-
-### BUG ARCH-003: Dual event system with incompatible formats (SEVERITY: MEDIUM)
-
-The codebase has two parallel event systems:
-
-1. **JSONL append log** (`run.sh` lines 893-966): Events appended to
-   `.loki/events.jsonl` with format `{"timestamp":..., "type":..., "data":...}`
-   - Used by `emit_event()` and `emit_event_json()` -- 28 call sites in run.sh
-   - Consumed by: dashboard (reads JSONL), CLI (reads JSONL)
-
-2. **File-per-event directory** (`events/bus.py`, `events/emit.sh`):
-   Events written as individual JSON files to `.loki/events/pending/` with
-   format `{"id":..., "type":..., "source":..., "timestamp":..., "payload":...}`
-   - Used by `emit_event_pending()` -- separate call sites
-   - Consumed by: EventBus subscribers, MCP server, state manager
-
-These two systems have different schemas, different consumers, and no bridge
-between them. Events emitted via `emit_event()` are invisible to the EventBus
-and vice versa. The `emit_event_pending()` function in run.sh (line 971) writes
-to the pending directory but most call sites still use the JSONL functions.
-
-**Impact:** Components subscribing to the EventBus miss events emitted to JSONL.
-Dashboard state pushes are based on file polling rather than events, adding
-unnecessary latency.
-
----
-
-## Systemic Architecture Issues
-
-### ISSUE 1: Extreme file sizes impair maintainability (SEVERITY: HIGH)
-
-| File | Lines | Functions | Concern |
-|------|-------|-----------|---------|
-| `autonomy/loki` | 20,300 | 130+ `cmd_*` functions | Single-file CLI with 130 subcommands |
-| `autonomy/run.sh` | 10,869 | 150+ functions | Orchestration engine |
-| `dashboard/server.py` | 5,244 | 121+ routes | Monolithic API server |
-
-The `loki` CLI has nearly doubled from the 10,820 lines documented in CLAUDE.md
-to 20,300 lines. At this size, bash's lack of namespacing means every function
-and variable is global. A naming collision between `cmd_test()` helpers and
-`cmd_report()` helpers is a constant risk.
-
-**Recommendation:** Split `autonomy/loki` by command group:
-- `autonomy/commands/start.sh` -- start/stop/pause/resume
-- `autonomy/commands/dashboard.sh` -- dashboard/web commands
-- `autonomy/commands/github.sh` -- github/import/issue commands
-- `autonomy/commands/memory.sh` -- memory subcommands
-- `autonomy/commands/config.sh` -- config/setup commands
-- Each file sources into the main `loki` dispatcher
-
-### ISSUE 2: 79 inline `python3 -c` calls in run.sh (SEVERITY: MEDIUM)
-
-The orchestrator shell script (`run.sh`) contains 79 inline Python one-liners
-for JSON parsing, state manipulation, and data extraction. For example, reading
-a single field from `orchestrator.json` spawns a new Python process each time:
-
-```bash
-current_phase=$(python3 -c "import json; print(json.load(open('.loki/state/orchestrator.json')).get('currentPhase', 'BOOTSTRAP'))" 2>/dev/null || echo "BOOTSTRAP")
-```
-
-In `write_dashboard_state()` alone, there are 8+ separate `python3 -c` calls
-that each open, parse, and close the same JSON files. Each call has ~50ms
-startup overhead, making the dashboard state writer measurably slow.
-
-**Recommendation:** Create a single helper script
-`autonomy/json-helper.py` that accepts commands like
-`json-helper.py get orchestrator.json currentPhase BOOTSTRAP` and can batch
-multiple reads in one invocation. Or use `jq` consistently (already used in
-some places).
-
-### ISSUE 3: Dashboard has two separate FastAPI apps (SEVERITY: MEDIUM)
-
-The dashboard package contains two independent FastAPI applications:
-
-1. `dashboard/server.py` -- Main API with 121+ routes, SQLAlchemy DB, auth, WebSocket
-2. `dashboard/control.py` -- Separate FastAPI app with its own CORS, models, routes
-
-Both define their own `CORSMiddleware` configurations (with slightly different
-wildcard policies -- control.py uses `allow_methods=["*"]` and `allow_headers=["*"]`
-while server.py restricts both). Both define their own status/health models.
-The control.py `app` is never mounted into server.py's `app`; it appears to be
-a legacy standalone server that was partially superseded by server.py.
-
-The `atomic_write_json` function from `control.py` is imported by `server.py`
-(line 55: `from .control import atomic_write_json`), but the rest of control.py's
-routes are unreachable when the dashboard is run via server.py.
-
-**Recommendation:** Either mount control.py's router into server.py, or extract
-`atomic_write_json` into a standalone utility module and deprecate the separate
-app.
-
-### ISSUE 4: State manager (state/manager.py) is underutilized (SEVERITY: LOW)
-
-A proper `StateManager` class exists at `state/manager.py` (1,896 lines) with:
-- File-based caching with watchdog
-- Thread-safe operations with file locking
-- Event bus integration
-- Version vectors for conflict resolution
-- Subscription system
-
-However, only the MCP server uses it (`mcp/server.py:48`). The main orchestrator
-(`run.sh`) uses raw file I/O. The dashboard (`server.py`) uses its own
-`_safe_json_read()`. This means the architecture has a proper abstraction layer
-that is largely bypassed.
-
-**Recommendation:** Gradually migrate dashboard state reads to use StateManager.
-The shell-based orchestrator cannot use it directly (Python vs Bash boundary),
-but the Python components should converge on this single state access layer.
-
-### ISSUE 5: No file locking on orchestrator.json writes (SEVERITY: MEDIUM)
-
-While `save_state()` (line 7938) uses atomic temp-file+mv for
-`autonomy-state.json`, and `control.py` uses `fcntl.flock` for atomic writes,
-the `set_phase()` function and multiple inline Python snippets write to
-`.loki/state/orchestrator.json` without any locking or atomicity.
-
-The `write_dashboard_state()` function (line 3272) reads from `orchestrator.json`
-at the same time that `set_phase()` might be writing to it. Since these run in
-the same process (run.sh), there is no parallelism risk in the normal case.
-However, during parallel mode (worktrees), multiple run.sh instances could
-write to the same orchestrator.json if they share a `.loki/` directory.
-
-**Impact:** Potential data corruption in parallel mode.
-
----
-
-## Security Architecture Review
-
-### Strengths
-- **Path traversal protection** in MCP server (`validate_path()` with symlink chain checking)
-- **PRD path validation** in control.py (blocks `..`, verifies file exists within allowed dirs)
-- **Provider name validation** prevents shell injection in loader.sh
-- **CORS restricted to localhost** by default (both server.py and control.py)
-- **OIDC/SSO support** with proper JWT validation (PyJWT + cryptography)
-- **Token-based auth** with role/scope hierarchy
-- **Rate limiting** on control and read endpoints
-- **Atomic writes** in many critical paths (BUG-XC-004, BUG-ST-008)
-
-### Concerns
-
-1. **CORS wildcard inconsistency** (LOW): `control.py` uses `allow_methods=["*"]`
-   and `allow_headers=["*"]` while `server.py` restricts to specific methods/headers.
-   If control.py routes are ever exposed, the broader CORS policy applies.
-
-2. **subprocess.Popen in control.py** (LOW): The `start_session` endpoint
-   (line 410) passes `request.provider` into a command-line argument list.
-   This IS validated: `request.validate_provider()` is called at line 379
-   and raises ValueError (caught at line 381) before the Popen call. The
-   validation is correct. However, `validate_provider()` is a manual method
-   call rather than a Pydantic `@field_validator`, so it could be bypassed
-   if someone adds a new endpoint that creates a StartRequest without calling
-   validate. Converting to a Pydantic validator would make this defense
-   automatic.
-
-3. **No auth on event bus** (LOW): Any process that can write to
-   `.loki/events/pending/` can inject events. This is acceptable for
-   local single-user use but should be noted for multi-tenant deployments.
-
----
-
-## Component Boundary Analysis
-
-### CLI (loki) to Orchestrator (run.sh)
-
-**Boundary:** Clean. CLI `exec`s run.sh as a subprocess (via `cmd_start()`),
-passing args via command-line flags. State handoff is filesystem-based.
-
-**Issue:** The CLI re-implements some orchestrator functionality (e.g., memory
-loading at `loki:274`, status file reading) rather than delegating to run.sh.
-This creates subtle divergence risk.
-
-### Web Server (server.py) to CLI
-
-**Boundary:** Indirect via filesystem. The dashboard reads `.loki/` state files
-that the orchestrator writes. For control operations, `control.py` spawns
-run.sh via `subprocess.Popen`.
-
-**Issue:** The dashboard does NOT call the CLI (`loki` binary) -- it calls
-`run.sh` directly. This bypasses any CLI-level validation, setup, or event
-emission that `cmd_start()` performs.
-
-### Dashboard API to Orchestrator
-
-**Boundary:** File-based polling. The orchestrator writes `dashboard-state.json`
-every iteration. The dashboard reads it via `_push_loki_state_loop()` (every 2s
-when running, 30s when idle) and pushes to WebSocket clients.
-
-**Issue:** This is polling-based, not event-driven. The event bus exists but is
-not used for this communication path. Adding event bus integration would reduce
-latency from 2s to near-instant.
-
-### Memory System to Everything
-
-**Boundary:** Clean Python API via `memory/engine.py`. The shell orchestrator
-bridges to it via Python one-liners.
-
-**Issue:** The memory engine is initialized independently by each consumer
-(run.sh via inline Python, MCP server via its own import, dashboard indirectly
-via state files). There is no shared singleton across components, so memory
-operations from different components may see inconsistent state.
-
-### MCP Server to Dashboard + Memory
-
-**Boundary:** MCP server uses StateManager for state access and direct memory
-imports. It has no dependency on the dashboard.
-
-**Issue:** MCP tools overlap significantly with dashboard API endpoints (task
-management, memory retrieval, state queries). There is no deduplication or
-shared implementation between `/api/tasks` and the MCP `loki_queue_*` tools.
-
-### Event Bus to All Components
-
-**Boundary:** File-based pub/sub via `.loki/events/pending/`.
-
-**Issue:** As documented in BUG ARCH-003, the event bus is underutilized.
-The main orchestrator (run.sh) emits to JSONL, not the event bus. The
-dashboard does not consume events from either system for real-time updates;
-it polls `dashboard-state.json` instead.
-
----
-
-## Recommendations Summary (Prioritized)
-
-| Priority | Issue | Effort | Impact |
-|----------|-------|--------|--------|
-| P0 | Fix non-atomic write in `set_phase()` | 15 min | Prevents state corruption |
-| P0 | Unify LOKI_DIR resolution in dashboard package | 30 min | Prevents policy lookups from wrong directory |
-| P1 | Consolidate dual event systems | 2-4 hours | Consistent event propagation |
-| P3 | Convert `validate_provider()` to Pydantic field_validator | 10 min | Defense-in-depth validation |
-| P2 | Split `autonomy/loki` into command modules | 1-2 days | Maintainability |
-| P2 | Replace inline `python3 -c` with helper script | 4 hours | Performance improvement |
-| P2 | Merge control.py routes into server.py | 2 hours | Eliminate duplicate FastAPI app |
-| P3 | Adopt StateManager in dashboard | 1 day | Consistent state access |
-| P3 | Connect event bus to dashboard WebSocket push | 4 hours | Real-time updates |
-| P3 | Standardize CORS configuration | 30 min | Security consistency |
-
----
-
-## Feedback Loop Verification
-
-### Loop 1: Self-review
-- All findings reference specific file paths and line numbers
-- Severity ratings consider both likelihood and impact
-- Recommendations are actionable with effort estimates
-
-### Loop 2: Code verification
-- BUG ARCH-001: Confirmed by reading `run.sh:3254-3261` -- no temp file + mv pattern
-- BUG ARCH-002: Confirmed by reading `control.py:30`, `api_v2.py:68`, `server.py:1869`
-- BUG ARCH-003: Confirmed by comparing `emit_event()` at line 893 vs `emit_event_pending()` at line 971
-- File sizes confirmed via `wc -l` (loki=20,300, run.sh=10,869, server.py=5,244)
-- Python3 -c count confirmed via grep (79 occurrences)
-
-### Loop 3: Priority validation
-- P0 items are data integrity issues that can cause silent corruption
-- P1 items are defense-in-depth security and consistency
-- P2/P3 items are maintainability and performance improvements