@nomos-arc/arc 0.1.0

package/docs/phase7/TEAM_RED.md

@@ -0,0 +1,249 @@
+# Adversarial Architect — Zero-Trust Audit Report
+
+**Subject:** Phase 7: Global Semantic Search — Atomic Implementation Blueprint  
+**Auditor Role:** Chief Adversarial Architect & Lead SRE  
+**Date:** 2026-04-06  
+**Classification:** Production Readiness Assessment  
+
+---
+
+## 1. VETO STATUS: CONDITIONAL
+
+The plan is **not safe for production execution** in its current form. While architecturally competent at surface level, it contains multiple race conditions, unguarded failure modes, and silent data corruption vectors that would cause outages in a live system under non-trivial load. Execution is conditional on resolution of all Critical Blockers below.
+
+---
+
+## 2. CRITICAL BLOCKERS (System Killers)
+
+### BLOCKER-1: Full Index `reset()` + `upsert()` Is Not Atomic — Zero-Availability Window
+
+**Location:** Step 7.4.1, Full Index Flow, steps 5–6.
+
+The plan calls `VectorStore.reset()` (drop table) followed by `VectorStore.upsert(records)`. Between these two operations, the vector index is **empty**. Any concurrent `arc search` query hitting the store during this window will either:
+- Return zero results (silent failure — user believes nothing matches).
+- Throw an error because the table does not exist (crash).
+
+The plan states "LanceDB uses Lance format with transactional writes — no partial corruption on crash." This is true for individual writes, but **the plan does not wrap reset+upsert in a single transaction**. LanceDB's transaction guarantee applies per-operation, not across two sequential operations. The plan conflates operation-level atomicity with pipeline-level atomicity.
+
+**Severity:** Data loss / availability gap during every full re-index.
+
+**Mitigation:** Write to a new table (`nomos_vectors_tmp`), then atomically swap by dropping old and renaming new. Or use LanceDB's versioning/overwrite mode if available.
+
+---
+
+### BLOCKER-2: `index-meta.json` Write Is Inconsistent With Vector Store State
+
+**Location:** Step 7.4.1, Full Index Flow, steps 6–7.
+
+The upsert to LanceDB (step 6) and the metadata write to `index-meta.json` (step 7) are two separate, non-transactional operations. If the process crashes after step 6 but before step 7:
+- The vector store contains the new data.
+- The metadata file references the **old** state (or doesn't exist).
+- Incremental index reads stale metadata → re-indexes everything unnecessarily, or worse, computes wrong diffs.
+
+If the process crashes after step 7's `.tmp` write but before the rename:
+- Metadata is lost. The `.tmp` file is orphaned.
+- Next incremental index falls back to full index (stated behavior), but the full index calls `reset()` — destroying a perfectly valid vector store.
+
+**Severity:** State corruption on crash during indexing.
+
+**Mitigation:** Write metadata **before** the upsert with a `status: "in_progress"` field. Mark `status: "complete"` after upsert succeeds. On startup, if status is `in_progress`, force a full re-index.
+
+---
+
+### BLOCKER-3: No Vector Dimension Validation on Incremental Index
+
+**Location:** Step 7.4.1, Incremental Index Flow. Risk Register mentions this but provides no implementation.
+
+The Risk Register identifies "Vector dimensions mismatch" and states "Validate on incremental index — force full re-index if dimensions changed." However, the incremental index flow (steps 1–11) contains **zero** dimension validation logic. If the `embedding_model` config changes between a full index and an incremental index:
+- New 768-dim vectors get upserted alongside old vectors of a different dimension.
+- Cosine similarity calculations between mismatched dimensions produce garbage results.
+- LanceDB may or may not reject this at the Arrow schema level — the plan does not verify.
+
+**Severity:** Silent data corruption. Search returns nonsense results with no error signal.
+
+**Mitigation:** Step 2 of incremental flow must compare `IndexMetadata.embedding_model` and `IndexMetadata.vector_dimensions` against current config. Mismatch → force full re-index with explicit log message.
+
+---
+
+### BLOCKER-4: Unbounded Memory During Embedding Composition
+
+**Location:** Step 7.4.1, Full Index Flow, step 4.
+
+The plan states: "Never hold all vectors in memory simultaneously." However, step 4 composes `VectorRecord[]` from **all** `TextChunks + vectors`. This means:
+- All `Float32Array` vectors from `embedBatch()` are held in memory.
+- All `VectorRecord` objects (which include the vectors) are constructed in a single array.
+- Then passed to `VectorStore.upsert(records)` as a single batch.
+
+For a 500-file project with ~1500 chunks, each with a 768-dim Float32Array (3KB per vector): `1500 * 3KB = 4.5MB` in vectors alone, plus the record metadata. For a 5000-file project: ~45MB. This is manageable, but the plan claims streaming batch processing (Risk Register) while the implementation design does not deliver it.
+
+**Severity:** Memory pressure / OOM on large projects. The plan's own Risk Register contradicts its implementation.
+
+**Mitigation:** Upsert in batches matching the embedding batch size. After each embedding batch completes, compose records and upsert immediately, then release references.
+
+---
+
+## 3. AMBIGUITY TRAPS
+
+### TRAP-1: `mergeInsert` Fallback Strategy Is Undefined
+
+**Location:** Step 7.1.1, Internal Design point 4.
+
+> "If `mergeInsert` is unavailable in the installed version, fall back to delete-then-add."
+
+The plan fails to specify **how** to detect whether `mergeInsert` is available. Runtime feature detection? Try-catch? Version string comparison? An AI agent implementing this will likely:
+- Wrap in try-catch, masking real errors as "feature unavailable."
+- Implement the fallback path (delete-then-add) which is **not atomic** and creates a window where records are deleted but not yet re-added.
+
+This is an instruction to an implementer that will produce non-deterministic behavior.
+
+---
+
+### TRAP-2: "Process Chunks Sequentially" vs. `max_concurrent_requests` Config
+
+**Location:** Step 7.2.1, Internal Design point 4 vs. Config Step 7.0.4.
+
+The config defines `max_concurrent_requests: 5`, implying concurrent batch processing. The embedder design says "Process chunks sequentially to respect rate limits." These are contradictory. An implementer will either:
+- Ignore `max_concurrent_requests` entirely (dead config).
+- Implement concurrency, violating the sequential processing instruction and creating rate limit violations.
+
+The plan must pick one: sequential with rate-limit delays, or concurrent with a semaphore. Not both.
+
+---
+
+### TRAP-3: De-duplication Logic Lacks Determinism
+
+**Location:** Step 7.5.2, Internal Design step 7.
+
+> "If same file_path appears as both 'file' and 'symbol' type, the symbol entry takes priority in ranking when scores are within 0.05."
+
+What does "takes priority" mean? Swap their positions? Remove the file-level result? Boost the symbol score? And "within 0.05" — is that absolute difference or relative? If two symbol chunks from the same file both appear alongside the file chunk, which symbol "takes priority"?
+
+This is insufficiently specified for deterministic implementation.
+
+---
+
+### TRAP-4: `graph_depth: -1` Sentinel Value
+
+**Location:** Step 7.5.1, Internal Design point 3.
+
+Using `-1` as a sentinel for "file was deleted after indexing" is a type-unsafe convention that will leak into downstream formatting. The CLI output (Step 7.6.2) shows `depth {N}` — will it print `depth -1`? The plan does not address how the CLI formats stale results. An implementer may print `depth -1` to the user, or crash on negative depth, or silently omit the result.
+
+---
+
+### TRAP-5: `.semantic.md` Path Derivation Is Fragile
+
+**Location:** Step 7.3.1, Internal Design point 3.
+
+> `fs.readFile(path.join(projectRoot, file_path.replace(/\.[^.]+$/, '.semantic.md')))`
+
+This regex replaces the **last** extension. For files like `config.test.ts`, this produces `config.test.semantic.md`, not `config.semantic.md`. For extensionless files (e.g., `Makefile`), the regex matches nothing and appends `.semantic.md` nowhere — the path is unchanged. The plan does not specify the actual `.semantic.md` naming convention from prior phases, leaving the implementer to guess.
+
+---
+
+## 4. RESILIENCE GAPS
+
+### GAP-1: Gemini API Outage During Indexing — Partial State, No Recovery
+
+**Location:** Step 7.4.1, Error handling.
+
+The plan states: "Embedding failure on a single batch: log error, continue with remaining batches. Track failed file paths."
+
+This means if batches 3–7 fail (e.g., Gemini 503 for 2 minutes), the resulting index is **partial**: it contains vectors for batches 1–2 but not 3–7. The metadata is written with `total_chunks` reflecting only the successfully embedded count. On next incremental index, the content hashes for batches 3–7 files are unchanged — so they are **not re-indexed**. The partial index becomes permanent.
+
+The plan has no mechanism to detect or repair a partial index. There is no `failed_files` field in `IndexMetadata`. The incremental diff only checks `content_hash`, not "was this file actually embedded."
+
+**Impact:** Permanent search blind spots after any transient API failure.
+
+**Mitigation:** Record failed file paths in `IndexMetadata.failed_files[]`. On next incremental index, treat failed files as "changed" regardless of hash. Or: fail the entire index operation on any batch failure (safer, simpler).
+
+---
+
+### GAP-2: LanceDB Corruption / Version Mismatch Has No Detection
+
+The plan pins no specific LanceDB version and acknowledges the package is "pre-1.0" (Risk Register). If `npm install` pulls a new minor version that changes the on-disk format:
+- Existing vector index may fail to open.
+- No error handling is specified for `lancedb.connect()` failure.
+- No schema migration path exists.
+
+The plan's `VectorStore.init()` has no try-catch, no version validation, no schema compatibility check.
+
+---
+
+### GAP-3: SIGINT During `reset()` + `upsert()` Sequence
+
+**Location:** Step 7.6.1, SIGINT handling.
+
+The SIGINT handler sets `cancellationFlag.cancelled = true`, which is checked "between embedding batches." But the critical danger zone is between `reset()` and `upsert()` (BLOCKER-1). If SIGINT fires after `reset()` but before `upsert()` completes:
+- The vector store is empty.
+- The "partial metadata" write records zero chunks.
+- The system is in an unrecoverable state without manual intervention (`arc index --force`).
+
+The cancellation check granularity is at the **embedding** level, not the **store write** level. The most dangerous operation (reset) is not cancellation-aware.
+
+---
+
+### GAP-4: No Timeout on Gemini API Calls
+
+**Location:** Step 7.2.1.
+
+The embedder implements retry with backoff on 429/5xx, but specifies **no request timeout**. If the Gemini API hangs (TCP connection established, no response), the embedder will block indefinitely. The CLI becomes unresponsive. There is no `AbortController`, no socket timeout, no `Promise.race` with a deadline.
+
+In a "3 AM" scenario: the on-call runs `arc index`, it hangs forever. No log output. No error. No timeout. They kill -9 it. See GAP-3.
+
+---
+
+### GAP-5: `arc search` Loads Full `project_map.json` On Every Query
+
+**Location:** Step 7.5.2, Internal Design step 5.
+
+Every `arc search` call loads and parses the full `project_map.json` for graph enrichment. For a large project, this file could be 5–50MB. The plan specifies AC-15 as "Search < 2s for 500 indexed files" — but if `project_map.json` is 20MB, just parsing it takes measurable time.
+
+No caching, no lazy loading, no pre-computed graph metadata stored alongside the vector index.
+
+---
+
+## 5. MANDATORY MITIGATIONS
+
+The following are **non-negotiable** requirements before this plan is approved for execution:
+
+| # | Mitigation | Addresses |
+|---|-----------|-----------|
+| M-1 | Implement table-swap strategy for full re-index (write to temp table, atomic rename). Do not use `reset()` + `upsert()` as separate operations. | BLOCKER-1 |
+| M-2 | Add `status` field to `IndexMetadata` (`in_progress` / `complete`). Write metadata at start and end of indexing. On startup with `in_progress`, force full re-index. | BLOCKER-2 |
+| M-3 | Validate `embedding_model` and `vector_dimensions` in incremental index flow. Mismatch → automatic full re-index with warning log. | BLOCKER-3 |
+| M-4 | Upsert records in batches immediately after embedding each batch. Do not accumulate all records in memory. | BLOCKER-4 |
+| M-5 | Remove `max_concurrent_requests` config or implement it. Dead config is worse than no config — it implies a capability that doesn't exist. | TRAP-2 |
+| M-6 | Add `failed_files: string[]` to `IndexMetadata`. On incremental index, re-index any file in `failed_files` regardless of content hash. | GAP-1 |
+| M-7 | Add request-level timeout (30s default) to all Gemini API calls using `AbortController` or equivalent. | GAP-4 |
+| M-8 | Wrap `lancedb.connect()` and table operations in try-catch with specific error messages. A corrupted index should produce a clear "run `arc index --force`" message, not an unhandled exception. | GAP-2 |
+| M-9 | Pin `@lancedb/lancedb` to an exact version (`"x.y.z"`, not `"^x.y.z"`). Pre-1.0 packages have no semver stability guarantees. | GAP-2 |
+| M-10 | Define deterministic de-duplication: if a symbol result and its parent file result are both present and within 0.05 similarity, **remove the file-level result** (keep only the more specific symbol result). Document this as a hard rule, not a vague "priority." | TRAP-3 |
+
+---
+
+## 6. SECONDARY CONCERNS (Non-Blocking)
+
+| # | Concern | Note |
+|---|---------|------|
+| S-1 | Pre-flight check PF-5 (`src/search/` does not exist) prevents re-execution. If Phase 7 partially fails and is retried, this check blocks it. Idempotency violation at the plan level. | Change to: "If `src/search/` exists, verify it was created by Phase 7 (check for known files)." |
+| S-2 | No `--dry-run` flag on `arc index`. For a command that makes external API calls (costs money) and writes to disk, dry-run should be mandatory for production systems. | Add `--dry-run` that extracts and counts chunks without embedding. |
+| S-3 | Cosine distance range stated as `[0, 2]` (Step 7.1.1, point 5). This is correct for cosine distance but the plan conflates distance and similarity in multiple places. The threshold config is in similarity space (0.0–1.0), but LanceDB returns distance. Multiple conversion points = multiple bug opportunities. | Centralize the `1 - distance = similarity` conversion in exactly one place (`VectorStore.query()`). Assert in tests that returned similarity is in `[0, 1]`. |
+| S-4 | Integration test (Step 7.7.1) requires `GEMINI_API_KEY` — skipped in CI. This means the most critical test (full pipeline) never runs in automated pipelines. The plan has no CI-compatible integration test strategy. | Add a mock-embedder integration test that runs in CI with deterministic fake vectors. |
+| S-5 | `Float32Array` serialization to JSON (for `--json` output) will produce `{}` in most JS runtimes (`JSON.stringify(new Float32Array([1,2,3]))` → `{"0":1,"1":2,"2":3}`). The plan doesn't specify that vectors should be excluded from JSON search output. | Ensure `SearchResult` type does not include `vector` field. Verify JSON output does not leak raw vectors. |
+| S-6 | The `content_hash` is computed from the composed text string, not the source file. If the text composition logic changes (e.g., different field ordering), all hashes change, forcing a full re-index. This couples hashing to presentation logic. | Hash the raw inputs (file path + semantic data + symbol signatures) separately from the composed text. Or accept the coupling and document it. |
+| S-7 | Step 7.0.2 says "Append `--external:@lancedb/lancedb` to the build script." If the build script is already long and complex, appending may break argument ordering. The plan does not show the current build script. | Read the current build script first. Insert externals in the correct position relative to other esbuild flags. |
+| S-8 | No rate limit backpressure signal to the user. When rate-limited, the CLI silently sleeps. For a large index (1000+ chunks, 20+ batches), the user sees no output for long periods and may assume it's hung. | Log rate limit delays: `"[nomos:search:warn] Rate limited. Waiting {N}ms before next batch..."` |
+
+---
+
+## 7. AUDIT SUMMARY
+
+The plan is architecturally sound in its decomposition — the module boundaries, type definitions, and dependency graph are well-structured. The risk register shows awareness of real failure modes. The idempotency checks per-step are a notable positive.
+
+However, the plan fails to account for the **composition** of its own components under failure. Individual modules are designed with care; the pipeline that connects them is not. The gap between `reset()` and `upsert()`, the gap between upsert and metadata write, the gap between embedding failure and incremental recovery — these interstitial failures are where production systems die.
+
+The plan is one good afternoon of mitigations away from being production-ready. It is not there yet.
+
+---
+
+*End of Adversarial Audit. All findings are technical in nature and assume worst-case operational conditions.*