@nomos-arc/arc 0.1.0

package/docs/phase7/task.md

@@ -0,0 +1,275 @@
+# Phase 7: Global Semantic Search (Vector-Based)
+
+## Overview
+
+Build a semantic search engine for the codebase that understands **meaning**, not just keywords. Instead of searching for the symbol `Payment`, a developer can ask: *"Where is the refund logic handled in this project?"* — and get accurate, context-aware results across the entire codebase.
+
+This phase transforms `project_map.json` and `.semantic.md` files into vector embeddings stored in a local database, enabling natural language queries over code architecture.
+
+---
+
+## Problem Statement
+
+Traditional code search (grep, ripgrep, IDE find) is limited to exact or fuzzy keyword matching. This fails when:
+
+- The developer doesn't know the exact symbol name or terminology used in the codebase.
+- The concept spans multiple files with no shared keyword (e.g., "error recovery flow" touches handlers, middleware, and retry logic).
+- Non-technical stakeholders (PMs, architects) need to locate functionality using domain language rather than code symbols.
+
+**Semantic search closes this gap** by matching on intent and meaning rather than lexical tokens.
+
+---
+
+## Core Concepts
+
+### Embeddings
+
+The process of converting text (e.g., contents of `.semantic.md` files) into numerical vectors in a high-dimensional space. Texts with similar meanings are mathematically "close" to each other, regardless of the specific words used.
+
+### Vector Database
+
+A local-first storage engine (e.g., LanceDB) that indexes these vectors for fast similarity search. This avoids reprocessing the entire project on every query and supports incremental updates.
+
+### Similarity Search
+
+When a user issues a query, it is converted to a vector using the same embedding model, then compared against stored vectors using cosine similarity (or similar distance metric) to find the closest matches.
+
+---
+
+## Execution Flow
+
+### Stage 1 — Indexing
+
+**Input:** `project_map.json` + `.semantic.md` files from the mapping phase.
+
+**Process:**
+1. Extract structured fields from each file entry: `purpose`, `key_logic`, `exports`, `dependencies`.
+2. Compose a search-optimized text chunk per file by concatenating these fields with clear delimiters.
+3. **Symbol-level chunking:** For each file, also extract individual classes and significant functions (using data from `project_map.json` exports/symbols). Each symbol gets its own embedding with a reference back to the parent file and line range. This enables search results that point to a specific function, not just a file.
+4. Send each chunk (file-level + symbol-level) to the embedding API (Gemini `text-embedding-004` or equivalent) to generate vectors.
+5. Store each vector alongside metadata (file path, module name, symbol name, line range, last modified timestamp).
+
+**Output:** A local vector index (`.nomos/vectors/`) containing file-level and symbol-level embeddings.
+
+### Stage 2 — Storage
+
+**Technology:** LanceDB (embedded, local-first, no server required).
+
+**Schema (file-level entry):**
+```json
+{
+  "id": "src/services/payment.ts",
+  "type": "file",
+  "vector": [0.012, -0.034, ...],
+  "file_path": "src/services/payment.ts",
+  "module": "payment-service",
+  "purpose": "Handles payment processing and refund logic",
+  "key_logic": "Stripe integration, idempotency keys, retry with backoff",
+  "graph_depth": 5,
+  "dependents_count": 10,
+  "last_indexed": "2026-04-06T12:00:00Z"
+}
+```
+
+**Schema (symbol-level entry):**
+```json
+{
+  "id": "src/services/payment.ts::processRefund",
+  "type": "symbol",
+  "vector": [0.008, -0.041, ...],
+  "file_path": "src/services/payment.ts",
+  "symbol_name": "processRefund",
+  "symbol_type": "function",
+  "line_start": 45,
+  "line_end": 82,
+  "purpose": "Processes a refund request via Stripe, validates eligibility, and updates order state",
+  "parent_file_id": "src/services/payment.ts",
+  "last_indexed": "2026-04-06T12:00:00Z"
+}
+```
+
+**Why local-first:** No external infrastructure dependency. The vector DB lives inside the project (`.nomos/vectors/`), is version-controllable, and works offline.
+
+### Stage 3 — Querying
+
+**Trigger:** `arc search "<natural language query>"`
+
+**Process:**
+1. Convert the user's query string into a vector using the same embedding model.
+2. Perform a nearest-neighbor search against the stored vectors.
+3. Rank results by similarity score (0.0–1.0), merging file-level and symbol-level matches.
+4. **Dependency-aware enrichment:** For each result, look up the file's graph metadata from Phase 3 (`project_map.json`) and attach impact context — graph depth, number of dependents, and core/leaf classification.
+5. Return the top-K results with file path, symbol (if applicable), line range, purpose summary, similarity score, and dependency impact.
+
+**Example:**
+```bash
+$ arc search "how is refund handled?"
+
+Results (top 3):
+
+  1. src/services/payment.ts :: processRefund()    [0.96]  L45-82
+     "Processes a refund request via Stripe, validates eligibility, updates order state"
+     ⚠ Core Module (depth 5) — modifying this affects 10 dependents
+
+  2. src/services/payment.ts                       [0.91]
+     "Handles payment processing and refund logic"
+     ⚠ Core Module (depth 5) — modifying this affects 10 dependents
+
+  3. src/middleware/billing.ts                      [0.84]
+     "Validates billing state before checkout"
+     Leaf Module (depth 1) — 2 dependents
+```
+
+---
+
+## CLI Commands
+
+| Command | Description |
+|---|---|
+| `arc index` | Build or rebuild the vector index from `project_map.json` and `.semantic.md` files |
+| `arc index --incremental` | Only re-index files modified since the last indexing run |
+| `arc search "<query>"` | Perform a semantic search and display ranked results |
+| `arc search "<query>" --top <N>` | Limit results to top N matches (default: 5) |
+| `arc search "<query>" --threshold <score>` | Only show results above the similarity threshold (default: 0.7) |
+| `arc search "<query>" --json` | Output results as JSON for programmatic consumption |
+
+---
+
+## Architecture
+
+```
+arc search "query"
+       │
+       ▼
+┌─────────────┐     ┌──────────────────┐
+│  Query       │────▶│  Embedding API   │
+│  Processor   │     │  (Gemini/OpenAI) │
+└─────────────┘     └──────────────────┘
+       │                      │
+       │ query vector         │
+       ▼                      │
+┌─────────────┐               │
+│  Vector DB   │◀──────────────┘
+│  (LanceDB)  │   file + symbol vectors (at index time)
+└─────────────┘
+       │
+       │ ranked results
+       ▼
+┌──────────────────┐     ┌──────────────────┐
+│  Graph Enricher   │────▶│  project_map.json │
+│  (Phase 3 data)  │     │  (dependency graph)│
+└──────────────────┘     └──────────────────┘
+       │
+       │ enriched results (+ depth, dependents, impact)
+       ▼
+┌─────────────┐
+│  Formatter   │──▶  CLI output / JSON
+└─────────────┘
+```
+
+### Module Breakdown
+
+| Module | Responsibility |
+|---|---|
+| `src/search/indexer.ts` | Reads project map + semantic files, chunks text at file-level and symbol-level, calls embedding API, writes to vector DB |
+| `src/search/symbol-extractor.ts` | Extracts individual classes/functions from project map exports for symbol-level indexing |
+| `src/search/embedder.ts` | Wraps the embedding API (Gemini/OpenAI), handles batching and rate limits |
+| `src/search/vector-store.ts` | LanceDB interface — create table, upsert vectors, query by similarity |
+| `src/search/graph-enricher.ts` | Reads Phase 3 dependency graph from `project_map.json`, attaches depth/dependents/impact metadata to search results |
+| `src/search/query-engine.ts` | Orchestrates the search flow: embed query → search → enrich → rank → format |
+| `src/commands/search.ts` | CLI command handler for `arc search` |
+| `src/commands/index.ts` | CLI command handler for `arc index` |
+
+---
+
+## State Management
+
+Following the project convention: **JSON is the source of truth**.
+
+**Index metadata** is stored at `.nomos/vectors/index-meta.json`:
+```json
+{
+  "last_full_index": "2026-04-06T12:00:00Z",
+  "total_files_indexed": 142,
+  "embedding_model": "text-embedding-004",
+  "vector_dimensions": 768,
+  "files": {
+    "src/services/payment.ts": {
+      "last_indexed": "2026-04-06T12:00:00Z",
+      "content_hash": "sha256:abc123..."
+    }
+  }
+}
+```
+
+**Incremental indexing** compares `content_hash` of each file against the stored hash. Only changed files are re-embedded, reducing API calls and indexing time.
+
+---
+
+## Configuration
+
+Added to `.nomos-config.json`:
+```json
+{
+  "search": {
+    "embedding_provider": "gemini",
+    "embedding_model": "text-embedding-004",
+    "vector_db": "lancedb",
+    "vector_store_path": ".nomos/vectors",
+    "default_top_k": 5,
+    "default_threshold": 0.7,
+    "batch_size": 50,
+    "max_concurrent_requests": 5
+  }
+}
+```
+
+---
+
+## Edge Cases and Constraints
+
+| Concern | Mitigation |
+|---|---|
+| **Large projects (1000+ files)** | Batch embedding requests (50 per batch), incremental indexing, progress bar in CLI |
+| **API rate limits** | Exponential backoff with jitter, configurable concurrency limit |
+| **Stale index** | Warn the user if index is older than the latest file modification; suggest `arc index --incremental` |
+| **No `.semantic.md` files** | Fall back to indexing raw `project_map.json` entries only; warn that results may be less accurate |
+| **Offline usage** | If embedding API is unreachable, search against the existing local index (no re-indexing) |
+| **Cost control** | Track token usage per indexing run, log to `.nomos/vectors/usage.json`, respect budget limits from `.nomos-config.json` |
+
+---
+
+## Success Criteria
+
+- [ ] `arc index` builds a complete vector index from project map and semantic files
+- [ ] `arc index` generates both file-level and symbol-level embeddings
+- [ ] `arc index --incremental` only re-indexes changed files (verified by content hash)
+- [ ] `arc search` returns semantically relevant results for natural language queries
+- [ ] Results can point to a specific function/class with line range, not just the file
+- [ ] Results include dependency impact context (graph depth, dependents count, core/leaf classification)
+- [ ] Results include file path, similarity score, and purpose summary
+- [ ] Search completes in under 2 seconds for projects with up to 500 indexed files
+- [ ] JSON output mode (`--json`) produces valid, parseable output
+- [ ] Graceful degradation when API is unavailable (search existing index, skip re-indexing)
+
+---
+
+## Dependencies
+
+| Dependency | Purpose |
+|---|---|
+| `lancedb` | Local embedded vector database |
+| `@google/generative-ai` or equivalent | Embedding API client |
+| Existing `project_map.json` | Source data (from mapping phase) — used for file content, symbol exports, AND dependency graph |
+| Existing `.semantic.md` files | Enriched semantic descriptions (from mapping phase) |
+| Phase 3 dependency graph | Graph depth, dependents count, and impact data for dependency-aware enrichment |
+
+---
+
+## Out of Scope (Phase 7)
+
+- Cross-project search (searching across multiple repositories)
+- Real-time index updates (watching file changes via filesystem events)
+- Custom embedding model training or fine-tuning
+- UI/dashboard for search results (CLI only)
+- Hybrid search (combining keyword + semantic results)

package/docs/prompts/USAGE.md

@@ -0,0 +1,312 @@
+# AI Agent Pipeline — Usage Guide
+
+## The 4-Agent Pipeline
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                                                                   │
+│   TASK.md / TASK.yaml                                             │
+│        │                                                          │
+│        ▼                                                          │
+│  ┌─────────────┐    plan.yaml     ┌─────────────┐                │
+│  │  ARCHITECT  │ ───────────────► │  RED TEAM   │                │
+│  │  architect  │                  │  red_team   │                │
+│  └─────────────┘                  └──────┬──────┘                │
+│                                          │ audit.yaml            │
+│                                          ▼                        │
+│                                   ┌─────────────┐                │
+│                                   │   RESOLVER  │                │
+│                         ┌─────────│   hardener  │                │
+│                         │ REVISE  └──────┬──────┘                │
+│                         │               │ blueprint.yaml         │
+│                         │               ▼                        │
+│                         │        ┌─────────────┐                │
+│                         │        │  OPERATOR   │                │
+│                         │        │   executer  │                │
+│                         │        └──────┬──────┘                │
+│                         │               │                        │
+│                         │    ┌──────────▼──────────┐            │
+│                         │    │  Phase 1 → Session 1 │            │
+│                         │    │  Phase 2 → Session 2 │            │
+│                         │    │  Phase N → Session N │            │
+│                         │    └─────────────────────┘            │
+│                         │                                        │
+│                         └── (if audit verdict = REVISE,          │
+│                              loop back to Architect)             │
+└─────────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Agent 1: Architect
+
+**File:** `docs/prompts/architect.md`
+**Role:** Analyzes the task and produces the initial atomic execution plan.
+**Output:** `plan.yaml`
+
+### What to send
+
+```
+Use this prompt @docs/prompts/architect.md to plan this task.
+
+<task>
+  [Full task description, requirements, and acceptance criteria]
+</task>
+
+<codebase>
+  @src/relevant-file-1.ts
+  @src/relevant-file-2.ts
+</codebase>
+
+<tree>
+  [Output of: tree -L 3 or paste directory structure]
+</tree>
+
+<config>
+  @package.json
+  @tsconfig.json
+</config>
+
+<rules>
+  @rules/core.json
+  @rules/backend.json
+</rules>
+```
+
+### What you get back
+
+A `plan.yaml` file with:
+- `context_analysis` — tech stack, touch zone, fragile zone
+- `steps[]` — atomic steps with `step_id`, `action`, `file_path`, `validation`, `rollback`
+- `risk_assessment` — overall risk and failure scenarios
+- `compliance` — rules check results
+
+### Save the output as
+
+```
+tasks-management/plans/TASK-001/plan.yaml
+```
+
+---
+
+## Agent 2: Red Team
+
+**File:** `docs/prompts/red_team.md`
+**Role:** Audits the plan for security vulnerabilities, architectural flaws, AI divergence risks, and rollback failures.
+**Input:** `plan.yaml` from Architect
+**Output:** `audit.yaml`
+
+### What to send
+
+```
+Use this prompt @docs/prompts/red_team.md to audit this plan.
+
+<proposed_plan>
+  @tasks-management/plans/TASK-001/plan.yaml
+</proposed_plan>
+
+<codebase>
+  @src/relevant-file-1.ts
+  @src/relevant-file-2.ts
+</codebase>
+
+<tree>
+  [directory structure]
+</tree>
+
+<config>
+  @package.json
+</config>
+
+<rules>
+  @rules/core.json
+</rules>
+```
+
+### What you get back
+
+An `audit.yaml` file with:
+- `verdict` — `APPROVE` | `APPROVE_WITH_NOTES` | `REVISE`
+- `system_integrity_score` — 0-100
+- `findings[]` — each finding with `severity`, `step_id`, `description`, `recommendation`
+- `negative_constraints[]` — list of hard "DO NOT" rules
+- `rollback_assessment` — which rollbacks are weak or broken
+
+### Save the output as
+
+```
+tasks-management/plans/TASK-001/audit.yaml
+```
+
+### Decision after audit
+
+| Verdict | Next Step |
+|---------|-----------|
+| `APPROVE` | Skip Resolver → go directly to Operator |
+| `APPROVE_WITH_NOTES` | Send to Resolver (optional improvements) |
+| `REVISE` | Send to Resolver (mandatory) |
+
+---
+
+## Agent 3: Resolver
+
+**File:** `docs/prompts/hardener.md`
+**Role:** Takes the original plan + audit findings and produces the Final Hardened Blueprint. Resolves every finding, injects constraints, strengthens rollbacks, and eliminates ambiguity.
+**Input:** `plan.yaml` + `audit.yaml`
+**Output:** `blueprint.yaml`
+
+### What to send
+
+```
+Use this prompt @docs/prompts/hardener.md to resolve this audit.
+
+<original_plan>
+  @tasks-management/plans/TASK-001/plan.yaml
+</original_plan>
+
+<audit_report>
+  @tasks-management/plans/TASK-001/audit.yaml
+</audit_report>
+
+<codebase>
+  @src/relevant-file-1.ts
+  @src/relevant-file-2.ts
+</codebase>
+
+<tree>
+  [directory structure]
+</tree>
+
+<rules>
+  @rules/core.json
+</rules>
+```
+
+### What you get back
+
+A `blueprint.yaml` file with:
+- `version: "2.0-HARDENED"` (or `"1.0-APPROVED"` if audit was clean)
+- `resolution_log` — what was changed and why
+- `steps[]` — same structure as plan but hardened, with `CONSTRAINT:` injections
+- `integrity_check` — confirms all findings resolved, dependency chain valid
+- `changelog` — diff: steps modified/added/removed
+
+### Save the output as
+
+```
+tasks-management/plans/TASK-001/blueprint.yaml
+```
+
+---
+
+## Agent 4: Operator
+
+**File:** `docs/prompts/executer.md`
+**Role:** Executes one phase of the blueprint per session. Stops after the phase is complete or if any step fails. Outputs a session state for the next session to resume.
+**Input:** `blueprint.yaml` + phase number + (optional) previous session state
+**Output:** `execution_report.yaml` + updated `session_state.yaml`
+
+### First session (Phase 1)
+
+```
+Use this prompt @docs/prompts/executer.md to execute phase 1
+from this blueprint @tasks-management/plans/TASK-001/blueprint.yaml
+
+Environment:
+- OS: Ubuntu 22.04
+- Node: 20.x
+- DB: connected (postgres://localhost:5432/nomos_dev)
+```
+
+### Next sessions (Phase 2, 3, ...)
+
+```
+Use this prompt @docs/prompts/executer.md to execute phase 2
+from this blueprint @tasks-management/plans/TASK-001/blueprint.yaml
+
+<session_state>
+  @tasks-management/plans/TASK-001/session_state.yaml
+</session_state>
+
+Environment:
+- OS: Ubuntu 22.04
+- Node: 20.x
+- DB: connected (postgres://localhost:5432/nomos_dev)
+```
+
+### What you get back
+
+An `execution_report.yaml` with:
+- `status` — `SUCCESS` | `PARTIAL_FAILURE` | `ROLLED_BACK` | `ABORTED`
+- `steps_log[]` — step-by-step results with validation status
+- `session_state` — save this for the next session
+- `state_delta` — files created/modified/deleted
+- `post_mortem` — (only on failure) root cause + recommended action
+- `phase_summary.ready_for_next_phase` — `true` / `false`
+
+### Save the outputs as
+
+```
+tasks-management/plans/TASK-001/execution_report_phase1.yaml
+tasks-management/plans/TASK-001/session_state.yaml   ← pass this to next session
+```
+
+---
+
+## Full Example — End to End
+
+```
+TASK: Add JWT authentication to the API
+
+Step 1 — Architect
+  Input:  task description + src/ + package.json + rules/
+  Output: plan.yaml  (12 atomic steps across 3 phases)
+
+Step 2 — Red Team
+  Input:  plan.yaml + src/ + rules/
+  Output: audit.yaml  (verdict: REVISE — 1 critical finding: JWT secret in config)
+
+Step 3 — Resolver
+  Input:  plan.yaml + audit.yaml + src/ + rules/
+  Output: blueprint.yaml v2.0-HARDENED  (secret moved to env var, rollback strengthened)
+
+Step 4a — Operator Session 1
+  Input:  blueprint.yaml + phase 1 + environment
+  Output: execution_report_phase1.yaml (SUCCESS) + session_state.yaml
+
+Step 4b — Operator Session 2
+  Input:  blueprint.yaml + phase 2 + session_state.yaml + environment
+  Output: execution_report_phase2.yaml (SUCCESS) + session_state.yaml
+
+Step 4c — Operator Session 3
+  Input:  blueprint.yaml + phase 3 + session_state.yaml + environment
+  Output: execution_report_phase3.yaml (SUCCESS) — ready_for_next_phase: false ✓
+```
+
+---
+
+## File Naming Convention
+
+```
+tasks-management/
+└── plans/
+    └── TASK-001/
+        ├── plan.yaml                      ← Architect output
+        ├── audit.yaml                     ← Red Team output
+        ├── blueprint.yaml                 ← Resolver output
+        ├── session_state.yaml             ← Operator: updated each session
+        ├── execution_report_phase1.yaml   ← Operator: one per session
+        ├── execution_report_phase2.yaml
+        └── execution_report_phase3.yaml
+```
+
+---
+
+## Quick Reference
+
+| Agent | Prompt File | Takes | Produces | Key Field |
+|-------|-------------|-------|----------|-----------|
+| Architect | `architect.md` | task + code + rules | `plan.yaml` | `steps[]` |
+| Red Team | `red_team.md` | `plan.yaml` + code | `audit.yaml` | `verdict` |
+| Resolver | `hardener.md` | `plan.yaml` + `audit.yaml` | `blueprint.yaml` | `integrity_check` |
+| Operator | `executer.md` | `blueprint.yaml` + phase# | `execution_report.yaml` | `ready_for_next_phase` |