@nathapp/nax 0.18.1

package/docs/v0.10-prompt-optimizer.md

@@ -0,0 +1,234 @@
+# v0.10 — Prompt Optimizer
+
+**Status:** Draft  
+**Author:** Subrina  
+**Date:** 2026-02-22  
+**Base:** v0.9.2  
+
+## Overview
+
+Abstract prompt optimization layer that sits between prompt assembly and agent execution. Reduces token usage while preserving semantic meaning. Supports built-in strategies (rule-based, LLM) and external/custom optimizers.
+
+## Motivation
+
+- Prompts assembled by `promptStage` contain verbose formatting, redundant context, and boilerplate that inflates token counts
+- Coding agents (Claude, etc.) perform equally well with concise instructions
+- No standardized hook point exists for prompt transformation before execution
+- External teams may want to plug in specialized compressors (LLMLingua, custom APIs)
+
+## Architecture
+
+```
+promptStage → optimizerStage (NEW) → executionStage
+                    ↓
+           IPromptOptimizer (interface)
+            ├── NoopOptimizer (default, passthrough)
+            ├── RuleBasedOptimizer (built-in)
+            │   ├── strip redundant whitespace/formatting
+            │   ├── deduplicate context vs constitution overlap
+            │   └── compact acceptance criteria
+            └── LlmOptimizer (built-in, optional)
+                └── uses fast-tier model to rewrite verbose prompts
+```
+
+### File Structure
+
+```
+src/optimizer/
+  types.ts              # IPromptOptimizer, PromptOptimizerInput, PromptOptimizerResult
+  index.ts              # resolveOptimizer() factory
+  noop.optimizer.ts     # NoopOptimizer (passthrough)
+  rule-based.optimizer.ts  # RuleBasedOptimizer
+  llm.optimizer.ts      # LlmOptimizer
+  loader.ts             # Custom module loader
+src/pipeline/stages/
+  optimizer.ts          # optimizerStage
+```
+
+## Interface
+
+```typescript
+export interface IPromptOptimizer {
+  /** Unique optimizer name */
+  name: string;
+
+  /**
+   * Optimize a prompt before it is sent to the coding agent.
+   *
+   * Implementations MUST preserve all technical requirements,
+   * acceptance criteria semantics, and code references.
+   */
+  optimize(input: PromptOptimizerInput): Promise<PromptOptimizerResult>;
+}
+
+export interface PromptOptimizerInput {
+  /** Assembled prompt from promptStage */
+  prompt: string;
+  /** Stories being executed (for context) */
+  stories: UserStory[];
+  /** Raw context markdown (pre-assembly, for dedup detection) */
+  contextMarkdown?: string;
+  /** Nax configuration */
+  config: NaxConfig;
+}
+
+export interface PromptOptimizerResult {
+  /** Optimized prompt */
+  prompt: string;
+  /** Estimated token count before optimization */
+  originalTokens: number;
+  /** Estimated token count after optimization */
+  optimizedTokens: number;
+  /** Savings percentage (0-1) */
+  savings: number;
+  /** List of applied optimization rules/passes */
+  appliedRules: string[];
+}
+```
+
+## Built-in Optimizers
+
+### NoopOptimizer
+
+Passthrough — returns prompt unchanged with zero savings. Default when optimizer is disabled or strategy is unrecognized.
+
+### RuleBasedOptimizer
+
+Deterministic, zero-cost transformations:
+
+| Rule | Description |
+|:-----|:------------|
+| `stripWhitespace` | Collapse multiple blank lines to single, trim trailing whitespace |
+| `compactCriteria` | Convert verbose acceptance criteria to terse bullet format |
+| `deduplicateContext` | Remove context sections that duplicate constitution content |
+| `maxPromptTokens` | If prompt exceeds threshold, aggressively trim context (not story/AC) |
+
+### LlmOptimizer
+
+Uses a fast-tier model (e.g., haiku) to rewrite the prompt:
+
+- Only triggers when prompt exceeds `minPromptTokens` threshold
+- System prompt: "Rewrite this coding task prompt to be {targetReduction}% shorter. Preserve ALL technical requirements, acceptance criteria, file paths, and code references exactly. Remove only verbose prose, redundant explanations, and formatting fluff."
+- Falls back to original prompt if LLM call fails
+
+## Configuration
+
+New `optimizer` section in `nax/config.json`:
+
+```json
+{
+  "optimizer": {
+    "enabled": true,
+    "strategy": "rule-based",
+    "strategies": {
+      "rule-based": {
+        "stripWhitespace": true,
+        "compactCriteria": true,
+        "deduplicateContext": true,
+        "maxPromptTokens": 8000
+      },
+      "llm": {
+        "model": "fast",
+        "targetReduction": 0.4,
+        "minPromptTokens": 2000
+      },
+      "custom": {
+        "module": "./my-optimizer.js",
+        "options": {}
+      }
+    }
+  }
+}
+```
+
+All fields are optional. When `optimizer` is absent or `enabled: false`, the NoopOptimizer is used.
+
+## Pipeline Integration
+
+```typescript
+// src/pipeline/stages/optimizer.ts
+export const optimizerStage: PipelineStage = {
+  name: "optimizer",
+  enabled: (ctx) => ctx.config.optimizer?.enabled ?? false,
+  async execute(ctx: PipelineContext): Promise<StageResult> {
+    const optimizer = resolveOptimizer(ctx.config);
+    const result = await optimizer.optimize({
+      prompt: ctx.prompt!,
+      stories: ctx.stories,
+      contextMarkdown: ctx.contextMarkdown,
+      config: ctx.config,
+    });
+    ctx.prompt = result.prompt;
+    logger.info("optimizer", `${optimizer.name}: ${result.savings}% savings`, {
+      originalTokens: result.originalTokens,
+      optimizedTokens: result.optimizedTokens,
+      rules: result.appliedRules,
+    });
+    return { action: "continue" };
+  },
+};
+```
+
+Stage order: `routing → context → constitution → prompt → **optimizer** → execution → review → acceptance → completion`
+
+## Custom/External Optimizer
+
+Custom optimizers are loaded via the **plugin system** (see `v0.10-plugin-system.md`). A plugin providing an optimizer implements the `IPromptOptimizer` interface:
+
+```javascript
+// nax/plugins/my-optimizer/index.js
+module.exports = {
+  name: "my-custom-optimizer",
+  version: "1.0.0",
+  provides: ["optimizer"],
+  extensions: {
+    optimizer: {
+      name: "llmlingua",
+      async optimize(input) {
+        const optimized = await externalService.compress(input.prompt);
+        return {
+          prompt: optimized,
+          originalTokens: estimateTokens(input.prompt),
+          optimizedTokens: estimateTokens(optimized),
+          savings: calculateSavings(input.prompt, optimized),
+          appliedRules: ["external-compression"],
+        };
+      },
+    },
+  },
+};
+```
+
+Plugin optimizers override the built-in strategy when loaded. See plugin system spec for loading and conflict resolution.
+
+## User Stories
+
+| # | Title | Complexity | Est. LOC | Dependencies |
+|:--|:------|:-----------|:---------|:-------------|
+| US-001 | Define IPromptOptimizer interface, types, and NoopOptimizer | simple | 60 | — |
+| US-002 | Implement RuleBasedOptimizer | medium | 150 | US-001 |
+| US-003 | Implement LlmOptimizer with fast-tier rewrite | medium | 120 | US-001 |
+| US-004 | Add optimizerStage to pipeline + config schema + resolveOptimizer factory | medium | 100 | US-001 |
+| US-005 | Custom module loader for external optimizers | simple | 60 | US-001, US-004 |
+
+## Token Estimation
+
+Use a simple heuristic for token counting (no external tokenizer dependency):
+
+```typescript
+/** ~4 chars per token for English text (rough estimate) */
+export function estimateTokens(text: string): number {
+  return Math.ceil(text.length / 4);
+}
+```
+
+This is sufficient for savings calculation. Exact counts are not required.
+
+## Design Decisions
+
+1. **NoopOptimizer as default** — zero breaking changes, fully opt-in
+2. **Strategy pattern** — `resolveOptimizer()` factory selects implementation based on config
+3. **Composable (future)** — interface supports chaining (rule-based → LLM) in a future version
+4. **Metrics in result** — always report token savings for cost tracking and logging
+5. **Fail-safe** — all optimizers fall back to original prompt on error
+6. **No external dependencies** — rule-based and LLM optimizers use only existing nax infrastructure

package/docs/v0.3-spec.md

@@ -0,0 +1,244 @@
+# ngent v0.3 Specification
+**Date:** 2026-02-17
+**Status:** Draft
+
+## Overview
+
+v0.3 adds LLM-enhanced analysis and a constitution system inspired by Relentless. The goal: replace keyword-based routing with codebase-aware intelligence, and give ngent a governance framework that agents reference during execution.
+
+## Feature 1: LLM-Enhanced Analyze
+
+### Problem
+`ngent analyze` currently uses keyword matching to classify stories. "Add caching" looks simple by keywords, but if the codebase has no cache infrastructure, it's actually complex. Only an LLM reading the code can catch that.
+
+### Solution
+During `ngent analyze`, make one cheap LLM call (haiku) that reads the codebase structure + story descriptions and returns structured routing decisions.
+
+### Flow
+```
+ngent analyze
+  1. Parse tasks.md → extract stories (existing)
+  2. Scan codebase → generate summary (NEW)
+     - file tree (src/ only, max depth 3)
+     - package.json dependencies
+     - existing test patterns
+  3. LLM call → classify all stories in one shot (NEW)
+     Input: codebase summary + all stories
+     Output: JSON array with per-story:
+       - complexity: simple/medium/complex/expert
+       - relevantFiles: string[] (files the story will likely touch)
+       - reasoning: string (why this classification)
+       - estimatedLOC: number
+       - risks: string[] (potential issues)
+  4. Merge LLM output into prd.json (existing routing fields)
+  5. Fall back to keyword matching if LLM call fails (graceful degradation)
+```
+
+### Classification Prompt (single call)
+```
+You are a code complexity classifier. Given a codebase summary and user stories,
+classify each story's implementation complexity.
+
+CODEBASE:
+{fileTree}
+{dependencies}
+{existingPatterns}
+
+STORIES:
+{storiesJson}
+
+For each story, respond with JSON:
+[{
+  "storyId": "US-001",
+  "complexity": "simple|medium|complex|expert",
+  "relevantFiles": ["src/path/to/file.ts"],
+  "reasoning": "Why this complexity level",
+  "estimatedLOC": 50,
+  "risks": ["No existing cache layer"]
+}]
+```
+
+### Config
+```json
+{
+  "analyze": {
+    "llmEnhanced": true,
+    "classifierModel": "fast",
+    "fallbackToKeywords": true,
+    "maxCodebaseSummaryTokens": 5000
+  }
+}
+```
+
+### Cost
+~$0.01-0.02 per analyze run (one haiku call). Pays for itself by preventing one misrouted story.
+
+### Acceptance Criteria
+- [ ] Codebase scanner generates file tree + dependency summary
+- [ ] Single LLM call classifies all stories with structured JSON output
+- [ ] relevantFiles populated in prd.json for context builder to use
+- [ ] Falls back to keyword matching on LLM failure (timeout, parse error)
+- [ ] Config option to disable LLM-enhanced analysis
+- [ ] Works with existing tasks.md format (no changes required)
+
+---
+
+## Feature 2: Constitution System
+
+### Problem
+ngent agents execute stories with a generic prompt. There's no project-specific governance — coding standards, architectural rules, testing requirements. Relentless solves this with a versioned constitution that agents reference.
+
+### Solution
+Add a `constitution.md` file to the ngent project directory that gets injected into every agent session prompt.
+
+### Structure
+```
+project/
+  ngent/
+    config.json
+    constitution.md    ← NEW
+    tasks.md
+    prd.json
+```
+
+### Constitution Format
+```markdown
+# Project Constitution
+
+## Coding Standards
+- Use Bun-native APIs, not Node.js
+- Functional style for pure logic
+- Max 400 lines per file
+
+## Testing Requirements  
+- All new code must have tests
+- Use bun:test (describe/test/expect)
+- Target 80%+ coverage
+
+## Architecture Rules
+- Each module: types.ts, implementation, index.ts barrel
+- No circular dependencies
+- Immutable data patterns
+
+## Forbidden Patterns
+- No `any` type
+- No console.log in production code
+- No hardcoded secrets
+```
+
+### How It Works
+1. `ngent init` creates a starter `constitution.md` (template with common rules)
+2. Human customizes it for their project
+3. During `ngent run`, constitution is injected into every agent prompt:
+   ```
+   CONSTITUTION (follow these rules strictly):
+   {constitutionContent}
+   
+   STORY:
+   {storyPrompt}
+   ```
+4. Constitution content counts against context token budget (priority 95 — highest after story itself)
+5. Large constitutions get truncated with warning
+
+### Config
+```json
+{
+  "constitution": {
+    "enabled": true,
+    "path": "constitution.md",
+    "maxTokens": 2000
+  }
+}
+```
+
+### Acceptance Criteria
+- [ ] `ngent init` generates starter constitution.md
+- [ ] Constitution loaded and injected into all agent prompts
+- [ ] Counts against context token budget (priority 95)
+- [ ] Warning if constitution exceeds maxTokens
+- [ ] Config option to disable
+- [ ] Works with both single-session and TDD modes
+
+---
+
+## Feature 3: Simplified Review Phase
+
+### Problem
+Stories are marked complete after tests pass. No automated quality gate for lint, typecheck, or code quality issues that tests don't catch.
+
+### Solution
+After each story's implementation, run a quick verification phase before marking it as passed.
+
+### Flow
+```
+Story implementation complete (tests pass)
+  → Review phase:
+    1. bun run typecheck (if tsconfig exists)
+    2. bun run lint (if lint script exists)
+    3. bun test (final verification)
+  → All pass → mark story as passed
+  → Any fail → mark story as failed (with review failure reason)
+```
+
+### Config
+```json
+{
+  "review": {
+    "enabled": true,
+    "checks": ["typecheck", "lint", "test"],
+    "commands": {
+      "typecheck": "bun run typecheck",
+      "lint": "bun run lint",
+      "test": "bun test"
+    }
+  }
+}
+```
+
+### Acceptance Criteria
+- [ ] Review phase runs after each story implementation
+- [ ] Configurable checks (typecheck, lint, test)
+- [ ] Custom commands per check
+- [ ] Failure in review marks story as failed with reason
+- [ ] Can be disabled via config
+- [ ] Review results logged in progress.txt
+
+---
+
+## Feature 4: SpecKit-Inspired Workflow Commands (Optional)
+
+Inspired by Relentless's command chain: specify → plan → tasks → analyze → implement.
+
+### New Commands
+```bash
+ngent specify "Add URL shortener with analytics"
+  → Generates spec.md from natural language (sonnet call)
+
+ngent plan
+  → LLM-enhanced analyze (Feature 1 above)
+  → Auto-populates relevantFiles, complexity, risks
+
+ngent run
+  → Execute with constitution (Feature 2) + review (Feature 3)
+```
+
+`ngent specify` is the DECOMPOSE stage discussed earlier — optional, human can still write tasks.md manually.
+
+### Acceptance Criteria
+- [ ] `ngent specify` generates structured spec.md from description
+- [ ] `ngent plan` is alias for LLM-enhanced analyze
+- [ ] Both commands optional — existing tasks.md workflow still works
+
+---
+
+## Priority Order
+
+| # | Feature | Effort | Impact |
+|:---|:---|:---|:---|
+| 1 | LLM-Enhanced Analyze | M | High — fixes routing accuracy |
+| 2 | Constitution System | S-M | High — governs all agent behavior |
+| 3 | Review Phase | S | Medium — quality gate |
+| 4 | SpecKit Commands | M | Low — convenience, not essential |
+
+## Estimated Total
+~600-800 LOC. Features 1-3 are the core of v0.3. Feature 4 is nice-to-have.

package/docs/v0.4-spec.md

@@ -0,0 +1,140 @@
+# ngent v0.4 Specification
+**Date:** 2026-02-17
+**Status:** Draft
+
+## Feature: Acceptance Validation & Self-Correcting Loop
+
+### Problem
+ngent verifies stories individually (tests pass?) but never validates the feature as a whole against the human's acceptance criteria. Stories can all pass but the feature can still mismatch what was specified.
+
+### Solution
+Two-phase acceptance validation:
+
+1. **At analyze time:** Generate acceptance tests from spec.md ACs
+2. **After all stories complete:** Run acceptance tests as final gate
+3. **On failure:** Auto-generate fix stories targeting only the gaps, retry (max 2 loops)
+
+### Flow
+```
+ngent analyze
+  → Decompose spec → stories (existing)
+  → NEW: Generate acceptance tests from spec.md ACs
+  → Write to ngent/features/<name>/acceptance.test.ts
+
+ngent run
+  → Execute all stories (existing pipeline)
+  → All stories pass
+  → NEW: Acceptance stage — run acceptance.test.ts
+    → All pass → ✅ Feature complete
+    → Some fail →
+        Map failed ACs → relevant stories
+        Create fix stories (US-FIX-001, etc.)
+        Run fix stories only
+        Re-run acceptance tests
+        Pass? → ✅ Done
+        Fail again? → Loop (max 2 retries)
+        Still failing? → ⏸ Pause, ask human
+```
+
+### Acceptance Test Generation
+
+During `ngent analyze`, after decomposing stories, make one additional LLM call:
+
+```
+Input: spec.md acceptance criteria + codebase context + story list
+Output: acceptance.test.ts with one test per AC
+
+Example:
+  AC-2: "set(key, value, ttl) expires after ttl milliseconds"
+  →
+  test("AC-2: TTL expiry", async () => {
+    const store = new KVStore();
+    store.set("key", "value", 50);
+    await Bun.sleep(60);
+    expect(store.get("key")).toBeUndefined();
+  });
+```
+
+**Rules for acceptance tests:**
+- Test observable behavior only (not implementation details)
+- Each AC maps to exactly one test
+- Tests are independent (no shared state)
+- Tests must be runnable without mocking (integration-level)
+
+### Fix Story Generation
+
+When acceptance tests fail:
+
+```typescript
+interface FixStory {
+  id: string;           // "US-FIX-001"
+  title: string;        // "Fix: AC-2 TTL expiry timing"
+  failedAC: string;     // "AC-2"
+  testOutput: string;   // actual vs expected from test failure
+  relatedStories: string[]; // ["US-002"] — original stories that built this
+  description: string;  // LLM-generated fix description
+}
+```
+
+The fix story prompt includes:
+- The failed acceptance test output
+- The relevant source code (from relatedStories)
+- The original AC text
+- Instruction: "Fix the code to make this acceptance test pass"
+
+### Pipeline Integration
+
+New pipeline stage: `acceptanceStage` — runs after `completionStage` when all stories are done.
+
+```typescript
+const defaultPipeline: PipelineStage[] = [
+  queueCheckStage,
+  routingStage,
+  constitutionStage,
+  contextStage,
+  promptStage,
+  executionStage,
+  verifyStage,
+  reviewStage,
+  completionStage,
+  acceptanceStage,    // NEW — runs only when all stories complete
+];
+```
+
+### Config
+```json
+{
+  "acceptance": {
+    "enabled": true,
+    "maxRetries": 2,
+    "generateTests": true,
+    "testPath": "acceptance.test.ts"
+  }
+}
+```
+
+### Human Override
+```bash
+ngent accept --override AC-2 "intentional: using lazy expiry instead of exact timing"
+```
+Marks an AC as accepted despite test failure. Stored in prd.json:
+```json
+{
+  "acceptanceOverrides": {
+    "AC-2": "intentional: using lazy expiry instead of exact timing"
+  }
+}
+```
+
+### Acceptance Criteria for This Feature
+- [ ] Acceptance tests generated from spec.md ACs during analyze
+- [ ] Tests run after all stories complete
+- [ ] Failed ACs mapped to relevant stories
+- [ ] Fix stories auto-generated and executed
+- [ ] Max 2 retry loops before pausing for human
+- [ ] `ngent accept --override` skips specific ACs
+- [ ] Works with existing pipeline (new stage, not inline)
+- [ ] Config to disable acceptance validation
+
+### Estimated Effort
+~400-600 LOC. Medium complexity — mostly LLM prompt engineering for test generation and fix story creation.