prjct-cli 1.8.0 → 1.9.0

package/CHANGELOG.md

@@ -1,5 +1,107 @@
 # Changelog
 
+## [1.9.0] - 2026-02-07
+
+### Features
+
+- add structured output schema to all LLM prompts (PRJ-264) (#150)
+- add mandatory model specification to AI provider (PRJ-265) (#149)
+
+### Bug Fixes
+
+- replace keyword domain detection with LLM semantic classification (PRJ-299) (#148)
+
+
+## [1.10.0] - 2026-02-07
+
+### Features
+- **Add structured output schema to all LLM prompts (PRJ-264)**: LLM prompts now include explicit JSON output schemas. Responses are validated with Zod before use. Invalid responses trigger re-prompt with structured error feedback.
+
+### Implementation Details
+- New `core/schemas/llm-output.ts`: Zod schemas for task classification, agent assignment, and subtask breakdown responses. Schema registry (`OUTPUT_SCHEMAS`) with examples that self-validate. `renderSchemaForPrompt()` serializes schemas as markdown format instructions for prompt injection.
+- New `core/agentic/response-validator.ts`: `validateLLMResponse()` handles JSON parsing (plain and markdown-wrapped `\`\`\`json` fences), Zod validation, and typed results. `buildReprompt()` generates retry messages with specific validation errors.
+- Replaced manual field-by-field validation in `domain-classifier.ts` with `TaskClassificationSchema.safeParse()` — the schema existed (PRJ-299) but was unused.
+- Added output schema injection to `prompt-builder.ts` `build()` method with `getSchemaTypeForCommand()` mapping commands to schemas.
+- 20 new unit tests in `core/__tests__/agentic/response-validator.test.ts`
+
+### Test Plan
+
+#### For QA
+1. Run `bun test core/__tests__/agentic/response-validator.test.ts` — all 20 tests pass
+2. Run `bun test` — full suite (677 tests) passes with no regressions
+3. Run `bun run build` — build succeeds cleanly
+4. Verify `renderSchemaForPrompt('classification')` returns markdown with OUTPUT FORMAT header
+5. Verify `validateLLMResponse()` handles plain JSON, markdown-wrapped JSON, and rejects non-JSON
+6. Verify OUTPUT_SCHEMAS registry examples validate against their own schemas
+
+#### For Users
+**What changed:** LLM prompts include explicit JSON output schemas. Domain classifier uses Zod validation. Response validator provides structured error handling with re-prompt.
+**How to use:** Automatic — schemas injected into prompts and validation runs transparently.
+**Breaking changes:** None — all changes are additive.
+
+## [1.9.0] - 2026-02-07
+
+### Features
+- **Add mandatory model specification to AI provider (PRJ-265)**: Provider configs now include `defaultModel`, `supportedModels`, and `minCliVersion` fields. Analysis and task metadata can record which model was used, enabling consistency tracking and mismatch warnings.
+
+### Implementation Details
+- New `core/schemas/model.ts`: Zod schemas defining supported models per provider (Claude: opus/sonnet/haiku, Gemini: 2.5-pro/2.5-flash/2.0-flash), default model resolution, semver comparison utilities, minimum CLI version validation, and model mismatch detection
+- Extended `AIProviderConfig` interface in `core/types/provider.ts` with `defaultModel`, `supportedModels`, `minCliVersion` fields
+- All 5 provider configs (Claude, Gemini, Cursor, Windsurf, Antigravity) updated with model specification fields
+- Added `modelMetadata` (optional) to `CurrentTaskSchema` in `core/schemas/state.ts` and `AnalysisSchema` in `core/schemas/analysis.ts`
+- Added `preferredModel` to `ProjectSettings` in `core/types/config.ts`
+- Added `validateCliVersion()` to `core/infrastructure/ai-provider.ts` with version warning integration into `detectProvider()`
+- Added `versionWarning` field to `ProviderDetectionResult`
+- 32 new unit tests in `core/__tests__/schemas/model.test.ts`
+
+### Test Plan
+
+#### For QA
+1. Verify `ClaudeProvider.defaultModel` is `'sonnet'` and `supportedModels` includes `['opus', 'sonnet', 'haiku']`
+2. Verify `GeminiProvider.defaultModel` is `'2.5-flash'` and `supportedModels` includes `['2.5-pro', '2.5-flash', '2.0-flash']`
+3. Verify multi-model IDEs (Cursor, Windsurf) have `null` defaultModel and empty supportedModels
+4. Run `bun test core/__tests__/schemas/model.test.ts` — all 32 tests pass
+5. Run `bun test` — full suite (657 tests) passes with no regressions
+6. Run `bun run build` — build succeeds cleanly
+
+#### For Users
+**What changed:** Provider configs now include model specification fields. Analysis and task metadata can record which model was used. Version validation warns if CLI is outdated.
+**How to use:** Existing configs work unchanged — model fields have sensible defaults. New `preferredModel` setting available in project settings.
+**Breaking changes:** None — all new fields are optional or have defaults.
+
+## [1.8.1] - 2026-02-07
+
+### Bug Fixes
+- **Replace keyword domain detection with LLM semantic classification (PRJ-299)**: Eliminated substring false positives in domain classification. "author" no longer matches "auth" → backend, "Build responsive dashboard" correctly routes to frontend.
+
+### Implementation Details
+- New `core/agentic/domain-classifier.ts`: LLM-based classifier with 4-level fallback chain (cache → confirmed history → Claude Haiku API → word-boundary heuristic)
+- New `core/schemas/classification.ts`: Zod schemas for TaskClassification, cache entries, and confirmed patterns
+- Replaced substring `includes()` matching in `smart-context.ts` and `orchestrator-executor.ts` with word-boundary regex (`\b`)
+- Removed ~230 lines of hardcoded keyword lists from both files
+- Classification results cached per (project + description hash) with 1-hour TTL
+- Successful classifications auto-persisted as confirmed patterns via `confirmClassification()`
+
+### Learnings
+- Word-boundary regex (`\b`) correctly rejects "author" matching "auth" because there's no boundary between "auth" and "or" in "author"
+- Using raw `fetch` to Claude API avoids adding `@anthropic-ai/sdk` dependency while keeping vendor-neutral design
+- Centralized classifier in `domain-classifier.ts` consumed by both `smart-context.ts` and `orchestrator-executor.ts` eliminates duplication
+
+### Test Plan
+
+#### For QA
+1. Run `bun test` — all 625 tests should pass
+2. Verify `detectDomain('Fix the author display on profile page')` returns `frontend` (not `backend`)
+3. Verify `detectDomain('Build responsive dashboard')` returns `frontend` (not `general`)
+4. Verify `detectDomain('Fix the auth middleware')` returns `backend` (standalone "auth" still works)
+5. Verify `classifyWithHeuristic` returns `general` with confidence 0.3 for unrecognizable tasks
+6. Run `bun run build` — build should succeed
+
+#### For Users
+**What changed:** Domain classification uses smarter word-boundary matching, eliminating false positives.
+**How to use:** No user-facing changes — classification happens automatically during `p. task`.
+**Breaking changes:** None for end users.
+
 ## [1.8.0] - 2026-02-07
 
 ### Features

package/core/__tests__/agentic/domain-classifier.test.ts

@@ -0,0 +1,330 @@
+/**
+ * Domain Classifier Tests
+ * PRJ-299: LLM-based domain classification with fallback chain
+ */
+
+import { describe, expect, it } from 'bun:test'
+import {
+  classifyWithHeuristic,
+  hashDescription,
+  type ProjectContext,
+} from '../../agentic/domain-classifier'
+
+// Default project context for testing (all domains available)
+const fullContext: ProjectContext = {
+  domains: {
+    hasFrontend: true,
+    hasBackend: true,
+    hasDatabase: true,
+    hasTesting: true,
+    hasDocker: true,
+  },
+  agents: ['frontend', 'backend', 'database', 'testing', 'devops'],
+  stack: { language: 'TypeScript', framework: 'Hono' },
+}
+
+// Backend-only project context
+const backendOnlyContext: ProjectContext = {
+  domains: {
+    hasFrontend: false,
+    hasBackend: true,
+    hasDatabase: false,
+    hasTesting: false,
+    hasDocker: false,
+  },
+  agents: ['backend'],
+  stack: { language: 'TypeScript', framework: 'Hono' },
+}
+
+describe('DomainClassifier PRJ-299', () => {
+  describe('classifyWithHeuristic', () => {
+    // =================================================================
+    // Substring Trap Tests (the whole reason for PRJ-299)
+    // =================================================================
+    describe('substring traps (critical fixes)', () => {
+      it('should NOT match "author" to "auth" domain', () => {
+        const result = classifyWithHeuristic('Fix the author display on profile page', fullContext)
+        // "author" should NOT trigger backend (auth)
+        // "profile page" and "display" should trigger frontend
+        expect(result.primaryDomain).not.toBe('backend')
+        expect(result.primaryDomain).toBe('frontend')
+      })
+
+      it('should match standalone "auth" to backend', () => {
+        const result = classifyWithHeuristic(
+          'Fix the auth middleware for JWT validation',
+          fullContext
+        )
+        expect(result.primaryDomain).toBe('backend')
+      })
+
+      it('should NOT match "testament" to "test" domain', () => {
+        const result = classifyWithHeuristic(
+          'Update the testament of the old testament module',
+          fullContext
+        )
+        expect(result.primaryDomain).not.toBe('testing')
+      })
+
+      it('should NOT match "button" to "but" in other domains', () => {
+        const result = classifyWithHeuristic('Add a button component', fullContext)
+        expect(result.primaryDomain).toBe('frontend')
+      })
+
+      it('should NOT match "configure" to "config" in devops', () => {
+        // "configure" without a devops context word should not go to devops
+        const result = classifyWithHeuristic('Configure the React component props', fullContext)
+        expect(result.primaryDomain).toBe('frontend')
+      })
+    })
+
+    // =================================================================
+    // Correct Classification Tests
+    // =================================================================
+    describe('frontend detection', () => {
+      it('should detect "Build responsive dashboard" as frontend', () => {
+        const result = classifyWithHeuristic('Build responsive dashboard', fullContext)
+        expect(result.primaryDomain).toBe('frontend')
+      })
+
+      it('should detect React component tasks', () => {
+        const result = classifyWithHeuristic('Create a modal dialog for user settings', fullContext)
+        expect(result.primaryDomain).toBe('frontend')
+      })
+
+      it('should detect CSS/styling tasks', () => {
+        const result = classifyWithHeuristic(
+          'Fix the layout for mobile responsive view',
+          fullContext
+        )
+        expect(result.primaryDomain).toBe('frontend')
+      })
+
+      it('should detect page/navigation tasks', () => {
+        const result = classifyWithHeuristic(
+          'Add sidebar navigation with dropdown menus',
+          fullContext
+        )
+        expect(result.primaryDomain).toBe('frontend')
+      })
+    })
+
+    describe('backend detection', () => {
+      it('should detect API endpoint tasks', () => {
+        const result = classifyWithHeuristic(
+          'Create REST API endpoint for user management',
+          fullContext
+        )
+        expect(result.primaryDomain).toBe('backend')
+      })
+
+      it('should detect middleware tasks', () => {
+        const result = classifyWithHeuristic('Add rate limiting middleware', fullContext)
+        expect(result.primaryDomain).toBe('backend')
+      })
+
+      it('should detect authentication tasks', () => {
+        const result = classifyWithHeuristic('Implement JWT authentication flow', fullContext)
+        expect(result.primaryDomain).toBe('backend')
+      })
+    })
+
+    describe('database detection', () => {
+      it('should detect schema/migration tasks', () => {
+        const result = classifyWithHeuristic(
+          'Create database migration for users table',
+          fullContext
+        )
+        expect(result.primaryDomain).toBe('database')
+      })
+
+      it('should detect connection pooling as database (not schema)', () => {
+        const result = classifyWithHeuristic('Optimize database connection pooling', fullContext)
+        expect(result.primaryDomain).toBe('database')
+      })
+
+      it('should detect ORM/Prisma tasks', () => {
+        const result = classifyWithHeuristic('Update Prisma schema with new entity', fullContext)
+        expect(result.primaryDomain).toBe('database')
+      })
+    })
+
+    describe('devops detection', () => {
+      it('should detect Docker tasks', () => {
+        const result = classifyWithHeuristic(
+          'Create Docker container for production deployment',
+          fullContext
+        )
+        expect(result.primaryDomain).toBe('devops')
+      })
+
+      it('should detect CI/CD tasks', () => {
+        const result = classifyWithHeuristic(
+          'Fix the CI pipeline for automated deployment',
+          fullContext
+        )
+        expect(result.primaryDomain).toBe('devops')
+      })
+    })
+
+    describe('testing detection', () => {
+      it('should detect test writing tasks', () => {
+        const result = classifyWithHeuristic('Add unit tests for the payment service', fullContext)
+        expect(result.primaryDomain).toBe('testing')
+      })
+
+      it('should detect coverage improvement tasks', () => {
+        const result = classifyWithHeuristic('Improve test coverage for auth module', fullContext)
+        expect(result.primaryDomain).toBe('testing')
+      })
+    })
+
+    // =================================================================
+    // Multi-domain Tasks
+    // =================================================================
+    describe('multi-domain tasks', () => {
+      it('should detect secondary domains', () => {
+        const result = classifyWithHeuristic(
+          'Add API endpoint with React frontend component',
+          fullContext
+        )
+        expect(result.secondaryDomains.length).toBeGreaterThan(0)
+      })
+
+      it('should limit secondary domains to 2', () => {
+        const result = classifyWithHeuristic(
+          'Add API endpoint with React component and Docker deploy with test coverage and database migration',
+          fullContext
+        )
+        expect(result.secondaryDomains.length).toBeLessThanOrEqual(2)
+      })
+    })
+
+    // =================================================================
+    // Project Context Filtering
+    // =================================================================
+    describe('project context filtering', () => {
+      it('should not classify as frontend when project has no frontend', () => {
+        const result = classifyWithHeuristic(
+          'Add a button component with responsive layout',
+          backendOnlyContext
+        )
+        // Can't be frontend since project doesn't have it
+        // Falls through to general or docs (always available)
+        expect(result.primaryDomain).not.toBe('frontend')
+      })
+
+      it('should respect available agents', () => {
+        const result = classifyWithHeuristic('Create REST API endpoint', backendOnlyContext)
+        expect(result.primaryDomain).toBe('backend')
+      })
+    })
+
+    // =================================================================
+    // Confidence Scoring
+    // =================================================================
+    describe('confidence scoring', () => {
+      it('should have higher confidence for strong signals than multi-domain', () => {
+        // Single-domain (strong frontend signal) vs multi-domain (split between frontend and backend)
+        const strong = classifyWithHeuristic(
+          'Create React component with jsx tsx ui button form modal',
+          fullContext
+        )
+        const split = classifyWithHeuristic(
+          'Add API endpoint with React component and database query',
+          fullContext
+        )
+        expect(strong.confidence).toBeGreaterThanOrEqual(split.confidence)
+      })
+
+      it('should cap confidence at 0.85 for heuristic', () => {
+        const result = classifyWithHeuristic(
+          'ui component react vue angular css style button form modal layout responsive animation',
+          fullContext
+        )
+        expect(result.confidence).toBeLessThanOrEqual(0.85)
+      })
+
+      it('should return 0.3 confidence for unknown domains', () => {
+        const result = classifyWithHeuristic(
+          'Do something completely unrelated to any domain',
+          fullContext
+        )
+        expect(result.confidence).toBe(0.3)
+        expect(result.primaryDomain).toBe('general')
+      })
+    })
+
+    // =================================================================
+    // Edge Cases
+    // =================================================================
+    describe('edge cases', () => {
+      it('should handle empty description', () => {
+        const result = classifyWithHeuristic('', fullContext)
+        expect(result.primaryDomain).toBe('general')
+        expect(result.confidence).toBe(0.3)
+      })
+
+      it('should handle very long descriptions', () => {
+        const longDesc = 'Fix the bug in the component '.repeat(100)
+        const result = classifyWithHeuristic(longDesc, fullContext)
+        expect(result.primaryDomain).toBeDefined()
+      })
+
+      it('should be case-insensitive', () => {
+        const lower = classifyWithHeuristic('add react component', fullContext)
+        const upper = classifyWithHeuristic('ADD REACT COMPONENT', fullContext)
+        expect(lower.primaryDomain).toBe(upper.primaryDomain)
+      })
+    })
+  })
+
+  // =================================================================
+  // Hash Function
+  // =================================================================
+  describe('hashDescription', () => {
+    it('should produce consistent hashes', () => {
+      const hash1 = hashDescription('Fix the auth middleware')
+      const hash2 = hashDescription('Fix the auth middleware')
+      expect(hash1).toBe(hash2)
+    })
+
+    it('should be case-insensitive', () => {
+      const hash1 = hashDescription('Fix the Auth Middleware')
+      const hash2 = hashDescription('fix the auth middleware')
+      expect(hash1).toBe(hash2)
+    })
+
+    it('should trim whitespace', () => {
+      const hash1 = hashDescription('  Fix the auth middleware  ')
+      const hash2 = hashDescription('Fix the auth middleware')
+      expect(hash1).toBe(hash2)
+    })
+
+    it('should produce different hashes for different descriptions', () => {
+      const hash1 = hashDescription('Fix frontend component')
+      const hash2 = hashDescription('Fix backend service')
+      expect(hash1).not.toBe(hash2)
+    })
+
+    it('should return a 16-character hex string', () => {
+      const hash = hashDescription('Test description')
+      expect(hash).toMatch(/^[a-f0-9]{16}$/)
+    })
+  })
+
+  // =================================================================
+  // File Patterns
+  // =================================================================
+  describe('file patterns', () => {
+    it('should return frontend file patterns for frontend domain', () => {
+      const result = classifyWithHeuristic('Add React component', fullContext)
+      expect(result.filePatterns.length).toBeGreaterThan(0)
+    })
+
+    it('should return relevant agents', () => {
+      const result = classifyWithHeuristic('Create REST API endpoint', fullContext)
+      expect(result.relevantAgents).toContain('backend')
+    })
+  })
+})