prjct-cli 1.19.0 → 1.21.0

package/CHANGELOG.md

@@ -1,11 +1,135 @@
 # Changelog
 
-## [1.19.0] - 2026-02-09
+## [1.21.0] - 2026-02-10
+
+### Features
+
+- add semantic verification for analysis results (PRJ-270) (#163)
+
+
+## [1.20.0] - 2026-02-10
 
 ### Features
 
-- implement aggressive archival of stale storage data (PRJ-267) (#161)
+- add retry with exponential backoff for agent and tool operations (#162)
+
+
+## [1.20.0] - 2026-02-09
+
+### Features
+
+- **Semantic verification for analysis results** (PRJ-270): Validate analysis consistency before sealing
+  - Framework verification: checks frameworks exist in package.json dependencies (case-insensitive matching)
+  - Language verification: validates languages match actual file extensions (.ts → TypeScript)
+  - Pattern location verification: confirms pattern files exist in project
+  - File count verification: validates count accuracy (within 10% tolerance)
+  - Anti-pattern file verification: ensures anti-pattern files exist when referenced
+  - CLI command: `prjct verify --semantic` with human-readable and JSON output
+  - Parallel execution of all 5 checks using Promise.all() (~100-200ms for typical projects)
+  - Comprehensive test suite with 10 new test cases covering all scenarios (28 tests total, 63 assertions)
+
+- **Retry with exponential backoff for agent and tool operations** (PRJ-271): Comprehensive retry infrastructure with error classification and circuit breaker
+  - RetryPolicy utility with configurable attempts, delays, and exponential backoff (1s→2s→4s)
+  - Automatic error classification: transient (EBUSY, EAGAIN, ETIMEDOUT) vs permanent (ENOENT, EPERM)
+  - Circuit breaker protection: opens after 5 consecutive failures, auto-closes after 60s
+  - Agent initialization retries (3 attempts with 1s base delay)
+  - Tool operations retry (Read/Write/Bash with 2 attempts)
+  - Resilient parallel agent generation using Promise.allSettled()
+
+### Implementation Details
+
+#### Semantic Verification (PRJ-270)
+
+Added semantic verification functions to validate analysis results match actual project state. The verification system runs 5 parallel checks to detect logical inconsistencies before sealing analysis data.
+
+**Modified modules:**
+- `core/schemas/analysis.ts` — Added 5 verification functions, 1 orchestrator, 2 helpers, and 2 interfaces (SemanticCheckResult, SemanticVerificationReport)
+- `core/storage/analysis-storage.ts` — Added semanticVerify() method to AnalysisStorage class
+- `core/commands/analysis.ts` — Extended verify() with --semantic flag and added semanticVerify() helper
+- `core/commands/commands.ts` — Updated verify() signature to include semantic?: boolean
+- `core/index.ts` — Updated verify command handler to pass semantic flag
+- `core/__tests__/storage/analysis-storage.test.ts` — Added 10 comprehensive test cases (+345 lines)
+- `README.md` — Added analysis verification documentation section
+- `CHANGELOG.md` — Documented semantic verification feature
+
+**Verification functions:**
+1. `verifyFrameworks()` — Checks frameworks exist in package.json (case-insensitive partial matching)
+2. `verifyLanguages()` — Validates languages match file extensions (.ts → TypeScript)
+3. `verifyPatternLocations()` — Confirms pattern files exist in project
+4. `verifyFileCount()` — Validates count accuracy (10% tolerance for temporary files/caches)
+5. `verifyAntiPatternFiles()` — Ensures anti-pattern files exist when referenced
+
+**Helper functions:**
+- `getProjectExtensions()` — Recursively scans project for file extensions (ignores node_modules, .git, dist, build, .next, .turbo, coverage)
+- `countProjectFiles()` — Counts all files in project with same ignore patterns
+
+**Key features:**
+- Parallel execution using Promise.all() for performance (~100-200ms typical)
+- Skip conditions for null/undefined/empty arrays (prevents false failures)
+- 10% tolerance for file count (accounts for temporary files, caches)
+- Case-insensitive partial matching for frameworks
+- Returns SemanticVerificationReport following VerificationReport pattern from sync-verifier
+- Zero breaking changes: all 18 existing tests pass
+
+#### Retry Infrastructure (PRJ-271)
+
+Built RetryPolicy utility with exponential backoff, error classification, and circuit breaker. Integrated across agent initialization, tool operations, and parallel agent generation. The system now automatically retries transient failures while failing fast on permanent errors.
+
+**New modules:**
+- `core/utils/retry.ts` (320 lines) — Core retry infrastructure with RetryPolicy class, error classification, circuit breaker
+- `core/__tests__/utils/retry.test.ts` (380 lines) — 21 comprehensive tests with 53 assertions
+- `ACCEPTANCE-PRJ-271.md` — Full acceptance criteria verification (22 criteria verified)
+
+**Modified modules:**
+- `core/services/agent-service.ts` — Wrapped initialize() with retry policy (3 attempts)
+- `core/agentic/tool-registry.ts` — Added retry to Read/Write/Bash tools (2 attempts each)
+- `core/services/agent-generator.ts` — Changed to Promise.allSettled() with per-agent retry
+
+**Key features:**
+- Exponential backoff: 1s, 2s, 4s (configurable base/max)
+- Error classification: automatic transient vs permanent detection
+- Circuit breaker: per-operation tracking, 5 failure threshold, 60s cooldown
+- Two default policies: defaultAgentRetryPolicy (3 attempts), defaultToolRetryPolicy (2 attempts)
+- Zero breaking changes: all 968 existing tests pass
+
+### Learnings
+
+- **RetryPolicy pattern:** Wrapping operations with retry execution provides clean separation of retry logic from business logic
+- **Error classification strategies:** Using error code sets (EBUSY, EAGAIN) for transient vs (ENOENT, EPERM) for permanent enables automatic decision-making
+- **Promise.allSettled() for resilient parallel operations:** Prevents one failure from blocking other operations, enables partial success
+- **Circuit breaker implementation:** Per-operation state tracking prevents cascading failures while allowing recovery
+
+### Test Plan
+
+#### For QA
+
+1. **Agent Initialization Retry**
+   - Temporarily make file system busy during agent initialization
+   - Verify agent initialization retries up to 3 times
+   - Confirm permanent errors (unsupported agent) fail immediately
+
+2. **Tool Operations Retry**
+   - Test Read/Write/Bash with transient errors (EBUSY, ETIMEDOUT)
+   - Verify operations retry automatically (2 attempts)
+   - Confirm permanent errors (ENOENT, EPERM) return null/false without retry
+
+3. **Circuit Breaker**
+   - Trigger 5 consecutive failures on same operation
+   - Verify circuit breaker opens and blocks further attempts
+   - Wait 60 seconds and verify circuit closes automatically
+
+4. **Parallel Agent Generation**
+   - Simulate one agent generation failure during sync
+   - Verify other agents generate successfully (Promise.allSettled behavior)
+   - Check logs for failure warnings
+
+#### For Users
+
+**What changed:** The system is now more resilient against transient failures. Operations like agent initialization, file reads/writes, and command execution will automatically retry when they encounter temporary errors (disk busy, timeouts, etc).
+
+**How to use:** No action required - retry logic works automatically. Users will experience fewer random failures during normal operations.
 
+**Breaking changes:** None. All changes are backward compatible. Existing tests (968 total) all pass.
 
 ## [1.19.0] - 2026-02-09
 

package/README.md

@@ -128,10 +128,59 @@ All agents share the same project storage, so you can switch between them freely
 ```bash
 prjct start          # First-time setup (Claude/Gemini)
 prjct init           # Initialize project (+ Cursor setup)
+prjct sync           # Analyze project and generate context
+prjct verify         # Verify analysis integrity (cryptographic)
+prjct verify --semantic  # Verify analysis consistency (semantic)
 prjct --version      # Show version + provider status
 prjct --help         # Show help
 ```
 
+### Analysis Verification
+
+prjct provides two types of analysis verification to ensure data integrity and logical consistency:
+
+#### Cryptographic Verification (Default)
+```bash
+prjct verify [--json]
+```
+
+Verifies the integrity of sealed analysis results using cryptographic signatures. This ensures:
+- Analysis data hasn't been tampered with
+- Sealed analysis matches the original analysis
+- Hash signatures are valid
+
+**When to use:** After sealing an analysis (`prjct seal`) to confirm data integrity.
+
+#### Semantic Verification (PRJ-270)
+```bash
+prjct verify --semantic [--json]
+```
+
+Validates that analysis results match the actual project state. This checks:
+- ✓ **Frameworks** exist in `package.json` dependencies
+- ✓ **Languages** match actual file extensions (.ts → TypeScript)
+- ✓ **Pattern locations** reference real files in the project
+- ✓ **File count** is accurate (within 10% tolerance)
+- ✓ **Anti-pattern files** exist when referenced
+
+**When to use:** Before sealing an analysis to catch logical inconsistencies or after project changes to validate analysis accuracy.
+
+**Example output:**
+```
+Semantic Verification Report
+────────────────────────────
+  ✓ Framework verification: passed (2 frameworks validated)
+  ✓ Language verification: passed (1 language validated)
+  ✓ Pattern locations: passed (12 patterns verified)
+  ✓ File count verification: passed (324 files, within tolerance)
+  ✓ Anti-pattern files: passed (3 anti-patterns verified)
+
+Result: PASSED (5/5 checks)
+Total time: 145ms
+```
+
+Both verification modes support `--json` flag for programmatic use.
+
 ## Environment Variables
 
 ### Configuration

package/core/__tests__/storage/analysis-storage.test.ts

@@ -14,6 +14,7 @@ import {
   AnalysisItemSchema,
   AnalysisStatusSchema,
   parseAnalysis,
+  type SemanticCheckResult,
   safeParseAnalysis,
 } from '../../schemas/analysis'
 
@@ -275,3 +276,366 @@ describe('backward compatibility', () => {
     expect(result.modelMetadata?.provider).toBe('claude')
   })
 })
+
+// =============================================================================
+// Semantic Verification (PRJ-270)
+// =============================================================================
+
+describe('semantic verification', () => {
+  const { semanticVerify } = require('../../schemas/analysis')
+  const fs = require('node:fs/promises')
+  const path = require('node:path')
+  const os = require('node:os')
+
+  // Helper to create a temporary test project
+  async function createTestProject(options: {
+    hasPackageJson?: boolean
+    packageJsonDeps?: Record<string, string>
+    files?: { path: string; content: string }[]
+    fileCount?: number
+  }) {
+    const tmpDir = await fs.mkdtemp(path.join(os.tmpdir(), 'prjct-test-'))
+
+    // Create package.json if requested
+    if (options.hasPackageJson) {
+      const pkg = {
+        name: 'test-project',
+        dependencies: options.packageJsonDeps || {},
+      }
+      await fs.writeFile(path.join(tmpDir, 'package.json'), JSON.stringify(pkg, null, 2))
+    }
+
+    // Create additional files
+    if (options.files) {
+      for (const file of options.files) {
+        const filePath = path.join(tmpDir, file.path)
+        await fs.mkdir(path.dirname(filePath), { recursive: true })
+        await fs.writeFile(filePath, file.content)
+      }
+    }
+
+    return tmpDir
+  }
+
+  // Helper to cleanup test project
+  async function cleanupTestProject(tmpDir: string) {
+    try {
+      await fs.rm(tmpDir, { recursive: true, force: true })
+    } catch {
+      // Ignore cleanup errors
+    }
+  }
+
+  it('should pass all checks for valid analysis', async () => {
+    const projectPath = await createTestProject({
+      hasPackageJson: true,
+      packageJsonDeps: { hono: '^3.0.0', zod: '^3.0.0' },
+      files: [
+        { path: 'src/index.ts', content: 'export const app = {}' },
+        { path: 'src/server.ts', content: 'import { serve } from "bun"' },
+        { path: 'patterns/service.ts', content: 'export class UserService {}' },
+      ],
+    })
+
+    const analysis = {
+      projectId: 'test',
+      languages: ['TypeScript'],
+      frameworks: ['Hono'],
+      configFiles: [],
+      fileCount: 4, // package.json + 3 TypeScript files
+      patterns: [
+        { name: 'Service pattern', description: 'DI pattern', location: 'patterns/service.ts' },
+      ],
+      antiPatterns: [],
+      analyzedAt: '2026-02-10T00:00:00.000Z',
+      status: 'draft' as const,
+    }
+
+    const result = await semanticVerify(analysis, projectPath)
+
+    expect(result.passed).toBe(true)
+    expect(result.failedCount).toBe(0)
+    expect(result.passedCount).toBeGreaterThan(0)
+    expect(result.checks.every((c: SemanticCheckResult) => c.passed)).toBe(true)
+
+    await cleanupTestProject(projectPath)
+  })
+
+  it('should fail when frameworks are not in package.json', async () => {
+    const projectPath = await createTestProject({
+      hasPackageJson: true,
+      packageJsonDeps: { express: '^4.0.0' }, // Different framework
+      files: [{ path: 'src/index.ts', content: '' }],
+    })
+
+    const analysis = {
+      projectId: 'test',
+      languages: ['TypeScript'],
+      frameworks: ['Hono', 'Zod'], // These are not in dependencies
+      configFiles: [],
+      fileCount: 1,
+      patterns: [],
+      antiPatterns: [],
+      analyzedAt: '2026-02-10T00:00:00.000Z',
+      status: 'draft' as const,
+    }
+
+    const result = await semanticVerify(analysis, projectPath)
+
+    expect(result.passed).toBe(false)
+    expect(result.failedCount).toBeGreaterThan(0)
+
+    const frameworkCheck = result.checks.find(
+      (c: SemanticCheckResult) => c.name === 'Framework verification'
+    )
+    expect(frameworkCheck?.passed).toBe(false)
+    expect(frameworkCheck?.error).toContain('not found in dependencies')
+
+    await cleanupTestProject(projectPath)
+  })
+
+  it('should fail when package.json is missing', async () => {
+    const projectPath = await createTestProject({
+      hasPackageJson: false, // No package.json
+      files: [{ path: 'src/index.ts', content: '' }],
+    })
+
+    const analysis = {
+      projectId: 'test',
+      languages: ['TypeScript'],
+      frameworks: ['Hono'],
+      configFiles: [],
+      fileCount: 1,
+      patterns: [],
+      antiPatterns: [],
+      analyzedAt: '2026-02-10T00:00:00.000Z',
+      status: 'draft' as const,
+    }
+
+    const result = await semanticVerify(analysis, projectPath)
+
+    const frameworkCheck = result.checks.find(
+      (c: SemanticCheckResult) => c.name === 'Framework verification'
+    )
+    expect(frameworkCheck?.passed).toBe(false)
+    expect(frameworkCheck?.error).toContain('package.json not found')
+
+    await cleanupTestProject(projectPath)
+  })
+
+  it('should fail when declared languages have no matching files', async () => {
+    const projectPath = await createTestProject({
+      hasPackageJson: true,
+      files: [
+        { path: 'src/index.js', content: '' }, // Only JS files
+      ],
+    })
+
+    const analysis = {
+      projectId: 'test',
+      languages: ['TypeScript', 'Go'], // Declared but no .ts or .go files
+      frameworks: [],
+      configFiles: [],
+      fileCount: 1,
+      patterns: [],
+      antiPatterns: [],
+      analyzedAt: '2026-02-10T00:00:00.000Z',
+      status: 'draft' as const,
+    }
+
+    const result = await semanticVerify(analysis, projectPath)
+
+    const languageCheck = result.checks.find(
+      (c: SemanticCheckResult) => c.name === 'Language verification'
+    )
+    expect(languageCheck?.passed).toBe(false)
+    expect(languageCheck?.error).toContain('without matching files')
+
+    await cleanupTestProject(projectPath)
+  })
+
+  it('should fail when pattern locations reference missing files', async () => {
+    const projectPath = await createTestProject({
+      hasPackageJson: true,
+      files: [{ path: 'src/index.ts', content: '' }],
+    })
+
+    const analysis = {
+      projectId: 'test',
+      languages: ['TypeScript'],
+      frameworks: [],
+      configFiles: [],
+      fileCount: 1,
+      patterns: [
+        { name: 'Service pattern', description: 'DI', location: 'src/service.ts' }, // Doesn't exist
+        { name: 'Repository', description: 'Data access', location: 'src/repo.ts' }, // Doesn't exist
+      ],
+      antiPatterns: [],
+      analyzedAt: '2026-02-10T00:00:00.000Z',
+      status: 'draft' as const,
+    }
+
+    const result = await semanticVerify(analysis, projectPath)
+
+    const patternCheck = result.checks.find(
+      (c: SemanticCheckResult) => c.name === 'Pattern location verification'
+    )
+    expect(patternCheck?.passed).toBe(false)
+    expect(patternCheck?.error).toContain('not found')
+
+    await cleanupTestProject(projectPath)
+  })
+
+  it('should fail when file count is inaccurate', async () => {
+    const projectPath = await createTestProject({
+      hasPackageJson: true,
+      files: [
+        { path: 'src/a.ts', content: '' },
+        { path: 'src/b.ts', content: '' },
+        { path: 'src/c.ts', content: '' },
+      ], // 4 files total (package.json + 3 .ts files)
+    })
+
+    const analysis = {
+      projectId: 'test',
+      languages: ['TypeScript'],
+      frameworks: [],
+      configFiles: [],
+      fileCount: 100, // Way off (actual is ~4)
+      patterns: [],
+      antiPatterns: [],
+      analyzedAt: '2026-02-10T00:00:00.000Z',
+      status: 'draft' as const,
+    }
+
+    const result = await semanticVerify(analysis, projectPath)
+
+    const fileCountCheck = result.checks.find(
+      (c: SemanticCheckResult) => c.name === 'File count verification'
+    )
+    expect(fileCountCheck?.passed).toBe(false)
+    expect(fileCountCheck?.error).toContain('mismatch')
+
+    await cleanupTestProject(projectPath)
+  })
+
+  it('should fail when anti-pattern files are missing', async () => {
+    const projectPath = await createTestProject({
+      hasPackageJson: true,
+      files: [{ path: 'src/index.ts', content: '' }],
+    })
+
+    const analysis = {
+      projectId: 'test',
+      languages: ['TypeScript'],
+      frameworks: [],
+      configFiles: [],
+      fileCount: 1,
+      patterns: [],
+      antiPatterns: [
+        { issue: 'Missing types', file: 'src/bad.ts', suggestion: 'Add types' }, // File doesn't exist
+      ],
+      analyzedAt: '2026-02-10T00:00:00.000Z',
+      status: 'draft' as const,
+    }
+
+    const result = await semanticVerify(analysis, projectPath)
+
+    const antiPatternCheck = result.checks.find(
+      (c: SemanticCheckResult) => c.name === 'Anti-pattern file verification'
+    )
+    expect(antiPatternCheck?.passed).toBe(false)
+    expect(antiPatternCheck?.error).toContain('not found')
+
+    await cleanupTestProject(projectPath)
+  })
+
+  it('should skip checks when no data to verify', async () => {
+    const projectPath = await createTestProject({
+      hasPackageJson: true,
+    })
+
+    const analysis = {
+      projectId: 'test',
+      languages: [], // No languages declared
+      frameworks: [], // No frameworks declared
+      configFiles: [],
+      fileCount: 1,
+      patterns: [], // No patterns with locations
+      antiPatterns: [], // No anti-patterns
+      analyzedAt: '2026-02-10T00:00:00.000Z',
+      status: 'draft' as const,
+    }
+
+    const result = await semanticVerify(analysis, projectPath)
+
+    expect(result.passed).toBe(true) // All checks skipped, so passes
+    expect(
+      result.checks.every((c: SemanticCheckResult) => c.output?.includes('skipped') || c.passed)
+    ).toBe(true)
+
+    await cleanupTestProject(projectPath)
+  })
+
+  it('should accept file count within tolerance (10%)', async () => {
+    const projectPath = await createTestProject({
+      hasPackageJson: true,
+      files: [
+        { path: 'src/a.ts', content: '' },
+        { path: 'src/b.ts', content: '' },
+        { path: 'src/c.ts', content: '' },
+        { path: 'src/d.ts', content: '' },
+        { path: 'src/e.ts', content: '' },
+      ], // 6 files total
+    })
+
+    const analysis = {
+      projectId: 'test',
+      languages: ['TypeScript'],
+      frameworks: [],
+      configFiles: [],
+      fileCount: 6, // Within 10% tolerance
+      patterns: [],
+      antiPatterns: [],
+      analyzedAt: '2026-02-10T00:00:00.000Z',
+      status: 'draft' as const,
+    }
+
+    const result = await semanticVerify(analysis, projectPath)
+
+    const fileCountCheck = result.checks.find(
+      (c: SemanticCheckResult) => c.name === 'File count verification'
+    )
+    expect(fileCountCheck?.passed).toBe(true)
+
+    await cleanupTestProject(projectPath)
+  })
+
+  it('should handle partial framework matches (case insensitive)', async () => {
+    const projectPath = await createTestProject({
+      hasPackageJson: true,
+      packageJsonDeps: { '@hono/node-server': '^1.0.0' }, // Hono as part of package name
+    })
+
+    const analysis = {
+      projectId: 'test',
+      languages: [],
+      frameworks: ['Hono'], // Should match @hono/node-server
+      configFiles: [],
+      fileCount: 1,
+      patterns: [],
+      antiPatterns: [],
+      analyzedAt: '2026-02-10T00:00:00.000Z',
+      status: 'draft' as const,
+    }
+
+    const result = await semanticVerify(analysis, projectPath)
+
+    const frameworkCheck = result.checks.find(
+      (c: SemanticCheckResult) => c.name === 'Framework verification'
+    )
+    expect(frameworkCheck?.passed).toBe(true)
+
+    await cleanupTestProject(projectPath)
+  })
+})