prjct-cli 1.16.0 → 1.18.0

package/CHANGELOG.md

@@ -1,12 +1,103 @@
 # Changelog
 
+## [1.18.0] - 2026-02-09
+
+### Features
+
+- implement incremental sync with file hashing (PRJ-305) (#160)
+
+
+## [1.18.0] - 2026-02-09
+
+### Features
+
+- **Incremental sync**: `prjct sync` now only re-analyzes files that changed since last sync (PRJ-305)
+  - File hashing with Bun.hash (xxHash64) — <100ms for 500 files
+  - Change propagation through import graph (1-level reverse edges)
+  - Conditional index rebuilds: BM25, import graph, co-change only when source files change
+  - Conditional agent regeneration: only when config files (package.json, tsconfig.json) change
+  - `prjct sync --full` flag to force complete re-analysis
+
+### Implementation Details
+
+New modules:
+- `core/domain/file-hasher.ts` — Hash computation via Bun.hash, SQLite registry using `index_checksums` table, diff detection (added/modified/deleted/unchanged)
+- `core/domain/change-propagator.ts` — Import graph reverse-edge lookup for 1-level change propagation, domain classification for affected files
+
+Modified:
+- `core/services/sync-service.ts` — Incremental decision logic: detect changes → propagate → conditionally rebuild indexes and agents
+- `core/services/watch-service.ts` — Passes accumulated `changedFiles` to sync options
+- `core/types/project-sync.ts` — Added `full`, `changedFiles` to `SyncOptions` + `IncrementalInfo` result type
+- CLI chain (`core/index.ts` → `commands.ts` → `analysis.ts`) — Wired `--full` flag through
+
+### Learnings
+
+- Bun's `fs.readdir` with `withFileTypes` returns `Dirent<NonSharedBuffer>` — need `String()` cast for `.name`
+- Existing `index_checksums` SQLite table was already set up (PRJ-303) — zero schema changes needed
+- Import graph reverse edges (from PRJ-304) enable efficient 1-level propagation without rebuilding the graph
+
+### Test Plan
+
+#### For QA
+1. Run `prjct sync` on fresh project (no hash cache) — should behave as full sync
+2. Run `prjct sync` again without changes — should skip index rebuilds and agent regeneration
+3. Modify a `.ts` file, run `prjct sync` — should detect change and rebuild indexes
+4. Modify `package.json`, run `prjct sync` — should regenerate agents
+5. Run `prjct sync --full` — should force complete re-analysis
+6. Run `prjct watch`, change a file — should pass changedFiles to sync
+
+#### For Users
+**What changed:** `prjct sync` is now incremental by default.
+**How to use:** No changes needed. Use `prjct sync --full` to force complete re-analysis.
+**Breaking changes:** None
+
+## [1.17.0] - 2026-02-09
+
+### Features
+
+- implement BM25 + import graph + git co-change for zero-cost file selection (PRJ-304) (#159)
+
+
+## [1.17.0] - 2026-02-08
+
+### Features
+- **BM25 + import graph + git co-change file selection** (PRJ-304): Zero-cost file selection using three mathematical signals combined into a weighted ranker. Replaces keyword matching in smart-context with precision that matches LLM-based classification — at zero API cost.
+
+### Implementation Details
+- `core/domain/bm25.ts` — BM25 indexer: tokenizes files (exports, functions, imports, comments, path segments), builds inverted index, scores queries using Okapi BM25 (k1=1.2, b=0.75). Stores in SQLite kv_store.
+- `core/domain/import-graph.ts` — Import graph builder: parses TS/JS imports to build directed adjacency list, follows chains 2 levels deep, scores by proximity (1/(depth+1)).
+- `core/domain/git-cochange.ts` — Git co-change analyzer: parses last 100 commits, builds Jaccard similarity matrix for file pairs that change together.
+- `core/domain/file-ranker.ts` — Combined ranker: `BM25 × 0.5 + imports × 0.3 + cochange × 0.2`, normalizes each signal to [0,1], returns top 15 files.
+- `core/agentic/smart-context.ts` — Uses ranker when indexes exist, graceful fallback to regex-based domain filtering.
+- `core/services/sync-service.ts` — Builds all 3 indexes in parallel during `prjct sync`.
+
+### Learnings
+- `tokenizeQuery` must split camelCase BEFORE lowercasing — otherwise "getUserById" becomes "getuserbyid" and doesn't split
+- Jaccard similarity > cosine for co-change because data is binary (file present or not in commit)
+- Batch file reads (50 at a time) needed for indexing performance on large projects
+- Stop words list must include code keywords (import, export, const) to reduce noise in scoring
+
+### Test Plan
+
+#### For QA
+1. Run `prjct sync` — verify BM25, import graph, and co-change indexes build without errors
+2. Query "Fix auth middleware" — verify auth-related files rank higher than unrelated files
+3. Query "Build responsive dashboard" — verify frontend files rank higher than backend files
+4. Verify index rebuild time <5 seconds on 300+ file project
+5. Verify query time <50ms
+6. Verify zero API calls during file selection
+
+#### For Users
+**What changed:** File selection during task context is now powered by BM25 text search, import graph proximity, and git co-change analysis instead of keyword matching.
+**How to use:** Run `p. sync` to build indexes — file selection is automatic and more accurate.
+**Breaking changes:** None. Falls back to previous filtering if indexes don't exist.
+
 ## [1.16.0] - 2026-02-09
 
 ### Features
 
 - remove JSON storage redundancy, SQLite-only backend (PRJ-303) (#158)
 
-
 ## [1.16.0] - 2026-02-08
 
 ### Features

package/core/__tests__/domain/bm25.test.ts

@@ -0,0 +1,225 @@
+/**
+ * Tests for BM25 Text Search Index
+ */
+
+import { afterEach, beforeEach, describe, expect, it } from 'bun:test'
+import fs from 'node:fs/promises'
+import os from 'node:os'
+import path from 'node:path'
+import { buildIndex, score, tokenizeFile, tokenizeQuery } from '../../domain/bm25'
+
+// =============================================================================
+// Tokenization Tests
+// =============================================================================
+
+describe('BM25', () => {
+  describe('tokenizeFile', () => {
+    it('should extract path segments', () => {
+      const tokens = tokenizeFile('', 'core/domain/bm25.ts')
+      expect(tokens).toContain('core')
+      expect(tokens).toContain('domain')
+      expect(tokens).toContain('bm25')
+    })
+
+    it('should extract function names and split camelCase', () => {
+      const content = 'export function getUserById(id: string) { return id }'
+      const tokens = tokenizeFile(content, 'user-service.ts')
+      expect(tokens).toContain('get')
+      expect(tokens).toContain('user')
+      // 'by' and 'id' are filtered (stop word / too short)
+    })
+
+    it('should extract class names', () => {
+      const content = 'export class AuthMiddleware { handle() {} }'
+      const tokens = tokenizeFile(content, 'middleware.ts')
+      expect(tokens).toContain('auth')
+      expect(tokens).toContain('middleware')
+    })
+
+    it('should extract interface names', () => {
+      const content = 'export interface JwtPayload { sub: string }'
+      const tokens = tokenizeFile(content, 'types.ts')
+      expect(tokens).toContain('jwt')
+      expect(tokens).toContain('payload')
+    })
+
+    it('should extract import sources', () => {
+      const content = `import { Router } from './router'\nimport express from 'express'`
+      const tokens = tokenizeFile(content, 'app.ts')
+      expect(tokens).toContain('router')
+      expect(tokens).toContain('express')
+    })
+
+    it('should extract words from comments', () => {
+      const content = '// Handle authentication for JWT tokens'
+      const tokens = tokenizeFile(content, 'auth.ts')
+      expect(tokens).toContain('handle')
+      expect(tokens).toContain('authentication')
+      expect(tokens).toContain('jwt')
+      expect(tokens).toContain('tokens')
+    })
+
+    it('should extract words from JSDoc comments', () => {
+      const content = '/** Validates user session and refreshes token */'
+      const tokens = tokenizeFile(content, 'session.ts')
+      expect(tokens).toContain('validates')
+      expect(tokens).toContain('session')
+      expect(tokens).toContain('refreshes')
+      expect(tokens).toContain('token')
+    })
+
+    it('should filter out stop words', () => {
+      const content = 'export function getTheData() {}'
+      const tokens = tokenizeFile(content, 'data.ts')
+      expect(tokens).not.toContain('the')
+      expect(tokens).not.toContain('export')
+      expect(tokens).not.toContain('function')
+    })
+
+    it('should handle empty content', () => {
+      const tokens = tokenizeFile('', 'empty.ts')
+      // Should still have path segments
+      expect(tokens).toContain('empty')
+    })
+  })
+
+  describe('tokenizeQuery', () => {
+    it('should tokenize a task description', () => {
+      const tokens = tokenizeQuery('Fix the auth middleware for JWT validation')
+      expect(tokens).toContain('fix')
+      expect(tokens).toContain('auth')
+      expect(tokens).toContain('middleware')
+      expect(tokens).toContain('jwt')
+      expect(tokens).toContain('validation')
+    })
+
+    it('should split camelCase in queries', () => {
+      const tokens = tokenizeQuery('update getUserById function')
+      expect(tokens).toContain('update')
+      expect(tokens).toContain('get')
+      expect(tokens).toContain('user')
+    })
+
+    it('should remove stop words from queries', () => {
+      const tokens = tokenizeQuery('Fix the bug in the login')
+      expect(tokens).not.toContain('the')
+      expect(tokens).not.toContain('in')
+      expect(tokens).toContain('fix')
+      expect(tokens).toContain('bug')
+      expect(tokens).toContain('login')
+    })
+  })
+
+  // =============================================================================
+  // Index Building & Scoring Tests
+  // =============================================================================
+
+  describe('buildIndex + score', () => {
+    let testDir: string
+
+    beforeEach(async () => {
+      testDir = path.join(os.tmpdir(), `prjct-bm25-test-${Date.now()}`)
+      await fs.mkdir(testDir, { recursive: true })
+    })
+
+    afterEach(async () => {
+      try {
+        await fs.rm(testDir, { recursive: true, force: true })
+      } catch {
+        // Ignore cleanup errors
+      }
+    })
+
+    it('should build an index from project files', async () => {
+      // Create test files
+      await fs.writeFile(
+        path.join(testDir, 'auth.ts'),
+        `export class AuthService {\n  validateJwt(token: string) {}\n  refreshSession() {}\n}`
+      )
+      await fs.writeFile(
+        path.join(testDir, 'middleware.ts'),
+        `import { AuthService } from './auth'\nexport function authMiddleware(req: any) {}`
+      )
+      await fs.writeFile(
+        path.join(testDir, 'button.tsx'),
+        `export function Button({ label }: { label: string }) {\n  return <button>{label}</button>\n}`
+      )
+
+      const index = await buildIndex(testDir)
+
+      expect(index.totalDocs).toBe(3)
+      expect(index.avgDocLength).toBeGreaterThan(0)
+      expect(Object.keys(index.documents)).toContain('auth.ts')
+      expect(Object.keys(index.documents)).toContain('middleware.ts')
+      expect(Object.keys(index.documents)).toContain('button.tsx')
+    })
+
+    it('should rank auth files higher for auth query', async () => {
+      await fs.writeFile(
+        path.join(testDir, 'auth.ts'),
+        `export class AuthService {\n  // Authenticate user with JWT\n  validateJwt(token: string) {}\n  refreshSession() {}\n}`
+      )
+      await fs.writeFile(
+        path.join(testDir, 'middleware.ts'),
+        `import { AuthService } from './auth'\n// Auth middleware for JWT validation\nexport function authMiddleware(req: any) {}`
+      )
+      await fs.writeFile(
+        path.join(testDir, 'button.tsx'),
+        `// Render a UI button component\nexport function Button({ label }: { label: string }) {\n  return <button>{label}</button>\n}`
+      )
+
+      const index = await buildIndex(testDir)
+      const results = score('Fix the auth middleware for JWT validation', index)
+
+      // Auth and middleware should rank higher than button
+      expect(results.length).toBeGreaterThan(0)
+      const authIndex = results.findIndex((r) => r.path === 'auth.ts')
+      const middlewareIndex = results.findIndex((r) => r.path === 'middleware.ts')
+      const buttonIndex = results.findIndex((r) => r.path === 'button.tsx')
+
+      expect(authIndex).toBeLessThan(buttonIndex === -1 ? Infinity : buttonIndex)
+      expect(middlewareIndex).toBeLessThan(buttonIndex === -1 ? Infinity : buttonIndex)
+    })
+
+    it('should rank frontend files higher for UI query', async () => {
+      await fs.writeFile(
+        path.join(testDir, 'dashboard.tsx'),
+        `// Responsive dashboard with charts and data grid\nexport function Dashboard() {\n  return <div className="dashboard">Charts here</div>\n}`
+      )
+      await fs.writeFile(
+        path.join(testDir, 'api-handler.ts'),
+        `// Handle API requests for user data\nexport function handleRequest(req: any) { return {} }`
+      )
+
+      const index = await buildIndex(testDir)
+      const results = score('Build responsive dashboard', index)
+
+      expect(results.length).toBeGreaterThan(0)
+      expect(results[0].path).toBe('dashboard.tsx')
+    })
+
+    it('should skip node_modules', async () => {
+      await fs.mkdir(path.join(testDir, 'node_modules', 'pkg'), { recursive: true })
+      await fs.writeFile(path.join(testDir, 'node_modules', 'pkg', 'index.ts'), 'export default {}')
+      await fs.writeFile(path.join(testDir, 'app.ts'), 'export function main() {}')
+
+      const index = await buildIndex(testDir)
+      expect(index.totalDocs).toBe(1)
+      expect(Object.keys(index.documents)).toContain('app.ts')
+    })
+
+    it('should handle empty query', async () => {
+      await fs.writeFile(path.join(testDir, 'app.ts'), 'export function main() {}')
+
+      const index = await buildIndex(testDir)
+      const results = score('', index)
+      expect(results).toEqual([])
+    })
+
+    it('should handle empty project', async () => {
+      const index = await buildIndex(testDir)
+      expect(index.totalDocs).toBe(0)
+      expect(score('anything', index)).toEqual([])
+    })
+  })
+})

package/core/__tests__/domain/change-propagator.test.ts

@@ -0,0 +1,100 @@
+import { describe, expect, test } from 'bun:test'
+import { affectedDomains, propagateChanges } from '../../domain/change-propagator'
+import type { FileDiff } from '../../domain/file-hasher'
+
+describe('change-propagator', () => {
+  // =========================================================================
+  // propagateChanges
+  // =========================================================================
+
+  describe('propagateChanges', () => {
+    test('returns direct changes when no import graph exists', () => {
+      const diff: FileDiff = {
+        added: ['src/new.ts'],
+        modified: ['src/changed.ts'],
+        deleted: ['src/removed.ts'],
+        unchanged: ['src/same.ts'],
+      }
+
+      // Use a fake projectId that won't have a graph
+      const result = propagateChanges(diff, 'nonexistent-project')
+
+      expect(result.directlyChanged).toEqual(['src/new.ts', 'src/changed.ts'])
+      expect(result.affectedByImports).toEqual([])
+      expect(result.allAffected).toEqual(['src/new.ts', 'src/changed.ts'])
+      expect(result.deleted).toEqual(['src/removed.ts'])
+    })
+
+    test('empty diff returns empty propagation', () => {
+      const diff: FileDiff = {
+        added: [],
+        modified: [],
+        deleted: [],
+        unchanged: ['src/a.ts', 'src/b.ts'],
+      }
+
+      const result = propagateChanges(diff, 'nonexistent-project')
+
+      expect(result.directlyChanged).toEqual([])
+      expect(result.affectedByImports).toEqual([])
+      expect(result.allAffected).toEqual([])
+    })
+  })
+
+  // =========================================================================
+  // affectedDomains
+  // =========================================================================
+
+  describe('affectedDomains', () => {
+    test('detects frontend files', () => {
+      const domains = affectedDomains([
+        'src/components/Button.tsx',
+        'src/pages/Home.jsx',
+        'styles/main.css',
+      ])
+      expect(domains.has('frontend')).toBe(true)
+      expect(domains.has('uxui')).toBe(true)
+    })
+
+    test('detects backend files', () => {
+      const domains = affectedDomains(['core/services/auth.ts', 'core/domain/user.ts'])
+      expect(domains.has('backend')).toBe(true)
+    })
+
+    test('detects testing files', () => {
+      const domains = affectedDomains([
+        'core/__tests__/auth.test.ts',
+        'src/components/Button.spec.tsx',
+      ])
+      expect(domains.has('testing')).toBe(true)
+    })
+
+    test('detects devops files', () => {
+      const domains = affectedDomains(['Dockerfile', '.github/workflows/ci.yml'])
+      expect(domains.has('devops')).toBe(true)
+    })
+
+    test('detects database files', () => {
+      const domains = affectedDomains(['prisma/schema.prisma', 'db/migrations/001.sql'])
+      expect(domains.has('database')).toBe(true)
+    })
+
+    test('handles mixed domain files', () => {
+      const domains = affectedDomains([
+        'src/components/Form.tsx', // frontend + uxui
+        'core/services/api.ts', // backend
+        'Dockerfile', // devops
+        'core/__tests__/api.test.ts', // testing + backend
+      ])
+      expect(domains.has('frontend')).toBe(true)
+      expect(domains.has('backend')).toBe(true)
+      expect(domains.has('devops')).toBe(true)
+      expect(domains.has('testing')).toBe(true)
+    })
+
+    test('empty file list returns empty domains', () => {
+      const domains = affectedDomains([])
+      expect(domains.size).toBe(0)
+    })
+  })
+})

package/core/__tests__/domain/file-hasher.test.ts

@@ -0,0 +1,146 @@
+import { describe, expect, test } from 'bun:test'
+import path from 'node:path'
+import { computeHashes, diffHashes, type FileHash } from '../../domain/file-hasher'
+
+describe('file-hasher', () => {
+  // =========================================================================
+  // diffHashes
+  // =========================================================================
+
+  describe('diffHashes', () => {
+    const makeHash = (filePath: string, hash: string): FileHash => ({
+      path: filePath,
+      hash,
+      size: 100,
+      mtime: '2026-01-01T00:00:00.000Z',
+    })
+
+    test('detects added files', () => {
+      const current = new Map<string, FileHash>([
+        ['src/new-file.ts', makeHash('src/new-file.ts', 'xxh64:abc')],
+        ['src/existing.ts', makeHash('src/existing.ts', 'xxh64:def')],
+      ])
+      const stored = new Map<string, FileHash>([
+        ['src/existing.ts', makeHash('src/existing.ts', 'xxh64:def')],
+      ])
+
+      const diff = diffHashes(current, stored)
+      expect(diff.added).toEqual(['src/new-file.ts'])
+      expect(diff.modified).toEqual([])
+      expect(diff.unchanged).toEqual(['src/existing.ts'])
+      expect(diff.deleted).toEqual([])
+    })
+
+    test('detects modified files', () => {
+      const current = new Map<string, FileHash>([
+        ['src/changed.ts', makeHash('src/changed.ts', 'xxh64:new-hash')],
+      ])
+      const stored = new Map<string, FileHash>([
+        ['src/changed.ts', makeHash('src/changed.ts', 'xxh64:old-hash')],
+      ])
+
+      const diff = diffHashes(current, stored)
+      expect(diff.added).toEqual([])
+      expect(diff.modified).toEqual(['src/changed.ts'])
+      expect(diff.unchanged).toEqual([])
+      expect(diff.deleted).toEqual([])
+    })
+
+    test('detects deleted files', () => {
+      const current = new Map<string, FileHash>()
+      const stored = new Map<string, FileHash>([
+        ['src/removed.ts', makeHash('src/removed.ts', 'xxh64:abc')],
+      ])
+
+      const diff = diffHashes(current, stored)
+      expect(diff.added).toEqual([])
+      expect(diff.modified).toEqual([])
+      expect(diff.unchanged).toEqual([])
+      expect(diff.deleted).toEqual(['src/removed.ts'])
+    })
+
+    test('handles empty maps', () => {
+      const diff = diffHashes(new Map(), new Map())
+      expect(diff.added).toEqual([])
+      expect(diff.modified).toEqual([])
+      expect(diff.unchanged).toEqual([])
+      expect(diff.deleted).toEqual([])
+    })
+
+    test('handles first sync (no stored hashes)', () => {
+      const current = new Map<string, FileHash>([
+        ['src/a.ts', makeHash('src/a.ts', 'xxh64:1')],
+        ['src/b.ts', makeHash('src/b.ts', 'xxh64:2')],
+        ['src/c.ts', makeHash('src/c.ts', 'xxh64:3')],
+      ])
+      const stored = new Map<string, FileHash>()
+
+      const diff = diffHashes(current, stored)
+      expect(diff.added).toHaveLength(3)
+      expect(diff.modified).toEqual([])
+      expect(diff.unchanged).toEqual([])
+      expect(diff.deleted).toEqual([])
+    })
+
+    test('mixed changes: added + modified + deleted + unchanged', () => {
+      const current = new Map<string, FileHash>([
+        ['src/new.ts', makeHash('src/new.ts', 'xxh64:new')],
+        ['src/changed.ts', makeHash('src/changed.ts', 'xxh64:v2')],
+        ['src/same.ts', makeHash('src/same.ts', 'xxh64:same')],
+      ])
+      const stored = new Map<string, FileHash>([
+        ['src/changed.ts', makeHash('src/changed.ts', 'xxh64:v1')],
+        ['src/same.ts', makeHash('src/same.ts', 'xxh64:same')],
+        ['src/gone.ts', makeHash('src/gone.ts', 'xxh64:gone')],
+      ])
+
+      const diff = diffHashes(current, stored)
+      expect(diff.added).toEqual(['src/new.ts'])
+      expect(diff.modified).toEqual(['src/changed.ts'])
+      expect(diff.unchanged).toEqual(['src/same.ts'])
+      expect(diff.deleted).toEqual(['src/gone.ts'])
+    })
+  })
+
+  // =========================================================================
+  // computeHashes (integration — reads actual files)
+  // =========================================================================
+
+  describe('computeHashes', () => {
+    test('computes hashes for project files', async () => {
+      // Hash the prjct-cli project itself (small subset)
+      const projectPath = path.resolve(__dirname, '..', '..', '..')
+      const hashes = await computeHashes(projectPath)
+
+      // Should find many files
+      expect(hashes.size).toBeGreaterThan(50)
+
+      // Check a known file exists
+      const packageJson = hashes.get('package.json')
+      expect(packageJson).toBeDefined()
+      expect(packageJson!.hash).toMatch(/^(xxh64|fnv1a):/)
+      expect(packageJson!.size).toBeGreaterThan(0)
+    })
+
+    test('excludes node_modules and .git', async () => {
+      const projectPath = path.resolve(__dirname, '..', '..', '..')
+      const hashes = await computeHashes(projectPath)
+
+      for (const [filePath] of hashes) {
+        expect(filePath).not.toContain('node_modules')
+        expect(filePath).not.toContain('.git/')
+      }
+    })
+
+    test('hash is deterministic', async () => {
+      const projectPath = path.resolve(__dirname, '..', '..', '..')
+      const hashes1 = await computeHashes(projectPath)
+      const hashes2 = await computeHashes(projectPath)
+
+      // Same file should produce same hash
+      const pkg1 = hashes1.get('package.json')
+      const pkg2 = hashes2.get('package.json')
+      expect(pkg1?.hash).toBe(pkg2?.hash)
+    })
+  })
+})