prjct-cli 1.17.0 → 1.19.0

package/CHANGELOG.md

@@ -1,5 +1,106 @@
 # Changelog
 
+## [1.19.0] - 2026-02-09
+
+### Features
+
+- implement aggressive archival of stale storage data (PRJ-267) (#161)
+
+
+## [1.19.0] - 2026-02-09
+
+### Features
+
+- **Aggressive archival of stale storage data** (PRJ-267): Automatic archival during `prjct sync` to keep LLM context lean
+  - Shipped features >90 days archived to SQLite `archives` table with 1-line summary
+  - Pending ideas >180 days marked `dormant` and excluded from LLM context
+  - Completed queue tasks >7 days auto-removed and archived
+  - Paused tasks >30 days archived with persistence (previously discarded)
+  - Memory log capped at 500 active entries, overflow archived
+
+### Implementation Details
+
+New modules:
+- `core/storage/archive-storage.ts` — Archive infrastructure: SQLite `archives` table, batch archival via transactions, restore, prune, stats
+- `core/__tests__/storage/archive-storage.test.ts` — 13 tests covering all archival paths
+
+Modified:
+- `core/storage/database.ts` — Migration v2: `archives` table with entity_type, entity_id, entity_data, summary, reason columns
+- `core/storage/shipped-storage.ts` — `archiveOldShipped()` method with 90-day retention policy
+- `core/storage/ideas-storage.ts` — `markDormantIdeas()` method, `dormant` status excluded from markdown context
+- `core/storage/queue-storage.ts` — `removeStaleCompleted()` method with 7-day retention
+- `core/storage/state-storage.ts` — `archiveStalePausedTasks()` now persists to archive table before removal
+- `core/services/memory-service.ts` — `capEntries()` method with 500-entry cap
+- `core/services/sync-service.ts` — `archiveStaleData()` orchestrates all archival in parallel during sync
+- `core/schemas/ideas.ts` + `core/types/storage.ts` — Added `dormant` to IdeaStatus enum
+
+### Test Plan
+
+#### For QA
+1. Run `prjct sync` with >90d shipped features — verify archive and removal from context
+2. Run sync with >180d pending ideas — verify dormant status, excluded from `ideas.md`
+3. Run sync with >7d completed queue tasks — verify removal and archival
+4. Run sync with >30d paused tasks — verify archival to SQLite
+5. Create >500 memory entries, sync — verify cap at 500
+6. `bun test` — all 947+ tests pass
+7. Verify recent items are NOT archived
+
+#### For Users
+**What changed:** Storage data automatically cleaned up during sync. Old data archived, not deleted.
+**How to use:** No action needed — runs automatically on every sync.
+**Breaking changes:** Ideas can now have `dormant` status (new enum value).
+
+## [1.18.0] - 2026-02-09
+
+### Features
+
+- implement incremental sync with file hashing (PRJ-305) (#160)
+
+
+## [1.18.0] - 2026-02-09
+
+### Features
+
+- **Incremental sync**: `prjct sync` now only re-analyzes files that changed since last sync (PRJ-305)
+  - File hashing with Bun.hash (xxHash64) — <100ms for 500 files
+  - Change propagation through import graph (1-level reverse edges)
+  - Conditional index rebuilds: BM25, import graph, co-change only when source files change
+  - Conditional agent regeneration: only when config files (package.json, tsconfig.json) change
+  - `prjct sync --full` flag to force complete re-analysis
+
+### Implementation Details
+
+New modules:
+- `core/domain/file-hasher.ts` — Hash computation via Bun.hash, SQLite registry using `index_checksums` table, diff detection (added/modified/deleted/unchanged)
+- `core/domain/change-propagator.ts` — Import graph reverse-edge lookup for 1-level change propagation, domain classification for affected files
+
+Modified:
+- `core/services/sync-service.ts` — Incremental decision logic: detect changes → propagate → conditionally rebuild indexes and agents
+- `core/services/watch-service.ts` — Passes accumulated `changedFiles` to sync options
+- `core/types/project-sync.ts` — Added `full`, `changedFiles` to `SyncOptions` + `IncrementalInfo` result type
+- CLI chain (`core/index.ts` → `commands.ts` → `analysis.ts`) — Wired `--full` flag through
+
+### Learnings
+
+- Bun's `fs.readdir` with `withFileTypes` returns `Dirent<NonSharedBuffer>` — need `String()` cast for `.name`
+- Existing `index_checksums` SQLite table was already set up (PRJ-303) — zero schema changes needed
+- Import graph reverse edges (from PRJ-304) enable efficient 1-level propagation without rebuilding the graph
+
+### Test Plan
+
+#### For QA
+1. Run `prjct sync` on fresh project (no hash cache) — should behave as full sync
+2. Run `prjct sync` again without changes — should skip index rebuilds and agent regeneration
+3. Modify a `.ts` file, run `prjct sync` — should detect change and rebuild indexes
+4. Modify `package.json`, run `prjct sync` — should regenerate agents
+5. Run `prjct sync --full` — should force complete re-analysis
+6. Run `prjct watch`, change a file — should pass changedFiles to sync
+
+#### For Users
+**What changed:** `prjct sync` is now incremental by default.
+**How to use:** No changes needed. Use `prjct sync --full` to force complete re-analysis.
+**Breaking changes:** None
+
 ## [1.17.0] - 2026-02-09
 
 ### Features

package/core/__tests__/domain/change-propagator.test.ts

@@ -0,0 +1,100 @@
+import { describe, expect, test } from 'bun:test'
+import { affectedDomains, propagateChanges } from '../../domain/change-propagator'
+import type { FileDiff } from '../../domain/file-hasher'
+
+describe('change-propagator', () => {
+  // =========================================================================
+  // propagateChanges
+  // =========================================================================
+
+  describe('propagateChanges', () => {
+    test('returns direct changes when no import graph exists', () => {
+      const diff: FileDiff = {
+        added: ['src/new.ts'],
+        modified: ['src/changed.ts'],
+        deleted: ['src/removed.ts'],
+        unchanged: ['src/same.ts'],
+      }
+
+      // Use a fake projectId that won't have a graph
+      const result = propagateChanges(diff, 'nonexistent-project')
+
+      expect(result.directlyChanged).toEqual(['src/new.ts', 'src/changed.ts'])
+      expect(result.affectedByImports).toEqual([])
+      expect(result.allAffected).toEqual(['src/new.ts', 'src/changed.ts'])
+      expect(result.deleted).toEqual(['src/removed.ts'])
+    })
+
+    test('empty diff returns empty propagation', () => {
+      const diff: FileDiff = {
+        added: [],
+        modified: [],
+        deleted: [],
+        unchanged: ['src/a.ts', 'src/b.ts'],
+      }
+
+      const result = propagateChanges(diff, 'nonexistent-project')
+
+      expect(result.directlyChanged).toEqual([])
+      expect(result.affectedByImports).toEqual([])
+      expect(result.allAffected).toEqual([])
+    })
+  })
+
+  // =========================================================================
+  // affectedDomains
+  // =========================================================================
+
+  describe('affectedDomains', () => {
+    test('detects frontend files', () => {
+      const domains = affectedDomains([
+        'src/components/Button.tsx',
+        'src/pages/Home.jsx',
+        'styles/main.css',
+      ])
+      expect(domains.has('frontend')).toBe(true)
+      expect(domains.has('uxui')).toBe(true)
+    })
+
+    test('detects backend files', () => {
+      const domains = affectedDomains(['core/services/auth.ts', 'core/domain/user.ts'])
+      expect(domains.has('backend')).toBe(true)
+    })
+
+    test('detects testing files', () => {
+      const domains = affectedDomains([
+        'core/__tests__/auth.test.ts',
+        'src/components/Button.spec.tsx',
+      ])
+      expect(domains.has('testing')).toBe(true)
+    })
+
+    test('detects devops files', () => {
+      const domains = affectedDomains(['Dockerfile', '.github/workflows/ci.yml'])
+      expect(domains.has('devops')).toBe(true)
+    })
+
+    test('detects database files', () => {
+      const domains = affectedDomains(['prisma/schema.prisma', 'db/migrations/001.sql'])
+      expect(domains.has('database')).toBe(true)
+    })
+
+    test('handles mixed domain files', () => {
+      const domains = affectedDomains([
+        'src/components/Form.tsx', // frontend + uxui
+        'core/services/api.ts', // backend
+        'Dockerfile', // devops
+        'core/__tests__/api.test.ts', // testing + backend
+      ])
+      expect(domains.has('frontend')).toBe(true)
+      expect(domains.has('backend')).toBe(true)
+      expect(domains.has('devops')).toBe(true)
+      expect(domains.has('testing')).toBe(true)
+    })
+
+    test('empty file list returns empty domains', () => {
+      const domains = affectedDomains([])
+      expect(domains.size).toBe(0)
+    })
+  })
+})

package/core/__tests__/domain/file-hasher.test.ts

@@ -0,0 +1,146 @@
+import { describe, expect, test } from 'bun:test'
+import path from 'node:path'
+import { computeHashes, diffHashes, type FileHash } from '../../domain/file-hasher'
+
+describe('file-hasher', () => {
+  // =========================================================================
+  // diffHashes
+  // =========================================================================
+
+  describe('diffHashes', () => {
+    const makeHash = (filePath: string, hash: string): FileHash => ({
+      path: filePath,
+      hash,
+      size: 100,
+      mtime: '2026-01-01T00:00:00.000Z',
+    })
+
+    test('detects added files', () => {
+      const current = new Map<string, FileHash>([
+        ['src/new-file.ts', makeHash('src/new-file.ts', 'xxh64:abc')],
+        ['src/existing.ts', makeHash('src/existing.ts', 'xxh64:def')],
+      ])
+      const stored = new Map<string, FileHash>([
+        ['src/existing.ts', makeHash('src/existing.ts', 'xxh64:def')],
+      ])
+
+      const diff = diffHashes(current, stored)
+      expect(diff.added).toEqual(['src/new-file.ts'])
+      expect(diff.modified).toEqual([])
+      expect(diff.unchanged).toEqual(['src/existing.ts'])
+      expect(diff.deleted).toEqual([])
+    })
+
+    test('detects modified files', () => {
+      const current = new Map<string, FileHash>([
+        ['src/changed.ts', makeHash('src/changed.ts', 'xxh64:new-hash')],
+      ])
+      const stored = new Map<string, FileHash>([
+        ['src/changed.ts', makeHash('src/changed.ts', 'xxh64:old-hash')],
+      ])
+
+      const diff = diffHashes(current, stored)
+      expect(diff.added).toEqual([])
+      expect(diff.modified).toEqual(['src/changed.ts'])
+      expect(diff.unchanged).toEqual([])
+      expect(diff.deleted).toEqual([])
+    })
+
+    test('detects deleted files', () => {
+      const current = new Map<string, FileHash>()
+      const stored = new Map<string, FileHash>([
+        ['src/removed.ts', makeHash('src/removed.ts', 'xxh64:abc')],
+      ])
+
+      const diff = diffHashes(current, stored)
+      expect(diff.added).toEqual([])
+      expect(diff.modified).toEqual([])
+      expect(diff.unchanged).toEqual([])
+      expect(diff.deleted).toEqual(['src/removed.ts'])
+    })
+
+    test('handles empty maps', () => {
+      const diff = diffHashes(new Map(), new Map())
+      expect(diff.added).toEqual([])
+      expect(diff.modified).toEqual([])
+      expect(diff.unchanged).toEqual([])
+      expect(diff.deleted).toEqual([])
+    })
+
+    test('handles first sync (no stored hashes)', () => {
+      const current = new Map<string, FileHash>([
+        ['src/a.ts', makeHash('src/a.ts', 'xxh64:1')],
+        ['src/b.ts', makeHash('src/b.ts', 'xxh64:2')],
+        ['src/c.ts', makeHash('src/c.ts', 'xxh64:3')],
+      ])
+      const stored = new Map<string, FileHash>()
+
+      const diff = diffHashes(current, stored)
+      expect(diff.added).toHaveLength(3)
+      expect(diff.modified).toEqual([])
+      expect(diff.unchanged).toEqual([])
+      expect(diff.deleted).toEqual([])
+    })
+
+    test('mixed changes: added + modified + deleted + unchanged', () => {
+      const current = new Map<string, FileHash>([
+        ['src/new.ts', makeHash('src/new.ts', 'xxh64:new')],
+        ['src/changed.ts', makeHash('src/changed.ts', 'xxh64:v2')],
+        ['src/same.ts', makeHash('src/same.ts', 'xxh64:same')],
+      ])
+      const stored = new Map<string, FileHash>([
+        ['src/changed.ts', makeHash('src/changed.ts', 'xxh64:v1')],
+        ['src/same.ts', makeHash('src/same.ts', 'xxh64:same')],
+        ['src/gone.ts', makeHash('src/gone.ts', 'xxh64:gone')],
+      ])
+
+      const diff = diffHashes(current, stored)
+      expect(diff.added).toEqual(['src/new.ts'])
+      expect(diff.modified).toEqual(['src/changed.ts'])
+      expect(diff.unchanged).toEqual(['src/same.ts'])
+      expect(diff.deleted).toEqual(['src/gone.ts'])
+    })
+  })
+
+  // =========================================================================
+  // computeHashes (integration — reads actual files)
+  // =========================================================================
+
+  describe('computeHashes', () => {
+    test('computes hashes for project files', async () => {
+      // Hash the prjct-cli project itself (small subset)
+      const projectPath = path.resolve(__dirname, '..', '..', '..')
+      const hashes = await computeHashes(projectPath)
+
+      // Should find many files
+      expect(hashes.size).toBeGreaterThan(50)
+
+      // Check a known file exists
+      const packageJson = hashes.get('package.json')
+      expect(packageJson).toBeDefined()
+      expect(packageJson!.hash).toMatch(/^(xxh64|fnv1a):/)
+      expect(packageJson!.size).toBeGreaterThan(0)
+    })
+
+    test('excludes node_modules and .git', async () => {
+      const projectPath = path.resolve(__dirname, '..', '..', '..')
+      const hashes = await computeHashes(projectPath)
+
+      for (const [filePath] of hashes) {
+        expect(filePath).not.toContain('node_modules')
+        expect(filePath).not.toContain('.git/')
+      }
+    })
+
+    test('hash is deterministic', async () => {
+      const projectPath = path.resolve(__dirname, '..', '..', '..')
+      const hashes1 = await computeHashes(projectPath)
+      const hashes2 = await computeHashes(projectPath)
+
+      // Same file should produce same hash
+      const pkg1 = hashes1.get('package.json')
+      const pkg2 = hashes2.get('package.json')
+      expect(pkg1?.hash).toBe(pkg2?.hash)
+    })
+  })
+})