@hardlydifficult/repo-processor 1.0.75 → 1.0.76

package/README.md

@@ -11,276 +11,277 @@ npm install @hardlydifficult/repo-processor
 ## Quick Start
 
 ```typescript
-import { RepoProcessor, GitYamlStore } from "@hardlydifficult/repo-processor";
-import { createGitHubClient } from "@hardlydifficult/github";
+import { RepoProcessor, GitYamlStore, type ProcessingProgress } from "@hardlydifficult/repo-processor";
+import { GitHubClient } from "@hardlydifficult/github";
 
-// 1. Configure git-backed YAML store
+// Setup the git-backed YAML store for state persistence
 const store = new GitYamlStore({
-  cloneUrl: "https://github.com/owner/repo.git",
-  localPath: ".results",
+  cloneUrl: "https://github.com/owner/repo-data.git",
+  localPath: "./repo-data",
   resultDir: (owner, repo) => `repos/${owner}/${repo}`,
-  gitUser: { name: "CI", email: "ci@example.com" },
+  gitUser: { name: "Processor Bot", email: "bot@example.com" },
 });
 
-// 2. Define processing callbacks
+// Build a GitHub client with token authentication
+const github = new GitHubClient({ token: process.env.GITHUB_TOKEN });
+
+// Configure callbacks to define how files and directories are processed
 const callbacks = {
-  shouldProcess: (entry) => entry.type === "blob",
+  shouldProcess: (entry) => entry.path.endsWith(".ts"),
   processFile: async ({ entry, content }) => ({
     path: entry.path,
+    sha: entry.sha,
     length: content.length,
   }),
-  processDirectory: async (ctx) => ({
-    path: ctx.path,
-    fileCount: ctx.subtreeFilePaths.length,
+  processDirectory: async ({ path, subtreeFilePaths }) => ({
+    path,
+    fileCount: subtreeFilePaths.length,
   }),
 };
 
-// 3. Create and run processor
+// Create and run the processor
 const processor = new RepoProcessor({
-  githubClient: createGitHubClient({ token: process.env.GITHUB_TOKEN! }),
+  githubClient: github,
   store,
   callbacks,
+  concurrency: 5, // Optional: default 5
+  branch: "main", // Optional: default "main"
 });
 
-const result = await processor.run("owner", "repo");
-// { filesProcessed: 10, filesRemoved: 1, dirsProcessed: 3 }
-```
-
-## RepoProcessor: Incremental Repository Processing
-
-The `RepoProcessor` class implements an incremental pipeline for processing GitHub repository file trees. It detects changes by comparing current and previous file SHAs, processes only changed files in parallel batches, and processes affected directories bottom-up.
+const result = await processor.run("hardlydifficult", "typescript", (progress) => {
+  console.log(`${progress.phase}: ${progress.message}`);
+});
 
-### Configuration
+// Example result:
+// {
+//   filesProcessed: 12,
+//   filesRemoved: 0,
+//   dirsProcessed: 5
+// }
+```
 
-| Field | Type | Default | Description |
-|-------|------|:-------:|-------------|
-| `githubClient` | `GitHubClient` | — | GitHub API client from `@hardlydifficult/github` |
-| `store` | `ProcessorStore` | — | Persistence layer for file/dir results and manifests |
-| `callbacks` | `ProcessorCallbacks` | — | Domain logic for filtering, file processing, and directory processing |
-| `concurrency` | `number` | `5` | Max concurrent file/dir processing per batch |
-| `branch` | `string` | `"main"` | Git branch to fetch tree from |
+## RepoProcessor
 
-### ProcessingResult
+Processes GitHub repositories incrementally by fetching file trees, detecting changes via SHA diffing, and persisting results.
 
-Result returned by `RepoProcessor.run()`.
+### Constructor
 
-```typescript
-interface ProcessingResult {
-  filesProcessed: number; // Count of files processed
-  filesRemoved: number;   // Count of deleted files
-  dirsProcessed: number;  // Count of directories processed
-}
-```
+| Parameter | Description |
+|-----------|-------------|
+| `githubClient` | GitHub client for fetching trees and file contents |
+| `store` | Persistence layer implementing `ProcessorStore` |
+| `callbacks` | Domain logic callbacks (`shouldProcess`, `processFile`, `processDirectory`) |
+| `concurrency?` | Parallel batch size (default: `5`) |
+| `branch?` | Git branch to fetch from (default: `"main"`) |
 
-### File and Directory Contexts
+### `run(owner, repo, onProgress?)`
 
-Files are processed top-down (all changed files) and directories bottom-up (deepest first), ensuring child directories are processed before parents.
+Processes the repository and returns a summary.
 
+**Returns:**
 ```typescript
-interface FileContext {
-  readonly entry: TreeEntry;
-  readonly content: string;
-}
-
-interface DirectoryContext {
-  path: string;
-  sha: string;
-  subtreeFilePaths: readonly string[];
-  children: readonly DirectoryChild[];
-  tree: readonly TreeEntry[];
-}
-
-interface DirectoryChild {
-  readonly name: string;
-  readonly isDir: boolean;
-  readonly fullPath: string;
+{
+  filesProcessed: number;
+  filesRemoved: number;
+  dirsProcessed: number;
 }
 ```
 
-### Usage Example
-
+**Example:**
 ```typescript
-import { RepoProcessor } from "@hardlydifficult/repo-processor";
-import { createGitHubClient } from "@hardlydifficult/github";
-
-const processor = new RepoProcessor({
-  githubClient: createGitHubClient({ token: process.env.GITHUB_TOKEN! }),
-  store: new GitYamlStore({
-    cloneUrl: "https://github.com/hardlydifficult/results.git",
-    localPath: ".results",
-    resultDir: () => "repos",
-    gitUser: { name: "CI", email: "ci@example.com" },
-  }),
-  callbacks: {
-    shouldProcess: (entry) => entry.path.endsWith(".ts"),
-    processFile: async ({ entry, content }) => ({
-      lines: content.split("\n").length,
-      checksum: crypto.createHash("sha256").update(content).digest("hex"),
-    }),
-    processDirectory: async (ctx) => ({
-      path: ctx.path,
-      fileCount: ctx.subtreeFilePaths.length,
-      hasSubdirs: ctx.children.some((c) => c.isDir),
-    }),
-  },
-  concurrency: 10,
-});
-
 const result = await processor.run("hardlydifficult", "typescript", (progress) => {
-  console.log(`Phase: ${progress.phase} | Files: ${progress.filesCompleted}/${progress.filesTotal}`);
+  console.log(`Phase: ${progress.phase}, Files: ${progress.filesCompleted}/${progress.filesTotal}`);
 });
-// => { filesProcessed: 12, filesRemoved: 0, dirsProcessed: 4 }
 ```
 
-## RepoWatcher: SHA-based Change Monitoring
+### Directory Processing
 
-The `RepoWatcher` class watches GitHub repositories for SHA changes and triggers processing. It supports automatic state persistence, concurrent run prevention, pending SHA re-triggers, and retry logic.
+Directories are processed bottom-up (deepest first) after all files. Each directory context includes:
+- `path`: Directory path (empty string for root)
+- `sha`: Git tree SHA
+- `subtreeFilePaths`: All file paths beneath the directory
+- `children`: Immediate children (files and subdirectories)
+- `tree`: Full repo tree entries
 
-### Configuration
+## RepoWatcher
 
-| Field | Type | Default | Description |
-|-------|------|:-------:|-------------|
-| `stateKey` | `string` | — | Key used for persisting state (e.g., `"repo-processor"`). |
-| `stateDirectory` | `string` | — | Directory where state is persisted. |
-| `autoSaveMs` | `number` | `5000` | Auto-save interval in milliseconds. |
-| `run` | `(owner: string, name: string) => Promise<T>` | — | Function to execute when processing is triggered. |
-| `onComplete` | `(owner, name, result, sha) => void` | — | Called after a successful run (optional). |
-| `onError` | `(owner, name, error) => void` | — | Called when a run fails (optional). |
-| `onEvent` | `(event) => void` | — | Logger/event callback (optional). |
-| `maxAttempts` | `number` | `1` | Max attempts for each run (includes initial + retries). |
+Watches repositories for SHA changes and triggers processing with retry logic and deduplication.
 
-### Usage
+### Constructor
 
 ```typescript
-import { RepoWatcher } from "@hardlydifficult/repo-processor";
-import { RepoProcessor } from "@hardlydifficult/repo-processor";
+interface RepoWatcherConfig<TResult> {
+  stateKey: string;              // Key for persisting state
+  stateDirectory: string;        // Directory for state files
+  autoSaveMs?: number;           // Auto-save interval (default: 5000)
+  run: (owner: string, name: string) => Promise<TResult>;
+  onComplete?: (owner: string, name: string, result: TResult, sha: string) => void;
+  onError?: (owner: string, name: string, error: unknown) => void;
+  onEvent?: (event: StateTrackerEvent) => void;
+  maxAttempts?: number;          // Retry attempts (default: 1)
+}
+```
+
+### Methods
 
+| Method | Description |
+|--------|-------------|
+| `init()` | Load persisted state from disk |
+| `handlePush(owner, name, sha)` | Handle GitHub push event. Queues processing if SHA changed |
+| `trigger(owner, name)` | Manually trigger processing (skips SHA check) |
+| `triggerManual(owner, name)` | Synchronous trigger. Returns `{ success, result }` |
+| `isRunning(owner, name)` | Check if repo is currently processing |
+| `getLastSha(key)` | Get last processed SHA for a repo key |
+| `setLastSha(key, sha)` | Manually set last processed SHA |
+
+**Example:**
+```typescript
 const watcher = new RepoWatcher({
   stateKey: "repo-processor",
-  stateDirectory: ".state",
+  stateDirectory: "./state",
   run: async (owner, name) => {
-    const processor = new RepoProcessor({
-      githubClient: createGitHubClient({ token: process.env.GITHUB_TOKEN! }),
-      store: new GitYamlStore({
-        cloneUrl: "https://github.com/hardlydifficult/results.git",
-        localPath: ".results",
-        resultDir: () => "repos",
-        gitUser: { name: "CI", email: "ci@example.com" },
-      }),
-      callbacks: {
-        shouldProcess: (entry) => entry.path.endsWith(".ts"),
-        processFile: async ({ entry, content }) => ({
-          path: entry.path,
-          lines: content.split("\n").length,
-        }),
-        processDirectory: async (ctx) => ({
-          path: ctx.path,
-          fileCount: ctx.subtreeFilePaths.length,
-        }),
-      },
-    });
-    return processor.run(owner, name);
+    const processor = new RepoProcessor({ /* ... */ });
+    return await processor.run(owner, name);
   },
+  onComplete: (owner, name, result, sha) => {
+    console.log(`Processed ${owner}/${name}, SHA: ${sha}`);
+  },
+  maxAttempts: 3,
 });
 
-await watcher.init(); // Load persisted state
+await watcher.init();
 watcher.handlePush("hardlydifficult", "typescript", "abc123");
-// Triggers processing if SHA differs from last tracked SHA
 ```
 
-### Trigger Methods
-
-```typescript
-// Handle push events (SHA comparison performed automatically)
-watcher.handlePush("hardlydifficult", "typescript", "abc123...");
-
-// Manual trigger (no SHA comparison)
-watcher.trigger("hardlydifficult", "typescript");
+## GitYamlStore
 
-// Synchronous trigger (blocks until complete)
-const response = await watcher.triggerManual("hardlydifficult", "typescript");
-// => { success: true, result: ProcessingResult } | { success: false, reason: string }
-```
+Persists processing results as YAML files in a git repository.
 
-## GitYamlStore: Git-Backed YAML Persistence
+### Constructor
 
-The `GitYamlStore` class implements `ProcessorStore` by persisting results as YAML files in a git repository. Each result includes the tree SHA for change detection. It supports authenticated cloning, auto-pull, batch commits, and push with conflict resolution.
+| Parameter | Description |
+|-----------|-------------|
+| `cloneUrl` | URL of the git repository to clone/pull |
+| `localPath` | Local directory to clone the repo into |
+| `resultDir` | Function mapping `(owner, repo)` to result subdirectory |
+| `authToken?` | GitHub token for authenticated operations (fallback: `GITHUB_TOKEN` env) |
+| `gitUser` | Git committer identity: `{ name: string, email: string }` |
 
-### Configuration
+### Result Storage
 
-| Field | Type | Default | Description |
-|-------|------|:-------:|-------------|
-| `cloneUrl` | `string` | — | URL of the git repository to clone (e.g., `"https://github.com/user/results.git"`). |
-| `localPath` | `string` | — | Local directory to clone the repo into. |
-| `resultDir` | `(owner, repo) => string` | — | Function mapping owner/repo to result subdirectory. |
-| `authToken` | `string` | `process.env.GITHUB_TOKEN` | GitHub token for authenticated clone/push. |
-| `gitUser` | `{ name: string; email: string }` | — | Git user identity used when committing. |
+- File results: `<resultDir>/<filePath>.yml`
+- Directory results: `<resultDir>/<dirPath>/dir.yml`
 
-### Loading Results with Schema Validation
+Each YAML file includes a `sha` field for change detection.
 
+**Example:**
 ```typescript
-import { GitYamlStore } from "@hardlydifficult/repo-processor";
-import { z } from "zod";
-
 const store = new GitYamlStore({
-  cloneUrl: "https://github.com/hardlydifficult/results.git",
-  localPath: ".results",
-  resultDir: (owner, repo) => `repos/${owner}/${repo}`,
-  gitUser: { name: "CI", email: "ci@example.com" },
+  cloneUrl: "https://github.com/owner/repo-data.git",
+  localPath: "./data",
+  resultDir: (owner, repo) => `results/${owner}/${repo}`,
+  gitUser: { name: "Processor Bot", email: "bot@example.com" },
 });
+```
 
-const FileResultSchema = z.object({
-  path: z.string(),
-  lines: z.number(),
-  sha: z.string(),
-});
-const fileResult = await store.loadFileResult(
-  "hardlydifficult",
-  "typescript",
+### Typed Load Helpers
+
+```typescript
+// Load file result with Zod validation
+const result = await store.loadFileResult(
+  "owner",
+  "repo",
   "src/index.ts",
-  FileResultSchema
+  z.object({ path: z.string(), sha: z.string(), length: z.number() })
+);
+
+// Load directory result with Zod validation
+const dirResult = await store.loadDirResult(
+  "owner",
+  "repo",
+  "src/utils",
+  z.object({ path: z.string(), fileCount: z.number() })
 );
-// { path: "src/index.ts", lines: 12, sha: "abc..." }
 ```
 
-## resolveStaleDirectories: Stale Directory Detection
+## resolveStaleDirectories
+
+Identifies directories requiring reprocessing by combining SHA-based stale detection with diff-derived stale directories.
 
-The `resolveStaleDirectories` function identifies directories requiring reprocessing by combining diff-based stale directories with SHA-based comparison.
+### Signature
 
 ```typescript
-import {
-  resolveStaleDirectories,
-  GitYamlStore,
-} from "@hardlydifficult/repo-processor";
-import { createGitHubClient } from "@hardlydifficult/github";
+async function resolveStaleDirectories(
+  owner: string,
+  repo: string,
+  staleDirsFromDiff: readonly string[],
+  allFilePaths: readonly string[],
+  tree: readonly TreeEntry[],
+  store: ProcessorStore
+): Promise<string[]>
+```
 
-const client = createGitHubClient({ token: process.env.GITHUB_TOKEN! });
-const { entries, rootSha } = await client.repo("owner", "repo").getFileTree();
-const tree = [...entries, { path: "", type: "tree", sha: rootSha }];
+**Logic:**
+1. Start with stale directories from file diff
+2. Add any directory whose stored SHA is missing or differs from current tree SHA
+3. Always include root directory if missing
 
+**Example:**
+```typescript
 const staleDirs = await resolveStaleDirectories(
   "owner",
   "repo",
-  [], // stale dirs from diff
-  entries.filter((e) => e.type === "blob").map((e) => e.path),
-  tree,
+  ["src"], // directories with changed children
+  ["src/index.ts", "src/utils/helper.ts"],
+  treeEntries,
   store
 );
-// ["src/utils", "src", ""] (bottom-up order inferred later)
-```
 
-### Stale Directory Logic
+// Returns ["src", "src/utils"] if SHAs differ or missing
+```
 
-1. **Diff-based stale dirs** — directories with changed/removed children
-2. **SHA mismatch** — any directory whose stored SHA differs from the current tree SHA
+## Types
 
-Directories whose stored SHA is missing are always included.
+### ProcessorStore Interface
 
-## Progress Reporting
+| Method | Description |
+|--------|-------------|
+| `ensureReady?(owner, repo)` | One-time init (e.g. clone repo). Optional |
+| `getFileManifest(owner, repo)` | Get manifest of previous file SHAs |
+| `getDirSha(owner, repo, dirPath)` | Get stored directory SHA |
+| `writeFileResult(owner, repo, path, sha, result)` | Persist file result |
+| `writeDirResult(owner, repo, path, sha, result)` | Persist directory result |
+| `deleteFileResult(owner, repo, path)` | Delete result for removed file |
+| `commitBatch(owner, repo, count)` | Commit changes |
 
-Progress reported by the `onProgress` callback during `RepoProcessor.run()`.
+### Callback Interfaces
 
 ```typescript
+interface FileContext {
+  entry: TreeEntry;
+  content: string;
+}
+
+interface DirectoryContext {
+  path: string;
+  sha: string;
+  subtreeFilePaths: readonly string[];
+  children: readonly DirectoryChild[];
+  tree: readonly TreeEntry[];
+}
+
+interface DirectoryChild {
+  name: string;
+  isDir: boolean;
+  fullPath: string;
+}
+
+interface ProcessorCallbacks {
+  shouldProcess(entry: TreeEntry): boolean;
+  processFile(ctx: FileContext): Promise<unknown>;
+  processDirectory(ctx: DirectoryContext): Promise<unknown>;
+}
+
 interface ProcessingProgress {
   phase: "loading" | "files" | "directories" | "committing";
   message: string;
@@ -289,45 +290,37 @@ interface ProcessingProgress {
   dirsTotal: number;
   dirsCompleted: number;
 }
-```
-
-| Phase | Description |
-|-------|-------------|
-| `"loading"` | Initial fetching of file tree. |
-| `"files"` | Processing of files. |
-| `"directories"` | Processing of directories (bottom-up). |
-| `"committing"` | Final commit to persistence. |
 
-## ProcessorStore Interface
-
-Consumer-implemented persistence layer.
-
-| Method | Description |
-|--------|-------------|
-| `ensureReady?(owner, repo)` | One-time initialization (optional). |
-| `getFileManifest(owner, repo)` | Return manifest of previously processed file SHAs (path → blob SHA). |
-| `getDirSha(owner, repo, dirPath)` | Return stored SHA for a directory. Null if not stored. |
-| `writeFileResult(owner, repo, path, sha, result)` | Persist result for a processed file. |
-| `writeDirResult(owner, repo, path, sha, result)` | Persist result for a processed directory. |
-| `deleteFileResult(owner, repo, path)` | Delete stored result for a removed file. |
-| `commitBatch(owner, repo, count)` | Commit current batch of changes. |
-
-## ProcessorCallbacks Interface
+interface ProcessingResult {
+  filesProcessed: number;
+  filesRemoved: number;
+  dirsProcessed: number;
+}
+```
 
-Consumer-provided domain logic.
+## Error Handling
 
-| Method | Description |
-|--------|-------------|
-| `shouldProcess(entry)` | Filter: which tree entries should be processed? |
-| `processFile(ctx)` | Process a single changed file. Return value passed to `store.writeFileResult`. |
-| `processDirectory(ctx)` | Process a directory after all children. Return value passed to `store.writeDirResult`. |
+File and directory processing failures throw descriptive errors:
 
-## Setup
+```typescript
+try {
+  await processor.run("owner", "repo");
+} catch (error) {
+  // Error includes all failed paths and messages
+  // e.g., "2 file(s) failed to process:\nfile1.ts: Connection timeout\nfile2.ts: Invalid format"
+}
+```
 
-No external service setup beyond GitHub is required. The package uses `@hardlydifficult/github` for tree fetches and `simple-git` for git operations.
+### Retries
 
-### Environment Variables
+`RepoWatcher` includes automatic retries for transient failures:
 
-| Variable | Usage |
-|---------|-------|
-| `GITHUB_TOKEN` | Used by `GitYamlStore` for authenticated git operations if `authToken` not provided |
+```typescript
+const watcher = new RepoWatcher({
+  // ...
+  maxAttempts: 3, // Initial + 2 retries
+  onError: (owner, name, error) => {
+    console.error(`Failed for ${owner}/${name}: ${error}`);
+  },
+});
+```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hardlydifficult/repo-processor",
-  "version": "1.0.75",
+  "version": "1.0.76",
   "main": "./dist/index.js",
   "types": "./dist/index.d.ts",
   "files": [
@@ -15,19 +15,19 @@
     "clean": "rm -rf dist"
   },
   "dependencies": {
-    "@hardlydifficult/collections": "1.0.10",
-    "@hardlydifficult/github": "1.0.33",
-    "@hardlydifficult/state-tracker": "2.0.21",
-    "@hardlydifficult/text": "1.0.30",
+    "@hardlydifficult/collections": "1.0.11",
+    "@hardlydifficult/github": "1.0.34",
+    "@hardlydifficult/state-tracker": "2.0.22",
+    "@hardlydifficult/text": "1.0.31",
     "simple-git": "3.32.2",
     "yaml": "2.8.2",
     "zod": "4.3.6"
   },
   "peerDependencies": {
-    "@hardlydifficult/collections": "1.0.10",
-    "@hardlydifficult/github": "1.0.33",
-    "@hardlydifficult/state-tracker": "2.0.21",
-    "@hardlydifficult/text": "1.0.30"
+    "@hardlydifficult/collections": "1.0.11",
+    "@hardlydifficult/github": "1.0.34",
+    "@hardlydifficult/state-tracker": "2.0.22",
+    "@hardlydifficult/text": "1.0.31"
   },
   "devDependencies": {
     "@types/node": "25.3.0",