@eldrforge/kodrdriv 1.2.24 → 1.2.26

package/MONOREPO-PUBLISH-IMPROVEMENTS.md

@@ -0,0 +1,281 @@
+# Kodrdriv Monorepo Publish Workflow Improvements
+
+## Implementation Summary
+
+This document summarizes the improvements made to the `kodrdriv tree publish --parallel` workflow based on real-world usage feedback with the Fjell monorepo (16 packages).
+
+## Completed Improvements
+
+### 1. ✅ Fixed Recovery Mode to Actually Continue Execution (#2 - High Priority)
+
+**Problem:** When using `--continue --mark-completed`, the system would apply recovery options but then exit immediately without continuing execution.
+
+**Root Cause:** The sequential execution context loading was overwriting the `runConfig`, removing the `--parallel` flag, causing the system to skip parallel execution entirely.
+
+**Solution:**
+- Modified `src/commands/tree.ts` to detect parallel mode and skip sequential context loading
+- Parallel execution now properly continues after recovery options are applied
+- Recovery is now atomic: apply recovery + continue execution in one step
+
+**Files Changed:**
+- `src/commands/tree.ts` (lines 1093-1144)
+
+### 2. ✅ Git Submodule Support (#3 - High Priority)
+
+**Status:** Already implemented and tested in previous work.
+
+**Implementation:** The file-based lock mechanism in `src/util/fileLock.ts` already handles both regular repositories and git submodules by:
+- Detecting if `.git` is a file (submodule) vs directory (regular repo)
+- Reading and parsing the `gitdir:` reference for submodules
+- Creating lock files in the actual git directory
+
+**Files:**
+- `src/util/fileLock.ts`
+- `tests/fileLock.test.ts` (comprehensive test coverage)
+- `SUBMODULE-LOCK-FIX.md` (documentation)
+
+### 3. ✅ Better Status Distinctions (#1/#4 - High Priority)
+
+**Problem:** The system reported "Completed successfully" for packages that were skipped due to no code changes, making it impossible to tell what was actually published.
+
+**Solution:**
+- Added new `skippedNoChanges` field to `ExecutionState` and `ExecutionResult` types
+- Modified `DynamicTaskPool` to track packages skipped due to no changes separately from those skipped due to failed dependencies
+- Updated progress logger to show distinct icons and messages:
+  - ✅ Published (actually executed)
+  - ⊘ Skipped (no code changes)
+  - ⊘ Skipped (dependency failed)
+  - ❌ Failed
+- Enhanced result summary to show detailed breakdown
+
+**Files Changed:**
+- `src/types/parallelExecution.ts` - Added `skippedNoChanges` to state and result types
+- `src/execution/DynamicTaskPool.ts` - Track and report skip reasons
+- `src/execution/TreeExecutionAdapter.ts` - Pass through skip status and format results
+- `src/execution/RecoveryManager.ts` - Include skippedNoChanges in validation
+- `src/commands/tree.ts` - Detect and return skip status from executePackage
+
+**Example Output:**
+```
+📊 Execution Summary:
+
+✅ Published: 3 package(s)
+   @fjell/common-config, @fjell/logging, @fjell/docs-template
+
+⊘ Skipped (no code changes): 5 package(s)
+   @fjell/core, @fjell/http-api, @fjell/registry, @fjell/client-api, @fjell/lib
+
+⊘ Skipped (dependency failed): 8 package(s)
+   @fjell/cache, @fjell/providers, @fjell/sample-app, ...
+   Blocked by: @fjell/core
+
+❌ Failed: 0 package(s)
+```
+
+### 4. ✅ Show Actual Errors Inline (#9 - High Priority)
+
+**Problem:** When packages failed, only generic error messages were shown. The actual error (test failure, build error, merge conflict) was buried in log files.
+
+**Solution:**
+- Added `errorDetails` field to `FailedPackageSnapshot` type with structured error information
+- Implemented `extractErrorDetails()` method in `DynamicTaskPool` to parse errors and extract:
+  - Error type (test_coverage, build_error, merge_conflict, test_failure, timeout, unknown)
+  - Context (specific details about the error)
+  - Log file location
+  - Suggested fix command
+- Enhanced `ProgressFormatter.createErrorSummary()` to display detailed error information
+
+**Files Changed:**
+- `src/types/parallelExecution.ts` - Added errorDetails to FailedPackageSnapshot
+- `src/execution/DynamicTaskPool.ts` - Extract and attach error details
+- `src/ui/ProgressFormatter.ts` - Display detailed error information
+
+**Example Output:**
+```
+❌ Failure Summary:
+
+  @fjell/core:
+    Type: Test Coverage
+    Details: Lines: 89.5% (threshold: 90%)
+    Log: /path/to/core/output/kodrdriv/publish_*.log
+    💡 Suggestion: cd /path/to/core && npm test -- --coverage
+    Blocked: @fjell/cache, @fjell/providers +6 more
+```
+
+### 5. ✅ Add Dry-Run Mode (#6 - Medium Priority)
+
+**Problem:** No way to preview what will happen without actually executing.
+
+**Solution:**
+- Added `generateDryRunPreview()` function that analyzes the dependency graph and shows:
+  - Build order grouped by dependency level
+  - Status for each package (will publish, will skip, etc.)
+  - For publish commands, checks git diff to determine if packages have code changes
+  - Summary statistics
+- Integrated with parallel execution to show preview before executing
+
+**Files Changed:**
+- `src/commands/tree.ts` - Added generateDryRunPreview() and integrated with parallel execution
+
+**Example Output:**
+```
+🔍 DRY RUN MODE - No changes will be made
+
+Build order determined:
+
+Level 1: (1 package)
+  @fjell/common-config
+    Status: 📝 Has changes (23 files), will publish
+    Path: /path/to/common-config
+
+Level 2: (1 package)
+  @fjell/logging
+    Status: ⊘ Only version bump, will skip
+    Path: /path/to/logging
+
+...
+
+Summary:
+  Total packages: 16
+  Dependency levels: 6
+  Command: kodrdriv publish
+  Max concurrency: 8
+
+To execute for real, run the same command without --dry-run
+```
+
+## Not Yet Implemented (Lower Priority)
+
+The following improvements were identified but not implemented in this session:
+
+### 5. Progress Indicators for Long-Running Operations
+- Show sub-step progress during PR checks
+- Display which checks are passing/failing
+- Estimate completion time
+
+### 7. Better Checkpoint Management
+- Add `kodrdriv tree --status` command
+- Add `kodrdriv tree --reset` command
+- Auto-detect and prompt when checkpoint exists
+
+### 8. Fix Concurrency Recommendation Inconsistency
+- Use recommended concurrency by default or explain why not
+
+### 10. Interactive Conflict Resolution
+- Offer to auto-resolve common conflicts (package.json versions, lockfiles)
+- Interactive prompts for manual resolution
+
+### 11. Smart Dependency Updating
+- Auto-update dependent packages when a package is published
+- `kodrdriv tree update-deps --package "@fjell/core@4.4.72"`
+
+### 12. Publish Groups/Profiles
+- Define groups of packages to publish together
+- `kodrdriv tree publish --group core`
+
+### 13. Better npm Registry Integration
+- Check npm for latest versions before publishing
+- Warn about version conflicts
+
+### 14. Automatic Changelog Generation
+- Generate changelogs based on commits since last release
+- Include in release notes automatically
+
+## Testing
+
+### Manual Testing Recommended
+
+1. **Recovery Mode:**
+   ```bash
+   # Start a publish
+   kodrdriv tree publish --parallel
+
+   # If it fails, mark packages as completed and continue
+   kodrdriv tree publish --parallel --continue --mark-completed "pkg1,pkg2"
+
+   # Verify it actually continues execution (not just exits)
+   ```
+
+2. **Status Distinctions:**
+   ```bash
+   # Publish a monorepo where some packages have no changes
+   kodrdriv tree publish --parallel
+
+   # Verify the summary shows:
+   # - ✅ Published: X packages
+   # - ⊘ Skipped (no changes): Y packages
+   # - ⊘ Skipped (dependency failed): Z packages
+   ```
+
+3. **Error Details:**
+   ```bash
+   # Cause a test failure in one package
+   # Run publish and verify the error summary shows:
+   # - Error type
+   # - Specific details
+   # - Log file location
+   # - Suggested fix
+   ```
+
+4. **Dry Run:**
+   ```bash
+   kodrdriv tree publish --parallel --dry-run
+
+   # Verify it shows:
+   # - Build order by level
+   # - Status for each package
+   # - Summary statistics
+   # - Does not actually execute
+   ```
+
+### Unit Tests
+
+The following test files should be updated to cover the new functionality:
+
+- `tests/execution/DynamicTaskPool.test.ts` - Test skippedNoChanges tracking
+- `tests/execution/TreeExecutionAdapter.test.ts` - Test result formatting
+- `tests/execution/RecoveryManager.test.ts` - Test validation with skippedNoChanges
+- `tests/commands/tree.test.ts` - Test dry-run preview and recovery continuation
+
+## Migration Notes
+
+### Breaking Changes
+
+None. All changes are backward compatible.
+
+### API Changes
+
+- `ExecutionState` now includes `skippedNoChanges: string[]`
+- `ExecutionResult` now includes `skippedNoChanges: string[]`
+- `PackageResult` now includes optional `skippedNoChanges?: boolean`
+- `FailedPackageSnapshot` now includes optional `errorDetails?: { type, context, logFile, suggestion }`
+- `executePackage()` return type now includes optional `skippedNoChanges?: boolean`
+
+### Configuration Changes
+
+None required. All new features work with existing configurations.
+
+## Performance Impact
+
+Minimal. The changes primarily affect:
+- Error handling (extracting details from error messages)
+- Status tracking (additional array in state)
+- Dry-run preview (only runs when --dry-run is specified)
+
+## Documentation Updates Needed
+
+- Update `docs/public/commands/tree.md` with:
+  - New status distinctions
+  - Dry-run mode usage
+  - Enhanced error reporting
+  - Recovery mode improvements
+- Update examples to show new output format
+
+## Related Issues/PRs
+
+This implementation addresses the feedback document "Kodrdriv Monorepo Publish Workflow Improvements" which identified 14 pain points based on real-world usage with the Fjell monorepo.
+
+## Contributors
+
+Implementation based on detailed feedback from production usage of kodrdriv with a 16-package monorepo.
+

package/SUBMODULE-LOCK-FIX.md

@@ -0,0 +1,132 @@
+# Git Submodule Lock File Fix
+
+## Problem
+
+The file-based lock mechanism used to prevent concurrent git operations was failing when used with git submodules. The issue occurred because:
+
+1. In regular git repositories, `.git` is a directory
+2. In git submodules, `.git` is a file containing a `gitdir:` reference pointing to the actual git directory
+3. The code assumed `.git` was always a directory and tried to create lock files in `.git/kodrdriv.lock`
+4. This caused `ENOTDIR` errors when operating on submodules
+
+## Example Submodule Structure
+
+```
+my-submodule/
+├── .git              # FILE (not directory) containing: "gitdir: ../.git/modules/my-submodule"
+├── src/
+└── package.json
+
+parent-repo/
+└── .git/
+    └── modules/
+        └── my-submodule/  # Actual git directory for the submodule
+            ├── HEAD
+            ├── refs/
+            └── objects/
+```
+
+## Solution
+
+Modified `src/util/fileLock.ts` to handle both regular repositories and submodules:
+
+### Key Changes
+
+1. **Added `resolveGitDirectory()` method** that:
+   - Checks if `.git` is a directory (regular repo) or file (submodule)
+   - If it's a file, reads and parses the `gitdir:` reference
+   - Resolves the gitdir path (handles both relative and absolute paths)
+   - Returns the actual git directory path where locks can be created
+
+2. **Updated `getRepositoryLock()` method** to:
+   - Use `resolveGitDirectory()` instead of assuming `.git` is a directory
+   - Create lock files in the resolved git directory
+   - Log the actual lock path for debugging
+
+### Code Implementation
+
+```typescript
+private resolveGitDirectory(repoPath: string): string {
+    const gitPath = path.join(repoPath, '.git');
+
+    try {
+        const stat = fs.statSync(gitPath);
+
+        if (stat.isDirectory()) {
+            // Regular git repository
+            return gitPath;
+        } else if (stat.isFile()) {
+            // Git submodule - .git is a file with format: gitdir: <path>
+            const gitFileContent = fs.readFileSync(gitPath, 'utf-8').trim();
+            const match = gitFileContent.match(/^gitdir:\s*(.+)$/);
+
+            if (match && match[1]) {
+                // Resolve the gitdir path (it's relative to the repo path)
+                const gitDirPath = path.resolve(repoPath, match[1]);
+                this.logger.debug(`Resolved submodule gitdir: ${gitDirPath}`);
+
+                // Ensure the git directory exists
+                if (!fs.existsSync(gitDirPath)) {
+                    throw new Error(`Submodule git directory does not exist: ${gitDirPath}`);
+                }
+
+                return gitDirPath;
+            }
+
+            throw new Error(`Invalid .git file format in ${gitPath}: ${gitFileContent}`);
+        }
+    } catch (error: any) {
+        // Check if error is from statSync (file doesn't exist)
+        if (error.code === 'ENOENT') {
+            throw new Error(`No .git directory or file found in ${repoPath}`);
+        }
+        throw new Error(`Failed to resolve git directory for ${repoPath}: ${error.message}`);
+    }
+
+    throw new Error(`No .git directory or file found in ${repoPath}`);
+}
+```
+
+## Testing
+
+Comprehensive test suite added in `tests/fileLock.test.ts` covering:
+
+1. **Basic lock operations**
+   - Acquire and release locks
+   - Block concurrent lock acquisition
+   - Handle stale locks
+
+2. **Regular repository support**
+   - Create locks in `.git` directory
+
+3. **Submodule support** (NEW)
+   - Handle `.git` file with `gitdir:` reference
+   - Resolve relative gitdir paths
+   - Create locks in the actual git directory
+   - Proper error handling for missing submodule directories
+   - Proper error handling for invalid `.git` file format
+
+4. **Lock manager functionality**
+   - Execute operations under lock
+   - Release locks on operation failure
+   - Serialize multiple operations on same repo
+
+All 12 tests pass successfully with 86%+ coverage on the fileLock module.
+
+## Impact
+
+This fix enables kodrdriv to work correctly in monorepo setups where packages are organized as git submodules, such as:
+
+- Multi-repository projects using git submodules for shared libraries
+- Projects with external dependencies included as submodules
+- Monorepos with complex submodule hierarchies
+
+The fix maintains backward compatibility with regular git repositories while adding robust support for submodules.
+
+## Related Files
+
+- `src/util/fileLock.ts` - Core fix implementation
+- `tests/fileLock.test.ts` - Comprehensive test suite
+- `src/util/gitMutex.ts` - Uses RepositoryFileLockManager
+- `src/commands/publish.ts` - Uses git locks during publish operations
+