vibe-validate 0.17.0 → 0.17.1-rc.2

package/skill/resources/extending-extraction.md

@@ -0,0 +1,451 @@
+# Creating Custom Extractors for vibe-validate
+
+## Purpose
+This guide helps you create custom error extractors when vibe-validate isn't capturing errors properly for your tools.
+
+## When to Use This Guide
+
+**You're in the right place if:**
+1. Validation fails but no errors were extracted (exitCode !== 0 but totalErrors === 0)
+2. Generic extractor is being used (extractor: "generic" in metadata)
+3. You want to create a custom extractor for an unsupported tool
+4. Errors aren't being captured properly
+
+**Signs extraction is failing:**
+```yaml
+command: ./gradlew build
+exitCode: 1
+extraction:
+  totalErrors: 0  # ❌ No errors despite failure
+  metadata:
+    detection:
+      extractor: generic  # ❌ Fell back to generic
+      confidence: 50
+```
+
+**Want to understand how extractors work first?** See [Error Extractors Guide](error-extractors-guide.md) for complete details on the extraction system.
+
+## Quick Start: Using the CLI Tool
+
+**CRITICAL**: vibe-validate includes a scaffolding command for creating extractors. **Always use this command first** before manually implementing:
+
+```bash
+vv create-extractor <tool-name> \
+  --description "Brief description of the tool" \
+  --author "User Name <email@example.com>" \
+  --detection-pattern "ERROR_KEYWORD"
+```
+
+### Command Options
+
+- `<tool-name>`: Name of the tool (e.g., `gradle`, `maven`, `webpack`)
+- `--description`: What the extractor does
+- `--author`: Author information
+- `--detection-pattern`: Key pattern that appears in error output (e.g., "ERROR:", "FAILED:", "✖")
+- `--force`: Overwrite existing plugin directory (optional)
+
+### What Gets Generated
+
+```
+vibe-validate-plugin-<tool-name>/
+├── package.json          # NPM package metadata
+├── tsconfig.json         # TypeScript configuration
+├── index.ts              # Plugin implementation (edit this!)
+└── README.md             # Usage instructions
+```
+
+## Step-by-Step Workflow
+
+### Step 1: Analyze Failed Output
+
+When a user encounters extraction failure:
+
+1. **Ask to see the actual error output**:
+   ```
+   "Can you show me the raw output from the failed command?
+   I need to see what error patterns appear."
+   ```
+
+2. **Identify key patterns**:
+   - Error markers: "ERROR:", "FAILED:", "Exception", "✖", etc.
+   - File/line references: `file.ts:42:5`, `(file.ts:42)`, etc.
+   - Message structure: How errors are formatted
+
+### Step 2: Generate Plugin Scaffold
+
+**Run the create-extractor command:**
+
+```bash
+vv create-extractor <tool-name> \
+  --description "Extracts errors from <tool-name> output" \
+  --author "User Name <user@example.com>" \
+  --detection-pattern "<DETECTED_PATTERN>"
+```
+
+**Example:**
+```bash
+# For Gradle builds
+vv create-extractor gradle \
+  --description "Extracts compilation errors from Gradle build output" \
+  --author "Jane Developer <jane@example.com>" \
+  --detection-pattern "FAILURE: Build failed"
+```
+
+### Step 3: Customize the Generated Plugin
+
+Open `vibe-validate-plugin-<tool-name>/index.ts` and customize:
+
+#### Detection Logic
+
+Update the `detect()` function to identify your tool's output:
+
+```typescript
+detect(output: string) {
+  // Check for tool-specific markers
+  const hasToolMarker = output.includes('> Task :') || output.includes('BUILD FAILED');
+  const hasErrorMarker = /FAILURE: Build failed/.test(output);
+
+  if (hasToolMarker && hasErrorMarker) {
+    return {
+      confidence: 95,  // High confidence = specific patterns
+      hints: {
+        required: ['FAILURE: Build failed'],  // Must have these
+        optional: ['> Task :'],                 // Nice to have
+      },
+    };
+  }
+
+  return { confidence: 0, hints: { required: [], optional: [] } };
+}
+```
+
+**Confidence Levels:**
+- `95-100`: Very specific patterns (e.g., "TypeScript compilation error TS2322")
+- `80-94`: Tool-specific but generic (e.g., "BUILD FAILED" + tool name)
+- `60-79`: Generic patterns (e.g., "ERROR:" alone)
+- `0-59`: Not confident, don't use this extractor
+
+#### Extraction Logic
+
+Update the `extract()` function to parse errors:
+
+```typescript
+extract(output: string) {
+  const lines = output.split('\n');
+  const errors: any[] = [];
+
+  for (const line of lines) {
+    // Example: Parse "file.ts:42:5 - error TS2322: Type mismatch"
+    const match = line.match(/^(.+?):(\d+):(\d+) - error (\w+): (.+)$/);
+    if (match) {
+      errors.push({
+        file: match[1],
+        line: parseInt(match[2], 10),
+        column: parseInt(match[3], 10),
+        code: match[4],
+        message: match[5],
+      });
+    }
+  }
+
+  return {
+    errors,
+    totalErrors: errors.length,
+    summary: `${errors.length} error(s) in build`,
+    guidance: 'Fix errors shown above',
+    errorSummary: errors.map(e => `${e.file}:${e.line} - ${e.message}`).join('\n'),
+  };
+}
+```
+
+### Step 4: Build the Plugin
+
+```bash
+cd vibe-validate-plugin-<tool-name>
+npm install
+npm run build
+```
+
+### Step 5: Test with Real Output
+
+**Move to the auto-discovery location:**
+```bash
+cd ..
+mkdir -p vibe-validate-local-plugins
+mv vibe-validate-plugin-<tool-name> vibe-validate-local-plugins/
+```
+
+**Run the command that was failing:**
+```bash
+vv run <failing-command>
+```
+
+**Check if extraction worked:**
+```bash
+vv state
+
+# Look for:
+# metadata:
+#   detection:
+#     extractor: <tool-name>  # Your plugin!
+#     confidence: 95
+#   errors:
+#     - file: ...
+#       line: ...
+#       message: ...
+```
+
+### Step 6: Iterate and Refine
+
+**If extraction isn't working:**
+
+1. **Check detection**:
+   - Is confidence high enough? (should be 60+)
+   - Are the patterns matching?
+   - Add debug logging to `detect()` function
+
+2. **Check extraction**:
+   - Are regex patterns matching the actual output format?
+   - Test regex patterns in isolation first
+   - Handle multi-line errors if needed
+
+3. **Add more examples**:
+   - Collect multiple failure samples
+   - Ensure patterns match all cases
+
+## Common Patterns
+
+### Pattern: Line-based Errors
+
+```typescript
+// Format: "ERROR: Something went wrong at line 42"
+const errorPattern = /ERROR: (.+?) at line (\d+)/;
+const match = line.match(errorPattern);
+if (match) {
+  errors.push({
+    message: match[1],
+    line: parseInt(match[2], 10),
+  });
+}
+```
+
+### Pattern: File:Line:Column Errors
+
+```typescript
+// Format: "src/index.ts:42:5 - Type mismatch"
+const errorPattern = /^(.+?):(\d+):(\d+) - (.+)$/;
+const match = line.match(errorPattern);
+if (match) {
+  errors.push({
+    file: match[1],
+    line: parseInt(match[2], 10),
+    column: parseInt(match[3], 10),
+    message: match[4],
+  });
+}
+```
+
+### Pattern: Multi-line Context
+
+```typescript
+// Errors span multiple lines
+let currentError: any = null;
+
+for (const line of lines) {
+  if (line.startsWith('ERROR:')) {
+    if (currentError) errors.push(currentError);
+    currentError = { message: line.replace('ERROR:', '').trim(), details: [] };
+  } else if (currentError && line.trim()) {
+    currentError.details.push(line.trim());
+  }
+}
+
+if (currentError) errors.push(currentError);
+```
+
+## Plugin Discovery
+
+**Automatic discovery locations:**
+1. `vibe-validate-local-plugins/` directory (relative to project root)
+2. Directories matching `vibe-validate-plugin-*` pattern in project root
+
+**Manual registration (optional):**
+```yaml
+# vibe-validate.config.yaml
+extractors:
+  - path: ./vibe-validate-local-plugins/vibe-validate-plugin-gradle
+    trust: sandbox  # Run in sandboxed environment (recommended)
+```
+
+## Security Considerations
+
+**Sandboxing:**
+- By default, plugins run in a sandboxed environment
+- Limited access to Node.js APIs (no `fs`, `child_process`, network)
+- Can only parse strings and return structured data
+
+**Trust levels:**
+- `sandbox` (default): Safe, limited APIs
+- `trusted`: Full access (requires explicit user consent)
+
+**Best practices:**
+- Use only string operations, regex, and array/object manipulation
+- Don't try to execute commands or access files
+- Don't use `eval()` or `Function()` constructor
+
+## Troubleshooting
+
+### Plugin Not Loading
+
+**Check:**
+1. Is the plugin in the right location? (`vibe-validate-local-plugins/` or `vibe-validate-plugin-*` name)
+2. Did you build it? (`npm run build` in plugin directory)
+3. Is the compiled JS present? (check `dist/` directory)
+4. Are there any warnings when running `vv run`? (look for plugin loading errors)
+
+### Detection Not Working
+
+**Debug detection:**
+```typescript
+detect(output: string) {
+  const hasMarker = output.includes('YOUR_PATTERN');
+  console.log('[DEBUG] hasMarker:', hasMarker);  // Shows in vv output
+  // ... rest of detection
+}
+```
+
+**Common issues:**
+- Pattern is case-sensitive but output varies
+- Pattern includes ANSI color codes (strip them first)
+- Pattern appears in success cases too (add more specificity)
+
+### Extraction Returning No Errors
+
+**Debug extraction:**
+```typescript
+extract(output: string) {
+  const lines = output.split('\n');
+  console.log('[DEBUG] Total lines:', lines.length);
+
+  for (const line of lines) {
+    console.log('[DEBUG] Testing line:', line.substring(0, 100));
+    // ... rest of extraction
+  }
+}
+```
+
+**Common issues:**
+- Regex pattern doesn't match actual output format
+- Output has unexpected whitespace or formatting
+- Error lines are multi-line but code assumes single-line
+
+## Advanced: Contributing Back
+
+Once your plugin is working well:
+
+1. **Prepare for contribution:**
+   ```bash
+   # Ensure tests are added
+   npm test
+
+   # Ensure no sensitive data in test fixtures
+   # Redact any company-specific paths or identifiers
+   ```
+
+2. **Submit to vibe-validate repo:**
+   - Fork the repository
+   - Add plugin to `packages/extractors/src/extractors/`
+   - Add tests to `packages/extractors/test/`
+   - Update `extractor-registry.ts` to include in built-in list
+   - Submit pull request
+
+3. **Benefits of contributing:**
+   - Other users automatically get your extractor
+   - Built-in extractors are maintained and optimized
+   - Community recognition
+
+## Example Workflows
+
+### Workflow 1: Gradle Build Failures
+
+**Problem:** Gradle build fails but errors aren't captured
+
+**Solution:**
+```bash
+# 1. Create plugin
+vv create-extractor gradle \
+  --description "Gradle build error extractor" \
+  --author "User <user@example.com>" \
+  --detection-pattern "BUILD FAILED"
+
+# 2. Customize detection in index.ts (add "Task :" pattern)
+# 3. Customize extraction (parse "error:" lines)
+
+# 4. Build and test
+cd vibe-validate-plugin-gradle
+npm install && npm run build
+cd ..
+mv vibe-validate-plugin-gradle vibe-validate-local-plugins/
+
+# 5. Verify
+vv run ./gradlew build
+vv state  # Should show extracted errors
+```
+
+### Workflow 2: Custom Internal Tool
+
+**Problem:** Company uses proprietary build tool
+
+**Solution:**
+```bash
+# 1. Capture real failure output
+./company-tool build > /tmp/failure.log 2>&1
+
+# 2. Analyze patterns in failure.log
+cat /tmp/failure.log | grep -i error
+
+# 3. Create plugin with identified pattern
+vv create-extractor company-tool \
+  --description "Company tool error extractor" \
+  --author "Developer <dev@company.com>" \
+  --detection-pattern "COMPILATION FAILURE"
+
+# 4. Edit index.ts based on failure.log patterns
+# 5. Test iteratively until all errors extracted
+```
+
+## Related Documentation
+
+- [Error Extractors Guide](../../../docs/error-extractors-guide.md) - Comprehensive extractor documentation
+- [Extractor Plugin Architecture](../../../docs/extractor-plugin-architecture.md) - Technical architecture details
+- [CLI Reference](../../../docs/cli-reference.md) - Full CLI command documentation
+
+## Quick Reference
+
+**Create plugin:**
+```bash
+vv create-extractor <name> --description "..." --author "..." --detection-pattern "..."
+```
+
+**Build plugin:**
+```bash
+cd vibe-validate-plugin-<name> && npm run build
+```
+
+**Move to auto-discovery:**
+```bash
+mkdir -p vibe-validate-local-plugins
+mv vibe-validate-plugin-<name> vibe-validate-local-plugins/
+```
+
+**Test:**
+```bash
+vv run <failing-command>
+vv state  # Check extraction results
+```
+
+**Iterate:**
+1. Edit `index.ts`
+2. Rebuild (`npm run build`)
+3. Test (`vv run ...`)
+4. Repeat until errors extracted correctly

package/skill/resources/run-capability.md

@@ -0,0 +1,229 @@
+# Run Capability: Immediate Caching + Extraction
+
+## Overview
+
+The `vv run` capability provides **immediate benefits without any project configuration**. Just wrap any command to get:
+- **312x speedup** via git tree hash caching
+- **95% token reduction** via smart error extraction
+- **Zero configuration** required
+
+## Quick Start
+
+```bash
+# No installation needed
+npx vibe-validate run <any-command>
+
+# Examples
+npx vibe-validate run npm test
+npx vibe-validate run pytest
+npx vibe-validate run gradle build
+```
+
+## After Installing
+
+Once installed (`npm install vibe-validate` or `npm install -g vibe-validate`):
+
+```bash
+vv run <command>                # Shorthand
+vibe-validate run <command>     # Full name
+
+# Examples
+vv run npm test
+vv run pnpm typecheck
+vv run ./gradlew build
+```
+
+## How It Works
+
+### 1. Git Tree Hash Caching
+
+```bash
+# First run: executes command
+vv run npm test
+
+# Second run (no code changes): uses cache
+vv run npm test  # Instant! (312x faster)
+
+# After code change: re-runs automatically
+vv run npm test  # Executes again
+```
+
+**Cache key:** Git tree hash of your code (deterministic, content-based)
+
+**Force re-run:**
+```bash
+vv run --force npm test  # Skip cache
+```
+
+### 2. Error Extraction
+
+**Without vibe-validate:**
+```
+# 1500+ lines of verbose test output
+Running tests...
+  ✓ test 1
+  ✓ test 2
+  ...
+  ✗ test 42
+    Expected: 5
+    Received: 4
+    at foo.test.ts:42:10
+  ...
+# (1400 more lines)
+```
+
+**With vibe-validate:**
+```yaml
+exitCode: 1
+extraction:
+  totalErrors: 1
+  errors:
+    - file: foo.test.ts
+      line: 42
+      column: 10
+      message: "Expected: 5, Received: 4"
+  summary: "1 test failure"
+  guidance: "Fix assertion at foo.test.ts:42"
+```
+
+**Result:** 1500 tokens → 75 tokens (95% reduction)
+
+## Supported Tools (Built-in Extractors)
+
+vibe-validate automatically detects and extracts errors from:
+
+- **TypeScript**: `tsc --noEmit`
+- **ESLint**: All output formats
+- **Vitest/Jest**: Test failures
+- **Prettier**: Formatting issues
+- **OpenAPI**: Validation errors
+- **Maven**: Compilation errors, test failures
+- **Generic**: Fallback for unknown tools
+
+## YAML Output Format
+
+```yaml
+passed: false
+command: npm test
+exitCode: 1
+durationSecs: 2.5
+extraction:
+  totalErrors: 3
+  errors:
+    - file: src/index.ts
+      line: 42
+      column: 5
+      message: "error TS2322: Type 'string' is not assignable to type 'number'"
+    - file: src/auth.ts
+      line: 128
+      column: 10
+      message: "error TS2345: Argument of type 'null' is not assignable"
+    - file: tests/foo.test.ts
+      line: 15
+      message: "Expected: 5, Received: 4"
+  summary: "3 errors found"
+  guidance: "Fix type errors in src/index.ts and src/auth.ts"
+  metadata:
+    detection:
+      extractor: typescript
+      confidence: 95
+```
+
+## When to Use
+
+### ✅ Use `vv run` when:
+- Running repetitive commands (tests, lint, build)
+- Want immediate caching without configuration
+- Need concise error output (LLM-friendly)
+- Testing in any project (Node.js, Python, Java, etc.)
+
+### ❌ Don't use `vv run` for:
+- Interactive commands (`git log`, `npm init`)
+- Watch modes (`npm run dev`, `vitest --watch`)
+- Commands that already cache well
+- One-time commands
+
+## Common Patterns
+
+### Pattern 1: Test-Driven Development
+```bash
+# Edit code
+vim src/index.ts
+
+# Run tests (fast if code unchanged, extracts errors)
+vv run npm test
+
+# Fix errors, re-run (fast due to cache)
+vv run npm test
+```
+
+### Pattern 2: Pre-Commit Checks
+```bash
+# Run multiple checks
+vv run npm test
+vv run pnpm lint
+vv run pnpm typecheck
+
+# All cached if code unchanged!
+```
+
+### Pattern 3: Monorepo Testing
+```bash
+# Test all packages (caching per-package)
+vv run pnpm -r test
+
+# Only changed packages re-run
+```
+
+## Checking Cache Status
+
+```bash
+# Run command
+vv run npm test
+
+# Check if result is cached
+vv run --check npm test
+
+# Output:
+# ✓ Cached (tree: a1b2c3d4...)
+# Last run: 2 minutes ago
+# exitCode: 0
+```
+
+## Advanced: Debugging Extraction
+
+If errors aren't being extracted properly:
+
+```bash
+# 1. Check what extractor was used
+vv run npm test  # Look for metadata.detection.extractor
+
+# 2. If "generic" extractor used, may need custom extractor
+# See: resources/extending-extraction.md
+
+# 3. Force verbose output temporarily
+npm test  # Run without vibe-validate to see raw output
+```
+
+## Transition to Full Adoption
+
+Once you see the benefits with `vv run`, consider configuring the project:
+
+```bash
+# Install in project
+npm install vibe-validate
+
+# Initialize configuration
+vv init
+
+# Now `vv validate` runs full validation suite with caching
+```
+
+See: **resources/configure-project.md** for full adoption workflow.
+
+## Related Documentation
+
+- **Main docs:** `docs/error-extractors-guide.md` (comprehensive extractor documentation)
+- **CLI reference:** `docs/cli-reference.md`
+- **Extending extraction:** `resources/extending-extraction.md` (if errors not captured)
+- **Project configuration:** `resources/configure-project.md` (adopt for team)