@boshu2/vibe-check 1.0.1 → 1.1.0

package/README.md

@@ -1,10 +1,56 @@
 # vibe-check
 
-Measure vibe coding effectiveness with git commit analysis.
+> ⚠️ **Experimental** - Metrics correlations with actual productivity outcomes have not been independently validated. Use as a directional signal, not ground truth.
 
-## What is Vibe Coding?
+**Track patterns in your AI-assisted coding workflow.**
 
-Vibe coding is AI-assisted development where you collaborate with an AI coding assistant. This tool measures how effectively that collaboration is working by analyzing your git commit patterns.
+## The Problem
+
+You're using AI to write code, but how do you know if it's actually helping?
+
+Are you shipping features faster, or just generating more commits? Building new features, or debugging AI mistakes? Moving forward, or stuck in fix-fix-fix loops?
+
+Without data, you're guessing.
+
+## The Insight
+
+vibe-check analyzes your git history and tells you:
+
+| What You'll Learn | Why It Matters |
+|-------------------|----------------|
+| **Trust Pass Rate** | Are you accepting AI code that works, or code that breaks immediately? |
+| **Debug Spirals** | Are you stuck in fix loops on the same component? |
+| **Rework Ratio** | What percentage of your work is building vs. cleaning up? |
+| **Pattern Detection** | What types of problems keep recurring? (auth, config, APIs...) |
+
+## Quick Demo
+
+```bash
+$ npx @boshu2/vibe-check --since "1 week ago"
+
+================================================================
+                    VIBE-CHECK RESULTS
+================================================================
+  Period: Nov 21 - Nov 28, 2025 (12.5h active)
+  Commits: 47 total (28 feat, 15 fix, 4 docs)
+
+  METRIC                      VALUE      RATING
+  --------------------------------------------------
+  Iteration Velocity          4.2/hr     HIGH
+  Rework Ratio                35%        MEDIUM
+  Trust Pass Rate             92%        HIGH
+  Debug Spiral Duration       18min      HIGH
+  Flow Efficiency             85%        HIGH
+
+  DEBUG SPIRALS (2 detected):
+  - auth: 4 commits, 25m (SECRETS_AUTH)
+  - api: 3 commits, 12m (API_MISMATCH)
+
+  OVERALL: HIGH
+================================================================
+```
+
+**What this tells you:** You're productive (4.2 commits/hour, 92% trust pass rate), but 35% of your work is fixing things—room to improve. OAuth caused a 25-minute spiral. Next time: validate auth flows with a tracer test before full implementation.
 
 ## Installation
 
@@ -12,7 +58,7 @@ Vibe coding is AI-assisted development where you collaborate with an AI coding a
 npm install -g @boshu2/vibe-check
 ```
 
-Or run directly with npx:
+Or run directly:
 
 ```bash
 npx @boshu2/vibe-check
@@ -21,94 +67,250 @@ npx @boshu2/vibe-check
 ## Usage
 
 ```bash
-# Analyze current repository
+# Analyze current repository (all history)
 vibe-check
 
-# Analyze last week
+# Analyze specific time period
 vibe-check --since "1 week ago"
+vibe-check --since "2025-11-01"
 
-# JSON output for CI/automation
-vibe-check --format json
-
-# Markdown report
-vibe-check --format markdown
+# Different output formats
+vibe-check --format json      # For CI/automation
+vibe-check --format markdown  # For reports
 
-# Analyze specific repo
+# Analyze a different repo
 vibe-check --repo /path/to/repo
 ```
 
-## The 5 Core Metrics (FAAFO)
+## The 5 Metrics
 
-| Metric | What It Measures | Target |
-|--------|------------------|--------|
-| **Iteration Velocity** | Commits per hour | >5/hr = Elite |
-| **Rework Ratio** | % commits that are fixes | <30% = Elite |
-| **Trust Pass Rate** | % commits that stick (no immediate fix) | >95% = Elite |
-| **Debug Spiral Duration** | Time spent in fix chains | <15m = Elite |
-| **Flow Efficiency** | % time in productive work vs debugging | >90% = Elite |
+| Metric | What It Measures | Elite | Good | Needs Work |
+|--------|------------------|-------|------|------------|
+| **Iteration Velocity** | Commits per hour | >5/hr | 3-5/hr | <3/hr |
+| **Rework Ratio** | % of commits that are fixes | <30% | 30-50% | >50% |
+| **Trust Pass Rate** | % of commits without immediate fix | >95% | 80-95% | <80% |
+| **Debug Spiral Duration** | Avg time stuck in fix chains | <15m | 15-30m | >30m |
+| **Flow Efficiency** | % time building vs debugging | >90% | 75-90% | <75% |
 
-## Output Ratings
+### What the ratings suggest
 
-| Rating | Meaning |
-|--------|---------|
-| **ELITE** | Vibe coding working excellently |
-| **HIGH** | Good effectiveness, minor improvements possible |
-| **MEDIUM** | Room for improvement |
-| **LOW** | Process issues, reassess approach |
+- **ELITE**: Commit patterns suggest smooth workflow
+- **HIGH**: Generally healthy patterns, some areas to watch
+- **MEDIUM**: Mixed signals—review individual metrics
+- **LOW**: Commit patterns suggest friction—investigate causes
 
-## Example Output
-
-```
-╔════════════════════════════════════════════════════════════╗
-║                    VIBE CHECK RESULTS                       ║
-╠════════════════════════════════════════════════════════════╣
-║  Period: Nov 21 - Nov 28, 2025                             ║
-║  Commits Analyzed: 47                                       ║
-╠════════════════════════════════════════════════════════════╣
-║                                                             ║
-║  Iteration Velocity    ████████████░░░░  4.2/hr    HIGH    ║
-║  Rework Ratio          ██████░░░░░░░░░░  35%       MEDIUM  ║
-║  Trust Pass Rate       ████████████████  92%       HIGH    ║
-║  Debug Spiral Duration ██████████████░░  18min     HIGH    ║
-║  Flow Efficiency       ██████████████░░  85%       HIGH    ║
-║                                                             ║
-╠════════════════════════════════════════════════════════════╣
-║  Overall Rating: HIGH                                       ║
-╚════════════════════════════════════════════════════════════╝
-```
+*Note: These ratings reflect commit patterns, not actual code quality or productivity.*
 
 ## Debug Spiral Detection
 
-When the tool detects fix chains (multiple fix commits in a row), it identifies:
-- Which component had issues
-- Pattern category (e.g., `API_MISMATCH`, `SSL_TLS`, `VOLUME_CONFIG`)
-- Duration of the spiral
-- Suggested tracer test to prevent recurrence
+When vibe-check detects 3+ consecutive fix commits on the same component, it flags a "debug spiral" and categorizes the pattern:
 
-## Integration with Claude Code
+| Pattern | What It Means | Prevention |
+|---------|---------------|------------|
+| `SECRETS_AUTH` | Auth/OAuth/credentials issues | Validate auth flow before implementation |
+| `API_MISMATCH` | API version or schema problems | Check API docs, deploy minimal test first |
+| `VOLUME_CONFIG` | Mount/path/permission issues | Test volume config in isolation |
+| `SSL_TLS` | Certificate/HTTPS problems | Verify certs before deploying |
+| `IMAGE_REGISTRY` | Container pull/tag issues | Test image pull separately |
 
-This tool is designed to work with the RPI (Research-Plan-Implement) workflow:
+## When to Run
 
-1. **Before implementation**: Run `vibe-check` to establish baseline
-2. **After implementation**: Run `vibe-check --since "<start-time>"` to measure session
-3. **Weekly review**: Run `vibe-check --since "1 week ago" --format markdown`
+- **Before starting work**: Establish your baseline
+- **After a session**: Measure what just happened
+- **Weekly**: Track trends over time
+- **After frustrating sessions**: Identify what went wrong
 
-## Options
+## CLI Options
 
 ```
 -V, --version        Output version number
---since <date>       Start date for analysis (e.g., "1 week ago", "2025-11-01")
---until <date>       End date for analysis (default: now)
--f, --format <type>  Output format: terminal, json, markdown (default: terminal)
+--since <date>       Start date (e.g., "1 week ago", "2025-11-01")
+--until <date>       End date (default: now)
+-f, --format <type>  Output: terminal, json, markdown
 -r, --repo <path>    Repository path (default: current directory)
--v, --verbose        Show verbose output
+-o, --output <file>  Write JSON results to file
+-v, --verbose        Show detailed output
+--score              Include VibeScore (semantic-free metrics)
+--recommend          Include level recommendation
+--simple             Show simplified output (less verbose)
 -h, --help           Display help
 ```
 
+## Gamification
+
+vibe-check tracks your progress over time with XP, levels, streaks, and achievements.
+
+### Levels
+
+| Level | Name | XP Required |
+|-------|------|-------------|
+| 1 | Newbie | 0 |
+| 2 | Regular | 100 |
+| 3 | Committed | 300 |
+| 4 | Dedicated | 600 |
+| 5 | Expert | 1000 |
+| 6 | Master | 1500 |
+| 7 | Grandmaster | 2100 |
+| 8 | Guru | 2800 |
+| 9 | Sage | 3600 |
+| 10 | Legend | 4500 |
+
+### Achievements
+
+Unlock achievements by hitting milestones:
+
+- 🩸 **First Blood** - Run your first vibe-check
+- ⚔️ **Week Warrior** - Maintain a 7-day streak
+- 👑 **Monthly Master** - Maintain a 30-day streak
+- ✨ **Elite Vibes** - Achieve ELITE rating
+- 🏅 **Ninety Club** - Vibe Score of 90%+
+- 🧘 **Zen Master** - 50+ commits, 0 debug spirals
+- ...and 12 more, including 2 hidden achievements!
+
+### Profile Command
+
+View your stats, achievements, and progress:
+
+```bash
+# View your profile
+vibe-check profile
+
+# See all achievements
+vibe-check profile --achievements
+
+# Detailed statistics
+vibe-check profile --stats
+
+# JSON output
+vibe-check profile --json
+```
+
+### Save Results to JSON
+
+```bash
+# Write JSON to file while showing terminal output
+vibe-check --since "1 week ago" --score -o results.json
+
+# Combine with other formats
+vibe-check --format markdown -o results.json  # Terminal gets markdown, file gets JSON
+```
+
+## GitHub Action
+
+Add automated vibe-check to your PRs:
+
+```yaml
+# .github/workflows/vibe-check.yml
+name: Vibe Check
+
+on:
+  pull_request:
+    branches: [main]
+
+jobs:
+  vibe-check:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: write
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Run Vibe Check
+        uses: boshu2/vibe-check@v1
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+```
+
+### Action Inputs
+
+| Input | Description | Default |
+|-------|-------------|---------|
+| `github-token` | GitHub token for PR comments | `${{ github.token }}` |
+| `since` | Start date for analysis | PR base commit |
+| `threshold` | Minimum rating to pass (elite, solid, needs-work) | none |
+| `include-score` | Include VibeScore | `true` |
+| `include-recommendation` | Include level recommendation | `true` |
+| `output-file` | Path to write JSON results | none |
+| `comment-on-pr` | Post results as PR comment | `true` |
+
+### Action Outputs
+
+| Output | Description |
+|--------|-------------|
+| `overall` | Overall rating (elite, solid, needs-work, struggling) |
+| `vibe-score` | Numeric score (0-100) |
+| `json` | Full JSON results |
+
+### Example: Fail PR if Below Threshold
+
+```yaml
+- uses: boshu2/vibe-check@v1
+  with:
+    github-token: ${{ secrets.GITHUB_TOKEN }}
+    threshold: 'solid'  # Fails if below solid
+```
+
+### Example: Save Results to File
+
+```yaml
+- uses: boshu2/vibe-check@v1
+  with:
+    github-token: ${{ secrets.GITHUB_TOKEN }}
+    output-file: 'vibe-check-results.json'
+
+- name: Upload results
+  uses: actions/upload-artifact@v4
+  with:
+    name: vibe-check-results
+    path: vibe-check-results.json
+```
+
 ## Requirements
 
-- Node.js >= 18.0.0
+- Node.js >= 20.0.0
 - Git repository with commit history
+- Conventional commits recommended (but not required)
+
+## Limitations & Caveats
+
+### What This Tool Does NOT Measure
+
+| Claim | Reality |
+|-------|---------|
+| Code quality | Measures commit patterns, not code correctness |
+| Actual productivity | Measures velocity signals, not shipped value |
+| AI effectiveness | Measures workflow patterns, not AI contribution |
+
+### Known Limitations
+
+1. **No ground truth validation**: The correlation between these metrics and actual productivity outcomes has not been independently validated.
+
+2. **Threshold sensitivity**: Magic numbers (5 min spiral threshold, 3-file churn) are based on practitioner intuition, not empirical studies.
+
+3. **Goodhart's Law risk**: Once you know the metrics, you may unconsciously optimize for them rather than actual outcomes.
+
+4. **Cold start**: New repositories have no calibration data. Default model weights are educated guesses.
+
+5. **Sample size**: The ML model requires 20+ calibration samples for meaningful learning. Results with fewer samples are unreliable.
+
+### When NOT to Use
+
+- As a performance review metric (easily gamed)
+- To compare across teams or developers (different baselines)
+- As the sole indicator of AI tool effectiveness
+- Without understanding what each metric actually measures
+
+### Recommended Use
+
+Use vibe-check as **one signal among many**:
+- Combine with code review feedback
+- Track alongside deployment success rates
+- Use for self-reflection, not external judgment
+- Treat as directional, not precise
 
 ## License
 

package/action.yml

@@ -0,0 +1,270 @@
+name: 'Vibe Check'
+description: 'Analyze git commit patterns and post vibe coding metrics to PR comments'
+author: 'boshu2'
+branding:
+  icon: 'activity'
+  color: 'purple'
+
+inputs:
+  github-token:
+    description: 'GitHub token for posting PR comments'
+    required: true
+    default: ${{ github.token }}
+  since:
+    description: 'Start date for analysis (e.g., "1 week ago", or leave empty for PR commits only)'
+    required: false
+    default: ''
+  threshold:
+    description: 'Minimum overall rating to pass (elite, solid, needs-work, struggling)'
+    required: false
+    default: ''
+  include-score:
+    description: 'Include VibeScore in output'
+    required: false
+    default: 'true'
+  include-recommendation:
+    description: 'Include level recommendation in output'
+    required: false
+    default: 'true'
+  output-file:
+    description: 'Path to write JSON results (optional)'
+    required: false
+    default: ''
+  comment-on-pr:
+    description: 'Post results as PR comment'
+    required: false
+    default: 'true'
+
+outputs:
+  overall:
+    description: 'Overall rating (elite, solid, needs-work, struggling)'
+    value: ${{ steps.vibe-check.outputs.overall }}
+  vibe-score:
+    description: 'Numeric vibe score (0-100)'
+    value: ${{ steps.vibe-check.outputs.vibe-score }}
+  json:
+    description: 'Full JSON results'
+    value: ${{ steps.vibe-check.outputs.json }}
+
+runs:
+  using: 'composite'
+  steps:
+    - name: Setup Node.js
+      uses: actions/setup-node@v4
+      with:
+        node-version: '20'
+
+    - name: Install vibe-check
+      shell: bash
+      run: npm install -g @boshu2/vibe-check
+
+    - name: Fetch full git history
+      shell: bash
+      run: |
+        git fetch --unshallow 2>/dev/null || true
+        git fetch origin ${{ github.base_ref }} 2>/dev/null || true
+
+    - name: Run vibe-check
+      id: vibe-check
+      shell: bash
+      env:
+        GITHUB_TOKEN: ${{ inputs.github-token }}
+      run: |
+        # Determine the date range
+        if [ -n "${{ inputs.since }}" ]; then
+          SINCE_ARG="--since \"${{ inputs.since }}\""
+        elif [ "${{ github.event_name }}" = "pull_request" ]; then
+          # For PRs, analyze commits since the base branch
+          BASE_SHA=$(git merge-base origin/${{ github.base_ref }} HEAD 2>/dev/null || echo "")
+          if [ -n "$BASE_SHA" ]; then
+            BASE_DATE=$(git show -s --format=%ci $BASE_SHA)
+            SINCE_ARG="--since \"$BASE_DATE\""
+          else
+            SINCE_ARG="--since \"1 week ago\""
+          fi
+        else
+          SINCE_ARG="--since \"1 week ago\""
+        fi
+
+        # Build command
+        CMD="vibe-check"
+        [ "${{ inputs.include-score }}" = "true" ] && CMD="$CMD --score"
+        [ "${{ inputs.include-recommendation }}" = "true" ] && CMD="$CMD --recommend"
+
+        # Run with JSON output
+        echo "Running: $CMD $SINCE_ARG --format json"
+        RESULT=$(eval "$CMD $SINCE_ARG --format json" 2>&1) || true
+
+        # Parse results
+        if echo "$RESULT" | jq . > /dev/null 2>&1; then
+          OVERALL=$(echo "$RESULT" | jq -r '.overall')
+          VIBE_SCORE=$(echo "$RESULT" | jq -r '.vibeScore // "N/A"')
+
+          echo "overall=$OVERALL" >> $GITHUB_OUTPUT
+          echo "vibe-score=$VIBE_SCORE" >> $GITHUB_OUTPUT
+
+          # Save JSON (escape for multiline)
+          EOF=$(dd if=/dev/urandom bs=15 count=1 status=none | base64)
+          echo "json<<$EOF" >> $GITHUB_OUTPUT
+          echo "$RESULT" >> $GITHUB_OUTPUT
+          echo "$EOF" >> $GITHUB_OUTPUT
+
+          # Write to file if requested
+          if [ -n "${{ inputs.output-file }}" ]; then
+            echo "$RESULT" > "${{ inputs.output-file }}"
+            echo "Results written to ${{ inputs.output-file }}"
+          fi
+        else
+          echo "overall=error" >> $GITHUB_OUTPUT
+          echo "vibe-score=0" >> $GITHUB_OUTPUT
+          echo "json={\"error\": \"Failed to parse vibe-check output\"}" >> $GITHUB_OUTPUT
+          echo "::warning::vibe-check output was not valid JSON: $RESULT"
+        fi
+
+    - name: Generate markdown comment
+      id: markdown
+      if: inputs.comment-on-pr == 'true' && github.event_name == 'pull_request'
+      shell: bash
+      run: |
+        JSON='${{ steps.vibe-check.outputs.json }}'
+
+        # Parse metrics
+        OVERALL=$(echo "$JSON" | jq -r '.overall')
+        COMMITS=$(echo "$JSON" | jq -r '.commits')
+        VIBE_SCORE=$(echo "$JSON" | jq -r '.vibeScore // "N/A"')
+
+        # Get individual metrics
+        VELOCITY=$(echo "$JSON" | jq -r '.metrics.iterationVelocity.value')
+        VELOCITY_RATING=$(echo "$JSON" | jq -r '.metrics.iterationVelocity.rating')
+        REWORK=$(echo "$JSON" | jq -r '.metrics.reworkRatio.value')
+        REWORK_RATING=$(echo "$JSON" | jq -r '.metrics.reworkRatio.rating')
+        TRUST=$(echo "$JSON" | jq -r '.metrics.trustPassRate.value')
+        TRUST_RATING=$(echo "$JSON" | jq -r '.metrics.trustPassRate.rating')
+        SPIRAL=$(echo "$JSON" | jq -r '.metrics.debugSpiralDuration.value')
+        SPIRAL_RATING=$(echo "$JSON" | jq -r '.metrics.debugSpiralDuration.rating')
+        FLOW=$(echo "$JSON" | jq -r '.metrics.flowEfficiency.value')
+        FLOW_RATING=$(echo "$JSON" | jq -r '.metrics.flowEfficiency.rating')
+
+        # Map ratings to emoji
+        rating_emoji() {
+          case "$1" in
+            elite) echo "🟢" ;;
+            solid) echo "🔵" ;;
+            needs-work) echo "🟡" ;;
+            struggling) echo "🔴" ;;
+            *) echo "⚪" ;;
+          esac
+        }
+
+        # Overall emoji
+        case "$OVERALL" in
+          elite) OVERALL_EMOJI="🚀" ;;
+          solid) OVERALL_EMOJI="✅" ;;
+          needs-work) OVERALL_EMOJI="⚠️" ;;
+          struggling) OVERALL_EMOJI="🔴" ;;
+          *) OVERALL_EMOJI="❓" ;;
+        esac
+
+        # Build comment
+        COMMENT="## $OVERALL_EMOJI Vibe Check: **${OVERALL^^}**
+
+**Commits analyzed:** $COMMITS"
+
+        [ "$VIBE_SCORE" != "N/A" ] && [ "$VIBE_SCORE" != "null" ] && COMMENT="$COMMENT | **VibeScore:** $VIBE_SCORE/100"
+
+        COMMENT="$COMMENT
+
+| Metric | Value | Rating |
+|--------|-------|--------|
+| Iteration Velocity | ${VELOCITY}/hr | $(rating_emoji $VELOCITY_RATING) $VELOCITY_RATING |
+| Rework Ratio | ${REWORK}% | $(rating_emoji $REWORK_RATING) $REWORK_RATING |
+| Trust Pass Rate | ${TRUST}% | $(rating_emoji $TRUST_RATING) $TRUST_RATING |
+| Debug Spiral Duration | ${SPIRAL}m | $(rating_emoji $SPIRAL_RATING) $SPIRAL_RATING |
+| Flow Efficiency | ${FLOW}% | $(rating_emoji $FLOW_RATING) $FLOW_RATING |
+
+<details>
+<summary>What do these metrics mean?</summary>
+
+- **Iteration Velocity**: Commits per active hour (higher = faster iteration)
+- **Rework Ratio**: Percentage of commits that are fixes (lower = cleaner first attempts)
+- **Trust Pass Rate**: Commits without immediate follow-up fixes (higher = more reliable)
+- **Debug Spiral Duration**: Average time stuck in fix chains (lower = faster recovery)
+- **Flow Efficiency**: Time building vs debugging (higher = more productive)
+
+</details>
+
+---
+*Generated by [vibe-check](https://github.com/boshu2/vibe-check)*"
+
+        # Save comment for next step
+        EOF=$(dd if=/dev/urandom bs=15 count=1 status=none | base64)
+        echo "comment<<$EOF" >> $GITHUB_OUTPUT
+        echo "$COMMENT" >> $GITHUB_OUTPUT
+        echo "$EOF" >> $GITHUB_OUTPUT
+
+    - name: Post PR comment
+      if: inputs.comment-on-pr == 'true' && github.event_name == 'pull_request'
+      uses: actions/github-script@v7
+      with:
+        github-token: ${{ inputs.github-token }}
+        script: |
+          const comment = `${{ steps.markdown.outputs.comment }}`;
+
+          // Find existing vibe-check comment
+          const { data: comments } = await github.rest.issues.listComments({
+            owner: context.repo.owner,
+            repo: context.repo.repo,
+            issue_number: context.issue.number,
+          });
+
+          const existingComment = comments.find(c =>
+            c.body.includes('Generated by [vibe-check]')
+          );
+
+          if (existingComment) {
+            // Update existing comment
+            await github.rest.issues.updateComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              comment_id: existingComment.id,
+              body: comment
+            });
+            console.log('Updated existing vibe-check comment');
+          } else {
+            // Create new comment
+            await github.rest.issues.createComment({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: context.issue.number,
+              body: comment
+            });
+            console.log('Created new vibe-check comment');
+          }
+
+    - name: Check threshold
+      if: inputs.threshold != ''
+      shell: bash
+      run: |
+        OVERALL="${{ steps.vibe-check.outputs.overall }}"
+        THRESHOLD="${{ inputs.threshold }}"
+
+        # Rating order (higher = better)
+        rating_to_num() {
+          case "$1" in
+            elite) echo 4 ;;
+            solid) echo 3 ;;
+            needs-work) echo 2 ;;
+            struggling) echo 1 ;;
+            *) echo 0 ;;
+          esac
+        }
+
+        ACTUAL=$(rating_to_num "$OVERALL")
+        REQUIRED=$(rating_to_num "$THRESHOLD")
+
+        if [ "$ACTUAL" -lt "$REQUIRED" ]; then
+          echo "::error::Vibe check failed: $OVERALL is below threshold $THRESHOLD"
+          exit 1
+        else
+          echo "Vibe check passed: $OVERALL meets threshold $THRESHOLD"
+        fi