llmsbrieftxt 1.3.1__tar.gz → 1.11.0__tar.gz

llmsbrieftxt-1.11.0/.github/copilot-instructions.md

@@ -0,0 +1,115 @@
+# GitHub Copilot Instructions for llmsbrieftxt
+
+## Project Overview
+
+This is `llmsbrieftxt`, a Python package that generates llms-brief.txt files by crawling documentation websites and using OpenAI to create structured descriptions. The CLI command is `llmtxt` (not `llmsbrieftxt`).
+
+## Architecture and Code Patterns
+
+### Async-First Design
+All main functions use async/await patterns. Use `asyncio.gather()` for concurrent operations and semaphore control for rate limiting. The processing pipeline flows: URL Discovery → Content Extraction → LLM Summarization → File Generation.
+
+### Module Organization
+- **cli.py**: Simple CLI with positional URL argument (no subcommands)
+- **main.py**: Orchestrates the async generation pipeline
+- **crawler.py**: RobustDocCrawler for breadth-first URL discovery
+- **doc_loader.py**: DocLoader wraps crawler with document loading
+- **extractor.py**: HTML to markdown via trafilatura
+- **summarizer.py**: OpenAI integration with retry logic (tenacity)
+- **url_utils.py**: URLNormalizer for deduplication
+- **url_filters.py**: Filter non-documentation URLs
+- **schema.py**: Pydantic models (PageSummary)
+- **constants.py**: Configuration constants
+
+### Type Safety
+Use Pydantic models for all structured data. The OpenAI integration uses structured output with the PageSummary model.
+
+### Error Handling
+Failed URL loads should be logged but not stop processing. LLM failures use exponential backoff retries via tenacity. Never let one failure break the entire pipeline.
+
+## Development Practices
+
+### Testing Requirements
+Write tests before implementing features. Use pytest with these markers:
+- `@pytest.mark.unit` for fast, isolated tests
+- `@pytest.mark.requires_openai` for tests needing OPENAI_API_KEY
+- `@pytest.mark.slow` for tests making external API calls
+
+Tests go in:
+- `tests/unit/` for fast tests with no external dependencies
+- `tests/integration/` for tests requiring OPENAI_API_KEY
+
+### Code Quality Tools
+Before committing, always run:
+1. Format: `uv run ruff format llmsbrieftxt/ tests/`
+2. Lint: `uv run ruff check llmsbrieftxt/ tests/`
+3. Type check: `uv run pyright llmsbrieftxt/`
+4. Tests: `uv run pytest tests/unit/`
+
+### Package Management
+Use `uv` for all package operations:
+- Install: `uv sync --group dev`
+- Add dependency: `uv add package-name`
+- Build: `uv build`
+
+## Design Philosophy
+
+### Unix Philosophy
+This project follows "do one thing and do it well":
+- Generate llms-brief.txt files only (no built-in search/list features)
+- Compose with standard Unix tools (rg, grep, ls)
+- Simple CLI: URL is a positional argument, no subcommands
+- Plain text output for scriptability
+
+### Simplicity Over Features
+Avoid adding functionality that duplicates mature Unix tools. Every line of code must serve the core mission of generating llms-brief.txt files.
+
+## Configuration Defaults
+
+- **Crawl Depth**: 3 levels (hardcoded in crawler.py)
+- **Output**: `~/.claude/docs/<domain>.txt` (override with `--output`)
+- **Cache**: `.llmsbrieftxt_cache/` for intermediate results
+- **OpenAI Model**: `gpt-5-mini` (override with `--model`)
+- **Concurrency**: 10 concurrent LLM requests (prevents rate limiting)
+
+## Commit Convention
+
+Use conventional commits for automated versioning:
+- `fix:` → patch bump (1.0.0 → 1.0.1)
+- `feat:` → minor bump (1.0.0 → 1.1.0)
+- `BREAKING CHANGE` or `feat!:`/`fix!:` → major bump (1.0.0 → 2.0.0)
+
+Examples:
+```bash
+git commit -m "fix: handle empty sitemap gracefully"
+git commit -m "feat: add --depth option for custom crawl depth"
+git commit -m "feat!: change default output location"
+```
+
+## Non-Obvious Behaviors
+
+1. URL Discovery discovers ALL pages up to depth 3, not just direct links
+2. URLs like `/page`, `/page/`, and `/page#section` are deduplicated as the same URL
+3. Summaries are automatically cached in `.llmsbrieftxt_cache/summaries.json`
+4. Content extraction uses trafilatura to preserve HTML structure in markdown
+5. File I/O is synchronous (uses standard `Path.write_text()` for simplicity)
+
+## Known Limitations
+
+1. Only supports OpenAI API (no other LLM providers)
+2. Crawl depth is hardcoded to 3 in crawler.py
+3. No CLI flag to force resume from cache (though cache exists)
+4. No progress persistence if interrupted
+5. Prompts and parsing assume English documentation
+
+## Code Review Checklist
+
+When reviewing code changes:
+- Ensure async patterns are used correctly (no blocking I/O in async functions)
+- Verify all functions have type hints
+- Check that tests are included for new functionality
+- Confirm error handling doesn't break the pipeline
+- Validate that conventional commit format is used
+- Ensure code follows Unix philosophy (simplicity, composability)
+- Check that ruff and pyright pass without errors
+- **IMPORTANT**: Always include specific file names and line numbers when providing review feedback (e.g., "main.py:165" or "line 182 in cli.py")

llmsbrieftxt-1.11.0/.github/workflows/claude-cli-qa.yml

@@ -0,0 +1,296 @@
+name: Claude CLI QA Tests
+
+on:
+  pull_request:
+    types: [opened, synchronize, reopened, ready_for_review]
+    branches: [main]
+    paths:
+      - 'llmsbrieftxt/**'
+      - 'tests/**'
+      - 'pyproject.toml'
+      - 'uv.lock'
+      - '.github/workflows/claude-cli-qa.yml'
+  workflow_dispatch:
+    inputs:
+      branch:
+        description: 'Branch to test (leave empty for current branch)'
+        required: false
+        type: string
+      test_scope:
+        description: 'Test scope (quick, standard, thorough)'
+        required: false
+        default: 'standard'
+        type: choice
+        options:
+          - quick
+          - standard
+          - thorough
+      single_test_case:
+        description: 'Run single test case (e.g., TS-001). Leave empty to run full test scope.'
+        required: false
+        type: choice
+        options:
+          - ''
+          - TS-001
+          - TS-002
+          - TS-003
+          - TS-004A
+          - TS-004B
+          - TS-005
+          - TS-006
+          - TS-007
+          - TS-008
+          - TS-009
+          - TS-010
+          - TS-011
+          - TS-012
+
+jobs:
+  cli-qa-tests:
+    name: Agent-Based CLI QA
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    permissions:
+      contents: read
+      pull-requests: write
+      issues: read
+      id-token: write
+    env:
+      OLLAMA_MODEL: gemma3:270m
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.inputs.branch || github.head_ref || github.ref }}
+          fetch-depth: 1
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+
+      - name: Install uv
+        run: |
+          curl -LsSf https://astral.sh/uv/install.sh | sh
+          echo "$HOME/.cargo/bin" >> $GITHUB_PATH
+
+      - name: Install package with dependencies
+        run: |
+          uv sync --all-groups
+
+      - name: Install Ollama
+        run: |
+          echo "Installing Ollama for local LLM testing..."
+          curl -fsSL https://ollama.com/install.sh | sh
+
+          # Verify installation
+          ollama --version
+
+      - name: Start Ollama service
+        run: |
+          echo "Starting Ollama service..."
+          ollama serve > ollama.log 2>&1 &
+          echo $! > ollama.pid
+
+          # Wait for Ollama to be ready
+          echo "Waiting for Ollama to be ready..."
+          timeout 60 bash -c 'until curl -s http://localhost:11434/api/tags > /dev/null; do sleep 2; done'
+
+          echo "✓ Ollama service is running"
+
+      - name: Pull Ollama model
+        run: |
+          echo "Pulling ${{ env.OLLAMA_MODEL }} model for testing..."
+          ollama pull ${{ env.OLLAMA_MODEL }}
+
+          echo "✓ Model pulled successfully"
+
+      - name: Verify Ollama setup
+        run: |
+          echo "Verifying Ollama models..."
+          ollama list
+
+          echo "✓ Model ready"
+
+      - name: Verify CLI installation
+        run: |
+          # Ensure CLI is accessible
+          uv run llmtxt --help
+
+          echo "✓ CLI installed and accessible"
+
+      - name: Determine test scope
+        id: test-scope
+        run: |
+          SCOPE="${{ github.event.inputs.test_scope || 'standard' }}"
+          echo "scope=$SCOPE" >> $GITHUB_OUTPUT
+
+          case "$SCOPE" in
+            quick)
+              echo "max_urls=2" >> $GITHUB_OUTPUT
+              echo "depth=1" >> $GITHUB_OUTPUT
+              ;;
+            thorough)
+              echo "max_urls=5" >> $GITHUB_OUTPUT
+              echo "depth=1" >> $GITHUB_OUTPUT
+              ;;
+            *)
+              echo "max_urls=3" >> $GITHUB_OUTPUT
+              echo "depth=1" >> $GITHUB_OUTPUT
+              ;;
+          esac
+
+      - name: Determine test argument
+        id: test-arg
+        run: |
+          # If single test case is specified, use that; otherwise use test scope
+          if [ -n "${{ github.event.inputs.single_test_case }}" ]; then
+            echo "arg=${{ github.event.inputs.single_test_case }}" >> $GITHUB_OUTPUT
+          else
+            # Default to 'standard' for PR events, or use the specified scope
+            SCOPE="${{ github.event.inputs.test_scope || 'standard' }}"
+            echo "arg=$SCOPE" >> $GITHUB_OUTPUT
+          fi
+
+      - name: Run Claude CLI QA Agent
+        id: claude-cli-qa
+        uses: anthropics/claude-code-action@v1
+        with:
+          claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
+
+          # Allow Claude bot to trigger this workflow
+          allowed_bots: '*'
+
+          # Use Haiku model for faster/cheaper testing
+          claude_args: |
+            --model claude-haiku-4-5-20251001
+            --allowedTools "Bash,Read,Write,Glob,Grep,TodoWrite,Task,SlashCommand"
+
+          # Enable progress tracking for PR events only
+          track_progress: ${{ github.event_name == 'pull_request' }}
+
+          # Show full output in workflow logs for debugging
+          show_full_output: true
+
+          prompt: |
+            Run the QA test suite using the slash command: /qa-test ${{ steps.test-arg.outputs.arg }}
+
+      - name: Validate test execution
+        id: validate-tests
+        if: always()
+        run: |
+          echo "=== Validating CLI QA Test Results ==="
+
+          # Check if CLI is accessible via uv run
+          if ! uv run llmtxt --help &> /dev/null; then
+            echo "❌ ERROR: llmtxt CLI not accessible via 'uv run'"
+            exit 1
+          fi
+
+          echo "✅ CLI validation passed (accessible via 'uv run llmtxt')"
+
+      - name: Evaluate test results
+        id: evaluate-results
+        if: always()
+        run: |
+          echo "=== Evaluating Test Results ==="
+
+          # Check Claude agent execution status
+          if [ "${{ steps.claude-cli-qa.outcome }}" = "failure" ]; then
+            echo "❌ Claude QA agent failed to complete"
+            echo "status=failure" >> $GITHUB_OUTPUT
+            exit 1
+          fi
+
+          # Parse output for test failures (these patterns match the prompt output)
+          CLAUDE_OUTPUT="${{ steps.claude-cli-qa.outputs.text }}"
+
+          # Check for critical failures in output
+          if echo "$CLAUDE_OUTPUT" | grep -q "NO-GO\|❌ FAIL\|CRITICAL"; then
+            echo "❌ Test failures detected in QA results"
+            echo "status=failure" >> $GITHUB_OUTPUT
+            exit 1
+          fi
+
+          # Check for test summary with failures
+          if echo "$CLAUDE_OUTPUT" | grep -q "FAIL.*\|Failed:.*[1-9]"; then
+            echo "⚠️  Some tests failed but not critical - checking severity"
+            if echo "$CLAUDE_OUTPUT" | grep -q "Exit Code Bug\|returns exit code 0\|CRITICAL"; then
+              echo "❌ Critical test failures detected"
+              echo "status=failure" >> $GITHUB_OUTPUT
+              exit 1
+            fi
+          fi
+
+          echo "✅ All critical tests passed"
+          echo "status=success" >> $GITHUB_OUTPUT
+
+      - name: Generate test summary
+        if: always()
+        env:
+          CLAUDE_STATUS: ${{ steps.claude-cli-qa.outcome }}
+          EVALUATION_STATUS: ${{ steps.evaluate-results.outputs.status }}
+          GITHUB_SERVER_URL: ${{ github.server_url }}
+          GITHUB_REPOSITORY: ${{ github.repository }}
+          GITHUB_RUN_ID: ${{ github.run_id }}
+        run: |
+          cat <<'EOF' >> $GITHUB_STEP_SUMMARY
+          ## llmtxt CLI QA Test Summary
+
+          **Test Mode**: ${{ github.event.inputs.single_test_case != '' && format('🔍 Single Test (Troubleshooting) - {0}', github.event.inputs.single_test_case) || format('📦 Full Test Suite - {0}', steps.test-arg.outputs.arg) }}
+          **Status**: ${{ steps.evaluate-results.outputs.status == 'success' && '✅ All Tests Passed' || '❌ Tests Failed' }}
+
+          ### Environment
+          - Python: 3.11
+          - Package Manager: uv
+          - LLM Provider: Ollama (${{ env.OLLAMA_MODEL }})
+          - Max URLs: ${{ steps.test-scope.outputs.max_urls }}
+          - Crawl Depth: ${{ steps.test-scope.outputs.depth }}
+
+          ### Test Categories
+          - Basic Functionality Tests
+          - CLI Flags and Options Tests
+          - Cache Behavior Tests
+          - Error Handling Tests
+          - Output Format Validation
+
+          ### Results
+          ${{ github.event.inputs.single_test_case != '' && format('The CLI QA agent executed test case **{0}** for troubleshooting.', github.event.inputs.single_test_case) || 'The CLI QA agent executed comprehensive tests using the `cli-qa-tester` agent.' }}
+          Detailed test report and findings are available in the workflow logs.
+
+          **View full logs**: [${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
+
+          ---
+
+          🤖 **Agent-Based Testing**: This workflow uses Claude Code agents to perform intelligent,
+          adaptive CLI testing without traditional E2E test frameworks.
+          EOF
+
+      - name: Upload test artifacts
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: cli-qa-artifacts
+          path: |
+            ~/.claude/docs/
+            .llmsbrieftxt_cache/
+            ollama.log
+            **/test-*.txt
+            **/test-*.log
+          if-no-files-found: warn
+          retention-days: 7
+
+      - name: Fail workflow if tests failed
+        if: always() && steps.evaluate-results.outputs.status == 'failure'
+        run: |
+          echo "❌ CLI QA tests have critical failures - blocking PR"
+          exit 1
+
+      - name: Cleanup
+        if: always()
+        run: |
+          if [ -f ollama.pid ]; then
+            kill $(cat ollama.pid) || true
+          fi
+          echo "✓ Cleanup complete"