paqad-ai 0.1.9 → 0.2.0

package/README.md

@@ -7,9 +7,9 @@
 ╚═╝     ╚═╝  ╚═╝ ╚══▀▀═╝ ╚═╝  ╚═╝╚═════╝
 ```
 
-**AI agents that think before they type.** paqad-ai reads your codebase, detects your stack, and generates everything your AI coding agents need to actually understand your project — docs, rules, security checks, MCP configs, and structured workflows. One command. No configuration.
+**AI agents that think before they type.** paqad-ai reads your codebase, detects your stack, builds optional hybrid RAG context, and generates the docs, rules, MCP configs, security checks, and workflows your coding agents need to operate with real project context.
 
-[![npm version](https://img.shields.io/npm/v/paqad-ai)](https://npmjs.com/package/paqad-ai) [![license](https://img.shields.io/npm/l/paqad-ai)](./LICENSE) [![website](https://img.shields.io/badge/website-paqad.ai-0A7A5C)](https://paqad.ai)
+[![npm version](https://img.shields.io/npm/v/paqad-ai)](https://npmjs.com/package/paqad-ai) [![license](https://img.shields.io/npm/l/paqad-ai)](./LICENSE) [![website](https://img.shields.io/badge/website-paqad.ai-0A7A5C)](https://paqad.ai) [![docs](https://img.shields.io/badge/docs-paqad.ai-1f6feb)](https://paqad.ai)
 
 ---
 
@@ -17,9 +17,11 @@
 
 You install an AI coding agent. You ask it to help with your Laravel + React project. It doesn't know your folder structure, conventions, test runner, or which version of anything you're running.
 
-So you spend an hour writing `CLAUDE.md` by hand. Then you do it again for Cursor. And again for Copilot. And again when your stack changes.
+Then the context problem gets worse: the agent still cannot retrieve the right files when the repo is large, ambiguous, or spread across app code, docs, workflows, and generated artifacts. It guesses. It misses the right modules. It burns tokens on the wrong files.
 
-**paqad-ai does all of this automatically — for 10 agents at once.**
+So you spend an hour writing `CLAUDE.md` by hand. Then you do it again for Cursor. And again for Copilot. And again when your stack changes. Then you still have to solve retrieval, documentation drift, and workflow consistency yourself.
+
+**paqad-ai solves both setup and retrieval: shared instructions for multiple external agent platforms, plus a built-in multi-agent runtime and optional hybrid RAG for real project context.**
 
 ---
 
@@ -37,9 +39,9 @@ paqad-ai reads your lockfiles, detects your stack, asks you to confirm, and gene
 ✓ Detected: laravel + react (inertia, tailwind, pest, docker)
 ✓ Generated: CLAUDE.md, AGENTS.md, ANTIGRAVITY.md, GEMINI.md, .cursor/rules/paqad.mdc
 ✓ Generated: .github/copilot-instructions.md, .windsurfrules
-✓ Generated: docs/instructions/stack/* (6 files)
-✓ Generated: docs/instructions/rules/*
-✓ Configured: MCP servers (figma, laravel-boost, tailwind-mcp)
+✓ Generated: docs/instructions/rules/* and stack-specific tool guides
+✓ Wrote: .paqad/project-profile.yaml, onboarding-manifest.json, detection-report.json
+✓ Configured: adapter entry files and MCP settings for selected platforms
 
 Ready. Your AI agents now understand your project.
 ```
@@ -52,11 +54,27 @@ npx paqad-ai onboard
 
 ---
 
-## 10 supported agents
+## Core features
+
+| Area                     | What you get                                                                         |
+| ------------------------ | ------------------------------------------------------------------------------------ |
+| **Agent onboarding**     | Shared instructions and MCP config for supported external agent platforms            |
+| **Stack detection**      | Lockfile-driven framework, trait, and archetype detection across 9 ecosystems        |
+| **Documentation**        | Stack, architecture, design-system, registry, and module documentation flows         |
+| **Hybrid RAG**           | Optional vector indexing, hybrid retrieval, reranking, evals, and benchmarks         |
+| **Security**             | OWASP-mapped pentest workflow with incremental retests                               |
+| **Built-in agent roles** | 20 internal specialist roles for routing, review, design, verification, and security |
+| **Context controls**     | Chunking, scoring, dedup, summarization, and budget enforcement                      |
+| **Pack system**          | Built-in and custom stack packs for framework-specific behavior                      |
+| **Operations**           | `doctor`, `refresh`, `update`, `patterns`, and capability toggles                    |
+
+---
+
+## Supported adapters
 
-One onboarding. Every major AI coding agent. Thin entry files that all point to the same shared instruction bundles:
+One onboarding. Thin entry files that all point to the same shared instruction bundles:
 
-| Agent                  | What it generates                                      |
+| Platform               | What it generates                                      |
 | ---------------------- | ------------------------------------------------------ |
 | **Claude Code**        | `CLAUDE.md` + MCP config                               |
 | **Codex CLI**          | `AGENTS.md` + MCP config                               |
@@ -69,13 +87,25 @@ One onboarding. Every major AI coding agent. Thin entry files that all point to
 | **Continue**           | `.continue/rules/paqad.md` + MCP, prompts              |
 | **Aider**              | `CONVENTIONS.md` (config-only)                         |
 
-Pick one or pick all — `paqad-ai onboard` lets you multi-select. If no `--providers` flag is passed, onboarding defaults to `claude-code`.
+Pick one or pick all. `paqad-ai onboard` lets you multi-select. If no `--providers` flag is passed, onboarding defaults to `claude-code`.
+
+## 20 built-in specialist agents
+
+Beyond adapter targets, paqad-ai also ships internal agent roles used by the framework's runtime content, reviews, workflows, and specialized guidance.
+
+| Group                    | Count | Examples                                                                     |
+| ------------------------ | ----- | ---------------------------------------------------------------------------- |
+| **Base workflow agents** | 11    | `router`, `verifier`, `story-designer`, `test-planner`, `product-owner`      |
+| **Coding specialists**   | 8     | `solution-architect`, `database-expert`, `devops-engineer`, `doc-maintainer` |
+| **Security specialists** | 1     | `security-auditor`                                                           |
+
+That is **20 built-in agent roles** in the shipped runtime, in addition to the supported external adapters above.
 
 ---
 
 ## 17 built-in Stack Packs
 
-paqad-ai parses **26 manifest and lockfile formats** across 9 ecosystems. It doesn't ask what stack you use — it reads your lockfiles to figure it out.
+paqad-ai parses manifests and lockfiles across 9 ecosystems. It doesn't ask what stack you use — it reads your repo to figure it out.
 
 **14 framework packs:**
 
@@ -85,7 +115,7 @@ paqad-ai parses **26 manifest and lockfile formats** across 9 ecosystems. It doe
 
 `node-cli` · `node-library` · `node-service`
 
-Each pack is a declarative `pack.yaml` that drives detection, documentation, MCP configuration, security mappings, and more. Don't see your framework? [Author your own pack](./docs/framework/authoring-stack-packs.md).
+Each pack is a declarative `pack.yaml` that drives detection, documentation, MCP configuration, security mappings, and more. Don't see your framework? [Author your own pack](./docs/modules/packs/features/authoring/technical.md).
 
 **What it reads:**
 
@@ -129,6 +159,17 @@ Laravel also detects `pest`, `phpunit`, `sail`, `docker`, and `compose` automati
 
 paqad-ai isn't just a setup tool. Here's the full feature set:
 
+### Stack intelligence and pack resolution
+
+Detection is lockfile-first and feeds the rest of the framework. Onboarding, refresh, docs, MCP config, and security workflows all consume the same resolved stack profile and effective pack set.
+
+| System                  | What it does                                                                     |
+| ----------------------- | -------------------------------------------------------------------------------- |
+| **Framework detection** | Detects frameworks, traits, and archetypes from manifests, lockfiles, and config |
+| **Pack resolution**     | Merges built-in, global, and project pack overrides by precedence                |
+| **Capability routing**  | Activates `content`, `coding`, and `security` based on the effective stack       |
+| **Drift tracking**      | Stores stack snapshots and reports when the live repo diverges from onboarding   |
+
 ### Documentation as a first-class output
 
 Docs aren't an afterthought — they're a primary framework output. There's no standalone `document` command. Instead, ask any connected agent to `create documentation` and the workflow:
@@ -183,6 +224,24 @@ Your AI agent has a limited context window. paqad-ai makes every token count:
 | **Turn summarizer**  | Old turns → structured extracts (decisions, files, blockers, next steps). No LLM, pure regex |
 | **Stream truncator** | Per-tier output limits (fast: 2K, medium: 5K, reasoning: 15K) with sentence-boundary cuts    |
 
+### Optional hybrid RAG retrieval
+
+RAG is optional, but when enabled it plugs directly into the semantic loader instead of living off to the side as a separate experiment.
+
+| Feature                 | What it does                                                                                |
+| ----------------------- | ------------------------------------------------------------------------------------------- |
+| **Vector index**        | Builds an AST-aware code/document index under `.paqad/vectors/` from the chunk index        |
+| **Embedding providers** | Supports `local` (default), `openai`, and `voyageai` with provider-specific default models  |
+| **Hybrid scoring**      | Fuses vector similarity, keyword overlap, symbol hits, and file-path proximity              |
+| **Adaptive depth**      | Routes retrieval as `none`, `standard`, or `deep` based on task complexity, risk, and scope |
+| **Metadata filtering**  | Narrows candidates by extension, module path, framework, and recency with safe fallbacks    |
+| **Reranking**           | Supports `passthrough`, local cross-encoder reranking, or Cohere reranking                  |
+| **Pattern vectors**     | Extends retrieval with vectorized cross-project patterns from `~/.paqad/patterns/`          |
+| **Audit + fallbacks**   | Logs build/fallback/provider mismatch events to `.paqad/audit.log`                          |
+| **Eval gates**          | Benchmarks hit@5, task success, correction turns, and prompt-token deltas                   |
+
+RAG configuration lives in `.paqad/project-profile.yaml`. Project-specific corpus exclusions can be tuned with `.paqad/rag.ignore.yaml`, and remote provider keys are stored in `.paqad/secrets.env`.
+
 ### Smart session handoff
 
 Structured v2 handoff captures active task, decisions, files, blockers, and next steps. **70–85% smaller** than raw dumps. Both JSON and Markdown.
@@ -217,6 +276,18 @@ Conditional steps, parallel execution, failure handling (`abort` / `skip` / `ret
 
 Verified solutions stored globally at `~/.paqad/patterns/`. Scored by framework overlap + keyword match, staleness penalty after 180 days.
 
+### Health, refresh, and update workflows
+
+The operational side matters too: paqad-ai keeps generated artifacts and project state from quietly drifting.
+
+| Workflow         | What it does                                                                        |
+| ---------------- | ----------------------------------------------------------------------------------- |
+| **Doctor**       | Validates framework artifacts, copied instructions, MCP config, docs, and RAG state |
+| **Refresh**      | Re-detects stack and regenerates stack or design-system outputs                     |
+| **Update**       | Rewrites framework-managed artifacts against the current package version            |
+| **Capabilities** | Toggles active capability layers with dependency rules                              |
+| **Patterns**     | Lists, prunes, and exports the global pattern library                               |
+
 ---
 
 ## The capability model
@@ -338,68 +409,103 @@ paqad-ai packs validate <path>   # Validate a pack before installing
 paqad-ai packs create <name>     # Scaffold a new pack
 ```
 
-### Security workflows
+### `paqad-ai rag`
+
+Optional RAG management, indexing, and benchmark tooling.
 
 ```bash
-paqad-ai pentest                           # Full scan
-paqad-ai pentest --incremental             # Only changed files
-paqad-ai pentest-retest --source <report>  # Re-check prior findings
+paqad-ai rag init                          # Enable RAG and build the initial vector index
+paqad-ai rag init --provider local         # Local embeddings via Xenova/all-MiniLM-L6-v2
+paqad-ai rag init --provider openai        # Remote embeddings via text-embedding-3-small
+paqad-ai rag init --provider voyageai      # Remote embeddings via voyage-code-3
+paqad-ai rag rebuild                       # Force a full rebuild
+paqad-ai rag status                        # Show provider, model, validity, size, chunk count
+paqad-ai rag clear --yes                   # Delete the vector index and disable RAG
+paqad-ai rag eval                          # Run deterministic evals on the current index
+paqad-ai rag eval --mode lexical-vs-rag
+paqad-ai rag eval --baseline ./baseline.json --mode rag-vs-candidate
+paqad-ai rag eval --model-graded           # Also run the optional model-graded lane
 ```
 
+**What `rag init` configures:**
+
+- Builds a vector index from the existing chunk index and enables `intelligence.rag_enabled`
+- Persists provider/model selection in `.paqad/project-profile.yaml`
+- Uses a free local model by default, or prompts for `OPENAI_API_KEY` / `VOYAGE_API_KEY` when needed
+- Reuses cached local models under `~/.paqad/models` to avoid repeat downloads
+
+**What `rag eval` measures:**
+
+- `hit_at_5`
+- `task_success_rate`
+- `correction_turns`
+- `prompt_tokens_sent`
+
+Benchmark comparisons support `lexical-vs-rag`, `rag-vs-candidate`, and `feature-off-vs-on`.
+
+### Security workflows
+
+Security and retest flows are part of the framework runtime and agent workflows in this package version. They are not exposed as standalone top-level CLI commands in `paqad-ai` today.
+
 ### Pattern library
 
 ```bash
-paqad-ai patterns list [--domain <d>] [--framework <f>] [--category <c>]
+paqad-ai patterns list [--domain <d>] [--frameworks <f1,f2>] [--category <c>]
 paqad-ai patterns prune [--older-than <days>]
 paqad-ai patterns export <output> [--format json|markdown]
 ```
 
 ---
 
-## What it generates
+## What paqad-ai manages
 
-```
+```text
 your-project/
-├── CLAUDE.md                              # → docs/instructions/
-├── AGENTS.md                              # → docs/instructions/
-├── ANTIGRAVITY.md                         # → docs/instructions/
-├── GEMINI.md                              # → docs/instructions/
-├── CONVENTIONS.md                         # Aider
-├── .junie/AGENTS.md                       # → docs/instructions/
-├── .cursor/rules/paqad.mdc               # + MCP, skills, agents
-├── .github/copilot-instructions.md        # + .vscode/mcp.json
-├── .windsurfrules                         # + MCP, skills, agents
-├── .continue/rules/paqad.md               # + MCP, prompts
+├── CLAUDE.md                              # onboarding, if Claude Code is selected
+├── AGENTS.md                              # onboarding, if Codex CLI is selected
+├── ANTIGRAVITY.md                         # onboarding, if Antigravity is selected
+├── GEMINI.md                              # onboarding, if Gemini CLI is selected
+├── CONVENTIONS.md                         # onboarding, if Aider is selected
+├── .junie/AGENTS.md                       # onboarding, if Junie is selected
+├── .cursor/rules/paqad.mdc               # onboarding, if Cursor is selected
+├── .github/copilot-instructions.md        # onboarding, if GitHub Copilot is selected
+├── .windsurfrules                         # onboarding, if Windsurf is selected
+├── .continue/rules/paqad.md               # onboarding, if Continue is selected
 │
 ├── docs/
 │   ├── instructions/
-│   │   ├── stack/                         # Auto-generated stack docs
+│   │   ├── stack/                         # generated later by refresh/documentation workflows
 │   │   │   ├── overview.md
 │   │   │   ├── frameworks.md
 │   │   │   ├── dependencies.md
 │   │   │   ├── tooling.md
 │   │   │   ├── version-rules.md
+│   │   │   ├── sources.md
 │   │   │   └── drift-report.md
-│   │   ├── rules/                         # Conventions, writing style
-│   │   ├── tools/                         # Stack-specific tool guides
-│   │   └── workflows/                     # Your custom YAML templates
-│   ├── modules/                           # Per-feature business + technical docs
-│   └── pentest/                           # Security reports + retests
+│   │   ├── rules/                         # onboarding-generated conventions and writing rules
+│   │   ├── tools/                         # onboarding-generated stack tool guides
+│   │   └── workflows/                     # user-authored workflow templates
+│   ├── modules/                           # generated later by documentation workflows
+│   └── pentest/                           # generated later by security workflows
 │
 └── .paqad/                                # Framework metadata (machine-managed)
     ├── project-profile.yaml
     ├── stack-snapshot.json
     ├── stack-drift.json
     ├── onboarding-manifest.json
+    ├── rag.ignore.yaml                    # Optional RAG corpus include/exclude rules
+    ├── secrets.env                        # Optional API keys for remote RAG providers
+    ├── audit.log                          # Build, fallback, and provider mismatch events
     ├── doc-progress.json
-    ├── pentest/runs/
+    ├── pentest/runs/                      # generated when pentest workflows run
     ├── session/                           # Handoff, budget, dedup stats
     ├── context/                           # Chunk index, load stats
+    ├── vectors/                           # Optional RAG vector index + metadata
     ├── cache/                             # Transition log, metrics
     └── workflows/                         # Custom workflow run state
 ```
 
-Entry files stay thin. Knowledge lives in shared instruction bundles that all 10 agents read.
+Entry files stay thin. Knowledge lives in shared instruction bundles that external platforms and built-in runtime roles consume consistently.
 
 ---
 
@@ -411,11 +517,13 @@ Entry files stay thin. Knowledge lives in shared instruction bundles that all 10
 | **Health check**     | `paqad-ai doctor`                      | Validate artifacts, config, docs                          |
 | **Refresh**          | `paqad-ai refresh`                     | Re-detect stack, update derived docs                      |
 | **Update**           | `paqad-ai update`                      | Regenerate framework files after version bump             |
+| **RAG**              | `paqad-ai rag ...`                     | Build, inspect, clear, and benchmark hybrid retrieval     |
 | **Documentation**    | Ask agent: _"create documentation"_    | Stack → architecture → design system → modules            |
 | **Pentest**          | Ask agent: _"run pentest"_             | 5-step scan → findings → report                           |
 | **Retest**           | Ask agent: _"retest pentest"_          | Replay findings → fixed / still-open / needs-verification |
 | **Capabilities**     | `paqad-ai capabilities add/remove`     | Toggle content / coding / security                        |
 | **Pack management**  | `paqad-ai packs install/remove`        | Install, validate, scaffold stack packs                   |
+| **Pattern library**  | `paqad-ai patterns ...`                | Query, prune, and export reusable solutions               |
 | **Custom workflows** | YAML at `docs/instructions/workflows/` | Your own resumable skill sequences                        |
 
 ---
@@ -433,16 +541,20 @@ Entry files stay thin. Knowledge lives in shared instruction bundles that all 10
 
 ```bash
 pnpm install
-pnpm run ci        # typecheck → lint → format → test (80%+ coverage) → build
+npm run format     # required before completion
+npm run ci         # required before completion
 ```
 
 ```bash
 pnpm run test       # Tests only
+pnpm run test:coverage  # Coverage gate summary
 pnpm run typecheck  # TypeScript checks
 pnpm run lint       # ESLint
 pnpm run build      # Production build
 ```
 
+Every completed change in this repo is expected to keep 100% lines, 100% statements, 100% functions, and 100% branches coverage, add positive and negative E2E coverage for changed user flows, and update the owning `docs/modules/**` plus `website/` surfaces when their contract changes.
+
 ---
 
 ## Contributing
@@ -451,4 +563,4 @@ Coming soon
 
 ---
 
-MIT License · [npm](https://npmjs.com/package/paqad-ai) · [Docs](./docs/framework/specifications.md)
+MIT License · [npm](https://npmjs.com/package/paqad-ai) · [Docs](https://paqad.ai) · [Maintainer Overview](./docs/maintainers/project-overview.md) · [Modules](./docs/modules/README.md)