codebase_index 0.2.1 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8ba2df9baa16005b8f3981639c0e1bca59bbff3382c1cd483e2a686d399054f4
-  data.tar.gz: '0983e4f7e63febbe63631bff76caab5d11cc88dec9e360eab2a3ea33f6adb025'
+  metadata.gz: 982e7949df0e0db9249705ab9f009121c3c8156582c63712f0613fccc998337d
+  data.tar.gz: 4fb41c658901cd26606e44164da7059a7d62aa39c795a682020cfbb6252311be
 SHA512:
-  metadata.gz: 25af1daf07cdbdbf810919db95f8dd9535b79c784e6fc74aad5885931221ca9c389507fd67eb1a1e5ec2cb3f5415fc111c4946db27a29df1882975a0741eb2f8
-  data.tar.gz: 5a9e35f69d822a8cc0e12a82a0d33fad0fdc6fac61d65c88e4c00ccfe8fbda5b7f7bcde0a82f7051a3238e43b60f1a8143dc35a28486fbc8e6818770b9d98693
+  metadata.gz: 6b62fe0a1d8b0db683744214461ec5d0029e41cf4538b7313ce2701a3a985bf9b1d06ce955acb225db09e93aaa72bcbc5e3b40cd534f7d682d98d190a670d722
+  data.tar.gz: f0a948295982aa85951fa8cca96cc8c30b317176a53424fe18e50cc1d3e28b17df5fc9dfb6a191e6450a1318b3b0f81a0f2cdf20fe7326c0c4e492a7f8b47f70

data/CHANGELOG.md

@@ -5,6 +5,66 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.1] - 2026-03-04
+
+### Fixed
+
+- **Gemspec version** now reads from `version.rb` instead of being hardcoded — prevents version mismatch during gem builds
+- **Release workflow** replaced `rake release` (fails on tag-triggered detached HEAD) with `gem build` + `gem push`
+
+## [0.3.0] - 2026-03-04
+
+### Added
+
+- **Redis/SolidCache caching layer** for retrieval pipeline with TTL, namespace isolation, and nil-caching
+- **Engine classification** — engines tagged as `:framework` or `:application` based on install path (handles Docker vendor paths)
+- **Graph analysis staleness tracking** — `generated_at` timestamp and `graph_sha` for detecting stale analysis
+- **Docker setup guide** (`docs/DOCKER_SETUP.md`) — split architecture, volume mounts, bridge mode, troubleshooting
+- **Context7 documentation suite** — 10 new user-facing docs optimized for AI retrieval: FAQ, Troubleshooting, Architecture, Extractor Reference, WHY CodebaseIndex, MCP Tool Cookbook, and 3 Context7 skills
+- **`context7.json`** configuration for controlling Context7 indexing scope
+
+### Fixed
+
+- **Vendor path leak** in source file resolution across 9 extractors — framework gems under `vendor/bundle` no longer produce empty source
+- **Prism cross-version compatibility** — handle API differences between Prism versions
+- **`schema_sha`** now supports `db/structure.sql` fallback (not just `db/schema.rb`)
+- **ViewComponent extractor** skips framework-internal components with no resolvable source file
+- **HTTP connection reuse** and retry handling in embedding providers
+- **DependencyGraph `to_h`** returns a dup to prevent cache pollution
+- **MCP tool counts** corrected across all documentation (27 index / 31 console)
+- **TROUBLESHOOTING.md** corrected: `config.extractors` controls retrieval scope, not which extractors run
+
+### Changed
+
+- **README streamlined** from 620 to 325 lines — added Quick Start, Documentation table; removed verbose sections in favor of links to dedicated docs
+- **Internal rake tasks** (`retrieve`, `self_analyze`) hidden from `rails -T`
+- **Estimated tokens memoization** removed to prevent stale values after source changes
+- **Simplification sweep** — dead code removal, shared helper extraction, bug fixes across caching and retrieval layers
+
+### Performance
+
+- Critical hotspots fixed across extraction, storage, and retrieval pipelines
+- `fetch_key` optimization for falsy value handling in cache layer
+
+## [0.2.1] - 2026-02-19
+
+### Changed
+
+- Switch release workflow to RubyGems trusted publishing
+
+## [0.2.0] - 2026-02-19
+
+### Added
+
+- **Embedded console MCP server** for zero-config Rails querying (no bridge process needed)
+- **Console MCP setup guide** (`docs/CONSOLE_MCP_SETUP.md`) — stdio, Docker, HTTP/Rack, SSH bridge options
+- **CODEOWNERS** and issue template configuration
+
+### Fixed
+
+- MCP gem compatibility and symbol key handling in embedded executor
+- Duplicate URI warning in gemspec
+
 ## [0.1.0] - 2026-02-18
 
 ### Added

data/README.md

@@ -18,6 +18,21 @@ CodebaseIndex solves this by:
 - **Tracking dependencies** bidirectionally so you can trace impact across the codebase
 - **Enriching with git data** so you know what's actively changing vs. dormant
 
+See [Why CodebaseIndex?](docs/WHY_CODEBASE_INDEX.md) for concrete before/after examples.
+
+## Quick Start
+
+```bash
+# Add to your Rails app's Gemfile, then:
+bundle install
+rails generate codebase_index:install
+bundle exec rake codebase_index:extract
+bundle exec rake codebase_index:stats
+# Add the MCP server to .mcp.json (see below) and start asking questions
+```
+
+See [Getting Started](docs/GETTING_STARTED.md) for the full walkthrough including Docker, storage presets, and CI setup.
+
 ## Installation
 
 Add to your Gemfile:
@@ -30,15 +45,26 @@ Then:
 
 ```bash
 bundle install
+rails generate codebase_index:install
+rails db:migrate
+```
+
+Create a minimal configuration:
+
+```ruby
+# config/initializers/codebase_index.rb
+CodebaseIndex.configure do |config|
+  config.output_dir = Rails.root.join('tmp/codebase_index')
+end
 ```
 
-Or install directly:
+Or install the gem directly:
 
 ```bash
 gem install codebase_index
 ```
 
-> **Requires Rails.** Extraction runs inside a booted Rails application using runtime introspection (`ActiveRecord::Base.descendants`, `Rails.application.routes`, etc.). The gem cannot extract from source files alone. See [Getting Started](docs/GETTING_STARTED.md) for setup.
+> **Requires Rails.** Extraction runs inside a booted Rails application using runtime introspection (`ActiveRecord::Base.descendants`, `Rails.application.routes`, etc.). The gem cannot extract from source files alone. See [Getting Started](docs/GETTING_STARTED.md) for full setup details.
 
 ## Target Environment
 
@@ -99,85 +125,19 @@ Extraction runs inside the Rails application (via rake task) to access runtime i
 
 ### Extractors (34)
 
-**Core Application**
-
-| Extractor | What it captures |
-|-----------|-----------------|
-| **ModelExtractor** | Schema (columns, indexes, FKs), associations, validations, callbacks (all 13 types), scopes, enums, inlined concerns. Chunks large models into summary/associations/callbacks/validations. |
-| **ControllerExtractor** | Route mapping (verb → path → action), filter chains per action, response formats, permitted params. Per-action chunks with applicable filters and route context. |
-| **ServiceExtractor** | Scans `app/services`, `app/interactors`, `app/operations`, `app/commands`, `app/use_cases`. Entry points, dependency injection, custom errors, return type inference. |
-| **JobExtractor** | ActiveJob and Sidekiq workers. Queue config, retry/concurrency options, perform arguments, callbacks. |
-| **MailerExtractor** | ActionMailer classes with defaults, per-action templates, callbacks, helper usage. |
-| **ConfigurationExtractor** | Rails initializers from `config/initializers` and `config/environments`, plus behavioral profile from resolved `Rails.application.config`. |
-| **RouteExtractor** | All Rails routes via runtime introspection of `Rails.application.routes`. |
-| **MiddlewareExtractor** | Rack middleware stack as a single ordered unit. |
-
-**UI Components**
-
-| Extractor | What it captures |
-|-----------|-----------------|
-| **PhlexExtractor** | Phlex component slots, initialize params, sub-components, Stimulus controller references, route helpers. |
-| **ViewComponentExtractor** | ViewComponent slots, template paths, preview classes, collection support. |
-| **ViewTemplateExtractor** | ERB view templates with render calls, instance variables, helper usage. |
-| **DecoratorExtractor** | Decorators, presenters, and form objects from `app/decorators`, `app/presenters`, `app/form_objects`. |
-
-**Data Layer**
-
-| Extractor | What it captures |
-|-----------|-----------------|
-| **ConcernExtractor** | ActiveSupport::Concern modules from `app/models/concerns` and `app/controllers/concerns`. |
-| **PoroExtractor** | Plain Ruby objects in `app/models` (non-ActiveRecord classes, excluding concerns). |
-| **SerializerExtractor** | ActiveModelSerializers, Blueprinter, Alba, and Draper. Auto-detects loaded serialization gems. |
-| **ValidatorExtractor** | Custom ActiveModel validator classes with validation rules. |
-| **ManagerExtractor** | SimpleDelegator subclasses — wrapped model, public methods, delegation chain. |
-
-**API & Authorization**
-
-| Extractor | What it captures |
-|-----------|-----------------|
-| **GraphQLExtractor** | graphql-ruby types, mutations, queries, resolvers, field metadata, authorization patterns. Produces 4 unit types. |
-| **PunditExtractor** | Pundit authorization policies with action methods (index?, show?, create?, etc.). |
-| **PolicyExtractor** | Domain policy classes with decision methods and eligibility rules. |
-
-**Infrastructure**
-
-| Extractor | What it captures |
-|-----------|-----------------|
-| **EngineExtractor** | Mounted Rails engines via runtime introspection with mount points and route counts. |
-| **I18nExtractor** | Locale files from `config/locales` with translation key structures. |
-| **ActionCableExtractor** | ActionCable channels with stream subscriptions, actions, broadcast patterns. |
-| **ScheduledJobExtractor** | Scheduled jobs from `config/recurring.yml`, `config/sidekiq_cron.yml`, `config/schedule.rb`. |
-| **RakeTaskExtractor** | Rake tasks from `lib/tasks/*.rake` with namespaces, dependencies, descriptions. |
-| **MigrationExtractor** | ActiveRecord migrations with DDL metadata, table operations, reversibility, risk indicators. |
-| **DatabaseViewExtractor** | SQL views from `db/views` (Scenic convention) with materialization and table references. |
-| **StateMachineExtractor** | AASM, Statesman, and state_machines DSL definitions with states and transitions. |
-| **EventExtractor** | Event publish/subscribe patterns (ActiveSupport::Notifications, Wisper). |
-| **CachingExtractor** | Cache usage across controllers, models, and views — strategies, TTLs, cache keys. |
-
-**Testing & Source**
-
-| Extractor | What it captures |
-|-----------|-----------------|
-| **FactoryExtractor** | FactoryBot factory definitions with traits and associations. |
-| **TestMappingExtractor** | Test file → subject class mapping with test counts and framework type. |
-| **LibExtractor** | Ruby files from `lib/` (excluding tasks and generators). |
-| **RailsSourceExtractor** | High-value Rails framework source and gem source pinned to exact installed versions. |
-
-### Key Design Decisions
-
-**Concern inlining.** When extracting a model, included concerns are read from disk and embedded as formatted comments directly in the model's source. This means the full behavioral picture is in one unit — no separate lookups needed during retrieval.
+34 extractors cover every major Rails concept: models (with inlined concerns and schema), controllers (with route context), services, jobs, mailers, GraphQL types/mutations/resolvers, serializers, view components (Phlex and ViewComponent), ERB templates, decorators, concerns, validators, policies, routes, middleware, engines, i18n, Action Cable, rake tasks, migrations, database views, state machines, events, caching patterns, factories, test mappings, and Rails framework source pinned to exact installed versions.
 
-**Route prepending.** Controller source gets a header block showing the HTTP routes that map to it, so the relationship between URLs and actions is immediately visible.
+See [docs/EXTRACTOR_REFERENCE.md](docs/EXTRACTOR_REFERENCE.md) for per-extractor documentation with configuration, edge cases, and example output.
 
-**Semantic chunking.** Large models are split into purpose-specific chunks (summary, associations, callbacks, validations) rather than arbitrary size-based splits. Controllers chunk per-action with the relevant filters and route attached.
+### Key Design Decisions
 
-**Dependency graph with BFS blast radius.** The graph tracks both forward dependencies (what this unit uses) and reverse dependencies (what uses this unit). Changed-file impact is computed via breadth-first traversal — if a concern changes, every model including it gets re-indexed.
+**Concern inlining** — included concerns are embedded directly in the model's source. **Route prepending** — controllers get a route header showing HTTP verb → path → action. **Semantic chunking** — models split by purpose (associations, callbacks, validations), controllers split per-action. **Dependency graph with BFS blast radius** — forward and reverse edges enable change-impact traversal.
 
 ## MCP Servers
 
 CodebaseIndex ships two [MCP](https://modelcontextprotocol.io/) servers for integrating with AI development tools (Claude Code, Cursor, Windsurf, etc.).
 
-**Index Server** (26 tools) — Reads pre-extracted data from disk. No Rails boot required. Provides code lookup, dependency traversal, graph analysis, semantic search, pipeline management, feedback collection, and temporal snapshots.
+**Index Server** (27 tools) — Reads pre-extracted data from disk. No Rails boot required. Provides code lookup, dependency traversal, graph analysis, semantic search, pipeline management, feedback collection, and temporal snapshots.
 
 ```bash
 codebase-index-mcp /path/to/rails-app/tmp/codebase_index
@@ -199,7 +159,7 @@ Add the servers to your project's `.mcp.json`:
 {
   "mcpServers": {
     "codebase-index": {
-      "command": "codebase-index-mcp",
+      "command": "codebase-index-mcp-start",
       "args": ["/path/to/rails-app/tmp/codebase_index"]
     },
     "codebase-console": {
@@ -211,217 +171,44 @@ Add the servers to your project's `.mcp.json`:
 }
 ```
 
-The **index server** reads from a pre-extracted directory — run `bundle exec rake codebase_index:extract` in your Rails app first.
-
-The **console server** runs embedded inside your Rails app (no config file needed). For Docker:
+> **Recommended**: Use `codebase-index-mcp-start` instead of `codebase-index-mcp` for Claude Code. It validates the index directory exists, checks for a manifest, ensures dependencies are installed, and restarts automatically on failure.
 
-```json
-{
-  "mcpServers": {
-    "codebase-console": {
-      "command": "docker",
-      "args": ["exec", "-i", "my_container", "bundle", "exec", "rake", "codebase_index:console"]
-    }
-  }
-}
-```
-
-### Validation
-
-Verify each server starts and lists its tools:
-
-```bash
-# Index server — should list 27 tools
-echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | \
-  codebase-index-mcp /path/to/rails-app/tmp/codebase_index
+The **index server** reads from a pre-extracted directory — run `bundle exec rake codebase_index:extract` in your Rails app first.
 
-# Console server — should list 31 tools (requires Rails app)
-echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | \
-  bundle exec rake codebase_index:console
-```
+The **console server** runs embedded inside your Rails app (no config file needed). For Docker setups, see [docs/DOCKER_SETUP.md](docs/DOCKER_SETUP.md).
 
 ## Subsystems
 
 ```
-lib/
-├── codebase_index.rb                          # Module interface, Configuration, entry point
-├── codebase_index/
-│   ├── extracted_unit.rb                       # Core value object
-│   ├── extractor.rb                            # Orchestrator — coordinates all extractors
-│   ├── dependency_graph.rb                     # Directed graph + PageRank scoring
-│   ├── graph_analyzer.rb                       # Structural analysis (orphans, hubs, cycles, bridges)
-│   ├── model_name_cache.rb                     # Precomputed regex for dependency scanning
-│   ├── retriever.rb                            # Retriever orchestrator with degradation tiers
-│   ├── builder.rb                              # DSL builder for configuration
-│   ├── version.rb                              # Gem version
-│   ├── railtie.rb                              # Rails integration
-│   │
-│   ├── extractors/                             # 34 extractors (one per Rails concept)
-│   │   ├── model_extractor.rb                  # ActiveRecord models
-│   │   ├── controller_extractor.rb             # ActionController
-│   │   ├── service_extractor.rb                # Service objects
-│   │   ├── job_extractor.rb                    # ActiveJob/Sidekiq workers
-│   │   ├── mailer_extractor.rb                 # ActionMailer
-│   │   ├── phlex_extractor.rb                  # Phlex components
-│   │   ├── view_component_extractor.rb         # ViewComponent
-│   │   ├── graphql_extractor.rb                # GraphQL types, mutations, queries
-│   │   ├── serializer_extractor.rb             # Serializers/decorators
-│   │   ├── manager_extractor.rb                # SimpleDelegator managers
-│   │   ├── policy_extractor.rb                 # Policy classes
-│   │   ├── validator_extractor.rb              # Standalone validators
-│   │   ├── rails_source_extractor.rb           # Framework/gem source
-│   │   ├── shared_dependency_scanner.rb        # Shared dependency detection
-│   │   ├── shared_utility_methods.rb           # Shared extractor utilities
-│   │   └── ast_source_extraction.rb            # AST-based source extraction
-│   │
-│   ├── ast/                                    # Prism-based AST layer
-│   │   ├── parser.rb                           # Source parsing adapter
-│   │   ├── node.rb                             # Normalized AST node
-│   │   ├── method_extractor.rb                 # Method boundary detection
-│   │   └── call_site_extractor.rb              # Call site analysis
-│   │
-│   ├── ruby_analyzer/                          # Static analysis
-│   │   ├── class_analyzer.rb                   # Class structure analysis
-│   │   ├── method_analyzer.rb                  # Method complexity/dependencies
-│   │   ├── dataflow_analyzer.rb                # Data flow tracing
-│   │   ├── trace_enricher.rb                   # Enriches flow traces
-│   │   ├── fqn_builder.rb                      # Fully-qualified name resolution
-│   │   └── mermaid_renderer.rb                 # Diagram generation
-│   │
-│   ├── flow_analysis/                          # Execution flow tracing
-│   │   ├── operation_extractor.rb              # Extract operations from AST
-│   │   └── response_code_mapper.rb             # HTTP response mapping
-│   ├── flow_assembler.rb                       # Assembles execution flows
-│   ├── flow_document.rb                        # Flow documentation format
-│   │
-│   ├── chunking/                               # Semantic chunking
-│   │   ├── chunk.rb                            # Chunk value object
-│   │   └── semantic_chunker.rb                 # Type-aware splitting
-│   │
-│   ├── embedding/                              # Embedding pipeline
-│   │   ├── provider.rb                         # Provider interface
-│   │   ├── openai.rb                           # OpenAI adapter
-│   │   ├── text_preparer.rb                    # Text preparation for embedding
-│   │   └── indexer.rb                          # Batch indexing with resumability
-│   │
-│   ├── storage/                                # Storage backends
-│   │   ├── vector_store.rb                     # Vector store interface + InMemory
-│   │   ├── metadata_store.rb                   # Metadata store interface + InMemory/SQLite
-│   │   ├── graph_store.rb                      # Graph store interface + InMemory
-│   │   ├── pgvector.rb                         # PostgreSQL pgvector adapter
-│   │   └── qdrant.rb                           # Qdrant adapter
-│   │
-│   ├── retrieval/                              # Retrieval pipeline
-│   │   ├── query_classifier.rb                 # Intent/scope/type classification
-│   │   ├── search_executor.rb                  # Multi-strategy search
-│   │   ├── ranker.rb                           # RRF-based ranking
-│   │   └── context_assembler.rb                # Token-budgeted context assembly
-│   │
-│   ├── formatting/                             # LLM context formatting
-│   │   ├── base.rb                             # Base formatter
-│   │   ├── claude_adapter.rb                   # Claude-optimized output
-│   │   ├── gpt_adapter.rb                      # GPT-optimized output
-│   │   ├── generic_adapter.rb                  # Generic LLM output
-│   │   └── human_adapter.rb                    # Human-readable output
-│   │
-│   ├── mcp/                                    # MCP Index Server (26 tools)
-│   │   ├── server.rb                           # Tool definitions + dispatch
-│   │   └── index_reader.rb                     # JSON index reader
-│   │
-│   ├── console/                                # Console MCP Server (31 tools)
-│   │   ├── server.rb                           # Console server + tool registration
-│   │   ├── bridge.rb                           # JSON-lines protocol bridge
-│   │   ├── safe_context.rb                     # Transaction rollback + timeout
-│   │   ├── connection_manager.rb               # Docker/direct/SSH modes
-│   │   ├── model_validator.rb                  # AR schema validation
-│   │   ├── sql_validator.rb                    # SQL statement validation
-│   │   ├── audit_logger.rb                     # JSONL audit logging
-│   │   ├── confirmation.rb                     # Human-in-the-loop confirmation
-│   │   ├── tools/
-│   │   │   ├── tier1.rb                        # 9 safe read-only tools
-│   │   │   ├── tier2.rb                        # 9 domain-aware tools
-│   │   │   ├── tier3.rb                        # 10 analytics tools
-│   │   │   └── tier4.rb                        # 3 guarded tools
-│   │   └── adapters/
-│   │       ├── sidekiq_adapter.rb              # Sidekiq job backend
-│   │       ├── solid_queue_adapter.rb          # Solid Queue job backend
-│   │       ├── good_job_adapter.rb             # GoodJob job backend
-│   │       └── cache_adapter.rb                # Cache backend adapters
-│   │
-│   ├── coordination/                           # Multi-agent coordination
-│   │   └── pipeline_lock.rb                    # File-based pipeline locking
-│   │
-│   ├── feedback/                               # Agent self-service
-│   │   ├── store.rb                            # JSONL feedback storage
-│   │   └── gap_detector.rb                     # Feedback-driven gap detection
-│   │
-│   ├── operator/                               # Pipeline management
-│   │   ├── status_reporter.rb                  # Pipeline status
-│   │   ├── error_escalator.rb                  # Error classification
-│   │   └── pipeline_guard.rb                   # Rate limiting
-│   │
-│   ├── observability/                          # Instrumentation
-│   │   ├── instrumentation.rb                  # ActiveSupport::Notifications
-│   │   ├── structured_logger.rb                # JSON structured logging
-│   │   └── health_check.rb                     # Component health checks
-│   │
-│   ├── resilience/                             # Fault tolerance
-│   │   ├── circuit_breaker.rb                  # Circuit breaker pattern
-│   │   ├── retryable_provider.rb               # Retry with backoff
-│   │   └── index_validator.rb                  # Index integrity validation
-│   │
-│   ├── db/                                     # Schema management
-│   │   ├── schema_version.rb                   # Version tracking
-│   │   ├── migrator.rb                         # Standalone migration runner
-│   │   └── migrations/
-│   │       ├── 001_create_units.rb
-│   │       ├── 002_create_edges.rb
-│   │       └── 003_create_embeddings.rb
-│   │
-│   ├── session_tracer/                          # Session tracing middleware + stores
-│   │   ├── middleware.rb                        # Rack middleware
-│   │   ├── file_store.rb                        # File-based trace storage
-│   │   ├── redis_store.rb                       # Redis trace storage
-│   │   └── solid_cache_store.rb                 # SolidCache trace storage
-│   │
-│   ├── temporal/                                # Temporal snapshot system
-│   │   ├── snapshot_store.rb                    # Snapshot persistence + diff
-│   │   └── snapshot_metadata.rb                 # Snapshot metadata
-│   │
-│   └── evaluation/                             # Retrieval evaluation
-│       ├── query_set.rb                        # Evaluation query loading
-│       ├── metrics.rb                          # Precision@k, Recall, MRR
-│       ├── evaluator.rb                        # Query evaluation
-│       ├── baseline_runner.rb                  # Grep/random/file baselines
-│       └── report_generator.rb                 # JSON report generation
-│
-├── generators/codebase_index/                  # Rails generators
-│   ├── install_generator.rb                    # Initial setup
-│   └── pgvector_generator.rb                   # pgvector migration
-│
-├── tasks/
-│   └── codebase_index.rake                     # Rake task definitions
-│
-exe/
-├── codebase-index-mcp                          # MCP Index Server executable (stdio)
-├── codebase-index-mcp-start                    # Self-healing MCP wrapper
-├── codebase-index-mcp-http                     # MCP Index Server (HTTP/Rack)
-└── codebase-console-mcp                        # Console MCP Server executable
-```
+lib/codebase_index/
+├── extractor.rb              # Orchestrator — coordinates all 34 extractors
+├── extracted_unit.rb         # Core value object (the universal currency)
+├── dependency_graph.rb       # Directed graph + PageRank scoring
+├── graph_analyzer.rb         # Structural analysis (orphans, hubs, cycles, bridges)
+├── retriever.rb              # Retrieval orchestrator with degradation tiers
+├── extractors/               # 34 extractors (one per Rails concept)
+├── ast/                      # Prism-based AST layer
+├── ruby_analyzer/            # Static analysis (class, method, dataflow)
+├── chunking/                 # Semantic chunking (type-aware splitting)
+├── embedding/                # Embedding pipeline (OpenAI, Ollama)
+├── storage/                  # Storage backends (pgvector, Qdrant, SQLite)
+├── retrieval/                # Retrieval pipeline (classify, search, rank, assemble)
+├── mcp/                      # MCP Index Server (27 tools)
+├── console/                  # Console MCP Server (31 tools, 4 tiers)
+├── coordination/             # Multi-agent pipeline locking
+├── notion/                   # Notion export
+├── session_tracer/           # Session tracing middleware
+├── temporal/                 # Temporal snapshot system
+└── evaluation/               # Retrieval evaluation harness
 
-## Context Assembly
-
-When serving context to an LLM, token budget is allocated in layers:
-
-```
-Budget Allocation:
-├── 10%  Structural overview (always included)
-├── 50%  Primary relevant units
-├── 25%  Supporting context (dependencies)
-└── 15%  Framework reference (when needed)
+exe/
+├── codebase-index-mcp        # Index Server executable (stdio)
+├── codebase-index-mcp-start  # Self-healing MCP wrapper
+├── codebase-index-mcp-http   # Index Server (HTTP/Rack)
+└── codebase-console-mcp      # Console MCP Server executable
 ```
 
-Queries are classified to determine whether framework source context is needed. "What options does has_many support?" routes to Rails source; "how do we handle checkout?" routes to application code.
+See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full pipeline explanation — extraction phases, dependency graph, retrieval pipeline, storage backends, and semantic chunking.
 
 ## Usage
 
@@ -434,43 +221,40 @@ bundle exec rake codebase_index:extract
 ### Incremental (CI)
 
 ```bash
-# Auto-detects GitHub Actions / GitLab CI environment
 bundle exec rake codebase_index:incremental
 ```
 
-```yaml
-# .github/workflows/index.yml
-jobs:
-  index:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          fetch-depth: 2
-      - name: Update index
-        run: bundle exec rake codebase_index:incremental
-        env:
-          GITHUB_BASE_REF: ${{ github.base_ref }}
-```
+Auto-detects GitHub Actions / GitLab CI environment. See [Getting Started](docs/GETTING_STARTED.md) for CI workflow YAML.
+
+### Docker
 
-### Framework-Only (on dependency changes)
+Extraction runs inside the container; the Index Server runs on the host reading volume-mounted output. See [docs/DOCKER_SETUP.md](docs/DOCKER_SETUP.md) for Docker setup, MCP config, and troubleshooting.
 
 ```bash
-bundle exec rake codebase_index:extract_framework
+docker compose exec app bundle exec rake codebase_index:extract
 ```
 
 ### Other Tasks
 
 ```bash
-rake codebase_index:validate  # Check index integrity
-rake codebase_index:stats     # Show unit counts, sizes, graph stats
-rake codebase_index:clean     # Remove index
+rake codebase_index:validate          # Check index integrity
+rake codebase_index:stats             # Show unit counts, sizes, graph stats
+rake codebase_index:clean             # Remove index
+rake codebase_index:embed             # Embed all extracted units
+rake codebase_index:embed_incremental # Embed changed units only
+rake codebase_index:flow[EntryPoint]  # Generate execution flow for an entry point
+rake codebase_index:console           # Start console MCP server
+rake codebase_index:notion_sync       # Sync models/columns to Notion databases
 ```
 
+See [docs/NOTION_INTEGRATION.md](docs/NOTION_INTEGRATION.md) for Notion export configuration.
+
 ### Ruby API
 
+> **Requires a booted Rails environment.** These methods use runtime introspection and must be called from within a Rails process (console, rake task, initializer).
+
 ```ruby
-# Full extraction
+# Full extraction (output_dir from configuration)
 CodebaseIndex.extract!
 
 # Incremental
@@ -510,13 +294,24 @@ tmp/codebase_index/
 
 Each unit JSON contains: `identifier`, `type`, `file_path`, `source_code` (annotated), `metadata` (rich structured data), `dependencies`, `dependents`, `chunks` (if applicable), and `estimated_tokens`.
 
-## Development
+## Documentation
 
-After checking out the repo:
+| Guide | Purpose |
+|-------|---------|
+| [Getting Started](docs/GETTING_STARTED.md) | Install, configure, extract, inspect |
+| [FAQ](docs/FAQ.md) | Common questions about setup, extraction, MCP, Docker |
+| [Troubleshooting](docs/TROUBLESHOOTING.md) | Symptom → cause → fix for common problems |
+| [Architecture](docs/ARCHITECTURE.md) | Pipeline stages, dependency graph, retrieval, storage |
+| [Extractor Reference](docs/EXTRACTOR_REFERENCE.md) | What each of the 34 extractors captures |
+| [MCP Servers](docs/MCP_SERVERS.md) | Full tool catalog and setup for Claude Code, Cursor, Windsurf |
+| [MCP Tool Cookbook](docs/MCP_TOOL_COOKBOOK.md) | Scenario-based examples for common tasks |
+| [Configuration Reference](docs/CONFIGURATION_REFERENCE.md) | All options with defaults |
+| [Backend Matrix](docs/BACKEND_MATRIX.md) | Supported infrastructure combinations |
+
+## Development
 
 ```bash
 bin/setup          # Install dependencies
-bin/console        # Interactive prompt
 bundle exec rake spec      # Run tests
 bundle exec rubocop        # Lint
 ```

data/exe/codebase-index-mcp

@@ -15,40 +15,12 @@ require_relative '../lib/codebase_index'
 require_relative '../lib/codebase_index/dependency_graph'
 require_relative '../lib/codebase_index/graph_analyzer'
 require_relative '../lib/codebase_index/mcp/server'
+require_relative '../lib/codebase_index/mcp/bootstrapper'
 require_relative '../lib/codebase_index/embedding/text_preparer'
 require_relative '../lib/codebase_index/embedding/indexer'
 
-index_dir = ARGV[0] || ENV['CODEBASE_INDEX_DIR'] || Dir.pwd
-
-unless Dir.exist?(index_dir)
-  warn "Error: Index directory does not exist: #{index_dir}"
-  exit 1
-end
-
-unless File.exist?(File.join(index_dir, 'manifest.json'))
-  warn "Error: No manifest.json found in: #{index_dir}"
-  warn 'Run `bundle exec rake codebase_index:extract` in your Rails app first.'
-  exit 1
-end
-
-# Attempt to build a retriever for semantic search.
-# Auto-configures from environment variables when no explicit configuration exists.
-retriever = begin
-  config = CodebaseIndex.configuration
-
-  if !config.embedding_provider && ENV.fetch('OPENAI_API_KEY', nil)
-    config.vector_store = :in_memory
-    config.metadata_store = :in_memory
-    config.graph_store = :in_memory
-    config.embedding_provider = :openai
-    config.embedding_options = { api_key: ENV.fetch('OPENAI_API_KEY', nil) }
-  end
-
-  CodebaseIndex::Builder.new(config).build_retriever if config.embedding_provider
-rescue StandardError => e
-  warn "Note: Semantic search unavailable (#{e.message}). Using pattern-based search only."
-  nil
-end
+index_dir = CodebaseIndex::MCP::Bootstrapper.resolve_index_dir(ARGV)
+retriever = CodebaseIndex::MCP::Bootstrapper.build_retriever
 
 server = CodebaseIndex::MCP::Server.build(index_dir: index_dir, retriever: retriever)