woods 1.6.1 → 2.0.0.beta2

data/docs/MCP_WORKTREE_SETUP.md

@@ -0,0 +1,127 @@
+# MCP Registration in Git Worktrees
+
+> **Claude Code specific.** This page covers Claude Code's MCP registration model (`/mcp`, `~/.claude/plugins/`); other MCP clients manage per-directory registration their own way.
+
+When you work in a git worktree, a separate directory checked out from the same repository, your MCP tools may not be available to subagents running in that directory. This page explains why, how to fix it, and how to confirm the registration took effect.
+
+## Why Worktree Subagents May Not See Woods Tools
+
+MCP server registration in Claude Code is controlled by `.mcp.json` files. Claude Code discovers these files by walking up the directory tree from the working directory. It stops at the first `.mcp.json` it finds (or at `~/.claude/settings.json` for global registrations).
+
+A git worktree has its own root directory separate from the main repository checkout. When a subagent starts inside the worktree root, it walks up from that path, not from the main repository root. Unless a `.mcp.json` exists inside the worktree directory tree or in an ancestor shared with both checkouts, the subagent sees no MCP servers.
+
+Example directory layout:
+
+```
+~/work/my-app/            ← main checkout, has .mcp.json here
+~/work/my-app-feature/    ← worktree, no .mcp.json, so MCP tools are missing
+```
+
+A subagent spawned in `~/work/my-app-feature/` will not find the `.mcp.json` from `~/work/my-app/`.
+
+## Fix: Add a `.mcp.json` to the Worktree Root
+
+Create a `.mcp.json` in the worktree's root directory with the same woods server entries you use in the main checkout:
+
+```json
+{
+  "mcpServers": {
+    "woods": {
+      "command": "woods-mcp-start",
+      "args": ["./tmp/woods"]
+    },
+    "woods-console": {
+      "command": "docker",
+      "args": [
+        "compose", "exec", "-T", "app",
+        "bundle", "exec", "rake", "woods:console"
+      ],
+      "cwd": "/absolute/host/path/to/worktree"
+    }
+  }
+}
+```
+
+Adjust the paths and arguments to match your project's setup. In particular:
+
+- `./tmp/woods` is a relative path, it resolves against the worktree root, which is correct if extraction output is written into each worktree separately.
+- If you share a single extraction output directory between checkouts, use the absolute path to the shared output: `"/absolute/path/to/main-checkout/tmp/woods"`.
+- For Docker projects, the `docker compose exec` command works the same from any host path.
+
+## How Plugin Discovery Works
+
+Claude Code also discovers MCP servers registered in plugin manifests. A plugin at `~/.claude/plugins/<plugin-name>/admin-tools/.mcp.json` is loaded globally, its servers are available in any session regardless of working directory.
+
+If your team distributes the woods MCP registration through a shared plugin, the worktree problem does not apply. Check whether woods is already registered this way:
+
+```bash
+ls ~/.claude/plugins/
+# look for a directory containing admin-tools/.mcp.json
+cat ~/.claude/plugins/<plugin-name>/admin-tools/.mcp.json
+```
+
+If you find the woods servers registered there, subagents in any worktree will have access automatically, you do not need a per-worktree `.mcp.json`.
+
+## Verifying MCP Registration for a Subagent
+
+### Option 1: List tools from a claude session in the worktree
+
+Open a new Claude Code session with the worktree as the working directory and run:
+
+```
+/mcp
+```
+
+This lists all connected MCP servers and their tools. If `woods` and/or `woods-console` appear, registration is working.
+
+### Option 2: Check via the woods status tool
+
+Ask Claude to call the status tool:
+
+```
+Use woods-console console_status to check what models are available.
+```
+
+If the tool runs successfully, MCP is registered and the console server is reachable.
+
+### Option 3: Inspect the MCP config that Claude Code loaded
+
+From the worktree directory, run:
+
+```bash
+cat .mcp.json
+```
+
+If the file exists and contains the woods entries, Claude Code will use it. If the file is missing, check parent directories up to your home directory for any `.mcp.json` that would be discovered.
+
+## Extraction Provenance in Worktrees (`git_branch` / `git_sha`)
+
+`manifest.json` records the `git_branch` and `git_sha` the extraction ran against. In a linked worktree, `.git` is a **file** containing a `gitdir:` pointer to the real git directory, frequently an absolute host path, rather than a `.git` directory.
+
+Woods resolves provenance with worktree-aware git plumbing (`git -C <root> rev-parse`), so an ordinary worktree reports the correct branch and SHA. When a `.git` is present but the pointed-to git directory **cannot be resolved**: most commonly a worktree extracted inside a container where the host path isn't mounted. Woods records `git_branch: "unknown"` / `git_sha: "unknown"` rather than a stale, misleading value: a baked `GIT_BRANCH`/`GIT_SHA` build arg is **not** trusted here (it could be stale). The env vars are honored only when there is no `.git` at the root at all (a non-repo checkout, e.g. a Docker `COPY` that excludes `.git`) or git is unavailable.
+
+To get correct provenance from a containerized worktree, mount the directory the `gitdir:` pointer references (the parent repository's `.git`) into the container so git can resolve it. Temporal snapshots skip an `"unknown"` SHA, so misleading provenance never keys a snapshot.
+
+## Troubleshooting
+
+**"Unknown tool" or "no MCP server named woods"**
+
+The woods MCP server is not registered for this session. Add a `.mcp.json` to the worktree root as shown above, then restart the session.
+
+**Console server starts but returns "unsupported: Not yet implemented in embedded mode" for console_sql / console_query**
+
+The console server is registered and reachable, but `embedded_read_tools` is disabled (the default). To enable console_sql and console_query in embedded mode, see the [Console MCP Setup guide](CONSOLE_MCP_SETUP.md), specifically the `embedded_read_tools: true` option for the Rack middleware.
+
+**Extraction output is empty or stale in the worktree**
+
+Extraction writes to `tmp/woods/` relative to the Rails application root (inside the container). If the worktree's volume mount points to a different host path than the main checkout, run extraction again from the worktree:
+
+```bash
+docker compose exec app bundle exec rake woods:extract
+```
+
+See [DOCKER_SETUP.md](DOCKER_SETUP.md) for the full Docker workflow.
+
+**Worktree `.mcp.json` conflicts with main checkout `.mcp.json`**
+
+Each file is independent. Claude Code loads the one closest to the working directory. There is no inheritance or merging between them. Keep both files in sync manually, or move the shared configuration into a global plugin manifest.

data/docs/NOTION_INTEGRATION.md

@@ -0,0 +1,283 @@
+# Notion Integration
+
+Sync your Rails application's data model documentation to Notion databases, making schema, associations, validations, callbacks, and column-level detail accessible to non-technical stakeholders without GitHub access.
+
+## What Gets Synced
+
+Woods extracts your Rails application via runtime introspection and pushes structured data to two Notion databases:
+
+**Data Models Database**: One page per ActiveRecord model:
+- Table name, model class name, file path
+- Associations (has_many, belongs_to, has_one, through, polymorphic)
+- Validations (grouped by attribute)
+- Callbacks with side-effect analysis (jobs enqueued, services called)
+- Scopes
+- Column count
+- Git metadata (last modified date, change frequency)
+- Dependencies (services, jobs, other models referenced)
+- Last schema change (from migration history)
+
+**Columns Database**: One page per database column:
+- Column name, data type, nullable, default value
+- Validation rules (matched from model-level validations)
+- Relation to parent Data Models page
+
+All syncs are **idempotent**: existing pages are updated, new pages are created. Re-running the sync is always safe.
+
+## Sync Manifest (Incremental Sync)
+
+Sync is incremental. A **sync manifest** (`<output_dir>/notion_sync_manifest.json`) records the content hash and Notion page id of every page last pushed. On each run the exporter:
+
+- **skips** a page whose built content hash is unchanged (zero API calls),
+- **updates** only pages whose content actually changed, via the cached page id,
+- **self-heals** the cached id if the page was deleted or archived behind it (one lookup, then a normal create).
+
+Manifest entries for models/columns that vanished from the current extraction are pruned, but **no Notion page is ever deleted** by the sync, there is no delete path. A renamed or removed model just leaves its old page in Notion untouched.
+
+If the manifest is missing (first run, or a CI cache miss), the exporter falls back to the full lookup/create path for every page and rebuilds the manifest, correct, just more API calls than a steady-state run.
+
+### Escape hatch
+
+Set `WOODS_NOTION_FORCE=1` (or pass `force_full: true` to the exporter) to ignore the manifest for one run and re-check every page, useful after a mapper/format change that alters every page's content, where the unchanged-hash skip would otherwise mask the update.
+
+## Setup
+
+### 1. Create a Notion Integration
+
+1. Go to [notion.so/my-integrations](https://www.notion.so/my-integrations)
+2. Create a new internal integration
+3. Copy the API token (starts with `secret_`)
+
+### 2. Create Notion Databases
+
+Create two databases in your Notion workspace with these exact property names and types:
+
+**Data Models Database:**
+
+| Property | Type |
+|---|---|
+| Table Name | Title |
+| Model Name | Text |
+| Description | Text |
+| Associations | Text |
+| Validations | Text |
+| Callbacks | Text |
+| Scopes | Text |
+| Column Count | Number |
+| Last Modified | Date |
+| Change Frequency | Select (options: new, hot, active, stable, dormant) |
+| Last Schema Change | Date |
+| File Path | Text |
+| Dependencies | Text |
+
+**Columns Database:**
+
+| Property | Type |
+|---|---|
+| Column Name | Title |
+| Table | Relation (→ Data Models database) |
+| Data Type | Select (options: string, integer, bigint, boolean, datetime, text, decimal, float, date, binary, json, jsonb) |
+| Nullable | Checkbox |
+| Default Value | Text |
+| Validation Rules | Text |
+
+### 3. Share Databases with Integration
+
+Open each database in Notion, click the `...` menu → "Connections" → add your integration.
+
+### 4. Get Database IDs
+
+Each database URL contains its ID: `https://notion.so/{workspace}/{database_id}?v=...`
+
+### 5. Configure Woods
+
+```ruby
+# config/initializers/woods.rb
+Woods.configure do |config|
+  config.notion_api_token = Rails.application.credentials.dig(:notion, :api_token)
+  config.notion_database_ids = {
+    data_models: 'your-data-models-database-id',
+    columns: 'your-columns-database-id'
+  }
+end
+```
+
+Or via environment variables:
+
+```bash
+export NOTION_API_TOKEN=secret_...
+```
+
+## Common Workflows
+
+### Full Extraction + Sync
+
+```bash
+# Extract everything from the Rails app, then push to Notion
+bundle exec rake woods:extract
+bundle exec rake woods:notion_sync
+```
+
+### Post-Migration Sync
+
+```bash
+# Re-extract changed files after a migration, then sync
+bundle exec rake woods:incremental
+bundle exec rake woods:notion_sync
+```
+
+### Buildkite CI Integration
+
+Add to your `pipeline.yml`:
+
+```yaml
+steps:
+  - label: ":database: Extract & Sync to Notion"
+    command: |
+      bundle exec rake woods:extract
+      bundle exec rake woods:notion_sync
+    if: build.branch == "main" && build.state == "passed"
+    soft_fail: true
+    env:
+      NOTION_API_TOKEN: $NOTION_API_TOKEN  # Set in pipeline environment settings
+```
+
+For post-migration syncs only:
+
+```yaml
+steps:
+  - label: ":database: Sync Schema to Notion"
+    command: |
+      bundle exec rake woods:incremental
+      bundle exec rake woods:notion_sync
+    if: |
+      build.message =~ /migrate/i ||
+      build.env("FORCE_SCHEMA_SYNC") == "true"
+    soft_fail: true
+```
+
+### MCP Server
+
+If using the MCP Index Server, the `notion_sync` tool is available:
+
+```json
+{
+  "tool": "notion_sync",
+  "arguments": {}
+}
+```
+
+### Selective Sync
+
+Only sync Data Models (skip Columns):
+
+```ruby
+Woods.configure do |config|
+  config.notion_database_ids = { data_models: 'db-uuid' }
+  # columns key omitted → column sync is skipped
+end
+```
+
+## What the Gem Handles vs. the Host App
+
+### Gem (automated)
+
+| Database | Content | Source |
+|---|---|---|
+| Data Models | Schema, associations, validations, callbacks, scopes, git metadata | ModelExtractor + MigrationExtractor |
+| Columns | Column details, types, defaults, validation rules | ModelExtractor metadata |
+
+### Host App (you build these)
+
+| Database | Content | How to Build |
+|---|---|---|
+| Features | Feature ownership, status, user stories, acceptance criteria | Human-authored in Notion |
+| User Flows | User-facing flow documentation, triggers, endpoints | Human-authored (future: gem can auto-populate from controller/route extraction) |
+| Data Handling | PII classification, encryption, retention policies | Human policy decisions |
+| Deploys | Build numbers, environments, deployers, commit SHAs | Buildkite webhook → Rails controller → Notion API |
+
+### Example: Deploy Tracking (Host App)
+
+```ruby
+# app/controllers/webhooks/buildkite_controller.rb
+module Webhooks
+  class BuildkiteController < ApplicationController
+    skip_before_action :verify_authenticity_token
+    before_action :verify_buildkite_token
+
+    def create
+      payload = JSON.parse(request.body.read)
+      NotionDeploySyncJob.perform_later(payload)
+      head :ok
+    end
+  end
+end
+
+# app/jobs/notion_deploy_sync_job.rb
+class NotionDeploySyncJob < ApplicationJob
+  def perform(payload)
+    build = payload["build"]
+    # Use your own NotionClient to create a Deploys page
+    # This is outside Woods's scope
+  end
+end
+```
+
+## Rate Limiting
+
+Notion's API allows 3 requests per second. The gem's built-in rate limiter handles this automatically. For large codebases (100+ models), expect the sync to take a few minutes.
+
+If you see 429 errors, the client retries up to 3 times with exponential backoff using the `Retry-After` header.
+
+### Retries and duplicates
+
+A 429 is always retried, for any request, the server rejected it before doing any work. A 503 is different: an intermediary in front of Notion's API can synthesize a 503 for a request the origin already committed, so blindly retrying it risks creating a duplicate page. The client only retries a 503 for **idempotent** requests (reads, and page updates keyed by an existing page id). Page *creation* has no idempotency key, so a 503 on `create_page` is raised immediately instead of retried, you'll see the error and can re-run the sync, which is safe because the manifest either already recorded the created page or will create it fresh.
+
+## Error Handling
+
+The sync collects errors per-model and per-column without stopping. The stats hash returned by `sync_all` includes an `errors` array:
+
+```ruby
+stats = exporter.sync_all
+stats[:errors]  # => ["User: Notion API error 400: ...", ...]
+```
+
+Common errors:
+- **401 Unauthorized**: Check your API token
+- **404 Not Found**: Check database IDs and ensure the integration has access
+- **400 Validation Error**: Check that Notion database properties match the expected schema above
+- **429 Rate Limited**: Automatic retry (up to 3 times)
+
+`rake woods:notion_sync` prints the first five errors and then **exits 1**, matching `woods:unblocked_sync` and `woods:obsidian`. A printed-but-green run is invisible in a post-merge pipeline: a dead token would otherwise keep the job green while the Notion database went stale indefinitely. Whatever synced before the failure is durable, so re-running after the fix converges. The Buildkite examples above use `soft_fail: true`; drop it if you want the sync failure to fail the build.
+
+## Architecture
+
+```
+Extraction Output (JSON on disk)
+       ↓
+┌─────────────────────┐
+│ IndexReader          │ ← Reads model, column, migration data
+└─────────────────────┘
+       ↓
+┌─────────────────────┐
+│ Exporter             │ ← Orchestrates sync flow
+├─────────────────────┤
+│ ModelMapper          │ ← Maps ExtractedUnit → Notion Data Models properties
+│ ColumnMapper         │ ← Maps column metadata → Notion Columns properties
+│ MigrationMapper      │ ← Extracts latest migration dates per table
+└─────────────────────┘
+       ↓
+┌─────────────────────┐
+│ Client               │ ← Notion API wrapper (Net::HTTP, rate-limited)
+│ RateLimiter          │ ← 3 req/sec token bucket
+└─────────────────────┘
+       ↓
+    Notion API
+```
+
+## Future Extensions
+
+- **User Flows**: Auto-populate from controller/route extraction (controller actions, HTTP methods, filters, dependencies)
+- **Data Handling**: Surface column types and model metadata to assist with PII classification
+- **ERD Diagrams**: Generate and attach relationship diagrams to Data Models pages
+- **Diff Reporting**: Use temporal snapshots to show what changed between syncs

data/docs/OBSIDIAN_INTEGRATION.md

@@ -0,0 +1,170 @@
+# Obsidian Integration
+
+Export Woods extraction artifacts to a self-contained [Obsidian](https://obsidian.md) vault, a
+folder of interlinked Markdown notes, one per extracted unit. The vault is designed to be read two
+ways at once:
+
+- **By humans**: explore the app's structure in Obsidian's graph view, filter and sort units in an
+  Obsidian [Bases](https://help.obsidian.md/bases) table, and drill into a single unit's note with its
+  dependencies and dependents as clickable wikilinks.
+- **By agents**: load the entire dependency topology from a single `_woods/` sidecar (one read,
+  no per-note fan-out), with a stable `id → note path` manifest for navigation.
+
+Unlike the Notion and Unblocked exporters, this one writes **local files only**: there is no API
+token, no network call, and no rate limit. An Obsidian vault is just a folder.
+
+## Quick start
+
+```bash
+# 1. Extract (produces tmp/woods/ with the dependency graph + per-unit JSON)
+bundle exec rake woods:extract
+
+# 2. Generate the vault (defaults to tmp/woods/obsidian_vault/)
+bundle exec rake woods:obsidian
+```
+
+Then in Obsidian: **Open folder as vault** → point it at `tmp/woods/obsidian_vault`. For the full
+experience (graph colors, Bases, link format) open the generated folder *as its own vault* rather than
+nesting it inside an existing one, a nested folder inherits the host vault's config and ignores the
+shipped `.obsidian/` settings.
+
+## What gets generated
+
+```
+obsidian_vault/
+├── .woods-vault              # ownership sentinel (marks this dir as woods-managed)
+├── .obsidian/                # vault config, only written into a woods-owned vault
+│   ├── app.json              #   newLinkFormat: absolute, useMarkdownLinks: false
+│   ├── types.json            #   property types (pagerank → number, tags → tags)
+│   └── graph.json            #   color groups by #woods/<type> (global graph)
+├── _woods/                   # machine sidecar (the agent interface)
+│   ├── manifest.json         #   { notes: {id → {path,type,pagerank,tags}}, paths: {path → id} }
+│   ├── dependency_graph.json #   verbatim copy, full topology for traversal
+│   └── graph_analysis.json   #   verbatim copy, hubs / cycles / orphans / bridges
+├── Units.base                # Obsidian Bases view (filterable/sortable unit inventory)
+├── _Overview.md              # top-level map of contents
+├── models/
+│   ├── _index.md             # MOC: every model as a wikilink
+│   └── User.md
+├── controllers/
+│   └── Users__RegistrationsController.md
+└── …                         # one folder per extracted unit type
+```
+
+## A unit note
+
+Each note carries **flat YAML frontmatter** (the machine-queryable surface) and a Markdown body with
+wikilinks (the human surface). Frontmatter is kept flat on purpose. Obsidian's Properties UI does
+not support nested objects or arrays-of-objects, so the structured edge data lives in the `_woods/`
+sidecar instead.
+
+```markdown
+---
+woods_managed: true
+id: User
+type: model
+file: app/models/user.rb
+source_hash: a3c5f8e9
+pagerank: 0.0421
+dependency_count: 3
+dependent_count: 12
+tags:
+- woods/model
+- woods/hub
+aliases:
+- User
+---
+
+# User
+
+**File:** `app/models/user.rb` | **LOC:** 84 | **Table:** users (17 columns)
+
+> [!warning] Hub, high blast radius
+> 12 units depend on this (PageRank 0.0421).
+
+## Depends on
+- [[models/Account|Account]], *belongs_to*
+- [[mailers/WelcomeMailer|WelcomeMailer]], *job_enqueue*
+
+## Used by (12)
+**controllers:** [[controllers/Users__RegistrationsController|Users::RegistrationsController]]
+**jobs:** [[jobs/SyncProfileJob|SyncProfileJob]]
+```
+
+Wikilinks are path-qualified with an alias (`[[models/Account|Account]]`): the target is the
+sanitized vault path so the link always resolves, and the alias shows the original identifier. The
+note's `# H1` carries the clean identifier so the sanitized filename never shows as the title.
+
+## The three visualizer surfaces
+
+| Surface | Best for | Notes |
+|---|---|---|
+| **Graph view** | seeing connections / blast radius at a glance | colored by `#woods/<type>` tag (global graph); for the *local* graph, create the color-by-tag groups once. Obsidian persists them per-vault |
+| **Bases (`Units.base`)** | inventory & triage, "all models sorted by PageRank", "show me the hubs" | requires Obsidian ≥ 1.9 (cards/list views ≥ 1.10); inert and harmless on older versions |
+| **Note bodies + backlinks** | drilling into one unit and navigating outward | "Depends on" / "Used by" are clickable; Obsidian's backlink panel shows inbound links |
+
+## Configuration
+
+All options are passed to the rake task via environment variables, there are **no global
+`Woods.configure` settings** for this exporter (the vault path is an output location, not a
+credential):
+
+| Env var | Default | Effect |
+|---|---|---|
+| `WOODS_OUTPUT` | `config.output_dir` (`tmp/woods`) | extraction directory to read from |
+| `WOODS_OBSIDIAN_VAULT` | `<output>/obsidian_vault` | where to write the vault |
+| `WOODS_OBSIDIAN_INCLUDE_SOURCE` | off | embed each unit's source code (credential-scrubbed) in its note |
+| `WOODS_OBSIDIAN_INCLUDE_FRAMEWORK` | off | include `rails_source` units (large; off by default) |
+| `WOODS_OBSIDIAN_FORCE_PURGE` | off | bypass the 30% mass-deletion guard during the stale-note sweep |
+
+```bash
+WOODS_OBSIDIAN_VAULT=~/notes/myapp-code WOODS_OBSIDIAN_INCLUDE_SOURCE=1 bundle exec rake woods:obsidian
+```
+
+Or call the exporter directly:
+
+```ruby
+require 'woods/obsidian/vault_exporter'
+
+Woods::Obsidian::VaultExporter.new(
+  index_dir: 'tmp/woods',
+  vault_path: 'tmp/woods/obsidian_vault',
+  include_source: false,
+  include_framework: false
+).export_all
+# => { exported: 412, indexes: 19, swept: 3, skipped: 0, errors: [] }
+```
+
+## Re-running and safety
+
+The exporter **fully regenerates** the vault on every run (no incremental manifest, local writes are
+cheap). Output is deterministic: re-running against an unchanged extraction produces byte-identical
+notes, so unchanged units never show up in a git diff.
+
+Notes Woods manages carry `woods_managed: true` in their frontmatter. On each run, after all notes are
+written successfully, a **sweep** removes managed notes whose unit no longer exists, so deletions in
+your code propagate. Several guards make the sweep safe to point at a real vault:
+
+- It only runs against a directory carrying the `.woods-vault` sentinel (written by Woods on first run).
+- It only deletes notes with the `woods_managed: true` marker, your own notes are never touched.
+- It refuses to delete more than 30% of managed notes at once (the signature of a partial extraction)
+  unless `WOODS_OBSIDIAN_FORCE_PURGE=1` is set.
+- It resolves symlinks and confirms every deletion target is inside the vault root.
+- It is skipped entirely if any note failed to write that run (a stale note is harmless; a deleted
+  reviewed note is not).
+
+The same ownership check guards the `.obsidian/` config: Woods will **not** overwrite an existing,
+foreign `.obsidian/` folder, it leaves your Obsidian settings untouched and warns instead.
+
+## Limitations
+
+- **Bases needs Obsidian ≥ 1.9** (≥ 1.10 for card/list views). The `.base` file is harmless on older
+  versions, it simply doesn't render.
+- **`gem_source` units are not exported.** They aren't reachable through the index reader; only
+  `rails_source` is covered by `include_framework`.
+- **Hand-edits diverge.** The vault is meant to be regenerated. Editing a note's properties in
+  Obsidian rewrites its frontmatter, after which a re-export will overwrite your changes.
+- **Nested vaults ignore the shipped config.** Open the generated folder as its own vault for graph
+  colors, Bases, and link-format settings to apply.
+
+See `docs/AGENT_GUIDE.md` for how an agent consumes the `_woods/` sidecar.