woods 1.6.1 → 2.0.0.beta2

data/SECURITY.md

@@ -0,0 +1,92 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported | Until |
+|---------|-----------|-------|
+| 2.0.x   | Yes, all fixes | Current release line |
+| 1.6.x   | Security fixes only | 2027-02-20 |
+| < 1.6   | No | n/a |
+| 0.x     | No | n/a |
+
+Only the newest patch release of a supported line receives fixes. A report
+against 1.6.x is assessed against `main` first; if the current release line is
+unaffected, the 1.6.x backport is still issued until the date above.
+
+Upgrading from 1.x to 2.0 requires one clean re-index, see the Upgrade Notes in
+[CHANGELOG.md](CHANGELOG.md) and [docs/UPGRADING_TO_2.md](docs/UPGRADING_TO_2.md).
+
+## Reporting a Vulnerability
+
+If you discover a security vulnerability in Woods, please report it responsibly.
+
+**Do not open a public GitHub issue for security vulnerabilities.**
+
+Preferred: use GitHub's private vulnerability reporting, the **Report a
+vulnerability** button under the repository's
+[Security tab](https://github.com/lost-in-the/woods/security/advisories/new).
+It keeps the report, the discussion and the eventual advisory in one place, and
+it does not expose anything publicly until an advisory is published.
+
+If you cannot use GitHub, contact the maintainer privately through the address listed in the gem metadata.
+
+Either way, please include:
+
+1. A description of the vulnerability
+2. Steps to reproduce
+3. The potential impact
+4. The Woods version and Rails version you observed it on
+5. Any suggested fix (optional)
+
+## What to Expect
+
+- **Acknowledgment** within 48 hours of your report
+- **Assessment** within 1 week, we'll confirm whether it's a valid vulnerability and its severity
+- **Fix timeline** depends on severity:
+  - **Critical** (remote code execution, data exfiltration): Patch within 7 days
+  - **High** (privilege escalation, injection): Patch within 14 days
+  - **Medium/Low** (information disclosure, DoS): Patch in the next release
+
+## Disclosure Timeline
+
+- We follow a 90-day coordinated disclosure timeline
+- We'll credit you in the release notes (unless you prefer to remain anonymous)
+- We'll publish a security advisory on GitHub once the fix is released
+
+## Security Considerations
+
+Woods runs inside your Rails application and has access to:
+
+- **Application source code**: extracted and written to the output directory as JSON
+- **Database schema**: column names, types, indexes, and foreign keys (no row data)
+- **Git metadata**: commit history, contributors, file change frequency
+- **Runtime state** (Console MCP Server only), live database queries within a rolled-back transaction
+
+### Output Directory
+
+Extracted data is written to `tmp/woods/` by default. This directory contains your application's source code and schema in structured JSON format. Treat it with the same sensitivity as your source code, do not expose it to untrusted parties.
+
+### Console Server
+
+The Console MCP Server provides live database access through a five-layer defense-in-depth stack (feature gate, blocked tables, credential scanner, column redaction, and SqlValidator + rolled-back transactions). Only 9 read-only tools register by default; the optional Tier 4 read tools (`console_sql`, `console_query`) require explicit opt-in via `console_embedded_read_tools` and are constrained by `SqlValidator`'s read-only function allowlist plus rolled-back transactions. No executable tool requires confirmation or writes a privileged audit log, the confirmation/audit contracts belong to Tier 2/3 and `console_eval`, which are inventory-only and never registered. Rolled-back transactions do not undo async side effects (`perform_later`, `deliver_later`, HTTP egress), so treat the Console Server as an admin-trust boundary, not a sandbox: use it in development/staging only, never in production. See [docs/CONSOLE_MCP_SETUP.md. Safety Model](docs/CONSOLE_MCP_SETUP.md#safety-model) for the full breakdown.
+
+### MCP Transport
+
+The MCP Index Server supports both stdio and HTTP transports. stdio is the default and has no network exposure. The HTTP transport (`exe/woods-mcp-http`) refuses to bind a non-loopback host unless `WOODS_MCP_HTTP_TOKEN` is set and validates incoming `Authorization: Bearer …` headers. It also enforces a default `Origin` allow-list via `OriginGuard` to mitigate DNS-rebinding attacks. TLS is not terminated in-process, front the HTTP transport with a reverse proxy (nginx, caddy) when exposing it beyond loopback. See [docs/MCP_HTTP_TRANSPORT.md](docs/MCP_HTTP_TRANSPORT.md) for the full deployment guide.
+
+## Blast Radius
+
+If extraction output leaks, what can an attacker do with it?
+
+**What the output contains.** Application source code (inlined concerns, callback-resolved behavior), database schema (column names, types, indexes, foreign keys), route tables, migration history, gem versions, and git metadata (commit history, contributor emails, file change frequency).
+
+**What the output does not contain.** No row-level data from your database. Woods extracts schema only. No environment variables, no `Rails.application.credentials`, no API keys, no session state, no request logs, no customer data.
+
+| Leak scenario | Attacker gains | Attacker does not gain |
+|---|---|---|
+| `tmp/woods/` directory exfiltrated | Source code + schema equivalent to a git clone + `rails db:schema:dump` | Database rows, secrets, tokens, customer data |
+| MCP Index Server token leaked (HTTP transport) | Read-only query access to the extracted index, no write or execution paths | Shell access, database row data, secrets |
+| Notion sync database compromised | Model and column summaries synced to Notion | Anything not mirrored, source code stays local |
+| Console MCP Server exposed (dev/staging) | Read-only database access through a rolled-back transaction, bounded by TableGate + Redactor + SqlValidator | Write access (rolled back), full credentials (redacted), blocked tables |
+
+**Mitigation.** Treat `tmp/woods/` as source-equivalent, keep it out of world-readable directories and public container images. Rotate `WOODS_MCP_HTTP_TOKEN` on compromise. Keep `console_mcp_enabled = false` in production regardless of environment, since the console layers are defense-in-depth and not primary controls.

data/docs/AGENT_GUIDE.md

@@ -0,0 +1,204 @@
+# Woods agent guide
+
+This guide is for coding agents using an already connected Woods MCP server. Woods is evidence from the running Rails application and its extracted graph; it complements file search, tests, git history, and direct source inspection.
+
+## Start every session with status
+
+Call `woods_status` before relying on the index. Check:
+
+- the index is ready and has a current generation;
+- unit counts are non-zero for relevant types;
+- retrieval is enabled before choosing `codebase_retrieve`;
+- warnings do not indicate a stale or partial index.
+
+If status is unhealthy, report the evidence and ask the owner to extract or refresh. Do not fill gaps by asserting that Woods found nothing.
+
+## The default query loop
+
+Use this four-step loop for most codebase questions:
+
+1. **Discover** with `search` when you do not know the exact identifier.
+2. **Inspect** the best match with `lookup`.
+3. **Traverse** from that identifier with `dependencies`, `dependents`, or `trace_flow`.
+4. **Verify** important claims against the returned source paths and current repository files.
+
+Identifiers are namespaced and typed. Never invent one from a filename when `search` can return the exact value.
+
+## Pick the smallest useful tool
+
+| Need | Start with | Continue with |
+|---|---|---|
+| Find a class, route, callback, or phrase | `search` | `lookup` |
+| Understand one exact unit | `lookup` | `dependencies` or `dependents` |
+| Find what a change may break | `dependents` | `trace_flow`, then tests/source |
+| Understand what a unit calls or includes | `dependencies` | `lookup` on important nodes |
+| Follow request-to-model-to-view/job behavior | `trace_flow` | `lookup` at ambiguous steps |
+| Understand a unit's neighborhood | `structure` | targeted traversal |
+| Inspect Rails or gem behavior | `framework` | current source files |
+| Find recently changed indexed units | `recent_changes` | git diff/history |
+| Discover architectural domains | `domain_clusters` | `graph_analysis` |
+| Check a join or foreign key across databases before writing it | `graph_analysis` with `analysis: "cross_database_edges"` | `lookup` on both models |
+| Find dependencies that change faster than their dependents | `graph_analysis` with `analysis: "volatile_dependencies"` | `recent_changes` |
+| See a Packwerk boundary before calling across it | `graph_analysis` with `analysis: "undeclared_package_edges"` | `lookup` on the package unit |
+| Find central or high-impact units | `pagerank` | `dependents` |
+| Ask a conceptual question | `codebase_retrieve` if status says ready | `lookup` and graph tools |
+| Refresh after a published extraction | `reload` | `woods_status` |
+
+Do not start with a broad graph or semantic query when an exact search will answer the question with less noise.
+
+## Core workflows
+
+### Understand a model
+
+1. `search(query: "^Order$", types: ["model"])`
+2. `lookup(identifier: <returned identifier>)`
+3. Read resolved schema, associations, validations, scopes, enums, callbacks, and included concerns.
+4. `dependencies(identifier: ..., depth: 1)` for collaborators.
+5. `dependents(identifier: ..., depth: 1)` for callers and affected features.
+
+Woods may inline concern behavior beside the owning model. Distinguish the resolved runtime view from the physical file that originally defined a method.
+
+### Trace a feature flow
+
+1. Search for the route, controller action, job, mailer, or service at the user-visible entry point.
+2. Call `trace_flow` on the exact identifier.
+3. Inspect important or ambiguous nodes with `lookup`.
+4. Follow missing branches with `dependencies` and a narrow `via` filter when useful.
+5. Verify behavior that depends on conditions, dynamic dispatch, or runtime data in source and tests.
+
+### Assess change impact
+
+1. Search and look up the unit being changed.
+2. Call `dependents` at depth 1 before increasing depth.
+3. Group results by relationship type and application layer.
+4. Trace the most relevant user-facing or asynchronous flows.
+5. Use test mappings and repository search to select tests; do not equate a graph edge with test coverage.
+
+Report direct dependents separately from inferred downstream impact.
+
+### Diagnose missing context
+
+If an expected unit is absent:
+
+1. Check `woods_status` and generation time.
+2. Search by a broader literal prefix or suffix.
+3. Search relevant source or metadata fields.
+4. Confirm the extractor supports that unit type in [Extractor reference](EXTRACTOR_REFERENCE.md).
+5. Ask for `woods:incremental` or a full `woods:extract` when the index predates the code.
+
+“Not found in this generation” is evidence about the index, not proof that the code does not exist.
+
+## Search precisely
+
+`search` accepts a Ruby regular expression in `query`. It also supports literal `exact_prefix` and `exact_suffix`, which are safer for namespaced identifiers.
+
+Good patterns:
+
+```text
+query: "Order|Purchase", types: ["model", "service"]
+exact_prefix: "Admin::Billing::"
+exact_suffix: "Controller"
+fields: ["identifier", "source_code", "metadata"]
+```
+
+Start with identifier search. Add source or metadata only when name discovery fails. Restrict types and keep result limits small enough to inspect.
+
+## Traverse deliberately
+
+`dependencies` means “what this unit uses.” `dependents` means “what uses this unit.” Both default to bounded breadth-first traversal and accept type or relationship filters.
+
+Start at depth 1 or 2. A deeper unfiltered traversal can obscure the direct evidence that matters. Common relationship values include associations (`belongs_to`, `has_many`, `has_one`), code references, renders, redirects, form actions, and navigation links.
+
+Both return at most 50 nodes and say so with a `Showing N of M (truncated)`
+line. Narrow with `depth`, `types` and `via` before paging with `limit` and
+`offset`: narrowing answers the question, paging only splits the same answer
+across turns. In a multi-database app each row names the unit's database.
+
+Use returned relationship labels as evidence. Do not infer call order from a dependency edge alone.
+
+## Use semantic retrieval only when ready
+
+`codebase_retrieve` answers natural-language questions with token-budgeted context. Use it when `woods_status` reports a configured embedding provider and current vector data.
+
+Important parameters:
+
+- `query`: the conceptual question;
+- `budget`: token budget, default 8,000;
+- `types`: restrict results and opt specific types in;
+- `exclude_types`: remove noisy types.
+
+Do not pass `limit`; retrieval is governed by `budget`. Test mappings are excluded by default so filenames do not dominate semantic rank. After retrieval, verify key units with `lookup` rather than treating ranked context as exhaustive.
+
+If retrieval is disabled, use `search`, `lookup`, and graph tools. Do not request credentials or reconfigure the project without authorization.
+
+## Index Server boundaries
+
+The normal packaged server registers 14 tools:
+
+`woods_status`, `search`, `lookup`, `dependencies`, `dependents`, `structure`, `trace_flow`, `framework`, `recent_changes`, `graph_analysis`, `domain_clusters`, `pagerank`, `reload`, and `codebase_retrieve`.
+
+Source inventory contains conditional tools for sessions, pipeline operation, feedback, snapshots, and Notion. Do not call or promise them unless they appear in the connected server's tool list and their backing collaborator is configured.
+
+## Console Server boundaries
+
+Console MCP is live-data access, not another code-search mode. Use it only when the user has authorized that environment and question.
+
+The default executable registers:
+
+- health/schema: `console_status`, `console_schema`;
+- bounded records: `console_find`, `console_recent`, `console_sample`;
+- projections/counts: `console_pluck`, `console_count`, `console_aggregate`, `console_association_count`.
+
+`console_sql` and `console_query` appear only when embedded read tools are explicitly enabled. Tier 2 and Tier 3 inventory schemas and `console_eval` are not callable in supported packaged modes.
+
+Before every Console call:
+
+1. confirm the authorized environment;
+2. use the narrowest model, fields, filters, and limit;
+3. avoid retrieving sensitive columns when aggregates or counts answer the question;
+4. treat redaction and scanners as defense in depth, not permission;
+5. do not work around a block or validation error.
+
+See [Console MCP setup](CONSOLE_MCP_SETUP.md) for the safety model.
+
+## Report evidence clearly
+
+When answering from Woods:
+
+- name the tools and exact identifiers used;
+- distinguish direct Woods evidence from your inference;
+- cite source paths returned by Woods when available;
+- state the index generation or staleness caveat when relevant;
+- say when a conditional path still needs source or test verification;
+- never claim the index is complete merely because a tool returned successfully.
+
+A useful answer shape is:
+
+```text
+Finding: <plain-language result>
+Woods evidence: <tool + identifier + relationship/source>
+Inference: <what follows from that evidence>
+Verification: <source/test/history checked or still needed>
+```
+
+## Common mistakes
+
+| Mistake | Better approach |
+|---|---|
+| Guessing an identifier | Discover it with `search` |
+| Starting with semantic retrieval for an exact class | Use `search` then `lookup` |
+| Treating no result as proof of absence | Check status, generation, extractor coverage, and source |
+| Traversing deeply from the start | Begin at depth 1 or 2 and filter |
+| Calling inventory-only tools | Use the connected server's registered list |
+| Using Console for code structure | Use Index tools |
+| Asking for `limit` on retrieval | Use `budget` |
+| Trusting graph edges as execution order | Verify conditions and order in source/tests |
+| Changing MCP or credentials while answering | Report the missing capability and ask the owner |
+
+## Related documentation
+
+- [MCP servers](MCP_SERVERS.md): installation, client configuration, and exact surfaces.
+- [MCP tool cookbook](MCP_TOOL_COOKBOOK.md): detailed parameters and response examples.
+- [Extractor reference](EXTRACTOR_REFERENCE.md): indexed unit and edge contracts.
+- [Retrieval guide](RETRIEVAL_GUIDE.md): embeddings, ranking, and token budgets.
+- [Troubleshooting](TROUBLESHOOTING.md): stale indexes, disabled retrieval, and startup failures.

data/docs/AGENT_SETUP.md

@@ -0,0 +1,205 @@
+# Agent setup runbook
+
+Use this runbook when a coding agent installs or configures Woods 2.0 in an existing Rails repository. The goal is a small, reviewable change and a working structural Index Server. Semantic retrieval and live-data access are separate opt-ins.
+
+## Default decision
+
+Install the structural index only unless the user explicitly asks for another capability.
+
+| Capability | Default | Requires explicit approval when agent-operated |
+|---|---:|---|
+| Extract Rails code and schema structure | On | No |
+| Local or hosted embeddings | Off | Yes: adds a service, credentials, cost, or generated vectors |
+| Console MCP | Off | Yes: boots Rails and can read live application data |
+| HTTP MCP transport | Off | Yes: expands network exposure |
+| Purging or rebuilding durable stores | Off | Yes: can remove Woods-owned data |
+
+## 1. Preflight without changing files
+
+Read the repository's agent instructions first. Then record:
+
+```bash
+git status --short --branch
+ruby --version
+bundle exec rails --version
+bundle exec rails runner 'puts Rails.application.class.name'
+```
+
+Also determine:
+
+- whether Rails commands run on the host or through Docker Compose;
+- which Compose service owns the Rails process, if applicable;
+- the database adapter and whether migrations are allowed in this environment;
+- whether `woods` already appears in the Gemfile or lockfile;
+- whether `config/initializers/woods.rb`, Woods migrations, or Woods tables already exist;
+- whether `tmp/woods/` is ignored or intentionally published.
+
+If the worktree contains unrelated changes, preserve them. Do not overwrite an existing initializer or migration without showing the conflict to the user.
+
+## 2. Choose the installation path
+
+Use structural-only setup when the user wants code navigation, runtime Rails structure, dependencies, flows, or blast-radius analysis. Fourteen tools register in the normal packaged launch without an embedding provider.
+
+Discuss semantic retrieval only if the user needs natural-language `codebase_retrieve`. The choice depends on whether they prefer local Ollama or hosted OpenAI and which vector store fits their environment. See [Backend matrix](BACKEND_MATRIX.md).
+
+Do not infer permission to configure Console MCP from a request to “set up Woods” or “set up MCP.” The Index Server reads generated code context; the Console Server can read live data.
+
+## 3. Install on a branch
+
+Create or switch to the branch requested by the repository owner. Add only the development dependency:
+
+```ruby
+# Gemfile
+group :development do
+  gem "woods", "~> 2.0"
+end
+```
+
+Run the repository's normal dependency command:
+
+```bash
+bundle install
+bundle exec ruby -rwoods/version -e 'puts Woods::VERSION'
+```
+
+Do not broadly update unrelated gems. Review the Gemfile and lockfile diff before continuing.
+
+## 4. Generate and inspect
+
+```bash
+bin/rails generate woods:install
+git diff -- config/initializers/woods.rb db/migrate
+```
+
+The generator emits a legacy application migration for:
+
+- `woods_units`
+- `woods_edges`
+- `woods_embeddings`
+
+Woods 2's shipped structural index and storage backends do not use those application tables. For a new default installation, propose removing the generated migration from the working tree and get approval before doing so. Keep or run it only when repository history shows an older/custom integration uses the tables; confirm name conflicts and obtain explicit migration authorization first.
+
+Follow repository policy for generated files and schema changes. Never run a production migration as an incidental setup step.
+
+## 5. Extract and validate
+
+Use the same execution environment and boot variables the Rails app normally needs:
+
+```bash
+bin/rails woods:extract
+bin/rails woods:validate
+bin/rails woods:stats
+```
+
+If extraction fails, reproduce Rails boot and eager loading outside Woods before changing configuration:
+
+```bash
+bin/rails runner 'puts Rails.application.class.name'
+bin/rails runner 'Rails.application.eager_load!; puts "eager load ok"'
+```
+
+Fix one root cause at a time. Do not suppress an application boot error to make extraction appear successful.
+
+## 6. Configure the Index MCP client
+
+Prefer a project-scoped configuration so the executable, bundle, and index all belong to the same repository:
+
+```json
+{
+  "mcpServers": {
+    "woods": {
+      "command": "bundle",
+      "args": ["exec", "woods-mcp-start", "./tmp/woods"],
+      "cwd": "/absolute/path/to/the-rails-app"
+    }
+  }
+}
+```
+
+For Docker, extraction runs inside the Rails container. If Woods is installed only there, configure the client to launch `docker compose exec -T app bundle exec woods-mcp /app/tmp/woods` with the host application root as `cwd`. Use a host-side `bundle exec woods-mcp-start` only after verifying the host has a supported Ruby, the application bundle, and a host-visible index. In either mode, supply the path visible to the server process.
+
+Reconnect the client and call `woods_status`. Confirm a current generation and non-zero unit counts before claiming setup works.
+
+## 7. Verify useful behavior
+
+Use a class known to exist in the application:
+
+1. Call `search` to obtain its exact identifier.
+2. Call `lookup` to confirm source and metadata are present.
+3. Call `dependents` with depth 1 or 2 to confirm graph edges are queryable.
+
+If `codebase_retrieve` reports that semantic search is disabled, that is expected for structural-only setup. Do not configure credentials merely to remove the message.
+
+## 8. Offer automatic index maintenance
+
+Ask whether the owner wants Woods added to the development process manager. If authorized, use the repository's existing Procfile or equivalent convention:
+
+```text
+web:   bin/rails server
+woods: bundle exec rake woods:watch
+```
+
+The watcher catches up missed changes, maintains the structural index as files change, and publishes generations the Index MCP server detects automatically. Ordinary edits then need no manual re-extraction or MCP restart. It should run in development, not production.
+
+Report these boundaries in the handoff:
+
+- boot-captured changes make the watcher exit 75 and require supervisor restart;
+- container bind mounts may require `WOODS_WATCH_POLL=1`;
+- semantic vectors still require `woods:embed_incremental`;
+- without a resident watcher, the fallback is `woods:incremental` after changes.
+
+## Stop and ask before
+
+Get explicit user approval before:
+
+- enabling Console MCP or granting access to a live database;
+- enabling `console_embedded_read_tools`, `console_sql`, or `console_query`;
+- adding an API key, hosted embedding provider, Qdrant, pgvector, or Ollama service;
+- exposing MCP over HTTP, selecting bind addresses, or configuring bearer tokens;
+- overriding a purge guard or deleting/rebuilding Woods durable data;
+- overwriting an existing Woods initializer, migration, or MCP configuration;
+- changing production or shared infrastructure.
+
+## Handoff report
+
+Return a concise report the owner can verify:
+
+```text
+Woods version:
+Branch:
+
+Files changed:
+- Gemfile / lockfile:
+- initializer:
+- migration/schema:
+- MCP client configuration:
+
+Commands run:
+- install:
+- migrate:
+- extract:
+- validate/stats:
+
+Verified capabilities:
+- Index Server connected: yes/no
+- woods_status current: yes/no
+- search/lookup/dependents checked: yes/no
+- semantic retrieval: disabled/enabled (provider)
+- Console MCP: disabled/enabled (authorization)
+- automatic structural updates: disabled/enabled (process manager)
+
+Follow-up or unresolved risk:
+```
+
+Never report a capability as enabled solely because its schema exists in source. Report what the packaged executable actually registered and what you called successfully.
+
+## Copyable prompt for an installation agent
+
+> Install Woods 2.x in this Rails repository using `docs/AGENT_SETUP.md`. Start with read-only preflight and preserve unrelated changes. Default to the structural Index Server; do not enable embeddings, Console MCP, HTTP transport, secrets, or purge overrides without asking me. Inspect generated files before migrating, run extraction and validation in the app's normal execution environment, configure a project-scoped MCP server in the same filesystem context as the application bundle and index, and verify `woods_status`, `search`, `lookup`, and `dependents`. Finish with the runbook's handoff report.
+
+## Related guides
+
+- [Getting started](GETTING_STARTED.md) for the human walkthrough.
+- [MCP servers](MCP_SERVERS.md) for client-specific configuration and server boundaries.
+- [Upgrade to Woods 2.0](UPGRADING_TO_2.md) for an existing 1.x installation.
+- [Troubleshooting](TROUBLESHOOTING.md) for extraction and connection failures.