viepilot 2.4.0 → 2.12.0

package/workflows/crystallize.md

@@ -104,6 +104,520 @@ Ask the user for project information:
 Store all metadata for template generation.
 </step>
 
+<step name="brownfield_detection">
+## Step 0-B: Brownfield Mode Detection (FEAT-018)
+
+**Trigger brownfield mode when ANY of the following is true:**
+- `--brownfield` flag passed explicitly, OR
+- `docs/brainstorm/` directory does not exist or is empty (no `session-*.md` files) **AND** `.viepilot/` directory does not yet exist
+
+**If NEITHER condition is met** (greenfield path): skip this entire step; proceed to Step 1.
+
+**If brownfield triggered AND `.viepilot/` already exists:**
+- Warn: "`.viepilot/` already exists. Re-running brownfield mode will overwrite artifacts."
+- Ask: "Continue? (y/n)" — abort if n.
+
+**When brownfield mode is active:**
+1. Run the full 12-category codebase scanner (Signal Categories 1–12 below).
+2. Produce a structured **Scan Report** (see schema at end of this step).
+3. Classify every field as DETECTED / ASSUMED / MISSING per Gap Detection Rules.
+4. Present Scan Report summary to user; interactively fill every MISSING MUST-DETECT field.
+5. User confirms ASSUMED fields (may accept all or override individually).
+6. After confirmation: proceed to Step 0-C (brainstorm stub generation).
+7. Then skip Step 1 (`analyze_brainstorm`); continue from Step 0 (metadata collection) → Step 2 onward using the confirmed Scan Report as input.
+
+---
+
+### Signal Category 1 — Build Manifest & Package Identity
+
+Probe the following files in order of priority (first match per language wins):
+
+| Language / Platform | Files to probe |
+|---------------------|----------------|
+| Node.js | `package.json` |
+| Java / Maven | `pom.xml` |
+| Java / Gradle | `build.gradle`, `build.gradle.kts`, `settings.gradle` |
+| Python | `pyproject.toml`, `setup.py`, `setup.cfg`, `requirements.txt`, `Pipfile` |
+| Rust | `Cargo.toml` |
+| Go | `go.mod` |
+| PHP | `composer.json` |
+| Ruby | `Gemfile`, `*.gemspec` |
+| .NET | `*.csproj`, `*.sln`, `global.json` |
+| Elixir / Erlang | `mix.exs` |
+| Swift | `Package.swift` |
+
+**Fields to extract per manifest:**
+- `project_name` — package name / artifactId / module name
+- `current_version` — version field
+- `primary_language` — derived from manifest type
+- `runtime_version` — engines/node, java.version, python_requires, go version
+- `entry_points` — main, bin, spring-boot:run target, etc.
+- `raw_dependencies[]` — full dependency list (used by Signal Category 2)
+- `raw_dev_dependencies[]` — dev/test deps
+
+**Monorepo detection** (check before single-manifest scan):
+- npm workspaces: `package.json` → `"workspaces"` field + `packages/*/package.json`
+- Nx / Turborepo: `nx.json`, `turbo.json` + `apps/*/` and `libs/*/`
+- Maven multi-module: `pom.xml` with `<modules>`
+- Gradle multi-project: `settings.gradle` with `include`
+- Cargo workspace: `Cargo.toml` with `[workspace]`
+- Go workspace: `go.work`
+
+If monorepo detected → scan each module separately; aggregate into `modules[]` in Scan Report.
+
+If no manifest found → `primary_language` = MISSING; user must provide.
+
+---
+
+### Signal Category 2 — Framework & Library Detection
+
+Infer from `raw_dependencies[]` + `raw_dev_dependencies[]`:
+
+**Backend frameworks:**
+| Signal pattern | Framework |
+|---------------|-----------|
+| `spring-boot-starter-*` | Spring Boot |
+| `express`, `fastify`, `koa`, `hapi` | Node.js HTTP |
+| `nestjs/core` | NestJS |
+| `fastapi`, `flask`, `django` | Python HTTP |
+| `gin-gonic/gin`, `gofiber/fiber`, `labstack/echo` | Go HTTP |
+| `rails`, `sinatra` | Ruby |
+| `laravel/framework`, `slim/slim` | PHP |
+| `actix-web`, `axum`, `rocket` | Rust HTTP |
+| `phoenix` (mix.exs) | Elixir |
+| `Microsoft.AspNetCore.*` | .NET ASP.NET Core |
+
+**Frontend frameworks:**
+| Signal pattern | Framework |
+|---------------|-----------|
+| `react`, `react-dom` | React |
+| `vue` | Vue.js |
+| `@angular/core` | Angular |
+| `svelte` | Svelte |
+| `next` | Next.js |
+| `nuxt` | Nuxt.js |
+| `remix` | Remix |
+| `astro` | Astro |
+| `solid-js` | SolidJS |
+
+**ORM / Database clients:**
+| Signal pattern | ORM / DB |
+|---------------|----------|
+| `mybatis-spring-boot-starter`, `mybatis` | MyBatis |
+| `spring-boot-starter-data-jpa`, `hibernate-*` | Hibernate / JPA |
+| `typeorm` | TypeORM |
+| `prisma` | Prisma |
+| `sequelize` | Sequelize |
+| `sqlalchemy`, `alembic` | SQLAlchemy |
+| `gorm.io/gorm` | GORM |
+| `mongoose` | Mongoose (MongoDB) |
+| `pg`, `mysql2`, `mariadb`, `better-sqlite3` | Raw SQL clients |
+| `redis`, `ioredis` | Redis client |
+
+**Message broker clients:**
+| Signal pattern | Broker |
+|---------------|--------|
+| `spring-kafka`, `kafka-clients` | Apache Kafka |
+| `amqplib`, `spring-rabbit` | RabbitMQ |
+| `@aws-sdk/client-sqs` | AWS SQS |
+| `nats` | NATS |
+| `bullmq`, `bull` | Redis-based queue |
+
+**Auth libraries:**
+| Signal pattern | Auth mechanism |
+|---------------|----------------|
+| `spring-security`, `spring-boot-starter-security` | Spring Security |
+| `jsonwebtoken`, `jose`, `passport-jwt` | JWT |
+| `passport` | OAuth / multi-strategy |
+| `keycloak-*`, `keycloak-connect` | Keycloak |
+| `next-auth`, `@auth/core` | NextAuth |
+| `authlib`, `python-jose`, `djangorestframework-simplejwt` | Python auth |
+
+**Test frameworks:**
+| Signal pattern | Framework |
+|---------------|-----------|
+| `jest`, `@jest/core` | Jest |
+| `vitest` | Vitest |
+| `mocha` | Mocha |
+| `jasmine` | Jasmine |
+| `junit`, `spring-boot-starter-test` | JUnit |
+| `pytest` | pytest |
+| `rspec-rails`, `rspec` | RSpec |
+| `cypress` | Cypress E2E |
+| `playwright` | Playwright E2E |
+| `@testing-library/*` | Testing Library |
+| `testify` | Testify (Go) |
+
+Rule: If zero backend AND zero frontend frameworks detected → add entry to `open_questions[]` and ask user.
+
+---
+
+### Signal Category 3 — Architecture Layer Inference (Directory Structure)
+
+Glob the following path patterns; presence implies the corresponding layer:
+
+| Pattern(s) | Inferred layer |
+|------------|----------------|
+| `src/main/java/**`, `src/main/kotlin/**` | Java/Kotlin Maven standard layout |
+| `src/controllers/`, `app/controllers/`, `**/controller/**` | MVC — Controller layer |
+| `src/api/`, `api/`, `routes/`, `src/routes/` | API / Router layer |
+| `src/services/`, `services/`, `**/service/**` | Service / Business logic layer |
+| `src/repositories/`, `repositories/`, `**/repository/**`, `**/dao/**` | Repository / DAO layer |
+| `src/models/`, `models/`, `**/model/**`, `**/entity/**` | Domain models / Entities |
+| `src/middleware/`, `middleware/` | Middleware layer |
+| `src/utils/`, `utils/`, `helpers/`, `common/` | Utilities / Shared |
+| `frontend/`, `client/`, `web/`, `src/client/` | Frontend module |
+| `backend/`, `server/`, `src/server/` | Backend module |
+| `packages/`, `apps/`, `libs/` | Monorepo layout |
+| `infrastructure/`, `infra/`, `terraform/`, `helm/`, `k8s/`, `kubernetes/` | Infrastructure / IaC |
+| `scripts/`, `bin/`, `tools/` | Developer tooling |
+| `public/`, `static/`, `assets/` | Static assets |
+| `docs/`, `documentation/` | Documentation |
+| `config/`, `configs/`, `settings/` | Configuration |
+| `tests/`, `test/`, `__tests__/`, `spec/`, `e2e/` | Test suite root |
+| `migrations/`, `db/migrate/`, `alembic/` | Database migrations |
+
+Output: `architecture_layers[]` — each with `{ layer, evidence_path }`.
+
+---
+
+### Signal Category 4 — Database Schema Signals
+
+Probe for migration/schema evidence:
+
+| Pattern | Evidence type |
+|---------|---------------|
+| `db/migrate/*.rb` | Rails ActiveRecord migrations |
+| `migrations/*.sql`, `migrations/*.js`, `migrations/*.ts` | Generic migrations |
+| `src/main/resources/db/migration/V*.sql` | Flyway |
+| `db/changelog*.xml`, `db/changelog*.yaml` | Liquibase |
+| `alembic/versions/*.py` | SQLAlchemy Alembic |
+| `src/migrations/**` | TypeORM / Sequelize migrations |
+| `prisma/schema.prisma` | Prisma schema |
+| `**/schema.sql`, `**/init.sql`, `**/seed.sql` | Raw SQL schema / seed |
+| `docker-compose.yml` → service names | Implied DB types (postgres, mysql, mongo, redis…) |
+
+Output: `database_signals[]` — each with `{ type, evidence_path, migration_tool }`.
+
+Rule: If none found but ORM dep exists → `database_signals` = ASSUMED with rationale note.
+
+---
+
+### Signal Category 5 — API Contract Files
+
+| File pattern | Contract type |
+|-------------|---------------|
+| `openapi.yaml`, `openapi.json`, `swagger.yaml`, `swagger.json` | OpenAPI / Swagger |
+| `api-docs.yaml`, `api.yaml`, `api-spec.yaml`, `api/*.yaml` | OpenAPI (alternate) |
+| `*.proto`, `proto/**/*.proto` | gRPC / Protocol Buffers |
+| `schema.graphql`, `**/*.graphql` | GraphQL |
+| `src/main/resources/static/v3/api-docs*` | Spring Swagger (generated) |
+| `postman_collection.json`, `*.postman_collection.json` | Postman |
+| `insomnia.yaml`, `.insomnia/` | Insomnia |
+
+`api_style` inference:
+- REST → if OpenAPI/Swagger file found
+- gRPC → if `*.proto` found (set `mixed` if REST also found)
+- GraphQL → if `schema.graphql` / `*.graphql` found
+- ASSUMED REST → if REST framework detected but no contract file found
+- MISSING → if api_style still unknown after all signals; user prompted
+
+---
+
+### Signal Category 6 — Infrastructure & Deployment Configuration
+
+| File pattern | Deployment signal |
+|-------------|------------------|
+| `Dockerfile` | Containerized — read `FROM` for base image |
+| `docker-compose.yml`, `docker-compose*.yml` | Service topology (list all services) |
+| `.github/workflows/*.yml` | GitHub Actions CI/CD |
+| `.gitlab-ci.yml` | GitLab CI |
+| `Jenkinsfile` | Jenkins pipeline |
+| `.circleci/config.yml` | CircleCI |
+| `bitbucket-pipelines.yml` | Bitbucket Pipelines |
+| `k8s/**/*.yaml`, `kubernetes/**/*.yaml` | Kubernetes manifests |
+| `helm/**`, `Chart.yaml` | Helm charts |
+| `terraform/**/*.tf` | Terraform IaC |
+| `pulumi/**`, `Pulumi.yaml` | Pulumi IaC |
+| `ansible/**`, `playbook.yml` | Ansible |
+| `nginx.conf`, `nginx/**` | Nginx reverse proxy |
+| `serverless.yml`, `serverless.ts` | Serverless Framework |
+| `fly.toml` | Fly.io |
+| `vercel.json`, `.vercel/` | Vercel |
+| `netlify.toml` | Netlify |
+| `render.yaml` | Render |
+
+Output: `deployment_signals[]` — each with `{ platform, file_path, notes }`.
+
+Rule: If no deployment signal found → `deployment_signals` = MISSING; user prompted.
+
+---
+
+### Signal Category 7 — Environment & Configuration Shape
+
+| File pattern | Purpose |
+|-------------|---------|
+| `.env.example`, `.env.sample`, `.env.template` | Required env key shape |
+| `application.properties`, `application.yml`, `application-*.yml` | Spring Boot config |
+| `config/database.yml` | Rails DB config |
+| `config/settings.py`, `config/*.py` | Django / Python config |
+| `appsettings.json`, `appsettings.*.json` | .NET config |
+| `config/config.exs`, `config/runtime.exs` | Elixir config |
+| `config.yaml`, `config.yml` (project root or config/) | Generic config |
+
+**Rule:** Read `.env.example` / `.env.sample` / `.env.template` to extract key names into `config_keys[]`.
+**SAFETY: Never read `.env` (live secrets file) — scanner explicitly skips it.**
+
+---
+
+### Signal Category 8 — Test Coverage Signals
+
+| File pattern | Test framework |
+|-------------|----------------|
+| `jest.config.js`, `jest.config.ts`, `jest.config.mjs` | Jest |
+| `vitest.config.ts`, `vitest.config.js` | Vitest |
+| `.mocharc.js`, `.mocharc.yaml` | Mocha |
+| `pytest.ini`, `setup.cfg [tool:pytest]`, `pyproject.toml [tool.pytest.ini_options]` | pytest |
+| `karma.conf.js` | Karma |
+| `cypress.config.js`, `cypress.config.ts`, `cypress.json` | Cypress |
+| `playwright.config.ts`, `playwright.config.js` | Playwright |
+| `phpunit.xml`, `phpunit.xml.dist` | PHPUnit |
+
+**Coverage indicators:** presence of `coverage/`, `htmlcov/`, `.nyc_output/`, `target/site/jacoco/` → `has_coverage_reports = true`.
+
+Output: `test_frameworks[]`, `test_root_dirs[]`, `has_coverage_reports`.
+
+Rule: If no test signals found → `test_frameworks` = MISSING; note added to generated `SYSTEM-RULES.md` quality gates section.
+
+---
+
+### Signal Category 9 — Code Quality & Tooling
+
+| File pattern | Tool |
+|-------------|------|
+| `.eslintrc*`, `eslint.config.*` | ESLint |
+| `.prettierrc*`, `prettier.config.*` | Prettier |
+| `sonar-project.properties`, `sonar-project.yml` | SonarQube |
+| `.pre-commit-config.yaml` | pre-commit hooks |
+| `checkstyle.xml` | Checkstyle (Java) |
+| `.pylintrc`, `pylint.cfg` | Pylint |
+| `.flake8`, `setup.cfg [flake8]` | Flake8 |
+| `mypy.ini`, `pyproject.toml [tool.mypy]` | mypy |
+| `rustfmt.toml`, `.rustfmt.toml` | rustfmt |
+| `.golangci.yml` | golangci-lint |
+| `.editorconfig` | EditorConfig |
+| `commitlint.config.js`, `.commitlintrc*` | Conventional Commits enforcement |
+| `.husky/` | Husky git hooks |
+| `lint-staged.config.*`, `package.json "lint-staged"` | lint-staged |
+
+Output: `quality_tools[]` — used to populate `SYSTEM-RULES.md` quality gates section.
+
+---
+
+### Signal Category 10 — Documentation Files
+
+| File | Priority |
+|------|----------|
+| `README.md` | MUST-READ — extract: project name, description, quickstart |
+| `CHANGELOG.md`, `HISTORY.md`, `RELEASES.md` | SHOULD-READ — extract: version history, latest changes |
+| `CONTRIBUTING.md` | SHOULD-READ — extract: contribution rules |
+| `ARCHITECTURE.md`, `docs/architecture*.md` | SHOULD-READ — extract: any existing arch notes |
+| `docs/adr/`, `ADR/`, `decisions/` | SHOULD-READ — Architecture Decision Records |
+| `docs/**/*.md` (top 10 by mtime) | NICE-TO-READ — project-specific docs |
+| `LICENSE` | MUST-READ — extract license type |
+
+Output: `docs_extracted[]` — each with `{ file, summary, key_facts[] }`.
+
+Rule: If `README.md` absent → `project_name` elevated to MISSING (must ask user).
+
+---
+
+### Signal Category 11 — Git History & Version Signals
+
+Run the following git commands (read-only):
+
+| Command | Purpose |
+|---------|---------|
+| `git log --oneline -100` | Commit message patterns (Conventional Commits? Jira refs? free-form?) |
+| `git tag --sort=-version:refname \| head -20` | Version history + tag naming convention |
+| `git log --format="%H %s" --diff-filter=A -- "*.md" \| head -20` | When key docs were added |
+| `git branch -a \| head -20` | Branch naming convention |
+| `git log --stat -3` | Most recently changed files |
+| `git shortlog -sn --no-merges \| head -10` | Top contributors |
+| `git remote get-url origin` | Repository URL |
+
+Extract: `commit_convention` (Conventional Commits / Jira-ref / free-form / mixed), `version_pattern` (semver / calver / custom), `latest_tag`, `active_branches[]`, `top_contributors[]`, `repo_url`.
+
+Rule: If not a git repo → all git fields = MISSING; user warned.
+
+---
+
+### Signal Category 12 — File Extension Language Survey
+
+Glob source file extensions to detect secondary languages:
+
+```
+Glob: src/**/*.{ts,tsx,js,jsx,py,java,kt,go,rs,rb,php,cs,swift,ex,exs,scala,clj,elm,dart}
+(also check project root if no src/ directory)
+```
+
+Count files per extension → `language_distribution{}` (e.g. `{ ts: 142, java: 38, sql: 12 }`).
+
+Rule: `secondary_languages[]` = languages with ≥5 files that are not `primary_language`.
+
+---
+
+### Scan Report Schema (finalized)
+
+After running all 12 signal categories, produce this structured Scan Report:
+
+```yaml
+# ViePilot Brownfield Scan Report
+project_name: string           # from manifest or README
+current_version: string        # from manifest or latest git tag
+primary_language: string       # from manifest
+secondary_languages: []        # from file extension survey
+runtime_version: string        # node/java/python/go version
+frameworks:
+  backend: []
+  frontend: []
+  orm: []
+  auth: []
+  message_broker: []
+build_tool: string
+package_manager: string
+monorepo: bool
+modules: []                    # if monorepo
+architecture_layers: []        # { layer, evidence_path }
+database_signals: []           # { type, evidence_path, migration_tool }
+api_contracts: []              # { style, file_path }
+api_style: string              # REST | GraphQL | gRPC | mixed | unknown
+deployment_signals: []         # { platform, file_path }
+test_frameworks: []
+test_root_dirs: []
+has_coverage_reports: bool
+quality_tools: []
+config_keys: []                # from .env.example (key names only — no values)
+commit_convention: string      # conventional | jira | free-form | mixed
+version_pattern: string        # semver | calver | custom
+latest_tag: string
+repo_url: string
+top_contributors: []
+docs_extracted: []             # { file, summary, key_facts[] }
+language_distribution: {}      # { ts: 142, java: 38, ... }
+open_questions: []             # fields not resolved by user input
+```
+
+---
+
+### Gap Detection Rules
+
+Every field in Scan Report is classified as:
+
+| Status | Meaning | Required action |
+|--------|---------|-----------------|
+| **DETECTED** | Inferred from codebase with high confidence | Show for confirmation — no user action required |
+| **ASSUMED** | Inferred with low confidence or indirect signal | Show to user with rationale; user may correct |
+| **MISSING** | Not found anywhere in codebase | **Must ask user** before generating artifacts |
+
+**MUST-DETECT fields** (MISSING = hard blocker — cannot generate artifacts until user fills):
+- `project_name`
+- `primary_language`
+- At least one entry in `frameworks.backend` OR `frameworks.frontend`
+- `current_version`
+
+**SHOULD-DETECT fields** (MISSING = warning — document assumption, continue with user acknowledgment):
+- `api_style`
+- At least one entry in `database_signals` (if ORM dep found)
+- `test_frameworks`
+- `commit_convention`
+- `deployment_signals`
+
+**NICE-TO-DETECT fields** (MISSING = note only — generate with placeholder):
+- `auth` frameworks, `message_broker`, `config_keys`, `top_contributors`, `has_coverage_reports`
+
+**Assumption documentation rule:** For every ASSUMED or MISSING field that proceeds without user fill, append to `open_questions[]` and insert a `> ⚠️ Assumed: {rationale}` callout in the relevant `.viepilot/` artifact section.
+
+---
+
+### Interactive Gap-Filling (Step 0-B-ii)
+
+After scanner completes:
+
+1. Display Scan Report summary table to user (field | value | status).
+2. For each MUST-DETECT field that is MISSING → **pause and ask user to provide value**. Do not proceed until all MUST-DETECT fields are filled.
+3. Present ASSUMED fields in a confirmation table → user may accept all with "y" or override individually.
+4. Capture all user responses; update Scan Report fields accordingly.
+5. All remaining unresolved items → `open_questions[]`.
+
+---
+
+### Brownfield Brainstorm Stub Generation (Step 0-C)
+
+After gap-filling is complete, write:
+
+**Path:** `docs/brainstorm/session-brownfield-import.md`
+
+**Content:**
+```markdown
+# Brownfield Import — {project_name}
+
+## Meta
+- **Import date**: {scan_date}
+- **Import source**: `vp-crystallize --brownfield`
+- **Scanner version**: FEAT-018
+
+## Scan Report
+
+\`\`\`yaml
+{full confirmed Scan Report YAML}
+\`\`\`
+```
+
+**Purpose:** This stub allows `vp-audit` and other ViePilot tools to not error on missing brainstorm session files. The presence of `session-brownfield-import.md` is treated as a valid brownfield import.
+
+---
+
+### Safety Rules
+
+The brownfield scanner MUST:
+
+```
+NEVER read:
+  .env                    (live secrets)
+  *.key, *.pem, *.p12, *.jks, id_rsa, id_ed25519
+
+ALWAYS skip these directories:
+  node_modules/
+  .git/
+  target/
+  build/
+  dist/
+  __pycache__/
+  .venv/
+  vendor/
+
+NEVER write any files until the user has confirmed the Scan Report.
+
+NEVER overwrite existing .viepilot/ without explicit user confirmation (y/n prompt).
+```
+
+---
+
+### TRACKER.md Continuity Annotation
+
+When generating TRACKER.md in Step 9 (brownfield mode only), append:
+
+```markdown
+## Brownfield Import
+- **Import date**: {scan_date}
+- **Imported version**: {current_version}
+- **Note**: Project history pre-dates ViePilot adoption.
+- **Scan Report**: `docs/brainstorm/session-brownfield-import.md`
+```
+
+</step>
+
 <step name="analyze_brainstorm">
 ## Step 1: Analyze Brainstorm