@libar-dev/architect 1.0.0-pre.3

package/docs-live/decisions/adr-021-doc-generation-proof-of-concept.md

@@ -0,0 +1,489 @@
+# ADR-021: Doc Generation Proof Of Concept
+
+**Purpose:** Architecture decision record for Doc Generation Proof Of Concept
+
+---
+
+## Overview
+
+| Property | Value         |
+| -------- | ------------- |
+| Status   | superseded    |
+| Category | documentation |
+| Phase    | 27            |
+
+**Status: SUPERSEDED** - This POC has been implemented. See:
+
+- Convention-tagged decision records (ADR/PDR) with @architect-convention tags
+- `docs-generated/ANNOTATION-GUIDE.md` - Comprehensive guide for fixing generated docs
+
+This decision establishes the pattern for generating technical documentation
+from annotated source files. It serves as both the DECISION (why/how) and
+the PROOF OF CONCEPT (demonstrating the pattern works).
+
+## Context
+
+**Invariant:** Documentation must be generated from annotated source code, never manually maintained as a separate artifact.
+
+**Rationale:** Manual documentation drifts from source as the codebase evolves, creating stale references that mislead both humans and AI coding sessions.
+
+| Document                     | Lines    | Maintenance Burden                |
+| ---------------------------- | -------- | --------------------------------- |
+| docs/PROCESS-GUARD.md        | ~300     | High - duplicates code behavior   |
+| docs/METHODOLOGY.md          | ~400     | Medium - conceptual, changes less |
+| \_claude-md/validation/\*.md | ~50 each | High - must match detailed docs   |
+| CLAUDE.md                    | ~800     | Very High - aggregates everything |
+
+| Gap                              | Impact | Solution                               |
+| -------------------------------- | ------ | -------------------------------------- |
+| Shape extraction from TypeScript | High   | New @extract-shapes tag                |
+| Convention-tagged content        | Medium | Decision records as convention sources |
+| Durable intro/context content    | Medium | Decision Rule: Context sections        |
+
+**The Problem:**
+
+    Common technical documentation is the hardest part to maintain in a repository.
+    The volume constantly grows, and AI coding sessions are drastically less effective
+    at updating documentation compared to code. Documentation drifts from source.
+
+    Current state in this package:
+
+    **Root Causes:**
+
+    1. **Duplication** - Same information exists in code comments, feature files,
+       and markdown docs. Changes require updating multiple places.
+
+    2. **No Single Source** - Documentation is authored separately from the code
+       it describes. There's no compilation step to catch drift.
+
+    3. **Detail Level Mismatch** - Compact docs for AI context and detailed docs
+       for humans are maintained separately despite sharing content.
+
+    **What We Have:**
+
+    The Architect package already has the required ingredients:
+    - Pattern extraction from TypeScript JSDoc and Gherkin tags
+    - Rich content support (DocStrings, tables, code blocks in features)
+    - Multi-source aggregation via tag taxonomy
+    - Progressive disclosure via codec detail levels
+    - Relationship tags for cross-references
+
+    **What's Missing:**
+
+**Verified by:**
+
+- Full pipeline generates documentation from decision documents
+
+## Decision
+
+**Invariant:** Each content type (intro/rationale, rules/examples, API types) is owned by exactly one source type (decision, behavior spec, or code).
+
+**Rationale:** Shared ownership leads to conflicting updates and ambiguous authority over what the "correct" version is.
+
+| Source Type                  | Durability | Content Ownership                      |
+| ---------------------------- | ---------- | -------------------------------------- |
+| Decision documents (ADR/PDR) | Permanent  | Intro, context, rationale, conventions |
+| Behavior specs (.feature)    | Permanent  | Rules, examples, acceptance criteria   |
+| Implementation code (.ts)    | Compiled   | API types, error messages, signatures  |
+
+| Rule Prefix      | ADR Section            | Doc Section                  |
+| ---------------- | ---------------------- | ---------------------------- |
+| `Context...`     | context                | ## Background / Introduction |
+| `Decision...`    | decision               | ## How It Works              |
+| `Consequence...` | consequences           | ## Trade-offs                |
+| Other rules      | other (warning logged) | Custom sections              |
+
+| Target Document                         | Sources                               | Detail Level | Effect                   |
+| --------------------------------------- | ------------------------------------- | ------------ | ------------------------ |
+| docs/PROCESS-GUARD.md                   | This decision + behavior specs + code | detailed     | All sections, full JSDoc |
+| \_claude-md/validation/process-guard.md | This decision + behavior specs + code | summary      | Rules table, types only  |
+
+| Level    | Content Included                     | Rendering Style                |
+| -------- | ------------------------------------ | ------------------------------ |
+| summary  | Essential tables, type names only    | Compact - lists vs code blocks |
+| standard | Tables, types, key descriptions      | Balanced                       |
+| detailed | Everything including JSDoc, examples | Full - code blocks with JSDoc  |
+
+| Source                          | What's Extracted                 | How                       |
+| ------------------------------- | -------------------------------- | ------------------------- |
+| Decision Rule: Context          | Intro/background section         | Rule description text     |
+| Decision Rule: Decision         | How it works section             | Rule description text     |
+| Decision Rule: Consequences     | Trade-offs section               | Rule description text     |
+| Decision DocStrings             | Code examples (Husky, API)       | Fenced code blocks        |
+| Behavior spec Rules             | Validation rules, business rules | Rule names + descriptions |
+| Behavior spec Scenario Outlines | Decision tables, lookup tables   | Examples tables           |
+| TypeScript @extract-shapes      | API types, interfaces            | AST extraction            |
+| TypeScript JSDoc                | Implementation notes             | Markdown in comments      |
+
+**The Pattern:**
+
+    Documentation is generated from three source types with different durability:
+
+
+    **Why Decisions Own Intro Content:**
+
+    Tier 1 specs (roadmap features) become clutter after implementation - their
+    deliverables are done, status is completed, they pile up. Behavior specs stay
+    current because tests must pass. But neither is appropriate for intro content.
+
+    Decisions (ADR/PDR) are durable by design - they remain valid until explicitly
+    superseded. The `Rule: Context` section of a decision IS the background/intro
+    for any documentation about that topic.
+
+    **Extends Existing ADR Codec:**
+
+    The doc-from-decision generator extends the existing `AdrDocumentCodec` which
+    already parses Rule: prefixes via `partitionAdrRules()` (see adr.ts:627-663):
+
+
+    **Source Mapping Pattern:**
+
+    Each documentation decision declares its target documents and source mapping:
+
+
+    **Detail Level Mapping:**
+
+    Uses existing `DetailLevel` enum from `renderable/codecs/types/base.ts`:
+
+
+    **Extraction by Source Type:**
+
+
+    **The Generator Command:**
+
+```bash
+generate-docs --decisions 'specs/**/*.feature' --features 'tests/**/*.feature' --typescript 'src/**/*.ts' --generators doc-from-decision --output docs
+```
+
+**Verified by:**
+
+- Decision Rule descriptions become documentation sections; Decision DocStrings become code examples
+
+**Invariant:** The source mapping table in a decision document defines how documentation sections are assembled from multiple source files.
+
+**Rationale:** Without a declarative mapping, generators must hard-code source-to-section relationships, making the system brittle to new document types.
+
+| Column            | Purpose                                      | Example                          |
+| ----------------- | -------------------------------------------- | -------------------------------- |
+| Section           | Target section heading in generated doc      | "Intro & Context", "API Types"   |
+| Source File       | Path to source file or self-reference marker | "src/types.ts", "THIS DECISION"  |
+| Extraction Method | How to extract content from source           | "@extract-shapes", "Rule blocks" |
+
+| Marker                    | Meaning                                            |
+| ------------------------- | -------------------------------------------------- |
+| THIS DECISION             | Extract from the current decision document         |
+| THIS DECISION (Rule: X)   | Extract specific Rule: block from current document |
+| THIS DECISION (DocString) | Extract fenced code blocks from current document   |
+
+| Extraction Method          | Source Type              | Action                                               |
+| -------------------------- | ------------------------ | ---------------------------------------------------- |
+| Decision rule description  | Decision (.feature)      | Extract Rule: block content (Invariant, Rationale)   |
+| @extract-shapes tag        | TypeScript (.ts)         | Invoke shape extractor for @architect-extract-shapes |
+| Rule blocks                | Behavior spec (.feature) | Extract Rule: names and descriptions                 |
+| Scenario Outline Examples  | Behavior spec (.feature) | Extract Examples tables as markdown                  |
+| JSDoc section              | TypeScript (.ts)         | Extract markdown from JSDoc comments                 |
+| createViolation() patterns | TypeScript (.ts)         | Extract error message literals                       |
+| Fenced code block          | Decision (.feature)      | Extract DocString code blocks with language          |
+
+**Table Format:**
+
+    **Self-Reference Markers:**
+
+
+    **Extraction Method Dispatch:**
+
+
+    **Path Resolution:**
+
+    - Relative paths are resolved from project root
+    - `THIS DECISION` resolves to the current decision document
+    - Missing files produce warnings but generation continues
+
+**Verified by:**
+
+- Source mapping aggregates multiple files; Source mapping validated at generation time
+
+## Consequences
+
+**Invariant:** Decision documents remain the authoritative source for intro, rationale, and convention content until explicitly superseded.
+
+**Rationale:** Without durable ownership, documentation sections lose their authoritative source and degrade into unattributed prose that no one updates.
+
+| Benefit                | How                                       |
+| ---------------------- | ----------------------------------------- |
+| Single source of truth | Each content type owned by one source     |
+| Always-current docs    | Generated from tested/compiled sources    |
+| Reduced maintenance    | Change source once, docs regenerate       |
+| Progressive disclosure | Same sources → compact + detailed outputs |
+| Clear ownership        | Decisions own "why", code owns "what"     |
+
+| Trade-off                                         | Mitigation                               |
+| ------------------------------------------------- | ---------------------------------------- |
+| Decisions must be updated for fundamental changes | Appropriate - fundamentals ARE decisions |
+| New @extract-shapes capability required           | Spec created (shape-extraction.feature)  |
+| Initial annotation effort on existing code        | One-time migration, then maintained      |
+| Generated docs in git history                     | Same as current manual approach          |
+
+| Content Type                | Owner                     | Update Trigger                  |
+| --------------------------- | ------------------------- | ------------------------------- |
+| Intro, rationale, context   | Decision document         | Fundamental change to approach  |
+| Rules, examples, edge cases | Behavior specs            | Behavior change (tests fail)    |
+| API types, signatures       | Code with @extract-shapes | Interface change (compile fail) |
+| Error messages              | Code patterns             | Message text change             |
+| Code examples               | Decision DocStrings       | Example needs update            |
+
+**Benefits:**
+
+    **Trade-offs:**
+
+
+    **Ownership Boundaries:**
+
+**Verified by:**
+
+- Decision Rule descriptions become documentation sections
+
+**Invariant:** Pre-implementation design stubs must reside in `architect/stubs/`, never in `src/`.
+
+**Rationale:** Stubs in `src/` require ESLint exceptions, create confusion between production and design code, and risk accidental imports of unimplemented functions.
+
+| Issue                    | Impact                                      |
+| ------------------------ | ------------------------------------------- |
+| ESLint exceptions needed | Rules relaxed for "not-yet-real" code       |
+| Confusion                | What's production vs. what's design?        |
+| Pollution                | Stubs mixed with implemented code           |
+| Import accidents         | Other code might import unimplemented stubs |
+| Maintenance burden       | Must track which files are stubs            |
+
+| Location                        | Content                                       | When Moved to src/     |
+| ------------------------------- | --------------------------------------------- | ---------------------- |
+| architect/stubs/{pattern}/\*.ts | API shapes, interfaces, throw-not-implemented | Implementation session |
+| src/\*_/_.ts                    | Production code only                          | Already there          |
+
+| Benefit               | How                                                           |
+| --------------------- | ------------------------------------------------------------- |
+| No ESLint exceptions  | Stubs aren't in src/, no relaxation needed                    |
+| Clear separation      | architect/stubs/ = design, src/ = production                  |
+| Documentation source  | Stubs with @extract-shapes generate API docs                  |
+| Safe iteration        | Can refine stub APIs without breaking anything                |
+| Implementation signal | Moving from architect/stubs/ to src/ = implementation started |
+
+| Document               | Decision Source                                 |
+| ---------------------- | ----------------------------------------------- |
+| docs/METHODOLOGY.md    | ADR for delivery process methodology            |
+| docs/TAXONOMY.md       | PDR-006 TypeScript Taxonomy (exists)            |
+| docs/VALIDATION.md     | ADR for validation approach                     |
+| docs/SESSION-GUIDES.md | ADR for session workflows                       |
+| \_claude-md/\*_/_.md   | Corresponding decisions with compact extraction |
+
+**The Problem:**
+
+    Design stubs (pre-implementation API shapes) placed in `src/` cause issues:
+
+
+    Example of the anti-pattern (from monorepo eslint.config.js):
+
+```javascript
+// TODO: Delivery process design artifacts: Relax unused-vars
+    {
+      files: [
+        "**/packages/platform-core/src/durability/durableAppend.ts",
+        "**/packages/platform-core/src/durability/intentCompletion.ts",
+        // ... more stubs in src/ ...
+      ],
+      rules: {
+        "@typescript-eslint/no-unused-vars": "off",
+      },
+    }
+```
+
+**The Solution:**
+
+    Design stubs live in `architect/stubs/`:
+
+
+    **Design Stub Pattern:**
+
+```typescript
+// architect/stubs/shape-extractor/shape-extractor.ts
+/**
+ * @architect
+ * @architect-pattern ShapeExtractorStub
+ * @architect-status roadmap
+ *
+ * ## Shape Extractor - Design Stub
+ *
+ * API design for extracting TypeScript types from source files.
+ */
+
+export interface ExtractedShape {
+  name: string;
+  kind: 'interface' | 'type' | 'enum' | 'function';
+  sourceText: string;
+}
+
+export function extractShapes(
+  sourceCode: string,
+  shapeNames: string[]
+): Map<string, ExtractedShape> {
+  throw new Error('ShapeExtractor not yet implemented - roadmap pattern');
+}
+```
+
+**Benefits:**
+
+    **Workflow:**
+
+    1. **Design session:** Create stub in `architect/stubs/{pattern-name}/`
+    2. **Iterate:** Refine API shapes, add JSDoc, test with docs generation
+    3. **Implementation session:** Move/copy to `src/`, implement real logic
+    4. **Stub becomes example:** Original stub stays as reference (optional)
+
+    **What This Enables:**
+
+    Once proven with Process Guard, the pattern applies to all documentation:
+
+**Verified by:**
+
+- N/A - architectural constraint verified by code structure
+- not runtime scenario
+
+## Rules
+
+### Proof of Concept - Self-documentation validates the pattern
+
+**Invariant:** The documentation generation pattern must be validated by generating documentation about itself from its own annotated sources.
+
+**Rationale:** A self-referential proof of concept exposes extraction gaps and source mapping issues that synthetic test data would miss.
+
+| Output                                                   | Purpose            | Detail Level |
+| -------------------------------------------------------- | ------------------ | ------------ |
+| docs/DOC-GENERATION-PROOF-OF-CONCEPT.md                  | Detailed reference | detailed     |
+| \_claude-md/generated/doc-generation-proof-of-concept.md | AI context         | summary      |
+
+| Section           | Source File                                     | Extraction Method          |
+| ----------------- | ----------------------------------------------- | -------------------------- |
+| Intro & Context   | THIS DECISION (Rule: Context above)             | Decision rule description  |
+| How It Works      | THIS DECISION (Rule: Decision above)            | Decision rule description  |
+| Validation Rules  | tests/features/validation/process-guard.feature | Rule blocks                |
+| Protection Levels | architect/specs/process-guard-linter.feature    | Scenario Outline Examples  |
+| Valid Transitions | architect/specs/process-guard-linter.feature    | Scenario Outline Examples  |
+| API Types         | src/lint/process-guard/types.ts                 | @extract-shapes tag        |
+| Decider API       | src/lint/process-guard/decider.ts               | @extract-shapes tag        |
+| CLI Options       | src/cli/lint-process.ts                         | JSDoc section              |
+| Error Messages    | src/lint/process-guard/decider.ts               | createViolation() patterns |
+| Pre-commit Setup  | THIS DECISION (DocString)                       | Fenced code block          |
+| Programmatic API  | THIS DECISION (DocString)                       | Fenced code block          |
+
+| Situation                     | Solution              | Example                                   |
+| ----------------------------- | --------------------- | ----------------------------------------- |
+| Fix bug in completed spec     | Add unlock reason tag | `@architect-unlock-reason:'Fix-typo'`     |
+| Modify outside session scope  | Use ignore flag       | `lint-process --staged --ignore-session`  |
+| CI treats warnings as errors  | Use strict flag       | `lint-process --all --strict`             |
+| Skip workflow (legacy import) | Multiple transitions  | Set roadmap then completed in same commit |
+
+**Process Guard docs are generated separately from `adr-006-process-guard.feature`.**
+
+    **Source Mapping for POC Self-Documentation:**
+
+    This source mapping demonstrates all extraction methods by extracting content
+    from this POC's own sources. The table serves as both documentation AND test data.
+
+
+    **Pre-commit Hook Setup:**
+
+    File: `.husky/pre-commit`
+
+```bash
+npx lint-process --staged
+```
+
+**Package.json Scripts:**
+
+```json
+{
+  "scripts": {
+    "lint:process": "lint-process --staged",
+    "lint:process:ci": "lint-process --all --strict"
+  }
+}
+```
+
+**Programmatic API Example:**
+
+```typescript
+import {
+  deriveProcessState,
+  detectStagedChanges,
+  validateChanges,
+  hasErrors,
+  summarizeResult,
+} from '@libar-dev/architect/lint';
+
+// 1. Derive state from annotations
+const state = (await deriveProcessState({ baseDir: '.' })).value;
+
+// 2. Detect changes
+const changes = detectStagedChanges('.').value;
+
+// 3. Validate
+const { result } = validateChanges({
+  state,
+  changes,
+  options: { strict: false, ignoreSession: false },
+});
+
+// 4. Handle results
+if (hasErrors(result)) {
+  console.log(summarizeResult(result));
+  process.exit(1);
+}
+```
+
+**Escape Hatches:**
+
+**Verified by:**
+
+- Full pipeline generates documentation from decision documents
+
+  This POC demonstrates the doc-from-decision pattern by generating docs
+  about ITSELF. The DocGenerationProofOfConcept pattern produces:
+
+### Expected Output - Compact claude module structure
+
+**Invariant:** Compact output must contain only essential tables and type names, with no JSDoc comments or implementation details.
+
+**Rationale:** AI context windows are finite; including non-essential content displaces actionable information and degrades session effectiveness.
+
+| Section            | Content                                                     |
+| ------------------ | ----------------------------------------------------------- |
+| Header + Intro     | Pattern name, problem/solution summary                      |
+| API Types          | Core interface definitions (DeciderInput, ValidationResult) |
+| 7 Validation Rules | Rule table with severity and description                    |
+| Protection Levels  | Status-to-protection mapping table                          |
+| CLI                | Essential command examples                                  |
+| Link               | Reference to full documentation                             |
+
+**File:** `_claude-md/validation/process-guard.md`
+
+    The compact module extracts only essential content for AI context.
+    Output size depends on source mapping entries - there is no artificial line limit.
+
+    **Expected Sections:**
+
+
+    **Key Characteristics:**
+
+    - Summary detail level (essential tables only)
+    - No JSDoc comments or implementation details
+    - Tables for structured data (rules, protection levels)
+    - Inline code blocks for CLI examples
+    - Cross-reference to detailed documentation
+
+**Verified by:**
+
+- Compact and detailed outputs from same sources
+
+---
+
+[← Back to All Decisions](../DECISIONS.md)