@tyvm/knowhow 0.0.109 → 0.0.111

package/autodoc/generate-guide.md

@@ -1,477 +1,437 @@
 # `knowhow generate` Guide
 
-`knowhow generate` is the Knowhow CLI command that turns your input files into generated documentation (and other artifacts) by running an AI prompt over the selected sources and writing the results to your configured outputs.
-
-It is driven entirely by your local **`.knowhow/knowhow.json`** config—specifically the **`sources`** array.
+`knowhow generate` is Knowhow’s “code-to-documents” generator. It reads your `sources` from `.knowhow/knowhow.json`, resolves prompts, runs AI summarization (or plugin handlers), and writes output files (or sets of files) to your configured `output` paths.
 
 ---
 
-## What `knowhow generate` does
-
-When you run:
-
-```bash
-knowhow generate
-```
-
-the CLI:
+## 1) What `knowhow generate` does
 
-1. Loads your Knowhow config from **`.knowhow/knowhow.json`**
-2. For each entry in **`config.sources`**:
-   - Expands `source.input` into a list of matching files
-   - Resolves `source.prompt` into an MDX prompt template or prompt string
-   - Uses the selected `model`/`agent` to summarize/generate content
-   - Writes output to either:
-     - a single output file, or
-     - multiple output files inside an output directory
-3. Skips work when inputs and prompt haven’t changed (hash-based caching)
+At a high level, for each entry in `config.sources`, it:
 
----
+1. **Finds input files** using `source.input` (glob / comma list / brace expansion).
+2. **Loads the prompt** via `source.prompt`.
+3. **Chooses output mode** based on `source.output`:
+   - If `source.output` ends with `/` → **multi-output mode** (one output per input file).
+   - Otherwise → **single-output mode** (one combined output file).
+4. **Skips work using caching** (`.knowhow/.hashes.json`):
+   - It computes an **MD5 hash of the prompt** and an **MD5 hash of each input file’s converted text**.
+   - If the prompt+file hashes match what was last generated, the output is skipped.
+5. **Writes generated documents** to disk.
 
-## `sources` configuration structure
+### Output mode behavior (from the code)
 
-`sources` is an array of objects in `.knowhow/knowhow.json`.
+#### A) Multi-output mode (directory output)
+If `source.output` ends with `/`:
 
-At runtime, each source object may be handled in one of two ways:
+- For each matched input file:
+  - Compute an output folder that preserves subpaths **under the base prefix before `**`**.
+    - Example: with `input: "src/**/*.ts"`, the base prefix is `"src/"`.
+    - A file like `src/cli/index.ts` becomes an output at:
+      - `<output>/<nestedFolder>/<outputName>.<outputExt>`
+  - Default `outputExt` is `"mdx"` unless overridden by `outputExt`.
 
-- **File generation mode** (default): when `kind` is `"file"` or missing/empty
-- **Plugin/“special kind” mode**: when `kind` is set and is not `"file"` (see `kind` below)
+#### B) Single-output mode (single file output)
+If `source.output` does **not** end with `/`:
 
-### Common fields
+- It writes **one** `source.output` file that contains combined results for **all** matched inputs.
 
-```jsonc
-{
-  "input": "src/**/*.ts",
-  "output": ".knowhow/docs/",
-  "prompt": "MyPrompt",
-  "kind": "file",
-  "agent": "Developer",
-  "model": "gpt-4o-mini",
-  "outputExt": "mdx",
-  "outputName": "README"
-}
-```
+---
 
-All fields described below are part of `GenerationSource` usage in the generator.
+## 2) `sources` configuration structure
 
----
+Your `sources` array lives in `.knowhow/knowhow.json` under `sources`.
 
-## Field reference
+Each source entry is a `GenerationSource` with these commonly used fields:
 
-### `input` (required)
-Defines which files to read. Supported formats are:
+### `input`
+Controls which files are processed. Supported formats (as implemented):
 
 - **Single file**
   ```json
-  { "input": "src/index.ts" }
+  "input": "src/index.ts"
   ```
 - **Glob**
   ```json
-  { "input": "src/**/*.ts" }
+  "input": "src/**/*.ts"
   ```
 - **Comma-separated list**
   ```json
-  { "input": "src/a.ts,src/b.ts,src/c.ts" }
+  "input": "src/a.ts,src/b.ts,src/c.ts"
   ```
-  Comma-separated values are auto-normalized into brace expansion internally.
+  Comma lists are normalized into brace expansion internally.
 - **Brace expansion**
   ```json
-  { "input": "{src/a.ts,src/b.ts}" }
+  "input": "{src/a.ts,src/b.ts}"
   ```
 
-**Normalization behavior (important):**
-- If `input` contains `{`, `*`, or `?`, it is used as-is.
-- If `input` contains commas and no glob/braces, commas are converted into `{a,b,c}`.
-- Otherwise, it’s treated as a glob/pattern directly.
+> Implementation note: Knowhow normalizes comma-separated lists into `{a,b,c}` form (so it can pass it to the glob library).
 
 ---
 
-### `output` (required)
-Defines where generated content is written.
-
-Two output modes exist:
-
-#### 1) Directory mode (multi-output)
-If `output` **ends with `/`**, each input file becomes its own output file:
-
-```json
-{
-  "input": "src/**/*.ts",
-  "output": ".knowhow/docs/",
-  "outputExt": "mdx"
-}
-```
-
-- Output file name defaults to the input file’s base name
-- You can override the output name (see `outputName`)
-- You can override the output extension (see `outputExt`)
+### `output`
+Controls where results are written:
 
-#### 2) Single file mode (single-output)
-If `output` **does not end with `/`**, all matched input files are combined into **one** output file.
+- **Directory output** (multi-output mode)
+  - Must end with `/`:
+  ```json
+  "output": ".knowhow/docs/"
+  ```
+- **Single file output** (single-output mode)
+  ```json
+  "output": ".knowhow/docs/README.mdx"
+  ```
 
-```json
-{
-  "input": "src/**/*.ts",
-  "output": ".knowhow/docs/ALL.md"
-}
-```
+Knowhow creates directories automatically in multi-output mode.
 
 ---
 
-### `prompt` (required or optional depending on your desired behavior)
-Controls the AI instruction template.
+### `prompt`
+Controls the prompt to use for AI generation.
 
-You can supply `prompt` in any of these ways:
+Knowhow resolves `prompt` in this order:
 
-1. **Prompt name** (looked up in `promptsDir`)
-   ```json
-   { "prompt": "BasicCodeDocumenter" }
-   ```
-   The generator looks for:
-   - `<promptsDir>/<promptName>.mdx`
-   - Default `promptsDir` is: **`.knowhow/prompts`**
+1. **Prompt name in `promptsDir`**  
+   If `prompt` is a name like `"BasicCodeDocumenter"`, it will try:
+   - `<promptsDir>/<prompt>.mdx`  
+   Example: `.knowhow/prompts/BasicCodeDocumenter.mdx`
 
 2. **Direct prompt file path**
-   ```json
-   { "prompt": "prompts/MyPrompt.mdx" }
-   ```
-   If that file exists, it is read as prompt content.
+   - If `prompt` points to an existing file, it reads that file.
 
 3. **Inline prompt string**
-   ```json
-   { "prompt": "Summarize the following file as a short doc for newcomers.\n\n{text}" }
-   ```
-
-#### `{text}` placeholder requirement
-If the prompt template does **not** include `{text}`, Knowhow automatically appends:
-
-```mdx
-{text}
-```
+   - If neither a known prompt file nor a direct path exists, Knowhow treats the `prompt` value itself as the prompt text.
 
-so the input is included when rendering.
+> Also: Knowhow ensures your prompt contains a `{text}` placeholder. If it’s missing, it automatically appends:
+> `\n\n{text}`
 
 ---
 
-### `kind` (optional)
-Controls whether generation is treated as “file generation” or a “special processing kind”.
+### `kind`
+Special processing kind.
+
+From the code:
 
-- If `kind` is `"file"` **or missing**, Knowhow performs the normal file summarization flow.
-- Otherwise, Knowhow checks whether `kind` matches a registered plugin:
-  - If it is a plugin, the plugin is executed and its returned data is written to `output` (and directory output is rejected for plugin-only output).
-  - Then Knowhow still proceeds with file generation for that same source.
+- If `kind` is `"file"` **or missing** → **standard file summarization** pipeline runs.
+- Otherwise → Knowhow attempts a **plugin-based handler**:
+  - It checks `Plugins.isPlugin(kind)`.
+  - If it is a plugin:
+    - It runs `Plugins.call(kind, input)` and writes the result to `source.output` (but **plugin output must be a single file**; directory outputs are rejected for plugins).
+  - After the plugin step (or if the kind is not a plugin), Knowhow still proceeds to the normal file summarization generation for that source.
 
-> In other words: non-`file` kinds can run a plugin and then also run the file-based generation pipeline.
+**Practical takeaway:**  
+Use `kind` for plugin integrations; otherwise default file generation happens.
 
 ---
 
-### `agent` (optional)
-Selects which agent to use.
+### `agent`
+Selects which agent to use when calling the AI summarizer.
 
-- Default agent (as noted in your config requirements): **`Developer`**
-- This value is passed through to summarization functions.
+- If omitted, a default is used (the docs mention default agent: **Developer**).
 
 ---
 
-### `model` (optional)
-Overrides the AI model for this source.
+### `model`
+Overrides the model for this source.
+
+- The value is passed to `summarizeFile(...)` / `summarizeFiles(...)`.
 
-- If omitted, the summarization functions will use Knowhow’s configured default model/provider logic (outside the code shown here).
-- If provided, the source-specific `model` is used when calling `summarizeFile` / `summarizeFiles`.
+> If you don’t set `model`, Knowhow will use whatever default model the summarization layer chooses (not shown in the provided code).
 
 ---
 
-### `outputExt` (optional; multi-output mode only)
-Overrides the extension used for generated files when `output` ends with `/`.
+### `outputExt`
+Overrides the extension used for multi-output mode outputs.
 
-Default: **`"mdx"`**
+- Default in multi-output handler: `"mdx"`
+- Only applied when `output` ends with `/` (directory mode).
 
-Example:
+---
 
-```json
-{
-  "input": "src/**/*.ts",
-  "output": ".knowhow/docs/",
-  "outputExt": "md"
-}
-```
+### `outputName`
+Overrides the filename portion used for multi-output mode.
+
+- Only applied when `output` ends with `/`.
+- For each input file, output filename becomes:
+  - `<outputFolder>/<outputName>.<outputExt>` (instead of using the input’s basename)
 
 ---
 
-### `outputName` (optional; multi-output mode only)
-Overrides the generated output file’s base name (instead of using the input file’s name).
+## 3) Prompt resolution (where your prompt comes from)
 
-Example:
+Knowhow loads prompts with `loadPrompt(promptName)` and supports:
 
+### A) Prompt file in `promptsDir`
+- Default `promptsDir`: `.knowhow/prompts`
+- Example prompt file:
+  - `.knowhow/prompts/MyPrompt.mdx`
+
+Config:
 ```json
-{
-  "input": "src/**/*.ts",
-  "output": ".knowhow/docs/",
-  "outputName": "REFERENCE",
-  "outputExt": "mdx"
-}
+"prompt": "MyPrompt"
 ```
 
-Each file would still map into the directory (including nested folders), but each output filename would be `REFERENCE.<outputExt>`.
-
----
+Knowhow reads:
+- `.knowhow/prompts/MyPrompt.mdx`
 
-## Prompt resolution (how Knowhow finds your prompt)
-
-Knowhow resolves `source.prompt` using the following order (from `loadPrompt()`):
+### B) Direct prompt file path
+Config:
+```json
+"prompt": "./prompts/MyPrompt.mdx"
+```
 
-1. **Look in `promptsDir`**
-   - Default `promptsDir`: **`.knowhow/prompts`**
-   - It checks for:  
-     `path.join(promptsDir, promptName + ".mdx")`
+Knowhow reads that file if it exists.
 
-2. **Try `prompt` as a direct file path**
-   - If `fs.existsSync(promptName)` succeeds, it reads that file.
+### C) Inline prompt string
+Config:
+```json
+"prompt": "Summarize the following code and output markdown with sections..."
+```
 
-3. **Treat `prompt` itself as the prompt content**
-   - If neither file exists, the value is used as an inline prompt string.
+### `{text}` placeholder requirement
+Knowhow requires/ensures the prompt contains `{text}`.
 
-Finally, it ensures `{text}` exists (appends it if missing).
+- If your prompt doesn’t include it, Knowhow appends it automatically.
+- `{text}` is where Knowhow inserts the content it’s summarizing (the code/text extracted from each input file).
 
 ---
 
-## Input → output mapping (output modes)
+## 4) Input patterns: examples
 
-### A) Directory output (`output` ends with `/`)
-For each input file:
-
-- Knowhow computes a nested output folder based on the prefix before `**`.
+### Single file
+```json
+{
+  "input": "src/index.ts",
+  "output": ".knowhow/out/index.mdx",
+  "prompt": "BasicCodeDocumenter"
+}
+```
 
-Given:
+### Glob (multiple files → combined single output)
+```json
+{
+  "input": "src/**/*.ts",
+  "output": ".knowhow/docs/SDK.md",
+  "prompt": "BasicCodeDocumenter"
+}
+```
 
+### Glob (multiple files → one output per file)
 ```json
-"input": "src/**/*.ts",
-"output": ".knowhow/docs/"
+{
+  "input": "src/**/*.ts",
+  "output": ".knowhow/docs/",
+  "prompt": "BasicCodeDocumenter",
+  "outputExt": "mdx"
+}
 ```
 
-- `inputPath` becomes `"src/"` (everything before the `**`)
-- Each input file’s directory is transformed into an equivalent subfolder under the output directory
+### Comma-separated inputs (combined single output)
+```json
+{
+  "input": "src/a.ts,src/b.ts,src/c.ts",
+  "output": ".knowhow/docs/selected.md",
+  "prompt": "BasicCodeDocumenter"
+}
+```
 
-Then each file is written as:
+### Brace expansion
+```json
+{
+  "input": "{src/a.ts,src/b.ts}",
+  "output": ".knowhow/docs/selected.md",
+  "prompt": "BasicCodeDocumenter"
+}
+```
 
-- `output/<nestedFolder>/<outputFileName>.<outputExt>`
+---
 
-Where:
-- `outputFileName` = `outputName` if provided, otherwise the input file’s base name
-- `outputExt` defaults to `mdx` unless overridden
+## 5) Output modes: directory vs single file
 
----
+### Directory output (`output` ends with `/`)
+Produces **N output files** (one per input file matched).
 
-### B) Single file output (`output` does NOT end with `/`)
-All matched files are combined and written into the one `output` path.
+Each output file is placed under your output directory while preserving nested folders relative to the prefix before `**`.
 
 ---
 
-## Hashing / caching behavior (skips unchanged files)
+### Single-file output (`output` is a file path)
+Produces **one output file** that combines all matched inputs into a single document.
 
-Knowhow uses a persistent hash file:
+---
 
-- **`.knowhow/.hashes.json`**
+## 6) Hashing / caching: when outputs are skipped
 
-For each generation operation, it computes:
+Knowhow uses `.knowhow/.hashes.json` and stores hashes keyed by the input file path (and the prompt hash).
 
-- `promptHash` = `md5(promptString)`
-- `fileHash` = `md5(convertToText(file))` (input content)
-- For directory mode, it checks both:
-  - the input file, and
-  - the output file
+In both output modes:
 
-If `checkNoFilesChanged()` decides nothing has changed, Knowhow prints a “Skipping…” message and does not regenerate.
+- It computes:
+  - `promptHash = md5(prompt)`
+  - `fileHash = md5(convertToText(file))`
+- It calls `checkNoFilesChanged(...)` for the “to check” set:
+  - **Multi-output mode** checks: `[file, outputFile]`
+  - **Single-output mode** checks: `[outputFile, ...files]`
+- If nothing changed, Knowhow logs a skip and moves on.
 
-This means:
-- Changing the prompt will invalidate cache for those files
-- Changing input file content will invalidate cache for those files
+**Important:** If the input or output file is missing on disk, regeneration will happen.
 
 ---
 
-## Writing good prompt templates (`.mdx`)
+## 7) Writing good `.mdx` prompt files
 
-Prompt files are typically stored under:
+Knowhow expects prompt files to be **MDX** templates (e.g. `.knowhow/prompts/MyPrompt.mdx`).
 
-- **`.knowhow/prompts/YourPromptName.mdx`**
+### Minimum requirement
+Your prompt must include a `{text}` placeholder (Knowhow will add it if you forget).
 
-### Minimum guidance
+### Recommended structure
+A solid prompt usually asks for:
 
-- Include a **`{text}`** placeholder somewhere in the prompt.
-- If you forget it, Knowhow will append it automatically at the end—but you should still place it where it makes sense.
+- Clear sections (e.g. Overview / API / Usage / Notes)
+- Output format (markdown headings, lists)
+- Constraints (brevity, accuracy)
+- Interpretation rules (assume code is partial, etc.)
 
-#### Example: basic prompt file
+### Example prompt file: `.knowhow/prompts/BasicCodeDocumenter.mdx`
 
-**`.knowhow/prompts/BasicCodeDocumenter.mdx`**
 ```mdx
-# Document this file
+# File documentation
+
+You are a technical writer.
 
-Write a clear developer-focused documentation page.
+Given this code, produce documentation with:
+1. **Purpose** (1-3 sentences)
+2. **Key exported items** (functions/classes/types)
+3. **How it works** (step-by-step at a high level)
+4. **Important caveats** (edge cases, assumptions)
+5. **Examples** (if obvious)
 
-Requirements:
-- 1–2 paragraph summary
-- list of key exported functions/classes
-- notable pitfalls or usage notes
+Return valid Markdown.
 
-Input:
+Source content:
+```text
 {text}
 ```
-
-Then reference it in `knowhow.json`:
-
-```json
-{
-  "input": "src/**/*.ts",
-  "output": ".knowhow/docs/",
-  "prompt": "BasicCodeDocumenter"
-}
 ```
 
+### About “file: src/cli.ts”
+Many Knowhow prompt templates also include a filename header (often formatted like `file: src/cli.ts`) above the `{text}` block. In your own prompts, follow the same pattern you see in the built-in prompts under `.knowhow/prompts/`.
+
 ---
 
-## Practical examples
+## 8) Pipeline examples (transcribe → summarize → embed)
 
-### 1) Generate docs for every source file (directory mode)
+`knowhow generate` performs “summarize / document from inputs” steps. Embeddings are handled by a separate command (`knowhow embed`), but you can model a pipeline with multiple config sections.
 
-```json
-{
-  "sources": [
-    {
-      "input": "src/**/*.ts",
-      "output": ".knowhow/docs/",
-      "prompt": "BasicCodeDocumenter",
-      "outputExt": "mdx",
-      "agent": "Developer"
-    }
-  ]
-}
-```
+### Example: Transcribe audio → generate summaries → embed them
 
-Result:
-- `.knowhow/docs/<same-folder-structure-as-src>/<filename>.mdx`
+1. **Transcribe step**  
+   Use `sources` with an appropriate `kind` if you have a transcription plugin installed (or you can generate from already-transcribed text).
 
----
+2. **Summarize step**  
+   Use `knowhow generate` to turn transcripts into markdown documents.
+
+3. **Embed step**  
+   Use `embedSources` for chunking and embedding.
 
-### 2) Generate one “project overview” doc from many inputs (single file mode)
+Example config outline:
 
 ```json
 {
   "sources": [
     {
-      "input": "src/**/*.ts",
-      "output": ".knowhow/docs/OVERVIEW.mdx",
-      "prompt": "BasicProjectDocumenter"
+      "input": "audio/**/*.txt",
+      "output": ".knowhow/transcripts/",
+      "prompt": "TranscriptSummarizer",
+      "outputExt": "mdx"
+    },
+    {
+      "input": ".knowhow/transcripts/**/*.mdx",
+      "output": ".knowhow/docs/AllSummaries.mdx",
+      "prompt": "ProjectSummary"
+    }
+  ],
+  "embedSources": [
+    {
+      "input": ".knowhow/docs/**/*.mdx",
+      "output": ".knowhow/embeddings/docs.json",
+      "prompt": "BasicEmbeddingExplainer",
+      "chunkSize": 2000
     }
   ]
 }
 ```
 
-Result:
-- One file: `.knowhow/docs/OVERVIEW.mdx`
+Run sequence:
+- `knowhow generate`
+- `knowhow embed`
 
 ---
 
-### 3) Use comma-separated input (combined behavior)
+## 9) Running for specific sources
 
-Comma-separated inputs are treated as brace-expanded patterns, but functionally they still match multiple files.
+From the provided code, `generate()` loops through **all** entries in `config.sources`:
 
-```json
-{
-  "sources": [
-    {
-      "input": "src/a.ts,src/b.ts,src/c.ts",
-      "output": ".knowhow/docs/SUBSET.mdx",
-      "prompt": "BasicCodeDocumenter"
-    }
-  ]
+```ts
+for (const source of config.sources) {
+  ...
 }
 ```
 
-Because `output` does not end with `/`, all matched inputs are combined into **one** output.
+So, there isn’t an obvious built-in “select by name” mechanism shown here.
+
+### Practical approaches
+- **Temporarily edit** `.knowhow/knowhow.json` to include only the sources you want to regenerate.
+- Split sources into multiple config files/directories and swap configs (if your workflow supports it).
+- Rely on caching: if your inputs and prompts haven’t changed, Knowhow will skip work automatically.
 
 ---
 
-### 4) Use brace expansion directly
+## 10) Practical `sources` config examples
 
+### Example A: Generate one file per component (directory output)
 ```json
 {
   "sources": [
     {
-      "input": "{src/controllers/*.ts,src/services/*.ts}",
-      "output": ".knowhow/docs/",
-      "prompt": "BasicCodeDocumenter"
+      "kind": "file",
+      "input": "src/components/**/*.ts",
+      "output": ".knowhow/docs/components/",
+      "prompt": "BasicCodeDocumenter",
+      "outputExt": "mdx",
+      "agent": "Developer",
+      "model": "gpt-5.4-nano"
     }
   ]
 }
 ```
 
-Directory mode: each input file becomes its own output file.
-
----
-
-### 5) Inline prompt (no prompt file)
-
+### Example B: Generate a single README from multiple files (single-file output)
 ```json
 {
   "sources": [
     {
-      "input": "src/index.ts",
-      "output": ".knowhow/docs/index.mdx",
-      "prompt": "Explain what this module does in plain English.\n\n{text}"
+      "input": "src/**/*.ts",
+      "output": ".knowhow/docs/README.mdx",
+      "prompt": "BasicProjectDocumenter"
     }
   ]
 }
 ```
 
----
-
-## Pipeline example: generate → embed
-
-A common workflow is:
-
-1. **Generate documentation files**
-2. **Embed the generated docs** for search/RAG
-
-Your config already supports embedding in a separate `embedSources` array. A typical setup:
-
-```jsonc
+### Example C: Combine a custom set of files using comma-separated input
+```json
 {
   "sources": [
     {
-      "input": "src/**/*.ts",
-      "output": ".knowhow/docs/",
+      "input": "src/cli.ts,src/config.ts,src/index.ts",
+      "output": ".knowhow/docs/CLI-Notes.mdx",
       "prompt": "BasicCodeDocumenter"
     }
-  ],
-  "embedSources": [
-    {
-      "input": ".knowhow/docs/**/*.mdx",
-      "output": ".knowhow/embeddings/docs.json",
-      "prompt": "BasicEmbeddingExplainer",
-      "chunkSize": 2000
-    }
   ]
 }
 ```
 
-Then run:
-
-```bash
-knowhow generate
-knowhow embed
-```
-
----
-
-## Running for specific sources
-
-There is **no built-in CLI flag** in the shown `knowhow generate` command to select only one `sources[]` entry.
-
-To target a specific source you typically have two options:
-
-- **Temporarily edit** `.knowhow/knowhow.json` to comment out other `sources`
-- **Create a separate config** and adjust your workflow (the current CLI wiring shown does not expose a `--config` argument for `generate`)
-
 ---
 
-If you want, paste your current `.knowhow/knowhow.json` `sources` array and tell me whether you want **single-file** or **directory-per-input** outputs—I can help you rewrite it for the behavior you want.
+If you want, paste your current `.knowhow/knowhow.json` `sources` section and the prompt(s) you’re using, and I’ll help you validate the config (including output folder structure expectations and caching behavior).