@taurgis/bonsai 1.0.0

package/README.md

@@ -0,0 +1,239 @@
+# Bonsai
+
+Bonsai is a standalone local research cache CLI for AI agents. It turns official documentation and web pages into source-cited Markdown artifacts, stores them in a durable local cache, and returns deterministic output that agents can reuse instead of repeatedly scraping the same pages.
+
+The package is published as `@taurgis/bonsai` and installs a `bonsai` binary.
+
+> Because the npm package is scoped, the correct one-shot command is `npx @taurgis/bonsai ...`. The unscoped `npx bonsai` name resolves to the separate `bonsai` package on npm unless an unscoped shim is published later.
+
+---
+
+## What it Does
+
+| Feature | How it works | Typical use case |
+| --- | --- | --- |
+| **Scrape and convert** | Fetches HTML, extracts main content, sanitizes unsafe markup, and converts it to deterministic Markdown | Capturing API docs, guides, standards, and changelogs |
+| **Cache-first lookup** | Normalizes URLs to collision-resistant SHA-256 keys before fetching | Avoiding duplicate network requests and repeated agent research |
+| **Token budgeting** | Returns `compressed` or `detailed` Markdown variants | Fitting research into limited context windows without losing full detail |
+| **Freshness tiers** | Supports `stable`, `standard`, `volatile`, custom TTLs, and stale revalidation | Handling standards differently from release notes or beta docs |
+| **Browser fallback** | Uses `--rendered` for pages that need client-side JavaScript | Capturing SPA documentation when static HTML is incomplete |
+| **Manual import** | Stores agent-supplied Markdown from stdin or files | Caching synthesized notes, private docs, or manually extracted pages |
+| **Search** | Ranks local cache entries by topic, tags, snippets, fuzzy terms, and phrases | Finding existing research before fetching again |
+
+---
+
+## Installation
+
+Requires Node.js 22 or newer.
+
+Run Bonsai without installing it globally:
+
+```bash
+npx @taurgis/bonsai https://nodejs.org/api/url.html
+```
+
+Install it as a normal CLI:
+
+```bash
+npm install -g @taurgis/bonsai
+bonsai https://nodejs.org/api/url.html
+```
+
+Use the local development binary inside this repository:
+
+```bash
+pnpm install
+pnpm build
+node bin/cli.mjs --help
+```
+
+## Quick Start
+
+### 1. Fetch and cache a URL
+
+```bash
+npx @taurgis/bonsai https://nodejs.org/api/url.html
+```
+
+The first run fetches the page, extracts the main article content, converts it to Markdown, estimates token size, and stores it locally.
+
+### 2. Reuse the cache
+
+```bash
+npx @taurgis/bonsai https://nodejs.org/api/url.html
+```
+
+The second run returns from cache when the entry is still fresh.
+
+### 3. Check cache state and metadata
+
+```bash
+npx @taurgis/bonsai status https://nodejs.org/api/url.html
+npx @taurgis/bonsai inspect https://nodejs.org/api/url.html
+```
+
+### 4. Import manual research notes
+
+```bash
+echo "# My Custom Node API Notes" | npx @taurgis/bonsai import https://nodejs.org/api/url.html --stdin
+
+echo "# Synthesized React Cache Guide" | npx @taurgis/bonsai import --stdin --topic "React Suspense" --source-url https://react.dev/a --source-url https://react.dev/b
+```
+
+### 5. Search before fetching
+
+```bash
+npx @taurgis/bonsai search "node api url"
+```
+
+---
+
+## Command Reference
+
+### `<url>`
+
+Fetch and format a webpage, or retrieve it from cache.
+
+```bash
+npx @taurgis/bonsai <url> [flags]
+```
+
+Common flags:
+
+* `--topic`, `-t`: Primary category/topic for metadata tagging.
+* `--tags`, `-g`: Searchable tags. Can be repeated.
+* `--format`, `-f`: Output density: `compressed` or `detailed`.
+* `--tier`: Freshness tier: `stable`, `standard`, or `volatile`.
+* `--ttl`, `-l`: Custom predicted lifespan, such as `2h`, `7d`, or `30d`.
+* `--max-age`: Read-time freshness threshold.
+* `--force`: Force a fresh fetch.
+* `--dry-run`: Scrape and validate without writing to cache.
+* `--allow-stale`: Serve stale cache if the remote site is offline.
+* `--rendered`: Use browser-rendered extraction for SPA pages.
+* `--json`: Return structured machine-readable output.
+
+### `import [url]`
+
+Save custom Markdown notes directly to the cache.
+
+```bash
+npx @taurgis/bonsai import [url] --stdin [flags]
+```
+
+### `status <url>`
+
+Inspect cache state and planned action without fetching.
+
+```bash
+npx @taurgis/bonsai status <url> [flags]
+```
+
+### `inspect <url>`
+
+Display stored YAML frontmatter metadata for a URL.
+
+```bash
+npx @taurgis/bonsai inspect <url>
+```
+
+### `search <query>`
+
+Search cached research by metadata and content snippets.
+
+```bash
+npx @taurgis/bonsai search "<keywords>" [flags]
+```
+
+### `config`
+
+Choose global or project-local cache storage.
+
+```bash
+npx @taurgis/bonsai config set storage project --local
+```
+
+Project config is stored in `.bonsai.json`; project cache files are stored under `.bonsai/research/`.
+
+---
+
+## Freshness and Cache Rules
+
+Global cache files live in Bonsai's oclif data directory, typically:
+
+* **macOS**: `~/Library/Application Support/bonsai/research/`
+* **Linux**: `~/.local/share/bonsai/research/`
+* **Windows**: `%LOCALAPPDATA%\bonsai\research\`
+
+Project-local storage uses:
+
+* `.bonsai.json` for the project config
+* `.bonsai/research/` for Markdown cache artifacts
+
+If no `--ttl` is specified, freshness is computed from the tier:
+
+| Tier | Fresh duration | Grace window | Use case |
+| --- | --- | --- | --- |
+| `volatile` | 7 days | 5 days | Latest releases, volatile changelogs, beta docs |
+| `standard` | 30 days | 14 days | General API docs and developer guides |
+| `stable` | 180 days | 60 days | RFCs, standards, long-lived references |
+
+When stale entries still have `ETag` or `Last-Modified` metadata, Bonsai attempts cheap revalidation before a full refetch. If revalidation fails and `--allow-stale` is omitted, the CLI can serve the stale content while exiting with code `5`.
+
+---
+
+## JSON Envelope
+
+When run with `--json`, commands return a stable envelope:
+
+```json
+{
+  "schemaVersion": 1,
+  "command": "bonsai",
+  "ok": true,
+  "exitCode": 0,
+  "stdout": "",
+  "stderr": "",
+  "data": {
+    "schemaVersion": 1,
+    "command": "bonsai",
+    "cache": {
+      "key": "0f115db062b7c0dd030b16878c99dea5c354b49dc37b38eb8846179c7783e9d7",
+      "status": "hit",
+      "freshness": "fresh",
+      "path": "/Users/user/Library/Application Support/bonsai/research/0f115db062b7c0dd030b16878c99dea5c354b49dc37b38eb8846179c7783e9d7.md"
+    },
+    "source": {
+      "url": "https://example.com",
+      "normalizedUrl": "https://example.com/",
+      "captureMethod": "static_fetch",
+      "extractionStatus": "extracted",
+      "extractionConfidence": "low",
+      "qualityNotes": ["readability extracted main article"],
+      "fetchedAt": "2026-06-24T07:33:20.519Z",
+      "validatedAt": "2026-06-24T07:33:20.519Z",
+      "staleAfter": "2026-07-24T07:33:20.519Z"
+    },
+    "format": "compressed",
+    "tokenEstimate": 29,
+    "content": "Cleaned main content markdown text..."
+  }
+}
+```
+
+---
+
+## Security and Limits
+
+* Fetched HTML is treated as untrusted input.
+* Only `http:` and `https:` URLs are accepted.
+* Private and local IP ranges are blocked to reduce SSRF risk.
+* Static extraction does not execute JavaScript; use `--rendered` when a page needs browser execution.
+* Authenticated/private scraping is not supported in the default workflow. Import trusted Markdown manually instead.
+
+---
+
+## Reference Documentation
+
+* [Command Reference](docs/commands.md)
+* [Caching Protocol Specification](docs/cache-protocol.md)
+* [Troubleshooting and Limitations](docs/troubleshooting.md)

package/agents/agents/official-docs-researcher.agent.md

@@ -0,0 +1,98 @@
+---
+name: 'Official Docs Researcher'
+description: 'Researches official documentation through Bonsai and returns source-cited findings'
+model: 'Auto'
+tools: ['vscode/askQuestions', 'execute', 'read', 'search', 'web', 'vscode/memory']
+argument-hint: 'What topic should I research in official docs?'
+metadata:
+  version: '3.0.0'
+---
+
+# Official Docs Researcher Agent
+
+You are a Forward documentation research specialist. Your job is to find current official documentation, cache it through Bonsai, and return concise source-cited findings that another agent can apply safely.
+
+## Scope
+
+Use this agent for technical changes, platform behavior, APIs, SDKs, standards, release notes, changelogs, beta docs, security advisories, and other documentation-sensitive work.
+
+Do not research from memory alone. Training-data knowledge does not satisfy this workflow.
+
+## Default Workflow
+
+1. Identify the product, version, edition, and source authority needed for the request.
+2. Search the Bonsai cache first:
+
+   ```bash
+   bonsai search "<topic or keywords>"
+   ```
+
+3. If the cache misses or does not cover the question, locate official source URLs. Prefer vendor docs, standards bodies, API references, and official release notes.
+4. Capture each source through Bonsai:
+
+   ```bash
+   bonsai <official-url> --format detailed
+   ```
+
+5. Use `--rendered` for SPAs or pages where static extraction is incomplete:
+
+   ```bash
+   bonsai <official-url> --rendered --format detailed
+   ```
+
+6. Summarize only what the official sources support. Include source URLs, validation time when available, version notes, and any important limitations.
+
+Inside the Bonsai repository, use the development binary instead of an installed global command:
+
+```bash
+node bin/cli.mjs search "<topic or keywords>" --json
+node bin/cli.mjs <official-url> --format detailed --json
+```
+
+For one-shot published usage outside this repo, use the scoped package name:
+
+```bash
+npx @taurgis/bonsai <official-url> --format detailed
+```
+
+Do not document or run bare `npx bonsai` unless an unscoped npm shim is actually published.
+
+## Freshness Policy
+
+Choose the freshness tier based on the source:
+
+- `stable`: standards, specs, MDN reference pages, pinned major-version docs.
+- `standard`: vendor API docs, SDK docs, normal technical references.
+- `volatile`: release notes, changelogs, latest pages, security advisories, beta docs.
+
+Stale volatile sources must be revalidated before they are trusted. Never overwrite a good cache entry with an error response.
+
+## Fallbacks
+
+If direct web access was unavoidable because Bonsai could not fetch the content, import the synthesized notes back into Bonsai before returning:
+
+```bash
+bonsai import <url> --file <path>
+```
+
+For multi-source synthesis:
+
+```bash
+bonsai import \
+  --topic "<descriptive topic>" \
+  --source-url <url1> \
+  --source-url <url2> \
+  --file <path>
+```
+
+## Response Contract
+
+Return:
+
+- Answer summary.
+- Official source URLs used.
+- Version, edition, or date constraints.
+- Cache status or validation timestamp when Bonsai output provides it.
+- Gaps or uncertainty where official docs do not answer the question.
+
+Keep stdout clean when using `--json`; warnings and diagnostic notes belong in stderr or in the final prose summary.

package/agents/instructions/web-research.instructions.md

@@ -0,0 +1,29 @@
+---
+description: 'Require the Official Docs Researcher subagent before technical repo changes.'
+applyTo: '**'
+skills:
+  - web-research
+metadata:
+  version: '3.0.0'
+---
+
+# Web Research Requirement
+
+## Mandatory Pre-Step
+
+- Run the **Official Docs Researcher** subagent before creating, updating, refactoring, scaffolding, or deleting technical content.
+- Use its findings to apply current official documentation and references in the change.
+- The researcher keeps a freshness-tiered Bonsai cache (its data directory or project-local `.bonsai/research/`): it reuses fresh notes, cheaply revalidates stale ones, and re-fetches only on a miss. Re-running on a recent topic is cheap — invoke it rather than skipping research to "save" a fetch.
+
+## When Not to Use
+
+- No technical content is being modified (purely editorial changes).
+- You already fetched and applied official docs **in the current task**. Training-data knowledge does not satisfy this — only docs fetched in the same task do.
+- The request is too simple to warrant research (typo, lint fix not involving platform behavior).
+
+## Examples
+
+- ✅ Before editing instructions, prompts, agents, skills, workflows, or docs.
+- ✅ Before auditing or refactoring source — reading source code does not replace checking current platform guidance.
+- ✅ Include official source URLs when the change references platform behavior or standards.
+- ❌ Don't modify technical content without running the subagent first, or treat cached/training knowledge as equivalent — official guidance evolves and must be verified per task.

package/agents/skills/web-research/SKILL.md

@@ -0,0 +1,126 @@
+---
+name: web-research
+description: 'Bonsai-backed official documentation and web research workflow. Use before technical changes that depend on platform behavior, when fetching documentation or web pages, when searching/listing/pruning the local research cache, or when importing manually gathered notes.'
+license: Forward Proprietary
+compatibility: VS Code 1.x+, GitHub Copilot
+metadata:
+  version: '2.1.0'
+---
+
+# Web Research Skill
+
+Use Bonsai as the cache-first research path for official documentation and web content. Prefer current official sources, keep research reusable, and avoid direct one-off web fetches when Bonsai can capture the same page.
+
+## Required Pre-Step
+
+Before creating, updating, refactoring, scaffolding, or deleting technical content, verify relevant current official documentation in the same task.
+
+Search the cache first:
+
+```bash
+bonsai search "<topic or keywords>"
+```
+
+If the cache misses or does not cover the question, fetch the source through Bonsai:
+
+```bash
+bonsai <official-url> --format detailed
+```
+
+Inside the Bonsai repository, use the development binary:
+
+```bash
+node bin/cli.mjs search "<topic or keywords>" --json
+node bin/cli.mjs <official-url> --format detailed --json
+```
+
+For one-shot published usage outside this repo, use the scoped package name:
+
+```bash
+npx @taurgis/bonsai <official-url> --format detailed
+```
+
+Do not document or run bare `npx bonsai` unless an unscoped npm shim is actually published.
+
+## Source Rules
+
+- Prefer official vendor docs, standards, API references, SDK docs, release notes, changelogs, and security advisories.
+- Include official source URLs when the change relies on platform behavior or standards.
+- Use `--tier stable`, `--tier standard`, or `--tier volatile` when the source class is clear.
+- Treat volatile sources as needing fresh validation before trusting them.
+- Do not use the retired manual `artifacts/online-research/` protocol for new research.
+
+## Fetch Rules
+
+Use `--format compressed` for context-budgeted reading and `--format detailed` for exact technical details, links, tables, and code examples.
+
+Use `--rendered` when static extraction is incomplete or the page is an SPA:
+
+```bash
+bonsai <official-url> --rendered --format detailed
+```
+
+Never reach for direct `WebFetch` or `WebSearch` to retrieve a specific page when Bonsai can fetch it. Bonsai returns reusable Markdown and keeps it cached for future agents.
+
+Pure discovery searches are allowed when you do not yet know which URL to fetch. Once a URL is selected, capture it with Bonsai.
+
+## Manual Fallbacks
+
+If direct web access was unavoidable because of authentication, browser interaction, or a tool constraint, import the result into Bonsai before returning.
+
+Single-source import:
+
+```bash
+bonsai import <url> --file path/to/notes.md
+```
+
+Stdin import:
+
+```bash
+echo "# My Synthesis Note" | bonsai import <url> --stdin
+```
+
+Multi-source synthesis:
+
+```bash
+bonsai import \
+  --topic "<descriptive topic>" \
+  --source-url <url1> \
+  --source-url <url2> \
+  --file path/to/synthesized-notes.md
+```
+
+## Cache Operations
+
+Check status without fetching:
+
+```bash
+bonsai status <url>
+```
+
+Inspect stored metadata:
+
+```bash
+bonsai inspect <url>
+```
+
+List or search cached entries:
+
+```bash
+bonsai list --tags node
+bonsai search "react suspense"
+```
+
+Preview pruning before deleting:
+
+```bash
+bonsai prune --older-than 90d --dry-run
+bonsai prune --older-than 90d --yes
+```
+
+## When This Does Not Apply
+
+- No technical content or web content is involved.
+- You already fetched and applied current official docs in the current task.
+- The request is a trivial typo or formatting fix that does not involve platform behavior.
+- Bonsai is not installed and `npx @taurgis/bonsai ...` is not available; use the best available official-source workflow and import notes later.

package/bin/cli.mjs

@@ -0,0 +1,22 @@
+#!/usr/bin/env node
+
+import { execute } from '@oclif/core';
+import { dirname } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const __dirname = dirname(fileURLToPath(import.meta.url));
+
+const rawArgv = process.argv.slice(2);
+const [firstArg, ...restArgv] = rawArgv;
+const args =
+  firstArg === 'help'
+    ? [...restArgv, '--help']
+    : firstArg?.startsWith('http://') || firstArg?.startsWith('https://')
+      ? ['fetch', ...rawArgv]
+      : rawArgv;
+
+await execute({
+  args,
+  development: false,
+  dir: __dirname + '/../',
+});

package/dist/base-command.d.ts

@@ -0,0 +1,14 @@
+import { Command, Interfaces } from '@oclif/core';
+export declare abstract class BaseCommand<T extends typeof Command> extends Command {
+    static jsonEnvelope: boolean;
+    static baseFlags: {
+        json: Interfaces.BooleanFlag<boolean>;
+    };
+    flags: Interfaces.InferredFlags<T['flags'] & typeof BaseCommand.baseFlags>;
+    args: Interfaces.InferredArgs<T['args']>;
+    protected requestedJson(): boolean;
+    protected envelopeCommandId(): string;
+    protected _run<R>(): Promise<R>;
+    abstract execute(): Promise<unknown>;
+    run(): Promise<unknown>;
+}

package/dist/base-command.js

@@ -0,0 +1,56 @@
+import { Command, Flags } from '@oclif/core';
+export class BaseCommand extends Command {
+    static jsonEnvelope = false;
+    static baseFlags = {
+        json: Flags.boolean({
+            description: 'Return the output as a machine-readable JSON object.',
+        }),
+    };
+    flags;
+    args;
+    requestedJson() {
+        return Boolean(this.flags?.json);
+    }
+    envelopeCommandId() {
+        return this.ctor.id || this.config.bin;
+    }
+    async _run() {
+        // Check argv directly for --json because if parsing fails in init(),
+        // we still want to output the failure in the JSON envelope format.
+        const isJson = this.argv.includes('--json');
+        if (isJson) {
+            try {
+                await this.init();
+                const result = await this.run();
+                const exitCode = process.exitCode || 0;
+                console.log(JSON.stringify({
+                    schemaVersion: 1,
+                    command: this.envelopeCommandId(),
+                    ok: exitCode === 0 || exitCode === 5,
+                    exitCode,
+                    stdout: '',
+                    stderr: '',
+                    data: result,
+                }, null, 2));
+            }
+            catch (error) {
+                const exitCode = error.oclif?.exit ?? 1;
+                console.log(JSON.stringify({
+                    schemaVersion: 1,
+                    command: this.envelopeCommandId(),
+                    ok: false,
+                    exitCode,
+                    stdout: '',
+                    stderr: error.message || String(error),
+                    data: null,
+                }, null, 2));
+                process.exitCode = exitCode;
+            }
+            return undefined;
+        }
+        return super._run();
+    }
+    async run() {
+        return this.execute();
+    }
+}

package/dist/commands/config/base.d.ts

@@ -0,0 +1,23 @@
+import { Command } from '@oclif/core';
+import { BaseCommand } from '../../base-command.js';
+import type { ConfigKey, ConfigScope } from '../../lib/config/index.js';
+/**
+ * Shared base for the four `config` subcommands: the mutually-exclusive
+ * scope-flag guard, the binary write-scope helper, and known-key validation.
+ */
+export declare abstract class ConfigCommand<T extends typeof Command> extends BaseCommand<T> {
+    protected assertScopeFlagsExclusive(global?: boolean, local?: boolean): void;
+    /** `--local` writes/reads the project file; otherwise the user-level file. */
+    protected writeScope(local?: boolean): 'user' | 'project';
+    /** Read scope for get/list: an explicit `--global`/`--local`, else the merged effective view. */
+    protected readScope(global?: boolean, local?: boolean): ConfigScope;
+    protected assertKnownKey(keyArg: string): asserts keyArg is ConfigKey;
+}
+/** Build the `--global`/`--local` scope-flag pair with per-command descriptions. */
+export declare function configScopeFlags(descriptions: {
+    global: string;
+    local: string;
+}): {
+    global: import("@oclif/core/interfaces").BooleanFlag<boolean>;
+    local: import("@oclif/core/interfaces").BooleanFlag<boolean>;
+};

package/dist/commands/config/base.js

@@ -0,0 +1,46 @@
+import { Flags } from '@oclif/core';
+import { BaseCommand } from '../../base-command.js';
+import { isKnownKey, suggestKey, validKeysHint } from '../../lib/config/index.js';
+/**
+ * Shared base for the four `config` subcommands: the mutually-exclusive
+ * scope-flag guard, the binary write-scope helper, and known-key validation.
+ */
+export class ConfigCommand extends BaseCommand {
+    assertScopeFlagsExclusive(global, local) {
+        if (global && local) {
+            this.error('--global and --local are mutually exclusive.', { exit: 2 });
+        }
+    }
+    /** `--local` writes/reads the project file; otherwise the user-level file. */
+    writeScope(local) {
+        return local ? 'project' : 'user';
+    }
+    /** Read scope for get/list: an explicit `--global`/`--local`, else the merged effective view. */
+    readScope(global, local) {
+        if (global)
+            return 'global';
+        if (local)
+            return 'local';
+        return 'effective';
+    }
+    assertKnownKey(keyArg) {
+        if (!isKnownKey(keyArg)) {
+            const suggestion = suggestKey(keyArg);
+            const hint = suggestion ? ` Did you mean: ${suggestion}?` : '';
+            this.error(`Unknown config key: "${keyArg}".${hint} Valid keys: ${validKeysHint()}.`, {
+                exit: 2,
+            });
+        }
+    }
+}
+/** Build the `--global`/`--local` scope-flag pair with per-command descriptions. */
+export function configScopeFlags(descriptions) {
+    return {
+        global: Flags.boolean({ char: 'g', description: descriptions.global }),
+        local: Flags.boolean({
+            description: descriptions.local,
+            aliases: ['project'],
+            charAliases: ['p'],
+        }),
+    };
+}

package/dist/commands/config/get.d.ts

@@ -0,0 +1,20 @@
+import { ConfigCommand } from './base.js';
+export default class ConfigGet extends ConfigCommand<typeof ConfigGet> {
+    static id: string;
+    static summary: string;
+    static description: string;
+    static examples: {
+        description: string;
+        command: string;
+    }[];
+    static args: {
+        key: import("@oclif/core/interfaces").Arg<string, Record<string, unknown>>;
+    };
+    static flags: {
+        global: import("@oclif/core/interfaces").BooleanFlag<boolean>;
+        local: import("@oclif/core/interfaces").BooleanFlag<boolean>;
+    };
+    static stdoutIsPrimaryData: boolean;
+    init(): Promise<void>;
+    execute(): Promise<unknown>;
+}