@mikedown/core 0.2.0 → 0.4.0

package/CHANGELOG.md

@@ -1,5 +1,24 @@
 # Changelog
 
+## 0.4.0
+
+- New `ai/docs/` module (public through `@mikedown/core/ai`): the documentation base model, home document precedence, link graph, keyword fallback, secrets scan, citations and the navigation loop. Hosts provide `host.docs` (`MikeDownDocsHost`) to back it; without it the chat shows "This document" only.
+- `@mikedown/core/ai` adds the `chat` feature (status `available`), the `docs` task, the `docsBase` value of `AiSendsScope` and an optional `enabled` flag per base.
+- `@mikedown/core` exports the chat panel (`ai-chat-panel.ts`), the documentation bases page (`ai-docs-page.ts`) and the run wiring (`ai-docs-run.ts`) through the AI UI, loaded only when the host has `host.ai`.
+- Run requests can be pinned to one engine and model (`AiRunRequest.pin`). A pinned run never re-routes or falls back within the turn.
+- Inline suggestions: `ai-ghost.ts`, `ai-ghost-plugin.ts` and `ai-ghost-session.ts` in the AI UI, with `ghostContext`, `ghostTrim` and the `continue` prompt in `@mikedown/core/ai`. New `ghostText` feature id and `cursor` consent scope.
+- AI Activity panel: an opt-in, in-memory record of each AI request (`ai-activity.ts`, `ai-activity-panel.ts`), with an optional `saveTextFile` on the host.
+- Insert from chat goes through an allow-list of node and mark types (`ai-insert-guard.ts`).
+- A bare URL at the end of a document's first block is no longer turned into a link on load.
+
+## 0.3.0
+
+- New prompt modules under `ai/prompts/` (`rewrite`, `explain`, `structureCheck`, `summary`, `title`, `properties`) for the AI writing features.
+- `ai/json.ts` parses model JSON leniently and makes one repair retry before reporting `badOutput`. `ai/documentBudget.ts` takes sections from the top of a long document and reports the coverage. `ai-prefs.ts` stores the per-session AI preferences.
+- `AiCompletionRequest.requestOverrides` carries per-request body fields (for example `reasoning_effort`). Hosts merge them over the model's configured overrides, so the request wins. Tier 1 hosts may ignore them.
+- The `summarize` and `structure` tasks now require chat only; `json` and `context` are preferences for them. There is a new `title` feature id, and a `badOutput` error code.
+- The rewrite popover gained modes for insert and choose, used by the summary and title commands.
+
 ## 0.2.0
 
 - `bindMikeDownValue` (`@mikedown/core/binding`): the framework-neutral value binding the React, Vue, Angular and web component wrappers are built on (pristine writes, debounced change, flush on blur, read-only, disabled and touched state).

package/README.md

@@ -86,11 +86,127 @@ field.destroy();              // flushes, then removes observers and listeners
 | `onHeadingRenamed`, `onHeadingRenameAmbiguous` | no rename callbacks |
 | `exportHtml`, `printDocument`, `viewInBrowser` | export buttons hidden |
 | `sidebarPrefs` | sidebar keeps in-memory defaults |
+| `ai` | no AI features, no AI tab (see [Bring your own AI](#bring-your-own-ai)) |
 
 Request and response methods return Promises. For example `uploadImage(file)` receives `{ mime, bytes, name }` and resolves `{ src, alt }`.
 
 The returned `MikeDownEditorHandle` takes pushes from the host: `setMarkdown`, `markSaved`, `updateSettings`, `setReadOnly`, `setTags`, `setBacklinks`, `setDocMeta`, `scrollToAnchor`, `getHtml`, `command(name)` and more. `handle.on('ready' | ...)` subscribes to events. All types are exported from the package root.
 
+## Bring your own AI
+
+`host.ai` turns on the editor's AI features (selection rewrite today, more later) and an AI tab in the Settings modal. Leave it out and nothing AI shows. **`@mikedown/core` ships no provider SDK and makes no AI network calls of its own.** Your `host.ai` does the transport; `@mikedown/core/ai` holds only pure helpers (settings schema, router, request builders, stream parsers) you may use or ignore.
+
+There are two tiers. Pick by how much of the engine setup you want the editor to own.
+
+**Tier 1: one method.** Implement `complete` and proxy it to your backend. Your server picks the model, holds the keys and decides what may leave your network. The editor shows the on/off switch and the feature toggles in Settings, but no engine list.
+
+```ts
+import { createMikeDownEditor, type MikeDownAiHost } from '@mikedown/core';
+
+const ai: MikeDownAiHost = {
+  async complete(req, onToken, signal) {
+    const res = await fetch('/api/ai/complete', {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      // Image parts carry raw bytes; encode them first (bytesToBase64 from '@mikedown/core/ai') if you forward them.
+      body: JSON.stringify({ feature: req.feature, task: req.task, messages: req.messages, maxTokens: req.maxTokens }),
+      signal,
+    });
+    if (!res.ok) throw { code: res.status === 429 ? 'rateLimited' : 'unknown', message: await res.text() };
+    // This backend streams plain text. Rejecting with an AbortError (or aborting `signal`) cancels cleanly.
+    const reader = res.body!.getReader();
+    const decoder = new TextDecoder();
+    let text = '';
+    for (;;) {
+      const { done, value } = await reader.read();
+      if (done) break;
+      const delta = decoder.decode(value, { stream: true });
+      text += delta;
+      onToken(delta);
+    }
+    return { text, served: { engineName: 'Our AI service', modelId: 'server-chosen', isLocal: false }, finishReason: 'stop' };
+  },
+};
+
+createMikeDownEditor(el, { host: { onChange, ai } });
+```
+
+**Tier 2: an engine registry.** Add `listEngines` (and optionally `testEngine`, `discoverModels`, `setSecret`, `requestCloudConsent`) and the editor shows the full AI tab: engines, models, capabilities and per-task routing. Core resolves which engine and model serves each request and passes it as `req.route`; your host sends it. The example calls a local Ollama from the browser, which suits development (start Ollama with `OLLAMA_ORIGINS` set to your dev origin so the browser may call it; in production, send these calls through your server).
+
+```ts
+import { createMikeDownEditor, type MikeDownAiHost } from '@mikedown/core';
+import {
+  aiErrorCodeFromHttp, buildAuthHeaders, buildCompletionBody, completeCapabilities, createCompletionStreamReader,
+  engineEndpoints, makeAiError, parseAiSettings, parseOllamaTags, type AiEngineStatus,
+} from '@mikedown/core/ai';
+
+const settings = parseAiSettings({
+  enabled: true,
+  engines: [{ id: 'ollama', name: 'Ollama (dev)', kind: 'ollama', baseUrl: 'http://127.0.0.1:11434/v1', models: [{ id: 'qwen3:8b' }] }],
+  routing: { default: { engineId: 'ollama', modelId: 'qwen3:8b' }, fallback: 'none', tasks: {} },
+});
+const engine = settings.engines[0];
+const urls = engineEndpoints(engine);
+
+async function probe(): Promise<AiEngineStatus> {
+  const started = performance.now();
+  try {
+    const res = await fetch(urls.nativeModels!);
+    if (!res.ok) throw makeAiError(aiErrorCodeFromHttp(res.status));
+    const ids = parseOllamaTags(await res.json());
+    return {
+      engineId: engine.id, ok: true, isLocal: true, hasKey: false, latencyMs: Math.round(performance.now() - started),
+      models: ids.map((id) => ({ id, capabilities: completeCapabilities({}, id, engine.kind), source: 'heuristic' as const })),
+    };
+  } catch (err) {
+    return { engineId: engine.id, ok: false, isLocal: true, hasKey: false, error: err instanceof Error ? err.message : 'Ollama is not reachable', models: [] };
+  }
+}
+
+const ai: MikeDownAiHost = {
+  async complete(req, onToken, signal) {
+    const model = engine.models.find((m) => m.id === req.route?.modelId) ?? engine.models[0];
+    const res = await fetch(urls.completions, {
+      method: 'POST',
+      headers: buildAuthHeaders(engine.kind, null),
+      body: JSON.stringify(buildCompletionBody(engine.kind, model, req)),
+      signal,
+    });
+    if (!res.ok) throw makeAiError(aiErrorCodeFromHttp(res.status, await res.text()));
+    const stream = createCompletionStreamReader(engine.kind);
+    const reader = res.body!.getReader();
+    const decoder = new TextDecoder();
+    for (;;) {
+      const { done, value } = await reader.read();
+      if (done) break;
+      for (const delta of stream.push(decoder.decode(value, { stream: true }))) onToken(delta);
+    }
+    const out = stream.finish();
+    return {
+      text: out.text, usage: out.usage, finishReason: out.finishReason,
+      served: { engineId: engine.id, engineName: engine.name, modelId: model.id, isLocal: true },
+    };
+  },
+  listEngines: async () => [await probe()],
+  testEngine: probe,
+  discoverModels: probe,
+};
+
+createMikeDownEditor(el, { host: { onChange, ai, saveSettings /* persists the ai.* keys */ }, settings: { ai: settings } });
+```
+
+**Consent.** Cloud engines are off until the user turns on *Allow cloud engines* in the AI tab. Even then, in tier 2 the first request a cloud engine would serve calls your `requestCloudConsent(engineId)`. Show your own dialog naming the engine, its URL and what is sent (the selection and one paragraph for rewrite; `aiConsentText` in `@mikedown/core/ai` has the words), and resolve `true` (allowed for the workspace; core remembers it for the session), `'once'` (this request only; the next one calls you again; a chat turn asks the optional `requestTurnConsent(engineId, about)` first, where it means the whole turn: each of its requests carries `req.turn`, and the optional `endTurn(turnId)` says when it is over), `false` (declined; it counts as unusable for that request, and you may answer later calls with `false` without a dialog, as VS Code and Desktop do until a restart) or `'never'` (a stored refusal; core disables the entries for that engine for the session). **A host without `requestCloudConsent` sends to cloud engines without asking** (the *Allow cloud engines* switch still applies), so implement it if users can add cloud engines. Optionally add `listConsents()` (records of `{ engineId, answer, at, scope }`) and `forgetConsent(engineId)`: the AI tab then shows a *Cloud consents* list with a Forget button for each. In tier 1 core never sees which model answers, so consent is your backend's and your UI's to handle.
+
+**Setup wizard.** With no engines configured, the toolbar's AI button opens a setup wizard (on the AI Engines page) instead of the engine form. It uses three more optional methods, each of which just hides its part of the wizard when absent: `detectLocalServers(candidates, signal)` looks for running local servers (one GET each, 1.5 s cap, only on request, never in the background; send a key only to the preset's own address), `hardwareInfo()` reports memory, CPU and GPU for the local model size hint, and `detectHostEngines()` lists models the host itself offers without an address or key (VS Code uses it for GitHub Copilot, the `vscode-lm` engine kind). Set `shellEnv: false` if your process cannot see shell variables, so the engine form points at a key file or a pasted key instead of `apiKeyEnv`. Choosing a cloud provider in the wizard turns *Allow cloud engines* on, and AI turns on when the user finishes with a usable default model.
+
+The wrappers pass `host.ai` through untouched: `host={{ ai }}` in React, `:host` in Vue, the `[ai]` input in Angular, and the `ai` property (properties only, no attribute) on `<mikedown-editor>`. Wrapper docs list the exact binding. The [developers hub](https://www.iadev.net/mikedown/developers/#byo-ai) has a live demo with a stub host that streams canned text.
+
+### Feature ids and request overrides
+
+Each AI feature has an id, and `host.ai` receives requests tagged with it. `rewrite` covers the rewrite popover, Explain and Translate; `summary` covers `/summary` and Summarize document; `title` covers `/title` and Suggest titles; `frontmatter` covers Properties Suggest (tags, title, description); `altText` covers image alt text. The full list is in `packages/core/src/ai/features.ts`. Use the id to route or log a feature, or to turn one off in your own UI.
+
+`AiCompletionRequest.requestOverrides` is a set of extra request body fields (for example `reasoning_effort` or `chat_template_kwargs`). Core fills it when the user changes the thinking dial in the rewrite popover, and merges it over the model's configured overrides, so the request wins. Hosts that do not support per-request fields can ignore it.
+
 ## Features
 
 Switch UI blocks with `options.features`. Defaults shown.
@@ -107,9 +223,12 @@ Switch UI blocks with `options.features`. Defaults shown.
 | `findReplace` | on | `Cmd/Ctrl+F` and `Cmd/Ctrl+H` |
 | `frontmatter` | on | Collapsible YAML block |
 | `supportCard` | off | VS Code only |
+| `customizeToolbar` | on | Settings > Toolbar section (Customize, Reset). Off keeps the toolbar fixed for the user |
 
 Also built in: tables, task lists, GFM callouts (`> [!NOTE]`), `==highlight==`, footnotes, emoji, wikilinks, code blocks with language picker.
 
+**Toolbar layout.** Users can reorder, remove and add toolbar buttons in the editor (Settings, then Toolbar). The result is one JSON value, `settings.toolbarLayout` (`{ version, items, seen }`, typed `ToolbarLayout`). Pass it in `settings` to preset or restore a layout, and store the value your `host.saveSettings` receives; `null` means Reset to Default, and an absent or invalid value shows the default toolbar. Set `features.customizeToolbar: false` to hide the customize controls; a `toolbarLayout` you pass still applies. `ToolbarLayout`, `ToolbarItemId`, `TOOLBAR_ITEM_IDS`, `DEFAULT_TOOLBAR_LAYOUT` and `normalizeToolbarLayout` are exported from `@mikedown/core`.
+
 ## Theming
 
 Colors, fonts and radii come only from `--mikedown-*` custom properties, with light and dark defaults in `@mikedown/core/tokens.css` (already included in `styles.css`). Every selector is scoped under `.mikedown-root` and `.mikedown-portal`, so the stylesheet cannot restyle your page. Set `theme: 'light' | 'dark' | 'auto'` or call `handle.setTheme()`.
@@ -130,7 +249,7 @@ Without `spellcheckDictionaryUrl`, spell check stays off and nothing is fetched.
 
 ## Shortcuts
 
-The core binds `Mod-/` (Cmd on macOS, Ctrl elsewhere) to the source toggle in both the rich editor and source mode. A host that binds the shortcut itself, such as the VS Code keybinding, passes `hostOwnsShortcuts: true` so one keystroke never toggles twice.
+The core binds `Mod-/` (Cmd on macOS, Ctrl elsewhere) to the source toggle in both the rich editor and source mode. A host that binds the shortcut itself, such as the VS Code keybinding, passes `hostOwnsShortcuts: true` so one keystroke never toggles twice. The same goes for `Mod-Alt-K`, which opens the AI rewrite popover when the host has an `ai` group (VS Code binds it as `mikedown.ai.rewrite`).
 
 ## Peer dependencies
 
@@ -197,7 +316,7 @@ Current Chrome, Edge, Firefox and Safari (ES2020 output, ESM only).
 
 ## Privacy
 
-No telemetry. The editor makes no network calls except the ones you configure: spell check dictionaries (when `spellcheckDictionaryUrl` is set), your `host` callbacks, and the lazy `import()` of KaTeX and Mermaid from your own bundle.
+No telemetry. The editor makes no network calls except the ones you configure: spell check dictionaries (when `spellcheckDictionaryUrl` is set), your `host` callbacks (including `host.ai`, which is the only way AI requests leave the page), and the lazy `import()` of KaTeX and Mermaid from your own bundle.
 
 ## Links
 

package/dist/ai/engines-page.d.ts

@@ -0,0 +1,137 @@
+import { MikeDownAiHost, AiSettings, AiEngineStatus, MikeDownDocsHost } from './index.js';
+import { AiSettingsPatch } from './index.js';
+import { MikeDownSettingsSectionContext } from '../index.js';
+import '@tiptap/core';
+
+/**
+ * One editing session over the AI settings, shared by the Settings modal's AI
+ * tab (`ai-settings.ts`) and the AI Engines page (`ai-engines-page.ts`): the
+ * draft, the save patch, and the bookkeeping that keeps API keys and early
+ * saves honest. No DOM; each view owns its own rendering and announcements.
+ *
+ * Everything edits a draft copy of the settings; `collect` gives only the
+ * `ai.*` keys that changed. API keys never touch the draft: they go through
+ * `host.ai.setSecret` as soon as a field settles, and a deleted engine's key
+ * is dropped on `commit`. Test connection and fetching models need the host
+ * to know the engine, so `ensureSaved` saves the engines first and waits for
+ * the host's settings echo. `discard` takes back engines added in this
+ * session that reached the host that way, and their keys.
+ *
+ * The page opened from the tab runs a child session (`fork`): it starts from
+ * the tab's draft, shares what the session knows about the host (statuses,
+ * stored keys), and on Done the tab `absorb`s the saved result.
+ */
+
+interface AiSettingsSessionOptions {
+    ai: MikeDownAiHost;
+    /** The settings the session starts from (the host's, or a parent session's draft). */
+    settings: AiSettings;
+    /** Last known engine statuses. */
+    statuses: readonly AiEngineStatus[];
+    /** The host's settings now (they change when the session saves engines before a test). */
+    getHostSettings(): AiSettings;
+    /** Persist an AI patch right away. Absent when the host cannot persist settings. */
+    saveNow?(patch: AiSettingsPatch): void;
+    /** Resolves with the host's next settings `accept` likes, or null after `timeoutMs`. */
+    waitForSettings(accept: (s: AiSettings) => boolean, timeoutMs: number): Promise<AiSettings | null>;
+    /** Fresh statuses (list, test, discover) for the editor's cache and the indicator. */
+    onStatuses(statuses: AiEngineStatus[]): void;
+    /** Statuses that arrive without the view asking (a host push, a settings refresh). Returns an unsubscribe. */
+    subscribeStatuses?(listener: (statuses: AiEngineStatus[]) => void): () => void;
+    /**
+     * A child session opened while the parent's "Enable AI features" is ticked
+     * but not saved: when this session saves anything, it saves
+     * `ai.enabled: true` with it, so finishing on the Engines page keeps the
+     * tick (Q2).
+     */
+    pendingEnable?: boolean;
+}
+
+/**
+ * The AI Engines page (M3c): engines and routing on a full-size page of
+ * their own, laid out like VS Code's Settings editor or the Network pane of
+ * Apple's System Settings. A header with Back and Done; on the left a quiet
+ * list of engines, then Tasks (which model does what), then Add engine; on
+ * the right the selected engine's Express form (`ai-engine-form.ts`), the
+ * Tasks section (`ai-routing-section.ts`) or the quick-start cards. Under 720 px the page
+ * is one column: the list first, and picking an entry pushes its detail,
+ * with Back returning to the list.
+ *
+ * Standalone and mountable: it renders into the container it is given and
+ * touches nothing outside it (no modal, no editor, no `document.body`), so a
+ * host can show it as an overlay (`editor.ts#showAiEnginesPage`) or, later,
+ * in its own editor tab or window. Dialog semantics, the focus trap and
+ * focus restore belong to whoever hosts it.
+ *
+ * Edits run in an `AiSettingsSession` (`ai-settings-session.ts`): Done hands
+ * the changed `ai.*` keys to `onDone`; Back with unsaved changes asks
+ * "Discard changes?" in the page, and discarding takes back engines added
+ * here and their keys before `onCancel`. Opened by the first-run panel's
+ * "Set up" (`turnOnWhenDone`), the header says AI turns on at Done, and Done
+ * adds `ai.enabled: true` when the result has a usable default route; Back
+ * leaves AI off.
+ *
+ * On a host that can look for local servers (`host.ai.detectLocalServers`),
+ * the quick-start cards look when they first show and on "Look again": a
+ * local card says Running · 3 models (and leads), Not running, or Added
+ * (it opens the engine already set up there). Without the method the cards
+ * are the plain list.
+ *
+ * The cloud cards carry the **Allow cloud engines** switch, one line on what
+ * it means and the privacy card folded under "What this means"; it edits the
+ * page's draft, so Done saves it with the rest. A cloud engine's head, while
+ * the switch is off, says so with a "Turn on cloud engines" button (F4). On
+ * VS Code, a "Built into VS Code" group offers GitHub Copilot when the host
+ * reports a Copilot model (`detectHostEngines`).
+ *
+ * Opened on the `'wizard'` section (the first-run panel's "Set up" with no
+ * engines yet, tier 2 only), the page shows the setup wizard
+ * (`ai-setup-wizard.ts`) instead of the list and detail: the header says
+ * "Set up AI" with the step dots and Skip, Back walks back through the
+ * steps, and the wizard's Done is the page's Done. "Change settings in AI Engines"
+ * leaves the wizard for the full page, in the same session.
+ *
+ * On a host with `host.docs` the list also has Documentation bases
+ * (`ai-docs-page.ts`), the `'bases'` section. A base saves at once through the
+ * host; only the bases' advanced options wait for Done.
+ *
+ * Modal UI only: nothing here touches the editor document or its DOM.
+ */
+
+type AiEnginesPageSection = 'engines' | 'routing' | 'wizard' | 'bases';
+interface AiEnginesPageOptions extends AiSettingsSessionOptions {
+    /**
+     * What to show first: an engine, the routing section, the documentation
+     * bases (with `docs`) or the setup wizard (tier 2 only). Default: the first
+     * engine (the cards when there is none).
+     */
+    initial?: {
+        engineId?: string;
+        section?: AiEnginesPageSection;
+    };
+    /** Shared with the session that opened the page (stored-key knowledge). */
+    keyOverride?: Map<string, boolean>;
+    /** Shared with the session that opened the page (engine statuses). */
+    statusMap?: Map<string, AiEngineStatus>;
+    /** Done: the `ai.*` keys that changed (possibly none) and the settings they lead to. */
+    onDone(patch: AiSettingsPatch, settings: AiSettings): void;
+    /** Back without saving; the page already took back what it added. */
+    onCancel(): void;
+    /**
+     * Opened to set AI up while it is off: Done also turns AI on (`ai.enabled`
+     * in the patch) when the result has a usable default route.
+     */
+    turnOnWhenDone?: boolean;
+    /** The host's documentation bases. Absent: no Documentation bases section. */
+    docs?: MikeDownDocsHost;
+}
+interface AiEnginesPageHandle {
+    /** Tear down (timers, listeners) and leave the container empty. Unresolved changes are discarded. */
+    dispose(): void;
+    /** Focus where a keyboard user should land: the selected list entry, or the detail when it was pushed. */
+    focus(): void;
+}
+declare function mountAiEnginesPage(container: HTMLElement, ctx: MikeDownSettingsSectionContext, opts: AiEnginesPageOptions): AiEnginesPageHandle;
+
+export { mountAiEnginesPage };
+export type { AiEnginesPageHandle, AiEnginesPageOptions, AiEnginesPageSection };

package/dist/ai/engines-page.js

@@ -0,0 +1,11 @@
+import {
+  mountAiEnginesPage
+} from "../chunk-DWXGWVA3.js";
+import "../chunk-N4N54LK2.js";
+import "../chunk-IWCXWUWW.js";
+import "../chunk-APPBNQEY.js";
+import "../chunk-TSP7BAV3.js";
+export {
+  mountAiEnginesPage
+};
+//# sourceMappingURL=engines-page.js.map

package/dist/ai/engines-page.js.map

@@ -0,0 +1 @@
+{"version":3,"sources":[],"sourcesContent":[],"mappings":"","names":[]}