@aliou/pi-dev-kit 0.6.5 → 0.7.0

package/src/skills/pi-extension/references/providers.md

@@ -1,15 +1,16 @@
 # Providers
 
-Providers add LLM backends to pi. They connect pi to model APIs, proxies, gateways, and custom streaming implementations.
+Providers add or override LLM backends through `pi.registerProvider(name, config)`. Use them for proxies, custom endpoints, OAuth/SSO, and custom streaming APIs.
 
-This reference tracks the current `pi.registerProvider(name, config)` API from pi-mono. For full provider details and advanced examples, also read pi-mono `packages/coding-agent/docs/custom-provider.md`.
+Read Pi `docs/custom-provider.md` and `docs/models.md` before implementing a provider.
 
-## Registration
+## Quick Registration
 
 ```typescript
-import type { ExtensionAPI, ProviderConfig } from "@mariozechner/pi-coding-agent";
+import type { ExtensionAPI, ProviderConfig } from "@earendil-works/pi-coding-agent";
 
-const myProviderConfig: ProviderConfig = {
+const myProvider: ProviderConfig = {
+  name: "My Provider",
   baseUrl: "https://api.example.com/v1",
   apiKey: "MY_API_KEY",
   api: "openai-completions",
@@ -26,134 +27,292 @@ const myProviderConfig: ProviderConfig = {
   ],
 };
 
-export default function (pi: ExtensionAPI) {
-  pi.registerProvider("my-provider", myProviderConfig);
+export default function providersExtension(pi: ExtensionAPI) {
+  pi.registerProvider("my-provider", myProvider);
 }
 ```
 
-## Common Registration Patterns
+Use legacy `@mariozechner/*` imports only when the target `@earendil-works/*` package is not available yet.
 
-### Override an existing provider
+## Async Model Discovery
 
-```typescript
-import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
+If models come from a remote endpoint, fetch them in an async extension factory, not `session_start`. Pi waits for the factory before startup continues, so models are available during startup and `pi --list-models`.
 
-export default function (pi: ExtensionAPI) {
-  pi.registerProvider("anthropic", {
-    baseUrl: "https://proxy.example.com",
+```typescript
+export default async function providersExtension(pi: ExtensionAPI) {
+  const response = await fetch("http://localhost:1234/v1/models");
+  const payload = (await response.json()) as {
+    data: Array<{ id: string; name?: string; context_window?: number; max_tokens?: number }>;
+  };
+
+  pi.registerProvider("local-openai", {
+    name: "Local OpenAI",
+    baseUrl: "http://localhost:1234/v1",
+    apiKey: "LOCAL_OPENAI_API_KEY",
+    api: "openai-completions",
+    models: payload.data.map((model) => ({
+      id: model.id,
+      name: model.name ?? model.id,
+      reasoning: false,
+      input: ["text"],
+      cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
+      contextWindow: model.context_window ?? 128000,
+      maxTokens: model.max_tokens ?? 4096,
+    })),
   });
 }
 ```
 
-Use this when you want to keep the built-in provider and model list, but change the endpoint and/or headers.
+## Override Existing Providers
 
-### Register a new provider
+When only `baseUrl` and/or `headers` are provided, Pi keeps built-in models and auth.
 
 ```typescript
-import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
-
-export default function (pi: ExtensionAPI) {
-  pi.registerProvider("my-provider", {
-    baseUrl: "https://api.example.com/v1",
-    apiKey: "MY_API_KEY",
-    api: "openai-completions",
-    models: [
-      {
-        id: "my-model",
-        name: "My Model",
-        reasoning: false,
-        input: ["text"],
-        cost: { input: 0.5, output: 1.5, cacheRead: 0, cacheWrite: 0 },
-        contextWindow: 128000,
-        maxTokens: 8192,
-      },
-    ],
-  });
-}
+pi.registerProvider("anthropic", {
+  baseUrl: "https://proxy.example.com",
+});
+
+pi.registerProvider("openai", {
+  headers: {
+    "X-Custom-Header": "MY_HEADER_ENV_OR_LITERAL",
+  },
+});
 ```
 
-### Unregister a provider
+## Unregister Providers
 
 ```typescript
 pi.unregisterProvider("my-provider");
 ```
 
-This takes effect immediately at runtime.
+Unregistering removes dynamic models, API key fallback, OAuth registration, and stream handlers. Built-in behavior overridden by that provider is restored.
 
 ## ProviderConfig Fields
 
-| Field | Type | Description |
-|---|---|---|
-| `baseUrl` | `string` | Base URL for the provider or proxy. |
-| `headers` | `Record<string, string>` | Optional static headers to add to requests. |
-| `apiKey` | `string` | Environment variable name containing the API key. |
-| `api` | `"openai-completions" \| "openai-responses"` | Compatibility mode for request/response handling. |
-| `models` | `ProviderModelConfig[]` | Model definitions exposed by this provider. |
-| `streamSimple` | `function` | Optional custom streaming implementation for non-standard APIs. |
-| `oauth` | `object` | Optional OAuth config for providers that need browser-based auth. |
+| Field | Notes |
+|---|---|
+| `name` | Display name for `/login` and UI. |
+| `baseUrl` | Endpoint URL. Required when defining models. |
+| `apiKey` | API key, env var name, or auth value. Required unless `oauth` handles auth. |
+| `api` | API type. Required at provider or model level when defining models. |
+| `headers` | Static custom headers. Values can be env var names. |
+| `authHeader` | When `true`, Pi sends `Authorization: Bearer <key>`. |
+| `models` | Replaces registered models for this provider when provided. |
+| `oauth` | Adds `/login` support. |
+| `streamSimple` | Custom streaming implementation for non-standard APIs. |
+
+Supported API types include:
+
+- `anthropic-messages`
+- `openai-completions`
+- `openai-responses`
+- `azure-openai-responses`
+- `openai-codex-responses`
+- `mistral-conversations`
+- `google-generative-ai`
+- `google-vertex`
+- `bedrock-converse-stream`
+
+Prefer a built-in API type. Use `streamSimple` only when the upstream API cannot be adapted with config and compatibility flags.
+
+## Model Fields
+
+| Field | Required | Notes |
+|---|---:|---|
+| `id` | Yes | Model ID sent to the API. |
+| `name` | Yes for provider extensions | Human-readable display name. |
+| `api` | No | Model-level API override. |
+| `baseUrl` | No | Model-level endpoint override. |
+| `reasoning` | Yes | Whether extended thinking is supported. |
+| `thinkingLevelMap` | No | Maps Pi thinking levels to provider-specific values; `null` hides unsupported levels. |
+| `input` | Yes | `Array<"text" | "image">`. |
+| `cost` | Yes | Per-million token cost: `{ input, output, cacheRead, cacheWrite }`. |
+| `contextWindow` | Yes | Context window in tokens. |
+| `maxTokens` | Yes | Maximum output tokens. |
+| `headers` | No | Model-specific headers. |
+| `compat` | No | Provider compatibility flags. |
+
+### Thinking level map
+
+Use model-level `thinkingLevelMap`; do not use older `compat.reasoningEffortMap`.
 
-Use the built-in OpenAI-compatible path when possible. Reach for `streamSimple` only when the upstream API is not compatible enough.
+```typescript
+{
+  id: "custom-reasoner",
+  name: "Custom Reasoner",
+  reasoning: true,
+  thinkingLevelMap: {
+    minimal: null,
+    low: null,
+    medium: null,
+    high: "default",
+    xhigh: "max",
+  },
+  input: ["text"],
+  cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
+  contextWindow: 128000,
+  maxTokens: 8192,
+}
+```
+
+## Compatibility Flags
 
-## Model Definition
+For OpenAI-compatible servers, use `compat` instead of custom streaming when possible.
 
-The exact model type has more fields, but these are the ones you will usually need:
+```typescript
+compat: {
+  supportsDeveloperRole: false,
+  supportsReasoningEffort: false,
+  supportsUsageInStreaming: false,
+  maxTokensField: "max_tokens",
+  requiresToolResultName: true,
+  thinkingFormat: "qwen-chat-template",
+  cacheControlFormat: "anthropic",
+}
+```
 
-| Field | Type | Description |
-|---|---|---|
-| `id` | `string` | Model identifier within the provider config. |
-| `name` | `string` | Display name shown in model selection UI. |
-| `reasoning` | `boolean` | Whether the model is a reasoning model. |
-| `input` | `Array<"text" \| "image" \| "audio" \| "pdf">` | Input modalities supported by the model. |
-| `cost` | `object` | `{ input, output, cacheRead, cacheWrite }` cost values. |
-| `contextWindow` | `number` | Maximum context window. |
-| `maxTokens` | `number` | Maximum output tokens. |
+Common flags:
 
-## API Key Gating
+- `supportsDeveloperRole`
+- `supportsReasoningEffort`
+- `supportsUsageInStreaming`
+- `maxTokensField`
+- `requiresToolResultName`
+- `requiresAssistantAfterToolResult`
+- `requiresThinkingAsText`
+- `requiresReasoningContentOnAssistantMessages`
+- `thinkingFormat`
+- `cacheControlFormat`
+- `supportsStrictMode`
+- `supportsLongCacheRetention`
+- `openRouterRouting`
+- `vercelGatewayRouting`
 
-Provider registration and extension tool registration are separate concerns.
+For Anthropic-compatible proxies, check `supportsEagerToolInputStreaming` and `supportsLongCacheRetention` in Pi docs.
 
-For providers:
-- Register the provider with `pi.registerProvider(name, config)`.
-- Point `apiKey` at the environment variable name that holds the credential.
-- If the provider should exist even when tools are disabled, still register it.
+## OAuth Providers
 
-For tools and commands that require the same credential:
-- Gate those registrations separately in your extension entry point.
+OAuth integrates with `/login`.
 
 ```typescript
-export default function (pi: ExtensionAPI) {
-  pi.registerProvider("my-provider", {
-    baseUrl: "https://api.example.com/v1",
-    apiKey: "MY_API_KEY",
-    api: "openai-completions",
-    models: [
-      {
-        id: "my-model",
-        name: "My Model",
-        reasoning: false,
-        input: ["text"],
-        cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
-        contextWindow: 128000,
-        maxTokens: 4096,
-      },
-    ],
-  });
+import type { OAuthCredentials, OAuthLoginCallbacks } from "@earendil-works/pi-ai";
+
+pi.registerProvider("corporate-ai", {
+  name: "Corporate AI",
+  baseUrl: "https://ai.corp.com/v1",
+  api: "openai-responses",
+  models,
+  oauth: {
+    name: "Corporate AI (SSO)",
+    async login(callbacks: OAuthLoginCallbacks): Promise<OAuthCredentials> {
+      callbacks.onAuth({ url: "https://sso.corp.com/authorize" });
+      const code = await callbacks.onPrompt({ message: "Enter code:" });
+      return exchangeCodeForTokens(code);
+    },
+    async refreshToken(credentials) {
+      return refreshTokens(credentials.refresh);
+    },
+    getApiKey(credentials) {
+      return credentials.access;
+    },
+    modifyModels(models, credentials) {
+      return models;
+    },
+  },
+});
+```
+
+Use `modifyModels` when subscription, tenant, or region information in credentials changes model endpoints or availability.
 
-  if (!process.env.MY_API_KEY) return;
+## API Key Gating
+
+Provider registration and tool registration are separate.
+
+- Register providers even when the API key is absent; Pi resolves auth and can show login/setup UI.
+- Gate tools and commands that directly call the same API.
+
+```typescript
+export default function extension(pi: ExtensionAPI) {
+  pi.registerProvider("my-provider", providerConfig);
 
-  pi.registerTool(mySearchTool);
-  pi.registerCommand("quota", { handler: showQuota });
+  if (!process.env.MY_API_KEY) {
+    pi.on("session_start", (_event, ctx) => {
+      ctx.ui.notify("MY_API_KEY not set. my-provider tools disabled.", "warning");
+    });
+    return;
+  }
+
+  pi.registerTool(createSearchTool(process.env.MY_API_KEY));
+}
+```
+
+Missing keys should disable affected tools gracefully, not crash extension loading.
+
+## Custom Streaming
+
+Use `streamSimple` only for non-standard APIs. Follow Pi provider implementations and tests.
+
+Basic pattern:
+
+```typescript
+import {
+  calculateCost,
+  createAssistantMessageEventStream,
+  type AssistantMessage,
+  type Context,
+  type Model,
+  type SimpleStreamOptions,
+} from "@earendil-works/pi-ai";
+
+function streamMyProvider(model: Model<any>, context: Context, options?: SimpleStreamOptions) {
+  const stream = createAssistantMessageEventStream();
+
+  (async () => {
+    const output: AssistantMessage = {
+      role: "assistant",
+      content: [],
+      api: model.api,
+      provider: model.provider,
+      model: model.id,
+      usage: {
+        input: 0,
+        output: 0,
+        cacheRead: 0,
+        cacheWrite: 0,
+        totalTokens: 0,
+        cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
+      },
+      stopReason: "stop",
+      timestamp: Date.now(),
+    };
+
+    try {
+      stream.push({ type: "start", partial: output });
+      // Push text/thinking/toolcall events as data arrives.
+      calculateCost(model, output.usage);
+      stream.push({ type: "done", reason: "stop", message: output });
+      stream.end();
+    } catch (error) {
+      output.stopReason = options?.signal?.aborted ? "aborted" : "error";
+      output.errorMessage = error instanceof Error ? error.message : String(error);
+      stream.push({ type: "error", reason: output.stopReason, error: output });
+      stream.end();
+    }
+  })();
+
+  return stream;
 }
 ```
 
-That pattern keeps provider setup accurate while still hiding tools that cannot work without credentials.
+Test custom providers against streaming, abort, usage, unicode, tool call, and context-overflow cases.
 
-## When to read the upstream docs
+## Checklist
 
-Also read pi-mono `packages/coding-agent/docs/custom-provider.md` when you need:
-- custom streaming via `streamSimple`
-- OAuth support
-- proxying existing providers
-- header injection
-- provider teardown with `pi.unregisterProvider()`
-- advanced model config details
+- [ ] Provider uses `pi.registerProvider(name, config)`.
+- [ ] Dynamic model discovery happens in an async factory.
+- [ ] Existing provider overrides do not redefine models unless needed.
+- [ ] New models include current fields and `thinkingLevelMap` when relevant.
+- [ ] Compatibility flags are used before custom streaming.
+- [ ] OAuth providers implement login, refresh, getApiKey, and optional modifyModels.
+- [ ] Tools needing the same credential are gated separately.
+- [ ] Missing credentials produce a notification, not a crash.

package/src/skills/pi-extension/references/publish.md

@@ -1,12 +1,10 @@
 # Publishing
 
-Extensions are published to npm and installed with `pi install`.
+Pi packages are published to npm or installed from git/local paths with `pi install`.
 
 ## Package Setup
 
-The `package.json` must have the `pi` key declaring extension resources. See `references/structure.md` for the full template.
-
-Key fields for publishing:
+A publishable package needs Pi metadata and discoverability fields.
 
 ```json
 {
@@ -15,20 +13,42 @@ Key fields for publishing:
   "type": "module",
   "license": "MIT",
   "private": false,
+  "keywords": ["pi-package", "pi-extension", "pi"],
   "publishConfig": { "access": "public" },
   "files": ["src", "README.md"],
   "pi": {
-    "extensions": ["./src/index.ts"]
+    "extensions": ["./src/tools/index.ts", "./src/commands/index.ts"]
   },
   "peerDependencies": {
-    "@mariozechner/pi-coding-agent": ">=0.51.0"
+    "@earendil-works/pi-coding-agent": "*",
+    "typebox": "*"
+  },
+  "peerDependenciesMeta": {
+    "@earendil-works/pi-coding-agent": { "optional": true },
+    "typebox": { "optional": true }
   }
 }
 ```
 
+Use the legacy `@mariozechner/*` namespace only while the target Pi packages are not published under `@earendil-works/*`.
+
+Pi core packages imported at runtime belong in optional `peerDependencies` with `"*"`. Keep exact target versions in `devDependencies` for local type checking. Third-party runtime packages belong in `dependencies`.
+
+## Installation Specs
+
+Users can install with:
+
+```bash
+pi install npm:@scope/pi-my-extension
+pi install git:github.com/org/pi-my-extension@v1.0.0
+pi install ./relative/path/to/package
+```
+
+By default, global installs write to `~/.pi/agent/settings.json`; `-l` writes project settings.
+
 ## Versioning with Changesets
 
-Use [changesets](https://github.com/changesets/changesets) for versioning and changelogs.
+Use Changesets for versioning and changelogs.
 
 ### `.changeset/config.json`
 
@@ -48,92 +68,86 @@ Use [changesets](https://github.com/changesets/changesets) for versioning and ch
 
 ### Creating a changeset
 
-The interactive CLI (`pnpm changeset`) is not available in headless environments. Write the file directly instead. Create `.changeset/<descriptive-name>.md`:
+In headless agent sessions, write the file directly instead of running the interactive CLI.
 
 ```md
 ---
 "@scope/pi-my-extension": patch
 ---
 
-Description of what changed and why it matters to users.
+Describe the user-visible change.
 ```
 
-Bump types: `patch` for fixes and internal changes, `minor` for new user-facing features, `major` for breaking changes.
+Use:
+
+- `patch` for fixes and internal compatibility updates.
+- `minor` for new user-facing features.
+- `major` for breaking changes.
 
-Commit the changeset file alongside the changes it describes. Multiple changeset files can coexist — they are all consumed together on the next release.
+Commit the changeset with the change it describes.
 
-### Releasing manually (no CI)
+### Manual release
 
 ```bash
-pnpm changeset version   # consumes .changeset/*.md, bumps version, updates CHANGELOG.md
-pnpm changeset publish   # publishes to npm
+pnpm changeset version
+pnpm changeset publish
 ```
 
 ## GitHub Actions Automation
 
-The recommended setup uses a single `publish.yml` workflow that runs on every push to `main`. It handles two cases automatically:
-
-- **Pending changesets present**: opens (or updates) a version PR titled `Updating @scope/pi-my-extension to version X.Y.Z`.
-- **Version PR merged**: publishes the package to npm and creates a GitHub release with a matching git tag.
-
-Copy `.github/workflows/publish.yml` from `pi-extension-template` into the new repo. It uses `changesets/action@v1` under the hood.
+Use the template publish workflow from `pi-extension-template`.
 
-The workflow requires two secrets, configured in the repo's GitHub settings under **Settings → Secrets and variables → Actions**:
+It handles:
 
-- `GITHUB_TOKEN` — automatically provided by GitHub Actions, no setup needed.
-- `NPM_TOKEN` — an npm automation token with publish access to the `@scope` org. Create one at npmjs.com under **Access Tokens → Generate New Token → Automation**. Add it as a repository secret named `NPM_TOKEN`. Without this, the publish step will fail silently on the version PR merge.
+- Pending changesets: opens or updates a version PR.
+- Version PR merged: publishes to npm and creates a GitHub release.
 
-The workflow also sets `NPM_CONFIG_PROVENANCE=true`, which links the published package to the GitHub Actions run for supply chain transparency (requires the `id-token: write` permission, already included in the template).
+Required secrets:
 
-## First-time Setup for a New Package
+- `GITHUB_TOKEN`: provided by GitHub Actions.
+- `NPM_TOKEN`: npm automation token with publish access.
 
-Before the workflow can publish a package that has never been on npm:
+Keep `NPM_CONFIG_PROVENANCE=true` and `id-token: write` for provenance.
 
-1. Make sure `"private": false` and `"publishConfig": { "access": "public" }` are in `package.json`.
-2. Add the `NPM_TOKEN` secret to the repo (see above).
-3. The first time the version PR is merged, the workflow publishes the package. npm will create the package entry automatically — no manual `npm publish` needed.
+## First Publish
 
-If the package name is scoped (e.g., `@aliou/pi-my-extension`) and the scope is new to your npm account, you may need to create the scope first at npmjs.com or run `npm publish --access public` once manually to register it.
+Before first publish:
 
-## Installation
-
-Users install extensions with:
-
-```bash
-pi install @scope/pi-my-extension
-```
+1. Set `"private": false`.
+2. Set `"publishConfig": { "access": "public" }` for scoped public packages.
+3. Add `NPM_TOKEN` to repo secrets.
+4. Merge the first Changesets version PR.
 
-Pi reads the `pi` key from the package's `package.json` to discover extensions, skills, themes, and prompts.
+npm creates the package entry on first publish. If the npm scope is new to your account, you may need to create the scope or publish once manually with `--access public`.
 
-## Dependency Management in Monorepos
+## Monorepo Dependency Rules
 
-If publishing from a monorepo that contains both public and private packages:
+Public packages cannot depend on private workspace packages. Users installing from npm will not have those private packages.
 
-**Critical rule**: Public packages cannot depend on private workspace packages. This will break when users try to install your package from npm.
+When adding a dependency:
 
-In the pi-extensions monorepo, this is enforced by:
-- Pre-commit hook that blocks commits with invalid dependencies
-- CI check that prevents merging bad dependencies
-- `pnpm run check:public-deps` validates all dependencies
+1. If it is a public workspace package, use `workspace:^` in the monorepo and make sure it is published.
+2. If it is private, do not depend on it from a public package.
+3. For external packages, use a normal npm range.
 
-When adding a workspace dependency to a `package.json`:
-1. Check if the dependency is public (`"private": false` or `"publishConfig": { "access": "public" }`).
-2. If the dependency is private, either make it public, make your package private, or remove the dependency.
+Run the repo's public-dependency check when available, for example `pnpm run check:public-deps`.
 
 ## Pre-publish Checklist
 
-- [ ] `"private": false` is set.
-- [ ] `"publishConfig": { "access": "public" }` is set.
-- [ ] `"files"` lists only what users need (`["src", "README.md"]`).
-- [ ] `peerDependencies` version range is correct (`>=` minimum supported version).
-- [ ] `@mariozechner/pi-tui` is in `peerDependencies` with `optional: true` in `peerDependenciesMeta` if imported at runtime.
-- [ ] `prepare` script is `[ -d .git ] && husky || true`, not bare `husky`.
-- [ ] `check:lockfile` script is present.
-- [ ] `description` is clear and concise.
-- [ ] `pi.extensions` paths are correct.
-- [ ] `NPM_TOKEN` secret is set on the GitHub repo.
-- [ ] `.github/workflows/publish.yml` is present.
-- [ ] If in a monorepo: no dependency on private workspace packages (`pnpm run check:public-deps` if available).
-- [ ] README documents what the extension does, required environment variables, and available tools/commands.
-- [ ] If wrapping a third-party API: extension handles missing API key gracefully (notification, not crash).
+- [ ] `private` is `false`.
+- [ ] `publishConfig.access` is `public` for scoped public packages.
+- [ ] `keywords` includes `pi-package`.
+- [ ] `files` lists only shipped files users need.
+- [ ] `pi.extensions`, `pi.skills`, `pi.prompts`, and `pi.themes` paths are correct.
+- [ ] Demo `pi.video` or `pi.image` metadata is present when available.
+- [ ] Imported Pi core packages are optional peers with `"*"`.
+- [ ] Imported Pi core packages are exact dev dependencies for type checking.
+- [ ] Third-party runtime packages are in `dependencies`.
+- [ ] No private workspace dependencies in public packages.
+- [ ] `prepare` is `[ -d .git ] && husky || true`, not bare `husky`.
+- [ ] `check:lockfile` exists.
+- [ ] README documents setup, tools, commands, providers, env vars, and limitations.
+- [ ] Missing API keys are handled with notifications or disabled features, not crashes.
 - [ ] `pnpm typecheck` and `pnpm lint` pass.
+- [ ] `.github/workflows/publish.yml` is present if using CI publish.
+- [ ] `NPM_TOKEN` is configured before relying on CI publish.