@khanglvm/llm-router 1.0.6 → 1.0.9

package/CHANGELOG.md

@@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.0.9] - 2026-03-03
+
+### Added
+- Added dedicated modules for Cloudflare API preflight checks and Wrangler TOML target handling.
+- Added runtime policy and route-debug helpers so stateful routing can be safely disabled by default on Cloudflare Worker.
+- Added reusable timeout-signal utility and start-command port reclaim utilities with test coverage.
+
+### Changed
+- Refactored CLI deploy/runtime handler code into focused modules with cleaner boundaries.
+- Updated provider-call timeout handling to support both `AbortSignal.timeout` and `AbortController` fallback.
+- Documented Worker safety defaults and switched README release/security links to canonical GitHub URLs.
+
+## [1.0.8] - 2026-02-28
+
+### Changed
+- Added focused npm `keywords` metadata in `package.json` to improve package discoverability.
+
+## [1.0.7] - 2026-02-28
+
+### Added
+- Added `llm-router ai-help` to generate an agent-oriented operating guide with live gateway checks and coding-tool patch instructions.
+- Added tests covering `ai-help` discovery output and first-run setup guidance.
+
+### Changed
+- Rewrote `README.md` into a shorter setup and operations guide focused on providers, aliases, rate limits, and local/hosted usage.
+
 ## [1.0.6] - 2026-02-28
 
 ### Added

package/README.md

@@ -1,440 +1,193 @@
 # llm-router
 
-`llm-router` is a gateway api proxy for accessing multiple models across any provider that supports OpenAI or Anthropic formats.
+`llm-router` exposes unified API endpoint for multiple AI providers and models.
 
-It supports:
-- local route server `llm-router start`
-- Cloudflare Worker route runtime deployment `llm-router deploy`
-- CLI + TUI management `config`, `start`, `deploy`, `worker-key`
-- Seamless model fallback
+## Main feature
 
-## Install
-
-```bash
-npm i -g @khanglvm/llm-router
-```
-
-## Versioning
+1. Single endpoint, unified providers & models
+2. Support grouping models with rate-limit and load balancing strategy
+3. Configuration auto reload in real time, no interruption
 
-- Follows [Semantic Versioning](https://semver.org/).
-- Release notes live in [`CHANGELOG.md`](./CHANGELOG.md).
-- npm publishes are configured for the public registry package.
-
-Release checklist:
-- Update `README.md` if user-facing behavior changed.
-- Add a dated entry in `CHANGELOG.md`.
-- Bump the package version before publish.
-- Publish with `npm publish`.
-
-## Quick Start
+## Install
 
 ```bash
-# 1) Open config TUI (default behavior) to manage providers, models, fallbacks, and auth
-llm-router
-
-# 2) Start local route server
-llm-router start
+npm i -g @khanglvm/llm-router@latest
 ```
 
-Local endpoints:
-- Unified (Auto transform): `http://127.0.0.1:8787/route` (or `/` and `/v1`)
-- Anthropic: `http://127.0.0.1:8787/anthropic`
-- OpenAI: `http://127.0.0.1:8787/openai`
-
-## Usage Example
-
-```bash
-# Your AI Agent can help! Ask them to manage api router via this tool for you.
-
-# 1) Add provider + models + provider API key. You can ask your AI agent to do it for you, or manually via TUI or command line:
-llm-router config \
-  --operation=upsert-provider \
-  --provider-id=openrouter \
-  --name="OpenRouter" \
-  --base-url=https://openrouter.ai/api/v1 \
-  --api-key=sk-or-v1-... \
-  --models=claude-3-7-sonnet,gpt-4o \
-  --format=openai \
-  --skip-probe=true
-
-# 2) (Optional) Configure model fallback order for direct provider/model requests
-llm-router config \
-  --operation=set-model-fallbacks \
-  --provider-id=openrouter \
-  --model=claude-3-7-sonnet \
-  --fallback-models=openrouter/gpt-4o
-
-# 3) (Optional) Create a model alias with a routing strategy and weighted targets
-llm-router config \
-  --operation=upsert-model-alias \
-  --alias-id=chat.default \
-  --strategy=auto \
-  --targets=openrouter/claude-3-7-sonnet@2,openrouter/gpt-4o@1 \
-  --fallback-targets=openrouter/gpt-4o-mini
-
-# 4) (Optional) Add provider request-cap bucket (models: all)
-llm-router config \
-  --operation=set-provider-rate-limits \
-  --provider-id=openrouter \
-  --bucket-name="Monthly cap" \
-  --bucket-models=all \
-  --bucket-requests=20000 \
-  --bucket-window=month:1
-
-# 5) Set master key (this is your gateway key for client apps)
-llm-router config --operation=set-master-key --master-key=gw_your_gateway_key
-
-# 6) Start gateway with auth required
-llm-router start --require-auth=true
-```
+## Usage
 
-Claude Code example (`~/.claude/settings.local.json`):
+Copy/paste this short instruction to your AI agent:
 
-```json
-{
-  "env": {
-    "ANTHROPIC_BASE_URL": "http://127.0.0.1:8787/anthropic",
-    "ANTHROPIC_AUTH_TOKEN": "gw_your_gateway_key"
-  }
-}
+```text
+Run `llm-router ai-help` first, then set up and operate llm-router for me using CLI commands.
 ```
 
-## Smart Fallback Behavior
-
-`llm-router` can fail over from a primary model to configured fallback models with status-aware logic:
-- `429` (rate-limited): immediate fallback (no origin retry), with `Retry-After` respected when present.
-- Temporary failures (`408`, `409`, `5xx`, network errors): origin-only bounded retries with jittered backoff, then fallback.
-- Billing/quota exhaustion (`402`, or provider-specific billing signals): immediate fallback with longer origin cooldown memory.
-- Auth and permission failures (`401` and relevant `403` cases): no retry; fallback to other providers/models when possible.
-- Policy/moderation blocks: no retry; cross-provider fallback is disabled by default (`LLM_ROUTER_ALLOW_POLICY_FALLBACK=false`).
-- Invalid client requests (`400`, `413`, `422`): no retry and no fallback short-circuit.
+## Main Workflow
 
-## Model Alias Routing Strategies
+1. Add Providers + models into llm-router
+2. Optionally, group models as alias with load balancing and auto fallback support
+3. Start llm-router server, point your coding tool API and model to llm-router
 
-A model alias groups multiple models from different providers under one model name.
+## What Each Term Means
 
-Use `--strategy` when creating or updating a model alias:
+### Provider
+The service endpoint you call (OpenRouter, Anthropic, etc.).
 
-- `auto`: Recommended set-and-forget mode. Automatically routes using quota, cooldown, and health signals to reduce rate-limit failures.
-- `ordered`: Tries targets in list order.
-- `round-robin`: Rotates evenly across eligible targets.
-- `weighted-rr`: Rotates like round-robin, but favors higher weights.
-- `quota-aware-weighted-rr`: Weighted routing plus remaining-capacity awareness.
-
-Example:
-
-```bash
-llm-router config \
-  --operation=upsert-model-alias \
-  --alias-id=coding \
-  --strategy=auto \
-  --targets=rc/gpt-5.3-codex,zai/glm-5
-```
-
-Concrete model alias example with provider-specific caps:
-
-```bash
-llm-router config \
-  --operation=upsert-model-alias \
-  --alias-id=coding \
-  --strategy=auto \
-  --targets=rc/gpt-5.3-codex,zai/glm-5
-
-llm-router config \
-  --operation=set-provider-rate-limits \
-  --provider-id=rc \
-  --bucket-name="Minute cap" \
-  --bucket-models=gpt-5.3-codex \
-  --bucket-requests=60 \
-  --bucket-window=minute:1
-
-llm-router config \
-  --operation=set-provider-rate-limits \
-  --provider-id=zai \
-  --bucket-name="5-hours cap" \
-  --bucket-models=glm-5 \
-  --bucket-requests=600 \
-  --bucket-window=hour:5
-```
-
-## What Is A Bucket?
-
-A rate-limit bucket is a request cap for a time window.
+### Model
+The actual model ID from that provider.
 
+### Rate-Limit Bucket
+A request cap for a time window.
 Examples:
-- `40 req / 1 minute`
-- `600 req / 6 hours`
-
-Multiple buckets can apply to the same model scope at the same time. A candidate is treated as exhausted if any matching bucket is exhausted.
+- `40 requests / minute`
+- `20,000 requests / month`
 
-## TUI Bucket Walkthrough
+### Model Load Balancer
+Decides how traffic is distributed across models in an alias group.
 
-Use the config manager and select:
-- `Manage provider rate-limit buckets`
-- `Create bucket(s)`
-
-The TUI now guides you through:
-- Bucket name (friendly label)
-- Model scope (`all` or selected models with multiselect checkboxes)
-- Request cap
-- Window unit (`minute`, `hour(s)`, `week`, `month`)
-- Window size (hours support `N`, other preset units lock to `1`)
-- Review + optional add-another loop for combined policies
-
-Internal bucket ids are generated automatically from the name when omitted and shown as advanced detail in review.
-
-## Combined-Cap Recipe (`40/min` + `600/6h`)
-
-```bash
-llm-router config \
-  --operation=set-provider-rate-limits \
-  --provider-id=openrouter \
-  --bucket-name="Minute cap" \
-  --bucket-models=all \
-  --bucket-requests=40 \
-  --bucket-window=minute:1
-
-llm-router config \
-  --operation=set-provider-rate-limits \
-  --provider-id=openrouter \
-  --bucket-name="6-hours cap" \
-  --bucket-models=all \
-  --bucket-requests=600 \
-  --bucket-window=hour:6
-```
-
-This keeps both limits active together for the same model scope.
-
-## Rate-Limit Troubleshooting
-
-- Check routing decisions with `LLM_ROUTER_DEBUG_ROUTING=true` and inspect `x-llm-router-skipped-candidates`.
-- `quota-exhausted` means proactive pre-routing skip happened before an upstream call.
-- For provider `429`, cooldown is tracked from `Retry-After` when present, or from `LLM_ROUTER_ORIGIN_RATE_LIMIT_COOLDOWN_MS`.
-- Local mode persists state by default (file backend), while Worker defaults to in-memory state.
-
-## Main Commands
-
-```bash
-llm-router config
-llm-router start
-llm-router stop
-llm-router reload
-llm-router update
-llm-router deploy
-llm-router worker-key
-```
-
-## Non-Interactive Config (Agent/CI Friendly)
-
-```bash
-llm-router config \
-  --operation=upsert-provider \
-  --provider-id=openrouter \
-  --name="OpenRouter" \
-  --base-url=https://openrouter.ai/api/v1 \
-  --api-key=sk-or-v1-... \
-  --models=gpt-4o,claude-3-7-sonnet \
-  --format=openai \
-  --skip-probe=true
-
-llm-router config \
-  --operation=upsert-model-alias \
-  --alias-id=chat.default \
-  --strategy=auto \
-  --targets=openrouter/gpt-4o-mini@3,anthropic/claude-3-5-haiku@2 \
-  --fallback-targets=openrouter/gpt-4o
-
-llm-router config \
-  --operation=set-provider-rate-limits \
-  --provider-id=openrouter \
-  --bucket-name="Monthly cap" \
-  --bucket-models=all \
-  --bucket-requests=20000 \
-  --bucket-window=month:1
-```
-
-Alias target syntax:
-- `--targets` / `--fallback-targets`: `<routeRef>@<weight>` or `<routeRef>:<weight>`
-- route refs: direct `provider/model` or alias id
-
-Routing strategy values:
+Available strategies:
 - `auto` (recommended)
 - `ordered`
 - `round-robin`
 - `weighted-rr`
 - `quota-aware-weighted-rr`
 
-Rate-limit bucket window syntax:
-- `--bucket-window=month:1`
-- `--bucket-window=1w`
-- `--bucket-window=7day`
-
-Routing summary:
-
-```bash
-llm-router config --operation=list-routing
-```
+### Model Alias (Group models)
+A single model name that auto route/rotate across multiple models.
 
-Explicit schema migration with backup:
+Example:
+- alias: `opus`
+- targets:
+  - `openrouter/claude-opus-4.6`
+  - `anthropic/claude-opus-4.6`
 
-```bash
-llm-router config --operation=migrate-config --target-version=2 --create-backup=true
-```
+Your app can use `opus` model and `llm-router` chooses target models based on your routing settings.
 
-Automatic version handling:
-- Local config loads with silent forward-migration to latest supported schema.
-- Migration is persisted automatically on read when possible (best-effort, no interactive prompt).
-- Future/newer version numbers do not fail only because of version mismatch; known fields are normalized best-effort.
+## Setup using Terminal User Interface (TUI)
 
-Set local auth key:
+Open the TUI:
 
 ```bash
-llm-router config --operation=set-master-key --master-key=your_local_key
-# or generate a strong key automatically
-llm-router config --operation=set-master-key --generate-master-key=true
+llm-router
 ```
 
-Start with auth required:
+Then follow this order.
+
+### 1) Add Provider
+Flow:
+1. `Config manager`
+2. `Add/Edit provider`
+3. Enter provider name, endpoint, API key
+4. Enter model list
+5. Save
+
+### 2) Configure Model Fallback (Optional)
+Flow:
+1. `Config manager`
+2. `Set model silent-fallbacks`
+3. Pick main model
+4. Pick fallback models
+5. Save
+
+### 3) Configure Rate Limits (Optional)
+Flow:
+1. `Config manager`
+2. `Manage provider rate-limit buckets`
+3. `Create bucket(s)`
+4. Set name, model scope, request cap, time window
+5. Save
+
+### 4) Group Models With Alias (Recommended)
+Flow:
+1. `Config manager`
+2. `Add/Edit model alias`
+3. Set alias ID (example: `chat.default`)
+4. Select target models
+5. Save
+
+### 5) Configure Model Load Balancer
+Flow:
+1. `Config manager`
+2. `Add/Edit model alias`
+3. Open the alias you want to balance
+4. Choose strategy (`auto` recommended)
+5. Review alias targets
+6. Save
+
+### 6) Set Gateway Key
+Flow:
+1. `Config manager`
+2. `Set worker master key`
+3. Set or generate key
+4. Save
+
+## Start Local Server
 
 ```bash
-llm-router start --require-auth=true
+llm-router start
 ```
 
-## Cloudflare Worker Deploy
+Local endpoints:
+- Unified: `http://127.0.0.1:8787/route`
+- Anthropic-style: `http://127.0.0.1:8787/anthropic`
+- OpenAI-style: `http://127.0.0.1:8787/openai`
 
-Worker project name in `wrangler.toml`: `llm-router-route`.
+## Connect your coding tool
 
-### Option A: Guided deploy
+After setting master key, point your app/agent to local endpoint and use that key as auth token.
 
-```bash
-llm-router deploy
-```
+Claude Code example (`~/.claude/settings.local.json`):
 
-If `LLM_ROUTER_CONFIG_JSON` exceeds Cloudflare Free-tier secret size (`5 KB`), deploy now warns and requires explicit confirmation (default is `No`). In non-interactive environments, pass `--allow-large-config=true` to proceed intentionally.
+```json
+{
+  "env": {
+    "ANTHROPIC_BASE_URL": "http://127.0.0.1:8787",
+    "ANTHROPIC_AUTH_TOKEN": "gw_your_gateway_key",
+    "ANTHROPIC_DEFAULT_OPUS_MODEL": "provider_name/model_name_1",
+    "ANTHROPIC_DEFAULT_SONNET_MODEL": "provider_name/model_name_2",
+    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "provider_name/model_name_3"
+  }
+}
+```
 
-`deploy` requires `CLOUDFLARE_API_TOKEN` for Cloudflare API access. Create a **User Profile API token** at <https://dash.cloudflare.com/profile/api-tokens> (do not use Account API Tokens), then choose preset/template `Edit Cloudflare Workers`. If the env var is missing in interactive mode, the CLI will show the guide and prompt for token input securely.
+## Real-Time Update Experience
 
-For multi-account tokens, set account explicitly in non-interactive runs:
-- `CLOUDFLARE_ACCOUNT_ID=<id>` or
-- `llm-router deploy --account-id=<id>`
+When local server is running:
+- open `llm-router`
+- change provider/model/load-balancer/rate-limit/alias in TUI
+- save
+- the running proxy updates instantly
 
-`llm-router deploy` resolves deploy target from CLI/TUI input (workers.dev or custom route), generates a temporary Wrangler config at runtime, deploys with `--config`, then removes that temporary file. Personal route/account details are not persisted back into repo `wrangler.toml`.
+No stop/start cycle needed.
 
-For custom domains, the deploy helper now prints a DNS checklist and connectivity commands. Common setup for `llm.example.com`:
-- Create a DNS record in Cloudflare for `llm` (usually `CNAME llm -> @`)
-- Set **Proxy status = Proxied** (orange cloud)
-- Use route target `--route-pattern=llm.example.com/* --zone-name=example.com`
-- Claude Code base URL should be `https://llm.example.com/anthropic` (**no `:8787`**; that port is local-only)
+## Cloudflare Worker (Hosted)
 
-```bash
-llm-router deploy --export-only=true --out=.llm-router.worker.json
-wrangler secret put LLM_ROUTER_CONFIG_JSON < .llm-router.worker.json
-wrangler deploy
-```
+Use when you want a hosted endpoint instead of local server.
 
-Rotate worker auth key quickly:
+Guided deploy:
 
 ```bash
-llm-router worker-key --master-key=new_key
-# or generate and rotate immediately
-llm-router worker-key --env=production --generate-master-key=true
+llm-router deploy
 ```
 
-If you intentionally need to bypass weak-key checks (not recommended), add `--allow-weak-master-key=true` to `deploy` or `worker-key`.
-
-Cloudflare hardening and incident-response checklist: see [`SECURITY.md`](./SECURITY.md).
-
-## Runtime Secrets / Env
-
-Primary:
-- `LLM_ROUTER_CONFIG_JSON`
-- `LLM_ROUTER_MASTER_KEY` (optional override)
-
-Also supported:
-- `ROUTE_CONFIG_JSON`
-- `LLM_ROUTER_JSON`
+You will be guided in TUI to select account and deploy target.
 
-Optional resilience tuning:
-- `LLM_ROUTER_ORIGIN_RETRY_ATTEMPTS` (default `3`)
-- `LLM_ROUTER_ORIGIN_RETRY_BASE_DELAY_MS` (default `250`)
-- `LLM_ROUTER_ORIGIN_RETRY_MAX_DELAY_MS` (default `3000`)
-- `LLM_ROUTER_ORIGIN_FALLBACK_COOLDOWN_MS` (default `45000`)
-- `LLM_ROUTER_ORIGIN_RATE_LIMIT_COOLDOWN_MS` (default `30000`)
-- `LLM_ROUTER_ORIGIN_BILLING_COOLDOWN_MS` (default `900000`)
-- `LLM_ROUTER_ORIGIN_AUTH_COOLDOWN_MS` (default `600000`)
-- `LLM_ROUTER_ORIGIN_POLICY_COOLDOWN_MS` (default `120000`)
-- `LLM_ROUTER_ALLOW_POLICY_FALLBACK` (default `false`)
-- `LLM_ROUTER_FALLBACK_CIRCUIT_FAILURES` (default `2`)
-- `LLM_ROUTER_FALLBACK_CIRCUIT_COOLDOWN_MS` (default `30000`)
-- `LLM_ROUTER_MAX_REQUEST_BODY_BYTES` (default `1048576`, min `4096`, max `20971520`)
-- `LLM_ROUTER_UPSTREAM_TIMEOUT_MS` (default `60000`, min `1000`, max `300000`)
+Worker safety defaults:
+- `LLM_ROUTER_STATE_BACKEND=file` is ignored on Worker (auto-fallback to in-memory state).
+- Stateful timing-dependent routing features (cursor balancing, local quota counters, cooldown persistence) are auto-disabled by default to keep route flow safe across Worker isolates.
+- To opt in to best-effort stateful behavior on Worker, set `LLM_ROUTER_WORKER_ALLOW_BEST_EFFORT_STATEFUL_ROUTING=true`.
 
-Optional browser access (CORS):
-- By default, cross-origin browser reads are denied unless explicitly allow-listed.
-- `LLM_ROUTER_CORS_ALLOWED_ORIGINS` (comma-separated exact origins, e.g. `https://app.example.com`)
-- `LLM_ROUTER_CORS_ALLOW_ALL=true` (allows any origin; not recommended for production)
+## Config File Location
 
-Optional source IP allowlist (recommended for Worker deployments):
-- `LLM_ROUTER_ALLOWED_IPS` (comma-separated client IPs; denies requests from all other IPs)
-- `LLM_ROUTER_IP_ALLOWLIST` (alias of `LLM_ROUTER_ALLOWED_IPS`)
-
-## Default Config Path
+Local config file:
 
 `~/.llm-router.json`
 
-Minimal shape:
-
-```json
-{
-  "version": 2,
-  "masterKey": "local_or_worker_key",
-  "defaultModel": "chat.default",
-  "modelAliases": {
-    "chat.default": {
-      "strategy": "auto",
-      "targets": [
-        { "ref": "openrouter/gpt-4o" },
-        { "ref": "anthropic/claude-3-5-haiku" }
-      ],
-      "fallbackTargets": [
-        { "ref": "openrouter/gpt-4o-mini" }
-      ]
-    }
-  },
-  "providers": [
-    {
-      "id": "openrouter",
-      "name": "OpenRouter",
-      "baseUrl": "https://openrouter.ai/api/v1",
-      "apiKey": "sk-or-v1-...",
-      "formats": ["openai"],
-      "models": [{ "id": "gpt-4o" }],
-      "rateLimits": [
-        {
-          "id": "openrouter-all-month",
-          "name": "Monthly cap",
-          "models": ["all"],
-          "requests": 20000,
-          "window": { "unit": "month", "size": 1 }
-        }
-      ]
-    }
-  ]
-}
-```
+## Security
 
-Direct vs model alias routing:
-- Direct route: request `model=provider/model` and optional model-level `fallbackModels` applies.
-- Model alias route: request `model=alias.id` (or set as `defaultModel`) and the model alias `targets` + `strategy` drive balancing. `auto` is the recommended default for new model aliases.
+See [`SECURITY.md`](https://github.com/khanglvm/llm-router/blob/master/SECURITY.md).
 
-State durability caveats:
-- Local Node (`llm-router start`): routing state defaults to file-backed local persistence, so cooldowns/caps survive restarts.
-- Cloudflare Worker: default state is in-memory per isolate for now; long-window counters are best-effort until a durable Worker backend is configured.
-
-## Smoke Test
-
-```bash
-npm run test:provider-smoke
-```
+## Versioning
 
-Use `.env.test-suite.example` as template for provider-based smoke tests.
+- Semver: [Semantic Versioning](https://semver.org/)
+- Release notes: [`CHANGELOG.md`](https://github.com/khanglvm/llm-router/blob/master/CHANGELOG.md)

package/package.json

@@ -1,7 +1,19 @@
 {
   "name": "@khanglvm/llm-router",
-  "version": "1.0.6",
+  "version": "1.0.9",
   "description": "Single gateway endpoint for multi-provider LLMs with unified OpenAI+Anthropic format and seamless fallback",
+  "keywords": [
+    "llm-router",
+    "llm-gateway",
+    "ai-proxy",
+    "openai-compatible",
+    "anthropic-compatible",
+    "model-routing",
+    "fallback",
+    "load-balancing",
+    "cloudflare-workers",
+    "agent-infra"
+  ],
   "type": "module",
   "main": "src/index.js",
   "bin": {