open-classify 0.4.0 → 0.6.0

package/README.md

@@ -6,12 +6,10 @@
   Decide what should happen to a user message <em>before</em> it reaches your downstream model.
 </p>
 
-Open Classify is a pre-routing layer for AI products. It runs a small set of fast classifiers in parallel against the latest user message, then tells your app one of three things: **route** it, **reply** immediately, or **block** it.
+Open Classify is a pre-routing layer for AI products. It runs a small set of fast classifiers in parallel against the latest user message, then returns a single `PipelineResult` your app can act on: an action (`route`, `block`, or `reply`), a downstream model recommendation, a tool exposure list, an optional immediate reply, and any custom signals your own classifiers contribute.
 
 Use it when your frontier model should not be the first thing every request touches. Open Classify can handle tiny terminal replies before they hit an expensive model, recommend the right downstream model for the actual task, suggest what tools or context the downstream model should receive, and add a focused prompt-injection pass.
 
-The result is a small, auditable decision envelope your app can act on before spending the big tokens.
-
 ```
 message
   │
@@ -19,29 +17,29 @@ message
 normalize + trim classifier context
   │
   ├─► preflight ─────────────► final_reply? / ack_reply?
-  ├─► routing ───────────────► model_tier?
-  ├─► model_specialization ──► specialization?
+  ├─► model_tier ────────────► model_tier?
+  ├─► model_specialization ──► model_specialization?
   ├─► tools ─────────────────► tools?
   ├─► prompt_injection ─────► risk_level?
-  └─► custom classifiers ────► JSON-Schema output
-        (run in parallel)
+  └─► your own classifiers ──► any JSON-Schema-validated payload
+        (all run in parallel, capped by maxConcurrency)
   │
   ▼
 aggregator + model catalog
   │
   ▼
-route / reply / block
+PipelineResult { action, model_id, tools, reply, ... }
 ```
 
-Stock classifiers have fixed typed signals. Custom classifiers carry their own JSON-Schema-validated payload. The aggregator merges everything, resolves a concrete model from your catalog, and short-circuits when preflight has a terminal reply or prompt injection is detected.
+Every classifier uses the same manifest shape and emits the same output envelope: `{ reason, certainty, ...payload }`. Some payload fields are **reserved** — like `model_tier`, `final_reply`, and `risk_level` — and the aggregator knows how to consume them into a routing decision. Everything else is your classifier's own data and passes through to the caller untouched.
 
 ## Why Open Classify
 
-- **Spend frontier tokens only when they matter.** Simple greetings, thanks, spelling checks, and small arithmetic can return `action: "reply"` with `reply.text` and skip downstream work entirely.
-- **Keep the user interface responsive.** For complex work, preflight can return an `ack_reply` while your app routes the request to the real worker.
+- **Spend frontier tokens only when they matter.** Simple greetings, thanks, spelling checks, and small arithmetic can be answered immediately (`action: "reply"`) without sending the request downstream.
+- **Keep the user interface responsive.** For complex work, preflight emits an `ack_reply` — a task-specific acknowledgement your UI can show while routing the real request.
 - **Pick the right model per message.** Classifiers emit soft constraints like tier and specialization; your catalog turns those into a concrete model optimized for cost, capability, and fit.
 - **Shape downstream context intentionally.** Built-in and custom classifiers can recommend tools, retrieval queries, summaries, or other context hints without passing the full conversation history back to the caller.
-- **Add another defensive layer.** The `prompt_injection` classifier can block instruction override attempts like “forget previous instructions” without treating ordinary tool requests as injection.
+- **Add another defensive layer.** The `prompt_injection` classifier surfaces instruction-override attempts. High-risk or unknown injection risk automatically sets `action: "block"`.
 
 ## Install
 
@@ -56,7 +54,7 @@ Node 18+. The packaged runner is local Ollama and ships with `gemma4:e4b-it-q4_K
 ```ts
 import { createClassifier } from "open-classify";
 
-const classify = createClassifier();
+const { classify, inspect } = createClassifier();
 
 const result = await classify({
   messages: [
@@ -64,120 +62,172 @@ const result = await classify({
   ],
 });
 
-if (result.action === "route") {
-  // result.downstream.model_id is a concrete model from your catalog.
-  // result.downstream.tools is the recommended tool exposure.
-  // result.classifier_outputs holds any custom classifier payloads.
+if (result.action === "block") {
+  // classification error or prompt injection — handle appropriately
+  console.error(result.block_reason, result.failed_classifiers);
+} else if (result.action === "reply") {
+  // preflight can answer this immediately — skip the downstream model
+  respondToUser(result.reply.text);
+} else {
+  // route to the downstream model
+  callDownstream(result.model_id, result.tools);
+  respondToUser(result.reply?.text); // show the ack while it works
 }
+
+const queries = result.classifier_outputs.memory_retrieval_queries?.queries;
 ```
 
-`createClassifier` builds the runner and loads the model catalog once. Reuse the returned `classify` function across your app — every call is a plain function invocation, no re-initialization.
+`createClassifier` builds the runner and loads the model catalog once. Reuse the returned `classify` and `inspect` functions across your app — every call is a plain function invocation, no re-initialization.
 
-## What you get back
+### Classifying assistant output
+
+`inspect()` is a lean second pass for the **assistant's reply**. It only runs classifiers tagged `applies_to: "both"` (or `"assistant"`) in their manifest, and returns the per-classifier outputs plus the message that was inspected — no routing, no action, no block logic.
+
+```ts
+const result = await inspect({
+  messages: [
+    { role: "user", text: "Summarize the contract." },
+    { role: "assistant", text: "The contract has three notable risks…" },
+  ],
+});
 
-Every call returns a `PipelineResult` with one of three `action` values:
+// result.message is { role: "assistant", text: "..." }
+const risk = result.classifier_outputs.prompt_injection?.risk_level;
+```
+
+Use it for things like prompt-injection checks on model output, summarized slugs, or any classifier you want to apply post-hoc. The built-in `prompt_injection` classifier ships tagged `"both"`, so it runs in both passes; everything else is `"user"` by default.
 
-| `action` | When | Key fields |
-|---|---|---|
-| `route` | Default — downstream work should continue | `downstream.{model_id, target_message, tools}`, `audit.ack_reply?` |
-| `reply` | Preflight had a tiny terminal reply | `reply.text` |
-| `block` | Prompt injection flagged confident `high_risk` / `unknown`, or the certainty gate fired | `reason.kind` plus prompt-injection or low-certainty details |
+## What you get back
 
-All three also carry `target_message_hash` (the stable 8-hex fingerprint of the target message), `classifier_outputs` (custom classifier payloads, keyed by name), and an `audit` block. Route results include the downstream target message, not the caller's message history. Short-circuit results include the firing classifier's audit context.
+Every `classify()` call returns a `PipelineResult`:
 
-For complex requests, look for `audit.ack_reply` on `route` results. It is the immediate acknowledgement your UI can show while the downstream model works. For trivial requests, `result.reply.text` is the complete response and no downstream model is needed.
+| Field | What it is |
+|---|---|
+| `action` | `"route"` \| `"block"` \| `"reply"` |
+| `block_reason` | `"prompt_injection"` \| `"classification_error"` (only when `action === "block"`) |
+| `target_message_hash` | Stable 8-hex fingerprint of the target message |
+| `model_id` | Concrete model id chosen from your catalog (or `null` if unresolvable) |
+| `tools` | Recommended tool ids (always an array; empty if not emitted) |
+| `reply` | `{ text }` — the `ack_reply` or `final_reply` text, if any |
+| `prompt_injection` | `{ risk_level }` from the injection classifier, or `null` |
+| `avg_certainty` | Arithmetic mean certainty score (float 0–1) across all classifiers |
+| `min_certainty` | Minimum certainty score (float 0–1) across all classifiers |
+| `failed_classifiers` | Names of classifiers that errored or timed out (always present; may be empty) |
+| `classifier_outputs` | Each classifier's payload with `reason` (string) and `certainty` (float) |
 
-Example `route` result:
+Example result:
 
 ```json
 {
   "action": "route",
   "target_message_hash": "b11d5268",
-  "downstream": {
-    "model_id": "gpt-5.5",
-    "tools": { "tools": ["workspace"] },
-    "target_message": { "role": "user", "text": "...", "hash": "b11d5268" }
-  },
+  "model_id": "gpt-5.5",
+  "tools": ["workspace"],
+  "reply": { "text": "On it — I'll review the contract now." },
+  "prompt_injection": { "risk_level": "normal" },
+  "avg_certainty": 0.84,
+  "min_certainty": 0.75,
+  "failed_classifiers": [],
   "classifier_outputs": {
-    "memory_retrieval_queries": { "queries": ["user code review preferences"] }
-  },
-  "audit": {
-    "ack_reply": { "text": "Let me check." },
-    "routing": { "model_tier": "frontier_strong" },
-    "model_specialization": { "specialization": "coding" },
-    "tools": { "tools": ["workspace"] },
-    "model_recommendation": {
-      "id": "gpt-5.5",
-      "context_window": 1050000,
-      "input_tokens_cpm": 5,
-      "cached_tokens_cpm": 0.5,
-      "output_tokens_cpm": 30,
-      "resolution": { "...": "..." }
-    },
-    "meta": { "classifiers": { "...": "..." } }
+    "model_tier": { "model_tier": "frontier_strong", "reason": "...", "certainty": 0.88 },
+    "model_specialization": { "model_specialization": "coding", "reason": "...", "certainty": 0.75 },
+    "tools": { "tools": ["workspace"], "reason": "...", "certainty": 0.88 },
+    "prompt_injection": { "risk_level": "normal", "reason": "...", "certainty": 0.97 },
+    "memory_retrieval_queries": { "queries": ["user code review preferences"], "reason": "...", "certainty": 0.75 }
   }
 }
 ```
 
-## Stock classifiers
+## Classifier model
 
-Stock classifiers are built in and have fixed, typed output shapes. Each one owns exactly one signal. Its manifest lives in `src/classifiers/stock/<name>/manifest.json`; the shared stock prompt building blocks live in `src/classifiers/stock/prompts/`.
+Open Classify ships with eight built-in classifiers; all use the same manifest shape. There is no distinction between "stock" and "custom" — the runtime only cares about which **reserved fields** a classifier declares.
 
-Every classifier prompt includes a shared header with its `Classifier` name, `Purpose`, and an instruction to treat that purpose as a hard scope boundary. In practice:
+| Name | dispatch_order | Reserved fields | What the aggregator does with it |
+|---|---|---|---|
+| `preflight` | 10 | `final_reply`, `ack_reply` | Sets `action: "reply"` or populates `result.reply` |
+| `model_tier` | 20 | `model_tier` | Feeds the catalog resolver as a soft constraint |
+| `model_specialization` | 30 | `model_specialization` | Feeds the catalog resolver as a soft constraint |
+| `tools` | 40 | `tools` | Sets `result.tools` |
+| `prompt_injection` | 50 | `risk_level` | High-risk/unknown → `action: "block"`; suspicious → advisory |
+| `memory_retrieval_queries` | 60 | — | Passes through to `classifier_outputs` |
+| `conversation_digest` | 70 | — | Passes through |
+| `context_shift` | 80 | — | Passes through |
 
-- `routing` chooses only `model_tier`
-- `model_specialization` chooses only `specialization`
-- `prompt_injection` is only for prompt injection, not harmfulness, authorization, contradiction, feasibility, or freshness checks
+Reserved fields are well-known output keys with canonical JSON Schemas and prompt fragments baked into the runtime. When you declare one in your manifest, you don't have to redeclare its enum values or shape — the runtime injects them.
 
-| Name | Signal | Short-circuits? |
-|---|---|---|
-| `preflight` | `final_reply?` / `ack_reply?` | `final_reply` → `reply` |
-| `routing` | `model_tier?` | no |
-| `model_specialization` | `specialization?` | no |
-| `tools` | `{ tools[] }` | no |
-| `prompt_injection` | `{ risk_level }` | confident `high_risk` or `unknown` → `block` |
+## Adding a classifier
 
-Each output must carry `reason` (≤120 chars) and `certainty` (`no_signal` through `near_certain`). The aggregator maps certainty tags to numeric scores and drops below-threshold signals; the default threshold is `0.65`.
+Every classifier is two files in `src/classifiers/<name>/`:
 
-## Custom classifiers
-
-A custom classifier is two files in `src/classifiers/custom/<name>/`:
-
-`manifest.json`:
+`manifest.json` describes the output shape and a fallback for when the classifier errors:
 
 ```json
 {
-  "kind": "custom",
-  "name": "memory_retrieval_queries",
+  "name": "topic_tags",
   "version": "1.0.0",
-  "purpose": "Generate retrieval queries likely to surface helpful user-specific context for the downstream model.",
-  "order": 60,
-  "fallback": {
-    "reason": "Classifier failed; no memory queries generated.",
-    "certainty": "no_signal",
-    "output": { "queries": [] }
-  },
+  "purpose": "Tag the message with a small set of topic labels for analytics.",
+  "dispatch_order": 70,
   "output_schema": {
-    "type": "object",
-    "additionalProperties": false,
-    "required": ["queries"],
+    "required": ["tags"],
     "properties": {
-      "queries": {
+      "tags": {
         "type": "array", "maxItems": 5,
-        "items": { "type": "string", "minLength": 1, "maxLength": 120 }
+        "items": { "type": "string", "minLength": 1, "maxLength": 40 }
       }
     }
+  },
+  "fallback": {
+    "reason": "Classifier failed; no tags generated.",
+    "certainty": "no_signal",
+    "tags": []
   }
 }
 ```
 
-`prompt.md`: your classifier-specific instructions.
+`prompt.md` describes the classification rule in plain language. You don't need to write JSON examples — the runtime synthesizes one from your schema and shows it to the model — and you don't paste enum values for reserved fields. Just describe what each output key means and when to abstain:
+
+```markdown
+You are the topic_tags classifier.
+
+`tags` are short single-word topic labels (lowercase, no spaces). Use at most five.
+Return an empty array when no clear topic applies.
+Do not invent tags for vague or ambiguous messages.
+```
+
+Rules:
+
+- `name` must match the directory name.
+- Reserved field names cannot appear in `output_schema.properties` — declare them in `reserved_fields` instead.
+- `fallback` requires only `reason` and `certainty`; reserved and custom required fields are exempt from the fallback check.
+- If you want hand-picked examples (preflight does this), add an `output_schema.examples` array. Each entry must validate against the composed schema at load time. Otherwise the runtime synthesizes a skeleton example for you.
+
+Consume your output:
+
+```ts
+const result = await classify(input);
+const tags = result.classifier_outputs.topic_tags?.tags ?? [];
+```
 
-Custom classifiers receive the same shared `Classifier` + `Purpose` header and the same scope-boundary instruction, so keep the manifest `purpose` specific and operational.
+See [docs/adding-a-classifier.md](docs/adding-a-classifier.md) for a full walkthrough and [docs/manifests.md](docs/manifests.md) for the field reference.
 
-The runtime auto-discovers it, validates outputs against your schema, and surfaces them on `result.classifier_outputs.<name>`. No TypeScript edits required.
+## Using reserved fields in your own classifier
 
-See [docs/adding-a-classifier.md](docs/adding-a-classifier.md) for a full walkthrough.
+Any classifier can emit reserved fields. If you write your own `task_router` that emits `model_tier`, the aggregator will fold it into the model resolution alongside the built-in `model_tier` classifier — highest-certainty contributor wins, ties broken by manifest `dispatch_order` ascending.
+
+```json
+{
+  "name": "task_router",
+  "version": "1.0.0",
+  "purpose": "Pick the downstream model tier and specialization for code-heavy tasks.",
+  "dispatch_order": 25,
+  "reserved_fields": ["model_tier", "model_specialization"],
+  "fallback": { "reason": "Classifier failed.", "certainty": "no_signal" }
+}
+```
+
+The runtime injects canonical sub-schemas and prompt fragments for each declared reserved field — the model is told the exact enum values it may emit. You don't paste enum values into `prompt.md`, and you don't have to hand-write a JSON example; the runtime synthesizes one from the schema and shows it to the model.
+
+The available reserved fields are: `final_reply`, `ack_reply`, `model_tier`, `model_specialization`, `tools`, `risk_level`. The `tools` field additionally requires an `allowed_tools` array on the manifest listing the tool ids the classifier may pick from.
 
 ## Model catalog
 
@@ -212,9 +262,7 @@ Classifiers never emit model ids. They emit constraints; your catalog maps const
 }
 ```
 
-OpenAI's GPT-5.5 model page lists text and image input, text output, a 1,050,000-token context window, 128,000 max output tokens, and text-token pricing of $5.00 input, $0.50 cached input, and $30.00 output per 1M tokens. OpenAI does not publish parameter counts, so use `null` for `params_in_billions`. See the [GPT-5.5 model details](https://developers.openai.com/api/docs/models/gpt-5.5) for current pricing and capability details.
-
-The resolver picks the cheapest model matching `specialization` and `tier`, relaxing constraints in order when nothing fits, and reports what it dropped on `audit.model_recommendation.resolution`. See [docs/resolver.md](docs/resolver.md) for ranking details.
+The resolver picks the cheapest model matching `model_specialization` and `model_tier`, relaxing constraints in order when nothing fits. See [docs/resolver.md](docs/resolver.md) for ranking details.
 
 ## Input contract
 
@@ -244,24 +292,16 @@ cp open-classify.config.example.json open-classify.config.json
     "provider": "ollama",
     "defaultModel": "gemma4:e4b-it-q4_K_M",
     "models": {
-      "stock": {
-        "routing": "qwen2.5:7b-instruct-q4_K_M",
-        "prompt_injection": "llama-guard3:8b"
-      },
-      "custom": {
-        "memory_retrieval_queries": "qwen2.5:7b-instruct-q4_K_M"
-      }
+      "model_tier": "qwen2.5:7b-instruct-q4_K_M",
+      "prompt_injection": "llama-guard3:8b",
+      "memory_retrieval_queries": "qwen2.5:7b-instruct-q4_K_M"
     }
   },
-  "aggregator": {
-    "certaintyThreshold": 0.65,
-    "certaintyGate": "min_score"
-  },
   "catalog": "downstream-models.json"
 }
 ```
 
-`runner.provider` currently supports `"ollama"` only. `runner.defaultModel` applies to any classifier without an explicit entry. `runner.models.stock` configures built-in classifiers; `runner.models.custom` configures custom classifiers by manifest name. `aggregator.certaintyGate` can be `"min_score"` (lowest score across all stock and custom classifiers), `"avg_score"`, or `"off"`. The setup script and `loadOpenClassifyConfig()` read `open-classify.config.json`, or `OPEN_CLASSIFY_CONFIG` when you want a different path.
+`runner.provider` currently supports `"ollama"` only. `runner.defaultModel` applies to any classifier without an explicit `runner.models` entry. `runner.models` is a flat map keyed by classifier name.
 
 ## Bring your own backend
 
@@ -284,15 +324,15 @@ const runClassifier: RunClassifier = async (name, input, signal) => {
   // call your provider of choice, return a ClassifierOutput
 };
 
-const classify = createClassifier({ runClassifier });
+const { classify, inspect } = createClassifier({ runClassifier });
 ```
 
-For the lowest-level entry point, `classifyOpenClassifyInput(input, { runClassifier, catalog })` skips the factory entirely.
+For the lowest-level entry points, `classifyOpenClassifyInput(input, { runClassifier, catalog })` and `inspectOpenClassifyInput(input, { runClassifier })` skip the factory entirely.
 
 ## Further reading
 
-- [docs/signals.md](docs/signals.md) — full signal contracts and validation rules
-- [docs/manifests.md](docs/manifests.md) — manifest reference (stock and custom)
+- [docs/signals.md](docs/signals.md) — reserved field reference
+- [docs/manifests.md](docs/manifests.md) — manifest reference
 - [docs/resolver.md](docs/resolver.md) — aggregation and model resolution
 - [docs/adding-a-classifier.md](docs/adding-a-classifier.md) — author guide
 

package/dist/src/aggregator.d.ts

@@ -1,21 +1,12 @@
-import type { AggregatorConfig, Catalog, ClassifierRegistry, ClassifierResults, Envelope, ModelRecommendation, ModelRecommendationResolution } from "./manifest.js";
-import type { AckReplySignal, ModelSpecializationClassifierOutput, FinalReplySignal, RoutingClassifierOutput, RoutingSignal } from "./stock.js";
-import type { ClassifierInput } from "./types.js";
-export declare const DEFAULT_CERTAINTY_THRESHOLD = 0.65;
-/** @deprecated Use DEFAULT_CERTAINTY_THRESHOLD. */
-export declare const DEFAULT_CONFIDENCE_THRESHOLD = 0.65;
-export interface ComposeEnvelopeArgs {
+import type { Catalog, ClassifierPublicOutputs, ClassifierRegistry, ClassifierResults, PipelineResult } from "./manifest.js";
+import type { AckReplySignal, FinalReplySignal, ToolsSignal } from "./stock.js";
+export interface AssembleResultArgs {
     readonly registry: ClassifierRegistry;
     readonly results: ClassifierResults;
+    readonly failedClassifiers: ReadonlyArray<string>;
     readonly catalog: Catalog;
-    readonly input: ClassifierInput;
-    readonly config?: AggregatorConfig;
 }
-export declare function composeEnvelope(args: ComposeEnvelopeArgs): Envelope;
-export declare function certaintyThreshold(config: AggregatorConfig | undefined): number;
-export declare function resolveModelFromRouting(routing: RoutingSignal | undefined, catalog: Catalog, confidence: number | undefined, ignoredConstraints?: ModelRecommendationResolution["constraints_dropped"]): ModelRecommendation;
-export declare function resolveModel(results: Readonly<{
-    routing?: RoutingClassifierOutput;
-    model_specialization?: ModelSpecializationClassifierOutput;
-}>, catalog: Catalog, threshold: number): ModelRecommendation;
-export type { FinalReplySignal, AckReplySignal };
+type AssembledResult = Omit<PipelineResult, "target_message_hash">;
+export declare function assembleResult(args: AssembleResultArgs): AssembledResult;
+export declare function buildPublicOutputs(registry: ClassifierRegistry, results: ClassifierResults): ClassifierPublicOutputs;
+export type { FinalReplySignal, AckReplySignal, ToolsSignal };