open-classify 0.4.0 → 0.5.0

package/README.md

@@ -6,12 +6,10 @@
   Decide what should happen to a user message <em>before</em> it reaches your downstream model.
 </p>
 
-Open Classify is a pre-routing layer for AI products. It runs a small set of fast classifiers in parallel against the latest user message, then tells your app one of three things: **route** it, **reply** immediately, or **block** it.
+Open Classify is a pre-routing layer for AI products. It runs a small set of fast classifiers in parallel against the latest user message, then returns a single decision envelope your app can act on: a downstream model recommendation, a tool exposure list, an optional acknowledgement, and any custom signals your own classifiers contribute.
 
 Use it when your frontier model should not be the first thing every request touches. Open Classify can handle tiny terminal replies before they hit an expensive model, recommend the right downstream model for the actual task, suggest what tools or context the downstream model should receive, and add a focused prompt-injection pass.
 
-The result is a small, auditable decision envelope your app can act on before spending the big tokens.
-
 ```
 message
   │
@@ -20,28 +18,28 @@ normalize + trim classifier context
   │
   ├─► preflight ─────────────► final_reply? / ack_reply?
   ├─► routing ───────────────► model_tier?
-  ├─► model_specialization ──► specialization?
+  ├─► model_specialization ──► model_specialization?
   ├─► tools ─────────────────► tools?
   ├─► prompt_injection ─────► risk_level?
-  └─► custom classifiers ────► JSON-Schema output
-        (run in parallel)
+  └─► your own classifiers ──► any JSON-Schema-validated payload
+        (all run in parallel, capped by maxConcurrency)
   │
   ▼
 aggregator + model catalog
   │
   ▼
-route / reply / block
+route
 ```
 
-Stock classifiers have fixed typed signals. Custom classifiers carry their own JSON-Schema-validated payload. The aggregator merges everything, resolves a concrete model from your catalog, and short-circuits when preflight has a terminal reply or prompt injection is detected.
+Every classifier uses the same manifest shape and emits the same output envelope: `{ reason, certainty, ...payload }`. Some payload fields are **reserved** — like `model_tier`, `final_reply`, and `risk_level` — and the aggregator knows how to consume them into a routing decision. Everything else is your classifier's own data and passes through to the caller untouched.
 
 ## Why Open Classify
 
-- **Spend frontier tokens only when they matter.** Simple greetings, thanks, spelling checks, and small arithmetic can return `action: "reply"` with `reply.text` and skip downstream work entirely.
-- **Keep the user interface responsive.** For complex work, preflight can return an `ack_reply` while your app routes the request to the real worker.
+- **Spend frontier tokens only when they matter.** Simple greetings, thanks, spelling checks, and small arithmetic can be answered immediately via `audit.final_reply` without sending the request downstream.
+- **Keep the user interface responsive.** For complex work, preflight can suggest an `ack_reply` while your app routes the request to the real worker.
 - **Pick the right model per message.** Classifiers emit soft constraints like tier and specialization; your catalog turns those into a concrete model optimized for cost, capability, and fit.
 - **Shape downstream context intentionally.** Built-in and custom classifiers can recommend tools, retrieval queries, summaries, or other context hints without passing the full conversation history back to the caller.
-- **Add another defensive layer.** The `prompt_injection` classifier can block instruction override attempts like “forget previous instructions” without treating ordinary tool requests as injection.
+- **Add another defensive layer.** The `prompt_injection` classifier surfaces instruction-override attempts so your app can decide whether to continue.
 
 ## Install
 
@@ -56,7 +54,7 @@ Node 18+. The packaged runner is local Ollama and ships with `gemma4:e4b-it-q4_K
 ```ts
 import { createClassifier } from "open-classify";
 
-const classify = createClassifier();
+const { classify, inspect } = createClassifier();
 
 const result = await classify({
   messages: [
@@ -64,30 +62,49 @@ const result = await classify({
   ],
 });
 
-if (result.action === "route") {
-  // result.downstream.model_id is a concrete model from your catalog.
-  // result.downstream.tools is the recommended tool exposure.
-  // result.classifier_outputs holds any custom classifier payloads.
-}
+// result.action is always "route". Use the audit envelope and the per-classifier
+// outputs to decide what to do next.
+const { model_id, target_message, tools } = result.downstream;
+const ackReply = result.audit.ack_reply?.text;
+const queries = result.classifier_outputs.memory_retrieval_queries?.queries;
 ```
 
-`createClassifier` builds the runner and loads the model catalog once. Reuse the returned `classify` function across your app — every call is a plain function invocation, no re-initialization.
+`createClassifier` builds the runner and loads the model catalog once. Reuse the returned `classify` and `inspect` functions across your app — every call is a plain function invocation, no re-initialization.
 
-## What you get back
+### Classifying assistant output
 
-Every call returns a `PipelineResult` with one of three `action` values:
+`inspect()` is a lean second pass for the **assistant's reply**. It only runs classifiers tagged `applies_to: "both"` (or `"assistant"`) in their manifest, and returns just the per-classifier outputs — no routing, no model resolution, no audit envelope.
 
-| `action` | When | Key fields |
-|---|---|---|
-| `route` | Default — downstream work should continue | `downstream.{model_id, target_message, tools}`, `audit.ack_reply?` |
-| `reply` | Preflight had a tiny terminal reply | `reply.text` |
-| `block` | Prompt injection flagged confident `high_risk` / `unknown`, or the certainty gate fired | `reason.kind` plus prompt-injection or low-certainty details |
+```ts
+const reply = await inspect({
+  messages: [
+    { role: "user", text: "Summarize the contract." },
+    { role: "assistant", text: "The contract has three notable risks…" },
+  ],
+});
+
+const risk = reply.classifier_outputs.prompt_injection?.risk_level;
+```
+
+Use it for things like prompt-injection checks on model output, summarized slugs, or any classifier you want to apply post-hoc. The built-in `prompt_injection` classifier ships tagged `"both"`, so it runs in both passes; everything else is `"user"` by default. Tag your own classifiers with `applies_to` in their manifest to opt into either side.
 
-All three also carry `target_message_hash` (the stable 8-hex fingerprint of the target message), `classifier_outputs` (custom classifier payloads, keyed by name), and an `audit` block. Route results include the downstream target message, not the caller's message history. Short-circuit results include the firing classifier's audit context.
+## What you get back
 
-For complex requests, look for `audit.ack_reply` on `route` results. It is the immediate acknowledgement your UI can show while the downstream model works. For trivial requests, `result.reply.text` is the complete response and no downstream model is needed.
+Every call returns a `PipelineResult`:
 
-Example `route` result:
+| Field | What it is |
+|---|---|
+| `action` | Always `"route"` — the worker pool always runs every classifier and returns aggregated results |
+| `target_message_hash` | Stable 8-hex fingerprint of the target message |
+| `downstream.model_id` | Concrete model id chosen from your catalog |
+| `downstream.target_message` | The sanitized target message that should be sent downstream |
+| `downstream.tools` | The recommended tool exposure (may be `{ tools: [] }`) |
+| `classifier_outputs[name]` | Each classifier's payload (reserved + custom fields) with `reason` and `certainty` stripped |
+| `audit` | The full envelope: reserved-field slots, every classifier's full output, model resolution details, and run metadata |
+
+For complex requests, look for `audit.ack_reply` — that's the immediate acknowledgement your UI can show while the downstream model works. For trivial requests, `audit.final_reply.text` is a tiny terminal answer your app can return directly without ever calling the downstream model. The pipeline never decides for you; the caller chooses what to act on.
+
+Example result:
 
 ```json
 {
@@ -99,19 +116,21 @@ Example `route` result:
     "target_message": { "role": "user", "text": "...", "hash": "b11d5268" }
   },
   "classifier_outputs": {
+    "routing": { "model_tier": "frontier_strong" },
+    "model_specialization": { "model_specialization": "coding" },
+    "tools": { "tools": ["workspace"] },
+    "prompt_injection": { "risk_level": "normal" },
     "memory_retrieval_queries": { "queries": ["user code review preferences"] }
   },
   "audit": {
     "ack_reply": { "text": "Let me check." },
-    "routing": { "model_tier": "frontier_strong" },
-    "model_specialization": { "specialization": "coding" },
+    "routing": { "model_tier": "frontier_strong", "model_specialization": "coding" },
     "tools": { "tools": ["workspace"] },
+    "prompt_injection": { "risk_level": "normal" },
+    "classifier_outputs": [ /* every classifier's full output, with reason + certainty */ ],
     "model_recommendation": {
       "id": "gpt-5.5",
       "context_window": 1050000,
-      "input_tokens_cpm": 5,
-      "cached_tokens_cpm": 0.5,
-      "output_tokens_cpm": 30,
       "resolution": { "...": "..." }
     },
     "meta": { "classifiers": { "...": "..." } }
@@ -119,65 +138,96 @@ Example `route` result:
 }
 ```
 
-## Stock classifiers
-
-Stock classifiers are built in and have fixed, typed output shapes. Each one owns exactly one signal. Its manifest lives in `src/classifiers/stock/<name>/manifest.json`; the shared stock prompt building blocks live in `src/classifiers/stock/prompts/`.
-
-Every classifier prompt includes a shared header with its `Classifier` name, `Purpose`, and an instruction to treat that purpose as a hard scope boundary. In practice:
+## Classifier model
 
-- `routing` chooses only `model_tier`
-- `model_specialization` chooses only `specialization`
-- `prompt_injection` is only for prompt injection, not harmfulness, authorization, contradiction, feasibility, or freshness checks
+Open Classify ships with eight built-in classifiers; all use the same manifest shape. There is no distinction between "stock" and "custom" — the runtime only cares about which **reserved fields** a classifier declares.
 
-| Name | Signal | Short-circuits? |
+| Name | Reserved fields | What the aggregator does with it |
 |---|---|---|
-| `preflight` | `final_reply?` / `ack_reply?` | `final_reply` → `reply` |
-| `routing` | `model_tier?` | no |
-| `model_specialization` | `specialization?` | no |
-| `tools` | `{ tools[] }` | no |
-| `prompt_injection` | `{ risk_level }` | confident `high_risk` or `unknown` → `block` |
+| `preflight` | `final_reply`, `ack_reply` | Surfaces the highest-certainty reply in `audit.final_reply` / `audit.ack_reply` |
+| `routing` | `model_tier` | Feeds the catalog resolver as a soft constraint |
+| `model_specialization` | `model_specialization` | Feeds the catalog resolver as a soft constraint |
+| `tools` | `tools` | Sets `downstream.tools` |
+| `prompt_injection` | `risk_level` | Surfaces in `audit.prompt_injection` |
+| `memory_retrieval_queries` | — | Passes through to `classifier_outputs.memory_retrieval_queries` |
+| `conversation_digest` | — | Passes through |
+| `context_shift` | — | Passes through |
 
-Each output must carry `reason` (≤120 chars) and `certainty` (`no_signal` through `near_certain`). The aggregator maps certainty tags to numeric scores and drops below-threshold signals; the default threshold is `0.65`.
+Reserved fields are well-known output keys with canonical JSON Schemas and prompt fragments baked into the runtime. When you declare one in your manifest, you don't have to redeclare its enum values or shape — the runtime injects them.
 
-## Custom classifiers
+## Adding a classifier
 
-A custom classifier is two files in `src/classifiers/custom/<name>/`:
+Every classifier is two files in `src/classifiers/<name>/`:
 
-`manifest.json`:
+`manifest.json` describes the output shape and a fallback for when the classifier errors:
 
 ```json
 {
-  "kind": "custom",
-  "name": "memory_retrieval_queries",
+  "name": "topic_tags",
   "version": "1.0.0",
-  "purpose": "Generate retrieval queries likely to surface helpful user-specific context for the downstream model.",
-  "order": 60,
-  "fallback": {
-    "reason": "Classifier failed; no memory queries generated.",
-    "certainty": "no_signal",
-    "output": { "queries": [] }
-  },
+  "purpose": "Tag the message with a small set of topic labels for analytics.",
+  "dispatch_order": 70,
   "output_schema": {
-    "type": "object",
-    "additionalProperties": false,
-    "required": ["queries"],
+    "required": ["tags"],
     "properties": {
-      "queries": {
+      "tags": {
         "type": "array", "maxItems": 5,
-        "items": { "type": "string", "minLength": 1, "maxLength": 120 }
+        "items": { "type": "string", "minLength": 1, "maxLength": 40 }
       }
     }
+  },
+  "fallback": {
+    "reason": "Classifier failed; no tags generated.",
+    "certainty": "no_signal",
+    "tags": []
   }
 }
 ```
 
-`prompt.md`: your classifier-specific instructions.
+`prompt.md` describes the classification rule in plain language. You don't need to write JSON examples — the runtime synthesizes one from your schema and shows it to the model — and you don't paste enum values for reserved fields. Just describe what each output key means and when to abstain:
+
+```markdown
+You are the topic_tags classifier.
+
+`tags` are short single-word topic labels (lowercase, no spaces). Use at most five.
+Return an empty array when no clear topic applies.
+Do not invent tags for vague or ambiguous messages.
+```
+
+Rules:
+
+- `name` must match the directory name.
+- Reserved field names cannot appear in `output_schema.properties` — declare them in `reserved_fields` instead.
+- `fallback` must validate against the composed schema; reserved fields are optional in fallback since "I failed" means "no signal."
+- If you want hand-picked examples (preflight does this), add an `output_schema.examples` array. Each entry must validate against the composed schema at load time. Otherwise the runtime synthesizes a skeleton example for you.
 
-Custom classifiers receive the same shared `Classifier` + `Purpose` header and the same scope-boundary instruction, so keep the manifest `purpose` specific and operational.
+Consume your output:
+
+```ts
+const result = await classify(input);
+const tags = result.classifier_outputs.topic_tags?.tags ?? [];
+```
 
-The runtime auto-discovers it, validates outputs against your schema, and surfaces them on `result.classifier_outputs.<name>`. No TypeScript edits required.
+See [docs/adding-a-classifier.md](docs/adding-a-classifier.md) for a full walkthrough and [docs/manifests.md](docs/manifests.md) for the field reference.
 
-See [docs/adding-a-classifier.md](docs/adding-a-classifier.md) for a full walkthrough.
+## Using reserved fields in your own classifier
+
+Any classifier can emit reserved fields. If you write your own `task_router` that emits `model_tier`, the aggregator will fold it into the model resolution alongside the built-in `routing` classifier — highest-certainty contributor wins, ties broken by manifest `dispatch_order` ascending.
+
+```json
+{
+  "name": "task_router",
+  "version": "1.0.0",
+  "purpose": "Pick the downstream model tier and specialization for code-heavy tasks.",
+  "dispatch_order": 25,
+  "reserved_fields": ["model_tier", "model_specialization"],
+  "fallback": { "reason": "Classifier failed.", "certainty": "no_signal" }
+}
+```
+
+The runtime injects canonical sub-schemas and prompt fragments for each declared reserved field — the model is told the exact enum values it may emit. You don't paste enum values into `prompt.md`, and you don't have to hand-write a JSON example; the runtime synthesizes one from the schema and shows it to the model.
+
+The available reserved fields are: `final_reply`, `ack_reply`, `model_tier`, `model_specialization`, `tools`, `risk_level`. The `tools` field additionally requires an `allowed_tools` array on the manifest listing the tool ids the classifier may pick from.
 
 ## Model catalog
 
@@ -212,9 +262,7 @@ Classifiers never emit model ids. They emit constraints; your catalog maps const
 }
 ```
 
-OpenAI's GPT-5.5 model page lists text and image input, text output, a 1,050,000-token context window, 128,000 max output tokens, and text-token pricing of $5.00 input, $0.50 cached input, and $30.00 output per 1M tokens. OpenAI does not publish parameter counts, so use `null` for `params_in_billions`. See the [GPT-5.5 model details](https://developers.openai.com/api/docs/models/gpt-5.5) for current pricing and capability details.
-
-The resolver picks the cheapest model matching `specialization` and `tier`, relaxing constraints in order when nothing fits, and reports what it dropped on `audit.model_recommendation.resolution`. See [docs/resolver.md](docs/resolver.md) for ranking details.
+The resolver picks the cheapest model matching `model_specialization` and `model_tier`, relaxing constraints in order when nothing fits, and reports what it dropped on `audit.model_recommendation.resolution`. See [docs/resolver.md](docs/resolver.md) for ranking details.
 
 ## Input contract
 
@@ -244,24 +292,19 @@ cp open-classify.config.example.json open-classify.config.json
     "provider": "ollama",
     "defaultModel": "gemma4:e4b-it-q4_K_M",
     "models": {
-      "stock": {
-        "routing": "qwen2.5:7b-instruct-q4_K_M",
-        "prompt_injection": "llama-guard3:8b"
-      },
-      "custom": {
-        "memory_retrieval_queries": "qwen2.5:7b-instruct-q4_K_M"
-      }
+      "routing": "qwen2.5:7b-instruct-q4_K_M",
+      "prompt_injection": "llama-guard3:8b",
+      "memory_retrieval_queries": "qwen2.5:7b-instruct-q4_K_M"
     }
   },
   "aggregator": {
-    "certaintyThreshold": 0.65,
-    "certaintyGate": "min_score"
+    "certaintyThreshold": 0.65
   },
   "catalog": "downstream-models.json"
 }
 ```
 
-`runner.provider` currently supports `"ollama"` only. `runner.defaultModel` applies to any classifier without an explicit entry. `runner.models.stock` configures built-in classifiers; `runner.models.custom` configures custom classifiers by manifest name. `aggregator.certaintyGate` can be `"min_score"` (lowest score across all stock and custom classifiers), `"avg_score"`, or `"off"`. The setup script and `loadOpenClassifyConfig()` read `open-classify.config.json`, or `OPEN_CLASSIFY_CONFIG` when you want a different path.
+`runner.provider` currently supports `"ollama"` only. `runner.defaultModel` applies to any classifier without an explicit `runner.models` entry. `runner.models` is a flat map keyed by classifier name.
 
 ## Bring your own backend
 
@@ -284,15 +327,15 @@ const runClassifier: RunClassifier = async (name, input, signal) => {
   // call your provider of choice, return a ClassifierOutput
 };
 
-const classify = createClassifier({ runClassifier });
+const { classify, inspect } = createClassifier({ runClassifier });
 ```
 
-For the lowest-level entry point, `classifyOpenClassifyInput(input, { runClassifier, catalog })` skips the factory entirely.
+For the lowest-level entry points, `classifyOpenClassifyInput(input, { runClassifier, catalog })` and `inspectOpenClassifyInput(input, { runClassifier })` skip the factory entirely.
 
 ## Further reading
 
-- [docs/signals.md](docs/signals.md) — full signal contracts and validation rules
-- [docs/manifests.md](docs/manifests.md) — manifest reference (stock and custom)
+- [docs/signals.md](docs/signals.md) — reserved field reference
+- [docs/manifests.md](docs/manifests.md) — manifest reference
 - [docs/resolver.md](docs/resolver.md) — aggregation and model resolution
 - [docs/adding-a-classifier.md](docs/adding-a-classifier.md) — author guide
 

package/dist/src/aggregator.d.ts

@@ -1,5 +1,6 @@
 import type { AggregatorConfig, Catalog, ClassifierRegistry, ClassifierResults, Envelope, ModelRecommendation, ModelRecommendationResolution } from "./manifest.js";
-import type { AckReplySignal, ModelSpecializationClassifierOutput, FinalReplySignal, RoutingClassifierOutput, RoutingSignal } from "./stock.js";
+import type { AckReplySignal, Certainty, FinalReplySignal, RoutingSignal, ToolsSignal } from "./stock.js";
+import type { DownstreamModelTier, ModelSpecialization } from "./enums.js";
 import type { ClassifierInput } from "./types.js";
 export declare const DEFAULT_CERTAINTY_THRESHOLD = 0.65;
 /** @deprecated Use DEFAULT_CERTAINTY_THRESHOLD. */
@@ -15,7 +16,13 @@ export declare function composeEnvelope(args: ComposeEnvelopeArgs): Envelope;
 export declare function certaintyThreshold(config: AggregatorConfig | undefined): number;
 export declare function resolveModelFromRouting(routing: RoutingSignal | undefined, catalog: Catalog, confidence: number | undefined, ignoredConstraints?: ModelRecommendationResolution["constraints_dropped"]): ModelRecommendation;
 export declare function resolveModel(results: Readonly<{
-    routing?: RoutingClassifierOutput;
-    model_specialization?: ModelSpecializationClassifierOutput;
+    routing?: {
+        model_tier?: DownstreamModelTier;
+        certainty?: Certainty;
+    };
+    model_specialization?: {
+        model_specialization?: ModelSpecialization;
+        certainty?: Certainty;
+    };
 }>, catalog: Catalog, threshold: number): ModelRecommendation;
-export type { FinalReplySignal, AckReplySignal };
+export type { FinalReplySignal, AckReplySignal, ToolsSignal };