open-classify 0.5.0 → 0.7.0

package/docs/adding-a-classifier.md

@@ -1,20 +1,27 @@
 # Adding a classifier
 
-Every classifier — reserved-field-bearing or pure custom — uses the same two-file layout. There is no separate "stock" vs "custom" distinction; the runtime only cares about which reserved fields a classifier opts into.
+Every classifier — bundled or your own — uses the same two-file layout. There is no separate "stock" vs "custom" distinction; the runtime only cares about which reserved fields a classifier opts into.
+
+There are two places a classifier can live:
+
+- **In your own app**, in a directory you pass to `extraClassifierDirs` (almost always `./classifiers/` after `npx open-classify init`). This is the right path when you've installed Open Classify as a dependency.
+- **In this repo**, under `src/classifiers/<name>/`. Only do this when you're contributing a new mandatory built-in back to Open Classify.
+
+Either way, the layout and contract are identical.
 
 ## 1. Create the directory
 
 ```
-src/classifiers/<name>/
+classifiers/<name>/
 ├── manifest.json
 └── prompt.md
 ```
 
-The directory name must match `manifest.json`'s `name` field. Top-level directories starting with `_` (like `_prompts/`) are reserved for shared assets and skipped by the loader.
+The directory name must match `manifest.json`'s `name` field. Directories starting with `_` are skipped by the loader — that's the deactivation mechanism (`_topic_tags/` is inert; rename to `topic_tags/` to activate).
 
 ## 2. Write the manifest
 
-Minimal example — a pure-custom classifier that emits tags. You don't need to provide JSON examples; the runtime synthesizes one from your schema and shows it to the model.
+Minimal example — a pure-custom classifier that emits tags. The runtime synthesizes a JSON example from your schema, so you don't need to write one.
 
 ```json
 {
@@ -39,8 +46,6 @@ Minimal example — a pure-custom classifier that emits tags. You don't need to
 }
 ```
 
-If your classifier's behavior is nuanced enough that hand-picked examples would help the model (preflight is one), add an `output_schema.examples` array. The runtime validates each example against the composed schema at load time, so a broken example fails the build.
-
 To also influence routing, opt into a reserved field:
 
 ```json
@@ -71,8 +76,9 @@ Rules:
 - `name` must match the directory name.
 - Reserved field names cannot appear in `output_schema.properties`; declare them in `reserved_fields` instead.
 - `reason` and `certainty` are added to the composed schema by the runtime — don't declare them.
-- `fallback` must validate against the composed schema. Reserved fields are optional in fallback (a "no signal" fallback usually omits them).
+- `fallback` must validate against the composed schema. Only `reason` and `certainty` are required in fallback; reserved fields and `output_schema.required` fields are exempt (a "no signal" fallback usually omits them).
 - `output_schema.examples` (JSON Schema standard) must validate against the composed schema at load time, so a broken example fails the build, not the model call.
+- **Name collisions throw.** Extras cannot override the mandatory built-ins (`preflight`, `model_tier`, `model_specialization`, `prompt_injection`). To customize one of those, use a custom `RunClassifier` to intercept it (see "Replacing the backend" below).
 
 See [manifests.md](manifests.md) for the full field list.
 
@@ -90,24 +96,37 @@ Do not invent tags for vague or ambiguous messages.
 
 Don't paste enum values for reserved fields — the runtime injects them with canonical wording so they never drift from `src/enums.ts`.
 
-## 4. Build and test
+## 4. Use it
 
-```sh
-npm run build   # validates the manifest, composes the schema, copies assets
-npm test
-```
+After `npx open-classify init`, your `classifiers/` directory already exists. Drop your folder in and point `createClassifier` at the parent dir:
 
-If the manifest is malformed, the loader throws `ClassifierManifestError` with the path and a specific reason.
+```ts
+import { createClassifier } from "open-classify";
 
-## 5. Consume the output
+const { classify } = createClassifier({
+  extraClassifierDirs: ["./classifiers"],
+});
+
+const result = await classify({
+  messages: [{ role: "user", text: "Can you review the attached contract?" }],
+});
 
-```ts
-const { classify } = createClassifier({ catalog });
-const result = await classify(input);
 const tags = result.classifier_outputs.topic_tags?.tags ?? [];
 ```
 
-`result.audit.classifier_outputs[]` carries the same data with `reason` and `certainty` attached if you need to inspect them.
+> Production tip: `"./classifiers"` resolves against `process.cwd()`, which is fine for `npm start` but breaks if the process launches from a different directory. For long-running services, resolve absolutely via `fileURLToPath(import.meta.url) + path.resolve(...)`.
+
+If the manifest is malformed, `createClassifier` throws `ClassifierManifestError` at startup with the path and a specific reason — typos fail loud.
+
+## Activating one of the bundled templates
+
+`npx open-classify init` copies four templates (`tools`, `memory_retrieval_queries`, `conversation_digest`, `context_shift`) into your `classifiers/` directory as `_<name>/` — inactive because of the underscore prefix. To turn one on:
+
+```sh
+mv classifiers/_tools classifiers/tools
+```
+
+Edit `manifest.json` first if you need to (`tools` in particular ships with an opinionated `allowed_tools` list you'll almost certainly want to tailor). The reverse works on any classifier: rename `<name>/` → `_<name>/` to deactivate without deleting.
 
 ## Targeting the assistant response
 
@@ -117,7 +136,7 @@ Classifiers run against the user message by default. To run a classifier against
 - `"assistant"` — only `inspect()` runs it.
 - `"both"` — both passes run it.
 
-Use `inspect()` from `createClassifier()` for the assistant-side pass. It returns a lean shape (`target_message_hash` + `classifier_outputs`) — no routing, no audit envelope. The built-in `prompt_injection` ships tagged `"both"` so it runs on both sides.
+Use `inspect()` from `createClassifier()` for the assistant-side pass. It returns a lean shape: `target_message_hash`, the `message` that was inspected, and `classifier_outputs`. No routing, no action, no block logic.
 
 ```ts
 const { inspect } = createClassifier({ catalog });
@@ -130,9 +149,11 @@ const post = await inspect({
 const risk = post.classifier_outputs.prompt_injection?.risk_level;
 ```
 
+The built-in `prompt_injection` ships tagged `"both"` so it runs on both sides.
+
 ## Choosing the classifier model
 
-For apps and OSS installs, prefer `open-classify.config.json`:
+In `open-classify.config.json`:
 
 ```json
 {
@@ -146,9 +167,9 @@ For apps and OSS installs, prefer `open-classify.config.json`:
 }
 ```
 
-`runner.defaultModel` applies to every classifier without an override. `runner.models` is a flat map keyed by classifier name — there is no separate stock/custom split.
+`runner.defaultModel` applies to every classifier without an override. `runner.models` is a flat map keyed by classifier name — works for built-ins, templates, and your own.
 
-Classifier manifests may also carry an Ollama hint for packaged classifiers:
+Classifier manifests may also carry an Ollama hint:
 
 ```json
 {
@@ -160,15 +181,17 @@ Config file and function options take precedence over manifest hints.
 
 ## Replacing the backend
 
-For full backend control, implement your own `RunClassifier` and pass it to `classifyOpenClassifyInput`:
+For full backend control — including replacing a mandatory built-in like `preflight` — implement your own `RunClassifier` and pass it to `createClassifier`:
 
 ```ts
-import { classifyOpenClassifyInput, loadCatalog } from "open-classify";
+import { createClassifier, type RunClassifier } from "open-classify";
 
 const runClassifier: RunClassifier = async (name, input, signal) => {
-  // call OpenAI, Anthropic, a remote service, etc.
-  // return a ClassifierOutput matching the classifier's composed schema.
+  if (name === "preflight") {
+    // call OpenAI / Anthropic / your own logic; return a ClassifierOutput.
+  }
+  // …handle other classifiers, or delegate to the Ollama runner you imported.
 };
 
-await classifyOpenClassifyInput(input, { runClassifier, catalog: loadCatalog(...) });
+const { classify } = createClassifier({ runClassifier });
 ```

package/docs/manifests.md

@@ -17,7 +17,7 @@ The loader skips any top-level directory whose name starts with `_` (those are s
 | Field | Required | Description |
 |---|---|---|
 | `name` | yes | Classifier id. Must match the directory name. |
-| `version` | yes | Contract version surfaced in `meta.classifiers[name].version`. |
+| `version` | yes | Contract version string for this classifier. |
 | `purpose` | yes | Human-readable description of the classifier's job. Treated as a hard scope boundary in the prompt. |
 | `dispatch_order` | no | Non-negative integer scheduling priority. Lower runs first. Omit to schedule this classifier last (treated as +Infinity). Duplicate names are rejected; duplicate dispatch_orders are allowed and schedule adjacent. |
 | `applies_to` | no | One of `"user"`, `"assistant"`, `"both"`. Controls which pipeline pass the classifier participates in: `classify()` runs `"user"` + `"both"`; `inspect()` runs `"assistant"` + `"both"`. Defaults to `"user"`. |
@@ -34,12 +34,12 @@ Reserved fields are well-known output keys the aggregator knows how to consume.
 
 | Reserved field | Shape | What the aggregator does with it |
 |---|---|---|
-| `final_reply` | `{ text: string ≤200 chars }` | Surfaced in `audit.final_reply`; caller can return as the terminal reply |
-| `ack_reply` | `{ text: string ≤200 chars }` | Surfaced in `audit.ack_reply`; caller can show as an acknowledgement |
+| `final_reply` | `{ text: string ≤200 chars }` | Sets `result.action = "reply"` and `result.reply`; caller returns it as the terminal reply |
+| `ack_reply` | `{ text: string ≤200 chars }` | Sets `result.reply` (when action is `"route"`); caller shows it as an acknowledgement while downstream works |
 | `model_tier` | one of `DOWNSTREAM_MODEL_TIER_VALUES` | Soft constraint for catalog resolver |
 | `model_specialization` | one of `MODEL_SPECIALIZATION_VALUES` | Soft constraint for catalog resolver |
-| `tools` | array of allowed tool ids | Sets `downstream.tools` |
-| `risk_level` | one of `PROMPT_INJECTION_RISK_LEVEL_VALUES` | Surfaced in `audit.prompt_injection` |
+| `tools` | array of allowed tool ids | Sets `result.tools` |
+| `risk_level` | one of `PROMPT_INJECTION_RISK_LEVEL_VALUES` | Surfaced in `result.prompt_injection`; `"high_risk"` or `"unknown"` triggers `action: "block"` |
 
 `final_reply` and `ack_reply` are mutually exclusive — a single output may contain at most one.
 
@@ -49,7 +49,7 @@ When multiple classifiers emit the same reserved field, the highest-certainty co
 
 ```json
 {
-  "name": "routing",
+  "name": "model_tier",
   "version": "1.0.0",
   "purpose": "Recommend the downstream model tier.",
   "dispatch_order": 20,

package/docs/resolver.md

@@ -1,12 +1,10 @@
 # Aggregation and model resolution
 
-The aggregator merges classifier outputs into an `Envelope`, picks a concrete model from the catalog, and returns a `PipelineResult`.
+The aggregator merges classifier outputs into a `PipelineResult` with a flat shape — no nested `audit` or `downstream` envelope.
 
-## Certainty threshold
+## Certainty labels
 
-Default: `0.65`. Configurable via `aggregator.certaintyThreshold` on `classifyOpenClassifyInput`. `aggregator.confidenceThreshold` remains as a deprecated compatibility alias.
-
-Per-classifier outputs carry `certainty` tags. The aggregator maps tags to scores:
+Classifier outputs carry a `certainty` label. The aggregator maps labels to numeric scores for comparison and reporting:
 
 ```ts
 {
@@ -21,23 +19,27 @@ Per-classifier outputs carry `certainty` tags. The aggregator maps tags to score
 }
 ```
 
-Reserved-field values from below-threshold classifiers are dropped from the named envelope slots. The full underlying output still appears in `audit.classifier_outputs[]` and `audit.meta.classifiers[name]`, so the caller can inspect or override.
+Labels stay in classifier prompts (the model understands them as semantic grades). Floats appear only in the final `PipelineResult` fields: `avg_certainty`, `min_certainty`, and `classifier_outputs[name].certainty`.
 
-Dropped routing axes are reported on `audit.model_recommendation.resolution.constraints_dropped` with `reason: "low_confidence"`.
+## Reserved-field merging
 
-Custom (non-reserved) outputs are surfaced regardless of certainty — callers can decide what to do with them — but the value still goes through schema validation.
+When multiple classifiers emit the same reserved field, the aggregator picks the highest-certainty contributor. Ties are broken by manifest `dispatch_order` ascending (first wins). Classifiers without `dispatch_order` sort last for tie-break purposes.
 
-## Reserved-field merging
+There is no certainty threshold gate — the highest-certainty value always wins, regardless of score. Values below any particular threshold are still reported in `classifier_outputs` for the caller to inspect.
 
-When multiple classifiers emit the same reserved field, the aggregator picks the highest-certainty contributor that meets the threshold. Ties are broken by manifest `dispatch_order` ascending (first wins). Classifiers without `dispatch_order` sort last for tie-break purposes.
+## Action
 
-The built-in classifiers each own a distinct reserved field, so the tie-break only matters if you add your own classifier that emits a field already covered by a built-in.
+Every result has `action: "route" | "block" | "reply"`.
 
-## Whole-run certainty summary
+**`"reply"`** — `preflight` emitted `final_reply`. The classifier determined it can answer the message immediately; no downstream model is needed. `result.reply` contains the text. All other classifiers still ran.
 
-Every run includes `audit.meta.certainty.{min, avg}`. These are the lowest and arithmetic-mean certainty scores across all classifiers, including failed classifiers that fell back to their manifest fallback (which use `no_signal` and score 0).
+**`"block"`** — something prevented routing. `result.block_reason` names the cause:
+- `"prompt_injection"` — `risk_level` is `"high_risk"` or `"unknown"`, regardless of certainty. This takes priority over other causes.
+- `"classification_error"` — one or more classifiers failed or timed out, or preflight provided no reply (which means the pipeline cannot fulfill its reply contract), or no model could be resolved.
 
-The pipeline does not block based on this summary — the worker pool always returns a `route` action. Callers inspect `audit.meta.certainty` and decide whether to trust the result or fall back to a safer behavior (e.g., force a frontier model, return an apology).
+**`"route"`** — all classifiers succeeded and `result.model_id` names the downstream model to call.
+
+Even on `"block"`, `model_id` and `reply` are populated when they can be (the caller may want to store them). `failed_classifiers` lists every classifier that errored or timed out.
 
 ## Model resolution
 
@@ -60,36 +62,10 @@ Within a pass, candidates are ranked:
 3. larger `context_window`
 4. earlier catalog order
 
-If every pass returns no candidates, the resolver returns `catalog.default` with `fell_back_to_default: true`. (In practice the no-constraints pass always finds at least one model unless the catalog is empty, so the default-fallback path is defensive.)
-
-## Resolution audit
+If every pass returns no candidates, the resolver uses `catalog.default`. In practice the no-constraints pass always finds at least one model unless the catalog is empty, so the default-fallback path is defensive.
 
-Every result carries a resolution report:
-
-```ts
-{
-  constraints_used: { model_specialization?: ..., model_tier?: ... },
-  constraints_dropped: Array<{
-    axis: "model_specialization" | "model_tier",
-    reason: "low_confidence" | "no_match_relaxed" | "default_fallback"
-  }>,
-  confidences: { routing?: number },
-  fell_back_to_default: boolean,
-}
-```
-
-Drop reasons:
-
-- `low_confidence` — a classifier emitted the axis but its certainty was below threshold.
-- `no_match_relaxed` — the axis was requested but no model matched, so the resolver relaxed it.
-- `default_fallback` — every pass failed; the resolver used `catalog.default`.
-
-## Audit envelope
+## Whole-run certainty summary
 
-The full `audit` envelope contains:
+Every run includes `avg_certainty` and `min_certainty` at the top level of `PipelineResult`. These are the arithmetic mean and minimum certainty scores across all classifiers, including failed classifiers that fell back to their manifest fallback (which use `no_signal` and score `0`).
 
-- Reserved-field slots that survived the certainty threshold: `final_reply`, `ack_reply`, `routing`, `tools`, `prompt_injection`
-- `classifier_outputs[]` — every classifier's full output, in registry order, including `reason`, `certainty`, all reserved fields, and all custom fields
-- `model_recommendation` with the resolution audit above
-- `meta.classifiers[name]` — per-classifier full output plus `status` and `version`
-- `meta.certainty.{min, avg}` — whole-run certainty summary
+The pipeline does not block based on these values — the caller inspects them and decides whether to trust the result or fall back to a safer behavior.

package/docs/signals.md

@@ -14,7 +14,7 @@ type Certainty =
   | "near_certain";
 ```
 
-The aggregator maps certainty tags to numeric scores. Reserved-field values from classifiers below the configured threshold (default `0.65`) are dropped from the envelope; the underlying output still appears in `audit.classifier_outputs` and `meta.classifiers` so the caller can decide whether to trust the run.
+The aggregator maps certainty tags to numeric scores. `classifier_outputs[name].certainty` is a float; `avg_certainty` and `min_certainty` on the top-level result are also floats. Certainty labels are internal to classifier prompts; floats are what callers see.
 
 ## Reserved fields
 
@@ -26,9 +26,9 @@ A manifest declares which reserved fields its classifier may emit via the `reser
 { text: string }  // 1–200 chars; must contain at least one non-whitespace character
 ```
 
-Use only for tiny terminal answers (greetings, thanks, spelling, simple arithmetic). The text IS the complete answer to the user — nothing else happens after this. Mutually exclusive with `ack_reply`.
+Use only for tiny terminal answers (greetings, thanks, spelling, simple arithmetic). The text IS the complete answer — nothing else happens after this. Mutually exclusive with `ack_reply`.
 
-When emitted with sufficient certainty, the highest-certainty value is surfaced in `audit.final_reply`. The pipeline does not short-circuit; the caller decides whether to return the reply or continue to the downstream model.
+When emitted, the pipeline sets `action: "reply"` and surfaces the text in `result.reply`. All other classifiers still run to completion.
 
 ### `ack_reply`
 
@@ -36,7 +36,9 @@ When emitted with sufficient certainty, the highest-certainty value is surfaced
 { text: string }  // 1–200 chars; must contain at least one non-whitespace character
 ```
 
-A brief acknowledgement to show while downstream work continues. Surfaced in `audit.ack_reply`. Mutually exclusive with `final_reply`.
+A brief, task-specific acknowledgement to show while downstream work continues. Mutually exclusive with `final_reply`.
+
+When emitted (and the action is `"route"`), the text is surfaced in `result.reply`. This is the immediate response your UI can show while the downstream model works.
 
 ### `model_tier`
 
@@ -62,22 +64,30 @@ Soft constraint for the catalog resolver. The resolver picks the cheapest catalo
 string[]   // each id must appear in the manifest's allowed_tools list
 ```
 
-Sets `downstream.tools.tools`. Any classifier emitting this reserved field must declare `allowed_tools` on its manifest — that menu of allowed ids becomes both the JSON Schema constraint and the prompt listing.
+Sets `result.tools`. Any classifier emitting this reserved field must declare `allowed_tools` on its manifest — that menu of allowed ids becomes both the JSON Schema constraint and the prompt listing.
 
 Common tool-id aliases (`browser`, `browsing`, `internet`, `web_browsing`, `web_search`) are normalized to `web` before validation, so the model can drift on phrasing without breaking.
 
+`result.tools` is always an array (empty if no classifier emitted it or no tools were selected).
+
 ### `risk_level`
 
 ```ts
 "normal" | "suspicious" | "high_risk" | "unknown"
 ```
 
-Prompt-injection posture for the target message. Surfaced in `audit.prompt_injection`. The pipeline does not short-circuit; the caller decides whether to block based on the risk level and certainty.
+Prompt-injection posture for the target message. Surfaced in `result.prompt_injection`.
+
+`"high_risk"` and `"unknown"` trigger `action: "block"` with `block_reason: "prompt_injection"`, regardless of certainty. `"suspicious"` is advisory — the pipeline routes normally and the caller decides whether to act on it.
+
+When the `prompt_injection` classifier fails (runtime error or timeout), it uses its fallback which does **not** include `risk_level`. The pipeline then blocks with `block_reason: "classification_error"`, not `"prompt_injection"` — a classifier failure is distinct from an assessed injection risk.
 
 ## Custom fields
 
-Anything not in the reserved list lives in your manifest's `output_schema.properties`. The runtime validates each output against the composed schema (custom properties + reserved sub-schemas + `reason` + `certainty`) at runtime, and surfaces the full output on `result.classifier_outputs[name]` with `reason` and `certainty` stripped for ergonomic access. The full output, including metadata, appears in `result.audit.classifier_outputs[]` and `result.audit.meta.classifiers[name]`.
+Anything not in the reserved list lives in your manifest's `output_schema.properties`. The runtime validates each output against the composed schema (custom properties + reserved sub-schemas + `reason` + `certainty`) at runtime, and surfaces the full output on `result.classifier_outputs[name]`.
+
+`classifier_outputs[name]` contains all payload fields plus `reason` (string) and `certainty` (float). The raw certainty label is not exposed; only the float score.
 
 ## Picking between reserved-field contributors
 
-When two classifiers declare the same reserved field, the aggregator picks the highest-certainty value above the threshold. Ties are broken by manifest `dispatch_order` ascending (first in registry order keeps the slot). Both classifiers' full outputs still appear in `audit.classifier_outputs` regardless of which one "won" the slot.
+When two classifiers declare the same reserved field, the aggregator picks the highest-certainty value. Ties are broken by manifest `dispatch_order` ascending (first in registry order keeps the slot). Both classifiers' full outputs still appear in `classifier_outputs` regardless of which one "won" the slot.

package/open-classify.config.example.json

@@ -11,15 +11,10 @@
     },
     "models": {
       "preflight": "gemma4:e4b-it-q4_K_M",
-      "routing": "gemma4:e4b-it-q4_K_M",
+      "model_tier": "gemma4:e4b-it-q4_K_M",
       "model_specialization": "gemma4:e4b-it-q4_K_M",
-      "tools": "gemma4:e4b-it-q4_K_M",
-      "prompt_injection": "gemma4:e4b-it-q4_K_M",
-      "memory_retrieval_queries": "gemma4:e4b-it-q4_K_M"
+      "prompt_injection": "gemma4:e4b-it-q4_K_M"
     }
   },
-  "aggregator": {
-    "certaintyThreshold": 0.65
-  },
   "catalog": "downstream-models.json"
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "open-classify",
-  "version": "0.5.0",
+  "version": "0.7.0",
   "description": "Manifest-driven classifier runtime for routing user messages to downstream AI models",
   "license": "MIT",
   "author": "Taylor Bayouth",
@@ -29,11 +29,16 @@
       "default": "./dist/src/index.js"
     }
   },
+  "bin": {
+    "open-classify": "./bin/open-classify.mjs"
+  },
   "files": [
+    "bin",
     "dist/src",
     "docs",
     "downstream-models.json",
     "open-classify.config.example.json",
+    "templates",
     "LICENSE",
     "README.md"
   ],