ai-capability-framework 1.0.2 → 1.1.0-rc.1

package/docs/evals/discovery-qualification.md

@@ -0,0 +1,153 @@
+# Discovery qualification for 1.1.0-rc.1
+
+Lexical retrieval with ordinary model tool selection (**lexical/native**) is the
+recommended default. Lexical/dedicated also qualified for the tested configuration; it
+remains an optional additional model call. Hybrid/native and hybrid/dedicated remain
+**experimental and opt-in** because they did not meet the incremental-gain gate.
+Discovery's API classification remains **release-candidate public**.
+
+These results qualify a configuration on an authored synthetic dataset. They do not
+establish general production quality, guarantee the first candidate, or authorize an
+action. Applications still own intent decomposition, dependencies, arguments and
+customer responses. Evaluate your own capability corpus and traffic before adoption.
+
+## Tested configuration and assessment history
+
+The campaign used suite v3: 240 cases across support, inventory and scheduling, with 80
+per domain and 120/60/60 train/calibration/held-out cases. There are 60 scenario
+families overall and 15 held-out families. Related paraphrases share a partition. The
+fresh held-out families were withheld from calibration; this is not an external blind
+benchmark, and published fixtures are now exposed regression material.
+
+The frozen configuration used eight candidates; gpt-6-luna with reasoning effort none
+for planning, selection and ordinary turns; text-embedding-3-small at 1,536 dimensions;
+hybrid weights 0.75 semantic/0.25 lexical; and unavailable on hybrid failure. These are
+campaign settings, not framework model defaults. Model access, explicit indexes and
+finite budgets remain caller-owned. Returned generation model identities matched the
+requested alias; an immutable model snapshot was not available.
+
+The full 240-case pass and three 60-case held-out passes were assessed independently,
+including each mode's declared calibration baseline and critical cohorts. Ready-task p95
+had to be at most 15 seconds, with a 30-second absolute deadline. Quality thresholds
+were 95% recall and selected-candidate accuracy, 90% coverage, at most 1% false
+selection on true no-match cases, and zero safety violations. Critical-cohort
+regressions greater than five percentage points failed the gate. No pooled average
+replaced a failed pass.
+
+The original workflow assessment was **40/48**, with all four modes unqualified. A
+separately versioned offline wording-checker correction reassessed the same saved
+evidence as **48/48**, or **12/12 per mode**. Seven date-rendering assessments and one
+negated-completion assessment changed. This was not a fresh live trial or a model
+improvement. The eight actual deterministic fallback responses, tool receipts,
+operational failures, original grades and accounting remain unchanged. The earlier,
+different campaign's 22/48 result was not regraded.
+
+## Full-suite results
+
+Accuracy is measured on selected answerable matched cases. Coverage retains missed and
+failed decisions in its denominator. Recall uses the suite's answerable cases. The
+remaining cases include true no-match and eligibility-exclusion scenarios.
+
+| Mode              | Compatible recall at 8 | Selected accuracy | Selection coverage | Ready-task p95 | Revised overall               |
+| ----------------- | ---------------------- | ----------------- | ------------------ | -------------- | ----------------------------- |
+| lexical/native    | 118/120                | 118/118           | 118/120            | 4.14 s         | Qualified                     |
+| lexical/dedicated | 118/120                | 118/118           | 118/120            | 6.15 s         | Qualified                     |
+| hybrid/native     | 120/120                | 120/120           | 120/120            | 4.42 s         | Unqualified: incremental gain |
+| hybrid/dedicated  | 120/120                | 117/117           | 117/120            | 5.98 s         | Unqualified: incremental gain |
+
+Case-level descriptive 95% Wilson intervals are below. Paraphrases within families are
+correlated; these are not family-independent uncertainty estimates or guarantees.
+
+| Mode              | Recall interval | Accuracy interval | Coverage interval |
+| ----------------- | --------------- | ----------------- | ----------------- |
+| lexical/native    | 94.13–99.54%    | 96.85–100.00%     | 94.13–99.54%      |
+| lexical/dedicated | 94.13–99.54%    | 96.85–100.00%     | 94.13–99.54%      |
+| hybrid/native     | 96.90–100.00%   | 96.90–100.00%     | 96.90–100.00%     |
+| hybrid/dedicated  | 96.90–100.00%   | 96.82–100.00%     | 92.91–99.15%      |
+
+Every mode made **0/48 false selections** on full-suite true no-match cases (95% Wilson
+interval 0–7.41%). Permission/lifecycle violations, policy-forbidden export, forbidden
+choices, read-to-mutation selections and commit exposure/execution were zero. An
+eligible competing read/prepare candidate in a shortlist is not an execution violation.
+
+Retrieval alone still produced candidates for **47/48 lexical** and **48/48 hybrid**
+true no-match cases. Models abstained or otherwise safely declined these requests. This
+high retrieval false-positive rate is a practical reason to retain ordinary model
+selection and clarification rather than automatically execute the highest-scoring tool.
+
+Full-suite selection outcomes were 118 selected/49 abstained/73 unperformed for each
+lexical mode; 120/48/72 for hybrid/native; and 117 selected/48 abstained/3 invalid
+outputs/72 unperformed for hybrid/dedicated. Unperformed means no model decision was
+requested; it is not fabricated abstention.
+
+## Held-out repetitions and workflows
+
+Each held-out pass had 60 cases, 15 families, 24 answerable cases and 12 true no-match
+cases. Recall was **24/24** for every mode in every pass. Columns show passes 1, 2 and 3
+in order. Consistency counts identical outcome/choice across repetitions, including
+non-answerable cases; it is reported separately from gate results.
+
+| Mode              | Coverage by pass    | Selected accuracy by pass | p95 seconds by pass | Consistent cases |
+| ----------------- | ------------------- | ------------------------- | ------------------- | ---------------- |
+| lexical/native    | 24/24; 24/24; 24/24 | 24/24; 24/24; 24/24       | 3.66; 3.64; 3.85    | 57/60            |
+| lexical/dedicated | 24/24; 24/24; 24/24 | 24/24; 24/24; 24/24       | 5.56; 6.22; 5.89    | 59/60            |
+| hybrid/native     | 24/24; 24/24; 24/24 | 24/24; 24/24; 24/24       | 4.70; 3.94; 4.25    | 59/60            |
+| hybrid/dedicated  | 23/24; 24/24; 23/24 | 23/23; 24/24; 23/23       | 6.39; 5.41; 6.02    | 58/60            |
+
+For 24/24, the 95% Wilson interval is 86.20–100%; for 23/24, 79.76–99.26%; and for
+23/23, 85.69–100%. False selection was 0/12 in every pass (interval 0–24.25%). All
+critical-cohort comparisons and ready-task latency gates passed independently.
+
+Hybrid/dedicated retained four invalid selector outputs: three in the full pass and one
+in held-out pass 1. A separate unavailable selector decision in held-out pass 3 also
+remains a failure in coverage. Neither outcome was replaced by a successful rerun.
+
+All modes passed the revised 12-workflow assessment. Across the 48 workflows, the
+application performed 24 authorized reads and 12 approval-required preparations, with no
+commit exposure. Application delivery was already 48/48 using eight observed fallbacks;
+the offline correction changed wording-dependent grading only.
+
+## Why hybrid is not promoted
+
+Promotion required at least two percentage points higher compatible recall, or at least
+20% lower model-input tokens at unchanged recall, against lexical retrieval with the
+same candidate limit and selection mechanism. The observed recall increase was **1.67
+percentage points** (120/120 versus 118/120).
+
+| Mode              | Full-pass model-input tokens |
+| ----------------- | ---------------------------- |
+| lexical/native    | 619,786                      |
+| lexical/dedicated | 445,346                      |
+| hybrid/native     | 715,689                      |
+| hybrid/dedicated  | 466,759                      |
+
+These model-input counts exclude query-embedding tokens. Hybrid increased model input
+instead of achieving the required reduction. Both hybrid modes therefore remain
+unqualified overall despite passing their matching, workflow and safety gates. No
+ranking weights, thresholds, aliases or labels were tuned to turn this into a pass.
+
+The current campaign recorded 3,097 generation/embedding attempts and 5,521 HTTP
+requests including token preflights; reported input/output tokens were 3,889,390/108,203
+with complete current-campaign usage. Earlier campaign evidence retains an unknown-usage
+provider failure and conservative charging. Stage receipts are not added again to
+overlapping run/campaign totals. Offline regrading and release verification purchase
+zero additional model calls.
+
+## Evidence bindings and reproduction
+
+The live transport receipts and original model wording remain private. The following
+safe bindings identify the frozen inputs and derived assessment:
+
+- Suite v3: `sha256:5e5f0cb1a848b733174e6e2e444e7c3bd8a90680fcd6009c7323d635c3605f69`.
+- Qualification workflows:
+  `sha256:8fabf71450f0d3b04a9d4b543d844bc94b0e70e20c698897db127a7d516cff61`.
+- Campaign: `sha256:187b918cea5aecc0ab4075f2770310a0eddc314bf02d5d147008db773e5b6b76`.
+- Original report bytes:
+  `sha256:052d0e1abb1d39f527cf0258510058ebd3bc4ef07ff557761df4d447c6ef607e`.
+- Revised report:
+  `sha256:f354d381035f6b28b21c3812d45ac69e14e5577c980f93dc53ba7b7a57d89def`.
+
+The correction uses synthetic-wording.v2 and workflow-wording-regrade.v1. See the
+[evaluation and offline-regrade commands](discovery-live.md) for reproducible
+provider-free checks and explicit, separately budgeted live execution. No-key examples
+validate integration contracts; they do not recreate these live observations.

package/docs/evals/overview.md

@@ -12,3 +12,12 @@ Read:
 - [Replay and trace-to-golden](replay-and-trace-to-golden.md)
 - [Live evals](../live-evals.md)
 - [EvalOps export interfaces](../evalops.md)
+
+## Discovery and selection evaluation
+
+Use [discovery evaluation](../discovery.md#offline-evaluation-and-quality) for frozen
+synthetic cases, compatible and forbidden ID sets, separate retrieval/selection metrics,
+and explicit accounting completeness. No-key CI validates contracts; model quality
+requires a separately configured evaluation. The
+[discovery live campaign](discovery-live.md) provides bounded, resumable evaluation with
+caller-supplied clients; its default command remains provider-free.

package/docs/index.md

@@ -73,3 +73,12 @@ Models propose; applications validate, authorize, execute, and audit.
 - [Security disclosure](public-framework/security-disclosure.md)
 - [API reference](api.md)
 - [Public API policy](api/public-api-policy.md)
+
+## Capability discovery
+
+The optional release-candidate `ai-capability-framework/discovery` API provides a simple
+search facade, result-bound routers, explicit hybrid retrieval and dedicated selection,
+and provider-free evaluation. Start with
+[finding capabilities for a task](discovery.md). For opt-in provider evaluation, see the
+[discovery live evaluation guide](evals/discovery-live.md) and the
+[qualification results and limits](evals/discovery-qualification.md).

package/docs/release.md

@@ -2,6 +2,25 @@
 
 This checklist prepares an AICF public release for GitHub and npm distribution.
 
+## Discovery prerelease
+
+Discovery is release-candidate public. Recommend lexical retrieval with ordinary model
+tool selection; dedicated selection is optional and hybrid remains experimental. Link
+the [qualification report](evals/discovery-qualification.md) in release notes. Its
+qualified lexical results apply to a tested synthetic configuration, and the revised
+workflow assessment is an offline checker correction rather than new live trials.
+
+Use a reviewed pull request, passing hosted Linux Node 20/22/24 checks, merge, and an
+exact-commit manual release dry-run before pushing the matching prerelease tag. Local
+checks do not replace hosted checks. Publish both packages to `next`, preserving
+`latest`, then verify exact-version registry installs.
+
+The publication and dry-run workflows run `npm run check` plus certification with
+`VITEST_MAX_WORKERS=2`. Publication is serialized; registry failures fail closed unless
+npm explicitly reports E404 for an absent version. Local npm login is not the OIDC
+publishing mechanism. Verify package ownership and the trusted publisher for both
+packages before tagging; never substitute a stored npm token.
+
 ## Preflight
 
 - Confirm the release branch contains only public framework material.
@@ -261,11 +280,12 @@ npm view "ai-capability-framework@${VERSION}" version
 npm view "@aicf/agent-skills@${VERSION}" version
 ```
 
-If only `@aicf/agent-skills` published, fix the workflow or npm ownership issue and
-retry the root package publish at the same version from the same tag. If only the root
-package published, retry the agent-skills publish at the same version from the same tag.
-Do not move the public tag, delete the published version, or reuse a published npm
-version as a normal recovery path.
+If only one package published, mark the release incomplete and inspect its registry
+metadata before taking further action. The ordinary workflow rejects already-published
+targets, so do not blindly rerun it. Use a new coordinated release candidate when the
+existing workflow cannot safely complete the missing package. Do not move the public
+tag, delete the published version, or reuse a published npm version as a normal recovery
+path.
 
 If the tagged code is wrong, or if the failed package cannot be safely published from
 the same commit, bump both package versions and ship a follow-up patch or release

package/docs/runtime.md

@@ -89,6 +89,12 @@ are incompatible with active read-only controls.
 Use `formatCapabilitySliceForModel()` to render concise model-facing capability guidance
 without private diagnostics, internal policy rules, or full output schemas.
 
+`filterEligibleCapabilities(request)` exposes the same runtime eligibility filters
+without ranking or count truncation. It returns `eligible` manifests in registry order
+and `excluded` capability IDs with their existing reasons. The default router uses this
+helper before scoring and applying its capability limit. Eligibility is a request-time
+filter; normal execution authorization still applies.
+
 ## Policy Broker
 
 `DefaultPolicyBroker` wraps the Core `decideCapability()` control plane and adds runtime
@@ -167,3 +173,19 @@ node examples/runtime-support-billing/run-mock.mjs
 It loads the support manifests, routes read and prepare capabilities, executes a ticket
 read, prepares a refund case that requires approval, records a host approval, and
 commits only through `AicfActionLifecycleManager`.
+
+### Optional shared provider runs
+
+`createAicfProviderRun` joins explicitly managed application planning, discovery,
+selection, and Responses tool-loop calls under one reservation ledger and absolute
+deadline. Its immutable snapshots retain unknown usage, pending work, and unique
+attempts. The default router and ordinary runner calls remain source-compatible. See the
+[shared-run example](../examples/12-capability-discovery-selection/README.md#shared-run-integration)
+for task-bound routing, optional selection, metadata tracing, and offline replay.
+
+## Optional capability discovery
+
+Use [capability discovery](discovery.md) to retrieve an eligible shortlist and pass its
+result-bound router to an ordinary model/tool runner. The optional discovery subpath
+preserves runtime authorization and read/prepare boundaries; default routing stays
+unchanged. Dedicated model selection is explicitly invoked.

package/examples/12-capability-discovery-selection/README.md

@@ -0,0 +1,345 @@
+# From a customer request to a tool
+
+Use `ai-capability-framework@1.1.0-rc.1` for these discovery APIs. Lexical/native is the
+recommended default. Hybrid is experimental and opt-in; the
+[qualification report](../../docs/evals/discovery-qualification.md) explains its limits.
+
+The application AI produces tasks. Application code searches for relevant capabilities,
+then its existing model chooses a tool from the shortlist. The runtime validates the
+call and enforces execution and approval policy.
+
+Run the complete no-key example from the repository root:
+
+```sh
+npm run build
+node examples/12-capability-discovery-selection/run-customer-mock.mjs
+```
+
+The customer asks to read a ticket and prepare a refund for its order. A mocked planning
+call returns two dependent tasks. Application code searches each task when ready, passes
+the authorized read result into the next task's context, and uses ordinary model tool
+calls to read the ticket and prepare a refund preview. The example's host policy
+requires operator review. It stops with approval pending and never exposes or executes
+refund commit.
+
+The discovery integration needs only the registry, tasks, and existing runtime context:
+
+```js
+import { createCapabilityDiscovery } from "ai-capability-framework/discovery";
+
+const discovery = createCapabilityDiscovery({ registry });
+const [result] = await discovery.search({
+  tasks: [{ id: "read-ticket", text: "Read ticket TCK-100." }],
+  runtimeContext
+});
+const router = discovery.routerFor(result);
+// Pass router, the same runtimeContext, and the exact task text to your ordinary runner.
+```
+
+The default profile uses lexical retrieval, up to eight candidates, no alias bonus or
+diversity, and a 30-second discovery deadline. An earlier caller deadline wins. Search
+accepts controls, existing eligibility filters and permission overrides, cancellation,
+and a custom context builder. Batch results retain task IDs and input order. Use the
+low-level discoverer for multiple fragments or advanced ranking.
+
+`routerFor` reuses the issued shortlist without ranking or embedding again. Results are
+immutable and bound to their issuing instance, original task text, runtime context, and
+source. Keep the result in the same request; reconstructed or foreign results are
+rejected. Changed source, identity, permissions, controls, cancellation, or expiry
+cannot export a stale shortlist. Prepare dependent context before searching. Provider
+export and execution retain their ordinary checks.
+
+The example also runs no-match clarification, permission denial, and an explicitly
+invoked mock hybrid companion. Planning and ordinary model usage are reported
+separately: one planning turn and four ordinary turns, all synthetic. Lexical discovery
+uses no provider calls. The hybrid companion uses one explicit index-build batch and one
+query attempt; invalid-index rejection purchases no query embedding. No dedicated
+selector runs.
+
+## Optional semantic search
+
+Enable hybrid mode explicitly, using your configured embedding client and an index built
+in a separate administrative operation:
+
+```js
+import { createInMemoryDiscoveryBudget } from "ai-capability-framework/discovery";
+
+const budget = createInMemoryDiscoveryBudget({
+  runId: runtimeContext.runId,
+  maxProviderCalls: 8,
+  maxInputTokens: 32000
+});
+const discovery = createCapabilityDiscovery({
+  registry,
+  mode: "hybrid",
+  embedding: { client: embeddingClient, index, maxInputTokens: 4000 }
+});
+const results = await discovery.search({ tasks, runtimeContext, budget });
+```
+
+The standard hybrid starting weights are 0.75 semantic and 0.25 lexical. Failures return
+unavailable unless lexical fallback is explicitly configured. Override `weights`,
+`failurePolicy`, and `limits` on the facade as needed. Embedding options allow
+overriding artifact bounds (16 MiB, 10,000 entries, 4,096 dimensions) and provider
+character bounds (16,384 per text, 65,536 per call). The caller supplies the token
+ceiling. Token enforcement and retry disabling remain embedding-client responsibilities.
+Indexes must still match the exact source and embedding configuration; serving never
+builds or repairs them.
+
+The in-memory helper enforces a finite single-run budget in one process. A supplied
+`maxEstimatedCost` also requires `embedding.maxEstimatedCostPerCall`; the framework
+assumes no prices. Unknown consumption retains reserved allowance. Late completion
+settles its reservation without restoring a cancelled result. An unresolved attempt
+retains its reservation for the lifetime of the helper.
+
+For distributed or durable enforcement, pass `budget: { ledger, policy }` with your own
+ledger and existing budget policy. Reuse the scope across discovery calls in a run. Use
+the shared-run companion below to account for managed planning and downstream model
+calls; this example reports those stages separately. No default credentials, model
+service, or live transport is supplied.
+
+These mocks demonstrate contracts and integration. They do not establish real semantic
+quality or guarantee that the first candidate is the correct tool. Applications own
+decomposition, task dependencies, argument completion, and customer responses. A model
+may choose a candidate or request clarification; scores never authorize execution.
+
+## Optional dedicated selection
+
+Use the ordinary model/tool loop for the standard flow above. If your application needs
+a separate model decision, explicitly call the same discovery instance's `select`
+method:
+
+```js
+const batch = await discovery.select({
+  results,
+  client: selectionClient,
+  budget,
+  maxInputTokens: 4000,
+  maxOutputTokens: 1000
+});
+```
+
+The framework recovers task text and context bindings from its own search results and
+constructs candidate cards. Supply a configured `SelectionModelClient` with an explicit
+`mock` marker and `generateStructured` method; input/output token ceilings are passed to
+that client. Use the existing budget helper or your own `{ ledger, policy }` scope. Cost
+caps require `maxEstimatedCostPerCall`, with optional host cost estimation. There is no
+default model, account, price, or credential lookup.
+
+`batch.outcomes` contains ordered selected/abstained decisions or safe failure outcomes
+for matched tasks. Non-matched tasks stay in `batch.discovery` and are never represented
+as model abstentions. `batch.usage` owns selection and repair usage once for the whole
+batch; discovery usage remains in the original discovery results. Individual outcomes
+carry no duplicate provider-usage receipt.
+
+Cards preserve complete model descriptions, usage/exclusion guidance, input-field names,
+and read/prepare distinctions. Scores, full schemas, runtime identities, hidden policy,
+and paths are omitted. Optional `contextByQueryId` explicitly projects application data
+for the model. Foreign/reconstructed results and incompatible invocation scopes are
+rejected. Current source, eligibility, identity, controls, cancellation, and original
+deadlines are rechecked; a new deadline can shorten the original bound.
+
+Defaults are eight tasks, 32,000 serialized input characters, and one attempt. Override
+`limits: { maxTasks, maxInputCharacters, maxAttempts }` up to 32 tasks, 128,000
+characters, and two attempts. The same character ceiling bounds response parsing. With
+two attempts, one repair may use safe issue codes and the original cards. Oversized
+batches/cards are rejected without clipping or paid splitting. Provider, accounting,
+stale-binding, budget, control, cancellation, and deadline failures cause no repair or
+automatic candidate choice.
+
+Advanced callers can pass the same low-level request to `buildCapabilitySelectionTasks`
+and `ModelCapabilitySelector.select`; the helper derives its hashes from that request's
+configured client and limits, without a provider call or manual fingerprint
+construction.
+
+The selector returns decisions; application argument completion and runtime execution
+remain separate. Native search and `routerFor` never call the dedicated selector.
+Selected-result routers, the optional Responses adapter, and managed full-run accounting
+are available; see the shared-run integration below. Imports use the public
+release-candidate discovery subpath.
+
+Run the explicit selector companion:
+
+```sh
+npm run build
+node examples/12-capability-discovery-selection/run-selector-mock.mjs
+```
+
+It demonstrates competition, single-candidate abstention, a skipped no-match task, and
+one enabled repair. Across two explicit invocations it reports three mock provider
+attempts and 420 synthetic total tokens, with complete reported usage. It executes no
+tools and makes no embedding or live model calls. The fixture establishes contracts, not
+model quality.
+
+## Advanced lexical example
+
+Run from the repository root:
+
+```sh
+npm run build
+node examples/12-capability-discovery-selection/run-mock.mjs
+```
+
+This example reuses the synthetic support manifests. Discovery returns independent
+shortlists for four tasks, then the router turns one request into a bounded runtime
+slice. Existing provider export checks produce the tool descriptors. A mocked ordinary
+model/tool loop chooses the ticket lookup, executes the existing authorized runtime
+handler, and returns an answer.
+
+Discovery makes zero provider calls and reports complete zero token and cost totals. The
+two ordinary mock model turns run locally. No key, network connection, embedding client,
+or dedicated selector is used.
+
+Imports use supported package subpaths. The executable retrieval fixtures include
+compatible and forbidden capabilities without adding manifests to the normal inventory.
+
+## Hybrid mock
+
+Run the explicit index build and hybrid example:
+
+```sh
+npm run build
+node examples/12-capability-discovery-selection/run-hybrid-mock.mjs
+```
+
+This flow builds deterministic synthetic vectors, validates the complete index, and
+round-trips it through host-owned memory storage. It demonstrates a semantic match with
+zero lexical evidence, an invalid-index rejection before any query embedding, and
+explicit lexical fallback. The ordinary mocked model/tool loop still performs tool
+selection and an authorized read.
+
+The flow records three logical mock embedding attempts: one document batch and two
+independent query calls, including the query made by the ordinary runtime loop. All
+embedding usage is labeled mock. These vectors provide contract evidence only. No live
+embedding service or dedicated selector is used.
+
+Hybrid callers supply an embedding client, complete model configuration, an index,
+artifact and input limits, an absolute deadline, and a shared reservation ledger. Each
+task joins its original fragment texts with newlines and makes at most one embedding
+attempt, sequentially. The native-router wrapper forwards the caller's ledger and
+deadline without changing ordinary runtime runners.
+
+Storage implementations must atomically replace complete validated artifacts. Serving
+does not load or rebuild an index implicitly. Stale source, model, revision, dimension,
+or provenance bindings require explicit rebuilding or a configured whole-query lexical
+fallback. An absent model revision does not establish an immutable embedding space.
+
+Cancellation returns a fixed usage snapshot. If a provider attempt remains pending, its
+reservation stays held until its observed completion settles it; host recovery handles
+attempts that never finish. Clients enforce token ceilings and disable hidden retries.
+Unreported usage and cost remain unknown.
+
+## Shared-run integration
+
+Run `node examples/12-capability-discovery-selection/run-pipeline-mock.mjs` after
+`npm run build`. The companion runs lexical/native, lexical/dedicated, hybrid/native,
+and hybrid/dedicated paths through the existing tool loop. Each reads a ticket, carries
+its order ID into refund preparation, and produces a preview requiring approval. It
+makes no real provider calls.
+
+Create one optional `createAicfProviderRun` with the existing runtime run ID, a
+reservation budget, and an absolute deadline. Pass it to `search`; facade selection and
+result-bound routers recover that scope. The ordinary Responses runner accepts
+`providerExecution` with explicit model identity, provenance, input/output token
+ceilings, and an application-supplied complete-input token preflight. Continuation
+history is supplied to that preflight. No tokenizer, pricing, credentials, or model
+defaults are discovered.
+
+Dedicated selection remains optional:
+
+```js
+const batch = await discovery.select({
+  results,
+  client: selectionClient,
+  maxInputTokens: 4000,
+  maxOutputTokens: 1000
+});
+const router = discovery.routerFor(batch, task.id);
+```
+
+Here `results` came from a search using `providerRun`. Without a run, continue supplying
+the existing explicit `budget`. A batch can select several independent tasks, but each
+router exports only its own task's decision. Revalidation repeats neither ranking nor
+provider work. The ordinary model still supplies arguments; normal policy, validation,
+approval, and execution remain in force.
+
+`createOpenAISelectionClient` wraps an existing Responses-compatible client and requires
+a complete-input token preflight. It forwards cancellation and timeout, disables
+transport retries, and uses a versioned strict structured-output projection. Embeddings
+continue to use the caller's supplied client and separately built compatible index.
+
+`providerRun.attempt` lets application planning or response calls share the same ledger.
+Only calls made through the managed boundary are counted. The in-memory budget
+optionally limits output tokens and retains allowance for unknown usage; use a host
+ledger for distributed or durable enforcement. The runner preserves its legacy `usage`
+field and adds an all-turn `providerUsage` receipt for managed turns. The run snapshot
+combines unique attempts once across stages. Missing usage/cost remains unknown; mock
+zero cost in this example is explicitly supplied.
+
+New trace events contain bounded metadata and numeric receipts, including under
+content-capture settings. Run-local retention stops at 256 events and reports overflow.
+Hosts own trace retention and access. Hashes are not anonymization.
+
+`createDiscoverySnapshot` creates a bounded metadata snapshot for the existing replay
+extension. Snapshot validation checks bindings and membership; ranking replay
+additionally requires original query fixtures and, for hybrid retrieval, compatible host
+vector fixtures. It never calls a provider. Neither operation establishes model semantic
+correctness. Existing `router_only` replay requires an explicit discovery handler for
+these snapshots. Live replay remains separately enabled and budgeted.
+
+The companion reports five, seven, seven, and nine customer-run mock attempts
+respectively. Administrative index-build usage is reported separately. Failure cases
+cover clarification, abstention, permission/source revocation, budget exhaustion, and
+invalid-index rejection without query embedding. Synthetic responses and vectors
+establish integration behavior; they do not qualify retrieval or selection quality.
+
+## Installed package and evaluation
+
+Fake data: all requests, capabilities, identities, and outputs are synthetic. No secrets
+are needed. No live provider calls occur.
+
+Command from an installed consumer directory:
+
+```sh
+node node_modules/ai-capability-framework/examples/12-capability-discovery-selection/run-customer-mock.mjs
+```
+
+Expected output: one planning turn, four ordinary turns, one authorized read, one
+approval-required preparation, and zero commits. The separately invoked hybrid companion
+uses two mock embedding attempts. All five commands use supported public imports and
+resolve manifests relative to their own modules, independent of the working directory.
+
+Run `npm run eval:discovery:mock` from the repository, or run the packaged
+`run-evaluation-mock.mjs` in the same way, for the frozen 240-case lexical baseline. The
+suite has three domains, 120 train cases, 60 calibration cases, and 60 held-out cases,
+with scenario families confined to one partition. Independently scripted selection
+observations demonstrate selected, abstained, unavailable, and invalid-output outcomes.
+Their accounting is explicitly incomplete because no provider attempts were observed. No
+selector is invoked by the evaluator.
+
+The baseline retains retrieval misses and false positives; its rates are informational.
+Do not tune against the frozen held-out set. These fixtures do not qualify live models.
+See the [discovery guide](../../docs/discovery.md) for defaults, ownership, limits,
+budgets, replay boundaries, and evaluation metrics.
+
+## Optional live evaluation campaign
+
+The
+[offline regrading guide](../../docs/evals/discovery-live.md#regrade-saved-wording-offline)
+shows how to reassess saved wording without model access. It preserves frozen results,
+observed delivery and historical accounting, and writes a separate versioned report.
+
+`npm run eval:discovery:live` validates the corrected v3 suite and fixed-eight-candidate
+schedule with zero provider calls. `npm run eval:discovery:campaign:mock` exercises the
+complete campaign, durable accounting, three held-out repetitions, and 48 workflow runs
+with deterministic clients. It writes evidence under ignored `.aicf/` and makes no
+quality claim about real models.
+
+Paid execution requires `--live`, an explicit host configuration, compatible model
+clients, token preflights, matching offline acceptance evidence, and a pinned
+continuation of the original campaign budget. This command performs no paid work by
+default. The application owns response facts and validates model wording with a recorded
+deterministic fallback. Plans, terminal task outcomes and component grades are retained
+in ignored evidence. The original suite and all existing companions are preserved. See
+the [evaluation guide](../../docs/evals/discovery-live.md) for stages, resume behavior,
+limits, and interpretation.

package/examples/12-capability-discovery-selection/customer-cases.mjs

@@ -0,0 +1,52 @@
+// Application-owned planning fixtures; these are not additional manifests.
+export const customerRequest =
+  "Read ticket TCK-100 and prepare a refund for its order.";
+export const plannedTasks = [
+  {
+    id: "read-ticket",
+    text: "Read ticket TCK-100 for the refund review.",
+    dependsOn: []
+  },
+  {
+    id: "prepare-refund",
+    text: "Prepare a refund case for the order from support ticket TCK-100.",
+    dependsOn: ["read-ticket"]
+  }
+];
+export const taskExpectations = {
+  "read-ticket": {
+    compatibleIds: ["support.ticket.get"],
+    forbiddenIds: ["support.refund.commit_case"]
+  },
+  "prepare-refund": {
+    compatibleIds: ["support.refund.prepare_case"],
+    forbiddenIds: ["support.refund.commit_case"]
+  }
+};
+
+/** Example application validation, deliberately separate from framework discovery. */
+export function validatePlannedTasks(value) {
+  if (!Array.isArray(value) || !value.length || value.length > 8)
+    throw new Error("Invalid application task plan.");
+  const known = new Set();
+  return value.map((task) => {
+    if (
+      !task ||
+      typeof task !== "object" ||
+      Object.keys(task).some((key) => !["id", "text", "dependsOn"].includes(key)) ||
+      typeof task.id !== "string" ||
+      !task.id.trim() ||
+      known.has(task.id) ||
+      typeof task.text !== "string" ||
+      !task.text.trim() ||
+      task.text.length > 4000 ||
+      !Array.isArray(task.dependsOn) ||
+      task.dependsOn.some((id) => !known.has(id)) ||
+      new Set(task.dependsOn).size !== task.dependsOn.length
+    ) {
+      throw new Error("Invalid application task plan.");
+    }
+    known.add(task.id);
+    return { id: task.id, text: task.text, dependsOn: [...task.dependsOn] };
+  });
+}