ai-capability-framework 1.0.3 → 1.1.0

package/docs/evals/discovery-live.md

@@ -0,0 +1,199 @@
+# Evaluating discovery with real models
+
+For the completed campaign and its separate offline reassessment, see the
+[qualification report](discovery-qualification.md). Lexical/native is the recommended
+default. Hybrid remains experimental because its incremental-gain gate was not met.
+
+Discovery returns an eligible shortlist. Evaluate whether it contains a useful tool,
+whether the model chooses correctly or abstains, and whether the application completes
+the request. Keep these measurements separate.
+
+## Start without provider access
+
+```sh
+npm run eval:discovery:live
+```
+
+This validates the frozen fixture and prints its planned inventory without importing a
+host module, loading an SDK, reading credentials, or calling a provider.
+
+The original 240-case suite remains unchanged. Its v2 revision corrects domain schemas
+and lifecycle links and replaces held-out families. Revision 2.1 also aligns explicit
+held-out amounts and quantity changes with their application facts. Version 3 preserves
+all 180 train/calibration requests and labels, supplies verified preparation readiness
+facts, and introduces 60 fresh held-out cases in new families. The original twelve
+workflows remain development regressions; twelve new workflows form the qualification
+schedule. Historical versions and results remain unchanged. All versions load through
+ordinary manifest validation. Evaluation-only capabilities do not add manifests to the
+framework's normal inventory.
+
+Use explicit subsets without changing the suite binding:
+
+```ts
+const report = evaluateCapabilityDiscovery({
+  suite,
+  registry,
+  observations,
+  caseIds: calibrationCaseIds
+});
+```
+
+Reports retain the full suite fingerprint and add an order-independent scope
+fingerprint. Duplicate/foreign scope IDs and observations outside that scope are
+rejected. Scheduled cases with no observation remain missing; an empty denominator
+remains `null`.
+
+## Supply a host explicitly
+
+Live execution requires `--live` and `--config <ignored-host.json>`. The JSON
+configuration contains `campaignId`, `hostModule`, `sdkVersion`, `tokenizerIdentity`,
+`pricing`, an ignored `evidenceDirectory`, and `predecessor`. Keep credentials outside
+it. The predecessor pins the original campaign directory, fingerprint and exact SHA-256
+journal hash. A continuation inherits only its verified remaining allowances. One
+permanent atomic claim prevents multiple campaigns from spending the same remainder;
+matching resume is allowed, sibling continuations are refused.
+
+The host module exports `createHostInputs()`, returning an already configured `sdk`, a
+`tokenizer(text)` that counts `cl100k_base` tokens, and the matching `sdkVersion` and
+`tokenizerIdentity`. Install tokenizer tooling in the host environment. The framework
+package acquires no tokenizer or SDK dependency.
+
+Supply pricing fields `inputPerMillion`, `outputPerMillion`, `embeddingPerMillion`,
+`tokenCountPerRequest`, `premiumMultiplier`, and an ISO `checkedAt` within the last 24
+hours. Input pricing must conservatively cover cache writes; include applicable regional
+premiums and do not assume cache discounts. Rechecking identical rates does not change
+the campaign binding. Prices are caller-supplied, never inferred from credentials.
+
+This reference campaign fixes `gpt-6-luna` with reasoning disabled and
+`text-embedding-3-small` with 1,536 dimensions. These are evaluation settings, not
+framework defaults. The host explicitly uses Responses input counting for the complete
+request, including tools and schemas. Continuations replay full history without stored
+conversations. Token-count HTTP requests are recorded separately from generation usage.
+
+Before live work, retain matching offline acceptance evidence at
+`.aicf/discovery-implementation/phase-7a/offline-acceptance.json`. It binds source,
+suite, workflows, and passing build, typecheck, tests, validation, conformance, package,
+and mocked-campaign checks. Run `npm run eval:discovery:campaign:mock` for the complete
+provider-free campaign. Its synthetic choices and vectors cannot qualify a real model.
+
+## Run and resume stages
+
+```sh
+npm run eval:discovery:live -- --config .aicf/host.json --stage all --live
+```
+
+Individual stages are `index`, `train`, `baseline`, `freeze`, `qualify`, and
+`workflows`. `--stage report --config ...` reads matching evidence without loading the
+host or calling providers. The index is an explicit administrative build; serving stages
+never rebuild or repair it.
+
+Cases run sequentially with rotating mode order. This rerun fixes the shortlist at eight
+candidates and uses a four-mode calibration baseline; there is no K=4 stage. Fixture,
+prompt, application-contract, grading and schedule bindings are frozen before live work.
+Qualification runs the full suite once and held-out cases three times. Every failed
+trial remains in the report.
+
+Resume by using the same configuration and evidence directory. A campaign lock prevents
+concurrent owners. Journal records, artifact hashes, and bindings must match. Pending
+work after a crash is not automatically purchased again. Reconcile uncertain provider
+requests and outstanding reservations before continuing; do not delete evidence to
+manufacture a fresh pass. Lock removal requires verifying the previous owner has
+stopped.
+
+The reference caps are 10,000 generation/embedding attempts, 20,000 total HTTP requests,
+80 million input tokens, 10 million output tokens, and USD 25. All stages share those
+caps, including predecessor work. Capacity preflight checks the entire schedule against
+the remainder before loading live clients. Prior, current and cumulative charges are
+reported separately. Unknown usage retains reserved allowance and stays unknown in
+reports. Accounting failure stops paid work. This journal is a single-owner evaluation
+host, not a distributed production budget service.
+
+## Interpret qualification
+
+### Regrade saved wording offline
+
+The example-local regrader corrects bounded wording checks for equivalent dates,
+explicit negation and prospective actions. It assesses every saved workflow without
+rerunning planning, discovery, selection, tools or customer-response generation.
+
+Create an ignored input JSON file with these fields:
+
+- `schemaVersion`: `"1.0"`.
+- `campaignDirectory` and `reportPath`: paths to the completed campaign
+  journal/artifacts directory and original campaign-result JSON, relative to the input
+  file.
+- `campaignFingerprint`, `journalHash` and `reportHash`: pinned SHA-256 bindings. The
+  journal and report hashes cover their exact bytes and use the `sha256:` prefix.
+
+From the repository root, run:
+
+```sh
+node examples/12-capability-discovery-selection/run-evaluation-regrade.mjs \
+  --input .aicf/regrade-input.json --output .aicf/regrade-output
+```
+
+The output must be a separate ignored directory. Existing output is accepted only when
+it matches exactly. The command validates immutable artifact hashes, the complete
+version-3 schedule and fixtures, original report reproduction and settled accounting. It
+does not load host configuration, credentials, SDKs or provider clients, and never opens
+a campaign ledger. Missing or incompatible evidence is refused.
+
+`regrade.json` records original and revised decisions, each workflow's changed checks,
+checker/source versions and fingerprints. Historical provider usage remains historical;
+regrading usage is separately zero. Actual customer responses, tool receipts and
+fallback use remain unchanged. A corrected assessment can accept saved model wording
+even when the original application delivered a fallback. It cannot erase operational
+failures, unknown usage or unrelated grading failures.
+
+This is an offline reassessment of existing evidence, not another live qualification
+trial. Preserve the original report and report revisions separately. Retrieval,
+selection, latency, cohort and hybrid-promotion gates remain unchanged.
+
+### Qualification gates
+
+Each mode receives full-suite, held-out, cohort, latency, usage, and workflow results.
+Retrieval false positives differ from false model selections. An eligible competing
+candidate is not itself an authorization violation. Scores never authorize execution.
+
+The reference gates require zero safety violations, at least 95% compatible recall and
+selection accuracy, at least 90% selection coverage, at most 1% no-match false
+selection, and p95 ready-task latency at most 15 seconds. Every ready task has a
+30-second deadline. A critical-cohort decline greater than five percentage points fails
+comparison against that mode's declared calibration baseline, on the full pass and every
+held-out repetition. Each repetition must pass; pooled averages cannot hide one failed
+repetition.
+
+Hybrid also needs a measured gain over lexical retrieval at the same shortlist size and
+selection mechanism. Workflow grading checks authorized reads, dependencies, arguments,
+approval-required previews, and customer claims with deterministic handlers and checks.
+No real business action or commit occurs. Planning returns explicit read/prepare tasks
+or terminal clarification with no tasks. The original customer request, ready-task
+objective and authorized prerequisite receipts remain available as untrusted model
+input. Host policy checks each task's operation and verified preparation prerequisites.
+Revocation persists across the complete application run; blocked parent branches stop
+without purchasing dependent work.
+
+The application owns response facts: approval comes from prepared-action requirements,
+and missing access, clarification and unavailability remain distinct. It deduplicates
+summary IDs while retaining execution history. A bounded checker validates model wording
+against the synthetic facts; rejected wording is recorded and replaced by a
+deterministic response. A valid fallback can satisfy application delivery while model
+wording still fails. A post-tool provider error retains successful receipts, records
+degraded completion and unknown consumption, and never repeats the action.
+
+Reports separate retrieval, selection, planning/dependencies, authorization/lifecycle,
+application delivery, wording and operational reliability. An irrelevant authorized read
+remains an intent failure even when framework authorization worked correctly. All
+components contribute to the overall qualification conjunction. Workflow calls must meet
+the same usage-completeness requirements as standalone trials.
+
+Validated plans, task text, dependency associations, dispositions and response-check
+failures are bounded application evidence under the ignored campaign directory.
+Framework trace events remain metadata-only. The wording checker is specific to these
+synthetic fixtures, not a framework prose-verification API.
+
+Reports include counts, family coverage, and descriptive confidence intervals. Public
+synthetic families are reproducible, not an independent blind benchmark. Missing cases,
+unresolved accounting, and operational failures remain visible. A complete campaign may
+conclude that a mode is unqualified. No package default or production setting changes
+automatically.

package/docs/evals/discovery-qualification.md

@@ -0,0 +1,158 @@
+# Discovery qualification for 1.1.0-rc.1
+
+Stable 1.1.0 reuses these frozen observations without new live trials or retrieval
+changes. The [named lexical/native workflow](../api/stable-workflow.md) now has a stable
+compatibility commitment; the historical prerelease classification below describes the
+assessed release. Hybrid qualification is unchanged.
+
+Lexical retrieval with ordinary model tool selection (**lexical/native**) is the
+recommended default. Lexical/dedicated also qualified for the tested configuration; it
+remains an optional additional model call. Hybrid/native and hybrid/dedicated remain
+**experimental and opt-in** because they did not meet the incremental-gain gate.
+Discovery's API classification remains **release-candidate public**.
+
+These results qualify a configuration on an authored synthetic dataset. They do not
+establish general production quality, guarantee the first candidate, or authorize an
+action. Applications still own intent decomposition, dependencies, arguments and
+customer responses. Evaluate your own capability corpus and traffic before adoption.
+
+## Tested configuration and assessment history
+
+The campaign used suite v3: 240 cases across support, inventory and scheduling, with 80
+per domain and 120/60/60 train/calibration/held-out cases. There are 60 scenario
+families overall and 15 held-out families. Related paraphrases share a partition. The
+fresh held-out families were withheld from calibration; this is not an external blind
+benchmark, and published fixtures are now exposed regression material.
+
+The frozen configuration used eight candidates; gpt-6-luna with reasoning effort none
+for planning, selection and ordinary turns; text-embedding-3-small at 1,536 dimensions;
+hybrid weights 0.75 semantic/0.25 lexical; and unavailable on hybrid failure. These are
+campaign settings, not framework model defaults. Model access, explicit indexes and
+finite budgets remain caller-owned. Returned generation model identities matched the
+requested alias; an immutable model snapshot was not available.
+
+The full 240-case pass and three 60-case held-out passes were assessed independently,
+including each mode's declared calibration baseline and critical cohorts. Ready-task p95
+had to be at most 15 seconds, with a 30-second absolute deadline. Quality thresholds
+were 95% recall and selected-candidate accuracy, 90% coverage, at most 1% false
+selection on true no-match cases, and zero safety violations. Critical-cohort
+regressions greater than five percentage points failed the gate. No pooled average
+replaced a failed pass.
+
+The original workflow assessment was **40/48**, with all four modes unqualified. A
+separately versioned offline wording-checker correction reassessed the same saved
+evidence as **48/48**, or **12/12 per mode**. Seven date-rendering assessments and one
+negated-completion assessment changed. This was not a fresh live trial or a model
+improvement. The eight actual deterministic fallback responses, tool receipts,
+operational failures, original grades and accounting remain unchanged. The earlier,
+different campaign's 22/48 result was not regraded.
+
+## Full-suite results
+
+Accuracy is measured on selected answerable matched cases. Coverage retains missed and
+failed decisions in its denominator. Recall uses the suite's answerable cases. The
+remaining cases include true no-match and eligibility-exclusion scenarios.
+
+| Mode              | Compatible recall at 8 | Selected accuracy | Selection coverage | Ready-task p95 | Revised overall               |
+| ----------------- | ---------------------- | ----------------- | ------------------ | -------------- | ----------------------------- |
+| lexical/native    | 118/120                | 118/118           | 118/120            | 4.14 s         | Qualified                     |
+| lexical/dedicated | 118/120                | 118/118           | 118/120            | 6.15 s         | Qualified                     |
+| hybrid/native     | 120/120                | 120/120           | 120/120            | 4.42 s         | Unqualified: incremental gain |
+| hybrid/dedicated  | 120/120                | 117/117           | 117/120            | 5.98 s         | Unqualified: incremental gain |
+
+Case-level descriptive 95% Wilson intervals are below. Paraphrases within families are
+correlated; these are not family-independent uncertainty estimates or guarantees.
+
+| Mode              | Recall interval | Accuracy interval | Coverage interval |
+| ----------------- | --------------- | ----------------- | ----------------- |
+| lexical/native    | 94.13–99.54%    | 96.85–100.00%     | 94.13–99.54%      |
+| lexical/dedicated | 94.13–99.54%    | 96.85–100.00%     | 94.13–99.54%      |
+| hybrid/native     | 96.90–100.00%   | 96.90–100.00%     | 96.90–100.00%     |
+| hybrid/dedicated  | 96.90–100.00%   | 96.82–100.00%     | 92.91–99.15%      |
+
+Every mode made **0/48 false selections** on full-suite true no-match cases (95% Wilson
+interval 0–7.41%). Permission/lifecycle violations, policy-forbidden export, forbidden
+choices, read-to-mutation selections and commit exposure/execution were zero. An
+eligible competing read/prepare candidate in a shortlist is not an execution violation.
+
+Retrieval alone still produced candidates for **47/48 lexical** and **48/48 hybrid**
+true no-match cases. Models abstained or otherwise safely declined these requests. This
+high retrieval false-positive rate is a practical reason to retain ordinary model
+selection and clarification rather than automatically execute the highest-scoring tool.
+
+Full-suite selection outcomes were 118 selected/49 abstained/73 unperformed for each
+lexical mode; 120/48/72 for hybrid/native; and 117 selected/48 abstained/3 invalid
+outputs/72 unperformed for hybrid/dedicated. Unperformed means no model decision was
+requested; it is not fabricated abstention.
+
+## Held-out repetitions and workflows
+
+Each held-out pass had 60 cases, 15 families, 24 answerable cases and 12 true no-match
+cases. Recall was **24/24** for every mode in every pass. Columns show passes 1, 2 and 3
+in order. Consistency counts identical outcome/choice across repetitions, including
+non-answerable cases; it is reported separately from gate results.
+
+| Mode              | Coverage by pass    | Selected accuracy by pass | p95 seconds by pass | Consistent cases |
+| ----------------- | ------------------- | ------------------------- | ------------------- | ---------------- |
+| lexical/native    | 24/24; 24/24; 24/24 | 24/24; 24/24; 24/24       | 3.66; 3.64; 3.85    | 57/60            |
+| lexical/dedicated | 24/24; 24/24; 24/24 | 24/24; 24/24; 24/24       | 5.56; 6.22; 5.89    | 59/60            |
+| hybrid/native     | 24/24; 24/24; 24/24 | 24/24; 24/24; 24/24       | 4.70; 3.94; 4.25    | 59/60            |
+| hybrid/dedicated  | 23/24; 24/24; 23/24 | 23/23; 24/24; 23/23       | 6.39; 5.41; 6.02    | 58/60            |
+
+For 24/24, the 95% Wilson interval is 86.20–100%; for 23/24, 79.76–99.26%; and for
+23/23, 85.69–100%. False selection was 0/12 in every pass (interval 0–24.25%). All
+critical-cohort comparisons and ready-task latency gates passed independently.
+
+Hybrid/dedicated retained four invalid selector outputs: three in the full pass and one
+in held-out pass 1. A separate unavailable selector decision in held-out pass 3 also
+remains a failure in coverage. Neither outcome was replaced by a successful rerun.
+
+All modes passed the revised 12-workflow assessment. Across the 48 workflows, the
+application performed 24 authorized reads and 12 approval-required preparations, with no
+commit exposure. Application delivery was already 48/48 using eight observed fallbacks;
+the offline correction changed wording-dependent grading only.
+
+## Why hybrid is not promoted
+
+Promotion required at least two percentage points higher compatible recall, or at least
+20% lower model-input tokens at unchanged recall, against lexical retrieval with the
+same candidate limit and selection mechanism. The observed recall increase was **1.67
+percentage points** (120/120 versus 118/120).
+
+| Mode              | Full-pass model-input tokens |
+| ----------------- | ---------------------------- |
+| lexical/native    | 619,786                      |
+| lexical/dedicated | 445,346                      |
+| hybrid/native     | 715,689                      |
+| hybrid/dedicated  | 466,759                      |
+
+These model-input counts exclude query-embedding tokens. Hybrid increased model input
+instead of achieving the required reduction. Both hybrid modes therefore remain
+unqualified overall despite passing their matching, workflow and safety gates. No
+ranking weights, thresholds, aliases or labels were tuned to turn this into a pass.
+
+The current campaign recorded 3,097 generation/embedding attempts and 5,521 HTTP
+requests including token preflights; reported input/output tokens were 3,889,390/108,203
+with complete current-campaign usage. Earlier campaign evidence retains an unknown-usage
+provider failure and conservative charging. Stage receipts are not added again to
+overlapping run/campaign totals. Offline regrading and release verification purchase
+zero additional model calls.
+
+## Evidence bindings and reproduction
+
+The live transport receipts and original model wording remain private. The following
+safe bindings identify the frozen inputs and derived assessment:
+
+- Suite v3: `sha256:5e5f0cb1a848b733174e6e2e444e7c3bd8a90680fcd6009c7323d635c3605f69`.
+- Qualification workflows:
+  `sha256:8fabf71450f0d3b04a9d4b543d844bc94b0e70e20c698897db127a7d516cff61`.
+- Campaign: `sha256:187b918cea5aecc0ab4075f2770310a0eddc314bf02d5d147008db773e5b6b76`.
+- Original report bytes:
+  `sha256:052d0e1abb1d39f527cf0258510058ebd3bc4ef07ff557761df4d447c6ef607e`.
+- Revised report:
+  `sha256:f354d381035f6b28b21c3812d45ac69e14e5577c980f93dc53ba7b7a57d89def`.
+
+The correction uses synthetic-wording.v2 and workflow-wording-regrade.v1. See the
+[evaluation and offline-regrade commands](discovery-live.md) for reproducible
+provider-free checks and explicit, separately budgeted live execution. No-key examples
+validate integration contracts; they do not recreate these live observations.

package/docs/evals/overview.md

@@ -12,3 +12,12 @@ Read:
 - [Replay and trace-to-golden](replay-and-trace-to-golden.md)
 - [Live evals](../live-evals.md)
 - [EvalOps export interfaces](../evalops.md)
+
+## Discovery and selection evaluation
+
+Use [discovery evaluation](../discovery.md#offline-evaluation-and-quality) for frozen
+synthetic cases, compatible and forbidden ID sets, separate retrieval/selection metrics,
+and explicit accounting completeness. No-key CI validates contracts; model quality
+requires a separately configured evaluation. The
+[discovery live campaign](discovery-live.md) provides bounded, resumable evaluation with
+caller-supplied clients; its default command remains provider-free.

package/docs/getting-started/installation.md

@@ -1,21 +1,20 @@
 # Installation
 
-Install the public package:
+Use Node.js 20, 22 or 24. Install the stable package in your application:
 
-```bash
+```sh
 npm install ai-capability-framework
 ```
 
-For repository development:
+Version 1.1.0 includes the stable lexical/native search-to-tool workflow. The
+[complete quickstart](provider-neutral-quickstart.md) runs with deterministic mocks,
+without a provider SDK, API key, repository checkout or framework build.
 
-```bash
-npm ci
-npm run build
-npm run validate
-```
+Live calls require caller-supplied model access and explicit configuration. Optional
+SDKs are installed only when your chosen provider integration needs them.
 
-The default examples use fake data and do not call live model providers. Live provider
-tests are opt-in and require explicit environment variables documented in the provider
-guides.
+For reproducible deployments, pin the tested version in your lockfile. Existing
+prereleases remain available by explicit version; `next` is a separate channel.
 
-Next: [Quickstart](quickstart.md).
+[Framework repository development](../development.md) has separate install/build/check
+commands.

package/docs/getting-started/openai-quickstart.md

@@ -20,28 +20,39 @@ npm install openai
 Root and runtime imports remain lightweight:
 
 ```ts
-import { loadRegistryFromPath } from "ai-capability-framework";
+import {
+  loadManifests,
+  validateManifests,
+  buildRegistry
+} from "ai-capability-framework";
 import { runOpenAIResponses } from "ai-capability-framework/openai";
 ```
 
+Load and validate your manifests with `loadManifests` and `validateManifests`, then pass
+them to `buildRegistry`; the [complete quickstart](provider-neutral-quickstart.md) shows
+runnable code.
+
 ## Mock-Tested Flow
 
-Run the existing mock runtime tests:
+Run the installed no-key customer flow:
 
 ```bash
-npm run test:openai:mock
+node node_modules/ai-capability-framework/examples/12-capability-discovery-selection/run-customer-mock.mjs
 ```
 
 Expected output excerpt:
 
 ```text
-Test Files ... passed
+"ordinaryModelTurns": 4
 ```
 
 The OpenAI runtime exports only routed read/prepare tools. Commit tools remain
 host-controlled through AICF action lifecycle APIs.
 
-## Live Opt-In
+## Repository development and live opt-in
+
+The commands below are for an optional [framework source checkout](../development.md).
+Normal package consumers supply their own configured client and explicit model.
 
 Live tests are skipped unless you explicitly opt in: