@ccoalm/ccl-skills 0.1.0

package/dist/assets/marketplace/plugins/ccl-skills/skills/llm-inference-integration/references/llm-client-gateway.md

@@ -0,0 +1,156 @@
+# LLM Client Gateway
+
+## Boundary Shape
+
+- Put all LLM calls behind a small gateway/client, not inside handlers or domain services.
+- Accept typed messages, model selection, response mode, timeout, trace/request id, and optional tool schema.
+- Return a structured response with content, reasoning/think content when available, tool calls, token usage, finish reason, raw provider metadata when allowed, and classified error.
+- Keep provider credentials, URLs, and headers inside provider adapters or secret-backed config.
+
+## Provider Adapter Rules
+
+- Normalize each provider behind one internal request/response contract.
+- Key adapters by wire protocol, not by brand: providers speaking the same OpenAI-compatible protocol are configuration variants (base URL, model list, credentials) of one adapter, not new adapters; add an adapter only for a genuinely different wire protocol (e.g. Anthropic-native). Otherwise adapter count grows linearly with every onboarded brand. Shared transport does NOT mean shared policy: each provider still gets its own policy profile — error-class mapping locked with per-provider tests, capability/tool/safety flags, usage-accounting quirks — and "compatible" is a claim to verify per provider, not assume; when a provider's capability, auth, safety, streaming, or accounting semantics genuinely diverge, split the adapter rather than stretching config.
+- Do not mix incompatible API formats in the same fallback chain unless an adapter converts payloads and response semantics explicitly.
+- Maintain a per-provider/model parameter policy:
+  - remove unsupported streaming options;
+  - remove unsupported sampling parameters for reasoning models;
+  - preserve max output tokens or equivalent caps;
+  - validate response-format support before using strict JSON mode.
+- Treat silent fallback as dangerous: record the failed provider, error class, fallback provider, and final outcome.
+
+## Streaming
+
+- Use a single stream event envelope for content deltas, reasoning deltas, tool-call deltas, usage summary, completion, and error.
+- Parse SSE defensively: ignore empty lines, handle `[DONE]`, tolerate non-JSON keepalives, and surface malformed chunks as telemetry.
+- **A streaming response needs an inter-event idle timeout, separate from the total/request timeout.** The total deadline (below) bounds the *whole* call, but a stream can connect, emit a few tokens, then silently stall with the socket open and no further bytes — a wedged upstream or a dropped network path that never sends a close. The total timeout does not catch this *promptly* (it may be minutes long to allow legitimate long generations), and "stream closed before the terminal event" handling never fires because it never closes; without an inter-event idle timeout the turn is pinned until the total deadline — or forever if no total deadline is enforced. Wrap each read for the next event in an idle deadline; on expiry, cancel and treat it as a retryable stream failure (per the gateway's retry classification), not a hang. Get the details right or it misfires:
+  - **Reset on meaningful progress, not on any byte.** If you reset the idle timer on every SSE line, periodic comments/keepalives/heartbeats keep it alive forever while no actual content or terminal event arrives — a dead generation masked by liveness pings. Reset only on real progress events (content deltas, tool-call deltas, the terminal event); treat keepalives as "connection alive" but track a separate no-content-progress budget.
+  - **Size it well above normal token cadence, and make it configurable per model/operation.** Reasoning- or tool-heavy models legitimately pause tens of seconds (sometimes >60s) between tokens; a tight fixed idle timeout aborts valid work. Default generously and allow per-model/per-method override, separate from any time-to-first-token budget.
+  - **Terminal success wins the race atomically.** The terminal event can arrive as the idle timer fires; serialize the read-vs-timer completion so a completed stream is never cancelled-and-retried after success was already in hand (a false retry that duplicates the call).
+  - **One idempotent cancellation path; cancel the stream, not the connection; await cleanup before retry.** The idle timeout, the caller/total deadline, and a consumer disconnect can all abort the same stream concurrently. Route every abort through a single idempotent cancel that awaits reader cleanup before any retry, so two cancel paths don't race or leak a half-closed reader. Cancel the specific **request/stream context** and close *that* response body with a deadline — do not unconditionally *drain* it on an idle timeout, since draining a wedged stream can block until the same stalled upstream produces bytes (defeating the timeout); drain only when bounded and worth it for connection reuse. On a multiplexed transport (HTTP/2) cancel only the stream, not the shared socket, or you kill unrelated healthy in-flight streams; on a non-multiplexed connection (HTTP/1.1 keep-alive) there are no sibling streams, so closing the underlying socket is the normal way to abort a stuck read.
+  - **Back off on idle-timeout retries.** A stalling upstream (overload) makes *every* client hit the idle timeout at once; immediate retry multiplies the load into a storm. Count idle-timeout failures against the same bounded retry budget with jittered backoff, and honor retry-after / circuit-breaker signals — a stall is a load signal, not just a transport blip.
+  - **Measure idle only while waiting on the upstream read.** If the read loop is blocked pushing a delta to a slow downstream consumer (backpressure), that elapsed time is not upstream stall — counting it cancels a perfectly healthy stream. Start the idle clock when you begin awaiting the next upstream event and pause it while blocked on client/consumer delivery.
+  - **An idle timeout after partial progress is not freely retryable — it inherits the partial-stream rule.** Once the stream has emitted client-visible deltas or dispatched tool calls, a blind retry replays/duplicates them (the same double-dispatch hazard as any mid-stream failure). Apply the turn loop's uncommitted-partial discipline: only auto-retry an idle-timed-out stream that produced no committed progress/side effect, otherwise require a resumable or idempotent continuation rather than re-running the call. (See `agent-turn-lifecycle.md` — buffer uncommitted until the terminal event; dedupe dispatch by response + tool-call id.)
+  - **Stop TTFT/idle timers on every termination branch.** A timer stopped only on the success path — but not on error, cancel, fallback, or early-EOF branches — fires after the stream already ended and reports a false timeout against a finished call; audit every exit path when adding a termination branch. Stopping alone is not enough against a concurrent fire: the timer's callback must re-check terminal/generation state under the same lock or CAS that guards its side effect (a bare flag read can still lose the race; for channel timers, drain the fired channel before reuse) so a stale expiry racing the terminal event cannot cancel or re-dispatch a stream that already finished.
+  - **Only an *upstream* stall enters retry — a downstream/caller cancellation is terminal.** The same cancel path fires for "upstream went silent" (retryable per above) and "the consumer disconnected / caller deadline elapsed" (the client is gone — see the client-disconnect-cancels-upstream rule above). Classify the cancellation cause: replaying a generation and re-running tool work for a consumer that no longer exists is pure waste and duplicated side effects. Downstream/caller cancellation is terminal non-retryable; only upstream idle/transport failure feeds the retry policy.
+- If usage only appears in the final chunk, emit a final usage event and record it once.
+- Timeouts and client disconnects must cancel upstream requests; verify the disconnect→upstream-cancel path over the real transport, not only in unit tests with fake streams.
+- **When evolving the stream protocol, answer identity/lifecycle questions before designing event fields.** "What fields does the new event type carry" and "which existing identity/idempotency/billing model do these events bind to" are two different questions, and the second must be answered first: which answer/revision object the events belong to, what key their usage is accounted under, and whether a retry hides a provider-side billing fork. Event designs that skip this get reworked at review or implementation time.
+- **Answer-replacing stream events need explicit commit-point semantics — per plane, not one global commit.** For *answer visibility*, bind the revision's commit point to its first visible content delta: when cancel or degrade lands before commit, the previous answer must be preserved while the failure still surfaces to the caller, and caller-visible mutations (staged references/metadata) plus caller-side effects (tool dispatch) stay buffered until commit so a never-committed revision leaves no executed side effects to duplicate on replay. *Accounting is a separate plane*: provider-attempt usage/ledger records book per attempt under their own keys (per the usage rules below) regardless of whether the revision ever commits — an uncommitted revision still consumed provider tokens, and deferring or dropping its usage record is a billing hole, not tidiness. Post-commit failure semantics must also be an explicit spec decision, not an accident: when a revision commits (content already visible) and the stream later fails, either keep the partial replacement visible with the failure surfaced (progressive replacement) or roll back to the prior answer at terminal — name the choice and test it; leaving it implicit is how a partial revision silently clobbers a good answer. Define the semantics explicitly for streams with no content plane (tool-only, refusal, error-terminal): name which event commits them and how their outcome surfaces. This boundary is what chain-level replay tests force out — for stateful stream protocols, write chain-level (multi-event-sequence) tests at spec time, not after implementation.
+- **OpenAI Responses API supersedes Chat Completions as the recommended surface for new integrations** per `platform.openai.com/docs/guides/migrate-to-responses` and `developers.openai.com/blog/responses-api`. Two load-bearing differences: (a) **reasoning state preservation across turns** — Responses keeps the model's reasoning context alive between API calls, whereas Chat Completions drops it; OpenAI's published evals show ~3% SWE-bench Verified improvement vs Chat Completions with identical prompts on reasoning models, and ~40-80% better cache utilization. (b) **agentic-by-default**: a single API request can call multiple built-in tools (`web_search`, `image_generation`, `file_search`, `code_interpreter`, custom functions) in one round-trip — the orchestration that previously required client-side tool-loop scaffolding moves provider-side. **CRITICAL P0**: provider-side built-in tools bypass the team's own auth / audit / rate-limit / data-egress boundaries — calls to `web_search` happen at OpenAI's edge with OpenAI's network, calls to `file_search` index data the team uploaded to OpenAI's storage, calls to `code_interpreter` execute code in OpenAI's sandbox. Required: explicitly DISABLE the built-in tools in the Responses request unless a specific tool is approved for the use case (default-on is the wrong stance for any team with compliance / data-residency / audit obligations); when enabled, require post-response reconciliation that emits one audit event per built-in tool invocation (which tool, with what arguments, returning what footprint), enforce per-tool rate limits the team owns rather than relying on provider defaults, and document the data-egress policy (what user content can the team's prompts contain when web_search / file_search is on). Migration mechanics: structured-output config moved from `response_format` to `text.format`; `reasoning_effort` default is model-specific and drifts across model versions (see `model-prompt-evaluation.md`) — do not hardcode a flat default; read the current model's documented default per route. **When to migrate**: new integrations on GPT-5.x reasoning models default to Responses; existing Chat Completions integrations stay until reasoning-state preservation OR multi-tool orchestration is load-bearing — do not migrate as drive-by during feature work. Cross-provider abstraction layers (LangChain, LlamaIndex, an in-house adapter) need to surface the API choice explicitly because SDK shapes differ; verify the adapter speaks both forms before flipping callers.
+- **Prompt caching is a 2024-2025 industry-standard cost lever, not a niche optimization** — all three majors support it with different ergonomics. **Anthropic**: explicit `cache_control` blocks on prompt segments per `docs.anthropic.com/en/docs/build-with-claude/prompt-caching`; cache-read tokens are billed at ~10% of standard input price (NOT free — the 90% savings is the discount on those read tokens, not zero cost); cache-write is 1.25x base for 5-minute default TTL, 2x base for 1-hour TTL; the 1-hour TTL pays back vs uncached at roughly the third cache read once the write premium is amortized. Anthropic claims up to 90% cost savings on cache hits, but a workload that writes once and reads zero is a net loss. **OpenAI**: automatic caching on prompts ≥1024 tokens with no API change; Responses API improves cache utilization 40-80% vs Chat Completions per OpenAI's own announcement. **Google Gemini**: **implicit caching enabled by default for all Gemini 2.5+ models** per `developers.googleblog.com/en/gemini-2-5-models-now-support-implicit-caching/` — no API change needed; minimum cache-eligible request 1024 tokens (2.5 Flash) / 2048 tokens (2.5 Pro). **Architecture impact**: structure prompts so the cacheable prefix is stable (system prompt + tool/function defs + few-shot examples on top; per-request user content at the bottom). **Tenant-isolation P0**: cache keys / cacheable prefixes MUST be tenant-isolated — NEVER place tenant id, tenant-private context, tenant-specific tool definitions, or tenant-scoped few-shot examples inside the shared stable prefix unless the provider's account-isolation contract is verified AND tested (Anthropic and OpenAI account-scope caches per organization, but the moment two tenants share one API account / organization, a shared prefix that includes one tenant's context is a cross-tenant leak vector). Default position: tenant-scoped content goes BELOW the cache boundary; the stable prefix carries only tenant-neutral system prompts / tool defs / generic examples. Verify with a per-tenant cache-hit-rate audit — if tenant A's content hash ever produces a cache hit on tenant B's first request, the isolation is broken. **Footgun (Anthropic-specific)**: per Anthropic's docs, extended-thinking blocks interact with prompt caching in ways that invalidate cache entries more aggressively when the thinking state changes between turns; pin per-route extended-thinking decisions to avoid silent cost drift. For OpenAI / Google equivalents, this drop is plausible but not documented as a generalized rule at writing — measure cache-hit-rate per route before assuming the same dynamic. Treat cache-hit-rate as a per-route SLI emitted into the observability stack.
+- **Keeping the prefix stable means making cost-only prefix churn cache-aware — and never deferring an authority/correctness change to save a cache write.** Any mid-conversation change to the cached prefix invalidates the cache from that point and pays the full uncached prefill on the next turn (plus, on providers with explicit cache-write pricing, a fresh cache-write charge), silently every turn after if it keeps mutating. Split such changes into two classes and handle them differently:
+  - **Authority-neutral, cost-only churn** (re-ordering stable prefix content, refreshing an unchanged or non-authority system-prompt section, adding authority-neutral formatting/few-shot guidance that grants no new capability) — here the recommended pattern is **deferred invalidation**: don't rebuild the prefix as a side effect of an unrelated command; instead record the change as a durable `pending` mutation bound to principal/session/config-generation, apply it **before the next model request** (not at some far-off boundary, and never silently dropped — if applying it fails, fail closed to a degraded/visible state rather than continuing on stale-but-uncommitted intent), and offer an explicit opt-in (e.g. a `--now` / "apply immediately" path) to rebuild the prefix this turn.
+  - **Authority/correctness/capability-surface changes** — ANY change to the callable/visible tool or capability surface (add, remove, revoke, schema/trust/exposure change), plus policy/authorization changes, memory deletion, model/behavior changes, and any privacy/principal/tenant/auth change — must invalidate / fence / re-authorize **NOW, never deferred for cost**. These follow the fail-closed snapshot/re-verify/re-authorize rules above and in `references/retrieval-agent-safety.md` / `references/agent-tool-dispatch.md`, which take precedence over cost-driven deferral. Deferring a revocation that leaves a revoked tool callable for the rest of a long turn — or deferring a newly-added tool so the model plans around a surface that isn't really there — is a correctness/security bug, not a saving. The only other routinely-acceptable mid-conversation prefix change is the deliberate context compaction already governed above (which invalidates and re-writes by design).
+- **Structured output strict mode is supported by all three majors but with non-equivalent guarantees**. Anthropic: tool-use with JSON-schema definitions on `input_schema`. The provider enforces the tool-call SHAPE the model emits matches the declared `input_schema` at the wire level, but application-side validation against the schema is still required — output-token truncation can produce partial JSON, `oneOf`-style unions need explicit workaround, and edge-case constraint enforcement (`pattern` / `format` / nested `$ref`) varies; do not trust the provider as the only validator. OpenAI: `text.format` (Responses API) / `response_format` (Chat Completions) with `json_schema` strict mode per `developers.openai.com/api/docs/guides/structured-outputs` — strict mode guarantees the model output adheres to the schema, but only a subset of JSON Schema features is enforced (verify the docs page for the specific feature: `pattern` / `format` / `oneOf` / nested `$ref` support varies). Google Gemini: `responseMimeType: "application/json"` + `responseSchema` for JSON mode. **Per-feature audit before relying on strict mode**: write the schema, send a request with deliberately ambiguous prompt, verify the response shape matches AND the values are coherent (strict mode prevents shape violations, NOT semantic correctness — a model can still hallucinate field values that satisfy the schema). For high-risk extraction (billing / identity / permissions / money / quota), validate the output AGAIN with a second schema-aware parser at the application boundary; do not trust strict-mode output as the only validation layer. **Schema evolution discipline**: version the schema (carry a `schema_id` in both the prompt's schema declaration AND the model's expected output structure when the surface allows); treat adding a required field as a BREAKING change for downstream consumers including any cached responses or replay datasets; reject outputs whose `finish_reason` indicates `max_tokens` truncation as parse failures (truncated JSON looks valid until a key is cut mid-string) and retry-or-repair only against the raw captured response, not against a partially-parsed object that already dropped the truncation signal.
+- **Truncation on free-text / long-form output is a continue-or-raise decision, distinct from the structured-output case above.** When a *non-structured* response stops at the output-token cap the body is a valid prefix cut at the limit; do NOT return it as if complete. **The truncation signal is per-provider AND per-API — enumerate it for every surface you support**, since a detector hard-coded to one field silently accepts a capped result from another API as complete: OpenAI Chat Completions → `finish_reason: "length"`; OpenAI Responses API → `status: "incomplete"` with `incomplete_details.reason: "max_output_tokens"` (NOT a `finish_reason`); Anthropic → `stop_reason: "max_tokens"`; Google Gemini → `candidates[].finishReason: "MAX_TOKENS"`. On a streamed response read this signal from the *terminal* event (final chunk / `message_delta` / completion), never an intermediate delta — mid-stream the field is absent, so checking an early delta sees no truncation flag and returns the capped stream as complete. Recoveries (`platform.claude.com/docs/en/build-with-claude/handling-stop-reasons` maps `max_tokens` to "raise `max_tokens` or continue the response"): **(a) raise the output-cap and re-run** when the shortfall is small and the input leaves context headroom — the simplest fix, and the one to prefer because (b)'s seam is lossy; but a *full* rerun is safe only when the capped response committed no client-visible delta or dispatched tool call — after committed streaming progress it replays/duplicates them (per the partial-stream rule above), so there use continuation (b) or surface incompleteness instead; **(b) continue-and-concatenate** for genuinely long output — re-send the *complete original request state* — EITHER via the provider's response handle (e.g. OpenAI Responses `previous_response_id`, which chains prior input/output items so you do NOT replay the transcript — but it does NOT carry the request's `instructions`, so re-send the system/developer `instructions` alongside the handle) OR by manual replay of system/developer instructions, tool definitions, and prior turns — with the assistant partial appended and a "continue from where you left off" turn, then append each continuation to an accumulator; combining the handle AND manual replay duplicates context and cost, and the bare `[user, assistant-partial, continue]` triple is only the minimal single-turn illustration — dropping the original instructions/tools lets the continuation violate the original constraints or drift to a different answer. This is the **opposite** of the JSON rule above: a truncated *structured* body is a parse failure to reject/repair, never to splice across continuations (concatenating across a cut key produces invalid JSON); the same applies when the cap is hit mid-**tool-call** — a truncated tool-call / arguments block is structured output to repair or re-request, never fed through this free-text append loop (continuation is for text content only). Guardrails the naive loop omits: **the continuation loop only decides truncated-vs-not — it does NOT own the success predicate.** Continue only while the stop signal is a *truncation* one (the per-surface signals above); on any other stop, exit the loop and hand the response to the gateway's normal terminal-outcome classification (success vs refusal vs `tool_use` / `pause_turn` vs safety/recitation vs error — owned elsewhere in this skill), which must inspect the actual output content, not just the top-level finish/status field. Never equate "not truncated" with "complete successful answer": an OpenAI refusal arrives as `finish_reason: "stop"` + `message.refusal` (Chat) or `status: "completed"` with a `refusal` content item (Responses) — both pass a naive natural-finish check yet carry no answer text, so returning them as complete ships an empty accumulator and skips refusal/fallback/telemetry. A configured `stop_sequence` end is a genuine complete answer; refusal / safety / `tool_use` / `pause_turn` stops are terminal-but-not-a-final-answer and must route to their own handling, never be returned as the finished text; **bound resources, not just attempts** — a fixed max-attempts (e.g. 3) stops an infinite loop, but also hard-cap total input+output tokens / cost / wall-time and stop on a no-progress turn (an empty or near-empty continuation), because every continuation re-sends the whole growing transcript so input cost climbs super-linearly and can itself approach the window; **distinguish the output cap from the context wall** — Anthropic surfaces the window limit as a *separate* `model_context_window_exceeded` stop reason (model/version-dependent — default on recent models, beta header on older, and an input already over the window is an HTTP 400, not this reason), whereas OpenAI reports both the output cap and the window limit under the same `incomplete_details.reason: "max_output_tokens"`, so gate continuation on token-usage-vs-window rather than the reason string alone: once the transcript fills the window, continuing just re-hits the wall — stop and surface it / raise the window (a reasoning-model cap can also return truncated with *no visible output* because reasoning tokens consumed the budget, which a blind loop would append as an empty chunk); **the seam is semantic, not byte-exact** — "continue" can re-emit or skip tokens at the boundary, so validate the join for duplication/gaps rather than trusting a raw append, and do not summarize the carried partial (it removes the exact suffix the continuation must follow). When you stop without a complete answer, surface incompleteness **out of band** (a status flag / metadata / UI marker), NOT by concatenating a notice into the accumulated output — downstream persistence, evaluation, or a further model turn will otherwise treat that notice as model-generated text.
+- **Inline content-type markers within `content` deltas are a parser contract, not free-form text**: when the product needs the model to emit mixed-modality output inside one `content` stream — markdown plus structured renderable blocks (e.g. a card type, a citation block, a chart spec, a code-editable cell) — the markers MUST be a typed contract (well-named XML-style tags such as `<card-type-N>...</card-type-N>`, fenced code blocks with a typed language tag, JSON-mode segments delimited by a sentinel). The parser MUST be an incremental state machine over arbitrary chunk boundaries: SSE can split the opening delimiter, the tag name, attributes, a code fence, a JSON sentinel, or the closing delimiter across multiple chunks (e.g. one chunk ends `<car` and the next begins `d-type-1>`). A naive implementation that only buffers after a complete opening tag will emit the prefix as plain text and silently lose the structured block. Specify bounded buffering, partial-state carry-over per chunk, and explicit EOF/error behavior when a started block never closes. Free-form delimiters chosen ad-hoc per feature (one feature picks a short tag, another picks a bracket sentinel, a third nests JSON in markdown) fork the renderer per feature and break feature-cross-cutting concerns (copy, regenerate, history replay). Land the marker schema in the same contract artifact as the `tool-call` schema. **Unknown markers MUST surface to the renderer as escaped plain text / a text event, never as raw HTML or a renderable node, and surface telemetry with the payload redacted** — a permissive Markdown or `dangerouslySetInnerHTML` path turns an unknown custom tag with attributes into an XSS sink. Reasoning content is a separate channel: provider adapters MUST translate provider-specific reasoning tags (e.g. `<think>` blocks, `reasoning_content` fields) into reasoning stream events upstream of this content-marker parser; the renderable content-marker contract NEVER uses a reasoning-tag name as a content block, otherwise reasoning may leak into user-visible content or be persisted as user content by replay/history.
+
+## Reliability
+
+- Every request needs context/deadline, total timeout, per-attempt timeout, retry budget, and fallback budget. The per-attempt timeout must exist explicitly in the orchestration layer that runs the retry/fallback chain — an adapter's internal HTTP-client timeout is a private implementation detail, not an orchestration guarantee, and swapping the adapter must not silently remove the bound.
+- Retry only retryable network, timeout, rate-limit, and transient server failures.
+- Do not retry non-idempotent tool execution without an idempotency key.
+- Use bounded concurrency per route/model/provider and reject or queue when saturated.
+- Separate user-visible failure from best-effort telemetry persistence; failed call-record writes should not fail successful inference.
+
+## Observability
+
+Record at least:
+
+- caller or feature route;
+- request id / trace id;
+- model and model version;
+- prompt key and prompt version;
+- latency;
+- prompt/completion/reasoning/total tokens;
+- finish reason;
+- success/failure and error class;
+- fallback path;
+- tool calls requested/executed;
+- redacted payload and response where policy permits.
+
+Never log secrets, raw private user content, credentials, or unredacted files unless the product has an explicit data-retention policy and access control.
+
+## Generated Content Lifecycle
+
+When model output can affect users, publishing, permissions, billing, support, or downstream automation, treat the output as a lifecycle object rather than a string response.
+
+| State | Meaning | Allowed operations | Required metadata |
+| --- | --- | --- | --- |
+| candidate | Generated but not yet trusted or user accepted | preview, inspect sources, regenerate, edit, reject | model, prompt/policy version, request id, latency, token/cost, source/context summary, safety/error class |
+| reviewed | Human, rule, or evaluator has checked the candidate | accept, revise, compare, reject | reviewer/evaluator, review time, decision reason, changed fields |
+| accepted | User or workflow owner chose this output | publish, save, enqueue downstream action, rollback before publish when possible | accepting actor, accepted version, source candidate id |
+| published | Output is visible or has affected another system | view, audit, retract/rollback if supported | publication target, trace id, published version, rollback/retraction path |
+| rejected | Output must not proceed | regenerate from a new candidate or close | rejection reason, unsafe/low-quality class where relevant |
+
+Do not silently move from generated text to published final state on high-impact routes. The release gate should prove the candidate/review/accept/publish transitions, metadata visibility, rejection path, and audit or rollback evidence.
+
+## Inference-As-RPC (Service-To-Service Inference Calls)
+
+LLM clients are one kind of inference client; the broader pattern is service-to-service inference where one backend service (often Go or Java) calls an inference service (often Python over HTTP, sometimes a Triton native gRPC client). The rules below apply when the inference target is internal infrastructure, not a SaaS provider.
+
+- **Transport**: HTTP POST + JSON body is the common cross-language transport even when the inference server can speak gRPC or Triton-native; the trade-off is universal client support and easy debugging vs the protocol-level efficiency of native binary. Document the choice per service.
+- **URL construction**: derive the inference endpoint from a method-name → PSM (service identity) mapping plus a `predict` path helper, not by hard-coding the URL in every caller. A `GeneratePredictPath(method, version)` helper centralizes routing so a model migration is one change, not N.
+- **Hard-coded inference IPs (`<private-ip>:<port>`) in business code are a finding**, not a shortcut. Every inference target must be resolved through service discovery (NACOS/Consul/k8s service DNS); the discovery layer owns load balancing, health filtering, and the lane/canary route.
+- **Method-level timeout per model SLA**: each inference method has its own timeout (image feature extract ~90 s, image match correction ~120 s, OCR ~90 s, classification ~10 s). Choose the per-request timeout as `min(method-timeout, config-override, context-deadline)` so the caller never holds longer than the strictest applicable budget.
+- **Per-attempt context** must be a sub-context of the request context; the client cancels the in-flight HTTP call when the caller's deadline elapses. Long-running inference holding the socket past the caller's cancellation is a leak.
+- **Large payload routing**: when the inference call carries an image, document, or multi-MB binary, the client converts the in-memory object to a signed object-storage URL and sends the URL plus metadata, not the inline bytes. For backward compatibility, support base64 inline as a fallback with a clear size threshold. When the inference response is itself a binary (corrected image, mask, rendered overlay), the server side should return a signed URL when possible; if the contract requires inline bytes, do not strip them from the response (that breaks callers) — apply size limits at the logger / persistence layer so log bloat is bounded without changing the wire contract.
+- **Signed URL trust model**: a signed URL is a bearer credential plus a server-side fetch target — it is not a "free" payload mechanism. Architecture defines: (1) allowed bucket/host allowlist (the inference server refuses to fetch from a URL outside it — SSRF defense); (2) allowed object namespace / prefix per caller; (3) maximum expiry window (short — minutes, not hours); (4) HTTP method scope (GET only for fetch URLs, PUT only for upload URLs); (5) content-length and content-type caps validated server-side before processing; (6) checksum or `If-Match` digest validation against an expected hash when the registry supports it; (7) signed query parameters redacted from logs and trace attributes. A URL that fails any of these is rejected before the model is invoked.
+- **Authentication and authorization across internal inference calls**: "internal infrastructure" is not a security boundary. The inference RPC contract must specify caller identity (mTLS service identity, signed JWT, or platform-issued workload identity), caller authorization (allowlist of services / accounts permitted to call this model), tenant / resource authorization (the call carries a tenant id and the inference service confirms the caller is allowed to use that tenant's data), and quota enforcement (rate limit per caller / per model / per tenant). A model endpoint reachable by anyone who can discover it via service discovery is a finding.
+- **Discovery cache miss policy is per-route, not portfolio default**: read-only idempotent inference calls (classification, OCR on already-uploaded content) MAY fall back to a documented baseline lane on cache miss. **Canary / stress / shadow / tenant-sensitive / write-with-side-effects routes MUST fail closed** when the requested lane has no healthy instance — silent fallback to production can leak canary traffic, stress traffic, or one tenant's processing into another lane. Architecture records the per-route fallback policy explicitly; the default for unmarked routes is fail-closed.
+- **Trace propagation**: every outbound inference request carries W3C `traceparent` + `tracestate` (and `baggage` when needed) via the OpenTelemetry propagator's `inject`; the inference server reads them via `extract` so spans become children of the parent trace. In addition, the platform's request-id / log-id header (`X-Request-ID` / `X-Log-Id` or equivalent) travels alongside for log-level correlation. **Edge trust rules**: at the public trust boundary, the server regenerates `X-Request-ID` if the caller supplied one (or validates against a strict shape) so callers cannot spoof log correlation; internal hop-to-hop, preserve the id without rewriting. For `asyncio` / Ray / thread-pool boundaries, capture the OTel context at task creation via `opentelemetry.context.get_current()` and reattach inside the worker (`context.attach` / `context.detach`); without this, child-task spans become orphans of the parent trace.
+- **Cross-stack ctx key discipline**: define context-key constants in one shared package (e.g. `TrafficRequestIdCtxKey`, `BusinessShardCtxKey`, `ExperimentRouterCtxKey`); business code reads through typed accessors, not raw string keys. The set is part of the inference contract.
+- **Failure mapping**: HTTP 2xx + envelope `code != success` is a domain error; HTTP non-2xx is a transport error; both must be distinguishable in the caller's error type. A flattened generic `ServerError` that erases the inference reason makes the call chain undebuggable.
+- **Inference-specific error codes**: reserve a numeric range (or a typed error class) for inference-side failures (model not found, GPU OOM, input shape mismatch, timeout on inner model call) so business retry and circuit-breaker rules can act on them differently from generic transport errors.
+- **Bounded retry without circuit breaker is incomplete**: short retry loops (2-3 attempts with backoff) handle transient blips; sustained failure needs a circuit breaker per upstream / per model so the service can fail fast and shed load instead of amplifying the upstream incident.
+
+## Experiment And Traffic Routing For Inference
+
+Production inference often runs more than one model variant simultaneously. Encode the routing as data, not code:
+
+- **Experiment strategy types**: A/B test (fixed ratio), Canary (small percentage rolling up), Orthogonal (independent dimensions multiplied), Shadow (replay candidate alongside live without affecting users). Pick by goal; reuse one engine across all four.
+- **ExperimentRule shape**: traffic ratio + matcher condition (tenant / route / lane / user cohort) + target model/version. Rules sum to 1.0 within a route or fail validation; silent ratio drift is a release-time bug.
+- **Per-request experiment metadata**: record which experiment id, which variant, and which model version served each request. Evaluation joins to this metadata; without it, comparing model A vs B is guesswork.
+- **Quality + cost metrics** (accuracy, precision, recall, F1, custom domain score, token cost, latency p95) are first-class fields on the experiment result, not afterthoughts. `confidence_threshold` and `max_retry` per experiment let downstream services react to low-confidence outputs deterministically.
+- **Shadow inference** runs the candidate without returning its output to the user; persist both candidate and production output and diff offline. Shadow is the safest activation gate for high-impact changes — **but it doubles the GPU / object-storage / queue load**, so it needs its own budget. Architecture defines: (1) shadow traffic ratio (start at low single-digit %, ramp deliberately); (2) separate GPU / replica budget for shadow so it cannot starve production; (3) sampled rollout (not 100% mirror — pick representative traffic); (4) automatic cut-off when shadow error rate, latency p99, or cost-per-request exceeds the configured threshold; (5) explicit termination when the comparison concludes. A shadow that runs forever is a hidden production dependency.
+
+## Service Discovery, Lane Isolation, And Lifecycle
+
+The inference fleet often spans many service instances across many environments. The discovery layer is the integration boundary; the service-side discipline determines whether it works.
+
+- **Three-tier isolation**: service identity (PSM, project.service.kind format like `project.service.api` or `project.service.rpc`) → lane (online / canary / offline / stress) → instance (healthy / draining / down). A request resolves through all three before hitting an inference replica.
+- **PSM format is strict**: a three-segment dot-delimited identity (`project.service.kind`) parsed at client construction; mistyped PSMs fail at startup, not at first request.
+- **Lane derivation from environment**: read `LANE` (or the project's equivalent) at startup and tag every outbound and inbound request; in-cluster discovery filters by lane metadata so a canary call cannot fall back to production by accident.
+- **Three-stage instance lifecycle**: `register → heartbeat → graceful_shutdown`. Register only after readiness probe passes (model loaded, health endpoint green). Heartbeat at a bounded cadence with explicit failure handling. On SIGTERM, deregister first, wait for in-flight requests to drain (bounded), then exit. Hidden infinite loops, unbounded waits, or process-kill exits are anti-patterns.
+- **Discovery client behavior**: cache the resolved instance list with a short refresh interval (typically 5-15 s); filter to healthy + matching lane; load-balance across the survivors (round-robin or random); on cache miss fall back to a documented baseline (cross-cluster k8s service, baseline lane) before failing the call.
+- **Internal-SDK as a vehicle**: the discovery + transport + retry + tracing concerns live in one internal Python WHEEL (or Go module) that every inference caller imports. Each business service does not re-implement; new patterns land in the SDK and propagate through dependency updates.
+
+## Multi-Language SDK Parity
+
+When the same client/gateway SDK is mirrored across languages:
+
+- Use one language-agnostic case manifest (case id + description) as the parity contract; adding a shared behavior forces a linked edit in every language. Assert with relational invariants (manifest ⊆ covered AND covered ⊆ manifest), not count snapshots, so drift is named, not just detected.
+- Align parity guards to strength, not intent: a same-named guard that self-verifies in one language (AST scan of real tests) but is a hand-maintained string table in another looks equivalent while protecting unequally. At design time ask "does this guard fail the same way in each language?"
+- Mirror protocol semantics and boundary behavior, not language idioms: error-model or reserved-field implementations may diverge per language convention, but each divergence carries a comment explaining it.
+- Audit error-model field completeness across languages: one SDK exposing a raw-response escape hatch while another silently drops `retryable`/provider error code is a parity break, not an idiom difference.
+
+## Inference failure classification before retry
+
+Classify inference failures before retrying: transport blips, stale auth, retryable server capacity, explicit rate limits with reset time, subscription or quota exhaustion, entitlement denial, context-window overflow, user abort, and malformed request are different outcomes. Build this as a closed taxonomy with explicit retryable semantics, and lock each provider's mapping with tests against real captured error responses, before building any fallback/cooldown policy on top: the same terminal condition surfaces under different wire shapes per provider (context-window overflow as 413 or as 400 plus an overflow error code; balance/quota exhaustion as 402), and misclassifying such terminal client errors as retryable server errors is the most common gateway defect class — retries that can never succeed, burning quota and latency. Honor retry-after/reset directives with capped waits, bound total retry amplification across SDK, gateway, queue/job, and user-visible retry layers, and let background or non-user-visible calls fail fast during capacity cascades unless their result is required for safety or permission correctness. Subscription or quota exhaustion is terminal until explicit reset or renewed-quota evidence exists; entitlement denial is terminal until fresh entitlement authorization evidence exists; user abort is terminal unless a fresh user action restarts the request; malformed requests must not retry unchanged. After stale auth or stale connection, retry only if the full original tuple still matches: principal, account or tenant, workspace, privacy state, route/adapter, provider credential/client generation, provider account/project or API-key scope, quota bucket, billing namespace, region or data-residency endpoint, entitlement state/version, authorization or policy version, prompt or policy version, model generation/params, tool-schema or capability generation, session or job id, session incarnation, and canonical rendered request digest that includes policy/prompt version, model generation/params, and tool-schema or capability generation. On any tuple drift, cancel the pending retry or rebuild, re-authorize, and re-render from scratch. Persistent unattended retry needs abort handling, periodic progress/keep-alive, and a maximum reset horizon.
+
+## Conversation compaction and context reconstruction
+
+Treat conversation compaction, summary replacement, tool-result deletion, and session-memory compaction as model-visible state transformations, not token housekeeping. Every compaction path needs a durable boundary marker, preserved-segment relink metadata when raw messages survive, an explicit lossy-deletion policy, and post-compact cache invalidation for file reads, nested memory, prompt sections, classifier/speculative approvals, micro-compact state, and session-message caches. Summaries must preserve current user intent, pending tasks, unresolved errors, tool/action finality, permission and policy decisions, source-scope labels, files or skills needed for continuation, and known uncertainty; media, large tool outputs, or old turns may be replaced only with typed markers or bounded summaries that cannot be mistaken for fresh evidence. Prompt-too-long retry may drop old API-round groups only after keeping at least one summarizable group, inserting a synthetic continuation marker when needed, preserving tool-use/tool-result pairing and assistant thinking blocks, preserving or revalidating required authorization, privacy, policy, source ACL/provenance, current-intent, pending-task, and tool/action finality evidence, and recording that the summary is lossy; fail closed if the dropped groups contained the only copy of required evidence or unresolved finality that cannot be reconstructed. Session-memory compaction must wait briefly for in-flight extraction, fence every extraction by attempt id and context generation, reject late or stale extraction writes after boundary drift, session incarnation change, privacy opt-out, deletion, ignore-memory instruction, account/workspace switch, or capability/policy generation change, fall back when the summarized boundary is unknown or stale, keep enough recent text and token context under a max cap, avoid splitting tool pairs or merged assistant messages, remove stale compact boundaries from the kept segment, and fail over to ordinary compaction if the resulting context still exceeds threshold. Post-compact rehydration of files, plans, invoked skills, deferred tools, agent listings, external instructions, and hooks must be size-bounded, treated as untrusted context until revalidated, and bound to the full current authorization/source tuple before becoming model-visible: principal, account or tenant, workspace or repository identity, privacy or data-residency state, session id and incarnation, authorization or policy version, tool/source trust identity, capability or tool-schema generation, file and memory snapshot generation, and canonical source digest where available.
+
+## Fallback, cooldown, and degraded modes
+
+Fallback, cooldown, and degraded modes must preserve user and product semantics. Short retry windows may preserve cache locality; long or unknown waits should switch to an approved degraded path, visible refusal with a sanitized reason code, cooldown, or standard-capacity path instead of silently changing model class, quality, speed, or cost. Any fallback/cooldown/degraded transition that changes provider, route/adapter, model class or params, region or data-residency endpoint, quota bucket, billing namespace, provider credential/client generation, provider account/project or API-key scope, entitlement state/version, authorization or policy version, prompt or policy version, tool-schema or capability generation, or canonical rendered request digest must become a new authorized and rendered attempt with separate usage attribution, not a silent continuation. Fail open only for non-critical advisory policy fetches with a safe stale cache that matches the full authorization/targeting tuple, maximum stale age, and monotonic policy/revocation epoch or fresh deny/opt-out watermark; stale advisory data must never override a newer deny, revocation, privacy opt-out, logout, account switch, or entitlement change. Fail closed for safety, entitlement, privacy, permission, destructive-action, or policy gates when no fresh-enough allow exists.
+
+## Usage, latency, and call-record accounting
+
+Record token usage, latency, request success, finish reason, tool calls, retry attempt, retry-inclusive and retry-exclusive duration, fallback/cooldown/degraded-mode decision, unknown-cost markers, and caller/request id. Persist usage/cost only to the same full tuple used for retry/re-render, including principal, account or tenant, workspace, privacy/data-residency scope, route/adapter, provider credential/client generation, provider account/project or API-key scope, quota bucket, billing namespace, region or data-residency endpoint, entitlement state/version, authorization or policy version, prompt or policy version, model generation/params, tool-schema or capability generation, session or job id, session incarnation, and canonical rendered request digest; purge or invalidate restored counters after any member of that tuple changes. Deduplicate and finalize usage by request id, provider request id where available, idempotency key or usage-event id, and attempt id so retry, stream reconnect, and late provider completion paths cannot double-count or lose a terminal usage event. Use terminal ledger semantics for late, duplicate, retried, reconnected, streamed, and provider-completion usage events. Keep per-model/per-route accounting separate enough to explain quota, billing, or incident questions without leaking prompts or business data.
+
+## Prompt cache miss attribution
+
+Treat prompt cache miss detection as a diagnostic control plane, not as ordinary usage telemetry. A detector must take a pre-call snapshot of the rendered prompt/cache tuple before comparing post-call cache-read tokens: system/instruction digest, tool-schema digest, cache-control scope or TTL class, route/model/effort/output parameters, mode or feature toggles that affect the provider cache key, extra request-body digest, query source class, session or agent generation, principal/workspace/privacy tuple, and capability/tool generation. Fence the pre-call snapshot, post-call comparison, and baseline mutation by immutable request/attempt identity, prompt or message generation, abort generation, provider-response identity where available, session incarnation, and rendered-request digest; reject late, retried, reconnected, or aborted completions when any identity or generation no longer matches. Bound tracked sources and evict stale entries so background agents or short-lived sessions cannot grow unbounded memory or cross-contaminate attribution. Classify cache-read drops with explicit precedence and multi-cause support: expected drops from first call, known TTL windows, intentional cache-edit deletion, compaction or context reset, and baseline reset must suppress incidents; likely server-side or routing causes must not be over-attributed to prompt drift; rendered prompt/tool/parameter drift may be claimed only when the bound pre/post tuple proves it and higher-precedence expected-drop classes are excluded; otherwise emit a bounded `unknown` or `ambiguous` cause. Diagnostic events may expose only booleans, counts, bounded deltas, enum-like cause classes, and sanitized fixed-vocabulary tool/category labels; user-configured tool names, connector names, prompts, schemas, request bodies, local paths, credentials, raw request or session ids, and free-form errors must not enter telemetry. Any local diff or support artifact that compares rendered prompts, instructions, tool descriptions, schemas, or request bodies is a privileged support artifact: write it only under an approved local diagnostic directory with bounded size and retention, never upload it automatically, gate sharing on explicit support policy, label it as raw-sensitive, and keep only a sanitized pointer or artifact class in logs.

package/dist/assets/marketplace/plugins/ccl-skills/skills/llm-inference-integration/references/model-prompt-evaluation.md

@@ -0,0 +1,146 @@
+# Model, Prompt, And Evaluation Control
+
+## Model Registry
+
+Track models as controlled runtime assets:
+
+- model id, display name, type, provider adapter, base model, active version;
+- version id, parent version, status, changelog, activation time;
+- default parameters and parameter policy;
+- allowed use cases or routes;
+- deprecation and rollback policy.
+
+Avoid hard-coded model strings scattered through product code. Business logic should ask the registry/router for an active model by capability or route.
+
+## Model Version Baseline 2025-2026
+
+- **Three major-provider model families are the credible production options at 2026-Q2**; the registry/router MUST verify exact current model strings against the live docs page before pinning, because provider naming cadence has accelerated (Anthropic ships ~quarterly minor revs, OpenAI ships sub-version reasoning-effort variants, Google ships Pro/Flash/Flash-Lite tiers separately). Authoritative-doc sources for live verification: `docs.anthropic.com/en/docs/about-claude/models` (Anthropic) / `developers.openai.com/api/docs/changelog` + OpenAI Help Center model release notes / `ai.google.dev/gemini-api/docs/changelog` (Google). General shape: Anthropic ships Opus (frontier reasoning) + Sonnet (workhorse, 200K default + 1M beta context per `anthropic.com/news/1m-context`) + Haiku (cost/speed); OpenAI ships GPT-5.x with a `reasoning_effort` parameter (default value and available effort tiers are model-version-specific and drift — do not hardcode; verify per target model per the next bullet) plus reasoning summaries; Google ships Gemini 2.5 Pro + Flash + Flash-Lite with explicit thinking-budget control. Per-route choice is a registry decision recorded in the project's model registry, not hardcoded. **Pinning + eval-baseline discipline (P0)**: pin the exact model version string in the registry config (NOT just the family name like `claude-sonnet`); block ENV-var-only flips between model versions in production (flipping `MODEL=...` to a new minor rev silently invalidates every prompt eval result against the old version); require evaluation replay or shadow comparison for ANY model-string or default-param change before flipping production traffic; emit a metric on per-route model-string drift so an unintended autoupgrade is observable, not just discoverable at the next eval. The recurring failure is: team upgrades minor rev for cost/quality reason, eval suite still passes BECAUSE eval prompts didn't exercise the regressed code path, production sees the regression a week later.
+- **Extended thinking / reasoning effort is a per-request product decision, not an account-level default**. Anthropic's extended thinking (off by default) trades latency + token cost for accuracy on multi-step reasoning per `docs.anthropic.com/en/docs/build-with-claude/extended-thinking`; impacts prompt-caching efficiency (cache invalidation more aggressive with thinking on). OpenAI's `reasoning_effort` tunes the same axis per `developers.openai.com/api/docs/guides/reasoning`, but the **default differs by model version and is not monotonic** — defaults have varied across GPT-5.x revs (different revs ship different defaults; do not assume a trend) and new effort tiers (e.g. `xhigh`) appear in some revs. Do NOT hardcode a version-specific default here; confirm the default for the exact model you target against its live docs page before relying on "default behavior" — drift in this default is the most common silent cost/latency change across an OpenAI minor-version bump. `revalidate-when: OpenAI ships a new GPT-5.x reasoning rev`. Gemini exposes a thinking budget knob per `developers.googleblog.com/en/gemini-2-5-thinking-model-updates/`. **Pattern**: per-route declare reasoning intent (off / low / medium / high) based on the task's complexity AND record the decision in the prompt/route config so latency and cost regressions are traceable to an explicit knob, not provider-default drift. **Production-safety contract for reasoning enablement (P0)**: declaring reasoning intent is not enough — the per-route config MUST also carry (a) p95/p99 **latency budget** with downstream timeout propagation (reasoning can add seconds-to-tens-of-seconds to single-turn latency; downstream HTTP timeouts that worked at non-reasoning latency will fail), (b) **reasoning-token cap** to prevent runaway thinking burning 10-50× the non-reasoning token budget on a hard prompt, (c) **cost guardrail** as a per-request token + per-user-session aggregate ceiling, (d) **rollback path** to flip reasoning off at the route level under incident. Mid-route enablement of reasoning (turning thinking on after launch) is a load-bearing change that requires fresh load/eval evidence and SRE sign-off, NOT a config-only flip. Reasoning content is a separate channel from user-visible content — never persist reasoning into product history or display it as final output (see streaming rules in `llm-client-gateway.md`).
+
+## Prompt Registry
+
+Track prompts as versioned assets:
+
+- prompt key with optional system/type namespace;
+- version id and parent version;
+- status: draft, active, deprecated, archived;
+- content, variables, defaults, tags, author, changelog;
+- activation and rollback operation;
+- local file or DB-backed storage depending on product maturity.
+
+Handlers should render a named prompt version, not concatenate ad hoc prompt strings inline.
+
+### Prompt evolution as a first-class engineering doc
+
+A prompt registry tracks the **current** asset set; without a parallel evolution surface, "why does this prompt look the way it does" becomes git archaeology that decays as branches merge, rebase, or get force-pushed:
+
+- **Per-prompt evolution doc, keyed by stable IDs**: each materially-iterated prompt asset has a doc that lists, per significant version, `version-id → content-hash → landed-change-id → change-reason`. The primary key is a **stable monotonic prompt version id** (`v1, v2, ...`) — git commit hash and branch are NOT stable across rebase / squash / cherry-pick / branch-rename and must be treated as informational pointers, not identity. A content-hash (SHA-256 of the canonical prompt body) is the secondary identity that survives history rewrites; the landed-change id (PR number / merge-request id, NOT commit hash) links to the durable review record. Reasons are operational (the actual design change — a scoring-tier split, a new output field, a finer-grained output unit) not commit-message-restating. New columns are added when a new dimension matters (active branch, eval-result delta, rollback candidate).
+- **Plain-text version snapshots, with separate escape hatches for size vs sensitivity**: when a registry entry would otherwise drop the prior content on activation, persist the prior version under a stable path (`<repo>/docs/prompt_versions/<prompt-key>/<version-id>__<short-descriptor>.txt`). The filename combines the stable version id with a descriptor (never just one or the other; descriptor-only collides on similar changes, id-only is unreadable in PR review). Two distinct escape hatches:
+  - **Size-only** (large few-shot corpora, generated examples, prompts > ~50 KB, DB-backed prompt registries with already-versioned bodies): Git LFS pointer or the registry's own versioned backend is acceptable; the access audience is unchanged from the repo. Keep a `<version-id>__<short-descriptor>.ref` pointer file in `docs/prompt_versions/` that resolves the external location.
+  - **Sensitivity** (prompts referencing customer data, trade-secret rubric, partner-confidential instructions): the storage backend must enforce **strictly narrower access controls than the repo** — an artifact/registry store with per-identity authorization, audit log, and short-lived access tokens. Git LFS in the common setup grants any repo-cloner access to the LFS object and is NOT a sensitivity hatch; using LFS for a sensitive prompt moves the body out of text review while keeping the same audience, which is worse than the in-repo snapshot it replaces. Keep only a sanitized title/version-id in the repo pointer.
+
+  Repo-direct snapshots are for small reviewable non-sensitive prompts; the size hatch keeps the discipline workable as the asset grows; the sensitivity hatch is non-negotiable when the body crosses a confidentiality boundary.
+- **Branch fork tracking is explicit**: when one experiment branch keeps iterating a prompt while another stops at an earlier point, the evolution doc records "branch X stopped at commit Y" so a future engineer can pick the correct version when rebasing or cherry-picking. Multi-branch prompt forks are not theoretical — fix-branch + feat-branch each touching the same prompt is common, and silent fork is a real regression source.
+- **Evolution doc is updated in the same PR that ships the prompt change**: a prompt change without an evolution-doc append is incomplete; the doc append is part of the code review surface so the rationale is captured at the time the engineer remembers it, not reconstructed later from commit messages.
+- **Anti-pattern**: keeping prompt history only in git log + reading `git blame` to recover rationale. `git log -p consts/<prompt>.go` becomes unreadable after the first few iterations; rebases and squashes lose the intermediate states; and the change reason — the operationally important field — was never in git in the first place.
+
+### Automated prompt/artifact optimization — optimizer output is an untrusted candidate behind an adoption gate
+
+Beyond hand-iterating prompts, the prompts / skill files / tool descriptions / few-shot sets can be **treated as candidate text artifacts and optimized programmatically** — wrap the artifact in an optimizer + eval harness, let the optimizer propose variants and select against a metric. This can run at the prompt/instruction level with no model-weight training (API-cost only): prompt-optimizing frameworks (e.g. DSPy's prompt optimizers; trace-reflective evolutionary optimizers such as GEPA, which read the *execution traces* of failures to propose targeted mutations rather than mutating blindly). It is a real lever, but an optimizer maximizes its *metric*, not your intent — so an evolved variant is an **untrusted candidate** until it clears an adoption gate, and is never auto-deployed:
+
+- **Improve on a held-out split, and don't burn the holdout across runs.** Split eval data into train / val / **holdout**; the optimizer sees only train/val; report the gain on the untouched holdout. A gain only on the set the optimizer searched against is metric-overfitting (Goodhart) — the holdout must never enter the optimization loop. Repeated *adoption attempts* against the same holdout also consume it: run 30 variants and report only the winner's holdout gain and you have silently turned the holdout into validation (multiple-comparison). Track run/version counts and rotate or refresh the holdout; this is the anti-contamination rule below applied to optimization, not just to measurement.
+- **Pass a behavioral gate, not just the score or the code tests.** The variant must pass the owning repo's full test/eval suite AND a prompt-**behavioral** regression suite — code/schema/parser tests do not exercise prompt behavior, so a candidate can be green while it changes refusal behavior, tool-choice propensity, verbosity, escalation, or data handling. Include negative and security cases and tool-use traces. Plus size/token **and growth** limits (an optimizer will happily 3× the prompt for +1%) and structural integrity (valid frontmatter; tool schema — param names/types — unchanged). Failed gate = reject regardless of fitness.
+- **Gate on broad regression benchmarks the optimizer is NOT optimizing against — benchmarks are gates, not the fitness.** The fitness metric is task-specific (did *this* artifact do *its own* job better?); an optimizer maximizing it can silently regress unrelated capabilities or multi-turn coherence (Goodhart, one level up from the held-out point). Boundary so these don't collapse into "one eval suite": held-out catches same-task overfit, the behavioral suite catches *this* artifact's own contracts, the broad benchmark catches *other* capabilities. Run a broad capability/regression benchmark and a long-horizon / multi-turn coherence check as **separate veto gates** — with care:
+  - **Significance, not raw percent.** "Regressed −Y%" is a decision only with predeclared statistical handling (paired inputs, minimum N, CI/bootstrap, per-slice floors); a small delta is *inconclusive → rerun / enlarge sample*, not an automatic pass or reject (per Eval Reliability below).
+  - **Don't let the cheap tier become a second fitness.** Tier for cost — a cheap fast subset on every candidate — but use it as an early veto / stratified screen only, never as the finalist *ranking* objective (the optimizer will Goodhart the subset and breed cheap-set specialists). Pick finalists on primary fitness + slice coverage/diversity + predeclared constraints, and promote some diverse/random challengers so robust candidates aren't filtered out before the real gate; run the full benchmark + coherence check on those finalists.
+  - **The coherence gate needs a run contract, or it gets skipped.** Give the long-horizon check an owner, budget, timeout, sample shape, and evidence artifact; if it can't run, the variant is `blocked` / `deferred` with residual risk — never silently "passed" because the cheaper gates ran (per the blocked-verification discipline).
+  - **Benchmark, cheap subset, and coherence suite are eval assets** — version / use-count / rotate them (the cheap subset, run on every candidate, contaminates fastest); the anti-contamination rule below applies to them too.
+
+  A variant that improves fitness +X% but *significantly* regresses a broad gate is rejected — net-negative is not an improvement, and held-out-on-the-same-task does not catch a cross-capability regression. (This is the optimization-loop form of the layered-acceptance / "launch follows the product baseline, not the best component metric" rule — see `retrieval-agent-safety.md` composite-chain acceptance and `testing-strategy`'s composite-pipeline rule.)
+- **Security review is a mandatory gate, not an afterthought — the evolved text is a durable injection surface.** An optimizer maximizing a helpfulness metric will, unprompted, weaken a refusal boundary, add a prompt-injection affordance, ask the model to reveal hidden/system instructions, widen a data egress, or soften a safety caveat — and that text passes score + similarity + tests, then persists in a shipped artifact. Every candidate diff must clear an adversarial security review / scan for instruction-hierarchy violations, jailbreak/override language, secret-exfiltration patterns, and safety/data-boundary weakening (owned by `references/retrieval-agent-safety.md` safety + output rules); never let fitness substitute for it.
+- **Trace-derived eval data must be sanitized before it enters the optimizer, and the candidate diff audited for leakage.** When the eval/optimization set is built from real session traces, an optimizer (especially a trace-reflective one) can copy a customer name, token, internal URL, proprietary rubric, or prompt fragment straight into the "better" artifact — a durable PII/secret leak, worse than ordinary eval leakage. Only minimized/redacted traces may enter optimization; the candidate diff must pass a secret/PII/provenance-leakage audit before the PR (memory/trace hygiene per `references/retrieval-agent-safety.md`; trajectory-dataset hygiene per `inference-capacity-operations.md`).
+- **Contract preservation, not just semantic similarity.** Similarity against the original is auxiliary, not sufficient — it misses small critical edits (removing a negation, "do not expose secrets" → "expose secrets when an admin asks", narrowing a safety caveat, adding a higher-priority exception). Clause-diff the must/shall/never rules, safety caveats, data boundaries, tool preconditions, and escalation rules against the original; "better at a different job" or "same words, flipped a critical clause" is a regression. For tool descriptions this is its own gate: the schema may be fixed but the *description* still drives when the model calls the tool, what it sends, whether it honors caveats, and whether it overuses a privileged tool — preserve invocation boundaries, allowed/disallowed use cases, safety caveats, and required-confirmation language.
+- **Cache-stable adoption — but not for an authority/safety change.** An authority-neutral, cost-only evolved artifact (a clearer instruction, a better few-shot) deploys as a new version effective on the **next fresh session**, never hot-swapped mid-conversation — it lives in the cached prefix. But when the evolved change tightens a safety/refusal boundary, narrows a privilege, changes a tool's invocation boundary, or fixes a privacy/authorization surface, deferring it to "next session" leaves an active conversation running the stale, less-safe surface: those follow the immediate **invalidate / fence / re-authorize now** path, never deferred for cache stability (see the prefix-mutation rule in `llm-client-gateway.md`, which already splits authority-neutral churn from authority/safety/capability changes). If it can't be applied safely mid-conversation, hold or disable the affected route.
+- **Human PR with the right reviewers, never direct commit.** Land via a PR carrying before/after on train/val/**holdout**, the diff, the run cost, and the constraint violations caught-and-rejected during the run; for high-risk artifacts (system prompts, safety/refusal text, privileged tool descriptions) the security/privacy/eval owner must be a required reviewer — a human rubber-stamping the fitness delta is not the gate. Risk-gate per `feature-risk-router` / `product-rd-workflow`. The optimizer proposes; a human merges. When the evolved artifact is a **CCL skill/reference** (this section lists skill files as optimizable), this gate owns only the optimizer/eval/adoption mechanics — *landing* the change still goes through `skill-extraction-workflow`'s shared-skill gates (charter, R0 leakage audit, dual-track review + challenge, owner map); an optimizer's adoption gate does not replace them.
+
+The fitness function may be a scalar metric, unit tests / validators / schema checks, or LLM-as-judge (see Eval below); when it is LLM-as-judge it inherits all those reliability caveats (rubric drift, judge bias, position effects). A metric the optimizer can game is worse than no optimization.
+
+## Evaluation
+
+Every material model or prompt change should define:
+
+- dataset or replay source;
+- expected output, rubric, or comparator;
+- quality metrics such as accuracy, recall, consistency, parse success, or groundedness;
+- runtime metrics such as latency, token usage, success rate, fallback rate, and cost;
+- regression examples and human review notes when judgment is subjective.
+
+Store evaluation reports with the model/prompt versions compared so future changes can reproduce the decision.
+
+## Eval Reliability
+
+- Keep a stable golden set for regression checks and a rotating challenge set for newly discovered failures.
+- **Separate the fixed benchmark from the regression bad-case set, and govern both.** Mapping to the two sets named above: the **stable golden set IS the fixed benchmark** (answers "is the new version at least as good" under one stable comparison), and the **rotating challenge set is where the regression bad-case set accumulates** (real failures that must never recur). Track each set's version, source, time window, sample count, slice distribution, labeler, and labeling rule.
+- **Anti-contamination is the load-bearing rule: a set that has been used to train, fine-tune, tune prompts, or develop hand rules is contaminated and can no longer measure generalization — it must NOT serve as the fixed benchmark.** Benchmark/eval contamination is a top cause of inflated, misleading scores in LLM evaluation; reported gains evaporate in production because the model memorized the test. When reusing public or prior data, screen for leakage (exact-string / n-gram overlap / hash / embedding-similarity checks) before trusting a set as held-out, and prefer a benchmark refreshed/rotated over time so it does not ossify into a memorized target. Contamination is not only training: repeated inspection, manual error analysis, prompt iteration, threshold tuning, hand-rule writing, relabeling, or cherry-picking against a set all contaminate it for benchmark use ("we only looked at it" is still leakage). A contaminated set is **reclassified, retained, and versioned** as a dev / training / challenge asset — not deleted — and a fresh held-out benchmark is cut. Few-shot examples are allowed, but only sourced from non-benchmark / dev data, never from the fixed benchmark.
+- **Role separation for high-risk evals**: the party optimizing the model/prompt should not also own or silently edit the benchmark set — otherwise the bar quietly bends to pass. Benchmark changes (add/remove/relabel) go through a recorded change with reason, author, and reviewer, kept auditable; keep the optimizer and the benchmark-owner roles distinct where the launch decision is high-stakes. Small-team fallback: one person may both optimize and maintain the benchmark only if benchmark edits are append-only or asynchronously reviewed by another accountable person, and high-risk launch decisions still carry a recorded independent review.
+- **Continuous re-injection (the eval set is a living asset, not a one-time artifact)**: online failures, user-flagged bad outputs, and human-review findings feed back into the regression bad-case set, which runs every release. Tier the run so the gate stays affordable and honest: deterministic frozen cases run in the blocking release gate; cases that need live model calls / real retrieval index / integration run in the release / pre-ramp gate with an explicit marker, owner, and timeout; human-review-only cases produce release evidence and are NOT mislabeled as automated tests. Any **material change** re-runs the core eval before re-ramp — a passing eval from before the change does not transfer. Material = any change that can affect input distribution, retrieved context, model behavior, output schema/parser, tool availability, fallback/safety policy, or dependency/provider behavior (model version, prompt, retrieval, index, knowledge base, tool set, third-party service, or a provider default-behavior shift all qualify). If unsure, treat it as material; record any non-material classification with owner, reason, and affected surface.
+- Track sample size, dataset slice, evaluator version, and run timestamp with every report.
+- For LLM-as-judge, version the judge prompt/model, calibrate against human-reviewed examples, and watch for judge drift.
+- Prefer paired comparisons on the same inputs when comparing model or prompt versions.
+- Report both aggregate scores and concrete regression examples; aggregate-only evals hide product risk.
+- Treat small score changes as inconclusive unless variance and sample size justify the decision.
+
+## Production Output Review (Human-In-The-Loop Sampling)
+
+Offline eval proves a version at launch; it does not catch drift after launch. Continuous full-coverage human review does not scale, so sample deliberately — but sampling strategy must be risk-weighted, not uniform.
+
+- **Baseline**: review a small random fraction of production outputs (a common starting range is on the order of ~1–5%, or a fixed quota such as N traces/week — tune to volume and risk) for an unbiased quality view. Random-only is insufficient — it under-samples rare, novel, and high-impact failures.
+- **Priority + stratified on top of random**: actively pull outputs that users flagged negative, that an LLM-as-judge scored low, or that are novel/uncertain; stratify so each trace type, user segment, and risk class is covered, not just the high-volume happy path.
+- **High-risk output classes are not random-sampled.** For classes where a single wrong output is severe — financial-fact error, mistranslation, safety/red-line answer, an irreversible user action triggered by model output — define a dedicated review rate (up to 100% for the top tier), a processing time-limit, and a named owner. An LLM-as-judge may **prioritize, annotate, or route** these items — it must NOT exclude a high-risk item from the human-eligible queue, or the worst false negatives never reach a human. Keep an independent random/stratified human sample drawn from ALL high-risk outputs (not only judge-flagged ones), audit the count of judge-dropped cases, and run periodic false-negative spot checks. For these classes the human is the final authority on legal / brand / reputational / safety calls.
+- **Review budget + degradation path** (so "up to 100%" does not become an ignored ideal at volume): set a queue cap, an SLA, a sampling floor, and an escalation owner. If full review of the top tier is infeasible at the current volume, the honest responses are to reduce launch scope or obtain explicit product/compliance approval for the residual risk — never silently drop the high-risk review rate to zero.
+- **Close the loop**: every confirmed bad case from this review feeds the regression bad-case set in Eval Reliability above, so the same failure is caught automatically next release. Review without re-injection is wasted signal.
+- The launch/quality gate (which risk classes need this review, and the pass bar) is owned by `testing-strategy` (high-risk resilience) and `product-rd-workflow` (launch gate); this section owns the inference-side sampling + re-injection mechanics.
+
+## Replay And Shadow
+
+Use replay when existing request logs can safely represent expected traffic. Use shadow when candidate output should be generated alongside production without affecting users.
+
+For replay/shadow comparison:
+
+- filter sensitive or disallowed traffic;
+- preserve enough request context to reproduce behavior;
+- define comparator thresholds before running;
+- store diff examples, not only aggregate scores;
+- require explicit activation after comparison passes.
+
+Do not copy business-specific comparator labels into new products. Reuse the mechanics: input capture, candidate execution, scoring, diff retention, and rollout gate.
+
+### Freeze Comparison Criteria Before The Run (No Backfill)
+
+- Define the comparator, success thresholds, sample scope, observation window, and stop rules **before** executing replay/shadow/A-B/canary. Reading the online or candidate result first and then choosing the success criterion the result happens to clear is result-fitting, not evaluation — it invalidates the comparison. This is the same discipline as the anti-contamination rule above, applied to the success bar rather than the dataset.
+- **The freeze blocks moving the bar to PASS; it does not suppress a newly-discovered failure.** If a run surfaces a severe failure class the frozen rubric did not anticipate, that finding is not ignored because it was not predeclared — it becomes a launch blocker through a documented amendment plus a re-run on a fresh or re-scored sample. Forbidden: relaxing a predeclared criterion the result just missed. Required: adding a newly-found severe-failure blocker and re-validating.
+- A/B experiment and shadow/canary rollout answer different questions and their conclusions are not interchangeable. A/B measures *effect difference* (is the new version better on the business/quality metric) and needs a pre-declared randomization, sample size, and stop rule. Shadow/canary controls *rollout risk* (does the new version break under real traffic) and needs a pre-declared ramp step, pause condition, rollback condition, and fallback version. Do not present a risk-control ramp as proof of effect, or an effect experiment as proof of operational safety. A combined design that measures effect while guarding rollout is valid only when randomization, power/stop rules, ramp gates, rollback, and safety monitoring are all predeclared together.
+- **A continuously-monitored *statistical* comparison needs a sequential design, not repeated peeking at a fixed-horizon test.** This applies to inferential effect/quality comparisons (is the new version significantly better/worse). "Freeze before run" does not mean "look only once" — but checking a fixed-sample-size test repeatedly and stopping as soon as it looks significant (the peeking problem) inflates the false-positive rate far above the nominal level. So for an effect/quality test that is watched continuously, the stopping rule must be a pre-declared sequential method (alpha-spending or an anytime-valid confidence sequence) sized for repeated looks — not a fixed-horizon p-value glanced at every hour. Declare up front whether the comparison is fixed-horizon (analyze once at N) or sequential (continuous, alpha-adjusted); do not run fixed-horizon and then peek. This is NOT required for pure operational/SLO safety gates (error rate, latency, cost) — those use predeclared thresholds with minimum volume, persistence, and hysteresis (see `inference-capacity-operations.md`), not a hypothesis test.
+- The A/B-vs-canary rollout-strategy choice is owned by `product-rd-workflow` and `platform-release-engineering`; this section owns the freeze-before-run / no-backfill comparison mechanic.
+
+## Offline Evaluation Pipelines For Multi-Stage Inference
+
+When a production inference system spans multiple stages (e.g. OCR → matching → scoring) and evaluation needs end-to-end + per-stage signals:
+
+- **Pipeline as ordered scripts**, not just one notebook: `1_download_inputs.py` → `2_run_candidate.py` → `3_evaluate_metrics.py` → `4_emit_badcase_report.py`. Each script is idempotent on its own checkpoint; rerunning step 3 does not require redownloading.
+- **Database-backed inventory**: store the dataset row, the candidate output, and the evaluation metric in a relational store. The badcase report is a SQL projection, not a runtime artifact. Crashes mid-pipeline resume from the row's last-completed column.
+- **Skip-on-cache + strict-mode override**: the download/run scripts should detect prior completion via a stored marker column (`regulated_local_path`, `candidate_response_hash`) and skip; a `--strict` or `--force` flag re-runs from scratch when validating reproducibility.
+- **Badcase HTML report** is the human review surface; it should bundle input image / text, ground truth, candidate output, diff highlight, and a link back to the row id so reviewers can update the dataset. **Treat all model-emitted text as untrusted when rendering**: escape HTML entities on every model output before embedding into the report (`html.escape` in Python, or a templating engine with auto-escape on); render images via `<img>` with `Content-Security-Policy` or a sanitized origin so an OCR output containing `<script>` does not become stored XSS in the reviewer's browser. Reviewers open badcase reports from many sources; a sanitized report is the safe default.
+- **Scheduling**: cron or manual `python pipeline/N_step.py` is acceptable for evaluation pipelines (not user-facing); event-driven triggering is over-engineering when the input is a frozen dataset. Document the cadence. **Add a run lease so cron + manual cannot overlap**: each pipeline step acquires a per-step + per-dataset lease (database row lock, Redis `SET NX EX`, or filesystem flock) before doing work and releases it on terminal state. Two concurrent runs of the same step on the same dataset must result in one waiting / skipping, not double-running. Per-row idempotency lives below the lease: each dataset row carries a `processed_at` / `result_hash` so a partial-completion replay only retries the unfinished tail.
+- **Pipeline as evaluation artifact**: every model/prompt change cites the pipeline run id and links to the badcase report. Aggregate-only "F1 went up" reports without badcase examples hide regression risk.
+
+## Inference Quality Metrics Beyond LLM Defaults
+
+Multi-modal inference systems (OCR, detection, structure recognition, matching, marking) need metrics beyond the LLM-defaults of accuracy / token cost:
+
+- **Per-stage quality**: per-stage precision, recall, and F1 — pipeline-level F1 can mask a 30% recall drop in stage 3.
+- **Per-class quality** when the model returns class labels: aggregate F1 is meaningless if one important class regressed.
+- **Calibration metrics**: confidence score vs actual correctness — over-confident wrong outputs hurt downstream decisions even when accuracy is acceptable.
+- **Throughput-quality trade-off**: per-batch latency vs per-batch quality; some optimizations (smaller batch, lower precision) gain throughput at quality cost.
+- **Domain-specific rule pass-rate**: rule-based post-validators (output shape / range / cross-field constraint) catch issues that statistical metrics miss.
+- **Cost per successful output** (not just per request): a model that retries to success costs more than its base latency suggests.