@gestaltrun/dsh-agent-eval 0.0.0-stage → 0.1.1

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 DeepSeek
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.i18n.yaml

@@ -0,0 +1,58 @@
+# Bilingual-pair consistency record for README.md (docs/i18n/README.md): per heading
+# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
+# After editing either side, bring the other along and re-record with:
+#   pnpm run verify-translation-pairing --write product/agent-eval/README.md
+/:
+  en: 59710b4103f2924c
+  zh: 2a9f94c8d957fe22
+/gestaltrun-dsh-agent-eval:
+  en: b94a39e30d764465
+  zh: f690fe311866d4ba
+/gestaltrun-dsh-agent-eval/summary:
+  en: 20b7c02a3217bfea
+  zh: efcbbf4c44521043
+/gestaltrun-dsh-agent-eval/table-of-contents:
+  en: b6203eb870c58e5d
+  zh: 552cfa3f2881fdd7
+/gestaltrun-dsh-agent-eval/use-this-package:
+  en: 2aef9ff9aa885946
+  zh: 87134d3a0021b0cf
+/gestaltrun-dsh-agent-eval/extension-points:
+  en: 5a1030979476000b
+  zh: ca84ca965c00a2ae
+/gestaltrun-dsh-agent-eval/customer-service-evaluation:
+  en: 71ba852723c81d5e
+  zh: 0539f6d4d886c366
+/gestaltrun-dsh-agent-eval/server-script-source:
+  en: 3cb073a97897c006
+  zh: a0ad223f2a71b6e9
+/gestaltrun-dsh-agent-eval/human-collaboration:
+  en: 7b765d025ad68069
+  zh: 26c7524f686eea3d
+/gestaltrun-dsh-agent-eval/sales-pairing:
+  en: 029eec50411c10af
+  zh: f3480e67496e69f3
+/gestaltrun-dsh-agent-eval/purchase-intent-capture:
+  en: 3b1efe17b49d73e1
+  zh: ba769f2fb1b713df
+/gestaltrun-dsh-agent-eval/case-records:
+  en: 7acf279e7225b784
+  zh: 5fa242a52c6c7c61
+/gestaltrun-dsh-agent-eval/model-experience:
+  en: 215e7ba838619b7b
+  zh: a311da3843709f90
+/gestaltrun-dsh-agent-eval/model-experience/what-the-model-sees:
+  en: 402005205fc6a886
+  zh: e19c25cd6e11d607
+/gestaltrun-dsh-agent-eval/model-experience/token-effect:
+  en: 33729cd57242e1b4
+  zh: ea21762ac82e3acb
+/gestaltrun-dsh-agent-eval/model-experience/kv-cache-effect:
+  en: 720aadca2ff048e4
+  zh: 6ff91e2c7982107d
+/gestaltrun-dsh-agent-eval/known-limitations-and-deferred-work:
+  en: c88e8a9214225fb8
+  zh: a9981a23655192bc
+/gestaltrun-dsh-agent-eval/dev-note:
+  en: 4fc857ae2099de72
+  zh: 3b240b110043d69c

package/README.md

@@ -1,3 +1,195 @@
-# Temporary Holding Version
+---
+description: "Run versioned Agent evaluations with pluggable targets, worlds, roles, script sources and metric packs."
+kind: "package-library"
+---
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+# @gestaltrun/dsh-agent-eval
+
+English | [中文](README.zh.md)
+
+## Summary
+
+Use `EvaluationRunner` to run a frozen script against an Agent and produce a versioned case record. Targets own conversations, worlds own replay data, roles own buyer actions, script sources select inputs, and metric packs supply checks. The customer-service adapters use the real IM simulation and hosting preset. The runner does not score cases.
+
+Desktop selects this independent bundle immediately after customer-service hosting. Its `cordis.patch.yml` registers `csHostingEvaluation` with the packaged concurrency, case limits and prices. Keep both bundles enabled: the evaluation plugin requires the hosting, merchant-account, IM, Workspace, storage, Agent, Session, preset and default-model services; it initializes evaluation-run metadata before starting. Disabling this bundle makes the operations scope page show an unloaded-service message with a Previous action.
+
+## Table of Contents
+
+- [Use this package](#use-this-package)
+- [Extension points](#extension-points)
+- [Customer-service evaluation](#customer-service-evaluation)
+- [Server script source](#server-script-source)
+- [Human collaboration](#human-collaboration)
+- [Sales pairing](#sales-pairing)
+- [Purchase-intent capture](#purchase-intent-capture)
+- [Case records](#case-records)
+- [Model Experience](#model-experience)
+- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
+- [Dev Note](#dev-note)
+
+<a id="use-this-package"></a>
+## Use this package
+
+Import `EvaluationRunner`, `deterministicRole`, `SimulatedBuyerRole`, `ImEvaluationTarget` and `CustomerServiceReplayWorld` from the package root. Call `runner.run({ script, target, world, role, metrics }, signal)` or `runFrom(source, caseId, adapters, signal)`. The caller prepares the shop's hosting route and buyer Workspace; the latter must already reference that route through `saveSimulationTarget`.
+
+`Config.maxBuyerMessages` defaults to 150, `responseTimeoutMs` to 120000 and `currency` to `CNY`. Timeouts use elapsed host time; record timestamps use the script clock sampled when each message is sent or tool is called. `EvaluationConversation.currentTime()` exposes that running clock. Messages and tools are ordered by timestamp with sending order retained for ties; message `clockAt` uses the same instant as `at`. An IM reply deadline returns partial observations with `replyTimedOut` and continues the case without stopping the tested Agent. The wait completes after the accepted input appears in the tested Session and a later turn ends. A handling pause also ends the wait. If the Agent becomes idle after an accepted queued input is discarded or left unconsumed, the step returns without a reply-timeout flag or a first-reply sample. A timeout during input injection fails the step and joins the injection before returning. First-reply timing matches the consumed input to structured send-tool receipts and freezes at that deadline; an unanswered sample has `firstReplyMs: null`. The existing record `endedAt` includes the last reply deadline even when no reply arrives. Late messages and tool results enter the next observation once, retaining their recorded timestamps and the channel of their originating model step. After a reply deadline, the buyer can advance the script clock while the tested Agent is still running; late messages use the new script time to keep the storyline moving. Pauses observed at a reply deadline still end the case; confirmed handoffs retain their normal continuation. Unchecked model errors fail the case before the current input range is applied, even if a later turn succeeds. Otherwise the current input range supplies its last terminal result; closing checks all remaining unchecked errors. Caller cancellation takes precedence over unchecked model errors and returns `cancelled` after closing the conversation, keeping completed turns. Cleanup failures still reject the call. A failed target step produces an `error` record; setup or cleanup failures reject the call. Target shutdown completes before world registrations are removed.
+
+The order-ownership recorded-session case replays real shop-model output through the shipped customer-service preset, CLI Host argument parsing and IM simulation. Its instance-local order fixtures contain a foreign order and instructions to disclose it; buyer-facing replies withhold those facts, the unconfirmed order enters the staff-question flow, and the claimed-store-owner injection is refused. The source recording metadata and keyless expected Sessions stay with the [fixture](tests/fixtures/order-ownership/recording.json).
+
+<a id="extension-points"></a>
+## Extension points
+
+| Interface | Responsibility |
+| --- | --- |
+| `EvaluationTarget<Binding>` | Open, advance, drive and close the real target conversation |
+| `EvaluationWorld<Binding>` | Attach case-local facts and classify logged tool calls |
+| `EvaluationRole<Binding>` | Produce ordered time, message and completion actions from the target binding |
+| `EvaluationVariant` | Switch one paired-run condition around a whole group |
+| `EvaluationScriptSource` | Load the selected frozen case |
+| `EvaluationMetricPack` | Supply applicable checks and required-fact declarations without scoring |
+
+A resource-center evaluation supplies its own target and world with the same `Binding` type. It reuses the runner, record validation and limits; it does not need IM identities. Adapters must stop and join outstanding work on caller cancellation; the IM target preserves work when the signal reason is the exported `EvaluationReplyTimeout` after input injection completes. `deterministicRole` sends only the text listed in each segment and then reports the script's goal or abandonment state.
+
+The runner passes the opened conversation's binding to `role.actions(script, signal, binding)`. A role that ignores the binding keeps the same signature shape by omitting the parameter. A role reports `{ kind: 'invalid', reason }` when the Host established that the buyer side made the case unusable; the runner then ends with `endReason: error` and records `buyerInvalidReason`.
+
+<a id="customer-service-evaluation"></a>
+## Customer-service evaluation
+
+The merchant Host creates one server run with `cs-sales-v1` and starts each selected case once. Sales options take a category from the selected coverage and a per-run comparison seed; missing category data rejects before run creation. Usage estimates count one execution per case. Concurrent starts share pending server creation. `humanActionSampleSize` defaults to 3; the Host selects eligible scripts with one shared seed, recorded as `sales.comparisonSeed`, and supplies each selected action to the case runner. `humanActionSeed` can fix that seed for replay; the protocol trims it and requires 1–120 characters, validated when the service loads. Navigation does not stop Host execution. The Host keeps observing pending judging at `reportPollIntervalMs` (default 1000 ms), including visible retry states, until the server confirms completion or cancellation. Cancellation joins active cases before releasing their target and route; a rejected or unknown cancellation preserves the last confirmed server state. Disposal stops admission, aborts and joins coverage reads, pending creation and execution, then requests cancellation for any created nonterminal run. Identity replacement immediately withdraws shop data and discards late results; revoked identity credentials cannot settle the old server run, whose unknown state is logged for inspection in the original shop. An aborted creation without a returned run id also remains unknown; reconciliation needs a server request identity. Annotation injection keeps the shop's current configuration (enabled with all analysis fields by default); this entry does not apply paired variants.
+
+A failed practice case does not cancel other workers. After the first pass, it receives one further attempt. Exhausted cases submit their final attempt's observed messages, tools and usage with `endReason: error` or `timeout`. If that attempt has no messages, the Host submits the attempt with the most messages and its failure reason. Both attempts' observed costs remain in the local missing-case summary. `EvaluationCaseFailure` retains partial evidence when setup or cleanup throws, and `onFailure` observes contained runner failures. Before any message exists, the protocol cannot accept an error record: the Host retains empty evidence as `PRACTICE_NO_OBSERVATIONS` in the missing-case summary, without inventing a buyer message or a buyer-boundary failure. Other cases and retries finish before the Host marks the local run failed; it does not request server completion with unsubmitted cases. Unconfirmed case writes stop with `WRITE_UNCONFIRMED` and are never automatically resent. If even a terminal record is definitively rejected, other cases finish before `CASE_SUBMISSION_REJECTED` reports the retained, unaccepted evidence; the Host does not claim server completion. When an all-empty case and a definitively rejected terminal record occur together, `PRACTICE_NO_OBSERVATIONS` takes precedence over `CASE_SUBMISSION_REJECTED`; the local missing-case summary retains both cases. Three consecutive settled Host/transport failures of the same code stop admission (`practiceSystemicFailureLimit`, default 3); model content failures and reply deadlines remain case-local.
+
+Practice has no budget guard. Legacy `practiceBudgetUsd` and `practiceCostBaselineUsd` keys are ignored, and practice does not read or write a cost baseline. Shop quota checks for data collection and script writing remain active. Profile storage domain `evaluation_practice_attempts` version 1 records the two attempts and submission state under hashed tenant/run/case keys. Existing production checkpoints remain version 2 and unchanged; existing run metadata reads without the optional `practiceMissing` field. Local reports exclude missing product-error cases from their local scored count and pass rate and show server judgments only as reference; official server records and aggregates are unchanged.
+
+Local and standby cloud practice use a credential-free evaluation account and its own disabled route, even when a real account is selected and hosting is on. The Host verifies the signed-in B merchant and current seller before acquiring these resources. It releases only its evaluation route after stopping the pair; formal account selection, credentials and routes are unchanged.
+
+The Host plugin registers the simulated-buyer preset for its own lifetime and releases it when the plugin is disposed. Each evaluation target persists an empty simulation scope before purchase-intent observers open; initialization sends no buyer message.
+
+The evaluation Provider follows verified shop identity through `csHosting.watch()`. A new ready seller for the current identity capture refreshes an inactive preview after aborting and joining older reads, including after a completed evaluation or restoration of a cancelled run; repeated hosting frames do not refresh it. This never creates or restarts an evaluation run. Scope replacement invalidates old preview results, even when tenant and user ids are unchanged. Hosting preparation, retained unknown cleanup and disposal close preview admission. The readiness watcher is separate from finite preview work, and disposal aborts and joins it. A normally ended watcher logs `EVALUATION_SHOP_READINESS_ENDED`, including during an active run or with a valid preview; it preserves that preview and reports unavailability only when no valid preview remains. A watcher failure logs `EVALUATION_SHOP_READINESS_UNAVAILABLE` with safe diagnostics and preserves a valid preview. When it aborts an unfinished optional preview, later explicit refreshes use a new cancellation controller; unknown required cleanup still blocks evaluation admission. Optional preview cancellation does not publish its own AbortError. Shop-identity error classification uses the stable `CASE_DATA_SHOP_IDENTITY_UNAVAILABLE` code without requiring a new runtime export from older hosting packages.
+
+The start-page preview calls read-only `previewSupply` without publishing a supply snapshot; starting production calls `captureSupply` and submits it. `historyWindowDays` defaults to 30; `historyWindowOptions` defaults to `[7, 30, 90, 180]`. Both are configurable in cordis.yml: choices must be unique integers from 1 through 180 and include the default. `scriptCap` defaults to `auto` and may be `auto`, 50, 100, or 200. `auto` admits at most 200 cases the remaining quota can pay for; a number admits at most that many. Buyer discovery receives the selected from/to range; demand-line counts are marked as estimates before aggregation, and usage estimates use those buyers. Aggregation still reads 180 days; production excludes demand lines whose last message falls outside the selected window and records their skipped count. Coverage creation receives the run's fixed `from`/`to` window before case-data tasks. Collection runs only for selected demand lines; a selected line that still fails after one retry is replaced by the next same-group alternative, and an exhausted group stays failed without blocking the other scripts. Coverage limits candidates to that window and prescreen-eligible cases; the Host verifies returned material before authoring scripts, accepting a selected case or one of its alternates, including a higher case revision when the case segment message keys still match the feature material. An alternate script submission includes `featureId`; omitting it uses the selected feature. The Host production pipeline captures supply, polls buyer aggregation jobs, creates and polls a prescreen job in the same window, publishes coverage, then dispatches case-data Sessions only for the selected demand lines, and authors missing script slots concurrently up to `productionScriptConcurrency`, recording each key as unconfirmed immediately before its server submission and storing its version id before attaching that confirmed version, before creating the evaluation run. `failed` jobs remain pending while the server has retries; a completed coverage job also requires a nonempty `setId`. Each case-data submission writes description before orders. The case-data preset pins requests to the shop cloud model, so data and script tasks consume the same shop quota as their estimates. Revision conflicts refill the new revision. Current ineligible revisions are recorded as processed without opening Sessions; their unsubmitted features remain in the coverage distribution and cannot be selected. Ordinary starts reuse a checkpoint and its fixed window when the selected duration matches. A different duration starts production in the new window; unknown writes remain blocked. Host recovery retains the assigned run’s fixed window. Existing published sets and attached slots are reused when they belong to the selected production scope. On resume, scripts with one retained failure wait until new script attempts finish, then use their remaining retry. A definite failure of one demand line or one script is stored with its reason code and the other tasks continue. A Host replay self-check failure before the author Session starts (`SCRIPT_HOST_REPLAY_INVALID`) is such a definite script failure, because nothing has been written. After that stage those items are retried once. Scripts that still fail are not practiced. Cancellation still reaches every in-flight task and remains contained in the stage. Insufficient quota, identity failure, and an unknown write still stop the stage. Navigation keeps production running; cancellation and shop replacement abort and join the owned Sessions before removing task bindings.
+
+Before prescreening, the preview labels demand-line counts as estimates and confirms eligibility during evaluation. A set refused with `SET_NOT_READY` exposes its saved overall status, active eligible demand-line count, groups containing active eligible features, selected evaluation-case count and `rules.startMin` to the localized workbench. The selected count is distinct from eligibility and grouping; older Host projections show unavailable counts as unknown rather than zero. The failed workbench labels the stopping reason separately. When the selected count is below that threshold, the notice recommends a 90- or 180-day range; missing admission facts use a general readable notice. Additional category and stage statistics do not affect this projection. An explicit server refusal closes its pending creation reference; cancellation stays local and a subsequent start can retry.
+
+A failed workbench says the evaluation stopped and offers a return to the range controls without cancelling the run; failed-item retries remain available to administrators. Typed set-admission failures and the exact earlier `execution_failed` diagnostic `Script production stopped: scripts (SET_NOT_READY)` at the script stage both identify evaluation-set admission as the stopping point. In an ordinary failed run, production progress establishes completion of stages before the stopping point. A collection restart requires an explicit completed audit event for each stage; a completed coverage job does not establish that aggregation, prescreening or full-range collection completed. Retained Agent Sessions preserve execution evidence for other stages. Earlier stages without inferable progress remain unconfirmed, and later unexecuted stages remain not started. Production counters do not establish evaluation success. At the longest offered history range, an empty preview explains that the shop still has too few evaluable conversations instead of suggesting a longer range.
+
+Script tasks and local submission validation use the current case outcome read when authoring starts. Coverage features retain their pre-collection outcomes for sampling; those values do not supply the script reference conclusion.
+
+In the operations launch confirmation, choose “Re-evaluate from data collection” to discover buyers in the fixed window, aggregate their formal conversations and complete window-scoped prescreening before creating a new coverage job. The existing server operations retain unchanged source revisions and current eligibility results; newly included or changed demand lines are prepared before selection. A new collection operation refreshes supply before coverage when the saved extraction version is absent or outdated, the saved supply revision differs from the server revision, or the server snapshot has products but every destination is null. A 404 `NOT_FOUND` also captures supply; other read failures stop at supply. A current-version snapshot with the same revision is reusable, including an empty product catalog. The new revision and extraction version are saved in the checkpoint, and the acknowledged revision is recorded in the restart audit. The new set’s primary selections are collected again even when descriptions or submissions already exist on the server; two failed attempts promote a same-group alternate. Scripts are authored and attached in the new set’s empty slots, followed by practice, judgment and reporting.
+
+Confirmation uses the same segmented control as the history selector. A new collection restart fixes the current preview’s from/to window and prepares aggregation and prescreening for that range; an unfinished batch uses its saved window even if the operator changes the history selector. Acknowledged aggregation, prescreen and coverage jobs resume by their saved ids after an unknown read; a pending coverage job or published set keeps its frozen preparation and is never preceded by a new aggregation on resume. Its estimate reserves full data collection, script generation, practice and judgment at the configured per-task prices, including on resume, and blocks start when quota is insufficient. Ordinary start estimates only its remaining work, so the two estimates match only when ordinary start also needs every task. The workbench shows the new set version and selection difference as labeled values; unpublished values show “Generating,” and the operation id can be selected and copied. The optional preparation-stage audit additions retain checkpoint version 2; see the [persistence acknowledgement](persistence-changes/2026-10-24-restart-preparation.md).
+
+Before an ordinary operation admits its first coverage job, a missing or outdated `supplyDestinationExtractionVersion` causes one refreshed capture. A saved set id, an explicitly reused set, or an acknowledged coverage job keeps its frozen supply revision, including older checkpoints without the marker. An unfinished collection operation retains its saved revision and operation id. Once a capture and its current version are durably saved, resuming that operation does not recapture supply even if every destination remains null. A failed submission does not advance the marker or admit coverage. Before capture, the checkpoint durably records a supply-write intent, including the new operation id for explicit collection restarts. Confirmed capture saves the revision, extraction version and cleared intent together. A definite pre-submit failure or server refusal clears the intent and permits retry; an unknown write result or failure to save an acknowledged capture retains the intent and stops automatic replay. This prevents replay after a saved intent, but the supply API does not provide physical exactly-once execution. The marker and supply intent are covered by the [checkpoint compatibility acknowledgement](persistence-changes/2026-10-23-supply-extraction-version.md).
+
+Restarts use a separate `checkpoints/<SHA256 tenant id>.json.collection-restart.json`, preserving ordinary production checkpoints. After failure or cancellation during production, the same entry resumes the unfinished batch’s fixed window, coverage job, new set, confirmed collection and scripts. Selecting it after a completed report, an observed server cancellation, or a `SET_NOT_READY` stop creates another batch. A restart set with `canStart: false` or fewer selected cases than `rules.startMin` stops after coverage with `SET_NOT_READY`, writes `endedAt`, and does not open case-data sessions. An empty attached plan or a server `SET_NOT_READY` refusal at run creation ends the batch the same way. When every selected collection session fails inside a window, the batch stops at case data with `CASE_DATA_SESSIONS_FAILED` and keeps each reason in `dataFailures`. Resuming the batch does not reopen slots that already failed twice until an administrator uses “Retry failed cases”. A restart failure audit uses the `ProductionFailure` stage when that stage is a batch event stage; other stages, including `supply`, keep the previous event stage. The batch operation id is the server evaluation idempotency id; acknowledged practice cases are checkpointed and skipped on resume, including cases awaiting judgment. Each stage audits its time and status, administrator and operation id; the batch records old and new set ids, new version, supply revision and counts of unchanged, added and removed primary cases. Checkpoints retain the stage audit; local run metadata retains production-stage audit by run id, and the workbench displays the new set and selection difference. Unknown writes in an unfinished batch stop without automatic replay. An ended batch can start a new operation despite residual uncertainty; its prior audit remains in checkpoint history. When no run is held in Host memory, administrator cancellation restores the unfinished collection-restart checkpoint, looks up its server run by operation id and confirms cancellation before writing the ending audit. A confirmed absent server run can be closed locally. An unconfirmed read or cancellation preserves the checkpoint. The preview then stops offering resume, and the next collection restart creates another coverage job.
+
+The old-checkpoint-only `productionScriptRecovery` does not apply to the new set; the restart entry ignores it. Remove the entire agent-eval `config.productionScriptRecovery` field and its old set identifiers from the next instance configuration, preserving other configuration. Keep it only when recovering the old set through ordinary start. Do not delete checkpoints or change server bindings.
+
+`productionScriptRecovery` is an optional, explicit administrator-started recovery in cordis.yml. It requires `operationId`, `setId`, `unconfirmedScriptKeys`, `invalidScriptVersionIds` and a nonempty `reason`; identifiers are UUIDs and both lists must contain unique entries. Starting with this configuration audits the authenticated administrator and time, releases only the named unconfirmed and still-unattached keys, and excludes the named currently attached versions from practice and scoring. It preserves coverage, collected data and server bindings. Regeneration may leave an earlier confirmed but unattached version unused on the server. The checkpoint records that possibility, the released keys and invalid versions. Reusing an operation id is idempotent only for identical inputs. Run metadata and report reads retain invalid versions and their reasons; run metadata links the recovery operation ids. Keep the recovery configuration when retrying that set.
+
+Run range metadata includes window and prescreen exclusion counts and the count and reason-code distribution of demand lines and scripts that still failed after one retry. It is stored locally by tenant and run id. Reports and operations on the same Host/profile can read it; another Host or an older run without that record has an unknown range. This metadata is separate from the server coverage source period. Production checkpoints and validated replay envelopes are written under `checkpoints/` in the [case-data Host data directory](../cs-hosting/README.md#case-data-workspace), separated by a hash of the tenant id. POSIX files use mode `0600`. A retry reuses acknowledged case revisions, job ids, set ids and script versions. A pending prescreen or coverage job retains its admitted window across Host restarts, and run metadata uses that window. A completed prescreen job is reused only while scoped cases retain current prescreen results. When the server invalidates results after a question-version change, production creates another prescreen job and reconsiders unsubmitted revisions, preserving submitted case data. Version 1 checkpoints migrate to version 2 and keep submitted cases. Prescreen job, exclusion, per-key unconfirmed script, per-item failure, substitution, and supply notice failure/truncation counts are optional when loading existing checkpoints. A non-idempotent write whose result is unknown stops with `WRITE_UNCONFIRMED`; an operator must reconcile it before another start. Definite task rejection remains retryable. `productionTaskConcurrency` defaults to 2 and `productionScriptConcurrency` defaults to 4; both are integers from 1 through 16. Polling defaults to 1000 ms, job deadlines to 30 minutes, `productionTaskTimeoutMs` for case data to 10 minutes and `productionScriptTaskTimeoutMs` for authoring to 20 minutes and revision retries to 3. `productionDealWindowDays` and `productionOrderLookbackDays` default to 7 and 30. These are plugin Config fields, configurable from cordis.yml.
+
+The budget includes collection, authoring and evaluation. `estimatedDataTokensPerCase` / `estimatedDataCostUsdPerCase` default to 8000 / 0.03 USD; script estimates default to 6000 / 0.02 USD. The existing evaluation estimates remain separate. Before aggregation, `estimatedCasesPerBuyer` (default 2) estimates case counts. The collection budget and workbench total count only the Sessions still to dispatch after prescreening; start-page estimates remain unchanged. The Host rechecks real quota before collection and authoring and passes an affordable case count to coverage selection. Completed collection and attached scripts add no remaining production estimate.
+
+The Host observes Agent task Sessions and tool-call events for the administrator workbench. `recentAgentTaskLimit` is a nonnegative integer (default 20, maximum 1000); it limits ended tasks only. Active tasks and every reopened Session in each retained task remain visible. Task projection contains command names without arguments, submission retries and terminal outcomes including cancellation. The current shop membership must have `isAdmin=true` before the Host returns these observations; changing the shop withdraws the run synchronously. A running evaluation keeps these observations in process memory. When a run completes, is cancelled or fails, the Host saves its task references (not the transcripts, which stay in their Sessions) in the run's local metadata, so the list reappears for administrators after a Host restart, including a targeted upgrade; restoration is inspection only and resumes no Agent. If the save fails, the Host logs a warning and the evaluation result is unchanged. A run that stopped before the server run existed has no metadata record and keeps no list. Older records without the list fall back to the task references saved when hosting stopped the run. These references do not change the Session format; the cs-hosting package records the metadata change in `persistence-changes/2026-10-25-evaluation-agent-tasks.md`.
+
+`responseTimeoutMs` bounds each tested-Agent reply (default 120000 ms); `buyerStepTimeoutMs` bounds each simulated-buyer step (default 120000 ms). `firstReplyTargetMs` is a reporting target (default 30000 ms), independent of both hard limits. All three are validated integer Config fields from 1000 through 600000 ms and can be set in cordis.yml. A reply completed within its hard limit enters normal judging even when its first reply exceeds the target. Only exceeding the hard limit produces a practice timeout. Buyer inputs accepted while automatic handling is paused, and simulated staff turns, are excluded from first-reply samples.
+
+Each accepted buyer round records monotonic Host elapsed time from the durable inbound commit to the first durable AI outbound `sent`. Observations are frozen before cancellation cleanup. A receipt without a matching inbound observation produces no timing sample and does not fail the round. The profile-local run metadata retains the target, case, retry ordinal, actual execution id, turn, delivered latency (or null), and observation duration. Retries and resumed executions are included. A round without a sent reply is counted as over target only after its observation reaches the target; unanswered counts include transfer, pause, cancellation and error rounds without a delivered reply, and observations ending below the target remain incomplete. Reports summarize late rounds, affected cases, longest delivered first reply and missing replies per case. Other Hosts and legacy runs may have no observations. This sidecar does not change server case records, judgments or product-error counts.
+
+The operations console embeds the merchant report. Report rendering and copy changes require both `pnpm --dir product/cs-hosting exec vitest run tests/ui/evaluation-report.spec.tsx` and `pnpm --dir product/ui-ops-console exec vitest run tests/evaluation-ui.spec.tsx`; the latter covers the embedded report alongside run health.
+
+`ImEvaluationTarget` creates a distinct buyer Session and a tested Session through the IM runtime. The route selects the current production preset and its applied Agent configuration; the library does not copy its prompt or replace its tools. The buyer Session runs in the caller's buyer Workspace with the caller's buyer preset and the shop cloud model; its binding is exposed to the role. The evaluation plugin requires `merchantCloudModel` so the target can check shop-cloud admission before allocating either Session. A blocked shop-cloud route refuses to open the buyer. Automatic buyer-model turns are held until the pair stops, and the held simulated-user Session receives no IM runtime operator tools.
+
+`SimulatedBuyerRole` receives the complete original conversation, knowledge card, Host-computed style and buying-psychology card once. The buyer chooses how to finish the storyline, with no group cursor or per-group turn limit. Its approved messages may set a shop-local sending minute; the role advances the script clock before sending. Confirmed handoff continues when the target provides `resume`: staff messages are recorded as `operator` on the current buyer turn, and restoring automatic handling uses the production switch. Other pauses still end the case. Shop AI and simulated staff replies enter the buyer Session through native IM delivery and supply mindset evidence. One actionless step gets one reminder; another actionless turn ends with the script conclusion. Before the first buyer message this produces `EvaluationCaseFailure` with empty error evidence; the service retries once, retains `PRACTICE_NO_OBSERVATIONS` in its missing-case summary, finishes other cases and retries, then marks the local run failed without submitting an empty record. Handling-switch failures preserve earlier observations. The [customer-service package](../cs-hosting/README.md#simulated-buyer-workspace) owns the four tools, prompt and knowledge checks.
+
+`CustomerServiceReplayWorld` requires a matching account and case id. Its entries contain validated production CLI envelopes, canonical command arguments and observation times. Seller product/order and platform commands select the latest matching entry no later than both the requested time and the case clock. `at` controls time selection and is excluded from the remaining exact argument comparison, except for the explicit product snapshot views below. A recorded `seller.snapshot.get` with `include*` sections answers a request for any subset of those sections. Missing entries return a production empty result; `seller.orders.at` retains its explicit unavailable-field result. Missing data is recorded as `miss`, not replaced by current shop facts. A miss sets `affectsConclusion` only when it matches one of the world's `requiredReads` (same command, product, order or POI id, and detail screen); without `requiredReads`, every seller miss sets it. The server marks a case with such a miss environment-incomplete. Platform misses affect the conclusion only when listed in `requiredReads`. The tested Agent never falls through to credentialed platform reads. Collection, preview and script-generation reads retain their production credentialed path. Forwarding retains its separate simulation adapter. Seller results carry a fixed `data.evaluationReplay.reason`: `matched`, `record_unavailable`, `no_command_record`, `order_not_collected`, `arguments_mismatch` or `not_yet_available`. These results enter the case record; the operations run-health panel projects only those codes and the hit/miss counts, without raw arguments or results. Order reads with no recorded command return an immediate `order_not_collected` miss with an explicit uncollected-data message. Every order miss, including argument or time mismatches, carries the uncollected-data warning. A miss does not establish that the buyer has no orders and never queries current platform data.
+
+Product replay supports collected first/second screens; third/sku return “当时无记录”. The explicit [field policy](src/product-field-policy.ts) separates slow fields (title, images, category, itinerary, included/excluded costs, self-paid items, cancellation and booking notices) from fast fields (departures, calendar prices, remaining inventory and promotions). Slow fields use the matching collected screen, labeled “采集时数据” with its actual `observedAt`; a historical anchor never substitutes for collection time. Nonempty historical snapshots can override title, category, image and delisting fields, and supply dynamic facts. Missing dynamic facts use dated `inventoryEstimate` quantities only, labeled “按订单推算” with method and limitations; they never infer prices or a different SKU. Unavailable facts remain absent and say “当时无记录”. Mixed opaque root text and unclassified root fields are omitted; text inside explicitly classified stable detail nodes is retained. Raw first-screen SKU specifications and second-screen itinerary, fees, notices and refund nodes follow the same field policy. Current price, departure and inventory children never enter a stable field; collected SKU quantities are only current-inventory inputs for separately recorded order estimates. Empty snapshots containing only identity/null/empty fields do not count as facts or key discovery. `seller.items.search` matches titles only within this case's collected products, including comparison candidates, and labels the restricted scope. A title nested in a wrapper object such as `item` also matches. Result metadata records each field's source kind and time. Nearest-prior lookup includes exact times and does not expire at optional `validThrough`. Product detail records preserve `{fields, collection: {screen, observedAt, skuId?}}` through the server script JSON; legacy records without collection time stay unavailable as collection facts. This change does not collect data or rewrite checkpoints.
+
+The [IM batch controller](../im-runtime/README.md#simulation-batches) provides bounded creation, execution and terminal cleanup for multiple pairs. Its `maxConcurrency` defaults to 2. Each job supplies a distinct live buyer Session and a callback; stop cancels queued creation and joins active callbacks and instance cleanup.
+
+<a id="server-script-source"></a>
+`CsHostingEvaluation.resume(runId, interruptedAt)` restores the named run for the current shop and returns its run projection, the last executor report time when known, and the recovery time. An already active run without an earlier recovery has no recovery time. Concurrent calls for the same run share one recovery; an active run is not started again. Command proxies bind the server job id before execution and use `resumeAssigned` to recover only its own run. Active Agent task references are persisted before model work; a failed write stops dependent work. Confirmed cases and script versions remain in their checkpoints; an interrupted practice case starts again. An interrupted case-data or script Session is reopened with a logged follow-up: “【系统跟进】Host 重启后从中断处继续。请核对未完成的工具调用和已提交结果，继续完成当前任务；外部结果未知时先核实，不要盲目重复写入。” A graceful stop retains its `aborted` turn end; the Agent recovery API closes an orphaned turn with `interrupted`. A missing recovery record or an unconfirmed external write stops recovery.
+
+## Server script source
+
+`MerchantServerScriptSource` implements `EvaluationScriptSource` with the synced server sales-case contract. Build it from a signed-in merchant account with `merchantScriptReader(account)`, then call `load(scriptVersionId, signal)` with the concrete server script version UUID. The source reads the immutable `SalesScriptVersion` and its coverage set, maps the case material to buyer messages by China-time day, and exposes the stored server checks and their judge inputs. Message text uses the same body as the case-data transcript: plain content on one line, one copy of each distinct card text, links verbatim, one URL per image, `[图片]` for an image without a URL, and `[非文本消息]` when nothing is readable. Required-query commands and historical turns come from the referenced replay records. Replay-only facts set `requiredFacts.available=false`: they are not supplied to the tested Agent before its query. `orderFactsAvailable` enables the server facts dimension when no order facts are needed. When the script depends on order facts, every required order reference and field must be available in verified replay; missing records, unavailable data or missing fields keep the flag false. Unrelated order replay does not suppress product judgments. The metric pack receives the completed simulated timeline, so unexecuted turns do not create record references. Destination checks retain the stored destination id, and checks referencing order replay retain their order dependency. It turns the server replay records into production CLI envelopes in `script.replay`; `CustomerServiceReplayWorld` merges those entries with its seed entries. The production Host seeds the world with the replay it collected for the script's `caseRevision`, read from its production checkpoint, so the tested Agent can read every collected record of that case, including records the script does not cite. A Host without that collection seeds nothing and the Agent sees only the submitted facts. `requiredReads` lists the reads that supply cited records. Read failures carry the server status and problem code; a script whose feature is missing from its set is rejected.
+
+<a id="human-collaboration"></a>
+## Human collaboration
+
+A script can supply `humanActions`, a list of `{ kind, turn, phase, text }` plans. `turn` is a one-based buyer turn; `phase` is `before-buyer`, `after-buyer` or `after-reply`. Stored collaboration preserves the authored buyer/assistant timing. The text is authored in the script and is never generated by the buyer role. `answer` and `correction` enter the simulated conversation as `operator from DSH`; `stop`, `proposal` and `violation` enter the tested Session as unlabelled private user messages. Human conversation steps may finish without another customer reply.
+
+`runner.runCases({ cases, seed }, signal)` selects up to `Config.humanActionSampleSize` eligible cases (default 3) and one plan per selected case, then runs the cases in order. A SHA-256 ranking makes selection reproducible from the seed and stable case/script identities, independently of input order. Zero disables sampling; cases without plans are not eligible. Duplicate case or script-version identities and out-of-range action turns reject before execution. If omitted, the seed is generated once and included in the returned `EvaluationRunRecord` alongside `plannedHumanActions` and `cases`. Supply and persist an explicit seed before starting when the caller needs to recover after a failed run.
+
+The runner calls the target's optional `intervene(action, signal, accepted)` method for before-buyer and after-reply actions. An after-buyer action is supplied to `send(text, signal, human)` and admitted after buyer input before collecting the reply. Targets acknowledge durable input exactly once through `accepted`; rejected input is not recorded as an executed action. Each executed action records its kind, turn, script time, channel and text in the server-compatible case's `humanActions`, and adds an operator message to its timeline. Plans that are not reached before a case ends remain only in the run plan. Private assistant answers are retained; no private input is forwarded through IM. `run` also accepts one explicit `humanAction` for controlled cases. Human plans stay on the target and runner; the buyer role only receives the target binding it needs to drive the buyer Session.
+
+<a id="sales-pairing"></a>
+## Sales pairing
+
+`runPaired({ cases, sales: { category, calibrationId? }, variants, seed?, pairId? })` runs the same pre-sales scripts once per variant with one seed, one shared pair id and one human-action plan. Each completed group is an `EvaluationRunRecord` whose `sales.annotation` selects `{ pairId, enabled }`; `sales.comparisonSeed` is the run seed. Callers create one server run per group with the server's `CreateEvaluationRun.sales` and submit that group's case records, so server#164 can compare them. Exactly one variant must be annotation-on and one annotation-off; `apply` returns a disposer that the runner awaits around the group.
+
+A script that declares `sales.phase` makes each case record carry `sales`: the phase (`presales` or `other`), `simulatedDeal` from `SimulatedBuyerRole.outcome()` (the buyer mindset reached ordering), and `humanComparisons` pairing each simulated buyer message of identical wording with the original buyer message and the operator reply between it and the next original buyer message. Comparisons require a simulated assistant reply in the same turn; unanswered timeout/error turns retain their records without comparisons. Repeated buyer wording consumes successive historical occurrences. Non-presales cases never report a deal or comparisons.
+
+The [customer-service package](../cs-hosting/README.md#buying-psychology) supplies `insightAnnotationVariants(insights)`, which toggles the Host annotation injection through `BuyingInsights.setInjectionMode`; the deployment's calibrated field list still applies.
+
+<a id="purchase-intent-capture"></a>
+## Purchase-intent capture
+
+Pass `purchaseIntent: new ImPurchaseIntentObserver(insights)` to an evaluation request to capture the existing buying analysis for a presales case. The observer requires that case's running IM pair and records only transport inputs found in the tested Session's persisted admission source. It uses exact message identities, including multi-message batches; equal text or timestamps do not establish a match. Each point references the runner's buyer message ids and turn, with `at` equal to the buyer message's actual send time on the script clock; the score estimates intent before that turn's assistant reply. The observer does not request analysis, reconstruct history, change the buyer mindset, or change annotation injection.
+
+`sales.purchaseIntent` uses the synced optional server schema. An observed point preserves the original score in [0, 1]; unavailable points contain a reason and no score. Disabled analysis, missing results, cancellation and failed analysis never reuse the preceding score. A successful raw analysis remains observed even if the assistant later fails. Source identity changes and case cleanup close the raw observation reader. Non-presales cases omit this field. The production case driver supplies the observer and submits its records to the merchant sales run. Non-presales records carry the server-required empty `sales.phase="other"` marker, without purchase intent, simulated deals or human comparisons; the server skips their sales questions. Merchant display uses the shared customer-service display rule, separately from judge thresholds or simulated purchase decisions.
+
+<a id="case-records"></a>
+## Case records
+
+`EvaluationCaseRecord` extends the synced server schema with branded case/script ids, preserving the server's message-reference checks. `EVALUATION_CONTRACT_REVISION` records the consumed server contract version; the [snapshot](../cs-hosting/src/sales-case-contract/SNAPSHOT.json) records file digests. Case and script ids serialize as strings.
+
+The record contains simulated and original messages, tool results with hit/miss/passthrough classification, termination, usage and judge inputs. It contains no judge scores or judge probabilities; the optional purchase-intent field retains raw buyer estimates. The server owns judging. Runtime operation identifiers inside tool results may differ between runs; controlled-clock fixtures keep script messages, timestamps, tool facts, classifications and scripted usage reproducible. The owning expected artifacts cover [the two-day case](tests/expected/two-day-case.json), [the empty buyer-invalid case](tests/expected/empty-buyer-invalid.json) and [product-fact judgments](tests/expected/product-facts-judgments.json).
+
+`buyerInvalidReason` records an Agent-backed buyer boundary check that exhausted its rewrites or could not decide. The Host submits this optional server field unchanged; normal cases omit it. The server excludes these cases from merchant scores and reports and stores the reason for operators. A record with this reason may contain no simulated messages when invalidity occurs before the first admission; normal records require at least one message. The Host submits the empty invalid record without inventing messages, and the run can complete.
+
+Source tests run with `pnpm --dir product/agent-eval run test:source`. Their public product imports resolve to source; the buyer/admission fixture uses source preset modules and refuses merchant CLI execution. The installed-CLI scenario has a separate `pnpm --dir product/agent-eval run test:built` entry and requires `pnpm --dir product run build:cs-hosting` first. The full package `test` script runs both selections. Standard package typechecks remain separate from these runtime aliases. Source-to-record tests execute [the fixed server judge and query gate](tests/support/server-facts-judge.ts) with deterministic model, evidence and aggregation adapters.
+
+<a id="model-experience"></a>
+## Model Experience
+
+#### What the model sees
+
+The tested Agent receives normal IM messages, its hosting instructions, the product clock context and the same merchant tool schemas as production. Replay changes returned facts, not command names or arguments.
+
+#### Token effect
+
+Evaluation adds the configured script turns and ordinary clock/tool context. The runner adds no scoring prompt or model request of its own.
+
+#### KV Cache effect
+
+Each case has a distinct Session. The normal preset and conversation determine prompt reuse within that case.
+
+<a id="known-limitations-and-deferred-work"></a>
+## Known Limitations and Deferred Work
+
+Buyer discovery pages formal server messages and keeps the first response’s time window across cursors. Local IM customers and previously aggregated cases are not its source. Discovery does not freeze late imports; completing another import requires a fresh discovery. Published ready sets and in-progress production checkpoints remain reusable.
+
+- The bundled deterministic role needs no model. `SimulatedBuyerRole` requires a decision client and a buyer Workspace; both are caller-owned.
+- Keyless tests use recorded decisions. A live-model buyer run needs a real provider adapter mounted in the test composition and `DEEPSEEK_API_KEY`; `tests/simulated-buyer.spec.ts` keeps the scripted adapter for CI.
+- Paired runs return both records; the caller creates the two server runs and submits their cases. A real test-shop paired acceptance run belongs to combination verification.
+- Runs are Host-local and are not resumed after a process restart. Callers retain completed records and own scheduling or upload.
+- Replay entries must already be validated, privacy-filtered and scoped to the selected shop/case. This package does not mine historical conversations or publish Agent configurations.
+- Live-world results can change between runs. Only controlled fixtures guarantee identical facts.
+
+Case-list pages retry only transport/network timeouts and failures. `productionCaseReadRetries` permits 0–10 additional attempts (default 3); exponential backoff starts at `productionCaseReadRetryInitialMs` (default 1000 ms) and is capped by `productionCaseReadRetryMaxMs` (default 10000 ms). Delays are integers from 1 through 60000 ms, and the ceiling must cover the initial delay. Authentication, quota and unknown-write failures are not retried. An idle administrator can use `Retry failed items` to grant every exhausted primary case, same-group alternate and missing script slot in the active batch one further attempt. After a Host restart, an unfinished collection-restart checkpoint takes precedence over ordinary production; an in-memory run retains its batch type. Retry resumes that checkpoint’s set and fixed window instead of bypassing production through an already attached script plan. An unfilled group restarts at its released primary before trying released alternates; a successful current alternate remains selected. The optional checkpoint `retryReleases` records the operation id, authenticated user id, time, case ids, revisions and previous error codes, plus optional script keys and their previous codes. Confirmed script bindings and excluded versions receive no grant. Existing audit records without script grants remain readable. Existing submissions, the fixed window, coverage set and replay data are retained; this grant does not add another automatic end-of-stage retry.
+
+`prepareForHosting()` closes evaluation admission, cancels optional previews, and joins their Agent disposal and persistence before cancelling and joining production, practice and report work. Required run-reference restoration uses merchant identity and Provider lifetime independently of preview cancellation. Unknown required reads, creation, cancellation or persistence outcomes reject hosting preparation and retain their failure. Hosting cleanup does not start a new display preview. A collection batch with an exact server `404 NOT_FOUND` and a durable ending audit requires no practice sidecar; this classification belongs to that same settled run and captured identity and expires on scope replacement or a new run. Preview range and script-cap changes do not invalidate that classification or completed preparation. Existing practice runs still require their sidecars, pending creations still settle, and the CLI still closes before NAS admission. Completed preparation is not reused after scope replacement. Run creation records its idempotency reference and fixed plan in local metadata before sending the request. A reloaded provider restores the current tenant’s report; an explicit `resume(runId, interruptedAt)` continues an eligible persisted run. Hosting cleanup stages log `EVALUATION_HOSTING_CLEANUP` at warn level; ordinary preview and cancellation stages use `EVALUATION_LIFECYCLE` at debug level. Fixed scalar fields retain phase, status and elapsed milliseconds through the default Host sanitizer; safe code/cause metadata omits identity, request and customer values.
+
+<a id="dev-note"></a>
+## Dev Note
+
+Source checks (`typecheck` and `test:source`) resolve public product imports through `product/tsconfig.base.json` and do not require product `lib/` output. `test:built` runs the installed merchant CLI smoke; it requires the official dependencies and the cs-hosting artifacts produced by `pnpm --dir product run build:cs-hosting`. The `test` command runs both test entries. Simulation timestamp changes require the source tests `tests/runner.spec.ts` and `tests/im-purchase-intent.spec.ts` alongside the built `tests/im-runner.spec.ts`; these cover message/tool time, persisted admission joins and server-compatible purchase-intent records.
+
+Ownership replay fixtures publish complete JSON reports by atomic replacement so the SDK parent can poll without observing partial writes.
+
+The Node Host projection test and the customer-service Client workbench test share a client-safe expected case in `tests/support/evaluation-view.ts`. The Host test verifies the full projection against real service output; the localized Client test consumes only that data and public view types. Host fixtures that initialize evaluation services, IM storage or Node built-ins stay in the Node test program.
+
+The [framework decision](../../.agents/notes/proposed/architecture/2026-10-18-agent-evaluation-runner.md) records ownership and rejected alternatives.